The Gaussian Sequence Model and Pinsker’s Theorem

The Gaussian sequence model separates an estimation problem into noisy coordinates. Keeping more coordinates reduces approximation bias but adds variance; shrinking the coordinates gives a finer compromise. We solve the linear minimax problem over an ellipsoid and then prove that the resulting risk is asymptotically optimal among all estimators. The Gaussian-prior lower bound requires an explicit estimate of the mass outside the ellipsoid.

1. Notations

Use the real trigonometric orthonormal basis of \(L^2[0,1]\): \(\varphi_1=1\), \(\varphi_{2m}(t)=\sqrt2\cos(2\pi mt)\), and \(\varphi_{2m+1}(t)=\sqrt2\sin(2\pi mt)\). For an integer \(\beta\ge1\) and \(L>0\), put

\[Q=L^2/\pi^{2\beta},\qquad a_1=0,\quad a_{2m}=a_{2m+1}=(2m)^\beta,\qquad \Theta(\beta,Q)=\{\theta\in\ell^2:\sum_j a_j^2\theta_j^2\le Q\}.\]

Define \(\widetilde W(\beta,L)=\{f\in L^2[0,1]:(\langle f,\varphi_j\rangle)_j\in\Theta(\beta,Q)\}\). This definition, rather than an unspecified convention for Sobolev norms, fixes the parameter class and sharp constant below. The constant coefficient is unrestricted; its estimation contributes only \(\varepsilon^2\) to risk.

Observe \(dY(t)=f(t)dt+\varepsilon dW(t)\), with \(f\in L^2[0,1]\) and \(0<\varepsilon<1\). The coordinates are

\[y_j=\int_0^1\varphi_j(t)dY(t)=\theta_j+\varepsilon\xi_j,\qquad \theta_j=\langle f,\varphi_j\rangle.\]

The Wiener integrals \(\xi_j\) are jointly Gaussian with covariance \(\int\varphi_i\varphi_j=\mathbf1\{i=j\}\), and hence are independent standard normals. The coordinate sequence contains the same statistical information as the path, up to completion of sigma-fields: expanding \(\mathbf1_{[0,t]}\) in the basis reconstructs \(Y(t)\) in mean square, and a subsequence reconstructs it almost surely for each rational \(t\). Path continuity then gives all \(t\). Conversely, the integrals are measurable functions of the path, obtained as limits of deterministic step integrals.

The weighted projection estimator is \(\widehat f_\varepsilon=\sum_j\ell_j^*y_j\varphi_j\), where

\[\ell_j^*=(1-\kappa^*a_j)_+,\qquad \kappa^*=\left(\frac{\beta}{(2\beta+1)(\beta+1)Q}\right)^{\beta/(2\beta+1)}\varepsilon^{2\beta/(2\beta+1)}.\tag{1}\]

2. Linear Minimax Theorem

First allow general nonnegative \(a_j\), and write \(\Theta=\{\theta\in\ell^2:\sum_j a_j^2\theta_j^2\le Q\}\), with \(Q>0\). For the diagonal linear estimator \(\widehat\theta_j=\lambda_jy_j\), Parseval and Tonelli give

\[R(\lambda,\theta)=\mathbb E_\theta\|\widehat\theta-\theta\|^2=\sum_j\{(1-\lambda_j)^2\theta_j^2+\varepsilon^2\lambda_j^2\}.\]

The infimum over square-summable deterministic weights defines \(R^L=\inf_\lambda\sup_{\theta\in\Theta}R(\lambda,\theta)\). Completing the square in each coordinate gives the oracle identity

\[\inf_\lambda R(\lambda,\theta)=\sum_j\frac{\varepsilon^2\theta_j^2}{\varepsilon^2+\theta_j^2}.\]

Indeed the minimizer is \(\lambda_j=\theta_j^2/(\varepsilon^2+\theta_j^2)\); it is square-summable because \(\sum_j\lambda_j^2\le\varepsilon^{-2}\sum_j\theta_j^2\). Define \(\kappa>0\) by

\[\frac{\varepsilon^2}{\kappa}\sum_j a_j(1-\kappa a_j)_+=Q.\tag{2}\]

Lemma 1 (Existence and active coordinates). If \(a_j\) is nondecreasing and tends to infinity, (2) has a unique solution. With

\[N=\max\{j:\varepsilon^2\sum_{m\le j}a_m(a_j-a_m)<Q\},\]

it is \(\kappa=\varepsilon^2\sum_{m\le N}a_m/(Q+\varepsilon^2\sum_{m\le N}a_m^2)\), and \(a_j<1/\kappa\) exactly for \(j\le N\).

Proof. The left side of (2), viewed as \(F(t)=\varepsilon^2\sum_j a_j(1/t-a_j)_+\), has finitely many nonzero summands on every compact subinterval of \((0,\infty)\), so it is continuous. It is strictly decreasing wherever it is positive. A fixed positive \(a_j\) shows \(F(t)\to\infty\) as \(t\downarrow0\), and \(F(t)=0\) once \(t\ge1/\min\{a_j:a_j>0\}\). This proves existence and uniqueness.

Let \(S_j=\varepsilon^2\sum_{m\le j}a_m(a_j-a_m)\). The sequence is nondecreasing and diverges, since any fixed positive \(a_m\) contributes \(\varepsilon^2a_m(a_j-a_m)\to\infty\). Also \(S_1=0\), so \(N\) is well defined and finite. Write \(A=\varepsilon^2\sum_{m\le N}a_m\) and \(B=\varepsilon^2\sum_{m\le N}a_m^2\). Then \(S_N=Aa_N-B<Q\), whereas \(S_{N+1}=Aa_{N+1}-B\ge Q\). Thus \(Aa_N<Q+B\le Aa_{N+1}\), proving the asserted active set for \(\kappa=A/(Q+B)\). Substitution into (2) finishes the proof. Repeated values of \(a_j\) and coordinates with \(a_j=0\) cause no problem. ◻

Theorem 2 (Linear minimax theorem). Suppose there are finitely many zero \(a_j\), a solution \(\kappa>0\) of (2) exists, and \(D^*=\varepsilon^2\sum_j\ell_j<\infty\), where \(\ell_j=(1-\kappa a_j)_+\). Then

\[\inf_\lambda\sup_{\theta\in\Theta}R(\lambda,\theta)=\sup_{\theta\in\Theta}\inf_\lambda R(\lambda,\theta)=\sup_{\theta\in\Theta}R(\ell,\theta)=D^*.\]

Proof. The elementary minimax inequality has the direction

\[\sup_\theta\inf_\lambda R(\lambda,\theta)\le\inf_\lambda\sup_\theta R(\lambda,\theta)\le\sup_\theta R(\ell,\theta).\]

We prove matching bounds on the two ends. Since \(0\le\ell_j\le1\) and \(\sum_j\ell_j<\infty\), these are admissible square-summable weights.

For the upper bound, \(1-\ell_j=0\) when \(a_j=0\), and \((1-\ell_j)/a_j\le\kappa\) otherwise. Hence

\[R(\ell,\theta)\le\varepsilon^2\sum_j\ell_j^2+Q\kappa^2=\varepsilon^2\sum_j\{\ell_j^2+\kappa a_j\ell_j\}=\varepsilon^2\sum_j\ell_j=D^*.\]

The middle equality is (2) multiplied by \(\kappa^2\). In the last equality, either \(\ell_j=0\), or \(\ell_j+\kappa a_j=1\).

For the lower bound, set \(v_j^2=\varepsilon^2\ell_j/(\kappa a_j)\) on positive-weight coordinates \(a_j>0\). Their weighted squared sum is \(Q\) by (2). To avoid any summability issue, first use these coordinates only on a finite set and set all other positive-weight coordinates to zero. On the finitely many zero-weight coordinates, set \(v_j=M\). The resulting finite vector lies in \(\Theta\). Its oracle risk is

\[\sum_{j:a_j=0}\frac{\varepsilon^2M^2}{\varepsilon^2+M^2}+\varepsilon^2\sum_{j\text{ in the finite positive set}}\ell_j.\]

Let the finite sets exhaust the positive coordinates and then let \(M\to\infty\). Monotone convergence gives \(D^*\), proving the lower bound. The supremum need not be attained on the unbounded zero-weight coordinates. ◻

3. Pinsker’s Theorem

Theorem 3 (Pinsker). For the trigonometric weights in Section 1, put \(r=4\beta/(2\beta+1)\) and

\[C^*=[Q(2\beta+1)]^{1/(2\beta+1)}\left(\frac\beta{\beta+1}\right)^{2\beta/(2\beta+1)}.\]

Then the estimator in (1) satisfies

\[\lim_{\varepsilon\downarrow0}\varepsilon^{-r}\sup_{f\in\widetilde W(\beta,L)}\mathbb E_f\|\widehat f_\varepsilon-f\|_2^2=\lim_{\varepsilon\downarrow0}\varepsilon^{-r}\inf_{T_\varepsilon}\sup_{f\in\widetilde W(\beta,L)}\mathbb E_f\|T_\varepsilon-f\|_2^2=C^*.\]

The infimum is over all measurable estimators based on the observed path, with squared \(L^2\) loss.

Proof, upper bound and constant. For a continuous function \(g\) on \([0,1]\), the paired weights satisfy the Riemann-sum relation

\[2\sum_{m\ge1}g(2m\kappa^{1/\beta})\mathbf1\{2m\kappa^{1/\beta}\le1\}=\kappa^{-1/\beta}\left(\int_0^1g(x)dx+o(1)\right).\]

The endpoint term can be included or excluded when \(g(1)=0\), as in all applications below. Equation (2) and \(g(x)=x^\beta(1-x^\beta)\) yield

\[Q=\varepsilon^2\kappa^{-2-1/\beta}(A+o(1)),\qquad A=\int_0^1x^\beta(1-x^\beta)dx=\frac\beta{(\beta+1)(2\beta+1)}.\]

Here \(\kappa\to0\): if it were bounded below, the left side of (2) would tend to zero. Solving gives \(\kappa\sim\kappa^*\). Likewise

\[D^*=\varepsilon^2\left[1+\kappa^{-1/\beta}\left(\frac\beta{\beta+1}+o(1)\right)\right]\sim C^*\varepsilon^r.\]

The constant-coordinate term \(\varepsilon^2=o(\varepsilon^r)\). For the explicit weights \(\ell^*\), the squared-bias and variance bound is

\[\sup_\theta R(\ell^*,\theta)\le Q(\kappa^*)^2+\varepsilon^2\left[1+(\kappa^*)^{-1/\beta}(B+o(1))\right],\quad B=\int_0^1(1-x^\beta)^2dx=2\beta A.\]

Using \(Q=\varepsilon^2(\kappa^*)^{-2-1/\beta}A\), its leading coefficient is \(A+B=(2\beta+1)A=\beta/(\beta+1)\). Thus this upper bound is \(C^*\varepsilon^r(1+o(1))\).

Scalar Gaussian Bayes calculation. If \(x=a+\varepsilon\xi\), with independent \(a\sim \mathcal{N}(0,s^2)\) and \(\xi\sim \mathcal{N}(0,1)\), completing the square in the joint density gives

\[a\mid x\sim N\left(\frac{s^2}{s^2+\varepsilon^2}x,\frac{s^2\varepsilon^2}{s^2+\varepsilon^2}\right).\]

For every estimator \(d(x)\), conditional orthogonality gives

\[\mathbb E(d(x)-a)^2=\mathbb E(d(x)-\mathbb E[a\mid x])^2+\mathbb E\operatorname{Var}(a\mid x).\]

The cross term is zero after conditioning on \(x\). Consequently the posterior mean minimizes Bayes risk, and that risk is \(s^2\varepsilon^2/(s^2+\varepsilon^2)\).

Finite-dimensional reduction. Let \(N=\max\{j:a_j<1/\kappa\}\), fix \(\theta_1=0\) and \(\theta_j=0\) for \(j>N\), and let

\[\Theta_N=\{(\theta_2,\ldots,\theta_N):\sum_{j=2}^N a_j^2\theta_j^2\le Q\}.\]

This is a submodel of the original class. For any \(L^2\)-valued estimator, first take its coefficients \(2,\ldots,N\); discarding the other squared errors can only reduce loss. Then project this finite vector orthogonally onto the closed convex ellipsoid \(\Theta_N\). If \(\Pi(z)\) is that projection, the minimizing property gives \(\langle z-\Pi(z),\theta-\Pi(z)\rangle\le0\) for every \(\theta\in\Theta_N\), and expansion of \(\|z-\theta\|^2\) shows \(\|\Pi(z)-\theta\|\le\|z-\theta\|\). The projection is continuous and hence measurable. We may therefore restrict the finite problem to estimators taking values in \(\Theta_N\).

Under this finite submodel, all observed coordinates outside \(2,\ldots,N\) are independent pure noise. Conditioning an estimator on \((y_2,\ldots,y_N)\) averages over a distribution independent of \(\theta\), so conditional Jensen cannot increase its risk; convexity preserves the constraint \(\Theta_N\). This also follows directly in the path model: its orthogonal Gaussian residual is independent of these coordinates and parameter-free. Its finite-dimensional likelihood ratio is

\[\exp\left\{\varepsilon^{-2}\sum_{j=2}^N\theta_jy_j-(2\varepsilon^2)^{-1}\sum_{j=2}^N\theta_j^2\right\}.\]

Thus no unrestricted path-based estimator can escape the finite Gaussian Bayes lower bound.

Prior and Bayes risk. Fix \(0<\delta<1/2\) and independently draw \(\theta_j\sim \mathcal{N}(0,s_j^2)\), \(j=2,\ldots,N\), with

\[v_j^2=\frac{\varepsilon^2(1-\kappa a_j)}{\kappa a_j},\qquad s_j^2=(1-\delta)v_j^2.\]

Write \(\mu\) for this prior. It is not supported entirely on \(\Theta_N\), so an outside-mass correction is essential. Its full Bayes risk factorizes by coordinates, giving exactly

\[B^*=\sum_{j=2}^N\frac{\varepsilon^2s_j^2}{\varepsilon^2+s_j^2}\ge(1-\delta)\sum_{j=2}^N\frac{\varepsilon^2v_j^2}{\varepsilon^2+v_j^2}=(1-\delta)(D^*-\varepsilon^2).\]

For any \(\Theta_N\)-valued estimator \(d\), its worst-case risk on \(\Theta_N\) is at least its risk integrated over that set against \(\mu\) (the restricted mass is at most one). Subtract the integral over its complement from its full integrated risk. Taking infima and using a uniform bound on the complement gives

\[R_\varepsilon^*\ge B^*-r^*,\qquad r^*=\sup_{d:\ d(y)\in\Theta_N}\int_{\Theta_N^c}\mathbb E_\theta\|d(y)-\theta\|^2\,d\mu(\theta).\]

Notice that this supremum is over estimators, not over the integration variable \(\theta\).

Controlling the outside contribution. Let \(p_\varepsilon=\mu(\Theta_N^c)\). Every constrained estimate has squared norm at most \(Q/a_2^2\), and

\[\mathbb E_\mu\|\theta\|^4=3\sum_js_j^4+\sum_{i\ne j}s_i^2s_j^2\le3\left(\sum_js_j^2\right)^2\le3Q^2/a_2^4.\]

Indeed \(\sum_ja_j^2v_j^2=Q\) by (2), so \(\sum_ja_j^2s_j^2=(1-\delta)Q\). The inequality \(\|d-\theta\|^2\le2\|d\|^2+2\|\theta\|^2\) and Cauchy–Schwarz therefore imply

\[r^*\le\frac{2Q}{a_2^2}\{p_\varepsilon+\sqrt{3p_\varepsilon}\}\le\frac{6Q}{a_2^2}\sqrt{p_\varepsilon}.\]

Put \(b_j^2=a_j^2s_j^2\), \(S=\sum_jb_j^2=(1-\delta)Q\), and \(b_{\max}^2=\max_jb_j^2\). Under the prior,

\[p_\varepsilon=P\left\{\sum_{j=2}^Nb_j^2(\xi_j^2-1)>\delta Q\right\}.\]

Remark 4 below, with \(t=\delta/(1-\delta)<1\), bounds this by \(\exp\{-t^2S/(8b_{\max}^2)\}\). For every active coordinate, \(u=\kappa a_j\in(0,1)\), and hence

\[b_j^2=(1-\delta)\frac{\varepsilon^2}{\kappa^2}u(1-u)\le\frac{(1-\delta)\varepsilon^2}{4\kappa^2}=O(\varepsilon^{2/(2\beta+1)}).\]

For fixed \(\delta\), it follows that \(p_\varepsilon\le\exp\{-c_\delta\varepsilon^{-2/(2\beta+1)}\}\). This decays faster than every power of \(\varepsilon\), so \(r^*=o(\varepsilon^r)\). We have proved \(\liminf\varepsilon^{-r}R_\varepsilon^*\ge(1-\delta)C^*\). First take \(\varepsilon\downarrow0\), then \(\delta\downarrow0\), and combine with the upper bound. Since the minimax risk is no larger than the risk of \(\widehat f_\varepsilon\), both limits in the theorem follow. ◻

Remark 4 (Weighted chi-square tail bound). Let \(Z_j=b_j^2(\xi_j^2-1)\), with independent standard normal \(\xi_j\), \(S=\sum_jb_j^2>0\), and \(b_{\max}^2=\max_jb_j^2\). For \(0<t<1\),

\[P\{\sum_jZ_j>tS\}\le\exp\{-t^2S/(8b_{\max}^2)\}.\]

Indeed Gaussian integration gives \(\mathbb Ee^{\gamma Z_j}=e^{-\gamma b_j^2}(1-2\gamma b_j^2)^{-1/2}\). If \(0\le u\le1/4\), then

\[-u-\frac12\log(1-2u)=\sum_{k\ge2}\frac{2^{k-1}}k u^k\le\sum_{k\ge2}2^{k-2}u^k=\frac{u^2}{1-2u}\le2u^2.\]

Thus Markov and independence yield

\[P\{\sum_jZ_j>x\}\le\exp\{-\gamma x+2\gamma^2\sum_jb_j^4\}\le\exp\{-\gamma x+2\gamma^2b_{\max}^2S\}.\]

Choose \(x=tS\) and \(\gamma=t/(4b_{\max}^2)\); then each \(\gamma b_j^2<1/4\), and substitution gives the asserted exponent. This is why the lower-bound proof fixes \(\delta<1/2\).