Regular Estimators, Limit Experiments, and Asymptotic Efficiency
Consider a parametric experiment \((P_\theta^n)\) near a fixed \(\theta\). Local alternatives have the form \(\theta+h/\sqrt n\); \(I_\theta\) is the Fisher information. The likelihood expansion is proved in local asymptotic normality, and the change-of-measure argument uses contiguity.
A likelihood expansion tells us what a sample can distinguish locally. We now ask what this implies for estimation. Suppose we want to estimate a parameter \(\varphi(\theta)\) based on observations from a model governed by the parameter \(\theta\). We seek a lower bound on the asymptotic error of an estimator sequence \((T_n)\) for \(\varphi(\theta)\).
To simplify the situation, we investigate the case where the sequence \(\sqrt{n} (T_n - \varphi(\theta))\) converges in distribution under every possible value of \(\theta\). This reduces the comparison to limit distributions, but pointwise limits can hide poor behavior at nearby parameter values. In analogy with the Cramér-Rao theorem, a “best” limiting distribution is referred to as an asymptotic lower bound. Under certain restrictions the normal distribution with mean zero and covariance the inverse Fisher information is an asymptotic lower bound for estimating \(\varphi(\theta) = \theta\) in a smooth parametric model. The qualification is essential: the next example improves the limit at a single parameter by changing the estimator on a shrinking neighborhood.
A loss function gives one way to compare limiting errors. For instance, a limiting distribution is “good” if discrepancies of the type \(\int \ell \dif L_{\theta}\) are small (where \(\ell\) is some loss function, and \(L_{\theta}\) is the limiting distribution). Such a comparison depends on the loss. A distributional lower bound can support several loss comparisons at once, under the appropriate assumptions.
Example 1 (Hodges' estimator). Suppose \(\sqrt n(T_n-\theta)\Rightarrow L_\theta\) at every fixed real \(\theta\), and set
For \(\theta\ne0\), consistency gives \(\Pr_\theta(S_n=T_n)\to1\), so \(\sqrt n(S_n-\theta)\Rightarrow L_\theta\). At \(\theta=0\), tightness of \(\sqrt nT_n\) gives \(\Pr_0(|T_n|\ge n^{-1/4})\to0\). Consequently, for every deterministic scalar sequence \(r_n\), \(r_nS_n\to_P0\), since the probability that it is nonzero tends to zero.
The improvement at zero can come at a large cost nearby. In the normal mean model \(X_i\sim\mathcal{N}(\theta,1)\), take \(T_n=\overline X_n\) and \(\theta_n=\frac12n^{-1/4}\). The sampling error is \(O_P(n^{-1/2})\), so \(\Pr_{\theta_n}(S_n=0)\to1\), and
Pointwise limits therefore conceal deterioration at parameter values that approach zero with the sample size.
For two asymptotically normal procedures, relative efficiency compares the sample sizes needed for the same limiting precision. Suppose, at a fixed parameter,
Choose deterministic sample sizes \(n_{k,j}\) satisfying \(n_{k,j}/k\to\sigma_j^2(\theta)\). Then both procedures have the same precision on the common scale:
Definition 2. The asymptotic relative efficiency of procedure 1 with respect to procedure 2 is the ratio of the sample size required by procedure 2 to that required by procedure 1:
Under this convention, a value greater than one favors procedure 1. The ratio generally depends on the parameter.
1. Statistical Experiments
Clearly, by Example 1, it is not enough to consider pointwise behaviour of statistical experiments, even for every \(\theta\). The first step we can take is to describe the local behaviour of estimators for some fixed \(\theta\), i.e. vary only by small “distances” away from the fixed parameter. The displacement will have the same order as the estimator’s usual sampling error.
For fixed \(\theta\), let us look at \(\theta + \frac{h}{\sqrt{n}}\) for \(h\) ranging over \(\mathop{\mathrm{\mathbf{R}}}^k\). Suppose that for certain limiting distributions \(L_{\theta,h}\),
Assume differentiability of the map \(\theta \mapsto \varphi(\theta)\), and fix the definition \(\varphi: \Theta \to \mathop{\mathrm{\mathbf{R}}}^m\) for some open subset \(\Theta \subset \mathop{\mathrm{\mathbf{R}}}^k\).
Theorem 3 (Lower Bound). Let the collection of experiments \((P_{\theta} : \theta \in \Theta)\) be differentiable in quadratic mean, i.e. Definition 1. At the fixed parameter \(\theta\), assume the Fisher information matrix \(I_{\theta}\) is invertible, and \(\varphi\) is differentiable at \(\theta\). Suppose (4) holds for all \(h \in \mathop{\mathrm{\mathbf{R}}}^k\). Then, there exists a randomized statistic \(T\) in the experiments \((\mathcal{N}(h,I_{\theta}^{-1}): h\in\mathbb R^k)\) such that \((T-\varphi'_{\theta} h)\) has distribution \(L_{\theta,h}\) for all \(h \in \mathop{\mathrm{\mathbf{R}}}^k\).
Proof. Write \(A=\varphi'_\theta\), \(P_{n,h}=P_{\theta+h/\sqrt n}^n\), \(Z_n=\sqrt n(T_n-\varphi(\theta))\), and \(\Delta_n=n^{-1/2}\sum_{i=1}^n\ell'_\theta(X_i)\). Under \(P_{n,0}\), both \(Z_n\) and \(\Delta_n\) are tight. Every joint convergent subsequence has a limit \((S,\Delta)\), with \(\Delta\sim\mathcal{N}(0,I_\theta)\). LAN gives, for every fixed \(h\),
The limiting likelihood ratio has expectation one. Le Cam's third lemma therefore gives the limit law of \(Z_n\) under \(P_{n,h}\) as the law of \(S\) under the tilted joint measure with density \(\exp(h^\top\Delta-\frac12h^\top I_\theta h)\). By differentiability, this is the translate of \(L_{\theta,h}\) by \(Ah\).
Let \(K(\delta,\cdot)\) be the conditional law of \(S\) given \(\Delta=\delta\); such a probability kernel exists because both spaces are Euclidean. In the Gaussian experiment observe \(X\sim\mathcal{N}(h,I_\theta^{-1})\), and draw \(T\) conditionally from \(K(I_\theta X,\cdot)\). Under \(h=0\), the pair \((T,I_\theta X)\) has the law of \((S,\Delta)\). Under general \(h\), its likelihood ratio relative to the central experiment is
This tilts only the observed coordinate and leaves the conditional kernel unchanged. Thus \(T-Ah\) has law \(L_{\theta,h}\) for every \(h\). The same joint subsequence supplies one kernel for all fixed alternatives; no separate choice of a kernel for each \(h\) is made. This is also the implication stated in van der Vaart, Theorem 8.3. □
Definition 4. A sequence of estimators \((T_n)\) is regular at \(\theta \in \Theta\) for estimating \(\varphi(\theta)\) if
for all \(h \in \mathop{\mathrm{\mathbf{R}}}^k\).
Because the right-hand side of the Definition 4 is independent of \(h\), it must be that \(\theta \in \Theta\) that is regular determines the limiting distribution. By the proof of Theorem 3, we see that every estimator sequence corresponds to a limiting normal experiment (i.e. some \(T\) in \((\mathcal{N}(h,I_{\theta}^{-1}) : h\in\mathbb R^k)\)). Therefore, a regular estimator sequence is matched by an equivariant-in-law estimator for \(\varphi'_{\theta} h\). An alternative name for a regular estimator sequence is asymptotic equivariant-in-law.
Theorem 5. Let \((P_{\theta} : \theta \in \Theta)\) be differentiable in quadratic mean (i.e. Definition 1) at a point \(\theta\), where \(I_{\theta}\) is invertible. Let \(\varphi\) be differentiable at \(\theta\). Let \(T_n\) be a regular estimator sequence in the experiments \((P^n_{\theta} : \theta \in \Theta)\) at \(\theta\), with limiting distribution \(L_{\theta}\). Then, there exists a probability measure \(M_{\theta}\) such that \(L_{\theta} = \mathcal{N}(0, \varphi'_{\theta} I_{\theta}^{-1} {\varphi'_{\theta}}^\intercal) \ast M_{\theta}\). If \(L_{\theta}\) has covariance matrix \(\Sigma_{\theta}\), then the matrix \(\Sigma_{\theta} - \varphi'_{\theta} I_{\theta}^{-1} {\varphi'_{\theta}}^\intercal\) is nonnegative-definite.
Proof. Use the joint limit \((S,\Delta)\) constructed above, and put \(A=\varphi'_\theta\), \(I=I_\theta\), \(V=AI^{-1}A^\top\). Regularity and likelihood tilting imply, for every \(t\in\mathbb R^m\) and \(h\in\mathbb R^k\),
The normal marginal has exponential moments of every linear functional. Thus, for fixed \(t\), the map \(z\mapsto E\exp(it^\top S+z^\top\Delta)\) is entire on \(\mathbb C^k\): on each compact set of \(z\)'s, its derivatives are dominated by a polynomial in \(\|\Delta\|\) times \(e^{C\|\Delta\|}\), which is Gaussian-integrable. Fixing all but one coordinate and using the one-variable identity theorem extends the preceding identity from real \(h\) to complex \(z\), one coordinate at a time. In particular,
To compute the law of the residual, substitute \(v=u-I^{-1}A^\top t\). Expanding the quadratic form cancels the two mixed terms and leaves \(-u^\top Iu/2+t^\top AI^{-1}A^\top t/2\).
Gaussian exponential integrability of \(\Delta\) permits analytic continuation in each coordinate of \(h\). Substituting imaginary arguments and then subtracting \(AI^{-1}\Delta\) gives, for \(R=S-AI^{-1}\Delta\),
The bracket is the characteristic function of \(R\), obtained by setting \(u=0\). Thus \(R\) and \(\Delta\) are independent, and \(S=AI^{-1}\Delta+R\). This proves the convolution assertion, including its joint form. If \(S\) has a finite covariance, so does \(R\), and \(\operatorname{Cov}(S)=V+\operatorname{Cov}(R)\). □
Pointwise limits need not be regular at every parameter. The next argument explains why the convolution conclusion nevertheless holds almost everywhere.
Lemma 6 (Regular Subsequence). Let \(T_n\) be estimators in the experiments \((P_{n,\theta} : \theta \in \Theta)\) indexed by open \(\Theta \subset \mathop{\mathrm{\mathbf{R}}}^k\). Assume \(\theta \mapsto P_{n,\theta} (A)\) is measurable for every measurable subset \(A\) and every \(n\). Assume \(\theta \mapsto \varphi(\theta)\) is measurable. Furthermore, suppose there exists distributions \(L_{\theta}\) such that for Lebesgue a.e. \(\theta\), the following holds:
Then, for every sequence \((\tau_n)\) tending to zero, there exists a subsequence \((\tau_{n_k})\) such that for Lebesgue a.e. \((\theta,h)\), along the subsequence,
Proof. Choose a countable family of bounded continuous functions that determines weak convergence. For one such test function \(f\), put \(g_n(\theta)=\mathbb{E}_\theta f(r_n(T_n-\varphi(\theta)))\) and \(g(\theta)=\int f\,dL_\theta\). These functions are measurable and uniformly bounded, and \(g_n\to g\) almost everywhere. Extend both by zero outside the open parameter set.
Let \(V,H\) be independent standard Gaussian vectors, and let \(p_n\) and \(p\) be the densities of \(V+\tau_nH\) and \(V\). Then \(\|p_n-p\|_1\to0\). If \(|g_n|,|g|\le C\),
Approximate \(g\) by a bounded continuous function \(g_\epsilon\) in \(L^1(p)\). The same density comparison and dominated convergence give
Letting the approximation error tend to zero proves \(g_n(V+\tau_nH)\to g(V)\) in \(L^1\). For the first \(j\) members of the countable determining family, choose a common subsequence whose expected errors are summable. Markov's inequality and Borel–Cantelli give simultaneous almost-sure convergence. The joint Gaussian density is everywhere positive, so the exceptional set is also Lebesgue-null. For \(\theta\) in the open parameter set, \(\theta+\tau_nh\) eventually remains inside it. This proves the lemma. □
The common subsequence in Lemma 6 can be chosen explicitly. Normalize the determining functions to be bounded by one, and let \(e_{n,j}\) be the expected absolute error for the \(j\)-th function. After proving \(e_{n,j}\to0\) for each \(j\), choose \(n_k>n_{k-1}\) so that \(\sum_{j\le k}e_{n_k,j}\le2^{-k}\). For each fixed \(j\), the sum of its expected errors over \(k\ge j\) is finite. Tonelli's theorem makes the corresponding sum of nonnegative random errors finite almost surely, and hence those errors tend to zero. Intersecting these probability-one sets over the countable family proves simultaneous weak convergence. Positivity of the Gaussian joint density converts this into the required Lebesgue almost-everywhere statement.
Combining the subsequence lemma with the Gaussian representation yields the almost-everywhere convolution theorem.
Theorem 7. Suppose the model is differentiable in quadratic mean at every point of an open \(\Theta\subset\mathbb R^k\), its Fisher information is nonsingular, and \(\varphi\) is differentiable. Assume the parameterized probabilities are measurable and \(\sqrt n(T_n-\varphi(\theta))\Rightarrow L_\theta\) under every fixed \(\theta\). Then for Lebesgue-almost every \(\theta\),
for some probability measure \(M_\theta\). If \(L_\theta\) has finite covariance, its covariance minus the displayed Gaussian covariance is nonnegative definite.
Proof. Apply Lemma 6 with \(r_n=\sqrt n\) and \(\tau_n=n^{-1/2}\). At almost every \(\theta\), the local centered limit along its subsequence is \(L_\theta\) for almost every \(h\). Extract a further joint limit with the central sequence and use the proof of Theorem 3. For each bounded continuous test function, the expectation in the tilted Gaussian experiment is continuous in \(h\): Gaussian likelihood ratios are continuous in \(L^1\), and the centering \(\varphi'_\theta h\) is continuous. Equality on almost every \(h\) therefore extends to every \(h\). The joint convolution argument of Theorem 5 applies. This is the statement of van der Vaart, Theorem 8.9. □
While all the results above indicate that the most suitable (i.e. best) possible limiting distribution is \(\mathcal{N}(0, \varphi'_{\theta} I_{\theta}^{-1} {\varphi'_{\theta}}^{\intercal})\), it is certainly not the most ideal in every situation. The results above only indicate the “optimal” normal limit of a sequence of statistical experiments, which is just the distribution of the estimator \(\varphi'_{\theta} X\) in the normal limit experiments. There are examples where this is not optimal per se.
Example 8 (Stein’s Shrinkage Estimator). Let \(X_1,\dots,X_n\ \text{are iid }\mathcal{N}_k(\theta,I_k),\ k\ge3\). Let the dimension of the observations be at least \(3\). Consider the estimator
If \(\theta \neq 0\), then \(T_n = \overline{X}_n - (k-2) \frac{\overline{X}_n}{n\norm{\overline{X}_n}^2} =\overline X_n+O_{P_\theta}(n^{-1})\) by the weak law of large numbers. Therefore, \(\sqrt{n} (T_n - \overline{X}_n) \xrightarrow{\rm P} 0\), so \(T_n\) is regular for every \(\theta \neq 0\). For \(\theta = \frac{h}{\sqrt{n}}\), we have that \(\sqrt{n} \overline{X}_n \sim X \sim \mathcal{N}(h,I_k)\). Furthermore, the standardized estimator \(\sqrt{n} (T_n - \frac{h}{\sqrt{n}}) \sim T-h\), with \(T(X) = X - (k-2) \frac{X}{\norm{X}^2}\). This is Stein’s shrinkage estimator.
Note that the distribution of \(T-h\) depends on \(h\), so the sequence \(T_n\) is not regular at \(\theta=0\).
It follows the remarkable fact that, for every \(h \in \mathop{\mathrm{\mathbf{R}}}^k\),
we have local limiting distributions \(L_{0,h}\) of \(\sqrt{n} (T_n - \frac{h}{\sqrt{n}})\) (under \(\theta = \frac{h}{\sqrt{n}}\)) are all better than \(\mathcal{N}(0,I_k)\) limiting distribution of the best regular estimator sequence \(\overline{X}_n\).
The strict risk improvement in Example 8 follows by a direct calculation. For \(X\sim N_k(h,I_k)\), write \(a=k-2\) and \(b(x)=-ax/\|x\|^2\). Gaussian integration by parts gives \(E_h[(X-h)^\top b(X)]=E_h\operatorname{div}b(X)\). Since
expanding the squared error yields
The reciprocal-square moment is finite for \(k>2\): near zero its radial integral is bounded by a constant times \(\int_0^1 r^{k-3}\,dr\), and outside the unit ball it is bounded by one. Integration by parts can first be performed outside a ball of radius \(\varepsilon\); its boundary contribution is \(O(\varepsilon^{k-2})\), which vanishes. The Gaussian tail removes the outer boundary. This proves the strict inequality and explains the dimension restriction.
The comparison must specify both where the risk is measured and which loss is used. The preceding examples raise two restrictions:
For the Gaussian concentration comparison furnished by the convolution theorem, an improvement in the limiting law can occur only on a Lebesgue-null set of fixed parameters, as evidenced by Theorem 7.
Risk comparisons must specify a loss function; the convolution theorem does not order arbitrary losses or finite-sample risks.
In light of these facts, we could ask if, locally, we can find a best-worst case of estimator sequence \(T_n\) for \(\varphi(\theta)\). Therefore, for a given loss function \(\ell\), we have the following definition.
Definition 9. An estimator sequence \((T_n)\) is locally asymptotically minimax for a given loss function \(\ell\) (for estimating \(\varphi(\theta)\)) if the following holds:
Proposition 10. Assume the LAN and differentiability conditions of Theorem 5, with a real-valued target and positive efficient variance \(v=\varphi'_\theta I_\theta^{-1}(\varphi'_\theta)^\top\). Let \(T_n\) be regular. Suppose \(\ell\) is a nonnegative lower-semicontinuous symmetric bowl-shaped loss, nonconstant on \((0,\infty)\), with \(\mathbb{E}\ell(G)<\infty\) for \(G\sim\mathcal{N}(0,v)\). If \(T_n\) satisfies Definition 9 for this loss, then its regular limit is \(\mathcal{N}(0,v)\).
Proof. By Theorem 5 the limit is \(G+R\), with \(G\) and \(R\) independent. The risk condition at the singleton local set \(\{0\}\), together with the portmanteau inequality, gives \(\mathbb{E}\ell(G+R)\le\mathbb{E}\ell(G)\). For every nonzero shift \(r\) and every finite \(a>0\), strict unimodality of the Gaussian density gives \(\Pr(|G+r|\le a)<\Pr(|G|\le a)\). Integrating these strict inequalities over the sublevel sets of the nonconstant bowl-shaped loss yields \(\mathbb{E}\ell(G+r)>\mathbb{E}\ell(G)\). Conditioning on \(R\) therefore forces \(R=0\) almost surely.
The scalar, nondegenerate, nonconstant-loss assumptions matter: a constant loss cannot distinguish efficient from inefficient limits, and the scalar interval argument is not a claim of uniqueness of all multivariate minimax procedures. Conversely, an efficient weak limit gives convergence of these risks only with additional control, such as uniform integrability of the losses under the relevant local alternatives. □
The preceding shrinkage example also warns against identifying pointwise loss improvement with regularity. We next characterize efficient regular estimators by their first-order expansion in the score.
Proposition 11. Let \((P_{\theta} : \theta \in \Theta)\) be differentiable in quadratic mean (i.e. Definition 1) at a point \(\theta\), where \(I_{\theta}\) is invertible. Let \(\varphi\) be differentiable at \(\theta\). Let \(T_n\) be a regular estimator sequence in the experiments \((P^n_{\theta} : \theta \in \Theta)\) at \(\theta\), such that
Then, \(T_n\) is best regular estimator for \(\varphi(\theta)\) at \(\theta\). Conversely, every best regular estimator sequence satisfies this expansion.
Proof. Write \(\Delta_{n,\theta}=n^{-1/2}\sum_{k=1}^n\ell'_\theta(X_k)\). Under \(P_\theta^n\), the central limit theorem gives \(\Delta_{n,\theta}\rightsquigarrow\Delta_\theta\sim \mathcal{N}(0,I_\theta)\), and local asymptotic normality gives
If the stated asymptotically linear expansion holds, the joint limit of the estimator and log likelihood ratio is
Its covariance matrix is
Le Cam’s third lemma shifts the estimator’s mean by \(\varphi'_\theta h\) under the local alternative. Differentiability of \(\varphi\) then gives the centered limit \(\mathcal{N}(0,\varphi'_\theta I_\theta^{-1}(\varphi'_\theta)^\top)\), independently of \(h\). This is the efficient regular limit.
For the converse, use the equality case of the convolution theorem in the joint limit with the central sequence. A regular limit has the representation \(\varphi'_\theta I_\theta^{-1}\Delta_\theta+Z\), where \(Z\) is independent of \(\Delta_\theta\). If the limit is the efficient normal distribution, its characteristic function forces \(Z=0\) almost surely. Applying this to every joint convergent subsequence yields
which is the desired expansion. ◻
As a final remark, a general method that can be used to strengthen consistency of an estimator sequence \(T_n\) is to make \(\mathop{\mathrm{\mathds{P}}}(d(T_n, \varphi(\theta) > \varepsilon)\) as small as possible—and as quickly as possible—under fixed \(\varepsilon> 0\). An equivalent statement is to investigate what the value of this probability is in the worst, rarest case possible. This is (very, very roughly speaking) a large deviation result.
Proposition 12. Suppose \(T_n\) is based on \(n\) i.i.d. observations and is consistent for \(\varphi(\theta)\) under every \(\theta\). For a metric \(d\), every \(\varepsilon>0\), and fixed \(\theta_0\),
where \(D(P\Vert Q)=\int\log(dP/dQ)\,dP\), with value \(+\infty\) when appropriate.
Proof. Fix an alternative in the infimum with finite divergence \(D\). Consistency and the strict separation imply that the event \(A_n=\{d(T_n,\varphi(\theta_0))>\varepsilon\}\) has \(P_\theta^n\)-probability tending to one. Under \(P_\theta^n\), the law of large numbers gives \(n^{-1}\log(dP_\theta^n/dP_{\theta_0}^n)\to D\). Thus, for each \(\eta>0\),
The last probability tends to one. Take logarithms, then let \(\eta\downarrow0\), and take the infimum over alternatives. ◻