Local Asymptotic Normality

How much can a sample tell us about two nearby parameter values? If their separation is comparable to the sampling error, neither value can be discarded simply by taking a large sample. Local asymptotic normality describes the likelihood at this scale. The contiguity note motivates that scale with a calibration example and gives a full square-root-likelihood proof of QMD-to-LAN. The calculations below show how the score determines its random part and Fisher information determines its curvature. When using LAN to study an estimator, distinguish a fixed-direction expansion from the compact-uniform expansion required by the argmax theorem.

We want to describe the likelihood near a fixed parameter \(\theta\). The relevant alternatives are \(\theta+h/\sqrt n\), where \(h\) remains bounded as the sample size grows. Write \(p_\theta\) for a density relative to a common dominating measure \(\mu\), and let \(\ell_\theta(x)=\log p_\theta(x)\). At this scale, summing the small changes in log likelihood produces two terms of comparable size: a centered random sum and a deterministic quadratic correction. The expansion below makes this balance explicit.

In a one-dimensional smooth model, a Taylor expansion suggests

\[\ell_{\theta+t}(x)-\ell_\theta(x) =t\dot\ell_\theta(x)+\frac{1}{2}t^2\ddot\ell_\theta(x)+o(t^2).\]

Under the usual conditions for differentiating the integral of the density, \(E_\theta\dot\ell_\theta(X)=0\) and \(-E_\theta\ddot\ell_\theta(X)=I_\theta=E_\theta\dot\ell_\theta(X)^2\). Taking \(t=h/\sqrt n\) and summing over the observations gives the heuristic expansion

\[\sum_{i=1}^n\log\frac{p_{\theta+h/\sqrt n}(X_i)}{p_\theta(X_i)} =h\Delta_{n,\theta}-\frac{1}{2} h^2I_\theta+o_{P_\theta}(1), \qquad \Delta_{n,\theta}=n^{-1/2}\sum_{i=1}^n\dot\ell_\theta(X_i).\]

The formal calculation leaves a question about the remainder. There are more observations as the local displacement shrinks, so the individual Taylor errors must remain small after summation. A pointwise Taylor expansion alone does not establish this.

The central limit theorem gives \(\Delta_{n,\theta}\Rightarrow \mathcal{N}(0,I_\theta)\) when the score is square-integrable. Thus the local log likelihood has a Gaussian linear term and a quadratic drift. Differentiability in quadratic mean gives a way to prove this without requiring a pointwise second derivative of the log density.

The square root of a density belongs to a Hilbert space, where differentiability controls the integrated error directly. We therefore impose a smoothness condition on the square root of the density. The score below is denoted \(\dot\ell_\theta\).

Assumption 1. \(\theta \mapsto \sqrt{\mathbb{P}_{\theta}}\) is differentiable in quadratic mean, i.e.

\[\begin{aligned} \int \left( \sqrt{\mathbb{P}_{\theta+h}} - \sqrt{\mathbb{P}_{\theta}} - \frac{1}{2} h^{\intercal} \dot{\ell}_{\theta} \sqrt{\mathbb{P}_{\theta}} \right)^2 \dif \mu = o(\norm{h}^2) \end{aligned}\]

Theorem 2. Let \(\Theta\subset\mathbb R^k\) be open, and suppose the model with densities \(p_\theta\) is differentiable in quadratic mean at \(\theta\), with square-integrable score \(\dot\ell_\theta\). Then \(E_\theta\dot\ell_\theta=0\) and \(I_\theta=E_\theta[\dot\ell_\theta\dot\ell_\theta^\top]\) is finite. If \(h_n\to h\), then for i.i.d. observations from \(P_\theta\),

\[\log\prod_{i=1}^n\frac{p_{\theta+h_n/\sqrt n}(X_i)}{p_\theta(X_i)} =\frac1{\sqrt n}\sum_{i=1}^n h^\top\dot\ell_\theta(X_i)-\frac12h^\top I_\theta h+o_{P_\theta}(1).\]

Proof. Fix \(h_n\to h\), put \(p_n=p_{\theta+h_n/\sqrt n}\), \(p=p_\theta\), and \(g=h^\top\dot\ell_\theta\). Differentiability in quadratic mean gives

\[\sqrt n(\sqrt{p_n}-\sqrt p)\longrightarrow\frac{1}{2}g\sqrt p \quad\text{in }L^2(\mu).\]

Since \(\int(p_n-p)\,d\mu=0\), taking the inner product with \(\sqrt{p_n}+\sqrt p\) shows that \(E_\theta g=0\). Applying this to the coordinate directions gives the mean-zero score. Define, on \(\{p>0\}\),

\[ W_{n,i}=2\left(\sqrt{p_n/p}(X_i)-1\right).\tag{1}\]

The quadratic-mean condition implies \(E_\theta(\sqrt nW_{n,1}-g(X_1))^2\to0\), and \(n\int_{\{p=0\}}p_n\,d\mu\to0\). Normalization of \(p_n\) therefore gives

\[E_\theta W_{n,1}+\frac{1}{4}E_\theta W_{n,1}^2 =-\int_{\{p=0\}}p_n\,d\mu, \qquad nE_\theta W_{n,1}\to-\frac{1}{4}E_\theta g^2.\]

The next step separates the mean of the square-root increment from its fluctuation. Independence and the \(L^2\) approximation show that the centered difference between \(\sum_iW_{n,i}\) and \(n^{-1/2}\sum_i g(X_i)\) has variance tending to zero. Thus

\[\sum_iW_{n,i}=n^{-1/2}\sum_i g(X_i)-\frac{1}{4}E_\theta g^2+o_P(1).\]

To see the variance bound explicitly, put \(A_{n,i}=\sqrt nW_{n,i}-g(X_i)\). The variables within each row are independent and identically distributed, so

\[\operatorname{Var}\!\left\{\sum_{i=1}^nW_{n,i}-n^{-1/2}\sum_{i=1}^ng(X_i)\right\}=\operatorname{Var}(A_{n,1})\le E A_{n,1}^{2}\longrightarrow0.\]

Its mean is \(nEW_{n,1}\), because \(Eg=0\). Combining the mean limit with Chebyshev's inequality gives the preceding expansion. The contribution outside the support of \(p\) must be retained in the normalization identity: on \(\{p=0\}\) the QMD approximation term vanishes, so QMD itself bounds \(\int_{\{p=0\}}p_n\,d\mu=o(n^{-1})\).

Moreover, \(\sum_i(W_{n,i}-g(X_i)/\sqrt n)^2\to_P0\). Cauchy–Schwarz and the law of large numbers imply \(\sum_iW_{n,i}^2\to_P E_\theta g^2\). The \(L^2\) approximation also makes \((\sqrt nW_{n,1})^2\) uniformly integrable, so

\[P\!\left(\max_i|W_{n,i}|>\varepsilon\right) \leqslant\varepsilon^{-2}nE[W_{n,1}^2\mathbf1\{|W_{n,1}|>\varepsilon\}]\to0.\]

Here is the squared-sum calculation. Markov's inequality gives \(n^{-1}\sum_i A_{n,i}^2\to_P0\), while \(n^{-1}\sum_i g(X_i)^2\to_P Eg^2\). Hence

\[\left|\sum_iW_{n,i}^2-\frac1n\sum_i g(X_i)^2\right|\le2\left(\frac1n\sum_i g(X_i)^2\right)^{1/2}\left(\frac1n\sum_i A_{n,i}^2\right)^{1/2}+\frac1n\sum_i A_{n,i}^2\to_P0.\]

Also, \(\sqrt nW_{n,1}\to g(X_1)\) in \(L^2\) implies convergence of their squares in \(L^1\), by Cauchy–Schwarz. The squares are therefore uniformly integrable. In the maximum bound above the truncation level is \(\varepsilon^2 n\), which tends to infinity; uniform integrability makes that tail expectation vanish.

The maximum bound ensures that every observed increment lies in the region where the logarithm has a uniform Taylor expansion, with probability tending to one. We can consequently apply \(2\log(1+w/2)=w-w^2/4+o(w^2)\) uniformly over the observations. Summing gives

\[\log\prod_i\frac{p_n(X_i)}{p(X_i)} =\sum_iW_{n,i}-\frac{1}{4}\sum_iW_{n,i}^2+o_P(1) =n^{-1/2}\sum_i g(X_i)-\frac{1}{2}E_\theta g^2+o_P(1).\]

More explicitly, define \(r(w)=[2\log(1+w/2)-w+w^2/4]/w^2\) for \(w\ne0\), with \(r(0)=0\). On \(|w|\le1\), this is continuous and \(r(w)\to0\) as \(w\to0\). With probability tending to one all \(|W_{n,i}|\le1\), so all likelihood ratios at the observed sample points are positive and

\[\left|\sum_iW_{n,i}^2r(W_{n,i})\right|\le\sup_{|w|\le\max_i|W_{n,i}|}|r(w)|\sum_iW_{n,i}^2=o_P(1)O_P(1)=o_P(1).\]

This justifies summing the Taylor errors; pointwise small errors without the maximum bound would not suffice.

Since \(E_\theta g^2=h^\top I_\theta h\), this is the desired expansion. ◻