Central Limit Theorems and Worked Examples

A central limit theorem needs both a scale and a reason that no small part of the sum dominates that scale. For independent arrays, Lindeberg’s condition controls the variance carried by large summands. For martingales, the corresponding variance is conditional on the past. The examples below show how these conditions enter a permutation statistic and an autoregressive estimator.

Theorem 1 (Lindeberg–Lévy). Let \((X_i)_{i\geq1}\) be i.i.d. with mean \(m\) and variance \(0<\sigma^2<\infty\). With \(S_n=\sum_{i=1}^nX_i\),

\[\frac{1}{\sqrt{n}} \frac{S_n - nm}{\sigma} \xrightarrow{d} \mathcal{N}(0,1) ~.\]

Proof from Theorem 2 below. Apply the independent-array theorem to \(Y_{n,k}=(X_k-m)/(\sigma\sqrt n)\), whose total variance is one. For each \(\varepsilon>0\), its Lindeberg sum is

\[\sum_{k=1}^n\mathbb{E}[Y_{n,k}^2 1_{\{|Y_{n,k}|>\varepsilon\}}]=\frac1{\sigma^2}\mathbb{E}[(X_1-m)^2 1_{\{|X_1-m|>\varepsilon\sigma\sqrt n\}}]\longrightarrow0.\]

Dominated convergence applies because the dominating square has finite expectation. Theorem 2 therefore proves the result. ◻

When the summands change with sample size, the i.i.d. theorem no longer applies directly. A triangular array keeps track of this dependence on sample size. Consider random variables

\[X_{n,1}, \dots, X_{n, r_n} ~,\]

where \(r_n\) changes with \(n\). This is known as a triangular array of random variables (though there are disputes about terminology; for instance, most texts only use the case \(r_n = n\) when defining triangular arrays, and all other choices of \(r_n\) are called double arrays). We require that for each \(n\), the sequence above is a collection of independent random variables. For each \(n\), adopt the following notations:

\[\begin{aligned} \mathbb{E} X_{n,k} &= 0 ~, \\ \sigma_n^2 &= \sum_{1 \leq k \leq r_n} \sigma_{n,k}^2 \triangleq \sum_{1 \leq k \leq r_n} \mathbb{E} X_{n,k}^2 ~. \end{aligned}\]

Assume throughout that the row variances are finite and \(\sigma_n^2>0\). That the mean of each \(X_{n,k}\) is zero is without loss of generality. To establish the asymptotic normality of \(S_n = \sum_{1 \leq k \leq r_n} X_{n,k}\), we need to expand the characteristic function of each \(X_{n,k}\), to the second order term, and estimate the remaining terms. The remainder is controlled by splitting the summands into small values and values large relative to the total standard deviation.

Theorem 2 (Lindeberg). Suppose, for each \(n\), \(X_{n,1},\dots,X_{n,r_n}\) is independent, and satisfies the Lindeberg condition

\[\lim\limits_{n\to\infty} \sum_{1 \leq k \leq r_n} \frac{1}{\sigma_n^2} \mathbb{E} [X_{n,k}^2 \mathbb 1 \{ |X_{n,k}| \geq \varepsilon \sigma_n \}] = 0\]

for any \(\varepsilon > 0\). Then, \(\frac{S_n}{\sigma_n} \xrightarrow{d} \mathcal{N} (0,1)\).

Proof.

Assume \(\sigma_n>0\) and divide the row by \(\sigma_n\) so that its total variance is one. Write \(v_{n,k}=\mathbb{E}X_{n,k}^2\). Taylor’s formula gives

\[|e^{iu}-1-iu+u^2/2|\leq C\min(|u|^2,|u|^3).\]

Consequently, for fixed \(t\) and any \(\varepsilon>0\),

\[\sum_k\left|Ee^{itX_{n,k}}-\left(1-\frac{t^2v_{n,k}}2\right)\right|\leq C_t\left(\varepsilon+\sum_k\mathbb{E}X_{n,k}^21_{\{|X_{n,k}|>\varepsilon\}}\right).\]

Take \(n\to\infty\) first and then \(\varepsilon\downarrow0\). Also \(\max_kv_{n,k}\leq\varepsilon^2+\sum_k\mathbb{E}X_{n,k}^21_{\{|X_{n,k}|>\varepsilon\}}\to0\) in the same order. For large \(n\), the quadratic factors lie in \([0,1]\). Telescoping products bounds the difference between the characteristic function of the row sum and \(\prod_k(1-t^2v_{n,k}/2)\) by the preceding sum of errors. Finally,

\[\sum_k\log(1-t^2v_{n,k}/2)=-t^2/2+O_t\left(\sum_kv_{n,k}^2\right)\longrightarrow-t^2/2,\]

because \(\sum_kv_{n,k}^2\leq\max_kv_{n,k}\). The characteristic functions converge to \(e^{-t^2/2}\), which proves the theorem.

◻

Example 3. Let \((X_i)\) be i.i.d., centered, with variance \(0<\sigma^2<\infty\). Put \(T_n=n^{-1/2}\sum_{i=1}^na_iX_i\), \(s_n=(\sum_{i=1}^na_i^2)^{1/2}>0\), and \(m_n=\max_{i\leq n}|a_i|\). If \(m_n/s_n\to0\), then \(T_n/(\sigma s_n/\sqrt n)\Rightarrow \mathcal{N}(0,1)\).

The normalized summands are \(a_iX_i/(\sigma s_n)\). Their Lindeberg sum is bounded by

\[\sum_i\frac{a_i^2}{\sigma^2s_n^2}\mathbb{E}\left[X_i^21_{\{|a_iX_i|>\varepsilon\sigma s_n\}}\right]\leq\frac1{\sigma^2}\mathbb{E}\left[X_1^21_{\{|X_1|>\varepsilon\sigma s_n/m_n\}}\right]\longrightarrow0.\]

The threshold tends to infinity and \(\mathbb{E}X_1^2<\infty\), so dominated convergence applies.

Lindeberg’s condition controls the contribution of unusually large summands to the total variance. A higher-moment bound can be easier to check: Lyapunov’s condition gives sufficient tail control by a single moment estimate.

Theorem 4 (Lyapunov). Suppose, for each \(n\), \(X_{n,1},\dots,X_{n,r_n}\) is independent, and satisfies the Lyapunov condition

\[\lim\limits_{n\to\infty} \sum_{1 \leq k \leq r_n} \frac{1}{\sigma_n^{2+\delta}} \mathbb{E} |X_{n,k}|^{2+\delta} = 0\]

for some \(\delta > 0\). Then, \(\frac{S_n}{\sigma_n} \xrightarrow{d} \mathcal{N} (0,1)\).

Proof.

On \(|X_{n,k}|\geq\varepsilon\sigma_n\), we have \(X_{n,k}^2\leq|X_{n,k}|^{2+\delta}/(\varepsilon\sigma_n)^\delta\). Hence

\[\frac1{\sigma_n^2}\sum_k\mathbb{E}X_{n,k}^21_{\{|X_{n,k}|\geq\varepsilon\sigma_n\}}\leq\frac1{\varepsilon^\delta\sigma_n^{2+\delta}}\sum_k\mathbb{E}|X_{n,k}|^{2+\delta}\longrightarrow0.\]

Lindeberg’s theorem applies.

◻

Dependence can change the limiting distribution even when each increment is centred. The next example gives a mixture of normal laws because a random scale remains in the limit.

Example 5. Let \((\Delta_i)_{i\geq1}\) be independent random signs. Put \(W=1_{\{\Delta_1=1\}}+2\,1_{\{\Delta_1=-1\}}\) and \(X_n=W\sum_{k=2}^n\Delta_k\). With \(\mathscr F_n=\sigma(\Delta_1,\ldots,\Delta_n)\), \((X_n)\) is a martingale. Conditional on \(W\), the ordinary CLT applies to the remaining signs, so \(X_n/\sqrt n\) converges to \(\frac12\mathcal{N}(0,1)+\frac12\mathcal{N}(0,4)\). The random conditional variance prevents a single normal limit.

For a square-integrable martingale \((X_n,\mathscr F_n)_{n\geq0}\) with \(X_0=0\), the increments \(Y_n=X_n-X_{n-1}\) have conditional mean zero. Their accumulated conditional variance is the relevant scale. One can introduce the variance-clock stopping time

\[\sigma_n^2=\mathbb{E}[Y_n^2\mid\mathscr F_{n-1}],\qquad v_t=\inf\left\{n\geq1:\sum_{k=1}^n\sigma_k^2\geq t\right\},\]

with the infimum equal to infinity if the level is never reached. This is a stopping time because the partial conditional-variance sums are adapted. A normal limit at this random time requires control of large jumps, the variance overshoot, and finiteness of the stopping time. The following theorem states explicit sufficient conditions directly for finite rows of martingale differences.

Theorem 6. (Martingale CLT, finite rows). Let \(Y_{n,k},\ 1\leq k\leq r_n\) be square-integrable martingale differences for \((\mathscr F_{n,k})\): \(\mathbb{E}[Y_{n,k}\mid\mathscr F_{n,k-1}]=0\). Write \(v_{n,k}=\mathbb{E}[Y_{n,k}^2\mid\mathscr F_{n,k-1}]\). Suppose

\[V_n=\sum_{k=1}^{r_n}v_{n,k}\xrightarrow{P}\sigma^2>0,\qquad \sum_{k=1}^{r_n}\mathbb{E}[Y_{n,k}^21_{\{|Y_{n,k}|>\varepsilon\}}]\longrightarrow0\quad(\varepsilon>0).\]

Then \(\sum_{k=1}^{r_n}Y_{n,k}\Rightarrow \mathcal{N}(0,\sigma^2)\). One may append zero increments after \(r_n\) without changing the assertion.

Proof.

First suppose \(V_n\leq c\) almost surely. Suppress \(n\) and put \(S_k=\sum_{j\leq k}Y_j\), \(V_k=\sum_{j\leq k}v_j\). The process \(Z_k=\exp(itS_k+t^2V_k/2)\) satisfies

\[\mathbb{E}(Z_k-Z_{k-1})=\mathbb{E}\left[e^{itS_{k-1}+t^2V_k/2}\{\mathbb{E}[e^{itY_k}\mid\mathscr F_{k-1}]-e^{-t^2v_k/2}\}\right].\]

Here \(S_{k-1}\) and \(V_k\) are \(\mathscr F_{k-1}\)-measurable. To spell out the error bound, conditional centering and the same Taylor remainder as in Theorem 2 give

\[\left|\mathbb{E}[e^{itY_k}\mid\mathscr F_{k-1}]-(1-t^2v_k/2)\right|\leq C_t\left(\mathbb{E}[Y_k^2 1_{\{|Y_k|>\varepsilon\}}\mid\mathscr F_{k-1}]+\varepsilon v_k\right).\]

Also \(|e^{-t^2v_k/2}-(1-t^2v_k/2)|\leq C_t v_k^2\). The multiplier in the preceding telescoping identity has modulus at most \(e^{t^2c/2}\). These observations justify the next bound term by term.

Conditional Taylor expansion, centering, and \(V_k\leq c\) bound the sum of the absolute expectations by

\[C_{t,c}\left(\sum_k\mathbb{E}Y_k^21_{\{|Y_k|>\varepsilon\}}+\varepsilon\sum_kEv_k+\sum_kEv_k^2\right).\]

Here \(\sum_kEv_k\leq c\) and \(\sum_kEv_k^2\leq c\mathbb{E}\max_kv_k\). Moreover,

\[\mathbb{E}\max_kv_k\leq\varepsilon^2+\sum_k\mathbb{E}Y_k^21_{\{|Y_k|>\varepsilon\}}.\]

Taking \(n\to\infty\) and then \(\varepsilon\downarrow0\) proves \(\mathbb{E}Z_{r_n}\to1\). Since \(V_n\to\sigma^2\) in probability and \(|Z_{r_n}|\leq e^{t^2c/2}\),

\[\left|Ee^{itS_{r_n}}-e^{-t^2\sigma^2/2}\mathbb{E}Z_{r_n}\right|\leq e^{t^2c/2}\mathbb{E}\left|e^{-t^2V_n/2}-e^{-t^2\sigma^2/2}\right|\longrightarrow0.\]

This proves characteristic-function convergence in the bounded case.

For the general case choose \(c>\sigma^2\) and replace \(Y_{n,k}\) by \(\widetilde Y_{n,k}=Y_{n,k}1_{\{\sum_{j\leq k}v_{n,j}\leq c\}}\). The indicator is \(\mathscr F_{n,k-1}\)-measurable. The new total conditional variance is at most \(c\), its Lindeberg sums do not increase, and it agrees with \(V_n\) on \(\{V_n\leq c\}\), an event whose probability tends to one. The bounded case applies. The original and truncated sums agree on that same event, proving the result.

◻

1. Examples

Example 7. (Inversions). Let \((a_1,\ldots,a_n)\) be a uniform permutation and let \(I_n=\sum_{i<j}1_{\{a_i>a_j\}}\). Define \(Y_j=\#\{i<j:a_i>a_j\}\), so \(I_n=\sum_jY_j\).

The rank of the \(j\)th entry among the first \(j\) entries is \(j-Y_j\). Every sequence of ranks with the \(j\)th rank in \(\{1,\ldots,j\}\) determines exactly one permutation: insert position \(j\) in the prescribed place in the ordered list of previous positions, and finally assign values by order. Thus the \(n!=\prod_{j=1}^nj\) possible rank sequences are equally likely. This proves that the \(Y_j\) are independent, each uniform on \(\{0,\ldots,j-1\}\).

\[\mathbb{E}Y_j=\frac{j-1}{2},\quad \operatorname{Var}(Y_j)=\frac{j^2-1}{12},\quad \mathbb{E}I_n=\frac{n(n-1)}4,\quad s_n^2=\operatorname{Var}(I_n)=\frac{n(n-1)(2n+5)}{72}.\]

The summands are independent but not identically distributed. Since \(|Y_j-\mathbb{E}Y_j|\leq n\) and \(s_n\sim n^{3/2}/6\), every Lindeberg indicator is zero for sufficiently large \(n\). The independent-array theorem gives

\[\frac{I_n-n(n-1)/4}{s_n}\Rightarrow \mathcal{N}(0,1),\qquad \frac{3\sqrt n}{2}\left(\frac{4I_n}{n^2}-1\right)\Rightarrow \mathcal{N}(0,1).\]

The second statement follows from the first because the difference in centering is \(3/(2\sqrt n)\) and the ratio of the two scales tends to one.

Example 8. (AR(1)). Let \(X_0=0,\ X_k=\rho X_{k-1}+E_k\), where \(|\rho|<1\) and the innovations are i.i.d., centered, with \(0<\omega^2=EE_1^2<\infty\) and \(|E_k|\leq c\). Each \(E_k\) is independent of \(\mathscr F_{k-1}=\sigma(E_1,\ldots,E_{k-1})\), hence of the past observations; it is not independent of the whole process.

The least-squares estimator and its error are

\[\widehat\rho_n=\frac{\sum_{k=1}^nX_kX_{k-1}}{\sum_{k=1}^nX_{k-1}^2},\qquad (\widehat\rho_n-\rho)\sum_{k=1}^nX_{k-1}^2=S_n:=\sum_{k=1}^nE_kX_{k-1}.\]

Set the estimator to zero if its denominator vanishes; the denominator limit below makes this event negligible. Iterating the recursion gives \(X_k=\sum_{j=1}^k\rho^{k-j}E_j\). Hence \(|X_k|\leq C:=c/(1-|\rho|)\) and \(\mathbb{E}X_k^2=\omega^2\sum_{j=0}^{k-1}\rho^{2j}\to v:=\omega^2/(1-\rho^2)\).

For \(\ell>k\), write \(X_\ell=\rho^{\ell-k}X_k+R_{\ell,k}\), where \(R_{\ell,k}\) is centered and independent of \(\mathscr F_k\). Expanding the squares gives

\[\operatorname{Cov}(X_k^2,X_\ell^2)=\rho^{2(\ell-k)}\operatorname{Var}(X_k^2),\qquad \operatorname{Var}\left(\frac1n\sum_{k=1}^nX_{k-1}^2\right)\leq\frac{C^4}{n}\left(1+2\sum_{j\geq1}|\rho|^{2j}\right).\]

The expectation converges to \(v\) by Cesàro averaging and the variance tends to zero. Thus \(n^{-1}\sum_kX_{k-1}^2\to v\) in probability.

Now \(Y_{n,k}=E_kX_{k-1}/\sqrt{nv\omega^2}\) is a martingale difference array. Its total conditional variance is \((nv)^{-1}\sum_kX_{k-1}^2\to1\). Its increments are bounded by \(cC/\sqrt{nv\omega^2}\), so the Lindeberg condition holds with zero tail terms for large \(n\). Applying Theorem 6 and then Slutsky gives

\[\frac{S_n}{\sqrt{nv\omega^2}}\Rightarrow \mathcal{N}(0,1),\qquad \sqrt{\frac{n}{1-\rho^2}}(\widehat\rho_n-\rho)\Rightarrow \mathcal{N}(0,1).\]