Stochastic Convergence Exercises: Poisson Limits, Cauchy Averages, and Joint CLTs

The number of summands alone does not determine the limit of a sum. Rare Bernoulli successes retain jumps of fixed size and give a Poisson law; Cauchy averages retain the original Cauchy law at every sample size. With a finite fourth moment, however, the sample mean and second moment have a joint Gaussian limit. Characteristic functions make these three mechanisms explicit and show where the usual central limit argument applies.

These are Exercises 21, 22, 23 from A. W. van der Vaart, Asymptotic Statistics (1998), Chapter 2, p. 24. The question numbers identify the original exercises. Write \(X_n\Rightarrow X\) for convergence in distribution and \(X_n\xrightarrow{P}X\) for convergence in probability.

For the underlying definitions, see joint convergence.

1. Rare successes give a Poisson limit

Question (Exercise 21). Use characteristic functions to prove \(\operatorname{Binomial}(n,\lambda/n)\Rightarrow\operatorname{Poisson}(\lambda)\). Why does a central limit theorem not give a normal limit here?

Answer.

Fix \(\lambda>0\), and let

\[ B_n\sim\operatorname{Binomial}(n,\lambda/n), \]

for sufficiently large \(n\). Its characteristic function is

\[ \begin{aligned} \varphi_{B_n}(t) &= \left(1-\frac{\lambda}{n} +\frac{\lambda}{n}e^{it}\right)^n\\ &= \left(1+\frac{\lambda(e^{it}-1)}n\right)^n \longrightarrow \exp\!\bigl(\lambda(e^{it}-1)\bigr). \end{aligned} \]

If \(P_\lambda\sim\operatorname{Poisson}(\lambda)\), then

\[ \begin{aligned} \varphi_{P_\lambda}(t) &= \sum_{k=0}^{\infty} e^{itk}e^{-\lambda}\frac{\lambda^k}{k!}\\ &= e^{-\lambda}\exp(\lambda e^{it}) = \exp\!\bigl(\lambda(e^{it}-1)\bigr). \end{aligned} \]

Lévy’s continuity theorem therefore gives

\[ {B_n\Rightarrow P_\lambda.} \]

For fixed \(t\in\mathbb R\), put \(z=\lambda(e^{it}-1)\). For all sufficiently large \(n\), \(1+z/n\) lies in a neighborhood of \(1\) where the logarithm is analytic. Taylor’s expansion gives

\[ \begin{aligned} n\log(1+z/n) &=n\left[\frac zn-\frac{z^2}{2n^2}+O_t(n^{-3})\right]\\ &=z-\frac{z^2}{2n}+O_t(n^{-2}) \longrightarrow z. \end{aligned} \]

Here the bound implicit in \(O_t\) may depend on the fixed \(t\) and \(\lambda\). Exponentiation yields

\[ \varphi_{B_n}(t)=(1+z/n)^n\longrightarrow e^z =e^{\lambda(e^{it}-1)}. \]

The proposed limiting masses are normalized because

\[ \sum_{k=0}^\infty e^{-\lambda}\frac{\lambda^k}{k!} =e^{-\lambda}e^\lambda=1. \]

Their characteristic function is continuous at zero, as required for the continuity theorem.

The Poisson limit already suggests why a normal approximation cannot become exact here: individual successes remain visible on the scale of the total standard deviation. To check the standardized limit, note that

\[ \mathbb{E} B_n=\lambda,\qquad \operatorname{Var}(B_n) =\lambda\left(1-\frac{\lambda}{n}\right) \longrightarrow\lambda. \]

Slutsky’s lemma gives the actual standardized limit:

\[ { \frac{B_n-\lambda} {\sqrt{\lambda(1-\lambda/n)}} \Rightarrow \frac{P_\lambda-\lambda}{\sqrt\lambda}. } \]

For \(\lambda>0\), let \(s_n=\sqrt{\lambda(1-\lambda/n)}\), taking \(n>\lambda\). The characteristic function of the standardized variable is

\[ \begin{aligned} \mathbb{E}\exp\left(it\frac{B_n-\lambda}{s_n}\right) &=e^{-it\lambda/s_n} \left[1+\frac\lambda n(e^{it/s_n}-1)\right]^n\\ &\longrightarrow e^{-it\sqrt\lambda} \exp\left[\lambda(e^{it/\sqrt\lambda}-1)\right]\\ &=\exp\left[\lambda\left(e^{it/\sqrt\lambda}-1-\frac{it}{\sqrt\lambda}\right)\right]. \end{aligned} \]

For the second line, the same logarithmic expansion applies with the bounded sequence \(z_n=\lambda(e^{it/s_n}-1)\to\lambda(e^{it/\sqrt\lambda}-1)\). This is the characteristic function of \((P_\lambda-\lambda)/\sqrt\lambda\). In particular,

\[ \mathbb P\left(\frac{P_\lambda-\lambda}{\sqrt\lambda}=-\sqrt\lambda\right) =\mathbb P(P_\lambda=0)=e^{-\lambda}>0. \]

Every nondegenerate normal law assigns zero mass to a singleton, so the standardized limit cannot be standard normal.

The ordinary i.i.d. CLT assumes a fixed summand law; here the Bernoulli law changes with the row. The failure of the more general Lindeberg condition is explicit. Write \(B_n=\sum_{i=1}^nY_{n,i}\), with independent \(Y_{n,i}\sim\operatorname{Bernoulli}(p_n)\) within each row. For \(p_n=\lambda/n\),

\[ Y_{n,i}-p_n= \begin{cases} 1-p_n,&\text{with probability }p_n,\\ -p_n,&\text{with probability }1-p_n. \end{cases} \]

With \(s_n^2=np_n(1-p_n)\), the normalized truncated second-moment sum equals

\[ \begin{aligned} L_n(\varepsilon) &=\frac n{s_n^2}\left[ p_n(1-p_n)^2\mathbf1_{\{1-p_n>\varepsilon s_n\}} +(1-p_n)p_n^2\mathbf1_{\{p_n>\varepsilon s_n\}} \right]. \end{aligned} \]

For \(0<\varepsilon<1/\sqrt\lambda\),

\[ \frac{1-p_n}{s_n}\to\frac1{\sqrt\lambda}>\varepsilon, \qquad \frac{p_n}{s_n}\to0<\varepsilon. \]

Thus, eventually, the first indicator is one and the second is zero, giving

\[ L_n(\varepsilon) =\frac{np_n(1-p_n)^2}{np_n(1-p_n)} =1-p_n\to1. \]

A success has nonvanishing size after standardization, and these jumps retain a nonzero share of the total variance. The displayed limit is precisely the obstruction to the Lindeberg condition.

For \(\lambda=0\), both laws are the point mass at zero. Standardization is relevant only for \(\lambda>0\).

2. Cauchy averages do not concentrate

Question (Exercise 22). Let \(X_1,\ldots,X_n\) be i.i.d. standard Cauchy. (i) Use characteristic functions to show that their average is standard Cauchy. (ii) Explain why the weak law of large numbers does not hold.

Answer.

(i) Characteristic functions. A standard Cauchy variable has density

\[ f(x)=\frac1{\pi(1+x^2)} \]

and characteristic function

\[ \varphi_X(t) = \frac1\pi\int_{-\infty}^{\infty} \frac{e^{itx}}{1+x^2}\,dx = e^{-|t|}. \]

To compute the Cauchy transform using real integrals, express its density as an integral of Gaussian kernels. We first need the transform of each kernel. Write \(\phi(x)=(2\pi)^{-1/2}e^{-x^2/2}\) and first compute the Gaussian transform. Let

\[ \gamma(s)=\int_{\mathbb R}e^{isx}\phi(x)\,dx. \]

Differentiation under the integral is justified by integrability of \(|x|\phi(x)\). Integration by parts then yields

\[ \begin{aligned} \gamma'(s) &=i\int_{\mathbb R}x e^{isx}\phi(x)\,dx\\ &=-i\int_{\mathbb R}e^{isx}\phi'(x)\,dx\\ &=-i[e^{isx}\phi(x)]_{-\infty}^{\infty} +i(is)\int_{\mathbb R}e^{isx}\phi(x)\,dx\\ &=-s\gamma(s). \end{aligned} \]

Since \(\gamma(0)=1\), solving the differential equation gives \(\gamma(s)=e^{-s^2/2}\). For \(u>0\), the substitution \(y=\sqrt{2u}\,x\) therefore gives

\[ \begin{aligned} \int_{\mathbb R}e^{-ux^2+itx}\,dx &=\frac1{\sqrt{2u}} \int_{\mathbb R}e^{-y^2/2}e^{i(t/\sqrt{2u})y}\,dy\\ &=\sqrt{\frac\pi u}\, \gamma\left(\frac{t}{\sqrt{2u}}\right)\\ &=\sqrt{\frac\pi u}\,e^{-t^2/(4u)}. \end{aligned} \]

The following identity supplies the required Gaussian kernels

\[ \frac1{1+x^2} =\int_0^\infty e^{-u(1+x^2)}\,du. \]

The absolute integral needed for Fubini’s theorem is finite:

\[ \begin{aligned} \int_0^\infty\int_{\mathbb R} |e^{itx}e^{-u(1+x^2)}|\,dx\,du &=\int_0^\infty e^{-u}\sqrt{\frac\pi u}\,du\\ &=\sqrt\pi\,\Gamma(1/2)=\pi. \end{aligned} \]

Consequently,

\[ \begin{aligned} \varphi_X(t) &=\frac1\pi\int_{\mathbb R}\frac{e^{itx}}{1+x^2}\,dx\\ &=\frac1\pi\int_0^\infty e^{-u} \left(\int_{\mathbb R}e^{-ux^2+itx}\,dx\right)du\\ &=\frac1{\sqrt\pi}\int_0^\infty u^{-1/2}\exp\left(-u-\frac{t^2}{4u}\right)du\\ &=\frac2{\sqrt\pi}\int_0^\infty \exp\left(-y^2-\frac{t^2}{4y^2}\right)dy, \end{aligned} \]

where the last substitution is \(u=y^2\).

Define, for \(a\ge0\),

\[ J(a)=\int_0^\infty e^{-y^2-a^2/y^2}\,dy. \]

For \(a>0\), differentiation under the integral gives

\[ J'(a)=-2a\int_0^\infty y^{-2}e^{-y^2-a^2/y^2}\,dy. \]

This differentiation is justified locally in \(a>0\): the factor \(e^{-c/y^2}\), for a fixed \(c>0\), controls the derivative near zero, and \(e^{-y^2}\) controls it at infinity. In the last integral, substitute \(z=a/y\), so that \(y=a/z\) and \(dy=-az^{-2}dz\). Then

\[ \begin{aligned} \int_0^\infty y^{-2}e^{-y^2-a^2/y^2}\,dy &=\frac1a\int_0^\infty e^{-a^2/z^2-z^2}\,dz\\ &=\frac{J(a)}a. \end{aligned} \]

Therefore

\[ J'(a)=-2J(a), \qquad J(a)=Ce^{-2a}\quad(a>0). \]

Dominated convergence, with dominating function \(e^{-y^2}\), gives

\[ \lim_{a\downarrow0}J(a) =\int_0^\infty e^{-y^2}\,dy=\frac{\sqrt\pi}{2}. \]

Hence \(C=\sqrt\pi/2\), and substitution of \(a=|t|/2\) yields

\[ \varphi_X(t) =\frac2{\sqrt\pi}J(|t|/2) =\frac2{\sqrt\pi}\frac{\sqrt\pi}{2}e^{-|t|} =e^{-|t|}. \]

For

\[ \overline X_n=\frac1n\sum_{j=1}^{n}X_j, \]

independence gives

\[ \begin{aligned} \varphi_{\overline X_n}(t) &= \prod_{j=1}^{n} \mathbb{E} e^{i(t/n)X_j}\\ &= \left(e^{-|t|/n}\right)^n =e^{-|t|}. \end{aligned} \]

By uniqueness of characteristic functions,

\[ {\overline X_n\sim\operatorname{Cauchy}(0,1) \quad\text{for every }n.} \]

(ii) Why the weak law does not hold. Part (i) shows directly that the averages do not concentrate. To locate the failed hypothesis of the usual weak law, check the first absolute moment:

\[ \begin{aligned} \mathbb{E} X_1^+ &=\lim_{R\to\infty}\int_0^R\frac{x}{\pi(1+x^2)}\,dx\\ &=\lim_{R\to\infty}\frac1{2\pi}[\log(1+x^2)]_0^R\\ &=\lim_{R\to\infty}\frac1{2\pi}\log(1+R^2)=\infty. \end{aligned} \]

Symmetry gives \(\mathbb{E} X_1^-=\infty\). Thus \(\mathbb{E} X_1\) is undefined; symmetry does not give a mean of zero.

The Cauchy distribution function follows by integration:

\[ F(x)=\int_{-\infty}^x\frac{du}{\pi(1+u^2)} =\frac1\pi[\arctan u]_{-\infty}^x =\frac12+\frac1\pi\arctan x. \]

The failure of concentration follows directly from part (i). For every \(\varepsilon>0\),

\[ \begin{aligned} \mathbb P(|\overline X_n|>\varepsilon) &= \mathbb P(|X_1|>\varepsilon)\\ &= 1-\frac2\pi\arctan(\varepsilon)>0, \end{aligned} \]

independently of \(n\). Therefore

\[ {\overline X_n\not\xrightarrow{P}0.} \]

More generally, for every finite \(c\) and \(\varepsilon>0\),

\[ \mathbb P(|\overline X_n-c|>\varepsilon) =1-\frac{\arctan(c+\varepsilon)-\arctan(c-\varepsilon)}\pi>0. \]

The strict inequality follows because a finite interval has strictly less than full probability under the everywhere-positive Cauchy density. The probability is independent of \(n\), excluding convergence in probability to any constant.

3. The mean and second moment converge jointly

Question (Exercise 23). For i.i.d. \(X_i\) with finite fourth moment, find constants \(a,b\) and a deterministic normalization \(c_n\) such that \(c_n(\overline X_n-a,M_{2,n}-b)\) has a limiting distribution, and identify that law. Here \(M_{2,n}=n^{-1}\sum_iX_i^2\).

Answer.

Write

\[ m_r=\mathbb{E} X_1^r,\qquad r=1,2,3,4, \]

and define

\[ \overline X_n=\frac1n\sum_{i=1}^{n}X_i, \qquad M_{2,n}=\frac1n\sum_{i=1}^{n}X_i^2. \]

Thus \(M_{2,n}\) is the average of the squares, not \((\overline X_n)^2\).

A suitable choice is

\[ {a=m_1,\qquad b=m_2,\qquad c_n=\sqrt n.} \]

The limit is

\[ { \sqrt n \begin{pmatrix} \overline X_n-m_1\\ M_{2,n}-m_2 \end{pmatrix} \Rightarrow \mathcal{N}_2(0,\Sigma), } \]

where

\[ { \Sigma= \begin{pmatrix} m_2-m_1^2 & m_3-m_1m_2\\ m_3-m_1m_2 & m_4-m_2^2 \end{pmatrix}. } \]

The fourth-moment assumption guarantees, for \(1\le r\le4\),

\[ \mathbb{E}|X_1|^r \le(\mathbb{E}|X_1|^4)^{r/4}<\infty, \]

by Hölder’s inequality (with equality of the two expressions when \(r=4\)). Thus all products in the following calculations are integrable. Put

\[ \mu_V=\begin{pmatrix}m_1\\m_2\end{pmatrix}, \qquad V_i=\begin{pmatrix}X_i\\X_i^2\end{pmatrix}, \qquad \Sigma=\mathbb{E}[(V_i-\mu_V)(V_i-\mu_V)^\top]. \]

The first diagonal entry is

\[ \begin{aligned} \Sigma_{11} &=\mathbb{E}(X_i-m_1)^2\\ &=\mathbb{E} X_i^2-2m_1\mathbb{E} X_i+m_1^2\\ &=m_2-2m_1^2+m_1^2=m_2-m_1^2. \end{aligned} \]

The off-diagonal entries are

\[ \begin{aligned} \Sigma_{12}=\Sigma_{21} &=\mathbb{E}[(X_i-m_1)(X_i^2-m_2)]\\ &=\mathbb{E}[X_i^3-m_2X_i-m_1X_i^2+m_1m_2]\\ &=m_3-m_2m_1-m_1m_2+m_1m_2\\ &=m_3-m_1m_2. \end{aligned} \]

The second diagonal entry is

\[ \begin{aligned} \Sigma_{22} &=\mathbb{E}(X_i^2-m_2)^2\\ &=\mathbb{E}[X_i^4-2m_2X_i^2+m_2^2]\\ &=m_4-2m_2^2+m_2^2=m_4-m_2^2. \end{aligned} \]

To pass from separate variances to a joint limit, examine every fixed linear combination of the two coordinates. For fixed \(q=(s,t)^\top\), define

\[ W_i=q^\top(V_i-\mu_V) =s(X_i-m_1)+t(X_i^2-m_2). \]

Then \(\mathbb{E} W_i=0\), and expansion gives

\[ \begin{aligned} \mathbb{E} W_i^2 &=s^2\mathbb{E}(X_i-m_1)^2 +2st\mathbb{E}[(X_i-m_1)(X_i^2-m_2)] +t^2\mathbb{E}(X_i^2-m_2)^2\\ &=s^2(m_2-m_1^2) +2st(m_3-m_1m_2) +t^2(m_4-m_2^2)\\ &=s^2m_2+2stm_3+t^2m_4-(sm_1+tm_2)^2\\ &=q^\top\Sigma q\ge0. \end{aligned} \]

This also proves that \(\Sigma\) is positive semidefinite. These vectors are i.i.d., and \(\mathbb{E}\|V_i\|^2=m_2+m_4<\infty\). The centered and scaled statistic is exactly

\[ \begin{aligned} \sqrt n \begin{pmatrix} \overline X_n-m_1\\ M_{2,n}-m_2 \end{pmatrix} &=\sqrt n\left(\frac1n\sum_{i=1}^nV_i-\mu_V\right)\\ &=\frac1{\sqrt n}\sum_{i=1}^n(V_i-\mu_V). \end{aligned} \]

Each such linear combination has finite variance. The scalar CLT therefore applies; its characteristic-function proof uses a second-order characteristic-function expansion. Let \(W\) have mean zero and finite variance \(v\). Set

\[ H(u)=\frac{e^{iu}-1-iu+u^2/2}{u^2}\quad(u\ne0), \qquad H(0)=0. \]

Taylor’s formula at zero shows that \(H(u)\to0\) as \(u\to0\). The function is bounded: it is continuous on compact intervals, and for \(|u|\ge1\),

\[ |H(u)|\le\frac2{u^2}+\frac1{|u|}+\frac12\le\frac72. \]

Consequently, dominated convergence, with dominating variable a constant times \(W^2\), gives

\[ \begin{aligned} \mathbb{E} e^{ihW} &=1+ih\mathbb{E} W-\frac{h^2}{2}\mathbb{E} W^2 +h^2\mathbb{E}[W^2H(hW)]\\ &=1-\frac{vh^2}{2}+o(h^2). \end{aligned} \]

For independent copies \(W_1,\ldots,W_n\) and fixed \(t\),

\[ \begin{aligned} \mathbb{E}\exp\left(\frac{it}{\sqrt n}\sum_{j=1}^nW_j\right) &=\left[\mathbb{E} e^{itW/\sqrt n}\right]^n\\ &=\left[1-\frac{vt^2}{2n}+o(n^{-1})\right]^n\\ &\longrightarrow e^{-vt^2/2}. \end{aligned} \]

To justify the final limit, write the bracket as \(1+z_n\), where \(nz_n\to-vt^2/2\). For large \(n\), the logarithm defined near \(1\) satisfies

\[ n\log(1+z_n)=nz_n+O(n|z_n|^2) \longrightarrow-\frac{vt^2}{2}. \]

Exponentiation gives the claimed limit, and Lévy’s continuity theorem identifies the normal law.

Applying this expansion to \(W_i\), for every fixed \(q\),

\[ \begin{aligned} \mathbb{E}\exp\left[ iq^\top\frac1{\sqrt n}\sum_{i=1}^n(V_i-\mu_V) \right] &=\left[\mathbb{E} e^{iW_1/\sqrt n}\right]^n\\ &=\left[1-\frac{q^\top\Sigma q}{2n}+o(n^{-1})\right]^n\\ &\longrightarrow\exp\left(-\frac12q^\top\Sigma q\right). \end{aligned} \]

Since \(\Sigma\) is positive semidefinite, it has a symmetric square root \(\Sigma^{1/2}\). If \(G\sim \mathcal{N}_2(0,I_2)\), then

\[ \mathbb{E} e^{iq^\top\Sigma^{1/2}G} =\exp\left(-\frac12\|\Sigma^{1/2}q\|_2^2\right) =\exp\left(-\frac12q^\top\Sigma q\right). \]

The limiting function is continuous at zero and is the characteristic function of \(\mathcal{N}_2(0,\Sigma)\). Lévy’s continuity theorem proves the desired joint limit directly.

If \(q^\top\Sigma q=0\), then \(\mathbb{E} W_i^2=0\), so \(W_i=0\) almost surely. Thus the corresponding linear combination is exactly zero for every \(n\), consistent with the degenerate normal limit. No positive-definiteness assumption is missing.

For example, if \(\mathbb P(X_i=1)=\mathbb P(X_i=-1)=1/2\), then \(M_{2,n}=1\) almost surely and \(\Sigma=\operatorname{diag}(1,0)\). Only the first coordinate fluctuates.

The two limiting coordinates need not be independent. For example, if \(X_i\sim\operatorname{Bernoulli}(p)\), \(0<p<1\), then \(X_i^2=X_i\), all four raw moments equal \(p\), and

\[ \Sigma=p(1-p) \begin{pmatrix}1&1\\1&1\end{pmatrix}. \]

Both coordinates of the scaled statistic are identical in this example, so their separate limiting variances cannot describe the joint law. The off-diagonal entries record their dependence and are essential to the answer.

Source: A. W. van der Vaart, Asymptotic Statistics, Chapter 2, “Stochastic Convergence”, pp. 5–24. The questions above are stated in the notation used in these solutions.