Stochastic Convergence Exercises: Rare Events and Consistent Estimation

A rare error can have a small probability and still make a large contribution to an expectation. Its occurrence can also move through the probability space so that every sample path sees infinitely many errors. The first examples separate these two phenomena. Moment bounds and summable tail probabilities then give sufficient conditions for convergence, before a likelihood calculation applies them to estimation of the endpoint and rate of a shifted exponential distribution.

These are Exercises 13, 14, 19, 20, 15 from A. W. van der Vaart, Asymptotic Statistics (1998), Chapter 2, p. 24. The question numbers identify the original exercises. Write \(X_n\Rightarrow X\) for convergence in distribution and \(X_n\xrightarrow{P}X\) for convergence in probability.

For the underlying definitions, see Slutsky and stochastic order.

1. Vanishing probabilities with growing expectations

Question (Exercise 13). Find \(X_n\Rightarrow0\) with \(\mathbb{E} X_n\to\infty\).

Answer.

Let \(U\sim\operatorname{Uniform}(0,1)\), and define

\[ X_n=n^2\mathbf1_{\{U\le1/n\}}. \]

There are two quantities to keep separate: the probability of a nonzero value and the size of that value. The law is

\[ \mathbb P(X_n=n^2)=\frac1n, \qquad \mathbb P(X_n=0)=1-\frac1n. \]

For every \(\varepsilon>0\),

\[ \mathbb P(|X_n|>\varepsilon) \le\mathbb P(U\le1/n) =\frac1n\longrightarrow0. \]

Thus \(X_n\xrightarrow{P}0\), and consequently \(X_n\Rightarrow0\). Nevertheless,

\[ {\mathbb{E} X_n=n^2\frac1n=n\longrightarrow\infty.} \]

On the specified probability space the sequence even converges almost surely: for each \(u>0\), if \(n>1/u\), then \(1/n<u\) and \(X_n(u)=0\).

The expectation depends on the product of size and probability. Here the increasing size outweighs the decreasing probability, and the upper-tail expectations are not uniformly small. For any \(K>0\) and any integer \(n>\sqrt K\),

\[ \mathbb{E}[X_n\mathbf1_{\{X_n>K\}}] =n^2\mathbb P(U\le1/n)=n. \]

Thus the sequence is not uniformly integrable. This calculation explains why even almost-sure convergence in this coupling does not justify passing the limit through the expectation.

2. The typewriter sequence

Question (Exercise 14). Find \(X_n\xrightarrow{P}0\) such that \(X_n\) does not converge almost surely.

Answer.

Let \(U\sim\operatorname{Uniform}[0,1)\). Every integer \(n\ge1\) can be written uniquely as

\[ n=2^k+j,\qquad k\ge0,\quad 0\le j<2^k. \]

Define

\[ X_{2^k+j} = \mathbf1_{\left\{ j2^{-k}\le U<(j+1)2^{-k} \right\}}. \]

For \(0<\varepsilon<1\),

\[ \mathbb P(|X_{2^k+j}|>\varepsilon) =2^{-k}\longrightarrow0 \qquad(k\to\infty). \]

Since \(2^k\le n<2^{k+1}\), we also have \(2^{-k}<2/n\). For \(\varepsilon\ge1\), this probability is zero. Since \(n\to\infty\) implies \(k\to\infty\),

\[ X_n\xrightarrow{P}0. \]

Fix any \(u\in[0,1)\). In each block

\[ X_{2^k}(u),\,X_{2^k+1}(u),\,\ldots,\,X_{2^{k+1}-1}(u), \]

the intervals sweep through the whole probability space. For this fixed point, exactly one term equals \(1\), because the corresponding intervals partition \([0,1)\). When \(k\ge1\), the block also contains a term equal to \(0\). Hence

\[ \limsup_{n\to\infty}X_n(u)=1, \qquad \liminf_{n\to\infty}X_n(u)=0. \]

Thus the sequence fails to converge at every point of this probability space:

\[ {\mathbb P\bigl((X_n)\text{ converges}\bigr)=0.} \]

For fixed \(u\in[0,1)\), let

\[ j_k=\lfloor2^ku\rfloor, \qquad n_k=2^k+j_k. \]

Then \(0\le j_k<2^k\), and

\[ \frac{j_k}{2^k}\le u<\frac{j_k+1}{2^k}, \qquad X_{n_k}(u)=1. \]

For \(k\ge1\), choose the next interval cyclically within the same block:

\[ \ell_k= \begin{cases} j_k+1,&j_k<2^k-1,\\ 0,&j_k=2^k-1, \end{cases} \qquad m_k=2^k+\ell_k. \]

The intervals are disjoint and \(\ell_k\ne j_k\), so \(X_{m_k}(u)=0\). Both \(n_k\) and \(m_k\) increase from block to block and tend to infinity. Thus the two subsequences have different limits for every \(u\), proving failure of convergence to any random variable almost surely.

3. Bias, variance, and consistency

Question (Exercise 19). If \(\mathbb{E} X_n\to\mu\) and \(\operatorname{Var}(X_n)\to0\), show that \(X_n\xrightarrow{P}\mu\).

Answer.

Let \(\mu_n=\mathbb{E} X_n\). Since

\[ X_n-\mu=(X_n-\mu_n)+(\mu_n-\mu), \]

we obtain

\[ \begin{aligned} \mathbb{E}(X_n-\mu)^2 &=\mathbb{E}\left[(X_n-\mu_n)^2 +2(X_n-\mu_n)(\mu_n-\mu) +(\mu_n-\mu)^2\right]\\ &=\mathbb{E}(X_n-\mu_n)^2 +2(\mu_n-\mu)\mathbb{E}(X_n-\mu_n) +(\mu_n-\mu)^2\\ &=\operatorname{Var}(X_n) +2(\mu_n-\mu)(\mu_n-\mu_n) +(\mu_n-\mu)^2\\ &=\operatorname{Var}(X_n)+(\mu_n-\mu)^2 \longrightarrow0. \end{aligned} \]

For every \(\varepsilon>0\), the pointwise inequality

\[ \varepsilon^2\mathbf1_{\{|X_n-\mu|>\varepsilon\}} \le(X_n-\mu)^2 \]

gives, after taking expectations and dividing by \(\varepsilon^2\),

\[ \mathbb P(|X_n-\mu|>\varepsilon) \le \frac{ \operatorname{Var}(X_n)+(\mathbb{E} X_n-\mu)^2 }{\varepsilon^2} \longrightarrow0. \]

Hence

\[ {X_n\xrightarrow{P}\mu.} \]

The calculation also proves the stronger conclusion \(X_n\to\mu\) in \(L^2\).

4. Summable deviations and almost-sure convergence

Question (Exercise 20). If \(\sum_{n=1}^{\infty}\mathbb P(|X_n|>\varepsilon)<\infty\) for every \(\varepsilon>0\), prove that \(X_n\to0\) almost surely.

Answer.

Assume

\[ \sum_{n=1}^{\infty}\mathbb P(|X_n|>\varepsilon)<\infty \qquad\text{for every }\varepsilon>0. \]

For each positive integer \(m\), define

\[ A_m= \bigcap_{N=1}^{\infty} \bigcup_{n\ge N}\{|X_n|>1/m\}. \]

This is the event that \(|X_n|>1/m\) occurs infinitely often. A union bound over all sufficiently late deviations makes its probability small. For every \(N\),

\[ \mathbb P(A_m) \le \mathbb P\!\left(\bigcup_{n\ge N}\{|X_n|>1/m\}\right) \le \sum_{n\ge N}\mathbb P(|X_n|>1/m). \]

The last expression tends to zero as \(N\to\infty\), so \(\mathbb P(A_m)=0\).

Taking a countable union,

\[ \mathbb P\!\left(\bigcup_{m=1}^{\infty}A_m\right)=0. \]

Outside this null event, for every \(m\) there exists \(N_m\) such that

\[ n\ge N_m\quad\Longrightarrow\quad |X_n|\le\frac1m. \]

Given \(\varepsilon>0\), choose \(m\) with \(1/m<\varepsilon\). Then \(|X_n|<\varepsilon\) eventually. Thus

\[ {X_n\xrightarrow{\mathrm{a.s.}}0.} \]

This is the first Borel–Cantelli argument; independence is unnecessary.

With \(E_{n,m}=\{|X_n|>1/m\}\), the exact event identity is

\[ \{X_n\not\to0\} =\bigcup_{m=1}^\infty\limsup_{n\to\infty}E_{n,m}. \]

For the forward inclusion, failure of convergence means that some \(\varepsilon>0\) is exceeded infinitely often; choose \(m\) with \(1/m<\varepsilon\). For the reverse inclusion, infinitely many deviations larger than \(1/m\) prevent convergence to zero. Thus

\[ \mathbb P(X_n\not\to0) \le\sum_{m=1}^\infty \mathbb P\left(\limsup_nE_{n,m}\right)=0. \]

Only the countable family of hypotheses with \(\varepsilon=1/m\) is needed. This avoids taking an uncountable intersection of probability-one events.

5. A shifted exponential likelihood

Question (Exercise 15). Let \(X_1,\ldots,X_n\) be i.i.d. with density

\[ f_{\lambda,a}(x)=\lambda e^{-\lambda(x-a)}\mathbf1_{\{x\ge a\}}, \qquad \lambda>0,\quad a\in\mathbb R. \]

Find the maximum likelihood estimator of \((\lambda,a)\) and prove its consistency.

Answer.

Assume \(\lambda>0\) and \(a\in\mathbb R\). Write

\[ X_{(1)}=\min_{1\le i\le n}X_i, \qquad \overline X_n=\frac1n\sum_{i=1}^{n}X_i. \]

The endpoint parameter enters through the support as well as through the exponential factor. Keep the support restriction when writing the likelihood. For candidate parameters \(r>0\) and \(b\in\mathbb R\), the likelihood is

\[ \begin{aligned} L_n(r,b) &= \prod_{i=1}^{n} r e^{-r(X_i-b)}\mathbf1_{\{X_i\ge b\}}\\ &= r^n \exp\!\left[-r\left(\sum_{i=1}^{n}X_i-nb\right)\right] \mathbf1_{\{b\le X_{(1)}\}}. \end{aligned} \]

For each fixed \(r>0\), this is strictly increasing in \(b\) on the admissible region \(b\le X_{(1)}\). Therefore

\[ \widehat a_n=X_{(1)}. \]

Set

\[ D_n=\sum_{i=1}^{n}(X_i-X_{(1)}) =n(\overline X_n-X_{(1)}). \]

If \(D_n>0\), the profile log-likelihood is

\[ \ell_n(r)=n\log r-rD_n. \]

Its derivatives are

\[ \ell_n'(r)=\frac nr-D_n, \qquad \ell_n''(r)=-\frac n{r^2}<0. \]

Hence its unique maximizer is \(r=n/D_n\), giving

\[ { \widehat a_n=X_{(1)},\qquad \widehat\lambda_n =\frac1{\overline X_n-X_{(1)}}. } \]

The derivative calculation identifies a candidate; the following comparison checks that it maximizes the likelihood over the whole positive half-line. For any \(r>0\), set \(u=r/\widehat\lambda_n\). Then

\[ \begin{aligned} \ell_n(r)-\ell_n(\widehat\lambda_n) &=n\log(r/\widehat\lambda_n)-(r-\widehat\lambda_n)D_n\\ &=n\log u-n(u-1)\\ &=n(\log u-u+1)\le0. \end{aligned} \]

The last inequality follows from \(q(u)=u-1-\log u\), for which

\[ q'(u)=1-1/u, \qquad q(1)=0, \]

so \(u=1\) is its unique global minimum.

The included boundary \(x\ge a\) in the stated density is what permits \(\widehat a_n=X_{(1)}\).

For \(n\ge2\), \(D_n>0\) almost surely because the sampling distribution is continuous. If all observations coincide, including when \(n=1\), the profile likelihood is \(r^n\), which is unbounded; no finite MLE exists in that case. For \(n\ge2\), the event \(\{D_n=0\}\) is contained in \(\{X_1=X_2\}\), which has probability zero by independence and continuity. We may assign any fixed admissible values to the estimators on this null event.

Consistency requires two different estimates: a tail bound for the sample minimum and a moment bound for the sample mean. For the minimum, observe that \(X_i-a\sim\operatorname{Exp}(\lambda)\). Its survival function is

\[ \mathbb P(X_i>a+u) =\int_u^\infty\lambda e^{-\lambda y}\,dy =e^{-\lambda u},\qquad u\ge0. \]

At \(u=0\) this also checks that the density integrates to one. Since \(\widehat a_n\ge a\) almost surely, for every \(\varepsilon>0\),

\[ \begin{aligned} \mathbb P(\widehat a_n-a>\varepsilon) &= \mathbb P(X_1>a+\varepsilon,\ldots,X_n>a+\varepsilon)\\ &= \bigl(e^{-\lambda\varepsilon}\bigr)^n =e^{-n\lambda\varepsilon}\longrightarrow0. \end{aligned} \]

This calculation identifies the exact law \(X_{(1)}-a\sim\operatorname{Exp}(n\lambda)\) and proves \(\widehat a_n\xrightarrow{P}a\).

For \(E=X_i-a\) and integers \(r=1,2\), the substitution \(u=\lambda y\) yields

\[ \begin{aligned} \mathbb{E} E^r &=\int_0^\infty y^r\lambda e^{-\lambda y}\,dy\\ &=\lambda^{-r}\int_0^\infty u^re^{-u}\,du =\lambda^{-r}\Gamma(r+1). \end{aligned} \]

Thus

\[ \mathbb{E} E=\lambda^{-1}, \quad \mathbb{E} E^2=2\lambda^{-2}, \quad \operatorname{Var}(E)=2\lambda^{-2}-\lambda^{-2}=\lambda^{-2}. \]

Adding the shift \(a\) gives

\[ \mathbb{E} X_i=a+\frac1\lambda, \qquad \operatorname{Var}(X_i)=\frac1{\lambda^2}. \]

Chebyshev’s inequality gives

\[ \mathbb P\!\left( \left|\overline X_n-a-\frac1\lambda\right|>\varepsilon \right) \le\frac1{n\lambda^2\varepsilon^2} \longrightarrow0. \]

Consequently,

\[ \overline X_n-X_{(1)} \xrightarrow{P}\frac1\lambda. \]

The positive limit keeps the estimated denominator away from zero with probability tending to one. Continuity of the reciprocal therefore yields

\[ \widehat\lambda_n\xrightarrow{P}\lambda. \]

Componentwise convergence in probability implies joint convergence, so

\[ { (\widehat\lambda_n,\widehat a_n) \xrightarrow{P}(\lambda,a). } \]

Let \(H_n=\overline X_n-X_{(1)}\). Its error decomposes as

\[ H_n-\lambda^{-1} =\left(\overline X_n-a-\lambda^{-1}\right) -(X_{(1)}-a). \]

Therefore, for \(\delta>0\),

\[ \begin{aligned} \mathbb P(|H_n-\lambda^{-1}|>\delta) &\le \mathbb P\left(\left|\overline X_n-a-\lambda^{-1}\right|>\delta/2\right) +\mathbb P(X_{(1)}-a>\delta/2)\\ &\le\frac4{n\lambda^2\delta^2}+e^{-n\lambda\delta/2} \longrightarrow0. \end{aligned} \]

On \(\{|H_n-\lambda^{-1}|\le(2\lambda)^{-1}\}\),

\[ H_n\ge\frac1{2\lambda}, \qquad \left|\frac1{H_n}-\lambda\right| =\frac{\lambda|H_n-\lambda^{-1}|}{H_n} \le2\lambda^2|H_n-\lambda^{-1}|. \]

Given \(\varepsilon>0\), set

\[ \delta_\varepsilon =\min((2\lambda)^{-1},\varepsilon/(2\lambda^2)). \]

Ignoring the already specified null event,

\[ \mathbb P(|\widehat\lambda_n-\lambda|>\varepsilon) \le\mathbb P(|H_n-\lambda^{-1}|>\delta_\varepsilon) \longrightarrow0. \]

For Euclidean joint convergence,

\[ \begin{aligned} &\mathbb P\left( \left\|(\widehat\lambda_n-\lambda,\widehat a_n-a)\right\|_2>\varepsilon \right)\\ &\quad\le \mathbb P(|\widehat\lambda_n-\lambda|>\varepsilon/\sqrt2) +\mathbb P(|\widehat a_n-a|>\varepsilon/\sqrt2) \longrightarrow0. \end{aligned} \]

Source: A. W. van der Vaart, Asymptotic Statistics, Chapter 2, “Stochastic Convergence”, pp. 5–24. The questions above are stated in the notation used in these solutions.