Stochastic Convergence Exercises: Slutsky, Tightness, and Order Symbols

A deterministic error that tends to zero remains small after multiplication by a bounded sequence. What replaces boundedness when both factors are random? Tightness gives the required control: restrict one factor to a large bounded set, and estimate the probability of its complement. The exercises below develop this argument, interpret stochastic order notation, and explain why a local expansion may fail when its random argument does not approach the expansion point.

These are Exercises 9, 10, 11, 12, 3, 18 from A. W. van der Vaart, Asymptotic Statistics (1998), Chapter 2, p. 24. The question numbers identify the original exercises. Write \(X_n\Rightarrow X\) for convergence in distribution and \(X_n\xrightarrow{P}X\) for convergence in probability.

For the underlying definitions, see modes of stochastic convergence.

1. Deterministic limits

Question (Exercise 9). If \(\mathbb P(X_n=x_n)=1\) and \(x_n\to x\), prove \(X_n\Rightarrow x\): (i) using distribution functions; (ii) using Theorem 2.7 of the chapter.

Answer.

(i) Using distribution functions. A point mass has a single jump in its distribution function. To prove weak convergence, we compare the functions away from the limiting jump:

\[ F_n(t)=\mathbf1_{\{x_n\le t\}}, \qquad F(t)=\mathbf1_{\{x\le t\}}, \]

where \(F\) is the distribution function of the point mass at \(x\).

If \(t<x\), choose \(\delta=(x-t)/2\). Eventually \(|x_n-x|<\delta\), so

\[ x_n>x-\delta=\frac{x+t}{2}>t, \qquad F_n(t)=0=F(t). \]

If \(t>x\), choose \(\delta=(t-x)/2\). Eventually,

\[ x_n<x+\delta=\frac{x+t}{2}<t, \qquad F_n(t)=1=F(t). \]

These cover all continuity points of \(F\); convergence at \(t=x\) is not required. Hence \(X_n\Rightarrow x\).

(ii) Using Theorem 2.7. For every \(\varepsilon>0\),

\[ \mathbb P(|X_n-x|>\varepsilon) = \mathbf1_{\{|x_n-x|>\varepsilon\}} \longrightarrow0. \]

Thus \(X_n\xrightarrow{P}x\). Theorem 2.7(ii), which states that convergence in probability implies convergence in distribution, gives

\[ {X_n\Rightarrow x.} \]

The same argument extends to random limits by allowing a small error in the threshold. Outside the event where the approximation error is large, the following inclusions compare the two distribution functions. If \(Y_n\xrightarrow{P}Y\), then, for every \(t\in\mathbb R\) and \(\varepsilon>0\),

\[ \begin{aligned} \{Y\le t-\varepsilon\}\cap\{|Y_n-Y|\le\varepsilon\} &\subseteq\{Y_n\le t\},\\ \{Y_n\le t\}\cap\{|Y_n-Y|\le\varepsilon\} &\subseteq\{Y\le t+\varepsilon\}. \end{aligned} \]

Taking probabilities gives

\[ F_Y(t-\varepsilon)-\mathbb P(|Y_n-Y|>\varepsilon) \le F_{Y_n}(t) \le F_Y(t+\varepsilon)+\mathbb P(|Y_n-Y|>\varepsilon). \]

For a continuity point \(t\) of \(F_Y\), first let \(n\to\infty\), and then let \(\varepsilon\downarrow0\). The resulting bounds are

\[ F_Y(t) \le\liminf_nF_{Y_n}(t) \le\limsup_nF_{Y_n}(t) \le F_Y(t). \]

Thus \(F_{Y_n}(t)\to F_Y(t)\).

For the converse when the limit is a constant \(c\), define

\[ g_\varepsilon(y)=\min(1,|y-c|/\varepsilon). \]

This function is bounded and continuous, \(g_\varepsilon(c)=0\), and

\[ \mathbf1_{\{|y-c|>\varepsilon\}}\le g_\varepsilon(y). \]

If \(Y_n\Rightarrow c\), the bounded-continuous criterion therefore implies

\[ \mathbb P(|Y_n-c|>\varepsilon) \le\mathbb{E} g_\varepsilon(Y_n) \longrightarrow g_\varepsilon(c)=0. \]

This proves the convergence-to-a-constant equivalence also used in Exercise 18.

2. Adding a negligible term to a tight sequence

Question (Exercise 10). State the rule \(o_P(1)+O_P(1)=O_P(1)\) in terms of random vectors and prove it.

Answer.

For random vectors in \(\mathbb R^d\),

\[ A_n=o_P(1) \quad\Longleftrightarrow\quad \forall\varepsilon>0,\ \mathbb P(\|A_n\|>\varepsilon)\longrightarrow0. \]

Also, \(B_n=O_P(1)\) means

\[ \forall\eta>0\ \exists M<\infty,\ N\in\mathbb N: \quad \sup_{n\ge N}\mathbb P(\|B_n\|>M)<\eta. \]

This is equivalent to uniform tightness: increasing \(M\) handles the finitely many initial terms.

The rule asserts

\[ A_n\xrightarrow{P}0,\quad B_n=O_P(1) \quad\Longrightarrow\quad A_n+B_n=O_P(1). \]

Fix \(\eta>0\). Choose \(M<\infty\) and \(N_1\) such that

\[ \mathbb P(\|B_n\|>M)<\frac{\eta}{2} \qquad(n\ge N_1). \]

Since \(A_n\xrightarrow{P}0\), choose \(N_2\) such that

\[ \mathbb P(\|A_n\|>1)<\frac{\eta}{2} \qquad(n\ge N_2). \]

The triangle inequality implies

\[ \{\|A_n+B_n\|>M+1\} \subseteq \{\|A_n\|>1\}\cup\{\|B_n\|>M\}. \]

Therefore, for \(n\ge\max(N_1,N_2)\),

\[ \begin{aligned} \mathbb P(\|A_n+B_n\|>M+1) &\le \mathbb P(\|A_n\|>1) +\mathbb P(\|B_n\|>M)\\ &<\eta. \end{aligned} \]

This proves

\[ {A_n+B_n=O_P(1).} \]

The argument uses only the triangle inequality and the union bound, so it applies to any joint distribution of the two sequences. Independence would add no useful information here.

Suppose that, for each \(\eta>0\), there exist \(M_0<\infty\) and \(N\) such that

\[ \sup_{n\ge N}\mathbb P(\|B_n\|>M_0)<\eta. \]

For each of the finitely many indices \(1\le j<N\), finiteness of \(B_j\) almost surely implies

\[ \mathbb P(\|B_j\|>M)\longrightarrow0\qquad(M\to\infty). \]

Choose \(M_j\) with \(\mathbb P(\|B_j\|>M_j)<\eta\), and put

\[ M=\max(M_0,M_1,\ldots,M_{N-1}). \]

If \(N=1\), take \(M=M_0\). Monotonicity of the tail probabilities gives

\[ \sup_{n\ge1}\mathbb P(\|B_n\|>M) \le\max\left( \max_{1\le j<N}\mathbb P(\|B_j\|>M_j), \sup_{n\ge N}\mathbb P(\|B_n\|>M_0) \right)<\eta, \]

with the finite maximum omitted when \(N=1\). Thus the eventual definition of \(O_P(1)\) is equivalent to uniform tightness.

3. Order symbols express implications

Question (Exercise 11). In what sense is \(o_P(1)=O_P(1)\) true? Is \(O_P(1)=o_P(1)\) true?

Answer.

Here the equality sign records an implication between classes of sequences. Written without order notation, the assertion is

\[ {A_n=o_P(1)\quad\Longrightarrow\quad A_n=O_P(1).} \]

Indeed, \(A_n=o_P(1)\) gives

\[ \mathbb P(\|A_n\|>1)\longrightarrow0. \]

Given \(\eta>0\), this probability is less than \(\eta\) for all sufficiently large \(n\). Thus the definition of \(O_P(1)\) holds with \(M=1\).

The converse is false. Take a fixed vector \(v\ne0\) and let

\[ A_n=v\qquad\text{almost surely for every }n. \]

For \(M\ge\|v\|\),

\[ \mathbb P(\|A_n\|>M)=0, \]

so \(A_n=O_P(1)\). However,

\[ \mathbb P\!\left(\|A_n\|>\frac{\|v\|}{2}\right)=1 \qquad\text{for every }n, \]

so \(A_n\ne o_P(1)\).

Consequently, when viewed as classes of sequences,

\[ {o_P(1)\subsetneq O_P(1).} \]

The equality sign in \(o_P(1)=O_P(1)\) is conventional shorthand for inclusion, not a symmetric equality of these classes.

Let \(\mathcal o\) and \(\mathcal O\) denote the classes of sequences that are \(o_P(1)\) and \(O_P(1)\), respectively. There is a related exact set identity for the sum rule:

\[ \mathcal o+\mathcal O=\mathcal O. \]

Exercise 10 proves the inclusion from left to right. For the reverse inclusion, every \((B_n)\in\mathcal O\) has the decomposition

\[ B_n=0+B_n, \qquad (0)\in\mathcal o. \]

This set identity for sums does not identify \(\mathcal o\) with \(\mathcal O\).

4. A local remainder needs a local argument

Question (Exercise 12). Show why the local remainder rules in Lemma 2.12 cannot be applied by formal substitution alone.

Find \(R(h)=o(|h|)\) as \(h\to0\) and random variables \(X_n\) for which \(R(X_n)\ne o_P(X_n)\).
Find \(R(h)=O(|h|)\) as \(h\to0\) and \(X_n=O_P(1)\) for which \(R(X_n)\ne O_P(X_n)\).

Answer.

A local bound controls the remainder only near zero. To use it at a random argument, the argument must enter that neighbourhood with probability tending to one; the lemma therefore requires \(X_n\xrightarrow{P}0\). Both examples below show why this condition matters. See Lemma 2.12, p. 13.

(i) Take

\[ R(h)=h^2,\qquad X_n=1\quad\text{almost surely}. \]

Then

\[ \frac{|R(h)|}{|h|}=|h|\longrightarrow0 \qquad(h\to0), \]

so \(R(h)=o(|h|)\). However,

\[ \frac{R(X_n)}{X_n}=1 \quad\text{almost surely for every }n. \]

In particular, the probability that the absolute ratio exceeds \(1/2\) is one for every \(n\), so it does not converge to zero in probability. Thus

\[ {R(X_n)\ne o_P(X_n).} \]

(ii) Define the finite-valued Borel function

\[ R(h)= \begin{cases} \dfrac{h}{1-h},&h\ne1,\\[2mm] 0,&h=1, \end{cases} \qquad X_n=\frac{n}{n+1}\quad\text{almost surely}. \]

If \(|h|\le\frac12\), then

\[ |R(h)|=\frac{|h|}{|1-h|}\le2|h|. \]

Hence \(R(h)=O(|h|)\) as \(h\to0\).

Also \(0<X_n<1\), so \(X_n=O_P(1)\). But

\[ R(X_n) = \frac{n/(n+1)}{1-n/(n+1)} =n, \qquad \frac{R(X_n)}{X_n}=n+1. \]

The latter sequence is not bounded in probability: for every \(M<\infty\),

\[ \mathbb P\!\left(\left|\frac{R(X_n)}{X_n}\right|>M\right)=1 \quad\text{eventually}. \]

Therefore

\[ {X_n=O_P(1),\qquad R(X_n)\ne O_P(X_n).} \]

The condition on \(R\) only controls its behavior near zero, whereas this sequence approaches \(1\).

The function in (ii) is Borel measurable and finite everywhere, including at \(h=1\). Because \(X_n\ne0\), a representation \(R(X_n)=Y_nX_n\) forces \(Y_n=n+1\); a different choice of bounded factor cannot repair the conclusion. Lemma 2.12 requires \(X_n\xrightarrow{P}0\), precisely the condition missing in both examples.

5. Multiplication by a consistent factor

Question (Exercise 3). If \(X_n\Rightarrow \mathcal{N}(0,1)\) and \(Y_n\xrightarrow{P}\sigma\), show that \(X_nY_n\Rightarrow \mathcal{N}(0,\sigma^2)\).

Answer.

Subtract the proposed limiting multiplier and estimate the resulting error. It is enough to prove that \(X_n(Y_n-\sigma)\xrightarrow{P}0\). This uses tightness of \(X_n\), not independence of the two factors.

Let \(Z\sim \mathcal{N}(0,1)\). Weak convergence \(X_n\Rightarrow Z\) implies boundedness in probability. In this case it also follows directly from the normal distribution function: for \(M>0\),

\[ \mathbb P(|X_n|>M) \le1-F_n(M)+F_n(-M) \longrightarrow1-\Phi(M)+\Phi(-M). \]

The expression on the right tends to zero as \(M\to\infty\). Given \(\eta>0\), choose \(M>0\) and then \(N\) such that

\[ \mathbb P(|X_n|>M)<\eta\qquad(n\ge N). \]

Put \(R_n=X_n(Y_n-\sigma)\). For every \(\varepsilon>0\),

\[ \begin{aligned} \{|R_n|>\varepsilon\} &\subseteq \{|X_n|>M\}\cup \{|Y_n-\sigma|>\varepsilon/M\},\\ \limsup_{n\to\infty}\mathbb P(|R_n|>\varepsilon) &\le\eta+0. \end{aligned} \]

Since \(\eta\) is arbitrary, \(R_n\xrightarrow{P}0\). Therefore

\[ X_nY_n=\sigma X_n+R_n\Rightarrow\sigma Z. \]

The characteristic function of the limiting variable is

\[ \mathbb{E} e^{it\sigma Z} =\exp\left(-\frac{\sigma^2t^2}{2}\right), \]

which identifies the \(\mathcal{N}(0,\sigma^2)\) law for every real \(\sigma\), including \(\sigma=0\).

6. A limiting distribution implies consistency

Question (Exercise 18). If \(\sqrt n(T_n-\theta)\) converges in distribution, show that \(T_n\xrightarrow{P}\theta\).

Answer.

Let \(W_n=\sqrt n(T_n-\theta)\Rightarrow W\), where \(W\) is finite almost surely. For \(M>0\), the set \(\{w:|w|\ge M\}\) is closed, and the portmanteau inequality gives

\[ \limsup_{n\to\infty}\mathbb P(|W_n|\ge M) \le\mathbb P(|W|\ge M). \]

Fix \(\varepsilon>0\). Once \(\varepsilon\sqrt n\ge M\),

\[ \begin{aligned} \mathbb P(|T_n-\theta|>\varepsilon) &=\mathbb P(|W_n|>\varepsilon\sqrt n)\\ &\le\mathbb P(|W_n|\ge M). \end{aligned} \]

Therefore, for every \(M>0\),

\[ \limsup_{n\to\infty}\mathbb P(|T_n-\theta|>\varepsilon) \le\mathbb P(|W|\ge M). \]

As \(M\to\infty\), the right-hand side decreases to zero. Tightness bounds the scaled error outside an event of arbitrarily small probability. Dividing that bound by the diverging scale then proves consistency; here the scale is \(\sqrt n\). The same proof works for random vectors upon replacing absolute values by Euclidean norms.

Source: A. W. van der Vaart, Asymptotic Statistics, Chapter 2, “Stochastic Convergence”, pp. 5–24. The questions above are stated in the notation used in these solutions.