Stochastic Convergence Exercises: Joint Laws and Continuity Sets

Knowing the limiting law of each coordinate leaves their dependence undetermined. The first two exercises compare an alternating dependence pattern with independence, which does determine the joint limit. The remaining exercises ask how much weak convergence says about individual sets: a common integer support gives convergence in total variation, whereas a rational grid approaching the uniform law still assigns the wrong probability to the set of rational points.

These are Exercises 5, 6, 7, 8 from A. W. van der Vaart, Asymptotic Statistics (1998), Chapter 2, p. 24. The question numbers identify the original exercises. Write \(X_n\Rightarrow X\) for convergence in distribution and \(X_n\xrightarrow{P}X\) for convergence in probability.

For the underlying definitions, see weak convergence and tightness.

1. Marginals do not determine joint convergence

Question (Exercise 5). Find sequences with \(X_n\Rightarrow X\) and \(Y_n\Rightarrow Y\) for which \((X_n,Y_n)\) does not converge in distribution.

Answer.

Let \(R\) be a Rademacher random variable:

\[ \mathbb P(R=1)=\mathbb P(R=-1)=\frac12. \]

Define

\[ X_n=R,\qquad Y_n=(-1)^nR. \]

For every \(n\), both \(X_n\) and \(Y_n\) have the same distribution as \(R\). Hence

\[ X_n\Rightarrow R,\qquad Y_n\Rightarrow R. \]

The coordinates agree at even indices and have opposite signs at odd indices. This change leaves both marginals unchanged but produces two different joint laws:

\[ (X_{2m},Y_{2m})=(R,R), \qquad (X_{2m+1},Y_{2m+1})=(R,-R). \]

In terms of point masses, these two laws are

\[ \begin{aligned} \mu_{\mathrm{even}} &=\frac{1}{2}\delta_{(1,1)}+\frac{1}{2}\delta_{(-1,-1)},\\ \mu_{\mathrm{odd}} &=\frac{1}{2}\delta_{(1,-1)}+\frac{1}{2}\delta_{(-1,1)}. \end{aligned} \]

Each coordinate marginal still assigns mass \(1/2\) to \(1\) and \(-1\).

A product detects the sign of this dependence. To make it a valid test for weak convergence, replace the product by the following bounded continuous function

\[ f(x,y)=\frac{xy}{1+|xy|}. \]

The expectations along the two subsequences are

\[ \begin{aligned} \int f\,d\mu_{\mathrm{even}} &=\frac{1}{2}\frac{1}{2}+\frac{1}{2}\frac{1}{2}=\frac{1}{2},\\ \int f\,d\mu_{\mathrm{odd}} &=\frac{1}{2}\frac{-1}{2}+\frac{1}{2}\frac{-1}{2}=-\frac{1}{2}. \end{aligned} \]

Thus \(\mathbb{E} f(X_n,Y_n)=(-1)^n/2\), which does not converge. Weak convergence would force convergence of the expectation of every bounded continuous function. Therefore

\[ {(X_n,Y_n)\text{ does not converge in distribution}.} \]

The boundedness condition matters: an oscillating expectation of an unbounded function would not, on its own, rule out weak convergence. Here \(|f|<1\) everywhere, so the bounded-continuous characterization of weak convergence applies directly.

2. Independent marginals and their joint limit

Question (Exercise 6). Suppose \(X_n\) and \(Y_n\) are independent random vectors for every \(n\), and \(X_n\Rightarrow X\), \(Y_n\Rightarrow Y\). Show that the joint laws converge to the product of the limiting marginal laws.

Answer.

Suppose

\[ X_n\in\mathbb R^d,\qquad Y_n\in\mathbb R^m, \]

with \(X_n\) independent of \(Y_n\) for every \(n\), and

\[ X_n\Rightarrow X,\qquad Y_n\Rightarrow Y. \]

Let \(\mu=\mathcal L(X)\) and \(\nu=\mathcal L(Y)\). The precise conclusion is

\[ {\mathcal L(X_n,Y_n)\Rightarrow\mu\otimes\nu.} \]

For \(s\in\mathbb R^d\) and \(t\in\mathbb R^m\), independence gives

\[ \begin{aligned} \varphi_{(X_n,Y_n)}(s,t) &= \mathbb{E}\exp\!\bigl(i(s^\top X_n+t^\top Y_n)\bigr)\\ &= \mathbb{E} e^{is^\top X_n}\, \mathbb{E} e^{it^\top Y_n}\\ &= \varphi_{X_n}(s)\varphi_{Y_n}(t). \end{aligned} \]

Marginal weak convergence implies convergence of the characteristic functions, so

\[ \varphi_{(X_n,Y_n)}(s,t) \longrightarrow \varphi_X(s)\varphi_Y(t). \]

Independence has reduced the joint characteristic function to two marginal factors, and both factors converge. Their product is the characteristic function of \(\mu\otimes\nu\). Lévy’s continuity theorem therefore proves the asserted joint convergence.

Equivalently, choose independent random vectors \(X^\ast,Y^\ast\) with

\[ \mathcal L(X^\ast)=\mu,\qquad \mathcal L(Y^\ast)=\nu. \]

Then

\[ (X_n,Y_n)\Rightarrow(X^\ast,Y^\ast). \]

The exercise denotes this independent pair by \((X,Y)\). Marginal convergence does not determine a preassigned coupling of the limiting variables.

To construct this independent pair, use the product probability space

\[ (\mathbb R^d\times\mathbb R^m, \mathcal B(\mathbb R^d)\otimes\mathcal B(\mathbb R^m), \mu\otimes\nu), \]

and define \(X^\ast(x,y)=x\) and \(Y^\ast(x,y)=y\). For Borel sets \(A,B\),

\[ \begin{aligned} \mathbb P(X^\ast\in A,Y^\ast\in B) &=(\mu\otimes\nu)(A\times B)\\ &=\mu(A)\nu(B)\\ &=\mathbb P(X^\ast\in A)\mathbb P(Y^\ast\in B). \end{aligned} \]

Thus these coordinate maps are independent and have the desired marginal laws. Fubini’s theorem applies because the complex exponential has modulus one, and gives

\[ \begin{aligned} \varphi_{(X^\ast,Y^\ast)}(s,t) &=\int\!\int e^{i(s^\top x+t^\top y)}\,\mu(dx)\nu(dy)\\ &=\left(\int e^{is^\top x}\,\mu(dx)\right) \left(\int e^{it^\top y}\,\nu(dy)\right)\\ &=\varphi_X(s)\varphi_Y(t). \end{aligned} \]

Weak convergence of the marginals implies convergence of these characteristic functions by applying the bounded-continuous criterion to their real and imaginary parts. The limiting product is continuous at zero, since dominated convergence gives continuity at zero of each factor. This verifies the hypotheses of Lévy’s continuity theorem for the joint vectors. No independence assumptions between different sample sizes are used.

3. Integer-valued laws and total variation

Question (Exercise 7). If \(X_n\) and \(X\) are integer-valued, show that \(X_n\Rightarrow X\) if and only if \(\mathbb P(X_n=k)\to\mathbb P(X=k)\) for every integer \(k\).

Answer.

Put

\[ p_n(k)=\mathbb P(X_n=k),\qquad p(k)=\mathbb P(X=k),\qquad k\in\mathbb Z. \]

Suppose first that \(X_n\Rightarrow X\). Let \(F_n,F\) be their distribution functions. Since \(X\) is integer-valued, \(F\) is continuous at every noninteger. Thus

\[ \begin{aligned} p_n(k) &= F_n\!\left(k+\frac12\right) -F_n\!\left(k-\frac12\right)\\ &\longrightarrow F\!\left(k+\frac12\right) -F\!\left(k-\frac12\right) =p(k). \end{aligned} \]

Conversely, suppose \(p_n(k)\to p(k)\) for every \(k\in\mathbb Z\). Since both mass functions sum to \(1\),

\[ \sum_{k\in\mathbb Z}|p_n(k)-p(k)| = 2-2\sum_{k\in\mathbb Z}\min(p_n(k),p(k)). \]

For each \(k\),

\[ \min(p_n(k),p(k))\longrightarrow p(k), \qquad 0\le\min(p_n(k),p(k))\le p(k). \]

Because \(\sum_k p(k)=1\), dominated convergence for the series yields

\[ \sum_{k\in\mathbb Z}\min(p_n(k),p(k))\longrightarrow1. \]

Consequently,

\[ \sum_{k\in\mathbb Z}|p_n(k)-p(k)|\longrightarrow0. \]

For every \(t\in\mathbb R\),

\[ |F_n(t)-F(t)| = \left|\sum_{\substack{k\in\mathbb Z\\k\le t}} \bigl(p_n(k)-p(k)\bigr)\right| \le \sum_{k\in\mathbb Z}|p_n(k)-p(k)|. \]

Thus \(F_n\to F\) uniformly, which implies \(X_n\Rightarrow X\). Hence

\[ { X_n\Rightarrow X \quad\Longleftrightarrow\quad \forall k\in\mathbb Z,\ \mathbb P(X_n=k)\to\mathbb P(X=k). } \]

The summability step can also be proved by first keeping a finite set that carries almost all of the limiting mass. Convergence on that finite set then prevents mass from escaping into its complement. Fix \(\varepsilon>0\) and choose \(m\) such that

\[ r_m\triangleq\sum_{|k|>m}p(k)<\varepsilon. \]

Set

\[ D_{n,m}=\sum_{|k|\le m}|p_n(k)-p(k)|\longrightarrow0. \]

Since both mass functions sum to one,

\[ \begin{aligned} \sum_{|k|>m}p_n(k) &=1-\sum_{|k|\le m}p_n(k)\\ &\le1-\sum_{|k|\le m}p(k)+D_{n,m}\\ &=r_m+D_{n,m}. \end{aligned} \]

Therefore

\[ \begin{aligned} \sum_{k\in\mathbb Z}|p_n(k)-p(k)| &\le D_{n,m}+\sum_{|k|>m}p_n(k)+\sum_{|k|>m}p(k)\\ &\le2D_{n,m}+2r_m,\\ \limsup_{n\to\infty}\sum_k|p_n(k)-p(k)| &\le2\varepsilon. \end{aligned} \]

Letting \(\varepsilon\downarrow0\) gives the asserted \(\ell^1\) convergence.

In fact the conclusion is convergence in total variation. If \(d_n(k)=p_n(k)-p(k)\), then \(\sum_kd_n(k)=0\), so

\[ \sum_{d_n(k)>0}d_n(k) =-\sum_{d_n(k)<0}d_n(k) =\frac12\sum_k|d_n(k)|. \]

The set \(\{k:d_n(k)>0\}\) attains the largest possible positive difference of probabilities. Hence

\[ \sup_{B\in\mathcal B(\mathbb R)} |\mathbb P(X_n\in B)-\mathbb P(X\in B)| =\frac12\sum_{k\in\mathbb Z}|p_n(k)-p(k)|\longrightarrow0. \]

4. A fine grid and a limiting null set

Question (Exercise 8). Let \(\mathbb P(X_n=i/n)=1/n\) for \(i=1,\ldots,n\). Find the weak limit \(X\), and a Borel set \(B\) such that \(\mathbb P(X_n\in B)=1\) for every \(n\) but \(\mathbb P(X\in B)=0\).

Answer.

Assume

\[ \mathbb P(X_n=i/n)=\frac1n,\qquad i=1,\ldots,n. \]

For \(0\le t<1\), the integers with \(i/n\le t\) are exactly \(1,\ldots,\lfloor nt\rfloor\) (none when \(\lfloor nt\rfloor=0\)). Thus

\[ \mathbb P(X_n\le t) =\frac{\#\{i\in\{1,\ldots,n\}:i\le nt\}}n =\frac{\lfloor nt\rfloor}{n}. \]

The full distribution function is

\[ F_n(t)= \begin{cases} 0,&t<0,\\[2mm] \dfrac{\lfloor nt\rfloor}{n},&0\le t<1,\\[2mm] 1,&t\ge1. \end{cases} \]

For \(0\le t<1\),

\[ 0\le t-\frac{\lfloor nt\rfloor}{n}<\frac1n. \]

Therefore \(F_n\) converges uniformly to

\[ F(t)= \begin{cases} 0,&t<0,\\ t,&0\le t<1,\\ 1,&t\ge1, \end{cases} \]

the distribution function of \(X\sim\operatorname{Uniform}[0,1]\). Hence

\[ {X_n\Rightarrow X\sim\operatorname{Uniform}[0,1].} \]

Uniform convergence of distribution functions still does not control every Borel set. To see the obstruction, take the countable Borel set

\[ B=\mathbb Q\cap[0,1]. \]

Every possible value \(i/n\) belongs to \(B\), so

\[ \mathbb P(X_n\in B)=1\qquad\text{for every }n. \]

But the uniform distribution assigns probability zero to every countable set:

\[ \mathbb P(X\in B) =\sum_{q\in\mathbb Q\cap[0,1]}\mathbb P(X=q)=0. \]

Every open interval meeting \((0,1)\) contains both rational and irrational points. Thus \(B\) has empty interior and closure \([0,1]\), giving

\[ \partial B=[0,1], \qquad \mathbb P(X\in\partial B)=1, \]

so \(B\) is not a continuity set of the limiting law.

There is also a direct coupling: round a single uniform variable up to the nearest grid point. Take \(U\sim\operatorname{Uniform}(0,1)\) and

\[ \widetilde X_n=\frac{\lceil nU\rceil}{n}. \]

Indeed,

\[ \mathbb P(\widetilde X_n=i/n) =\mathbb P((i-1)/n<U\le i/n)=\frac1n, \]

and

\[ 0\le\widetilde X_n-U<\frac1n. \]

This constructed sequence converges almost surely to \(U\), which also proves convergence of the prescribed laws. It does not impose a coupling on an arbitrary original sequence \((X_n)\).

Source: A. W. van der Vaart, Asymptotic Statistics, Chapter 2, “Stochastic Convergence”, pp. 5–24. The questions above are stated in the notation used in these solutions.