Modes of Stochastic Convergence

A statement that random variables converge is incomplete until the mode of convergence is specified. The definitions below distinguish convergence of paths, probabilities, and laws. We then develop three tools used in asymptotic calculations: continuous mappings, boundedness in probability, and control of moments through uniform integrability.

Definition 1. A sequence of random vectors, denoted $(X_n)$ is said to converge in probability to $X$ if, for every $\varepsilon>0$,

\[\mathop{\mathrm{\mathds{P}}}\left( \abs{X_n - X} \geqslant\varepsilon\right) \xrightarrow{n \to \infty} 0\]

This is denoted ${X_n}\xrightarrow{\mathds{P}}X$.

Definition 2. A sequence of random vectors, denoted $(X_n)$ is said to converge in distribution to $X$ if

\[\mathop{\mathrm{\mathds{P}}}(X_n \leqslant x) \xrightarrow{n \to \infty} \mathop{\mathrm{\mathds{P}}}(X \leqslant x)\]

at all continuity points $x$ of the map $x \mapsto \mathop{\mathrm{\mathds{P}}}(X \leqslant x)$. This is denoted ${X_n}\xrightarrow{\mathrm{d}}X$.

Definition 3. A sequence of random vectors, denoted $(X_n)$ is said to converge almost surely to $X$ if

\[\mathop{\mathrm{\mathds{P}}}\left( \lim\limits_{n \to \infty} \abs{X_n - X} = 0 \right) = 1\]

This is denoted ${X_n}\xrightarrow{\mathrm{a.s.}}X$.

Theorem 4 (Portmanteau). The following are equivalent for random vectors $X_n$ and random vector $X$:

$\mathop{\mathrm{\mathds{P}}}(X_n \leqslant x) \to \mathop{\mathrm{\mathds{P}}}(X \leqslant x)$ at all continuity points of $x \mapsto \mathop{\mathrm{\mathds{P}}}(X \leqslant x)$
$\mathbb{E}f(X_n) \to \mathbb{E}f(X)$ for all bounded, continuous functions $f$
$\mathbb{E}f(X_n) \to \mathbb{E}f(X)$ for all bounded, Lipschitz functions $f$
$\liminf \mathbb{E}f(X_n) \geqslant\mathbb{E}f(X)$ for all nonnegative, continuous functions $f$
$\liminf \mathop{\mathrm{\mathds{P}}}(X_n \in G) \geqslant\mathop{\mathrm{\mathds{P}}}(X \in G)$ for all open sets $G$
$\limsup \mathop{\mathrm{\mathds{P}}}(X_n \in H) \leqslant\mathop{\mathrm{\mathds{P}}}(X \in H)$ for all closed sets $H$
$\mathop{\mathrm{\mathds{P}}}(X_n \in B) \to \mathop{\mathrm{\mathds{P}}}(X \in B)$ for all Borel sets $B$ with $\mathop{\mathrm{\mathds{P}}}(X \in \partial B) = 0$.

All convergence statements in this theorem are understood as $n \to \infty$.

Proof. Number the assertions (i)–(vii) in their displayed order. Write $F_n(x)=\mathbb P(X_n\leq x)$ and $F(x)=\mathbb P(X\leq x)$, with inequalities between vectors understood coordinatewise.

(i) ⇒ (ii). Let $f$ be bounded and continuous. By rescaling, it suffices to consider $|f|\leq1$. Fix $\varepsilon>0$ and choose a large half-open rectangle $I=\prod_{r=1}^d(a_r,b_r]$ with $\mathbb P(X\notin I)<\varepsilon$. Choose every endpoint outside the atoms of the corresponding marginal of $X$. This is possible because each marginal has at most countably many atoms.

At every vertex of such a rectangle, $F$ is continuous: a small change in the coordinates can change the lower-orthant event only when a coordinate of $X$ lies in a small interval around the corresponding vertex coordinate. Those probabilities tend to zero because the coordinates are not atoms. For any rectangle $J=\prod_{r=1}^d(u_r,v_r]$ with the same endpoint property, inclusion–exclusion gives

\[\mathbb P(X_n\in J) =\sum_{e\in\{0,1\}^d}(-1)^{d-\sum_r e_r}F_n(c_e) \longrightarrow \sum_{e\in\{0,1\}^d}(-1)^{d-\sum_r e_r}F(c_e) =\mathbb P(X\in J),\]

where $(c_e)_r=u_r$ if $e_r=0$ and $(c_e)_r=v_r$ if $e_r=1$. In particular, $\mathbb P(X_n\notin I)<\varepsilon$ for all sufficiently large $n$.

The function $f$ is uniformly continuous on the compact rectangle $\overline I$. Partition $I$ into finitely many disjoint half-open rectangles $I_1,\ldots,I_m$, using cut points that also avoid marginal atoms, so that the oscillation of $f$ on each cell is at most $\varepsilon$. Pick $x_j\in I_j$ and set

\[f_\varepsilon(x)=\sum_{j=1}^m f(x_j)\mathbf1_{I_j}(x).\]

Then $|f-f_\varepsilon|\leq\varepsilon$ on $I$, while $f_\varepsilon=0$ outside $I$. Consequently, for all sufficiently large $n$,

\[\begin{aligned} \left|\mathbb E f(X_n)-\mathbb E f_\varepsilon(X_n)\right| &\leq\varepsilon+\mathbb P(X_n\notin I)<2\varepsilon,\\ \left|\mathbb E f(X)-\mathbb E f_\varepsilon(X)\right| &\leq\varepsilon+\mathbb P(X\notin I)<2\varepsilon. \end{aligned}\]

For the step function itself, there are only finitely many cell probabilities to compare:

\[\left|\mathbb E f_\varepsilon(X_n)-\mathbb E f_\varepsilon(X)\right| \leq\sum_{j=1}^m|f(x_j)|\left|\mathbb P(X_n\in I_j)-\mathbb P(X\in I_j)\right| \longrightarrow0.\]

Taking $n$ large enough to make this last difference smaller than $\varepsilon$, the triangle inequality yields

\[\begin{aligned} \left|\mathbb E f(X_n)-\mathbb E f(X)\right| &\leq\left|\mathbb E f(X_n)-\mathbb E f_\varepsilon(X_n)\right|\\ &\quad+\left|\mathbb E f_\varepsilon(X_n)-\mathbb E f_\varepsilon(X)\right|\\ &\quad+\left|\mathbb E f_\varepsilon(X)-\mathbb E f(X)\right|\\ &<2\varepsilon+\varepsilon+2\varepsilon=5\varepsilon. \end{aligned}\]

Since $\varepsilon$ is arbitrary, (ii) follows. Choosing endpoints outside marginal atoms makes this argument valid even when $F$ has discontinuities elsewhere.

(ii) ⇒ (iii). Every bounded Lipschitz function is bounded and continuous.

(iii) ⇒ (v). Let $G$ be open; the cases $G=\varnothing$ and $G=\mathbb R^d$ are immediate. Define

\[f_m(x)=\min(1,m\,d(x,G^c)),\qquad d(x,G^c)=\inf_{y\in G^c}\|x-y\|.\]

Distance to a nonempty set is $1$-Lipschitz, so $f_m$ is bounded and Lipschitz. Moreover, $0\leq f_m\leq\mathbf1_G$ and $f_m\uparrow\mathbf1_G$: every point of the open set has positive distance from its complement. For each fixed $m$, (iii) gives

\[\mathbb E f_m(X)=\lim_{n\to\infty}\mathbb E f_m(X_n) \leq\liminf_{n\to\infty}\mathbb P(X_n\in G).\]

Letting $m\to\infty$ and applying monotone convergence proves (v).

(v) ⇔ (vi). If $H$ is closed, then $H^c$ is open. Taking complements gives

\[\begin{aligned} \limsup_{n\to\infty}\mathbb P(X_n\in H) &=1-\liminf_{n\to\infty}\mathbb P(X_n\in H^c)\\ &\leq1-\mathbb P(X\in H^c)=\mathbb P(X\in H). \end{aligned}\]

The same calculation starting with an open set and its closed complement proves the converse.

(v), (vi) ⇒ (vii). For a Borel set $B$, use $B^\circ\subseteq B\subseteq\overline B$ to obtain

\[\begin{aligned} \mathbb P(X\in B^\circ) &\leq\liminf_{n\to\infty}\mathbb P(X_n\in B)\\ &\leq\limsup_{n\to\infty}\mathbb P(X_n\in B) \leq\mathbb P(X\in\overline B). \end{aligned}\]

If $\mathbb P(X\in\partial B)=0$, the two outside probabilities both equal $\mathbb P(X\in B)$, since $\partial B=\overline B\setminus B^\circ$. The desired convergence follows by squeezing.

(vii) ⇒ (i). Fix a continuity point $x$ of $F$. Choose vectors $a^{(m)}\uparrow x$ and $b^{(m)}\downarrow x$, strictly below and above $x$ in every coordinate, with all coordinates avoiding marginal atoms. The lower orthants with these corners are continuity sets: their boundaries are contained in the union of the coordinate hyperplanes through their corners, each of which has probability zero. Thus (vii) and monotonicity give, for each fixed $m$,

\[\begin{aligned} F(a^{(m)}) &=\lim_{n\to\infty}F_n(a^{(m)}) \leq\liminf_{n\to\infty}F_n(x)\\ &\leq\limsup_{n\to\infty}F_n(x) \leq\lim_{n\to\infty}F_n(b^{(m)})=F(b^{(m)}). \end{aligned}\]

Let $m\to\infty$. Continuity of $F$ at $x$ makes both outside terms tend to $F(x)$, proving (i).

(ii) ⇒ (iv). Let $f$ be nonnegative and continuous, and fix $M>0$. The truncation $h_M=\min(f,M)$ is bounded and continuous, and $h_M\leq f$. Hence

\[\liminf_{n\to\infty}\mathbb E f(X_n) \geq\lim_{n\to\infty}\mathbb E h_M(X_n) =\mathbb E h_M(X).\]

Now let $M\to\infty$. Since $h_M(X)\uparrow f(X)$, monotone convergence gives $\mathbb E h_M(X)\uparrow\mathbb E f(X)$, including when this expectation is infinite. This proves (iv); the limit in $n$ is taken before the truncation level increases.

(iv) ⇒ (ii). For bounded continuous $f$, choose $M\geq\|f\|_\infty$. Both $M+f$ and $M-f$ are nonnegative and continuous. Applying (iv) separately to them gives

\[\begin{aligned} M+\liminf_{n\to\infty}\mathbb E f(X_n) &\geq M+\mathbb E f(X),\\ M-\limsup_{n\to\infty}\mathbb E f(X_n) &\geq M-\mathbb E f(X). \end{aligned}\]

After subtracting $M$ and rearranging,

\[\mathbb E f(X)\leq\liminf_{n\to\infty}\mathbb E f(X_n) \leq\limsup_{n\to\infty}\mathbb E f(X_n) \leq\mathbb E f(X).\]

Thus the expectations converge. Together, these implications prove all seven assertions equivalent. ◻

Theorem 5 (Continuous mapping). Let $g:\mathbb R^k\to\mathbb R^\ell$ be Borel measurable and continuous at every point of a Borel set $C$ with $P(X\in C)=1$. Then:

${X_n}\xrightarrow{\mathrm{a.s.}}X \implies {g(X_n)}\xrightarrow{\mathrm{a.s.}}g(X)$.
${X_n}\xrightarrow{\mathrm{d}}X \implies {g(X_n)}\xrightarrow{\mathrm{d}}g(X)$.
${X_n}\xrightarrow{\mathds{P}}X \implies {g(X_n)}\xrightarrow{\mathds{P}}g(X)$.

Proof. Borel measurability of $g$ ensures that $g(X_n)$ and $g(X)$ are random vectors. In the almost sure and in-probability assertions, the variables are defined on a common probability space.

(i) Almost sure convergence. Let $A=\{\omega:X_n(\omega)\to X(\omega)\}\cap\{\omega:X(\omega)\in C\}$. Both events have probability one, so $\mathbb P(A)=1$. Fix $\omega\in A$ and $\varepsilon>0$. Continuity of $g$ at $X(\omega)$ provides a $\delta>0$ such that

\[\|y-X(\omega)\|<\delta \quad\Longrightarrow\quad \|g(y)-g(X(\omega))\|<\varepsilon.\]

Since $X_n(\omega)\to X(\omega)$, the left inequality holds with $y=X_n(\omega)$ for every sufficiently large $n$. Thus $g(X_n(\omega))\to g(X(\omega))$ on $A$, proving almost sure convergence. The choices of $\delta$ and the starting index may depend on $\omega$; uniform continuity is not required.

(ii) Convergence in distribution. Let $H\subseteq\mathbb R^\ell$ be closed and put $B=g^{-1}(H)$. The set $B$ is Borel, but it need not be closed because $g$ need not be continuous everywhere. We claim that

\[\overline B\subseteq B\cup C^c.\]

Indeed, take $x\in\overline B\cap C$. There is a sequence $x_j\in B$ with $x_j\to x$. Continuity of $g$ at $x$ gives $g(x_j)\to g(x)$, and $g(x_j)\in H$ for every $j$. Since $H$ is closed, $g(x)\in H$, so $x\in B$. This proves the inclusion. Consequently, using $\mathbb P(X\in C^c)=0$,

\[\mathbb P(X\in\overline B) \leq\mathbb P(X\in B)+\mathbb P(X\in C^c) =\mathbb P(g(X)\in H).\]

Now apply the closed-set form of Portmanteau to the closed set $\overline B\subseteq\mathbb R^k$:

\[\begin{aligned} \limsup_{n\to\infty}\mathbb P(g(X_n)\in H) &=\limsup_{n\to\infty}\mathbb P(X_n\in B)\\ &\leq\limsup_{n\to\infty}\mathbb P(X_n\in\overline B)\\ &\leq\mathbb P(X\in\overline B)\\ &\leq\mathbb P(g(X)\in H). \end{aligned}\]

This holds for every closed $H$. The converse direction of Portmanteau, now in $\mathbb R^\ell$, therefore gives $g(X_n)\Rightarrow g(X)$.

(iii) Convergence in probability. We first extract an almost surely convergent further subsequence from any subsequence. Let $(X_{n_j})$ be arbitrary. Since $X_{n_j}\to X$ in probability, we can choose increasing indices $j_m$ such that

\[\mathbb P\left(\|X_{n_{j_m}}-X\|>2^{-m}\right)<2^{-m} \qquad(m\geq1).\]

The sum of these probabilities is finite. By the first Borel–Cantelli lemma, with probability one only finitely many of the events occur. Hence $\|X_{n_{j_m}}-X\|\leq2^{-m}$ eventually, and

\[X_{n_{j_m}}\longrightarrow X\quad\text{almost surely}. \qquad\text{By (i),}\qquad g(X_{n_{j_m}})\longrightarrow g(X)\quad\text{almost surely}.\]

For every fixed $\varepsilon>0$, the indicators of $\{\|g(X_{n_{j_m}})-g(X)\|>\varepsilon\}$ then tend to zero almost surely and are bounded by one. Dominated convergence gives

\[\mathbb P\left(\|g(X_{n_{j_m}})-g(X)\|>\varepsilon\right)\longrightarrow0.\]

To conclude for the entire sequence, suppose instead that $g(X_n)$ failed to converge to $g(X)$ in probability. Then there would be $\varepsilon,\eta>0$ and a subsequence $(n_j)$ such that

\[\mathbb P\left(\|g(X_{n_j})-g(X)\|>\varepsilon\right)\geq\eta \qquad\text{for every }j.\]

The construction above supplies a further subsequence along which these same probabilities tend to zero, a contradiction. Therefore $g(X_n)\to g(X)$ in probability. ◻

Theorem 6 (Prokhorov).

${X_n}\xrightarrow{\mathrm{d}}X \implies (X_n)$ is uniformly tight.
$(X_n)$ is uniformly tight implies there exists $X$ such that ${(X_{n_j})}\xrightarrow{\text{d}}X$.

Proof. Uniform tightness means that for every $\varepsilon>0$ some compact $K\subset\mathbb R^d$ satisfies $\sup_n\mathbb P(X_n\notin K)<\varepsilon$. In Euclidean space it is enough to use closed balls.

Suppose first that $X_n\Rightarrow X$. Fix $\varepsilon>0$ and choose $R$ with $\mathbb P(\|X\|\geq R)<\varepsilon$. This is possible because $X$ is finite almost surely. The set $\{x:\|x\|\geq R\}$ is closed, so

\[\limsup_n\mathbb P(\|X_n\|\geq R)\leq\mathbb P(\|X\|\geq R)<\varepsilon.\]

There is consequently $N$ such that the left-hand tail probability is below $\varepsilon$ for all $n\geq N$. Each of the finitely many variables $X_1,\ldots,X_{N-1}$ also has tails tending to zero. Increase $R$ to make their tail probabilities smaller than $\varepsilon$ as well. The resulting closed ball proves uniform tightness.

Conversely, write $\mu_n$ for the law of $X_n$. Helly selection, proved below without using this theorem, gives a subsequence converging vaguely to a measure $\mu$ with total mass at most one. Fix $\varepsilon>0$, choose a compact $K$ with $\mu_n(K)>1-\varepsilon$ for every $n$, and take $h\in C_c(\mathbb R^d)$ with $0\leq h\leq1$ and $h=1$ on $K$. For example, a sufficiently large ball contains $K$, and a continuous radial cutoff is one on that ball and zero outside a larger ball. Vague convergence gives

\[\int h\,d\mu=\lim_j\int h\,d\mu_{n_j}\geq1-\varepsilon.\]

Thus $1\geq\mu(\mathbb R^d)\geq1-\varepsilon$ for every $\varepsilon>0$, and $\mu$ is a probability measure. For the same cutoff, both $\int(1-h)\,d\mu_{n_j}\leq\varepsilon$ and $\int(1-h)\,d\mu\leq\varepsilon$. If $f$ is bounded and continuous, then $fh$ is continuous with compact support and

\[\left|\int f\,d\mu_{n_j}-\int f\,d\mu\right| \leq\left|\int fh\,d\mu_{n_j}-\int fh\,d\mu\right|+2\varepsilon\|f\|_\infty.\]

Let $j\to\infty$, then $\varepsilon\downarrow0$. Portmanteau proves weak convergence to $\mu$. A random vector $X$ with law $\mu$ may be realized as the identity map on $(\mathbb R^d,\mathcal B(\mathbb R^d),\mu)$. ◻

Lemma 7. (Helly selection). Every sequence of probability measures $\mu_n$ on $\mathbb R^d$ has a subsequence and a finite measure $\mu$ of total mass at most one such that $\int f\,d\mu_{n_j}\to\int f\,d\mu$ for every continuous compactly supported $f$. This is vague convergence; it need not be convergence in distribution to a probability law.

Proof.

Map each coordinate by $x\mapsto\frac12+\pi^{-1}\arctan x$ into $(0,1)$ and regard the image laws as measures on the compact cube $[0,1]^d$. On a compact cube, take nested finite dyadic partitions. A diagonal subsequence makes the mass of every cell converge. The limiting masses are nonnegative, sum to one at each level, and add consistently from children to parents. They define a probability on nested cell choices: divide the unit interval into intervals of these masses, and subdivide each interval according to its children. A uniform point selects a nested sequence, outside a countable set of endpoints.

The closures of the selected cells are nested compact sets whose diameters tend to zero, so they determine a unique point $Z$. Choose a representative $z_A$ in every cell $A$ of the level-$m$ partition $\mathcal P_m$. The representative selected by the uniform point is a measurable, finite-valued random vector $Z_m$; since $Z_m\to Z$, the limit $Z$ is measurable. Write $p_A$ for the limiting mass of cell $A$, and $\nu_n$ for the image laws on the cube. For continuous $f$ on the cube, uniform continuity supplies a modulus $\omega_f(\delta)\to0$. If $\delta_m$ is the largest cell diameter, then

\[\left|\int f\,d\nu_{n_j}-\sum_{A\in\mathcal P_m}f(z_A)\nu_{n_j}(A)\right|\leq\omega_f(\delta_m),\qquad \left|\mathbb E f(Z)-\sum_{A\in\mathcal P_m}f(z_A)p_A\right|\leq\omega_f(\delta_m).\]

For fixed $m$, the finite sums converge by construction. Hence the limit superior of $|\int f\,d\nu_{n_j}-\mathbb E f(Z)|$ is at most $2\omega_f(\delta_m)$; now let $m\to\infty$. This proves compact-cube weak selection, even if the limiting point lies on a cell boundary.

Restrict the law of $Z$ to the open cube and pull it back through the inverse coordinate map. The resulting measure $\mu$ has mass at most one. If $f\in C_c(\mathbb R^d)$, its transformed function is zero near the boundary of the cube, so extension by zero is continuous on the whole compact cube. Applying the convergence just proved gives $\int f\,d\mu_{n_j}\to\int f\,d\mu$. Boundary mass is the mass lost at infinity.

◻

Lemma 8. If ${X_n}\xrightarrow{\mathrm{d}}X$ and $X$ has a continuous distribution function, then

\[\sup_{x} \abs{\mathop{\mathrm{\mathds{P}}}(X_n \leqslant x) - \mathop{\mathrm{\mathds{P}}}(X \leqslant x)} \xrightarrow{n\to\infty} 0\]

Proof. Write $F_n,F$ for the distribution functions on $\mathbb R^d$. First, continuity of $F$ implies that each coordinate law is atomless. If $\mathbb P(X_r=a)>0$, then for some finite $b$ the event $\{X_r=a,\ X_j\leq b\ (j\ne r)\}$ has positive probability, by continuity from below. The jump of $F$ in coordinate $r$ at $(b,\ldots,a,\ldots,b)$ would then be positive, contradicting continuity.

Fix $\varepsilon>0$. For each coordinate $r$, choose a finite grid with endpoints $-\infty,+\infty$ such that every interval between consecutive grid points has marginal probability at most $\varepsilon/d$. Such a grid exists: first make the two tails small, then use uniform continuity of the marginal distribution function on the remaining compact interval. Let $G$ be the finite product grid. Lower orthants with corners in $G$ are continuity sets for the law of $X$, since their boundaries lie in finitely many coordinate hyperplanes of probability zero. Infinite coordinates simply remove a constraint, or give the empty set. Portmanteau therefore gives

\[\Delta_n\triangleq\max_{z\in G}|F_n(z)-F(z)|\longrightarrow0.\]

For any $x\in\mathbb R^d$, choose adjacent grid corners $a\leq x\leq b$. The difference between the lower orthants with corners $a,b$ is contained in the union of the coordinate strips $\{a_r<X_r\leq b_r\}$. Hence

\[0\leq F(b)-F(a)\leq\sum_{r=1}^d\mathbb P(a_r<X_r\leq b_r)\leq\varepsilon.\]

Monotonicity yields $F_n(a)\leq F_n(x)\leq F_n(b)$ and the corresponding inequalities for $F$, so

\[F_n(x)-F(x)\leq\Delta_n+\varepsilon, \qquad F(x)-F_n(x)\leq\Delta_n+\varepsilon.\]

This bound holds uniformly in $x$. Taking the supremum, then the limit superior in $n$, and finally letting $\varepsilon\downarrow0$ proves the result. ◻

Stochastic order notation records the size of a remainder relative to a chosen scale. An additive remainder that vanishes in probability does not change a weak limit; a term bounded in probability can still contribute to the limit.

Definition 9. A real sequence $Y_n$ is $O_{\mathbb P}(1)$, or bounded in probability, when for every $\varepsilon>0$ some $M<\infty$ satisfies $\limsup_n\mathbb P(|Y_n|>M)<\varepsilon$. For finite-valued variables this is equivalent to tightness of all their laws, because the finitely many initial indices can be handled by increasing $M$. The following definitions use real-valued random scales $R_n$, which may vanish; the factorization must hold also on $\{R_n=0\}$.

$X_n = o_{\mathop{\mathrm{\mathds{P}}}} (R_n)$ means there exists ${Y_n}\xrightarrow{\mathds{P}}0$ such that $X_n = Y_n R_n$.
$X_n = O_{\mathop{\mathrm{\mathds{P}}}} (R_n)$ means there exists $Y_n = O_{\mathop{\mathrm{\mathds{P}}}} (1)$ such that $X_n = Y_n R_n$.

Fact 10 ($o$ and $O$ rules).

$o_{\mathop{\mathrm{\mathds{P}}}} (1) + o_{\mathop{\mathrm{\mathds{P}}}} (1) = o_{\mathop{\mathrm{\mathds{P}}}} (1)$
$o_{\mathop{\mathrm{\mathds{P}}}} (1) + O_{\mathop{\mathrm{\mathds{P}}}} (1) = O_{\mathop{\mathrm{\mathds{P}}}} (1)$
$o_{\mathop{\mathrm{\mathds{P}}}} (1) O_{\mathop{\mathrm{\mathds{P}}}} (1) = o_{\mathop{\mathrm{\mathds{P}}}} (1)$
$[1 + o_{\mathop{\mathrm{\mathds{P}}}} (1)]^{-1} = O_{\mathop{\mathrm{\mathds{P}}}} (1)$
$o_{\mathop{\mathrm{\mathds{P}}}} (R_n) = R_n o_{\mathop{\mathrm{\mathds{P}}}} (1)$
$O_{\mathop{\mathrm{\mathds{P}}}} (R_n) + o_{\mathop{\mathrm{\mathds{P}}}} (R_n) = R_n O_{\mathop{\mathrm{\mathds{P}}}} (1)$
$o_{\mathop{\mathrm{\mathds{P}}}} (O_{\mathop{\mathrm{\mathds{P}}}} (1))= o_{\mathop{\mathrm{\mathds{P}}}} (1)$

Proof of the order rules. Write $A_n\to0$ in probability and $B_n=O_{\mathbb P}(1)$. The union bounds

\[\mathbb P(|A_n+B_n|>M+1) \leq\mathbb P(|A_n|>1)+\mathbb P(|B_n|>M)\]

prove (i) and (ii). For the product, fix $\varepsilon>0$ and use

First choose $M$ so that the limit superior of the first term is arbitrarily small; for this fixed $M$, the second term tends to zero. This proves (iii), without any independence assumption.

For (iv), define the reciprocal arbitrarily as a finite value when $1+A_n=0$; that event has probability tending to zero. On $\{|A_n|<1/2\}$,

\[\left|\frac1{1+A_n}-1\right|\leq2|A_n|.\]

Thus the reciprocal in fact equals $1+o_{\mathbb P}(1)$, which implies boundedness in probability. Assertion (v) is the defining factorization. For (vi), write the two summands as $R_nB_n$ and $R_nA_n$ and use (ii). Finally, (vii) is (iii) applied to the factorization defining its left side. All of these bounds remain valid when the factors are dependent. ◻

The next lemma transfers an ordinary small-argument estimate to a random argument converging to zero. This is the step used when applying Taylor expansions to estimators.

Lemma 11. $R:\mathop{\mathrm{\mathbb{R}}}^k \to \mathop{\mathrm{\mathbb{R}}}$ is Borel measurable and satisfies $R(0) = 0$. $X_n$ are random vectors on domain of $R$, and ${X_n}\xrightarrow{\mathds{P}}0$. Then, for all $p > 0$,

$R(h) = o(\norm{h}^p)$ as $h \to 0$ implies $R(X_n) = o_{\mathop{\mathrm{\mathds{P}}}} (\norm{X_n}^p)$.
$R(h) = O(\norm{h}^p)$ as $h \to 0$ implies $R(X_n) = O_{\mathop{\mathrm{\mathds{P}}}} (\norm{X_n}^p)$.

Proof. Set

\[g(h) = \begin{cases*} \frac{R(h)}{\norm{h}^p} & if $h \neq 0$ \\ 0 & if $h = 0$ \end{cases*}\]

it follows $R(X_n) = g(X_n) \norm{X_n}^p$.

The first assertion follows from the continuous mapping theorem, since $g$ is continuous at $h=0$.

For the second assertion, choose $M,\delta>0$ with $|g(h)|\leq M$ whenever $\|h\|<\delta$. Then $P(|g(X_n)|>M)\leq P(\|X_n\|\geq\delta)\to0$, which proves boundedness in probability. ◻

Weak convergence controls expectations of bounded continuous functions, as stated in Theorem 4, assertion (ii). Moment functions are unbounded, so rare large values can prevent their expectations from converging. Uniform integrability supplies the required control of the tails.

Definition 12. A sequence $(X_n)$ is asymptotically uniformly integrable if

\[\lim\limits_{M \to \infty} \limsup_{n \to \infty} \mathbb{E}\abs{X_n} \mathop{\mathrm{\mathbf{1}}}\set{\abs{X_n} \geqslant M} = 0\]

Theorem 13. Let $X_n\Rightarrow X$ and let $f$ be Borel measurable and continuous at $X$ almost surely. Put $Y_n=f(X_n),\ Y=f(X)$, and assume each $Y_n$ is integrable. If $(Y_n)$ is asymptotically uniformly integrable, then $Y$ is integrable and $\mathbb{E}Y_n\to \mathbb{E}Y$. The converse holds when $Y_n\geq0$ and $\mathbb{E}Y<\infty$. For signed variables, convergence of means alone is insufficient; convergence $\mathbb{E}|Y_n|\to \mathbb{E}|Y|<\infty$ is equivalent to asymptotic uniform integrability.

Proof.

Continuous mapping gives $Y_n\Rightarrow Y$. For $T_M(y)=(-M)\vee(y\wedge M)$, weak convergence controls $\mathbb{E}T_M(Y_n)$. The identity $|y-T_M(y)|=(|y|-M)_+$ and Portmanteau applied to the nonnegative continuous tail function give

\[\mathbb{E}(|Y|-M)_+\leq\liminf_n\mathbb{E}(|Y_n|-M)_+.\]

Asymptotic uniform integrability makes the right side tend to zero. It also bounds $\limsup_n\mathbb{E}|Y_n|$, so Portmanteau gives $\mathbb{E}|Y|<\infty$. Splitting the means at $T_M$ now gives

\[|\mathbb{E}Y_n-\mathbb{E}Y|\leq|\mathbb{E}T_M(Y_n)-\mathbb{E}T_M(Y)|+\mathbb{E}(|Y_n|-M)_++\mathbb{E}(|Y|-M)_+.\]

Take $n\to\infty$ and then $M\to\infty$. For the converse when $Y_n\geq0$ and the finite means converge,

\[\lim_n\mathbb{E}(Y_n-M)_+=\mathbb{E}Y-\mathbb{E}(Y\wedge M)=\mathbb{E}(Y-M)_+.\]

Since $y1_{\{y>2M\}}\leq2(y-M)_+$ for $y\geq0$, these excess-tail bounds imply asymptotic uniform integrability. Apply this nonnegative argument to $|Y_n|\Rightarrow|Y|$ to obtain the absolute-moment equivalence. Finally, the variables taking values $n,-n,0$ with probabilities $1/(2n),1/(2n),1-1/n$ converge in probability to zero and have mean zero, but their absolute-tail expectations remain one. This disproves the signed converse.

◻