Stochastic Equicontinuity

Taylor expansion often reduces an estimator to a linear term and a remainder evaluated at a random parameter. Pointwise convergence of that remainder is insufficient: the estimated parameter depends on the same sample. Stochastic equicontinuity supplies control over a whole neighborhood, allowing the random parameter to be substituted into the expansion.

Let \(X_1,\ldots,X_n\) be i.i.d. with law \(P\). For a function \(f\), write \(Pf=\mathbb{E} f(X_1)\), \(P_nf=n^{-1}\sum_{i=1}^n f(X_i)\), and \(\mathbb G_nf=\sqrt n(P_n-P)f\). Rademacher variables are independent random signs, independent of the observations. See the empirical-process setup for measurability conventions.

Analyses of statistical procedures can rely heavily on Taylor’s theorem and regularity conditions. For example, maximum likelihood estimation relies on maximizing random criterion functions: if \(X_1, \dots, X_n\) are independent draws from a distribution \(\mathbf P\,\,\!\!\) (admitting a family of density functions \(\{p(\cdot,\theta)\}\)), then the true and unknown parameter \(\theta_0\) can be estimated by finding the argmax of \(G_n (\theta) = \frac1n \sum_{1 \leq i \leq n} \ln p(X_i,\theta)\).

To identify the remainder that needs uniform control, write \(g_0 (\cdot,\theta) = \ln p(\cdot, \theta)\), and let \(g_i (\cdot,\theta)\) be the \(i\)th derivative of \(g_0\) for \(i = 1,2,3\). Taylor’s theorem implies

\[\begin{aligned} g_0(\cdot, \theta) = g_0(\cdot, \theta_0) + g_1(\cdot, \theta_0) (\theta-\theta_0) + \frac{g_2(\cdot, \theta_0)}{2} (\theta-\theta_0)^2 + \frac{g_3(\cdot, \theta^*)}{6} (\theta-\theta_0)^3 ~. \end{aligned}\]

Here, \(\theta^*\) lies between \(\theta\) and \(\theta_0\). Integrating with respect to the empirical measure over \(X_1,\dots,X_n\), denoted \(\widehat P_n\), implies

\[\begin{aligned} G_n(\theta) = G_n(\theta_0) + (\theta-\theta_0) \widehat P_n g_1 + \frac{(\theta-\theta_0)^2}{2} \widehat P_n g_2 + R_n (\theta) ~, \end{aligned}\]

The intermediate point in the pointwise Taylor formula may depend on the observation as well as on \(\theta\). A concrete sufficient condition for controlling the remainder is an integrable envelope \(H\) with \(\sup_{|\theta-\theta_0|\leq\delta}|g_3(x,\theta)|\leq H(x)\). Then

\[|R_n(\theta)|\leq\frac{|\theta-\theta_0|^3}{6}P_nH,\qquad\sup_{0<|\theta-\theta_0|\leq\delta_n}\frac{|R_n(\theta)|}{|\theta-\theta_0|^2}\leq\frac{\delta_n}{6}P_nH=o_P(1)\]

for every deterministic \(\delta_n\downarrow0\), by the law of large numbers. If \(P|g_2(\cdot,\theta_0)|<\infty\), its empirical average converges to the population curvature. A score central limit theorem also requires finite score variance; consistency, an interior optimum, nonzero curvature, and sufficiently accurate optimization are separate requirements. Integrability of the second derivative alone does not imply an estimator CLT.

Definition 1. Let \((T,d)\) be a pseudometric space where \(d\) is a semimetric. Let \(\{Z_n(t): t \in T\}\) be a stochastic process. We say \(\{Z_n\}\) is stochastically equicontinuous at \(t_0\) if, for each \(\eta > 0\) and \(\varepsilon > 0\), there exists a neighbourhood \(U\) of \(t_0\) where

\[\begin{aligned} \limsup_{n \to \infty} \mathbf P\,\,\!\!\left( \sup_U |Z_n(t) - Z_n(t_0)| > \eta \right) < \varepsilon ~. \end{aligned}\]

Probabilities of potentially nonmeasurable suprema are understood as outer probabilities; under a countable separability condition they are ordinary probabilities. If \(t_n\to_Pt_0\), stochastic equicontinuity immediately implies \(Z_n(t_n)-Z_n(t_0)=o_P(1)\): for a fixed neighborhood \(U\), the probability of a deviation exceeds at most \(P(t_n\notin U)+P^*(\sup_U|Z_n(t)-Z_n(t_0)|>\eta)\). First take the limsup in \(n\), then shrink \(U\).

The same issue appears for general minimum-criterion estimators, even when the sample criterion is not classically differentiable. Suppose we study \(\mathcal F = \{f(\cdot,t): t \in T\}\), \(T \subset \mathbb R^k\), where \(f\) is \(\mathbf P\,\,\!\!\)-integrable functions on the set \(S\) where \(\mathbf P\,\,\!\!\) lives. Let \(P_n\) be the empirical measure formed from \(n\) independent observations drawn from \(\mathbf P\,\,\!\!\). Let the empirical process be written as \(\mathbb G_n (f) = \sqrt n(\widehat P_n - \mathbf P\,\,\!\!) (f)\) as applied to \(f \in \mathcal F\). Write \(F(t) = \mathbf P\,\,\!\!f(\cdot, t)\) and \(F_n(t) = \widehat P_n f(\cdot,t)\). It will be shown that if we want to minimize \(\{F_n(\cdot)\}\), then we can study a suitable sequence \(\{\tau_n\} \subset T\) that actually comes close to minimizing the family of functions. To proceed with showing this, we can write

\[\begin{aligned} f(\cdot, t) = f(\cdot, t_0) + (t-t_0)' \Delta(\cdot) + |t-t_0| r(\cdot,t) ~, \end{aligned}\tag{1}\]

where \(\Delta(\cdot)\) is a vector of \(k\) real functions defined on \(S\) and \(r(\cdot,t_0) = 0\). The expansion is useful when the normalized remainder becomes small in mean square. We therefore impose size constraints on \(r(\cdot,t)\) and therefore \(f(\cdot,t)\). In particular, a sensible definition is \(\mathbf P\,\,\!\!|r(\cdot,t)|^2 \xrightarrow{t \to t_0} 0\), so that the criterion is differentiable as a map into \(L^2(P)\). This is a property of the criterion functions; it is distinct from differentiability in quadratic mean of square-root densities in a statistical model. We can work with \(L^2(\mathbf P\,\,\!\!)\) directly, consider a seminorm \(\rho_P\) on it, and focus our attention on the set \(\mathcal R = \{r(\cdot,t)\}\). A sufficient route to stochastic equicontinuity of \(\{\mathbb G_n r(\cdot,t)\}\) is the following condition, provided \(\rho_P(r_t)\to0\) as \(t\to t_0\): for \(\varepsilon > 0\), \(\eta > 0\), there exists a neighbourhood \(V \subset \mathcal R\) such that \(\limsup_{n \to \infty} \mathbf P\,\,\!\!(\sup_V |\mathbb G_n r| > \eta) < \varepsilon\), i.e. \(V = \{r \in \mathcal R: \rho_P(r) \leq \delta\}\) for some \(\delta > 0\).

Remark 2. If \(t_0\) is an interior local minimum of \(F\) and \(Pr_t^2\to0\), integration of the linear approximation gives \(F(t)-F(t_0)=(t-t_0)^\top P\Delta+o(|t-t_0|)\). Hence \(P\Delta=0\), by taking directional derivatives in both directions. Mean-square differentiability by itself, without the interior-minimum hypothesis, would not give this conclusion.

Theorem 3. Let \(\{\tau_n\}\) be a sequence of random vectors converging in probability to \(t_0\) where \(F(t_0)\) is the minimum in the function class considered. Let \(r(\cdot,t)\) and \(\Delta(\cdot)\) be the functions defined in the linear approximation (1). Assume the mean-square remainder condition \(P r_t^2\to0\) as \(t\to t_0\), with \(r_{t_0}=0\). Suppose the following conditions hold.

\(t_0 \in T^\circ\).
\(F(\cdot)\) is twice differentiable at the interior minimizer and has an invertible Hessian matrix \(V\) at \(t_0\).
\(F_n(\tau_n) \le \inf_{t\in T}F_n(t)+\varepsilon_n,\quad \varepsilon_n\ge0,\quad n\varepsilon_n\xrightarrow{P}0\).
The components of \(\Delta(\cdot)\) are all elements of \(L^2(\mathbf P\,\,\!\!)\).
The sequence \(\{\mathbb G_n r(\cdot,t)\}\) is stochastic equicontinuous at \(t_0\).

Then, \(\sqrt n (\tau_n - t_0) \xrightarrow{d} \mathcal{N}(0,V^{-1} (\mathbf P\,\,\!\!(\Delta \Delta') - (\mathbf P\,\,\!\!\Delta)(\mathbf P\,\,\!\!\Delta)')V^{-1})\).

For background on this approach see Pollard (1984). The argument below proves the stated version directly: it first establishes the rate and then substitutes the random estimator into a compact uniform expansion.

Proof of Theorem 3. Write \(P g = \int g \, dP\), \(P_n g = \frac1n\sum_{i=1}^n g(X_i)\), and \(\mathbb G_n g = \sqrt n (P_n-P)g\). For \(t\in T\) write \(f_t(\cdot)=f(\cdot,t)\) and \(r_t(\cdot)=r(\cdot,t)\).

Step 1 (first-order condition and positivity). Because \(t_0\in T^\circ\) and \(t_0\) is a (local) minimizer of the differentiable map \(t\mapsto F(t)=P f_t\), we have \(\nabla F(t_0)=0\). Integrating the linear approximation (1) gives, for \(t\) near \(t_0\),

\[\begin{aligned} F(t)-F(t_0) &= P(f_t-f_{t_0}) = (t-t_0)' P\Delta + |t-t_0| \, P r_t . \end{aligned}\tag{2}\]

For any fixed unit vector \(v\), put \(t=t_0+av\). Cauchy–Schwarz gives \(Pr_t\to0\), so the directional derivative of \(F\) is \(v'P\Delta\). At an interior differentiable minimum this derivative is zero for every \(v\), hence

\[\begin{aligned} P\Delta = 0. \end{aligned}\tag{3}\]

Assumption (2) gives that the Hessian matrix \(V=\nabla^2 F(t_0)\) exists and is invertible. Since \(t_0\) is a local minimizer, \(V\) must be symmetric positive definite; denote

\[\lambda_{\min} \equiv \lambda_{\min}(V)>0.\]

Step 2 (second-order expansion of \(F\) and a quadratic lower bound). By twice differentiability of \(F\) at \(t_0\),

\[\begin{aligned} F(t_0+u) = F(t_0) + \nabla F(t_0)'u + \frac12 u'Vu + o(\|u\|^2) = F(t_0) + \frac12 u'Vu + o(\|u\|^2). \end{aligned}\tag{4}\]

Therefore there exists \(\delta_0>0\) such that whenever \(\|u\|\le \delta_0\),

\[\begin{aligned} \left|o(\|u\|^2)\right| \le \frac{\lambda_{\min}}{4}\|u\|^2, \qquad\text{hence}\qquad F(t_0+u)-F(t_0) \ge \frac{\lambda_{\min}}{4}\|u\|^2. \end{aligned}\tag{5}\]

Step 3 (local expansion of \(F_n\) on \(n^{-1/2}\)-scales). For \(h\in\mathbb R^k\) define \(t_{n,h}\equiv t_0+h/\sqrt n\) (which lies in \(T\) for all large \(n\) because \(t_0\in T^\circ\)). Consider the local criterion

\[Q_n(h)\ \equiv\ n\left(F_n(t_{n,h})-F_n(t_0)\right) \quad\text{where}\quad F_n(t)=P_n f_t.\]

Decompose

\[\begin{aligned} Q_n(h) &= n\left(F(t_{n,h})-F(t_0)\right) + n\left[(F_n-F)(t_{n,h})-(F_n-F)(t_0)\right]. \end{aligned}\tag{6}\]

For the deterministic part, set \(u=h/\sqrt n\) in (4) to get

\[\begin{aligned} n\left(F(t_{n,h})-F(t_0)\right) = \frac12 h'Vh + nR(h/\sqrt n) = \frac12 h'Vh + o(1), \end{aligned}\tag{7}\]

uniformly over \(\|h\|\leq M\) for each fixed \(M\). Here \(R(u)=F(t_0+u)-F(t_0)-u'Vu/2=o(\|u\|^2)\), and the precise uniform bound is

\[\sup_{\|h\|\leq M}n|R(h/\sqrt n)|\leq M^2\sup_{0<\|u\|\leq M/\sqrt n}\frac{|R(u)|}{\|u\|^2}\longrightarrow0.\]

For the stochastic part, use (1):

\[f_{t_{n,h}}-f_{t_0} = (t_{n,h}-t_0)'\Delta + |t_{n,h}-t_0|\, r_{t_{n,h}} = \frac1{\sqrt n} h'\Delta + \frac{\|h\|}{\sqrt n}\, r_{t_{n,h}}.\]

Hence

\[\begin{aligned} n\left[(F_n-F)(t_{n,h})-(F_n-F)(t_0)\right] &= n(P_n-P)\left(f_{t_{n,h}}-f_{t_0}\right) \notag\\ &= \sqrt n\, h'(P_n-P)\Delta + \|h\|\,\sqrt n\,(P_n-P) r_{t_{n,h}} \notag\\ &= h'\,\mathbb G_n\Delta + \|h\|\, \mathbb G_n r_{t_{n,h}}. \end{aligned}\tag{8}\]

Step 4 (use stochastic equicontinuity to control the remainder term). By assumption (5), \(\{\mathbb G_n r_t: t\in T\}\) is stochastically equicontinuous at \(t_0\). Because \(r_{t_0}\equiv 0\), we have \(\mathbb G_n r_{t_0}=0\) for every \(n\). Therefore, for any fixed \(\eta,\varepsilon>0\), there exists a neighborhood \(U\) of \(t_0\) such that

\[\limsup_{n\to\infty}\mathbf P\,\,\!\!\Big(\sup_{t\in U}|\mathbb G_n r_t|>\eta\Big)<\varepsilon.\]

Fix \(M<\infty\) and choose \(U\) so small that \(t_{n,h}\in U\) for all \(\|h\|\le M\) and all large \(n\). Then

\[\begin{aligned} \sup_{\|h\|\le M} |\mathbb G_n r_{t_{n,h}}| \ \xrightarrow{p}\ 0, \end{aligned}\tag{9}\]

and consequently,

\[\begin{aligned} \sup_{\|h\|\le M} \|h\|\,|\mathbb G_n r_{t_{n,h}}| \ \xrightarrow{p}\ 0. \end{aligned}\tag{10}\]

Combining (6)–(8) with (7) and (10), we have: for every fixed \(M<\infty\),

\[\begin{aligned} \sup_{\|h\|\le M}\Big|Q_n(h) - \Big(\frac12 h'Vh + h'\mathbb G_n\Delta\Big)\Big| \ \xrightarrow{p}\ 0. \end{aligned}\tag{11}\]

Step 5 (prove the rate before using a compact local expansion). Let \(u_n=\tau_n-t_0\), \(h_n=\sqrt n\,u_n\). The empirical near-minimization assumption is

\[F_n(\tau_n)\le\inf_{t\in T}F_n(t)+\varepsilon_n,\qquad n\varepsilon_n=o_P(1).\tag{12}\]

Consistency and stochastic equicontinuity give \(\mathbb G_nr_{\tau_n}=o_P(1)\): outside a small consistency event the index lies in a fixed neighborhood on which the oscillation probability is arbitrarily small. Also \(\mathbb G_n\Delta=O_P(1)\) by the multivariate CLT. On the event \(\|u_n\|\le\delta_0\), whose probability tends to one, use the quadratic lower bound (5) and the exact empirical decomposition from (1):

\[\frac{\lambda_{\min}}4\|h_n\|^2\le\|h_n\|\bigl(\|\mathbb G_n\Delta\|+|\mathbb G_nr_{\tau_n}|\bigr)+n\varepsilon_n.\]

The right side is \(\|h_n\|O_P(1)+o_P(1)\). The positive root of this quadratic inequality is bounded in probability, so \(h_n=O_P(1)\). This establishes the rate without presuming that the random index already lies in a fixed compact set.

Set \(b_n=-V^{-1}\mathbb G_n\Delta=O_P(1)\). Since \(t_0\) is interior, \(t_0+b_n/\sqrt n\in T\) with probability tending to one; otherwise set the comparison point to \(t_0\). Completing the square gives

\[\frac12h'Vh+h'\mathbb G_n\Delta=\frac12(h-b_n)'V(h-b_n)-\frac12b_n'Vb_n.\tag{13}\]

Now the compact uniform approximation (11) applies at both tight indices: restrict them to a ball of radius \(M\), use that approximation, and then let \(M\to\infty\). Near-minimization yields

\[\frac12(h_n-b_n)'V(h_n-b_n)=Q_n(h_n)-Q_n(b_n)+o_P(1)\le n\varepsilon_n+o_P(1)=o_P(1).\tag{14}\]

The left side is nonnegative and at least \(\lambda_{\min}\|h_n-b_n\|^2/2\). Therefore

\[\sqrt n(\tau_n-t_0)=-V^{-1}\mathbb G_n\Delta+o_P(1).\tag{15}\]

Step 6 (CLT and conclusion). By assumption (4), \(\Delta\in L^2(P)^k\), hence

\[\mathbb G_n\Delta =\frac1{\sqrt n}\sum_{i=1}^n\left(\Delta(X_i)-P\Delta\right) \xrightarrow{d} \mathcal{N} \left(0,\ \Sigma\right), \qquad \Sigma \equiv P(\Delta\Delta')-(P\Delta)(P\Delta)'.\]

Combine this with (15) and Slutsky’s theorem to obtain

\[\sqrt n(\tau_n-t_0) \xrightarrow{d} \mathcal{N} \left(0,\ V^{-1}\Sigma V^{-1}\right),\]

which is exactly the claim. ◻

The argument separates the curvature of the population criterion from the random linear term. Step 5 supplies the rate bound that permits the random estimator to enter the uniform local expansion.

Pollard, David. 1984. Convergence of Stochastic Processes. Springer Series in Statistics. New York, NY: Springer New York. https://doi.org/10.1007/978-1-4612-5254-2.