Uniform Laws of Large Numbers and Stochastic Equicontinuity

An estimator is evaluated at a parameter chosen from the same data that define the empirical criterion. A law of large numbers at each fixed parameter does not by itself justify that substitution. Uniform laws control the error over a parameter set, while stochastic equicontinuity controls it locally. The binary-choice example below shows where these requirements enter an estimator’s expansion.

Begin with a binary-choice model and a likelihood estimator. The score expansion will expose an empirical average evaluated at the estimated parameter.

Example 1 (Binary choice model). Let \(X_i = (Y_i, Z_i)\), where

\[Y_i = \begin{cases*} 1 & with a job \\ 0 & otherwise \end{cases*}\]

and \(Z_i\) be the level of education (measured in years).

A logistic or probit model specifies a relationship between \(Y_i\) and \(Z_i\) through a chosen link function. For the logistic model, consider

\[\mathop{\mathrm{\mathds{P}}}(Y=1|Z=z) = F_0 (\alpha_0 + \theta_0 z)\]

where \(F_0(\xi) = \frac{\exp \xi}{ 1 + \exp \xi }\) is the logistic function. For simplicity, set \(\alpha_0 = 0\), so we focus on the relationship captured by \(\theta_0\).

By standard M-estimation, we have

\[\widehat{\theta}_n \in \arg\max_{\theta \in \Theta} \sum_{i=1}^{n} \log p_{\theta} (Y_i | Z_i)\]

where \(p_{\theta} (\cdot)\) is the conditional density function

\[p_{\theta} (y | z) = F_0 (\theta z)^y \left[ 1 - F_0 (\theta z) \right]^{1-y},\qquad y\in\{0,1\}\]

One can then introduce the score function

\[\begin{aligned} \ell_{\theta} (y,z) &= \dod{}{\theta} \log p_{\theta} (y | z) \\ &= z (y-F_0 (\theta z)) \end{aligned}\]

Assume the observations are i.i.d., the logistic model is correctly specified, and \(\mathbb{E} Z^2<\infty\). Define the secant slope of the score by

\[g_\theta(z)=z^2\int_0^1 F_0'\big((\theta_0+t(\theta-\theta_0))z\big)\,dt.\]

Then \(0\le g_\theta(z)\le z^2/4\), \(g_\theta(z)\to g_{\theta_0}(z)\), and the exact identity

\[\ell_\theta(y,z)=\ell_{\theta_0}(y,z)-(\theta-\theta_0)g_\theta(z)\]

holds also at \(\theta=\theta_0\). Write \(Ph=\mathbb{E} h(X)\), \(P_nh=n^{-1}\sum_i h(X_i)\), and \(\mathbb G_nh=\sqrt n(P_n-P)h\). The scalar information is \(I_{\theta_0}=Pg_{\theta_0}=\mathbb{E}[Z^2F_0(\theta_0Z)(1-F_0(\theta_0Z))]\); assume it is positive. Conditional centering of the Bernoulli response gives \(P\ell_{\theta_0}=0\) and \(P\ell_{\theta_0}^2=I_{\theta_0}\).

Lemma 2. Suppose the estimator solves \(P_n\ell_{\widehat\theta_n}=0\) with probability tending to one, and

\[P_ng_{\widehat\theta_n}\xrightarrow{P}I_{\theta_0}.\tag{1}\]

Then \(\sqrt n(\widehat\theta_n-\theta_0)\Rightarrow\mathcal{N}(0,I_{\theta_0}^{-1})\).

Proof. An interior maximizer has zero score, rather than zero derivative of the score. Substituting the exact secant identity gives

\[0=P_n\ell_{\widehat\theta_n}=P_n\ell_{\theta_0}-(\widehat\theta_n-\theta_0)P_ng_{\widehat\theta_n},\qquad \sqrt n(\widehat\theta_n-\theta_0)=\frac{\mathbb G_n\ell_{\theta_0}}{P_ng_{\widehat\theta_n}}.\]

The iid central limit theorem applies to the numerator, whose variance is \(I_{\theta_0}\); Slutsky's theorem proves the assertion. The denominator assumption also follows from consistency here: for fixed \(\delta>0\), the local difference is bounded by \(P_n\sup_{|\theta-\theta_0|\le\delta}|g_\theta-g_{\theta_0}|\). The fixed-function law of large numbers and dominated convergence, using the envelope \(Z^2/2\), make this bound tend to zero as first \(n\to\infty\) and then \(\delta\downarrow0\). □

The local law used in Lemma 2 can be written as a three-event estimate. Define \(H_\delta(z)=\sup_{|\theta-\theta_0|\le\delta}|g_\theta(z)-g_{\theta_0}(z)|\). Continuity in \(\theta\) means the supremum can be taken over a countable dense subset, so it is measurable. Moreover \(H_\delta\to0\) pointwise as \(\delta\downarrow0\) and \(H_\delta\le Z^2/2\). On \(\{|\widehat\theta_n-\theta_0|\le\delta\}\),

\[|P_ng_{\widehat\theta_n}-I_{\theta_0}|\le P_nH_\delta+|(P_n-P)g_{\theta_0}|.\]

For fixed \(\delta\), the two sample averages converge by the integrable LLN. Since \(PH_\delta\to0\) by dominated convergence, consistency of \(\widehat\theta_n\) finishes the argument. Thus evaluation at a data-dependent parameter is justified by a local supremum, not by a pointwise LLN applied after observing that parameter.

A score equation need not be solved exactly. Stochastic equicontinuity gives a version for approximate roots.

Lemma 3. Under the same model and moment assumptions, suppose \(\widehat\theta_n\to_P\theta_0\), \(P_n\ell_{\widehat\theta_n}=o_P(n^{-1/2})\), and

\[\mathbb G_n(\ell_{\widehat\theta_n}-\ell_{\theta_0})=o_P(1).\tag{2}\]

Then \(\sqrt n(\widehat\theta_n-\theta_0)\Rightarrow\mathcal{N}(0,I_{\theta_0}^{-1})\).

Proof. The approximate score equation and (2) imply \(-\sqrt nP\ell_{\widehat\theta_n}=\mathbb G_n\ell_{\theta_0}+o_P(1)\). Taking population expectations in the secant identity gives \(P\ell_\theta=-(\theta-\theta_0)Pg_\theta\). Dominated convergence and consistency give \(Pg_{\widehat\theta_n}\to_P I_{\theta_0}>0\). Therefore

\[\sqrt n(\widehat\theta_n-\theta_0)=\frac{\mathbb G_n\ell_{\theta_0}+o_P(1)}{Pg_{\widehat\theta_n}}.\]

The same central limit theorem applies. This argument establishes the rate as part of the conclusion; it does not assume it in order to control a Taylor remainder. □

Another approach is to consider the slightly more general case, where we just want to estimate a distribution function:

\[\mathop{\mathrm{\mathds{P}}}(Y=1|Z=z) = F_0(z)\]

for any \(F_0 \in \Lambda \triangleq \left\{ F:\mathop{\mathrm{\mathbb{R}}}\to [0,1] : \: F(x) \geqslant F(y) , \: \forall x \geqslant y \right\}\). As such, the maximum likelihood estimate satisfies

\[\widehat{F}_n \in \arg\max_{F \in \Lambda} \sum_{i=1}^{n} Y_i \log F(Z_i) + (1-Y_i) \log (1-F(Z_i))\]

The objective is smooth in the fitted probabilities when they lie strictly between zero and one, but the monotonicity constraint couples those probabilities and the parameter is a function. A fixed-dimensional unconstrained score equation does not describe this optimization problem.

Criterion comparisons provide another route when a finite-dimensional score expansion is unavailable. Their empirical error must again be controlled uniformly.

For a dominated model \(\{p_\theta:\theta\in\Theta\}\) with true density \(p_{\theta_0}\), let \(q_\theta=\log(p_{\theta_0}/p_\theta)\), assuming these contrasts are integrable. Their population means are the Kullback–Leibler divergences

\[Pq_\theta=\mathrm{KL}(p_{\theta_0}\Vert p_\theta)=\int\log\frac{p_{\theta_0}}{p_\theta}\,p_{\theta_0}\,d\mu\ge0.\]

An exact likelihood maximizer satisfies \(P_nq_{\widehat\theta_n}\le0\). Consequently,

\[0\le\mathrm{KL}(p_{\theta_0}\Vert p_{\widehat\theta_n}) =Pq_{\widehat\theta_n}\le |(P_n-P)q_{\widehat\theta_n}| \le\sup_{\theta\in\Theta}|(P_n-P)q_\theta|.\]

A uniform law for the contrast class therefore gives convergence of the fitted density in KL divergence. Parameter consistency additionally requires separation: outside each neighborhood of \(\theta_0\), the population divergence must have a positive infimum. This argument also applies to an approximate maximizer with an added \(o_P(1)\) optimization error. The ordinary strong law at each fixed contrast would not justify substitution of the fitted index.

The KL separation step is also an explicit probability bound. If \(c_\varepsilon=\inf_{d(\theta,\theta_0)\ge\varepsilon}Pq_\theta>0\) and the likelihood optimization error is bounded by a nonnegative \(r_n=o_P(1)\), then

\[P\{d(\widehat\theta_n,\theta_0)\ge\varepsilon\}\le P\left\{\sup_{\theta\in\Theta}|(P_n-P)q_\theta|+r_n\ge c_\varepsilon\right\}\longrightarrow0.\]

KL nonnegativity follows from \(-\log u\ge1-u\), applied to \(u=p_\theta/p_{\theta_0}\) on the support of the true density and then integrated. The likelihood comparison controls a divergence; the positive separation constant is what converts that control into convergence of the parameter.

1. Uniform Law of Large Numbers

For the ensuing discussions, fix an underlying probability space \((\mathscr{X},\mathop{\mathrm{\mathfrak{F}}},\mathop{\mathrm{\mathds{P}}})\). Consider the space of functions

\[\mathop{\mathrm{\mathcal{G}}}\triangleq \left\{ g_{\theta} : \: \theta \in \Theta \right\}\]

defined on \(\mathscr{X}\). Furthermore, let \(X_1,X_2,\dots\) denote independent copies of random variable \(X\) defined on \((\mathscr{X},\mathop{\mathrm{\mathfrak{F}}},\mathop{\mathrm{\mathds{P}}})\).

Definition 4. \(\mathop{\mathrm{\mathcal{G}}}\) satisfies the uniform law of large numbers (ULLN) if

\[\sup_{\theta \in \Theta} \abs{\frac{1}{n} \sum_{i=1}^{n} g_{\theta} (X_i) - \mathbb{E}g_{\theta} (X_i) } \xrightarrow{n \to \infty} 0, \quad \text{a.s.}\]

If the above holds, then clearly

\[\abs{\frac{1}{n} \sum_{i=1}^{n} g_{\widehat{\theta}_n} (X_i) - \mathbb{E}g_{\widehat{\theta}_n} (X_i) } \xrightarrow{n \to \infty} 0, \quad \text{a.s.}\]

the slight abuse of notation above is where if \(\widehat{\theta}_n\) is random, then \(\mathbb{E}g_{\widehat{\theta}_n} (X_i)\) is not an expectation! Notice that

\[\begin{aligned} \mathbb{E}g_{\theta} (X_i) &= \int g_{\theta} \dif \mathop{\mathrm{\mathds{P}}}\\ \mathbb{E}g_{\widehat{\theta}_n} (X_i) &= \int g_{\widehat{\theta}_n} \dif \mathop{\mathrm{\mathds{P}}} \end{aligned}\]

Also note that neither of the above represents the empirical mean; instead, that is defined as

\[\begin{aligned} \frac{1}{n} \sum_{i=1}^{n} g_{\theta} (X_i) = \int g_{\theta} \dif \mathbb{P}_n \end{aligned}\]

and the difference between

\[\frac{1}{n} \sum_{i=1}^{n} g_{\theta} (X_i) - \mathbb{E}g_{\theta} = \int g_{\theta} \dif (\mathbb{P}_n - \mathop{\mathrm{\mathds{P}}})\]

For a random index, the population integral means \(Pg_{\widehat\theta_n}(\omega)=\int g_{\widehat\theta_n(\omega)}(x)\,P(dx)\), with the fitted index held fixed while integrating over a fresh variable \(x\). It generally differs from the unconditional expectation of \(g_{\widehat\theta_n}(X_i)\), because \(X_i\) helped choose the index. The pathwise inequality

\[|(P_n-P)g_{\widehat\theta_n}|\le\sup_{\theta\in\Theta}|(P_n-P)g_\theta|\]

is the entire justification for the substitution. As usual, the supremum must be measurable (for example, for a pointwise measurable class); otherwise the statement is interpreted using outer probability and outer almost-sure convergence.