Bootstrapping the Mean and Coverage Accuracy
Let \(X_1,\ldots,X_n\) be i.i.d. from \(F\), with mean \(\mu=\theta(F)\) and variance \(0<\sigma^2(F)<\infty\). Write \(\hat F_n\) for the empirical distribution and \(P^*\) for probability conditional on the observations. The setup and conditional convergence conventions distinguish this conditional law from repeated sampling under \(F\).
We first approximate the distribution of the unstudentized error. Its limit retains the unknown population scale:
Proposition 1. Let \(\mathbf S_F\) consist of deterministic sequences \(F_n\rightsquigarrow F\) for which \(\theta(F_n)\to\mu\) and \(\sigma^2(F_n)\to\sigma^2(F)\).
(a) For every such sequence, \(J_n(F_n)\rightsquigarrow \mathcal{N}(0,\sigma^2(F))\).
(b) For i.i.d. observations from \(F\), the conditional bootstrap law \(J_n(\hat F_n)\) is strongly consistent. In particular, the cdf, quantile and coverage conclusions (2), (3) and (4) hold for each fixed probability level in \((0,1)\).
Proof. In row \(n\), let \(X_{n,1},\ldots,X_{n,n}\) be independent with law \(F_n\), and put \(Y_{n,i}=X_{n,i}-\theta(F_n)\). Then \(Y_{n,1}\rightsquigarrow X_1-\mu\) and its second moment converges to \(\sigma^2(F)\). These two facts imply uniform integrability of \(Y_{n,1}^2\), as justified in Lemma 4 below. For every \(\varepsilon>0\),
This is the Lindeberg condition at the correct \(\sqrt n\) scale. The triangular-array CLT proves (a). For (b), the empirical cdf converges weakly almost surely, and the strong law applied to \(X_i\) and \(X_i^2\) gives convergence of its mean and variance. Thus \(\{\hat F_n\}\in\mathbf S_F\) almost surely. Apply Proposition 3 of the preceding post. ◻
To remove the unknown scale from the limiting law, estimate it in the same sample as the numerator. Throughout this post use the empirical variance with divisor \(n\):
Define the ratio arbitrarily, say as zero, when the denominator is zero; the probability of this event tends to zero under the assumptions here. The bootstrap recomputes both the mean and the standard deviation in each resample. To justify this step along changing populations, we need a weak law for triangular arrays.
Lemma 2. Suppose each row \(Y_{n,1},\ldots,Y_{n,n}\) is i.i.d., \(Y_{n,1}\rightsquigarrow Y\), and \(\mathbb{E}|Y_{n,1}|\to \mathbb{E}|Y|<\infty\). Then \(n^{-1}\sum_iY_{n,i}\to \mathbb{E}Y\) in probability.
The full Lindeberg normalization in Proposition 1 is \(s_n^2=n\sigma_n^2\), where \(\sigma_n^2=\operatorname{Var}_{F_n}(X)\to\sigma^2>0\). Thus
The variance stays bounded away from zero, and the threshold tends to infinity, so uniform integrability is exactly the required tail control. The CLT gives \(\sqrt n(\bar X_n-\theta(F_n))/\sigma_n\Rightarrow \mathcal{N}(0,1)\); multiplying by \(\sigma_n\to\sigma\) gives the unstudentized conclusion.
The proof separates two tasks: control the tails uniformly, then apply a weak law to bounded truncations.
Lemma 3. Suppose each row is i.i.d. with finite mean \(\mu_n\), and
Then \(\bar Y_n-\mu_n\to0\) in probability.
Proof. Center the variables and write \(Z_{n,i}=Y_{n,i}-\mu_n\), \(Z_{n,i}^{A}=Z_{n,i}\mathbf1\{|Z_{n,i}|\le A\}\). For fixed \(A\), independence and Chebyshev give
The expected absolute difference between the centered truncated average and the original average is at most \(2\mathbb{E}[|Z_{n,1}|\mathbf1\{|Z_{n,1}|>A\}]\). Markov's inequality, followed first by \(n\to\infty\) and then by \(A\to\infty\), completes the proof. ◻
Writing \(T_{n,i}^A=Z_{n,i}\mathbf1\{|Z_{n,i}|>A\}\), the difference of averages in Lemma 3 is \(n^{-1}\sum_i(T_{n,i}^A-ET_{n,i}^A)\). Its absolute expectation is at most \(2E|T_{n,1}^A|\). Consequently the complete probability bound is
The order of limits matters: first let \(n\to\infty\) at fixed \(A\), and only then let \(A\to\infty\). No second-moment assumption on the untruncated variables is used.
Lemma 4. Suppose \(X_n\rightsquigarrow X\).
(a) If \(|X_n|\le M\) almost surely for one finite \(M\), then \(\mathbb{E}X_n\to \mathbb{E}X\).
(b) At every \(A>0\) with \(P(|X|=A)=0\), the expectations of \(|X_n|\mathbf1\{|X_n|\le A\}\) converge. If each variable is integrable and also \(\mathbb{E}|X_n|\to \mathbb{E}|X|<\infty\), then \(\{|X_n|\}\) is uniformly integrable and \(\mathbb{E}X_n\to \mathbb{E}X\).
Proof. Bounded continuous test functions prove (a); for (b), the bounded truncation has discontinuities only at \(\pm A\), which have zero limiting probability. Subtracting its expectation from \(\mathbb{E}|X_n|\) gives convergence of the tail expectations at such \(A\). Let these continuity points increase to infinity. The finitely many early indices have integrable tails too, so the resulting bound is uniform in all \(n\). Truncating the signed variables then proves convergence of their means. ◻
Proof of Lemma 2. Lemma 4 gives \(\mathbb{E}Y_{n,1}\to \mathbb{E}Y\) and uniform integrability. Since the means are bounded, centering preserves the tail condition (3). Lemma 3 now proves the conclusion. ◻
Proposition 5. Under the assumptions of Proposition 1:
(a) \(K_n(F_n)\rightsquigarrow \mathcal{N}(0,1)\) for every \(\{F_n\}\in\mathbf S_F\).
(b) The conditional studentized bootstrap law \(K_n(\hat F_n)\) is strongly consistent, with the same cdf, quantile and coverage conclusions as in Proposition 1.
Proof. The assumptions on \(F_n\) give convergence of the first two moments. Lemma 4 makes \(X_{n,1}^2\) uniformly integrable, and hence makes \(|X_{n,1}|\) uniformly integrable as well. Apply Lemma 2 to \(X_{n,i}\) and to \(X_{n,i}^2\):
Combine this with Proposition 1 and Slutsky's theorem. The almost-sure empirical-distribution argument used there also proves (b). ◻
1. Bootstrap Coverage Errors
First-order consistency gives limiting coverage, but does not distinguish the error rates of confidence intervals. Edgeworth expansions describe the first departure from normality. Put \(\gamma=\mathbb{E}[(X-\mu)^3]/\sigma^3\), and let \(\Phi,\phi\) be the standard normal cdf and density.
Theorem 6. If \(\mathbb{E}|X|^4<\infty\) and \(\limsup_{|t|\to\infty}|Ee^{itX}|<1\), then, uniformly in \(x\),
Theorem 7. If \(\mathbb{E}|X|^8<\infty\) and the vector \((X,X^2)\) satisfies Cramér's condition, then, uniformly in \(x\),
These are applications of the smooth-function Edgeworth theorem in Hall, The Bootstrap and Edgeworth Expansion, Theorem 2.2 and the mean example in Section 2.4. For Theorem 7 take the sample-mean vector of \((X,X^2)\) and the smooth map \((a,b)\mapsto(a-\mu)/(b-a^2)^{1/2}\), in a neighborhood of the population moments where the denominator is positive. The moment assumption supplies the fourth moment of this vector. The displayed coefficients use the divisor-\(n\) variance fixed above.
For the bootstrap coverage comparisons below, impose the following sufficient conditions: \(\mathbb{E}|X|^r<\infty\) for every positive integer \(r\), and, for each such \(r\),
These explicit assumptions allow the joint expansions of every finite vector of empirical moments needed below. They are stronger than the finite-variance assumptions for consistency. We use Hall's Theorems 5.1–5.2 for conditional cdfs and quantiles, and Section 5.3 for coverage probabilities with estimated critical values. In particular, the coverage claims use those joint expansions, rather than replacing an \(O_P(n^{-1})\) error inside an event by an \(O(n^{-1})\) error in probability. All rates below are for fixed \(0<\alpha<1/2\), or uniformly over a compact subinterval of this range.
Normal intervals. The upper confidence bound is \(U_N=\bar X_n+\hat\sigma_n z_{1-\alpha}/\sqrt n\). Since \(\mu\le U_N\) is the event \(T_n\ge z_\alpha\), Theorem 7 gives
For the symmetric interval \(I_N=[\bar X_n\pm\hat\sigma_n z_{1-\alpha}/\sqrt n]\), the two cdf corrections cancel: \(q_1\phi\) is even, so
Here the minus after the lower endpoint denotes the left limit of the cdf. The uniform continuous Edgeworth approximation bounds any jump by its remainder order, so the same expansion applies at that left limit. Thus two-sided normal coverage is already accurate to order \(n^{-1}\). This cancellation does not say that the errors in the two individual tails are of that order.
The inversion step can be checked before considering random critical values. Suppose \(H_n(x)=\Phi(x)+n^{-1/2}a(x)\phi(x)+O(n^{-1})\) uniformly near \(z=z_\alpha\), with \(a\) smooth. Set \(x_n=z+n^{-1/2}b\). Taylor expansion gives
Since \(\phi(z)>0\), choose \(b=-a(z)\). Evaluating at \(x_n\pm C/n\) for a sufficiently large constant traps the generalized inverse, giving \(H_n^{-1}(\alpha)=z-n^{-1/2}a(z)+O(n^{-1})\). The same trapping works in probability for the conditional cdf and estimated coefficients. This derives the signs in both bootstrap quantile formulas below; the coverage remainder still requires the separate joint expansion already specified.
The unstudentized bootstrap. Put \(\hat\gamma=\gamma(\hat F_n)\). The conditional expansion and its inverse are
Here the upper basic-bootstrap bound is \(U_B=\bar X_n-J_n^{-1}(\alpha,\hat F_n)/\sqrt n\). Hall's coverage expansion (3.39), justified in Section 5.3, applies to the smooth empirical-moment coefficients in this quantile. Its first correction is obtained by adding the quantile shift \(\gamma(z_\alpha^2-1)/6\) to the studentized cdf correction \(q_1(z_\alpha)\):
The random estimation of \(\gamma\) enters at the next order in that joint expansion. There is no general ordering between this error constant and the normal-interval constant. The equal-tailed basic interval has endpoints
Its coverage is \(1-2\alpha+O(n^{-1})\), by cancellation of the even first correction.
In the basic-bootstrap coverage calculation the first coefficient is
Coverage of an upper bound is one minus the lower-tail probability, so this coefficient appears with a minus sign. For an equal-tailed interval the same expression, multiplied by the even density \(\phi\), takes equal values at \(z_\alpha\) and \(-z_\alpha\); subtracting the two cdf values removes the entire \(n^{-1/2}\) term.
The studentized bootstrap. The corresponding expansion is
For \(U_t=\bar X_n-\hat\sigma_nK_n^{-1}(\alpha,\hat F_n)/\sqrt n\), the first quantile correction is \(-q_1(z_\alpha)\). It cancels the first cdf correction in the same coverage theorem, giving
The equal-tailed interval uses the \(1-\alpha\) and \(\alpha\) quantiles in its lower and upper endpoints respectively, both with a minus sign; its coverage is \(1-2\alpha+O(n^{-1})\). Studentization improves one-sided coverage because the normal limit no longer contains an unknown scale.
A different construction uses the absolute studentized root, with both the absolute value and resampled denominator included:
Under the same sufficient conditions, the symmetric bootstrap interval \([\bar X_n\pm\hat\sigma_nc_n^*/\sqrt n]\) has coverage \(1-2\alpha+O(n^{-2})\), by Hall, Section 3.6, equations (3.53)–(3.58), with the remainder justification of Sections 5.2–5.3. In that argument the even first correction cancels from the distribution of \(|T_n|\); estimating its next correction changes the critical value by \(O_P(n^{-3/2})\), and the odd-order term in the joint coverage expansion cancels between the two tails. The last cancellation, with the stated remainder control, is what gives \(n^{-2}\); critical-value accuracy alone would not establish it.