Post-Double Selection: Orthogonality, Assumptions, and Inference
A control can contribute little to outcome prediction and still affect inference for a treatment coefficient. If it strongly predicts treatment, omitting even a small outcome effect can shift the coefficient on the confidence-interval scale. Post-double selection (Belloni, Chernozhukov, and Hansen 2014) gives such a control two opportunities to enter: through the outcome regression and through the treatment regression.
1. The result and its scope
The partially linear model separates the treatment coefficient from two nuisance functions:
A large dictionary \(X=p(Z)\) approximates both nuisance functions using at most \(s\) terms. Run Lasso of \(Y\) on \(X\) and of \(D\) on \(X\), take the union of the selected controls, and regress \(Y\) on \(D\) and that union.
Theorem 1. Under the paper’s approximate-sparsity, sparse-eigenvalue, moment, and penalty conditions, including \(s^2\log^2(p\vee n)/n\to0\), the resulting estimator satisfies
The residual-based variance estimator is consistent under the accompanying conditions. The result is uniform over the specified approximately sparse class; exact recovery of a particular model is not required.
The sparsity rate states how fast approximation errors must shrink. It must be accompanied by moment and design assumptions: treatment needs residual variation, and the relevant sparse submodels need well-conditioned quadratic forms.
2. Why two selections help
Suppose a control has a small outcome coefficient but strongly predicts \(D\). Outcome selection may discard it because its contribution to prediction error is small. Its omission can nevertheless change the coefficient of \(D\) appreciably. The treatment regression gives this control a second route into the final model.
For a direct calculation, let \(D=\pi Z+v\), \(Y=\alpha_0D+\beta Z+\zeta\), and normalize \(\operatorname{Var}(Z)=1\). Assume \(v\) is uncorrelated with \(Z\) and \(\zeta\) is uncorrelated with both. Regressing \(Y\) only on \(D\) targets
Even \(\beta=c/\sqrt n\) gives a nonvanishing displacement on the scale of a confidence interval, although the omitted term’s prediction loss is only \(c^2/n\). Selection based only on predictive fit need not distinguish this error from noise. A strong coefficient \(\pi\) in the treatment equation makes the same control easier to retain there.
To understand why two approximations help, consider a moment that residualizes both treatment and outcome:
Replace \(m\) by \(m+a\) and \(g\) by \(g+b\). At the true treatment coefficient,
Conditional mean zero eliminates both terms linear in a nuisance error. The remaining product is bounded by \(\|a\|_2\|b\|_2\). Prediction errors of order \(\sqrt{s\log(p\vee n)/n}\) therefore give a bias smaller than \(n^{-1/2}\) under the stated sparsity rate.
In derivative notation, for perturbations \(h_g,h_m\),
The first equality identifies the two potentially harmful terms; the second follows by conditioning on \(Z\). The mixed derivative need not vanish. Orthogonality removes first-order nuisance bias, leaving a product that still has to be small.
Population cancellation is only part of the proof when the same observations select and fit the controls. The selected subspace depends on the noise. Model-size bounds and the probability inequalities behind the argument control this additional dependence.
3. Reading the final regression
Frisch–Waugh–Lovell writes the final coefficient as a regression between residualized treatment and outcome. The selected union approximates both nuisance regressions, leaving
The CLT applies once selection errors are absent from the leading term. Establishing that reduction is most of the argument. The fragility of sparsity concerns whether a proposed dictionary admits the required approximation; the inference theorem assumes that property within its specified class.
4. Expand the regression remainder
Let \(P\) project onto the selected union, \(M=I-P\), and put \(r_m=Mm\) and \(r_g=Mg\), treating the evaluated functions as sample vectors. Since \(D=m+v\), the residualized treatment is \(MD=r_m+Mv\). The final regression obeys the exact identity
Use \(M^2=M\) and the fact that \(r_m,r_g\) lie in the range of \(M\). The numerator becomes
This displays the influence sum \(v'\zeta\) and four remainders. The last is the sample counterpart of the nuisance-product bias:
Because selection uses the outcomes and treatments, conditioning on \(P\) does not make the errors independent of it. Instead, bound the noise scores simultaneously over all eligible sparse submodels. The event then also covers the randomly selected union.
The denominator has the corresponding expansion
Under the stated assumptions, division by \(n\) leaves \(\mathbb{E}[v^2]+o_p(1)\). The lower bound on this expectation prevents a small denominator from magnifying otherwise negligible remainder terms. Combining numerator and denominator gives the displayed influence expansion.
4.1. Control the two projected noise vectors together
The term \(v'P\zeta\) shows why a prediction rate alone is insufficient. For any subset \(J\) of at most \(Cs\) dictionary columns, put \(\widehat Q_J=X_J'X_J/n\), and suppose its smallest eigenvalue is at least \(\kappa_0>0\). Then
A simultaneous score bound of order \(\sqrt{\log p/n}\) gives \(\|P_Jv\|_{2,n}=O_p(\sqrt{s\log p/n})\), uniformly over these subsets. The same argument applies to \(\zeta\). Because the event holds for every eligible \(J\), it also holds for the random selected union. Therefore
The simultaneous score event makes the bound valid for the selected model even though selection used the noise. Together with prediction and size control, it recovers the first-order influence function of the infeasible regression with known nuisance functions.
Double selection thus supplies two approximations, and the proof bounds their products and their correlations with noise at the inference scale. Good predictive fit in one selected regression alone does not establish those bounds.
5. Studentization
Write \(\widehat v_i\) for the final treatment residual and \(\widehat\zeta_i=Y_i-\widehat\alpha D_i-X_i'\widehat\beta\) for the outcome residual. The natural variance estimate for the limiting distribution of \(\sqrt n(\widehat\alpha-\alpha_0)\) is
Consistency of this variance estimate requires control of squared residual products as well as the coefficient. Under the stated integrability and nuisance-rate conditions, Slutsky’s theorem gives \(\widehat\alpha\pm z_{1-\alpha/2}\widehat\sigma/\sqrt n\). The argument does not require selection of one exactly correct model.
The remaining correlations with selected-model errors
The two terms \(r_m'\zeta\) and \(v'r_g\) need an argument beyond the product bound. Here is one sufficient version, under independent observations, bounded conditional error variances, and the simultaneous sparse-projection bound already established. Write \(m=X\beta_m+a_m\) with oracle support of size at most \(s\), and suppose both \(\|a_m\|_{2,n}\) and \(\|r_m\|_{2,n}\) are \(O_p(r_n)\), where \(r_n=\sqrt{s\log(p\vee n)/n}\). Use the same approximation for \(g\).
Let \(A\) be the span of the selected dictionary columns together with the oracle support for \(m\). Its dimension is \(O(s)\). The vector \(MX\beta_m\) lies in \(A\), even though the choice of \(A\) is random, and
The score bound holds simultaneously over all such supports, so \(\|P_A\zeta\|_{2,n}=O_p(r_n)\). Consequently
For the approximation residual, decompose \(a_m'M\zeta=a_m'\zeta-a_m'P\zeta\). Conditional on the covariates, \(a_m\) is fixed and the first term divided by \(\sqrt n\) has mean zero and variance at most \(C\|a_m\|_{2,n}^2=o_p(1)\). Conditional Chebyshev, restricting first to events where this norm is small, makes it \(o_p(1)\). The second term divided by \(\sqrt n\) is bounded by \(\sqrt n\|a_m\|_{2,n}\|P\zeta\|_{2,n}=O_p(\sqrt n r_n^2)\). This proves \(r_m'\zeta/\sqrt n=o_p(1)\).
The identical decomposition of \(r_g\) uses \(\mathbb E[v\mid Z]=0\) and the sparse-projection bound for \(v\) to prove \(v'r_g/\sqrt n=o_p(1)\). All four numerator remainders are now controlled. These sufficient bounded-variance conditions explain the mechanism; the paper’s more general uniform theorem uses its stated moment and approximation conditions. Conditioning on the selected model itself would not justify either calculation.