The Fragility of Sparsity: Coordinates, Geometry, and Specification Tests
Changing coordinates can leave a regression’s fitted control space unchanged and alter which coefficients are zero. Variable selection depends on those zeros. (Kolesár, Müller, and Roelsgaard 2024) studies this dependence on representation and asks what supports the coordinates in which sparsity is assumed.
1. What the rotation result says
Consider \(Y_i=\beta D_i+W_i'\gamma+U_i\). Write \(\Sigma=\mathbb{E}[W_iW_i']\) and suppose its eigenvalues are bounded above and away from zero, with \(\|\gamma\|_2\asymp1\). If \(\widetilde W_i=RW_i\) for an orthogonal matrix \(R\), the same control function has coefficient vector \(R\gamma\).
Proposition 1. For a Haar-distributed rotation \(R\), let
For fixed constants in the approximation requirement \(r_s(R)\lesssim s/p\), a random rotation is very unlikely to preserve a sparse approximation when \(s\) is small relative to \(p\). Under the paper’s sparsity regime \(s=o(\sqrt p/\log p)\), the probability decays on a super-exponential scale in \(p\); its log is bounded above at the \(-p\log p\) scale.
The result concerns random representations of one fixed function. A scientifically chosen basis may still admit a sparse approximation; that choice requires a justification beyond invariance of the underlying regression.
2. The geometric calculation
Substituting the rotated variables leaves prediction error equal to
The eigenvalue bounds turn prediction error into Euclidean approximation error. A Haar rotation sends \(\gamma/\|\gamma\|\) to a uniform point on the unit sphere, which can be represented as \(G/\|G\|\) for independent standard normal coordinates \(G_j\).
Fix a support \(J\) of size \(s\). The squared distance to its coordinate subspace is
Thus, for a constant \(C\) determined by the eigenvalue and norm bounds,
For each support, almost all of a random direction’s mass must lie in \(s\) coordinates. A union bound compares the lower-tail factor \((s/p)^{(p-s)/2}\) with the number of supports, at most \((ep/s)^s\). For small \(s\), concentration near any one sparse subspace is too unlikely for the number of choices to compensate. The beta law applies to a fixed support; the union bound is what permits choosing the support after observing the vector.
Here is a coarse bound that makes the rate explicit. Put \(a=(p-s)/2\), \(b=s/2\), and \(\epsilon=C's/p\le1/2\). The beta density gives
For \(b\ge1\), the factor \((1-t)^{b-1}\) is at most one; for \(b=1/2\), it is at most \(\sqrt2\) on this interval. Bounding the beta integral below on \([1/4,1/2]\) shows \(\mathrm B(a,b)^{-1}\le e^{Cp}\) for \(1\le s\le p/2\), with a numerical constant. Hence
Under the stated regime, the first term is negative with magnitude of order at least \(p\log p\), while the support-count term is smaller. This proves a bound \(e^{-c p\log p}\) for some \(c>0\) and large enough \(p\). The calculation concerns a union over fixed supports; it does not pretend the support selected after seeing the rotation was fixed in advance.
A complementary calculation describes a typical rotated vector. First take \(\Sigma=I_p\) and \(\|\gamma\|=1\). If \(G_{(1)}^2\geq\cdots\geq G_{(p)}^2\) are the ordered squared Gaussian coordinates, then the best \(s\)-term approximation error is exactly
The retained energy is at most \(s\max_jG_j^2/\sum_jG_j^2\). A Gaussian tail bound and a union bound give
Also \(\sum_jG_j^2/p\to1\). Thus the retained fraction is \(O_p(s\log p/p)\), and \(r_s(R)\to1\) whenever \(s\log p/p\to0\). Bounded eigenvalues extend this to prediction error up to fixed constants.
The two calculations answer different questions. The beta tail bounds the rare event of an unusually accurate sparse approximation; the maximum-coordinate bound shows that a typical small set retains little signal. The full control space has lost no information: OLS projects onto the same space after rotation. Coordinatewise selection changes because it discards directions according to the basis.
The variance cost of using all controls can likewise be made explicit in the Gaussian homoskedastic illustration below. If \(p/n\to\kappa\in(0,1)\), then \(v'M_Wv\approx\tau^2n(1-\kappa)\) and \(v'v\approx\tau^2n\). The variance of the OLS-minus-oracle difference is approximately
With few controls relative to sample size, the two estimators may be close. Using many residual degrees of freedom increases the noise in their difference. Since both use the same outcomes, that scale must account for their covariance.
3. Other changes of representation
The paper also considers categorical reference choices and shifts of the origin in polynomial controls. These are familiar modelling choices, not arbitrary rotations. Under its categorical recoding scheme and regularity conditions, the chance of satisfying the required sparse approximation is exponentially small when a positive fraction of the relevant coordinates change.
For a polynomial, the issue can be seen without an asymptotic argument. A single term in one coordinate system becomes
The polynomial function is unchanged, yet its coefficient vector can change from one nonzero term to many. Changing an indicator reference category similarly changes which level differences are represented by zero coefficients.
The geometric objection concerns the assumptions behind post-double selection. When unrestricted OLS is available, it also suggests a way to examine those assumptions.
4. OLS as a specification check
Use the model
with zero conditional means for the errors. Let \(P_W\) project onto the columns of \(W\), \(M_W=I-P_W\), and \(\ddot D=M_WD\).
Theorem 2. Under conditional independence across observations, uniformly bounded \(2+\eta\) moments of \(U_i\) and fourth moments of \(\widetilde D_i\), conditional variances bounded away from zero, and \(\limsup p/n<1\), the paper establishes
For an estimator \(\widehat\beta_\star\) having the paper’s efficient sparse-model expansion, the appropriately standardized difference from OLS is also asymptotically normal. Feasible testing additionally requires the paper’s nuisance-approximation and variance-consistency conditions.
This is an oracle standard error because it contains \(U_i\). With many controls, obtaining a feasible consistent estimate is a separate argument. The displayed CLT does not validate every residual-based variance formula.
5. Start with Frisch–Waugh–Lovell
Partialling out \(W\) gives the exact identity
Conditional on \((D,W)\), this is a weighted sum of centered independent errors. Its conditional variance is
The squared denominator matters for the variance and the Lyapunov normalization. Given the moment bounds, the remaining CLT requirement is that no residualized treatment observation dominate \(\sum_i\ddot D_i^2\).
Quadratic-form bounds control this denominator because \(M_W\) is a projection. In particular, \(M_W^2=M_W\) and \(\operatorname{tr}(M_W)=n-\operatorname{rank}(W)\) explain why some residual variation remains when \(p/n\) stays below one.
For example, suppose \(\widetilde D\mid W\sim \mathcal{N}(0,\tau^2I_n)\) and let \(r=\operatorname{rank}(W)\). Then
The variance follows by diagonalizing the projection: it has \(n-r\) eigenvalues equal to one and the remainder equal to zero. The quadratic form is a sum of \(n-r\) independent squared Gaussian coordinates. Its relative variance is \(2/(n-r)\), so it concentrates if \(n-r\to\infty\). This illustrates the role of residual degrees of freedom without identifying the sample covariance of regressors with the population covariance.
For the weighted-error CLT, define \(w_i=\ddot D_i/(\ddot D'\ddot D)\) and \(V=\sum_iw_i^2\mathbb{E}[U_i^2\mid D,W]\). If conditional \(2+\eta\) moments are uniformly bounded and conditional error variances are bounded below, the Lyapunov ratio obeys
The final CLT step is therefore a bound on the largest contribution relative to the total. Invertibility of the regression matrix alone supplies no such bound.
6. Standardize the difference, not each estimator separately
The efficient expansion uses the population treatment residual \(\widetilde D_i\). Subtracting it from the OLS expansion gives, to first order,
with \(s_H^2=\sum_i Z_i^2U_i^2\). The paper’s nondegeneracy condition on the control projection and its moment restrictions ensure a useful normal limit for this difference.
Subtracting influence weights before squaring preserves the covariance between the estimators. Thus \(Z_i^2U_i^2\) is the appropriate contribution for their difference; adding separate variance estimates would omit the covariance.
The cancellation is especially transparent with homoskedastic errors of variance \(\sigma_U^2\). Temporarily use the oracle treatment residual \(v=\widetilde D\) and compare the two linear errors \((v'M_Wv)^{-1}v'M_WU\) and \((v'v)^{-1}v'U\). Their conditional covariance is
It equals the oracle estimator’s variance. The variance of the difference is therefore
This is nonnegative because projecting out controls can only reduce the treatment sum of squares. It can also be much smaller than either variance separately. If the projection removes no component of \(v\), the difference has zero variance and there is nothing to studentize. The paper’s nondegeneracy conditions rule out that case for its comparison test.
To make the statistic feasible, estimate the treatment residuals and outcome errors, obtaining \(\widehat Z_i\) and \(\widehat U_i\). What must be proved is
Feasibility requires nuisance errors small relative to the variance of the estimator difference. Consistency in an unspecified norm gives no such comparison. The paper’s rate and variance conditions justify the studentized normal approximation.
Rejection challenges the sparse-model restrictions used to justify the efficient estimator in this comparison. Nonrejection establishes less: it does not certify that sparse approximation holds in other coordinate systems.