When to Cluster: Sampling, Assignment, and Fixed Effects
A variance describes how an estimator changes when data are collected again. To decide whether to cluster, one must specify what is sampled again and how treatment is reassigned. Holding potential outcomes fixed, (Abadie et al. 2022) separates these sources of randomness and shows why residual correlation alone cannot determine the answer.
1. The design
A finite population of \(n\) units is partitioned into clusters of sizes \(n_m\). Sample each cluster with probability \(q\) and each unit within a sampled cluster with probability \(p\). Independently, draw a treatment probability \(A_m\) for each cluster, with mean \(\mu\) and variance \(\sigma_A^2\), then assign treatment independently conditional on \(A_m\).
The estimand is the population ATE \(\tau=\sum_m(n_m/n)\tau_m\). Under the paper’s growing-cluster-count, cluster-size, bounded-outcome, and nondegeneracy conditions, the difference in means has a CLT around \(\tau\).
2. What the variance comparison says
Proposition 1. In this design, the usual heteroskedasticity-robust variance can be either too small or too large. The limiting cluster-robust variance is conservative, with excess
Here \(v\) is the asymptotic variance for the estimator scaled by the square root of the realized sample size.
The comparison belongs to this sampling and assignment model. Changing which clusters are sampled or how treatments are assigned changes the covariance terms and can change the ordering of variance formulas.
3. Derive the two sources of covariance
Write sample inclusion as \(R_i=C_mB_i\) for two distinct units in cluster \(m\). Since \(C_m\) has mean \(q\) and the \(B_i\) have mean \(p\),
Likewise, iterating over the random assignment probability gives
Both mechanisms create within-cluster cross-products in the estimator’s variance. Sampling every cluster eliminates common cluster-inclusion randomness. A fixed common treatment probability eliminates common assignment-probability randomness. These are properties of the design, whether or not a residual plot displays a pattern.
4. Why the conventional cluster formula can overshoot
The displayed excess variance depends on between-cluster treatment-effect heterogeneity and the fraction of the population observed. With equal cluster sizes \(n_m=h\), it reduces to
Thus the excess can remain large when many units per cluster are observed and the sampled clusters represent much of the population. Small \(p\) alone is not sufficient to dismiss clustering if \(h\) grows fast enough.
Cluster fixed effects remove cluster-specific outcome levels, but they do not remove all randomness in assignment or all treatment-effect heterogeneity. Their variance calculation uses within-cluster treatment variation, proportional in expectation to \(\mu(1-\mu)-\sigma_A^2\). Under fully clustered assignment that quantity is zero, so a within-cluster treatment coefficient is not identified.
The paper’s design-based variance adjustments use this information about sampling fractions and assignment. The corresponding issue for network clusters is different: one must first establish that the proposed groups are sufficiently independent.
5. A score expansion for the difference in means
Let \(\overline Y(t)=n^{-1}\sum_iY_i(t)\) and \(e_i(t)=Y_i(t)-\overline Y(t)\). Linearizing the treated and untreated sample means around their expected counts gives
This follows from the ratio identity \((A+a)/(B+b)-A/B\approx(a-(A/B)b)/B\) for each arm. The expected treated count is \(npq\mu\) and the expected control count is \(npq(1-\mu)\). The paper’s count and nondegeneracy conditions justify replacing the random denominators in this way.
Now write \(K_i=W_ie_i(1)/\mu-(1-W_i)e_i(0)/(1-\mu)\). Since sampling and assignment are independent,
For distinct units in the same cluster, \(\mathbb{E}[R_iR_j]=qp^2\). Also
and, because \(K_i\) is affine in \(W_i\),
The two cross-unit terms involve different products of fixed potential outcomes, whose signs need not agree. Deleting all off-diagonal terms can therefore make the robust formula too large or too small, depending on the design and outcomes.
5.1. Assemble the full variance
To obtain the variance of the full score, retain diagonal and off-diagonal terms separately before summing. Define
For a single unit, \(R_i^2=R_i\) and \(W_i(1-W_i)=0\). Thus \(\mathbb{E}H_i^2=pqb_i\), whereas \((\mathbb{E}H_i)^2=p^2q^2d_i^2\). For distinct units in one cluster, the preceding calculation gives
Different clusters are independent. Divide the variance of the total score by \(npq\), and write \(D_m=\sum_{i\in m}d_i\), \(A_m^{\rm score}=\sum_{i\in m}a_i\). The identity \(\sum_{i\ne j}a_ia_j=(\sum_i a_i)^2-\sum_i a_i^2\) now yields
The coefficient of \(\sum_i d_i^2\) is \(-pq-p(1-q)=-p\). This is an easy term to lose if the diagonal and off-diagonal contributions are combined too quickly. The formula is exact for the variance of the linearized score. The ratio-estimation remainder is what requires the paper’s asymptotic assumptions.
The first term is the probability limit of the usual robust variance on this scale. The negative terms remove variability that the realized design does not generate. The two cluster-square terms add back the common variation induced by sampling clusters and drawing their assignment probabilities. In particular, when \(q=1\) and \(\sigma_A^2=0\),
Even without clustered assignment, the robust formula can be conservative for a finite-population ATE. Conversely, if \(d_i=0\) and the \(a_i\) have the same sign within a large cluster, clustered assignment creates positive cross-products and the robust formula can be too small. These are different designs applied to fixed potential outcomes. They need not have different-looking regression residuals.
The covariance matrix must correspond to the sampling and assignment repetitions used to define the target. A superpopulation error model introduces another source of variation, potentially including changes in cluster effects. That variation is absent when the finite population of clusters is held fixed.
6. Deriving the conservative term
A cluster variance estimator aggregates scores first and then squares. At the population linearization above, the expected cluster sum is
The second moment of this sum differs from its variance by the square of that expectation. Scaling the estimator by \(\sqrt{npq}\) divides the score variance by \(npq\). Consequently the extra term is
Although the total score is centered, a cluster’s score can have nonzero mean when its treatment effect differs from the population average. Squaring each cluster sum before aggregation retains the squares of these means. They give the excess variance in the proposition.
Equal cluster-average treatment effects remove this particular excess. Observing every cluster does not: assignment remains random, but selection of cluster effects no longer is. Sampling fractions therefore determine which components belong in the target variance.
7. Fixed effects and the source of within-cluster variation
Conditional on \(A_m\), individual assignments are independent Bernoulli variables. For a cluster of size \(h\), let \(\overline W_m=h^{-1}\sum_iW_i\). Then
To derive it, use \(\sum_i(W_i-\overline W_m)^2=\sum_iW_i-h\overline W_m^2\) and the conditional mean and variance of \(\overline W_m\). Averaging over \(A_m\) gives
When \(A_m\) is always zero or one, all treatment varies between clusters and this quantity is zero. Cluster fixed effects then absorb treatment completely. When it is positive, within-cluster variation identifies a coefficient, but the weighting and variance depend on how much such variation each cluster supplies.
Fixed effects specify which treatment variation remains available. They do not settle how the residualized score co-varies across observations. The clustering calculation must be performed again for that residualized design.
To see the weighting directly, take a census and write \(N_{m1}=\sum_{i\in m}W_i\), \(N_{m0}=n_m-N_{m1}\). In a cluster with both arms observed,
Frisch–Waugh–Lovell therefore makes the fixed-effect coefficient a weighted average of within-cluster differences, with weights proportional to \(N_{m1}N_{m0}/n_m\). Clusters containing only one arm contribute zero. This identity also explains why a cluster with many observations but almost no within-cluster treatment variation contributes little to this regression.
For large \(n_m\), the weight is approximately \(n_m A_m(1-A_m)\). Under the common assignment law and suitable averaging over many clusters, its expectation is proportional to \(n_m\), recovering the population ATE under the paper’s conditions. If assignment laws instead vary systematically with cluster treatment effects, that step fails: the coefficient can target an assignment-variance-weighted effect. The algebra exposes the extra assumption needed to interpret the fixed-effect coefficient.
8. What to ask of the design
The formulas separate cluster sampling, sampling within clusters, and common variation in treatment probabilities. Each contributes different terms. Residuals alone cannot reveal the sampling fractions or establish which assignment mechanism generated treatment.
Under interference, outcomes also depend on other assignments, changing the score representation itself. Network dependence may cross the proposed cluster boundaries as well. Applying these variance calculations in either setting requires a corresponding identification model and covariance argument.