Many statistics average terms that share observations, so their summands are dependent. Projection separates the part explained by individual observations from higher-order interactions. For a nondegenerate U-statistic, the first-order projection is a sum of independent terms and the remainder is smaller at the central-limit scale. We develop the orthogonal decomposition and then apply it to variance, rank, and association statistics.
If \(T_n-S_n\to_P0\) and \(S_n\Rightarrow Z\), Slutsky's lemma gives \(T_n\Rightarrow Z\). Orthogonal projection supplies useful approximations because it minimizes a mean square error and makes the error orthogonal to the approximation. We work in the real Hilbert space \(L^2(P)\), identifying random variables equal almost surely, with inner product \(\langle U,V\rangle=E(UV)\).
Let \(\mathsf S\) be a closed linear subspace of \(L^2(P)\), and let \(T\in L^2(P)\). The subspace is a chosen class of approximating variables; taking all of \(L^2(P)\) would give the trivial projection \(T\) itself.
Definition 1. The projection \(\widehat S\) of \(T\) onto \(\mathsf S\) is the element minimizing \(E(T-S)^2\) over \(S\in\mathsf S\).
Proposition 2. The projection exists, is unique almost surely, and is characterized by
because the last squared norm is at least \(d^2\). Completeness and closedness give a limit \(\widehat S\in\mathsf S\) with distance \(d\). For any \(S\in\mathsf S\), the quadratic \(t\mapsto\|T-\widehat S-tS\|_2^2\) is minimized at zero, so its linear coefficient vanishes. Conversely, orthogonality gives
This proves optimality and uniqueness. Taking \(S=1\) proves equality of means; the covariance assertion then follows from the same orthogonality. ◻
Theorem 3. Suppose \(\mathsf S_n\) contains constants, \(T_n\in L^2\), and \(\widehat S_n\) is its projection. Assume both variances are positive eventually and \(\operatorname{Var}(T_n)/\operatorname{Var}(\widehat S_n)\to1\). Then
Proof. Put \(v_n=\operatorname{Var}T_n\) and \(w_n=\operatorname{Var}\widehat S_n\). Orthogonality implies \(E(T_n-\widehat S_n)^2=v_n-w_n\). The difference in the display is the sum of the residual divided by \(\sqrt{v_n}\) and \((\widehat S_n-E\widehat S_n)(v_n^{-1/2}-w_n^{-1/2})\). These two terms are orthogonal, so its squared \(L^2\) norm is
For the remainder of the projection discussion, let \(X_1,\ldots,X_n\) be independent. They need not have the same distribution. Define the additive subspace
It is the orthogonal sum of the constants and the \(n\) centered coordinate spaces \(\{g_i(X_i):Eg_i=0\}\). Each is closed, and a finite orthogonal sum is closed: convergence of a sum in \(L^2\) forces convergence of each component by Pythagoras.
Definition 4. Projection onto \(\mathfrak S\) is called the Hájek projection.
Theorem 5. For \(T\in L^2\), its Hájek projection is
Proof. The constant projection is \(ET\). Conditional expectation is an \(L^2\) contraction, so each summand is square integrable and centered. For a centered \(g_i(X_i)\), conditioning shows \(E[Tg_i]=E[E(T\mid X_i)g_i]\). Every other centered coordinate term has zero inner product with \(g_i\), by independence. Thus the displayed residual is orthogonal to all additive functions, and Proposition 2 applies. ◻
For \(A\subseteq[n]=\{1,\ldots,n\}\), write \(E_AT=E[T\mid X_i:i\in A]\), with \(E_\emptyset T=ET\). Distinguish the full coordinate space \(L_A=L^2(\sigma(X_i:i\in A))\) from the canonical interaction space
\[\mathcal H_A=\{g\in L_A:E_Bg=0\text{ for every }B\subsetneq A\},\qquad \mathcal H_\emptyset=\{\text{constants}\}.\]
The zero conditional means are part of the definition; arbitrary functions of \(X_A\) do not have this property. These spaces are closed since conditional expectations are continuous linear maps. Independence gives the commuting identity \(E_BE_AT=E_{A\cap B}T\). To see it, first replace \(T\) by its conditional expectation on all coordinates; integrating a product of coordinate functions proves the identity, and approximation by finite linear combinations of such products extends it to \(L^2\).
If \(A\ne B\), choose, after interchanging them if needed, an index in \(A\setminus B\). For \(g\in\mathcal H_A\), the commuting identity gives \(E_Bg=E_{A\cap B}g=0\). Hence \(E(gh)=0\) for every \(h\in\mathcal H_B\). Thus distinct canonical interactions are orthogonal.
Theorem 6. The Hoeffding projection onto \(\mathcal H_A\) is
In particular, if \(T\) is a function of all \(n\) coordinates, then \(T=\sum_{A\subseteq[n]}P_AT\) in \(L^2\), and \(\operatorname{Var}T=\sum_{A\ne\emptyset}E(P_AT)^2\).
Proof. Fix a proper subset \(C\subsetneq A\) and an index \(i\in A\setminus C\). In \(E_CP_AT\), pair each \(B\) not containing \(i\) with \(B\cup\{i\}\). Their signs are opposite and their intersections with \(C\) coincide, so every pair cancels. Consequently \(E_CP_AT=0\) and \(P_AT\in\mathcal H_A\).
For \(g\in\mathcal H_A\), we have \(E[(T-E_AT)g]=0\) by conditioning on \(X_A\). Every other term \(E_BT\), \(B\subsetneq A\), is orthogonal to \(g\), since \(E_Bg=0\). Thus \(E[(T-P_AT)g]=0\), proving the projection formula.
To sum the components, the coefficient of \(E_CT\) in \(\sum_{B\subseteq A}P_BT\) is
which is one when \(C=A\) and zero otherwise. This proves the identity and the direct-sum decomposition. It also proves the source's assertion that \(T\perp\mathcal H_B\) for every \(B\subseteq A\) implies \(E_AT=0\). For a general \(T\), the sum over \(B\subseteq A\) is \(E_AT\), not \(T\) unless \(T\) is \(X_A\)-measurable. ◻
The first components are \(P_\emptyset T=ET\), \(P_{\{i\}}T=E[T\mid X_i]-ET\), and
The projection on interactions of order at most \(p\) is \(\sum_{|A|\le p}P_AT\).
1. Basics of Unbiased Statistics
Now let the observations be i.i.d., and let \(h\) be a measurable symmetric kernel of fixed order \(r\). We estimate \(\theta=Eh(X_1,\ldots,X_r)\). Symmetrizing a nonsymmetric kernel over its \(r!\) permutations preserves its expectation. Using only the first \(r\) observations wastes the rest of the sample; averaging over all subsets removes that dependence on the labels.
Definition 7. For \(n\ge r\), the U-statistic with kernel \(h\) is
Here the arguments can be listed in increasing index order, since \(h\) is symmetric. Every summand has mean \(\theta\), so \(EU_n=\theta\).
When \(Eh^2<\infty\), Jensen's inequality applied pointwise to the average of \(h(X_A)-\theta\) gives \(\operatorname{Var}U_n\le\operatorname{Var}h(X_1,\ldots,X_r)\); strict inequality is not automatic. For real observations, the conditional expectation identity is
Indeed, averaging over all permutations of the sample gives \(U_n\), and exchangeability makes the conditional expectation of each permuted kernel the same given the ordered full sample. This remains true with ties. Conditioning only on a kernel evaluated at a few order statistics would not give this identity.
Example 8. The kernel \(h(x)=x\) gives the sample mean. With finite variance, its asymptotic normality is the usual i.i.d. central limit theorem.
Example 9. For \(h(x,y)=(x-y)^2/2\), the parameter is \(\operatorname{Var}X\) and
The identity follows by expanding the squares: \(\sum_{i<j}(X_i-X_j)^2=n\sum_iX_i^2-(\sum_iX_i)^2\). Finite second moment is enough for unbiasedness; applying the following square-integrable-kernel theorem to this example requires a finite fourth moment.
Let \(h_1(x)=E[h(x,X_2,\ldots,X_r)]-\theta\). Among the \(\binom nr\) subsets, exactly \(\binom{n-1}{r-1}\) contain a given index \(i\). The others have conditional mean \(\theta\), so
where \(\zeta_1=E h_1(X_1)^2\). The result includes \(\zeta_1=0\), with a point mass at zero as the limit; a different scaling may then be needed for a nondegenerate limit.
Proof. For \(0\le c\le r\), let \(\zeta_c\) be the covariance of two kernel evaluations sharing exactly \(c\) observations and using independent observations otherwise. Conditional independence given the shared variables gives
Each is finite. Fix the first subset \(A\). A second subset overlaps it in \(c\) indices in exactly \(\binom rc\binom{n-r}{r-c}\) ways. Consequently, with impossible binomial coefficients interpreted as zero,
For fixed \(r\), the coefficient for \(c=1\) is \(r^2/n+O(n^{-2})\), by expanding its finite product of linear factors; the coefficient for \(c\ge2\) is \(O(n^{-c})\), by comparing polynomial degrees. Therefore
The projection has variance exactly \(r^2\zeta_1/n\). Orthogonality now gives \(E(U_n-\theta-\widehat U_n)^2=O(n^{-2})\), proving the first assertion without dividing by a potentially zero variance. The i.i.d. CLT applied to \(h_1(X_i)\), followed by Slutsky, proves the second. ◻
Example 11. For the signed-rank kernel \(h(x,y)=\mathbf1\{x+y>0\}\), put \(\theta=P(X_1+X_2>0)\). If \(F\) is the distribution function, then
The strict inequality in the kernel explains the use of \(F(-x)\), even when there are atoms.
Under the null that \(F\) is continuous and symmetric about zero, \(F(-x)=1-F(x)\), \(\theta=1/2\), and \(F(X)\) is uniform on \((0,1)\). Hence \(\zeta_1=1/12\) and \(\sqrt n(U_n-1/2)\Rightarrow \mathcal{N}(0,1/3)\). A one-sided test rejects when \(\sqrt{3n}(U_n-1/2)>z_{1-\alpha}\), where \(z_p=\Phi^{-1}(p)\). Symmetry implies \(\theta=1/2\) here; that single probability equality does not characterize symmetry or ensure the same null variance.
Let \(R_i^+\) be the rank of \(|X_i|\), and suppose the sample has no zeros or tied absolute values. Wilcoxon's statistic is \(W^+=\sum_iR_i^+\mathbf1\{X_i>0\}\). The diagonal part of each rank contributes \(\mathbf1\{X_i>0\}\). For each unordered pair, the remaining contribution is one precisely when the observation of larger absolute value is positive, which is equivalent to a positive pair sum. Thus
\[W^+=\binom n2U_n+\sum_i\mathbf1\{X_i>0\}.\]
Under the continuous symmetric null, \(EW^+=n(n+1)/4\). The centered last sum is bounded in absolute value by \(n\), hence is negligible after multiplication by \(n^{-3/2}\). Since \(n^{-3/2}\binom n2=(\sqrt n/2)(1-1/n)\),
Assume continuous marginals so ties have probability zero. It is the U-statistic with kernel twice the concordance indicator minus one. Write \(\theta=E\tau_n\), \(F^\ell(x,y)=P(X<x,Y<y)\), \(F^r(x,y)=P(X>x,Y>y)\), and \(q=F^\ell+F^r\). Conditional concordance probability is \(q(x,y)\), so the first kernel is \(h_1(x,y)=2q(x,y)-1-\theta\). Thus
If \(X\) and \(Y\) are independent, \(A=F_X(X)\) and \(B=F_Y(Y)\) are independent uniforms, and \(q(X,Y)=AB+(1-A)(1-B)\). Hence \(2q-1=(2A-1)(2B-1)\), of mean zero and variance \((1/3)^2\). Therefore \(\sqrt n\tau_n\Rightarrow \mathcal{N}(0,4/9)\), and a two-sided level-\(\alpha\) asymptotic test rejects if \(\sqrt{9n/4}|\tau_n|>z_{1-\alpha/2}\). This detects departures with nonzero population Kendall correlation; zero correlation does not imply independence.
For two samples, let \(X_1,\ldots,X_m\) be i.i.d. from \(F\), and independently let \(Y_1,\ldots,Y_n\) be i.i.d. from \(G\). The distributions may differ. For a fixed kernel symmetric separately in its \(r\) first and \(s\) last arguments, define
Here \(\zeta_{a,b}\) is the covariance of kernel evaluations sharing \(a\) of the \(X\)'s and \(b\) of the \(Y\)'s, with independent remaining coordinates. Equivalently it is the variance of the kernel's conditional expectation given those shared variables.
Proof. Independently count the overlapping subsets in the two samples to obtain
The \((0,0)\) term is zero. The coefficients of \((1,0)\) and \((0,1)\) are \(r^2/m+O(N^{-2})\) and \(s^2/n+O(N^{-2})\): the zero-overlap coefficient in the other sample is \(1+O(N^{-1})\). All terms with \(a+b\ge2\) are \(O(N^{-2})\). It follows that
The first two terms are exactly the projection variance, so orthogonality proves the remainder bound. The two independent sample means in the projection satisfy their respective CLTs; multiplying them by \(r\sqrt{N/m}\) and \(s\sqrt{N/n}\) gives the stated variance. This also covers a zero limiting variance. ◻
Example 14. For Mann–Whitney, \(h(x,y)=\mathbf1\{x\le y\}\), \(r=s=1\), and
The left limit is necessary for the weak inequality when \(G\) has atoms. For continuous \(G\), this becomes the centered negative average of \(G(X_i)\) plus the centered average of \(F(Y_j)\). Under \(F=G\) continuous, both first-kernel variances equal \(1/12\) and \(\theta=1/2\), giving