Kernel Density Estimation: Pointwise and Integrated Risk
Kernel density estimation replaces each observation by a small local bump. The bandwidth determines the tradeoff between approximation bias and variance. We calculate pointwise risk, integrated risk, and then an exact Fourier representation that also identifies an oracle smoother.
Let \(X_1,\ldots,X_n\) be independent with density \(p\), cdf \(F\), and empirical cdf \(F_n\). The Glivenko–Cantelli theorem gives \(\sup_x|F_n(x)-F(x)|\to0\) almost surely. At a continuity point of \(p\), \([F(x+h)-F(x-h)]/(2h)\to p(x)\). This motivates the Rosenblatt estimator
More generally, for \(h>0\), a real kernel \(K\in L^1(\mathbb R)\cap L^2(\mathbb R)\), and \(\int K=1\), define \(\widehat p_n(x)=(nh)^{-1}\sum_iK((X_i-x)/h)\). Higher-order kernels may take negative values, so the estimate need not itself be a nonnegative density. The shrinking denominator means cdf consistency alone does not prove density consistency; the risk bounds below supply the needed bandwidth control.
1. Mean Squared Error
At a fixed \(x\), expand the square around the mean. The centered cross term has expectation zero, giving
The first term is squared bias, rather than bias itself.
1.1. Variance of kernel density estimators
Independence eliminates the covariance terms. Changing variables \(z=x+hu\) gives the exact expression and its upper bound:
If \(\|p\|_\infty\le\overline p\), this is at most \(\overline p\|K\|_2^2/(nh)\). The bound is uniform in \(x\), and uniform over a density class only when a common \(\overline p\) is available.
1.2. Bias of kernel density estimators
For \(\beta>0\), let \(\ell=\lceil\beta\rceil-1\), so that \(0<\beta-\ell\le1\), including integer \(\beta\). This avoids using the usual floor symbol for a different integer convention. The Hölder class \(\Sigma(\beta,L)\) consists of \(\ell\)-times continuously differentiable functions satisfying \(|p^{(\ell)}(x+t)-p^{(\ell)}(x)|\le L|t|^{\beta-\ell}\) for all \(x,t\). For \(\ell=0\), this is a condition directly on \(p\).
A kernel of order \(\ell\) has \(\int K=1\), finite absolute moments through order \(\ell\), and \(\int u^jK(u)du=0\) for \(1\le j\le\ell\). Also require \(M_\beta(K)=\int|u|^\beta|K(u)|du<\infty\). Since
the order-zero case immediately gives \(|b(x)|\le LM_\beta(K)h^\beta\). For \(\ell\ge1\), the correct integral Taylor formula is
It follows by integrating the \(\ell\)-th derivative remainder and subtracting \(t^\ell p^{(\ell)}(x)/\ell!\). All polynomial terms cancel upon integration against \(K\), except the constant \(p(x)\), which is subtracted in the bias. Thus
with \(C_\beta=1\) for \(\ell=0\). This also justifies the integrability of the remainder for signed kernels.
Let \(\mathcal P(\beta,L)\) be the densities in this Hölder class. They have a common finite sup-norm bound. To see this without adding an assumption, use the compact, bounded order-\(\ell\) kernel \(K_0\) constructed in Remark 1 at bandwidth one. The bias argument gives
Here \(p\ge0\) and \(\int p=1\) bound the convolution, irrespective of the signs of \(K_0\). Therefore the pointwise bound is uniform over both \(p\) and \(x\): \(\operatorname{MSE}(x)\le Ah^{2\beta}+B/(nh)\), where \(A=[C_\beta LM_\beta(K)]^2\) and \(B=\overline p\|K\|_2^2\). When \(A,B>0\), the minimizer of this upper bound is
The constants cannot be dropped when claiming an exact minimizing bandwidth. For any fixed \(\alpha>0\), \(h=\alpha n^{-1/(2\beta+1)}\) nevertheless gives \(\sup_{p\in\mathcal P(\beta,L)}\sup_x\operatorname{MSE}(x)\le Cn^{-2\beta/(2\beta+1)}\).
Remark 1. An explicit compact kernel is obtained from the orthonormal Legendre polynomials
The constant term \(m=0\), with \(\varphi_0=1/\sqrt2\), is essential. For any polynomial \(q\) of degree at most \(\ell\), expand \(q=\sum_{m=0}^\ell b_m\varphi_m\). Orthonormality gives \(\int q(u)K_0(u)du=\sum_mb_m\varphi_m(0)=q(0)\). Taking \(q=1,u,\ldots,u^\ell\) proves all moment conditions. Compact support and boundedness give every absolute moment and square integrability.
2. Mean Integrated Squared Error
Assume \(p\in L^2(\mathbb R)\). Tonelli applied to squared errors gives \(\operatorname{MISE}=\|b\|_2^2+\int\sigma^2(x)dx\). Unlike the pointwise variance bound, integrated variance needs no sup-norm bound on \(p\):
For an \(L^2\) smoothness condition, use the Nikol’ski class \(\mathcal H(\beta,L)\): \(p\in L^2\), with weak derivatives through order \(\ell=\lceil\beta\rceil-1\) in \(L^2\), and
For \(\ell\ge1\), choose the locally absolutely continuous representatives of the lower derivatives, so the preceding integral Taylor formula is valid almost everywhere. Minkowski's integral inequality first in \(u\), then in \(v\), gives
For \(\ell=0\), apply Minkowski directly to \(b=\int K(u)[p(\cdot+hu)-p]du\); no negative factorial or nonexistent derivative is needed. For integer \(\beta\), the Sobolev condition \(p\in W^{\beta,2}\), \(\|p^{(\beta)}\|_2\le L\), implies the stated translation bound with \(\ell=\beta-1\): integrate the last derivative over a segment and use Minkowski, obtaining \(\|p^{(\beta-1)}(\cdot+t)-p^{(\beta-1)}\|_2\le |t|L\).
Consequently \(\operatorname{MISE}\le Ah^{2\beta}+\|K\|_2^2/(nh)\). Its minimizing bandwidth includes \(A\), just as in the pointwise calculation. In particular,
3. Fourier Analysis of Kernel Density Estimators
Use the convention \(\mathcal F[g](\omega)=\int e^{it\omega}g(t)dt\). Plancherel says \(\|g\|_2^2=(2\pi)^{-1}\|\mathcal Fg\|_2^2\), first for \(L^1\cap L^2\), then for all \(L^2\) by its isometric extension. Scaling and reflection/translation give
The characteristic function is \(\phi(\omega)=\int e^{ix\omega}p(x)dx\), the transform of the density (equivalently the probability measure), not the cdf as an ordinary function. Its empirical version is \(\phi_n(\omega)=n^{-1}\sum_je^{iX_j\omega}\). For a real symmetric kernel, \(\widehat K=\mathcal FK\) is real and even, and
In this section we may enlarge the smoothing class to all real symmetric \(K\in L^2\). The estimator is then a finite sum of \(L^2\) translates, its transform identities hold in \(L^2\), and neither \(\int K=1\) nor nonnegativity is imposed. This distinction matters for the oracle and clipping arguments below.
Independence gives \(\mathbb E\phi_n=\phi\) and
For complex numbers the cross term in the variance is a real part with a conjugate, so
Decompose \(\phi_n\widehat K-\phi=(\phi_n-\phi)\widehat K+(\widehat K-1)\phi\). The cross term has expectation zero because the first factor is centered. Tonelli and Plancherel now yield the exact risk
All terms are integrable when \(p,K\in L^2\): \(|\phi|\le1\), \(\phi\in L^2\), and \(\widehat K(h\cdot)\in L^2\). Equivalently, this is the integrated squared bias plus \((2\pi n)^{-1}\int|\widehat K(h\omega)|^2d\omega\), minus \((2\pi n)^{-1}\int|\phi(\omega)\widehat K(h\omega)|^2d\omega\).
Remark 2. At a fixed frequency write \(a=|\phi|^2\), \(b=e^2=(1-a)/n\), and \(w=\widehat K(h\omega)\). Since \(a+b\ge1/n>0\), completing the square gives
Thus the oracle multiplier and risk are
This is an actual admissible \(L^2\) multiplier: it is real and even, and \(0\le w^*\le\min(1,n|\phi|^2)\), so \(\int|w^*|^2\le n\int|\phi|^2<\infty\). Its inverse transform defines a real symmetric \(L^2\) kernel at any fixed bandwidth. It depends on unknown \(p\), and need not define a nonnegative normalized density estimator; its role is a benchmark in the enlarged linear-smoothing class.
Call a smoother inadmissible in this \(L^2\) class if another has no larger risk for every square-integrable density, every \(h>0\), and every integer \(n\ge1\), and has strictly smaller risk for some density. If the set where \(\widehat K\notin[0,1]\) has positive Lebesgue measure, a dominating kernel is obtained by clipping:
Clipping preserves reality and evenness, and \(|\widehat{\widetilde K}|\le|\widehat K|\), so its inverse transform lies in the same \(L^2\) class. At each frequency the optimum \(a/(a+b)\) lies in \([0,1]\); projecting \(w\) onto this interval cannot increase its squared distance from that optimum. The completed-square identity proves risk domination. For a Gaussian density, \(0<|\phi(\omega)|^2<1\) at every \(\omega\ne0\), so the optimum is strictly inside \((0,1)\). On the positive-measure clipping set the decrease is strict, and integration proves a strict risk improvement for every \(n,h\). This establishes the omitted inadmissibility argument. It does not assert that clipping preserves \(L^1\) integrability or density normalization.
If \(K\in L^1\cap L^2\), its transform is continuous and \(\widehat K(0)=\int K\). Thus \(\int K>1\) gives a positive-measure clipping set and inadmissibility in the enlarged class. If instead \(\widehat K(0)\ne1\), continuity gives \(\delta,t_0>0\) with \(|1-\widehat K(t)|\ge\delta\) for \(|t|\le t_0\). Hence the squared bias obeys
This rules out consistency as \(h\downarrow0\), even if \(n\to\infty\). For integrals strictly between zero and one, inconsistency alone does not prove the domination criterion above.
Remark 3. A Fourier Sobolev class is defined by \(p\ge0\), \(\int p=1\), and
This implies \(\phi\in L^2\), since \(|\phi|\le1\) controls \(|\omega|\le1\). For integer \(\beta\) and \(p\in W^{\beta,2}\), the weak-derivative transform is \(\mathcal F[p^{(\beta)}]=(-i\omega)^\beta\phi\); Plancherel gives the equality \(\int|\omega|^{2\beta}|\phi|^2=2\pi\|p^{(\beta)}\|_2^2\). For noninteger \(\beta\), the displayed Fourier condition itself defines the class.
Suppose \(A=\operatorname*{ess\,sup}_{t\ne0}|1-\widehat K(t)|/|t|^\beta<\infty\). Factoring \(|h\omega|^{2\beta}\) out of the bias integrand gives
For \(h=\alpha n^{-1/(2\beta+1)}\), the constant is \(A^2L^2\alpha^{2\beta}+\|K\|_2^2/\alpha\). The factor \(2\pi\) cancels with Plancherel's prefactor. This recovers (1) under a Fourier smoothness condition.
Remark 4. The global multiplier bound can be checked locally if \(\widehat K\) is essentially bounded, for example if \(K\in L^1\cap L^2\). If the quotient is bounded by \(A_0\) for \(0<|t|\le t_0\), then for \(|t|>t_0\) it is at most \((1+\|\widehat K\|_\infty)/t_0^\beta\); the converse is immediate. The local bound lets us choose a representative with \(\widehat K(0)=1\) continuous at zero: change its values on the exceptional null set to one and use \(|\widehat K(t)-1|\le A_0|t|^\beta\).
Square integrability alone does not make the local and global conditions equivalent. For example, take an even multiplier equal to one near zero and with spikes of height \(m\) on disjoint intervals of length \(2^{-m}m^{-4}\) inside \((2,3)\), with reflected spikes in \((-3,-2)\), and zero elsewhere. It lies in \(L^2\) and satisfies the local condition, while the global essential supremum is infinite. The boundedness hypothesis is therefore substantive.