Confidence Bounds, Conditioning, and Unbiased Tests

A confidence set can be constructed by retaining every parameter value whose hypothesis test accepts the observed data. This correspondence transfers optimality questions from confidence bounds to tests. We begin with one-sided bounds, then examine how nuisance parameters change the problem and how conditioning leads to unbiased and permutation tests.

Let \(0<\alpha<1\). A lower confidence bound \(L=L(X)\) for a real parameter satisfies \(P_\theta(L\le\theta)\ge1-\alpha\) for every \(\theta\in\Theta\). We allow extended endpoints when inversion reaches the edge of the parameter space. Randomized procedures can be represented as functions of the observations and an independent uniform variable.

Definition 1. A lower bound is uniformly most accurate if, among all bounds satisfying the coverage constraint, it minimizes \(P_\theta(L\le\theta')\) for every pair \(\theta'<\theta\). This asks for the smallest probability of including each false value below the truth.

Lemma 2. A uniformly most accurate bound simultaneously minimizes expected losses of the form

\[\mathcal L(\theta,b)=\int_{t<\theta}\mathbf1\{b\le t\}\,\nu_\theta(dt),\]

where \(\nu_\theta\) is any nonnegative measure, subject to coverage. Conversely, minimizing all these losses implies uniform accuracy, by taking point-mass measures.

Proof. Tonelli gives \(E_\theta\mathcal L(\theta,L)=\int_{t<\theta}P_\theta(L\le t)\,\nu_\theta(dt)\); every integrand is minimized by the uniformly most accurate bound. This includes the underestimation loss \((\theta-b)_+\) by taking Lebesgue measure. An arbitrary loss, or minimization for just one loss, does not imply the assertion in the archival statement.

More generally, a measurable random set \(S(X)\subseteq\Theta\) has confidence level \(1-\alpha\) if \(P_\theta(\theta\in S(X))\ge1-\alpha\) for every \(\theta\). A lower bound gives \(S(X)=[L(X),\infty)\cap\Theta\).

Proposition 3. Let \(A(\theta_0)\) be a level-\(\alpha\) acceptance region for testing \(\theta=\theta_0\), and define \(S(x)=\{\theta_0\in\Theta:x\in A(\theta_0)\}\). Then \(S\) has confidence level \(1-\alpha\). If each test is uniformly most powerful against \(K(\theta_0)\), then \(S\) minimizes \(P_\theta(\theta_0\in S(X))\) for every \(\theta_0\in\Theta\) and \(\theta\in K(\theta_0)\).

Proof. The event \(\theta_0\in S(X)\) is exactly \(X\in A(\theta_0)\). Setting the data-generating parameter to \(\theta_0\) proves coverage. Conversely, any competing confidence set \(S^*\) defines a level-\(\alpha\) test with acceptance event \(\theta_0\in S^*(X)\). At an alternative \(\theta\in K(\theta_0)\), UMP optimality therefore gives

\[P_\theta(\theta_0\in S^*(X))=P_\theta(X\in A^*(\theta_0))\ge P_\theta(X\in A(\theta_0))=P_\theta(\theta_0\in S(X)).\]

The false value is \(\theta_0\), not the true parameter \(\theta\), throughout this comparison.

Remark 4. This is test inversion: retain precisely the parameter values whose tests accept the observed sample. The same construction applies to randomized tests by using an auxiliary uniform variable. Confidence coverage concerns repeated samples at each fixed true parameter.

Lemma 5. Here is a convenient version with explicit endpoint conditions. Let \(\Theta\) be an interval and assume monotone likelihood ratio in \(T\): for \(\theta_1>\theta_0\), the density ratio is nondecreasing in \(T\). Suppose the cdf \(F_\theta(t)\) is continuous and strictly increasing in \(t\) on a common support interval, continuous in \(\theta\), and strictly decreasing in \(\theta\) there. Then the \((1-\alpha)\)-quantile \(c(\theta)\) is continuous and strictly increasing. Inverting the tests \(T>c(\theta_0)\) gives a uniformly most accurate lower bound. Whenever a solution exists,

\[F_{L(x)}(T(x))=1-\alpha\]

uniquely determines it. Outside the range of \(c\), use the corresponding extended boundary of \(\Theta\).

Proof. Neyman–Pearson applied to each \(\theta_1>\theta_0\) shows that an upper-tail test of null size \(\alpha\) is most powerful; the cutoff depends only on \(\theta_0\), hence is UMP. Strict stochastic ordering implies \(c(\theta_1)>c(\theta_0)\). For continuity at \(\theta\), values just below and above \(c(\theta)\) have cdf values strictly below and above \(1-\alpha\); continuity in the parameter brackets all nearby quantiles between them. Thus \(\{\theta:T(x)\le c(\theta)\}\) is an upper interval, with a closed endpoint whenever that endpoint lies in \(\Theta\). Proposition 3 proves accuracy. Strict monotonicity gives the unique inverse equation. Without strictness or the common-support conditions, one must retain the acceptance sets and use generalized inverses; one cannot simply assert ordinary inverse functions or finite endpoints.

1. Testing the mean of a normal distribution

Let \(X_1,\ldots,X_n\) be independent \(\mathcal{N}(\mu,\sigma^2)\), with \(n\ge2\), \(\sigma>0\), and put \(\nu=n-1\),

\[R=\sum_i(X_i-\overline X)^2=\sum_iX_i^2-n\overline X^2,\qquad S^2=R/\nu.\]

An orthogonal transformation of the standardized Gaussian vector separates its coordinate in the direction \((1,\ldots,1)/\sqrt n\) from its \(n-1\) perpendicular coordinates. These coordinates are independent standard normals. Consequently

\[\frac{\sqrt n(\overline X-\mu)}\sigma\sim \mathcal{N}(0,1),\qquad\frac R{\sigma^2}\sim\chi^2_\nu,\qquad\overline X\perp R.\]

This establishes the sampling identities directly, rather than deriving them from optimality claims.

Under \(\mu=\mu_0\), \(t=\sqrt n(\overline X-\mu_0)/S\) has a Student \(t_\nu\) law. With \(t_{\nu,p}\) denoting its \(p\)-quantile, the two-sided acceptance region is \(|t|\le t_{\nu,1-\alpha/2}\); a one-sided upper rejection region is \(t>t_{\nu,1-\alpha}\).

To justify unbiased optimality, write the normal likelihood in exponential-family form:

\[p_{\mu,\sigma}(x)=C(\mu,\sigma)\exp\left\{-\frac1{2\sigma^2}\sum_i x_i^2+\frac\mu{\sigma^2}\sum_i x_i\right\}.\]

The natural parameter space \(\{(\eta_1,\eta_2):\eta_1\in\mathbb R,\eta_2<0\}\) is open and convex. The conditional exponential-family UMP-unbiased theorem applies: condition on the nuisance sufficient statistic, require conditional rejection probability \(\alpha\) at a one-sided boundary, and additionally a zero centered first moment for a two-sided point null. Completeness of the nuisance family turns the unconditional unbiased constraints into these conditional constraints. Conditional likelihood-ratio comparisons then give optimality at each conditioning value and hence after integration.

The precise external result used for this optimality step is Lehmann and Romano, Testing Statistical Hypotheses, third edition, Theorem 4.4.1, pp. 119–123; the normal-model application is in §5.2, pp. 154–156. The calculations below verify the parameters and conditional regions for this application.

For testing \(\sigma\ge\sigma_0\) against \(\sigma<\sigma_0\), condition on \(\overline X\). The remaining statistic \(R\) is independent of it, so the UMP-unbiased test rejects for

\[R/\sigma_0^2<\chi^2_{\nu,\alpha}.\]

For \(\sigma\le\sigma_0\) against \(\sigma>\sigma_0\), reject for \(R/\sigma_0^2>\chi^2_{\nu,1-\alpha}\). The chi-square scale representation shows that the size is maximized at the null boundary and the power is at least \(\alpha\) on the respective alternative.

The union null \(\sigma\le\sigma_0\) or \(\sigma\ge\sigma_1\), where \(0<\sigma_0<\sigma_1\), is a different problem. Its UMP-unbiased rejection region is \(C_1<R<C_2\), with

\[\int_{C_1/\sigma_0^2}^{C_2/\sigma_0^2}f_{\chi^2_\nu}(z)\,dz=\int_{C_1/\sigma_1^2}^{C_2/\sigma_1^2}f_{\chi^2_\nu}(z)\,dz=\alpha.\]

These are the two conditional boundary constraints of the same theorem. The denominators are variances, not standard deviations.

For the point null \(\sigma=\sigma_0\), the UMP-unbiased acceptance region is \(a<R/\sigma_0^2<b\), where

\[\int_a^bf_{\chi^2_\nu}(z)\,dz=1-\alpha,\qquad\int_a^bz f_{\chi^2_\nu}(z)\,dz=\nu(1-\alpha).\]

The second equation is the zero power-derivative constraint. For example the scale score at the null is proportional to \(z-\nu\), so it requires \(E[(Z-\nu)\mathbf1\{a<Z<b\}]=0\). It is not generally the equal-tailed interval. Existence can be seen by sliding an interval carrying probability \(1-\alpha\) from its lowest possible position to its highest: its truncated first moment varies continuously from below \(\nu(1-\alpha)\) to above it. The theorem supplies the corresponding optimality.

For mean tests, center the observations at \(\mu_0\) and let \(Q=\sum_i(X_i-\mu_0)^2\). Under the null, the standardized Gaussian vector has an independent radius and uniform direction on the sphere. Therefore

\[U=\frac{\overline X-\mu_0}{\sqrt Q}\]

is independent of \(Q\), is symmetric about zero, and lies in \((-1/\sqrt n,1/\sqrt n)\) almost surely. Given \(Q=q\), the mean natural parameter \(\mu/\sigma^2\) exponentially tilts the sum of the centered observations. The upper-tail conditional rule has constant null size; the symmetric two-tail rule also has zero conditional first moment by symmetry. The conditional theorem therefore identifies the UMP-unbiased rules. Finally,

\[t=\frac{\sqrt{n\nu}\,U}{\sqrt{1-nU^2}},\]

which is strictly increasing in \(U\), and whose absolute value increases with \(|U|\). Thus these conditional rules are exactly the usual one- and two-sided \(t\)-tests.

Remark 6. Here is the full density calculation, using a single degrees-of-freedom symbol \(\nu\). Let \(Z\sim \mathcal{N}(0,1)\) and \(V\sim\chi^2_\nu\) be independent, with densities \((2\pi)^{-1/2}e^{-z^2/2}\) and \(v^{\nu/2-1}e^{-v/2}/[2^{\nu/2}\Gamma(\nu/2)]\), respectively. Set \(T=Z\sqrt{\nu/V}\), \(W=V\). The inverse transformation is \(Z=T\sqrt{W/\nu}\), \(V=W\), with Jacobian matrix

\[\begin{pmatrix}\sqrt{W/\nu}&T/(2\sqrt{\nu W})\\0&1\end{pmatrix}\]

and determinant \(\sqrt{W/\nu}\). Therefore

\[f_{T,W}(t,w)=\frac{w^{(\nu+1)/2-1}\exp\{-w(1+t^2/\nu)/2\}}{\sqrt{2\pi\nu}\,2^{\nu/2}\Gamma(\nu/2)},\quad w>0.\]

Using \(\int_0^\infty w^{a-1}e^{-bw}\,dw=\Gamma(a)b^{-a}\), with \(a=(\nu+1)/2\), \(b=(1+t^2/\nu)/2\), yields

\[f_T(t)=\frac{\Gamma((\nu+1)/2)}{\sqrt{\pi\nu}\,\Gamma(\nu/2)}\left(1+\frac{t^2}\nu\right)^{-(\nu+1)/2}.\]

The denominator of \(T\) is the square root of \(V/\nu\), not \(V/\nu\) itself. For a noncentral numerator \(Z+\delta\), the density is instead the exact integral \(\int_0^\infty\varphi(t\sqrt{v/\nu}-\delta)\sqrt{v/\nu}\,f_{\chi^2_\nu}(v)\,dv\), where \(\varphi\) is the standard normal density.

Remark 7. Inverting the two-sided mean test gives \(\overline X\pm t_{\nu,1-\alpha/2}S/\sqrt n\). Its expected length is

\[\frac{2t_{\nu,1-\alpha/2}\sigma}{\sqrt n}\sqrt{\frac2\nu}\frac{\Gamma((\nu+1)/2)}{\Gamma(\nu/2)},\]

since \(S=\sigma\sqrt{V/\nu}\) and the chi-square moment follows from the same gamma integral. Thus expected length is proportional to \(\sigma\) and decreases at the usual \(n^{-1/2}\) rate. For fixed nonzero mean displacement and fixed \(\sigma\), the noncentrality \(\sqrt n(\mu-\mu_0)/\sigma\) grows with sample size. In contrast, at a fixed sample size, allowing arbitrarily large \(\sigma\) makes that noncentrality approach zero, so no uniform power guarantee above \(\alpha\) follows without a variance restriction.

2. Unbiased confidence sets

With nuisance parameter \(\eta\), a usable confidence procedure remains a function of observed data and any auxiliary randomization, not of the unknown \(\eta\). Its coverage must hold for every nuisance value. For tests of the point hypotheses \(\theta=\theta'\), unbiased confidence sets satisfy

\[P_{\theta,\eta}(\theta\in S(X))\ge1-\alpha,\qquad P_{\theta,\eta}(\theta'\in S(X))\le1-\alpha\quad(\theta'\ne\theta).\]

A uniformly most accurate unbiased set minimizes every false-value coverage probability within this class. By the same inversion argument as Proposition 3, UMP-unbiased tests give such sets. Nuisance parameters often prevent unrestricted UMP procedures from existing, but their presence alone does not prove nonexistence.

For the normal variance, inversion of \(a<R/\sigma_0^2<b\) gives the UMA-unbiased interval \((R/b,R/a)\) for \(\sigma^2\); boundary choices do not affect continuous sampling probabilities.

Lemma 8. Suppose real densities are positive on a common interval and have monotone likelihood ratio in \(x\). Suppose the UMP-unbiased level-\(\alpha\) tests of \(\theta=\theta_0\) have acceptance intervals \([C_1(\theta_0),C_2(\theta_0)]\), and have power strictly greater than \(\alpha\) at every other parameter. Then both endpoints increase strictly with \(\theta\). Whenever ordinary inverses at \(x\) exist, the inverted interval is

\[C_2^{-1}(x)\le\theta\le C_1^{-1}(x).\]

Otherwise the precise definition is \(\{\theta:C_1(\theta)\le x\le C_2(\theta)\}\), with generalized endpoints and the boundary inclusion inherited from these inequalities.

Proof. Take \(\theta_0<\theta_1\), with tests \(\phi_0,\phi_1\). Strict unbiasedness gives \(E_{\theta_0}(\phi_1-\phi_0)>0\) and \(E_{\theta_1}(\phi_1-\phi_0)<0\). Hence neither acceptance interval contains the other. If the second interval were shifted to the left, \(d=\phi_1-\phi_0\) would be negative on its left excess and positive on the first interval's right excess. Choose a separating point \(x_*\), and put \(k=p_{\theta_1}(x_*)/p_{\theta_0}(x_*)\). Monotone likelihood ratio gives \(d(x)(p_{\theta_1}(x)/p_{\theta_0}(x)-k)\ge0\). Integrating implies \(E_{\theta_1}d\ge kE_{\theta_0}d\ge0\), a contradiction. Thus the second interval must shift to the right, with both endpoints strictly larger. Solving the acceptance inequalities proves the inverse formula. The optimality class here is unbiased tests; unrestricted two-sided UMP tests are not being assumed.

Permutation inference removes the normal shape assumption. For orientation, in the two-sample normal model with a common variance, the contrast \(\overline Y-\overline X\) divided by its pooled standard error has a \(t_{m+n-2}\) null law and gives the UMP-unbiased one-sided test. Orthogonally separate this contrast, the grand mean, and the within-sample residual coordinates; the conditional exponential-family argument above applies after conditioning on the grand mean and the remaining nuisance quadratic statistic. Under a common nonnormal distribution with positive finite variance and \(m/(m+n)\to\lambda\in(0,1)\), the ordinary CLT for the two means and consistency of the pooled variance instead give asymptotic normal calibration. This is an asymptotic assertion, not finite-sample exactness.

For exact inference, take independent strata \(i=1,\ldots,a\), each with \(m_i\) control and \(n_i\) treated observations, \(N_i=m_i+n_i\). Let their joint density be

\[p_\Delta(z)=\prod_{i=1}^a\left\{\prod_{j=1}^{m_i}f_i(x_{ij})\prod_{j=1}^{n_i}f_i(y_{ij}-\Delta)\right\}.\]

Thus \(\Delta>0\) shifts treatment outcomes upward. The null \(\Delta=0\) is invariant under all permutations within each stratum. Let \(G\) be this finite product group, of size \(M=\prod_iN_i!\), and let \(T(z)\) collect the ordered observations within each stratum.

If a level-\(\alpha\) test is unbiased for every \(\Delta>0\), then it has exact null rejection probability \(\alpha\) for every collection of densities. Indeed translations are continuous in \(L^1\), so the finite-product density \(p_\Delta\to p_0\) in \(L^1\) as \(\Delta\downarrow0\). For \(0\le\phi\le1\), power is therefore continuous at zero; power at least \(\alpha\) on the alternative and at most \(\alpha\) at zero force equality.

Proposition 9. For the family of all probability densities continuous almost everywhere, a bounded critical function \(\phi\) has \(E_0\phi=\alpha\) for every collection \(f_1,\ldots,f_a\) if and only if

\[\frac1M\sum_{g\in G}\phi(gz)=\alpha\quad\text{for Lebesgue-almost every }z.\]

Equivalently, \(E_0[\phi(Z)\mid T]=\alpha\) almost surely for every such null distribution. The group sum includes multiplicities; under the density model coordinates are distinct within strata almost surely.

Proof. Null invariance immediately proves sufficiency by integrating the group average. For necessity, define the bounded, separately symmetric function \(b(z)=M^{-1}\sum_g\phi(gz)-\alpha\). Its integral against \(\prod_{i,j}f_i(z_{ij})\) is zero for every choice of \(f_i\).

For one stratum of size \(N\), take \(f=(\sum_{j=1}^N t_j f_j)/(\sum_jt_j)\) with \(t_j>0\). Multiplying the zero integral by \((\sum_jt_j)^N\) gives a polynomial identically zero on an open orthant. The coefficient of \(t_1\cdots t_N\) is \(N!\int b(z)\prod_j f_j(z_j)\,dz\), because symmetry makes all permutations equal. Hence every such mixed integral vanishes. Apply this coefficient argument separately to every stratum. Taking the \(f_j\)'s to be uniform densities on bounded intervals shows that the integral of \(b\) on every bounded coordinate rectangle is zero. These densities are continuous except at their endpoints and belong to the specified family. Uniqueness of the locally finite signed measure with density \(b\), or Lebesgue differentiation on rectangles, yields \(b=0\) almost everywhere.

Finally, exchangeability makes the conditional distribution given the ordered samples uniform on the within-stratum permutations. Thus the group average is exactly the conditional expectation. This supplies the completeness argument left implicit in the archival proof.

For a fixed alternative density \(h\), conditional probabilities on a permutation set with positive total alternative mass are proportional to \(h(gz)\). Its conditional power is

\[\frac{\sum_g\phi(gz)h(gz)}{\sum_g h(gz)}.\]

To maximize overall power among tests exact for every null density, maximize this quantity separately on each set, subject to \(\sum_g\phi(gz)=\alpha M\). The finite Neyman–Pearson comparison says to assign rejection probability to the largest \(h(gz)\), randomizing at the cutoff. Permutation sets of zero alternative mass do not affect power and can use any rule satisfying the null constraint.

Remark 10. If all the ordered likelihood values differ, reject the largest \(k=\lfloor\alpha M\rfloor\) with probability one and the next with probability \(\alpha M-k\). With ties, let \(a\) be the number strictly above a boundary value and \(b\) the number equal to it; use boundary probability \((\alpha M-a)/b\). Each permutation, including repeated statistic values, retains its probability weight. The ranking generally depends on the specified alternative, so this construction by itself does not give a UMP test over all alternative densities.

Remark 11. For normal alternatives with stratum means \(\mu_i\), common variance \(\sigma^2\), and common positive treatment shift \(\Delta\), expand the squared residuals. On a permutation set, the sums of all observations and of all their squares in each stratum are fixed, as are the treatment counts. The only varying log-likelihood term is

\[\frac\Delta{\sigma^2}\sum_{i=1}^a\sum_{j=1}^{n_i}y_{ij}.\]

Thus large treatment sums are optimal against every such fixed positive normal alternative within the class exact for all null densities. One may enumerate the \(\prod_i\binom{N_i}{n_i}\) treatment assignments instead of all \(M\) coordinate permutations, since each assignment occurs the same \(\prod_i m_i!n_i!\) times. Different assignments can still produce tied sums; these must keep their multiplicities.

Lemma 12. Suppose a two-sample test has rejection probability \(\alpha\) whenever the distributions agree, and is nondecreasing in every treatment observation with controls fixed. If \(G\) is stochastically larger than \(F\), meaning \(G(t)\le F(t)\) for every real \(t\), then its power under \((F,G)\) is at least \(\alpha\). The reverse ordering gives power at most \(\alpha\).

Proof. For independent uniforms \(V_j\), generalized quantiles satisfy \(F^{-1}(V_j)\le G^{-1}(V_j)\) almost surely. Couple the controls as \(F^{-1}(V_1),\ldots,F^{-1}(V_m)\), and compare treatment draws \(F^{-1}(V_{m+j})\) with \(G^{-1}(V_{m+j})\). The test inequality holds pointwise, and taking expectations proves the result. The same proof works separately within independent strata.

For the treatment-sum permutation test, the required monotonicity includes randomization at ties. Fix the observed assignment \(A_0\). For any competing assignment \(A\) of the same stratum sizes,

\[T_{A_0}(z)-T_A(z)=\sum_{j\in A_0\setminus A}z_j-\sum_{j\in A\setminus A_0}z_j.\]

Increasing any observed treatment outcome can only increase this difference. Hence both the number \(a\) of competing statistics strictly greater than the observed one and the number \(a+b\) greater than or equal to it are nonincreasing. With an independent uniform \(V\), the randomized upper-tail rank is \(p=(a+Vb)/M\), a convex combination of these two counts divided by \(M\); it is nonincreasing under the same change. Rejecting for \(p\le\alpha\) is therefore coordinatewise nondecreasing, and its conditional rejection probability equals the boundary rule above. Lemma 12 proves unbiasedness for positive shifts, and more generally for the stated stratumwise stochastic order alternatives. Exact null calibration alone would not prove unbiasedness.