Log-Sobolev Inequality
Entropy tensorization reduces a concentration problem to an inequality in one coordinate. For Gaussian measure, the needed estimate bounds entropy by the squared gradient. This post develops the geometric inequalities used in that argument, then derives the Gaussian log-Sobolev inequality and explains where the gradient enters.
Let \(X_1,\ldots,X_n\) be i.i.d. with law \(P\). For a function \(f\), write \(Pf=\mathbb{E} f(X_1)\), \(P_nf=n^{-1}\sum_{i=1}^n f(X_i)\), and \(\mathbb G_nf=\sqrt n(P_n-P)f\). Rademacher variables are independent random signs, independent of the observations. See the empirical-process setup for measurability conventions.
The proof begins with addition of sets and an integral inequality for log-concave functions. These facts allow us to compare a function with a small perturbation of itself under Gaussian measure. Expanding that comparison gives an entropy bound.
To begin, let \(\lambda\) denote the Lebesgue measure on \(\mathbb R^n\) for any \(n\). Note the following result follows from properties of \(\lambda\).
Lemma 1. Suppose \(A\) and \(B\) are nonempty Borel measurable sets in \(\mathbb R\). Then, \(\lambda(A + B) \geq \lambda(A) + \lambda(B)\), where \(A + B = \{a + b : a \in A ~,~ b \in B\}\).
Proof. A continuous image of a Borel set need not be Borel. Here \(A+B\) is analytic, hence measurable for completed Lebesgue measure. We use the theorem that analytic subsets of Polish spaces are universally measurable; see Tserunyan, Theorem 7.6. For nonempty compact sets, translate separately so that \(\max A=0=\min B\). Both sets then contain zero, so \(A\cup B\subset A+B\), and their intersection has measure zero. Translation invariance gives the inequality. For general Borel sets, choose nonempty compact subsets with measures approaching the respective measures (or arbitrarily large measures in the infinite case), and use monotonicity. Nonemptiness is necessary: adding the empty set always gives the empty set. ◻
The following result is the main ingredient to results on volume comparisons.
Theorem 2 (Précopa-Leindler). Let \(f\), \(g\), and \(\varphi\) be Lebesgue measurable functions on \(\mathbb R^n\) taking values in \([0,\infty]\). For some \(\eta \in (0,1)\) and for all \(u, v \in \mathbb R^n\), suppose
holds. Then,
Proof. We use the convention \(0\cdot\infty=0\) for the nonnegative products. Measurability can first be reduced to the Borel case: replace \(f,g\) by Borel minorants agreeing almost everywhere and \(\varphi\) by a Borel majorant agreeing almost everywhere. This preserves the pointwise hypothesis and all three integrals.
In dimension one, first suppose \(f,g\) are bounded and not identically zero. Set \(M=\sup f\), \(N=\sup g\), \(\tilde f=f/M\), \(\tilde g=g/N\), and \(\tilde\varphi=\varphi/(M^\eta N^{1-\eta})\). The transformed functions still satisfy (1), and both suprema are one. For every \(0<t<1\), the sets \(A_t=\{\tilde f>t\}\) and \(B_t=\{\tilde g>t\}\) are nonempty, and
Lemma 1, including the scaling of Lebesgue measure, and the layer-cake formula therefore give
Multiplying by \(M^\eta N^{1-\eta}\) proves the desired inequality. If either integral is zero, the conclusion is immediate. For unbounded functions apply this argument to \(f\wedge m\) and \(g\wedge m\), keeping \(\varphi\) unchanged, and let \(m\to\infty\). Truncation to bounded intervals also handles infinite integrals without an indeterminate normalization.
For the induction step write points as \((u,a)\in\mathbb R^{n-1}\times\mathbb R\), and define \(F(a)=\int f(u,a)du\), \(G(b)=\int g(v,b)dv\), and \(H(c)=\int\varphi(w,c)dw\). For each \(a,b\), the induction hypothesis applied to the slices gives
The slice integrals are measurable by Tonelli. The one-dimensional result applied to \(F,G,H\), followed by Tonelli again, proves the claim in dimension \(n\). ◻
Précopa–Leindler implies log-concavity of Gaussian measure on \(\mathbb R^n\): a property we will use to compare the mass of translated convex sets.
Proposition 3 (Gaussian measures on \(\mathbb R^n\) are log-concave). Let \(\mu\) be a centred Gaussian measure on \(\mathbb R^n\). Then, for any nonempty Borel sets \(A, B \subset \mathbb R^n\) and \(\eta \in [0,1]\), we have
Proof. Interpret \(\mu\) on its completion; the linear image of \(A\times B\) is analytic and therefore measurable for this measure. The endpoint values \(\eta=0,1\) are immediate. If \(\mu\) is a point mass at zero, the assertion follows directly. Otherwise let \(V\) be its supporting subspace. On \(V\), with its own Euclidean Lebesgue measure, \(\mu\) has density \(\psi(x)=c\exp(-\|\Gamma x\|^2/2)\), where \(\Gamma\) is the inverse square root of the covariance restricted to \(V\). Thus \(\log\psi=\log c-\|\Gamma x\|^2/2\) is concave.
First take compact \(K\subset A\cap V\) and \(L\subset B\cap V\). Apply Theorem 2 within \(V\) to \(f=\psi\mathbf1_K\), \(g=\psi\mathbf1_L\), and \(\varphi=\psi\mathbf1_{\eta K+(1-\eta)L}\). The hypothesis follows from log-concavity of \(\psi\), and hence
Inner regularity on \(V\) lets the last two measures approach \(\mu(A)\) and \(\mu(B)\). If either is zero the inequality is immediate. This also avoids identifying intrinsic Lebesgue measure on a lower-dimensional subspace with the restriction of ambient Lebesgue measure. ◻
Lemma 4 (Anderson). Suppose \(X = (g_1, \dots, g_n)\) is a centred, jointly normal vector in \(\mathbb R^n\). Let \(C\) be a Borel measurable, convex, symmetric set in \(\mathbb R^n\). Then, for all \(x \in \mathbb R^n\),
Proof. If \(C\) is empty there is nothing to prove. Otherwise put \(A=C+x\) and \(B=C-x\). Symmetry of both \(C\) and the centered Gaussian law \(\mu\) gives \(\mu(A)=\mu(B)\), while convexity gives \((A+B)/2=(C+C)/2=C\). Gaussian log-concavity at \(\eta=1/2\) therefore yields \(\mu(C)\geq\sqrt{\mu(A)\mu(B)}=\mu(C-x)=\mathbb P(X+x\in C)\). ◻
There are two useful direct consequences of all the constructions above.
Theorem 5 (Brunn-Minkowski). Let \(\operatorname{Vol}\) denote the volume on \(\mathbb R^n\). Then, for any nonempty compact subsets \(A\) and \(B\) of \(\mathbb R^n\), we have
Proof. If \(a=\operatorname{Vol}(A)^{1/n}\) and \(b=\operatorname{Vol}(B)^{1/n}\) are positive, the sets \(A/a\) and \(B/b\) have unit volume. Apply Theorem 2 to their indicator functions with \(\eta=a/(a+b)\):
The set inside the volume is \((A+B)/(a+b)\), so rescaling proves the result. If one volume is zero, a translate of the other nonempty set is contained in \(A+B\), which gives the same conclusion. ◻
Lemma 6 (Khatri-Sidak). Let \(n \geq 2\) and \(g_1, \dots, g_n\) be centred, jointly normal random variables. Then, for all \(x \geq 0\), we have
More generally,
Proof. If \(\operatorname{Var}(g_1)=0\), then \(g_1=0\) almost surely and the first inequality is an equality. Otherwise Gaussian regression writes \((g_2,\ldots,g_n)=a g_1+Z\), where \(Z\) is a centered Gaussian vector independent of \(g_1\). For a symmetric convex Borel set \(C\), put \(h(u)=P(Z+au\in C)\). Gaussian log-concavity shows that \(h\) is log-concave, and symmetry makes it even. Thus \(h\) is nonincreasing on \([0,\infty)\): if \(0\le u\le v\), write \(u\) as a convex combination of \(v\) and \(-v\).
Let \(\psi(t)=P(|g_1|\le t)\) and \(\phi(t)=E[h(g_1)\mathbf1\{|g_1|\le t\}]\). For \(t>0\), \(h(t)\le\phi(t)/\psi(t)\), because the latter is an average of values at \(|u|\le t\). If \(f_1\) is the density of \(g_1\), differentiation almost everywhere gives
The ratio is absolutely continuous on intervals bounded away from zero. Its limit as \(t\to\infty\) is \(P((g_2,\ldots,g_n)\in C)\). Taking \(C=[-x,x]^{n-1}\) proves the first inequality for \(x>0\); continuity from above gives \(x=0\). Iteration proves the product bound. ◻
The discussions above give us the ability to derive log-Sobolev inequality for Gaussian measures on \(\mathbb R^n\). To do so, we make the notational change to how we discuss entropy of a nonnegative, real-valued function (before, \(\operatorname{Ent}(X)\) is a specific way of writing what follows): for any measurable, real-valued, nonnegative function on some measurable space, the entropy of \(f\) with respect to a probability measure \(\mu\) (defined on this measurable space) is written as \(\operatorname{Ent}_\mu (f) = \int f \log f \dif \mu - \left( \int f \dif \mu \right) \left( \log \int f \dif \mu \right)\). The tensorization and variational formulas apply with this notation as well.
For Lemmas 7–9, take a nonnegative measurable \(f\) with finite mean and finite \(\int f\log^+f\,d\mu\) (with the product measure in Lemma 8). We use \(0\log0=0\). In Lemma 7 the supremum can be taken over bounded measurable \(g\); allowing unbounded \(g\) for which the integral is defined gives the same value.
Lemma 7.
Proof. If \(m=\int f\,d\mu=0\), then \(f=0\) almost everywhere and both sides vanish. Otherwise put \(q=f/m\), a probability density. For any admissible \(g\), Jensen under \(q\,d\mu\) gives
Thus \(\int fg\,d\mu\leq m\int q\log q\,d\mu=\operatorname{Ent}_\mu(f)\). For the reverse bound, let \(h_N=\max(-N,\min(\log q,N))\), taking \(\log0=-\infty\), and normalize \(g_N=h_N-\log\int e^{h_N}d\mu\). Then \(\int e^{g_N}d\mu=1\). Since \(e^{h_N}\leq q+1\) and it converges to \(q\), its integral tends to one. Also \(q h_N\to q\log q\) in \(L^1\): the positive part is controlled by \(q\log^+q\), and the negative part by the integrable bound \(1/e\). Consequently \(\int fg_N\to\operatorname{Ent}_\mu(f)\). The upper bound for unbounded admissible \(g\) follows by the same Jensen argument and truncation; \(qg^+\) is integrable since \(qg^+\leq q\log q-q+e^{g^+}\) and \(e^{g^+}\leq1+e^g\). ◻
Lemma 8. Let \(\mu_1, \dots, \mu_n\) be probability measures on some measurable space, \(P = \mu_1 \times \cdots \times \mu_n\), and \(f\) be a measurable, real-valued, and nonnegative function of \(n\) variables. With the understanding that \(\operatorname{Ent}_{\mu_i} (f)\) denotes the entropy of \(f\) computed with respect to the \(i\)th variable (and \(\mu_i\)), we have
Proof. The two-coordinate identity is
It follows by adding and subtracting the entropy of the coordinate mean. Convexity of the entropy functional, proved using the joint convexity of \((u,m)\mapsto u\log(u/m)\) in the preceding entropy note, bounds the last term by \(\int\operatorname{Ent}_\nu(f(x,\cdot))d\mu(x)\). This proves the two-coordinate inequality. Inductively separate the last coordinate from the first \(n-1\), apply the same identity and convexity, then apply the induction hypothesis to each fixed last-coordinate slice. Fubini is justified by the stated logarithmic integrability and the bound \(u\log u\geq-1/e\). Summing gives exactly the displayed \(n\)-coordinate bound. ◻
Lemma 9.
Proof. Put \(m=\int f\,d\mu\). The expression being minimized is \(\int f\log f\,d\mu-m(1+\log t)+t\). If \(m>0\), its derivative in \(t>0\) is \(1-m/t\), negative below \(m\) and positive above it. The minimum is therefore attained at \(t=m\), where the expression equals \(\operatorname{Ent}_\mu(f)\). If \(m=0\), then \(f=0\) almost everywhere and the expression is simply \(t\), whose infimum is zero as \(t\downarrow0\). This handles the boundary case without writing \(\log0\). ◻
Theorem 10 (Log-Sobolev inequality for Gaussian measures on \(\mathbb R^n\)). Let \(\mu\) be the standard Gaussian measure on \(\mathbb R^n\). Let \(f : \mathbb R^n \to \mathbb R\) be a \(\mathscr C^2\) function such that \(f^2\) and \(|\nabla f|^2\) are \(\mu\)-integrable, where \(\nabla f = (\frac{\partial f}{\partial x_1}, \dots, \frac{\partial f}{\partial x_n})\). Then,
Proof. Tensorization allows us to sum the one-coordinate gradient bounds. It is therefore sufficient to prove the statement for \(n=1\), i.e. on the real-line. Since \(f\) is twice continuous differentiable, we first prove the inequality when \(f^2 = e^g\), where \(g\) is twice continuously differentiable with compact support; the approximation needed for general functions is given below. Therefore, if we set \(V(x) = \frac{x^2 + \ln 2\pi}{2}\), then the standard normal density becomes \(e^{-V(x)}\). So, for \(\lambda \in (0,1)\), set
Exponentiating, set \(\phi(z) = e^{g_\lambda (z) - V(z)}\), so
Therefore, \(\phi\), \(e^{\frac{g}{\lambda} - V}\), and \(e^{-V}\) satisfy the assumption of Précopa-Leindler, meaning we can apply the theorem to allow us to conclude
Write \(A(\lambda)=\int e^{g/\lambda}d\mu\) and \(H(\lambda)=A(\lambda)^\lambda\). Boundedness of \(g\) permits differentiation under the integral near one. Logarithmic differentiation gives
Taylor expanding about \(\lambda = 1\) gives
which is valid as \(\lambda \to 1\). To extract the gradient term from this comparison, it remains to bound \(\int e^{g_\lambda} \dif \mu\). To do so, note that if we use the definition of \(V\) and make changes of variables \(h = z-v\) and \(\eta = \frac{1-\lambda}{\lambda}\), we have
and, because \(g''\) is compactly supported and continuous, we have, for some constant \(d\) independent of \(z\),
For \(\eta<1/(2d)\) (with no restriction if \(d=0\)), maximize the quadratic in \(h\) to obtain the explicit bound
The lower bound uses \(h=0\). Since \(g'\) is bounded, the last expression equals \(g(z)+\eta(g'(z))^2/2+O(\eta^2)\), uniformly in \(z\). The supremum is measurable: continuity in \(h\) lets us take it over rational \(h\).
Exponentiating and using the uniform bound gives
Combine (2), (3), and the integrated upper bound. Subtract \(\int e^g d\mu\), divide by \(1-\lambda>0\), and let \(\lambda\uparrow1\). Since \(\eta/(1-\lambda)=1/\lambda\to1\) and the uniform quadratic remainder divided by \(1-\lambda\) vanishes,
For \(f=e^{g/2}\), the chain rule gives \(4(f')^2=(g')^2e^g\), proving the one-dimensional bound. Applying it to each coordinate slice and summing by Lemma 8 proves the corresponding bound in any finite dimension. The approximation below removes positivity and compact-support restrictions. ◻
For completeness, the approximation also covers zeros and signs of \(f\). The argument for \(e^g\) works when \(g\) is constant outside a compact set, since adding a constant only scales entropy and the gradient integral. For a smooth compactly supported function \(u\), apply it to \(u^2+\varepsilon\). Its entropy is at most
Let \(\varepsilon\downarrow0\); boundedness gives convergence of entropy. Finally take \(u_R=\chi_R f\), where smooth cutoffs satisfy \(\chi_R=1\) on the ball of radius \(R\), vanish outside the ball of radius \(2R\), and \(\|\nabla\chi_R\|\le C/R\). The assumed integrability gives \(u_R\to f\) and \(\nabla u_R\to\nabla f\) in Gaussian \(L^2\). Fatou's lemma applied to \(u_R^2\log u_R^2+1/e\), together with convergence of \(\int u_R^2d\mu\), gives lower semicontinuity of entropy and proves Theorem 10 in full.