Conditioning and Conditional Expectation
Conditioning on an event of positive probability gives a ratio of probabilities. Conditioning on a continuously distributed observation cannot generally be defined by a ratio at a single observed value, because that event may have probability zero. Conditional expectation avoids this division: it is specified by the information it can depend on and the integrals it must preserve.
Remark 1. For any event \(\{X\in A\}\) of positive probability, the function
is a valid distribution function, whether or not \(X,Y\) are independent. The difficulty arises when conditioning on a single value \(X=x\) of probability zero. For real random variables, a regular conditional distribution \(K(x,\cdot)\) gives the precise replacement:
The kernel is determined only for \(P_X\)-almost every \(x\). Independence allows a version constant in \(x\); it is not needed for the integral identity.
The information available after an observation is described by a sigma-field. Rather than beginning with the joint distribution of random variables \(X\) and \(Y\) directly, we can represent our limited knowledge of the distribution of \(Y\) by specifying a \(\sigma\)-field of events \(\mathscr G \subset \mathscr F\) such that, for all \(G \in \mathscr G\), we know whether an outcome (that is realized) belongs to \(G\). To make this concrete, we need to know about subfields, and what it means for a random variable to be measurable with respect to a subfield.
Recall that a random variable \(X\) is measurable with respect to a \(\sigma\)-field \(\mathscr G \subset \mathscr F\) if \(X^{-1} (B) = \{ \omega : X(\omega) \in B \} \in \mathscr G\) for all Borel measurable \(B\). For a random variable \(X\), there is a “smallest \(\sigma\)-field” with respect to which \(X\) is measurable; formally, that is the intersection of all \(\sigma\)-fields with respect to which \(X\) is measurable. Denote this “smallest” \(\sigma\)-field using \(\sigma(X)\). The next result concerns how this information changes when a measurable function is applied to an observation.
Theorem 2. Let \(X\) be real valued, let \(g:\mathbb R\to\mathbb R\) be Borel, and put \(Y=g(X)\). Then \(\sigma(Y)\subseteq\sigma(X)\). Equality holds if and only if there is a Borel \(h:\mathbb R\to\mathbb R\) such that \(X=h(Y)\) everywhere. After adjoining all ambient probability-zero sets to both sigma-fields, the analogous statement holds modulo null sets, with the representation required almost surely.
Proof. Measurability of composition gives the inclusion. If the sigma-fields agree, the measurable-factorisation result for real random variables gives \(X=h(Y)\). Conversely, this representation gives \(\sigma(X)\subseteq\sigma(Y)\). A global Borel isomorphism is sufficient but unnecessary: when \(X\) is constant, both generated sigma-fields are trivial even for a constant \(g\).
◻
For a countable partition, observing the experiment tells us which cell contains the outcome. Conditional probabilities are therefore constant on each cell. The sigma-field formulation extends this description beyond countable partitions. Let \((\Omega, \mathscr F, \mathop{\mathrm{\mathbf{P}}})\) be the underlying probability space for this discussion. Let \(\mathscr H\) be a \(\sigma\)-field generated by a partition \(B_1, B_2, \dots\). This means an element of \(\mathscr H\) is of the form \(\bigcup_{k \in \mathbb N} B_{n_k}\), which can be a finite or infinite (countable!) union of \(B_1, B_2, \dots\). The question is, given \(\omega \in \Omega\), how do we know which set \(B_n\) contains \(\omega\)? By this construction, the answer to this question is the same as that of which element of \(\mathscr H\) contains \(\omega\); it turns out the latter question is well-defined even if \(\mathscr H \subset \mathscr F\), and when \(\mathscr H\) is not generated from a partition, as above (generally, it is not the case that \(\mathscr H\) is generated from a partition). The well-definedness comes from the following heuristic: given \(\omega \in \Omega\), we can systematically identify, for each \(H \in \mathscr H\), whether \(\omega \in H\) or \(\omega \in H^c\). Therefore, principally, we can identify \(\mathscr H\) with an experiment.
The Radon–Nikodym theorem provides the conditional probability once we specify its integrals over all observable events. Thinking of \(\mathscr H\) as an experiment, we define \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H)\) for every \(A \in \mathscr F\), as follows. Fixing an event \(A \in \mathscr F\), define a (finite!) measure on \(\mathscr H\) via the formula
This is well-defined, again, because \(\mathscr H \subset \mathscr F\). In particular, this means \(\mathop{\mathrm{\mathbf{P}}}(H) = 0\) implies \(\mathbf Q(H) = 0\). Therefore, applying the Radon-Nikodym theorem (Theorem 3), we have that there exists a function (or, random variable) \(f\) such that \(\mathop{\mathrm{\mathbf{P}}}(A \cap H) = \mathbf Q(H) = \int_H f \dif \mathop{\mathrm{\mathbf{P}}}\). Here, \(f\) is \(\mathscr H\)-measurable and integrable with respect to \(\mathop{\mathrm{\mathbf{P}}}\). Note that the Radon-Nikodym theorem does apply because \(\mathbf Q \ll \mathop{\mathrm{\mathbf{P}}}\): by a restriction argument, let \(\tilde{\mathop{\mathrm{\mathbf{P}}}}\) be the restriction of \(\mathop{\mathrm{\mathbf{P}}}\) to \(\mathscr H \subset \mathscr F\). Clearly, \(\int_H f \dif \mathop{\mathrm{\mathbf{P}}}= \int_H f \dif \tilde{\mathop{\mathrm{\mathbf{P}}}}\) for all \(H \in \mathscr H\). Take \(\dif \mathbf Q = f \dif \tilde{\mathop{\mathrm{\mathbf{P}}}}\) to conclude.
The construction determines an observable random variable through its averages over observable events. Thus \(f\) is usually denoted \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H)\), which is a random variable, satisfying the following properties:
\(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H)\) is \(\mathscr H\)-measurable and integrable.
The following functional equation is satisfied:
\[\mathop{\mathrm{\mathbf{P}}}(A \cap H) = \int_H \mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) \dif \mathop{\mathrm{\mathbf{P}}}\quad \text{for all } H \in \mathscr H ~.\]
This construction does not necessarily permit uniqueness of \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H)\); in fact, there are generally many such random variables that agree with each other, with probability one. We call one of these random variables a version of the conditional probability.
Remark 3. A set \(A \in \mathscr F\) is, by definition, independent of the \(\sigma\)-field \(\mathscr H\) if it is independent of every \(H \in \mathscr H\), in the usual sense: \(\mathop{\mathrm{\mathbf{P}}}(A \cap H) = \mathop{\mathrm{\mathbf{P}}}(A) \mathop{\mathrm{\mathbf{P}}}(H)\). The product on the right equals \(\int_H \mathop{\mathrm{\mathbf{P}}}(A) \dif \mathop{\mathrm{\mathbf{P}}}\). Therefore, \(A\) is independent of \(\mathscr H\) if and only if \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) = \mathop{\mathrm{\mathbf{P}}}(A)\) with probability one.
Proposition 4. For each fixed event in the assertion below, the following identities and bounds hold almost surely: \(\mathop{\mathrm{\mathbf{P}}}(\emptyset \mid \mathscr H) = 0\), \(\mathop{\mathrm{\mathbf{P}}}(\Omega \mid \mathscr H) = 1\), and \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) \in [0,1]\) for all \(A \in \mathscr F\). Furthermore, if \(A_1,A_2,\dots\) is a finite or infinite (countable!) sequence of disjoint sets, then
with probability one.
Proof. The constant functions zero and one have the required defining integrals for \(\varnothing\) and \(\Omega\). The Radon–Nikodym construction gives \(P(A\mid\mathscr H)\geq0\) almost surely. Also \(P(A\mid\mathscr H)+P(A^c\mid\mathscr H)\) has the same integrals as the constant one, so uniqueness gives the upper bound one.
For a fixed disjoint sequence \((A_n)\), choose nonnegative versions \(Z_n=P(A_n\mid\mathscr H)\). The sum \(Z=\sum_nZ_n\) is measurable, and monotone convergence gives, for every \(H\in\mathscr H\),
Taking \(H=\Omega\) shows \(Z\) is integrable and finite almost surely. Uniqueness makes it a version of the conditional probability of the union. The exceptional null set may depend on the fixed event or countable family. These statements alone do not give a single null set valid for every event in an uncountable sigma-field; Theorem 6 supplies compatible versions when the events arise from one real random variable. ◻
Proposition 5. For each fixed choice of the events or sequence below, almost surely:
If \(A, B \in \mathscr F\) such that \(A \subset B\), then
\[\begin{aligned} \mathop{\mathrm{\mathbf{P}}}(B \setminus A \mid \mathscr H) &= \mathop{\mathrm{\mathbf{P}}}(B \mid \mathscr H) - \mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) ~, \\ \mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) &\leq \mathop{\mathrm{\mathbf{P}}}(B \mid \mathscr H) ~. \end{aligned}\]\(\mathop{\mathrm{\mathbf{P}}}(A) = 0\) implies \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) = 0\), and \(\mathop{\mathrm{\mathbf{P}}}(A) = 1\) implies \(\mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H) = 1\).
(Inclusion-Exclusion)
\[\mathop{\mathrm{\mathbf{P}}}\left( \bigcup_{1 \leq n \leq N} A_n \, \Bigg| \, \mathscr H \right) = \sum_{1 \leq i \leq N} \mathop{\mathrm{\mathbf{P}}}(A_i \mid \mathscr H) - \sum_{1 \leq i < j \leq N} \mathop{\mathrm{\mathbf{P}}}(A_i \cap A_j \mid \mathscr H) + \cdots\]If \(A_n \uparrow A\), then \(\mathop{\mathrm{\mathbf{P}}}(A_n \mid \mathscr H) \uparrow \mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H)\).
If \(A_n \downarrow A\), then \(\mathop{\mathrm{\mathbf{P}}}(A_n \mid \mathscr H) \downarrow \mathop{\mathrm{\mathbf{P}}}(A \mid \mathscr H)\).
Proof. Apply disjoint additivity to \(B=A\sqcup(B\setminus A)\); nonnegativity gives monotonicity. If \(P(A)=0\), the nonnegative variable \(P(A\mid\mathscr H)\) has expectation zero and is therefore zero almost surely. The case \(P(A)=1\) follows by complements.
For inclusion–exclusion, the finite indicator identity holds pointwise. Replace each indicator by its conditional probability and integrate over any \(H\in\mathscr H\); the integrals still agree, so uniqueness gives the displayed identity. If \(A_n\uparrow A\), apply disjoint additivity to \(A_1,A_2\setminus A_1,A_3\setminus A_2,\ldots\). Its partial sums are versions of \(P(A_n\mid\mathscr H)\), and its full sum is a version of \(P(A\mid\mathscr H)\). Countably many almost-sure equalities can be imposed on one common probability-one event, proving the increasing-limit assertion. Apply it to complements for decreasing sequences. ◻
Theorem 6 (Existence of Conditional Probability Distribution). Let \(X\) be a random variable on \((\Omega,\mathscr F,\mathop{\mathrm{\mathbf{P}}})\), and \(\mathscr H \subset \mathscr F\) be a \(\sigma\)-field. Then, there exists a function \(\mu(B,\omega)\), defined for each \(B \in \mathscr B_{\mathbb R}\) and \(\omega \in \Omega\), such that
for each \(\omega \in \Omega\), \(\mu(\cdot,\omega)\) is a probability measure on \(\mathscr B_{\mathbb R}\); and
for each \(B \in \mathscr B_{\mathbb R}\), \(\mu(B,\cdot)\) is a version of \(\mathop{\mathrm{\mathbf{P}}}(X \in B \mid \mathscr H)\).
Proof. Choose a version \(f_q=P(X\leq q\mid\mathscr H)\) for each rational \(q\). Proposition 5 implies, almost surely, that these versions lie in \([0,1]\), are nondecreasing in \(q\), satisfy
For the last identity, use a fixed rational sequence decreasing to each \(q\) and continuity from above. There are only countably many rational comparisons and limits here. Their exceptional sets have a union \(N\in\mathscr H\) with \(P(N)=0\). On \(N\), redefine every \(f_q\) to be \(\mathbf1_{\{q\geq0\}}\). All the displayed properties now hold at every \(\omega\).
For real \(x\), set \(F_\omega(x)=\inf_{q\in\mathbb Q,\ q>x}f_q(\omega)\). This is nondecreasing, tends to zero and one at the two ends, and is right-continuous. For the last property, if \(x_j\downarrow x\), the rational sets \(\{q:q>x_j\}\) increase to \(\{q:q>x\}\), so their infima decrease to the stated infimum. Thus \(F_\omega\) is a distribution function and determines a unique probability measure \(\mu(\cdot,\omega)\). On rational arguments \(F_\omega(q)=f_q(\omega)\).
Let \(\mathcal D\) be the Borel sets \(B\) for which \(\omega\mapsto\mu(B,\omega)\) is \(\mathscr H\)-measurable and
The preceding construction puts every rational lower half-line in \(\mathcal D\). The class contains \(\mathbb R\), is closed under complements by subtracting from one, and under disjoint countable unions by pointwise countable additivity and monotone convergence. It is a Dynkin class. Rational lower half-lines form a generating pi-system, so \(\mathcal D=\mathcal B(\mathbb R)\). This proves both assertions.
In particular, conditioning a real \(Y\) on \(\sigma(X)\) yields the kernel in Remark 1. Factor each rational conditional probability as \(f_q=g_q(X)\) with Borel \(g_q\), using measurable factorization. Perform the same countable correction on a Borel set of \(P_X\)-measure zero in the \(x\)-space. The construction above then gives \(K(x,B)\), Borel in \(x\) for every Borel \(B\), with the required integral identities. Two such kernels agree outside one \(P_X\)-null set: first compare their rational half-line probabilities, then use uniqueness of a probability measure from those probabilities. ◻
1. Conditional Expectation
Conditional probability is the conditional expectation of an indicator. To extend the construction to general integrable random variables, preserve the same two requirements: measurability with respect to the available information, and equality of integrals over every event in that information. As always, let \((\Omega, \mathscr F, \mathop{\mathrm{\mathbf{P}}})\) be the underlying probability space; without further qualifications, every random variable defined below is assumed to be defined on this probability space.
Let \(X\) be an integrable real random variable, and let \(\mathscr G\subseteq\mathscr F\) be a sigma-field.
Definition 7. A random variable, denoted \(\mathbb{E}[X \mid \mathscr G]\), exists, and is called the conditional expectation of \(X\) given \(\mathscr G\). This random variable satisfies the following conditions:
\(\mathbb{E}[X \mid \mathscr G]\) is \(\mathscr G\)-measurable and integrable; and
\(\mathbb{E}[X \mid \mathscr G]\) satisfies
\[\int_G \mathbb{E}[X \mid \mathscr G] \dif \mathop{\mathrm{\mathbf{P}}}= \int_G X \dif \mathop{\mathrm{\mathbf{P}}}\quad \text{for all } G \in \mathscr G ~.\]
The existence of conditional expectation follows from the Radon-Nikodym theorem (Theorem 3). The measure to which the theorem is applied is obtained by integrating the random variable over observable events. First consider the case where \(X \geq 0\). Then, the measure \(\mathbf Q(G) = \int_G X \dif \mathop{\mathrm{\mathbf{P}}}\), defined for all \(G \in \mathscr G\), is absolutely continuous with respect to \(\mathop{\mathrm{\mathbf{P}}}\), and is finite (because \(X\) is integrable!). By the Radon-Nikodym theorem, this function (density!) \(f\) has the required properties in Definition 7. Then, in the general case where \(X\) is not necessarily nonnegative, by definition \(X^+ = \max(X,0)\) and \(X^- = -\min(X,0)\), it follows directly that \(\mathbb{E}[X^+ \mid \mathscr G] - \mathbb{E}[X^- \mid \mathscr G]\) has the desired properties in Definition 7.
Two cases make the definition concrete. For a simple random variable, conditional expectation is a linear combination of conditional probabilities. For a sigma-field generated by a partition, it is constant on each cell and equals the average over that cell.
Let \(X = \sum_{1 \leq i \leq n} a_i \mathbb 1_{A_i}\), where \(\{A_i\}_{1 \leq i \leq n}\) are events. For each \(\mathbb 1_{A_i}\), the defining properties of \(\mathbb{E}[\mathbb 1_{A_i} \mid \mathscr G]\) and conditional probability \(\mathop{\mathrm{\mathbf{P}}}(A_i \mid \mathscr G)\) coincide. This means \(\mathbb{E}[X \mid \mathscr G] = \sum_{1 \leq i \leq n} a_i \mathop{\mathrm{\mathbf{P}}}(A_i \mid \mathscr G)\).
If \(\mathscr G\) is a \(\sigma\)-field generated from \(B_1,B_2,\dots\), where \(\{B_1,B_2,\dots\}\) is a finite or infinite (countable!) partition of \(\Omega\), then it follows that \(\mathbb{E}[X \mid \mathscr G]\) has a constant value over each \(B_i\) (note that \(\mathbb{E}[X \mid \mathscr G]\) is, by definition, \(\mathscr G\)-measurable). Call this constant value \(b_i\). The second condition in Definition 7 then implies
\[(\mathbb{E}[X \mid \mathscr G])_\omega = (\mathop{\mathrm{\mathbf{P}}}(B_i))^{-1} \int_{B_i} X \dif \mathop{\mathrm{\mathbf{P}}}\quad \text{for } \omega \in B_i \text{ and } \mathop{\mathrm{\mathbf{P}}}(B_i) > 0 ~.\]If \(\mathop{\mathrm{\mathbf{P}}}(B_i)=0\), then \(\mathbb{E}[X \mid \mathscr G]\) takes some arbitrary constant.
Proposition 8 (Properties of Conditional Expectation). Let \(X\), \(X_n\), and \(Y\) be integrable random variables, and \(a,b\) be constants.
If \(X = a\) with probability one, then \(\mathbb{E}[X \mid \mathscr G] = a\).
If \(X \leq Y\) with probability one, then \(\mathbb{E}[X \mid \mathscr G] \leq \mathbb{E}[Y \mid \mathscr G]\).
(Linearity) \(\mathbb{E}[aX + bY \mid \mathscr G] = a \mathbb{E}[X \mid \mathscr G] + b \mathbb{E}[Y \mid \mathscr G]\).
\(|\mathbb{E}[X \mid \mathscr G]| \leq \mathbb{E}[|X| \mid \mathscr G]\).
If \(X_n \xrightarrow{n\to\infty} X\) with probability one, and \(|X_n| \leq Y\) for all \(n \in \mathbb N\), then \(\mathbb{E}[X_n \mid \mathscr G] \xrightarrow{n\to\infty} \mathbb{E}[X \mid \mathscr G]\) with probability one.
Proof.
First note uniqueness: if integrable \(\mathscr G\)-measurable \(U,V\) have the same integrals over every \(G\in\mathscr G\), integrate \(U-V\) on \(\{U>V\}\) and \(V-U\) on \(\{V>U\}\). Both nonnegative integrals vanish, so \(U=V\) almost surely. The constant \(a\) itself satisfies the defining identity in (i). For (iii), the proposed linear combination is measurable and integrable, and integrating it over any \(G\) gives \(\int_G(aX+bY)\,dP\); uniqueness proves linearity.
For (ii), the Radon–Nikodym construction gives a nonnegative version of \(\mathbb E(Y-X\mid\mathscr G)\) when \(Y-X\geq0\). Apply linearity to obtain the order inequality. Since \(-|X|\leq X\leq|X|\), applying that inequality twice proves (iv). Taking \(G=\Omega\) in the defining identity also gives \(\mathbb E\mathbb E(X\mid\mathscr G)=\mathbb EX\). These facts imply the useful contraction
For the dominated-convergence assertion, let \(Z_m=\sup_{n\geq m}|X_n-X|\). Then \(Z_m\downarrow0\) almost surely and \(Z_m\leq2Y\). The variables \(\mathbb{E}[Z_m\mid\mathscr G]\) decrease almost surely; their nonnegative limit has expectation zero because \(\mathbb{E}Z_m\to0\) by ordinary dominated convergence. Thus their limit is zero, and \(|\mathbb{E}[X_n-X\mid\mathscr G]|\leq \mathbb{E}[Z_m\mid\mathscr G]\) for \(n\geq m\) proves the claim.
◻
Proposition 9. If \(X\) is \(\mathscr G\)-measurable, and \(Y\), \(XY\) are integrable, then \(\mathbb{E}[XY \mid \mathscr G] = X \mathbb{E}[Y \mid \mathscr G]\) with probability one.
Proof. If \(X=\mathbf1_B\) with \(B\in\mathscr G\), then for every \(G\in\mathscr G\),
The proposed product is measurable and integrable, so uniqueness proves the assertion for indicators, and linearity proves it for simple \(X\). Uniform approximation by simple functions, together with the \(L^1\) contraction above, proves it for bounded measurable \(X\).
For general real-valued measurable \(X\), the bounded case and monotone convergence give
Thus \(X\mathbb E(Y\mid\mathscr G)\) is integrable. Set \(X_k=(-k)\vee(X\wedge k)\). Dominated convergence gives \(X_kY\to XY\) in \(L^1\), and also \(X_k\mathbb E(Y\mid\mathscr G)\to X\mathbb E(Y\mid\mathscr G)\) in \(L^1\), using the preceding integrable bound. The contraction gives convergence in \(L^1\) of \(\mathbb E(X_kY\mid\mathscr G)\) to \(\mathbb E(XY\mid\mathscr G)\). Pass to the limit in the bounded-case identity. ◻
Proposition 10. Let \(X\) be an integrable random variable, and \(\mathscr G_1\), \(\mathscr G_2\) are \(\sigma\)-fields satisfying \(\mathscr G_1 \subset \mathscr G_2\). Then, \(\mathbb{E}[\mathbb{E}[X \mid \mathscr G_2] \mid \mathscr G_1] = \mathbb{E}[X \mid \mathscr G_1]\).
Proof. The left-hand side is integrable and \(\mathscr G_1\)-measurable. For every \(G\in\mathscr G_1\subseteq\mathscr G_2\), its defining integral is
It therefore satisfies both defining properties of \(\mathbb E(X\mid\mathscr G_1)\), and uniqueness proves the equality almost surely. ◻