Probability Spaces, Independence, and Countable Constructions
An infinite sequence of experiments requires more than a list of finite-dimensional probabilities. We must specify which events are measurable, extend probabilities to those events, and understand which properties of independence survive countable constructions. Cylinder sets provide the starting point. The extension theorem, Borel–Cantelli lemmas, and zero-one law then address these three questions.
Let \((\Omega,\mathscr F,\mathop{\mathrm{\mathbf{P}}})\) denote a probability space.
Definition 1. A class of subsets of \(\Omega\), denoted \(\mathcal F\), is called a field (in the algebraic sense) if (i) \(\Omega \in \mathcal F\), (ii) \(A \in \mathcal F \Rightarrow A^c \in \mathcal F\), and (iii) \(A,B \in \mathcal F \Rightarrow A \cup B \in \mathcal F\). A \(\sigma\)-field is a field with the additional requirement that (iv) if \(A_i \in \mathcal F\) for all \(i \in \mathbb N\), then \(\bigcup_{i \in \mathbb N} A_i \in \mathcal F\).
1. Denumerable Probabilities
Let \(S\) be a finite set (say, consisting of possible outcomes of an experiment). Let \(\Omega = S^\infty\) be the space of infinite sequences of elements of \(S\), i.e. \(\omega \in \Omega\) takes the form
Here, \(z_k (\omega) \in S\) for all \(\omega \in S^\infty\) and \(k \geq 1\). One can think of this as repeating a simple experiment, represented by \(S\), infinitely often.
Some nomenclature from geometry carries over to this setting. Each \(z_k : S^\infty \to S\) is called a coordinate function or canonical projection. Letting \(S^n \triangleq \underbrace{S \times \dots \times S}_{n \text{ times}}\), we write \((s_1,\dots,s_n) \in S^n\) to denote an element of \(S^n\). We can write realizations of a \(n\)-sequence as
where we can think of this as the event where first \(n\) repetitions of the experiment are given by the outcomes \(s_1,\dots,s_n\). Accordingly, for \(H \subset S^n\), we define a cylinder of rank \(n\) to be
Clearly, if we let \(\mathcal C_0\) denote the class of cylinders of all ranks, then \(\mathcal C_0\) is a field.
In what follows, we adopt the standard notation: for a collection of sets \(A_n\) and another set \(A\), we write \(A_n \uparrow A\) to mean \(A_1 \subset A_2 \subset \dots\), and \(A = \bigcup_{i \in \mathbb N} A_i\). Furthermore, we write \(A_n \downarrow A\) to mean \(A_1 \supset A_2 \supset \dots\), and \(A = \bigcap_{i \in \mathbb N} A_i\).
Lemma 2. Let \(A_n\) denote nonempty cylinders. If \(A_n \downarrow A\), then \(A\) is nonempty.
Proof. Choose \(\omega^{(n)}\in A_n\). Because \(S\) is finite, infinitely many of these sequences have the same first coordinate. Among that subsequence, infinitely many have the same second coordinate; continue recursively. The diagonal subsequence \(\omega^{(n_j)}\), with increasing \(n_j\), eventually has each of its first \(r\) coordinates fixed, for every \(r\). Let \(\omega\in S^\infty\) have those fixed coordinates.
Fix \(m\), and let \(r_m\) be the rank of the cylinder \(A_m\). For sufficiently large \(j\), both \(n_j\geq m\) and the first \(r_m\) coordinates of \(\omega^{(n_j)}\) equal those of \(\omega\). Since \(\omega^{(n_j)}\in A_{n_j}\subseteq A_m\), and membership in \(A_m\) depends only on those coordinates, \(\omega\in A_m\). This holds for every \(m\), so \(\omega\in\bigcap_mA_m\). Finiteness of \(S\) is the compactness input in this elementary version of the product-compactness argument. ◻
Theorem 3. Every finitely additive probability measure on \(\mathcal C_0\) (field of cylinders in \(S^\infty\)) is countably additive.
Proof. Let disjoint cylinders \(B_1,B_2,\ldots\) have union \(B\in\mathcal C_0\). The remainders \(R_n=B\setminus\bigcup_{j=1}^nB_j\) are cylinders decreasing to the empty set. Lemma 2 implies that one of them, say \(R_N\), must already be empty. Disjointness then forces \(B_j=\varnothing\) for \(j>N\), so finite additivity gives \(P(B)=\sum_{j=1}^NP(B_j)=\sum_{j\geq1}P(B_j)\). This is precisely countable additivity on the field; it does not assert that an arbitrary countable union of cylinders is a cylinder. ◻
Specifying probabilities on events involving finitely many coordinates is only the first step. We also need probabilities for countable combinations of those events. The extension theorem gives a unique measure on the generated sigma-field.
Theorem 4. A nonnegative, countably additive set function on a field, with total mass one, admits a unique probability extension to the generated \(\sigma\)-field.
The extension is constructed by covering an arbitrary set with measurable sets whose probabilities are already known. Taking the cheapest such cover gives an outer measure; the measurability criterion identifies the sets on which this outer measure is additive.
Let \(\mathop{\mathrm{\mathbf{P}}}\) denote a probability measure on a field \(\mathcal F_0\), and let \(\sigma(\mathcal F_0)\) denote the generated \(\sigma\)-field.
For each subset \(A\) of \(\Omega\), define the outer measure as
\[\mathop{\mathrm{\mathbf{P}}}^*(A) = \inf_{A_1,A_2,\dots} \sum_{n \in \mathbb N} \mathop{\mathrm{\mathbf{P}}}(A_n) ~,\]where the infimum is taken over all (in)finite sequences of \(\mathcal F_0\)-sets satisfying \(A \subset \bigcup_{n \in \mathbb N} A_n\). The infimum permits us to improve the bound by choosing more efficient covers. Overlap between the covering sets can cause overcounting, so covers of \(A\) with little overlap are often useful.
Define the inner measure as
\[\mathop{\mathrm{\mathbf{P}}}_* (A) = 1 - \mathop{\mathrm{\mathbf{P}}}^*(A^c) ~.\]For a probability measure, complementation turns outer approximation into inner approximation. If \(A\) is measurable, then \(P_*(A)=P(A)=P^*(A)\). For instance, the rational numbers in the unit interval have both inner and outer measure zero: enumerate them and cover the \(j\)th rational by an interval of length less than \(\varepsilon 2^{-j}\). Thus inner and outer measure can differ only for a nonmeasurable set; the set of normal numbers is not such an example.
The Carathéodory criterion tests whether splitting any set along \(A\) preserves its outer measure. It requires the following identity for every subset \(E\subseteq\Omega\):
If this relation holds, we say the set \(A\) is \(\mathop{\mathrm{\mathbf{P}}}^*\)-measurable. The set function \(\mathop{\mathrm{\mathbf{P}}}^*\) requires the following four properties:
\(\mathop{\mathrm{\mathbf{P}}}^*(\emptyset)=0\);
\(\mathop{\mathrm{\mathbf{P}}}^*\) is nonnegative;
\(\mathop{\mathrm{\mathbf{P}}}^*\) is monotone;
\(\mathop{\mathrm{\mathbf{P}}}^*\) is countably subadditive.
Now, let \(\mathscr M\) denote the collection of \(\mathop{\mathrm{\mathbf{P}}}^*\)-measurable sets.
Lemma 5.
The class \(\mathscr M\) is a field.
If \(A_1,A_2,A_3,\dots\) is a(n) (in)finite sequence of disjoint sets in \(\mathscr M\), then, for each \(E \subset \Omega\),
\[\mathop{\mathrm{\mathbf{P}}}^*\left( E \cap \left( \bigcup_{n \in \mathbb N} A_n \right) \right) = \sum_{n \in \mathbb N} \mathop{\mathrm{\mathbf{P}}}^*\left( E \cap A_n \right) ~.\]\(\mathscr M\) is a \(\sigma\)-field, and \(\mathop{\mathrm{\mathbf{P}}}^*\) restricted to \(\mathscr M\) is countably additive.
For the outer measure \(\mathop{\mathrm{\mathbf{P}}}^*\), \(\mathcal F_0 \subset \mathscr M\), and \(\mathop{\mathrm{\mathbf{P}}}^*(A)=\mathop{\mathrm{\mathbf{P}}}(A)\) for \(A \in \mathcal F_0\).
Proof. First verify that the covering construction is an outer measure. The empty cover gives \(P^*(\varnothing)=0\), and nonnegativity and monotonicity follow directly. For sets \(E_j\), choose a countable field-set cover of each whose cost is below \(P^*(E_j)+\varepsilon2^{-j}\). Combining the covers gives
Let \(\varepsilon\downarrow0\). All outer measures here are at most one, since \(\Omega\) is an available cover.
The empty set satisfies the Carathéodory identity, and the identity is unchanged on replacing a set by its complement. If \(A,B\in\mathscr M\), first split an arbitrary \(E\) along \(A\), and then split both pieces along \(B\). The resulting four-piece identity and subadditivity imply
The reverse inequality is always true by subadditivity, so \(A\cup B\in\mathscr M\). Thus \(\mathscr M\) is a field.
For pairwise disjoint \(A_j\in\mathscr M\), put \(U=\bigcup_{j\geq1}A_j\). Repeated splitting gives, for every \(N\),
Let \(N\to\infty\). Subadditivity on \(E\cap U\) shows that the right-hand side is at least \(P^*(E\cap U)+P^*(E\setminus U)\). This proves \(U\in\mathscr M\). Setting \(E\) in the same finite-splitting formula equal to \(E\cap U\) also gives
Pass to the supremum over \(N\) and use subadditivity for the reverse inequality, obtaining assertion (ii). Arbitrary countable unions can be made disjoint by taking successive differences, so \(\mathscr M\) is a sigma-field. Taking \(E=\Omega\) in (ii) gives countable additivity on it.
Finally let \(A\in\mathcal F_0\). For any field-set cover \(E\subseteq\bigcup_nC_n\), finite additivity gives
Taking the infimum over covers shows that \(A\in\mathscr M\). The single-set cover gives \(P^*(A)\leq P(A)\). Conversely, if \(A\subseteq\bigcup_n C_n\), the disjoint field sets \(D_n=(A\cap C_n)\setminus\bigcup_{j<n}C_j\) have union \(A\). Countable additivity of the original set function gives
Taking the infimum proves \(P^*(A)=P(A)\). This last step explains why countable additivity on the original field is necessary. ◻
Proof of Theorem 4. Lemma 5 gives \(\mathcal F_0\subseteq\mathscr M\), hence \(\sigma(\mathcal F_0)\subseteq\mathscr M\), and restriction of \(P^*\) to this generated sigma-field is the desired extension. For uniqueness, if \(P_1,P_2\) are probability extensions, the class \(\{A:P_1(A)=P_2(A)\}\) contains \(\Omega\), is closed under complements, and is closed under countable disjoint unions. It is therefore a Dynkin class containing the pi-system \(\mathcal F_0\). The pi–lambda theorem implies that it contains \(\sigma(\mathcal F_0)\), proving uniqueness. ◻
The Borel–Cantelli lemmas answer a different question about countably many events: do infinitely many of them occur? A finite sum of probabilities rules this out without any independence assumption. For independent events, a divergent sum forces infinitely many occurrences.
Lemma 6 (Borel-Cantelli).
If \(\sum_{n \in \mathbb N} \mathop{\mathrm{\mathbf{P}}}(A_n) < \infty\), then \(\mathop{\mathrm{\mathbf{P}}}(\limsup_{n\to\infty} A_n) = 0\).
If \(\{A_n\}_{n \in \mathbb N}\) is a sequence of independent events, and \(\sum_{n \in \mathbb N} \mathop{\mathrm{\mathbf{P}}}(A_n)\) diverges, then \(\mathop{\mathrm{\mathbf{P}}}(\limsup_{n\to\infty} A_n) = 1\).
Proof. For the first assertion, Tonelli’s theorem gives \(\mathbb{E}\sum_n1_{A_n}=\sum_nP(A_n)<\infty\). Hence only finitely many events occur almost surely.
For the second, fix \(M\). Independence gives
Continuity from above therefore gives \(P(\bigcap_{k=M}^{\infty}A_k^c)=0\). Thus \(P(\bigcup_{k=M}^{\infty}A_k)=1\) for each \(M\). Intersecting these countably many probability-one events proves \(P(\limsup_n A_n)=1\).
◻
Proposition 7. Put \(X=(X_1,\ldots,X_n):\Omega\to\mathbb R^n\). Then \(\sigma(X)=\{X^{-1}(B):B\in\mathcal B(\mathbb R^n)\}\). A real random variable \(Y\) is \(\sigma(X)\)-measurable if and only if \(Y=f(X)\) for a Borel function \(f:\mathbb R^n\to\mathbb R\).
Proof. The displayed collection is a sigma-field and is exactly the one generated by the coordinates of \(X\). If \(Y\) is measurable for this sigma-field, choose finite-valued measurable \(Y_m\to Y\). Each level set of \(Y_m\) is an inverse image under \(X\) of a Borel set. Making these Borel sets disjoint produces a Borel \(f_m\) with \(Y_m=f_m(X)\). Define \(f(x)=\lim_m f_m(x)\) where the limit exists and is finite, and set it to zero elsewhere. This function is Borel and \(Y=f(X)\). The reverse implication follows from measurability of compositions.
◻
A tail event is unaffected by discarding any finite initial segment. This makes it a natural object when studying eventual behaviour of an infinite sequence. Formally, let a sequence of events be denoted as \(A_1,A_2,A_3,\dots\) (with respect to a probability space \((\Omega, \mathscr F, \mathop{\mathrm{\mathbf{P}}})\)). Define
We call (1) the tail \(\sigma\)-field associated with \(\{A_n,A_{n+1},\dots\}\). The elements of the tail \(\sigma\)-field are called tail events. For independent events, Kolmogorov’s zero-one law shows that every tail event has probability zero or one.
Theorem 8 (Zero-One Law). Let \(A_1, A_2, \dots\) be a sequence of independent events. Then, for each event \(A\) in the tail \(\sigma\)-field, \(\mathop{\mathrm{\mathbf{P}}}(A)\) is either zero or one.
Proof. Fix a tail event \(A\), and put \(\mathscr G_n=\sigma(A_1,\ldots,A_n)\) and \(\mathscr G_\infty=\sigma(A_1,A_2,\ldots)\). Independence of finite collections, followed by the pi-system extension proved in Theorem 10, shows that \(\mathscr G_n\) and \(\sigma(A_{n+1},A_{n+2},\ldots)\) are independent. Since \(A\) belongs to the latter sigma-field for every \(n\),
The union on the right is an algebra: two of its sets lie in some common \(\mathscr G_n\). The class of all \(B\) satisfying the displayed identity is a Dynkin class, by the same complement/disjoint-union argument used in Theorem 10. It therefore contains \(\mathscr G_\infty\). But \(A\in\mathscr G_\infty\), so taking \(B=A\) yields \(P(A)=P(A)^2\). The only solutions in \([0,1]\) are zero and one. ◻
2. Independence
A sequence of real random variables is independent when the generated sigma-fields are independent. Here \(\sigma(X_i)=\{X_i^{-1}(H):H\in\mathcal B(\mathbb R)\}\). Equivalently, for every finite selection of distinct indices and every choice of Borel sets, the following product identity holds (with the selected indices relabelled \(1,\ldots,n\)):
If we take \(H_i = \{x_i\}\), then this is the familiar special case
This point-mass identity characterizes independence for countably valued variables. For general real variables it is only a necessary condition; for continuous laws all its terms can vanish even under dependence.
Checking independence on every measurable set would be cumbersome. The next theorem reduces the check to classes closed under finite intersections, then extends the result to the \(\sigma\)-fields they generate.
Definition 9. We often use class to refer to a collection of sets “following the same properties”, e.g. closed under finite intersections, countable unions, and so on. A collection of classes \(\{\mathscr K_1,\dots,\mathscr K_n\}\) in a given \(\sigma\)-field \(\mathscr F\) is independent if, for each choice of \(K_i \in \mathscr K_i\) for \(1 \leq i \leq n\), \(K_1,\dots,K_n\) are independent events.
Theorem 10. If each \(\mathscr K_i\) is a \(\pi\)-system (i.e. closed under finite intersections), and \(\mathscr K_1,\dots, \mathscr K_n\) are independent, then \(\sigma(\mathscr K_1), \dots, \sigma(\mathscr K_n)\) are independent.
Proof. Add \(\Omega\) to each generating pi-system; this preserves the hypotheses because independence includes every subcollection. First fix \(K_2,\ldots,K_n\) in their generating classes, and let \(B=\bigcap_{i=2}^nK_i\). Independence of these events gives \(P(B)=\prod_{i=2}^nP(K_i)\). The class
is a Dynkin class: it contains \(\Omega\), is closed under complements by subtracting from \(P(B)\), and under disjoint countable unions by countable additivity. It contains \(\mathscr K_1\) by hypothesis, so the pi–lambda theorem gives \(\sigma(\mathscr K_1)\subseteq\mathcal D\). This proves the product formula with the first coordinate allowed to range over its whole sigma-field. Repeat with the second coordinate, then the third, and so on. Choosing \(\Omega\) in unused coordinates proves the identities for all subcollections, which is independence of the generated sigma-fields. ◻
Theorem 11. Suppose \(\{X_{i,j} : i \in \mathbb N, 1 \leq j \leq n_i\}\) is an array of independent random vectors. Let \(\mathscr F_i\) be the \(\sigma\)-field generated by the \(i\)th row. Then, \(\mathscr F_1, \mathscr F_2, \dots\) are independent.
The finite intersections of coordinate inverse images form a generating pi-system \(\mathscr H_i\) for each row sigma-field. Independence of all array entries gives the required product identities for every finite selection from different row pi-systems. Theorem 10 therefore makes the generated row sigma-fields independent. ◻