Probability axioms

A probability measure obeys three axioms, usually known as the Kolmolgorov axioms:

1. For any set \(A \in \mathcal{F}\), \(P(A) \geq 0\).
2. \(P(\Omega) = 1\).
3. For any mutually disjoint sets \(A_1, A_2 \ldots A_3 \ldots\), where each \(A_i \in \mathcal{F}\), then \[ P\left(\bigcup_{i=1}^\infty A_i \right) = \sum_{i=1}^\infty P(A_i). \]

Put informally, \(P\) is a function that assigns a measure, which we can view as something like mass, to each subset of \(\Omega\). This measure is non-negative, and the total amount of measure it assigns to the whole of \(\Omega\) is \(1\). The measure assigned to any subset of \(\Omega\) is the sum of the measures assigned to its disjoint subsets. Put another way, if \(A\) and \(B\) are two disjoint subsets of \(\Omega\), then the measure assigned to \(A \cup B\) is equal to the sum of the measure assigned to \(A\) an \(B\).

Probability theorems

Having defined our probability space where the probability measure \(P\) obeys the probability axioms, we can now state endless numbers of theorems. Here are some examples.

• \(0 \leq P(A) \leq 1\)
• \(P(A^c) = 1 - P(A)\)
• \(P(\emptyset) = 0\)
• \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\)

These can be proved using basic set theory and the probability axioms, as follows.

• \(0 \leq P(A) \leq 1\): Given that \(A \subseteq \Omega\), \(P(A) \leq 1\), and by the first axiom, \(P(A) \geq 0\).
• \(P(A^c) = 1 - P(A)\): If \(A\) is a subset of \(\Omega\) and \(A^c\) is its complement, then \(A \cup A^c = \Omega\), and so \(P(A \cup A^c) = P(\Omega) = 1\). By definition, \(A\) and \(A^c\) are disjoint, and so by by axiom 3, \(P(A \cup A^c) = P(A) + P(A^C)\). Therefore, \(P(A) + P(A^C) = 1\) and \(P(A^c) = 1 - P(A)\).
• \(P(\emptyset) = 0\): The complement of \(\Omega\) is \(\Omega^c = \emptyset\). By the previous theorem, \(P(\Omega^c) = 1 - P(\Omega\)), and so \(P(\emptyset) = 1 - 1 = 0\).
• \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\): By the algebra of sets, \(A \cup B = (A \cup B) \cap \Omega = (A \cup B) \cap (A \cup A^c)\) and so from this, we have \(A \cup B = A \cup (B \cap A^c)\), and \(A\) and \(B \cap A^c\) are disjoint. Similarly, \(B = B \cap \Omega = B \cap (A \cup A^c)\), and so from this, we have \(B = (A \cap B) \cup (B \cap A^c)\). And so \(P(A \cup B) = P(A) + P(B \cap A^c)\). Given that \(P(B) = P(A \cap B) + P(B \cap A^c)\), we have \(P(B \cap A^c) = P(B) - P(A \cap B)\). Therefore, \(P(A \cup B) = P(A) + P(B) - P(A \cap B)\).

Conditional probability

For any two events \(A\) and \(B\), the conditional probability of \(A\) given that \(B\) has occurred, written as \(P(A \vert B)\), is defined as follows: \[ P(A \vert B) = \frac{P(A \cap B)}{P(B)}. \] For example, using the die throw example, if \(A = \{2, 4, 6\}\) and \(B = \{4, 5, 6\}\), the conditional probability of \(A\) given that \(B\) has been observed is \[ \frac{P(A \cap B)}{P(B)} = \frac{P(\{4, 6\})}{P(\{4, 5, 6\})}. \]

In general then, conditional probability effectively defines a new probability space with \(B\) as the sample space, the subsets of \(B\) as the set of events, and the probability measure assigned to these events is the probability measure from the original probability space divided by \(P(B)\).

Note that by the definition of conditional probability above, we also have \[ P(A \cap B) = P(A \vert B) P(B), \] and also \[ P(A \cap B) = P(B \vert A) P(A), \] and this means, for example, \[ P(A \vert B) P(B) = P(B \vert A) P(A), \] and \[ P(B \vert A) = \frac{P(A \vert B) P(B)}{P(A)}, \] and so on.

Random variables

A random variable is not like a variable in the usual mathematical sense of the term. It is in fact a function mapping from the sample space \(\Omega\) to a new measurable space, which is usually the real line \(\mathbb{R}\). We define a random variable \(X\) more formally as follows: \[ X \colon \Omega \mapsto \mathbb{R}. \]

For any \(s \subseteq \mathbb{R}\), the probability that \(X\) takes a value in \(s\) is \[ \mathrm{P}( X \in s ) = P(\{\omega\colon X(\omega) \in s\}). \] We can read this as follows: the probability that \(X\) takes a value in the set \(s\) is the probability measure assigned to the set of all outcomes \(\omega\) such that \(X(\omega) \in s\).

As an example, again using our die throw example where \(\Omega = \{1, 2, 3, 4, 5, 6\}\), we can define a random variable \(X\) that maps each outcome in the set \(\{1, 2, 3, 4\}\) to \(0\), and each outcome in the set \(\{5, 6\}\) to \(1\). If our probability measure on \(\Omega\) is simply that each of the 6 possible outcomes has a probability of \(\frac{1}{6}\), then we have \[ \begin{aligned} \mathrm{P}( X = 0 ) &= P(\{\omega\colon X(\omega) = 0\}),\\ &= P(\{1, 2, 3, 4\}) = \frac{2}{3}. \end{aligned} \]

Somewhat informally, the probability assigned by the random variable \(X\) to each possible value in \(\mathbb{R}\) is known probability distribution of \(X\), and written \(\mathrm{P}( X )\). This \(\mathrm{P}( X )\) is a function (possibly continuous, possibly discontinuous) over \(\mathbb{R}\). Every value of this function must be non-negative and it must integrate to 1.0.

Independent random variables

Two variables random variables \(X\) and \(Y\) are independent if and only if, for all possible values of \(s\) and \(r\), we have \[ \mathrm{P}( X = s, Y = r ) = \mathrm{P}( X = s )\mathrm{P}( Y = r ). \] Given that, as we have seen, \[ \mathrm{P}( X = s \vert Y = r ) = \frac{\mathrm{P}( X = s, Y = r )}{\mathrm{P}( Y = r )}, \] then \[ \mathrm{P}( X = s, Y = r ) = \mathrm{P}( X = s \vert Y = r )\mathrm{P}( Y = r ), \] and so, if \(X\) and \(Y\) are independent, we see that \(\mathrm{P}( X = s \vert Y = r ) = \mathrm{P}( X = s )\). From this perspective, we that independence means that the probability that \(X\) takes on any given value does not vary with the value that \(Y\) takes.

Chain rule

We have seen above that for random variables \(X\), \(Y\) \[\mathrm{P}( X, Y ) = \mathrm{P}( X\vert Y )\mathrm{P}( Y ) = \mathrm{P}( Y\vert X )\mathrm{P}( X ).\]

Similarly, for \(X\), \(Y\), \(Z\), \[\begin{aligned} \mathrm{P}( X, Y, Z ) &= \mathrm{P}( X \vert Y, Z )\mathrm{P}( Y \vert Z )\mathrm{P}( Z ),\\ &=\mathrm{P}( Y \vert X, Z )\mathrm{P}( X \vert Z )\mathrm{P}( Z ),\\ &=\mathrm{P}( Z \vert X, Y )\mathrm{P}( X \vert Y )\mathrm{P}( Y ). \end{aligned} \]

This means that any joint probability distribution can be decomposed in the products conditional probability distributions.

Reasoning with probabilistic models

If we have, for example, a set of random variables \(y_1, y_2 \ldots y_n\) that are independently and identically distributed as \(N(\mu, \sigma^2)\), which is a normal distribution with mean \(\mu\) and variance \(\sigma^2\), then if \(\mu\) and \(\sigma\) also defined by probability distributions, our joint probabilistic model over all these variables is a probability distribution over \(n + 2\) random variables \[ \mathrm{P}( y_1, y_2, y_2 \ldots y_n, \mu, \sigma^2 ). \]

Because of conditional independence, namely that \(y_1, y_2 \ldots y_n\) are mutually independent given knowledge of \(\mu\) and \(\sigma^2\), this joint probability decomposes as follows: \[ \prod_{i=1}^n \mathrm{P}( y_i \vert\mu, \sigma^2 )\mathrm{P}( \mu, \sigma^2 ). \] The marginal probability of \(y_1 \ldots y_n\) is \[ \mathrm{P}( y_1 \ldots y_n ) = \int \mathrm{P}( y_1 \ldots y_n, \mu, \sigma^2 ) d\mu d\sigma^2 \]

Bayes theorem

Given a joint probability distribution over \(y_1 \ldots y_n\) and \(\mu\), \(\sigma^2\), we define the conditional probability of \(\mu\) and \(\sigma^2\) given \(y_1 \ldots y_n\), using definitions above, as follows: \[ \begin{aligned} \mathrm{P}( \mu, \sigma^2 \vert y_1 \ldots y_n ) &= \frac{\mathrm{P}( y_1 \ldots y_n, \mu, \sigma^2 )}{\mathrm{P}( y_1 \ldots y_n )},\\ &= \frac{ \prod_{i=1}^n \mathrm{P}( y_i \vert\mu, \sigma^2 )\mathrm{P}( \mu, \sigma^2 ). }{ \mathrm{P}( y_i \ldots y_n ) },\\ &= \frac{ \prod_{i=1}^n \mathrm{P}( y_i \vert\mu, \sigma^2 )\mathrm{P}( \mu, \sigma^2 ). }{ \int \prod_{i=1}^n \mathrm{P}( y_i \vert\mu, \sigma^2 )\mathrm{P}( \mu, \sigma^2 ) d\mu d\sigma^2. } \end{aligned} \]

This is Bayes theorem, which the basis of statistical inference in Bayesian data analysis.