Probability
Measuring uncertainty — from counting equally likely outcomes to updating beliefs with new evidence, and the distributions that model repeated chance
- Sample spaces and events, and how probability assigns numbers to them.
- Classical probability and the addition theorem for combining events.
- Conditional probability, the multiplication theorem, and independence.
- The total probability theorem and Bayes' theorem for updating with evidence.
- Random variables, with their mean and variance.
- The binomial distribution for repeated independent trials.
Sample Space & Events
A random experiment is one whose outcome cannot be predicted with certainty. The set of all possible outcomes is the sample space \(S\), and any subset of it is an event. Rolling a die has \(S=\{1,2,3,4,5,6\}\), and "an even number" is the event \(\{2,4,6\}\).
The whole space \(S\) is the sure event and the empty set is the impossible event. Two events are mutually exclusive if they cannot occur together (\(A\cap B=\varnothing\)), and the complement \(A'\) is "A does not happen," with \(P(A')=1-P(A)\).
Classical Probability & the Addition Theorem
When every outcome is equally likely, probability is just counting: favourable outcomes over total outcomes. To find the chance that either of two events happens, add their probabilities — but subtract the overlap, so it is not counted twice.
For mutually exclusive events the overlap is empty, so the formula collapses to \(P(A\cup B)=P(A)+P(B)\). The classical definition applies only when outcomes are equally likely; otherwise probabilities come from data or a model.
Conditional Probability
Learning that one event has occurred can change the odds of another. Conditional probability \(P(A\mid B)\) answers "given that \(B\) has happened, how likely is \(A\)?" — and the trick is that the sample space shrinks down to \(B\).
Dividing by \(P(B)\) rescales so that \(B\) becomes the new certainty. Note the direction matters: \(P(A\mid B)\) and \(P(B\mid A)\) are generally different — reversing them is a classic error that Bayes' theorem later corrects.
Multiplication & Independence
Rearranging the conditional formula gives the probability that both events occur. When knowing one event tells you nothing about the other, they are independent, and the rule simplifies to a plain product.
The first rule always holds. If the events are independent, \(P(A\mid B)=P(A)\) — the condition adds no information — and the joint probability is just the product. Independence is the natural model for repeated tosses, draws with replacement, and separate trials.
These two ideas are often confused but are nearly opposites. Mutually exclusive events cannot happen together (\(P(A\cap B)=0\)), so if one occurs the other certainly did not — that is a very strong dependence. Independent events satisfy \(P(A\cap B)=P(A)P(B)\). Two events with non-zero probability cannot be both at once.
The Total Probability Theorem
Suppose the sample space is carved into several non-overlapping cases that together cover everything — a partition. The probability of any event is then a weighted average of its chances within each case, weighted by how likely each case is.
The events \(B_1,B_2,\dots\) must be mutually exclusive and exhaustive (their union is \(S\)), each with \(P(B_i)>0\). The theorem assembles \(P(A)\) from the easier conditional pieces — ideal when \(A\) can arrive by several routes, such as a defective item from one of several machines.
Bayes' Theorem
Total probability runs forwards, from causes to an effect. Bayes' theorem runs it backwards: having observed the effect \(A\), what is the probability it came from a particular cause \(B_i\)? It is the engine of evidence-based reasoning.
The numerator is one path through the tree; the denominator is the total probability of \(A\) over all paths. The \(P(B_i)\) are prior probabilities and the \(P(B_i\mid A)\) are posteriors — beliefs revised in the light of the evidence \(A\).
Random Variables
A random variable \(X\) attaches a number to each outcome — the number of heads in three tosses, say. Listing its values with their probabilities gives a probability distribution, summarised by a mean (its centre) and a variance (its spread).
The probabilities satisfy \(\sum_i p_i=1\). The mean (or expectation) \(\mu=E(X)\) is the long-run average value, and the variance \(\sigma^2\) measures how widely the values scatter about \(\mu\); its square root \(\sigma\) is the standard deviation.
The Binomial Distribution
Repeat the same success-or-failure trial \(n\) times, independently, with a fixed success probability \(p\). The number of successes follows the binomial distribution — the most important discrete model in the course.
Here \(q=1-p\) and \(r=0,1,\dots,n\). The four conditions are: a fixed number of trials, only two outcomes per trial, a constant \(p\), and independence. The distribution is symmetric when \(p=\tfrac12\) and skewed otherwise.
Putting It to Work
Problem. A card is drawn from a standard pack of \(52\). Find the probability that it is a king.
Solution. There are \(4\) kings among \(52\) equally likely cards:
Problem. Find the probability that the card is a king or a heart.
Solution. There are \(4\) kings, \(13\) hearts, and \(1\) king of hearts shared:
Problem. A card is known to be a face card (J, Q, K). Find the probability that it is a king.
Solution. The condition shrinks the space to the \(12\) face cards, of which \(4\) are kings:
Problem. If \(P(A)=0.3\) and \(P(B)=0.4\) are independent, find \(P(A\cap B)\) and \(P(A\cup B)\).
Solution. Multiply for the intersection, then use the addition theorem:
Problem. A disease affects \(1\%\) of people. A test detects it in \(99\%\) of carriers but is also positive for \(2\%\) of healthy people. A test is positive — find the probability the person has the disease.
Solution. With \(P(D)=0.01\), total positive probability is \(0.01(0.99)+0.99(0.02)=0.0297\):
Problem. A fair coin is tossed \(5\) times. Find the probability of exactly \(3\) heads.
Solution. Use \(n=5,\ p=q=\tfrac12,\ r=3\):
Chapter Summary
\(E\subseteq S\); \(P(A')=1-P(A)\).
\(P(A\cup B)=P(A)+P(B)-P(A\cap B)\).
\(P(A\mid B)=\tfrac{P(A\cap B)}{P(B)}\).
\(P(A\cap B)=P(A)P(B)\); not the same as exclusive.
\(P(A)=\sum P(B_i)P(A\mid B_i)\); Bayes reverses it.
\(E(X)=\sum x_ip_i\); binomial mean \(np\), variance \(npq\).
Problems
Name the tool first — counting, addition, conditioning, independence, total probability, Bayes, or a distribution — then apply it. Difficulty rises down the list.
- A die is rolled. Find the probability of getting an even number.
- Two coins are tossed. Find the probability of at least one head.
- A card is drawn from \(52\). Find the probability that it is a heart or a queen.
- If \(P(A)=0.5,\ P(B)=0.3,\ P(A\cap B)=0.1\), find \(P(A\cup B)\).
- A die shows an even number. Find the probability that it is a multiple of \(3\).
- If \(P(A)=0.6,\ P(B)=0.5\) are independent, find \(P(A\cap B)\) and \(P(A\cup B)\).
- A bag has \(3\) red and \(5\) black balls; two are drawn without replacement. Find \(P(\text{both red})\).
- A random variable \(X\) takes values \(0,1,2\) with probabilities \(0.2,0.5,0.3\). Find \(E(X)\).
- For the distribution in Problem 8, find \(\operatorname{Var}(X)\).
- A fair die is thrown \(4\) times. Find the probability of exactly two sixes.
- A coin is tossed \(6\) times. Find the probability of at least one head.
- Machines \(A\) and \(B\) make \(60\%\) and \(40\%\) of items, with \(2\%\) and \(3\%\) defective. An item is defective; find the probability it was made by \(A\).
- A student passes Maths with probability \(0.7\), Physics with \(0.6\), and at least one with \(0.8\). Find the probability of passing both.