Probability and Statistics
Conditional probability and Bayes
Updating beliefs when evidence arrives
You start with a belief. Then a piece of evidence arrives. How much should you move? Too little and you ignore data; too much and you panic at every false alarm. Bayes' theorem is the one correct amount to move - and its most famous lesson is deeply counterintuitive: a 90%-accurate test that comes back positive can still mean you're almost certainly fine.
The core move
A probability like is a belief with no strings attached. A conditional probability is that belief after you've been told is true - you zoom into the slice of the world where happened and re-measure how much of it also has . Learning updates belief; conditioning is how the update is done exactly.
Conditioning: zoom into the world where B happened
The definition is a fraction. Take the outcomes where both and occur, and divide by all the outcomes where occurs - because once you know , the rest of the world is off the table.
Definition, independence, joint vs marginal
If being told changes nothing, and are : , equivalently . The is a single cell of a two-way table; a like is a row or column total - you get it by summing the joint over everything you don't care about, e.g. .
Bayes' theorem: flip the condition you have into the one you want
You usually know - that's the test's advertised accuracy, measured on people already known to be sick. But you want - given your positive result, are you actually sick? Bayes flips one into the other.
The theorem, and its four named parts
Read right to left: is the (belief before evidence), the (how well predicts the evidence), the (a marginal - the total chance of seeing at all), and the (belief after). For the medical test, expanding over the two ways to test positive gives:
The base-rate trap, made visible
Here are 10,000 people as 10,000 dots. The sliders set how rare the disease is and how accurate the test is. The catch lives in the denominator above: false positives are drawn from the enormous healthy majority, so even a small false-positive rate produces a flood of false alarms. Tick "show only people who tested positive" and watch the true cases (green) get swamped by false alarms (red).
Out of 10,000 people, 100 truly have the disease.
Why "90% accurate" collapses to 8%
At 1% prevalence, 10,000 people hide only 100 sick ones - the test catches 90. But it also mislabels 10% of the 9,900 healthy people: 990 false alarms. So a positive result is 91 red dots for every 9 green ones in miniature - . The test barely changed: the prior did the damage. Now drag prevalence up to 50% and the same test suddenly means ~90% - evidence is only as strong as the base rate it lands on.
The whole update in eight lines:
def posterior(prior, sensitivity, specificity):
# P(disease | positive) via Bayes' theorem
p_pos_if_sick = sensitivity # true-positive rate
p_pos_if_healthy = 1 - specificity # false-positive rate
joint_sick = p_pos_if_sick * prior
joint_healthy = p_pos_if_healthy * (1 - prior)
evidence = joint_sick + joint_healthy # P(+), the marginal
return joint_sick / evidence # the posterior
posterior(0.01, 0.90, 0.90) # -> 0.0833... (about 8%)Check yourself
A test that is 90% accurate both ways, run for a disease affecting 1% of people, comes back positive. Roughly how likely are you to be sick?
In Bayes' theorem, the prior P(D) is combined with the evidence to produce the:
Recall: why can a 90%-accurate test still be wrong most times it says "positive"?
The relationship between accuracy and base rate is the crux. Try to state it, then check.
Lock it in
- zooms into the world where happened and re-measures .
- Bayes flips the condition: .
- A rare base rate swamps even a good test - 90% accuracy at 1% prevalence gives only ~8% posterior.
- False positives come from the huge healthy majority; true positives from the tiny sick minority.
Primary source
The tie forward