Probability and Statistics

Probability, random variables, and distributions

Formalizing chance and its shapes

~12 minLesson 15 of 19Written by Claude Opus 4.8

Most of the world you care about is uncertain - the next token, the next click, the next measurement. Probability is the one language that lets you reason about that uncertainty quantitatively instead of hand-waving. You start with a set of things that can happen, attach numbers that obey three tiny rules, and out of that falls everything: expectation, variance, the bell curve, and - you will see it here - the very layer that lets a language model choose what to say.

The whole subject in one picture

Pin down what can happen (the sample space), spread a total of exactly 1.0 of "belief mass" across those outcomes (a distribution), then ask two questions of that mass: where is its center of balance (the expectation) and how spread out is it (the variance). That is the entire toolkit - the rest is just naming the common shapes the mass likes to take.

Three rules pin down all of chance

A sample space $Ω$ is the set of all possible outcomes of an experiment - for one die, $Ω = {1, 2, 3, 4, 5, 6}$ . An event $A$ is any subset of $Ω$ ("rolled even" is ${2, 4, 6}$ ). A probability is a function $P$ that assigns each event a number, and it only has to obey three axioms (Kolmogorov, 1933):

The Kolmogorov axioms

P (A) \geq 0, P (Ω) = 1, P (A \cup B) = P (A) + P (B) when A \cap B = \emptyset .

In words: probabilities are never negative, something in $Ω$ is certain to happen, and the chance of " $A$ or $B$ " for two outcomes that cannot co-occur is just their sum. Every identity you will ever use - $P (not A) = 1 - P (A)$ , the whole distribution summing to $1$ - is squeezed out of these three lines.

Random variables turn outcomes into numbers

A random variable $X$ is a function that maps each outcome to a number - the count of heads in 3 flips, the height of a person. Two flavors matter:

Discrete $X$ takes countable values. Its PMF (probability mass function) $p (x) = P (X = x)$ gives an honest probability to each value, and $\sum_{x} p (x) = 1$ .
Continuous $X$ takes a whole range of real values. Its PDF (probability density function) $f (x) \geq 0$ gives density, not probability - you get a probability only by taking area: $P (a \leq X \leq b) = \int_{a}^{b} f (x) d x$ , with $\int_{- \infty}^{\infty} f (x) d x = 1$ . A single exact point has $P (X = x) = 0$ .

From either one you read off the two summary numbers. The expectation is the mass-weighted average - the balance point:

E [X] = x \sum x p (x) (discrete), E [X] = \int x f (x) d x (continuous) .

The variance measures spread - the average squared distance from that balance point, with standard deviation $σ = Var (X)$ :

Var (X) = E [(X - μ)^{2}] = E [X^{2}] - μ^{2}, μ = E [X] .

The distribution lab

Four shapes cover most of what you will meet. Uniform: every outcome equally likely. Bernoulli( $p$ ): one trial, success with probability $p$ . Binomial( $n, p$ ): how many successes in $n$ independent trials - $P (X = k) = (k n) p^{k} (1 - p)^{n - k}$ , mean $n p$ . Normal( $μ, σ^{2}$ ): the continuous bell curve, $f (x) = \frac{1}{σ 2 π} e^{- (x - μ)^{2} /2 σ^{2}}$ . Below, the amber is the true distribution. Press Sample and watch teal draws pile into a histogram that converges to it - the law of large numbers, live - while the green fulcrum marks $E [X]$ , the balance point of the mass.

The amber is the true distribution. Press Sample and watch teal draws pile into a histogram that converges to it - the law of large numbers, live - while the green fulcrum marks E[X], the balance point of the mass.

true P(X) - theorysampled - empiricalE[X] balance pointsample mean x̄

trials n = 10success p = 0.50

E[X] theory

Var theory

2.50

samples N

sample mean x̄

A softmax is a probability distribution

An LLM's output layer takes raw scores (logits)

z

and runs

softmax (z)_{i} = \frac{e ^{z_{i}}}{\sum _{j} e ^{z_{j}}}

. Every entry is

\geq 0

and they sum to exactly

1

- that is precisely a PMF, but over the entire vocabulary instead of over

{0, 1, \dots, n}

. Picking a token is nothing more exotic than the Draw 1 button above: one draw from a distribution. Temperature just reshapes that distribution before you sample - flatter for more surprise, peakier for less.

import numpy as np

# 10,000 binomial draws: mean → n·p, variance → n·p·(1-p)
n, p = 10, 0.5
draws = np.random.binomial(n, p, size=10_000)
print(draws.mean(), draws.var())   # ~5.0   ~2.5

# a normal, and a softmax that IS a distribution over 3 tokens
x      = np.random.normal(loc=0.0, scale=1.0, size=10_000)
logits = np.array([2.0, 1.0, 0.1])
probs  = np.exp(logits) / np.exp(logits).sum()   # sums to 1.0
token  = np.random.choice(3, p=probs)             # one draw = one token

Sampling in NumPy - the mean lands where the theory says

Check yourself

For a continuous random variable, the probability that X equals one exact value is:

As you draw more and more samples in the lab, the histogram of outcomes will:

What is the difference between a PMF and a PDF, and what does each add up to?

The distinction discrete vs continuous governs how you compute probabilities. Try to state it, then check.

Lock it in

Three axioms (non-negative, total = 1, additive for disjoint events) pin down all of probability.
A random variable maps outcomes to numbers; its distribution (PMF or PDF) spreads exactly 1.0 of mass across them.
Expectation is the balance point; variance is the average squared distance from it.
Binomial counts successes in n trials; Normal is the continuous bell curve; both converge empirically via the law of large numbers.
A softmax IS a PMF over a vocabulary - sampling a token is one draw from a distribution.

Where this leads

Want to see why the binomial becomes the normal as

n

grows (the Central Limit Theorem), or how expectation behaves under sums and scaling? The payoff is close: an LLM's output layer is a softmax - a probability distribution over the vocabulary - and sampling a token is exactly drawing from a distribution like the ones you just built.

Primary source

Play with distributions visually at Brown's Seeing Theory - Probability Distributions, an interactive treatment that inspired this lab. For the same ideas explained slowly and from scratch, work through Josh Starmer's StatQuest statistics playlist.