Information theory: entropy and cross-entropy

When a friend says "the sun rose this morning," you learn nothing - you already knew it. When they say "it snowed in the desert," you sit up. Information theory turns that gut feeling into arithmetic: the surprise of an event is $-\log p$, measured in bits. Average that surprise over a whole distribution and you get entropy. Average it while using the wrong probabilities and you get cross-entropy - which, it turns out, is the exact number an LLM is trained to push down, token after token.

Surprise, measured in bits

The information content (or "surprise") of an outcome $x$ with probability $p(x)$ is

The minus sign is only there because probabilities are $\le 1$, so their logs are negative - surprise should be a positive number. Two properties make this the only sensible choice. First, it is monotone: smaller $p$ means larger surprise, and a sure thing ($p=1$) has surprise $0$. Second, it is additive for independent events - the surprise of two coin flips is $-{\log }_2(\frac{1}{2}\cdot \frac{1}{2})=2$ bits, not because we multiplied but because $\log$ turns the product of probabilities into a sum of surprises.

Now average that surprise over the whole distribution, weighting each outcome by how often it actually happens. That average is Shannon entropy:

Entropy is the irreducible uncertainty of a source: the average number of yes/no questions you must ask to pin down its outcome, and - by Shannon's coding theorem - the shortest average code length in bits that any scheme can achieve. A predictable source (one likely outcome) has low entropy; a uniform source has the most, ${\log }_{2}n$ bits for $n$ equally likely options.

Cross-entropy and KL: paying for a wrong map

Here is the twist that matters for machine learning. Suppose reality follows a distribution $p$, but you build your code - your model - as if the distribution were $q$. You still get charged the real surprise $-{\log }_{2}q(x)$ every time $x$ occurs, but $x$ occurs at the true rate $p(x)$. Your average bill is cross-entropy:

It is the average surprise of the truth, scored with the model's probabilities. Cross-entropy is never smaller than the true entropy, and the gap between them has a name - the Kullback-Leibler divergence, the number of extra bits you waste by using $q$ instead of $p$:

Drag the model's guesses

Below is a miniature language model. The prompt is "The cat sat on the ___" and the vocabulary has four tokens. The dashed ticks are the true next-token distribution $p$ (what the corpus actually does); the solid bars are the model's guess $q$, which you can drag up and down. As you make $q$ resemble $p$, the cross-entropy loss slides toward its floor $H(p)$. Starve the true winner of probability - make the model confident and wrong - and watch the loss bolt toward infinity. Hit "Draw the actual token" to sample from reality and see the surprise $-{\log }_{2}q$ you paid on that one draw.

Cross-entropy, and the one-hot special case, in a few lines:

# p: the true distribution, q: the model's guess (both sum to 1)
import numpy as np

def cross_entropy(p, q):
    return -np.sum(p * np.log2(q))     # bits; use np.log for nats (frameworks' default)

# The LM case: the target is one-hot on the real next token.
loss = -np.log2(q[true_token])         # surprise of the actual token, in bits
ppl  = 2 ** loss                       # perplexity: the "effective number of choices"

PyTorch's cross_entropy uses the natural log, so its numbers are in nats rather than bits - but the story is identical, and perplexity is just $e^{\text{loss}}$ instead of $2^{\text{loss}}$. Base changes the unit, not the meaning.

Check yourself