Language Modeling and Decoding

Language modeling and next-token prediction

The training objective behind every LLM

~12 minLesson 31 of 57Written by Claude Opus 4.8

Strip away the mystique and a large language model does exactly one thing: it reads the tokens so far and outputs a probability distribution over what comes next. Show it "The cat sat on the ___" and it does not know the answer - it assigns 41% to mat, 22% to floor, a sliver to banana. Training nudges those numbers so the right word gets more mass. Generating text is then just doing this over and over, feeding each guess back in. That's the whole machine.

One task to rule them all

A language model assigns a probability to any sequence of tokens $x_{1}, x_{2}, \dots, x_{T}$ . That sounds like it needs a probability for every possible sentence - astronomically many. The escape hatch is the chain rule of probability, which factors any joint probability into a product of one-step-ahead conditionals:

Factor the whole into next-token bets

p (x_{1}, x_{2}, \dots, x_{T}) = t = 1 \prod T p (x_{t} x_{1}, \dots, x_{t - 1})

Read it left to right: the probability of the sentence is the probability of the first token, times the probability of the second given the first, times the third given the first two, and so on. Every factor has the identical shape - predict one token given all the tokens before it. Learn that single conditional $p_{θ} (x_{t} ∣ x_{< t})$ well and you have modeled all of language.

Because each token is predicted from everything to its left, the model is called autoregressive - auto (self) + regressive (predicts from its own past). To generate, you sample a token from the distribution, glue it onto the end of the context, and ask again. The output becomes the next input: text generation is recursion with feedback.

Why "just predict the next word" is not a toy

To reliably continue "The capital of France is ___", "2 + 2 = ___", or "The murderer turned out to be ___", a model must absorb facts, arithmetic, grammar, and narrative logic - because all of those are needed to lower its error on real text scraped from the internet. The task is trivial to state and bottomless to master. Squeezing every drop of predictive accuracy out of "next token" is what forces the network to learn how the world is described.

Logits to a bet: the softmax

The network's final layer emits one raw score per vocabulary token - a vector $z \in R^{V}$ (with $V$ often above 50,000) called the logits. Logits are unbounded real numbers: they can be negative, they don't sum to anything nice. To turn them into a valid probability distribution we apply the softmax:

Softmax: scores in, probabilities out

p_{i} = softmax (z)_{i} = \frac{e ^{z_{i}}}{j = 1 \sum V e ^{z_{j}}}

Exponentiating makes every entry positive and blows up the gaps (a logit that is $2$ larger becomes $e^{2} \approx 7.4 \times$ more likely); dividing by the sum forces the outputs to be non-negative and to add to exactly $1$ . A quick concrete pass over three tokens with logits $[2.0, 1.0, 0.1]$ : the exponentials are $[7.39, 2.72, 1.11]$ , the sum is $11.21$ , so

softmax ([2.0, 1.0, 0.1]) = [0.66, 0.24, 0.10] .

The biggest logit wins the largest share, but every option keeps some probability - the model hedges.

Watch it predict, token by token

Below is a tiny hand-built model. Pick a starting prefix; the bars show its predicted distribution for the next token (correctly normalized by softmax over this toy vocabulary - the amber bar is the argmax). Hit Greedy to append the tallest bar, or Sample to roll a weighted die and let chance pick. Each appended token grows the prefix - and the causal-mask grid, which shows exactly which earlier positions each token is allowed to look at.

Next-token predictor softmax over the vocabulary

Start fromthe cat sat on the

0the

1cat

2sat

3on

4the

Model's guess for the next token

catz = 3.1

40.6%

dogz = 2.8

30.1%

matz = 1.9

12.2%

sunz = 1.5

8.2%

parkz = 1.2

6.1%

endz = 0.4

2.7%

Append a token to watch the sequence grow.

running avg surprise = - · perplexity = -

Causal mask - who may look at whom

the

cat

sat

the

✓

cat

✓

sat

✓

the

✓

attends (key ≤ query)masked (future - forbidden)

The "aha": generation is a loop, not a plan

Press Greedy a few times, then Reset and press Sample - same prompt, different sentence. The model never plotted a sentence out; it only ever answered "what's the single next token?", appended it, and asked again. That feedback loop is why the same prompt can produce different text, and why one confident wrong token can send the whole continuation off the rails. Notice the mask grid: every new row is exactly one square wider - position $t$ can see positions $0 \dots t$ and nothing ahead.

Grading the guess, and the no-peeking rule

How do we train those logits? We already know the right answer for every position in the training text - the next token is sitting right there. So the target is a one-hot vector (all mass on the true token), and we punish the model by how little probability it put on that correct token. That penalty is the cross-entropy loss (the same cross-entropy you met as a cost function in Lesson 17):

Cross-entropy = average surprise, in bits

L = - \frac{1}{T} t = 1 \sum T lo g p_{θ} (x_{t} x_{< t})

The quantity $- lo g_{2} p$ is the surprise of an event in bits: a token the model gave $p = 1.0$ costs $0$ bits (no surprise), $p = 0.5$ costs $1$ bit, $p = 0.25$ costs $2$ bits. Cross-entropy is just the model's average surprise per token on the true text - the exact information-theory idea. Minimizing it means "stop being surprised by real language." A closely related headline number is perplexity:

PPL = exp (L) = exp (- \frac{1}{T} \sum_{t} lo g p_{θ} (x_{t} ∣ x_{< t})) .

Perplexity is the model's effective branching factor: a perplexity of $10$ means the model is, on average, as unsure as if it were choosing uniformly among $10$ equally-likely words. Lower is better; a perfect oracle scores $1$ . (The demo above tracks both live as you append.)

One subtlety makes training efficient and honest. We want to predict the next token at every position at once - position 3 predicts token 4, position 4 predicts token 5, all in a single parallel pass. But position 3 must not be allowed to peek at token 4, because token 4 is the answer. Enter the causal mask: inside self-attention, before the softmax, we forcibly forbid every position from attending to any later position.

The causal mask: set the future to -infinity

Attention scores $s_{ij}$ (how much query $i$ wants to attend to key $j$ ) are masked additively, then softmaxed within each row:

M_{ij} = {0, - \infty, j \leq i j > i ⟹ a_{ij} = softmax_{j} (s_{ij} + M_{ij}) .

Because $e^{- \infty} = 0$ , any future position gets exactly zero attention weight - the softmax renormalizes over only the allowed positions. The result is the lower-triangular pattern in the demo: token $i$ attends to $0 \dots i$ and is blind to everything after. Forget this mask and the model "cheats" by reading the answer, trains to near-zero loss, and generates pure garbage at inference - where the future genuinely isn't there.

# x: (B, T) token ids.  The whole model reduces to one tensor:
# logits (B, T, V) - a score for every vocab token, at every position.
logits = model(x)

# TRAIN: predict token t+1 from everything up to t. Shift targets left by one.
loss = F.cross_entropy(
    logits[:, :-1, :].reshape(-1, V),  # guesses at positions 0 .. T-2
    x[:, 1:].reshape(-1)              # the true "next" token at each
)                                     # = mean over positions of -log p(correct)

# CAUSAL MASK inside attention: forbid looking ahead BEFORE the softmax.
scores = q @ k.transpose(-2, -1) / math.sqrt(d)
scores = scores.masked_fill(tril == 0, float("-inf"))  # upper triangle -> -inf
attn   = scores.softmax(dim=-1)              # masked cells become exactly 0

# GENERATE: autoregression - sample, append, feed back, repeat.
for _ in range(max_new_tokens):
    logits = model(x)[:, -1, :]        # only the last position predicts next
    probs  = logits.softmax(dim=-1)
    nxt    = torch.multinomial(probs, 1)  # roll the weighted die
    x      = torch.cat([x, nxt], dim=1)   # glue it on, loop again

The entire objective, in a few lines of PyTorch

Check yourself

The softmax layer at the top of a language model exists to:

Applied before the attention softmax, the causal mask lets each position:

Recall: why is minimizing cross-entropy on next-token prediction the same as reducing the model's "surprise"?

Nail down the information-theory connection. Try to state it, then check.

Cross-entropy is

- \frac{1}{T} \sum_{t} lo g p_{θ} (x_{t} ∣ x_{< t})

- the average of

- lo g p

over the true tokens, and

- lo g_{2} p

is precisely the surprise of an event in bits. So the training loss is the model's mean surprise per real token; driving it down means the model stops being surprised by actual language. Perplexity,

exp (cross-entropy)

, repackages the same number as an effective branching factor.

Lock it in

A language model factors the probability of any sentence into a product of next-token conditionals via the chain rule - learn one conditional and you have modeled all of language.
Softmax turns the raw logit vector into a valid probability distribution; cross-entropy measures how surprised the model is by the true next token.
The causal mask sets future attention scores to -infinity before softmax, enforcing the "no peeking" rule that makes autoregressive training honest.
Generation is recursion with feedback: sample a token, append it, predict again - the same prompt can produce different text each time.

Primary source

For the picture-by-picture story of a decoder-only model doing masked self-attention to predict the next token, read Jay Alammar's The Illustrated GPT-2. Then watch Andrej Karpathy build one from scratch - tokenization, the causal mask, cross-entropy training, and autoregressive generation - in "Let's build GPT: from scratch, in code, spelled out". Between the two you will have seen every idea on this page as running code.

Ask your teacher

Here is the thread that ties it all together: the cross-entropy we minimize is the information-theory notion of surprise measured in bits - the very same "how many bits to encode this outcome" idea - and predicting the next token is the entire training objective of every GPT. Generating text then is nothing but recursion with feedback: sample, append, repeat. Want me to go deeper on how the logits are produced (self-attention and the residual stream), why we sample instead of always taking the argmax (temperature, top-k, top-p - that's next), or how perplexity numbers translate to model quality? Ask and I'll build the follow-up.