Language Modeling and Decoding

Decoding and sampling strategies

How a model picks which token comes next

~11 minLesson 32 of 57Written by Claude Opus 4.8

The transformer's last layer produces one raw score - a logit - for every token in the vocabulary. Softmax turns those logits into a probability distribution. But a distribution isn't text. Something has to select an actual token, and that selector has knobs: temperature, top-k, and top-p. Turn them one way and the model is a careful, repetitive clerk; turn them the other and it's a reckless poet. This lesson is those knobs.

The core question

Given a distribution like { mat: 39%, rug: 17%, floor: 13%, ... }, which token do we emit? The simplest answer is greedy decoding: always take the argmax - the single highest-probability token. It's deterministic and safe, but it's also bland and prone to loops. Every other strategy in this lesson is a way to inject controlled randomness so the text stays coherent without becoming robotic.

The knobs, one at a time

Temperature reshapes the distribution before we sample by dividing every logit by a scalar $T$ inside the softmax. Small $T$ sharpens the peak (toward greedy); large $T$ flattens it (toward uniform). Top-k keeps only the $k$ likeliest tokens and throws the rest away. Top-p (nucleus) is smarter about the tail: instead of a fixed count, it keeps the smallest set of tokens whose probabilities add up past a threshold $p$ - so it keeps many tokens when the model is unsure and few when it's confident. After any cut, the survivors are re-normalized so their probabilities sum to 1 again.

Temperature-scaled softmax

For logits $z_{1}, \dots, z_{V}$ and temperature $T > 0$ , the probability of token $i$ is:

p_{i} = \frac{exp ( z _{i} / T )}{j = 1 \sum V exp ( z _{j} / T )} .

As $T \to 0^{+}$ the largest logit dominates and $p$ collapses onto the argmax - that is greedy decoding. As $T \to \infty$ every $z_{i} / T \to 0$ , so $p$ flattens toward the uniform distribution $1/ V$ . Top-p then keeps the smallest set $S$ with $\sum_{i \in S} p_{i} \geq p$ , and we re-normalize over $S$ .

Play the sampler

Below is a real next-token distribution for the prompt, built from 12 fixed logits. Drag the three sliders and watch the bars re-shape and re-normalize in real time: grey bars are tokens the filters have rejected, the amber dashed tick marks where each surviving bar sat before re-normalizing, and the solid accent shows where it lands after. Then press Sample to draw one token from the surviving distribution.

Sampling control paneldrag T / k / p, then sample

The cat sat on the ___

mat

39%

rug

18%

floor

13%

sofa

couch

bed

chair

table

windowsill

roof

lap

throne

surviving (after re-norm)probability before re-normrejected by a filter

temperature T = 1.00top-k = 12 (all)top-p = 1.00 (all)

12 of 12 survive - 100% of the raw mass

Move the sliders, then press Sample to draw a token.

Two aha moments to trigger

Slide T down to 0.1: the top bar swallows almost everything - that's greedy. Now slide top-k to 1: exactly one bar survives, and every sample returns mat - top-k = 1 is greedy. Then set T back to 1 and drag top-p to 0.90: the tail greys out, and every surviving accent bar grows past its amber tick. That growth is re-normalization - the removed mass gets redistributed to the tokens you kept.

The whole pipeline in one function

import numpy as np

def sample(logits, T=1.0, k=12, p=1.0):
    logits = logits / T                    # temperature: scale the logits
    order  = np.argsort(logits)[::-1]      # tokens high -> low
    order  = order[:k]                     # top-k: keep the k largest (a heap)
    probs  = softmax(logits[order])         # re-normalize the survivors
    cum    = np.cumsum(probs)             # prefix sums over sorted probs
    m      = np.searchsorted(cum, p) + 1   # top-p: smallest prefix reaching p
    keep   = order[:m]
    probs  = softmax(logits[keep])          # re-normalize again over the nucleus
    return np.random.choice(keep, p=probs)  # draw ONE token from the survivors

Two more you'll meet

Beam search doesn't commit token-by-token - it keeps the b most probable partial sequences alive at once and expands them, a best-first search driven by a priority queue. It maximizes total sequence probability, which is great for translation but tends to produce dull, repetitive open-ended text. Repetition penalty patches a common failure of all these methods: it subtracts from (or divides) the logits of tokens already generated, discouraging the model from looping the the the. Real systems stack these - e.g. temperature + top-p + a repetition penalty.

Check yourself

Raising the temperature T before the softmax:

Top-p (nucleus) sampling keeps the smallest set of tokens whose...

Recall: why must we re-normalize the survivors after top-k or top-p?

This is the key step that the demo makes visible. Try to state it, then check.

Lock it in

A language model outputs logits; softmax converts them to a probability distribution over the vocabulary.
Temperature divides logits before softmax: low T sharpens toward greedy, high T flattens toward uniform.
Top-k keeps the k likeliest tokens; top-p (nucleus) keeps the smallest set whose cumulative probability exceeds p - adaptive to model confidence.
After any filter, survivors are re-normalized (divide by their sum) so probabilities add to 1 again.
Real systems stack multiple strategies (temperature + top-p + repetition penalty) to balance coherence and diversity.

Primary source

The Hugging Face LLM Course walks through these generation strategies with runnable model.generate() examples. For the visual intuition of how a transformer produces that next-token distribution in the first place, Jay Alammar's The Illustrated GPT-2 is the classic companion.

Ask your teacher

Notice that decoding is really a selection problem in disguise. Greedy is just argmax. Top-k is "find the k largest logits" - a job for a heap. Top-p is prefix-sums plus a binary search into the cumulative array. And beam search is a best-first search over sequences, powered by a priority queue. Want me to build the beam-search visualizer, or derive why low temperature causes repetition loops? Ask and I'll add the follow-up.