From GPT to Production

Decoder-only models and the GPT family

Why everything became a causal LM

~12 minLesson 48 of 57Written by Claude Opus 4.8

GPT is the Transformer with a single rule added - you may only look left - trained to do exactly one thing: guess the next token.

The original Transformer (Lesson 34) had two halves: an encoder that reads a whole input at once and a decoder that writes an output. GPT throws away the encoder entirely. What is left is a stack of decoder blocks with one crucial constraint on their attention - each position may attend only to itself and the positions before it, never to the future - trained with a single objective: predict the next token (Lesson 37). That "look only left" constraint is causal masking, and it is the entire difference between a model that can read and a model that can write. Everything about the GPT family - GPT-2, GPT-3, and their descendants - is that one idea scaled up.

The whole model in one breath

Take a Transformer decoder. Mask the future so token $t$ can only see tokens $1 \dots t$ . Train it on oceans of text to answer one question at every position - what comes next? - and you have GPT. To generate, you run it forward, take the next-token distribution from the last position, pick a token, glue it onto the end, and run again. Read left-to-right, predict the next word, repeat. That is a decoder-only language model.

Decoder-only: masked, and that's the difference

Three shapes of Transformer exist, and they differ almost entirely in who is allowed to see whom inside attention. An encoder lets every token see every other token (great for understanding a fixed input). A decoder forbids looking ahead (necessary for generating, because at write-time the future genuinely does not exist yet). GPT is pure decoder.

Family	Attention	Trained to...	Example
Encoder-only	bidirectional (no mask)	fill in a masked-out token	BERT
Encoder-decoder	encoder bi + decoder causal + cross	map input sequence → output	T5, original Transformer
Decoder-only	causal (masked) self-attention	predict the next token	GPT-2 / GPT-3

Mechanically, causal masking is one line. Self-attention scores every query against every key as $S = Q K^{⊤} / d_{k}$ (Lesson 32). Before the softmax, we add a mask $M$ that is $0$ where a token is allowed to look and $- \infty$ where it is not. Because $e^{- \infty} = 0$ , every forbidden position gets exactly zero attention weight - the future contributes nothing.

Causal masking, and why it is next-token prediction

Attn (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}} + M) V, M_{ij} = {0 - \infty j \leq i j > i

The mask $M$ is a lower-triangular wall: row $i$ (the token doing the looking) may attend to column $j$ only when $j \leq i$ . That single constraint lets the model be trained on a whole sequence in one forward pass - position $i$ 's output predicts token $i + 1$ and provably never peeked at it. So the training objective is the autoregressive factorization from Lesson 37:

p_{θ} (x_{1}, \dots, x_{T}) = t = 1 \prod T p_{θ} (x_{t} ∣ x_{< t}), where softmax (z)_{k} = \frac{e ^{z_{k}}}{\sum _{j} e ^{z_{j}}}

turns the final layer's logits $z$ into that next-token probability distribution over the vocabulary.

Watch a tiny GPT generate one token at a time

Below is a toy decoder-only model seeded with the prefix the cat. Each press of Generate next token reads the current context, produces logits for the strongest next-token candidates, runs a real softmax to turn them into probabilities (the bars), appends the argmax, and grows the context by one. The causal-mask grid on the right grows too: each new row can attend to everything before it, and every earlier row gains one more masked cell - it is forbidden from seeing the token that arrived after it. Then branch: click any bar (not just the top one) to force a different token, or Rewind and choose again - and watch the whole continuation change.

Toy GPT - generate + causal mask step - click a bar to branch

thecat

Next-token distribution (logits - softmax)

tokenlogitsoftmaxp

This toy lists its 3 strongest candidates; the bars are a softmax over them (a real GPT softmaxes over a whole vocabulary). Click any bar to append that token instead of the argmax.

Causal attention mask

the

cat

the

●

cat

●

can attend (j ≤ i)masked future (j > i)

Highlighted row 1 (“cat”) is the query: it attends to all 2 tokens up to itself, and predicts the next.

position 1 → predict token 2 · argmax = “sat” (70%)

The two aha moments to catch

Causal mask (the grid). Read a row as "what this token is allowed to look at." Row 0 sees only column 0. Row 3 sees columns 0-3 but the cells to its right are blocked - that lower-triangular shape is the mask $M$ made visible. The highlighted last row is the query whose output becomes the next-token bars. Branching (the bars). Force ran instead of sat at the first step and the sentence veers into a completely different future - because every prediction is conditioned on the entire prefix, one different token reshapes everything downstream. That is generation as recursion with feedback (Lesson 48): the output is fed back in as the next input.

The GPT family, context windows, and one weight-saving trick

The lineage is the same architecture at growing scale. GPT-1 (2018, ~117M parameters) showed the recipe works. GPT-2 (2019, up to 1.5B, a 1024-token context) showed it writes fluent paragraphs. GPT-3 (2020, 175B, a 2048-token context) showed that at scale it does tasks it was never explicitly trained for. No architectural revolution between them - more layers, wider blocks, more data, longer context. The context window is the maximum number of tokens the model can attend over at once (the width of that mask grid); anything older simply falls off the left edge and is forgotten. One quiet efficiency trick threads through all of them - weight tying: the input token-embedding matrix $E \in R^{V \times d}$ is reused (transposed) as the output projection that produces logits, $logits = h E^{⊤}$ , so the same table that turns tokens into vectors also turns vectors back into token scores - halving a large chunk of the parameters.

tril = torch.tril(torch.ones(T, T))                 # lower-triangular 1s = the causal mask

# --- causal self-attention: the one thing that makes it "decoder-only" ---
att = (q @ k.transpose(-2, -1)) / math.sqrt(head_dim)   # scores, shape (T, T)
att = att.masked_fill(tril[:T, :T] == 0, float('-inf'))  # block the future → −∞
att = F.softmax(att, dim=-1)                          # each row sums to 1; masked → 0
y   = att @ v                                         # mix ONLY past + present values

# --- autoregressive generation: recursion with feedback (Lesson 48) ---
idx = prompt                                          # (T,) token ids
for _ in range(max_new_tokens):
    logits = model(idx)[-1]                       # next-token logits from the LAST position
    probs  = F.softmax(logits, dim=-1)             # over the whole vocabulary
    nxt    = torch.argmax(probs)                      # greedy - or torch.multinomial to sample
    idx    = torch.cat([idx, nxt])                    # append & feed back in - the recurrence
    if nxt == EOS: break                          # base case: the model chose to stop

nanoGPT, distilled: the mask and the loop

Check yourself

In a decoder-only model, what does causal masking guarantee?

A GPT is trained with a single objective - to...

Recall: why can a decoder-only model process a whole training sequence in parallel, yet must generate one token at a time?

Nail down the training vs. inference asymmetry. Try to state it, then check.

In training, every position's target (its next token) is already in the data, so a single masked forward pass yields all

T

next-token predictions at once - the causal mask

M

keeps each position from peeking ahead. At generation time the future tokens don't exist yet: each must be predicted from the last position, appended, and fed back before the next can be predicted, so it is inherently sequential. The KV cache is the memoization that stops it from recomputing every past token on each step.

Lock it in

GPT is a Transformer decoder stack with causal masking - each token attends only to itself and the tokens before it.
Causal masking adds $- \infty$ to attention scores for future positions, yielding zero weight after softmax - the future contributes nothing.
The same masked forward pass trains all $T$ next-token predictions in parallel; generation is inherently sequential (emit, append, repeat).
GPT-1, GPT-2, GPT-3 are the same idea at growing scale: more layers, wider blocks, longer context window, more data.
Weight tying reuses the input embedding matrix (transposed) as the output projection, halving a large chunk of parameters.

Primary source

Jay Alammar's The Illustrated GPT-2 walks through a decoder-only model visually - the masked self-attention, the token-by-token generation loop, and how the pieces stack. To build the entire thing in ~300 lines of PyTorch and read the exact causal-mask and generation code shown above, Andrej Karpathy's nanoGPT is the canonical reference implementation.

Ask your teacher

This lesson stitches together everything before it: the Transformer (Lesson 34) with its encoder removed and its decoder masked, driven by the next-token objective (Lesson 37). The generation loop is exactly the recursion-with-feedback from Lesson 48 - emit a token, feed it back, repeat until $⟨ eos ⟩$ . And the trick that makes it fast is a bridge worth its own visit: the KV cache is memoization. Instead of recomputing every past token's keys and values on every step, the model stores each one the first time it sees it and reuses it forever after - the same "trade memory to stop redoing work" you met in recursion. Want me to trace exactly what the KV cache holds on each step, or show what changes when we sample instead of taking the argmax? Ask and I'll add the follow-up.