The Transformer

The transformer architecture

Encoder, decoder, and the stack that scales

~13 minLesson 28 of 57Written by Claude Opus 4.8

One block, repeated. Attention mixes the tokens, a feed-forward layer rethinks each one - stack that recipe a dozen times and you have the engine behind every modern LLM.

You already have every piece. Self-attention lets tokens look at each other; a feed-forward layer is the plain neural net from Lesson 20. The Transformer just bolts them together into one block - attention, then a feed-forward - and stacks that identical block $N$ times. Feed a sentence in at the bottom, and the answer (a probability for the next word) reads out the top. That top-to-bottom trip is the same forward pass you already know, now running on a whole sequence at once.

The block is a fixed four-move recipe

Inside one Transformer block, the same four moves happen in the same order, every time: (1) self-attention - every token gathers context from the others; (2) add & norm - keep the original vector and gently re-center it; (3) feed-forward - a small neural net re-thinks each token on its own; (4) add & norm again. That's it. A deep model is not a new idea - it's this one block copied $N$ times.

Attention mixes, feed-forward rethinks

The two halves of a block do genuinely different jobs. Self-attention is the only step that lets positions talk: the vector at "it" pulls in information from "the cat" earlier in the sentence. The feed-forward half never looks sideways - it applies the same little two-layer network (the $a = σ (W x + b)$ from Lesson 20) to each token independently. Mix across tokens, then think about each token: that alternation, repeated, is the whole design.

One block, written out (light math)

Let $x$ be the stack of token vectors entering the block. Each sub-layer is wrapped in a residual add and a LayerNorm - exactly the pattern from the original paper:

u = LN (x + Attn (x)), block (x) = LN (u + FFN (u)) .

The $+ x$ is the residual: the block adds a correction rather than replacing the vector. Because the output is the same shape as the input, blocks snap together like Lego - and the running vector is often called the residual stream. (We unpack residuals, LayerNorm and the FFN next lesson.)

Trace one token up the stack

Below is a simplified decoder-only stack - the GPT shape. Follow a single sentence, "the cat sat", as its vectors climb from token IDs at the bottom to a next-word probability at the top. Press Next (or Run) to move the glowing marker one stage at a time. Watch the readout: through every block the shape stays $[seq, d_{model}]$ - only the final linear layer changes it.

Decoder-only stack tracer step the marker up the stack

Softmax -> probabilitieslast row = next-token distribution

Final Linear (unembed)each vector -> one score per word

Blocks 2 ... Nidentical block, stacked (GPT-2: N=12)

Transformer block 1

Add & Normresidual + LayerNorm

Feed-Forwardper-token MLP, shared weights

Add & Normresidual + LayerNorm

Masked Self-Attentionmix across positions (causal)

Embedding + PositionalIDs -> vectors, add position

Tokens"the" . "cat" . "sat"

tensor shape[seq] = [3]

Three tokens enter as integer IDs - one per word. Nothing is a vector yet.

The 'aha': the shape never changes - until the very end

Notice the marker's shape readout freeze at $[seq, d_{model}]$ from the embedding all the way to the last block. That constancy is the whole trick that lets you stack blocks arbitrarily deep: each block returns exactly what it was handed, just improved. Depth here means the same thing it did for a plain network in Lesson 20 - more chances to re-transform the data - except each "mixing" step is now self-attention instead of a fixed weight matrix. Only the final linear layer breaks the shape, projecting to $[seq, vocab]$ so softmax can name a next token.

Decoder-only, and why the mask matters

The original Transformer had two halves, an encoder and a decoder. GPT-style models keep only the decoder stack - hence decoder-only. Its one non-negotiable rule: when predicting the word at position $i$ , a token may attend to itself and everything before it, but never to words that come later. This is masked (causal) self-attention: before the softmax, every attention score pointing to a future position is set to $- \infty$ , so its weight becomes $0$ .

Without the mask, the model cheats

Training teaches the model to predict the next token. If position $i$ could see the future, it would simply read off the answer sitting one step ahead - learning nothing. The causal mask forces every prediction to rest only on the past, which is exactly the setting the model faces at generation time, when the future genuinely does not exist yet.

The whole architecture, as a short forward pass:

# one decoder block: attention mixes, FFN rethinks
def block(x):                       # x: [seq, d_model]
    a = masked_self_attention(x)     # mix across positions
    x = layer_norm(x + a)            # add & norm  (residual)
    f = feed_forward(x)             # per-position MLP
    x = layer_norm(x + f)            # add & norm  (residual)
    return x                        # [seq, d_model] - same shape

def transformer(tokens):             # tokens: [seq] of int ids
    x = embed(tokens) + positional(tokens)   # [seq, d_model]
    for _ in range(N):                # stack N identical blocks
        x = block(x)
    logits = x @ W_unembed           # [seq, vocab]
    return softmax(logits[-1])        # next-token distribution

The whole architecture, as a short forward pass

That is the entire Transformer: embed, loop the block $N$ times, project, softmax. Everything else - the exact attention math, LayerNorm, the FFN - is detail packed inside those helper calls.

Check yourself

What are the four moves inside one Transformer block, in order?

In a decoder-only (GPT) stack, masked self-attention lets each token attend to:

Recall: what shape does a token's vector keep through all N blocks, and what turns that vector into a next-token prediction?

Nail down the shape invariant. Try to state it, then check.

Lock it in

A Transformer block is four moves: self-attention (mix), add & norm, feed-forward (rethink), add & norm. Stack it $N$ times and you have a deep model.
Self-attention is the only step that lets positions talk; the feed-forward layer processes each token independently using the same weights.
The tensor shape stays $[seq, d_{model}]$ through every block - only the final linear layer breaks it to $[seq, vocab]$ .
Decoder-only (GPT) stacks use a causal mask so each token attends only to itself and earlier tokens - never the future.

Primary source

The architecture is defined in Vaswani et al., "Attention Is All You Need" (2017) - Figure 1 is the exact block-and-stack diagram this lesson simplifies. To watch a real GPT-2 run this forward pass token by token, open the interactive Transformer Explainer.

Ask your teacher

Here's the thread that ties it all together: a Transformer is just a deep stack of the neural-network layers from Lesson 20, where each block's "mixing" step - the one that used to be a fixed weight matrix - is now self-attention. The trip you just traced is the very same forward pass, only running on a whole sequence at once. Want to see how the residual add and LayerNorm actually stabilize a stack this deep, or why the FFN is where much of the "knowledge" seems to live? Ask, and I'll build the follow-up.