The attention mechanism

Instead of cramming a whole sentence through one fixed vector, let every word look back at every other and keep a weighted blend of what actually matters.

Early translation models read an entire source sentence, squeezed it into one fixed-length vector, and then tried to write the translation out of that single summary. It worked for short phrases and fell apart on long ones - the vector was a bottleneck, a funnel everything had to pass through. Attention removes the funnel. It lets each output position reach back and look at every input position directly, decide how relevant each one is, and take a weighted blend of them. That one idea - score, normalise, blend - is the engine inside every modern language model.

The bottleneck it fixes

A sequence-to-sequence model without attention has a hard job: an encoder compresses "the cat that the dog next door had chased all afternoon finally climbed the tree" into a single vector of, say, 512 numbers, and a decoder must reconstruct the meaning from only that. The longer the sentence, the more gets lost - earlier words get overwritten by later ones. Attention lets the decoder skip the summary and consult the original words on demand: when it is about to translate "climbed," it can look straight back at "cat" and "tree," wherever they sit in the sentence.

Three steps: score, softmax, blend

Give every position three roles. A query ${\mathbf{q}}_i$ is what position $i$ is looking for. A key ${\mathbf{k}}_j$ advertises what position $j$ offers. A value ${\mathbf{v}}_j$ is the content position $j$ will hand over if chosen. (Lesson 32 shows how $\mathbf{q},\mathbf{k},\mathbf{v}$ are actually produced; here we just need their shapes.)

Step 1 - score. How well does query $i$ match key $j$? Use the dot product from Lesson 9 - big when the two vectors point the same way - divided by $\sqrt{d}$ to keep the numbers from blowing up as the dimension $d$ grows:

Step 2 - softmax. The raw scores can be any size, positive or negative. Exponentiate each one (forcing it positive) and divide by the row's total, so the weights for one query are all in $[0,1]$ and sum to exactly $1$ - a genuine probability distribution over the keys:

Step 3 - blend. The output for position $i$, its context vector, is the weighted sum of every value, using those attention weights as the mixing proportions:

Do this for all $n$ positions and you have filled an $n\times n$ grid of weights ${\alpha }_{ij}$ - the alignment matrix. Row $i$ says where word $i$ looked; it always sums to 1. Play with it below.

The three steps in NumPy

import numpy as np

def softmax(z, T=1.0):
    z = z / T
    z = z - z.max()             # subtract max: numerically stable, same result
    e = np.exp(z)
    return e / e.sum()          # nonnegative, sums to 1

# Q, K, V come from Lesson 32; d is the key/value dimension.
scores  = Q @ K.T / np.sqrt(d)   # (n, n): every query dotted with every key
weights = np.stack([softmax(r) for r in scores])  # softmax along each row
context = weights @ V             # (n, d): each row is a weighted blend of values

# one position alone - the verb "saw" looking back at the 5 words
row     = np.array([1., 5., 3., 1., 6.])   # saw vs [the, cat, saw, the, bird]
w       = softmax(row, T=1.0)              # [.005, .257, .035, .005, .699]
c_saw   = w @ V                            # ~70% bird + 26% cat + a little of the rest

Check yourself