Tokens, Embeddings, and Attention

Self-attention: query, key, value

The three projections that let tokens talk

~12 minLesson 26 of 57Written by Claude Opus 4.8

Every token asks a question, every token offers an answer, and attention is the soft, weighted lookup that connects them - the single mechanism the whole transformer is built around.

Last lesson gave us the idea of attention: let each token look at every other token and pull in what's relevant. Now we make it precise. Every token in a sequence sends out a Query ("what am I looking for?"), advertises a Key ("here's what I'm about"), and holds a Value ("here's the content I'll hand over"). Attention dots each query against every key to score relevance, softmaxes those scores into weights, and returns a weighted blend of the values. That's it. This is scaled dot-product attention - and it is self-attention because the queries, keys, and values all come from the same sequence attending to itself.

The one-line idea

Attention is a soft dictionary lookup. A normal dictionary matches your key exactly and returns one value. Here, a token's query is compared against every key by dot product, the matches are turned into fractional weights that sum to 1, and the answer is a blended average of all the values - weighted toward whichever tokens matched best.

Three projections of the same token

Each token arrives as a single embedding vector $x_{i}$ (Lesson 12). Self-attention doesn't use that vector raw - it first makes three different views of it, using three learned weight matrices $W^{Q}, W^{K}, W^{V}$ that the network trains by gradient descent:

q_{i} = x_{i} W^{Q}, k_{i} = x_{i} W^{K}, v_{i} = x_{i} W^{V} .

Same token, three roles. The query $q_{i}$ is what token $i$ is searching for; the key $k_{i}$ is how token $i$ describes itself so others can find it; the value $v_{i}$ is the information token $i$ contributes once it's been attended to. Because $W^{Q}, W^{K}, W^{V}$ are learned, the model discovers on its own what "relevant" should mean for the task. Stack all the per-token vectors as rows and you get the matrices $Q$ , $K$ , $V$ .

Scaled dot-product attention

With $Q$ , $K$ , $V$ in hand, the entire operation is one compact formula:

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V .

Read it inside-out, and every piece is something you already know. $Q K^{⊤}$ is a matrix multiply: row $i$ (a query) dotted against every column (a key) gives a full grid of similarity scores - entry $(i, j)$ is how much token $i$ should attend to token $j$ . Dividing by $d_{k}$ (with $d_{k}$ the key/query dimension) keeps those scores from growing too large. The softmax turns each row of scores into a probability distribution that sums to 1. Multiplying by $V$ then takes, for each token, the weighted average of all the value vectors. Walk one token through all four stages below.

Q/K/V attention calculatorpick a token - step the 4 stages

Attend from token:

Query for sat = [1, 2]

Key K

Value V

Q·Kᵀ

÷ √dₖ

softmax weight w

the

cat

sat

mat

Meet the three vectors each token carries: a Query, a Key, and a Value.

Query - what token "sat" is looking for

Key - the label each token advertises to be found by

Value - the content each token hands back once attended to

Vectors are 2-D here (d_k=2) to keep the arithmetic visible. Pick a different token to watch its query land on a different neighbour.

The aha: the output is a blend, not a pick

Attention never "chooses" one token. For the query from sat, the softmax spreads weight 0.54 on cat, 0.27 on mat, 0.13 on sat, 0.06 on the - and the output is that exact mixture of their value vectors. Because cat's key matched the query best, its value dominates the blend, but every token still contributes a little. Change what a token is looking for (its query) and the whole distribution shifts. That soft, differentiable blending is why attention can be trained end-to-end.

Why divide by sqrt(d_k)?

If a query and key each have $d_{k}$ independent components with roughly unit variance, their dot product $q \cdot k = \sum_{t = 1}^{d_{k}} q_{t} k_{t}$ is a sum of $d_{k}$ such terms, so its variance grows like $d_{k}$ and its typical size like $d_{k}$ . For a real model with $d_{k} = 64$ , raw scores swing by $\pm 8$ or more. Feed numbers that large into softmax and it saturates: one weight goes to nearly 1, the rest to nearly 0, and the gradient there is almost flat - learning stalls. Dividing by $d_{k}$ rescales the scores back to unit size, keeping softmax in its responsive, trainable region. It's a variance fix, not a cosmetic one.

import numpy as np

def softmax(x, axis=-1):
    x = x - x.max(axis=axis, keepdims=True)   # subtract max for numerical stability
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

def attention(Q, K, V):
    d_k    = K.shape[-1]                 # dimension of each key / query
    scores = Q @ K.T / np.sqrt(d_k)     # (n, n) similarity grid, scaled
    weights = softmax(scores, axis=-1)  # each ROW sums to 1 across the keys
    return weights @ V                   # (n, d_v) weighted blend of the values

Scaled dot-product attention in NumPy

Check yourself

In scaled dot-product attention, the softmax weights are multiplied by which vectors?

Why are the raw scores divided by sqrt(d_k) before the softmax?

Recall the full scaled dot-product attention pipeline: the formula and what each of the four stages does.

Nail down the core formula and its four stages. Try to state it, then check.

Lock it in

Every token gets three views via learned matrices: Query (what it looks for), Key (how it advertises itself), Value (what it hands over).
Attention scores are dot products of queries against keys, scaled by $d_{k}$ to prevent softmax saturation, then softmaxed into weights that sum to 1.
The output for each token is the weighted average of all value vectors - a soft blend, never a hard pick.
The entire operation is differentiable end-to-end, so the model learns what "relevant" means for the task through gradient descent.

Primary source

For the definitive visual walkthrough - the same query/key/value story, illustrated step by step with matrices - read Jay Alammar's The Illustrated Transformer. For the original definition of scaled dot-product attention and the $d_{k}$ argument in its own words, see the paper that introduced it, Vaswani et al., "Attention Is All You Need" (2017), section 3.2.1.

Ask your teacher

Notice that $Q K^{⊤}$ is exactly the matrix multiplication from Lesson 13 - the very same cell-by-cell row·column dot products you stepped through there, only now each cell is scoring how much one token should attend to another (connections C2 and C7). Nothing new in the arithmetic; the whole leap is interpreting that grid as attention. Ask me why we need three separate projections instead of dotting the embeddings directly, why attention has to be masked for language models so a token can't peek at the future, or how this single head becomes the many parallel heads of the next lesson.