The Transformer

Positional encodings

How the model knows word order without recurrence

~10 minLesson 30 of 57Written by Claude Opus 4.8

Self-attention reads a bag of tokens with no sense of order - so before it runs, we stamp each token with a signature that says exactly where it sits.

Attention has a secret weakness: it is order-blind. Mathematically it is an operation on a set, so shuffling the words of a sentence leaves its output completely unchanged. To a raw attention layer, "dog bites man" and "man bites dog" are the same three vectors in a different pile - identical. Positional encodings are the fix: a small vector added to each token that quietly injects where it is, turning a pile of words back into a sequence.

The picture to hold in your head

Imagine handing three name-tags - "dog", "bites", "man" - to someone who can read them but was never told the order you laid them out. They see a set, not a line. To restore the order, you clip a second tag onto each: "position 0", "position 1", "position 2". Now the same reader can tell the two sentences apart. Positional encoding is that second tag, written not as a number but as a rich vector of sine waves the model can actually compute with.

Why attention can't feel order

Self-attention builds each token's new representation as a weighted average of every token's value vector, where the weights come only from how well queries match keys:

Attention is permutation-equivariant

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}}) V

Nothing in this formula mentions an index. The weight linking two tokens depends on their content, never on their positions. So if you permute the input rows by any shuffle $P$ , the output is just the same rows shuffled the same way:

Attention (P X) = P Attention (X) .

That is the whole problem in one line. Because "dog bites man" and "man bites dog" are built from the same three tokens, they are the same set $X$ up to a permutation - so attention produces the same three output vectors for both. The meaning that lives in the order is invisible.

We fix it before attention ever runs. Each token embedding $x_{p os}$ gets a position vector $P E_{p os}$ added to it: $\tilde{x}_{p os} = x_{p os} + P E_{p os}$ . Now two identical words in different slots arrive as different vectors, and attention can finally tell them apart.

A fingerprint made of sine waves

The trick from the original Transformer is elegant: describe each position not with one number but with a vector of sine and cosine values at many different frequencies. Even dimensions get a sine, odd dimensions the matching cosine, and each pair oscillates at its own speed:

Sinusoidal positional encoding

P E_{(p os, 2 i)} = sin (\frac{p os}{1000 0 ^{2 i / d}}), P E_{(p os, 2 i + 1)} = cos (\frac{p os}{1000 0 ^{2 i / d}})

Here $p os$ is the position, $d$ is the model width, and $i$ runs over the $d /2$ frequency pairs. The denominator $1000 0^{2 i / d}$ stretches the wavelength as $i$ grows: the low-index dimensions ( $i$ small) have the shortest wavelength $2 π$ and cycle fast; the high-index dimensions crawl with wavelengths up to $2 π \cdot 10000$ . Reading down a single position gives a unique fingerprint; reading across the dimensions gives a whole spectrum of clocks.

It's a binary odometer made of waves

Think of how a car's odometer counts: the ones-digit spins fast, the tens-digit slower, the hundreds slower still - together they pin down any number. Sinusoidal encodings do the same with continuous dials. The fast, low-index dimensions distinguish neighbours; the slow, high-index dimensions capture coarse, long-range position. A bonus falls out for free: because $sin$ and $cos$ obey angle-addition, the encoding of position $p os + k$ is a fixed linear rotation of the encoding at $p os$ - so the model can learn to attend by relative offset, not just absolute slot.

See the encoding

Below is the real sinusoidal table for a tiny model ( $d = 48$ dimensions, positions $0$ - $23$ ). Every cell is one $P E$ value, colored from amber (-1) through the page tone (0) to teal (+1). Drag the position slider to spotlight a row and watch its signature; click any column to inspect that dimension's wave in the panel below. Notice the left columns strobe quickly down the rows while the right columns drift - a full ladder of frequencies, exactly as the formula promises.

Sinusoidal encoding, live drag the slider - click a column - sweep

Heatmap - rows = positions, columns = dimensions

value:-1+1outlined row = selected positionoutlined column = inspected dimension

position = 5

One dimension as a wave - position along x, value along y

inspected dimensionother dimensions (the frequency ladder)

position5

dimension0 · sin

PE value-0.959

wavelength6.3

Position 5 is the outlined row. Column 0 is a sine of wavelength ≈ 6.3 positions.

The "aha" to take away

Push the slider from 0 to 23 and watch the highlighted row's pattern change smoothly and uniquely - no two positions share a fingerprint, yet nearby positions look similar, which is exactly the smoothness a model wants. Then click a far-left column (fast wave) versus a far-right column (nearly flat): the same table encodes both fine and coarse position at once. That is the entire idea.

Learned tables and rotary twists

Sinusoids are not the only option - they were just the first. Two other schemes dominate today:

Learned positional embeddings. Instead of a fixed formula, keep a lookup table with one trainable vector per position and let gradient descent discover whatever pattern helps. BERT and the original GPT do this. It is dead simple, but it can only address positions it saw in training - feed it a longer sequence and it has no row to return.
Rotary embeddings (RoPE). The modern favourite in LLaMA-style models. Rather than adding a position vector, RoPE rotates each query and key by an angle proportional to its position. Because a dot product between two rotated vectors depends only on the difference of their angles, attention scores automatically encode relative distance, and the scheme extends gracefully to longer contexts.

import numpy as np

def sinusoidal(n_pos, d, base=10000):
    pos  = np.arange(n_pos)[:, None]        # column of positions
    i    = np.arange(d)                       # 0 .. d-1 (one per column)
    freq = 1.0 / base ** (2 * (i // 2) / d)   # one frequency per sin/cos pair
    ang  = pos * freq                          # (n_pos, d) grid of angles
    return np.where(i % 2 == 0, np.sin(ang), np.cos(ang))  # add to embeddings

The sinusoidal table in five lines of NumPy

Check yourself

Why must we add positional information to token vectors before self-attention?

In the sinusoidal scheme, which dimensions oscillate fastest as the position increases?

Recall: what property of self-attention makes positional encodings necessary, and how do sinusoidal encodings supply position?

Nail down the core reason and the core mechanism. Try to state it, then check.

Lock it in

Self-attention is permutation-equivariant - it treats tokens as an unordered set, so word order is invisible without positional encoding.
Sinusoidal encodings give each position a unique fingerprint built from sine and cosine waves at geometrically spaced frequencies - low indices cycle fast, high indices crawl.
The angle-addition property means the encoding of position $p os + k$ is a fixed rotation of the encoding at $p os$ , enabling relative-distance attention.
Learned embeddings (BERT, GPT) trade the formula for a trainable table but cannot extrapolate past training length; RoPE (LLaMA) rotates queries and keys so attention scores encode relative distance natively.

Primary source

Jay Alammar's The Illustrated Transformer has the clearest visual walk-through of the positional-encoding heatmap and how it is added to the embeddings. The scheme itself is introduced in Vaswani et al., "Attention Is All You Need" (2017) - see Section 3.5, which gives the exact sine/cosine formula reproduced above.

Ask your teacher

Sit with the tie-in: without positions, "dog bites man" and "man bites dog" are literally the same input to the model - same tokens, reshuffled, same output. Position is what makes them different sentences. Later, the causal mask plus these positions will turn the fully-connected token graph into an ordered, one-directional DAG, where each token may only look leftward. Ask me how the mask and the encodings work together, why RoPE has largely replaced additive encodings in new LLMs, and what "length extrapolation" means when a model meets positions longer than it trained on.