Tokens, Embeddings, and Attention

Word embeddings

Turning words into vectors that carry meaning

~12 minLesson 24 of 57Written by Claude Opus 4.8

A computer has no idea what "king" means. So we do something audacious: we represent each word as a list of numbers - a point in a high-dimensional space - and we learn those numbers so that words used in similar ways land near each other. Once meaning is a set of coordinates, similarity is an angle, and analogies become addition and subtraction. This is a preview built directly on Vectors (L8) and the Dot Product (L9).

One-hot: every word a lonely island

The naive encoding gives each word its own slot. With a vocabulary of $V$ words, "king" is a vector of all zeros with a single $1$ in its slot - a vector living in ${0, 1}^{V}$ :

the0

cat0

king1

queen0

run0

…0

It is honest but useless for meaning. Every pair of distinct one-hot vectors is orthogonal: their dot product is exactly $0$ . So the geometry insists that "king" is precisely as related to "queen" as it is to "banana" - no notion of similarity survives. And with $V$ often above 50,000, the vectors are enormous and almost entirely zeros.

The distributional hypothesis

The fix comes from a 1957 slogan by linguist J.R. Firth: "You shall know a word by the company it keeps." Words that appear in the same contexts - the handful of words to their left and right, the context window - tend to mean similar things. "The ___ ruled the kingdom" is filled by king and queen alike. So we throw away the one-hot slots and instead learn a short, dense vector

king \in R^{d}

(real numbers,

d \approx 300 ≪ V

) whose coordinates are tuned so that words sharing company end up as neighbours.

Nobody hands the model the meaning of each coordinate; the numbers are learned by gradient descent so that the geometry pays off. The remarkable result: individual directions in that space line up with human-recognisable concepts - a "gender" direction, a "royalty" direction, a "plural" direction. That is what makes the toy below possible.

Meaning becomes geometry

Below is a hand-built 2-D embedding space. The x-axis behaves like a gender dimension and the y-axis like a royalty / seniority dimension. The dashed cross marks the neutral centre of the space. Real embeddings have 300 dimensions and are learned, not placed - but the geometry works exactly like this.

Analogy mode: pick A - B + C and watch the relationship arrow copy onto C. Neighbours mode: pick a word (or click one on the map) to rank its nearest neighbours by cosine.

Analogy mode: pick A - B + C below.

A− B+ C

king − man + woman → queen (cos 1)

ABCresult / nearest word

The magic moment

Leave the analogy on its default, king − man + woman. The solid arrow from man to king is the "gain royalty" step. Copy that exact same arrow, start it at woman, and where do you land? On queen - dead on. The relationship king : man is the same direction and length as queen : woman, so the four words form a parallelogram. That is not a coincidence baked into this toy; it falls out of real trained embeddings, and it is the first sign that these vectors have captured something about meaning.

The arithmetic of meaning

Two operations do all the work, and you already met both in Lessons 8 and 9.

Analogy = vector arithmetic. Subtracting two word vectors isolates a relationship; adding it elsewhere applies that relationship:

The two formulas

The famous analogy is just addition and subtraction of vectors, then a search for the closest real word:

"royalty" direction king - man + woman \approx queen

Similarity = angle, not distance. To decide which real word a result is closest to, we use cosine similarity - the cosine of the angle between two vectors (Lesson 9). It ignores length and keeps only direction, so it reads as "how aligned in meaning are these?":

cos (θ) = \frac{a \cdot b}{∥ a ∥ ∥ b ∥} \in [- 1, 1]

A value near $1$ means "point the same way" (synonyms), near $0$ means "unrelated" (orthogonal, as the one-hot vectors were), and near $- 1$ means "opposite". Switch the demo to Neighbours and click king: its top match is prince, because their vectors point in nearly the same direction from the centre.

# E is the embedding matrix: one learned row per word  (V rows, d cols)
# "looking up" a word is literally indexing into a row by its token id
king  = E[ vocab["king"] ]
man   = E[ vocab["man"] ]
woman = E[ vocab["woman"] ]

def cosine(a, b):                 # similarity = angle, not distance
    return (a @ b) / (norm(a) * norm(b))

target  = king - man + woman            # vector arithmetic isolates + applies "royalty"
nearest = max(vocab, key=lambda w: cosine(target, E[vocab[w]]))
# nearest -> "queen"

The whole trick, in a few lines of NumPy

Check yourself

In an embedding space, cosine similarity between two word vectors measures their:

Reading king - man + woman = queen, the vector (king - man) captures the idea of:

Recall: why can a model do arithmetic like king - man + woman and land on queen?

Try to state it, then check.

Lock it in

One-hot vectors are orthogonal and enormous (one slot per vocabulary word), so they carry no similarity - "king" is as unrelated to "queen" as to "banana".
The distributional hypothesis - know a word by the company it keeps - lets gradient descent learn short dense vectors ( $d \approx 300$ ) where words sharing contexts land near each other.
Similarity is angle, not distance: cosine similarity reads meaning off direction and ignores length, landing in $[- 1, 1]$ .
Consistent relationships become consistent directions, so analogies are plain vector arithmetic: $king - man + woman \approx queen$ .
These learned vectors are the embedding matrix - the first layer of every LLM, where a token id simply indexes its row.

Primary source

The clearest visual walkthrough is Jay Alammar's The Illustrated Word2vec - start there for the intuition and the training story. Then go play with real 300-dimensional embeddings in the TensorFlow Embedding Projector: type a word, watch its true nearest neighbours appear, and rotate the projected cloud to see the structure for yourself.

Ask your teacher

Here is the connection that makes this a preview of every LLM: these vectors are the first layer of the model - the embedding matrix. A token id simply looks up its row, exactly like the code above. From there, attention compares those vectors against each other using the very dot product from Lesson 9 to decide which words should influence which. Want me to unpack how the embeddings are actually trained (skip-gram vs. CBOW), or how positional information gets added on top? Ask and I'll build the follow-up.