Linear Algebra

Dot product and cosine similarity

Measuring alignment, length, and similarity

~11 minLesson 4 of 19Written by Claude Opus 4.8

One number decides whether two arrows point the same way, sit at right angles, or pull dead against each other. Divide out the two lengths and it becomes the exact yardstick a language model uses to decide what means the same thing.

The asks a single question: how much of one vector points along the other? It is largest when the two are parallel, exactly zero when they are perpendicular, and negative when they oppose. Reduce "similar" to geometry and this is what you get.

The dot product, two ways

The dot product takes two vectors and returns a single scalar. There are two recipes for it, and the small miracle of linear algebra is that they always agree. The algebraic one multiplies matching components and adds them up. The geometric one multiplies the two lengths by the cosine of the angle between them.

a \cdot b = i \sum a_{i} b_{i} = ∣ a ∣ ∣ b ∣ cos θ Read left to right it is arithmetic; read right to left it is an angle. The lesson lives in the equals sign.

Length, right angles, and shadows

A vector dotted with itself gives its length squared, which defines the norm (magnitude):

∣ a ∣ = a \cdot a

. Set

cos θ = 0

and two nonzero vectors are orthogonal (perpendicular) exactly when

a \cdot b = 0

. And the piece of

b

that lies along

a

, its projection or shadow, has signed length

\frac{a \cdot b}{∣ a ∣} = ∣ b ∣ cos θ

Turn the dial

Rotate and stretch $b$ below. $a$ is fixed along the horizontal. Watch the readouts: the dot product and $cos θ$ swing from $+ 1$ (aligned) through $0$ (perpendicular) to $- 1$ (opposite). The shaded bar along $a$ is $b$ 's projection, the shadow $∣ b ∣ cos θ$ .

Vector a is fixed. Drag b's tip on the canvas to rotate and stretch it, or use the sliders.

rotate b: 40°length of b: 3.0

a·b = 9.19|a| = 4.00|b| = 3.00cos θ = 0.77θ = 40°projection = 2.30

-1 · opposite0 · orthogonal+1 · aligned

These two vectors are loosely similar.

Notice what moves the numbers

Stretch

b

straight outward without turning it: the dot product grows (it scales with length), but

cos θ

does not budge. Now rotate it:

cos θ

swings across its whole range. The raw dot product tangles up length and angle; cosine similarity keeps only the angle. That is the distinction the next section cashes in.

Similarity is just the angle

To compare direction alone, strip the lengths out of the dot product by dividing by both magnitudes. What survives is the cosine of the angle, .

cos θ = \frac{a \cdot b}{∣ a ∣ ∣ b ∣} \in [- 1, 1]

Because it is bounded in $[- 1, 1]$ and blind to magnitude, it is the natural way to score "how alike are these two directions?". A long vector and a short vector that point the same way score a perfect $1$ . This is precisely how a model compares embeddings, the vectors it stores for words and documents. Below, drag the query direction and the words re-rank live by cosine similarity to wherever you point it.

Drag the query dot anywhere and watch the words re-rank by cosine similarity. Drag it far out - the ranking does not budge, only its angle matters.

1. cat0.99
2. dog0.99
3. bus0.31
4. car0.08
5. king-0.11
6. queen-0.31

query direction: 63°query length: 2.2

Length is irrelevant, angle is everything

Swing the query far out toward the edge and the ranking does not change; only its direction matters. Point it toward the animals and cat and dog rise to the top; point it toward the vehicles and car and bus win. Semantic neighbors are simply vectors pointing the same way. Similarity is geometry.

The whole computation is two lines.

import numpy as np

a = np.array([4.0, 0.0])
b = np.array([3.0, 2.0])

dot = a @ b                                            # 12.0 = 4*3 + 0*2
cos = dot / (np.linalg.norm(a) * np.linalg.norm(b))   # 0.832 = pure angle

The dot product is a sum of products; cosine divides the lengths out.

Check yourself

Two nonzero vectors have a dot product of exactly zero. The vectors must be:

Cosine similarity divides the dot product by both magnitudes. The main effect is to:

Give both forms of a·b, and say what cosine similarity does that the raw dot product does not.

One recipe is arithmetic, the other is geometry. Try to state it, then check.

Lock it in

The dot product is sum of products, and also length times length times cos.
Zero dot product means orthogonal; positive means aligned, negative opposed.
Divide out both lengths to get cosine similarity: pure direction in [-1, 1].
This is how attention scores tokens and how RAG ranks documents.

Primary source

For the visual intuition of the dot product as projection, watch 3Blue1Brown, Essence of Linear Algebra. For a figure-driven derivation, work through the dot product chapter of Immersive Linear Algebra.

Where this reappears

This exact operation is how self-attention scores tokens in a transformer, a query dotted against every key, and how retrieval-augmented generation finds relevant documents: cosine similarity between the question's embedding and each stored chunk.