Advanced Linear Algebra

Eigenvectors and eigenvalues

The directions a transformation only scales

~12 minLesson 7 of 19Written by Claude Opus 4.8

A matrix takes the whole plane and warps it - most arrows come out pointing somewhere new. But hidden inside every transformation are a handful of special directions that refuse to turn. Feed one in, and what comes out points the exact same way - only longer or shorter. Those unbending directions are its eigenvectors; the amount each one stretches is its eigenvalue. Find them and you've found the transformation's own private set of axes - the frame in which its tangled behaviour becomes pure, simple scaling.

The one-line idea

Spin an arrow all the way around and watch its image under the matrix. Almost always the image is rotated off the original - knocked off its line. But at a few magic angles the image lands right back on the same line, just scaled. That input direction is an eigenvector; the scale factor is its eigenvalue

λ

. Eigen means "own" in German - these are the transformation's own directions.

One equation says it all

Everything above collapses into a single, deceptively small equation. For a square matrix $A$ , a nonzero vector $v$ is an eigenvector, with eigenvalue $λ$ , exactly when:

A v = λ v .

Read it slowly. The left side, $A v$ , is the full transformation - matrix multiply, the machinery that normally rotates, shears, and stretches. The right side, $λ v$ , is nothing but scalar times the same vector - pure stretching, no turning. The equation demands they be equal: for this one special $v$ , the whole matrix does no more than a single number would. All the complicated off-axis mixing cancels out along that direction. If $λ = 3$ the vector triples; if $λ = 1$ it sits perfectly still; if $λ$ is negative it flips through the origin.

Go on an eigen-hunt

Below, a fixed matrix $A = [2112]$ warps the plane. The faint dashed circle is every possible unit input direction; the amber loop is where that circle lands - an ellipse, because the matrix stretches some directions more than others. Spin the input arrow (drag it, or use the slider) and watch its image. The rotation meter shows how far the image is turned off the input's line. Hunt for the angles where it drops to zero: there, input and output share a line, the arrow snaps and glows, and you've caught an eigenvector with its $λ$ .

Eigen-hunt: drag the arrow, or spin theta. The dashed circle is every unit input; the amber loop is where it lands. Hunt the angles where the image snaps back onto the input line.

input angle θ = 20°

input v = (0.94, 0.34)image Av = (2.22, 1.62)|Av| = 2.75turn = 16°

0° · on an eigenvector90° · fully sideways

The image is turned 16° off the input's line - not an eigenvector. Keep hunting.

eigenvector #1- not found -

eigenvector #2- not found -

The "aha": the meter hits exactly zero

For this matrix the meter bottoms out at 45° and 135° - and nowhere else. At 45° the arrow triples (

λ = 3

); at 135° it doesn't move at all (

λ = 1

, the matrix leaves that whole direction frozen). Notice those two eigenvectors sit at right angles to each other, and they are precisely the long and short axes of the amber ellipse. That's no accident: a symmetric matrix always has perpendicular eigenvectors, and its eigenvectors are the axes of the ellipse it draws. Everywhere else on the circle, the image is turned off its line - no eigenvector there.

Why this is the "skeleton" of a transformation

Once you know the eigenvectors, you understand the whole matrix. Line them up as the columns of a matrix $Q$ and put the eigenvalues on the diagonal of $Λ$ , and the transformation factors into three clean moves:

A = Q Λ Q^{- 1} .

This is diagonalization, and it reads right-to-left as a story: $Q^{- 1}$ rotates the world so the eigenvectors become the ordinary axes, $Λ$ does nothing but scale each axis by its eigenvalue, and $Q$ rotates back. In its own eigenbasis, a matrix is just a set of independent stretch knobs - no rotation, no shear, no coupling. That is why eigenvalues govern stability: repeatedly applying $A$ (as dynamical systems, Markov chains, and training loops all do) multiplies each eigendirection by $λ^{k}$ . Eigenvalues with $∣ λ ∣ > 1$ blow up, $∣ λ ∣ < 1$ decay to zero, and $∣ λ ∣ = 1$ persist forever. The largest eigenvalue decides the long-run fate.

How to actually find them: the characteristic equation

Rewrite $A v = λ v$ as $(A - λ I) v = 0$ . For a nonzero $v$ to solve this, the matrix $A - λ I$ must squash space flat - it must be singular - which means its determinant is zero:

det (A - λ I) = 0.

For our $A = [2112]$ that is $det [2 - λ 1 1 2 - λ] = (2 - λ)^{2} - 1 = 0$ , so $2 - λ = \pm 1$ , giving $λ = 3$ and $λ = 1$ - exactly the two stretch factors the demo snapped onto. Plug each $λ$ back in and solve $(A - λ I) v = 0$ to recover its eigenvector.

Why PCA hunts these exact directions

Here is the payoff that reaches all the way into machine learning. Take a cloud of data points and compute its covariance matrix $C$ - a symmetric matrix whose entries measure how the features vary together. Its eigenvectors are the directions along which the data spreads out the most, and each eigenvalue is the amount of variance captured in that direction. The top eigenvector points along the cloud's longest axis; the next, perpendicular to it, along the second-longest; and so on. That is exactly Principal Component Analysis: keep the few eigenvectors with the largest eigenvalues and you've compressed high-dimensional data down to the directions that actually carry information, with provably minimal loss. The special axes of a transformation and the meaningful axes of a dataset turn out to be the same idea.

import numpy as np

A = np.array([[2, 1], [1, 2]])
vals, vecs = np.linalg.eig(A)
# vals  -> [3., 1.]                 the eigenvalues (stretch factors)
# vecs  -> columns are unit eigenvectors: [.707, .707] and [-.707, .707]

# PCA is exactly this, run on the data's covariance matrix:
C = np.cov(data, rowvar=False)   # symmetric covariance
variances, axes = np.linalg.eigh(C)  # eigh: for symmetric matrices
# largest 'variances' -> the principal component directions in 'axes'

Eigen-decomposition and PCA in a handful of lines

Check yourself

A nonzero vector v is an eigenvector of A exactly when Av is:

In the eigen-hunt, the rotation meter reads exactly zero when:

Recall: state the eigen-equation, say what v and λ mean geometrically, and name the ML method that finds the eigenvectors of a covariance matrix.

The equation, its geometry, and its ML payoff. Try to state it, then check.

Lock it in

An eigenvector is a direction a matrix only stretches; the eigenvalue is how much, all captured by Av = λv.
The rotation meter reads zero exactly on an eigenvector, where Av lands back on v's line.
Diagonalization A = Q Λ Q^-1 turns a matrix into pure per-axis scaling, so eigenvalues govern stability under repeated application.
PCA is this idea on data: the eigenvectors of the covariance matrix are the perpendicular directions of maximum variance.

Primary source

For the definitive visual intuition - watching space warp and the eigenvectors emerge as the axes that don't turn - see 3Blue1Brown's Essence of Linear Algebra (the eigenvectors chapter). To grab the transformation and its eigenvectors with your own mouse, play with Explained Visually's interactive eigenvectors & eigenvalues, and its companion page on principal component analysis.

Ask your teacher

Eigenvectors are the bridge from linear algebra to embeddings. PCA finds the eigenvectors of the data's covariance - the directions of maximum variance - and those are the classical, hand-computed cousin of the learned embedding space you'll meet in Lesson 12: both hunt for a small set of meaningful axes hidden in high-dimensional data. Ask me how PCA relates to the SVD, why symmetric matrices are guaranteed real eigenvalues, or how power iteration finds the top eigenvector without ever solving a polynomial.