Matrices and matrix multiplication

A matrix looks like a spreadsheet, a rectangle of numbers. But it is secretly a machine: feed it a vector and it hands you back a transformed vector. Matrix multiplication chains those machines, and it is the beating heart of every neural network.

When people say a language model is "just a lot of matmuls," this lesson is the "just." A matrix stores a transformation as a grid: each column tells you where one input direction lands. Multiplying a matrix by a vector mixes the columns; multiplying two matrices takes a dot product of every row against every column. Two pictures, one operation.

A grid of rows and columns

A matrix with $m$ rows and $n$ columns is an $m\times n$ matrix; you read the shape as "rows by columns," always. The entry in row $i$, column $j$ is written $A_{ij}$. Here are the two matrices we will work with for the whole lesson.

You can slice a matrix two ways. Its rows are horizontal vectors; its columns are vertical vectors. That column view is the key to everything below.

Matrix times a vector: mix the columns

The most useful way to read $A\mathbf{x}$ is not "row by row." It is: use the entries of $\mathbf{x}$ as weights, and add up the columns of $A$. Each input coordinate $x_j$ says "how much of column $j$" to include.

So $A\mathbf{x}$ is a linear combination of $A$'s columns. Every output the matrix can ever produce is some weighted sum of those columns, a set called the column space. Turn the weights below and watch three columns get scaled and stacked tip-to-tail into a single result.

Matrix times a matrix: rows meet columns

To multiply $A$ by $B$, march through the output grid one cell at a time. The entry $C_{ij}$ is the dot product of row $i$ of $A$ with column $j$ of $B$.

This only type-checks when the shapes line up: an $(m\times n)$ times an $(n\times p)$ gives an $(m\times p)$. The two inner numbers must match; the outer numbers become the answer's shape. Our $A$ is $2\times 3$ and $B$ is $3\times 2$, so $C=AB$ is $2\times 2$. Step through it, cell by cell.

The same three products in NumPy.

import numpy as np

A = np.array([[2, 1, 3], [1, 4, 2]])    # shape (2, 3)
B = np.array([[1, 2], [3, 1], [2, 1]])  # shape (3, 2)

C = A @ B          # (2, 2) -> [[11, 8], [17, 8]]

x = np.array([1, 1, 1])
y = A @ x          # (2,) -> [6, 7] = a1 + a2 + a3, the sum of A's columns

I = np.eye(3)             # 3x3 identity
(A @ I == A).all()        # True: the identity leaves A untouched

Check yourself