Tensors, broadcasting, and autograd

What every deep-learning framework quietly does for you: n-dimensional arrays, the broadcasting rules that stretch shapes to fit, and an engine that hands you every gradient for free.

Open any PyTorch or NumPy program and you will find just three ideas doing all the work. Data lives in tensors - n-dimensional arrays. Operations between tensors of different shapes are made to line up by broadcasting. And every operation is quietly recorded so an autograd engine can replay it backward and compute gradients. Two of these you already half-know; the third - broadcasting - is where the majority of real bugs actually live. Let's make all three concrete.

Tensors: one array type for everything

A tensor is the generalisation of the containers you already met: a scalar is a 0-D tensor, a vector a 1-D tensor, a matrix a 2-D tensor (Lesson 13), and beyond that we just keep adding axes. Its shape is the tuple of sizes along each axis, and its rank is how many axes it has. A batch of $32$ RGB images at $224\times 224$ is one tensor of shape $(32,3,224,224)$ - rank 4 - and the whole point of a framework is to run one operation across that entire block at once, on a GPU, instead of looping.

Axes are read left-to-right but the rules that follow always align from the right, so it pays to think of the last axis as the innermost, fastest-moving one. Get a shape wrong by a single axis and nothing crashes politely - you get silently wrong numbers. That is exactly the failure broadcasting both enables and hides.

Broadcasting: how shapes stretch to fit

You constantly want to combine tensors of different shapes - add a length-$d$ bias vector to a whole batch of activations, multiply a matrix by a per-row scale. Broadcasting is the deterministic rule that decides whether that is legal and what the result shape is. It is identical across NumPy, PyTorch, and JAX.

Now play with it. Resize the two tensors below and watch the rule decide: green means an axis lined up, an amber highlight marks a size-$1$ axis being stretched to fill the result, and red is the shape error you will meet in real code.

Autograd: the graph that differentiates itself

Every tensor operation is a node in a computational graph (Lesson 22): the forward pass fills each node's value; the framework simultaneously records how that node was produced. To get gradients it walks the graph backward in reverse order, multiplying in each node's local derivative - reverse-mode automatic differentiation. Below is a single neuron, $L=(x\cdot w+b-y{)}^2$, unrolled into its graph. Step forward to fill the values, then keep stepping to watch the gradient flow back to every input at once.

The same engine, in a dozen lines

What makes autograd feel like magic is almost embarrassingly small. Each operation returns a node that remembers how to push gradient back to its inputs. Build the graph forward, seed the tail with $1$, then call those closures in reverse - this is the beating heart of Karpathy's micrograd.

# each op returns a node that remembers its local derivative
def mul(a, b):
    out = node(a.data * b.data)
    def _backward():                # local: d(a*b)/da = b,  d(a*b)/db = a
        a.grad += b.data * out.grad
        b.grad += a.data * out.grad
    out._backward = _backward
    return out

# ... build the graph forward, collecting nodes in order, then:
loss.grad = 1.0                     # seed the tail: dL/dL
for n in reversed(graph):          # reverse topological order
    n._backward()                    # each node hands gradient to its inputs

Check yourself