From Neurons to Networks

Backpropagation

The chain rule that teaches the network

~13 minLesson 15 of 57Written by Claude Opus 4.8

The chain rule, run in reverse: one backward sweep hands every weight in the network its exact share of the blame.

Gradient descent already told us the plan: nudge each weight against its gradient and the loss goes down. It left one enormous question open - with millions of tangled weights, how do you get the gradient for every one of them? Backpropagation is the answer, and it is not magic. It is bookkeeping: run the chain rule backward through the network's computation, and each weight's contribution to the error falls out of a single, orderly sweep.

The idea in one line

A forward pass computes the answer; the error at the end is one number. Backprop pushes that error backward along the same wires it flowed forward through, and at every junction asks: "how much of this mistake is yours?" The answer at each weight is exactly its gradient - the number gradient descent needs to know which way to nudge.

Everything is a computational graph

Before we can differentiate a network, we rewrite it as a : break the whole calculation into tiny elementary operations - one multiply, one add, one square - each a node with inputs and one output. Even a giant LLM is just an astronomically large version of this. Our toy example predicts a number from inputs $a, b, c$ and measures how far off it lands from a target $t$ :

The expression, node by node

u = a \cdot b, p = u + c, L = (p - t)^{2} .

Here $p$ is the prediction and $L$ is the loss - a single number for "how wrong." Each node's local derivative (how its output reacts to a tiny wiggle of one input) is elementary calculus:

\frac{\partial ( a \cdot b )}{\partial a} = b, \frac{\partial ( u + c )}{\partial u} = 1, \frac{\partial ( p - t ) ^{2}}{\partial p} = 2 (p - t) .

The chain rule (Lesson 14) is the glue. To learn how the loss depends on a buried input like $a$ , you multiply the local derivatives along the path from $a$ to $L$ :

The chain rule, spelled out

\frac{\partial L}{\partial a} = loss \to p \frac{\partial L}{\partial p} \cdot p \to u \frac{\partial p}{\partial u} \cdot u \to a \frac{\partial u}{\partial a} .

Backprop's trick is to compute these right to left and reuse the shared front part. The quantity $\frac{\partial L}{\partial u}$ is needed by both $a$ and $b$ - so we compute it once, cache it, and hand it down. That reuse is why one backward sweep is cheap even for a trillion weights.

Forward, then backward - step through it

Press ▶ Forward to fill every node's value left to right, then ◀ Backward to watch the gradient flow right to left, one edge at a time - each edge multiplying in its local derivative (the chain rule, live). Or walk it yourself with Next. Then drag a slider: the graph and the gradient readouts recompute instantly, so you can see exactly which gradients each input controls.

Micrograd-style backprop. Run the forward pass to fill every value, then the backward pass to flow the gradient right to left. Drag the sliders and every number recomputes live.

input

-3

input

a·b

u+c

(p−t)²

value (forward)gradient (backward)× local derivative on each edge

a = 2b = -3c = 10target t = 1 (fixed)

Forward pass

Forward pass. The inputs are known: a = 2, b = -3, c = 10. Now fill every node left to right.

prediction p4

loss L9

∂L / ∂a-18

∂L / ∂b12

∂L / ∂c6

Two rules run the whole show

Watch the backward pass and you only ever see two moves. A plus node has local derivative

1

, so it copies the incoming gradient unchanged to both inputs - a

+

is a gradient distributor. A times node routes the gradient to each input scaled by the other input - that is why

\partial L / \partial a = \partial L / \partial u \cdot b

. Every network, however deep, is built from junctions like these.

The same thing, in code

This is exactly what a framework does automatically, but written by hand it fits in a dozen lines. Notice the shape: forward first (top to bottom), then backward (bottom to top), each gradient built from the one just above it.

# inputs
a, b, c, t = 2.0, -3.0, 10.0, 1.0

# --- forward pass: fill every value left to right ---
u = a * b            # -6.0
p = u + c            #  4.0   (prediction)
L = (p - t) ** 2       #  9.0   (loss)

# --- backward pass: reverse-mode autodiff ---
dL = 1.0              # seed: dL/dL
dp = dL * 2 * (p - t)  #  6.0   through the square
du = dp * 1          #  6.0   + passes gradient through
dc = dp * 1          #  6.0
da = du * b          # -18.0  * swaps in the OTHER input
db = du * a          # 12.0

Forward must finish first

The backward pass needs values the forward pass produced - the times node's gradient uses

b

, the square's gradient uses

p

. That is the whole reason for the two-phase structure: compute and cache every node's value on the way forward, then reuse those cached values on the way back. Skip the caching and you would recompute the forward pass over and over - turning a cheap sweep into an expensive one.

Check yourself

When the gradient reaches an add node p = u + c, each input receives:

Why does backpropagation visit the graph nodes in reverse order?

Recall: what are backprop's two passes, and what does each one compute?

Nail down the two-phase structure. Try to state it, then check.

Lock it in

A network is a computational graph: tiny multiply / add / square nodes, each with a simple local derivative.
Backprop is the chain rule run right to left, reusing and caching shared partials so one sweep prices out every weight.
A plus node distributes the gradient unchanged; a times node scales it by the other input.
Forward first (cache every value), then backward, because the backward pass reads those cached values.

Primary source

3Blue1Brown's Backpropagation calculus builds the chain-rule intuition frame by frame. To see this exact node-by-node engine implemented from scratch in Python, work through Andrej Karpathy's Neural Networks: Zero to Hero and its micrograd - the demo above is a miniature of it.

Ask your teacher

Here is the connection to sit with: this exact algorithm trains every LLM. Backprop is nothing more than reverse-mode automatic differentiation - visiting the nodes of the computational graph in reverse topological order (the payoff of C14, once you know topological sort). Do a forward sweep to fill values, then one reverse sweep, and a trillion weights all receive their gradient together. Ask me how frameworks build the graph automatically, why reverse mode beats forward mode when there is one loss and many weights, or how vanishing gradients threaten the very deepest chains.