Calculus for Learning

Derivatives and the chain rule

Slope, sensitivity, and composing change

~12 minLesson 8 of 19Written by Claude Opus 4.8

A function tells you where you are. Its derivative tells you which way and how fast things are changing right now: the slope of the curve at a single point. That one number is what gradient descent reads to decide which way is downhill.

Zoom far enough into any smooth curve and it looks straight. The is the slope of that straight zoom-in, the tilt of the tangent line that kisses the curve at $x$ . Positive slope means rising, negative means falling, and exactly zero means you are standing on a peak or in a valley. Everything else in this lesson is a consequence of that single picture.

The slope that never stops shrinking

Average speed is easy: distance over time. But what is your speed at one instant, when no time has passed at all? The trick is to measure the slope of a line through two nearby points, a secant, and slide the second point closer and closer until the gap $h$ vanishes. The secant tips over into the tangent, and its slope stops changing. That limiting slope is the derivative.

f^{'} (x) = h \to 0 lim \frac{f ( x + h ) - f ( x )}{h}

The payoff: local linear approximation

You will rarely compute this by hand, but it buys you something priceless: near any point, the function behaves like a straight line,

f (x + Δ x) \approx f (x) + f^{'} (x) Δ x

. Take a small step

Δ x

and the output changes by about slope times step. Gradient descent is nothing but this sentence, run over and over.

Scrub the tangent

Drag the point along the curve and watch the amber tangent line pivot. Read its slope live, and see the sign flip from negative to positive as you cross the bottom of a valley. Where the tangent lies perfectly flat, the slope is zero.

point x = -1.50

f(x) = 2.25slope f'(x) = -3.00the curve is falling

Where the slope hits zero

Scrub the parabola through

x = 0

: the slope goes

- 3, - 1, 0, + 1, + 3

. The tangent lies perfectly flat exactly at the minimum. That is the tool calculus gives us for finding the best (lowest or highest) point of anything: solve

f^{'} (x) = 0

. Training a model is exactly this, hunting for where the error's slope vanishes.

Two rules you will use forever

You almost never take a limit by hand. Instead you memorize a handful of rules; two carry most of the weight in machine learning. The power rule handles any $x$ raised to a power: bring the exponent down front, then subtract one.

\frac{d}{d x} x^{n} = n x^{n - 1} ⟹ \frac{d}{d x} x^{2} = 2 x, \frac{d}{d x} x^{3} = 3 x^{2}

The handles a function inside a function, $f (g (x))$ . Its slope is the outer function's slope (measured at the inner value) times the inner function's slope.

Write it as fractions and the intermediate du cancels: rates along a chain multiply.

The gears make this physical. Gear $g$ turns $g^{'} (x)$ times for each turn of the input; gear $f$ turns $f^{'} (g)$ times for each turn of $g$ . So per turn of the input, gear $f$ turns $g^{'} (x) \times f^{'} (g)$ times, and the middle gear's size literally cancels out of the ratio, just as $d u$ cancels in the algebra.

A chain of functions x → g → f. Size the gears by teeth, spin the input, and watch the rates multiply.

gear g teeth = 6gear f teeth = 4

g'(x) = 12/6 = 2.00 f'(g) = 6/4 = 1.50

f'(g)·g'(x) = 6/4 × 12/6 = 12/4 = 3.00 turns of f per input turn

Why this matters next

Every layer of a neural network is a function wrapped inside another. The chain rule is the bookkeeping that lets an error at the output be traced back, layer by layer, to every weight. That backward pass is backpropagation, on the LLM path.

The definition and the chain rule, in a few lines of Python.

# numerical derivative: slope of a tiny secant (the limit, approximated)
def deriv(f, x, h=1e-6):
    return (f(x + h) - f(x)) / h

# chain rule by hand for y = (3x + 1) ** 2
g  = lambda x: 3 * x + 1     # inner:  u = g(x)
f  = lambda u: u ** 2        # outer:  y = f(u)
dg = 3                       # g'(x)
df = lambda u: 2 * u         # f'(u)

x = 2
chain = df(g(x)) * dg        # f'(g(x)) * g'(x) = 2*(7)*3 = 42

A numerical derivative approximates the limit; the chain rule multiplies the rates.

Check yourself

At the exact bottom of a smooth valley, the derivative f'(x) is:

For y = f(g(x)), the chain rule says the overall rate is:

State the chain rule for f(g(x)) and say why the rates multiply.

Think of the fraction form. Try to state it, then check.

Lock it in

The derivative is the tangent's slope: the local rate of change.
Near a point, f(x + dx) is about f(x) + f'(x)·dx. That is one descent step.
A flat tangent (f'(x) = 0) marks a peak or valley: how we find best points.
The chain rule multiplies rates along a chain, the engine of backprop.

Primary source

For the tangent-as-zoom intuition and the chain rule built visually, watch 3Blue1Brown, Essence of Calculus, chapters 2 to 4.