Calculus for Learning

Partial derivatives and gradients

The gradient: which way is uphill

~12 minLesson 9 of 19Written by Claude Opus 4.8

A one-input function has one slope. But a loss function has millions of inputs, every weight in a model, and at any point it slopes differently in every direction. The gradient is the elegant trick that tames this.

Stand on a hillside. In every direction you could face, the ground tilts by a different amount, steep behind you, flat to your left. A is the slope when you look along one fixed compass line. The bundles all those slopes together and reports the single direction that climbs fastest, along with how steep that climb is. To go down the hill, which is all a model ever wants, turn around and step the opposite way.

Partial derivatives: wiggle one input, freeze the rest

Take a function of two inputs, $f (x, y) = x^{2} + y^{2}$ , a perfectly round bowl sitting above the plane. To find its slope we borrow the one-variable derivative and add a rule: vary a single input and treat every other input as a constant. The result is a partial derivative, written with a curly $\partial$ instead of $d$ .

\frac{\partial f}{\partial x} = 2 x \frac{\partial f}{\partial y} = 2 y Read df/dx as: how fast does f change if I nudge only x?

At the point $(3, 1)$ the $x$ -slope is $2 (3) = 6$ : push $x$ a hair and the height climbs six times as fast. Push $y$ instead and it climbs only $2 (1) = 2$ . Same spot, two very different slopes.

The gradient: all the partials, stacked into one arrow

Nothing forces us to pick a direction. Stack the partials into a vector and you get the gradient, written $\nabla f$ (say "grad f" or "del f").

\nabla f = [\partial f / \partial x \partial f / \partial y] = [2 x 2 y]

At $(3, 1)$ the gradient is $(6, 2)$ , an arrow pointing mostly east and a little north. Its direction is where the hill rises fastest; its length $∥ \nabla f ∥$ is how steep that fastest climb is. Why does that one arrow capture the steepest direction? Because the slope along any unit direction $u$ is just a dot product with the gradient, the directional derivative.

D_{u} f = \nabla f \cdot u = ∥ \nabla f ∥ cos θ cos(theta) is largest at theta = 0, so the slope is maximized facing along the gradient.

Perpendicular to the contour

Turn 90 degrees away (

θ = 9 0^{\circ}

cos θ = 0

) and the slope is zero: that direction runs along a level curve, staying at constant height. So the gradient is always perpendicular to the contour you are standing on.

Contour maps, and descending the negative gradient

A contour map draws the loops where $f$ holds one fixed value. Rings packed tightly mean a steep slope; rings spread apart mean gentle ground. The gradient stabs straight across the rings, uphill. And here is the entire idea of learning, distilled: to shrink the loss, step in the negative gradient direction, $- \nabla f$ , the steepest way down. Click the map, press Descend or Run, then drag the learning rate up and watch a calm descent turn into overshoot, oscillation, and finally divergence.

Click the map to drop a point. Amber is the gradient (uphill, cutting across the rings); blue is the step we take (downhill). The trail records the path.

alpha preset

learning rate α = 0.10

point = (4.00, 4.00)f = 32.00∇f = (8.0, 8.0)

Ready - press Descend for one step, or Run to animate.

The stretched valley is where alpha earns its keep

Switch to the stretched valley

f = 0.3 x^{2} + y^{2}

: gentle east-west, steep north-south. Now the gradient does not point at the minimum, it points across the narrow direction, so descent bounces from wall to wall in a zig-zag. Crank

α

toward 1 and the bounces stop shrinking; past 1 they grow and the point escapes. This mismatch is exactly why real training uses momentum and adaptive step sizes.

The update rule, in vector form

Every step is the one-dimensional rule from the last lesson, applied to the whole vector of inputs at once:

p := p - α \nabla f (p)

. For the round bowl this scales the point by

(1 - 2 α)

each step, smooth for small

α

, oscillating near

α = 1

, exploding beyond. One rule, whether

p

holds two numbers or two hundred billion.

Gradient descent in two dimensions, in nine lines.

import numpy as np

def grad(p):                  # grad f for f = x^2 + y^2 -> [2x, 2y]
    return np.array([2 * p[0], 2 * p[1]])

p = np.array([4.0, 4.0])      # start somewhere on the map
alpha = 0.1                   # the learning rate (step size)
for step in range(50):
    p = p - alpha * grad(p)   # step DOWN the negative gradient
print(p)                      # -> near [0, 0], the bottom of the bowl

Step against the gradient; the round bowl converges to (0, 0).

Check yourself

A partial derivative df/dx measures the slope when you:

Relative to the contour line you are standing on, the gradient points:

What does the gradient point toward, and how do we use it to learn?

Direction and length both carry meaning. Try to state it, then check.

Lock it in

A partial derivative varies one input and freezes the rest.
The gradient stacks the partials; it points uphill, steepest first.
The gradient is perpendicular to the contour you stand on.
Learning is p := p - alpha·grad f, one rule at any number of dimensions.

Primary source

For gradients and directional derivatives built visually, see 3Blue1Brown's multivariable calculus, and Khan Academy's multivariable derivatives unit.