What Learning Means

Gradient descent

Rolling downhill to the best parameters

~12 minLesson 4 of 57Written by Claude Opus 4.8

The one algorithm behind nearly all of modern machine learning: how a model, told only how wrong it is, feels its way downhill to being right.

A model "learns" by adjusting numbers - its weights - until its mistakes shrink. But which way should each weight move, and by how much? Gradient descent gives a stunningly simple answer: measure how wrong you are, find the downhill direction, and take a small step. Repeat. That's the whole trick that trains linear regression, every neural network, and the largest language models on Earth.

The picture to hold in your head

Imagine the model's error as a landscape. Each spot on the ground is one setting of the weights; the height at that spot is how wrong the model is there - call it the loss. Learning is just walking downhill to the lowest valley. You can't see the whole map, but at your feet you can always feel which way is down. So you step that way, and again, and again.

A loss landscape you can roll down

Let's make it concrete with a single weight $w$ and the simplest possible landscape - a bowl. The loss is $J (w) = (w - 3)^{2}$ : it's zero when $w = 3$ (the model is perfect) and rises the further $w$ drifts either way. The whole game is to find the bottom of the bowl without being told that it sits at 3.

The $\frac{dJ}{d w}$ is the slope of the landscape under your feet - and crucially, it points uphill, toward more error. So to go down, we step in the opposite direction. That single sentence is the entire update rule:

The update rule

w := w - α \frac{dJ}{d w}

Read it as "the new $w$ is the old $w$ , nudged against the slope." The minus sign is what makes it descent - flip it to a plus and you'd climb toward more error. For our bowl the slope has a tidy form (you'll derive rules like this later):

\frac{dJ}{d w} = 2 (w - 3) .

The number $α$ (alpha) is the - the size of each step. It's the one knob you get to tune, and getting it wrong is the classic beginner trap. Let's see why.

Roll the ball yourself

Click anywhere on the curve to drop a ball at that weight. The amber line is the gradient - the slope where the ball sits - and it always tilts uphill. The blue arrow is the step we actually take, pointing the opposite way, into the hollow ring where the ball will land next. Press Step for one move, or Run to animate the whole descent and leave a trail. Then drag $α$ and watch everything change.

Gradient descent on a loss bowl. Amber is the gradient (the slope, pointing uphill); blue is the step we take (downhill), landing in the hollow ring. Click the curve to drop a ball, press Step or Run, then drag alpha.

gradient = slope (points uphill)the step we take (downhill)where one step lands

learning rate α = 0.30

drop point at w = 6

try a preset

weight w6

loss J(w)9

gradient dJ/dw6

Ready - press Step for one move, or Run to animate.

The learning rate is everything - crank it past 1.0

Hit the α 1.10 · blow up preset and watch: instead of settling, each step overshoots the bottom and lands higher on the far wall, then higher still, until the ball rockets off the chart. That's divergence, and it's the single most instructive failure in all of ML. Too small (try 0.05) and the ball barely creeps - you'd wait forever. Too big and it explodes. Real training lives on the knife's edge between them, and picking

α

well is half the craft.

Why does that happen? With our bowl, one step multiplies the distance-to-the-bottom by $(1 - 2 α)$ . When $α$ is small that factor is just under 1, so the gap shrinks a little each time - slow but safe. At $α = 0.5$ the factor is exactly 0, so you leap to the bottom in a single step. Past $α = 1$ the factor exceeds 1 in size, so the gap grows every step - the ball climbs out. Same algorithm, same bowl; only the step size changed.

One valley, or many?

Our bowl has exactly one low point, so descent can't miss - it's a landscape. Real models rarely enjoy that luxury. Their loss surfaces are lumpy, with many dips: a is a valley you can get stuck in, while the is the deepest valley of all. Gradient descent only feels the ground at its feet, so it can settle into a shallow dip and stop. In practice a bag of tricks - momentum, randomness, and the sheer high-dimensional roominess of big models - usually finds a valley that's good enough, even if it isn't provably the deepest.

# descend the bowl J(w) = (w - 3)^2
w = 6.0                      # start the ball somewhere
alpha = 0.3                  # the learning rate
for step in range(15):
    grad = 2 * (w - 3)       # dJ/dw: the slope here
    w = w - alpha * grad        # step against the slope
    print(w, (w - 3) ** 2)     # weight, and the loss

The entire algorithm, in eight lines of Python

Check yourself

If the learning rate α is set far too large, gradient descent will most likely:

The update rule w := w - α·(dJ/dw) moves the weight:

Recall: what does the learning rate α control, and what goes wrong at each extreme?

Tie the one knob to both failure modes. Try to state it, then check.

Primary source

Andrew Ng's Machine Learning Specialization builds gradient descent from the ground up with the clearest intuition anywhere. For a hands-on companion where you drag the learning rate and watch it converge or explode, Google's Machine Learning Crash Course has an excellent interactive section on gradient descent and learning rate.

Ask your teacher

Here's the connection worth sitting with: this one algorithm trains everything - linear regression, every neural network, and the largest LLMs alike. The only thing that scales up is the number of weights, from one in our bowl to hundreds of billions. Backpropagation, which you'll meet later, isn't a different learning method - it's just the clever way we compute the gradient efficiently for all those millions of weights at once. Ask me about momentum, stochastic gradient descent, or how a landscape with billions of dimensions can still be walked downhill.