What Learning Means

The cost function

Turning 'wrong' into a number to minimize

~10 minLesson 3 of 57Written by Claude Opus 4.8

A model can't fix a mistake it can't measure. The cost function is the measurement: feed it the model's current settings and it hands back a single number - how wrong you are, all in. Big number, bad model. Small number, good model. Every training run in machine learning, from a two-parameter line to a trillion-parameter language model, is nothing more than the hunt for the settings that drive this one number as low as it will go.

The idea in one line

"Learning" sounds mysterious until you make it a score. Pick a way to measure total wrongness, then adjust the model's numbers to shrink that score. The cost function is the scorekeeper - and once you have it, learning becomes an ordinary optimization problem: find the inputs that minimize an output.

Mean squared error: the workhorse score

Take our line from last lesson, $\overset{y}{^} = w x + b$ . For a data point $(x_{i}, y_{i})$ the model predicts $\overset{y}{^}_{i} = w x_{i} + b$ , and it misses by the error $\overset{y}{^}_{i} - y_{i}$ . The most common way to turn a whole dataset of misses into one score is mean squared error - square each miss, then average:

Mean squared error (MSE)

J (w, b) = \frac{1}{n} i = 1 \sum n (\overset{y}{^}_{i} w x_{i} + b - y_{i})^{2}

Two words that get used loosely but mean different things: the loss is the error on a single example, $ℓ_{i} = (\overset{y}{^}_{i} - y_{i})^{2}$ ; the cost $J$ is the average loss over all $n$ examples. Loss is per-point, cost is the whole-batch verdict - and $J$ depends only on the parameters $w$ and $b$ , since the data is fixed.

Steer the line, watch the score

Here are two views of the same two numbers. On the left, the line $w x + b$ sitting over five data points, with each error drawn as a literal square - its area is that point's squared loss. On the right, the whole cost landscape: every $(w, b)$ pair is a spot on the map, and its height (shown as rings and shading) is the cost there. The glowing dot is you. Drag it across the landscape - or nudge the sliders - and feel the line and the squares respond in lockstep. Your job: steer to the bottom of the bowl, the center of the rings.

Two views of the same two numbers. Left: the line wx+b over five points, every squared error drawn as a square whose area is that point's loss. Right: the cost surface. Drag the dot, drag the line, or move the sliders.

The fit line + squared errors

dataline wx+bsquared error = area

The cost surface contours over (w, b)

you (w, b)★ minimumequal-cost rings

slope w = 1.5intercept b = 0

show squared errors

slope w1.50

intercept b0

cost J(w,b)7.55

Steer the dot toward the center of the rings to shrink the cost.

Why square the error, and not just add it up?

Three reasons, all visible in the demo. (1) Signs would cancel. A point above the line (+3) and one below (−3) are both wrong, but their raw errors sum to zero - squaring makes every miss positive. (2) Big misses hurt more. One point off by 4 contributes

4^{2} = 16

; four points each off by 1 contribute only

4

total - so the score screams loudest about the worst outlier, and the model works hardest to reel it in. (3) It makes a smooth bowl. Squaring gives a single, gently curved minimum you can slide down; the absolute value

∣ \overset{y}{^} - y ∣

has a sharp kink that trips up the calculus you'll use to descend.

Parameters are coordinates; cost is the height

The right-hand panel is the whole point. Stop thinking of $w$ and $b$ as knobs and start thinking of them as coordinates: the pair $(w, b)$ is an address on a map, and the cost $J (w, b)$ is the altitude at that address. Plot the altitude everywhere and you get a surface - for mean squared error it's a bowl (convex, one lowest point), which is why the contour rings close neatly around a single ★. Notice the rings are stretched into a tilted valley: this dataset has $x$ -values that couple $w$ and $b$ , so the cheapest way to raise the slope is to lower the intercept. Real models have millions of coordinates instead of two, but the picture is identical - and gradient descent is simply the act of walking downhill on this surface until the altitude stops dropping.

import numpy as np
x = np.array([1, 2, 3, 4, 5])
y = np.array([2, 6, 7, 10, 10])

def cost(w, b):
    y_hat  = w * x + b        # the model's predictions
    errors = y_hat - y        # how far off each point is
    return np.mean(errors ** 2)  # square, then average = the cost

print(cost(1.5, 0.0))   # 7.55  - a so-so line
print(cost(2.0, 1.0))   # 0.80  - the best fit, bottom of the bowl

MSE is four lines of NumPy

Check yourself

In MSE, if one example's error doubles, its contribution to the score becomes:

On the cost surface, a single point corresponds to:

Recall: how do loss and cost differ, and what is the MSE formula?

Pin down the per-point vs whole-batch distinction and the formula. Try to state it, then check.

Primary source

StatQuest's Machine Learning playlist walks through cost functions and gradient descent with the friendliest visual intuition anywhere. For the rigorous build-up - squared error, the cost surface, and why we minimize it - see Andrew Ng's Machine Learning Specialization.

Ask your teacher

Here's the thread that runs all the way to the top: cross-entropy - the exact loss a large language model minimizes - is a cost function just like this one, only its "error" measures how much probability the model put on the wrong next token. Training an LLM is gradient descent walking downhill on that surface, and next-token prediction is minimizing this number, billions of parameters at a time. Ask me why classification swaps MSE for cross-entropy, or how a bowl in two dimensions becomes a landscape in a trillion.