What Learning Means

Linear regression

The simplest model: a line that predicts

~11 minLesson 2 of 57Written by Claude Opus 4.8

Give a machine a scatter of dots and ask it to predict a number, and the simplest thing it can learn is a straight line. This is the "hello world" of machine learning: teach a machine to predict a number by finding the single best straight line through a cloud of points.

Give a machine a scatter of dots and ask it to predict $y$ from $x$ , and the simplest thing it can learn is a straight line. Linear regression finds the one line $\overset{y}{^} = w x + b$ that sits as close as possible to every point at once. It is the first real learned function - and the very same "fit a function to examples" idea that, scaled up to trillions of knobs, becomes a language model.

The hypothesis: a line with two knobs

Picture each dot as one house: its size $x$ and its price $y$ . Our model's guess for the price is written $\overset{y}{^}$ (read "y-hat"), and the model is nothing more than a line:

The model (hypothesis)

\overset{y}{^} = w x + b

Two numbers define it completely. The weight $w$ is the slope - how many dollars the price climbs for each extra unit of size. The bias $b$ is the intercept - the line's height where $x = 0$ . "Learning" a linear model means nothing more mysterious than choosing good values for $w$ and $b$ .

The picture to hold in your head

Imagine laying a rigid rod across the scatter of dots. You can tilt it (that changes

w

) and slide it up or down (that changes

b

). Every possible line is just one tilt-and-slide away from another. Somewhere among all those positions is the one that hugs the dots best - and our job is to find it.

Measuring wrongness: the squared error

To find the best line we first need to say what "best" means. For each point, the model's mistake is the residual - the vertical gap between the true value $y_{i}$ and the prediction $\overset{y}{^}_{i}$ :

r_{i} = y_{i} - \overset{y}{^}_{i} = y_{i} - (w x_{i} + b) .

Some residuals are positive (point above the line), some negative (below). If we just added them, they would cancel and a terrible line could score zero. So we square each one - squaring makes every miss positive, and it punishes big misses far more than small ones. Sum those squares over all $n$ points and you get the model's total cost:

Sum of squared errors - your first loss function

J (w, b) = i = 1 \sum n (y_{i} - (w x_{i} + b))^{2}

The best line is simply the $w$ and $b$ that make $J$ as small as possible. Fitting by minimizing this sum has a name three centuries old: least squares.

Below, each red bar is one residual, and its faint square is that residual squared - a literal square whose area is the point's contribution to the cost. Least squares is the search for the line with the smallest total shaded area.

Hunt for the best-fit line. Drag either endpoint of the line to tilt it, drag the middle to slide it, or use the sliders. Each red bar is a residual and its faint square is that residual squared.

data points (real houses)your line ŷ = wx + bresidual (error)squared errortrue best fit

your model: ŷ = 0.6·x + 6

weight w (slope) = 0.6bias b (intercept) = 6

reveal target lineshow squares

weight w0.6

bias b6

squared error177.4

total squared error J(w,b)177.4

floor 8

Drag an endpoint of the line, or nudge the sliders, to shrink the total squared error.

The meter has a floor - and it is unique

No matter how you tilt and slide, you cannot push the error below 8. That floor is the least-squares minimum, and exactly one line reaches it. For a straight line that best line even has a closed-form formula - no searching required:

w = \frac{\sum _{i} ( x _{i} - x ˉ ) ( y _{i} - y ˉ )}{\sum _{i} ( x _{i} - x ˉ ) ^{2}}, b = \overset{y}{ˉ} - w \overset{x}{ˉ} .

Here it lands on $w = 2, b = 3$ . But that tidy formula only exists because the model is so simple. Give a model millions of weights and there is no formula - instead you slide downhill on $J$ using the gradient descent you met earlier, one step at a time.

From a line to a plane: more features

Real predictions lean on more than one number. Add a second feature - say size and number of bedrooms - and the model gets a second weight:

\overset{y}{^} = w_{1} x_{1} + w_{2} x_{2} + b .

That is no longer a line but a tilted plane floating over the two inputs, and "least squares" now means the plane that minimizes the squared vertical gaps to every point. With $d$ features it becomes a flat sheet in $d + 1$ dimensions, written compactly with the dot product as $\overset{y}{^} = w \cdot x + b$ . The picture is impossible to draw past three dimensions, but the recipe never changes: pick the weights that make the sum of squared errors smallest.

import numpy as np
x = np.array([1, 2, 3, 4, 5, 6, 7, 8])
y = np.array([4, 8, 10, 10, 12, 16, 18, 18])

def sse(w, b):                       # sum of squared residuals
    pred = w * x + b
    return np.sum((y - pred) ** 2)

# closed-form least-squares fit
w = np.sum((x - x.mean()) * (y - y.mean())) / np.sum((x - x.mean()) ** 2)
b = y.mean() - w * x.mean()
print(w, b, sse(w, b))               # -> 2.0  3.0  8.0

Predictions, error, and the exact best fit in NumPy

Check yourself

Least-squares regression chooses the line that minimizes the:

In the model y-hat = wx + b, the weight w sets the line's:

Recall: write the linear-regression hypothesis and the quantity least squares minimizes.

Tie the hypothesis, the two knobs, and the loss together in one recall. Try to state it, then check.

Lock it in

The hypothesis is a line $\overset{y}{^} = w x + b$ : the weight $w$ is the slope, the bias $b$ the intercept.
The residual is the vertical gap $y_{i} - \overset{y}{^}_{i}$ ; squaring it makes every miss positive and punishes big misses hardest.
Least squares minimizes the total squared error $J (w, b)$ , which here has a unique floor at 8.
More features turn the line into a plane, and the dragged line is your first learned function and first loss - the same idea an LLM scales to billions of weights.

Primary source

Andrew Ng's Machine Learning Specialization (audit free) opens with linear regression and the squared-error cost, building every piece from intuition. For a friendly visual walk through residuals and the least-squares fit, StatQuest's Machine Learning playlist is the clearest companion on YouTube.

Ask your teacher

Sit with this: the line you just dragged is the simplest possible learned function - fit

f

to data by shrinking its mistakes. A large language model is the exact same idea, only

f

has hundreds of billions of weights instead of two, and it is fit to trillions of examples of text. The "sum of squared errors" you watched fill and drain is your first loss function; every model, up to the biggest LLM, is trained by making a loss like it as small as possible. Ask me why we square instead of taking absolute values, how this connects to gradient descent, or what changes when the thing being predicted is a word rather than a price.