Training Deep Networks

Overfitting, bias-variance, and regularization

Generalizing instead of memorizing

~12 minLesson 19 of 57Written by Claude Opus 4.8

A model that memorizes its training data looks brilliant on the exam it already saw - and falls apart on every new question. Regularization is how we stop it.

Give a flexible model a handful of noisy points and it will happily thread a curve through every single one - driving its training error to zero by contorting itself around the random noise. Then a fresh point arrives and the curve is wildly wrong. The whole discipline of generalization is one fight: learn the signal, not the noise. Regularization wins that fight by deliberately handicapping the model.

Two ways to be wrong

Underfitting is a model too simple to capture the real pattern - a straight line trying to trace a curve. It is wrong on the training data and on new data. Overfitting is the opposite failure: a model so flexible it fits the noise, acing the training set while flunking everything new. The sweet spot lives between them - and a validation set (from the train / validation / test split) is how you find it.

Play with the trade-off

Below are 12 noisy training points (amber) drawn from a hidden true curve (dashed). Drag degree up: the fit goes from a rigid underfit line to a wildly wiggly curve that threads every point. Then drag regularization $λ$ up: it shrinks the model's weights and smooths that wiggle back toward the true trend. Watch the two error bars - training error and held-out test error - pull apart and come back together.

Polynomial-fit playground drag degree & λ

training pointsheld-out test pointstrue curve (signal)model's fit

degree = 1regularization λ = 0

Train error

0.256

Test error

0.218

Underfitting - high biaslargest weight 0.7

The tell-tale gap

Overfitting has a fingerprint: training error near zero while test error stays high. The model aced the questions it memorized and guessed the rest. Notice too what happens to the largest weight - at degree 10 with

λ = 0

it balloons past 90. Those enormous coefficients are exactly what regularization exists to punish; push

λ

up and watch them collapse back toward sanity.

The bias-variance trade-off

Every model's expected error on new data splits into three parts. Write $\hat{f}$ for the model we fit, $y$ for the true value plus noise of variance $σ^{2}$ :

Error decomposition

E [(y - \hat{f})^{2}] = too rigid (Bias [\hat{f}])^{2} + too jumpy Var [\hat{f}] + noise floor σ^{2}

Bias is error from wrong assumptions - a straight line cannot bend, so it is biased no matter how much data you feed it. Variance is how much the fit would jerk around if you resampled the training set - a degree-10 curve swings wildly, so it has high variance. The last term $σ^{2}$ is irreducible: noise you can never fit. Simpler models trade high bias for low variance; complex models do the reverse. Total error is a U-shape in model complexity, and regularization slides you toward its bottom.

Underfitting is the high-bias end; overfitting is the high-variance end. You cannot drive both to zero - you trade them. That is why "just use a bigger model" is not automatically better: past the sweet spot, extra capacity buys variance you did not want.

The regularizer's toolbox

Regularization is any deliberate handicap that biases the model toward simpler solutions. The workhorse is a penalty added to the loss - the model now pays for large weights, so it only grows a weight when the data really earns it:

Technique	What it does	Effect
L2 (weight decay)	adds $λ \sum_{j} w_{j}^{2}$ to the loss	shrinks all weights smoothly toward zero
L1 (lasso)	adds $λ \sum_{j} ∣ w_{j} ∣$ to the loss	drives some weights to exactly zero → sparse
Dropout	randomly zeros neurons each step	stops co-adaptation; an implicit ensemble
Early stopping	halt when validation error turns up	quits before the model starts memorizing
More data	collect more real examples	the noise averages out - the best regularizer

The $λ$ in your playground is exactly the L2 knob: at $λ = 0$ the model minimizes only the fit error and overfits; crank it up and the weight penalty dominates, forcing a flat, high-bias line. Somewhere in between is the bottom of the U.

# penalized loss: fit error + weight penalty
loss = mse(y_pred, y) + lam * np.sum(w ** 2)

# closed form: solve (X^T X + lam*I) w = X^T y
A = X.T @ X + lam * np.eye(d)   # lam > 0 makes A invertible & smooth
w = np.linalg.solve(A, X.T @ y)

# same idea in SGD: weight decay shrinks w every step
w = w - lr * (grad + lam * w)     # the "+ lam*w" pull toward 0

L2-regularized least squares (ridge)

Check yourself

You fit a model and see these two numbers. Which pattern screams overfitting?

Turning the regularization strength lambda up toward very large values does what to the model?

Why is "more data" often called the best regularizer of all?

Try to state it, then check.

Lock it in

Generalization is one fight - learn the signal, not the noise: a flexible model will thread a curve through every noisy point, driving training error to zero and failing on anything new.
Two ways to be wrong: underfitting (too simple, high bias, wrong on train and test) and overfitting (too flexible, high variance, aces train and flunks new).
Overfitting's fingerprint is a large gap - training error near zero while test error stays high - usually with weights ballooning out of sane range.
Expected error decomposes into $bias^{2} + variance + σ^{2}$ ; the noise floor is irreducible and total error is a U-shape in complexity, so you trade bias against variance rather than zeroing both.
Regularizers deliberately handicap the model toward simpler fits: L2 shrinks weights smoothly, L1 drives some to exactly zero (sparse), dropout and early stopping help, and more data is the best regularizer of all - it lowers variance with no extra bias.

Primary source

Andrew Ng derives L2 regularization and the bias-variance intuition gently in the Machine Learning Specialization (the regularization week). For a concise, code-first treatment with interactive figures on generalization and regularization, Google's Machine Learning Crash Course is excellent.

Ask your teacher

Overfitting is the central risk of all learning, and the same fight shows up everywhere in LLMs wearing different costumes: the KL penalty in RLHF (Lesson 53) that stops a policy from drifting too far, the weight decay baked into AdamW (Lesson 96), and the industry mantra that more tokens beats a cleverer model. Want me to trace L2 weight decay straight into an AdamW update, or show how dropout behaves as an ensemble? Ask and I'll build the follow-up.