Feature scaling and encoding

A five-minute preprocessing step that turns a slow, zig-zagging descent into a straight sprint to the answer - and turns words into numbers a model can actually learn from.

Suppose one feature is a house's area in square feet (values around 2,000) and another is its number of bedrooms (values around 3). To the learning algorithm those are just two numbers, but one is a thousand times bigger than the other - and that mismatch quietly wrecks training. Gradient descent starts zig-zagging down a narrow canyon instead of walking straight to the answer, and it crawls. The fix is almost embarrassingly cheap: rescale every feature to a comparable range before training, and turn each category into a number in a way that doesn't invent a false order. Do that, and the same optimizer converges in a handful of steps instead of hundreds.

Two ways to put features on the same footing

There are two standard rescalings, and you will meet both constantly. Min-max normalization squeezes a feature into the interval $[0,1]$ by subtracting its minimum and dividing by its range:

Standardization - the z-score from Lesson 59 - recenters a feature to mean $0$ and rescales it to standard deviation $1$:

Min-max is handy when you need bounded inputs (pixels in $[0,1]$, say); standardization is the default for most models because it is robust to differing spreads and doesn't let a single outlier squash everything else into a corner. Either way the goal is identical: no feature gets to dominate simply because its raw numbers happen to be large.

Why unscaled features make descent zig-zag

Picture the cost as a landscape over the two weights (Lesson 11). For a linear model the landscape is a bowl, and its contour lines - the level sets of equal cost - tell you the shape. When the features share a scale, the bowl is round and the contours are circles: the downhill direction points straight at the minimum, so descent walks there directly. When the features differ wildly in scale, the bowl is a long, narrow canyon and the contours are stretched ellipses. Now the steepest-downhill direction points mostly across the canyon rather than along it, so each step slams into the far wall and the path zig-zags, inching toward the minimum. Flip the toggle below and watch the canyon become a bowl.

Turning categories into numbers

Models do arithmetic, so a categorical column like color ∈ {red, green, blue} has to become numeric. The tempting shortcut - red $=1$, green $=2$, blue $=3$ - is a trap: it tells the model that green sits "between" red and blue and that blue is "three times" red, an ordering that doesn't exist. One-hot encoding avoids the fiction by giving each category its own column: red $=[1,0,0]$, green $=[0,1,0]$, blue $=[0,0,1]$. No category is larger than another; they are simply different directions. (Ordinal encoding as $1,2,3$ is fine only when the categories are genuinely ordered, like small < medium < large.)

import numpy as np

# --- learn the statistics from TRAIN ONLY, then reuse them ---
mu    = X_train.mean(axis=0)      # per-feature mean   (z-score)
sigma = X_train.std(axis=0)       # per-feature std
def zscore(X): return (X - mu) / sigma       # -> mean 0, std 1

lo    = X_train.min(axis=0)       # per-feature min    (min-max)
hi    = X_train.max(axis=0)
def minmax(X): return (X - lo) / (hi - lo)   # -> into [0, 1]

X_train_s = zscore(X_train)             # never call .mean() on X_test!
X_val_s   = zscore(X_val)               # reuse the TRAIN mu, sigma
X_test_s  = zscore(X_test)

# --- one-hot a categorical column: 3 categories -> 3 columns ---
cats   = ["red", "green", "blue"]        # the vocabulary, learned from TRAIN
onehot = np.array([[c == k for k in cats] for c in colors], dtype=float)
# "green" -> [0, 1, 0]   an unseen color -> [0, 0, 0]

Check yourself