Logistic regression

Bend a regression line through an S-shaped curve and it stops predicting numbers and starts predicting probabilities - the leap from "how much?" to "which class?"

Linear regression answers how much - a house price, a temperature, a number on a line. But an enormous share of real questions are yes / no: spam or not, fraud or not, this word or that word. The fix is small and beautiful: take the model's raw score and squeeze it through a smooth S-curve called the sigmoid. Out comes a number pinned between 0 and 1 - a probability - and a probability is one threshold away from a decision.

From a straight line to a probability

Start with the familiar linear score - the same weighted sum linear regression uses. For a single feature $x$ it is just

In the demo below it's handier to write the score as $z=w(x-t)$. That's the same line - just with $b=-wt$ - but now the two knobs mean something you can see: $t$ is the input value where the score crosses zero (so $p=0.5$), and $w$ is the steepness of the S.

Where the decision happens

A probability isn't yet a class. To decide, pick a threshold - the natural default is $0.5$ - and call it class 1 when $p\ge 0.5$, else class 0. Because $\sigma$ is $0.5$ exactly at $z=0$, that rule is astonishingly clean:

Now play. Below is a 1-D dataset: blue points are true class 0 (sitting on the bottom, $y=0$), amber points are true class 1 (on top, $y=1$). Drag the curve left/right to move the boundary $t$, and use the steepness slider to sharpen or soften the S. Each point is filled by the probability the curve assigns it, and ringed green if that prediction is right, red if wrong. Watch the accuracy climb as you separate the colors.

Grading a probability: log loss

Accuracy only asks "right or wrong?" It can't tell a lucky 0.51 from a confident 0.99. To train with gradient descent we need a smooth score that rewards calibrated confidence. That's log loss, a.k.a. binary cross-entropy. For one labelled point with true label $y\in \{0,1\}$ and predicted probability $p$:

This is the cost function you met last lesson, specialized for classification, and it's what gradient descent actually minimizes when it fits $w$ and $b$. In the demo, watch the log loss readout react to steepness even while accuracy sits still - that's the model getting more (or less) sure of itself.

Straight boundaries, and bending them

With one feature the boundary is a single point; with two features it becomes a straight line; with many, a flat hyperplane. That's why plain logistic regression is a linear classifier - its dividing surface is always flat. When the classes coil around each other so no straight cut works, you either hand it richer, transformed features (say $x^2$ or $x_1{x}_2$) so a curve becomes possible, or you stack many logistic-style units into a neural network - the very next idea, and the reason the humble sigmoid keeps showing up.

Logistic regression, in six lines of Python:

import math

def sigmoid(z):   # squash any score into (0, 1)
    return 1 / (1 + math.exp(-z))

def prob(x, w, t):    # score z = w*(x - t), then squash
    return sigmoid(w * (x - t))

def predict(x, w, t): # threshold at 0.5 -> a hard class
    return 1 if prob(x, w, t) >= 0.5 else 0

def log_loss(y, p):   # binary cross-entropy for one point
    return -(y * math.log(p) + (1 - y) * math.log(1 - p))

Check yourself