Evaluating classifiers: precision, recall, and ROC

A model can be 99% accurate and completely useless. The confusion matrix, the precision-recall trade-off, and the threshold knob are how you find out which.

Your spam filter reports 97% accuracy. Sounds great - until you notice that 97% of the mail was ham anyway, so a filter that flags nothing would score the same and catch zero spam. Accuracy hides this failure completely. To see what a classifier actually does you need four raw counts - the confusion matrix - and two numbers built from them, precision and recall, that pull in opposite directions as you turn one knob: the decision threshold. Sweep that knob across every setting and you trace the ROC curve, a single picture of how good the model really is.

The confusion matrix

Fix a threshold and every prediction lands in one of four bins, tallied in a 2×2 table. Rows are the truth, columns are the prediction:

• TP (true positive) - real spam, flagged spam. ✔
• FP (false positive) - real ham, flagged spam. A good email in the junk folder.
• FN (false negative) - real spam, flagged ham. Spam slips into the inbox.
• TN (true negative) - real ham, flagged ham. ✔

The diagonal (TP, TN) is what you got right; the off-diagonal (FP, FN) is every mistake, split by kind. From these four counts, three definitions:

Why accuracy lies on rare classes

Accuracy divides correct predictions by all predictions - so it is dominated by whichever class is biggest. Imagine a fraud detector where only $1\%$ of transactions are fraud. A model that outputs "legit" for every transaction achieves $99\%$ accuracy while catching zero fraud: its recall is $0$. The same trap hits medical screening, spam, anomaly detection, and safety filters - anywhere the class you care about is rare, accuracy will happily hide total failure behind a big number.

The threshold knob and the ROC curve

The model's probability is fixed; the threshold is yours to set. Slide it toward $1$ and you flag only your most confident predictions - precision rises, recall falls. Slide it toward $0$ and you flag almost everything - recall rises, precision falls. Plot two rates as you sweep the threshold through every value - the true positive rate (recall) against the false positive rate - and you get the ROC curve. Its area, the AUC, summarises the whole model in one number.

Below are 24 emails a model has already scored - 9 real spam (top lane), 15 real ham (bottom lane) - placed on the probability axis. Drag the blue threshold line (or the slider). Watch the confusion matrix, the precision/recall bars, and the moving dot on the ROC curve all update from that single motion. Dots turn green when classified correctly, red when they become an error.

import numpy as np

# fixed model outputs: P(spam) for 24 emails, and the true labels (1 = spam)
scores = np.array([.97,.90,.83,.77,.71,.64,.56,.47,.34,           # 9 real spam
                   .04,.08,.11,.15,.19,.23,.27,.32,.37,.43,.49,.57,.66,.78,.88])  # 15 real ham
labels = np.array([1]*9 + [0]*15)

def confusion(t):
    pred = scores >= t                        # decide: flag as spam?
    tp = np.sum(pred & (labels == 1))
    fp = np.sum(pred & (labels == 0))
    fn = np.sum(~pred & (labels == 1))
    tn = np.sum(~pred & (labels == 0))
    return tp, fp, fn, tn

tp, fp, fn, tn = confusion(0.5)
precision = tp / (tp + fp)                     # of flagged, how many were spam  -> 0.64
recall    = tp / (tp + fn)                     # of real spam, how much we caught -> 0.78

# sweep every threshold to trace the ROC curve, then integrate for the AUC
ts  = np.sort(np.unique(scores))[::-1]
tpr = [confusion(t)[0] / (labels == 1).sum() for t in ts]
fpr = [confusion(t)[1] / (labels == 0).sum() for t in ts]
auc = np.trapz(tpr, fpr)                     # area under the ROC -> 0.82

Check yourself