Floating point and numerical stability

A real number line is infinite and continuous; a 32-bit register holds exactly $2^{32}$ patterns. So the machine cannot store most numbers - it stores the nearest representable one and quietly rounds. Usually the error is invisible. But in machine learning you sum millions of these, exponentiate them, and divide - and the rounding compounds until a probability that should be $1.0$ comes back as NaN. This lesson opens the bits, shows exactly where they bite, and gives you the standard fixes: subtract the max, and work in log-space.

A real number in 32 bits

The IEEE-754 float32 ("single precision") splits its 32 bits into three fields: 1 sign bit, 8 exponent bits, and 23 mantissa bits (also called the fraction). For a normal number the value is reconstructed as:

Enough theory - open the hood. The demo below has two panels. The first is a live bit inspector: toggle any bit and watch the exact value and the gap to the next representable number update. The second runs softmax, the function that turns model scores (logits) into probabilities, and shows it detonate into NaN - then survive. Softmax and its overflow-proof cousin log-sum-exp are defined here so both panels make sense:

Four ways floats bite

Every one of these is the same rounding, viewed from a different angle:

• Rounding error. $0.1$, $0.2$, $0.3$ are all non-terminating in binary, so each is stored approximately. In double precision the errors don't cancel: $0.1+0.2=0.30000000000000004\ne 0.3$. This is not a bug - it is the format working exactly as specified.
• Overflow. Exceed the largest finite value ( $\approx 3.4\times 10^{38}$ for float32) and you get $+\infty$. One stray $\infty$ then poisons everything: $\infty -\infty$ and $\infty /\infty$ are both NaN, and NaN spreads through every later operation.
• Underflow. Numbers smaller than $\approx 1.4\times 10^{-45}$ round to $0$. A product of many probabilities underflows to exactly zero, and $\log 0=-\infty$ - which is precisely why training works with sums of logs rather than products.
• Catastrophic cancellation. Subtract two nearly equal numbers and the matching leading digits annihilate, leaving only the noisy low bits. The relative error of the tiny result can explode by orders of magnitude - the classic trap in computing a variance as $E[x^2]-E[x{]}^2$.

The fix: subtract the max, live in log-space

The stable panel above is the recipe every deep-learning framework uses internally. Subtract the max before exponentiating (kills overflow at no mathematical cost), and carry log-probabilities rather than probabilities (kills underflow, turns fragile products into robust sums). Cross-entropy loss reads those log-probs directly, which is why libraries pair softmax and log into one fused, numerically-stable log_softmax.

import numpy as np

def softmax_naive(z):          # z: logits, float32
    e = np.exp(z)              # exp(z) overflows to inf past ~88.7
    return e / e.sum()         # inf / inf -> nan: output destroyed

def softmax_stable(z):
    m = z.max()                # the largest logit
    e = np.exp(z - m)          # every exponent <= 0, so exp(...) in (0, 1]
    return e / e.sum()         # identical result, nothing overflows

def logsumexp(z):              # log of the denominator, overflow-proof
    m = z.max()
    return m + np.log(np.exp(z - m).sum())

def log_softmax(z):            # what training actually optimizes
    return z - logsumexp(z)    # cross-entropy reads these log-probs

z = np.array([120., 2., 0.], dtype=np.float32)
softmax_naive(z)               # [nan, 0., 0.]  -- exp(120) is inf in float32
softmax_stable(z)              # [ 1., 0., 0.]  -- correct and finite

Check yourself