Scaling laws and emergent abilities

Here is the surprise at the heart of modern AI: the test loss of a language model is not a mystery you discover after training - it is a number you can predict in advance from three inputs. Give me the parameter count, the number of training tokens, and the compute budget, and I can tell you roughly how good the model will be, because loss falls as a smooth power law in each of them. And every so often, riding on top of that smooth curve, a brand-new ability appears almost overnight.

Loss is a straight line on log-log

A power law ties a quantity to a resource through an exponent. Writing $L$ for loss and $C$ for compute, the reducible part of the loss behaves like $L-L_{\infty }\propto C^{-{\alpha }_C}$. There is a floor $L_{\infty }$ - the irreducible entropy of language itself, which no amount of compute removes - and everything above it shrinks as a power of the budget:

Take the logarithm of the reducible part and the power law unmasks itself as a straight line - slope $-{\alpha }_C$, no curvature anywhere:

Drag the compute budget along the frontier

The top panel is a log-log plot of the reducible loss (everything above the floor $E$) against the compute budget $C$. Drag the slider to spend more or less compute. For each budget the plotter places you at the compute-optimal point - the best split of that budget into parameters and tokens - and reads off the predicted loss. The bottom panel tracks one downstream skill's accuracy at the same scale. Watch it: near zero, near zero... then a sudden cliff.

Compute-optimal training

Because compute is pinned at $C\approx 6ND$, choosing $N$ forces $D$. Chinchilla asks: across all valid $(N,D)$ splits of a fixed $C$, which minimizes the loss law above? The answer is that both grow as roughly the square root of compute:

Kaplan et al. (2020) first drew these lines and - with a different treatment of the learning-rate schedule - leaned toward spending most new compute on parameters. Chinchilla (2022) re-ran the experiment more carefully and found data had been badly under-weighted. Same phenomenon, sharper measurement, a very different prescription. Both papers agree on the shape; they disagree on the slope of the split, and that disagreement reshaped how every frontier model since is trained.

# Chinchilla: pick params & tokens for a fixed compute budget C.
E, A, B     = 1.69, 406.4, 410.7      # loss-law constants
alpha, beta = 0.34, 0.28              # the two exponents

def optimal(C):                        # C in FLOPs
    N = (C / 120) ** 0.5              # since C = 6*N*D and D = 20*N
    D = 20 * N                          # ~20 tokens per parameter
    loss = E + A / N**alpha + B / D**beta
    return N, D, loss                    # optimal(5.88e23) -> 70B, 1.4T, 1.94

When abilities emerge

The loss curve is smooth, yet specific capabilities - three-digit arithmetic, following an instruction, chain-of-thought - can stay pinned near zero for model after model and then jump sharply once scale crosses some threshold, as the bottom panel shows. Wei et al. called these emergent abilities: not present in smaller models, present in larger ones, and not obviously extrapolable from the small-scale trend.

There is an honest caveat worth carrying. Schaeffer et al. argued some emergence is partly a measurement artifact: a harsh all-or-nothing metric like exact-match makes a smoothly improving model look like it snaps on, whereas a softer metric reveals steady gains underneath. So the loss is genuinely smooth; whether a skill looks emergent depends partly on how you score it. Either way, the practical lesson stands - you cannot always predict which skills a bigger model will unlock, only that the loss will keep obeying its law.

Check yourself