Pretraining at scale

How a Transformer swallows trillions of tokens of raw internet and, with no human labels at all, teaches itself to model language - one gradient step at a time.

You already know the two halves of this puzzle. From Lesson 37 you know the objective - read the tokens so far, predict the next one, and grade the bet with cross-entropy. From Lesson 11 you know the learning rule - nudge every weight against its gradient. Pretraining is nothing more than these two ideas run at absurd scale: a data pipeline that feeds trillions of tokens through the same loop, millions of times, until the loss stops falling. This lesson is about what breaks - and what you must tune - when that loop gets enormous.

The fuel: web-scale text, deduplicated

A frontier model is not shown a curated textbook. It is shown a firehose: snapshots of the open web (Common Crawl), code repositories, books, encyclopedias, and forums - cleaned, filtered for quality, and stitched into one gigantic stream of tokens. The single most important cleaning step is deduplication: the raw web is full of near-identical copies (mirrors, boilerplate, reposts), and letting the model see the same passage thousands of times wastes compute and encourages brittle memorization. Dedup, quality filtering, and blending the right mixture of sources are where a huge fraction of real pretraining effort goes.

One objective, a trillion times over

The entire training signal is the next-token cross-entropy you met in Lesson 37 - the same cross-entropy that first appeared as a cost function in Lesson 17 - now averaged over the whole corpus of $N$ tokens:

Scaling this loop introduces knobs that did not matter on a single neuron. Because the weights start random, the very first gradients are huge and chaotic, so we warm up the learning rate from zero over the first few thousand steps, then decay it back down (usually on a cosine) so the model can settle into a sharp minimum instead of bouncing around it:

Run the loop yourself

Below is a live pretraining run. Press Run and watch the tokens-seen counter race into the billions while the loss curve falls along a power-law-ish descent toward the dashed ideal floor (the best loss this data can reach - a preview of scaling laws). Then reshape it: drag the learning rate too high and the loss spikes and diverges; drag it too low and it crawls, never catching the floor. Shrink the batch and the curve turns jagged with gradient noise; grow it and it smooths. Finally, rebalance the data mixture and watch the floor itself rise or fall.

The loop, in real code

Here is the whole machine, stripped to essentials. Note three scale-only tricks: AdamW (an adaptive optimizer with decoupled weight decay, the default for LLMs); gradient accumulation, which sums gradients over several micro-batches so you can simulate a giant batch that would never fit in one GPU's memory; and checkpointing, so a crash three weeks into a run resumes instead of restarting. Distributed training simply replicates this loop across thousands of GPUs, averaging gradients between them each step.

# one pretraining loop, scaled to trillions of tokens
opt = torch.optim.AdamW(model.parameters(), lr=3e-4,
                        betas=(0.9, 0.95), weight_decay=0.1)
sched = warmup_cosine(opt, warmup=2000, total=600_000)  # steps

model.train()
for step, batch in enumerate(loader):          # batch: (B, T) token ids
    # gradient accumulation: fake a huge batch on limited memory
    for micro in batch.split(micro_bs):
        logits = model(micro[:, :-1])
        loss = F.cross_entropy(                   # the ONLY objective
            logits.reshape(-1, V),
            micro[:, 1:].reshape(-1)
        ) / accum_steps
        loss.backward()                          # backprop accumulates grads
    torch.nn.utils.clip_grad_norm_(model.parameters(), 1.0)  # tame spikes
    opt.step(); sched.step(); opt.zero_grad()   # one AdamW update
    if step % 1000 == 0:
        save_checkpoint(model, opt, step)      # so a crash resumes here

Check yourself