Reasoning and test-time compute

Some problems cannot be solved in a single shot - the model has to think first. This lesson shows you the mechanism behind reasoning models, why spending more tokens at inference buys accuracy on hard tasks, and when that trade is worth it for the agents you build.

Why one token at a time is a bottleneck

From the last lesson: an LLM is a next-token predictor. It commits to one token at a time, and the compute it spends per token is roughly fixed - about twice the model's parameter count in FLOPs for each token generated.^[1] That has a sharp consequence.

There is no hidden plan the model executes silently. Anything that looks like deliberation has to be written down as tokens in the context - a scratchpad the model reads back to itself. The scratchpad is the computation.

Chain of thought: thinking out loud

The seed of everything here is chain of thought (CoT) prompting. Wei et al. (2022) showed that simply asking a model to produce intermediate reasoning steps - or just appending "Let's think step by step" - sharply improves arithmetic, commonsense, and symbolic reasoning. Two details matter: it worked on an off-the-shelf model with no fine-tuning, and the benefit emerges only at scale (roughly 100B-parameter models; smaller ones barely budge).^[2]

The classic worked example is "How many r's in strawberry?". The model sees tokens like ["str","aw","berry"], not letters, so it guesses. A model that first spells the word out token-by-token in its scratchpad turns a character task into a sequence it can actually process - and gets it right. That is the whole value of thinking tokens in miniature.

Reasoning models: the 2024 to 2026 shift

Chain of thought put the burden on you to prompt for steps. The leap of 2024 to 2026 was to train the model to generate a long internal chain of thought automatically, then let it spend a variable - often large - number of tokens deliberating before it answers. These are reasoning models. The current families are OpenAI o1 and o3, Claude extended and adaptive thinking, DeepSeek-R1, and Gemini thinking.

OpenAI's o1 is trained with large-scale reinforcement learning to produce a productive private chain of thought - breaking problems into steps, catching its own mistakes, and trying alternative strategies. The raw reasoning is hidden: you see the final answer plus a count of "reasoning tokens," not their contents.^[3] Anthropic instead makes thinking visible (as summarized thinking blocks), trading a little competitive secrecy for transparency and debuggability.^[5]

DeepSeek-R1 proved the punchline openly: reasoning can emerge from pure RL with no supervised reasoning traces. The reward was just rule-based - is the final answer verifiably correct, and did it wrap its reasoning in <think>...</think>? Self-reflection, verification, and spontaneous "aha moments" appeared on their own, and AIME accuracy climbed from 15.6% to 71.0% as the model taught itself to think longer.^[4]

Test-time compute: a new scaling axis

The headline empirical result from o1 is the reason this matters: accuracy improves smoothly (roughly log-linear) with both train-time RL compute and test-time thinking compute.^[3] For decades the only way to make a model smarter was to make it bigger and pre-train it longer. Test-time compute is a genuinely new lever: you can buy accuracy at inference by letting the model think longer, without retraining anything.

The curve rises, but with plainly diminishing returns. o3 on ARC-AGI goes from 75.7% to 87.5% by spending about 170 times more per task - real gains, at real cost.^[6] There are two knobs for spending that compute.^[1]

Thinking budgets: the engineer's dial

Because thinking costs tokens, most reasoning APIs let you cap it. With Claude's extended thinking you set a budget_tokens (which must be less than max_tokens). The model emits thinking blocks before the final text, and - this is the gotcha - you are billed for every thinking token it generates, even though the API may return only a shorter summary of them.^[7]

# Standard model: answer appears immediately, low latency
resp = model.generate(prompt, temperature=0.7, top_p=0.9)

# Reasoning model: private thinking first, then the answer
resp = client.messages.create(
    model="claude-...",
    max_tokens=16000,
    thinking={"type": "enabled", "budget_tokens": 10000},  # < max_tokens
    messages=[{"role": "user", "content": "Prove ..."}],
)
# resp.content = [ {type:"thinking", ...}, {type:"text", ...} ]
# Billed for ALL thinking tokens, even if 'thinking' is only summarized.

Match the budget to difficulty: small budgets for easy questions, large for genuinely hard ones. The budget is a cap, not a target - Claude often will not use the whole thing (especially above about 32k tokens), and newer models expose an adaptive effort control that lets the model decide how much to think for you.^[7] For agents that call tools, interleaved thinking lets the model reason between tool calls and over their results - keep its thinking blocks in the loop so the reasoning stays continuous.

1. Set a floor by task. Give hard, multi-step requests a generous budget and easy ones little or none. Route by request type rather than using one setting for all.
2. Watch the curve. Measure accuracy as you raise the budget on your own evals. Stop near where the gains flatten instead of maxing it out by default.
3. Cap for safety. A hard ceiling stops a pathological input from spending an unbounded pile of tokens on a single question.

When reasoning helps, and when it hurts

Reasoning models are not a free upgrade. More thinking means more latency (seconds to minutes), more cost, and a new failure mode: overthinking trivial questions or underthinking hard ones. Choosing the right tool per task is the actual engineering skill here.

And the limits are genuinely contested. Apple's "Illusion of Thinking" (June 2025) reported that reasoning models hit a complexity threshold beyond which accuracy collapses and the models paradoxically think less.^[9] A rebuttal showed much of that "collapse" was an experimental artifact - puzzles that exceeded output-token limits or were mathematically unsolvable.^[10] The honest takeaway: real limits exist, and benchmark design is treacherous. Hold both.

What this means for agents

An agent runs the model in a loop, so every reasoning choice multiplies. A reasoning model deciding which tool to call next is often worth it, because a bad tool choice wastes a whole loop iteration and its observed result. But turning maximum thinking on for every step of a long trajectory is how cost and latency quietly explode. The lever you actually pull in production is per-step: cheap fast passes for routine steps, and a deep reasoning pass reserved for the genuinely hard decisions.

Check yourself