Training agents and agentic RL

Almost every agent you build should be prompted, not trained. But the reasoning models underneath them, o1 and DeepSeek-R1, were made by reinforcement learning. This lesson demystifies how: the training ladder, RLHF versus verifiable rewards, the GRPO algorithm behind R1, and what "agentic RL" adds when you optimize a whole tool-use trajectory instead of one reply.

First decision: train or prompt?

Training changes weights; prompting changes context. Both change behavior, but they are not interchangeable. Prompting, tools, and retrieval are cheap, instant, and endlessly revisable. Fine-tuning is a bet that the task will not move. Make that bet only when the task is stable, high-volume, and well-specified, and you already have prompts that work and clean examples to learn from.

Meta deliberately post-trained Llama 3 with SFT plus rejection sampling plus DPO rather than full RL, calling RL "less stable and harder to scale" at the time.^[7] Even frontier labs climb the cheap rungs first.

The training ladder

"Training" is not one thing. It is a ladder of techniques, roughly ordered by cost and instability. Frontier post-training pipelines chain several of them; you rarely pick just one.

1. SFT / behavior cloning. Train on (input, ideal output) pairs with cross-entropy loss. For agents, the "outputs" are high-quality tool-use trajectories. It is imitation: the model can only be as good as its demonstrations, and it never sees the consequences of its own mistakes. Coverage and quality of the data are the ceiling.
2. Rejection sampling (best-of-N). Generate N candidates per prompt from the current model, score them with a verifier or reward model, keep only the winners, and SFT on those. You distill the model's own best behavior back into itself: cheap, stable, and a pillar of real pipelines.^[1]
3. Preference optimization (DPO). Learn from pairs (chosen, rejected) instead of a scalar reward. DPO reparameterizes the RLHF objective into a simple classification-style loss: no reward model, no sampling loop, no RL machinery. Stable and cheap, but limited to a static offline dataset.
4. Full RL (PPO, GRPO). The model generates, gets scored, and updates via policy gradient, a closed loop where it learns from the consequences of its own sampled behavior (on-policy). The only rung that can push behavior beyond the demonstration data, and the most expensive and least stable.

That last jump, from imitation to learning-from-consequences, is the whole story. Hold these two side by side, because everything below is a variation on the right-hand column.

Aligning behavior: RLHF, RLAIF, reward models

The technique behind the ChatGPT-era chat revolution is RLHF, and it has three stages:

1. SFT an instruction-following base model.
2. Train a reward model. Collect human pairwise comparisons (is A or B better?) and fit a reward model that outputs a scalar "how much a human would prefer this."
3. RL (PPO). Optimize the policy to maximize reward-model score, with a KL penalty to the SFT reference so it cannot drift into gibberish that games the score.

Human preferences are slow and expensive to collect, so RLAIF swaps the human labeler for a (usually stronger) model applying a written rubric: orders of magnitude cheaper, and consistent because the same principles apply every time. Anthropic's Constitutional AI is the canonical recipe: the model critiques and revises its own answers against a written "constitution," then judges which of two answers better follows it, producing an AI-labeled preference dataset to train on.^[6]

RLVR: rewards you cannot fake

In domains where correctness is objectively checkable, you can throw the learned reward model away and reward the policy directly from a deterministic verifier. This is RLVR, RL with Verifiable Rewards, and it is the paradigm behind modern reasoning models.

GRPO: the workhorse algorithm

GRPO (Group Relative Policy Optimization) is the de-facto RLVR algorithm, introduced by DeepSeek. Its key move: delete PPO's critic/value network (which is often as large as the policy, doubling memory). Instead of a learned baseline, it uses the model's own peers.^[5]

Formally: for prompt q, sample a group of G outputs, score each with the verifier to get rewards r1 ... r_G, and compute each output's advantage relative to its own group, then update with a PPO-style clipped objective:

A_i = ( r_i - mean(r_1 ... r_G) ) / std(r_1 ... r_G)

Here it is worked by hand. Prompt: "Compute 17 x 23", the verifiable answer is 391. Sample a group of G = 4:

Sampling G completions per prompt is the price you pay for skipping the critic. Now the full loop: the policy samples rollouts, the verifier scores them, group-relative advantages are computed, and the policy is updated, then it repeats.

How reasoning models are actually made (o1, R1)

This is the payoff. Earlier lessons showed you what reasoning models do; here is how they are built. OpenAI o1 was trained with large-scale RL "to think before answering," to generate a long internal chain of thought, then answer. Through RL it learns to refine its reasoning, catch its own mistakes, decompose hard steps, and switch strategy when one is not working. The headline: accuracy improves with both more training-time RL compute and more test-time thinking, making inference compute a genuine new scaling axis.^[3]

DeepSeek-R1 (Jan 2025) open-sourced a credible recipe and made two sharp points that demystify the whole thing:^[1]

V3 base --RL (GRPO, verifiable rewards)--> R1-Zero   # proof: reasoning emerges from pure RL - but messy

V3 base --cold-start SFT (~k readable CoTs)--> ckptA
ckptA   --reasoning RL (+ language-consistency reward)--> ckptB
ckptB   --rejection sampling--> 600K reasoning + 200K general SFT data
V3 base --SFT on that 800K--> ckptC
ckptC   --RL over ALL tasks (rule + preference rewards)--> R1

R1      --generate 800K samples --SFT--> distilled 1.5B / 7B / 8B / ... / 70B  # no RL in the small models

Classic RLHF gives one reward for a whole trajectory, so the model cannot tell which step was wrong. o1-style training is widely believed to lean on process reward models that score each reasoning step, plus searching over multiple candidate reasoning paths.^[9]

Agentic RL: optimizing the whole trajectory

Everything so far optimizes a single response. Agentic RL optimizes the entire tool-use trajectory. The cleanest mental model reframes what the LLM is:^[2]

This shift, from imitation (SFT on trajectories) to outcome-driven optimization (closed-loop RL), is what lets an agent discover when, how, and which tool to use, self-correct faulty tool calls, and compose tools, rather than parroting demonstrated patterns. Here is one RLVR trajectory for a search agent:

s0  user: "What year did the author of <book> win the Nobel?"
 a0 (text)  reason: "find the author, then their Nobel year"
 a1 (tool)  search("author of <book>")          -> obs: "<Author>"
 a2 (text)  reason: "now the Nobel year for <Author>"
 a3 (tool)  search("<Author> Nobel Prize year") -> obs: "1982"
 a4 (text)  answer: "1982"
 verifier: exact-match vs gold "1982" -> reward = 1   # else 0

Under SFT you would imitate one fixed gold trajectory. Under agentic RLVR you roll out many trajectories per question, reward only the final answer, and let GRPO raise the probability of whichever action sequences, which searches in which order, led to correct answers.

Distillation and the "does RL teach anything new?" debate

Distillation trains a small student on a stronger teacher's outputs. DeepSeek showed it is remarkably potent for reasoning: SFT'ing open models on 800K R1-generated samples produced strong reasoners with no RL at all, a distilled 7B hit 55.5% on AIME 2024, beating QwQ-32B-Preview.^[1] Practically, distillation is often the cheapest way to get most of a big RL run's gains into a small, fast production model.^[10]

Check yourself