Alignment: RLHF, reward models, and DPO

Turn a fluent-but-aimless base model into a helpful assistant by teaching it human taste - and keeping it on a short leash.

A base language model has read the internet and can continue any text - but it has no idea it is supposed to be helpful. Ask it a question and it might answer, or it might write three more questions, or drift into an ad. Alignment is the step that turns raw fluency into an assistant. The catch: "a good, helpful, honest answer" is not something you can write down as a loss function. So we do something sneakier - we let humans compare answers, learn a model of what they prefer, and steer the language model toward it. This lesson builds that loop from scratch: preference pairs, a reward model, RLHF with a KL leash, and the closed-form shortcut called DPO.

Preferences, not scores

Ask ten people to rate an answer from 1 to 10 and you get ten incompatible scales - one person's 7 is another's 4. Ask instead which of two answers is better and they mostly agree. Comparisons are cheap, stable, and human-friendly. So the raw material of alignment is the preference pair: one prompt, two candidate responses, and a human who picks a winner. We write the winner as $y_w$ and the loser as $y_l$. Collect tens of thousands of these pairs and you have a dataset that encodes taste without ever naming a numeric target.

The reward model: a learned taste

A number that says "how good is this answer" is exactly a learned function from Lesson 2: feed it a response, out comes a scalar reward $r_{\varphi }(y)$. We fit its parameters $\varphi$ by gradient descent so that on every recorded pair the winner outscores the loser. The reward model never sees an absolute "correct score" - it only ever learns from comparisons, using the Bradley-Terry loss in the math box below. Once trained, it can score answers it has never seen, standing in for a human labeler at scale.

PPO and the KL leash

Now improve the policy - the language model itself. Sample answers, score them with the reward model, and push the model to produce higher-reward answers. (PPO, Proximal Policy Optimization, is the reinforcement-learning algorithm that does the pushing; the RL machinery is not the point here - the objective is.) Left unchecked, the policy learns to exploit the reward model: it discovers that padding, flattery, or one favorite phrase reliably scores high even when the answer got worse. That is reward hacking - Goodhart's law, "a measure that becomes a target stops being a good measure." The fix is a leash: penalize drifting away from the frozen base model ${\pi }_{\text{ref}}$, measured as KL divergence, with strength $\beta$. It is the same idea you met as overfitting control - stay close to something already trusted. Play the labeler below and feel the leash pull.

Notice the trap the demo makes visible. Keep preferring the longer, friendlier reply and the reward model happily concludes "more is better." With a loose leash the policy chases that signal all the way into gushing, emoji-laden padding - the reward-model bar climbs while the hidden true-helpfulness bar collapses. Tighten $\beta$ and the leash drags the policy back near the base, trading a little reward for staying sane. That balance - enough optimization to become helpful, enough leash to stay grounded - is the whole art of RLHF.

DPO: the reward model in closed form

Training a separate reward model and then running an RL loop is fiddly and unstable - two moving models, reward-hacking to police, hyperparameters everywhere. Direct Preference Optimization (DPO) asks: what if the policy is its own reward model? The RLHF objective has a known closed-form optimum, and inverting it shows the reward is already implicit in the policy as $\beta \log (\pi /{\pi }_{\text{ref}})$. Substitute that into Bradley-Terry and the reward model cancels out - leaving one plain supervised loss over the preference pairs, with the KL leash baked in through $\beta$. No reward model, no RL, same math. It is simpler, steadier, and now the default recipe for many open models.

# Reward model: Bradley-Terry loss on one preference pair (y_w beat y_l)
def rm_loss(r, y_w, y_l):
    return -log(sigmoid(r(y_w) - r(y_l)))     # push winner's reward above loser's

# DPO: no reward model, no RL loop. beta is the same KL leash;
# ref is the frozen base model, pi is the policy we are training.
def dpo_loss(pi, ref, y_w, y_l, beta):
    lw = logp(pi, y_w) - logp(ref, y_w)     # how far the policy moved on the winner
    ll = logp(pi, y_l) - logp(ref, y_l)     # ...and on the loser
    return -log(sigmoid(beta * (lw - ll)))     # raise winner, lower loser, leash by beta

Check yourself