RLHF and DPO: Aligning Models to Be Helpful

A pretrained causal LM completes text. Base models are talented but not particularly cooperative — they don't follow instructions, they don't refuse harmful requests, they don't behave as a helpful assistant. Alignment is the post-training stage that turns a base model into something you'd want to talk to.

RLHF (Reinforcement Learning from Human Feedback)

The classic recipe (Ouyang et al., 2022, behind ChatGPT) has three stages:

Supervised fine-tuning (SFT). Fine-tune the base model on curated demonstrations — humans writing what good answers look like.
Reward model. Show humans pairs of completions for the same prompt; have them pick the better one. Train a small model to predict human preferences from completion pairs.
PPO (Proximal Policy Optimization). Use the reward model as a reward signal in reinforcement learning. The policy (the LLM) generates completions; the reward model scores them; PPO updates the policy to increase reward, with a KL-divergence penalty against the SFT model to prevent reward hacking.

DPO (Direct Preference Optimization)

DPO (Rafailov et al., 2023) skips the explicit reward model and PPO loop. The insight: you can derive a closed-form objective that directly optimizes the policy on preference pairs. Loss is:

L = -log σ(β × (log P(y_w|x) - log P_ref(y_w|x) - log P(y_l|x) + log P_ref(y_l|x)))

where y_w is the preferred completion, y_l is the dispreferred, and P_ref is the SFT-only model's probability. Simpler, more stable, less compute. Matches or exceeds PPO quality on most benchmarks. Used by Llama 3 fine-tunes, Mistral Instruct variants, and a large fraction of community chat models.

Code

DPO loss in PyTorch·python

def dpo_loss(policy_logits_w, policy_logits_l,
             ref_logits_w, ref_logits_l,
             beta=0.1):
    # logits_*: (B, L, vocab); we need log-probs of the actual tokens.
    # Pseudocode — assume you have already gathered:
    #   policy_logp_w  = log P(y_w | x, policy)   # (B,)
    #   policy_logp_l  = log P(y_l | x, policy)
    #   ref_logp_w     = log P(y_w | x, reference)
    #   ref_logp_l     = log P(y_l | x, reference)
    diff = beta * ((policy_logp_w - ref_logp_w)
                   - (policy_logp_l - ref_logp_l))
    return -torch.nn.functional.logsigmoid(diff).mean()
# That's it. No reward model, no value head, no PPO bookkeeping.

RLHF and DPO: Aligning Models to Be Helpful

RLHF (Reinforcement Learning from Human Feedback)

DPO (Direct Preference Optimization)

Code

External links

Exercise

Progress

Comments 0