TRL: RLHF, DPO, and Preference Optimization

~24 min · ops, trl, rlhf

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

TRL is the alignment toolkit

trl (Transformer Reinforcement Learning) is the canonical Python library for: SFT (we covered this), DPO (Direct Preference Optimization), PPO (Proximal Policy Optimization for RLHF), KTO, IPO, ORPO, GRPO, and other preference / RL methods. The dominant 2026 default for alignment is DPO — it skips the reward-model + PPO loop and trains directly on preference pairs.

DPO in 30 seconds

DPO takes pairs (prompt, chosen, rejected). The objective increases probability of chosen relative to rejected using a closed-form bound that mirrors RLHF without the RL. Training looks like SFT with a different loss; converges fast; doesn't need a separate reward model.

Code

DPO with trl on a preference dataset·python

# pip install trl
from trl import DPOTrainer, DPOConfig
from transformers import AutoModelForCausalLM, AutoTokenizer
from datasets import load_dataset
import torch

base = "Qwen/Qwen2.5-1.5B-Instruct"
tok = AutoTokenizer.from_pretrained(base)
model = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)

# Reference model (frozen) — DPO compares trained model's logprobs against this baseline
ref = AutoModelForCausalLM.from_pretrained(base, torch_dtype=torch.bfloat16)

# Preference dataset: each row has 'prompt', 'chosen', 'rejected'
ds = load_dataset("trl-lib/ultrafeedback_binarized", split="train")

cfg = DPOConfig(
    output_dir="./dpo-out",
    per_device_train_batch_size=2,
    num_train_epochs=1,
    learning_rate=5e-7,           # DPO lr is low (often 1e-6 to 5e-7)
    bf16=True,
    beta=0.1,                     # KL strength
)

trainer = DPOTrainer(model=model, ref_model=ref, args=cfg, tokenizer=tok, train_dataset=ds)
trainer.train()

External links

Exercise

Take an SFT-tuned model from the training track. DPO-tune it on a small preference dataset (e.g. trl-lib/ultrafeedback_binarized, 200 examples). Compare on 10 held-out prompts: SFT-only output vs SFT+DPO output. Note: which axes change (helpfulness, conciseness, refusal style).

Progress

Progress is local-only — sign in to sync across devices.

← PreviousOptimum: Hardware-Specific Optimization Next →End-to-End: Hub → Train → Serve → Space

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.