Alignment: DPO, ORPO, SimPO

Beyond supervised fine-tuning

SFT teaches the model what to say. Alignment teaches it to prefer better responses over worse ones. Use alignment when your data is preference pairs (preferred / rejected) rather than instruction → response.

DPO (Direct Preference Optimization)

Train with pairs of (preferred, rejected) responses. Simpler than RLHF — no separate reward model needed. Requires a reference model (frozen base) for the KL constraint.

ORPO (Odds Ratio Preference Optimization)

Combines SFT and alignment into a single training step — no reference model. Faster and more memory-efficient than DPO. In TRL v1.0, available under trl.experimental.orpo.

SimPO (Simple Preference Optimization)

Uses average log probability as the implicit reward (no reference model). Reported to outperform DPO by ~6.4 points on AlpacaEval 2 with 20% less compute. Available via CPOTrainer with loss_type="simpo".

Method comparison

Method	Data format	Reference model?	Best for
SFT	instruction → response	No	Teaching specific behaviors
DPO	preferred / rejected pairs	Yes	Improving response quality
ORPO	preferred / rejected pairs	No	Single-stage alignment
SimPO	preferred / rejected pairs	No	Best quality, efficient

Code

DPO training in TRL·python

from trl import DPOConfig, DPOTrainer

# Data format: {"prompt": "...", "chosen": "...", "rejected": "..."}
dpo_config = DPOConfig(
    output_dir="./dpo-results",
    beta=0.1,                    # KL penalty strength
    learning_rate=5e-5,
    num_train_epochs=1,
    per_device_train_batch_size=2,
    gradient_accumulation_steps=4,
    bf16=True,
)

trainer = DPOTrainer(
    model=model,                 # the SFT-fine-tuned model
    ref_model=None,              # None = use frozen base for KL
    args=dpo_config,
    train_dataset=preference_dataset,
    processing_class=tokenizer,
    peft_config=peft_config,
)
trainer.train()

Beyond supervised fine-tuning

DPO (Direct Preference Optimization)

ORPO (Odds Ratio Preference Optimization)

SimPO (Simple Preference Optimization)

Method comparison

Code

External links

Exercise

Progress

Comments 0