DPO & Preference Optimization

← RLHF (Reinforcement Learning from Human Feedback)GraphRAG →

Direct Preference Optimization tunes a model on pairs of preferred and rejected answers with a simple classification-style loss, no reward model and no RL loop. It made preference tuning accessible to anyone with a GPU, and most open models now align with DPO or one of its descendants.

L7 · LLM and generative AI engineerDevelopers Tech leaders QA and testing

Key terms

Preference pairsImplicit rewardReference modelDPO vs RLHFChosen and rejected

Learn these first

Where you meet it in the real world

Alignment of most open-weights chat models, style tuning, harmlessness training on a budget

Videos

▶ Stanford CS234 I Guest Lecture on DPO: Rafael Rafailov, Archit Sharma, Eric Mitchell I Lecture 9 ↗

Stanford Online · YouTube

Guides and articles

The RLHF Book ↗

Nathan Lambert