DPO and GRPO
Two simpler routes from preferences and checkable answers to a better model.
Read first:RLHF: learning from human feedback
Step 1 of 6· Illustrative answers, log-probabilities and training steps; the DPO loss, implicit reward and group-advantage maths is real; AIME figures from DeepSeek-AI (2025)
RLHF with PPO juggles several models at once: the one being trained, a frozen reference, a reward model and a value model. Two newer methods cut that down. Direct Preference Optimization (DPO) trains straight on pairs of answers where people preferred one: it raises the chosen answer's probability and lowers the rejected one's, both measured against the frozen reference, with no reward model and no sampling. Group Relative Policy Optimization (GRPO) keeps reinforcement learning but drops the value model: it samples a group of answers to each question, scores them, and pushes up the ones that beat the group's average.
Why it matters for your product
Both methods make preference and reasoning training cheaper and simpler to run. DPO needs only a dataset of chosen and rejected answers and fits into an ordinary fine-tuning pipeline, which makes it practical for adjusting tone, format or refusals. GRPO suits tasks whose answers can be checked automatically, such as maths, code with tests or structured output, because a rule is harder to game than a learned reward model. For a team, the deciding question is the signal: clean preference pairs for DPO, a reliable checker for GRPO.
For engineersShow the maths
L_DPO = −log σ( β·log[π(y_w) / π_ref(y_w)] − β·log[π(y_l) / π_ref(y_l)] )
Measure how far the policy has raised the chosen answer and lowered the rejected one, each against the frozen reference, and push that margin up.
Worked example: With β = 0.1, if the chosen answer's log-probability rose by 5 and the rejected one's fell by 5, the margin is 0.1 × 10 = 1 and the loss is −ln σ(1) ≈ 0.31.
Âᵢ = (rᵢ − mean(r)) / std(r)
An answer's advantage is how much better it scored than its own group, in units of the group's spread.
Worked example: In a group of 8 with 2 correct (reward 1) and 6 wrong (0): mean 0.25, std ≈ 0.43, so each correct answer gets +1.73 and each wrong one −0.58.
π*(y) ∝ π_ref(y) · exp(r(y) / β)
The best policy under a KL leash re-weights the reference by the exponentiated reward. DPO turns this around: any policy implies a reward, r = β·log(π / π_ref) plus a constant, so the policy can be trained directly.
DPO learns from a fixed set of pairs and never tries new answers during training, so it is only as good as those pairs and how closely they match what the model itself writes. GRPO needs a reward it can trust: on open-ended tasks with no checker it falls back on a learned reward model, with the same over-optimisation risk as RLHF.
The words you will hear
DPO
Primary sources
- 01Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafailov et al. · 2023
- 02DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language ModelsShao et al. · 2024
- 03DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement LearningDeepSeek-AI · 2025
- 04Training language models to follow instructions with human feedbackOuyang et al. · 2022
Connected ideas
Training and alignment
RLHF: learning from human feedback
How people's choices between answers become a reward that steers a model.
ExploreTraining and alignment
Fine-tuning and LoRA
Teaching a pretrained model a new job by training a small add-on instead of every weight.
ExploreObservability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreTraining and alignment
Distillation
Training a small model to imitate a large one, including what the large one nearly said.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.