RLHF: learning from human feedback
How people's choices between answers become a reward that steers a model.
Read first:Fine-tuning and LoRANext-token prediction
Step 1 of 6· Illustrative answers, scores and quality; the sigmoid, PPO clipping and KL-optimal policy maths is real
A pretrained model continues text; it does not follow instructions or know which of two answers people would find more helpful. Reinforcement learning from human feedback adds that in three stages. First, fine-tune the model on example answers written by people. Second, show people several answers to one prompt, ask which they prefer, and train a reward model to predict those choices as a score. Third, let the model write answers, score them with the reward model and update it with reinforcement learning, usually PPO, while a penalty keeps it close to where it started, so it does not drift into answers the reward model cannot judge well.
Why it matters for your product
RLHF is a large part of why assistants answer the question asked and keep a consistent tone, and in the InstructGPT study it mattered more to human raters than a hundredfold increase in model size. It has a weak point, though: the model learns whatever the reward model rewards, including its mistakes, and pushing too hard makes real quality fall. For teams, the preference data and reward signal define the behaviour: know who labelled it, and test the behaviours you care about directly instead of trusting a reward score.
For engineersShow the maths
loss = −log σ(r(x, y_w) − r(x, y_l))
Train the reward model so the answer people preferred, y_w, scores higher than the one they rejected, y_l; σ turns the score gap into a probability.
Worked example: A gap of 2 gives σ(2) ≈ 0.88 and a loss of −ln 0.88 ≈ 0.13; a gap of 0 gives a loss of 0.69.
R(x, y) = r(x, y) − β · log[π(y|x) / π_SFT(y|x)]
The reward PPO maximises is the reward model's score minus a penalty for making an answer more likely than the reference model did.
Worked example: If the policy makes an answer 3 times likelier than the reference and β = 0.1, the penalty is 0.1 × ln 3 ≈ 0.11.
L_CLIP = E[ min(ρ · Â, clip(ρ, 1 − ε, 1 + ε) · Â) ], ρ = π_new / π_old
PPO rewards making better-than-expected answers (positive advantage Â) more likely, but gives no extra credit beyond a ratio of 1 + ε, so one batch cannot move the policy too far.
Worked example: With ε = 0.2, raising an answer from 10% to 15% (ratio 1.5) earns no more credit than raising it to 12%.
The reward model is only a learned stand-in for human judgement: optimised hard enough, its scores and people's real preferences come apart. RLHF also inherits the preferences and blind spots of the people who wrote and ranked the answers, and can cost performance on other tasks, an effect InstructGPT called the alignment tax.
The words you will hear
SFT model
Primary sources
- 01Training language models to follow instructions with human feedbackOuyang et al. · 2022
- 02Learning to summarize from human feedbackStiennon et al. · 2020
- 03Deep reinforcement learning from human preferencesChristiano et al. · 2017
- 04Proximal Policy Optimization AlgorithmsSchulman et al. · 2017
- 05Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafailov et al. · 2023
Connected ideas
Training and alignment
DPO and GRPO
Two simpler routes from preferences and checkable answers to a better model.
ExploreObservability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreInterpretability and safety
Hallucination
Why models state wrong answers with confidence, and how to trade a few answers for a lot of trust.
ExploreTraining and alignment
Fine-tuning and LoRA
Teaching a pretrained model a new job by training a small add-on instead of every weight.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.