Evals
How to know a change made your AI system better, before your users find out it did not.
Step 1 of 6· Illustrative golden set, replies and scores; the gate logic and the swap test follow the cited practice
An eval is a test suite for an AI system. You collect real cases, each with an input and a way to check the answer, and run every version of the system against them. Some checks are plain code: did it quote the right price, call the right tool, keep card numbers out of the reply? Open-ended answers, such as tone or policy explanations, are graded by a second model against a written rubric. Comparing versions case by case shows what a change fixed and what it broke. A release gate then turns the results into a decision, so a better average cannot hide a broken safety rule.
Why it matters for your product
Without evals, every prompt tweak or model upgrade is a guess, and regressions reach customers first. A set of real cases with automated checks runs on every change, the way unit tests do for ordinary code. Keep must-pass rules separate from the average, check your judge against human labels, and add every real failure you see as a new case. The discipline matters more than any particular tool.
For engineersShow the maths
pass rate = cases passed ÷ cases run
The headline number: the share of golden-set cases whose check says pass.
Worked example: v1 passes 19 of 24 = 79.2%; v2 passes 21 of 24 = 87.5%, a rise of 8.3 points.
net change = cases fixed − cases broken
The pass rate only shows the net. The case-by-case diff shows both sides, and a single broken must-pass case outweighs any net gain.
Worked example: v2 fixed 4 cases and broke 2: net +2, with the safety case X3 among the broken.
judge agreement = cases where judge and humans agree ÷ cases both graded
How often the judge reaches the same verdict as people on a sample you have labelled by hand.
Worked example: 42 of 50 cases agree, so agreement is 84%.
A golden set measures only what it contains: a single held-out score tends to overestimate how well a model works in the wild, and if test cases leak into a model's training data, scores rise without any real improvement. With a few dozen cases, small differences between versions are mostly noise, which is why eval results need error bars.
The words you will hear
Golden set
Primary sources
- 01Judging LLM-as-a-Judge with MT-Bench and Chatbot ArenaZheng et al. · 2023
- 02Define success criteria and build evaluationsAnthropic · 2026
- 03Evaluating Large Language Models Trained on CodeChen et al. · 2021
- 04Beyond Accuracy: Behavioral Testing of NLP Models with CheckListRibeiro et al. · 2020
- 05NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each BenchmarkSainz et al. · 2023
- 06Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsMiller · 2024
Connected ideas
Observability
Error bars on evals
Why 82% against 78% on 100 questions can mean nothing, and how many questions you need before it does.
ExploreInterpretability and safety
Hallucination
Why models state wrong answers with confidence, and how to trade a few answers for a lot of trust.
ExploreInterpretability and safety
Prompt injection
When text the model reads on your behalf starts giving it orders.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.