Error bars on evals
Why 82% against 78% on 100 questions can mean nothing, and how many questions you need before it does.
Read first:Evals
Step 1 of 6· Illustrative accuracies of 78% and 82% with simulated re-runs; the error-bar maths is real
An eval score is a measurement taken on a sample of questions, so it wobbles. Score the same model on a different 100 questions and 78% could easily become 74% or 83%. The standard error says how big that wobble is, and it shrinks only with the square root of the number of questions: four times the questions halves the error bar. So when two models score 78% and 82%, the honest question is whether the 4-point gap is bigger than the noise. Often it is not until you have run many hundreds of questions. Scoring both models on the same questions helps a great deal.
Why it matters for your product
Teams switch models and prompts on eval gaps of a few points. On a 100-question eval, such gaps are usually noise, so decisions flip back and forth and real regressions slip through. Report every score with its error bar, compare models on the same questions with a paired test, and size the eval before you run it: detecting a 4-point gap reliably can take well over a thousand questions. It is cheaper to add questions than to ship the wrong model.
For engineersShow the maths
SE = √(p × (1 − p) / n), 95% interval = p ± 1.96 × SE
For pass or fail scoring, the wobble depends only on the score p and the number of questions n.
Worked example: p = 0.78, n = 100: SE = √(0.1716 / 100) = 0.041, so the interval is 78% ± 8.1 points.
SE_diff = √(SE_A² + SE_B²) unpaired; √((Var_A + Var_B − 2 × Cov_AB) / n) paired
The gap between two scores has its own standard error. Scoring both models on the same questions subtracts their covariance, which shrinks it.
Worked example: At n = 1,000: SE_diff = √(0.3192 / 1,000) = 1.79 points, so the gap is +4 ± 3.5, from +0.5 to +7.5.
n = (z₀.₀₂₅ + z₀.₂₀)² × (Var_A + Var_B − 2 × Cov_AB) / δ²
How many questions an eval needs to detect a true gap δ with 80% power at the 5% significance level.
Worked example: δ = 0.04, unpaired: (1.96 + 0.84)² × 0.3192 / 0.0016 ≈ 1,566 questions. Paired at a correlation of 0.5: about 786.
These formulas assume questions are independent draws: related questions, such as several about one passage, need clustered standard errors, and below a few hundred questions normal-approximation error bars come out too narrow. No amount of statistics rescues a test set that does not resemble real use.
The words you will hear
Standard error
Primary sources
- 01Adding Error Bars to Evals: A Statistical Approach to Language Model EvaluationsMiller · 2024
- 02Position: Don't Use the CLT in LLM Evals With Fewer Than a Few Hundred DatapointsBowyer et al. · 2025
- 03With Little Power Comes Great ResponsibilityCard et al. · 2020
Connected ideas
Observability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreGenerating text
Sampling and decoding
How a list of probabilities becomes one word, and the dials that trade steadiness for variety.
ExploreInterpretability and safety
Hallucination
Why models state wrong answers with confidence, and how to trade a few answers for a lot of trust.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.