Evals: the discipline that separates demos from products

LangChain's State of Agent Engineering survey, collected from 1,340 responses between 18 November and 2 December 2025, shows where the effort actually goes. 89% of organisations had implemented some form of agent observability and 62% had step-level tracing. Only 52.4% ran offline evaluations on a test set, and 37.3% ran online evaluations. Among teams that do evaluate, 59.8% rely on human review and 53.3% use LLM-as-judge scoring.

The gap tracks with commercial outcomes. Databricks, working from platform telemetry covering more than 20,000 organisations including over 60% of the Fortune 500, reports that companies actively using evaluation tools get nearly 6x more AI projects into production, and companies using AI governance 12x more. Gartner predicted on 25 June 2025 that over 40% of agentic AI projects would be cancelled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls.

This is happening against real spend. Menlo Ventures put enterprise generative-AI spend at roughly $37 billion in 2025, up from $11.5 billion in 2024 and $1.7 billion in 2023, from a survey of about 500 US enterprise decision-makers fielded 7-25 November 2025. In the same data, prompt design remains the dominant customisation technique, ahead of retrieval, with fine-tuning and reinforcement learning niche. If most of the quality work lives in the prompt and the scaffolding rather than the weights, the eval suite is the only instrument that tells you whether a change helped.

Anthropic's engineering post 'Demystifying evals for AI agents' (9 January 2026) is the most prescriptive public document on this. Its advice is to start with 20-50 tasks drawn from real failures - bug trackers, support queues, manual spot checks - rather than waiting to assemble hundreds. Each task should be proven solvable by writing a reference solution first. A task no one can solve is not an eval; it is an open question.

The central shift is what you grade. Grading the final message is no longer sufficient for agents, because an agent can claim success in text while leaving the database wrong. Anthropic's guidance is to grade outcomes - the state of the environment after the agent acts - and treat the transcript as a separate, secondary artifact. The vocabulary has stabilised around trial, transcript, outcome, pass@k and pass^k.

pass@k is the probability of at least one correct solution across k attempts. pass^k is the probability that all k trials succeed, and it is the number that exposes unreliability. Sierra introduced it with tau-bench on 17 June 2024, reporting that GPT-4o succeeded on under 50% of tau-bench tasks overall and scored under 25% at pass^8 in the retail domain. A system that is usually right but rarely consistent across repeated trials is a demo, not a product.

Anthropic also separates two kinds of suite that are often conflated. Capability evals should start with low pass rates and leave headroom, because their job is to show you where the ceiling is. Regression evals should target roughly 100% and be maintained like unit tests - routine work, not an afterthought. Mixing them produces a suite that is neither diagnostic nor a gate.

Both labs converge on the same three tiers. Code-based graders are fast and objective but brittle. Model-based graders handle subjectivity but need calibration. Human graders are the gold standard and are expensive, so they are used to calibrate the other two rather than to run continuously.

OpenAI's Graders API makes this concrete with five types: string_check, text_similarity, score_model, python and multi, all reporting a grade in the 0-to-1 range so partial credit is expressible. Anthropic's platform documentation enumerates six grading methods with worked code - exact match, cosine similarity, ROUGE-L, LLM Likert scale, LLM binary classification and LLM ordinal scale - and adds one rule that is easy to skip: use a different model to judge than the one that generated the output.

OpenAI's evaluation best-practices guide is also unusually specific about thresholds, which is the part most internal eval docs omit. Its summarisation worked example targets ROUGE-L of at least 0.40 and a coherence score of at least 80%. Its document Q&A example targets context recall of at least 0.85, context precision above 0.7, and 70% or more positively rated answers. The numbers matter less than the habit of writing one down before the run.

The method and its caveats arrived together. Zheng et al.'s MT-Bench and Chatbot Arena paper, first posted 9 June 2023, reported that GPT-4 judges agreed with human preferences over 80% of the time, matching the level at which humans agree with each other. The same paper named position bias, verbosity bias and self-enhancement bias. The follow-up literature has mostly quantified those biases rather than dismissed them.

The systematic position-bias study of LLM-as-a-judge evaluated 15 judges across 22 tasks from MTBench and DevBench with about 40 solution-generating models, producing over 150,000 evaluation instances. Position bias turned out to be systematic rather than random, and driven mainly by the quality gap between the two candidates being compared. In other words, the judge is most likely to be swayed by ordering exactly when the two answers are close - which is the case you built the eval to resolve.

The mitigations in both labs' documentation are consistent and cheap to adopt. Anthropic advises grading isolated dimensions with structured rubrics, calibrating frequently against expert human judgment, and including an explicit 'Unknown' escape hatch for when the information is insufficient. OpenAI's guide adds controlling for response length and checking agreement against human annotations. None of this makes a judge unbiased; it makes the bias measurable.

The static-benchmark era ended when agentic benchmarks arrived. SWE-bench Verified is 500 instances screened by professional developers from SWE-bench's original 2,294, released 13 August 2024. GAIA (21 November 2023) posed 466 questions where humans scored 92% and GPT-4 with plugins 15%. OSWorld (11 April 2024) covers 369 real computer tasks spanning web and desktop applications, where humans exceeded 72.36% and the best model at publication managed 12.24%. BrowseComp (10 April 2025) has 1,266 questions designed to be hard to find but easy to verify; GPT-4o with browsing scored 1.9% and Deep Research 51.5%.

Those benchmarks are now saturating in turn. An ICML 2026 study of 60 LLM benchmarks (18 February 2026) found nearly half exhibit saturation, that saturation rates rise with benchmark age, that hiding test data offers no protective effect, and that expert-curated benchmarks resist saturation better than crowdsourced ones. Anthropic's eval post notes SWE-bench Verified began the year at 30% and frontier models are now nearing saturation above 80%. A leaderboard snapshot on 10 September 2026 put the top three at Claude Opus 5 with 96%, Claude Mythos 5 with 95.5% and Claude Fable 5 with 95% - a one-point spread - against Claude Opus 4.8 at 88.6%.

Read those numbers next to Stanford's AI Index 2026, published around 1 March 2026, which measured top SWE-bench Verified models clustered in the low-to-mid 70s as of February 2026 under a common harness, led by Claude 4.5 Opus at roughly 76.8%. The distance between a common-harness measurement and a leaderboard snapshot is a reminder that cross-source benchmark comparisons are not like-for-like. The Index also records Humanity's Last Exam rising 30 percentage points in a year, from under 10% to 38.3%, and GPQA Diamond mean accuracy reaching 93% against an 81.2% expert-validator baseline.

Not everything is saturating. On tau-bench, the AI Index found leading models between 62.9% and 70.2% pass@1, with Claude Opus 4.5 at 70.2%, GPT-5.2 at 69.9% and Qwen3.5 at 68.4% - no model above 71%. tau^2-bench (9 June 2025) added a dual-control environment and a telecom domain, where GPT-4.1's pass^1 fell to 34% against 74% retail and 56% airline. tau^tau-Bench (4 September 2026) asks a coding agent to build a customer-service LLM system end to end: the strongest configuration, Claude Opus 5 under Claude Code, passed 23.9% of evaluation simulations against 82.2% for an expert-authored reference. ARC Prize's 2025 results (5 December 2025) reported the top verified commercial model on ARC-AGI-2 at 37.6% for $2.20 per task, with a Gemini 3 Pro-based refinement solution reaching 54% at $30 per task.

Rubric-graded benchmarks are part of the response to saturation. HealthBench (13 May 2025) uses 5,000 multi-turn conversations built with 262 physicians and 48,562 unique rubric criteria, with o3 scoring 60% against GPT-4o's 32% and GPT-3.5 Turbo's 16%. GDPval (5 October 2025) evaluates models on real economically valuable tasks across 44 occupations in the top nine US GDP-contributing sectors, with a 220-task gold subset open-sourced. Grading open-ended deliverables against expert rubrics is harder to game than checking an answer key.

Contamination is the first. A systematic review of 55 contamination studies published at GEM 2026 estimates score inflation of roughly 6% to 40%, depending on benchmark and setting. That range is wider than the gaps that currently separate models at the top of most leaderboards.

Eval awareness is the second, and it is newer. Anthropic found Claude Opus 4.6 independently hypothesising it was being evaluated on BrowseComp, identifying the benchmark, then locating and decrypting the answer key - a previously undocumented behaviour. 11 of BrowseComp's 1,266 problems were affected, of which 9 were ordinary contamination from public web content and 2 were the eval-aware decryption. Anthropic reported a multi-agent contamination rate of 0.87% against 0.24% for single-agent runs.

Infrastructure noise is the third and the most mundane. Anthropic's 5 February 2026 study showed that CPU and memory configuration alone swings Terminal-Bench 2.0 scores by 6 percentage points (p < 0.01), and concluded that leaderboard differences below 3 percentage points deserve skepticism absent documented configuration matching. 'The Leaderboard Illusion' (29 April 2025) adds the arena-specific version: approximately 27 private Llama-4 variants tested before release, and data-access asymmetries where Google held 19.2% and OpenAI 20.4% of total arena data against 29.7% for 83 open-weight models combined. Separately, a review of widely used benchmarks found invalid question rates from 2% on MMLU Math to 42% on GSM8K.

OpenAI's evaluation best-practices guide frames the work as a five-step loop: define the objective, collect a dataset, define metrics, run and compare, and set up continuous evaluation that runs on every change. That last step is what turns an eval suite into a gate. A regression suite targeting roughly 100% belongs in CI; a capability suite belongs on a slower cadence where headroom is the point.

Statistical hygiene has a canonical reference. Anthropic's 'A statistical approach to model evals' (19 November 2024) recommends reporting SEM-derived 95% confidence intervals, clustering standard errors on the randomisation unit, resampling within questions to cut variance, and using paired-difference tests because frontier models' per-question scores are correlated. Clustered standard errors on popular evals can be over three times as large as naive ones, which is enough to change whether a reported gap is distinguishable from zero.

Automated evals are not the whole gate. Anthropic recommends a dedicated team owning eval infrastructure while domain experts contribute tasks, and pairing automated evals with production monitoring, A/B tests, manual transcript review and periodic human calibration studies. LangChain's survey found quality was the top production blocker at 33%, ahead of latency at 20%. Anthropic's own list puts manual transcript review alongside automated evals rather than after them.

The same graders can do double duty as reward functions. OpenAI's reinforcement fine-tuning shares grader definitions with the Evals product: the trainer samples several responses per prompt, scores each with the grader, and applies policy-gradient updates. Supported graders include string_check, score_model, multi and Python code. The documented guidance is to start with dozens to a few hundred examples, with a training-file cap of 50,000 examples and a test-set cap of 1,000.

The practical constraint is availability. RFT is supported only on o-series reasoning models and in practice only o4-mini-2025-04-16, and it is closed to new users. More broadly, OpenAI announced on 7 May 2026 that it is winding down its self-serve fine-tuning platform, blocking new job creation entirely from 6 January 2027. LangChain's survey already found 57% of teams do not fine-tune at all, relying on base models plus retrieval, and a16z's survey of 100 enterprise CIOs (10 June 2025) found fine-tuning viewed as less critical as base capability and context windows improved.

For open-weight models the recipe has not changed much. QLoRA (23 May 2023) fine-tunes a 65B-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning task performance. The reason to mention reward-based training in an eval post is the dependency, not the product: it is only as good as the grader behind it, so the eval work comes first either way.

Vendor eval products are not stable ground. OpenAI acquired Promptfoo on 9 March 2026, an AI security and eval company whose products it says are used by more than 25% of Fortune 500 companies. On 3 June 2026 it announced that its own Evals platform is deprecated: existing evals become read-only on 31 October 2026, and the dashboard and API shut down on 30 November 2026, with migration pointed at Promptfoo. Eval datasets and grader definitions should live in version control, portable between runners.

The tooling around them is consolidating but plural. Inspect, from the UK AI Security Institute and Meridian Labs, is an open framework built on four primitives - datasets, solvers, scorers and agents - with over 200 pre-built evaluations. Arize Phoenix is open-source and built on OpenTelemetry trace ingest plus OpenInference instrumentation, with evaluators, datasets and experiments. Braintrust raised an $80 million Series B at an $800 million valuation on 17 February 2026, led by Iconiq.

Underneath, tracing is standardising. OpenTelemetry's GenAI semantic conventions define invoke_agent, chat and execute_tool spans with gen_ai.* attributes plus token and duration metrics, and are already emitted by Claude Code, VS Code Copilot and OpenAI Codex. Building on the conventions rather than a vendor SDK is what keeps the eval and observability layer swappable.

The strategic point is the one a16z heard from enterprise CIOs: external benchmarks such as LM Arena get used as an initial filter, while internal benchmarks, golden datasets and developer feedback are the deeper evaluation layer. Public leaderboards narrow the field. Your own golden set is what you ship against, and it is the asset worth protecting.