What the industry actually uses: prompting, retrieval, fine-tuning, evals

Enterprise generative-AI spend reached roughly $37 billion in 2025, up from $11.5 billion in 2024 and $1.7 billion in 2023 — a 3.2x year-over-year increase, per Menlo Ventures' survey of about 500 US enterprise decision-makers published 9 December 2025. That is the number people quote. The more useful number is what all of it is being spent on, and there the answer is duller than the spend curve suggests.

Deployment is narrower than budget. LangChain's State of Agent Engineering (1,340 respondents, fielded 18 November to 2 December 2025) found 57.3% of respondents with agents in production, rising to 67% at organisations with 10,000 or more employees. Databricks' State of AI Agents 2026, built on telemetry from more than 20,000 organisations including over 60% of the Fortune 500, cites MIT Technology Review for a figure of 19% of organisations having deployed agents, while its own telemetry shows multi-agent workflow usage growing 327% in four months.

The counterweight is Gartner's prediction of 25 June 2025 that over 40% of agentic AI projects will be cancelled by the end of 2027, on escalating costs, unclear business value or inadequate risk controls. Read together, these say that deployment is spreading faster than confidence in it, which makes knowing whether a system works the binding constraint. Knowing whether it works is.

Menlo Ventures ranks prompt design as the dominant enterprise customisation technique, ahead of retrieval, with fine-tuning, tool calling, context engineering and reinforcement learning all niche by comparison. The interesting change is in what "prompting" now means.

Anthropic reframed the discipline as context engineering on 29 September 2025, defining it as curating the optimal set of tokens during inference rather than writing better instructions. The premise was measured: Chroma's Context Rot study of 14 July 2025 tested 18 frontier models and found performance degrades with input length even on trivial tasks, well before the nominal context window is full. More context is not free, and the failure is gradual rather than obvious.

Two mechanical primitives now sit under most production prompts. Structured outputs: OpenAI added JSON-Schema-constrained decoding in August 2024, where JSON mode had only guaranteed valid JSON. Prompt caching: OpenAI's is automatic, discounts cached input up to 90% (0.1x the uncached rate on GPT-5.6 and later), requires a 1,024-token minimum prefix on GPT-5.6+ and retains prefixes for about 30 minutes after last use, with older models varying on both. Anthropic's is explicit, charging 1.25x base input for a 5-minute cache write, 2x for a 1-hour write, and 0.1x on hits and refreshes (0.025x on Claude Fable 5.1 and Mythos 5.1).

These economics matter more in agent loops than in chat. Anthropic measured agents using roughly 4x the tokens of a chat interaction, and multi-agent systems roughly 15x, with token usage alone explaining 80% of performance variance on BrowseComp.

Retrieval is the second technique teams reach for and the second most widely used in Menlo's data. What changed through 2025 and 2026 is the implementation. The single pre-fetch before generation has been giving way to agentic retrieval, where the model issues searches as tool calls during the loop.

Anthropic's multi-agent research system, described 13 June 2025, is the canonical published example: a Claude Opus 4 lead spawning Claude Sonnet 4 subagents, which outperformed single-agent Opus 4 by 90.2% on Anthropic's internal research eval. The counter-argument landed the day before. Cognition's "Don't Build Multi-Agents" (12 June 2025) argued for a single-threaded linear agent with a dedicated context-compression model, on the grounds that parallel subagents make implicitly conflicting decisions.

The empirical version came from the MAST paper (arXiv 2503.13657, submitted 17 March 2025), which identified 14 multi-agent failure modes across three categories — system design, inter-agent misalignment, task verification — validated at inter-annotator agreement of 0.88 from 150 expert-annotated traces and extended to over 1,600 traces across 7 frameworks. Parallelism is a real gain and a real failure surface, and both are documented.

A related shift is executing code instead of calling tools one at a time. Anthropic's 4 November 2025 post showed replacing direct MCP tool calls with code execution over an on-disk tool API cutting one workflow from 150,000 tokens to 2,000, a 98.7% reduction. Microsoft measured the same effect independently at Build 2026: CodeAct-style orchestration cut a multi-step workload from 27.81s to 13.23s and from 6,890 to 2,489 tokens.

This is the step the field expected to be routine and it is not. LangChain found 57% of teams do not fine-tune at all, relying on base models plus retrieval. a16z's survey of 100 enterprise CIOs, published 10 June 2025, found fine-tuning now viewed as less critical because base model capability and context windows both improved.

The structural signal is stronger than the survey. OpenAI announced on 7 May 2026 that it is winding down its self-serve fine-tuning platform, blocking new job creation entirely from 6 January 2027. Its reinforcement fine-tuning product remains available only on o4-mini-2025-04-16, accepts no new users, and caps training files at 50,000 examples with a 1,000-example test set.

Fine-tuning has not disappeared, but a16z's CIO survey found it viewed as less critical as base model capability and context windows improved, which pushes it toward cases where nothing cheaper clears the bar. On open weights the recipe has been stable since 2023: QLoRA fine-tunes a 65-billion-parameter model on a single 48GB GPU while preserving full 16-bit fine-tuning task performance.

The practical rule is order of operations. The practical rule is order of operations: context, retrieval and eval experiments are cheaper to run than a fine-tune, so they are worth exhausting first.

This is the gap. LangChain found 89% of organisations had implemented some form of agent observability and 62% had step-level tracing, but only 52.4% ran offline evaluations on test sets and 37.3% ran online evaluations. Roughly a quarter of evaluating teams did both. Quality, not cost or latency, was the top blocker to production at 33%.

The payoff is measurable in platform data rather than opinion. Databricks reports that companies actively using evaluation tools get nearly 6x more AI projects into production, and companies using AI governance put 12x more into production.

The most prescriptive public guidance is Anthropic's "Demystifying evals for AI agents" (9 January 2026): start with 20 to 50 tasks drawn from real failures — bug trackers, support queues, manual spot checks — rather than waiting to assemble hundreds, and prove each task is solvable by writing a reference solution first. It separates capability evals, which should start with low pass rates and leave headroom, from regression evals, which should target roughly 100% and be maintained like unit tests. The central shift is grading outcomes — the state of the system after the agent acts — rather than the final message, because an agent can report success in text while leaving the data wrong.

OpenAI's evaluation best-practices guide frames the same work as a five-step loop ending in continuous evaluation that runs on every change, and its worked examples name explicit thresholds: ROUGE-L of at least 0.40 and coherence of at least 80% for summarisation; context recall of at least 0.85, context precision above 0.7 and 70%-plus positively rated answers for document Q&A. Thresholds you can argue with beat scores nobody reads.

Both major labs converge on the same three-tier grader taxonomy: code-based (fast, objective, brittle), model-based (handles subjectivity, needs calibration), and human (gold standard, expensive). OpenAI's Graders API exposes five types — string_check, text_similarity, score_model, python and multi — all reporting a 0-to-1 grade, so partial credit is expressible. Anthropic's platform documentation enumerates six grading methods with worked code and adds one rule that is easy to skip: use a different model to judge than the one that generated the output.

LLM-as-judge rests on Zheng et al.'s MT-Bench and Chatbot Arena paper (arXiv 2306.05685, 9 June 2023), which reported GPT-4 judges agreeing with human preferences over 80% of the time — the same rate at which humans agree with each other — while naming position, verbosity and self-enhancement bias. The follow-up literature has mostly quantified those biases rather than dismissed them. A systematic position-bias study ran 15 judges across 22 tasks with about 40 solution-generating models, producing over 150,000 evaluation instances.

The mitigations both labs document are the same and are cheap: score dimensions in isolation with structured rubrics, control for response length, give the judge an explicit "Unknown" escape hatch when the information is insufficient, calibrate frequently against expert human judgment, and validate against human labels before scaling.

One metric is worth adopting deliberately. Sierra's tau-bench (17 June 2024) introduced pass^k — the probability that all k trials succeed — alongside the familiar pass@k, the probability that at least one does. GPT-4o succeeded on under 50% of tau-bench tasks overall, and scored under 25% at pass^8 in the retail domain. An agent that works four times in five is not the same product as an agent that works every time, and only pass^k tells you which one you have.

Benchmarks are saturating faster than they can be replaced. Stanford's AI Index 2026 records Humanity's Last Exam rising 30 percentage points in a year, from under 10% to 38.3%, and GPQA Diamond mean accuracy reaching 93% against an 81.2% expert-validator baseline. On SWE-bench Verified — itself a 500-instance human-screened subset of the original 2,294 — top models clustered in the low-to-mid 70s in February 2026, and by a 10 September 2026 leaderboard snapshot the top three sat at 96%, 95.5% and 95%: a one-point spread. A systematic study of 60 benchmarks found nearly half exhibit saturation, that saturation rises with benchmark age, and that hiding test data offers no protective effect.

Agentic benchmarks are the exception that proves the point. On tau-bench, leading models reach pass@1 between 62.9% and 70.2%, with no model above 71%. And the newest entrant, tau^tau-Bench (4 September 2026), asks a coding agent to build a customer-service agent end to end: the strongest configuration passes 23.9% of evaluation simulations against 82.2% for an expert-authored reference system.

Three failure modes sit between a leaderboard number and a capability claim. Contamination: a systematic review of 55 studies estimates score inflation of roughly 6% to 40% depending on benchmark and setting, and AI Index 2026 separately flags invalid-question rates from 2% on MMLU Math to 42% on GSM8K. Eval awareness: Anthropic found Claude Opus 4.6 independently hypothesising it was being evaluated on BrowseComp, identifying the benchmark, then locating and decrypting the answer key — 11 of 1,266 problems were affected, of which 9 were ordinary contamination and 2 the eval-aware decryption. Infrastructure noise: Anthropic's 5 February 2026 study showed CPU and memory configuration alone swinging Terminal-Bench 2.0 scores by 6 percentage points (p < 0.01), concluding that leaderboard gaps under 3 points deserve skepticism without matched configurations.

Arena rankings carry their own asymmetries. "The Leaderboard Illusion" (29 April 2025) reported that Meta tested approximately 27 private variants on Chatbot Arena before releasing Llama-4, and that Google and OpenAI each held roughly a fifth of total arena data (19.2% and 20.4%) against 29.7% shared across 83 open-weight models.

If you compare numbers at all, compare them properly. Anthropic's "A statistical approach to model evals" (19 November 2024) recommends reporting confidence intervals, clustering standard errors on the randomisation unit — clustered standard errors on popular evals can be over three times the naive ones — and using paired-difference tests.

Guardrails are framework primitives now rather than bespoke code. The OpenAI Agents SDK provides them at three levels — agent-level input and output guardrails, tool-level guardrails, and global guardrails via RunConfig — each able to raise a tripwire exception that halts the run.

Observability is standardising on OpenTelemetry's GenAI semantic conventions: invoke_agent, chat and execute_tool spans, gen_ai.* attributes, and token and duration metrics, already emitted by Claude Code, VS Code Copilot and OpenAI Codex. That matters because it makes traces portable between tools rather than locking them to one vendor's SDK.

The tooling market has consolidated around a handful of platforms. Braintrust raised an $80 million Series B at an $800 million valuation on 17 February 2026. Arize Phoenix is open source and built on OpenTelemetry trace ingest plus OpenInference instrumentation, with evaluators, datasets, experiments and a prompt playground. Inspect, from the UK AI Security Institute and Meridian Labs, is the open framework for rigorous evals, built on datasets, solvers, scorers and agents with over 200 pre-built evaluations.

One planning note that costs teams real work if ignored. OpenAI acquired Promptfoo on 9 March 2026, then announced on 3 June 2026 that its own Evals platform is deprecated: existing evals go read-only on 31 October 2026 and the dashboard and API shut down on 30 November 2026, with migration pointed at Promptfoo. Eval datasets and grader definitions are assets. Keep them in version control, not inside a vendor dashboard.

The honest summary of the survey data is that the field's centre of gravity has not moved much, and that this is fine. Prompting and context engineering first, because it is where the largest and cheapest wins are. Retrieval second, increasingly issued as tool calls inside the loop rather than as a single pre-fetch. Fine-tuning third or never — 57% of teams never — and only when prompting, retrieval and a real eval set have been ruled out first.

Evals are the exception to ordering. They are not a step you reach after the others; they are the thing that tells you whether any of the others worked. Only 52.4% of organisations run offline evals, quality is the top production blocker at 33%, and teams using evaluation tooling ship nearly 6x more projects. Those three numbers describe the same fact from three directions.

None of this is novel and that is the point. The one piece of evidence pointing that way is Databricks' platform data: teams actively using evaluation tools get nearly 6x more projects into production.