Eval-driven development: the missing discipline in LLM applications

Teams shipping LLM features usually know something is wrong before they can say what. A prompt tweak improves one case and quietly breaks three others. A provider upgrades a model and answers get longer, or shorter, or subtly different. Spot checks in a chat window cannot catch this. Evaluation can.

Start with a golden set from real traffic

Pull a few hundred real inputs, group them by intent, and have domain reviewers write the expected output or the criteria a good output must meet. Include the ugly cases: ambiguous questions, adversarial inputs, empty context. This set is your definition of quality.

Calibrate the judge before you trust it

LLM-as-judge scoring scales, but only after you have checked that it agrees with humans. Score a sample both ways, measure agreement, and tune the rubric until it is high. We have found rubrics that seemed obvious disagreeing with reviewers on a third of cases before calibration.

Gate releases on the numbers

Run the set in CI on every change to a prompt, model, retrieval setting or tool schema. Set thresholds per feature. A change that drops faithfulness below the line does not ship, however good it looked in the demo.

Watch production with sampled evaluation

Offline sets go stale. Sample live traffic, score it with the calibrated judge, and alert on drift. Feed user corrections back into the golden set so it grows with the product.

One of our evaluation runs uncovered a 43-point accuracy gap between two models on a task everyone assumed was equivalent, and a batch of machine-generated ground-truth labels that had been skewing every previous result. That is the kind of thing you only find when measurement is part of the workflow.