Skip to content
Observability

Monitoring and drift

After launch the data, the documents and the model itself keep changing. Monitoring is how you notice before your customers do.

Intermediate · 7 steps

Read first:EvalsLLM observability

Step 1 of 7· Illustrative weekly metrics, histograms and traffic split; the PSI, burn-rate and sample-size maths is real

in one minute

An AI feature that scored well before launch does not stay that way on its own. New kinds of questions arrive, the documents it searches change, costs creep, and the model vendor retires the version you tested and replaces it with another. Monitoring means measuring the live system on a schedule: grading a sample of real conversations every week, watching retrieval quality, accuracy and cost per request, and comparing any new model or prompt on a small slice of traffic before everyone gets it. Alerts fire when a measure breaks an agreed target, and every failure found in production goes back into the test set so it cannot happen quietly twice.

Why it matters for your product

The most common way an AI feature fails is slowly and quietly: a little more off-topic traffic, a stale index, a vendor upgrade nobody re-tested. A weekly graded sample, a handful of golden signals from traces, and one or two targets with error budgets turn that into a graph someone owns. Pin model versions and treat every vendor deprecation notice as a scheduled regression test with a canary. And route every production failure back into the eval set: the loop from monitoring to evals is what makes the system get better instead of merely older.

For engineersShow the maths

error budget = 1 − SLO; burn rate = observed error rate ÷ (1 − SLO); time to exhaust = window ÷ burn rate

From the Google SRE workbook: burn rate 1 spends the budget exactly over the window; higher spends it sooner. The workbook's starting alert set for a 99.9% SLO pages at burn rate 14.4 over one hour (2% of budget) and 6 over six hours (5%), and opens a ticket at burn rate 1 over three days (10%).

Worked example: SLO 92% pass rate on a 30-day window: budget 8%. A week with 14% failures burns at 1.75, so the budget would be gone after about 17 days.

PSI = Σ over bins (a_i − e_i) × ln(a_i ÷ e_i) = KL(a ‖ e) + KL(e ‖ a)

a_i is the share of this week's scores in bin i, e_i the share at launch. It is the symmetric sum of the two Kullback-Leibler divergences, KL(p ‖ q) = Σ p_i ln(p_i ÷ q_i). Rules of thumb read below 0.10 as little shift and 0.10 to 0.25 as a significant shift requiring action, though those cut-offs were set without reference to error rates.

Worked example: Launch shares (0.05, 0.10, 0.20, 0.35, 0.30) against week 6 shares (0.12, 0.18, 0.25, 0.28, 0.17): PSI ≈ 0.21.

n ≈ z² × p × (1 − p) ÷ E², with z = 1.96 for a 95% interval

How many sampled conversations a weekly review needs to estimate a pass rate p to within E points. It follows from the standard error of a proportion; see the error-bars topic.

Worked example: p = 0.9, E = 0.03: 3.84 × 0.09 ÷ 0.0009 ≈ 384 cases a week. Accept ± 5 points and about 138 will do.

budget consumed by an alert window = burn rate × window ÷ period

The same relation, rearranged, says how much of the budget a burst of errors has already spent by the time an alert fires.

Worked example: Burn rate 36 over one hour of a 30-day period: 36 × 1 ÷ 720 = 5% of the budget, the workbook's own example.

where it stops working

A weekly sample measures the average; a failure that hits one customer segment can hide inside a healthy overall number, so slice by feature, language and channel. Labels arrive late or never for many requests, which is why sampled review and proxy signals stand in for true accuracy. Drift statistics such as PSI say a distribution moved, not whether the move matters, and their conventional thresholds were chosen by custom rather than by error rates. Monitoring also creates its own feedback loops: what you alert on shapes what gets fixed.

Key terms

The words you will hear

Drift

A change over time in the inputs a system sees or in how well it handles them, without any change to the code.
Where it came from

Primary sources

  1. 01Monitoring Distributed Systems (Site Reliability Engineering)Beyer et al. (eds.) · 2016
  2. 02Service Level Objectives (Site Reliability Engineering)Beyer et al. (eds.) · 2016
  3. 03Alerting on SLOs (The Site Reliability Workbook)Beyer et al. (eds.) · 2018
  4. 04Canarying Releases (The Site Reliability Workbook)Beyer et al. (eds.) · 2018
  5. 05Hidden Technical Debt in Machine Learning SystemsSculley et al. · 2015
  6. 06The ML Test Score: A Rubric for ML Production Readiness and Technical Debt ReductionBreck et al. · 2017
  7. 07Model deprecationsAnthropic · 2026
  8. 08DeprecationsOpenAI · 2026
  9. 09Statistical Properties of Population Stability IndexYurdakul · 2018
  10. 10On Information and SufficiencyKullback and Leibler · 1951

Want this working on your data?

We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.