LLM observability
What one request looks like as a trace, and why a pile of logs cannot tell you what a trace can.
Read first:The agent loopRetrieval-augmented generation (RAG)
Step 1 of 7· Illustrative request, timings, tokens and prices; span names and attribute keys follow the OpenTelemetry GenAI conventions
When someone asks your AI assistant a question, the answer is built in stages: the app assembles a prompt, searches your documents, calls the model, perhaps calls a tool such as an order lookup, then calls the model again and streams the reply. Observability means recording each stage as a span, with its timing, tokens, cost and whether it succeeded, all linked into one trace for that request. With traces you can see which stage made an answer slow or wrong, replay a bad request with exactly the inputs it had, and add up what each feature costs. Plain logs record events, but not how they connect.
Why it matters for your product
A request to an LLM app passes through retrieval, one or more model calls and tools, and any of them can be the reason an answer was slow, wrong or expensive. Traces turn 'the assistant is bad today' into 'the retrieval span returned nothing after the index rebuild'. They also make evals and monitoring possible: sampled traces become review cases, production failures become golden-set entries, and token counts become a cost line per feature. Instrument with the OpenTelemetry conventions from day one, capture prompts and outputs where privacy rules allow, and sample deliberately rather than by accident.
For engineersShow the maths
cost(request) = Σ over model spans ( tokens_in × price_in + tokens_out × price_out )
Every model call in the trace reports its input and output tokens; multiply by the per-token prices for that model and sum across the calls. Embedding calls add their own, usually tiny, line.
Worked example: Two calls of 1,840 + 62 and 2,010 + 118 tokens at an illustrative $3 per million input and $15 per million output tokens: $0.0065 + $0.0078 = $0.0143 per request.
cost per feature = Σ cost(request) over traces tagged with that feature; cost per good answer = cost per feature ÷ answers rated good
Tag each trace with the feature it served and the sum is a cost line per feature. Dividing by the number of answers a sampled review rated good gives the number that matters for a product decision.
Worked example: 12,000 order-support requests a day at $0.0143 is about $172 a day; if 90% of sampled answers pass, that is about $0.016 per good answer.
trace duration = longest path of sequential spans, not the sum of all spans
Spans that run in parallel overlap, so the request takes as long as its critical path. Shortening a span off that path changes nothing the user can feel.
Worked example: Retrieval (65 ms) and a customer-profile lookup (120 ms) run side by side: the stage takes 120 ms, not 185 ms.
traces kept ≈ requests × rate (head); = every trace matching the keep rules + rate × the rest (tail)
OpenTelemetry notes that for high-volume systems a rate of 1% or lower is often enough to represent the rest, and that head sampling alone cannot guarantee every trace with an error is kept.
Worked example: 2 million requests a day at 1% is 20,000 traces. With tail sampling and a rule to keep every error, all 6,000 failed requests are kept plus 1% of the rest.
A trace shows what happened, not whether it was good: a request can finish with every span marked OK and still give a wrong answer, which is why evals and sampled review sit beside tracing. Capturing prompts and outputs in spans records personal data, so retention, access and redaction need the same care as any customer record. The GenAI conventions are still in Development status, so attribute names can change between releases, and sampling by design throws information away.
The words you will hear
Trace
Primary sources
- 01Semantic conventions for GenAI spansOpenTelemetry · 2026
- 02Semantic conventions for GenAI metricsOpenTelemetry · 2026
- 03Semantic conventions for generative AI systemsOpenTelemetry · 2026
- 04Observability primerOpenTelemetry · 2026
- 05SamplingOpenTelemetry · 2026
- 06Monitoring Distributed Systems (Site Reliability Engineering)Beyer et al. (eds.) · 2016
- 07The ML Test Score: A Rubric for ML Production Readiness and Technical Debt ReductionBreck et al. · 2017
Connected ideas
Observability
Evals
How to know a change made your AI system better, before your users find out it did not.
ExploreObservability
Monitoring and drift
After launch the data, the documents and the model itself keep changing. Monitoring is how you notice before your customers do.
ExploreGenerating text
Prefill and decode
Why the first word of an answer takes a moment and the rest stream out at a steady pace.
ExploreGenerating text
KV cache
Why a model keeps the keys and values of earlier tokens instead of recomputing them, and what that memory costs.
ExploreWant this working on your data?
We design and build the systems these ideas power: retrieval, agents, voice and the models behind them. Start with a free discovery call.