AI features break the assumptions behind traditional observability: outputs are non-deterministic, requests cost real money per call, and failures often look like unhelpful text instead of exceptions. In 2026, teams instrumenting LLM workloads converged on a third pillar alongside traces and logs — token economics as a first-class signal.
Token spend is a first-class metric
AI features have a variable marginal cost that traditional metrics ignore. Track tokens per request, cost per user, and cache hit rate alongside latency, because a prompt change can double your bill without any visible performance regression.
Trace the whole chain
A single AI response may involve retrieval, multiple model calls, and tool execution. Spanning each step shows where time and tokens actually go, and it is the only way to diagnose a slow or expensive answer to its root cause.
Watch quality signals over time
User corrections, regeneration rates, and thumbs-down feedback are quality metrics that degrade gradually. Trending them catches model drift and prompt regressions that unit-style evals run only at release time will miss.
Tracing through model calls
A useful trace for an AI feature spans the whole chain: retrieval results, prompt template version, model and parameters, tool calls, and the final output — all under one trace ID so a bad answer can be replayed to its inputs. Tool calls and retries are where latency hides, and spans reveal whether slowness is the model or the eight sequential lookups around it. Sampling should be biased toward requests users flagged or retried.
Tokens, cost and quality on one chart
Per-request token spend belongs next to latency: an unplanned prompt change can double cost without breaking a single test. Track input versus output tokens by feature, cache hit rates, and cost per successful outcome rather than per call — the last metric is what finance actually understands. Pair it with quality signals (user retries, escalations) so cost optimization never quietly trades away answer quality.
Key takeaways
- Trace the full chain: retrieval, prompt version, model, tools, output.
- Chart cost per successful outcome, not per API call.
- Bias sampling toward flagged and retried requests.
Final thoughts
AI workload observability is traditional distributed tracing plus an economic dimension, and teams that instrument both discover that the same dashboards catch latency regressions, quality slips and cost surprises in one pass.