Observability for LLM systems

Your APM doesn't know what a hallucination is. Your APM doesn't know what a citation is. Your APM doesn't know what an off-topic refusal is. To debug an LLM system at 3 a.m., you need observability your APM doesn't ship.

What to log, per request

1. The full prompt — system, user, retrieved context, tool definitions. With size budgets so it doesn't bloat.
2. The full completion — and any tool calls the completion implied.
3. The model and version. Don't trust the SDK's default; pin it explicitly.
4. Latency, broken down — by phase. End-to-end is meaningless without the breakdown.
5. Tokens in / tokens out / cost. Per request and aggregated.
6. Eval scores, where applicable — sampled production traffic scored by your judges.
7. Tenant, feature, environment. The tags that let you slice the dashboard.

Sampling

You will not log everything. At scale, full logging is prohibitive. Sample 1–5% of all traffic, 100% of errors, 100% of HITL escalations, 100% of any request flagged by your guardrails. The full prompt for the sampled subset; metadata-only for the rest.

Retention

Compliance regimes will tell you the floor. Cost will tell you the ceiling. Our default: 30 days of full prompts, 1 year of metadata, indefinite for the eval-scored subset. Document the policy so you can defend it in an audit.

Finding a regression at 3 a.m.

You get paged because the eval score on production traffic dropped. Walk down: which model? Which feature? Which tenant? Did the prompt change? Did the retrieval change? Did the model provider release a new version? The right observability lets you answer each question in under a minute. The wrong observability turns the page into an investigation that lasts the rest of the night.

What's specific to LLMs

• Trace tool calls as spans. Each tool is a child span of the agent run.
• Capture the system prompt diff across deploys. Most regressions are prompt regressions.
• Sample completions and re-score for hallucination, citation correctness, refusal rate.
• Alert on drift, not just on errors. A silently-getting-worse system never throws an exception.