3.4 — Observability and Monitoring at Scale

CCAR-P Domain 3 · Task 3.4

What You Need to Know

LLM systems break anecdote debugging: non-deterministic, so a single re-run proves nothing. A green uptime dashboard says nothing about answer quality.

The Trace Is the Unit

Capture per request: prompt version, model id, retrieved context, tool calls + results, tokens, cost, latency, response. Correlation ids stitch multi-stage pipelines into one record.

Two Tiers at Scale

1. Cheap aggregates on every request — latency percentiles, token spend, error/refusal/tool-failure rates
2. Expensive full payloads on a sample + every detected failure — sampling rates, retention, redaction are architecture decisions

Quality Is Not an Infrastructure Metric

Uptime/error/latency can all be green while the system produces rubbish. Quality needs its own signal: automated evaluation on sampled production traffic, scored against a calibrated rubric/judge.

Drift Shows in Cheap Signals

Token spend per request, refusal rates, tool-failure rates, input-length drift. Alert on tail percentiles, not averages.

Exam Traps

• Adding more uptime/latency alerting when support reports wrong answers
• Debugging by re-running the input a few times
• Persisting full prompts/outputs for every request indefinitely
• Tracing single calls without correlation ids in a multi-stage pipeline