4.6 — Monitoring System Performance in Production

CCAR-P Domain 4 · Task 4.6

What You Need to Know

Launch-time evals prove readiness once; monitoring proves it stays true. Domain 3 built the plumbing; Domain 4 runs the quality practice on top.

Continuous Evaluation

Formal evals are lagging proof at release boundaries. Continuous evaluation scores sampled production traffic automatically (programmatic checks + calibrated judge). Quality becomes a trend line against the launch baseline.

Drift & Leading Indicators

Indicator	Implicates
Token spend per request	Inputs grew / prompt edit
Refusal rate	Traffic shifted / policy interaction
Tool-failure rate	Upstream API changed
Input-length distribution	New client integrated
Retrieval-quality signals	Corpus refreshed

Waiting for user complaints is the most lagging (and expensive) signal.

Alerts, Baselines, Incident Loop

• Alert on guard metrics at tail percentiles, thresholds from measured baselines
• Protect signal: deliberately choose what not to alert on
• Incident loop: Alert → Traces → Diagnose (4.4 order) → Fix at failing layer → Mint regression cases → Re-baseline if floor legitimately moved

Key Concept: Uptime does not measure wrongness. A system can be fully up, fast on average, and confidently incorrect.

Exam Traps

• Presenting green uptime and mean-latency dashboards as evidence the system is healthy
• Skipping continuous evaluation because the team evals before every release
• Closing incidents without minting regression cases from the failures