7.3 — Debugging and Operational Resolution: Evidence Before Change

CCAR-P Domain 7 · Task 7.3

What You Need to Know

Read the Evidence First

When a production agent behaves unexpectedly, the first move is to read what happened: inputs, retrieved context, tool calls and results, outputs, versions in play.

Editing the agent’s instructions first is wrong for three stacked reasons: nothing yet implicates the instructions; the edit changes the system under investigation; the failing runs become unreproducible.

Containment ≠ Diagnosis

• Blast radius decides containment. Ongoing external effect (money moving, messages sent, records altered outside the team) → contain first, diagnose second.
• Internal and reversible degradation → the failing cases are the asset; a rollback spends them.
• Containment is not a diagnosis. A reverted system is quiet, not understood. Blanket containment also costs attribution.

Change History Narrows the Search

Most incidents are caused by a change. Correlate symptom onset against instruction versions, model versions, corpus refreshes, configuration changes, dependency/connector updates. When two changes land in the same window, test them one at a time (Domain 4’s one-variable rule).

The Agent Investigates; It Does Not Adjudicate

AI tooling shortens root-cause work (logs at volume, timeline reconstruction, correlation, reproduction, hypothesis generation). Its account of its own behaviour is a hypothesis — fluent, from the same context that produced the behaviour, with no privileged access to the cause. Confirm against the trace. One clean re-run does not clear a hypothesis under non-determinism.

Enablement Is Built Before the Incident

Traces that exist and are readable; pager-holder who can open them at 2am; rollback available to more than one person; runbook naming the first three moves; shared investigation tooling.

What the Incident Must Leave Behind

A fix at the failing layer, verified against the real case; a regression case; a runbook update; any access gap closed or recorded.

Exam Traps

• Rewriting the prompt/shared instructions as the first move
• Reverting several recent changes at once as a precaution, with no ongoing harm
• Accepting the agent’s account of its own behaviour as the root cause, or clearing a hypothesis on one clean re-run
• Closing the incident with a fix and nothing else, or planning an investigation that assumes access nobody has granted