Domain 5: Context Management & Reliability (15%)

6 Task Statements / Lessons

5.1 Context Window Management

Strategies when the window fills: summarisation, sliding window, subagent isolation, progressive disclosure, prompt caching of static content.

5.2 Escalation & Ambiguity Resolution

Detecting ambiguity, asking clarifying questions, or escalating to a human with the right context package.

5.3 Error Propagation in Multi-Agent Systems

How failures in one agent affect downstream agents; circuit breakers, partial-result handling, and recovery.

5.4 Codebase Exploration & Context Degradation

Techniques for exploring large codebases without drowning the context window; when context quality degrades and how to detect it.

5.5 Human Review & Confidence Calibration

When to surface confidence scores or force human review; calibrating the model’s expressed confidence.

5.6 Information Provenance & Multi-Source Synthesis

Tracking where information came from, reconciling conflicting sources, citing or attributing correctly.

---

Practical tip: Prompt caching of the system prompt + static knowledge is almost always the first optimisation for cost and latency.

CCAR-F Domain 5 · Task 5.1

What You Need to Know

The context window is everything Claude reads on one request. Managing it is choosing what goes in.

The Progressive Summarisation Trap

Progressive summarisation systematically destroys numerical values, dates, percentages and customer-stated expectations.

Fix: Persistent case facts block — extract transactional facts into a structured block included in every prompt, outside the summarised history. Never summarise this block.

The "Lost in the Middle" Effect

Models process beginning and end of long inputs more reliably. Put a summary of main findings at the start of aggregated input, with detailed results under explicit section headers.

Tool Result Trimming

Trim verbose tool outputs to only relevant fields before they enter conversation history (PostToolUse hook or tool implementation).

Full Conversation History

Claude API is stateless. Each request must include the complete conversation history. The case facts block resolves the tension with context limits.

Upstream Agent Optimisation

Upstream agents should return structured data (facts, citations, relevance scores) rather than verbose reasoning chains.

Prompt Caching

Mark a stable prefix with cache_control. Put constant content first (system, tools, reference docs); volatile content after. Static block belongs in the top-level system parameter.

Key Concept: The persistent case facts block is the single most important pattern: transactional facts in every prompt, never summarised.

Exam Traps

• Thinking progressive summarisation is safe for transactional data
• Assuming "pay attention to everything" fixes lost-in-the-middle
• Keeping full tool results "just in case"
• Selectively truncating conversation history

Practice Scenario

Support agent refers to "your recent refund request" instead of the specific $247.83 for order #8891 after summarisation. Most effective fix?

Build Exercise: Build a Persistent Case Facts Context Manager (45 min)

CCAR-F Domain 5 · Task 5.2

What You Need to Know

Three Valid Escalation Triggers

1. Customer explicitly requests a human — honour immediately; no exceptions
2. Policy exceptions or gaps (not violations) — human judges the exception
3. Inability to make meaningful progress — after a genuine attempt

Two Unreliable Triggers (Anti-patterns)

• Sentiment-based escalation — frustration ≠ case complexity
• Self-reported confidence scores — poorly calibrated

Frustrated Customers

• Straightforward issue + frustration → acknowledge + resolve (do not escalate)
• Explicit "I want a human" → escalate immediately
• Customer reiterates preference after offer to help → then escalate

Ambiguous Customer Matching

When a tool returns multiple matches: always ask for additional identifiers. Never pick by "most recent" or "most active" heuristic (privacy risk).

Explicit Escalation Criteria in System Prompts

Add explicit criteria + few-shot examples covering the three valid triggers and when to resolve autonomously. Prompt optimisation before architectural changes.

Key Concept: Valid triggers = explicit human request, policy gaps, inability to progress. Unreliable = sentiment and self-reported confidence.

Exam Traps

• Escalating on frustration/sentiment threshold
• Trusting self-reported confidence for escalation
• Attempting to resolve before honouring explicit human request
• Selecting from ambiguous matches heuristically

Practice Scenario

Agent achieves 55% first-contact resolution. Escalates straightforward cases, attempts complex policy exceptions. Rates itself highly on both. Most effective improvement?

Build Exercise: Build an Escalation Decision Engine (40 min)

CCAR-F Domain 5 · Task 5.3

What You Need to Know

Structured Error Context (4 elements)

1. Failure type (transient / validation / business / permission)
2. What was attempted (query, parameters, target)
3. Partial results gathered before failure
4. Potential alternative approaches

Two Anti-Patterns

1. Silent suppression — return empty results marked as success (worst; invisible gaps)
2. Workflow termination — kill entire pipeline on one failure (wastes completed work)

Middle ground: structured error propagation → coordinator decides recovery.

Access Failure vs Valid Empty Result

	Access failure	Valid empty result
Meaning	Tool never reached data	Query ran, found nothing
Retry?	Consider yes	No — this IS the answer

Coverage Annotations

Synthesis output should note which topics are well-supported and which have gaps due to unavailable sources.

Local Recovery

Subagents recover locally from transient failures before propagating. Only escalate persistent failures with structured context.

Key Concept: Structured error context enables intelligent coordinator recovery. Avoid silent suppression and workflow termination.

Exam Traps

• Catching timeout and returning empty results as success
• Terminating entire pipeline on one subagent timeout
• Returning generic "search unavailable" status
• Retrying a valid empty result

Practice Scenario

Web search subagent times out. How should failure information flow back to the coordinator to enable intelligent recovery?

Build Exercise: Build a Structured Error Propagation System (50 min)

CCAR-F Domain 5 · Task 5.4

What You Need to Know

Context degradation = model loses grip on earlier specific findings (refers to "typical patterns" instead of class names). It is an attention quality problem, not a token-limit problem. Larger context windows do not fix it.

Primary Mitigations

1. Scratchpad files — write key findings outside conversation context; refer back later
2. Subagent delegation — isolate verbose exploration in separate contexts; coordinator keeps only summaries
3. Summary injection between phases — inject Phase 1 summary into Phase 2 subagent prompts (prevents cold start)
4. /compact — use proactively during extended sessions to protect context quality
5. Structured state manifests — for crash recovery; load on resume and inject into agent prompts

Key Concept: Context degradation is lost grip on specific findings, not a token limit problem. Scratchpad files keep findings outside the context. Subagents isolate verbose exploration. Manifests prevent lost progress.

Exam Traps

• Increasing the context window to solve degradation
• Expecting next-phase subagents to inherit phase-one history
• Restarting a session without saving state
• Using /compact only when hitting limits

Practice Scenario

Agent exploring unfamiliar codebase starts referencing "typical repository patterns" instead of specific class names and dependency chains discovered earlier. Most effective mitigation?

Build Exercise: Build a Context-Resilient Codebase Explorer (60 min)

CCAR-F Domain 5 · Task 5.5

What You Need to Know

The Aggregate Metrics Trap

97% overall accuracy can hide catastrophic failure rates on specific document types or fields (e.g. handwritten receipts at 60%). Always validate by document type and field segment before automating.

Stratified Random Sampling

Sample high-confidence extractions as well as low-confidence ones. High-confidence items are automated; only stratified sampling detects novel error patterns on them.

Field-Level Confidence Calibration

Raw model confidence is uncalibrated. Calibrate against labelled validation sets (ground truth). Build calibration curves per field type per document type. Then set thresholds for automation vs human review.

Reviewer Capacity Prioritisation

Route highest-uncertainty items first. Do not spread capacity evenly. Dynamic priority queue ordered by uncertainty.

Validation Sequence Before Automation

1. Measure accuracy by document type and field segment
2. Calibrate confidence scores on labelled sets
3. Set calibrated thresholds
4. Implement stratified random sampling
5. Only then reduce human review on validated segments

Key Concept: 97% aggregate can hide 40% error rates on specific types. Validate by segment, calibrate against ground truth, sample high-confidence items, prioritise highest-uncertainty.

Exam Traps

• Using aggregate accuracy to justify automating all high-confidence extractions
• Only sampling low-confidence extractions
• Using raw confidence scores without calibration
• Spreading reviewer capacity evenly

Practice Scenario

System achieves 97% overall accuracy. Team proposes automating all extractions where confidence > 95%. Critical risk?

Build Exercise: Build a Confidence-Calibrated Review Router (50 min)

CCAR-F Domain 5 · Task 5.6

What You Need to Know

Structured Claim-Source Mappings

Every finding must include:

• Claim
• Source URL
• Document name
• Relevant excerpt
• Publication date

Why Attribution Dies in Synthesis

Synthesis compresses and paraphrases; source details drop out unless explicitly preserved. Downstream agents must maintain and merge claim-source mappings. Explicit instructions required.

Conflict Handling

When two credible sources disagree: annotate both values with full attribution and dates. Never arbitrarily select one (most recent, average, most authoritative).

Temporal Awareness

Different publication/data-collection dates often explain different numbers (trend, not contradiction). Require dates in all structured outputs.

Content-Appropriate Rendering

Content type	Format
Financial data	Tables
News / current events	Prose
Technical findings	Structured lists

Completing Analysis with Conflicts Intact

Analysis agent completes work with conflicts annotated; does not resolve them. Coordinator or consumer decides.

Key Concept: Map every claim to source URL, document name, excerpt and publication date; keep mappings through synthesis. Annotate conflicts with both values; never pick one arbitrarily. Match format to content.

Exam Traps

• Selecting the most recent source when sources conflict
• Assuming different numbers are contradictions (check dates)
• Allowing synthesis to paraphrase without preserving mappings
• Rendering all content in a uniform format

Practice Scenario

Two credible sources report 12% and 8% growth for the same sector (different data years). Synthesis currently selects the more recent value. Correct approach?

Build Exercise: Build a Provenance-Preserving Synthesis Pipeline (60 min)