Domain 4: Prompt Engineering & Structured Output (20%)

6 Task Statements / Lessons

4.1 System Prompts with Explicit Criteria

Write system prompts that contain measurable success criteria, role, constraints, and output format expectations.

4.2 Few-Shot Prompting

When and how to include examples; quality vs quantity of examples; placing examples in system vs user turns.

4.3 Structured Output with Tool Use

Prefer tool/function calling (or JSON schema mode) over free-text parsing for any data that must be machine-readable.

4.4 Validation, Retry, and Feedback Loops

Detect invalid output, feed the error back, and retry with corrective instructions.
Know when to escalate to a human.

4.5 Batch Processing Strategies

Handling many independent items efficiently (parallelism limits, cost control, failure isolation).

4.6 Multi-Instance and Multi-Pass Review

Using multiple Claude calls (or different models) for generation + critique + refinement pipelines.

---

Exam pattern: Questions often present a broken prompt and ask for the minimal change that fixes the failure mode.

CCAR-F Domain 4 · Task 4.1

What You Need to Know

Vague instructions ("be conservative", "only report high-confidence findings") give the model no actionable decision boundary. Explicit categorical criteria define precisely what to flag and what to skip.

Explicit Criteria vs Vague Instructions

Vague: "Review this code. Be conservative. Only report high-confidence findings."

Explicit: "Flag comments only when claimed behaviour contradicts actual code behaviour. Report bugs and security vulnerabilities. Skip minor style preferences and local patterns."

The False Positive Trust Problem

High false positive rates in one category destroy trust in all categories. Temporarily disable high-FP categories while you rework their prompts. System-wide trust comes ahead of category completeness.

Severity Calibration with Code Examples

Define severity with concrete code examples, not prose descriptions.

Key Concept: Explicit categorical criteria always outperform vague instructions and confidence-based filtering.

Why Confidence-Based Filtering Fails

LLM self-reported confidence is poorly calibrated. Use explicit criteria first; confidence routing second.

Exam Traps

• Choosing "be conservative" or "only report high-confidence findings" as valid improvements
• Assuming confidence thresholds fix false positive problems
• Keeping all categories active while iterating on high-FP ones

Practice Scenario

CI review pipeline has 40% FP rate on "documentation mismatch". Telling the model to report only high-confidence findings changed nothing. Budget allows no extra model call. Most effective fix?

Build Exercise: Build an Explicit Criteria Code Review Prompt (45 min)

CCAR-F Domain 4 · Task 4.2

What You Need to Know

Few-shot prompting puts a few worked examples (input + desired output + reasoning) in the prompt. It is the most effective technique for consistent, well-formatted output.

When to Deploy Few-Shot Examples

1. Detailed instructions alone produce inconsistent formatting
2. Model makes inconsistent judgement calls on ambiguous cases
3. Extraction returns empty/null fields for information the document contains

How to Construct Effective Examples

• Use 2 to 4 targeted examples
• Each example must show reasoning (why this action over alternatives)
• Cover the failing cases specifically

Key Concept: Few-shot examples are the most effective technique for consistency. Use 2–4 targeted examples with reasoning when instructions alone give inconsistent results, ambiguous judgements, or empty fields for data that exists.

Matching Technique to Problem

Problem	Correct Technique
Inconsistent output formatting	Few-shot examples
Malformed JSON	tool_use with JSON schemas
Fabricated values for missing fields	Optional/nullable schema fields
Wrong tool selection	Better tool descriptions first, then few-shot
Extraction sum ≠ total	Validation-retry loop

Exam Traps

• Choosing "add more detailed instructions" when formatting is inconsistent
• Adding many examples to cover every case
• Using confidence thresholds to fix inconsistent judgements

Practice Scenario

Extraction works on tables but returns empty fields for the same data in narrative paragraphs. Detailed instructions specify all fields. What should you try first?

Build Exercise: Build a Few-Shot Enhanced Extraction Prompt (45 min)

CCAR-F Domain 4 · Task 4.3

What You Need to Know

For guaranteed schema-compliant output: tool_use with JSON schemas (eliminates syntax errors) > prompt-based JSON (no structural guarantees).

tool_choice: Three Modes

Mode	Behaviour	Use when
"auto"	Model decides whether to call a tool	Conversational flexibility needed
"any"	Must call a tool; chooses which	Guaranteed structured output; unknown document type
{"type":"tool","name":"..."}	Must call one specific tool	Enforce a mandatory first step

What tool_use Does NOT Prevent

• Sum discrepancies
• Field placement errors
• Fabrication of values for required fields when source lacks the data

Semantic validation needs additional logic (Task 4.4).

Schema Design for Production

• Optional/nullable fields — primary defence against fabrication
• "unclear" enum value — for genuinely unclear sources
• "other" + detail string — extensible categorisation
• Format normalisation rules in the prompt alongside the schema

Key Concept: tool_use with a JSON schema stops syntax errors, not semantic ones. Optional/nullable fields stop fabrication. tool_choice "any" guarantees structured output for unknown document types.

Exam Traps

• Believing tool_use prevents all extraction errors
• Relying on tool_choice "auto" to guarantee a tool call
• Making all schema fields required to ensure completeness

Practice Scenario

Extraction uses tool_use with strict JSON schema; all fields required. Model invents dates/amounts when documents lack them. Instruction not to invent has not stopped it. Best fix?

Build Exercise: Build a Structured Extraction Tool with JSON Schema (45 min)

CCAR-F Domain 4 · Task 4.4

What You Need to Know

Schema syntax errors (malformed JSON, wrong types) are eliminated by tool_use + schemas. Semantic validation errors (sums don’t match, values in wrong fields) need external validation logic and retry loops.

Retry-with-Error-Feedback

Send three things back to the model:

1. The original document
2. The failed extraction
3. The specific validation error

Retry Effectiveness Boundary

Effective for: format mismatches, structural errors, misplaced values, mathematical errors (missed line items).
Not effective for: information genuinely absent from the source, data only in external documents, fields requiring unknown knowledge → human review or null.

Self-Correction Schema Design

• calculated_total vs stated_total + discrepancy flag
• conflict_detected booleans for contradictory source info
• detected_pattern fields for systematic prompt improvement

Pydantic as Validation Layer

Pydantic enforces structure and semantic rules; its ValidationError supplies the specific per-field messages the retry loop needs.

Key Concept: Retry-with-error-feedback sends original document + failed extraction + specific error. Retries fix format/structural errors but cannot create absent information.

Exam Traps

• Assuming retries always work
• Retrying without the specific validation error
• Retrying or picking one value when the source contradicts itself
• Treating Pydantic as redundant once tool_use enforces a schema

Practice Scenario

Document A: line items sum £450, stated total £500. Document B: department field missing from source. Only documents the model cannot fix should reach a person. Correct retry strategy?

Build Exercise: Build a Validation-Retry Loop for Document Extraction (60 min)

CCAR-F Domain 4 · Task 4.5

What You Need to Know

Message Batches API: 50% cost savings, up to 24-hour window, no latency SLA, no multi-turn client tool calling within a single batch request, custom_id for correlation.

The Matching Rule

Workflow type	API
Blocking (someone waiting)	Synchronous
Latency-tolerant (results consumed later)	Batch

Blocking examples: pre-merge CI checks, real-time code review.
Latency-tolerant: overnight debt reports, nightly test generation, batch document extraction.

Multi-Turn Tool Calling Limitation

Batch does not support multi-turn tool calling within a single request (client tools). Workflows needing mid-processing tool execution must use the synchronous API for that step.

SLA Calculation

Plan for the full 24-hour maximum. Work backwards from SLA (e.g. 30h SLA − 24h max = 6h buffer → submit every ~4h).

Batch Failure Handling

1. Identify failures by custom_id
2. Resubmit only failures with modifications (chunk, simplify prompt, add examples)
3. Refine prompts on a sample set before full batch submission

Key Concept: Batch only latency-tolerant work. Keep blocking work synchronous. Refine prompts on a sample set first.

Exam Traps

• Switching all workflows to batch for cost savings
• Assuming batch results arrive quickly
• Using batch for a workflow that needs a client tool mid-request

Practice Scenario

Manager proposes switching both blocking pre-merge review and overnight technical debt report to Batches for 50% savings. How evaluate?

Build Exercise: Design a Batch Processing Strategy (45 min)

CCAR-F Domain 4 · Task 4.6

What You Need to Know

The Self-Review Limitation

Same-session self-review retains the original reasoning chain and tends to confirm its own decisions. Use an independent instance (fresh session, no prior context) for review.

Multi-Pass Review Architecture

Large reviews suffer from attention dilution in a single pass (uneven depth, missed bugs, contradictory findings).

1. Pass 1 – Per-file local analysis: each file gets a focused independent review
2. Pass 2 – Cross-file integration: separate pass checks data flow, consistency, contradictions

Larger context windows do not fix attention quality.

Confidence-Based Routing

• High confidence → report directly
• Low confidence → human review
• Calibrate thresholds on labelled validation sets before automated routing

Key Concept: Independent instances for review. Split large reviews into per-file + cross-file passes. Calibrate confidence on labelled sets.

Production Architecture

1. Generation (instance A)
2. Per-file review (independent instances)
3. Integration review (separate instance)
4. Confidence routing
5. Calibration loop

Exam Traps

• Self-review in the same session
• Single pass for large multi-file reviews
• Switching to a larger context window to fix attention dilution
• Using uncalibrated confidence scores for automated routing

Practice Scenario

PR modifying 14 files receives inconsistent review (detailed on some, superficial on others, contradictory findings). Best restructuring?

Build Exercise: Build a Multi-Pass Code Review System (60 min)