Domain 4 – Evaluation, Testing & Optimisation (16%)

• Define multi-dimensional metrics (accuracy, latency, cost, safety, security)
• Build mixed-methodology evaluation datasets and test frameworks
• A/B testing and iterative improvement loops
• Diagnose prompt failure, hallucinations, model mismatch
• Token / latency / cost optimisation techniques
• Logging and observability tooling

CCAR-P Domain 4 · Task 4.1

What You Need to Know

Every metric has three parts: definition, measurement method, and target with baseline. Define metrics before the build.

Five Metric Families

Family	Professional distinction	Single-number mistake
Accuracy	Match metric to task shape (exact match, per-field P/R, rubric)	One aggregate “% correct” over heterogeneous tasks
Latency	Percentiles; distinguish TTFT vs total completion	Mean that conceals the tail
Cost	Per task at projected volume	Per-token figures with no volume
Safety	Refusal correctness in both directions	Counting refusals as if more were always safer
Security	Resistance measured under adversarial testing	Resistance claimed on paper

Success Metrics vs Guard Metrics

Success metric = what a change should improve. Guard metrics = what it must not damage (cost/task, tail latency, safety rates). A variant that lifts the headline while breaching a guard has failed.

Leading Signals, Lagging Proof, Proxy Trap

• Leading signals (token spend, refusal rate) → early warning
• Lagging proof (formal eval) → decisions
• Proxy trap: a proxy optimised hard enough stops tracking the outcome it proxied. Audit periodically.

Key Concept: Success criteria should be specific, measurable, achievable, and relevant. Most use cases need multidimensional evaluation.

Exam Traps

• Accepting mean latency as the latency metric/SLA target
• Deferring metric definition until after launch
• Scoring safety as harmful-output rate alone

Practice Scenario

Contract-extraction service proposes one metric: overall accuracy ≥ 90% on spot-checked requests. Sponsor wants targets fixed before launch, re-measurable the same way after every release, and showing where the service fails. What should the architect do?

Build Exercise: Write the Metric Sheet for One Claude System (30 min)

CCAR-P Domain 4 · Task 4.2

What You Need to Know

Four Case Populations (risk coverage)

1. Representative — mirror production distribution (the bulk)
2. Edge — boundaries: irrelevant data, overly long inputs, poor input, ambiguous cases
3. Adversarial — injection attempts, leading questions, malformed input
4. Regression — minted from production failures so fixed bugs do not return

Golden-Set Discipline

1. Versioned — scores comparable over time
2. Held out — never appears in the prompt as few-shot (tests capability, not recall)
3. Refreshed — production traffic drifts; refresh cadence is a design decision

Sizing

Non-deterministic outputs → report SEM and CIs; use clustered SEs for related questions; resample and average; run power analysis before the eval. High-stakes/high-variance behaviours need more cases.

Mixed Methodologies (Pairing Rule)

Method	Use for
Code-graded (exact match, schema, regex)	Everything it can grade (cheap, objective)
LLM-as-judge	Open-ended quality (tone, faithfulness); use different model; explicit rubric; versioned
Human review	Calibrate the judge, audit samples, adjudicate high-stakes/ambiguous cases

Uncalibrated judge is not evidence. Validate against human labels before trusting scores.

Exam Traps

• Trusting an LLM judge nobody has checked against human labels
• Choosing full human grading as the rigorous option
• Pasting eval cases into the prompt as few-shot, then celebrating the pass rate
• Adding more happy-path cases to fix recurring production incidents

Practice Scenario

Team evaluates customer-email assistant with 40 hand-written happy-path cases graded by an uncalibrated LLM judge. Production incidents keep recurring. What should change?

Build Exercise: Design the Dataset Composition for One System (45 min)

CCAR-P Domain 4 · Task 4.3

What You Need to Know

The Gate and the Validation

• Offline evals gate — quick, repeatable; failing stops the candidate
• Online A/B validates — proves real-world impact on live traffic
• Passing offline is necessary but not sufficient for user-facing changes

One Variable at a Time

Only an isolated change can be credited. Land the first, establish the new baseline, then test the second. “Since we’re touching it anyway” turns experiments into deployments.

The Evidence Bar

• Report SEM and confidence intervals, not bare scores
• Paired-difference tests when variants answer the same questions
• Resample non-deterministic outputs and average per question
• Power analysis first so sample size can resolve the effect you care about

Staged Exposure for Risky Swaps

• Shadow — new variant handles real traffic; outputs compared offline; zero user risk
• Canary — small live slice first; guard metrics watched; rollback armed
• Versioned configs + clean rollback. After migration, re-baseline every downstream measurement.

The Improvement Loop

Measure → hypothesise → change one thing → eval offline → A/B online (success + guard metrics) → ship or revert → mint losing variant’s failures as regression cases.

Exam Traps

• Bundling a model migration with a prompt rewrite
• Treating a passing offline suite as licence to roll to everyone
• Declaring a winner from a handful of cases
• Offering a rollback plan as a substitute for staged exposure

Practice Scenario

Team wants to migrate support agent to a newer model and simultaneously rewrite the system prompt. Must attribute any behaviour shift; no change may reach every user untested. Right approach?

Build Exercise: Write the Experiment Plan for Your Next Change (30 min)

CCAR-P Domain 4 · Task 4.4

What You Need to Know

Diagnosis finds why a system fails before changing anything. Order:

1. Trace and reproduce (pattern across cases)
2. Isolate the stage (evidence first)
3. Classify against signatures
4. Run the cheapest decisive test
5. Fix at the failing layer
6. Mint regression cases

Failure Classes & Signatures

Class	Signature	Decisive test	Failing layer
Prompt failure	Format drift, rules ignored, varies with phrasing	Clarified structured prompt, all else constant	Prompt
Hallucination (grounding)	Wrongness correlates with corpus refresh/re-import	Inspect retrieved passages in the trace	Retrieval/indexing
Hallucination (generation)	Right passages retrieved; answer contradicts/exceeds them	Grounding discipline; measure faithfulness	Prompt-level grounding
Model mismatch	Uniform mediocrity on hard cases under every prompt variant; context verified	Run failing set on stronger tier	Model tier

Hallucination Fixes

• Grounding: re-index, re-embed, repair retrieval (no instruction fixes bad context)
• Generation: allow “I don’t know”; extract quotes first; require citations; restrict to provided text; CoT / best-of-N

Model Mismatch Runs Both Ways

Upgrade only after stronger-tier test confirms capability gap. Premium tier far above a simple task is the same diagnosis, opposite prescription (downsize).

Exam Traps

• Upgrading the model tier as the first response to wrong answers
• Rewriting the prompt to suppress hallucinations that began after a corpus refresh
• Reaching for temperature zero to stop incorrect answers
• Calling every incorrect output a hallucination and treating it as one problem

Practice Scenario

Policy assistant invents clause numbers two days after policy corpus was re-imported. Traces show retrieved passages no longer contain the cited sections; latency, model, prompt unchanged. What should the team do?

Build Exercise: Write the Diagnosis Runbook for One System (45 min)

CCAR-P Domain 4 · Task 4.5

What You Need to Know

Two Rules

1. Measure first — traces/usage name the dominant term (stage, token type, call pattern)
2. Re-gate with evals — change ships only if cheaper and still clears the quality bar

Token & Cost Levers

Lever	When
Prompt caching	Repeated stable prefixes
Context curation	Send fields, not files
Tight output contracts	Cap expensive output tokens
Effort / reasoning budgets	Tune within one model
Right-sizing per stage + cascade	Cheapest model that clears measured bar
Batch processing	No waiting user (“overnight”, “bulk”)

Latency Levers

Lever	What it attacks
Streaming	Perceived latency (humans watching)
Smallest model that clears bar	Real latency
Parallel independent tool calls	Serial waits
Caching	TTFT for long stable prefixes

Key Concept: Optimisations move along the frontier — they do not repeal it. Name the spend and the saving, with eval evidence the spend is affordable.

Exam Traps

• Offering batch for a latency-sensitive user-facing flow
• Switching to the smallest model without re-running evals
• Offering prompt caching for context-window overflow, wrong answers, or unique one-off documents

Practice Scenario

Nightly compliance sweep summarises 30k different documents on a premium model with per-request calls. Finance wants cost cut by changing the pipeline. Mid-tier matches summary quality on a representative sample; no model may replace premium unless tests show it meets the bar. What should the architect do?

Build Exercise: Run a Measured Optimisation Pass on One System (45 min)

CCAR-P Domain 4 · Task 4.6

What You Need to Know

Launch-time evals prove readiness once; monitoring proves it stays true. Domain 3 built the plumbing; Domain 4 runs the quality practice on top.

Continuous Evaluation

Formal evals are lagging proof at release boundaries. Continuous evaluation scores sampled production traffic automatically (programmatic checks + calibrated judge). Quality becomes a trend line against the launch baseline.

Drift & Leading Indicators

Indicator	Implicates
Token spend per request	Inputs grew / prompt edit
Refusal rate	Traffic shifted / policy interaction
Tool-failure rate	Upstream API changed
Input-length distribution	New client integrated
Retrieval-quality signals	Corpus refreshed

Waiting for user complaints is the most lagging (and expensive) signal.

Alerts, Baselines, Incident Loop

• Alert on guard metrics at tail percentiles, thresholds from measured baselines
• Protect signal: deliberately choose what not to alert on
• Incident loop: Alert → Traces → Diagnose (4.4 order) → Fix at failing layer → Mint regression cases → Re-baseline if floor legitimately moved

Key Concept: Uptime does not measure wrongness. A system can be fully up, fast on average, and confidently incorrect.

Exam Traps

• Presenting green uptime and mean-latency dashboards as evidence the system is healthy
• Skipping continuous evaluation because the team evals before every release
• Closing incidents without minting regression cases from the failures

Practice Scenario

Three months after launch, procurement assistant’s satisfaction scores sag. Release evals all passed; uptime and mean-latency dashboards green. Team must learn what has changed before altering the system, and catch further decline before users report it. What should the team do?

Build Exercise: Write the Monitoring Page for One System (45 min)