4.2 — Designing Evaluation Datasets and Test Frameworks

CCAR-P Domain 4 · Task 4.2

What You Need to Know

Four Case Populations (risk coverage)

1. Representative — mirror production distribution (the bulk)
2. Edge — boundaries: irrelevant data, overly long inputs, poor input, ambiguous cases
3. Adversarial — injection attempts, leading questions, malformed input
4. Regression — minted from production failures so fixed bugs do not return

Golden-Set Discipline

1. Versioned — scores comparable over time
2. Held out — never appears in the prompt as few-shot (tests capability, not recall)
3. Refreshed — production traffic drifts; refresh cadence is a design decision

Sizing

Non-deterministic outputs → report SEM and CIs; use clustered SEs for related questions; resample and average; run power analysis before the eval. High-stakes/high-variance behaviours need more cases.

Mixed Methodologies (Pairing Rule)

Method	Use for
Code-graded (exact match, schema, regex)	Everything it can grade (cheap, objective)
LLM-as-judge	Open-ended quality (tone, faithfulness); use different model; explicit rubric; versioned
Human review	Calibrate the judge, audit samples, adjudicate high-stakes/ambiguous cases

Uncalibrated judge is not evidence. Validate against human labels before trusting scores.

Exam Traps

• Trusting an LLM judge nobody has checked against human labels
• Choosing full human grading as the rigorous option
• Pasting eval cases into the prompt as few-shot, then celebrating the pass rate
• Adding more happy-path cases to fix recurring production incidents