Domain 4 – Evaluation, Testing & Optimisation (16%)

6 lessons, each ending with a practice scenario (Pro)

Log in to save your progress.

1. ✓4.1 — Defining Evaluation MetricsEvery metric has three parts: definition, measurement method, and target with baseline. Define metrics before the build.Scenario: Pro
2. ✓4.2 — Designing Evaluation Datasets and Test FrameworksNon-deterministic outputs → report SEM and CIs; use clustered SEs for related questions; resample and average; run power analysis before the eval. High-stakes/high-variance behaviours need more cases.Scenario: Pro
3. ✓4.3 — A/B Testing and Iterative ImprovementOnly an isolated change can be credited. Land the first, establish the new baseline, then test the second. “Since we’re touching it anyway” turns experiments into deployments.Scenario: Pro
4. ✓4.4 — Diagnosing Prompt Failure, Hallucination, and Model MismatchDiagnosis finds why a system fails before changing anything. Order: Trace and reproduce (pattern across cases); Isolate the stage (evidence first); Classify against signatures; Run the cheapest decisive test; Fix at the failing layer; Mint regression cases.Scenario: Pro
5. ✓4.5 — Optimising Token Usage, Latency, and Cost-Performance Trade-offsKey Concept: Optimisations move along the frontier — they do not repeal it. Name the spend and the saving, with eval evidence the spend is affordable.Scenario: Pro
6. ✓4.6 — Monitoring System Performance in ProductionLaunch-time evals prove readiness once; monitoring proves it stays true. Domain 3 built the plumbing; Domain 4 runs the quality practice on top.Scenario: Pro