4.3 — A/B Testing and Iterative Improvement

CCAR-P Domain 4 · Task 4.3

What You Need to Know

The Gate and the Validation

• Offline evals gate — quick, repeatable; failing stops the candidate
• Online A/B validates — proves real-world impact on live traffic
• Passing offline is necessary but not sufficient for user-facing changes

One Variable at a Time

Only an isolated change can be credited. Land the first, establish the new baseline, then test the second. “Since we’re touching it anyway” turns experiments into deployments.

The Evidence Bar

• Report SEM and confidence intervals, not bare scores
• Paired-difference tests when variants answer the same questions
• Resample non-deterministic outputs and average per question
• Power analysis first so sample size can resolve the effect you care about

Staged Exposure for Risky Swaps

• Shadow — new variant handles real traffic; outputs compared offline; zero user risk
• Canary — small live slice first; guard metrics watched; rollback armed
• Versioned configs + clean rollback. After migration, re-baseline every downstream measurement.

The Improvement Loop

Measure → hypothesise → change one thing → eval offline → A/B online (success + guard metrics) → ship or revert → mint losing variant’s failures as regression cases.

Exam Traps

• Bundling a model migration with a prompt rewrite
• Treating a passing offline suite as licence to roll to everyone
• Declaring a winner from a handful of cases
• Offering a rollback plan as a substitute for staged exposure