4.1 — Defining Evaluation Metrics

CCAR-P Domain 4 · Task 4.1

What You Need to Know

Every metric has three parts: definition, measurement method, and target with baseline. Define metrics before the build.

Five Metric Families

Family	Professional distinction	Single-number mistake
Accuracy	Match metric to task shape (exact match, per-field P/R, rubric)	One aggregate “% correct” over heterogeneous tasks
Latency	Percentiles; distinguish TTFT vs total completion	Mean that conceals the tail
Cost	Per task at projected volume	Per-token figures with no volume
Safety	Refusal correctness in both directions	Counting refusals as if more were always safer
Security	Resistance measured under adversarial testing	Resistance claimed on paper

Success Metrics vs Guard Metrics

Success metric = what a change should improve. Guard metrics = what it must not damage (cost/task, tail latency, safety rates). A variant that lifts the headline while breaching a guard has failed.

Leading Signals, Lagging Proof, Proxy Trap

• Leading signals (token spend, refusal rate) → early warning
• Lagging proof (formal eval) → decisions
• Proxy trap: a proxy optimised hard enough stops tracking the outcome it proxied. Audit periodically.

Key Concept: Success criteria should be specific, measurable, achievable, and relevant. Most use cases need multidimensional evaluation.

Exam Traps

• Accepting mean latency as the latency metric/SLA target
• Deferring metric definition until after launch
• Scoring safety as harmful-output rate alone