3.3 — Accuracy–Latency Trade-offs

CCAR-P Domain 3 · Task 3.3

What You Need to Know

Every integration has a latency requirement (e.g. p95) and a quality requirement. The skill is holding both at once.

The Budget Is a Sum

Latency terms: TTFT + generation, retrieval/rerank, tool round-trips, agent-loop iterations. Serial chains multiply; only independent calls parallelise. Find the dominant term first (traces).

Accuracy Levers Pull the Other Way

More context, reranking, verification, reasoning, bigger model — each adds latency. Spend selectively: verify high-value items, fast-path the rest. Never delete a control tied to a named requirement.

Streaming and Caching

• Streaming improves perceived latency for interactive humans; total generation time unchanged. Irrelevant to machine consumers/parsers.
• Prefix caching is a double win (cost + TTFT) for stable prefixes.

Justification Names Both Numbers

“Meets the stated p95 while holding the measured mis-route rate.” One-sided justifications are distractors.

Exam Traps

• Dropping verification in a compliance/safety-critical workflow to hit latency
• Offering streaming as the latency fix for a machine consumer or batch pipeline
• Running expensive verification on every request when only a subset is high-stakes
• Parallelising a chain where step B consumes output of step A