4.5 — Optimising Token Usage, Latency, and Cost-Performance Trade-offs

CCAR-P Domain 4 · Task 4.5

What You Need to Know

Two Rules

1. Measure first — traces/usage name the dominant term (stage, token type, call pattern)
2. Re-gate with evals — change ships only if cheaper and still clears the quality bar

Token & Cost Levers

Lever	When
Prompt caching	Repeated stable prefixes
Context curation	Send fields, not files
Tight output contracts	Cap expensive output tokens
Effort / reasoning budgets	Tune within one model
Right-sizing per stage + cascade	Cheapest model that clears measured bar
Batch processing	No waiting user (“overnight”, “bulk”)

Latency Levers

Lever	What it attacks
Streaming	Perceived latency (humans watching)
Smallest model that clears bar	Real latency
Parallel independent tool calls	Serial waits
Caching	TTFT for long stable prefixes

Key Concept: Optimisations move along the frontier — they do not repeal it. Name the spend and the saving, with eval evidence the spend is affordable.

Exam Traps

• Offering batch for a latency-sensitive user-facing flow
• Switching to the smallest model without re-running evals
• Offering prompt caching for context-window overflow, wrong answers, or unique one-off documents