4.5 — Batch Processing Strategies

CCAR-F Domain 4 · Task 4.5

What You Need to Know

Message Batches API: 50% cost savings, up to 24-hour window, no latency SLA, no multi-turn client tool calling within a single batch request, custom_id for correlation.

The Matching Rule

Workflow type	API
Blocking (someone waiting)	Synchronous
Latency-tolerant (results consumed later)	Batch

Blocking examples: pre-merge CI checks, real-time code review.
Latency-tolerant: overnight debt reports, nightly test generation, batch document extraction.

Multi-Turn Tool Calling Limitation

Batch does not support multi-turn tool calling within a single request (client tools). Workflows needing mid-processing tool execution must use the synchronous API for that step.

SLA Calculation

Plan for the full 24-hour maximum. Work backwards from SLA (e.g. 30h SLA − 24h max = 6h buffer → submit every ~4h).

Batch Failure Handling

1. Identify failures by custom_id
2. Resubmit only failures with modifications (chunk, simplify prompt, add examples)
3. Refine prompts on a sample set before full batch submission

Key Concept: Batch only latency-tolerant work. Keep blocking work synchronous. Refine prompts on a sample set first.

Exam Traps

• Switching all workflows to batch for cost savings
• Assuming batch results arrive quickly
• Using batch for a workflow that needs a client tool mid-request