4.4 — Validation, Retry, and Feedback Loops

CCAR-F Domain 4 · Task 4.4

What You Need to Know

Schema syntax errors (malformed JSON, wrong types) are eliminated by tool_use + schemas. Semantic validation errors (sums don’t match, values in wrong fields) need external validation logic and retry loops.

Retry-with-Error-Feedback

Send three things back to the model:

1. The original document
2. The failed extraction
3. The specific validation error

Retry Effectiveness Boundary

Effective for: format mismatches, structural errors, misplaced values, mathematical errors (missed line items).
Not effective for: information genuinely absent from the source, data only in external documents, fields requiring unknown knowledge → human review or null.

Self-Correction Schema Design

• calculated_total vs stated_total + discrepancy flag
• conflict_detected booleans for contradictory source info
• detected_pattern fields for systematic prompt improvement

Pydantic as Validation Layer

Pydantic enforces structure and semantic rules; its ValidationError supplies the specific per-field messages the retry loop needs.

Key Concept: Retry-with-error-feedback sends original document + failed extraction + specific error. Retries fix format/structural errors but cannot create absent information.

Exam Traps

• Assuming retries always work
• Retrying without the specific validation error
• Retrying or picking one value when the source contradicts itself
• Treating Pydantic as redundant once tool_use enforces a schema