4.4 — Diagnosing Prompt Failure, Hallucination, and Model Mismatch

CCAR-P Domain 4 · Task 4.4

What You Need to Know

Diagnosis finds why a system fails before changing anything. Order:

1. Trace and reproduce (pattern across cases)
2. Isolate the stage (evidence first)
3. Classify against signatures
4. Run the cheapest decisive test
5. Fix at the failing layer
6. Mint regression cases

Failure Classes & Signatures

Class	Signature	Decisive test	Failing layer
Prompt failure	Format drift, rules ignored, varies with phrasing	Clarified structured prompt, all else constant	Prompt
Hallucination (grounding)	Wrongness correlates with corpus refresh/re-import	Inspect retrieved passages in the trace	Retrieval/indexing
Hallucination (generation)	Right passages retrieved; answer contradicts/exceeds them	Grounding discipline; measure faithfulness	Prompt-level grounding
Model mismatch	Uniform mediocrity on hard cases under every prompt variant; context verified	Run failing set on stronger tier	Model tier

Hallucination Fixes

• Grounding: re-index, re-embed, repair retrieval (no instruction fixes bad context)
• Generation: allow “I don’t know”; extract quotes first; require citations; restrict to provided text; CoT / best-of-N

Model Mismatch Runs Both Ways

Upgrade only after stronger-tier test confirms capability gap. Premium tier far above a simple task is the same diagnosis, opposite prescription (downsize).

Exam Traps

• Upgrading the model tier as the first response to wrong answers
• Rewriting the prompt to suppress hallucinations that began after a corpus refresh
• Reaching for temperature zero to stop incorrect answers
• Calling every incorrect output a hallucination and treating it as one problem