3.5 — RAG Pipeline Design: Chunking and Indexing

CCAR-P Domain 3 · Task 3.5

What You Need to Know

RAG stages: ingest → chunk → embed → index → retrieve → (rerank) → generate. Generation quality is capped by retrieval quality.

Chunking Is a Trade-off

Too small → strips needed context. Too large → dilutes relevance. Structure-aware chunking (sections, headings, tables) with modest overlap outperforms fixed-size splitting.

Contextual Retrieval

Prepend a brief, chunk-specific explanation situating each chunk in its document before embedding. Restores context lost at the chunk boundary. Measurably cuts retrieval failures.

Index for How It Will Be Asked

• Exact strings / identifiers → BM25 / keyword
• Conceptual queries → embedding search
• Identifier-heavy corpora → hybrid (embeddings + keyword)
• Metadata filters narrow the candidate set before similarity ranks

Freshness Is Operational

• Refresh triggers re-embed + re-index atomically
• Pin the embedding-model version everywhere
• Monitor retrieval quality (recall on labelled set)
• Evaluate retrieval separately from generation

Refresh signature: quality breaks immediately after a content refresh, model and latency unchanged → investigate retrieval/indexing first.

When RAG Is Not the Answer

A corpus small and stable enough for the cached prompt is faster and cheaper served whole.

Exam Traps

• Model upgrade when answers went wrong right after a document refresh
• Retrieving more chunks to fix a stale/broken index
• Tuning chunk size without measuring where retrieval fails
• Evaluating the pipeline only end-to-end