RAG Systems Usually Fail at Retrieval, Not Generation

Retrieval-augmented generation is often presented as a language-model pattern: find some documents, put them in a prompt and ask for an answer. In production, the model is usually the most visible component and the least useful place to begin debugging. If the evidence does not enter the context window, no prompt can recover it. If contradictory or obsolete evidence does enter, a more capable model may merely produce a more persuasive mistake. The engineering problem is therefore an information-retrieval system with a generative interface. Its quality depends on source selection, parsing, segmentation, indexing, candidate retrieval, ranking and context assembly, all before the first output token is sampled.

Define retrieval success before tuning it

A useful evaluation starts with answerable questions and labelled evidence, not preferred wording for answers. For each question, identify the smallest source passages that justify a correct response and record acceptable alternatives where several documents say the same thing. Then measure whether those passages appear in the first k retrieved candidates. Recall@k answers the basic question: did the pipeline make the evidence available? Precision matters too, because a context packed with loosely related text increases cost and gives the model distracting claims. Track retrieval separately from end-to-end answer quality. Otherwise a correct answer from model memory can make broken retrieval look healthy, while cautious wording can make good retrieval look poor.

Ingestion errors become retrieval errors

The index can only represent what the parser preserved. PDF columns may be interleaved, tables flattened without headers, footnotes inserted mid-sentence and scanned pages silently omitted. Web pages contribute navigation, cookie notices and duplicated responsive content. Before experimenting with embeddings, inspect extracted text and retain provenance: source identifier, revision, page or section, access policy and extraction timestamp. Build ingestion checks for empty pages, implausible character counts, repeated boilerplate and encoding damage. Content should be re-indexable from an immutable or versioned source snapshot. Without that discipline, a parser upgrade cannot be compared cleanly and citations may point to a document that has changed since the answer was generated.

Chunking is a choice about answer boundaries

Fixed token windows are simple, but they cut through headings, list scopes and table rows. Very small chunks match narrow questions yet lose the qualifications around a statement. Large chunks preserve context but dilute embedding similarity and consume the model context budget. A practical strategy follows document structure first, splitting on semantic boundaries and only then applying a token ceiling. Keep headings with their children, repeat necessary table headers and attach metadata rather than injecting it awkwardly into prose. Overlap should restore continuity at genuine boundaries, not duplicate half the corpus. Evaluate chunk sizes against real query types: a policy lookup, a multi-row comparison and a procedural question need different evidence shapes.

• Store stable document and chunk identifiers so evaluation results survive re-indexing.
• Preserve hierarchy, including section titles and table headers, as ranking features.
• Record effective dates and versions so obsolete passages can be filtered deliberately.
• Test extracted chunks directly; a visually correct source file proves nothing about parsing.
• Treat access-control attributes as mandatory filters, never as instructions to the model.

Embeddings are one retrieval signal

Dense vectors are strong at semantic similarity but weak at some exact distinctions. Product codes, legal clause numbers, error identifiers and personal names may be better served by lexical search. Hybrid retrieval combines dense and term-based candidates, often followed by a reranker that scores query-passage pairs more precisely. Candidate generation should favour recall; reranking can then improve ordering. Filters must be applied with care. Filtering after approximate vector search may leave too few authorised results, while pre-filter support varies by database and index. Measure latency and recall under realistic tenant and date filters rather than against an unfiltered demonstration corpus. Changing the embedding model also requires a controlled re-index and a comparison on the same labelled queries.

Context assembly is not concatenation

The highest-ranked chunks are not automatically the best prompt. Near-duplicates waste space, adjacent chunks may need merging, and a single source can crowd out independent evidence. Context assembly should deduplicate, enforce token budgets and preserve citations through every transformation. It can diversify by source or section and include neighbouring text only when that improves labelled evidence coverage. Ordering has effects too: models do not attend uniformly across long contexts, and critical evidence buried among repetitive passages may be missed. Ask the model to distinguish supported facts from inference and to decline when evidence is insufficient, but do not mistake that instruction for a retrieval control. The application should expose which passages were selected so failures remain inspectable.

Use failure buckets, not one aggregate score

An average hides the work. Classify failures as missing source, extraction damage, bad boundary, vocabulary mismatch, filter error, ranking error, context truncation, unsupported synthesis or citation mismatch. Slice results by document type, query intent, tenant, language and content age. A regression suite should include simple lookups as well as adversarial cases: similarly named products, superseded policies and questions whose correct response is that the corpus does not say. Log query rewrites, candidate scores, filters, chosen chunks, prompt version and model version with appropriate redaction. This gives engineers a replayable trace and gives reviewers a way to decide whether an apparent model problem began much earlier.

Freshness, permissions and citations are retrieval concerns

Production corpora are not static collections. A policy can be replaced, a customer can lose access and a document can be removed while its old vector remains searchable. Indexing therefore needs a lifecycle. Use source events or scheduled comparison to create, update and delete chunks, and monitor lag between the authoritative source and the searchable index. Version documents so retrieval can prefer the current revision while preserving history where policy requires it. Deletes must propagate to vector, lexical and cache layers. Treat authorisation as part of the query plan: attach access attributes during ingestion and enforce them before content enters candidates or prompts. Post-generation redaction is too late because the model has already received the text. Citations need the same discipline. Keep offsets or stable anchors through parsing, chunk merging and reranking, then link the displayed citation to the exact revision used. Verify that the cited passage entails the nearby claim rather than merely sharing its topic. If a source has moved, the application can resolve its stable identifier to the permitted current location without changing the historical trace. Test permission changes, document replacement and cache invalidation as explicit scenarios. These operational details are part of retrieval quality because the right passage delivered to the wrong user, or an obsolete passage presented as current, is still a failed retrieval.

A practical order of work

Begin with a small, representative labelled set and a transparent lexical baseline. Validate parsing and provenance, establish recall@k, then compare chunking and hybrid candidate strategies. Add reranking only after candidate recall is adequate, and tune context assembly against both answer support and cost. Finally run end-to-end evaluation with human review for claims, citations and abstentions. This order prevents an expensive model from masking weak foundations. A good RAG system is not one that always answers. It is one that retrieves the right evidence, shows where it came from and recognises when the available evidence is not enough.