Where does a RAG answer go wrong? Debugging in four stages
A customer asks a document assistant: “How long can I return the item?” The assistant retrieves the right policy — “Returns are accepted within 14 days of purchase” — but answers “within 30 days [1]”. The answer has a source and looks credible. A RAG pipeline can be examined in four stages, and the error sits in one of them.
1. Chunking: split without losing the source
Documents are split into chunks before they are searched. Every chunk must keep which document it came from, otherwise the answer cannot be backed up. A chunk must not mix two documents, and no words may disappear in the split.
Size is a trade-off: chunks that are too small lack context, chunks that are too large dilute what is relevant and take up the model's context. If you change the size, measure the effect instead of guessing.
2. Retrieval: find the right evidence — and measure it
Search ranks chunks by how well they match the question, lexically (BM25, keywords) or semantically (embeddings). Common question words such as “how” and “when” should not produce hits.
Measure search on its own, separate from the answer. Recall@k is the share of questions where the right source is among the first k hits. If recall is low it does not matter how good the model is — it never sees the right text. In the example above the right source was retrieved, so search is not the problem.
3. Grounding: the answer must be supported by the source
This is where the example fails. The source says 14 days, the answer says 30. The model mixed in something it “knows” from training instead of sticking to the evidence. The citation makes it worse, not better: the reader believes the claim has been checked.
A simple, checkable requirement is that every quote must appear verbatim in the cited source. That can be checked automatically. Whether the claim is otherwise supported by the quote must be reviewed — by a person or a calibrated judge.
4. Abstain when evidence is missing
If the customer asks something the documents do not cover, the assistant should say so, not guess. A grounded abstention — “I find no support in the material” — is a correct answer. Test it explicitly with questions that have no answer in the documents.
- Does every chunk keep its source?
- Is recall@k measured on questions that were not used to build the search?
- Does every quote appear verbatim in the source?
- Does the system abstain when relevant evidence is missing?