Common RAG Failure Modes

RAG failures rarely look like errors#

A RAG system's most common failures don't raise exceptions or show up in an error log — they show up as a plausible-looking but subtly wrong answer, which is exactly what makes them dangerous in production. This page catalogs the specific, well-documented failure patterns worth actively testing for, rather than discovering them from a user complaint.

Lost-in-the-middle#

Research on long-context LLMs has repeatedly found that models are noticeably better at using information placed near the beginning or end of their context window than information buried in the middle — a pattern commonly called "lost in the middle." For a RAG system retrieving, say, 10 chunks and stuffing all of them into the prompt, the one chunk that actually contains the answer can end up in the middle of that block and get under-weighted relative to chunks at the edges, even though it's the most relevant one retrieved.

Fix: order retrieved chunks by relevance with the most relevant chunks placed at the start and/or end of the context block rather than the middle, and keep the total number of chunks handed to the model as small as it can be while still covering the answer — fewer, more relevant chunks reduce how much "middle" there is for an important one to get lost in.

Chunk boundary problems#

A fact can be split across a chunk boundary in a way that makes it unretrievable, or unusable, from either resulting chunk on its own:

Chunk 1: "...the refund policy for enterprise accounts changed as of the
new fiscal year. Under the updated terms,"

Chunk 2: "customers now have 45 days instead of the previous 30 days to
request a refund. This applies to all..."

A query asking "how many days do enterprise customers have for a refund" might retrieve Chunk 2, which has the number but no context establishing this is about enterprise accounts specifically — or retrieve Chunk 1, which has the context but not the actual number.

Fix: use overlap between consecutive chunks (repeating the tail of one chunk at the start of the next, covered in this track's building-a-rag-pipeline page) so a fact sitting near a boundary appears intact in at least one chunk, and prefer chunking along natural document structure (paragraphs, sections) over a rigid fixed-size cut that ignores where sentences and ideas actually end.

Stale indexes#

A RAG system's answers are only as current as its vector store's contents. If the underlying source documents change (a policy update, a corrected error, a new price) but the vector store isn't re-embedded and re-indexed to match, the system will confidently retrieve and present outdated information as if it were current — arguably worse than an LLM admitting it doesn't know, because it looks fully sourced and grounded while being wrong.

Source document updated: refund window changed from 30 to 45 days.
Vector store: still contains the old chunk saying "30 days," never re-indexed.

User: "What's the refund window?"
RAG system: "30 days" -- confidently wrong, and looks grounded because
it's citing a real (but stale) retrieved chunk.

Fix: treat ingestion as a continuously running pipeline (see this track's production-rag-architecture page), not a one-time job — re-index on every source document change where possible, or on a short enough schedule that staleness windows stay within an acceptable bound for how fast the underlying data actually changes.

Other well-documented failure modes#

• Irrelevant-but-confident retrieval — the vector search returns the k closest chunks even when none of them are actually a good match for the query (there's no minimum similarity threshold enforced), and the model then tries to answer from genuinely unhelpful context rather than recognizing there was no good match at all.
• Over-chunking a single coherent idea — splitting one continuous explanation into many small chunks can mean no single chunk, embedded alone, captures enough of the idea to rank highly for a related query, even though the full passage would have answered it well.
• Query-document vocabulary mismatch — a user's casual phrasing ("get my money back") not matching the source documents' formal phrasing ("refund policy") closely enough for either keyword or (to a lesser extent) vector search to bridge — addressed by hybrid search and query expansion, covered in this track's advanced-retrieval-techniques page.

Common mistakes#

• Assuming a bigger k (more retrieved chunks) always improves answer quality — beyond a point, more chunks mostly adds "lost in the middle" risk and irrelevant noise, not more useful information.
• Never enforcing a minimum similarity threshold on retrieval — without one, a query with genuinely no good match in the corpus still returns something, and the model may treat that irrelevant "something" as if it were real, usable context.
• Assuming an ingested corpus stays accurate indefinitely once it's indexed — real source documents change, and a RAG system without a re-indexing strategy will drift out of sync with reality, silently.