RAG Interview Questions

A curated set of RAG interview questions covering the design trade-offs that come up when discussing retrieval-augmented systems in practice.

Fundamentals#

Q: Does RAG eliminate hallucination? No — it substantially reduces it by grounding the model's answer in real, retrieved text instead of relying purely on memorized training data, but it doesn't eliminate the risk entirely. The model can still misread, misquote, or blend the retrieved context incorrectly, and if retrieval itself returns irrelevant or missing chunks, the model may fall back on its own (possibly wrong) prior knowledge unless explicitly instructed not to.

Q: What are the trade-offs in choosing a chunking strategy? Smaller chunks retrieve more precisely, since each one is narrowly about one thing, but risk losing surrounding context needed to make sense of a fact on its own. Larger chunks preserve more context but dilute relevance — a large chunk embeds as one blurred vector even if only a small part of it is actually relevant to a given query, and wastes context window space if retrieved. Overlapping consecutive chunks helps avoid losing information that happens to fall right at a chunk boundary.

Q: How is vector search different from keyword search? Keyword search matches on the literal presence of search terms (or their stems/synonyms via specific rules); it's precise for exact terms but misses semantically related content phrased differently. Vector search compares the meaning of the query and documents as embeddings, using similarity metrics like cosine similarity, so it can find relevant content that shares no exact words with the query, at the cost of sometimes missing an exact rare term or identifier that doesn't carry strong semantic signal. Many production systems combine both ("hybrid search") to get the strengths of each.

Q: When would you choose fine-tuning over RAG, or vice versa? RAG is the better fit when the model needs access to specific, current, or private facts/documents, and you want answers grounded in a checkable source with easy updates — just update the document store, no retraining needed. Fine-tuning is the better fit when you need to change the model's behavior, its style, tone, output format, or how it performs a specialized task, rather than give it new facts; fine-tuning doesn't reliably teach a model new, precise factual knowledge, and updating facts learned this way means retraining. In practice, many production systems use both: fine-tuning for behavior/format, RAG for facts.

Q: Why does chunk retrieval quality matter more than people initially assume? Because the generation step can only work with what it's given — if the retrieval step returns irrelevant, incomplete, or missing chunks, the model has no good material to answer from, regardless of how capable it is. RAG system quality is very often bottlenecked by retrieval (chunking strategy, embedding model quality, k value) rather than by the language model's generation ability.

Advanced retrieval#

Q: Why do production RAG systems typically combine keyword search with vector search instead of using vector search alone? Vector search is strong at semantic similarity but can miss exact identifiers, rare technical terms, or specific codes that don't carry strong "meaning" signal — precisely where keyword search (like BM25) excels. Combining both ("hybrid search") catches both semantically related content phrased differently and exact-term matches that pure vector search sometimes ranks too low, which is why most production systems that care about retrieval quality use both rather than picking one.

Q: What does a reranker add on top of standard vector or hybrid search? The first retrieval pass (vector and/or keyword search) is optimized to be fast across a large corpus, using relatively cheap scoring to produce a shortlist. A reranker is a slower but more accurate model that re-scores just that shortlist (often the top 20-50 candidates) directly against the query, producing a better final ordering than the fast first pass alone could — a two-stage pattern used broadly in search systems, not unique to RAG.

Evaluation and production#

Q: Why does a RAG system need to evaluate retrieval and generation separately, rather than just checking if the final answer looks right? Because the two stages can fail independently: a system can retrieve exactly the right chunks and still generate a poor or unfaithful answer from them, or retrieve nothing useful and still produce a correct-looking answer from the model's own training data by coincidence. Measuring retrieval (precision/recall@k) and generation (answer relevance, faithfulness) separately tells you specifically which stage to fix when something goes wrong.

Q: What's the difference between "answer relevance" and "faithfulness" in RAG evaluation? Answer relevance asks whether the generated answer actually addresses the question that was asked. Faithfulness asks whether every claim in that answer is actually supported by the retrieved context, rather than added from the model's own prior knowledge. A faithful answer can still score poorly on relevance if it's grounded but off-topic; a relevant answer can score poorly on faithfulness if it's on-topic but includes a fabricated detail not present in the retrieved context.

Q: What is "lost in the middle," and how does it affect RAG system design? It's a well-documented tendency for LLMs to use information placed near the start or end of their context window more reliably than information buried in the middle. For a RAG system stuffing several retrieved chunks into one prompt, this means the single most relevant chunk can be under-weighted if it happens to land in the middle of that block. The practical fix is ordering chunks by relevance toward the edges of the context and keeping the total number of chunks handed to the model as small as it can be while still covering the answer.

Q: Why is a stale vector index a particularly dangerous kind of RAG failure? Because it doesn't look like a failure at all — the system retrieves a real, previously-correct chunk and answers confidently, citing what appears to be a legitimate source, even though the underlying source document has since changed. This can be more misleading than an outright "I don't know," since it presents outdated information with the full appearance of being grounded and current. The fix is treating ingestion as a continuously running pipeline that re-indexes on source changes, not a one-time job.