Evaluating RAG Systems

Why RAG needs two separate kinds of evaluation#

A RAG system has two genuinely distinct stages — retrieval and generation — and each can fail independently of the other. A system can retrieve exactly the right chunks and still generate a poor answer from them (a generation failure), or retrieve irrelevant chunks and still stumble into a correct-sounding answer from the model's own training data (a retrieval failure masked by a lucky generation). Evaluating "did the final answer look right" alone conflates these two failure modes and makes it much harder to know what to actually fix.

Retrieval metrics#

These measure whether the retrieval step found the right source material, independent of what the model did with it afterward — typically measured against a labeled evaluation set where each test question has known relevant document(s):

• Precision@k — of the k chunks retrieved, what fraction are actually relevant to the query? Low precision means the model is being handed a lot of irrelevant noise alongside (or instead of) the useful chunks.
• Recall@k — of all the chunks that were actually relevant somewhere in the corpus, what fraction did the top-k retrieval actually surface? Low recall means the genuinely relevant chunk exists in the corpus but the retrieval step never found it at all — no matter how good the generation step is, it has nothing to work with.
• Mean Reciprocal Rank (MRR) — for each query, how high up the ranked results did the first relevant chunk appear? Useful when getting at least one good chunk near the top matters more than retrieving every relevant chunk.

def precision_at_k(retrieved_ids, relevant_ids, k):
    top_k = retrieved_ids[:k]
    relevant_in_top_k = len(set(top_k) & set(relevant_ids))
    return relevant_in_top_k / k

def recall_at_k(retrieved_ids, relevant_ids, k):
    top_k = retrieved_ids[:k]
    relevant_in_top_k = len(set(top_k) & set(relevant_ids))
    return relevant_in_top_k / len(relevant_ids) if relevant_ids else 0

Generation metrics: relevance and faithfulness#

Once retrieval is measured separately, generation quality itself is usually evaluated along two distinct dimensions, often using an LLM-as-judge (see this site's LLM tutorials on evaluation) since there's rarely one single "correct" phrasing to match exactly:

• Answer relevance — does the generated answer actually address the question that was asked? A technically accurate answer that dodges the actual question, or answers a related-but-different question, scores poorly here even if every individual claim in it is true.
• Faithfulness (groundedness) — is every claim in the answer actually supported by the retrieved context, or did the model add something not present in the source material (a subtle hallucination on top of otherwise-correct retrieval)? This is checked by comparing each claim in the generated answer against the retrieved chunks, not against general world knowledge.

You are evaluating a RAG system's answer for faithfulness.

Context provided to the model:
"Enterprise customers may request a refund within 45 days of purchase."

Generated answer:
"Enterprise customers get a 45-day refund window, and can also receive a
20% loyalty discount on their next purchase."

Is every claim in the answer supported by the context? Answer NO -- the
45-day refund claim is supported, but the loyalty discount claim is not
present anywhere in the provided context and appears to be fabricated.

Putting it together#

A RAG system with strong retrieval metrics but poor faithfulness scores points squarely at the generation prompt (tighten the "answer only using the context" instruction); strong faithfulness but poor recall points squarely at the retrieval pipeline (chunking strategy, embedding model, or k) — this separation is exactly why measuring both stages independently is worth the extra evaluation setup.

Common mistakes#

• Evaluating only the final answer's apparent correctness, without separately checking retrieval quality — this makes it impossible to tell whether a bad answer came from bad retrieval, bad generation, or both.
• Treating faithfulness and factual correctness as the same thing — a faithful answer is one that accurately reflects its retrieved context, even if that context itself happens to be wrong; faithfulness measures whether the model stuck to its sources, not whether the sources were true.
• Skipping retrieval evaluation because "the vector database seems to be working" — a retrieval system can run without errors and still consistently retrieve mediocre chunks; that failure mode is invisible without a labeled evaluation set to measure against.