RAG Evaluation

Two pipelines, two failure modes, four metrics

A RAG system has two stages and at least four ways to fail. The retrieval stage may pull wrong or insufficient docs; the generation stage may hallucinate, summarize wrong, or answer the wrong question. You must evaluate each stage independently.

Retrieval-stage metrics

Recall@k — fraction of relevant docs retrieved in the top k.
Precision@k — fraction of top-k docs that are relevant.
nDCG@k — normalized Discounted Cumulative Gain. Rewards relevant docs ranked higher.
MRR — Mean Reciprocal Rank. Average of 1/rank of the first relevant doc.

Generation-stage metrics

Faithfulness — does the answer make claims supported by the retrieved context?
Answer Relevancy — does the answer address the user's question?
Citation accuracy — if your system cites sources, do citations link to actual supporting passages?

Why decomposed metrics matter

An aggregate "RAG quality score" is useless for debugging. Per-stage metrics tell you exactly where to spend engineering effort:

Low recall → fix the retriever (better embeddings, hybrid search, query rewriting).
High recall, low faithfulness → fix the generator prompt, lower temperature, enforce citations.
High faithfulness, low relevance → the retriever finds correct docs but the generator answers a different question — usually a prompt issue.

Principle: Always measure retrieval and generation independently in RAG. A composite score lies; per-stage metrics tell the truth.

Build a RAG eval set

For each case, capture: question, expected_relevant_doc_ids, reference_answer. The first lets you score retrieval; the third lets you score generation. The middle is the bridge. Without all three, you can only do half the eval.

Code

Retrieval metrics — recall and nDCG·python

import math

def recall_at_k(retrieved_ids, relevant_ids, k):
    top_k = retrieved_ids[:k]
    return sum(1 for d in top_k if d in relevant_ids) / max(len(relevant_ids), 1)

def ndcg_at_k(retrieved_ids, relevant_ids, k):
    relevance_scores = [1.0 if d in relevant_ids else 0.0 for d in retrieved_ids[:k]]
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores))
    ideal = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), k)))
    return dcg / ideal if ideal > 0 else 0.0

retrieved = ["doc_5", "doc_2", "doc_9", "doc_1", "doc_7"]
relevant = {"doc_2", "doc_7", "doc_3"}
print(recall_at_k(retrieved, relevant, 5))   # 0.667
print(ndcg_at_k(retrieved, relevant, 5))     # ~0.59

End-to-end RAG eval with RAGAS·python

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

data = Dataset.from_dict({
    "question": ["How many users does Acme have?"],
    "answer": ["Acme has 2.5M users as of Q1 2026."],
    "contexts": [["Acme grew to 2.5M users in Q1 2026.", "Acme has offices in 12 countries."]],
    "ground_truth": ["2.5 million users"],
})

result = evaluate(
    data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# Each metric scored independently — you can see exactly which axis failed.

Citation-accuracy check·python

import re

def citation_accuracy(answer, retrieved_docs):
    """Does every numbered citation point to a real retrieved doc?"""
    cited_ids = set(re.findall(r'\[doc_(\d+)\]', answer))
    retrieved_ids = {d['id'] for d in retrieved_docs}
    bogus = cited_ids - retrieved_ids
    if bogus:
        return 0.0, f"hallucinated citations: {bogus}"
    if not cited_ids:
        return 0.0, "no citations in answer"
    return 1.0, "all citations point to retrieved docs"

Two pipelines, two failure modes, four metrics

Retrieval-stage metrics

Generation-stage metrics

Why decomposed metrics matter

Build a RAG eval set

Code

External links

Exercise

Progress

Comments 0