RAG Evaluation

두 pipeline, 두 실패 mode, 4 metric

RAG system 은 단계 둘과 적어도 4가지 fail 방법 가져. Retrieval 단계가 wrong 또는 insufficient doc 가져올 수 있고; generation 단계가 hallucinate, summarize wrong, wrong 질문 답할 수 있어. 각 단계 독립적으로 평가해야 해.

Retrieval-단계 metric

Recall@k — top k 에서 retrieve 된 relevant doc 의 fraction.
Precision@k — top-k doc 중 relevant 한 fraction.
nDCG@k — normalized Discounted Cumulative Gain. 더 높이 ranked 된 relevant doc 보상.
MRR — Mean Reciprocal Rank. 첫 relevant doc 의 1/rank 평균.

Generation-단계 metric

Faithfulness — 답이 retrieved context 에 의해 supported 된 claim 만들어?
Answer Relevancy — 답이 user 의 질문 다뤄?
Citation accuracy — system 이 source cite 하면, citation 이 실제 supporting passage 에 link?

왜 분해된 metric 이 중요해

Aggregate "RAG quality score" 는 디버깅에 무용. Per-stage metric 이 정확히 어디 engineering effort 쓸지 알려줘:

낮은 recall → retriever 고침 (better embedding, hybrid search, query rewriting).
높은 recall, 낮은 faithfulness → generator prompt 고침, temperature 낮춤, citation 강제.
높은 faithfulness, 낮은 relevance → retriever 가 옳은 doc 찾는데 generator 가 다른 질문에 답 — 보통 prompt issue.

원칙: RAG 에서 retrieval 과 generation 항상 독립적으로 측정. Composite score 는 거짓말, per-stage metric 이 truth.

RAG eval set 만들기

각 case 에 대해 캡처: question, expected_relevant_doc_ids, reference_answer. 첫째가 retrieval score 가능하게; 셋째가 generation score 가능하게. 가운데가 bridge. 셋 다 없으면 eval 의 절반만 가능.

Code

Retrieval metric — recall 과 nDCG·python

import math

def recall_at_k(retrieved_ids, relevant_ids, k):
    top_k = retrieved_ids[:k]
    return sum(1 for d in top_k if d in relevant_ids) / max(len(relevant_ids), 1)

def ndcg_at_k(retrieved_ids, relevant_ids, k):
    relevance_scores = [1.0 if d in relevant_ids else 0.0 for d in retrieved_ids[:k]]
    dcg = sum(rel / math.log2(i + 2) for i, rel in enumerate(relevance_scores))
    ideal = sum(1.0 / math.log2(i + 2) for i in range(min(len(relevant_ids), k)))
    return dcg / ideal if ideal > 0 else 0.0

retrieved = ["doc_5", "doc_2", "doc_9", "doc_1", "doc_7"]
relevant = {"doc_2", "doc_7", "doc_3"}
print(recall_at_k(retrieved, relevant, 5))   # 0.667
print(ndcg_at_k(retrieved, relevant, 5))     # ~0.59

RAGAS 와 end-to-end RAG eval·python

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy, context_precision, context_recall

data = Dataset.from_dict({
    "question": ["How many users does Acme have?"],
    "answer": ["Acme has 2.5M users as of Q1 2026."],
    "contexts": [["Acme grew to 2.5M users in Q1 2026.", "Acme has offices in 12 countries."]],
    "ground_truth": ["2.5 million users"],
})

result = evaluate(
    data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# Each metric scored independently — you can see exactly which axis failed.

Citation-accuracy check·python

import re

def citation_accuracy(answer, retrieved_docs):
    """Does every numbered citation point to a real retrieved doc?"""
    cited_ids = set(re.findall(r'\[doc_(\d+)\]', answer))
    retrieved_ids = {d['id'] for d in retrieved_docs}
    bogus = cited_ids - retrieved_ids
    if bogus:
        return 0.0, f"hallucinated citations: {bogus}"
    if not cited_ids:
        return 0.0, "no citations in answer"
    return 1.0, "all citations point to retrieved docs"

두 pipeline, 두 실패 mode, 4 metric

Retrieval-단계 metric

Generation-단계 metric

왜 분해된 metric 이 중요해

RAG eval set 만들기

Code

External links

Exercise

Progress

댓글 0