RAGAS: RAG-Specific Evaluation

The framework built specifically for RAG pipelines

RAGAS (Retrieval-Augmented Generation Assessment) provides metrics designed for the four-stage failure space of a RAG system: retrieval quality, context relevance, answer faithfulness, and answer relevance. If you ship RAG, you want it.

The four core RAGAS metrics

Faithfulness — Does the answer make claims supported by the retrieved context? (catches hallucination)
Answer Relevancy — Does the answer address the question? (catches off-topic responses)
Context Precision — Of the retrieved chunks, how many are actually relevant? (catches retriever pulling junk)
Context Recall — Did the retriever find all the chunks needed to answer? (catches retriever missing crucial info)

Together they decompose "is this RAG output good?" into the four sub-questions that tell you which part failed.

Why this decomposition matters

A vague "answer is bad" tells the team nothing. RAGAS scores tell you: high faithfulness + low recall = retriever missing docs. Low faithfulness + high recall = generator hallucinating despite having the right context. Each diagnosis points to a different fix.

Principle: If you ship RAG, you must measure faithfulness and recall as independent metrics. A composite score hides which half of your pipeline broke.

Implementation reality

RAGAS uses LLM-as-Judge under the hood for most of its metrics, so it inherits judge costs and biases. Calibrate against humans on your specific RAG corpus before trusting absolute numbers. Use the metrics as relative signals (this version vs that version) more than as absolute quality measurements.

Code

Install and basic eval·python

# pip install ragas
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)
from datasets import Dataset

data = Dataset.from_dict({
    "question": ["What is the capital of France?"],
    "answer": ["Paris is the capital of France."],
    "contexts": [["France's capital is Paris. It has a population of 2.1M."]],
    "ground_truth": ["Paris"],
})

result = evaluate(
    data,
    metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
)
print(result)
# {'faithfulness': 1.0, 'answer_relevancy': 0.95, 'context_precision': 1.0, 'context_recall': 1.0}

Decompose a failing case·python

# Three diagnostic profiles, three different fixes:
#
# 1) low faithfulness, high recall
#    → generator is hallucinating despite having the right context
#    → fix: tighter prompt, citation enforcement, smaller model temperature
#
# 2) high faithfulness, low recall
#    → retriever is not finding the right chunks
#    → fix: better embeddings, hybrid search, query rewriting
#
# 3) high faithfulness, high recall, low answer relevancy
#    → retrieved correct docs, generated faithful summary, but didn't
#      answer the question the user asked
#    → fix: prompt engineering — explicitly route the answer to the question

RAGAS: RAG-Specific Evaluation

The framework built specifically for RAG pipelines

The four core RAGAS metrics

Why this decomposition matters

Implementation reality

Code

External links

Exercise

Progress

Comments 0