DeepEval: Pytest-Native Python Framework

Evals that feel like Python tests

DeepEval (v3.x as of mid-2026) is the Python-native eval framework. It integrates with pytest, ships 30+ named metrics across RAG / agentic / safety / custom categories, and offers an optional cloud dashboard via Confident AI. Best fit when your eval logic lives in Python and you already use pytest.

What you get

Pytest integration — write evals as pytest tests with assert_test.
Built-in metrics — Faithfulness, AnswerRelevancy, Hallucination, Toxicity, Bias, ContextualPrecision/Recall/Relevancy, ToolCallAccuracy, AgentGoalAccuracy, plus GEval (custom LLM-as-Judge).
End-to-end and tracing modes — black-box on final output, or white-box on traces of your pipeline.
Cloud dashboard via deepeval login for shared experiment history.
Synthesizer for generating test data.

Where it shines

DeepEval fits naturally into Python codebases. Your eval suite is one more tests/eval/ directory; pytest collects it; CI runs it. Custom metrics via GEval let you define LLM-as-Judge logic in declarative criteria without writing prompts manually.

Where it doesn't fit

If your team is not on Python, the friction is real. If you want a no-code GUI, look at Braintrust or Confident AI's hosted UI instead.

Principle: DeepEval is the right pick when your eval lives next to your Python code and you want it to feel like every other test.

Code

Install and login·bash

pip install -U deepeval

# Optional: login to Confident AI for cloud dashboard
deepeval login

# Run a single test file
deepeval test run test_eval.py

# Or via pytest
pytest test_eval.py -v

test_eval.py — GEval custom metric·python

from deepeval import assert_test
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import GEval

def test_correctness():
    correctness = GEval(
        name="Correctness",
        criteria="Determine whether the actual output is factually correct given the expected output.",
        evaluation_params=[
            LLMTestCaseParams.ACTUAL_OUTPUT,
            LLMTestCaseParams.EXPECTED_OUTPUT,
        ],
        threshold=0.5,
    )
    case = LLMTestCase(
        input="What is the capital of France?",
        actual_output="Paris is the capital of France.",
        expected_output="Paris",
    )
    assert_test(case, [correctness])

RAG-specific built-in metrics·python

from deepeval.metrics import (
    AnswerRelevancyMetric,
    FaithfulnessMetric,
    ContextualPrecisionMetric,
    ContextualRecallMetric,
    HallucinationMetric,
)

faithfulness = FaithfulnessMetric(threshold=0.7)
relevancy = AnswerRelevancyMetric(threshold=0.7)

case = LLMTestCase(
    input="What is the latest iPhone?",
    actual_output="The iPhone 17 Pro has a 48MP main camera.",
    retrieval_context=["iPhone 17 Pro features a 48MP main camera, A19 chip, ..."],
)
assert_test(case, [faithfulness, relevancy])

DeepEval: Pytest-Native Python Framework

Evals that feel like Python tests

What you get

Where it shines

Where it doesn't fit

Code

External links

Exercise

Progress

Comments 0