Evaluation vs Testing

Why software tests don't survive contact with LLMs

Twenty years of software engineering wisdom says: write a unit test, assert exact equality, run on every commit, fail the build on any regression. That entire muscle memory breaks the moment your function calls a probabilistic model.

The same input produces different outputs across calls (sampling temperature, top-p, stochastic decoding).
Even with temperature=0, model providers update model weights silently — the snapshot you tested last week may have shifted.
The set of acceptable outputs is rarely a single string. "Paris is the capital of France." and "The capital of France is Paris." are both correct.
Quality is graded, not binary. A 90% useful answer is materially different from a 60% one — both pass assert is_string(output).

What changes when you move to evals

Aspect	Testing	Evaluation
Output	Deterministic	Stochastic
Pass criterion	Exact equality	Threshold on a graded score
Failure semantics	Bug	Regression in distribution
Retry policy	Flaky test	Statistical variance — run N times, average
Coverage	Lines / branches	Behaviors / edge cases / personas
Cost per run	Free (CPU)	Paid (LLM API calls × N samples × graders)

Principle: Evals are tests with a budget, a tolerance band, and a sampling strategy. They behave like statistical hypothesis tests, not boolean assertions.

Where they overlap — and where they don't

You should still write deterministic tests. Test that your retrieval function returns the right document IDs given a known query. Test that your tool-call parser raises on malformed JSON. Test that your prompt template substitutes variables correctly. Those are software bugs and software tests catch them. Evals layer on top, measuring the model's behavior given a correctly-wired system. Conflating the two — running an LLM call inside a unit test — produces flaky CI and angry teammates.

Code

The wrong way — deterministic test on a stochastic system·python

# This will pass on Tuesday, fail on Wednesday, and you will blame CI.
def test_summarizer():
    output = summarize_article("...long text...")
    assert output == "This article is about climate change."  # ❌ brittle

The right way — eval with threshold and N samples·python

from statistics import mean

def eval_summarizer(article, expected_topics, n_samples=5):
    """Run the summarizer N times, score each, return distribution."""
    scores = []
    for _ in range(n_samples):
        output = summarize_article(article)
        coverage = topic_coverage_grader(output, expected_topics)  # 0..1
        scores.append(coverage)
    return {
        "mean": mean(scores),
        "min": min(scores),  # worst-case is what production users feel
        "n": n_samples,
        "pass": min(scores) >= 0.7,  # threshold, not equality
    }

Pytest hybrid pattern — deterministic plumbing + eval blocks·python

import pytest

# Deterministic — runs on every commit, fast, free.
def test_prompt_template_substitution():
    rendered = render_prompt("hello {name}", name="world")
    assert rendered == "hello world"

# Eval — runs on a schedule or before release, slower, costs money.
@pytest.mark.eval
def test_summarizer_quality():
    result = eval_summarizer(SAMPLE_ARTICLE, EXPECTED_TOPICS, n_samples=10)
    assert result["pass"], f"summarizer regression: {result}"

Evaluation vs Testing

Why software tests don't survive contact with LLMs

What changes when you move to evals

Where they overlap — and where they don't

Code

External links

Exercise

Progress

Comments 0