What Evaluation Means for LLMs

Evaluation is not testing — and it is not benchmarking

People talk past each other when they say "evals." Three overlapping but distinct activities live under that word, and conflating them produces the wrong tool every time.

Testing answers does my code do what I told it? Inputs and outputs are deterministic. assert add(2, 3) == 5 either passes or it does not.
Benchmarking answers how does this model rank against others on a public dataset? Useful for model selection, irrelevant for product quality.
Evaluation answers does my system produce outputs that are good for my users on inputs that look like my users' inputs? Outputs are non-deterministic, "good" is contextual, and users keep moving the goalposts.

What an LLM eval actually measures

An eval combines four ingredients into a measurement:

A test dataset — inputs that resemble the real workload, plus optional reference answers.
A system under test — a prompt, a model, a retrieval pipeline, a tool loop, a multi-agent system, or any composition.
A grader — the function that decides whether an output is acceptable. Can be exact-match, regex, embedding similarity, an LLM judge, a Python validator, or a human rater.
A reporting layer — the dashboard, CI gate, or notebook that turns numbers into decisions.

Principle: An eval is only as honest as its weakest ingredient. A pristine grader on a junk dataset measures nothing. A perfect dataset with a sloppy grader measures the sloppy grader.

Online vs offline evals

Offline evals run on a fixed dataset before deployment. Think CI/CD. They protect you from the obvious regressions. Online evals run on live traffic after deployment — sampled production logs, A/B comparisons, user feedback, telemetry. They catch what your offline dataset does not represent. You need both. Teams that only do offline are blind to drift; teams that only do online are blind to regressions until users complain.

Code

Anatomy of an eval — pseudo-code·python

# The four ingredients in their simplest form.
def run_eval(system, dataset, grader):
    results = []
    for case in dataset:
        output = system(case["input"])
        score, reason = grader(output, case.get("reference"))
        results.append({"input": case["input"], "output": output, "score": score, "reason": reason})
    return summarize(results)


def summarize(results):
    pass_rate = sum(r["score"] >= 1.0 for r in results) / len(results)
    avg = sum(r["score"] for r in results) / len(results)
    return {"pass_rate": pass_rate, "avg_score": avg, "n": len(results)}

Three graders, three measurement philosophies·python

def exact_match_grader(output, reference):
    score = 1.0 if output.strip() == reference.strip() else 0.0
    return score, f"exact match: {score==1.0}"


def contains_grader(output, reference):
    score = 1.0 if reference.lower() in output.lower() else 0.0
    return score, f"reference appears in output: {score==1.0}"


def llm_judge_grader(output, reference, judge_model):
    prompt = f"Output: {output}\nReference: {reference}\nIs the output factually consistent with the reference? Reply 'yes' or 'no' followed by one sentence."
    verdict = judge_model.complete(prompt).lower()
    score = 1.0 if verdict.startswith("yes") else 0.0
    return score, verdict

What Evaluation Means for LLMs

Evaluation is not testing — and it is not benchmarking

What an LLM eval actually measures

Online vs offline evals

Code

External links

Exercise

Progress

Comments 0