Treating Evals as Code

~12 min · evaluation, engineering

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Evals deserve the same hygiene as the system under test

Evals are software. They have versions, dependencies, tests of their own (judge calibration), and lifecycle. Treat them with the same engineering discipline as the application code.

Practical hygiene

Eval scripts in source control alongside prompts.
Golden sets versioned (semantic version or date).
Run via a single command (pytest evals/) or a dedicated runner (Promptfoo, Langfuse).
Reports persisted in a queryable form (dashboards, sheets, internal app).
Eval failures triaged like test failures — flake, real regression, or eval bug.

The cultural piece

If only one person on the team can run the eval, the eval is fragile. Make it boringly easy. The reward is that prompt iteration becomes data-driven instead of vibes-driven.

Code

Eval as a pytest suite·python

# tests/test_support_prompt.py
import pytest
from evals.runner import run_case

@pytest.mark.parametrize("case", load_jsonl("goldens/support/v3.jsonl"))
def test_support_prompt(case, prompt_v8):
    out = call(prompt_v8, case["input"])
    assert all(s in out for s in case["must_contain"])
    assert not any(s in out for s in case["must_not_contain"])
    if "schema" in case:
        case["schema"].validate(out)

External links

Exercise

Wire your eval suite into pytest (or your test runner). Add a CI step that runs it on every PR touching prompts. Document the one-line command in the README.