Eval Discipline: Catch Regressions Before Users Do

~14 min · eval, regression, quality

Level 0Observer

0 XP0/64 lessons0/13 achievements

0/150 XP to next level150 XP to go0% complete

Three classes of eval

You need unit evals (single prompt → expected pattern), integration evals (full tool loop → expected behavior), and quality evals (LLM-as-judge or human spot-check on a held-out set). Skip any one and regressions slip through that class.

Pin the model and the prompt

An eval suite is a measurement instrument. The model id must be date-pinned, the prompts must be checked into git, and the dataset must be versioned. Without these, the suite drifts and 'the eval improved' becomes meaningless.

Run on every prompt change

Treat prompts like code: changes flow through a PR, the eval suite runs in CI, regressions block merge. cwk-site uses this pattern for its content-generation prompts; cwkPippa uses it for the persona system prompt. The discipline is what keeps changes safe.

Principle: If the prompt is in git but the eval is not, the prompt is not really versioned. Version both, gate both.

Code

Unit eval pattern with pytest·python

import pytest
from anthropic import Anthropic

client = Anthropic()
MODEL = "claude-haiku-4-5-20251001"  # date-pinned for reproducibility

CLASSIFY_SYSTEM = "Reply with exactly one word: positive, negative, or neutral."

@pytest.mark.parametrize("text,expected", [
    ("I love this", "positive"),
    ("this is awful", "negative"),
    ("meh", "neutral"),
])
def test_classify(text, expected):
    r = client.messages.create(
        model=MODEL,
        max_tokens=8,
        system=CLASSIFY_SYSTEM,
        messages=[{"role": "user", "content": text}],
    )
    assert r.content[0].text.strip().lower() == expected

LLM-as-judge for quality scoring·python

JUDGE_PROMPT = """
Evaluate the candidate answer against the expected answer for a customer
support reply. Score 1-5 on factual accuracy, tone, and completeness.
Reply with JSON: {"factual": int, "tone": int, "completeness": int, "reasoning": str}
"""

def judge(question: str, expected: str, candidate: str) -> dict:
    r = client.messages.create(
        model="claude-sonnet-4-6",  # judge stays on a stronger model
        max_tokens=512,
        system=JUDGE_PROMPT,
        messages=[{
            "role": "user",
            "content": f"Question:\n{question}\n\nExpected:\n{expected}\n\nCandidate:\n{candidate}",
        }],
    )
    import json
    return json.loads(r.content[0].text)

External links

Exercise

Pick one critical prompt in your project. Write 5 unit eval cases, 1 integration case, and 1 LLM-as-judge case. Wire them into CI as a required check on changes to that prompt file.

Hint

If 'CI cost is too high' blocks you, run quality evals nightly via Batch API instead of per-PR.

Eval Discipline: Catch Regressions Before Users Do

Three classes of eval

Pin the model and the prompt

Run on every prompt change

Code

External links

Exercise

Progress

Comments 0