The Evaluation Lifecycle

Evals are not a one-shot deliverable

The most common eval failure on a real project is "we wrote evals during the launch sprint and never looked at them again." Three months later the evals still pass while users complain on Slack. The dataset froze in time while the product moved on.

Healthy evals run a continuous loop:

Define — pick a behavior worth protecting and write what success looks like.
Curate — gather a dataset that represents that behavior, including failure-prone edge cases.
Measure — run the system, score outputs, summarize results.
Diagnose — when something fails, look at the actual outputs, not just the aggregate.
Improve — change the prompt, model, retrieval, or tool, then re-measure.
Monitor — sample production traffic; when the live distribution drifts from the offline dataset, refresh.

The diagnose step is where teams cheat

It is tempting to look at "pass rate dropped from 92% to 88%" and start tweaking the prompt. Don't. Open the failures. Read the outputs. The aggregate score is a smoke alarm; the failures are the fire. Nine times out of ten, the regression is concentrated in a specific input cluster — a new entity type, a longer document, a new language — and the right fix is targeted, not a global prompt rewrite.

Principle: Treat aggregate scores as the question, not the answer. Always read the failures.

The monitor step is where teams stop

An eval suite that does not get refreshed becomes a museum exhibit: nostalgic, irrelevant, and politically expensive to remove. Schedule a quarterly review. Sample 50 recent production conversations. Check whether your offline dataset still looks like them. If not, refresh.

Code

The lifecycle as a make/just/task target·bash

# Each step is a command anyone on the team can run.
just eval-define     # opens BEHAVIOR.md and the rubric template
just eval-curate     # samples production logs into datasets/staging.jsonl
just eval-run        # runs the full suite, writes results/<timestamp>.json
just eval-diff       # diffs the latest run against the previous baseline
just eval-failures   # opens failures in a side-by-side viewer
just eval-monitor    # samples last 7 days of prod, flags distribution drift

Diagnose pattern — cluster failures before fixing·python

from collections import Counter

def cluster_failures(results):
    """Group failures by tag so you don't fix the wrong thing."""
    failures = [r for r in results if not r["passed"]]
    by_tag = Counter()
    for f in failures:
        for tag in f.get("tags", ["untagged"]):
            by_tag[tag] += 1
    return by_tag.most_common()

# Output:
# [("long_documents", 14), ("polish_language", 8), ("untagged", 3), ...]
# Now you know: 14 of your 25 failures are long documents.
# Fix that cluster, not the prompt at large.

The Evaluation Lifecycle

Evals are not a one-shot deliverable

The diagnose step is where teams cheat

The monitor step is where teams stop

Code

External links

Exercise

Progress

Comments 0