Regression Testing and the Eval-Improvement Cycle

The regression suite is the system's institutional memory

Every fixed bug deserves a regression test. Every reported failure deserves a case in the dataset. Over months, the regression suite becomes a record of every mistake the system has made — and a guarantee none of them will silently come back.

The eval-improvement cycle

Eval flags failure — pre-release suite catches a regression, or a production sample shows a new failure mode.
Failure becomes a case — added to the regression dataset with the right tag.
Fix the system — prompt, retrieval, model upgrade, whatever the diagnosis suggests.
Eval confirms fix — the new case passes, no other cases regress.
Ship + monitor — production sampling watches for the same failure pattern recurring.

The regression suite never shrinks

It grows with every shipped fix. Cases that have been "easy passes" for six months don't get retired — that's exactly the moment a model upgrade silently regresses on them. Retire only when the underlying behavior is no longer relevant (a feature is removed).

Principle: Every fixed bug becomes a regression test. The dataset that grows with every release is the most valuable asset your team owns.

Diff against baseline, not against ideal

The right CI gate is "did pass rate drop vs the previous release?" not "did pass rate clear an absolute threshold?" Absolute thresholds drift. Diff-against-baseline naturally tightens as the system improves.

Code

Eval-improvement cycle as scripts·bash

# 1. Eval flags failure on PR
$ just eval-pr
  qa.long_doc.045  FAIL  judge: 'response cited a doc that wasn't retrieved'

# 2. Add the case to the regression set with a tag
$ python scripts/add_to_regression.py \
    --id qa.long_doc.045 \
    --tag 'citation-hallucination' \
    --reason 'Added after regression spotted on 2026-04-15'

# 3. Fix the prompt / retrieval / model / etc.
$ vim prompts/citation.txt

# 4. Re-run eval. The new case must pass; nothing else may regress.
$ just eval-pr
  qa.long_doc.045  PASS
  pass_rate: 0.94 → 0.96  ✓

# 5. Ship.
$ just deploy

Diff-against-baseline gate·python

def gate(current_results, baseline_results, max_drop=0.02):
    cur_rate = sum(r["pass"] for r in current_results) / len(current_results)
    base_rate = sum(r["pass"] for r in baseline_results) / len(baseline_results)
    diff = cur_rate - base_rate
    if diff < -max_drop:
        regressed = [r for r in current_results
                     if r["pass"] is False and any(b["id"] == r["id"] and b["pass"] for b in baseline_results)]
        return False, f"pass-rate dropped {diff:.1%}; specifically: {[r['id'] for r in regressed[:5]]}"
    return True, f"pass-rate diff {diff:+.1%} within tolerance"

Regression Testing and the Eval-Improvement Cycle

The regression suite is the system's institutional memory

The eval-improvement cycle

The regression suite never shrinks

Diff against baseline, not against ideal

Code

External links

Exercise

Progress

Comments 0