Regression Tests for Prompts

~12 min · evaluation, regression

Level 0Apprentice

0 XP0/100 lessons0/14 achievements

0/120 XP to next level120 XP to go0% complete

Prompt CI

If your prompts live in source control, the eval suite should run on every change as part of CI. The merge gate is "new prompt does not regress on the golden set." Without this, prompts drift across sprints and quality changes are invisible until a user complains.

What to enforce

Pass rate on the golden set must be ≥ baseline.
Per-category pass rate (refunds, complaints, etc.) must not regress more than X%.
Cost per call must not increase more than Y% without sign-off.
Latency p95 must not regress more than Z%.

What this changes

Prompt PRs include eval results in their description.
Failures are visible to reviewers, not buried.
The golden set becomes the team's shared definition of quality.

Code

GitHub Action gating prompt merges·yaml

name: prompt-eval
on: pull_request
jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
      - run: pip install -e .
      - run: python scripts/run_eval.py --golden-set goldens/v3.jsonl --gate

External links

Exercise

Add a CI job that runs your golden-set eval on every prompt change. Block merges on regression. Document the baseline.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousLLM-as-Judge — Use Cases and Pitfalls Next →Cost-Quality-Latency Triangles

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.