The Eval-Gate Pattern

The pattern that ties this whole track together

An eval gate is a job in your deploy pipeline that runs the eval suite against the candidate version and refuses to promote it if any agreed metric regressed.

The shape

Build the candidate (model checkpoint, prompt set, code).
Run eval against the candidate. Compare to the current production baseline.
If candidate ≥ baseline within tolerance: promote, deploy.
If candidate < baseline: fail the deploy. Surface the diff (which prompts regressed) on the PR or deploy failure.

What makes this hard

Baseline drift — when do you update the baseline? Auto-update on each successful deploy is fine for monotonic improvements; for behavior changes, human approval is needed.
Score noise — eval scores have variance. A 1% drop might be noise. Define tolerance bands.
Cost — full eval on every PR is expensive. Smoke gate on PRs, full gate on merge to main.
Slice regressions — overall score might be flat, but Korean inputs degraded by 8%. Slice metrics catch this.

The promise

An eval gate means the next time someone tweaks a prompt 'just to make this one case work', the gate will tell them what they broke. No more silent regressions in production.

Code

Full eval-gate workflow·yaml

name: eval-gate
on:
  push: { branches: [main] }
  pull_request: {}
permissions:
  contents: read
  pull-requests: write

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: astral-sh/setup-uv@v3
      - run: uv sync --all-extras --dev
      - name: Fetch baseline
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
        run: |
          gh release download eval-baseline --pattern 'baseline.json' --output baseline.json
      - name: Run eval
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: uv run python -m evals.run --output candidate.json
      - name: Gate
        run: |
          uv run python -m evals.compare \
            --baseline baseline.json \
            --candidate candidate.json \
            --tolerance 0.02 \
            --report report.md
      - if: failure() && github.event_name == 'pull_request'
        uses: marocchino/sticky-pull-request-comment@v2
        with:
          path: report.md

  promote-baseline:
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    needs: eval
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: |
          # On a successful main merge, promote the candidate to baseline
          gh release upload eval-baseline candidate.json --clobber
        env:
          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The pattern that ties this whole track together

The shape

What makes this hard

The promise

Code

External links

Exercise

Progress

Comments 0