Evals in the AI Workflow

Where evals plug in

An eval that runs only when a researcher remembers it is not an eval. The infrastructure question is: at which moments in the development loop does the suite fire?

Local development. Quick smoke evals (5-20 cases) run in seconds, before the engineer even pushes. just eval-fast.
Pull request. Full deterministic eval (100-500 cases) runs as a CI step. PR cannot merge if pass rate drops below threshold.
Pre-release. Full eval including paid LLM judges and human-rated cases. Gates the release tag.
Scheduled (nightly). Long-running suite over the full dataset, plus drift checks against production logs.
Online (production). Sampling layer captures live traffic, scores it, alerts on regressions, feeds new edge cases back into the offline dataset.

The PR-level eval is the highest-leverage integration

If you do exactly one thing, make it the PR gate. Most regressions are introduced one prompt edit at a time, and most of them survive only because nobody re-ran the suite. A green eval check on every PR catches them at the cheapest possible moment.

Principle: The earlier in the workflow an eval fires, the cheaper a regression is to fix. Local < PR < pre-release < production.

Production telemetry closes the loop

Sampling 1-5% of production traffic into an online eval pipeline is the single biggest force multiplier on offline evals. Anything you find in the wild becomes tomorrow's regression test. Tools like Braintrust and Inspect AI specifically support this loop.

Code

GitHub Actions — eval as a PR gate·yaml

# .github/workflows/evals.yml
name: Evals
on: [pull_request]
jobs:
  fast-eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with: { python-version: '3.12' }
      - run: pip install -r requirements.txt
      - name: Run fast eval suite
        env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: just eval-fast
      - name: Compare to baseline
        run: python scripts/eval_diff.py --base origin/main --threshold 0.02
      # PR fails if pass rate dropped by more than 2 points.

Production sampler — feed the offline dataset·python

import random
import time

def log_for_eval(request, response):
    """Sample 2% of production calls into the eval queue."""
    if random.random() < 0.02:
        record = {
            "timestamp": time.time(),
            "input": request["input"],
            "output": response["text"],
            "model": response["model"],
            "latency_ms": response["latency_ms"],
        }
        eval_queue.publish(record)

# Downstream worker scores each record async, alerts on anomalies,
# and adds the most informative ones to the offline regression set.

Evals in the AI Workflow

Where evals plug in

The PR-level eval is the highest-leverage integration

Production telemetry closes the loop

Code

External links

Exercise

Progress

Comments 0