LLM Evaluation in CI

Behavior is the regression target

For a service that wraps an LLM, the thing your tests should check is not 'did the API respond' (it almost always does). It's 'did the model produce the kind of answer we expect for our prompts'. That's an eval: a structured set of (input, expected behavior) pairs that you score against the model.

Eval shape

• Dataset — N examples covering golden cases, edge cases, regressions you've fixed. Versioned in the repo.
• Scorer — function that takes a model output and returns a number (0–1, or pass/fail). Common scorers: exact match, contains, regex, semantic similarity, rubric judge (LLM-as-judge), pairwise preference.
• Threshold — minimum aggregate score to pass.

Where it runs in CI

1. On every PR — fast smoke subset (10–20 examples).
2. On every push to main — full eval (all examples).
3. Nightly — full eval against the latest model versions to catch upstream changes.

Cost discipline

An eval with 200 examples × 3 model variants × 5 PRs/day burns API budget. Cache results keyed on (prompt SHA, model version, scorer); only re-run when one of those changed. The Eval Quest covers this in depth.