Meta-Evaluation and Building Eval Culture

Are your evaluations actually measuring quality?

The hardest question in evals: "is the metric I'm watching the metric I should be watching?" Meta-evaluation is the practice of evaluating your evaluations.

Three meta-eval questions

Correlation with humans — does the metric move when human-rated quality moves? Compute correlation between automated scores and human ratings on the same set.
Correlation with users — does the metric move when user-perceived quality moves? Compute correlation between automated scores and downstream user signals (thumbs, retention, completion).
Robustness to gaming — if a model is optimized against the metric, does the optimization improve real quality or just the score?

If correlation is weak, the metric is theatre. Replace it.

Building eval culture — the team practices

Make it easy — running evals is a single command, not a 30-step ceremony.
Make it visible — dashboards in the team room. Eval scores quoted in PR descriptions.
Make it required — no PR merges without an eval result. No release without a green eval check.
Make it shared — every team member contributes test cases. The dataset is communal.
Make it celebrated — when someone catches a regression with an eval, recognize it. The first time the suite saves an embarrassment is the moment culture takes hold.

Principle: Evals stick when the team feels them. A metric in a dashboard nobody opens is no metric at all.

The production-stack eval pipeline

A mature stack: JSONL datasets in git → Python or YAML eval runner → Braintrust or self-hosted dashboard → CI/CD gate → production sampling feeding back into the dataset → quarterly meta-eval against humans. Each piece is small; the loop is the value.

Code

Meta-eval — correlate metric with human ratings·python

from scipy.stats import spearmanr

# 100 outputs, each with: (automated_metric_score, human_rating_1_to_5)
automated = [...]   # list of floats from your metric
human = [...]       # list of integers from human rater

rho, pval = spearmanr(automated, human)
print(f"Spearman correlation: {rho:.3f} (p={pval:.4f})")

# rho > 0.7 → metric tracks human judgment well
# rho 0.4-0.7 → useful but noisy; treat with care
# rho < 0.4 → metric is theatre; replace it

End-to-end eval stack as a Makefile·text

Test data        +   Metrics       +   Runner          +   Dashboard       +   CI/CD
   |                  |                  |                    |                    |
   └── JSONL files    └── Python /       └── promptfoo /      └── Braintrust /     └── GitHub Actions
       in git             scoring             pytest /             custom              blocks bad PRs
                          functions           DeepEval /           webapp
                                              hand-rolled

Meta-Evaluation and Building Eval Culture

Are your evaluations actually measuring quality?

Three meta-eval questions

Building eval culture — the team practices

The production-stack eval pipeline

Code

External links

Exercise

Progress

Comments 0