Benchmark Limitations and Custom Benchmarks

Why public benchmarks are not enough

Every public benchmark has the same fatal flaw for product evaluation: it is public. That means models are evaluated on (and often trained on) the same questions you want to use to compare them. Three structural problems follow:

Contamination — model has seen test items in training.
Goodhart — once a benchmark is known, model labs optimize for it specifically.
Distribution mismatch — your product's input distribution looks nothing like the benchmark's.

Why custom benchmarks always win in production

The benchmark that matters for your product is your product's own eval suite. Public benchmarks help you screen out obviously weak models; custom benchmarks discriminate between the survivors.

How to build a defensible custom benchmark

Sample real production traffic (with privacy controls).
Stratify across the dimensions that matter (intent, language, length, difficulty).
Curate 500-1000 cases with reference answers or rubric grading.
Hold out 100-200 cases that NEVER touch any model fine-tuning.
Version it like code; refresh it quarterly.

Principle: Public benchmarks tell you which models are in the running. Custom benchmarks tell you which one to ship.

The future-proofing argument

When the next model lands ("Claude X.Y", "GPT-N+1"), your custom benchmark is the only artifact that lets you decide whether to switch. Without it, every model upgrade is a leap of faith. With it, the upgrade is a numerical decision: ran our suite, beat our baseline, ship.

Code

Custom benchmark structure·json

{
  "benchmark": "acme-product-v3",
  "version": "2026-q2",
  "description": "500 cases sampled from production logs, stratified by intent and language",
  "strata": {
    "intent": ["qa", "navigation", "transactional", "exploratory"],
    "language": ["en", "es", "fr", "ja"],
    "difficulty": ["easy", "medium", "hard"]
  },
  "holdout_count": 100,
  "refresh_cadence": "quarterly",
  "contamination_policy": "no case ever appears in any fine-tuning corpus"
}

Detecting public-benchmark contamination·python

# A simple test: paraphrase a benchmark question and ask the model both versions.
# If it answers the original near-perfectly but the paraphrase poorly,
# memorization is the likely explanation.
import difflib

def paraphrase_test(model, original_q, paraphrased_q, expected):
    a1 = model.complete(original_q)
    a2 = model.complete(paraphrased_q)
    return {
        "original_correct": expected in a1.lower(),
        "paraphrase_correct": expected in a2.lower(),
        "answer_similarity": difflib.SequenceMatcher(None, a1, a2).ratio(),
    }

Benchmark Limitations and Custom Benchmarks

Why public benchmarks are not enough

Why custom benchmarks always win in production

How to build a defensible custom benchmark

The future-proofing argument

Code

External links

Exercise

Progress

Comments 0