The skepticism stack
Benchmark numbers in announcements are rarely outright lies, but they are almost always presented to maximize favorable framing. Below is the skepticism stack to apply to every benchmark claim you read.
1. Cherry-picking
Companies show benchmarks where they win and quietly omit ones where they don't. If a coding-focused announcement skips HumanEval, that is information. If a math-focused announcement skips GSM8K, that is information. The omissions tell you what the announcement does not want to talk about.
2. Benchmark contamination
Pretraining data crawls the internet. The internet contains benchmark test sets. Many widely cited benchmarks are partially or fully memorized by frontier models. Inflated scores from contamination look exactly like real capability gains, and only careful held-out evaluation can distinguish them.
3. Evaluation protocol differences
Few-shot count (0-shot vs 5-shot vs CoT-prompted), prompt format, answer extraction method (regex vs structured output vs LLM-judge) — small changes can swing scores by 5–10%. Two announcements quoting the same benchmark may not be using the same protocol.
4. Saturation
MMLU, HellaSwag, ARC-Easy are nearly saturated — the top models score within 1–2% of each other, well within evaluation noise. A "1.5% gain on MMLU" headline is meaningless. Look for benchmarks that still discriminate (Frontier Math, ARC-AGI, SWE-Bench Verified, GPQA, AIME).
5. The real test
How does the model perform on YOUR actual workload? No published benchmark substitutes for this. The cheapest sanity check is to run a small custom eval (10–50 examples that mirror your real use case) on every candidate model. Half the time the leaderboard ordering reverses on your data — and that is the only ordering that matters.
Useful third-party sources
For latency, cost, and basic capability evaluation, Artificial Analysis maintains independent benchmarks across providers. The Open LLM Leaderboard at Hugging Face has continuously updated rankings. LMSYS Chatbot Arena uses pairwise human preference and resists single-axis cherry-picking better than most. None of these substitute for your own eval, but they are useful triangulation points.