Benchmark Literacy: MMLU to GPQA to HLE

Knowing what each benchmark measures, and which ones are still hard, is the difference between hype and useful comparison. The frontier saturates fast — what was a state-of-the-art number two years ago is sometimes uninteresting today.

Benchmark	Tests	Status (2026)
MMLU	57-subject multiple choice	Saturated — top models 88-92%
HumanEval	164 Python programming problems	Saturated — top models 95%+
GSM8K	8.5K grade-school math word problems	Saturated — top models 95%+
AIME 2024 / 2025	Competition math (USA Math Olympiad qualifier)	Hard — frontier 80-95%
SWE-bench	Real GitHub issues, agentic patches	Hard — frontier 60-75%
GPQA	PhD-level science multiple choice	Hard — frontier 80-85%
HLE (Humanity's Last Exam)	Hardest open multi-domain	Very hard — frontier <30%
RULER	Long-context retrieval at varying depths	Hard at long depths

For reference: Gemini 2.5 Pro AIME 2024 92%, GPQA 83%, SWE-bench 63%; Claude 3.7 Sonnet GPQA 84.8%; GPT-5 AIME 2025 94.6%, SWE-bench Verified 74.9%, hallucination rate ~4.8% in thinking mode.

Reading benchmark numbers responsibly

Benchmarks can be gamed (training on test sets, optimizing for benchmark style). Numbers without protocols are nearly meaningless. Look for: was the eval done with thinking enabled? what was the temperature? was the test set contaminated? did the model's training data leak the answers? When in doubt, run your own eval on tasks closer to your real use case.

Code

Building your own eval harness·python

# A 30-line harness beats most marketing benchmark numbers
import json

def evaluate(model_fn, eval_set):
    results = []
    for item in eval_set:
        out = model_fn(item['prompt'])
        passed = grade(out, item['expected'])  # task-specific grader
        results.append({**item, 'output': out, 'passed': passed})
    return results

def grade(output, expected):
    # For multiple-choice: exact match.
    # For code: run the code, check tests pass.
    # For free-form: rubric (consistency, length, on-topic).
    # The grader IS your eval — make it match what you actually care about.
    return output.strip() == expected.strip()
# Run this nightly against your candidates. The 'best model'
# becomes a measurable, reproducible answer.

Exercise

Build a 30-prompt eval harness covering three categories your application actually uses (e.g., 'follow instructions', 'extract structured data', 'short factual answer'). Run three candidate models. Compare results to whatever public benchmarks each model advertises. How well does public correlate with private?

Benchmark Literacy: MMLU to GPQA to HLE

Reading benchmark numbers responsibly

Code

External links

Exercise

Progress

Comments 0