MMLU: Massive Multitask Language Understanding

The most-cited LLM benchmark in the world

MMLU (Massive Multitask Language Understanding), introduced by Hendrycks et al. in 2020, is a multiple-choice benchmark covering 57 academic subjects from elementary mathematics to professional law. Each item is a question with four options; the model picks one. Scored on accuracy.

Why MMLU mattered

Broad coverage — STEM, humanities, social sciences, business, professional exams. Makes it harder to memorize one domain to inflate the score.
Multiple-choice format — deterministic scoring, easy to run.
Adopted as a default reporting metric by every major model lab.

What MMLU does NOT tell you

It does not measure reasoning. The questions test recall and comprehension; multi-step reasoning is rarely required.
It does not measure long-context understanding. Each question is a few sentences.
It does not measure agentic capability, tool use, or planning.
It does not predict product quality. A 85% MMLU model and a 75% MMLU model can perform identically on your specific RAG pipeline.

Principle: MMLU is the SAT of LLMs — useful as a coarse capability sort, useless as a quality predictor for any specific product.

The contamination problem

MMLU was published in 2020. Every major model trained since has seen it (or close paraphrases) in training data. A high MMLU score may reflect memorization, not understanding. The community responded by creating MMLU-Pro (harder questions, more options) and similar harder follow-ups, but the original is still widely cited.

MMLU-Pro and follow-ups

MMLU-Pro (Wang et al., 2024) increases option count from 4 to 10, requires multi-step reasoning, and reduces dataset overlap with public training data. It is the more honest 2025-2026 successor when comparing modern models.

Code

Run MMLU via lm-eval-harness·bash

lm_eval --model hf \
  --model_args pretrained=Qwen/Qwen2.5-7B-Instruct \
  --tasks mmlu \
  --num_fewshot 5 \
  --batch_size auto \
  --output_path results/mmlu.json

# Reports per-subject + macro-average accuracy.
# Headline number is usually the macro-average across 57 subjects.

Sample MMLU question (anatomy)·text

Subject: anatomy

Which of the following structures is part of the small intestine?
A) Ascending colon
B) Cecum
C) Sigmoid colon
D) Jejunum

Correct: D

# Multiple-choice format. Model picks the letter. Scored on accuracy.

MMLU: Massive Multitask Language Understanding

The most-cited LLM benchmark in the world

Why MMLU mattered

What MMLU does NOT tell you

The contamination problem

MMLU-Pro and follow-ups

Code

External links

Exercise

Progress

Comments 0