MTEB, ARC-AGI, and Modern Frontier Benchmarks

Specialized and frontier benchmarks worth knowing

MTEB — Massive Text Embedding Benchmark

MTEB (Muennighoff 2023) is the de-facto benchmark for embedding models. 56+ tasks across retrieval, classification, clustering, semantic similarity, and reranking. If you're picking an embedding model for retrieval, MTEB is your first reference.

What to read on MTEB:

Retrieval sub-leaderboard — most relevant for RAG.
Reranking — relevant if you use a cross-encoder reranker.
Multilingual variants (MIRACL, MMTEB) — for non-English production.

ARC-AGI (Chollet 2019, ARC-AGI-2 in 2025)

Abstract-pattern visual puzzles where each task has a few input/output examples and a held-out test input. Designed specifically to resist memorization — every test item is novel. Frontier models scored 5-15% in 2023; reached 50-87% in 2025 with extensive test-time compute. Still considered a meaningful frontier challenge for general intelligence.

BIG-Bench Hard

BIG-Bench (Google 2022) bundles 200+ diverse tasks. BIG-Bench Hard is the 23 hardest. Useful for stress-testing a model's range across unfamiliar problem types.

FrontierMath

Research-mathematician-level problems. Designed to be impossible for memorization (problems are crafted by professional mathematicians and held secret). 2025 frontier models score <10%, making it one of the few unsaturated benchmarks for reasoning.

Principle: When public benchmarks saturate, the field invents harder ones. The "right" benchmark for 2026 evaluation is whatever frontier-relevant benchmark hasn't yet been beaten.

Code

MTEB — pick an embedding model·python

# pip install mteb
import mteb
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("intfloat/e5-large-v2")
tasks = mteb.get_tasks(task_types=["Retrieval"], languages=["eng"])
results = mteb.MTEB(tasks=tasks).run(model, output_folder="results/mteb")

# Each retrieval task reports nDCG@10. Average gives a single comparison number.
# Pick the best model for the retrieval pattern that matches your corpus.

MTEB, ARC-AGI, and Modern Frontier Benchmarks

Specialized and frontier benchmarks worth knowing

MTEB — Massive Text Embedding Benchmark

ARC-AGI (Chollet 2019, ARC-AGI-2 in 2025)

BIG-Bench Hard

FrontierMath

Code

External links

Exercise

Progress

Comments 0