GSM8K and Math Reasoning

Word problems are how we measure reasoning

GSM8K (Grade School Math 8K) is 8,500 grade-school math word problems requiring 2-8 reasoning steps to solve. Released by OpenAI in 2021, it became the canonical "does this model reason?" benchmark.

What GSM8K tests

Multi-step arithmetic in natural language.
Reading comprehension under quantitative constraints.
Robustness to irrelevant information ("Tom buys 3 apples and goes to the park...").
Chain-of-thought reasoning — historically a major win factor.

The chain-of-thought story

GSM8K is where the chain-of-thought (CoT) trick was popularized. Models that scored 17% accuracy direct-answering scored 50%+ when prompted to think step-by-step before answering. CoT is now baked into most modern models via training; explicit CoT prompts matter less than they used to.

Principle: GSM8K is for measuring multi-step reasoning. If your product needs reasoning, this benchmark is more relevant than MMLU.

Saturation and follow-ups

GSM8K is now ~90%+ on frontier models. The community moved to:

MATH — competition-level math problems, much harder than grade school.
AIME / HMMT — actual high school math competitions.
GSM-Hard — adversarial GSM8K with larger numbers and trickier wording.
FrontierMath (2024) — research-mathematician-level problems where 2025 frontier models still score <5%.

Code

Sample GSM8K problem·text

Question:
Natalia sold clips to 48 of her friends in April, and then she sold half
as many clips in May. How many clips did Natalia sell altogether in April
and May?

Answer (chain-of-thought):
Natalia sold 48 clips in April.
In May, she sold half as many: 48 / 2 = 24 clips.
Total: 48 + 24 = 72 clips.

#### 72

# The harness extracts the final number after `####` for scoring.

Run GSM8K with chain-of-thought·bash

lm_eval --model openai-completions \
  --model_args model=gpt-4o-mini \
  --tasks gsm8k_cot \
  --num_fewshot 8 \
  --output_path results/gsm8k.json

# 8-shot CoT prompting is the canonical setting.
# Frontier models score 90%+ on this, so it is mostly a sanity check
# rather than a discriminator anymore.

GSM8K and Math Reasoning

Word problems are how we measure reasoning

What GSM8K tests

The chain-of-thought story

Saturation and follow-ups

Code

External links

Exercise

Progress

Comments 0