Code Benchmarks: HumanEval, MBPP, SWE-bench

Three benchmarks that measure different code skills

Code generation has its own benchmark family. The three most-cited tell you different things.

HumanEval (Chen et al. 2021, OpenAI)

164 hand-written Python programming problems with unit tests. Models generate code; the harness runs the tests; score is fraction passing (pass@1, pass@10). Saturated for top models in 2024 — frontier scores hit 90%+. Useful as a basic capability check, not as a discriminator anymore.

MBPP (Mostly Basic Python Problems, Google 2021)

1,000 simpler Python problems. Lower ceiling than HumanEval; mostly used in conjunction with HumanEval for cross-validation.

SWE-bench (Princeton 2023, with extensions)

The interesting one. SWE-bench gives the model a real GitHub issue from a real OSS repo and asks it to produce a patch that passes the project's existing test suite. This is a multi-file, repo-context, agent-style task. SWE-bench-Verified (a subset blessed by maintainers) is the de-facto modern code-agent benchmark. Top scores in 2026 sit around 60-70% — still far from saturated, much more predictive of "can this model help with real software work."

Principle: HumanEval tells you a model can write Python; SWE-bench tells you it can do software engineering. The gap is enormous.

What none of them test

Code review — reading code is not in scope.
Tooling — using shell, debuggers, IDE features.
Long-running iteration — most are one-shot or small-loop.
Languages other than Python and TypeScript (most of them).

Code

HumanEval problem (sample)·python

# HumanEval/0

def has_close_elements(numbers: List[float], threshold: float) -> bool:
    """ Check if in given list of numbers, are any two numbers closer to each other than
    given threshold.
    >>> has_close_elements([1.0, 2.0, 3.0], 0.5)
    False
    >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3)
    True
    """
    # Model generates the body. Harness runs hidden test cases.
    # pass@k = fraction of attempts where ALL tests pass.

SWE-bench task structure·json

{
  "repo": "django/django",
  "instance_id": "django__django-12345",
  "problem_statement": "... (real GitHub issue text)",
  "base_commit": "abc123...",
  "patch": "... (the maintainer's actual fix)",
  "test_patch": "... (tests that verify the fix)",
  "FAIL_TO_PASS": ["tests.test_foo.test_bar"],
  "PASS_TO_PASS": ["tests.test_foo.test_baz"]
}

# Model gets the repo at base_commit + the issue.
# Must produce a patch that flips FAIL_TO_PASS to passing
# without breaking any PASS_TO_PASS test.

Code Benchmarks: HumanEval, MBPP, SWE-bench

Three benchmarks that measure different code skills

HumanEval (Chen et al. 2021, OpenAI)

MBPP (Mostly Basic Python Problems, Google 2021)

SWE-bench (Princeton 2023, with extensions)

What none of them test

Code

External links

Exercise

Progress

Comments 0