Code Generation Evaluation

Code is the easiest output to evaluate — and the hardest

Easy because you can run it; hard because "the code runs" is necessary but not sufficient. A passing test does not mean the code is good. Three quality layers matter.

Layer 1: execution correctness

Runs the generated code; checks against unit tests. pass@k is the fraction of attempts where ALL tests pass on at least one of k samples. This is what HumanEval and SWE-bench measure.

Layer 2: code quality

Static analysis — linters, type checkers, security scanners (bandit, semgrep).
Style — ruff, prettier, black. Doesn't catch bugs but signals "would a human accept this PR?"
Complexity — cyclomatic, nesting depth.
Documentation — docstrings, comments, type annotations.

Layer 3: behavioral correctness beyond the test

The generated code passes its tests but does it match the intent? An LLM judge can compare implementation against the natural-language spec. This is where most "looks correct, isn't" bugs hide.

Principle: Always run generated code in a sandbox. Always run linters. Always have a judge compare implementation against intent. Skip any layer and you ship the bug it would have caught.

Code-specific safety checks

Generated code can have command injection, SQL injection, hardcoded secrets, unsafe deserialization, or insecure dependencies. A security scanner (semgrep, bandit, npm audit) on the generated code is part of the eval, not an afterthought.

Code

pass@k computation·python

import math
import itertools

def pass_at_k(n, c, k):
    """Standard HumanEval-style pass@k. n=samples, c=correct, k=evaluation k."""
    if n - c < k:
        return 1.0
    return 1.0 - math.comb(n - c, k) / math.comb(n, k)

# Generate n samples for each problem; run tests; count correct.
# Then compute pass@1, pass@10, pass@100 from the same data.
results = []
for problem in benchmark:
    samples = [model.generate(problem) for _ in range(20)]
    n_correct = sum(run_tests(s, problem.tests) for s in samples)
    results.append({
        "problem": problem.id,
        "pass@1": pass_at_k(20, n_correct, 1),
        "pass@10": pass_at_k(20, n_correct, 10),
    })

Security scan as part of code eval·python

import subprocess, tempfile, os

def security_scan(code: str):
    with tempfile.NamedTemporaryFile(suffix=".py", delete=False) as f:
        f.write(code.encode())
        path = f.name
    try:
        # bandit for Python security antipatterns
        result = subprocess.run(
            ["bandit", "-q", "-f", "json", path],
            capture_output=True, text=True,
        )
        # also: semgrep, ruff, mypy --strict
    finally:
        os.unlink(path)
    return result.stdout

Sandboxed test execution·python

import subprocess, tempfile, os, signal

def run_in_sandbox(code: str, test_code: str, timeout=10):
    """Run with subprocess timeout. For real safety use Docker/firejail."""
    with tempfile.TemporaryDirectory() as d:
        with open(f"{d}/sol.py", "w") as f:
            f.write(code)
        with open(f"{d}/test.py", "w") as f:
            f.write(test_code)
        try:
            r = subprocess.run(
                ["python", f"{d}/test.py"],
                capture_output=True, text=True, timeout=timeout,
                cwd=d,
            )
            return r.returncode == 0, r.stdout + r.stderr
        except subprocess.TimeoutExpired:
            return False, "timeout"

Code Generation Evaluation

Code is the easiest output to evaluate — and the hardest

Layer 1: execution correctness

Layer 2: code quality

Layer 3: behavioral correctness beyond the test

Code-specific safety checks

Code

External links

Exercise

Progress

Comments 0