Code is the easiest output to evaluate — and the hardest
Easy because you can run it; hard because "the code runs" is necessary but not sufficient. A passing test does not mean the code is good. Three quality layers matter.
Layer 1: execution correctness
Runs the generated code; checks against unit tests. pass@k is the fraction of attempts where ALL tests pass on at least one of k samples. This is what HumanEval and SWE-bench measure.
Layer 2: code quality
Static analysis — linters, type checkers, security scanners (bandit, semgrep).
Style — ruff, prettier, black. Doesn't catch bugs but signals "would a human accept this PR?"
Complexity — cyclomatic, nesting depth.
Documentation — docstrings, comments, type annotations.
Layer 3: behavioral correctness beyond the test
The generated code passes its tests but does it match the intent? An LLM judge can compare implementation against the natural-language spec. This is where most "looks correct, isn't" bugs hide.
Principle: Always run generated code in a sandbox. Always run linters. Always have a judge compare implementation against intent. Skip any layer and you ship the bug it would have caught.
Code-specific safety checks
Generated code can have command injection, SQL injection, hardcoded secrets, unsafe deserialization, or insecure dependencies. A security scanner (semgrep, bandit, npm audit) on the generated code is part of the eval, not an afterthought.
Code
pass@k computation·python
import math
import itertools
def pass_at_k(n, c, k):
"""Standard HumanEval-style pass@k. n=samples, c=correct, k=evaluation k."""
if n - c < k:
return 1.0
return 1.0 - math.comb(n - c, k) / math.comb(n, k)
# Generate n samples for each problem; run tests; count correct.
# Then compute pass@1, pass@10, pass@100 from the same data.
results = []
for problem in benchmark:
samples = [model.generate(problem) for _ in range(20)]
n_correct = sum(run_tests(s, problem.tests) for s in samples)
results.append({
"problem": problem.id,
"pass@1": pass_at_k(20, n_correct, 1),
"pass@10": pass_at_k(20, n_correct, 10),
})
Security scan as part of code eval·python
import subprocess, tempfile, os
def security_scan(code: str):
with tempfile.NamedTemporaryFile(suffix=".py", delete=False) as f:
f.write(code.encode())
path = f.name
try:
# bandit for Python security antipatterns
result = subprocess.run(
["bandit", "-q", "-f", "json", path],
capture_output=True, text=True,
)
# also: semgrep, ruff, mypy --strict
finally:
os.unlink(path)
return result.stdout
Sandboxed test execution·python
import subprocess, tempfile, os, signal
def run_in_sandbox(code: str, test_code: str, timeout=10):
"""Run with subprocess timeout. For real safety use Docker/firejail."""
with tempfile.TemporaryDirectory() as d:
with open(f"{d}/sol.py", "w") as f:
f.write(code)
with open(f"{d}/test.py", "w") as f:
f.write(test_code)
try:
r = subprocess.run(
["python", f"{d}/test.py"],
capture_output=True, text=True, timeout=timeout,
cwd=d,
)
return r.returncode == 0, r.stdout + r.stderr
except subprocess.TimeoutExpired:
return False, "timeout"
Build a 20-task code-eval suite for one small domain. Score each generated solution on three layers: execution correctness, lint/security pass, and intent match (LLM judge). Report a triple-score per task.
Progress
Progress is local-only — sign in to sync across devices.