Benchmark Locally

~18 min · ops, benchmark

Level 0Downloader

0 XP0/41 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

What to measure

Three numbers per (model, hardware) pair:

Tokens / second (decode). The headline number. Streaming responsiveness depends on it.
Time to first token (TTFT). Includes model load + prompt eval. Drops to ~0 when the model is warm.
Memory cost. What ollama ps reports under SIZE.

Document, don't trust memory

Six months later you won't remember whether a 32B model was viable on this Mac. Write down: hardware, model, quant, num_ctx, env vars, the three numbers. A simple Markdown table is fine.

Fair benchmarks

Run with the model already warm (one warmup prompt before timing).
Use a fixed seed so the response length is roughly stable.
Use a fixed prompt across runs.
Run three times and take the median.

Code

Benchmark harness·python

import httpx, json, time, statistics

OLLAMA = "http://localhost:11434"

def bench(model: str, prompt: str, runs: int = 3) -> dict:
    """Warm the model, then run N timed iterations."""
    # Warmup
    httpx.post(f"{OLLAMA}/api/chat", json={
        "model": model,
        "messages": [{"role": "user", "content": "ok"}],
        "stream": False, "options": {"num_predict": 1, "seed": 1},
    }, timeout=300.0)

    tps_runs, ttft_runs = [], []
    for i in range(runs):
        t0 = time.time()
        first_token_at = None
        with httpx.stream("POST", f"{OLLAMA}/api/chat", json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": True, "options": {"seed": 42, "num_predict": 200},
        }, timeout=None) as r:
            for line in r.iter_lines():
                if not line:
                    continue
                chunk = json.loads(line)
                if first_token_at is None and chunk.get("message", {}).get("content"):
                    first_token_at = time.time() - t0
                if chunk.get("done"):
                    ec = chunk.get("eval_count", 0)
                    ed = chunk.get("eval_duration", 1) or 1
                    tps_runs.append(ec / (ed / 1e9))
                    break
        ttft_runs.append(first_token_at or 0)

    return {
        "model": model,
        "runs": runs,
        "tps_median": statistics.median(tps_runs),
        "ttft_median_s": statistics.median(ttft_runs),
    }

print(bench("qwen2.5:7b", "Explain unified memory in 4 bullets."))

External links

Exercise

Run the benchmark on three of your installed models against one fixed prompt. Record the three numbers per model in a Markdown table. Identify the best tok/s on your hardware and decide if it's good enough for your real use cases.