Benchmark Locally

What to measure

Three numbers per (model, hardware) pair. Adding more makes the table heavier without making the decision faster.

Tokens / second (decode). The headline number. Streaming responsiveness depends on it. Comfortable reading sits somewhere around 10 tok/s, so once you fall below that the waiting becomes visible.
Time to first token (TTFT). Includes model load + prompt eval. Drops to ~0 when the model is warm and climbs into the tens of seconds when it's cold. This is why the same model feels fast one day and sluggish the next — it's usually TTFT that moved, not decode speed.
Memory cost. What ollama ps reports under SIZE. This is what ultimately decides what you can keep loaded side by side on one machine.

Document, don't trust memory

Six months later you won't remember whether a 32B model was viable on this Mac. Write down: hardware, model, quant, num_ctx, env vars, the three numbers. A simple Markdown table is fine. Once that table has a few rows, your next hardware decision gets made against your own measurements instead of somebody's spec sheet.

Fair benchmarks

Run with the model already warm (one warmup prompt before timing). Otherwise model load time bleeds into your decode number.
Use a fixed seed so the response length is roughly stable. When length wobbles, tok/s wobbles with it.
Use a fixed prompt across runs. Change the prompt and you change how much prompt eval happens, which makes TTFT incomparable.
Run three times and take the median. The callout below is why.

Code

Benchmark harness·python

import httpx, json, time, statistics

OLLAMA = "http://localhost:11434"

def bench(model: str, prompt: str, runs: int = 3) -> dict:
    """Warm the model, then run N timed iterations."""
    # Warmup
    httpx.post(f"{OLLAMA}/api/chat", json={
        "model": model,
        "messages": [{"role": "user", "content": "ok"}],
        "stream": False, "options": {"num_predict": 1, "seed": 1},
    }, timeout=300.0)

    tps_runs, ttft_runs = [], []
    for i in range(runs):
        t0 = time.time()
        first_token_at = None
        with httpx.stream("POST", f"{OLLAMA}/api/chat", json={
            "model": model,
            "messages": [{"role": "user", "content": prompt}],
            "stream": True, "options": {"seed": 42, "num_predict": 200},
        }, timeout=None) as r:
            for line in r.iter_lines():
                if not line:
                    continue
                chunk = json.loads(line)
                if first_token_at is None and chunk.get("message", {}).get("content"):
                    first_token_at = time.time() - t0
                if chunk.get("done"):
                    ec = chunk.get("eval_count", 0)
                    ed = chunk.get("eval_duration", 1) or 1
                    tps_runs.append(ec / (ed / 1e9))
                    break
        ttft_runs.append(first_token_at or 0)

    return {
        "model": model,
        "runs": runs,
        "tps_median": statistics.median(tps_runs),
        "ttft_median_s": statistics.median(ttft_runs),
    }

print(bench("qwen2.5:7b", "Explain unified memory in 4 bullets."))

What to measure

Document, don't trust memory

Fair benchmarks

Code

External links

Exercise

Progress

Comments 0