Why 90% Context Is the Honest Benchmark

"The only benchmark that matters is decode tokens per second and time to first token at 90% context — and nobody publishes it." — the founder, 2026-08-26

The Empty-Context Number

Almost every tokens-per-second figure you will ever see — vendor, reviewer, forum — is measured on a short prompt with an empty cache. The lab track's ladder is that kind of number too, deliberately, because it isolates the weights. But nobody uses a model that way. A real session carries a long system prompt, a document, a repository, a conversation; the cache is large before the first useful token, and the two numbers that decide whether the machine is usable are how long you wait for the first token (prefill of everything so far) and how fast the answer streams once it starts (decode with a full cache). The founder's rule is that a benchmark that does not report both at a nearly full window is not measuring the workload. This lesson measures them.

The Curve, Measured

On office, with a 32,768-token window, prompts at 10, 50 and 90% of it, 200 tokens generated, three models chosen to span the cache-to-weights ratio:

Model	Context	Prompt tokens	TTFT	Prefill tok/s	Decode tok/s	Peak GB	Evidence
Llama-3.2-1B (pure GQA, 32 KB/token)	10%	3,254	0.56 s	6,572	343.6	1.63	measured, office, 2026-09-15
	50%	16,302	2.54 s	6,562	237.4	1.97	measured
	90%	29,406	5.82 s	5,105	208.5	2.38	measured
Qwen3.5-9B (hybrid, 32 KB/token)	10%	3,233	2.69 s	1,277	94.4	6.77	measured
	50%	16,337	13.12 s	1,259	86.3	8.13	measured
	90%	29,385	24.90 s	1,187	78.1	9.39	measured
Qwen3.5-27B (hybrid, 64 KB/token)	10%	3,233	8.96 s	367	32.3	18.21	measured
	50%	16,337	45.94 s	357	29.3	20.46	measured
	90%	29,385	90.38 s	326	27.5	22.57	measured

Reading the Two Curves

Decode falls from 10% to 90% by 39% for the 1B, 17% for the 9B and 15% for the 27B. The previous lesson predicted the slope from the cache-to-weights ratio — (weights + cache) ÷ weights — as 2.3x more bytes for the 1B, 1.21x for the 9B, 1.13x for the 27B. The 9B and 27B land on the prediction (1.21x and 1.17x measured); the 1B falls short of its 2.3x because a millisecond and a half of fixed overhead per token dilutes a 3-millisecond token. So the founder's sentence from the direction-is-not-timing doctrine — bandwidth divides the slope, it does not change the curve's shape — is exact, with one refinement this quest adds: the shape is the cache-to-weights ratio, and a hybrid architecture has a much flatter one than a pure transformer of the same cache per token.

Time to first token is the number that actually changes a session. Ninety seconds on the 27B at 90% of a 32K window; twenty-five on the 9B; and these are one-shot prefills — a chat client that re-sends the whole history every turn pays a bill of that order every turn, unless the runtime keeps and extends the cache between turns. The physics is unforgiving here: prefill is compute-bound, so the fix is more matrix throughput, which is exactly what Apple's M5 Neural Accelerators are for and exactly what this quest has not measured.

Code

context_curve.py — the lab's curve experiment, condensed·python

#!/usr/bin/env python3
"""Decode and TTFT at fractions of a context window. Build a prompt of the
target token count, generate, and record what mlx-lm reports. This is the
`curve` experiment of silicon_lab.py, trimmed to its essentials."""
import sys, time
import mlx.core as mx
from mlx_lm import load, stream_generate
from mlx_lm.sample_utils import make_sampler

repo = sys.argv[1] if len(sys.argv) > 1 else "mlx-community/Qwen3.5-9B-4bit"
window = int(sys.argv[2]) if len(sys.argv) > 2 else 32768
FILLER = "Decode re-reads the weights for every generated token, which is why its ceiling is bandwidth divided by bytes per token. "
QUESTION = "Explain the passage in detail, in at least 400 words."

model, tok = load(repo)


def prompt_of(target: int) -> list[int]:
    reps = max(1, target // 24)                          # ~24 tokens per filler sentence
    ids = tok.apply_chat_template([{"role": "user", "content": FILLER * reps + QUESTION}], add_generation_prompt=True)
    while len(ids) > target and reps > 1:
        reps -= 1
        ids = tok.apply_chat_template([{"role": "user", "content": FILLER * reps + QUESTION}], add_generation_prompt=True)
    return ids


for frac in (0.1, 0.5, 0.9):
    ids = prompt_of(int(window * frac))
    mx.reset_peak_memory(); t0 = time.perf_counter(); first = None
    for r in stream_generate(model, tok, ids, max_tokens=200, sampler=make_sampler(temp=0.0)):
        if first is None:
            first = time.perf_counter() - t0
    print(f"{frac:.0%} of {window}: prompt {r.prompt_tokens:6d}  TTFT {first:6.2f}s  prefill {r.prompt_tps:7.1f} tok/s  "
          f"decode {r.generation_tps:6.2f} tok/s  peak {r.peak_memory:5.2f} GB")

# office, mlx 0.32.2, 2026-09-15, Qwen3.5-9B-4bit:
# 10%: prompt 3233  TTFT  2.69s  prefill 1277  decode 94.38  peak 6.77
# 50%: prompt 16337 TTFT 13.12s  prefill 1259  decode 86.31  peak 8.13
# 90%: prompt 29385 TTFT 24.90s  prefill 1187  decode 78.10  peak 9.39

Exercise

Run context_curve.py on your Mac with the model and window you actually use (pick a window your working set can hold — the previous track's peak table is the guide). Add the 90% row to your card as your honest benchmark. Then compare the decode slope you measured with the ratio you predicted in the previous lesson's exercise, and explain any gap in one sentence.

Hint

If your slope is shallower than the ratio predicts, fixed overhead is diluting it (small model) or your runtime is doing something clever with the cache. If it is steeper, your machine may be near its working set and the peak-memory column will show it — that is the swap cliff, not the cache.