Decode and TTFT at 10, 50 and 90% Context

"A headline decode number describes a state the machine leaves within minutes of real use. Measure where you will actually be: near the end of the window, with the cache full."

The Only Benchmark That Matters

The household's doctrine on hardware timing has one line about benchmarks: the only one that matters is decode tokens per second and time to first token at ninety per cent of the context window. Everything a ladder measures happens at a 209-token prompt, which is where nobody works; a coding session or a long document lives at tens of thousands of tokens, where the KV cache is a second model's worth of bytes and the prefill is a wall the user waits behind. So the lab's second experiment fixes a 32,768-token window and runs three models at 10, 50 and 90% of it, 200 tokens out, on office. Three models chosen for their cache-to-weights ratio: a pure-attention 1B (32 KB of cache per token against 0.7 GB of weights), a hybrid 9B (32 KB against 4.5 GB), a hybrid 27B (64 KB against 14.4 GB).

Model, office	Context	TTFT	Prefill tok/s	Decode tok/s	ms per token	Peak GB	Evidence
Llama-3.2-1B, 4-bit	3,254 (10%)	0.56 s	6,572	343.6	2.91	1.63	measured 2026-09-15
	16,302 (50%)	2.54 s	6,562	237.4	4.21	1.97	measured
	29,406 (90%)	5.82 s	5,105	208.5	4.80	2.38	measured
Qwen3.5-9B, 4-bit	3,233	2.69 s	1,277	94.4	10.60	6.77	measured
	16,337	13.12 s	1,259	86.3	11.59	8.13	measured
	29,385	24.90 s	1,187	78.1	12.80	9.39	measured
Qwen3.5-27B, 4-bit	3,233	8.96 s	367	32.3	31.00	18.21	measured
	16,337	45.94 s	357	29.3	34.09	20.46	measured
	29,385	90.38 s	326	27.5	36.35	22.57	measured

Two Walls, Read Separately

The prefill wall. TTFT is linear in prompt length at a nearly constant prefill rate — 9 seconds at 10% and 90 seconds at 90% for the 27B, on the fastest Mac in the house. Prefill barely slows with context (367 → 326 tokens per second), so the wall is simply the size of the prompt divided by the compute stage's rate; a 32K document through a 27B is a minute and a half before the first token, and nothing about bandwidth changes that. The decode slope. Milliseconds per token rise linearly with context, because each token now reads the cache as well as the weights: the 1B loses 39% of its decode rate across the window, the 9B 17%, the 27B 15%. The physics track predicted the ratios from cache-to-weights and this is where they were measured; the 1B's shortfall from its predicted 2.3× is the fixed cost diluting a three-millisecond token, which the bandwidth lesson's fit already priced at 1.67 ms.

The Number the Slope Hides

Divide the cache bytes added per context token by the milliseconds added per context token and you get the bandwidth at which the cache was actually read: 444 GB/s for the 1B, 379 for the 9B, 313 for the 27B. The weights were streamed at 497 on this Mac. The cache is read less efficiently than the weights, and less efficiently the larger the model's cache per token — an attention kernel reading a long, strided cache is not a matrix-vector product reading a contiguous weight, and the lab reports the ratio without naming the kernel-level cause. For the card: the 90% numbers, not the 10% ones, are the ones to compare between machines; a Mac that wins at 209 tokens and loses at 29,000 is the one whose cache read is slower, and that is a measurement the ladder cannot make.

Code

lab_curve.py — the curve as a table, and the cache-read bandwidth from its slope·python

#!/usr/bin/env python3
"""Decode and TTFT at 10 / 50 / 90 % of a 32K window. Seconds per token is linear
in context: t(N) = t0 + (KV bytes per token × N) / BW_cache, so the slope of the
three points gives the bandwidth at which the cache was actually read -- compare it
with the weight-streaming bandwidth the ladder fit. Standard library only.
Usage: lab_curve.py curve-office.jsonl"""
import json, sys

KV_PER_TOKEN = {"Llama-3.2-1B-Instruct-4bit": 32e3, "Qwen3.5-9B-4bit": 32e3, "Qwen3.5-27B-4bit": 64e3}   # bytes, from the configs (T6)
WEIGHT_BW = 497e9                                                                                          # office ladder fit (previous lesson)

rows = [json.loads(l) for l in open(sys.argv[1])]
by = {}
for r in rows:
    if r.get("experiment") == "curve":
        by.setdefault(r["model"].split("/")[-1], []).append(r)
print(f"{'model':28} {'ctx tokens':>10} {'TTFT s':>8} {'prefill':>8} {'decode':>8} {'ms/token':>9} {'peak GB':>8}")
for name, rs in by.items():
    rs.sort(key=lambda r: r["prompt_tokens"])
    for r in rs:
        print(f"{name:28} {r['prompt_tokens']:10d} {r['ttft_s']:8.2f} {r['prompt_tps']:8.0f} {r['generation_tps']:8.1f} {1e3/r['generation_tps']:9.2f} {r['peak_memory_gb']:8.2f}")
    n0, n2 = rs[0]["prompt_tokens"], rs[-1]["prompt_tokens"]
    t0, t2 = 1 / rs[0]["generation_tps"], 1 / rs[-1]["generation_tps"]
    slope = (t2 - t0) / (n2 - n0)                                     # seconds per token per context token
    bw_cache = KV_PER_TOKEN[name] / slope
    w = rs[0]["weight_bytes_per_token"]
    naive = rs[0]["generation_tps"] * (w + KV_PER_TOKEN[name] * n0) / (w + KV_PER_TOKEN[name] * n2)   # every byte at one bandwidth
    print(f"{'':28} decode 10% -> 90%: {100*(rs[-1]['generation_tps']-rs[0]['generation_tps'])/rs[0]['generation_tps']:+.0f}%   "
          f"TTFT at 90%: {rs[-1]['ttft_s']:.1f} s   cache read at {bw_cache/1e9:.0f} GB/s (weights: {WEIGHT_BW/1e9:.0f})   "
          f"naive one-bandwidth prediction at 90%: {naive:.1f} tok/s\n")

# office, M3 Ultra, 2026-09-15, window 32,768, 200 generated:
# Llama-3.2-1B-Instruct-4bit   3254 / 16302 / 29406 ctx: TTFT 0.56 / 2.54 / 5.82 s; decode 343.6 / 237.4 / 208.5 (-39%); cache read at 444 GB/s
# Qwen3.5-9B-4bit              3233 / 16337 / 29385:      TTFT 2.69 / 13.12 / 24.90;  decode 94.4 / 86.3 / 78.1 (-17%);   cache read at 379 GB/s
# Qwen3.5-27B-4bit             3233 / 16337 / 29385:      TTFT 8.96 / 45.94 / 90.38;  decode 32.3 / 29.3 / 27.5 (-15%);   cache read at 313 GB/s

Exercise

Run the curve on your Mac for one model at 10, 50 and 90% of a window your memory holds (check peak GB before choosing the window). Write TTFT and decode at 90% on your card as the row that matters, then derive the cache-read bandwidth from the slope and compare it with your fitted weight bandwidth from lesson 2.

Hint

If decode at 90% is more than half of decode at 10% for a hybrid model, your window is small relative to the weights — that is fine, just say the window. If TTFT at 90% is not roughly nine times TTFT at 10%, the prompt fill did not tokenize to the count you expected.