Prefill and Decode Are Two Different Workloads

"A language model does two things with the same weights, and the hardware experiences them as two different programs."

What Happens When You Press Enter

Every response from a language model has two phases. Prefill reads your whole prompt at once: every token in it goes through every layer together, as one big matrix of activations multiplied against the weight matrices. The work is a matrix-matrix product; the output is the first generated token and a KV cache holding what the model computed about the prompt. Decode then produces the rest one token at a time: a single token's activations — a vector — go through every layer, multiplied against the same weight matrices, and the KV cache grows by one entry. The work is a matrix-vector product, repeated once per token.

The weights are read from memory in both phases. The difference is how much arithmetic each read pays for. In prefill a weight matrix is read once and used against hundreds or thousands of tokens; the GPU has plenty to compute per byte fetched, and the phase is limited by how fast it can multiply — compute-bound. In decode the same matrix is read once and used against one token; there is almost nothing to compute per byte, and the phase is limited by how fast the bytes arrive — bandwidth-bound. NVIDIA's own inference guide describes it in the same terms: prefill is "a matrix-matrix operation that's highly parallelized", while in decode "the speed at which the data (weights, keys, values, activations) is transferred to the GPU from memory dominates the latency, not how fast the computation actually happens."

Measured, on the Ladder

The lab track recorded both rates for every model on every lab Mac. Prefill divided by decode is the ratio of how many tokens each phase processes per weight read:

Model (4-bit)	office prefill / decode tok/s	ratio	pro2023 prefill / decode	ratio	air prefill / decode	ratio	Evidence
Qwen3.5-0.8B	6,312 / 338	19	4,017 / 417	10	1,707 / 168	10	measured 2026-09-15
Qwen3.5-9B	1,043 / 95	11	642 / 73	9	172 / 19	9	measured
Qwen3.5-27B	315 / 33	10	195 / 23	8	49 / 6	8	measured

Two readings. First, prefill is roughly ten times faster than decode per token on every machine, for a 209-token prompt — a small prompt; with a thousand-token prompt the ratio grows, because prefill amortizes better and decode cannot amortize at all. Second, look at what changes between machines. From the M3 Max to the M3 Ultra, prefill on the 27B rises 62% (195 → 315) — twice the GPU cores — while decode rises 41% (23 → 33), tracking the achieved bandwidth ratio (391 → 638 GB/s, 63%) less overhead. The GPU generation lesson in the lab track is the same split seen the other way: the M3 Ultra out-prefills the M2 Ultra by 35% and decodes slower than it — 32.6 against 35.3.

Why This Is the Whole Quest in One Lesson

Every vendor claim in track one was a prefill claim or a peak-compute claim; every household experience of a model "feeling slow" is a decode experience or a time-to-first-token experience, and the two have different remedies. More GPU cores, Neural Accelerators, a better matrix unit — those move prefill. Only bandwidth and fewer bytes per token move decode. When Apple says the M5 is "up to 4x" faster at prompt processing and "19-27%" faster at generation "thanks to its greater memory bandwidth", it is describing exactly this split in its own numbers. The next lesson turns the decode half into a ceiling you can compute before you download anything.

Code

two_workloads.py — prefill and decode rates from the lab's JSONL, and the ratio between them·python

#!/usr/bin/env python3
"""Read silicon_lab.py ladder records and print prefill vs decode per model per
machine. The ratio is how many more tokens prefill handles per weight read."""
import glob
import json
from collections import defaultdict

rows = defaultdict(dict)
for path in glob.glob("results/ladder-*.jsonl"):
    for line in open(path):
        r = json.loads(line)
        rows[r["model"].split("/")[-1]][r["alias"]] = (r["prompt_tps"], r["generation_tps"])

for model, per_alias in sorted(rows.items()):
    print(model)
    for alias, (pre, dec) in sorted(per_alias.items()):
        print(f"   {alias:8} prefill {pre:8.1f} tok/s   decode {dec:7.2f} tok/s   ratio {pre/dec:5.1f}x")

# office, 2026-09-15 (prompt 209 tokens):
#   Qwen3.5-9B-4bit    prefill 1042.6   decode 95.06   ratio 11.0x
#   Qwen3.5-27B-4bit   prefill  315.4   decode 32.60   ratio  9.7x

Watch the two phases yourself with mlx-lm's streaming API·python

import time
from mlx_lm import load, stream_generate

model, tok = load("mlx-community/Qwen3.5-9B-4bit")
prompt = tok.apply_chat_template([{"role": "user", "content": "Explain unified memory in 300 words."}],
                                 add_generation_prompt=True)
t0 = time.perf_counter(); first = None
for r in stream_generate(model, tok, prompt, max_tokens=200):
    if first is None:
        first = time.perf_counter() - t0             # time to first token = prefill
print(f"prefill: {r.prompt_tokens} tokens at {r.prompt_tps:.0f} tok/s (TTFT {first:.2f}s)")
print(f"decode:  {r.generation_tokens} tokens at {r.generation_tps:.1f} tok/s")
# GenerationResponse carries prompt_tps and generation_tps separately — the
# library measures the two phases as two phases, because they are.

Exercise

Run the streaming block on your Mac with a 30-word prompt and then with a 600-word prompt (paste in any article). Record TTFT, prefill tok/s and decode tok/s for both and add them to your card. Then explain why decode tok/s barely changed while TTFT rose, and what that predicts for a workload that pastes a whole file into every request.

Hint

Prefill cost scales with prompt length; decode cost per token barely does (until the cache grows large — lesson four). A workload that re-sends a long context every turn pays the prefill bill every turn, which is why local coding agents feel slow on long files and why the edge-era track calls coding the wrong edge workload.