A Batch of One vs a Thousand Users

"The weights are read once per step. Whether that read serves one person or a thousand is the whole difference between a Mac and a data centre."

The Same Read, Shared

A decode step reads every active weight once. If one sequence is being generated, that read produces one token. If thirty-two sequences are being generated together — a batch — the same read produces thirty-two tokens, because the matrix-vector product becomes a matrix-matrix product against thirty-two vectors and the weights stream through the GPU once for all of them. Bandwidth-bound work that costs the same per step whether it serves one or many is the economic core of cloud inference: aggregate tokens per second rises almost linearly with batch size until the arithmetic, or the growing KV traffic of many sequences, becomes the limit.

The lab measured it on office with mlx-lm's batch_generate, the 9B model, 128 tokens per sequence:

Batch	Tokens generated	Wall time (incl. prefill)	Aggregate tok/s	Per sequence	Evidence
1	128	1.70 s	75	75	measured, office, 2026-09-15
2	256	1.60 s	160	80	measured
4	512	1.95 s	263	66	measured
8	1,024	3.08 s	332	42	measured
16	2,048	5.34 s	384	24	measured
32	4,096	7.76 s	528	16.5	measured

Seven times the aggregate throughput at batch 32, from the same weights and the same bus — and each user waits longer for their own tokens. The wall times include prefill for every prompt in the batch, so the per-sequence figures understate decode alone; the shape is what matters. This is the curve a serving provider lives on, and the reason a token from a cloud API can cost a hundredth of what the same token costs on your own hardware: their weights are read once for a thousand people, yours are read once for you.

Why a Mac Is a Batch of One

A household's inference is one session at a time. There is no second user whose tokens can share the read, and the two things that make cloud long-context economical — batching, and a prompt cache shared across many requests with the same prefix — are exactly the two things a single-user machine cannot exploit. The household's own serving hub is the honest test: its MLX server runs continuous batching (up to eight concurrent requests, per its settings) and a tiered prefix cache with an SSD cold tier. Its statistics on 2026-09-15: 143,141 requests, 328.8 million prompt tokens, zero completion tokens, zero cached tokens. Every request was an embedding or a reranking call — prefill only, batched thirty-two at a time — and the decode-side machinery has never had a second sequence to batch or a prefix to reuse, because the household's generation load is one person typing.

That is not a failing of the server; it is the shape of the workload. The founder's doctrine states it as a cost: local inference "has no prompt cache and no batching to hide a full-history replay per turn". The physics track's previous lessons said the same in numbers: at batch 1 you get the ceiling divided by overhead, and every token of context is prefilled by you, for you, once per turn unless your client keeps the cache. The edge-era track asks which workloads are worth that; this lesson only wants the asymmetry clear.

Where Batching Comes Back Locally

Two places. Prefill is already a batch — every token of the prompt shares the weight read, which is why prefill runs ten times faster per token than decode and why the MoE prefills faster than the dense 9B. And batch jobs are batches: tagging ten thousand images, embedding a corpus, summarizing a library overnight. Those are the workloads where a Mac's aggregate throughput climbs the table above, and they are the workloads the household actually runs on its fleet at scale. The batch-of-one cost applies to the chat window, not to the machine.

Code

batch_scaling.py — aggregate decode throughput vs batch size with mlx-lm·python

#!/usr/bin/env python3
"""Aggregate decode throughput vs batch size with mlx-lm's batch_generate.
Weights are read once per step for the whole batch, so aggregate tokens/s
rises with batch until compute or KV traffic takes over. Wall time includes
prefill of every prompt in the batch."""
import sys, time
from mlx_lm import load, batch_generate
from mlx_lm.sample_utils import make_sampler

repo = sys.argv[1] if len(sys.argv) > 1 else "mlx-community/Qwen3.5-9B-4bit"
model, tok = load(repo)
base = tok.apply_chat_template([{"role": "user", "content": "Write 300 words about unified memory."}],
                               add_generation_prompt=True)

for batch in (1, 2, 4, 8, 16, 32):
    prompts = [base] * batch
    t = time.perf_counter()
    resp = batch_generate(model, tok, prompts, max_tokens=128, sampler=make_sampler(temp=0.0), verbose=False)
    dt = time.perf_counter() - t
    total = sum(len(tok.encode(text)) for text in resp.texts)
    print(f"batch {batch:3d}: {total:5d} tokens in {dt:6.2f}s -> {total/dt:7.1f} tok/s aggregate, {total/dt/batch:6.1f} per sequence")

# office, M3 Ultra, Qwen3.5-9B-4bit, mlx-lm 0.31.3, 2026-09-15:
# batch   1:  128 tokens in 1.70s ->  75.2 tok/s aggregate,  75.2 per sequence
# batch   8: 1024 tokens in 3.08s -> 332.4 tok/s aggregate,  41.6 per sequence
# batch  32: 4096 tokens in 7.76s -> 527.7 tok/s aggregate,  16.5 per sequence

Exercise

Run batch_scaling.py on your Mac (drop the 32 row if memory is tight) and add the batch-1 and largest-batch aggregate figures to your card. Then list every AI job your machine actually runs and sort them into batch-of-one and batch jobs. For the batch jobs, estimate how much of the table's gain you are currently leaving on the floor by running them one item at a time.

Hint

Anything with a queue — tagging, embedding, summarizing, transcribing a folder — is a batch job and should be dispatched as one. The chat window is the only batch-of-one workload that has to be, and it is the one everyone benchmarks.