Memory is the real constraint
The hard ceiling on which model you can run is memory, not compute. Weights have to fit. Anything that doesn't fit either spills to slow paths or refuses to load. Memory bandwidth mostly determines tokens-per-second, as the decode section below works out; memory capacity decides whether you get any tokens at all.
Two memory architectures
- Discrete GPU. NVIDIA RTX 4090 (24 GB), RTX 5090 (32 GB), data-center H100 / H200 / B200 (80–192 GB). Weights live in VRAM, separate from system RAM. Fast bandwidth (1+ TB/s). Expensive per gigabyte.
- Apple Silicon unified memory. CPU and GPU share the same pool. Whatever your Mac has — 16 GB, 64 GB, 128 GB, 192 GB, 512 GB — is available for weights. Bandwidth is lower than data-center GPUs, and it varies a lot across the lineup: about 100 GB/s on a base M2 or M3, 120 on M4, 153 on M5, 273 on M4 Pro, and 819 GB/s at the top on M3 Ultra (Apple announced 1.2 TB/s for M5 Ultra in August 2026). Capacity is huge for the price; speed depends on which chip you bought.
Sizing rule of thumb
At Q4 quantization (the daily-driver default), the weights take about 0.6 GB per billion parameters. On paper 4 bits is half a byte, so 0.5; in practice Q4_K_M keeps some tensors at higher precision and averages about 4.9 bits per weight. llama.cpp's own table for Llama 3.1 puts 8B at 4.9 GB, 70B at 43 GB, and 405B at 249 GB. Add 10–20% for context, KV cache, and runtime overhead. If you're tight, drop to a smaller quant; if you have headroom, take a higher quant.
Bandwidth determines speed
Decode is bandwidth-bound. To produce each new token, the chip reads every weight that token passes through, plus the KV cache for everything generated so far. For a single stream that decodes one token per pass, that gives you a ceiling you can work out on a napkin: tokens per second ≲ memory bandwidth ÷ bytes read per token. A dense 70B at Q4_K_M reads its whole 43 GB for every token, so an M3 Ultra at 819 GB/s tops out near 19 tok/s before the KV cache takes its share. A 64 GB M4 Pro can still load the same file, but at 273 GB/s its ceiling sits near 6 tok/s. Capacity decides whether a model loads; bandwidth decides how fast it talks. A mixture-of-experts model still has to fit whole, but each token reads only the experts it is routed to, so a 30B model with 3B active reads about a 3B model's worth of bytes per token and decodes far faster than a dense 30B.
A ceiling is not a benchmark. Plain decoding always measures below it, because compute, kernels, and software overhead each take a cut. llama.cpp's Apple Silicon benchmark runs Llama 2 7B at Q4_0 (a 3.56 GiB file, ceiling around 210 tok/s) at 92 tok/s on an M3 Ultra — under half the ceiling. The known ways past the line don't beat the physics; they change how many tokens one read of the weights buys. Speculative decoding has a small draft model propose several tokens and the big model check them all in one pass, so a single read can confirm more than one token. Batching spreads one read across many users' streams, which raises total throughput while each stream still sits under its own ceiling.