Hardware Reality

Memory is the real constraint

The hard ceiling on which model you can run is memory, not compute. Weights have to fit. Anything that doesn't fit either spills to slow paths or refuses to load. Compute determines tokens-per-second, but memory decides whether you get any tokens at all.

Two memory architectures

Discrete GPU. NVIDIA RTX 4090 (24 GB), RTX 5090 (32 GB), data-center H100 / H200 / B200 (80–192 GB). Weights live in VRAM, separate from system RAM. Fast bandwidth (1+ TB/s). Expensive per gigabyte.
Apple Silicon unified memory. CPU and GPU share the same pool. Whatever your Mac has — 16 GB, 64 GB, 128 GB, 192 GB, 512 GB — is available for weights. Bandwidth is lower than data-center GPUs (273–819 GB/s) but capacity is huge for the price.

Sizing rule of thumb

At Q4 quantization (the daily-driver default), ~1 GB of memory per billion parameters. A 7B model wants ~4 GB; a 70B model wants ~40 GB; a 405B model wants ~200 GB. Add 10–20% for context, KV cache, and runtime overhead. If you're tight, drop to a smaller quant; if you have headroom, take a higher quant.

Bandwidth determines speed

Decode is bandwidth-bound for autoregressive generation: each new token re-streams the weights. Memory bandwidth in GB/s sets a hard ceiling on tokens/second for a given model size. M3 Ultra at 819 GB/s decodes a 70B INT4 around 95 tok/s; the same model on a slower-bandwidth machine drops to 30–60 tok/s even when capacity is fine.

Code

Memory-fit calculation·python

def fits(params_billions: float, bits: int, total_ram_gb: float, ctx_overhead_gb: float = 4) -> bool:
    weight_gb = params_billions * bits / 8
    return weight_gb + ctx_overhead_gb < total_ram_gb * 0.85  # leave 15% headroom

# Examples on a 64 GB Mac
print(fits(7,  4, 64))    # True  — 7B Q4 trivially fits
print(fits(70, 4, 64))    # True  — tight but fits
print(fits(70, 8, 64))    # False — 70B Q8 needs 70 GB
print(fits(405, 4, 192))  # False — 405B Q4 needs ~200 GB

Memory is the real constraint

Two memory architectures

Sizing rule of thumb

Bandwidth determines speed

Code

External links

Exercise

Progress

Comments 0