Memory Math: Bytes per Parameter

~10 min · memory, deployment

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Memory is the binding constraint for almost every LLM deployment decision. The calculation is simple, but you have to track all the pieces.

Total memory ≈ params × bytes_per_param + KV_cache + activation buffer + framework overhead

Format	Bytes / param	7B model weights	70B model weights
FP32	4	28 GB	280 GB
FP16 / BF16	2	14 GB	140 GB
INT8 / Q8	1	7 GB	70 GB
INT4 / Q4	0.5	3.5 GB	35 GB

This is just the weights. Add:

KV cache — proportional to context length × n_layers × n_kv_heads × d_head × precision. For Llama 3.3 70B at 128K context FP16: ~32 GB.
Activations during inference — typically a few GB, much larger during training.
Framework overhead — CUDA driver, allocator, optimizer state. Often 5-15% of total.

An H100 80 GB can hold a 70B INT4 model (35 GB) with ~32 GB of KV cache room and overhead. FP16 needs at least 2 H100s. This single arithmetic shapes nearly every "should we use model X?" conversation in deployment.

Code

Honest memory budget for an inference deployment·python

def deployment_memory_gb(
    params_b, dtype_bytes, n_layers, n_kv_heads, d_head,
    max_context, batch_size=1, overhead_pct=0.10,
):
    weights = params_b * 1e9 * dtype_bytes / 1e9
    kv_cache = (
        2 * batch_size * max_context * n_layers
        * n_kv_heads * d_head * dtype_bytes / 1e9
    )
    activations = 2.0   # rough; depends on framework
    base = weights + kv_cache + activations
    return base * (1 + overhead_pct)

# Llama 3.3 70B, INT4 weights, FP16 KV cache, 128K context, batch=1
mem = deployment_memory_gb(70, 0.5, 80, 8, 128, 128_000, batch_size=1)
print(f"~{mem:.0f} GB")    # ≈ 70 GB total — fits on a single H100

External links

Hugging Face — Model Memory Calculator (interactive)

Exercise

For a model you'd consider serving (pick one), compute total deployment memory at: (a) batch=1, context=8K, FP16; (b) batch=1, context=128K, FP16; (c) batch=8, context=8K, INT4; (d) batch=8, context=128K, INT4. Which configurations actually fit on the GPUs you have access to?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousParameter Count Intuition: 1B to 1T Next →Quantization: Q4, Q5, Q8 in Practice

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.