The Memory Paradox — Total in VRAM, Active in FLOPs

The number that surprises people

DeepSeek-V3 has 671B total parameters and ~37B active per token. That ratio (~5.5%) means the per-token compute is roughly 5.5% of a 671B dense model. But the memory required is 100% of a 671B dense model. All 671B parameters must be loaded — they all have different weights, and any one of them might fire on the next token.

Why memory cannot follow active

The router decision is per-token and depends on input. There is no way to know in advance which experts will fire, so all experts must be ready. This is fundamentally different from compute, which only happens for the chosen experts. Memory is "what is reachable", compute is "what is reached". MoE shrinks the second without shrinking the first.

The serving-cost shape

For an MoE model:

VRAM cost ∝ total parameters. You need a fleet sized for the total.
FLOP cost per token ∝ active parameters. You can serve more tokens per second per FLOP than a same-total dense model.
Throughput depends on batch composition. If many tokens in a batch all want the same experts, expert-level GPUs get overloaded; if tokens spread across experts, parallelism is good. This is why MoE batch behavior is harder to predict.

Why MoE is harder to run locally

A single 80GB H100 cannot hold DeepSeek-V3 even at FP8 (it would need ~700GB). Even a 4-bit quantized version is ~170GB. So MoE at frontier scale is fundamentally a multi-GPU or even multi-node story. Dense at the same compute equivalence (a hypothetical ~37B dense) would fit in a fraction of one H100.

The pricing implication

API providers price MoE models based on total cost of serving (memory + utilization), not just per-token FLOPs. So a 671B-A37B MoE model is rarely 5× cheaper than a 70B dense model on the API even though its FLOPs/token are similar to a 37B dense. The economics flow through total parameters, not active.

Code

Memory vs compute estimation for MoE·python

def memory_gb_bf16(total_B):           # weights only; +KV cache on top
    return total_B * 2

def memory_gb_fp8(total_B):
    return total_B * 1.0

def memory_gb_int4(total_B):
    return total_B * 0.5

# DeepSeek-V3 has 671B total, ~37B active.
print("BF16:", memory_gb_bf16(671), "GB")  # ~1342 GB
print("FP8:",  memory_gb_fp8(671),  "GB")  # ~671 GB
print("INT4:", memory_gb_int4(671), "GB")  # ~336 GB

# Per-token FLOPs (very rough; just FFN portion)
flops_per_token = 2 * 37 * 1e9             # ~74 GFLOP/token

Exercise

Compute the memory required to host DeepSeek-V3 at BF16, FP8, and INT4 quantization. Then look at consumer hardware budgets — single 80GB H100, 4×H100, 8×H100, 1×Mac Studio M2 Ultra (192GB unified). Which configurations can hold each quantization? This kind of napkin math is the everyday work of architectural literacy.