Depth vs Width: Where to Spend Parameters

Given a fixed parameter budget, should you stack more layers (deeper) or widen each layer (larger d_model)? Empirically, both matter, but the right ratio is non-obvious.

Model	Layers	d_model	Params	Layers / d_model
Transformer Base	6	512	65M	1/85
GPT-2	12	768	117M	1/64
GPT-3	96	12,288	175B	1/128
Llama 3 (8B)	32	4,096	8B	1/128
Llama 3.3 (70B)	80	8,192	70B	1/102
DeepSeek-V3	61	—	671B (37B active, MoE)	—

The pattern: as models scale, depth and width grow together. The d_head per layer stays around 64-128, and roughly d_model ≈ 64-128 × n_layers across most modern architectures. This isn't a theorem but a strong empirical regularity.

Why both matter: deeper models can express more compositional functions (each layer composes its predecessors). Wider models have richer per-token representations. Practical evidence suggests depth is somewhat more important for language understanding, but with diminishing returns past ~80 layers — at which point widening or sparsifying (MoE) is more productive than going deeper still.

Code

Param budget split·python

def transformer_total(vocab, d, n_layers, d_ff_mul=4):
    embed = vocab * d
    attn  = 4 * d * d * n_layers
    ffn   = 2 * d * (d_ff_mul * d) * n_layers
    return embed + attn + ffn

# Same total budget, different shapes:
budget = 8_000_000_000
# Deep+narrow: 64 layers, d_model=2048
deep   = transformer_total(128_000, 2048, 64)
# Shallow+wide: 16 layers, d_model=4096
wide   = transformer_total(128_000, 4096, 16)
print(deep/1e9, wide/1e9)
# Both ~8B, but train very differently.

Depth vs Width: Where to Spend Parameters

Code

External links

Exercise

Progress

Comments 0