The Four Numbers: d_model, d_ff, n_heads, n_layers

~8 min · hyperparameters, synthesis

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The shape of every modern Transformer is a four-tuple. Read these from any model's config and you know the architecture.

Symbol	Meaning	Typical 2026 values
d_model	Hidden dimension. Width of every internal representation.	768 – 12,288
d_ff	FFN intermediate dimension. With SwiGLU, ≈ 8/3 × d_model.	2,048 – 28,672
n_heads (Q)	Number of query heads.	12 – 96
n_kv_heads	Number of KV heads (GQA). Often n_heads / 4 or /8.	4 – 96
n_layers	Number of stacked Transformer blocks.	12 – 126
d_head	= d_model / n_heads. Modern default 128.	64 – 128

The original Transformer Base used d_model=512, d_ff=2048, n_heads=8, n_layers=6, d_head=64. Llama 3.3 70B uses d_model=8192, d_ff~28000, n_q=64, n_kv=8 (GQA), n_layers=80, d_head=128. Same architecture, ~1000× scale, different rectangles.

Code

Read the rectangle from any HF model·python

from transformers import AutoConfig

def rectangle(name):
    cfg = AutoConfig.from_pretrained(name)
    return {
        'd_model': cfg.hidden_size,
        'd_ff': cfg.intermediate_size,
        'n_q_heads': cfg.num_attention_heads,
        'n_kv_heads': getattr(cfg, 'num_key_value_heads', cfg.num_attention_heads),
        'n_layers': cfg.num_hidden_layers,
        'd_head': cfg.hidden_size // cfg.num_attention_heads,
    }

for m in ["meta-llama/Meta-Llama-3-8B",
          "mistralai/Mistral-7B-v0.3",
          "Qwen/Qwen2.5-7B"]:
    print(m, rectangle(m))

External links

Hugging Face model configs (search any model + /raw/main/config.json)

Exercise

Build a small CLI tool that takes a Hugging Face model ID and prints its rectangle plus an estimated parameter count. Run it on five models you'd consider serving. Sort by per-token KV-cache size at 32K context. Which is most memory-efficient to serve? Which most representationally rich?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousParameter Count Calculator: Reading Any Model Card Next →Dropout and Regularization (and Why It's Out of Fashion)

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.