Head Count Across Models: From 8 to 96

~8 min · heads, model-survey

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Different models pick different head counts based on their d_model and design philosophy. The pattern: d_head has stabilized at 64 or 128, while d_model and head count grow together.

Model	d_model	Q heads	KV heads	d_head
Transformer Base (2017)	512	8	8	64
BERT-base	768	12	12	64
GPT-2	768	12	12	64
GPT-3	12,288	96	96	128
Llama 3 (8B)	4,096	32	8 (GQA)	128
Llama 3.3 (70B)	8,192	64	8 (GQA)	128
Mixtral 8×22B	6,144	48	8 (GQA)	128
Qwen 2.5-7B	3,584	28	4 (GQA)	128

The trend is unmistakable: modern models have d_head=128 and use GQA to keep KV heads small. The decoupling of Q heads (representational capacity) from KV heads (cache memory) is one of the most consequential design moves of the past three years.

Code

Read it from a model config·python

from transformers import AutoConfig

for name in ["meta-llama/Meta-Llama-3-8B",
             "meta-llama/Meta-Llama-3.1-70B",
             "mistralai/Mixtral-8x22B-v0.1"]:
    cfg = AutoConfig.from_pretrained(name)
    print(f"{name}")
    print(f"  d_model       = {cfg.hidden_size}")
    print(f"  Q heads       = {cfg.num_attention_heads}")
    print(f"  KV heads      = {getattr(cfg, 'num_key_value_heads', cfg.num_attention_heads)}")
    print(f"  d_head        = {cfg.hidden_size // cfg.num_attention_heads}")
    print()

External links

Exercise

For five open-weight models you might serve in production, pull the config and tabulate (d_model, n_heads, n_kv_heads, d_head). Which models would have the smallest KV cache per token at 128K context? Which the largest? (Hint: KV-cache size is proportional to n_kv_heads × d_head × n_layers.)

Progress

Progress is local-only — sign in to sync across devices.

← PreviousMulti-Head Attention: Parallel Subspaces Next →Self-Attention vs Cross-Attention

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.