Embedding Dimension (d_model): 768 to 12,288

~10 min · d-model, scaling, capacity

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

d_model is the most important hyperparameter you'll see in this quest. It's the dimensionality of every internal representation: the embedding rows, the attention outputs, the FFN outputs, the residual stream that flows through every layer. Bigger d_model means more capacity per token, at quadratic cost in attention parameters and linear cost in FFN parameters.

Model	d_model	Layers	Total params
BERT-base	768	12	110M
GPT-2	768	12	117M
GPT-2-XL	1,600	48	1.5B
Llama 3 (8B)	4,096	32	8B
Llama 3.3 (70B)	8,192	80	70B
GPT-3	12,288	96	175B

Why larger d_model helps: every dimension can encode a different feature of meaning — syntactic role, sentiment, formality, topic, named-entity type, and many more we don't have names for. Interpretability research (sparse autoencoders) suggests there are tens of thousands of distinct "features" packed into superposition in modern d_model=4096+ representations.

Code

Capacity scaling — what doubling d_model costs·python

def transformer_param_estimate(vocab, d, n_layers, d_ff_mul=4):
    embed = vocab * d                     # input embedding
    attn  = 4 * d * d * n_layers           # Q, K, V, O per layer
    ffn   = 2 * d * (d_ff_mul * d) * n_layers   # 2 matrices per layer
    return embed + attn + ffn

# Doubling d_model from 4096 -> 8192 with 32 layers, vocab 128K:
small = transformer_param_estimate(128_000, 4_096, 32)
big   = transformer_param_estimate(128_000, 8_192, 32)
print(f"d=4096:  {small/1e9:5.2f}B params")
print(f"d=8192:  {big/1e9:5.2f}B params  (~{big/small:.1f}x)")
# Note: attention scales as d^2, FFN scales as d^2 (with mul=4).

External links

Exercise

Pull the architecture configs of three open-weight models (e.g., Mistral 7B, Llama 3 8B, Qwen 2.5-7B) from their config.json files. Tabulate d_model, n_layers, n_heads, vocab_size, intermediate_size. Where do the differences live? What design decisions can you read off?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousWord Embeddings: Token IDs Become Dense Vectors Next →Semantic Arithmetic: king − man + woman ≈ queen

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.