Parameter Count Calculator: Reading Any Model Card

~10 min · params, model-card

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

You should be able to estimate any modern Transformer's parameter count from its (vocab, d_model, n_layers, d_ff, n_q_heads, n_kv_heads) tuple in your head, or at worst on a napkin. The formula is simple:

N ≈ vocab · d_model + n_layers · ((n_q + 2 n_kv) · d_model · d_head + d_model² + 3 · d_model · d_ff + 2 · d_model)

Breaking it down per block:

Q projection: n_q_heads · d_model · d_head — same as d_model² when n_q_heads = d_model / d_head.
K, V projections: n_kv_heads · d_model · d_head each. With GQA, much smaller than Q.
Output projection: d_model² (mixes heads).
FFN (SwiGLU): 3 · d_model · d_ff.
RMSNorm × 2: 2 · d_model (negligible).

The dominant terms are the FFN (about 70% of per-block params) and the embedding (a one-time cost that becomes proportionally smaller in large models).

Code

Parameter calculator·python

def count_params(vocab, d_model, n_layers, d_ff,
                 n_q_heads, n_kv_heads, weight_tied=True):
    d_head = d_model // n_q_heads
    embed = vocab * d_model
    per_layer = (
        # Q, K, V, O projections (GQA-aware)
        (n_q_heads + 2 * n_kv_heads) * d_model * d_head
        + d_model * d_model
        # SwiGLU FFN
        + 3 * d_model * d_ff
        # 2 RMSNorms
        + 2 * d_model
    )
    final_norm = d_model
    lm_head = 0 if weight_tied else vocab * d_model
    return embed + n_layers * per_layer + final_norm + lm_head

# Llama 3 8B: 128K vocab, 4096, 32 layers, 14336 ffn, 32 Q / 8 KV heads
print(count_params(128_000, 4096, 32, 14336, 32, 8) / 1e9, "B")
# ~8.0 B

External links

Hugging Face — model parameter calculator

Exercise

For each model in {GPT-2 small, Llama 3 8B, Mistral 7B, Mixtral 8x22B, Qwen 2.5 32B}, pull its config, compute the predicted parameter count with the formula, and compare to the announced number. Where do you mis-estimate? (Common: forgetting to include both K and V, or assuming GQA where there isn't.)

Progress

Progress is local-only — sign in to sync across devices.

← PreviousDepth vs Width: Where to Spend Parameters Next →The Four Numbers: d_model, d_ff, n_heads, n_layers

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.