GQA in the Block: A Concrete Example

~8 min · gqa, block-summary

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

You've seen GQA in Track 4. Now place it inside the block. The change is local — only the K and V projections shrink — but the implications cascade through inference.

For Llama 3.3 70B (d_model=8192, n_q_heads=64, n_kv_heads=8, d_head=128):

Q projection: 8192 × (64 × 128) = 8192 × 8192 = 67M params per block.
K projection: 8192 × (8 × 128) = 8192 × 1024 = 8.4M params per block (was 67M with full MHA).
V projection: 8192 × 1024 = 8.4M params per block (was 67M with full MHA).
Output projection: 8192 × 8192 = 67M.

Total attention params per block: 151M (vs ~268M with full MHA). Per-block savings: ~117M. Across 80 blocks: ~9.4B parameters saved by GQA. But the bigger win is at inference: the KV cache per token shrinks 8× because we store only 8 KV heads × 128 dims instead of 64 × 128.

This is what 'GQA' means in practice: a structural decision in the block that pays off again and again every time you run inference.

Code

Per-block memory accounting with GQA·python

def per_block_kv_cache(seq_len, n_kv_heads, d_head, dtype_bytes=2):
    # K and V each
    return 2 * seq_len * n_kv_heads * d_head * dtype_bytes

# Llama 3.3 70B per layer at 32K context, FP16
mha = per_block_kv_cache(32_000, 64, 128) / 1e6
gqa = per_block_kv_cache(32_000, 8, 128) / 1e6
print(f"Per-layer KV cache:  MHA {mha:.0f} MB,  GQA {gqa:.0f} MB,  ratio {mha/gqa:.0f}x")
# Across 80 layers, GQA saves tens of GB of cache.

External links

Llama 3 tech report (architecture details)

Exercise

Compute the per-layer KV-cache size at 128K context for: Llama 3.3 70B (8 KV heads), Mixtral 8x22B (8 KV heads), Mistral 7B (8 KV heads), Qwen 2.5-7B (4 KV heads). All use d_head=128. Which has the smallest cache? What does that mean for serving?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousMixture of Experts: Sparse Compute, Dense Knowledge Next →Quiz · 4 questions

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.