Why Divide by √d_k? The Scaling Factor Explained

~10 min · scaled-dot-product, softmax

Level 0Token

0 XP0/94 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The 1/√d_k factor in the attention formula isn't decorative. Without it, large d_k destroys the softmax.

Here's why. Dot products of two random vectors of dimension d_k have variance proportional to d_k. So if Q and K are roughly unit-norm in expectation, Q · K typically has magnitude on the order of √d_k. For d_k = 64, that's about 8; for d_k = 128, about 11. Plug values of magnitude 8 into a softmax and the largest one almost wins outright — softmax saturates, the other entries become near-zero, and gradients through them die.

Dividing by √d_k keeps the variance of the score O(1) regardless of dimension. Softmax then produces a useful distribution that responds smoothly to changes in Q and K, and gradients flow.

This is one of the few places in modern Transformers where a simple constant correction makes the difference between "trains" and "doesn't train." It's also the reason the architecture is called scaled dot-product attention.

Code

See the saturation yourself·python

import torch

d_k = 128
q = torch.randn(d_k)
k = torch.randn(50, d_k)              # 50 candidate keys

raw = k @ q                            # ~N(0, sqrt(d_k))
scaled = raw / d_k**0.5

print('raw weights (top 5):',
      torch.softmax(raw, 0).topk(5).values.tolist())
print('scaled weights (top 5):',
      torch.softmax(scaled, 0).topk(5).values.tolist())
# The unscaled top weight is often > 0.9 (saturated);
# the scaled top weight is typically 0.05-0.2 (useful).

External links

Vaswani et al. — sec. 3.2.1 explains the scaling

Exercise

Train a tiny Transformer (2 layers, d_model=64) on a copy task with and without the √d_k scaling. Plot loss curves on the same axes. The unscaled version should plateau earlier or fail to learn entirely. How much earlier?

Progress

Progress is local-only — sign in to sync across devices.

← PreviousScaled Dot-Product Attention: Q · K^T → softmax → · V Next →Multi-Head Attention: Parallel Subspaces

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.