The Position Problem: Attention Is Permutation-Invariant

Here is a critical, easy-to-miss property of self-attention: it is permutation-equivariant. Shuffle the tokens of the input and the attention scores get shuffled the same way — the relationships are preserved, but the model has no idea which token came first, second, third, or last.

RNNs got order for free: their hidden state is updated step by step, so position is implicit in the dynamics. Transformers don't have that; they need an explicit mechanism to inject position. Without it, "the dog bit the man" and "the man bit the dog" are literally identical to the model — same set of tokens, same set of pairwise relations, no order.

Three families of solutions

Sinusoidal positional encoding (original 2017 paper): add a fixed sinusoid pattern to the input embeddings. Different positions get different sums of sines and cosines, the model learns to read it.
Learned positional embeddings (BERT, GPT-2): learn a separate embedding per position from 0 to max_len, just like token embeddings. Add to the input.
RoPE / ALiBi (modern standard): inject position directly into Q and K inside attention, not at the input layer. Better extrapolation, no extra parameters in some variants.

The next four lessons cover these in detail. They look like trivia but they are the difference between a model that handles 4K context and one that handles 1M.

Code

Demonstrating permutation equivariance·python

import torch
def attn(X):
    Q, K, V = X, X, X        # use embeddings directly for the demo
    s = Q @ K.T / X.shape[-1] ** 0.5
    return torch.softmax(s, -1) @ V

X = torch.randn(4, 8)         # 4 tokens, d=8
P = torch.tensor([[0,0,1,0],[1,0,0,0],[0,0,0,1],[0,1,0,0]]).float()
# Apply permutation: PX rearranges rows of X.
out_orig = attn(X)
out_perm = attn(P @ X)
# out_perm should equal P @ out_orig (permutation-equivariant).
print(torch.allclose(P @ out_orig, out_perm, atol=1e-5))  # True

The Position Problem: Attention Is Permutation-Invariant

Three families of solutions

Code

External links

Exercise

Progress

Comments 0