C.W.K.
Stream
Lesson 09 of 13 · published

Attention Patterns: What Heads Actually Learn

~12 min · interpretability, induction-heads

Level 0Token
0 XP0/94 lessons0/10 achievements
0/120 XP to next level120 XP to go0% complete

If you visualize the attention weights of a trained Transformer layer-by-layer and head-by-head, you'll find that heads specialize in remarkably consistent ways. This is one of the few interpretability findings that holds across model families.

Common emergent head types

  • Positional heads. Always attend to a fixed offset (most often −1, sometimes +1 in encoder models). These are how the network "knows" word order even after the position encoding has been processed.
  • Syntactic heads. Attend along grammatical relations: subjects to their verbs, adjectives to the nouns they modify, pronouns to their antecedents.
  • Induction heads (Anthropic, 2022). When the model has seen "A B" earlier in the context and then sees "A" again, an induction head attends specifically to the earlier "B." This is a key mechanism behind in-context learning and copying behavior.
  • Semantic heads. Attend to topically related tokens regardless of distance — useful for long-range coherence.
  • Rare-token heads. Concentrate attention on unusual or important tokens (named entities, low-frequency vocabulary).

Nobody hand-codes these. They emerge from training on next-token prediction. Anthropic's "induction heads" paper made the case that some of these specialized heads are not just decorative — they are the proximate cause of capabilities like few-shot learning.

Code

Visualizing one head's attention·python
from transformers import AutoModel, AutoTokenizer
import torch

model = AutoModel.from_pretrained("gpt2", output_attentions=True)
tok = AutoTokenizer.from_pretrained("gpt2")

inputs = tok("The cat sat on the mat", return_tensors='pt')
with torch.no_grad():
    out = model(**inputs)

# out.attentions: tuple of (B, n_heads, L, L) per layer
layer_idx, head_idx = 5, 3
attn = out.attentions[layer_idx][0, head_idx]   # (L, L)

import matplotlib.pyplot as plt
plt.imshow(attn, cmap='viridis')
labels = tok.tokenize(inputs['input_ids'][0])
plt.xticks(range(len(labels)), labels, rotation=45)
plt.yticks(range(len(labels)), labels)
plt.title(f"Layer {layer_idx} Head {head_idx}")
plt.savefig("head.png")

External links

Exercise

Use BertViz or write your own visualization for GPT-2 small. Find at least three different head types from the list above (positional, syntactic, induction). Capture screenshots and annotate what each head appears to be doing. Submit your gallery.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.