If you visualize the attention weights of a trained Transformer layer-by-layer and head-by-head, you'll find that heads specialize in remarkably consistent ways. This is one of the few interpretability findings that holds across model families.
Common emergent head types
- Positional heads. Always attend to a fixed offset (most often −1, sometimes +1 in encoder models). These are how the network "knows" word order even after the position encoding has been processed.
- Syntactic heads. Attend along grammatical relations: subjects to their verbs, adjectives to the nouns they modify, pronouns to their antecedents.
- Induction heads (Anthropic, 2022). When the model has seen "A B" earlier in the context and then sees "A" again, an induction head attends specifically to the earlier "B." This is a key mechanism behind in-context learning and copying behavior.
- Semantic heads. Attend to topically related tokens regardless of distance — useful for long-range coherence.
- Rare-token heads. Concentrate attention on unusual or important tokens (named entities, low-frequency vocabulary).
Nobody hand-codes these. They emerge from training on next-token prediction. Anthropic's "induction heads" paper made the case that some of these specialized heads are not just decorative — they are the proximate cause of capabilities like few-shot learning.