Self-attention is a learned database lookup
Self-attention's superpower is that every token can look at every other token and decide, per-query, how much to weight each one. That's a flexibility no previous sequence architecture offered. RNNs forced information through a single hidden state — a fixed-size bottleneck that lost long-range detail. Convolutions had a fixed receptive field — local by construction. Attention let the model build pairwise relationships at any distance, with weights computed at runtime from the actual content of the tokens.
Mechanically it's a three-tensor dance. For each token you compute a query Q ("what am I looking for?"), a key K ("what do I contain?"), and a value V ("what do I output?"). The pairwise score Q·Kᵀ/√d_k tells you how relevant every token is to every other token. Softmax turns those scores into a distribution. Multiplying by V produces a context-aware output for every position. The whole thing is one matrix multiply, one elementwise softmax, one more matrix multiply.
Why it stuck
Three properties made attention dominant once compute caught up. First, it's fully parallelizable across the sequence dimension during training — unlike RNNs which had to step through tokens sequentially. Second, multi-head attention learns multiple relationship patterns at the same time — different heads end up specializing in syntax, coreference, position, semantics. Third, it composes cleanly with residual streams and layer norms into a stack that scales: 2017's 65M-parameter Transformer and 2025's 2T-parameter mixture-of-experts use the same fundamental block.
This is the architecture you're competing against. Every alternative in this quest gives up something attention has — usually exact recall over the full context — to get something attention doesn't — usually subquadratic compute or constant inference memory. Knowing what attention is good at is the first step to knowing when to leave it behind.