The key constraint: data-independent decay
The retention weight on token m when computing the output for token n is γ^(n-m), regardless of the content of the tokens. The weight is determined entirely by recency — older tokens decay exponentially with a fixed rate per head, full stop.
Standard attention is the opposite: weights are content-dependent, determined by the dot product of Q_n and K_m. The model can learn to weight a particular distant token highly because of what it contains, not just because of where it is.
The expressiveness tradeoff
Data-independent decay is what gives RetNet its O(1) inference state. The convolution view of the recurrence works because γ is fixed. But it's also what limits expressiveness. RetNet cannot say "this particular earlier token is important — pay extra attention to it". Importance is solely determined by recency.
For tasks where recency is a good proxy for relevance — most natural-language modeling, especially summarization-style tasks — this is fine. For tasks that require the model to selectively focus on a specific earlier moment — the same recall-heavy tasks that defeat pure SSMs — retention falls short.
The mental model
Think of retention as automatic exponential forgetting: the model can't choose what to forget; the past simply fades on a fixed schedule. Standard attention is learned selective focus: the model can choose, per query, what to attend to, regardless of distance. The architectures sit at opposite ends of the recall-vs-cost spectrum.
Mamba's selectivity, in retrospect, was the obvious next move: keep the O(1) inference state, but give the model some ability to vary the decay per token. That's the missing piece RetNet didn't have, and it's why Mamba ended up with more production traction.