C.W.K.
Stream
Lesson 01 of 05 · published

Perfect Recall and In-Context Learning

~13 min · recall, icl, kv-cache

Level 0Observer
0 XP0/50 lessons0/14 achievements
0/100 XP to next level100 XP to go0% complete

The capability gap that doesn't go away

Transformers maintain an explicit KV cache where every past token is directly accessible to every later position. SSMs compress history into a fixed state. Hyena's filters are content-independent. RWKV's state decays. RetNet's decay is per-head fixed. Only Transformers preserve the original tokens at lookup time, and that's the property that makes in-context learning genuinely robust.

The 2025 NeurIPS spotlight "Achilles' Heel of Mamba" made this rigorous: pure SSMs systematically fail on copy and recall tasks where Transformers succeed, and the failure can't be patched with more data — it's structural. Five-shot MMLU is the canonical practical demonstration: pure SSMs lag substantially because in-context learning requires looking back at the few-shot examples and pattern-matching, and compressed state blurs that lookup.

This is why Transformers won't be replaced wholesale

The recall gap is the single most important fact for understanding why production frontier models in 2026 are still Transformers (or hybrids with significant attention components). Tasks like multi-step tool use, agentic workflows that accumulate observations, code generation that must respect distant function signatures, and RAG that retrieves and reasons over 10–100 chunks at once — all of these depend on exactly the kind of precise per-token recall that Transformers do natively and that compressed-state alternatives cannot.

You can build hybrids that recover most of this capability with much less attention. You cannot build a pure-replacement architecture that recovers it. That asymmetry is why every successful production deployment we've discussed in this quest is either a Transformer, a hybrid with attention layers, or a domain-specific model where exact recall doesn't matter.

The frontier reasoning angle

Frontier reasoning workloads — complex math, multi-step proof, sophisticated tool use, code-wide refactors — are particularly recall-heavy. The model needs to maintain coherent reference to many specific facts, intermediate results, and constraint definitions over thousands of tokens. This is the hardest workload in modern AI, and it's the one where Transformer's advantage is largest. Don't expect the frontier to leave the Transformer family until something seriously addresses this gap, which no current alternative does at production scale.

External links

Exercise

Construct a synthetic in-context learning task where the model is given 5 examples of a pattern (e.g., "input → reversed input") and asked to apply it to a new input. Run a small Transformer and a small Mamba at similar parameter counts on this task. Vary the example separator and the input length. The Transformer should learn the pattern reliably; the Mamba will degrade much faster as the examples get further from the query token. That degradation is the recall gap, in your own training run.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.