🌐

Beyond Attention Quest

Updated: 2026-05-04

A field guide to Transformer alternatives — Mamba, RWKV, RetNet, Hyena, hybrids

Transformers won the 2017–2025 cycle. The 2026 frontier is messier: SSM blocks, recurrent gates, implicit convolutions, and hybrids that splice attention into a mostly-not-attention stack. This quest walks the whole landscape with honest tradeoffs, real production deployments, and a practical decision guide so you stop chasing benchmarks and start matching architectures to workloads.

10 tracks · 50 lessons · ~11h · difficulty: intermediate-to-advanced

Level 0Observer

0 XP0/50 lessons0/14 achievements

0/100 XP to next level100 XP to go0% complete

Beyond Attention Quest is a 10-track tour through every credible post-Transformer architecture: why O(n²) attention finally hits a wall, how State Space Models compress history into a fixed state (S4 → Mamba → Mamba-3), how RWKV trains like a Transformer and runs like an RNN, how RetNet's retention seeded the entire SSD framework, how Hyena's implicit convolutions found their killer app in genomics, where linear/efficient attention variants like Kimi Linear actually beat full attention, and why every successful production deployment in 2026 is a hybrid (Jamba, Nemotron-H, Granite 4.0). The final tracks explain why Transformers still dominate despite all of this — explicit KV recall, a 750K-checkpoint ecosystem, GPU-friendly GEMM patterns — and give you a concrete decision framework: when to default to Transformer + FlashAttention-3 + GQA, when to reach for a hybrid, when SSM/Mamba is worth the bet, and how to read architecture papers without getting fooled by unfair baselines. By the end you can tell a builder which architecture to use for which job and defend the answer with numbers, not vibes.

Tracks

01❓Why Look Beyond Attention?
0/5 lessons
What Transformers do well, where they hit walls, and what "alternative" actually means
Transformers won the last decade because self-attention solved the right problem at the right time. They did not, however, repeal the laws of computation. This track lays out exactly what Transformers do well, where the O(n²) attention matrix and growing KV-cache turn into hard practical walls, and what families of architecture have emerged in response. Once you can name the bottlenecks precisely you can stop arguing about "is the Transformer dead" and start asking the only question that matters: which workloads pay the quadratic tax, and which don't.
Lesson list (5)
Quiz · 3 questions→
02📡State Space Models — The Big Alternative Family
0/5 lessons
From classical control theory to S4, S4D, H3, and the LTI ceiling Mamba broke
State Space Models are the largest family of Transformer alternatives because they sit on top of decades of classical control theory and signal processing — there's a deep mathematical bench to draw from. This track walks the lineage from the abstract SSM equations through HiPPO-based S4, the diagonal simplification S4D, the gated H3, and explains exactly why every pre-Mamba SSM hit the same Linear Time-Invariant ceiling. By the end you understand the dual training/inference trick that makes SSMs practical on GPUs, and you can name the precise capability they were missing before selectivity was added.
Lesson list (5)
Quiz · 3 questions→
03🐍Mamba and the Selective SSM Breakthrough
0/5 lessons
S6 selectivity, hardware-aware scan, State Space Duality, Mamba-3's inference-first design
Mamba is the moment SSMs stopped being a research footnote and started showing up in production. The unlock was selectivity — making B, C, and Δ functions of the input — which broke the LTI ceiling that capped every prior SSM. Selectivity also broke the convolution training trick, which forced the invention of a hardware-aware parallel scan that lives in GPU SRAM. This track walks the full Mamba 1/2/3 lineage: what selectivity is, why it required new training kernels, what the State Space Duality framework unified, and what Mamba-3's exponential-trapezoidal discretization and complex-valued states bought at inference time.
Lesson list (5)
Quiz · 4 questions→
04🦉RWKV — RNN Spirit, Transformer Flavor
0/5 lessons
Dual training/inference formulation, the v4 → v7 lineage, the TC⁰ exit, and the largest post-Transformer deployment in the world
RWKV is the alternative architecture you've actually used without knowing it — Microsoft shipped RWKV v5 to about 1.5 billion Windows machines for on-device Copilot, the largest deployment of any post-Transformer architecture by orders of magnitude. The headline trick is dual formulation: train as a Transformer (parallel, GPU-friendly), run as an RNN (constant memory, O(1) per token). This track walks the lineage from v4 Dove through v7 Goose, explains the formal expressivity result that put RWKV-7 outside the TC⁰ complexity class that bounds standard Transformers, and gives you an honest read on where RWKV's recall limits show up.
Lesson list (5)
Quiz · 3 questions→
05🔁RetNet — Retention and the Memory-Based Path
0/5 lessons
Three-paradigm computation, fixed exponential decay, and why RetNet's biggest contribution was inspiring everything that came after
RetNet is the alternative architecture you've heard mentioned in every post-Transformer overview but never seen in production at scale. Microsoft Research's 2023 paper introduced retention — a recurrent operation with fixed per-head exponential decay — that supports parallel, recurrent, and chunkwise modes from the same parameters. RetNet's headline numbers were impressive at 6.7B scale and the math was elegant. But its real contribution turned out to be conceptual: directly seeding Mamba-2's State Space Duality, Google's Hawk/Griffin, and Gated Linear Attention. This track walks the architecture, the three paradigms, the data-independent decay tradeoff, and why the architecture's deepest impact lives downstream.
Lesson list (5)
Quiz · 3 questions→
06🐾Hyena and Implicit Convolutions
0/5 lessons
FFT-based long convolutions, the StripedHyena lineage, and why the killer app turned out to be DNA
Hyena was the architecture that asked: what if instead of attention or recurrence, you used learned convolution filters for sequence mixing? The trick is making the filters <em>implicit</em> — generated by a small FFN from positional encodings — so they can be arbitrarily long without proportional memory. FFT-based application brings the cost down to O(L log L). Hyena got famous in 2023, struggled to outcompete Mamba on language modeling, and then quietly found its real home in genomics — where StripedHyena 2 powers Evo 2, a 40B model with 1M context that generates viable mitochondrial genomes. This track walks the operator, the multi-scale hybrid lineage, and explains why a niche-excellent architecture is sometimes more valuable than a generally-mediocre one.
Lesson list (5)
Quiz · 3 questions→
07⚡Linear & Efficient Attention Variants
0/5 lessons
Performer, Longformer, BigBird, sliding window, NSA, MoBA, Kimi Linear — staying inside the attention frame
Not every alternative architecture leaves attention behind. A whole family of approaches keeps the attention frame and reduces its O(n²) cost through kernelization (linear attention, Performer), sparsity (Longformer, BigBird, sliding window), or restructuring (NSA, MoBA, Kimi Linear). These are the Transformer-friendly alternatives — drop-in compatible with the ecosystem, easier to adopt, with most of the production wins in 2024–2026 happening in this lane. This track walks the lineage from the early kernelization attempts that didn't quite work, through sliding-window attention as Mistral and Griffin productionized it, to Kimi Linear's October 2025 result claiming first-ever across-the-board superiority over full attention.
Lesson list (5)
Quiz · 3 questions→
08🧬Hybrid Models — Where the Field Actually Converged
0/5 lessons
Jamba's 1:7 ratio, Hymba's parallel fusion, Nemotron-H's enterprise scale, and the MIRAS theoretical unification
If you read only one track of this quest, read this one. Every successful production deployment of a non-Transformer architecture in 2024–2026 is a hybrid — Jamba (1:7 attention:Mamba), Nemotron-H (~8% attention layers), Granite 4.0 (9:1 ratio), Bamba, Zamba2, Hymba, SAMBA. The field independently converged on a small range of attention:SSM ratios from labs that weren't talking to each other. This track explains why hybrids work, walks the production zoo, and points at the MIRAS framework that unifies all sequence models as variants of associative memory.
Lesson list (5)
Quiz · 4 questions→
09👑Why Transformers Still Dominate
0/5 lessons
Explicit recall, ecosystem inertia, GPU-friendly GEMM, training stability — and a healthy-skepticism framework for new claims
After eight tracks of alternative architectures, this is where you learn to defend the incumbent. Transformers hold nearly every frontier position in 2026 not out of inertia but because of concrete advantages: explicit per-token KV cache that makes in-context learning robust, a 750K-checkpoint ecosystem with mature serving infrastructure, GPU GEMM patterns that map perfectly onto modern hardware, and predictable training stability at scale. This track walks each of these advantages and ends with a framework for evaluating new architecture claims so you don't get fooled by paper-flavored speedups that don't reproduce in production.
Lesson list (5)
Quiz · 3 questions→
10🧭Practical Decision Guide for Builders
0/5 lessons
When to default to Transformer, when to reach for hybrid, when SSM/RWKV/Hyena actually pencil out, and how to stay current without chasing every paper
You've walked the entire post-Transformer landscape. Now turn it into a decision tree. This track gives you concrete guidance for the four most common builder questions: when to stick with Transformer, when efficient attention is enough, when SSM/Mamba progress is worth tracking for your stack, and how to stay current without drowning in papers. By the end you have an architecture decision table you can pin next to your screen — and a defensible answer when a teammate asks why you picked what you picked.
Lesson list (5)
Quiz · 4 questions→

Tracks

01❓Why Look Beyond Attention?

02📡State Space Models — The Big Alternative Family

03🐍Mamba and the Selective SSM Breakthrough

04🦉RWKV — RNN Spirit, Transformer Flavor

05🔁RetNet — Retention and the Memory-Based Path

06🐾Hyena and Implicit Convolutions

07⚡Linear & Efficient Attention Variants

08🧬Hybrid Models — Where the Field Actually Converged

09👑Why Transformers Still Dominate

10🧭Practical Decision Guide for Builders