The walls aren't theoretical — they block real applications
It's easy to read complexity arguments as academic. They're not. Every workload that wants to consume more than ~32K tokens — full legal briefs, entire codebases, hours of audio at frame rate, biological sequences, multi-document RAG with all sources concatenated, long agentic traces — is already paying real money to the quadratic tax, or being denied by it.
For consumer hardware the wall is closer than most people realize. The 2025 study cited in the previous lesson — Transformers stopping at ~25K tokens on a 24GB consumer card while SSMs reach 220K+ — is not an edge case. It's the daily experience of anyone trying to run a coding assistant against their own repo on their own machine.
The deployment economics are the real motivator
A 128K-context Transformer model costs roughly 3–5× per request to serve compared to the same model at 8K context. That ratio is a structural tax paid by every long-context API consumer. It's why Anthropic's 1M-context Claude has a separate pricing tier. It's why some providers cap context at 32K-64K for the cheap tier. It's why an open-weights long-context model that runs cheaply on commodity GPUs would be a wedge product.
What "alternative" actually means
Given that landscape, an "alternative" architecture is selling at least one of three things. Linear or near-linear training compute, so longer contexts don't quadratically blow up training cost. Constant-memory inference, so per-token cost stays flat regardless of how long the context grows. Better hardware mapping, so the same FLOP count translates to more wall-clock throughput. The rest of this quest is, basically, a tour of who's selling what — and what they give up to sell it.