The signal-to-noise problem
The post-Transformer space publishes more papers per month than any single engineer can read carefully. Trying to read everything is the wrong strategy — you'll burn out, you'll get fooled by the most-marketed papers (which aren't always the best), and you'll lose track of fundamentals while chasing the latest acronyms.
The right strategy is to follow a small number of high-signal sources and apply the four-question framework from the previous track to anything new that crosses your desk. Skim widely, read deeply only when something passes the four-question filter.
The high-signal source list
- state-spaces/mamba on GitHub (18K+ stars). Issues, PRs, and discussions are where the working knowledge lives. Read commits more than blog posts.
- RWKV Foundation (rwkv.com) for RWKV roadmap and version tracking.
- Hugging Face blog for new model architecture announcements that will actually ship to ecosystem support.
- Latent Space podcast / blog for high-quality interviews with architecture researchers — the framing is often clearer than the papers themselves.
- Papers With Code, sequence modeling section, for benchmark-tracked architecture comparisons.
- NeurIPS / ICML / ICLR architecture tracks for the year's most-cited architecture work — read the spotlight papers, skip most of the rest.
What to read with caution
- Benchmark results — always check whether the comparison is against optimized FA3+GQA Transformers or naive baselines. Many "X beats Transformer" results disappear under fair baselines.
- Scaling claims — most architecture comparisons are tested at <10B scale. Behavior at 70B+ can be different. Don't assume "works at 1B" means "works at 70B".
- Production claims — distinguish paper demos from actual production deployments. "Could be deployed" and "is deployed at scale" are very different things.
The fundamentals don't move
Through all the architecture churn, the fundamental tradeoff is stable: recall vs. compression. Every architecture in this quest makes a different choice on this spectrum. Transformer maximizes recall at cost. Pure SSM maximizes efficiency at recall cost. Hybrids try to keep both, paying a small efficiency tax for usable recall.
If you understand where your workload sits on the recall-vs-compression spectrum, you have a more durable mental model than someone who's memorized the latest twenty architecture acronyms but not the underlying tradeoff. Architectures will keep evolving; the underlying tradeoff will keep being the underlying tradeoff. Anchor on the fundamentals; sample the new work.