Recall — the same wall, in a different shape
RWKV-7 maintains good performance up to about 28K tokens on passkey-retrieval tests. Past that, performance degrades. The shape of the failure mode differs from Mamba's (RWKV's exponential decay produces a different pattern of forgetting than Mamba's selective compression), but the category of failure is the same: finite state can't store arbitrary amounts of information.
This is the recurring theme of this quest. Every architecture that tries to replace attention runs into the same wall at some context length, just at different lengths and with different failure shapes. RWKV's wall is at ~28K. Mamba's is similar. Hyena's is different in shape but present. Transformers don't have the wall, but they have the cost.
Ecosystem reality check
RWKV's ecosystem is impressively broad for an alternative architecture, but it remains much smaller than Transformer's. Fine-tuning recipes, instruct-tuned variants, evaluation harnesses, RLHF infrastructure — all are thinner. Benchmark competitiveness still requires more training tokens to compensate for the architectural constraints, which means small fine-tuning runs hit harder ceilings than equivalent Transformer fine-tunes.
The upside: a smaller ecosystem means more opportunity to contribute. If you're a researcher or hobbyist who wants to leave a mark, RWKV's tooling has more low-hanging fruit than the Transformer ecosystem in 2026.
What's coming: v8 Heron and RWKV-X
RWKV-8 "Heron" introduces two new ideas: DeepEmbed (an edge-MoE-style sparse mechanism without VRAM overhead, allowing larger effective model capacity at fixed inference memory) and ROSA (a neurosymbolic suffix automaton for structured knowledge — bringing explicit symbolic state into a recurrent architecture).
RWKV-X (April 2025) is the family's hybrid response to the recall limitation: RWKV layers + sparse attention layers, mirroring the SSM-attention hybrid pattern from Jamba and Nemotron-H. The convergence to hybrids is happening across families, not just within Mamba.