The bet
RWKV ("Receptance-Weighted Key-Value") asks: can we get Transformer-quality outputs with RNN-style O(1) per-token inference? The answer, as of RWKV-7 ("Goose", March 2025), is "surprisingly close". RWKV is a different bet from Mamba — it leans even harder into the recurrent direction.
The key idea
RWKV uses time-mixing and channel-mixing operations that are reformulations of attention into a form that can be computed recurrently at inference. During training, attention-like parallel computation is possible; during inference, the same model can be evaluated step-by-step like an RNN.
Why this matters
- Constant memory at inference. RWKV processes sequences in O(1) memory regardless of length. Ideal for streaming applications.
- O(1) per-token inference cost. No attention to compute, no KV cache to grow. Fits naturally on resource-constrained devices.
- Edge / streaming applications. Live transcription, on-device assistants, low-power scenarios where Transformer attention is impractical.
RWKV-7 "Goose" advances
Full-matrix dynamic state decay — earlier RWKV versions used vector decay; v7 uses matrix decay, making the state-update step more expressive. Theoretically more powerful than earlier RWKV variants while keeping the recurrent inference property.
Where RWKV is now
RWKV is genuinely competitive at small-to-medium scale (1B–14B). Frontier-scale RWKV (70B+) is still in early development as of 2026. The community is small but active, and the architecture has its own ecosystem of inference engines.
RWKV vs Mamba — different bets
Both reject quadratic attention. Mamba leans on selective state spaces with structured dynamics; RWKV leans on attention reformulations with explicit recurrent inference. They are not converging — they are exploring different parts of the post-Transformer design space.