A formal expressivity result, not just a benchmark win
Standard Transformers, by a 2023 result, are bounded by the TC⁰ complexity class — they can only solve problems that can be expressed by constant-depth threshold circuits with polynomial size. There are languages and computational problems strictly outside TC⁰, and a vanilla Transformer provably cannot solve them, no matter how big you make it.
RWKV-7's headline formal result: RWKV-7 provably exceeds TC⁰. Specifically, it can solve the S5 state-tracking problem using only 2 layers (NC¹ complexity — circuits with logarithmic depth, strictly more expressive than TC⁰), and it can recognize all regular languages using 4 layers. These are concrete computational problems that Transformers structurally cannot do at any size.
Where this shows up in practice
The TC⁰ exit isn't a benchmark trick; it predicts a real capability difference. RWKV-7 maintains richer sequential state — counts, parity, regular-language structure — that Transformers struggle with. RWKV-7 2.9B achieves 72.8 average across standard benchmarks, matching Qwen2.5-3B (71.4) and beating Llama 3.2 3B (69.7), despite being trained on only 5.6T tokens compared to 15–18T for those competitors.
That training-token-efficiency story matters. If RWKV is hitting Transformer-comparable benchmarks with one-third the training data, the architecture is doing something the Transformer isn't — likely the richer state dynamics letting the model learn faster from each example. This is one of the clearest "RWKV is not just a Transformer with different math" data points.
The honest caveat
TC⁰ exit is for state-tracking problems. It is not a general expressivity advantage. Transformers excel at parallel associative lookup over the full context — that's a different axis, and on it RWKV (like all recurrent-state architectures) gives ground. The lesson here isn't "RWKV is better than Transformer" — it's "different architectures sit in different complexity classes, and the choice matters more than people credit".