What RWKV Is Trying to Do

The dual-form trick, again — but different

RWKV (Receptance Weighted Key Value, pronounced RwaKuv) is a linear-attention/RNN hybrid that was conceived from the start with one goal: train in parallel like a Transformer, run as an RNN at inference. SSMs have a similar duality, but RWKV is structurally closer to attention — its parameter naming (R, W, K, V) is not coincidence; it's a deliberate echo of attention's QKV.

The four parameters: R is the receptance gate (how much should this token absorb the running summary?), W is the exponential decay weight (how fast do older tokens fade?), K is the key (what do I contain?), V is the value (what do I output?). The WKV mechanism computes weighted key-value sums with exponential decay over time — it never materializes a T×T matrix.

Why this is a different sales pitch from SSMs

SSMs sell efficiency from a control-theory frame — "compress history into a state". RWKV sells the same efficiency from an attention-frame — "approximate attention with a recurrent state that decays exponentially". The math comes out adjacent (and the State Space Duality framework eventually showed they're related), but the engineering and ergonomic story is different. RWKV models look like Transformer-shaped checkpoints, can be ported to llama.cpp/MLX/ggml-style stacks more easily, and run as RNNs without any "is this state stable" worry.

The constant-time-per-token guarantee

The thing RWKV gives you that nothing else gives you in such a clean form is constant per-token cost at inference, by construction. The model is provably equivalent to an RNN with a fixed-size state, so generation latency is flat in context length. This is what makes RWKV the right pick for the on-device serving case — the cost of generating token N+1 is the same whether you're at token 1 or token 100,000.

Code

RWKV's WKV recurrence (simplified)·python

# At each timestep t, given input x_t:
#   k_t, v_t = K(x_t), V(x_t)
#   r_t = sigmoid(R(x_t))
# Maintain two running quantities (the 'state'):
#   num   <- num * exp(-w) + exp(k_t) * v_t
#   denom <- denom * exp(-w) + exp(k_t)
# Output:
#   wkv_t = num / denom
#   y_t   = r_t * wkv_t
# 'num' and 'denom' are the entire state. Constant size, regardless of t.

Exercise

Pull a small RWKV model from Hugging Face (e.g., RWKV/rwkv-4-169m-pile) and generate text with it. Then increase the prompt length from 256 → 8K → 64K tokens and measure per-new-token latency. The latency should be flat across all prompt lengths (the WKV state is the same size), in contrast to a Transformer where prefill grows quadratically and KV-cache memory grows linearly.