Five versions, one family
RWKV evolves quickly. Each version has a bird codename, and each one fixes a concrete limitation of the previous.
v4 "Dove" (2023, EMNLP). Established the stable token-shift mechanism — every layer mixes the current token with the previous one, giving the model implicit access to short-range context without needing to attend explicitly. v4 also was the first RWKV to be trained at the 14B parameter scale, demonstrating that the architecture scales credibly.
v5 "Eagle" (2024, COLM). The big upgrade: multi-headed 64×64 matrix-valued states. Instead of a per-head vector state, each head maintains a full matrix. This dramatically expands the state's information-carrying capacity per parameter. v5 is the version Microsoft shipped to ~1.5 billion Windows machines.
v6 "Finch" (2024). Added LoRA-based data-dependent token shift, making the recurrence dynamic — the mixing weights between current and previous token can now adapt per input. About +5–7% benchmark improvement, and the first step toward content-aware adaptive recurrence.
v7 "Goose" (March 2025). The headline change: a generalized delta rule with full state evolution: S_t = S_{t-1} · (diag(w_t) + a_t^T · b_t) + v_t^T · k_t. This effectively performs gradient descent on prediction error within the recurrence — a form of in-context learning that was previously thought to require explicit attention.
Why version-tracking matters here
RWKV's version cadence is unusually fast for a serious architecture, and the differences are big. v4 results don't transfer to v6, v6 results don't fully transfer to v7. When you read "RWKV beats X on benchmark Y", check the version. The v7 results are what's current; older versions are mostly historical.