Same parameters, three execution patterns
RetNet's headline engineering claim is that the same trained parameters can be executed in three interchangeable modes, each optimal for a different scenario:
Parallel mode (O(n²)). Identical to attention computation: build the n×n score matrix (with the exponential-decay mask), softmax-normalize, multiply by V. Use this during training, where you have the full sequence at once and want to maximize GPU parallelism. Same matmul-friendly compute pattern as standard attention.
Recurrent mode (O(1) per token). Maintain the state s and update it per token. Use this during autoregressive inference. Constant memory, constant per-token compute. This is the mode that wins on long-context generation.
Chunkwise recurrent mode (O(N·B)). Split the sequence into chunks of size B. Within each chunk, use parallel mode. Across chunks, use recurrent mode (passing the chunk-end state forward). Use this during long-sequence training or pre-fill of long inference prompts. Trades a bit of training-time parallelism for the ability to handle sequences longer than what fits in a single attention matrix.
Why three modes is more than two
Many architectures have a parallel and a recurrent form. RetNet's chunkwise mode is the bridge that makes the dual-form actually scale — without chunking, you either pay O(n²) memory at long sequences (parallel) or you give up GPU parallelism entirely (pure recurrent). Chunked is what lets you train at long context while still using the GPU well.
The benchmark numbers
Original RetNet paper at 6.7B scale reported: 8.4× faster decode, 70% less memory, and 7× training speedup versus a comparably-sized Transformer. These are big numbers and they are real for the regime they were measured in. Whether they hold against today's optimized FA3+GQA Transformers at the same scale is — like all architecture comparison numbers — a function of which baseline you trust.