RoPE: Rotary Position Embeddings (the Modern Standard)

Rotary Position Embedding (RoPE), introduced by Su et al. (2021), is the de-facto position-encoding standard for modern decoder-only LLMs. Llama 1/2/3/4, Mistral, Mixtral, Qwen, Phi, and Falcon all use RoPE or a close variant. It is worth understanding properly because nearly every long-context paper builds on top of it.

The core idea

Instead of adding a position vector to the input embedding, RoPE rotates the Q and K vectors at every position by a position-dependent angle. The rotation is applied inside attention, after the Q and K projections. The key property: the dot product between rotated Q at position m and rotated K at position n depends only on the relative distance m − n, not on the absolute positions.

θ_i = base^(−2i / d_head), where base = 10,000
(rotated_Q · rotated_K) at positions (m, n) ≡ f(content, m − n)

Why RoPE replaced its competitors

Relative position falls out for free, without dedicated relative-position parameters.
Zero learned parameters for position — it's a deterministic geometric transform.
Extrapolation works much better than learned PE. With YaRN-style scaling tricks (next lesson but one), models trained on 8K can be extended to 128K with modest fine-tuning.
Easy to implement. Element-wise complex-number multiplication; no extra layer.

Llama 4 uses iRoPE — interleaved RoPE — alternating layers with and without RoPE for even better long-context behavior. The base architecture remains the same; only the position-injection knob is being tuned.

Code

RoPE — minimal but real·python

import torch

def rope(x, base=10000.0):
    # x: (batch, seq_len, n_heads, d_head) — d_head must be even
    seq_len = x.shape[1]
    d_head = x.shape[-1]
    half = d_head // 2

    # Frequency per dimension (paired)
    inv_freq = 1.0 / (base ** (torch.arange(0, d_head, 2).float() / d_head))
    # Position-frequency outer product
    t = torch.arange(seq_len, dtype=inv_freq.dtype)
    freqs = torch.outer(t, inv_freq)            # (seq_len, half)
    cos, sin = freqs.cos(), freqs.sin()

    # Split last dim into pairs (even, odd) and rotate each pair
    x_even = x[..., 0::2]
    x_odd  = x[..., 1::2]
    rot_even = x_even * cos - x_odd * sin
    rot_odd  = x_even * sin + x_odd * cos

    out = torch.empty_like(x)
    out[..., 0::2] = rot_even
    out[..., 1::2] = rot_odd
    return out

# Inside attention:
# Q = rope(W_q(x))
# K = rope(W_k(x))
# V = W_v(x)   # V is NOT rotated

RoPE: Rotary Position Embeddings (the Modern Standard)

The core idea

Why RoPE replaced its competitors

Code

External links

Exercise

Progress

Comments 0