Sinusoidal Positional Encoding (Original Paper)

The 2017 Transformer paper proposed encoding position with a fixed combination of sines and cosines at different frequencies:

PE(pos, 2i) = sin(pos / 10000^(2i / d_model))
PE(pos, 2i+1) = cos(pos / 10000^(2i / d_model))

Each position gets a unique vector of length d_model. Different dimensions oscillate at different rates: dimension 0 wraps around quickly (high frequency), dimension d_model−1 oscillates very slowly. Together, they form a unique fingerprint per position.

Why the design is clever

Unique per position. No two positions share the same encoding within the trained range.
Linear transformation captures relative position. PE(pos + k) = M_k @ PE(pos), where M_k depends only on k. This means the model can in principle learn relative-position attention as a linear operation on the encoding.
Zero parameters. The encoding is fully determined by the formula — no training needed.
Extrapolation potential. The formula is defined for all integer positions, so longer sequences than training are at least syntactically valid (though practically the quality often degrades).

Sinusoidal encoding has largely been superseded by RoPE in modern decoder-only LLMs, but it's still the cleanest mental model. If you understand sinusoidal first, RoPE is just "the same idea, but applied as a rotation inside attention instead of an addition at the input."

Code

Sinusoidal PE in NumPy·python

import numpy as np

def sinusoidal_positional_encoding(max_len, d_model):
    pe = np.zeros((max_len, d_model))
    position = np.arange(max_len)[:, None]                # (L, 1)
    div_term = np.exp(np.arange(0, d_model, 2) *
                      -(np.log(10000.0) / d_model))        # (d/2,)
    pe[:, 0::2] = np.sin(position * div_term)
    pe[:, 1::2] = np.cos(position * div_term)
    return pe

PE = sinusoidal_positional_encoding(512, 64)
# Add to input embeddings (broadcasting):
# x = token_embeddings + PE[:seq_len]

Sinusoidal Positional Encoding (Original Paper)

Why the design is clever

Code

External links

Exercise

Progress

Comments 0