Learned Positional Embeddings (BERT, GPT-2)

The simplest alternative to sinusoidal encoding: just learn a per-position embedding from scratch, the same way you learn token embeddings. The embedding matrix has shape (max_seq_len × d_model). Position 0 has its own row, position 1 has its own row, and so on.

BERT and GPT-2 use this. It is straightforward to implement, behaves predictably during training, and can capture position-dependent patterns that sinusoidal encoding cannot (e.g., "position 0 always gets a [CLS] token, so its embedding can specialize"). The cost: an extra max_seq_len × d_model parameters, and — critically — no extrapolation past max_seq_len. A model trained with max_seq_len=1024 has no embedding for position 1500.

This hard cap is why both BERT (max 512) and GPT-2 (max 1024) had relatively short context windows for their era. To get longer contexts you either (a) pre-train with longer sequences from the start, (b) interpolate the position embeddings post-hoc and fine-tune (the "Position Interpolation" technique), or (c) use a different positional scheme entirely. Modern LLMs chose (c).

Code

Learned positional embedding·python

import torch.nn as nn

class GPT2Embeddings(nn.Module):
    def __init__(self, vocab, d_model, max_len=1024):
        super().__init__()
        self.tok = nn.Embedding(vocab, d_model)
        self.pos = nn.Embedding(max_len, d_model)
        self.max_len = max_len
    def forward(self, ids):
        seq_len = ids.shape[-1]
        if seq_len > self.max_len:
            raise ValueError(f"seq {seq_len} > max_len {self.max_len}")
        positions = torch.arange(seq_len, device=ids.device)
        return self.tok(ids) + self.pos(positions)

Learned Positional Embeddings (BERT, GPT-2)

Code

External links

Exercise

Progress

Comments 0