ALiBi: Attention with Linear Biases

ALiBi (Attention with Linear Biases, Press et al. 2022) takes the most radical approach to position: skip positional embeddings entirely. Add a fixed linear bias to attention scores instead, where the bias is proportional to the distance between query and key positions:

attention_score(i, j) = q_i · k_j^T − m × |i − j|

Each attention head has its own slope m, drawn from a fixed geometric sequence (no training needed). Closer positions get higher scores; distant positions get penalized linearly. The model never sees a positional embedding at all — position lives only in the bias added to the attention logits.

What this buys you

No positional parameters or computation at the input. ~11% less memory, ~11% faster training compared to learned PE.
Excellent extrapolation. A model trained on 1024-token sequences runs at 2048 tokens with no quality degradation, just by extending the bias matrix.
Dead simple. One subtraction in the attention computation.

ALiBi is used by BLOOM and MPT models. It's less popular than RoPE in 2026 because RoPE-with-YaRN scaling has caught up on extrapolation while keeping the relative-position-as-rotation elegance. Still worth knowing — the underlying insight (position can live in the bias, not the embedding) influenced subsequent research.

Code

ALiBi bias matrix·python

import torch

def alibi_bias(seq_len, n_heads):
    # Slopes are a fixed geometric sequence
    def get_slopes(n):
        start = 2 ** (-2 ** -(torch.log2(torch.tensor(n)).item() - 3))
        return torch.tensor([start ** (i + 1) for i in range(n)])

    slopes = get_slopes(n_heads)              # (n_heads,)
    # Distance matrix (n, n) of |i - j|
    pos = torch.arange(seq_len)
    rel = (pos[None, :] - pos[:, None]).abs().float()   # (seq_len, seq_len)

    # Per-head bias: -m * |i - j|
    bias = -slopes[:, None, None] * rel[None]            # (n_heads, n, n)
    return bias

# Inside attention:
# scores = Q @ K.transpose(-2, -1) / d_k**0.5
# scores = scores + alibi_bias(seq_len, n_heads)        # add per-head bias
# weights = softmax(scores, dim=-1)

Exercise

Take a small Transformer encoder. Train it briefly on a synthetic copy task with sequence length 256. Then evaluate at sequence length 1024 with three position encodings: (a) learned PE up to 256, (b) sinusoidal extrapolated to 1024, (c) ALiBi. Where does each scheme break or hold? Plot accuracy vs. test sequence length.

ALiBi: Attention with Linear Biases

What this buys you

Code

External links

Exercise

Progress

Comments 0