The Attention Insight: Direct Pairwise Access

The breakthrough behind the Transformer is almost embarrassingly simple to state: let every token directly attend to every other token, no matter how far apart they are. No recurrence, no convolution window, no sequential state. Just a learned similarity function between every pair of positions.

This single move is what unlocks everything else. It is why the architecture parallelizes across positions, why long-range dependencies stop suffering from gradient decay, and why the same architecture transfers from text to images to audio to protein sequences without changing.

The three-line core

You will see this formula every track from now on. Memorize the shape:

scores = Q @ K.T / sqrt(d_k) — every pair (i, j) gets a similarity score.
weights = softmax(scores) — each row becomes a probability distribution over positions.
output = weights @ V — each position's output is a weighted blend of all values.

Three matmuls and a softmax. That's the entire mechanism. Everything else in modern Transformer architecture — multi-head, positional encoding, KV-cache, GQA, RoPE, Flash Attention — is engineering polish on top of this core.

Where the cost lives

The price of "every token sees every token" is the (n × n) score matrix: O(n²) compute and memory in sequence length. For sequences up to a few thousand tokens this is cheap. For 1M tokens it isn't, and that's why a sub-industry exists around making attention efficient (Track 4).

Code

Self-attention from scratch — no PyTorch nn modules·python

import numpy as np

def self_attention(X, W_q, W_k, W_v):
    # X: (n, d_model); W_*: (d_model, d_k or d_v)
    Q = X @ W_q
    K = X @ W_k
    V = X @ W_v
    d_k = K.shape[-1]
    scores = Q @ K.T / np.sqrt(d_k)         # (n, n)
    weights = softmax(scores, axis=-1)       # row-wise softmax
    return weights @ V                       # (n, d_v)

def softmax(x, axis):
    x = x - x.max(axis=axis, keepdims=True)  # numerical stability
    e = np.exp(x)
    return e / e.sum(axis=axis, keepdims=True)

Path length comparison·text

RNN:        position 1 ----step----step----step--... ----> position n   (depth n)
CNN k=3:    position 1 -conv-conv-conv-... -> position n               (depth log_k n)
Attention:  position 1 -------- direct weight --------> position n      (depth 1)

Exercise

Implement self_attention(X) from scratch in NumPy with shapes annotated at every step. Verify against torch.nn.functional.scaled_dot_product_attention on a small example with the same Q/K/V. They should match to within float precision. Save this file — you'll extend it with multi-head and causal masks in Track 4.