The Attention Breakthrough

Attention in one sentence

Attention computes a weighted average of value vectors, where the weights come from comparing a query vector to a set of key vectors. Each output position 'attends to' the inputs that matter most for it, with the weights learned end-to-end via backprop.

Self-attention is the special case where queries, keys, and values all come from the same sequence. Now every token can directly look at every other token in one step — no recurrence, no fixed-size hidden state. Long-range dependencies become trivially expressible.

Tip: If you read one paper from this whole quest, make it 'Attention Is All You Need' (Vaswani et al., 2017). It's eight pages, and it explains the architecture that powers every modern LLM.

The math of one attention head

Given Q, K, V (each shape [B, T, d_k]): attention(Q, K, V) = softmax(Q K^T / sqrt(d_k)) V. The sqrt(d_k) prevents the dot products from growing too large with dimension. The softmax turns the dot products into a probability distribution over input tokens. The result is [B, T, d_k] — same shape as the input.

Multi-head attention

One attention head learns one type of relationship. Multi-head attention runs many heads in parallel (each with its own Q, K, V projections) and concatenates the results. The intuition: different heads can learn different relationship types (syntax, coreference, position, semantic similarity) without competing for one set of weights.

Why attention is parallelizable

Unlike RNNs, attention can compute the output for every position simultaneously — all the dot products are independent. This is what made transformers train so much faster than RNNs on GPUs.

Principle: Attention is the most important architectural primitive of the last decade. Knowing exactly what its forward pass computes (Q, K, V → softmax → weighted sum of V) is the foundation for understanding every modern LLM.

Code

Scaled dot-product attention from scratch·python

import torch
import torch.nn.functional as F
import math

def scaled_dot_product_attention(Q, K, V, mask=None):
    # Q, K, V: [B, H, T, d_k]
    d_k = Q.size(-1)
    scores = Q @ K.transpose(-2, -1) / math.sqrt(d_k)   # [B, H, T, T]
    if mask is not None:
        scores = scores.masked_fill(mask == 0, float("-inf"))
    weights = F.softmax(scores, dim=-1)                 # [B, H, T, T]
    return weights @ V                                  # [B, H, T, d_k]

# PyTorch has a fused, fast version too
out = F.scaled_dot_product_attention(Q, K, V, attn_mask=None)

Multi-head attention via nn.MultiheadAttention·python

import torch
import torch.nn as nn

mha = nn.MultiheadAttention(embed_dim=512, num_heads=8, batch_first=True)
x = torch.randn(2, 16, 512)            # [B, T, d_model]
out, weights = mha(x, x, x)            # self-attention: q=k=v=x
print(out.shape)                       # [2, 16, 512]
print(weights.shape)                   # [2, 16, 16] averaged over heads

Attention in one sentence

The math of one attention head

Multi-head attention

Why attention is parallelizable

Code

External links

Exercise

Progress

Comments 0