Self-Attention vs Cross-Attention

Two flavors of attention, used in different parts of an architecture:

Self-attention

Q, K, V all come from the same sequence. Each token attends to other tokens in the same input. This is what encoder-only models like BERT use throughout, and what decoder-only models like Llama use as the only attention type.

Cross-attention

Q comes from one sequence (the decoder), K and V come from another (the encoder's output). The decoder's tokens "look at" the encoder's representation of the input. Used in encoder-decoder models like T5 and Whisper to connect the encoder's understanding of the source to the decoder's generation of the target.

Architecture	Self-attn	Cross-attn
BERT (encoder-only)	Yes (bidirectional)	No
GPT / Llama (decoder-only)	Yes (causal)	No
T5 / Whisper (encoder-decoder)	Yes in encoder, yes in decoder (causal)	Yes (decoder Q attending encoder K, V)

For a translation task with a source sentence S and a target sentence T being generated, cross-attention is what lets the decoder, while producing the n-th target token, look back at the entire encoder representation of S and figure out which source tokens are relevant right now.

Code

Cross-attention in PyTorch·python

class CrossAttention(nn.Module):
    def __init__(self, d_model, n_heads):
        super().__init__()
        self.attn = nn.MultiheadAttention(d_model, n_heads, batch_first=True)
    def forward(self, decoder_x, encoder_out):
        # Q from decoder, K/V from encoder
        # PyTorch's MultiheadAttention takes (query, key, value)
        out, _ = self.attn(decoder_x, encoder_out, encoder_out)
        return out
# Inside an encoder-decoder block:
# x = self_attn(x)              # decoder self-attention (causal)
# x = cross_attn(x, enc_out)    # decoder looks at encoder output
# x = ffn(x)

Self-Attention vs Cross-Attention

Self-attention

Cross-attention

Code

External links

Exercise

Progress

Comments 0