Transformer Blocks

A transformer is a stack of nearly-identical blocks

Every transformer block has two sublayers: multi-head self-attention and a position-wise feedforward network (a small MLP applied independently to each position). Both sublayers are wrapped in residual connections and layer normalization.

That's it. Stack 12 of these blocks for a small transformer (BERT-base), 96 for a frontier LLM. Add an embedding layer at the input, an LM head or classifier at the output, and a position encoding so the model knows token order.

Tip: Once you can draw a transformer block from memory (residual + LN, attention, residual + LN, FFN), you can read every modern LLM paper. The architectural variations (RMSNorm, rotary position embeddings, GLU activations, mixture of experts) are all swap-outs for the basic pieces.

Position encodings

Self-attention is permutation-invariant on its own. To give the model a sense of order, we add positional information to the token embeddings. Original transformer used sinusoidal encodings; modern LLMs use RoPE (rotary position embeddings) which integrate position into the query/key projections themselves, allowing relative-position reasoning at any context length.

The two flavors of attention you'll see

Encoder (BERT, ViT) — bidirectional self-attention; every token attends to every other token. Used for classification, retrieval, embedding.
Decoder (GPT, LLaMA) — causal self-attention with a triangular mask; each token only attends to itself and previous tokens. Used for generation.
Encoder-decoder (T5, BART) — both, plus cross-attention from decoder to encoder. Used for translation, summarization, text-to-text tasks.

Principle: Modern transformers are mostly variations on one block. Learn that block deeply — embedding + positional encoding + N×(LayerNorm + Attention + LayerNorm + FFN with residuals) + final norm + head — and you can read every model card on Hugging Face.

Code

A pre-norm transformer block·python

import torch
import torch.nn as nn

class TransformerBlock(nn.Module):
    def __init__(self, d_model=512, n_heads=8, d_ff=2048, dropout=0.1):
        super().__init__()
        self.norm1 = nn.LayerNorm(d_model)
        self.attn  = nn.MultiheadAttention(d_model, n_heads, dropout=dropout,
                                           batch_first=True)
        self.norm2 = nn.LayerNorm(d_model)
        self.ffn   = nn.Sequential(
            nn.Linear(d_model, d_ff),
            nn.GELU(),
            nn.Dropout(dropout),
            nn.Linear(d_ff, d_model),
        )
        self.drop = nn.Dropout(dropout)

    def forward(self, x, attn_mask=None):
        # Pre-norm: norm before each sublayer, residual after
        h = self.norm1(x)
        x = x + self.drop(self.attn(h, h, h, attn_mask=attn_mask, need_weights=False)[0])
        h = self.norm2(x)
        x = x + self.drop(self.ffn(h))
        return x

A transformer is a stack of nearly-identical blocks

Position encodings

The two flavors of attention you'll see

Code

External links

Exercise

Progress

Comments 0