The Full Block: Putting Everything Together

You now have all the parts. A complete modern decoder-only Transformer block looks like this:

x ← x + MultiHeadAttention(RMSNorm(x))
x ← x + SwiGLU_FFN(RMSNorm(x))

That's it. Two sublayers, each preceded by a normalization and followed by a residual addition. Stack N copies, and you have the full Transformer body. The output of the last block goes through a final RMSNorm and the LM head (a projection back to vocab size, often weight-tied with the input embedding).

What changes between models

The block is remarkably stable across the major model families. What varies:

Attention type: dense MHA / GQA / MQA / sliding-window / sparse.
Position scheme: RoPE / ALiBi / iRoPE / Sandwich / yarn-scaled.
Activation: GELU / SwiGLU / GeGLU.
FFN: standard / Mixture-of-Experts replacing the FFN with a router + many experts.
Normalization: LayerNorm / RMSNorm / DeepNorm (rare).

Once you can read this diagram, you can read essentially every modern model card. The 95% of the architecture that doesn't change is what you've now internalized.

Code

A Llama-style block, end to end·python

class LlamaBlock(nn.Module):
    def __init__(self, d_model, n_q_heads, n_kv_heads, d_ff):
        super().__init__()
        self.norm1 = RMSNorm(d_model)
        self.attn = GroupedQueryAttention(d_model, n_q_heads, n_kv_heads)
        self.norm2 = RMSNorm(d_model)
        self.ffn = SwiGLU(d_model, d_ff)
    def forward(self, x, rope_cache=None, mask=None):
        x = x + self.attn(self.norm1(x), rope_cache=rope_cache, mask=mask)
        x = x + self.ffn(self.norm2(x))
        return x

# Stack:
class LlamaModel(nn.Module):
    def __init__(self, vocab_size, d_model, n_layers, n_q_heads, n_kv_heads, d_ff):
        super().__init__()
        self.tok_emb = nn.Embedding(vocab_size, d_model)
        self.blocks = nn.ModuleList([
            LlamaBlock(d_model, n_q_heads, n_kv_heads, d_ff)
            for _ in range(n_layers)
        ])
        self.final_norm = RMSNorm(d_model)
        # Weight tying: LM head reuses tok_emb.weight
    def forward(self, ids):
        x = self.tok_emb(ids)
        for block in self.blocks:
            x = block(x)
        x = self.final_norm(x)
        logits = x @ self.tok_emb.weight.T
        return logits

Exercise

Read karpathy/nanoGPT/model.py end to end. Identify every component you've seen so far in this quest (tokenizer → embedding → causal mask → multi-head attention → residual → norm → MLP → LM head). Annotate the file with a comment per section linking to the lesson that covered it. Submit your annotated copy.

The Full Block: Putting Everything Together

What changes between models

Code

External links

Exercise

Progress

Comments 0