Input Pipeline: Putting It All Together

You now have all the moving parts of the input pipeline. Let's assemble them.

Tokenization. Raw text → token IDs. Frozen at training time, must be the model's tokenizer.
Embedding lookup. Token IDs → dense vectors via emb.weight[ids]. Output shape (seq_len, d_model).
Position injection. One of:
- Add sinusoidal PE to the embedding (original Transformer).
- Add learned positional embeddings to the embedding (BERT, GPT-2).
- Apply RoPE to Q and K inside attention; embedding is unchanged at the input (Llama, Mistral, Qwen).
- Apply ALiBi bias to attention scores; embedding is unchanged at the input (BLOOM, MPT).
Layer-norm and dropout (optional). Some models normalize and apply dropout to the input embedding before the first block.
The result enters the first Transformer block.

The key conceptual point: position is information and the architecture chooses where to insert it. There is no neutral choice. RoPE is currently winning, but the rest of the architecture is independent of which scheme you pick — you can swap RoPE for ALiBi in any modern decoder-only Transformer with a few hundred lines of code change.

Code

End-to-end input pipeline (RoPE-style)·python

class TransformerInput(nn.Module):
    def __init__(self, vocab_size, d_model):
        super().__init__()
        self.emb = nn.Embedding(vocab_size, d_model)
        # No learned positional embedding — RoPE is applied later
    def forward(self, ids):
        return self.emb(ids)             # (B, L, d_model)

# Inside the attention layer:
def attention_with_rope(x, W_q, W_k, W_v, n_heads):
    B, L, d = x.shape
    Q = (x @ W_q).view(B, L, n_heads, d // n_heads)
    K = (x @ W_k).view(B, L, n_heads, d // n_heads)
    V = (x @ W_v).view(B, L, n_heads, d // n_heads)
    Q = rope(Q)                          # rotate by position
    K = rope(K)                          # rotate by position
    # ... standard scaled dot-product attention from here ...

Exercise

In one Jupyter notebook, write a 50-line function that takes a string, a tokenizer, and an embedding matrix, and returns the (seq_len, d_model) tensor that goes into the first attention block. Make the position scheme a function argument so you can swap between sinusoidal, learned, and a placeholder for RoPE-style (which is just identity at the input).

Input Pipeline: Putting It All Together

Code

External links

Exercise

Progress

Comments 0