During training, randomly zeros each element with probability p, scaling the rest by 1/(1-p) so the expected output is preserved. During eval, becomes a no-op. The classic regularizer — prevents the model from depending too strongly on any single neuron.
Common values: 0.1 for Transformer FFN/attention, 0.5 for older fully-connected layers. Modern models use it more sparingly than 2015-era papers.
nn.Embedding
A lookup table from integer indices to dense vectors. Used wherever you have categorical inputs:
Token embeddings in NLP (vocab → vector).
Positional embeddings.
User / item embeddings in recommendation systems.
Conceptually it's just a (num_embeddings, embedding_dim) weight matrix where embed(idx) returns weight[idx]. The reason to use nn.Embedding instead of weight[idx] manually: PyTorch handles sparse gradients efficiently, supports padding indices that don't get updated, and integrates cleanly with the optimizer.
nn.ModuleList — the dynamic-depth pattern
Already covered in the Sequential lesson, but worth highlighting once more here: when your forward needs to iterate (Transformer with N layers, ResNet stack), use nn.ModuleList. Plain Python lists are invisible to PyTorch.
Code
Dropout — train vs eval behavior·python
import torch
import torch.nn as nn
drop = nn.Dropout(p=0.5)
x = torch.ones(1, 8)
drop.train()
print(drop(x)) # ~half the entries zeroed, rest scaled by 2
# tensor([[2., 0., 0., 2., 0., 2., 2., 0.]]) (varies)
drop.eval()
print(drop(x)) # tensor([[1., 1., 1., 1., 1., 1., 1., 1.]]) — no-op
Embedding — the lookup table·python
import torch
import torch.nn as nn
# Vocab of 10,000 tokens, each represented by a 256-dim vector
embed = nn.Embedding(num_embeddings=10000, embedding_dim=256)
token_ids = torch.tensor([42, 100, 7, 2023])
vectors = embed(token_ids)
print(vectors.shape) # torch.Size([4, 256])
# Batched
batch = torch.randint(0, 10000, (32, 50)) # batch=32, seq=50
print(embed(batch).shape) # torch.Size([32, 50, 256])
# padding_idx — vector for index 0 stays zero and isn't trained
embed_pad = nn.Embedding(10000, 256, padding_idx=0)
print(embed_pad.weight[0].sum()) # tensor(0.) — guaranteed
Putting it together — a minimal Transformer block·python
import torch
import torch.nn as nn
class TransformerBlock(nn.Module):
def __init__(self, d_model=512, n_head=8, d_ff=2048, drop=0.1):
super().__init__()
self.attn = nn.MultiheadAttention(d_model, n_head, dropout=drop, batch_first=True)
self.ln1 = nn.LayerNorm(d_model)
self.ln2 = nn.LayerNorm(d_model)
self.ff = nn.Sequential(
nn.Linear(d_model, d_ff),
nn.GELU(),
nn.Dropout(drop),
nn.Linear(d_ff, d_model),
nn.Dropout(drop),
)
def forward(self, x, attn_mask=None):
# Pre-LN style — modern Transformer convention
h = self.ln1(x)
h, _ = self.attn(h, h, h, attn_mask=attn_mask, need_weights=False)
x = x + h # residual connection
h = self.ln2(x)
h = self.ff(h)
x = x + h # residual connection
return x
block = TransformerBlock()
x = torch.randn(2, 16, 512) # batch=2, seq=16, dim=512
print(block(x).shape) # torch.Size([2, 16, 512])
Build a TransformerEncoder by stacking 6 of the TransformerBlock from the third code block via nn.ModuleList. Use a config dict to drive the depth. Verify the parameter count: each block ≈ 3.15M for d_model=512, so the stack should be around 18.9M.
Progress
Progress is local-only — sign in to sync across devices.