Quiz · 4 questions
🧱 The Transformer Block
Pre-LN, RMSNorm, SwiGLU, residual — the unit cell
Level 0Token
0 XP0/94 lessons0/10 achievements
0/120 XP to next level120 XP to go0% complete
Quiz
01What activation function do Llama and Mistral use in their FFN?
Hint
It uses gating and three weight matrices, not two.
02What normalization scheme do modern decoder-only LLMs typically use?
Hint
Modern means: norm comes before the sublayer, and the norm itself is simpler than the original.
03Where do most parameters live in a typical Transformer block?
Hint
Despite the name 'attention is all you need,' it's not where the params are.
04How many active parameters does Mixtral 8×22B use per forward pass?
Hint
Total params ≠ active params. Look at the routing K.
Comments 0
🔔 Reply notifications (sign in)Sign in — Please sign in to comment.
No comments yet — be the first.