Quiz · 6 questions
🔀 Mixture of Experts (MoE)
Many experts per layer, only a few fire per token — frontier capacity at affordable compute
Level 0Scout
0 XP0/41 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
Quiz
01DeepSeek-V3 has 671B total parameters and 37B active per token. What does this mean in practice?
Hint
Memory is what is *reachable*; compute is what is *reached*.
02What is the ONLY structural difference between a dense Transformer and an MoE Transformer?
Hint
If you printed both architectures side by side, only one box would differ.
03Why is 'expert collapse' a problem in MoE training?
Hint
It's the failure mode where the rich get richer.
04DeepSeek-R1 and DeepSeek-V3 share the same MoE backbone (671B-A37B, 256+1 experts top-8). What makes R1 a 'reasoning model'?
Hint
Same wiring, different schooling.
05What did DeepSeek-V3 introduce as an alternative to the standard auxiliary load-balancing loss?
Hint
The trick is to make the router *naturally* prefer underused experts without telling it to.
06True or false: 'Each expert in an MoE model specializes in a semantic topic (e.g., one expert for science, one for code).'
Hint
Reality is more mundane than the marketing diagrams.
Comments 0
🔔 Reply notifications (sign in)Sign in — Please sign in to comment.
No comments yet — be the first.