Putting it all together
You now have three lenses — dense, MoE, reasoning — and four axes — backbone, training, inference, product. The combinations are not infinite, but they are too many to remember as a list. This lesson is the master decision table: for each dimension that actually matters in production, which paradigm wins?
| Dimension | Dense | MoE | Reasoning-oriented |
|---|---|---|---|
| Per-token compute | Highest (all params active) | Lowest (only top-K experts) | Variable, depends on thinking depth |
| Memory footprint | = total params | = ALL params (≫ active) | = base model + KV for thinking tokens |
| Latency (per request) | Predictable | Lower compute but routing overhead | High and variable (5–20× standard) |
| Serving complexity | Simple — standard tensor parallelism | Complex — expert parallelism, balancing | Moderate — long sequences, budget control |
| Fine-tuning | Easy (mature LoRA ecosystem) | Hard (routing complicates adaptation) | Very hard (RL pipelines, reward models) |
| Local deployment | Best (llama.cpp, Ollama) | Challenging (huge total memory) | OK if base fits, just slower |
| Cost per token (API) | Proportional to size | Lower per-FLOP, but memory amortization | Much higher due to thinking tokens |
| Quality on hard reasoning | Good at scale | Good (training-dependent) | Excellent on hard problems |
| Quality on simple Q&A | Excellent (cheap and fast) | Excellent (cheap and fast) | Wasteful — overthinks simple questions |
The two big patterns to internalize
Pattern 1. Dense and MoE compete on the backbone axis (cost per token vs cost per memory). Reasoning is orthogonal — it can be layered on top of either. So "MoE vs reasoning" is a malformed comparison; the real comparison is "MoE vs dense" and "reasoning on vs off".
Pattern 2. Each row in this table maps to a real production constraint. If your bottleneck is memory, MoE makes it worse; if your bottleneck is compute, MoE helps; if your bottleneck is serving simplicity, dense wins; if your bottleneck is hard-task accuracy, reasoning helps; if your bottleneck is latency, reasoning hurts.