Three Lenses — Dense, MoE, Reasoning · Beyond the Transformer Quest

Three primary lenses, not a taxonomy

The rest of this quest uses three primary lenses to look at the modern LLM landscape. They are not mutually exclusive categories — DeepSeek-R1 is MoE and reasoning-oriented at the same time. They are independent dimensions, like RGB color channels.

Lens 1 — Dense

Every token activates every parameter. The well-understood baseline. Llama, Gemma, Qwen dense, Mistral, Phi-4. Simple to fine-tune, simple to serve, simple to debug, well-supported by every inference engine on earth. Caps out somewhere around 100–400B before the FLOP cost per token gets nasty.

Lens 2 — Mixture of Experts

Many small expert FFNs per layer, only a few fire per token. Mixtral, DeepSeek-V2/V3/R1, Llama 4, Qwen3 MoE, Gemma 4 MoE, Mistral Large 3. Higher quality ceiling at the cost of more memory and serving complexity. The way frontier scale gets affordable.

Lens 3 — Reasoning-oriented

Spends extra inference compute to think before answering. o-series, Claude extended thinking, Gemini thinking modes, DeepSeek-R1, Qwen3 thinking mode. Almost always built on top of a dense or MoE backbone — the reasoning is in the post-training and inference axes, not the wiring.

How they combine

Combination	Example	Behavior
Dense + standard	Llama 3.3 70B	Predictable, fast, cheap, the conversational baseline.
Dense + reasoning	Qwen3 32B (thinking on)	Same checkpoint, higher latency, much better on hard tasks.
MoE + standard	DeepSeek-V3	Frontier-scale knowledge at near-30B inference cost.
MoE + reasoning	DeepSeek-R1	The current frontier paradigm. Capacity + careful thinking.

Code

All four combinations live in the same code path·python

def call_model(model, prompt, *, thinking=False):
    # The backbone (dense vs MoE) decides cost shape and engine config.
    # The thinking flag toggles inference strategy without touching weights.
    return model.generate(prompt, extended_thinking=thinking)

Three Lenses — Dense, MoE, Reasoning

Three primary lenses, not a taxonomy

Lens 1 — Dense

Lens 2 — Mixture of Experts

Lens 3 — Reasoning-oriented

How they combine

Code

External links

Exercise

Progress

Comments 0