When Dense Is Still the Right Choice

Five strong reasons to choose dense over MoE in 2026

1. Local inference

llama.cpp, Ollama, MLX, and most local serving stacks are still dense-first. Quantization (Q4_K_M, Q5_K_M, AWQ) is mature for dense and immature or quirky for many MoE models. If your target is "runs on a MacBook Pro" or "runs on a single 24GB consumer card", dense is almost always the right answer.

2. Fine-tuning

LoRA, QLoRA, and full fine-tuning all work straightforwardly on dense models. PEFT, Unsloth, and Axolotl are dense-first ecosystems. MoE fine-tuning is possible but full of routing-related sharp edges (expert collapse during adaptation, capacity issues, balancing losses interfering with task losses).

3. Predictable serving costs

Dense gives you uniform per-token cost. Capacity planning is just multiplication: tokens × per-token cost = budget. MoE has more variable serving behavior (batch-size sensitivity, expert load skew, longer tail latencies) that complicates capacity planning.

4. Below ~30B parameters

The fixed costs of MoE (router parameters, load-balancing logic, expert-parallelism plumbing) eat the efficiency gains at small scale. Below 30B-class total parameters, dense generally beats MoE on quality-per-FLOP. Above 100B-class, MoE pulls ahead. The crossover region is the interesting design space.

5. Debuggability and interpretability

Most interpretability tools (attention visualization, activation steering, sparse autoencoder analysis) assume dense architectures. If you are doing research, alignment work, or any kind of "why did the model do that?" debugging, the dense ecosystem is much more developed.

The honest summary

MoE is the way to climb to frontier capacity affordably. Dense is the way to ship a serveable, fine-tunable, debuggable, locally-runnable model below frontier scale. They are not competitors — they are tools for different jobs. The mistake is treating MoE as "the future" and dense as "the past". They will both still matter in five years.

Code

Quick decision — dense or MoE?·python

def pick_backbone(scale_B, *, fine_tune=False, local=False):
    if scale_B < 30:
        return "dense"
    if local or fine_tune:
        return "dense (consider quantization for memory)"
    if scale_B >= 100:
        return "MoE (frontier capacity at manageable serving cost)"
    return "either — depends on your serving stack and ecosystem"

Exercise

Take the last LLM-using project you built or worked on. Re-evaluate: was the choice of model on the dense or MoE axis explicit? If it was implicit, write a one-paragraph note describing the workload (latency tolerance, scale, fine-tuning requirements, deployment surface) and which backbone family is the right fit. This is the muscle the rest of this quest is building.