Spotting Dense vs MoE From Public Material

The signals

You can almost always tell whether a model is dense or MoE without ever reading the technical report. Watch for:

Parameter notation

Single number ("70B", "405B") almost always means dense. Dense doesn't need a second number because total = active.
"XB-AYB" notation ("235B-A22B", "671B-A37B") is the standard MoE shorthand: total / active.
"X×YB" notation ("8×7B", "8×22B") is the older style for MoE — N experts of size Y. This was Mixtral's branding; many newer MoE models have moved to XB-AYB.

Vocabulary

Any mention of "experts", "router", "gate", "top-K", "routing", "expert collapse", "load balancing" is MoE.
Any mention of "fine-grained experts", "shared experts", "auxiliary-loss-free balancing" is specifically modern MoE (DeepSeek-style).

Cost-shape clues

"Cheaper than its size suggests" or "compute-efficient at scale" or "frontier capability at lower per-token cost" almost always = MoE.
"Lightweight, fast, predictable" or "easy to fine-tune, easy to quantize" almost always = dense.

Config files don't lie

If a Hugging Face repo exists, check config.json for fields like num_experts, num_experts_per_tok, num_local_experts, router_aux_loss_coef. Their presence is dispositive. Their absence almost certainly means dense.

Code

Quick check on a Hugging Face config·python

import json, urllib.request

def is_moe(repo):
    url = f"https://huggingface.co/{repo}/raw/main/config.json"
    cfg = json.loads(urllib.request.urlopen(url).read())
    moe_signals = ["num_experts", "num_local_experts",
                   "num_experts_per_tok", "router_aux_loss_coef"]
    return any(k in cfg for k in moe_signals)

The signals

Parameter notation

Vocabulary

Cost-shape clues

Config files don't lie

Code

External links

Exercise

Progress

Comments 0