Why Similar Parameter Counts Feel Different

The riddle

A 70B dense model and a 70B MoE model with 12B active parameters are both called "70B", yet on the same hardware:

The dense model has predictable per-token cost — every token costs the same.
The MoE model has a much smaller per-token compute cost (close to a 12B dense), but needs the full 70B in VRAM.
The MoE model has potentially higher capability ceiling — different experts can specialize.
The MoE model is harder to fine-tune cleanly with LoRA because routing complicates which weights matter for what.

The framing rule

When two models have similar total parameter counts but feel different to use, the differences almost always come from three sources:

Active parameters — same total, different active means different per-token cost and different speed.
Post-training recipe — same backbone, different RL or SFT means radically different behavior.
Inference strategy — same checkpoint, extended thinking on vs off can 10× the latency and cost.

Worked example

"Why does Qwen3 235B-A22B feel about as snappy as a 30B dense, but answer questions like a 70B+ dense?" Answer: 22B active per token gives it ~30B-class FLOP cost, but the 235B total parameter capacity (plus extensive post-training) gives it a bigger knowledge surface than a 30B-class dense. Same "235B" can mean two completely different things depending on which axis you're standing on.

The mantra

"Same number, different shape." When you see two models with the same parameter count but different cost or behavior — ask which axis is doing the work. It is never "the parameter count is wrong". It is always "the parameter count alone is no longer enough to describe the model".

Code

Two 70Bs, two cost shapes·python

models = [
    {"name": "Llama 3 70B (dense)",    "total": 70,  "active": 70},
    {"name": "Hypothetical 70B-A12B",  "total": 70,  "active": 12},
]

for m in models:
    # Memory: similar (both ~140 GB BF16)
    mem_gb = m["total"] * 2
    # Per-token compute: very different
    rel_cost = m["active"] / 70
    print(f"{m['name']:24s}  mem ~{mem_gb} GB   rel-FLOP/tok {rel_cost:.2f}")

Exercise

Find two models in your normal use that have a similar advertised parameter count but feel different to you (one might be cheaper, slower, smarter, or more verbose). Without looking anything up, write down which axis you suspect is doing the work. Then check the model cards and see if you were right.