Qwen, Mistral, Phi — The Other Dense Families · Beyond the Transformer Quest

Qwen — the widest tier coverage

Alibaba's Qwen family has the broadest dense parameter coverage of any open-weight series. Qwen2.5 ships 7 dense tiers from 0.5B to 72B with a 152K vocabulary. Qwen3 (2025) refined this with 0.6B, 1.7B, 4B, 8B, 14B, and 32B dense variants — and crucially, all of them support dual-mode operation in a single checkpoint. Same weights run in fast mode (no thinking) or thinking mode (extended CoT). One model, two inference behaviors.

Mistral — efficiency-first dense

Mistral AI built its reputation on "small but punchy" dense models. Mistral 7B (2023) pioneered sliding-window attention in production LLMs, predating Gemma's interleaving by a year. Mistral NeMo 12B (2024, with NVIDIA) uses 40 layers, 5120 hidden dim, and the Tekken tokenizer, and remains a strong choice for resource-constrained deployment. Mistral Large 2 (123B) is one of the most capable dense closed-weight models from a non-US lab.

Phi — quality-from-data, not size

Microsoft's Phi line (Phi-3, Phi-4) chases a different lever entirely: extreme attention to training data quality. Phi-4 (14B, MIT-licensed) demonstrates that careful synthetic-data curation can let a small model punch well above its weight class. Useful as a counterexample to "scale solves everything" — sometimes the data axis matters more than the parameter-count axis.

Other notable dense families

Cohere Command A (111B): enterprise-focused dense, strong for agentic tool use.
Falcon 3: TII's continued work on efficient dense architectures.
InternLM and Yi: Chinese-lab dense families with their own specialization.

Reading dense families through the four axes

Most differences between Qwen, Mistral, and Phi at a given size live on axis 2 (post-training data and recipe) and axis 4 (deployment posture and license), not on axis 1. The backbones are remarkably similar — almost all are decoder-only with GQA, RoPE, SwiGLU. The personality lives elsewhere.

Code

Qwen3 dense lineup, with dual-mode flag·python

qwen3_dense = [
    ("Qwen3 0.6B",  0.6),
    ("Qwen3 1.7B",  1.7),
    ("Qwen3 4B",    4),
    ("Qwen3 8B",    8),
    ("Qwen3 14B",   14),
    ("Qwen3 32B",   32),
]

# Same checkpoint, two inference modes.
def chat_qwen3(prompt, *, thinking=False):
    return generate(prompt, enable_thinking=thinking)

Exercise

Pick one dense model in the 7–14B class from each of three families (Llama, Qwen, Mistral, Phi, Gemma — pick three). Read each model card's 'training data' section. Note one substantive difference in their training-data philosophy. This is the post-training axis showing through into model behavior at a similar size.

Qwen, Mistral, Phi — The Other Dense Families