Small Models: Phi, Gemma, Qwen, and the Cost-Quality Frontier

Frontier-quality models are reaching billions of parameters, but a different lineage of small models — 1B to 30B — has become extraordinarily competitive on real tasks. They run on consumer hardware, fit on a single GPU, and are often the right choice for production.

Microsoft Phi

Phi-4 (14B) was trained on 9.8T tokens (much of it synthetic, curated for reasoning) using 1920 H100 GPUs for 21 days. Achieves GPQA 56.1%. Phi-4-mini (3.8B) uses GQA, supports 128K context, and runs comfortably on a 16GB GPU.

Google Gemma 3

Variants at 1B, 4B, 12B, 27B. Architectural twists: 5:1 ratio of local-window attention to global attention layers, sliding window of 1024 tokens. Vision via SigLIP ViT (400M params) backbone. SentencePiece tokenizer with 262K vocabulary covering 140+ languages.

Alibaba Qwen 3

Qwen 3-235B-A22B (MoE, 235B total / 22B active, 128K context) is the flagship. Dense variants like Qwen 3-32B and Qwen 3-7B are widely used in open-source. Qwen 3-30B-A3B (30B / 3B active) is an extreme-MoE design for efficient inference.

The lesson

Smaller models trained on better data + longer schedules can match much larger ones at fixed quality budgets. Phi-4 (14B) competes with models 5× its size on reasoning benchmarks. Picking the right small model often beats picking the largest possible model.

Code

Comparing small-model size and ability·python

from transformers import AutoConfig

models = [
    "microsoft/Phi-4",
    "google/gemma-3-12b-it",
    "Qwen/Qwen2.5-7B-Instruct",
    "meta-llama/Meta-Llama-3.1-8B",
]
for m in models:
    cfg = AutoConfig.from_pretrained(m)
    params_billion_estimate = (
        cfg.vocab_size * cfg.hidden_size +
        cfg.num_hidden_layers * (
            4 * cfg.hidden_size * cfg.hidden_size
            + 3 * cfg.hidden_size * cfg.intermediate_size
        )
    ) / 1e9
    print(f"{m:<40} ~{params_billion_estimate:.1f}B params")

Small Models: Phi, Gemma, Qwen, and the Cost-Quality Frontier

Microsoft Phi

Google Gemma 3

Alibaba Qwen 3

The lesson

Code

External links

Exercise

Progress

Comments 0