The Transformer Is Not the Whole Story

The label stopped meaning anything around 2022

When GPT-3 dropped in 2020, "it's a Transformer" actually carried information. By 2022 the entire field had converged on the Transformer decoder, so saying a model "is a Transformer" became a tautology — like saying a car "has wheels". Two cars with wheels can be a Smart Fortwo or an F-150, and two Transformer LLMs can be a 7B you run on a MacBook or a 671B that needs a rack of H100s.

The information now lives in what was done to the backbone, not in the backbone itself. By 2026 the difference between two models is almost never "different attention" or "novel architecture". The differences live on four independent axes.

Why this quest exists

If you cannot name the axis a change lives on, you cannot reason about its cost shape, its latency, or its transferability to your use case. You will treat every announcement as either equally exciting or equally suspect. This quest gives you a vocabulary that survives the next news cycle.

Three things this quest is not: it is not a paper-by-paper survey, it is not benchmark cheerleading, and it is not "every architecture from Mamba to Hyena explained". It is the working compass the rest of those things hang off.

The minimum claim before continuing

If by the end of the next four lessons you cannot read "DeepSeek-V3 671B-A37B MoE with auxiliary-loss-free balancing, FP8 training, GRPO post-training, sigmoid routing, MLA attention, hidden CoT" and know which words live on which axis, the quest has not done its job. Don't memorize the words. Learn the axes that hold them.

Code

Pseudocode: what 'a Transformer' tells you in 2026·python

# In 2020 this told you a lot:
def what_is(model):
    return "a Transformer"

# In 2026 you have to ask four follow-ups:
def what_actually_is(model):
    return {
        "backbone":       model.architecture,        # dense / MoE / hybrid
        "post_training":  model.training_recipe,     # SFT / RLHF / GRPO / DPO
        "inference":      model.inference_strategy,  # std / extended-thinking / TTS
        "product":        model.product_layer,       # RAG / tools / agents
    }

Exercise

Pick three model announcements from the last six months (any LLM family — Llama, Qwen, DeepSeek, Gemini, Claude). For each one, write down the single sentence that the announcement leads with. Then circle each phrase in that sentence and label it 'backbone', 'training', 'inference', or 'product'. You will probably find two of the three sentences are mostly training/inference/product, with backbone barely mentioned.