The Transformer was published in 2017 as a translation model. Within two years it had escaped translation and become the universal substrate for almost every NLP system that mattered. Within five years it had escaped language entirely.
The pivotal early follow-ups:
- GPT-1 (June 2018, OpenAI): a 117M-parameter decoder-only Transformer pre-trained on next-token prediction over BooksCorpus, then fine-tuned for downstream tasks. The "pre-train then fine-tune" recipe is born.
- BERT (October 2018, Google): a 340M-parameter encoder-only Transformer pre-trained with masked language modeling. State of the art on every NLU benchmark for years.
- T5 (October 2019, Google): "every NLP task is text-to-text." Uses the full encoder-decoder shape and unifies translation, summarization, classification, and QA.
- GPT-3 (May 2020, OpenAI): 175B parameters, demonstrating that scale unlocks in-context learning — solving new tasks just from examples in the prompt, no fine-tuning required.
By 2024-2026, every frontier model — GPT-4/5, LLaMA 4, Gemini 2.5 Pro, Claude 3.7 Sonnet, Mistral Large, Qwen 3 — is a Transformer variant. The unit cell from the 2017 paper is unchanged; only its scale, training data, and post-training (RLHF, DPO, GRPO, Constitutional AI) have evolved.