The History Arc

Three inflection points

Deep learning's modern history is short and sharp. 2012 — AlexNet: a CNN trained on two GPUs cuts the ImageNet top-5 error roughly in half overnight. The community, until then sceptical of neural nets, pivots within a year. 2014–2017 — the architecture rush: VGG, GoogLeNet, ResNet, attention mechanisms, sequence-to-sequence models, batch normalization, Adam. The toolbox we still use today gets built. 2017 — Transformer: Attention Is All You Need proposes a recurrence-free architecture that scales horizontally on GPUs/TPUs. By 2018 BERT and GPT had eaten NLP; by 2020 it had eaten vision (ViT) and speech.

The ingredients that made it possible

None of this was algorithmic genius alone. Three things had to align: data (ImageNet, the open web), compute (GPUs going from a graphics oddity to general-purpose tensor engines), and differentiable software (Theano, Caffe, then PyTorch and TensorFlow). Researchers in 1995 had similar ideas; they did not have any of the three.

Tip: When you read 'X invented backpropagation in 1986,' the right reaction is: 'and it sat unused for 25 years because the data and the GPUs were not there yet.' Algorithms wait for infrastructure.

What the arc tells you about the next ten years

Each inflection looks obvious in retrospect and shocking in the moment. The arc has not stopped — foundation models, multi-modal training, and reasoning-oriented post-training are the current frontier. The pattern is the same: better representations, more data, more compute, more ergonomic software. Building intuition about which ingredient is binding for your problem is more useful than memorizing the dates.

Principle: Every named architecture in this quest is a frozen snapshot of one inflection. The point is not to memorize them — the point is to recognize the pattern when the next one arrives.

Code

The architecture rush in one screen·python

milestones = [
    (2012, "AlexNet",        "First CNN to win ImageNet by a huge margin"),
    (2014, "VGG",             "Deeper is better, with simple 3x3 convs"),
    (2014, "GoogLeNet",       "Inception modules; multi-scale within a layer"),
    (2014, "Seq2Seq",         "Encoder-decoder for translation"),
    (2015, "Batch Norm",      "Normalization that made deeper nets trainable"),
    (2015, "ResNet",          "Residual connections; 100+ layer networks work"),
    (2014, "Adam",            "Per-parameter learning rates; default optimizer"),
    (2017, "Transformer",     "Attention-only; no recurrence; massively parallel"),
    (2018, "BERT / GPT-1",    "Pretrain a transformer, fine-tune on tasks"),
    (2020, "ViT",             "Transformers eat vision too"),
    (2022, "Diffusion + LLM", "Generative scaling on text and images"),
    (2024, "Reasoning LLMs",  "Test-time compute as a first-class scaling axis"),
]
for year, name, why in milestones:
    print(f"{year}  {name:14}  {why}")

Three inflection points

The ingredients that made it possible

What the arc tells you about the next ten years

Code

External links

Exercise

Progress

Comments 0