The data type that broke Dense layers
Most data we've worked with so far has fixed shape and order-independent meaning. Text, audio, time series, DNA — all order-dependent and variable-length. Consider "the dog bit the man" vs "the man bit the dog": same words, opposite meaning. A standard Dense layer treats positions independently and symmetrically — it has no notion of "this token came before that token." That's why we need architectures designed for sequential data.
Three challenges sequences create:
- Variable length — one review is 10 words, another 1,000.
- Long-range dependencies — "it" might refer to a noun 40 words earlier.
- Compositionality — "not bad" means something different from "bad".
The history runs: SimpleRNN → LSTM/GRU → Attention → Transformer. Each one solved a problem the previous couldn't. We'll walk that history because the architectural decisions made along the way explain the AI landscape today.