Why Sequence Data Is Hard

Variable length, order matters, long-range dependencies

An image is a fixed-size grid. A sentence is a variable-length sequence where the meaning depends on word order ("dog bites man" ≠ "man bites dog"). Audio is a long sequence at high sampling rates. Code is a structured sequence. Each token's meaning depends on tokens far away — sometimes the relevant context is hundreds of tokens back.

Three core challenges: variable length (no fixed input shape), order sensitivity (permuting tokens changes meaning), long-range dependencies (token N may depend on token N-1000).

Tip: Every sequence architecture in this track is an answer to one of those three challenges. RNNs handle variable length sequentially. Attention handles long-range dependencies in parallel. Transformers do both, with a quadratic cost in length that newer models (Mamba, RWKV) try to fix.

The sequence task shapes

Sequence classification (sentiment, intent) — one label for a whole sequence.
Sequence labeling (NER, POS) — one label per token.
Sequence-to-sequence (translation, summarization) — output sequence given input sequence.
Causal language modeling (GPT) — predict the next token given all previous tokens.

What you'd need from an ideal sequence model

Constant-time per-token inference, true long-range memory, parallelizable training, and reasonable parameter efficiency. No single architecture has all four. RNNs gave constant per-token cost but bad long-range memory; transformers gave great long-range memory and parallel training at the cost of quadratic attention.

Principle: Sequence modeling is a tradeoff space, not a solved problem. Knowing which axis your task is bound on (latency, context length, training compute) is half of picking the right architecture.

Code

Three sequence task shapes in one screen·python

from transformers import pipeline

# Sequence classification (one label per sequence)
clf = pipeline("text-classification", model="distilbert-base-uncased-finetuned-sst-2-english")
print(clf("I love this so much"))   # [{'label': 'POSITIVE', 'score': 0.999}]

# Sequence labeling (one label per token)
ner = pipeline("ner", model="dbmdz/bert-large-cased-finetuned-conll03-english")
print(ner("Pippa lives in Seoul."))

# Causal language modeling (next-token prediction)
gen = pipeline("text-generation", model="gpt2")
print(gen("Deep learning is", max_length=30))

Variable length, order matters, long-range dependencies

The sequence task shapes

What you'd need from an ideal sequence model

Code

External links

Exercise

Progress

Comments 0