Choosing: MLP, CNN, Transformer, Foundation Model

Match architecture to data shape

MLP — Tabular numeric features, fixed-size embeddings, simple regression/classification on flat vectors. Often loses to gradient boosting on real tabular data; useful as a head on top of a learned backbone.
CNN — Images, 2-D spatial signals (spectrograms), video frames, low-dimensional structured grids. Strong inductive biases for locality and translation. Pretrained backbones available for almost everything.
RNN / LSTM / GRU — Streaming sequences with constant per-token cost requirement, RL policies, embedded systems where transformer attention's quadratic cost hurts. Largely superseded by transformers for general use.
Transformer — Text, code, long sequences with global dependencies, multi-modal. The default for almost everything sequence-shaped in 2026.
State-Space Models (Mamba, Mamba-2, RWKV) — Very long sequences where transformer's O(n²) attention is too expensive. Linear in sequence length. Still emerging.
Foundation models (CLIP, DINOv2, SAM, GPT-class LLMs) — Use as black-box embeddings or as fine-tunable bases. The 'just use a foundation model' answer covers a surprising fraction of real applications.

Tip: Match the architecture's inductive biases to the data's structure. Don't apply attention to a 100-feature tabular dataset; don't apply CNNs to text. Mismatched architectures still 'work' but waste parameters and give worse generalization.

The 'foundation model first' shortcut

For most application work, the right move is: pick a relevant foundation model (CLIP for vision-language, DINOv2 for general vision, a Llama / Qwen / GPT-class LLM for text), embed your data, and train a tiny head. You skip 90% of the architecture decision.

Principle: In 2026, the architecture decision is mostly 'which foundation model do I start from?' rather than 'what shape is my custom CNN?' Make peace with that — the leverage is in the data and the head, not the backbone.

Code

Five architectures, one task type each·python

# Tabular: classical wins
import lightgbm as lgb
clf = lgb.LGBMClassifier(n_estimators=500).fit(X_train, y_train)

# Vision: foundation model + head
import torch.nn as nn
import torchvision.models as tvm
vit = tvm.vit_b_16(weights="IMAGENET1K_V1"); vit.heads = nn.Linear(768, n_cls)

# Text classification: pretrained transformer
from transformers import AutoTokenizer, AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained(
    "distilbert-base-uncased", num_labels=n_cls)

# Long-sequence text generation: causal LLM
from transformers import AutoModelForCausalLM
gen = AutoModelForCausalLM.from_pretrained("Qwen/Qwen2.5-7B-Instruct",
                                            device_map="auto")

# Time-series forecasting: foundation forecaster
# from chronos import ChronosPipeline
# forecaster = ChronosPipeline.from_pretrained("amazon/chronos-t5-large")

Choosing: MLP, CNN, Transformer, Foundation Model

Match architecture to data shape

The 'foundation model first' shortcut

Code

External links

Exercise

Progress

Comments 0