Classical ML vs Deep Learning

Two different jobs that share a name

Classical machine learning — logistic regression, gradient-boosted trees, SVMs, random forests — is brilliant at structured tabular data, small to medium dataset sizes, and problems where the predictive signal is mostly in a handful of well-engineered columns. It is fast to train, easy to interpret, and runs on a laptop. XGBoost and LightGBM still win most Kaggle tabular competitions, and that is not a stale meme.

Deep learning is brilliant when the predictive signal is buried in raw, high-dimensional, redundant data — pixels, waveforms, character streams — and you have enough examples or a good pretrained backbone. The cost is real: GPUs, careful initialization, normalization, regularization, and a willingness to debug shapes for hours.

Tip: If your dataset fits in a CSV with a few hundred columns and a few hundred thousand rows, your first move is gradient boosting, not a neural network. Then escalate only if a strong baseline plateaus.

The baseline reflex

The single most common failure mode in deep-learning projects is skipping baselines. A team spends three weeks tuning a transformer on a tabular dataset where a tuned LightGBM model would have hit the same accuracy in twenty minutes. The cost is not just compute — it is the wrong mental model of which knobs matter. Baselines tell you how much room deep learning has to earn.

The shape of the right call

Reach for deep learning when at least two of these are true: the input is raw and high-dimensional, the relationship to the label is non-linear and compositional, you have enough data (or a strong pretrained model), and you can afford the training and serving cost. Otherwise, classical ML is still the adult choice.

Principle: Always train a stupid baseline first. The number you should beat is not zero — it is what the simplest reasonable model achieves on the same split.

Code

Baseline first, escalate later·python

from sklearn.dummy import DummyClassifier
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import f1_score
import lightgbm as lgb

dummy = DummyClassifier(strategy="most_frequent").fit(X_train, y_train)
print("dummy f1:", f1_score(y_val, dummy.predict(X_val), average="macro"))

lr = LogisticRegression(max_iter=1000).fit(X_train, y_train)
print("lr f1:   ", f1_score(y_val, lr.predict(X_val), average="macro"))

gbm = lgb.LGBMClassifier(n_estimators=500).fit(X_train, y_train)
print("lgbm f1: ", f1_score(y_val, gbm.predict(X_val), average="macro"))
# Only escalate to a neural network if the gap to the ceiling
# justifies the engineering cost.

Two different jobs that share a name

The baseline reflex

The shape of the right call

Code

External links

Exercise

Progress

Comments 0