Classical ML, Deep Learning, LLMs, and RAG

The map of approaches

Classical ML (linear models, trees, gradient boosting) dominates tabular problems and most production scoring systems. Deep learning dominates perception (vision, audio, text) where features are hard to hand-craft. Large language models excel at instruction-following over unstructured text. Retrieval-augmented generation grafts an LLM onto a search index so the answer comes from your data, not from the LLM's frozen weights.

What still wins on tabular data

For most enterprise scoring, ranking, and risk problems, gradient-boosted trees (xgboost, lightgbm, catboost) still beat neural networks at less compute and far less plumbing. Reach for deep learning when the input is naturally a tensor (image, waveform, sequence) or when you need an embedding to feed downstream classical models.

When to use an LLM

LLMs shine for tasks with no labeled data, where instructions are easier to write than code, and where 95% accuracy at zero training cost beats 98% accuracy after months of labeling. They are the wrong tool for high-precision scoring on structured data and for any problem where the cost of a hallucinated answer is large.

Code

Pick the right tool for the row of data·python

def pick_approach(row):
    if row.input_kind == "tabular" and row.label_count > 1000:
        return "gradient_boosted_trees"
    if row.input_kind in {"image", "audio", "raw_text"}:
        return "deep_learning"
    if row.input_kind == "unstructured_text" and row.labels_scarce:
        return "llm_few_shot"
    if row.needs_facts_from_corpus:
        return "rag"
    return "clarify_problem"

A classical baseline before reaching for an LLM·python

from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline

tfidf_lr = Pipeline([
    ("tfidf", TfidfVectorizer(min_df=5, ngram_range=(1, 2))),
    ("clf", LogisticRegression(max_iter=1000)),
]).fit(X_text_train, y_train)

The map of approaches

What still wins on tabular data

When to use an LLM

Code

External links

Exercise

Progress

Comments 0