Better Data Beats Fancier Models

Why data wins

For most production problems the model architecture is not the bottleneck. The bottleneck is label quality, label coverage, feature freshness, and the gap between training data and production data. A team that spends a week on labels and a day on the model usually beats a team that does the opposite.

Labels are a product, not a natural resource

A human label reflects a policy about which evidence counts and how ambiguity is resolved. If two experienced reviewers disagree on many examples, the boundary is not yet teachable. Write positive, negative, edge, and abstain examples so a new reviewer can reproduce the judgment.

Five high-leverage data moves

Write a labeling guide that turns judgment calls into reproducible rules.
Sample disagreements and re-label them as a team.
Add explicit "I don't know" and "ambiguous" labels so noise has a home.
Track label quality over time the way you track model quality.
Inspect rare classes and edge cases manually before any modeling.

Andrew Ng's data-centric flip

The data-centric AI message is that for fixed model code, improving data labels often beats tuning the model. Hold the model and split constant, repair a diagnosed class of data errors, and measure the change. This isolates the value of the data work instead of mixing it with a new estimator.

Quality needs an owner

Label review is engineering work, not assistant work. Name who approves the guide, adjudicates disagreements, backfills old data after a policy change, and versions the label definition. Otherwise a larger dataset simply mixes several incompatible eras of judgment.

Code

Track inter-annotator agreement·python

from sklearn.metrics import cohen_kappa_score

kappa = cohen_kappa_score(annotator_a, annotator_b)
print(f"agreement κ={kappa:.2f}  (>0.6 acceptable, >0.8 good)")

Sample low-confidence predictions for re-labeling·python

import numpy as np

probs = model.predict_proba(X_unlabeled)
uncertainty = 1 - np.abs(probs[:, 1] - 0.5) * 2
to_review = X_unlabeled.iloc[np.argsort(-uncertainty)[:200]]

Why data wins

Labels are a product, not a natural resource

Five high-leverage data moves

Andrew Ng's data-centric flip

Quality needs an owner

Code

External links

Exercise

Progress

Comments 0