Error Analysis

~28 min · errors, diagnostics

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Errors are the cheapest data you have

Error analysis is the discipline of looking at what your model gets wrong, in detail, with your eyes. The single highest-leverage activity in classical ML is sampling 50 errors after every training run and tagging them by failure mode. Most teams skip it; the teams that ship reliably never do.

A reusable triage tag set

Bad label — the ground truth itself is wrong.
Hard ambiguous — even a senior reviewer could see both sides.
Missing feature — a feature the model needs is not in the dataset.
Distribution shift — the example does not look like the training data.
Genuine model error — the model just made a mistake on a normal example.

The follow-through

Each tag points to a different fix. Bad labels go back to labeling. Missing features become work tickets. Distribution shift triggers a coverage check. Only "genuine model error" justifies more model tuning. Tagging errors prevents months of futile hyperparameter search.

Code

Sample errors stratified by error type·python

import numpy as np
import pandas as pd

preds = (probs >= 0.5).astype(int)
errors = X_val.assign(y_true=y_val.values, y_pred=preds, prob=probs)
errors = errors[errors["y_true"] != errors["y_pred"]]
fp = errors[errors["y_pred"] == 1].sample(min(25, len(errors)), random_state=7)
fn = errors[errors["y_pred"] == 0].sample(min(25, len(errors)), random_state=7)
review = pd.concat([fp, fn]).sample(frac=1, random_state=7)
review.to_csv("errors_to_review.csv", index=False)

Slice metrics by subgroup to find systemic errors·python

from sklearn.metrics import classification_report

for plan in df_val["plan_tier"].unique():
    mask = df_val["plan_tier"] == plan
    print(f"--- {plan} (n={mask.sum()}) ---")
    print(classification_report(y_val[mask], preds[mask], digits=3))

External links

Exercise

Sample 50 errors from your validation set. Tag each as bad-label, ambiguous, missing-feature, distribution-shift, or genuine-model-error. Report the tag distribution. Decide which fix has the highest expected lift.