Missing Values and Outliers

~30 min · missing, outliers, preprocessing

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Missing means something

Missingness carries information. The pattern of which rows have missing values often correlates with the target. Adding a binary was_missing indicator beside the imputed value preserves that signal.

Imputation strategies, ranked by lift

Domain-specific fill (zero for "never bought", -1 for "unknown").
Median or mode imputation with a missingness indicator.
Iterative or KNN imputation for tabular with strong feature correlations.
Native missing-handling in tree models (lightgbm, xgboost) — let them split on missing.

Outliers as evidence

Outliers are sometimes data-entry bugs and sometimes the most important rows in the dataset. Decide per column whether to clip, log-transform, model-as-is, or split into a separate model. Removing outliers blindly throws away the customers who matter most.

Code

SimpleImputer with a missingness indicator column·python

from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer

numeric_imputer = SimpleImputer(strategy="median", add_indicator=True)
ct = ColumnTransformer([
    ("num", numeric_imputer, numeric_cols),
])

Robust outlier handling: clip, do not delete·python

import numpy as np

def robust_clip(s):
    q1, q3 = s.quantile([0.01, 0.99])
    return s.clip(q1, q3)

df["amount_clipped"] = robust_clip(df["amount"])

Let lightgbm handle missing natively·python

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=400, learning_rate=0.05, max_depth=-1
).fit(X_train, y_train)

External links

Exercise

Pick three columns with the highest missing fraction in your dataset. For each, choose between domain fill, median + indicator, or native tree handling. Justify the choice in one sentence per column.