Missing Values and Outliers

Missing means something

Missingness can carry information. A person was never tested, a customer skipped an optional field, or an order has not completed yet. Before filling a blank, ask who creates the value, when it appears, and why it can be absent. Compare missing rates across time and important groups to find process changes and access differences.

Imputation strategies

Use a domain-specific value when the meaning is known, such as zero for "never bought."
Use median or mode imputation with a binary was_missing indicator for unknown measurements.
Consider iterative or KNN imputation when feature relationships are strong and the extra assumptions are justified.
Compare native missing handling in tree models such as LightGBM and XGBoost.

Fit imputers after the split

A median or neighbor structure computed on the full dataset leaks validation and test distribution into training. Fit the imputer inside the pipeline on training data only, repeat it inside each cross-validation fold, and use the same fitted object for serving.

Outliers as evidence

An age of 420 is probably an input error; a top-revenue customer may be the most important row in the dataset. Trace extreme values back to source records and units before deleting them. Preserve rare-but-valid cases as a dedicated evaluation slice.

Choose treatment for the model and loss

Linear models may need log transforms, robust scaling, clipping, or a robust loss because one extreme value can dominate a coefficient. Trees are less sensitive to scale but can still make unstable splits on tiny extreme groups. Document every clipping boundary and compare subgroup errors before and after.

Deletion is a decision too

Dropping missing or extreme rows can remove a whole user group. Compare counts, target prevalence, and key demographics before and after filtering. The same inputs will arrive in production, so define whether serving rejects them, uses a safe default, or routes them to human review.

Code

SimpleImputer with a missingness indicator column·python

from sklearn.impute import SimpleImputer
from sklearn.compose import ColumnTransformer

numeric_imputer = SimpleImputer(strategy="median", add_indicator=True)
ct = ColumnTransformer([
    ("num", numeric_imputer, numeric_cols),
])

Robust outlier handling: clip, do not delete·python

import numpy as np

def robust_clip(s):
    q1, q3 = s.quantile([0.01, 0.99])
    return s.clip(q1, q3)

df["amount_clipped"] = robust_clip(df["amount"])

Let lightgbm handle missing natively·python

import lightgbm as lgb

model = lgb.LGBMClassifier(
    n_estimators=400, learning_rate=0.05, max_depth=-1
).fit(X_train, y_train)

Missing means something

Imputation strategies

Fit imputers after the split

Outliers as evidence

Choose treatment for the model and loss

Deletion is a decision too

Code

External links

Exercise

Progress

Comments 0