Null and Outlier Handling — Strategies, Not Defaults

Don't paper over nulls; understand them first

The instinct "I have nulls, let me fill them with the mean / 0 / forward-fill" is the most common bad reflex in data work. Nulls always carry information — they're either missing because the field doesn't apply, missing because the source didn't record it, or missing because of a bug. Each origin needs a different response, and "fill with mean" is almost never the right one.

Four legitimate strategies

Drop the row — when the row is unusable without the field. df.dropna(subset=['order_id', 'amount_usd']).
Drop the column — when the field is mostly null and removing it doesn't lose information.
Impute with a sentinel — explicit 'unknown', -1, or a domain default — when downstream logic needs a value but the missingness is meaningful.
Impute with statistics — mean/median/forward-fill — only when the field is genuinely interchangeable across rows. Rare in practice; often masks bugs.

Outliers — same discipline

Outliers can be (a) data-entry errors (amount = 99999999 instead of 9999.99), (b) legitimate but unusual values, or (c) the entire signal you're trying to detect (fraud, anomalies). Don't auto-cap or auto-remove them. Surface them, count them, and decide per pipeline what they mean.

Code

Explicit null/outlier handling — log what you did, don't silently fix·python

import logging
import pandas as pd

log = logging.getLogger('quality')

def handle_nulls(df: pd.DataFrame, required: list[str]) -> pd.DataFrame:
    n_before = len(df)
    df = df.dropna(subset=required)
    log.info('nulls.dropped=%d required=%s', n_before - len(df), required)
    return df

def handle_outliers(df: pd.DataFrame, col: str, low_q: float = 0.01, high_q: float = 0.99) -> pd.DataFrame:
    lo, hi = df[col].quantile([low_q, high_q])
    mask = df[col].between(lo, hi)
    n_drop = (~mask).sum()
    log.info('outliers.col=%s low_q=%.4f high_q=%.4f dropped=%d', col, lo, hi, n_drop)
    # IMPORTANT: write the dropped rows somewhere for audit
    df.loc[~mask].to_parquet(f'audit/outliers_{col}.parquet', index=False)
    return df.loc[mask]

Exercise

Pick any DataFrame with at least one column that has >5% nulls. For that column, write a one-paragraph note explaining what the nulls represent (look at the rows, not just the count). Then pick the right strategy from the four above — and be honest if the answer is "go ask the upstream owner because I don't know."