Tabular Data Intuition

Look before you model

Every minute spent in pandas before training pays back tenfold. The goal of tabular EDA is not to produce a beautiful notebook; it is to discover the surprises that would otherwise hide inside the model: skewed distributions, sentinel values, mislabeled categories, time gaps, duplicated keys, and the few rows that contain most of the signal.

The first ten things to check

Row count and unique-key count.
Null fraction per column and by important group.
Cardinality and common values of every categorical.
Numerical summary (minimum, quartiles, median, maximum) for every numeric.
Sentinel values masquerading as numbers (-1, 999, 9999).
Class balance or target distribution.
Time range, timezone, and collection gaps.
Fully duplicated rows and duplicated entity keys.
Outliers in the inputs and target.
Top linear and rank correlations with the target.

Translate numbers back into reality

A median of 42 is meaningless until you know the unit, collection process, and time at which the value becomes available. Inspect raw rows behind extreme summaries. Confirm whether blank, unknown, zero, and not-applicable are distinct states rather than trusting a column name.

The rule of suspicion

If a single feature seems too predictive on its own, it is probably leakage. If the target is too clean, it was probably already curated by a downstream system. Trace the source and creation time of suspicious columns before you celebrate.

Close every finding with an action

Turn an unexpected value into a source fix, schema rule, transformation, exclusion decision, or documented limitation. EDA that remains a notebook annotation will not protect the next training run. Move stable checks into code and fail clearly when the data contract changes.

Code

First-ten EDA in one screen·python

import pandas as pd

df = pd.read_parquet("data/processed/train.parquet")
print(df.shape, df.duplicated().sum(), "duplicates")
print(df.isna().mean().sort_values(ascending=False).head(10))
print(df.select_dtypes("object").nunique().sort_values(ascending=False).head(10))
print(df["target"].value_counts(normalize=True))

Spot sentinel values that masquerade as numbers·python

for col in df.select_dtypes("number").columns:
    counts = df[col].value_counts().head(3)
    if counts.iloc[0] > 0.05 * len(df) and counts.index[0] in {-1, 0, 999, 9999}:
        print(f"⚠️  {col}: {counts.index[0]} appears in {counts.iloc[0]} rows")

Look before you model

The first ten things to check

Translate numbers back into reality

The rule of suspicion

Close every finding with an action

Code

External links

Exercise

Progress

Comments 0