Tabular Data Intuition

~26 min · tabular, eda, pandas

Level 0Scout

0 XP0/48 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Look before you model

Every minute spent in pandas before training pays back tenfold. The goal of tabular EDA is not to produce a beautiful notebook; it is to discover the surprises that would otherwise hide inside the model: skewed distributions, sentinel values, mislabeled categories, time gaps, duplicated keys, and the few rows that contain most of the signal.

The first ten things to check

Row count and unique-key count.
Null fraction per column.
Cardinality of every categorical.
Numerical summary (min, p25, median, p75, max) for every numeric.
Sentinel values masquerading as numbers (-1, 999, 9999).
Class balance for the target.
Time range and gaps if there is a timestamp.
Duplicated rows.
Outliers in the target.
Top correlations with the target (linear and rank).

The rule of suspicion

If a single feature seems too predictive on its own, it is probably leakage. If the target is too clean, it was probably already curated by a downstream system. Be suspicious before you celebrate.

Code

First-ten EDA in one screen·python

import pandas as pd

df = pd.read_parquet("data/processed/train.parquet")
print(df.shape, df.duplicated().sum(), "duplicates")
print(df.isna().mean().sort_values(ascending=False).head(10))
print(df.select_dtypes("object").nunique().sort_values(ascending=False).head(10))
print(df["target"].value_counts(normalize=True))

Spot sentinel values that masquerade as numbers·python

for col in df.select_dtypes("number").columns:
    counts = df[col].value_counts().head(3)
    if counts.iloc[0] > 0.05 * len(df) and counts.index[0] in {-1, 0, 999, 9999}:
        print(f"⚠️  {col}: {counts.index[0]} appears in {counts.iloc[0]} rows")

External links

Exercise

Run the first-ten EDA list against your dataset. Write down three surprises you found. For each, decide whether it changes a feature, the target definition, or the labeling guide. Save the notebook in notebooks/01_eda.ipynb.