Data Profiling — Knowing Your Data Before You Validate It

You can't validate what you haven't profiled

Schema validation is most useful when you actually know the shape of your data. Profiling — automated exploratory analysis that produces summary statistics, distributions, missing-value patterns, correlations, and anomalies — is the upstream activity that makes good validation possible. Profile first, then write the schema based on what you saw.

The two flavors of profiling

Manual / quick — df.describe(), df.info(), df.isna().mean(), df['col'].value_counts(dropna=False). The everyday muscle memory.
Automated reports — ydata-profiling (formerly pandas-profiling) generates an interactive HTML report with type detection, distributions, correlations, missing-value patterns, and warnings. Useful for handing a profile to a colleague who hasn't seen the data.

Code

The everyday quick-profile snippet·python

import pandas as pd

def quick_profile(df: pd.DataFrame) -> None:
    print(f'shape: {df.shape}')
    print(f'memory: {df.memory_usage(deep=True).sum() / 1e6:.2f} MB')
    print()
    print('dtypes:')
    print(df.dtypes)
    print()
    print('missingness:')
    print((df.isna().mean().sort_values(ascending=False).head(10) * 100).round(2).to_string())
    print()
    print('cardinality (unique counts):')
    print(df.nunique().sort_values().head(10).to_string())
    print()
    print('numeric describe:')
    print(df.describe(include='number').T.head(10).to_string())

quick_profile(orders_df)

ydata-profiling — full HTML report with one call·python

from ydata_profiling import ProfileReport

profile = ProfileReport(orders_df,
                        title='Orders profile (2026-04-30)',
                        minimal=False,           # set True for big DataFrames
                        explorative=True)
profile.to_file('reports/orders_profile.html')

Exercise

Pick any DataFrame with at least 50,000 rows and 10 columns. Run the quick-profile snippet. Note three things that surprised you. Then run ydata-profiling and open the HTML report. Compare the two — quick-profile gives you the daily reflex; ydata gives you the deep dive when you need to share with a colleague.