Catching Type Drift Before It Bites

~10 min · validation, drift, monitoring

Level 0Curious Reader

0 XP0/47 lessons0/11 achievements

0/120 XP to next level120 XP to go0% complete

Type drift is the silent killer

The pipeline that worked yesterday breaks today, with no code change, because the upstream source quietly changed a type. Common patterns:

An ID column that was always numeric starts coming in as strings ('12345').
A date column changes format from YYYY-MM-DD to DD/MM/YYYY.
A previously-non-null column starts having nulls.
A free-text field that used to be plain text starts containing JSON.
A currency field changes from "123.45" to "$123.45" or "123,45".

Two-tier defense

At the gate. Schema validation (Pandera/GX) catches the drift the same day it happens, before downstream consumers see it.
At rest. A scheduled "profile diff" comparing this run's profile to last week's — surfaces drift as warnings even when validation hasn't been written for a particular constraint.

Code

Profile-diff: detect drift between two runs·python

import pandas as pd

def profile_diff(prev: pd.DataFrame, curr: pd.DataFrame, *, tol: float = 0.05) -> dict:
    '''Return a dict of columns whose null rate or unique count drifted beyond tolerance.'''
    drift = {}
    for col in set(prev.columns) & set(curr.columns):
        prev_null = prev[col].isna().mean()
        curr_null = curr[col].isna().mean()
        if abs(prev_null - curr_null) > tol:
            drift[col] = {'kind': 'null_rate',
                          'prev': round(prev_null, 4),
                          'curr': round(curr_null, 4)}
            continue
        prev_uniq = prev[col].nunique()
        curr_uniq = curr[col].nunique()
        if prev_uniq and abs(prev_uniq - curr_uniq) / prev_uniq > tol:
            drift[col] = {'kind': 'cardinality',
                          'prev': prev_uniq, 'curr': curr_uniq}
    # Schema diff (added/removed columns)
    drift['__added__']   = sorted(set(curr.columns) - set(prev.columns))
    drift['__removed__'] = sorted(set(prev.columns) - set(curr.columns))
    return drift

External links

Exercise

Take two snapshots of the same data source — yesterday's and today's. Run the profile_diff function and look at what's drifted. Pick one drift signal that would warrant a Slack alert (a column's null rate jumped from 0.5% to 8%, say) and design what the alert message would contain.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousData Profiling — Knowing Your Data Before You Validate It Next →Null and Outlier Handling — Strategies, Not Defaults

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.