Data Validation

~11 min · data, validation, quality

Level 0Apprentice

0 XP0/101 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Data is a build artifact too

If your service relies on a dataset (training data, RAG corpus, knowledge base), the data is part of what gets shipped. Validate it in CI like you'd validate code.

What to check

Schema — every row has the expected columns / fields with the expected types.
Cardinality — row count is within expected bounds (no silent data loss).
Nulls / completeness — required fields are populated.
Distribution — no big shift in basic statistics (mean, label balance) compared to a baseline.
PII / sensitive data — no leaked emails, phone numbers, API keys.
Encoding / format — UTF-8, no BOM in CSVs, no mixed line endings.

Tools

Great Expectations — declarative expectations, full reports.
Pandera — schema validation as Python decorators.
dbt tests — for SQL-based pipelines.
Custom pytest — for small datasets.

Code

Pandera schema check in CI·python

# tests/test_data_validity.py
import pandera.pandas as pa
import pandas as pd

schema = pa.DataFrameSchema({
    'id': pa.Column(int, unique=True),
    'text': pa.Column(str, checks=pa.Check.str_length(min_value=1)),
    'label': pa.Column(str, checks=pa.Check.isin(['pos', 'neg', 'neu'])),
    'lang': pa.Column(str, checks=pa.Check.str_matches(r'^(en|ko|ja|zh)$')),
})

def test_dataset_conforms():
    df = pd.read_parquet('data/training.parquet')
    schema.validate(df, lazy=True)  # raises with all errors collected
    assert 1000 < len(df) < 1_000_000   # cardinality bounds

External links

Exercise

Add a data validation step to your CI for one dataset you ship with. Define schema, expected row-count bounds, and a PII check. Push and watch the validation either pass or surface a problem you didn't know about.