Schema Validation with Pandera

The lightest-weight validation that pays its keep

Pandera is a Pythonic schema validation library for Pandas (and Polars, and Modin, and Dask). The pitch: declare the schema you expect, call schema.validate(df), and either get a validated DataFrame back or get an exception with every violation listed in detail. It's the cheapest defensive measure that turns silent corruption into a loud, debuggable failure.

What you get

Type checks — column dtypes match the declared types, with coerce=True for automatic casting.
Constraint checks — unique, non-null, regex, range, custom predicates.
Lazy collection — schema.validate(df, lazy=True) collects every violation across the whole DataFrame instead of failing on the first one.
DataFrameModel classes — declarative schema definitions that look like Pydantic models.
Polars support — same schema, applied to a Polars DataFrame, since Pandera 0.20+.

Code

Pandera schema with the four most useful constraint types·python

import pandas as pd
import pandera.pandas as pa
from pandera.errors import SchemaErrors

schema = pa.DataFrameSchema({
    'order_id':    pa.Column(str, unique=True, regex=r'^O\d{6}$'),
    'customer_id': pa.Column(str, regex=r'^C\d{4,8}$'),
    'order_date':  pa.Column(pa.DateTime, coerce=True,
                              checks=pa.Check.greater_than('2020-01-01')),
    'amount_usd':  pa.Column(float, coerce=True,
                              checks=[pa.Check.ge(0), pa.Check.le(1_000_000)]),
    'status':      pa.Column(str, checks=pa.Check.isin(['pending', 'completed', 'cancelled'])),
    'country':     pa.Column(str, nullable=True, regex=r'^[A-Z]{2}$'),
})

try:
    clean = schema.validate(raw_df, lazy=True)        # collect all errors
except SchemaErrors as e:
    # e.failure_cases is a DataFrame of every failed row × failed check
    print(e.failure_cases.head(20))
    raise

DataFrameModel — declarative, IDE-friendly schemas·python

import pandera.pandas as pa
import pandas as pd

class OrderSchema(pa.DataFrameModel):
    order_id:    pa.typing.Series[str]   = pa.Field(unique=True, str_matches=r'^O\d{6}$')
    customer_id: pa.typing.Series[str]   = pa.Field(str_matches=r'^C\d{4,8}$')
    order_date:  pa.typing.Series[pd.Timestamp] = pa.Field(coerce=True)
    amount_usd:  pa.typing.Series[float] = pa.Field(coerce=True, ge=0, le=1_000_000)
    status:      pa.typing.Series[str]   = pa.Field(isin=['pending', 'completed', 'cancelled'])

    class Config:
        strict = True       # extra columns raise
        coerce = True       # auto-cast where possible

clean = OrderSchema.validate(raw_df, lazy=True)

Exercise

Define a Pandera DataFrameModel for any DataFrame you've worked with. Include at least one regex check, one range check, and one isin check. Run validation on real data with lazy=True. Look at e.failure_cases for everything that failed — even on data you thought was clean, you'll usually find at least one surprise.

The lightest-weight validation that pays its keep

What you get

Code

External links

Exercise

Progress

Comments 0