Great Expectations — Cross-Team Data Tests · Data Engineering Quest

The heavier-weight cousin of Pandera

Great Expectations (GX) is a more elaborate validation framework than Pandera. Where Pandera is a Python library you use inline, GX is a project: it has a persistent context with a config directory, expectation suites you save and version, a Data Docs renderer, and integrations with Airflow/dbt/Dagster. The shape is heavier; the payoff is that the expectations become a first-class, version-controlled, human-readable artifact that non-engineers can review.

When to pick GX over Pandera

You need data contracts that non-engineers (analysts, product, business stakeholders) can read and review.
You want the auto-generated data docs — a static site that shows expected schemas, recent run results, and drift over time.
You're already running Airflow/Dagster and want validation as a first-class operator/asset.
You want to validate data at rest — running expectations against a Snowflake table on a schedule, not just inline against a DataFrame.

When Pandera is enough

If your pipeline lives in Python, your validation stays inline, and your team is the only audience — Pandera is the lighter answer. The decision tree: Pandera by default; GX when expectations need to leave the codebase.

Code

Minimal Great Expectations validation in code·python

import great_expectations as gx
import great_expectations.expectations as gxe

context = gx.get_context()

# 1. Data source -> asset -> batch definition (GX 1.x wiring)
ds        = context.data_sources.add_pandas('orders_ds')
asset     = ds.add_dataframe_asset('orders')
batch_def = asset.add_batch_definition_whole_dataframe('batch')

# 2. The suite IS the artifact — this is the thing you version and review
suite = context.suites.add(gx.ExpectationSuite(name='orders.suite'))
suite.add_expectation(gxe.ExpectColumnValuesToNotBeNull(column='order_id'))
suite.add_expectation(gxe.ExpectColumnValuesToBeUnique(column='order_id'))
suite.add_expectation(gxe.ExpectColumnValuesToMatchRegex(
    column='order_id', regex=r'^O\d{6}$'))
suite.add_expectation(gxe.ExpectColumnValuesToBeBetween(
    column='amount_usd', min_value=0, max_value=1_000_000))
suite.add_expectation(gxe.ExpectColumnValuesToBeInSet(
    column='status', value_set=['pending', 'completed', 'cancelled']))

# 3. Bind suite to data, then run — the DataFrame arrives as a batch parameter
validation = context.validation_definitions.add(
    gx.ValidationDefinition(name='orders.vd', data=batch_def, suite=suite))
result = validation.run(batch_parameters={'dataframe': raw_df})

if not result.success:
    raise RuntimeError('orders failed expectations')

Exercise

If you have time and motivation: pip install great_expectations in a fresh project and run the snippet above against a small DataFrame with a deliberate violation in it — a duplicate id, a negative amount. Watch result.success come back False and read which expectation failed. Then build the Data Docs site and open it in a browser. The point of the exercise is to feel the weight difference vs Pandera so you can pick on purpose.

Great Expectations — Cross-Team Data Tests