The DE Mental Model — Schemas, Lineage, Idempotency, Contracts

Four ideas separate a script that ran once from a pipeline that runs every night for two years. Memorize them now; you'll see them in every track that follows.

1. Schema

A schema is a declarative description of what your data is supposed to look like: column names, types, constraints ("this is unique," "this is non-negative," "this matches a regex"). Auto-detection is fine for exploration. For a pipeline that runs unattended, the schema must be explicit, version-controlled, and validated at every step where data crosses a tool boundary.

2. Lineage

Lineage answers the question "where did this number come from?" — at the table level, the column level, and ideally the row level. When the CFO asks why Q1 revenue moved by 0.3%, you need to be able to walk backwards: dashboard → final mart → intermediate join → source extract. Modern tools (dbt, Dagster, OpenLineage) can produce this graph automatically if you let them.

3. Idempotency

An idempotent pipeline produces the same result whether you run it once, twice, or twenty times for the same input window. This sounds trivial; it isn't. Naive append-style writes double-count rows on retry. Naive incremental loads get out of sync after backfills. The discipline is to design your write step so that re-running a partition replaces it cleanly. Once you internalize this, debugging stops being terrifying — "just rerun it" becomes a real answer.

4. Contracts

A data contract is the shared, durable agreement between the team that produces data and the team that consumes it: schema, freshness, ownership, breaking-change policy. Without a contract, every upstream change is an outage waiting to happen. With one, the producer knows what they're not allowed to break, and the consumer knows what they can rely on.

How they fit together

Schemas are local — at every step, this is what should be true. Lineage is global — across all steps, here is how data flowed. Idempotency is temporal — across reruns, the result is stable. Contracts are social — across teams, here is what we promise. A serious pipeline has all four; a one-off script has none.

Code

Idempotent partition write — the canonical pattern·python

from pathlib import Path
import shutil
import pandas as pd

def write_partition(df: pd.DataFrame, root: Path, partition: str) -> Path:
    '''Replace a partition atomically. Rerunning is safe — same result.'''
    target = root / f'date={partition}'
    staging = root / f'.tmp_{partition}'

    # 1. Write to a staging dir nobody reads from
    staging.mkdir(parents=True, exist_ok=True)
    df.to_parquet(staging / 'data.parquet', index=False)

    # 2. Atomically swap staging → target. If anything above failed, target is untouched.
    if target.exists():
        shutil.rmtree(target)
    staging.rename(target)
    return target

# Re-run for the same partition? Same result. No double-count, no orphaned rows.
write_partition(today_df, Path('warehouse/orders'), '2026-04-30')

The DE Mental Model — Schemas, Lineage, Idempotency, Contracts

1. Schema

2. Lineage

3. Idempotency

4. Contracts

How they fit together

Code

External links

Exercise

Progress

Comments 0