The DE Mental Model — Schemas, Lineage, Idempotency, Contracts · Data Engineering Quest

Four ideas separate a script that ran once from a pipeline that runs every night for two years. Memorize them now; you'll see them in every track that follows.

1. Schema

A schema is a declarative description of what your data is supposed to look like: column names, types, constraints ("this is unique," "this is non-negative," "this matches a regex"). Auto-detection is fine for exploration. For a pipeline that runs unattended, the schema must be explicit, version-controlled, and validated at every step where data crosses a tool boundary.

2. Lineage

Lineage answers the question "where did this number come from?" — at the table level, the column level, and ideally the row level. When the CFO asks why Q1 revenue moved by 0.3%, you need to be able to walk backwards: dashboard → final mart → intermediate join → source extract. Modern tools (dbt, Dagster, OpenLineage) can produce this graph automatically if you let them.

3. Idempotency

An idempotent pipeline produces the same result whether you run it once, twice, or twenty times for the same input window. This sounds trivial; it isn't. Naive append-style writes double-count rows on retry. Naive incremental loads get out of sync after backfills. The discipline is to design your write step so that re-running a partition replaces it cleanly. Once you internalize this, debugging stops being terrifying — "just rerun it" becomes a real answer.

One word of care: people call the stage-and-swap pattern below atomic, and it isn't quite. A filesystem can rename a single path atomically; it cannot swap out a directory that already has contents in one step. What the pattern actually buys you is a rerun-safe write with a very short window where the partition is in flight — which is enough, as long as you order the steps so a crash inside that window is recoverable. Track 4 takes this apart properly.

4. Contracts

A data contract is the shared, durable agreement between the team that produces data and the team that consumes it: schema, freshness, ownership, breaking-change policy. Without a contract, every upstream change is an outage waiting to happen. With one, the producer knows what they're not allowed to break, and the consumer knows what they can rely on.

How they fit together

Schemas are local — at every step, this is what should be true. Lineage is global — across all steps, here is how data flowed. Idempotency is temporal — across reruns, the result is stable. Contracts are social — across teams, here is what we promise. A serious pipeline has all four; a one-off script has none.

Code

Idempotent partition write — the canonical pattern·python

from pathlib import Path
import shutil
import pandas as pd

def write_partition(df: pd.DataFrame, root: Path, partition: str) -> Path:
    '''Replace a partition. Rerunning is safe — same result.

    Not atomic, and the docstring should not claim it is: a rename onto a
    directory that already has contents raises OSError(ENOTEMPTY), so this
    kind of directory swap is always two steps. Order them so a crash
    between them is recoverable.
    '''
    target = root / f'date={partition}'
    staging = root / f'.tmp_{partition}'
    retired = root / f'.retired_{partition}'

    # 1. Write to a staging dir nobody reads from
    if staging.exists():
        shutil.rmtree(staging)
    staging.mkdir(parents=True)
    df.to_parquet(staging / 'data.parquet', index=False)

    # 2. Move the old partition aside, THEN move the new one in. Crash
    #    between these two renames and the old data still exists under
    #    .retired_* — you can put it back. Deleting first cannot be undone.
    if target.exists():
        if retired.exists():
            shutil.rmtree(retired)
        target.rename(retired)
    staging.rename(target)

    # 3. Only now is it safe to throw the old partition away
    if retired.exists():
        shutil.rmtree(retired)
    return target

# Re-run for the same partition? Same result. No double-count, no orphaned rows.
write_partition(today_df, Path('warehouse/orders'), '2026-04-30')

The DE Mental Model — Schemas, Lineage, Idempotency, Contracts

1. Schema

2. Lineage

3. Idempotency

4. Contracts

How they fit together

Code

External links

Exercise

Progress

Comments 0