The Production Mindset — Rerunnable, Observable, Reviewable

The transition from "someone who can wrangle data" to "data engineer" happens the day you stop optimizing for does it work right now and start optimizing for does it survive Monday morning. Three properties separate the two:

Rerunnable

Anyone — you in six months, your colleague tomorrow, the on-call engineer at 3 AM — can run the pipeline from a clean checkout against any historical input window and get the same result. This means: no hard-coded paths, no implicit "only run on Tuesdays after the upstream feed," no "I always do step 4 manually." Configuration is explicit; inputs are addressable; the entry point is one command.

Observable

When the pipeline succeeds, you can prove it (row counts, freshness, distribution metrics, lineage). When it fails, you know which step, on which input, with what error, fast enough to act. Logs are structured, metrics are emitted, dashboards exist before the first incident — not after.

Reviewable

Every change goes through code review. The diff tells the reviewer what business meaning changed, not just what code changed. Tests demonstrate that the new behavior matches the new expectation. Migrations are explicit. The graveyard of "someone changed the SQL on the dashboard four years ago and nobody knows why" is paved with non-reviewable changes.

The cultural shift

This isn't a tooling choice — it's a posture. The exact same Pandas script can be rerunnable + observable + reviewable, or it can be a notebook in someone's home directory. The bytes on disk are similar; the surrounding discipline is what makes one a system and the other a fragile artifact. The good news: the discipline is learnable, and most of this quest is teaching it.

Code

A pipeline entry point that's rerunnable + observable·python

import argparse, logging, sys, time
from datetime import date
from pathlib import Path

log = logging.getLogger('orders_pipeline')
logging.basicConfig(level=logging.INFO,
    format='%(asctime)s %(levelname)s %(name)s %(message)s')

def run(window: str, root: Path) -> None:
    t0 = time.monotonic()
    log.info('start window=%s root=%s', window, root)

    df = extract(window, root)
    log.info('extract.rows=%d', len(df))

    clean = transform(df)
    log.info('transform.rows=%d', len(clean))

    written = load(clean, root, window)
    log.info('load.path=%s rows=%d', written, len(clean))

    log.info('done elapsed=%.2fs', time.monotonic() - t0)

if __name__ == '__main__':
    p = argparse.ArgumentParser()
    p.add_argument('--window', default=str(date.today().isoformat()))
    p.add_argument('--root',   default=Path('warehouse'), type=Path)
    args = p.parse_args()
    try:
        run(args.window, args.root)
    except Exception:
        log.exception('pipeline failed')
        sys.exit(1)

Exercise

Take any script you've written. Time-box yourself to 30 minutes and add three things: (1) a --window CLI argument so you can rerun it for any date, (2) structured log.info() calls at the start and end of each stage with row counts, (3) a non-zero exit code on failure. You won't reach "production" — but you'll feel the shift.