Why You Need an Orchestrator (Eventually)

The cron-and-bash phase is fine — until it isn't

Every data team starts the same way: a Python script, a cron entry, a Slack notification on failure. That stack works for one or two pipelines. It breaks the moment you have ten, because cron has none of the things you need: dependency ordering, retries, backfills, observability, structured failure handling, scheduling visibility, run history. You either build all of that yourself or you adopt an orchestrator that already has it.

What every modern orchestrator gives you

DAG / dependency graph — "task B runs after task A succeeds."
Scheduling — cron-style or interval-based, with timezone-aware semantics.
Retries — automatic, configurable, idempotent.
Backfills — "run last week again, in order."
Run history — every execution recorded with status, duration, logs.
UI — see what's running, drill into failures, kick off manual reruns.
Alerting — Slack/PagerDuty integrations on failure, SLA misses, etc.

The three contenders in 2026

Apache Airflow — most-deployed, biggest ecosystem, oldest of the three. Mature, opinionated, requires a real deployment (database, scheduler, webserver, workers). Best when you have lots of operators talking to many systems.

Dagster — asset-first model. You declare data assets and Dagster figures out the dependency graph. Best when your team thinks in terms of "the customers table" rather than "the customers ETL task."

Prefect — most Pythonic. Flows are decorated functions. Lightest deployment story (Cloud option, or self-host). Best when you want orchestration that feels like a library, not a platform.

Code

What "cron-and-bash" looks like — and why it doesn't scale·bash

# crontab -e
# 0 3 * * * /usr/bin/env python /opt/pipelines/orders.py >>/var/log/orders.log 2>&1
# 0 4 * * * /usr/bin/env python /opt/pipelines/customers.py >>/var/log/customers.log 2>&1
# 0 5 * * * /usr/bin/env python /opt/pipelines/revenue_dashboard.py >>/var/log/revenue.log 2>&1

# Problems with this once you have 10+ pipelines:
#  - revenue_dashboard runs at 5 AM whether or not orders+customers succeeded
#  - no retry on transient API failures
#  - no backfill mechanism — if Tuesday's run fails, fixing it is manual
#  - no central place to see "what's running right now"
#  - alerts mean parsing log files in a shell loop

Exercise

Inventory the scheduled jobs in your environment (crontab, launchd, GitHub Actions cron, whatever). For each one, write down: (1) what it depends on, (2) what depends on it, (3) what happens when it fails. Most teams discover that they already have a fragile DAG written across multiple cron entries — and that's the actual case for adopting an orchestrator.