Apache Airflow — DAGs, Operators, the Scheduler

The default for many production teams

Apache Airflow (2.10 as of April 2026) is the most-deployed orchestrator in the world. It originated at Airbnb in 2014, became an Apache top-level project in 2019, and powers production pipelines at Lyft, Stripe, Netflix, Spotify, and most every "data platform team" you've heard of. The model is the DAG — Directed Acyclic Graph — of tasks.

The three architectural pieces

Webserver / UI — see DAGs, runs, task instances, logs.
Scheduler — reads DAG definitions, decides what to run, queues tasks.
Workers / executor — actually run the tasks. Local executor for one machine; Celery or Kubernetes for clusters.

You also need a metadata database (Postgres in production). This is the friction Dagster and Prefect chip away at — Airflow has more deployment moving parts than its competitors, but the mature ecosystem (1,500+ provider operators, huge community) makes it worth it for many teams.

Code

An Airflow DAG with the modern @task decorator API·python

from datetime import datetime, timedelta
from airflow.decorators import dag, task

@dag(
    dag_id='orders_pipeline',
    schedule='0 3 * * *',
    start_date=datetime(2026, 4, 1),
    catchup=False,
    default_args={'retries': 3, 'retry_delay': timedelta(minutes=5)},
    tags=['orders', 'analytics'],
)
def orders_pipeline():

    @task
    def extract(ds: str) -> str:
        # ds is the logical date as YYYY-MM-DD
        path = f'raw/orders/{ds}.json'
        # ... actually fetch and write ...
        return path

    @task
    def transform(raw_path: str, ds: str) -> str:
        out_path = f'warehouse/orders/date={ds}/'
        # ... read raw_path, transform, write Parquet to out_path ...
        return out_path

    @task
    def validate(parquet_path: str) -> None:
        # ... pandera schema validation ...
        return None

    raw = extract('{{ ds }}')
    out = transform(raw, '{{ ds }}')
    validate(out)

orders_pipeline()

Exercise

If you have Docker available, run astro dev start from the Astronomer CLI to bring up a local Airflow. Write a 3-task DAG (extract → transform → validate) for any toy dataset. Trigger it from the UI, watch the task instances run, click into a failure to see the logs. The point is to feel the loop — most of Airflow's value is the visibility, not the scheduler.