Apache Airflow — DAGs, Operators, the Scheduler · Data Engineering Quest

The default for many production teams

Apache Airflow (3.3 as of August 2026) is the most-deployed orchestrator in the world. It originated at Airbnb in 2014, became an Apache top-level project in 2019, and powers production pipelines at Lyft, Stripe, Netflix, Spotify, and most every "data platform team" you've heard of. The model is the DAG — Directed Acyclic Graph — of tasks.

Airflow 3 is not a version you can skip past

Airflow 3.0 landed in April 2025, and it invalidated two things at once: the DAG files people had been copying around, and the deployment diagram in every blog post written before it. If you learned Airflow from 2.x material — and most material still on the internet is 2.x — this is the section that matters.

DAG authoring moved to airflow.sdk. from airflow.decorators import dag, task still works and emits a deprecation warning; from airflow.sdk import dag, task is the stable interface now. Underneath, task execution became a client-server split, which is what makes non-Python task runtimes possible at all.
DAG versioning. A run finishes against the version of the DAG it started with. Editing a file mid-run no longer quietly rewrites what a running pipeline is doing — the single most-requested feature in the project's own survey.
Datasets became Assets, and scheduling became event-driven. A DAG can now wake because an asset changed somewhere outside Airflow, not only because the clock moved.
Removed outright: SubDAGs (use TaskGroups), SLAs (replaced by Deadline Alerts), the Sequential Executor, and the entire execution_date / tomorrow_ds / yesterday_ds family of context variables. logical_date is the survivor.
Defaults flipped: catchup_by_default is now False. Deploying a new DAG no longer sets off a stampede of historical runs the moment it is parsed.

The architectural pieces

API server — serves the REST API and the rebuilt React UI. This is what used to be called the webserver.
Scheduler — decides what is due and hands tasks to the executor.
DAG processor — parses your DAG files and serializes them into the database. In Airflow 3 this is a required standalone process, not an optional split you turn on at scale.
Metadata database — Postgres in production.
Workers, and a triggerer for deferred tasks — optional, depending on which executor you pick.

That is one more mandatory moving part than Airflow 2 had. This is the friction Dagster and Prefect chip away at, and Airflow 3 widened rather than narrowed it — the trade is the same shape, just steeper. Airflow costs more to stand up than either competitor, and the ecosystem (80+ community-maintained provider packages, and the largest community in the category) is what buys it back.

EV v2 · 2026-08-01: This lesson used to describe Airflow 2.10, and it should not have: Airflow 3.0 had already been generally available for a year when the lesson was first written. Everything above the fold has been re-truthed against the current docs — the required DAG processor, the API server that replaced the webserver, the airflow.sdk import path, the removals, and the flipped catchup default. If you completed this quest before today, the concrete thing to check is your own imports: a DAG that starts from airflow.decorators import dag, task is running on a deprecated path, and anything reading execution_date is running on one that is already gone.

Code

An Airflow 3 DAG — the @task API, imported from airflow.sdk·python

from datetime import datetime, timedelta
from airflow.sdk import dag, task

@dag(
    dag_id='orders_pipeline',
    schedule='0 3 * * *',
    start_date=datetime(2026, 8, 1),
    catchup=False,          # the default in Airflow 3; still worth saying out loud
    default_args={'retries': 3, 'retry_delay': timedelta(minutes=5)},
    tags=['orders', 'analytics'],
)
def orders_pipeline():

    @task
    def extract(ds: str) -> str:
        # ds is the logical date as YYYY-MM-DD. It exists only on runs that
        # HAVE a logical date, and in Airflow 3 logical_date can be None —
        # asset-triggered and manually-triggered runs are the cases to expect.
        path = f'raw/orders/{ds}.json'
        # ... actually fetch and write ...
        return path

    @task
    def transform(raw_path: str, ds: str) -> str:
        out_path = f'warehouse/orders/date={ds}/'
        # ... read raw_path, transform, write Parquet to out_path ...
        return out_path

    @task
    def validate(parquet_path: str) -> None:
        # ... pandera schema validation ...
        return None

    raw = extract('{{ ds }}')
    out = transform(raw, '{{ ds }}')
    validate(out)

orders_pipeline()

Exercise

If you have Docker available, run astro dev start from the Astronomer CLI to bring up a local Airflow. Write a 3-task DAG (extract → transform → validate) for any toy dataset, importing from airflow.sdk. Trigger it from the UI, watch the task instances run, click into a failure to see the logs. The point is to feel the loop — most of Airflow's value is the visibility, not the scheduler.

Apache Airflow — DAGs, Operators, the Scheduler