One stack, glued by Apache Arrow
For most of the 2010s the Python data stack meant Pandas, sometimes NumPy, occasionally a database. That world is gone. The modern stack is multi-tool, polyglot, and held together by a single in-memory format: Apache Arrow. Knowing the cast is half of knowing what tool to reach for.
| Tool | Version (Aug 2026) | Role |
|---|---|---|
| Python | 3.14 | The language. Free-threaded builds are officially supported as of 3.14 (PEP 779) — no longer experimental. |
| NumPy | 2.5.1 | Fast n-dimensional arrays, the numerical bedrock under Pandas/PyArrow/sklearn/PyTorch. |
| Pandas | 3.0.5 | Tabular data: cleaning, grouping, joining, reshaping. Copy-on-Write is now the default; PyArrow-backed strings are widely used. |
| Polars | 1.43.1 | Rust-backed DataFrame library. Eager and lazy APIs. Handles bigger-than-memory data Pandas would choke on. |
| PyArrow | 25.0.0 | Apache Arrow in Python. The columnar in-memory format that lets Pandas, Polars, DuckDB, and storage talk without copies. |
| DuckDB | 1.5.5 | In-process SQL engine. Queries Parquet/Arrow/CSV/Pandas/Polars directly. "SQLite for analytics." |
| Parquet | format spec | Columnar binary file format. Compressed, statistics-rich, column-pruned reads. Default for analytics-at-rest. |
| Pandera | 0.32.1 | Schema validation that integrates with Pandas and Polars. |
| Great Expectations | 1.19.1 | Heavier-weight expectation framework, good for cross-team data contracts. |
| Apache Airflow | 3.3.0 | Most-deployed orchestrator. Tasks in a DAG, plus a scheduler, a DAG processor, and an API server. Airflow 3 made scheduling asset- and event-aware. |
| Dagster | 1.13.16 | Asset-first orchestrator. Models data assets, not just tasks. |
| Prefect | 3.8.1 | Pythonic flows. Lightweight, less infrastructure than Airflow. |
| dbt | 1.12.0 | SQL transformations as version-controlled code. Where modeling actually happens. dbt v2, on the Fusion engine, is in preview alongside it. |
| JupyterLab | 4.6.2 | Interactive notebook IDE for exploration. Not a deployment target. |
Arrow as the glue
Apache Arrow is a language-independent columnar memory format. The reason this matters in practice: when you read a Parquet file with PyArrow, hand the resulting Arrow Table to Pandas, query it with DuckDB, and pass the result to Polars — none of those steps copy the data. They share the same memory buffers. This is the difference between modern data work and the old serialize-deserialize-everywhere world.
You don't need to learn Arrow as a separate tool. You need to know that it's there, and that picking Arrow-aware libraries (Pandas 3.0, Polars, DuckDB, PyArrow) is what makes the stack feel fast.