The Modern Python Data Stack (April 2026)

One stack, glued by Apache Arrow

For most of the 2010s the Python data stack meant Pandas, sometimes NumPy, occasionally a database. That world is gone. The modern stack is multi-tool, polyglot, and held together by a single in-memory format: Apache Arrow. Knowing the cast is half of knowing what tool to reach for.

Tool	Version (Apr 2026)	Role
Python	3.13	The language. Free-threaded build is experimental but landing.
NumPy	2.4.4	Fast n-dimensional arrays, the numerical bedrock under Pandas/PyArrow/sklearn/PyTorch.
Pandas	3.0.2	Tabular data: cleaning, grouping, joining, reshaping. Copy-on-Write is now the default; PyArrow-backed strings are widely used.
Polars	1.39.3	Rust-backed DataFrame library. Eager and lazy APIs. Handles bigger-than-memory data Pandas would choke on.
PyArrow	23.0.1	Apache Arrow in Python. The columnar in-memory format that lets Pandas, Polars, DuckDB, and storage talk without copies.
DuckDB	1.5.2	In-process SQL engine. Queries Parquet/Arrow/CSV/Pandas/Polars directly. "SQLite for analytics."
Parquet	format spec	Columnar binary file format. Compressed, statistics-rich, column-pruned reads. Default for analytics-at-rest.
Pandera	0.24+	Schema validation that integrates with Pandas and Polars.
Great Expectations	1.x	Heavier-weight expectation framework, good for cross-team data contracts.
Apache Airflow	2.10	Most-deployed orchestrator. DAGs of operators, scheduler, web UI.
Dagster	1.10	Asset-first orchestrator. Models data assets, not just tasks.
Prefect	3.x	Pythonic flows. Lightweight, less infrastructure than Airflow.
dbt	1.9	SQL transformations as version-controlled code. Where modeling actually happens.
JupyterLab	4.5+	Interactive notebook IDE for exploration. Not a deployment target.

Arrow as the glue

Apache Arrow is a language-independent columnar memory format. The reason this matters in practice: when you read a Parquet file with PyArrow, hand the resulting Arrow Table to Pandas, query it with DuckDB, and pass the result to Polars — none of those steps copy the data. They share the same memory buffers. This is the difference between modern data work and the old serialize-deserialize-everywhere world.

You don't need to learn Arrow as a separate tool. You need to know that it's there, and that picking Arrow-aware libraries (Pandas 3.0, Polars, DuckDB, PyArrow) is what makes the stack feel fast.

Code

The whole stack in one snippet — Parquet → Arrow → DuckDB → Pandas, no copies·python

import pyarrow.parquet as pq
import duckdb

# Read a Parquet file as an Arrow Table — zero copy from disk
arrow_tbl = pq.read_table('orders.parquet')
print(f'Loaded {arrow_tbl.num_rows:,} rows, {arrow_tbl.num_columns} cols')

# Query that Arrow Table with SQL via DuckDB — also zero copy
result = duckdb.sql(
    '''
    SELECT date_trunc('month', order_date) AS month,
           SUM(amount_usd) AS revenue
    FROM arrow_tbl
    WHERE country = 'KR'
    GROUP BY 1
    ORDER BY 1
    '''
).to_df()                # convert only the result to a Pandas DataFrame

print(result.head())

Exercise

In a fresh Python REPL, install pandas, pyarrow, and duckdb. Create any small Pandas DataFrame, write it to a Parquet file with df.to_parquet('demo.parquet'), then run duckdb.sql('SELECT * FROM "demo.parquet" LIMIT 5').to_df(). You just queried a file with SQL without spinning up a database server. Notice that.