Apache Arrow — The Glue Format · Data Engineering Quest

An in-memory standard, not a file format

Apache Arrow is the language-independent columnar memory format that Pandas (3.0+), Polars, DuckDB, PyArrow, and many database client libraries use internally. It's not a file format you write to disk for archival (though Arrow IPC files exist); it's the in-memory representation that makes data movement between tools cheap.

Why this matters in practice

The old world: a Pandas DataFrame in memory was bytes laid out one way; a Spark DataFrame was laid out another way; a PostgreSQL row was a third way. Moving data between them meant serialize-deserialize, every time. Arrow flattens this: when both sides speak Arrow, they share the same memory buffers — no copy, no serialization, no format translation. pl.from_pandas(df), duckdb.sql(...).arrow(), arrow_table.to_pandas(zero_copy_only=True) all become microsecond operations instead of milliseconds.

When you'll touch Arrow directly

When you want a tool-neutral format on disk that's faster to load than Parquet (Arrow IPC files for short-term scratch).
When you ship data over the network with Arrow Flight instead of REST/JSON.
When you call pyarrow.parquet directly because pd.read_parquet doesn't expose a feature you need.

Most of the time, you don't import Arrow yourself. You just notice that picking Arrow-aware libraries makes the rest of the stack feel fast.

Code

Round-trip Pandas ↔ Arrow ↔ Polars without copies·python

import pandas as pd
import pyarrow as pa
import polars as pl

pdf = pd.DataFrame({'order_id': ['A1', 'A2'], 'amount': [100.0, 250.0]})

# Pandas → Arrow (zero-copy when dtypes are Arrow-backed already)
tbl = pa.Table.from_pandas(pdf)

# Arrow → Polars (zero-copy)
pl_df = pl.from_arrow(tbl)

# Polars → Arrow → Pandas
back_to_pandas = pl_df.to_arrow().to_pandas(types_mapper=pd.ArrowDtype)
print(back_to_pandas.dtypes)        # Arrow-backed dtypes preserved

Exercise

Build a 1M-row Pandas DataFrame. Convert it to Arrow with pa.Table.from_pandas. Print tbl.nbytes (Arrow's idea of the table size) and compare with pdf.memory_usage(deep=True).sum() (Pandas' idea). The closer they are, the more cleanly your Pandas DataFrame mapped to Arrow — that's a useful proxy for "is this DataFrame using PyArrow-backed dtypes already?"

Apache Arrow — The Glue Format