Parquet — Columnar Binary, Why It Matters

The format that should be your default

Apache Parquet is a columnar binary file format originally from Hadoop, now ubiquitous across the analytics world. The file stores each column's data in a separate, compressed chunk, with rich statistics (min/max/null-count) and a footer that lets readers skip whole row groups based on filter predicates. The result: queries that read a few columns from a 100GB file touch only the bytes they need, and pay compression-aware cost.

What you get for free vs CSV

Types. Schema is part of the file. amount_usd is double, order_date is timestamp[us] — no guessing on read.
Compression. Snappy is the default; zstd typically gives 2–3× better ratios at similar speed. A 10GB CSV often becomes a 1–2GB Parquet.
Column pruning. Reading only order_date and amount_usd from a 50-column file skips the other 48.
Predicate pushdown. WHERE country = 'KR' can use row-group statistics to skip whole row groups whose country column never contained 'KR'.
Splittable. Multiple readers can process different row groups in parallel.

Partitioning — the next level

Parquet writers can split a logical "table" into a directory tree like orders/year=2026/month=04/data.parquet. Readers can prune entire partitions before opening any files. Combined with column pruning and predicate pushdown, this is how you get sub-second queries over hundreds of GB of data on a single machine with DuckDB.

Code

Writing partitioned Parquet — the production-default pattern·python

import pandas as pd

df = pd.read_csv('orders.csv', dtype=str)
df['order_date'] = pd.to_datetime(df['order_date'])
df['amount_usd'] = pd.to_numeric(df['amount_usd'])
df['year']  = df['order_date'].dt.year.astype('Int16')
df['month'] = df['order_date'].dt.month.astype('Int8')

df.to_parquet(
    'warehouse/orders',           # directory, not a single file
    partition_cols=['year', 'month'],
    compression='zstd',
    index=False,
)
# Result: warehouse/orders/year=2026/month=04/<uuid>.parquet

Reading with column pruning and partition filters·python

import pandas as pd

# Pandas + PyArrow auto-discovers the partition layout
df_apr = pd.read_parquet(
    'warehouse/orders',
    columns=['order_date', 'amount_usd', 'country'],   # only these columns are read
    filters=[('year', '=', 2026), ('month', '=', 4)],   # only this partition is scanned
)
print(f'rows: {len(df_apr):,}')

Parquet — Columnar Binary, Why It Matters

The format that should be your default

What you get for free vs CSV

Partitioning — the next level

Code

External links

Exercise

Progress

Comments 0