Choosing a Format for Production

The working decision tree

You've seen the players. Here's the decision tree I actually use, in the order it gets applied:

Will a downstream pipeline read it? → Parquet (partitioned, zstd). This is 90% of the answer.
Is the consumer a database that already has the data? → Don't write a file. Compute in the warehouse with SQL/dbt.
Is the consumer a human with Excel? → CSV at the boundary, with a sidecar schema. Don't make it a pipeline stage.
Is the data nested/log-shaped? → Newline-delimited JSON (NDJSON / JSONL). Compress with gzip or zstd.
Is this short-term scratch between notebook stages? → Arrow IPC (Feather). Fastest load, because it skips Parquet's decode step — not because it skips compression. Reach it through Pandas' df.to_feather(), which still defaults to an LZ4 frame (on one 400k-row frame: 6.5MB against 15.2MB uncompressed — your columns will give their own ratio), so you rarely want compression='uncompressed'. If you drop to raw PyArrow instead, read Track 3's Arrow lesson first — that path defaults the other way.
Is the data tiny and you want diffability? → CSV in version control. Past ~10MB, switch to Parquet.

What about HDF5, Avro, ORC, Iceberg, Delta Lake, Hudi?

Quick gloss:

HDF5 — scientific data, hierarchical, predates the modern stack. Still used in physics/genomics. Don't pick it for new analytics work.
Avro — row-oriented, common in Kafka pipelines. Fine for streams; Parquet for analytics-at-rest.
ORC — Hadoop-era columnar competitor to Parquet. If you're not already in the ORC world, pick Parquet.
Iceberg / Delta Lake / Hudi — table formats on top of Parquet. They add ACID transactions, schema evolution, time travel. Critical for warehouse-scale lakehouses; overkill for a single-machine pipeline. Learn them when you outgrow the file-tree approach.

Code

A pipeline-friendly write helper that picks the right format by purpose·python

from pathlib import Path
import pandas as pd

def write(df: pd.DataFrame, path: Path, *, purpose: str) -> Path:
    '''Pick a format based on what the data is for.'''
    if purpose == 'pipeline':
        # Default analytics format
        path = path.with_suffix('.parquet')
        df.to_parquet(path, index=False, compression='zstd')
    elif purpose == 'human':
        # Excel/CSV boundary — humans look at this
        path = path.with_suffix('.csv')
        df.to_csv(path, index=False, encoding='utf-8', lineterminator='\n')
    elif purpose == 'scratch':
        # Notebook scratch cache — fastest load. Pandas' wrapper keeps the
        # LZ4 default; raw pyarrow.ipc does not (see the Arrow lesson).
        path = path.with_suffix('.feather')
        df.to_feather(path)
    elif purpose == 'log':
        # Append-only newline-delimited JSON
        path = path.with_suffix('.jsonl')
        df.to_json(path, orient='records', lines=True)
    else:
        raise ValueError(f'unknown purpose {purpose!r}')
    return path

Exercise

Look at the data your team produces today. For each output, classify the purpose (pipeline, human, scratch, log) and the current format. Note one mismatch — somewhere a CSV is being used for a pipeline stage, or a Parquet is being shipped to a human who only opens Excel — and propose the right format swap.

The working decision tree

What about HDF5, Avro, ORC, Iceberg, Delta Lake, Hudi?

Code

External links

Exercise

Progress

Comments 0