Batch vs Streaming — The Real Tradeoff

The discourse vs the reality

Conference talks make it sound like everyone is moving to streaming. The reality on the ground in 2026 is that most data engineering remains batch, and the ratio is something like 80/20 in favor of batch even at sophisticated companies. Here's how to think about which one you actually need.

What batch buys you

Simpler mental model — "every day at 3 AM, the pipeline runs."
Cheaper compute — you can size the cluster for the daily peak instead of the running 24/7 worst case.
Easier debugging — there are discrete runs, with logs, that either succeeded or didn't.
Easier backfills — "rerun for last week" is meaningful.

What streaming buys you

Lower latency — events flow continuously, downstream consumers see them within seconds/minutes.
Always-on aggregations — useful for fraud detection, real-time dashboards, online ML feature serving.
Decoupled producers/consumers — Kafka/Pulsar acts as a bus.

The right question

"What's the freshness SLA the consumer needs?" If the answer is "daily is fine," you're batch. If "five minutes is fine," you're micro-batch (run a batch job every 5 minutes). If "sub-second," you're streaming. Most consumers say "daily." Streaming has real benefits when you actually need it — and a real cost (operational complexity, cost, debugging difficulty) when you don't.

Code

Micro-batch — the practical middle ground for most "streaming" needs·python

# Schedule this with cron / Airflow / Dagster every 5 minutes.
# Reads everything since the last watermark, writes to a partitioned target.
import datetime as dt
from pathlib import Path
import pandas as pd

def micro_batch_run(window_minutes: int = 5) -> None:
    now = dt.datetime.utcnow()
    since = now - dt.timedelta(minutes=window_minutes)
    bucket = now.strftime('%Y/%m/%d/%H%M')

    df = fetch_events(since=since.isoformat(), until=now.isoformat())
    if df.empty:
        return

    target = Path(f'warehouse/events/{bucket}')
    target.mkdir(parents=True, exist_ok=True)
    df.to_parquet(target / 'part-0.parquet', index=False)

# Run this on a 5-minute cron and you have practical near-real-time without Kafka.

Exercise

For three real consumers of data in your environment (a dashboard, an ML model, a finance report), write down the freshness SLA each one actually needs (be honest — not what they wish for, what they need). Classify each as batch / micro-batch / streaming. Notice that the answer is almost always "batch is fine."