Streaming Datasets and IterableDataset

~26 min · datasets, streaming

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

When streaming is the only option

For datasets larger than your disk — FineWeb, RedPajama, common-crawl-derived sets — streaming=True is mandatory. The returned IterableDataset never materializes; it pulls examples on demand from the underlying Parquet shards.

What changes

No len(), no ds[i]. Forward-only iteration.
map still works, but is lazy (transforms on iteration, not pre-baked).
filter works, but the rejection rate matters — if you filter 99%, you waste 99% of your I/O.
shuffle(buffer_size=N) uses a reservoir sample of size N, not a global shuffle.

Multi-shard sharding for distributed training

iter_dataset = ds.shard(num_shards=8, index=worker_id) hands different shards to different DataLoader workers. The distributed hint in load_dataset(..., streaming=True) handles this automatically when you wrap with PyTorch's DataLoader in a DDP setup.

Code

Streaming + map + shuffle·python

from datasets import load_dataset

ds = load_dataset(
    "HuggingFaceFW/fineweb",
    name="sample-10BT",
    streaming=True,
    split="train",
)

# Filter, then map, then shuffle (with a buffer)
ds = ds.filter(lambda ex: len(ex["text"]) > 500)
ds = ds.map(lambda ex: {"upper_first_word": ex["text"].split(" ")[0].upper()})
ds = ds.shuffle(buffer_size=10000, seed=42)

for i, ex in enumerate(ds):
    if i >= 3: break
    print(ex["upper_first_word"], ex["text"][:80])

Wrap streaming dataset for PyTorch DataLoader·python

from datasets import load_dataset
from torch.utils.data import DataLoader

ds = load_dataset("HuggingFaceFW/fineweb", name="sample-10BT", streaming=True, split="train")
ds = ds.with_format("torch")

loader = DataLoader(ds, batch_size=4, num_workers=2)
for i, batch in enumerate(loader):
    if i >= 2: break
    print(type(batch), list(batch.keys()))

External links

Exercise

Stream the first 1k rows of FineWeb's sample-10BT. Apply a filter that keeps only English-looking text (cheap heuristic — text.isascii() ratio), then shuffle with buffer_size=1000. Time the whole pipeline. Compare with downloading the full sample-10BT and doing the same offline.

Progress

Progress is local-only — sign in to sync across devices.

← Previousdataset.map(): Transformations that Cache Next →Custom Datasets from Local Files

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.