Hugging Face Datasets

One library, ten thousand datasets

datasets (the Hugging Face library) gives you a uniform API to load any dataset on the Hugging Face Hub — and many local formats too. Two big features make it special:

Apache Arrow backing — datasets are memory-mapped from disk. Loading a 100GB dataset uses near-zero RAM; you read what you iterate.
Streaming mode — for datasets too large to download, set streaming=True and iterate without downloading the full file.

The standard workflow

load_dataset("name") — fetch (or stream) and cache.
.map(fn, batched=True) — apply preprocessing in parallel, results cached on disk.
.set_format("torch", columns=[...]) — make it return PyTorch tensors directly when iterated, no manual conversion.
Pass it to a DataLoader like any other Dataset.

Why this matters even if you have your own data

Even for local data, the Datasets library gives you: parallel preprocessing with disk-backed caching, lossless format conversion (CSV/JSON/Parquet/Arrow), and trivial train/val/test split management. For NLP and vision projects of any reasonable size, it's worth using as your data layer.

Code

Loading from the Hub·python

# pip install datasets
from datasets import load_dataset

# Load IMDB sentiment dataset
ds = load_dataset("imdb")
print(ds)
# DatasetDict({
#     train: Dataset({features: ['text', 'label'], num_rows: 25000})
#     test:  Dataset({features: ['text', 'label'], num_rows: 25000})
# })

train = ds['train']
print(train[0])    # {'text': '...', 'label': 1}

Preprocessing with .map — parallel and cached·python

from datasets import load_dataset
from transformers import AutoTokenizer

ds = load_dataset("imdb")
tok = AutoTokenizer.from_pretrained("distilbert-base-uncased")

def encode(batch):
    return tok(batch['text'], padding='max_length', truncation=True, max_length=512)

# .map runs in parallel by default; result is cached to disk
ds_enc = ds.map(encode, batched=True, num_proc=4)

# Tell datasets to return PyTorch tensors when iterated
ds_enc = ds_enc.with_format('torch', columns=['input_ids', 'attention_mask', 'label'])

print(ds_enc['train'][0])
# {'input_ids': tensor([...]), 'attention_mask': tensor([...]), 'label': tensor(1)}

Streaming for huge datasets·python

from datasets import load_dataset

# C4 is hundreds of GB. Stream it instead of downloading.
ds = load_dataset("c4", "en", split="train", streaming=True)

# Iterate without downloading the whole thing
for i, ex in enumerate(ds):
    if i >= 3: break
    print(ex['text'][:80])

# Streaming datasets are IterableDatasets — wrap with DataLoader
from torch.utils.data import DataLoader
loader = DataLoader(ds.with_format('torch'), batch_size=8)

Loading your own files·python

from datasets import load_dataset

# CSV
csv_ds = load_dataset("csv", data_files="my_data.csv")

# Parquet — fast columnar format, good for large tabular data
pq_ds = load_dataset("parquet", data_files="my_data.parquet")

# JSON Lines
jsonl_ds = load_dataset("json", data_files="my_data.jsonl")

# Even your own image folder, in ImageFolder layout
img_ds = load_dataset("imagefolder", data_dir="./images")

One library, ten thousand datasets

The standard workflow

Why this matters even if you have your own data

Code

External links

Exercise

Progress

Comments 0