load_dataset(): The Universal Loader

~26 min · datasets, load

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

One function, many sources

datasets.load_dataset() is the entry point for the entire Datasets library. It accepts a Hub dataset id, a local path, a directory of files, or one of HF's named loader scripts. Behind the scenes it figures out the format (Parquet, CSV, JSONL, TSV, audio folder, image folder, …), downloads to HF_DATASETS_CACHE, and returns a Dataset or DatasetDict.

The two return shapes

If the loader script defines splits (train/validation/test), you get a DatasetDict — a dict-of-Datasets keyed by split name.
If you ask for a specific split (split="train") or the source has only one, you get a single Dataset.

Streaming changes the contract

By default, load_dataset downloads the full dataset to disk and gives you a fully indexed array-like. With streaming=True it returns an IterableDataset: lazy, never written to disk, no random access. For multi-TB datasets that's the only sensible option.

Code

Three load patterns that cover 90%·python

from datasets import load_dataset

# 1. Hub dataset, all splits
ds = load_dataset("stanfordnlp/imdb")
print(ds)  # DatasetDict({train, test})
print(ds["train"][0])

# 2. Hub dataset, specific split
train = load_dataset("stanfordnlp/imdb", split="train")
print(len(train), train.features)

# 3. Local CSV / JSONL — points the loader at files directly
data_files = {"train": "data/train.csv", "test": "data/test.csv"}
ds_local = load_dataset("csv", data_files=data_files)
print(ds_local)

Streaming a too-big-to-disk dataset·python

from datasets import load_dataset

# 1.5T tokens, no full download
ds = load_dataset("HuggingFaceFW/fineweb", streaming=True, split="train")

# IterableDataset — only forward iteration
for i, ex in enumerate(ds):
    if i >= 3: break
    print(ex.keys(), ex["text"][:100])

External links

Exercise

Load stanfordnlp/imdb and inspect its features, num_rows per split, and a few random samples. Then load wikipedia with streaming=True and pull the first 5 examples without downloading the full dataset. Print HF_DATASETS_CACHE size before and after.

Progress

Progress is local-only — sign in to sync across devices.

← 🤖 Transformers LibraryQuiz · 5 questions Next →Dataset vs DatasetDict, Features, and Schema

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.