Dataset vs DatasetDict, Features, and Schema

~24 min · datasets, schema

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Two classes, one mental model

Dataset is a single Arrow-backed table with named columns and rows. DatasetDict is just {split_name: Dataset} with convenience methods that broadcast operations to every split. Anything you can do on a single Dataset — map, filter, select, shuffle — you can do on a DatasetDict and it applies to all splits.

Features are the schema

ds.features is the schema dict. Each column has a Value, ClassLabel, Sequence, Audio, Image, or nested type. For classification, ClassLabel carries the label-int-to-name mapping — so ds.features['label'].int2str(1) yields 'positive'. Treat the features dict like a database schema: type-aware, queryable, and the source of truth for what the data means.

Code

Inspect the schema·python

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb")
print(ds["train"].features)
# {'text': Value(dtype='string'), 'label': ClassLabel(names=['neg', 'pos'])}

# Convert label int → name
print(ds["train"].features["label"].int2str(1))  # 'pos'

# Slice access
sample = ds["train"][:3]  # dict of column → list
print(sample["label"])

Operations broadcast on DatasetDict·python

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb")  # DatasetDict
small = ds.shuffle(seed=42).select(range(1000))  # NOTE: select is per-split, but shuffle works on DatasetDict
# ds.map / .filter / .remove_columns all broadcast similarly
small_filtered = ds.filter(lambda ex: len(ex["text"]) > 200)
print({k: len(v) for k, v in small_filtered.items()})

External links

Exercise

Load any classification dataset on the Hub. Print features, num_rows per split, and label distribution. Then create a stratified 80/20 train/dev split using train_test_split(stratify_by_column='label') and verify class balance is preserved.

Progress

Progress is local-only — sign in to sync across devices.

← Previousload_dataset(): The Universal Loader Next →Exploring with Pandas, Arrow, and the Viewer

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.