Custom Datasets from Local Files

~24 min · datasets, custom

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Built-in loaders cover most local cases

For local files you rarely need a loading script anymore. The named loaders "json", "csv", "parquet", "text", "imagefolder", "audiofolder" handle the common shapes. Pass data_files as a string, list, or split-keyed dict.

imagefolder / audiofolder conventions

imagefolder looks for {root}/{class_name}/{file} and infers labels from the folder names. Add a metadata.jsonl or metadata.csv alongside the files to attach extra columns (caption, bounding box, transcription).

Dataset.from_dict / from_list / from_pandas

For programmatic construction, Dataset.from_dict({...}) takes column lists; Dataset.from_list([{...}, ...]) takes row dicts; Dataset.from_pandas(df) wraps a DataFrame. All three return a single Dataset; combine via DatasetDict({"train": ..., "test": ...}).

Code

Local CSV with named splits·python

from datasets import load_dataset

# Treat each path as a split
data_files = {
    "train": "data/train.csv",
    "validation": "data/dev.csv",
    "test": "data/test.csv",
}
ds = load_dataset("csv", data_files=data_files)
print(ds)

Build a Dataset from in-memory rows·python

from datasets import Dataset, DatasetDict

rows = [
    {"prompt": "Hello", "completion": "Hi there"},
    {"prompt": "How are you?", "completion": "I'm doing well"},
]
train = Dataset.from_list(rows)

# Wrap as DatasetDict if you have multiple splits
ddict = DatasetDict({"train": train})

# Now usable like any Hub dataset
ddict.push_to_hub("yourname/scratch-prompts", private=True)

imagefolder convention·python

# Folder structure on disk:
# imgs/train/cat/001.jpg
# imgs/train/dog/001.jpg
# imgs/val/cat/001.jpg
# imgs/val/dog/001.jpg

from datasets import load_dataset
ds = load_dataset("imagefolder", data_dir="imgs")
print(ds["train"].features)  # {'image': Image(...), 'label': ClassLabel(...)}

External links

Exercise

Build a small chat dataset (10-20 prompt/completion pairs) as JSONL. Load it with load_dataset('json', data_files=...). Then push it to a private Hub repo via push_to_hub. Verify the dataset card shows the right columns in the viewer.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousStreaming Datasets and IterableDataset Next →Splitting, Combining, Concatenating

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.