Caching, push_to_hub, and Reproducibility

~24 min · datasets, hub, cache

Level 0Scout

0 XP0/50 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Where the bytes go (datasets edition)

Default cache: ~/.cache/huggingface/datasets. Override with HF_DATASETS_CACHE or the umbrella HF_HOME. The library writes Arrow files keyed by (dataset id, config name, processing fingerprint). The fingerprint includes the function source for any map/filter — same input + same function = cache hit.

Saving processed datasets

After heavy preprocessing you usually want to save_to_disk(path) the result and load_from_disk(path) next run, bypassing the loader entirely. Reproducibility is then a tarball, not a recipe.

push_to_hub with version pinning

ds.push_to_hub("yourname/my-set", commit_message="initial", private=True) uploads as Parquet. The repo gets a Dataset Viewer for free. Pin SHA the same way you would for a model: load_dataset(..., revision="abc123def").

Code

save_to_disk + load_from_disk·python

from datasets import load_dataset

ds = load_dataset("stanfordnlp/imdb")
small = ds["train"].select(range(2000))

# Bake the result
small.save_to_disk("/tmp/imdb-2k")

# Reload — no internet required
from datasets import load_from_disk
ds2 = load_from_disk("/tmp/imdb-2k")
print(len(ds2), ds2[0])

push_to_hub a processed dataset·python

ds_processed = ds["train"].map(lambda ex: {"text_len": len(ex["text"])}, num_proc=4)

ds_processed.push_to_hub(
    "yourname/imdb-with-length",
    commit_message="add text_len",
    private=True,
)

# Anyone with read access can now:
# load_dataset("yourname/imdb-with-length", revision="<sha>")

External links

Exercise

Process a small public dataset (filter, map, select). save_to_disk it. load_from_disk to verify. Then push_to_hub to a private repo. Verify the repo's viewer shows the new columns. Pin a SHA and load_dataset from the repo using the SHA.