Downloads, Caching, and Pinning

Where the bytes go

Default cache root: ~/.cache/huggingface/hub on macOS/Linux, %USERPROFILE%\.cache\huggingface\hub on Windows. Override with HF_HOME (recommended — covers tokens, the hub cache, and the datasets cache in one shot) or HF_HUB_CACHE (just the model/Space cache).

Inside, each repo has a models--{org}--{name} directory containing blobs (content-addressed by hash), refs (branch/tag pointers), and snapshots (symlinked checkouts of a particular commit). Symlinks are why a 70B model on disk is only one copy of weights even if you've checked out three different commits.

Pin or pay later

Every loader (from_pretrained, snapshot_download, load_dataset) accepts a revision= argument: a branch, a tag, or a commit SHA. Without it you get whatever the default branch happens to be, which silently moves under you. The first time a model author force-pushes a tokenizer fix, your pipeline output changes character — with no error and no log line. Pin the SHA in production. Tags are convenient but mutable.

Air-gapped / offline

HF_HUB_OFFLINE=1 hard-fails any HTTP call. Combined with a pre-warmed cache, this is how you ship to environments without outbound internet. The whole point of the content-addressed cache is that air-gap mode just works as long as the blobs are there.

Code

Pinned snapshot download·python

from huggingface_hub import snapshot_download

# Pin a SHA (immutable). For tags, pass revision="v1.0".
local_path = snapshot_download(
    repo_id="meta-llama/Llama-3.1-8B-Instruct",
    revision="0e9e39f249a16976918f6564b8830bc894c89659",  # example SHA
    allow_patterns=["*.safetensors", "*.json", "tokenizer*"],
    ignore_patterns=["*.bin"],  # skip .bin if .safetensors is present
)
print("downloaded to:", local_path)

Cache layout and offline mode·bash

# Inspect the cache
huggingface-cli scan-cache

# Move the cache root before downloading
export HF_HOME=/big-disk/hf-home

# Air-gapped run
export HF_HUB_OFFLINE=1
python my_pipeline.py  # will hard-fail on any HTTP call

Exercise

Set HF_HOME=/tmp/hf-test and pre-download a small open model (e.g. prajjwal1/bert-tiny). Run huggingface-cli scan-cache and inspect the layout. Then set HF_HUB_OFFLINE=1 and load the model again with from_pretrained — it should succeed. Delete one snapshot dir, re-run offline, and see the failure.