LFS Pointers and Hub Digests

Why the repo you clone is not the repo you download

Git was built to track text. Multi-gigabyte tensors would make every clone and every diff catastrophic, so model hubs bolt on Git LFS: the repository proper stores a tiny pointer file for each large artifact, and the real bytes live in a content-addressed store behind the hub. When you clone, you get the pointers; when you download through the hub's file endpoints, you get the real files.

That architecture puts the digest exactly where an archivist needs it. A pointer file is three or four lines:

version https://git-lfs.github.com/spec/v1
oid sha256:<64 hex characters> — the SHA-256 of the real bytes
size <bytes> — the true size of the real file

The oid line is the reference side of your digest contract, published by the platform the moment the file was uploaded. It is what your post-download shasum -a 256 should be compared against — not something you take on faith from a README.

Getting the reference without cloning

You never need to clone a 14 GB repo to learn its digests. The hub's raw-file endpoint serves the pointer file itself (it is small — that is the point), and the API's file-tree listing carries the LFS digest and size per file. Either route gives you the verification targets in seconds, before you commit to any transfer. Note the size field too: it is your first, cheapest check — a download that finishes at the wrong size has already failed, no hashing required.

One subtlety worth internalizing: the digest is computed over the blob, independent of filename and path. Renaming a file or moving it between directories does not change its digest; editing one byte does. This is what lets you verify a copy under a different name in a different layout — identity lives in the bytes, not the path.

The pointer file is a free certificate. Every large file in an LFS-backed repo ships with a platform-published SHA-256 and true size. Collect them before downloading; verify against them after.

The workflow, assembled

Reconnaissance reads the pointers; acquisition moves the bytes; verification closes the loop: fetch the pointer (or API record), download the real file, recompute SHA-256 locally, compare, record both digests in the acquisition note. Three of those steps are commands you already know from this track; the lesson's contribution is only knowing where the reference comes from — which turns verification from a ritual into a check.

Code

Pull the reference digests, then verify against them·bash

REPO=<org>/<model>
F=model-00001-of-00003.safetensors

# 1) The pointer file itself — tiny, instant
curl -s "https://huggingface.co/$REPO/raw/main/$F"
# version https://git-lfs.github.com/spec/v1
# oid sha256:3f2a9b...        <- the reference digest
# size 4920320256             <- the true byte size

# 2) Or the whole tree with LFS digests via API:
curl -s "https://huggingface.co/api/models/$REPO/tree/main?recursive=true" \
  | python3 -c "
import json, sys
for f in json.load(sys.stdin):
    lfs = f.get('lfs') or {}
    if lfs: print(f['path'], lfs['oid'], lfs['size'])"

# 3) After download — the comparison the contract promised:
shasum -a 256 $F   # must equal the oid from step 1
stat -f%z $F       # must equal the size from step 1 (macOS; %s on Linux)

Exercise

For one repo you plan to acquire: pull the pointer files (or API LFS records) for every large file into a references file — path, oid, size, one line each. After any download from that repo, run the comparison. Report any file where local digest or size disagrees with the reference, and what you would do about it.

Hint

Keep the references file next to the acquisition — it is simultaneously the verification target and the provenance record. A disagreement means re-download first (transient corruption is more common than upstream tampering), then investigate if it repeats.

Why the repo you clone is not the repo you download

Getting the reference without cloning

The workflow, assembled

Code

External links

Exercise

Progress

Comments 0