Anatomy of a Model Repository

The page is not the repo

A model's web page is marketing wrapped around a directory. The directory is the artifact. Learning to read that directory — before downloading anything — is the single cheapest skill in this quest: everything later (dtype forensics, provenance, digest verification) starts from knowing what files a well-formed repo contains and what each one is for.

A typical Transformers-style model repository on a git-based model hub contains some mix of the following:

config.json — the architecture declaration: model type, hidden sizes, layer counts, vocabulary size, and (notoriously) a dtype field. Loaders trust it; you should read it.
model.safetensors (or a shard set + model.safetensors.index.json) — the weights themselves, in the format that won (next lesson).
tokenizer.json / tokenizer_config.json / special_tokens_map.json / vocab files — the token pipeline. Models are useless without these; archives that copy only the tensor file discover this at load time, in the dark.
README.md — the model card: intended use, training summary, license declaration, and the machine-readable metadata block at the top (library_name, license tags, base-model pointers). This is both documentation and evidence.
generation_config.json — decoding defaults the maker recommends.
Optional artifacts — chat templates, processor configs for multimodal models, preprocessor configs, and occasionally legacy pytorch_model.bin files from the pickle era.

What the file list already tells you

Before any download, the file list answers three acquisition questions. What format generation is this? — safetensors only, mixed, or pickle-era; whether shards exist and how many. Is this repo whole? — a config without weights, or weights without a tokenizer, are red flags (or signs you are looking at a LoRA adapter repo, which is legitimate but different). How big will this be? — the API exposes per-file sizes; sum them and know before you commit the disk and the transfer.

The branch structure is part of the anatomy too. A repository is not one snapshot but a tree: main is whatever the author currently wants there, while other refs — conversion branches, legacy snapshots, PR heads — hold alternates. The commit sha of the ref you read is the revision you are actually acquiring; record it or you have acquired nothing addressable.

Read the repo before you read the page. The card tells you what the author believes; the file list tells you what actually shipped. When they disagree, believe the files.

A reading pass you can run now

The hub exposes a JSON API for exactly this reconnaissance: one request returns the metadata (current commit sha, tags, pipeline type), another lists the file tree with sizes. Make these two calls your first move on any candidate acquisition — they cost nothing and they discipline everything after.

Code

Reconnaissance before acquisition: metadata + file tree·bash

# 1) Repository metadata — current revision sha, tags, pipeline
curl -s https://huggingface.co/api/models/mistralai/Mistral-7B-Instruct-v0.2 \
  | python3 -m json.tool | grep -E '"sha"|"pipeline_tag"|"tags"' | head

# 2) The file tree with sizes (resolve endpoint, main branch)
curl -s "https://huggingface.co/api/models/mistralai/Mistral-7B-Instruct-v0.2/tree/main?recursive=true" \
  | python3 -c "
import json, sys
files = json.load(sys.stdin)
total = 0
for f in files:
    if f['type'] == 'file':
        total += f.get('size', 0)
        print(f\"{f['size']:>12,}  {f['path']}\")
print(f'{total:>12,}  TOTAL bytes ({total/1e9:.1f} GB)')"

# What you now know without downloading a byte:
#   format generation (safetensors vs bin), shard count,
#   tokenizer presence, total size, and the revision sha to pin.

Exercise

Pick a model you might acquire. Using only API calls (no download), produce a one-paragraph acquisition brief: current revision sha, complete file list with sizes, total bytes, format generation, and any file you would expect but do not see. End with the sentence you would write into your archive log.

Hint

The revision sha comes from the metadata call; the file tree from the tree endpoint. Missing-file candidates: README.md (no card), tokenizer files, index for a shard set.

The page is not the repo

What the file list already tells you

A reading pass you can run now

Code

External links

Exercise

Progress

Comments 0