Large File Handling

~10 min · lfs, large-files, s3

Level 0Apprentice

0 XP0/101 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

Don't put model weights in Git

A 4 GB model checkpoint in Git is bad: clones take forever, push limits will reject single files >100 MB, and Git stores binaries inefficiently. Use one of the alternatives.

Options

Git LFS — files are kept in a separate LFS store, the repo tracks pointers. Free for the first GB; storage tier costs after.
Hugging Face Hub — purpose-built for model weights, free for public, paid for large private. Native to most ML pipelines.
Cloud object storage (S3, R2, GCS) — write your own download step in CI; cheapest at scale.
DVC — Git-like CLI on top of object storage; tracks data versions alongside code.

CI implications

Cache model weights aggressively. Pulling a 4 GB file each run is wasteful.
Use actions/cache with a key based on the model version, not the Git SHA.
If the model lives in HF Hub, use huggingface_hub.snapshot_download with a local cache path inside ~/.cache/huggingface — that path is what actions/cache targets.

Code

Cache HF model weights between runs·yaml

      - name: Cache HF model
        uses: actions/cache@v4
        with:
          path: ~/.cache/huggingface
          # Key on the model name + revision, not the Git SHA
          key: hf-${{ vars.MODEL_NAME }}-${{ vars.MODEL_REVISION }}
          restore-keys: |
            hf-${{ vars.MODEL_NAME }}-

      - name: Pre-download model
        env:
          HF_TOKEN: ${{ secrets.HF_TOKEN }}
        run: |
          uv run python -c "
          from huggingface_hub import snapshot_download
          snapshot_download('${{ vars.MODEL_NAME }}', revision='${{ vars.MODEL_REVISION }}')
          "

External links

Exercise

If your repo has any large file in Git history, audit how it got there. Migrate to LFS, HF Hub, or object storage based on usage pattern. Verify CI still works with the new download path + cache.