Resumable Transfers

Long transfers will be interrupted — plan for it

A multi-gigabyte pull at archival pacing takes hours. Across hours, the probability of an interruption converges toward certainty: laptops sleep, Wi-Fi changes towers, VPNs renegotiate, power blips, hubs reset connections. This is not bad luck — it is the expected operating condition of large-file work. The design question is what an interruption costs: a full restart (the naive policy) or a continuation (the resumable policy).

HTTP gives the mechanism: Range requests. A client that knows how many bytes it already holds can ask the server for only the remainder — Range: bytes=4920320256- — and append. Servers that store files statically (model hubs' CDN endpoints among them) support this. The client-side ritual that makes it safe:

Download to a .part (or similarly marked) filename — an incomplete file must never wear a complete file's name.
On resume, request exactly current-size onward and append.
Only when the byte count matches the expected total does the file graduate to its real name.
Then — and only then — the digest check.

That last ordering matters: resumption assembles the file, verification proves the assembly. A resumed file has been written by multiple sessions across hours; if any window's bytes were corrupted in transit, only the end-of-transfer digest catches it. Resume without verify is plumbing without inspection.

The tools you already have

curl has resume as a first-class flag (-C -: "continue where the file on disk ends"), which turns any scripted download into a resumable one. The official hub CLIs implement resume internally — another reason to prefer them for acquisition. rsync is the workhorse for the storage-side copies of the same files, and it brings its own block-level delta logic and partial-file handling.

What none of the tools can do is decide policy for you: the .part convention, the expected-size check before renaming, and the digest-after-assembly are the archivist's three rules, applied in that order, every time.

An incomplete file must never wear a complete file's name. The .part convention plus size-then-digest graduation is what lets a months-long acquisition campaign survive its interruptions without lying about what exists.

Pacing as part of resumability

There is a second, quieter reason archival transfers run slow on purpose: a saturated link for hours raises the interruption rate (router reboots under memory pressure, overheating adapters, family members rebooting things) and multiplies the cost of each restart. A paced transfer — deliberately under the link's ceiling — trades a little speed for a much lower expected number of interruptions. Patience is not just politeness to the source; it is resumability's ally.

Code

The resumable ritual with curl, end to end·bash

URL=https://huggingface.co/<org>/<model>/resolve/main/model-00001-of-00003.safetensors
EXPECTED=4920320256   # from the pointer file's size field
PART=model-00001.safetensors.part

resume_pull() {
  while true; do
    curl -L -C - --fail -o "$PART" "$URL" && break
    echo "interrupted at $(stat -f%z "$PART") bytes; retrying in 10s"; sleep 10
  done
}
resume_pull

# Graduate only on the right size, then verify:
ACTUAL=$(stat -f%z "$PART")
[ "$ACTUAL" = "$EXPECTED" ] && mv "$PART" model-00001.safetensors \
  || echo "SIZE MISMATCH: $ACTUAL != $EXPECTED -- do not rename"
shasum -a 256 model-00001.safetensors   # vs the pointer oid

# rsync for the storage-side copy (resume + verification built in):
# rsync -avh --partial --progress src/ archive/models/<model>/

Exercise

Build the resumable ritual for one real URL: write the resume_pull shell function (or your equivalent), run it on a moderately large file, and interrupt it deliberately at least twice (Ctrl-C mid-transfer). Verify that it completes, graduates on the right size, and passes the digest check. Note what the .part file's size showed after each interruption.

Hint

Pick a file whose expected size you know from its pointer (previous lesson). The interruption sizes tell you the resume offset is working — each restart should log a nonzero starting point.

Long transfers will be interrupted — plan for it

The tools you already have

Pacing as part of resumability

Code

External links

Exercise

Progress

Comments 0