Lesson 04 of 07 · published

Resumable by Record

~15 min · resumable-transfer, fsync, atomic-rename, append-only-log, concurrency

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

What "Resume" Has to Promise

Resuming sounds simple: remember how far you got, start from there. The hard part is the word remember. A record that says chunk 7 is done must never be wrong about it. If it claims bytes the disk does not hold, the resumed file has a hole that nothing will ever fetch again. If the source changed in between, the resumed file is two versions spliced together. Both failures produce a file of the right size that passes a size-and-date check and is quietly wrong.

Delta's answer is a partial: a file being received, plus a record of exactly which chunks of it are proven. Both directions use it. A pull lands in a partial on the pulling Mac. A push lands in one on the receiving Mac.

Disk First, Then the Record

Every chunk takes the same four steps, in this order:

Check. The chunk's length must be what its index implies, and its SHA-256 must equal the hash that travelled with it. A mismatch is refused right here.
Write at its offset. os.pwrite into the partial's data file, at index × 16 MiB. Chunks arrive out of order from parallel workers, and position is all that matters.
fsync. The bytes are on the disk before anything says they are.
Record. One line, "<index> <sha256>\n", appended to the partial's .chunks file.

A stop between steps 3 and 4 costs one chunk, fetched again on the next run. A stop in the middle of step 4 leaves a line that does not parse as an index plus 64 hex characters, and that line is no record either. The order means the record can lag behind the disk but never run ahead of it.

The Key Holds the Version

A partial is named by a hash of four things: the destination path, the file's size, the chunk size and the source's validator (its etag). If the source is edited between two attempts, its validator changes. The next run looks for a different partial, finds none, and starts over. The two versions are never mixed. If the source changes during a transfer, a chunk arrives with a different etag. The partial is discarded and the file fails with "changed during the transfer; run again".

Landing: Prove, Then Rename

When the last chunk is recorded, the partial is committed:

every chunk must be recorded, or the commit refuses with "N of M chunks still missing";
the recorded chunk hashes must equal the source's chunk list. If not, the partial is discarded. The whole-file SHA-256 is the fallback proof, and a file with neither proof is not landed at all;
mode and modification time are set on the data file;
os.replace moves it onto the destination name. On one file system that rename is atomic. A reader sees the old file or the new one, never half of each.

One subtle case: a source file rewritten in place with its size and time kept. The source's hash record is keyed by device, inode, size and modification time, so it can still describe the old bytes. If the chunks that arrived disagree with that record, the pulling side asks the source once more with fresh=1, and the source hashes the file again before the commit decides.

Two Bugs in the Record

The push that lost its own chunks. On 2026-11-09 a 40 MiB push of a model file from the service host to a laptop (Cradle delivering a model) failed at commit: "2 of 3 chunks still missing". All three chunks had arrived. A push sends its chunks as parallel requests, and each request had built its own Partial object with its own copy of the record. Each saved its own list, and the last save erased the others' chunks. Pulls never hit it, because one transfer shares one Partial, and every earlier big-file test had been a pull. Now every request for one partial shares one in-memory chunk list, under that partial's lock.

The record that cost more every chunk. The fix above first kept the whole list in one JSON header, re-read and rewritten on every chunk. The review on 2026-11-10 measured where that leads: at 65,536 chunks (a 1 TiB file), each chunk cost a 5 MB read and a 5 MB write, so the bookkeeping grew with the square of the chunk count. The header is now written once, and the chunk list is append-only, one line per chunk. Recording a chunk costs the same at the end of a 1 TiB file as at its start. A test asserts exactly that.

What Resuming Looks Like

A re-run of the same command plans again. Files whose size and modification time match are already in place. A partial's recorded chunks are counted as done, and only the missing ones are requested. The job says so: "resuming: N of M files already in place". In the measurements, a 1 GiB pull stopped at 560 MiB and run again finished the rest in 0.17 s.

An abandoned partial (never run again, or its source changed so the key moved on) is kept while anything touches it. After 14 days untouched it is swept, at start and a few times a day, and health shows how many partials are waiting and how many bytes they hold. The data file sits in Delta's own state folder when that is on the destination's volume, so a home folder never fills with stray halves.

Code

Delta's partial: check, write at the offset, fsync, then append one line (trimmed excerpt)·python

# delta/partial.py (trimmed excerpt)
class Partial:
    def __init__(self, state_root, dest, *, size, chunk, source_tag=""):
        self.key = hashlib.sha256(f"{dest}|{size}|{chunk}|{source_tag}".encode()).hexdigest()[:32]
        ...
        self.slot = _slot_for(self.key)   # one chunk list per partial, shared by every request
        self.lock = self.slot.lock

    def write_chunk(self, index: int, data, sha256: str | None) -> None:
        want = self.expected_length(index)
        if len(data) != want:
            raise Integrity(f"chunk {index} of {self.dest.name} is {len(data)} bytes, expected {want}")
        actual = hashlib.sha256(data).hexdigest()
        if sha256 and actual != sha256:
            raise Integrity(f"chunk {index} of {self.dest.name} failed its SHA-256 check")
        fd = os.open(str(self.data_path), os.O_WRONLY)
        try:
            view = memoryview(data)
            offset = index * self.chunk
            written = 0
            while written < want:
                written += os.pwrite(fd, view[written:], offset + written)
            os.fsync(fd)
        finally:
            os.close(fd)
        with self.lock:
            done = self._list()
            fd = os.open(str(self.log_path), os.O_WRONLY | os.O_APPEND | os.O_CREAT, 0o644)
            try:
                os.write(fd, f"{index} {actual}\n".encode())
            finally:
                os.close(fd)
            done.add(index)
            self.slot.shas[index] = actual

A tiny partial: stop after five chunks, resume, prove by the chunk list, rename (runnable)·python

import hashlib
import os
import tempfile

CHUNK = 4  # bytes per chunk, tiny so the demo is readable


class Partial:
    """A file being received: chunks land at their offset, then one line is appended."""

    def __init__(self, folder, dest, size):
        self.dest, self.size = dest, size
        self.data = os.path.join(folder, "x.part")
        self.log = os.path.join(folder, "x.chunks")
        if not os.path.exists(self.data):
            open(self.data, "wb").close()

    def chunks(self):
        return (self.size + CHUNK - 1) // CHUNK

    def recorded(self):
        """Index -> SHA-256 for every complete line; anything else is no record."""
        done = {}
        if os.path.exists(self.log):
            for line in open(self.log).read().splitlines():
                index, _, sha = line.partition(" ")
                if index.isdigit() and len(sha) == 64:
                    done[int(index)] = sha
        return done

    def write_chunk(self, index, data, sha):
        if hashlib.sha256(data).hexdigest() != sha:
            raise ValueError(f"chunk {index} failed its SHA-256 check")
        fd = os.open(self.data, os.O_WRONLY)
        try:
            os.pwrite(fd, data, index * CHUNK)
            os.fsync(fd)                      # on disk first ...
        finally:
            os.close(fd)
        with open(self.log, "a") as log:      # ... then recorded, one line
            log.write(f"{index} {sha}\n")

    def commit(self, source_chunk_shas):
        done = self.recorded()
        if len(done) != self.chunks():
            raise RuntimeError(f"{self.chunks() - len(done)} of {self.chunks()} chunks still missing")
        if [done[i] for i in range(self.chunks())] != source_chunk_shas:
            raise RuntimeError("chunk hashes differ from the source's; partial discarded")
        os.replace(self.data, self.dest)      # atomic: the old file or the new one, never half
        os.remove(self.log)


source = b"the family keeps its files on its own channel"
pieces = [source[i:i + CHUNK] for i in range(0, len(source), CHUNK)]
shas = [hashlib.sha256(p).hexdigest() for p in pieces]

with tempfile.TemporaryDirectory() as folder:
    dest = os.path.join(folder, "landed.txt")
    first = Partial(folder, dest, len(source))
    for i in range(5):                        # five chunks land, then the run stops
        first.write_chunk(i, pieces[i], shas[i])
    fd = os.open(first.data, os.O_WRONLY)     # chunk 5's bytes reach the disk,
    os.pwrite(fd, pieces[5], 5 * CHUNK)       # and the stop comes before its line
    os.close(fd)
    again = Partial(folder, dest, len(source))
    missing = [i for i in range(again.chunks()) if i not in again.recorded()]
    print(f"resuming: {again.chunks() - len(missing)} of {again.chunks()} chunks in place")
    for i in missing:
        again.write_chunk(i, pieces[i], shas[i])
    again.commit(shas)
    print(open(dest, "rb").read().decode())

External links

Exercise

Run the runnable block, then break it on purpose in two ways and see which guard, if any, catches each. First, swap the order inside write_chunk so the line is appended before the bytes are written, and stop the first run right after the append of chunk 5. Second, change one byte of source before the resumed run, as if the source had been edited between attempts. Describe what the commit says in each case, and what Delta's partial key would have done about the second one.

Hint

In the first case the commit passes and the landed file is wrong at chunk 5. The recorded hash came from the bytes that arrived, not from what is on disk, so the chunk-list proof never reads the disk; only a whole-file SHA-256 over the data file would notice. That is exactly why the order of the steps is the guard. In the second, Delta would never mix the versions: the source's etag is part of the partial's key, so the resumed run would look for a different partial and start fresh.