Chunks and Bundles

Files Are the Wrong Unit of Work

The natural way to write a copy tool is a loop over files: open one, send it, close it, next. It is simple and it is wrong at both ends of the size range. A single 40 GiB model file becomes one stream on one TCP connection. On a Thunderbolt bridge that can carry 28 Gbit/s, one Python stream leaves most of the cable idle. A folder of 4,000 small notes becomes 4,000 request round trips. On the tailnet or a hotel link, each one pays the latency again, and the latency, not the bandwidth, decides how long it takes.

Delta's transfer engine works in work units, not files. The planner sorts every file that has to move into one of two shapes, and a pool of workers drains a shared queue of units.

The Two Shapes

File size	Unit	Why
over 4 MiB	16 MiB chunks, each its own request	one big file can use every worker at once
4 MiB or less	bundles of up to 400 files or 16 MiB	thousands of files become a few requests

The number of workers depends on the route Track 3 chose: eight on a Thunderbolt bridge, six on the house LAN, three on the tailnet. The tailnet figure is deliberately small. A laptop away from home should not be flooded by its own transfer.

How the Queue Decides

Each worker asks for the next unit in a fixed order of preference:

A chunk of a file that is already open. Finishing what has started keeps the number of half-written files small.
The next big file to open, largest first, but only while fewer big files are open than there are workers. Opening means reading or creating its partial (pull) or asking the destination which chunks it already holds (push).
A bundle, a folder or a link.

The limit in step 2 came from a review. The first version opened every big file before the first chunk moved, each with its own thread asking the source for its hashes. A folder of a thousand such files became a thousand requests at once, and over a slow link, an hour of silence before any progress. Now a worker opens a file only when it is free to work on it.

Every Chunk Carries Its Own Proof

A pulled chunk is a ranged GET on the source's file with verify=1. The source reads the range and sends its SHA-256 in an X-Delta-SHA256 header beside the bytes. A pushed chunk is a PUT to the destination's upload endpoint with X-Delta-Chunk-SHA256. Either way, the receiving side hashes what arrived and refuses a mismatch before the chunk is recorded as done. The next lesson covers what happens when bytes stop arriving. Lesson 4 covers how a recorded chunk survives a restart.

While the chunks move, a pull also asks the source for the file's whole chunk list (hash?whole=0). The source hashes it on a few threads beside the transfer, or answers from its record if it hashed this version before. When the last chunk lands, the receiver compares its own list with the source's. Equal lists prove the file without reading it a second time. The whole-file SHA-256 is one more full pass, so it is computed only for those who ask for it (a pair's archive, sync --checksum).

Inside a Bundle

A bundle is one request whose body is a simple stream: for each file, one JSON header line (path, size, mtime_ns, mode, sha256, the source's etag), then exactly size bytes. The receiver reads a line, reads the bytes, checks the hash, and lands the file whole: a hidden temporary file beside the destination, fsync, mode and modification time set on the open descriptor, then a rename. A file the source cannot open becomes a header line with an error field instead of bytes. It is that file's failure, and the rest of the bundle still lands.

Two Smaller Savings

The plan measures the link only when it matters. For 64 MiB or more to send, the plan pulls a probe (32 MiB, or 8 MiB on the tailnet) to estimate the time, and keeps the figure for ten minutes. Smaller plans carry no estimate. Every plan used to pull 8 to 32 MiB through the link first, even a sync with a few kilobytes to send.
Only empty folders are asked for. A file landing makes the folders above it. A folder needs its own mkdir request only if nothing will land inside it. A tree of 50,000 folders used to be 50,000 requests, one after another, before its first file moved.

Code

Delta's work units: sizes, workers and the queue's order (trimmed excerpt)·python

# delta/serving.py and delta/transfer.py (trimmed excerpt)
MIB = 1024 * 1024
CHUNK = 16 * MIB
BUNDLE_FILE_MAX = 4 * MIB      # a file up to this size travels in a bundle, many to one request

WORKERS = {"thunderbolt": 8, "lan": 6, "tailnet": 3, "local": 4}
BUNDLE_FILES = 400          # small files per bundle request
BUNDLE_BYTES = 16 * MIB     # and their bytes: every worker holds one bundle in memory
ESTIMATE_FROM = 64 * MIB    # a plan measures the link only for a transfer at least this large


def take():
    """The next unit: a chunk of a file already open, then a big file to open
    while fewer than ``limit`` are, then a bundle, a folder or a link."""
    with turn:
        while not job.cancel.is_set():
            if chunks:
                unit = chunks.popleft()
            elif big and counts["open"] < limit:
                unit = ("file", big.popleft())
                counts["open"] += 1
            elif others:
                unit = others.popleft()
            elif counts["busy"] == 0:
                return None          # nothing queued and nobody who could queue more
            else:
                turn.wait(0.25)
                continue
            counts["busy"] += 1
            return unit
        return None

Split a file list into chunk units and bundles, then stream and verify one bundle (runnable)·python

import hashlib
import io
import json

MIB = 1024 * 1024
CHUNK = 16 * MIB
BUNDLE_FILE_MAX = 4 * MIB
BUNDLE_FILES, BUNDLE_BYTES = 400, 16 * MIB


def units(files):
    """files: {path: size}. Big files become chunk indexes; small ones are bundled."""
    out, batch, batch_bytes = [], [], 0
    for path, size in sorted(files.items(), key=lambda kv: -kv[1]):
        if size > BUNDLE_FILE_MAX:
            count = (size + CHUNK - 1) // CHUNK
            out.extend(("chunk", path, i) for i in range(count))
    for path, size in sorted(files.items()):
        if size > BUNDLE_FILE_MAX:
            continue
        if batch and (len(batch) >= BUNDLE_FILES or batch_bytes + size > BUNDLE_BYTES):
            out.append(("bundle", batch))
            batch, batch_bytes = [], 0
        batch.append(path)
        batch_bytes += size
    if batch:
        out.append(("bundle", batch))
    return out


def write_bundle(items):
    """items: {path: bytes}. One JSON header line per file, then its bytes."""
    stream = io.BytesIO()
    for path, data in items.items():
        head = {"path": path, "size": len(data), "sha256": hashlib.sha256(data).hexdigest()}
        stream.write(json.dumps(head).encode() + b"\n" + data)
    return stream.getvalue()


def read_bundle(body):
    stream, landed = io.BytesIO(body), {}
    while True:
        line = stream.readline()
        if not line:
            return landed
        head = json.loads(line)
        data = stream.read(head["size"])
        if len(data) != head["size"]:
            raise ValueError(f"the bundle ended inside {head['path']}")
        if hashlib.sha256(data).hexdigest() != head["sha256"]:
            raise ValueError(f"{head['path']} failed its SHA-256 check")
        landed[head["path"]] = data


plan = units({"model.bin": 40 * MIB, "notes/a.md": 2_000, "notes/b.md": 3_500, "photo.jpg": 3 * MIB})
for unit in plan:
    print(unit)

body = write_bundle({"notes/a.md": b"alpha\n", "notes/b.md": b"beta\n"})
print(sorted(read_bundle(body)))

Exercise

Take one real folder on your Mac (a project with a node_modules, or a photo library export) and feed its sizes to the units() function above with os.walk and os.path.getsize. Report three numbers: how many files, how many requests the chunk-and-bundle plan makes, and how many requests a one-file-per-request tool would make. Then change BUNDLE_FILES to 50 and to 2,000 and explain which limit (file count or bytes) closes each bundle for your folder.

Hint

Most source trees are dominated by files under a few kilobytes, so the 400-file limit closes nearly every bundle. A folder of camera RAW files is the opposite: the 16 MiB byte limit closes a bundle after three or four files, or the files are over 4 MiB and become chunks.