Lesson 05 of 07 · published

Never Done on Silence

~14 min · error-handling, retries, threads, invariants, http

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

The Worst Outcome Is "Done"

A transfer that fails loudly is an annoyance. You read the error and run it again. A transfer that says done while files are missing is a disaster you find weeks later, when the copy you trusted turns out to be incomplete. Delta's review on 2026-11-10 found exactly that path. One unreadable small file cut the source's bundle answer short. The worker thread that was reading it died on the cut. The job counted no errors, so at the end it reported done, with nothing landed.

The invariant that came out of it is short: a job never says done on silence. Every planned file either landed or is named in the job's errors. Making that true took four separate changes, because silence had four ways in.

1. A Worker Catches Everything

The dead worker had caught Delta's own DeltaError and OSError. The cut raised http.client.IncompleteRead, which is neither, so the exception ended the thread, and a thread that ends takes its unit of work with it. Now each worker catches three tiers and turns each into a line on the job:

Caught	Recorded as
`DeltaError`	`<unit>: <cause>: <detail>`, e.g. `timeout`, `integrity`, `conflict`
`OSError`	`<unit>: unreadable: <strerror> (<path>)`
anything else	`<unit>: internal: <type>: <message>`, with the traceback in the log

A bug in Delta is still a bug, but now it is a named failure on the job instead of a worker that quietly vanished.

2. One File's Error Is That File's

The source side changed too. When the bundle server cannot open one file, it now writes that file's header line with "error": "unreadable" and moves on to the next file. Before, the exception ended the response mid-stream and took every other file in the bundle with it. The push direction follows the same rule: a local file that will not open is recorded as its own error, and the rest of the bundle is still sent. The receiver also checks the other way round. A file it asked for that never appeared in the answer is failed by name: "the bundle did not carry it".

3. A Hiccup Is Tried Again, and Said So

On a weak link, every hiccup used to fail the file and end the job, to be run again by hand. Now a transient failure (cause unavailable or timeout) is tried again inside the job after 0.5, 2, 5 and 10 seconds. Every retry appends a note such as "retrying chunk 3 of model.bin in 2 s after: …" and increments a counter, and the CLI prints each note as it appears. Retrying is careful about what it repeats:

a retried pull bundle asks only for the files that have not landed yet;
a commit sent twice lands once and answers twice, so retrying the last step of a big file is safe;
a bundle that names unique or an expected version is never sent twice. The first send may have landed, and a second would take a second name or refuse itself.

Anything that is not transient (an integrity failure, a refusal, a conflict) is not retried. After the last delay, a transient failure becomes the file's error too.

4. The Final Count

The end of a job is now an audit, not a formality:

cancelled → interrupted, "run the same command again to resume";
any error → failed, "N problem(s), the rest landed; first: …";
no errors, but fewer files landed than planned → failed, "N of M file(s) neither landed nor reported an error";
only then done, with what moved, what was already in place, what was deleted, and any links left out or deletes that did not happen.

Step 3 found a bug the day it shipped. A sync counted the files it was about to delete among the files it should have in place, so any sync that deleted something could never reach its own total. Five delete tests failed on the lab Mac. The fix was a definition: a file to delete is not one of the files a job is to have in place. Deletes got the same honesty. One counts only when it happened, and one that did not is listed in not_deleted with its reason.

The Server's Half: Read Before You Refuse

One more silent failure lived on the server. When Delta refused a request (a forbidden path, a bad ticket) before reading its body, the unread body stayed in the connection. The next request on that keep-alive connection was parsed from the middle of it, and produced a 400 or 501 that nobody could explain. Now a refusal first drains the request body, up to 96 MiB (beyond that the connection is closed instead), and only then sends its answer. The caller reads the real reason, and the connection stays usable.

Code

Delta's job end: never done while a planned file is unaccounted for (trimmed excerpt)·python

# delta/transfer.py (trimmed excerpt)
RETRY_DELAYS = (0.5, 2.0, 5.0, 10.0)        # a transient failure is tried again after each, then named
TRANSIENT = ("unavailable", "timeout")


def worker():
    while True:
        unit = take()
        if unit is None:
            return
        try:
            do(unit, name)
        except DeltaError as exc:
            job.fail(f"{label_of(unit)}: {exc.cause}: {exc.detail}")
        except OSError as exc:
            job.fail(f"{label_of(unit)}: unreadable: {exc.strerror or exc}"
                     f"{' (' + str(exc.filename) + ')' if exc.filename else ''}")
        except Exception as exc:  # a bug is a named failure, never a worker that vanished
            log.exception("job %s: %s crashed", job.id, label_of(unit))
            job.fail(f"{label_of(unit)}: internal: {type(exc).__name__}: {exc}")


# at the end of _run
lost = job.files_total - job.files_done
if job.cancel.is_set():
    job.state = "interrupted"
elif job.errors:
    job.state = "failed"
    job.message = (f"{len(job.errors)} problem(s), the rest landed; first: {job.errors[0]}. "
                   "Run again to retry.")
elif lost > 0:
    # Never "done" on silence: every planned file either landed or said why not.
    job.state = "failed"
    job.errors.append(f"internal: {lost} of {job.files_total} file(s) neither landed nor reported "
                      "an error")
else:
    job.state = "done"

Retries with a schedule, three tiers of caught errors, and a final count (runnable)·python

import random
import time

RETRY_DELAYS = (0.05, 0.2, 0.5, 1.0)   # Delta's are 0.5, 2, 5 and 10 s; shortened to run fast
TRANSIENT = ("unavailable", "timeout")


class NamedError(Exception):
    def __init__(self, cause, detail):
        super().__init__(detail)
        self.cause, self.detail = cause, detail


def retrying(job, fn, what):
    delays = iter(RETRY_DELAYS)
    while True:
        try:
            return fn()
        except NamedError as exc:
            delay = next(delays, None)
            if exc.cause not in TRANSIENT or delay is None:
                raise
            job["notes"].append(f"retrying {what} in {delay:g} s after: {exc.detail}")
            time.sleep(delay)


def move(path, flaky):
    if path.endswith(".locked"):
        raise PermissionError(13, "Permission denied", path)     # not transient: the file's own error
    if flaky.random() < 0.3:
        raise NamedError("unavailable", f"the peer closed the connection mid-body during {path}")
    return path


def run(paths, seed):
    flaky = random.Random(seed)
    job = {"planned": len(paths), "landed": 0, "errors": [], "notes": []}
    for path in paths:
        try:
            retrying(job, lambda: move(path, flaky), path)
            job["landed"] += 1
        except NamedError as exc:
            job["errors"].append(f"{path}: {exc.cause}: {exc.detail}")
        except OSError as exc:
            job["errors"].append(f"{path}: unreadable: {exc.strerror}")
        except Exception as exc:                      # a bug is a named failure, never silence
            job["errors"].append(f"{path}: internal: {type(exc).__name__}: {exc}")
    lost = job["planned"] - job["landed"] - len(job["errors"])
    if job["errors"]:
        state = f"failed: {len(job['errors'])} problem(s), the rest landed; first: {job['errors'][0]}"
    elif lost:
        state = f"failed: {lost} file(s) neither landed nor reported an error"
    else:
        state = f"done: {job['landed']} files"
    return state, job["notes"]


for paths, seed in (([f"notes/{i}.md" for i in range(8)], 7), (["a.md", "b.locked", "c.md"], 1)):
    state, notes = run(paths, seed)
    print(state)
    for note in notes:
        print("   ", note)

External links

Exercise

Run the runnable block. Then add a fourth kind of failure to move() that none of the handlers expects. For example, raise a KeyError for one path, and remove the last except Exception tier from run(). Watch the program crash instead of reporting. Then restore the tier, and change run() so each file runs on its own threading.Thread without any handler at all. Show that the final count still catches the lost file even though no error was recorded.

Hint

An exception inside a thread is printed to stderr by threading.excepthook and then forgotten: landed simply never increases for that path. With the handlers gone, only lost = planned - landed - len(errors) notices, which is why Delta keeps both the handlers and the count.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousResumable by Record Next →Links, Patterns and Folders That Will Not Open

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.