Stall, Not Duration

Two Questions a Timeout Can Answer

A timeout can ask either of two questions. Has this taken too long? Or: has anything happened lately? They sound alike and they fail in opposite directions. A deadline on the whole operation punishes a slow but healthy link: a large upload over a hotel uplink is doing everything right and still runs out of time. A limit on silence punishes only a dead link: as long as bytes keep moving, the operation may take as long as it needs.

Delta's rule for moving bytes is the second one. Nothing moving for 30 seconds is the failure. Taking a long time is not. A 16 MiB chunk over a 2 Mbit/s uplink takes about a minute and is fine.

The Bug Hidden in sendall

The first version believed it already had that rule. Every connection set a 30 second socket timeout, and a socket timeout sounds like a stall timeout. It is not, for one call in particular. Since Python 3.5, the timeout on socket.sendall() is the maximum total duration to send all the data. It is no longer reset each time some data goes out. http.client hands a request body that is one bytes object to a single sendall. So a 16 MiB chunk had to leave within 30 seconds, which is about 4.5 Mbit/s per stream. Below that, a push failed on every run, while every byte was still flowing.

The review that found it wrote the failing test first (test_a_slow_uplink_does_not_fail_a_body_that_keeps_moving), marked as an expected failure, and then fixed the code under it.

The Fix: Bodies Leave in Pieces

A request body larger than 256 KiB is now passed to http.client as a generator of 256 KiB memoryview slices, with an explicit Content-Length. http.client sends each piece with its own sendall, so each piece gets its own 30 seconds. The body as a whole has no deadline. Three details make it work:

No copies. A memoryview slice of the chunk's buffer is a window, not a new 256 KiB object. A bundle's parts (header lines and file bytes) are sent as they are, not joined into a second copy first.
Progress per piece. Each piece that leaves reports its size to the job, so a slow link shows bytes moving instead of a frozen counter that jumps every minute.
Progress taken back. If an attempt fails and is tried again, the bytes it reported are subtracted first. The counter never claims bytes that did not arrive.

The Receiving Side Has the Same Rule

Reading is easier to get right, because each recv is its own blocking call, so each one gets the full timeout. Delta reads a chunk straight into one preallocated buffer, up to 1 MiB per call. Every read goes through one wrapper that turns the two ways it can fail into named causes:

What happened	What the caller is told
no response header within 30 s	`timeout`: "first byte from the peer for GET … did not arrive within 30 s"
the body stopped for 30 s	`timeout`: "the peer stopped sending for 30 s at 1.2 of 16.0 MiB via lan"
the connection ended inside the body	`unavailable`: "closed the connection mid-body at 1.2 of 16.0 MiB"
sending stalled	`timeout`: "sending PUT … stalled for 30 s"

The third row matters more than it looks. http.client's read(n) returns a short body without a word when the connection closes early. Only a read with no size raises IncompleteRead. Delta compares what arrived with Content-Length after every body. A short read is a cut and becomes an error, never a smaller file.

Where the Limits Sit

The stall limit is not the only clock. Connecting has its own short timeouts per route (2 s on Thunderbolt, 3 s on the LAN, 8 s on the tailnet), because a Mac that is off should be noticed quickly. Track 3 covers what happens to the callers after that. Hash requests may wait longer for their first byte (up to 900 s), because the source may have to read a large file before it can answer. Everything else that moves bytes lives under the one rule: no deadline on the work, and 30 seconds of patience for silence.

When the stall limit fires, the job does not give up at once either. Lesson 5 covers the retries that sit on top of it.

Code

Delta's peer client: a body leaves in pieces, each with its own stall limit (trimmed excerpt)·python

# delta/peer.py (trimmed excerpt)
READ_TIMEOUT = 30.0
# A request body goes out in pieces, each with its own stall timeout. Sent whole,
# the timeout bounded the entire upload: a 16 MiB chunk had to leave within 30 s,
# so a push over an uplink under about 4.5 Mbit/s per stream failed on every run.
SEND_PIECE = 256 * 1024


def _raw(self, route, label, method, path, *, body=None, headers=None, read_timeout=READ_TIMEOUT,
         progress=None):
    parts = list(body) if isinstance(body, (list, tuple)) else None
    length = sum(len(part) for part in parts) if parts is not None else (len(body) if body is not None else 0)
    pieces = parts is not None or length > SEND_PIECE
    if pieces:
        hdrs["Content-Length"] = str(length)   # with it, http.client sends the pieces as they are

    sent = [0]

    def in_pieces():
        for part in (parts if parts is not None else (body,)):
            view = memoryview(part)
            for start in range(0, len(view), SEND_PIECE):
                piece = view[start:start + SEND_PIECE]
                yield piece
                sent[0] += len(piece)
                if progress is not None:
                    progress(len(piece))

    def unsend():   # an attempt that is tried again took its progress back
        if progress is not None and sent[0]:
            progress(-sent[0])
        sent[0] = 0
    ...

The same 2 MiB to a slow reader, sent whole and in pieces, with a 1 s timeout (runnable)·python

import socket
import threading
import time

PIECE = 64 * 1024
BODY = b"x" * (2 * 1024 * 1024)       # 2 MiB to send
TIMEOUT = 1.0                         # the stall limit, in seconds


def slow_reader(server, rate_bytes_per_s):
    """Accept one connection and drain it at a fixed rate, like a weak uplink."""
    conn, _ = server.accept()
    conn.setsockopt(socket.SOL_SOCKET, socket.SO_RCVBUF, 64 * 1024)
    step = 32 * 1024
    while True:
        data = conn.recv(step)
        if not data:
            break
        time.sleep(len(data) / rate_bytes_per_s)
    conn.close()


def send(mode):
    server = socket.socket()
    server.bind(("127.0.0.1", 0))
    server.listen(1)
    threading.Thread(target=slow_reader, args=(server, 600 * 1024), daemon=True).start()
    client = socket.create_connection(server.getsockname())
    client.setsockopt(socket.SOL_SOCKET, socket.SO_SNDBUF, 64 * 1024)
    client.settimeout(TIMEOUT)
    started = time.monotonic()
    try:
        if mode == "whole":
            client.sendall(BODY)       # one call: the timeout bounds all of it
        else:
            view = memoryview(BODY)
            for start in range(0, len(view), PIECE):
                client.sendall(view[start:start + PIECE])   # each piece gets its own limit
        result = "sent"
    except socket.timeout:
        result = "timed out"
    finally:
        client.close()
        server.close()
    print(f"{mode:>6}: {result} after {time.monotonic() - started:.1f} s")


send("whole")      # times out after 1.0 s, with bytes still flowing
send("pieces")     # finishes in about 3 s: no piece ever waited 1 s

Exercise

Run the runnable block, then find the edge. Lower the reader's rate step by step (600, 300, 100 KiB/s) and note when the piece-by-piece send starts to time out too. Work out from the numbers how large a piece can be for a given stall limit and link rate, and check your formula against Delta's choice: 256 KiB pieces and a 30 s limit.

Hint

A piece times out when it cannot leave within the limit: piece size divided by rate, plus whatever the socket buffers absorb. 256 KiB in 30 s is about 70 Kbit/s, so Delta's pieces only stall on a link that has effectively stopped.