Lesson 05 of 05 · published

Pooled Connections, Named Failures

~14 min · http, keep-alive, connection-pooling, error-handling, python, http-client

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

384 Handshakes Was the Bug

Go back to where Delta began. Loomis needed 384 Eagle thumbnails from another Mac, and each one was fetched by a fresh SSH session: a new TCP connection, a new key exchange, a new login, for a few kilobytes of image. sshd treats a crowd of half-finished logins as an attack and starts dropping them (MaxStartups), so 152 of the 384 failed, and the ones that succeeded had a median of 3.8 seconds. The thumbnails were never the expensive part. The connections were.

Through Delta the same grid finished with zero failures in 1.94 seconds cold, with a median of 0.19 seconds and a p99 of 0.30, and warm runs were all 304s with no new bytes. The cache in front (track 5) explains the warm runs; the cold run is mostly this lesson: one pool of persistent connections per road, so a burst of 384 requests rides a handful of TCP connections instead of 384 handshakes.

The Pool

PeerClient keeps idle HTTP/1.1 connections keyed by everything that makes a connection what it is: host, port, whether it is TLS, and the pinned fingerprint. A connection made for the Thunderbolt address is never handed to a request for the LAN address, and a TLS connection pinned to one certificate is never reused after the peer published another. After a request finishes reading its body cleanly, the connection goes back into its pool. Up to 16 idle connections are kept per route; anything beyond that is closed.

The pool also has an expiry, and the number is chosen against the other end. The peer's gateway idles a keep-alive out after 75 seconds; the client throws away anything idle for more than 50. The gap matters: if both used 75, the client would sometimes pick a connection at the very moment the server closed it, and the request would die in the socket. Retiring connections earlier than the server does makes that race rare.

Rare is not never. A pooled connection can still have died while idle (the peer restarted, a cable was swapped). When a reused connection fails in the way a dead socket fails (the remote end disconnected, a reset, a broken pipe) and the request is idempotent, Delta closes that connection, drops all its idle siblings on the same road (they almost certainly died with it), and tries again. With the road's pool now empty, that second try opens a fresh connection, so in practice the silent retry happens once; the loop's ceiling of three attempts is reached only if another request hands a connection back to the pool in between. This retry is silent, and that is correct: the request never reached the peer, so nothing happened that anyone needs to hear about. A fresh connection that fails the same way is a real error and is named as one.

Two small settings complete the picture. Every new connection turns on TCP_NODELAY, so small requests are not held back by Nagle's algorithm waiting to fill a packet. And every response is read with Accept-Encoding: identity, so the byte count on the wire is the byte count of the file.

Every Failure Names Its Phase and Its Road

A request can fail in four places, and Delta's error says which, on which route, after how long:

Phase	What the error says
Connect	`connect to … via thunderbolt (… :5201 on bridge0) timed out after 2.0 s`
First byte	`first byte from … for GET … via lan did not arrive within 30 s`
Stall	`… stopped sending for 30 s at 1.2 of 4.0 MiB via tailnet`
Cut	`… closed the connection mid-body at 10 of 100 bytes via lan`

These messages travel. An error crosses the wire as {"error", "detail", "status"} and is rebuilt on the other side with the same class (Unavailable is 503, Timeout is 504), so an app three hops away reads the sentence the failing hop wrote, not a bare "502".

The Silent Short Read

The last row of that table exists because of a trap in Python's standard library. http.client's read() with no argument raises IncompleteRead if the connection ends before Content-Length bytes arrived. But read(n) with a size, which is what any program streaming a large body uses, returns whatever arrived, short, without a word. A peer that died halfway through a 16 MiB chunk would hand the caller fewer bytes and a clean return.

So everything that reads from a peer in Delta goes through peer.Response. It counts every byte it hands out, and when the body ends it compares the count with Content-Length (a 304 and a HEAD carry no body and are exempt). Short means Unavailable: closed the connection mid-body at N of M: a named error, never a short answer. Run shortread.py below to watch the standard library return 10 bytes out of a promised 100 and raise nothing.

Code

The pool checkout and the body checks in peer.py (trimmed excerpt)·python

# delta/peer.py: the pool, and a body that cannot come back short (trimmed excerpt)

MAX_IDLE_PER_ROUTE = 16
IDLE_SECONDS = 50.0          # the peer's gateway idles a keep-alive out after 75 s
READ_TIMEOUT = 30.0


def _checkout(self, route, label):
    key = (route.host, route.port, route.tls, route.fingerprint)
    with self._lock:
        idle = self._idle.get(key) or []
        while idle:
            conn, since = idle.pop()
            if time.monotonic() - since < IDLE_SECONDS:
                self.counters["reused"] += 1
                return key, conn, True        # warm: no handshake, no TLS, no new socket
            conn.close()                      # too old: the peer may already have closed it
    return key, self._connect(route, label), False


class Response:
    def _take(self, read):
        """A stall is a timeout, and a connection that ends inside a body is a cut,
        never a short answer."""
        try:
            return read()
        except socket.timeout:
            self.close()
            raise Timeout(f"{self.label} stopped sending for {self.read_timeout:.0f} s "
                          f"at {self._where()}") from None
        except (http.client.HTTPException, OSError) as exc:
            self.close()
            raise Unavailable(f"{self.label} closed the connection mid-body at {self._where()}: "
                              f"{type(exc).__name__}") from None

    def _ended(self):
        total = self.header("content-length")
        if total and self.status != 304 and self.received < int(total) and self.method != "HEAD":
            where = self._where()
            self.close()
            raise Unavailable(f"{self.label} closed the connection mid-body at {where}")

shortread.py: the standard library returning a short body without a word·python

# shortread.py: a server that promises 100 bytes and sends 10, and a client that believes it.
import http.client
import http.server
import threading


class Liar(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        self.send_response(200)
        self.send_header("Content-Length", "100")
        self.end_headers()
        self.wfile.write(b"0123456789")   # then the connection closes
        self.close_connection = True

    def log_message(self, *args):
        pass


server = http.server.HTTPServer(("127.0.0.1", 0), Liar)
threading.Thread(target=server.handle_request, daemon=True).start()

conn = http.client.HTTPConnection("127.0.0.1", server.server_port, timeout=5)
conn.request("GET", "/")
resp = conn.getresponse()
promised = int(resp.getheader("Content-Length"))
body = resp.read(promised)          # a sized read: no exception
print(f"promised {promised}, got {len(body)}, no error raised")

received = len(body)
if received < promised:
    print(f"the check Delta adds: closed the connection mid-body at {received} of {promised} bytes")

External links

Exercise

Run shortread.py with Apple's /usr/bin/python3 and with any other Python you have; confirm that both return a short body from read(100) without an exception. Then change only the client line to resp.read() with no size and run it again: what happens now? Finally, write a small wrapper function read_exactly(resp, chunk=8192) that streams the body in chunks and raises a ConnectionError naming "N of M bytes" when the total falls short of Content-Length.

Hint

With no size, read() raises http.client.IncompleteRead, whose partial attribute holds what did arrive. Your wrapper should loop on resp.read(chunk) until it returns b"", add up the lengths, and compare with int(resp.getheader("Content-Length")) only when that header exists and the status is not 304.