Drop It and Try Once More

A Fast Road That Stops Being One

A route chosen thirty seconds ago was true thirty seconds ago. Since then the Thunderbolt cable may have been pulled to plug in a display, the house switch may be rebooting after a firmware update, or the peer may have fallen asleep. The route book's clocks (last lesson) bound how long a stale belief can live, but they do not help the request that is in flight right now, on a road that just disappeared under it. That request needs its own rule, and Delta's is short: drop the road, choose again, try exactly once more.

The Rule, Precisely

When a request on a route fails as Unavailable (the connection could not be made or was cut), Timeout (a connect, a first byte or a stall ran out) or Forbidden (for instance, a LAN certificate that does not match the one published over the tailnet), PeerClient.request looks at three things before doing anything clever:

Was the failed route the tailnet? Then there is nowhere left to fall back to. The error goes to the caller as it is, with its named cause.
Was the request idempotent? A read can be asked twice; asking twice is harmless. A request that is not idempotent is never repeated on a second road, because nobody can say whether the first road delivered it. (The write call in track 7 takes this to its conclusion.)
Was the route forced? A caller that pinned a road, like the bench measuring Thunderbolt, wants that road's failure, not a quiet detour that would make its numbers lie.

If none of those apply, the failure is a LAN route that just died. Delta increments route_fallbacks, invalidates the route with the failure written into the reason (the log line reads route to … via lan dropped: Timeout: …), closes every idle pooled connection on that road, asks the book for a fresh choice with refresh=True, and sends the request once more. Usually the fresh choice is the tailnet. If that second attempt fails too, its error is the answer. There is no third attempt and no loop: a retry policy that keeps trying is how one dead Mac turns into a minute of hanging callers.

Connect Timeouts Sized to the Road

How long to wait for a connection depends on what a healthy connection looks like on that road. Delta waits 2 seconds on Thunderbolt, 3 on the LAN and 8 on the tailnet. A cable between two Macs connects in well under a millisecond; if it has not answered in two seconds, nothing is coming. The tailnet may need to punch through NAT or go via a relay, so it gets more room before it is declared dead.

The Second Caller Should Not Pay Again

Now picture the peer asleep, and a Loomis grid that keeps asking it for thumbnails: the next tile, the next row, the grid's own retries. Without a memory, each request dials the same address and waits the full timeout to learn the same fact. Before the 1110-7 review that was exactly what happened: a Mac that was off cost every request the whole timeout.

The fix is a hold-down. When a connect times out, _connect writes the address into _down with the moment and the full error sentence. For the next 4 seconds on Thunderbolt, 5 on the LAN or 10 on the tailnet, any other request to that address fails immediately with Unavailable: "did not answer 3 s ago (connect … timed out after 8.0 s); not dialing it again for another 7 s". The first caller pays; every caller that arrives after its timeout is told at once, and told the original cause, not a vague "unavailable". After the hold, the address is dialed again normally, so a Mac that wakes up is back within seconds.

Know the limit, because it is real. The hold is written only after a connect has timed out, and nothing makes simultaneous dials wait for one another. If sixteen requests dial the sleeping peer in the same instant, all sixteen are already inside connect when the first one gives up, so each of them pays its own 8 seconds on the tailnet. The hold-down spares the requests that come after the first failure, not the ones already waiting beside it.

Only a timeout arms the hold. A connection that is refused fails in a millisecond already, so remembering it would save nothing, and the next attempt might find Delta restarted and listening.

What the Caller Sees

Put the pieces together for a read from an app while the service host's Thunderbolt cable is being swapped. The request fails on the bridge with a named connect timeout; the route is dropped with that reason; a fresh selection finds the bridge no longer on-link and chooses the tailnet; the request is sent once more over the tunnel and succeeds. The app receives its bytes a couple of seconds late, with X-Delta-Route: tailnet on the answer, and the log holds the whole story. And if the peer is truly gone, a read the cache can still answer comes back labeled stale with the reason (track 5), rather than as an error.

Code

PeerClient.request and _connect: one fallback, one hold-down (trimmed excerpt)·python

# delta/peer.py: one retry on a fresh route, and a dial that remembers a dead address
# (trimmed excerpt)

CONNECT_TIMEOUT = {"thunderbolt": 2.0, "lan": 3.0, "tailnet": 8.0}
DOWN_SECONDS = {"thunderbolt": 4.0, "lan": 5.0, "tailnet": 10.0}


def request(self, label, method, path, *, body=None, headers=None, app=None,
            read_timeout=READ_TIMEOUT, force=None, idempotent=True, progress=None):
    route = self.delta.routes.get(label, force=force)
    try:
        return self._raw(route, label, method, path, body=body, headers=headers, app=app,
                         read_timeout=read_timeout, idempotent=idempotent, progress=progress)
    except (Unavailable, Timeout, Forbidden) as exc:
        if route.kind == "tailnet" or not idempotent or force:
            raise
        # A LAN route that just failed: drop it, choose again, try once more.
        self.counters["route_fallbacks"] += 1
        self.delta.routes.invalidate(label, f"{type(exc).__name__}: {exc.detail}")
        self.drop_route_connections(route)
        fresh = self.delta.routes.get(label, refresh=True)
        log.warning("retrying %s %s on %s via %s after: %s", method, path, label, fresh.kind, exc.detail)
        return self._raw(fresh, label, method, path, body=body, headers=headers, app=app,
                         read_timeout=read_timeout, idempotent=idempotent, progress=progress)


def _connect(self, route, label):
    timeout = CONNECT_TIMEOUT.get(route.kind, 5.0)
    down = self._down.get((route.host, route.port))
    if down:
        age = time.monotonic() - down[0]
        hold = DOWN_SECONDS.get(route.kind, 5.0)
        if age < hold:
            raise Unavailable(f"{label} did not answer {age:.0f} s ago ({down[1]}); not dialing it "
                              f"again for another {hold - age:.0f} s")
        self._down.pop((route.host, route.port), None)
    conn = http.client.HTTPConnection(route.host, route.port, timeout=timeout)
    try:
        conn.connect()
    except socket.timeout:
        detail = (f"connect to {label} via {route.kind} ({route.host}:{route.port}"
                  f"{' on ' + route.interface if route.interface else ''}) timed out after {timeout:.1f} s")
        self._down[(route.host, route.port)] = (time.monotonic(), detail)
        raise Timeout(detail) from None
    ...

holddown.py: watch one caller pay and three be told·python

# holddown.py: one caller pays the timeout; the callers right behind it are told at once.
import socket
import time

CONNECT_TIMEOUT = 2.0
HOLD_SECONDS = 4.0
_down = {}   # (host, port) -> (when it timed out, what it said)


def dial(host, port):
    held = _down.get((host, port))
    if held:
        age = time.monotonic() - held[0]
        if age < HOLD_SECONDS:
            raise ConnectionError(f"{host}:{port} did not answer {age:.0f} s ago ({held[1]}); "
                                  f"not dialing it again for another {HOLD_SECONDS - age:.0f} s")
        _down.pop((host, port), None)
    sock = socket.socket()
    sock.settimeout(CONNECT_TIMEOUT)
    try:
        sock.connect((host, port))
    except socket.timeout:
        sock.close()
        detail = f"connect to {host}:{port} timed out after {CONNECT_TIMEOUT:.1f} s"
        _down[(host, port)] = (time.monotonic(), detail)
        raise ConnectionError(detail) from None
    except OSError as exc:
        sock.close()
        raise ConnectionError(f"connect to {host}:{port} failed: {exc.strerror or exc}") from None
    return sock


# 192.0.2.1 is a documentation address (RFC 5737): nothing should answer there.
for caller in range(4):
    started = time.monotonic()
    try:
        dial("192.0.2.1", 5201).close()
    except ConnectionError as exc:
        print(f"caller {caller}: {time.monotonic() - started:5.2f} s  {exc}")

Exercise

Run holddown.py and confirm that caller 0 waits about two seconds while callers 1 to 3 return immediately with the first caller's reason. Then extend it: put the four callers in four threads started at the same moment, and run it twice, once as written and once with the hold-down check removed. Record the wall time and each thread's elapsed time in both runs. Finally, add a sleep of five seconds before a fifth call and show that the address is dialed again after the hold expires.

Hint

With threads starting together, all four may reach connect before the first timeout is recorded, so the hold-down helps the callers that arrive after the first failure, not the ones already waiting. That is also true in Delta. If 192.0.2.1 fails instantly with "network is unreachable" on your network, nothing timed out and nothing is held: try another address in a private range of your LAN that no device uses.