Lesson 03 of 05 · published

A Choice Kept Sixty Seconds

~12 min · routing, caching, concurrency, locks, laptop, cli

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

A Decision Is Expensive; Remembering It Is Not

The last two lessons built a careful decision: read the peer's identity over the tailnet, read this Mac's interfaces, ask the routing table about every advertised address, sort what survives, and send a real health request down each candidate until one answers. That is the right way to decide. It is also far too slow to do before every request. A Loomis grid asks for 384 thumbnails in a burst, and a sync lists thousands of files; nobody wants three subprocesses and a probe in front of each one.

So Delta remembers. RouteBook keeps one chosen Route per peer, with the time it was chosen and the reason it was chosen in a sentence. The interesting part is not that it caches, but how long each kind of belief is allowed to live, and what makes it die early.

Three Clocks

Belief	Lives	Why that long
The chosen route	60 s	Long enough that a burst rides one decision, short enough that the network's real state wins within a minute
A LAN route is still on-link	re-proved every 10 s	A pulled cable or a laptop carried out the door is noticed within ten seconds
This Mac's interfaces	10 s memo	Three subprocesses per read; re-read no faster than anything can physically change

When a kept LAN route is older than ten seconds since its last proof, get does not re-run the whole decision. It asks one cheap question: does route -n get on the route's address still name the same interface, with no gateway? If yes, the proof is renewed and the route is returned. If no, the full selection runs again. A tailnet route is not re-proved at all within its minute: being away is the safe answer, and there is nothing cheap to check.

The Asymmetry Is Deliberate

Walk through a laptop's day. At the desk, its route to the service host is Thunderbolt. The owner unplugs and walks out. Within ten seconds the next request finds the bridge no longer on-link, selects again, finds no on-link address at all, and switches to the tailnet with the reason away: no address of the peer is on-link with an active physical interface here. If a request happens to be in flight when the cable comes out, it fails, and the failure drops the route at once (next lesson); the ten seconds are a ceiling, not a delay.

Coming home is slower. The kept tailnet route lives out its full minute before anything re-examines it, so for up to sixty seconds the laptop keeps using the tunnel while a cable sits right there. That is the right side to be wrong on. Using the slow road for a minute costs time; using a fast road that has gone away costs a timeout on every request and, on a foreign network, dials a stranger. When the owner wants the fast road now, delta route @pippa --refresh forgets the kept choice and decides again on the spot.

One Decision Per Peer, Shared

Selection takes a lock per peer, so when 384 thumbnail requests arrive at the same instant on a cold book, one of them does the identity read and the probes while the other 383 wait on the lock, then all find the fresh route and go. Without that lock, a burst would probe the peer hundreds of times in parallel, which is exactly the kind of connection storm that broke SSH in the first place (track 1). Different peers have different locks, so a slow decision about one Mac never blocks requests to another.

Overrides, and a Question That Never Waits

Two escape hatches sit in front of the clocks. DELTA_FORCE_ROUTE, or a routes entry in this Mac's own host config, can pin a peer to the tailnet: that is how the bench measures the away road from inside the house. And the CLI's --tailnet flag does the same for a single command. A peer that is this Mac never reaches the book at all: it is local, with no network. Finally, known(peer) answers "which route is in use?" without choosing: a reply served from the cache names its route in a header, and that header must never wait on a probe or on an eight-second connect to a Mac that is asleep.

The Book Keeps Its Own History

A route book that only knew the current answer would be useless the morning after something went wrong. Delta's keeps a second map beside the routes: failures, the last reason each peer's LAN route was dropped or each candidate failed its probe. delta status prints every kept route with its kind, address and reason, and the full health document carries both maps. So the question "why has the laptop been on the tunnel all afternoon?" has an answer written down at the moment it happened, in the words of the check that failed.

Code

RouteBook.get: one minute of memory, ten seconds of proof (trimmed excerpt)·python

# delta/routes.py: RouteBook.get, the memory in front of every decision (trimmed excerpt)

CACHE_SECONDS = 60.0
ONLINK_RECHECK_SECONDS = 10.0  # a cable pulled is noticed within 10 s; a failed request notices at once


def get(self, peer, *, refresh=False, force=None):
    if peer == config.this_label():
        return Route(peer, "local", "", 0, reason="this Mac", checked_at=time.time())
    force = force or self.forced(peer)          # DELTA_FORCE_ROUTE, or the host config's "routes"
    if force == "tailnet":
        return self.tailnet(peer, "forced to the tailnet (host config or caller)")
    with self._lock:
        lock = self._peer_locks.setdefault(peer, threading.Lock())
    with lock:  # one selection per peer at a time; parallel callers share it
        cached = self._routes.get(peer)
        if cached and not refresh and time.time() - cached.checked_at < CACHE_SECONDS:
            if cached.kind == "tailnet":
                return cached
            verified = self._verified.get(peer, float("-inf"))
            if time.monotonic() - verified < ONLINK_RECHECK_SECONDS:
                return cached
            if self._still_on_link(cached):     # route -n get still names the same interface
                self._verified[peer] = time.monotonic()
                return cached
        route = self._select(peer, force)       # identity, on-link, order, health probes
        with self._lock:
            self._routes[peer] = route
            self._verified[peer] = time.monotonic()
        return route

routebook.py: the same clocks in thirty lines you can run·python

# routebook.py: a decision kept for a minute, re-proved cheaply, dropped at once on failure.
import threading
import time

KEEP_SECONDS = 60.0      # how long a choice is trusted at all
RECHECK_SECONDS = 10.0   # how often a kept LAN choice re-proves itself


class Book:
    def __init__(self, choose, still_true):
        self.choose = choose            # the expensive decision: probes, subprocesses
        self.still_true = still_true    # the cheap re-proof of a kept LAN choice
        self.routes = {}                # peer -> (route, chosen_at, proved_at)
        self.locks = {}
        self.lock = threading.Lock()

    def get(self, peer, refresh=False):
        with self.lock:
            lock = self.locks.setdefault(peer, threading.Lock())
        with lock:                      # one decision per peer; parallel callers share it
            now = time.monotonic()
            kept = self.routes.get(peer)
            if kept and not refresh and now - kept[1] < KEEP_SECONDS:
                route, chosen, proved = kept
                if route == "tailnet" or now - proved < RECHECK_SECONDS:
                    return route
                if self.still_true(peer, route):
                    self.routes[peer] = (route, chosen, now)
                    return route
            route = self.choose(peer)
            self.routes[peer] = (route, now, now)
            return route

    def invalidate(self, peer, why):
        if self.routes.pop(peer, None):
            print(f"route to {peer} dropped: {why}")


if __name__ == "__main__":
    at_home = {"value": True}
    choose = lambda peer: "thunderbolt" if at_home["value"] else "tailnet"
    still = lambda peer, route: at_home["value"]
    book = Book(choose, still)
    print(book.get("service-host"))          # thunderbolt, chosen
    at_home["value"] = False                 # the laptop leaves the house
    print(book.get("service-host"))          # still thunderbolt: proved under 10 s ago
    book.invalidate("service-host", "connect timed out")   # a failed request does not wait
    print(book.get("service-host"))          # tailnet, chosen again
    at_home["value"] = True                  # home again
    print(book.get("service-host"))          # tailnet: kept until 60 s pass
    print(book.get("service-host", refresh=True))   # thunderbolt: asked for a fresh look

Asking your own Delta which way it would go·bash

# Ask this Mac's Delta which way it would go, and why (the label and address shown are stand-ins)
delta route @pippa
# service-host: thunderbolt 192.0.2.20:5201 — same house: thunderbolt on bridge0 (1.4 ms health)

delta route @pippa --refresh     # forget the kept choice and decide again now
delta route @pippa --tailnet     # what the away road looks like from here
delta route @pippa --json        # the whole Route: kind, host, port, tls, interface, link, reason

External links

Exercise

Run routebook.py and match each printed line to the comment beside it. Then change it into an experiment of your own: make choose sleep for one second, start twenty threads that call book.get("service-host") at the same moment on an empty book, and count how many times choose ran. Remove the per-peer lock and run it again. Write down both counts and the wall time of each run.

Hint

Wrap choose in a function that increments a counter under its own lock before sleeping. With the per-peer lock, expect one call and about one second of wall time: the other nineteen threads wait and then find the kept route. Without it, expect close to twenty calls in about the same wall time, which is twenty probes of the same peer for one answer.