The 152 Thumbnails

A Grid of 384 Tiles

Loomis is the family's drawing-construction engine: it reads reference images and practice drawings and measures them. Its engine runs on the service host. The references it reads live somewhere else: in Dad's Eagle library, on the workbench, where Dad collects and tags images by hand. So the most ordinary thing Loomis does, opening its reference grid, is a cross-machine read. One screen asks for 384 thumbnails at once.

On the morning of 2026-11-09 that screen came up with holes in it. Of the 384 tiles, 152 failed with a 502. The whole grid took 32.6 seconds to settle, and the median tile took 3.8 seconds. Nothing was wrong with the images, the library, or the network cable between the two Macs. Every failure came from the way the bytes were being carried.

What One Tile Cost

Before that morning, Loomis read the workbench the way most of us would on day one: over SSH. A request went out as ssh <workbench> 'python -m loomis.eagle_worker', the request JSON on stdin, the answer JSON on stdout, the image inside that JSON as base64. It was careful code. Credentials never crossed the wire, user input never touched the remote shell, and a write was never retried after a lost reply.

It was also expensive in a way that does not show on one request. Each thumbnail opened two or three fresh SSH sessions, and each session started a fresh Python process on the far side. So one grid was not 384 reads. It was somewhere between 768 and 1,152 SSH handshakes, each with a key exchange, an authentication, a process spawn, and a base64 round trip, all arriving at the workbench's sshd within a few seconds.

MaxStartups Is Not a Bug

OpenSSH's server protects itself from a flood of half-open logins with one setting, MaxStartups. It counts unauthenticated connections: ones that have connected but not yet finished logging in. The default is 10:30:100, read as start:rate:full. Below 10 pending handshakes, every new one is admitted. At 10, a new one is refused with probability 30%. The probability climbs linearly from there, and at 100 pending every new connection is dropped.

A burst of short SSH sessions is exactly what this rule was written against. Each session spends its first fraction of a second unauthenticated, and if hundreds arrive in that window, sshd starts dropping them at random. Loomis saw the dropped session as a failed remote process, and the browser saw a 502. That is how 152 tiles out of 384 went blank.

Notice where the fault sits. sshd did exactly what it was configured to do. Loomis' code did exactly what it said. The failure lived between two correct programs, in a component nobody in the family wrote and nobody in the family should be tuning. You could raise MaxStartups on one Mac, then on every Mac, and you would have spent the fix on a daemon whose job is to guard logins, not to move thumbnails.

The Same Grid, After

The same day, the grid moved onto Delta. The workbench's Delta serves Loomis' Eagle library as a share, and Loomis on the service host asks its own Mac's Delta for each tile. Measured on the same 384 tiles:

Path	Failed	Wall time	Per tile
per-item SSH	152 of 384	32.6 s	p50 3.8 s
Delta, cold	0 of 384	1.94 s	p50 0.19 s, p99 0.30 s
Delta, warm	0 of 384	1.67 to 1.88 s	every tile a 304, no new bytes

Three things changed, and none of them is "Delta is faster than SSH" in the abstract. First, the tiles ride a handful of keep-alive connections instead of hundreds of handshakes, so there is no burst of unauthenticated anything. Second, the caller is authenticated once per connection, by the route it arrived on, not once per item. Third, each tile carries the owner's validator, so the second visit asks "has this changed?" and gets a 304 with no bytes, which is why the warm grid moves no image data at all.

What Stayed on SSH

One Loomis operation did not move: setting an item's star rating. That is a write, a single small action, and the commit that moved the reads says so in as many words: the star rating remains the one command over SSH, and the old worker now refuses get, image and stat with "reads ride cwkDelta". The split is the whole lesson in miniature. SSH is a fine way to run a command. It was never meant to be a data pipe that hundreds of requests a second pour through.

Loomis' frontend also got a small belt to go with the new braces: a thumbnail that still fails is retried at 0.6, 1.8 and 5 seconds, and then shows a tap-to-retry tile instead of a blank square. A fix that removes a failure class is the main event. A user interface that names the rare leftover failure, instead of hiding it, is how you keep trusting the fix.

Code

Loomis before Delta: one SSH session per remote read (trimmed excerpt)·python

# loomis/eagle.py, before Coop 1109-10 (trimmed excerpt)
# Every remote read was a fresh ssh session running a Python worker on the
# workbench; the request rode stdin and the answer (base64 image inside JSON)
# rode stdout.
def _remote(request: dict) -> object:
    try:
        route = select_route('@workbench', purpose='control')
        command = 'cd <loomis checkout> && <loomis python> -m loomis.eagle_worker'
        result = subprocess.run([*route.ssh(), route.label, command],
                                input=json.dumps(request).encode(),
                                capture_output=True, timeout=45)
        if result.returncode:
            raise OSError('remote process failed')
        reply = json.loads(result.stdout)
    except (OSError, subprocess.SubprocessError, ValueError, TypeError) as exc:
        raise EagleError('Eagle workbench not reachable or its response unreadable') from exc
    return reply['data']

A burst meets MaxStartups: random early drop, simulated·python

# Random early drop, the way sshd's MaxStartups start:rate:full decides.
# An illustration, not a replay: how a burst of handshakes meets the limit.
import random


def refused(pending, start=10, rate=30, full=100):
    if pending < start:
        return False
    if pending >= full:
        return True
    chance = rate + (100 - rate) * (pending - start) / (full - start)
    return random.random() * 100 < chance


def burst(tiles, sessions_per_tile=2, handshake_s=0.25, arrivals_per_s=400):
    random.seed(7)
    pending = []          # when each unauthenticated handshake will finish
    failed = set()
    now = 0.0
    for tile in range(tiles):
        for _ in range(sessions_per_tile):
            now += 1.0 / arrivals_per_s
            pending = [done for done in pending if done > now]
            if refused(len(pending)):
                failed.add(tile)
            else:
                pending.append(now + handshake_s)
    return len(failed)


for rate in (30, 100, 400):
    print(rate, 'handshakes/s ->', burst(384, arrivals_per_s=rate), 'of 384 tiles failed')

Exercise

Find one place in your own tooling that reads data from another machine per item over SSH (a loop of ssh host cat …, scp per file, a remote script per request). Count the SSH sessions one realistic run opens, then run it ten times concurrently and record how many fail and with what message. Compare the count with your server's MaxStartups setting.

Hint

sshd -T | grep -i maxstartups (run with sudo) prints the effective value. On the client, ssh -v shows each handshake; a dropped one usually surfaces as kex_exchange_identification: Connection closed by remote host.