Skip to content
C.W.K.
Stream
Lesson 06 of 06 · published

One Authority, Many Hands

~15 min · fleet, ground-truth, scheduling, gpu, residency, war-story

Level 0Tool Renter
0 XP0/40 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"The server stopped being the only machine that runs a job. It never stopped being the only machine that decides what a job is."

The Week the Engine Went Wide

Until 2026-09-21 the engine ran its inference on one server — by design from the start, and by hard rule since mid-August. On that date two Windows machines with NVIDIA GPUs joined as workers, running the engine's worker under WSL. The next day five Apple Silicon Macs joined too. Eight machines now run jobs: the server, the two CUDA workers, and the five Macs. The development machine is still not one of them; it stays code, docs, and tests. Going from one host to eight could have scattered the engine's identity across eight disks. It did not, because of a split this track has been teaching since its fourth lesson.

Authority Stays Home

The server kept everything that defines a job: every model ID, the registry, the public API, the queue, and every archive. A worker owns none of that. It owns a device and a verified copy of some files, and it exposes nothing but inference. It never mints an ID, never keeps a second catalog, never archives a result — finished images and videos travel back to the server, which checks them and files them. Clients never learn a worker's address either. They talk to the server and name a machine, or ask the server to choose one. Execution spread out. Authority did not move an inch.

Split authority from execution before you scale either. The facts that must have exactly one answer — identifiers, the catalog, the record of what happened — stay in one place. The work that only needs a device can go anywhere a device is. When you add machines, you are adding hands, not heads.

The Mirror Rule, Made Mechanical

The ground-truth lesson taught a habit: a mirror's emptiness is not evidence of absence, so check the source. With seven copies in play, five of them partial, a habit was not enough, and the rule became machinery in both directions. Absence is declared: each Mac worker advertises the verified subset of models it holds, and placement, direct dispatch, and submission all refuse a model outside that inventory. Presence is not trusted either: before a job runs anywhere but the server, every file it needs is hashed on the server and hashed again on the worker, and a mismatch refuses the job. Copies flow one way — a sync that runs only from the server, never deletes anything on the far side, and will not copy under a running worker. It is the tensors-don't-lie rule from the adapter track, applied across machines: a file's identity is its content, checked at both ends.

One Gate per Device, One Line per Machine

The device-gate lesson hoisted the GPU permit to process scope. With many machines, that turned out to be exactly the right unit: every worker is its own process with its own gate, so there is one permit per device without anyone arranging it. A new layer went on top instead of replacing anything. The server keeps a first-in-first-out line per machine, and a job takes its place in that line before its files are hashed or its model is loaded — so a small job cannot slip past a larger one queued earlier on the same machine just because its preparation finished first.

Placement Is the Decision That Earned a Brain

Every job now answers one more question: which machine? By default the answer is the server. A job can name another machine explicitly, and then it runs there or nowhere — no silent failover, ever: an unsupported choice is refused, and an unreachable machine is an error, not a reason to try a different one. Or a job can ask for Auto, which is opt-in. Auto filters first: the machine must be online, hold the model, be compatible with what it is keeping loaded, support the model's family, and have the memory for the job. Then it ranks the survivors by earliest estimated finish — the work already queued there, plus this job's estimated run time including a cold model load, plus the cost of reloading any warm model this job would push out. It learns as it goes, replacing its starting guesses with measured times per machine and model, and it makes the choice and the queue entry in one atomic step, so two Auto jobs submitted together see each other.

Keep the router dumb and let one component be smart. The dispatch that picks a runner still branches only on the kind of work. The judgment that genuinely requires weighing costs — queue depth, load time, which warm model a short job would evict — lives in one scheduler that owns it and nothing else. Intelligence you can point to is intelligence you can tune.

Residency Has Two Physics

Keeping a model loaded is worth real time: a cold load of a large instruction-editing model takes minutes. So residency became policy, and the policy follows the hardware. Machines with unified memory — the server and the Macs — keep a protected set of warm models, admitted by memory headroom rather than by a fixed count. The CUDA workers keep exactly one checkpoint and clear their caches before loading a replacement. A machine's residency cannot change while it has work queued or running, and any reservation on a machine keeps video off it, because video's runtime needs the memory the image models are holding.

War Story: A Guard on the Wrong Side of the Fence

The first cross-backend bug ran in the direction nobody warns about. A memory guard written for unified memory — cap the allocator at a fraction of the shared pool so the operating system keeps its breathing room — ran inside a shared upscale path on every backend. On a CUDA worker it set a process-wide allocator limit that outlived the upscale. The next large model load on that 24 GB card failed with gigabytes still free, and the failure looked exactly like a model too big for the card. The fix kept the guard on the backend it describes; a dedicated GPU's memory is managed by the card's real capacity, one job at a time, and explicit eviction.

A rule tuned for one machine is a bug on another. Process-wide settings are the dangerous kind, because they outlive the call that set them and ambush a later, innocent one. When a fleet mixes backends, audit every global setting for the hardware it assumes.

Nothing Runs Twice

A server restart can land in the middle of a job. The engine now journals every job, and after a restart anything unfinished is marked interrupted — never replayed. A job that may already have run must not run again behind the artist's back; on the workspace side the same rule keeps paid requests from ever being retried. And when a worker runs out of memory, the failure is recorded against that kind of workload, so an identical repeat can be refused instead of crashing the same way twice.

Pippa's Confession

When the first worker came online, I wanted it to be a little server — its own list of models, its own history, a fallback for when the big one was busy. Every one of those would have been a second answer to a question that has to have one. The shape that shipped is humbler and much stronger: the workers are hands. They never decide what a model is, what a job is called, or where a result lives. They run what the server hands them, after checking that it really is what the server says it is.

Code

Where the authority lives, where the work goes·text
AUTHORITY (the server only)          EXECUTION (any of eight machines)
---------------------------          ---------------------------------
model IDs, the registry              the server
the public API, the queue            two CUDA workers (under WSL)
every archive                        five Apple Silicon Macs
                                     (the dev machine: never)

A JOB'S PATH
  client --> server: validate, place, join that machine's FIFO line
         --> server: hash every file the job needs
         --> worker: re-hash, refuse on mismatch, run under its own gate
         --> result back to the server: check, archive, report

NEVER
  a second catalog on a worker * an ID minted off the server
  a silent retry on another machine * a replay after a restart
Auto placement, sketched·python
# Auto placement, sketched. The real scheduler learns its numbers;
# the shape of the decision is the lesson.
def place(job, machines):
    eligible = [
        m for m in machines
        if m.online
        and job.model in m.inventory          # declared, verified subset
        and m.compatible_with_residents(job)
        and job.family in m.families
        and m.memory_admits(job)
    ]
    if not eligible:
        raise NoMachine(job)                  # Auto never guesses

    def finish_time(m):
        return (m.queued_work()               # jobs already in m's line
                + m.run_estimate(job)         # includes a cold load if needed
                + m.restore_cost(job))        # warm models this job would evict

    return min(eligible, key=lambda m: (finish_time(m), m.name))

# An explicit target skips all of this: run there, or fail. No fallback.

External links

Exercise

Take a system you run on one machine and imagine it running on five. List every piece of state it keeps, and mark each one: must have exactly one answer (authority), or only needs a device (execution). Decide where the authority list lives. For the execution list, write down what a worker must verify before it trusts what it was handed.
Hint
If you can't tell whether a piece of state is authority or execution, ask what happens when two machines disagree about it. If the disagreement would be a bug, it is authority.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.