Skip to content
C.W.K.
Stream
More →
Lesson 05 of 05 · published

Gate the Device, Not the Object

~14 min · semaphore, gpu, concurrency, cancellation, war-story, scope

Level 0Tool Renter
0 XP0/36 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"A permit is a promise about hardware. The moment the permit's scope or its lifetime stops matching what the hardware is actually doing, you own a very convincing piece of decoration."

The Gate Was Right and Still Leaked — Twice

The previous lesson put a semaphore of one around the GPU section so two inferences could never stack and double peak memory. The principle was stated correctly: gate the scarce resource. The implementation put the semaphore on the adapter object. Those two sentences are not the same sentence, and the difference cost two separate bugs — both of them the same out-of-memory crash coming back through doors the gate couldn't see.

Leak One: A Second Consumer Outside the Object

The engine grew upscalers — a pixel-space upscaler path and a slow, minutes-long enhancement model. Those run on the same device, but they run from a different route, not through the adapter. The adapter's semaphore was an attribute on the adapter instance, so the upscale path simply never asked for the permit. A generation and a long upscale could sit on the device together, which is the doubled peak memory the whole gate existed to prevent. Worse, two upscales never serialized against each other either — from the gate's point of view they didn't exist.

The fix is small and the lesson is large: hoist the semaphore out of the object and into the process, so every entry point that touches the device shares one permit. The adapter binds to it, the upscale route acquires it around its own thread hand-off, and any future device consumer inherits the discipline by construction instead of by remembering.

Scope a lock to the resource it protects, not to the object that first used it. An instance-local gate protects that instance's calls — which is only the same thing as protecting the device while the instance is the device's only caller. That coincidence expires the first time a second consumer appears, and it expires silently, because nothing about the new code looks like it's breaking a rule. Give the shared thing a shared gate.

Leak Two: Cancelling the Wait Is Not Cancelling the Work

The second leak is subtler and has nothing to do with scope. GPU work runs in a worker thread so the event loop stays responsive; the async side awaits that thread inside the gate. Then a user cancels a generation — the normal flow, cancel, nudge a setting, regenerate. Cancelling the task unwinds the await immediately, which exits the block holding the permit. But the worker thread doesn't know anything happened. It is still mid-step on the device.

So the permit was free while the device was busy. The regenerate then acquired the permit legitimately and started a second inference on a device that already had one running: the exact double-peak path, reached this time not by bypassing the gate but by obeying it.

Cancelling an await cancels your waiting, not the work you were waiting on. Async cancellation propagates through awaits; it does not reach into a running thread, a native call, or a remote job. If a resource is held for the duration of an await, cancellation hands the resource back while the work is still happening. Anywhere a permit, a lock, or a lease wraps an await over work that isn't itself cancellable, that's the bug — and it only shows up under cancellation, which is exactly when nobody is watching.

The Fix: Hold Until the Thread Actually Exits

The repair shields the thread's future from the cancellation, sets a flag the sampler's per-step callback checks, waits for the thread to notice and unwind — absorbing further cancels while it waits — and only then lets the cancellation propagate. The permit is released when the device is free, not when the caller stopped caring. The same treatment goes on the model-load path, because a client disconnecting mid-preload hits the identical race.

Tie a resource's release to the resource becoming free, not to the caller's control flow returning. Ask of every acquire/release pair: is the release triggered by the work finishing, or by my function exiting? Those coincide on the happy path and diverge under cancellation, timeout, and error. The version that survives is the one where the release is a statement about the resource.

The Tax a Process-Wide Singleton Charges

Hoisting to process scope buys correctness and bills you in tests. A concurrency primitive built lazily binds to whatever event loop is running when it's first created, and a test suite that gives each case a fresh loop will hand the second test a gate bound to the first test's dead loop. The answer is an automatic per-test reset of the singleton — a small, deliberate piece of test wiring that exists purely because the thing is now process-scoped. Worth naming, because it's the moment a lot of people quietly revert to instance scope to make the tests pass.

I wrote the first gate, and I wrote it as an attribute because that's where state goes — inside the object that uses it. It felt like good encapsulation. What I'd actually done was encapsulate a fact about the hardware inside one of its several users, which meant the fact stopped being true the moment a second user existed. And I never would have caught the cancel bug by reading: it requires someone to cancel and immediately regenerate, which is what Dad does constantly and what I, testing my own code, never did. Both bugs were me protecting the object I was thinking about instead of the device everyone shares.

Code

Hoist the permit to the resource's scope·python
# BEFORE: the permit lives on the adapter instance.
class LocalAdapter(Adapter):
    def __init__(self):
        self._gpu = asyncio.Semaphore(1)   # protects THIS OBJECT's calls

# ...meanwhile, in a completely different route:
async def upscale(req):
    # Same device. Different object. Never asks for the permit.
    return await asyncio.to_thread(run_upscaler, req)   # UNGATED


# AFTER: the permit lives with the resource, at process scope.
_gate: asyncio.Semaphore | None = None

def gpu_gate() -> asyncio.Semaphore:
    """The shared GPU permit. Acquire around ANY device work."""
    global _gate
    if _gate is None:                # lazy: binds to the running loop
        _gate = asyncio.Semaphore(1)
    return _gate

# Every entry point now shares one permit:
async with gpu_gate():
    await run_gpu_thread(infer, ...)      # generation
async with gpu_gate():
    await run_gpu_thread(upscale, ...)    # upscaling

def reset_gpu_gate_for_tests() -> None:
    """Process scope has a process lifetime -- including the test process.
    Autouse fixture drops the singleton so it rebinds per test loop."""
    global _gate
    _gate = None
The permit outlives the caller's patience·python
# THE CANCEL RACE: releasing the permit when the AWAIT unwinds.
async with gpu_gate():
    await asyncio.to_thread(infer_sync, ...)
# task.cancel() -> the await unwinds NOW -> the `async with` exits NOW
# -> permit free. The worker thread is still mid-step on the device.
# Next job acquires the permit legally and stacks a second inference.

# THE FIX: hold the permit until the THREAD exits, not until we stop waiting.
async def run_gpu_thread(fn, /, *args, record=None):
    inner = asyncio.ensure_future(asyncio.to_thread(fn, *args))
    try:
        return await asyncio.shield(inner)        # cancel can't kill `inner`
    except asyncio.CancelledError:
        if record is not None:
            record.cancel_requested = True        # the step callback reads this
        while not inner.done():                   # wait for the device to be free
            try:
                await asyncio.shield(inner)
            except asyncio.CancelledError:
                continue                          # absorb repeated cancels
            except BaseException:
                break
        raise                                     # only now propagate

# Release is now a statement about the device, not about our control flow.

External links

Exercise

Find a lock or semaphore in code you know and answer the two audit questions on paper. First, list every code path that touches the protected resource and mark which ones acquire the gate — the unmarked ones are leak one. Second, trace what happens if the holder is cancelled or times out mid-operation: does the release wait for the operation, or for your function to exit? If it's the latter, write the sequence that produces two simultaneous holders.
Hint
Leak one is found by grepping for the resource, not for the lock — the bypassing caller never mentions the lock, which is why searching for the lock can't find it. Leak two is found by asking what a cancellation actually reaches: awaits, yes; threads, native calls, and remote work, no.

Progress

Progress is local-only — sign in to sync across devices.
← PreviousGround Truth and the Dev MirrorNext →Quiz · 4 questions
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.