C.W.K.
Stream
More →
Lesson 02 of 04 · published

Accept Concurrently, Run One at a Time

~12 min · semaphore, serialization, gpu, concurrency, memory

Level 0Tool Renter
0 XP0/33 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"The API can juggle a hundred requests. The GPU can run exactly one. Those are two different numbers, and the design has to honor both."

Two Kinds of Concurrency, Often Confused

An async server is built to accept many requests at once — that's what keeps it responsive. The trap is assuming that 'accept many' should mean 'execute many.' For most web work it does. For GPU inference it absolutely must not: the API layer should stay happily concurrent while the actual GPU work runs strictly one job at a time. Conflating these two is how a responsive server turns into a crashed one under load.

Why Two Inferences at Once Is the OOM Again

Recall the previous lesson: a single diffusion inference has a real peak memory cost. Run two simultaneously and you double that peak. On a finite device, doubling peak memory is the exact road back to the out-of-memory crash you just fixed. The inference_mode fix bounded one job's memory; running two jobs at once unbounds it again. Concurrency at the GPU undoes the memory discipline everywhere else.

Separate admission from execution. Accepting a job and running a job are different concerns with different limits. Admit liberally (stay responsive); execute within the resource's real capacity (stay alive). A queue between the two is what lets each obey its own correct limit.

The Semaphore of One

The mechanism is small: a concurrency gate that permits exactly one holder at a time, wrapped around the GPU section of the adapter. The job queue still accepts everything and runs all the non-GPU work concurrently — validation, queueing, response streaming. But when a job reaches the actual inference, it must acquire the single permit. The second job waits at that gate until the first releases it. The API never blocks; only the GPU section serializes.

The gate belongs around the scarce resource, not around the whole request. If you serialize the entire request handler, you've made the server single-threaded and killed responsiveness. Wrap the semaphore only around the GPU-bound section. Everything before and after that section stays concurrent — the gate is as narrow as the scarcity it protects.

Why Not Just a Bigger Machine?

You could buy more memory and run more concurrent inferences — but that just moves the ceiling, it doesn't remove it. However much memory you have, some number of concurrent inferences exceeds it, and without a gate you'll find that number the hard way, in production. The semaphore makes the engine correct at any memory size: it never tries to run more GPU work than the device can hold, regardless of how big the device is. Correctness beats capacity.

More capacity hides the bug; it doesn't fix it. Throwing hardware at an unbounded-concurrency problem raises the crash threshold without removing the crash. The system still has no principled limit — it just fails later, and more confusingly. Fix the bound in the design, then enjoy the extra capacity as headroom, not as a substitute for the fix.

Pippa's Confession

My async instinct screamed 'parallelize the slow part!' — the GPU work is the slowest thing, so surely run more of it at once. Dad pointed out I was parallelizing the one thing that physically can't be doubled without doubling memory. The slow part is slow and scarce; those need opposite treatment. I learned to ask not just 'is this slow?' but 'is this scarce?' — and to serialize the scarce thing even when every fiber of async-me wants to fan it out.

Code

The gate wraps only the scarce section·python
import asyncio

class LocalAdapter(Adapter):
    def __init__(self):
        # Exactly one permit. Only ONE GPU inference runs at a time.
        self._gpu = asyncio.Semaphore(1)

    async def run(self, request):
        # Everything up to here is concurrent: many requests in flight.
        prepared = await self.prepare(request)      # concurrent OK

        async with self._gpu:                        # the narrow gate
            # Only the GPU-bound section is serialized. Job 2 waits HERE
            # until job 1 releases — never doubling peak device memory.
            result = await self._infer(prepared)

        return await self.finalize(result)           # concurrent again

External links

Exercise

Find a resource in one of your systems that is scarce, not just slow (a single GPU, a connection pool of size N, a rate-limited external call). Is your code admitting work and executing work at the same concurrency level? Sketch where a semaphore gate would go to separate the two — and what concurrency number the gate should hold.
Hint
The gate's number is the real capacity of the scarce resource: 1 for a single GPU's memory, N for a pool of N connections. Admission stays unbounded (or bounded only by a queue); execution is bounded by the gate.

Progress

Progress is local-only — sign in to sync across devices.
← PreviousThe 75 GB SurpriseNext →Loading FLUX Without the Repo
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.