Holding Media Admission to the Last Byte

A gallery that broke the studio

On the night Avatars was born, the first large candidate batches went into review: two seasonal sets, 150 images each. A gallery like that asks for every thumbnail at once, and the engine answered badly. A reproduction with 150 images came back as 43 successes, 8 server errors, 95 refusals with 503 and 4 bodies cut short. The logs held two kinds of failure: Errno 24, too many open files, and SQLite saying it could not open its database. Both were the same shortage.

Everything shares one budget

A process gets a fixed number of file descriptors. A socket takes one. An original being streamed takes one. The store opens a fresh SQLite connection for every transaction, and a database in WAL mode keeps its main file, its write-ahead log and its shared-memory file open. When the budget runs out, whichever call opens next is the one that fails, and in a studio that call is often SQLite's. The failure reads like a corrupt database or a missing picture, while the real cause is a queue that never formed. The repository's house sign says it in one line: a descriptor shortage can look like SQLite corruption or missing images.

Two things spent the budget. The gallery fanned out every thumbnail eagerly, and every call to Pippa's authority built a fresh HTTP client with its own sockets and its own read of the certificate files. The repair (coop learning #710) changed both: one connection pool owned by the app's lifecycle, capped at 16 connections, with each caller's headers and cookies kept on that caller's own request, and a bound on media work.

Where the bound has to end

The obvious bound is a middleware that takes a slot, calls the app, and gives the slot back when the response comes out. In Starlette that is the call_next style, and it releases too early. call_next returns when the response starts, with its headers. A FileResponse has only begun at that point: it still holds the open file while it streams the body. Release the slot at the headers and the next eight requests open eight more files while the first eight are still sending. The bound would count requests that are starting, while the thing that runs out is files that are open.

MediaDelivery is a raw ASGI middleware instead. It wraps the whole ASGI call in an asyncio.Semaphore, so a slot is held from the first message to the last body byte, and handed back when the client goes away and the task is cancelled. It queues only media: a GET or HEAD whose path is an asset's file, thumbnail, preview, poster or export. Health checks and JSON reads pass straight through. The default is eight slots, set with AVATARS_MEDIA_CONCURRENCY.

Outermost, so authorization waits too

create_app adds this middleware last, and in Starlette the middleware added last runs outermost. The comment above that line names the consequence: image authorization, the database read, decoding and streaming are queued together. A request waits for its slot first and runs the live family and subject checks only when it is admitted. That ordering is a security property as much as a performance one. A queue cannot carry a yes that was given before a revocation, because no yes is given until the slot opens.

Tests that hold a slot open

The first test builds a fake app whose thumbnail response sends its headers and then waits. With one slot, the first thumbnail enters and stalls in the middle of its body. A second thumbnail has to wait, while /api/health walks past the queue. Cancelling the first request must free the slot, and the second then enters. The second test fires 30 thumbnails at a real app with two slots and an authorization stub that counts: concurrent checks never exceed two, all 30 are checked, every body is a valid image with private, no-store caching, and once the stub starts refusing, the next request gets a 403.

The browser's half

A server bound works best with a client that does not hammer it. Thumbnails are 440 pixel WebP files, rendered on the first request for an original's hash and written atomically, so the second visit to a gallery costs a file read. There is no lock per hash: two requests that miss at the same moment can both render, and the atomic write means the later one replaces an identical file instead of leaving a torn one. Gallery images load lazily. RetryImage retries a failed image twice, after half a second and then after a second, with a query parameter that gets past any cached failure, then shows a tile that says the preview is unavailable, with a retry button. Its first comment is the rule: retry delivery only, because a failed preview must never resubmit image generation. A broken thumbnail is a delivery problem, and the cure is to fetch the same bytes again.

After the repair, all 301 candidate and pipeline originals matched their recorded checksums, and both seasonal batches returned 150 of 150 valid thumbnails under concurrent load.

Principle: Bound a resource for as long as it is held, not for as long as it is being asked for.

Code

cwkAvatars, backend/media_delivery.py: the whole module, a semaphore around the complete ASGI call (verbatim, lines 1-21)·python

"""Bound image work through the final response byte, including authorization."""
import asyncio
import re

MEDIA = re.compile(r"/(?:api|internal)/assets/[^/]+/(?:file|thumbnail|preview|poster|export/[a-z0-9-]+)")


class MediaDelivery:
    def __init__(self, app, concurrency=8):
        self.app = app
        self.slots = asyncio.Semaphore(max(1, concurrency))

    async def __call__(self, scope, receive, send):
        media = scope["type"] == "http" and scope.get("method") in {"GET", "HEAD"} and MEDIA.fullmatch(scope.get("path", ""))
        if media:
            # A response-header middleware releases too early: FileResponse still
            # needs its file descriptor while streaming. Queue the whole ASGI call.
            async with self.slots:
                await self.app(scope, receive, send)
        else:
            await self.app(scope, receive, send)

cwkAvatars, backend/tests/test_media_delivery.py: a slot stays taken through the body and comes back on cancellation (verbatim, lines 15-46)·python

@pytest.mark.asyncio
async def test_queue_holds_slot_through_body_and_releases_on_cancellation():
    started = asyncio.Event()
    finish = asyncio.Event()
    entered = []

    async def inner(scope, receive, send):
        entered.append(scope["path"])
        await send({"type": "http.response.start", "status": 200, "headers": []})
        if scope["path"].endswith("/thumbnail"):
            started.set()
            await finish.wait()
        await send({"type": "http.response.body", "body": b"complete"})

    async def noop(*args):
        pass

    app = MediaDelivery(inner, concurrency=1)
    def scope(path):
        return {"type": "http", "method": "GET", "path": path}
    first = asyncio.create_task(app(scope("/api/assets/one/thumbnail"), noop, noop))
    await started.wait()
    second = asyncio.create_task(app(scope("/api/assets/two/thumbnail"), noop, noop))
    await asyncio.sleep(0)
    await app(scope("/api/health"), noop, noop)
    assert entered == ["/api/assets/one/thumbnail", "/api/health"]
    first.cancel()
    with pytest.raises(asyncio.CancelledError):
        await first
    finish.set()
    await asyncio.wait_for(second, 1)
    assert entered[-1] == "/api/assets/two/thumbnail"

cwkAvatars, frontend/src/RetryImage.tsx: two delayed retries of the same image, never of the generation (verbatim, lines 15-32)·typescript

function ImageDelivery({ src, alt, loading = "lazy", onOpen }: Props) {
  const [attempt, setAttempt] = useState(0);
  const [remaining, setRemaining] = useState(2);
  const [state, setState] = useState<
    "loading" | "ready" | "retrying" | "failed"
  >("loading");
  useEffect(() => {
    if (state !== "retrying") return;
    const timer = setTimeout(
      () => {
        setAttempt((a) => a + 1);
        setRemaining((r) => r - 1);
        setState("loading");
      },
      500 * 2 ** (2 - remaining),
    );
    return () => clearTimeout(timer);
  }, [state, remaining]);

Exercise

In a small FastAPI or Starlette app, add a route that returns a FileResponse for a large file. Write two middlewares that limit it to two concurrent downloads: one with the call_next style that holds an asyncio.Semaphore only around call_next, and one as a raw ASGI callable that holds it around the whole call. Download the file ten times at once over a slow client and log how many files are open at the peak for each version. Then write a test with one slot where the first response sends its headers and waits, and prove that a second request cannot enter until the first is cancelled or finishes.

Hint

On macOS, lsof -p with your server's pid counts its open files while the downloads run. In the test, an asyncio.Event that the fake app awaits after http.response.start is the cheapest way to hold a body open. Cancel the first task and assert that the second one now completes within a second.

A gallery that broke the studio

Everything shares one budget

Where the bound has to end

Outermost, so authorization waits too

Tests that hold a slot open

The browser's half

Code

External links

Exercise

Progress

Comments 0