One Identity, One Paid Call

"A hundred identical requests at once should cost exactly one paid call. The other ninety-nine wait for it."

The Burst Problem

A content-addressed cache stops you from paying twice in sequence: the second request, later, is a hit. But what about ten identical requests arriving in the same instant, before any of them has finished and populated the cache? A naive engine lets all ten see a cache miss and all ten make a paid call — ten charges for one piece of audio. This is the classic thundering-herd, and money makes it sting.

Single-Flight on the Identity

Bellows guarantees one cache identity has at most one in-process paid synthesis at a time. The first request to miss the cache starts the paid call and registers itself as in-flight for that identity; every other request with the same identity attaches to the same in-flight call and awaits its result instead of starting its own. One payer, many riders. When it completes, the cache is populated and the identity is cleared for next time.

Scope the Lock to the Key, Not the Engine

The subtlety is granularity. You don't want a single global lock that serializes all synthesis — that would make ten different sentences queue behind each other for no reason. The lock is per-identity: different identities run fully in parallel, and only identical ones collapse onto a single call. The key you built in the last lesson is exactly the right lock name.

Code

Single-flight, keyed by the synthesis identity·python

import asyncio

_inflight: dict[str, asyncio.Future] = {}
_guard = asyncio.Lock()

async def synthesize_once(identity: str, do_paid_call) -> Audio:
    if hit := cache.get(identity):
        return hit                              # fast path: already cached

    # Single-flight: at most ONE paid call per identity at a time.
    async with _guard:
        fut = _inflight.get(identity)
        if fut is None:
            fut = _inflight[identity] = asyncio.ensure_future(do_paid_call())

    try:
        return await fut                        # every rider awaits the one payer
    finally:
        async with _guard:
            _inflight.pop(identity, None)        # cleared for next time

# Different identities never block each other -- only identical ones collapse.

Exercise

Find a place in a system you know where a burst of identical requests could each trigger the same expensive work before the first one caches its result — a cold cache on deploy, a popular key expiring, a webhook retried in parallel. Describe the duplicate cost, then sketch the single-flight fix and name what you'd use as the lock key.

Hint

The lock key is whatever makes two requests 'the same work.' If you built a correct cache key already, you've already built the lock key — they're the same string.