Skip to content
C.W.K.
Lesson 04 of 05 · published

Idempotent Capture by Request Id

~14 min · idempotency, request-id, concurrency, http

Level 0Mise en Place
0 XP0/43 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

A retry must never make a second plate

Picture a phone on a weak connection. It sends a plate. The engine stores it and answers, and the answer is lost on the way back. From the phone's side the request failed, so it sends the plate again. Without something to recognise the second copy, Dad's record now has the same dinner twice.

Sizzle's answer is a request id that the client mints before the first attempt and reuses on every retry. The web client builds one with newRequestId("plate"): the prefix, a dash, and 24 hex characters from crypto.randomUUID. The phone mints one with the kit's ClientRequestId when it puts the plate in its outbox, and that outbox entry's id is the request id for as long as the plate waits. The route accepts only ids matching ^[A-Za-z0-9_-]{8,80}$.

On the engine side, the id travels inside the encounter.captured event, and the mirror's encounters.request_id column is UNIQUE. Because the id lives in the journal, the deduplication survives a restart. The engine's first capture test checks exactly that: a replay with a different note returns the same plate, the list still holds one plate, and a fresh engine over the same data still finds it after rebuilding.

Three checks, three jobs

The same question, "have I seen this request id?", is asked three times on the way in. Each one sits where it does for a reason.

WhereWhenWhat it saves
The route, backend/app.pyBefore reading the body's photos or any staged idA retry whose staged photos were already released still gets its plate
capture, first lookBefore decoding and re-encoding the photosThe slow image work for a plain replay
capture, under the write lockJust before recordThe journal itself, when two copies race

The route's check carries a comment that explains it. Once a plate commits, the route releases its staged photos. A retry that arrives after that names staged ids that are gone, and take would refuse them. Answering the replay first means the retry lands anyway, with the stored plate.

Why the third check guards the journal

Now picture two copies of the same request arriving together: a client retry fires while the first attempt is still decoding photos. The route hands each one to asyncio.to_thread, so they run in separate threads. Both pass the first two checks, because nothing is stored yet. Both ingest their photos.

Remove the third check and both would call record. Remember the order from lesson 1: append first, then apply. The second copy's event would land in the journal. Then its apply would hit the UNIQUE request id and raise. I tried this in a scratch directory by recording the same encounter.captured twice. The second call raised IntegrityError, the journal held two lines, and a fresh Sizzle over that directory refused to open with the same error. One duplicate event would stop every future start.

The re-check under self._write closes that window. Whichever thread takes the lock second finds the first one's plate and returns it without writing anything. The comment in the code names it: a concurrent replay won the race. This is also why _write is re-entrant: capture holds it and then calls record, which takes it again.

One honest leftover. The losing thread already ingested its photos under its own encounter id, outside the lock. Those files stay on disk in a directory no event names. That costs disk space and nothing else, and it only happens in this race. The engine suite has no concurrent-capture test, so this guard rests on reading the code. No test exercises it.

Principle: In a journal-first system, deduplicate before the append. A constraint that only the projection enforces turns a duplicate into a journal that cannot replay.

A replay is an ordinary success

When a request id is already known, Sizzle returns the stored plate as an ordinary success. There is no 409 and no special error for the client to handle. From the client's side, the retry simply worked. Once a replay gets as far as the lookup, everything else in it is ignored: the stored note, kind and photos win, which is what the test's changed note checks. Getting that far has two gates, though. FastAPI validates the form fields before the handler runs: the id's pattern, a note of at most 4000 characters, a timezone of at most 64, latitude and longitude in range. Then the route checks kind before it looks the id up. A replay that fails either gate is refused with an error even though its plate is stored. A client that resends exactly the body it sent the first time passes both, and that is the retry this design is for.

The same shape appears wherever a client can resend. Conversation turns carry their own request_id, unique in the turns table (track 6). Staged photo uploads are idempotent on a client key (lesson 5). The rule underneath is the one http-rest-quest's Semantics track spells out for idempotent methods: sending a request twice must leave the server where sending it once would.

Two more decisions happen on the capture path. _capture_moment decides when the plate happened: the camera's own clock first, then the client's captured_at, then now. And the first photo that carries its own GPS wins over the place the client sent. Lesson 5 explains where those facts come from.

Code

cwkSizzle, backend/domain.py: Sizzle.capture, the replay check before the image work and again under the write lock (verbatim, lines 77-107)·python
        existing = self.store.encounter_by_request(request_id)
        if existing is not None:
            return existing
        if kind not in KINDS:
            raise DomainError(f"kind must be one of {', '.join(KINDS)}")
        if not files:
            raise DomainError("an encounter needs at least one photo")
        encounter_id = new_ulid()
        photos = [self.photos.ingest(encounter_id, new_ulid(), name, data) for name, data in files]
        moment, zone = _capture_moment(photos, captured_at, timezone_name)
        with self._write:
            again = self.store.encounter_by_request(request_id)
            if again is not None:  # a concurrent replay won the race
                return again
            place = next((p["gps"] for p in photos if p.get("gps")), place)
            self.record(
                "encounter.captured",
                {
                    "id": encounter_id,
                    "request_id": request_id,
                    "captured_at": moment.isoformat(),
                    "local_date": moment.date().isoformat(),
                    "timezone": zone,
                    "kind": kind,
                    "note": _text(note),
                    "place": place,
                    "photos": photos,
                },
            )
        self.on_reading_requested()
        return self.store.encounter(encounter_id)
cwkSizzle, backend/app.py: the capture route answers a replay before it reads anything, and releases staged photos only after the commit (verbatim, lines 203-222)·python
        if kind not in KINDS:
            raise DomainError(f"kind must be one of {', '.join(KINDS)}")
        # A replay answers the stored plate before anything is read, so a retry
        # after its staged photos were released still lands.
        existing = store.encounter_by_request(request_id)
        if existing is not None:
            staged.release(staged_ids)
            return encounter_payload(existing["id"])
        files = [(upload.filename or "photo", await upload.read()) for upload in photos]
        try:
            files += [(name, data) for name, data, _ in staged.take(staged_ids)]
        except StagedUploadError as exc:
            raise DomainError(str(exc)) from exc
        encounter = await asyncio.to_thread(
            sizzle.capture, request_id=request_id, kind=kind, note=note, captured_at=captured_at,
            timezone_name=timezone, files=files,
            place=None if lat is None or lon is None else {"lat": round(lat, 6), "lon": round(lon, 6)},
        )
        staged.release(staged_ids)
        return encounter_payload(encounter["id"])

External links

Exercise

Pick a create endpoint in your own software that a mobile or flaky client calls. Add a client-minted request id, store it with the record, and return the stored record when the same id comes back. Then write a test that fires the same request from two threads at once and asserts there is exactly one record afterwards, in your event log as well as your table.
Hint
A threading.Barrier lets both threads pass your first check before either writes, which is the case you want to provoke. If your writes go log-first, count lines in the log, not only rows in the table: the table can look right while the log already holds the duplicate.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.