Skip to content
C.W.K.
Lesson 06 of 06 · published

A Worker With No Timer

~14 min · asyncio, workers, fail-closed, concurrency

Level 0Mise en Place
0 XP0/43 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

Nothing polls

The docstring of backend/reading.py ends with two words: no timer. The reading worker drains pending plates at startup and whenever a capture, a correction or a re-read request kicks it, and nothing wakes it on a schedule. That works because every way a plate can come to need reading is a write through the domain, and every such write rings the same bell.

What happenedWhat gets recordedWho kicks
Dad captures a platethe encounter, pending by defaultthe capture itself
Dad presses Read againreading.requestedrequest_reading
Dad corrects a readingthe correction, then reading.requestedrequest_reading
The engine starts under a newer rubricreading.requested for each stale platestart()
A new reading brain is savednothing newthe slot's after-update hook
A read ends and the plate is still pendingnothing newthe worker itself

One bell, any thread

The bell is an asyncio.Event. kick() sets it through call_soon_threadsafe, because captures run in a worker thread and an event loop's objects must not be touched from another thread. The loop waits on the event, clears it, and asks the store for every pending plate. For each one not already being read, it starts a task. Three details in that loop are easy to miss.

  • The scan decides the work, not the kick. A kick carries no plate id. Ten kicks before the loop wakes collapse into one scan, and the scan finds whatever is pending at that moment. The store is the queue.
  • The tasks set holds references. Each task is added to a set and removes itself when done. The event loop keeps only weak references to tasks, so a task nobody holds can vanish before it finishes; Python's documentation asks you to save a reference for exactly this reason.
  • Two at a time. A semaphore of READING_CONCURRENCY, two by default, keeps a burst of captures moving without stacking a queue of vision calls on cwkPippa. A _running set stops the same plate being read twice at once, and SIZZLE_READINGS_ENABLED=0 leaves the loop awake but idle, so pending plates simply wait.

A restart loses nothing either. Pending is a status in a mirror rebuilt from the journal, so when the engine comes back, start() kicks once and the scan finds every plate that was waiting or half read.

Fail closed, and say why

A read resolves the reading slot when it starts, never earlier. The vendored kit client has one bridge before it gives up. Every successful fetch from the family authority is saved as a last-known-good cache, and when the authority cannot be reached, the client serves that cache, marked stale, and the read goes ahead on the cached assignment. The worker records the brain, effort and model it used, but not the stale flag, so a reading made during an outage looks like any other in the journal. Only when there is no assignment to use does resolution raise BrainControlUnavailable: the authority is unreachable and no cache exists yet, the slot has no assignment even after the engine declares it again, or the authority marks the assignment invalid. Then the read stops before any model is called and records reading.failed with "No brain assignment for plate reading" and the reason. It never falls back to a brain it guessed. Invariant 10 is that sentence: every Pippa call runs on its function's slot, or not at all, and a stale copy of the slot's own row still counts as the slot. An engine test takes the fake authority down before the engine starts, so no cache is ever written, captures a plate, and asserts both halves: the plate shows the failure, and no reading was ever sent.

The other failures are the ones earlier lessons raise: cwkPippa unreachable or answering with an error, a reply the parser cannot read or the models reject, a photo that cannot be read from disk. Each is caught, logged, and recorded as reading.failed with its message cut to 500 characters. The plate page then shows "Pippa could not read this plate:" with that message, and a Read again button. The phone shows the same.

There is no retry with backoff

The architecture's operating rules say transport failures retry with backoff. The code has no such thing. A failed read is recorded once and stays failed. The worker only ever scans pending plates, so a failed plate is not picked up at the next start. A rubric bump does not bring it back either, since it re-requests finished plates only. Saving a new reading brain wakes the worker, which again finds pending plates only. A failed plate returns in two ways: Dad presses Read again, which records a fresh request, or Dad corrects the plate, which requests a re-read carrying his word.

The backoff sentence entered the architecture on the design day, and no commit has ever touched backoff in the engine. The same section says that while Pippa is unavailable new readings show as waiting; in the code, an unreachable cwkPippa is a failed reading with its reason on the page. Read both lines as design that was never built. The retry policy that exists is in Dad's hands: pressing Read again, or correcting the plate.

A correction in the middle of a read

One race is handled with care. Dad can correct a plate while I am still reading it. The worker counts the plate's corrections before it starts. If the read then fails at cwkPippa, the parser or a photo, and the count has grown, it records no failure: the correction already set the plate back to pending, and when the guarded read finishes it sees the plate still pending and kicks again. If the read succeeds and the count has grown, it records the reading, because it is real, and then requests another, because it predates his word. Either way the next read carries the correction. The guard has one gap. The branch for a missing brain assignment records reading.failed without comparing the count, so if a correction lands while the slot is being resolved and the resolution then fails, the plate stays failed: the correction set it back to pending, the failure set it to failed, and the guarded read finds nothing pending to kick. The correction is not lost. It is stored, shown, and carried by the next read Dad asks for.

Principle: Let writes wake the worker and let the store be the queue. A worker that scans for pending work on every kick needs no timer, survives a restart, and never holds a job anywhere except in the record.

Code

cwkSizzle, backend/reading.py: kick, the loop and the guarded read (verbatim, lines 320-353)·python
    def kick(self) -> None:
        """Safe from any thread (captures run in a worker thread)."""
        if self._loop is None:
            return
        self._loop.call_soon_threadsafe(self._wake.set)

    def running(self, encounter_id: str) -> bool:
        return encounter_id in self._running

    async def _run(self) -> None:
        semaphore = asyncio.Semaphore(max(1, settings.READING_CONCURRENCY))
        tasks: set[asyncio.Task[None]] = set()
        while True:
            await self._wake.wait()
            self._wake.clear()
            if not settings.READINGS_ENABLED:
                continue
            for encounter_id in self.sizzle.store.pending_encounters():
                if encounter_id in self._running:
                    continue
                self._running.add(encounter_id)
                task = asyncio.create_task(self._read_guarded(encounter_id, semaphore))
                tasks.add(task)
                task.add_done_callback(tasks.discard)

    async def _read_guarded(self, encounter_id: str, semaphore: asyncio.Semaphore) -> None:
        try:
            async with semaphore:
                await self.read(encounter_id)
        finally:
            self._running.discard(encounter_id)
            # A correction that arrived mid-reading left the plate pending.
            if encounter_id in self.sizzle.store.pending_encounters():
                self.kick()
cwkSizzle, backend/reading.py: one read, resolved at job start and failing closed (verbatim, lines 355-382)·python
    async def read(self, encounter_id: str) -> None:
        encounter = self.sizzle.store.encounter(encounter_id)
        if encounter is None or encounter["reading_status"] != "pending":
            return
        seen = len(self.sizzle.store.corrections(encounter_id))
        picks: dict = {}
        try:
            # The reading slot's brain, resolved at job start (fail-closed:
            # no reachable assignment means no reading, never a guessed brain).
            picks = await asyncio.to_thread(brain_control.resolve, brain_control.READING)
            images = await asyncio.to_thread(self._images, encounter_id)
            raw = await pippa.read_plate(INSTRUCTION, input_text(self.sizzle, encounter_id), images,
                                         brain=picks["brain"], effort=picks.get("effort") or "",
                                         model=picks.get("model"))
            reading = parse_reading(raw)
        except brain_control.BrainControlUnavailable as exc:
            logger.warning("reading %s has no brain: %s", encounter_id, exc)
            self.sizzle.record("reading.failed", {"encounter_id": encounter_id, "rubric": RUBRIC_VERSION,
                                                  "error": f"No brain assignment for plate reading: {exc}"[:500]})
            return
        except (pippa.PippaError, ReadingError, OSError) as exc:
            logger.warning("reading %s failed: %s", encounter_id, exc)
            # A correction that arrived mid-read keeps the plate pending: the
            # worker reads it again with Dad's word instead of showing a failure.
            if len(self.sizzle.store.corrections(encounter_id)) == seen:
                self.sizzle.record("reading.failed", {"encounter_id": encounter_id, "error": str(exc)[:500],
                                                      "rubric": RUBRIC_VERSION})
            return
cwkSizzle, frontend/src/pages/Plate.tsx: a failed reading and its Read again button, one of Dad's two ways back (verbatim, lines 116-123)·typescript
      {encounter.reading_status === "failed" && (
        <div className="reading-failed">
          <span>Pippa could not read this plate: {encounter.reading_error}</span>
          <button className="ghost" disabled={busy} onClick={() => void run(() => api.reread(id))}>
            <RefreshCw size={14} /> Read again
          </button>
        </div>
      )}

External links

Exercise

Build a small background worker for a job your app runs after a write: it waits on an asyncio.Event, scans a table for rows marked pending on every wake, runs at most two jobs at once, and records success or failure on the row. Resolve whatever the job depends on (a model choice, a credential) at job start, and record a failure with its reason when that is unavailable. Write three tests: a kick from a plain thread wakes it; with the dependency down, the row fails and no job is attempted; a change to the input during a job causes one more run instead of a failure.
Hint
Use loop.call_soon_threadsafe(event.set) from the thread. Keep each task in a set and remove it in a done callback. For the third test, make the fake job wait on a second event so the test can change the input while the job is in flight, then release it.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.