Expired Lease Reclaim

"Whatever the claim touched, the reclaim must untouch — all of it, atomically."

The Heartbeat Stops

A worker dies mid-transcription: power loss, a crash, someone restarting the machine. The heartbeats stop. The lease runs out. Now the control plane has to clean up, and this is where a subtle, brutal bug lives.

The obvious cleanup is: expired lease → set the job back to queued so another worker can claim it. Recall does exactly that, and it does it before any new claim is granted, so a stale lease can never be double-claimed. But 'set the job back to queued' is only half the story, and the missing half is what bites.

The Claim Touched Two Things, So the Reclaim Must Too

Remember what claiming did: it moved the job from queued to leased, and it moved that job's batch item from queued to transcribing. Two rows changed. So if the reclaim only resets the job, the batch item is left saying 'transcribing' forever — with nobody transcribing it. The queue is technically correct (the job is back, another worker takes it) while the batch's progress view is permanently lying: an item eternally in-progress, an ETA computed from a fiction, an operator staring at a batch that looks alive and isn't.

So Recall's reclaim returns both the job and its batch item to queued, atomically — one transaction, both rows, all-or-nothing. That word matters: a reclaim that resets one row and crashes before the other would recreate the exact orphan it was cleaning up. The general rule is symmetry: whatever state a claim mutates, the reclaim must restore, in a single atomic step. A partial reclaim isn't a smaller fix — it's a new inconsistency wearing the costume of a cleanup.

And Heal What Crashed Between the Cracks

Even with atomic reclaim, one gap remains: a crash landing in the narrow window between 'job completed' and 'batch item updated.' The job is genuinely done, but its item still reads transcribing — and no lease will expire to fix it, because the job isn't leased anymore. So Recall adds a second safety net: on startup and on job claim, it reconciles — it looks for completed jobs whose batch items were left mid-flight and repairs them. This is the reconciliation-loop instinct: rather than trusting that every transition was perfectly paired, periodically compare reality to what the records claim and heal the difference. Leases handle the crashes you expect; reconciliation handles the crash windows you didn't.

Code

Reclaim both rows atomically, then heal the crash windows·python

# The claim mutated TWO things:
#   job:  queued -> leased
#   item: queued -> transcribing

# So the reclaim must restore BOTH, in one transaction.
with db.transaction():                 # all-or-nothing
    job.status = 'queued'
    job.lease_token = None
    item.status = 'queued'             # <-- forget this and the
                                       #     item says 'transcribing'
                                       #     forever, with no worker

# Reclaim runs BEFORE any new claim is granted, so an expired
# lease can never be double-claimed.

# Safety net for the crash window between 'job done' and
# 'item updated': reconcile on startup and on each claim.
for job in db.completed_jobs_with_item_in_transcribing():
    repair(job.batch_item)   # reality vs records -> heal the gap

Exercise

Find a claim/lock/checkout in your own system and enumerate every row or field it mutates — the primary record, plus any status, counter, or progress field elsewhere. Now write the cleanup path and check: does it restore every one of them, in a single transaction? Then find the crash window your atomicity can't cover, and sketch the reconciliation pass that would heal it on startup.

Hint

Two questions find the bugs: (1) list everything the claim writes, then confirm the release path un-writes all of it atomically — the usual miss is a progress/status row in a different table. (2) Ask what happens if the process dies between two transitions that aren't in the same transaction. If nothing would ever notice, you need a reconciliation pass that compares reality to the records.