Lease Token + Heartbeat

"A worker doesn't own a job. It rents one, on a timer, with a token that proves the rental is still valid."

The Problem: Who Owns This Job Right Now?

Multiple workers pull from one queue of long jobs. The instant more than one exists, a dangerous question appears: what stops two workers from grabbing the same job, or a worker everyone thought was dead from waking up and stomping on a job that was reassigned? If ownership is fuzzy, two workers transcribe the same video, or a zombie worker submits results for a job someone else is already redoing. You need a way to say, with certainty, 'this exact worker owns this job right now, and no one else.'

Claim Atomically, Prove With a Token

Recall's answer is a lease. Claiming a job happens inside a single atomic database transaction: the job moves from queued to leased, and the claiming worker receives a random lease token. That token is the proof of ownership. From then on, every action the worker takes against that job — heartbeating, reporting a failure, submitting the result — must present the token. The control plane checks it and rejects any mutation whose owner, token, or status doesn't match. A second worker that never claimed the job has no token; a worker whose lease has expired has a token the control plane no longer honors. Ownership stops being a matter of trust and becomes a matter of proof.

This is the fencing-token idea from distributed locking: a lock you merely 'hold' can be silently lost while you think you still have it, so the safe pattern is to carry a token that the resource itself validates on every write. A worker resumed from a stall might believe it still owns its job — but its token is stale, so its writes bounce. The token, checked at the point of mutation, is what makes a lease safe rather than a hopeful convention.

Heartbeats Keep the Lease Alive

A lease is time-limited on purpose — Recall uses a two-hour window — so a worker that vanishes can't hold a job forever. But real transcription work takes long, blocking stretches: preparing audio, waiting on the provider. To keep the lease from expiring mid-work, the worker sends a heartbeat — before the paid call, after every completed chunk, and, via a background heartbeater, even during long blocking operations. Each heartbeat renews the lease. The presence of heartbeats says 'I'm alive and still working'; their absence is precisely the signal the control plane uses to reclaim the job (next lesson). So the lease is a living thing: held by a token, kept warm by a heartbeat, and released — willingly or by timeout — the moment the worker can no longer prove it's there.

Code

Claim in a transaction, then prove ownership with the token·python

# Claim: atomic transition + a random token bound to this worker.
with db.transaction():
    job = db.claim_next_queued(worker_id)   # queued -> leased
    token = random_token()
    job.lease_token = token
    job.lease_expires_at = now_plus(hours=2)

# Every later action must present the token. Mismatch -> rejected.
control_plane.heartbeat(job.id, worker_id, token)   # renews the lease
control_plane.submit_result(job.id, worker_id, token, result)

# A stale/second worker without the current token cannot mutate
# the job. Ownership is proven per-action, not assumed once.

Exercise

Find a place in your own work where a worker, cron job, or process 'takes' a task and works on it — a queue consumer, a scheduled job, a lock file. Ask: if two of them ran, or one stalled and resumed, could they both act on the same task? Design a lease: an atomic claim, a token bound to the worker, and a validation of that token on every state-changing action. Note where a stale holder's write would now be rejected.

Hint

The vulnerability is any 'claim' that isn't re-proven at the moment of writing results. A worker can claim, stall past its lease, get its job reassigned, then wake and submit — corrupting the reassigned work. The fix is a token checked at every mutation, plus a heartbeat to renew ownership while alive. Ownership must be provable per-write, not just at claim time.