Locks and Idempotency

"In distributed work, 'it ran twice' isn't a rare bug. It's Tuesday. Design so twice equals once."

Retries are the normal weather

Connections drop mid-command. Workers restart and re-pick their queue. A human clicks the button again because the spinner looked stuck. None of these are edge cases — they're the ordinary conditions of any work that crosses a network. If your operation is only correct when it runs exactly once, it isn't correct.

Per-host locks: one mutation at a time on a machine

A per-host mutation lock ensures two operations can't act on the same Mac simultaneously — an update and a reboot shouldn't interleave on one machine. Crucially, the lock is scoped to the host, not the whole fleet, so independent machines keep running in parallel. You serialize contention, not the whole world.

Idempotency keys: the same request twice is one effect

Every mutating request carries an idempotency key. The executor remembers keys it has already applied, so a replayed request — from a reconnect, a retry, a double-click — returns the first result instead of doing the thing again. One key, one effect, no matter how many times the request arrives.

Prefer commands you can verify apart from their exit code

An exit code can lie: a command can time out after it already succeeded, or return zero while its real effect never landed. So prefer operations whose success you can check independently. After a restart, don't trust that SSH returned 0 — confirm the service is loaded and its last-run timestamp advanced. Verify the world, not the return value.

Make replay safe, then stop fearing retries. A per-host lock keeps two operations from colliding; an idempotency key keeps one operation from counting twice. Together they turn 'ran again' from a disaster into a yawn.

Code

Lock the host, key the request·python

request = {
    "op": "restart_service",
    "host": "server",
    "idempotency_key": "op_7f3a:server:restart",   # stable per intended effect
}

with per_host_lock("server"):                 # one mutation at a time on THIS host
    if executor.already_applied(request["idempotency_key"]):
        return executor.first_result(request["idempotency_key"])   # replay -> no-op
    result = executor.apply(request)
    executor.remember(request["idempotency_key"], result)

# A dropped connection + retry now produces ONE restart, not two.
# The lock stops a concurrent reboot interleaving on the same Mac.
# Independent hosts hold independent locks, so the fleet still runs wide.

Exercise

Take an action that would be bad to run twice (charge a card, send an email, reboot a machine). Design an idempotency key for it: what makes two requests 'the same request'? Then name one way you'd verify the effect actually happened without trusting the command's exit code.

Hint

A good key encodes the intended effect, not the moment: 'reboot host server for operation op_7f3a' is stable across retries; 'reboot at 09:03:11.442' is not — the retry has a new timestamp and looks like a fresh request, which is exactly the bug.

Retries are the normal weather

Per-host locks: one mutation at a time on a machine

Idempotency keys: the same request twice is one effect

Prefer commands you can verify apart from their exit code

Code

External links

Exercise

Progress

Comments 0