Durable Jobs

"The browser tab is not the operation. Close the tab; the deploy keeps its promise anyway."

Some Operations Outlive the Request

A birth, a deploy, a tree fan-out, a restart, a fetch, a commit, a push — these can take longer than a browser is willing to wait, and they touch the real world. Firelink runs them as durable jobs, not as work bolted to an HTTP request. Each run has an explicit state — planned, running, succeeded, failed, or needs-attention — and its progress and bounded output are persisted as it goes. Close the tab, lose the network, refresh the page: the run doesn't vanish and it doesn't silently restart. It's a durable record you can come back to.

Disconnects Don't Create Mystery Retries

The dangerous moment with any side-effecting operation is uncertainty: did it happen or not? A browser disconnect mid-deploy must never turn into 'I'll just try again' when the first attempt may have already pushed. Firelink handles this with idempotency keys — a double-click or a retry with the same key doesn't run the effect twice — and with locks that serialize conflicting operations by member, adapter, and target, so two deploys of the same thing can't interleave. A retry either resumes the existing run or creates a clearly linked new one; it never silently repeats an external side effect whose outcome is unknown.

'Needs-Attention' Is an Honest Ending

Notice that the states include needs-attention, not just success and failure. Some operations end in genuine ambiguity — a commit succeeded but the push failed, an external step returned something the verifier can't confirm. Collapsing that into 'failed' invites a blind retry that might duplicate the part that already worked; collapsing it into 'succeeded' hides a real problem. needs-attention is the honest terminal state that says 'a human should look before anything else happens here' — the same honesty as 'unknown' in the census, applied to operations.

Any operation with an external side effect must be a durable, idempotent, lockable run — never work tied to the request that started it. If closing a browser tab can leave you unsure whether a deploy happened, you don't have an operation, you have a coin flip. Durability plus an idempotency key turns 'did it run?' from a guess into a lookup.

Code

A durable run: stateful, idempotent, locked·python

def start_operation(plan, idempotency_key) -> Run:
    existing = runs.by_key(idempotency_key)
    if existing:
        return existing            # double-click / retry -> same run, no re-effect

    lock = locks.acquire(plan.member, plan.adapter, plan.target)  # serialize conflicts
    run = Run(
        id=new_run_id(),
        key=idempotency_key,
        state="planned",           # -> running -> succeeded | failed | needs-attention
    )
    runs.save(run)                 # persisted BEFORE the external effect starts
    spawn_durable(run, plan, lock) # survives browser disconnect; progress persisted
    return run

# commit ok but push failed?  ->  state = "needs-attention", not "failed":
#   retrying a 'failed' might re-commit; 'needs-attention' asks a human first.
# The browser only WATCHES this run. It never IS the run.

Exercise

Find an operation you've built or used that has a real side effect (charge, email, deploy, file write) and is triggered from a UI. Ask: if the user double-clicks, or the network drops mid-request and they retry, can the effect happen twice? If yes, design the idempotency-key + durable-run fix. Then find one ambiguous outcome that isn't cleanly success or failure, and give it its own 'needs-attention' state.

Hint

The double-effect bug hides wherever a retry re-sends the whole request. The fix is a client-generated key carried on every attempt, checked server-side before the effect runs. And the ambiguous-outcome state is usually 'step 1 of 2 succeeded' — name it instead of forcing it into success or failure.

Some Operations Outlive the Request

Disconnects Don't Create Mystery Retries

'Needs-Attention' Is an Honest Ending

Code

External links

Exercise

Progress

Comments 0