Resumable per Target

"A campaign is not a transaction that all-or-nothing succeeds. It's a set of tracks, each of which can be picked up where it stopped."

Many resumable tracks, not one transaction

It's tempting to think of a fleet campaign as one big all-or-nothing thing that either fully works or fully fails. That framing is wrong and dangerous. A campaign is a set of per-target tracks, each with its own state, each resumable from where it left off. Losing one target should never mean redoing the seven that finished.

A sleeping Mac is pending work

An expected reboot disconnect. A laptop asleep in a bag. These are target states, not failures. The campaign records them honestly — pending, not broken — and resumes the moment the host comes back. Treating an intentional absence as a failure is how you turn a calm, resumable rollout into a false alarm that trains you to ignore the real ones.

Isolated failure vs systemic failure

When a target does fail, the question is: is this its problem, or everyone's? An isolated failure — one host, its own reason — becomes a repair task, and the healthy peers keep finishing. A systemic failure — failures that share a cause, the kind that will break every remaining target the same way — triggers a narrow hold: pause the unstarted targets so you don't repeat the same break nine times. One is a repair; the other is a stop.

No heavyweight rollback by default

High-risk campaigns get explicit exception handling and that narrow systemic-failure rule — but they do not require a full rollback pipeline by default. And some things simply aren't reversible: a major OS downgrade is never pretended to be an automatic rollback. Honesty about what can't be undone is safer than a rollback button that lies.

Resume per target; pause only what shares the failure. A sleeping machine is work to finish, an isolated break is a repair, and only a shared cause earns a stop — nothing here is ever all-or-nothing.

Code

Each target resumes from where it stopped·python

for host in campaign.targets:
    state = campaign.state_of(host)
    if state == "succeeded": continue            # done; don't touch it
    if state == "pending":   resume(host)         # sleeping/disconnected -> pick up later
    if state == "failed":    repair_task(host)    # isolated: fix this one; peers roll on

# A SHARED cause is the only thing that halts the rest:
if systemic_failure(campaign):
    pause_unstarted(campaign)   # narrow hold -- don't break every remaining target identically

# There is no automatic 'undo the whole campaign.' Some steps (an OS
# downgrade) are not reversible, and the system never pretends otherwise.

Exercise

Design a batch job over ten items that can be interrupted at any point (a laptop sleeps, the process is killed). Write how each item records its own state so a re-run finishes only the unfinished ones. Then decide: which single failure would make you pause the whole batch, and which would you just log and continue past?

Hint

The 'pause the whole batch' failure is the systemic one — a shared cause that will hit every remaining item identically (the API is down, the disk is full). Everything else is per-item: log it, keep going, and let the re-run pick up exactly what's left.

Many resumable tracks, not one transaction

A sleeping Mac is pending work

Isolated failure vs systemic failure

No heavyweight rollback by default

Code

External links

Exercise

Progress

Comments 0