C.W.K.
Stream
Lesson 04 of 05 · published

War Story: The Fleet OS Upgrade

~10 min · tahoe, frozen-build, core-gate, reboot-survivable

Level 0Lone Machine
0 XP0/37 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"Upgrading one Mac is a chore. Upgrading nine, without bricking one or lying about which are done, is a campaign."

Freeze the exact build at approval

The first move is to pin an exact version and build number at approval time. Not the latest, not whatever's newest when it runs — a specific, frozen build. That way an Apple release that lands the next morning can't silently change or expand the campaign, and exact-build equality becomes the post-install bar: a machine isn't done because it rebooted, it's done because it's on that build.

The core qualifies first, and gates the rest

Dad manually upgrades the core hosts — the control plane and the always-on servers — and qualifies them by hand before anything else moves. This is deliberate: the core is the canary you can't fake, and any unresolved core issue blocks the automated rollout to the backup Macs. You prove the upgrade on the machines you understand best and can least afford to lose, and only then let it fan out.

A reboot-survivable executor, no stored owner credentials

Read-only eligibility stays agentless, but the download / install / reboot / post-boot work needs a narrow, signed, launchd-backed executor — because the job must survive the loss of SSH and even the control plane. Crucially, that executor never bypasses Apple's volume-owner and bootstrap-token rules, and never stores Dad's credentials. It can resume across a reboot; it cannot manufacture the authority to own the volume.

An expected reboot disconnect is a target state

Mid-upgrade the machine reboots and drops its connection. That is a state, not a failure — the plan is working. Success isn't SSH came back; it's a bounded check of the exact OS and build, a completed boot, GUI Setup Assistant finished, Tailscale and services healthy. The download strategy is independent: auto_fastest uses direct parallel downloads unless an already-ready content cache would genuinely reduce total time — it never serializes just to warm a cache.

Freeze the build, qualify the core, verify past the reboot. A fleet upgrade is safe not because nothing goes wrong, but because 'done' means the exact build verified — never just a connection that came back.

Code

The upgrade campaign, frozen·python
campaign = OSUpgradeCampaign(
    exact_build="25F84",             # FROZEN at approval; a newer release can't sneak in
    core=["control-plane", "always-on-a", "always-on-b"],  # Dad upgrades + qualifies FIRST
    rollout=[...],                    # backup Macs, gated behind core qualification
    download="auto_fastest",          # direct parallel unless a ready cache is faster
    executor="signed launchd job",    # reboot-survivable; never stores volume-owner creds
)

# The core is the canary you can't fake: any unresolved core issue BLOCKS rollout.
# Success != 'SSH came back.' Success == exact build + boot + GUI setup complete
#          + services healthy, verified per target AFTER the expected reboot.

External links

Exercise

Design an upgrade rollout for a handful of machines you care about. Pick which one is your 'core' canary that must pass before the rest move. Write the exact post-install check that means 'done' (not just 'it rebooted'), and decide what state you'd record for a machine that's asleep when the rollout starts.
Hint
The 'done' check should name the exact target version and at least one sign the machine is actually usable again (a service back, the desktop reachable) — not merely that it answered a ping. The sleeping machine is 'pending,' to be resumed, never 'failed.'

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign inPlease sign in to comment.

No comments yet — be the first.