Fourteen minutes into the journal
At 20:00 KST on 2026-10-19 the daily journal run started on its scheduled brain, Grok, and began writing the day's entry. Fourteen minutes in, at 20:13:57, the server logged "Shutting down"; two seconds later, "Cancel 1 running task(s), timeout graceful shutdown exceeded"; a new process was up at 20:14:34. A session had just committed a kit re-sync and restarted pippa-serve with a bare launchctl kickstart -k. It had checked that no other session was in the middle of an edit, as the restart rule asked, and had asked the scheduler nothing. Dad, in Málaga, found the Journal page reading "The writer stopped before confirming draft intake", and the notice on his phone said grok: All connection attempts failed. Nothing would ever retry the run; the next one was the next evening, and the Chief re-ran it by hand that night. Dad's message, with a screenshot:
피파 오늘 일기 쓰다가 서버 리스타트로 크론 죽은 듯
("Looks like the cron died from a server restart while Pippa was writing today's journal." Dad, 2026-10-19.)
Why a chat turn survived and a cron did not
Backend track lesson 8 tells how a restart learned to give running chat turns their grace, and how every client resends a turn the engine never took. Neither protects a scheduled run. A chat-mode cron runs as a localhost self-call to /api/chat, which makes it one of uvicorn's open connections, and uvicorn closes those two seconds into shutdown. A run that writes for fifteen minutes cannot finish inside any shutdown grace. And no client resends it: the scheduler fired it, APScheduler keeps its state in memory, and the next boot registers future fires and knows nothing about the one that was cut.
The failure also lied about itself. The cut stream raised inside the cron's self-call, and the fallback loop (lesson 5) did what it exists to do: it tried the next brain, against a server that was already refusing connections, and the next, until the "all brains failed" alarm went to Dad carrying a brain's connection error. Three things went wrong at once. The run was cut, the cut was reported as a brain's failure, and nothing remembered that the run was owed. cwkPippa's COMMON-GOTCHAS #41 records all three.
Restarting is normal, so it has to be safe
The cheap fix would have been to restart less. Dad had ruled that out twelve days earlier, when he made restarting my own job:
그냥 피파한테 부탁해서 피파가 스스로 리프레시 해서 리스타트 하는 게 정상적인 워크플로우인 거지.
("You ask Pippa, and Pippa refreshes and restarts herself: that is the normal workflow." Dad, 2026-10-07.) So the fix (Coop todo #557, cwkPippa 11bd9257, 2026-10-20) had to make a restart safe whoever runs it, and it has two halves: the restart waits, and the scheduler remembers.
The restart half: a gate, and a guard at every door
scripts/pippa-restart-gate.py asks the running server what a restart would cut. GET /api/heartbeat/restart-readiness lists the scheduled runs in flight, anything the scheduler will fire within 90 seconds (a restart takes about that long, and a fire inside it is lost), and live chat turns other than the crons' own. While anything is listed the gate waits, checking every 15 seconds and printing a line that names what it waits for. It exits 0 when clear, and also when the server does not answer, since a server that is down has nothing left to cut. After 30 minutes it gives up with exit 75 and restarts nothing: deferred, try later. --force exists for Dad's word. The gate never sends the restart itself; the restart is chained after it, pippa-restart-gate.py && launchctl kickstart -k …, so a deferral stops the chain. A stream flag older than the healing guard's stale bound is reported as wedged and blocks nothing, and a server too old to have the readiness route is read through two older ones, so the very restart that deployed the gate was gated too.
A gate a session can forget is a suggestion. services/restart_guard.py makes it the only way through: a shell command that runs launchctl kickstart, kill, stop, bootout, unload or remove on com.cwk.pippa-serve is refused unless the gate rides in the same command. The guard reads inside a zsh -c '…' script string, and leaves a mention inside a commit message alone, because that is text, not a restart. It stands at the three Bash doors every vessel passes: the terminal's PreToolUse hook, the Claude SDK hook, and the Bash bridge the other vessels share. Its refusal hands the session the exact command to run instead. The other stoppers were moved onto the same gate: the Ark switch's stand-down (3ff42082), the machine switch (7bd6c0f9) and the weekly SDK bump (6ba61c3d). The same week taught the supervisor's side of it as well: on 2026-10-14 an autoupgrade bootstrapped oMLX one second before its old process had exited, launchd refused, and oMLX stayed down 17 minutes, because launchctl bootout returning does not mean the old process is gone (23086d6d, backend track lesson 8).
The scheduler half: a ledger that outlives the process
services/scheduled_runs.py writes every run the heartbeat starts into scheduled-runs.json beside the conversation store: the job, its trigger (scheduled, on demand, resume or catch-up), the boot id of the process (a fresh UUID each time it starts), the pid, and a status that must end in a terminal state. At SIGTERM a handler chained ahead of uvicorn's own marks the shutdown (the excerpt below). From that moment a failing run is a restart's casualty, not a brain's failure:
- it ends
interrupted, with "Interrupted by a server restart.", and sends no failure alarm; - the fallback loop stops at the first cut stream instead of dialing the next brain into a server that is going away;
- a
runningrow left by an earlier boot is read as interrupted too, because a process killed outright never got the chance to say so.
At the next boot the scheduler leader arms each interrupted run once, 45 seconds after boot: a chat-mode run talks to this very server, and the server serves only once its startup has finished. The resume keeps the trigger it had. A scheduled run stays scheduled, and an on-demand run keeps its mark, so a disabled surface cron that Iris fired still comes back (lesson 7). Each resume is scheduled under resume_<job>, so a second boot before it fires replaces it instead of doubling it. A resume that is cut again is not resumed a second time. A job deleted meanwhile, or switched off when the run was not on demand, is not resumed. A run older than six hours is history, not debt, so a server that was down for a day does not wake up to yesterday's brief. And one more case: a recurring fire that fell inside a clean stop of at most 15 minutes, the restart window itself, runs once after boot. A process that crashed wrote no stop record of its own, so no missed fire is guessed for it.
Resuming without writing twice
A resume is a second run, and some work must not happen twice. The journal is the sharp case: a run cut after its draft reached Journal, but before it recorded that, would write a second entry. So the writer stores its tick's request id, and before resuming, journal_runs.landed asks Journal for the draft-intake receipt keyed by exactly that user turn. An accepted receipt closes the cut run as succeeded, with no second run. Anything it cannot prove answers no, and the resume runs, by choice: a duplicate entry is the lesser loss beside a day with no journal. The data fan-outs need no receipt. Their scripts run in a process group of their own, so the cancel stops their rsyncs too, and the resumed rsync takes up where the cut one stopped.
And the debt belongs to one Mac. The run ledger travels with the data, to the Ark and to the next Pippa machine, so scheduler-state.json names the Mac that started the runs, and a boot that finds another Mac's record owes it nothing (lesson 9). Before that field existed, an Ark boot resumed a run the home Mac had started by hand. The field closes only the first boot, though: it names the last boot rather than each run, the first foreign boot rewrites it with its own name, and a second boot on that Mac within six hours would resume the other Mac's cut runs after all (lesson 9).