The FileProvider Stall

"It works when I run it by hand. It hangs when the service manager runs it. Same path, same code, same machine."

The Bug That Refuses to Reproduce

Here's a genuinely maddening one. Recall's code lives at a conventional fleet path, and on some machines that path is backed by a file-sync provider — the kind that keeps files 'in the cloud' and materializes them on demand. Everything works perfectly when you SSH in and run the code by hand. Then the service manager starts the same code, from the same path, on the same machine — and the background process hangs, sometimes just resolving its own working directory, before it executes a single meaningful line.

Every debugging instinct fails here, because your instinct is to reproduce it — and it won't reproduce. Your interactive shell can drive the sync provider's on-demand materialization; a process launched by the service manager, in a different session context, can sit there waiting for a file that never materializes. The code is fine. The path is 'the same' only in the sense that the string matches. The context that resolves that string is not the same at all.

The Fix: Separate Where You Edit From Where You Run

You cannot fix this inside the program, because the program never gets to run. The fix is structural: don't run the service from the synced path at all. Recall's service controller builds an atomic, generated runtime snapshot into a plain local application-support directory — excluding git metadata, caches, dependency folders, and build output, and including the runtime environment file. The service manager then starts the process from that snapshot, on an ordinary local path with no sync provider in the way. The stall is designed out of existence.

The discipline that makes this safe is the part worth memorizing: the snapshot is not a second repository. It is generated, disposable, and must never be edited. The conventional checkout and git remain the sole truth for code. Edit in the repo, push through git, regenerate the snapshot on deploy. The instant someone 'just fixes something quickly' in the snapshot, you've forked your truth and created a machine whose behavior no commit explains — which is exactly the ghost this whole architecture is built to avoid.

The Companion Detail: Build Before You Stop

One more piece of hard-won ordering lives in the same command. On the control-plane host, the restart builds the production frontend before stopping the currently-running process. It sounds like a trivial sequencing choice; it's the difference between a failed build being a non-event and a failed build being an outage. Build first, and a broken build leaves the healthy old service still serving. Stop first, and a broken build leaves you with nothing running and a broken tree to fix under pressure. Do the thing that can fail before you take down the thing that works.

Code

Edit in the repo; run from a generated snapshot·text

CODE TRUTH (edit here, never run the service here):
  conventional checkout on the sync-backed path
    -> git commit / push / pull

RUNTIME (generated, disposable, NEVER edited):
  service restart|auto:
    1. build the frontend        <- do the failable thing FIRST
    2. stop the old process         (a failed build = old one still up)
    3. write an atomic runtime snapshot to a plain local dir
         excludes: .git, caches, node_modules, dist
         includes: runtime .env
    4. start the service FROM THE SNAPSHOT

# The snapshot is not a second repo. Editing it forks your truth.

Exercise

Find a service, cron job, or scheduled task in your own setup and check where it actually runs from. Is that path a cloud-synced folder, a network mount, or a symlinked convenience path? Then compare the launch contexts: what environment, working directory, and session does it get from the service manager versus from your interactive shell? If they differ meaningfully, design the separation — edit location versus generated run location — and note what you'd have to forbid to keep the run copy from becoming a second source of truth.

Hint

Two probes: (1) run the job exactly the way the service manager does (not the way you'd test it by hand) and see whether it still works — the gap between those two is where this class of bug lives. (2) Ask whether the runtime path involves anything that materializes files lazily or resolves over a network. If so, move the runtime to a plain local path and generate it on deploy, keeping version control as the only truth.