Lesson 01 of 05 · published

One Service per Mac

~13 min · launchd, launchagent, install, keepalive, health-check, operations

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Seven Tracks of Mechanism, One Process to Keep Alive

Everything so far (shares, routes, tickets, the cache, jobs, pairs, the write call) lives inside one Python process per Mac. If that process is down, an app on that Mac gets local_delta_down, and a peer that wanted this Mac's files gets a refused connect. So the most important operational fact about Delta is boring on purpose: one LaunchAgent per Mac, labelled com.cwk.delta, started at login and restarted whenever it exits. This lesson is that agent: what installs it, what it runs, which Macs carry it, and why it is the only background process Delta has.

The Install Script Is the Only Door

Delta's repository ships one installer, scripts/install.sh, and it is run on the Mac it installs, as the account Delta will serve. It never reaches another Mac. Read it top to bottom and every step is a refusal or a check before it is an action:

The interpreter must be usable. It imports ssl, sqlite3 and http.server with /usr/bin/python3. If that fails, it stops and says to install the Xcode Command Line Tools. There is no conda environment to build, because Delta is standard library only (Track 1).
This Mac must be in Delta's fleet. It asks Delta's own config for this Mac's fleet label, the two ports from the family port ledger and the state root, and checks the label against fleet_labels(). A Mac outside the fleet is refused with not in Delta's fleet (every family Mac but the household's); not installing.
State is created private. The state folder (~/cwk-dbs/delta-db unless the host profile moves it) is made with mode 700. The local token and this Mac's TLS certificate are generated there if they do not exist yet. The script prints only the first 16 hex digits of the certificate's fingerprint, enough for a person to compare and nothing a caller could use.
The plist is rendered, then linted, then moved. deploy/com.cwk.delta.plist.template has two placeholders, the repository path and the logs folder. The script fills them into a temporary file, runs plutil -lint, and only a plist that lints replaces the installed one.
The service is replaced, not stacked. If the agent is loaded, launchctl bootout first, then bootstrap and kickstart -k.
Done means it answers. The script polls http://127.0.0.1:<engine>/v1/health for up to ten seconds. If Delta never answers, it prints the last twenty lines of launchd.err.log and exits non-zero. "launchctl returned 0" is never reported as success.

install.sh --check runs the same identification, then reports whether the plist exists, whether the service is loaded and what health answers, and changes nothing.

What the Plist Says

Key	Value	Why
`ProgramArguments`	`/usr/bin/python3 -B -E -s -m delta serve`	Apple's python (the Local Network gate lets a platform binary through); `-B` no bytecode files, `-E` ignore `PYTHON*` variables, `-s` no user site-packages
`WorkingDirectory`	the repository	so `-m delta` imports the checkout; it is also why a relative local path is refused (Track 6)
`RunAtLoad`, `KeepAlive`	true, true	start at login; restart on any exit
`ThrottleInterval`	10	a crash loop restarts at most every ten seconds
`NumberOfFiles`	8192 (soft limit)	pooled connections, partials and listings hold many descriptors at once
`StandardOutPath`, `StandardErrorPath`	`logs/launchd.out.log`, `logs/launchd.err.log`	what the process printed before its own logging started, and every crash

Nothing in the plist names a Mac, an address or a port. The ports come from the ledger when the process starts, so the same template installs on every Mac (invariant 8).

Which Macs Carry It

Delta's fleet is computed, not listed: every Mac in Kit's maintenance group (the Macs Dad manages) and its fanout_excluded group (Pippa's home Mac), minus any Mac in the household group. Pippa's home Mac runs it in her own pippa account and serves that account's home; Dad's ruling the day Delta was born was that her home gets one too, as a matter of course. The household's Mac (Mom's) runs no background service at all, so it runs no Delta; when it needs one specific share, its client asks a gateway directly (Track 4).

Because every Mac runs the same code and the same install, there is no "Delta server". The workbench and the service host are peers to each other in exactly the way a laptop is.

No Scheduler of Its Own

Delta moves nothing on its own. A transfer happens only when a person types delta cp or delta sync, or an app asks through kit_delta. The vault pair, the memory pair and Lantern's mirror pull all have their own schedulers in their own apps; Delta is the channel they ride. The one loop inside the daemon is housekeeping: every six hours it sweeps abandoned partials and bounds the logs (next lesson). That split matters when something goes wrong: if files are moving at three in the morning, the question is never "what did Delta decide", it is "which app asked".

The Family Sees It as a Card

Delta serves no web page and docks no Pippa panel, so its face in the family is Firelink's card. Firelink's operations registry declares the service label com.cwk.delta and a health check: GET /v1/health must contain "member":"cwkDelta". The short health answer carries the label, the version, the protocol, the uptime and this Mac's clock (a pair reads it for clock skew). The full one, which only this Mac's engine gives, adds shares and their load failures, the cache, routes, peers, jobs, partials waiting and the counters. A share whose child failed to load turns the status to degraded, never to down: one broken app share must not take the channel away from everyone else.

Code

Delta's installer: the checks before the action (trimmed excerpt)·bash

# scripts/install.sh (trimmed excerpt)
REPO="$(cd "$(dirname "$0")/.." && pwd)"
LABEL=com.cwk.delta
PLIST="$HOME/Library/LaunchAgents/$LABEL.plist"
PY=/usr/bin/python3

"$PY" -c 'import ssl, sqlite3, http.server' 2>/dev/null \
  || die "$PY is not usable (install the Xcode Command Line Tools on this Mac)"

[ "$MEMBER" = yes ] || die "$MAC is not in Delta's fleet (every family Mac but the household's); not installing"

mkdir -p "$STATE/logs" "$HOME/Library/LaunchAgents" "$HOME/bin"
chmod 700 "$STATE"

tmp=$(mktemp "$PLIST.XXXXXX")
sed -e "s#__REPO__#$REPO#g" -e "s#__LOGS__#$STATE/logs#g" "$REPO/deploy/$LABEL.plist.template" >"$tmp"
plutil -lint "$tmp" >/dev/null || { rm -f "$tmp"; die "rendered plist does not lint"; }
mv "$tmp" "$PLIST"

if launchctl print "gui/$(id -u)/$LABEL" >/dev/null 2>&1; then
  launchctl bootout "gui/$(id -u)/$LABEL" 2>/dev/null || true
  sleep 1
fi
launchctl bootstrap "gui/$(id -u)" "$PLIST"
launchctl kickstart -k "gui/$(id -u)/$LABEL" >/dev/null 2>&1 || true

for _ in 1 2 3 4 5 6 7 8 9 10; do
  if out=$(curl -s -m 2 "http://127.0.0.1:$ENGINE/v1/health"); then
    say "running: $out"
    exit 0
  fi
  sleep 1
done
tail -20 "$STATE/logs/launchd.err.log" >&2 || true
die "Delta did not answer on 127.0.0.1:$ENGINE within 10 s (log above)"

Install your own stdlib service the same way: render, lint, replace, prove it answers·bash

#!/bin/bash
# install-agent.sh: a LaunchAgent for a small stdlib HTTP service of your own.
# Usage: ./install-agent.sh /absolute/path/to/server.py 8765
set -euo pipefail
SERVER="$1"; PORT="$2"
LABEL=com.example.minichannel
PLIST="$HOME/Library/LaunchAgents/$LABEL.plist"
LOGS="$HOME/Library/Logs/$LABEL"
mkdir -p "$LOGS" "$HOME/Library/LaunchAgents"

tmp=$(mktemp "$PLIST.XXXXXX")
cat >"$tmp" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0"><dict>
  <key>Label</key><string>$LABEL</string>
  <key>ProgramArguments</key><array>
    <string>/usr/bin/python3</string><string>-B</string><string>-E</string><string>-s</string>
    <string>$SERVER</string><string>$PORT</string>
  </array>
  <key>RunAtLoad</key><true/>
  <key>KeepAlive</key><true/>
  <key>ThrottleInterval</key><integer>10</integer>
  <key>StandardOutPath</key><string>$LOGS/out.log</string>
  <key>StandardErrorPath</key><string>$LOGS/err.log</string>
</dict></plist>
EOF
plutil -lint "$tmp" >/dev/null || { rm -f "$tmp"; echo "plist does not lint" >&2; exit 1; }
mv "$tmp" "$PLIST"

launchctl bootout "gui/$(id -u)/$LABEL" 2>/dev/null || true
launchctl bootstrap "gui/$(id -u)" "$PLIST"

# Done means it answers, not that launchctl returned 0.
for _ in $(seq 10); do
  if curl -s -m 2 "http://127.0.0.1:$PORT/" >/dev/null; then echo "running on $PORT"; exit 0; fi
  sleep 1
done
tail -20 "$LOGS/err.log" >&2 || true
echo "no answer on 127.0.0.1:$PORT within 10 s" >&2; exit 1

External links

Exercise

Write a twenty-line stdlib HTTP server that answers GET / with a small JSON health document (a status, its uptime and its process id). Install it as a LaunchAgent with the second script above. Then prove KeepAlive: read the pid from health, kill -9 it, and time how long until health answers again with a new pid. Finally break it on purpose (a syntax error in the server) and run the installer again: it must fail loudly and show you the traceback.

Hint

os.getpid() and time.monotonic() give the pid and uptime. After the kill, expect a gap close to the ThrottleInterval only if the job had been running less than ten seconds; launchd throttles restarts, not the first one after a long run. Remove the agent afterwards with launchctl bootout gui/$(id -u)/com.example.minichannel and delete the plist.