Monitoring Without Watching

The transfer engine that does not need you

The pacing and resumability lessons of the archive-design track described a campaign that survives inattention. The Monitor surface is that campaign, running: a queue of approved download jobs moving at the configured rate under the daily budget, each job resumable across interruptions, each landing verified — and none of it requiring a human to keep a page open or a terminal attached.

The Woodpile Monitor surface: a queue of download job cards, each showing the model name, total size, and a progress bar, with per-job transfer statistics such as received bytes, average speed, and elapsed or remaining time. — The campaign, running: paced jobs under the daily budget, resumable and verified — reported, not watched.

Look at what the cards carry: per-job progress against expected size, received bytes and average speed, elapsed and remaining estimates. Those are the planning numbers from the budgets lesson, surfaced live. The queue heartbeats on its own; recovery from interruption is a state transition the engine performs, not a task it assigns to you. When you check in, the state is already true — the monitor's job is to report, not to be watched.

Why autonomy is a design consequence, not a feature

The autonomy follows from the failure analysis, not from ambition. Because interruptions were treated as certainties, resume became mandatory; because resume is mandatory, no job depends on a continuous session; because pacing is configured, no job depends on opportunistic bandwidth; because verification is automatic, no landing depends on a human running the digest check. Each survival trait removes a reason to watch; together they remove all of them. The monitor is boring on purpose — boring is what reliable looks like from the outside.

This is also the honest division of labor between the human gate and the machine queue, the one the Composer lesson set up: people decide (slowly, occasionally, with judgment), machines move (continuously, patiently, without judgment). Confusing the two assignments is how tools end up needing babysitters — or how collections end up curated by whatever automated at 3 a.m.

Attention is not a transfer resource. A pipeline that spends human attention to move bytes is mispriced; pacing, resume, and verification exist precisely so the campaign runs on electricity instead of vigilance.

The home version

Your own queue can be this boring with a scheduler and the scripts from earlier tracks: a launchd job or cron entry that runs the paced, resumable pull loop each night, honors a daily byte budget by checking the day's counter, and appends each file's digest result to a log you read at your leisure. Ten lines of shell buy the same property: check in when curious, never to keep things alive.

Code

A boring home queue: nightly, paced, budgeted·bash

# ~/bin/woodpull-nightly.sh — run from cron/launchd each night
#!/bin/zsh
BUDGET_FILE=$HOME/.woodpull-day-$(date +%Y-%m-%d)
SPENT=$(cat "$BUDGET_FILE" 2>/dev/null || echo 0)
BUDGET=$((150*1024*1024*1024))   # daily bytes allowance
QUEUE=$HOME/.woodpull-queue       # url<TAB>expected-size<TAB>expected-sha256
DEST=$HOME/models                 # landing directory

while IFS=$'\t' read -r url expected sha256; do
  dest="$DEST/$(basename "$url")"
  [ -f "$dest" ] && continue                 # already landed: skip, never re-pull
  [ "$SPENT" -ge "$BUDGET" ] && { echo "budget reached"; exit 0; }
  part="$dest.part"
  curl -L -C - --fail --limit-rate 1500K -o "$part" "$url" || continue
  sz=$(stat -f%z "$part")
  [ "$sz" = "$expected" ] || { echo "SIZE FAIL $part ($sz != $expected)"; continue; }
  mv "$part" "$dest"
  # verification is part of landing, not an afterthought:
  got=$(shasum -a 256 "$dest" | awk '{print $1}')
  if [ -n "$sha256" ] && [ "$got" = "$sha256" ]; then
    echo "$(date -I) OK $dest" >> ~/models/digests.log
    SPENT=$((SPENT+sz)) && echo $SPENT > "$BUDGET_FILE"
  else
    echo "$(date -I) VERIFY FAIL $dest (got $got)" >> ~/models/digests.log
    mv "$dest" "$dest.bad-$$"   # quarantine; never count unverified as landed
  fi
done < "$QUEUE"
# Read digests.log whenever you like. Nothing here needs you awake.

Exercise

Deploy the boring home queue: write the nightly script (adapt paths and budget), seed a one-entry queue with a real URL and its expected size, and let it run on schedule for at least two nights. Check in once mid-campaign without touching anything, then verify the digest log caught the completed landing. Note what required you and what did not.

Hint

The one thing that should require you: choosing the queue's contents (the gate from lesson one). The transfer itself should survive your absence — if it did not, which survival trait was missing?

The transfer engine that does not need you

Why autonomy is a design consequence, not a feature

The home version

Code

External links

Exercise

Progress

Comments 0