Test the Installed App, With a Deadline on Every Wait

"A hang costs more than a red. A red at least names itself."

The Artifact a Person Runs

A unit suite proves the code in a test host. The family's desktop apps also carry an installed-app gate that proves the bundle in /Applications: launched through LaunchServices with open, so it is its own responsible process with its own permissions; started as a fresh instance with -ApplePersistenceIgnoreState YES and an explicit result file, so it neither depends on nor overwrites the owner's saved windows; and judged by markers in its own log. The first check of that gate is the one from the LaunchServices lesson: the installed executable's hash equals this checkout's build product, because another checkout can replace the slot between install and test.

open -W Waits Forever

Each stage of the coding engine's gate ran open -n -W -a App --env … and then looked for a success marker. When one stage's app stopped printing its marker and never quit, the gate did not fail. It blocked, with no timeout and no output. A sample of the app showed a healthy main thread in the ordinary run loop: the app simply had nothing left to do, and the wait was the defect. The regression had been invisible for months, because an earlier stage had been failing first. The rule is short: every wait gets a bound. Background the launcher, poll it against a deadline, and on expiry stop the launcher and only the app instance that stage started, print the stage log, and fail with the stage's name and the bound. Deadlines are generous on purpose; the point is that a stage that stops answering ends in seconds with a name attached. The same script had a second silent death: a helper whose output was redirected to a file failed under set -e and ended the gate with nothing on screen, so every redirected failure is re-read on the failure path.

Stages Only a Person's Terminal Can Run

One stage dragged a split-view divider through the Accessibility API. From an agent's shell it failed at once, because Accessibility belongs to the responsible process and the agent's host app did not hold it. No retry fixes that. The stage stays in the script, fails with a named reason when run without the grant, and is reported as operator-only, while the stage after it is still proven by seeding the state the drag would have produced. Report the boundary honestly rather than a green.

Proofs That Cannot Pass by Accident

To prove a desktop client reads files from a remote Mac, the smoke test gave both machines the same absolute path with different bytes and a different file count, and asserted on both; typing hostname into the remote terminal did the same for the shell. A pass could not come from the local disk. On the iOS side, a Simulator with a Korean keyboard turned ASCII typed through the hardware keyboard into Korean letters until the Simulator's keyboard list was changed and the app relaunched.

Code

A gate stage: the installed app, through LaunchServices, with a deadline and its own log·bash

#!/bin/zsh
set -euo pipefail
app=${APP:-/Applications/Spark.app}
executable=$(/usr/libexec/PlistBuddy -c 'Print :CFBundleExecutable' "$app/Contents/Info.plist")
hard_fail() { print -u2 "HARD_FAIL $1"; exit 1; }

# One stage: launch the INSTALLED app through LaunchServices, bounded, and judge by its log.
run_stage() {   # label, deadline in seconds, marker, then open(1) arguments for this stage
  local label=$1 deadline=$2 marker=$3; shift 3
  local log="build/stage-$label.log" ticks=0
  local before=(${(f)"$(/usr/bin/pgrep -x "$executable" || true)"})   # instances that are not ours
  mkdir -p build && : > "$log"
  /usr/bin/open -n -W -a "$app" --stdout "$log" --stderr "$log" "$@" \
    --args -ApplePersistenceIgnoreState YES &
  local launcher=$!
  while /bin/kill -0 $launcher 2>/dev/null && (( ticks < deadline * 10 )); do
    /bin/sleep 0.1; (( ticks += 1 ))
  done
  if /bin/kill -0 $launcher 2>/dev/null; then   # still waiting: the app never finished
    /bin/kill $launcher 2>/dev/null || true
    for pid in ${(f)"$(/usr/bin/pgrep -x "$executable" || true)"}; do
      (( ${before[(Ie)$pid]} )) || /bin/kill "$pid" 2>/dev/null || true   # only the one this stage started
    done
    /bin/cat "$log" >&2
    hard_fail "stage $label did not finish within ${deadline}s"
  fi
  wait $launcher 2>/dev/null || true
  /usr/bin/grep -q "$marker" "$log" || { /bin/cat "$log" >&2; hard_fail "stage $label ended without $marker"; }
  print "stage $label: ok"
}

run_stage launch 20 SPARK_STAGE_OK --env SPARK_STAGE=ok
run_stage restore 5 SPARK_STAGE_OK --env SPARK_STAGE=${SECOND_STAGE:-ok}

Give a Simulator an English keyboard before a UI test types ASCII·bash

# A Simulator whose keyboard is Korean turns typed ASCII into jamo: "echo" arrives as "ㄷㅊㅗㅐ".
xcrun simctl spawn "$SPARK_SIMULATOR_ID" defaults write .GlobalPreferences \
  AppleKeyboards -array 'en_US@sw=QWERTY;hw=Automatic'   # the spelling the Simulator itself stores
xcrun simctl terminate "$SPARK_SIMULATOR_ID" com.example.spark.mobile   # running apps cache defaults
xcrun simctl launch "$SPARK_SIMULATOR_ID" com.example.spark.mobile

Exercise

Build a tiny stand-in app that prints a marker and quits when an environment variable says ok, and never quits otherwise. Run the stage runner against it with one finishing stage and one hanging stage, and record the total time and the failure line. Start an unrelated instance of the same app before the gate and confirm it survives. Then write the one sentence your gate prints for a stage that needs Accessibility and was run without it.

Hint

Print and fflush(stdout) before quitting, or the marker may never reach the log file that open --stdout created. pgrep -x matches the executable name from CFBundleExecutable, not the bundle's display name.