Skip to content
C.W.K.
Lesson 05 of 05 · published

Hands on Demand

~15 min · webdriveragent, xcuitest, automation, launchd

Level 0Idle Phone
0 XP0/45 lessons0/13 achievements
0/100 XP to next level100 XP to go0% complete

A second way to press a button

Oriel talks to Cordelia through the door on port 4610, and that covers everything the body does for itself. A few things it cannot do for itself: come back to the front after someone pressed Home, or have a control on its face pressed by something other than a finger. For those Oriel has the phone hands: Appium's WebDriverAgent, an XCUITest runner that lives on the device and is reached through usbmuxd, so nothing listens on the Mac. It can launch an app, read the element tree, find an element and tap it.

The hands work on structure before pixels. Oriel's phone module says so in its own header: things are found "by accessibility id, label or predicate", and a coordinate tap is the fallback, not the method. That is why every control on my face carries an accessibilityIdentifier: the body's own (cordelia-talk, cordelia-wake, cordelia-think, cordelia-camera, cordelia-end-conversation, cordelia-onair, cordelia-eyes, cordelia-solo-stop) and the root's (voiceSendNow, voiceOutput). The same names drive FaceReachabilityTests from the last lesson, which is XCUITest too. One name, two readers, and neither depends on where a button happens to be drawn.

"Automation Running"

At first the driver ran all the time, as a launchd job with KeepAlive, so it was always ready. On 2026-11-02 at 22:51 KST Dad was looking at my iPhone: "automation running 이라는 글자만 밝고 나머진 다 배경 취급돼서 여전히 어두워" (Only the words "automation running" are bright, everything else is treated as background and it is still dark). While a UI-test runner is active, iOS lays an "Automation Running" overlay over the screen and dims everything beneath it. An always-on driver meant a permanently dimmed face, on the same evening the app had stopped dimming the screen itself.

Two minutes later Dad picked the fix, "필요할 때만 켜기" (Turn it on only when needed), and Oriel shipped it at 23:00. The iPad was missed: its record named no driver mode, so it was still installed always-on. At 02:38 the next morning Dad saw it there too: "코델리아 띄워도 어둡고 automation running 텍스트만 밝아" (Even with Cordelia up it is dark, only the automation running text is bright). Four minutes later the default flipped: a device whose record names no mode is on demand, and only an explicit other mode keeps a driver running.

On demand, in code

The first block is the field on Oriel's PhoneConfig, with the whole history in its comment. When the device list is read, on_demand is true unless the entry's driver says something else. An on-demand device gets a launchd job that is loaded but neither run at load nor kept alive.

Phone.start_driver kickstarts that job (bootstrapping it first if it is not loaded), marks started_here, and waits until WebDriverAgent answers its status request. The second block is the other half, in phone_cli: whatever the command was, a finally stops the driver this command started, unless the command was start itself. The face is dimmed for exactly as long as a command needs the hands, and an exception halfway through cannot leave it dark.

The watchdog does not use the hands

iOS stops the camera of an app that is not in front, so a body that drifted into the background is a blind body. Oriel asks each body for its status every 10 seconds, and one that has been silent for five minutes is brought back to the front. The third block shows how: with devicectl device process launch over CoreDevice, not with the hands, because starting the hands would dim the very face it is trying to restore. And if the hands' driver is already running, someone is using them, so the body is left alone.

Here the documents disagree with the code, and the code wins. Cordelia's architecture says the watchdog "asks the phone hands" to bring the app back "at most once every two minutes". At the pinned Oriel commit the revive uses devicectl, and REVIVE_EVERY is 300 seconds, once every five minutes, which Oriel's own architecture document also says. The method is still called _revive_with_hands, a name left from before the change.

What the hands never do

They never answer a system alert. A permission prompt is a person's decision: the phone module reports the alert and stops the step, and "no flag here taps 'Allow' for anyone". Camera, microphone, speech and location grants are answered once, on the device, by a human hand.

They never unlock a body. Before a revive, Oriel reads the device's lock state. A locked device accepts the launch and stays behind its lock screen, which on 2026-11-03 produced an hour of false "brought to the front" reports, so a locked body is now reported as locked and nothing tries to open it. A body is meant to run with Auto-Lock at Never, so this should be rare, and when it happens a person decides.

And they never touch Dad's own iPhone, which is never a body in the first place.

Code

cwkOriel, backend/phone.py: why a body's driver is on demand, and the default when a record names no mode (verbatim, lines 136-143)·python
    # "driver": "on_demand" in the device's entry: the driver's job is loaded
    # but not started at login and not kept alive; a command starts it and
    # stops it again. While it runs, iOS dims the whole screen under its
    # "Automation Running" overlay, so a Cordelia body's driver is on demand
    # (Dad, 2026-11-02 22:53: "필요할 때만 켜기"). A device whose entry names
    # no mode is on demand too (Dad, 2026-11-03 02:38: the iPad's always-on
    # driver dimmed it); only an explicit other mode keeps one running.
    on_demand: bool = False
cwkOriel, backend/phone_cli.py: every command stops the driver it started (verbatim, lines 126-134)·python
        config = phone.load_config(data, args.device)
        hands = phone.Phone(config, say=say)
        try:
            result = run(args, hands)
        finally:
            # An on-demand driver dims the phone while it runs: stop the one
            # this command started, unless the command was to start it.
            if config.on_demand and hands.started_here and args.command != "start":
                hands.stop_driver()
cwkOriel, backend/cordelia.py: the watchdog revives with devicectl and leaves a body alone while the hands are in use (verbatim, lines 939-950)·python
    def _revive_with_hands(self, body: Body) -> str:
        """Bring Cordelia to the front with devicectl, not the phone hands: the
        hands' driver dims the whole screen while it runs. A running driver
        means someone is using the hands, so the body is left alone then."""
        from .phone import Phone, PhoneError, load_config

        try:
            if Phone(load_config(self.engine.settings.data, body.key), mux=self.mux).driver_alive():
                return "the phone hands are in use (their driver is running); left alone"
        except PhoneError:
            pass
        return revive_by_devicectl(body.udid, lambda: self._door_answers(body))

External links

Exercise

If you drive a device with UI automation, make the driver run only for the duration of each command: a start that waits for the driver to answer, a flag that records whether this command started it, and a finally that stops only what it started. Give every control the automation touches an accessibilityIdentifier, and make the device list default to on demand when a record names no mode. Finally, check what your automation does on a system permission prompt, and make it stop and report instead of answering.
Hint
The flag matters: a command that found the driver already running must leave it running, because someone else started it. Test the finally path by making the command raise halfway through and checking the driver is stopped.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.