The Library Path

Hands, Not Just Eyes

Watching Dad play is half of what Oriel was built for. The other half arrived on its first night: hands. Pippa should be able to press a game console's buttons herself, to navigate a menu, to try a section of a game, eventually to play. The console speaks Remote Play, the protocol a phone or computer uses to stream the console's picture and send controller input over the network. The open-source client for it is chiaki-ng, a desktop app with a window.

There were two ways to give Pippa hands. One was to run that app and inject keystrokes into its window: quick, fragile, tied to a GUI and to whatever has focus. The other was to build a client of her own on libchiaki, the library inside chiaki-ng, with no window at all. Dad chose on 2026-11-01 in a few words: "정석대로 가자. 1번." Do it properly; option one, the library.

oriel-hands

The result is oriel-hands, a headless Remote Play client built from a pinned chiaki-ng tree (the build script pins the tag and the commit, so an upstream change never arrives by surprise). It sends controller state (buttons, sticks, triggers) and decodes the console's video stream with VideoToolbox. It links only system frameworks, so the home Mac's own package manager is never touched, and it is signed with the family's code-signing identity like the recorder. It has no launchd job: it runs only when a command asks, or while a play session Dad opened is running.

Two operational lessons from building it are worth keeping. The client carries a tiny pipe2 shim: a static library from Homebrew was built on a newer macOS than the home Mac runs, and the dynamic loader failed with "Symbol not found: _pipe2" until the shim provided it. And "Remote Play on Console is already in use" means exactly that: another client (often a forgotten desktop window) holds the console's single Remote Play session. The rule is to close the other client first, never to fight it.

A Console Is Chosen by Registration, Never by Address

Remote Play requires registering a client with a console once, which produces keys. oriel-hands never asks for those keys in a prompt or a config file of its own. It reads the registration (the console's nickname, its hardware address, the registration key and the session key) from chiaki-ng's own settings in her account. The console's network address is not stored at all: each time, LAN discovery finds consoles and the right one is matched by its hardware address, so a router handing out a new IP changes nothing.

The keys are handled the way every secret in the family is: they reach the client on stdin only, never as a command-line argument, a log line, an event or an error message. Account tokens for the console's online service are never read.

How Fast Are the Hands?

Before anyone built play on top of them, the hands were measured: 30 D-pad presses per run on the console's Settings screen, with control trials (no button pressed) to measure the change detector's own noise. Input to the first changed decoded frame on the Remote Play stream: 79.1 ms median, 88.5 ms p95 in the clean run. Through the capture card instead (the console's HDMI output recorded by Oriel), input to captured frame: 81.7 ms median, landing a median of 6 ms after the stream frame. The control trials were essential: the Settings background animates, and some no-button trials crossed the detector's threshold around 106 ms, so a reading slower than that on that screen could not be told from background motion. The report says so, and names a focus-region detector as the follow-up.

Private Use, Never Distributed

libchiaki is licensed AGPL-3.0 (with an OpenSSL exception). That licence's obligations attach to distribution and to offering the software to others over a network. oriel-hands is for private use on Pippa's home Mac and is never distributed or published, which is why this quest describes it and does not offer it.

Code

The hands' build refuses anything it should not link (trimmed excerpt, native/OrielHands/build.sh)·bash

set -euo pipefail
step "configuring libchiaki (lib only: no GUI, no CLI, no audio device code, no FFmpeg)"
# ...configure and build the pinned tree, link oriel-hands...

# Only system libraries: the home Mac's package manager is never touched.
foreign=$(otool -L "$out" | tail -n +2 | awk '{print $1}' \
          | grep -v -E '^(/usr/lib/|/System/Library/)' || true)

# Never an audio framework: the hands have no microphone.
if otool -L "$out" | grep -q -E 'CoreAudio|AudioToolbox|AVFAudio'; then
  echo "oriel-hands links an audio framework; it must not" >&2; exit 1
fi

Measure input-to-screen latency honestly: trials plus control trials (runnable simulation of the method)·python

import random
import statistics

random.seed(4)


def first_change_ms(pressed):
    """When a 64x36 change detector first fires after the moment of an input."""
    background = random.choice([None, None, None, random.uniform(100, 110), random.uniform(800, 1100)])
    press = random.uniform(72, 90) if pressed else None
    candidates = [t for t in (background, press) if t is not None]
    return min(candidates) if candidates else None


trials = [first_change_ms(True) for _ in range(30)]
controls = [first_change_ms(False) for _ in range(10)]
floor = min(c for c in controls if c is not None) if any(controls) else float("inf")
clean = [t for t in trials if t < floor]
print(f"control floor: {floor:.0f} ms (anything slower may be background motion)")
print(f"trials below the floor: {len(clean)} of 30, median {statistics.median(clean):.1f} ms")

Exercise

Pick any interactive program you can script (a terminal game, a browser page, a local web app). Measure input-to-screen latency for one action thirty times, using screenshots or a frame grab before and after, and run ten control trials with no input at all. Report the median and p95 of the trials that land below your control floor, and say how many did not.

Hint

A simple change detector is enough: downscale both frames to 64x36 grey, count cells that changed by more than a threshold, and call it a change when at least three cells moved. The controls tell you whether your threshold is catching the action or the background.