The Loop, and Who Owns Each State

"Okay. Let's go to step 2." — Dad, starting hands-free, 2026-09-25

The Loop on Paper

On paper, a hands-free talk is a circle: idle, listening, sending, thinking, speaking, and back to listening. The design drew exactly that. The implementation taught something the drawing hides: those states don't all belong to the same component. Listening belongs to the microphone loop. Thinking belongs to the chat stream. Speaking belongs to the audio player. A loop that tries to own all of them ends up guessing about the two it doesn't control.

The Listener's Own Phases

So the hands-free loop, on the web and on the phone alike, owns only its own phases: off, connecting (asking the brain for a listening session and opening the socket), listening (streaming 16 kHz audio and showing partial words), waiting (an utterance has ended; it is being kept, sent, and answered), paused and stopped. Thinking and speaking it simply observes. It starts listening again only when two independent facts are both true: the reply's stream has ended, and the player reports that nothing is playing or queued. That second fact needed its own fix on the web, where the player hadn't been releasing finished audio elements, so "is anything playing?" could still answer yes after the last word had played.

EV v6 · 2026-10-27: connecting no longer makes the microphone wait for the brain. The phone used to ask for the listening grant first and open the microphone only after that round trip, so on a slow link a working microphone looked dead. Since 2026-10-20 (cbe99bd0) the phone opens its microphone first, at the grant's fixed 16 kHz, and then asks for the grant; whatever Dad says before the socket is up is kept and goes in first. The web followed on 2026-10-22 (050bae9d): it asks for the grant while the microphone opens instead of after it, and the words said while a listen is connecting are kept and replayed when the socket opens. Until then his first words after her reply were dropped on the web while the grant and the socket opened.

The Microphone Stays Open; the Socket Doesn't Always Listen

The browser opens the microphone once per talk, with echo cancellation on, and keeps it. Audio chunks go to the transcriber only while the loop is listening; while the soul thinks or speaks they go nowhere except to the barge-in detector of Track 7. Before barge-in existed, the rule was simpler still: listening ended at the commit and reopened after the reading, so the soul could never hear herself.

A Quiet Room Is Not a Talk

An open realtime socket bills for the audio it hears, silence included. After 60 seconds with nothing heard, the loop stops itself and says so on the face screen. One tap listens again.

The Phone Needs One Owner for Audio

On iOS the loop taught one more lesson. At first the player and the recorder each configured the audio session on their own. Worse, iOS suspends an app whose audio session goes idle, and in a talk the idle part was the soul's thinking. Now a single owner sets the session to play-and-record once and keeps the microphone open from the first listen until hands-free ends, and the player leaves the session alone while that owner holds it. The talk continues with the screen off and the phone in a pocket, and each turn gets a Live Activity that stays on the lock screen until the next turn replaces it, showing "listening…" or "speaking…".

EV v4 · 2026-10-12: One owner still left one hole, and it was iOS itself. A phone call in the middle of a talk, or AirPods moving between the watch and the phone, stops the talk's audio engine, and iOS restarts nothing. Her reply sat on a dead engine with no sound and no end. Since 2026-10-10 the shared voice kit's audio owner watches the system's notices (an interruption ending, a route change, an engine configuration change, a media-services reset) and rebuilds a microphone whose engine has stopped; the phone then plays the unit she was reading again on the new output, and listening carries on over the same socket. An interruption's beginning only stops the engine, because iOS refuses a restart until it ends. The cause was read from the code and from what Dad heard, never reproduced on a device, and Dad closed the report on the build that carried the fix.

EV v6 · 2026-10-27: Playing that unit again changed meaning once a spoken answer became one take again (Track 4, Whole Again: Zero Breaks Before a Fast Start): the top of the unit was now the top of the whole answer. Since 2026-10-22 (f58e7fea) the phone holds the reading the moment the route changes or the interruption begins, and restarts the same take at its heard position, half a second before her last sound so the word that was cut is heard whole (VoiceLoopSpeechPlayer.play(chunks:skippingSeconds:)). A reading that plays off the talk's engine restarts the same way, and the owner's rebuild and the playback's own 0.8-second fallback cannot both restart it. A day later (5c43315a) a stream the kit reports as .interrupted, stopped by the system with no route change behind it, is held as well and reads on from where she was, instead of counting as the end of the reading. This track's lesson The Phone Reads It to Your Ears has the navigation prompt that found it.

Code

The listener's phases as a pure reducer·typescript

// The hands-free loop as a pure reducer: every transition is explicit and testable.
type Phase = 'off' | 'connecting' | 'listening' | 'waiting' | 'paused' | 'stopped';

type Event =
  | { kind: 'start' }
  | { kind: 'socketOpen' }
  | { kind: 'utterance'; text: string }        // end of speech decided; its upload begins
  | { kind: 'uploaded' }                        // the finished utterance reached the server
  | { kind: 'replyDone'; playerBusy: boolean }  // the reply's stream ended
  | { kind: 'playerIdle' }                      // nothing playing or queued
  | { kind: 'quietMinute' }
  | { kind: 'socketClosed'; code: number; reason: string }
  | { kind: 'pause' }
  | { kind: 'resume' }
  | { kind: 'close' };

interface Loop { phase: Phase; reason: string | null; streamEnded: boolean; delivering: boolean }

const OFF: Loop = { phase: 'off', reason: null, streamEnded: false, delivering: false };
const LISTEN_AGAIN: Loop = { phase: 'connecting', reason: null, streamEnded: false, delivering: false };

export function step(loop: Loop, event: Event): Loop {
  switch (event.kind) {
    case 'start':
      return loop.phase === 'off' ? LISTEN_AGAIN : loop;
    case 'resume':
      if (loop.phase !== 'paused' && loop.phase !== 'stopped') return loop;
      // Words he finished before the pause are still on their way: wait for
      // them and her reply, as if he had never paused.
      return loop.phase === 'paused' && loop.delivering ? { ...loop, phase: 'waiting' } : LISTEN_AGAIN;
    case 'socketOpen':
      return loop.phase === 'connecting' ? { ...loop, phase: 'listening' } : loop;
    case 'utterance':
      return loop.phase === 'listening'
        ? { ...loop, phase: 'waiting', streamEnded: false, delivering: true } : loop;
    case 'uploaded':
      return { ...loop, delivering: false };
    case 'replyDone':
      if (loop.phase !== 'waiting') return loop;
      // Listen again only when the reply is finished AND nothing is playing.
      return event.playerBusy ? { ...loop, streamEnded: true } : LISTEN_AGAIN;
    case 'playerIdle':
      // An idle player before the reply has ended is not a reason to listen.
      return loop.phase === 'waiting' && loop.streamEnded ? LISTEN_AGAIN : loop;
    case 'quietMinute':
      return loop.phase === 'listening'
        ? { ...loop, phase: 'stopped', reason: 'Nothing heard for a minute; tap to listen again.' } : loop;
    case 'socketClosed':
      // A close he caused (the close button, a pause) is not news; any other is.
      if (loop.phase === 'off' || loop.phase === 'paused' || loop.phase === 'stopped') return loop;
      return { ...loop, phase: 'stopped', reason: `Listening closed (${event.code}): ${event.reason || 'no reason given'}` };
    case 'pause':
      return loop.phase === 'off' || loop.phase === 'stopped' ? loop : { ...loop, phase: 'paused' };
    case 'close':
      return OFF;
  }
}

function run(title: string, events: Event[]) {
  let loop = OFF;
  console.log(`-- ${title}`);
  for (const event of events) {
    loop = step(loop, event);
    console.log('  ' + event.kind.padEnd(12), '->', loop.phase, loop.reason ?? '');
  }
}

run('a turn, then a quiet minute', [
  { kind: 'start' }, { kind: 'socketOpen' }, { kind: 'utterance', text: '내일 날씨 어때?' },
  { kind: 'uploaded' },
  { kind: 'playerIdle' },                        // too early: the reply hasn't ended
  { kind: 'replyDone', playerBusy: true }, { kind: 'playerIdle' },
  { kind: 'socketOpen' }, { kind: 'quietMinute' },
]);
run('paused while the words were still uploading', [
  { kind: 'start' }, { kind: 'socketOpen' }, { kind: 'utterance', text: '잠깐만' },
  { kind: 'pause' }, { kind: 'resume' },          // back to waiting, not listening
  { kind: 'uploaded' }, { kind: 'replyDone', playerBusy: false },
]);
run('a close he caused, then the socket reports it', [
  { kind: 'start' }, { kind: 'socketOpen' }, { kind: 'close' },
  { kind: 'socketClosed', code: 1000, reason: 'closed by client' },   // stays off
]);

Exercise

Run the reducer (npx tsx loop.ts) and read its three runs. Then add two events you think are missing: the microphone refused before the socket ever opens, and the user sending a typed message while the loop is waiting. Decide the next phase for each and add a run that exercises it.

Hint

A refused microphone is a stop with its reason on screen, like any other failure: the face stays up and says why, and a tap listens again. A typed message while waiting doesn't change the listener at all: the reply to it will end and the player will go idle, and the loop listens again from those facts.