~14 min · reliability, failure-paths, code-review, observability
Level 0Muted
0 XP0/64 lessons0/14 achievements
0/100 XP to next level100 XP to go0% complete
"모두 이 세션에서 고쳐" (Fix all of it, in this session.) — Dad, on the voice-mode review's findings, 2026-10-22
The Happy Path Held
On 2026-10-21 the Pippa machine moved to a new Mac, and Dad set an order of code reviews to follow it. Voice mode came second. The engine, the web, the phone, the watch and the shared voice kit were read side by side, and every major finding was reopened at its own line in the code before it counted.
The logs were read first, and they were good. 103 phone voice turns on the new machine over 2026-10-21 and 22, 90 of them over cellular, played with underruns=0 on every one; Track 4's Whole Again: Zero Breaks Before a Fast Start tells why. But a log is a sample of what happened, and on those two days the bad things had not happened often enough to show. The review asked a different question: what does the code do when something goes wrong? Its findings were almost all on those exception paths, and Dad's word on them is the line above: everything, the same day.
A Failure Is a State
This track's second lesson set a rule for Dad's words: a finished utterance has three places to go and no fourth, and "silently nothing" is not one of them. The review turned the same rule on failures. Before it, a spoken turn whose provider erred in the middle of a reply left the phone with nowhere to go. The turn stayed awaited, the lines it had already streamed kept their entry, the microphone never opened again, and the last stage line stood frozen until Dad tapped twice. On the engine, the stage board stood at whatever step the dead turn had reached. Nothing had crashed, and nothing said anything.
So every fix below follows one rule: a failure ends in a named state, and the record and the screen agree on what it was.
Fixes That Keep a Turn Moving
A failed turn frees the talk. The phone's spoken-reply gate forgets the turn (SpokenReplyGate.forget), its units and its stage line go, and the talk listens again once nothing of hers is sounding. Only the first sighting does this, because a failed record stays in the outbox and a later drain must not reopen the microphone under a newer turn. On the engine, a turn that fails before its done leaves the stage board as done with the detail failed:<reason>: http_<status>, cancelled or error, from turn_clock.marks_failure on the chat routes and from each stream's own end. An entry quiet for an hour is pruned.
A dropped send is sent again. A transient failure while the app stayed in front used to wait for a launch, a return to the foreground, a push or a new send; nothing else drained the outbox, and the line read "Link dropped, sending again · 240s". The lane is now drained again on its own after 5, 15 and 45 seconds, then every 60, with the screen off too while a talk is open, and "sending again" has a stall limit and is diagnosed like any other quiet step.
A lost stream is not the whole answer. On the web, a stream that died mid-reply used to be read as final, so the partial tail became her last word and the full reply was never read. The reading now keeps its cursor, follows the recovery to the stored reply, and reads it from where it stopped, while the face says "Connection lost — following her answer".
A route change resumes where she was. AirPods moving, a call or a navigation prompt holds the reading and restarts the same take at its heard position. The first lesson of this track and The Phone Reads It to Your Ears tell those two.
Fixes That Say What Happened
"끊기는 데다 아예 건너 뛰어." (It breaks up, and on top of that it skips right over parts.) — Dad, 2026-10-19, on the web voice screen
Until 2026-10-20 a web take whose fetch failed twice was dropped without a word, and Dad heard an answer with a hole in it. Now a unit whose voice never came is said and reported, never skipped. The web did it first (89bf9fe9); the phone followed in the review (f58e7fea), where how a take fails decides what comes first. A stream that never sounded falls back to the finished take, and is said only if that fails too. A private talk keeps no finished take, so there the first failure is said. In a hands-free talk a take that stalls is asked for again once and read on from its heard position, and only a second failure is said. Any other break is said at once. The line is "Part of her answer was not read aloud: …" when nothing of the unit sounded, and "Her voice stopped early: …" when it broke off. Both go to the talk's record as a voice_reading_stopped line (take_failed or audio_error); a private talk keeps none.
The opposite mistake turned up three days later, on 2026-10-22. Dad's iPhone put "Part of her answer was not voiced: the stream could not open (Pippa answered HTTP 400: …)" above the composer, over much of the screen, with no way to close it. The reply was a lone fenced spark block, which holds no words a voice can read. Nothing to speak is not an error (671e6ee4). The engine knows it before Bellows is asked: /api/tts/prepare answers {"ticket": null, "enhanced": false, "nothing_to_speak": true}, /api/tts/ensure answers {"filename": null, "nothing_to_speak": true}, and a stream answers 204; Bellows' own empty_text refusal is answered the same way. Both clients pass the unit over in silence: no line, no report.
And every voice problem is one plain line. The words come from the error's own text, never from the HTTP chain: "her voice took too long to arrive", "the link to Pippa dropped", "the voice service refused it". The technical detail waits behind a tap or a hover, a cross dismisses the line, and Dad's next turn clears it. That is the Seventh Rule's addendum, "A Wrong Status Is Worse Than a Silent One", written into one function on each side (voiceProblem.ts on the web, plainCause on the phone).
The Thread
Read the fixes again as states, not patches. A failed turn: the board says failed:<reason> and the talk is listening. A dropped send: queued, with its next attempt on the screen. A lost stream: following, with the cursor kept. A failed take: one sentence on the screen and one line in the record. A reply with nothing to speak: the next unit, silently, because nothing went wrong. Each path ends in a state the record and the screen agree on, and silence remains only where silence is the truth. It is the never-dropped rule, applied to failures.
The fixes landed the same day: in cwkPippa, engine cb63d4f7, web 050bae9d, a kit sync d3b3965f and phone and watch f58e7fea, beside the kit's own commits in cwkSiblingKit, with the suites green at each: 7780 backend tests, 1193 on the web, 536 on the phone's iOS 27 simulator. The connection-pool and "Reconnecting" findings of the same review are Track 10's A False Alarm, and One Measurement per Cause; the deadlines on her voice are Track 4's.
Code
A turn that fails leaves the board as done / failed:<reason> (verbatim, backend/services/turn_clock.py, as of 2026-10-27)·python
# backend/services/turn_clock.py:125-156 (cwkPippa)
def failure_reason(exc: BaseException) -> str:
"""What ``failed:<reason>`` names: an HTTP refusal by its status, a
request that went away, or any other error."""
status = getattr(exc, "status_code", None)
if isinstance(status, int):
return f"http_{status}"
if isinstance(exc, asyncio.CancelledError):
return "cancelled"
return "error"
def marks_failure(endpoint: Callable[..., Awaitable[T]]) -> Callable[..., Awaitable[T]]:
"""Wrap a chat route so a turn that fails before its stream starts leaves
the board as ``done`` / ``failed:<reason>`` (todo #598): a 404 for a
conversation the viewer may not write, a 500 for a row that would not
save, or any error in the preparation. ``functools.wraps`` keeps the
route's signature visible to FastAPI. The stream reports its own end."""
@functools.wraps(endpoint)
async def wrapper(*args: Any, **kwargs: Any) -> T:
token = _route_turn.set(None)
try:
return await endpoint(*args, **kwargs)
except BaseException as exc:
clock = _route_turn.get()
if clock is not None:
clock.failed(failure_reason(exc))
raise
finally:
_route_turn.reset(token)
return wrapper
The phone lets go of a failed spoken turn, once (verbatim, clients/apple/PippaMobile/Sources/AppModel.swift, as of 2026-10-27)·swift
// clients/apple/PippaMobile/Sources/AppModel.swift:4966-4979 (cwkPippa)
/// A spoken turn failed (todo #598): after a provider error mid-reply the
/// turn stayed awaited, the lines it had streamed kept a `spokenUnits`
/// entry, the mic never opened again and the last stage line stood frozen
/// until Dad tapped twice. It leaves the gate and the line now, and the
/// talk listens again once nothing of hers is sounding. Only the first time
/// the failure is seen: a failed record stays in the outbox, and every
/// later drain must not reopen the mic under a newer turn.
private func spokenTurnFailed(_ turnId: String) {
let awaited = spokenReplies.forget(turnId)
spokenUnits[turnId] = nil
guard awaited else { return }
if voiceStageTurnId == turnId || voiceStage?.turnId == turnId { clearVoiceStage() }
if !voice.isReading { resumeHandsFree() }
}
A sketch: every failure settles into a named state (npx tsx settle.ts)·typescript
// A sketch, not cwkPippa's code: the ways a spoken turn can fail, and the named
// state each one must end in. The rule it checks: a real failure is never
// silent, and "nothing to speak" is not a failure at all.
type Failure =
| { kind: 'turnFailed'; reason: string; firstSeen: boolean } // provider error mid-reply
| { kind: 'sendDropped'; attempt: number } // a transient link drop
| { kind: 'streamLost'; cursor: number } // web: the stream died mid-answer
| { kind: 'routeChanged'; heardSeconds: number } // AirPods moved, a call came
| { kind: 'takeFailed'; how: 'neverSounded' | 'stalled' | 'brokeOff'; restarted: boolean;
detail: string; privately: boolean } // a take in a hands-free talk
| { kind: 'nothingToSpeak' }; // a lone ```spark block
interface Outcome {
record: string | null; // what the engine or the talk's record keeps
screen: string | null; // the one line Dad sees
next: string; // the state the turn moves on to
}
const RETRY_SECONDS = [5, 15, 45, 60]; // the last delay repeats
function retryAfter(attempt: number): number {
return RETRY_SECONDS[Math.min(attempt, RETRY_SECONDS.length - 1)];
}
/** Words Dad can act on, from the error's own text; never the HTTP chain. */
function plainCause(detail: string): string {
const text = detail.toLowerCase();
if (/timed out|stalled|no audio in time/.test(text)) return 'her voice took too long to arrive';
if (/offline|connection was lost|network/.test(text)) return 'the link to Pippa dropped';
if (/\b5\d\d\b/.test(text)) return 'the voice service failed';
if (/\b4\d\d\b/.test(text)) return 'the voice service refused it';
return 'her voice did not come through';
}
export function settle(f: Failure): Outcome {
switch (f.kind) {
case 'turnFailed':
// Only the first sighting frees the talk; a later drain must not reopen
// the mic under a newer turn.
return f.firstSeen
? { record: `stage board: done / failed:${f.reason}`, screen: null, next: 'gate forgets the turn; listening again' }
: { record: null, screen: null, next: 'unchanged (already settled)' };
case 'sendDropped':
return { record: null, screen: `Link dropped, sending again in ${retryAfter(f.attempt)} s`, next: 'queued' };
case 'streamLost':
return { record: null, screen: 'Connection lost — following her answer',
next: `read the stored reply from character ${f.cursor}` };
case 'routeChanged':
return { record: null, screen: null,
next: `same take from ${Math.max(0, f.heardSeconds - 0.5).toFixed(1)} s` };
case 'takeFailed': {
// A stream that never sounded falls back to the finished take, except in a
// private talk, which keeps no finished take: there the first failure is said.
if (f.how === 'neverSounded' && !f.privately)
return { record: null, screen: 'Her voice came from the finished take: the live stream did not open.',
next: 'play the finished take (take_failed if it fails too)' };
// A stall (no audio for 20 s) is asked for again once, from where she was heard.
if (f.how === 'stalled' && !f.restarted)
return { record: null, screen: null, next: 'reopen the take at the heard position' };
// A second stall, or any other break, is said at once.
const voiced = f.how !== 'neverSounded';
const cause = voiced ? 'audio_error' : 'take_failed';
const line = voiced
? `Her voice stopped early: ${plainCause(f.detail)}.`
: `Part of her answer was not read aloud: ${plainCause(f.detail)}.`;
return { record: f.privately ? null : `voice_reading_stopped cause=${cause}`, screen: line, next: 'the next unit' };
}
case 'nothingToSpeak':
return { record: null, screen: null, next: 'the next unit, silently' };
}
}
const cases: Failure[] = [
{ kind: 'turnFailed', reason: 'http_500', firstSeen: true },
{ kind: 'turnFailed', reason: 'http_500', firstSeen: false },
{ kind: 'sendDropped', attempt: 0 },
{ kind: 'sendDropped', attempt: 7 },
{ kind: 'streamLost', cursor: 412 },
{ kind: 'routeChanged', heardSeconds: 37.2 },
{ kind: 'takeFailed', how: 'neverSounded', restarted: false, detail: 'HTTP 504', privately: false },
{ kind: 'takeFailed', how: 'neverSounded', restarted: false, detail: 'HTTP 504', privately: true },
{ kind: 'takeFailed', how: 'stalled', restarted: false, detail: 'stalled 20 s', privately: false },
{ kind: 'takeFailed', how: 'stalled', restarted: true, detail: 'stalled 20 s', privately: true },
{ kind: 'takeFailed', how: 'brokeOff', restarted: false, detail: 'the network connection was lost', privately: false },
{ kind: 'nothingToSpeak' },
];
for (const f of cases) {
const o = settle(f);
console.log(f.kind.padEnd(15), '| record:', o.record ?? '-', '| screen:', o.screen ?? '-', '| next:', o.next);
// Only a first stall is asked for again in silence; every other failed take says so on screen.
const retrying = f.kind === 'takeFailed' && f.how === 'stalled' && !f.restarted;
if (f.kind === 'takeFailed' && !retrying && !o.screen) throw new Error('a failed take went silent');
}
Run the sketch (npx tsx settle.ts) and read each line as a state. Then add the one failure it does not know yet, the listening grant refused while Dad is mid-talk, and walk the private-take cases it already holds. For each, decide what the record keeps, what the screen says and what the next state is, and add an assertion that would have caught the 2026-10-22 mistake: a reply with nothing to speak must never produce a screen line.
Hint
A refused grant is a stop with its reason on the face, as in this track's fourth lesson; the record has nothing to keep because no turn was sent. A private take depends on how it fails. A stream that cannot open is said at once, because a private reply keeps no finished take to fall back on; a stall is asked for again once from where she was heard, like any other. Either way the record keeps nothing, because a private talk keeps no record.
Progress
Progress is local-only — sign in to sync across devices.