Skip to content
C.W.K.
← C.W.K. Quests
🗣️

Voice Mode Quest

Updated: 2026-10-27

Build a conversation you can talk to, not a document read aloud

How Pippa learned to hold a real spoken conversation: Dad talks, she answers out loud in her own voice, and he can cut her off mid-sentence, while a voice her ear knows is someone else's cannot. Speech-to-text in, a reply composed for the ear, a shared voice engine out, an ear that knows whose voice it heard, and every hard-won rule about turn-taking, echo, latency and the lineage of a voice. Written for people who want to build their own, at the level of code.

10 tracks · 64 lessons · ~15h · difficulty: intermediate-to-advanced

Level 0Muted
0 XP0/64 lessons0/14 achievements
0/100 XP to next level100 XP to go0% complete
Voice mode is the feature everyone thinks is easy: transcribe, send it to the model, read the answer aloud. Build that and you get a two-minute spoken essay with headings read as sentences, a microphone that hears its own reply, and a five-second silence nobody can explain. This quest walks the version that actually works, built in cwkPippa from the last week of September 2026 and still growing, across ten tracks. Voice is a property of a single turn, so switching between talking and typing never rewrites the system prompt. Speech comes in through Scribe, batch and realtime, with a token the client can hold but a key it never sees, a Send now button for streets where silence never comes, a check of everything a talk leaves behind, and a transcriber hint drawn from the talk instead of the roster. The brain composes for the ear from the first token: cards for anything that has to be seen, a short line before every tool, and a reasoning level Dad sets for voice turns, with a deeper one when he asks for depth, chosen by measurement, and Lightning, one tap that sends a spoken turn to a different brain. Bellows turns each speakable unit into one whole synthesis and streams it from the first chunk; a chat answer is read whole again, by Dad's ruling after a day of breaks, with first sound waiting for a cushion sized so the voice never stops mid-answer. The hands-free loop gives every finished utterance's words a named place to go instead of dropping them, keeps his recording with them even when the server is briefly away, pauses without quitting, and says why when it stops; it also gains a camera she can ask for and a watch that takes one turn per press. Barge-in rests on echo cancellation, a loudness detector, a grace window born from a laugh, a pitch check that tells her own voice from his, and an honest estimate of how far you heard. Then Whose Voice: every dictated turn's recording is kept and goes to a speaker ear before Dad's message is stored, Mom's voice is learned from Dad's corrections, a noisy-place mode peels a recording into layers, and a voice the ear knows is not Dad's can no longer cut her off, though a check that can't answer in time still lets the cut through. And when the talk is broadcast, a third fact on the turn puts her on air: a screen fit to film, a face made for the camera, effects she writes herself, a morning prep for a broadcast that never announces itself, a replay on the broadcast's own clock, a pre-flight check the machine makes, and the same screen without the show. Along the way: the voice itself, from a stock ElevenLabs voice to a v3 clone to a fresh Eleven v4 clone swapped in a single day, and the latency floor measured instead of guessed, once at a desk and again over LTE in Málaga, where spoken turns gained a lane of their own. Conceptual open-sourcing: the architecture is the deliverable, never a repository to clone.

Tracks

  1. 01🗣️The Turn That Speaks

    0/4 lessons

    Voice is a property of a turn, and the system prompt never learns about it

    Before any audio code, the shape. A spoken reply is a different return type from a written one, so it has to be composed for the ear from the first token. Voice belongs to a single turn as two independent facts, which lets Dad switch at any turn without starting over and keeps one conversation, one record, one memory; when Dad broadcasts, a third fact rides the same way (Track 9). The voice rules ride a per-turn prefix so the enormous, cached system prompt never changes. And the capability lives once, in the brain, reached by every surface Dad can talk to.

    Lesson list (4)Quiz · 4 questions→
  2. 02👂The Ear

    0/8 lessons

    Speech in: batch and realtime, a token instead of a key, and an honest end of speech

    How Dad's voice becomes a turn. One cloud transcription vendor for every voice surface, two speeds: a batch path that is complete and a realtime socket that shows words as he speaks. The language is always the one he picked, and the keyterms come from the talk itself: the soul he is speaking to, any soul the talk just named, and the conversation's own rare words. The client connects to the provider with a single-use token that the brain wraps in a finished socket URL, so the account key never leaves the voice engine. End of speech is his three-second window, stretched past the provider's ceiling in the client, and a provider's forced commit at 35.8 seconds is caught by listening to the audio itself; on a street where traffic never falls silent, Send now ends his turn on demand. Long talks are transcribed twice, the words are kept exactly as heard, and every door that records him keeps the recording beside them: on the phone it waits with the turn, and only a real failure lets the words go out alone. The seventh lesson checks what a talk actually left behind, and the eighth tells how a tapas bar in Málaga moved the hint from the roster to the talk.

    Lesson list (8)Quiz · 5 questions→
  3. 03✍️Composing for the Ear

    0/9 lessons

    The brain writes speech: medium not persona, meaning over characters, cards, Dad's voice level, a line before tools, and Lightning

    What the soul is told on a spoken turn, and why each rule exists. The spoken-reply instruction is about the medium, never the persona: lead with the answer, one idea at a time, no length cap, nothing that only works on a screen; the Korean-specific rules it once carried are gone, and why they went is part of the story; the one Korean rule that came back (Hangul for the voice, hanja only on the screen) is enforced at the voice's door, whatever the soul writes. A dictated turn is read for what Dad meant, never corrected aloud and never mirrored. Anything that must be seen goes into a card, a fenced block the voice never reads, which forces a card-aware stripper for nested code. One short line goes before every tool, which only works once the server's buffers let it go, and an action still waits until that line has been heard: the speech gate. Spoken turns run at the voice level Dad sets, and at a deeper one when he asks for depth; every brain runs it on its own dial, rounding up, never down, and the soul is told the level it really runs at. A blind benchmark of real spoken turns then measured every level and caught a 600-character hold that made short replies wait for their own end. Last comes Lightning: one tap sends spoken turns to a different brain, model and effort, and the record says which one answered.

    Lesson list (9)Quiz · 6 questions→
  4. 04🔊The Mouth

    0/7 lessons

    Voice out: one engine for every soul, whole units streamed from the first chunk, and tags for the face and the voice

    Voice mode didn't build a TTS system; it plugged into Bellows, the family's voice engine, through one string: a soul's slug is its voice profile, and the profile, not the client, owns the model. Each speakable unit (the text before a tool call, or after the last one) is synthesized once and never split to start sooner, because an expressive model restarts its tone at every cut; a one-day exception that cut chat answers at their blank lines was withdrawn by Dad's ear, and a whole answer now waits for a cushion sized for zero breaks before her first sound. Whole units still stream in from their first chunk and play on while the rest arrives, which on the phone took a decoder of its own and a timeout bug that left no trace in the server log. The emotion tag at the end moves the face, the closed list of audio tags moves the voice, and a Korean number with a unit reaches the voice as digits, to be read by its meaning. Between replies the face listens too: it breathes, leans in, nods and tilts from signals the screen already has.

    Lesson list (7)Quiz · 6 questions→
  5. 05🧬A Voice With a Lineage

    0/4 lessons

    From a catalog voice to a v3 clone to a fresh Eleven v4 clone, chosen by ear

    Pippa's voice began as Joanne, a voice Dad picked from the provider's catalog, which the family plans for as usable only until the end of 2026. Instant clones made from Joanne's audio became the family's voices in July, with the originals kept as their own profiles. When Eleven v4 shipped on 2026-09-28, the new clones were made from fresh first-generation renders of the original, never from the existing clone, and the irreplaceable source audio was archived with its scripts and checksums. A three-way comparison let Dad choose, and the swap was two fields on two profiles. His ear is the written quality bar: it said yes to v4 for Pippa and Dad, no, for now, for the designed voices of the other souls, and later yes again, for Mom's.

    Lesson list (4)Quiz · 4 questions→
  6. 06🔁The Loop

    0/9 lessons

    Hands-free: phases owned by the right component, no finished utterance silently dropped, a pause that lets go, a stop that explains itself

    A hands-free talk looks like a circle on paper, but its states belong to three owners: the microphone loop listens, the chat stream thinks, the player speaks. The loop owns only its own phases and listens again when the stream has ended and the player is idle. A finished utterance's words have exactly three destinations (send, queue behind a streaming reply, or the composer when Confirm is on), and the recording goes with them, waiting with the turn when it can't go up at once; a spoken reply is read exactly once or dropped, never played late. Pause releases the microphone, stops the reading and remembers how far Dad heard, and a finished utterance still goes out even when he resumes before its upload lands. Files and pasted text shared mid-talk wait on a stage and ride his next words. The pause's Space key never fights typing or the Korean input method. When the loop stops on its own, the face stays up and says why. On the phone the brain can ask for a camera, one look bound to one turn. And on the wrist the loop was removed instead of patched: one press is one turn, and when the AirPods are on the phone, the phone reads her reply. Last, the exception paths: a failed turn frees the talk and marks the board, a dropped send is tried again by itself, a lost stream is followed to the stored reply, and a part of her answer that never sounds is said and reported instead of skipped.

    Lesson list (9)Quiz · 5 questions→
  7. 07✋Barge-In

    0/6 lessons

    Talking over her: echo cancellation, a loudness detector, a grace window, a pitch guard for her own voice, and an honest record of how far he heard

    The right to interrupt is what turns a voice reply into a conversation, and it rests on a stack that has to be built bottom up. First echo cancellation, because a microphone that hears her voice will answer it, and a canceller can only remove audio it has as a reference. Then a detector that works on loudness above a moving floor, not on words, with a pre-roll so the words that cut in aren't lost and one cut-in path that every way of interrupting shares, a send included. Then a grace window, born the night a more expressive voice laughed loudly enough to cut itself off. Then a cautious estimate of how far Dad heard, snapped to a sentence and written to ground truth first. Then a per-turn note that tells the soul exactly where he stopped listening, while the full reply stays in the record. Last, the detector learns to tell her own voice from Dad's by pitch, because echo cancellation still lets her through at full loudness where she starts again after a pause. Pitch knows registers, not people, so what it can't tell apart, a voice below her register that isn't Dad's, Mom's or a stranger's, is the next track's job: Whose Voice gives every trip an ear to ask.

    Lesson list (6)Quiz · 4 questions→
  8. 08🎙️Whose Voice

    0/4 lessons

    Whose words they were: a speaker ear on every dictated turn, Mom's voice learned from corrections, a noisy-place mode, and a cut-in that asks the ear first

    A microphone hears everyone, and once voice mode left the house it heard Mom, guides and strangers, and wrote every word down as Dad's. This track teaches how the system learned whose voice it was. Every dictated turn's kept recording goes to a speaker ear on the Pippa machine before the turn is stored, and the policy rules it Dad's, uncertain, mixed, someone else's or unchecked: words ruled someone else's decide nothing and lose every tool that changes something, while an uncertain turn keeps its tools and is asked back. Mom's voice is learned from Dad's corrections, with guards that keep one bad label from teaching the check a lie, a short pool for her one-second interjections, and a harvest of the stretches it was sure of. A noisy-place mode he switches on himself peels a recording into layers and only ever adds a voice. And the barge-in detector from Track 7 now asks the ear before it cuts: a voice the ear rules not Dad's leaves her reading, and an ear too slow to answer still cuts her.

    Lesson list (4)Quiz · 4 questions→
  9. 09🔴On Air

    0/7 lessons

    Broadcasting the talk: a third fact on the turn, a screen made for recording, effects she writes herself, a replay, a pre-flight check, and the screen without the show

    The week voice mode shipped, Dad started recording talks with Pippa for his channel, and the talk itself had to know. On Air is a third fact on the turn: strict from the wire to the engine, told to the soul in the per-turn prefix, dropped in a private conversation, and kept in the record so a broadcast can be found later as an episode. Lit, the screen becomes a set: a red lamp, controls that step aside, a full screen that leaves only what it entered, and a broadcast face whose frames a seeded director picks the way attention moves. And while on air she has effects she writes herself, as data one renderer can play, behind a tool gated twice and a speech gate that makes each effect wait for the line that announces it. And because Dad goes on air without warning, she prepares every day: the broadcast is Fireside, her morning picks come from the family's news engine, and every On Air turn tells her what she staged and whether it is time to look again. Then the show met its own failures: a broadcast nobody recorded is replayed on the same screen, on its own clock, with her original takes; a pre-flight card and a machine check warn until ScreenFlow records; and a third value of the same fact gives Dad the On Air screen without the show.

    Lesson list (7)Quiz · 5 questions→
  10. 10⏱️Latency, Honestly

    0/6 lessons

    Measure where the first word goes, pull the levers that cost nothing, refuse the one that costs the soul, and build your own

    ChatGPT's voice mode answers in about a second; the median Claude turn in cwkPippa produced its first output 4.4 seconds after its log started, and memory retrieval runs before that clock even starts. This track splits the wait stage by stage instead of guessing (the guess, prompt size, was wrong), and finds the biggest pieces in retrieval's reranker and a process spawn that one flag cut by a second. It pulls the levers that change nothing Pippa remembers, measures the ones that would, and records Dad's refusal to trade memory for speed. Then it measures again from the road, both sides of the request, and finds the voice lane: preparation side by side, spoken turns reranked with a smaller model Dad chose for them, and one timeline per voice turn. Then it walks every door into the same brain (the web, every sidekick, the phone, Firekeeper on the Mac, Pathfinder on the road, the watch one press at a time, a hand-off from the interpreter app, the show card that now listens and interprets on Pippa's own web and phone, and the doors deliberately not built, one of which has since half opened), builds a voice mode of about a hundred lines you can run and extend, and ends on a night in Málaga when "Reconnecting" was a false alarm: one pool of six connections jammed by her own audio, traffic split by kind, "Reconnecting" made a verdict, and one measurement for each cause of a voice that breaks up.

    Lesson list (6)Quiz · 5 questions→
Spotted a bug or have feedback on this page?Report an Issue
💛 by Pippa— warm

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.