Listen, Never Speak

A device voice, briefly

For about a day, Spark could read aloud without me. Listen on lines and expressions used the phone's installed voices through AVSpeechSynthesizer, and the browser's through Web Speech. The reason looked sensible on paper: a weak connection should not leave Dad with silence. Then came the request that a card shown to the other person should also be playable, and the kit's phrase card gained a device Speak button. I wrote that change myself, and it went out in a TestFlight build.

Dad heard it and stopped it.

피파가 bellows 통해서 제대로 읽게 해. 우리가 언제부터 이런 엉터리 tts 썼어?

"Have Pippa read it properly, through Bellows. Since when did we use junk TTS like this?"

언어 하드코딩하지마
spark 당연히 고쳐

"Don't hard-code the language." And, about the Listen on Spark's own rows: "Of course fix Spark."

The device Speak on the card was gone less than twenty minutes after it was committed. The row's device voices followed in the next hour. Nothing of device read-aloud is left in Spark today except a file name: the row's button is still called SourceSpeechButton, and it plays my voice.

Why the reversal is right

Spark's founding rule is that the person across the table hears me, in my Bellows voice, speaking their language. A card with my face on it that then talks in the operating system's voice says two different things about who is speaking. For Dad it is worse: his own phrases read back by a voice he never chose.

The weak-signal reason did not need a second voice. It needed the first one to be kept. The phone already saves each reading's chunk list and audio bytes, so a reading heard once replays without the engine, and that removed the only argument device voices had. New words still need the network, because generating audio is an online action. Spark waits for my voice rather than substituting another speaker.

Principle: A fallback that changes who is speaking is not a fallback. Keep the real voice close instead of swapping in a different one.

No language, except where it was chosen

Dad's second line became a rule in the code. Listen on a reading's line or expression, and the voice on a reading result's Show card, pass no language to Bellows. My Bellows profile runs ElevenLabs v4, which reads words in their own language. AppModel.read (first code block) is the single path for both. The source plays as the result's own voice, reusing a saved take when the engine made one. A reverse card's translation is synthesized as its own take, with language: nil.

The conversation screen is where a language still travels. The card's voice is made by the engine in the scene's language (lesson 2), and the answers and expressions of a conversation play through the same player with that language too (lesson 3), as does the big card any of them opens. It is the language Dad picked and I was told to write in, a choice recorded on the moment. The language tags on results stay for display and for the reverse-card rule in lesson 4.

The button on the card had one more lesson to teach. For a while it said Speak. A waiter, shown a card, read that as an instruction: speak into this, and it translates. And talked into it. The word was right from Dad's side of the table and wrong from the side the card faces.

The fix moved the control into the recipient's language. On the card it says Listen in the card's language, and Stop while playing: Escuchar and Parar on a Spanish card, Écouter on a French one, 聞く on a Japanese one. The web's table is in the vendored kit component (second code block), and the phone uses the kit's PhraseListenButton on the same players as the row. A language the table does not know falls back to English. Dad's own controls on the card stay English. The same change gave every card a Dark card and Light card toggle, light by default and chosen per card, and retired the old Brighten button.

The waiter who talked to the card also started the next track. If people are going to speak into a card, the card can learn to listen. Track 6 is that interpreter.

Heard once, replays offline

The phone's EngineClient.speech (third code block) looks up a saved chunk list before it asks the engine. The key is built from the language and the exact text, so the same words read the same way map to the same saved list. The audio files are saved by their URL when they are first downloaded, and the next request for that URL reads the saved bytes. Bellows' cache keys are content-addressed, so the same text in the same voice always has the same key and the saved audio never goes stale. A reading Dad has heard once plays again on a plane.

Bellows Quest covers the server side of that bargain: paying for a sentence once, and owning the audio after.

Code

cwkSpark, clients/apple/SparkMobile/Sources/AppModel.swift: one path for Listen and the Show card's voice, with no language (verbatim, lines 693-717)·swift

    /// A line's or expression's Listen, and a Show card's voice: Pippa reads
    /// the chosen words through Bellows, never a device voice, and with no
    /// language, because ElevenLabs v4 reads the words in their own (cwkPippa
    /// todo #357, Dad 2026-10-09). The source is the result's own voice (a
    /// saved take when the engine made one); a translation is its own take.
    func read(_ item: PippaResult, _ reading: SourceReading) {
        if reading.translated {
            speak(text: reading.text ?? "", id: reading.playerId(for: item))
        } else {
            speak(item, language: nil)
        }
    }

    private func speak(text: String, id: String) {
        let words = text.trimmingCharacters(in: .whitespacesAndNewlines)
        guard !words.isEmpty else { return }
        let client = self.client
        voice.toggle(id: id) {
            var out: [Data] = []
            for chunk in try await client.speech(text: words, language: nil) {
                out.append(try await client.audio(cacheKey: chunk.cacheKey))
            }
            return out
        }
    }

cwkSpark, frontend/src/kit/components/KitPhraseCard.tsx (vendored from cwkSiblingKit): Listen and Stop in the card's language (verbatim, lines 38-54)·typescript

  /** The recipient's control speaks the card's language; Dad's stay English. */
  listen: {
    'en': { listen: 'Listen', stop: 'Stop' },
    'es': { listen: 'Escuchar', stop: 'Parar' },
    'ca': { listen: 'Escoltar', stop: 'Aturar' },
    'fr': { listen: 'Écouter', stop: 'Arrêter' },
    'it': { listen: 'Ascolta', stop: 'Ferma' },
    'pt': { listen: 'Ouvir', stop: 'Parar' },
    'de': { listen: 'Anhören', stop: 'Stopp' },
    'nl': { listen: 'Luisteren', stop: 'Stoppen' },
    'ja': { listen: '聞く', stop: '停止' },
    'zh': { listen: '收听', stop: '停止' },
    'zh-hant': { listen: '收聽', stop: '停止' },
    'zh-tw': { listen: '收聽', stop: '停止' },
    'zh-hk': { listen: '收聽', stop: '停止' },
    'ko': { listen: '듣기', stop: '정지' },
  } as Readonly<Record<string, { listen: string; stop: string }>>,

cwkSpark, clients/apple/SparkMobile/Sources/EngineClient.swift: a reading's chunk list is saved for offline replay (verbatim, lines 246-263)·swift

    /// Pippa saying `text` in `language`: synthesized once by cwkBellows and
    /// cached forever; the cache keys play from /api/tts/audio/{key}.
    func speech(text: String, language: String?) async throws -> [PippaResult.VoiceChunk] {
        struct Body: Encodable {
            var text: String
            var languageCode: String?
            enum CodingKeys: String, CodingKey { case text, languageCode = "language_code" }
        }
        struct Reply: Decodable { var chunks: [PippaResult.VoiceChunk] }
        let key = "speech/" + OfflineKey.filename((language ?? "") + "\u{0}" + text)
        if let saved: [PippaResult.VoiceChunk] = offline?.value(key) { return saved }
        var request = try request("POST", "/api/tts/ensure")
        request.setValue("application/json", forHTTPHeaderField: "Content-Type")
        request.httpBody = try JSONEncoder().encode(Body(text: text, languageCode: language))
        let chunks = (try decode(try await mediaTransport(request)) as Reply).chunks
        offline?.save(chunks, key: key)
        return chunks
    }

Exercise

Add offline replay to a text-to-speech feature in your own app. Cache the server's chunk list under a key built from the language and the exact text, and cache each audio file by its URL on first download. Turn the network off and play a phrase you heard once. Then play a new phrase and make the app say plainly that it needs the network, with no fallback to a system voice. Finally, label the play control on any screen that faces another person in that person's language, from a table with an English fallback.

Hint

Keep the two caches separate. The list of keys for a text can change if the server's chunking changes, while the audio behind a content-addressed key never does, so the audio needs no invalidation. Test the offline path with the network really off, not with a mocked client. For labels, normalize the language tag (lowercase, underscores to hyphens), then try the full tag, the first two parts, and the bare language, in that order.

Listen, Never Speak

A device voice, briefly

Why the reversal is right

No language, except where it was chosen