The Speaker Check and Its Verdict

"It could be picked out from the waveform anyway... It should be something you can handle in the audio." — Dad

The Microphone Hears Everyone

Track 7 ended with pitch: a loud chunk in her register is her own voice coming back, and it never counts as Dad. That guard knows one voice, hers. It knows nothing about Mom, a tour guide, or a waiter. While Dad traveled with Mom and talked to Pippa out loud on streets, in cafes and in tour groups, the microphone heard all of them, and every word the transcriber wrote came in as his. A guide's sentence and Dad's approval looked exactly alike in the conversation.

The transcriber can't fix that. Scribe's realtime socket has no diarization at all, and the batch call's diarization gives relative labels, speaker one and speaker two, never Dad. Words alone can't say whose they were. Dad's guess was that the sound can, and he was right.

The Audio Is Already There

What made a server-side check cheap is a ruling from Track 2: keep his voice. Every dictated turn already names its kept recording in the chat request, so the audio sits on office before the turn is stored. Every chat route (Claude and all six variants) sends that recording (the 16 kHz take the listening socket heard, when there is one) to a speaker ear before it writes the user row.

The work splits across two houses. cwkBellows owns the ear: the models, the speaker profiles (kept as embeddings, never as audio) and the routes under /api/speaker/. cwkPippa owns the policy, in services/speaker_check.py: what a verdict lets a turn do, what the stored message says, and what the soul is told. The ear names voices. It never decides what a turn may do.

How the Ear Hears

The recording is decoded to 16 kHz mono and its first 120 seconds are examined. Silero VAD finds the speech, and the speech is cut into chunks of about two seconds. NVIDIA TitaNet-large, run on the CPU through sherpa-onnx with no torch, turns each chunk into an embedding, and the chunk is named for the closest enrolled profile if its cosine score reaches 0.30. A lone chunk between two of one person's is taken as that person's unless it was a sure match for someone else. Each run of one name becomes one voice, embedded again over up to 30 seconds of its speech and scored: match at 0.45 or more, uncertain from 0.25, nobody below that. Less than a second of speech is a match only at 0.60.

The model was chosen on Dad's own data: 769 kept turns against 227 recordings of 49 other voices.

Model	Dad's median	Other voices' max
3D-Speaker CAM++	0.79	0.40
WeSpeaker ResNet34	0.94	0.88
3D-Speaker ERes2NetV2	0.88	0.53
TitaNet-large	0.84	0.21

The highest median wasn't the point. The gap was: TitaNet put every other voice near zero. Identifying is free and local, 0.08 s for a 4-second turn and 2.1 s for a 110-second one. An earlier design ran pyannote segmentation and clustering and was dropped: slower, and it answers how many voices when the question is whose.

Five Verdicts

Verdict	When	What the turn does
`dad`	every voice that spoke a second or more is his, and one is sure	Nothing changes. No block.
`uncertain`	his voice, but not sure; or a word or two nobody could rule on	Text unchanged. The block says: answer, but ask "아빠가 말한 거 맞아?" before acting on a decision, approval or command.
`mixed`	Dad and someone else both spoke	One paid Scribe pass with word timestamps labels the lines `[Dad]`, `[Mom]`, `[someone else]`.
`other`	no voice of his, and 2 s or more of the other one, or a sure match to another enrolled voice	The stored message becomes `[Mom] …` or `[someone else] …`; nothing in it decides anything.
`unchecked`	no recording, no speech, or the ear did not answer	Exactly as before the check existed.

Nothing is dropped. The text as the client sent it rides voice_meta.speaker.original_text, and the labels ride the row itself, so every replay, every brain and the search index read the same thing.

Two Rules the First Live Run Wrote

A word or two is never ruled a stranger's: Dad's one-word turns scored 0.37 at the median, and calling his "응" someone else's would throw his approvals away. And an unknown voice alone with under two seconds of speech is uncertain, not other: a 1.4-second turn of his on the road scored 0.20 against his own voice, and asking him back costs less than dismissing him. A third came with them: when the other voice said no word Scribe heard (a laugh, a murmur), the turn stays his.

Where the Hard Gate Sits

The ask-back is an instruction in the per-turn block, the Admin-editable speaker_check_instruction, not a code gate. Code gates sit in two places. A turn ruled other loses every tool that changes something (clamp_tools); reading, searching and answering stay. uncertain keeps its tools on purpose: a hard strip would refuse Dad's own "응" again on the very turn that confirms it. And the voice_check door, the one thing that changes whom the check trusts, refuses unless the turn asking is typed or spoken in a voice ruled dad.

Replayed over the same data, Dad's 769 turns came out 709 dad, 34 uncertain, 15 mixed, 3 other and 8 unchecked; 200 LibriSpeech recordings came out 0 dad. No other voice was ever taken for his.

What It Can't Do

Two people at the same moment are not separated: an embedding of a mixture follows the stronger voice, and Dad talking over Mom scored Dad 0.55 and Mom 0.09 even with her twice as loud. With office unreachable there is no check. Bellows down means unchecked and a chat that runs as it always did; PIPPA_SPEAKER_CHECK=0 closes the check, and Bellows' speaker_thresholds setting moves any line without a deploy.

Code

The ear's bands, verbatim (cwkBellows backend/bellows/speaker.py:112-121)·python

# Cosine similarity to a profile's centroid, calibrated on Dad's own 769 kept
# turns against 227 recordings of 49 other voices (cwkPippa doc §7). A voice
# at or above ``match`` is the profile's; below ``uncertain`` it is nobody's;
# between, nobody can say. Measured: other voices topped out at 0.21 (a
# whole turn) and 0.24 (a two-second chunk); 95% of Dad's turns scored above
# 0.49 and 95% of his chunks above 0.38. A chunk is named at ``chunk``. Less
# than ``MIN_IDENTITY_SECONDS`` of speech is a match only at ``short_match``
# (Dad's one-word turns scored 0.37 at the median). The
# ``speaker_thresholds`` setting overrides any of them without a deploy.
DEFAULT_THRESHOLDS = {"match": 0.45, "uncertain": 0.25, "chunk": 0.30, "short_match": 0.60}

A minimal sketch: the verdict rule from speaker_check.verdict_of, type checks trimmed, run on nine turns·python

# The verdict rule, as verdict_of in cwkPippa's speaker check rules it.
DAD_PROFILE = "dad"
SIGNIFICANT_SECONDS = 1.0     # less speech than this never changes a verdict alone
SURE_STRANGER_SECONDS = 2.0   # an unknown voice alone needs this much to be "other"


def is_dad(v):
    return v.get("profile_id") == DAD_PROFILE and v.get("band") in ("match", "uncertain")


def verdict_of(voices):
    if not voices:
        return "unchecked"
    significant = [v for v in voices if v["seconds"] >= SIGNIFICANT_SECONDS]
    if not significant:
        if any(is_dad(v) and v["band"] == "match" for v in voices):
            return "dad"
        if any(v.get("profile_id") not in (None, DAD_PROFILE) and v["band"] == "match" for v in voices):
            return "other"
        return "uncertain"
    dad = [v for v in significant if is_dad(v)]
    others = [v for v in significant if not is_dad(v)]
    if not others:
        return "dad" if any(v["band"] == "match" for v in dad) else "uncertain"
    if dad:
        return "mixed"
    enrolled = any(v.get("profile_id") not in (None, DAD_PROFILE) and v["band"] == "match" for v in others)
    if enrolled or sum(v["seconds"] for v in others) >= SURE_STRANGER_SECONDS:
        return "other"
    return "uncertain"


def voice(profile, band, seconds):
    return {"profile_id": profile, "band": band, "seconds": seconds}


turns = {
    "an ordinary turn of his": [voice("dad", "match", 6.2)],
    "his voice through wind": [voice("dad", "uncertain", 4.0)],
    "a one-word yes, unsure": [voice("dad", "uncertain", 0.6)],
    "he talks, Mom answers": [voice("dad", "match", 3.1), voice("mom", "match", 2.4)],
    "a guide, 5 s, nobody known": [voice(None, "none", 5.0)],
    "1.4 s of nobody known": [voice(None, "none", 1.4)],
    "Mom alone, short but sure": [voice("mom", "match", 0.8)],
    "a cough beside his sentence": [voice("dad", "match", 5.0), voice(None, "none", 0.4)],
    "no speech found": [],
}
for name, voices in turns.items():
    print(f"{name:<30} -> {verdict_of(voices)}")

# Output:
# an ordinary turn of his        -> dad
# his voice through wind         -> uncertain
# a one-word yes, unsure         -> uncertain
# he talks, Mom answers          -> mixed
# a guide, 5 s, nobody known     -> other
# 1.4 s of nobody known          -> uncertain
# Mom alone, short but sure      -> other
# a cough beside his sentence    -> dad
# no speech found                -> unchecked

Exercise

Run the sketch and read the nine verdicts. Then add three turns of your own: Mom (a sure match) and a stranger together with no Dad, 1.5 s each, Dad unsure for 3 s beside 1.2 s of nobody, and a 0.9-second sure match to Dad. Predict each verdict before you run it. Finally, pick the one line in verdict_of you would most like to change, and write down which mistake the change would buy and which it would cost.

Hint

Mom and a stranger are both 'others': with a sure match to an enrolled voice among them the turn is other, without one it needs 2 s of their speech together. Dad unsure plus a significant stranger is mixed, because any significant Dad voice beside another voice is mixed whatever his band. The short sure match is dad. Every line here trades a stranger slipping through against Dad being doubted; the code leans toward asking him rather than dismissing him.