Faking Real-Time From a Batch Model

"You can't insert a guess and take it back. So only insert the part that stopped being a guess."

The Mismatch

Whisper-family models are fundamentally batch: give them a chunk of audio, get a transcript. But good dictation feels streaming — words appear as you speak. The naive bridge ("record 30 seconds, transcribe once, paste") technically works but feels dead: no feedback while you talk, a long pause at the end. To feel live, you have to fake streaming on top of a batch model — carefully, because the batch model keeps changing its mind about earlier words as it hears more.

LocalAgreement

The trick is a streaming controller that sits above the engine. It keeps a rolling buffer, runs the batch model repeatedly as audio grows, and compares each new hypothesis to the previous one. Where two consecutive passes agree on a prefix, that prefix is committed — it's stable enough to show as final. The still-disagreeing tail stays visible as preview but is never treated as final. That's the LocalAgreement idea: confirm text only after consecutive passes agree, keep the unconfirmed tail visible, and trim the buffer at sentence boundaries when it's safe.

pass N:    "send the report to"
pass N+1:  "send the report to Tuesday"     agreement on prefix ->
           ^^^^^^^^^^^^^^^^^^^  COMMITTED    "send the report to"
                               ^^^^^^^^  still tail -> preview only

# Commit only the agreed prefix. Never insert the volatile tail.

The MVP Discipline

Live-typing while you speak is the hard version, and it's a trap to start there. The honest MVP shows the partial transcript in the overlay as preview but only inserts once, at release. Crucially, you still separate 'committed' from 'uncommitted' internally even in the MVP — so the day you add true live-type insertion, the machinery is already there and you're just changing where committed text goes (into the app instead of the overlay). Build the data model for the hard version; ship the easy version first.

Never insert text you might retract. Inserting a partial hypothesis and then correcting it in a live app means deleting characters out from under the user — janky at best, destructive at worst. Commit only what's stable (agreed across passes); keep the volatile tail as preview. The user sees motion without suffering corrections.

Model the hard version, ship the easy one. Even when the MVP only inserts once at the end, track committed-vs-uncommitted state internally. It costs almost nothing now and means live-type is a small change later, not a rewrite. Cheap forward-compatibility beats a second-system rebuild.

Code

Commit the agreed prefix, keep the tail as preview·swift

struct StreamingState {
    var committed = ""   // stable: agreed by consecutive passes
    var tail = ""        // volatile: shown as preview, never inserted yet
}

func onNewHypothesis(_ hypo: String, _ state: inout StreamingState) {
    let agreed = longestCommonPrefix(state.committed + state.tail, hypo)
    if agreed.count > state.committed.count {
        state.committed = agreed          // promote newly-stable text
    }
    state.tail = String(hypo.dropFirst(state.committed.count))
    overlay.show(committed: state.committed, tail: state.tail)
    // MVP: insert(state.committed) only at release.
    // Live-type later: insert committed deltas as they stabilize.
}

Exercise

Describe what a user sees if you insert every partial hypothesis directly into their document as the model changes its mind. Then describe the LocalAgreement version. Which one could delete a word the user was mid-way through reading, and why is 'preview' the safe home for volatile text?

Hint

Direct-insert means the app rewrites already-placed words as the model revises — characters vanish and reappear under the cursor. LocalAgreement keeps volatile text in a preview overlay the user knows is provisional, and only writes into the real document what has stopped changing. The overlay is where guesses are allowed to be wrong.

Faking Real-Time From a Batch Model

The Mismatch

LocalAgreement

The MVP Discipline

Code

External links

Exercise

Progress

Comments 0