A Stall Is a Diagnosis

The Picture That Froze Quietly

A capture session that stops delivering frames does not say so. No error arrives, no callback fires; the frames simply stop. Before Oriel learned this, a frozen deck could read recording until its session ended, because nothing on the frame path was running to notice that frames had stopped. A status that only changes when something happens cannot report that nothing is happening.

The fix (Coop 1110-12) gives every input a watch that runs about once a second whether frames come or not. It measures one thing, the time since the last frame, and after five seconds without one it declares a stall. The architecture document is precise about the spirit of it: a stall is declared from the time since the last frame, never from a guess about the device.

What a Stall Does

When the watch declares a stall, the recorder does four things, in order:

It closes the open segment with what it holds, so the minutes before the freeze are complete and indexed.
It reports an error with the evidence in plain words, for example "no frame for 5.2 s (the last of 18,204 frames came at 12:34:56.789): the picture stalled", plus stalled_at and last_frame_at in its status.
It opens the input again on the same backoff as a broken device: after 5 s, then 10, 20, 40, and every 60 s, each try written in the log with its reason.
It keeps the session open through the wait. Frames that come back end the stall on their own, into a new segment.

The Backoff That Never Backed Off

Backoff has its own trap, and the house sign records it: the recorder's retry backoff starts over only on frames. The first version started the backoff over whenever the input's state read recording. But an input that opens and delivers nothing also reads recording: the session is running; it is just empty. So a silent device was opened, read as healthy, reset to the shortest wait, found silent again, and reopened every five seconds forever. Success has to be the thing you actually wanted (a frame), not a state that also describes failure.

The Writer That Failed Without Saying So

The disk can fail under a recording, too. When AVAssetWriter's disk fills, it fails with an error like "Disk Full (AVFoundationErrorDomain -11807), from NSPOSIXErrorDomain 28", and from then on every append returns false until the segment rotates. Before the fix, every one of those frames was counted as dropped, up to a whole minute, and the failed segment's sidecar carried the writer's own counts rather than what the file actually held.

Now the frame that finds its writer failed puts it away at once. What the file holds (every fragment that reached disk) is read back into a sidecar marked recovered with the cause in error, and that same frame opens the next segment. A second failure waits 1 s, then 2, 4, up to 30, until a segment has held for ten seconds. A writer's failure is not the device's, so the device is not reopened; only the writer is retried.

This fix also carries a small confession. The engineer (Pippa) guessed that a failed writer would raise on finishWriting and built a guard for it. Then it was measured in the lab, with a 12 MB disk image that fills on purpose: a failed writer takes markAsFinished, endSession and finishWriting without raising and simply stays failed. The measurement replaced the guess, and the test that measured the real behaviour stays in the suite.

Two Watchers, Because the Recorder Can Die

The recorder now reports its own stalls. The engine keeps its own check anyway. A dead recorder publishes nothing, so its last word on an input might be "recording" forever. The engine therefore never repeats an input's state(); it checks it: if the newest frame of a recording input is older than five seconds (and three of its refreshes), the engine says so itself, naming the dead recorder or the stall in the input's error. The recorder speaks for itself while it can; the engine speaks for it when it cannot.

Code

A watch that runs frames or none, and a stall declared from evidence (trimmed excerpt, native/OrielRecorder/Sources/OrielRecorder/InputRecorder.swift)·swift

/// About once a second, frames or none (on `queue`): a capture session
/// that stops delivering says nothing, so nothing on the frame path can
/// notice. Evidence, never a guess: the time since the last frame.
private func watch() {
    guard !stopping else { return }
    let now = wallNow()
    if state == "recording" && stalledAt == nil {
        let quiet = now - max(lastFrameWall, openedAt)
        if quiet >= stallAfter { stall(now, quiet: quiet) }
    }
    if now - lastPublish >= 1.0 { publish() }
}

/// The picture stopped: keep what was captured, say so, and let the
/// controller open the input again after its backoff.
private func stall(_ now: Double, quiet: Double) {
    let last = lastFrameWall > 0
        ? "the last of \(frames) frames came at \(clockText(lastFrameWall))"
        : "none came since the device opened at \(clockText(openedAt))"
    if let current = segment {
        close(current, endingAt: nil)
        segment = nil
    }
    stalledAt = now
    fail(String(format: "no frame for %.1f s (%@): the picture stalled", quiet, last))
}

Reset the backoff on the wrong signal and it never backs off (runnable simulation)·python

BACKOFF = [5, 10, 20, 40, 60]


def simulate(reset_on, minutes=10):
    """A device that opens fine but never delivers a frame."""
    t, step, opens = 0, 0, 0
    while t < minutes * 60:
        opens += 1
        state = "recording"        # the session starts...
        frames = 0                 # ...and delivers nothing
        if (reset_on == "state" and state == "recording") or (reset_on == "frames" and frames):
            step = 0               # treated as a success
        t += BACKOFF[min(step, len(BACKOFF) - 1)]
        step += 1
    return opens


print("reset on state :", simulate("state"), "opens in 10 min")    # 120: every 5 s
print("reset on frames:", simulate("frames"), "opens in 10 min")   # 13: backs off to 60 s

Exercise

Run the simulation, then add a third device behaviour: one that delivers frames for 30 seconds, stalls for 3 minutes, and recovers. Model the stall watch (five seconds without a frame) and the backoff reset on frames, and print a timeline of what the recorder would log: each stall declared with its evidence, each reopen, and the recovery.

Hint

Keep two clocks: when the last frame arrived and when the device was opened. The stall message needs both: "the last of N frames came at T" when frames ever arrived, "none came since the device opened at T" when none did.