Every Read Starts at the Index

The Two-Step Read

Every frame Oriel reads from a recording, for any purpose, takes the same two steps. First, one index lookup: Store.segment_at(input, t) asks SQLite for the closed segment whose start is at or before t and whose end is after it. No file is opened for that. A moment inside the segment the recorder still has open, which has no sidecar yet, is the one exception: the engine takes that file's path and start from the recorder's status instead. Second, one short ffmpeg process on that one segment, seeking to the offset t - start_at inside it. The house sign states the rule as an invariant: every segment has a sidecar, and every frame read starts from the index; no read decodes a whole file.

The measured costs on the home Mac show why it matters: an exact 4K frame in 82 ms; on the 1080p, 40 Mb/s recordings since, an exact frame in 185 ms. A design that had to open and scan an hour-long file would be orders of magnitude slower, and the cost would grow with the length of the recording. Here it does not grow at all.

Three Ways to Read a Frame

backend/frames.py has three readers, and choosing between them is most of the craft:

Reader	What it returns	How	Measured
`extract`	the exact frame at a moment, full resolution	`-ss` before `-i`, decode to that frame, raw RGB out	185 ms for 1080p (470 ms as PNG, same pixels)
`view`	a keyframe near a moment, fitted small	keyframes only, no accurate seek, ffmpeg scales and encodes JPEG	97 ms of ffmpeg CPU per 1080p frame
keyframe pass	every keyframe in a window	non-key packets dropped at the demuxer	0.38 s for a 20 s window

Two details carry most of the speed. In extract, the frame comes out as raw RGB bytes of the size the sidecar already knows, so ffmpeg does not pack a PNG and Python does not unpack one; a frame that comes out a different size is read again the slow way. In the keyframe pass, the flags -discard nokey plus -skip_frame nokey drop non-key packets at the demuxer. With -skip_frame nokey alone, ffmpeg still hands every packet to the decoder to parse and throw away: 2.3 s for a 20 s window and 6.2 s for 60 s, against 0.38 s and 0.69 s with the demuxer dropping them, the same frames byte for byte.

The Read That Answered Nothing

Fast readers have sharp edges, and one of them cost Oriel four days. A sidecar's duration ends one frame after the last frame starts. A keyframe-only read answers with a keyframe near its moment, usually the next one, and sometimes the one just before ( on a test clip with a keyframe every 0.25 s, asking ffmpeg 9.0.2 for 1.01 s or 1.02 s still returned the 1.00 s keyframe, so it is a sample, never an exact frame). Ask such a read for a moment near the very end of a segment, far enough past the last keyframe, and ffmpeg finds nothing and writes nothing. (Not just any moment past it: on the test clip, asking 0.01 to 0.03 s after the last keyframe still returned that keyframe, and 0.05 s after it returned nothing.) On the ffmpeg 8 the house ran then, it also exited with status 0; ffmpeg 9.0.2 answers the same request with an error status (234) and still no bytes, so a reader has to treat both as "no frame". The background pass that checks recordings for people (track six) sampled exactly such a moment, and that one unreadable read stopped the pass every time it came round: from 2026-11-02 to 11-06 nothing behind that segment was checked. The fix was twofold: sample a frame and a GOP short of a segment's end, and hold back an unreadable segment for an hour so the pass moves on to the ones behind it.

The general lesson is the one every pipeline eventually learns: a process that exits 0 has not necessarily done anything. Check the output, not only the status.

"Now" Is a Few Seconds Ago

A read of the live moment has its own wrinkle. A fragment reaches disk about a second after its frames are captured, so the engine treats "now" in a recording as LIVE_LAG_SECONDS (2.5 s) ago, which guarantees the newest frame it asks for is readable. Even then, that moment can sit in a segment that rotated a moment ago and whose sidecar is not yet written, so the index has no row for it yet. Instead of refusing at once ("nothing recorded"), a live read waits up to four seconds, retrying every half second, for the moment to appear in the index. A refusal after that wait is a real refusal; one before it would have been a race.

Code

The index lookup and the exact-frame reader (trimmed excerpts, backend/store.py and backend/frames.py)·python

def segment_at(self, input_id: str, at: float):
    """The closed segment that holds wall time ``at``: one index lookup, no decode."""
    return self._one(
        "SELECT * FROM segments WHERE input_id = ? AND start_at <= ? AND start_at + duration > ? "
        "ORDER BY start_at DESC LIMIT 1",
        (input_id, at, at),
    )


# Keyframes only, dropped at the demuxer before the decoder ever sees the rest.
KEYFRAMES_ONLY = ("-discard", "nokey", "-skip_frame", "nokey")


def extract(ffmpeg, path, offset, size=None):
    """The frame at ``offset`` seconds into one segment, at full resolution."""
    seek = [ffmpeg, "-hide_banner", "-loglevel", "error", "-ss", f"{max(0.0, offset):.4f}",
            "-i", str(path), "-frames:v", "1"]
    if size and size[0] > 0 and size[1] > 0:
        data = _run([*seek, "-f", "rawvideo", "-pix_fmt", "rgb24", "-"])
        if len(data) == size[0] * size[1] * 3:
            return Image.frombytes("RGB", (int(size[0]), int(size[1])), data)
    data = _run([*seek, "-f", "image2pipe", "-c:v", "png", "-"])   # the slow way
    image = Image.open(io.BytesIO(data))
    image.load()
    return image.convert("RGB")

Feel the demuxer difference on a recording of your own (use a segment from the previous lesson's exercise)·bash

f=seg-001.mp4

# Decoder skips non-key frames, but every packet is still read and parsed
time ffmpeg -hide_banner -loglevel error -skip_frame nokey -i "$f" \
  -vf scale=160:-2 -f null -

# Demuxer drops non-key packets before the decoder ever sees them
time ffmpeg -hide_banner -loglevel error -discard nokey -skip_frame nokey -i "$f" \
  -vf scale=160:-2 -f null -

# An exact frame: seek before -i, one frame, raw bytes out
time ffmpeg -hide_banner -loglevel error -ss 37.25 -i "$f" -frames:v 1 \
  -f rawvideo -pix_fmt rgb24 - | wc -c     # 1920*1080*3 = 6220800 bytes

Let the output decide a frame read, not the exit status (runnable; needs ffmpeg)·python

import subprocess


def read_frame(path, offset):
    done = subprocess.run(
        ["ffmpeg", "-hide_banner", "-loglevel", "error", "-skip_frame", "nokey",
         "-noaccurate_seek", "-ss", f"{offset:.3f}", "-i", path, "-frames:v", "1",
         "-f", "image2pipe", "-c:v", "mjpeg", "-"],
        capture_output=True, timeout=30)
    if not done.stdout:
        # no bytes: past the last keyframe. ffmpeg 8 exited 0 here, ffmpeg 9 exits
        # with an error status, so the output decides, never the status alone
        raise RuntimeError(f"no frame at {offset} s (status {done.returncode}, past the last keyframe?)")
    if done.returncode != 0:
        raise RuntimeError(f"ffmpeg failed: {done.stderr[-200:]!r}")
    return done.stdout


print(len(read_frame("seg-001.mp4", 10.0)), "bytes")
try:
    read_frame("seg-001.mp4", 59.99)
except RuntimeError as exc:
    print("refused:", exc)

Exercise

Using the segments and the SQLite index from the previous exercise, write frame_at(t): one segment_at-style query, then one ffmpeg exact-frame read at t - start_at, returning raw RGB bytes and checking their length. Time it for t = 5, 95 and 175. Then add a keyframe-only variant and find the largest offset in a segment for which it still returns a frame.

Hint

The exact read's time should not depend on t at all. For the keyframe-only read, the last keyframe in a 60 s, 60 fps segment with GOP 15 starts at 59.75 s; ask for a moment well after it (59.9 s, say) and you get no bytes (exit status 0 on ffmpeg 8, an error status on ffmpeg 9). A few hundredths of a second after it can still return the 59.75 s keyframe, so find the boundary by measuring, not by assuming.