Skip to content
C.W.K.
Lesson 01 of 05 · published

Segments, Keyframes, Sidecars

~15 min · fragmented-mp4, gop, hevc, avassetwriter, sidecar

Level 0Closed Lens
0 XP0/45 lessons0/13 achievements
0/100 XP to next level100 XP to go0% complete

Record Everything, Then Read Almost None of It

Oriel's recording philosophy fits in one sentence from its house sign: record everything, delete nothing. The game card writes about 18 GB an hour and the camera card about 5.4. A long broadcast is tens of gigabytes; the disk cap Dad approved holds a little over a hundred hours of both feeds. Yet what Pippa usually wants from all that is one frame from three seconds ago, or nine frames that summarize the last twenty seconds.

So the file layout is designed backwards, from the read. Every choice below exists so that a single frame anywhere in a hundred hours can be reached by opening one small file, seeking once and decoding a handful of frames.

The Layout, Choice by Choice

ChoiceValueWhy
Containerfragmented MP4a crash loses at most the last second; the open segment can be read while it is still being written
Segment length60 sa read opens one modest file, never an hour-long one
Keyframe intervalevery 15th framefour keyframes a second at 60 fps, so a seek never lands far from a decodable frame
B-framesnoneframes decode in the order they are stored, which keeps a seek-and-decode short and predictable
CodecHEVC through VideoToolboxthe Mac's hardware encoder; the bitrate per role was set by measurement

The keyframe interval was Dad's suggestion, in one short line: "15프레임에 한 번씩", once every fifteen frames. It is a trade. More keyframes make files larger for the same quality, because a keyframe is a whole picture rather than a difference. Fewer keyframes make every random read slower, because a decoder must start at the last keyframe and work forward. At fifteen, the worst case is decoding fourteen frames to reach the one you want, and the architecture document says exactly that: a read seeks to the keyframe before the moment and decodes at most 14 frames.

Fragmented MP4, Briefly

A classic MP4 keeps its index (the moov box, which says where every sample is) in one place, usually written at the end when the file is finished. Kill the writer before that and the file may be unreadable. A fragmented MP4 writes a small header up front and then a series of self-contained fragments, each with its own little index. The recorder asks AVAssetWriter for a fragment every second, so a crash costs at most that last second, and a reader can open a segment that is still being written and read every fragment already on disk. That second property is what lets a "look now" read a moment from a few seconds ago without waiting for the minute to close.

The Sidecar: Facts the Index Needs

Each closed segment gets a JSON sidecar beside it: the session, the input, the wall-clock time of its first frame, its duration, frame count, measured fps, size, GOP and byte count (and, for a deck card, its audio's details). The engine indexes sidecars into one SQLite table, segments, with an index on (input_id, start_at). That table is the map of everything Oriel has ever recorded, and it is small: one row per minute per input.

Why a sidecar instead of having the engine probe each MP4? Because the recorder already knows these facts at the moment it closes the file, and probing a video file costs a process launch and a parse. Writing the facts once, next to the file, means the engine never has to open a recording just to learn what is in it. And when a segment closes without its sidecar (a recorder crash, a full disk), the recorder writes one later from the file itself, marked recovered, so no minute is ever invisible to the index.

Code

The writer behind every segment: HEVC, a fragment a second, a capped keyframe interval, no reordering (trimmed excerpt, native/OrielRecorder/Sources/OrielRecorder/InputRecorder.swift)·swift
writer = try AVAssetWriter(outputURL: url, fileType: .mp4)
writer.movieFragmentInterval = CMTime(seconds: 1, preferredTimescale: 600)
writer.shouldOptimizeForNetworkUse = false
let compression: [String: Any] = [
    AVVideoAverageBitRateKey: spec.bitrate,
    AVVideoMaxKeyFrameIntervalKey: spec.gop,          // 15: four keyframes a second at 60 fps
    AVVideoAllowFrameReorderingKey: false,            // no B-frames
    AVVideoExpectedSourceFrameRateKey: spec.fps,
    AVVideoProfileLevelKey: "HEVC_Main_AutoLevel",
]
input = AVAssetWriterInput(mediaType: .video, outputSettings: [
    AVVideoCodecKey: AVVideoCodecType.hevc,
    AVVideoWidthKey: width,
    AVVideoHeightKey: height,
    AVVideoCompressionPropertiesKey: compression,
])
input.expectsMediaDataInRealTime = true
The map of everything recorded: one row per segment (trimmed excerpt, backend/store.py schema)·text
CREATE TABLE IF NOT EXISTS segments (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    session_id TEXT NOT NULL,
    input_id TEXT NOT NULL,
    path TEXT NOT NULL UNIQUE,
    start_at REAL NOT NULL,      -- wall-clock time of the first frame
    duration REAL NOT NULL,
    frames INTEGER NOT NULL,
    fps REAL NOT NULL,
    width INTEGER NOT NULL,
    height INTEGER NOT NULL,
    gop INTEGER NOT NULL,
    bytes INTEGER NOT NULL,
    codec TEXT NOT NULL
);
CREATE INDEX IF NOT EXISTS segments_by_time ON segments (input_id, start_at);
Make the same layout with ffmpeg and check the keyframes (runs anywhere ffmpeg has an HEVC encoder)·bash
mkdir -p segs && cd segs
# 3 minutes of test pattern at 60 fps: HEVC, keyframe every 15 frames, no B-frames,
# fragmented MP4, cut into 60 s segments.
ffmpeg -hide_banner -f lavfi -i testsrc2=size=1920x1080:rate=60 -t 180 \
  -c:v libx265 -x265-params keyint=15:min-keyint=15:bframes=0 -b:v 12M \
  -f segment -segment_time 60 -reset_timestamps 1 \
  -segment_format_options movflags=+frag_keyframe+empty_moov+default_base_moof \
  seg-%03d.mp4

# Every 15th frame should be a keyframe: list the first few key frame times.
ffprobe -v error -select_streams v -skip_frame nokey -show_entries frame=pts_time \
  -of csv=p=0 seg-000.mp4 | head -n 8
# Expect 0.000000, 0.250000, 0.500000, ... (four a second)

External links

Exercise

Run the ffmpeg command from this lesson, then write a tiny sidecar for each segment yourself: a Python script that uses ffprobe once per file to get duration, frame count and size, writes seg-000.json beside seg-000.mp4, and loads all sidecars into a SQLite table with an index on start time. Finally, query which segment holds second 125.
Hint
Second 125 is in the third segment (120 to 180 s), at offset 5.0. The query is the same shape as Oriel's segment_at: start_at <= t AND start_at + duration > t, newest first, one row.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.