The Clock Under the Picture

The Card That Recorded at 30

The first time both capture cards recorded together, each was asked for 60 frames a second and each delivered exactly 30.000. Nothing was dropped, nothing errored. A probe read the device's frame duration back at each step and found the cause: 16.667 ms (60 fps) inside the capture session's configuration, still 16.667 ms after the configuration was committed, and 33.367 ms (29.97 fps) the moment startRunning() returned. AVFoundation puts a UVC device back on its default rate when the session starts.

The fix is three lines in the recorder: set the frame duration again on the running device. The recorder then reads the duration back and reports fps_applied in its status, so the engine and the page see the rate the device actually took, not the rate that was asked for. There is a quiet lesson inside the fix too. Only a duration the device itself reports may be set; an unsupported one raises an Objective-C exception that Swift cannot catch. One card's "60" turned out to be 1000000/60000240 of a second, not 1/60, so the recorder takes the exact value from the device's own ranges.

One Clock for Two Pictures

Oriel records two feeds at once and promises that a moment means the same instant in both. Each input maps its segment's first capture timestamp to wall time on the host's clock, and every frame after it is placed from its own capture timestamp. A five-minute recording of both feeds checked this: the segment joints were off by the same small amounts on both feeds (around a millisecond, and once about 23 ms, on both at once), so the two timelines move together and stay aligned to within a frame.

Note what that check is and is not. It is a timestamp check. A visual check needs one event both cards can see, like a clap in front of the camera shown on the game screen at the same moment. None was available, so the architecture document says plainly that the visual frame-level check was not run. Saying which check you did is part of the measurement.

Two Clocks That Disagree

The external recorder whose decks feed the cards keeps its own files, with its own clock, and the family's editing workflow sometimes needs to line Oriel's frames up with those files. Two measured facts shape how that is done:

60.000 is not 59.94. The capture card delivers exactly 60.000 frames a second while the external recorder runs at 59.94. Oriel's frames are therefore exactly 16.667 ms apart, and the card repeats a source frame about every 16.7 seconds. Mapping a frame to the other recording by its index drifts a frame every 16.7 s; mapping it by time does not.
The external recorder's clock drifts. Against the Mac's, it gains or loses about 2.2 ms a minute (34 to 38 parts per million, on both decks), and its timecode restarts in each clip. One offset carried across hours is wrong by seconds. So the bridge is made from the card's audio, which both recordings carry, by cross-correlation, and refitted along the way.

Sound on the Picture's Clock

Since a deck card's HDMI audio is now recorded inside the same segment files as its picture, the sound has to live on the picture's clock too. The probe stamps each audio buffer with the host time of its first sample. The recorder converts that host time to the capture session's synchronization clock with CMSyncConvertTime, the clock the video frames are stamped on, and writes AAC beside the HEVC.

Two small rules keep it clean. A buffer that starts within 2 ms of where the last one ended continues it exactly, because the card's clock and the host's differ by parts per million and each buffer is stamped on its own; a larger gap stays a gap, and an overlap is cut. And the picture never waits on the sound: a segment gets an audio track only if sound is flowing as it opens, sound that stops for a second finishes that segment's audio early, and sound that comes back opens the next segment at once.

Code

Set the frame rate again once the session runs, and report what the device took (trimmed excerpt, native/OrielRecorder/Sources/OrielRecorder/InputRecorder.swift)·swift

session.startRunning()
// startRunning puts a UVC device back on its default 29.97 fps
// (measured on both capture cards, 2026-11-02: set to 60 inside the
// configuration, read back 33.367 ms once running). The duration
// holds only when it is set again on the running device.
if let wanted = frameDuration {
    try device.lockForConfiguration()
    device.activeVideoMinFrameDuration = wanted
    device.activeVideoMaxFrameDuration = wanted
    device.unlockForConfiguration()
}
fpsApplied = 1 / device.activeVideoMinFrameDuration.seconds

Audio buffers move onto the capture session's clock, and near-contiguous ones join exactly (trimmed excerpts, InputRecorder.swift)·swift

let clock = session.synchronizationClock ?? CMClockGetHostTimeClock()
let host = CMTime(value: chunk.hostNanos, timescale: 1_000_000_000)
let onClock = CMSyncConvertTime(host, from: CMClockGetHostTimeClock(), to: clock)

// later, appending a piece to the open segment:
let gap = (piece.pts - audioCursor).seconds
let begins = abs(gap) < 0.002 ? audioCursor : piece.pts   // within 2 ms: continue exactly

Why index mapping drifts and one offset is not enough (runnable)·python

card_fps = 60.0
recorder_fps = 60000 / 1001          # 59.94...

# A source frame is repeated whenever the card has delivered one more frame
# than the source produced.
repeat_every_s = 1 / (card_fps - recorder_fps)
print(f"a repeated frame every {repeat_every_s:.1f} s")

# Clock drift between the external recorder and the Mac
ppm = 36
for hours in (0.5, 1, 3):
    drift_ms = ppm * 1e-6 * hours * 3600 * 1000
    frames = drift_ms / (1000 / card_fps)
    print(f"after {hours:>3} h: {drift_ms:6.0f} ms, about {frames:4.1f} frames apart")

Exercise

Run the Python block, then extend it: simulate two recordings of the same event, one at exactly 60 fps and one at 60000/1001 fps with a 36 ppm clock error, and write a function that maps a frame index in the first to the nearest frame index in the second by time. Compare its answer after one hour with naive index mapping.

Hint

Convert index to seconds on one side (index / 60), correct for drift (multiply by 1 + 36e-6), then convert back on the other side (seconds * 60000/1001) and round. After an hour, naive index mapping is off by about 216 frames (3.6 s); time mapping without the drift correction is off by about 8, and with it by less than one.