A Replay in One Pass

"Look at What Just Happened"

The most common thing Dad asks during a game is some version of "피파, 방금 거 봐": look at what just happened. A single frame from now is useless for that; the interesting moment (a hit, a death, a boss appearing) is somewhere in the last twenty seconds. A video clip is too expensive to put in a model's context. Oriel's answer is the replay action: the last N seconds (20 by default, 120 at most), summarized as up to nine frames, the ones with the most change, laid out in time order on one 3×3 contact sheet, each labeled with how many seconds before "now" it was. One picture, 1536×864, within a 2,100-token budget.

Which Nine Frames?

Choosing the nine is the interesting part, and pick_changes does it with a deliberately simple rule. Every keyframe in the window is turned small and grey. Each one is scored by how much it differs from the one before it (the mean of the pixel-wise difference). The frames with the highest scores, at least 0.75 seconds apart, are kept, because a hit, a death or a phase change is exactly a big difference between neighbours. Two details matter:

The last frame always rides along. "What just happened" ends at now, so the final keyframe is chosen first, whatever its score.
A quiet window still gets nine. If fewer than nine frames changed enough, the rest are filled from a regular stride through the window's frames, taken from the start until there are nine, so a calm window is never shown as one frame (the filled ones lean toward the window's beginning, which is a known imperfection).

The 13-Second Replay

The first version read the window twice. A cheap small grey pass over the keyframes chose the nine moments; then nine separate full-size reads fetched those frames as PNGs, each its own ffmpeg process on the open segment. It was correct, and on the 2026-11-07 broadcast a replay took 13.2 seconds on average. For a coach, thirteen seconds is the difference between "look at that" and "what were we talking about?".

The fix (Coop 1110-5) is the title of this lesson: one pass. keyframe_tiles reads every keyframe in the window once, already at tile size (512×288) and already as JPEG, in one ffmpeg process per segment, with non-key packets dropped at the demuxer. The change score is computed from those same tiles, decoded small and grey at a fraction of their size. The nine shown are picked from what was already read, so no second read follows. Measured on a closed segment: a 20-second window went from 6.03 s to 0.38 s, a 60-second window from 6.19 s to 0.69 s, with the same nine frames chosen and the tiles on average 1.6 levels (out of 255) apart.

Spans, Not Files

A replay window can cross a segment boundary, and the newest part of it lies in the segment still being written. The engine lists the window's segments in one index lookup, the open one last, and the replay reads each span inside its own segment. A stretch that cannot be read is not hidden: the answer says how many stretches could not be read, and if nothing at all could be, the refusal names the first stretch and its cause instead of the vague "nothing was recorded".

One edge of the one-pass reader is worth knowing because it surprises everyone once. When a window holds no keyframe at all (nothing written there yet), ffmpeg never opens its JPEG encoder and exits with an error code (234) instead of 0. The raw-video grey pass on the same window exits 0. So the tile reader treats "error, no times, no pictures, and the input itself opened fine" as no frames, and only a real failure to open the input as an error.

Code

Keep the frames that changed most, at least 0.75 s apart, always ending at now (trimmed excerpt, backend/frames.py)·python

def pick_changes(frames, count, min_gap=0.75):
    """The times of the ``count`` frames that changed most from the one before
    (hits, deaths, phase changes), at least ``min_gap`` apart, in time order.
    The last frame always rides along: "what just happened" ends there."""
    if not frames:
        return []
    if len(frames) <= count:
        return [t for t, _ in frames]
    scores = [(0.0, frames[0][0])]
    for (_, before), (at, after) in zip(frames, frames[1:]):
        diff = ImageStat.Stat(ImageChops.difference(before, after)).mean[0]
        scores.append((diff, at))
    chosen = [frames[-1][0]]
    for _, at in sorted(scores, key=lambda s: s[0], reverse=True):
        if len(chosen) >= count:
            break
        if all(abs(at - c) >= min_gap for c in chosen):
            chosen.append(at)
    ...  # a quiet window is filled from a regular stride, from the start, up to count
    return sorted(chosen)

A one-pass replay of your own: tiles once, pick, then a sheet (runnable; needs ffmpeg and Pillow)·python

import io
import re
import subprocess

from PIL import Image, ImageChops, ImageStat


def tiles(path, start, seconds, size=(512, 288)):
    w, h = size
    done = subprocess.run(
        ["ffmpeg", "-hide_banner", "-discard", "nokey", "-skip_frame", "nokey",
         "-ss", str(start), "-t", str(seconds), "-i", path,
         "-vf", f"scale={w}:{h}:force_original_aspect_ratio=decrease,showinfo",
         "-fps_mode", "passthrough", "-f", "image2pipe", "-c:v", "mjpeg", "-q:v", "2", "-"],
        capture_output=True, timeout=60)
    times = [float(t) for t in re.findall(r"pts_time:([0-9.]+)", done.stderr.decode())]
    jpegs = [b"\xff\xd8" + part for part in done.stdout.split(b"\xff\xd8")[1:]]
    return list(zip(times, jpegs))


def grey(jpeg):
    return Image.open(io.BytesIO(jpeg)).convert("L").resize((160, 90))


shots = tiles("seg-001.mp4", 10, 20)
scores = [0.0] + [ImageStat.Stat(ImageChops.difference(grey(a), grey(b))).mean[0]
                  for (_, a), (_, b) in zip(shots, shots[1:])]
last = len(shots) - 1
picked = [last]                                      # "what just happened" ends at now
for i in sorted(range(last), key=lambda i: scores[i], reverse=True):
    if len(picked) == 9:
        break
    if all(abs(shots[i][0] - shots[j][0]) >= 0.75 for j in picked):
        picked.append(i)
picked.sort()
sheet = Image.new("RGB", (1536, 864))
for n, i in enumerate(picked):
    sheet.paste(Image.open(io.BytesIO(shots[i][1])), ((n % 3) * 512, (n // 3) * 288))
sheet.save("replay.jpg")
print(len(shots), "keyframes read once,", len(picked), "shown")

Exercise

Run the one-pass replay on a recording with some action in it (a screen recording of you scrolling and clicking works). Time it. Then write the two-pass version (grey scan, then nine separate exact-frame reads) and time that too. Finally, check that both choose the same nine moments.

Hint

The two-pass version spends most of its time in the nine extra ffmpeg launches and full-size decodes. If the two versions choose different moments, compare their grey images: different scaling or decoding sizes change the difference scores slightly, which is why Oriel decodes the score from the very tiles it shows.