Exit Codes Answer a Narrower Question Than You Asked

Success Reported, Garbage Produced

An encode ran. It printed no errors and exited zero. The resulting clip was roughly twenty-five times longer than it should have been.

Nothing malfunctioned. The command was asked to apply a filter with certain parameters, and it did exactly that; the parameters produced a duration nobody wanted. The exit code answered the question did I complete the operation you specified, which is a much narrower question than is the output correct, and confusing those two is the failure mode this whole discipline exists to prevent.

Measure the Artifact, Not the Process

The rule that follows is simple to state and slightly tedious to obey: after every operation that produces a file, measure the file. Probe its duration rather than computing what the duration should have been. Analyze its loudness rather than trusting the gain you applied. Enumerate its streams rather than assuming the encoder produced what you asked for.

The distinction is between verifying intent and verifying result. Checking that you passed the right arguments verifies intent, and it is what most error handling actually does. Opening the artifact and asking what it is verifies result, and it is the only one that catches a tool doing precisely what it was told to do with the wrong outcome.

Documentation Lies in the Same Direction

A related trap from the same domain: media tooling can report capabilities it does not have in the build you are running. A help listing may enumerate filters from a static table, so a filter appears supported while the actual binary lacks the library that implements it.

The workshop's response is to treat capability questions like output questions — do not read the documentation, run a real one-frame render and see whether it works. That is a small habit with a general shape: when a system can report about itself, its report is a claim, and a claim from the thing being tested is the weakest evidence available.

Assert on artifacts, not on the absence of errors. "No exception was raised" is compatible with an empty file, a truncated file, a file of the wrong format, and a file that is correct. Those four outcomes are indistinguishable to error handling and trivially distinguishable to a measurement — so any pipeline step whose output matters should end by looking at what it produced.

Code

The difference between checking intent and checking result·python

# INTENT ONLY - this is what most pipelines do
def encode(src: Path, dst: Path) -> Path:
    subprocess.run([FFMPEG, "-i", str(src), *PROFILE, str(dst)], check=True)
    return dst        # exit 0. we know nothing about dst.


# RESULT - open the artifact and ask what it actually is
def encode_checked(src: Path, dst: Path, expect_s: float) -> Path:
    subprocess.run([FFMPEG, "-i", str(src), *PROFILE, str(dst)], check=True)

    got = probe_duration(dst)                       # measured, not computed
    if abs(got - expect_s) > 0.05:
        raise BadEncode(f"{dst.name}: {got:.2f}s, expected {expect_s:.2f}s")

    streams = probe_streams(dst)                    # enumerated, not assumed
    if [s["codec_type"] for s in streams] != ["video", "audio"]:
        raise BadEncode(f"{dst.name}: unexpected stream layout {streams}")

    return dst

# The 25x-too-long clip passed the first version and fails the second
# on its first assertion - with the actual number in the message.

Do not ask a tool what it can do — make it do it·bash

# WEAK: a help listing may be a static table compiled in,
# not a reflection of what this build can actually run.
ffmpeg -h filter=somefilter        # "supported" proves little

# STRONG: render one frame through it and see.
ffmpeg -v error -f lavfi -i color=c=black:s=64x64:d=1 \
       -vf "somefilter=..." -frames:v 1 -f null - \
  && echo "actually works in THIS build"

# Same principle as measuring output: a system's report about
# itself is a claim. Exercise the capability instead.

Exercise

Pick a step in one of your pipelines that produces a file, a record, or a message and currently succeeds on the absence of an exception. Add one measurement of the artifact itself — a size, a count, a checksum, a round-trip read — and assert on it with the observed value in the failure message. Then deliberately break the input and confirm the assertion fires with a message you could act on.

Hint

Start with the step whose output is consumed furthest downstream, because that is where a silent defect costs most to trace. A wrong artifact produced at the start and detected at the end forces you to search the whole pipeline for where it went wrong.

Success Reported, Garbage Produced

Measure the Artifact, Not the Process

Documentation Lies in the Same Direction

Code

External links

Exercise

Progress

Comments 0