Batch Runs, and the Art of Being Interruptible

A Workshop Needs a Different Kind of Resilience

An always-on service gets its resilience from supervision. Something watches it, notices when it dies, and starts it again; the state it needs is in a database that survived the crash. A workshop has none of that, and it needs an answer, because a video render is a long job made of expensive steps — speech synthesis that costs money per character, image generation that costs money per image, encodes that cost minutes.

The answer is that every stage is cached and individually re-runnable. A run that dies at the encode step does not redo the synthesis. You re-invoke, the finished stages notice their outputs already exist, and the work resumes where it stopped. Rebuilding anyway is a deliberate act — a force flag — rather than something that happens because you ran the command twice.

Freshness Is a Question, Not an Assumption

The whole design lives in one small function: given an output path and a force flag, is what is on disk still good? Naive answers cause the two failures that actually happen in practice. Answer "yes, if the file exists" and you will happily serve a stale artifact from before your edit. Answer "no, always rebuild" and you have thrown away the property you were trying to buy.

The useful version keys the cached artifact to the content that produced it. If the scene definition changes, the path changes, and the old pixels are simply not where anyone looks — the cache cannot serve a stale answer because it no longer has that address. You will meet a memorable failure of exactly this later in the quest, when a cache with a hole in it served an image three hours old into a finished render.

Where You Cut the Stages Is a Design Decision

It is tempting to treat the stage list as an implementation detail — some arbitrary chopping-up of a long script. It isn't. Every stage boundary is doing two jobs at once, and both of them are about people rather than machines.

A boundary is a place work can resume, which is the mechanical job. But it is also a place a human can look, which is the important one. Speech synthesis is its own stage because a person needs to listen to the takes before anything gets built on top of them. Plates are their own stage because someone has to look at the pictures. If synthesis and assembly were one step, the first opportunity to hear a bad take would be after the encode — and the review that catches it would cost a full rebuild instead of a reroll.

So the rule for where to cut is not "wherever the code gets long." It is: cut where a person would want to stop and judge, and cut before anything expensive that depends on that judgment. Get that ordering right and review becomes cheap enough to actually happen. Get it wrong and you build a pipeline that is technically resumable and practically un-reviewable, because every inspection point sits downstream of the spend it was supposed to prevent.

Cache invalidation by naming. Rather than storing one path and deciding whether it is fresh, derive the path from a digest of the inputs. Then "is it stale?" stops being a judgment call: a stale artifact lives at an address nobody asks for, and the correct one is either present or absent.

Code

The freshness question, and the better answer beside it·python

def fresh(path: Path, force: bool) -> bool:
    """Naive: is there something already at this path?"""
    return path.exists() and not force


# The problem: edit the scene, and this still says True.
# The fix is not a smarter check - it is a different address.

def scene_digest(sc: dict) -> str:
    """Everything that can change the pixels goes into the key."""
    payload = json.dumps(sc, sort_keys=True, ensure_ascii=False)
    return hashlib.sha256(payload.encode()).hexdigest()[:12]


def scene_plate_path(ep: Path, sc: dict) -> Path:
    # content-keyed: editing a scene cannot serve last run's pixels,
    # because the edited scene simply has a different filename.
    return ep / "plates" / f"{sc['key']}-{scene_digest(sc)}.png"


def scene_clip_path(ep: Path, sc: dict, dur: float) -> Path:
    # keyed by the scene AND its length on the timeline - the same
    # picture held for 4s and for 9s are two different artifacts.
    return ep / "clips" / f"{sc['key']}-{scene_digest(sc)}-{dur:.2f}.mp4"

What being interruptible looks like from the outside·bash

# a full run: each stage caches its outputs
pipeline.py all --episode my-episode

# ...dies during encode. re-invoke; synthesis and plates are kept.
pipeline.py all --episode my-episode

# rebuild one stage on purpose
pipeline.py plates --episode my-episode --force

# one bad speech take: purge exactly that segment, keep the rest.
# without this, re-synthesizing identical text is a cache hit
# that cheerfully returns the same glitched audio forever.
pipeline.py reroll seg-014 --episode my-episode

Exercise

Take a multi-step script you own where a later step is expensive — a download, a model call, a build. Give each step a content-keyed output path derived from a digest of its inputs, and make the step skip when that path already exists. Then test the case that matters: change an input in the middle and confirm that exactly the affected steps rebuild, not everything and not nothing.

Hint

The bug to look for is an input that affects the output but is not in the digest — a flag, an environment variable, a template file read from disk. Anything that can change the result and is not part of the key is a stale artifact waiting to be served.

A Workshop Needs a Different Kind of Resilience

Freshness Is a Question, Not an Assumption

Where You Cut the Stages Is a Design Decision

Code

External links

Exercise

Progress

Comments 0