The Killable Child

"You can't time out a thread that's stuck in the kernel. But you can kill a process."

The Hang You Can't Interrupt

Recall's executor reads video from a network-mounted archive. And here is a fact that surprises people who've only dealt with well-behaved local disks: opening a file on a network filesystem can block indefinitely. Not slowly — indefinitely. The mount goes unresponsive, and the thread that called open() is parked inside the kernel, in a state your Python code cannot reach. There's no timeout parameter that saves you. You can't interrupt it. You can't KeyboardInterrupt it. The thread is simply gone until the filesystem decides to answer, which may be never.

Now trace what that does to a worker. The main process hangs on that open. It stops sending heartbeats. Its lease expires. The control plane, correctly, decides the worker is dead and hands the job to another worker — which reaches for the same stuck mount and hangs identically. One unresponsive network path has just taken your entire queue hostage, and no amount of clever retry logic inside the hung process can help, because the process itself is frozen.

If You Can't Interrupt It, Isolate It Where You Can Kill It

The fix is structural, and it's the lesson of this lesson: run the risky operation in a separate child process. The parent spawns a child to do the audio proxy preparation — the part that touches the network filesystem — and then simply watches it. The parent stays fully responsive. It keeps renewing the lease. It enforces a wall-clock timeout that it controls, generously sized (a floor of about 30 minutes, scaled up by the source's size, capped at a few hours) so a genuinely slow-but-working transfer isn't murdered. And if the child exceeds that budget, the parent kills it. A process can always be killed, even when a thread inside it cannot be interrupted. The process boundary is the timeout you couldn't get from the syscall.

What makes this land cleanly is where it sits: entirely before the paid boundary. A timeout here is recorded as a pre-provider failure — it consumed no transcription credits, so it's in the category of failures that are provably safe to retry (Track 4's safe-only retry). The daemon then moves on to the next durable job instead of stalling. An indefinite hang becomes a bounded, free, recoverable failure.

And Don't Let the Slow Thing Block the Ready Work

There's a companion rule that follows the same instinct. The executor also periodically scans the archive for new videos — and that scan touches the same potentially-slow filesystem. So the daemon drains its existing durable job queue first and only then attempts inventory. A degraded network mount can make discovery slow, but it must never hold hostage the work that's already registered and ready to run. Same principle in a different costume: don't let an operation whose latency you don't control sit on the critical path of work that could otherwise proceed.

Code

The parent watches; the child takes the risk·python

import multiprocessing as mp

# The risky part (network-fs open can block in the kernel, forever)
# runs in a child we can KILL — not a thread we could only beg.
child = mp.Process(target=prepare_audio_proxy, args=(video_path,))
child.start()

deadline = timeout_for(source_size)   # ~30 min floor, scales, capped
while child.is_alive():
    control_plane.heartbeat(job.id, worker_id, token)  # parent stays alive
    if past(deadline):
        child.kill()                  # a process can ALWAYS be killed
        raise PreProviderTimeout      # no credits spent -> safe to retry
    sleep_briefly()

# Key: this whole risk sits BEFORE the paid boundary.
# A timeout here is free, bounded, and safely retryable.

Exercise

Find an operation in your own code that touches something you don't control the latency of — a network mount, an external drive, a remote filesystem, a C library call. Ask: if it blocks forever, what actually stops it? If your answer is a thread timeout or a signal, verify it would really work for a call stuck in the kernel. Redesign the risky part into a child process with a parent-enforced deadline, and note what stays responsive because of it.

Hint

The test: could your timeout stop the operation, or would it just stop *waiting* for it while the work stays hung? Those are very different — the second leaks a frozen thread every attempt. If you can't interrupt the call itself, the only reliable deadline is a process you can kill. Check also that the risky step sits before any irreversible or paid boundary, so a timeout costs nothing.