Not Complete Until Accepted

"A computation is not complete until the control plane durably accepts it."

The Most Important Sentence in the System

Everything in Recall's distributed design collapses to one rule. The executor can finish computing a transcript — the function returned, the file is on disk, the log says success — and none of that means the job is complete. Completion is a property of the source of truth, not of the machine that did the work. The job is done when, and only when, the control plane has durably recorded that it accepted the result.

Why be so strict? Because between 'the executor finished' and 'the control plane knows' there is a network, and networks fail. The executor might compute a perfect result and then lose its connection before reporting it. If you called that 'done,' the truth and the work would silently disagree — the executor thinks it's finished, the database thinks the job is still pending. That gap is where the worst bugs live.

Finishing Is Cheap. Acceptance Is Durable.

So the executor's local sense of 'done' is treated as a proposal, never a fact. It computes a result, then submits it to the control plane, which validates it and commits a state transition in one atomic database write. Only that commit is completion. A returned function is a rumor; a committed transition is truth.

This is what makes the whole thing survivable. Kill the executor after it computed but before it reported — the control plane never saw acceptance, the lease expires, the job returns to the queue, and it runs again. Kill it after the control plane committed — the job is genuinely done and won't be redone. The one moment that matters is the durable commit, and it lives on the host that owns the truth.

Replay Must Be Safe

There's a subtlety: what if the executor computed, the control plane committed, but the acknowledgment back to the executor got lost? The executor doesn't know it succeeded, so it retries — and now the same result arrives twice. The system has to make that harmless. Recall does it with identity: a resubmission of an already-accepted result returns the existing record instead of creating a duplicate. Retrying is always safe because acceptance is idempotent. On top of that, when the control plane restarts, it reconciles any job left mid-flight by a crash. 'Accept once, replay safely, heal on startup' is how two hosts stay honest across an unreliable network.

Code

Returned is a rumor; committed is truth·python

# WRONG: the executor decides it's done
result = transcribe(chunk)
return result            # the function returned... so what?
                         # a crash here and the truth never learns

# RIGHT: completion lives on the control plane
result = transcribe(chunk)
ack = control_plane.submit_result(job_id, lease_token, result)
# control_plane commits ONE atomic transition, then acknowledges.
# If this call is lost, the executor retries; resubmitting an
# already-accepted result returns the existing row (idempotent).
# The job is complete only because the DB says so.

Exercise

Find a place in your own code where a background task marks itself done — sets a flag, returns success, writes a file. Now imagine the process is killed one line after the work finishes but before the 'done' is durably recorded somewhere authoritative. What happens on restart? Is the work redone, silently lost, or duplicated? Rewrite the completion so it only counts when an owner-of-truth durably accepts it, and so a retry is safe.

Hint

Ask two questions: (1) where does 'done' actually get recorded durably, and does the worker own that record or merely propose it? (2) if the same result is submitted twice — because an acknowledgment was lost — does the second submission create a duplicate, or return the existing one? If duplicates are possible, add an identity key so acceptance is idempotent.