Unknown Is Not Failed

Three possible answers

Most code treats a remote call as a yes or a no. For a call that may spend money, that is one answer short. After the worker posts a task to Ember, exactly one of three things is true:

Ember said no. A 4xx is an explicit rejection: Ember read the request and refused it, so nothing was generated. The task becomes failed with the phase Submission refused and Ember's own sentence as the error.
Ember said yes. A 202 with a job id. The task becomes running and the id is recorded.
Nobody knows. The connection dropped, the call timed out, Ember answered with a 5xx, or the answer could not be read. The comment beside that line in the worker: transport errors and 5xx cannot prove whether a metered request was accepted. The task becomes uncertain, phase Reconcile before retry.

The plumbing makes that split easy to get right. Avatars' upstream client turns every transport failure and every unreadable body into a 503, and turns any 5xx Ember sends into a 503 as well, while a 4xx keeps its own status and carries Ember's message. So the worker needs one comparison: 400 up to 499 is a refusal, everything else it cannot vouch for.

Where unknowns come from

Submission is only the first place an outcome can go missing. The worker files every other doubt the same way:

What happened	Resulting state
The engine restarted while a task was `submitting`	`uncertain`
Polling hit a transport error, a non-200 status, or the 1800-second bound	`uncertain`
Ember reported the job failed	`failed`
Ember reported the job cancelled	`cancelled`
Ember answered 404 for a job it had accepted	reconciled at once (below)
Fetching the original from Ember failed, after Ember's answer was saved	`uncertain`
Anything else, after Ember's answer was saved (writing the asset, say)	`copy_failed`
Anything else, after a job id existed or mid-submit	`uncertain`
Anything else, before anything was sent	`failed`

The download row is easy to misread. It looks like a copy problem, but the kit's client raises a failed image fetch as an EmberJobError, and the worker files every such error that is not Ember's own report of a failed or cancelled job as uncertain. Ember's answer is still saved on the task, so a reconcile finds the same history row and runs the copy again. Nothing is generated.

The 404 case is worth a sentence. Ember keeps live job records in memory, so after Ember restarts, a job it accepted an hour ago simply is not there. The Sibling Kit's client reads that 404 as the job being gone, never as still running. The worker does not wait for a person: it marks the task uncertain, phase Reconciling with Ember history, and reconciles straight away, because Ember's archive outlives its memory.

Reconcile by identity

Every job Avatars submits carries its own task id inside source_metadata, and Ember stores that metadata with the history row of each image it makes. Reconciling is a search for that id. reconcile() pages through Ember's Pro history a hundred rows at a time and collects every row whose metadata names this task. Then:

One match. That row is the result. The worker stores it as Ember's answer and runs the normal finish: copy the original, release protection, mark done. No generation is repeated.
Several matches. It stops, records the match ids and says Multiple archived outputs match this task. Review them before retrying.
No match. The task stays uncertain with a recorded reconciliation (when it checked, and an empty match list) and the sentence No Pro history match found. The upstream request may still be running; an explicit retry can create another billed image.

Zero matches is information, and it is still not proof. A job that is generating right now has no history row yet. That is why the empty result is recorded with its time and the sentence keeps both possibilities open. A copy_failed task with Ember's answer already saved needs no search at all; reconcile sends it back to copying.

The batch view offers the same thing in bulk. Reconcile N with Ember runs it over every task that needs it and reports how many were recovered from history, how many still need a decision, and how many could not be checked. It never retries anything.

Retry is a decision

retry() accepts a failed or cancelled task without a reconciliation: Ember has reported the job's end, so there is nothing left to reconcile. That is not proof that nothing was spent. Ember's comment beside its vendor calls says a call already sent cannot be aborted: cancelling only stops Ember from waiting, the call runs to completion in the background, may still be billed, and its result is discarded. So retrying a cancelled task can pay a second time for work the vendor already finished. An uncertain task needs three things together: a recorded reconciliation, zero matches in it, and the caller's explicit acknowledge_uncertain. Without all three the answer is a 409, Reconcile history and explicitly acknowledge the uncertain outcome before retrying. In the batch view that acknowledgement is a separate button, Retry despite uncertain outcome, behind a confirmation that says the prior request may still finish and be billed. A history_missing or copy_failed task is never retried as a new generation; its image already exists.

A retry keeps the past. The old state, error and reconciliation are appended to attempts, the job id is cleared, and the task goes back to queued. Subject policy is checked again first, because a subject may have become protected since.

Cancel tells the truth too

Before submission, cancel is local and certain. While submitting or uncertain, it is refused, because nobody knows what there is to cancel: Reconcile this task first; its upstream outcome is not known. While running, it asks Ember to cancel and answers honestly: Cancellation requested. A metered request already sent may still be billed by the provider.

Principle: When you cannot tell whether money was spent, the safe default is to find out. Guessing in either direction loses something: retry and you may pay twice, mark it failed and you may throw away an image you paid for.

Code

cwkAvatars, backend/upstream.py: transport and 5xx become 503, a 4xx keeps its status and Ember's sentence (verbatim, lines 20-45)·python

    async def json(self, origin, path, *, method="GET", body=None, params=None, headers=None, timeout=30):
        try:
            # Build independently of the client's cookie jar/default headers.
            # Connections are shared; each visiting device's credentials are not.
            request = httpx.Request(
                method, origin.rstrip("/") + "/" + path.lstrip("/"),
                json=body, params=params, headers=headers,
                extensions={"timeout": httpx.Timeout(timeout).as_dict()},
            )
            result = await self.client.send(request)
            self.refuse(result)
            return result.json()
        except (httpx.HTTPError, ValueError) as exc:
            raise StudioError(f"The upstream service is unavailable: {origin}", 503) from exc

    @staticmethod
    def refuse(result):
        if result.is_success:
            return
        try:
            message = result.json().get("detail", result.text)
        except ValueError:
            message = result.text
        if isinstance(message, dict) and message.get("message"):
            message = message["message"]  # Ember's structured errors carry the sentence here
        raise StudioError(str(message)[:1500], result.status_code if result.status_code < 500 else 503)

cwkAvatars, backend/worker.py: reconcile by task id in Ember's history (verbatim, lines 256-282)·python

    async def reconcile(self, id):
        task = await self.studio.io("get", id, "task")
        if task["state"] not in RECONCILABLE:
            raise StudioError("This task does not need reconciliation.")
        if task.get("upstream_result") and task["state"] == "copy_failed":
            return await self.studio.io("update", id, state="copying", error=None)
        upstream = self.studio.upstream
        if is_clip(task):
            return await self.reconcile_clip(task)
        matches, offset = [], 0
        while True:
            page = await upstream.json(upstream.settings.ember, "/api/v1/api-generators/history", params={"limit": 100, "offset": offset})
            for record in page["items"]:
                if (record.get("source_metadata") or {}).get("task_id") == id:
                    matches.append(record)
            offset += len(page["items"])
            if not page["items"] or offset >= page.get("total", 0):
                break
        if len(matches) == 1:
            record = {"status": "done", "result": {"image": matches[0]["output_image"], "history": matches[0]}}
            await self.studio.io("update", id, upstream_result=record)
            await self.finish(id, record)
        elif matches:
            await self.studio.io("update", id, phase="Reconcile before retry", reconciliation={"checked_at": now(), "matches": [m["id"] for m in matches]}, error="Multiple archived outputs match this task. Review them before retrying.")
        else:
            await self.studio.io("update", id, phase="Reconcile before retry", reconciliation={"checked_at": now(), "matches": []}, error="No Pro history match found. The upstream request may still be running; an explicit retry can create another billed image.")
        return await self.studio.io("get", id, "task")

cwkAvatars, backend/worker.py: retry only what is known, or acknowledged (verbatim, lines 323-332)·python

    async def retry(self, id, acknowledge_uncertain=False):
        task = await self.studio.io("get", id, "task")
        if task["state"] == "uncertain":
            if not acknowledge_uncertain or not task.get("reconciliation") or task["reconciliation"].get("matches"):
                raise StudioError("Reconcile history and explicitly acknowledge the uncertain outcome before retrying.", 409)
        elif task["state"] not in {"failed", "cancelled"}:
            raise StudioError("This task cannot be retried as a new generation.", 409)
        await self.studio.subjects(task["subjects"], task.get("actor_soul_id"))
        attempts = [*task["attempts"], {"state": task["state"], "error": task.get("error"), "reconciliation": task.get("reconciliation"), "ended_at": now()}]
        return await self.studio.io("update", id, state="queued", phase="Queued for retry", job_id=None, upstream_result=None, error=None, attempts=attempts)

Exercise

Take one call in your own code that costs money or has a side effect you cannot undo, and that has no idempotency key. Add your own work-item id to the request's metadata. Classify every outcome into rejected, accepted or unknown, and store unknown as a state of its own. Write a reconcile function that searches the remote side's records for your id and handles zero, one and several matches differently. Finally, make retry refuse an unknown item unless a reconciliation with zero matches is recorded and the caller passes an explicit acknowledgement.

Hint

Simulate the unknown case by dropping the connection after the remote side has received the request: a proxy that closes the socket, or a server that sleeps past your timeout and then succeeds. If the remote side keeps no searchable record of your id, your reconcile can only ever say 'not found', and that limit is worth knowing before you need it.

Three possible answers

Where unknowns come from

Reconcile by identity

Retry is a decision

Cancel tells the truth too

Code

External links

Exercise

Progress

Comments 0