One Key per Attempt

An answer that never comes back

The worker posts a clip to Ember and the connection drops. Did Ember get it? From the worker's side there is no way to know. A request can be lost on the way out, or it can arrive, start a paid render, and lose only its reply on the way back. This is the Two Generals' Problem in miniature, and no amount of waiting settles it.

For images, Avatars answers carefully and slowly: the task becomes uncertain, and reconciliation searches Ember's history for the task id before anyone may retry. The Plan Before You Spend track walks through that. Clips get a better answer, because Ember's Pro Video accepts an idempotency key chosen by the caller. Ember's journal stores request_id in a unique column, and its create() looks the key up before it plans or spends anything. The same key returns the clip it already made.

Naming the attempt

clip_request_id() is one line of logic. The key is avatars-, then the task id, then the attempt number, which is the length of the task's attempt log plus one. Nothing about it is random, and nothing is stored apart from the task. Anyone holding the task record can compute the key again, and that is exactly what a worker that just restarted needs.

The key changes only when the attempt log grows, and it grows in two places. When Ember accepts a submission, the worker appends the accepted clip id. When Dad retries a task, retry() appends the attempt that just ended, with its state and error. So a first submission Ember refused goes out as avatars-<task>-1, and the retry after it as -2. A clip Ember accepted and the vendor then failed has logged both, so its retry goes out as -3. The number is not a strict count of tries. It only has to be new on the occasions a person has decided to pay again, and the same on every other occasion.

Inputs first, then one request

Before it builds anything, run() asks Pippa again whether the acting soul may depict the task's subjects, because a subject can become protected between review and submission. Then each reference goes up to Ember's video input store, POST /api-video/inputs, and comes back as an input id with its role. The body carries the key, the resolved generator, the prompt, the settled settings, those inputs, confirm: true for the consent already given at review, and source_metadata naming the batch, the task, the attempt and the depicted souls. The worker writes submitting with a timestamp, and only then posts.

Three answers, and one of them is again

What came back	An image task	A clip task
A 4xx refusal	`failed`, Submission refused	`failed`, Submission refused, nothing charged
A transport error or a 5xx	`uncertain`, reconcile before retry	back to `queued`, Ember unreachable; will retry, a 10-second pause, the same key
A 202	`running` with the job id	`running` with the clip id, and the attempt logged
The worker died while submitting	`uncertain` on restart	`queued` on restart

The middle rows are the whole point. With a key, an unknown outcome stops being dangerous. Posting the same key again either creates the clip that never arrived or returns the one that did, and in both cases exactly one clip is paid for. The references are uploaded a second time on the next pass, which costs disk space in Ember and no credits.

A 4xx is different on purpose. A refusal is Ember or the vendor saying no before anything is charged, such as a Veo model that needs the 8-second length when reference images are given. Resending a refused request would only be refused again, so the task stops as failed with the vendor's own words, and a new attempt is a person's decision.

The restart sweep

When a worker starts, it first takes a file lock so that only one worker owns the database. Then it sweeps tasks that were in flight when the last one died. A task caught in preparing returns to the queue. Usually it had sent nothing yet. A clip is the exception: one whose earlier submit Ember accepted before the answer was lost went back to queued under the same key, and its next pass moved it into preparing again. Requeueing it is still safe, because its attempt log has not grown, so the next post carries the same key and Ember returns the clip it already made. A task caught in submitting splits by kind. An image goes to uncertain, because the previous submit may have reached Ember and there is no key to ask by. A clip goes straight back to queued, with the comment that its request id makes the submit idempotent in Ember.

The test that proves it

test_a_lost_answer_or_a_restart_never_pays_twice walks the dangerous paths in order:

Ember answers 503. The task is back in the queue, its attempt log is empty, and Ember has created nothing.
The test forces the task into submitting and starts a fresh worker. The clip returns to the queue, goes out, and finishes. The list of request ids Ember received is exactly one, avatars-<task>-1, and Ember holds one clip.
A second request is refused with a 422 about the 8-second length. The task is failed, carrying those words.
An explicit retry goes out, and the request Ember sees ends in -2.

Principle: The key belongs to the decision to pay. One decision keeps one key, however many times the wire drops the call.

Code

cwkAvatars, backend/worker.py: one idempotency key per attempt (verbatim, lines 16-23)·python

def is_clip(task):
    return task.get("media") == "clip"


def clip_request_id(task):
    """One idempotency key per attempt: re-posting it after a crash returns
    the clip Ember already made instead of paying for another."""
    return f"avatars-{task['id']}-{len(task['attempts']) + 1}"

cwkAvatars, backend/worker.py: the submit, and why unreachable means queued (verbatim, lines 158-174)·python

            settings = task["clip"]
            body = {"request_id": clip_request_id(task), "generator": task["provider"], "prompt": task["prompt"], "aspect_ratio": settings["aspect_ratio"], "resolution": settings["resolution"], "duration_secs": settings["duration_secs"], "audio": settings["audio"], "negative_prompt": settings.get("negative_prompt"), "inputs": inputs,
                    # Consent was settled when the batch was accepted (studio.generate).
                    "confirm": True,
                    "source_metadata": {**batch["request"].get("source_metadata", {}), "source_app": "cwkAvatars", "batch_id": batch["id"], "task_id": id, "attempt": len(task["attempts"]) + 1, "soul_ids": task["subjects"], "ordinal": task["ordinal"]}}
            task = await io("update", id, expected=task["version"], state="submitting", phase="Submitting to Ember", submitted_at=now())
            try:
                accepted = await upstream.video_create(body)
            except StudioError as exc:
                if 400 <= exc.status < 500:
                    await io("update", id, state="failed", error=str(exc), phase="Submission refused")
                else:
                    # The same request id is safe to send again: leave it for the next pass.
                    await io("update", id, state="queued", phase="Ember unreachable; will retry", error=str(exc))
                    await asyncio.sleep(10)
                return
            task = await io("update", id, state="running", phase=accepted.get("phase") or "Generating", job_id=accepted["id"], error=None, attempts=[*task["attempts"], {"job_id": accepted["id"], "submitted_at": task["submitted_at"]}])

cwkAvatars, backend/worker.py: on restart, an image is uncertain and a clip is queued (verbatim, lines 43-50)·python

        for task in await self.studio.io("list", "task", states=("preparing",)):
            await self.studio.io("update", task["id"], state="queued", phase="Queued")
        for task in await self.studio.io("list", "task", states=("submitting",)):
            if is_clip(task):
                # The clip's request id makes the submit idempotent in Ember.
                await self.studio.io("update", task["id"], state="queued", phase="Queued")
                continue
            await self.studio.io("update", task["id"], state="uncertain", phase="Reconcile before retry", error="The previous submit may have reached Ember. Check history before submitting again.")

Exercise

Find a call in your own software that creates something paid on another service and is retried after errors. Derive an idempotency key from your own durable record (the task id plus an attempt number), never from a random value made at send time. Send it with every post. On a transport error or a 5xx, leave the task queued with the same key. On a 4xx, fail it. Bump the attempt number only when a person or a rule explicitly decides to try again. Then kill your worker between writing submitting and receiving the reply, restart it, and confirm the service saw one key and made one thing.

Hint

If the service has no idempotency key, you cannot use this pattern; fall back to the uncertain state and reconcile by an id you put in the request's metadata. A fake service that stores keys in a dictionary and returns the stored result on a repeat is enough to test the whole flow without spending anything.

An answer that never comes back

Naming the attempt

Inputs first, then one request

Three answers, and one of them is again

The restart sweep

The test that proves it

Code

External links

Exercise

Progress

Comments 0