Release Only After the Copy Is Durable

Two archives and a handoff

Every image Avatars asks for is born in Ember. Ember saves the original and a Pro history row first, and the architecture calls Ember first-level generated storage. Avatars then keeps a second durable copy, so that Ember's history cleanup can never destroy an avatar Dad chose. Two archives holding the same bytes raise one question: when may the first one let go?

Ember answers it with a protection flag. Avatars marks every job purpose: "exploration" with source_app: "cwkAvatars", and Ember's documentation for exploration consumers is short: new exploration results are protected until the consumer copies the actual bytes durably and then acknowledges with PATCH /api/v1/explorations/{id}/protection and the body {"protected": false, "durable_copy": true}. Submission metadata that promises a copy does not release anything. Ember's per-app expiry is off until someone enables it for that app, and its retention pass and manual cleanup only ever select unprotected images of the named app.

So the rule on Avatars' side is about order. The comment above the call says it: protection is released only after both the native bytes and the metadata are durable in Avatars.

The order inside finish()

finish() is where a done job becomes an Avatars asset. Step by step:

Look for an asset this task already made. If one exists, the bytes are already safe; at most the Ember history id is filled in. A second call never fetches or saves again.
Otherwise fetch the bytes from Ember's image file route through the kit's client.
Save the original. save_asset opens the bytes as an image to prove they are one, hashes them with SHA-256, and writes them under their own hash: to a temporary file opened for exclusive creation, flushed, fsynced, and moved into place with an atomic os.replace. If a file with that name already exists, it must still match its hash. Then the asset row goes into SQLite with its subjects, references, task id, batch id, provider, model and prompt.
Only now, release. The kit's release sends the PATCH and returns True only on a 200.
Record the outcome. The task becomes done, or history_missing when Ember never gave a history id, and the result of the release is kept on the task as protection_released.

The Record Keeper track covers why originals are content-addressed and written once. Here the point is the position of step 4.

A failed ack is the safe failure

Look at what each crash leaves behind. If the engine dies before step 3 finishes, Ember still holds a protected image and Avatars has nothing; the task is copying with Ember's answer saved, so the next start copies it again without generating anything. If it dies between 3 and 4, both sides hold the image and Ember's copy stays protected; the task is still copying, so the next start runs finish() again, finds the saved asset and asks for the release then. If the PATCH itself fails, release catches the transport error and returns False; it never raises, so a flaky network cannot turn a saved image into a failed task. Read that False as unconfirmed, not as still protected. A PATCH that Ember applied before the connection dropped also comes back False, with protection already gone. The kit's docstring and the comment above the call both say a failed acknowledgement leaves the image protected; that is the usual case, not a guarantee. It costs nothing here, because by step 4 Avatars already holds the durable copy.

Now reverse the order and the same failures look very different. Release first, crash before the copy, and the only original is now eligible for Ember's cleanup with nothing in Avatars. Every failure in the chosen order leaves the original in Avatars, or still protected in Ember, or both; the price is at most an extra copy in Ember. The worst case of the other order is losing an image someone paid for.

Nothing in Avatars retries a failed release on a timer. The flag stays on the task, the image most likely stays protected in Ember, and the next run of finish() for that task, such as a history repair, asks again.

history_missing: the bytes are safe, the record is not

Ember can save an image and still fail to write its Pro history row. It then reports history_error beside the image instead of failing the job. Avatars keeps the original anyway, marks the task history_missing with the phase Original saved; history needs repair, and carries Ember's error text.

That state blocks one thing: publishing. A set whose chosen images include a generated one with no Ember history id is refused with Ember history must be recovered before publishing generated images. Repair goes through reconciliation (the previous lesson): once a history row carrying this task id exists in Ember, the search finds it and finish() runs again. This time the asset already exists, so only the history id is written. The test for it calls finish() twice, first without and then with a history row, and checks that Ember's image file was fetched exactly once. The architecture's sentence for this case: an archived image with failed history insertion is recoverable metadata work, not permission to bill another generation.

Code

cwkAvatars, backend/worker.py: copy and record first, release last (verbatim, lines 222-241)·python

    async def finish(self, id, record):
        io, upstream = self.studio.io, self.studio.upstream
        task = await io("get", id, "task")
        existing = next((a for a in await io("list", "asset") if a.get("task_id") == id), None)
        result = record.get("result") or {}
        if existing:
            asset = existing
            history_id = (result.get("history") or {}).get("id")
            if history_id and not asset.get("ember_history_id"):
                asset = await io("update", asset["id"], ember_history_id=history_id, upstream=result)
        else:
            batch = await io("get", task["parent"], "batch")
            image = await upstream.jobs.fetch(record)
            metadata = (result.get("image") or {}).get("metadata") or {}
            label = f"{batch['title']} · {task['provider_label']} {task['ordinal']:02d}" + (f" · {task['frame']}" if task["frame"] != "image" else "")
            asset = await io("save_asset", image.data, label=label, subjects=task["subjects"], origin="generated", references=[r["id"] for r in task["references"]], task_id=id, batch_id=batch["id"], workflow=batch["request"].get("workflow", "studio"), ember_image_id=image.image_id, ember_history_id=(result.get("history") or {}).get("id"), provider=metadata.get("provider") or task["provider"], model=metadata.get("model"), prompt=task["prompt"], upstream=result)
        # Protection is released only after both native bytes and metadata
        # are durable in Avatars. A failed ack safely leaves it protected.
        released = await upstream.jobs.release(asset["ember_image_id"])
        await io("update", id, state="done" if asset.get("ember_history_id") else "history_missing", phase="Complete" if asset.get("ember_history_id") else "Original saved; history needs repair", asset_id=asset["id"], protection_released=released, error=result.get("history_error"))

cwkAvatars, backend/kit_ember_jobs.py (vendored from the Sibling Kit): the acknowledgement that never raises (verbatim, lines 174-186)·python

    async def release(self, image_id: str) -> bool:
        """Acknowledge a durable copy: Ember may now clean the exploration up.
        False (never raised) when Ember did not take it; the image then stays
        protected, the safe side."""
        try:
            async with self._client() as client:
                response = await client.patch(
                    f"/api/v1/explorations/{image_id}/protection",
                    json={"protected": False, "durable_copy": True},
                )
        except httpx.HTTPError:
            return False
        return response.status_code == 200

cwkAvatars, backend/store.py: written once, under its hash, through fsync and an atomic rename (verbatim, lines 174-187)·python

    def write_original(self, filename: str, data: bytes, digest: str) -> Path:
        """Content-addressed and written once; an existing file must still match."""
        path = self.root / "originals" / filename
        if path.exists():
            if hashlib.sha256(path.read_bytes()).hexdigest() != digest:
                raise StudioError("Stored original failed its checksum.", 503)
        else:
            temporary = path.with_name(f".{identity()}.tmp")
            with temporary.open("xb") as handle:
                handle.write(data)
                handle.flush()
                os.fsync(handle.fileno())
            os.replace(temporary, path)
        return path

Exercise

Model a two-store handoff in your own stack. One side produces a file and keeps it protected from cleanup. The other side fetches it, writes it to a temporary file, flushes and fsyncs it, renames it into place, records it in its database, and only then calls an acknowledgement that unprotects it on the first side. Make the acknowledgement return a boolean and never raise. Then kill the consumer at each step and run the producer's cleanup each time. Confirm that no step can leave the file missing from both sides.

Hint

Write down, for each kill point, which side holds the file and whether it is protected. If any row of that table reads 'consumer: none, producer: unprotected', your order is wrong. A second run of the consumer should find its own earlier copy and skip the fetch.