Lesson 02 of 05 · published

Publish Against an Expected Revision

~14 min · idempotency, concurrency, publishing, api-design

Level 0Sketch

0 XP0/43 lessons0/13 achievements

0/100 XP to next level100 XP to go0% complete

Preview first

The Publish button on a collection does not publish. It asks for GET /api/sets/{id}/publish-preview, and the answer has four parts: the set with its current selection, the frames still missing, the chosen assets, and expected_revision, which is what Pippa says its live copy of this soul, channel and collection looks like right now. Avatars gets that last value from Pippa's own door, /api/avatar-studio/revision, which answers only a sibling calling on the same machine.

The modal is titled Publish to Pippa. It names the soul, the channel and the collection, says The current live set is backed up before these selections are installed., and shows every frame that will be installed. If a frame is missing it shows Choose missing frames: with their names instead, and the publish button stays disabled. The other button is Back to review.

When the preview opens, the page mints a request id with crypto.randomUUID() and keeps it with the preview. One id per preview, never one per click. Hold on to that; it is what makes the button safe to press twice.

What a revision is

Pippa computes a revision as a hash of hashes: the SHA-256 of every frame file, keyed by frame name, serialised with sorted keys, hashed once more. It depends on frame names and file contents only, never on modification times. A folder that does not exist yet has a revision too, the hash of an empty map, so a brand new collection publishes against a real value. Avatars computes the very same formula from its own recorded hashes when it audits copies, which is the last lesson of this track.

Three values in one request

Back up & publish sends the request id, the set's version as the preview saw it, and the expected revision. Studio.publish takes the studio's intent lock and first asks whether this request id is already known, with a fingerprint over the set id, version, expected revision and mode. A known id with different input is a 409. A known id whose publication already applied simply returns it.

A new id rebuilds the preview from scratch and compares. If the set's version moved or Pippa's live revision moved, the publish is refused with The selection or live set changed. Review the preview again. Then come the content checks: no missing frame, and no generated image whose Ember history id is missing (Ember history must be recovered before publishing generated images.). Only then does it freeze a payload, every frame with its asset id, original path, SHA-256 and subjects, and commit a pending publication bound to the request id. Then it calls Pippa. When Pippa's receipt comes back, complete_publication marks the publication applied and writes the set's published, live_revision, publication_id and owner in one transaction.

Pippa checks again, on its side

Pippa does not trust that Avatars checked. Its publish fingerprints the whole body, keeps one receipt file per request id, and under its own lock compares the live revision once more: The live avatar set changed; refresh the publication preview. The second check is the one that counts, because only Pippa can see its folder at the moment it writes. Pippa's route turns a ValueError into 409 and a PermissionError into 403, and Avatars passes the status and the sentence through to the screen.

Check	Where	Refusal
same request id, different input	Avatars and Pippa	409
set version or live revision moved since the preview	Avatars	409
a frame left empty, or Ember history missing	Avatars	422 or 409
live revision moved since Avatars looked	Pippa	409
original missing or changed, protected subject under a public soul	Pippa	409 or 403

When the answer is lost

Avatars waits up to 120 seconds for Pippa. Suppose Pippa installs the set and the connection drops before the receipt arrives. Avatars' publication is still pending, and Dad sees an error. He clicks again in the same modal: same request id, same version, same revision. The lookup finds the pending publication, and Avatars resends the payload it froze the first time, not a fresh preview. Pippa finds its receipt for that id, the fingerprint matches, and it returns the applied receipt without touching a file. Avatars completes the record. The test below plays the Avatars half of this against a fake Pippa, with one twist: between the two attempts Dad chooses a newer candidate. The retry still installs what was reviewed, the newer pick stays a selection, and a third call returns the applied publication without calling Pippa at all.

The publication history covers the case where nobody clicks again. A pending publication reads Awaiting recovery with a Resume publication button, which rebuilds the same call from the stored record. One honest gap: if Pippa refused after the pending record was committed, resuming repeats the refusal. That record stays in the history, and the way forward is a fresh preview.

What the modal does with a conflict

It shows the error and stays open. It does not reload and resends nothing on its own. A second click repeats the same request, which is safe either way: a request that reached Pippa completes, and one that was refused is refused again, identically. To publish over a live set that really changed, Dad closes the modal and opens a new preview, which brings a new request id, the new revision, and a fresh look at what he is about to replace.

Principle: a write that replaces something names the version it expects to replace. Without that, the last writer wins, and nobody learns what was lost.

Code

cwkAvatars, backend/studio.py: publish, recall a known request, compare, freeze, then call Pippa (verbatim, lines 606-634)·python

    async def publish(self, set_id, request_id, version, expected_revision, *, mode="publish", publication_id=None):
        async with self.intent_lock:
            request = {"set_id": set_id, "version": version, "expected_revision": expected_revision, "mode": mode}
            if publication_id:
                request["publication_id"] = publication_id
            existing = await self.io("request", request_id, request)
            if existing:
                if existing["state"] == "applied":
                    return existing
                receipt = await self.upstream.publish(existing["payload"])
                return await self.io("complete_publication", existing["id"], receipt)
            preview = await self.publish_preview(set_id, publication_id)
            collection = preview["set"]
            if mode == "adopt":
                collection = {**collection, "selection": collection["published"]}
                preview["assets"] = await self.reference_assets(list(collection["published"].values()))
                preview["missing"] = [f for f in collection["frames"] if f not in collection["published"]]
            if collection["version"] != version or preview["expected_revision"] != expected_revision:
                raise Conflict("The selection or live set changed. Review the preview again.")
            if preview["missing"]:
                raise StudioError("Choose every frame before publishing: " + ", ".join(preview["missing"]))
            assets = {a["id"]: a for a in preview["assets"]}
            if any(a.get("origin") == "generated" and not a.get("ember_history_id") for a in assets.values()):
                raise StudioError("Ember history must be recovered before publishing generated images.", 409)
            frames = {frame: {"asset_id": id, "path": str(self.store.asset_path(assets[id])), "sha256": assets[id]["sha256"], "subjects": assets[id]["subjects"]} for frame, id in collection["selection"].items()}
            payload = {"request_id": request_id, "collection_id": set_id, "mode": mode, "soul_pk": collection["soul_pk"], "channel": collection["channel"], "name": collection["name"], "expected_revision": expected_revision, "frames": frames}
            record = await self.io("create_intent", request_id, request, "publication", {"state": "pending", "request": request, "payload": payload, "frames": collection["selection"], "restored_from": publication_id, "provenance": "legacy-adoption" if mode == "adopt" else "studio-publication"}, parent=set_id)
            receipt = await self.upstream.publish(payload)
            return await self.io("complete_publication", record["id"], receipt)

cwkPippa, backend/services/avatar_studio.py: publish, the receipt per request id and the second revision check (verbatim, lines 198-217)·python

def publish(body: dict) -> dict:
    """Idempotent whole-directory activation with a durable prior backup."""
    request_id = body["request_id"]
    if not re.fullmatch(r"[a-zA-Z0-9-]{1,100}", request_id):
        raise ValueError("Invalid publication request ID")
    fingerprint = hashlib.sha256(json.dumps(body, sort_keys=True).encode()).hexdigest()
    with _publication_lock:
        STATE.mkdir(parents=True, exist_ok=True)
        receipt_path = STATE / f"{request_id}.json"
        if receipt_path.exists():
            receipt = json.loads(receipt_path.read_text())
            if receipt["fingerprint"] != fingerprint:
                raise ValueError("Publication ID belongs to different input")
            if receipt["status"] == "applied":
                return receipt
            return finish_publication(receipt, receipt_path)
        target = publication_target(body["soul_pk"], body["channel"], body["name"])
        previous = revision(frame_files(target))
        if previous != body["expected_revision"]:
            raise ValueError("The live avatar set changed; refresh the publication preview")

cwkAvatars, backend/tests/test_studio.py: a lost answer is recovered with the frozen selection (verbatim, lines 132-148)·python

@pytest.mark.asyncio
async def test_publish_recovers_response_loss_using_frozen_selection(studio):
    current = await collection(studio)
    studio.upstream.fail_after_apply = True
    with pytest.raises(StudioError):
        await studio.publish(current["id"], "publish-once", current["version"], "before")
    # Dad can review a newer candidate without changing the in-flight publish.
    newer = await studio.upload(image("yellow"), "new candidate", ["pippa"])
    await studio.select(current["id"], Selection(version=current["version"], choices={"default": newer["id"]}))
    receipt = await studio.publish(current["id"], "publish-once", current["version"], "before")
    assert receipt["state"] == "applied"
    assert studio.upstream.publish_calls[0] == studio.upstream.publish_calls[1]
    saved = await studio.io("get", current["id"])
    assert saved["published"] == current["selection"]
    assert saved["selection"]["default"] == newer["id"]
    await studio.publish(current["id"], "publish-once", current["version"], "before")
    assert len(studio.upstream.publish_calls) == 2

External links

Exercise

Build a publish endpoint for something your app installs elsewhere: a config file, a static page, a set of images. The client first fetches a preview that includes a revision of the current target, and gets a request id minted with that preview. The publish call sends both. The server refuses with 409 when the target's revision moved, and stores a receipt per request id so a repeat returns the first result. Then fake a lost response: apply, drop the answer, and call again with the same id.

Hint

Compute the revision from content (a hash of each file's hash, keyed by name and sorted), never from modification times. Fingerprint the request body and refuse a known id with a different fingerprint, or a buggy client can reuse an id for a different publication and receive the old receipt.

Publish Against an Expected Revision

Preview first

What a revision is

Three values in one request

Pippa checks again, on its side

When the answer is lost

What the modal does with a conflict

Code

External links

Exercise

Progress

Comments 0