Skip to content
C.W.K.
Lesson 04 of 06 · published

A Photo per Request and a Live Line

~14 min · uploads, progress, idempotency, slow-networks

Level 0Mise en Place
0 XP0/43 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

A minute with nothing to count

Until kit todo #545, the phone sent a plate as one multipart body: every photo and the form fields in a single request. On a slow uplink that one request was the whole wait, and nothing on the screen moved while it lasted. Dad's report was short:

이미지 첨부하고 save 누르면 멍 때림

Attach the images, press Save, and it just zones out.

The fix was measured on a throttling proxy from the kit's bench: a 61.4 KB/s uplink with a 256 KB queue and 150 ms each way, against a scratch engine. Before, three photos of 2.54 MB made their first visible change at 44.5 seconds, when the sheet closed, with no progress line at all; four photos of 3.53 MB took 62.3 seconds the same way. After, the first line appeared at 2.0 and 2.4 seconds. The totals barely moved, 44 and 62 seconds, because time and bytes belong to the link. What changed is that every second now says which photo is going, how many megabytes of how many, the rate and the seconds. The web followed fifteen minutes later and measured a first line at 0.4 seconds and the three photos done at 46.4.

The staging door

The engine mounts the kit's kit_staged_uploads at POST /api/staged-uploads. A request carries a client_key and one file. Four rules make it safe on a bad link:

  • Idempotent on the key. The staged id is the first 32 hex characters of the key's SHA-256, so the same key always names the same id. On a slow link the bytes can land while the reply is lost; the retry gets the first copy's record back and nothing is written twice, even after an engine restart.
  • Taken, then released. Reading staged files deletes nothing. The app releases them only after its own record has committed, so a crash between the two still leaves a retry that works.
  • Swept. Anything unclaimed for a week goes on the next upload.
  • Bounded while reading. Only image types are accepted, and the size cap (60 MB for Sizzle) is enforced chunk by chunk as the body is read, so an oversized part is refused before it all sits in memory.

Keys that make a retry free

Both clients build every key from the plate's request id, in the form <request id>:files-<from>-<to>. The web sends one file per key through the kit's kitUploadFiles and keeps the ids that landed in a ref. Changing the photo set forgets them, because the keys would no longer describe the same files. Forgetting is not enough, though, and this is a live bug. The request id stays the same until Save succeeds or the card is reset, so after a failed Save a changed set asks again under the same positional keys, and the staging door answers each repeated key with the copy it already holds. Swap the second photo after a half-finished Save, and the plate can come back with the old second photo. The fix is a fresh request id whenever the set changes. The phone is clear of it, because a plate's photos are fixed the moment its outbox record commits.

The phone uses the kit's FileUploadRun, which plans batches of consecutive files up to about 1 MB. A phone photo of that size goes alone, and several small ones share one batch key. Even then EngineClient.stage sends one request per file, appending the file's offset to the key. The same sizes always give the same batches, so a resumed upload asks under the same keys. After each batch lands, the ids so far are written to the outbox record as stagedIds, and the next pass starts at the first batch that did not. The checkpoint is per batch, not per photo. If the second photo of a three-photo batch fails, the first one landed but was never recorded, so the retry sends it again. That costs its bytes once more and nothing else: the same key reaches the staging door, which answers with the copy it already holds.

Then the plate names them

With every photo staged, the plate's own request goes to POST /api/encounters with the request id, the form fields and one staged field per id. The engine checks the request id first and answers a replay with the stored plate. Otherwise it takes the staged files, stores the plate, and releases the staged copies.

One case needed its own branch. A plate that waited on the phone for more than a week names staged ids the sweep already removed, and the engine refuses it with a 400 saying the upload is not here. A 400 would normally park the plate. The phone recognises this one, clears stagedIds, and turns it into a transient failure, so the next pass stages the photos again from the bytes it still holds.

A line that counts

The line is the kit's: UploadStatusLine on the phone, KitUploadStatus on the web. It reads like "Uploading photo 2 of 3 · 1.1 of 2.9 MB · 52 KB/s · 21s", then "Saving the plate · 3s" once the files are up. The web stages with XMLHttpRequest, because its upload progress events count the body's bytes and fetch cannot. Once the engine has confirmed a file, the rate shown is the confirmed one, so the browser's send buffer cannot inflate it.

Quiet is a diagnosis, never a timer. After 20 seconds with no byte moving (longer while a large body may still be draining the device's send buffer), the line checks the device's own link and then whether the engine answers a health read, and says which of the two went silent. On the phone, Close leaves the plate sending, and its row on Plates shows the same line. The same commit fixed Dad's second report:

x 눌러도 첨부 이미지 삭제 안 되는 버그

The bug where pressing the X does not remove the attached image. The photo strip now gives every prepared photo its own id, so the X removes that photo and no other.

A doc line that lived fifteen minutes

The architecture doc's paragraph on phone uploads ends by saying the web client still sends one multipart body. That was written in 1021db2 and stopped being true fifteen minutes later, when 00c4e83 moved the web to staging. The web code in this lesson is the current shape.

Principle: Make the unit of upload small enough to count and keyed so that repeating it is free. Then progress can be shown honestly, and a dropped link costs only the piece that was in flight.

Code

cwkSizzle, clients/apple/SizzleMobile/Sources/Outbox.swift: the drain's send, staging then the plate, with the swept-photo branch (verbatim, lines 118-134)·swift
        let detail: EncounterDetail
        do {
            let ids = try await run.run(stored: already, send: { batch, key in
                try await Self.classified { try await self.stage(batch, key: key) }
            }, landed: { ids in try await outbox.setStaged(entry.id, ids) })
            await run.finishing()
            do {
                detail = try await Self.classified { try await capture(entry.payload, requestId: entry.id, staged: ids) }
            } catch OutboxFailure.permanent(let reason) where reason.contains("is not here") {
                // Staged a week ago and swept: stage them again on the next pass.
                try await outbox.setStaged(entry.id, nil)
                throw OutboxFailure.transient("The engine no longer had this plate's photos; they go up again.")
            }
        } catch let failure as OutboxFailure {
            await run.fail(failure)
            throw failure
        }
cwkSizzle, frontend/src/pages/Plates.tsx: the web's Save, a photo per request and then the plate naming them (verbatim, lines 106-139)·typescript
  // A photo per request with the line live (cwkSiblingKit todo #545), then the
  // plate naming them. Everything is keyed by the plate's request id, so a
  // retry resends only what did not land and never makes a second plate.
  const save = async () => {
    if (!files.length || saving) return;
    setSaving(true);
    try {
      const ids = await kitUploadFiles(files, {
        stored: staged.current,
        send: (file, index, onProgress) => api.stage(file, `${requestId.current}:files-${index}-${index + 1}`, onProgress),
        onStatus: setUpload,
        landed: (stored) => {
          staged.current = stored;
        },
      });
      setUpload((s) => s && kitUploadPhase(s, "finishing"));
      const form = new FormData();
      form.append("request_id", requestId.current);
      form.append("kind", kind);
      form.append("note", note);
      form.append("captured_at", new Date().toISOString());
      form.append("timezone", Intl.DateTimeFormat().resolvedOptions().timeZone);
      ids.forEach((id) => form.append("staged", id));
      await api.capture(form);
      reset();
      setError(null);
      onSaved();
    } catch (exc) {
      // The request id and the staged photos stay: Retry resends only what was lost.
      setUpload((s) => s && kitUploadPhase(s, "failed", (exc as Error).message));
    } finally {
      setSaving(false);
    }
  };
cwkSizzle, backend/tests/test_capture.py: both steps replayed, one plate, staged files released (verbatim, lines 62-83)·python
def test_photos_staged_one_per_request_become_the_plate(make_client, fake_pippa, tmp_path):
    """The phone since cwkSiblingKit #545: each photo to /api/staged-uploads under
    its own key, then the plate names them. A replay of either step is free."""
    client = make_client()
    ids = []
    for index in range(3):
        key = f"plate-req-staged-01:files-{index}-{index + 1}"
        files = {"file": (f"plate-{index + 1}.jpg", jpeg_bytes(), "image/jpeg")}
        first = client.post("/api/staged-uploads", data={"client_key": key}, files=files)
        assert first.status_code == 200, first.text
        again = client.post("/api/staged-uploads", data={"client_key": key}, files=files)
        assert again.json() == first.json()
        ids.append(first.json()["id"])
    body = client.post("/api/encounters", data={"request_id": "req-staged-01", "kind": "out", "staged": ids})
    assert body.status_code == 200, body.text
    plate = body.json()
    assert len(plate["photos"]) == 3
    # Released once the plate committed; a lost reply's retry still answers the plate.
    assert not list((tmp_path / "sizzle-db" / "staged").glob("*.bin"))
    replay = client.post("/api/encounters", data={"request_id": "req-staged-01", "kind": "out", "staged": ids})
    assert replay.status_code == 200
    assert replay.json()["encounter"]["id"] == plate["encounter"]["id"]

External links

Exercise

Find an upload in your own app that sends several files in one request. Add a staging endpoint that stores one file per request under a client key, derives the stored id from that key, and answers a repeated key with the first copy. Change the client to stage each file under a key built from the record's request id and the file's position, keep the ids that landed, and resume after the last one on a retry. Then post the record naming the staged ids. Measure the first visible feedback before and after on a throttled link.
Hint
Browser developer tools and macOS's Network Link Conditioner can both throttle an uplink. Hash the client key for the id so a server restart cannot forget a file that already landed. Delete staged files only after the record that claims them has committed, and sweep the unclaimed ones on a schedule of days. When the file set changes, mint a new request id as well, or a repeated key hands back a file that is no longer in the set.

Progress

Progress is local-only — sign in to sync across devices.
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.