The Offline Shelf and Keyed Photo Uploads

Reading without the studio

The studio runs on one Mac, and the phone reaches it over Tailscale. Away from a signal, the app can still show what it has already read. StudioShelf keeps those reads, and its first comment sets the limits: no authorization or policy verdict and no paid request is ever a replica, and only a successful engine read may add bytes.

Scoped twice. The shelf lives under a digest of the studio's address, and inside that, under a digest of the current family credential. A different login never sees the last one's reads.
An allow-list of routes. Souls, sets, generators, templates, batches, the library pages, single records, an asset's file, thumbnail and preview, publication history and a soul's references. Health, auth status, the publish preview and the export-policy check are never saved, so the shelf can never answer "you may".
Budgets. Page data up to 32 MiB and 2,000 entries, media up to 512 MiB, oldest copies evicted first.
Prepared galleries. When a saved page lists images, the shelf reads each image's thumbnail, original and preview while the link is up, so the gallery opens offline too.

The read path decides per answer. Offline with a copy: serve the copy. Online: ask the engine. A 200 is saved, a 401 or 403 revokes the whole scope, a 404 or 410 deletes that copy, and a 5xx with a copy at hand marks the studio offline and serves the copy. Revocation also bumps an epoch, and every read checks the epoch it started under, so a read that was in flight when access ended cannot write its answer back afterwards. One test holds a read open, revokes, releases it, and asserts that nothing reappears.

Clips follow the same rule at block size. Every block read while the link was up is saved with the clip's length and type, so a cold start can reopen it. That is more than what was watched: a block is read whole, 1 MiB at a time, around whatever range the player asked for, and each block the player uses starts a read of the next one. A range outside every saved block fails visibly offline, because a partially fetched clip is not a downloaded clip.

Writes refuse before they leave

Generating, editing, uploading and publishing all fail before transmission when the studio is offline, with one sentence: saved studio reads are available, reconnect before changing work or generating, nothing was queued. Nothing queues, so nothing paid can replay later behind your back. While the app is in the foreground it checks every 30 seconds: health first, then a live read of the soul registry that a saved copy can never satisfy.

Photos on a slow uplink

Dad was abroad on a slow mobile uplink and chose photos to add to the library. The picker stayed on screen with touch disabled until every original had gone up, one request each, with no word. And a retry after a lost reply kept a twin of the photo. The fix changed what the screen says and what a retry does, not how fast the link is. A bench throttled to 61.4 KB/s with three 12 MP JPEGs, 10.87 MB in all, measured first visible feedback at 197.1 seconds before the change and 0.7 seconds after. Total time stayed the link's.

The picker hands back its picks and closes at once.
Each pick is copied out of the photo library into the app's caches and read back memory-mapped, so a 100 MB original costs no resident copy until it is sent.
Each original goes up untouched, in its own request, under a key shaped <pick>:files-n-m.
A status line under the connection bar shows which photo, how many MB, the rate and the seconds. On failure, Retry sends only what did not land, and Dismiss hands the page what did.
The transport is sized to the body: 120 seconds with no byte moving is a dead link, and the whole body gets time at 5 KB/s, never less than 300 seconds. That is a quarter of the slowest uplink the family measured abroad, 20 KB/s. The read transport's flat 960-second ceiling had failed a 60 MB original moving at 61 KB/s.

The key on the engine's side

POST /api/assets takes an optional client_key. The engine looks it up as upload:<key> in the same requests table that guards generation, with a fingerprint of the bytes' SHA-256 and the sorted subjects. Same key, same bytes and same subjects return the asset stored the first time. Same key with other bytes, or with the same bytes under other subjects, is a 409, never a silent substitute. The label is not in the fingerprint, so a retry that renames the photo still gets the first asset back.

One gap is worth saying plainly. Saving the asset and binding the key are two commits. The upload lock keeps two retries in one engine process from racing, but if the engine dies after the asset row is committed and before the binding is, the retry finds no binding and saves again. Because originals are content-addressed, the bytes on disk stay a single file under their hash; what doubles is the asset record in the library. Generation avoids this by writing the intent and its key in one transaction. An upload has a file to write first, but the file could be written before both rows, and the two rows could share one commit.

Which studio, by address

The app's default address is the studio's HTTPS twin from the family registry, and that exact string keys the offline shelf. Dad can point the app at another Mac in the information sheet. A saved address that names a registered family host in an older spelling is rewritten to the twin; anything else must be a whole https://host:port. A new address opens a fresh studio with its own shelf, and the app adopts that Mac's host profile from Pippa's open host-profile route.

Principle: Offline may read what it was shown. It may never decide, spend or queue.

Code

cwkAvatars, clients/apple/AvatarsMobile/Sources/StudioShelf.swift: one read, decided by the answer and guarded by the revocation epoch (verbatim, lines 53-81)·swift

    func read(_ request: URLRequest, path: String, scope: String, transport: HTTPTransport) async throws -> HTTPReply {
        let ticket = epoch
        if !connected, let copy = await cached(path, scope: scope) {
            guard epoch == ticket else { throw CancellationError() }
            return HTTPReply(statusCode: 200, data: copy.data)
        }
        do {
            let result = try await transport(request)
            try Task.checkCancellation()
            guard epoch == ticket else { throw CancellationError() }
            if [401,403].contains(result.statusCode) { await revoke(scope) }
            else if result.statusCode == 200 {
                if path.range(of: "/(file|thumbnail|preview)$", options: .regularExpression) != nil || (try? JSONSerialization.jsonObject(with: result.data, options: [.fragmentsAllowed])) != nil {
                    await save(result.data, path: path, scope: scope)
                }
            }
            else if [404,410].contains(result.statusCode) { await store(path, scope: scope).remove(path) }
            else if result.statusCode >= 500, let copy = await cached(path, scope: scope) {
                connected = false; return HTTPReply(statusCode: 200, data: copy.data)
            }
            return result
        } catch {
            try Task.checkCancellation()
            guard epoch == ticket else { throw CancellationError() }
            connected = false
            if let copy = await cached(path, scope: scope) { return HTTPReply(statusCode: 200, data: copy.data) }
            throw error
        }
    }

cwkAvatars, clients/apple/AvatarsMobile/Sources/Engine.swift: an upload's timeout sized to its bytes (verbatim, lines 43-45)·swift

    static func uploadPolicy(bytes: Int) -> HTTPSessionPolicy {
        HTTPSessionPolicy(requestTimeout: 120, resourceTimeout: max(300, 60 + Double(bytes) / 5_000), byteBudget: 8 << 20)
    }

cwkAvatars, backend/studio.py: a keyed upload answers the asset it stored, and other bytes or subjects under the key are a conflict (verbatim, lines 106-122)·python

    async def upload(self, data, label, subjects, actor=None, client_key=None):
        bound = [s["soul_pk"] for s in await self.subjects(subjects, actor)]
        if not client_key:
            return await self.save_upload(data, label, bound, actor)
        # A photo per request, keyed by the client (cwkSiblingKit todo #545): on
        # a slow link the original can land while the reply is lost, and the
        # retry answers the asset stored the first time instead of a twin. The
        # same key with other bytes is a conflict, never a silent substitute.
        request_id = f"upload:{client_key}"
        payload = {"sha256": hashlib.sha256(data).hexdigest(), "subjects": sorted(bound)}
        async with self.upload_lock:
            existing = await self.io("request", request_id, payload)
            if existing:
                return existing
            asset = await self.save_upload(data, label, bound, actor)
            await self.io("bind_request", request_id, payload, asset["id"])
        return asset

Exercise

Add keyed uploads to an upload endpoint of your own. The client sends one file per request with a key made of a pick id and the file's position. The server stores a fingerprint of the bytes' SHA-256 next to the key: the same key and bytes return the stored record, the same key with other bytes returns 409. Then close the gap this lesson describes: write the file first, and insert the record and the key binding in a single transaction. Prove it with a test that raises an exception between the file write and the commit, retries, and finds exactly one record.

Hint

Content-address the file (name it by its hash) and the file write becomes safe to repeat, so a crash after it costs nothing. The only thing that must be atomic is the pair of rows, which SQLite can give you with one BEGIN IMMEDIATE.

Reading without the studio

Writes refuse before they leave

Photos on a slow uplink

The key on the engine's side

Which studio, by address

Code

External links

Exercise

Progress

Comments 0