Lesson 06 of 06 · published

Read APIs and Derived Outputs

~14 min · etag, conditional-requests, derived-files, lru, subprocess

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Not Everything Worth Reading Is a File

Loomis does not only want thumbnails. It wants Eagle's folder tree, an item list, an item's metadata. Soul Vault does not only want note bytes; it wants a manifest of every note with its hash, a vault's change events after a cursor, a note's history. These are questions, not paths, and the root class has a verb for them: api(op, params), a read-only operation that returns JSON. A caller sends GET /v1/api/<share>/<op>?…; the server finds the share and the operation name, checks can_read exactly as for a file, and hands the rest to the child.

Read-only is a hard line. Every operation behind api is a GET, its answer may be cached on the receiving Mac like a file's bytes, and nothing about it may change the owner's data. The one way a share's owner writes on another Mac's behalf is a different verb with different rules entirely (call, track 7).

Two Ways to Answer "Nothing Changed"

An API answer can be large, and most of the time the caller already holds the current one. HTTP's conditional requests exist for that: the caller sends the ETag it holds in If-None-Match, and the owner answers 304 Not Modified with no body when it still matches. Delta's serve_api gets a 304 out of a child in one of two ways, and the order matters:

The child's validator, before any work. The server first calls validator(op, params). A child that can tell cheaply whether an answer changed returns a tag; if it matches what the caller sent, the server answers 304 and api is never called. Loomis' child does this for item-info: the answer changes only when Eagle rewrites that item's metadata.json, so the tag is that file's mtime and size, and a revalidation costs one stat and no Eagle API call. Soul Vault's child returns the store's own change tag for vaults, manifest and notes: no note is read to decide.
The answer's own content, after the work. When the child has no cheap tag, the server runs api, serializes the JSON, and uses the first 32 hex digits of its SHA-256 as the ETag. If that matches, it still answers 304: the owner did the work, but the bytes do not cross the network again.

Both answers carry Cache-Control: private, no-cache, which in HTTP means you may keep this, but ask me before you use it. That is the contract Delta's receiving cache follows (track 5), and it is why validators come from the owner instead of a guessed time-to-live.

Derived Outputs: Build Once Per Source Version

Some reads want a file that does not exist yet: a thumbnail of an image that was never given one, a PNG of a TIFF a browser cannot show. Building it on every request would be wasteful; building it once and forgetting when the source changes would be wrong. The root class reserves a hook for exactly this, derive(entry, kind, params), and Delta's serving side owns the machinery around it:

The child supplies only a recipe: a file suffix and an argument list with {src} and {out} placeholders, or None when that kind is not derivable. A caller asks for it with ?derive=<kind> on an ordinary file read.
The key is the source's version: SHA-256 over the share id, path, kind, parameters, the source's ETag (size, mtime, inode) and the recipe itself. Edit the source, or change the recipe, and the key changes, so a stale output is never served; it simply stops being asked for.
Two callers build it once: a per-key lock makes the second caller wait for the first one's result.
It never fills the disk: before building, Delta checks the volume against the family's disk floor and refuses with a named Unavailable when it is under.
It is built whole or not at all: the recipe writes to a temporary file, runs with a 60-second limit, and a non-zero exit or an empty output is a named derive_failed error carrying the tail of the tool's stderr. Only a good result is moved into place with an atomic os.replace.
It is bounded: outputs live under Delta's state root, never beside the source, capped at 4 GiB by default (derived_cap_gib in the host config). Past the cap, the least recently used outputs are deleted until the total is under 90% of it.

The served output gets its own ETag, the source's tag with the kind appended, so a receiving cache keeps the original and its thumbnail as two different things that each revalidate against the same source.

A Hook That Is Ready Before Its First User

Honesty about the present: as this quest is written, no shipped child overrides derive. Loomis' Eagle child answers a thumbnail request with Eagle's own generated thumbnail and falls back to the original when Eagle made none; the docstring of Delta's serving module names "an Eagle thumbnail when Eagle made none, a PNG of a TIFF" as the cases the hook exists for. The machinery (keys, locks, the floor, the cap, the eviction) is built and tested in the root, so the first child that needs it writes a two-line recipe, not a cache.

Code

The API verb: a cheap validator first, the answer's own hash second (trimmed excerpt)·python

# delta/server.py — serve_api (trimmed excerpt)

def serve_api(h, tail):
    share, op = _share_and_op(h, tail)
    _require_read(h, share)
    params = dict(h.query)
    inm = h.headers.get("If-None-Match")
    tag = share.validator(op, params)  # a child that can tell cheaply answers before any work
    if tag and etag_matches(inm, tag):
        h.send_head(304, {"ETag": tag, "Content-Length": 0, "Cache-Control": "private, no-cache"})
        return
    result = share.api(op, params)
    body = json.dumps(result, separators=(",", ":"), default=str).encode()
    tag = tag or '"' + hashlib.sha256(body).hexdigest()[:32] + '"'  # otherwise the answer's own content
    if etag_matches(inm, tag):
        h.send_head(304, {"ETag": tag, "Content-Length": 0, "Cache-Control": "private, no-cache"})
        return
    h.send_head(200, {"Content-Type": "application/json", "Content-Length": len(body), "ETag": tag,
                      "Cache-Control": "private, no-cache", "X-Delta-Share": share.id})
    h.write(body)


# loomis/delta_eagle.py — the child's cheap validator (trimmed excerpt)
def validator(self, op, params):
    """item-info changes only when Eagle rewrites the item's metadata.json, so
    its stat answers a revalidation without asking Eagle at all."""
    if op != "item-info" or not ITEM_RE.match(params.get("id", "")):
        return None
    st = (self.library() / "images" / f"{params['id']}.info" / "metadata.json").stat()
    return f'"ii-{st.st_mtime_ns:x}-{st.st_size:x}"'

Runnable: derived outputs keyed by source version, built once, evicted least-recently-used·python

# Derived outputs: a recipe from the share, built once per source version, kept under a size cap.
import hashlib
import json
import os
import subprocess
import sys
import tempfile
import time

UPPER = [sys.executable, "-c", "import sys; open(sys.argv[2], 'w').write(open(sys.argv[1]).read().upper())",
         "{src}", "{out}"]


def etag(path):
    st = os.stat(path)
    return f"{st.st_size:x}-{st.st_mtime_ns:x}-{st.st_ino:x}"


class Derived:
    def __init__(self, root, cap_bytes):
        self.root, self.cap, self.index, self.built = root, cap_bytes, {}, 0

    def get(self, src, kind, suffix, argv):
        key = hashlib.sha256(json.dumps([src, kind, etag(src), argv]).encode()).hexdigest()
        hit = self.index.get(key)
        if hit and os.path.exists(hit["file"]):
            hit["last_access"] = time.time()
            return hit["file"], "hit"
        out = os.path.join(self.root, key + suffix)
        fd, tmp = tempfile.mkstemp(dir=self.root, suffix=suffix)
        os.close(fd)
        try:
            cmd = [a.replace("{src}", src).replace("{out}", tmp) for a in argv]
            done = subprocess.run(cmd, capture_output=True, timeout=60)
            if done.returncode != 0 or os.path.getsize(tmp) == 0:
                raise RuntimeError(f"deriving {kind} failed: {done.stderr.decode()[-200:] or done.returncode}")
            os.replace(tmp, out)  # whole or not at all
        finally:
            if os.path.exists(tmp):
                os.unlink(tmp)
        self.index[key] = {"file": out, "size": os.path.getsize(out), "last_access": time.time()}
        self.built += 1
        self.evict()
        return out, "built"

    def evict(self):
        total = sum(e["size"] for e in self.index.values())
        if total <= self.cap:  # nothing to do until the cap itself is passed
            return
        for key, e in sorted(self.index.items(), key=lambda kv: kv[1]["last_access"]):
            if total <= self.cap * 0.9:
                break
            os.unlink(e["file"])
            total -= e["size"]
            del self.index[key]


if __name__ == "__main__":
    with tempfile.TemporaryDirectory() as data, tempfile.TemporaryDirectory() as state:
        src = os.path.join(data, "note.txt")
        with open(src, "w") as f:
            f.write("delta derives once per version\n")
        d = Derived(state, cap_bytes=1024)
        print(d.get(src, "upper", ".txt", UPPER)[1])   # built
        print(d.get(src, "upper", ".txt", UPPER)[1])   # hit: same source version
        time.sleep(0.01)
        with open(src, "a") as f:
            f.write("and again after an edit\n")
        out, how = d.get(src, "upper", ".txt", UPPER)
        print(how, "->", open(out).read().strip().splitlines()[-1])
        print("builds:", d.built, "entries kept:", len(d.index))

External links

Exercise

Run the runnable block. Then make it fail honestly: add a second recipe whose command exits non-zero (for example a Python one-liner that raises), call it, and confirm that nothing is left in the state folder and nothing is recorded in the index. Finally, lower cap_bytes to 40, derive three different kinds from the same source, and print which outputs survive the eviction.

Hint

List the state folder with os.listdir(state) after the failing call: the temporary file is removed in the finally, and the index only gains an entry after os.replace succeeded. With a 40-byte cap each output is about 30 to 55 bytes, so once the total passes 40 bytes, eviction trims oldest access first until the total is at most 36 bytes (90% of the cap).