Lesson 02 of 05 · published

Four Answers to One Read

~14 min · cache, http, etag, conditional-requests, observability

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

One Read, Four Ways to Answer It

An app on the service host asks its own Delta for a file that lives on the workbench. Before the engine decides anything, it looks the request up in the cache index. What happens next is one of four things, and Delta always says which, in a response header named X-Delta-Cache:

`X-Delta-Cache`	What happened	What crossed the network
`hit`	The caller named the version it expects, and the cached entry is that version	Nothing. No route was even chosen
`revalidated`	The engine asked the owner "still this one?" and the owner answered `304 Not Modified`	A request and an empty answer: a few hundred bytes
`miss`	Nothing usable was cached, or the owner sent new bytes	The whole body, which is kept for next time
`stale`	The owner could not be reached, and a cached copy existed	Nothing; the copy is served with a label saying why

There is a fifth value, pass, for an answer the engine relayed without keeping: the app opted out with cache=0, the owner sent no validator, or the body was over the per-entry cap. And a second header, X-Delta-Route, says which way the bytes came (thunderbolt, lan, tailnet). On a hit or a stale answer no bytes crossed, so the header carries the route this Mac last chose to that peer, if it still remembers one, and otherwise cache (? for a stale copy). Read X-Delta-Cache to learn whether anything moved; X-Delta-Route alone can say lan on a hit.

Revalidated: the Default Path

When an entry exists and the caller said nothing, the engine sends the owner a conditional request: the same GET, plus If-None-Match carrying the validator it stored with those bytes. The owner's gateway computes the file's current validator from a stat, compares, and if they match answers 304 with no body. The engine serves the blob it already holds. This is HTTP's own conditional-request machinery (RFC 9110, section 13), used between two programs that both speak it precisely.

A revalidation is cheap, but it is not free. It is a round trip, and the owner still has to look at the file. The architecture document records what that costs at scale: Soul Vault reading the canonical vault note by note made 2,123 round trips a pass, every one a 304, each spending 4.9 ms inside its share's file() on the owner. Correct, and still 2,123 conversations about bytes that had not changed.

Hit: When the Caller Already Knows

Often the caller does know. It just listed the folder, and a listing carries every file's etag. A caller that passes that tag back as expect is saying "I want exactly this version". If the cached entry's validator is that tag, the engine answers from disk without a single packet leaving the Mac. The kit client exposes it directly: Delta.read(peer, share, path, expect=tag). A caller that already holds the validator a listing gave it turns a round trip into a local read. The vault reader above does not do that today, and not out of carelessness: its manifest carries Soul Vault's own hash, size and modification time, not the validator Delta's share hands out, and its note() reads with no expect at all. The architecture document's line about those 2,123 round trips becoming cache hits describes what expect would buy once the reader can name the version, not what the code does now.

The 1110-7 review found one leftover cost on that path. The first version chose a route before it checked for a hit, and choosing a route can mean probing the peer. When the owner was switched off, a hit (bytes already on the disk) waited about eight seconds for a probe that did not matter. Now a pure hit chooses no route at all; the route header reads whatever route was last known, or cache. A regression test, test_a_cache_hit_chooses_no_route, fails if a hit ever asks for one again.

Miss and Stale

A miss is the ordinary first read, or a read after the owner's file changed: the owner answers 200 with the body and its new validator, the engine streams it to the app and into a blob, and the index row moves to the new version. If the owner answers 404 for an entry that was cached, the entry is forgotten: gone at the source means gone here.

Stale is the one answer that is not about the bytes at all. It is about the owner being unreachable. Lesson 4 is entirely about it; for now, notice the shape: stale is only possible when a copy exists. A read with nothing cached and nobody answering does not invent anything. It fails with the named cause the routing layer produced.

Counted, So You Can See It Working

Every outcome is counted. delta status and the health document show hits, revalidated, misses, stale_served, bytes_from_cache (reported as bytes_saved), the per-app entry counts, and a hit_ratio that counts hits and revalidations together, since both answered from local bytes. The Loomis grid shows the payoff: a warm open of 384 thumbnails took 1.67 to 1.88 s with every tile a 304 and no new bytes crossing.

Code

Delta's engine: the read path through the cache (trimmed excerpt)·python

# delta/proxy.py: the read path an app's GET takes through the engine (trimmed excerpt)

def proxy_fs(h, peer, tail):
    cache = h.delta.cache
    key = cache.key(peer, tail, "fs", json.dumps(_variant(h.query)))
    hit = cache.lookup(key) if cacheable else None
    expect = h.query.get("expect")
    if hit and expect and expect == hit.etag:
        # No network at all, so no route is chosen either: choosing one can probe the
        # peer, and when the owner is off that was an 8 s wait in front of bytes that
        # were already here.
        cache.touch(hit, "hits")
        _serve_hit(h, hit, "hit", peer, h.delta.routes.known(peer) or "cache")
        return
    headers = {"If-None-Match": hit.etag} if hit else {}
    try:
        resp = h.delta.peers.request(peer, h.method, wire_path("v1", "fs", tail) + _qs(h.query),
                                     headers=headers, app=h.app or None)
    except (Unavailable, Timeout) as exc:
        if hit:                                   # the owner is out of reach
            cache.touch(hit, "stale_served")
            _serve_hit(h, hit, "stale", peer, h.delta.routes.known(peer) or "?", stale=exc.detail)
            return
        raise                                     # nothing kept: the named cause goes back
    if resp.status == 304 and hit:
        cache.touch(hit, "revalidated")
        _serve_hit(h, hit, "revalidated", peer, resp.route.kind)
        return
    if resp.status == 404 and hit:
        cache.forget(key)                         # gone at the source, gone here
    ...                                           # a 200: stream it, keep it ("miss")

Run all four answers on one machine: an owner with ETags and 304s, a cache that reports what happened·python

# Four answers to one read: an owner that serves one file with an ETag and 304s,
# and a receiving cache that reports hit, revalidated, miss or stale.
import hashlib
import os
import tempfile
import threading
import urllib.error
import urllib.request
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer

SOURCE = os.path.join(tempfile.mkdtemp(prefix="owner-"), "note.txt")


def etag_of(path):
    st = os.stat(path)
    return '"%x-%x-%x"' % (st.st_size, st.st_mtime_ns, st.st_ino)


class Owner(BaseHTTPRequestHandler):
    def do_GET(self):
        tag = etag_of(SOURCE)
        if self.headers.get("If-None-Match") == tag:
            self.send_response(304)
            self.send_header("ETag", tag)
            self.end_headers()
            return
        body = open(SOURCE, "rb").read()
        self.send_response(200)
        self.send_header("ETag", tag)
        self.send_header("Content-Length", str(len(body)))
        self.end_headers()
        self.wfile.write(body)

    def log_message(self, *args):
        pass


class Cache:
    def __init__(self, url):
        self.url = url
        self.entry = None   # (etag, sha256, bytes)

    def read(self, expect=None):
        if self.entry and expect and expect == self.entry[0]:
            return "hit", self.entry[2]              # no network at all
        request = urllib.request.Request(self.url)
        if self.entry:
            request.add_header("If-None-Match", self.entry[0])
        try:
            with urllib.request.urlopen(request, timeout=2) as resp:
                data = resp.read()
                tag = resp.headers["ETag"]
        except urllib.error.HTTPError as exc:
            if exc.code == 304 and self.entry:
                return "revalidated", self.entry[2]  # a few hundred bytes crossed
            raise
        except OSError as exc:
            if self.entry:
                return "stale (%s)" % exc.__class__.__name__, self.entry[2]
            raise
        self.entry = (tag, hashlib.sha256(data).hexdigest(), data)
        return "miss", data


if __name__ == "__main__":
    with open(SOURCE, "w") as handle:
        handle.write("first version\n")
    server = ThreadingHTTPServer(("127.0.0.1", 0), Owner)
    threading.Thread(target=server.serve_forever, daemon=True).start()
    cache = Cache("http://127.0.0.1:%d/note.txt" % server.server_address[1])

    print(cache.read()[0])                          # miss: first time
    print(cache.read()[0])                          # revalidated: owner says 304
    print(cache.read(expect=etag_of(SOURCE))[0])    # hit: caller named the version
    with open(SOURCE, "w") as handle:
        handle.write("second version, longer\n")
    print(cache.read()[0])                          # miss: the validator moved
    server.shutdown()
    server.server_close()
    outcome, data = cache.read()
    print(outcome, data)                            # stale, and it says so

External links

Exercise

Run the script above and watch the five lines it prints. Then change it: give the owner a counter of how many times it opened the file, and add a sixth read that passes expect with a tag that is no longer current (save the first version's tag before the rewrite). Put that read just before server.shutdown(), while the owner still answers; after the shutdown every read is the stale case. Predict before you run it: is that read a hit, a revalidation or a miss, and how many times did the owner open the file in total?

Hint

A mismatched expect is not an error; it simply does not qualify as a hit, so the engine falls through to the conditional request. In Delta's code the comparison is expect == hit.etag, nothing cleverer.