Lesson 04 of 05 · published

Stale Is a Label, Not a Lie

~13 min · stale-if-error, availability, disk-space, macos, cache-eviction

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

The Owner Is Off. Now What?

Laptops sleep. A Mac restarts for an update. The tailnet takes a few seconds to find a path. Any read across machines will eventually meet an owner that does not answer, and the receiving side has two honest options and one dishonest one.

Fail with the cause. Say exactly what did not answer, on which route, in which phase.
Serve the copy you have, and say it is a copy. The bytes were right the last time the owner confirmed them; whether that is good enough is not the cache's call.
Serve the copy and say nothing. This is the dishonest one, and it is the default of more caches than anyone would like.

Delta does the first when it holds nothing, and the second when it holds something. It never does the third.

The Label

When the request to the owner raises Unavailable or Timeout and a cached entry exists, the engine serves the entry with X-Delta-Cache: stale and a second header, X-Delta-Stale, carrying the reason (URL-quoted, the first 200 characters): the same named cause the routing layer would have raised, for example that a connect to the peer over the tailnet timed out after eight seconds. The serve is logged as a warning and counted under stale_served. When nothing is cached, there is no copy to label: the error itself goes back, 503 for an unreachable owner or 504 for a timeout, with the same detail.

HTTP has a standard version of this idea (RFC 5861's stale-if-error), and it carries the same lesson: a stale answer is a legitimate answer only when it is announced. A connection that breaks in the middle of a small body is handled the same way: the engine reads a body of up to 4 MiB whole before sending a single header, so the caller receives the stale copy if one exists, or the named cause otherwise. A large body that was already streaming cannot be taken back; its connection ends short of the promised Content-Length, which the caller's client reads as an error, never as a complete file.

The Caller Decides What Stale Is Worth

Delta labels; it does not judge. Two of its readers make opposite decisions, and both are right:

Caller	Policy on `stale`	Why
A thumbnail grid	Draw it	A picture from an hour ago is still the right picture
Soul Vault reading its canonical store	Refuse: raise `CanonicalUnreachable`, quoting `X-Delta-Stale`	A store that decides what is true may not decide from an old copy

This is why the label matters more than the copy. If Delta silently served stale bytes, the vault reader could not tell, and would treat a snapshot as the present. Because the outcome is in a header, a policy that needs freshness can enforce it in one line, and a policy that tolerates staleness costs nothing.

Bounded Twice: Size, and the Disk Under It

A cache that grows forever eventually becomes the reason a Mac runs out of space. Delta's has two bounds. The first is a size cap, 20 GiB unless the Mac's host config sets cache_cap_gib. When the blobs exceed it, the least recently read entries are dropped until the total is back under 90% of the cap. The total is a scan of every blob row, so it is taken at most once a second, not after every stored thumbnail.

The second is a disk floor, 50 GiB unless cache_floor_gib says otherwise, and it is read the family way. macOS answers "how much room is left?" with two true numbers. statfs (what df and shutil.disk_usage report) counts only blocks free right now; System Settings and Finder add the purgeable space macOS frees on demand. On the workbench, on 2026-11-06, the first said 489.5 GB and the second 3.98 TB. A floor judged on the first would have called a Mac with almost four free terabytes nearly full. So the cache judges by the System Settings figure (through the family's kit_disk_space reader), plus a small hard stop on free-right-now space (2 GiB) as a last line, since nobody has measured that macOS purges fast enough for a writer that never pauses.

Under the floor, the cache does not fail anything. It stops storing new entries and serves straight through, logs the change once (cache paused (no new entries), and later resumed), counts each skipped entry under skipped_disk, and shows storing: false with the figures in health. Reads keep working; only the remembering pauses.

Code

Delta: the stale answer and the disk floor (trimmed excerpts)·python

# Delta: a stale answer, and the disk floor that pauses storing (trimmed excerpts)

# delta/proxy.py: the owner could not be reached
def proxy_fs(h, peer, tail):
    ...
    try:
        resp = delta.peers.request(peer, h.method, wire_path("v1", "fs", tail) + _qs(h.query),
                                   headers=headers, app=h.app or None, force=force)
    except (Unavailable, Timeout) as exc:
        if hit:
            log.warning("serving %s/%s stale from the cache: %s", peer, tail, exc.detail)
            cache.touch(hit, "stale_served")
            _serve_hit(h, hit, "stale", peer, delta.routes.known(peer) or "?", stale=exc.detail)
            return
        raise      # nothing kept: 503 or 504 with the named cause
    ...


def _serve_hit(h, hit, outcome, peer, route_kind, stale=""):
    extra = {"X-Delta-Cache": outcome, "X-Delta-Route": route_kind, "X-Delta-Peer": peer}
    if stale:
        extra["X-Delta-Stale"] = quote(stale[:200])
    h.send_path(hit.blob, size=hit.size, etag=hit.etag, mtime_ns=hit.mtime_ns,
                name=hit.name or "data", extra=extra)


# delta/cache.py: judged the way System Settings counts, re-read at most every 30 s
FREE_NOW_HARD_STOP = 2 * GIB


def disk_ok(self) -> bool:
    now = time.monotonic()
    if now - self._disk_checked < 30:
        return self._disk_ok
    self._disk_checked = now
    space = disk.volume_space(self.root)
    ok = space.available_bytes > self.floor and space.free_bytes > FREE_NOW_HARD_STOP
    if ok != self._disk_ok:
        log.warning("cache %s: %s", "resumed" if ok else "paused (no new entries)", note)
    self._disk_ok = ok
    return ok

One stale answer, two callers, two policies·python

# The cache labels; the caller decides. Two callers, one stale answer, two policies.
from urllib.parse import quote, unquote


class StaleRefused(Exception):
    pass


def answer_from_cache(blob, reason):
    """What the engine does when the owner cannot be reached and a copy exists."""
    return {"status": 200, "body": blob,
            "headers": {"X-Delta-Cache": "stale", "X-Delta-Stale": quote(reason[:200])}}


def thumbnail_caller(resp):
    # A picture from an hour ago is still the right picture to draw.
    return resp["body"]


def canonical_store_caller(resp):
    # A store that decides what is true may not decide on an old copy.
    if resp["headers"].get("X-Delta-Cache") == "stale":
        raise StaleRefused("canonical store out of reach: " + unquote(resp["headers"]["X-Delta-Stale"]))
    return resp["body"]


resp = answer_from_cache(b"cached bytes", "connect to the service host via tailnet timed out after 8.0 s")
print("thumbnail grid:", thumbnail_caller(resp))
try:
    canonical_store_caller(resp)
except StaleRefused as exc:
    print("vault reader:  ", exc)

macOS free space two ways: statfs and the figure System Settings shows (macOS only)·python

# Free space two ways on macOS: what statfs says is free now, and what
# System Settings calls available (free now plus purgeable space).
import ctypes
import os
import shutil

CF = ctypes.CDLL("/System/Library/Frameworks/CoreFoundation.framework/CoreFoundation")
CF.CFURLCreateFromFileSystemRepresentation.argtypes = [ctypes.c_void_p, ctypes.c_char_p, ctypes.c_long, ctypes.c_bool]
CF.CFURLCreateFromFileSystemRepresentation.restype = ctypes.c_void_p
CF.CFURLCopyResourcePropertyForKey.argtypes = [ctypes.c_void_p, ctypes.c_void_p,
                                               ctypes.POINTER(ctypes.c_void_p), ctypes.POINTER(ctypes.c_void_p)]
CF.CFURLCopyResourcePropertyForKey.restype = ctypes.c_bool
CF.CFNumberGetValue.argtypes = [ctypes.c_void_p, ctypes.c_int, ctypes.c_void_p]
CF.CFNumberGetValue.restype = ctypes.c_bool
CF.CFRelease.argtypes = [ctypes.c_void_p]


def important_usage_bytes(path):
    raw = os.fsencode(path)
    url = CF.CFURLCreateFromFileSystemRepresentation(None, raw, len(raw), True)
    key = ctypes.c_void_p.in_dll(CF, "kCFURLVolumeAvailableCapacityForImportantUsageKey")
    value, error = ctypes.c_void_p(), ctypes.c_void_p()
    try:
        if not CF.CFURLCopyResourcePropertyForKey(url, key, ctypes.byref(value), ctypes.byref(error)) or not value:
            return None
        number = ctypes.c_int64()
        CF.CFNumberGetValue(value, 4, ctypes.byref(number))   # 4 = kCFNumberSInt64Type
        return number.value
    finally:
        for ref in (value, error):
            if ref:
                CF.CFRelease(ref)
        CF.CFRelease(url)


home = os.path.expanduser("~")
free_now = shutil.disk_usage(home).free
available = important_usage_bytes(home)
print("free now (statfs, df):        %8.1f GB" % (free_now / 1e9))
if available is None:
    print("available (System Settings):  no answer from macOS")
else:
    print("available (System Settings):  %8.1f GB" % (max(free_now, available) / 1e9))
    print("purgeable macOS frees on demand: %5.1f GB" % (max(0, available - free_now) / 1e9))

External links

Exercise

Run the two-way free-space script on your Mac and note the gap between the two figures. Then, in the four-answers script from lesson 2, make the stale branch return the reason in a header-like field, and write two callers over it: one that accepts stale for display, and one that refuses it and says why. Finally, decide a floor for your own cache: what number, judged by which figure, and what the cache should do below it.

Hint

On a Mac with Photos or iCloud Drive optimization turned on, the purgeable share is often large. The disk script must run on macOS: it loads CoreFoundation through ctypes, which needs no third-party package.