Lesson 01 of 05 · published

Bytes Named by Their Hash

~13 min · content-addressed-storage, sha256, cache, sqlite, atomic-rename

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Why the Receiving Side Keeps Anything at All

Track 2 was about what a Mac serves. This track is about the other end: the Mac that asks. When an app on the service host draws a grid of thumbnails that live on the workbench, the same few hundred files get asked for again every time the grid is opened, scrolled or refreshed. Sending them across the house each time is wasteful, and sending them across the tailnet from a café is slow. So every Delta keeps what it has already received. Not as a feature an app can forget to turn on, but as a part of the engine every read passes through: the cache sits in front of every GET /v1/peer/<mac>/fs/<share>/<path> an app makes on loopback.

The interesting decisions are not whether to cache. They are where the bytes live, what they are called, and how the receiving side ever learns that they are out of date. This lesson is the first two. The rest of the track is the third.

A Blob Is Named by What It Contains

A file that crossed the network once is kept as a blob: a plain file whose name is the SHA-256 of its bytes. It lives under Delta's own state root, at ~/cwk-dbs/delta-db/cache/blobs/<first two hex digits>/<the whole digest>. The two-character fan-out folder is the same trick Git uses for its object store: it keeps any one directory from holding hundreds of thousands of entries.

Naming by content buys three things at once:

Duplicates cost nothing. The same logo saved in two folders, or on two Macs, is one blob with two references.
A blob is verifiable. Hash the file, compare with its name. A blob whose bytes do not match its name is not a blob.
A blob never changes. New bytes have a new name, so an entry that moves on simply points somewhere else. Nothing is ever rewritten in place, which means no reader can see half of an update.

The Index Says Which Blob Answers Which Question

The blob store alone cannot answer "what is Pictures/logo.png on the workbench?". That is the job of an index row in Delta's SQLite file, table cache_entries. Each row is keyed by a hash of four things: the peer, the share-and-path the app asked for, the verb (fs for files, api for an app's read API), and a variant, the sorted query parameters that change what the answer is (a thumbnail size, a derived format). Parameters that only steer the request (expect, cache, app, force) are left out of the variant, so they never split one file into two entries.

Beside the blob's SHA-256, the row records the validator the owner sent with those bytes (its ETag), the size, the content type, the display name, the source's modification time, the app that asked, when it was stored, when it was last read, and how often. A second table, cache_blobs, counts references per blob. When the last row pointing at a blob moves on or is evicted, the blob file is deleted inside that same transaction.

Piece	Where	Named by
Bytes	`cache/blobs/ab/ab12…`	SHA-256 of the bytes
Which bytes answer a read	`cache_entries` row	hash of (peer, path, verb, variant)
How many rows share a blob	`cache_blobs.refs`	SHA-256
A blob still arriving	`cache/tmp/in-…`	a temporary name

Arriving, Then Landing

A blob is never written under its final name, because its final name is not known until its last byte has been hashed. A Writer opens a temporary file in cache/tmp, and every block that streams through is written, fed into a running SHA-256 and counted. When the body is complete, commit flushes and fsyncs the file, takes the digest, and moves the file into place with os.replace, an atomic rename on the same volume. If that blob already exists, the temporary file is just deleted: those bytes were already kept. A crash in the middle leaves only a temporary file, and a starting Delta empties cache/tmp before it does anything else.

A writer also stops on size. Each entry has a cap of 512 MiB; a body larger than that is still served to the app, but it is passed through rather than kept, and counted under skipped_large.

Never Inside the Data It Caches

The cache lives under Delta's state root and nowhere else. It is never a hidden folder beside the files it mirrors, never a copy dropped into a user's project, never written into the share it came from. Two reasons. The data folders belong to people; a tool that leaves its own debris in them gets turned off. And the cache is derived: every byte in it can be fetched again from its owner, so it may be cleared at any time (all of it, or one app's entries with delta cache --clear --app <name>) and nothing is lost. The architecture's ownership table says it in one row: the receiving cache is derived, revalidated against the owner's validator, and disposable.

Code

Delta's cache: the key, the blob path and the commit (trimmed excerpt)·python

# delta/cache.py: the cache key, where a blob lives, and the commit (trimmed excerpt)

@staticmethod
def key(peer: str, share: str, path: str, variant: str = "") -> str:
    return hashlib.sha256(json.dumps([peer, share, path, variant]).encode()).hexdigest()

def _blob_path(self, sha: str) -> Path:
    return self.blobs / sha[:2] / sha

def commit(self, writer, *, peer, share, path, app, etag, content_type, name, mtime_ns) -> str:
    writer.handle.flush()
    os.fsync(writer.handle.fileno())
    writer.handle.close()
    sha = writer.digest.hexdigest()          # hashed while it streamed in
    blob = self._blob_path(sha)
    blob.parent.mkdir(parents=True, exist_ok=True)
    if blob.exists():
        writer.path.unlink()                 # these bytes are already kept
    else:
        os.replace(writer.path, blob)
    with self.store.transaction() as db:
        old = db.execute("SELECT sha256 FROM cache_entries WHERE key=?", (writer.key,)).fetchone()
        db.execute("INSERT OR REPLACE INTO cache_entries(key, peer, share, path, app, etag, sha256, ...) "
                   "VALUES (...)", (writer.key, peer, share, path, app, etag, sha, ...))
        if not (old and old[0] == sha):
            db.execute("INSERT INTO cache_blobs(sha256, size, refs) VALUES (?,?,1) "
                       "ON CONFLICT(sha256) DO UPDATE SET refs=refs+1", (sha, writer.size))
            if old:
                self._unref(db, old[0])      # the last reference deletes the blob
    self._evict()
    return sha

A content-addressed blob store you can run: duplicates kept once, atomic landing, reference counts·python

# A content-addressed blob store in forty lines: bytes are named by their SHA-256,
# an index maps a logical key to a blob, and identical bytes are stored once.
import hashlib
import json
import os
import tempfile
from pathlib import Path


class BlobStore:
    def __init__(self, root):
        self.blobs = Path(root) / "blobs"
        self.tmp = Path(root) / "tmp"
        self.blobs.mkdir(parents=True, exist_ok=True)
        self.tmp.mkdir(parents=True, exist_ok=True)
        self.index = {}   # logical key -> sha256
        self.refs = {}    # sha256 -> how many keys point at it

    @staticmethod
    def key(peer, share, path, variant=""):
        return hashlib.sha256(json.dumps([peer, share, path, variant]).encode()).hexdigest()

    def blob_path(self, sha):
        return self.blobs / sha[:2] / sha

    def put(self, key, data):
        fd, name = tempfile.mkstemp(dir=str(self.tmp), prefix="in-")
        with os.fdopen(fd, "wb") as handle:
            handle.write(data)
            handle.flush()
            os.fsync(handle.fileno())
        sha = hashlib.sha256(data).hexdigest()
        blob = self.blob_path(sha)
        blob.parent.mkdir(parents=True, exist_ok=True)
        if blob.exists():
            os.unlink(name)          # the same bytes are already here
        else:
            os.replace(name, blob)   # atomic: a reader sees all of it or none
        old = self.index.get(key)
        if old != sha:
            self.refs[sha] = self.refs.get(sha, 0) + 1
            if old:
                self.unref(old)
        self.index[key] = sha
        return sha

    def unref(self, sha):
        self.refs[sha] -= 1
        if self.refs[sha] <= 0:
            del self.refs[sha]
            self.blob_path(sha).unlink()

    def get(self, key):
        sha = self.index.get(key)
        return self.blob_path(sha).read_bytes() if sha else None


if __name__ == "__main__":
    store = BlobStore(tempfile.mkdtemp(prefix="cas-"))
    logo = b"\x89PNG pretend this is a logo"
    a = store.put(store.key("peer-a", "home", "Pictures/logo.png"), logo)
    b = store.put(store.key("peer-b", "home", "Desktop/logo copy.png"), logo)
    print("same bytes, same blob:", a == b, "| blobs on disk:", sum(1 for p in store.blobs.rglob("*") if p.is_file()))
    store.put(store.key("peer-a", "home", "Pictures/logo.png"), b"a new version")
    print("refs after one key moved on:", sorted(store.refs.values()))
    print("read back:", store.get(store.key("peer-b", "home", "Desktop/logo copy.png"))[:12])

External links

Exercise

Run the blob store above, then extend it: give BlobStore a verify() method that walks every blob, hashes it, and reports any whose bytes no longer match their name. Corrupt one blob by hand (append a byte with echo x >>) and confirm verify() names it. Then add a size cap with least-recently-read eviction and decide what get() should do when the index points at a blob that is gone.

Hint

Delta's answer to the last question is in lookup(): when the row's blob file does not exist, the row is dropped and the read is treated as a miss. A missing blob is never an error the app sees, because everything in a cache can be fetched again.

Progress

Progress is local-only — sign in to sync across devices.

← 🪪 Identity Without Copied SecretsQuiz · 5 questions Next →Four Answers to One Read

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.