Commit Before the Last Byte

A Bug That Only Showed Up When Someone Was Fast

Here is a sequence that looks correct line by line. The engine receives a file from its owner. It streams the bytes to the app that asked. When the last byte has gone out, it finishes the bookkeeping: closes the temporary blob, hashes it, writes the index row. The app got its file, the cache got its entry. Nothing is wrong.

Now add a caller that is quick. The moment its response ends, it asks for the same file again, or reads the health page to check the counters. For a few milliseconds, the app has the whole body and the cache does not yet have the entry. The second request finds nothing, and becomes a second miss: another full transfer of bytes that were sitting in a temporary file one function call away from being kept. The health page, read at the same instant, shows a miss count one lower than the truth.

It was intermittent, as races are: whether the second request lost depended on a few milliseconds of scheduling, so the same sequence was fine one time and a second miss the next. Nothing in a single response looked wrong, because nothing in a single response was wrong. The defect was in the order of two steps that each worked. Delta's house sign records the rule that came out of it, in two parts: count a cache outcome when it is decided, and land a small body in the cache before its last byte goes out.

Count When Decided

The first part is about the counters. The moment the engine knows this read is a miss (the owner sent a 200, it has a validator, it fits the cap, and the disk floor allows storing), it increments misses, before any byte of the body moves. Anyone who reads health after the response began sees a count that already includes it. Hits, revalidations and stale serves are counted the same way, at the instant they are chosen, inside touch(), before the blob is sent.

Commit Before the Last Byte, for Small Bodies

The second part changes the order of work for every body up to 4 MiB, which covers every thumbnail and nearly every small file an app reads: read the whole body from the owner first, write it into the blob, commit the blob and the index row, and only then send the headers and the body to the caller. By the time the caller can possibly know its read is over, the entry already exists. Its next request is a revalidation or a hit, never a second miss.

The same order brings a second guarantee for free. Because the body is read whole before a single header goes out, a connection that the owner cuts halfway through is discovered while the engine can still choose what to answer. If an older copy is cached, the caller gets it labeled stale; otherwise it gets the named cause (the peer client reports exactly how far the body got). It never receives a 200 with a short body.

Streaming for Large Bodies, and Why

Larger bodies, up to the 512 MiB per-entry cap, cannot be held whole in memory per request without making the daemon's memory a function of how many big files are in flight. They stream: each block is written to the caller and into the blob writer as it passes. The commit happens after the last block, so the race exists for them in principle, but large files are not what a quick caller re-reads in a loop, and the trade is bounded memory. The stream has its own checks. A body that grows past the per-entry cap (512 MiB by default) overflows the blob writer; it is still served through to the caller, but not kept. (Delta's own comment on that line says "larger than it said", but the writer counts against the cap, and http.client never hands over more than the declared length anyway.) A body that ends short of its declared length is not committed at all; the blob writer is aborted and its temporary file removed. A cache entry is only ever a complete body.

Body	Order	If the owner cuts it short
≤ 4 MiB	read whole → commit → answer	stale copy or named cause; no header was sent
4 MiB to 512 MiB	stream to caller and blob → commit	caller's client sees the short body as an error; nothing kept
> 512 MiB, no validator, or below the floor	stream through (`pass`)	caller's client sees the short body as an error

The General Shape

Whenever an operation has a visible end (a response finishing, a function returning, a message being acknowledged), any state that a follow-up action depends on must be in place before that end becomes visible. Bookkeeping after the signal is a race with whoever trusts the signal. The same reasoning shows up again in this quest wherever a record and a signal meet: a transfer's chunk is recorded only after its bytes are verified and on disk, and a write call is answered only after its owner has committed it.

Code

Delta's engine: count when decided, commit small bodies before answering, stream big ones (trimmed excerpt)·python

# delta/proxy.py: a 200 from the owner, kept and served (trimmed excerpt)
SMALL_BODY = 4 * 1024 * 1024


def proxy_fs(h, peer, tail):
    ...
    writer = cache.begin(key) if (cacheable and etag and size <= cache.item_cap) else None
    out_headers["X-Delta-Cache"] = "miss" if writer else "pass"
    if writer:
        cache.count("misses")  # counted when decided, before a caller can ask the health page
    if writer and size <= SMALL_BODY:
        # A small body (every thumbnail) lands in the cache before the caller sees
        # its last byte, so the caller's next request finds it: no second miss. A
        # body cut short raises here, before any header: the caller gets the named
        # cause (or the stale copy), never a short answer.
        try:
            body = resp.read()
        except (Unavailable, Timeout) as exc:
            writer.abort()
            if hit:
                cache.touch(hit, "stale_served")
                _serve_hit(h, hit, "stale", peer, resp.route.kind, stale=exc.detail)
                return
            raise
        try:
            writer.write(body)
            cache.commit(writer, **meta)
        except OverflowError:
            writer.abort()
        h.send_head(resp.status, out_headers)
        h.write(body)
        return
    h.send_head(resp.status, out_headers)          # a big body streams through
    for block in resp.iter_chunks():
        if writer:
            try:
                writer.write(block)
            except OverflowError:                  # larger than it said: served, not kept
                writer.abort()
                writer = None
        h.write(block)
    if writer:
        if writer.size != size:                    # short of its own length: not kept
            writer.abort()
            return
        cache.commit(writer, **meta)

The second miss, reproduced and removed: the same caller, two commit orders·python

# The second miss, reproduced: a caller that asks again the instant its first
# answer ends races a cache commit made after the body went out.
import threading
import time


class Cache:
    def __init__(self):
        self.entries = {}
        self.counts = {"miss": 0, "hit": 0}
        self.lock = threading.Lock()

    def lookup(self, key):
        with self.lock:
            return self.entries.get(key)

    def commit(self, key, body):
        with self.lock:
            self.entries[key] = body


def fetch_from_owner():
    time.sleep(0.01)          # the network
    return b"thumbnail bytes"


def serve(cache, key, reply, commit_first):
    hit = cache.lookup(key)
    if hit is not None:
        cache.counts["hit"] += 1
        reply(hit)
        return
    cache.counts["miss"] += 1          # counted when decided
    body = fetch_from_owner()
    if commit_first:                   # small body: land it, then answer
        cache.commit(key, body)
        reply(body)
    else:                              # answer first, then land it
        reply(body)
        time.sleep(0.005)              # bookkeeping after the last byte
        cache.commit(key, body)


def caller(cache, commit_first):
    done = threading.Event()
    worker = threading.Thread(target=serve, args=(cache, "k", lambda b: done.set(), commit_first))
    worker.start()
    done.wait()                        # our last byte arrived: ask again at once
    serve(cache, "k", lambda b: None, commit_first)
    worker.join()


for commit_first in (False, True):
    cache = Cache()
    caller(cache, commit_first)
    label = "commit before the last byte" if commit_first else "commit after the last byte "
    print(label, cache.counts)

Exercise

Run the race script and confirm both lines. Then combine it with the four-answers script from lesson 2: make the owner's file 8 MiB, keep the commit-after order for bodies above 4 MiB, and add a quick second read. Measure how often the second read is a second miss over 50 runs, then decide whether the memory you would need to buffer the body is worth removing that window for your workload.

Hint

The window for a streamed body is the time between the last block written to the caller and the commit: a flush, an fsync, a rename and one SQLite transaction. On a fast SSD that is short but not zero, and it grows when the disk is busy.