What Accumulates Is Bounded

A Service That Never Restarts Meets Every Slow Leak

A command-line tool forgets everything when it exits. A LaunchAgent with KeepAlive exits only when it crashes or the Mac restarts, so anything it keeps "for a while" it keeps for weeks. A dictionary that gains one key per request, a table that gains one row per job, a log nobody rotates: each is harmless in a test and a problem on day forty. Delta's architecture names this as an invariant of its own, number 10: what accumulates is bounded. It is not a general wish. It is a list, and every item on the list has a number and a place in the code that enforces it.

The List

What grows	Bound	Who lets go
Finished jobs held in memory (what `/v1/jobs` lists)	200	`JobRunner._remember` drops the oldest finished ones as each new job arrives; running ones are never dropped
Job rows in the database	30 days	the sweep deletes rows older than that that are not running or planning
Hash records (one per file version a peer asked about)	45 days unused	`Hasher.prune`; a record still in use has its last-used time refreshed at most weekly, which keeps it out of the sweep
Partials (resumable receives)	14 days untouched	`partial.sweep`, at start and every six hours; a live partial is touched by every chunk
Per-key locks (one per file version being hashed or derived)	dropped after use	`KeyedLocks` removes a key's lock when its last holder or waiter leaves
The content cache	20 GiB, 512 MiB per item, and the family disk floor (50 GiB available)	`ContentCache`; under the floor it stops storing and serves through
Derived outputs	4 GiB	`Derived`, least recently used first out
`delta.log`	20 MiB, five backups	Python's `RotatingFileHandler`
`launchd.out.log`, `launchd.err.log`	5 MiB, then cut back to empty	`bound_launchd_logs`, at start and every six hours

The cache, the floor and the derived cap are per-Mac settings in ~/.config/cwk/delta.json; the rest are constants. A host config with an unknown key or a non-positive size is refused at start, never run on a guessed default.

Two Leaks the Review Found

The 1110-7 code review (2026-11-10) found two of these the hard way, and both are worth recognising because they look correct.

The lock dictionary. The hasher computes a file's SHA-256 once per version and keeps the answer, and it must not compute the same version twice at once, so it took a lock per version key. The first version kept those locks in a plain dict and removed one only on the path that actually computed something. Every request answered from the record left its lock behind: one lock per file version, for the life of the daemon. The fix is a lock with a reference count: hold(key) increments under a guard, and the last one out removes the entry. The second code block below shows both shapes side by side, and the difference after ten thousand versions.

The job list. /v1/jobs answered from an in-memory dict of every job since start, so it grew by one entry for every distinct copy for as long as the daemon ran. Every job is also in the database, so memory only needs the recent ones: the runner now keeps at most 200 finished jobs, oldest out first, and never lets go of one that is still running. The test for it submits 250 finished jobs plus a running one and checks that the running one and the newest finished one are still there and the oldest is gone.

The Two Files Nobody Rotates

Delta's own log is easy: a rotating handler. The hard case is the pair of files launchd writes for it. StandardOutPath and StandardErrorPath are opened by launchd, in append mode, before Python starts, and launchd never rotates them. Every warning and every crash lands there too (the stderr handler is set to WARNING on purpose, so a crash is visible without the rotating log).

You cannot rename a file another process holds open and expect it to follow; launchd would keep writing to the renamed file. What you can do is cut it in place. Past 5 MiB, bound_launchd_logs truncates the file to zero bytes. Because launchd opened it with O_APPEND, its next write goes to the current end of the file, which is now zero: the file simply starts again. Nothing is lost that matters, because the same lines are in delta.log, which rotates. The test reproduces exactly this: it holds the file open in append mode the way launchd does, truncates it, writes one more line, and checks that the file now holds only that line.

Health Shows What Is Waiting

A bound you cannot see is a bound you cannot trust. The full health document reports partials as a count and the bytes they hold, the cache's entries and bytes against its cap and floor, the derived outputs, and the jobs summary. So "is something piling up on this Mac?" is one request to its engine, not a walk through its state folder. When the six-hourly sweep removes something it logs what it removed and how much space it freed ("removed 3 partial(s) untouched for 14 days"), and when nothing needed removing it says nothing.

How to Find Your Own Unbounded Piles

Read every self.something = {}, [] and set() on a long-lived object and ask one question: what removes an entry? If the answer is "nothing" or "only on the success path", that is a leak with a delay. Do the same for every table that gains rows and every file that gains lines. Then give each one a number and a place that enforces it, and put the number in the health document.

Code

Delta: the two launchd logs cut back in place (trimmed excerpt)·python

# delta/server.py (trimmed excerpt)
LAUNCHD_LOG_MAX = 5 * 1024 * 1024


def bound_launchd_logs(logs: Path) -> None:
    """launchd appends this process's stdout and stderr (every warning, every crash)
    to two files nothing rotates. Past 5 MiB one is cut back to empty: launchd
    opened it for appending, so what it writes next starts the file again. The
    same lines are in delta.log, which rotates."""
    for name in ("launchd.err.log", "launchd.out.log"):
        path = logs / name
        try:
            if path.stat().st_size > LAUNCHD_LOG_MAX:
                os.truncate(path, 0)
                log.info("%s passed %d MiB and was emptied (delta.log keeps the same lines)", name,
                         LAUNCHD_LOG_MAX // (1024 * 1024))
        except OSError:
            pass


# in serve(): at start, then a few times a day
swept = time.monotonic()
while not stop.wait(1.0):
    if time.monotonic() - swept > 6 * 3600:  # abandoned partials and the like
        swept = time.monotonic()
        delta.jobs.sweep_partials()
        bound_launchd_logs(delta.state / "logs")

Run it: a lock dict that leaks, one that lets go, and a held-open log cut in place·python

# bounded.py: two piles that grow for ever unless something lets go.
import contextlib, os, tempfile, threading
from pathlib import Path


class KeyedLocks:
    """One lock per key while someone holds or waits for it, and none afterwards."""

    def __init__(self):
        self._locks = {}
        self._guard = threading.Lock()

    @contextlib.contextmanager
    def hold(self, key):
        with self._guard:
            entry = self._locks.setdefault(key, [threading.Lock(), 0])
            entry[1] += 1
        try:
            with entry[0]:
                yield
        finally:
            with self._guard:
                entry[1] -= 1
                if not entry[1]:
                    self._locks.pop(key, None)


class LeakyLocks:
    """The first shape: a dict that only ever gains keys."""

    def __init__(self):
        self._locks = {}

    @contextlib.contextmanager
    def hold(self, key):
        with self._locks.setdefault(key, threading.Lock()):
            yield


for cls in (LeakyLocks, KeyedLocks):
    locks = cls()
    for version in range(10000):          # one key per file version, as a hasher sees them
        with locks.hold(("file.bin", version)):
            pass
    print(f"{cls.__name__:10} holds {len(locks._locks):5} locks after 10,000 versions")

# A log launchd appends to: nobody rotates it, so cut it back while it is held open.
LIMIT = 1024
with tempfile.TemporaryDirectory() as tmp:
    log = Path(tmp, "launchd.err.log")
    log.write_bytes(b"w" * (LIMIT + 1))
    with open(log, "ab") as held:          # the way launchd holds it
        if log.stat().st_size > LIMIT:
            os.truncate(log, 0)
        held.write(b"the next warning\n")
    print("after the cut, the file holds:", log.read_bytes())

Exercise

Pick a long-running service you own (a web app, a bot, a daemon). List every container on a long-lived object, every table that gains rows and every file that gains lines. For each, write down what removes an entry, or "nothing". Pick the worst one, add a bound with a number, and write a test that pushes it past the bound in a loop and asserts the size stays at or under it.

Hint

Search for = {}, = [], = set() and .append( inside classes, and for INSERT without a matching DELETE. A cache keyed by request input (a path, a user, a version) is the usual suspect. The test does not need to wait weeks: a loop of ten thousand iterations reproduces a month of traffic in a second.