Measured Against SSH

A Claim Needs a Table

"Our own service is better than SSH" is a claim, and a claim about speed is cheap until someone measures it. Delta shipped with a bench script, bench/compare.py, that runs the same pulls with Delta, rsync and scp against the same peer. It has two rules that make its numbers worth reading:

Every result is checked. Each copy is compared with its source by SHA-256, and the line that reports its time says verified or MISMATCH beside it. The script does not throw a mismatched time away; it brands it, so a fast tool that landed the wrong bytes is caught in the very row it would have won. The architecture document's pull rows come from these checked runs; its two push rows were timed through the CLI, outside this check.
Same wire for everyone. In the house, ssh dials the same address Delta's route chose, with the peer's trusted host key. On the tailnet, both go over the tailnet. For "away", both run through bench/away_proxy.py, a loopback proxy that gives every byte 40 Mbit/s each way and an 80 ms round trip, the way one hotel link would. The peer still sees the real caller, so neither tool's authentication changes.

The Results (2026-11-09)

One Mac pulling from another (bench/compare.py), plus two push rows run the other way through the CLI; for the first of them, all three tools dialed the same Thunderbolt address. "Small" is 4,000 files (74 MB). "Away" is a 128 MiB file, and separately 500 small files (9.5 MB).

Case	Delta	rsync	scp
LAN, 1 GiB (Delta on Thunderbolt)	0.34 s	4.36 s	1.73 s
LAN, 4,000 small files	1.98 s	1.15 s	2.16 s
tailnet, 1 GiB	19.65 s	23.05 s	22.84 s
tailnet, 4,000 small files	2.27 s	1.44 s	23.47 s
away, 128 MiB	28.98 s	28.14 s	28.56 s
away, 500 small files	2.37 s	3.23 s	187.8 s
resume: 1 GiB stopped at 560 MiB, run again	0.17 s for the rest	—	—
push, 1 GiB	1.03 s (1.49 s first, with the link probe)	2.67 s	1.56 s
push again, the file already in place	0.41 s	—	—

Reading It Honestly

rsync still wins on many small files over a fast link. The doc says so, and so does this lesson. rsync sends everything in one stream and does not fsync each file. Delta fsyncs every file it lands, because a file Delta reports as landed must survive a power cut. That is a deliberate cost, paid where it is visible.

Away from home, the link decides. For the 128 MiB file, all three tools take about 28 to 29 seconds, because 40 Mbit/s is the limit and every tool reaches it. The small-file case shows the cost of round trips instead. scp pays at least one round trip per file and takes 187.8 s. Delta's bundles take 2.37 s, ahead of rsync's 3.23 s.

The big wins are the fast wires and the parallel chunks. 0.34 s for 1 GiB is about 25 Gbit/s, which needs both the Thunderbolt route (Track 3) and several workers on one file (lesson 2).

And the case that started all of this is not in the table. Loomis's Eagle grid, 384 thumbnails read from another Mac: through per-item SSH, 152 failed, 32.6 s wall, p50 3.8 s. Through Delta, none failed, 1.94 s cold (p50 0.19 s, p99 0.30 s), and 1.67 to 1.88 s warm, with every tile a 304 and no new bytes.

The Second Measurement: Before and After a Review

A day later, bench/measure.py measured Delta against itself: one peer, both directions, with the daemon's CPU and peak memory beside each case. "Before" is build 24, "after" is build 30 on both ends.

Case (Thunderbolt)	before	after
push 4 GiB	1.65 s	1.23 s
pull 4 GiB, the source never hashed it	3.59 s	1.44 s
push 4,000 small files	2.70 s	1.11 s
list 4,040 entries	0.267 s	0.032 s
daemon peak memory in the run	616 MB	302 MB

Listing was the steady cost, because two background users list trees all day: the vault pair once a minute, and a site-assets check every five minutes. A listing cost 104 µs per entry, 62% of it a privacy guard asked once per entry and 23% a loop over refused names. After the review it cost 12 µs per entry. A content tree of 17,861 entries went from 1,865 ms to 221 ms.

Anything Per Piece Is Constant-Cost

The sharpest lesson came from a regression the review itself introduced. Lesson 3 made progress report per 256 KiB piece. At 20 Gbit/s that is about ten thousand reports a second. The job's rate window kept one entry per report and summed the whole window on every report. Right after the first rollout, a 4 GiB push over Thunderbolt took 4.06 s where it had taken 1.65 s. The test suite passed both before and after, because tests run on small files and slow paths. Only a measurement on a fast link showed it. The window is now a handful of tenth-of-a-second buckets, and a report adds to the newest bucket. The rule went into the house sign: anything called per piece or per entry must cost the same no matter how many came before, and you measure after a change, on a fast link.

Code

Delta's job: bytes counted per piece into tenth-of-a-second buckets (trimmed excerpt)·python

# delta/transfer.py (trimmed excerpt)
def note_bytes(self, n: int) -> None:
    """Count bytes as they move (``n`` is negative when a failed chunk takes its
    bytes back) and keep the rate over the last three seconds. The window is a few
    tenth-of-a-second buckets: bytes are now reported per piece, some ten thousand
    times a second on a fast link, and a window of one entry per report was summed
    whole on every report (a 4 GiB push went from 1.6 s to 4.1 s that way)."""
    with self._lock:
        now = time.monotonic()
        self.bytes_done += n
        window = self._rate_window
        if window and now - window[-1][0] < 0.1:
            window[-1][1] += n
            return
        window.append([now, n])
        while now - window[0][0] > 3.0:
            window.popleft()
        span = now - window[0][0]
        if span > 0.2:
            moved = sum(bucket[1] for bucket in window) - window[0][1]
            self.rate_bps = max(0.0, moved / span)

The same reports into a summed window and a bucketed one, timed (runnable)·python

import collections
import time

REPORTS = 15_000           # 1.5 s of a 20 Gbit/s push: one report per 256 KiB piece


class SummedWindow:
    """One entry per report, summed whole on every report (the slow shape)."""

    def __init__(self):
        self.window = collections.deque()

    def note(self, now, n):
        self.window.append((now, n))
        while now - self.window[0][0] > 3.0:
            self.window.popleft()
        return sum(b for _, b in self.window)


class BucketWindow:
    """Tenth-of-a-second buckets: a report adds to the newest bucket (Delta's shape)."""

    def __init__(self):
        self.window = collections.deque()

    def note(self, now, n):
        if self.window and now - self.window[-1][0] < 0.1:
            self.window[-1][1] += n
            return None
        self.window.append([now, n])
        while now - self.window[0][0] > 3.0:
            self.window.popleft()
        return sum(b for _, b in self.window)


for cls in (SummedWindow, BucketWindow):
    meter = cls()
    started = time.perf_counter()
    for i in range(REPORTS):
        meter.note(i / 10_000, 256 * 1024)        # simulated clock: 10,000 reports per second
    cost = time.perf_counter() - started
    print(f"{cls.__name__:>12}: {cost * 1e6 / REPORTS:7.2f} us per report, {cost:5.2f} s in all")

Exercise

Pick two machines you can reach, or one Mac and a second account on it. Copy a 1 GiB file and a folder of 2,000 small files with scp and with rsync, three runs each, and verify every result with shasum -a 256 before you record its time. Then repeat the small-file case with a 100 ms delay added (macOS: dnctl and pfctl, or simply over a phone hotspot). Write one paragraph saying which tool won each case and why, in terms of streams, round trips and fsync.

Hint

Run each case once to warm the caches and throw that time away. Expect scp's small-file time to grow roughly with file count multiplied by round-trip time, while rsync's barely moves. That difference is exactly what Delta's bundles were built to close.