Lesson 03 of 05 · published

A Test per Failure

~14 min · testing, pytest, xfail, integration-tests, benchmarking, regression

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Two Real Daemons on One Mac

Most of what can go wrong in Delta happens between two Macs: a ticket, a route, a body cut halfway, a partial resumed by a second request. Mocking the network away would test the parts that rarely break and skip the ones that do. So Delta's suite starts the real thing. tests/conftest.py launches two daemons, alpha and beta, each a real delta serve process on Apple's /usr/bin/python3, each with its own fake home folder, state folder, projects folder, fleet label and two free ports.

The trick that makes one Mac behave like two is in the environment. Each node's gateway is told that a loopback caller is the other node (DELTA_TEST_PEERS=127.0.0.1=<peer>), the route is forced to the tailnet kind, and the fleet is exactly the two labels. So alpha copying from beta:~/mixed exercises the same planner, the same gateway verbs, the same tailnet-route authentication and the same job runner as a copy between two Macs, without a second Mac, the LAN or SSH. Production sets none of these DELTA_* variables; they exist only so a test can describe a world.

The suite itself runs in the cwk-delta conda environment; only the daemons it starts run on Apple's python. That split still has a payoff: every daemon imports Delta on 3.9, so 3.10-only syntax anywhere in the package fails at start-up here, and a 3.10-only call fails on any path a test drives, in the test instead of in the LaunchAgent on some Mac a week later (Track 1).

Scripted Peers for What Healthy Daemons Cannot Do

Two healthy daemons on one Mac never produce a slow uplink, a connection cut inside a body, or a chunk that fails once and then succeeds. Those are exactly the failures Delta's transfer code exists to survive. So test_failure_modes.py has two halves. The first half drives alpha and beta the way a person or an app does. The second half exercises one part in-process against a scripted peer: a raw socket that promises 1,000 bytes in its Content-Length, sends 100 and closes; a receiver that takes a 32 MiB upload at 8 MiB/s, about four seconds, while the client's read timeout is one second, so a body that keeps moving must succeed even though it outlasts the timeout; and, for the hold-down, HTTPConnection.connect monkeypatched to time out. The second code block below is a scripted peer of the first kind in thirty lines, scaled down to 100 and 40.

Defect First, Fix Second, and Strict About It

On 2026-11-10 a code review (Coop 1110-7) found eighteen failure modes. The way they entered the repository is the part worth copying. Commit 0458b9a added only tests: one per defect, each one reproducing the defect on the code as it was, each marked

DEFECT = pytest.mark.xfail(strict=True, reason="a defect the 1110-7 review found; fixed in the commit after this one")

An xfail test that fails is reported as expected, so the suite stays green with every defect still in the code. strict=True is what makes it honest: if a marked test unexpectedly passes, strict mode fails the suite. So the mark cannot outlive the fix. The next commit, 22c331c, fixed the defects and removed the marks, and each test then had to pass on its own.

Why not write test and fix together? Because of where the tests run. Pippa's home Mac keeps a checkout of Delta that follows origin by itself, and that checkout is also the one her live Delta runs from. Tests run there from origin. Anything pushed soon becomes running code on a real Mac. So the rule is: a test that shows a defect goes in first as xfail(strict), the commit that fixes it removes the mark, and nothing half-finished is pushed "to see if it works there". The record of the defect is then a commit you can check out and watch fail.

Tests Named Like Sentences

Read the test names in that file and you have the failure catalogue:

test_one_unreadable_small_file_fails_by_name_and_its_neighbours_land
test_an_error_before_the_body_is_read_leaves_the_connection_usable
test_a_connection_cut_inside_a_body_is_named_not_read_as_a_short_answer
test_a_slow_uplink_does_not_fail_a_body_that_keeps_moving
test_a_mac_that_timed_out_is_not_dialed_again_at_once
test_the_logs_launchd_appends_to_are_cut_back_when_they_pass_their_size

Each name states the behaviour that must hold, not the function it calls. When one fails months later, its name tells you which promise broke before you open the file. Each test's docstring keeps the story of what used to happen ("the worker thread died on the cut, and the job said done with nothing landed").

Green Is Not Fast: Measure on a Fast Link

Tests prove behaviour. They do not prove cost. During the same review, a change that reported transferred bytes on every 256 KiB piece made a 4 GiB push two and a half times slower, because the rate window was summed whole on every report. The suite passed. The guard checked on every listed entry was 62% of a listing's time. The suite passed again. Only a measurement on a fast link (a Thunderbolt cable, where Delta itself is the bottleneck) showed either.

So Delta has a second instrument, bench/measure.py. Run on a Mac whose Delta is up, against one peer, it pushes and pulls a big file (cold, then again in place) and thousands of small ones, lists a tree five times, and makes 200 JSON round trips through the engine. Each pull that lands new bytes is compared with its source by SHA-256, and its row says verified or MISMATCH beside the time. Read that column: the script still prints a mismatched row's time, and the all-in-place re-runs, which land nothing, are not checked at all. Beside each case it prints this Mac's daemon CPU seconds and peak memory, sampled with ps while the case runs. That is how the before-and-after table in Track 6 was made, and why its memory column fell from about 600 MB to about 300 MB.

The rule from the house sign: anything called per piece or per entry is constant-cost, and you check it by measuring after the change, on a fast link, because a slow link hides your own overhead behind the wire.

Code

Delta's test harness: one Mac playing two (trimmed excerpt)·python

# tests/conftest.py (trimmed excerpt)
APPLE_PYTHON = "/usr/bin/python3"


class Node:
    def env(self, peer: "Node") -> dict:
        env = {k: v for k, v in os.environ.items() if not k.startswith(("DELTA_", "PYTHON"))}
        env.update({
            "HOME": str(self.home), "DELTA_LABEL": self.label, "DELTA_STATE": str(self.state),
            "DELTA_ENGINE_PORT": str(self.engine), "DELTA_GATEWAY_PORT": str(self.gateway),
            "DELTA_FLEET": f"{self.label},{peer.label}", "DELTA_PROJECTS": str(self.projects),
            "DELTA_HOST_CONFIG": "", "DELTA_TEST_PEERS": f"127.0.0.1={peer.label}",
            f"DELTA_HOST_{peer.label.upper()}": "127.0.0.1",
            f"DELTA_PEER_PORT_{peer.label.upper()}": str(peer.gateway),
            "DELTA_FORCE_ROUTE": "tailnet", "CWK_PORT_LEDGER": "", "CWK_HOST_PROFILE": "",
        })
        return env

    def start(self, peer: "Node") -> None:
        self.proc = subprocess.Popen([APPLE_PYTHON, "-B", "-s", "-m", "delta", "serve"], cwd=str(REPO),
                                     env=self.env(peer), stdout=subprocess.DEVNULL, stderr=subprocess.PIPE)
        deadline = time.monotonic() + 15
        while time.monotonic() < deadline:
            try:
                if self.get("/v1/health")[0] == 200:
                    return
            except OSError:
                time.sleep(0.1)
        raise RuntimeError(f"{self.label} did not start: {self.proc.stderr.read().decode()[-2000:]}")


@pytest.fixture(scope="module")
def pair(tmp_path_factory):
    if not os.access(APPLE_PYTHON, os.X_OK):
        pytest.skip("Apple's /usr/bin/python3 is not installed on this Mac")
    root = tmp_path_factory.mktemp("delta-pair")
    alpha = Node("alpha", root, "beta")
    beta = Node("beta", root, "alpha")
    beta.start(alpha)
    alpha.start(beta)
    yield alpha, beta
    alpha.stop()
    beta.stop()

Run it: a scripted peer, a defect shown as xfail(strict), and the fix beside it·python

# test_cut_body.py: a failure two healthy servers cannot produce, scripted.
# Run: python3 -m pytest -q test_cut_body.py
import http.client
import socket
import threading

import pytest

DEFECT = pytest.mark.xfail(strict=True, reason="shown here; fixed in the next commit")


def read_all(port):
    """The code under test: fetch one body from a peer."""
    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
    conn.request("GET", "/file")
    resp = conn.getresponse()
    return resp.read(int(resp.getheader("Content-Length")))   # read(n) never complains


def read_all_checked(port):
    """The fix: compare what arrived with what was promised."""
    conn = http.client.HTTPConnection("127.0.0.1", port, timeout=5)
    conn.request("GET", "/file")
    resp = conn.getresponse()
    promised = int(resp.getheader("Content-Length"))
    data = resp.read(promised)
    if len(data) != promised:
        raise ConnectionError(f"closed the connection mid-body at {len(data)} of {promised}")
    return data


@pytest.fixture
def cutting_peer():
    """A peer that promises 100 bytes, sends 40, and hangs up."""
    server = socket.socket()
    server.bind(("127.0.0.1", 0))
    server.listen(1)

    def serve():
        conn, _ = server.accept()
        conn.recv(65536)
        conn.sendall(b"HTTP/1.1 200 OK\r\nContent-Length: 100\r\n\r\n" + b"x" * 40)
        conn.close()

    threading.Thread(target=serve, daemon=True).start()
    yield server.getsockname()[1]
    server.close()


@DEFECT
def test_a_cut_inside_a_body_is_an_error_not_a_short_answer(cutting_peer):
    with pytest.raises(ConnectionError):
        read_all(cutting_peer)


def test_the_checked_reader_names_how_far_it_got(cutting_peer):
    with pytest.raises(ConnectionError, match="mid-body at 40 of 100"):
        read_all_checked(cutting_peer)

External links

Exercise

Take a bug you fixed recently in your own code. Check out the commit before the fix, write a test that reproduces it, mark it xfail(strict=True) and confirm the suite is green. Check out the fix and run again: the suite must now fail with XPASS(strict). Remove the mark and confirm it passes. Then write a scripted peer for one network failure your code claims to survive (a cut body, a stall, a refused connect) and test that claim.

Hint

git stash the test, git checkout <fix>^, git stash pop, run, then check out the fix again. For a stall, have the fake server send headers and a few bytes, then time.sleep longer than your reader's timeout; assert the error names the phase it stopped in, not just "timed out".