Whose Suite Runs It

Where the Failure Has to Land

Picture the moment the whole mechanism exists to survive. Someone is deep in a consumer repository, chasing a bug, and the fix is one line inside a vendored file sitting right there in the editor. The header says do not edit here. They have already stopped reading headers, because they are three hours into a bug.

A check that runs only in the shared repository reports this later, to somebody else, about a repository the editor was not looking at. A check that runs in the consumer's own test suite goes red for them, in the repository they are already in, with a message naming the file to edit instead. Same check, same script, completely different outcome — because the correction arrives while the context is still loaded.

So each consumer runs the drift check from its own suite. It shells out to the shared repository's script in check mode, scoped to that consumer, and asserts a zero exit.

The Rule Was Written as a Path, and That Was the Bug

The convention had been recorded like this: every consumer has a test file at a particular path under its backend directory. It worked for every consumer that had a backend directory.

Then one joined that is frontend-only. It vendored shared files for a week with no check at all. Nothing failed, because a test that does not exist cannot fail — and there was no inventory anywhere asking whether each consumer's check was present. The repository quietly complied with nothing while looking exactly like a compliant member of the family.

The correction was not to give that repository a backend. It was to notice that the rule's real content had been mis-stated. The requirement is every consumer enforces the check; the path was only ever an implementation detail of how the majority happened to do it. The frontend-only consumer now runs the same check from its own JavaScript test runner. Same assertion, different host.

State the requirement, not the implementation — and be most suspicious when the implementation is a file path. A rule expressed as a location silently exempts anyone whose structure cannot host that location, and the exemption is invisible because it manifests as an absence. When a new participant joins, the right question is never "can you host this file" but "which of your existing mechanisms will enforce this".

Verify That the Check Can Fail

When the frontend-only consumer's check was added, it was not merely observed to pass. A vendored file was deliberately edited, the suite was run, and the failure was confirmed — then the edit was reverted.

That step is easy to skip and it is the only thing separating a real check from a decorative one. A green test proves nothing about a check's sensitivity: a check with an inverted condition, a wrong path, a swallowed exception, or a filter that matches nothing all pass beautifully forever. The first thing a new guard should do is fail on purpose.

Code

The same assertion, hosted by whichever suite the consumer has·python

# --- Consumer with a Python suite --------------------------------
import subprocess, sys
from pathlib import Path
import pytest

KIT_ROOT = Path(__file__).resolve().parents[2] / "the-kit"
SYNC = KIT_ROOT / "sync" / "kit_sync.py"


@pytest.mark.skipif(not SYNC.is_file(), reason="kit not present")
def test_vendored_files_match_the_kit() -> None:
    result = subprocess.run(
        [sys.executable, str(SYNC), "--check", "--repo", "this-app"],
        capture_output=True, text=True, timeout=30,
    )
    assert result.returncode == 0, (
        "drift detected - edit the kit, not the vendored copy:\n"
        + result.stdout + result.stderr
    )


# --- Consumer with only a JS suite -------------------------------
# Same script, same flags, same assertion. The rule is "a suite runs
# this", and it never cared which language that suite is written in.
#
#   import { execFileSync } from "node:child_process";
#   import { existsSync } from "node:fs";
#   import { test, expect } from "vitest";
#
#   const SYNC = "../the-kit/sync/kit_sync.py";
#
#   test.skipIf(!existsSync(SYNC))("vendored files match", () => {
#     expect(() =>
#       execFileSync("python3", [SYNC, "--check", "--repo", "this-app"])
#     ).not.toThrow();
#   });


# --- The inventory that would have caught the gap ------------------
# One assertion in the KIT's own suite: every repo that appears as a
# deploy target must also appear in the enforcement registry. This is
# the check on the checks, and it is what an absence-shaped failure
# needs, because absences never raise on their own.
def test_every_target_repo_enforces_the_check(manifest, enforcement):
    targets = {repo for e in manifest["files"] for repo in e["targets"]}
    for repo in sorted(targets):
        assert repo in enforcement, (
            f"{repo} receives vendored files but no suite checks them"
        )

Exercise

Find a rule in your team's documentation that is phrased as a location — a file that must exist at a path, a directory every service must have, a config key in a specific place. Ask who cannot comply with it structurally. Then rewrite it as the requirement it was trying to express, and add one enumeration that lists who is supposed to satisfy it and checks that they do.

Hint

Look at the newest participant first, and the most structurally unusual one second. The rule was almost certainly written when everyone had the same shape, and the exemption arrived silently with the first member that did not.

Where the Failure Has to Land

The Rule Was Written as a Path, and That Was the Bug

Verify That the Check Can Fail

Code

External links

Exercise

Progress

Comments 0