Where the Check Stands

Four Greens That Depended on Where They Ran

The previous lessons built a check that compares the right bytes, renders templates before comparing, refuses placeholders, and runs from each consumer's own suite. In the weeks after, the family found four ways that check could be green and wrong at once. None was a bug in the comparison. Each was about where the check was standing when it looked.

An absent kit. The check skipped when the kit was not checked out beside the consumer, on the reasoning that no deploy could have happened there either. A review found the hole: a hand-edited copy passed green on exactly such a checkout, because a skipped check reports nothing. The owner chose a middle path. On a fleet Mac, where the kit is always expected, the same absence now fails and names the path where the kit should be; off the fleet — a cloud session, an archive restored years later — it skips and says why.
A worktree. A desktop session runs a consumer's suite in a git worktree nested inside the repository. The test found the kit by counting directories up from its own file, and from a nested worktree that count lands inside the repo, so a fleet Mac failed a suite on a kit that only looked absent. The other half was worse: the check, given a repository name, located that repository beside the kit — the main checkout — and reported its copies clean while the suite was running the worktree's copies. The fix asks git for the main checkout instead of counting, and passes the checkout under test explicitly, so the worktree's own copies are the ones judged.
A copy git ignores. A vendored file can sit on disk, match its source byte for byte, and never be committed, because an ignore rule swallows it. A global ignore rule for any directory named like a common script folder silently dropped a newly vendored helper from three repositories while their tracked install scripts called it. The check now reports a vendored copy that git ignores, by name, with the override to add.
A header that names the wrong commit. A copy's body can match while its provenance header names a commit whose copy of the source is not the one that body was rendered from. The check now warns about exactly that, and the deployer refuses to deploy any source that is not committed, so the header can only ever name a commit that holds what it delivered.

What the Four Have in Common

Each green described a different tree from the one that ships. The skip described a checkout with no kit in it. The worktree check described the main checkout. The ignored copy described a working directory that git would never commit. The stale header described a commit that never held the source of that body. In every case the comparison was correct about the thing it looked at, and the thing it looked at was not the thing that mattered.

So the check now has to answer four questions before its verdict means anything: which machine am I on, which checkout am I judging, which of these files will git actually keep, and does the commit each header names really hold the source its body was rendered from. Those are questions about vantage, and none of them is visible in a byte comparison.

A check is a claim about a specific tree; make it say which tree, and make it judge the one that ships. Comparing the right bytes is necessary and not sufficient. Before trusting a green, ask where the check stood — which machine, which checkout, which files the version-control system will keep — because a check that looks at the wrong tree is green about something nobody is going to run.

Do Not Fix an Honest Red

One consequence surprised people. A worktree whose vendored copies are simply older than the kit's — because its branch has not taken main since the kit last deployed — now fails. That is not a bug to route around. The worktree's tree is not what the kit ships, and a merge of main is what brings the newer copies. The failure message says which tree was compared and that a merge is the remedy. A check that could be made green by skipping this case would be back to describing a different tree from the one under test.

Code

The four vantage questions, asked before the verdict·python

import subprocess
from pathlib import Path


def git(*args: str, cwd: Path) -> str:
    out = subprocess.run(["git", "-C", str(cwd), *args],
                         capture_output=True, text=True)
    return out.stdout.strip() if out.returncode == 0 else ""


def main_checkout(checkout: Path) -> Path:
    """WHICH CHECKOUT. Ask git; never count directories. From a
    worktree nested inside the repo, counting lands inside the repo."""
    top, common = git("rev-parse", "--show-toplevel", "--git-common-dir",
                      cwd=checkout).splitlines() or ("", "")
    if Path(top) == checkout and Path(common).name == ".git":
        return (checkout / common).resolve().parent
    return checkout                     # no answer from git: depth rule


def ignored(checkout: Path, rel: str) -> bool:
    """WHICH FILES GIT KEEPS. On disk and identical is not committed."""
    return bool(git("check-ignore", rel, cwd=checkout))


def header_holds_body(kit: Path, source: str, sha: str) -> bool:
    """WHICH COMMIT. The source as committed at the header's sha must
    be the source this body was rendered from today. Compare SOURCE
    with SOURCE: the copy on disk is rendered (vars substituted, a
    header added), so comparing it with raw committed source would
    flag every template. No sha, or a git read that fails, is a NO -
    never an empty string that might happen to match. Compare exact
    text: whitespace in a source is content."""
    if not sha:
        return False
    out = subprocess.run(["git", "-C", str(kit), "show", f"{sha}:{source}"],
                         capture_output=True, text=True)
    if out.returncode != 0:
        return False
    return out.stdout == (kit / source).read_text()


def verdict(checkout: Path, kit: Path, on_fleet: bool) -> list[str]:
    """WHICH MACHINE first: absence is a defect on the fleet."""
    if not kit.is_dir():
        if on_fleet:
            return [f"FAIL: no kit beside {main_checkout(checkout)}"]
        return ["SKIP: off the fleet, drift not checked here"]
    problems = []
    for rel, body, source, sha in vendored_copies(checkout):  # THIS tree
        if ignored(checkout, rel):
            problems.append(f"IGNORED: {rel} is never committed")
        if body != expected(source):
            problems.append(f"DRIFT: {rel}")
        elif not header_holds_body(kit, source, sha):
            problems.append(f"STALE PROVENANCE (warning): {rel}")
    return problems

Exercise

Take one check in your own continuous integration or local test suite and answer the four vantage questions for it: on which machines does it run, which checkout does it judge, can any file it reads be ignored by version control, and does any provenance it reports point at a commit that holds what it describes? Then run it from one unusual place — a worktree, a fresh clone, a different machine — and see whether its verdict is about the tree you are in.

Hint

The fastest way to catch a vantage bug is to make the check print the absolute path of what it compared. If that path is not the directory you are standing in, you have found the class of bug this lesson is about.

Four Greens That Depended on Where They Ran

What the Four Have in Common

Do Not Fix an Honest Red

Code

External links

Exercise

Progress

Comments 0