War Story: The Root Migration

"Moving the ground everyone stands on is the scariest campaign there is. You do it by remaking the ground, never by editing it in place."

Remake, don't remaster

The fleet's second big campaign retired a legacy cloud-provider root — the folder every project lived under — and replaced it with an ordinary one. The temptation is to convert the old data in place: mutate it into the new shape as fast as possible. That's a remaster, and it's a trap. The migration instead remade the root from current ground truth: an authoritative host's live projects were the single source, and each peer's active project tree was rebuilt fresh from it rather than transformed from old bytes.

Writers validate the destination and fail closed

Before the cutover even began, a narrower guard landed everywhere: every local and cross-host writer had to validate its destination root by identity and fail closed if it was wrong — without ever recreating the old root. This guard migrated no data and granted no authority. Its entire job was to make sure a stray reboot or a background sync couldn't quietly spawn a second, split-brain root while the real migration was still being set up.

Visibility is not local bytes

A cloud file provider showing a file in Finder does not mean the bytes are on the disk. So a complete copy blocked on unexpected dataless objects rather than silently copying empty placeholders; the cloud's cheerful it's right here was never accepted as proof. Trusting visibility over materialization is exactly how a migration ends up with a directory full of confident, empty files.

Preserve the sources; a recreated old root is drift

The original provider and any split-brain roots stayed untouched, retained for a separate, later cleanup campaign — cutover approval explicitly did not include deletion. And if the old root reappeared afterward because some stale writer recreated it, that was recorded as observed drift to audit, never as a fallback the system was allowed to lean on. You keep the past around as evidence, and you treat its unexpected return as a signal, not a safety net.

Remake from truth, fence every writer, keep the old ground as evidence. The safest way to move the floor is to build a new one from what's true today — and to make absolutely sure nothing can quietly rebuild the old one behind you.

Code

Fail closed; never recreate the old root·python

def safe_write(path, data):
    dest_root = resolve_root(path)
    # Validate the destination is the NEW ordinary root, by IDENTITY (lstat),
    # not by name. If it's missing or wrong, STOP -- do not recreate the old one.
    if not is_expected_root(dest_root):        # lstat-based identity check
        fail_closed(f"destination root not verified: {dest_root}")
        return                                  # migrates no data; prevents split-brain
    write(path, data)

# A file provider showing a file is NOT proof the bytes are local:
if has_dataless_objects(source):
    block("unexpected dataless objects -- a complete copy must not proceed")

# Original provider / split-brain roots stay RETAINED for a later cleanup campaign.
# A recreated old root is observed DRIFT to audit, never a fallback to lean on.

Exercise

You need to move everything from an old base folder to a new one across several machines. Write the migration as a remake, not an in-place convert: name the authoritative source, how each writer proves it's targeting the new root, and what happens if the old root reappears. Then decide what you keep from the old location, and for how long.

Hint

The remake version never mutates the old data — it rebuilds the new tree from an authoritative source and leaves the old one retained as evidence. The reappearing old root is drift to audit, not a fallback to use; and a compatibility symlink is the exact convenience you should refuse, because it hides the references you still need to fix.

War Story: The Root Migration

Remake, don't remaster

Writers validate the destination and fail closed

Visibility is not local bytes

Preserve the sources; a recreated old root is drift

Code

External links

Exercise

Progress

Comments 0