Lesson 06 of 07 · published

Links, Patterns and Folders That Will Not Open

~14 min · symlinks, include-exclude, rsync, delete-safety, create-only

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

The Edges of a Tree

Copying a folder of regular files is the easy case. Real trees have edges: symbolic links, files you want and files you do not, a destination where a name is already taken, and folders that refuse to open. Each edge has a tempting shortcut, and each shortcut has damaged someone's data with some tool. Delta takes rsync's answers where rsync had good ones, and the code says so in its comments.

Links Cross as Links

A symbolic link is copied as a link, never followed into a copy of its target. Its target may be anything that stays inside the share and that the share would serve. node_modules/.bin/tool pointing at ../pkg/bin/tool.js crosses intact, so a JavaScript project still works on the other Mac. A link whose target leaves the share, or leads into a refused place (a credential folder, a folder macOS guards), is not copied. It is left out, listed in the job's left_out as name -> target, and the job goes on. This is rsync's --safe-links. The final message says how many links were left out and names the first.

Two follow-ups came from the review. The first version refused the whole job on one relative link with .. in it, so every node_modules tree failed. And a link whose target had changed was never updated. Now a changed target replaces the old link on the next sync. A link is also deleted and set aside as what it is, a link, and is never followed or written through.

Patterns, Read the Way rsync Reads Them

--exclude skips names anywhere in the tree:

a bare pattern (*.tmp) matches a file or a folder by name at any depth, and a folder takes its whole subtree with it;
a path tail (.editor/cache) matches those trailing parts at any depth;
a trailing slash (.locks/) matches folders only, so a file named .locks still moves;
a leading slash (/session_index.jsonl) matches only at the transfer's top level.

--include keeps only the files that match, like rsync's --include=*/ --include=<p> --exclude=* --prune-empty-dirs. A bare pattern matches a file's name, a pattern with a slash matches the same number of trailing path parts (evidence/*), and a leading slash anchors it at the root. Folders that end up holding no wanted file are not created.

Two rules make patterns safe with --delete. Excluded paths neither move nor count for deletion, on either side, so excluding .locks/ never deletes the destination's lock folder. And files outside an include set on the destination are not deleted either. When both are given, an excluded name never moves, even if it also matches an include. Each list takes at most 50 patterns. An exclude pattern with an empty, . or .. part is refused; an include pattern only has to be a non-empty string.

Create-Only Delivery

delta cp --unique never replaces anything. The file takes its own name if that name is free, otherwise name (2), then name (3), up to name (50). The job reports the name it took in landed. The landing is a hard link from the verified data file to the candidate name. os.link fails if the name exists, so "is it free?" and "take it" are one atomic step and two deliveries can never claim the same name. This is the shape for an inbox: things arrive, and nothing that was already there is overwritten.

Unreadable Is Not Gone

This is the edge with the sharpest teeth. In the first version, the listing skipped a folder it could not open and logged a warning. Think through a sync --delete with that listing. The source folder is there, but a permission problem makes it look absent. The planner sees the destination's copy of that folder, with no counterpart in the source, and deletes it. A permission error would have erased a mirror.

The fix borrows rsync's rule ("IO error encountered -- skipping file deletion") and makes it explicit:

a folder that will not open is listed with the kind unreadable, never as an empty folder and never silently skipped;
nothing under it moves, in either direction;
a run that met any unreadable folder, on either side, deletes nothing at all;
the job fails, naming the folder, while everything else lands;
a transfer root that will not open fails the plan before anything moves.

The sync plan shows it before you confirm: UNREADABLE: <where> (nothing under it moves, and this run deletes nothing). The principle generalizes to any tool that compares two listings. "I could not look" is a third answer, separate from "present" and "absent", and it must never collapse into either.

The Plan Comes First

All of these rules land in the sync plan, and the plan is shown before anything changes: files to send and their size, files already in place, paths to delete, file-versus-folder conflicts, unreadable folders, and an estimate when there is enough to measure. --dry-run prints the plan and stops. --yes skips the question, and without a terminal to ask, the CLI refuses to proceed unless --yes was given. Deleting on the destination happens only with --delete. Nothing in Delta infers it.

Code

Delta's include and exclude rules, with checks you can run (from delta/transfer.py)·python

import fnmatch


def included(rel, pattern):
    """rsync's include match for a file (Delta's rule, copied whole)."""
    if pattern.startswith("/"):
        return fnmatch.fnmatchcase(rel, pattern[1:])
    if "/" not in pattern:
        return fnmatch.fnmatchcase(rel.rsplit("/", 1)[-1], pattern)
    parts, want = rel.split("/"), pattern.count("/") + 1
    return len(parts) >= want and fnmatch.fnmatchcase("/".join(parts[-want:]), pattern)


def excluded(rel, kind, patterns):
    """rsync's exclude: a name anywhere, a trailing / for folders only, a leading / for the top."""
    parts = rel.split("/")
    for pattern in patterns:
        folders_only = pattern.endswith("/")
        body = pattern.strip("/").split("/")
        width = len(body)
        ends = [width] if pattern.startswith("/") else range(width, len(parts) + 1)
        for end in ends:
            if end > len(parts):
                continue
            window = parts[end - width:end]
            if all(fnmatch.fnmatchcase(part, want) for part, want in zip(window, body)):
                is_dir = end < len(parts) or kind in ("dir", "unreadable")
                if is_dir or not folders_only:
                    return True
    return False


assert included("runs/7/evidence/a.png", "evidence/*")
assert not included("runs/7/a.png", "evidence/*")
assert included("train.log", "/train.log") and not included("runs/train.log", "/train.log")
assert excluded("app/.locks/x", "file", [".locks/"])          # inside a folder named .locks
assert not excluded("app/.locks", "file", [".locks/"])        # a FILE named .locks is kept
assert excluded("session_index.jsonl", "file", ["/session_index.jsonl"])
assert not excluded("old/session_index.jsonl", "file", ["/session_index.jsonl"])
print("patterns behave as rsync reads them")

A sync planner where one unreadable folder blocks every delete (runnable)·python

def plan_sync(source, target, delete):
    """source/target: {rel: kind}. A folder that would not open blocks every delete."""
    blind = [rel for side in (source, target) for rel, kind in side.items() if kind == "unreadable"]

    def outside(rel):
        return not any(rel == b or rel.startswith(b + "/") for b in blind)

    send = sorted(r for r, k in source.items() if k == "file" and r not in target and outside(r))
    gone = sorted(r for r in target if r not in source and outside(r))
    if blind:
        return send, [], [f"unreadable: {b}; nothing under it moved and this run deleted nothing" for b in blind]
    return send, gone if delete else [], []


source = {"a.md": "file", "private": "unreadable"}
target = {"a.md": "file", "private": "dir", "private/keep.md": "file", "stale.md": "file"}
print(plan_sync(source, target, delete=True))
print(plan_sync({"a.md": "file", "new.md": "file"}, {"a.md": "file", "stale.md": "file"}, delete=True))

Patterns and a reviewed plan from the CLI (paths are examples)·bash

# Only the evidence images and the top-level training log, from a peer's run folder
delta cp @workbench:~/Projects/run-42/ ~/Projects/ --include 'evidence/*' --include /train.log

# Mirror a project, skipping lock folders and caches, and see the plan before anything changes
delta sync ~/Projects/site @pippa:~/Projects/site --exclude .locks/ --exclude '*.tmp' --delete --dry-run

# Deliver into an inbox without ever replacing a file that is already there
delta cp ~/Projects/report.pdf @workbench:~/Inbox/ --unique

External links

Exercise

In a scratch folder, build a small tree: a regular file, a link that points to a sibling file inside the tree, a link that points to /etc/hosts, and a subfolder you make unreadable with chmod 000. Write a walker with os.scandir that reports each entry as file, link-inside, link-outside or unreadable, without ever following a link. Then feed its output into plan_sync above against a fake destination and confirm that no delete is planned. Restore the folder with chmod 755 when you are done.

Hint

Use entry.is_symlink() before entry.is_dir(), and resolve a link's target with os.path.normpath(os.path.join(parent, os.readlink(path))), never os.path.realpath on the whole chain. An unreadable folder shows up as a PermissionError from os.scandir on that folder, and that exception is the signal to record unreadable.