Jobs, Not Commands

Who Is Doing the Copying?

With scp or rsync, the program you typed is the transfer. It holds the sockets, it holds the progress, and when its terminal closes, the copy dies with it. A laptop lid shuts, an SSH session to the workbench drops, a tab is closed by accident: an hour of copying becomes a half-written file and no record of how far it got.

Delta separates the two. delta cp and delta sync are thin clients. They describe what should move, post that description to this Mac's own Delta on loopback, and then only watch. The work is a job that runs on a thread inside the daemon, under launchd, with the daemon's routes, tickets and Local Network standing. Close the terminal and the job keeps going. Open a new one and follow it again.

What the CLI Actually Sends

A spec. POST /v1/jobs with op (copy or sync), src, dst, and the options: delete, checksum, include, exclude, unique, a worker count, or a forced tailnet route.
A job id derived from the spec. The id is the first 16 hex digits of a SHA-256 over the spec's meaningful fields. The same command always names the same job. If that job is still planning or running, the daemon answers "attached": true instead of starting a second copy of the same work. If it already ended, the re-run plans again and finds what is in place.
A follow. The CLI opens GET /v1/jobs/<id>/events, a stream of newline-delimited JSON snapshots: one at once, then after pauses of 20, 50, 100, 200, 400 and 700 ms (so at about 20, 70, 170, 370, 770 and 1,470 ms), then one a second until a terminal state (done, failed, interrupted, cancelled, planned). A copy that lands in 300 ms is reported at the 370 ms look, not after a whole second.

Ctrl-C in the CLI is not a lost transfer either. The CLI sends DELETE /v1/jobs/<id>, the job stops at its next check and ends interrupted, and the CLI prints that the partial work is kept and the same command resumes it. --detach prints the id and returns at once; delta jobs <id> --follow picks the stream up later, from any terminal on that Mac.

One End Is This Mac

An endpoint is [mac:]path. Without a Mac it is here; with one (@workbench:~/Projects/a.mov, a role resolved through the family registry) it is that Mac's Delta. The planner refuses a job whose two ends are both here or both remote: "one end of a Delta transfer is this Mac and the other another Delta Mac". Remote-to-remote is out of scope on purpose. It would need one Mac to authorize a write between two others, and that rule has not been designed. The constraint keeps every transfer between two parties who already trust each other.

Paths Become Shares

A transfer never touches a raw filesystem path on another Mac. It maps each end onto a share that Mac serves:

You type	It means
`~/x`, or that Mac's own home path	the `home` share, path `x` (the remote home comes from the peer's identity document)
`/Volumes/Disk/x`	the `volumes` share, only where that Mac enabled it
`@cwkLoomis/eagle/x`	an app share, `<app>/<share>/<path>`
anything else	refused: "outside every share Delta serves"

So every rule from Track 2 rides along: credential stores and guarded folders are refused, read-only shares stay read-only, and a destination path inside a refused place fails before a byte moves.

The Relative Path That Landed Beside the Code

One early bug shows why the split between client and daemon needs care. The daemon's working folder is its own repository (the plist's WorkingDirectory). delta cp x ./out was sent as typed, the daemon resolved ./out against its folder, and the copy landed beside Delta's source code. The fix has two halves. The daemon now refuses a relative local path outright ("Delta runs in its own folder and cannot know the caller's"). The CLI resolves a relative path where it runs, with os.path.abspath, before it sends anything. A client and a server in different working directories have to agree on whose directory a path means. Here, only the client can know.

When the Daemon Goes Away Mid-Job

The follow stream can also end while the job is still running, because this Mac's Delta restarted. A naive client would print the last progress line it saw, which reads like an outcome. Delta's CLI checks the state of the last snapshot. If it is still planning or running, the CLI says it lost contact with this Mac's Delta, that what landed is kept, and that the same command resumes. It never reports progress as a result. The next lessons show why "the same command resumes" is a promise the daemon can keep.

Code

Delta's transfer planner: the job id and the one-end rule (trimmed excerpt)·python

# delta/transfer.py (trimmed excerpt)

def job_id(spec: dict) -> str:
    key = {k: spec.get(k) for k in ("op", "src", "dst", "delete", "checksum", "include", "exclude", "unique",
                                     "pair", "mode")}
    return hashlib.sha256(json.dumps(key, sort_keys=True).encode()).hexdigest()[:16]


class JobRunner:
    def submit(self, spec: dict) -> dict:
        ...
        jid = job_id(clean)
        with self._lock:
            running = self.jobs.get(jid)
            if running and running.state in ("planning", "running"):
                return {"id": jid, "state": running.state, "attached": True}
            job = Job(jid, clean)
            self._remember(job)
        thread = threading.Thread(target=self._run, args=(job,), name=f"delta-job-{jid}", daemon=True)
        thread.start()
        return {"id": jid, "state": "planning"}

    def plan(self, job: Job):
        src, dst = self.parse(spec["src"]), self.parse(spec["dst"])
        if src.local == dst.local:
            raise BadRequest("one end of a Delta transfer is this Mac and the other another Delta Mac "
                             f"({src.text} and {dst.text} are both {'here' if src.local else 'remote'})")
        ...

A job that outlives its caller: a tiny runner with an NDJSON-style follow (runnable)·python

import hashlib
import json
import threading
import time


class Runner:
    """Work runs on the runner's threads; a caller only submits and follows."""

    def __init__(self):
        self.jobs = {}
        self.lock = threading.Lock()

    def submit(self, spec):
        jid = hashlib.sha256(json.dumps(spec, sort_keys=True).encode()).hexdigest()[:16]
        with self.lock:
            job = self.jobs.get(jid)
            if job and job["state"] == "running":
                return {"id": jid, "attached": True}
            job = self.jobs[jid] = {"state": "running", "done": 0, "total": spec["files"]}
        threading.Thread(target=self._run, args=(job,), daemon=True).start()
        return {"id": jid, "attached": False}

    def _run(self, job):
        while job["done"] < job["total"]:
            time.sleep(0.05)          # one file moving
            job["done"] += 1
        job["state"] = "done"

    def follow(self, jid):
        for delay in (0.02, 0.05, 0.1, 0.2, 0.4, 0.7):
            yield json.dumps({"id": jid, **self.jobs[jid]})
            if self.jobs[jid]["state"] != "running":
                return
            time.sleep(delay)
        while True:
            yield json.dumps({"id": jid, **self.jobs[jid]})
            if self.jobs[jid]["state"] != "running":
                return
            time.sleep(1.0)


runner = Runner()
spec = {"op": "copy", "src": "~/a", "dst": "@peer:~/a", "files": 30}
first = runner.submit(spec)
second = runner.submit(spec)           # the same command again
print(first, second)                   # same id; the second one attached
for line in runner.follow(first["id"]):
    print(line)

Exercise

Run the runnable block above, then change it so the caller can disconnect: start a job, stop iterating follow() after two lines, sleep two seconds, and follow the same id again. Then add a cancel(jid) that ends the job as interrupted with a message saying the same submit resumes it, and make a second submit of the same spec continue from done instead of zero.

Hint

Keep the progress on the job record, not in the follower. Resuming is just a new thread that starts its loop from the recorded done count. An interrupted job is not running, so submit must not attach to it.