Lesson 04 of 06 · published

An App Ships a Child

~14 min · plugins, importlib, declarations, fault-isolation, eagle

Level 0Loose File

0 XP0/43 lessons0/12 achievements

0/100 XP to next level100 XP to go0% complete

Domain Code Never Moves Into Delta

Delta knows files, folders, links, hashes and HTTP. It does not know what an Eagle item is, which image in an item's folder is the real original, or what a soul vault's committed bytes are. It should not have to. The design rule in Delta's house sign is short: root class first; apps are children. A plain tree is a DirectoryShare; an app whose data needs domain knowledge ships a child of DataShare in its own repository and declares it in a file named delta.json at that repository's root. Domain code stays with the domain.

The Declaration

Loomis, the drawing-study app, was the first to need it. Its engine runs on the service host; the Eagle library it studies lives on the workbench. Its delta.json (shown below) says everything Delta needs and nothing it should not know:

version is 1; a file with any other version declares nothing.
name becomes the second half of the share id: cwkLoomis/eagle.
host says which Mac serves it, as a role (@workbench), a member name, or a fleet label. Every Mac reads every declaration and loads only the shares whose host resolves to itself; the rest are skipped without a word.
class is file:Class, a path inside the app's repository and the class name in it: loomis/delta_eagle.py:EagleLibraryShare.
readers, writers, apps: who may ask (lesson 5). Loomis lets only the Mac that hosts Loomis' engine, plus the workbench itself, read it, and only for the app cwkLoomis.
options: free settings the child reads itself. Loomis passes env_file, the name of the file on that Mac that holds the Eagle API token.

Loading by File Path, Fenced to the Repository

At start, load_shares walks the family's projects folder for repositories that carry a delta.json, and for each declaration placed on this Mac it calls _load_child. That function does four things, and the order is the safety:

It resolves the class file and refuses unless the real path is a file inside the declaring repository. A declaration cannot point Delta at another app's code, or at anything outside the projects tree, with ../.
It imports the file under a module name of its own making (delta_app_<repo>_<share>), with importlib.util.spec_from_file_location. No package install, no entry point, no sys.path edit by Delta.
It refuses unless the named attribute is a class and a subclass of DataShare.
It constructs the child with the declaration's fields and the repository's path as repo_root.

A Child's Bug Is Its Own Problem

The whole call is wrapped. Any exception while loading one share, a syntax error in the child, a missing option, a constructor that raises, is logged with its traceback and recorded as that share's failure: {"id": "cwkLoomis/eagle", "error": "ValueError: …"}. Every other share loads. Delta keeps serving the home folder and every app share that did load, its health document lists the failure and turns its status from ok to degraded, delta status prints it, and a caller asking for the broken share gets a named answer rather than a vague 404: share cwkLoomis/eagle failed to load on <mac>: ValueError: ….

That is the right blast radius. A daemon that every Mac's data rides on cannot fall over because one app pushed a typo. Fixing it is an app-side change plus a reload: the engine answers POST /v1/reload on loopback by loading every share again.

What a Child Must Live With

The child runs inside Delta's process, so it inherits Delta's constraints. Delta runs on Apple's /usr/bin/python3 with the standard library only, so Loomis' child imports nothing from Loomis and nothing from pip: its docstring says so in its first paragraph. To decide whether an item sits inside Loomis' declared folder scope, it reads the item's own metadata.json from disk and checks it against Eagle's folder tree. The tree comes from Eagle's /api/folder/list, kept for five seconds, so a burst of 384 thumbnails asks Eagle for it about once rather than 384 times; when Eagle is closed, the child reads the same tree from the library's own metadata.json instead. The library's path is asked once, on first use, and remembered. For its four read-only API operations it calls Eagle's localhost API with the token from Loomis' .env on that Mac, and when that call fails it deliberately does not echo the URL, because the URL carries the token. The token never leaves the workbench.

Soul Vault's child takes the other path a child can take: it puts its own repository on sys.path and imports its store code, and on construction it drops any soulvault modules a long-lived Delta imported earlier, so that a reload after a git pull serves the code on disk instead of yesterday's.

And the child serves only what its declaration grants. Loomis' one write, an item's star rating, is not part of the share at all; it stays a command over SSH, which is what SSH is for in this family.

Code

Loomis' delta.json: one share, placed by role, read by one app·json

{
  "version": 1,
  "shares": [
    {
      "name": "eagle",
      "host": "@workbench",
      "class": "loomis/delta_eagle.py:EagleLibraryShare",
      "readers": {"members": ["cwkLoomis"], "roles": ["workbench"]},
      "apps": ["cwkLoomis"],
      "description": "Dad's Eagle library, read-only: Loomis' source items (original, thumbnail) and its read API",
      "options": {"env_file": ".env"}
    }
  ]
}

How Delta loads a child, and what one failure costs (trimmed excerpt)·python

# delta/shares.py — _load_child and load_shares (trimmed excerpt)

def _load_child(repo, decl):
    ref = decl["class"]
    file_part, _, class_name = ref.partition(":")
    path = (repo / file_part).resolve()
    if not str(path).startswith(str(repo.resolve()) + "/") or not path.is_file():
        raise ValueError(f"class file {file_part} is not a file inside {repo.name}")
    module_name = f"delta_app_{repo.name}_{decl['name']}".replace("-", "_").replace(".", "_")
    spec = importlib.util.spec_from_file_location(module_name, str(path))
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    cls = getattr(module, class_name, None)
    if not (isinstance(cls, type) and issubclass(cls, DataShare)):
        raise ValueError(f"{ref} is not a DataShare child")
    return cls(decl["name"], app=repo.name, readers=decl.get("readers"),
               writers=decl.get("writers"), apps=decl.get("apps") or [repo.name],
               cache=decl.get("cache", True), description=decl.get("description", ""),
               options=decl.get("options"), repo_root=repo)


def load_shares(host_cfg=None):
    ...
    for repo in repos:                      # every repository with a delta.json
        ...
        for decl in decls:
            share_id = f"{repo.name}/{decl.get('name')}"
            host = _host_label(str(decl.get("host") or ""))
            if host != me:                  # placed on another Mac: not ours to load
                continue
            try:
                shares.append(_load_child(repo, decl))
            except Exception as exc:  # a child's bug disables its own share only
                log.exception("share %s failed to load", share_id)
                failures.append({"id": share_id, "error": f"{type(exc).__name__}: {exc}"})
    return ShareSet(shares, failures)

Runnable: a loader that fences class files to their app and isolates a broken child·python

# Load share children by file path from app folders; a broken child disables its own share and nothing else.
import importlib.util
import json
import os
import sys
import tempfile


class Share:
    def __init__(self, name, app):
        self.name, self.app = name, app

    @property
    def id(self):
        return f"{self.app}/{self.name}"


def load_child(repo, decl):
    file_part, _, class_name = decl["class"].partition(":")
    path = os.path.realpath(os.path.join(repo, file_part))
    if not path.startswith(os.path.realpath(repo) + "/") or not os.path.isfile(path):
        raise ValueError(f"class file {file_part} is not a file inside {os.path.basename(repo)}")
    module_name = f"app_{os.path.basename(repo)}_{decl['name']}"
    spec = importlib.util.spec_from_file_location(module_name, path)
    module = importlib.util.module_from_spec(spec)
    sys.modules[module_name] = module
    spec.loader.exec_module(module)
    cls = getattr(module, class_name, None)
    if not (isinstance(cls, type) and issubclass(cls, Share)):
        raise ValueError(f"{decl['class']} is not a Share child")
    return cls(decl["name"], os.path.basename(repo))


def load_shares(projects):
    shares, failures = {}, []
    for repo in sorted(os.path.join(projects, d) for d in os.listdir(projects)):
        decl_file = os.path.join(repo, "share.json")
        if not os.path.isfile(decl_file):
            continue
        for decl in json.load(open(decl_file))["shares"]:
            try:
                share = load_child(repo, decl)
                shares[share.id] = share
            except Exception as exc:  # a child's bug disables its own share only
                failures.append({"id": f"{os.path.basename(repo)}/{decl['name']}",
                                 "error": f"{type(exc).__name__}: {exc}"})
    return shares, failures


if __name__ == "__main__":
    sys.modules["sharekit"] = sys.modules[__name__]  # what the children import
    with tempfile.TemporaryDirectory() as projects:
        apps = {
            "goodApp": ("from sharekit import Share\nclass Gallery(Share):\n    pass\n", "gallery.py:Gallery"),
            "brokenApp": ("from sharekit import Share\nclass Notes(Share):\n    def __init__(self, *a):\n        raise RuntimeError('config missing')\n", "notes.py:Notes"),
            "sneakyApp": ("", "../goodApp/gallery.py:Gallery"),
        }
        for app, (source, ref) in apps.items():
            os.makedirs(os.path.join(projects, app))
            if source:
                with open(os.path.join(projects, app, ref.split(":")[0]), "w") as f:
                    f.write(source)
            with open(os.path.join(projects, app, "share.json"), "w") as f:
                json.dump({"shares": [{"name": "main", "class": ref}]}, f)
        shares, failures = load_shares(projects)
        print("serving:", sorted(shares))
        for failure in failures:
            print("disabled:", failure["id"], "-", failure["error"])

External links

Exercise

Run the runnable block and confirm one share serves and two are disabled with different reasons. Then add a fourth app whose class file is fine but whose named class is a plain function rather than a Share subclass, and a fifth whose declaration places the share on a different host (add a host field and skip declarations whose host is not yours). Which of those two should appear in failures, and which should not appear anywhere?

Hint

The function case must be a failure: it was meant to load here and could not. The other-host case is not a failure at all, because it was never this Mac's to load; Delta skips it before trying, exactly as load_shares does with if host != me: continue.