C.W.K.
Stream
More →
Lesson 03 of 04 · published

Absorb In Place, Never Migrate

~12 min · absorb-in-place, model-storage, no-migration, registry

Level 0Tool Renter
0 XP0/33 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete
"The best migration is the one you never run. Read the files where they already live."

The Migration Temptation

When you build a new engine that needs to use existing model files, the instinct is to bring them into your world: copy them into your folder structure, rename them to your convention, maybe symlink them so they look like they're yours. It feels like taking ownership. It's actually taking on risk — a migration step that can fail, duplicate gigabytes, and orphan the files the old tool still expects.

Absorb In Place Instead

The engine takes the opposite stance: scan the existing model directory and read every file where it already lives. No moving, no renaming, no symlinking into a private structure. The engine is a parallel reader of the model directory, not a new installer that relocates it. The files stay exactly where they are; the engine just builds a registry that points at them in place.

Read in place; don't relocate. When you need to use someone else's (or a previous tool's) data layout, build a reader over it rather than a migration into your own. The data has one home; you visit it. The moment you copy it into your structure, you own a second copy that can drift, and a migration that can break.

Why This Avoids a Whole Class of Bugs

Migrations are where data goes to die. A copy that half-completes leaves you with an inconsistent state. A rename that the old tool didn't expect breaks the old tool. Symlinks rot when paths change. Every one of these is a failure mode you simply don't have if you never move the files. Absorbing in place isn't just tidier — it deletes an entire category of failure by refusing to do the thing that causes it.

The cheapest bug is the one the design makes impossible. You can't have a half-completed migration if there's no migration. Architectural choices that remove a failure mode entirely beat any amount of careful code that merely handles it well. Prefer the design with fewer ways to break.

Coexistence Is a Feature

Because the engine reads in place and never mutates the layout, the old tool can keep running from the exact same files. You don't have to choose between the new engine and the old one during the transition — both read the same directory, neither steps on the other. That coexistence is what makes adopting the new engine low-risk: if something's wrong, the old tool is right there, untouched, still working.

A migration forces a cutover; absorbing in place doesn't. Move-and-rename means the old tool stops working the moment you migrate — you've burned the bridge before crossing it. Reading in place keeps both tools alive simultaneously, so adoption is reversible. Reversible adoption is how you try a new tool without betting the farm.

Pippa's Confession

My instinct was to make the model directory 'mine' — proper folders, my naming, everything neat under the engine's roof. Dad stopped me: the files aren't ours to reorganize, they're shared ground the old tool still stands on. Reading them in place felt like less ownership, but it was more respect — for the data, for the old tool, for future-me who might need to fall back. The neatest design was the one that touched the least.

Code

Point at the file; don't copy it·python
# ABSORB IN PLACE: scan the existing layout, point at files where they live.
from pathlib import Path

def scan_models(models_root: Path) -> list[dict]:
    rows = []
    for path in models_root.rglob("*.safetensors"):
        rows.append({
            "id": stable_id(path),
            "abs_path": str(path),     # registry POINTS at the file in place
            "adapter": "local",
            # architecture filled in by reading the file's tensors (next lesson)
        })
    return rows
# No copy. No rename. No symlink. The old tool still reads the same files.

# MIGRATION (rejected): the failure-prone path we refuse to take.
# for path in models_root.rglob("*.safetensors"):
#     shutil.copy(path, OUR_DIR / our_name(path))   # duplicates GBs,
#                                                    # can half-fail,
#                                                    # orphans the old tool

External links

Exercise

Think of a time you (or a tool you used) migrated data into a new structure. What broke, half-completed, or got duplicated? Now redesign it as 'absorb in place': could a reader over the original layout have avoided the migration entirely? What would the reader need to know that the migration encoded in folder structure?
Hint
Migrations often encode meaning in the new folder structure ('all the SDXL models are in this folder now'). A reader has to derive that same meaning from the data itself instead — which is exactly what the next lesson's tensor classification does.

Progress

Progress is local-only — sign in to sync across devices.
← PreviousA Dumb Router Is a Smart RouterNext →The Tensors Don't Lie
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.