Metadata Is the Real Index

Files are contents; records are the library

A room full of books is not a library until someone can answer questions about them. The same distinction runs through an archive of weights: the tensors are the collection, but the records — acquisition notes, digests, license lines, chain scores, decision history — are what make it usable, defensible, and extensible. Strip the records from a healthy archive and you get a very well-preserved junk drawer.

Notice how much of this quest has secretly been about metadata. The pin log (revisions), the walk-away log (refusals), SHA256SUMS files (identity), NOTES per directory (context), license lines (terms), chain records (trust). The craft was never only "get the bytes" — it was "make the bytes accompanied by their story." An acquisition without its record is half an acquisition, and the missing half is the half that appreciates over time.

What the catalog must answer

The catalog — whatever form it takes, from per-directory NOTES files to a small database — earns its keep by answering, without touching the tensors:

What do I hold? Maker, model, variant, revision — the acquisition coordinate, per holding.
Why is it here? The named reason from the woodpile rules. Reasons rot faster than files; the record keeps them alive.
On what terms? The license line and its conditions.
Verified how, and when? Digest status, chain score, last sweep date.
What did I consider and decline? The walk-away entries — the catalog's immune memory.

That last item is the one people forget and archivists don't: a catalog that records only holdings invites re-litigating every refused candidate forever. The refusals are as much the collection's shape as the acquisitions.

The bytes answer 'what'; the records answer 'why, under what terms, verified how'. An archive is the pairing, and the records are the half that lets the whole thing be trusted, inherited, and defended.

Form follows fragility

Keep the records where failure can't reach them easily: duplicated alongside the files (NOTES per directory — survives database loss), and summarized in one index (a single catalog file or small database — survives directory-by-directory archaeology). Two representations, one truth maintained in step. The test of the setup is the disaster drill: if the index vanished tonight, could you rebuild it from the per-directory notes? If the directories vanished, does the index still tell the story? Both answers should be yes.

Code

Rebuild the index from the tree — the disaster drill·bash

# The catalog is derivable; prove it to yourself:
find archive -name NOTES -exec sh -c 'echo "== $1"; cat "$1"' _ {} \; \
  > catalog-rebuilt.txt
grep -c '^==' catalog-rebuilt.txt   # one block per holding

# The query the catalog exists for — answered without touching tensors:
grep -l 'license: research-only' $(find archive -name NOTES) \
  | while read n; do dirname "$n"; done
# -> every holding whose use is restricted, in two commands

# And the reverse question — what was refused and why:
cat walkaway.log | tail -20

# If the first command can rebuild your index and the last two
# can answer real questions, the metadata layer is doing its job.

Exercise

Run the disaster drill on your archive (or a practice directory): rebuild an index from per-directory NOTES, then answer two real questions from it alone — which holdings are research-only, and what was refused this quarter. If either question is unanswerable, name the missing record and add it to your NOTES convention.

Hint

The most commonly missing fields are 'why' and license — both feel obvious at acquisition time and both evaporate within months. If your NOTES lack them, that is the finding.

Files are contents; records are the library

What the catalog must answer

Form follows fragility

Code

External links

Exercise

Progress

Comments 0