Content-Addressed Ids

"Don't give the chunk a name. Let the chunk's content name itself."

The Formula

A Lantern chunk id is not assigned — it is computed. Take the document's own content hash, the start and end character offsets of the slice, and the id of the chunking profile that produced it. Join them, hash the result, and keep the first 32 hex characters. That string is the chunk id. Every input is a fact about the content itself, so the output is a fingerprint of the content, not a label the database stuck on at insert time.

Why Every Ingredient Is There

Each part of the formula earns its place:

doc_sha256 — which document, identified by its content, not its filename. Two files with identical bytes are the same document here; a renamed file is still the same document; an edited file is a different one.
char_start and char_end — which slice of that document, as character offsets into the decoded text. This pins the exact span.
profile_id — under which chunking rules. The same span cut by a different profile is a different chunk, and must get a different id.

Nothing in the formula mentions when the chunk was inserted or where it sits in a table. That omission is the whole point: remove every trace of position and time, and what's left can only change when the content changes.

What You Get for Free

Content-addressing hands you three properties at once, without extra bookkeeping:

Stability: re-ingest the same text under the same profile and the id is byte-identical — across rebuilds, across machines.
Deduplication: identical content computes to the same id, so it naturally collapses to one entry instead of many.
Verifiability: given the content, anyone can recompute the id and confirm it matches — the id is checkable, not just assigned.

This is the exact trick Git uses to name commits and blobs, and that container systems use to name image layers: the address is a hash of the content.

Truncating the hash is a deliberate, safe trade. A full SHA-256 is 64 hex characters; Lantern keeps 32. That's still 128 bits of address space — vastly more than enough that two different chunks colliding is astronomically unlikely for any real corpus. You trade a bit of theoretical collision margin for shorter, more manageable ids. Know why you truncate, and know it's safe at this scale.

The Id Travels With Full Provenance

The chunk id never travels alone. Every result carries the whole provenance bundle: the corpus, the document's path, the document hash, the character offsets, the chunk hash, and the profile. The id is the durable handle; the bundle is everything you'd need to re-find, re-verify, or re-derive the slice from scratch. Together they make a result something you can point at with total confidence, years later, on a machine that didn't exist when you first ran the search.

Code

The chunk id is a fingerprint, computed not assigned·python

import hashlib

def chunk_id(doc_sha256: str, char_start: int, char_end: int, profile_id: str) -> str:
    # Every input is a property of the CONTENT, never of insertion time or place.
    payload = f"{doc_sha256}:{char_start}:{char_end}:{profile_id}"
    return hashlib.sha256(payload.encode("utf-8")).hexdigest()[:32]

# Same content + same profile -> same id, forever, everywhere:
a = chunk_id("d41d8c...", 4120, 4229, "md-para-v1")
b = chunk_id("d41d8c...", 4120, 4229, "md-para-v1")
assert a == b            # deterministic

# Change the chunking rules -> a genuinely different chunk -> a different id:
c = chunk_id("d41d8c...", 4120, 4229, "md-para-v2")
assert c != a            # profile is part of identity

Exercise

Design a content-addressed id for something in your own world — a photo, a note, a record. List the properties that make it uniquely what it is (not when you saved it or what row it landed in). Write the formula: hash of which fields? Then test it in your head — if you copied that item to another machine, would your formula produce the same id? If not, you've included a position or time field that needs removing.

Hint

The failure mode is sneaking in a timestamp, a filename, or an auto id 'just to be safe'. Every one of those breaks portability. A content-addressed id must be computable from the content alone, so two independent machines reach the same answer.

The Formula

Why Every Ingredient Is There

What You Get for Free

The Id Travels With Full Provenance

Code

External links

Exercise

Progress

Comments 0