Metadata That Earns Its Keep

~20 min · chunking, metadata

Level 0Scout

0 XP0/41 lessons0/10 achievements

0/120 XP to next level120 XP to go0% complete

The minimum useful metadata schema

For every chunk you embed, store:

source — file path, URL, or document id
chunk_index — position in the source
updated_at — ISO timestamp of the source revision
doc_type — markdown, pdf, code, html, conversation, etc.
Anything that lets you filter at query time (tenant_id, language, project, classification level)

Filter before you rank

The single biggest performance win in production retrieval is metadata pre-filtering. Computing cosine against 10 million vectors is slow. Computing cosine against the 50,000 vectors that match {tenant: 'acme', lang: 'en'} is fast. Every serious vector store supports a where filter — use it.

Metadata becomes provenance

When the LLM answers, you want to show "source: handbook.md (updated 2026-04-12)". That citation comes straight from your metadata. Skip metadata at ingestion time and you cannot fake it later.

Code

Build chunks with disciplined metadata·python

import hashlib
from pathlib import Path
from datetime import datetime, timezone

def ingest_markdown(path: Path):
    text = path.read_text()
    chunks = splitter.split_text(text)
    updated = datetime.fromtimestamp(path.stat().st_mtime, tz=timezone.utc).isoformat()
    return [
        {
            'id': hashlib.sha1(f'{path}:{i}:{c[:80]}'.encode()).hexdigest(),
            'text': c,
            'metadata': {
                'source': str(path),
                'chunk_index': i,
                'updated_at': updated,
                'doc_type': 'markdown',
                'project': path.parts[-3] if len(path.parts) >= 3 else None,
            },
        }
        for i, c in enumerate(chunks)
    ]

External links

Exercise

Pick one corpus you maintain. Define the minimum metadata schema (5–7 fields) you would require for every chunk. Justify each field with a query you would actually want to run. Throw away any field that fails the test.

Progress

Progress is local-only — sign in to sync across devices.

← PreviousStructure-Aware: Markdown, HTML, Code Next →Semantic Splitting and When to Skip It

Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)

No comments yet — be the first.