Skip to content
C.W.K.
Lesson 03 of 06 · published

Same Dish Twice, One Class

~13 min · matching, unicode, normalization, deduplication

Level 0Mise en Place
0 XP0/43 lessons0/12 achievements
0/100 XP to next level100 XP to go0% complete

The duplicate problem

A model is an inconsistent speller. Over a month of plates, Pippa might write "Gambas al ajillo", "gambas al ajillo!" and "Gambas-al-Ajillo" for the same dish. If every spelling made a new class, one dish's plates would scatter across several rows. The badge would not be the problem, since one plate cooked at home is enough for a class to read as cooked. Where the plates land would be. Each row's page would list only its share of the plates, and a sibling Pippa proposed earlier could stay "Pippa's inference" after Dad had eaten the dish, because his plate went to a twin row. The tree would lie by fragmentation.

Sizzle has two defenses, and they run in order.

First: hand the model the ids

Every reading's input includes Dad's tree as data, built by store.catalog(): each principle's id, name, summary and status, and each dish or variant's id, kind, name, gloss, cuisine and status. The rubric tells Pippa to put a matching id in ref or dish_ref and keep the existing name.

The engine does not take that id on faith. _resolve_principle accepts a ref only if it names an existing class whose kind is principle. _resolve_dish accepts a dish_ref only for a dish or a variant. An id Pippa misremembers, or a dish id put where a principle id belongs, is ignored, and matching falls back to names.

Second: fold the name

When there is no usable id, the engine compares names through norm, a six-line function in backend/store.py. Its steps, in order:

  1. NFKD splits each accented letter into a base letter plus a combining mark.
  2. Every combining mark is dropped, which removes the accents.
  3. NFKC recomposes what is left and maps compatibility forms, such as full-width letters, to their plain equivalents.
  4. casefold() lowercases more aggressively than lower(), so "Straße" and "strasse" agree.
  5. Every run of characters that are not letters or digits, underscores included, becomes one space, and the ends are trimmed.

I ran it on origin/main against a handful of names. What folds together:

Name as writtenMatching key
Gambas-al-Ajillogambas al ajillo
gambas_al_ajillogambas al ajillo
Crème brûléecreme brulee
paella (valenciana)paella valenciana
ＫＩＭＣＨＩ (full-width)kimchi
김치-찌개김치 찌개

And what stays apart. 김치찌개 and 김치 찌개 do not match, because norm turns any separator into a single space but never deletes one. Spaghetti aglio e olio and Aglio e olio differ too, because there is no fuzzy or partial matching at all. A name made only of punctuation folds to an empty key, and find_class returns nothing for an empty key rather than matching every unnamed thing.

Hangul passes through intact. NFKD does split a syllable into its jamo, but jamo are not combining marks, so nothing is dropped and NFKC puts the syllable back together.

Two names, two chances

A dish has two names, and the lookup uses both. find_dish tries the reading's original name first and its Korean name second. For each candidate, find_class compares the key against both the name and the gloss column of every dish and variant. So a reading that writes the Korean name where the original belongs still finds the class whose gloss it matches. Variants are included, which means a plate of Dad's own version links to the variant itself when the reading names it the way he did. Principles are matched by name only, through find_principle.

The engine test for this lesson is short. Two plates of the same dish, both read, and the dish's page lists exactly those two plates on one class.

Three keys, three jobs

There are three name keys in Sizzle, and they do not agree on everything:

  • norm in the engine, for matching the tree. It folds accents and punctuation.
  • _key inside integrate, for matching a sibling's principle name to a principle named earlier in the same reading. It folds only spacing and case.
  • nameKey on the web and TreeIndex.key on the phone, for turning names inside a reading into links to class pages. They apply NFKC, lowercase and collapse spaces, and they keep accents.

That leaves one seam you can see. If the tree holds "Crème brûlée" and a reading writes "creme brulee", the engine links the plate to the right class, and the web plate's "In the tree" list shows it. The dish heading inside the reading stays plain text unless its Korean name happens to match. The record is right. Only a convenience link is missing.

Principle: Decide what "the same name" means once, write it as a function, and test it with pairs that must match and pairs that must not.

Code

cwkSizzle, backend/store.py: norm, the matching key for every class name (verbatim, lines 128-133)·python
def norm(text: str | None) -> str:
    """Matching key for names: case, accents, punctuation and spacing folded."""
    decomposed = unicodedata.normalize("NFKD", text or "")
    stripped = "".join(ch for ch in decomposed if not unicodedata.combining(ch))
    folded = unicodedata.normalize("NFKC", stripped).casefold()
    return re.sub(r"[\W_]+", " ", folded).strip()
cwkSizzle, backend/store.py: find_class compares one key against name and gloss (verbatim, lines 428-437)·python
    def find_class(self, kind: str | tuple[str, ...], name: str) -> dict[str, Any] | None:
        kinds = (kind,) if isinstance(kind, str) else kind
        key = norm(name)
        if not key:
            return None
        marks = ",".join("?" * len(kinds))
        for row in self._all(f"SELECT id, name, gloss FROM classes WHERE kind IN ({marks})", kinds):
            if norm(row["name"]) == key or (row["gloss"] and norm(row["gloss"]) == key):
                return self.klass(row["id"])
        return None
cwkSizzle, backend/domain.py: find_dish tries both names; find_principle matches principles only (verbatim, lines 321-330)·python
    def find_dish(self, name: str | None, gloss: str | None = None) -> dict[str, Any] | None:
        for candidate in (name, gloss):
            if candidate:
                found = self.store.find_class(("dish", "variant"), candidate)
                if found:
                    return found
        return None

    def find_principle(self, name: str | None) -> dict[str, Any] | None:
        return self.store.find_class("principle", name or "") if norm(name) else None

External links

Exercise

Write the name-normalizing function for entities in your own system: tags, product names, people, places. Then write a table-driven test with two lists, pairs that must produce the same key and pairs that must not. Include accents, full-width characters, punctuation, a spacing difference, and one pair your users would call the same that you decide to keep apart on purpose. Run the function over your real data and count how many existing rows would collide before you ship it.
Hint
Print the key next to the original for every collision you find. A collision list you can read is how you notice that your fold is too strong. Keep an empty key from matching anything, the way find_class returns None when norm comes back empty.

Progress

Progress is local-only — sign in to sync across devices.
← PreviousSiblings Across CuisinesNext →Proposal Versus Adoption: Who Owns the Words
Spotted a bug or have feedback on this page?Report an Issue

Comments 0

🔔 Reply notifications (sign in)
Sign in — Please sign in to comment.

No comments yet — be the first.