A Token in Everything Discriminates Nothing

The Report Was About Sports; the Bug Was About Grammar

The complaint from live use was ordinary: disliking sports articles did not make sports go away. Chasing it produced two corrections, and the second one is the interesting one because it had nothing to do with sports.

With muting wired up, one shelf started hiding articles that had no relationship to anything the reader had rejected — a restaurant piece and a drought story, each scoring around negative six point seven. They shared no subject with the disliked article. What they shared were ordinary Korean connective words, which had accumulated heavy negative weight because they appear in almost every Korean headline, including the ones that got disliked.

The Patch You Reach For First Is Language-Shaped

The obvious fix is to extend the stopword list. It works, for that language, until the next one — and the list will always be shaped like whatever language its author speaks. The original list here was English-shaped with a handful of Korean newsroom words bolted on, which is exactly how you would expect it to look, and exactly why it leaked.

Worse, a stopword list is a claim about vocabulary made ahead of time, by someone who is not looking at the corpus. It cannot know that a word which is a stopword in general prose is a meaningful term in a particular feed, and it cannot notice a new common word arriving.

Ask the Corpus Instead

The general rule is older than any of this and language-agnostic: a term's usefulness is inversely related to how many documents contain it. A token appearing in most of the pool cannot help you choose within that pool, whatever it means and whatever language it is in.

So compute it. Over the rows actually being filtered, count how many contain each token, and ignore any token above a share of the set. The threshold is deliberately low — a token in more than about a seventh of the shelf is already nearly useless as a discriminator — and there is a floor on the number of rows, because document frequency measured over a handful of items is noise.

This is self-maintaining in a way a list can never be. It adapts per shelf, so a word that is common on one topic shelf and rare on another is treated correctly on both. It handles a language nobody anticipated. And it needs no upkeep.

Measure It Per Filtered Set, Not Globally

One design choice worth defending: the frequency is computed over the rows being filtered right now, not over the whole store. That sounds like extra work for a worse estimate, and it is the right call — because "common" is a property relative to the choice you are making. On a shelf about one country, that country's name is in everything and discriminates nothing; on a general headlines shelf, the same token is a strong signal.

When a rule keeps needing per-language exceptions, the rule is at the wrong level. Ask what the exceptions have in common — here, that they were words carrying no information about which row to pick — and implement that, measured from the data, instead of enumerating instances of it forever.

Code

Document frequency computed per filtered set, replacing a per-language list·python

# A keyword only tells us something if it is rare. The stopword list is
# English-shaped, so Korean function words sailed straight into the
# profile: the heaviest negative weights were ordinary connective words,
# and one shelf was muting a restaurant piece and a drought story at
# -6.7 apiece for sharing them with a disliked article.
#
# Rather than chase stopwords per language forever, ignore any token
# that shows up in a large share of the shelf being filtered -- common
# words cannot discriminate, whatever language they are in.
MUTE_MAX_DOC_FREQ = 0.15
MUTE_DF_MIN_ROWS = 30      # DF over a handful of rows is noise


def _common_tokens(rows: list[dict]) -> set[str]:
    """Document frequency over the set BEING FILTERED, not the store.

    'Common' is relative to the choice being made: on a single-country
    shelf that country's name is in everything and discriminates
    nothing, while on a general shelf the same token is a strong signal.
    """
    if len(rows) < MUTE_DF_MIN_ROWS:
        return set()
    counts: dict[str, int] = {}
    for row in rows:
        for tok in tokens(row):
            counts[tok] = counts.get(tok, 0) + 1
    ceiling = len(rows) * MUTE_MAX_DOC_FREQ
    return {tok for tok, n in counts.items() if n > ceiling}

Exercise

Find a hard-coded list in your system that exists to exclude things — stopwords, ignored file names, known-noisy identifiers. Ask what property every entry shares. Then work out whether that property is measurable from the data at runtime, and what it would cost to compute it instead. Keep the list only if the property genuinely is not measurable.

Hint

The tell that a list should be a measurement: entries keep getting appended, and each addition is triggered by an incident rather than by a rule. A list that has grown by incident is a measurement someone has been performing by hand, one report at a time.

The Report Was About Sports; the Bug Was About Grammar

The Patch You Reach For First Is Language-Shaped

Ask the Corpus Instead

Measure It Per Filtered Set, Not Globally

Code

External links

Exercise

Progress

Comments 0