[Python] Chinese chapter lookup shows too many nodes because of variant characters – how can I...

I'm building an offline-only CLI tool that searches our company rulebook, stored as hierarchical JSON:
Levels like 3, 3.2, 3.2.1 …

router(query) chooses the node by comparing the query with each record’s "title" and "source".
Once a node ID is found, get_chapter_family() prints that node + its children.

When the user types:

• 程序說明 (“procedure explanation”) → Should limit output to 3.2 and its children, but prints 3.3 as well.
• 新進人員訓練作業 (“new-hire training procedure”) → Should limit to 3.2.1 (a)(b), but also prints 3.2.2. normalize_txt() already does NFKC + punctuation strip + lower-case, yet the JSON titles contain variant characters such as 說 vs 說, 練 vs 練.
  This makes exact string equality fail, so router() falls back to parent levels and the range explodes.

Here’s the normalization function:

def normalize_txt(t: str) -> str:
t = unicodedata.normalize('NFKC', t)
t = PUNCT_PAT.sub('', t)
return t.lower()


Tried:

1. Manual replace map – works, but maintenance nightmare.
2. OpenCC for T-Chinese variant folding – local, but adds a heavy dependency.
3. Slug field pre-computed at ETL time (title → canonical form) – helps, still doesn’t prevent fallback overflow.
4. Stop “infinite fallback” – only allow one level up; prevents 3 → 3.3 spill but still needs better matching.

Question: I’m open to any design, even a completely different search strategy, provided everything runs 100% locally (no cloud APIs).
What architecture or text-normalization approach would you recommend to keep the match precise without maintaining an endless variant-character list?

Continue reading...