cjkfold 0.1.0__tar.gz

cjkfold-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 cdcailf
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cjkfold-0.1.0/PKG-INFO

@@ -0,0 +1,247 @@
+Metadata-Version: 2.4
+Name: cjkfold
+Version: 0.1.0
+Summary: Fold look-alike CJK glyphs (Kangxi radicals, CJK radical supplements) so text extracted from PDFs matches again.
+Author: cdcailf
+License: MIT
+Project-URL: Homepage, https://github.com/cdcailf/cjkfold
+Project-URL: Repository, https://github.com/cdcailf/cjkfold
+Project-URL: Issues, https://github.com/cdcailf/cjkfold/issues
+Keywords: cjk,chinese,unicode,nfkc,normalization,pdf-extraction,rag,retrieval,text-matching
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: Chinese (Simplified)
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Dynamic: license-file
+
+# cjkfold
+
+Text pulled out of a PDF often contains characters that **look** like ordinary
+Han characters but are not them. Comparison against them quietly fails.
+
+```
+⻆  U+2EC6  CJK RADICAL SIMPLIFIED HORN   renders as  角  U+89D2
+⾊  U+2F8A  KANGXI RADICAL COLOR          renders as  色  U+8272
+⺮  U+2EAE  CJK RADICAL BAMBOO            folds to    𥫗  U+25AD7
+```
+
+`cjkfold` folds them back onto the characters you meant:
+
+```python
+from cjkfold import fold
+
+fold("标准颜⾊值与⻆色按钮")
+# '标准颜色值与角色按钮'
+
+"颜色" in "标准颜⾊值"          # False  ← the bug
+"颜色" in fold("标准颜⾊值")     # True
+```
+
+Zero runtime dependencies. Pure functions. Python 3.9+.
+
+---
+
+**背景阅读 / Background:** [PDF 抽出来的中文，为什么搜不到](docs/why-cjk-text-breaks.md)
+—— 完整的排查经过（中文），包括我自己踩的四个坑。
+社区版：[掘金](https://juejin.cn/post/7695671592383807542)
+
+**Built on this:** [corpusfold](https://github.com/cdcailf/corpusfold) — this
+character layer wired into a full PDF → knowledge-base pipeline (抽取 / 分栏 /
+表格 / 分块 / 混合检索).
+
+## The problem
+
+Nothing raises. The text displays correctly, because the glyphs really are
+identical. What breaks is everything downstream that compares characters:
+
+- `"紫色" in text` returns `False` for a document that shows 紫色
+- a keyword search misses
+- a vector retriever ranks an unrelated *short* document above the one that
+  contains the query verbatim, because a short document's embedding is tighter
+  and scores a higher cosine — measured on a real corpus, the relevant document
+  scored **0.29** against a decoy's **0.36**
+- deduplication, caching and ground-truth checks all start disagreeing
+
+## Why NFKC is not enough
+
+The reflex is `unicodedata.normalize("NFKC", text)`. It handles the **Kangxi
+Radicals** block completely — all 214 characters carry compatibility
+decompositions — and then stops:
+
+| Block | Characters | Repaired by NFKC | Repaired by cjkfold |
+|---|---|---|---|
+| Kangxi Radicals `U+2F00–U+2FDF` | 214 | **214** | 214 |
+| CJK Radicals Supplement `U+2E80–U+2EFF` | 115 | **2** | **114** |
+
+Unicode withholds compatibility mappings in the supplement block *on purpose*:
+those glyphs are distinct encoded radicals, not compatibility variants of an
+existing character. So NFKC leaves 113 of the 115 standing.
+
+`cjkfold` closes that gap with the `EquivalentUnifiedIdeograph` property from
+the Unicode Character Database (Unicode 15.1+), which maps 114 of them onto a
+unified ideograph. The one remainder, `⺀ U+2E80 CJK RADICAL REPEAT`, has no
+ideograph equivalent and is left alone.
+
+## Install
+
+Not published on PyPI yet, so install from a checkout:
+
+```bash
+git clone https://github.com/cdcailf/cjkfold
+cd cjkfold
+uv venv .venv --python 3.11
+uv pip install -e .
+```
+
+Once it is published this collapses to `pip install cjkfold`.
+
+## Usage
+
+```python
+from cjkfold import fold, has_lookalikes, iter_lookalikes
+
+fold("紫⾊的⻆落")            # '紫色的角落'
+fold("普通文本")              # unchanged; returns the same object
+
+has_lookalikes("⻆")           # True
+has_lookalikes("角")           # False
+
+for item in iter_lookalikes("第一行\n第⼆行⻆"):
+    print(item)
+# 2:2  ⼆ U+2F06 -> 二 U+4E8C  (KANGXI RADICAL TWO)
+# 2:4  ⻆ U+2EC6 -> 角 U+89D2  (CJK RADICAL SIMPLIFIED HORN)
+```
+
+`fold()` is idempotent, and it is deliberately **not** an NFKC pass: it will
+not touch full-width punctuation (`，` `；` `（` `）`), compatibility characters
+or ligatures, so it is safe to apply to text that is still going to be stored or
+displayed. Combine with NFKC yourself if you want both:
+
+```python
+import unicodedata
+clean = fold(unicodedata.normalize("NFKC", text))
+```
+
+### Command line
+
+```bash
+cjkfold check document.txt      # report look-alikes, exit 1 if any (CI-friendly)
+cjkfold check --quiet docs/*.txt
+cjkfold fold document.txt       # print the folded text
+cjkfold fold -w document.txt    # rewrite in place
+cat document.txt | cjkfold check -
+```
+
+```
+$ cjkfold check extracted.txt
+extracted.txt: 3 look-alike character(s) (1 kangxi, 2 radical-supplement)
+    1:4  ⾊ U+2F8A -> 色 U+8272  (KANGXI RADICAL COLOR)
+    1:7  ⻆ U+2EC6 -> 角 U+89D2  (CJK RADICAL SIMPLIFIED HORN)
+    2:5  ⺮ U+2EAE -> 𥫗 U+25AD7  (CJK RADICAL BAMBOO)
+```
+
+Drop `cjkfold check` into an ingest pipeline to catch extraction regressions
+before they reach the index.
+
+## Retrieval helper
+
+Folding is the fix; ranking is the follow-up. Dense retrieval is bad at literal
+tokens, so `exact_boost` adds a bonus for query n-grams that appear verbatim in a
+document:
+
+```python
+from cjkfold import exact_boost
+
+score = vector_similarity(query, document) + exact_boost(query, document)
+```
+
+Both sides are folded inside the helper. That is not a convenience — folding the
+index but not the query (or the reverse) silently disables the whole mechanism,
+because nothing ever matches. The API gives you no way to get it wrong.
+
+Queries shorter than the shingle size fall back to a containment test, so a
+two-character query is not silently scored zero.
+
+## Wiring it into a pipeline
+
+Three touch points, zero framework dependencies:
+
+```python
+from cjkfold import fold, fold_documents, rerank
+
+# 1. on the way in -- clean the corpus before it is embedded
+documents = fold_documents(loader.load())
+
+# 2. on the way out -- fold the query so it can match
+hits = vectorstore.similarity_search_with_score(fold(question), k=20)
+
+# 3. before you trust the order -- restore the verbatim matches
+documents, scores = zip(*hits)
+for document, score in rerank(question, documents, scores):
+    ...
+```
+
+`fold_documents` and `rerank` recognise LangChain documents (`page_content`),
+LlamaIndex nodes (`text`) and plain strings -- and import none of them. Making
+you install a framework in order to fold a string would be a bad trade:
+framework APIs move, string folding does not.
+
+## Caveats
+
+- **Glyph folding only.** No traditional⇄simplified conversion, no z-variant
+  normalisation. Those are different problems with different trade-offs.
+- **Table follows the latest UCD**, which can run ahead of the `unicodedata`
+  database bundled with an older Python — 3.9 ships Unicode 13, where one target
+  (`U+2B739`) is not yet assigned. Folding is unaffected (it is a plain code point
+  substitution), but `unicodedata.name()` on that result will come up empty.
+- **Existing stored text is not retroactively repaired.** Fold on write; if you
+  have already indexed unfolded text, re-ingest or re-fold it.
+- Characters above the BMP are written as `\U000xxxxx` in the table. A `\uXXXX`
+  escape silently truncates there and turns one character into two — the table
+  test suite guards against exactly that regression.
+
+## Regenerating the table
+
+`src/cjkfold/_table.py` is generated, never hand-edited:
+
+```bash
+python scripts/build_table.py                  # fetch from unicode.org
+python scripts/build_table.py CJKRadicals.txt  # or use a local copy
+```
+
+## Development
+
+```bash
+uv venv .venv --python 3.9
+uv pip install -e ".[dev]"
+.venv/bin/python -m pytest
+```
+
+## 中文简述
+
+PDF 抽出来的中文常常混着**形近异码**字符：`⻆` 和 `角` 长得一样、码位不同，于是
+`"颜色" in 文本` 返回 `False`，检索排错、去重失效、缓存打不中。
+
+`unicodedata.normalize("NFKC", ...)` 只能解决康熙部首区（214 个全都能还原），
+**解决不了 CJK 部首补充区**——那个区 115 个字符里 NFKC 只救得了 2 个。
+
+`cjkfold` 用 Unicode 官方的 `EquivalentUnifiedIdeograph` 属性补上这个缺口，
+115 个里能还原 114 个。同时提供 `exact_boost()`，把"查询片段在文档里逐字命中"
+补进向量相似度，避免短文档靠余弦虚高把真正命中的长文档挤下去。
+
+## License
+
+MIT

cjkfold-0.1.0/README.md

@@ -0,0 +1,218 @@
+# cjkfold
+
+Text pulled out of a PDF often contains characters that **look** like ordinary
+Han characters but are not them. Comparison against them quietly fails.
+
+```
+⻆  U+2EC6  CJK RADICAL SIMPLIFIED HORN   renders as  角  U+89D2
+⾊  U+2F8A  KANGXI RADICAL COLOR          renders as  色  U+8272
+⺮  U+2EAE  CJK RADICAL BAMBOO            folds to    𥫗  U+25AD7
+```
+
+`cjkfold` folds them back onto the characters you meant:
+
+```python
+from cjkfold import fold
+
+fold("标准颜⾊值与⻆色按钮")
+# '标准颜色值与角色按钮'
+
+"颜色" in "标准颜⾊值"          # False  ← the bug
+"颜色" in fold("标准颜⾊值")     # True
+```
+
+Zero runtime dependencies. Pure functions. Python 3.9+.
+
+---
+
+**背景阅读 / Background:** [PDF 抽出来的中文，为什么搜不到](docs/why-cjk-text-breaks.md)
+—— 完整的排查经过（中文），包括我自己踩的四个坑。
+社区版：[掘金](https://juejin.cn/post/7695671592383807542)
+
+**Built on this:** [corpusfold](https://github.com/cdcailf/corpusfold) — this
+character layer wired into a full PDF → knowledge-base pipeline (抽取 / 分栏 /
+表格 / 分块 / 混合检索).
+
+## The problem
+
+Nothing raises. The text displays correctly, because the glyphs really are
+identical. What breaks is everything downstream that compares characters:
+
+- `"紫色" in text` returns `False` for a document that shows 紫色
+- a keyword search misses
+- a vector retriever ranks an unrelated *short* document above the one that
+  contains the query verbatim, because a short document's embedding is tighter
+  and scores a higher cosine — measured on a real corpus, the relevant document
+  scored **0.29** against a decoy's **0.36**
+- deduplication, caching and ground-truth checks all start disagreeing
+
+## Why NFKC is not enough
+
+The reflex is `unicodedata.normalize("NFKC", text)`. It handles the **Kangxi
+Radicals** block completely — all 214 characters carry compatibility
+decompositions — and then stops:
+
+| Block | Characters | Repaired by NFKC | Repaired by cjkfold |
+|---|---|---|---|
+| Kangxi Radicals `U+2F00–U+2FDF` | 214 | **214** | 214 |
+| CJK Radicals Supplement `U+2E80–U+2EFF` | 115 | **2** | **114** |
+
+Unicode withholds compatibility mappings in the supplement block *on purpose*:
+those glyphs are distinct encoded radicals, not compatibility variants of an
+existing character. So NFKC leaves 113 of the 115 standing.
+
+`cjkfold` closes that gap with the `EquivalentUnifiedIdeograph` property from
+the Unicode Character Database (Unicode 15.1+), which maps 114 of them onto a
+unified ideograph. The one remainder, `⺀ U+2E80 CJK RADICAL REPEAT`, has no
+ideograph equivalent and is left alone.
+
+## Install
+
+Not published on PyPI yet, so install from a checkout:
+
+```bash
+git clone https://github.com/cdcailf/cjkfold
+cd cjkfold
+uv venv .venv --python 3.11
+uv pip install -e .
+```
+
+Once it is published this collapses to `pip install cjkfold`.
+
+## Usage
+
+```python
+from cjkfold import fold, has_lookalikes, iter_lookalikes
+
+fold("紫⾊的⻆落")            # '紫色的角落'
+fold("普通文本")              # unchanged; returns the same object
+
+has_lookalikes("⻆")           # True
+has_lookalikes("角")           # False
+
+for item in iter_lookalikes("第一行\n第⼆行⻆"):
+    print(item)
+# 2:2  ⼆ U+2F06 -> 二 U+4E8C  (KANGXI RADICAL TWO)
+# 2:4  ⻆ U+2EC6 -> 角 U+89D2  (CJK RADICAL SIMPLIFIED HORN)
+```
+
+`fold()` is idempotent, and it is deliberately **not** an NFKC pass: it will
+not touch full-width punctuation (`，` `；` `（` `）`), compatibility characters
+or ligatures, so it is safe to apply to text that is still going to be stored or
+displayed. Combine with NFKC yourself if you want both:
+
+```python
+import unicodedata
+clean = fold(unicodedata.normalize("NFKC", text))
+```
+
+### Command line
+
+```bash
+cjkfold check document.txt      # report look-alikes, exit 1 if any (CI-friendly)
+cjkfold check --quiet docs/*.txt
+cjkfold fold document.txt       # print the folded text
+cjkfold fold -w document.txt    # rewrite in place
+cat document.txt | cjkfold check -
+```
+
+```
+$ cjkfold check extracted.txt
+extracted.txt: 3 look-alike character(s) (1 kangxi, 2 radical-supplement)
+    1:4  ⾊ U+2F8A -> 色 U+8272  (KANGXI RADICAL COLOR)
+    1:7  ⻆ U+2EC6 -> 角 U+89D2  (CJK RADICAL SIMPLIFIED HORN)
+    2:5  ⺮ U+2EAE -> 𥫗 U+25AD7  (CJK RADICAL BAMBOO)
+```
+
+Drop `cjkfold check` into an ingest pipeline to catch extraction regressions
+before they reach the index.
+
+## Retrieval helper
+
+Folding is the fix; ranking is the follow-up. Dense retrieval is bad at literal
+tokens, so `exact_boost` adds a bonus for query n-grams that appear verbatim in a
+document:
+
+```python
+from cjkfold import exact_boost
+
+score = vector_similarity(query, document) + exact_boost(query, document)
+```
+
+Both sides are folded inside the helper. That is not a convenience — folding the
+index but not the query (or the reverse) silently disables the whole mechanism,
+because nothing ever matches. The API gives you no way to get it wrong.
+
+Queries shorter than the shingle size fall back to a containment test, so a
+two-character query is not silently scored zero.
+
+## Wiring it into a pipeline
+
+Three touch points, zero framework dependencies:
+
+```python
+from cjkfold import fold, fold_documents, rerank
+
+# 1. on the way in -- clean the corpus before it is embedded
+documents = fold_documents(loader.load())
+
+# 2. on the way out -- fold the query so it can match
+hits = vectorstore.similarity_search_with_score(fold(question), k=20)
+
+# 3. before you trust the order -- restore the verbatim matches
+documents, scores = zip(*hits)
+for document, score in rerank(question, documents, scores):
+    ...
+```
+
+`fold_documents` and `rerank` recognise LangChain documents (`page_content`),
+LlamaIndex nodes (`text`) and plain strings -- and import none of them. Making
+you install a framework in order to fold a string would be a bad trade:
+framework APIs move, string folding does not.
+
+## Caveats
+
+- **Glyph folding only.** No traditional⇄simplified conversion, no z-variant
+  normalisation. Those are different problems with different trade-offs.
+- **Table follows the latest UCD**, which can run ahead of the `unicodedata`
+  database bundled with an older Python — 3.9 ships Unicode 13, where one target
+  (`U+2B739`) is not yet assigned. Folding is unaffected (it is a plain code point
+  substitution), but `unicodedata.name()` on that result will come up empty.
+- **Existing stored text is not retroactively repaired.** Fold on write; if you
+  have already indexed unfolded text, re-ingest or re-fold it.
+- Characters above the BMP are written as `\U000xxxxx` in the table. A `\uXXXX`
+  escape silently truncates there and turns one character into two — the table
+  test suite guards against exactly that regression.
+
+## Regenerating the table
+
+`src/cjkfold/_table.py` is generated, never hand-edited:
+
+```bash
+python scripts/build_table.py                  # fetch from unicode.org
+python scripts/build_table.py CJKRadicals.txt  # or use a local copy
+```
+
+## Development
+
+```bash
+uv venv .venv --python 3.9
+uv pip install -e ".[dev]"
+.venv/bin/python -m pytest
+```
+
+## 中文简述
+
+PDF 抽出来的中文常常混着**形近异码**字符：`⻆` 和 `角` 长得一样、码位不同，于是
+`"颜色" in 文本` 返回 `False`，检索排错、去重失效、缓存打不中。
+
+`unicodedata.normalize("NFKC", ...)` 只能解决康熙部首区（214 个全都能还原），
+**解决不了 CJK 部首补充区**——那个区 115 个字符里 NFKC 只救得了 2 个。
+
+`cjkfold` 用 Unicode 官方的 `EquivalentUnifiedIdeograph` 属性补上这个缺口，
+115 个里能还原 114 个。同时提供 `exact_boost()`，把"查询片段在文档里逐字命中"
+补进向量相似度，避免短文档靠余弦虚高把真正命中的长文档挤下去。
+
+## License
+
+MIT

cjkfold-0.1.0/pyproject.toml

@@ -0,0 +1,48 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "cjkfold"
+version = "0.1.0"
+description = "Fold look-alike CJK glyphs (Kangxi radicals, CJK radical supplements) so text extracted from PDFs matches again."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+authors = [{ name = "cdcailf" }]
+keywords = [
+    "cjk", "chinese", "unicode", "nfkc", "normalization",
+    "pdf-extraction", "rag", "retrieval", "text-matching",
+]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "License :: OSI Approved :: MIT License",
+    "Natural Language :: Chinese (Simplified)",
+    "Operating System :: OS Independent",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Topic :: Text Processing :: Linguistic",
+]
+
+[project.urls]
+Homepage = "https://github.com/cdcailf/cjkfold"
+Repository = "https://github.com/cdcailf/cjkfold"
+Issues = "https://github.com/cdcailf/cjkfold/issues"
+
+[project.scripts]
+cjkfold = "cjkfold.cli:main"
+
+[project.optional-dependencies]
+dev = ["pytest>=7"]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+addopts = "-q"

cjkfold-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

cjkfold-0.1.0/src/cjkfold/__init__.py

@@ -0,0 +1,36 @@
+"""cjkfold -- fold look-alike CJK glyphs so extracted text matches again.
+
+    >>> from cjkfold import fold
+    >>> fold("⻆⺁⼀") == "角厂一"
+    True
+
+See :mod:`cjkfold.normalize` for the background, :mod:`cjkfold.score` for the
+retrieval-ranking helper that pairs with it, and :mod:`cjkfold.integrations` for
+framework-agnostic adapters that fold a whole document set.
+"""
+
+from __future__ import annotations
+
+from ._table import EQUIVALENT_IDEOGRAPH
+from .integrations import fold_document, fold_documents, rerank, text_of
+from .normalize import Lookalike, fold, has_lookalikes, iter_lookalikes, kind_of
+from .score import exact_boost, ngram_overlap, ngrams
+
+__version__ = "0.1.0"
+
+__all__ = [
+    "fold",
+    "has_lookalikes",
+    "iter_lookalikes",
+    "Lookalike",
+    "kind_of",
+    "ngrams",
+    "ngram_overlap",
+    "exact_boost",
+    "text_of",
+    "fold_document",
+    "fold_documents",
+    "rerank",
+    "EQUIVALENT_IDEOGRAPH",
+    "__version__",
+]

cjkfold-0.1.0/src/cjkfold/__main__.py

@@ -0,0 +1,3 @@
+from .cli import main
+
+raise SystemExit(main())