cjk-semantic-split 0.1.3__tar.gz → 0.2.0__tar.gz

{cjk_semantic_split-0.1.3 → cjk_semantic_split-0.2.0}/.gitignore

@@ -4,6 +4,8 @@ __pycache__/
 *$py.class
 *.so
 .Python
+
+# Build / distribution / packaging
 build/
 develop-eggs/
 dist/
@@ -17,17 +19,20 @@ sdist/
 var/
 wheels/
 *.egg-info/
-.installed.cfg
+*.egg-info
 *.egg
 MANIFEST
+.installed.cfg
 
 # Virtual envs
 .venv/
 venv/
 env/
 ENV/
+.env
+.env.local
 
-# Test / coverage
+# Test / coverage / type-checker caches
 .pytest_cache/
 .coverage
 .coverage.*
@@ -35,10 +40,34 @@ htmlcov/
 .tox/
 .cache
 coverage.xml
+coverage.txt
 *.cover
+.hypothesis/
+.mypy_cache/
+.pyre/
+.pytype/
+.ruff_cache/
 
-# IDE
+# IDEs / editors
 .idea/
 .vscode/
+.vscode-test/
 *.swp
+*.swo
+*~
 .DS_Store
+Thumbs.db
+
+# AI tooling session data
+.aider*
+.claude/
+.superpowers/
+
+# Project-specific scratch
+/dist/
+
+# Logs / runtime
+*.log
+*.pid
+*.seed
+*.pid.lock

{cjk_semantic_split-0.1.3 → cjk_semantic_split-0.2.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cjk-semantic-split
-Version: 0.1.3
+Version: 0.2.0
 Summary: Recursive atomic decomposition of CJK characters with 9-grid 2-digit spatial encoding. 将汉字递归拆解为原子部件，并在 9 宫格 2 位数字空间编码中标记每个组件的位置。
 Project-URL: Source, https://github.com/yueguanyu/Semantic-Character-Splitting-and-Structural-Coding-for-Chinese-Script.git
 Project-URL: Bug Tracker, https://github.com/yueguanyu/Semantic-Character-Splitting-and-Structural-Coding-for-Chinese-Script/issues

{cjk_semantic_split-0.1.3 → cjk_semantic_split-0.2.0}/docs/superpowers/specs/2026-07-16-cjk-semantic-split-design.md

@@ -25,10 +25,11 @@ A Python library that, given any CJK Unified Ideograph, returns a flat DSL strin
 
 ### 1.3 Reference Standards & Sources
 
-- ISO/IEC 10646 IDC (Ideographic Description Characters): `吅 吕 罒 目 回 冂 凵 匚 厂 勹 匕 .` — repurposed as initial structure inspiration
-- Wikimedia Commons:Chinese_characters_decomposition — primary data source for 2nd & 3rd layer harvesting
-- Unihan Database (`kIDS` field) — primary source for the 1st layer and base coverage
-- CHISE project — reference for canonical primitive set
+- ISO/IEC 10646 IDC (Ideographic Description Characters): `吅 吕 罒 目 回 冂 凵 匚 厂 勹 匕 .` — repurposed as initial structure inspiration; same notation appears as Unicode IDC `⿰ ⿱ ⿲ ⿳ ⿴ ⿵ ⿶ ⿷ ⿸ ⿹ ⿺ ⿻ ⿼ ⿽ ⿾ ⿿`.
+- Wikimedia Commons:Chinese_characters_decomposition — visual decomposition **principle** (primitives are visually irreducible units, not structurally minimal); also the planned source for Tier 2/3 harvesting.
+- **yi-bai/ids** (MIT-licensed) — current **operational** source for Tier 1 (~82k chars); provides the same IDC notation that Unihan `kIDS` historically used.
+- Unihan Database (`kIDS` field) — historically the primary Tier 1 source; no longer included as a separate file in current Unicode distributions (the `kIDS` field was removed from Unihan.txt in 9.x+).
+- CHISE project — historical reference for canonical primitive sets.
 
 ---
 
@@ -91,16 +92,18 @@ Full table (`P ⊙ Q → R`). `-` indicates a geometrically invalid combination
 |  P＼Q  | 00 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 9X |
 |-------|----|----|----|----|----|----|----|----|----|----|
 | **00**| 00 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | 9X |
-| **10**| 10 | 10 | -  | 50 | 60 | 50 | 60 | -  | -  | -  |
-| **20**| 20 | -  | 20 | 70 | 80 | -  | -  | 70 | 80 | -  |
-| **30**| 30 | 50 | 70 | 30 | 30 | 50 | 50 | 70 | 70 | -  |
-| **40**| 40 | 60 | 80 | 40 | 40 | 60 | 60 | 80 | 80 | -  |
-| **50**| 50 | 50 | 50 | 50 | 50 | 50 | 50 | 70 | 70 | -  |
-| **60**| 60 | 60 | 60 | 60 | 60 | 60 | 60 | 80 | 80 | -  |
-| **70**| 70 | 50 | 70 | 70 | 70 | 50 | 50 | 70 | 70 | -  |
-| **80**| 80 | 60 | 80 | 80 | 80 | 60 | 60 | 80 | 80 | -  |
+| **10**| 10 | 10 | -  | 50 | 60 | 50 | 60 | -  | -  | 9X |
+| **20**| 20 | -  | 20 | 70 | 80 | -  | -  | 70 | 80 | 9X |
+| **30**| 30 | 50 | 70 | 30 | 30 | 50 | 50 | 70 | 70 | 9X |
+| **40**| 40 | 60 | 80 | 40 | 40 | 60 | 60 | 80 | 80 | 9X |
+| **50**| 50 | 50 | 50 | 50 | 50 | 50 | 50 | 70 | 70 | 9X |
+| **60**| 60 | 60 | 60 | 60 | 60 | 60 | 60 | 80 | 80 | 9X |
+| **70**| 70 | 50 | 70 | 70 | 70 | 50 | 50 | 70 | 70 | 9X |
+| **80**| 80 | 60 | 80 | 80 | 80 | 60 | 60 | 80 | 80 | 9X |
 | **9X**| 00 | 10 | 20 | 30 | 40 | 50 | 60 | 70 | 80 | -  |
 
+**Note on the `9X` columns in rows 10–80**: spec §2.2 only formalized "surround parent passes children through" (the row `9X`). Real CJK data nests surrounds inside directional parents (e.g. `明` contains `日`, which decomposes as `⿴口一` — surround-with-content inside a left-zone). The implementation adds the **symmetric** rule "a surround child of any parent retains its own slot" so the table is closed for the surround column too. This is documented in `src/chinese_decompose/structures.py` near `COMPOSE_TABLE`.
+
 **Worked example** (踩):
 ```
 踩 → 足 at 30 + 采 at 40
@@ -109,7 +112,7 @@ Recurse:
   足_final = 30 (already primitive-leaf)
   爪_final = 40 ⊙ 10 = 60 (top-right)
   木_final = 40 ⊙ 20 = 80 (bottom-right)
-DSL: 踩[30]:足[60]:爪[80]:木
+DSL: 踩 [30]:足 [60]:爪 [80]:木
 ```
 
 ---
@@ -118,13 +121,16 @@ DSL: 踩[30]:足[60]:爪[80]:木
 
 ### 3.1 Definition
 
-A primitive is a non-decomposable semantic component. Examples:
+A primitive is a non-decomposable semantic component — a visual atom that the walker treats as a leaf. Examples:
 
 ```
 木 火 土 金 水 人 口 手 心 日 月 目 耳 鼻 舌 身 爪 戈 矢 石
 一 乙 二 三 四 五 六 七 八 九 十 丁 卜 厂 寸 干 弓 牛 犬 玉 王
+宀 冖 冫 氵 扌 忄 讠 纟 亻 刂 艹 阝 彳 凵 匚 廴 疒 攴
 ```
 
+`primitives.json` is **authoritative** for atomicity. This means: if the bundled JSON lists `日` as primitive, the walker stops at `日` even if a deeper data tier (e.g. yi-bai/ids) records `日 → ⿴口一`. The rule follows the **wikimedia commons** visual decomposition view — primitives are the smallest irreducible units, not whatever deeper structure the source suggests. See §6.3 for the runtime resolution order.
+
 ### 3.2 Storage & Override
 
 - Default set bundled as `chinese_decompose/data/primitives.json` (~250 entries).
@@ -147,12 +153,16 @@ A character is primitive if and only if it appears in the resolved primitive set
 
 ## 4. Decomposition Data Sources & Tiers
 
-| Tier | File                         | Source                        | Coverage target | Description |
-|------|------------------------------|-------------------------------|-----------------|-------------|
-| 1    | `data/decomp_l1.json`        | Unihan `kIDS`                 | ≥ 21,000 chars  | Normalized to 9-grid; covers CJK Unified base block |
-| 2    | `data/decomp_l2.json`        | wikimedia Commons + cleanup   | ≥ 22,000 chars  | Cross-checked against Tier 1; fills gaps |
-| 3    | `data/decomp_l3.json`        | wikimedia "next-layer" harvest | ≥ 80,000 chars  | Recursive harvest; supports 踩 → 足 爪 木 |
-| 4    | `data/decomp_patches.json`   | Manual curation               | dispute cases   | Last-wins overrides for ambiguous/dubious entries |
+| Tier | File                         | Source                                | Coverage target | Description |
+|------|------------------------------|---------------------------------------|-----------------|-------------|
+| 1    | `data/decomp_l1.json`        | **yi-bai/ids** (MIT, via jsDelivr CDN) | ≥ 81,000 chars  | Currently ~82k; covers CJK Unified + Ext-A |
+| 2    | `data/decomp_l2.json`        | wikimedia Commons + cleanup           | ≥ 22,000 chars  | Cross-checked against Tier 1; fills gaps |
+| 3    | `data/decomp_l3.json`        | wikimedia "next-layer" harvest       | ≥ 80,000 chars  | Recursive harvest; supports 踩 → 足 爪 木 |
+| 4    | `data/decomp_patches.json`   | Manual curation                     | dispute cases   | Last-wins overrides for ambiguous/dubious entries |
+
+Note on Unihan `kIDS`: the field that originally seeded Tier 1 was removed from current Unicode distributions (Unihan 9.x+ no longer ships a single combined `Unihan.txt`). The project switched to **yi-bai/ids** which provides the same IDC notation (⿰⿱⿲⿳⿴⿵…) and is actively maintained.
+
+Note on IDs that are skipped during harvest: 3-cell horizontal (⿲), 3-cell vertical (⿳), and overlaid (⿻/⿼/⿽) — see §11 (open questions).
 
 ### 4.1 Loader Resolution
 
@@ -196,39 +206,71 @@ The data pipeline is **not** part of the Python package; it is a separate offlin
 
 ## 5. DSL Format
 
-### 5.1 Grammar (informal EBNF)
+### 5.1 Format Convention
+
+The DSL is a **staggered** encoding: each segment's bracket carries the position of the *next* segment's character. The last leaf is bare. A single space separates every character from its following `[pos]` bracket.
 
 ```
-dsl        = char "[" pos "]" ":" component (":" component)*
-component  = char "[" pos "]"
-             | char "[" pos "]" ":" component (":" component)*   (* primitive expansion optional *)
-pos        = "00" | "10" | "20" | "30" | "40"
-           | "50" | "60" | "70" | "80"
-           | "14" | "16" | "34" | "36"
-           | "90" | "91" | "92" | "93" | "94" | "95" | "96" | "97" | "98" | "99"
+char_pos_segment = char " " "[" pos "]"     (* space is part of the format *)
+last_leaf        = char                   (* no bracket, no preceding space *)
+
+dsl_segment       = char_pos_segment (":" char_pos_segment)* ":" last_leaf
 ```
 
-Primitives terminate expansion. Non-primitives expand their one-level definition recursively, with each descendant's position computed by chaining `⊙`.
+Primitives terminate expansion — they emit only `char` (no brackets, no space).
 
-### 5.2 Examples
+The bracket carries the **next** leaf's position (not the current character's). Example walk for `踩`:
 
-| Input | Output DSL                                                |
-|-------|-----------------------------------------------------------|
-| 好    | `好[30]:女[40]:子`                                          |
-| 踩    | `踩[30]:足[60]:爪[80]:木`                                    |
-| 森    | `森[10]:木[20]:木[40]:木`                                    |
-| 回    | `回[90]:囗[00]:口`                                           |
-| 囚    | `囚[90]:囗[00]:人`                                           |
-| 目    | `目[10]: [sub-cells 14, 16] [20]` — a 4-cell vertical layout. Concrete example TBD by data fixtures. |
-| 一    | `一` (primitive, no brackets)                              |
-| ？    | `?[?]:?` (unknown char marker — see §6.2)                  |
+```
+踩 [30]:足 [60]:爪 [80]:木
+↑   ↑    ↑   ↑    ↑   ↑    ↑
+root pos   |   |   |   |   last
+           |   |   |
+           ↓   ↓   ↓
+          pos  pos  last (no bracket, no space)
+```
+
+Reading top-to-bottom:
+- `踩 [30]` — root `踩`, next leaf's position is 30 → meaning `足` will appear at composed position 30
+- `足 [60]` — first leaf `足`, next leaf's position is 60 → `爪` at position 60
+- `爪 [80]` — second leaf `爪`, next leaf's position is 80 → `木` at position 80
+- `木` — third (last) leaf, bare, no bracket
+
+The bracket offset-by-one encodes the same final positions as the explicit `tree.positions` field (`(30, 60, 80)` here) but inline for human reading.
 
-### 5.3 Multi-Character Input
+### 5.2 Grammar (informal EBNF)
+
+```
+dsl_segment = atom (":" atom)*    (* the leaf count is determined by data *)
+atom        = primitive        (* bare char, no brackets, no space *)
+           |  char " " "[" pos "]"
+primitve    = the char itself (no expansion needed)
+
+pos = "00" | "10" | "20" | "30" | "40"
+    | "50" | "60" | "70" | "80"
+    | "14" | "16" | "34" | "36"
+    | "90" | "91" | "92" | "93" | "94" | "95" | "96" | "97" | "98" | "99"
+```
+
+The parser tolerates the same character-pos-bracket form on every segment; the final segment is bare by convention (the parser also accepts a space-then-`[` on the final segment for backward compatibility with one-level emit).
+
+### 5.3 Examples
+
+| Input | Output DSL                                          | Notes |
+|-------|-----------------------------------------------------|-------|
+| 一    | `一`                                                | primitive — bare |
+| 好    | `好 [30]:女 [40]:子`                                    | 2 leaves, root + 2 atom segments |
+| 踩    | `踩 [30]:足 [60]:爪 [80]:木`                                | 3 leaves, intermediate 采 absorbed into final composed positions |
+| 森    | `森 [10]:木 [20]:木 [40]:木`                                | 3 leaves, vertical 3-cell |
+| 国    | `国 [90]:囗 [0]:玉`                                       | surround + content |
+| ？    | `?[?]:?`                                             | unknown-char marker (non-strict mode) — fixed sentinel, brackets use literal `?` |
+
+### 5.4 Multi-Character Input
 
 For an input string of length > 1, per-character DSL segments are joined by `|`:
 
 ```
-decompose("踩好")   →  踩[30]:足[60]:爪[80]:木|好[30]:女[40]:子
+decompose("踩好")   →  踩 [30]:足 [60]:爪 [80]:木|好 [30]:女 [40]:子
 ```
 
 ---
@@ -238,29 +280,36 @@ decompose("踩好")   →  踩[30]:足[60]:爪[80]:木|好[30]:女[40]:子
 ```python
 from chinese_decompose import (
     decompose,
+    read_structure,
     DecompositionTree,
-    DSLParser,
+    parse_dsl,
+    serialize_tree,
     set_primitives,
-    PrimitiveSet,
     get_primitives,
 )
 
-# Primary API
-decompose("踩")                              # → str DSL
-decompose("踩好")                            # → str DSL (multi-char)
-decompose("踩", as_tree=True)                # → DecompositionTree
+# Primary APIs
+decompose("踩")                              # → str DSL (staggered format with spaces)
+decompose("踩好")                            # → multi-char DSL joined by '|'
+decompose("踩", as_tree=True)                # → list[DecompositionTree] (FLAT: intermediates absorbed)
+
+# Structural read (NESTED — preserves intermediate non-leaf nodes)
+read_structure("踩")                         # → list[DecompositionTree]
+# trees[0].children[1] is 采 (intermediate node preserved)
+# trees[0].children[1].position == 40 (its slot in 踩's frame)
 
 # Programmatic access
-tree = decompose("踩", as_tree=True)
+tree = read_structure("踩")[0]
 tree.char           # '踩'
-tree.positions      # [30, 60, 80]
-tree.atoms          # ['足', '爪', '木']
-tree.children       # list[DecompositionTree] for subtrees
-tree.to_dsl()       # round-trip DSL
+tree.atoms          # ('足', '爪', '木')   — flat list of leaves in DFS order
+tree.positions      # (30, 60, 80)        — composed final position of each leaf
+tree.children       # tuple[DecompositionTree, ...]   — may include intermediates
+tree.to_dsl()       # same as serialize_tree(tree)
 
-# Round-trip
-parsed = DSLParser.parse("踩[30]:足[60]:爪[80]:木")
-parsed.to_dict()    # {'踩': [{'char': '足', 'pos': 30}, {'char': '爪', 'pos': 60}, ...]}
+# Round-trip (parser tolerant of the staggered form)
+parsed = parse_dsl("踩 [30]:足 [60]:爪 [80]:木")
+parsed.char         # '踩'
+parsed.atoms        # ('足', '爪', '木')
 
 # Primitive control
 set_primitives({"木", "火", "水", ...})
@@ -269,18 +318,30 @@ get_primitives()    # → frozenset[str]
 
 ### 6.1 Return Types
 
-- `decompose(text: str, as_tree: bool = False) -> str | DecompositionTree`
-- `DecompositionTree` is an immutable value object (no setters).
+- `decompose(text: str, as_tree: bool = False, strict: bool = True, loader=None) -> str | list[DecompositionTree]`
+- `read_structure(text: str, strict: bool = True, loader=None) -> list[DecompositionTree]`
+- `DecompositionTree` is an immutable `@dataclass(frozen=True)` with:
+  - `char: str`
+  - `children: tuple[DecompositionTree, ...]` (empty for primitives and unknown markers)
+  - `position: int | None` (composed final position; `None` means root)
+  - Derived properties: `atoms: tuple[str, ...]`, `positions: tuple[int, ...]` (flat over DFS)
 
 ### 6.2 Exceptions
 
 | Exception                    | Raised when |
 |------------------------------|-------------|
-| `UnknownCharacterError`      | A character in `text` has no entry across all tiers. |
-| `MalformedDSLParseError`     | Parser fails on input string. |
+| `UnknownCharacterError`      | A character in `text` has no entry across all tiers (and `strict=True`). |
+| `MalformedDSLParseError`     | DSL parser fails on input string (bad position, missing colons, etc.). |
 | `InvalidPositionCodeError`   | Data file contains a code outside the valid 2-digit set (see §2.1). |
+| `CompositionError`           | `compose(parent, child)` is called with a geometrically invalid pair (per §2.3 table). |
+
+Unknown character handling: with `strict=False`, decompose emits `?[?]:?` marker tokens in place of unknown chars. `read_structure` emits a marker tree `DecompositionTree(char="?", children=(DecompositionTree(char="?", children=(), position=0),))`.
 
-Unknown character handling for non-strict mode: return `?[?]:?` token instead of raising. Strict mode (default for the library; non-strict for CLI) controlled by `decompose(text, strict=True)`.
+### 6.3 Caller-Atomicity Rule
+
+The bundled `primitives.json` is the **authoritative atomicity definition**: any character in it is treated as atomic by the walker, even if some data tier records a further decomposition for it. This matches the wikimedia commons visual decomposition view — primitives are the visual irreducible units, not whatever deeper IDS structure the source data suggests.
+
+Priority on lookup: `is_primitive(char)` short-circuits before loader lookup, so prims win over data (last-wins at the loader level is moot for prims). This rule is **load-bearing** for characters that appear both in `primitives.json` and in `decomp_l{1,2,3}.json` (e.g. `日`).
 
 ---
 
@@ -347,28 +408,47 @@ Semantic-Character-Splitting-and-Structural-Coding-for-Chinese-Script/
 
 ### 9.1 `python -m chinese_decompose`
 
+The default output format is **JSON**. Use `--format dsl` for the staggered DSL string.
+
 ```bash
 $ python -m chinese_decompose 踩
-踩[30]:足[60]:爪[80]:木
+[
+  {
+    "char": "踩",
+    "atoms": ["足", "爪", "木"],
+    "positions": [30, 60, 80],
+    "dsl": "踩 [30]:足 [60]:爪 [80]:木"
+  }
+]
+
+$ python -m chinese_decompose 踩 --format dsl
+踩 [30]:足 [60]:爪 [80]:木
 
 $ python -m chinese_decompose 囚 --format json
-{"char": "囚", "atoms": ["囗", "人"], "positions": [90, 00], "dsl": "囚[90]:囗[00]:人"}
+[
+  {
+    "char": "囚",
+    "atoms": ["囗", "人"],
+    "positions": [90, 0],
+    "dsl": "囚 [90]:囗 [0]:人"
+  }
+]
 
 $ python -m chinese_decompose --self-test
-Coverage: 32784 chars indexed across 4 tiers
+chinese_decompose self-test
+  version: 0.1.3
+  characters indexed: 81686
+  tiers: 4 (l1, l2, l3, patches)
 ```
 
-Flags: `--format {dsl,json,tree}`, `--no-strict`, `--primitives <file>`.
+Flags: `--format {json,dsl,tree}`, `--no-strict`, `--primitives <file>`.
 
 ### 9.2 Offline Harvest Script (`scripts/harvest.py`)
 
-- Reads Unihan `Unihan_IRGSources.txt` (or its `kIDS` extract).
-- Reads wikimedia Commons decomposition XML.
-- Normalizes to data format.
-- Emits `decomp_l{1,2,3}.json` next to sources.
-- Logs tier conflicts to `scripts/conflicts.log`.
+- **Tier 1** (currently implemented): downloads **yi-bai/ids** (MIT-licensed, ~102k entries) via the jsDelivr CDN mirror, parses TSV-formatted IDC notation, and writes `decomp_l1.json`. Per the §6.3 rule, it never strips primitives from `primitives.json` — it only **adds** new leaves it discovered.
+- **Tier 2/3**: planned (wikimedia Commons scrape + next-layer recursive harvest); returns `NotImplementedError` if invoked with `--tier l2` or `--tier l3` today.
 
-This script is **not** part of the installable package; it lives in `scripts/` and is invoked manually when refreshing data.
+This script is **not** part of the installable package; it lives in `scripts/` and is invoked manually when refreshing data. It requires network access on first run.
 
 ---
 
@@ -386,8 +466,20 @@ This script is **not** part of the installable package; it lives in `scripts/` a
 
 ## 11. Open Questions & Future Work
 
+### Shipped (since the spec was written)
+
+- DSL format finalized: **staggered bracket + space between char-and-position** (`踩 [30]:足 [60]:爪 [80]:木`); last leaf bare.
+- CLI default format is **JSON**; `--format dsl` for the string form.
+- **`read_structure()`** API added: returns `list[DecompositionTree]` that preserves **intermediate non-leaf nodes** (e.g. `采` inside `踩`), unlike `decompose()` which collapses them into a flat atom list.
+- **Real-data harvest (v0.1.3)**: `scripts/harvest.py` now pulls ~82k characters from yi-bai/ids; bundled in `decomp_l1.json` at install time.
+- **Caller-atomicity rule** (§6.3): `primitives.json` is the authoritative primitive set; characters in it stay atomic even when layered data says otherwise.
+
+### Still open
+
 - **Position grammar extensions**: future versions may add codes for repeated patterns (e.g. repetitive 吕 罒 structures) as sub-positions.
-- **Bidirectional mapping**: `compose(characters_from_atoms(...))` for the inverse direction — not in v0.1.
-- **Heuristic decomposer**: for characters with no data entry, attempt primitive grouping — out of scope for v0.1.
-- **CJK Extensions B–G**: properly indexing beyond CJK Unified base block — relies on Tier 3 harvest quality.
-- **Performance**: prototype assumed sub-millisecond per character; v0.1 ships without benchmarks, defer to v0.2 if needed.
+- **3-cell IDC handling**: `⿲` (LMR) and `⿳` (TMB) entries are currently **skipped** by the harvester (out of scope for the 2-digit schema). They are needed for full coverage.
+- **Overlaid IDC handling**: `⿻`/`⿼`/`⿽` patterns skipped.
+- **Bidirectional mapping**: `compose(atoms → char)` inverse — not in v0.x.
+- **Heuristic decomposer**: for characters with no data entry, attempt primitive grouping — out of scope.
+- **CJK Extensions B–G yi-bai coverage**: ~82k chars covers most of the Unified Ideographs block + Extension A but doesn't sweep Extensions B–F systematically. Tiers 2/3 (wikimedia scrape + recursive next-layer) remain unimplemented.
+- **Performance**: prototype assumed sub-millisecond per character; v0.1.3 ships without benchmarks, defer to v0.2 if needed.

{cjk_semantic_split-0.1.3 → cjk_semantic_split-0.2.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "cjk-semantic-split"
-version = "0.1.3"
+version = "0.2.0"
 description = "Recursive atomic decomposition of CJK characters with 9-grid 2-digit spatial encoding. 将汉字递归拆解为原子部件，并在 9 宫格 2 位数字空间编码中标记每个组件的位置。"
 readme = "README.md"
 requires-python = ">=3.9"
@@ -40,7 +40,7 @@ dev = [
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/chinese_decompose"]
-include = ["src/chinese_decompose/data/*.json"]
+include = ["src/chinese_decompose/data/*.txt"]
 
 [tool.pytest.ini_options]
 testpaths = ["tests"]

{cjk_semantic_split-0.1.3 → cjk_semantic_split-0.2.0}/scripts/harvest.py

@@ -7,7 +7,7 @@ notation (⿰⿱⿴⿵⿶⿷⿸⿹⿺⿾⿿, plus 3-cell and overlaid variants w
 
 Usage:
     python -m scripts.harvest --tier l1 \
-        --output ../src/chinese_decompose/data/decomp_l1.json
+        --output ../src/chinese_decompose/data/decomp_l1.txt
 
 Source file format (tab-separated):
 
@@ -153,6 +153,11 @@ def _walk_kids(
             return False
         composed = compose_fn(parent_position, local_pos)
         if len(comp_str) == 1:
+            # Reject leaves that are themselves IDC codes (e.g. degenerate
+            # yi-bai entries where ⿰ appears as a slot — that's a parse
+            # artifact, not a real atom).
+            if comp_str in IDC_TABLE_2_CELL:
+                return False
             out.append((comp_str, composed))
         else:
             # Recurse into subtree
@@ -221,8 +226,40 @@ def parse_ids_source(text: str, compose_fn) -> tuple[dict[str, list[tuple[str, i
     return entries, refs
 
 
+def _serialize_dsl(char: str, comps: list[tuple[str, int]]) -> str:
+    """Build the staggered DSL string for a character + its components.
+
+    Matches the format used by ``DecompositionTree.to_dsl()``:
+    "踩 [30]:足 [60]:爪 [80]:木" — the bracket carries the *next* leaf's
+    position; the last leaf is bare. A primitive char with no children
+    yields just the bare char.
+    """
+    if not comps:
+        return char  # primitive — bare char
+    parts = [f"{char} [{comps[0][1]}]"]
+    leaves = [c for c, _ in comps]
+    leaf_positions = [p for _, p in comps]
+    for i in range(len(leaves) - 1):
+        parts.append(f"{leaves[i]} [{leaf_positions[i + 1]}]")
+    parts.append(leaves[-1])
+    return ":".join(parts)
+
+
+def _load_existing_primitives_txt(path: Path) -> set[str]:
+    if not path.exists():
+        return set()
+    chars: set[str] = set()
+    for line in path.read_text(encoding="utf-8").splitlines():
+        line = line.strip()
+        if not line or line.startswith("#"):
+            continue
+        if len(line) == 1:
+            chars.add(line)
+    return chars
+
+
 def emit_tier1(output_path: Path, compose_fn) -> dict:
-    """Download IDS dataset, parse, write decomp_l1.json and extend primitives.json."""
+    """Download IDS dataset, parse, write decomp_l1.txt and extend primitives.txt."""
     print(f"Downloading {IDS_SOURCES['lv0']}...", file=sys.stderr)
     text = fetch_text(IDS_SOURCES["lv0"])
     print(f"  fetched {len(text.splitlines())} lines", file=sys.stderr)
@@ -232,18 +269,26 @@ def emit_tier1(output_path: Path, compose_fn) -> dict:
     print(f"  entries: {len(entries)}, primitive refs: {len(refs)}", file=sys.stderr)
 
     output_path.parent.mkdir(parents=True, exist_ok=True)
-    payload = {"_meta": {"tier": 1, "source": "github.com/yi-bai/ids", "license": "MIT"}}
-    payload.update({char: [[c, p] for c, p in v] for char, v in entries.items()})
-    output_path.write_text(
-        json.dumps(payload, ensure_ascii=False, separators=(",", ":")), encoding="utf-8"
-    )
+
+    # Emit Tier 1 as TXT (DSL format, one line per char). Format:
+    #   <char>\t<DSL>
+    # Blank lines and lines starting with `#` are comments.
+    lines = [
+        f"# Tier 1 decomposition data harvested from yi-bai/ids (MIT) via jsDelivr",
+        "# Format per line: <char>\\t<DSL>",
+        "# DSL is the staggered form: each bracket carries the NEXT atom's",
+        "# position; the last atom is bare. Lines starting with # are comments.",
+        "# Empty DSL after the tab means the character is primitive at this tier.",
+    ]
+    for char in sorted(entries):
+        lines.append(f"{char}\t{_serialize_dsl(char, entries[char])}")
+    output_path.write_text("\n".join(lines) + "\n", encoding="utf-8")
     size_kb = output_path.stat().st_size // 1024
     print(f"  wrote {output_path} ({size_kb} KB)", file=sys.stderr)
 
     # Find all characters that get further decomposed in `entries`
-    # (recursively, since chars may appear nested). These MUST be removed
-    # from the primitive set so walker behavior is unambiguous — if a char
-    # is in both primitives and l1, l1 wins (per last-wins loader order).
+    # (recursively, since chars may appear nested). These MUST NOT be
+    # promoted into the primitive set.
     def _collect_decomposable(char: str, seen: set[str], into: set[str]) -> None:
         if char in seen:
             return
@@ -260,33 +305,28 @@ def emit_tier1(output_path: Path, compose_fn) -> dict:
     for char in entries:
         _collect_decomposable(char, set(), decomposable)
 
-    # Primitive leaves: characters in `refs` that NEVER get decomposed (i.e.,
-    # not in the `decomposable` set we just built). These stay in the
-    # primitive set; the rest get removed.
     leaves_without_decomp = {
         c for c in refs if c not in decomposable and _is_data_char(c)
     }
 
-    # Append new primitives to primitives.json. The bundled primitives.json is
-    # the AUTHORITATIVE atomicity definition (per wikimedia commons' visual
-    # decomposition view): any character in it is treated as atomic, even if
-    # the IDS source happens to have a further decomposition. We only ADD
-    # new leaves — never remove from the curated set.
+    # Append new leaves to primitives.txt. Per the wikimedia commons
+    # atomicity principle (§6.3 of the spec), primitives.txt is the
+    # authoritative atomicity definition — we only ADD new leaves; never
+    # remove from the curated set.
     primitives_path = (
-        Path(__file__).parent.parent / "src" / "chinese_decompose" / "data" / "primitives.json"
+        Path(__file__).parent.parent / "src" / "chinese_decompose" / "data" / "primitives.txt"
     )
-    if primitives_path.exists():
-        existing = set(json.loads(primitives_path.read_text(encoding="utf-8"))["primitives"])
-    else:
-        existing = set()
+    existing = _load_existing_primitives_txt(primitives_path)
     new_primitives = sorted(existing | leaves_without_decomp)
+    primitives_lines = [
+        "# chinese-decompose primitives (authoritative atomic components per wikimedia commons view)",
+        "# Format: one character per line. Lines starting with # are comments.",
+        "# Characters listed here are treated as atomic by the walker regardless",
+        "# of any deeper structure recorded in decomp_*.txt.",
+    ]
+    primitives_lines += new_primitives
     primitives_path.write_text(
-        json.dumps(
-            {"primitives": new_primitives},
-            ensure_ascii=False,
-            separators=(",", ":"),
-        ),
-        encoding="utf-8",
+        "\n".join(primitives_lines) + "\n", encoding="utf-8"
     )
     added = len(new_primitives) - len(existing)
     print(

{cjk_semantic_split-0.1.3 → cjk_semantic_split-0.2.0}/src/chinese_decompose/__init__.py

@@ -1,5 +1,5 @@
 """chinese_decompose: recursive atomic decomposition of CJK characters."""
-__version__ = "0.1.3"
+__version__ = "0.2.0"
 
 from chinese_decompose.api import (
     DecompositionTree,