khammoon 0.1.0__tar.gz

khammoon-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

khammoon-0.1.0/NOTICE

@@ -0,0 +1,18 @@
+khammoon
+Copyright 2026 The khammoon authors
+
+The source code in this repository is licensed under the Apache License,
+Version 2.0 (see LICENSE).
+
+The trained model weights in khammoon/model/ (model.onnx, weights.npz) are
+licensed separately under Creative Commons Attribution-NonCommercial-ShareAlike
+4.0 International (see khammoon/model/LICENSE), because the model was trained on
+data released under NonCommercial terms:
+  - BEST2009 corpus, NECTEC (CC BY-NC-SA 3.0)
+  - WangchanThaiInstruct, VISTEC/AIResearch (per row CC BY-NC 4.0 / CC BY-SA 4.0)
+
+This product includes software derived from PyThaiNLP
+(https://github.com/PyThaiNLP/pythainlp), licensed under the Apache License,
+Version 2.0: the Thai Character Cluster rules in khammoon/tcc.py
+(Theeramunkong et al. 2000; TCC by Jakkrit TeCho, grammar by Wittawat
+Jitkrittum, Python code by Korakot Chaovavanich).

khammoon-0.1.0/PKG-INFO

@@ -0,0 +1,225 @@
+Metadata-Version: 2.4
+Name: khammoon
+Version: 0.1.0
+Summary: Thai morpheme (คำมูล) segmenter: a 4 MB character model that runs on CPU
+License: Apache-2.0 (code); CC-BY-NC-SA-4.0 (bundled model weights)
+Project-URL: Homepage, https://github.com/warit-s/khammoon-segmenter
+Project-URL: Repository, https://github.com/warit-s/khammoon-segmenter
+Project-URL: Model, https://huggingface.co/typhoon-ai/khammoon
+Project-URL: Data, https://huggingface.co/datasets/typhoon-ai/khammoon-data
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Natural Language :: Thai
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: numpy>=1.21
+Provides-Extra: onnx
+Requires-Dist: onnxruntime>=1.15; extra == "onnx"
+Provides-Extra: train
+Requires-Dist: torch>=2.1; extra == "train"
+Requires-Dist: huggingface_hub; extra == "train"
+Requires-Dist: onnx; extra == "train"
+Provides-Extra: baselines
+Requires-Dist: pythainlp>=5; extra == "baselines"
+Requires-Dist: attacut; extra == "baselines"
+Requires-Dist: nlpo3; extra == "baselines"
+Provides-Extra: test
+Requires-Dist: pytest; extra == "test"
+Dynamic: license-file
+
+# khammoon — Thai morpheme segmenter
+
+khammoon splits Thai text into **morphemes** in the sense of Thai grammar (คำมูล, *khammoon*): the
+smallest units that still carry meaning on their own, as defined by the annotation guideline in
+[`linguistics/`](linguistics/).
+It is a 4 MB character-level model that runs on a CPU with nothing but `numpy`; no GPU and no LLM.
+
+```python
+from khammoon import segment
+
+segment("นักเรียนโรงเรียนสอนคนตาบอดทำงานขะมักเขม้น")
+# ['นัก', 'เรียน', 'โรง', 'เรียน', 'สอน', 'คน', 'ตา', 'บอด', 'ทำ', 'งาน', 'ขะมัก', 'เขม้น']
+```
+
+## What a morpheme is
+
+Ordinary Thai word segmenters keep compounds together; khammoon splits them down to their
+meaningful parts. `สินค้า` ("goods") is the compound สิน (wealth) + ค้า (trade), so it becomes
+`สิน|ค้า`, whereas `ราคา` ("price") has no meaningful parts (รา + คา) and stays whole. Following
+the guideline, Pali–Sanskrit compounds read back-to-front (`ประวัติศาสตร์`) and loanwords
+(`วิปครีม`) are not split, and foreign-script text is split only at whitespace (`COVID-19`).
+
+## Install
+
+```bash
+pip install .            # from this directory; needs only numpy
+pip install ".[onnx]"    # adds onnxruntime: about 2.4x faster, identical output
+pip install ".[train]"   # to train or evaluate (torch, pythainlp, ...)
+```
+
+| backend | dependencies | speed (1 CPU thread) |
+|---|---|---|
+| `onnx` | numpy + onnxruntime | ~30k characters/s |
+| `numpy` | numpy | ~13k characters/s |
+
+The default `backend="auto"` uses `onnx` when onnxruntime is installed and `numpy` otherwise.
+Both backends produce the same segmentation on every sentence of the development set.
+
+## Usage
+
+```python
+from khammoon import Segmenter
+
+seg = Segmenter()                                     # loads the bundled model
+seg("ไปที่ www.google.com เลย")                        # ['ไป', 'ที่', ' ', 'www.google.com', ' ', 'เลย']
+seg.segment_batch(["ข้อความแรก", "ข้อความที่สอง"])     # batching is faster than one call per text
+Segmenter(threshold=0.3)                              # lower threshold: more boundaries, higher recall
+Segmenter(backend="numpy")                            # force the pure-numpy backend
+```
+
+```bash
+khammoon "โรงเรียนสอนคนตาบอด"            # โรง|เรียน|สอน|คน|ตา|บอด
+cat corpus.txt | khammoon --sep " "      # reads stdin line by line
+```
+
+Guarantees: `"".join(tokens) == text` always holds (characters are never added, removed or
+normalised); every whitespace character is its own token; and boundaries are placed only between
+Thai Character Clusters, so no token is misspelled (for example, none starts with the vowel `า`).
+
+## Results
+
+The evaluation set is the **test set**: the Wangchan reference, 2,000 documents segmented by three
+trained annotators following the guideline and adjudicated (Fleiss κ 0.991). It was never used
+during development: not for training, model selection or threshold choice. The metric is
+boundary precision / recall / F1, pooled over all boundaries, ignoring boundaries adjacent to
+whitespace (those are fixed by the input, and the human annotation treats runs of whitespace
+inconsistently). Confidence intervals and the full discussion are in the technical report.
+
+| method | precision | recall | F1 |
+|---|---|---|---|
+| PyThaiNLP `newmm` (BEST-style words) | 0.962 | 0.687 | 0.802 |
+| nlpO3 (`newmm` in Rust) | 0.961 | 0.693 | 0.805 |
+| deepcut | 0.972 | 0.875 | 0.921 |
+| AttaCut | 0.981 | 0.874 | 0.925 |
+| syllable segmentation (PyThaiNLP `syllable_tokenize`) | 0.821 | 0.991 | 0.898 |
+| unigram Viterbi over the LLM-label lexicon | 0.975 | 0.881 | 0.925 |
+| Qwen3-30B-A3B two-pass LLM segmenter that produced the training labels | 0.975 | 0.948 | 0.962 |
+| the same BiLSTM trained on the uncorrected LLM labels | 0.977 | 0.948 | 0.962 |
+| **khammoon 0.1.0** | **0.992** | **0.993** | **0.992** |
+
+`newmm`, deepcut and AttaCut segment into BEST2009 *words*, not morphemes, so their recall is
+low; the table measures distance to the morpheme definition, not the quality of those tools
+at their own task (`scripts/baselines_test.py`). The Qwen3-30B-A3B row scores the saved outputs of
+the same LLM segmenter that labelled the training text, run on the test set texts
+(`scripts/eval_llm_outputs.py`). khammoon is about 30,000 times smaller than that LLM, runs on a
+CPU, and makes about 5 times fewer errors (1 − F1: 0.008 vs 0.038). Models trained on its output
+perform on par with those trained on word segmenters on POS tagging, NER and sentiment
+classification; see the technical report.
+
+The model has ~1.06M parameters (4.2 MB).
+
+## Data
+
+All data are released as [`typhoon-ai/khammoon-data`](https://huggingface.co/datasets/typhoon-ai/khammoon-data).
+
+| config | source | role |
+|---|---|---|
+| `llm_labelled` | BEST2009 train + WangchanThaiInstruct train, segmented by an LLM (Qwen3-30B-A3B, two passes following the guideline) and then corrected; 180,206 documents, 70M characters | training |
+| `dev` | development set: BEST2009 reference, 2,252 sentences, human-labelled | label correction, fine-tuning, model and threshold selection; split into dev-train (1,484) and dev-val (768), marked `devtrain` / `devval` in the `dev_split` field |
+| `test` | test set: Wangchan reference, 2,000 documents, human-labelled | evaluation only |
+
+Training rows whose text also occurs in either human set were removed before training. The
+cross-domain split (news, novels and encyclopedia text for development; instruction data in
+finance, legal, medical and retail for testing) is deliberate.
+
+**All of the data is open, for developers and researchers to build on.** Besides the model, we
+release the human-labelled development and test sets (a common benchmark with a fixed protocol), the 70M
+characters of LLM-labelled text with both the original LLM labels and the corrected ones (training
+data for new segmenters, and a record of where an LLM annotator goes wrong), and the LLM's outputs
+on the reference sets (the strongest baseline, reproducible without a GPU). Use them to train your
+own segmenter, compare against ours on the same test set, or study Thai morphology and
+tokenization; the licenses of the source texts apply (see [License](#license)).
+
+## Method
+
+1. **Character tagging.** The model predicts, before every character, whether a boundary
+   falls there. Boundaries are allowed only at Thai Character Cluster (TCC) edges and are forced
+   at whitespace edges; a boundary is placed where the predicted probability exceeds the
+   threshold (0.5).
+2. **Label correction, round 1.** The LLM systematically under-splits frequent Thai compounds
+   (`สินค้า`, `กฎหมาย`, `ข้อมูล`). We build a table of strings that dev-train always
+   segments the same way (at least twice) and rewrite every occurrence in the LLM labels to match, and we
+   merge foreign-script spans that the guideline keeps whole (`COVID-19`, `People's`) and
+   parenthesised heading markers (`(๒)`).
+3. **Train and fine-tune.** A BiLSTM (character embedding → convolutions with kernels 3/5/7 →
+   2-layer BiLSTM, 256-character windows) is trained on the round-1 labels and fine-tuned on
+   dev-train.
+4. **Label correction, round 2.** The fine-tuned model re-segments the whole LLM-labelled corpus;
+   only boundaries between two Thai letters are taken from it, all others keep the round-1
+   labels. This spreads the human conventions over the 70M characters of LLM-labelled text.
+5. **Mixed fine-tuning.** A new model is trained on the round-2 labels and fine-tuned with half
+   of every batch from dev-train and half from the LLM-labelled text, so it does not forget what
+   the development set lacks (it is 2009 text without `COVID-19` and splits every hyphen). The
+   mixing ratio (0% or 50% LLM-labelled data, chosen by a rule fixed beforehand: the largest
+   LLM-labelled share whose dev-val F1 is within 0.0005 of the best), the number of steps and the
+   threshold were selected on dev-val; the released model was then fine-tuned on the whole
+   development set with those values.
+6. **Export** to ONNX (`model.onnx`) and to raw weights (`weights.npz`) for the numpy backend.
+
+| dev-val F1 | before fine-tuning | human-only fine-tuning | 50% LLM-labelled fine-tuning |
+|---|---|---|---|
+| LLM labels after round 1 | 0.9916 | 0.9970 | 0.9957 |
+| LLM labels after round 2 | 0.9955 | 0.9974 | 0.9972 (selected) |
+
+Every step can be rerun from the released data; see [`scripts/README.md`](scripts/README.md). The technical report describing this work is linked from the model card on Hugging Face.
+
+## Limitations
+
+- Only one cross-domain test set; there is no human-annotated set for social-media or general
+  web text yet.
+- Most remaining errors are rare proper names and transliterations, and Pali–Sanskrit
+  compounds that are over-split.
+- The test set was evaluated three times. Each change of method was motivated by the
+  development set or by examples outside the test set, never by test results: (1) F1 0.9916;
+  (2) after fixing an inference bug found through a README example (text-final characters were
+  mis-segmented because training documents are newline-delimited, and batches contained padding),
+  0.9917; (3) after label-correction round 2 and mixed fine-tuning (prompted by `COVID-19` being
+  split), P 0.9917 / R 0.9928 / F1 0.9923, the numbers in the table.
+- The developer had seen error analyses on the Wangchan reference in an earlier experiment,
+  before this split was fixed. The foreign-script merge rule follows Step 2 of the guideline but
+  may have been influenced by what was seen.
+
+## License
+
+| part | license |
+|---|---|
+| code (all `.py` files, scripts, tests) | [Apache-2.0](LICENSE) |
+| model weights (`khammoon/model/model.onnx`, `weights.npz`) | [CC BY-NC-SA 4.0](khammoon/model/LICENSE); no commercial use, because the training data carry NonCommercial terms |
+| data (`typhoon-ai/khammoon-data`) | per row, inherited from the source: CC BY-NC-SA 3.0 (BEST2009), CC BY-NC 4.0 or CC BY-SA 4.0 (WangchanThaiInstruct) |
+
+The wheel installed by pip bundles the model weights, so that part falls under CC BY-NC-SA 4.0;
+see [NOTICE](NOTICE).
+
+## Citation
+
+If you use khammoon, the guideline or the data, please cite the technical report:
+
+```bibtex
+@techreport{khammoon2026,
+  title       = {khammoon: From a Linguistic Guideline to a Thai Morpheme Segmenter},
+  author      = {Sirichotedumrong, Warit and Na-Thalang, Adisai and Wittayasakpan, Chanakan and
+                 Phatcharoen, Kritsadha and Pipatanakul, Kunat},
+  institution = {Typhoon, SCB DataX},
+  year        = {2026},
+  url         = {https://github.com/warit-s/khammoon-segmenter}
+}
+```
+
+## Credits
+
+TCC rules ported from [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp) (Apache-2.0;
+Theeramunkong et al. 2000). Training data from the BEST2009 corpus (NECTEC) and
+WangchanThaiInstruct (VISTEC/AIResearch).

khammoon-0.1.0/README.md

@@ -0,0 +1,193 @@
+# khammoon — Thai morpheme segmenter
+
+khammoon splits Thai text into **morphemes** in the sense of Thai grammar (คำมูล, *khammoon*): the
+smallest units that still carry meaning on their own, as defined by the annotation guideline in
+[`linguistics/`](linguistics/).
+It is a 4 MB character-level model that runs on a CPU with nothing but `numpy`; no GPU and no LLM.
+
+```python
+from khammoon import segment
+
+segment("นักเรียนโรงเรียนสอนคนตาบอดทำงานขะมักเขม้น")
+# ['นัก', 'เรียน', 'โรง', 'เรียน', 'สอน', 'คน', 'ตา', 'บอด', 'ทำ', 'งาน', 'ขะมัก', 'เขม้น']
+```
+
+## What a morpheme is
+
+Ordinary Thai word segmenters keep compounds together; khammoon splits them down to their
+meaningful parts. `สินค้า` ("goods") is the compound สิน (wealth) + ค้า (trade), so it becomes
+`สิน|ค้า`, whereas `ราคา` ("price") has no meaningful parts (รา + คา) and stays whole. Following
+the guideline, Pali–Sanskrit compounds read back-to-front (`ประวัติศาสตร์`) and loanwords
+(`วิปครีม`) are not split, and foreign-script text is split only at whitespace (`COVID-19`).
+
+## Install
+
+```bash
+pip install .            # from this directory; needs only numpy
+pip install ".[onnx]"    # adds onnxruntime: about 2.4x faster, identical output
+pip install ".[train]"   # to train or evaluate (torch, pythainlp, ...)
+```
+
+| backend | dependencies | speed (1 CPU thread) |
+|---|---|---|
+| `onnx` | numpy + onnxruntime | ~30k characters/s |
+| `numpy` | numpy | ~13k characters/s |
+
+The default `backend="auto"` uses `onnx` when onnxruntime is installed and `numpy` otherwise.
+Both backends produce the same segmentation on every sentence of the development set.
+
+## Usage
+
+```python
+from khammoon import Segmenter
+
+seg = Segmenter()                                     # loads the bundled model
+seg("ไปที่ www.google.com เลย")                        # ['ไป', 'ที่', ' ', 'www.google.com', ' ', 'เลย']
+seg.segment_batch(["ข้อความแรก", "ข้อความที่สอง"])     # batching is faster than one call per text
+Segmenter(threshold=0.3)                              # lower threshold: more boundaries, higher recall
+Segmenter(backend="numpy")                            # force the pure-numpy backend
+```
+
+```bash
+khammoon "โรงเรียนสอนคนตาบอด"            # โรง|เรียน|สอน|คน|ตา|บอด
+cat corpus.txt | khammoon --sep " "      # reads stdin line by line
+```
+
+Guarantees: `"".join(tokens) == text` always holds (characters are never added, removed or
+normalised); every whitespace character is its own token; and boundaries are placed only between
+Thai Character Clusters, so no token is misspelled (for example, none starts with the vowel `า`).
+
+## Results
+
+The evaluation set is the **test set**: the Wangchan reference, 2,000 documents segmented by three
+trained annotators following the guideline and adjudicated (Fleiss κ 0.991). It was never used
+during development: not for training, model selection or threshold choice. The metric is
+boundary precision / recall / F1, pooled over all boundaries, ignoring boundaries adjacent to
+whitespace (those are fixed by the input, and the human annotation treats runs of whitespace
+inconsistently). Confidence intervals and the full discussion are in the technical report.
+
+| method | precision | recall | F1 |
+|---|---|---|---|
+| PyThaiNLP `newmm` (BEST-style words) | 0.962 | 0.687 | 0.802 |
+| nlpO3 (`newmm` in Rust) | 0.961 | 0.693 | 0.805 |
+| deepcut | 0.972 | 0.875 | 0.921 |
+| AttaCut | 0.981 | 0.874 | 0.925 |
+| syllable segmentation (PyThaiNLP `syllable_tokenize`) | 0.821 | 0.991 | 0.898 |
+| unigram Viterbi over the LLM-label lexicon | 0.975 | 0.881 | 0.925 |
+| Qwen3-30B-A3B two-pass LLM segmenter that produced the training labels | 0.975 | 0.948 | 0.962 |
+| the same BiLSTM trained on the uncorrected LLM labels | 0.977 | 0.948 | 0.962 |
+| **khammoon 0.1.0** | **0.992** | **0.993** | **0.992** |
+
+`newmm`, deepcut and AttaCut segment into BEST2009 *words*, not morphemes, so their recall is
+low; the table measures distance to the morpheme definition, not the quality of those tools
+at their own task (`scripts/baselines_test.py`). The Qwen3-30B-A3B row scores the saved outputs of
+the same LLM segmenter that labelled the training text, run on the test set texts
+(`scripts/eval_llm_outputs.py`). khammoon is about 30,000 times smaller than that LLM, runs on a
+CPU, and makes about 5 times fewer errors (1 − F1: 0.008 vs 0.038). Models trained on its output
+perform on par with those trained on word segmenters on POS tagging, NER and sentiment
+classification; see the technical report.
+
+The model has ~1.06M parameters (4.2 MB).
+
+## Data
+
+All data are released as [`typhoon-ai/khammoon-data`](https://huggingface.co/datasets/typhoon-ai/khammoon-data).
+
+| config | source | role |
+|---|---|---|
+| `llm_labelled` | BEST2009 train + WangchanThaiInstruct train, segmented by an LLM (Qwen3-30B-A3B, two passes following the guideline) and then corrected; 180,206 documents, 70M characters | training |
+| `dev` | development set: BEST2009 reference, 2,252 sentences, human-labelled | label correction, fine-tuning, model and threshold selection; split into dev-train (1,484) and dev-val (768), marked `devtrain` / `devval` in the `dev_split` field |
+| `test` | test set: Wangchan reference, 2,000 documents, human-labelled | evaluation only |
+
+Training rows whose text also occurs in either human set were removed before training. The
+cross-domain split (news, novels and encyclopedia text for development; instruction data in
+finance, legal, medical and retail for testing) is deliberate.
+
+**All of the data is open, for developers and researchers to build on.** Besides the model, we
+release the human-labelled development and test sets (a common benchmark with a fixed protocol), the 70M
+characters of LLM-labelled text with both the original LLM labels and the corrected ones (training
+data for new segmenters, and a record of where an LLM annotator goes wrong), and the LLM's outputs
+on the reference sets (the strongest baseline, reproducible without a GPU). Use them to train your
+own segmenter, compare against ours on the same test set, or study Thai morphology and
+tokenization; the licenses of the source texts apply (see [License](#license)).
+
+## Method
+
+1. **Character tagging.** The model predicts, before every character, whether a boundary
+   falls there. Boundaries are allowed only at Thai Character Cluster (TCC) edges and are forced
+   at whitespace edges; a boundary is placed where the predicted probability exceeds the
+   threshold (0.5).
+2. **Label correction, round 1.** The LLM systematically under-splits frequent Thai compounds
+   (`สินค้า`, `กฎหมาย`, `ข้อมูล`). We build a table of strings that dev-train always
+   segments the same way (at least twice) and rewrite every occurrence in the LLM labels to match, and we
+   merge foreign-script spans that the guideline keeps whole (`COVID-19`, `People's`) and
+   parenthesised heading markers (`(๒)`).
+3. **Train and fine-tune.** A BiLSTM (character embedding → convolutions with kernels 3/5/7 →
+   2-layer BiLSTM, 256-character windows) is trained on the round-1 labels and fine-tuned on
+   dev-train.
+4. **Label correction, round 2.** The fine-tuned model re-segments the whole LLM-labelled corpus;
+   only boundaries between two Thai letters are taken from it, all others keep the round-1
+   labels. This spreads the human conventions over the 70M characters of LLM-labelled text.
+5. **Mixed fine-tuning.** A new model is trained on the round-2 labels and fine-tuned with half
+   of every batch from dev-train and half from the LLM-labelled text, so it does not forget what
+   the development set lacks (it is 2009 text without `COVID-19` and splits every hyphen). The
+   mixing ratio (0% or 50% LLM-labelled data, chosen by a rule fixed beforehand: the largest
+   LLM-labelled share whose dev-val F1 is within 0.0005 of the best), the number of steps and the
+   threshold were selected on dev-val; the released model was then fine-tuned on the whole
+   development set with those values.
+6. **Export** to ONNX (`model.onnx`) and to raw weights (`weights.npz`) for the numpy backend.
+
+| dev-val F1 | before fine-tuning | human-only fine-tuning | 50% LLM-labelled fine-tuning |
+|---|---|---|---|
+| LLM labels after round 1 | 0.9916 | 0.9970 | 0.9957 |
+| LLM labels after round 2 | 0.9955 | 0.9974 | 0.9972 (selected) |
+
+Every step can be rerun from the released data; see [`scripts/README.md`](scripts/README.md). The technical report describing this work is linked from the model card on Hugging Face.
+
+## Limitations
+
+- Only one cross-domain test set; there is no human-annotated set for social-media or general
+  web text yet.
+- Most remaining errors are rare proper names and transliterations, and Pali–Sanskrit
+  compounds that are over-split.
+- The test set was evaluated three times. Each change of method was motivated by the
+  development set or by examples outside the test set, never by test results: (1) F1 0.9916;
+  (2) after fixing an inference bug found through a README example (text-final characters were
+  mis-segmented because training documents are newline-delimited, and batches contained padding),
+  0.9917; (3) after label-correction round 2 and mixed fine-tuning (prompted by `COVID-19` being
+  split), P 0.9917 / R 0.9928 / F1 0.9923, the numbers in the table.
+- The developer had seen error analyses on the Wangchan reference in an earlier experiment,
+  before this split was fixed. The foreign-script merge rule follows Step 2 of the guideline but
+  may have been influenced by what was seen.
+
+## License
+
+| part | license |
+|---|---|
+| code (all `.py` files, scripts, tests) | [Apache-2.0](LICENSE) |
+| model weights (`khammoon/model/model.onnx`, `weights.npz`) | [CC BY-NC-SA 4.0](khammoon/model/LICENSE); no commercial use, because the training data carry NonCommercial terms |
+| data (`typhoon-ai/khammoon-data`) | per row, inherited from the source: CC BY-NC-SA 3.0 (BEST2009), CC BY-NC 4.0 or CC BY-SA 4.0 (WangchanThaiInstruct) |
+
+The wheel installed by pip bundles the model weights, so that part falls under CC BY-NC-SA 4.0;
+see [NOTICE](NOTICE).
+
+## Citation
+
+If you use khammoon, the guideline or the data, please cite the technical report:
+
+```bibtex
+@techreport{khammoon2026,
+  title       = {khammoon: From a Linguistic Guideline to a Thai Morpheme Segmenter},
+  author      = {Sirichotedumrong, Warit and Na-Thalang, Adisai and Wittayasakpan, Chanakan and
+                 Phatcharoen, Kritsadha and Pipatanakul, Kunat},
+  institution = {Typhoon, SCB DataX},
+  year        = {2026},
+  url         = {https://github.com/warit-s/khammoon-segmenter}
+}
+```
+
+## Credits
+
+TCC rules ported from [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp) (Apache-2.0;
+Theeramunkong et al. 2000). Training data from the BEST2009 corpus (NECTEC) and
+WangchanThaiInstruct (VISTEC/AIResearch).

khammoon-0.1.0/khammoon/__init__.py

@@ -0,0 +1,5 @@
+"""khammoon — Thai morpheme (คำมูล) segmentation with a small CPU model."""
+from .segmenter import Segmenter, segment
+
+__version__ = "0.1.0"
+__all__ = ["Segmenter", "segment", "__version__"]