tujia-nlp 0.2.0__tar.gz

tujia_nlp-0.2.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 BaiBing
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

tujia_nlp-0.2.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include LICENSE
+include README.md
+include README.zh-CN.md
+include requirements.txt
+recursive-include schema *.json
+recursive-include tests *.py

tujia_nlp-0.2.0/PKG-INFO

@@ -0,0 +1,195 @@
+Metadata-Version: 2.4
+Name: tujia-nlp
+Version: 0.2.0
+Summary: Tujia orthography utilities: tone notation, IPA to Ye conversion, phone normalisation
+Author: TujiaNLP contributors
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/ibaibing/TujiaNLP
+Keywords: tujia,tji,orthography,ipa,ye,low-resource
+Classifier: Programming Language :: Python :: 3
+Classifier: Operating System :: OS Independent
+Classifier: Natural Language :: Chinese (Simplified)
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: test
+Requires-Dist: pytest>=7; extra == "test"
+Provides-Extra: zh
+Requires-Dist: pypinyin>=0.50; extra == "zh"
+Dynamic: license-file
+
+# TujiaNLP
+
+Preserving the endangered **Northern Tujia** language with AI.
+
+> Tujia is now spoken mainly by elderly native speakers. This open-source project aims to
+> digitise the language record and build Tujia speech, translation and dialogue models.
+
+[Chinese version](README.zh-CN.md)
+
+## Goals
+
+1. Collect field recordings and build a consistently annotated bilingual corpus.
+2. Train Tujia ASR (speech recognition) and TTS (speech synthesis).
+3. Build Tujia ↔ Chinese machine translation.
+4. Assemble a Tujia dialogue pipeline, modular first, LLM fine-tuning later.
+5. Ship lightweight learning tools for language revitalisation.
+
+## Scope
+
+**Northern Tujia only** (Longshan / Yongshun / Laifeng area, Miaotan standard).
+Southern Tujia is deliberately excluded for now, to avoid mixing dialects in training.
+
+## Orthography
+
+The project follows the **Yedshu–Peng (1984) Tujia romanisation** ("Ye"). Other
+transcriptions are not used as the working standard.
+
+Two IPA spellings appear in source materials for the same sounds:
+
+| Earlier spelling | Current standard |
+| --- | --- |
+| `aŋ eŋ iŋ oŋ`, `ʐ`, `iu` | `ã ẽ ĩ ũ`, `z`, `iou` |
+
+The `tujia_nlp` package normalises between them: input accepts either spelling, output is
+always the current convention.
+
+## Ecosystem
+
+| Repository | Role |
+| --- | --- |
+| **TujiaNLP** (this) | Corpus terms, collection standards, orthography, licensing |
+| [tujia-asr](https://github.com/ibaibing/tujia-asr) | Acoustic model releases: syllable-level CTC |
+| [tujia-tts](https://github.com/ibaibing/tujia-tts) | Speech synthesis releases |
+| [tujia-llm](https://github.com/ibaibing/tujia-llm) | LoRA adapters and the evaluation protocol |
+
+Where the work happens:
+
+```
+audio ──► tujia-asr        acoustic model, syllable-level output
+
+text  ──► tujia_nlp        IPA ↔ Ye, tone notation ──► tts / llm data, annotations
+```
+
+Rules of thumb:
+
+- **This repository** owns language resources and contracts: the syllable table, the
+  orthography rules, the corpus terms, and the dictionary entry/manifest schemas.
+- **tujia-asr / tujia-tts / tujia-llm** own model releases. Training code lives in private
+  `*-lib` / `*-lab` repositories.
+
+## Installation
+
+```bash
+pip install tujia-nlp
+```
+
+Requires Python 3.9+. No third-party dependencies.
+
+## Usage
+
+```python
+from tujia_nlp import format_syllable, normalize_shape, to_ipa, to_ye
+
+to_ye("la35")                    # "laf"
+to_ye("kaŋ35")                   # "gangf"   legacy input accepted
+to_ipa("gangf")                  # "kã35"    output in the current convention
+format_syllable("la35", "ipa")   # "la³⁵"
+normalize_shape("ʐuo")           # "zuo"
+```
+
+Notation rules:
+
+- `to_ipa` always emits the current convention: nasalised vowels `ã ẽ ĩ ũ`, `z`, `iou`.
+- `to_ye` accepts both: legacy `aŋ eŋ iŋ oŋ`, `ʐ`, `iu` are folded before lookup.
+- A few contrasts are not distinguished by the Ye scheme (`ɿ` / `i`, `ə` / `ɤ`), so conversion
+  is one-way: `to_ye` can produce Ye, `to_ipa` cannot recover them.
+
+The syllable table covers 291 Northern Tujia syllables, 227 attested in corpus material and
+64 derived by rule.
+
+Beyond IPA ↔ Ye, three helpers are available as submodules:
+
+```python
+from tujia_nlp import phone_norm, syllable, zh_sound
+
+syllable.split_head("a⁵⁵ji²¹")     # [("a", 55), ("ji", 21)]
+syllable.tone_stats(["la³⁵"])      # tone usage per shape
+phone_norm.canon("d")              # ["t"]  voiced stops fold onto voiceless
+zh_sound.pinyin_to_ipa("ni3")      # ("ni", 21)
+```
+
+`zh_sound.text_to_syllables` accepts hanzi only when the optional `pypinyin` package is
+installed (`pip install "tujia-nlp[zh]"`); plain pinyin input needs nothing extra.
+
+## Data coverage
+
+The published package contains only the syllable table. Recordings, transcripts and
+dictionary entries are not distributed.
+
+## Repository layout
+
+```
+src/tujia_nlp/     the package: tones.py, ye.py, syllable.py, phone_norm.py, zh_sound.py
+schema/            dictionary entry and pack manifest contracts (JSON Schema)
+tests/             unit tests (pytest)
+docs/              annotation standards and resources
+dataset/           corpus metadata and fetch scripts (raw data is not committed)
+tools/             early conversion scripts, superseded by the package
+```
+
+## Zenodo datasets
+
+Two public North Tujia text corpora are archived on Zenodo. Only fetch scripts and checksums
+live in this repository; the original files are not redistributed.
+
+1. **V1 (DOI: 10.5281/zenodo.4446698)** — incomplete, contains annotation errors, for
+   provenance only, **not for training**.
+2. **V2 (DOI: 10.5281/zenodo.4447924)** — the recommended source for translation, LLM, ASR
+   and TTS work; includes the orthography mapping, vocabulary, textbook sentences, folk
+   narratives and greetings.
+
+The upstream record is marked `Other (Public Domain)`, which is not a standard licence. This
+project does not redistribute the files. Standardised data derived by this project is
+released under CC-BY 4.0, covering only the new conversion and annotation work.
+
+### Citation
+
+Zhou, Y., & Hill, N. W. (2021). *A collection of North Tujia (Bifzivsar) vocabulary and
+textual passages for use in NLP* (Version 2) [Data set]. Zenodo.
+https://doi.org/10.5281/zenodo.4447924
+
+See `DATASET_LICENSE.md` for the full constraints.
+
+## Contributing
+
+- Tujia speakers: field recordings, semantic proofreading, pronunciation checks.
+- NLP developers: model fine-tuning, code review, front-end tools.
+- Linguists: vocabulary expansion and grammatical notes.
+
+### Ethics
+
+Every field recording requires prior **informed consent**. Speaker privacy must be protected
+and recorded material must not be used commercially without permission.
+
+## License
+
+- Source code: **MIT** (see `LICENSE`).
+- Third-party corpora: rights remain with the original authors; not licensed or redistributed here.
+- Data produced by this project: **CC-BY 4.0**, excluding the original word lists.
+- Field recordings: governed by each speaker's individual consent form.
+
+## Roadmap
+
+- [x] Repository bootstrapped
+- [x] `tujia_nlp` orthography package (IPA ↔ Ye, tone notation)
+- [x] Dictionary entry and pack manifest schemas
+- [ ] Annotation standard finalised
+- [ ] Base vocabulary assembled
+- [ ] Field recording collection started
+- [ ] Tujia ASR fine-tuning
+- [ ] Tujia TTS training
+- [ ] Tujia ↔ Chinese translation model
+- [ ] Dialogue web demo
+- [ ] LLM SFT (long-term)

tujia_nlp-0.2.0/README.md

@@ -0,0 +1,174 @@
+# TujiaNLP
+
+Preserving the endangered **Northern Tujia** language with AI.
+
+> Tujia is now spoken mainly by elderly native speakers. This open-source project aims to
+> digitise the language record and build Tujia speech, translation and dialogue models.
+
+[Chinese version](README.zh-CN.md)
+
+## Goals
+
+1. Collect field recordings and build a consistently annotated bilingual corpus.
+2. Train Tujia ASR (speech recognition) and TTS (speech synthesis).
+3. Build Tujia ↔ Chinese machine translation.
+4. Assemble a Tujia dialogue pipeline, modular first, LLM fine-tuning later.
+5. Ship lightweight learning tools for language revitalisation.
+
+## Scope
+
+**Northern Tujia only** (Longshan / Yongshun / Laifeng area, Miaotan standard).
+Southern Tujia is deliberately excluded for now, to avoid mixing dialects in training.
+
+## Orthography
+
+The project follows the **Yedshu–Peng (1984) Tujia romanisation** ("Ye"). Other
+transcriptions are not used as the working standard.
+
+Two IPA spellings appear in source materials for the same sounds:
+
+| Earlier spelling | Current standard |
+| --- | --- |
+| `aŋ eŋ iŋ oŋ`, `ʐ`, `iu` | `ã ẽ ĩ ũ`, `z`, `iou` |
+
+The `tujia_nlp` package normalises between them: input accepts either spelling, output is
+always the current convention.
+
+## Ecosystem
+
+| Repository | Role |
+| --- | --- |
+| **TujiaNLP** (this) | Corpus terms, collection standards, orthography, licensing |
+| [tujia-asr](https://github.com/ibaibing/tujia-asr) | Acoustic model releases: syllable-level CTC |
+| [tujia-tts](https://github.com/ibaibing/tujia-tts) | Speech synthesis releases |
+| [tujia-llm](https://github.com/ibaibing/tujia-llm) | LoRA adapters and the evaluation protocol |
+
+Where the work happens:
+
+```
+audio ──► tujia-asr        acoustic model, syllable-level output
+
+text  ──► tujia_nlp        IPA ↔ Ye, tone notation ──► tts / llm data, annotations
+```
+
+Rules of thumb:
+
+- **This repository** owns language resources and contracts: the syllable table, the
+  orthography rules, the corpus terms, and the dictionary entry/manifest schemas.
+- **tujia-asr / tujia-tts / tujia-llm** own model releases. Training code lives in private
+  `*-lib` / `*-lab` repositories.
+
+## Installation
+
+```bash
+pip install tujia-nlp
+```
+
+Requires Python 3.9+. No third-party dependencies.
+
+## Usage
+
+```python
+from tujia_nlp import format_syllable, normalize_shape, to_ipa, to_ye
+
+to_ye("la35")                    # "laf"
+to_ye("kaŋ35")                   # "gangf"   legacy input accepted
+to_ipa("gangf")                  # "kã35"    output in the current convention
+format_syllable("la35", "ipa")   # "la³⁵"
+normalize_shape("ʐuo")           # "zuo"
+```
+
+Notation rules:
+
+- `to_ipa` always emits the current convention: nasalised vowels `ã ẽ ĩ ũ`, `z`, `iou`.
+- `to_ye` accepts both: legacy `aŋ eŋ iŋ oŋ`, `ʐ`, `iu` are folded before lookup.
+- A few contrasts are not distinguished by the Ye scheme (`ɿ` / `i`, `ə` / `ɤ`), so conversion
+  is one-way: `to_ye` can produce Ye, `to_ipa` cannot recover them.
+
+The syllable table covers 291 Northern Tujia syllables, 227 attested in corpus material and
+64 derived by rule.
+
+Beyond IPA ↔ Ye, three helpers are available as submodules:
+
+```python
+from tujia_nlp import phone_norm, syllable, zh_sound
+
+syllable.split_head("a⁵⁵ji²¹")     # [("a", 55), ("ji", 21)]
+syllable.tone_stats(["la³⁵"])      # tone usage per shape
+phone_norm.canon("d")              # ["t"]  voiced stops fold onto voiceless
+zh_sound.pinyin_to_ipa("ni3")      # ("ni", 21)
+```
+
+`zh_sound.text_to_syllables` accepts hanzi only when the optional `pypinyin` package is
+installed (`pip install "tujia-nlp[zh]"`); plain pinyin input needs nothing extra.
+
+## Data coverage
+
+The published package contains only the syllable table. Recordings, transcripts and
+dictionary entries are not distributed.
+
+## Repository layout
+
+```
+src/tujia_nlp/     the package: tones.py, ye.py, syllable.py, phone_norm.py, zh_sound.py
+schema/            dictionary entry and pack manifest contracts (JSON Schema)
+tests/             unit tests (pytest)
+docs/              annotation standards and resources
+dataset/           corpus metadata and fetch scripts (raw data is not committed)
+tools/             early conversion scripts, superseded by the package
+```
+
+## Zenodo datasets
+
+Two public North Tujia text corpora are archived on Zenodo. Only fetch scripts and checksums
+live in this repository; the original files are not redistributed.
+
+1. **V1 (DOI: 10.5281/zenodo.4446698)** — incomplete, contains annotation errors, for
+   provenance only, **not for training**.
+2. **V2 (DOI: 10.5281/zenodo.4447924)** — the recommended source for translation, LLM, ASR
+   and TTS work; includes the orthography mapping, vocabulary, textbook sentences, folk
+   narratives and greetings.
+
+The upstream record is marked `Other (Public Domain)`, which is not a standard licence. This
+project does not redistribute the files. Standardised data derived by this project is
+released under CC-BY 4.0, covering only the new conversion and annotation work.
+
+### Citation
+
+Zhou, Y., & Hill, N. W. (2021). *A collection of North Tujia (Bifzivsar) vocabulary and
+textual passages for use in NLP* (Version 2) [Data set]. Zenodo.
+https://doi.org/10.5281/zenodo.4447924
+
+See `DATASET_LICENSE.md` for the full constraints.
+
+## Contributing
+
+- Tujia speakers: field recordings, semantic proofreading, pronunciation checks.
+- NLP developers: model fine-tuning, code review, front-end tools.
+- Linguists: vocabulary expansion and grammatical notes.
+
+### Ethics
+
+Every field recording requires prior **informed consent**. Speaker privacy must be protected
+and recorded material must not be used commercially without permission.
+
+## License
+
+- Source code: **MIT** (see `LICENSE`).
+- Third-party corpora: rights remain with the original authors; not licensed or redistributed here.
+- Data produced by this project: **CC-BY 4.0**, excluding the original word lists.
+- Field recordings: governed by each speaker's individual consent form.
+
+## Roadmap
+
+- [x] Repository bootstrapped
+- [x] `tujia_nlp` orthography package (IPA ↔ Ye, tone notation)
+- [x] Dictionary entry and pack manifest schemas
+- [ ] Annotation standard finalised
+- [ ] Base vocabulary assembled
+- [ ] Field recording collection started
+- [ ] Tujia ASR fine-tuning
+- [ ] Tujia TTS training
+- [ ] Tujia ↔ Chinese translation model
+- [ ] Dialogue web demo
+- [ ] LLM SFT (long-term)

tujia_nlp-0.2.0/README.zh-CN.md

@@ -0,0 +1,137 @@
+# TujiaNLP
+利用AI技术抢救、保护濒危**北部土家语（龙山苗儿滩标准音）**
+> 土家语目前仅高龄土家族母语者掌握，面临语言消亡风险。本开源公益项目旨在数字化留存语言资料，构建土家语AI语音、翻译与对话模型。
+
+[English](README.md)
+
+## 🎯 项目目标
+1. 田野采集土家语音频，建立规范双语标注数据集；
+2. 训练土家语ASR（语音识别）、TTS（语音合成）；
+3. 实现土家语 ↔ 汉语机器翻译；
+4. 搭建土家语对话AI（模块化流水线先行，远期开展LLM微调）；
+5. 产出轻量化学习工具，用于土家语传承科普。
+
+> ⚠️ 范围限定：**北部土家语（龙山/永顺/来凤片区）**，前期暂不纳入南部土家语，避免方言混杂干扰模型训练。
+
+## 📜 书写标准
+统一采用【叶德书-彭秀模1984土家语拼音方案】，不混用其他转写体系。
+拼音互转由 `tujia_nlp` 包提供（`tools/orthography_convert.py` 是早期脚本，功能已并入前者）。
+
+## 🌐 生态与分工
+
+| 仓库 | 职责 |
+| --- | --- |
+| **TujiaNLP**（本仓） | 语料授权、采集规范、正字法、数据契约 |
+| [tujia-asr](https://github.com/ibaibing/tujia-asr) | 声学模型发布：音节级 CTC |
+| [tujia-tts](https://github.com/ibaibing/tujia-tts) | 语音合成发布 |
+| [tujia-llm](https://github.com/ibaibing/tujia-llm) | LoRA 适配器与评测协议 |
+
+数据流：
+
+```
+音频 ──► tujia-asr       声学模型，输出音节级结果
+
+文本 ──► tujia_nlp       IPA ↔ Ye、四调记法 ──► TTS / LLM 数据与注音
+```
+
+边界原则：
+
+- **本仓**管语言资源与契约：音节表、正字法规则、语料授权、词典条目与包清单 schema。
+- **tujia-asr / tujia-tts / tujia-llm** 管模型发布。训练代码在私有的 `*-lib` / `*-lab`。
+
+## 📦 安装与使用
+
+```bash
+pip install tujia-nlp
+```
+
+```python
+from tujia_nlp import format_syllable, normalize_shape, to_ipa, to_ye
+
+to_ye("la35")                    # "laf"
+to_ye("kaŋ35")                   # "gangf"   旧记法输入同样接受
+to_ipa("gangf")                  # "kã35"    输出统一为现行口径
+format_syllable("la35", "ipa")   # "la³⁵"
+normalize_shape("ʐuo")           # "zuo"
+```
+
+记法约定：
+
+- 输出（`to_ipa`）一律是现行口径：鼻化元音 `ã ẽ ĩ ũ`、`z`、`iou`。
+- 输入（`to_ye`）两套都接受：早期材料的 `aŋ eŋ iŋ oŋ`、`ʐ`、`iu` 会先折算再查表。
+- `ɿ`、`ə` 这类 Ye 不区分的对立只能单向：`to_ye` 能转出 Ye，`to_ipa` 不能还原。
+
+音节表覆盖北部土家语（龙山/永顺/来凤）291 个音节，其中 227 条有语料实证、64 条按规则推断。
+
+除 IPA ↔ Ye 之外，还有三个子模块：
+
+```python
+from tujia_nlp import phone_norm, syllable, zh_sound
+
+syllable.split_head("a⁵⁵ji²¹")     # [("a", 55), ("ji", 21)]
+syllable.tone_stats(["la³⁵"])      # 每个音节的调值分布
+phone_norm.canon("d")              # ["t"]  浊塞音并入清音类
+zh_sound.pinyin_to_ipa("ni3")      # ("ni", 21)
+```
+
+`zh_sound.text_to_syllables` 处理汉字需要可选依赖 `pypinyin`（`pip install "tujia-nlp[zh]"`），
+纯拼音输入无需额外安装。
+
+## 🛠️ 硬件与技术栈
+- 开发设备：Mac Studio M2 Ultra 128GB，基于MLX实现本地离线训练/推理
+- 语音模型：OpenAI Whisper、VITS、FishSpeech
+- LLM底座：Qwen系列（原生适配MLX）
+- 开发语言：Python 3.11+
+
+## 📂 目录说明
+```
+src/tujia_nlp/     包本体：tones.py、ye.py、syllable.py、phone_norm.py、zh_sound.py
+schema/            词典条目与包清单契约（JSON Schema）
+tests/             单元测试（pytest）
+docs/              采集规范、资源清单
+dataset/           语料元数据与取数脚本（不含原始数据）
+tools/             早期转换脚本，功能已并入包
+```
+
+## 📚 公开Zenodo数据集说明
+项目归档两套官方开源北部土家语文本语料，原始文件存放路径：`dataset/raw_zenodo/`
+1. **V1 初代存档版（DOI：10.5281/zenodo.4446698）**
+   素材残缺、拼音映射存在标注错误，仅用于学术溯源查阅，**禁止用于模型训练**；
+2. **V2 扩充完整版（DOI：10.5281/zenodo.4447924）**
+   本项目翻译/LLM/ASR/TTS唯一官方训练数据源，完整包含拼音对照表、词汇、课本短句、民间叙事、日常问候语。
+
+原始数据由 Zenodo 记录 4446698 / 4447924 提供，上传者标注 `Other (Public Domain)`，该标注不是标准许可。本项目不再分发原始文件，仅提供取数脚本。经 `tools/orthography_convert.py` 转换、清洗后的标准化语料存放于 `dataset/processed/`，其许可仅覆盖本项目新增的转换与标注工作。
+
+### 学术引用格式
+#### V1 初代数据集
+Zhou, Y., & Hill, N. W. (2021). A collection of North Tujia (Bifzivsar 北部土家语) vocabulary and textual passages for use in NLP (Version 1) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.4446698
+#### V2 完整版数据集
+Zhou, Y., & Hill, N. W. (2021). A collection of North Tujia (Bifzivsar 北部土家语) vocabulary and textual passages for use in NLP (Version 2) [Data set]. Zenodo. https://doi.org/10.5281/zenodo.4447924
+
+完整数据集许可约束详见根目录 `DATASET_LICENSE.md`。
+
+## 🤝 如何参与贡献
+欢迎所有志愿者参与：
+1. 土家族同胞：协助田野录音、语义校对、发音校验；
+2. AI/NLP开发者：模型微调、代码优化、前端工具开发；
+3. 语言学爱好者：扩充词汇、整理语法资料。
+
+### ⚖️ 伦理规范
+所有田野录音必须提前获取受访者**知情同意**，严格保护老人隐私，采集素材不可私自商用。
+
+## 📄 开源协议
+- 程序源代码：**MIT License**（根目录 LICENSE 文件）
+- 第三方原始语料：版权归原权利人，本项目不授予许可、不再分发，详见 `dataset/raw_zenodo/README.md`
+- 项目自行生成的标准化数据与标注：**CC-BY 4.0**（不含原始词表全文）
+- 田野原始音频：遵循每位受访者单独签署的知情同意约定
+
+## 📌 项目进度清单
+- [x] Github仓库初始化
+- [ ] 完善《土家语数据采集与标注规范》
+- [ ] 整理公开文献，搭建基础词汇库
+- [ ] 启动田野录音采集
+- [ ] Whisper增量微调 土家语ASR
+- [ ] 训练土家语TTS语音合成
+- [ ] 汉-土双语翻译模型
+- [ ] 土家语语音对话Web Demo
+- [ ] 土家语大模型SFT微调（长期目标）

tujia_nlp-0.2.0/pyproject.toml

@@ -0,0 +1,40 @@
+[build-system]
+requires = ["setuptools>=68", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tujia-nlp"
+dynamic = ["version"]
+description = "Tujia orthography utilities: tone notation, IPA to Ye conversion, phone normalisation"
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+authors = [{ name = "TujiaNLP contributors" }]
+keywords = ["tujia", "tji", "orthography", "ipa", "ye", "low-resource"]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Operating System :: OS Independent",
+    "Natural Language :: Chinese (Simplified)",
+    "Topic :: Text Processing :: Linguistic",
+]
+
+[project.optional-dependencies]
+test = ["pytest>=7"]
+zh = ["pypinyin>=0.50"]
+
+[project.urls]
+Homepage = "https://github.com/ibaibing/TujiaNLP"
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+# The version lives in one place only: tujia_nlp.__version__.
+[tool.setuptools.dynamic]
+version = { attr = "tujia_nlp.__version__" }
+
+[tool.setuptools.package-data]
+tujia_nlp = ["data/*.json"]
+
+[tool.pytest.ini_options]
+testpaths = ["tests"]
+pythonpath = ["src"]

tujia_nlp-0.2.0/requirements.txt

@@ -0,0 +1,6 @@
+# tujia-nlp has no runtime dependencies; it uses the Python standard library only.
+# ML training packages (mlx / torch / whisper and friends) belong to other projects
+# and are not listed here.
+#
+# Development and tests:
+#   pip install -e ".[test]"

tujia_nlp-0.2.0/schema/entry.schema.json

@@ -0,0 +1,60 @@
+{
+  "$schema": "https://json-schema.org/draft/2020-12/schema",
+  "$id": "https://github.com/ibaibing/TujiaNLP/schema/entry.schema.json",
+  "title": "Dictionary entry (one line of lexicon.jsonl)",
+  "description": "One record per line of lexicon.jsonl: w is toned IPA, p is the per-character phone sequence, g is the gloss, pg is the source locator.",
+  "type": "object",
+  "required": ["w", "p", "g", "pos", "pg"],
+  "properties": {
+    "w": {
+      "type": "string",
+      "minLength": 1,
+      "description": "Toned IPA word form; tones use superscript digits, e.g. 'loŋ⁵⁵', 'a²¹pa²¹'."
+    },
+    "p": {
+      "type": "string",
+      "minLength": 1,
+      "description": "Per-character phone sequence, space separated, e.g. 'l o ŋ'; equals the cleaned and detoned w."
+    },
+    "g": {
+      "type": "string",
+      "description": "Gloss. Chinese dictionaries use Chinese; public packs may carry English only."
+    },
+    "pos": {
+      "type": "string",
+      "description": "Part of speech, one Chinese character (名/动/副/量/数/代/助/连/后缀...); empty string when unannotated."
+    },
+    "pg": {
+      "type": "string",
+      "pattern": "^[A-Za-z0-9_.-]+(#[0-9]+)?$",
+      "description": "Source locator. Single-dictionary use: 'page_0123'. Pack use: '<pack_id>#<row>', e.g. 'pack-a#1386'."
+    },
+    "src": {
+      "type": "string",
+      "description": "Source pack id. Optional for single-dictionary entries; required for provenance after merging."
+    },
+    "src_head": {
+      "type": "string",
+      "description": "Original headword in the source file, before transliteration."
+    },
+    "ye": {
+      "type": "string",
+      "description": "Yedshu-Peng (1984) spelling; optional, for display and manual checking."
+    },
+    "variants": {
+      "type": "array",
+      "items": {"type": "string"},
+      "description": "Alternative spellings or grammatical variants (future / perfect)."
+    },
+    "sense": {
+      "type": "integer",
+      "minimum": 1,
+      "description": "Sense number for the same form; identical forms from different packs keep their own sense."
+    },
+    "notes": {
+      "type": "string",
+      "description": "Free-form annotator note."
+    }
+  },
+  "additionalProperties": true
+}