cunei-tools 1.0.0__tar.gz

cunei_tools-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Anonymous
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

cunei_tools-1.0.0/PKG-INFO

@@ -0,0 +1,188 @@
+Metadata-Version: 2.4
+Name: cunei-tools
+Version: 1.0.0
+Summary: Cuneiform NLP utilities: word segmentation (with pre-trained Akkadian/Sumerian/Elamite models) and Latin-Unicode script conversion
+Author-email: Chuanjun Zhou <chuanjunzhou@berkeley.edu>, Adam Anderson <adam.anderson@factgrid.eu>
+License: MIT
+Project-URL: Homepage, https://github.com/ancient-world-citation-analysis/cunei-tools
+Project-URL: Repository, https://github.com/ancient-world-citation-analysis/cunei-tools
+Project-URL: Paper, https://openreview.net/forum?id=WdmrVUDpzt
+Project-URL: Experiments, https://github.com/ancient-world-citation-analysis/cuneiform-experiments
+Keywords: cuneiform,nlp,segmentation,akkadian,sumerian,elamite
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: convert
+Requires-Dist: pandas; extra == "convert"
+Provides-Extra: all
+Requires-Dist: pandas; extra == "all"
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Dynamic: license-file
+
+# cunei-tools
+
+Cuneiform NLP utilities for word segmentation and script conversion across Akkadian, Sumerian, and Elamite.
+
+Developed as part of *"What Transfers Across Cuneiform? Script-Level Substrate and Language-Specific Regularities via Segmentation and Representation"* (Chuanjun Zhou and Adam Anderson, EMNLP 2026 Main).
+
+## Installation
+
+```bash
+pip install cunei-tools
+# Or from source:
+git clone https://github.com/ancient-world-citation-analysis/cunei-tools
+cd cunei-tools && pip install -e .
+```
+
+## Quick start with pre-trained models
+
+Pre-trained segmentation models for all three languages ship with the
+package (trained on the paper's corpora with each language's published
+optimal threshold):
+
+```python
+from cunei_tools import CuneiSeg
+seg = CuneiSeg.pretrained("akk")   # or "sux", "elx"
+words = seg.segment(unicode_sign_stream)
+```
+
+## cunei-seg: Word Boundary Segmentation
+
+Recovers word boundaries from unsegmented Unicode cuneiform using transitional probability. Achieves **F1 > 0.96** across three languages.
+
+| Language | F1 | Precision | Recall | Threshold |
+|----------|------|-----------|--------|-----------|
+| Akkadian | 0.969 | 0.946 | 0.992 | 0.70 |
+| Sumerian | 0.971 | 0.946 | 0.998 | 0.60 |
+| Elamite  | 0.973 | 0.993 | 0.955 | 0.90 |
+
+### Python API
+
+```python
+from cunei_tools import CuneiSeg
+
+# Train on segmented documents
+seg = CuneiSeg(lang="akk")
+seg.train([
+    "𒀀𒈾 𒁀𒌅 𒊭",
+    "𒂍 𒈾 𒀀𒈾",
+    # ... your segmented corpus
+])
+
+# Segment new text
+seg.segment("𒀀𒈾𒁀𒌅𒊭")
+# → "𒀀𒈾 𒁀𒌅 𒊭"
+
+# Find optimal threshold
+metrics = seg.find_optimal_threshold(gold_documents)
+print(metrics)
+# → {'threshold': 0.70, 'f1': 0.969, 'precision': 0.946, 'recall': 0.992}
+
+# Save and load
+seg.save("akk_model.json")
+seg2 = CuneiSeg.load("akk_model.json")
+```
+
+### Command Line
+
+```bash
+# Segment from file
+cunei-seg --model akk_model.json --input unsegmented.txt --output segmented.txt
+
+# Segment single text
+cunei-seg --model akk_model.json --text "𒀀𒈾𒁀𒌅𒊭"
+```
+
+## cunei-conv: Script Conversion
+
+Convert between Latin transliteration and Unicode cuneiform using 18,000+ sign mappings from Nuolenna and Akkademia.
+
+### Python API
+
+```python
+from cunei_tools import CuneiConv
+
+conv = CuneiConv()
+conv.load_sign_lists()  # from GitHub
+
+# Latin → Unicode
+conv.to_unicode("a-na")        # → "𒀀 𒈾"
+conv.to_unicode("šu-un-ki-ik") # → "𒋗 𒌦 𒆠 𒅅"
+
+# Unicode → Latin
+conv.to_latin("𒀀𒈾")          # → "a na"
+
+# Detailed conversion with quality check
+result = conv.to_unicode_detailed("a-na ba-x")
+# → {'unicode': '𒀀 𒈾 𒁀 x', 'clean': False, 'unmatched': ['x']}
+
+# Batch conversion rate
+stats = conv.conversion_rate(["a-na", "ba-ab", "unknown-sign"])
+# → {'total': 3, 'clean': 2, 'rate': 0.667, ...}
+
+# Add manual corrections
+conv.load_manual_corrections("corrections.csv")
+
+# Save for offline use
+conv.save("sign_dict.json")
+conv2 = CuneiConv.load_from_file("sign_dict.json")
+```
+
+### Command Line
+
+```bash
+# Convert to Unicode
+cunei-conv --mode to-unicode --text "a-na ba-ab"
+
+# Convert file
+cunei-conv --mode to-unicode --input latin.txt --output unicode.txt --lang akk
+
+# Reverse: Unicode to Latin
+cunei-conv --mode to-latin --text "𒀀𒈾"
+```
+
+## Supported Languages
+
+- **Akkadian** (`akk`) — Semitic, ~1.25M tokens tested
+- **Sumerian** (`sux`) — language isolate, ~146K tokens tested
+- **Elamite** (`elx`) — isolating/agglutinative, ~13K tokens tested
+
+The tools are language-agnostic at their core — they work with any cuneiform script that uses Unicode code points in the U+12000–U+1254F range. Language-specific parameters (thresholds, normalization) improve accuracy.
+
+## Citation
+
+If you use cunei-tools, please cite:
+
+```bibtex
+@inproceedings{zhou-anderson-2026-transfers,
+  title = "What Transfers Across Cuneiform? Script-Level Substrate and Language-Specific Regularities via Segmentation and Representation",
+  author = "Zhou, Chuanjun and Anderson, Adam",
+  booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
+  year = "2026",
+  address = "Budapest, Hungary",
+  publisher = "Association for Computational Linguistics",
+}
+```
+
+## Acknowledgments
+
+Sign list data comes from [Nuolenna](https://github.com/situx/Nuolenna) and [Akkademia](https://github.com/gaigutherz/Akkademia).
+
+
+## Team
+
+Chuanjun Zhou and Adam Anderson (University of California, Berkeley; FactGrid / TokenWorks LLC)
+
+## License
+
+MIT

cunei_tools-1.0.0/README.md

@@ -0,0 +1,157 @@
+# cunei-tools
+
+Cuneiform NLP utilities for word segmentation and script conversion across Akkadian, Sumerian, and Elamite.
+
+Developed as part of *"What Transfers Across Cuneiform? Script-Level Substrate and Language-Specific Regularities via Segmentation and Representation"* (Chuanjun Zhou and Adam Anderson, EMNLP 2026 Main).
+
+## Installation
+
+```bash
+pip install cunei-tools
+# Or from source:
+git clone https://github.com/ancient-world-citation-analysis/cunei-tools
+cd cunei-tools && pip install -e .
+```
+
+## Quick start with pre-trained models
+
+Pre-trained segmentation models for all three languages ship with the
+package (trained on the paper's corpora with each language's published
+optimal threshold):
+
+```python
+from cunei_tools import CuneiSeg
+seg = CuneiSeg.pretrained("akk")   # or "sux", "elx"
+words = seg.segment(unicode_sign_stream)
+```
+
+## cunei-seg: Word Boundary Segmentation
+
+Recovers word boundaries from unsegmented Unicode cuneiform using transitional probability. Achieves **F1 > 0.96** across three languages.
+
+| Language | F1 | Precision | Recall | Threshold |
+|----------|------|-----------|--------|-----------|
+| Akkadian | 0.969 | 0.946 | 0.992 | 0.70 |
+| Sumerian | 0.971 | 0.946 | 0.998 | 0.60 |
+| Elamite  | 0.973 | 0.993 | 0.955 | 0.90 |
+
+### Python API
+
+```python
+from cunei_tools import CuneiSeg
+
+# Train on segmented documents
+seg = CuneiSeg(lang="akk")
+seg.train([
+    "𒀀𒈾 𒁀𒌅 𒊭",
+    "𒂍 𒈾 𒀀𒈾",
+    # ... your segmented corpus
+])
+
+# Segment new text
+seg.segment("𒀀𒈾𒁀𒌅𒊭")
+# → "𒀀𒈾 𒁀𒌅 𒊭"
+
+# Find optimal threshold
+metrics = seg.find_optimal_threshold(gold_documents)
+print(metrics)
+# → {'threshold': 0.70, 'f1': 0.969, 'precision': 0.946, 'recall': 0.992}
+
+# Save and load
+seg.save("akk_model.json")
+seg2 = CuneiSeg.load("akk_model.json")
+```
+
+### Command Line
+
+```bash
+# Segment from file
+cunei-seg --model akk_model.json --input unsegmented.txt --output segmented.txt
+
+# Segment single text
+cunei-seg --model akk_model.json --text "𒀀𒈾𒁀𒌅𒊭"
+```
+
+## cunei-conv: Script Conversion
+
+Convert between Latin transliteration and Unicode cuneiform using 18,000+ sign mappings from Nuolenna and Akkademia.
+
+### Python API
+
+```python
+from cunei_tools import CuneiConv
+
+conv = CuneiConv()
+conv.load_sign_lists()  # from GitHub
+
+# Latin → Unicode
+conv.to_unicode("a-na")        # → "𒀀 𒈾"
+conv.to_unicode("šu-un-ki-ik") # → "𒋗 𒌦 𒆠 𒅅"
+
+# Unicode → Latin
+conv.to_latin("𒀀𒈾")          # → "a na"
+
+# Detailed conversion with quality check
+result = conv.to_unicode_detailed("a-na ba-x")
+# → {'unicode': '𒀀 𒈾 𒁀 x', 'clean': False, 'unmatched': ['x']}
+
+# Batch conversion rate
+stats = conv.conversion_rate(["a-na", "ba-ab", "unknown-sign"])
+# → {'total': 3, 'clean': 2, 'rate': 0.667, ...}
+
+# Add manual corrections
+conv.load_manual_corrections("corrections.csv")
+
+# Save for offline use
+conv.save("sign_dict.json")
+conv2 = CuneiConv.load_from_file("sign_dict.json")
+```
+
+### Command Line
+
+```bash
+# Convert to Unicode
+cunei-conv --mode to-unicode --text "a-na ba-ab"
+
+# Convert file
+cunei-conv --mode to-unicode --input latin.txt --output unicode.txt --lang akk
+
+# Reverse: Unicode to Latin
+cunei-conv --mode to-latin --text "𒀀𒈾"
+```
+
+## Supported Languages
+
+- **Akkadian** (`akk`) — Semitic, ~1.25M tokens tested
+- **Sumerian** (`sux`) — language isolate, ~146K tokens tested
+- **Elamite** (`elx`) — isolating/agglutinative, ~13K tokens tested
+
+The tools are language-agnostic at their core — they work with any cuneiform script that uses Unicode code points in the U+12000–U+1254F range. Language-specific parameters (thresholds, normalization) improve accuracy.
+
+## Citation
+
+If you use cunei-tools, please cite:
+
+```bibtex
+@inproceedings{zhou-anderson-2026-transfers,
+  title = "What Transfers Across Cuneiform? Script-Level Substrate and Language-Specific Regularities via Segmentation and Representation",
+  author = "Zhou, Chuanjun and Anderson, Adam",
+  booktitle = "Proceedings of the 2026 Conference on Empirical Methods in Natural Language Processing",
+  year = "2026",
+  address = "Budapest, Hungary",
+  publisher = "Association for Computational Linguistics",
+}
+```
+
+## Acknowledgments
+
+Sign list data comes from [Nuolenna](https://github.com/situx/Nuolenna) and [Akkademia](https://github.com/gaigutherz/Akkademia).
+
+
+## Team
+
+Chuanjun Zhou and Adam Anderson (University of California, Berkeley; FactGrid / TokenWorks LLC)
+
+## License
+
+MIT

cunei_tools-1.0.0/cunei_tools/__init__.py

@@ -0,0 +1,6 @@
+"""
+cunei-tools: Cuneiform NLP utilities
+"""
+__version__ = "1.0.0"
+from cunei_tools.segmenter import CuneiSeg
+from cunei_tools.converter import CuneiConv

cunei_tools-1.0.0/cunei_tools/cli.py

@@ -0,0 +1,94 @@
+"""
+Command-line interface for cunei-tools.
+
+Usage:
+    # Segment unsegmented Unicode cuneiform
+    cunei-seg --model model.json --input text.txt --output segmented.txt
+    cunei-seg --model model.json --text "𒀀𒈾𒁀𒌅"
+
+    # Convert transliteration to Unicode
+    cunei-conv --mode to-unicode --text "a-na ba-ab"
+    cunei-conv --mode to-latin --text "𒀀𒈾"
+    cunei-conv --mode to-unicode --input texts.txt --output converted.txt
+"""
+
+import argparse
+import sys
+
+
+def seg_main():
+    parser = argparse.ArgumentParser(
+        description='cunei-seg: Word boundary segmentation for cuneiform'
+    )
+    parser.add_argument('--model', required=True, help='Path to trained model JSON')
+    parser.add_argument('--text', help='Single text to segment')
+    parser.add_argument('--input', help='Input file (one text per line)')
+    parser.add_argument('--output', help='Output file (default: stdout)')
+    parser.add_argument('--threshold', type=float, help='Override threshold')
+    args = parser.parse_args()
+
+    from cunei_tools import CuneiSeg
+    seg = CuneiSeg.load(args.model)
+
+    if args.threshold:
+        seg.threshold = args.threshold
+
+    if args.text:
+        print(seg.segment(args.text))
+    elif args.input:
+        with open(args.input, 'r', encoding='utf-8') as f:
+            lines = [l.strip() for l in f]
+        results = seg.segment_batch(lines)
+        out = open(args.output, 'w', encoding='utf-8') if args.output else sys.stdout
+        for r in results:
+            out.write(r + '\n')
+        if args.output:
+            out.close()
+    else:
+        parser.print_help()
+
+
+def conv_main():
+    parser = argparse.ArgumentParser(
+        description='cunei-conv: Cuneiform transliteration converter'
+    )
+    parser.add_argument('--mode', required=True, choices=['to-unicode', 'to-latin'],
+                        help='Conversion direction')
+    parser.add_argument('--dict', help='Path to sign dictionary JSON')
+    parser.add_argument('--text', help='Single text to convert')
+    parser.add_argument('--input', help='Input file (one text per line)')
+    parser.add_argument('--output', help='Output file (default: stdout)')
+    parser.add_argument('--lang', choices=['akk', 'sux', 'elx'],
+                        help='Language for specialized normalization')
+    args = parser.parse_args()
+
+    from cunei_tools import CuneiConv
+
+    if args.dict:
+        conv = CuneiConv.load_from_file(args.dict)
+    else:
+        conv = CuneiConv()
+        conv.load_sign_lists()
+
+    def convert(text):
+        if args.mode == 'to-unicode':
+            return conv.to_unicode(text, lang=args.lang)
+        else:
+            return conv.to_latin(text)
+
+    if args.text:
+        print(convert(args.text))
+    elif args.input:
+        with open(args.input, 'r', encoding='utf-8') as f:
+            lines = [l.strip() for l in f]
+        out = open(args.output, 'w', encoding='utf-8') if args.output else sys.stdout
+        for line in lines:
+            out.write(convert(line) + '\n')
+        if args.output:
+            out.close()
+    else:
+        parser.print_help()
+
+
+if __name__ == '__main__':
+    print("Use 'cunei-seg' or 'cunei-conv' commands.")