hmaraniam 0.1.0__tar.gz

hmaraniam-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hmar Heritage Project
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

hmaraniam-0.1.0/PKG-INFO

@@ -0,0 +1,199 @@
+Metadata-Version: 2.4
+Name: hmaraniam
+Version: 0.1.0
+Summary: High-precision language identification library for Hmar ('Hmar a ni am?')
+Author-email: Hmar Heritage Project <info@hmarheritage.org>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/hmar-heritage-org/hmaraniam
+Project-URL: Repository, https://github.com/hmar-heritage-org/hmaraniam
+Project-URL: Bug Tracker, https://github.com/hmar-heritage-org/hmaraniam/issues
+Keywords: hmar,language-identification,nlp,kuki-chin,zo-languages,language-detection,hmaraniam
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: license-file
+
+# hmaraniam 🇲z
+
+**High-precision, zero-dependency language identification library for Hmar.**
+
+> *"Hmar a ni am?"* — *"Is it Hmar?"*
+
+`hmaraniam` is a lightweight Python library designed to accurately distinguish Hmar text from English and other Kuki-Chin / Zo languages (Mizo, Kuki, Paite, Vaiphei).
+
+---
+
+## Key Features
+
+- **Microsecond Speed:** $O(1)$ dictionary lookups with no heavy ML dependencies (PyTorch/TensorFlow free).
+- **Dual-Lens Diacritic Engine:** Reports both `casual_hmar_ratio` (ASCII-normalized for standard QWERTY typing) and `formal_hmar_ratio` (exact diacritic matching for formal literary text).
+- **Dual Continuous Confidence Metrics:** Disambiguates language metrics into `hmar_confidence` (permanent metric answering *"How confident are we that this text is Hmar?"*) and `detected_language_confidence` (confidence in the overall classification choice).
+- **Standardized Schema:** Guarantees an immutable JSON output structure across all calls, including `non_hmar_words_count` and `diacritic_words_count`.
+- **Extensible & Customizable:** Allows developers to supply `custom_unigrams`, `extra_unigrams`, `custom_stopwords`, or disable default stopwords.
+- **Dual Offline/CDN Architecture:** Automatically syncs with the live `hmar-heritage-org/hmaraniam` unigram dataset via jsDelivr CDN, with automatic local disk caching and bundled fallback.
+
+---
+
+## Installation
+
+```bash
+pip install hmaraniam
+```
+
+---
+
+## Standard Output Schema
+
+```json
+{
+  "language": "hmar",
+  "hmar_confidence": 0.9842,
+  "detected_language_confidence": 0.9842,
+  "mode": "basic",
+  "scores": {
+    "casual_hmar_ratio": 0.9524,
+    "formal_hmar_ratio": 0.8095,
+    "english_stopword_ratio": 0.0000,
+    "sibling_zo_stopword_ratio": 0.0000,
+    "unknown_words_ratio": 0.0476,
+    "total_words": 21,
+    "hmar_words_count": 20,
+    "non_hmar_words_count": 1,
+    "unknown_words_count": 1,
+    "english_stopwords_count": 0,
+    "sibling_zo_stopwords_count": 0,
+    "hmar_diacritic_words_count": 17,
+    "non_hmar_diacritic_words_count": 0,
+    "total_diacritic_words_count": 17
+  }
+}
+```
+
+---
+
+## Usage
+
+### Quick Start
+
+```python
+import hmaraniam
+
+# Authentic text from L. Keivom archive
+sample_text = "Khawvel fe dan phung ei en chun, ram le hnam damna thuruk chu lien lema intel le insung khawm a nih."
+
+result = hmaraniam.detect(sample_text)
+
+print(result)
+```
+
+### Custom Unigrams & Stopwords
+
+```python
+from hmaraniam import Detector
+
+# Provide custom unigrams or extra domain vocabulary
+detector = Detector(
+    mode="basic",
+    extra_unigrams=["customworda", "customwordb"],
+    custom_stopwords=["and", "the", "with"],
+    disable_default_stopwords=False
+)
+
+result = detector.detect("Khawvel fe dan phung...")
+```
+
+### Modes & Advanced Options
+
+```python
+from hmaraniam import Detector
+
+# Basic Mode (Active default ~30k core unigrams)
+basic_detector = Detector(mode="basic")
+
+# High Mode (Scans data/shards/ and loads all available unigram shards, falling back seamlessly to basic)
+high_detector = Detector(mode="high")
+
+# Offline-only mode (uses cached/bundled dataset without network calls)
+offline_detector = Detector(offline_only=True)
+```
+
+---
+
+## Empirical Benchmarks & Performance
+
+`hmaraniam` has been empirically validated across both **controlled parallel corpus datasets** (Parallel Zo Bibles across multiple literary genres) and **unfiltered real-world web archives** (~1,300 scraped articles and raw HTML pages across 5 major publishers).
+
+### 1. Controlled Parallel Zo Bible Benchmark
+
+Evaluated across parallel chapters (Genesis 1, Exodus 20, Matthew 5, Luke 2, Romans 8, Revelation 21) across 10 parallel Bible translations in 8 Zo languages + English:
+
+| Language | Edition / Source | Evaluated Passages | Target Class | Engine Assigned Label | Classification Accuracy | Avg Hmar Confidence | Key Distinguishing Metrics |
+| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :--- |
+| **Hmar** | CLB (Contemporary) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `hmar` | `hmar` | **100%** | **1.0000** | `casual_hmar_ratio` $\ge 0.95$, `unknown_words_ratio` $\le 0.05$ |
+| **Hmar** | OV (Old Version) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `hmar` | `hmar` | **100%** | **1.0000** | Formal diacritic match + `hmar_diacritic_words_count` $>0$ |
+| **Mizo** | OV (Mizo Bible) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathian`, `hnenah`, `avangin`, `tichuan`) & `unknown_words_ratio` ($\approx 24\%$) |
+| **Paite** | Paite Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pasian`, `tungah`, `simhuai`) & `unknown_words_ratio` ($\approx 51\%$) |
+| **Vaiphei** | Vaiphei Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathian`, `tiu-in`, `apat`) & `unknown_words_ratio` ($\approx 38\%$) |
+| **Gangte** | Gangte Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathen`, `hepa`, `dih-in`) & `unknown_words_ratio` ($\approx 40\%$) |
+| **Zou** | Zou Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pasian`, `a-in`, `leh-in`) & `unknown_words_ratio` ($\approx 51\%$) |
+| **Thadou** | Thadou-Kuki Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathen`, `chun`, `tichun`) & `unknown_words_ratio` ($\approx 67\%$) |
+| **English** | WEB (World English) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `english` | `english` | **100%** | **0.0000** | `english_stopword_ratio` ($>0.08$) & `unknown_words_ratio` ($>0.70$) |
+
+> **Cognate Resolution:** Closely related sibling languages like Mizo share up to 78% unigram overlap with Hmar. `hmaraniam` cleanly resolves sibling Zo languages without requiring massive full dictionaries by combining vocabulary completeness (`unknown_words_ratio` $\le 0.18$) with curated structural markers (`sibling_zo_stopwords`).
+
+---
+
+### 2. Real-World Web Archive Benchmark (~1,300 Scraped Documents)
+
+Evaluated on real-world scraped web archives across 5 major Hmar/Zo web publishers:
+
+| Publisher Web Archive | Corpus Source / Format | Evaluated Items | Hmar Posts Detected (%) | English Posts Detected (%) | Other / Mixed Posts (%) | Avg Hmar Confidence | Mean Casual Hmar Ratio | Mean Formal Hmar Ratio | Mean Unknown Words Ratio | Primary Content Profile |
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- |
+| **L. Keivom Archive** (`keivom`) | Blogger API JSON | 300 | **199 (66.3%)** | 32 (10.7%) | 69 (23.0%) | **0.8320** | **79.3%** | 70.0% | 20.7% | Authentic Hmar literary essays & prose |
+| **Inpui Journal** (`inpui`) | Blogger API JSON | 291 | **124 (42.6%)** | 67 (23.0%) | 100 (34.4%) | **0.7400** | **63.7%** | 55.7% | 36.3% | Bilingual news journal & opinion pieces |
+| **HSA Portal** (`hsa`) | WordPress API JSON | 177 | **23 (13.0%)** | 38 (21.5%) | **116 (65.5%)** | 0.5730 | **53.7%** | 46.7% | 46.3% | Student association alerts & mixed posts |
+| **Hmarram.com** (`hmarram`) | WordPress API JSON | 235 | 12 (5.1%) | **209 (88.9%)** | 14 (6.0%) | 0.7762 | 31.2% | 23.8% | 68.8% | Tech articles & English press releases |
+| **Virthli News** (`virthli`) | Scraped Raw HTML | 296 | 5 (1.7%) | **267 (90.2%)** | 24 (8.1%) | 0.8721 | 17.6% | 13.4% | 82.4% | Employment alerts & exam guidelines |
+
+- **Literary Archives:** Archives like L. Keivom and Inpui Journal feature rich Hmar prose and opinion articles, achieving high Hmar classification rates ($42.6\% - 66.3\%$) and high token ratios ($63.7\% - 79.3\%$).
+- **Community News & Recruitment Notices:** Community portals like Hmarram and Virthli publish predominantly in English (recruitment notices, exam guidelines, press statements). `hmaraniam` accurately tags these as `"english"` or `"other"` without false-positive over-classification.
+
+---
+
+## Error Handling
+
+`hmaraniam` provides clear, descriptive error messages:
+
+```python
+import hmaraniam
+
+# Raises ValueError for unsupported modes
+try:
+    hmaraniam.detect("Text", mode="ultra")
+except ValueError as e:
+    print(e)
+
+# Raises TypeError for non-string input
+try:
+    hmaraniam.detect(12345)
+except TypeError as e:
+    print(e)
+```
+
+---
+
+## License
+
+Published under the MIT License by the **Hmar Heritage Project**.
+

hmaraniam-0.1.0/README.md

@@ -0,0 +1,173 @@
+# hmaraniam 🇲z
+
+**High-precision, zero-dependency language identification library for Hmar.**
+
+> *"Hmar a ni am?"* — *"Is it Hmar?"*
+
+`hmaraniam` is a lightweight Python library designed to accurately distinguish Hmar text from English and other Kuki-Chin / Zo languages (Mizo, Kuki, Paite, Vaiphei).
+
+---
+
+## Key Features
+
+- **Microsecond Speed:** $O(1)$ dictionary lookups with no heavy ML dependencies (PyTorch/TensorFlow free).
+- **Dual-Lens Diacritic Engine:** Reports both `casual_hmar_ratio` (ASCII-normalized for standard QWERTY typing) and `formal_hmar_ratio` (exact diacritic matching for formal literary text).
+- **Dual Continuous Confidence Metrics:** Disambiguates language metrics into `hmar_confidence` (permanent metric answering *"How confident are we that this text is Hmar?"*) and `detected_language_confidence` (confidence in the overall classification choice).
+- **Standardized Schema:** Guarantees an immutable JSON output structure across all calls, including `non_hmar_words_count` and `diacritic_words_count`.
+- **Extensible & Customizable:** Allows developers to supply `custom_unigrams`, `extra_unigrams`, `custom_stopwords`, or disable default stopwords.
+- **Dual Offline/CDN Architecture:** Automatically syncs with the live `hmar-heritage-org/hmaraniam` unigram dataset via jsDelivr CDN, with automatic local disk caching and bundled fallback.
+
+---
+
+## Installation
+
+```bash
+pip install hmaraniam
+```
+
+---
+
+## Standard Output Schema
+
+```json
+{
+  "language": "hmar",
+  "hmar_confidence": 0.9842,
+  "detected_language_confidence": 0.9842,
+  "mode": "basic",
+  "scores": {
+    "casual_hmar_ratio": 0.9524,
+    "formal_hmar_ratio": 0.8095,
+    "english_stopword_ratio": 0.0000,
+    "sibling_zo_stopword_ratio": 0.0000,
+    "unknown_words_ratio": 0.0476,
+    "total_words": 21,
+    "hmar_words_count": 20,
+    "non_hmar_words_count": 1,
+    "unknown_words_count": 1,
+    "english_stopwords_count": 0,
+    "sibling_zo_stopwords_count": 0,
+    "hmar_diacritic_words_count": 17,
+    "non_hmar_diacritic_words_count": 0,
+    "total_diacritic_words_count": 17
+  }
+}
+```
+
+---
+
+## Usage
+
+### Quick Start
+
+```python
+import hmaraniam
+
+# Authentic text from L. Keivom archive
+sample_text = "Khawvel fe dan phung ei en chun, ram le hnam damna thuruk chu lien lema intel le insung khawm a nih."
+
+result = hmaraniam.detect(sample_text)
+
+print(result)
+```
+
+### Custom Unigrams & Stopwords
+
+```python
+from hmaraniam import Detector
+
+# Provide custom unigrams or extra domain vocabulary
+detector = Detector(
+    mode="basic",
+    extra_unigrams=["customworda", "customwordb"],
+    custom_stopwords=["and", "the", "with"],
+    disable_default_stopwords=False
+)
+
+result = detector.detect("Khawvel fe dan phung...")
+```
+
+### Modes & Advanced Options
+
+```python
+from hmaraniam import Detector
+
+# Basic Mode (Active default ~30k core unigrams)
+basic_detector = Detector(mode="basic")
+
+# High Mode (Scans data/shards/ and loads all available unigram shards, falling back seamlessly to basic)
+high_detector = Detector(mode="high")
+
+# Offline-only mode (uses cached/bundled dataset without network calls)
+offline_detector = Detector(offline_only=True)
+```
+
+---
+
+## Empirical Benchmarks & Performance
+
+`hmaraniam` has been empirically validated across both **controlled parallel corpus datasets** (Parallel Zo Bibles across multiple literary genres) and **unfiltered real-world web archives** (~1,300 scraped articles and raw HTML pages across 5 major publishers).
+
+### 1. Controlled Parallel Zo Bible Benchmark
+
+Evaluated across parallel chapters (Genesis 1, Exodus 20, Matthew 5, Luke 2, Romans 8, Revelation 21) across 10 parallel Bible translations in 8 Zo languages + English:
+
+| Language | Edition / Source | Evaluated Passages | Target Class | Engine Assigned Label | Classification Accuracy | Avg Hmar Confidence | Key Distinguishing Metrics |
+| :--- | :--- | :--- | :---: | :---: | :---: | :---: | :--- |
+| **Hmar** | CLB (Contemporary) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `hmar` | `hmar` | **100%** | **1.0000** | `casual_hmar_ratio` $\ge 0.95$, `unknown_words_ratio` $\le 0.05$ |
+| **Hmar** | OV (Old Version) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `hmar` | `hmar` | **100%** | **1.0000** | Formal diacritic match + `hmar_diacritic_words_count` $>0$ |
+| **Mizo** | OV (Mizo Bible) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathian`, `hnenah`, `avangin`, `tichuan`) & `unknown_words_ratio` ($\approx 24\%$) |
+| **Paite** | Paite Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pasian`, `tungah`, `simhuai`) & `unknown_words_ratio` ($\approx 51\%$) |
+| **Vaiphei** | Vaiphei Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathian`, `tiu-in`, `apat`) & `unknown_words_ratio` ($\approx 38\%$) |
+| **Gangte** | Gangte Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathen`, `hepa`, `dih-in`) & `unknown_words_ratio` ($\approx 40\%$) |
+| **Zou** | Zou Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pasian`, `a-in`, `leh-in`) & `unknown_words_ratio` ($\approx 51\%$) |
+| **Thadou** | Thadou-Kuki Bible | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `other` | `other` | **100%** | **0.0000** | `sibling_zo_stopwords` (`pathen`, `chun`, `tichun`) & `unknown_words_ratio` ($\approx 67\%$) |
+| **English** | WEB (World English) | Gen 1, Ex 20, Matt 5, Luke 2, Rom 8, Rev 21 | `english` | `english` | **100%** | **0.0000** | `english_stopword_ratio` ($>0.08$) & `unknown_words_ratio` ($>0.70$) |
+
+> **Cognate Resolution:** Closely related sibling languages like Mizo share up to 78% unigram overlap with Hmar. `hmaraniam` cleanly resolves sibling Zo languages without requiring massive full dictionaries by combining vocabulary completeness (`unknown_words_ratio` $\le 0.18$) with curated structural markers (`sibling_zo_stopwords`).
+
+---
+
+### 2. Real-World Web Archive Benchmark (~1,300 Scraped Documents)
+
+Evaluated on real-world scraped web archives across 5 major Hmar/Zo web publishers:
+
+| Publisher Web Archive | Corpus Source / Format | Evaluated Items | Hmar Posts Detected (%) | English Posts Detected (%) | Other / Mixed Posts (%) | Avg Hmar Confidence | Mean Casual Hmar Ratio | Mean Formal Hmar Ratio | Mean Unknown Words Ratio | Primary Content Profile |
+| :--- | :--- | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :---: | :--- |
+| **L. Keivom Archive** (`keivom`) | Blogger API JSON | 300 | **199 (66.3%)** | 32 (10.7%) | 69 (23.0%) | **0.8320** | **79.3%** | 70.0% | 20.7% | Authentic Hmar literary essays & prose |
+| **Inpui Journal** (`inpui`) | Blogger API JSON | 291 | **124 (42.6%)** | 67 (23.0%) | 100 (34.4%) | **0.7400** | **63.7%** | 55.7% | 36.3% | Bilingual news journal & opinion pieces |
+| **HSA Portal** (`hsa`) | WordPress API JSON | 177 | **23 (13.0%)** | 38 (21.5%) | **116 (65.5%)** | 0.5730 | **53.7%** | 46.7% | 46.3% | Student association alerts & mixed posts |
+| **Hmarram.com** (`hmarram`) | WordPress API JSON | 235 | 12 (5.1%) | **209 (88.9%)** | 14 (6.0%) | 0.7762 | 31.2% | 23.8% | 68.8% | Tech articles & English press releases |
+| **Virthli News** (`virthli`) | Scraped Raw HTML | 296 | 5 (1.7%) | **267 (90.2%)** | 24 (8.1%) | 0.8721 | 17.6% | 13.4% | 82.4% | Employment alerts & exam guidelines |
+
+- **Literary Archives:** Archives like L. Keivom and Inpui Journal feature rich Hmar prose and opinion articles, achieving high Hmar classification rates ($42.6\% - 66.3\%$) and high token ratios ($63.7\% - 79.3\%$).
+- **Community News & Recruitment Notices:** Community portals like Hmarram and Virthli publish predominantly in English (recruitment notices, exam guidelines, press statements). `hmaraniam` accurately tags these as `"english"` or `"other"` without false-positive over-classification.
+
+---
+
+## Error Handling
+
+`hmaraniam` provides clear, descriptive error messages:
+
+```python
+import hmaraniam
+
+# Raises ValueError for unsupported modes
+try:
+    hmaraniam.detect("Text", mode="ultra")
+except ValueError as e:
+    print(e)
+
+# Raises TypeError for non-string input
+try:
+    hmaraniam.detect(12345)
+except TypeError as e:
+    print(e)
+```
+
+---
+
+## License
+
+Published under the MIT License by the **Hmar Heritage Project**.
+

hmaraniam-0.1.0/hmaraniam/__init__.py

@@ -0,0 +1,19 @@
+"""
+hmaraniam - High-precision language identification for Hmar.
+"Hmar a ni am?" -> "Is it Hmar?"
+
+Pure string language identification engine.
+"""
+
+from hmaraniam.detector import Detector, detect, get_default_detector
+
+__version__ = "0.1.0"
+__author__ = "Hmar Heritage Project"
+__license__ = "MIT"
+
+__all__ = [
+    "Detector",
+    "detect",
+    "get_default_detector",
+    "__version__",
+]