sindhi-tokenizer 1.1.0__tar.gz

sindhi_tokenizer-1.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Ali Nawaz, Muqadas Fareed, Soonh Sadaf
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

sindhi_tokenizer-1.1.0/PKG-INFO

@@ -0,0 +1,545 @@
+Metadata-Version: 2.4
+Name: sindhi-tokenizer
+Version: 1.1.0
+Summary: A SentencePiece Unigram tokenizer and lightweight Python SDK purpose-built for Sindhi.
+Author: Ali Nawaz, Muqadas Fareed, Soonh Sadaf
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/alinawazcode/sindhi-tokenizer
+Project-URL: Repository, https://github.com/alinawazcode/sindhi-tokenizer
+Project-URL: Models, https://huggingface.co/alinawazmahar/sindhi-tokenizer
+Keywords: sindhi,tokenizer,sentencepiece,unigram,nlp,low-resource-languages
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: sentencepiece>=0.1.99
+Requires-Dist: huggingface_hub>=0.23.0
+Provides-Extra: training
+Requires-Dist: datasets>=2.14.0; extra == "training"
+Requires-Dist: numpy>=1.22.0; extra == "training"
+Requires-Dist: pandas>=1.5.0; extra == "training"
+Provides-Extra: demo
+Requires-Dist: gradio>=4.0.0; extra == "demo"
+Requires-Dist: transformers>=4.30.0; extra == "demo"
+Provides-Extra: dev
+Requires-Dist: build>=1.2.0; extra == "dev"
+Requires-Dist: pytest>=7.4.0; extra == "dev"
+Requires-Dist: twine>=5.0.0; extra == "dev"
+Dynamic: license-file
+
+# Sindhi Tokenizer: A Corpus-Driven SentencePiece Unigram Tokenizer
+
+<p align="center">
+  <img src="assets/banner.png" alt="Sindhi Tokenizer" width="100%">
+</p>
+
+[![Python 3.9+](https://img.shields.io/badge/python-3.9%20%7C%203.10%20%7C%203.11%20%7C%203.12-blue.svg)](https://www.python.org/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-green.svg)](LICENSE)
+[![Hugging Face Models](https://img.shields.io/badge/%F0%9F%A4%97%20Hugging%20Face-alinawazmahar%2Fsindhi--tokenizer-yellow)](https://huggingface.co/alinawazmahar/sindhi-tokenizer)
+[![Documentation](https://img.shields.io/badge/Docs-GitHub%20Pages-informational)](https://alinawazcode.github.io/sindhi-tokenizer/)
+[![Tests](https://github.com/alinawazcode/sindhi-tokenizer/actions/workflows/python-tests.yml/badge.svg)](https://github.com/alinawazcode/sindhi-tokenizer/actions)
+[![Round-trip check: 0.000% normalized-length loss](https://img.shields.io/badge/Round--trip%20check-0.000%25%20normalized--length%20loss-brightgreen)](#round-trip-character-fidelity)
+
+A subword tokenizer purpose-built for the **Sindhi language** (سنڌي) in the Perso-Arabic script. Trained with **SentencePiece Unigram** with full observed-character coverage and byte-level fallback on 1.46M+ sentences. It was benchmarked against **14 baseline tokenizers** across four evaluation registers. The benchmark has 18 entries: 14 baseline entries and four custom vocabulary sizes. **Qwen2.5 and SindhiLM_v2 are identical in this evaluation**, so the entries represent 17 distinct tokenizer behaviors. The round-trip check reports **0.0000% normalized-length loss** on evaluated text; this does not establish exact identity for arbitrary input.
+
+---
+
+### Quick Links
+- 🌐 **Documentation Site**: [https://alinawazcode.github.io/sindhi-tokenizer/](https://alinawazcode.github.io/sindhi-tokenizer/)
+- 🤗 **Hugging Face Hub**: [alinawazmahar/sindhi-tokenizer](https://huggingface.co/alinawazmahar/sindhi-tokenizer)
+- 📄 **Research Manuscript Draft (Not Yet Published)**: [`docs/paper.md`](docs/paper.md)
+- 🔬 **Technical Explainer**: [`docs/explainer.md`](docs/explainer.md)
+- 📊 **Benchmark Results**: [`results/README.md`](results/README.md)
+- 💻 **Interactive Web Demo**: [`demo/app.py`](demo/app.py)
+
+---
+
+## Table of Contents
+1. [Project Overview](#project-overview)
+2. [Motivation and Problem Statement](#motivation-and-problem-statement)
+3. [Project Objectives and Scope](#project-objectives-and-scope)
+4. [Key Features](#key-features)
+5. [Technology Stack](#technology-stack)
+6. [Repository Architecture](#repository-architecture)
+7. [Architecture and Workflow Diagrams](#architecture-and-workflow-diagrams)
+8. [Dataset Documentation](#dataset-documentation)
+9. [Tokenizer Methodology](#tokenizer-methodology)
+10. [Training Pipeline & Negative Results](#training-pipeline--negative-results)
+11. [Installation and Usage](#installation-and-usage)
+12. [Evaluation and Comprehensive Benchmarks](#evaluation-and-comprehensive-benchmarks)
+13. [Linguistic & Morpheme Analysis](#linguistic--morpheme-analysis)
+14. [Downstream NLP Applications](#downstream-nlp-applications)
+15. [Limitations and Known Issues](#limitations-and-known-issues)
+16. [Complete Documentation Index](#complete-documentation-index)
+17. [References and Further Reading](#references-and-further-reading)
+18. [Team and Acknowledgments](#team-and-acknowledgments)
+19. [License and Citation](#license-and-citation)
+20. [Frequently Asked Questions (FAQ)](#frequently-asked-questions-faq)
+
+---
+
+## Project Overview
+
+Sindhi is an Indo-Aryan language spoken by an estimated 25–30 million people, predominantly in the Sindh province of Pakistan and parts of India. Despite its rich literary heritage, Sindhi remains a severely under-resourced language in Natural Language Processing (NLP).
+
+Existing modern Large Language Models (LLMs) and multilingual tokenizers (such as XLM-RoBERTa, LLaMA, Qwen, and BLOOM) under-represent Sindhi orthography, causing high token fertility, extreme sequence bloat, out-of-vocabulary (UNK) fallbacks, or destructive character drops. 
+
+This repository provides:
+- **Four Trained Models**: Pretrained SentencePiece Unigram models at 32k, 64k (flagship recommended), 100k, and 150k vocabulary sizes.
+- **Corpus Engineering Pipeline**: A 1.46M+ sentence corpus drawn from Sindhi and regional Wikipedia, mostly news text, a small Twitter component, and an Urdu/Persian/Arabic anchor that makes up 16.1% of the corpus by character count (89,449 unique sentences, repeated four times).
+- **Full-Scale Benchmark Suite**: Standardized evaluation of 18 tokenizers across 4 distinct registers (FLORES+, Wikipedia, Sentiment, Sindh Line News).
+- **Round-Trip Length Check**: A normalized-length check that can flag text shortening; targeted examples were also inspected for deleted characters.
+- **Interactive UI**: A Gradio web application for real-time subword tokenization inspection.
+
+---
+
+## Motivation and Problem Statement
+
+### 1. Unique Script and Extended Perso-Arabic Orthography
+Sindhi uses a 52-letter extended Perso-Arabic alphabet containing implosive consonants (ٻ, ڄ, ڏ, ڳ), aspirated digraphs (ڀ, ٿ, ڇ, ٺ), retroflex consonants (including ڻ), and specialized diacritics. General Arabic or Urdu tokenizers frequently split these single graphemes into multi-byte fallback sequences, dramatically inflating token counts.
+
+### 2. The Token Fertility Tax
+When a tokenizer splits a single word into 3–5 subword fragments, context window efficiency drops proportionally. A 2048-token context window accommodates only ~707 Sindhi words under LLaMA 3.2 (fertility 2.896) or ~1,354 words under XLM-RoBERTa (1.512), compared to ~1,786 Sindhi words under CustomSindhi_64k (1.147).
+
+### 3. Silent Character Deletion in Competing Models
+Standard tokenizer benchmarks often report token fertility and `UNK` rate without checking decoded output. During this project's audit, a **normalized-length round-trip check**, followed by inspection of affected examples, surfaced text-shortening defects in existing tokenizers:
+- **`TinyLlama_Sindhi`**: Drops native Sindhi letters such as `U+06C1` (HEH GOAL) and corrupts text encoded with Arabic Presentation Forms, destroying data in up to **24.9%** of sentences.
+- **`FahadQazi_BPE`**: Silently deletes Latin characters in code-switched technical text, corrupting up to **16.2%** of sentences.
+- **Our model**: Shows **0.0000% normalized-length loss** under this check on evaluated text with byte-level fallback; the metric is not proof of exact character identity.
+
+---
+
+## Project Objectives and Scope
+
+### Primary Objectives
+1. **Deliver Flagship Tokenizer**: Train and release a high-efficiency Sindhi subword tokenizer achieving the lowest fertility and highest single-token preservation across diverse registers.
+2. **Measure Round-Trip Integrity**: Use byte fallback and full observed-character coverage, then report the limits of the normalized-length check.
+3. **Rigorous Empirical Comparison**: Benchmark 14 baseline tokenizers under identical, unsampled test conditions.
+4. **Transparent Documentation**: Document negative results (what didn't work), corpus preparation heuristics, and evaluation caveats.
+
+### Supported Use Cases
+- Large Language Model (LLM) pretraining and fine-tuning for Sindhi.
+- Sequence classification (Sentiment Analysis, Topic Modeling).
+- Machine Translation (FLORES+ benchmarks).
+- Named Entity Recognition (NER) and Information Retrieval.
+
+### Known Exclusions
+- Speech recognition (ASR) acoustic unit modeling.
+- Devnagari script Sindhi (this release targets Arabic-script Sindhi).
+
+---
+
+## Key Features
+
+- **SentencePiece Unigram Architecture**: Statistically models subword probabilities using maximum likelihood estimation, proven superior to standard greedy BPE for morphologically rich languages.
+- **100% Character Coverage + Byte Fallback**: Zero unknown (`<unk>`) token emissions. Unknown Unicode characters decompose cleanly into raw UTF-8 byte tokens without data corruption.
+- **Flagship 64k Model**: Balances vocabulary efficiency (1.147 fertility) with embedding table size, yielding an **88.6% single-token word retention** rate.
+- **Round-Trip Check Result**: 0.0000% normalized-length loss across the evaluated corpora; this does not establish exact character identity.
+- **Clean Subword Boundaries**: Preserves leading whitespace symbols (` `, `\u2581`) for precise detokenization without text corruption.
+
+---
+
+## Technology Stack
+
+| Layer | Technologies | Purpose |
+|---|---|---|
+| **Core Engine** | Python 3.9+, [SentencePiece](https://github.com/google/sentencepiece) (C++ bindings) | Tokenizer training, Viterbi segmentation, byte fallback |
+| **NLP Framework** | [Hugging Face Transformers](https://github.com/huggingface/transformers), `huggingface_hub` | Adapter integration, model sharing and distribution |
+| **Datasets** | [Hugging Face Datasets](https://github.com/huggingface/datasets), Pandas, NumPy | Corpus ingestion, normalization, filtering, metrics computation |
+| **Interactive Demo** | [Gradio](https://gradio.app/) | Real-time interactive subword visualization UI |
+| **Quality & CI/CD** | Pytest, GitHub Actions | Automated regression tests across Python 3.9–3.12 |
+| **Documentation** | GitHub Pages, Markdown, Mermaid.js | Static web docs, draft research manuscript, technical explainer |
+
+---
+
+## Repository Architecture
+
+```text
+sindhi-tokenizer/
+├── .github/
+│   └── workflows/
+│       └── python-tests.yml        # Automated CI workflow testing Python 3.9-3.12
+├── .gitignore                      # Excludes models, artifacts, cache, and virtualenvs
+├── CITATION.cff                    # Citation File Format (v1.2.0)
+├── DATA_SOURCES.md                 # Complete provenance, volume, and licensing documentation
+├── LICENSE                         # MIT Open Source License
+├── README.md                       # Central documentation index and project guide
+├── pyproject.toml                  # PEP 517/621 package metadata and dependency configuration
+├── requirements.txt                # Exact runtime dependencies
+├── demo/
+│   ├── README.md                   # Instructions for running and deploying the demo
+│   └── app.py                      # Interactive Gradio comparison web app
+├── docs/
+│   ├── paper.md                    # Draft research manuscript, not yet published (14-baseline study)
+│   └── explainer.md                # 21-page technical companion log and negative results
+├── evaluation/
+│   ├── README.md                   # Evaluation metrics guide and CLI/API documentation
+│   ├── __init__.py                 # Package initialization
+│   └── metrics.py                  # Fertility, single-token, UNK, byte-fallback, and fidelity engine
+├── notebooks/
+│   ├── README.md                   # Reproducible notebook guide
+│   ├── training.ipynb              # Kaggle/Jupyter notebook: corpus download, split & training
+│   └── benchmark.ipynb             # Kaggle/Jupyter notebook: 18-tokenizer evaluation suite
+├── results/
+│   └── README.md                   # Benchmark tables and summary report across 4 registers
+└── training/
+    ├── README.md                   # Training recipes, regex splitters, and negative results
+    ├── __init__.py                 # Package initialization
+    └── train_tokenizer.py          # Standalone training script with optimal SentencePiece parameters
+```
+
+---
+
+## Architecture and Workflow Diagrams
+
+### 1. End-to-End System Architecture
+
+```mermaid
+flowchart TD
+    subgraph Data_Pipeline["1. Corpus Assembly & Preprocessing"]
+        A1["Sindhi Wikipedia<br/>(6,000 articles)"] --> P["Sentence Extraction & Cleaning<br/>Regex Split: [۔.!\n؟]+<br/>Drop fragments of 3 characters or fewer"]
+        A2["Sindhi-NLP Dataset<br/>(Twitter + Kawish + Awami Awaz)"] --> P
+        A3["shaikhsalman Corpus<br/>(20k rows, no language filter)"] --> P
+        A4["Multilingual Regional Anchor<br/>(Urdu, Persian, Arabic, repeated ×4, 16.1% of characters)"] --> P
+        P --> C["Balanced Training Corpus<br/>(1.46M+ Sentences)"]
+    end
+
+    subgraph Training_Engine["2. Model Training (SentencePiece)"]
+        C --> SP["SentencePiece Unigram Trainer<br/>character_coverage=1.0<br/>byte_fallback=True<br/>split_by_whitespace=True"]
+        SP --> M1["Custom 32k Model"]
+        SP --> M2["Custom 64k Model (Flagship)"]
+        SP --> M3["Custom 100k Model"]
+        SP --> M4["Custom 150k Model"]
+    end
+
+    subgraph Evaluation_Suite["3. Rigorous Evaluation Engine"]
+        M2 --> E["evaluation/metrics.py"]
+        REG1["FLORES+ (snd_Arab)"] --> E
+        REG2["Wikipedia Formal (25.4k sent)"] --> E
+        REG3["Sentiment Informal (3.5k sent)"] --> E
+        REG4["Sindh Line News (87.1k sent)"] --> E
+        E --> MET["Metrics Computed:<br/>• Token Fertility<br/>• Single-Token Word %<br/>• UNK Rate (0.000%)<br/>• Byte Fallback Rate<br/>• Normalized-Length Loss (0.000%)"]
+    end
+
+    subgraph Downstream["4. Downstream Deployment"]
+        M2 --> HF["Hugging Face Hub<br/>alinawazmahar/sindhi-tokenizer"]
+        M2 --> GR["Interactive Gradio Demo<br/>demo/app.py"]
+        M2 --> LLM["Sindhi LLM Pretraining & NLP"]
+    end
+```
+
+### 2. Text Encoding and Round-Trip Fidelity Flow
+
+```mermaid
+sequenceDiagram
+    autonumber
+    participant Input as Raw Sindhi Text
+    participant Norm as NFKC & Formatting Normalizer
+    participant Tokenizer as SPMTokenizerAdapter
+    participant Tokens as Subword Tokens / Byte IDs
+    participant Output as Decoded Reconstructed Text
+    participant Audit as Normalized-Length Check
+
+    Input->>Norm: Normalize Unicode (NFKC) & remove zero-width formatting
+    Norm->>Tokenizer: Clean text string
+    Tokenizer->>Tokens: Encode via Unigram Viterbi path (out_type=int)
+    Tokens->>Output: Decode token IDs back to string
+    Output->>Audit: Compare Original vs Decoded
+    Note over Audit: Loss = max(0, input_length - decoded_length) / input_length<br/>Custom Model Result: 0.0000% normalized-length loss
+```
+
+---
+
+## Dataset Documentation
+
+Training and evaluation records are separated where known, but the sentiment evaluation set shares publisher-adjacent provenance with a training source. See [`DATA_SOURCES.md`](DATA_SOURCES.md) for source details and licensing status.
+
+### Training Corpus (1.46M+ Sentences)
+
+| Source Dataset | Origin / Provider | Volume Used | Preprocessing & Filtering Rules | License |
+|---|---|---|---|---|
+| **Sindhi Wikipedia** | `wikimedia/wikipedia` (`20231101.sd`) | 6,000 articles | Articles 2,001–8,000 (first 2,000 reserved for evaluation). Fragments of 3 characters or fewer dropped; no word-count limit (the 3–100 word filter is applied only to the evaluation set). | CC BY-SA 3.0 |
+| **Sindhi-NLP** | `GitHub: Sindhi-NLP/sindhi-NLP-dataset` | 4 CSV collections | Primarily news (98.4% Awami Awaz and Daily Kawish articles) with 1.6% social tweets. | MIT |
+| **shaikhsalman Corpus** | `shaikhsalman/sindhi-llm-corpus` | 20,000 rows | Seed 7 shuffle; 630,180 sentences, about 57% of the Sindhi-side corpus. **No language filter was applied.** A diagnostic found about 11.3% of sampled sentences were mostly Latin script, and they remain in the training text. | Project owner reports free use; exact license and conditions are not shown on the card (see DATA_SOURCES.md) |
+| **Multilingual Anchor** | Urdu, Persian, Arabic Wikipedia | 1,500 articles each | 89,449 unique sentences (4.9M characters) repeated four times to reach 16.1% of total corpus characters (target was 15%). Exposes model to shared Perso-Arabic roots. | CC BY-SA 3.0 |
+
+### Evaluation Registers (Unsampled Benchmarks)
+
+| Register | Dataset Identifier | Sentences | Domain / Linguistic Style |
+|---|---|---|---|
+| **`flores200_anchor`** | `openlanguagedata/flores_plus` (`snd_Arab`) | 1,012 | Standardized translation benchmark; formal and modern register. |
+| **`wikipedia_formal`** | `wikimedia/wikipedia` (`20231101.sd`) | 25,480 | Formal encyclopedic Sindhi prose (first 2,000 articles, disjoint from train). |
+| **`sentiment_informal`** (legacy ID) | `alinawazmahar/Sindhi_Sentiment_dataset` | 3,536 | Newspaper text pseudo-labeled for sentiment; this is news, not social-media text. |
+| **`sindh_line_news`** | Mozilla Data Collective (`Sindh Line Publishers`) | 87,158 | Contemporary, daily regional news coverage (CC BY-SA 4.0). |
+
+---
+
+## Tokenizer Methodology
+
+### Why SentencePiece Unigram over Greedy BPE?
+Byte-Pair Encoding (BPE) merges the most frequent character pairs iteratively. In morphologically complex, agglutinative, or clitic-heavy languages like Sindhi, greedy merges frequently fuse grammatical suffixes onto root stems incorrectly. 
+
+The **Unigram language model algorithm** (Kudo, 2018) initializes an oversized vocabulary and optimizes subword segmentation using maximum likelihood:
+
+$$P(X) = \max_{\mathbf{x} \in S(X)} \prod_{i=1}^{M} P(x_i)$$
+
+This statistically discovers true morphological subwords while optimizing whole-sequence likelihood.
+
+### Step-by-Step Tokenization Example
+Consider the authentic Sindhi phrase:
+$$\text{خداوند ڪريم کيس اعليٰ ذهانت سان نوازيو هو}$$
+*(The Almighty Lord had blessed him with high intellect)*
+
+1. **Input Normalization**: NFKC normalization ensures canonical representation of Arabic script forms.
+2. **Whitespace Prefixing**: Words receive the boundary marker ` ` (`\u2581`).
+3. **Unigram Segmentation**:
+   ```text
+   [' خداوند', ' ڪريم', ' کيس', ' اعليٰ', ' ذهانت', ' سان', ' نوازيو', ' هو']
+   ```
+4. **Token ID Mapping**:
+   ```text
+   [12450, 4812, 1089, 7543, 14210, 521, 18920, 412]
+   ```
+5. **Round-trip check**: Evaluated text showed 0.0000% normalized-length loss. This length-based check can miss equal-length substitutions or reorderings.
+
+---
+
+## Training Pipeline & Negative Results
+
+### 1. Training Parameters (`training/train_tokenizer.py`)
+```bash
+python training/train_tokenizer.py \
+  --corpus corpus.txt \
+  --vocab_size 64000 \
+  --model_prefix models/sindhi_custom_64000
+```
+- `model_type="unigram"`
+- `character_coverage=1.0` (Ensures every single observed Unicode character is covered)
+- `byte_fallback=True` (Unknown characters fall back to raw byte tokens `<0xXX>`)
+- `split_by_whitespace=True`
+- `unk_id=0`, `bos_id=1`, `eos_id=2`, `pad_id=-1`
+
+### 2. Documented Negative Results (What Failed and Why)
+In accordance with rigorous research standards, two negative experimental results are preserved in [`docs/explainer.md`](docs/explainer.md):
+1. **Forced Postposition Tokens**: Forcing common postpositions (e.g., `جو`, `جي`, `کي`) and loanwords to remain single tokens degraded the training objective (final objective value rose from ≈9.6 unforced to ≈14.9 forced at 32k vocab). Given enough data, Unigram naturally discovers these tokens with superior probability allocations.
+2. **Pre-marking Clitic Boundaries**: Pre-segmenting supposed fused clitics with whitespace proved harmful. Investigation revealed that suspected "fused forms" were simply coincidental orthographic overlaps with unrelated lexical roots.
+
+---
+
+## Installation and Usage
+
+### Install the Python SDK
+
+Requires Python 3.9 or newer. The SDK only installs SentencePiece and the Hugging Face Hub client; training and demo dependencies are optional.
+
+```bash
+pip install sindhi-tokenizer
+```
+
+### Quickstart
+
+```python
+from sindhi_tokenizer import SindhiTokenizer
+
+# Downloads the recommended 64k model once, then reuses the Hub cache.
+tokenizer = SindhiTokenizer.from_pretrained()
+text = "سنڌ جي تاريخ تمام قديم آهي"
+
+pieces = tokenizer.tokenize(text)
+token_ids = tokenizer.encode(text)
+
+print("Pieces:", pieces)
+print("Token IDs:", token_ids)
+print("Decoded:", tokenizer.decode(token_ids))
+print("Vocabulary size:", tokenizer.vocab_size)  # 64000
+```
+
+Choose another released vocabulary with `vocab_size=32000`, `100000`, or `150000`. You can also load a local `.model` file with `SindhiTokenizer.from_file(path)`. Batch methods are `encode_batch()` and `decode_batch()`.
+
+To work on the repository and its training/demo tools:
+
+```bash
+git clone https://github.com/alinawazcode/sindhi-tokenizer.git
+cd sindhi-tokenizer
+pip install -e ".[dev,training,demo]"
+```
+
+The model download is cached by Hugging Face Hub. The SDK code is MIT licensed; the model files are separately licensed under CC BY-SA 4.0. Review the model card for attribution and sharing terms before redistributing model files.
+
+> [!CAUTION]
+> **Do not use `transformers.LlamaTokenizer` to load raw `.model` files**: it can silently return a 3-token fallback vocabulary for these custom SentencePiece Unigram files. Use `SindhiTokenizer`, `sentencepiece.SentencePieceProcessor`, or the benchmark adapter in [`evaluation/metrics.py`](evaluation/metrics.py).
+
+---
+
+## Evaluation and Comprehensive Benchmarks
+
+All 18 benchmark entries were evaluated across the four evaluation registers without subsampling. SindhiLM_v2 is identical to Qwen2.5 for this evaluation; the entries therefore represent 17 distinct tokenizer behaviors.
+
+### Macro Benchmark Summary (Unweighted Mean across 4 Registers)
+
+| Tokenizer | Vocab Size | Fertility (tokens/word) ↓ | Single-Token Word % ↑ | UNK Rate % ↓ | Byte-Fallback % ↓ | Worst Normalized-Length Loss % ↓ |
+|---|---|---|---|---|---|---|
+| **CustomSindhi_150k** | 150,000 | **1.125** | **89.7%** | **0.000%** | 0.154% | **0.0000%** |
+| **CustomSindhi_100k** | 100,000 | **1.133** | **89.3%** | **0.000%** | 0.149% | **0.0000%** |
+| **CustomSindhi_64k (Flagship)** | 64,000 | **1.147** | **88.6%** | **0.000%** | 0.138% | **0.0000%** |
+| **CustomSindhi_32k** | 32,000 | **1.184** | **86.5%** | **0.000%** | 0.113% | **0.0000%** |
+| TinyLlama_Sindhi | 42,681 | 1.208 | 84.4% | 0.000%* | 0.000%* | **0.4147% (Corrupts 24.9% sent)** |
+| hellosindh_SindhiBERT_NER | 32,001 | 1.218 | 83.7% | 0.194% | 0.197% | 0.0751% |
+| mBERT_Sindhi_Kashif786 | 119,547 | 1.278 | 78.4% | 0.143% | 0.000% | 0.0021% |
+| XLM-RoBERTa | 250,002 | 1.512 | 65.4% | 0.034% | 1.402% | 0.0040% |
+| SindhiLM_v3 | 32,000 | 1.730 | 55.8% | 3.605% | 0.000% | 0.0754% |
+| FahadQazi_BPE | 13,031 | 1.919 | 39.1% | 0.000%* | 7.127%* | **0.8461% (Corrupts 16.2% sent)** |
+| hellosindh_SindhiBERT | 32,000 | 1.960 | 35.5% | 0.122% | 0.114% | 0.0089% |
+| mT5 | 250,100 | 2.280 | 21.7% | 0.022% | 12.746% | 0.0040% |
+| Gemma3 | 262,144 | 2.308 | 26.8% | 0.000% | 0.105% | 0.0000% |
+| Qwen2.5 / SindhiLM_v2 | 151,643 | 2.580 | 23.8% | 0.000% | 8.665% | 0.0000% |
+| BLOOM | 250,680 | 2.627 | 24.7% | 0.000% | 31.660% | 0.0000% |
+| Llama3.2 | 128,000 | 2.896 | 18.2% | 0.000% | 28.472% | 0.0000% |
+| SindhiLM_v1 | 43 | 4.306 | 0.1% | 14.017% | 0.000% | 0.0790% |
+
+*\* TinyLlama_Sindhi and FahadQazi_BPE show apparent 0.000% UNK rates because both models silently delete unsupported characters rather than emitting a UNK token. FahadQazi_BPE shows real, severe character loss on round-trip fidelity testing alongside high byte-fallback. SindhiLM_v2 is numerically identical to Qwen2.5 across all metrics; both are listed in the table for completeness.*
+
+### Evaluation Takeaways
+1. **Lowest fertility and highest single-token retention at ≥64k**: CustomSindhi (64k, 100k, 150k) has the lowest fertility and highest single-token retention in all four registers. The margin on FLORES+ at 64k is 0.002 fertility and 0.3 points of single-token retention, which we treat as parity; no significance tests were run. At 32k, one narrow exception exists: TinyLlama_Sindhi beats CustomSindhi_32k on FLORES+ fertility (though it carries a critical character-deletion defect). On byte-fallback, CustomSindhi_64k (0.138%) achieves near-lowest fallback, with Gemma 3 slightly lower at 0.105%.
+2. **Efficiency gains at 64k**: Mean fertility improves by **24.1% over XLM-RoBERTa** and **60.4% over LLaMA 3.2**. This describes fertility and single-token retention, not every metric.
+3. **Round-trip check results**: Custom models show 0.0000% normalized-length loss under the stated check on all four registers. This length-based check does not prove exact character identity; TinyLlama_Sindhi and FahadQazi_BPE show measurable length loss.
+
+---
+
+## Linguistic & Morpheme Analysis
+
+To verify that vocabulary expansion did not simply overfit high-frequency substrings, all tokenizers were evaluated on a targeted 22-word linguistic test suite spanning four distinct categories:
+
+1. **Genitive & Possessive Constructions** (`دنيا جو`, `پاڻي جي`, `ماڻهن جو`): All Custom models split these cleanly into base noun + postposition clitic.
+2. **Inflectional Suffixes** (`ڪتابن`, `شهرن`, `توهانکي`, `منهنجو`): Preserved as single tokens in Custom 64k models.
+3. **Sindhi Implosives & Aspirated Digraphs** (`ٻار`, `ڄاڻ`, `ڏينهن`, `ڳوٺ`, `ڀاءُ`, `ٿورو`, `ڇوڪرو`): Never broken into disconnected diacritics or fallback bytes.
+4. **Loanwords & Technical Terms** (`ڪمپيوٽر`, `انٽرنيٽ`, `اسڪول`, `تعليم`): Retained as intact single lexical units.
+
+> [!NOTE]
+> **Aggregate vs. Word-Level Distinction**: Across the 22-word test suite, the Custom tokenizer ties the best competitor on 21 of 22 words and loses one (`هنجي`, to FahadQazi_BPE and mBERT_Sindhi jointly). The primary advantage of CustomSindhi is consistency across aggregate, full-scale text registers rather than hand-picked individual words. See [`docs/paper.md#appendix-b`](docs/paper.md#appendix-b) for the full 22-word breakdown.
+
+---
+
+## Intended Downstream NLP Applications & Hypotheses
+
+*Note: As documented in the paper (§6), the tokenizer is designed for textual NLP representation; downstream task-specific fine-tuning (e.g. BLEU/ROUGE benchmarks) represents intended future work.*
+
+| Intended Application | Hypothesized Architectural Benefit |
+|---|---|
+| **Sindhi Large Language Models (LLMs)** | Decreases token sequence length by 50–60% vs general multilingual LLMs (LLaMA, BLOOM), effectively doubling or tripling context length capacity during pretraining and inference. |
+| **Machine Translation (MT)** | Preserves postpositions and implosive graphemes intact, intended to reduce subword fragmentation and hallucination during sequence-to-sequence translation. |
+| **Sentiment Analysis & Text Classification** | Retains full grammatical morphemes, yielding richer dense embeddings for subword encoders. |
+| **Named Entity Recognition (NER)** | Eliminates character drop bugs that corrupt proper nouns, locations, and transliterated person names. |
+
+---
+
+## Limitations and Known Issues
+
+In accordance with rigorous academic transparency (see [`docs/paper.md` §6](docs/paper.md)), ten limitations are explicitly recorded:
+
+1. **Absence of Classical / Literary Register**: No classical or literary Sindhi register is included. Five independent candidate sources were investigated: a corrupted CSV export, an unrelated SQL dump, two unusable OCR scans (one incompatibly licensed), and a government-affiliated platform gated behind an institutional subscription.
+2. **Partial Source Overlap in Evaluation**: The Sentiment register and the Sindhi-NLP component of the training corpus share adjacent news-source provenance; full genre-level independence is not established for this pair.
+3. **Metric-specific results**: Against FLORES+ specifically, CustomSindhi_32k has one confirmed loss on fertility and single-token retention. The all-register lead at 64k and above refers to those two measures; byte-fallback is a separate trade-off.
+4. **Qualitative Test Does Not Differentiate at Word Level**: On the 22-word test list, the new tokenizer ties rather than beats the best competitor on 21 of 22 words; the demonstrated advantage is aggregate across full corpora, not example-level.
+5. **Search for Existing Tokenizers May Be Incomplete**: Four additional tokenizers were discovered after initial benchmarking; while subsequent searches found no further models, the open-source NLP space is active enough that this cannot be guaranteed exhaustive.
+6. **Downstream Applications Undetermined**: The tokenizer's target use is future Sindhi language modeling. It is not validated for Automatic Speech Recognition (ASR) or Text-to-Speech (TTS).
+7. **Register Weighting Choice**: The four evaluation registers differ substantially in size; an unweighted macro mean was used rather than a token-pooled mean, which would have been dominated by Sindh Line News (~2.5× the other three combined).
+8. **No language filter on training text**: About 11.3% of a diagnostic sample of the shaikhsalman text was mostly Latin script. No filter was applied before training, so English text is part of the training corpus.
+9. **Repeated cross-lingual text**: The Urdu, Persian and Arabic part of the corpus is 89,449 unique sentences repeated four times. Its share by character count is 16.1%, but its unique content is much smaller.
+10. **Possible overlap with baseline training data**: The SindhiLM model card names the ambile-official Sindhi Mega Corpus as training data, and that corpus lists Sindhi Wikipedia among its sources. SindhiLM may therefore have seen text from our Wikipedia evaluation set, which would favor that baseline, not ours.
+
+---
+
+## Complete Documentation Index
+
+| Document | Path | Purpose |
+|---|---|---|
+| **Root Guide & Navigation Hub** | [`README.md`](README.md) | Central repository overview, quickstart, and full benchmark summary |
+| **Research Manuscript Draft** | [`docs/paper.md`](docs/paper.md) | Draft for future publication; not yet published. Includes methodology and 18 benchmark rows |
+| **Technical Explainer Log** | [`docs/explainer.md`](docs/explainer.md) | In-depth engineering explainer into empirical failures, metric corrections, and design rationale |
+| **Dataset Provenance Card** | [`DATA_SOURCES.md`](DATA_SOURCES.md) | Complete documentation of all training corpora and evaluation sets, licensing, and volumes |
+| **Evaluation Module Guide** | [`evaluation/README.md`](evaluation/README.md) | Technical documentation of metrics algorithms, CLI arguments, and Python API |
+| **Training Pipeline Guide** | [`training/README.md`](training/README.md) | Exact SentencePiece training recipe, filtering thresholds, and negative findings |
+| **Benchmark Result Summaries** | [`results/README.md`](results/README.md) | Detailed register breakdowns across FLORES+, Wikipedia, Sentiment, and Sindh Line |
+| **Notebook Reproduction Guide** | [`notebooks/README.md`](notebooks/README.md) | Instructions for Kaggle/Colab execution of training and benchmarking pipelines |
+| **Interactive Demo Guide** | [`demo/README.md`](demo/README.md) | Local execution and Hugging Face Spaces deployment guide for Gradio UI |
+| **Citation Specification** | [`CITATION.cff`](CITATION.cff) | Machine-readable CFF citation metadata |
+| **Project Packaging** | [`pyproject.toml`](pyproject.toml) | Modern build-system and packaging metadata |
+
+---
+
+## References and Further Reading
+
+1. **Kudo, T. & Richardson, J. (2018).** *SentencePiece: A simple and language independent subword tokenizer and detokenizer for neural text processing.* EMNLP 2018 System Demonstrations. [https://aclanthology.org/D18-2012/](https://aclanthology.org/D18-2012/)
+2. **Kudo, T. (2018).** *Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates.* ACL 2018. [https://aclanthology.org/P18-1007/](https://aclanthology.org/P18-1007/)
+3. **Conneau, A., et al. (2020).** *Unsupervised Cross-lingual Representation Learning at Scale (XLM-RoBERTa).* ACL 2020. [arXiv:1911.02116](https://arxiv.org/abs/1911.02116)
+4. **NLLB Team et al. (2022).** *No Language Left Behind: Scaling Human-Centered Machine Translation (FLORES-200).* [arXiv:2207.04672](https://arxiv.org/abs/2207.04672)
+5. **Qwen Team (2024).** *Qwen2.5 Technical Report.* [arXiv:2412.15115](https://arxiv.org/abs/2412.15115)
+6. **Meta AI (2024).** *Llama 3.2 Model Card & Architecture Overview.* [https://huggingface.co/meta-llama](https://huggingface.co/meta-llama)
+
+*Note: For the complete list of 21 baseline model citations and references, please refer to [`docs/paper.md`](docs/paper.md).*
+
+---
+
+## Team and Acknowledgments
+
+This research and tokenizer implementation were developed at the **Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan**, under the supervision of **Dr. Attaullah Sahito** (academic supervisor, not an author).
+
+### Authors and Roles
+- **Ali Nawaz** (Lead author and project lead): [GitHub (@alinawazcode)](https://github.com/alinawazcode)
+- **Muqadas Fareed** (author): [LinkedIn Profile](https://www.linkedin.com/in/muqadas-fareed-24a07837b/)
+- **Soonh Sadaf (Sadaf Khaliq)** (author): [LinkedIn Profile](https://www.linkedin.com/in/sadaf-khaliq-a69215411/)
+- **Dr. Attaullah Sahito** (academic supervisor): [LinkedIn Profile](https://www.linkedin.com/in/attaullah-sahito/?isSelfProfile=false), Department of Computer Science, Shaikh Ayaz University.
+
+---
+
+## License and Citation
+
+### License
+- **Software Code**: Licensed under the open-source [MIT License](LICENSE).
+- **Trained Tokenizer Models**: The Hugging Face listing states [CC BY-SA 4.0](https://creativecommons.org/licenses/by-sa/4.0/). Source datasets are reported by the project lead to be free to use, mostly under MIT and other open licenses; exact terms and attribution conditions vary by source and are summarized in [`DATA_SOURCES.md`](DATA_SOURCES.md).
+- **Corpus Texts**: Refer to [`DATA_SOURCES.md`](DATA_SOURCES.md) for original source licenses.
+
+### Citation
+If you use this tokenizer, benchmark code, or findings in your academic research or applications, please cite:
+
+```bibtex
+@misc{nawaz2026sindhitokenizer,
+  author = {Nawaz, Ali and Fareed, Muqadas and Sadaf, Soonh},
+  title  = {A Corpus-Driven Sindhi Subword Tokenizer: Design, Empirical Validation, and Benchmark Comparison Against Fourteen Existing Tokenizers},
+  year   = {2026},
+  howpublished = {\url{https://github.com/alinawazcode/sindhi-tokenizer}},
+  note   = {Department of Computer Science, Shaikh Ayaz University, Shikarpur, Pakistan}
+}
+```
+
+---
+
+## Frequently Asked Questions (FAQ)
+
+<details>
+<summary><b>1. Why choose the 64k model over 32k or 150k?</b></summary>
+The 64,000 vocabulary size represents the empirical sweet spot between fertility efficiency (1.147 tokens/word) and embedding matrix parameter overhead. Scaling to 100k or 150k provides marginal fertility gains (1.133 and 1.125 respectively) but more than doubles the vocabulary table size, while the 32k model exhibits one minor loss on the standardized FLORES+ benchmark.
+</details>
+
+<details>
+<summary><b>2. What does the round-trip length check measure?</b></summary>
+It applies Unicode NFKC, removes zero-width formatting characters and whitespace, then compares normalized input and decoded-output lengths. This detects shortening but can miss substitutions or reorderings that preserve length; 0.0000% is not proof of exact text identity.
+</details>
+
+<details>
+<summary><b>3. Why is there a 15% mix of Urdu, Persian, and Arabic in the training corpus?</b></summary>
+Sindhi shares the Perso-Arabic alphabet and borrows significant religious, legal, and cultural vocabulary from Arabic and Persian, as well as shared regional terms with Urdu. This anchor (16.1% of characters, from 89,449 unique sentences repeated four times) was added by design to expose the subword vocabulary to shared Perso-Arabic root morphemes.
+</details>
+
+<details>
+<summary><b>4. Why should I avoid transformers.LlamaTokenizer for loading this model?</b></summary>
+Certain versions of Hugging Face Transformers' <code>LlamaTokenizer</code> fail to load custom SentencePiece Unigram models correctly, silently substituting an empty or 3-token fallback vocabulary without throwing an exception. Always use <code>sentencepiece.SentencePieceProcessor</code> directly or via our verified <code>SPMTokenizerAdapter</code>.
+</details>