uzbek-tokenizer 0.1.0__tar.gz

uzbek_tokenizer-0.1.0/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md
+include LICENSE
+recursive-include uzbek_tokenizer/data *.txt

uzbek_tokenizer-0.1.0/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: uzbek-tokenizer
+Version: 0.1.0
+Summary: A morphological tokenizer for Uzbek language
+Author-email: Ibrat Usmonov <usmonovibrat315@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/IbratDO/uzbek-tokenizer
+Project-URL: Documentation, https://github.com/IbratDO/uzbek-tokenizer#usage
+Project-URL: Repository, https://github.com/IbratDO/uzbek-tokenizer.git
+Project-URL: Issues, https://github.com/IbratDO/uzbek-tokenizer/issues
+Keywords: uzbek,tokenizer,morphology,nlp,segmentation
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: regex>=2022.0.0
+
+# Uzbek Tokenizer
+
+A Python library for morphological segmentation of Uzbek text. This tokenizer breaks down Uzbek words into their constituent morphemes (stems and affixes), which is useful for natural language processing tasks like machine learning, text analysis, and linguistic research.
+
+## Features
+
+- 🔤 **Morphological Segmentation**: Breaks Uzbek words into stems and affixes
+- 📝 **Text Normalization**: Handles Unicode, punctuation, and Uzbek-specific characters
+- ⚡ **Efficient**: Cached segmentation for repeated tokens
+- 📦 **Easy to Use**: Simple API for both single words and batch processing
+- 🎯 **Agglutinative Language Support**: Designed specifically for Uzbek's morphological structure
+
+## Installation
+
+Install via pip:
+
+```bash
+pip install uzbek-tokenizer
+```
+
+Or clone the repository:
+
+```bash
+git clone https://github.com/IbratDO/uzbek-tokenizer.git
+cd uzbek-tokenizer
+pip install -e .
+```
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from uzbek_tokenizer import apply_segmentation, normalize
+
+# Segment a word
+text = "kitoblarimizdan"
+result = apply_segmentation(text)
+print(result)
+# Output: ['kitob', 'lar', 'imiz', 'dan']
+
+# Segment a sentence
+sentence = "kitoblarimizdan o'qidim"
+result = apply_segmentation(sentence)
+print(result)
+# Output: ['kitob', 'lar', 'imiz', 'dan', "o'q", 'id', 'im']
+```
+
+### Normalization Only
+
+```python
+from uzbek_tokenizer import normalize
+
+text = "Salom, JAHON!"
+normalized = normalize(text)
+print(normalized)
+# Output: "salom , jahon !"
+```
+
+### Morphological Segmentation Only
+
+```python
+from uzbek_tokenizer import segment_morphological
+
+word = "o'qiyotgan"
+segments = segment_morphological(word)
+print(segments)
+# Output: ["o'q", 'i', 'yotgan']
+```
+
+## API Reference
+
+### `normalize(text: str) -> str`
+Normalizes Uzbek text by:
+- Converting to NFC Unicode form
+- Lowercasing
+- Standardizing punctuation
+- Handling Uzbek apostrophe variants
+- Collapsing whitespace
+
+**Parameters:**
+- `text` (str): Input text
+
+**Returns:** Normalized text
+
+### `segment_morphological(token: str) -> list[str]`
+Recursively segments a single token into morphemes.
+
+**Parameters:**
+- `token` (str): A single word/token
+
+**Returns:** List of morpheme components
+
+### `apply_segmentation(line: str) -> list[str]`
+Normalizes text and segments all tokens into morphemes.
+
+**Parameters:**
+- `line` (str): Input text (can be multiple words)
+
+**Returns:** List of all morphemes
+
+## How It Works
+
+The tokenizer uses a **greedy longest-match-first algorithm**:
+
+1. **Normalization**: Cleans and standardizes the input text
+2. **Prefix Stripping**: Removes longest matching prefixes (be-, no-)
+3. **Suffix Stripping**: Removes longest matching suffixes (47+ Uzbek suffixes)
+4. **Recursion**: Repeats until no more affixes can be removed
+5. **Output**: Returns the stem + all stripped affixes
+
+Example: `kitoblarimizdan`
+- Remove suffix "dan" → remainder: "kitoblarimiz"
+- Remove suffix "imiz" → remainder: "kitoblar"
+- Remove suffix "lar" → remainder: "kitob"
+- No more affixes → stem: "kitob"
+- Result: `[kitob, lar, imiz, dan]`
+
+## Supported Affixes
+
+### Prefixes (2)
+- be-, no-
+
+### Suffixes (47+)
+Includes: -cha, -lar, -dagi, -dan, -ga, -ni, -ning, -lik, -ish, and many more
+
+See `uzbek_tokenizer/data/prefixes.txt` and `uzbek_tokenizer/data/suffixes.txt` for the complete list.
+
+## Requirements
+
+- Python 3.8+
+- regex >= 2022.0.0
+
+## License
+
+MIT License - see LICENSE file for details
+
+## Contributing
+
+Contributions are welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+## Issues & Feedback
+
+Found a bug or have a suggestion? Please open an [issue on GitHub](https://github.com/IbratDO/uzbek-tokenizer/issues).
+
+## Citation
+
+If you use this tokenizer in your research, please cite:
+
+```bibtex
+@software{uzbek_tokenizer_2026,
+  author = {Ibrat Usmonov},
+  title = {Uzbek Tokenizer: Morphological Segmentation for Uzbek},
+  year = {2026},
+  url = {https://github.com/IbratDO/uzbek-tokenizer}
+}
+```
+
+## Roadmap
+
+- [ ] Vowel harmony validation
+- [ ] Stem lemmatization
+- [ ] Support for additional Uzbek dialects
+- [ ] Performance benchmarks
+- [ ] Web API
+
+## Acknowledgments
+
+Built for Uzbek language processing with love ❤️
+
+---
+
+**Status**: Early alpha (v0.1.0) - API may change. Contributions welcome!

uzbek_tokenizer-0.1.0/README.md

@@ -0,0 +1,179 @@
+# Uzbek Tokenizer
+
+A Python library for morphological segmentation of Uzbek text. This tokenizer breaks down Uzbek words into their constituent morphemes (stems and affixes), which is useful for natural language processing tasks like machine learning, text analysis, and linguistic research.
+
+## Features
+
+- 🔤 **Morphological Segmentation**: Breaks Uzbek words into stems and affixes
+- 📝 **Text Normalization**: Handles Unicode, punctuation, and Uzbek-specific characters
+- ⚡ **Efficient**: Cached segmentation for repeated tokens
+- 📦 **Easy to Use**: Simple API for both single words and batch processing
+- 🎯 **Agglutinative Language Support**: Designed specifically for Uzbek's morphological structure
+
+## Installation
+
+Install via pip:
+
+```bash
+pip install uzbek-tokenizer
+```
+
+Or clone the repository:
+
+```bash
+git clone https://github.com/IbratDO/uzbek-tokenizer.git
+cd uzbek-tokenizer
+pip install -e .
+```
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from uzbek_tokenizer import apply_segmentation, normalize
+
+# Segment a word
+text = "kitoblarimizdan"
+result = apply_segmentation(text)
+print(result)
+# Output: ['kitob', 'lar', 'imiz', 'dan']
+
+# Segment a sentence
+sentence = "kitoblarimizdan o'qidim"
+result = apply_segmentation(sentence)
+print(result)
+# Output: ['kitob', 'lar', 'imiz', 'dan', "o'q", 'id', 'im']
+```
+
+### Normalization Only
+
+```python
+from uzbek_tokenizer import normalize
+
+text = "Salom, JAHON!"
+normalized = normalize(text)
+print(normalized)
+# Output: "salom , jahon !"
+```
+
+### Morphological Segmentation Only
+
+```python
+from uzbek_tokenizer import segment_morphological
+
+word = "o'qiyotgan"
+segments = segment_morphological(word)
+print(segments)
+# Output: ["o'q", 'i', 'yotgan']
+```
+
+## API Reference
+
+### `normalize(text: str) -> str`
+Normalizes Uzbek text by:
+- Converting to NFC Unicode form
+- Lowercasing
+- Standardizing punctuation
+- Handling Uzbek apostrophe variants
+- Collapsing whitespace
+
+**Parameters:**
+- `text` (str): Input text
+
+**Returns:** Normalized text
+
+### `segment_morphological(token: str) -> list[str]`
+Recursively segments a single token into morphemes.
+
+**Parameters:**
+- `token` (str): A single word/token
+
+**Returns:** List of morpheme components
+
+### `apply_segmentation(line: str) -> list[str]`
+Normalizes text and segments all tokens into morphemes.
+
+**Parameters:**
+- `line` (str): Input text (can be multiple words)
+
+**Returns:** List of all morphemes
+
+## How It Works
+
+The tokenizer uses a **greedy longest-match-first algorithm**:
+
+1. **Normalization**: Cleans and standardizes the input text
+2. **Prefix Stripping**: Removes longest matching prefixes (be-, no-)
+3. **Suffix Stripping**: Removes longest matching suffixes (47+ Uzbek suffixes)
+4. **Recursion**: Repeats until no more affixes can be removed
+5. **Output**: Returns the stem + all stripped affixes
+
+Example: `kitoblarimizdan`
+- Remove suffix "dan" → remainder: "kitoblarimiz"
+- Remove suffix "imiz" → remainder: "kitoblar"
+- Remove suffix "lar" → remainder: "kitob"
+- No more affixes → stem: "kitob"
+- Result: `[kitob, lar, imiz, dan]`
+
+## Supported Affixes
+
+### Prefixes (2)
+- be-, no-
+
+### Suffixes (47+)
+Includes: -cha, -lar, -dagi, -dan, -ga, -ni, -ning, -lik, -ish, and many more
+
+See `uzbek_tokenizer/data/prefixes.txt` and `uzbek_tokenizer/data/suffixes.txt` for the complete list.
+
+## Requirements
+
+- Python 3.8+
+- regex >= 2022.0.0
+
+## License
+
+MIT License - see LICENSE file for details
+
+## Contributing
+
+Contributions are welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+## Issues & Feedback
+
+Found a bug or have a suggestion? Please open an [issue on GitHub](https://github.com/IbratDO/uzbek-tokenizer/issues).
+
+## Citation
+
+If you use this tokenizer in your research, please cite:
+
+```bibtex
+@software{uzbek_tokenizer_2026,
+  author = {Ibrat Usmonov},
+  title = {Uzbek Tokenizer: Morphological Segmentation for Uzbek},
+  year = {2026},
+  url = {https://github.com/IbratDO/uzbek-tokenizer}
+}
+```
+
+## Roadmap
+
+- [ ] Vowel harmony validation
+- [ ] Stem lemmatization
+- [ ] Support for additional Uzbek dialects
+- [ ] Performance benchmarks
+- [ ] Web API
+
+## Acknowledgments
+
+Built for Uzbek language processing with love ❤️
+
+---
+
+**Status**: Early alpha (v0.1.0) - API may change. Contributions welcome!

uzbek_tokenizer-0.1.0/pyproject.toml

@@ -0,0 +1,45 @@
+[build-system]
+requires = ["setuptools>=65.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "uzbek-tokenizer"
+version = "0.1.0"
+description = "A morphological tokenizer for Uzbek language"
+readme = "README.md"
+requires-python = ">=3.8"
+license = {text = "MIT"}
+authors = [
+    {name = "Ibrat Usmonov", email = "usmonovibrat315@gmail.com"}
+]
+keywords = ["uzbek", "tokenizer", "morphology", "nlp", "segmentation"]
+classifiers = [
+    "Development Status :: 3 - Alpha",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Topic :: Scientific/Engineering",
+    "Topic :: Text Processing :: Linguistic",
+]
+
+dependencies = [
+    "regex>=2022.0.0",
+]
+
+[project.urls]
+Homepage = "https://github.com/IbratDO/uzbek-tokenizer"
+Documentation = "https://github.com/IbratDO/uzbek-tokenizer#usage"
+Repository = "https://github.com/IbratDO/uzbek-tokenizer.git"
+Issues = "https://github.com/IbratDO/uzbek-tokenizer/issues"
+
+[tool.setuptools]
+packages = ["uzbek_tokenizer"]
+
+[tool.setuptools.package-data]
+uzbek_tokenizer = ["data/*.txt"]

uzbek_tokenizer-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

uzbek_tokenizer-0.1.0/tests/test_segmenter.py

@@ -0,0 +1,148 @@
+import unittest
+from uzbek_tokenizer import normalize, segment_morphological, apply_segmentation
+
+
+class TestNormalize(unittest.TestCase):
+    """Test cases for the normalize function."""
+    
+    def test_lowercase(self):
+        """Test that text is converted to lowercase."""
+        result = normalize("SALOM")
+        self.assertEqual(result, "salom")
+    
+    def test_unicode_normalization(self):
+        """Test Unicode NFC normalization."""
+        result = normalize("kafe")
+        self.assertIsInstance(result, str)
+    
+    def test_punctuation_separation(self):
+        """Test that punctuation is separated with spaces."""
+        result = normalize("Salom!")
+        self.assertIn("!", result)
+    
+    def test_apostrophe_handling(self):
+        """Test apostrophe normalization."""
+        # Various apostrophe types
+        result = normalize("o'q")
+        self.assertIn("ʼ", result)
+    
+    def test_whitespace_collapse(self):
+        """Test that multiple whitespaces are collapsed."""
+        result = normalize("salom    dunyo")
+        self.assertEqual(result, "salom dunyo")
+    
+    def test_ellipsis_handling(self):
+        """Test that ellipsis is normalized."""
+        result = normalize("salom...")
+        self.assertIn(".", result)
+
+
+class TestSegmentMorphological(unittest.TestCase):
+    """Test cases for the segment_morphological function."""
+    
+    def test_single_stem_no_affixes(self):
+        """Test word with no affixes."""
+        result = segment_morphological("kitob")
+        self.assertEqual(result, ["kitob"])
+    
+    def test_suffix_segmentation(self):
+        """Test basic suffix removal."""
+        result = segment_morphological("kitoblar")
+        self.assertIn("kitob", result)
+        self.assertIn("lar", result)
+    
+    def test_multiple_suffix_segmentation(self):
+        """Test multiple suffix removal."""
+        result = segment_morphological("kitoblarimiz")
+        self.assertIn("kitob", result)
+        self.assertTrue(any(morpheme in result for morpheme in ["lar", "lari"]))
+    
+    def test_prefix_segmentation(self):
+        """Test prefix removal."""
+        result = segment_morphological("bepartach")
+        self.assertEqual(result[0], "be")
+    
+    def test_single_character_token(self):
+        """Test single character returns as-is."""
+        result = segment_morphological("a")
+        self.assertEqual(result, ["a"])
+    
+    def test_empty_token_after_stripping(self):
+        """Test that minimum stem length is enforced."""
+        result = segment_morphological("dan")
+        self.assertEqual(result, ["dan"])
+
+
+class TestApplySegmentation(unittest.TestCase):
+    """Test cases for the apply_segmentation function."""
+    
+    def test_single_word(self):
+        """Test segmentation of single word."""
+        result = apply_segmentation("kitob")
+        self.assertEqual(result, ["kitob"])
+    
+    def test_single_word_with_affixes(self):
+        """Test segmentation of word with affixes."""
+        result = apply_segmentation("kitoblar")
+        self.assertIn("kitob", result)
+        self.assertIn("lar", result)
+    
+    def test_multiple_words(self):
+        """Test segmentation of multiple words."""
+        result = apply_segmentation("kitoblar o'qidim")
+        self.assertGreater(len(result), 2)
+        self.assertIn("kitob", result)
+        self.assertIn("lar", result)
+    
+    def test_text_with_punctuation(self):
+        """Test that punctuation is handled."""
+        result = apply_segmentation("Salom!")
+        self.assertIsInstance(result, list)
+        self.assertGreater(len(result), 0)
+    
+    def test_empty_string(self):
+        """Test handling of empty string."""
+        result = apply_segmentation("")
+        self.assertEqual(result, [])
+    
+    def test_whitespace_only(self):
+        """Test handling of whitespace-only string."""
+        result = apply_segmentation("   ")
+        self.assertEqual(result, [])
+    
+    def test_normalized_output(self):
+        """Test that output is normalized (lowercase)."""
+        result = apply_segmentation("KITOBLAR")
+        self.assertIn("kitob", result)
+
+
+class TestIntegration(unittest.TestCase):
+    """Integration tests for real-world Uzbek text."""
+    
+    def test_real_sentence(self):
+        """Test with real Uzbek sentence."""
+        text = "kitoblarimizdan o'qidim"
+        result = apply_segmentation(text)
+        
+        self.assertIsInstance(result, list)
+        self.assertGreater(len(result), 0)
+        
+        self.assertIn("kitob", result)
+        self.assertTrue(any(morpheme in result for morpheme in ["lar", "lari"]))
+    
+    def test_complex_word_segmentation(self):
+        """Test complex Uzbek word with multiple affixes."""
+        result = segment_morphological("borganmish")
+        self.assertIsInstance(result, list)
+        self.assertGreater(len(result), 1)
+    
+    def test_caching_consistency(self):
+        """Test that caching doesn't affect results."""
+        word = "kitoblar"
+        result1 = segment_morphological(word)
+        result2 = segment_morphological(word)
+        self.assertEqual(result1, result2)
+
+
+if __name__ == "__main__":
+    unittest.main()

uzbek_tokenizer-0.1.0/uzbek_tokenizer/__init__.py

@@ -0,0 +1,19 @@
+"""
+Uzbek Tokenizer - Morphological segmentation for Uzbek text
+"""
+
+from .segmenter import (
+    normalize,
+    segment_morphological,
+    apply_segmentation,
+)
+
+__version__ = "0.1.0"
+__author__ = "Ibrat Usmonov"
+__description__ = "O'zbek tili uchun morfologik segmentatsiya kutubxonasi"
+
+__all__ = [
+    "normalize",
+    "segment_morphological", 
+    "apply_segmentation",
+]

uzbek_tokenizer-0.1.0/uzbek_tokenizer/data/prefixes.txt

@@ -0,0 +1,2 @@
+be
+no

uzbek_tokenizer-0.1.0/uzbek_tokenizer/data/suffixes.txt

@@ -0,0 +1,77 @@
+cha
+choq
+chak
+jon
+xon
+loq
+bek
+lar
+niki
+dagi
+gacha
+dek
+day
+ish
+g'ish
+mtir
+roq
+ta
+tacha
+larcha
+ov
+ala
+ovlon
+tadan
+inchi
+g'i
+qa
+qil
+qi
+g'a
+sira
+gila
+nqira
+kila
+tir
+dir
+ar
+zor
+don
+giz
+sat
+kaz
+kiz
+qaz
+g'iz
+guncha
+may
+gach
+mayin
+gani
+gancha
+masdan
+gan
+jak
+yotgan
+vchi
+digan
+si
+miz
+ngiz
+lari
+ning
+ni
+ga
+dan
+da
+man
+san
+siz
+lik
+gin
+sin
+ka
+mi
+moq
+i
+ir

uzbek_tokenizer-0.1.0/uzbek_tokenizer/segmenter.py

@@ -0,0 +1,96 @@
+import os
+import unicodedata
+import regex as re
+from functools import lru_cache
+
+BASE_DIR = os.path.dirname(__file__)
+with open(os.path.join(BASE_DIR, "data/prefixes.txt"), encoding="utf-8") as f:
+    PREFIXES = sorted([line.strip() for line in f if line.strip()], key=len, reverse=True)
+
+with open(os.path.join(BASE_DIR, "data/suffixes.txt"), encoding="utf-8") as f:
+    SUFFIXES = sorted([line.strip() for line in f if line.strip()], key=len, reverse=True)
+
+
+def normalize(text: str) -> str:
+    """
+    Applies normalization rules to Uzbek text.
+    """
+    # 1) Unicode normalize
+    text = unicodedata.normalize("NFC", text)
+    # 2) Lowercase
+    text = text.lower()
+    # 3) Punctuation normalization
+    text = re.sub(r"[«»]", '"', text)
+    text = re.sub(r"[–—]", "-", text)
+    text = re.sub(r'([.!?;:,(){}[\]"`~@#$%^&*+=|\\/<>\-])', r' \1 ', text)
+    
+    # 4) Special handling for Uzbek apostrophes before O/G
+    text = re.sub(r"([OoGg])[\'ʻ''`ʼ]", r"\1ʻ", text)
+    
+    # 5) Normalize all other apostrophe variants to ʼ (U+02BC)
+    text = re.sub(r"[\'ʻ''`]", "ʼ", text)
+    
+    # 6) Handle special punctuation sequences
+    text = re.sub(r'\.{3,}', ' ... ', text)  # Ellipsis
+    text = re.sub(r'[!]{2,}', ' !! ', text)   # Multiple exclamations
+    text = re.sub(r'[?]{2,}', ' ?? ', text)   # Multiple questions
+    
+    # 7) Collapse whitespace
+    text = re.sub(r"\s+", " ", text).strip()
+    return text
+
+
+@lru_cache(maxsize=None)
+def segment_morphological(token: str) -> list[str]:
+    """
+    Recursively strip the longest matching prefix or suffix from `token`.
+    Returns a list of stems/affixes in order.
+    """
+    if len(token) <= 1:
+        return [token]
+    
+    # 2) Try stripping a prefix
+    for p in PREFIXES:
+        if token.startswith(p) and len(token) > len(p) + 2:
+            remainder = token[len(p):]
+            return [p] + segment_morphological(remainder)
+
+    # 3) Try stripping a suffix
+    for s in SUFFIXES:
+        if token.endswith(s) and len(token) > len(s) + 2:
+            remainder = token[:-len(s)]
+            return segment_morphological(remainder) + [s]
+
+    # 4) No more affixes found: return the token itself
+    return [token]
+
+
+def apply_segmentation(line: str) -> list[str]:
+    """
+    Takes a single line of whitespace-tokenized text (e.g. "kitoblarimizdan o'qidim")
+    and returns a list where each token is replaced by its
+    affix-segmented pieces, e.g.
+      ["kitob", "lar", "imiz", "dan", "o'q", "id", "im"]
+    """
+    pieces = []
+    norm = normalize(line)
+
+    for tok in norm.split(" "):
+        if tok:  # Skip empty tokens
+            pieces.extend(segment_morphological(tok))
+    return pieces
+
+
+# Example usage
+if __name__ == "__main__":
+    examples = [
+        "kitoblarimizdan o'qidim",
+        "yig'layvergan edilar",
+        "qayta ishlov berish uchun",
+        "borganmish"
+    ]
+    for ex in examples:
+        print(f"\nOriginal:  {ex}")
+        segmented = apply_segmentation(ex)
+        print(f"Segmented: {segmented}")
+        print(f"As string: {' '.join(segmented)}")

uzbek_tokenizer-0.1.0/uzbek_tokenizer.egg-info/PKG-INFO

@@ -0,0 +1,206 @@
+Metadata-Version: 2.4
+Name: uzbek-tokenizer
+Version: 0.1.0
+Summary: A morphological tokenizer for Uzbek language
+Author-email: Ibrat Usmonov <usmonovibrat315@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/IbratDO/uzbek-tokenizer
+Project-URL: Documentation, https://github.com/IbratDO/uzbek-tokenizer#usage
+Project-URL: Repository, https://github.com/IbratDO/uzbek-tokenizer.git
+Project-URL: Issues, https://github.com/IbratDO/uzbek-tokenizer/issues
+Keywords: uzbek,tokenizer,morphology,nlp,segmentation
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+Requires-Dist: regex>=2022.0.0
+
+# Uzbek Tokenizer
+
+A Python library for morphological segmentation of Uzbek text. This tokenizer breaks down Uzbek words into their constituent morphemes (stems and affixes), which is useful for natural language processing tasks like machine learning, text analysis, and linguistic research.
+
+## Features
+
+- 🔤 **Morphological Segmentation**: Breaks Uzbek words into stems and affixes
+- 📝 **Text Normalization**: Handles Unicode, punctuation, and Uzbek-specific characters
+- ⚡ **Efficient**: Cached segmentation for repeated tokens
+- 📦 **Easy to Use**: Simple API for both single words and batch processing
+- 🎯 **Agglutinative Language Support**: Designed specifically for Uzbek's morphological structure
+
+## Installation
+
+Install via pip:
+
+```bash
+pip install uzbek-tokenizer
+```
+
+Or clone the repository:
+
+```bash
+git clone https://github.com/IbratDO/uzbek-tokenizer.git
+cd uzbek-tokenizer
+pip install -e .
+```
+
+## Quick Start
+
+### Basic Usage
+
+```python
+from uzbek_tokenizer import apply_segmentation, normalize
+
+# Segment a word
+text = "kitoblarimizdan"
+result = apply_segmentation(text)
+print(result)
+# Output: ['kitob', 'lar', 'imiz', 'dan']
+
+# Segment a sentence
+sentence = "kitoblarimizdan o'qidim"
+result = apply_segmentation(sentence)
+print(result)
+# Output: ['kitob', 'lar', 'imiz', 'dan', "o'q", 'id', 'im']
+```
+
+### Normalization Only
+
+```python
+from uzbek_tokenizer import normalize
+
+text = "Salom, JAHON!"
+normalized = normalize(text)
+print(normalized)
+# Output: "salom , jahon !"
+```
+
+### Morphological Segmentation Only
+
+```python
+from uzbek_tokenizer import segment_morphological
+
+word = "o'qiyotgan"
+segments = segment_morphological(word)
+print(segments)
+# Output: ["o'q", 'i', 'yotgan']
+```
+
+## API Reference
+
+### `normalize(text: str) -> str`
+Normalizes Uzbek text by:
+- Converting to NFC Unicode form
+- Lowercasing
+- Standardizing punctuation
+- Handling Uzbek apostrophe variants
+- Collapsing whitespace
+
+**Parameters:**
+- `text` (str): Input text
+
+**Returns:** Normalized text
+
+### `segment_morphological(token: str) -> list[str]`
+Recursively segments a single token into morphemes.
+
+**Parameters:**
+- `token` (str): A single word/token
+
+**Returns:** List of morpheme components
+
+### `apply_segmentation(line: str) -> list[str]`
+Normalizes text and segments all tokens into morphemes.
+
+**Parameters:**
+- `line` (str): Input text (can be multiple words)
+
+**Returns:** List of all morphemes
+
+## How It Works
+
+The tokenizer uses a **greedy longest-match-first algorithm**:
+
+1. **Normalization**: Cleans and standardizes the input text
+2. **Prefix Stripping**: Removes longest matching prefixes (be-, no-)
+3. **Suffix Stripping**: Removes longest matching suffixes (47+ Uzbek suffixes)
+4. **Recursion**: Repeats until no more affixes can be removed
+5. **Output**: Returns the stem + all stripped affixes
+
+Example: `kitoblarimizdan`
+- Remove suffix "dan" → remainder: "kitoblarimiz"
+- Remove suffix "imiz" → remainder: "kitoblar"
+- Remove suffix "lar" → remainder: "kitob"
+- No more affixes → stem: "kitob"
+- Result: `[kitob, lar, imiz, dan]`
+
+## Supported Affixes
+
+### Prefixes (2)
+- be-, no-
+
+### Suffixes (47+)
+Includes: -cha, -lar, -dagi, -dan, -ga, -ni, -ning, -lik, -ish, and many more
+
+See `uzbek_tokenizer/data/prefixes.txt` and `uzbek_tokenizer/data/suffixes.txt` for the complete list.
+
+## Requirements
+
+- Python 3.8+
+- regex >= 2022.0.0
+
+## License
+
+MIT License - see LICENSE file for details
+
+## Contributing
+
+Contributions are welcome! Please:
+
+1. Fork the repository
+2. Create a feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -m 'Add amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Open a Pull Request
+
+## Issues & Feedback
+
+Found a bug or have a suggestion? Please open an [issue on GitHub](https://github.com/IbratDO/uzbek-tokenizer/issues).
+
+## Citation
+
+If you use this tokenizer in your research, please cite:
+
+```bibtex
+@software{uzbek_tokenizer_2026,
+  author = {Ibrat Usmonov},
+  title = {Uzbek Tokenizer: Morphological Segmentation for Uzbek},
+  year = {2026},
+  url = {https://github.com/IbratDO/uzbek-tokenizer}
+}
+```
+
+## Roadmap
+
+- [ ] Vowel harmony validation
+- [ ] Stem lemmatization
+- [ ] Support for additional Uzbek dialects
+- [ ] Performance benchmarks
+- [ ] Web API
+
+## Acknowledgments
+
+Built for Uzbek language processing with love ❤️
+
+---
+
+**Status**: Early alpha (v0.1.0) - API may change. Contributions welcome!

uzbek_tokenizer-0.1.0/uzbek_tokenizer.egg-info/SOURCES.txt

@@ -0,0 +1,13 @@
+MANIFEST.in
+README.md
+pyproject.toml
+tests/test_segmenter.py
+uzbek_tokenizer/__init__.py
+uzbek_tokenizer/segmenter.py
+uzbek_tokenizer.egg-info/PKG-INFO
+uzbek_tokenizer.egg-info/SOURCES.txt
+uzbek_tokenizer.egg-info/dependency_links.txt
+uzbek_tokenizer.egg-info/requires.txt
+uzbek_tokenizer.egg-info/top_level.txt
+uzbek_tokenizer/data/prefixes.txt
+uzbek_tokenizer/data/suffixes.txt

uzbek_tokenizer-0.1.0/uzbek_tokenizer.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

uzbek_tokenizer-0.1.0/uzbek_tokenizer.egg-info/requires.txt

@@ -0,0 +1 @@
+regex>=2022.0.0

uzbek_tokenizer-0.1.0/uzbek_tokenizer.egg-info/top_level.txt

@@ -0,0 +1 @@
+uzbek_tokenizer