kannada-tokenizer 0.1.0__tar.gz

kannada_tokenizer-0.1.0/PKG-INFO

@@ -0,0 +1,105 @@
+Metadata-Version: 2.4
+Name: kannada-tokenizer
+Version: 0.1.0
+Summary: Kannada tokenizer with sandhi splitting for Information Retrieval.
+Author: Hemanth HM
+License-Expression: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+
+# kannada-tokenizer
+
+Tokenize Kannada text with sandhi splitting for Information Retrieval.
+
+```bash
+pip install kannada-tokenizer
+```
+
+## Quick start
+
+```python
+from kannada_tokenizer import tokenize
+
+tokenize("ಕನ್ನಡ ಧರ್ಮ")
+# ['kannaḍa', 'dharma']
+
+tokenize("vidyālaya")
+# ['vidyālaya']
+
+tokenize("mahōtsava")
+# ['maha', 'ōtsava']
+```
+
+`tokenize()` normalizes to ISO 15919, splits on whitespace and punctuation, then applies reverse sandhi rules. Accepts both Kannada script and romanized input.
+
+## Sandhi splitting
+
+```python
+from kannada_tokenizer.sandhi import split_sandhi
+
+split_sandhi("rāmāyana")   # lōpa sandhi: ā → a + ā
+# ['rāma', 'āyana']
+
+split_sandhi("kannaḍa")    # no junction found
+# ['kannaḍa']
+```
+
+Rule-based engine covering lōpa sandhi (vowel elision), āgama sandhi (y/v insertion), ādeśa sandhi (guṇa-like substitution), and consonant sandhi (voicing, gemination, nasals). Uses longest-match heuristic when splits are ambiguous.
+
+## Transliteration
+
+```python
+from kannada_tokenizer.transliterate import (
+    kannada_to_iso15919,
+    iso15919_to_kannada,
+    is_kannada,
+)
+
+kannada_to_iso15919("ಬೆಂಗಳೂರು")
+# 'beṃgaḻūru'
+
+iso15919_to_kannada("kannaḍa")
+# 'ಕನ್ನಡ'
+
+is_kannada("ಕನ್ನಡ")
+# True
+```
+
+Handles Kannada's short/long e/o distinction (ಎ→e vs ಏ→ē) and the retroflex lateral ಳ→ḻ unique to Dravidian.
+
+## Word-level tokenization
+
+```python
+from kannada_tokenizer.tokenizer import tokenize_words
+
+tokenize_words("vidyālaya namaḥ")
+# ['vidyālaya', 'namaḥ']
+```
+
+`tokenize_words()` splits on whitespace and punctuation only — no sandhi splitting.
+
+## CLI
+
+```bash
+kannada-tokenize "ಕನ್ನಡ ಧರ್ಮ"
+# kannaḍa
+# dharma
+
+echo "vidyālaya" | kannada-tokenize
+# vidyālaya
+
+kannada-tokenize --no-sandhi "mahōtsava"
+# mahōtsava
+
+kannada-tokenize -s " " "dharma yōga"
+# dharma yōga
+```
+
+- `--no-sandhi` — word-level only, skip sandhi splitting
+- `--separator SEP` — output separator (default: newline)
+
+## License
+
+MIT © [Hemanth.HM](https://h3manth.com)

kannada_tokenizer-0.1.0/README.md

@@ -0,0 +1,94 @@
+# kannada-tokenizer
+
+Tokenize Kannada text with sandhi splitting for Information Retrieval.
+
+```bash
+pip install kannada-tokenizer
+```
+
+## Quick start
+
+```python
+from kannada_tokenizer import tokenize
+
+tokenize("ಕನ್ನಡ ಧರ್ಮ")
+# ['kannaḍa', 'dharma']
+
+tokenize("vidyālaya")
+# ['vidyālaya']
+
+tokenize("mahōtsava")
+# ['maha', 'ōtsava']
+```
+
+`tokenize()` normalizes to ISO 15919, splits on whitespace and punctuation, then applies reverse sandhi rules. Accepts both Kannada script and romanized input.
+
+## Sandhi splitting
+
+```python
+from kannada_tokenizer.sandhi import split_sandhi
+
+split_sandhi("rāmāyana")   # lōpa sandhi: ā → a + ā
+# ['rāma', 'āyana']
+
+split_sandhi("kannaḍa")    # no junction found
+# ['kannaḍa']
+```
+
+Rule-based engine covering lōpa sandhi (vowel elision), āgama sandhi (y/v insertion), ādeśa sandhi (guṇa-like substitution), and consonant sandhi (voicing, gemination, nasals). Uses longest-match heuristic when splits are ambiguous.
+
+## Transliteration
+
+```python
+from kannada_tokenizer.transliterate import (
+    kannada_to_iso15919,
+    iso15919_to_kannada,
+    is_kannada,
+)
+
+kannada_to_iso15919("ಬೆಂಗಳೂರು")
+# 'beṃgaḻūru'
+
+iso15919_to_kannada("kannaḍa")
+# 'ಕನ್ನಡ'
+
+is_kannada("ಕನ್ನಡ")
+# True
+```
+
+Handles Kannada's short/long e/o distinction (ಎ→e vs ಏ→ē) and the retroflex lateral ಳ→ḻ unique to Dravidian.
+
+## Word-level tokenization
+
+```python
+from kannada_tokenizer.tokenizer import tokenize_words
+
+tokenize_words("vidyālaya namaḥ")
+# ['vidyālaya', 'namaḥ']
+```
+
+`tokenize_words()` splits on whitespace and punctuation only — no sandhi splitting.
+
+## CLI
+
+```bash
+kannada-tokenize "ಕನ್ನಡ ಧರ್ಮ"
+# kannaḍa
+# dharma
+
+echo "vidyālaya" | kannada-tokenize
+# vidyālaya
+
+kannada-tokenize --no-sandhi "mahōtsava"
+# mahōtsava
+
+kannada-tokenize -s " " "dharma yōga"
+# dharma yōga
+```
+
+- `--no-sandhi` — word-level only, skip sandhi splitting
+- `--separator SEP` — output separator (default: newline)
+
+## License
+
+MIT © [Hemanth.HM](https://h3manth.com)

kannada_tokenizer-0.1.0/pyproject.toml

@@ -0,0 +1,22 @@
+[build-system]
+requires = ["setuptools>=68.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "kannada-tokenizer"
+version = "0.1.0"
+description = "Kannada tokenizer with sandhi splitting for Information Retrieval."
+readme = "README.md"
+requires-python = ">=3.9"
+license = "MIT"
+authors = [{name = "Hemanth HM"}]
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "Topic :: Text Processing :: Linguistic",
+]
+
+[project.scripts]
+kannada-tokenize = "kannada_tokenizer.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

kannada_tokenizer-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

kannada_tokenizer-0.1.0/src/kannada_tokenizer/__init__.py

@@ -0,0 +1,6 @@
+"""Kannada tokenizer with sandhi splitting for Information Retrieval."""
+
+from kannada_tokenizer.tokenizer import tokenize
+
+__version__ = "0.1.0"
+__all__ = ["tokenize"]

kannada_tokenizer-0.1.0/src/kannada_tokenizer/cli.py

@@ -0,0 +1,65 @@
+"""Command-line interface for the Kannada tokenizer."""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+from .tokenizer import tokenize, tokenize_words
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    """Build and return the argument parser."""
+    parser = argparse.ArgumentParser(
+        prog="kannada-tokenize",
+        description="Tokenize Kannada text with optional sandhi splitting.",
+    )
+    parser.add_argument(
+        "text",
+        nargs="?",
+        default=None,
+        help="Kannada text to tokenize. Reads from stdin if omitted.",
+    )
+    parser.add_argument(
+        "--no-sandhi",
+        action="store_true",
+        default=False,
+        help="Disable sandhi splitting (word-level tokenization only).",
+    )
+    parser.add_argument(
+        "--separator",
+        "-s",
+        default="\n",
+        help="Token separator in output (default: newline).",
+    )
+    return parser
+
+
+def main(argv: list[str] | None = None) -> None:
+    """Entry point for the ``kannada-tokenize`` CLI.
+
+    Args:
+        argv: Optional argument list (defaults to ``sys.argv[1:]``).
+    """
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+
+    # Determine input source
+    if args.text is not None:
+        text: str = args.text
+    elif not sys.stdin.isatty():
+        text = sys.stdin.read()
+    else:
+        parser.print_help()
+        sys.exit(1)
+
+    # Select tokenization strategy
+    tok_fn = tokenize_words if args.no_sandhi else tokenize
+    tokens: list[str] = tok_fn(text)
+
+    if tokens:
+        print(args.separator.join(tokens))
+
+
+if __name__ == "__main__":
+    main()