vettu 1.0.0__tar.gz

vettu-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 Tamil NLP Project
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

vettu-1.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE
+include README.md
+include pyproject.toml
+recursive-include tamil_tokenizer/data *.list

vettu-1.0.0/PKG-INFO

@@ -0,0 +1,159 @@
+Metadata-Version: 2.4
+Name: vettu
+Version: 1.0.0
+Summary: Multi-level tokenizer for Tamil text — sentence, word, character, and morpheme tokenization
+Home-page: https://github.com/tamil-phy/tamil_tokenizer
+Author: Tamil NLP Project
+License: MIT
+Project-URL: Homepage, https://github.com/tamil-phy/tamil_tokenizer
+Project-URL: Repository, https://github.com/tamil-phy/tamil_tokenizer
+Project-URL: Issues, https://github.com/tamil-phy/tamil_tokenizer/issues
+Keywords: tamil,tokenizer,nlp,morpheme,tamil-nlp,text-processing
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Natural Language :: Tamil
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: hf
+Requires-Dist: transformers>=4.20.0; extra == "hf"
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# Tamil Tokenizer
+
+A standalone, multi-level tokenizer for Tamil text. No external dependencies — uses only the Python standard library.
+
+## Features
+
+Four levels of tokenization:
+
+| Level | Description | Example |
+|-------|-------------|---------|
+| **sentence** | Split text into sentences | `"அவன் வந்தான். அவள் பார்த்தாள்."` → 2 sentences |
+| **word** | Split into words + punctuation | `"அவன் வந்தான்."` → `அவன்`, `வந்தான்`, `.` |
+| **character** | Split into Tamil letters with classification (உயிர்/மெய்/உயிர்மெய், வல்லினம்/மெல்லினம்/இடையினம்) | `"வந்தான்"` → `வ`, `ந்`, `தா`, `ன்` |
+| **morpheme** | Split into root + grammatical suffixes (case, tense, person) | `"பள்ளிக்கு"` → root `பள்ளி` + case suffix `க்கு` (Dative) |
+
+## Installation
+
+```bash
+# From the project directory
+pip install -e .
+
+# Or just use directly (no install needed)
+python -m tamil_tokenizer "அவன் வந்தான்."
+```
+
+## Usage
+
+### Command Line
+
+```bash
+# Word tokenization (default)
+python -m tamil_tokenizer "அவன் வந்தான்."
+
+# Character tokenization
+python -m tamil_tokenizer "தமிழ்நாடு" --level character
+
+# Sentence tokenization
+python -m tamil_tokenizer "அவன் வந்தான். அவள் பார்த்தாள்." --level sentence
+
+# Morpheme tokenization
+python -m tamil_tokenizer "பள்ளிக்கு சென்றான்." --level morpheme
+
+# JSON output
+python -m tamil_tokenizer "அவன் வந்தான்." --format json
+
+# Plain text output (just token strings)
+python -m tamil_tokenizer "அவன் வந்தான்." --format text
+
+# Interactive mode
+python -m tamil_tokenizer --interactive
+```
+
+### Python API
+
+```python
+from tamil_tokenizer import TamilTokenizer, Token, TokenType
+
+tokenizer = TamilTokenizer()
+
+# Sentence tokenization
+sentences = tokenizer.sentence_tokenize("அவன் வந்தான். அவள் பார்த்தாள்.")
+
+# Word tokenization
+words = tokenizer.word_tokenize("அவன் வந்தான்.")
+
+# Character tokenization
+letters = tokenizer.character_tokenize("வந்தான்")
+for letter in letters:
+    print(f"{letter.text} -> {letter.token_type.value} ({letter.metadata})")
+
+# Morpheme tokenization
+morphemes = tokenizer.morpheme_tokenize("பள்ளிக்கு")
+for m in morphemes:
+    print(f"{m.text} -> {m.token_type.value} ({m.metadata})")
+
+# Unified pipeline
+tokens = tokenizer.tokenize("அவன் வந்தான்.", level="word")
+
+# Convenience: get just strings
+strings = tokenizer.tokenize_to_strings("அவன் வந்தான்.", level="word")
+# ['அவன்', 'வந்தான்', '.']
+
+# Convenience: get dicts (useful for JSON serialization)
+dicts = tokenizer.tokenize_to_dicts("அவன் வந்தான்.", level="character")
+```
+
+## Token Types
+
+### Word-level
+- `word` — Tamil word
+- `number` — Numeric value
+- `punctuation` — Punctuation mark
+- `symbol` — Other symbol
+
+### Character-level
+- `vowel` — உயிரெழுத்து (அ, ஆ, இ, ...)
+- `consonant` — மெய்யெழுத்து (க், ங், ச், ...)
+- `vowel_consonant` — உயிர்மெய்யெழுத்து (க, கா, கி, ...)
+- `special` — ஆய்த எழுத்து (ஃ)
+
+### Morpheme-level
+- `root` — Root word
+- `suffix` — Generic suffix
+- `case_suffix` — வேற்றுமை உருபு (case marker)
+- `tense_marker` — கால இடைநிலை (tense marker)
+- `person_marker` — விகுதி (person/number marker)
+
+## Project Structure
+
+```
+tamil_tokenizer/
+├── __init__.py          # Package init + public API
+├── __main__.py          # CLI entry point
+├── tokenizer.py         # Main TamilTokenizer class
+├── constants/           # Tamil Unicode constants & letter groups
+├── grammar/             # Grammar analysis (util, case, tense)
+├── config/              # Configuration & data file loading
+├── parsers/             # Root word parser & core parsing
+├── utils/               # Iterator, splitting, word class utilities
+└── data/                # Grammar rule files (.list)
+```
+
+## Requirements
+
+- Python 3.8+
+- No external dependencies

vettu-1.0.0/README.md

@@ -0,0 +1,126 @@
+# Tamil Tokenizer
+
+A standalone, multi-level tokenizer for Tamil text. No external dependencies — uses only the Python standard library.
+
+## Features
+
+Four levels of tokenization:
+
+| Level | Description | Example |
+|-------|-------------|---------|
+| **sentence** | Split text into sentences | `"அவன் வந்தான். அவள் பார்த்தாள்."` → 2 sentences |
+| **word** | Split into words + punctuation | `"அவன் வந்தான்."` → `அவன்`, `வந்தான்`, `.` |
+| **character** | Split into Tamil letters with classification (உயிர்/மெய்/உயிர்மெய், வல்லினம்/மெல்லினம்/இடையினம்) | `"வந்தான்"` → `வ`, `ந்`, `தா`, `ன்` |
+| **morpheme** | Split into root + grammatical suffixes (case, tense, person) | `"பள்ளிக்கு"` → root `பள்ளி` + case suffix `க்கு` (Dative) |
+
+## Installation
+
+```bash
+# From the project directory
+pip install -e .
+
+# Or just use directly (no install needed)
+python -m tamil_tokenizer "அவன் வந்தான்."
+```
+
+## Usage
+
+### Command Line
+
+```bash
+# Word tokenization (default)
+python -m tamil_tokenizer "அவன் வந்தான்."
+
+# Character tokenization
+python -m tamil_tokenizer "தமிழ்நாடு" --level character
+
+# Sentence tokenization
+python -m tamil_tokenizer "அவன் வந்தான். அவள் பார்த்தாள்." --level sentence
+
+# Morpheme tokenization
+python -m tamil_tokenizer "பள்ளிக்கு சென்றான்." --level morpheme
+
+# JSON output
+python -m tamil_tokenizer "அவன் வந்தான்." --format json
+
+# Plain text output (just token strings)
+python -m tamil_tokenizer "அவன் வந்தான்." --format text
+
+# Interactive mode
+python -m tamil_tokenizer --interactive
+```
+
+### Python API
+
+```python
+from tamil_tokenizer import TamilTokenizer, Token, TokenType
+
+tokenizer = TamilTokenizer()
+
+# Sentence tokenization
+sentences = tokenizer.sentence_tokenize("அவன் வந்தான். அவள் பார்த்தாள்.")
+
+# Word tokenization
+words = tokenizer.word_tokenize("அவன் வந்தான்.")
+
+# Character tokenization
+letters = tokenizer.character_tokenize("வந்தான்")
+for letter in letters:
+    print(f"{letter.text} -> {letter.token_type.value} ({letter.metadata})")
+
+# Morpheme tokenization
+morphemes = tokenizer.morpheme_tokenize("பள்ளிக்கு")
+for m in morphemes:
+    print(f"{m.text} -> {m.token_type.value} ({m.metadata})")
+
+# Unified pipeline
+tokens = tokenizer.tokenize("அவன் வந்தான்.", level="word")
+
+# Convenience: get just strings
+strings = tokenizer.tokenize_to_strings("அவன் வந்தான்.", level="word")
+# ['அவன்', 'வந்தான்', '.']
+
+# Convenience: get dicts (useful for JSON serialization)
+dicts = tokenizer.tokenize_to_dicts("அவன் வந்தான்.", level="character")
+```
+
+## Token Types
+
+### Word-level
+- `word` — Tamil word
+- `number` — Numeric value
+- `punctuation` — Punctuation mark
+- `symbol` — Other symbol
+
+### Character-level
+- `vowel` — உயிரெழுத்து (அ, ஆ, இ, ...)
+- `consonant` — மெய்யெழுத்து (க், ங், ச், ...)
+- `vowel_consonant` — உயிர்மெய்யெழுத்து (க, கா, கி, ...)
+- `special` — ஆய்த எழுத்து (ஃ)
+
+### Morpheme-level
+- `root` — Root word
+- `suffix` — Generic suffix
+- `case_suffix` — வேற்றுமை உருபு (case marker)
+- `tense_marker` — கால இடைநிலை (tense marker)
+- `person_marker` — விகுதி (person/number marker)
+
+## Project Structure
+
+```
+tamil_tokenizer/
+├── __init__.py          # Package init + public API
+├── __main__.py          # CLI entry point
+├── tokenizer.py         # Main TamilTokenizer class
+├── constants/           # Tamil Unicode constants & letter groups
+├── grammar/             # Grammar analysis (util, case, tense)
+├── config/              # Configuration & data file loading
+├── parsers/             # Root word parser & core parsing
+├── utils/               # Iterator, splitting, word class utilities
+└── data/                # Grammar rule files (.list)
+```
+
+## Requirements
+
+- Python 3.8+
+- No external dependencies

vettu-1.0.0/pyproject.toml

@@ -0,0 +1,47 @@
+[build-system]
+requires = ["setuptools>=61.0", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "vettu"
+version = "1.0.0"
+description = "Multi-level tokenizer for Tamil text — sentence, word, character, and morpheme tokenization"
+readme = "README.md"
+license = {text = "MIT"}
+requires-python = ">=3.8"
+authors = [
+    {name = "Tamil NLP Project"},
+]
+keywords = ["tamil", "tokenizer", "nlp", "morpheme", "tamil-nlp", "text-processing"]
+classifiers = [
+    "Development Status :: 4 - Beta",
+    "Intended Audience :: Developers",
+    "Intended Audience :: Science/Research",
+    "License :: OSI Approved :: MIT License",
+    "Topic :: Text Processing :: Linguistic",
+    "Programming Language :: Python :: 3",
+    "Programming Language :: Python :: 3.8",
+    "Programming Language :: Python :: 3.9",
+    "Programming Language :: Python :: 3.10",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Natural Language :: Tamil",
+    "Operating System :: OS Independent",
+]
+
+[project.optional-dependencies]
+hf = ["transformers>=4.20.0"]
+
+[project.scripts]
+tamil-tokenize = "tamil_tokenizer.__main__:main"
+
+[project.urls]
+Homepage = "https://github.com/tamil-phy/tamil_tokenizer"
+Repository = "https://github.com/tamil-phy/tamil_tokenizer"
+Issues = "https://github.com/tamil-phy/tamil_tokenizer/issues"
+
+[tool.setuptools.packages.find]
+include = ["tamil_tokenizer*"]
+
+[tool.setuptools.package-data]
+tamil_tokenizer = ["data/*.list"]

vettu-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

vettu-1.0.0/setup.py

@@ -0,0 +1,39 @@
+"""Setup script for Tamil Tokenizer."""
+
+from setuptools import setup, find_packages
+
+setup(
+    name="vettu",
+    version="1.0.0",
+    description="Multi-level tokenizer for Tamil text",
+    long_description=open("README.md", encoding="utf-8").read(),
+    long_description_content_type="text/markdown",
+    author="Tamil NLP Project",
+    license="MIT",
+    url="https://github.com/tamil-phy/tamil_tokenizer",
+    python_requires=">=3.8",
+    packages=find_packages(),
+    package_data={
+        "tamil_tokenizer": ["data/*.list"],
+    },
+    include_package_data=True,
+    entry_points={
+        "console_scripts": [
+            "tamil-tokenize=tamil_tokenizer.__main__:main",
+        ],
+    },
+    classifiers=[
+        "Development Status :: 4 - Beta",
+        "Intended Audience :: Developers",
+        "Intended Audience :: Science/Research",
+        "Topic :: Text Processing :: Linguistic",
+        "Programming Language :: Python :: 3",
+        "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
+        "Programming Language :: Python :: 3.12",
+        "Natural Language :: Tamil",
+        "Operating System :: OS Independent",
+    ],
+)

vettu-1.0.0/tamil_tokenizer/__init__.py

@@ -0,0 +1,30 @@
+"""
+Tamil Tokenizer - Standalone multi-level tokenizer for Tamil text.
+
+Provides four levels of tokenization:
+- Sentence tokenization
+- Word tokenization
+- Character (letter) tokenization
+- Morpheme tokenization (root word + suffixes)
+
+Usage:
+    from tamil_tokenizer import TamilTokenizer, Token, TokenType
+
+    tokenizer = TamilTokenizer()
+    tokens = tokenizer.tokenize("அவன் வந்தான்.", level="word")
+"""
+
+__version__ = "1.0.0"
+
+from .tokenizer import TamilTokenizer, Token, TokenType
+
+def _import_hf_tokenizer():
+    """Lazy import to avoid hard dependency on transformers."""
+    from .hf_tokenizer import TamilHFTokenizer
+    return TamilHFTokenizer
+
+try:
+    from .hf_tokenizer import TamilHFTokenizer
+    __all__ = ['TamilTokenizer', 'TamilHFTokenizer', 'Token', 'TokenType']
+except ImportError:
+    __all__ = ['TamilTokenizer', 'Token', 'TokenType']

vettu-1.0.0/tamil_tokenizer/__main__.py

@@ -0,0 +1,124 @@
+"""
+Tamil Tokenizer CLI - Command-line interface for Tamil tokenization.
+
+Usage:
+    python -m tamil_tokenizer "அவன் வந்தான்."
+    python -m tamil_tokenizer "அவன் வந்தான்." --level character
+    python -m tamil_tokenizer "அவன் வந்தான். அவள் பார்த்தாள்." --level sentence
+    python -m tamil_tokenizer "பள்ளிக்கு சென்றான்." --level morpheme
+    python -m tamil_tokenizer --interactive
+"""
+
+import argparse
+import sys
+from typing import Optional
+
+from .tokenizer import TamilTokenizer, Token
+
+
+def print_tokens(tokens: list, level: str, text: str) -> None:
+    """Print tokens in a formatted table."""
+    print(f"\n{'='*60}")
+    print(f"Tokenization Level: {level}")
+    print(f"Input: {text}")
+    print(f"{'='*60}")
+    print(f"Tokens ({len(tokens)}):")
+
+    for i, token in enumerate(tokens):
+        meta_str = ""
+        if token.metadata:
+            meta_parts = [f"{k}={v}" for k, v in token.metadata.items()]
+            meta_str = f"  ({', '.join(meta_parts)})"
+        print(f"  {i+1}. [{token.token_type.value:>16}] '{token.text}'{meta_str}")
+
+
+def interactive_mode(tokenizer: TamilTokenizer) -> None:
+    """Run tokenizer in interactive mode."""
+    print("Tamil Tokenizer - Interactive Mode")
+    print("Type Tamil text to tokenize. Commands:")
+    print("  :level <sentence|word|character|morpheme>  - Change level")
+    print("  :quit                                      - Exit")
+    print(f"{'='*60}")
+
+    current_level = "word"
+
+    while True:
+        try:
+            text = input(f"\n[{current_level}] >>> ").strip()
+        except (EOFError, KeyboardInterrupt):
+            print("\nGoodbye!")
+            break
+
+        if not text:
+            continue
+
+        if text == ":quit":
+            print("Goodbye!")
+            break
+
+        if text.startswith(":level "):
+            new_level = text.split(maxsplit=1)[1].strip()
+            if new_level in ("sentence", "word", "character", "morpheme"):
+                current_level = new_level
+                print(f"Level set to: {current_level}")
+            else:
+                print(f"Unknown level: {new_level}. Use: sentence, word, character, morpheme")
+            continue
+
+        tokens = tokenizer.tokenize(text, level=current_level)
+        print_tokens(tokens, current_level, text)
+
+
+def main():
+    """Main entry point."""
+    parser = argparse.ArgumentParser(
+        description='Tamil Tokenizer - Multi-level tokenization for Tamil text',
+        formatter_class=argparse.RawDescriptionHelpFormatter,
+        epilog="""
+Examples:
+  %(prog)s "அவன் வந்தான்."                           Word tokenization (default)
+  %(prog)s "அவன் வந்தான்." --level character          Character tokenization
+  %(prog)s "அவன் வந்தான். அவள் பார்த்தாள்." --level sentence  Sentence tokenization
+  %(prog)s "பள்ளிக்கு சென்றான்." --level morpheme      Morpheme tokenization
+  %(prog)s --interactive                              Interactive mode
+        """
+    )
+
+    parser.add_argument('text', nargs='?', help='Tamil text to tokenize')
+    parser.add_argument('-l', '--level', default='word',
+                        choices=['sentence', 'word', 'character', 'morpheme'],
+                        help='Tokenization level (default: word)')
+    parser.add_argument('-i', '--interactive', action='store_true',
+                        help='Run in interactive mode')
+    parser.add_argument('-d', '--data', metavar='PATH',
+                        help='Path to data directory')
+    parser.add_argument('-f', '--format', default='table',
+                        choices=['table', 'text', 'json'],
+                        help='Output format (default: table)')
+    parser.add_argument('--version', action='version',
+                        version='Tamil Tokenizer 1.0.0')
+
+    args = parser.parse_args()
+
+    tokenizer = TamilTokenizer(args.data)
+
+    if args.interactive:
+        interactive_mode(tokenizer)
+    elif args.text:
+        if args.format == 'json':
+            import json
+            result = tokenizer.tokenize_to_dicts(args.text, level=args.level)
+            print(json.dumps(result, ensure_ascii=False, indent=2))
+        elif args.format == 'text':
+            result = tokenizer.tokenize_to_strings(args.text, level=args.level)
+            for t in result:
+                print(t)
+        else:
+            tokens = tokenizer.tokenize(args.text, level=args.level)
+            print_tokens(tokens, args.level, args.text)
+    else:
+        parser.print_help()
+
+
+if __name__ == '__main__':
+    main()

vettu-1.0.0/tamil_tokenizer/config/__init__.py

@@ -0,0 +1,13 @@
+"""Tamil tokenizer configuration module."""
+
+from .constants import ConfigConstants, DEFAULT_FILE_PATHS
+from .config_loader import ConfigLoader, ReadConfig
+from .constant_table import TamilConstantTable
+
+__all__ = [
+    'ConfigConstants',
+    'DEFAULT_FILE_PATHS',
+    'ConfigLoader',
+    'ReadConfig',
+    'TamilConstantTable',
+]