indicate 0.7.0__tar.gz → 0.7.1__tar.gz

indicate-0.7.1/.gitignore

@@ -0,0 +1,34 @@
+.*
+!.github
+!.gitignore
+!.gitattributes
+!.travis.yml
+!.pre-commit-config.yaml
+*.pyc
+*.log
+./data
+build/
+lit/
+dist/
+.ipynb_checkpoints
+*/.ipynb_checkpoints/*
+past/
+.venv
+*.egg-info*
+coverage.xml
+htmlcov/
+/test_install_tf220
+/test_tf219
+# Dakshina benchmark dataset (downloaded locally for training/eval.py)
+*.tar
+dakshina_dataset_v1.0/
+data/dakshina/
+# Large/raw Punjabi source data (silver, electoral-roll PII); committed data is data/punjabi.csv
+data/punjab_transliteration_subset.parquet
+training/IMPROVEMENTS.md
+# v2 (Aksharantar) — external download, derived training corpora, built eval sets
+data/aksharantar/
+data/eval/
+data/*_train_v2.csv.gz
+# Model weights are hosted on HF and lazy-downloaded; tokenizer JSONs stay tracked
+indicate/data/*/saved_weights/

indicate-0.7.1/.pre-commit-config.yaml

@@ -0,0 +1,40 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.8.7
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.12.1
+    hooks:
+      - id: mypy
+        additional_dependencies:
+          - types-requests
+        args: [--ignore-missing-imports]
+
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v5.0.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-toml
+      - id: check-merge-conflict
+      - id: check-case-conflict
+      - id: check-docstring-first
+      - id: debug-statements
+      - id: check-added-large-files
+
+  - repo: https://github.com/jsh9/pydoclint
+    rev: 0.8.6
+    hooks:
+      - id: pydoclint
+
+  - repo: https://github.com/pycqa/bandit
+    rev: 1.8.0
+    hooks:
+      - id: bandit
+        args: [-r, ., -ll]
+        exclude: ^tests/

indicate-0.7.1/CHANGELOG.md

@@ -0,0 +1,63 @@
+# Changelog
+
+## 0.7.0
+
+### Added
+- **Data-scaled v2 models** — both Hindi and Punjabi retrained on the public
+  **Aksharantar** corpus (AI4Bharat) merged with our own data: Hindi ~1.5M pairs,
+  Punjabi ~773k. A strict **leakage filter** drops every eval word from training.
+- **Batched inference** — `transliterate_batch(list[str])` is the decode engine
+  (batched greedy + beam); `transliterate(str)` is a thin wrapper. The CLI
+  `--batch` mode and bulk eval use it (≫ faster than one-at-a-time).
+- **Top-k / n-best** — `transliterate(text, n=k)` returns up to `k` ranked
+  candidates (a `list[str]`); `n=1` (default) still returns a `str`.
+
+### Changed
+- **Model weights hosted on Hugging Face** (`soodoku/indicate`, tag `v0.7.0`) and
+  lazy-downloaded (cached) on first use, so the wheel is ~50 KB instead of ~107 MB
+  (PyPI's 100 MB limit). Tokenizer JSONs still ship in the package; a local copy of
+  the weights, if present, is used in preference to the download. Weights are fp32
+  (`safetensors`) — fp16 was tried and rejected (it perturbed ~0.05% of outputs for
+  no size benefit worth keeping, since hosting solves the size problem).
+- **Input-adaptive decode cap** (`min(max_length_output, 2·len(input)+8)`).
+- Removed `func_timeout` (the decoder is hard-bounded, so the wall-clock guard
+  was dead weight — and a perf/flakiness drag); dropped the dependency.
+- Coverage reports to the terminal only (no `htmlcov`/`coverage.xml`).
+
+### Accuracy (Dakshina test, exact-match; vs AI4Bharat IndicXlit, same direction/data)
+| Model | Dakshina (gold) | Held-out-own names | 
+|-------|-----------------|--------------------|
+| Hindi v2 | **74.4%** (IndicXlit 73.2%) | **52.8%** (IndicXlit 49.7%) |
+| Punjabi v2 | 71.9% (IndicXlit 73.2%) | **56.9%** (IndicXlit 53.5%) |
+
+v2 matches/edges SOTA IndicXlit on the gold benchmark and **beats it on the
+deployment domain** (names IndicXlit never trained on). Eval is leakage-filtered;
+see `training/IMPROVEMENTS.md` and `training/compare.py`.
+
+## 0.6.0
+
+### Added
+- **Punjabi (Gurmukhi → English) transliteration model** — new `PunjabiToEnglish`
+  class, `indicate.punjabi2english(...)` API, and `indicate punjabi2english` CLI
+  command. Distilled from GPT-4o-labelled Punjab data
+  (`training/extract_punjabi.py` → `data/punjabi.csv.gz`).
+- **Beam-search decoding** (length-normalized, width 5) is now the default for
+  both models — +1.5 pts exact-match on the Hindi Dakshina test set vs greedy.
+- Evaluation tooling: `training/eval.py` (gold Dakshina), `training/oos_eval.py`
+  (held-out), and `training/metrics.py` reporting exact-match accuracy, CER, and
+  Acc@≤1 with edit-distance histograms.
+
+### Changed
+- **Migrated the modeling stack from TensorFlow to PyTorch** and the package to
+  **Python 3.13-only**. Weights now ship as `safetensors`.
+- Refactored the per-language logic into a shared `Seq2SeqTransliterator` base.
+
+### Accuracy (Google Dakshina test, exact-match / Acc@≤1 / CER)
+- Hindi → English: **77.32% / 91.16% / 5.65%**
+- Punjabi → English: **71.24% / 91.56% / 6.42%**
+
+### Notes
+- Char-LM beam re-ranking and attention padding masks are implemented but gated
+  off (no measured win for the name-heavy, native→Latin case); see
+  `training/IMPROVEMENTS.md`. The next milestone is a data-scale retrain on the
+  public Aksharantar corpus.

indicate-0.7.1/CITATION.cff

@@ -0,0 +1,11 @@
+cff-version: 1.2.0
+message: "If you use this software, please cite it as below."
+authors:
+- family-names: "Chintalapati"
+  given-names: "Rajashekar"
+- family-names: "Sood"
+  given-names: "Gaurav"
+title: "Indicate: Transliterate Indic Languages to English"
+version: 0.1.0
+date-released: 2022-08-21
+url: "https://github.com/in-rolls/indicate"

indicate-0.7.1/CLAUDE.md

@@ -0,0 +1,77 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## Project Overview
+
+Indicate is a Python package for transliterating Indic text to English using PyTorch-based encoder-decoder models with attention. It ships local models for **Hindi** (Devanagari) and **Punjabi** (Gurmukhi), plus an LLM backend for other languages. The local models are custom-trained neural networks with pre-trained weights.
+
+## Development Commands
+
+### Testing
+```bash
+python -m unittest tests.test_010_hindi_translate
+python -m unittest discover tests/
+```
+
+### Build and Installation
+```bash
+uv sync           # Install dependencies with uv (recommended)
+uv build          # Build package
+pip install -e .  # Install in development mode (alternative)
+```
+
+### CLI Usage
+```bash
+# Modern Click-based CLI
+indicate hindi2english "राजशेखर चिंतालपति"
+indicate hindi2english --input file.txt --output result.txt
+indicate info
+```
+
+### Documentation (Sphinx)
+```bash
+cd docs/
+make html         # Build HTML documentation
+make clean        # Clean build artifacts
+```
+
+## Architecture
+
+### Core Components
+
+1. **Seq2SeqTransliterator** (`indicate/transliterator.py`) - Base lazy-loaded singleton holding the load + greedy `transliterate()` logic, parameterized by class attrs (vocab/weights paths, max lengths)
+2. **HindiToEnglish** (`indicate/hindi2english.py`) / **PunjabiToEnglish** (`indicate/punjabi2english.py`) - Thin subclasses pointing at each language's tokenizers + safetensors
+3. **Encoder** (`indicate/encoder.py`) - `nn.Module` LSTM encoder
+4. **Decoder** (`indicate/decoder.py`) - `nn.Module` LSTM decoder with Luong (dot-product) attention
+5. **Utils** (`indicate/utils.py`) - Tokenizer loading (`load_tokenizer`) and greedy decoding (`translate`)
+
+### Model Architecture
+- Encoder-decoder with Luong attention mechanism
+- Embedding dimension: 256, LSTM units: 1024
+- Per-language safetensors weights + tokenizer JSONs under
+  `indicate/data/{hindi,punjabi}_to_english/`
+
+### Training
+- PyTorch training/extraction/eval scripts live in `training/` (see `training/README.md`)
+- Hindi corpus `data/hindi.csv.gz` and Punjabi corpus `data/punjabi.csv.gz` are committed
+  (Punjabi is extracted from the Dataverse-hosted parquet via `training/extract_punjabi.py`)
+- Raw/large source data and the Dakshina benchmark live on Dataverse (see `data/README.md`)
+
+### Data Pipeline
+- Training data from ESPN Cricinfo, election affidavits, Google Dakshina dataset, and IIT Bombay corpus
+- Character-level tokenization with special start (^) and end ($) tokens
+- Decoding is hard-bounded by an input-adaptive step cap (no wall-clock timeout)
+
+### Key Entry Points
+- CLI: `indicate` command (Click group) with `hindi2english` / `punjabi2english` subcommands
+- API: `indicate.hindi2english(text)` and `indicate.punjabi2english(text)` functions
+- Main module: `indicate/__init__.py` exposes both transliteration functions
+
+## Dependencies
+- Python 3.13+ (modern Python with enhanced type hints)
+- Click 8.0+ (modern CLI framework)
+- PyTorch 2.6+ (core ML framework)
+- safetensors 0.4+ (model weight serialization)
+- func-timeout 4.3.0+ (prevents hanging translations)
+- tqdm 4.60.0+ (progress bars)

indicate-0.7.1/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2021 Rajasekhar Chintalapati and Gaurav Sood
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

indicate-0.7.1/Makefile

@@ -0,0 +1,56 @@
+.PHONY: help clean install test test-cov lint format type-check docs docs-serve build upload dev-install
+
+help: ## Show this help message
+	@grep -E '^[a-zA-Z_-]+:.*?## .*$$' $(MAKEFILE_LIST) | sort | awk 'BEGIN {FS = ":.*?## "}; {printf "\033[36m%-20s\033[0m %s\n", $$1, $$2}'
+
+clean: ## Clean build artifacts
+	rm -rf build/
+	rm -rf dist/
+	rm -rf *.egg-info/
+	rm -rf .pytest_cache/
+	rm -rf .coverage
+	rm -rf htmlcov/
+	rm -rf docs/_build/
+	find . -type d -name __pycache__ -delete
+	find . -type f -name "*.pyc" -delete
+
+install: ## Install package
+	uv pip install -e .
+
+dev-install: ## Install package with development dependencies
+	uv sync --dev
+	pre-commit install
+
+test: ## Run tests
+	pytest
+
+test-cov: ## Run tests with coverage
+	pytest --cov=indicate --cov-report=html --cov-report=term
+
+lint: ## Run linter
+	ruff check .
+
+format: ## Format code
+	ruff format .
+	ruff check --fix .
+
+type-check: ## Run type checker
+	mypy indicate/
+
+docs: ## Build documentation
+	cd docs && make clean && make html
+
+docs-serve: ## Serve documentation locally
+	cd docs/build/html && python -m http.server 8000
+
+build: ## Build package
+	uv build
+
+upload: ## Upload to PyPI
+	uv publish
+
+ci: lint type-check test ## Run CI checks
+
+# Legacy aliases for compatibility
+install-dev: dev-install ## Legacy alias for dev-install
+typecheck: type-check ## Legacy alias for type-check

{indicate-0.7.0 → indicate-0.7.1}/PKG-INFO

@@ -1,28 +1,28 @@
-Metadata-Version: 2.3
+Metadata-Version: 2.4
 Name: indicate
-Version: 0.7.0
+Version: 0.7.1
 Summary: Transliterations to/from Indian languages
-Keywords: transliterate,indic,hindi,english
-Author: Rajashekar Chintalapati, Gaurav Sood
+Project-URL: Homepage, https://github.com/in-rolls/indicate
+Project-URL: Repository, https://github.com/in-rolls/indicate
 Author-email: Rajashekar Chintalapati <rajshekar.ch@gmail.com>, Gaurav Sood <gsood07@gmail.com>
 License: MIT
+License-File: LICENSE
+Keywords: english,hindi,indic,transliterate
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Topic :: Scientific/Engineering :: Information Analysis
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Classifier: Topic :: Utilities
+Requires-Python: >=3.13
 Requires-Dist: click>=8.0.0
-Requires-Dist: tqdm>=4.60.0
-Requires-Dist: numpy>=1.26
-Requires-Dist: torch>=2.6.0
-Requires-Dist: safetensors>=0.4.0
+Requires-Dist: google-genai>=1.0
 Requires-Dist: huggingface-hub>=0.23
 Requires-Dist: litellm>=1.0.0
-Requires-Dist: google-genai>=1.0
-Requires-Python: >=3.13
-Project-URL: Homepage, https://github.com/in-rolls/indicate
-Project-URL: Repository, https://github.com/in-rolls/indicate
+Requires-Dist: numpy>=1.26
+Requires-Dist: safetensors>=0.4.0
+Requires-Dist: torch>=2.6.0
+Requires-Dist: tqdm>=4.60.0
 Description-Content-Type: text/markdown
 
 # Indicate: Transliterate Indic Languages with PyTorch and LLMs

indicate-0.7.1/dist/.gitignore

@@ -0,0 +1 @@
+*

{indicate-0.7.0 → indicate-0.7.1}/indicate/__init__.py

@@ -1,11 +1,15 @@
+from .hindi2english import HindiToEnglish
 from .indic_utils import detect_indic_script, detect_language_from_script
 from .llm_indic import IndicLLMTransliterator
-from .transliterate import hindi2english, punjabi2english
+from .punjabi2english import PunjabiToEnglish
+
+hindi2english = HindiToEnglish.transliterate
+punjabi2english = PunjabiToEnglish.transliterate
 
 __all__ = [
-    "hindi2english",
-    "punjabi2english",
     "IndicLLMTransliterator",
     "detect_indic_script",
     "detect_language_from_script",
+    "hindi2english",
+    "punjabi2english",
 ]

{indicate-0.7.0 → indicate-0.7.1}/indicate/batch.py

@@ -30,6 +30,7 @@ import tempfile
 import time
 from dataclasses import asdict, dataclass, field
 from pathlib import Path
+from typing import Any, cast
 
 import litellm
 
@@ -175,8 +176,8 @@ def _content_to_text(content) -> str:
 def _parse_output_jsonl(content) -> dict[str, str]:
     """Map each request's ``custom_id`` to the model's text output.
 
-    Output lines follow the OpenAI batch schema:
-    ``{"custom_id", "response": {"body": {"choices": [{"message": {"content"}}]}}, "error"}``.
+    Output lines follow the OpenAI batch schema (``custom_id``,
+    ``response.body.choices[0].message.content``, ``error``).
     """
     results: dict[str, str] = {}
     for line in _content_to_text(content).splitlines():
@@ -257,7 +258,7 @@ def _parse_gemini_jsonl(content) -> dict[str, str]:
 
 
 def _write_jsonl(records: list[dict]) -> str:
-    tmp = tempfile.NamedTemporaryFile(
+    tmp = tempfile.NamedTemporaryFile(  # noqa: SIM115 - delete=False, path returned
         "w", suffix=".jsonl", delete=False, encoding="utf-8"
     )
     for record in records:
@@ -294,14 +295,21 @@ def _submit_openai_jobs(
         path = _write_jsonl(request_chunk)
         try:
             with open(path, "rb") as handle:
-                file_obj = litellm.create_file(
-                    file=handle, purpose="batch", custom_llm_provider=state.provider
+                # litellm types the sync variants as possibly-Coroutine
+                file_obj = cast(
+                    Any,
+                    litellm.create_file(
+                        file=handle, purpose="batch", custom_llm_provider=state.provider
+                    ),
                 )
-            batch = litellm.create_batch(
-                completion_window=completion_window,
-                endpoint=BATCH_ENDPOINT,
-                input_file_id=file_obj.id,
-                custom_llm_provider=state.provider,
+            batch = cast(
+                Any,
+                litellm.create_batch(
+                    completion_window=completion_window,
+                    endpoint=BATCH_ENDPOINT,
+                    input_file_id=file_obj.id,
+                    custom_llm_provider=state.provider,
+                ),
             )
         finally:
             os.unlink(path)
@@ -347,7 +355,11 @@ def _submit_gemini_jobs(
             uploaded = client.files.upload(
                 file=path, config=types.UploadFileConfig(mime_type="jsonl")
             )
+            if not uploaded.name:
+                raise RuntimeError("Gemini file upload returned no name")
             job = client.batches.create(model=gmodel, src=uploaded.name)
+            if not job.name:
+                raise RuntimeError("Gemini batch create returned no name")
         finally:
             os.unlink(path)
         chunk_map = {r["key"]: custom_id_to_tokens[r["key"]] for r in request_chunk}
@@ -364,7 +376,10 @@ def _submit_gemini_jobs(
 
 
 def _poll_job(job: BatchJob, provider: str) -> tuple[str, dict[str, str]]:
-    """Poll one job. Returns (status, by_custom_id); status in completed/running/failed."""
+    """Poll one job.
+
+    Returns (status, by_custom_id); status in completed/running/failed.
+    """
     if provider in _GEMINI_PROVIDERS:
         client = _gemini_client()  # keep a ref for the whole call (httpx lifecycle)
         info = client.batches.get(name=job.batch_id)
@@ -384,7 +399,7 @@ def _poll_job(job: BatchJob, provider: str) -> tuple[str, dict[str, str]]:
         return "running", {}
 
     retrieved = litellm.retrieve_batch(
-        batch_id=job.batch_id, custom_llm_provider=provider
+        batch_id=job.batch_id, custom_llm_provider=cast(Any, provider)
     )
     status = getattr(retrieved, "status", None)
     output_file_id = getattr(retrieved, "output_file_id", None)
@@ -535,7 +550,8 @@ def collect_transliteration_batches(
                 parsed = transliterator._parse_batch_response(text, len(group))
                 if len(raw_lines) != len(group) or any(not p for p in parsed):
                     logger.warning(
-                        "Batch %s group %s: %d output line(s) for %d tokens; requeueing",
+                        "Batch %s group %s: %d output line(s) for %d tokens; "
+                        "requeueing",
                         job.batch_id,
                         custom_id,
                         len(raw_lines),