bhashantar 0.1.0__tar.gz

bhashantar-0.1.0/.github/workflows/smoke.yml

@@ -0,0 +1,34 @@
+name: smoke
+
+# Failure mode 2 is breakage when a dependency releases. The weekly job
+# installs whatever is newest within the declared ranges and translates.
+on:
+  push:
+  pull_request:
+  schedule:
+    - cron: "17 4 * * 1"
+
+jobs:
+  offline:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.10", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python }}
+      - run: pip install --upgrade ".[dev]"
+      - run: pytest -q
+
+  model:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+      - run: pip install --upgrade ".[dev]"
+      - run: pytest -q -m model
+      - run: bhashantar translate --tgt hin_Deva "The train leaves at 6:45 from platform 3."

bhashantar-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+.venv/
+.venv-ref/
+.ref-model/
+__pycache__/
+*.egg-info/
+.pytest_cache/
+dist/
+bench/results/*.partial

bhashantar-0.1.0/HANDOFF.md

@@ -0,0 +1,5 @@
+# bhashantar handoff (2026-11-10)
+
+State: pushed to private zaidwhy/bhashantar. 13 offline tests; 27 model tests + 1 xfail (Santali). IndicTrans2 on CTranslate2 int8: chrF vs reference hin -0.52, ben +0.10, tam +0.27; about 0.3 s per sentence; 707 MB.
+
+Next: publishing int8 weights to HF and PyPI are Zaid's call.

bhashantar-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Zaid Ali Syed
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

bhashantar-0.1.0/NOTICE

@@ -0,0 +1,25 @@
+bhashantar includes code adapted from the following MIT-licensed projects.
+
+src/bhashantar/textprep.py
+  Adapted from IndicTransToolkit (processor.pyx, IndicProcessor)
+  https://github.com/VarunGumma/IndicTransToolkit
+  MIT License, Copyright (c) Varun Gumma.
+
+src/bhashantar/indic_script.py
+  Adapted from indic_nlp_library (unicode_transliterate.py, indic_detokenize.py, langinfo.py)
+  https://github.com/anoopkunchukuttan/indic_nlp_library
+  MIT License, Copyright (c) 2013-present Anoop Kunchukuttan.
+
+src/bhashantar/convert.py
+  Weight mapping follows ctranslate2/converters/fairseq.py
+  https://github.com/OpenNMT/CTranslate2
+  MIT License, Copyright (c) 2019 SYSTRAN.
+
+Model weights (downloaded at runtime, not shipped in this package)
+  IndicTrans2 en-indic-dist-200M by AI4Bharat
+  https://huggingface.co/ai4bharat/indictrans2-en-indic-dist-200M
+  MIT License. The converted CTranslate2 model directory carries a copy of
+  that LICENSE file.
+
+Both MIT notices require that the copyright and permission notice be included
+in copies or substantial portions; the full MIT text is in LICENSE.

bhashantar-0.1.0/PKG-INFO

@@ -0,0 +1,157 @@
+Metadata-Version: 2.5
+Name: bhashantar
+Version: 0.1.0
+Summary: IndicTrans2 English-to-Indic translation on CPU via CTranslate2 int8, no transformers or fairseq at runtime
+Author-email: Zaid Ali Syed <sidzaid72@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+License-File: NOTICE
+Keywords: ctranslate2,hindi,indic,indictrans2,machine-translation
+Classifier: Natural Language :: Hindi
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Requires-Dist: ctranslate2<5,>=4.4
+Requires-Dist: huggingface-hub>=0.24
+Requires-Dist: regex>=2023.0
+Requires-Dist: sacremoses>=0.1.1
+Requires-Dist: sentencepiece<0.3,>=0.2
+Provides-Extra: convert
+Requires-Dist: safetensors>=0.4; extra == 'convert'
+Provides-Extra: dev
+Requires-Dist: psutil>=5.9; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: sacrebleu>=2.4; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# bhashantar
+
+English to 25 Indic languages with AI4Bharat's IndicTrans2 (`en-indic-dist-200M`), on CPU, through CTranslate2 int8. No transformers, no torch, no fairseq and no IndicTransToolkit at runtime.
+
+```python
+from bhashantar import translate
+
+translate(["The weather in Mumbai is pleasant today."], tgt="hin_Deva")
+# ['मुंबई में आज मौसम सुखद है।']
+```
+
+```
+$ bhashantar translate --tgt hin_Deva "Send the report to ops@example.com by 15-03-2025."
+रिपोर्ट को 15-03-2025 द्वारा ops@example.com पर भेजें।
+```
+
+## Why this exists
+
+The official checkpoint loads through `trust_remote_code` files written against transformers 4.3x. On a fresh Python 3.12 venv with transformers 5.22.2 and IndicTransToolkit 1.1.1 (reproduced 2026-11-09):
+
+```
+from IndicTransToolkit.processor import IndicProcessor
+ImportError: cannot import name 'PreTrainedTokenizerBase' from 'transformers.tokenization_utils'
+
+AutoTokenizer.from_pretrained(..., trust_remote_code=True)
+ModuleNotFoundError: No module named 'transformers.onnx'
+
+AutoModelForSeq2SeqLM.from_pretrained(..., trust_remote_code=True)
+ModuleNotFoundError: No module named 'transformers.onnx'
+```
+
+This is [IndicTransToolkit#29](https://github.com/VarunGumma/IndicTransToolkit/issues/29). A fix for the remote code is open as discussion #15 on the model repo, unmerged at the time of writing. Pinning transformers 4.50.3 makes the stack load again; that pinned stack is the reference this package is measured against.
+
+## Data path
+
+```
+raw text
+  -> punctuation and digit normalisation, URLs/emails/numbers swapped for <IDn>   (textprep.prepare)
+  -> Moses tokenisation, "eng_Latn hin_Deva" tags, SentencePiece pieces           (engine)
+  -> CTranslate2 int8 beam search (beam 5), Devanagari-unified pieces out
+  -> detokenise pieces, restore <IDn>, Devanagari -> target script, reattach punctuation   (textprep.finish)
+```
+
+The model writes every Brahmic target in Devanagari; Tamil, Bengali and the rest come from a fixed Unicode offset afterwards. That step and the placeholder restore are where a port silently goes wrong, so both are vendored from IndicTransToolkit and indic_nlp_library (MIT, see `NOTICE`) with their regexes and quirks intact. Two deliberate changes: placeholder maps travel with each sentence instead of a shared queue, and placeholder numbering is sorted instead of following set order, which varied between processes.
+
+## Two ways this fails, and the guard for each
+
+**Silent quality loss from conversion or quantisation.** CTranslate2 has no converter for this checkpoint's custom class, so `bhashantar.convert` maps the Hugging Face tensors onto the same `TransformerSpec` the fairseq converter fills. A wrong position offset or a missed layer norm still produces fluent-looking text. The guard is `bench/parity.py`: chrF against human references for both this package and the upstream model, with the pass bar fixed before the first run.
+
+**Breakage on library upgrades.** Runtime dependencies are only what inference touches: `ctranslate2` (capped below 5, since `model.bin` is tied to the major), `sentencepiece`, `sacremoses`, `regex`, `huggingface_hub`. `.github/workflows/smoke.yml` installs the newest versions in range every week and translates.
+
+**The opinion:** parity is published as a number, not claimed. If the delta ever exceeds the bar, the table says so.
+
+## Parity
+
+FLORES-200 devtest, first 100 sentences, English source. The FLORES files come from the public `dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz`; AI4Bharat's IN22 and the Hugging Face FLORES+ copy are both gated. Reference = upstream model, transformers 4.50.3 + IndicTransToolkit 1.1.1, fp32, beam 5. Pass bar, set before running: |delta| <= 1.0 chrF.
+
+| pair | chrF reference | chrF bhashantar | delta | identical outputs | pass |
+|---|---|---|---|---|---|
+| eng_Latn-hin_Deva | 65.69 | 65.17 | -0.52 | 61/100 | yes |
+| eng_Latn-ben_Beng | 54.52 | 54.62 | +0.10 | 54/100 | yes |
+| eng_Latn-tam_Taml | 61.33 | 61.59 | +0.27 | 56/100 | yes |
+
+Reproduce: `python bench/parity.py score <flores devtest dir> bench/results` (outputs and `parity.json` are committed in `bench/results`).
+
+"Identical outputs" counts sentences where both systems produce the same string; the rest differ in word choice (for example कनाडाई मधुमेह संघ against कैनेडियन डायबिटीज एसोसिएशन), the expected effect of int8 weights on beam search. Conversion error and quantisation error are not separated: an fp32 CTranslate2 conversion did not fit on the build machine's disk.
+
+Reference weights: the official repo is gated, so the reference used `Raghavan/indictrans2-en-indic-dist-200M`, whose `pytorch_model.bin` has the same Git LFS pointer as the official file (sha256 `52671a26...0fef0`, verified locally); tokenizer files match the official Git blob ids.
+
+## Speed and memory
+
+| | bhashantar (CT2 int8) | upstream (transformers fp32) |
+|---|---|---|
+| model load | 0.47 s | not timed |
+| ms per sentence, eng to tam_Taml, beam 5 | 323 (all 100 in one call) | 2,786 (batch 2, 80 sentences timed) |
+| peak RSS | 707 MB | 1,472 MB |
+
+The two latency figures are not like for like: the reference had to run at batch 2 because the build machine ran out of commit memory at batch 8, and it was resumed once mid-run, so only the last 80 sentences were timed. bhashantar on Hindi and Bengali: 328 and 308 ms per sentence.
+
+Measured on a laptop (Windows 11, 16 GB RAM, 4 threads, other jobs running), so treat them as relative, not absolute.
+
+## Limits
+
+- One direction: English to Indic. The `indic-en` and `indic-indic` checkpoints are not converted.
+- Caps, refused with the cap named in the error: 1,000 characters per sentence, 512 sentences and 50,000 characters per call, 256 subword tokens per sentence after preprocessing. Long input is refused, not truncated.
+- Santali (`sat_Olck`) output mixes in Oriya and Arabic words for some sentences. The upstream model does the same; the test marks it as a known model limit.
+- Sentence-level: split paragraphs yourself.
+
+## Model files
+
+`translate()` looks for a model in this order: the `model_dir` argument, `$BHASHANTAR_MODEL`, a local conversion at `$HF_HOME/bhashantar/en-indic-dist-200M-ct2-int8`, then the Hugging Face repo `zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8` (cache first, so it works offline after the first download).
+
+That repo is not published yet. Until it is, convert once yourself (needs a Hugging Face login with the model's terms accepted):
+
+```
+pip install "bhashantar[convert]"
+hf auth login
+bhashantar convert
+```
+
+## Development
+
+```
+pip install -e ".[dev]"
+pytest                 # offline tests, no model
+pytest -m model        # end to end, needs the converted model
+```
+
+## Maintainer: publishing (not done yet)
+
+```
+# 1. weights, from a machine with the converted model
+hf auth login
+hf repo create zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8 --repo-type model
+hf upload zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8 ~/.cache/huggingface/bhashantar/en-indic-dist-200M-ct2-int8 .
+
+# 2. check that a clean install fetches it
+python -m venv /tmp/bt && /tmp/bt/bin/pip install . && /tmp/bt/bin/bhashantar translate --tgt tam_Taml "Good morning."
+
+# 3. package
+python -m pip install build twine
+python -m build
+twine check dist/*
+twine upload dist/*
+```
+
+The model directory carries AI4Bharat's MIT `LICENSE` and a `provenance.json` with the source weights' sha256; add a model card stating the parity numbers above before uploading.
+
+## License
+
+MIT. Vendored code and model weights keep their own MIT notices; see `NOTICE`.

bhashantar-0.1.0/README.md

@@ -0,0 +1,131 @@
+# bhashantar
+
+English to 25 Indic languages with AI4Bharat's IndicTrans2 (`en-indic-dist-200M`), on CPU, through CTranslate2 int8. No transformers, no torch, no fairseq and no IndicTransToolkit at runtime.
+
+```python
+from bhashantar import translate
+
+translate(["The weather in Mumbai is pleasant today."], tgt="hin_Deva")
+# ['मुंबई में आज मौसम सुखद है।']
+```
+
+```
+$ bhashantar translate --tgt hin_Deva "Send the report to ops@example.com by 15-03-2025."
+रिपोर्ट को 15-03-2025 द्वारा ops@example.com पर भेजें।
+```
+
+## Why this exists
+
+The official checkpoint loads through `trust_remote_code` files written against transformers 4.3x. On a fresh Python 3.12 venv with transformers 5.22.2 and IndicTransToolkit 1.1.1 (reproduced 2026-11-09):
+
+```
+from IndicTransToolkit.processor import IndicProcessor
+ImportError: cannot import name 'PreTrainedTokenizerBase' from 'transformers.tokenization_utils'
+
+AutoTokenizer.from_pretrained(..., trust_remote_code=True)
+ModuleNotFoundError: No module named 'transformers.onnx'
+
+AutoModelForSeq2SeqLM.from_pretrained(..., trust_remote_code=True)
+ModuleNotFoundError: No module named 'transformers.onnx'
+```
+
+This is [IndicTransToolkit#29](https://github.com/VarunGumma/IndicTransToolkit/issues/29). A fix for the remote code is open as discussion #15 on the model repo, unmerged at the time of writing. Pinning transformers 4.50.3 makes the stack load again; that pinned stack is the reference this package is measured against.
+
+## Data path
+
+```
+raw text
+  -> punctuation and digit normalisation, URLs/emails/numbers swapped for <IDn>   (textprep.prepare)
+  -> Moses tokenisation, "eng_Latn hin_Deva" tags, SentencePiece pieces           (engine)
+  -> CTranslate2 int8 beam search (beam 5), Devanagari-unified pieces out
+  -> detokenise pieces, restore <IDn>, Devanagari -> target script, reattach punctuation   (textprep.finish)
+```
+
+The model writes every Brahmic target in Devanagari; Tamil, Bengali and the rest come from a fixed Unicode offset afterwards. That step and the placeholder restore are where a port silently goes wrong, so both are vendored from IndicTransToolkit and indic_nlp_library (MIT, see `NOTICE`) with their regexes and quirks intact. Two deliberate changes: placeholder maps travel with each sentence instead of a shared queue, and placeholder numbering is sorted instead of following set order, which varied between processes.
+
+## Two ways this fails, and the guard for each
+
+**Silent quality loss from conversion or quantisation.** CTranslate2 has no converter for this checkpoint's custom class, so `bhashantar.convert` maps the Hugging Face tensors onto the same `TransformerSpec` the fairseq converter fills. A wrong position offset or a missed layer norm still produces fluent-looking text. The guard is `bench/parity.py`: chrF against human references for both this package and the upstream model, with the pass bar fixed before the first run.
+
+**Breakage on library upgrades.** Runtime dependencies are only what inference touches: `ctranslate2` (capped below 5, since `model.bin` is tied to the major), `sentencepiece`, `sacremoses`, `regex`, `huggingface_hub`. `.github/workflows/smoke.yml` installs the newest versions in range every week and translates.
+
+**The opinion:** parity is published as a number, not claimed. If the delta ever exceeds the bar, the table says so.
+
+## Parity
+
+FLORES-200 devtest, first 100 sentences, English source. The FLORES files come from the public `dl.fbaipublicfiles.com/nllb/flores200_dataset.tar.gz`; AI4Bharat's IN22 and the Hugging Face FLORES+ copy are both gated. Reference = upstream model, transformers 4.50.3 + IndicTransToolkit 1.1.1, fp32, beam 5. Pass bar, set before running: |delta| <= 1.0 chrF.
+
+| pair | chrF reference | chrF bhashantar | delta | identical outputs | pass |
+|---|---|---|---|---|---|
+| eng_Latn-hin_Deva | 65.69 | 65.17 | -0.52 | 61/100 | yes |
+| eng_Latn-ben_Beng | 54.52 | 54.62 | +0.10 | 54/100 | yes |
+| eng_Latn-tam_Taml | 61.33 | 61.59 | +0.27 | 56/100 | yes |
+
+Reproduce: `python bench/parity.py score <flores devtest dir> bench/results` (outputs and `parity.json` are committed in `bench/results`).
+
+"Identical outputs" counts sentences where both systems produce the same string; the rest differ in word choice (for example कनाडाई मधुमेह संघ against कैनेडियन डायबिटीज एसोसिएशन), the expected effect of int8 weights on beam search. Conversion error and quantisation error are not separated: an fp32 CTranslate2 conversion did not fit on the build machine's disk.
+
+Reference weights: the official repo is gated, so the reference used `Raghavan/indictrans2-en-indic-dist-200M`, whose `pytorch_model.bin` has the same Git LFS pointer as the official file (sha256 `52671a26...0fef0`, verified locally); tokenizer files match the official Git blob ids.
+
+## Speed and memory
+
+| | bhashantar (CT2 int8) | upstream (transformers fp32) |
+|---|---|---|
+| model load | 0.47 s | not timed |
+| ms per sentence, eng to tam_Taml, beam 5 | 323 (all 100 in one call) | 2,786 (batch 2, 80 sentences timed) |
+| peak RSS | 707 MB | 1,472 MB |
+
+The two latency figures are not like for like: the reference had to run at batch 2 because the build machine ran out of commit memory at batch 8, and it was resumed once mid-run, so only the last 80 sentences were timed. bhashantar on Hindi and Bengali: 328 and 308 ms per sentence.
+
+Measured on a laptop (Windows 11, 16 GB RAM, 4 threads, other jobs running), so treat them as relative, not absolute.
+
+## Limits
+
+- One direction: English to Indic. The `indic-en` and `indic-indic` checkpoints are not converted.
+- Caps, refused with the cap named in the error: 1,000 characters per sentence, 512 sentences and 50,000 characters per call, 256 subword tokens per sentence after preprocessing. Long input is refused, not truncated.
+- Santali (`sat_Olck`) output mixes in Oriya and Arabic words for some sentences. The upstream model does the same; the test marks it as a known model limit.
+- Sentence-level: split paragraphs yourself.
+
+## Model files
+
+`translate()` looks for a model in this order: the `model_dir` argument, `$BHASHANTAR_MODEL`, a local conversion at `$HF_HOME/bhashantar/en-indic-dist-200M-ct2-int8`, then the Hugging Face repo `zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8` (cache first, so it works offline after the first download).
+
+That repo is not published yet. Until it is, convert once yourself (needs a Hugging Face login with the model's terms accepted):
+
+```
+pip install "bhashantar[convert]"
+hf auth login
+bhashantar convert
+```
+
+## Development
+
+```
+pip install -e ".[dev]"
+pytest                 # offline tests, no model
+pytest -m model        # end to end, needs the converted model
+```
+
+## Maintainer: publishing (not done yet)
+
+```
+# 1. weights, from a machine with the converted model
+hf auth login
+hf repo create zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8 --repo-type model
+hf upload zaidwhy/indictrans2-en-indic-dist-200M-ct2-int8 ~/.cache/huggingface/bhashantar/en-indic-dist-200M-ct2-int8 .
+
+# 2. check that a clean install fetches it
+python -m venv /tmp/bt && /tmp/bt/bin/pip install . && /tmp/bt/bin/bhashantar translate --tgt tam_Taml "Good morning."
+
+# 3. package
+python -m pip install build twine
+python -m build
+twine check dist/*
+twine upload dist/*
+```
+
+The model directory carries AI4Bharat's MIT `LICENSE` and a `provenance.json` with the source weights' sha256; add a model card stating the parity numbers above before uploading.
+
+## License
+
+MIT. Vendored code and model weights keep their own MIT notices; see `NOTICE`.

bhashantar-0.1.0/bench/parity.py

@@ -0,0 +1,84 @@
+"""Parity of bhashantar (CTranslate2 int8) against the upstream transformers model.
+
+Step 1, `run`: translate the FLORES-200 devtest slice with bhashantar and record
+latency and peak RSS. Step 2, `score`: chrF of both systems against the human
+references, the delta, and how often the two outputs are identical.
+
+    .venv/Scripts/python bench/parity.py run   FLORES_DEVTEST_DIR bench/results
+    .venv/Scripts/python bench/parity.py score FLORES_DEVTEST_DIR bench/results
+
+Pass bar, fixed before the first run: |chrF(ours) - chrF(reference)| <= 1.0
+for every pair.
+"""
+
+import json
+import sys
+import time
+from pathlib import Path
+
+import psutil
+import sacrebleu
+
+from bhashantar import Translator
+
+PAIRS = ["hin_Deva", "ben_Beng", "tam_Taml"]
+SENTENCES = 100
+PASS_BAR = 1.0
+
+
+def run(flores_dir: str, out_dir: str) -> None:
+    out = Path(out_dir)
+    source = Path(flores_dir, "eng_Latn.devtest").read_text(encoding="utf-8").splitlines()[:SENTENCES]
+    started = time.perf_counter()
+    translator = Translator(threads=4)
+    translator.translate(["warm up"], tgt="hin_Deva")
+    stats = {"load_seconds": round(time.perf_counter() - started, 2)}
+    for tgt in PAIRS:
+        started = time.perf_counter()
+        hyps = translator.translate(source, tgt=tgt)
+        elapsed = time.perf_counter() - started
+        Path(out, f"ours.eng_Latn-{tgt}.txt").write_text("\n".join(hyps) + "\n", encoding="utf-8")
+        stats[tgt] = {"seconds": round(elapsed, 2), "ms_per_sentence": round(1000 * elapsed / len(source), 1)}
+        print(tgt, stats[tgt], flush=True)
+    stats["peak_rss_mb"] = round(psutil.Process().memory_info().peak_wset / 2**20)
+    Path(out, "ours.stats.json").write_text(json.dumps(stats, indent=2), encoding="utf-8")
+    print(stats)
+
+
+def _lines(path: Path) -> list[str]:
+    return path.read_text(encoding="utf-8").splitlines()[:SENTENCES]
+
+
+def score(flores_dir: str, out_dir: str) -> None:
+    out = Path(out_dir)
+    rows = []
+    for tgt in PAIRS:
+        refs = _lines(Path(flores_dir, f"{tgt}.devtest"))
+        upstream = _lines(out / f"ref.eng_Latn-{tgt}.txt")
+        ours = _lines(out / f"ours.eng_Latn-{tgt}.txt")
+        assert len(refs) == len(upstream) == len(ours) == SENTENCES, tgt
+        chrf_upstream = sacrebleu.corpus_chrf(upstream, [refs]).score
+        chrf_ours = sacrebleu.corpus_chrf(ours, [refs]).score
+        identical = sum(a == b for a, b in zip(upstream, ours))
+        rows.append(
+            {
+                "pair": f"eng_Latn-{tgt}",
+                "chrf_reference": round(chrf_upstream, 2),
+                "chrf_bhashantar": round(chrf_ours, 2),
+                "delta": round(chrf_ours - chrf_upstream, 2),
+                "identical_outputs": f"{identical}/{SENTENCES}",
+                "pass": abs(chrf_ours - chrf_upstream) <= PASS_BAR,
+            }
+        )
+    Path(out, "parity.json").write_text(json.dumps(rows, indent=2), encoding="utf-8")
+    print("| pair | chrF reference | chrF bhashantar | delta | identical outputs | pass |")
+    print("|---|---|---|---|---|---|")
+    for r in rows:
+        print(
+            f"| {r['pair']} | {r['chrf_reference']} | {r['chrf_bhashantar']} | {r['delta']:+} "
+            f"| {r['identical_outputs']} | {'yes' if r['pass'] else 'NO'} |"
+        )
+
+
+if __name__ == "__main__":
+    {"run": run, "score": score}[sys.argv[1]](*sys.argv[2:4])

bhashantar-0.1.0/bench/reference.py

@@ -0,0 +1,72 @@
+"""Reference translations from the upstream stack, for the parity table.
+
+Runs only in the pinned reference venv (torch, transformers==4.50.3,
+IndicTransToolkit 1.1.1), because that stack does not load on current
+transformers. Writes one hypothesis file per pair plus timing and peak RSS.
+
+    .venv-ref/Scripts/python bench/reference.py MODEL_DIR FLORES_DEVTEST_DIR OUT_DIR [tgt,tgt] [batch]
+
+One process per pair keeps peak memory to a single pair and gives each pair
+its own peak RSS figure.
+"""
+
+import json
+import sys
+import time
+from pathlib import Path
+
+import psutil
+import torch
+from IndicTransToolkit.processor import IndicProcessor
+from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
+
+PAIRS = ["hin_Deva", "ben_Beng", "tam_Taml"]
+SENTENCES = 100
+BATCH = 8
+
+
+def main(model_dir: str, flores_dir: str, out_dir: str, pairs: str = ",".join(PAIRS), batch: str = str(BATCH)) -> None:
+    step = int(batch)
+    torch.set_num_threads(4)
+    out = Path(out_dir)
+    out.mkdir(parents=True, exist_ok=True)
+    tokenizer = AutoTokenizer.from_pretrained(model_dir, trust_remote_code=True)
+    model = AutoModelForSeq2SeqLM.from_pretrained(model_dir, trust_remote_code=True, low_cpu_mem_usage=True).eval()
+    processor = IndicProcessor(inference=True)
+    source = Path(flores_dir, "eng_Latn.devtest").read_text(encoding="utf-8").splitlines()[:SENTENCES]
+    stats = {}
+    for tgt in pairs.split(","):
+        # Hypotheses are appended per batch so a run killed by memory pressure
+        # resumes where it stopped; timing then covers the last run only.
+        partial = Path(out, f"ref.eng_Latn-{tgt}.partial")
+        hyps = partial.read_text(encoding="utf-8").splitlines() if partial.exists() else []
+        started, done_before = time.perf_counter(), len(hyps)
+        for i in range(len(hyps), len(source), step):
+            chunk = processor.preprocess_batch(source[i : i + step], src_lang="eng_Latn", tgt_lang=tgt)
+            enc = tokenizer(chunk, truncation=True, padding="longest", return_tensors="pt")
+            with torch.inference_mode():
+                ids = model.generate(**enc, num_beams=5, max_length=256, min_length=0, use_cache=True)
+            decoded = tokenizer.batch_decode(ids, skip_special_tokens=True, clean_up_tokenization_spaces=True)
+            batch_hyps = processor.postprocess_batch(decoded, lang=tgt)
+            with partial.open("a", encoding="utf-8") as fh:
+                fh.write("".join(h + "\n" for h in batch_hyps))
+            hyps += batch_hyps
+        elapsed = time.perf_counter() - started
+        Path(out, f"ref.eng_Latn-{tgt}.txt").write_text("\n".join(hyps) + "\n", encoding="utf-8")
+        partial.unlink()
+        timed = len(source) - done_before
+        stats[tgt] = {
+            "batch": step,
+            "timed_sentences": timed,
+            "seconds": round(elapsed, 2),
+            "ms_per_sentence": round(1000 * elapsed / max(timed, 1), 1),
+        }
+        print(tgt, stats[tgt], flush=True)
+    # peak_wset is the Windows peak working set; this harness ran on Windows.
+    stats["peak_rss_mb"] = round(psutil.Process().memory_info().peak_wset / 2**20)
+    Path(out, f"ref.stats.{pairs.replace(',', '+')}.json").write_text(json.dumps(stats, indent=2), encoding="utf-8")
+    print(stats)
+
+
+if __name__ == "__main__":
+    main(*sys.argv[1:6])