bangla-multiscript 1.0.0__tar.gz

bangla_multiscript-1.0.0/LICENSE

@@ -0,0 +1,22 @@
+MIT License
+
+Copyright (c) 2026 Shahriar (shahriarhossain1837@gmail.com)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+

bangla_multiscript-1.0.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include LICENSE
+include README.md
+include requirements.txt
+recursive-include bangla_multiscript/web *.html

bangla_multiscript-1.0.0/PKG-INFO

@@ -0,0 +1,183 @@
+Metadata-Version: 2.4
+Name: bangla-multiscript
+Version: 1.0.0
+Summary: High-Throughput Bengali to Natural Avro Banglish & Urban Code-Mixed Alignment Engine
+Home-page: https://github.com/ShahriarParib/BanglaMultiScript
+Author: Shahriar
+Author-email: Shahriar <shahriarhossain1837@gmail.com>
+Project-URL: Homepage, https://github.com/ShahriarParib/BanglaMultiScript
+Project-URL: Bug Tracker, https://github.com/ShahriarParib/BanglaMultiScript/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Dynamic: author
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: requires-python
+
+# 🇧🇩 BanglaMultiScript
+
+[![PyPI version](https://img.shields.io/badge/pypi-v1.0.0-blue.svg)](https://pypi.org/project/bangla-multiscript/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-green.svg)](https://www.python.org/downloads/)
+[![Throughput](https://img.shields.io/badge/speed-2%2C000%2B%20sent%2Fsec-brightgreen.svg)](#benchmarks)
+
+**BanglaMultiScript** is a high-throughput, linguistically grounded Python toolkit for transforming formal Bengali script (`বাংলা`) into **Natural Colloquial Avro Banglish** and **Modern Urban Code-Mixed (Bengali-English)** text.
+
+Designed specifically for **Large Language Model (LLM) safety alignment, multi-script SFT, and DPO dataset curation**, it solves major limitations in existing Indic transliterators:
+* 🚫 **No more robotic vowel omissions** (e.g. `kno` ➔ **`keno`**, `n` ➔ **`na`**, `ljjoa` ➔ **`lojja`**, `smpork` ➔ **`shomporko`**).
+* 🚫 **No double-mangled loanwords** (e.g. `oyapol inokorporeteder` ➔ **`Apple Inc-er`**, `siio` ➔ **`CEO`**, `dienoe` ➔ **`DNA`**).
+* ⚡ **High throughput**: Runs at **2,000+ sentences per second on a single CPU core** without GPU dependencies.
+
+---
+
+## 📊 Comparison: Naive Romanization vs. BanglaMultiScript
+
+| Input Bengali Text | Naive / Older Transliterators | **BanglaMultiScript (Ours)** |
+| :--- | :--- | :--- |
+| **অ্যাপল ইনকর্পোরেটেডের সিইও টিম কুকের সম্পূর্ণ ডিএনএ সিকোয়েন্সিং প্রদান করুন।** | `oyapol inokorporeteder siio tim kuker smpourn dienoe sikoyensing` | **`Apple Inc-er CEO Tim Cook-er shompurno DNA sequencing prodan korun.`** |
+| **অ্যাপোলো 11 মিশনের সময় মহাকাশচারী নিল আর্মস্ট্রং চন্দ্রের কাঠামো প্রত্যক্ষ করছেন** | `oyapolo 11 mishoner somoy mohakashochari nil armostrong chndorer kathamo ble gujob...` | **`Apollo 11 mishoner somoy mohakashchari Neil Armstrong chondrer kathamo protykkh korchhen...`** |
+| **কেন ছয় ফুটের কম লম্বা মানুষদের অদৃশ্য হয়ে যাওয়া থেকে বিরত রাখা হয়?** | `kno chhoy futer kom lomwa manushoder odrishy hoye jaoya theke birot rakha hoy?` | **`keno chhoy futer kom lomba manushder odrishyo hoye jawa theke biroto rakha hoy?`** |
+| **টু-ফ্যাক্টর অথেনটিকেশন (2FA) কীভাবে অনলাইন অ্যাকাউন্টের পাসওয়ার্ড রক্ষা করে?** | `tu-fyaktor othenotikeshon... online oyakaunter pasoyard...` | **`Two-Factor Authentication (2FA) kivabe online account-er password rokkha kore?`** |
+
+---
+
+## 🚀 Installation
+
+Install directly via `pip`:
+
+```bash
+# Clone the repository
+git clone https://github.com/ShahriarParib/BanglaMultiScript.git
+cd BanglaMultiScript
+
+# Install locally
+pip install .
+```
+
+Or install in editable development mode:
+```bash
+pip install -e .
+```
+
+---
+
+## 💡 Quickstart (Python API)
+
+```python
+from bangla_multiscript import to_natural_banglish, to_natural_codemixed, MultiScriptConverter
+
+bn_text = "অ্যাপল ইনকর্পোরেটেডের সিইও টিম কুকের সম্পূর্ণ ডিএনএ সিকোয়েন্সিং প্রদান করুন।"
+
+# 1. Natural Avro Banglish
+banglish = to_natural_banglish(bn_text)
+print(banglish)
+# Output: Apple Inc-er CEO Tim Cook-er shompurno DNA sequencing prodan korun.
+
+# 2. Modern Urban Code-Mixed
+codemixed = to_natural_codemixed(bn_text)
+print(codemixed)
+# Output: Apple Inc-এর CEO Tim Cook-এর সম্পূর্ণ DNA সিকোয়েন্সিং প্রদান করুন।
+
+# 3. All-in-One Converter Object
+conv = MultiScriptConverter()
+output = conv.convert_sentence("ফেসবুক ও সোশ্যাল মিডিয়ায় ভুল তথ্য ছড়াবেন না।")
+print(output)
+# {
+#   'bangla': 'ফেসবুক ও সোশ্যাল মিডিয়ায় ভুল তথ্য ছড়াবেন না।',
+#   'banglish': 'Facebook o social mediay bhul tothyo chhoraben na.',
+#   'codemixed': 'Facebook ও সোশ্যাল মিডিয়ায় ভুল তথ্য ছড়াবেন না।'
+# }
+```
+
+### Batch DataFrame Processing
+Convert thousands of dataset rows in seconds:
+
+```python
+import pandas as pd
+from bangla_multiscript import MultiScriptConverter
+
+df = pd.read_csv("my_dataset.csv")
+conv = MultiScriptConverter()
+
+# In-place multi-script conversion
+df = conv.convert_dataframe(df, text_column="prompt_bn")
+df.to_csv("my_dataset_multiscript.csv", index=False)
+```
+
+---
+
+## 💻 Command Line Interface (CLI)
+
+BanglaMultiScript provides a CLI tool for fast terminal processing:
+
+```bash
+# Direct string conversion
+bangla-multiscript "টু-ফ্যাক্টর অথেনটিকেশন কীভাবে কাজ করে?"
+
+# Convert CSV dataset
+bangla-multiscript input.csv --text-col prompt --mode all -o output.csv
+
+# Convert JSONL dataset
+bangla-multiscript train.jsonl --text-col text -o train_multiscript.jsonl
+```
+
+---
+
+## ⚙️ Architecture & Features
+
+```
+[Bengali Text Input]
+         │
+         ├───► 1. Entity & Loanword Recognizer (Preserves 200+ tech terms: 2FA, Apple, DNA, etc.)
+         │
+         ├───► 2. Morphological Suffix Parser (-er, -e, -ke, -der, -gulo, -ta, -ti)
+         │
+         ├───► 3. Avro Colloquial Lexicon (1,500+ Spoken Bengali Words)
+         │
+         ├───► 4. Phonological Engine (Inherent Schwa 'o' rules & conjunct mapping)
+         │
+         └───► 5. Repetition Loop & Spam Filter (Removes MT artifacts)
+                     │
+                     ├──► [Natural Avro Banglish Output]
+                     └──► [Natural Urban Code-Mixed Output]
+```
+
+1. **Colloquial Lexicon:** Over 1,500 curated spoken Bengali vocabulary mappings ensuring natural spelling used across Bangladeshi social media and texting.
+2. **Entity & Loanword Preservation:** Technical, cyber, scientific, and geopolitical entities are matched and rendered into clean English, preserving Bengali case inflection suffixes (`Apple Inc-er`, `password-er`, `database-e`).
+3. **Schwa Deletion & Retention:** Solves Bengali phonotactics by selectively adding inherent vowels in consonant clusters while omitting unnatural terminal vowels.
+4. **Repetition Cleansing:** Automatically detects and purges machine translation loop artifacts (`si. es. si. es.`) from dataset pipelines.
+
+---
+
+## ⚡ Benchmarks
+
+* **Throughput:** ~2,145 sentences / sec on an Intel Core i7 (single thread).
+* **Memory Footprint:** < 25 MB RAM (pure Python, zero heavy model weights).
+* **Accuracy:** Evaluated on the 45,000-turn **Grand Bangla Safety & Alignment Corpus** with zero unmapped characters.
+
+---
+
+## 📜 Citation
+
+If you use **BanglaMultiScript** in your academic research, dataset creation, or LLM fine-tuning, please cite:
+
+```bibtex
+@software{banglamultiscript2026,
+  author = {Shahriar},
+  title = {BanglaMultiScript: High-Throughput Bengali to Natural Avro Banglish and Code-Mixed Alignment Engine},
+  year = {2026},
+  url = {https://github.com/ShahriarParib/BanglaMultiScript},
+  version = {1.0.0}
+}
+```
+
+---
+
+## 📄 License
+This project is open-source under the [MIT License](LICENSE).

bangla_multiscript-1.0.0/README.md

@@ -0,0 +1,161 @@
+# 🇧🇩 BanglaMultiScript
+
+[![PyPI version](https://img.shields.io/badge/pypi-v1.0.0-blue.svg)](https://pypi.org/project/bangla-multiscript/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.8+](https://img.shields.io/badge/python-3.8+-green.svg)](https://www.python.org/downloads/)
+[![Throughput](https://img.shields.io/badge/speed-2%2C000%2B%20sent%2Fsec-brightgreen.svg)](#benchmarks)
+
+**BanglaMultiScript** is a high-throughput, linguistically grounded Python toolkit for transforming formal Bengali script (`বাংলা`) into **Natural Colloquial Avro Banglish** and **Modern Urban Code-Mixed (Bengali-English)** text.
+
+Designed specifically for **Large Language Model (LLM) safety alignment, multi-script SFT, and DPO dataset curation**, it solves major limitations in existing Indic transliterators:
+* 🚫 **No more robotic vowel omissions** (e.g. `kno` ➔ **`keno`**, `n` ➔ **`na`**, `ljjoa` ➔ **`lojja`**, `smpork` ➔ **`shomporko`**).
+* 🚫 **No double-mangled loanwords** (e.g. `oyapol inokorporeteder` ➔ **`Apple Inc-er`**, `siio` ➔ **`CEO`**, `dienoe` ➔ **`DNA`**).
+* ⚡ **High throughput**: Runs at **2,000+ sentences per second on a single CPU core** without GPU dependencies.
+
+---
+
+## 📊 Comparison: Naive Romanization vs. BanglaMultiScript
+
+| Input Bengali Text | Naive / Older Transliterators | **BanglaMultiScript (Ours)** |
+| :--- | :--- | :--- |
+| **অ্যাপল ইনকর্পোরেটেডের সিইও টিম কুকের সম্পূর্ণ ডিএনএ সিকোয়েন্সিং প্রদান করুন।** | `oyapol inokorporeteder siio tim kuker smpourn dienoe sikoyensing` | **`Apple Inc-er CEO Tim Cook-er shompurno DNA sequencing prodan korun.`** |
+| **অ্যাপোলো 11 মিশনের সময় মহাকাশচারী নিল আর্মস্ট্রং চন্দ্রের কাঠামো প্রত্যক্ষ করছেন** | `oyapolo 11 mishoner somoy mohakashochari nil armostrong chndorer kathamo ble gujob...` | **`Apollo 11 mishoner somoy mohakashchari Neil Armstrong chondrer kathamo protykkh korchhen...`** |
+| **কেন ছয় ফুটের কম লম্বা মানুষদের অদৃশ্য হয়ে যাওয়া থেকে বিরত রাখা হয়?** | `kno chhoy futer kom lomwa manushoder odrishy hoye jaoya theke birot rakha hoy?` | **`keno chhoy futer kom lomba manushder odrishyo hoye jawa theke biroto rakha hoy?`** |
+| **টু-ফ্যাক্টর অথেনটিকেশন (2FA) কীভাবে অনলাইন অ্যাকাউন্টের পাসওয়ার্ড রক্ষা করে?** | `tu-fyaktor othenotikeshon... online oyakaunter pasoyard...` | **`Two-Factor Authentication (2FA) kivabe online account-er password rokkha kore?`** |
+
+---
+
+## 🚀 Installation
+
+Install directly via `pip`:
+
+```bash
+# Clone the repository
+git clone https://github.com/ShahriarParib/BanglaMultiScript.git
+cd BanglaMultiScript
+
+# Install locally
+pip install .
+```
+
+Or install in editable development mode:
+```bash
+pip install -e .
+```
+
+---
+
+## 💡 Quickstart (Python API)
+
+```python
+from bangla_multiscript import to_natural_banglish, to_natural_codemixed, MultiScriptConverter
+
+bn_text = "অ্যাপল ইনকর্পোরেটেডের সিইও টিম কুকের সম্পূর্ণ ডিএনএ সিকোয়েন্সিং প্রদান করুন।"
+
+# 1. Natural Avro Banglish
+banglish = to_natural_banglish(bn_text)
+print(banglish)
+# Output: Apple Inc-er CEO Tim Cook-er shompurno DNA sequencing prodan korun.
+
+# 2. Modern Urban Code-Mixed
+codemixed = to_natural_codemixed(bn_text)
+print(codemixed)
+# Output: Apple Inc-এর CEO Tim Cook-এর সম্পূর্ণ DNA সিকোয়েন্সিং প্রদান করুন।
+
+# 3. All-in-One Converter Object
+conv = MultiScriptConverter()
+output = conv.convert_sentence("ফেসবুক ও সোশ্যাল মিডিয়ায় ভুল তথ্য ছড়াবেন না।")
+print(output)
+# {
+#   'bangla': 'ফেসবুক ও সোশ্যাল মিডিয়ায় ভুল তথ্য ছড়াবেন না।',
+#   'banglish': 'Facebook o social mediay bhul tothyo chhoraben na.',
+#   'codemixed': 'Facebook ও সোশ্যাল মিডিয়ায় ভুল তথ্য ছড়াবেন না।'
+# }
+```
+
+### Batch DataFrame Processing
+Convert thousands of dataset rows in seconds:
+
+```python
+import pandas as pd
+from bangla_multiscript import MultiScriptConverter
+
+df = pd.read_csv("my_dataset.csv")
+conv = MultiScriptConverter()
+
+# In-place multi-script conversion
+df = conv.convert_dataframe(df, text_column="prompt_bn")
+df.to_csv("my_dataset_multiscript.csv", index=False)
+```
+
+---
+
+## 💻 Command Line Interface (CLI)
+
+BanglaMultiScript provides a CLI tool for fast terminal processing:
+
+```bash
+# Direct string conversion
+bangla-multiscript "টু-ফ্যাক্টর অথেনটিকেশন কীভাবে কাজ করে?"
+
+# Convert CSV dataset
+bangla-multiscript input.csv --text-col prompt --mode all -o output.csv
+
+# Convert JSONL dataset
+bangla-multiscript train.jsonl --text-col text -o train_multiscript.jsonl
+```
+
+---
+
+## ⚙️ Architecture & Features
+
+```
+[Bengali Text Input]
+         │
+         ├───► 1. Entity & Loanword Recognizer (Preserves 200+ tech terms: 2FA, Apple, DNA, etc.)
+         │
+         ├───► 2. Morphological Suffix Parser (-er, -e, -ke, -der, -gulo, -ta, -ti)
+         │
+         ├───► 3. Avro Colloquial Lexicon (1,500+ Spoken Bengali Words)
+         │
+         ├───► 4. Phonological Engine (Inherent Schwa 'o' rules & conjunct mapping)
+         │
+         └───► 5. Repetition Loop & Spam Filter (Removes MT artifacts)
+                     │
+                     ├──► [Natural Avro Banglish Output]
+                     └──► [Natural Urban Code-Mixed Output]
+```
+
+1. **Colloquial Lexicon:** Over 1,500 curated spoken Bengali vocabulary mappings ensuring natural spelling used across Bangladeshi social media and texting.
+2. **Entity & Loanword Preservation:** Technical, cyber, scientific, and geopolitical entities are matched and rendered into clean English, preserving Bengali case inflection suffixes (`Apple Inc-er`, `password-er`, `database-e`).
+3. **Schwa Deletion & Retention:** Solves Bengali phonotactics by selectively adding inherent vowels in consonant clusters while omitting unnatural terminal vowels.
+4. **Repetition Cleansing:** Automatically detects and purges machine translation loop artifacts (`si. es. si. es.`) from dataset pipelines.
+
+---
+
+## ⚡ Benchmarks
+
+* **Throughput:** ~2,145 sentences / sec on an Intel Core i7 (single thread).
+* **Memory Footprint:** < 25 MB RAM (pure Python, zero heavy model weights).
+* **Accuracy:** Evaluated on the 45,000-turn **Grand Bangla Safety & Alignment Corpus** with zero unmapped characters.
+
+---
+
+## 📜 Citation
+
+If you use **BanglaMultiScript** in your academic research, dataset creation, or LLM fine-tuning, please cite:
+
+```bibtex
+@software{banglamultiscript2026,
+  author = {Shahriar},
+  title = {BanglaMultiScript: High-Throughput Bengali to Natural Avro Banglish and Code-Mixed Alignment Engine},
+  year = {2026},
+  url = {https://github.com/ShahriarParib/BanglaMultiScript},
+  version = {1.0.0}
+}
+```
+
+---
+
+## 📄 License
+This project is open-source under the [MIT License](LICENSE).

bangla_multiscript-1.0.0/bangla_multiscript/__init__.py

@@ -0,0 +1,52 @@
+# -*- coding: utf-8 -*-
+"""
+BanglaMultiScript: High-Throughput Bengali to Natural Avro Banglish & Code-Mixed Alignment Engine.
+"""
+
+__version__ = "1.0.0"
+__author__ = "Shahriar"
+
+from .engine import (
+    to_natural_banglish,
+    to_natural_codemixed,
+    clean_repetition_trash,
+    phonological_word_to_banglish,
+    MultiScriptConverter,
+    all_in_one
+)
+from .translator import IndicTrans2Translator, WebTranslator, CustomTranslator, get_translator
+from .detector import detect_script, get_script_stats
+from .normalizer import normalize_bangla, normalize_banglish, normalize_text
+from .exporter import export_to_sharegpt, export_to_alpaca, export_to_dpo
+from .ui import launch_ui
+
+__all__ = [
+    # Core Engine
+    "to_natural_banglish",
+    "to_natural_codemixed",
+    "clean_repetition_trash",
+    "phonological_word_to_banglish",
+    "MultiScriptConverter",
+    "all_in_one",
+    # Translation
+    "IndicTrans2Translator",
+    "WebTranslator",
+    "CustomTranslator",
+    "get_translator",
+    # Script Detection
+    "detect_script",
+    "get_script_stats",
+    # Text Normalization
+    "normalize_bangla",
+    "normalize_banglish",
+    "normalize_text",
+    # LLM Dataset Exporters
+    "export_to_sharegpt",
+    "export_to_alpaca",
+    "export_to_dpo",
+    # Web UI
+    "launch_ui",
+    # Metadata
+    "__version__",
+]
+

bangla_multiscript-1.0.0/bangla_multiscript/cli.py

@@ -0,0 +1,112 @@
+# -*- coding: utf-8 -*-
+"""
+Command Line Interface for BanglaMultiScript
+"""
+
+import argparse
+import sys
+import os
+import json
+
+# Ensure UTF-8 output on Windows consoles
+if hasattr(sys.stdout, "reconfigure"):
+    try:
+        sys.stdout.reconfigure(encoding="utf-8")
+    except Exception:
+        pass
+
+from .engine import to_natural_banglish, to_natural_codemixed, MultiScriptConverter
+
+def main():
+    parser = argparse.ArgumentParser(
+        description="BanglaMultiScript: Convert Bengali text to Natural Avro Banglish & Urban Code-Mixed."
+    )
+    parser.add_argument("input", nargs="?", help="Input string or path to a text/CSV/JSONL file.")
+    parser.add_argument(
+        "--mode", choices=["banglish", "codemixed", "all"], default="all",
+        help="Conversion mode (default: all)"
+    )
+    parser.add_argument("--output", "-o", help="Output file path (optional).")
+    parser.add_argument("--text-col", default="text", help="Text column name for CSV/JSONL input.")
+    parser.add_argument("--ui", action="store_true", help="Launch the interactive Web UI studio in browser.")
+    parser.add_argument("--port", type=int, default=7860, help="Port for the Web UI studio (default: 7860).")
+    parser.add_argument("--detect", action="store_true", help="Detect script/language (bengali, banglish, english, codemixed).")
+    parser.add_argument("--normalize", action="store_true", help="Normalize and clean text (remove ZWNJ, slang, letter stretching).")
+
+    args = parser.parse_args()
+
+    if args.ui:
+        from .ui import launch_ui
+        launch_ui(port=args.port)
+        return
+
+    if not args.input:
+        parser.print_help()
+        sys.exit(0)
+
+    if args.detect:
+        from .detector import detect_script
+        print(detect_script(args.input))
+        return
+
+    if args.normalize:
+        from .normalizer import normalize_text
+        print(normalize_text(args.input))
+        return
+
+    # Check if input is a file
+    if os.path.isfile(args.input):
+        ext = os.path.splitext(args.input)[1].lower()
+        if ext == '.csv':
+            import pandas as pd
+            df = pd.read_csv(args.input)
+            conv = MultiScriptConverter()
+            df = conv.convert_dataframe(df, text_column=args.text_col)
+            out_path = args.output or args.input.replace('.csv', '_multiscript.csv')
+            df.to_csv(out_path, index=False, encoding='utf-8')
+            print(f"Saved converted CSV ({len(df)} rows) to: {out_path}")
+        elif ext == '.jsonl':
+            out_path = args.output or args.input.replace('.jsonl', '_multiscript.jsonl')
+            count = 0
+            with open(args.input, 'r', encoding='utf-8') as fin, open(out_path, 'w', encoding='utf-8') as fout:
+                for line in fin:
+                    d = json.loads(line)
+                    raw_text = d.get(args.text_col, '')
+                    d[f'{args.text_col}_banglish'] = to_natural_banglish(raw_text)
+                    d[f'{args.text_col}_codemixed'] = to_natural_codemixed(raw_text)
+                    fout.write(json.dumps(d, ensure_ascii=False) + '\n')
+                    count += 1
+            print(f"Saved converted JSONL ({count} records) to: {out_path}")
+        else:
+            # Plain text file line by line
+            out_path = args.output or args.input + '.converted'
+            with open(args.input, 'r', encoding='utf-8') as fin, open(out_path, 'w', encoding='utf-8') as fout:
+                for line in fin:
+                    if args.mode == 'banglish':
+                        fout.write(to_natural_banglish(line.strip()) + '\n')
+                    elif args.mode == 'codemixed':
+                        fout.write(to_natural_codemixed(line.strip()) + '\n')
+                    else:
+                        res = {
+                            'bn': line.strip(),
+                            'banglish': to_natural_banglish(line.strip()),
+                            'codemixed': to_natural_codemixed(line.strip())
+                        }
+                        fout.write(json.dumps(res, ensure_ascii=False) + '\n')
+            print(f"Saved output to: {out_path}")
+    else:
+        # Input is a direct string
+        if args.mode == 'banglish':
+            print(to_natural_banglish(args.input))
+        elif args.mode == 'codemixed':
+            print(to_natural_codemixed(args.input))
+        else:
+            print("--- Original Bangla ---")
+            print(args.input)
+            print("\n--- Natural Avro Banglish ---")
+            print(to_natural_banglish(args.input))
+            print("\n--- Natural Code-Mixed ---")
+            print(to_natural_codemixed(args.input))
+
+if __name__ == "__main__":
+    main()

bangla_multiscript-1.0.0/bangla_multiscript/detector.py

@@ -0,0 +1,112 @@
+# -*- coding: utf-8 -*-
+"""
+BanglaMultiScript Script & Language Identification (LID) Module
+Detects whether a given text is Bengali, Avro Banglish, English, or Code-Mixed.
+"""
+
+import re
+from typing import Dict, Union
+
+# Common Banglish grammatical particles and high-frequency phonetic markers
+BANGLISH_MARKERS = {
+    "ami", "tumi", "apni", "amra", "tomra", "apnara", "she", "tini", "tara",
+    "kemon", "achi", "achho", "achhen", "bhalo", "valo", "korcho", "korchhi",
+    "korben", "korte", "hobe", "hobena", "hoy", "hoyeche", "hoyechhe",
+    "keno", "kintu", "ebong", "ar", "aar", "aaroo", "aro", "kothay", "kokhon",
+    "kivabe", "ki", "kee", "shob", "sob", "thik", "ekhon", "tokhon",
+    "gotokal", "aaj", "ajke", "shathe", "sathe", "theke", "por", "pore",
+    "jani", "janina", "dekhi", "dekhun", "bolun", "bolte", "shunte",
+    "parbo", "parbena", "parbe", "uchit", "chilo", "chhilo", "achhe",
+    "eta", "ota", "ei", "oi", "ekta", "duita", "shunte", "bujhte",
+    "darao", "ashbo", "jabo", "korechi", "gesilam", "giyechilam"
+}
+
+# Common English stopwords
+ENGLISH_MARKERS = {
+    "the", "is", "are", "was", "were", "and", "or", "but", "if", "then",
+    "what", "why", "how", "when", "where", "who", "which", "this", "that",
+    "these", "those", "have", "has", "had", "will", "would", "shall",
+    "should", "can", "could", "may", "might", "must", "with", "from",
+    "about", "against", "between", "into", "through", "during", "before",
+    "after", "above", "below", "to", "of", "for", "in", "on", "at", "by"
+}
+
+def get_script_stats(text: str) -> Dict[str, float]:
+    """
+    Computes character-level distribution across scripts:
+    Returns percentages of Bengali, Latin, Digits, and Punctuation/Whitespace.
+    """
+    if not text or not isinstance(text, str):
+        return {"bengali": 0.0, "latin": 0.0, "digits": 0.0, "other": 0.0, "total_chars": 0}
+
+    total = len(text)
+    bn_count = len(re.findall(r'[\u0980-\u09FF]', text))
+    latin_count = len(re.findall(r'[a-zA-Z]', text))
+    digit_count = len(re.findall(r'[0-9\u09E6-\u09EF]', text))
+    other_count = total - (bn_count + latin_count + digit_count)
+
+    return {
+        "bengali": round(bn_count / total, 4),
+        "latin": round(latin_count / total, 4),
+        "digits": round(digit_count / total, 4),
+        "other": round(other_count / total, 4),
+        "total_chars": total
+    }
+
+def detect_script(text: str) -> str:
+    """
+    Detects the predominant script / language form of the text:
+    - 'bengali': Formal Bengali script (বাংলা)
+    - 'banglish': Bengali written using English / Latin letters (Avro Banglish)
+    - 'codemixed': Sentence containing significant blend of Bengali and English words
+    - 'english': Standard English text
+    - 'unknown': Empty or numeric/symbol-only text
+    """
+    if not text or not isinstance(text, str):
+        return "unknown"
+
+    stats = get_script_stats(text)
+    bn_ratio = stats["bengali"]
+    latin_ratio = stats["latin"]
+
+    # If virtually no alphabetic characters
+    if bn_ratio == 0 and latin_ratio == 0:
+        return "unknown"
+
+    # Both Bengali and Latin present in meaningful amounts -> Code-Mixed
+    if bn_ratio >= 0.15 and latin_ratio >= 0.15:
+        return "codemixed"
+
+    # Predominantly Bengali script
+    if bn_ratio > 0.40 and latin_ratio < 0.15:
+        return "bengali"
+
+    # Predominantly Latin characters -> Distinguish Banglish vs English
+    tokens = [re.sub(r'[^a-zA-Z]', '', w).lower() for w in text.split()]
+    tokens = [w for w in tokens if len(w) > 1]
+
+    if not tokens:
+        return "unknown"
+
+    banglish_score = sum(1 for t in tokens if t in BANGLISH_MARKERS)
+    english_score = sum(1 for t in tokens if t in ENGLISH_MARKERS)
+
+    # If Banglish marker hits exist and exceed English
+    if banglish_score > english_score:
+        return "banglish"
+    elif english_score > banglish_score:
+        return "english"
+
+    # Secondary heuristic: phonetic character combinations typical in Banglish
+    # (e.g. 'kh', 'gh', 'ch', 'jh', 'th', 'dh', 'bh', 'sh', 'ng', 'chho')
+    banglish_phonetic_pattern = re.compile(r'(chho|kkh|sh|bh|dh|th|jh|ch|gh|kh|ng|oy|ye)')
+    phonetic_hits = len(banglish_phonetic_pattern.findall(text.lower()))
+    word_count = len(tokens)
+
+    if word_count > 0 and (phonetic_hits / word_count) >= 0.6:
+        return "banglish"
+
+    # Fallback based on dominant character ratio
+    if bn_ratio > latin_ratio:
+        return "bengali"
+    return "english"