shobdosearch 2.0.0__tar.gz

shobdosearch-2.0.0/ARCHITECTURE.md

@@ -0,0 +1,69 @@
+# System Architecture: Banglish-to-Bangla NLP Engine
+
+This document outlines the technical design, architectural components, and data flow of the ShobdoSearch transliteration engine.
+
+---
+
+## 1. End-to-End Pipeline
+
+The engine combines fast-path seed caching, recursive candidate generation, dynamic vowel-expansion for shorthand words, and weighted multi-tier dictionary scoring.
+
+```mermaid
+graph TD
+    A[Input: 'tmi kmn aso?'] --> B[Token & Punctuation Splitter]
+    B --> C{In Seed Map or In-Memory Cache?}
+    C -- Yes --> D[Instant O-1 Return]
+    C -- No --> E[Longest-Match Phonetic Splitter]
+    E --> F[Recursive Candidate Generator]
+    F --> G{Strong Dict Match Found?}
+    G -- Yes --> H[Select Lowest-Weight Word]
+    G -- No --> I[Dynamic Vowel-Expansion Engine]
+    I --> J[Generate Shorthand Permutations]
+    J --> K[Weighted Multi-Tier Dict Scorer 464k+ Words]
+    K --> L[Select Best Valid Bangla Word]
+    H --> M[Store in b2b_cache & Return]
+    L --> M
+    D --> N[Assemble Sentence & Return]
+    M --> N
+```
+
+---
+
+## 2. Core Architectural Components
+
+### A. Fast-Path Base Map (`ben2bn.csv`) & In-Memory Cache (`b2b_cache`)
+- **Base Seed Map**: Contains 2,770+ curated, alphabetically sorted high-frequency words covering 80%+ daily conversational Bengali.
+- **In-Memory Cache**: Dynamically stores transliterated words during runtime to provide $O(1)$ response time for repeated tokens without disk writes.
+
+### B. Longest-Match Phonetic Splitter
+- Sorts phonetic keys in `banGenerator.csv` by length descending to match multi-character phonemes (`kh`, `sh`, `th`, `ch`, `gh`, `dh`, `bh`, `ph`, `ng`, `nd`, `st`) before single letters (`k`, `s`, `t`).
+
+### C. Recursive Candidate Generator with Diacritic Filtering
+- Explores valid character substitutions per phoneme.
+- Enforces strict orthographic rules:
+  - **Dependent Vowels Filter**: Prevents standalone vowel diacritics (kars like `া`, `ে`, `ি`) from appearing at word beginnings.
+  - **Implicit Vowel Handling**: Handles implicit vowels (`a`/`o` $\rightarrow$ `অ` / `""`) while preserving explicit hasantas (`্`).
+  - **Conjunct Formation**: Automatically generates conjuncts (`যুক্তবর্ণ`) between consonant clusters.
+
+### D. Dynamic Vowel-Expansion Engine
+- Recovers informal chat abbreviations and consonant skeletons (e.g. `tmi`, `vlo`, `kmn`, `apnr`, `bndhu`, `rsta`).
+- Detects adjacent consonants, interpolates candidate vowels (`a`, `o`, `e`, `u`, `i`), and scores them against the dictionary.
+
+### E. Weighted Multi-Tier Dictionary Validator
+- Validates candidates against **464,411 words** across 4 frequency tiers:
+  - Tier 1: Core vocabulary (40k words) - Weight 1
+  - Tier 2: Intermediate vocabulary (48k words) - Weight 2
+  - Tier 3: Large vocabulary (112k words) - Weight 3
+  - Tier 4: Comprehensive lexicon (439k words) - Weight 4
+- Ranks candidate words by priority score with penalties for single-character relics and trailing hasantas.
+
+---
+
+## 3. Data Structures & Performance
+
+| Component | Implementation | Complexity | Purpose |
+| :--- | :--- | :--- | :--- |
+| `b2b_map` | Python `dict` | $O(1)$ | 2,770+ high-frequency seed words |
+| `b2b_cache` | Python `dict` | $O(1)$ | In-memory session cache |
+| `generator_map` | Python `dict[str, list[str]]` | $O(L)$ | 111 phoneme rules (7-column aligned) |
+| `word_weights` | Python `dict[str, int]` | $O(1)$ | 464,411 weighted dictionary words |

shobdosearch-2.0.0/CHANGELOG.md

@@ -0,0 +1,41 @@
+# Changelog
+
+All notable changes to this project are documented in this file.
+
+---
+
+## [1.0.0] - 2026-08-22
+
+### Added
+- **Dynamic Vowel-Expansion Engine**: Algorithmic recovery for consonant-heavy chat shorthand and abbreviations (`tmi`, `vlo`, `kmn`, `amr`, `tmr`, `apnr`, `bndhu`, `rsta`, `khbr`).
+- **Comprehensive Base Vocabulary**: Expanded and alphabetically sorted [`data/ben2bn.csv`](file:///c:/Users/Khalid/OneDrive/Desktop/Git%20clone/Antigravity/ShobdoSearch/data/ben2bn.csv) to **2,770+** high-frequency words covering 80%+ daily conversational Bengali.
+- **Extended Phonetic Rules**: Added missing phonemes (`v`, `f`, `w`, `x`, `z`, `q`) and digraph conjuncts (`bd`, `bdh`, `kt`, `st`) to `data/banGenerator.csv`.
+- **Comprehensive Test Suite**: Added [`verify.py`](file:///c:/Users/Khalid/OneDrive/Desktop/Git%20clone/Antigravity/ShobdoSearch/verify.py) with 41 core test cases (100% pass rate).
+
+### Changed
+- **In-Memory Cache Architecture**: Replaced disk-writing in `converter.py` with in-memory `self.b2b_cache` to prevent dataset pollution during runtime.
+- **CSV Standardization**: Cleaned and standardized `data/banGenerator.csv` to an exact 7-column matrix across all 111 phoneme rows.
+- **Dictionary Priority Scoring**: Refined `get_word_priority` with trailing hasanta and 1-letter word penalties, removing detrimental length tie-breakers.
+
+### Fixed
+- **Leading Dependent Vowels**: Fixed bug causing vowel signs (kars like `া`, `ে`, `ি`) to attach at the beginning of words (e.g. `eta` → `এটা` instead of `েটা`).
+- **Implicit Vowel Dangling Hasanta**: Fixed implicit vowel suppression to preserve explicit hasanta endings.
+- **API Stats Endpoint**: Fixed dictionary size calculation in `/stats` to accurately return the 464,411 loaded dictionary words instead of 0.
+
+---
+
+## [0.2.0] - 2026-05-09
+### Changed
+- Refactored logic from Jupyter Notebook to modular `converter.py`.
+- Implemented recursive candidate generation.
+- Replaced linear dictionary search with set-based lookups.
+
+### Fixed
+- Windows UTF-8 terminal encoding support.
+- Longest-match phoneme splitting for `kh`, `sh`.
+
+---
+
+## [0.1.0] - 2026-05-09
+### Added
+- Initial project structure with `data/` and basic phoneme mappings.

shobdosearch-2.0.0/DATASET.md

@@ -0,0 +1,40 @@
+# Dataset Documentation: ShobdoSearch
+
+This document describes the datasets and corpora used by the ShobdoSearch transliteration engine.
+
+---
+
+## 1. Phonetic Rules Matrix (`banGenerator.csv`)
+The foundational phonetic mapping matrix. Maps Romanized character sequences to corresponding Bengali characters in order of phonetic likelihood.
+- **Structure**: Exactly 7 columns (`EngLit, Ben1, Ben2, Ben3, Ben4, Ben5, Ben6`).
+- **Total Rules**: 111 rules covering vowels, consonants, digraphs (`kh`, `sh`, `th`, `ch`, `gh`, `dh`, `bh`, `ph`, `ng`, `nd`), and conjuncts (`kt`, `bd`, `bdh`, `st`, etc.).
+
+---
+
+## 2. High-Frequency Baseline Dictionary (`ben2bn.csv`)
+A curated, pre-indexed mapping of high-frequency Banglish words to accurate Bangla Unicode text.
+- **Total Words**: **2,770+ unique mappings**.
+- **Organization**: Alphabetically sorted from A to Z (`a` through `z`).
+- **Coverage**: Top 80%+ of everyday spoken, written, and chat-shorthand vocabulary (pronouns, tenses, question words, numbers, adjectives, common nouns).
+
+---
+
+## 3. Weighted Dictionary Corpora (`BengaliWordList_*.txt`)
+Used for candidate validation and frequency-based ranking across **464,411 words**:
+
+| File Name | Word Count | Tier / Weight | Description |
+| :--- | :--- | :---: | :--- |
+| `BengaliWordList_40.txt` | ~40,000 | Weight 1 | Highest-frequency core vocabulary |
+| `BengaliWordList_48.txt` | ~48,000 | Weight 2 | Common everyday conversational words |
+| `BengaliWordList_112.txt` | ~112,000 | Weight 3 | Intermediate and domain-specific terms |
+| `BengaliWordList_439.txt` | ~439,000 | Weight 4 | Comprehensive Bengali lexicon & inflections |
+
+---
+
+## 4. In-Memory Session Cache (`b2b_cache`)
+Dynamic in-memory cache managed at runtime by `BanglishConverter`. Stores dynamically transliterated shorthand words for instantaneous $O(1)$ repeated access without modifying files on disk.
+
+---
+
+## 5. Sources & Acknowledgments
+Word lists are sourced, normalized, and compiled from open-source Bengali linguistic corpora and the [BengaliDictionary](https://github.com/MinhasKamal/BengaliDictionary) repository.

shobdosearch-2.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Khalid Mahmud & Antora Ghosh
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

shobdosearch-2.0.0/MANIFEST.in

@@ -0,0 +1,8 @@
+include README.md
+include LICENSE
+include ARCHITECTURE.md
+include CHANGELOG.md
+include DATASET.md
+include ROADMAP.md
+recursive-include data *
+recursive-include static *

shobdosearch-2.0.0/PKG-INFO

@@ -0,0 +1,130 @@
+Metadata-Version: 2.4
+Name: shobdosearch
+Version: 2.0.0
+Summary: High-accuracy Banglish-to-Bangla NLP transliteration engine with dynamic shorthand recovery
+Author: Khalid Mahmud
+License: MIT
+Project-URL: Homepage, https://github.com/skhalidmahmud/ShobdoSearch
+Project-URL: Bug Tracker, https://github.com/skhalidmahmud/ShobdoSearch/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Topic :: Text Processing :: Linguistic
+Classifier: Natural Language :: Bengali
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi>=0.100.0
+Requires-Dist: uvicorn>=0.20.0
+Requires-Dist: pydantic>=2.0.0
+Dynamic: license-file
+
+# 🇧🇩 Banglish-to-Bangla: Phonetic Smart Converter
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![Tests](https://img.shields.io/badge/Tests-41%2F41%20Passing-brightgreen.svg)](#testing--verification)
+
+An intelligent, rule-based, and dictionary-validated NLP engine designed to convert Romanized Bengali (Banglish) into authentic Bangla Unicode script. Features dynamic vowel-expansion for informal texting/chat shorthand, weighted multi-dictionary validation across **464,000+ words**, and an alphabetically sorted curated base dictionary of **2,770+ words** covering 80%+ of daily conversational Bengali.
+
+---
+
+## 🚀 Key Features
+
+- **Dynamic Vowel-Expansion & Shorthand Recovery**: Seamlessly transliterates informal chat shorthand (e.g., `tmi` → **তুমি**, `vlo` → **ভালো**, `kmn` → **কেমন**, `amr` → **আমার**, `bndhu` → **বন্ধু**, `rsta` → **রাস্তা**).
+- **Curated 2,770+ Baseline Dictionary**: Pre-indexed and alphabetically sorted mapping in [`data/ben2bn.csv`](file:///c:/Users/Khalid/OneDrive/Desktop/Git%20clone/Antigravity/ShobdoSearch/data/ben2bn.csv) covering top 80%+ everyday Bengali vocabulary.
+- **Weighted Multi-Tier Dictionary**: Validates candidates against 464,411 words across 4 frequency tiers for maximum accuracy.
+- **Modern Glassmorphic Web UI**: Responsive web app with live transliteration, history, voice typing, and clipboard utilities.
+- **FastAPI Backend**: Clean RESTful endpoints (`/convert`, `/stats`) for integration into web, mobile, and desktop applications.
+- **In-Memory Cache**: High-speed session caching (`b2b_cache`) without disk pollution.
+- **PWA & Offline Ready**: Service Worker and Web Manifest support for mobile/desktop installability.
+
+---
+
+## 💻 Installation & Quick Start
+
+### 1. Clone the Repository
+```bash
+git clone https://github.com/skhalidmahmud/ShobdoSearch.git
+cd ShobdoSearch
+```
+
+### 2. Start the Web App (Recommended)
+This installs required dependencies and launches the local dev server:
+```bash
+python run.py
+```
+Open your browser at **http://localhost:8080**.
+
+### 3. Using Docker
+```bash
+docker build -t banglish-converter .
+docker run -p 8080:8080 banglish-converter
+```
+
+### 4. Interactive CLI Mode
+```bash
+python converter.py
+```
+
+---
+
+## 🧪 Testing & Verification
+
+Run the built-in test suite to verify phonetic accuracy, edge cases, and chat shorthand recovery:
+```bash
+python verify.py
+```
+
+Output:
+```text
+===========================================
+      SHOBDOSEARCH CONVERSION TESTS       
+===========================================
+ [PASS] ka           -> কা
+ [PASS] kha          -> খা
+ [PASS] eta          -> এটা
+ [PASS] kemon        -> কেমন
+ [PASS] ami          -> আমি
+ [PASS] tumi         -> তুমি
+ [PASS] valo         -> ভালো
+ [PASS] shundor      -> সুন্দর
+ [PASS] manush       -> মানুষ
+ [PASS] ghor         -> ঘর
+ ...
+-------------------------------------------
+Total: 41 | Passed: 41 | Failed: 0
+ ALL TESTS PASSED SUCCESSFULLY!
+```
+
+---
+
+## 🛠️ API Reference
+
+- **Interactive Documentation**: `http://localhost:8080/docs`
+- **Transliterate Endpoint**:
+  ```http
+  POST /convert
+  Content-Type: application/json
+
+  {
+    "text": "tmi kmn aso? amr khub vlo lagse."
+  }
+  ```
+  **Response**:
+  ```json
+  {
+    "original": "tmi kmn aso? amr khub vlo lagse.",
+    "converted": "তুমি কেমন আছো? আমার খুব ভালো লাগছে."
+  }
+  ```
+- **Stats Endpoint**:
+  ```http
+  GET /stats
+  ```
+  Returns dictionary size, active rules, and baseline mappings count.
+
+---
+
+## 📄 License
+Distributed under the MIT License. See `LICENSE` for details.

shobdosearch-2.0.0/README.md

@@ -0,0 +1,109 @@
+# 🇧🇩 Banglish-to-Bangla: Phonetic Smart Converter
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+[![Python 3.12+](https://img.shields.io/badge/python-3.12+-blue.svg)](https://www.python.org/downloads/)
+[![Tests](https://img.shields.io/badge/Tests-41%2F41%20Passing-brightgreen.svg)](#testing--verification)
+
+An intelligent, rule-based, and dictionary-validated NLP engine designed to convert Romanized Bengali (Banglish) into authentic Bangla Unicode script. Features dynamic vowel-expansion for informal texting/chat shorthand, weighted multi-dictionary validation across **464,000+ words**, and an alphabetically sorted curated base dictionary of **2,770+ words** covering 80%+ of daily conversational Bengali.
+
+---
+
+## 🚀 Key Features
+
+- **Dynamic Vowel-Expansion & Shorthand Recovery**: Seamlessly transliterates informal chat shorthand (e.g., `tmi` → **তুমি**, `vlo` → **ভালো**, `kmn` → **কেমন**, `amr` → **আমার**, `bndhu` → **বন্ধু**, `rsta` → **রাস্তা**).
+- **Curated 2,770+ Baseline Dictionary**: Pre-indexed and alphabetically sorted mapping in [`data/ben2bn.csv`](file:///c:/Users/Khalid/OneDrive/Desktop/Git%20clone/Antigravity/ShobdoSearch/data/ben2bn.csv) covering top 80%+ everyday Bengali vocabulary.
+- **Weighted Multi-Tier Dictionary**: Validates candidates against 464,411 words across 4 frequency tiers for maximum accuracy.
+- **Modern Glassmorphic Web UI**: Responsive web app with live transliteration, history, voice typing, and clipboard utilities.
+- **FastAPI Backend**: Clean RESTful endpoints (`/convert`, `/stats`) for integration into web, mobile, and desktop applications.
+- **In-Memory Cache**: High-speed session caching (`b2b_cache`) without disk pollution.
+- **PWA & Offline Ready**: Service Worker and Web Manifest support for mobile/desktop installability.
+
+---
+
+## 💻 Installation & Quick Start
+
+### 1. Clone the Repository
+```bash
+git clone https://github.com/skhalidmahmud/ShobdoSearch.git
+cd ShobdoSearch
+```
+
+### 2. Start the Web App (Recommended)
+This installs required dependencies and launches the local dev server:
+```bash
+python run.py
+```
+Open your browser at **http://localhost:8080**.
+
+### 3. Using Docker
+```bash
+docker build -t banglish-converter .
+docker run -p 8080:8080 banglish-converter
+```
+
+### 4. Interactive CLI Mode
+```bash
+python converter.py
+```
+
+---
+
+## 🧪 Testing & Verification
+
+Run the built-in test suite to verify phonetic accuracy, edge cases, and chat shorthand recovery:
+```bash
+python verify.py
+```
+
+Output:
+```text
+===========================================
+      SHOBDOSEARCH CONVERSION TESTS       
+===========================================
+ [PASS] ka           -> কা
+ [PASS] kha          -> খা
+ [PASS] eta          -> এটা
+ [PASS] kemon        -> কেমন
+ [PASS] ami          -> আমি
+ [PASS] tumi         -> তুমি
+ [PASS] valo         -> ভালো
+ [PASS] shundor      -> সুন্দর
+ [PASS] manush       -> মানুষ
+ [PASS] ghor         -> ঘর
+ ...
+-------------------------------------------
+Total: 41 | Passed: 41 | Failed: 0
+ ALL TESTS PASSED SUCCESSFULLY!
+```
+
+---
+
+## 🛠️ API Reference
+
+- **Interactive Documentation**: `http://localhost:8080/docs`
+- **Transliterate Endpoint**:
+  ```http
+  POST /convert
+  Content-Type: application/json
+
+  {
+    "text": "tmi kmn aso? amr khub vlo lagse."
+  }
+  ```
+  **Response**:
+  ```json
+  {
+    "original": "tmi kmn aso? amr khub vlo lagse.",
+    "converted": "তুমি কেমন আছো? আমার খুব ভালো লাগছে."
+  }
+  ```
+- **Stats Endpoint**:
+  ```http
+  GET /stats
+  ```
+  Returns dictionary size, active rules, and baseline mappings count.
+
+---
+
+## 📄 License
+Distributed under the MIT License. See `LICENSE` for details.

shobdosearch-2.0.0/ROADMAP.md

@@ -0,0 +1,29 @@
+# Project Roadmap
+
+The goal of ShobdoSearch is to deliver a robust, high-accuracy, and frictionless Banglish-to-Bangla transliteration engine.
+
+---
+
+## 📍 Phase 1: Foundational Engine (Completed)
+- [x] Longest-match phonetic splitting logic.
+- [x] Recursive candidate generation.
+- [x] Multi-tier dictionary validation across 464k+ words.
+- [x] Modular object-oriented Python architecture (`converter.py`).
+
+## 📍 Phase 2: Shorthand & Accuracy Optimization (Completed)
+- [x] **Dynamic Vowel-Expansion Engine**: Algorithmic recovery for consonant-heavy chat slang (`tmi`, `vlo`, `kmn`, `apnr`, `bndhu`, `rsta`).
+- [x] **2,770+ Core Vocabulary Seed Map**: Alphabetically sorted base dictionary covering 80%+ daily vocabulary.
+- [x] **In-Memory Caching**: High-speed session caching (`b2b_cache`) with zero disk pollution.
+- [x] **Orthographic Corrections**: Fixed leading dependent vowel diacritics and dangling hasantas.
+- [x] **Standardized CSV Rulebook**: Aligned 111 phoneme rules into a strict 7-column matrix.
+
+## 📍 Phase 3: Web Platform & API (Completed)
+- [x] **Modern Glassmorphic UI**: Responsive web app with live transliteration.
+- [x] **FastAPI Backend**: Fully operational `/convert` and `/stats` endpoints.
+- [x] **Voice Typing Integration**: Web Speech API for voice-to-text input.
+- [x] **PWA & Offline Ready**: Service Worker and Web Manifest for desktop/mobile installation.
+
+## 📍 Phase 4: Future Enhancements (Planned)
+- [ ] **Context-Aware Language Model**: Bigram / Trigram n-gram scoring for adjacent word context.
+- [ ] **Browser Extension**: Chrome & Firefox extension for universal typing in web forms.
+- [ ] **Mobile Keyboard Layouts**: Android and iOS keyboard SDK integration.

shobdosearch-2.0.0/app.py

@@ -0,0 +1,55 @@
+from fastapi import FastAPI, HTTPException
+from fastapi.middleware.cors import CORSMiddleware
+from fastapi.staticfiles import StaticFiles
+from pydantic import BaseModel
+from converter import BanglishConverter
+import os
+
+app = FastAPI(title="Banglish-to-Bangla API")
+
+# Enable CORS for frontend integration
+app.add_middleware(
+    CORSMiddleware,
+    allow_origins=["*"],
+    allow_methods=["*"],
+    allow_headers=["*"],
+)
+
+# Initialize the converter
+converter = BanglishConverter()
+
+class TranslationRequest(BaseModel):
+    text: str
+
+@app.post("/convert")
+async def convert_text(request: TranslationRequest):
+    if not request.text:
+        return {"original": "", "converted": ""}
+    
+    try:
+        converted = converter.convert_sentence(request.text)
+        return {
+            "original": request.text,
+            "converted": converted
+        }
+    except Exception as e:
+        raise HTTPException(status_code=500, detail=str(e))
+
+@app.get("/stats")
+async def get_stats():
+    return {
+        "known_mappings": len(converter.b2b_map),
+        "generator_rules": len(converter.generator_map),
+        "dictionaries_loaded": 4,
+        "total_words_in_dict": len(converter.word_weights)
+    }
+
+# Serve static files (Frontend)
+current_dir = os.path.dirname(os.path.abspath(__file__))
+static_dir = os.path.join(current_dir, "static")
+if os.path.exists(static_dir):
+    app.mount("/", StaticFiles(directory=static_dir, html=True), name="static")
+
+if __name__ == "__main__":
+    import uvicorn
+    uvicorn.run(app, host="0.0.0.0", port=8080)