sinlib 0.2.2__tar.gz → 0.3.0__tar.gz

{sinlib-0.2.2 → sinlib-0.3.0}/.gitignore

@@ -17,3 +17,4 @@ website/dist/
 website/node_modules/
 website/.astro/
 
+notebooks/akshara_vocab.json

{sinlib-0.2.2 → sinlib-0.3.0}/.readthedocs.yaml

@@ -6,6 +6,6 @@ build:
     nodejs: "20"
   commands:
     - cd website && npm ci
-    - cd website && npm run build
+    - cd website && ASTRO_TELEMETRY_DISABLED=1 npm run build
     - mkdir -p $READTHEDOCS_OUTPUT/html
     - cp -r website/dist/* $READTHEDOCS_OUTPUT/html/

{sinlib-0.2.2 → sinlib-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: sinlib
-Version: 0.2.2
+Version: 0.3.0
 Summary: Sinhala NLP Toolkit
 Project-URL: Code, https://github.com/Ransaka/sinlib
 Project-URL: Docs, https://sinlib.readthedocs.io
@@ -39,6 +39,8 @@ Requires-Dist: matplotlib>=3.9.4
 Requires-Dist: numpy>=1.24.0
 Requires-Dist: tqdm>=4.64.1
 Requires-Dist: transformers>=4.31.0
+Provides-Extra: charbert
+Requires-Dist: torch>=2.0; extra == 'charbert'
 Provides-Extra: dev
 Requires-Dist: pytest; extra == 'dev'
 Description-Content-Type: text/markdown
@@ -105,6 +107,52 @@ detector.suggest_correction("අඩිරාජ")
 # ['අධිරාජ']
 ```
 
+### Neural Typo Correction (optional, CharBERT)
+
+TypoDetector can optionally delegate hard cases to a **Sinhala-CharBERT** neural
+corrector — a dual-channel (subword + phonological akshara) seq2seq model. This
+catches noise classes the statistical dictionary pipeline cannot fix: Singlish
+transliteration, dialectal morphology (`යන්ඩ` → `යන්න`), ZWJ-damaged ligatures
+(`ක්රීඩාව` → `ක්‍රීඩාව`), split/fused words, and Unicode decomposition errors.
+
+Install the optional dependency and pick a backend mode:
+
+```python
+from sinlib import TypoDetector
+
+# "denoise"  - bounded word-level neural fix when dictionary suggestions fail
+# "seq2seq"  - open-vocabulary sentence-level fix when structural noise is detected
+# "hybrid"   - both, in cascade (recommended)
+detector = TypoDetector(neural_backend="hybrid")
+
+detector("මම ගෙදර යන්ඩ ඕනේ")
+# 'මම ගෙදර යන්න ඕනේ'
+
+detector("මම gedara යන්න ඕනේ")
+# 'මම ගෙදර යන්න ඕනේ'
+
+detector("ක්රීඩාව")
+# 'ක්‍රීඩාව'
+```
+
+| Kwarg | Default | Description |
+|---|---|---|
+| `neural_backend` | `None` | `None`, `"denoise"`, `"seq2seq"`, or `"hybrid"` |
+| `backend_model` | `Ransaka/sinhala-charbert-seq2seq` | HF Hub repo id or local checkpoint dir (`pytorch_model.bin` + `char_vocab.json`) |
+| `backend_device` | auto (cuda > mps > cpu) | Torch device |
+| `backend_revision` | `None` | Pin a Hub revision |
+| `backend_num_beams` | `4` | Beam width for generation |
+
+Notes:
+
+- **Default behavior is unchanged** — without `neural_backend` the detector is
+  purely statistical and requires no torch.
+- Neural corrections are gated: clean in-dictionary sentences never hit the
+  model, and a candidate is accepted only if it scores no worse than the input
+  (hallucination guard).
+- If the checkpoint cannot be downloaded, the detector degrades gracefully to
+  statistical-only correction with a warning.
+
 ### Preprocessing
 
 ```python

{sinlib-0.2.2 → sinlib-0.3.0}/README.md

@@ -60,6 +60,52 @@ detector.suggest_correction("අඩිරාජ")
 # ['අධිරාජ']
 ```
 
+### Neural Typo Correction (optional, CharBERT)
+
+TypoDetector can optionally delegate hard cases to a **Sinhala-CharBERT** neural
+corrector — a dual-channel (subword + phonological akshara) seq2seq model. This
+catches noise classes the statistical dictionary pipeline cannot fix: Singlish
+transliteration, dialectal morphology (`යන්ඩ` → `යන්න`), ZWJ-damaged ligatures
+(`ක්රීඩාව` → `ක්‍රීඩාව`), split/fused words, and Unicode decomposition errors.
+
+Install the optional dependency and pick a backend mode:
+
+```python
+from sinlib import TypoDetector
+
+# "denoise"  - bounded word-level neural fix when dictionary suggestions fail
+# "seq2seq"  - open-vocabulary sentence-level fix when structural noise is detected
+# "hybrid"   - both, in cascade (recommended)
+detector = TypoDetector(neural_backend="hybrid")
+
+detector("මම ගෙදර යන්ඩ ඕනේ")
+# 'මම ගෙදර යන්න ඕනේ'
+
+detector("මම gedara යන්න ඕනේ")
+# 'මම ගෙදර යන්න ඕනේ'
+
+detector("ක්රීඩාව")
+# 'ක්‍රීඩාව'
+```
+
+| Kwarg | Default | Description |
+|---|---|---|
+| `neural_backend` | `None` | `None`, `"denoise"`, `"seq2seq"`, or `"hybrid"` |
+| `backend_model` | `Ransaka/sinhala-charbert-seq2seq` | HF Hub repo id or local checkpoint dir (`pytorch_model.bin` + `char_vocab.json`) |
+| `backend_device` | auto (cuda > mps > cpu) | Torch device |
+| `backend_revision` | `None` | Pin a Hub revision |
+| `backend_num_beams` | `4` | Beam width for generation |
+
+Notes:
+
+- **Default behavior is unchanged** — without `neural_backend` the detector is
+  purely statistical and requires no torch.
+- Neural corrections are gated: clean in-dictionary sentences never hit the
+  model, and a candidate is accepted only if it scores no worse than the input
+  (hallucination guard).
+- If the checkpoint cannot be downloaded, the detector degrades gracefully to
+  statistical-only correction with a warning.
+
 ### Preprocessing
 
 ```python

{sinlib-0.2.2 → sinlib-0.3.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "sinlib"
-version = "0.2.2"
+version = "0.3.0"
 description = "Sinhala NLP Toolkit"
 authors = [
     { name = "Ransaka", email = "ransaka.ravihara@gmail.com" }
@@ -28,6 +28,9 @@ classifiers = [
 dev = [
     "pytest",
 ]
+charbert = [
+    "torch >= 2.0",
+]
 
 [project.urls]
 Code = "https://github.com/Ransaka/sinlib"

{sinlib-0.2.2 → sinlib-0.3.0}/src/sinlib/__init__.py

@@ -17,7 +17,13 @@ BatchEncoding
 
 TypoDetector
     N-gram–based spell checker for Sinhala. Use
-    ``TypoDetector.from_pretrained()`` or instantiate directly.
+    ``TypoDetector.from_pretrained()`` or instantiate directly. Optionally
+    accepts ``neural_backend="seq2seq" | "denoise" | "hybrid"`` to delegate
+    hard cases to a CharBERT neural corrector.
+
+CharBERTBackend
+    Loader/corrector for published Sinhala-CharBERT seq2seq checkpoints
+    (HF Hub or local directory). See ``sinlib.charbert``.
 
 preprocessing
     Utility module exposing ``remove_english_characters``,
@@ -42,6 +48,7 @@ from sinlib.subword import SubwordTokenizer
 from sinlib.spellcheck import TypoDetector
 from sinlib.romanize import Romanizer
 from sinlib.transliterate import Transliterator  # noqa: F401
+from sinlib.charbert import CharBERTBackend
 from sinlib.utils import preprocessing, setup_matplotlib
 
 __all__: List[str] = [
@@ -49,9 +56,10 @@ __all__: List[str] = [
     "Tokenizer",
     "SubwordTokenizer",
     "TypoDetector",
+    "CharBERTBackend",
     "Romanizer",
     "preprocessing",
     "setup_matplotlib",
 ]
 
-__version__ = "0.2.2"
+__version__ = "0.3.0"

sinlib-0.3.0/src/sinlib/charbert/__init__.py

@@ -0,0 +1,15 @@
+"""
+Optional Sinhala-CharBERT neural backend for sinlib's TypoDetector.
+
+Provides inference-only model code and a high-level ``CharBERTBackend`` that
+loads published CharBERT seq2seq checkpoints (HF Hub or local directory).
+``torch`` is imported lazily; the base sinlib install does not require it.
+"""
+
+from sinlib.charbert.backend import CharBERTBackend
+from sinlib.charbert.config import SinhalaCharBERTConfig
+
+__all__ = [
+    "CharBERTBackend",
+    "SinhalaCharBERTConfig",
+]

sinlib-0.3.0/src/sinlib/charbert/alignment.py

@@ -0,0 +1,154 @@
+"""
+Sequence Alignment Engine for mapping subword tokens to phonological character unit boundaries.
+"""
+
+from dataclasses import dataclass, field
+from typing import Any, Dict, List, Optional, Tuple, Union
+from transformers import AutoTokenizer, PreTrainedTokenizerFast, PreTrainedTokenizer
+
+from sinlib.utils.preprocessing import normalize_sinhala, process_text
+from sinlib.charbert.char_tokenization import SinhalaCharTokenizer
+
+
+@dataclass
+class AlignedSequence:
+    """Represents a dual-channel aligned text sequence."""
+    text: str
+    tokens: List[str]
+    input_ids: List[int]
+    attention_mask: List[int]
+    char_units: List[str]
+    char_input_ids: List[int]
+    char_attention_mask: List[int]
+    start_char_idx: List[int]
+    end_char_idx: List[int]
+    subword_offsets: List[Tuple[int, int]] = field(default_factory=list)
+
+
+class SequenceAlignmentEngine:
+    """
+    Computes exact bidirectional boundary mappings between Subword Tokens (m)
+    and Phonological Units / Aksharas (N).
+    """
+
+    def __init__(
+        self,
+        subword_tokenizer: Union[PreTrainedTokenizerFast, PreTrainedTokenizer, Any],
+        char_tokenizer: SinhalaCharTokenizer,
+        max_subword_length: int = 256,
+        max_char_length: int = 512,
+    ):
+        self.subword_tokenizer = subword_tokenizer
+        self.char_tokenizer = char_tokenizer
+        self.max_subword_length = max_subword_length
+        self.max_char_length = max_char_length
+
+    def _get_phonological_spans(self, text: str, units: List[str]) -> List[Tuple[int, int]]:
+        """Calculates character string offsets [start_char, end_char) for each phonological unit in raw text."""
+        spans: List[Tuple[int, int]] = []
+        curr_idx = 0
+        for u in units:
+            # Locate unit starting from current offset
+            found_idx = text.find(u, curr_idx)
+            if found_idx == -1:
+                # Fallback if normalization shifted index
+                found_idx = curr_idx
+            end_idx = found_idx + len(u)
+            spans.append((found_idx, end_idx))
+            curr_idx = end_idx
+        return spans
+
+    def align(self, text: str) -> AlignedSequence:
+        """
+        Aligns a single sentence across subword and character channels.
+        Returns subword IDs, character IDs, and boundary coordinate arrays start_char_idx and end_char_idx.
+        """
+        norm_text = normalize_sinhala(text.strip())
+        if not norm_text:
+            norm_text = " "
+
+        # 1. Subword Tokenization with offset mapping
+        subword_encoding = self.subword_tokenizer(
+            norm_text,
+            truncation=True,
+            max_length=self.max_subword_length,
+            return_offsets_mapping=True,
+            add_special_tokens=True,
+        )
+
+        subword_ids: List[int] = subword_encoding["input_ids"]
+        subword_attn: List[int] = subword_encoding.get("attention_mask", [1] * len(subword_ids))
+        subword_tokens: List[str] = self.subword_tokenizer.convert_ids_to_tokens(subword_ids)
+        subword_offsets: List[Tuple[int, int]] = subword_encoding["offset_mapping"]
+
+        # 2. Phonological Unit Tokenization
+        raw_units = process_text(norm_text)
+        unit_spans = self._get_phonological_spans(norm_text, raw_units)
+
+        # Prepend <bos> and append <eos> to phonological units
+        char_units = [SinhalaCharTokenizer.BOS_TOKEN] + raw_units + [SinhalaCharTokenizer.EOS_TOKEN]
+        char_ids = (
+            [self.char_tokenizer.bos_token_id]
+            + [self.char_tokenizer.vocab_map.get(u, self.char_tokenizer.unk_token_id) for u in raw_units]
+            + [self.char_tokenizer.eos_token_id]
+        )
+
+        # Truncate character sequence if needed
+        if len(char_ids) > self.max_char_length:
+            char_ids = char_ids[: self.max_char_length - 1] + [self.char_tokenizer.eos_token_id]
+            char_units = char_units[: self.max_char_length - 1] + [SinhalaCharTokenizer.EOS_TOKEN]
+            unit_spans = unit_spans[: len(char_units) - 2]
+
+        char_attn = [1] * len(char_ids)
+        bos_idx = 0
+        eos_idx = len(char_ids) - 1
+
+        # 3. Align Subword Tokens to Phonological Units
+        start_char_idx: List[int] = []
+        end_char_idx: List[int] = []
+
+        for i, (tok_start, tok_end) in enumerate(subword_offsets):
+            # Special tokens [CLS], [SEP], [PAD]
+            if tok_start == 0 and tok_end == 0:
+                if i == 0:  # First token is [CLS] / [BOS]
+                    start_char_idx.append(bos_idx)
+                    end_char_idx.append(bos_idx)
+                else:  # Last or padding token is [SEP] / [EOS]
+                    start_char_idx.append(eos_idx)
+                    end_char_idx.append(eos_idx)
+                continue
+
+            # Find matching phonological units by offset overlap
+            matching_unit_indices = []
+            for u_idx, (u_start, u_end) in enumerate(unit_spans):
+                # Check for overlap: max(tok_start, u_start) < min(tok_end, u_end)
+                if max(tok_start, u_start) < min(tok_end, u_end):
+                    # Offset +1 because index 0 is <bos>
+                    matching_unit_indices.append(u_idx + 1)
+
+            if matching_unit_indices:
+                start_char_idx.append(matching_unit_indices[0])
+                end_char_idx.append(matching_unit_indices[-1])
+            else:
+                # Nearest fallback to prevent out-of-bounds indexing
+                closest_idx = min(
+                    range(len(unit_spans)),
+                    key=lambda idx: abs(unit_spans[idx][0] - tok_start),
+                    default=0,
+                )
+                adjusted_idx = min(closest_idx + 1, eos_idx - 1)
+                start_char_idx.append(adjusted_idx)
+                end_char_idx.append(adjusted_idx)
+
+        return AlignedSequence(
+            text=norm_text,
+            tokens=subword_tokens,
+            input_ids=subword_ids,
+            attention_mask=subword_attn,
+            char_units=char_units,
+            char_input_ids=char_ids,
+            char_attention_mask=char_attn,
+            start_char_idx=start_char_idx,
+            end_char_idx=end_char_idx,
+            subword_offsets=subword_offsets,
+        )