arabicocr-khatt 0.1.0__tar.gz

arabicocr_khatt-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Osama Sharaf
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

arabicocr_khatt-0.1.0/PKG-INFO

@@ -0,0 +1,203 @@
+Metadata-Version: 2.4
+Name: arabicocr-khatt
+Version: 0.1.0
+Summary: Arabic handwritten text recognition — CRNN-CTC with Arabic-specific multi-scale vertical encoding, trained on the KHATT dataset
+Author: Osama Sharaf
+License: MIT
+Project-URL: Homepage, https://github.com/FixFips/ArabicOCR_KHATT
+Project-URL: Repository, https://github.com/FixFips/ArabicOCR_KHATT
+Project-URL: Issues, https://github.com/FixFips/ArabicOCR_KHATT/issues
+Keywords: arabic,ocr,htr,handwriting,handwritten-text-recognition,khatt,crnn,ctc,pytorch,rtl
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: Arabic
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Scientific/Engineering :: Image Recognition
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.2
+Requires-Dist: numpy>=1.24
+Requires-Dist: pillow>=10.0
+Requires-Dist: opencv-python-headless>=4.8
+Requires-Dist: rapidfuzz>=3.6
+Requires-Dist: huggingface_hub>=0.23
+Provides-Extra: demo
+Requires-Dist: gradio>=4.0; extra == "demo"
+Requires-Dist: pandas>=2.0; extra == "demo"
+Requires-Dist: torchvision>=0.17; extra == "demo"
+Provides-Extra: train
+Requires-Dist: pandas>=2.0; extra == "train"
+Requires-Dist: scikit-learn>=1.3; extra == "train"
+Requires-Dist: torchvision>=0.17; extra == "train"
+Dynamic: license-file
+
+# ArabicOCR-KHATT — Arabic Handwritten Text Recognition (CRNN-CTC)
+
+[![PyPI](https://img.shields.io/pypi/v/arabicocr-khatt)](https://pypi.org/project/arabicocr-khatt/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Model on HF](https://img.shields.io/badge/%F0%9F%A4%97%20Hub-FixFips%2Farabicocr--khatt-blue)](https://huggingface.co/FixFips/arabicocr-khatt)
+
+**Arabic handwritten text recognition** — line-level OCR (a line or paragraph
+image → Arabic text), trained on the **KHATT** dataset.
+
+What makes it Arabic-specific rather than a generic CRNN:
+
+- **3-zone vertical pooling** — preserves *where* dots sit (above vs below the
+  baseline), the only difference between ب/ت/ث and ن/ي
+- **Input height 96** — keeps diacritic dots large enough (3–5 px) for 3×3
+  convolutions to detect
+- **Dot-safe augmentation** — shear, kashida stretch, mild rotation; erosion /
+  dilation / elastic are banned because they destroy 2–4 px dots
+- **Beam-search decoding with an Arabic character bigram LM**
+- **Dot-group CER metric** — tracks errors on dot-differentiated letter groups,
+  the #1 Arabic OCR error source
+
+---
+
+## Install
+
+```bash
+pip install arabicocr-khatt
+```
+
+## Quickstart
+
+```python
+from arabicocr_khatt import ArabicOCR
+
+ocr = ArabicOCR.from_pretrained()        # downloads weights from the HF Hub
+text = ocr.recognize("handwritten_page.jpg")   # multi-line pages segmented automatically
+print(text)
+```
+
+Or from the command line:
+
+```bash
+arabicocr handwritten_page.jpg
+arabicocr line.png --no-segment --greedy       # single line, fastest decode
+arabicocr scan.jpg --checkpoint runs/exp1/crnn_best.pt   # local weights
+```
+
+Try it in the browser: **[demo Space on Hugging Face](https://huggingface.co/spaces/FixFips/arabicocr-khatt-demo)**.
+
+---
+
+## Architecture
+
+```
+Input [B, 1, 96, 1536] grayscale
+  → CNN (7 conv layers, full BatchNorm, Dropout2d)
+  → adaptive pool to 3 vertical zones (dot position preserved)
+  → 2-layer BiLSTM(1536 → 384)
+  → FC → CTC loss
+  → greedy or beam-search decode (+ Arabic bigram LM), reversed back to RTL
+```
+
+Preprocessing (shared between training and inference): grayscale → CLAHE →
+dual-polarity Otsu binarization → LANCZOS resize to H=96 → pad to W=1536.
+
+## Project structure
+
+```text
+ArabicOCR_KHATT/
+├─ arabicocr_khatt/            # the installable package
+│  ├─ pipeline.py              # ArabicOCR API + CLI (inference single-source)
+│  ├─ model.py                 # CRNN + CTC decoders + bigram LM builder
+│  ├─ preprocess.py            # CLAHE, binarization, resize, padding
+│  ├─ augment.py               # Arabic-safe augmentation
+│  ├─ dataset.py               # KHATTDataset
+│  ├─ metrics.py               # CER / WER / dot-group CER
+│  ├─ train_crnn_ctc.py        # training script
+│  ├─ webocr.py                # Gradio test bench (rich debugging UI)
+│  ├─ show_metrics.py, monitor.py, compare_runs.py, eval_val.py, ...
+│  └─ charset_arabic.txt       # 75-class charset
+├─ scripts/upload_to_hf.py     # publish weights + model card to the HF Hub
+├─ space/                      # Hugging Face Space demo app
+├─ archive/                    # (gitignored) KHATT images + labels + splits
+├─ runs/                       # (gitignored) checkpoints + metrics.csv
+└─ pyproject.toml
+```
+
+---
+
+## Training your own model
+
+Get the KHATT dataset (request access at [khatt.ideas2serve.net](https://khatt.ideas2serve.net/))
+and lay it out as:
+
+```text
+archive/
+├─ images/   *.jpg            # line images
+├─ labels/   *.txt            # Arabic text (Windows-1256 / UTF-8)
+└─ splits/                    # auto-created train/val/test CSVs (80/10/10)
+```
+
+Then, with the training extras installed:
+
+```bash
+pip install -e ".[train]"
+python -m arabicocr_khatt.train_crnn_ctc                 # trains to runs/exp1
+python -m arabicocr_khatt.show_metrics --run ./runs/exp1 # inspect metrics
+python -m arabicocr_khatt.monitor                        # live training dashboard
+```
+
+Training uses OneCycleLR, gradient accumulation, early stopping, and logs
+per-epoch CER / WER / WER(norm) / dot-group CER to `runs/<exp>/metrics.csv`.
+Checkpoints are saved as `{"model": state_dict, "vocab": [...], "arch_version": 2}`.
+
+## Web test bench (Gradio)
+
+A rich debugging UI with crop/rotate, polarity control, decoder comparison,
+confidence heatmaps, char-level diff against ground truth, and batch evaluation:
+
+```bash
+pip install -e ".[demo]"
+python -m arabicocr_khatt.webocr
+```
+
+## Publishing weights to the Hub
+
+After training (on the machine that has `runs/` and `archive/`):
+
+```bash
+hf auth login
+python scripts/upload_to_hf.py --run-dir runs/exp1
+```
+
+This validates the checkpoint, builds the bigram LM from the training split,
+fills the model card with your best-epoch metrics, and uploads everything to
+[FixFips/arabicocr-khatt](https://huggingface.co/FixFips/arabicocr-khatt).
+
+---
+
+## Notes and limitations
+
+- Trained on KHATT: performance is best on handwriting similar to that dataset.
+  Printed fonts, very noisy backgrounds, or strongly curved text may fail.
+- Line-level model: pages are segmented into lines with classical morphology;
+  complex layouts may segment poorly.
+- Labels use Windows-1256 encoding (KHATT standard); the reader falls back to
+  UTF-8 automatically.
+
+## Contributing
+
+Contributions are very welcome — Arabic OCR is an underserved area and there is
+plenty to do. See [CONTRIBUTING.md](CONTRIBUTING.md) and the
+[good first issues](https://github.com/FixFips/ArabicOCR_KHATT/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+If you use this project in research, please also cite the KHATT dataset:
+
+> Mahmoud, S. A., et al. "KHATT: An open Arabic offline handwritten text
+> database." Pattern Recognition 47.3 (2014): 1096–1112.

arabicocr_khatt-0.1.0/README.md

@@ -0,0 +1,162 @@
+# ArabicOCR-KHATT — Arabic Handwritten Text Recognition (CRNN-CTC)
+
+[![PyPI](https://img.shields.io/pypi/v/arabicocr-khatt)](https://pypi.org/project/arabicocr-khatt/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Model on HF](https://img.shields.io/badge/%F0%9F%A4%97%20Hub-FixFips%2Farabicocr--khatt-blue)](https://huggingface.co/FixFips/arabicocr-khatt)
+
+**Arabic handwritten text recognition** — line-level OCR (a line or paragraph
+image → Arabic text), trained on the **KHATT** dataset.
+
+What makes it Arabic-specific rather than a generic CRNN:
+
+- **3-zone vertical pooling** — preserves *where* dots sit (above vs below the
+  baseline), the only difference between ب/ت/ث and ن/ي
+- **Input height 96** — keeps diacritic dots large enough (3–5 px) for 3×3
+  convolutions to detect
+- **Dot-safe augmentation** — shear, kashida stretch, mild rotation; erosion /
+  dilation / elastic are banned because they destroy 2–4 px dots
+- **Beam-search decoding with an Arabic character bigram LM**
+- **Dot-group CER metric** — tracks errors on dot-differentiated letter groups,
+  the #1 Arabic OCR error source
+
+---
+
+## Install
+
+```bash
+pip install arabicocr-khatt
+```
+
+## Quickstart
+
+```python
+from arabicocr_khatt import ArabicOCR
+
+ocr = ArabicOCR.from_pretrained()        # downloads weights from the HF Hub
+text = ocr.recognize("handwritten_page.jpg")   # multi-line pages segmented automatically
+print(text)
+```
+
+Or from the command line:
+
+```bash
+arabicocr handwritten_page.jpg
+arabicocr line.png --no-segment --greedy       # single line, fastest decode
+arabicocr scan.jpg --checkpoint runs/exp1/crnn_best.pt   # local weights
+```
+
+Try it in the browser: **[demo Space on Hugging Face](https://huggingface.co/spaces/FixFips/arabicocr-khatt-demo)**.
+
+---
+
+## Architecture
+
+```
+Input [B, 1, 96, 1536] grayscale
+  → CNN (7 conv layers, full BatchNorm, Dropout2d)
+  → adaptive pool to 3 vertical zones (dot position preserved)
+  → 2-layer BiLSTM(1536 → 384)
+  → FC → CTC loss
+  → greedy or beam-search decode (+ Arabic bigram LM), reversed back to RTL
+```
+
+Preprocessing (shared between training and inference): grayscale → CLAHE →
+dual-polarity Otsu binarization → LANCZOS resize to H=96 → pad to W=1536.
+
+## Project structure
+
+```text
+ArabicOCR_KHATT/
+├─ arabicocr_khatt/            # the installable package
+│  ├─ pipeline.py              # ArabicOCR API + CLI (inference single-source)
+│  ├─ model.py                 # CRNN + CTC decoders + bigram LM builder
+│  ├─ preprocess.py            # CLAHE, binarization, resize, padding
+│  ├─ augment.py               # Arabic-safe augmentation
+│  ├─ dataset.py               # KHATTDataset
+│  ├─ metrics.py               # CER / WER / dot-group CER
+│  ├─ train_crnn_ctc.py        # training script
+│  ├─ webocr.py                # Gradio test bench (rich debugging UI)
+│  ├─ show_metrics.py, monitor.py, compare_runs.py, eval_val.py, ...
+│  └─ charset_arabic.txt       # 75-class charset
+├─ scripts/upload_to_hf.py     # publish weights + model card to the HF Hub
+├─ space/                      # Hugging Face Space demo app
+├─ archive/                    # (gitignored) KHATT images + labels + splits
+├─ runs/                       # (gitignored) checkpoints + metrics.csv
+└─ pyproject.toml
+```
+
+---
+
+## Training your own model
+
+Get the KHATT dataset (request access at [khatt.ideas2serve.net](https://khatt.ideas2serve.net/))
+and lay it out as:
+
+```text
+archive/
+├─ images/   *.jpg            # line images
+├─ labels/   *.txt            # Arabic text (Windows-1256 / UTF-8)
+└─ splits/                    # auto-created train/val/test CSVs (80/10/10)
+```
+
+Then, with the training extras installed:
+
+```bash
+pip install -e ".[train]"
+python -m arabicocr_khatt.train_crnn_ctc                 # trains to runs/exp1
+python -m arabicocr_khatt.show_metrics --run ./runs/exp1 # inspect metrics
+python -m arabicocr_khatt.monitor                        # live training dashboard
+```
+
+Training uses OneCycleLR, gradient accumulation, early stopping, and logs
+per-epoch CER / WER / WER(norm) / dot-group CER to `runs/<exp>/metrics.csv`.
+Checkpoints are saved as `{"model": state_dict, "vocab": [...], "arch_version": 2}`.
+
+## Web test bench (Gradio)
+
+A rich debugging UI with crop/rotate, polarity control, decoder comparison,
+confidence heatmaps, char-level diff against ground truth, and batch evaluation:
+
+```bash
+pip install -e ".[demo]"
+python -m arabicocr_khatt.webocr
+```
+
+## Publishing weights to the Hub
+
+After training (on the machine that has `runs/` and `archive/`):
+
+```bash
+hf auth login
+python scripts/upload_to_hf.py --run-dir runs/exp1
+```
+
+This validates the checkpoint, builds the bigram LM from the training split,
+fills the model card with your best-epoch metrics, and uploads everything to
+[FixFips/arabicocr-khatt](https://huggingface.co/FixFips/arabicocr-khatt).
+
+---
+
+## Notes and limitations
+
+- Trained on KHATT: performance is best on handwriting similar to that dataset.
+  Printed fonts, very noisy backgrounds, or strongly curved text may fail.
+- Line-level model: pages are segmented into lines with classical morphology;
+  complex layouts may segment poorly.
+- Labels use Windows-1256 encoding (KHATT standard); the reader falls back to
+  UTF-8 automatically.
+
+## Contributing
+
+Contributions are very welcome — Arabic OCR is an underserved area and there is
+plenty to do. See [CONTRIBUTING.md](CONTRIBUTING.md) and the
+[good first issues](https://github.com/FixFips/ArabicOCR_KHATT/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22).
+
+## License
+
+MIT — see [LICENSE](LICENSE).
+
+If you use this project in research, please also cite the KHATT dataset:
+
+> Mahmoud, S. A., et al. "KHATT: An open Arabic offline handwritten text
+> database." Pattern Recognition 47.3 (2014): 1096–1112.

arabicocr_khatt-0.1.0/arabicocr_khatt/__init__.py

@@ -0,0 +1,7 @@
+"""Arabic handwritten text recognition (CRNN-CTC) trained on the KHATT dataset."""
+
+__version__ = "0.1.0"
+
+from .pipeline import ArabicOCR
+
+__all__ = ["ArabicOCR", "__version__"]

arabicocr_khatt-0.1.0/arabicocr_khatt/augment.py

@@ -0,0 +1,147 @@
+# arabicocr_khatt/augment.py
+"""
+Arabic-safe data augmentation for handwriting OCR.
+
+DESIGN PRINCIPLE: Arabic letters (ba/ta/tha/nun/ya) share identical base
+strokes and differ ONLY by dots that are 2-4 pixels tall.  Every transform
+here is chosen to preserve dots and cursive connections.
+
+BANNED transforms and why:
+  - Erosion:           destroys dots (2-4px) and breaks thin connections
+  - Dilation:          merges adjacent dots (ta → ba) and fills inter-letter gaps
+  - Elastic distortion: shifts dots away from their base letter
+  - Random crop/cutout: can remove dots above/below the text line
+  - Heavy rotation (>2°): shifts dot laterally to wrong letter position
+"""
+
+import random
+import cv2
+import numpy as np
+from PIL import Image
+
+
+class ArabicAugment:
+    """Apply Arabic-safe augmentations to a binarized grayscale PIL Image."""
+
+    def __init__(self, training: bool = True):
+        self.training = training
+
+    def __call__(self, img: Image.Image) -> Image.Image:
+        if not self.training:
+            return img
+
+        arr = np.array(img, dtype=np.uint8)  # binarized grayscale
+
+        # 1) Affine shear — slants all features equally, dots stay above/below their letter
+        if random.random() < 0.4:
+            arr = self._shear(arr)
+
+        # 2) Slight rotation — within ±2° dots keep their letter association
+        if random.random() < 0.3:
+            arr = self._rotate(arr)
+
+        # 3) Kashida (tatweel) stretch — uniquely Arabic horizontal elongation
+        #    Stretches connections between letters; dots sit above/below and are unaffected
+        if random.random() < 0.3:
+            arr = self._kashida_stretch(arr)
+
+        # 4) Vertical baseline shift — simulates off-center writing
+        if random.random() < 0.2:
+            arr = self._vertical_shift(arr)
+
+        # 5) Brightness/contrast jitter — simulates paper/ink variation
+        if random.random() < 0.3:
+            arr = self._brightness_jitter(arr)
+
+        # 6) Mild Gaussian noise — scanner/paper noise at safe levels
+        if random.random() < 0.2:
+            arr = self._gaussian_noise(arr)
+
+        # Re-binarize to maintain clean binary format
+        arr = (arr > 128).astype(np.uint8) * 255
+
+        return Image.fromarray(arr, mode="L")
+
+    @staticmethod
+    def _shear(arr: np.ndarray) -> np.ndarray:
+        h, w = arr.shape[:2]
+        shear_x = random.uniform(-0.12, 0.12)
+        M = np.array([[1, shear_x, 0], [0, 1, 0]], dtype=np.float32)
+        # Shift to keep content centered
+        M[0, 2] = -shear_x * h / 2
+        return cv2.warpAffine(
+            arr, M, (w, h),
+            flags=cv2.INTER_LINEAR,
+            borderMode=cv2.BORDER_CONSTANT,
+            borderValue=255,
+        )
+
+    @staticmethod
+    def _rotate(arr: np.ndarray) -> np.ndarray:
+        h, w = arr.shape[:2]
+        angle = random.uniform(-2.0, 2.0)
+        M = cv2.getRotationMatrix2D((w / 2, h / 2), angle, 1.0)
+        return cv2.warpAffine(
+            arr, M, (w, h),
+            flags=cv2.INTER_LINEAR,
+            borderMode=cv2.BORDER_CONSTANT,
+            borderValue=255,
+        )
+
+    @staticmethod
+    def _kashida_stretch(arr: np.ndarray) -> np.ndarray:
+        """
+        Simulate Arabic kashida (tatweel) by horizontally stretching
+        a random column range.  This stretches the horizontal connections
+        between letters — dots sit above/below and are spatially unaffected.
+        """
+        h, w = arr.shape[:2]
+        if w < 20:
+            return arr
+
+        # Pick a random stretch region (20-40% of width)
+        region_w = random.randint(w // 5, 2 * w // 5)
+        start = random.randint(0, w - region_w)
+        end = start + region_w
+
+        stretch_factor = random.uniform(1.0, 1.15)
+        new_region_w = int(region_w * stretch_factor)
+
+        region = arr[:, start:end]
+        stretched = cv2.resize(region, (new_region_w, h), interpolation=cv2.INTER_LINEAR)
+
+        new_w = w - region_w + new_region_w
+        result = np.full((h, new_w), 255, dtype=np.uint8)
+        result[:, :start] = arr[:, :start]
+        result[:, start:start + new_region_w] = stretched
+        result[:, start + new_region_w:] = arr[:, end:]
+
+        # Resize back to original width to maintain consistency
+        return cv2.resize(result, (w, h), interpolation=cv2.INTER_LINEAR)
+
+    @staticmethod
+    def _vertical_shift(arr: np.ndarray) -> np.ndarray:
+        shift = random.randint(-2, 2)
+        if shift == 0:
+            return arr
+        M = np.array([[1, 0, 0], [0, 1, shift]], dtype=np.float32)
+        return cv2.warpAffine(
+            arr, M, (arr.shape[1], arr.shape[0]),
+            flags=cv2.INTER_LINEAR,
+            borderMode=cv2.BORDER_CONSTANT,
+            borderValue=255,
+        )
+
+    @staticmethod
+    def _brightness_jitter(arr: np.ndarray) -> np.ndarray:
+        factor = random.uniform(0.85, 1.15)
+        offset = random.uniform(-10, 10)
+        result = arr.astype(np.float32) * factor + offset
+        return np.clip(result, 0, 255).astype(np.uint8)
+
+    @staticmethod
+    def _gaussian_noise(arr: np.ndarray) -> np.ndarray:
+        sigma = random.uniform(0, 8)
+        noise = np.random.randn(*arr.shape) * sigma
+        result = arr.astype(np.float32) + noise
+        return np.clip(result, 0, 255).astype(np.uint8)

arabicocr_khatt-0.1.0/arabicocr_khatt/charset_arabic.txt

@@ -0,0 +1,78 @@
+# Digits
+0
+1
+2
+3
+4
+5
+6
+7
+8
+9
+
+# Basic punctuation
+
+!
+.
+،
+؟
+؛
+"
+'
+-
+:
+#
+(
+)
+,
+%
+/
+
+# Arabic letters
+ء
+آ
+أ
+ؤ
+إ
+ئ
+ا
+ب
+ة
+ت
+ث
+ج
+ح
+خ
+د
+ذ
+ر
+ز
+س
+ش
+ص
+ض
+ط
+ظ
+ع
+غ
+ف
+ق
+ك
+ل
+م
+ن
+ه
+و
+ي
+ى
+
+# Kashida (tatweel) and diacritics
+ـ
+ً
+ٌ
+ٍ
+َ
+ُ
+ِ
+ّ
+ْ