arabic-fix 0.4.0__tar.gz

arabic_fix-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 arabic-fix contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

arabic_fix-0.4.0/PKG-INFO

@@ -0,0 +1,101 @@
+Metadata-Version: 2.4
+Name: arabic-fix
+Version: 0.4.0
+Summary: Fix Arabic text rendering across code, AI agents, and designs.
+Author: arabic-fix contributors
+License: MIT
+Project-URL: Homepage, https://github.com/your-org/arabic-fix
+Project-URL: Issues, https://github.com/your-org/arabic-fix/issues
+Keywords: arabic,rtl,bidirectional,reshaper,bidi,unicode,i18n,l10n
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Natural Language :: Arabic
+Classifier: Natural Language :: English
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Internationalization
+Classifier: Topic :: Text Processing :: Markup
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: arabic-reshaper>=3.0.0
+Requires-Dist: python-bidi>=0.4.2
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: pytest-cov>=4.1; extra == "dev"
+Requires-Dist: mypy>=1.7; extra == "dev"
+Dynamic: license-file
+
+# arabic-fix
+
+> Fix Arabic text rendering across code, AI agents, and designs.
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)]()
+[![Tests](https://img.shields.io/badge/tests-169%20passing-brightgreen)]()
+
+**One library, one CLI, one system prompt.** Letter-shaping, BiDi reordering,
+Unicode normalization, and a 3-section prompt template for any LLM. Arabic
+flows through terminals, AI agents, log files, and PDFs the way it should.
+
+```bash
+pip install arabic-fix
+```
+
+```python
+from arabic_fix import fix
+print(fix("User 42 من الرياض at 09:30"))
+# → a properly shaped, correctly ordered, NFC-normalized string
+```
+
+```bash
+# CLI
+cat arabic.txt | arabic-fix
+arabic-fix --check file.txt          # CI mode: exit non-zero if changes needed
+```
+
+```text
+# AI agent: paste one of the 3 sections from agents/system_prompt.md
+#   A · Read Arabic    — for agents that parse / summarize / classify
+#   B · Write Arabic   — for agents that emit to terminal / log / email
+#   C · Design Arabic  — for HTML / CSS / Tailwind / design tokens
+```
+
+## Why
+
+- **Letter shaping** — `ا ب ت` (isolated forms) become `اتصال` (connected) via
+  Unicode Presentation Forms-B.
+- **BiDi order** — `User 42 من الرياض` ends up with `42` on the visual left
+  and Arabic letters in their connected form, per UAX #9.
+- **Tashkeel preserved** — diacritics survive; `مَرْحَبًا` round-trips intact.
+- **Logical CSS only** — `margin-inline-start`, never `margin-left`.
+- **Real Arabic fonts** — Tajawal, IBM Plex Sans Arabic, Noto Naskh Arabic
+  in priority order; never Inter or Roboto.
+
+## What's in the box
+
+| Path | What it is |
+|---|---|
+| `arabic_fix/` | Python library: `fix()`, `shape()`, `reorder()`, `normalize()` |
+| `agents/system_prompt.md` | 3 splice-in-able sections for any LLM |
+| `agents/eval_cases.md` | 15 cases to verify the system prompt works |
+| `designs/tokens.json` | 73 design tokens (W3C DTCG), CSS / Tailwind / Figma exports |
+| `examples/sample-arabic-page.html` | 5 Arabic surfaces using the tokens |
+| `docs/roadmap.md` | v0.2 → v0.6 milestones |
+
+## Verify
+
+```bash
+pytest -q                          # 169 tests
+mypy arabic_fix                    # strict, clean
+python designs/validate_tokens.py  # 73 token schema check
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE). The brand `arabic-fix` is locked until v1.0.

arabic_fix-0.4.0/README.md

@@ -0,0 +1,68 @@
+# arabic-fix
+
+> Fix Arabic text rendering across code, AI agents, and designs.
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.9+](https://img.shields.io/badge/python-3.9+-blue.svg)]()
+[![Tests](https://img.shields.io/badge/tests-169%20passing-brightgreen)]()
+
+**One library, one CLI, one system prompt.** Letter-shaping, BiDi reordering,
+Unicode normalization, and a 3-section prompt template for any LLM. Arabic
+flows through terminals, AI agents, log files, and PDFs the way it should.
+
+```bash
+pip install arabic-fix
+```
+
+```python
+from arabic_fix import fix
+print(fix("User 42 من الرياض at 09:30"))
+# → a properly shaped, correctly ordered, NFC-normalized string
+```
+
+```bash
+# CLI
+cat arabic.txt | arabic-fix
+arabic-fix --check file.txt          # CI mode: exit non-zero if changes needed
+```
+
+```text
+# AI agent: paste one of the 3 sections from agents/system_prompt.md
+#   A · Read Arabic    — for agents that parse / summarize / classify
+#   B · Write Arabic   — for agents that emit to terminal / log / email
+#   C · Design Arabic  — for HTML / CSS / Tailwind / design tokens
+```
+
+## Why
+
+- **Letter shaping** — `ا ب ت` (isolated forms) become `اتصال` (connected) via
+  Unicode Presentation Forms-B.
+- **BiDi order** — `User 42 من الرياض` ends up with `42` on the visual left
+  and Arabic letters in their connected form, per UAX #9.
+- **Tashkeel preserved** — diacritics survive; `مَرْحَبًا` round-trips intact.
+- **Logical CSS only** — `margin-inline-start`, never `margin-left`.
+- **Real Arabic fonts** — Tajawal, IBM Plex Sans Arabic, Noto Naskh Arabic
+  in priority order; never Inter or Roboto.
+
+## What's in the box
+
+| Path | What it is |
+|---|---|
+| `arabic_fix/` | Python library: `fix()`, `shape()`, `reorder()`, `normalize()` |
+| `agents/system_prompt.md` | 3 splice-in-able sections for any LLM |
+| `agents/eval_cases.md` | 15 cases to verify the system prompt works |
+| `designs/tokens.json` | 73 design tokens (W3C DTCG), CSS / Tailwind / Figma exports |
+| `examples/sample-arabic-page.html` | 5 Arabic surfaces using the tokens |
+| `docs/roadmap.md` | v0.2 → v0.6 milestones |
+
+## Verify
+
+```bash
+pytest -q                          # 169 tests
+mypy arabic_fix                    # strict, clean
+python designs/validate_tokens.py  # 73 token schema check
+```
+
+## License
+
+MIT — see [LICENSE](LICENSE). The brand `arabic-fix` is locked until v1.0.

arabic_fix-0.4.0/arabic_fix/__init__.py

@@ -0,0 +1,13 @@
+"""arabic-fix: shape, BiDi-fix, and normalize Arabic text.
+
+A unified API for the three things that go wrong when Arabic text flows
+through systems built English-first: lost letter shaping, broken BiDi
+order, and inconsistent Unicode normalization.
+"""
+
+from __future__ import annotations
+
+from .fixer import fix, fix_report, FixOptions, FixReport
+
+__all__ = ["fix", "fix_report", "FixOptions", "FixReport"]
+__version__ = "0.2.0"

arabic_fix-0.4.0/arabic_fix/bidi.py

@@ -0,0 +1,86 @@
+"""Bidirectional (BiDi) reordering for mixed-direction text.
+
+Arabic is RTL. Numbers and Latin words embedded inside Arabic need
+explicit reordering or they land on the wrong side of the line.
+`python-bidi` does this correctly; we wrap it here so callers can pass a
+single string and get a display-ready one back.
+
+For HTML/CSS contexts (which is most web output), use the system's own
+BiDi engine via `direction: rtl` + `unicode-bidi: isolate` on the
+container — DO NOT pre-shape Arabic inside HTML, the browser does that
+better. This module is for terminals, log files, PDFs, AI chat where
+no rendering engine helps.
+"""
+
+from __future__ import annotations
+
+from typing import Optional
+
+try:
+    from bidi.algorithm import get_display  # type: ignore[import-untyped]
+    _HAS_BIDI = True
+    _ImportError: Exception | None = None
+except Exception as _exc:  # pragma: no cover
+    # bidi is untyped; plain `= None` is fine for mypy.
+    get_display = None
+    _HAS_BIDI = False
+    _ImportError = _exc
+
+
+# python-bidi v0.6+ uses single-char codes for base_dir
+_RTL_LTR_TO_BIDI = {"rtl": "R", "ltr": "L"}
+
+
+def reorder(
+    text: str,
+    *,
+    base_dir: str = "rtl",
+    upper_is_rtl: bool = False,
+) -> str:
+    """Apply BiDi reordering for display.
+
+    Parameters
+    ----------
+    text:
+        Input. May already be letter-shaped, but most callers pass
+        unshaped text here — shaping is a separate step.
+    base_dir:
+        'rtl' (default) or 'ltr'. Set 'ltr' if the dominant run is
+        English with an Arabic substring. Internally translated to
+        python-bidi's 'R' / 'L' codes.
+    upper_is_rtl:
+        Treat runs of uppercase Latin as RTL. Almost never what you
+        want; default False.
+
+    Returns
+    -------
+    str
+        Display-ordered string. If `python-bidi` is missing, the input
+        is returned unchanged.
+
+    Raises
+    ------
+    ValueError
+        If `base_dir` is not one of 'rtl' / 'ltr' (case-insensitive).
+        Previously this was silently passed through to `python-bidi`,
+        which would either raise an opaque error or fall back to a
+        default — masking caller bugs. Now we fail loud at the API edge.
+    """
+    if not text:
+        return text
+    if not _HAS_BIDI:
+        return text
+    bidi_dir = _RTL_LTR_TO_BIDI.get(base_dir.lower())
+    if bidi_dir is None:
+        raise ValueError(
+            f"unknown base_dir: {base_dir!r}; expected one of "
+            f"{sorted(_RTL_LTR_TO_BIDI)} (case-insensitive)"
+        )
+    return get_display(  # type: ignore
+        text,
+        base_dir=bidi_dir,
+        upper_is_rtl=upper_is_rtl,
+    )
+
+
+__all__ = ["reorder"]

arabic_fix-0.4.0/arabic_fix/cli.py

@@ -0,0 +1,161 @@
+"""CLI entry point — `arabic-fix` command."""
+
+from __future__ import annotations
+
+import argparse
+import io
+import sys
+from pathlib import Path
+
+from .fixer import fix, fix_report, FixOptions
+
+
+def _build_parser() -> argparse.ArgumentParser:
+    p = argparse.ArgumentParser(
+        prog="arabic-fix",
+        description=(
+            "Fix Arabic text in files: letter-shape, BiDi-reorder, normalize. "
+            "Reads UTF-8, writes UTF-8."
+        ),
+    )
+    p.add_argument(
+        "paths",
+        nargs="*",
+        type=Path,
+        help="Files to fix. Use '-' or omit to read from stdin.",
+    )
+    p.add_argument(
+        "-o", "--output",
+        type=Path,
+        help="Output file. Only valid with a single input file. "
+             "If omitted with one input, write in place.",
+    )
+    p.add_argument("--no-shape", action="store_true", help="Skip letter shaping.")
+    p.add_argument("--no-bidi", action="store_true", help="Skip BiDi reordering.")
+    p.add_argument(
+        "--no-normalize",
+        action="store_true",
+        help="Skip Unicode normalization.",
+    )
+    p.add_argument(
+        "--normalize-form",
+        default="NFC",
+        choices=["NFC", "NFD", "NFKC", "NFKD"],
+        help="Unicode normalization form (default NFC).",
+    )
+    p.add_argument(
+        "--bidi-base-dir",
+        default="rtl",
+        choices=["rtl", "ltr"],
+        help="Base direction for BiDi reorder (default rtl).",
+    )
+    p.add_argument(
+        "--no-ligatures",
+        action="store_true",
+        help="Disable reshape ligatures (e.g. لا, ﷲ).",
+    )
+    p.add_argument(
+        "--report",
+        action="store_true",
+        help="Print a JSON report per file (what was shaped, reordered, "
+             "normalized) to stderr.",
+    )
+    p.add_argument(
+        "--check",
+        action="store_true",
+        help="Exit non-zero if any file would change. Useful in CI to "
+             "enforce Arabic text is already fixed.",
+    )
+    p.add_argument(
+        "--encoding",
+        default="utf-8",
+        help="File encoding (default utf-8).",
+    )
+    return p
+
+
+def _process(text: str, opts: FixOptions) -> str:
+    return fix(text, **opts.__dict__)
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = _build_parser()
+    args = parser.parse_args(argv)
+    opts = FixOptions(
+        shape=not args.no_shape,
+        bidi=not args.no_bidi,
+        normalize_text=not args.no_normalize,
+        normalize_form=args.normalize_form,
+        bidi_base_dir=args.bidi_base_dir,
+        ligatures=not args.no_ligatures,
+    )
+
+    if not args.paths or (len(args.paths) == 1 and str(args.paths[0]) == "-"):
+        # stdin → stdout.
+        #
+        # `sys.stdin.read()` defaults to `locale.getpreferredencoding(False)`,
+        # which on many non-English terminals is ASCII or Latin-1 and will
+        # raise `UnicodeDecodeError` on Arabic UTF-8 bytes. Force UTF-8 by
+        # wrapping the binary buffer explicitly. This matches the encoding
+        # we use for file reads (`--encoding`, default utf-8).
+        #
+        # In tests, `sys.stdin` is sometimes monkeypatched to a `StringIO`
+        # (which has no `.buffer`); fall back to that wrapped as text.
+        if hasattr(sys.stdin, "buffer"):
+            text = io.TextIOWrapper(sys.stdin.buffer, encoding="utf-8").read()
+        else:
+            text = sys.stdin.read()
+        out = _process(text, opts)
+        sys.stdout.write(out)
+        return 0
+
+    if len(args.paths) > 1 and args.output:
+        print("error: --output can only be used with a single input file.",
+              file=sys.stderr)
+        return 2
+
+    exit_code = 0
+    for path in args.paths:
+        try:
+            original = path.read_text(encoding=args.encoding)
+        except OSError as exc:
+            print(f"error: cannot read {path}: {exc}", file=sys.stderr)
+            exit_code = 1
+            continue
+
+        report = fix_report(original, **opts.__dict__)
+
+        if args.report:
+            import json
+            payload = {
+                "path": str(path),
+                "changed": report.changed,
+                "shaped": report.shaped,
+                "bidi_reordered": report.bidi_reordered,
+                "normalized": report.normalized,
+                "contained_arabic": report.contained_arabic,
+                "warnings": report.warnings,
+            }
+            print(json.dumps(payload, ensure_ascii=False), file=sys.stderr)
+
+        if args.check:
+            if report.changed:
+                print(f"would change: {path}", file=sys.stderr)
+                exit_code = 1
+            continue
+
+        if not report.changed:
+            continue
+
+        target = args.output if args.output else path
+        try:
+            target.write_text(report.output, encoding=args.encoding)
+        except OSError as exc:
+            print(f"error: cannot write {target}: {exc}", file=sys.stderr)
+            exit_code = 1
+
+    return exit_code
+
+
+if __name__ == "__main__":  # pragma: no cover
+    sys.exit(main())

arabic_fix-0.4.0/arabic_fix/fixer.py

@@ -0,0 +1,163 @@
+"""The high-level `fix()` API — one call, all three fixes.
+
+Pipeline order matters: shape FIRST, then BiDi-reorder, then normalize.
+Skipping any step produces subtly-broken output in a different way:
+
+    normalize → shape  : marks get reshaped into presentation forms too
+                          late to fix; cached glyph tables may stay wrong.
+    shape → normalize → bidi : normalization AFTER shape may decompose
+                               presentation forms back to base chars.
+    bidi first         : without shaping, BiDi reorders disconnected
+                         glyphs and you can't fix it later.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Any
+
+from .shaper import shape
+from .bidi import reorder
+from .normalize import normalize, contains_arabic, NormForm
+from .bidi import _RTL_LTR_TO_BIDI  # for normalizing base_dir at the API edge
+
+
+@dataclass
+class FixOptions:
+    """Tune which fixes get applied and how."""
+    shape: bool = True
+    bidi: bool = True
+    normalize_text: bool = True
+    normalize_form: str = "NFC"        # 'NFC' | 'NFD' | 'NFKC' | 'NFKD'
+    bidi_base_dir: str = "rtl"         # 'rtl' | 'ltr'
+    bidi_upper_is_rtl: bool = False
+    ligatures: bool = True             # reshape-config: presentation-form ligatures
+
+
+@dataclass
+class FixReport:
+    """Returned by `fix_report()` so callers can see what changed."""
+    input: str
+    output: str
+    shaped: bool
+    bidi_reordered: bool
+    normalized: bool
+    contained_arabic: bool
+    warnings: list[str] = field(default_factory=list)
+
+    @property
+    def changed(self) -> bool:
+        return self.input != self.output
+
+
+def _coerce_options(opts: dict[str, Any]) -> FixOptions:
+    """Build FixOptions from arbitrary user kwargs, rejecting unknown keys.
+
+    We do this rather than `FixOptions(**opts)` so a typo'd option name
+    fails loudly with a helpful message instead of silently producing
+    a dataclass missing that field.
+    """
+    known = {f for f in FixOptions.__dataclass_fields__}
+    unknown = set(opts) - known
+    if unknown:
+        raise TypeError(
+            f"unknown fix() option(s): {sorted(unknown)}; "
+            f"valid options: {sorted(known)}"
+        )
+    # Normalize bidi_base_dir to its long form ('rtl'/'ltr') if the
+    # caller passed the single-char bidi code ('R'/'L').
+    if "bidi_base_dir" in opts:
+        b = opts["bidi_base_dir"]
+        if isinstance(b, str) and b in {"R", "L"}:
+            opts["bidi_base_dir"] = "rtl" if b == "R" else "ltr"
+    # Narrow normalize_form if it was passed as a valid literal.
+    if "normalize_form" in opts:
+        f = opts["normalize_form"]
+        if f not in {"NFC", "NFD", "NFKC", "NFKD"}:
+            raise ValueError(
+                f"normalize_form must be one of NFC/NFD/NFKC/NFKD, got {f!r}"
+            )
+        opts["normalize_form"] = f
+    return FixOptions(**opts)
+
+
+def fix(text: str, **opts: Any) -> str:
+    """One-call fix: shape, BiDi-reorder, normalize.
+
+    Returns the fixed string. For diagnostics, use `fix_report()`.
+
+    Examples
+    --------
+    >>> fix("العربية")
+    '\\u0627\\u0644\\u0639\\u0631\\u0628\\u064a\\u0629'  # shaped + BiDi-fixed
+    >>> fix("Hello مرحبا 123")
+    ...  # digits 123 end up on the left in display order
+    """
+    options = _coerce_options(opts) if opts else FixOptions()
+    return _apply(text, options).output
+
+
+def fix_report(text: str, **opts: Any) -> FixReport:
+    """Same as `fix()` but returns a `FixReport` describing what happened."""
+    options = _coerce_options(opts) if opts else FixOptions()
+    return _apply(text, options)
+
+
+def _apply(text: str, options: FixOptions) -> FixReport:
+    if not text:
+        return FixReport(
+            input=text,
+            output=text,
+            shaped=False,
+            bidi_reordered=False,
+            normalized=False,
+            contained_arabic=False,
+        )
+
+    warnings: list[str] = []
+    had_arabic = contains_arabic(text)
+    out = text
+    shaped_now = False
+    bidi_now = False
+    normalized_now = False
+
+    if options.normalize_text:
+        new = normalize(out, form=options.normalize_form)
+        if new != out:
+            normalized_now = True
+            out = new
+
+    if options.shape and had_arabic:
+        try:
+            new = shape(out, ligatures=options.ligatures)
+            if new != out:
+                shaped_now = True
+                out = new
+        except Exception as exc:  # pragma: no cover
+            warnings.append(f"shaper failed: {exc}")
+
+    if options.bidi and had_arabic:
+        try:
+            new = reorder(
+                out,
+                base_dir=options.bidi_base_dir,
+                upper_is_rtl=options.bidi_upper_is_rtl,
+            )
+            if new != out:
+                bidi_now = True
+                out = new
+        except Exception as exc:  # pragma: no cover
+            warnings.append(f"bidi reorder failed: {exc}")
+
+    return FixReport(
+        input=text,
+        output=out,
+        shaped=shaped_now,
+        bidi_reordered=bidi_now,
+        normalized=normalized_now,
+        contained_arabic=had_arabic,
+        warnings=warnings,
+    )
+
+
+__all__ = ["fix", "fix_report", "FixOptions", "FixReport"]