harfix 0.1.0__tar.gz

harfix-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,22 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python: ["3.10", "3.11", "3.12", "3.13"]
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+        with:
+          python-version: ${{ matrix.python }}
+      - run: uv sync --locked
+      - run: uv run ruff check .
+      - run: uv run ruff format --check .
+      - run: uv run pytest -q

harfix-0.1.0/.github/workflows/publish.yml

@@ -0,0 +1,21 @@
+name: Publish
+
+on:
+  push:
+    tags: ["v*"]
+
+jobs:
+  publish:
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write # PyPI trusted publishing; no API token stored in the repository
+    steps:
+      - uses: actions/checkout@v4
+      - uses: astral-sh/setup-uv@v6
+      - run: uv sync --locked
+      - run: uv run pytest -q
+      - name: Check the tag matches the package version
+        run: test "${GITHUB_REF_NAME#v}" = "$(uv run python -c 'import harfix; print(harfix.__version__)')"
+      - run: uv build
+      - uses: pypa/gh-action-pypi-publish@release/v1

harfix-0.1.0/.gitignore

@@ -0,0 +1,10 @@
+__pycache__/
+*.py[cod]
+.venv/
+build/
+dist/
+*.egg-info/
+.pytest_cache/
+.ruff_cache/
+.DS_Store
+corpus/

harfix-0.1.0/CHANGELOG.md

@@ -0,0 +1,20 @@
+# Changelog
+
+## 0.1.0
+
+First release.
+
+- `harfix scan` audits a PDF's text layer per page and per font and routes each page to
+  `keep`, `repair`, `reextract` or `ocr`.
+- `harfix check` and `assess_document` judge extractor output page by page (form feeds) or
+  in runs of lines, so a few bad pages are not averaged away by clean ones.
+- Detects wrong glyph-to-letter maps, unmapped glyphs, presentation forms, mirrored and
+  mixed reading order, lam-alef transposition and lost word boundaries.
+- Detects pre-Unicode Arabic fonts whose text layer extracts as ASCII gibberish (`AXt…`,
+  `Adv…` desktop-publishing fonts), not only those landing on Latin-1 symbols.
+- Detects Quran glyph-ID fonts (`QCF_P…`) as `quran_glyph_codes` and never sends them to
+  `repair`, where NFKC would turn them into wrong words.
+- Fonts too short to judge on any single page are judged on their short runs pooled across
+  the document (`"scope": "document"`).
+- Fonts are identified by their full PDF name including the subset tag, so two subsets of
+  one typeface with different glyph maps are reported separately.

harfix-0.1.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

harfix-0.1.0/NOTICE

@@ -0,0 +1,13 @@
+harfix
+Copyright 2026 Ahmad Sebaq
+
+This product includes a character n-gram model (src/harfix/data/arabic_char_lm.json.gz)
+whose letter statistics were derived from the following datasets, both licensed under the
+Apache License, Version 2.0:
+
+- Misraj/Misraj-DocOCR, revision 7177bf70f77ce259890d6af0bef53f18faf26ec9
+  https://huggingface.co/datasets/Misraj/Misraj-DocOCR
+- Misraj/KITAB_pdf_to_markdown_reviewed, revision 8890d721c660cd027839bc8bc48634552fe7aec6
+  https://huggingface.co/datasets/Misraj/KITAB_pdf_to_markdown_reviewed
+
+No text from these datasets is distributed with harfix; only aggregate character counts.

harfix-0.1.0/PKG-INFO

@@ -0,0 +1,212 @@
+Metadata-Version: 2.5
+Name: harfix
+Version: 0.1.0
+Summary: Quality gate for Arabic PDF text layers: detect garbled, reversed, or unmapped Arabic before it reaches your index.
+Project-URL: Homepage, https://github.com/asebaq/harfix
+Project-URL: Issues, https://github.com/asebaq/harfix/issues
+Project-URL: Changelog, https://github.com/asebaq/harfix/blob/main/CHANGELOG.md
+Author-email: Ahmad Sebaq <ahmad.sebaq@gmail.com>
+License-Expression: Apache-2.0
+License-File: LICENSE
+License-File: NOTICE
+Keywords: arabic,data-quality,ocr,pdf,rag,text-extraction
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Natural Language :: Arabic
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Text Processing :: Linguistic
+Requires-Python: >=3.10
+Requires-Dist: pdfminer-six>=20240706
+Description-Content-Type: text/markdown
+
+# harfix
+
+**A quality gate for Arabic text extracted from PDFs.** harfix tells you, per page and per
+font, whether the text you are about to index is trustworthy, needs a cheap repair, needs
+re-extraction with a different tool, or must go to OCR.
+
+Arabic PDFs fail silently. A page can render perfectly and still extract as the wrong
+letters (`المدعى عليه` → `الددعى عليهي`), as presentation-form code points that match no
+search query, as mirrored lines, or as pre-Unicode symbols (`Âd‡;« ≤∞`). Generic
+"broken text" detectors check for control characters and Latin mojibake, so they pass
+this text straight into your RAG index. harfix scores the letters themselves with a small
+Arabic character model and reports what is wrong and what to do about it.
+
+harfix never modifies text. It only decides.
+
+## Install
+
+```bash
+pip install harfix        # or: uv add harfix
+```
+
+Python 3.10+. The only runtime dependency is `pdfminer.six`; the two character models
+(~300 KiB) ship with the package.
+
+## Use
+
+Audit the text layer of PDFs:
+
+```console
+$ harfix scan --pages 8-12 --fail-on ocr mukhtasar_khuqayr.pdf
+mukhtasar_khuqayr.pdf
+  pages: keep=1, ocr=4
+  p9     ocr       legacy_encoding
+  p10    ocr       legacy_encoding
+  p11    ocr       legacy_encoding
+  p12    ocr       legacy_encoding
+```
+
+That book was typeset in pre-Unicode Arabic fonts (`AXt…`, `Adv…`), so its text layer
+extracts as Latin gibberish. `-v` lists every font on every page; `--json` prints the full
+report; `--fail-on repair|reextract|ocr` turns the scan into a CI or pipeline gate (exit
+code 1).
+
+Check text produced by any extractor (Docling, pdftotext, Marker, an OCR engine). Pages
+separated by form feeds (pdftotext's default) are judged one by one; other text in runs of
+whole lines holding about 500 Arabic letters, so a few bad pages are not averaged away:
+
+```console
+$ pdftotext -f 1 -l 12 al_mughalatat.pdf - | harfix check
+reextract: lam_alef_transposition, presentation_forms
+  pages: keep=2, repair=3, reextract=2
+  p4               repair    presentation_forms
+  p5               repair    presentation_forms
+  p6               reextract presentation_forms, lam_alef_transposition
+  p9               repair    presentation_forms
+  p11              reextract presentation_forms, lam_alef_transposition
+```
+
+Page numbers count from the start of the input; blank pages are skipped.
+
+From Python:
+
+```python
+from harfix import assess_document, assess_text, scan_pdf
+
+report = scan_pdf("book.pdf", pages=range(1, 11))
+for page in report.pages:
+    if page.route != "keep":
+        print(page.page, page.route, page.reasons)
+
+document = assess_document(extracted_text)  # extractor output, page by page
+document.route, [(s.page, s.assessment.route) for s in document.segments]
+
+verdict = assess_text(snippet)  # one piece of text: checks reading order too
+```
+
+## Routes and reasons
+
+| Route | Meaning | Reasons |
+|---|---|---|
+| `keep` | Index as is. | — |
+| `repair` | Text is right but encoded or ordered badly; a deterministic fix exists. | `presentation_forms` (apply NFKC), `visual_order` (reverse each line) |
+| `reextract` | The extractor damaged correct source text; try another extractor. | `mixed_reading_order`, `lam_alef_transposition`, `lost_word_boundaries` |
+| `ocr` | The text layer itself is wrong; read the pixels. | `same_script_substitution`, `unmapped_glyphs`, `legacy_encoding`, `quran_glyph_codes`, `no_text_layer` |
+
+`quran_glyph_codes`: Quran text set in the King Fahd Complex page fonts (`QCF_P…`) stores
+one glyph ID per word under Presentation Forms-A code points. NFKC would turn those IDs
+into wrong words, so they are never sent to `repair`; take the verse from a canonical Quran
+text or OCR it. The rest of the same text is still judged on its own letters.
+
+A page takes the most severe route among its font runs, so one broken heading font next to
+a healthy body font is reported precisely instead of being averaged away. A font whose runs
+are too short to judge on any one page (running heads, headings, verse numbers) is judged
+on its short runs pooled across the scanned pages; such font reports carry
+`"scope": "document"`. Fonts are identified by their PDF name including the subset tag
+(`ABCDEE+Lotus Linotype`), because two subsets of one typeface can map glyphs differently.
+
+`scan` (raw text layer) ignores reading order, which depends on the extractor. `check`
+(final extractor output) also reports order and spacing faults.
+
+## How it works
+
+1. Text is reduced to base Arabic letters (presentation forms decomposed, diacritics,
+   tatweel and decorative letter runs removed).
+2. Each line's reading direction is decided by comparing the model's score forwards and
+   backwards, so mirrored text is not mistaken for corruption.
+3. A 4-gram character model with interpolated Witten-Bell smoothing scores the corrected
+   letters. Wrong glyph-to-letter maps produce improbable letter sequences (high bits per
+   character) or isolated very improbable letters (spikes).
+4. A second model trained without spaces separates "the letters are wrong" from "the
+   extractor broke the word spacing".
+5. Character-class checks catch private-use glyphs, raw `(cid:N)` codes, presentation
+   forms, Quran glyph-ID fonts, and pre-Unicode encodings without any model. Pre-Unicode
+   fonts land either on Latin-1 and symbol characters or on ASCII, where they read as
+   Latin "words" with almost no vowels or with digits, symbols and case flips inside
+   (`otsrqponmlkA{zvjihgf`, `h&c%G`); English text has about 40% vowels and few such words.
+
+## Accuracy
+
+Full numbers: [`reports/calibration.md`](reports/calibration.md). Thresholds were fitted on
+one held-out split of 462 public Arabic pages and measured on another, then raised
+(`DOMAIN_FLOORS`) after inspecting clean runs in Egyptian court and statute PDFs.
+
+On the test split, windows of at least 80 Arabic letters:
+
+| Case | Flagged correctly |
+|---|---:|
+| Clean text left as `keep` (PDF scan / extractor output) | 100% / 99.5% |
+| Whole font with a wrong letter map | 100% |
+| Private-use glyphs, raw CID codes, Latin-1 legacy encodings, presentation forms | 100% |
+| Quran glyph-ID fonts | 100% |
+| ASCII legacy encodings | 96% (100% at ≥500 letters) |
+| Mirrored lines detected (`check`) | 100% |
+| Lam-alef transposition (`check`) | 90% (97% at ≥200 letters) |
+| Lost word boundaries (`check`) | 99.8% |
+| Padding letters inserted (seen in Egyptian court books) | 72% (87% at ≥500 letters) |
+| Only 6 of 36 letters swapped | 40% (52% at ≥500 letters) |
+
+Real books: `scripts/fetch_shamela.py` samples printed-edition PDFs listed in
+al-Maktaba al-Shamela (two per subject category, seed 0); none was used for any threshold.
+Of 46 distinct books (first 40 pages each), 33 are image-only scans (`no_text_layer`). Of
+the 13 with a text layer, 12 have fonts flagged on at least some pages: pre-Unicode `AXt`,
+`Adv…` and `MCS` fonts (`legacy_encoding`), fonts with no Unicode map at all
+(`unmapped_glyphs`), a `QCF_P206` verse font (`quran_glyph_codes`), Lotus and Uthman Taha
+fonts whose text is fragmented with stray dammas (`same_script_substitution`), Yakout text
+stored as presentation forms (`repair`), and OCR text layers with garbled title pages.
+Samples of flagged fonts in 8 of those 12 books, covering every kind above, were inspected
+by hand: all were genuinely broken, but one was misattributed (a Lotus layer with correct
+letters and no word spaces is reported as `same_script_substitution`, see Limitations).
+The inspection also caught stray Lotus ligature glyphs misread as a Quran font, which led
+to the consecutive-code-point condition. This is a spot check, not a labeled evaluation.
+
+## Limitations
+
+- The model is trained on about 300 pages of modern standard Arabic. Classical, dialectal,
+  or heavily domain-specific text scores higher; calibrate before relying on it there.
+- Precision is preferred over recall: partial letter swaps and light padding are missed
+  part of the time.
+- `scan` cannot tell a text layer without word spaces from one with wrong letters: both
+  are reported as `same_script_substitution`.
+- A pre-Unicode font is caught only once its runs hold about 15 Latin-looking words, and a
+  font mapped entirely onto digits and punctuation is not caught at all.
+- `check` splits text without form feeds at line ends, so a single very long line is
+  judged as one piece.
+- Scanned pages are routed to OCR but their image quality is not assessed.
+- PDF parsing uses pdfminer.six: about 0.1–0.2 s per page on the documents above.
+
+## Roadmap
+
+- Text-level repairs for the `repair` route (NFKC, line reversal) with provenance.
+- Per-font decoder for legacy and custom Arabic fonts: learn each font's glyph-to-letter
+  map once (from a typed edition of the same book, a library of known fonts, or the letter
+  model itself, with OCR only as a fallback), then decode the whole document.
+- Docling, LangChain and LlamaIndex integrations.
+- A public benchmark of broken Arabic PDFs with labels per failure class.
+
+## Rebuilding the model and thresholds
+
+```bash
+uv sync --group model
+uv run python scripts/build_model.py    # pinned Hugging Face revisions, reproducible output
+uv run python scripts/fetch_shamela.py  # needs the Shamela desktop app's catalogue
+uv run python scripts/calibrate.py --pages 1-40 --pdf corpus/shamela/*.pdf
+```
+
+The Shamela PDFs are third-party works: keep them local, never commit or redistribute them.
+
+## License
+
+Apache-2.0. The packaged character statistics are derived from Apache-2.0 datasets by
+Misraj; see [`NOTICE`](NOTICE).