indic-pdf-check 0.2.0__tar.gz

indic_pdf_check-0.2.0/LICENSE

@@ -0,0 +1,202 @@
+
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

indic_pdf_check-0.2.0/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: indic-pdf-check
+Version: 0.2.0
+Summary: Can a screen reader, search engine or AI read this PDF? Detects scanned and broken text layers in Indian and South/Southeast Asian script PDFs.
+Author-email: Novain Shibu <georgeshibu380@gmail.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/novain-shibu/indic-pdf-check
+Project-URL: Research report, https://github.com/novain-shibu/finepdfs-audit
+Keywords: pdf,accessibility,screen reader,indic,unicode,malayalam,hindi,tamil,khmer,myanmar
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Topic :: Text Processing
+Classifier: Intended Audience :: Developers
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pymupdf>=1.23
+Dynamic: license-file
+
+# indic-pdf-check
+
+**Can a screen reader, a search engine or an AI read the text of this PDF?**
+
+Many PDFs in Indian and other South/Southeast Asian scripts look perfect on screen but carry a broken text layer: copy-paste,
+search, screen readers and AI pipelines get garbled words. Microsoft Word's "Save as PDF" is a common cause (see the
+[research report](https://github.com/novain-shibu/finepdfs-audit)). Others are scans with no text at all. This tool reads a
+PDF's own text layer and tells you which case you have, in a second, on your own computer.
+
+**Next steps after `BROKEN`:** recover the text with [Unshape](https://github.com/novain-shibu/unshape); verify that extracted text matches what people see with [SeenRead](https://github.com/novain-shibu/seenread).
+
+```bash
+pip install indic-pdf-check
+indic-pdf-check circular.pdf notice.pdf
+```
+
+```
+circular.pdf: BROKEN             script=Malayalam letters=1994 broken=29.4% tagged=True creator='Microsoft® Word for Microsoft '
+notice.pdf:   NO_TEXT            script=None letters=0 broken=0.0% tagged=False creator='Canon '
+-- 2 PDFs: BROKEN 1, NO_TEXT 1
+```
+
+| verdict | meaning | what to do |
+|---|---|---|
+| `NO_TEXT` | almost no text at all: usually a scan or an image | run OCR (e.g. OCRmyPDF with Tesseract language packs) before publishing |
+| `INSUFFICIENT_TEXT` | some text in a supported script, but under 200 letters: too little to judge | check by hand (e.g. a short notice) |
+| `UNSUPPORTED_SCRIPT` | plenty of text, but none in a supported script (e.g. English only) | this tool cannot judge it |
+| `BROKEN` | the main script, or any other supported script with 200+ letters, has more than 5% impossible spellings (e.g. a vowel sign or virama with no consonant before it) | re-export with another tool and check again; or recover the text with [Unshape](https://github.com/novain-shibu/unshape) (needs the document's original font) |
+| `OK` | no problem detected by these checks | - (this is not a full accessibility check: reading order, headings, tables and alternative text are not checked) |
+
+Pages whose own text has a high share of impossible spellings are listed as `suspicious_pages` (with `--json`), so damage
+confined to some pages or to a second script is visible.
+
+Also reported: the script, the share of broken words, whether the PDF is tagged for accessibility, and the program that
+made it. `--json` gives machine-readable output; the exit code is 1 if any file is `NO_TEXT` or `BROKEN` (useful in CI).
+
+**Scripts:** Devanagari (Hindi, Marathi, Nepali...), Bengali/Assamese, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada,
+Malayalam, Sinhala, Thai, Lao, Khmer, Myanmar, Thaana (Divehi).
+
+**Python:**
+```python
+from indic_pdf_check import check, check_text
+check("notice.pdf")["verdict"]          # 'BROKEN' | 'OK' | 'NO_TEXT' | 'INSUFFICIENT_TEXT' | 'UNSUPPORTED_SCRIPT'
+```
+
+## How well it works (tested before release)
+- Word for Microsoft 365 PDFs with known-broken text in 14 scripts: 12 flagged `BROKEN`; Kannada was passed as `OK`
+  (only 4% impossible spellings although many words are wrong - see limits).
+- Clean source text in the same 14 scripts: 0 false alarms.
+- Microsoft Edge PDFs of the same text are flagged in 3 of 4 scripts: they really do contain doubled vowel signs, which a
+  screen reader also reads.
+
+## Limits
+- It finds *impossible* spellings. Garbled text can still be spelled legally (e.g. a wrong consonant), so `OK` means "no
+  sign of breakage", not "verified correct".
+- Scripts not listed (Arabic, Hebrew, Latin, Chinese...) come out as `UNSUPPORTED_SCRIPT` (version 0.1.0 said `NO_TEXT`).
+- Version 0.2.0 kept every verdict of the release tests above unchanged; it only splits the old `NO_TEXT` into three
+  clearer cases and adds per-script and per-page checks.
+- It reads only the text layer; it does not look at how the page renders.
+
+Apache-2.0 · Novain Shibu · [github.com/novain-shibu](https://github.com/novain-shibu)
+
+It uses PyMuPDF (installed separately), which is licensed under AGPL-3.0 or a commercial licence from Artifex; using the two
+together is subject to PyMuPDF's licence.

indic_pdf_check-0.2.0/README.md

@@ -0,0 +1,64 @@
+# indic-pdf-check
+
+**Can a screen reader, a search engine or an AI read the text of this PDF?**
+
+Many PDFs in Indian and other South/Southeast Asian scripts look perfect on screen but carry a broken text layer: copy-paste,
+search, screen readers and AI pipelines get garbled words. Microsoft Word's "Save as PDF" is a common cause (see the
+[research report](https://github.com/novain-shibu/finepdfs-audit)). Others are scans with no text at all. This tool reads a
+PDF's own text layer and tells you which case you have, in a second, on your own computer.
+
+**Next steps after `BROKEN`:** recover the text with [Unshape](https://github.com/novain-shibu/unshape); verify that extracted text matches what people see with [SeenRead](https://github.com/novain-shibu/seenread).
+
+```bash
+pip install indic-pdf-check
+indic-pdf-check circular.pdf notice.pdf
+```
+
+```
+circular.pdf: BROKEN             script=Malayalam letters=1994 broken=29.4% tagged=True creator='Microsoft® Word for Microsoft '
+notice.pdf:   NO_TEXT            script=None letters=0 broken=0.0% tagged=False creator='Canon '
+-- 2 PDFs: BROKEN 1, NO_TEXT 1
+```
+
+| verdict | meaning | what to do |
+|---|---|---|
+| `NO_TEXT` | almost no text at all: usually a scan or an image | run OCR (e.g. OCRmyPDF with Tesseract language packs) before publishing |
+| `INSUFFICIENT_TEXT` | some text in a supported script, but under 200 letters: too little to judge | check by hand (e.g. a short notice) |
+| `UNSUPPORTED_SCRIPT` | plenty of text, but none in a supported script (e.g. English only) | this tool cannot judge it |
+| `BROKEN` | the main script, or any other supported script with 200+ letters, has more than 5% impossible spellings (e.g. a vowel sign or virama with no consonant before it) | re-export with another tool and check again; or recover the text with [Unshape](https://github.com/novain-shibu/unshape) (needs the document's original font) |
+| `OK` | no problem detected by these checks | - (this is not a full accessibility check: reading order, headings, tables and alternative text are not checked) |
+
+Pages whose own text has a high share of impossible spellings are listed as `suspicious_pages` (with `--json`), so damage
+confined to some pages or to a second script is visible.
+
+Also reported: the script, the share of broken words, whether the PDF is tagged for accessibility, and the program that
+made it. `--json` gives machine-readable output; the exit code is 1 if any file is `NO_TEXT` or `BROKEN` (useful in CI).
+
+**Scripts:** Devanagari (Hindi, Marathi, Nepali...), Bengali/Assamese, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada,
+Malayalam, Sinhala, Thai, Lao, Khmer, Myanmar, Thaana (Divehi).
+
+**Python:**
+```python
+from indic_pdf_check import check, check_text
+check("notice.pdf")["verdict"]          # 'BROKEN' | 'OK' | 'NO_TEXT' | 'INSUFFICIENT_TEXT' | 'UNSUPPORTED_SCRIPT'
+```
+
+## How well it works (tested before release)
+- Word for Microsoft 365 PDFs with known-broken text in 14 scripts: 12 flagged `BROKEN`; Kannada was passed as `OK`
+  (only 4% impossible spellings although many words are wrong - see limits).
+- Clean source text in the same 14 scripts: 0 false alarms.
+- Microsoft Edge PDFs of the same text are flagged in 3 of 4 scripts: they really do contain doubled vowel signs, which a
+  screen reader also reads.
+
+## Limits
+- It finds *impossible* spellings. Garbled text can still be spelled legally (e.g. a wrong consonant), so `OK` means "no
+  sign of breakage", not "verified correct".
+- Scripts not listed (Arabic, Hebrew, Latin, Chinese...) come out as `UNSUPPORTED_SCRIPT` (version 0.1.0 said `NO_TEXT`).
+- Version 0.2.0 kept every verdict of the release tests above unchanged; it only splits the old `NO_TEXT` into three
+  clearer cases and adds per-script and per-page checks.
+- It reads only the text layer; it does not look at how the page renders.
+
+Apache-2.0 · Novain Shibu · [github.com/novain-shibu](https://github.com/novain-shibu)
+
+It uses PyMuPDF (installed separately), which is licensed under AGPL-3.0 or a commercial licence from Artifex; using the two
+together is subject to PyMuPDF's licence.

indic_pdf_check-0.2.0/pyproject.toml

@@ -0,0 +1,26 @@
+[build-system]
+requires = ["setuptools>=61"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "indic-pdf-check"
+version = "0.2.0"
+description = "Can a screen reader, search engine or AI read this PDF? Detects scanned and broken text layers in Indian and South/Southeast Asian script PDFs."
+readme = "README.md"
+license = {text = "Apache-2.0"}
+authors = [{name = "Novain Shibu", email = "georgeshibu380@gmail.com"}]
+requires-python = ">=3.9"
+dependencies = ["pymupdf>=1.23"]
+keywords = ["pdf", "accessibility", "screen reader", "indic", "unicode", "malayalam", "hindi", "tamil", "khmer", "myanmar"]
+classifiers = ["Programming Language :: Python :: 3", "License :: OSI Approved :: Apache Software License",
+               "Topic :: Text Processing", "Intended Audience :: Developers"]
+
+[project.urls]
+Homepage = "https://github.com/novain-shibu/indic-pdf-check"
+"Research report" = "https://github.com/novain-shibu/finepdfs-audit"
+
+[project.scripts]
+indic-pdf-check = "indic_pdf_check.cli:main"
+
+[tool.setuptools.packages.find]
+where = ["src"]

indic_pdf_check-0.2.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

indic_pdf_check-0.2.0/src/indic_pdf_check/__init__.py

@@ -0,0 +1,95 @@
+"""indic-pdf-check: can a screen reader, search engine or AI read the text of this PDF?
+
+Many PDFs in Indian and other South/Southeast Asian scripts look correct on screen but carry a broken text layer (for
+example Microsoft Word's "Save as PDF"), or carry no text at all (scans). This tool reads the PDF's own text layer - the
+text that copy-paste, search, screen readers and AI pipelines get - and reports one verdict per PDF:
+
+  NO_TEXT             almost no text at all: usually a scan/image (needs OCR)
+  INSUFFICIENT_TEXT   some text in a supported script, but too little to judge (under 200 letters)
+  UNSUPPORTED_SCRIPT  plenty of text, but none in a supported script, so this tool cannot judge it
+  BROKEN              the main script, or any other supported script with 200+ letters, has more than 5% impossible
+                      spellings (e.g. a vowel sign or virama with no consonant before it)
+  OK                  no problem detected by these checks (this does not mean "accessible": reading order, headings,
+                      tables and alternative text are not checked)
+Pages whose own text has a high share of impossible spellings are listed as "suspicious_pages" (for information).
+
+Supported scripts: Devanagari, Bengali/Assamese, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada, Malayalam, Sinhala,
+Thai, Lao, Khmer, Myanmar, Thaana. It cannot prove text is *correct* (a garbled text can still be spelled legally); it
+catches the impossible spellings that broken PDF text layers produce. Nothing is uploaded; files are only read.
+"""
+from pathlib import Path
+import unicodedata
+
+from .rules import BASE, OTHER, scan_text
+
+__version__ = "0.2.0"
+SCRIPTS = list(BASE) + list(OTHER)
+NAMES = {"Deva": "Devanagari", "Beng": "Bengali", "Guru": "Gurmukhi", "Gujr": "Gujarati", "Orya": "Odia", "Taml": "Tamil",
+         "Telu": "Telugu", "Knda": "Kannada", "Mlym": "Malayalam", "Sinh": "Sinhala", "Thai": "Thai", "Laoo": "Lao",
+         "Khmr": "Khmer", "Mymr": "Myanmar", "Thaa": "Thaana"}
+
+
+def _block(script):
+    if script in BASE:
+        return BASE[script], BASE[script] + 0x7F
+    return OTHER[script][0]
+
+
+MIN_LETTERS = 200   # below this the text is too short to judge
+NO_TEXT_CHARS = 20  # fewer non-space characters than this in total: treat as no text layer
+PAGE_WORDS = 30     # pages need at least this many words in a script to be listed as suspicious
+
+
+def _letters(text, script):
+    lo, hi = _block(script)
+    return sum(1 for c in text if lo <= ord(c) <= hi and unicodedata.category(c)[0] in "LM")
+
+
+def _script_stats(text):
+    """[(script, letters, words, bad)] for every supported script present, most letters first."""
+    out = []
+    for s in SCRIPTS:
+        n = _letters(text, s)
+        if n:
+            w, b = scan_text(text, s)
+            out.append((s, n, w, b))
+    return sorted(out, key=lambda t: -t[1])
+
+
+def check_text(text):
+    """Verdict for a text string (the text layer of a PDF)."""
+    stats = _script_stats(text)
+    script, letters, words, bad = stats[0] if stats else (None, 0, 0, 0)
+    share = bad / words if words else 0.0
+    others = [(s, b / w) for s, n, w, b in stats[1:] if n >= MIN_LETTERS and w and b / w > 0.05]
+    if letters >= MIN_LETTERS:
+        verdict = "BROKEN" if share > 0.05 or others else "OK"
+    elif letters > 0:
+        verdict = "INSUFFICIENT_TEXT"
+    elif sum(1 for c in text if not c.isspace()) < NO_TEXT_CHARS:
+        verdict = "NO_TEXT"
+    else:
+        verdict = "UNSUPPORTED_SCRIPT"
+    return {"verdict": verdict, "script": NAMES[script] if script else None, "letters": letters, "words": words,
+            "broken_share": round(share, 4),
+            "other_broken_scripts": [{"script": NAMES[s], "broken_share": round(x, 4)} for s, x in others]}
+
+
+def check(path):
+    """Verdict for a PDF file: dict with verdict, script, letters, words, broken_share, pages, tagged, producer, creator."""
+    import pymupdf
+    pymupdf.TOOLS.mupdf_display_errors(False)
+    doc = pymupdf.open(path)
+    pages = [page.get_text() for page in doc]
+    r = check_text("".join(pages))
+    sus = []
+    for i, t in enumerate(pages, 1):
+        for s, n, w, b in _script_stats(t):
+            if w >= PAGE_WORDS and b / w > 0.05:
+                sus.append({"page": i, "script": NAMES[s], "broken_share": round(b / w, 4)})
+    r["suspicious_pages"] = sus
+    md = doc.metadata or {}
+    cat = doc.pdf_catalog()
+    r.update({"file": Path(path).name, "pages": len(doc), "tagged": bool(cat and "StructTreeRoot" in doc.xref_object(cat)),
+              "producer": md.get("producer", ""), "creator": md.get("creator", "")})
+    return r

indic_pdf_check-0.2.0/src/indic_pdf_check/cli.py

@@ -0,0 +1,34 @@
+"""Command line: indic-pdf-check file1.pdf [file2.pdf ...] [--json]"""
+import collections, json, sys
+from . import __version__, check
+
+
+def main(argv=None):
+    argv = sys.argv[1:] if argv is None else argv
+    if not argv or argv[0] in ("-h", "--help"):
+        print(__doc__ + "\nVerdicts: NO_TEXT (no text layer, usually a scan), INSUFFICIENT_TEXT (too little text to judge), UNSUPPORTED_SCRIPT, BROKEN (garbled text layer), OK (no problem detected; not a full accessibility check)."); return 0
+    if argv[0] == "--version":
+        print(__version__); return 0
+    as_json = "--json" in argv
+    rows = []
+    for f in (a for a in argv if a != "--json"):
+        try:
+            rows.append(check(f))
+        except Exception as e:
+            rows.append({"file": f, "verdict": "ERROR", "error": repr(e)[:120]})
+    if as_json:
+        print(json.dumps(rows, ensure_ascii=False, indent=1))
+    else:
+        for r in rows:
+            if r["verdict"] == "ERROR":
+                print(f"{r['file']}: ERROR {r['error']}"); continue
+            print(f"{r['file']}: {r['verdict']:18s} script={r['script']} letters={r['letters']} broken={r['broken_share']:.1%} "
+                  f"tagged={r['tagged']} creator={r['creator'][:30]!r}"
+                  + (f" suspicious_pages={len(r['suspicious_pages'])}" if r.get("suspicious_pages") else ""))
+        c = collections.Counter(r["verdict"] for r in rows)
+        print(f"-- {len(rows)} PDFs: " + ", ".join(f"{k} {v}" for k, v in c.most_common()))
+    return 1 if any(r["verdict"] in ("BROKEN", "NO_TEXT") for r in rows) else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

indic_pdf_check-0.2.0/src/indic_pdf_check/rules.py

@@ -0,0 +1,119 @@
+"""Orthographic validity checker for Brahmic-script text (Devanagari, Bengali, Tamil, Malayalam).
+Scrambled PDF text layers (visual-order vowel signs, ToUnicode lies, dropped glyphs) create sequences that cannot occur in
+correctly spelled Unicode text. Rules (Unicode block offsets are shared across these ISCII-derived blocks):
+  R1 dependent vowel sign not preceded by a consonant / nukta (or a vowel sign for the length marks)
+  R2 virama not preceded by a consonant / nukta
+  R3 candrabindu / anusvara / visarga at the start of a word
+A word = maximal run of characters from the script's block (+ ZWJ/ZWNJ). Metric: share of words with >= 1 violation.
+Usage: python -I validity.py <finepdfs_sample.json> [...]   (also checks Wikipedia baselines via --wiki)
+"""
+import json, re, sys, statistics, unicodedata
+BASE = {"Deva": 0x0900, "Beng": 0x0980, "Guru": 0x0A00, "Gujr": 0x0A80, "Orya": 0x0B00, "Taml": 0x0B80,
+        "Telu": 0x0C00, "Knda": 0x0C80, "Mlym": 0x0D00}   # Sinhala (0D80) has a different layout: not covered
+
+def classes(base):
+    cons = set(range(base + 0x15, base + 0x3A)) | set(range(base + 0x58, base + 0x60))
+    if base == 0x0D00:
+        cons |= {0x0D3A}
+    if base == 0x0B00:
+        cons |= {0x0B71}
+    if base == 0x0980:
+        cons |= {0x09F0, 0x09F1}       # Assamese ra / wa (skeptic-style fix: Assamese Wikipedia scored 26% without these)
+    dv = set(range(base + 0x3E, base + 0x4D)) | {base + 0x62, base + 0x63}
+    length = {base + 0x55, base + 0x56, base + 0x57}
+    signs = {base + 0x01, base + 0x02, base + 0x03}
+    if base == 0x0B80:
+        signs = {0x0B82}               # Tamil aytham (U+0B83) is a letter, legal at word start
+    viramas, nukta = {base + 0x4D}, base + 0x3C
+    if base == 0x0D00:                 # skeptic fixes: U+0D3B/0D3C are viramas (no nukta in Malayalam); chillus 0D7A-0D7F
+        viramas, nukta = {0x0D4D, 0x0D3B, 0x0D3C}, None
+    return cons, dv, length, viramas, nukta, signs
+
+#   Non-ISCII complex scripts: (block range, base letters, marks that must follow a base or another mark, word-start-only bad)
+OTHER = {
+    "Thai": ((0x0E00, 0x0E7F), set(range(0x0E01, 0x0E2F)), {0x0E31} | set(range(0x0E34, 0x0E3B)) | set(range(0x0E47, 0x0E4F))),
+    "Khmr": ((0x1780, 0x17FF), set(range(0x1780, 0x17B4)), set(range(0x17B6, 0x17D4))),
+    "Sinh": ((0x0D80, 0x0DFF), set(range(0x0D85, 0x0DC7)), set(range(0x0DCA, 0x0DE0)) | {0x0DF2, 0x0DF3, 0x0D82, 0x0D83}),
+    "Mymr": ((0x1000, 0x109F), set(range(0x1000, 0x102B)) | set(range(0x103F, 0x1056)), set(range(0x102B, 0x103F)) | set(range(0x1056, 0x105A))),
+    "Laoo": ((0x0E80, 0x0EFF), set(range(0x0E81, 0x0EB1)) | {0x0EB2, 0x0EB3} | set(range(0x0EDC, 0x0EE0)), {0x0EB1} | set(range(0x0EB4, 0x0EBD)) | set(range(0x0EC8, 0x0ECE))),
+    "Thaa": ((0x0780, 0x07BF), set(range(0x0780, 0x07A6)) | {0x07B1}, set(range(0x07A6, 0x07B1))),
+}
+PUA_THAI = set(range(0xF700, 0xF71B))   # legacy Thai font glyph-variant code points leaking into text = broken text layer
+
+
+def other_word_ok(w, script):
+    (lo, hi), base, marks = OTHER[script]
+    cps = [ord(c) for c in w if c not in "‌‍"]
+    for i, c in enumerate(cps):
+        if c in PUA_THAI:
+            return False
+        if c in marks:
+            if i == 0:
+                return False
+            p = cps[i - 1]
+            if p not in base and p not in marks:
+                return False
+    return True
+
+
+def scan_other(text, script, collect=None):
+    text = unicodedata.normalize("NFC", text)
+    (lo, hi), base, marks = OTHER[script]
+    words = re.findall(f"[\\u{lo:04x}-\\u{hi:04x}\\u200c\\u200d\\uf700-\\uf71a]+", text)
+    words = [w for w in words if any(lo <= ord(c) <= hi for c in w)]
+    badw = [w for w in words if not other_word_ok(w, script)]
+    if collect is not None:
+        collect.extend(badw)
+    return len(words), len(badw)
+
+
+def word_ok(w, cl):
+    cons, dv, length, viramas, nukta, signs = cl
+    cps = [ord(c) for c in w if c not in "‌‍"]
+    mal = 0x0D4D in viramas
+    for i, c in enumerate(cps):
+        prev = cps[i - 1] if i else None
+        if c in dv and prev not in cons and not (nukta is not None and prev == nukta):
+            return False
+        if c in length and prev not in dv and prev not in cons:
+            return False
+        if c in viramas and prev not in cons and not (nukta is not None and prev == nukta):
+            # Malayalam: chillu + virama (nta form) and samvruthokaram (u-sign + virama) are standard spellings
+            if not (mal and prev is not None and (0x0D7A <= prev <= 0x0D7F or prev == 0x0D41)):
+                return False
+        if c in signs and i == 0:
+            return False
+    return True
+
+def scan_text(text, script, collect=None):
+    if script in OTHER:
+        return scan_other(text, script, collect)
+    text = unicodedata.normalize("NFC", text)
+    base = BASE[script]
+    cl = classes(base)
+    words = re.findall(f"[\\u{base:04x}-\\u{base + 0x7F:04x}\\u200c\\u200d]+", text)
+    words = [w for w in words if any(base <= ord(c) <= base + 0x7F for c in w)]
+    badw = [w for w in words if not word_ok(w, cl)]
+    if collect is not None:
+        collect.extend(badw)
+    return len(words), len(badw)
+
+def report(name, docs, script):
+    W = B = 0; per = []
+    for t in docs:
+        w, b = scan_text(t, script)
+        W += w; B += b
+        if w >= 50:
+            per.append(b / w)
+    over = sum(p > 0.05 for p in per)
+    print(f"{name:28s} docs {len(docs):4d} | words {W:8d} | invalid {B:7d} = {B / max(W, 1):6.2%} | "
+          f"docs(>=50 words) with >5% invalid: {over}/{len(per)}"
+          + (f" | median doc {statistics.median(per):.2%}" if per else ""))
+    return per
+
+if __name__ == "__main__":
+    for path in sys.argv[1:]:
+        script = re.search(r"_(" + "|".join(list(BASE) + list(OTHER)) + ")", path).group(1)
+        data = json.load(open(path, encoding="utf-8"))
+        docs = [d["text"] if isinstance(d, dict) else d for d in data]
+        report(path.split("/")[-1].split("\\")[-1], docs, script)

indic_pdf_check-0.2.0/src/indic_pdf_check.egg-info/PKG-INFO

@@ -0,0 +1,83 @@
+Metadata-Version: 2.4
+Name: indic-pdf-check
+Version: 0.2.0
+Summary: Can a screen reader, search engine or AI read this PDF? Detects scanned and broken text layers in Indian and South/Southeast Asian script PDFs.
+Author-email: Novain Shibu <georgeshibu380@gmail.com>
+License: Apache-2.0
+Project-URL: Homepage, https://github.com/novain-shibu/indic-pdf-check
+Project-URL: Research report, https://github.com/novain-shibu/finepdfs-audit
+Keywords: pdf,accessibility,screen reader,indic,unicode,malayalam,hindi,tamil,khmer,myanmar
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Topic :: Text Processing
+Classifier: Intended Audience :: Developers
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pymupdf>=1.23
+Dynamic: license-file
+
+# indic-pdf-check
+
+**Can a screen reader, a search engine or an AI read the text of this PDF?**
+
+Many PDFs in Indian and other South/Southeast Asian scripts look perfect on screen but carry a broken text layer: copy-paste,
+search, screen readers and AI pipelines get garbled words. Microsoft Word's "Save as PDF" is a common cause (see the
+[research report](https://github.com/novain-shibu/finepdfs-audit)). Others are scans with no text at all. This tool reads a
+PDF's own text layer and tells you which case you have, in a second, on your own computer.
+
+**Next steps after `BROKEN`:** recover the text with [Unshape](https://github.com/novain-shibu/unshape); verify that extracted text matches what people see with [SeenRead](https://github.com/novain-shibu/seenread).
+
+```bash
+pip install indic-pdf-check
+indic-pdf-check circular.pdf notice.pdf
+```
+
+```
+circular.pdf: BROKEN             script=Malayalam letters=1994 broken=29.4% tagged=True creator='Microsoft® Word for Microsoft '
+notice.pdf:   NO_TEXT            script=None letters=0 broken=0.0% tagged=False creator='Canon '
+-- 2 PDFs: BROKEN 1, NO_TEXT 1
+```
+
+| verdict | meaning | what to do |
+|---|---|---|
+| `NO_TEXT` | almost no text at all: usually a scan or an image | run OCR (e.g. OCRmyPDF with Tesseract language packs) before publishing |
+| `INSUFFICIENT_TEXT` | some text in a supported script, but under 200 letters: too little to judge | check by hand (e.g. a short notice) |
+| `UNSUPPORTED_SCRIPT` | plenty of text, but none in a supported script (e.g. English only) | this tool cannot judge it |
+| `BROKEN` | the main script, or any other supported script with 200+ letters, has more than 5% impossible spellings (e.g. a vowel sign or virama with no consonant before it) | re-export with another tool and check again; or recover the text with [Unshape](https://github.com/novain-shibu/unshape) (needs the document's original font) |
+| `OK` | no problem detected by these checks | - (this is not a full accessibility check: reading order, headings, tables and alternative text are not checked) |
+
+Pages whose own text has a high share of impossible spellings are listed as `suspicious_pages` (with `--json`), so damage
+confined to some pages or to a second script is visible.
+
+Also reported: the script, the share of broken words, whether the PDF is tagged for accessibility, and the program that
+made it. `--json` gives machine-readable output; the exit code is 1 if any file is `NO_TEXT` or `BROKEN` (useful in CI).
+
+**Scripts:** Devanagari (Hindi, Marathi, Nepali...), Bengali/Assamese, Gurmukhi, Gujarati, Odia, Tamil, Telugu, Kannada,
+Malayalam, Sinhala, Thai, Lao, Khmer, Myanmar, Thaana (Divehi).
+
+**Python:**
+```python
+from indic_pdf_check import check, check_text
+check("notice.pdf")["verdict"]          # 'BROKEN' | 'OK' | 'NO_TEXT' | 'INSUFFICIENT_TEXT' | 'UNSUPPORTED_SCRIPT'
+```
+
+## How well it works (tested before release)
+- Word for Microsoft 365 PDFs with known-broken text in 14 scripts: 12 flagged `BROKEN`; Kannada was passed as `OK`
+  (only 4% impossible spellings although many words are wrong - see limits).
+- Clean source text in the same 14 scripts: 0 false alarms.
+- Microsoft Edge PDFs of the same text are flagged in 3 of 4 scripts: they really do contain doubled vowel signs, which a
+  screen reader also reads.
+
+## Limits
+- It finds *impossible* spellings. Garbled text can still be spelled legally (e.g. a wrong consonant), so `OK` means "no
+  sign of breakage", not "verified correct".
+- Scripts not listed (Arabic, Hebrew, Latin, Chinese...) come out as `UNSUPPORTED_SCRIPT` (version 0.1.0 said `NO_TEXT`).
+- Version 0.2.0 kept every verdict of the release tests above unchanged; it only splits the old `NO_TEXT` into three
+  clearer cases and adds per-script and per-page checks.
+- It reads only the text layer; it does not look at how the page renders.
+
+Apache-2.0 · Novain Shibu · [github.com/novain-shibu](https://github.com/novain-shibu)
+
+It uses PyMuPDF (installed separately), which is licensed under AGPL-3.0 or a commercial licence from Artifex; using the two
+together is subject to PyMuPDF's licence.

indic_pdf_check-0.2.0/src/indic_pdf_check.egg-info/SOURCES.txt

@@ -0,0 +1,12 @@
+LICENSE
+README.md
+pyproject.toml
+src/indic_pdf_check/__init__.py
+src/indic_pdf_check/cli.py
+src/indic_pdf_check/rules.py
+src/indic_pdf_check.egg-info/PKG-INFO
+src/indic_pdf_check.egg-info/SOURCES.txt
+src/indic_pdf_check.egg-info/dependency_links.txt
+src/indic_pdf_check.egg-info/entry_points.txt
+src/indic_pdf_check.egg-info/requires.txt
+src/indic_pdf_check.egg-info/top_level.txt

indic_pdf_check-0.2.0/src/indic_pdf_check.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

indic_pdf_check-0.2.0/src/indic_pdf_check.egg-info/entry_points.txt

@@ -0,0 +1,2 @@
+[console_scripts]
+indic-pdf-check = indic_pdf_check.cli:main

indic_pdf_check-0.2.0/src/indic_pdf_check.egg-info/requires.txt

@@ -0,0 +1 @@
+pymupdf>=1.23

indic_pdf_check-0.2.0/src/indic_pdf_check.egg-info/top_level.txt

@@ -0,0 +1 @@
+indic_pdf_check