paperstack-cli 0.3.2__tar.gz → 0.4.0__tar.gz

paperstack_cli-0.3.2/README.md → paperstack_cli-0.4.0/PKG-INFO

@@ -1,5 +1,29 @@
+Metadata-Version: 2.5
+Name: paperstack-cli
+Version: 0.4.0
+Summary: Review, inspect, and retrieve research sources from one CLI
+Project-URL: Repository, https://github.com/MilkClouds/paperstack
+Project-URL: Issues, https://github.com/MilkClouds/paperstack/issues
+License-Expression: Apache-2.0
+License-File: LICENSE
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.11
+Requires-Dist: filelock>=3.20
+Requires-Dist: polars>=1.43
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: pyyaml>=6
+Provides-Extra: pdf
+Requires-Dist: pdf-inspector<2,>=1.17; extra == 'pdf'
+Description-Content-Type: text/markdown
+
 # Paperstack
 
+[![CI](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml/badge.svg)](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml) [![PyPI version](https://img.shields.io/pypi/v/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![Python versions](https://img.shields.io/pypi/pyversions/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![License](https://img.shields.io/pypi/l/paperstack-cli.svg)](https://github.com/MilkClouds/paperstack/blob/main/LICENSE)
+
 Paperstack is an open-source CLI for maintaining, querying, and publishing an independently authored research corpus.
 The software and corpus are separate: Paperstack can be public while each corpus and its editorial judgments remain
 under its maintainer's control.
@@ -21,6 +45,18 @@ PDF conversion is optional:
 uv tool install 'paperstack-cli[pdf]'
 ```
 
+The PDF extra uses `pdf-inspector` for native extraction and selective OCR.
+Clean text PDFs need no external runtime, while routed OCR pages require the pinned PDFium and ONNX Runtime libraries described in the [OCR runtime guide](https://github.com/firecrawl/pdf-inspector/blob/main/docs/ocr-runtime.md).
+The first routed page downloads and verifies the pinned OCR model set unless an offline model directory is configured.
+When the shared libraries are not on the platform search path, point `pdf-inspector` at the extracted files:
+
+```bash
+export PDFIUM_LIB_PATH=/absolute/path/to/libpdfium.so
+export ORT_DYLIB_PATH=/absolute/path/to/libonnxruntime.so
+```
+
+The runtime guide lists the matching downloads and filenames for Linux, macOS, and Windows.
+
 ## AI agent skill
 
 Paperstack ships an [Agent Skills](https://agentskills.io)-compatible skill for Codex, Claude Code, and other

paperstack_cli-0.3.2/PKG-INFO → paperstack_cli-0.4.0/README.md

@@ -1,22 +1,7 @@
-Metadata-Version: 2.5
-Name: paperstack-cli
-Version: 0.3.2
-Summary: Review, inspect, and retrieve research sources from one CLI
-Project-URL: Repository, https://github.com/MilkClouds/paperstack
-Project-URL: Issues, https://github.com/MilkClouds/paperstack/issues
-License-Expression: Apache-2.0
-License-File: LICENSE
-Requires-Python: >=3.11
-Requires-Dist: filelock>=3.20
-Requires-Dist: polars>=1.43
-Requires-Dist: python-dotenv>=1.2.2
-Requires-Dist: pyyaml>=6
-Provides-Extra: pdf
-Requires-Dist: pymupdf4llm>=0.0.17; extra == 'pdf'
-Description-Content-Type: text/markdown
-
 # Paperstack
 
+[![CI](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml/badge.svg)](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml) [![PyPI version](https://img.shields.io/pypi/v/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![Python versions](https://img.shields.io/pypi/pyversions/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![License](https://img.shields.io/pypi/l/paperstack-cli.svg)](https://github.com/MilkClouds/paperstack/blob/main/LICENSE)
+
 Paperstack is an open-source CLI for maintaining, querying, and publishing an independently authored research corpus.
 The software and corpus are separate: Paperstack can be public while each corpus and its editorial judgments remain
 under its maintainer's control.
@@ -38,6 +23,18 @@ PDF conversion is optional:
 uv tool install 'paperstack-cli[pdf]'
 ```
 
+The PDF extra uses `pdf-inspector` for native extraction and selective OCR.
+Clean text PDFs need no external runtime, while routed OCR pages require the pinned PDFium and ONNX Runtime libraries described in the [OCR runtime guide](https://github.com/firecrawl/pdf-inspector/blob/main/docs/ocr-runtime.md).
+The first routed page downloads and verifies the pinned OCR model set unless an offline model directory is configured.
+When the shared libraries are not on the platform search path, point `pdf-inspector` at the extracted files:
+
+```bash
+export PDFIUM_LIB_PATH=/absolute/path/to/libpdfium.so
+export ORT_DYLIB_PATH=/absolute/path/to/libonnxruntime.so
+```
+
+The runtime guide lists the matching downloads and filenames for Linux, macOS, and Windows.
+
 ## AI agent skill
 
 Paperstack ships an [Agent Skills](https://agentskills.io)-compatible skill for Codex, Claude Code, and other

{paperstack_cli-0.3.2 → paperstack_cli-0.4.0}/pyproject.toml

@@ -5,6 +5,13 @@ description = "Review, inspect, and retrieve research sources from one CLI"
 readme = "README.md"
 license = "Apache-2.0"
 requires-python = ">=3.11"
+classifiers = [
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+]
 dependencies = [
     "filelock>=3.20",
     "polars>=1.43",
@@ -13,7 +20,7 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
-pdf = ["pymupdf4llm>=0.0.17"]
+pdf = ["pdf-inspector>=1.17,<2"]
 
 [dependency-groups]
 lint = ["ruff>=0.12"]

{paperstack_cli-0.3.2 → paperstack_cli-0.4.0}/src/paperstack/cli.py

@@ -1004,10 +1004,13 @@ def _run_paper(a: argparse.Namespace) -> int:
         from .content import arxiv_pdf
 
         arxiv_pdf.CACHE_DIR = _paper_cache()
-        cached_pdf = arxiv_pdf.CACHE_DIR / ref.value / "paper.md"
-        if offline and (not cached_pdf.is_file() or cached_pdf.stat().st_size <= 1000):
-            die(f"no complete cached PDF conversion for arxiv:{ref.value}")
-        if not arxiv_pdf.convert(ref.value):
+        cached_pdf = arxiv_pdf._cached_conversion(
+            arxiv_pdf.CACHE_DIR / ref.value,
+            allow_native_fallback=offline,
+        )
+        if offline and cached_pdf is None:
+            die(f"no usable cached PDF conversion for arxiv:{ref.value}")
+        if not arxiv_pdf.convert(ref.value, allow_native_fallback=offline):
             return 1
         return 0
     raise AssertionError(a.paper_cmd)

paperstack_cli-0.4.0/src/paperstack/content/arxiv_pdf.py

@@ -0,0 +1,253 @@
+"""Convert arXiv PDF submissions to Markdown with selective OCR.
+
+Used by `paperstack paper pdf` after the source fetcher reports no TeX.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import importlib.metadata
+import json
+import os
+import sys
+import tempfile
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+_CACHE_ROOT = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+CACHE_DIR = Path(os.environ.get("PAPERSTACK_PAPERS_DIR", _CACHE_ROOT / "paperstack" / "papers"))
+CONVERTER = "pdf-inspector"
+MIN_MARKDOWN_CHARS = 100
+OCR_RUNTIME_GUIDE = "https://github.com/firecrawl/pdf-inspector/blob/main/docs/ocr-runtime.md"
+
+
+def _fetch(url: str, timeout: int = 60) -> bytes | None:
+    req = urllib.request.Request(url, headers={"User-Agent": "paperstack/1.0 (+arxiv pdf fetch)"})
+    for attempt in range(3):
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                return resp.read()
+        except urllib.error.HTTPError as e:
+            if e.code == 429:
+                time.sleep(15 * (attempt + 1))
+                continue
+            if attempt == 2:
+                return None
+            time.sleep(5)
+        except (urllib.error.URLError, OSError, TimeoutError):
+            if attempt == 2:
+                return None
+            time.sleep(5)
+    return None
+
+
+def _cached_conversion(d: Path, *, allow_native_fallback: bool = False) -> Path | None:
+    md_path = d / "paper.md"
+    meta_path = d / "meta.json"
+    if not md_path.is_file() or md_path.stat().st_size < MIN_MARKDOWN_CHARS or not meta_path.is_file():
+        return None
+    try:
+        markdown = md_path.read_bytes()
+        meta = json.loads(meta_path.read_text(encoding="utf-8"))
+    except (OSError, UnicodeDecodeError, json.JSONDecodeError):
+        return None
+    if not isinstance(meta, dict):
+        return None
+    valid = (
+        meta.get("converter") == CONVERTER
+        and meta.get("bytes") == len(markdown)
+        and meta.get("sha256") == hashlib.sha256(markdown).hexdigest()
+        and (allow_native_fallback or meta.get("conversion_mode") != "native_fallback")
+    )
+    return md_path if valid else None
+
+
+def _converter_version() -> str:
+    try:
+        return importlib.metadata.version(CONVERTER)
+    except importlib.metadata.PackageNotFoundError:
+        return "unknown"
+
+
+def _ocr_reasons(items) -> list[dict]:
+    return [{"page": item.page, "reasons": list(item.reasons)} for item in items]
+
+
+def _page_provenance(page) -> dict:
+    provenance = page.provenance
+    model = provenance.ocr_model
+    return {
+        "page": page.page_number,
+        "source": provenance.source,
+        "ocr_model": None if model is None else {"name": model.name, "revision": model.revision},
+        "render_dpi": provenance.render_dpi,
+        "ocr_confidence": provenance.ocr_confidence,
+        "hosted_recommended": provenance.hosted_recommended,
+        "warnings": list(provenance.warnings),
+        "timings": {
+            "render_ms": provenance.timings.render_ms,
+            "ocr_ms": provenance.timings.ocr_ms,
+            "assembly_ms": provenance.timings.assembly_ms,
+        },
+    }
+
+
+def _base_meta(arxiv_id: str, url: str, markdown: str) -> dict:
+    encoded = markdown.encode("utf-8")
+    return {
+        "schema_version": 1,
+        "arxiv_id": arxiv_id,
+        "bytes": len(encoded),
+        "sha256": hashlib.sha256(encoded).hexdigest(),
+        "url": url,
+        "converter": CONVERTER,
+        "converter_version": _converter_version(),
+    }
+
+
+def _ocr_meta(arxiv_id: str, url: str, markdown: str, result) -> dict:
+    meta = _base_meta(arxiv_id, url, markdown)
+    meta.update(
+        {
+            "conversion_mode": "selective_ocr",
+            "quality": "partial" if result.pages_recommending_hosted else "complete",
+            "page_count": result.page_count,
+            "pages_recommended_for_ocr": list(result.pages_recommended_for_ocr),
+            "pages_routed_to_ocr": list(result.pages_routed_to_ocr),
+            "pages_recommending_hosted": list(result.pages_recommending_hosted),
+            "ocr_reasons_by_page": _ocr_reasons(result.ocr_reasons_by_page),
+            "pages_with_tables": list(result.pages_with_tables),
+            "pages_with_columns": list(result.pages_with_columns),
+            "pages": [_page_provenance(page) for page in result.pages],
+        }
+    )
+    return meta
+
+
+def _native_fallback_meta(arxiv_id: str, url: str, markdown: str, result, error: ValueError) -> dict:
+    meta = _ocr_meta(arxiv_id, url, markdown, result)
+    meta["conversion_mode"] = "native_fallback"
+    meta["quality"] = "partial"
+    meta["ocr_error"] = str(error)
+    return meta
+
+
+def _usable(markdown: str | None) -> bool:
+    return markdown is not None and len(markdown.strip()) >= MIN_MARKDOWN_CHARS
+
+
+def _is_pdf(path: Path) -> bool:
+    try:
+        with path.open("rb") as stream:
+            return stream.read(4) == b"%PDF"
+    except OSError:
+        return False
+
+
+def _download_pdf(url: str, pdf_path: Path) -> bool:
+    raw = _fetch(url)
+    if raw is None:
+        print(f"could not fetch {url}", file=sys.stderr)
+        return False
+    if not raw.startswith(b"%PDF"):
+        print(f"{url} did not return a PDF", file=sys.stderr)
+        return False
+    descriptor, staged = tempfile.mkstemp(prefix=f".{pdf_path.name}-", suffix=".part", dir=pdf_path.parent)
+    try:
+        with os.fdopen(descriptor, "wb") as stream:
+            stream.write(raw)
+        os.replace(staged, pdf_path)
+    finally:
+        try:
+            os.unlink(staged)
+        except FileNotFoundError:
+            pass
+    return True
+
+
+def convert(arxiv_id: str, *, allow_native_fallback: bool = False) -> bool:
+    d = CACHE_DIR / arxiv_id
+    cached = _cached_conversion(d, allow_native_fallback=allow_native_fallback)
+    if cached is not None:
+        print(f"{arxiv_id}: cached at {cached}")
+        return True
+
+    try:
+        import pdf_inspector
+    except ImportError:
+        print(
+            "PDF conversion needs `paperstack-cli[pdf]`; reinstall with "
+            "`uv tool install --force 'paperstack-cli[pdf]'`.",
+            file=sys.stderr,
+        )
+        return False
+
+    url = f"https://arxiv.org/pdf/{arxiv_id}"
+    d.mkdir(parents=True, exist_ok=True)
+    pdf_path = d / "paper.pdf"
+    reused_cached_pdf = _is_pdf(pdf_path)
+    if not reused_cached_pdf and not _download_pdf(url, pdf_path):
+        return False
+
+    while True:
+        ocr_error = None
+        try:
+            result = pdf_inspector.process_pdf_with_ocr(str(pdf_path), mode="auto")
+            parsed = result
+            md = result.markdown
+            meta = _ocr_meta(arxiv_id, url, md, result)
+        except ValueError as error:
+            ocr_error = error
+            try:
+                native = pdf_inspector.process_pdf_with_ocr(str(pdf_path), mode="off")
+            except ValueError:
+                native = None
+            parsed = native
+            if native is None:
+                md = None
+            else:
+                md = native.markdown
+                if _usable(md):
+                    meta = _native_fallback_meta(arxiv_id, url, md, native, error)
+
+        if _usable(md):
+            if ocr_error is not None:
+                print(f"{arxiv_id}: OCR unavailable; cached {meta['quality']} native extraction", file=sys.stderr)
+            break
+        needs_ocr = parsed is not None and bool(parsed.pages_recommended_for_ocr)
+        if reused_cached_pdf and not needs_ocr and _download_pdf(url, pdf_path):
+            reused_cached_pdf = False
+            continue
+        if ocr_error is not None:
+            print(f"{arxiv_id}: PDF conversion failed: {ocr_error}", file=sys.stderr)
+            if needs_ocr:
+                print(
+                    "OCR needs PDFium and ONNX Runtime; set PDFIUM_LIB_PATH and ORT_DYLIB_PATH.",
+                    file=sys.stderr,
+                )
+                print(f"OCR runtime setup: {OCR_RUNTIME_GUIDE}", file=sys.stderr)
+            return False
+        chars = len(md.strip()) if md else 0
+        print(f"{arxiv_id}: converted to only {chars} chars, treat as a failure", file=sys.stderr)
+        return False
+
+    md_path = d / "paper.md"
+    md_path.write_bytes(md.encode("utf-8"))
+    (d / "meta.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")
+    print(f"{arxiv_id}: {len(md)} chars -> {md_path}")
+    return True
+
+
+def main(argv: list[str]) -> None:
+    if not argv:
+        sys.exit("pass one or more arXiv ids")
+    CACHE_DIR.mkdir(parents=True, exist_ok=True)
+    failures = [a for a in argv if not convert(a)]
+    if failures:
+        sys.exit(f"failed: {' '.join(failures)}")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

{paperstack_cli-0.3.2 → paperstack_cli-0.4.0}/src/paperstack/entrypoint.py

@@ -9,6 +9,8 @@ from dotenv import find_dotenv, load_dotenv
 
 from . import credentials
 
+_EXPORTED_ONLY = ("PDFIUM_LIB_PATH", "ORT_DYLIB_PATH")
+
 
 def load_environment() -> None:
     """Load the nearest .env without replacing exported variables."""
@@ -16,6 +18,9 @@ def load_environment() -> None:
     path = find_dotenv(usecwd=True)
     if path:
         load_dotenv(path, override=False)
+    for name in _EXPORTED_ONLY:
+        if name not in exported_keys:
+            os.environ.pop(name, None)
     credentials.set_environment_context(exported_keys, Path(path) if path else None)
 
 

paperstack_cli-0.3.2/src/paperstack/content/arxiv_pdf.py

@@ -1,103 +0,0 @@
-"""Convert native-PDF arXiv submissions to Markdown.
-
-Used by `paperstack paper pdf` after the source fetcher reports no TeX.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import os
-import sys
-import time
-import urllib.error
-import urllib.request
-from pathlib import Path
-
-_CACHE_ROOT = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
-CACHE_DIR = Path(os.environ.get("PAPERSTACK_PAPERS_DIR", _CACHE_ROOT / "paperstack" / "papers"))
-
-
-def _fetch(url: str, timeout: int = 60) -> bytes | None:
-    req = urllib.request.Request(url, headers={"User-Agent": "paperstack/1.0 (+arxiv pdf fetch)"})
-    for attempt in range(3):
-        try:
-            with urllib.request.urlopen(req, timeout=timeout) as resp:
-                return resp.read()
-        except urllib.error.HTTPError as e:
-            if e.code == 429:
-                time.sleep(15 * (attempt + 1))
-                continue
-            if attempt == 2:
-                return None
-            time.sleep(5)
-        except (urllib.error.URLError, OSError, TimeoutError):
-            if attempt == 2:
-                return None
-            time.sleep(5)
-    return None
-
-
-def convert(arxiv_id: str) -> bool:
-    d = CACHE_DIR / arxiv_id
-    md_path = d / "paper.md"
-    if md_path.exists() and md_path.stat().st_size > 1000:
-        print(f"{arxiv_id}: cached at {md_path}")
-        return True
-
-    try:
-        import pymupdf4llm
-    except ImportError:
-        print(
-            "PDF conversion needs `paperstack-cli[pdf]`; reinstall with "
-            "`uv tool install --force 'paperstack-cli[pdf]'`.",
-            file=sys.stderr,
-        )
-        return False
-
-    url = f"https://arxiv.org/pdf/{arxiv_id}"
-    raw = _fetch(url)
-    if raw is None:
-        print(f"{arxiv_id}: could not fetch {url}", file=sys.stderr)
-        return False
-    if not raw.startswith(b"%PDF"):
-        print(f"{arxiv_id}: {url} did not return a PDF", file=sys.stderr)
-        return False
-
-    d.mkdir(parents=True, exist_ok=True)
-    pdf_path = d / "paper.pdf"
-    pdf_path.write_bytes(raw)
-    md = pymupdf4llm.to_markdown(str(pdf_path))
-    if len(md) < 500:
-        print(f"{arxiv_id}: converted to only {len(md)} chars, treat as a failure", file=sys.stderr)
-        return False
-
-    md_path.write_text(md, encoding="utf-8")
-    (d / "meta.json").write_text(
-        json.dumps(
-            {
-                "arxiv_id": arxiv_id,
-                "bytes": len(md),
-                "sha256": hashlib.sha256(md.encode("utf-8")).hexdigest(),
-                "url": url,
-                "converter": "pymupdf4llm",
-            },
-            indent=2,
-        ),
-        encoding="utf-8",
-    )
-    print(f"{arxiv_id}: {len(md)} chars -> {md_path}")
-    return True
-
-
-def main(argv: list[str]) -> None:
-    if not argv:
-        sys.exit("pass one or more arXiv ids")
-    CACHE_DIR.mkdir(parents=True, exist_ok=True)
-    failures = [a for a in argv if not convert(a)]
-    if failures:
-        sys.exit(f"failed: {' '.join(failures)}")
-
-
-if __name__ == "__main__":
-    main(sys.argv[1:])