paperstack-cli 0.3.2__tar.gz → 0.4.1__tar.gz

paperstack_cli-0.3.2/README.md → paperstack_cli-0.4.1/PKG-INFO

@@ -1,5 +1,29 @@
+Metadata-Version: 2.5
+Name: paperstack-cli
+Version: 0.4.1
+Summary: Review, inspect, and retrieve research sources from one CLI
+Project-URL: Repository, https://github.com/MilkClouds/paperstack
+Project-URL: Issues, https://github.com/MilkClouds/paperstack/issues
+License-Expression: Apache-2.0
+License-File: LICENSE
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Requires-Python: >=3.11
+Requires-Dist: filelock>=3.20
+Requires-Dist: polars>=1.43
+Requires-Dist: python-dotenv>=1.2.2
+Requires-Dist: pyyaml>=6
+Provides-Extra: pdf
+Requires-Dist: pdf-inspector<2,>=1.17; extra == 'pdf'
+Description-Content-Type: text/markdown
+
 # Paperstack
 
+[![CI](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml/badge.svg)](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml) [![PyPI version](https://img.shields.io/pypi/v/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![Python versions](https://img.shields.io/pypi/pyversions/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![License](https://img.shields.io/pypi/l/paperstack-cli.svg)](https://github.com/MilkClouds/paperstack/blob/main/LICENSE)
+
 Paperstack is an open-source CLI for maintaining, querying, and publishing an independently authored research corpus.
 The software and corpus are separate: Paperstack can be public while each corpus and its editorial judgments remain
 under its maintainer's control.
@@ -21,6 +45,18 @@ PDF conversion is optional:
 uv tool install 'paperstack-cli[pdf]'
 ```
 
+The PDF extra uses `pdf-inspector` for native extraction and selective OCR.
+Clean text PDFs need no external runtime, while routed OCR pages require the pinned PDFium and ONNX Runtime libraries described in the [OCR runtime guide](https://github.com/firecrawl/pdf-inspector/blob/main/docs/ocr-runtime.md).
+The first routed page downloads and verifies the pinned OCR model set unless an offline model directory is configured.
+When the shared libraries are not on the platform search path, point `pdf-inspector` at the extracted files:
+
+```bash
+export PDFIUM_LIB_PATH=/absolute/path/to/libpdfium.so
+export ORT_DYLIB_PATH=/absolute/path/to/libonnxruntime.so
+```
+
+The runtime guide lists the matching downloads and filenames for Linux, macOS, and Windows.
+
 ## AI agent skill
 
 Paperstack ships an [Agent Skills](https://agentskills.io)-compatible skill for Codex, Claude Code, and other
@@ -98,6 +134,7 @@ These commands use external source records and do not select a citation or make
 
 ```bash
 paperstack paper search "Attention Is All You Need" --source dblp
+paperstack paper search "Exact Paper Title" --source openreview --exact-title --openreview-status accepted
 paperstack paper search "robot learning" --source semantic-scholar --year 2024-2026
 paperstack paper search 'ti:"robot learning"' --source arxiv --category cs.RO --sort date
 paperstack paper metadata arxiv:2106.09685
@@ -115,6 +152,10 @@ reference. `authors`, `citations`, and `references` use Semantic Scholar and als
 `--offline` flags. Paperstack reports source records but does not synthesize a citation entry or choose which version
 of a work should be cited.
 
+OpenReview exact-title search uses its title-only exact mode and verifies a normalized title match locally. Status
+filtering is conservative because venues encode decisions and withdrawals differently; Paperstack infers it from
+public invitation, venue, decision, and status fields rather than treating it as a universal field.
+
 ## Build a viewer
 
 ```bash

paperstack_cli-0.3.2/PKG-INFO → paperstack_cli-0.4.1/README.md

@@ -1,22 +1,7 @@
-Metadata-Version: 2.5
-Name: paperstack-cli
-Version: 0.3.2
-Summary: Review, inspect, and retrieve research sources from one CLI
-Project-URL: Repository, https://github.com/MilkClouds/paperstack
-Project-URL: Issues, https://github.com/MilkClouds/paperstack/issues
-License-Expression: Apache-2.0
-License-File: LICENSE
-Requires-Python: >=3.11
-Requires-Dist: filelock>=3.20
-Requires-Dist: polars>=1.43
-Requires-Dist: python-dotenv>=1.2.2
-Requires-Dist: pyyaml>=6
-Provides-Extra: pdf
-Requires-Dist: pymupdf4llm>=0.0.17; extra == 'pdf'
-Description-Content-Type: text/markdown
-
 # Paperstack
 
+[![CI](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml/badge.svg)](https://github.com/MilkClouds/paperstack/actions/workflows/ci.yml) [![PyPI version](https://img.shields.io/pypi/v/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![Python versions](https://img.shields.io/pypi/pyversions/paperstack-cli.svg)](https://pypi.org/project/paperstack-cli/) [![License](https://img.shields.io/pypi/l/paperstack-cli.svg)](https://github.com/MilkClouds/paperstack/blob/main/LICENSE)
+
 Paperstack is an open-source CLI for maintaining, querying, and publishing an independently authored research corpus.
 The software and corpus are separate: Paperstack can be public while each corpus and its editorial judgments remain
 under its maintainer's control.
@@ -38,6 +23,18 @@ PDF conversion is optional:
 uv tool install 'paperstack-cli[pdf]'
 ```
 
+The PDF extra uses `pdf-inspector` for native extraction and selective OCR.
+Clean text PDFs need no external runtime, while routed OCR pages require the pinned PDFium and ONNX Runtime libraries described in the [OCR runtime guide](https://github.com/firecrawl/pdf-inspector/blob/main/docs/ocr-runtime.md).
+The first routed page downloads and verifies the pinned OCR model set unless an offline model directory is configured.
+When the shared libraries are not on the platform search path, point `pdf-inspector` at the extracted files:
+
+```bash
+export PDFIUM_LIB_PATH=/absolute/path/to/libpdfium.so
+export ORT_DYLIB_PATH=/absolute/path/to/libonnxruntime.so
+```
+
+The runtime guide lists the matching downloads and filenames for Linux, macOS, and Windows.
+
 ## AI agent skill
 
 Paperstack ships an [Agent Skills](https://agentskills.io)-compatible skill for Codex, Claude Code, and other
@@ -115,6 +112,7 @@ These commands use external source records and do not select a citation or make
 
 ```bash
 paperstack paper search "Attention Is All You Need" --source dblp
+paperstack paper search "Exact Paper Title" --source openreview --exact-title --openreview-status accepted
 paperstack paper search "robot learning" --source semantic-scholar --year 2024-2026
 paperstack paper search 'ti:"robot learning"' --source arxiv --category cs.RO --sort date
 paperstack paper metadata arxiv:2106.09685
@@ -132,6 +130,10 @@ reference. `authors`, `citations`, and `references` use Semantic Scholar and als
 `--offline` flags. Paperstack reports source records but does not synthesize a citation entry or choose which version
 of a work should be cited.
 
+OpenReview exact-title search uses its title-only exact mode and verifies a normalized title match locally. Status
+filtering is conservative because venues encode decisions and withdrawals differently; Paperstack infers it from
+public invitation, venue, decision, and status fields rather than treating it as a universal field.
+
 ## Build a viewer
 
 ```bash

{paperstack_cli-0.3.2 → paperstack_cli-0.4.1}/pyproject.toml

@@ -5,6 +5,13 @@ description = "Review, inspect, and retrieve research sources from one CLI"
 readme = "README.md"
 license = "Apache-2.0"
 requires-python = ">=3.11"
+classifiers = [
+    "Programming Language :: Python :: 3 :: Only",
+    "Programming Language :: Python :: 3.11",
+    "Programming Language :: Python :: 3.12",
+    "Programming Language :: Python :: 3.13",
+    "Programming Language :: Python :: 3.14",
+]
 dependencies = [
     "filelock>=3.20",
     "polars>=1.43",
@@ -13,7 +20,7 @@ dependencies = [
 ]
 
 [project.optional-dependencies]
-pdf = ["pymupdf4llm>=0.0.17"]
+pdf = ["pdf-inspector>=1.17,<2"]
 
 [dependency-groups]
 lint = ["ruff>=0.12"]

{paperstack_cli-0.3.2 → paperstack_cli-0.4.1}/src/paperstack/cli.py

@@ -925,12 +925,42 @@ def _run_paper(a: argparse.Namespace) -> int:
         metadata.print_results(results, json_output=a.json)
         return 0 if any(item["status"] == "ok" for item in results) else 1
     if a.paper_cmd == "search":
+        semantic_filters = [
+            flag
+            for flag, selected in (
+                ("--offset", a.offset),
+                ("--year", a.year),
+                ("--field-of-study", a.fields_of_study),
+                ("--open-access", a.open_access),
+            )
+            if selected
+        ]
+        arxiv_filters = [
+            flag
+            for flag, selected in (
+                ("--category", a.categories),
+                ("--date-from", a.date_from),
+                ("--date-to", a.date_to),
+                ("--sort", a.sort != "relevance"),
+            )
+            if selected
+        ]
+        openreview_filters = [
+            flag
+            for flag, selected in (
+                ("--exact-title", a.exact_title),
+                ("--openreview-status", a.openreview_status),
+            )
+            if selected
+        ]
         try:
+            if a.source != "openreview" and openreview_filters:
+                die(f"--source openreview is required for {', '.join(openreview_filters)}")
             if a.source == "arxiv":
                 from . import arxiv
 
-                if a.year or a.fields_of_study or a.open_access or a.offset:
-                    die("--year, --field-of-study, --open-access, and --offset require --source semantic-scholar")
+                if semantic_filters:
+                    die(f"--source semantic-scholar is required for {', '.join(semantic_filters)}")
                 result = arxiv.search(
                     a.query,
                     categories=a.categories,
@@ -942,8 +972,8 @@ def _run_paper(a: argparse.Namespace) -> int:
             elif a.source == "semantic-scholar":
                 from . import semantic_scholar
 
-                if a.categories or a.date_from or a.date_to or a.sort != "relevance":
-                    die("--category, --date-from, --date-to, and --sort require --source arxiv")
+                if arxiv_filters:
+                    die(f"--source arxiv is required for {', '.join(arxiv_filters)}")
                 result = semantic_scholar.search(
                     a.query,
                     limit=a.limit,
@@ -953,19 +983,21 @@ def _run_paper(a: argparse.Namespace) -> int:
                     open_access=a.open_access,
                 )
             else:
-                if (
-                    a.categories
-                    or a.date_from
-                    or a.date_to
-                    or a.limit != 10
-                    or a.offset
-                    or a.sort != "relevance"
-                    or a.year
-                    or a.fields_of_study
-                    or a.open_access
-                ):
-                    die("search filters require --source arxiv or --source semantic-scholar")
-                result = metadata.search(a.source, a.query, local_only=offline)
+                requirements = []
+                if semantic_filters:
+                    requirements.append(f"--source semantic-scholar is required for {', '.join(semantic_filters)}")
+                if arxiv_filters:
+                    requirements.append(f"--source arxiv is required for {', '.join(arxiv_filters)}")
+                if requirements:
+                    die("; ".join(requirements))
+                result = metadata.search(
+                    a.source,
+                    a.query,
+                    limit=a.limit,
+                    local_only=offline,
+                    exact_title=a.exact_title,
+                    openreview_status=a.openreview_status,
+                )
         except credentials.CredentialsError as exc:
             die(f"configuration failed: {exc}")
         except RuntimeError as exc:
@@ -1004,10 +1036,13 @@ def _run_paper(a: argparse.Namespace) -> int:
         from .content import arxiv_pdf
 
         arxiv_pdf.CACHE_DIR = _paper_cache()
-        cached_pdf = arxiv_pdf.CACHE_DIR / ref.value / "paper.md"
-        if offline and (not cached_pdf.is_file() or cached_pdf.stat().st_size <= 1000):
-            die(f"no complete cached PDF conversion for arxiv:{ref.value}")
-        if not arxiv_pdf.convert(ref.value):
+        cached_pdf = arxiv_pdf._cached_conversion(
+            arxiv_pdf.CACHE_DIR / ref.value,
+            allow_native_fallback=offline,
+        )
+        if offline and cached_pdf is None:
+            die(f"no usable cached PDF conversion for arxiv:{ref.value}")
+        if not arxiv_pdf.convert(ref.value, allow_native_fallback=offline):
             return 1
         return 0
     raise AssertionError(a.paper_cmd)
@@ -1175,6 +1210,12 @@ Use `paperstack review ...` to find or read an authored critical judgment.""",
     s.add_argument("--date-from", help="earliest arXiv submission date")
     s.add_argument("--date-to", help="latest arXiv submission date")
     s.add_argument("--sort", choices=("relevance", "date"), default="relevance")
+    s.add_argument("--exact-title", action="store_true", help="require a normalized exact OpenReview title")
+    s.add_argument(
+        "--openreview-status",
+        choices=("submission", "accepted", "withdrawn"),
+        help="filter OpenReview forum records by inferred status",
+    )
     _output(s)
     _offline(s)
     for command in ("authors", "citations", "references"):

paperstack_cli-0.4.1/src/paperstack/content/arxiv_pdf.py

@@ -0,0 +1,253 @@
+"""Convert arXiv PDF submissions to Markdown with selective OCR.
+
+Used by `paperstack paper pdf` after the source fetcher reports no TeX.
+"""
+
+from __future__ import annotations
+
+import hashlib
+import importlib.metadata
+import json
+import os
+import sys
+import tempfile
+import time
+import urllib.error
+import urllib.request
+from pathlib import Path
+
+_CACHE_ROOT = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+CACHE_DIR = Path(os.environ.get("PAPERSTACK_PAPERS_DIR", _CACHE_ROOT / "paperstack" / "papers"))
+CONVERTER = "pdf-inspector"
+MIN_MARKDOWN_CHARS = 100
+OCR_RUNTIME_GUIDE = "https://github.com/firecrawl/pdf-inspector/blob/main/docs/ocr-runtime.md"
+
+
+def _fetch(url: str, timeout: int = 60) -> bytes | None:
+    req = urllib.request.Request(url, headers={"User-Agent": "paperstack/1.0 (+arxiv pdf fetch)"})
+    for attempt in range(3):
+        try:
+            with urllib.request.urlopen(req, timeout=timeout) as resp:
+                return resp.read()
+        except urllib.error.HTTPError as e:
+            if e.code == 429:
+                time.sleep(15 * (attempt + 1))
+                continue
+            if attempt == 2:
+                return None
+            time.sleep(5)
+        except (urllib.error.URLError, OSError, TimeoutError):
+            if attempt == 2:
+                return None
+            time.sleep(5)
+    return None
+
+
+def _cached_conversion(d: Path, *, allow_native_fallback: bool = False) -> Path | None:
+    md_path = d / "paper.md"
+    meta_path = d / "meta.json"
+    if not md_path.is_file() or md_path.stat().st_size < MIN_MARKDOWN_CHARS or not meta_path.is_file():
+        return None
+    try:
+        markdown = md_path.read_bytes()
+        meta = json.loads(meta_path.read_text(encoding="utf-8"))
+    except (OSError, UnicodeDecodeError, json.JSONDecodeError):
+        return None
+    if not isinstance(meta, dict):
+        return None
+    valid = (
+        meta.get("converter") == CONVERTER
+        and meta.get("bytes") == len(markdown)
+        and meta.get("sha256") == hashlib.sha256(markdown).hexdigest()
+        and (allow_native_fallback or meta.get("conversion_mode") != "native_fallback")
+    )
+    return md_path if valid else None
+
+
+def _converter_version() -> str:
+    try:
+        return importlib.metadata.version(CONVERTER)
+    except importlib.metadata.PackageNotFoundError:
+        return "unknown"
+
+
+def _ocr_reasons(items) -> list[dict]:
+    return [{"page": item.page, "reasons": list(item.reasons)} for item in items]
+
+
+def _page_provenance(page) -> dict:
+    provenance = page.provenance
+    model = provenance.ocr_model
+    return {
+        "page": page.page_number,
+        "source": provenance.source,
+        "ocr_model": None if model is None else {"name": model.name, "revision": model.revision},
+        "render_dpi": provenance.render_dpi,
+        "ocr_confidence": provenance.ocr_confidence,
+        "hosted_recommended": provenance.hosted_recommended,
+        "warnings": list(provenance.warnings),
+        "timings": {
+            "render_ms": provenance.timings.render_ms,
+            "ocr_ms": provenance.timings.ocr_ms,
+            "assembly_ms": provenance.timings.assembly_ms,
+        },
+    }
+
+
+def _base_meta(arxiv_id: str, url: str, markdown: str) -> dict:
+    encoded = markdown.encode("utf-8")
+    return {
+        "schema_version": 1,
+        "arxiv_id": arxiv_id,
+        "bytes": len(encoded),
+        "sha256": hashlib.sha256(encoded).hexdigest(),
+        "url": url,
+        "converter": CONVERTER,
+        "converter_version": _converter_version(),
+    }
+
+
+def _ocr_meta(arxiv_id: str, url: str, markdown: str, result) -> dict:
+    meta = _base_meta(arxiv_id, url, markdown)
+    meta.update(
+        {
+            "conversion_mode": "selective_ocr",
+            "quality": "partial" if result.pages_recommending_hosted else "complete",
+            "page_count": result.page_count,
+            "pages_recommended_for_ocr": list(result.pages_recommended_for_ocr),
+            "pages_routed_to_ocr": list(result.pages_routed_to_ocr),
+            "pages_recommending_hosted": list(result.pages_recommending_hosted),
+            "ocr_reasons_by_page": _ocr_reasons(result.ocr_reasons_by_page),
+            "pages_with_tables": list(result.pages_with_tables),
+            "pages_with_columns": list(result.pages_with_columns),
+            "pages": [_page_provenance(page) for page in result.pages],
+        }
+    )
+    return meta
+
+
+def _native_fallback_meta(arxiv_id: str, url: str, markdown: str, result, error: ValueError) -> dict:
+    meta = _ocr_meta(arxiv_id, url, markdown, result)
+    meta["conversion_mode"] = "native_fallback"
+    meta["quality"] = "partial"
+    meta["ocr_error"] = str(error)
+    return meta
+
+
+def _usable(markdown: str | None) -> bool:
+    return markdown is not None and len(markdown.strip()) >= MIN_MARKDOWN_CHARS
+
+
+def _is_pdf(path: Path) -> bool:
+    try:
+        with path.open("rb") as stream:
+            return stream.read(4) == b"%PDF"
+    except OSError:
+        return False
+
+
+def _download_pdf(url: str, pdf_path: Path) -> bool:
+    raw = _fetch(url)
+    if raw is None:
+        print(f"could not fetch {url}", file=sys.stderr)
+        return False
+    if not raw.startswith(b"%PDF"):
+        print(f"{url} did not return a PDF", file=sys.stderr)
+        return False
+    descriptor, staged = tempfile.mkstemp(prefix=f".{pdf_path.name}-", suffix=".part", dir=pdf_path.parent)
+    try:
+        with os.fdopen(descriptor, "wb") as stream:
+            stream.write(raw)
+        os.replace(staged, pdf_path)
+    finally:
+        try:
+            os.unlink(staged)
+        except FileNotFoundError:
+            pass
+    return True
+
+
+def convert(arxiv_id: str, *, allow_native_fallback: bool = False) -> bool:
+    d = CACHE_DIR / arxiv_id
+    cached = _cached_conversion(d, allow_native_fallback=allow_native_fallback)
+    if cached is not None:
+        print(f"{arxiv_id}: cached at {cached}")
+        return True
+
+    try:
+        import pdf_inspector
+    except ImportError:
+        print(
+            "PDF conversion needs `paperstack-cli[pdf]`; reinstall with "
+            "`uv tool install --force 'paperstack-cli[pdf]'`.",
+            file=sys.stderr,
+        )
+        return False
+
+    url = f"https://arxiv.org/pdf/{arxiv_id}"
+    d.mkdir(parents=True, exist_ok=True)
+    pdf_path = d / "paper.pdf"
+    reused_cached_pdf = _is_pdf(pdf_path)
+    if not reused_cached_pdf and not _download_pdf(url, pdf_path):
+        return False
+
+    while True:
+        ocr_error = None
+        try:
+            result = pdf_inspector.process_pdf_with_ocr(str(pdf_path), mode="auto")
+            parsed = result
+            md = result.markdown
+            meta = _ocr_meta(arxiv_id, url, md, result)
+        except ValueError as error:
+            ocr_error = error
+            try:
+                native = pdf_inspector.process_pdf_with_ocr(str(pdf_path), mode="off")
+            except ValueError:
+                native = None
+            parsed = native
+            if native is None:
+                md = None
+            else:
+                md = native.markdown
+                if _usable(md):
+                    meta = _native_fallback_meta(arxiv_id, url, md, native, error)
+
+        if _usable(md):
+            if ocr_error is not None:
+                print(f"{arxiv_id}: OCR unavailable; cached {meta['quality']} native extraction", file=sys.stderr)
+            break
+        needs_ocr = parsed is not None and bool(parsed.pages_recommended_for_ocr)
+        if reused_cached_pdf and not needs_ocr and _download_pdf(url, pdf_path):
+            reused_cached_pdf = False
+            continue
+        if ocr_error is not None:
+            print(f"{arxiv_id}: PDF conversion failed: {ocr_error}", file=sys.stderr)
+            if needs_ocr:
+                print(
+                    "OCR needs PDFium and ONNX Runtime; set PDFIUM_LIB_PATH and ORT_DYLIB_PATH.",
+                    file=sys.stderr,
+                )
+                print(f"OCR runtime setup: {OCR_RUNTIME_GUIDE}", file=sys.stderr)
+            return False
+        chars = len(md.strip()) if md else 0
+        print(f"{arxiv_id}: converted to only {chars} chars, treat as a failure", file=sys.stderr)
+        return False
+
+    md_path = d / "paper.md"
+    md_path.write_bytes(md.encode("utf-8"))
+    (d / "meta.json").write_text(json.dumps(meta, indent=2), encoding="utf-8")
+    print(f"{arxiv_id}: {len(md)} chars -> {md_path}")
+    return True
+
+
+def main(argv: list[str]) -> None:
+    if not argv:
+        sys.exit("pass one or more arXiv ids")
+    CACHE_DIR.mkdir(parents=True, exist_ok=True)
+    failures = [a for a in argv if not convert(a)]
+    if failures:
+        sys.exit(f"failed: {' '.join(failures)}")
+
+
+if __name__ == "__main__":
+    main(sys.argv[1:])

{paperstack_cli-0.3.2 → paperstack_cli-0.4.1}/src/paperstack/entrypoint.py

@@ -9,6 +9,8 @@ from dotenv import find_dotenv, load_dotenv
 
 from . import credentials
 
+_EXPORTED_ONLY = ("PDFIUM_LIB_PATH", "ORT_DYLIB_PATH")
+
 
 def load_environment() -> None:
     """Load the nearest .env without replacing exported variables."""
@@ -16,6 +18,9 @@ def load_environment() -> None:
     path = find_dotenv(usecwd=True)
     if path:
         load_dotenv(path, override=False)
+    for name in _EXPORTED_ONLY:
+        if name not in exported_keys:
+            os.environ.pop(name, None)
     credentials.set_environment_context(exported_keys, Path(path) if path else None)
 
 

{paperstack_cli-0.3.2 → paperstack_cli-0.4.1}/src/paperstack/metadata.py

@@ -6,11 +6,18 @@ import json
 import os
 import re
 import time
+import unicodedata
 import urllib.error
 import urllib.parse
 import urllib.request
 import xml.etree.ElementTree as ET
+from contextlib import contextmanager
 from dataclasses import dataclass
+from datetime import UTC, datetime
+from email.utils import parsedate_to_datetime
+from pathlib import Path
+
+from filelock import FileLock
 
 from . import credentials
 
@@ -45,6 +52,59 @@ class PaperRef:
 
 
 _last_request: dict[str, float] = {}
+_OPENREVIEW_HOSTS = {"api.openreview.net", "api2.openreview.net"}
+
+
+@contextmanager
+def _request_slot(key: str):
+    if key != "openreview":
+        yield None
+        return
+    base = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
+    root = base / "paperstack" / "http"
+    try:
+        root.mkdir(parents=True, exist_ok=True, mode=0o700)
+        lock = FileLock(root / "openreview.lock", timeout=120)
+        lock.acquire()
+    except OSError:
+        yield None
+        return
+    try:
+        yield root / "openreview.timestamp"
+    finally:
+        lock.release()
+
+
+def _shared_elapsed(path: Path | None) -> float:
+    if path is None:
+        return float("inf")
+    try:
+        return max(0.0, time.time() - path.stat().st_mtime)
+    except OSError:
+        return float("inf")
+
+
+def _mark_request(path: Path | None) -> None:
+    if path is not None:
+        try:
+            path.touch()
+        except OSError:
+            pass
+
+
+def _retry_delay(exc: urllib.error.HTTPError, attempt: int) -> float:
+    value = exc.headers.get("Retry-After") if exc.headers else None
+    if value:
+        try:
+            delay = float(value)
+        except ValueError:
+            try:
+                delay = (parsedate_to_datetime(value) - datetime.now(UTC)).total_seconds()
+            except (TypeError, ValueError, OverflowError):
+                delay = -1
+        if delay >= 0:
+            return min(delay, 120)
+    return min(2**attempt, 30)
 
 
 def request(
@@ -56,34 +116,41 @@ def request(
     if params:
         url += "?" + urllib.parse.urlencode(params)
     host = urllib.parse.urlparse(url).netloc
+    request_key = "openreview" if host in _OPENREVIEW_HOSTS else host
     interval = 3.0 if "arxiv.org" in host else 1.1 if "dblp.org" in host else 0.5
     request_headers = {
         "User-Agent": "paperstack (+https://github.com/MilkClouds/paperstack)",
         **(headers or {}),
     }
     req = urllib.request.Request(url, headers=request_headers, data=data)
-    for attempt in range(3):
-        elapsed = time.monotonic() - _last_request.get(host, 0.0)
-        if elapsed < interval:
-            time.sleep(interval - elapsed)
-        try:
-            with urllib.request.urlopen(req, timeout=30) as response:
-                _last_request[host] = time.monotonic()
-                return response.read()
-        except urllib.error.HTTPError as exc:
-            _last_request[host] = time.monotonic()
-            if exc.code != 429:
-                raise
-            if attempt == 2:
-                has_api_key = any(name.lower() == "x-api-key" for name in request_headers)
-                if host == "api.semanticscholar.org" and not has_api_key:
-                    message = (
-                        f"{exc.reason}; configure semantic-scholar.api-key with "
-                        "`paperstack config set semantic-scholar.api-key` for more reliable access"
-                    )
-                    raise urllib.error.HTTPError(exc.url, exc.code, message, exc.headers, exc.fp) from exc
-                raise
-            time.sleep(5 * (attempt + 1))
+    with _request_slot(request_key) as shared_timestamp:
+        for attempt in range(3):
+            elapsed = min(
+                time.monotonic() - _last_request.get(request_key, 0.0),
+                _shared_elapsed(shared_timestamp),
+            )
+            if elapsed < interval:
+                time.sleep(interval - elapsed)
+            try:
+                with urllib.request.urlopen(req, timeout=30) as response:
+                    _last_request[request_key] = time.monotonic()
+                    _mark_request(shared_timestamp)
+                    return response.read()
+            except urllib.error.HTTPError as exc:
+                _last_request[request_key] = time.monotonic()
+                _mark_request(shared_timestamp)
+                if exc.code != 429:
+                    raise
+                if attempt == 2:
+                    has_api_key = any(name.lower() == "x-api-key" for name in request_headers)
+                    if host == "api.semanticscholar.org" and not has_api_key:
+                        message = (
+                            f"{exc.reason}; configure semantic-scholar.api-key with "
+                            "`paperstack config set semantic-scholar.api-key` for more reliable access"
+                        )
+                        raise urllib.error.HTTPError(exc.url, exc.code, message, exc.headers, exc.fp) from exc
+                    raise
+                time.sleep(_retry_delay(exc, attempt))
     raise RuntimeError("unreachable request retry state")
 
 
@@ -305,11 +372,50 @@ def fetch_all(
     return results
 
 
-def search(source: str, query: str, *, local_only: bool = False) -> dict:
+def _content_value(note: dict, name: str):
+    value = (note.get("content") or {}).get(name)
+    return value.get("value") if isinstance(value, dict) and "value" in value else value
+
+
+def _normalized_title(value: object) -> str:
+    return re.sub(r"\W+", "", unicodedata.normalize("NFKC", str(value or "")).casefold())
+
+
+def _openreview_status(note: dict) -> str:
+    venue = _content_value(note, "venue") or _content_value(note, "venueid") or _content_value(note, "venue_id")
+    fields = [*note.get("invitations", []), venue, _content_value(note, "decision"), _content_value(note, "status")]
+    text = " ".join(str(value) for value in fields if value).casefold()
+    if "withdraw" in text:
+        return "withdrawn"
+    if "reject" in text:
+        return "rejected"
+    if "accept" in text or venue and not any(word in str(venue).casefold() for word in ("submission", "submitted")):
+        return "accepted"
+    return "submission"
+
+
+def search(
+    source: str,
+    query: str,
+    *,
+    limit: int = 10,
+    local_only: bool = False,
+    exact_title: bool = False,
+    openreview_status: str | None = None,
+) -> dict:
+    query = query.strip()
+    if not query:
+        raise ValueError("paper search query is required")
+    if not 1 <= limit <= 100:
+        raise ValueError("limit must be between 1 and 100")
+    if (exact_title or openreview_status) and source != "openreview":
+        raise ValueError("exact title and OpenReview status filters require the openreview source")
+    if openreview_status not in (None, "submission", "accepted", "withdrawn"):
+        raise ValueError("OpenReview status must be submission, accepted, or withdrawn")
     if source == "dblp":
         from . import dblp_index
 
-        if hits := dblp_index.search(query):
+        if hits := dblp_index.search(query, limit=limit):
             return _result("dblp", str(dblp_index.index_path()), {"query": query, "matches": hits})
         if local_only:
             return {
@@ -319,19 +425,25 @@ def search(source: str, query: str, *, local_only: bool = False) -> dict:
                 "reason": "not found in local index",
             }
         url = "https://dblp.org/search/publ/api"
+        has_field_token = re.search(r"(?:^|\s)(?:author|title|venue|year|type|stream|toc):[^:]+:", query)
+        remote_query = query if has_field_token else re.sub(r":\s+", " ", query)
         return _safe(
-            lambda: _result("dblp", url, _get_json(url, {"q": query, "format": "json", "h": 10})), "dblp", url
+            lambda: _result("dblp", url, _get_json(url, {"q": remote_query, "format": "json", "h": limit})),
+            "dblp",
+            url,
         )
     if source == "crossref":
         url = "https://api.crossref.org/works"
         return _safe(
-            lambda: _result("crossref", url, _get_json(url, {"query.title": query, "rows": 10})), "crossref", url
+            lambda: _result("crossref", url, _get_json(url, {"query.title": query, "rows": limit})),
+            "crossref",
+            url,
         )
     if source == "arxiv":
         url = "https://export.arxiv.org/api/query"
 
         def arxiv_search() -> dict:
-            root = ET.fromstring(_get_text(url, {"search_query": f'ti:"{query}"', "max_results": 10}))
+            root = ET.fromstring(_get_text(url, {"search_query": f'ti:"{query}"', "max_results": limit}))
             matches = [
                 {
                     "id": entry.findtext("atom:id", "", ARXIV_NS),
@@ -347,40 +459,75 @@ def search(source: str, query: str, *, local_only: bool = False) -> dict:
         token = os.environ.get("OPENREVIEW_ACCESS_TOKEN")
         headers = {"Cookie": f"openreview.accessToken={token}"} if token else {}
         endpoints = (
-            "https://api2.openreview.net/notes/search",
-            "https://api.openreview.net/notes/search",
+            ("https://api2.openreview.net/notes/search", True),
+            ("https://api.openreview.net/notes/search", False),
         )
         matches = []
         errors = []
-        for endpoint in endpoints:
-            result = _safe(
-                lambda endpoint=endpoint: _result(
+        for endpoint, is_v2 in endpoints:
+            endpoint_matches = []
+            local_filter = bool(openreview_status or (exact_title and not is_v2))
+            page_size = min(max(limit * 2, 20), 100) if local_filter else limit
+            params = {"limit": page_size, "source": "forum", "cache": "true"}
+            if exact_title and is_v2:
+                params.update({"term": query, "type": "exact", "content": "title"})
+            else:
+                params["query"] = query
+            for offset in range(0, 1000, page_size):
+                page_params = {**params, "offset": offset} if offset else params
+                result = _safe(
+                    lambda endpoint=endpoint, page_params=page_params: _result(
+                        "openreview",
+                        endpoint,
+                        _get_json(endpoint, page_params, headers),
+                    ),
                     "openreview",
                     endpoint,
-                    _get_json(endpoint, {"query": query, "limit": 10, "source": "forum"}, headers),
-                ),
-                "openreview",
-                endpoint,
-            )
-            if result["status"] == "ok":
-                matches.extend(result.get("response", {}).get("notes", []))
-            else:
-                errors.append(result.get("error", endpoint))
+                )
+                if result["status"] != "ok":
+                    errors.append(result.get("error", endpoint))
+                    break
+                notes = result.get("response", {}).get("notes", [])
+                matches.extend(notes)
+                endpoint_matches.extend(notes)
+                if not local_filter or len(notes) < page_size:
+                    break
+                eligible = endpoint_matches
+                if exact_title:
+                    wanted = _normalized_title(query)
+                    eligible = [
+                        note for note in eligible if _normalized_title(_content_value(note, "title")) == wanted
+                    ]
+                if openreview_status:
+                    eligible = [note for note in eligible if _openreview_status(note) == openreview_status]
+                eligible_ids = {note.get("forum") or note.get("id") for note in eligible}
+                if len(eligible_ids) >= limit:
+                    break
         if not matches and errors:
-            return _result("openreview", endpoints[0], error="; ".join(errors))
+            return _result("openreview", endpoints[0][0], error="; ".join(errors))
         unique = {}
         for note in matches:
             unique[note.get("forum") or note.get("id") or json.dumps(note, sort_keys=True)] = note
+        selected = list(unique.values())
+        if exact_title:
+            wanted = _normalized_title(query)
+            selected = [note for note in selected if _normalized_title(_content_value(note, "title")) == wanted]
+        if openreview_status:
+            selected = [note for note in selected if _openreview_status(note) == openreview_status]
         return _result(
             "openreview",
-            endpoints[0],
-            {"query": query, "matches": list(unique.values()), "api_endpoints": list(endpoints)},
+            endpoints[0][0],
+            {
+                "query": query,
+                "matches": selected[:limit],
+                "api_endpoints": [endpoint for endpoint, _ in endpoints],
+            },
         )
     if source == "s2":
         url = "https://api.semanticscholar.org/graph/v1/paper/search"
         api_key = credentials.get(credentials.SEMANTIC_SCHOLAR_API_KEY)
         headers = {"x-api-key": api_key} if api_key else {}
-        params = {"query": query, "limit": 10, "fields": S2_FIELDS}
+        params = {"query": query, "limit": limit, "fields": S2_FIELDS}
         return _safe(
             lambda: _result("semantic_scholar", url, _get_json(url, params, headers)), "semantic_scholar", url
         )

{paperstack_cli-0.3.2 → paperstack_cli-0.4.1}/src/paperstack/viewer.py

@@ -94,11 +94,12 @@ def build(root: Path, output: Path) -> int:
     ):
         raise ValueError("viewer output would replace a broad path, the corpus, or authored entries")
     backup = destination.parent / f".{destination.name}.backup"
-    if backup.exists() and not destination.exists():
+    if backup.exists():
         marker = backup / ".paperstack-viewer"
         if not backup.is_dir() or not marker.is_file() or marker.read_text(encoding="utf-8") != "1\n":
             raise ValueError(f"viewer backup is not owned by Paperstack: {backup}")
-        os.replace(backup, destination)
+        if not destination.exists():
+            os.replace(backup, destination)
     if destination.exists() and not destination.is_dir():
         raise ValueError(f"viewer output is not a directory: {destination}")
     if destination.exists() and any(destination.iterdir()):

paperstack_cli-0.3.2/src/paperstack/content/arxiv_pdf.py

@@ -1,103 +0,0 @@
-"""Convert native-PDF arXiv submissions to Markdown.
-
-Used by `paperstack paper pdf` after the source fetcher reports no TeX.
-"""
-
-from __future__ import annotations
-
-import hashlib
-import json
-import os
-import sys
-import time
-import urllib.error
-import urllib.request
-from pathlib import Path
-
-_CACHE_ROOT = Path(os.environ.get("XDG_CACHE_HOME", Path.home() / ".cache"))
-CACHE_DIR = Path(os.environ.get("PAPERSTACK_PAPERS_DIR", _CACHE_ROOT / "paperstack" / "papers"))
-
-
-def _fetch(url: str, timeout: int = 60) -> bytes | None:
-    req = urllib.request.Request(url, headers={"User-Agent": "paperstack/1.0 (+arxiv pdf fetch)"})
-    for attempt in range(3):
-        try:
-            with urllib.request.urlopen(req, timeout=timeout) as resp:
-                return resp.read()
-        except urllib.error.HTTPError as e:
-            if e.code == 429:
-                time.sleep(15 * (attempt + 1))
-                continue
-            if attempt == 2:
-                return None
-            time.sleep(5)
-        except (urllib.error.URLError, OSError, TimeoutError):
-            if attempt == 2:
-                return None
-            time.sleep(5)
-    return None
-
-
-def convert(arxiv_id: str) -> bool:
-    d = CACHE_DIR / arxiv_id
-    md_path = d / "paper.md"
-    if md_path.exists() and md_path.stat().st_size > 1000:
-        print(f"{arxiv_id}: cached at {md_path}")
-        return True
-
-    try:
-        import pymupdf4llm
-    except ImportError:
-        print(
-            "PDF conversion needs `paperstack-cli[pdf]`; reinstall with "
-            "`uv tool install --force 'paperstack-cli[pdf]'`.",
-            file=sys.stderr,
-        )
-        return False
-
-    url = f"https://arxiv.org/pdf/{arxiv_id}"
-    raw = _fetch(url)
-    if raw is None:
-        print(f"{arxiv_id}: could not fetch {url}", file=sys.stderr)
-        return False
-    if not raw.startswith(b"%PDF"):
-        print(f"{arxiv_id}: {url} did not return a PDF", file=sys.stderr)
-        return False
-
-    d.mkdir(parents=True, exist_ok=True)
-    pdf_path = d / "paper.pdf"
-    pdf_path.write_bytes(raw)
-    md = pymupdf4llm.to_markdown(str(pdf_path))
-    if len(md) < 500:
-        print(f"{arxiv_id}: converted to only {len(md)} chars, treat as a failure", file=sys.stderr)
-        return False
-
-    md_path.write_text(md, encoding="utf-8")
-    (d / "meta.json").write_text(
-        json.dumps(
-            {
-                "arxiv_id": arxiv_id,
-                "bytes": len(md),
-                "sha256": hashlib.sha256(md.encode("utf-8")).hexdigest(),
-                "url": url,
-                "converter": "pymupdf4llm",
-            },
-            indent=2,
-        ),
-        encoding="utf-8",
-    )
-    print(f"{arxiv_id}: {len(md)} chars -> {md_path}")
-    return True
-
-
-def main(argv: list[str]) -> None:
-    if not argv:
-        sys.exit("pass one or more arXiv ids")
-    CACHE_DIR.mkdir(parents=True, exist_ok=True)
-    failures = [a for a in argv if not convert(a)]
-    if failures:
-        sys.exit(f"failed: {' '.join(failures)}")
-
-
-if __name__ == "__main__":
-    main(sys.argv[1:])