PyPI - pyPaperFlow - Versions diffs - 0.3.0__tar.gz → 0.6.0__tar.gz - Mend

pyPaperFlow 0.3.0tar.gz → 0.6.0tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (36) hide show

pypaperflow-0.6.0/.claude/settings.local.json ADDED Viewed

@@ -0,0 +1,8 @@
+{
+  "permissions": {
+    "allow": [
+      "Bash(curl -s -m 40 --noproxy '*' \"https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=SRC:PPR%20AND%20zinc%20AND%20finger%20AND%20263%20AND%20FIRST_PDATE:%5B2026-08-01%20TO%202026-12-31%5D&format=json&pageSize=25&cursorMark=*&resultType=core\" -o /tmp/epmc_date.json -w \"HTTP=%{http_code}\\\\n\")",
+      "Bash(python3 -c ' *)"
+    ]
+  }
+}

pypaperflow-0.6.0/.github/workflows/docs.yml ADDED Viewed

@@ -0,0 +1,35 @@
+name: docs
+on:
+  push:
+    branches: [main]
+    paths:
+      - "README_zh.md"
+  workflow_dispatch:
+permissions:
+  contents: read
+  pages: write
+  id-token: write
+concurrency:
+  group: pages
+  cancel-in-progress: true
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - run: pip install -r requirements-docs.txt
+      - run: python scripts/sync_docs.py
+      - run: mkdocs build
+      - uses: actions/configure-pages@v6
+      - uses: actions/upload-pages-artifact@v5
+        with:
+          path: site
+      - id: deployment
+        uses: actions/deploy-pages@v5

pypaperflow-0.6.0/.gitignore ADDED Viewed

@@ -0,0 +1,24 @@
+# build product
+dist/
+build/
+*.egg-info/
+# Python cache
+__pycache__/
+*.py[cod]
+# environment
+.env
+venv/
+CLAUDE.md
+pdfs/
+refers/
+# mkdocs build output
+site/
+.cache/
+# generated docs (from scripts/sync_docs.py)
+# ignore everything under mkdoc_site/ except the hand-written homepage
+mkdoc_site/*
+!mkdoc_site/index.md

{pypaperflow-0.3.0 → pypaperflow-0.6.0}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyPaperFlow
-Version: 0.3.0
+Version: 0.6.0
 Summary: Automated paper fetching and analysis platform.
 Project-URL: Homepage, https://github.com/MaybeBio/pyPaperFlow
 Project-URL: Issues, https://github.com/MaybeBio/pyPaperFlow/issues
@@ -35,7 +35,7 @@ Description-Content-Type: text/markdown
 <p><strong>An automated literature processing platform for scientific researchers.</strong></p>
-<p>Batch retrieve, fetch, parse, and structure papers from PubMed, arXiv, bioRxiv, and DOI-based sources.</p>
+<p>Batch retrieve, fetch, parse, and structure papers from PubMed, arXiv, bioRxiv, ChemRxiv, and DOI-based sources.</p>
 <p>From paper retrieval to knowledge internalization, automate the heavy lifting and keep the judgment human.</p>
@@ -44,17 +44,20 @@ Description-Content-Type: text/markdown
 [![License: GPL v3](https://img.shields.io/badge/License-GPL%20v3-blue.svg)](https://www.gnu.org/licenses/gpl-3.0)
 [![PR's Welcome](https://img.shields.io/badge/PRs-welcome-brightgreen.svg?style=flat)](http://makeapullrequest.com)
 [![Workflow](https://img.shields.io/badge/Workflow-7%20Stages-0366d6)](docs/Design.md)
-[![Sources](https://img.shields.io/badge/Sources-PubMed%20%2F%20arXiv%20%2F%20bioRxiv-f59e0b)](#features)
+[![Sources](https://img.shields.io/badge/Sources-PubMed%20%2F%20arXiv%20%2F%20bioRxiv%20%2F%20medRxiv%20%2F%20chemRxiv-f59e0b)](#features)
 [![PyPI version](https://img.shields.io/pypi/v/pyPaperFlow.svg?logo=pypi&logoColor=white)](https://pypi.org/project/pyPaperFlow/)
 [![Python Versions](https://img.shields.io/pypi/pyversions/pyPaperFlow.svg?logo=python&logoColor=white)](https://pypi.org/project/pyPaperFlow/)
 [![Downloads](https://static.pepy.tech/badge/pyPaperFlow)](https://pepy.tech/project/pyPaperFlow)
+[![Docs](https://img.shields.io/badge/Docs-online_docs-2ea44f)](https://maybebio.github.io/pyPaperFlow/)
 <p>
-  Document here 👉
+  文档阅读👉
   <a href="README.md">English</a> |
-  <a href="README_zh.md">中文</a>
+  <a href="README_zh.md">中文</a> |
+  <a href="https://maybebio.github.io/pyPaperFlow/" target="_blank">中文在线文档</a>
 </p>
 <p>
   <a href="./docs/Design.md">Design</a> |
   <a href="./docs/Cases.md">Cases</a>
@@ -179,6 +182,8 @@ flowchart TD
 ## 📦 Installation
+> ⚠️ For typical usage, you only need to install our tool, seen below:
 ```bash
 # 1. install our tool
 ## ✏️1️⃣ option1: Install via pip (Recommended)
@@ -188,9 +193,11 @@ pip install pyPaperFlow
 git clone https://github.com/MaybeBio/pyPaperFlow.git
 cd pyPaperFlow
 pip install -e .
+```
---------------------------------------------------------
+> Some optional dependencies, if you do not need to use the corresponding functional modules, you can ignore the installation. seen below:
+```bash
 # 2. install MinerU
 # follow the official installation guide: https://github.com/opendatalab/MinerU
 # verify installation: mineru --help
@@ -210,10 +217,11 @@ pip install openai anthropic
 pip install paperscraper
 ```
-> ⚠️ For typical usage, you only need to install the repository from source and MinerU, which are steps 1 and 2.
 ## 🛠️ Usage
+> 📌 **Tip**: If you want to get started directly, please refer to the usage examples in [Cases.md](./docs/Cases.md). The content below is theoretical process analysis and can be skipped.
 We designed pyPaperFlow as a versatile academic research tool built strictly around the `real‑world workflow of researchers conducting literature investigation, paper reading, literature comprehension and analysis, and corpus utilization`.
 Therefore, please follow our step‑by‑step operations, which mirror your full literature research process. Through this hands‑on experience, you will fully grasp the design philosophy and usage of this tool.
@@ -227,33 +235,39 @@ Current available modules include (`will be continuously updated`):
 ```python
 ❯ paperflow --help
- Usage: paperflow [OPTIONS] COMMAND [ARGS]...
- pyPaperFlow CLI
-╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --install-completion          Install completion for the current shell.                                                                                                               │
-│ --show-completion             Show completion for the current shell, to copy it or customize the installation.                                                                        │
-│ --help                        Show this message and exit.                                                                                                                             │
-╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-╭─ Commands ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ pubmed-search      Search PubMed using Your customized query and return PMIDs.                                                                                                        │
-│ pubmed-meta        Fetch paper metadata from PubMed using Your customized query, pmid list file and save to storage.                                                                  │
-│ pubmed-content     Download full text (PMC) for given PMIDs if the paper has a PMC ID.                                                                                                │
-│ pubmed-all         Fetch BOTH metadata and full text (if available) for papers.                                                                                                       │
-│                    Also extracts URLs from full text and updates metadata links.                                                                                                      │
-│ pubmed-merge-json  Create a merged JSON (or JSONL) file from PubMed paper directories.                                                                                                │
-│ pubmed-export-md   Export a single Markdown view from a merged JSON file using optional YAML config.                                                                                  │
-│ arxiv-search       Search arXiv and write matching IDs to a text file.                                                                                                                │
-│ arxiv-fetch        Fetch arXiv metadata and attempt to download PDFs.                                                                                                                 │
-│ biorxiv-search     Search bioRxiv and write matching IDs to a text file.                                                                                                              │
-│ biorxiv-fetch      Fetch bioRxiv metadata and attempt to download PDFs.                                                                                                               │
-│ paper-fetch        Fetch PDFs by DOI — passes through to the paper-fetch engine.                                                                                                      │
-│ pdf-parse          Parse a PDF file using MinerU engine, and clean up the output directory.                                                                                           │
-│ mineru-parse       Parse mineru output content_list_v2.json into canonical sectioned JSON.                                                                                            │
-│ mineru-export-md   Export structured mineru JSON to a clean Markdown file for LLM processing.                                                                                         │
-╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+ Usage: paperflow [OPTIONS] COMMAND [ARGS]...
+ pyPaperFlow CLI
+╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --install-completion          Install completion for the current shell.                                                                    │
+│ --show-completion             Show completion for the current shell, to copy it or customize the installation.                             │
+│ --help                        Show this message and exit.                                                                                  │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+╭─ Commands ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ pubmed-search      Search PubMed using Your customized query and return PMIDs.                                                             │
+│ pubmed-meta        Fetch paper metadata from PubMed using Your customized query, pmid list file and save to storage.                       │
+│ pubmed-content     Download full text (PMC) for given PMIDs if the paper has a PMC ID.                                                     │
+│ pubmed-all         Fetch BOTH metadata and full text (if available) for papers.                                                            │
+│                    Also extracts URLs from full text and updates metadata links.                                                           │
+│ pubmed-merge-json  Create a merged JSON (or JSONL) file from PubMed paper directories.                                                     │
+│ pubmed-export-md   Export a single Markdown view from a merged JSON file using optional YAML config.                                       │
+│ arxiv-search       Search arXiv and write matching IDs to a text file.                                                                     │
+│ arxiv-fetch        Fetch arXiv metadata and attempt to download PDFs.                                                                      │
+│ biorxiv-search     Search bioRxiv and write matching IDs to a text file.                                                                   │
+│ biorxiv-fetch      Fetch bioRxiv metadata and attempt to download PDFs.                                                                    │
+│ medrxiv-search     Search medRxiv and write matching IDs to a text file.                                                                   │
+│ medrxiv-fetch      Fetch medRxiv metadata and attempt to download PDFs.                                                                    │
+│ chemrxiv-search    Search ChemRxiv and write matching IDs to a text file.                                                                  │
+│ chemrxiv-fetch     Fetch ChemRxiv metadata and attempt to download PDFs.                                                                   │
+│ paper-fetch        Fetch PDFs by DOI — passes through to the paper-fetch engine.                                                           │
+│ pdf-parse          Parse a PDF file using MinerU engine, and clean up the output directory.                                                │
+│ mineru-parse       Parse mineru output content_list_v2.json into canonical sectioned JSON.                                                 │
+│ mineru-export-md   Export structured mineru JSON to a clean Markdown file for LLM processing.                                              │
+│ github-export      Export GitHub links from merged PubMed JSON, validate accessibility,                                                    │
+│                    and aggregate `ghresearcher parse <owner/repo> --view` outputs into one markdown.                                       │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
@@ -275,11 +289,20 @@ bioRxiv Modules:
 - biorxiv-search # search bioRxiv and return matching IDs
 - biorxiv-fetch # fetch bioRxiv metadata and attempt to download PDFs
+medRxiv Modules:
+- medrxiv-search # search medRxiv and return matching IDs
+- medrxiv-fetch # fetch medRxiv metadata and attempt to download PDFs
+ChemRxiv Modules:
+- chemrxiv-search # search ChemRxiv and return matching IDs
+- chemrxiv-fetch # fetch ChemRxiv metadata and attempt to download PDFs
 Third-party Modules:
 - paper-fetch # fetch PDFs by DOI
 - pdf-parse # parse PDF files into JSON, Markdown format using the MinerU engine
 - mineru-parse # Based on your custom section configuration, re-parse the MinerU output file into a structured JSON format clustered by standard literature sections
 - mineru-export-md # Based on your custom section configuration, export the structured mineru JSON to a clean Markdown file for LLM processing (🌟 e.g., batch export of introductions as your research background)
+- github-export # export GitHub links from merged PubMed JSON, validate accessibility, and aggregate `ghresearcher parse <owner/repo> --view` outputs into one markdown
 ```
 > ⚠️ `Other preprint platforms modules are under development, please stay tuned!`
@@ -323,6 +346,20 @@ Our literature database primarily covers biomedical research and computational i
 - arXiv
 - bioRxiv，medRxiv，chemRxiv
+> **⚠️ — Preprint search = Crossref relevance search + local boolean re-check (not a full-corpus pull, and not each server's official API).** Each request asks Crossref to search ONLY that platform's prefix (`filter=prefix:10.64898 / 10.26434,type:posted-content`) — platform scoping happens server-side, not by post-filtering a global result set. bioRxiv and medRxiv share the openRxiv prefix `10.64898`, so those two are then told apart locally by DOI-accession digit count (6 = bioRxiv, 8 = medRxiv).
+>
+> How the relevance step behaves: `query.bibliographic` is a fuzzy, OR-like ranking (a two-term query returns more than either single term — e.g. chemRxiv "base editing" ≈ the union of "base" and "editing"), i.e. a **superset** of the strict matches. The fetcher cursor-paginates the whole result set (not a capped top-N), then keeps only records in which **every** query term is actually present in the metadata (local boolean AND over title/abstract/…). Because strict matches ⊆ relevance superset, ranking never drops a metadata-level exact match — it only changes the order.
+>
+> **Limitations of the relevance search:**
+>
+> ① **Deposit lag** — a preprint posted minutes ago may not be indexed in Crossref yet.
+>
+> ② **Metadata-only matching** — Crossref scores deposited metadata (title/abstract/…), so query terms that appear only in the paper body are invisible to it. Only the bioRxiv/medRxiv Europe PMC leg indexes full text; **ChemRxiv has no full-text leg at all**, so body-only-term misses are expected there.
+>
+> ③ **Version duplication** — every revision is its own DOI work, so `.../v1` and `.../v2` both match a search and may need manual dedup.
+>
+> **Difference vs. exhaustive full-corpus enumeration:** a relevance search is a heuristic over the deposited metadata. The "no-omission-by-construction" alternative is to list the *whole* platform corpus (`filter=prefix…` with no `query`, cursor-paging every record — ≈ 55k ChemRxiv / 436k openRxiv) and run the boolean AND locally, with no relevance engine in the loop; recall is then exactly "all records whose metadata fully matches the query" (a `--start/--end-date` window shrinks the pull). The cost is downloading the full corpus per search, and it still inherits the source-level boundaries above (deposit lag, metadata-only, version duplication). This tool's `search()` path is relevance-based today; the exhaustive mode is not currently exposed as a flag.
 We recommend that you proactively learn and master the search syntax of these databases, as our built‑in search module functions similarly to the search bar on official web portals.
 For instance, here is a typical complex query example tailored for PubMed:
@@ -599,6 +636,8 @@ examples:
 We acknowledge the work of [paper-fetch](https://github.com/Agents365-ai/paper-fetch)！We have modified, refactored, and encapsulated one of its core scripts for tailored integration into our pipeline.
+> 🔙 Rollback boundary for the paper-fetch module: commit `bc8394c` (`update paper-fetch module according to upstream repo`) contains the **original upstream script**. Commit `89eda06bac1f853254b04aee9e8916109c7771a1` is the first local **modification** to it. To restore the original module, use `89eda06` as the boundary — e.g. `git show 89eda06^:src/pyPaperFlow/integrations/pdf_fetch.py` (identical to `bc8394c`).
 The workflow of our paper acquisition module is outlined below:
 ```bash
@@ -624,6 +663,8 @@ The workflow of our paper acquisition module is outlined below:
 ┌─────────────────────────────────────────┐
 │  3. arXiv (via S2 externalIds.ArXiv)     │
 │  4. Europe PMC → PMC (via PMCID)         │
+│     No PMCID: DOI→PMCID recovery         │
+│     (Europe PMC hasPDF=Y / OpenAIRE)     │
 │  5. bioRxiv / medRxiv (DOI prefix: 10.1101/)
 └─────────────────────────────────────────┘
            Total Failure ↓
@@ -634,7 +675,12 @@ The workflow of our paper acquisition module is outlined below:
 └─────────────────────────────────────────┘
            Persistent Failure ↓
 ┌─────────────────────────────────────────┐
-│  7. Sci‑Hub Mirror Fallback (enabled by default, configurable)
+│  7. CORE Repository Aggregator (optional, requires CORE_API_KEY)
+│     → core.ac.uk aggregates OA full-text downloadUrl
+└─────────────────────────────────────────┘
+           Persistent Failure ↓
+┌─────────────────────────────────────────┐
+│  8. Sci‑Hub Mirror Fallback (enabled by default, configurable)
 │     → 1 request‑per‑second rate‑limiting to prevent CAPTCHA triggers
 │     → Automatic discovery of active new mirrors
 └─────────────────────────────────────────┘
@@ -647,9 +693,10 @@ Resolution Priority Sequence
 Unpaywall: The optimal open‑access source covering the broadest range of publishers with the highest hit rate.
 Semantic Scholar: Retrieves OA PDF links and cross‑platform external identifiers.
 arXiv: Activated when an arXiv identifier is available for the target paper.
-PubMed Central (PMC) OA Subset: Activated when a PMCID is associated with the paper.
+PubMed Central (PMC) OA Subset: Activated when a PMCID is associated with the paper; when no PMCID is known, a DOI→PMCID recovery is attempted first (Europe PMC search hasPDF=Y / OpenAIRE originalId).
 bioRxiv / medRxiv: Triggered for preprints with the DOI prefix 10.1101/.
 Publisher Direct Links: Enabled only under institutional mode (PAPER_FETCH_INSTITUTIONAL=1), authorized via the caller’s institutional subscription IP, cookies, or EZproxy access.
+CORE Repository Aggregator: Optional, requires CORE_API_KEY; aggregates full texts from many repositories, attempted only when all other OA sources miss (free tier ~5 req/10s).
 Sci‑Hub Mirror Fallback: Enabled by default as the final retrieval backup.
 Mirrors are attempted in the order specified by the environment variable PAPER_FETCH_SCIHUB_MIRRORS (default list: sci‑hub.ru, sci‑hub.st, sci‑hub.su, sci‑hub.box, sci‑hub.red, sci‑hub.al, sci‑hub.mk, sci‑hub.ee).
 If all predefined mirrors fail, the module fetches the latest live mirror list from https://www.sci‑hub.pub/ and retries.
@@ -665,6 +712,120 @@ export UNPAYWALL_EMAIL=you@example.com
 ```
+**Cloudflare-blocked PDFs (optional)**
+Some publishers (e.g. `science.org`) sit behind Cloudflare and return `403`/`429` or a "Just a moment…" JS challenge page instead of the PDF. Set `PAPER_FETCH_CLOAK=1` to retry those URLs through [CloakBrowser](https://github.com/CloakHQ/CloakBrowser) (a stealth Chromium that can pass the challenge). This fallback lives at the download layer (covers all sources), fails silently when CloakBrowser is unavailable, is operator-controlled (agents cannot enable it), and re-validates returned bytes through the same `%PDF` + 50 MB checks. Successful cloak downloads carry `via:"cloak"`.
+Setup — install once, then treat `PAPER_FETCH_CLOAK` as an always-on safety net:
+```bash
+# One-time install: put cloakbrowser into the same Python that runs paperflow
+# (auto-detected), or into a separate venv and point CLOAKBROWSER_PYTHON at it.
+pip install cloakbrowser
+# Recommended: keep as a safety net (fires ONLY when a download is blocked;
+# normal OA downloads are unaffected).
+export PAPER_FETCH_CLOAK=1
+# Cleaner per-command alternative, when you only occasionally hit a blocked URL:
+PAPER_FETCH_CLOAK=1 paperflow paper-fetch 10.1126/sciadv.aee6105 --out ./pdfs
+# ⚠️ PAPER_FETCH_CLOAK_HEADED=1 is NOT a default. Set it only for strong
+# challenges (e.g. science.org) AND only on a machine with a display:
+# export PAPER_FETCH_CLOAK_HEADED=1
+```
+> **Practical notes (from testing)**
+> - Cloak is **not** a paywall bypass. It only fires when an *OA* download is Cloudflare-blocked; a paywalled paper (no OA copy, e.g. `10.1016/j.cels.2025.101486`) never reaches the download layer, so Cloak is never invoked.
+> - `science.org` is a "strong" challenge that **headless cannot pass** (it stalls on "Just a moment…") — it requires `PAPER_FETCH_CLOAK_HEADED=1` on a real display (a desktop, or a headless server wrapped with `xvfb-run`).
+> - Cloak only has a URL to retry when a source returned a direct `url_for_pdf`. Papers whose only OA copy is in PMC (Unpaywall reports a landing page with no `url_for_pdf`) are **not** attempted by the stock script.
+> - Keeping `PAPER_FETCH_CLOAK=1` always on has no effect on normal downloads, but once `cloakbrowser` is installed, each blocked URL adds a ~30–90 s browser attempt before falling through; use the inline form if you'd rather not wait.
+> - Sci-Hub discovery contacts `www.sci-hub.pub` — on networks where that DNS is blocked you'll see `scihub_discover_failed`, which removes the last fallback. For genuinely unavailable papers, use institutional mode (`PAPER_FETCH_INSTITUTIONAL=1`) or download via a browser / interlibrary loan.
+**Institutional access (optional)**
+Some paywalled papers are exactly what your institution's subscription covers — a generic OA source just can't see them. Set `PAPER_FETCH_INSTITUTIONAL=1` to enable the publisher-direct-link source (step 6 of the chain): the script builds a direct PDF URL for the DOI's publisher and downloads it.
+```bash
+export PAPER_FETCH_INSTITUTIONAL=1   # only effective while on the institutional network
+```
+> **Key requirement (verified by testing):** authorization comes from the *caller's network*, not the script. Publisher-direct downloads succeed only when the machine runs **on-campus or on the institutional VPN** — the publisher recognizes your institution's IP range (or cookies / EZproxy). From an off-network IP (a datacenter or home machine) the URL is still built correctly but the publisher answers `HTTP 403 Forbidden`, and the run ends with `download_network_error` (retryable) instead of `not_found` — that error-type change is how you can tell institutional mode actually fired.
+> - The result envelope's `auth_mode` field reports `"institutional"` (vs `"public"`).
+> - Direct-link templates are matched by DOI prefix; Elsevier (`10.1016/`) needs an extra PII lookup via Crossref and `sciencedirect.com/.../pdfft` — verified working. Supported publishers: Nature, Science, Wiley, Springer, ACS, PNAS, NEJM, SAGE, Taylor & Francis, Elsevier, MDPI.
+> - Auto rate-limits to **1 req/s** to respect publisher ToS (protects your institution's IP from throttling).
+> - In public mode, when a paper looks paywalled the error payload sets `suggest_institutional: true` and hints to set this variable and re-run from on-campus / VPN.
+**CORE repository-aggregator fallback (optional)**
+When a paper misses across all OA sources (Unpaywall / Semantic Scholar / arXiv / PMC / bioRxiv) but an institutional or subject repository may still hold a copy, set `CORE_API_KEY` to enable the [CORE](https://core.ac.uk) (core.ac.uk) aggregator fallback. CORE aggregates full-text metadata from thousands of OA repositories and journals worldwide; its v3 search API queries by DOI and the `downloadUrl` field of a matching record is the directly downloadable OA full-text link (paywalled records leave this field empty and are skipped automatically).
+```bash
+export CORE_API_KEY=your_core_api_key   # free signup: https://core.ac.uk/services/api
+```
+> **Principle & notes**
+> - CORE is a *repository aggregator*, not a single publisher: it pools full texts from institutional and subject repositories, covering repository copies that other sources miss.
+> - This source fires **only** when all earlier OA sources (Unpaywall / Semantic Scholar / arXiv / PMC / bioRxiv) have missed — it never interferes with the normal OA download path, so keeping it on is harmless.
+> - Requires a free API key (Bearer auth); the source is silently skipped when `CORE_API_KEY` is unset.
+> - The free tier rate-limits to roughly **5 requests / 10 seconds** and returns `403` when exceeded — imperceptible for single papers, and batch fetching is serialized/throttled.
+> - Returned links pass the same `%PDF` magic-byte + 50 MB checks; successful hits carry `source:"core"`.
+> - Only records with a non-empty `downloadUrl` are used, naturally filtering out paywalled entries with no OA copy.
+**Recommended setup (best practice)**
+For a mixed workload — OA papers, Cloudflare-gated OA papers, and the occasional paywalled paper your institution subscribes to — keep three things on:
+```bash
+export UNPAYWALL_EMAIL=you@example.com   # fastest / broadest OA source
+export PAPER_FETCH_CLOAK=1               # safety net for Cloudflare-gated OA PDFs (harmless otherwise)
+# Run the line below only while on campus / the institutional VPN:
+export PAPER_FETCH_INSTITUTIONAL=1       # publisher-direct links for paywalled papers
+```
+Decision logic for a single paper:
+- **OA paper** → Unpaywall / Semantic Scholar / arXiv / PMC handle it; `PAPER_FETCH_CLOAK` only adds a retry when the download is Cloudflare-blocked.
+- **Cloudflare-blocked OA** (e.g. `science.org`) → Cloak retries it (headless may stall; use `PAPER_FETCH_CLOAK_HEADED=1` on a machine with a display).
+- **Paywalled paper** (e.g. `10.1016/j.cels.2025.101486`) → only the institutional chain can fetch it, and only from on-campus / VPN: Unpaywall → Semantic Scholar → publisher-direct (Elsevier via PII lookup → `sciencedirect.com/.../pdfft`) → Sci-Hub fallback. From an off-network IP the publisher answers `403` and no automated path remains — use your library portal / EZproxy or interlibrary loan.
+**Notes & Environment Variables**
+All settings are read from environment variables when the process starts — there is no config file. Every var above can be combined freely.
+| Env var | Effect | Default | When to set |
+| --- | --- | --- | --- |
+| `UNPAYWALL_EMAIL` | Contact email for the Unpaywall API (sent in the User-Agent); without it the Unpaywall source is skipped. | empty | Recommended — Unpaywall is the fastest / broadest source |
+| `CORE_API_KEY` | API key for the CORE (core.ac.uk) aggregator; without it the `core` repository source is skipped. | empty | When you need coverage for institutional / subject repository OA copies |
+| `PAPER_FETCH_NO_SCIHUB` | Set to `1` to disable the Sci-Hub mirror fallback. | Sci-Hub ON | If your institution / compliance forbids Sci-Hub |
+| `PAPER_FETCH_SCIHUB_MIRRORS` | Comma-separated mirror list, tried in priority order (hostnames only). | built-in default list | When the default mirrors stop working |
+| `PAPER_FETCH_INSTITUTIONAL` | Set to `1` to enable publisher direct links (authorized by your institutional IP / cookies / EZproxy). Auto rate-limits to 1 req/s to respect publisher ToS. | off | If you have an institutional subscription |
+| `PAPER_FETCH_CLOAK` | Set to `1` to retry Cloudflare-blocked PDFs through CloakBrowser. | off | For publishers behind Cloudflare (e.g. `science.org`) |
+| `CLOAKBROWSER_PYTHON` | Python interpreter that can `import cloakbrowser`. | auto-detected | Only if not auto-detected |
+| `PAPER_FETCH_CLOAK_HEADED` | Set to `1` for a headed browser (needs a display). | headless | For strong challenges that fail headless (e.g. `science.org`) |
+> ⚠️ **Boolean flags are presence-based, not value-based.** The code checks `os.environ.get(...)`, so *any* non-empty value enables the feature. Setting `PAPER_FETCH_CLOAK=0` or `=false` still **enables** Cloak; setting `PAPER_FETCH_NO_SCIHUB=0` still **disables** Sci-Hub. To turn a feature off, `unset` the variable — never write `=0`/`=false`.
+**Usage notes**
+- The output directory flag is `--out` — there is **no `-o` short option** (the `paperflow paper-fetch` passthrough does not rewrite args).
+- One DOI as a positional arg; `-` reads a single DOI from stdin; `--batch FILE` (or `--batch -`) reads DOIs line-by-line.
+- Files already downloaded are skipped by default; pass `--overwrite` to force a re-download.
+- stdout carries the machine-readable JSON envelope, stderr carries progress; `--format json|text` plus TTY auto-detection control the shape. Exit codes: `0` all resolved, `1` some unresolved, `3` bad arguments, `4` transport error (retryable).
+- `paperflow paper-fetch schema` prints the machine-readable CLI schema (no network).
+- `--idempotency-key KEY` replays the original result envelope on retry, with no network I/O.
+**Known limitations**
+- Some publisher redirects land on HTML pages instead of PDFs — the `%PDF` magic-byte check rejects them.
+- No browser automation by default (no CAPTCHA solving) — only the optional `PAPER_FETCH_CLOAK` CloakBrowser fallback.
+- SSRF protection rejects private IPs, non-`http(s)` schemes, non-80/443 ports, and cloud metadata hosts.
+- Each PDF is capped at 50 MB.
 Unlike PMC parsing, non‑PubMed papers can only be obtained as PDF files via the paper‑fetch module.
 We recommend standardizing all paper information into Markdown or JSON formats.
@@ -1328,8 +1489,8 @@ You may directly run the test scripts to verify the correctness and completeness
 <details>
 <summary><b>3. Literature Acquisition (and Full‑Text Download)</b></summary>
-> - [ ] Refine and encapsulate the `paper‑fetch` module. Refer to [2026‑05‑08 paper‑fetch Encapsulation](https://github.com/Agents365‑ai/paper‑fetch); evaluate integration or replacement with more robust modules offering higher hit rates.
-> - [ ] The `pdf‑parse` module currently wraps basic MinerU parsing commands with the CPU backend (`‑b pipeline`). Future integration of GPU‑accelerated features; see [MinerU Repository](https://github.com/opendatalab/MinerU) for details.
+> - [x] Refine and encapsulate the `paper‑fetch` module. Refer to [2026‑05‑08 paper‑fetch Encapsulation](https://github.com/Agents365‑ai/paper‑fetch); evaluate integration or replacement with more robust modules offering higher hit rates.
+> - [x] The `pdf‑parse` module currently wraps basic MinerU parsing commands with the CPU backend (`‑b pipeline`). Future integration of GPU‑accelerated features; see [MinerU Repository](https://github.com/opendatalab/MinerU) for details.
 </details>
@@ -1348,7 +1509,10 @@ You may directly run the test scripts to verify the correctness and completeness
 <details>
 <summary><b>5. Processing for Other Literature Databases</b></summary>
-> - [ ] Develop a unified `search‑fetch‑parse` pipeline for non‑PubMed databases and complete corresponding modules. Refer to open‑source implementations such as [paperscraper](https://github.com/jannisborn/paperscraper) and [paper‑tracker](https://github.com/RainerSeventeen/paper‑tracker).
+> - [x] Develop a unified `search‑fetch‑parse` pipeline for non‑PubMed databases and complete corresponding modules. Refer to open‑source implementations such as [paperscraper](https://github.com/jannisborn/paperscraper) and [paper‑tracker](https://github.com/RainerSeventeen/paper‑tracker).
+> - [ ] Multi‑source retrieval merge (primitive ready, orchestration pending): `preprint/source_merge.py` already provides `merge_papers` (backfill missing DOIs → dedupe by DOI, key cascade DOI → title+authors → source_id). The cross‑source orchestration command is not yet implemented; when adding a unified `search --sources arxiv,biorxiv,medrxiv,chemrxiv` that merges results into a single corpus, call `merge_papers` directly.
+> - [ ] Optional connectors (add on demand): skip a standalone Semantic Scholar metadata connector — it overlaps with the S2 usage inside `pdf_fetch.py` (`openAccessPdf`/`externalIds`), avoiding dual maintenance. Introduce an OpenAlex connector (`api.openalex.org/works`, inverted‑index abstract reconstruction + authors + citations/references) only when there is a concrete need for batch DOI‑based metadata completion or citation‑graph harvesting; emit `SourcePaper`.
+> - [ ] Infrastructure alignment (partially done): `extract_doi` unified into `source_merge.py`; `safe_filename` already present (`source_utils.py`); `get_env` deferred (mismatches the existing flat `os.environ.get` style); OAI‑PMH base class deferred until a generic OAI repository is actually integrated.
 </details>

pyPaperFlow 0.3.0__tar.gz → 0.6.0__tar.gz

pyPaperFlow 0.3.0tar.gz → 0.6.0tar.gz