pyPaperFlow 0.6.0__tar.gz → 0.6.2__tar.gz

{pypaperflow-0.6.0 → pypaperflow-0.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyPaperFlow
-Version: 0.6.0
+Version: 0.6.2
 Summary: Automated paper fetching and analysis platform.
 Project-URL: Homepage, https://github.com/MaybeBio/pyPaperFlow
 Project-URL: Issues, https://github.com/MaybeBio/pyPaperFlow/issues
@@ -112,6 +112,7 @@ This tool is designed to `complement rather than replace` reference management s
 
 - **Automated Retrieval from Multiple Sources**: Automatically search and retrieve paper metadata and full-text records from `PubMed/Medline, arXiv, medRxiv, chemRxiv and bioRxiv`. The repository focuses primarily on biomedical research and computational interdisciplinary fields (`Biomedicine + Computational Biology`).
 - **Full-Text Access**: Enable automatic downloading of open-access full texts in XML/Text format from `PMC`. For preprints and other publications without accessible PMC full texts, alternative acquisition modules are integrated to fetch `original PDFs`, with `Sci-Hub` set as the fallback provider.
+- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` reads the native bioRxiv/medRxiv full-text HTML first (browser User-Agent + rate-limit handling), then falls back to Europe PMC fullTextXML for preprints already published into PMC.
 - **Structured Storage**:
   - **Metadata**: Preserved in well-structured detailed JSON files.
   - **Full Text**: Stored in multiple formats including parsed JSON and Markdown for versatile downstream usage — JSON for programmatic data analysis, and Markdown optimized for LLM comprehension and processing.
@@ -360,6 +361,8 @@ Our literature database primarily covers biomedical research and computational i
 >
 > **Difference vs. exhaustive full-corpus enumeration:** a relevance search is a heuristic over the deposited metadata. The "no-omission-by-construction" alternative is to list the *whole* platform corpus (`filter=prefix…` with no `query`, cursor-paging every record — ≈ 55k ChemRxiv / 436k openRxiv) and run the boolean AND locally, with no relevance engine in the loop; recall is then exactly "all records whose metadata fully matches the query" (a `--start/--end-date` window shrinks the pull). The cost is downloading the full corpus per search, and it still inherits the source-level boundaries above (deposit lag, metadata-only, version duplication). This tool's `search()` path is relevance-based today; the exhaustive mode is not currently exposed as a flag.
 
+**Retries & backoff (all preprint commands).** Every preprint fetcher retries failed HTTP requests with exponential backoff — the delay doubles per attempt (`1.5s → 3s → 6s → 12s → …`, capped at 30s) and honors the server's `Retry-After` header when present. The default budget is **3 retries (≈4.5s of backoff)**, tuned to fail fast for interactive use so you get a quick answer rather than a ~22s silent stall. Override it per command with the `--max-retries` flag (e.g. `biorxiv-search ... --max-retries 5`); unattended jobs (e.g. `monitor.py`) pass a larger value explicitly. For **bioRxiv / medRxiv**, when the Europe PMC full-text leg is unreachable (e.g. a transient upstream outage) the search degrades to Crossref-only metadata matching and prints a `Warning: ... degraded ...` notice to stderr — this is a fallback, not a failure, but it drops body-only-term matches, so watch for the warning on unattended runs.
+
 We recommend that you proactively learn and master the search syntax of these databases, as our built‑in search module functions similarly to the search bar on official web portals.
 
 For instance, here is a typical complex query example tailored for PubMed:
@@ -1450,6 +1453,50 @@ In theory, all DOI‑driven literature workflows can be standardised following t
 
 > Modules dedicated to the aforementioned preprint platforms are still under development and refinement. Preprint‑related subcommands are provided for testing purposes only. For detailed test cases, refer to [Cases](./docs/Cases.md)
 
+#### Preprint full-text fetch (Python API)
+
+For preprints without an open-access PDF, each fetcher exposes a `fetch_full_text()` method that returns clean section-headed text without any PDF parsing:
+
+```python
+from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher
+from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher
+from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
+
+# arXiv → ar5iv rendered HTML (LaTeX → HTML)
+arxiv = ArxivFetcher(root_dir="./papers")
+text = arxiv.fetch_full_text("1706.03762")            # "" on failure
+
+# bioRxiv / medRxiv → native full-text HTML first, then Europe PMC fullTextXML
+biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
+text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
+
+# Any DOI-indexed preprint → Europe PMC fullTextXML directly
+epmc = EuropePMCFullText()
+xml = epmc.full_text_xml("10.1101/2023.06.22.546069")
+epmc.close()
+```
+
+All three return an empty string `""` on failure, so callers can gracefully fall back to the abstract. The returned text is section-headed (`## Section`) plain text ready for LLM input.
+
+**bioRxiv / medRxiv full-text route & rate-limit handling**
+
+`BioRxivFetcher.fetch_full_text(doi)` resolves full text in this order:
+
+1. **Native full-text HTML** (`{landing_base}/{doi}.full-text`) — the preprint's own rendered page, parsed into `## Section` text (stops at References, skips figure/table captions). Requires a browser `User-Agent`.
+2. **Europe PMC `fullTextXML`** — only present once the preprint has been published into PMC; preprints still at the PPR (preprint) stage return 404 here.
+3. Otherwise `""` (caller falls back to the abstract).
+
+Because bioRxiv/medRxiv sit behind a Cloudflare wall that returns `429` on rapid successive requests, the HTML route applies several safeguards so a batch fetch does not get silently rate-limited into `""`:
+
+- **Browser User-Agent** — non-browser UAs are always 429 on `.full-text`.
+- **Connection reuse + `trust_env=False`** — fewer handshakes, avoids local-proxy timeouts.
+- **Inter-request throttle** (2 s, shared across instances) — requests are never fired back-to-back.
+- **Status-code branching** — `404` returns immediately (genuinely no full text); only `429/403/5xx` retry.
+- **Exponential backoff** (3/6/12 s, capped at 20 s, jittered) — retried up to `max_retries` times.
+- **Global cooldown** (30 s) — after any `429/403`, the whole batch pauses before the next request instead of hammering the wall.
+
+Typical cost: ~2–4 s per successful paper, ~30–90 s when a rate limit is hit — a small fraction of any downstream LLM step, so it is not the pipeline bottleneck.
+
 ### 6. Critical Reading and Knowledge Graph Analysis: Downstream End‑Use
 
 Upon completing literature retrieval, parsing, and structured processing as outlined above, users obtain chapter‑organised Markdown files and structured JSON files, which serve as the fundamental inputs for subsequent critical reading and knowledge graph analysis.
@@ -1483,6 +1530,8 @@ You may directly run the test scripts to verify the correctness and completeness
 
 > - [ ] Supplement query syntax for various literature databases and implement skill‑based support. Currently only partial MeSH‑aware syntax priors for PubMed are integrated.
 > - [ ] Maintain and update the BioPython library (E‑utilities API) for PubMed parsing from this stage onward. Current version: BioPython 1.87; see [biopython Repository](https://github.com/biopython/biopython) for details.
+> - [x] Europe PMC can return errors inside an HTTP 200 body (e.g. `{"errCode":404,...}` or a bare `{"version":"6.9"}` with no `resultList`); `EuropePMCSearch` now raises on `errCode` / a missing `resultList` so `last_search_degraded` is set instead of silently returning an empty result set.
+> - [x] `SourcePaper.version` is populated for arXiv (from the `vN` suffix) and was hardcoded to `""` for bioRxiv/medRxiv/chemRxiv. Now derived from the DOI version suffix (e.g. `.../v2`) via `extract_version_from_doi`, so version dedup (# ③ above) works uniformly across sources.
 
 </details>
 
@@ -1524,5 +1573,12 @@ You may directly run the test scripts to verify the correctness and completeness
 
 </details>
 
+<details>
+<summary><b>7. Testing & CI</b></summary>
+
+> - [ ] Add a pytest suite for pyPaperFlow itself. Regression coverage for the retry/backoff + degradation changes currently lives only in the monitor repo's `tests/test_backfill.py`; `test/` here holds sample output data, and CI (`docs.yml`) builds docs only.
+
+</details>
+
 
 

{pypaperflow-0.6.0 → pypaperflow-0.6.2}/README.md

@@ -83,6 +83,7 @@ This tool is designed to `complement rather than replace` reference management s
 
 - **Automated Retrieval from Multiple Sources**: Automatically search and retrieve paper metadata and full-text records from `PubMed/Medline, arXiv, medRxiv, chemRxiv and bioRxiv`. The repository focuses primarily on biomedical research and computational interdisciplinary fields (`Biomedicine + Computational Biology`).
 - **Full-Text Access**: Enable automatic downloading of open-access full texts in XML/Text format from `PMC`. For preprints and other publications without accessible PMC full texts, alternative acquisition modules are integrated to fetch `original PDFs`, with `Sci-Hub` set as the fallback provider.
+- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` reads the native bioRxiv/medRxiv full-text HTML first (browser User-Agent + rate-limit handling), then falls back to Europe PMC fullTextXML for preprints already published into PMC.
 - **Structured Storage**:
   - **Metadata**: Preserved in well-structured detailed JSON files.
   - **Full Text**: Stored in multiple formats including parsed JSON and Markdown for versatile downstream usage — JSON for programmatic data analysis, and Markdown optimized for LLM comprehension and processing.
@@ -331,6 +332,8 @@ Our literature database primarily covers biomedical research and computational i
 >
 > **Difference vs. exhaustive full-corpus enumeration:** a relevance search is a heuristic over the deposited metadata. The "no-omission-by-construction" alternative is to list the *whole* platform corpus (`filter=prefix…` with no `query`, cursor-paging every record — ≈ 55k ChemRxiv / 436k openRxiv) and run the boolean AND locally, with no relevance engine in the loop; recall is then exactly "all records whose metadata fully matches the query" (a `--start/--end-date` window shrinks the pull). The cost is downloading the full corpus per search, and it still inherits the source-level boundaries above (deposit lag, metadata-only, version duplication). This tool's `search()` path is relevance-based today; the exhaustive mode is not currently exposed as a flag.
 
+**Retries & backoff (all preprint commands).** Every preprint fetcher retries failed HTTP requests with exponential backoff — the delay doubles per attempt (`1.5s → 3s → 6s → 12s → …`, capped at 30s) and honors the server's `Retry-After` header when present. The default budget is **3 retries (≈4.5s of backoff)**, tuned to fail fast for interactive use so you get a quick answer rather than a ~22s silent stall. Override it per command with the `--max-retries` flag (e.g. `biorxiv-search ... --max-retries 5`); unattended jobs (e.g. `monitor.py`) pass a larger value explicitly. For **bioRxiv / medRxiv**, when the Europe PMC full-text leg is unreachable (e.g. a transient upstream outage) the search degrades to Crossref-only metadata matching and prints a `Warning: ... degraded ...` notice to stderr — this is a fallback, not a failure, but it drops body-only-term matches, so watch for the warning on unattended runs.
+
 We recommend that you proactively learn and master the search syntax of these databases, as our built‑in search module functions similarly to the search bar on official web portals.
 
 For instance, here is a typical complex query example tailored for PubMed:
@@ -1421,6 +1424,50 @@ In theory, all DOI‑driven literature workflows can be standardised following t
 
 > Modules dedicated to the aforementioned preprint platforms are still under development and refinement. Preprint‑related subcommands are provided for testing purposes only. For detailed test cases, refer to [Cases](./docs/Cases.md)
 
+#### Preprint full-text fetch (Python API)
+
+For preprints without an open-access PDF, each fetcher exposes a `fetch_full_text()` method that returns clean section-headed text without any PDF parsing:
+
+```python
+from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher
+from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher
+from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
+
+# arXiv → ar5iv rendered HTML (LaTeX → HTML)
+arxiv = ArxivFetcher(root_dir="./papers")
+text = arxiv.fetch_full_text("1706.03762")            # "" on failure
+
+# bioRxiv / medRxiv → native full-text HTML first, then Europe PMC fullTextXML
+biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
+text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
+
+# Any DOI-indexed preprint → Europe PMC fullTextXML directly
+epmc = EuropePMCFullText()
+xml = epmc.full_text_xml("10.1101/2023.06.22.546069")
+epmc.close()
+```
+
+All three return an empty string `""` on failure, so callers can gracefully fall back to the abstract. The returned text is section-headed (`## Section`) plain text ready for LLM input.
+
+**bioRxiv / medRxiv full-text route & rate-limit handling**
+
+`BioRxivFetcher.fetch_full_text(doi)` resolves full text in this order:
+
+1. **Native full-text HTML** (`{landing_base}/{doi}.full-text`) — the preprint's own rendered page, parsed into `## Section` text (stops at References, skips figure/table captions). Requires a browser `User-Agent`.
+2. **Europe PMC `fullTextXML`** — only present once the preprint has been published into PMC; preprints still at the PPR (preprint) stage return 404 here.
+3. Otherwise `""` (caller falls back to the abstract).
+
+Because bioRxiv/medRxiv sit behind a Cloudflare wall that returns `429` on rapid successive requests, the HTML route applies several safeguards so a batch fetch does not get silently rate-limited into `""`:
+
+- **Browser User-Agent** — non-browser UAs are always 429 on `.full-text`.
+- **Connection reuse + `trust_env=False`** — fewer handshakes, avoids local-proxy timeouts.
+- **Inter-request throttle** (2 s, shared across instances) — requests are never fired back-to-back.
+- **Status-code branching** — `404` returns immediately (genuinely no full text); only `429/403/5xx` retry.
+- **Exponential backoff** (3/6/12 s, capped at 20 s, jittered) — retried up to `max_retries` times.
+- **Global cooldown** (30 s) — after any `429/403`, the whole batch pauses before the next request instead of hammering the wall.
+
+Typical cost: ~2–4 s per successful paper, ~30–90 s when a rate limit is hit — a small fraction of any downstream LLM step, so it is not the pipeline bottleneck.
+
 ### 6. Critical Reading and Knowledge Graph Analysis: Downstream End‑Use
 
 Upon completing literature retrieval, parsing, and structured processing as outlined above, users obtain chapter‑organised Markdown files and structured JSON files, which serve as the fundamental inputs for subsequent critical reading and knowledge graph analysis.
@@ -1454,6 +1501,8 @@ You may directly run the test scripts to verify the correctness and completeness
 
 > - [ ] Supplement query syntax for various literature databases and implement skill‑based support. Currently only partial MeSH‑aware syntax priors for PubMed are integrated.
 > - [ ] Maintain and update the BioPython library (E‑utilities API) for PubMed parsing from this stage onward. Current version: BioPython 1.87; see [biopython Repository](https://github.com/biopython/biopython) for details.
+> - [x] Europe PMC can return errors inside an HTTP 200 body (e.g. `{"errCode":404,...}` or a bare `{"version":"6.9"}` with no `resultList`); `EuropePMCSearch` now raises on `errCode` / a missing `resultList` so `last_search_degraded` is set instead of silently returning an empty result set.
+> - [x] `SourcePaper.version` is populated for arXiv (from the `vN` suffix) and was hardcoded to `""` for bioRxiv/medRxiv/chemRxiv. Now derived from the DOI version suffix (e.g. `.../v2`) via `extract_version_from_doi`, so version dedup (# ③ above) works uniformly across sources.
 
 </details>
 
@@ -1495,5 +1544,12 @@ You may directly run the test scripts to verify the correctness and completeness
 
 </details>
 
+<details>
+<summary><b>7. Testing & CI</b></summary>
+
+> - [ ] Add a pytest suite for pyPaperFlow itself. Regression coverage for the retry/backoff + degradation changes currently lives only in the monitor repo's `tests/test_backfill.py`; `test/` here holds sample output data, and CI (`docs.yml`) builds docs only.
+
+</details>
+
 
 

{pypaperflow-0.6.0 → pypaperflow-0.6.2}/README_zh.md

@@ -82,6 +82,7 @@
 
 - **多来源自动检索**：自动从 `PubMed/Medline`、`arXiv`、`medRxiv`、`chemRxiv` 和 `bioRxiv` 搜索并获取论文元数据与全文记录。项目主要聚焦于生物医学与计算交叉领域（`Biomedicine + Computational Biology`）。
 - **全文获取**：支持自动从 `PMC` 下载开放获取的 XML/Text 全文。对于预印本及其他没有 PMC 全文的文献，集成了额外的获取模块以下载 `原始 PDF`，并将 `Sci-Hub` 作为兜底来源。
+- **预印本全文获取（免 PDF 解析）**：对于没有开放获取 PDF 的预印本，提供专用方法直接返回带章节标题的纯文本——`ArxivFetcher.fetch_full_text(arxiv_id)` 读取 ar5iv 渲染 HTML（arXiv LaTeX→HTML），`BioRxivFetcher.fetch_full_text(doi)` 优先读取 bioRxiv / medRxiv 原生全文 HTML（浏览器 User-Agent + 限流处理），再回退到 Europe PMC fullTextXML（已正式收录进 PMC 的预印本）；`EuropePMCFullText.full_text_xml(doi)` 从 Europe PMC 读取 JATS 全文 XML。
 - **结构化存储**：
     - **元数据**：保存为结构清晰的详细 JSON 文件。
     - **全文**：保存为多种格式，包括解析后的 JSON 和 Markdown，方便下游使用。其中 JSON 适合程序化分析，Markdown 更适合 LLM 理解与处理。
@@ -335,6 +336,8 @@ ChemRxiv 相关模块：
 >
 > 全库拉取对于轻量级的文献调研并不适用，除非你有明确的理由需要获取某一特定数据库的全部文献，而且每年每月更新的文献本身就具有一定的冗余性，所以从效率+数量上考虑，单纯相关性检索应该能够满足绝大多数科研工作者的文献调研需求(`因为真正重要的内容一定会反复出现，往往不需要担心全量遗漏`)。当然，对于全量拉取，可以参考其他开源工具如 [paperscraper](https://github.com/jannisborn/paperscraper) 等的实现。
 
+**重试与退避（所有预印本命令）。** 每个预印本 fetcher 在 HTTP 请求失败时都会按指数退避重试——延迟逐次翻倍（`1.5s → 3s → 6s → 12s → …`，上限 30s），并在服务器返回 `Retry-After` 头时优先遵循该头。默认预算是 **3 次重试（约 4.5s 退避）**，为交互式使用而调成"快速失败"，让你尽快得到结果，而不是静默卡住约 22s。可用每条命令的 `--max-retries` 参数覆盖（例如 `biorxiv-search ... --max-retries 5`）；无人值守任务（如 `monitor.py`）会显式传入更大的值。对 **bioRxiv / medRxiv** 而言，当 Europe PMC 全文支路不可达时（如上游临时故障），检索会降级为 Crossref 纯元数据匹配，并向 stderr 打印 `Warning: ... degraded ...` 提示——这是降级而非失败，但会丢失仅出现在正文中的词项命中，因此无人值守运行时务必留意该警告。
+
 建议用户提前学习并熟练掌握上述数据库的检索语法，本工具内置搜索模块的运行逻辑与数据库网页端搜索框基本一致。
 
 > ✨ 这里我们为你提供了几个特定文献数据库构建搜索query的skill，[paper query skill](./docs/Skills.md)
@@ -1440,6 +1443,50 @@ mineru_config.yaml                mineru_export_config.yaml
 
 > ⚠️ `针对上述预印本平台的模块目前基本已经开发完毕，后续只对相关功能进行维护和优化`, 测试细节与pubmed合并，详情见[Cases](./docs/Cases.md)
 
+#### 预印本全文获取（Python API）
+
+对于没有开放获取 PDF 的预印本，各 fetcher 提供 `fetch_full_text()` 方法，免 PDF 解析直接返回带章节标题的纯文本：
+
+```python
+from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher
+from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher
+from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
+
+# arXiv → ar5iv 渲染 HTML（LaTeX → HTML）
+arxiv = ArxivFetcher(root_dir="./papers")
+text = arxiv.fetch_full_text("1706.03762")            # 失败时返回 ""
+
+# bioRxiv / medRxiv → 原生全文 HTML 优先，Europe PMC fullTextXML 兜底
+biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
+text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
+
+# 任意 DOI 收录预印本 → 直接取 Europe PMC fullTextXML
+epmc = EuropePMCFullText()
+xml = epmc.full_text_xml("10.1101/2023.06.22.546069")
+epmc.close()
+```
+
+三者失败时均返回空字符串 `""`，调用方可优雅回退到摘要。返回文本为带章节标题（`## Section`）的纯文本，可直接作为 LLM 输入。
+
+**bioRxiv / medRxiv 全文获取路径与限流处理**
+
+`BioRxivFetcher.fetch_full_text(doi)` 按以下顺序解析全文：
+
+1. **原生全文 HTML**（`{landing_base}/{doi}.full-text`）——预印本自身的渲染页面，解析为 `## Section` 纯文本（遇 References 停止，跳过图/表标题）。需要浏览器 `User-Agent`。
+2. **Europe PMC `fullTextXML`**——仅在预印本已正式收录进 PMC 后才存在；仍处于 PPR（预印本）阶段的记录在此返回 404。
+3. 否则返回 `""`（调用方回退到摘要）。
+
+由于 bioRxiv/medRxiv 位于 Cloudflare 防火墙之后，连续快速请求会触发 `429`，HTML 路径应用了多重保护，避免批量抓取被静默限流成 `""`：
+
+- **浏览器 User-Agent**——非浏览器 UA 在 `.full-text` 上恒为 429。
+- **连接复用 + `trust_env=False`**——减少握手、避免本地代理超时。
+- **请求间隔节流**（2 s，跨实例共享）——请求绝不背靠背连发。
+- **状态码分支**——`404` 立即返回（确实无全文）；仅 `429/403/5xx` 才重试。
+- **指数退避**（3/6/12 s，封顶 20 s，加抖动）——最多重试 `max_retries` 次。
+- **全局冷却**（30 s）——任一次 `429/403` 后，整批在下一次请求前暂停，而非继续撞击防火墙。
+
+典型耗时：成功约 2–4 s/篇，触发限流约 30–90 s——相对任何下游 LLM 步骤都只是零头，不构成流水线瓶颈。
+
 
 #### 1. 命令速查 (TL;DR)
 
@@ -1544,6 +1591,8 @@ Europe PMC 走的是预印本全文,能补上 Crossref 只看标题摘要而漏
 5. **ChemRxiv 检索走 Crossref,不用官方 API**:ChemRxiv 的公开 API(`chemrxiv.org/engage/chemrxiv/public-api/v1`)对非浏览器客户端(httpx/curl)返回 Cloudflare 403,而 Crossref 侧(prefix `10.26434`)是稳定、最全的元数据通道,故 `chemrxiv-*` 只查 Crossref(也不并入 Europe PMC)。⚠️ 代价见下(版本重复 / 新贴有入库延迟 / 只看标题摘要)。完整讨论见 README「注意点:为什么预印本检索走 Crossref 元数据」。
 6. **ChemRxiv PDF 直连可下**:PDF 端点固定为 `https://chemrxiv.org/doi/pdf/{doi}`,本网络实测经 httpx 直连即返回 `%PDF` 字节,**不需要** CloakBrowser / undetected_chromedriver 回退(与 bioRxiv/medRxiv 的 Cloudflare 403 相反)。万一某篇直连失败,`--download-pdf` 仍会自动走浏览器回退链。
 7. **版本重复(去重要手动)**:Crossref 把 ChemRxiv 每次改版都单独注册成一个 DOI work——`10.26434/chemrxiv-2025-tj4pr-v2` 与 `chemrxiv-2025-tj4pr`、`10.26434/chemrxiv.15007500/v2` 与 `/v1` 都会作为独立结果同时命中(见下方实测,3 条 DOI 实为 2 篇论文)。`chemrxiv-*` 不去重,用 `--file` 清单抓取前可自行剔除旧版 DOI。
+8. **重试与退避**:所有预印本命令(`arxiv-*` / `biorxiv-*` / `medrxiv-*` / `chemrxiv-*`)的 HTTP 请求失败都会按指数退避重试(延迟逐次翻倍 `1.5s→3s→6s→12s→…`,封顶 30s,并优先遵循 `Retry-After` 头)。默认 **3 次重试(约 4.5s)**,面向交互式使用快速失败;可用 `--max-retries` 覆盖(如 `biorxiv-search ... --max-retries 5`),无人值守任务(如 `monitor.py`)会显式传更大值。
+9. **bioRxiv/medRxiv 的 Europe PMC 降级**:当 Europe PMC 全文支路不可达(如上游临时 503)时,`biorxiv-*` / `medrxiv-*` 的搜索会自动降级为纯 Crossref 元数据匹配,并向 stderr 打印 `Warning: ... degraded ...`——这是降级而非失败,但会丢失仅出现在正文中的词项命中(如基因缩写),无人值守运行时务必留意该警告。
 
 #### 2. 搜索并获取 arXiv 论文
 如果你只想先拿到 ID，可以先搜索；如果想同时获取元数据和 PDF，可以直接 fetch。
@@ -2302,6 +2351,8 @@ paperflow pubmed-export-md -i IDR_all_20260520_2026-05-20_18-33-54.json -o ./IDR
 
 > - [ ] 各文献数据库Query搜索语法的补充，尝试skill化，目前仅实现pubmed mesh部分语法先验结合
 > - [ ] 从这一步开始，关于pubmed数据库解析部分，考虑BioPython库的更新与维护(E-utility的接口)。目前biopython version 1.87，详情参考[biopython仓库](https://github.com/biopython/biopython)
+> - [x] Europe PMC 可能在 HTTP 200 响应体内返回错误（如 `{"errCode":404,...}` 或缺少 `resultList` 的裸 `{"version":"6.9"}`）；`EuropePMCSearch` 现对 `errCode` / 缺失 `resultList` 抛异常，从而设置 `last_search_degraded`，不再静默返回空结果集。
+> - [x] `SourcePaper.version` 对 arXiv 有值（来自 `vN` 后缀），但此前对 bioRxiv/medRxiv/chemRxiv 硬编码为 `""`。现通过 `extract_version_from_doi` 从 DOI 版本后缀（如 `.../v2`）推导，使版本去重（上文 ③）在各源间一致生效。
 
 </details>
 
@@ -2350,6 +2401,14 @@ paperflow pubmed-export-md -i IDR_all_20260520_2026-05-20_18-33-54.json -o ./IDR
 </details>
 
 
+<details markdown="1">
+<summary><b>7. 测试与 CI</b></summary>
+
+> - [ ] 为 pyPaperFlow 本身补充 pytest 测试套件。目前退避/降级改动的回归覆盖只存在于 monitor 仓库的 `tests/test_backfill.py`；本仓库 `test/` 只有样例输出数据，CI（`docs.yml`）仅构建文档。
+
+</details>
+
+
 
 
  

pypaperflow-0.6.2/src/pyPaperFlow/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.6.2"

{pypaperflow-0.6.0 → pypaperflow-0.6.2}/src/pyPaperFlow/cli.py

@@ -8,6 +8,7 @@ from .pubmed.pubmed_fetcher import PubmedFetcher
 from .preprint.arxiv_fetcher import ArxivFetcher
 from .preprint.biorxiv_fetcher import BioRxivFetcher
 from .preprint.chemrxiv_fetcher import ChemRxivFetcher
+from .preprint.source_utils import DEFAULT_MAX_RETRIES
 from .pubmed.pubmed_merger import PubmedMerger
 from .integrations import pdf_fetch
 from .integrations.github_export import run_github_export
@@ -20,6 +21,7 @@ opt_storage = typer.Option("./Papers", "--storage-dir", "-s", help="Directory in
 opt_email = typer.Option(..., "--email", help="Entrez Email.")
 opt_api_key = typer.Option(None, "--api-key", help="NCBI API Key (recommended).")
 opt_max_retries = typer.Option(3, "--max-retries", help="Maximum number of retries for Entrez API calls.")
+opt_preprint_max_retries = typer.Option(DEFAULT_MAX_RETRIES, "--max-retries", help="Maximum number of retries per request. Backoff is exponential, capped at 30s.")
 opt_batch_size = typer.Option(50, "--batch-size", "-b", help="Batch size for fetching.")
 opt_arxiv_backend = typer.Option("native", "--backend", help="arXiv backend: 'native' or 'paperscraper'.")
 
@@ -338,9 +340,10 @@ def arxiv_search_cmd(
     start_date: Optional[str] = typer.Option(None, "--start-date", help="Optional start date in YYYY-MM-DD."),
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
     backend: str = opt_arxiv_backend,
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search arXiv and write matching IDs to a text file."""
-    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend)
+    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend, max_retries=max_retries)
     records = fetcher.search(query=query, max_results=max_results, start_date=start_date, end_date=end_date)
     typer.echo(f"Found {len(records)} arXiv papers.")
     if not records:
@@ -374,12 +377,13 @@ def arxiv_fetch_cmd(
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD (query mode only)."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
     backend: str = opt_arxiv_backend,
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch arXiv metadata and attempt to download PDFs.
 
     Provide one of: a positional query, --file, or one or more --id values.
     """
-    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend)
+    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--id", arxiv_id)) if value]
@@ -426,6 +430,7 @@ def biorxiv_search_cmd(
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="Also search Europe PMC full text (boolean AND) and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search bioRxiv and write matching IDs to a text file.
 
@@ -438,7 +443,7 @@ def biorxiv_search_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days, max_retries=max_retries)
     records = fetcher.search(query=query, start_date=start_date, end_date=end_date, max_results=max_results, use_europepmc=use_europepmc)
     typer.echo(f"Found {len(records)} bioRxiv papers.")
     for record in records:
@@ -462,6 +467,7 @@ def biorxiv_fetch_cmd(
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="In query mode, also search Europe PMC full text and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch bioRxiv metadata and attempt to download PDFs.
 
@@ -475,7 +481,7 @@ def biorxiv_fetch_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--doi", doi)) if value]
@@ -523,6 +529,7 @@ def medrxiv_search_cmd(
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="Also search Europe PMC full text (boolean AND) and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search medRxiv and write matching IDs to a text file.
 
@@ -535,7 +542,7 @@ def medrxiv_search_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days, max_retries=max_retries)
     records = fetcher.search(query=query, start_date=start_date, end_date=end_date, max_results=max_results, use_europepmc=use_europepmc)
     typer.echo(f"Found {len(records)} medRxiv papers.")
     for record in records:
@@ -559,6 +566,7 @@ def medrxiv_fetch_cmd(
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="In query mode, also search Europe PMC full text and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch medRxiv metadata and attempt to download PDFs.
 
@@ -572,7 +580,7 @@ def medrxiv_fetch_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--doi", doi)) if value]
@@ -618,13 +626,14 @@ def chemrxiv_search_cmd(
     output_dir: Optional[str] = typer.Option(None, "--output-dir", "-o", help="Directory to save searched ChemRxiv IDs."),
     start_date: Optional[str] = typer.Option(None, "--start-date", help="Optional start date in YYYY-MM-DD."),
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search ChemRxiv and write matching DOIs to a text file.
 
     Metadata is retrieved from Crossref over ChemRxiv records (prefix
     10.26434, publisher American Chemical Society (ACS)).
     """
-    fetcher = ChemRxivFetcher(root_dir=storage_dir)
+    fetcher = ChemRxivFetcher(root_dir=storage_dir, max_retries=max_retries)
     records = fetcher.search(query=query, start_date=start_date, end_date=end_date, max_results=max_results)
     typer.echo(f"Found {len(records)} ChemRxiv papers.")
     for record in records:
@@ -646,13 +655,14 @@ def chemrxiv_fetch_cmd(
     start_date: Optional[str] = typer.Option(None, "--start-date", help="Optional start date in YYYY-MM-DD (query mode only)."),
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD (query mode only)."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch ChemRxiv metadata and attempt to download PDFs.
 
     Metadata retrieval uses Crossref over ChemRxiv records (prefix 10.26434).
     Provide one of: a positional query, --file, or one or more --doi values.
     """
-    fetcher = ChemRxivFetcher(root_dir=storage_dir)
+    fetcher = ChemRxivFetcher(root_dir=storage_dir, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--doi", doi)) if value]

{pypaperflow-0.6.0 → pypaperflow-0.6.2}/src/pyPaperFlow/preprint/arxiv_fetcher.py

@@ -5,7 +5,6 @@ import importlib
 import time
 import xml.etree.ElementTree as ET
 from datetime import date, datetime, timezone
-from email.utils import parsedate_to_datetime
 from pathlib import Path
 from typing import Any, Dict, Iterable, List, Optional
 
@@ -14,12 +13,14 @@ from bs4 import BeautifulSoup
 
 from .source_models import SourcePaper
 from .source_utils import (
+    DEFAULT_MAX_RETRIES,
     build_source_record_dir,
     download_binary,
     extract_year,
     normalize_text,
     safe_filename,
     save_json,
+    sleep_before_retry,
 )
 
 
@@ -73,7 +74,7 @@ class ArxivFetcher:
         root_dir: str,
         backend: str = "native",
         batch_size: int = 100,
-        max_retries: int = 3,
+        max_retries: int = DEFAULT_MAX_RETRIES,
         request_timeout: float = 60.0,
     ):
         self.root_dir = root_dir
@@ -382,7 +383,7 @@ class ArxivFetcher:
                 if response.status_code == 429:
                     last_error = RuntimeError(f"arXiv API rate limited request for {description}")
                     if attempt + 1 < self.max_retries:
-                        self._sleep_before_retry(response, attempt)
+                        sleep_before_retry(response, attempt)
                         continue
                     break
                 response.raise_for_status()
@@ -391,19 +392,19 @@ class ArxivFetcher:
                 except ET.ParseError as exc:
                     last_error = exc
                     if attempt + 1 < self.max_retries:
-                        self._sleep_before_retry(response, attempt)
+                        sleep_before_retry(response, attempt)
                         continue
                     break
             except (httpx.HTTPStatusError, httpx.TimeoutException, httpx.TransportError) as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    self._sleep_before_retry(response, attempt)
+                    sleep_before_retry(response, attempt)
                     continue
                 break
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    self._sleep_before_retry(response, attempt)
+                    sleep_before_retry(response, attempt)
                     continue
                 break
 
@@ -426,32 +427,6 @@ class ArxivFetcher:
             self._http_client = httpx.Client(http2=False, **client_kwargs)
         return self._http_client
 
-    def _sleep_before_retry(self, response: Optional[httpx.Response], attempt: int) -> None:
-        retry_after = self._retry_after_seconds(response)
-        delay = retry_after if retry_after is not None else min(30.0, 1.5 * (2**attempt))
-        time.sleep(max(0.0, delay))
-
-    def _retry_after_seconds(self, response: Optional[httpx.Response]) -> Optional[float]:
-        if response is None:
-            return None
-
-        raw_retry_after = normalize_text(response.headers.get("Retry-After", ""))
-        if not raw_retry_after:
-            return None
-
-        if raw_retry_after.isdigit():
-            return float(raw_retry_after)
-
-        try:
-            retry_after_dt = parsedate_to_datetime(raw_retry_after)
-        except (TypeError, ValueError, IndexError):
-            return None
-
-        if retry_after_dt.tzinfo is None:
-            retry_after_dt = retry_after_dt.replace(tzinfo=timezone.utc)
-        now = datetime.now(retry_after_dt.tzinfo)
-        return max(0.0, (retry_after_dt - now).total_seconds())
-
     def _normalize_date_bounds(
         self,
         start_date: Optional[str],