pyPaperFlow 0.6.1__tar.gz → 0.6.2__tar.gz

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyPaperFlow
-Version: 0.6.1
+Version: 0.6.2
 Summary: Automated paper fetching and analysis platform.
 Project-URL: Homepage, https://github.com/MaybeBio/pyPaperFlow
 Project-URL: Issues, https://github.com/MaybeBio/pyPaperFlow/issues
@@ -112,7 +112,7 @@ This tool is designed to `complement rather than replace` reference management s
 
 - **Automated Retrieval from Multiple Sources**: Automatically search and retrieve paper metadata and full-text records from `PubMed/Medline, arXiv, medRxiv, chemRxiv and bioRxiv`. The repository focuses primarily on biomedical research and computational interdisciplinary fields (`Biomedicine + Computational Biology`).
 - **Full-Text Access**: Enable automatic downloading of open-access full texts in XML/Text format from `PMC`. For preprints and other publications without accessible PMC full texts, alternative acquisition modules are integrated to fetch `original PDFs`, with `Sci-Hub` set as the fallback provider.
-- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` / `EuropePMCFullText.full_text_xml(doi)` read JATS full-text XML from Europe PMC (bioRxiv / medRxiv and other DOI-indexed preprints).
+- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` reads the native bioRxiv/medRxiv full-text HTML first (browser User-Agent + rate-limit handling), then falls back to Europe PMC fullTextXML for preprints already published into PMC.
 - **Structured Storage**:
   - **Metadata**: Preserved in well-structured detailed JSON files.
   - **Full Text**: Stored in multiple formats including parsed JSON and Markdown for versatile downstream usage — JSON for programmatic data analysis, and Markdown optimized for LLM comprehension and processing.
@@ -361,6 +361,8 @@ Our literature database primarily covers biomedical research and computational i
 >
 > **Difference vs. exhaustive full-corpus enumeration:** a relevance search is a heuristic over the deposited metadata. The "no-omission-by-construction" alternative is to list the *whole* platform corpus (`filter=prefix…` with no `query`, cursor-paging every record — ≈ 55k ChemRxiv / 436k openRxiv) and run the boolean AND locally, with no relevance engine in the loop; recall is then exactly "all records whose metadata fully matches the query" (a `--start/--end-date` window shrinks the pull). The cost is downloading the full corpus per search, and it still inherits the source-level boundaries above (deposit lag, metadata-only, version duplication). This tool's `search()` path is relevance-based today; the exhaustive mode is not currently exposed as a flag.
 
+**Retries & backoff (all preprint commands).** Every preprint fetcher retries failed HTTP requests with exponential backoff — the delay doubles per attempt (`1.5s → 3s → 6s → 12s → …`, capped at 30s) and honors the server's `Retry-After` header when present. The default budget is **3 retries (≈4.5s of backoff)**, tuned to fail fast for interactive use so you get a quick answer rather than a ~22s silent stall. Override it per command with the `--max-retries` flag (e.g. `biorxiv-search ... --max-retries 5`); unattended jobs (e.g. `monitor.py`) pass a larger value explicitly. For **bioRxiv / medRxiv**, when the Europe PMC full-text leg is unreachable (e.g. a transient upstream outage) the search degrades to Crossref-only metadata matching and prints a `Warning: ... degraded ...` notice to stderr — this is a fallback, not a failure, but it drops body-only-term matches, so watch for the warning on unattended runs.
+
 We recommend that you proactively learn and master the search syntax of these databases, as our built‑in search module functions similarly to the search bar on official web portals.
 
 For instance, here is a typical complex query example tailored for PubMed:
@@ -1464,7 +1466,7 @@ from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
 arxiv = ArxivFetcher(root_dir="./papers")
 text = arxiv.fetch_full_text("1706.03762")            # "" on failure
 
-# bioRxiv / medRxiv → Europe PMC fullTextXML
+# bioRxiv / medRxiv → native full-text HTML first, then Europe PMC fullTextXML
 biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
 text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
 
@@ -1476,6 +1478,25 @@ epmc.close()
 
 All three return an empty string `""` on failure, so callers can gracefully fall back to the abstract. The returned text is section-headed (`## Section`) plain text ready for LLM input.
 
+**bioRxiv / medRxiv full-text route & rate-limit handling**
+
+`BioRxivFetcher.fetch_full_text(doi)` resolves full text in this order:
+
+1. **Native full-text HTML** (`{landing_base}/{doi}.full-text`) — the preprint's own rendered page, parsed into `## Section` text (stops at References, skips figure/table captions). Requires a browser `User-Agent`.
+2. **Europe PMC `fullTextXML`** — only present once the preprint has been published into PMC; preprints still at the PPR (preprint) stage return 404 here.
+3. Otherwise `""` (caller falls back to the abstract).
+
+Because bioRxiv/medRxiv sit behind a Cloudflare wall that returns `429` on rapid successive requests, the HTML route applies several safeguards so a batch fetch does not get silently rate-limited into `""`:
+
+- **Browser User-Agent** — non-browser UAs are always 429 on `.full-text`.
+- **Connection reuse + `trust_env=False`** — fewer handshakes, avoids local-proxy timeouts.
+- **Inter-request throttle** (2 s, shared across instances) — requests are never fired back-to-back.
+- **Status-code branching** — `404` returns immediately (genuinely no full text); only `429/403/5xx` retry.
+- **Exponential backoff** (3/6/12 s, capped at 20 s, jittered) — retried up to `max_retries` times.
+- **Global cooldown** (30 s) — after any `429/403`, the whole batch pauses before the next request instead of hammering the wall.
+
+Typical cost: ~2–4 s per successful paper, ~30–90 s when a rate limit is hit — a small fraction of any downstream LLM step, so it is not the pipeline bottleneck.
+
 ### 6. Critical Reading and Knowledge Graph Analysis: Downstream End‑Use
 
 Upon completing literature retrieval, parsing, and structured processing as outlined above, users obtain chapter‑organised Markdown files and structured JSON files, which serve as the fundamental inputs for subsequent critical reading and knowledge graph analysis.
@@ -1509,6 +1530,8 @@ You may directly run the test scripts to verify the correctness and completeness
 
 > - [ ] Supplement query syntax for various literature databases and implement skill‑based support. Currently only partial MeSH‑aware syntax priors for PubMed are integrated.
 > - [ ] Maintain and update the BioPython library (E‑utilities API) for PubMed parsing from this stage onward. Current version: BioPython 1.87; see [biopython Repository](https://github.com/biopython/biopython) for details.
+> - [x] Europe PMC can return errors inside an HTTP 200 body (e.g. `{"errCode":404,...}` or a bare `{"version":"6.9"}` with no `resultList`); `EuropePMCSearch` now raises on `errCode` / a missing `resultList` so `last_search_degraded` is set instead of silently returning an empty result set.
+> - [x] `SourcePaper.version` is populated for arXiv (from the `vN` suffix) and was hardcoded to `""` for bioRxiv/medRxiv/chemRxiv. Now derived from the DOI version suffix (e.g. `.../v2`) via `extract_version_from_doi`, so version dedup (# ③ above) works uniformly across sources.
 
 </details>
 
@@ -1550,5 +1573,12 @@ You may directly run the test scripts to verify the correctness and completeness
 
 </details>
 
+<details>
+<summary><b>7. Testing & CI</b></summary>
+
+> - [ ] Add a pytest suite for pyPaperFlow itself. Regression coverage for the retry/backoff + degradation changes currently lives only in the monitor repo's `tests/test_backfill.py`; `test/` here holds sample output data, and CI (`docs.yml`) builds docs only.
+
+</details>
+
 
 

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/README.md

@@ -83,7 +83,7 @@ This tool is designed to `complement rather than replace` reference management s
 
 - **Automated Retrieval from Multiple Sources**: Automatically search and retrieve paper metadata and full-text records from `PubMed/Medline, arXiv, medRxiv, chemRxiv and bioRxiv`. The repository focuses primarily on biomedical research and computational interdisciplinary fields (`Biomedicine + Computational Biology`).
 - **Full-Text Access**: Enable automatic downloading of open-access full texts in XML/Text format from `PMC`. For preprints and other publications without accessible PMC full texts, alternative acquisition modules are integrated to fetch `original PDFs`, with `Sci-Hub` set as the fallback provider.
-- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` / `EuropePMCFullText.full_text_xml(doi)` read JATS full-text XML from Europe PMC (bioRxiv / medRxiv and other DOI-indexed preprints).
+- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` reads the native bioRxiv/medRxiv full-text HTML first (browser User-Agent + rate-limit handling), then falls back to Europe PMC fullTextXML for preprints already published into PMC.
 - **Structured Storage**:
   - **Metadata**: Preserved in well-structured detailed JSON files.
   - **Full Text**: Stored in multiple formats including parsed JSON and Markdown for versatile downstream usage — JSON for programmatic data analysis, and Markdown optimized for LLM comprehension and processing.
@@ -332,6 +332,8 @@ Our literature database primarily covers biomedical research and computational i
 >
 > **Difference vs. exhaustive full-corpus enumeration:** a relevance search is a heuristic over the deposited metadata. The "no-omission-by-construction" alternative is to list the *whole* platform corpus (`filter=prefix…` with no `query`, cursor-paging every record — ≈ 55k ChemRxiv / 436k openRxiv) and run the boolean AND locally, with no relevance engine in the loop; recall is then exactly "all records whose metadata fully matches the query" (a `--start/--end-date` window shrinks the pull). The cost is downloading the full corpus per search, and it still inherits the source-level boundaries above (deposit lag, metadata-only, version duplication). This tool's `search()` path is relevance-based today; the exhaustive mode is not currently exposed as a flag.
 
+**Retries & backoff (all preprint commands).** Every preprint fetcher retries failed HTTP requests with exponential backoff — the delay doubles per attempt (`1.5s → 3s → 6s → 12s → …`, capped at 30s) and honors the server's `Retry-After` header when present. The default budget is **3 retries (≈4.5s of backoff)**, tuned to fail fast for interactive use so you get a quick answer rather than a ~22s silent stall. Override it per command with the `--max-retries` flag (e.g. `biorxiv-search ... --max-retries 5`); unattended jobs (e.g. `monitor.py`) pass a larger value explicitly. For **bioRxiv / medRxiv**, when the Europe PMC full-text leg is unreachable (e.g. a transient upstream outage) the search degrades to Crossref-only metadata matching and prints a `Warning: ... degraded ...` notice to stderr — this is a fallback, not a failure, but it drops body-only-term matches, so watch for the warning on unattended runs.
+
 We recommend that you proactively learn and master the search syntax of these databases, as our built‑in search module functions similarly to the search bar on official web portals.
 
 For instance, here is a typical complex query example tailored for PubMed:
@@ -1435,7 +1437,7 @@ from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
 arxiv = ArxivFetcher(root_dir="./papers")
 text = arxiv.fetch_full_text("1706.03762")            # "" on failure
 
-# bioRxiv / medRxiv → Europe PMC fullTextXML
+# bioRxiv / medRxiv → native full-text HTML first, then Europe PMC fullTextXML
 biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
 text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
 
@@ -1447,6 +1449,25 @@ epmc.close()
 
 All three return an empty string `""` on failure, so callers can gracefully fall back to the abstract. The returned text is section-headed (`## Section`) plain text ready for LLM input.
 
+**bioRxiv / medRxiv full-text route & rate-limit handling**
+
+`BioRxivFetcher.fetch_full_text(doi)` resolves full text in this order:
+
+1. **Native full-text HTML** (`{landing_base}/{doi}.full-text`) — the preprint's own rendered page, parsed into `## Section` text (stops at References, skips figure/table captions). Requires a browser `User-Agent`.
+2. **Europe PMC `fullTextXML`** — only present once the preprint has been published into PMC; preprints still at the PPR (preprint) stage return 404 here.
+3. Otherwise `""` (caller falls back to the abstract).
+
+Because bioRxiv/medRxiv sit behind a Cloudflare wall that returns `429` on rapid successive requests, the HTML route applies several safeguards so a batch fetch does not get silently rate-limited into `""`:
+
+- **Browser User-Agent** — non-browser UAs are always 429 on `.full-text`.
+- **Connection reuse + `trust_env=False`** — fewer handshakes, avoids local-proxy timeouts.
+- **Inter-request throttle** (2 s, shared across instances) — requests are never fired back-to-back.
+- **Status-code branching** — `404` returns immediately (genuinely no full text); only `429/403/5xx` retry.
+- **Exponential backoff** (3/6/12 s, capped at 20 s, jittered) — retried up to `max_retries` times.
+- **Global cooldown** (30 s) — after any `429/403`, the whole batch pauses before the next request instead of hammering the wall.
+
+Typical cost: ~2–4 s per successful paper, ~30–90 s when a rate limit is hit — a small fraction of any downstream LLM step, so it is not the pipeline bottleneck.
+
 ### 6. Critical Reading and Knowledge Graph Analysis: Downstream End‑Use
 
 Upon completing literature retrieval, parsing, and structured processing as outlined above, users obtain chapter‑organised Markdown files and structured JSON files, which serve as the fundamental inputs for subsequent critical reading and knowledge graph analysis.
@@ -1480,6 +1501,8 @@ You may directly run the test scripts to verify the correctness and completeness
 
 > - [ ] Supplement query syntax for various literature databases and implement skill‑based support. Currently only partial MeSH‑aware syntax priors for PubMed are integrated.
 > - [ ] Maintain and update the BioPython library (E‑utilities API) for PubMed parsing from this stage onward. Current version: BioPython 1.87; see [biopython Repository](https://github.com/biopython/biopython) for details.
+> - [x] Europe PMC can return errors inside an HTTP 200 body (e.g. `{"errCode":404,...}` or a bare `{"version":"6.9"}` with no `resultList`); `EuropePMCSearch` now raises on `errCode` / a missing `resultList` so `last_search_degraded` is set instead of silently returning an empty result set.
+> - [x] `SourcePaper.version` is populated for arXiv (from the `vN` suffix) and was hardcoded to `""` for bioRxiv/medRxiv/chemRxiv. Now derived from the DOI version suffix (e.g. `.../v2`) via `extract_version_from_doi`, so version dedup (# ③ above) works uniformly across sources.
 
 </details>
 
@@ -1521,5 +1544,12 @@ You may directly run the test scripts to verify the correctness and completeness
 
 </details>
 
+<details>
+<summary><b>7. Testing & CI</b></summary>
+
+> - [ ] Add a pytest suite for pyPaperFlow itself. Regression coverage for the retry/backoff + degradation changes currently lives only in the monitor repo's `tests/test_backfill.py`; `test/` here holds sample output data, and CI (`docs.yml`) builds docs only.
+
+</details>
+
 
 

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/README_zh.md

@@ -82,7 +82,7 @@
 
 - **多来源自动检索**：自动从 `PubMed/Medline`、`arXiv`、`medRxiv`、`chemRxiv` 和 `bioRxiv` 搜索并获取论文元数据与全文记录。项目主要聚焦于生物医学与计算交叉领域（`Biomedicine + Computational Biology`）。
 - **全文获取**：支持自动从 `PMC` 下载开放获取的 XML/Text 全文。对于预印本及其他没有 PMC 全文的文献，集成了额外的获取模块以下载 `原始 PDF`，并将 `Sci-Hub` 作为兜底来源。
-- **预印本全文获取（免 PDF 解析）**：对于没有开放获取 PDF 的预印本，提供专用方法直接返回带章节标题的纯文本——`ArxivFetcher.fetch_full_text(arxiv_id)` 读取 ar5iv 渲染 HTML（arXiv LaTeX→HTML），`BioRxivFetcher.fetch_full_text(doi)` / `EuropePMCFullText.full_text_xml(doi)` 从 Europe PMC 读取 JATS 全文 XML（bioRxiv / medRxiv 及其它 DOI 收录预印本）。
+- **预印本全文获取（免 PDF 解析）**：对于没有开放获取 PDF 的预印本，提供专用方法直接返回带章节标题的纯文本——`ArxivFetcher.fetch_full_text(arxiv_id)` 读取 ar5iv 渲染 HTML（arXiv LaTeX→HTML），`BioRxivFetcher.fetch_full_text(doi)` 优先读取 bioRxiv / medRxiv 原生全文 HTML（浏览器 User-Agent + 限流处理），再回退到 Europe PMC fullTextXML（已正式收录进 PMC 的预印本）；`EuropePMCFullText.full_text_xml(doi)` 从 Europe PMC 读取 JATS 全文 XML。
 - **结构化存储**：
     - **元数据**：保存为结构清晰的详细 JSON 文件。
     - **全文**：保存为多种格式，包括解析后的 JSON 和 Markdown，方便下游使用。其中 JSON 适合程序化分析，Markdown 更适合 LLM 理解与处理。
@@ -336,6 +336,8 @@ ChemRxiv 相关模块：
 >
 > 全库拉取对于轻量级的文献调研并不适用，除非你有明确的理由需要获取某一特定数据库的全部文献，而且每年每月更新的文献本身就具有一定的冗余性，所以从效率+数量上考虑，单纯相关性检索应该能够满足绝大多数科研工作者的文献调研需求(`因为真正重要的内容一定会反复出现，往往不需要担心全量遗漏`)。当然，对于全量拉取，可以参考其他开源工具如 [paperscraper](https://github.com/jannisborn/paperscraper) 等的实现。
 
+**重试与退避（所有预印本命令）。** 每个预印本 fetcher 在 HTTP 请求失败时都会按指数退避重试——延迟逐次翻倍（`1.5s → 3s → 6s → 12s → …`，上限 30s），并在服务器返回 `Retry-After` 头时优先遵循该头。默认预算是 **3 次重试（约 4.5s 退避）**，为交互式使用而调成"快速失败"，让你尽快得到结果，而不是静默卡住约 22s。可用每条命令的 `--max-retries` 参数覆盖（例如 `biorxiv-search ... --max-retries 5`）；无人值守任务（如 `monitor.py`）会显式传入更大的值。对 **bioRxiv / medRxiv** 而言，当 Europe PMC 全文支路不可达时（如上游临时故障），检索会降级为 Crossref 纯元数据匹配，并向 stderr 打印 `Warning: ... degraded ...` 提示——这是降级而非失败，但会丢失仅出现在正文中的词项命中，因此无人值守运行时务必留意该警告。
+
 建议用户提前学习并熟练掌握上述数据库的检索语法，本工具内置搜索模块的运行逻辑与数据库网页端搜索框基本一致。
 
 > ✨ 这里我们为你提供了几个特定文献数据库构建搜索query的skill，[paper query skill](./docs/Skills.md)
@@ -1454,7 +1456,7 @@ from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
 arxiv = ArxivFetcher(root_dir="./papers")
 text = arxiv.fetch_full_text("1706.03762")            # 失败时返回 ""
 
-# bioRxiv / medRxiv → Europe PMC fullTextXML
+# bioRxiv / medRxiv → 原生全文 HTML 优先，Europe PMC fullTextXML 兜底
 biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
 text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
 
@@ -1466,6 +1468,25 @@ epmc.close()
 
 三者失败时均返回空字符串 `""`，调用方可优雅回退到摘要。返回文本为带章节标题（`## Section`）的纯文本，可直接作为 LLM 输入。
 
+**bioRxiv / medRxiv 全文获取路径与限流处理**
+
+`BioRxivFetcher.fetch_full_text(doi)` 按以下顺序解析全文：
+
+1. **原生全文 HTML**（`{landing_base}/{doi}.full-text`）——预印本自身的渲染页面，解析为 `## Section` 纯文本（遇 References 停止，跳过图/表标题）。需要浏览器 `User-Agent`。
+2. **Europe PMC `fullTextXML`**——仅在预印本已正式收录进 PMC 后才存在；仍处于 PPR（预印本）阶段的记录在此返回 404。
+3. 否则返回 `""`（调用方回退到摘要）。
+
+由于 bioRxiv/medRxiv 位于 Cloudflare 防火墙之后，连续快速请求会触发 `429`，HTML 路径应用了多重保护，避免批量抓取被静默限流成 `""`：
+
+- **浏览器 User-Agent**——非浏览器 UA 在 `.full-text` 上恒为 429。
+- **连接复用 + `trust_env=False`**——减少握手、避免本地代理超时。
+- **请求间隔节流**（2 s，跨实例共享）——请求绝不背靠背连发。
+- **状态码分支**——`404` 立即返回（确实无全文）；仅 `429/403/5xx` 才重试。
+- **指数退避**（3/6/12 s，封顶 20 s，加抖动）——最多重试 `max_retries` 次。
+- **全局冷却**（30 s）——任一次 `429/403` 后，整批在下一次请求前暂停，而非继续撞击防火墙。
+
+典型耗时：成功约 2–4 s/篇，触发限流约 30–90 s——相对任何下游 LLM 步骤都只是零头，不构成流水线瓶颈。
+
 
 #### 1. 命令速查 (TL;DR)
 
@@ -1570,6 +1591,8 @@ Europe PMC 走的是预印本全文,能补上 Crossref 只看标题摘要而漏
 5. **ChemRxiv 检索走 Crossref,不用官方 API**:ChemRxiv 的公开 API(`chemrxiv.org/engage/chemrxiv/public-api/v1`)对非浏览器客户端(httpx/curl)返回 Cloudflare 403,而 Crossref 侧(prefix `10.26434`)是稳定、最全的元数据通道,故 `chemrxiv-*` 只查 Crossref(也不并入 Europe PMC)。⚠️ 代价见下(版本重复 / 新贴有入库延迟 / 只看标题摘要)。完整讨论见 README「注意点:为什么预印本检索走 Crossref 元数据」。
 6. **ChemRxiv PDF 直连可下**:PDF 端点固定为 `https://chemrxiv.org/doi/pdf/{doi}`,本网络实测经 httpx 直连即返回 `%PDF` 字节,**不需要** CloakBrowser / undetected_chromedriver 回退(与 bioRxiv/medRxiv 的 Cloudflare 403 相反)。万一某篇直连失败,`--download-pdf` 仍会自动走浏览器回退链。
 7. **版本重复(去重要手动)**:Crossref 把 ChemRxiv 每次改版都单独注册成一个 DOI work——`10.26434/chemrxiv-2025-tj4pr-v2` 与 `chemrxiv-2025-tj4pr`、`10.26434/chemrxiv.15007500/v2` 与 `/v1` 都会作为独立结果同时命中(见下方实测,3 条 DOI 实为 2 篇论文)。`chemrxiv-*` 不去重,用 `--file` 清单抓取前可自行剔除旧版 DOI。
+8. **重试与退避**:所有预印本命令(`arxiv-*` / `biorxiv-*` / `medrxiv-*` / `chemrxiv-*`)的 HTTP 请求失败都会按指数退避重试(延迟逐次翻倍 `1.5s→3s→6s→12s→…`,封顶 30s,并优先遵循 `Retry-After` 头)。默认 **3 次重试(约 4.5s)**,面向交互式使用快速失败;可用 `--max-retries` 覆盖(如 `biorxiv-search ... --max-retries 5`),无人值守任务(如 `monitor.py`)会显式传更大值。
+9. **bioRxiv/medRxiv 的 Europe PMC 降级**:当 Europe PMC 全文支路不可达(如上游临时 503)时,`biorxiv-*` / `medrxiv-*` 的搜索会自动降级为纯 Crossref 元数据匹配,并向 stderr 打印 `Warning: ... degraded ...`——这是降级而非失败,但会丢失仅出现在正文中的词项命中(如基因缩写),无人值守运行时务必留意该警告。
 
 #### 2. 搜索并获取 arXiv 论文
 如果你只想先拿到 ID，可以先搜索；如果想同时获取元数据和 PDF，可以直接 fetch。
@@ -2328,6 +2351,8 @@ paperflow pubmed-export-md -i IDR_all_20260520_2026-05-20_18-33-54.json -o ./IDR
 
 > - [ ] 各文献数据库Query搜索语法的补充，尝试skill化，目前仅实现pubmed mesh部分语法先验结合
 > - [ ] 从这一步开始，关于pubmed数据库解析部分，考虑BioPython库的更新与维护(E-utility的接口)。目前biopython version 1.87，详情参考[biopython仓库](https://github.com/biopython/biopython)
+> - [x] Europe PMC 可能在 HTTP 200 响应体内返回错误（如 `{"errCode":404,...}` 或缺少 `resultList` 的裸 `{"version":"6.9"}`）；`EuropePMCSearch` 现对 `errCode` / 缺失 `resultList` 抛异常，从而设置 `last_search_degraded`，不再静默返回空结果集。
+> - [x] `SourcePaper.version` 对 arXiv 有值（来自 `vN` 后缀），但此前对 bioRxiv/medRxiv/chemRxiv 硬编码为 `""`。现通过 `extract_version_from_doi` 从 DOI 版本后缀（如 `.../v2`）推导，使版本去重（上文 ③）在各源间一致生效。
 
 </details>
 
@@ -2376,6 +2401,14 @@ paperflow pubmed-export-md -i IDR_all_20260520_2026-05-20_18-33-54.json -o ./IDR
 </details>
 
 
+<details markdown="1">
+<summary><b>7. 测试与 CI</b></summary>
+
+> - [ ] 为 pyPaperFlow 本身补充 pytest 测试套件。目前退避/降级改动的回归覆盖只存在于 monitor 仓库的 `tests/test_backfill.py`；本仓库 `test/` 只有样例输出数据，CI（`docs.yml`）仅构建文档。
+
+</details>
+
+
 
 
  

pypaperflow-0.6.2/src/pyPaperFlow/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.6.2"

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/src/pyPaperFlow/cli.py

@@ -8,6 +8,7 @@ from .pubmed.pubmed_fetcher import PubmedFetcher
 from .preprint.arxiv_fetcher import ArxivFetcher
 from .preprint.biorxiv_fetcher import BioRxivFetcher
 from .preprint.chemrxiv_fetcher import ChemRxivFetcher
+from .preprint.source_utils import DEFAULT_MAX_RETRIES
 from .pubmed.pubmed_merger import PubmedMerger
 from .integrations import pdf_fetch
 from .integrations.github_export import run_github_export
@@ -20,6 +21,7 @@ opt_storage = typer.Option("./Papers", "--storage-dir", "-s", help="Directory in
 opt_email = typer.Option(..., "--email", help="Entrez Email.")
 opt_api_key = typer.Option(None, "--api-key", help="NCBI API Key (recommended).")
 opt_max_retries = typer.Option(3, "--max-retries", help="Maximum number of retries for Entrez API calls.")
+opt_preprint_max_retries = typer.Option(DEFAULT_MAX_RETRIES, "--max-retries", help="Maximum number of retries per request. Backoff is exponential, capped at 30s.")
 opt_batch_size = typer.Option(50, "--batch-size", "-b", help="Batch size for fetching.")
 opt_arxiv_backend = typer.Option("native", "--backend", help="arXiv backend: 'native' or 'paperscraper'.")
 
@@ -338,9 +340,10 @@ def arxiv_search_cmd(
     start_date: Optional[str] = typer.Option(None, "--start-date", help="Optional start date in YYYY-MM-DD."),
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
     backend: str = opt_arxiv_backend,
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search arXiv and write matching IDs to a text file."""
-    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend)
+    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend, max_retries=max_retries)
     records = fetcher.search(query=query, max_results=max_results, start_date=start_date, end_date=end_date)
     typer.echo(f"Found {len(records)} arXiv papers.")
     if not records:
@@ -374,12 +377,13 @@ def arxiv_fetch_cmd(
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD (query mode only)."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
     backend: str = opt_arxiv_backend,
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch arXiv metadata and attempt to download PDFs.
 
     Provide one of: a positional query, --file, or one or more --id values.
     """
-    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend)
+    fetcher = ArxivFetcher(root_dir=storage_dir, backend=backend, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--id", arxiv_id)) if value]
@@ -426,6 +430,7 @@ def biorxiv_search_cmd(
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="Also search Europe PMC full text (boolean AND) and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search bioRxiv and write matching IDs to a text file.
 
@@ -438,7 +443,7 @@ def biorxiv_search_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days, max_retries=max_retries)
     records = fetcher.search(query=query, start_date=start_date, end_date=end_date, max_results=max_results, use_europepmc=use_europepmc)
     typer.echo(f"Found {len(records)} bioRxiv papers.")
     for record in records:
@@ -462,6 +467,7 @@ def biorxiv_fetch_cmd(
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="In query mode, also search Europe PMC full text and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch bioRxiv metadata and attempt to download PDFs.
 
@@ -475,7 +481,7 @@ def biorxiv_fetch_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, window_days=window_days, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--doi", doi)) if value]
@@ -523,6 +529,7 @@ def medrxiv_search_cmd(
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="Also search Europe PMC full text (boolean AND) and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search medRxiv and write matching IDs to a text file.
 
@@ -535,7 +542,7 @@ def medrxiv_search_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days, max_retries=max_retries)
     records = fetcher.search(query=query, start_date=start_date, end_date=end_date, max_results=max_results, use_europepmc=use_europepmc)
     typer.echo(f"Found {len(records)} medRxiv papers.")
     for record in records:
@@ -559,6 +566,7 @@ def medrxiv_fetch_cmd(
     window_days: int = typer.Option(365, "--window-days", help="Compatibility-only option. Retained for older scripts; not used by current Crossref-backed direct query path."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
     use_europepmc: bool = typer.Option(True, "--europepmc/--no-europepmc", help="In query mode, also search Europe PMC full text and union with Crossref results. Default: enabled."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch medRxiv metadata and attempt to download PDFs.
 
@@ -572,7 +580,7 @@ def medrxiv_fetch_cmd(
             fg=typer.colors.YELLOW,
         )
 
-    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days)
+    fetcher = BioRxivFetcher(root_dir=storage_dir, platform="medrxiv", window_days=window_days, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--doi", doi)) if value]
@@ -618,13 +626,14 @@ def chemrxiv_search_cmd(
     output_dir: Optional[str] = typer.Option(None, "--output-dir", "-o", help="Directory to save searched ChemRxiv IDs."),
     start_date: Optional[str] = typer.Option(None, "--start-date", help="Optional start date in YYYY-MM-DD."),
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Search ChemRxiv and write matching DOIs to a text file.
 
     Metadata is retrieved from Crossref over ChemRxiv records (prefix
     10.26434, publisher American Chemical Society (ACS)).
     """
-    fetcher = ChemRxivFetcher(root_dir=storage_dir)
+    fetcher = ChemRxivFetcher(root_dir=storage_dir, max_retries=max_retries)
     records = fetcher.search(query=query, start_date=start_date, end_date=end_date, max_results=max_results)
     typer.echo(f"Found {len(records)} ChemRxiv papers.")
     for record in records:
@@ -646,13 +655,14 @@ def chemrxiv_fetch_cmd(
     start_date: Optional[str] = typer.Option(None, "--start-date", help="Optional start date in YYYY-MM-DD (query mode only)."),
     end_date: Optional[str] = typer.Option(None, "--end-date", help="Optional end date in YYYY-MM-DD (query mode only)."),
     download_pdf: bool = typer.Option(True, "--download-pdf/--no-download-pdf", help="Download PDFs when available."),
+    max_retries: int = opt_preprint_max_retries,
 ):
     """Fetch ChemRxiv metadata and attempt to download PDFs.
 
     Metadata retrieval uses Crossref over ChemRxiv records (prefix 10.26434).
     Provide one of: a positional query, --file, or one or more --doi values.
     """
-    fetcher = ChemRxivFetcher(root_dir=storage_dir)
+    fetcher = ChemRxivFetcher(root_dir=storage_dir, max_retries=max_retries)
     output = output_dir if output_dir else storage_dir
 
     provided = [name for name, value in (("query", query), ("--file", file), ("--doi", doi)) if value]

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/src/pyPaperFlow/preprint/arxiv_fetcher.py

@@ -5,7 +5,6 @@ import importlib
 import time
 import xml.etree.ElementTree as ET
 from datetime import date, datetime, timezone
-from email.utils import parsedate_to_datetime
 from pathlib import Path
 from typing import Any, Dict, Iterable, List, Optional
 
@@ -14,12 +13,14 @@ from bs4 import BeautifulSoup
 
 from .source_models import SourcePaper
 from .source_utils import (
+    DEFAULT_MAX_RETRIES,
     build_source_record_dir,
     download_binary,
     extract_year,
     normalize_text,
     safe_filename,
     save_json,
+    sleep_before_retry,
 )
 
 
@@ -73,7 +74,7 @@ class ArxivFetcher:
         root_dir: str,
         backend: str = "native",
         batch_size: int = 100,
-        max_retries: int = 3,
+        max_retries: int = DEFAULT_MAX_RETRIES,
         request_timeout: float = 60.0,
     ):
         self.root_dir = root_dir
@@ -382,7 +383,7 @@ class ArxivFetcher:
                 if response.status_code == 429:
                     last_error = RuntimeError(f"arXiv API rate limited request for {description}")
                     if attempt + 1 < self.max_retries:
-                        self._sleep_before_retry(response, attempt)
+                        sleep_before_retry(response, attempt)
                         continue
                     break
                 response.raise_for_status()
@@ -391,19 +392,19 @@ class ArxivFetcher:
                 except ET.ParseError as exc:
                     last_error = exc
                     if attempt + 1 < self.max_retries:
-                        self._sleep_before_retry(response, attempt)
+                        sleep_before_retry(response, attempt)
                         continue
                     break
             except (httpx.HTTPStatusError, httpx.TimeoutException, httpx.TransportError) as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    self._sleep_before_retry(response, attempt)
+                    sleep_before_retry(response, attempt)
                     continue
                 break
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    self._sleep_before_retry(response, attempt)
+                    sleep_before_retry(response, attempt)
                     continue
                 break
 
@@ -426,32 +427,6 @@ class ArxivFetcher:
             self._http_client = httpx.Client(http2=False, **client_kwargs)
         return self._http_client
 
-    def _sleep_before_retry(self, response: Optional[httpx.Response], attempt: int) -> None:
-        retry_after = self._retry_after_seconds(response)
-        delay = retry_after if retry_after is not None else min(30.0, 1.5 * (2**attempt))
-        time.sleep(max(0.0, delay))
-
-    def _retry_after_seconds(self, response: Optional[httpx.Response]) -> Optional[float]:
-        if response is None:
-            return None
-
-        raw_retry_after = normalize_text(response.headers.get("Retry-After", ""))
-        if not raw_retry_after:
-            return None
-
-        if raw_retry_after.isdigit():
-            return float(raw_retry_after)
-
-        try:
-            retry_after_dt = parsedate_to_datetime(raw_retry_after)
-        except (TypeError, ValueError, IndexError):
-            return None
-
-        if retry_after_dt.tzinfo is None:
-            retry_after_dt = retry_after_dt.replace(tzinfo=timezone.utc)
-        now = datetime.now(retry_after_dt.tzinfo)
-        return max(0.0, (retry_after_dt - now).total_seconds())
-
     def _normalize_date_bounds(
         self,
         start_date: Optional[str],

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/src/pyPaperFlow/preprint/biorxiv_fetcher.py

@@ -18,15 +18,18 @@ from ..integrations.undetected_fallback import is_undetected_enabled, undetected
 
 from .source_models import SourcePaper
 from .source_utils import (
+    DEFAULT_MAX_RETRIES,
     basic_boolean_text_match,
     build_source_record_dir,
     detect_platform_from_doi,
     download_binary,
     ensure_directory,
+    extract_version_from_doi,
     extract_year,
     normalize_text,
     safe_filename,
     save_json,
+    sleep_before_retry,
 )
 
 
@@ -123,7 +126,7 @@ class BioRxivFetcher:
         root_dir: str,
         platform: str = "biorxiv",
         window_days: int = 365,
-        max_retries: int = 3,
+        max_retries: int = DEFAULT_MAX_RETRIES,
         request_timeout: float = 60.0,
     ):
         if platform not in PLATFORM_CONFIG:
@@ -136,6 +139,10 @@ class BioRxivFetcher:
         self.window_days = max(1, int(window_days))
         self.max_retries = max(1, int(max_retries))
         self.request_timeout = float(request_timeout)
+        # Set by search() when Europe PMC was unreachable, so the results came from
+        # Crossref alone. Returning [] there is indistinguishable from a genuine
+        # empty result, so callers need this to tell a lossy week from a quiet one.
+        self.last_search_degraded: Optional[str] = None
         self.headers = {
             "User-Agent": "pyPaperFlow/0.1.0 (+https://github.com/MaybeBio/pyPaperFlow)",
             "Accept": "application/json,text/html;q=0.9,*/*;q=0.8",
@@ -178,6 +185,8 @@ class BioRxivFetcher:
         if not query_text:
             raise ValueError("query must be non-empty")
 
+        self.last_search_degraded = None
+
         if DOI_RE.match(query_text):
             return self._search_by_doi(query_text)
 
@@ -262,6 +271,7 @@ class BioRxivFetcher:
         try:
             from .europepmc_fetcher import EuropePMCSearch
         except Exception:
+            self.last_search_degraded = "Europe PMC module unavailable"
             return []
 
         start_dt, end_dt = self._normalize_date_range(start_date, end_date)
@@ -280,8 +290,15 @@ class BioRxivFetcher:
                 max_results=max_results,
             )
         except Exception as exc:
+            status = getattr(getattr(exc, "response", None), "status_code", None)
+            if status:
+                reason = f"HTTP {status}"
+            else:
+                message = str(exc).strip()
+                reason = message or type(exc).__name__
+            self.last_search_degraded = f"Europe PMC unavailable ({reason})"
             print(
-                f"[biorxiv] Europe PMC search failed ({exc}); returning Crossref-only results.",
+                f"[{self.platform}] Europe PMC search failed ({exc}); returning Crossref-only results.",
                 file=sys.stderr,
             )
             return []
@@ -341,7 +358,7 @@ class BioRxivFetcher:
             landing_url=landing_url,
             pdf_url=pdf_url,
             query=query,
-            version="",
+            version=extract_version_from_doi(doi),
             keywords=[],
             extra={
                 "provider": "europepmc",
@@ -398,6 +415,7 @@ class BioRxivFetcher:
         url = f"{BIO_RXIV_CROSSREF_API}/{quote(doi, safe='')}"
         last_error: Optional[Exception] = None
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._get_http_client().get(url)
                 if response.status_code == 404:
@@ -407,7 +425,7 @@ class BioRxivFetcher:
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
         if last_error is not None:
             raise last_error
         raise RuntimeError(f"Failed to fetch Crossref work for DOI {doi}")
@@ -591,6 +609,7 @@ class BioRxivFetcher:
         last_error: Optional[Exception] = None
 
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._get_http_client().get(BIO_RXIV_CROSSREF_API, params=params)
                 response.raise_for_status()
@@ -598,7 +617,7 @@ class BioRxivFetcher:
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
 
         if last_error is not None:
             raise last_error
@@ -665,7 +684,7 @@ class BioRxivFetcher:
             landing_url=landing_url,
             pdf_url=pdf_url,
             query=query,
-            version="",
+            version=extract_version_from_doi(doi),
             keywords=keywords,
             extra={
                 "publisher": record.get("publisher", ""),

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/src/pyPaperFlow/preprint/chemrxiv_fetcher.py

@@ -1,7 +1,6 @@
 from __future__ import annotations
 
 import re
-import time
 from datetime import datetime
 from pathlib import Path
 from typing import Any, Dict, Iterable, List, Optional
@@ -15,13 +14,16 @@ from ..integrations.undetected_fallback import is_undetected_enabled, undetected
 
 from .source_models import SourcePaper
 from .source_utils import (
+    DEFAULT_MAX_RETRIES,
     basic_boolean_text_match,
     build_source_record_dir,
     download_binary,
+    extract_version_from_doi,
     extract_year,
     normalize_text,
     safe_filename,
     save_json,
+    sleep_before_retry,
 )
 
 
@@ -48,7 +50,7 @@ class ChemRxivFetcher:
     def __init__(
         self,
         root_dir: str,
-        max_retries: int = 3,
+        max_retries: int = DEFAULT_MAX_RETRIES,
         request_timeout: float = 60.0,
     ):
         self.root_dir = root_dir
@@ -157,6 +159,7 @@ class ChemRxivFetcher:
         url = f"{CHEM_RXIV_CROSSREF_API}/{quote(doi, safe='')}"
         last_error: Optional[Exception] = None
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._get_http_client().get(url)
                 if response.status_code == 404:
@@ -166,7 +169,7 @@ class ChemRxivFetcher:
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
         if last_error is not None:
             raise last_error
         raise RuntimeError(f"Failed to fetch Crossref work for DOI {doi}")
@@ -254,6 +257,7 @@ class ChemRxivFetcher:
 
         last_error: Optional[Exception] = None
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._get_http_client().get(CHEM_RXIV_CROSSREF_API, params=params)
                 response.raise_for_status()
@@ -261,7 +265,7 @@ class ChemRxivFetcher:
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
 
         if last_error is not None:
             raise last_error
@@ -321,7 +325,7 @@ class ChemRxivFetcher:
             landing_url=landing_url,
             pdf_url=pdf_url,
             query=query,
-            version="",
+            version=extract_version_from_doi(doi),
             keywords=keywords,
             extra={
                 "publisher": record.get("publisher", ""),

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/src/pyPaperFlow/preprint/europepmc_fetcher.py

@@ -1,12 +1,11 @@
 from __future__ import annotations
 
 import re
-import time
 from typing import Any, Dict, List, Optional
 
 import httpx
 
-from .source_utils import normalize_text
+from .source_utils import DEFAULT_MAX_RETRIES, normalize_text, sleep_before_retry
 
 EUROPE_PMC_SEARCH_URL = "https://www.ebi.ac.uk/europepmc/webservices/rest/search"
 EUROPE_PMC_FULLTEXT_URL = "https://www.ebi.ac.uk/europepmc/webservices/rest/{source}/{id}/fullTextXML"
@@ -24,13 +23,16 @@ class EuropePMCSearch:
     Crossref's metadata-only relevance search cannot.
 
     The client bypasses the local HTTP proxy via ``trust_env=False`` because
-    the proxy commonly times out (HTTP 504) on ``ebi.ac.uk``.
+    the proxy commonly times out (HTTP 504) on ``ebi.ac.uk``. Note that the
+    503s occasionally seen against this endpoint are an upstream EBI outage,
+    not the proxy: direct and proxied requests fail (and recover) together, and
+    ``trust_env=False`` only sidesteps the proxy's own 504 flakiness.
     """
 
     def __init__(
         self,
         request_timeout: float = 60.0,
-        max_retries: int = 3,
+        max_retries: int = DEFAULT_MAX_RETRIES,
         page_size: int = 100,
     ):
         self.request_timeout = float(request_timeout)
@@ -129,18 +131,39 @@ class EuropePMCSearch:
         }
         last_error: Optional[Exception] = None
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._client.get(EUROPE_PMC_SEARCH_URL, params=params)
                 response.raise_for_status()
-                return response.json()
+                payload = response.json()
+                self._validate_payload(payload)
+                return payload
             except Exception as exc:
                 last_error = exc
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
         if last_error is not None:
             raise last_error
         raise RuntimeError("Failed to query Europe PMC")
 
+    @staticmethod
+    def _validate_payload(payload: Any) -> None:
+        """Raise when Europe PMC returns an error inside an HTTP 200 body.
+
+        The REST API can answer with a bare ``{"version":"6.9"}`` (no
+        ``resultList``) or an ``errCode``/``errMessage`` envelope on a 200,
+        which ``raise_for_status`` cannot see. Treat those as failures so
+        callers degrade instead of silently returning an empty result set.
+        """
+        if not isinstance(payload, dict):
+            raise RuntimeError(f"Europe PMC returned a non-object payload: {type(payload).__name__}")
+        if "errCode" in payload:
+            raise RuntimeError(
+                f"Europe PMC error: errCode={payload.get('errCode')} errMessage={payload.get('errMessage', '')}"
+            )
+        if "resultList" not in payload:
+            raise RuntimeError("Europe PMC returned an unexpected payload without resultList")
+
 
 class EuropePMCFullText:
     """Fetch full-text JATS XML for a DOI from Europe PMC REST.
@@ -149,7 +172,7 @@ class EuropePMCFullText:
     times out on ebi.ac.uk.
     """
 
-    def __init__(self, request_timeout: float = 60.0, max_retries: int = 3):
+    def __init__(self, request_timeout: float = 60.0, max_retries: int = DEFAULT_MAX_RETRIES):
         self.request_timeout = float(request_timeout)
         self.max_retries = max(1, int(max_retries))
         self.headers = {
@@ -177,19 +200,21 @@ class EuropePMCFullText:
         source = "PPR" if source_id.startswith("PPR") else "PMC"
         url = EUROPE_PMC_FULLTEXT_URL.format(source=source, id=source_id)
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._client.get(url)
                 response.raise_for_status()
                 return response.text
             except Exception:
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
         return ""
 
     def _resolve_id(self, doi: str) -> str:
         """Map a DOI to a Europe PMC id (prefer pmcid, then the PPR/PMC id)."""
         params = {"query": f'DOI:"{doi}"', "format": "json", "pageSize": 1, "resultType": "core"}
         for attempt in range(self.max_retries):
+            response: Optional[httpx.Response] = None
             try:
                 response = self._client.get(EUROPE_PMC_SEARCH_URL, params=params)
                 response.raise_for_status()
@@ -200,5 +225,5 @@ class EuropePMCFullText:
                 return normalize_text(record.get("pmcid") or record.get("id") or "")
             except Exception:
                 if attempt + 1 < self.max_retries:
-                    time.sleep(min(2.0, 0.5 * (attempt + 1)))
+                    sleep_before_retry(response, attempt)
         return ""

{pypaperflow-0.6.1 → pypaperflow-0.6.2}/src/pyPaperFlow/preprint/source_utils.py

@@ -2,7 +2,9 @@ from __future__ import annotations
 
 import json
 import re
-from datetime import datetime, timedelta
+import time
+from datetime import datetime, timedelta, timezone
+from email.utils import parsedate_to_datetime
 from pathlib import Path
 from typing import Any, Dict, Iterable, List, Optional
 
@@ -12,6 +14,7 @@ import httpx
 BOOLEAN_OR_SPLIT_RE = re.compile(r"\s+OR\s+", re.IGNORECASE)
 BOOLEAN_AND_SPLIT_RE = re.compile(r"\s+AND\s+", re.IGNORECASE)
 TOKEN_RE = re.compile(r'"([^"]+)"|\'([^\']+)\'|(\S+)')
+VERSION_SUFFIX_RE = re.compile(r"v(\d+)$", re.IGNORECASE)
 
 
 def normalize_text(value: Any) -> str:
@@ -54,6 +57,21 @@ def detect_platform_from_doi(doi: Any) -> str:
     return ""
 
 
+def extract_version_from_doi(doi: Any) -> str:
+    """Return the ``N`` of a trailing ``vN`` DOI version suffix, else "".
+
+    bioRxiv/medRxiv and chemRxiv register each revision as its own DOI work
+    (e.g. ``10.1101/2023.06.22.546069v2`` or ``10.26434/chemrxiv-2025-tj4pr-v2``),
+    so the version is recoverable from the DOI itself. A DOI without a suffix
+    (or an explicit ``v1``) is the first version and yields "".
+    """
+    text = normalize_text(doi)
+    if not text:
+        return ""
+    match = VERSION_SUFFIX_RE.search(text)
+    return match.group(1) if match else ""
+
+
 def extract_year(date_text: Any) -> str:
     text = normalize_text(date_text)
     if not text:
@@ -106,6 +124,48 @@ def download_binary(url: str, output_path: Path | str, headers: Optional[Dict[st
         return False
 
 
+def retry_after_seconds(response: Optional[httpx.Response]) -> Optional[float]:
+    """Seconds to wait per the Retry-After header, or None when unusable."""
+    if response is None:
+        return None
+
+    raw_retry_after = normalize_text(response.headers.get("Retry-After", ""))
+    if not raw_retry_after:
+        return None
+
+    if raw_retry_after.isdigit():
+        return float(raw_retry_after)
+
+    try:
+        retry_after_dt = parsedate_to_datetime(raw_retry_after)
+    except (TypeError, ValueError, IndexError):
+        return None
+
+    if retry_after_dt.tzinfo is None:
+        retry_after_dt = retry_after_dt.replace(tzinfo=timezone.utc)
+    now = datetime.now(retry_after_dt.tzinfo)
+    return max(0.0, (retry_after_dt - now).total_seconds())
+
+
+# Default retry budget for every preprint fetcher. This is the interactive/tool
+# default: fail fast (~4.5s of backoff) so a human gets a quick answer plus the
+# degradation notice, instead of a ~22s silent stall. Unattended callers (e.g.
+# monitor.py) pass a larger max_retries explicitly.
+DEFAULT_MAX_RETRIES = 3
+
+
+def sleep_before_retry(response: Optional[httpx.Response], attempt: int) -> None:
+    """Back off between attempts, capped at 30s.
+
+    A 503 usually asks for a pause via Retry-After; otherwise back off
+    exponentially. A short linear cap cannot outlast a transient outage, which
+    pushes callers into whatever lossy fallback they have.
+    """
+    retry_after = retry_after_seconds(response)
+    delay = retry_after if retry_after is not None else min(30.0, 1.5 * (2**attempt))
+    time.sleep(max(0.0, delay))
+
+
 def parse_boolean_query(query: str) -> List[List[str]]:
     text = normalize_text(query)
     if not text:

pypaperflow-0.6.1/.claude/settings.local.json

@@ -1,8 +0,0 @@
-{
-  "permissions": {
-    "allow": [
-      "Bash(curl -s -m 40 --noproxy '*' \"https://www.ebi.ac.uk/europepmc/webservices/rest/search?query=SRC:PPR%20AND%20zinc%20AND%20finger%20AND%20263%20AND%20FIRST_PDATE:%5B2026-08-01%20TO%202026-12-31%5D&format=json&pageSize=25&cursorMark=*&resultType=core\" -o /tmp/epmc_date.json -w \"HTTP=%{http_code}\\\\n\")",
-      "Bash(python3 -c ' *)"
-    ]
-  }
-}

pypaperflow-0.6.1/src/pyPaperFlow/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.6.1"