pyPaperFlow 0.6.0__tar.gz → 0.6.1__tar.gz

{pypaperflow-0.6.0 → pypaperflow-0.6.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: pyPaperFlow
-Version: 0.6.0
+Version: 0.6.1
 Summary: Automated paper fetching and analysis platform.
 Project-URL: Homepage, https://github.com/MaybeBio/pyPaperFlow
 Project-URL: Issues, https://github.com/MaybeBio/pyPaperFlow/issues
@@ -112,6 +112,7 @@ This tool is designed to `complement rather than replace` reference management s
 
 - **Automated Retrieval from Multiple Sources**: Automatically search and retrieve paper metadata and full-text records from `PubMed/Medline, arXiv, medRxiv, chemRxiv and bioRxiv`. The repository focuses primarily on biomedical research and computational interdisciplinary fields (`Biomedicine + Computational Biology`).
 - **Full-Text Access**: Enable automatic downloading of open-access full texts in XML/Text format from `PMC`. For preprints and other publications without accessible PMC full texts, alternative acquisition modules are integrated to fetch `original PDFs`, with `Sci-Hub` set as the fallback provider.
+- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` / `EuropePMCFullText.full_text_xml(doi)` read JATS full-text XML from Europe PMC (bioRxiv / medRxiv and other DOI-indexed preprints).
 - **Structured Storage**:
   - **Metadata**: Preserved in well-structured detailed JSON files.
   - **Full Text**: Stored in multiple formats including parsed JSON and Markdown for versatile downstream usage — JSON for programmatic data analysis, and Markdown optimized for LLM comprehension and processing.
@@ -1450,6 +1451,31 @@ In theory, all DOI‑driven literature workflows can be standardised following t
 
 > Modules dedicated to the aforementioned preprint platforms are still under development and refinement. Preprint‑related subcommands are provided for testing purposes only. For detailed test cases, refer to [Cases](./docs/Cases.md)
 
+#### Preprint full-text fetch (Python API)
+
+For preprints without an open-access PDF, each fetcher exposes a `fetch_full_text()` method that returns clean section-headed text without any PDF parsing:
+
+```python
+from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher
+from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher
+from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
+
+# arXiv → ar5iv rendered HTML (LaTeX → HTML)
+arxiv = ArxivFetcher(root_dir="./papers")
+text = arxiv.fetch_full_text("1706.03762")            # "" on failure
+
+# bioRxiv / medRxiv → Europe PMC fullTextXML
+biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
+text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
+
+# Any DOI-indexed preprint → Europe PMC fullTextXML directly
+epmc = EuropePMCFullText()
+xml = epmc.full_text_xml("10.1101/2023.06.22.546069")
+epmc.close()
+```
+
+All three return an empty string `""` on failure, so callers can gracefully fall back to the abstract. The returned text is section-headed (`## Section`) plain text ready for LLM input.
+
 ### 6. Critical Reading and Knowledge Graph Analysis: Downstream End‑Use
 
 Upon completing literature retrieval, parsing, and structured processing as outlined above, users obtain chapter‑organised Markdown files and structured JSON files, which serve as the fundamental inputs for subsequent critical reading and knowledge graph analysis.

{pypaperflow-0.6.0 → pypaperflow-0.6.1}/README.md

@@ -83,6 +83,7 @@ This tool is designed to `complement rather than replace` reference management s
 
 - **Automated Retrieval from Multiple Sources**: Automatically search and retrieve paper metadata and full-text records from `PubMed/Medline, arXiv, medRxiv, chemRxiv and bioRxiv`. The repository focuses primarily on biomedical research and computational interdisciplinary fields (`Biomedicine + Computational Biology`).
 - **Full-Text Access**: Enable automatic downloading of open-access full texts in XML/Text format from `PMC`. For preprints and other publications without accessible PMC full texts, alternative acquisition modules are integrated to fetch `original PDFs`, with `Sci-Hub` set as the fallback provider.
+- **Preprint Full-Text Fetch (no PDF parsing)**: For preprints without an OA PDF, dedicated methods return clean section-headed text directly — `ArxivFetcher.fetch_full_text(arxiv_id)` reads ar5iv-rendered HTML (arXiv LaTeX → HTML), and `BioRxivFetcher.fetch_full_text(doi)` / `EuropePMCFullText.full_text_xml(doi)` read JATS full-text XML from Europe PMC (bioRxiv / medRxiv and other DOI-indexed preprints).
 - **Structured Storage**:
   - **Metadata**: Preserved in well-structured detailed JSON files.
   - **Full Text**: Stored in multiple formats including parsed JSON and Markdown for versatile downstream usage — JSON for programmatic data analysis, and Markdown optimized for LLM comprehension and processing.
@@ -1421,6 +1422,31 @@ In theory, all DOI‑driven literature workflows can be standardised following t
 
 > Modules dedicated to the aforementioned preprint platforms are still under development and refinement. Preprint‑related subcommands are provided for testing purposes only. For detailed test cases, refer to [Cases](./docs/Cases.md)
 
+#### Preprint full-text fetch (Python API)
+
+For preprints without an open-access PDF, each fetcher exposes a `fetch_full_text()` method that returns clean section-headed text without any PDF parsing:
+
+```python
+from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher
+from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher
+from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
+
+# arXiv → ar5iv rendered HTML (LaTeX → HTML)
+arxiv = ArxivFetcher(root_dir="./papers")
+text = arxiv.fetch_full_text("1706.03762")            # "" on failure
+
+# bioRxiv / medRxiv → Europe PMC fullTextXML
+biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
+text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
+
+# Any DOI-indexed preprint → Europe PMC fullTextXML directly
+epmc = EuropePMCFullText()
+xml = epmc.full_text_xml("10.1101/2023.06.22.546069")
+epmc.close()
+```
+
+All three return an empty string `""` on failure, so callers can gracefully fall back to the abstract. The returned text is section-headed (`## Section`) plain text ready for LLM input.
+
 ### 6. Critical Reading and Knowledge Graph Analysis: Downstream End‑Use
 
 Upon completing literature retrieval, parsing, and structured processing as outlined above, users obtain chapter‑organised Markdown files and structured JSON files, which serve as the fundamental inputs for subsequent critical reading and knowledge graph analysis.

{pypaperflow-0.6.0 → pypaperflow-0.6.1}/README_zh.md

@@ -82,6 +82,7 @@
 
 - **多来源自动检索**：自动从 `PubMed/Medline`、`arXiv`、`medRxiv`、`chemRxiv` 和 `bioRxiv` 搜索并获取论文元数据与全文记录。项目主要聚焦于生物医学与计算交叉领域（`Biomedicine + Computational Biology`）。
 - **全文获取**：支持自动从 `PMC` 下载开放获取的 XML/Text 全文。对于预印本及其他没有 PMC 全文的文献，集成了额外的获取模块以下载 `原始 PDF`，并将 `Sci-Hub` 作为兜底来源。
+- **预印本全文获取（免 PDF 解析）**：对于没有开放获取 PDF 的预印本，提供专用方法直接返回带章节标题的纯文本——`ArxivFetcher.fetch_full_text(arxiv_id)` 读取 ar5iv 渲染 HTML（arXiv LaTeX→HTML），`BioRxivFetcher.fetch_full_text(doi)` / `EuropePMCFullText.full_text_xml(doi)` 从 Europe PMC 读取 JATS 全文 XML（bioRxiv / medRxiv 及其它 DOI 收录预印本）。
 - **结构化存储**：
     - **元数据**：保存为结构清晰的详细 JSON 文件。
     - **全文**：保存为多种格式，包括解析后的 JSON 和 Markdown，方便下游使用。其中 JSON 适合程序化分析，Markdown 更适合 LLM 理解与处理。
@@ -1440,6 +1441,31 @@ mineru_config.yaml                mineru_export_config.yaml
 
 > ⚠️ `针对上述预印本平台的模块目前基本已经开发完毕，后续只对相关功能进行维护和优化`, 测试细节与pubmed合并，详情见[Cases](./docs/Cases.md)
 
+#### 预印本全文获取（Python API）
+
+对于没有开放获取 PDF 的预印本，各 fetcher 提供 `fetch_full_text()` 方法，免 PDF 解析直接返回带章节标题的纯文本：
+
+```python
+from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher
+from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher
+from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
+
+# arXiv → ar5iv 渲染 HTML（LaTeX → HTML）
+arxiv = ArxivFetcher(root_dir="./papers")
+text = arxiv.fetch_full_text("1706.03762")            # 失败时返回 ""
+
+# bioRxiv / medRxiv → Europe PMC fullTextXML
+biorxiv = BioRxivFetcher(root_dir="./papers", platform="biorxiv")
+text = biorxiv.fetch_full_text("10.1101/2023.06.22.546069")
+
+# 任意 DOI 收录预印本 → 直接取 Europe PMC fullTextXML
+epmc = EuropePMCFullText()
+xml = epmc.full_text_xml("10.1101/2023.06.22.546069")
+epmc.close()
+```
+
+三者失败时均返回空字符串 `""`，调用方可优雅回退到摘要。返回文本为带章节标题（`## Section`）的纯文本，可直接作为 LLM 输入。
+
 
 #### 1. 命令速查 (TL;DR)
 

pypaperflow-0.6.1/src/pyPaperFlow/__init__.py

@@ -0,0 +1 @@
+__version__ = "0.6.1"

{pypaperflow-0.6.0 → pypaperflow-0.6.1}/src/pyPaperFlow/preprint/biorxiv_fetcher.py

@@ -1,6 +1,8 @@
 from __future__ import annotations
 
+import random
 import sys
+import threading
 import time
 import re
 from datetime import datetime, timedelta
@@ -35,6 +37,22 @@ MED_RXIV_LAUNCH_DATE = datetime(2019, 6, 1)
 
 DOI_RE = re.compile(r"^10\.\d{4,9}/[^\s]+$")
 
+# bioRxiv/medRxiv return HTTP 429 for non-browser User-Agents on the .full-text
+# route; a browser UA is required to fetch rendered full text.
+_FULLTEXT_HEADERS = {
+    "User-Agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/120.0 Safari/537.36",
+    "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+}
+
+# Rate-limit / bot-wall hygiene for the .full-text route. bioRxiv/medRxiv sit
+# behind a Cloudflare wall that 429s on rapid successive requests, so we (a)
+# space requests apart, and (b) back off exponentially on 429/403/5xx rather
+# than treating them like a missing article.
+_FULLTEXT_MIN_INTERVAL = 2.0  # seconds between .full-text requests (shared across instances)
+_FULLTEXT_BACKOFF_BASE = 3.0  # seconds; doubled per retry
+_FULLTEXT_BACKOFF_CAP = 20.0  # seconds; hard ceiling on a single backoff sleep
+_FULLTEXT_COOLDOWN = 30.0  # seconds to pause all requests after a 429/403
+
 
 def _jats_xml_to_text(xml: str) -> str:
     """Convert JATS full-text XML to section-headed plain text."""
@@ -50,6 +68,35 @@ def _jats_xml_to_text(xml: str) -> str:
                 parts.append(text)
     return "\n\n".join(parts).strip()
 
+
+def _biorxiv_html_to_text(html: str) -> str:
+    """Convert bioRxiv/medRxiv full-text HTML to section-headed plain text.
+
+    Walks the article body in document order, emitting headings as ``## ...``
+    and paragraphs as body text. Stops at the "References" section and skips
+    figure/table caption paragraphs.
+    """
+    soup = BeautifulSoup(html, "html.parser")
+    article = soup.find("div", class_="fulltext-view") or soup
+    parts: List[str] = []
+    for node in article.find_all(["h1", "h2", "h3", "p"]):
+        if node.name in ("h1", "h2", "h3"):
+            text = normalize_text(node.get_text(" ", strip=True))
+            if not text:
+                continue
+            if text.lower() == "references":
+                break
+            parts.append(f"\n## {text}")
+        else:
+            parent = node.find_parent("div")
+            classes = set(parent.get("class") or []) if parent is not None else set()
+            if classes & {"fig-caption", "table-caption"}:
+                continue
+            text = normalize_text(node.get_text(" ", strip=True))
+            if text:
+                parts.append(text)
+    return "\n\n".join(parts).strip()
+
 PLATFORM_CONFIG = {
     "biorxiv": {
         "journal": "bioRxiv",
@@ -65,6 +112,12 @@ PLATFORM_CONFIG = {
 
 
 class BioRxivFetcher:
+    # Shared across instances so sequential fetches (even from separate
+    # BioRxivFetcher objects) never fire back-to-back .full-text requests.
+    _fulltext_last_request = 0.0
+    _fulltext_cooldown_until = 0.0
+    _fulltext_lock = threading.Lock()
+
     def __init__(
         self,
         root_dir: str,
@@ -88,11 +141,15 @@ class BioRxivFetcher:
             "Accept": "application/json,text/html;q=0.9,*/*;q=0.8",
         }
         self._http_client: Optional[httpx.Client] = None
+        self._fulltext_client: Optional[httpx.Client] = None
 
     def close(self) -> None:
         if self._http_client is not None:
             self._http_client.close()
             self._http_client = None
+        if self._fulltext_client is not None:
+            self._fulltext_client.close()
+            self._fulltext_client = None
 
     def __del__(self) -> None:
         try:
@@ -398,12 +455,86 @@ class BioRxivFetcher:
         return records
 
     def fetch_full_text(self, doi: str) -> str:
-        """Return full text for a bioRxiv/medRxiv preprint via Europe PMC, or ""."""
-        from .europepmc_fetcher import EuropePMCFullText
+        """Return full text for a bioRxiv/medRxiv preprint, or "" on failure.
 
-        doi = normalize_text(doi)
+        Primary route: the preprint's own full-text HTML on biorxiv.org /
+        medrxiv.org. Fallback: Europe PMC fullTextXML (only present once the
+        preprint is published into PMC).
+        """
+        doi = re.sub(r"v\d+$", "", normalize_text(doi))
         if not doi:
             return ""
+        text = self._fetch_full_text_html(doi)
+        if text:
+            return text
+        return self._fetch_full_text_europepmc(doi)
+
+    def _fulltext_http_client(self) -> httpx.Client:
+        if self._fulltext_client is None:
+            self._fulltext_client = httpx.Client(
+                headers=_FULLTEXT_HEADERS,
+                timeout=self.request_timeout,
+                follow_redirects=True,
+                trust_env=False,
+            )
+        return self._fulltext_client
+
+    @staticmethod
+    def _throttle_fulltext() -> None:
+        """Space .full-text requests so they don't trip the Cloudflare wall.
+
+        Waits out both the minimum inter-request gap and any active post-429
+        cooldown, so a rate-limited fetch pauses the whole batch rather than
+        hammering the wall request after request.
+        """
+        with BioRxivFetcher._fulltext_lock:
+            now = time.monotonic()
+            wait = max(
+                _FULLTEXT_MIN_INTERVAL - (now - BioRxivFetcher._fulltext_last_request),
+                BioRxivFetcher._fulltext_cooldown_until - now,
+            )
+            if wait > 0:
+                time.sleep(wait)
+            BioRxivFetcher._fulltext_last_request = time.monotonic()
+
+    @staticmethod
+    def _mark_fulltext_throttled() -> None:
+        with BioRxivFetcher._fulltext_lock:
+            BioRxivFetcher._fulltext_cooldown_until = time.monotonic() + _FULLTEXT_COOLDOWN
+
+    @staticmethod
+    def _backoff_seconds(attempt: int) -> float:
+        base = _FULLTEXT_BACKOFF_BASE * (2 ** attempt)
+        return min(_FULLTEXT_BACKOFF_CAP, base) + random.uniform(0, 1)
+
+    def _fetch_full_text_html(self, doi: str) -> str:
+        url = f"{self.landing_base}/{doi}.full-text"
+        client = self._fulltext_http_client()
+        for attempt in range(self.max_retries):
+            self._throttle_fulltext()
+            try:
+                response = client.get(url)
+            except Exception:
+                if attempt + 1 < self.max_retries:
+                    time.sleep(self._backoff_seconds(attempt))
+                continue
+            status = response.status_code
+            if status == 200:
+                return _biorxiv_html_to_text(response.text)
+            if status == 404:
+                # No full-text page for this DOI; don't burn retries.
+                return ""
+            # 429 / 403 / 5xx: transient (rate limit / bot wall / server error).
+            # Mark a global cooldown so the rest of the batch pauses, then back
+            # off and retry.
+            self._mark_fulltext_throttled()
+            if attempt + 1 < self.max_retries:
+                time.sleep(self._backoff_seconds(attempt))
+        return ""
+
+    def _fetch_full_text_europepmc(self, doi: str) -> str:
+        from .europepmc_fetcher import EuropePMCFullText
+
         fetcher = EuropePMCFullText(
             request_timeout=self.request_timeout,
             max_retries=self.max_retries,

pypaperflow-0.6.0/src/pyPaperFlow/__init__.py

@@ -1 +0,0 @@
-__version__ = "0.6.0"

pypaperflow-0.6.0/tests/test_arxiv_fulltext.py

@@ -1,47 +0,0 @@
-from pyPaperFlow.preprint.arxiv_fetcher import ArxivFetcher, _html_to_text
-
-HTML = """<html><body>
-<h1 class="ltx_title">Title</h1>
-<section class="ltx_section"><h2 class="ltx_title">Introduction</h2>
-<p class="ltx_p">First paragraph.</p><p class="ltx_p">Second paragraph.</p></section>
-<section class="ltx_section"><h2 class="ltx_title">Results</h2><p class="ltx_p">Result text.</p></section>
-</body></html>"""
-
-
-def test_html_to_text_extracts_sections():
-    text = _html_to_text(HTML)
-    assert "## Introduction" in text
-    assert "First paragraph." in text
-    assert "## Results" in text
-
-
-def test_fetch_full_text_returns_body():
-    fetcher = ArxivFetcher(root_dir="/tmp")
-
-    class FakeResp:
-        status_code = 200
-        text = HTML
-
-    class FakeClient:
-        def get(self, url, follow_redirects=True):
-            assert url == "https://ar5iv.labs.arxiv.org/html/2301.00001"
-            return FakeResp()
-
-    fetcher._http_client = FakeClient()
-    text = fetcher.fetch_full_text("2301.00001v2")
-    assert "First paragraph." in text
-
-
-def test_fetch_full_text_404_returns_empty():
-    fetcher = ArxivFetcher(root_dir="/tmp")
-
-    class FakeResp:
-        status_code = 404
-        text = ""
-
-    class FakeClient:
-        def get(self, url, follow_redirects=True):
-            return FakeResp()
-
-    fetcher._http_client = FakeClient()
-    assert fetcher.fetch_full_text("2301.00001") == ""

pypaperflow-0.6.0/tests/test_biorxiv_fulltext.py

@@ -1,101 +0,0 @@
-import types
-
-from pyPaperFlow.preprint.biorxiv_fetcher import BioRxivFetcher, _jats_xml_to_text
-from pyPaperFlow.preprint.europepmc_fetcher import EuropePMCFullText
-
-XML = """<article><body><sec><title>Abstract</title><p>Abs text.</p></sec>
-<sec><title>Results</title><p>Result one.</p><p>Result two.</p></sec></body></article>"""
-
-
-def test_jats_xml_to_text():
-    text = _jats_xml_to_text(XML)
-    assert "## Abstract" in text
-    assert "Abs text." in text
-    assert "## Results" in text
-    assert "Result two." in text
-
-
-def test_full_text_xml_resolves_then_fetches():
-    ft = EuropePMCFullText()
-
-    def search_resp():
-        r = types.SimpleNamespace()
-        r.raise_for_status = lambda: None
-        r.json = lambda: {"resultList": {"result": [{"pmcid": "PMC123"}]}}
-        return r
-
-    def xml_resp():
-        r = types.SimpleNamespace()
-        r.raise_for_status = lambda: None
-        r.text = XML
-        return r
-
-    class FakeClient:
-        def __init__(self):
-            self.urls = []
-
-        def get(self, url, params=None):
-            self.urls.append(url)
-            if "fullTextXML" in url:
-                return xml_resp()
-            return search_resp()
-
-        def close(self):
-            pass
-
-    ft._client = FakeClient()
-    xml = ft.full_text_xml("10.1101/2026.01.01.123456")
-    assert "Results" in xml
-    assert any("PMC/PMC123/fullTextXML" in u for u in ft._client.urls)
-
-
-def test_full_text_xml_no_result_empty():
-    ft = EuropePMCFullText()
-    r = types.SimpleNamespace()
-    r.raise_for_status = lambda: None
-    r.json = lambda: {"resultList": {"result": []}}
-
-    class FakeClient:
-        def get(self, url, params=None):
-            return r
-
-        def close(self):
-            pass
-
-    ft._client = FakeClient()
-    assert ft.full_text_xml("10.1101/2026.01.01.999999") == ""
-
-
-def test_biorxiv_fetch_full_text(monkeypatch):
-    class FakeFT:
-        def __init__(self, request_timeout, max_retries):
-            pass
-
-        def full_text_xml(self, doi):
-            assert doi == "10.1101/2026.01.01.123456"
-            return XML
-
-        def close(self):
-            pass
-
-    monkeypatch.setattr("pyPaperFlow.preprint.europepmc_fetcher.EuropePMCFullText", FakeFT)
-    fetcher = BioRxivFetcher(root_dir="/tmp", platform="biorxiv")
-    text = fetcher.fetch_full_text("10.1101/2026.01.01.123456")
-    assert "## Results" in text
-    assert "Result one." in text
-
-
-def test_biorxiv_fetch_full_text_empty(monkeypatch):
-    class FakeFT:
-        def __init__(self, request_timeout, max_retries):
-            pass
-
-        def full_text_xml(self, doi):
-            return ""
-
-        def close(self):
-            pass
-
-    monkeypatch.setattr("pyPaperFlow.preprint.europepmc_fetcher.EuropePMCFullText", FakeFT)
-    fetcher = BioRxivFetcher(root_dir="/tmp", platform="biorxiv")
-    assert fetcher.fetch_full_text("10.1101/2026.01.01.999999") == ""