fetchpdf 0.1.7__tar.gz → 0.1.8__tar.gz

{fetchpdf-0.1.7/fetchpdf.egg-info → fetchpdf-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fetchpdf
-Version: 0.1.7
+Version: 0.1.8
 Summary: A comprehensive tool to download PDFs from DOIs using multiple sources
 Author-email: Dan Elton <delton17@gmail.com>
 License-Expression: MIT
@@ -578,7 +578,8 @@ suppressed when the answer is already known: any flag that turns the
 supplementary pass on (`--pull-supplementary`, `--refresh-supplementary`,
 `--download-data-artifacts`, `--fetch-everything`, `--llm-agent-retrieval`), an
 explicit `--on-existing skip` or `supplement`, `--abstract-only`, or the
-goal-aware flags (`--get-xml-or-html-and-pdf`, `--upgrade-existing` on the tiered path),
+goal-aware flags (`--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`,
+`--upgrade-existing` on the tiered path),
 which do not skip records at all. Single-identifier runs never ask.
 
 What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
@@ -678,11 +679,12 @@ Twenty routes, tried in a fixed order so the numbering is reproducible:
 | Figshare / Zenodo / OSF | repository DOIs, and datasets reached through link services |
 | DataCite reverse relations | any DOI — finds deposits whose own DataCite record points at the article through a non-citation relation (e.g. `IsSupplementTo`); deposits that merely cite it are dropped |
 | Crossref component DOIs | publishers that register components |
-| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks) — recovery of last resort |
+| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks, publisher Source Data links) — recovery of last resort. Read from the PMC copy or, with no PMCID, from a publisher `{stem}.xml` saved beside the record (e.g. by `--get-xml-and-pdf`); TEI and non-JATS XML are ignored |
 | OpenAIRE ScholeXplorer | any DOI — Scholix publication→dataset/software links |
 | Europe PMC datalinks | any PMID — text-mined accessions and data citations |
 | Full-text scan | deposits the paper names in its own prose — only under `--download-data-artifacts` |
 | Atypon publisher SI | PNAS, Science, Annual Reviews, SAGE, T&F — only when Europe PMC withholds the SI or the record has no PMCID; drives a headed browser, so it needs Playwright, Chromium and a display (or Xvfb) |
+| Internet Archive (Wayback Machine) | Last-ditch fallback for publisher-hosted SI (Atypon hosts incl. SAGE, INFORMS, AOM, PNAS; Elsevier `ars.els-cdn.com` mmc files; supp.apa.org): asks the Archive's open CDX index for captures under the article's supplement URLs and downloads the archived copy. Runs only when no publisher-supplement provider listed anything; an Archive outage is reported as `blocked`, not as "none found" |
 | LLM retrieval agent | only under [`--fetch-everything`](#--fetch-everything-the-second-layer) or `--llm-agent-retrieval` |
 | JCI | `10.1172/jci…` |
 
@@ -1134,6 +1136,11 @@ the DOI link, and the PDF the landing page declares is fetched: **AACR**
 (`10.1182`), **Company of Biologists** (`10.1242`), **ERS** (`10.1183`),
 **BMJ** (`10.1136`), **Bioscientifica** (`10.1530`) and **LWW** (`10.1097`).
 
+**IEEE Xplore** (`10.1109`) keys articles by a number the DOI does not
+contain, so the route reads it from the DOI link's redirect and fetches
+`stampPDF/getPDF.jsp` with your cookies. Sign in once (or just open any IEEE
+PDF from a network your library covers) and the session cookies carry over.
+
 Several of these are open access but sit behind a Cloudflare bot check that a
 plain download cannot pass. Opening any article on the site once in your
 everyday browser earns the clearance cookie, and the next run reuses it.
@@ -1229,9 +1236,9 @@ refusal, not a bare failure:
 | `path` | str \| null | Absolute path to the artifact, or null. |
 | `format` | str | From the artifact actually written: `pdf`, `xml`, `html`, `latex`, `landing`, `suppl`, `abstract`, `text`, `other`. **A successful record is not necessarily a PDF** — with the default XML fallback a record can succeed as `xml`. |
 | `source` | str \| null | The route that produced it (`unpaywall`, `core`, `osf`, …), `existing` for a skip, `structured_fallback` when no PDF was available and structured full text was saved instead. |
-| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
+| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
 | `reasons` | list[str] | Why a record failed, or why a kept PDF could not be verified. Empty on a verified success. |
-| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf`), empty otherwise. `path` is not repeated here. |
+| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf` and its `--get-xml-and-pdf` / `--get-xml-and-html-and-pdf` variants), empty otherwise. `path` is not repeated here. |
 | `supplementary` | object | Only with `--pull-supplementary` (or a flag that implies it): `status` (`ok`, `none_found`, `partial`, `incomplete`, `error`, `skipped`), `written`, `skipped`, `manifest` (absolute path or null), `missing_declared` (files the article declares that were not obtained) and `blocked_urls` (refused to this client; open them in a browser). |
 | `figures` | object | Only with `--pull-figures`: `status`, `written`, `failed`, `refusal` and `manifest`. |
 | `images` | object | Only with `--extract-images`: `status`, `written` and `manifest`. |
@@ -1469,7 +1476,7 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
       records whose key is entitled to the XML but not the PDF, and covers
       ScienceDirect DOIs outside `10.1016/*`. It does not repeat a PDF request the
       early step already made. The XML half is skipped with `--no-xml-fallback`.
-    - With `--prioritize-xml` / `--xml-only` / `--get-xml-or-html-and-pdf` the tiered
+    - With `--prioritize-xml` / `--xml-only` / any `--get-xml-*-and-pdf` flag the tiered
       ladder asks the same endpoint itself (`elsevier_tdm` at T1, `elsevier_pdf`
       at T5, ahead of the legacy chain).
     - **Off campus?** Entitlement is decided by your IP unless you also set

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/README.md

@@ -538,7 +538,8 @@ suppressed when the answer is already known: any flag that turns the
 supplementary pass on (`--pull-supplementary`, `--refresh-supplementary`,
 `--download-data-artifacts`, `--fetch-everything`, `--llm-agent-retrieval`), an
 explicit `--on-existing skip` or `supplement`, `--abstract-only`, or the
-goal-aware flags (`--get-xml-or-html-and-pdf`, `--upgrade-existing` on the tiered path),
+goal-aware flags (`--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`,
+`--upgrade-existing` on the tiered path),
 which do not skip records at all. Single-identifier runs never ask.
 
 What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
@@ -638,11 +639,12 @@ Twenty routes, tried in a fixed order so the numbering is reproducible:
 | Figshare / Zenodo / OSF | repository DOIs, and datasets reached through link services |
 | DataCite reverse relations | any DOI — finds deposits whose own DataCite record points at the article through a non-citation relation (e.g. `IsSupplementTo`); deposits that merely cite it are dropped |
 | Crossref component DOIs | publishers that register components |
-| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks) — recovery of last resort |
+| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks, publisher Source Data links) — recovery of last resort. Read from the PMC copy or, with no PMCID, from a publisher `{stem}.xml` saved beside the record (e.g. by `--get-xml-and-pdf`); TEI and non-JATS XML are ignored |
 | OpenAIRE ScholeXplorer | any DOI — Scholix publication→dataset/software links |
 | Europe PMC datalinks | any PMID — text-mined accessions and data citations |
 | Full-text scan | deposits the paper names in its own prose — only under `--download-data-artifacts` |
 | Atypon publisher SI | PNAS, Science, Annual Reviews, SAGE, T&F — only when Europe PMC withholds the SI or the record has no PMCID; drives a headed browser, so it needs Playwright, Chromium and a display (or Xvfb) |
+| Internet Archive (Wayback Machine) | Last-ditch fallback for publisher-hosted SI (Atypon hosts incl. SAGE, INFORMS, AOM, PNAS; Elsevier `ars.els-cdn.com` mmc files; supp.apa.org): asks the Archive's open CDX index for captures under the article's supplement URLs and downloads the archived copy. Runs only when no publisher-supplement provider listed anything; an Archive outage is reported as `blocked`, not as "none found" |
 | LLM retrieval agent | only under [`--fetch-everything`](#--fetch-everything-the-second-layer) or `--llm-agent-retrieval` |
 | JCI | `10.1172/jci…` |
 
@@ -1094,6 +1096,11 @@ the DOI link, and the PDF the landing page declares is fetched: **AACR**
 (`10.1182`), **Company of Biologists** (`10.1242`), **ERS** (`10.1183`),
 **BMJ** (`10.1136`), **Bioscientifica** (`10.1530`) and **LWW** (`10.1097`).
 
+**IEEE Xplore** (`10.1109`) keys articles by a number the DOI does not
+contain, so the route reads it from the DOI link's redirect and fetches
+`stampPDF/getPDF.jsp` with your cookies. Sign in once (or just open any IEEE
+PDF from a network your library covers) and the session cookies carry over.
+
 Several of these are open access but sit behind a Cloudflare bot check that a
 plain download cannot pass. Opening any article on the site once in your
 everyday browser earns the clearance cookie, and the next run reuses it.
@@ -1189,9 +1196,9 @@ refusal, not a bare failure:
 | `path` | str \| null | Absolute path to the artifact, or null. |
 | `format` | str | From the artifact actually written: `pdf`, `xml`, `html`, `latex`, `landing`, `suppl`, `abstract`, `text`, `other`. **A successful record is not necessarily a PDF** — with the default XML fallback a record can succeed as `xml`. |
 | `source` | str \| null | The route that produced it (`unpaywall`, `core`, `osf`, …), `existing` for a skip, `structured_fallback` when no PDF was available and structured full text was saved instead. |
-| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
+| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
 | `reasons` | list[str] | Why a record failed, or why a kept PDF could not be verified. Empty on a verified success. |
-| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf`), empty otherwise. `path` is not repeated here. |
+| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf` and its `--get-xml-and-pdf` / `--get-xml-and-html-and-pdf` variants), empty otherwise. `path` is not repeated here. |
 | `supplementary` | object | Only with `--pull-supplementary` (or a flag that implies it): `status` (`ok`, `none_found`, `partial`, `incomplete`, `error`, `skipped`), `written`, `skipped`, `manifest` (absolute path or null), `missing_declared` (files the article declares that were not obtained) and `blocked_urls` (refused to this client; open them in a browser). |
 | `figures` | object | Only with `--pull-figures`: `status`, `written`, `failed`, `refusal` and `manifest`. |
 | `images` | object | Only with `--extract-images`: `status`, `written` and `manifest`. |
@@ -1429,7 +1436,7 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
       records whose key is entitled to the XML but not the PDF, and covers
       ScienceDirect DOIs outside `10.1016/*`. It does not repeat a PDF request the
       early step already made. The XML half is skipped with `--no-xml-fallback`.
-    - With `--prioritize-xml` / `--xml-only` / `--get-xml-or-html-and-pdf` the tiered
+    - With `--prioritize-xml` / `--xml-only` / any `--get-xml-*-and-pdf` flag the tiered
       ladder asks the same endpoint itself (`elsevier_tdm` at T1, `elsevier_pdf`
       at T5, ahead of the legacy chain).
     - **Off campus?** Entitlement is decided by your IP unless you also set

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/__init__.py

@@ -15,7 +15,7 @@ from .fetch_metadata_from_doi import fetch_metadata_from_doi
 # behavioural difference, it is the same object.
 fetch_pdf_from_doi = fetch_pdf
 
-__version__ = "0.1.7"
+__version__ = "0.1.8"
 __all__ = [
     "fetch_pdf",
     "fetch_pdf_from_doi",  # deprecated alias for fetch_pdf

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/fetchpdf.py

@@ -5756,7 +5756,7 @@ def _append_missing_si_to_report(output_dir, si_statuses, run_timestamp, lock,
     for display_id, summary in (si_statuses or {}).items():
         status = getattr(summary, "status", "")
         blocked_urls = list(getattr(summary, "blocked_urls", None) or [])
-        if status not in ("partial", "incomplete"):
+        if status not in ("partial", "incomplete", "blocked"):
             continue
         missing = ", ".join(getattr(summary, "missing_declared", None) or []) or "—"
         # A withheld-but-existing supplement is the click-through candidate:
@@ -6426,6 +6426,11 @@ def _batch_fetch_pdfs_inner(dois, output_dir, email=None, verbose=False, delay=0
                 # point of the gate is that this cannot pass silently.
                 names = ", ".join(summary.missing_declared) or "unknown"
                 print(f"   ⚠️  {display_id}: SI INCOMPLETE — missing: {names}")
+            elif summary.status == "blocked":
+                # Not "none found": a source refused or was down, so the paper
+                # may well have supplements. Printed regardless of verbosity.
+                print(f"   ⚠️  {display_id}: SI BLOCKED — a supplement source refused "
+                      f"or was unavailable ({summary.detail or 'no detail'})")
             elif summary.written and summary.status != "skipped":
                 extra = f", {summary.skipped} skipped" if summary.skipped else ""
                 print(f"   📎 {display_id}: {summary.written} supplementary file(s){extra}")

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/get_cookies.py

@@ -70,11 +70,13 @@ PROBES: Dict[str, str] = {
     "informs": "10.1287/mnsc.2023.4866",
     "science": "10.1126/science.abn1725",
     "aha": "10.1161/circulationaha.121.058901",
+    "ieee": "10.1109/embc.2014.6943774",
 }
 
 PUBLISHER_NAMES = {"wiley": "Wiley", "tandf": "Taylor & Francis", "sage": "SAGE",
                    "springer": "Springer", "hogrefe": "Hogrefe", "informs": "INFORMS",
-                   "science": "Science", "aha": "AHA Journals"}
+                   "science": "Science", "aha": "AHA Journals",
+                   "ieee": "IEEE Xplore"}
 
 CONFIG_DIR = inst.CONFIG_DIR
 ACCESS_CONFIG = inst.ACCESS_CONFIG

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/retrieval/institutional.py

@@ -448,15 +448,58 @@ def _prefix(doi: str) -> str:
     return (doi or "").split("/")[0]
 
 
+# --------------------------------------------------------------------------
+# IEEE Xplore
+# --------------------------------------------------------------------------
+# IEEE keys every article by an "arnumber" that is not derivable from the DOI
+# (10.1109/aina.2010.102 is document 5474692). Its landing pages sit behind
+# AWS WAF -- a plain client gets HTTP 202 with an empty body -- so the arnumber
+# cannot be scraped from them. doi.org's own redirect names it, though, and
+# costs one request that never reaches IEEE. The PDF endpoint,
+# stampPDF/getPDF.jsp, is not behind the WAF: without a session it redirects
+# to Xplore/login.jsp, with one it serves the PDF. Measured 2026-11-08.
+IEEE_PREFIX = "10.1109"
+IEEE_SITE = "ieee.org"
+_IEEE_ARNUMBER_RE = re.compile(r"ieeexplore\.ieee\.org/(?:abstract/)?document/(\d+)")
+_ieee_lock = threading.Lock()
+#: DOI -> arnumber. Only successful lookups are kept. Seeded with the
+#: `fetchpdf cookies` test article so the wizard never needs the network to
+#: know where IEEE lives.
+_ieee_arnumbers = {"10.1109/embc.2014.6943774": "6943774"}
+
+
+def ieee_arnumber(doi: str, timeout: float = 30) -> Optional[str]:
+    """IEEE Xplore's article number for *doi*, from doi.org's redirect; None if unknown."""
+    key = (doi or "").strip().lower()
+    with _ieee_lock:
+        if key in _ieee_arnumbers:
+            return _ieee_arnumbers[key]
+    url = f"https://doi.org/{key}"
+    try:
+        r = requests.get(url, allow_redirects=False, timeout=timeout,
+                         headers=headers_for(url, {}))
+        m = _IEEE_ARNUMBER_RE.search(r.headers.get("location") or "")
+    except requests.RequestException:
+        return None
+    if not m:
+        return None
+    with _ieee_lock:
+        _ieee_arnumbers[key] = m.group(1)
+    return m.group(1)
+
+
 def covers(doi: str) -> bool:
     """True if this route knows where the publisher keeps the DOI's PDF."""
     prefix = _prefix(doi)
-    return prefix in PUBLISHER_PDF_TEMPLATES or prefix in PUBLISHER_LANDING_SITES
+    return (prefix in PUBLISHER_PDF_TEMPLATES or prefix in PUBLISHER_LANDING_SITES
+            or prefix == IEEE_PREFIX)
 
 
 def publisher_sites(doi: str) -> List[str]:
     """The publisher sites whose cookies this DOI's fetch may use, main one first."""
     prefix = _prefix(doi)
+    if prefix == IEEE_PREFIX:
+        return [IEEE_SITE]
     if prefix in PUBLISHER_LANDING_SITES:
         return list(PUBLISHER_LANDING_SITES[prefix])
     article = publisher_article_url(doi)
@@ -469,6 +512,7 @@ def cookie_sites() -> List[str]:
              for t in PUBLISHER_ARTICLE_TEMPLATES.values()}
     for more in PUBLISHER_LANDING_SITES.values():
         sites.update(more)
+    sites.add(IEEE_SITE)
     return sorted(sites)
 
 
@@ -476,8 +520,12 @@ def publisher_article_url(doi: str) -> Optional[str]:
     """The article landing page on the publisher's own host, or None.
 
     For a publisher in PUBLISHER_LANDING_SITES that is the DOI link, which
-    redirects there: its article URLs are not derivable from the DOI.
+    redirects there: its article URLs are not derivable from the DOI. For IEEE
+    it is the Xplore document page, which costs a doi.org lookup the first time.
     """
+    if _prefix(doi) == IEEE_PREFIX:
+        n = ieee_arnumber(doi)
+        return f"https://ieeexplore.ieee.org/document/{n}" if n else None
     template = PUBLISHER_ARTICLE_TEMPLATES.get(_prefix(doi))
     if template:
         return template.format(doi=doi)
@@ -522,6 +570,9 @@ def _count_download() -> None:
 
 def publisher_pdf_urls(doi: str) -> List[str]:
     """Canonical publisher PDF URLs for a DOI, or ``[]`` if none is derivable."""
+    if _prefix(doi) == IEEE_PREFIX:
+        n = ieee_arnumber(doi)
+        return [f"https://ieeexplore.ieee.org/stampPDF/getPDF.jsp?tp=&arnumber={n}"] if n else []
     return [t.format(doi=doi) for t in PUBLISHER_PDF_TEMPLATES.get(_prefix(doi), [])]
 
 
@@ -606,6 +657,9 @@ _ELSEVIER_REDIRECT_RE = re.compile(r'name=["\']redirectURL["\'][^>]*value=["\'](
 
 
 def _is_challenge(response) -> bool:
+    """A Cloudflare challenge, or AWS WAF's (IEEE: 202 + x-amzn-waf-action)."""
+    if response.status_code == 202:
+        return bool(response.headers.get("x-amzn-waf-action"))
     return (response.status_code in (403, 429, 503)
             and "cloudflare" in (response.headers.get("server") or "").lower())
 
@@ -795,14 +849,21 @@ _FETCH_JS = """async ([u, cap]) => {
 
 
 def wait_past_challenge(page, timeout_s: float = 30.0) -> bool:
-    """Wait until the page is no longer a bot-challenge interstitial."""
+    """Wait until the page is no longer a bot-challenge interstitial.
+
+    Cloudflare names itself in the title; AWS WAF (IEEE Xplore) does not, but
+    its page carries the challenge script's globals.
+    """
+    from .blocked import WAF_CHALLENGE_MARKERS
     deadline = time.time() + timeout_s
     while time.time() < deadline:
         try:
             title = (page.title() or "").lower()
+            body = page.content()[:20000] if "ieee.org" in (page.url or "") else ""
         except Exception:      # noqa: BLE001 - mid-navigation
-            title = "just a moment"
-        if not any(t in title for t in _CHALLENGE_TITLES):
+            title, body = "just a moment", ""
+        if not any(t in title for t in _CHALLENGE_TITLES) and \
+                not any(m in body for m in WAF_CHALLENGE_MARKERS):
             return True
         page.wait_for_timeout(1000)
     return False

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/retrieval/supplement_atypon.py

@@ -74,6 +74,17 @@ def atypon_host_for(doi: str) -> Optional[str]:
     return _ATYPON_HOSTS.get(prefix)
 
 
+def note_blocked(ctx, provider: str, url: str, detail: str) -> None:
+    """Record that a provider was refused, so the manifest can say "blocked".
+
+    An empty list alone reads as "nothing listed", which claims the article has
+    no supplements. A refused listing page proves nothing of the sort.
+    """
+    scratch = getattr(ctx, "scratch", None)
+    if isinstance(scratch, dict):
+        scratch.setdefault("_provider_blocked", {})[provider] = {"url": url, "detail": detail}
+
+
 def headed_browser_available() -> bool:
     """Whether a headed Chromium can actually be launched here.
 
@@ -190,6 +201,7 @@ def fetch_atypon_supplements(ids, ctx) -> List[SupplementFile]:
                 status = response.status if response else None
                 if status != 200:
                     ctx.log(f"    atypon SI: listing page HTTP {status}")
+                    note_blocked(ctx, "atypon_suppl", listing, f"listing page HTTP {status}")
                     return []
                 page.wait_for_timeout(_SETTLE_MS)
 

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/retrieval/supplement_index.py

@@ -716,6 +716,9 @@ def _providers() -> Tuple[Tuple[str, Any], ...]:
         # records Europe PMC has already said it is withholding, which on a
         # 46-record corpus was 1, not 46.
         ("atypon_suppl", _atypon_if_withheld),                 # E19
+        # E22 after every publisher route: archived copies, only when none of
+        # them listed a publisher supplement.
+        ("wayback", _wayback_if_nothing_found),                # E22
         # E20 truly last: the only provider that costs money, and the only one
         # that benefits from knowing what all the others already found.
         ("llm_agent", _llm_agent),                             # E20
@@ -723,6 +726,29 @@ def _providers() -> Tuple[Tuple[str, Any], ...]:
     )
 
 
+#: Providers that list the publisher's own supplement files. When one of them
+#: found something, the Internet Archive has nothing to add.
+_PUBLISHER_SI_PROVIDERS = frozenset({
+    "europepmc_supplements", "pmc_s3", "elsevier_objects", "springer_esm", "plos",
+    "preprint_supplements", "apa_supplemental", "jats_declared", "atypon_suppl", "jci",
+})
+
+
+def _wayback_if_nothing_found(ids, ctx) -> List[SupplementFile]:
+    """E22: the Internet Archive's copies, as a last-ditch fallback.
+
+    Skipped as soon as any publisher-supplement provider has listed a file, so a
+    record that is already served costs the Archive no request. Repository files
+    (OSF, Zenodo, figshare) do not count: they are data deposits, and a paper
+    can have those while its publisher-hosted supplement is still missing.
+    """
+    so_far = ctx.scratch.get("enumerated_so_far") or []
+    if any(str(f.provider or "").split(":", 1)[0] in _PUBLISHER_SI_PROVIDERS for f in so_far):
+        return []
+    from .supplement_wayback import enumerate_wayback
+    return enumerate_wayback(ids, ctx)
+
+
 def _llm_agent(ids, ctx):
     """Imported at call time: a run without the flag never loads a backend."""
     from .llm_agent_retrieval import enumerate_llm_agent
@@ -750,6 +776,7 @@ PROVIDER_NAMES: Tuple[str, ...] = (
     "epmc_datalinks",
     "fulltext_scan",
     "atypon_suppl",
+    "wayback",
     "llm_agent",
     "jci",
 )
@@ -799,6 +826,17 @@ def enumerate_all(ids, ctx, providers=None) -> Tuple[List[SupplementFile], List[
         for entry in files:
             entry.provider = entry.provider or name
         found.extend(files)
+        blocked = (ctx.scratch.get("_provider_blocked") or {}).pop(name, None)
+        if blocked and not files:
+            # Refused, not empty: the supplements may well exist.
+            reports.append({
+                "name": name,
+                "status": "blocked",
+                "listed": 0,
+                "url": blocked.get("url"),
+                "reason": blocked.get("detail"),
+            })
+            continue
         reports.append({
             "name": name,
             "status": "ok" if files else "nothing_listed",

{fetchpdf-0.1.7 → fetchpdf-0.1.8}/fetchpdf/retrieval/supplement_pmc.py

@@ -60,9 +60,9 @@ def enumerate_jats_manifest(ids, ctx) -> List[SupplementFile]:
     Returns no files by design -- see the module docstring. Runs first so every
     later provider can classify against ground truth instead of a filename guess.
     """
-    if not ids.pmcid:
-        return []
-
+    # No PMCID is fine when a T1 pass already saved the publisher's XML beside
+    # the record: _jats_for reads that copy first and only needs the PMCID for
+    # the EPMC fallback.
     content = _jats_for(ids, ctx)
     if not content:
         return []
@@ -71,11 +71,12 @@ def enumerate_jats_manifest(ids, ctx) -> List[SupplementFile]:
     except ElementTree.ParseError:
         return []
 
-    # A zero count is only trustworthy from a document that actually has a body.
-    # An abstract stub or an EPMC error document parses cleanly and has no
-    # <supplementary-material> either, and the two are otherwise identical.
-    if root.find(".//{*}body") is None and root.find(".//body") is None:
-        ctx.log("    JATS manifest: no <body>, so a zero supplement count proves nothing")
+    # A zero count is only trustworthy from a JATS document that actually has a
+    # body. An abstract stub or an EPMC error document parses cleanly and has no
+    # <supplementary-material> either, and neither does a TEI or Elsevier XML
+    # saved as {stem}.xml -- those declare supplements some other way, if at all.
+    if not _is_full_jats(root):
+        ctx.log("    JATS manifest: not a full-text JATS document, so a zero supplement count proves nothing")
         return []
 
     supplements, labels, raw = _supplementary_hrefs(root)
@@ -128,6 +129,8 @@ def _jats_for(ids, ctx) -> Optional[bytes]:
     except OSError:
         pass
 
+    if not ids.pmcid:
+        return None
     response = ctx.http.get(EPMC_FULLTEXT.format(pmcid=ids.pmcid), timeout=60)
     if response.ok and response.content:
         return response.content
@@ -145,12 +148,18 @@ def declared_from_xml(content: bytes) -> Optional[Dict[str, str]]:
         root = ElementTree.fromstring(content)
     except ElementTree.ParseError:
         return None
-    if root.find(".//{*}body") is None and root.find(".//body") is None:
+    if not _is_full_jats(root):
         return None
     _, _, raw = _supplementary_hrefs(root)
     return raw
 
 
+def _is_full_jats(root) -> bool:
+    """JATS (<article-meta>) with a <body>: the only kind whose count means anything."""
+    has = lambda name: any(_tag_is(element, name) for element in root.iter())
+    return has("article-meta") and has("body")
+
+
 def _supplementary_hrefs(root) -> Tuple[Set[str], Dict[str, str], Dict[str, str]]:
     """Every href under a <supplementary-material>: basenames, labels, raw hrefs."""
     hrefs: Set[str] = set()

fetchpdf-0.1.8/fetchpdf/retrieval/supplement_wayback.py

@@ -0,0 +1,178 @@
+"""E22: publisher supplements as archived by the Internet Archive.
+
+A last-ditch fallback. Several publishers refuse every automated request for
+their supplement pages (SAGE, INFORMS, AOM and PNAS answer 403 from a bot check;
+Elsevier's file host and supp.apa.org are often unreachable too), yet the
+Wayback Machine has crawled some of those same files. Its CDX index is an open
+API, so this asks it one question per record: "which captures do you hold under
+this article's supplement URL prefix?"
+
+Measured on a 26-paper SAGE set: 5 papers had at least one archived supplement
+file, 8 had only an archived listing page, 13 had nothing. So this recovers a
+minority -- which is why it only runs when no publisher-supplement provider
+listed anything (see supplement_index._wayback_if_nothing_found).
+
+What comes back is a capture, not the live file: the manifest keeps both the
+original publisher URL and the capture timestamp so a reader can tell.
+"""
+from __future__ import annotations
+
+import re
+from typing import List, Optional
+from urllib.parse import unquote, urljoin
+
+from .supplement_index import ROLE_SUPPLEMENT, SupplementFile
+
+CDX = "https://web.archive.org/cdx/search/cdx"
+#: `id_` asks for the capture's original bytes, without the Wayback toolbar or
+#: rewritten links.
+RAW = "https://web.archive.org/web/{timestamp}id_/{url}"
+
+#: Atypon-platform hosts, by DOI prefix. Wider than supplement_atypon's table on
+#: purpose: that one lists hosts a headed browser gets through, this one lists
+#: hosts whose /doi/suppl/ URLs the Archive may hold.
+_ATYPON_HOSTS = {
+    "10.1073": "www.pnas.org",
+    "10.1126": "www.science.org",
+    "10.1146": "www.annualreviews.org",
+    "10.1177": "journals.sagepub.com",
+    "10.1080": "www.tandfonline.com",
+    "10.1287": "pubsonline.informs.org",
+    "10.5465": "journals.aom.org",
+}
+
+_MAX_FILES = 20
+#: Listing pages worth opening to find file links the prefix query missed.
+_MAX_LISTINGS = 2
+_SUPPL_HREF_RE = re.compile(r'href="([^"]*suppl_file/[^"]+)"', re.IGNORECASE)
+_WAYBACK_PREFIX_RE = re.compile(r"^(?:https?:)?//web\.archive\.org/web/\d+[a-z_]*/", re.IGNORECASE)
+_PAGE_SUFFIXES = (".html", ".htm", ".css", ".js", ".php", ".asp", ".aspx")
+
+
+def url_prefixes(ids) -> List[str]:
+    """The URL prefixes under which this article's supplements are published."""
+    doi = (getattr(ids, "doi", None) or "").strip()
+    if not doi:
+        return []
+    prefix = doi.split("/", 1)[0]
+    out = []
+    host = _ATYPON_HOSTS.get(prefix)
+    if host:
+        out.append(f"{host}/doi/suppl/{doi}")
+    pii = getattr(ids, "elsevier_pii", None)
+    if pii:
+        out.append(f"ars.els-cdn.com/content/image/1-s2.0-{pii}-mmc")
+    if prefix == "10.1037":
+        code = doi.rsplit("/", 1)[-1].lower()
+        if code:
+            out.append(f"supp.apa.org/psycarticles/supplemental/{code}/")
+    return out
+
+
+def _captures(ctx, url: str, prefix: bool) -> List[dict]:
+    """Successful captures at (or, with prefix, under) a URL; one per distinct URL."""
+    params = {
+        "url": url + "*" if prefix else url,
+        "output": "json",
+        "fl": "timestamp,original,mimetype,length",
+        "filter": "statuscode:200",
+        "collapse": "urlkey",
+        "limit": "100" if prefix else "-1",
+    }
+    # polite=False: the Archive needs no contact address, so none is sent.
+    response = ctx.http.get(CDX, params=params, timeout=60, polite=False)
+    rows = None
+    if response.ok:
+        try:
+            rows = response.json() if response.content.strip() else []
+        except ValueError:
+            rows = None   # the Archive's "temporarily offline" page, served as HTML
+    if not isinstance(rows, list):
+        # An outage is not an answer. Without this the manifest would say
+        # "nothing listed" for a record the Archive was never actually asked about.
+        from .supplement_atypon import note_blocked
+        note_blocked(ctx, "wayback", f"https://web.archive.org/web/*/{url}*",
+                     f"Internet Archive index unavailable (HTTP {response.status})")
+        return []
+    out = []
+    for row in rows[1:]:
+        if isinstance(row, list) and len(row) >= 4:
+            out.append({"timestamp": row[0], "original": row[1],
+                        "mimetype": row[2] or "", "length": row[3]})
+    return out
+
+
+def _is_page(capture: dict) -> bool:
+    name = capture["original"].split("?")[0].lower()
+    return "html" in capture["mimetype"].lower() or name.endswith(_PAGE_SUFFIXES)
+
+
+def _listed_file_urls(ctx, capture: dict) -> List[str]:
+    """Supplement-file links on an archived Atypon listing page."""
+    response = ctx.http.get(
+        RAW.format(timestamp=capture["timestamp"], url=capture["original"]),
+        timeout=60, polite=False,
+    )
+    if not response.ok:
+        return []
+    urls = []
+    for href in _SUPPL_HREF_RE.findall(response.text or ""):
+        url = urljoin(capture["original"], href.replace("&", "&"))
+        url = _WAYBACK_PREFIX_RE.sub("", url)
+        if url not in urls:
+            urls.append(url)
+    return urls
+
+
+def _name(url: str) -> str:
+    return unquote(url.split("?")[0].rstrip("/").rsplit("/", 1)[-1])
+
+
+def _as_int(value) -> Optional[int]:
+    try:
+        return int(value)
+    except (TypeError, ValueError):
+        return None
+
+
+def enumerate_wayback(ids, ctx) -> List[SupplementFile]:
+    """Archived copies of the article's publisher-hosted supplement files."""
+    files: List[SupplementFile] = []
+    seen = set()
+
+    def add(capture: dict) -> None:
+        key = capture["original"].split("://", 1)[-1].lower()
+        name = _name(capture["original"])
+        if key in seen or not name or len(files) >= _MAX_FILES:
+            return
+        seen.add(key)
+        files.append(SupplementFile(
+            name=name,
+            url=RAW.format(timestamp=capture["timestamp"], url=capture["original"]),
+            provider="wayback",
+            listing_index=len(files),
+            mimetype=capture["mimetype"] or None,
+            # CDX `length` is the compressed record size, not the file's: leave
+            # size_bytes unset rather than declare a number that will not match.
+            role=ROLE_SUPPLEMENT,
+            origin_doi=getattr(ids, "doi", None),
+            polite=False,
+            extra={"original_url": capture["original"],
+                   "wayback_timestamp": capture["timestamp"]},
+        ))
+
+    for prefix in url_prefixes(ids):
+        captures = _captures(ctx, prefix, prefix=True)
+        for capture in captures:
+            if not _is_page(capture):
+                add(capture)
+        # A listing page may name files the prefix query did not return (they
+        # were captured under another scheme or letter case). Look each one up.
+        for listing in [c for c in captures if _is_page(c)][:_MAX_LISTINGS]:
+            for url in _listed_file_urls(ctx, listing):
+                if url.split("://", 1)[-1].lower() in seen:
+                    continue
+                for capture in _captures(ctx, url, prefix=False)[-1:]:
+                    if not _is_page(capture):
+                        add(capture)
+    return files