fetchpdf 0.1.6__tar.gz → 0.1.8__tar.gz

{fetchpdf-0.1.6/fetchpdf.egg-info → fetchpdf-0.1.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fetchpdf
-Version: 0.1.6
+Version: 0.1.8
 Summary: A comprehensive tool to download PDFs from DOIs using multiple sources
 Author-email: Dan Elton <delton17@gmail.com>
 License-Expression: MIT
@@ -578,7 +578,8 @@ suppressed when the answer is already known: any flag that turns the
 supplementary pass on (`--pull-supplementary`, `--refresh-supplementary`,
 `--download-data-artifacts`, `--fetch-everything`, `--llm-agent-retrieval`), an
 explicit `--on-existing skip` or `supplement`, `--abstract-only`, or the
-goal-aware flags (`--get-xml-or-html-and-pdf`, `--upgrade-existing` on the tiered path),
+goal-aware flags (`--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`,
+`--upgrade-existing` on the tiered path),
 which do not skip records at all. Single-identifier runs never ask.
 
 What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
@@ -678,11 +679,12 @@ Twenty routes, tried in a fixed order so the numbering is reproducible:
 | Figshare / Zenodo / OSF | repository DOIs, and datasets reached through link services |
 | DataCite reverse relations | any DOI — finds deposits whose own DataCite record points at the article through a non-citation relation (e.g. `IsSupplementTo`); deposits that merely cite it are dropped |
 | Crossref component DOIs | publishers that register components |
-| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks) — recovery of last resort |
+| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks, publisher Source Data links) — recovery of last resort. Read from the PMC copy or, with no PMCID, from a publisher `{stem}.xml` saved beside the record (e.g. by `--get-xml-and-pdf`); TEI and non-JATS XML are ignored |
 | OpenAIRE ScholeXplorer | any DOI — Scholix publication→dataset/software links |
 | Europe PMC datalinks | any PMID — text-mined accessions and data citations |
 | Full-text scan | deposits the paper names in its own prose — only under `--download-data-artifacts` |
 | Atypon publisher SI | PNAS, Science, Annual Reviews, SAGE, T&F — only when Europe PMC withholds the SI or the record has no PMCID; drives a headed browser, so it needs Playwright, Chromium and a display (or Xvfb) |
+| Internet Archive (Wayback Machine) | Last-ditch fallback for publisher-hosted SI (Atypon hosts incl. SAGE, INFORMS, AOM, PNAS; Elsevier `ars.els-cdn.com` mmc files; supp.apa.org): asks the Archive's open CDX index for captures under the article's supplement URLs and downloads the archived copy. Runs only when no publisher-supplement provider listed anything; an Archive outage is reported as `blocked`, not as "none found" |
 | LLM retrieval agent | only under [`--fetch-everything`](#--fetch-everything-the-second-layer) or `--llm-agent-retrieval` |
 | JCI | `10.1172/jci…` |
 
@@ -1134,6 +1136,11 @@ the DOI link, and the PDF the landing page declares is fetched: **AACR**
 (`10.1182`), **Company of Biologists** (`10.1242`), **ERS** (`10.1183`),
 **BMJ** (`10.1136`), **Bioscientifica** (`10.1530`) and **LWW** (`10.1097`).
 
+**IEEE Xplore** (`10.1109`) keys articles by a number the DOI does not
+contain, so the route reads it from the DOI link's redirect and fetches
+`stampPDF/getPDF.jsp` with your cookies. Sign in once (or just open any IEEE
+PDF from a network your library covers) and the session cookies carry over.
+
 Several of these are open access but sit behind a Cloudflare bot check that a
 plain download cannot pass. Opening any article on the site once in your
 everyday browser earns the clearance cookie, and the next run reuses it.
@@ -1229,9 +1236,9 @@ refusal, not a bare failure:
 | `path` | str \| null | Absolute path to the artifact, or null. |
 | `format` | str | From the artifact actually written: `pdf`, `xml`, `html`, `latex`, `landing`, `suppl`, `abstract`, `text`, `other`. **A successful record is not necessarily a PDF** — with the default XML fallback a record can succeed as `xml`. |
 | `source` | str \| null | The route that produced it (`unpaywall`, `core`, `osf`, …), `existing` for a skip, `structured_fallback` when no PDF was available and structured full text was saved instead. |
-| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
+| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
 | `reasons` | list[str] | Why a record failed, or why a kept PDF could not be verified. Empty on a verified success. |
-| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf`), empty otherwise. `path` is not repeated here. |
+| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf` and its `--get-xml-and-pdf` / `--get-xml-and-html-and-pdf` variants), empty otherwise. `path` is not repeated here. |
 | `supplementary` | object | Only with `--pull-supplementary` (or a flag that implies it): `status` (`ok`, `none_found`, `partial`, `incomplete`, `error`, `skipped`), `written`, `skipped`, `manifest` (absolute path or null), `missing_declared` (files the article declares that were not obtained) and `blocked_urls` (refused to this client; open them in a browser). |
 | `figures` | object | Only with `--pull-figures`: `status`, `written`, `failed`, `refusal` and `manifest`. |
 | `images` | object | Only with `--extract-images`: `status`, `written` and `manifest`. |
@@ -1469,7 +1476,7 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
       records whose key is entitled to the XML but not the PDF, and covers
       ScienceDirect DOIs outside `10.1016/*`. It does not repeat a PDF request the
       early step already made. The XML half is skipped with `--no-xml-fallback`.
-    - With `--prioritize-xml` / `--xml-only` / `--get-xml-or-html-and-pdf` the tiered
+    - With `--prioritize-xml` / `--xml-only` / any `--get-xml-*-and-pdf` flag the tiered
       ladder asks the same endpoint itself (`elsevier_tdm` at T1, `elsevier_pdf`
       at T5, ahead of the legacy chain).
     - **Off campus?** Entitlement is decided by your IP unless you also set

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/README.md

@@ -538,7 +538,8 @@ suppressed when the answer is already known: any flag that turns the
 supplementary pass on (`--pull-supplementary`, `--refresh-supplementary`,
 `--download-data-artifacts`, `--fetch-everything`, `--llm-agent-retrieval`), an
 explicit `--on-existing skip` or `supplement`, `--abstract-only`, or the
-goal-aware flags (`--get-xml-or-html-and-pdf`, `--upgrade-existing` on the tiered path),
+goal-aware flags (`--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`,
+`--upgrade-existing` on the tiered path),
 which do not skip records at all. Single-identifier runs never ask.
 
 What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
@@ -638,11 +639,12 @@ Twenty routes, tried in a fixed order so the numbering is reproducible:
 | Figshare / Zenodo / OSF | repository DOIs, and datasets reached through link services |
 | DataCite reverse relations | any DOI — finds deposits whose own DataCite record points at the article through a non-citation relation (e.g. `IsSupplementTo`); deposits that merely cite it are dropped |
 | Crossref component DOIs | publishers that register components |
-| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks) — recovery of last resort |
+| JATS-declared URLs | absolute URLs the JATS itself declares (e.g. LWW permalinks, publisher Source Data links) — recovery of last resort. Read from the PMC copy or, with no PMCID, from a publisher `{stem}.xml` saved beside the record (e.g. by `--get-xml-and-pdf`); TEI and non-JATS XML are ignored |
 | OpenAIRE ScholeXplorer | any DOI — Scholix publication→dataset/software links |
 | Europe PMC datalinks | any PMID — text-mined accessions and data citations |
 | Full-text scan | deposits the paper names in its own prose — only under `--download-data-artifacts` |
 | Atypon publisher SI | PNAS, Science, Annual Reviews, SAGE, T&F — only when Europe PMC withholds the SI or the record has no PMCID; drives a headed browser, so it needs Playwright, Chromium and a display (or Xvfb) |
+| Internet Archive (Wayback Machine) | Last-ditch fallback for publisher-hosted SI (Atypon hosts incl. SAGE, INFORMS, AOM, PNAS; Elsevier `ars.els-cdn.com` mmc files; supp.apa.org): asks the Archive's open CDX index for captures under the article's supplement URLs and downloads the archived copy. Runs only when no publisher-supplement provider listed anything; an Archive outage is reported as `blocked`, not as "none found" |
 | LLM retrieval agent | only under [`--fetch-everything`](#--fetch-everything-the-second-layer) or `--llm-agent-retrieval` |
 | JCI | `10.1172/jci…` |
 
@@ -1094,6 +1096,11 @@ the DOI link, and the PDF the landing page declares is fetched: **AACR**
 (`10.1182`), **Company of Biologists** (`10.1242`), **ERS** (`10.1183`),
 **BMJ** (`10.1136`), **Bioscientifica** (`10.1530`) and **LWW** (`10.1097`).
 
+**IEEE Xplore** (`10.1109`) keys articles by a number the DOI does not
+contain, so the route reads it from the DOI link's redirect and fetches
+`stampPDF/getPDF.jsp` with your cookies. Sign in once (or just open any IEEE
+PDF from a network your library covers) and the session cookies carry over.
+
 Several of these are open access but sit behind a Cloudflare bot check that a
 plain download cannot pass. Opening any article on the site once in your
 everyday browser earns the clearance cookie, and the next run reuses it.
@@ -1189,9 +1196,9 @@ refusal, not a bare failure:
 | `path` | str \| null | Absolute path to the artifact, or null. |
 | `format` | str | From the artifact actually written: `pdf`, `xml`, `html`, `latex`, `landing`, `suppl`, `abstract`, `text`, `other`. **A successful record is not necessarily a PDF** — with the default XML fallback a record can succeed as `xml`. |
 | `source` | str \| null | The route that produced it (`unpaywall`, `core`, `osf`, …), `existing` for a skip, `structured_fallback` when no PDF was available and structured full text was saved instead. |
-| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
+| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--get-xml-and-pdf`, `--get-xml-and-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
 | `reasons` | list[str] | Why a record failed, or why a kept PDF could not be verified. Empty on a verified success. |
-| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf`), empty otherwise. `path` is not repeated here. |
+| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf` and its `--get-xml-and-pdf` / `--get-xml-and-html-and-pdf` variants), empty otherwise. `path` is not repeated here. |
 | `supplementary` | object | Only with `--pull-supplementary` (or a flag that implies it): `status` (`ok`, `none_found`, `partial`, `incomplete`, `error`, `skipped`), `written`, `skipped`, `manifest` (absolute path or null), `missing_declared` (files the article declares that were not obtained) and `blocked_urls` (refused to this client; open them in a browser). |
 | `figures` | object | Only with `--pull-figures`: `status`, `written`, `failed`, `refusal` and `manifest`. |
 | `images` | object | Only with `--extract-images`: `status`, `written` and `manifest`. |
@@ -1429,7 +1436,7 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
       records whose key is entitled to the XML but not the PDF, and covers
       ScienceDirect DOIs outside `10.1016/*`. It does not repeat a PDF request the
       early step already made. The XML half is skipped with `--no-xml-fallback`.
-    - With `--prioritize-xml` / `--xml-only` / `--get-xml-or-html-and-pdf` the tiered
+    - With `--prioritize-xml` / `--xml-only` / any `--get-xml-*-and-pdf` flag the tiered
       ladder asks the same endpoint itself (`elsevier_tdm` at T1, `elsevier_pdf`
       at T5, ahead of the legacy chain).
     - **Off campus?** Entitlement is decided by your IP unless you also set

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/__init__.py

@@ -15,7 +15,7 @@ from .fetch_metadata_from_doi import fetch_metadata_from_doi
 # behavioural difference, it is the same object.
 fetch_pdf_from_doi = fetch_pdf
 
-__version__ = "0.1.6"
+__version__ = "0.1.8"
 __all__ = [
     "fetch_pdf",
     "fetch_pdf_from_doi",  # deprecated alias for fetch_pdf

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/fetchpdf.py

@@ -5756,7 +5756,7 @@ def _append_missing_si_to_report(output_dir, si_statuses, run_timestamp, lock,
     for display_id, summary in (si_statuses or {}).items():
         status = getattr(summary, "status", "")
         blocked_urls = list(getattr(summary, "blocked_urls", None) or [])
-        if status not in ("partial", "incomplete"):
+        if status not in ("partial", "incomplete", "blocked"):
             continue
         missing = ", ".join(getattr(summary, "missing_declared", None) or []) or "—"
         # A withheld-but-existing supplement is the click-through candidate:
@@ -6426,6 +6426,11 @@ def _batch_fetch_pdfs_inner(dois, output_dir, email=None, verbose=False, delay=0
                 # point of the gate is that this cannot pass silently.
                 names = ", ".join(summary.missing_declared) or "unknown"
                 print(f"   ⚠️  {display_id}: SI INCOMPLETE — missing: {names}")
+            elif summary.status == "blocked":
+                # Not "none found": a source refused or was down, so the paper
+                # may well have supplements. Printed regardless of verbosity.
+                print(f"   ⚠️  {display_id}: SI BLOCKED — a supplement source refused "
+                      f"or was unavailable ({summary.detail or 'no detail'})")
             elif summary.written and summary.status != "skipped":
                 extra = f", {summary.skipped} skipped" if summary.skipped else ""
                 print(f"   📎 {display_id}: {summary.written} supplementary file(s){extra}")

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/get_cookies.py

@@ -70,11 +70,13 @@ PROBES: Dict[str, str] = {
     "informs": "10.1287/mnsc.2023.4866",
     "science": "10.1126/science.abn1725",
     "aha": "10.1161/circulationaha.121.058901",
+    "ieee": "10.1109/embc.2014.6943774",
 }
 
 PUBLISHER_NAMES = {"wiley": "Wiley", "tandf": "Taylor & Francis", "sage": "SAGE",
                    "springer": "Springer", "hogrefe": "Hogrefe", "informs": "INFORMS",
-                   "science": "Science", "aha": "AHA Journals"}
+                   "science": "Science", "aha": "AHA Journals",
+                   "ieee": "IEEE Xplore"}
 
 CONFIG_DIR = inst.CONFIG_DIR
 ACCESS_CONFIG = inst.ACCESS_CONFIG

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/retrieval/institutional.py

@@ -448,15 +448,58 @@ def _prefix(doi: str) -> str:
     return (doi or "").split("/")[0]
 
 
+# --------------------------------------------------------------------------
+# IEEE Xplore
+# --------------------------------------------------------------------------
+# IEEE keys every article by an "arnumber" that is not derivable from the DOI
+# (10.1109/aina.2010.102 is document 5474692). Its landing pages sit behind
+# AWS WAF -- a plain client gets HTTP 202 with an empty body -- so the arnumber
+# cannot be scraped from them. doi.org's own redirect names it, though, and
+# costs one request that never reaches IEEE. The PDF endpoint,
+# stampPDF/getPDF.jsp, is not behind the WAF: without a session it redirects
+# to Xplore/login.jsp, with one it serves the PDF. Measured 2026-11-08.
+IEEE_PREFIX = "10.1109"
+IEEE_SITE = "ieee.org"
+_IEEE_ARNUMBER_RE = re.compile(r"ieeexplore\.ieee\.org/(?:abstract/)?document/(\d+)")
+_ieee_lock = threading.Lock()
+#: DOI -> arnumber. Only successful lookups are kept. Seeded with the
+#: `fetchpdf cookies` test article so the wizard never needs the network to
+#: know where IEEE lives.
+_ieee_arnumbers = {"10.1109/embc.2014.6943774": "6943774"}
+
+
+def ieee_arnumber(doi: str, timeout: float = 30) -> Optional[str]:
+    """IEEE Xplore's article number for *doi*, from doi.org's redirect; None if unknown."""
+    key = (doi or "").strip().lower()
+    with _ieee_lock:
+        if key in _ieee_arnumbers:
+            return _ieee_arnumbers[key]
+    url = f"https://doi.org/{key}"
+    try:
+        r = requests.get(url, allow_redirects=False, timeout=timeout,
+                         headers=headers_for(url, {}))
+        m = _IEEE_ARNUMBER_RE.search(r.headers.get("location") or "")
+    except requests.RequestException:
+        return None
+    if not m:
+        return None
+    with _ieee_lock:
+        _ieee_arnumbers[key] = m.group(1)
+    return m.group(1)
+
+
 def covers(doi: str) -> bool:
     """True if this route knows where the publisher keeps the DOI's PDF."""
     prefix = _prefix(doi)
-    return prefix in PUBLISHER_PDF_TEMPLATES or prefix in PUBLISHER_LANDING_SITES
+    return (prefix in PUBLISHER_PDF_TEMPLATES or prefix in PUBLISHER_LANDING_SITES
+            or prefix == IEEE_PREFIX)
 
 
 def publisher_sites(doi: str) -> List[str]:
     """The publisher sites whose cookies this DOI's fetch may use, main one first."""
     prefix = _prefix(doi)
+    if prefix == IEEE_PREFIX:
+        return [IEEE_SITE]
     if prefix in PUBLISHER_LANDING_SITES:
         return list(PUBLISHER_LANDING_SITES[prefix])
     article = publisher_article_url(doi)
@@ -469,6 +512,7 @@ def cookie_sites() -> List[str]:
              for t in PUBLISHER_ARTICLE_TEMPLATES.values()}
     for more in PUBLISHER_LANDING_SITES.values():
         sites.update(more)
+    sites.add(IEEE_SITE)
     return sorted(sites)
 
 
@@ -476,8 +520,12 @@ def publisher_article_url(doi: str) -> Optional[str]:
     """The article landing page on the publisher's own host, or None.
 
     For a publisher in PUBLISHER_LANDING_SITES that is the DOI link, which
-    redirects there: its article URLs are not derivable from the DOI.
+    redirects there: its article URLs are not derivable from the DOI. For IEEE
+    it is the Xplore document page, which costs a doi.org lookup the first time.
     """
+    if _prefix(doi) == IEEE_PREFIX:
+        n = ieee_arnumber(doi)
+        return f"https://ieeexplore.ieee.org/document/{n}" if n else None
     template = PUBLISHER_ARTICLE_TEMPLATES.get(_prefix(doi))
     if template:
         return template.format(doi=doi)
@@ -522,6 +570,9 @@ def _count_download() -> None:
 
 def publisher_pdf_urls(doi: str) -> List[str]:
     """Canonical publisher PDF URLs for a DOI, or ``[]`` if none is derivable."""
+    if _prefix(doi) == IEEE_PREFIX:
+        n = ieee_arnumber(doi)
+        return [f"https://ieeexplore.ieee.org/stampPDF/getPDF.jsp?tp=&arnumber={n}"] if n else []
     return [t.format(doi=doi) for t in PUBLISHER_PDF_TEMPLATES.get(_prefix(doi), [])]
 
 
@@ -606,6 +657,9 @@ _ELSEVIER_REDIRECT_RE = re.compile(r'name=["\']redirectURL["\'][^>]*value=["\'](
 
 
 def _is_challenge(response) -> bool:
+    """A Cloudflare challenge, or AWS WAF's (IEEE: 202 + x-amzn-waf-action)."""
+    if response.status_code == 202:
+        return bool(response.headers.get("x-amzn-waf-action"))
     return (response.status_code in (403, 429, 503)
             and "cloudflare" in (response.headers.get("server") or "").lower())
 
@@ -795,14 +849,21 @@ _FETCH_JS = """async ([u, cap]) => {
 
 
 def wait_past_challenge(page, timeout_s: float = 30.0) -> bool:
-    """Wait until the page is no longer a bot-challenge interstitial."""
+    """Wait until the page is no longer a bot-challenge interstitial.
+
+    Cloudflare names itself in the title; AWS WAF (IEEE Xplore) does not, but
+    its page carries the challenge script's globals.
+    """
+    from .blocked import WAF_CHALLENGE_MARKERS
     deadline = time.time() + timeout_s
     while time.time() < deadline:
         try:
             title = (page.title() or "").lower()
+            body = page.content()[:20000] if "ieee.org" in (page.url or "") else ""
         except Exception:      # noqa: BLE001 - mid-navigation
-            title = "just a moment"
-        if not any(t in title for t in _CHALLENGE_TITLES):
+            title, body = "just a moment", ""
+        if not any(t in title for t in _CHALLENGE_TITLES) and \
+                not any(m in body for m in WAF_CHALLENGE_MARKERS):
             return True
         page.wait_for_timeout(1000)
     return False

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/retrieval/pdf_identity.py

@@ -160,6 +160,44 @@ PAGE_COUNT_MIN_SPAN = 2
 
 _PAGE_RANGE_RE = re.compile(r"^\s*(?:[A-Za-z]*)(\d+)\s*(?:[-–—]\s*(?:[A-Za-z]*)(\d+))?\s*$")
 
+#: S1 on its own is not enough when the front pages are legible and do not carry
+#: the title (issue #5). The requested DOI is printed by the paper itself, but
+#: also by a paper that CITES it and by the article's own supplementary file:
+#: fetch_pdf("10.1037/0022-3514.74.6.1464") wrote 10.5964/meth.7155, whose
+#: abstract reads "(IAT; Greenwald et al., 1998. DOI: 10.1037/0022-3514.74.6.1464)",
+#: and the Supplementary Information PDF for 10.1038/nature08833 prints
+#: "doi: 10.1038/nature08833" on page 1. Measured over 121 cached PDFs whose DOI
+#: was found in the text, scoring the Crossref title against the front two pages:
+#: the citing paper 0.52, the supplement 0.64, a line-numbered medRxiv preprint
+#: (correct) 0.76, the other 118 correct papers >= 0.97. 0.70 sits in the gap; it
+#: is narrow, so a page too short to judge (LEGIBLE_FRONT_CHARS) keeps the DOI
+#: verdict rather than risk refusing a correct scan.
+DOI_TITLE_PAGES = 2
+DOI_TITLE_MIN = 0.70
+
+#: The same DOI printed inside an author-year citation -- "Greenwald et al.,
+#: 1998. DOI: 10.1037/...", "(Smith & Jones, 2020, https://doi.org/...)" -- is a
+#: reference to the paper, not the paper's own masthead. Matched on the text
+#: just before the DOI, so it also works when no title is known to compare.
+_CITED_BEFORE_DOI_RE = re.compile(
+    r"(?:et\s+al\.?|&\s*[a-z][a-z'\-]+|\band\s+[a-z][a-z'\-]+)"   # et al. / & Name / and Name
+    r"[\s,]*\(?\s*(?:1[89]|20)\d{2}[a-z]?\s*\)?"                     # , 1998 / (1998) / 2020a
+    r"\s*[.,;:]?\s*"
+    r"(?:doi\s*[:.]?\s*|(?:https?\s*:\s*//\s*)?(?:dx\s*\.\s*)?doi\s*\.\s*org\s*/\s*)?$",
+    re.IGNORECASE,
+)
+#: How far before the DOI the author-year pattern may end, and how far back a
+#: "cite this article" line may start. A paper's own front matter often prints
+#: a suggested citation in exactly the cited shape ("Cite this article as: Smith
+#: et al., 2021. https://doi.org/..."), and that is the paper's own DOI.
+CITED_CONTEXT_CHARS = 80
+OWN_CITATION_LINE_CHARS = 400
+_OWN_CITATION_RE = re.compile(
+    r"cite\s+(?:this|the)\s+(?:article|paper|chapter|work|preprint)|please\s+cite"
+    r"|how\s+to\s+cite|citation\s*:|recommended\s+citation|cite\s+as",
+    re.IGNORECASE,
+)
+
 #: Embedded titles that carry no identity: a converter's leftovers, a filename,
 #: a template default. Present in a large minority of real PDFs, and treating
 #: them as "declares a different article" would reject correct files.
@@ -279,11 +317,24 @@ def verify_pdf_identity(source, doi: Optional[str] = None,
 
     # S1: the requested DOI, printed on the paper. True of essentially every
     # version of record, and squashing means a line break inside it is harmless.
+    # Not on its own, though, when the DOI is only cited, or when the front pages
+    # are legible and do not carry the title (DOI_TITLE_MIN): then the later
+    # signals decide, and a refusal says the DOI was seen and why it did not count.
+    doi_discounted = ""
     if doi:
         wanted = squash(normalise_doi(doi))
         if wanted and wanted in squashed_page:
-            return truncated or IdentityVerdict(
-                VERIFIED, "requested DOI found in the PDF text", ["doi-in-text"])
+            if _doi_only_cited(text, doi):
+                doi_discounted = "only inside a citation"
+            elif title:
+                two_pages = squash(reader.text(DOI_TITLE_PAGES) or text)
+                fraction = matched_fraction(squash(title), two_pages)
+                if len(two_pages) >= LEGIBLE_FRONT_CHARS and fraction < DOI_TITLE_MIN:
+                    doi_discounted = ("but its front pages carry only {:.0%} of the "
+                                      "title".format(fraction))
+            if not doi_discounted:
+                return truncated or IdentityVerdict(
+                    VERIFIED, "requested DOI found in the PDF text", ["doi-in-text"])
 
     # S1b: the arXiv id, stamped down the margin of every PDF arXiv serves. It
     # is S1 for a class of record that S1 cannot reach and neither can S2-S4:
@@ -410,10 +461,23 @@ def verify_pdf_identity(source, doi: Optional[str] = None,
                 "so it cannot be checked against {}".format(
                     len(squashed_page), DOI_PAGES, doi or "the requested record"),
             )
+        if doi_discounted:
+            return IdentityVerdict(
+                WRONG,
+                'PDF prints the requested DOI {}, so it is not taken as "{}" '
+                "(likely a paper citing it, or its supplement)".format(
+                    doi_discounted, _clip(title)),
+            )
         return IdentityVerdict(
             WRONG,
             'PDF does not contain the requested DOI or the title "{}"'.format(_clip(title)),
         )
+    if doi_discounted:
+        return IdentityVerdict(
+            NO_REFERENCE,
+            "PDF prints {} {}, and no title is known to check it against".format(
+                doi or "the requested DOI", doi_discounted),
+        )
     return IdentityVerdict(
         NO_REFERENCE,
         "no title known for {} and its DOI is not printed in the PDF, so there "
@@ -630,6 +694,32 @@ def _is_generic_title(embedded: str) -> bool:
     return bool(_GENERIC_TITLE_RE.match(stripped) or _FILENAME_TITLE_RE.match(stripped))
 
 
+def _doi_only_cited(text: str, doi: str) -> bool:
+    """Whether every place `doi` is printed in `text` is inside a citation.
+
+    True only when the DOI appears at least once and each appearance follows an
+    author-year citation ("et al., 1998. DOI:") with no "cite this article" line
+    before it. A DOI printed once in a masthead and once in a citation is the
+    paper's own, so one non-citation appearance is enough to say False.
+    """
+    wanted = normalise_doi(doi)
+    if not wanted:
+        return False
+    # Line wraps land anywhere inside a DOI, so allow whitespace between its characters.
+    pattern = re.compile(r"\s*".join(re.escape(c) for c in wanted), re.IGNORECASE)
+    hits = list(pattern.finditer(text))
+    if not hits:
+        return False
+    for hit in hits:
+        before = text[max(0, hit.start() - CITED_CONTEXT_CHARS):hit.start()]
+        if not _CITED_BEFORE_DOI_RE.search(before):
+            return False
+        line_window = text[max(0, hit.start() - OWN_CITATION_LINE_CHARS):hit.start()]
+        if _OWN_CITATION_RE.search(line_window):
+            return False
+    return True
+
+
 def _clip(text: str, limit: int = 70) -> str:
     text = " ".join((text or "").split())
     return text if len(text) <= limit else text[:limit - 1] + "…"

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/retrieval/supplement_atypon.py

@@ -74,6 +74,17 @@ def atypon_host_for(doi: str) -> Optional[str]:
     return _ATYPON_HOSTS.get(prefix)
 
 
+def note_blocked(ctx, provider: str, url: str, detail: str) -> None:
+    """Record that a provider was refused, so the manifest can say "blocked".
+
+    An empty list alone reads as "nothing listed", which claims the article has
+    no supplements. A refused listing page proves nothing of the sort.
+    """
+    scratch = getattr(ctx, "scratch", None)
+    if isinstance(scratch, dict):
+        scratch.setdefault("_provider_blocked", {})[provider] = {"url": url, "detail": detail}
+
+
 def headed_browser_available() -> bool:
     """Whether a headed Chromium can actually be launched here.
 
@@ -190,6 +201,7 @@ def fetch_atypon_supplements(ids, ctx) -> List[SupplementFile]:
                 status = response.status if response else None
                 if status != 200:
                     ctx.log(f"    atypon SI: listing page HTTP {status}")
+                    note_blocked(ctx, "atypon_suppl", listing, f"listing page HTTP {status}")
                     return []
                 page.wait_for_timeout(_SETTLE_MS)
 

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/retrieval/supplement_index.py

@@ -716,6 +716,9 @@ def _providers() -> Tuple[Tuple[str, Any], ...]:
         # records Europe PMC has already said it is withholding, which on a
         # 46-record corpus was 1, not 46.
         ("atypon_suppl", _atypon_if_withheld),                 # E19
+        # E22 after every publisher route: archived copies, only when none of
+        # them listed a publisher supplement.
+        ("wayback", _wayback_if_nothing_found),                # E22
         # E20 truly last: the only provider that costs money, and the only one
         # that benefits from knowing what all the others already found.
         ("llm_agent", _llm_agent),                             # E20
@@ -723,6 +726,29 @@ def _providers() -> Tuple[Tuple[str, Any], ...]:
     )
 
 
+#: Providers that list the publisher's own supplement files. When one of them
+#: found something, the Internet Archive has nothing to add.
+_PUBLISHER_SI_PROVIDERS = frozenset({
+    "europepmc_supplements", "pmc_s3", "elsevier_objects", "springer_esm", "plos",
+    "preprint_supplements", "apa_supplemental", "jats_declared", "atypon_suppl", "jci",
+})
+
+
+def _wayback_if_nothing_found(ids, ctx) -> List[SupplementFile]:
+    """E22: the Internet Archive's copies, as a last-ditch fallback.
+
+    Skipped as soon as any publisher-supplement provider has listed a file, so a
+    record that is already served costs the Archive no request. Repository files
+    (OSF, Zenodo, figshare) do not count: they are data deposits, and a paper
+    can have those while its publisher-hosted supplement is still missing.
+    """
+    so_far = ctx.scratch.get("enumerated_so_far") or []
+    if any(str(f.provider or "").split(":", 1)[0] in _PUBLISHER_SI_PROVIDERS for f in so_far):
+        return []
+    from .supplement_wayback import enumerate_wayback
+    return enumerate_wayback(ids, ctx)
+
+
 def _llm_agent(ids, ctx):
     """Imported at call time: a run without the flag never loads a backend."""
     from .llm_agent_retrieval import enumerate_llm_agent
@@ -750,6 +776,7 @@ PROVIDER_NAMES: Tuple[str, ...] = (
     "epmc_datalinks",
     "fulltext_scan",
     "atypon_suppl",
+    "wayback",
     "llm_agent",
     "jci",
 )
@@ -799,6 +826,17 @@ def enumerate_all(ids, ctx, providers=None) -> Tuple[List[SupplementFile], List[
         for entry in files:
             entry.provider = entry.provider or name
         found.extend(files)
+        blocked = (ctx.scratch.get("_provider_blocked") or {}).pop(name, None)
+        if blocked and not files:
+            # Refused, not empty: the supplements may well exist.
+            reports.append({
+                "name": name,
+                "status": "blocked",
+                "listed": 0,
+                "url": blocked.get("url"),
+                "reason": blocked.get("detail"),
+            })
+            continue
         reports.append({
             "name": name,
             "status": "ok" if files else "nothing_listed",

{fetchpdf-0.1.6 → fetchpdf-0.1.8}/fetchpdf/retrieval/supplement_pmc.py

@@ -60,9 +60,9 @@ def enumerate_jats_manifest(ids, ctx) -> List[SupplementFile]:
     Returns no files by design -- see the module docstring. Runs first so every
     later provider can classify against ground truth instead of a filename guess.
     """
-    if not ids.pmcid:
-        return []
-
+    # No PMCID is fine when a T1 pass already saved the publisher's XML beside
+    # the record: _jats_for reads that copy first and only needs the PMCID for
+    # the EPMC fallback.
     content = _jats_for(ids, ctx)
     if not content:
         return []
@@ -71,11 +71,12 @@ def enumerate_jats_manifest(ids, ctx) -> List[SupplementFile]:
     except ElementTree.ParseError:
         return []
 
-    # A zero count is only trustworthy from a document that actually has a body.
-    # An abstract stub or an EPMC error document parses cleanly and has no
-    # <supplementary-material> either, and the two are otherwise identical.
-    if root.find(".//{*}body") is None and root.find(".//body") is None:
-        ctx.log("    JATS manifest: no <body>, so a zero supplement count proves nothing")
+    # A zero count is only trustworthy from a JATS document that actually has a
+    # body. An abstract stub or an EPMC error document parses cleanly and has no
+    # <supplementary-material> either, and neither does a TEI or Elsevier XML
+    # saved as {stem}.xml -- those declare supplements some other way, if at all.
+    if not _is_full_jats(root):
+        ctx.log("    JATS manifest: not a full-text JATS document, so a zero supplement count proves nothing")
         return []
 
     supplements, labels, raw = _supplementary_hrefs(root)
@@ -128,6 +129,8 @@ def _jats_for(ids, ctx) -> Optional[bytes]:
     except OSError:
         pass
 
+    if not ids.pmcid:
+        return None
     response = ctx.http.get(EPMC_FULLTEXT.format(pmcid=ids.pmcid), timeout=60)
     if response.ok and response.content:
         return response.content
@@ -145,12 +148,18 @@ def declared_from_xml(content: bytes) -> Optional[Dict[str, str]]:
         root = ElementTree.fromstring(content)
     except ElementTree.ParseError:
         return None
-    if root.find(".//{*}body") is None and root.find(".//body") is None:
+    if not _is_full_jats(root):
         return None
     _, _, raw = _supplementary_hrefs(root)
     return raw
 
 
+def _is_full_jats(root) -> bool:
+    """JATS (<article-meta>) with a <body>: the only kind whose count means anything."""
+    has = lambda name: any(_tag_is(element, name) for element in root.iter())
+    return has("article-meta") and has("body")
+
+
 def _supplementary_hrefs(root) -> Tuple[Set[str], Dict[str, str], Dict[str, str]]:
     """Every href under a <supplementary-material>: basenames, labels, raw hrefs."""
     hrefs: Set[str] = set()