fetchpdf 0.1.4__tar.gz → 0.1.6__tar.gz

{fetchpdf-0.1.4/fetchpdf.egg-info → fetchpdf-0.1.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: fetchpdf
-Version: 0.1.4
+Version: 0.1.6
 Summary: A comprehensive tool to download PDFs from DOIs using multiple sources
 Author-email: Dan Elton <delton17@gmail.com>
 License-Expression: MIT
@@ -30,6 +30,7 @@ Provides-Extra: better-text-images
 Requires-Dist: pymupdf>=1.24; extra == "better-text-images"
 Provides-Extra: render
 Requires-Dist: markdown-it-py>=3.0; extra == "render"
+Requires-Dist: Pillow>=9.0; extra == "render"
 Provides-Extra: dev
 Requires-Dist: pytest>=7.0.0; extra == "dev"
 Requires-Dist: pytest-xdist>=3.0; extra == "dev"
@@ -54,7 +55,7 @@ A comprehensive Python package to download academic papers (PDFs) from DOIs (or
 - [Batch Processing & Parallel Execution](#batch-processing--parallel-execution)
 - [Format-Prioritized Retrieval](#format-prioritized-retrieval)
 - [Supplementary Material](#supplementary-material)
-- [`--pull-everything`: the second layer](#--pull-everything-the-second-layer)
+- [`--fetch-everything`: the second layer](#--fetch-everything-the-second-layer)
 - [Figure Images](#figure-images)
 - [Institutional access](#institutional-access-via-cookies-ie-using-your-librarys-subscriptions) — papers your library subscribes to (`fetchpdf fetchcookies`)
 - [Calling fetchpdf from a script or an agent](#calling-fetchpdf-from-a-script-or-an-agent) — `--json`
@@ -84,7 +85,7 @@ A comprehensive Python package to download academic papers (PDFs) from DOIs (or
 - 🖼️ **[Figure Images](#figure-images)**: `--pull-figures` fetches the article's published figures from PMC (or the Elsevier object API) with a manifest of labels, captions and hashes
 - 🖨️ **[XML to HTML / PDF](#rendering-the-xml-as-html-or-pdf)**: `--generate-html-from-xml` / `--generate-pdf-from-xml` render a retrieved XML as one self-contained page, figures and tables embedded
 - 🔗 **[Linked Datasets & Code](#linked-datasets-and-code-stem_linked_artifactsjson)**: the same pass discovers each paper's external datasets/software via ScholeXplorer, Europe PMC and DataCite, downloads ownership-confirmed deposits from figshare/Zenodo/OSF/Dryad/Dataverse, and records every link in a sidecar
-- 🤖 **[LLM-assisted retrieval](#--pull-everything-the-second-layer)**: `--pull-everything` adds a second layer — a model reads the paper itself and goes after the SI and datasets the APIs missed
+- 🤖 **[LLM-assisted retrieval](#--fetch-everything-the-second-layer)**: `--fetch-everything` adds a second layer — a model reads the paper itself and goes after the SI and datasets the APIs missed
 - 🔐 **[Institutional access](#institutional-access-via-cookies-ie-using-your-librarys-subscriptions)** (opt-in): a one-time `fetchpdf fetchcookies` lets fetchpdf download papers your library subscribes to
 - 🚀 **Batch Processing**: Process multiple DOIs/PMIDs from CSV files or lists
 - ⚡ **Parallel Execution**: Download multiple papers simultaneously with configurable workers
@@ -105,7 +106,7 @@ pip install -e .
 
 ### Optional: Install Playwright and Chromium for Browser Automation
 
-Install Playwright and Chromium only if you want to try additional scrapers enabled by `--add-playwright`. The scapers are for SSRN, OSF (if direct download fails), Wiley 10.1111/ DOIs, HTML table retrieval (second route), and Atypon supplementary material (PNAS, Science, SAGE, T&F, Annual Reviews).
+Install Playwright and Chromium to enable additional browser-based scrapers. They are used automatically once Playwright is installed (pass `--no-playwright` to skip them). The scrapers are for SSRN, OSF (if direct download fails), Wiley 10.1111/ DOIs, HTML table retrieval (second route), and Atypon supplementary material (PNAS, Science, SAGE, T&F, Annual Reviews).
 
 ```bash
 pip install -e ".[browser]"   # or: pip install playwright
@@ -140,9 +141,10 @@ SERPAPI_API_KEY=your_serpapi_key           # Optional Google Scholar PDF fallbac
 
 # Optional — publisher-specific
 ELSEVIER_TDM_API_KEY=your_elsevier_key      # Elsevier text/data-mining access
+ELSEVIER_INSTTOKEN=your_inst_token          # Elsevier institutional token (needs the key above)
 SCOPUS_API_KEY=your_scopus_key              # Scopus abstract lookup during identifier resolution
 
-# Optional — only for --pull-everything --llm-backend openrouter
+# Optional — only for --fetch-everything --llm-backend openrouter
 OPENROUTER_API_KEY=your_openrouter_key      # the retrieval agent's second backend
 
 # Required by fetchpdf-pubpeer — PubPeer rejects keyless requests
@@ -193,7 +195,7 @@ papers your library subscribes to. See
 By default a record yields one file: the best PDF the source chain finds. Three
 flags widen that.
 
-1. **Structured full text** — `--get-xml-or-html` keeps a JATS/TEI XML (else
+1. **Structured full text** — `--get-xml-or-html-and-pdf` keeps a JATS/TEI XML (else
    publisher HTML) copy alongside the PDF; `--to-markdown` also renders it to
    `{stem}_from_xml.md` (or `{stem}_from_html.md`). XML and HTML are great
    formats for using with AI (LLMs): complex tables usually survive better than
@@ -213,7 +215,7 @@ flags widen that.
 
 ```bash
 fetchpdf papers.csv -o ./out -w 4 \
-    --get-xml-or-html --to-markdown \
+    --get-xml-or-html-and-pdf --to-markdown \
     --download-data-artifacts \
     --make-subfolder --provenance
 ```
@@ -442,10 +444,10 @@ fetchpdf papers.csv -o ./out --xml-html-only    # XML or publisher HTML
 fetchpdf papers.csv -o ./out --xml-only         # XML, nothing else
 
 # A structured copy AND the PDF for every record
-fetchpdf papers.csv -o ./out --get-xml-or-html
+fetchpdf papers.csv -o ./out --get-xml-or-html-and-pdf
 
 # ...and a {stem}_from_xml.md / _from_html.md for each XML/HTML artifact
-fetchpdf papers.csv -o ./out --get-xml-or-html --to-markdown
+fetchpdf papers.csv -o ./out --get-xml-or-html-and-pdf --to-markdown
 
 # Convert artifacts you already have, without re-downloading
 fetchpdf-md ./out
@@ -463,7 +465,7 @@ supplement keeps its own extension, e.g. `{stem}.xlsx`), `{stem}.pdf`,
 `{stem}.txt` and `{stem}.landing.html` — plus `{stem}_from_xml.md` /
 `{stem}_from_html.md` under `--to-markdown`.
 
-`--get-xml-or-html` doubles as a backfill: pointed at a directory of existing
+`--get-xml-or-html-and-pdf` doubles as a backfill: pointed at a directory of existing
 PDFs it fetches only the missing structured half and re-downloads nothing.
 
 ### Choosing a flag
@@ -473,7 +475,9 @@ PDFs it fetches only the missing structured half and re-downloads nothing.
 | `--prioritize-xml` | everything, best format first | yes |
 | `--xml-html-only` | T1 XML, T2 HTML | **no** — record fails |
 | `--xml-only` | T1 XML | **no** — record fails |
-| `--get-xml-or-html` | structured **and** PDF, both kept | yes, and keeps it |
+| `--get-xml-or-html-and-pdf` | structured **and** PDF, both kept | yes, and keeps it |
+| `--get-xml-and-pdf` | XML where available **and** PDF where available, both kept | yes, and keeps it |
+| `--get-xml-and-html-and-pdf` | XML, publisher HTML and PDF, each to the extent available | yes, and keeps it |
 
 `--xml-only` and `--xml-html-only` both imply `--prioritize-xml`. Given both,
 `--xml-only` is stricter and wins (with a warning).
@@ -572,9 +576,9 @@ unless stdin and stdout are both terminals** — pipes, cron and CI get a one-li
 count and today's behaviour, so no unattended run can block on it. It is also
 suppressed when the answer is already known: any flag that turns the
 supplementary pass on (`--pull-supplementary`, `--refresh-supplementary`,
-`--download-data-artifacts`, `--pull-everything`, `--llm-agent-retrieval`), an
+`--download-data-artifacts`, `--fetch-everything`, `--llm-agent-retrieval`), an
 explicit `--on-existing skip` or `supplement`, `--abstract-only`, or the
-goal-aware flags (`--get-xml-or-html`, `--upgrade-existing` on the tiered path),
+goal-aware flags (`--get-xml-or-html-and-pdf`, `--upgrade-existing` on the tiered path),
 which do not skip records at all. Single-identifier runs never ask.
 
 What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
@@ -590,6 +594,12 @@ retrieved, the PDF is kept and reported as unverified.
 When a PDF is unavailable, FetchPDF tries XML or HTML full text automatically.
 Use `--no-xml-fallback` to disable this fallback.
 
+A record that comes back as XML only is filed in its own folder,
+`{output_dir}/{stem}/`, with its figure images in `{stem}_figures/` and a
+rendering at `{stem}_from_xml.pdf` (figures embedded; a rendering of the XML,
+not the publisher's PDF). A later flat run finds it there. Use
+`--no-xml-fallback-extras` to leave the bare `.xml` instead.
+
 Publisher HTML (T2) uses lxml, which is installed with fetchpdf.
 
 Advanced users can customize retrieval order, source capabilities, and host rate
@@ -673,7 +683,7 @@ Twenty routes, tried in a fixed order so the numbering is reproducible:
 | Europe PMC datalinks | any PMID — text-mined accessions and data citations |
 | Full-text scan | deposits the paper names in its own prose — only under `--download-data-artifacts` |
 | Atypon publisher SI | PNAS, Science, Annual Reviews, SAGE, T&F — only when Europe PMC withholds the SI or the record has no PMCID; drives a headed browser, so it needs Playwright, Chromium and a display (or Xvfb) |
-| LLM retrieval agent | only under [`--pull-everything`](#--pull-everything-the-second-layer) or `--llm-agent-retrieval` |
+| LLM retrieval agent | only under [`--fetch-everything`](#--fetch-everything-the-second-layer) or `--llm-agent-retrieval` |
 | JCI | `10.1172/jci…` |
 
 Wiley, ACS and MDPI serve HTTP 403 to plain requests and are not yet covered;
@@ -723,7 +733,7 @@ Supplementary files never appear in `failed_dois.csv`, in `source_tracking.csv`,
 or in the format-composition tally; the missing-PDFs report lists gaps in them
 only in its own separate section. The same holds for linked datasets,
 [figures](#figure-images) and the
-[`--pull-everything`](#--pull-everything-the-second-layer) agent.
+[`--fetch-everything`](#--fetch-everything-the-second-layer) agent.
 
 ### Linked datasets and code: `{stem}_linked_artifacts.json`
 
@@ -812,18 +822,19 @@ GitLab repositories, and the institutional-repository long tail.
 Adding an enumerator is the same small pattern as `enumerate_dryad` /
 `enumerate_dataverse` in `fetchpdf/retrieval/supplement_index.py`.
 
-## `--pull-everything`: the second layer
+## `--fetch-everything`: the second layer
 
 Everything above is layer 1 — APIs, indexes and enumerators. It is very good at
 what publishers and repositories *declare*, and it cannot read. Layer 2 is a
 model that reads the paper itself and goes after what layer 1 did not get.
 
 ```bash
-fetchpdf papers.csv -o ./out --get-xml-or-html --make-subfolder --provenance \
-         --pull-everything
+fetchpdf papers.csv -o ./out --make-subfolder --provenance --fetch-everything
 ```
 
-`--pull-everything` implies `--pull-supplementary` and
+`--fetch-everything` implies `--get-xml-or-html-and-pdf` (XML, else HTML, beside the PDF
+where one exists), `--pull-figures` (the publisher's figure images beside the
+XML), `--pull-supplementary` and
 `--download-data-artifacts`, turns on adjudication of the deterministic scan's
 uncertain candidates, and adds the retrieval agent. Files land exactly where
 they always did: supplements as numbered siblings of the PDF, linked deposits
@@ -895,7 +906,7 @@ backend has no general-purpose harness to lock down, because its two tools are
 ours, so it downloads directly.
 
 `--llm-backend` selects the retrieval agent's backend only. The adjudication of
-uncertain full-text-scan candidates (on under `--pull-everything`, or with
+uncertain full-text-scan candidates (on under `--fetch-everything`, or with
 `--llm-adjudicate-artifacts`) always calls the Claude Code CLI, as a single-turn
 call with `--allowed-tools ""`; it does not get the `--disallowed-tools` /
 `--strict-mcp-config` lockdown described above.
@@ -999,7 +1010,7 @@ from the network. `--generate-pdf-from-xml` prints the same page to
 
 ```bash
 pip install 'fetchpdf[render,browser]' && playwright install chromium
-fetchpdf papers.csv -o ./out --get-xml-or-html --generate-pdf-from-xml
+fetchpdf papers.csv -o ./out --get-xml-or-html-and-pdf --generate-pdf-from-xml
 fetchpdf-render ./out --html --pdf --fetch-figures   # XML you already have
 ```
 
@@ -1017,7 +1028,7 @@ shaped by its aspect ratio and capped at the column width. Under the title come
 the authors, their affiliations and the journal citation (volume, issue, pages).
 Tables always fit the page width in the PDF; very wide ones print smaller.
 
-Only records whose XML was kept are rendered — add `--get-xml-or-html` to keep
+Only records whose XML was kept are rendered — add `--get-xml-or-html-and-pdf` to keep
 the XML beside the PDF.
 
 ## Institutional access via cookies (ie using your library's subscriptions)
@@ -1111,8 +1122,22 @@ your browser's cookies signs fetchpdf out as well.
 Publishers whose PDF address follows from the DOI: **Wiley** (`10.1002`,
 `10.1111`), **Taylor & Francis** (`10.1080`), **SAGE** (`10.1177`),
 **Springer** (`10.1007`, `10.1057`), the **Royal Society** (`10.1098`),
-**Hogrefe** (`10.1027`) and **INFORMS** (`10.1287`). Other DOIs are untouched
-and produce no extra output.
+**Hogrefe** (`10.1027`), **INFORMS** (`10.1287`), **PNAS** (`10.1073`),
+**Science** (`10.1126`), **ASCO** (`10.1200`), **AHA** (`10.1161`), **APS**
+(`10.1152`), **ASM** (`10.1128`), **ATS** (`10.1164`), **ACS** (`10.1021`) and
+**FASEB** (`10.1096`).
+
+Publishers whose PDF address is not derivable from the DOI are reached through
+the DOI link, and the PDF the landing page declares is fetched: **AACR**
+(`10.1158`), **Cold Spring Harbor** (`10.1101`), **Oncotarget** (`10.18632`),
+**Oxford** (`10.1093`, `10.1210`, `10.4049`), **JAMA** (`10.1001`), **ASH**
+(`10.1182`), **Company of Biologists** (`10.1242`), **ERS** (`10.1183`),
+**BMJ** (`10.1136`), **Bioscientifica** (`10.1530`) and **LWW** (`10.1097`).
+
+Several of these are open access but sit behind a Cloudflare bot check that a
+plain download cannot pass. Opening any article on the site once in your
+everyday browser earns the clearance cookie, and the next run reuses it.
+Other DOIs are untouched and produce no extra output.
 
 It runs right after the fast open-access lookups (repositories, PMC,
 Unpaywall), so a paper that is free anywhere is fetched from there and never
@@ -1120,7 +1145,7 @@ spends a subscription download. It runs before the slow long tail (Crossref
 landing pages, CORE, DOAJ, DataCite, ...), where a subscription answers in a
 second or two. The PDF meets the same identity check as every other source and
 is reported as `institutional_cookies` in source tracking and `--json`. It also
-fills the PDF half of `--get-xml-or-html` runs.
+fills the PDF half of `--get-xml-or-html-and-pdf` runs.
 
 ### Sign-in, the VPN, and Cloudflare
 
@@ -1204,9 +1229,9 @@ refusal, not a bare failure:
 | `path` | str \| null | Absolute path to the artifact, or null. |
 | `format` | str | From the artifact actually written: `pdf`, `xml`, `html`, `latex`, `landing`, `suppl`, `abstract`, `text`, `other`. **A successful record is not necessarily a PDF** — with the default XML fallback a record can succeed as `xml`. |
 | `source` | str \| null | The route that produced it (`unpaywall`, `core`, `osf`, …), `existing` for a skip, `structured_fallback` when no PDF was available and structured full text was saved instead. |
-| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
+| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
 | `reasons` | list[str] | Why a record failed, or why a kept PDF could not be verified. Empty on a verified success. |
-| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html`), empty otherwise. `path` is not repeated here. |
+| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf`), empty otherwise. `path` is not repeated here. |
 | `supplementary` | object | Only with `--pull-supplementary` (or a flag that implies it): `status` (`ok`, `none_found`, `partial`, `incomplete`, `error`, `skipped`), `written`, `skipped`, `manifest` (absolute path or null), `missing_declared` (files the article declares that were not obtained) and `blocked_urls` (refused to this client; open them in a browser). |
 | `figures` | object | Only with `--pull-figures`: `status`, `written`, `failed`, `refusal` and `manifest`. |
 | `images` | object | Only with `--extract-images`: `status`, `written` and `manifest`. |
@@ -1402,7 +1427,7 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
 9. **CORE** (core.ac.uk) — 40M+ repository papers; needs `COREAPIKEY`
 10. **DOAJ** — open-access full-text links
 11. **DataCite content URLs** — direct content and related resource links
-12. **Wiley rendered PDF** — only with `--add-playwright`
+12. **Wiley rendered PDF** — only when Playwright is installed
 13. **APA supplemental** — APA supplemental files, converting `.doc`/`.docx` to PDF when available
 14. **Direct DOI resolver**
     - Follows `https://doi.org/{doi}` to the landing page
@@ -1430,8 +1455,28 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
 
 18. **Elsevier full-text API**
     - Elsevier records only (`10.1016/*`, or a Crossref landing page on sciencedirect.com / elsevier.com)
-    - Tries the PDF first, then full-text XML (the XML half is skipped with `--no-xml-fallback`)
     - Requires `ELSEVIER_TDM_API_KEY`
+    - **`10.1016/*` DOIs are asked for their PDF first** — before PMC, Unpaywall,
+      institutional access and the long tail of landing-page and repository
+      probes (a dead repository host in Unpaywall's list alone can cost over a
+      minute). An entitled record is one request instead of a walk through every
+      other route. Only a *full* PDF ends
+      the search: Elsevier's first-page-only preview (served with HTTP 200 and a
+      real PDF when the key is not entitled) is discarded, so asking first cannot
+      pre-empt a source that holds the whole paper. When the PDF is obtained, the
+      full-text XML is fetched alongside it, so both are on disk.
+    - The deferred step at the end of the chain remains: it takes the XML for
+      records whose key is entitled to the XML but not the PDF, and covers
+      ScienceDirect DOIs outside `10.1016/*`. It does not repeat a PDF request the
+      early step already made. The XML half is skipped with `--no-xml-fallback`.
+    - With `--prioritize-xml` / `--xml-only` / `--get-xml-or-html-and-pdf` the tiered
+      ladder asks the same endpoint itself (`elsevier_tdm` at T1, `elsevier_pdf`
+      at T5, ahead of the legacy chain).
+    - **Off campus?** Entitlement is decided by your IP unless you also set
+      `ELSEVIER_INSTTOKEN`. Request an institutional token from Elsevier (the
+      developer portal at dev.elsevier.com, or your library); it is sent as
+      `X-ELS-Insttoken` on every `api.elsevier.com` request, never to any other
+      host, and is redacted from logs and sidecars.
 
 If the whole chain yields no PDF, `fetch_pdf` tries structured full text
 (T1 XML / T2 HTML) from the tier ladder before giving up; `--no-xml-fallback`

{fetchpdf-0.1.4 → fetchpdf-0.1.6}/README.md

@@ -15,7 +15,7 @@ A comprehensive Python package to download academic papers (PDFs) from DOIs (or
 - [Batch Processing & Parallel Execution](#batch-processing--parallel-execution)
 - [Format-Prioritized Retrieval](#format-prioritized-retrieval)
 - [Supplementary Material](#supplementary-material)
-- [`--pull-everything`: the second layer](#--pull-everything-the-second-layer)
+- [`--fetch-everything`: the second layer](#--fetch-everything-the-second-layer)
 - [Figure Images](#figure-images)
 - [Institutional access](#institutional-access-via-cookies-ie-using-your-librarys-subscriptions) — papers your library subscribes to (`fetchpdf fetchcookies`)
 - [Calling fetchpdf from a script or an agent](#calling-fetchpdf-from-a-script-or-an-agent) — `--json`
@@ -45,7 +45,7 @@ A comprehensive Python package to download academic papers (PDFs) from DOIs (or
 - 🖼️ **[Figure Images](#figure-images)**: `--pull-figures` fetches the article's published figures from PMC (or the Elsevier object API) with a manifest of labels, captions and hashes
 - 🖨️ **[XML to HTML / PDF](#rendering-the-xml-as-html-or-pdf)**: `--generate-html-from-xml` / `--generate-pdf-from-xml` render a retrieved XML as one self-contained page, figures and tables embedded
 - 🔗 **[Linked Datasets & Code](#linked-datasets-and-code-stem_linked_artifactsjson)**: the same pass discovers each paper's external datasets/software via ScholeXplorer, Europe PMC and DataCite, downloads ownership-confirmed deposits from figshare/Zenodo/OSF/Dryad/Dataverse, and records every link in a sidecar
-- 🤖 **[LLM-assisted retrieval](#--pull-everything-the-second-layer)**: `--pull-everything` adds a second layer — a model reads the paper itself and goes after the SI and datasets the APIs missed
+- 🤖 **[LLM-assisted retrieval](#--fetch-everything-the-second-layer)**: `--fetch-everything` adds a second layer — a model reads the paper itself and goes after the SI and datasets the APIs missed
 - 🔐 **[Institutional access](#institutional-access-via-cookies-ie-using-your-librarys-subscriptions)** (opt-in): a one-time `fetchpdf fetchcookies` lets fetchpdf download papers your library subscribes to
 - 🚀 **Batch Processing**: Process multiple DOIs/PMIDs from CSV files or lists
 - ⚡ **Parallel Execution**: Download multiple papers simultaneously with configurable workers
@@ -66,7 +66,7 @@ pip install -e .
 
 ### Optional: Install Playwright and Chromium for Browser Automation
 
-Install Playwright and Chromium only if you want to try additional scrapers enabled by `--add-playwright`. The scapers are for SSRN, OSF (if direct download fails), Wiley 10.1111/ DOIs, HTML table retrieval (second route), and Atypon supplementary material (PNAS, Science, SAGE, T&F, Annual Reviews).
+Install Playwright and Chromium to enable additional browser-based scrapers. They are used automatically once Playwright is installed (pass `--no-playwright` to skip them). The scrapers are for SSRN, OSF (if direct download fails), Wiley 10.1111/ DOIs, HTML table retrieval (second route), and Atypon supplementary material (PNAS, Science, SAGE, T&F, Annual Reviews).
 
 ```bash
 pip install -e ".[browser]"   # or: pip install playwright
@@ -101,9 +101,10 @@ SERPAPI_API_KEY=your_serpapi_key           # Optional Google Scholar PDF fallbac
 
 # Optional — publisher-specific
 ELSEVIER_TDM_API_KEY=your_elsevier_key      # Elsevier text/data-mining access
+ELSEVIER_INSTTOKEN=your_inst_token          # Elsevier institutional token (needs the key above)
 SCOPUS_API_KEY=your_scopus_key              # Scopus abstract lookup during identifier resolution
 
-# Optional — only for --pull-everything --llm-backend openrouter
+# Optional — only for --fetch-everything --llm-backend openrouter
 OPENROUTER_API_KEY=your_openrouter_key      # the retrieval agent's second backend
 
 # Required by fetchpdf-pubpeer — PubPeer rejects keyless requests
@@ -154,7 +155,7 @@ papers your library subscribes to. See
 By default a record yields one file: the best PDF the source chain finds. Three
 flags widen that.
 
-1. **Structured full text** — `--get-xml-or-html` keeps a JATS/TEI XML (else
+1. **Structured full text** — `--get-xml-or-html-and-pdf` keeps a JATS/TEI XML (else
    publisher HTML) copy alongside the PDF; `--to-markdown` also renders it to
    `{stem}_from_xml.md` (or `{stem}_from_html.md`). XML and HTML are great
    formats for using with AI (LLMs): complex tables usually survive better than
@@ -174,7 +175,7 @@ flags widen that.
 
 ```bash
 fetchpdf papers.csv -o ./out -w 4 \
-    --get-xml-or-html --to-markdown \
+    --get-xml-or-html-and-pdf --to-markdown \
     --download-data-artifacts \
     --make-subfolder --provenance
 ```
@@ -403,10 +404,10 @@ fetchpdf papers.csv -o ./out --xml-html-only    # XML or publisher HTML
 fetchpdf papers.csv -o ./out --xml-only         # XML, nothing else
 
 # A structured copy AND the PDF for every record
-fetchpdf papers.csv -o ./out --get-xml-or-html
+fetchpdf papers.csv -o ./out --get-xml-or-html-and-pdf
 
 # ...and a {stem}_from_xml.md / _from_html.md for each XML/HTML artifact
-fetchpdf papers.csv -o ./out --get-xml-or-html --to-markdown
+fetchpdf papers.csv -o ./out --get-xml-or-html-and-pdf --to-markdown
 
 # Convert artifacts you already have, without re-downloading
 fetchpdf-md ./out
@@ -424,7 +425,7 @@ supplement keeps its own extension, e.g. `{stem}.xlsx`), `{stem}.pdf`,
 `{stem}.txt` and `{stem}.landing.html` — plus `{stem}_from_xml.md` /
 `{stem}_from_html.md` under `--to-markdown`.
 
-`--get-xml-or-html` doubles as a backfill: pointed at a directory of existing
+`--get-xml-or-html-and-pdf` doubles as a backfill: pointed at a directory of existing
 PDFs it fetches only the missing structured half and re-downloads nothing.
 
 ### Choosing a flag
@@ -434,7 +435,9 @@ PDFs it fetches only the missing structured half and re-downloads nothing.
 | `--prioritize-xml` | everything, best format first | yes |
 | `--xml-html-only` | T1 XML, T2 HTML | **no** — record fails |
 | `--xml-only` | T1 XML | **no** — record fails |
-| `--get-xml-or-html` | structured **and** PDF, both kept | yes, and keeps it |
+| `--get-xml-or-html-and-pdf` | structured **and** PDF, both kept | yes, and keeps it |
+| `--get-xml-and-pdf` | XML where available **and** PDF where available, both kept | yes, and keeps it |
+| `--get-xml-and-html-and-pdf` | XML, publisher HTML and PDF, each to the extent available | yes, and keeps it |
 
 `--xml-only` and `--xml-html-only` both imply `--prioritize-xml`. Given both,
 `--xml-only` is stricter and wins (with a warning).
@@ -533,9 +536,9 @@ unless stdin and stdout are both terminals** — pipes, cron and CI get a one-li
 count and today's behaviour, so no unattended run can block on it. It is also
 suppressed when the answer is already known: any flag that turns the
 supplementary pass on (`--pull-supplementary`, `--refresh-supplementary`,
-`--download-data-artifacts`, `--pull-everything`, `--llm-agent-retrieval`), an
+`--download-data-artifacts`, `--fetch-everything`, `--llm-agent-retrieval`), an
 explicit `--on-existing skip` or `supplement`, `--abstract-only`, or the
-goal-aware flags (`--get-xml-or-html`, `--upgrade-existing` on the tiered path),
+goal-aware flags (`--get-xml-or-html-and-pdf`, `--upgrade-existing` on the tiered path),
 which do not skip records at all. Single-identifier runs never ask.
 
 What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
@@ -551,6 +554,12 @@ retrieved, the PDF is kept and reported as unverified.
 When a PDF is unavailable, FetchPDF tries XML or HTML full text automatically.
 Use `--no-xml-fallback` to disable this fallback.
 
+A record that comes back as XML only is filed in its own folder,
+`{output_dir}/{stem}/`, with its figure images in `{stem}_figures/` and a
+rendering at `{stem}_from_xml.pdf` (figures embedded; a rendering of the XML,
+not the publisher's PDF). A later flat run finds it there. Use
+`--no-xml-fallback-extras` to leave the bare `.xml` instead.
+
 Publisher HTML (T2) uses lxml, which is installed with fetchpdf.
 
 Advanced users can customize retrieval order, source capabilities, and host rate
@@ -634,7 +643,7 @@ Twenty routes, tried in a fixed order so the numbering is reproducible:
 | Europe PMC datalinks | any PMID — text-mined accessions and data citations |
 | Full-text scan | deposits the paper names in its own prose — only under `--download-data-artifacts` |
 | Atypon publisher SI | PNAS, Science, Annual Reviews, SAGE, T&F — only when Europe PMC withholds the SI or the record has no PMCID; drives a headed browser, so it needs Playwright, Chromium and a display (or Xvfb) |
-| LLM retrieval agent | only under [`--pull-everything`](#--pull-everything-the-second-layer) or `--llm-agent-retrieval` |
+| LLM retrieval agent | only under [`--fetch-everything`](#--fetch-everything-the-second-layer) or `--llm-agent-retrieval` |
 | JCI | `10.1172/jci…` |
 
 Wiley, ACS and MDPI serve HTTP 403 to plain requests and are not yet covered;
@@ -684,7 +693,7 @@ Supplementary files never appear in `failed_dois.csv`, in `source_tracking.csv`,
 or in the format-composition tally; the missing-PDFs report lists gaps in them
 only in its own separate section. The same holds for linked datasets,
 [figures](#figure-images) and the
-[`--pull-everything`](#--pull-everything-the-second-layer) agent.
+[`--fetch-everything`](#--fetch-everything-the-second-layer) agent.
 
 ### Linked datasets and code: `{stem}_linked_artifacts.json`
 
@@ -773,18 +782,19 @@ GitLab repositories, and the institutional-repository long tail.
 Adding an enumerator is the same small pattern as `enumerate_dryad` /
 `enumerate_dataverse` in `fetchpdf/retrieval/supplement_index.py`.
 
-## `--pull-everything`: the second layer
+## `--fetch-everything`: the second layer
 
 Everything above is layer 1 — APIs, indexes and enumerators. It is very good at
 what publishers and repositories *declare*, and it cannot read. Layer 2 is a
 model that reads the paper itself and goes after what layer 1 did not get.
 
 ```bash
-fetchpdf papers.csv -o ./out --get-xml-or-html --make-subfolder --provenance \
-         --pull-everything
+fetchpdf papers.csv -o ./out --make-subfolder --provenance --fetch-everything
 ```
 
-`--pull-everything` implies `--pull-supplementary` and
+`--fetch-everything` implies `--get-xml-or-html-and-pdf` (XML, else HTML, beside the PDF
+where one exists), `--pull-figures` (the publisher's figure images beside the
+XML), `--pull-supplementary` and
 `--download-data-artifacts`, turns on adjudication of the deterministic scan's
 uncertain candidates, and adds the retrieval agent. Files land exactly where
 they always did: supplements as numbered siblings of the PDF, linked deposits
@@ -856,7 +866,7 @@ backend has no general-purpose harness to lock down, because its two tools are
 ours, so it downloads directly.
 
 `--llm-backend` selects the retrieval agent's backend only. The adjudication of
-uncertain full-text-scan candidates (on under `--pull-everything`, or with
+uncertain full-text-scan candidates (on under `--fetch-everything`, or with
 `--llm-adjudicate-artifacts`) always calls the Claude Code CLI, as a single-turn
 call with `--allowed-tools ""`; it does not get the `--disallowed-tools` /
 `--strict-mcp-config` lockdown described above.
@@ -960,7 +970,7 @@ from the network. `--generate-pdf-from-xml` prints the same page to
 
 ```bash
 pip install 'fetchpdf[render,browser]' && playwright install chromium
-fetchpdf papers.csv -o ./out --get-xml-or-html --generate-pdf-from-xml
+fetchpdf papers.csv -o ./out --get-xml-or-html-and-pdf --generate-pdf-from-xml
 fetchpdf-render ./out --html --pdf --fetch-figures   # XML you already have
 ```
 
@@ -978,7 +988,7 @@ shaped by its aspect ratio and capped at the column width. Under the title come
 the authors, their affiliations and the journal citation (volume, issue, pages).
 Tables always fit the page width in the PDF; very wide ones print smaller.
 
-Only records whose XML was kept are rendered — add `--get-xml-or-html` to keep
+Only records whose XML was kept are rendered — add `--get-xml-or-html-and-pdf` to keep
 the XML beside the PDF.
 
 ## Institutional access via cookies (ie using your library's subscriptions)
@@ -1072,8 +1082,22 @@ your browser's cookies signs fetchpdf out as well.
 Publishers whose PDF address follows from the DOI: **Wiley** (`10.1002`,
 `10.1111`), **Taylor & Francis** (`10.1080`), **SAGE** (`10.1177`),
 **Springer** (`10.1007`, `10.1057`), the **Royal Society** (`10.1098`),
-**Hogrefe** (`10.1027`) and **INFORMS** (`10.1287`). Other DOIs are untouched
-and produce no extra output.
+**Hogrefe** (`10.1027`), **INFORMS** (`10.1287`), **PNAS** (`10.1073`),
+**Science** (`10.1126`), **ASCO** (`10.1200`), **AHA** (`10.1161`), **APS**
+(`10.1152`), **ASM** (`10.1128`), **ATS** (`10.1164`), **ACS** (`10.1021`) and
+**FASEB** (`10.1096`).
+
+Publishers whose PDF address is not derivable from the DOI are reached through
+the DOI link, and the PDF the landing page declares is fetched: **AACR**
+(`10.1158`), **Cold Spring Harbor** (`10.1101`), **Oncotarget** (`10.18632`),
+**Oxford** (`10.1093`, `10.1210`, `10.4049`), **JAMA** (`10.1001`), **ASH**
+(`10.1182`), **Company of Biologists** (`10.1242`), **ERS** (`10.1183`),
+**BMJ** (`10.1136`), **Bioscientifica** (`10.1530`) and **LWW** (`10.1097`).
+
+Several of these are open access but sit behind a Cloudflare bot check that a
+plain download cannot pass. Opening any article on the site once in your
+everyday browser earns the clearance cookie, and the next run reuses it.
+Other DOIs are untouched and produce no extra output.
 
 It runs right after the fast open-access lookups (repositories, PMC,
 Unpaywall), so a paper that is free anywhere is fetched from there and never
@@ -1081,7 +1105,7 @@ spends a subscription download. It runs before the slow long tail (Crossref
 landing pages, CORE, DOAJ, DataCite, ...), where a subscription answers in a
 second or two. The PDF meets the same identity check as every other source and
 is reported as `institutional_cookies` in source tracking and `--json`. It also
-fills the PDF half of `--get-xml-or-html` runs.
+fills the PDF half of `--get-xml-or-html-and-pdf` runs.
 
 ### Sign-in, the VPN, and Cloudflare
 
@@ -1165,9 +1189,9 @@ refusal, not a bare failure:
 | `path` | str \| null | Absolute path to the artifact, or null. |
 | `format` | str | From the artifact actually written: `pdf`, `xml`, `html`, `latex`, `landing`, `suppl`, `abstract`, `text`, `other`. **A successful record is not necessarily a PDF** — with the default XML fallback a record can succeed as `xml`. |
 | `source` | str \| null | The route that produced it (`unpaywall`, `core`, `osf`, …), `existing` for a skip, `structured_fallback` when no PDF was available and structured full text was saved instead. |
-| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
+| `identity` | str \| null | The verdict on the PDF **fetched in this run and reported in `path`**: `verified`, or a refusal — `wrong_article`, `truncated`, `unreadable`, `no_reference`, `no_engine`. Null when nothing was judged: an XML artifact, a file already on disk, or any run under the tiered flags (`--prioritize-xml`, `--get-xml-or-html-and-pdf`, `--xml-only`, `--xml-html-only`), where the engine validates per tier instead. A success with a non-`verified` identity is a PDF that was kept rather than deleted because the verdict could not be trusted — see [the identity check](#format-prioritized-retrieval) — not one that passed. |
 | `reasons` | list[str] | Why a record failed, or why a kept PDF could not be verified. Empty on a verified success. |
-| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html`), empty otherwise. `path` is not repeated here. |
+| `paths` | list[str] | The other artifacts a tiered run wrote (`--get-xml-or-html-and-pdf`), empty otherwise. `path` is not repeated here. |
 | `supplementary` | object | Only with `--pull-supplementary` (or a flag that implies it): `status` (`ok`, `none_found`, `partial`, `incomplete`, `error`, `skipped`), `written`, `skipped`, `manifest` (absolute path or null), `missing_declared` (files the article declares that were not obtained) and `blocked_urls` (refused to this client; open them in a browser). |
 | `figures` | object | Only with `--pull-figures`: `status`, `written`, `failed`, `refusal` and `manifest`. |
 | `images` | object | Only with `--extract-images`: `status`, `written` and `manifest`. |
@@ -1363,7 +1387,7 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
 9. **CORE** (core.ac.uk) — 40M+ repository papers; needs `COREAPIKEY`
 10. **DOAJ** — open-access full-text links
 11. **DataCite content URLs** — direct content and related resource links
-12. **Wiley rendered PDF** — only with `--add-playwright`
+12. **Wiley rendered PDF** — only when Playwright is installed
 13. **APA supplemental** — APA supplemental files, converting `.doc`/`.docx` to PDF when available
 14. **Direct DOI resolver**
     - Follows `https://doi.org/{doi}` to the landing page
@@ -1391,8 +1415,28 @@ These run first when the DOI matches (by prefix, or by the name appearing anywhe
 
 18. **Elsevier full-text API**
     - Elsevier records only (`10.1016/*`, or a Crossref landing page on sciencedirect.com / elsevier.com)
-    - Tries the PDF first, then full-text XML (the XML half is skipped with `--no-xml-fallback`)
     - Requires `ELSEVIER_TDM_API_KEY`
+    - **`10.1016/*` DOIs are asked for their PDF first** — before PMC, Unpaywall,
+      institutional access and the long tail of landing-page and repository
+      probes (a dead repository host in Unpaywall's list alone can cost over a
+      minute). An entitled record is one request instead of a walk through every
+      other route. Only a *full* PDF ends
+      the search: Elsevier's first-page-only preview (served with HTTP 200 and a
+      real PDF when the key is not entitled) is discarded, so asking first cannot
+      pre-empt a source that holds the whole paper. When the PDF is obtained, the
+      full-text XML is fetched alongside it, so both are on disk.
+    - The deferred step at the end of the chain remains: it takes the XML for
+      records whose key is entitled to the XML but not the PDF, and covers
+      ScienceDirect DOIs outside `10.1016/*`. It does not repeat a PDF request the
+      early step already made. The XML half is skipped with `--no-xml-fallback`.
+    - With `--prioritize-xml` / `--xml-only` / `--get-xml-or-html-and-pdf` the tiered
+      ladder asks the same endpoint itself (`elsevier_tdm` at T1, `elsevier_pdf`
+      at T5, ahead of the legacy chain).
+    - **Off campus?** Entitlement is decided by your IP unless you also set
+      `ELSEVIER_INSTTOKEN`. Request an institutional token from Elsevier (the
+      developer portal at dev.elsevier.com, or your library); it is sent as
+      `X-ELS-Insttoken` on every `api.elsevier.com` request, never to any other
+      host, and is redacted from logs and sidecars.
 
 If the whole chain yields no PDF, `fetch_pdf` tries structured full text
 (T1 XML / T2 HTML) from the tier ladder before giving up; `--no-xml-fallback`

{fetchpdf-0.1.4 → fetchpdf-0.1.6}/fetchpdf/__init__.py

@@ -15,7 +15,7 @@ from .fetch_metadata_from_doi import fetch_metadata_from_doi
 # behavioural difference, it is the same object.
 fetch_pdf_from_doi = fetch_pdf
 
-__version__ = "0.1.4"
+__version__ = "0.1.6"
 __all__ = [
     "fetch_pdf",
     "fetch_pdf_from_doi",  # deprecated alias for fetch_pdf

{fetchpdf-0.1.4 → fetchpdf-0.1.6}/fetchpdf/_env.py

@@ -59,6 +59,13 @@ PUBPEER_DEVKEY = os.getenv("PUBPEER_DEVKEY")
 
 # Elsevier full-text (TDM) retrieval.
 ELSEVIER_TDM_API_KEY = os.getenv("ELSEVIER_TDM_API_KEY")
+# Elsevier institutional token. Optional, and only meaningful alongside the key:
+# it is issued by Elsevier (developer portal / your library) and tells the API
+# to grant the key your institution's subscription entitlements even when the
+# request does not come from an institutional IP -- off campus, behind a
+# non-subscribing VPN, on a cloud host. Without it entitlement is decided by the
+# requesting IP alone.
+ELSEVIER_INSTTOKEN = os.getenv("ELSEVIER_INSTTOKEN")
 
 
 if not EMAIL:

{fetchpdf-0.1.4 → fetchpdf-0.1.6}/fetchpdf/_http.py

@@ -79,6 +79,30 @@ def looks_like_pdf(content: bytes, content_type: str = "") -> bool:
     return ctype in ("application/pdf", "application/octet-stream") and bool(content)
 
 
+def elsevier_headers(api_key: str, extra: dict = None) -> dict:
+    """Request headers for any api.elsevier.com call that carries the TDM key.
+
+    The one place the credentials are spelled, so a request can never carry the
+    key and forget the institutional token. ``X-ELS-Insttoken`` is added when
+    ELSEVIER_INSTTOKEN is set and omitted otherwise -- an empty header would be
+    read as a malformed token and refused. It is looked up at call time, not
+    bound at import, so a process that loads its .env late still gets it.
+
+    Both go in headers, never the URL: a URL surfaces in exception text, logs
+    and provenance, and a credential there is a credential leaked.
+
+    The key is a parameter, not read here, because each caller holds its own
+    binding of it (and its tests pin that binding).
+    """
+    from . import _env
+
+    out = {"X-ELS-APIKey": api_key}
+    if _env.ELSEVIER_INSTTOKEN:
+        out["X-ELS-Insttoken"] = _env.ELSEVIER_INSTTOKEN
+    out.update(extra or {})
+    return out
+
+
 def elsevier_first_page_only(headers) -> bool:
     """True when Elsevier's X-ELS-Status header marks a first-page-only PDF.
 

{fetchpdf-0.1.4 → fetchpdf-0.1.6}/fetchpdf/fetch_abstract_from_doi.py

@@ -261,7 +261,7 @@ def fetch_abstract_from_doi(doi, email=None, delay=0.1, verbose=False):
                 "title":    d.get("title"),
                 "abstract": abstract,
                 "authors":  authors,
-                "journal":  (d.get("primary_location") or {}).get("source", {}).get("display_name"),
+                "journal":  ((d.get("primary_location") or {}).get("source") or {}).get("display_name"),
                 "year":     d.get("publication_year"),
             })
             if verbose and abstract:
@@ -435,7 +435,8 @@ def fetch_abstract_from_doi(doi, email=None, delay=0.1, verbose=False):
                 if verbose and abstract:
                     print("  ✅ Abstract from Scopus")
         except Exception as e:
-            if verbose: print(f"  Scopus error: {e}")
+            # The exception text can carry the request URL, and with it the key.
+            if verbose: print(f"  Scopus error: {type(e).__name__}")
         time.sleep(delay)
         if _done(): return result