fetchpdf 0.1.0__py3-none-any.whl

fetchpdf-0.1.0.dist-info/METADATA

@@ -0,0 +1,1392 @@
+Metadata-Version: 2.4
+Name: fetchpdf
+Version: 0.1.0
+Summary: A comprehensive tool to download PDFs from DOIs using multiple sources
+Author-email: Dan Elton <delton17@gmail.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/The-Metascience-Observatory/fetchpdf
+Project-URL: Repository, https://github.com/The-Metascience-Observatory/fetchpdf
+Project-URL: Issues, https://github.com/The-Metascience-Observatory/fetchpdf/issues
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: requests>=2.31.0
+Requires-Dist: urllib3>=1.26
+Requires-Dist: playwright>=1.40.0
+Requires-Dist: python-dotenv>=1.0.0
+Requires-Dist: pypdf>=4.0
+Provides-Extra: html
+Requires-Dist: lxml>=4.9; extra == "html"
+Provides-Extra: images
+Requires-Dist: pymupdf>=1.24; extra == "images"
+Provides-Extra: text
+Requires-Dist: pymupdf>=1.24; extra == "text"
+Requires-Dist: pypdf>=4.0; extra == "text"
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-xdist>=3.0; extra == "dev"
+Requires-Dist: black>=23.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Dynamic: license-file
+
+# fetchpdf
+
+A comprehensive Python package to download academic papers (PDFs) from DOIs (or PMIDs resolved to DOIs) using multiple fallback sources.
+
+
+## Table of Contents
+
+- [Features](#features)
+- [Installation](#installation)
+- [Quick Start](#quick-start)
+  - [Getting more than the PDF](#getting-more-than-the-pdf) — XML/HTML, supplementary material, linked datasets
+- [Usage Examples](#usage-examples)
+- [Batch Processing & Parallel Execution](#batch-processing--parallel-execution)
+- [Format-Prioritized Retrieval](#format-prioritized-retrieval)
+- [Supplementary Material](#supplementary-material)
+- [`--pull-everything`: the second layer](#--pull-everything-the-second-layer)
+- [API Reference](#api-reference)
+- [Download Sources](#download-sources-in-order-of-priority)
+- [Troubleshooting](#troubleshooting)
+
+## Features
+
+- 🔍 **Multiple Sources**: Automatically tries 10+ sources including:
+  - OSF (Open Science Framework) - Projects & Preprints
+  - PubMed Central / Europe PMC
+  - OpenAlex
+  - Unpaywall
+  - Crossref
+  - Semantic Scholar
+  - Figshare
+  - SSRN (Social Science Research Network)
+
+- 🔄 **Smart Fallback**: If one source fails, automatically tries the next
+- 🧬 **Format Prioritization**: `--prioritize-xml` prefers structured full text (JATS/TEI) over PDF — see [below](#format-prioritized-retrieval)
+- 📎 **Supplementary Material**: `--pull-supplementary` fetches every supplementary file alongside the paper — see [below](#supplementary-material)
+- 🔗 **Linked Datasets & Code**: the same pass discovers each paper's external datasets/software via ScholeXplorer, Europe PMC and DataCite, downloads ownership-confirmed deposits from figshare/Zenodo/OSF/Dryad/Dataverse, and records every link in a sidecar — see [below](#linked-datasets-and-code-stem_linked_artifactsjson)
+- 🤖 **LLM-assisted retrieval**: `--pull-everything` adds a second layer — a model reads the paper itself and goes after the SI and datasets the APIs missed — see [below](#--pull-everything-the-second-layer)
+- 🎭 **Browser Automation**: Uses Playwright to bypass JavaScript-based protections
+- 🚀 **Batch Processing**: Process multiple DOIs/PMIDs from CSV files or lists
+- ⚡ **Parallel Execution**: Download multiple papers simultaneously with configurable workers
+
+## Installation
+
+```bash
+pip install fetchpdf
+```
+
+Or, to work from source:
+
+```bash
+git clone https://github.com/The-Metascience-Observatory/fetchpdf.git
+cd fetchpdf
+pip install -e .
+```
+
+### Post-Installation: Install Playwright Browsers
+
+After installing the package, you need to install Playwright browsers:
+
+```bash
+playwright install chromium
+```
+
+### Configuration: `.env.local`
+
+Create a `.env.local` file in the project root to configure API keys and settings. API keys are **optional** but significantly improve rate limits and reliability:
+
+```bash
+# Required — used by Unpaywall (mandatory) and Crossref polite pool (10 req/s vs 5)
+EMAIL=your@email.com
+
+# Optional — improves rate limits / avoids throttling
+OPENALEXAPIKEY=your_openalex_key           # OpenAlex: avoids 429 rate-limit errors
+SEMANTIC_SCHOLAR_API_KEY=your_s2_key        # Semantic Scholar: 1 → 100 req/s
+ENTREZ_EUTILS_API_KEY=your_ncbi_key         # NCBI E-utilities: 3 → 10 req/s
+COREAPIKEY=your_core_key                   # CORE: 40M+ OA papers from core.ac.uk
+
+# Optional — publisher-specific
+ELSEVIER_TDM_API_KEY=your_elsevier_key      # Elsevier text/data-mining access
+
+# Optional — only for --pull-everything --llm-backend openrouter
+OPENROUTER_API_KEY=your_openrouter_key      # the retrieval agent's second backend
+```
+
+**Rate limit improvements with API keys:**
+
+| API | Without Key | With Key | How to Get |
+|-----|-------------|----------|------------|
+| Crossref | 5 req/s | 10 req/s (polite pool) | Just set `EMAIL` |
+| OpenAlex | Severe throttling | Normal | [openalex.org/users](https://openalex.org/users) |
+| Semantic Scholar | 1 req/s | 100 req/s | [semanticscholar.org/product/api](https://www.semanticscholar.org/product/api) |
+| NCBI E-utilities | 3 req/s | 10 req/s | [ncbi.nlm.nih.gov/account](https://www.ncbi.nlm.nih.gov/account/) |
+| CORE | Unavailable | 1,000 tokens/day | [core.ac.uk/services/api](https://core.ac.uk/services/api) |
+
+## Quick Start
+
+### Command Line Interface
+
+```bash
+# Batch download from CSV (auto-detected by .csv extension)
+fetchpdf papers.csv -o ./pdfs
+fetchpdf papers.csv -o ./pdfs -w 4        # 4 parallel workers
+
+# Single DOI or PMID
+fetchpdf "10.1038/nature12373"             # saves to ./pdfs/
+fetchpdf "10.1038/nature12373" -o ./papers # custom output dir
+fetchpdf "10.1038/nature12373" output.pdf  # custom filename
+fetchpdf 33262244                          # PMID auto-resolved to DOI
+```
+
+### Getting more than the PDF
+
+By default a record yields one file: the best PDF the source chain finds. Three
+flags widen that, and they compose — use all three together to get the paper in
+both formats plus everything deposited with it.
+
+**1. Structured full text (XML / HTML) instead of, or alongside, the PDF**
+
+A PDF has no tables, only ink that looks like tables. For anything that parses
+the paper, ask for markup:
+
+```bash
+# Keep BOTH a structured copy (JATS/TEI XML, else publisher HTML) AND the PDF
+fetchpdf papers.csv -o ./out --get-xml-or-html
+
+# Prefer structured full text, but accept a PDF when there is none
+fetchpdf papers.csv -o ./out --prioritize-xml
+
+# Structured only — fail the record rather than write a PDF
+fetchpdf papers.csv -o ./out --xml-html-only    # XML or publisher HTML
+fetchpdf papers.csv -o ./out --xml-only         # XML, nothing else
+
+# Also render each XML/HTML artifact to {stem}.md for an LLM
+fetchpdf papers.csv -o ./out --get-xml-or-html --to-markdown
+```
+
+Artifacts land as suffixed siblings: `{stem}.xml`, `{stem}.fulltext.html`,
+`{stem}.pdf`, `{stem}.md`. `--get-xml-or-html` is also a backfill — pointed at a
+directory of PDFs you already have, it fetches only the missing structured half.
+Publisher HTML needs `pip install 'fetchpdf[html]'`; without it that tier is skipped
+with a logged reason. Full detail: [Format-Prioritized Retrieval](#format-prioritized-retrieval).
+
+**2. Supplementary material (SI/SM)**
+
+```bash
+# Every supplementary file the SI endpoints offer: PDFs, spreadsheets, docs, data
+fetchpdf papers.csv -o ./out --pull-supplementary
+
+# Raise or lower the per-file cap (default 300 MB)
+fetchpdf papers.csv -o ./out --pull-supplementary --max-supplementary-mb 500
+
+# Pick up files deposited since an earlier run
+fetchpdf papers.csv -o ./out --refresh-supplementary
+```
+
+Files arrive numbered — `{stem}_supplementary_info_1.xlsx`, `_2.docx`, … — with
+`{stem}_supplementary_info.json` recording what each number originally was, plus
+what was *skipped and why*. The numbering discards original filenames, so
+[read the manifest](#the-manifest-is-not-optional-reading). This runs for records
+whose PDF is already on disk, so it collects supplements for an old corpus
+without re-fetching a single paper, and a second run over the same directory
+costs zero requests.
+
+**3. Linked datasets and code**
+
+The supplementary pass also asks the link services (ScholeXplorer, Europe PMC
+datalinks, DataCite) what datasets and software each paper is connected to, and
+writes every answer to `{stem}_linked_artifacts.json`. Deposits affirmatively
+identified as *the paper's own* are downloaded and numbered alongside the
+supplements. `--download-data-artifacts` goes further:
+
+```bash
+# Supplements + the paper's own linked deposits (Zenodo, Dryad, OSF, figshare, Dataverse)
+fetchpdf papers.csv -o ./out --pull-supplementary
+
+# Also take citation-grade deposits whose own metadata names this article,
+# and repositories named in the paper's own full text
+fetchpdf papers.csv -o ./out --download-data-artifacts
+
+# Per-record dataset cap (default 500 MB), counted separately from supplements
+fetchpdf papers.csv -o ./out --download-data-artifacts --max-data-artifact-mb 2000
+
+# Expand replication-package zips (off by default — they have their own layout)
+fetchpdf papers.csv -o ./out --download-data-artifacts --unpack-data-artifacts
+```
+
+`--download-data-artifacts` implies `--pull-supplementary`. What is *not*
+downloaded still gets recorded: cited-but-not-owned datasets, tool citations,
+trial registrations. See [Linked datasets and code](#linked-datasets-and-code-stem_linked_artifactsjson).
+
+**All three at once**, which is the usual corpus-building invocation:
+
+```bash
+fetchpdf papers.csv -o ./out -w 4 \
+    --get-xml-or-html --to-markdown \
+    --download-data-artifacts \
+    --make-subfolder --provenance
+```
+
+```
+out/10.1371--journal.pone.0000308/
+  10.1371--journal.pone.0000308.pdf                     <- the paper
+  10.1371--journal.pone.0000308.xml                     <- structured full text
+  10.1371--journal.pone.0000308.md                      <- prose + HTML tables
+  10.1371--journal.pone.0000308_supplementary_info_1.xls
+  10.1371--journal.pone.0000308_supplementary_info_2.csv <- from a Dryad deposit
+  10.1371--journal.pone.0000308_supplementary_info.json  <- what each number is
+  10.1371--journal.pone.0000308_linked_artifacts.json    <- every dataset/code link
+  10.1371--journal.pone.0000308.provenance.json          <- source, tier, hashes
+```
+
+Supplements and datasets never change whether a record counts as a success: a
+paper with no SI is not a failure, and a repository timing out cannot turn a
+downloaded paper into a failed one.
+
+### Python API
+
+```python
+from fetchpdf import fetch_pdf
+
+# Download a PDF from a DOI (PMID also supported)
+doi = "10.1038/nature12373"
+save_path = "./papers/nature_paper.pdf"
+
+result = fetch_pdf(
+    doi=doi,
+    save_path=save_path,
+    verbose=True,
+    delay=0.1  # Polite delay between API calls
+)
+```
+
+The extra formats and the supplementary pass are keyword arguments on the batch
+entry point — `--pull-supplementary` is plumbed through `batch_fetch_pdfs` only,
+so use it even for a single DOI:
+
+```python
+from fetchpdf import batch_fetch_pdfs
+
+results = batch_fetch_pdfs(
+    dois=["10.1371/journal.pone.0000308"],
+    output_dir="./out",
+    get_xml_or_html=True,        # structured copy AND the PDF
+    to_markdown=True,            # plus {stem}.md
+    pull_supplementary=True,     # SI/SM as numbered siblings + manifest
+    download_data_artifacts=True,  # and the paper's linked deposits
+    max_supplementary_bytes=300 * 1024 * 1024,
+    want_provenance=True,
+    verbose=True,
+)
+# -> [(doi, success, save_path), ...]
+```
+
+To collect supplements for one record you already have on disk, call
+`pull_supplementary_for` directly — see [API Reference](#api-reference).
+
+
+
+### PMID Support
+
+You can provide a PMID anywhere a DOI is accepted. The tool first resolves PMID -> DOI (via NCBI), then runs the normal DOI download flow.
+
+**CLI examples:**
+```bash
+# Plain PMID
+fetchpdf "33262244" -o ./pdfs -v
+
+# PMID with prefix
+fetchpdf "PMID:33262244" -o ./pdfs
+
+# PubMed URL
+fetchpdf "https://pubmed.ncbi.nlm.nih.gov/33262244/" -o ./pdfs
+```
+
+If a PMID cannot be resolved to a DOI, that identifier will fail quickly and be reported in batch mode.
+
+**Batch Processing from CSV:**
+```bash
+# Sequential processing
+fetchpdf dois.csv -o ./pdfs
+
+# Parallel processing with 4 workers
+fetchpdf dois.csv -o ./pdfs -w 4
+
+# Custom DOI column name
+fetchpdf papers.csv --doi-column "paper_doi" -o ./pdfs -w 2
+```
+
+
+## Usage Examples
+
+### Example 1: Download Multiple Papers
+
+```python
+from fetchpdf import fetch_pdf
+import os
+
+dois = [
+    "10.1038/nature12373",
+    "10.1126/science.1241224",
+    "10.1016/j.cell.2019.05.031"
+]
+
+output_dir = "./papers"
+os.makedirs(output_dir, exist_ok=True)
+
+for doi in dois:
+    safe_doi = doi.replace("/", "--")
+    save_path = os.path.join(output_dir, f"{safe_doi}.pdf")
+
+    print(f"Downloading {doi}...")
+    result = fetch_pdf(doi, save_path, verbose=True)
+
+    if result:
+        print(f"✅ Success: {save_path}")
+    else:
+        print(f"❌ Failed: {doi}")
+```
+
+### Example 2: Batch Processing with Parallel Execution From CSV
+
+```python
+from fetchpdf import batch_fetch_pdfs
+
+# Parallel processing with 4 workers
+results = batch_fetch_pdfs(
+    dois="papers.csv",
+    output_dir="./downloaded_papers",
+    verbose=False,
+    workers=4,  # 4 parallel downloads
+    delay=0.2   # Delay per worker
+)
+
+# Get failed DOIs
+failed_dois = [doi for doi, success, _ in results if not success]
+print(f"Failed: {len(failed_dois)} DOIs")
+
+# Save failed DOIs for retry
+if failed_dois:
+    import csv
+    with open("failed_dois.csv", "w", newline="") as f:
+        writer = csv.writer(f)
+        writer.writerow(["DOI"])
+        writer.writerows([doi] for doi in failed_dois)
+```
+
+### Example 3: Batch Processing with DOI List
+
+```python
+from fetchpdf import batch_fetch_pdfs
+
+# Process a list of DOIs directly
+dois = [
+    "10.1038/nature12373",
+    "10.1126/science.1241224",
+    "10.1016/j.cell.2019.05.031"
+]
+
+results = batch_fetch_pdfs(
+    dois=dois,  # Pass list directly
+    output_dir="./papers",
+    workers=3  # Use 3 parallel workers
+)
+```
+
+
+### Example 3: Download with Metadata
+
+```python
+from fetchpdf import fetch_pdf, fetch_metadata_from_doi
+import os
+
+doi = "10.1038/nature12373"
+
+# Get metadata first
+metadata = fetch_metadata_from_doi(doi)
+print(f"Downloading: {metadata['title']}")
+
+# Use metadata for filename
+safe_title = metadata['title'][:50].replace(" ", "_").replace("/", "_")
+save_path = f"./papers/{safe_title}.pdf"
+
+# Download PDF
+result = fetch_pdf(doi, save_path)
+
+if result:
+    print(f"✅ Downloaded: {metadata['title']}")
+    print(f"   Authors: {metadata['authors']}")
+    print(f"   Year: {metadata['year']}")
+```
+
+### Example 4: Using the Missing PDFs Report
+
+When batch processing completes, an HTML report is automatically created for any failed downloads:
+
+```python
+from fetchpdf import batch_fetch_pdfs
+
+results = batch_fetch_pdfs(
+    dois="papers.csv",
+    output_dir="./papers",
+    workers=4
+)
+
+# After completion, check ./papers/missing_pdfs.html
+# The report includes:
+# - Paper title, authors, journal, year
+# - Clickable DOI links
+# - Expected filename for manual download
+```
+
+**The report helps you:**
+- Quickly identify which papers failed
+- Click DOI links to manually download
+- See expected filenames for manual organization
+- View paper metadata even without the PDF
+
+To disable the report:
+```python
+results = batch_fetch_pdfs(
+    dois="papers.csv",
+    output_dir="./papers",
+    create_missing_report=False  # Disable report
+)
+```
+
+Or via command line:
+```bash
+fetchpdf papers.csv -o ./papers --no-missing-report
+```
+
+## Batch Processing & Parallel Execution
+
+This package supports both **batch processing** and **parallel execution** for downloading multiple PDFs efficiently.
+
+### Batch Processing Capabilities
+
+#### 1. Process List of DOIs
+
+**Python API:**
+```python
+from fetchpdf import batch_fetch_pdfs
+
+dois = ["10.1038/nature12373", "10.1126/science.1241224", "10.1016/j.cell.2019.05.031"]
+
+results = batch_fetch_pdfs(
+    dois=dois,
+    output_dir="./papers",
+    workers=1  # Sequential
+)
+```
+
+#### 2. Process CSV File
+
+**CSV format:**
+```csv
+DOI
+10.1038/nature12373
+10.1126/science.1241224
+10.1016/j.cell.2019.05.031
+```
+
+**Python API:**
+```python
+from fetchpdf import batch_fetch_pdfs
+
+results = batch_fetch_pdfs(
+    dois="papers.csv",  # Path to CSV
+    output_dir="./papers",
+    workers=1
+)
+```
+
+**Command Line:**
+```bash
+fetchpdf papers.csv -o ./papers
+```
+
+#### 3. Custom DOI Column Name
+
+If your CSV uses a different column name:
+
+```bash
+fetchpdf data.csv --doi-column "paper_doi" -o ./pdfs
+```
+
+### Parallel Execution
+
+#### Why Use Parallel Execution?
+
+- **Faster downloads**: Download multiple PDFs simultaneously
+- **Better resource utilization**: Make use of network I/O waiting time
+- **Configurable workers**: Control parallelism based on your needs
+
+#### Performance Comparison
+
+**Example: Downloading 100 PDFs**
+
+| Workers | Time | Speedup |
+|---------|------|---------|
+| 1 (sequential) | ~500 seconds | 1x |
+| 2 workers | ~260 seconds | 1.9x |
+| 4 workers | ~140 seconds | 3.6x |
+| 8 workers | ~80 seconds | 6.3x |
+
+*Note: Actual speedup depends on network speed, API rate limits, and source availability.*
+
+#### Python API - Parallel
+
+```python
+from fetchpdf import batch_fetch_pdfs
+
+# Use 4 parallel workers
+results = batch_fetch_pdfs(
+    dois="papers.csv",
+    output_dir="./papers",
+    workers=4,  # 4 parallel downloads
+    delay=0.2   # Delay between API calls per worker
+)
+```
+
+#### Command Line - Parallel
+
+```bash
+# Use 4 parallel workers
+fetchpdf papers.csv -o ./papers -w 4
+```
+
+### Best Practices
+
+#### 1. Start with Sequential Processing
+
+For small batches (< 20 DOIs), sequential processing is sufficient:
+
+```python
+results = batch_fetch_pdfs(dois=dois, output_dir="./papers", workers=1)
+```
+
+#### 2. Use Moderate Parallelism
+
+For large batches, use 2-4 workers:
+
+```python
+results = batch_fetch_pdfs(dois=dois, output_dir="./papers", workers=4)
+```
+
+**Why not more?**
+- API rate limits
+- Respectful to servers
+- Diminishing returns beyond 4-8 workers
+
+#### 3. Increase Delay for Larger Workers
+
+More workers = more simultaneous requests. Be polite to APIs:
+
+```python
+results = batch_fetch_pdfs(
+    dois=dois,
+    output_dir="./papers",
+    workers=8,
+    delay=0.5  # Longer delay with more workers
+)
+```
+
+#### 4. Handle Failures and Retry
+
+```python
+# First attempt
+results = batch_fetch_pdfs(dois=dois, output_dir="./papers", workers=4)
+
+# Get failed DOIs
+failed_dois = [doi for doi, success, _ in results if not success]
+
+if failed_dois:
+    print(f"Retrying {len(failed_dois)} failed DOIs...")
+
+    # Retry with sequential processing and longer delay
+    retry_results = batch_fetch_pdfs(
+        dois=failed_dois,
+        output_dir="./papers",
+        workers=1,
+        delay=1.0,
+        verbose=True  # See what's happening
+    )
+```
+
+#### 5. Save Results for Tracking
+
+```python
+import csv
+
+results = batch_fetch_pdfs(dois="papers.csv", output_dir="./papers", workers=4)
+
+# Save results
+with open("download_results.csv", "w", newline="") as f:
+    writer = csv.writer(f)
+    writer.writerow(["DOI", "Success", "Path"])
+    writer.writerows(results)
+
+# Statistics
+successes = sum(1 for _, success, _ in results if success)
+print(f"Success: {successes}/{len(results)}")
+print(f"Success rate: {successes/len(results)*100:.1f}%")
+```
+
+## Format-Prioritized Retrieval
+
+The default chain walks *sources* and takes the first PDF each one yields. That
+is right for a paper you want to read, and wrong for feeding a table-extraction
+pipeline: what breaks there is not character accuracy but row/column/header
+association, and that only survives losslessly in markup.
+
+`--prioritize-xml` inverts the loop — format tiers outer, sources inner — so a
+JATS copy from a worse-ranked source beats a PDF from a better-ranked one.
+
+```bash
+# Prefer structured full text, fall back through the normal chain
+fetchpdf papers.csv -o ./out --prioritize-xml
+
+# Structured XML or nothing
+fetchpdf papers.csv -o ./out --xml-only
+
+# XML or publisher HTML, but never a PDF
+fetchpdf papers.csv -o ./out --xml-html-only
+
+# Re-run over an existing directory, writing only genuine upgrades
+fetchpdf papers.csv -o ./out --upgrade-existing
+
+# Audit sidecar per record
+fetchpdf papers.csv -o ./out --prioritize-xml --provenance
+```
+
+### Getting both formats, and Markdown for the model
+
+```bash
+# A structured copy AND the PDF for every record
+fetchpdf papers.csv -o ./out --get-xml-or-html
+
+# ...and a {stem}.md for each XML/HTML artifact
+fetchpdf papers.csv -o ./out --get-xml-or-html --to-markdown
+
+# Convert artifacts you already have, without re-downloading
+fetchpdf-md ./out
+```
+
+`--get-xml-or-html` doubles as a backfill: pointed at a directory of existing
+PDFs it fetches only the missing structured half and re-downloads nothing.
+
+### Re-running over a directory you already have
+
+There is no ledger and nothing records which flags built a directory. Skipping
+is decided per record from the filesystem: if an artifact already sits at the
+record's stem, that record is skipped. This is what makes an interrupted run
+resumable — re-run the same command and it picks up where it stopped — and it
+works identically flat or under `--make-subfolder`, since the two layouts differ
+by one path segment.
+
+The silent part is that a skip also skips the *choice*. A directory of papers
+you fetched last month can gain its supplementary material without re-fetching a
+single PDF, because the SI pass runs on the skip branch — but nothing says so.
+So a re-run that is about to skip records asks:
+
+```
+📁 1,284 of 5,000 records already have artifacts in ./out.
+
+   [s] skip them             download only the 3,716 still missing   (default)
+   [m] skip, but pull SI/SM  keep the papers, fetch supplementary material for all of them
+   [a] abort
+```
+
+Answering `m` is exactly `--pull-supplementary`: every PDF already on disk stays
+untouched, and records whose SI manifest is already written cost zero requests.
+
+`--on-existing {skip,supplement,ask}` pre-answers it. **The prompt never appears
+unless stdin and stdout are both terminals** — pipes, cron and CI get a one-line
+count and today's behaviour, so no unattended run can block on it. It is also
+suppressed when the answer is already known: `--pull-supplementary` given
+explicitly, `--abstract-only`, or the goal-aware flags (`--get-xml-or-html`,
+`--upgrade-existing` on the tiered path), which do not skip records at all.
+
+What is *not* remembered between runs: failures. `failed_dois.csv` is a report,
+never read back, so every re-run retries every previous failure. Feed it back in
+deliberately if you want only the retries.
+
+**Markdown output keeps tables as HTML, deliberately.** Markdown has no span
+mechanism, so a `<th colspan="2">` over two treatment arms collapses and every
+value to its right shifts one column — silently, in a way that still parses as a
+valid table. Prose becomes Markdown; every table stays canonical minimal HTML at
+its position in the document, with its caption and footnotes attached. Inline
+HTML is part of the CommonMark spec, so this is ordinary Markdown, not a hybrid.
+
+Every table is HTML, including simple ones with no spans. Mixing pipes and HTML
+would make a three-column header ambiguous — no spans, or spans lost in
+conversion? — and that ambiguity is worse than either format alone.
+
+Each `.md` opens with front matter recording table count, tables that were
+published as images (not machine-readable), figures referenced but not included,
+and separate prose/table token counts:
+
+```yaml
+---
+source_artifact: 10.3390--biom12111676.xml
+table_format: canonical-html
+tables: 2
+tables_not_machine_readable: 0
+figures_referenced_not_included: 1
+prose_tokens: 11980
+table_tokens: 189
+---
+```
+
+JATS references images and never contains them, so figures appear as visible
+placeholders rather than vanishing — a record whose key outcome sits in a forest
+plot should look incomplete rather than complete.
+
+### Choosing a flag
+
+| Flag | Accepts | Falls back to PDF? |
+|------|---------|--------------------|
+| `--prioritize-xml` | everything, best format first | yes |
+| `--xml-html-only` | T1 XML, T2 HTML | **no** — record fails |
+| `--xml-only` | T1 XML | **no** — record fails |
+| `--get-xml-or-html` | structured **and** PDF, both kept | yes, and keeps it |
+
+`--xml-only` and `--xml-html-only` both imply `--prioritize-xml`. Given both,
+`--xml-only` is stricter and wins (with a warning).
+
+### The tier ladder
+
+| Tier | Format | Why it sits here |
+|------|--------|------------------|
+| T1 | JATS/TEI XML | Cells, spans, headers and footnotes are markup. The only lossless tier. |
+| T2 | Publisher HTML | Real `<table>` with `<th>` and spans; at most OA publishers generated from the same JATS. |
+| T3 | LaTeX / e-print source | Lossless in principle, macro-hostile in practice. |
+| T4 | Structured supplements | Often the underlying data; coverage is partial. |
+| T5 | PDF | Structure must be reconstructed; introduces undetectable numeric corruption. |
+| T6 | Plain text | Flattening destroys row/column association **silently**. Screening only. |
+| T7 | Landing page | Last resort. |
+
+`--target-task` selects the ladder. `screening` ranks plain text 4th;
+`extraction` refuses it outright rather than degrading into it — every number is
+present but anchored to nothing, which yields confident, plausible, wrong
+arm-to-outcome assignments.
+
+Artifacts are written as suffixed siblings: `{stem}.xml`, `{stem}.fulltext.html`,
+`{stem}.source.tar.gz`, `{stem}.suppl.zip`, `{stem}.pdf`, `{stem}.txt`.
+
+### Reordering without editing code
+
+The ladder, the per-source capability map and the per-host rate limits live in
+[`fetchpdf/retrieval/ladder.json`](fetchpdf/retrieval/ladder.json). Point
+`FETCHPDF_LADDER` at your own copy to override it.
+
+### Validation
+
+HTTP 200 is not evidence of full text. Before a T1 artifact is accepted it must
+parse, have a populated `<body>`, clear a character threshold, and not be a
+publisher denial stub — NCBI returns those with status 200, full metadata, no
+body, and the explanation in an XML comment. T2 additionally requires at least
+one `<table>` with populated `<td>`, which is what rejects JS-rendered table
+containers and tables shipped as images. Failures demote and the walk continues;
+nothing is written unclassified or unvalidated.
+
+A PDF gets the same question asked of it: **is this the paper we asked for?**
+`%PDF` and a size floor are a file-type check, not an identity check, and the
+difference is not academic. A landing page's reference list is a list of other
+people's PDFs; when the article's own copy is unavailable — paywalled,
+bot-walled, moved — a link-scraping fallback will happily download one of them,
+and it arrives as a genuine `application/pdf` with a 200. That is how
+`10.1111/all.14949`, a Wiley allergy paper, came back as the USDA's 164-page
+*Dietary Guidelines for Americans*, a work it cites, saved next to the correct
+XML for the same record fetched in the same walk.
+
+So every retrieved PDF is now checked against the record it was fetched for
+(`retrieval/pdf_identity`). Any one signal is enough, because the documents
+that legitimately fail one are common:
+
+| signal | carries |
+|---|---|
+| the requested DOI in the front matter | essentially every version of record |
+| the article title on the front pages | accepted manuscripts (`nihms-*.pdf`), which print the title and not the publisher's DOI |
+| the title as a near match | Greek letters typeset in a Latin face, hyphenated line breaks, entity-mangled metadata |
+| the title embedded in the PDF's own metadata | documents whose text layer is too thin to carry one |
+
+The title costs no extra request: resolution has already memoised the Crossref
+payload by the time a PDF is judged.
+
+Refusals are kept distinct rather than collapsed into one message, because
+*"this is a different paper"* is a bug signal, *"this document has no text
+layer"* is a corpus-quality signal, and *"no PDF reader is installed"* is a
+broken install — and they send whoever reads the report to three different
+places. A document that says almost nothing machine-readable is reported as
+unreadable, never as somebody else's: a scanned paper often extracts to nothing
+but a library's download stamp, which is text, but not text that could ever
+have carried a title.
+
+The check refuses inside the candidate loop, not only at the end of the run.
+Refusing only at the end would downgrade "wrong PDF" to "no PDF" — the right
+copy is frequently two candidates further down, and the loop has to reach it.
+Artifacts already on disk are never re-judged, so re-running over a corpus will
+not delete files it did not fetch.
+
+Measured over 500 main-article PDFs from real corpora: 95.6% verified, 3.4%
+refused as a different document, 1.0% refused as illegible. The refusals were
+inspected by hand and were overwhelmingly genuine — publisher marketing pages
+("Why Publish in Spine?"), permissions boilerplate, a government citizens'
+charter, an R package manual, a JSTOR literature article, and cited documents
+like the one above.
+
+Because verification is not optional, **a PDF text engine is no longer
+optional**: `pypdf` is a base dependency and the CLI refuses to start without an
+engine rather than rejecting every PDF one at a time. PyMuPDF remains in the
+`text` extra as the faster reader and is preferred when installed.
+
+#### The right article, but not all of it
+
+A publisher's first-page preview is the hardest case, because every identity
+signal passes it: it carries the article's own DOI, its own title, its own
+journal furniture. It is simply two pages of seventeen, cut off mid-sentence,
+with nothing anywhere in the text admitting as much. The same is true of an
+article's own supplementary file, which is why an "APA supplemental PDF"
+route can hand back four pages of a thirteen-page record and look like success.
+
+Two structural signals catch these, both refusing with a `truncated` verdict
+that is deliberately distinct from "wrong article" — it *is* the right article,
+so the caller should keep walking the chain for a complete copy rather than
+conclude the source was bad:
+
+1. **Against the record's own structured full text**, when the walk already has
+   it. The strongest signal, because it compares the document with *itself* in a
+   format a paywall cannot truncate. Measured on records where both were
+   retrieved: complete PDFs scored 0.98, 1.02 and 1.04; a two-page Brill preview
+   of a thirty-two-page article scored **0.07**.
+2. **Against the printed page range.** Available far more often, since most
+   records never get a structured copy. Over 456 verified-correct corpus PDFs
+   the ratios run 0.22, 0.31, 0.33, 0.33, 0.42 and then jump to 0.73, so the 0.5
+   threshold sits in open space rather than on a slope.
+
+Two other signals were measured and **rejected** rather than shipped: 77.6% of
+known-good PDFs have no "References"/"Acknowledgements" marker in their last
+4,000 characters, and 14.8% never print their own last page number. Both would
+have thrown away one correct paper in seven.
+
+URLs containing `/previewpdf/` are skipped before any transfer — Human Kinetics
+and Brill both serve previews from that path — and a URL already refused for a
+record is not fetched again, since `try_landing_page_pdf_fallback` runs from
+nine call sites and several converge on the same publisher URL.
+
+#### Crossref under parallel load
+
+Verification needs a title for every record, which made metadata the most
+rate-limit-hungry step in the tool at exactly the moment large runs got faster.
+Crossref publishes its allowance in every response — `x-rate-limit-limit: 10`,
+`x-rate-limit-interval: 1s` — and three things now keep a `--workers 10` run
+inside it:
+
+- **One process-wide token bucket — genuinely one.** The tiered engine, the
+  supplementary pass and the legacy chain each used to build their own
+  `HostRateLimiter`, so a run touching two of them politely allowed 10/s twice
+  against an allowance of 10/s. They now share a single limiter
+  (`ratelimit.shared_host_limiter`), and every Crossref call in the package
+  draws from the same bucket. Politeness is a property of the process, not of a
+  batch.
+- **A 429 slows every worker.** The bucket is drained by `Retry-After` rather
+  than the receiving thread sleeping alone — otherwise one worker backs off
+  while the other nine keep spending the budget that just ran out.
+- **Titles fetched in bulk, before any worker starts.** Crossref accepts a
+  comma-joined `filter=doi:...` list; batches of 50 with `select=DOI,title,page`
+  return every item in about a fifth of a second. A 10,000-record run needs
+  **200 metadata requests instead of 10,000**.
+- **429 means wait, not "no such record".** `Retry-After` is honoured (clamped
+  to 60s), with bounded exponential backoff behind it. And **"not in Crossref"
+  is not "no title"** — Crossref does not register Zenodo, OSF, figshare or
+  Dryad DOIs, so those fall through to DataCite rather than being recorded as
+  titleless.
+
+Measured on 120 records at `--workers 10`: **19 Crossref requests, zero 429s,
+peak 9 req/s.**
+
+The bug that motivated this was worse than slowness. A failed lookup used to be
+cached as "this record has no title" — so one 429 became permanent, and since a
+record with no title cannot be verified and an unverifiable PDF is deleted, a
+brief rate-limit blip **deleted correct files**. Now the two are kept strictly
+apart: a 404 is an answer and is cached, while a 429 or a timeout is not an
+answer and is never cached. Deleting now requires a **positive** finding (`wrong_article` or `truncated`)
+**and** metadata we actually obtained. `no_reference`, `unreadable` and
+`no_engine` all mean "nobody checked", which is not a statement about the file;
+each of them used to delete it. If metadata cannot be fetched at all, the PDF is
+**kept unverified rather than deleted** — a wrong file kept can be found again
+by re-running the audit, a right file deleted cannot — and every such record is
+named at the end of the run, because they are the only artifacts in it that
+nobody checked.
+
+#### When there is no PDF, there may still be the paper
+
+A record whose PDF cannot be had falls back to structured full text **even when
+it was not asked for**. That is not a consolation prize: XML is tier 1 on the
+extraction ladder and the PDF is tier 5. "No PDF" and "no full text" are
+different answers, and only one of them is worth a human's time. The fallback is
+restricted to T1/T2, so it cannot re-enter the chain that called it, and
+`--no-xml-fallback` opts out.
+
+In practice this is what a bot-walled or preview-only record now returns: the
+Brill record above yields 715 KB of publisher HTML where it previously yielded
+two pages of PDF, and `10.1111/all.14949` yields the full JATS that NCBI efetch
+serves for records Europe PMC's OA route refuses.
+
+### T2 HTML support
+
+Publisher HTML needs a forgiving parser:
+
+```bash
+pip install 'fetchpdf[html]'
+```
+
+Without it the T2 rung demotes with a logged reason and the ladder descends
+normally. Every other tier needs only the standard library.
+
+## Supplementary Material
+
+`--pull-supplementary` fetches everything the authors deposited *alongside* the
+paper — supplementary PDFs, spreadsheets, documents, images, archives, raw data —
+and saves it as flat siblings of the main artifact.
+
+```bash
+fetchpdf papers.csv -o ./out --pull-supplementary
+fetchpdf papers.csv -o ./out --pull-supplementary -w 4
+fetchpdf "10.1371/journal.pone.0000308" -o ./out --pull-supplementary
+```
+
+```
+out/
+  10.1371--journal.pone.0000308.pdf
+  10.1371--journal.pone.0000308_supplementary_info_1.doc
+  10.1371--journal.pone.0000308_supplementary_info_2.doc
+  10.1371--journal.pone.0000308_supplementary_info_3.txt
+  10.1371--journal.pone.0000308_supplementary_info_4.xls
+  10.1371--journal.pone.0000308_supplementary_info.json    <- the manifest
+```
+
+It works with or without `--prioritize-xml`, and it runs for records whose PDF is
+already on disk — so pointing it at a corpus you downloaded months ago collects
+the supplements for all of it without re-fetching a single paper.
+
+### The manifest is not optional reading
+
+Flat numbering discards the original filenames, so
+`{stem}_supplementary_info.json` is the only record of what `_2` actually is:
+
+```json
+{"index": 2, "filename": "..._supplementary_info_2.xls",
+ "original_name": "pone.0000308.s004.xls", "label": "Table S1",
+ "provider": "europepmc_supplements", "container": "PMC1817752_SupplementaryFiles.zip",
+ "bytes": 42496, "sha256": "9f2c...", "role": "supplement"}
+```
+
+It also records what was *not* taken and why — `too-large` with the declared byte
+count, `duplicate-of` with the index it duplicates, `not-a-document` for a
+Cloudflare interstitial served as HTTP 200. "This record has no supplementary
+material" and "one 4 GB HDF5 was refused" are different facts, and the manifest is
+where they stay distinguishable.
+
+The manifest is also the skip signal: a second run over the same output directory
+costs zero HTTP requests. Delete a numbered file and the next run restores it *at
+its original index*, so removing `_2` never renumbers `_3`. Use
+`--refresh-supplementary` to re-enumerate and pick up newly deposited files.
+
+### Where the files come from
+
+Fifteen routes, tried in a fixed order so the numbering is reproducible:
+
+| Route | Applies to |
+|---|---|
+| JATS `<supplementary-material>` | any PMC record — *classifies*, telling supplements from figures |
+| Europe PMC `supplementaryFiles` | any PMCID (asked with `includeInlineImage=no`, or it returns every figure too) |
+| PMC AWS Open Data | the PMC OA subset; declares size and md5 before transfer |
+| Elsevier object API | Elsevier DOIs — the only ScienceDirect route that needs no browser |
+| Springer / Nature ESM | `10.1038`, `10.1007`, `10.1186`, … |
+| PLOS | `10.1371` |
+| bioRxiv / medRxiv | `10.1101` |
+| APA supplemental | `10.1037` |
+| Figshare / Zenodo / OSF / Dryad / Dataverse | repository DOIs, and datasets reached through link services |
+| DataCite reverse relations | any DOI — finds the deposited datasets that cite it |
+| Crossref component DOIs | publishers that register components |
+| OpenAIRE ScholeXplorer | any DOI — Scholix publication→dataset/software links |
+| Europe PMC datalinks | any PMID — text-mined accessions and data citations |
+
+Atypon (PNAS, Science, T&F, Sage), Wiley, ACS and MDPI serve HTTP 403 to plain
+requests and are not yet covered; arXiv ancillary files are not yet covered
+either. Dryad (`10.5061`) and Harvard Dataverse (`10.7910/DVN`) deposits are
+reached through the link services rather than as top-level providers.
+
+### Size cap
+
+`--max-supplementary-mb` defaults to 300, per file. A file whose `Content-Length`
+exceeds it is skipped without downloading anything; a server that under-reports is
+aborted mid-stream and its partial removed. Nothing is ever truncated — a
+truncated `.xlsx` is a corrupt zip that some readers open far enough to yield
+wrong numbers, which is worse than not having the file.
+
+### It never changes whether a record succeeded
+
+A paper with no supplementary material is not a failure, and a repository serving
+malformed JSON cannot turn a successful download into a failed one. Supplementary
+files never appear in `failed_dois.csv`, in the missing-PDFs report, in
+`source_tracking.csv`, or in the format-composition tally.
+
+### Linked datasets and code: `{stem}_linked_artifacts.json`
+
+The two link services (ScholeXplorer, Europe PMC datalinks) discover more than
+they download, and the rest lands in a second sidecar. Every
+publication→dataset and publication→software link they returned is recorded
+there with a classification:
+
+- `owned` — affirmatively the paper's own material: a supplement-grade
+  relation (`IsSupplementTo` and friends), or a repository deposit whose own
+  DataCite record names this article
+- `related` — everything citation-grade: datasets/software the paper cites,
+  text-mined data citations that no metadata ties back to the article, and
+  repositories with no file enumerator
+- `tool_citation` — "xgboost software on GitHub" and friends: somebody's tool,
+  not this paper's code. Recorded, never downloaded.
+- `registry` — ClinicalTrials.gov and other registrations; a link, not a file
+
+**Only `owned` links are downloaded.** A "cites" link cannot distinguish the
+paper's own deposit from a dataset the paper merely cites, so citation-grade
+links earn a download only when the deposit's DataCite record names the
+article — otherwise they stay sidecar links for the operator to judge.
+Ownership-confirmed deposits in Figshare, Zenodo, OSF, Dryad or Harvard
+Dataverse route to those enumerators and land as numbered supplements
+(`routed_to_download: true` in the sidecar). The same deposit named by several
+link services is listed and downloaded once.
+
+Empty answers are recorded too — `"queried": [...]` with zero links is a
+different fact from a sidecar that does not exist. On the corpora this was
+measured against (2026-08), most papers have no links at all: ~30% of a recent
+biomedical batch had any ScholeXplorer link, an older clinical-trial corpus had
+none, and Europe PMC's text-mined accessions were the widest net. The sidecar
+is written by the same `--pull-supplementary` pass, after the manifest; a run
+that skips enumeration (manifest already present, no `--refresh-supplementary`)
+does not rewrite it.
+
+#### Where dataset links come from, and what can be downloaded
+
+Discovery — the link services queried per record:
+
+| Service | What it contributes |
+|---|---|
+| OpenAIRE ScholeXplorer | Scholix links aggregated from Crossref, DataCite, EMBL-EBI, repositories, plus OpenAIRE's text-mining of ~14M PDFs |
+| Europe PMC datalinks | Text-mined accessions (trial registrations, GEO/SRA/PDB…), DOI data citations, BioStudies deposits — the widest net for biomedical papers (needs a PMID) |
+| DataCite reverse query | Deposits whose own metadata names the article — depositor-declared ground truth |
+
+Download — repositories with file enumerators (ownership-confirmed links only):
+
+| Repository | Routed on | Notes |
+|---|---|---|
+| figshare | `10.6084` / "figshare" | declared size + md5 before transfer |
+| Zenodo | `10.5281` / "zenodo" | restricted records skipped |
+| OSF | `10.17605` / "osf.io" | walks folder trees (depth/node caps) |
+| Dryad | `10.5061` / "dryad" | declared size + sha-256, paginated |
+| Harvard Dataverse | `10.7910/DVN` | declared md5; restricted files skipped |
+| BioStudies | — | content arrives via the Europe PMC supplementary bundle instead, never fetched twice |
+
+Not yet downloadable (recorded in the sidecar, left to the operator):
+Mendeley Data (`10.17632`), non-Harvard Dataverse installations, subject
+databases (GEO, SRA, PDB, UniProt — accession records, not file bundles),
+GitLab repositories, and the institutional-repository long tail.
+Adding an enumerator is the same small pattern as `enumerate_dryad` /
+`enumerate_dataverse` in `fetchpdf/retrieval/supplement_index.py`.
+
+## `--pull-everything`: the second layer
+
+Everything above is layer 1 — APIs, indexes and enumerators. It is very good at
+what publishers and repositories *declare*, and it cannot read. Layer 2 is a
+model that reads the paper itself and goes after what layer 1 did not get.
+
+```bash
+fetchpdf papers.csv -o ./out --get-xml-or-html --make-subfolder --provenance \
+         --pull-everything
+```
+
+`--pull-everything` implies `--pull-supplementary` and
+`--download-data-artifacts`, turns on adjudication of the deterministic scan's
+uncertain candidates, and adds the retrieval agent. Files land exactly where
+they always did: supplements as numbered siblings of the PDF, linked deposits
+in `{stem}_data_artifacts/`.
+
+**It is not literally everything, and the run says so.** Wiley, ACS and MDPI
+serve HTTP 403 to plain requests; Atypon needs the headed-browser path; arXiv
+ancillary files are uncovered; the repositories in the table above still have
+no enumerator. Those are recorded in the sidecar, not fetched.
+
+### What the agent is told
+
+Three things a model does not have and layer 1 does:
+
+1. the paper's own text, weighted toward its availability statements;
+2. what has already been obtained, so it does not fetch a second copy;
+3. **what the link services found and declined, with their reasons.** That is
+   the richest input it gets. Measured over four corpora, those services
+   returned 3,005 `related` links that were never routed, and of 280 `owned`
+   ones 264 were BioStudies mirrors of supplements already in hand.
+
+It proposes; the existing gates dispose. A proposal naming a repository routes
+through the same enumerator and the same ownership checks
+(`_should_route`, `_deposit_claims_another_article`) an index's link would, and
+anything it saves still meets `_commit`'s size cap, sha256, deduplication and
+challenge-page sniff. No failure of it can change whether a record succeeded.
+
+### `--llm-backend`: two sandboxes, one result
+
+| | `claude-cli` (default) | `openrouter` |
+|---|---|---|
+| auth | whatever the local Claude Code CLI already has | `OPENROUTER_API_KEY` in `.env.local` |
+| `--llm-model` | `haiku`, `sonnet`, … | the same names, translated to slugs |
+| tools | WebFetch only | `fetch_url` and `download`, both implemented here |
+| downloads? | **no** — it navigates and reports URLs, fetchpdf transfers | yes, into a staging directory |
+
+The asymmetry is not a preference, it is a measurement. Against Claude Code
+2.1.240:
+
+- **`--allowed-tools` grants permission; it does not restrict the tool set.** A
+  subprocess given `--allowed-tools "Bash(echo:*)"` ran
+  `cat ../outside_canary.txt` and returned the contents. So did
+  `--allowed-tools WebFetch`. With and without `--permission-mode
+  bypassPermissions`.
+- **`--disallowed-tools` does remove tools**, and was the only lever that
+  worked.
+- **A default subprocess inherits the operator's own MCP servers.** On the
+  machine this was written on, that put Gmail, Google Drive, Calendar and a
+  brokerage account in scope for a subprocess whose job is downloading
+  spreadsheets. `--strict-mcp-config` with an empty config removes them.
+
+So the CLI backend is locked to WebFetch and nothing else — verified by asking a
+locked-down subprocess to list its own tools, which answered `WebFetch` — and a
+process with no file tools cannot save a file. It reports URLs instead, which
+loses nothing: navigating is the part that needs a model, and transferring bytes
+under a cap with a hash is the part this package already does. The OpenRouter
+backend has no general-purpose harness to lock down, because its two tools are
+ours, so it downloads directly.
+
+### What it costs, and what to expect
+
+One model call per record. `--max-llm-records N` is the ceiling that stops an
+overnight batch spending without bound; records past it still get every API
+route, and the manifest records that the agent was *skipped* rather than that it
+found nothing.
+
+Measured on the corpora this was built against, **most calls will correctly find
+nothing**, and that is not a defect in the agent. Of 606 PDF-only records, 6
+name a repository at all and 17 carry a data-availability statement — the
+biomedical sets skew to 1991–2006, before data-sharing norms existed. Expect the
+yield on modern and social-science corpora instead: `fulltext_scan` alone
+produced 83 OSF files from ten such records, and the link indexes had surfaced
+none of the fourteen deposits those articles named in their own prose.
+
+Recall is also **model-dependent and not deterministic**. On one record whose
+GitHub URL is broken across a line by PDF layout — invisible to any regex —
+`haiku` recovered it on two runs out of four and returned nothing on the other
+two. Treat the agent as a second pass that sometimes finds what layer 1 missed,
+not as a guarantee.
+
+## API Reference
+
+### `fetch_pdf(doi, save_path, email, verbose, delay)`
+
+Download a PDF from a DOI (or a PMID that can be resolved to a DOI) using multiple fallback sources.
+
+**Parameters:**
+- `doi` (str): DOI or PMID identifier to download
+- `save_path` (str): Path where the PDF should be saved
+- `email` (str, optional): Email for API calls (default: `EMAIL` from `.env.local`)
+- `verbose` (bool, optional): Print detailed progress (default: False)
+- `delay` (float, optional): Delay between API calls in seconds (default: 0.1)
+
+**Returns:**
+- `str`: Path to downloaded PDF if successful, `None` otherwise
+
+### `batch_fetch_pdfs(dois, output_dir, email, verbose, delay, workers, create_missing_report)`
+
+Download PDFs for multiple DOI/PMID identifiers with optional parallel processing.
+
+**Parameters:**
+- `dois` (list or str): List of DOI/PMID identifiers or path to CSV file
+- `output_dir` (str): Directory to save PDFs
+- `email` (str, optional): Email for API calls (default: `EMAIL` from `.env.local`)
+- `verbose` (bool, optional): Print detailed progress (default: False)
+- `delay` (float, optional): Delay between API calls per worker (default: 0.1)
+- `workers` (int, optional): Number of parallel workers; 1 = sequential (default: 1)
+- `create_missing_report` (bool, optional): Create HTML report for failed downloads (default: True)
+
+**Returns:**
+- `list`: List of tuples `(doi, success, save_path)` for each DOI processed
+
+**Side Effects:**
+- Creates `missing_pdfs.html` in `output_dir` if any downloads fail (unless `create_missing_report=False`)
+
+### `pull_supplementary_for(raw_identifier, doi, save_path, ...)`
+
+Fetch every supplementary file for one record, as flat siblings of `save_path`.
+
+A separate call rather than a keyword on `fetch_pdf_from_doi`, deliberately: the
+tiered engine re-enters that function with a temporary path, so a flag threaded
+through it would write supplementary siblings next to a file that is about to be
+deleted. Two calls make that impossible rather than merely discouraged.
+
+```python
+from fetchpdf import fetch_pdf, pull_supplementary_for
+
+doi = "10.1371/journal.pone.0000308"
+path = fetch_pdf(doi, "out/paper.pdf")
+summary = pull_supplementary_for(doi, doi=doi, save_path="out/paper.pdf")
+print(summary.status, summary.written, summary.manifest_path)
+```
+
+**Key parameters:**
+- `raw_identifier` (str): the identifier as given (DOI or PMID)
+- `doi` (str, optional): the resolved DOI, when known
+- `save_path` (str): the main artifact's path; siblings and the manifest derive from its stem
+- `max_file_bytes` (int, optional): per-file cap (default 300 MB)
+- `refresh` (bool, optional): re-enumerate even when a manifest exists (default: False)
+- `resolver` / `http` (optional): reuse a batch's `BatchResolver` / `HttpClient` so the per-host rate budget stays shared
+
+**Returns:**
+- `SupplementarySummary`: `.status` (`ok` | `none_found` | `partial` | `error` | `skipped`), `.written`, `.skipped`, `.bytes_written`, `.manifest_path`, `.paths`
+
+Never raises for a network or provider failure, and its result must not be used to
+decide whether the record succeeded.
+
+### `fetch_metadata_from_doi(doi, email, delay)`
+
+Fetch metadata for a paper from its DOI.
+
+**Parameters:**
+- `doi` (str): The DOI to fetch metadata for
+- `email` (str, optional): Email for API calls (default: `EMAIL` from `.env.local`)
+- `delay` (float, optional): Delay between API calls in seconds (default: 0.2)
+
+**Returns:**
+- `dict`: Metadata dictionary with keys: `authors`, `title`, `journal`, `volume`, `issue`, `pages`, `year`, `url`
+
+## Download Sources (in order of priority)
+
+The tool tries multiple sources in sequence until a PDF is successfully downloaded. The order is optimized to check faster/more reliable sources first and preserve rate-limited API quotas.
+
+### Special DOI Handlers (Pattern-Matched)
+These run first if the DOI matches specific patterns:
+
+- **OSF** (Open Science Framework) - Projects (`10.17605/osf.io/*`) & Preprints (`10.31234/osf.io/*`)
+  - Direct download URLs, Playwright browser automation, API fallback
+- **SSRN** (Social Science Research Network) - `10.2139/ssrn.*`
+  - Playwright with Cloudflare bypass, download button detection
+- **Figshare** - `10.6084/m9.figshare.*`
+  - Figshare API for file metadata and download URLs
+- **PsychArchives** - `10.23668/psycharchives.*`
+  - Leibniz psychology repository bitstream extraction
+
+### Standard Fallback Chain (All DOIs)
+
+1. **PubMed Central (PMC)** via Europe PMC
+   - Open access papers via NCBI idconv → Europe PMC PDF endpoint
+   - Fast, reliable for biomedical papers
+
+2. **Unpaywall**
+   - Comprehensive legal open access aggregator
+   - Highly reliable, requires email
+
+3. **Crossref**
+   - Publisher metadata with direct PDF links
+   - Landing page scraping with `citation_pdf_url` extraction
+   - **NEW: Crossref chooser page handler** for multi-resolution DOIs
+   - Publisher-specific URL patterns (Wiley, T&F, SAGE, MIT Press, etc.)
+
+4. **Europe PMC**
+   - European PubMed Central search API
+   - Complementary to PMC direct access
+
+5. **Semantic Scholar**
+   - AI-powered academic search with `openAccessPdf` field
+   - Landing page fallback for OJS sites
+   - PsyArXiv→OSF URL conversion
+
+6. **OpenAlex** ⚠️ **Rate Limited**
+   - **1,000 downloads/day limit** (even with API key)
+   - Placed after other sources to preserve quota for harder-to-find papers
+   - Comprehensive metadata aggregator with open access locations
+
+7. **CORE** (core.ac.uk)
+   - 40M+ open access papers from repositories worldwide
+   - Search API with download URLs
+
+8. **Direct DOI Resolver**
+   - Follows `https://doi.org/{doi}` redirect to landing page
+   - Handles Crossref chooser pages (multiple resolution)
+   - Scrapes PDF links from HTML (`citation_pdf_url`, href patterns)
+   - Publisher-specific deterministic URL patterns
+
+9. **DataCite Related Identifiers**
+   - Supplementary material → main paper fallback
+   - Versioned DOI resolution (`IsIdenticalTo`, `IsVersionOf`)
+   - Figshare supplement handling
+
+10. **ResearchGate**
+    - DuckDuckGo search: `title + site:researchgate.net`
+    - Landing page PDF extraction
+
+11. **DOI → PMID Fallback**
+    - Convert DOI to PMID when DOI sources fail
+    - Try PMID-native sources (PubMed landing page `citation_pdf_url`)
+
+
+### Final Fallback
+
+12. **Elsevier XML API**
+    - Text/data-mining API for Elsevier articles
+    - Returns XML instead of PDF (last resort)
+    - Requires `ELSEVIER_TDM_API_KEY`
+
+---
+
+**Total: 12 standard sources + 4 special handlers** = 16 different download strategies
+
+## Requirements
+
+- Python 3.8+
+- requests
+- playwright
+- duckduckgo-search
+
+## Troubleshooting
+
+### Rate Limiting
+
+If you're downloading many papers, consider:
+- Increasing the `delay` parameter (e.g., `delay=1.0`)
+- Using the batch downloader with controlled parallelism
+
+
+## Ethical Considerations
+
+This tool is intended for:
+- ✅ Accessing open access papers
+- ✅ Retrieving papers you have legal access to
+- ✅ Academic research and education
+- ✅ Fair use purposes
+
+Please respect copyright laws and publisher terms of service in your jurisdiction.
+
+## Acknowledgments
+
+This package aggregates access to multiple open academic resources. Thanks to:
+- OpenAlex, Unpaywall, Crossref, PubMed Central, Semantic Scholar, and other open science initiatives
+- The Playwright team for browser automation tools