medsci-skills 5.0.0 → 5.1.0

package/metadata/distribution_files.json

@@ -1488,23 +1488,58 @@
     },
     {
       "path": "skills/fulltext-retrieval/SKILL.md",
-      "size": 5830,
-      "sha256": "0bdf94398af3d9f144b0b58fbce643d05f5486329472681a39090f135d3ec63f"
+      "size": 8248,
+      "sha256": "d97c19a64d92f66e3c8a1445d893c0f55364f21dc99404fb6f012a3566556e73"
     },
     {
       "path": "skills/fulltext-retrieval/fetch_oa.py",
-      "size": 15882,
-      "sha256": "b33feb91c7258d1986fbb6168a044b2abe21f3ba1cacadd3a52a38689477f86c"
+      "size": 24503,
+      "sha256": "768f95a30737e67b7f9edee72fe8815a795c9f83451b46551bbe25fc36094f4b"
+    },
+    {
+      "path": "skills/fulltext-retrieval/fetch_oa_report_challenge/expected/projection.json",
+      "size": 522,
+      "sha256": "f24bdb507f66932c0429acefc171abf5221bd644cacdeba5cd7b5c07670b04aa"
+    },
+    {
+      "path": "skills/fulltext-retrieval/fetch_oa_report_challenge/extracted_text.json",
+      "size": 351,
+      "sha256": "9b512bbea3780727e79355dd4b3689be182f086d4afb4184737524449fc44e4d"
+    },
+    {
+      "path": "skills/fulltext-retrieval/fetch_oa_report_challenge/results.json",
+      "size": 175,
+      "sha256": "e46d52e6a9c038ebd5cb8ce89a5961f32c8365e80d189d3ee80bed021a60296a"
+    },
+    {
+      "path": "skills/fulltext-retrieval/fetch_oa_report_challenge/run_challenge.py",
+      "size": 3138,
+      "sha256": "a3c194bb31f319bdd011d4436e5de22cfcc79b7979fbcba4883ed370e4acbeb4"
+    },
+    {
+      "path": "skills/fulltext-retrieval/fetch_oa_report_challenge/verify.sh",
+      "size": 286,
+      "sha256": "f96687fa9d1b213734810dcdfa8126a2efd1af106f5d1cdeb43d58b431ab915b"
+    },
+    {
+      "path": "skills/fulltext-retrieval/fetch_oa_report_challenge/worklist.tsv",
+      "size": 326,
+      "sha256": "d1c4c906052be198275155ee3e9ecf29b04ec8bdfeef97816ca15aa8cefa61fa"
     },
     {
       "path": "skills/fulltext-retrieval/pdf_to_md.py",
       "size": 5398,
       "sha256": "16ba8c61db254b4b356d85686946351daa5ea650d70dca2fd0d5677851c6df4d"
     },
+    {
+      "path": "skills/fulltext-retrieval/references/find_available_pdf.js",
+      "size": 3057,
+      "sha256": "04dc6d13d0bb7c43679e5386f1788edee6b187b2503cd9e250ae4ea63e11cec0"
+    },
     {
       "path": "skills/fulltext-retrieval/skill.yml",
-      "size": 1712,
-      "sha256": "43f84eeeeedb4676fb80fdca2ac0a84f6a90901b1d4e1d88359a532bb4ead835"
+      "size": 2420,
+      "sha256": "7a457e5f5fde09f57a8ef9d2207dbaab241e56a9bca59ba577d26351bf225346"
     },
     {
       "path": "skills/generate-codebook/SKILL.md",
@@ -1563,8 +1598,8 @@
     },
     {
       "path": "skills/lit-sync/SKILL.md",
-      "size": 16378,
-      "sha256": "e59b10d8dfdb52318a212e42d403a4372435697880e45643d3864f1374c7e82a"
+      "size": 21126,
+      "sha256": "4ba2d2b6dd4704803c0922993401b837d4135ac1d655d6064229315537a8e818"
     },
     {
       "path": "skills/lit-sync/references/locale/ko/note_templates.md",
@@ -1573,8 +1608,8 @@
     },
     {
       "path": "skills/lit-sync/skill.yml",
-      "size": 2522,
-      "sha256": "5f656fae91ba46d54f9d619785607bfb57773abc6a872207435ecf7a7315816b"
+      "size": 2780,
+      "sha256": "a04d160e9fc6382c783940b737c9d9348db45e34ea80c69ef905be0a3c357e34"
     },
     {
       "path": "skills/ma-scout/SKILL.md",
@@ -3238,8 +3273,8 @@
     },
     {
       "path": "skills/search-lit/SKILL.md",
-      "size": 21637,
-      "sha256": "35809486f31f1a26288a27ee0b1383760cf97e0287916ace2e9097eef88182e4"
+      "size": 20281,
+      "sha256": "608644e83ffc357f3b43b65cf7c439c9381f5948c51798672929f7a38aea5024"
     },
     {
       "path": "skills/search-lit/references/parse_pubmed.py",

package/metadata/distribution_manifest.json

@@ -1,6 +1,6 @@
 {
   "schema_version": 1,
-  "version": "5.0.0",
+  "version": "5.1.0",
   "owned_skills": [
     "academic-aio",
     "add-journal",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "medsci-skills",
-  "version": "5.0.0",
+  "version": "5.1.0",
   "description": "MedSci Skills — a medical/scientific research skill suite for AI coding agents (Claude Code, Codex, Cursor, Copilot). The npm package is a terminal-friendly installer shortcut; the canonical distribution remains the GitHub repository and the Claude Code plugin marketplace.",
   "license": "SEE LICENSE IN LICENSE",
   "homepage": "https://github.com/Aperivue/medsci-skills#readme",

package/skills/fulltext-retrieval/SKILL.md

@@ -13,10 +13,10 @@ Batch download open-access full-text PDFs from a DOI list using legitimate OA AP
 ## Pipeline
 
 ```
-DOI list → Unpaywall → PMC (Europe PMC / OA FTP / web) → OpenAlex → Crossref → landing page
+DOI → arXiv (10.48550/arXiv.* DOIs) → Unpaywall → PMC (Europe PMC / OA FTP / web) → OpenAlex → Crossref → landing page
 ```
 
-Each DOI goes through these sources in order until a valid PDF (≥10 KB, `%PDF-` header) is found.
+Each DOI goes through these sources in order until a valid PDF (≥10 KB, `%PDF-` header) is found. arXiv DOIs (`10.48550/arXiv.2401.01234`, version suffixes, old-style `hep-th/9901001`, or a bare `arXiv:` id) resolve directly to the arXiv PDF first.
 
 ## Quick Start
 
@@ -43,13 +43,20 @@ python fetch_oa.py dois.txt -o pdfs/ -e your@email.com --verbose
 10.1002/mp.12524
 ```
 
-**TSV with header** — must contain a `DOI` column, optional `PMID` column:
+**TSV / CSV with header** — must contain a `DOI` column; optional `PMID` and `Title` columns:
 ```tsv
 ID	Title	DOI	PMID	Year
 1	Some paper	10.1007/s00330-010-1783-x	20628747	2010
 ```
 
-When a PMID is available, the PMC lookup is more reliable (PMID → PMCID conversion).
+**Markdown table** — a pipe table with a `DOI` column also works:
+```markdown
+| DOI | PMID | Title |
+|-----|------|-------|
+| 10.1007/s00330-010-1783-x | 20628747 | Some paper |
+```
+
+When a PMID is available, the PMC lookup is more reliable (PMID → PMCID conversion). When a `Title` column is present, downloaded PDFs get a best-effort title cross-check (see *Retrieval report* below).
 
 ## PMC Download (JS-Challenge Resistant)
 
@@ -82,8 +89,48 @@ curl -s "https://www.ncbi.nlm.nih.gov/pmc/utils/idconv/v1.0/?ids=${DOI}&format=j
 ## Output
 
 - PDFs saved as `{DOI_safe}.pdf` (slashes replaced with underscores)
+- `pdfs/retrieval_report.json` — structured per-DOI report (see below)
 - `manual_needed.txt` — DOIs that could not be retrieved via OA
-- Summary with OA/PMC/fail/skip counts
+- Summary with arXiv/OA/PMC/fail/skip counts
+
+## Retrieval report (`--report`)
+
+Every run writes a structured report (default `<output>/retrieval_report.json`,
+override with `--report PATH`):
+
+```json
+{
+  "schema_version": 1,
+  "generated_by": "fetch_oa.py",
+  "counts": {"total": 10, "retrieved": 6, "not_retrieved": 4, "title_mismatch": 1},
+  "items": [
+    {"doi": "10.1007/...", "pmid": "20628747", "title": "...",
+     "status": "oa", "source": "unpaywall", "file": "10.1007_....pdf",
+     "size_bytes": 482113, "title_match": "match"}
+  ]
+}
+```
+
+- `status` ∈ `arxiv | oa | pmc | skip | fail`; `source` names the resolver that succeeded.
+- `title_match` ∈ `match | mismatch | unavailable` (tri-state). It is **best-effort**:
+  it needs a `Title` column **and** `pdftotext` (poppler). When either is missing it is
+  `unavailable`; a `mismatch` is **flagged** for review and **never** auto-rejects a PDF
+  (guards against a publisher serving a wrong/redirect PDF that still passes the `%PDF-` check).
+
+## Attach PDFs into Zotero ("Find Available PDF")
+
+OA-only resolvers miss paywalled-but-licensed papers. To attach full text **inside
+Zotero** at a much higher yield, use `references/find_available_pdf.js` — a user-run
+snippet for Zotero's *Tools → Developer → Run JavaScript*. It triggers Zotero's own
+`addAvailablePDF` / `addAvailablePDFs` and therefore reuses **your** OpenURL resolver /
+institutional proxy config; **no credentials, proxy hosts, or institutional identifiers
+are hard-coded or leave your Zotero client**. The no-code equivalent is right-click →
+"Find Available PDF".
+
+This path is **user-initiated** and depends on your live Zotero session, so its results
+are recorded manually (not reproducible CI evidence). `/lit-sync` Phase 2.7 orchestrates
+both routes (disk OA via this script + in-library via the snippet) and reconciles them in
+a report.
 
 ## Requirements
 

package/skills/fulltext-retrieval/fetch_oa.py

@@ -2,20 +2,28 @@
 """
 Open-access full-text PDF batch retrieval.
 
-Pipeline: Unpaywall → PMC (Europe PMC REST / OA FTP / web) →
-          OpenAlex → Crossref → landing-page scrape.
+Pipeline: arXiv (for 10.48550/arXiv.* DOIs) → Unpaywall →
+          PMC (Europe PMC REST / OA FTP / web) → OpenAlex → Crossref →
+          landing-page scrape.
 
 Usage:
     python fetch_oa.py dois.txt --output pdfs/ --email user@example.com
-    python fetch_oa.py dois.txt -o pdfs/ -e user@example.com --verbose
+    python fetch_oa.py worklist.tsv -o pdfs/ -e user@example.com --verbose
+    python fetch_oa.py worklist.csv -o pdfs/ -e user@example.com --report pdfs/retrieval_report.json
+
+Worklist formats: plain DOI-per-line, or TSV/CSV/Markdown-table with a DOI
+column (optional PMID and Title columns). A Title column enables a best-effort
+title cross-check (via `pdftotext` if installed) that flags mislabeled PDFs.
 """
 
 import argparse
+import csv
+import io
 import json
 import logging
-import os
 import re
-import sys
+import shutil
+import subprocess
 import time
 import urllib.error
 import urllib.parse
@@ -25,6 +33,9 @@ from pathlib import Path
 
 MIN_PDF_BYTES = 10 * 1024
 USER_AGENT = "medsci-skills/1.0"
+REPORT_SCHEMA_VERSION = 1
+TITLE_MATCH_THRESHOLD = 0.6
+RETRIEVED_STATUSES = ("oa", "pmc", "arxiv", "skip")
 
 log = logging.getLogger("fetch_oa")
 
@@ -38,6 +49,11 @@ def _ua(email: str) -> str:
     return f"{USER_AGENT} (mailto:{email})"
 
 
+def safe_doi_name(doi: str) -> str:
+    """Filesystem-safe filename stem for a DOI."""
+    return re.sub(r"[^\w\-.]", "_", doi)
+
+
 def is_valid_pdf(data: bytes) -> bool:
     return data.startswith(b"%PDF-") and len(data) >= MIN_PDF_BYTES
 
@@ -68,6 +84,84 @@ def existing_pdf_ok(path: Path) -> bool:
         return False
 
 
+# ============================================================
+# Title cross-check (pure, offline-testable)
+# ============================================================
+
+def normalize_title(text: str) -> str:
+    """Lowercase, strip punctuation, collapse whitespace."""
+    text = (text or "").lower()
+    text = re.sub(r"[^a-z0-9\s]", " ", text)
+    return " ".join(text.split())
+
+
+def title_overlap(expected_title: str, extracted_text: str) -> float:
+    """Fraction of meaningful expected-title tokens present in extracted text.
+
+    Tokens of length <= 2 are dropped as near-stopwords. Returns 0.0 when the
+    expected title has no usable tokens.
+    """
+    expected = {t for t in normalize_title(expected_title).split() if len(t) > 2}
+    if not expected:
+        return 0.0
+    got = set(normalize_title(extracted_text).split())
+    return len(expected & got) / len(expected)
+
+
+def classify_title_match(expected_title: str, extracted_text,
+                         threshold: float = TITLE_MATCH_THRESHOLD) -> str:
+    """Tri-state title check: 'match' | 'mismatch' | 'unavailable'.
+
+    'unavailable' when no expected title is supplied or no extracted text is
+    available (e.g. pdftotext absent). A low overlap is 'mismatch' — flagged,
+    never used to auto-reject a downloaded PDF.
+    """
+    if not expected_title or not extracted_text:
+        return "unavailable"
+    return "match" if title_overlap(expected_title, extracted_text) >= threshold else "mismatch"
+
+
+def extract_pdf_text(path: Path, max_pages: int = 2) -> str | None:
+    """Best-effort first-page text via `pdftotext` (poppler). None if unavailable."""
+    if not shutil.which("pdftotext"):
+        return None
+    try:
+        out = subprocess.run(
+            ["pdftotext", "-f", "1", "-l", str(max_pages), str(path), "-"],
+            capture_output=True, timeout=20,
+        )
+        if out.returncode == 0:
+            return out.stdout.decode("utf-8", errors="ignore")
+    except (OSError, subprocess.SubprocessError):
+        pass
+    return None
+
+
+# ============================================================
+# arXiv (direct, for 10.48550/arXiv.* DOIs)
+# ============================================================
+
+_ARXIV_DOI_RE = re.compile(r"^10\.48550/arxiv\.(.+)$", re.IGNORECASE)
+_ARXIV_ID_RE = re.compile(r"^arxiv:(.+)$", re.IGNORECASE)
+
+
+def arxiv_id_from_doi(doi: str) -> str | None:
+    """Extract an arXiv ID from a DataCite arXiv DOI or a bare arXiv: id.
+
+    Handles new-style (2401.01234, 2401.01234v2) and old-style
+    (hep-th/9901001) identifiers; version suffix preserved when present.
+    """
+    s = (doi or "").strip()
+    m = _ARXIV_DOI_RE.match(s) or _ARXIV_ID_RE.match(s)
+    return m.group(1).strip() if m else None
+
+
+def arxiv_pdf_url(doi: str) -> str | None:
+    """Direct arXiv PDF URL for an arXiv DOI/ID (None if not an arXiv id)."""
+    aid = arxiv_id_from_doi(doi)
+    return f"https://arxiv.org/pdf/{aid}" if aid else None
+
+
 # ============================================================
 # 1. Unpaywall
 # ============================================================
@@ -270,41 +364,34 @@ def download_pdf(url: str, outpath: Path, email: str) -> bool:
 # 5. Main pipeline
 # ============================================================
 
-def gather_candidates(doi: str, email: str) -> list[str]:
-    """Collect OA PDF candidate URLs from multiple sources."""
-    urls: list[str] = []
-
-    def add(v: str | None):
-        if v and v not in urls:
-            urls.append(v)
-
-    add(unpaywall_lookup(doi, email))
-    for v in openalex_lookup(doi, email):
-        add(v)
-    for v in crossref_lookup(doi, email):
-        add(v)
-    add(f"https://doi.org/{doi}")
-    return urls
-
-
 def process_doi(doi: str, outdir: Path, email: str,
-                pmid: str = "") -> str:
-    """Try to download a PDF for one DOI. Returns status string."""
-    safe_name = re.sub(r"[^\w\-.]", "_", doi)
-    outpath = outdir / f"{safe_name}.pdf"
+                pmid: str = "") -> tuple[str, str]:
+    """Try to download a PDF for one DOI.
+
+    Returns (status, source):
+      status ∈ {"arxiv", "oa", "pmc", "skip", "fail"}
+      source identifies the resolver that succeeded (e.g. "unpaywall", "pmc",
+      "openalex", "crossref", "landing", "arxiv", "existing", "").
+    """
+    outpath = outdir / f"{safe_doi_name(doi)}.pdf"
 
     if existing_pdf_ok(outpath):
-        return "skip"
+        return ("skip", "existing")
 
     # Remove stale stub
     if outpath.exists():
         outpath.unlink(missing_ok=True)
 
+    # Step 0: arXiv direct (for 10.48550/arXiv.* DOIs)
+    ax_url = arxiv_pdf_url(doi)
+    if ax_url and download_pdf(ax_url, outpath, email):
+        return ("arxiv", "arxiv")
+
     # Step 1: Unpaywall direct PDF URL (fastest path)
     uw_url = unpaywall_lookup(doi, email)
     if uw_url and ".pdf" in uw_url.lower():
         if download_pdf(uw_url, outpath, email):
-            return "oa"
+            return ("oa", "unpaywall")
         time.sleep(0.3)
 
     # Step 2: PMC (try before slow landing-page scraping)
@@ -312,59 +399,158 @@ def process_doi(doi: str, outdir: Path, email: str,
     if not pmcid:
         pmcid = id_to_pmcid(doi, email)
     if pmcid and download_pmc_pdf(pmcid, outpath, email):
-        return "pmc"
+        return ("pmc", "pmc")
 
     # Step 3: OA candidates from OpenAlex, Crossref, landing pages
-    candidates: list[str] = []
-    if uw_url and uw_url not in candidates:
-        candidates.append(uw_url)
+    candidates: list[tuple[str, str]] = []
+    seen: set[str] = set()
+
+    def add(source: str, url: str | None):
+        if url and url not in seen:
+            seen.add(url)
+            candidates.append((source, url))
+
+    add("unpaywall", uw_url)
     for v in openalex_lookup(doi, email):
-        if v not in candidates:
-            candidates.append(v)
+        add("openalex", v)
     for v in crossref_lookup(doi, email):
-        if v not in candidates:
-            candidates.append(v)
-    candidates.append(f"https://doi.org/{doi}")
+        add("crossref", v)
+    add("landing", f"https://doi.org/{doi}")
 
-    for url in candidates:
+    for source, url in candidates:
         if ".pdf" in url.lower():
             ok = download_pdf(url, outpath, email)
         else:
             ok = download_from_landing(url, outpath, email)
         if ok:
-            return "oa"
+            return ("oa", source)
         time.sleep(0.3)
 
-    return "fail"
+    return ("fail", "")
+
+
+def build_report(records: list[dict], results: dict[str, tuple[str, str]],
+                 outdir: Path, extracted_text_by_doi: dict[str, str] | None = None,
+                 threshold: float = TITLE_MATCH_THRESHOLD) -> dict:
+    """Assemble a deterministic retrieval report (no network, no I/O writes).
+
+    records: list of {"doi", "pmid", "title"}.
+    results: doi -> (status, source) as returned by process_doi.
+    outdir:  directory where PDFs were written (used for file/size lookup).
+    extracted_text_by_doi: optional doi -> first-page text for title cross-check.
+    """
+    extracted_text_by_doi = extracted_text_by_doi or {}
+    items = []
+    for rec in records:
+        doi = rec["doi"]
+        status, source = results.get(doi, ("fail", ""))
+        path = outdir / f"{safe_doi_name(doi)}.pdf"
+        have_file = status in RETRIEVED_STATUSES and path.exists()
+        size = path.stat().st_size if have_file else 0
+        if have_file:
+            title_match = classify_title_match(
+                rec.get("title", ""), extracted_text_by_doi.get(doi), threshold)
+        else:
+            title_match = "unavailable"
+        items.append({
+            "doi": doi,
+            "pmid": rec.get("pmid", ""),
+            "title": rec.get("title", ""),
+            "status": status,
+            "source": source,
+            "file": path.name if have_file else "",
+            "size_bytes": size,
+            "title_match": title_match,
+        })
+
+    retrieved = [i for i in items if i["status"] in RETRIEVED_STATUSES]
+    not_retrieved = [i for i in items if i["status"] == "fail"]
+    return {
+        "schema_version": REPORT_SCHEMA_VERSION,
+        "generated_by": "fetch_oa.py",
+        "counts": {
+            "total": len(items),
+            "retrieved": len(retrieved),
+            "not_retrieved": len(not_retrieved),
+            "title_mismatch": sum(1 for i in items if i["title_match"] == "mismatch"),
+        },
+        "items": items,
+    }
+
+
+def _norm_key(key: str) -> str:
+    return (key or "").strip().lstrip("#").strip().lower()
+
+
+def _records_from_dictrows(rows) -> list[dict]:
+    records = []
+    for row in rows:
+        rec = {"doi": "", "pmid": "", "title": ""}
+        for k, v in row.items():
+            nk = _norm_key(k)
+            if nk in rec:
+                rec[nk] = (v or "").strip()
+        if rec["doi"]:
+            records.append(rec)
+    return records
+
+
+def _records_from_markdown(lines: list[str]) -> list[dict]:
+    pipe_rows = [ln for ln in lines if ln.strip().startswith("|")]
+    if not pipe_rows:
+        return []
+
+    def cells(line: str) -> list[str]:
+        return [c.strip() for c in line.strip().strip("|").split("|")]
+
+    header = [_norm_key(c) for c in cells(pipe_rows[0])]
+    records = []
+    for line in pipe_rows[1:]:
+        c = cells(line)
+        # Skip the |---|---| separator row
+        if c and all(set(x) <= set("-: ") for x in c):
+            continue
+        row = dict(zip(header, c))
+        doi = (row.get("doi") or "").strip()
+        if doi:
+            records.append({
+                "doi": doi,
+                "pmid": (row.get("pmid") or "").strip(),
+                "title": (row.get("title") or "").strip(),
+            })
+    return records
 
 
 def read_doi_file(path: Path) -> list[dict]:
-    """Read DOI list. Supports plain DOIs or TSV with DOI/PMID columns."""
+    """Read a worklist of DOIs.
+
+    Supports: plain DOI-per-line; TSV/CSV with a DOI header (optional PMID,
+    Title columns); and a Markdown pipe table with a DOI column. Each record is
+    {"doi", "pmid", "title"}.
+    """
+    text = Path(path).read_text(encoding="utf-8")
+    lines = text.splitlines()
+    first = next((ln for ln in lines
+                  if ln.strip() and not ln.strip().startswith("#")), "")
+    low = first.lower()
+
+    # Markdown pipe table with a DOI column
+    if first.strip().startswith("|") and "doi" in low:
+        return _records_from_markdown(lines)
+
+    # Delimited (TSV or CSV) with a DOI header
+    if "doi" in low and ("\t" in first or "," in first):
+        delimiter = "\t" if "\t" in first else ","
+        body = "\n".join(ln for ln in lines if not ln.strip().startswith("#"))
+        reader = csv.DictReader(io.StringIO(body), delimiter=delimiter)
+        return _records_from_dictrows(reader)
+
+    # Plain text: one DOI per line
     records = []
-    with open(path, encoding="utf-8") as f:
-        first_line = f.readline().strip()
-        f.seek(0)
-
-        # TSV with header containing DOI column
-        if "\t" in first_line and "doi" in first_line.lower():
-            import csv
-            reader = csv.DictReader(f, delimiter="\t")
-            for row in reader:
-                doi = ""
-                pmid = ""
-                for k, v in row.items():
-                    if k.lower().strip() == "doi":
-                        doi = (v or "").strip()
-                    elif k.lower().strip() == "pmid":
-                        pmid = (v or "").strip()
-                if doi:
-                    records.append({"doi": doi, "pmid": pmid})
-        else:
-            # Plain text: one DOI per line
-            for line in f:
-                line = line.strip()
-                if line and not line.startswith("#"):
-                    records.append({"doi": line, "pmid": ""})
+    for line in lines:
+        line = line.strip()
+        if line and not line.startswith("#"):
+            records.append({"doi": line, "pmid": "", "title": ""})
     return records
 
 
@@ -372,11 +558,15 @@ def main():
     parser = argparse.ArgumentParser(
         description="Batch download open-access PDFs by DOI.")
     parser.add_argument("input", type=Path,
-                        help="File with DOIs (one per line, or TSV with DOI column)")
+                        help="Worklist: DOIs (one per line) or TSV/CSV/Markdown "
+                             "with a DOI column (optional PMID, Title)")
     parser.add_argument("-o", "--output", type=Path, default=Path("pdfs"),
                         help="Output directory (default: pdfs/)")
     parser.add_argument("-e", "--email", required=True,
                         help="Contact email (required by Unpaywall TOS)")
+    parser.add_argument("--report", type=Path, default=None,
+                        help="Path for the JSON retrieval report "
+                             "(default: <output>/retrieval_report.json)")
     parser.add_argument("-v", "--verbose", action="store_true",
                         help="Show debug messages")
     args = parser.parse_args()
@@ -387,33 +577,62 @@ def main():
     )
 
     args.output.mkdir(parents=True, exist_ok=True)
+    report_path = args.report or (args.output / "retrieval_report.json")
     records = read_doi_file(args.input)
     print(f"Loaded {len(records)} DOIs from {args.input}")
 
-    stats = {"oa": 0, "pmc": 0, "fail": 0, "skip": 0}
+    stats = {"arxiv": 0, "oa": 0, "pmc": 0, "fail": 0, "skip": 0}
+    results: dict[str, tuple[str, str]] = {}
 
     for i, rec in enumerate(records, 1):
         doi = rec["doi"]
         pmid = rec.get("pmid", "")
         print(f"  [{i}/{len(records)}] {doi}", end=" … ", flush=True)
 
-        status = process_doi(doi, args.output, args.email, pmid)
+        status, source = process_doi(doi, args.output, args.email, pmid)
+        results[doi] = (status, source)
         stats[status] += 1
 
-        labels = {"oa": "OK (OA)", "pmc": "OK (PMC)",
+        labels = {"arxiv": "OK (arXiv)", "oa": "OK (OA)", "pmc": "OK (PMC)",
                   "fail": "FAIL", "skip": "SKIP"}
         print(labels[status])
         time.sleep(0.5)
 
+    # Best-effort title cross-check on successful downloads (needs pdftotext).
+    extracted: dict[str, str] = {}
+    have_titles = any(r.get("title") for r in records)
+    if have_titles and shutil.which("pdftotext"):
+        for rec in records:
+            doi = rec["doi"]
+            if not rec.get("title"):
+                continue
+            status, _ = results.get(doi, ("fail", ""))
+            if status not in RETRIEVED_STATUSES:
+                continue
+            path = args.output / f"{safe_doi_name(doi)}.pdf"
+            if path.exists():
+                text = extract_pdf_text(path)
+                if text:
+                    extracted[doi] = text
+
+    report = build_report(records, results, args.output, extracted)
+    report_path.parent.mkdir(parents=True, exist_ok=True)
+    report_path.write_text(json.dumps(report, indent=2) + "\n", encoding="utf-8")
+
     print(f"\n--- Summary ---")
+    print(f"  arXiv:   {stats['arxiv']}")
     print(f"  OA:      {stats['oa']}")
     print(f"  PMC:     {stats['pmc']}")
     print(f"  Failed:  {stats['fail']}")
     print(f"  Skipped: {stats['skip']}")
-    total = stats["oa"] + stats["pmc"] + stats["fail"]
+    total = stats["arxiv"] + stats["oa"] + stats["pmc"] + stats["fail"]
     if total > 0:
-        pct = (stats["oa"] + stats["pmc"]) / total * 100
+        pct = (stats["arxiv"] + stats["oa"] + stats["pmc"]) / total * 100
         print(f"  Success: {pct:.0f}%")
+    mismatches = report["counts"]["title_mismatch"]
+    if mismatches:
+        print(f"  Title mismatches flagged: {mismatches} (see report)")
+    print(f"  Report:  {report_path}")
 
     # Write failed DOIs for manual retrieval
     if stats["fail"] > 0:
@@ -422,10 +641,10 @@ def main():
             f.write("# DOIs needing manual retrieval\n")
             f.write("# Options: institutional access, ILL\n\n")
             for rec in records:
-                safe = re.sub(r"[^\w\-.]", "_", rec["doi"])
-                pdf = args.output / f"{safe}.pdf"
+                doi = rec["doi"]
+                pdf = args.output / f"{safe_doi_name(doi)}.pdf"
                 if not existing_pdf_ok(pdf):
-                    f.write(f"{rec['doi']}\n")
+                    f.write(f"{doi}\n")
         print(f"  Manual list: {fail_path}")
 
 

package/skills/fulltext-retrieval/fetch_oa_report_challenge/expected/projection.json

@@ -0,0 +1,14 @@
+{
+  "counts": {
+    "total": 4,
+    "retrieved": 3,
+    "not_retrieved": 1,
+    "title_mismatch": 1
+  },
+  "items": [
+    {"doi": "10.1111/valid.match", "status": "oa", "source": "unpaywall", "title_match": "match"},
+    {"doi": "10.2222/label.mismatch", "status": "oa", "source": "openalex", "title_match": "mismatch"},
+    {"doi": "10.3333/no.text", "status": "pmc", "source": "pmc", "title_match": "unavailable"},
+    {"doi": "10.9999/not.retrieved", "status": "fail", "source": "", "title_match": "unavailable"}
+  ]
+}

package/skills/fulltext-retrieval/fetch_oa_report_challenge/extracted_text.json

@@ -0,0 +1,4 @@
+{
+  "10.1111/valid.match": "Deep Learning for Pulmonary Nodule Detection on Chest CT\nAbstract: we present a convolutional neural network trained to detect pulmonary nodules.",
+  "10.2222/label.mismatch": "Genome-wide association study of type 2 diabetes in an Asian cohort\nIntroduction: we genotyped participants to identify susceptibility loci."
+}

package/skills/fulltext-retrieval/fetch_oa_report_challenge/results.json

@@ -0,0 +1,6 @@
+{
+  "10.1111/valid.match": ["oa", "unpaywall"],
+  "10.2222/label.mismatch": ["oa", "openalex"],
+  "10.3333/no.text": ["pmc", "pmc"],
+  "10.9999/not.retrieved": ["fail", ""]
+}

package/skills/fulltext-retrieval/fetch_oa_report_challenge/run_challenge.py

@@ -0,0 +1,86 @@
+#!/usr/bin/env python3
+"""Offline, network-free challenge for fetch_oa.build_report + title tri-state.
+
+Loads committed fixtures (worklist.tsv, results.json, extracted_text.json),
+fabricates stub PDFs in a temp dir for the "retrieved" DOIs, runs
+fetch_oa.build_report, and asserts the resulting projection equals
+expected/projection.json. Exercises read_doi_file (TSV+title), build_report,
+and classify_title_match (match / mismatch / unavailable) without touching the
+network or pdftotext. Stdlib-only.
+"""
+import importlib.util
+import json
+import sys
+import tempfile
+from pathlib import Path
+
+HERE = Path(__file__).resolve().parent
+ENGINE = HERE.parent / "fetch_oa.py"
+
+
+def load_engine():
+    spec = importlib.util.spec_from_file_location("fetch_oa", ENGINE)
+    mod = importlib.util.module_from_spec(spec)
+    spec.loader.exec_module(mod)
+    return mod
+
+
+def projection(report: dict) -> dict:
+    items = sorted(
+        ({"doi": i["doi"], "status": i["status"], "source": i["source"],
+          "title_match": i["title_match"]} for i in report["items"]),
+        key=lambda x: x["doi"],
+    )
+    return {"counts": report["counts"], "items": items}
+
+
+def main() -> int:
+    assert ENGINE.exists(), f"ENV-ERR: {ENGINE} missing"
+    m = load_engine()
+
+    records = m.read_doi_file(HERE / "worklist.tsv")
+    results = {k: tuple(v) for k, v in
+               json.loads((HERE / "results.json").read_text()).items()}
+    extracted = json.loads((HERE / "extracted_text.json").read_text())
+    expected = json.loads((HERE / "expected" / "projection.json").read_text())
+
+    fails = []
+
+    def check(label, cond):
+        print(f"  {'PASS' if cond else 'FAIL'}  {label}")
+        if not cond:
+            fails.append(label)
+
+    with tempfile.TemporaryDirectory() as tmp:
+        outdir = Path(tmp)
+        for rec in records:
+            status, _ = results.get(rec["doi"], ("fail", ""))
+            if status in m.RETRIEVED_STATUSES:
+                stub = outdir / f"{m.safe_doi_name(rec['doi'])}.pdf"
+                stub.write_bytes(b"%PDF-1.4\n" + b"0" * (11 * 1024))
+        report = m.build_report(records, results, outdir, extracted)
+        proj = projection(report)
+
+    by_doi = {i["doi"]: i for i in proj["items"]}
+    check("valid -> title_match 'match'",
+          by_doi["10.1111/valid.match"]["title_match"] == "match")
+    check("mislabel -> title_match 'mismatch'",
+          by_doi["10.2222/label.mismatch"]["title_match"] == "mismatch")
+    check("no-text -> title_match 'unavailable'",
+          by_doi["10.3333/no.text"]["title_match"] == "unavailable")
+    check("missing -> status 'fail'",
+          by_doi["10.9999/not.retrieved"]["status"] == "fail")
+    check("counts.retrieved == 3", proj["counts"]["retrieved"] == 3)
+    check("counts.not_retrieved == 1", proj["counts"]["not_retrieved"] == 1)
+    check("counts.title_mismatch == 1", proj["counts"]["title_mismatch"] == 1)
+    check("projection matches expected/projection.json", proj == expected)
+
+    if fails:
+        print(f"FAILURES: {len(fails)}")
+        return 1
+    print("ALL PASS")
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/fulltext-retrieval/fetch_oa_report_challenge/verify.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+# Network-free challenge: fetch_oa report builder + title-match tri-state.
+# Mirrors CI usage: `bash skills/fulltext-retrieval/fetch_oa_report_challenge/verify.sh`
+set -euo pipefail
+DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+python3 "$DIR/run_challenge.py"

package/skills/fulltext-retrieval/fetch_oa_report_challenge/worklist.tsv

@@ -0,0 +1,5 @@
+DOI	PMID	Title
+10.1111/valid.match	1001	Deep learning for pulmonary nodule detection on chest CT
+10.2222/label.mismatch	1002	Automated segmentation of cardiac MRI using transformers
+10.3333/no.text	1003	Federated learning for multi-center radiology models
+10.9999/not.retrieved	1004	A paywalled study with no open-access copy

package/skills/fulltext-retrieval/references/find_available_pdf.js

@@ -0,0 +1,82 @@
+/*
+ * Find Available PDF — batch trigger for Zotero (user-run snippet)
+ * ----------------------------------------------------------------
+ * Attaches full-text PDFs to library items using Zotero's OWN
+ * "Find Available PDF" resolver. This reuses whatever OpenURL resolver,
+ * institutional proxy, or library configuration YOU have already set in
+ * Zotero — so it typically retrieves far more than open-access-only resolvers,
+ * yet no credentials, proxy hosts, or institutional identifiers ever leave
+ * your Zotero client. Nothing here is hard-coded to any institution.
+ *
+ * HOW TO RUN
+ *   1. In Zotero, select the items (or open the collection) you want PDFs for.
+ *   2. Tools → Developer → Run JavaScript.
+ *   3. Paste this whole snippet and click Run.
+ *   4. The result panel prints a JSON summary: considered / attached /
+ *      alreadyHadPDF / missing (DOIs still without a PDF).
+ *
+ * NO-CODE FALLBACK
+ *   Select items → right-click → "Find Available PDF" does the same thing
+ *   interactively. Use it if you prefer not to run a script.
+ *
+ * VERSION NOTE
+ *   Zotero 7 exposes a batch Zotero.Attachments.addAvailablePDFs(items);
+ *   Zotero 6 only has the per-item Zotero.Attachments.addAvailablePDF(item).
+ *   This snippet prefers the batch call and falls back to per-item.
+ *
+ * NOTE: results are user-initiated and depend on your live Zotero session;
+ * they are NOT reproducible CI evidence. Record retrieved/not-retrieved
+ * outcomes from the printed summary into your retrieval report manually.
+ */
+
+var pane = Zotero.getActiveZoteroPane();
+var items = pane.getSelectedItems().filter(function (it) { return it.isRegularItem(); });
+
+// Fall back to the whole selected collection if nothing is selected.
+if (!items.length) {
+  var collection = pane.getSelectedCollection();
+  if (collection) {
+    items = collection.getChildItems().filter(function (it) { return it.isRegularItem(); });
+  }
+}
+
+function hasPDF(item) {
+  return item.getAttachments().some(function (id) {
+    var att = Zotero.Items.get(id);
+    if (!att) return false;
+    if (typeof att.isPDFAttachment === "function") return att.isPDFAttachment();
+    return att.attachmentContentType === "application/pdf";
+  });
+}
+
+var todo = items.filter(function (it) { return !hasPDF(it); });
+var alreadyHadPDF = items.length - todo.length;
+
+if (typeof Zotero.Attachments.addAvailablePDFs === "function") {
+  await Zotero.Attachments.addAvailablePDFs(todo);   // Zotero 7 batch
+} else {
+  for (let it of todo) {                              // Zotero 6 per-item
+    try {
+      await Zotero.Attachments.addAvailablePDF(it);
+    } catch (e) {
+      Zotero.debug("addAvailablePDF failed for item " + it.id + ": " + e);
+    }
+  }
+}
+
+var attached = 0;
+var missing = [];
+for (let it of todo) {
+  if (hasPDF(it)) {
+    attached++;
+  } else {
+    missing.push(it.getField("DOI") || it.getField("title") || ("itemID:" + it.id));
+  }
+}
+
+return JSON.stringify({
+  considered: items.length,
+  attached: attached,
+  alreadyHadPDF: alreadyHadPDF,
+  missing: missing
+}, null, 2);

package/skills/fulltext-retrieval/skill.yml

@@ -8,12 +8,14 @@ when_to_use: "Batch-download open-access full-text PDFs from a DOI list; optiona
 when_NOT_to_use: "Finding or verifying citations (use search-lit / verify-refs). Retrieving paywalled or non-open-access content."
 
 inputs:
-  - path: "DOI list (.txt, one per line, or .tsv with a DOI column)"
+  - path: "DOI worklist (.txt one-per-line, or .tsv/.csv/.md with a DOI column; optional PMID, Title)"
     schema: csv
     required: true
 outputs:
   - path: "downloaded open-access PDFs (pdfs/)"
+  - path: "pdfs/retrieval_report.json (per-DOI status/source/title_match tri-state)"
   - path: "optional PDF-to-Markdown conversions"
+  - path: "references/find_available_pdf.js (user-run Zotero 'Find Available PDF' batch snippet)"
 
 deterministic_scripts:
   - fetch_oa.py
@@ -35,8 +37,11 @@ safety_boundaries:
   - "Validates each download (>=10 KB and a %PDF- header) before accepting it."
 known_limitations:
   - "Only open-access content is retrievable; non-OA DOIs fail by design rather than fetching from unauthorized sources."
+  - "Higher-yield in-library retrieval (find_available_pdf.js) is user-initiated inside Zotero and uses the user's own proxy/OpenURL config; it is not reproducible CI evidence."
+  - "Title cross-check is best-effort: it needs a Title column plus pdftotext (poppler); otherwise title_match is 'unavailable'. A mismatch is flagged, never auto-rejected."
   - "PDF-to-Markdown conversion requires the optional pymupdf4llm dependency (AGPL-3.0 or commercial license)."
 validation_commands:
-  - "python fetch_oa.py dois.txt -o pdfs/ -e <email> --verbose   # per-DOI source trace"
+  - "bash fetch_oa_report_challenge/verify.sh   # offline report-builder + title tri-state (CI-wired)"
+  - "python fetch_oa.py dois.txt -o pdfs/ -e <email> --report pdfs/retrieval_report.json --verbose   # per-DOI source trace"
   - "verify each output begins with %PDF- and is at least 10 KB"
 evidence_surface: bundled_script

package/skills/lit-sync/SKILL.md

@@ -65,6 +65,9 @@ Direct hand edits to `refs.bib` are drift — revert on sight.
     ▼ Phase 2.5: refs.bib snapshot refresh
     Trigger Better BibTeX auto-export → verify manuscript/_src/refs.bib mtime updated
     │
+    ▼ Phase 2.7: Fulltext Retrieval (opt-in)
+    Disk OA PDFs via /fulltext-retrieval + in-library via find_available_pdf.js → reconcile report
+    │
     ▼ Phase 3: Obsidian Literature Notes
     Create Literature/{citekey}.md (empty note OK — fill later with highlights)
     │
@@ -116,8 +119,16 @@ collection key for future use.
 For each entry:
 
 1. Use `zotero_search_items` to search by DOI or title — if already present, skip.
+   This search-first step is what prevents duplicates; `zotero_add_by_doi` does **not**
+   dedupe by itself (it fetches CrossRef and creates the item), so never skip the search.
 2. Otherwise call `zotero_add_by_doi` (when a DOI is available) or
    `zotero_add_by_url` (falling back to the PubMed URL when no DOI is available).
+   - `zotero_add_by_doi` accepts an `attach_mode` argument that governs the **OA child-PDF
+     attach attempt at add time** (the installed server treats `linked_url` as "bookmark the
+     PDF URL"; other values download/import). Set it when you want a PDF attached during the
+     add. Exact accepted values are server-version-specific — verify against the connected
+     server. Do **not** use `zotero_add_from_file` to attach a PDF to an item added here: it
+     has no parent-item argument and would create a duplicate parent item.
 3. Use `zotero_manage_collections` to place the item in the project collection.
 
 ### Step 2.3: Result report
@@ -214,6 +225,62 @@ If refresh failed, set `refs_bib_refreshed: false` and include `reason`. `/verif
 
 ---
 
+## Phase 2.7: Fulltext Retrieval (opt-in, owner-only)
+
+**Run only when the user asks for full text** (e.g. "download the PDFs", "fetch full
+text", or a worklist supplied with that intent). Default `/lit-sync` stays metadata-only
+and network-light — do not auto-run this phase. Runs after items are in Zotero (Phase 2)
+and the snapshot is verified (Phase 2.5), before Obsidian notes (Phase 3).
+
+There are two complementary retrieval routes; offer both and reconcile them in one report:
+
+### Route A — disk OA PDFs (for downstream skills)
+
+Delegate to the `/fulltext-retrieval` engine (do **not** re-implement the OA cascade or
+import its code; invoke it by path). Resolve the engine as:
+
+```bash
+ENGINE="${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/fetch_oa.py"
+python3 "$ENGINE" <worklist> -o pdfs/ -e <contact-email> --report pdfs/retrieval_report.json
+```
+
+`<worklist>` is the DOI/PMID(/Title) list — the Phase-1 `.bib` DOIs, the worklist supplied
+in the standalone mode below, or the project collection's DOIs. Output: `pdfs/*.pdf` for
+`/meta-analysis`, `/obsidian-paper-vault`, and `pdf_to_md.py`, plus
+`pdfs/retrieval_report.json` (per-DOI `status`/`source`/`title_match`).
+
+### Route B — in-library PDFs (Zotero-native, higher yield, proxy-aware)
+
+Emit `${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/references/find_available_pdf.js`
+for the user to paste into Zotero (*Tools → Developer → Run JavaScript*) with the project
+collection selected. It triggers Zotero's own `addAvailablePDF`/`addAvailablePDFs`, which
+reuse the **user's** OpenURL resolver / institutional proxy — so it typically retrieves more
+than OA-only, while **no credentials or institutional identifiers enter this skill**. The
+no-code equivalent is right-click → "Find Available PDF". This route is user-initiated and
+session-dependent; record its `{attached, missing}` summary from the printed JSON.
+
+### Report
+
+Merge Route A's `pdfs/retrieval_report.json` (and the user-reported Route B summary) into
+`references/fulltext_retrieval.json` (owner of this file is `/lit-sync`):
+
+```json
+{
+  "schema_version": 1,
+  "retrieved_oa_disk": [{"doi": "...", "source": "unpaywall", "file": "...", "title_match": "match"}],
+  "retrieved_zotero_native": [{"doi": "...", "via": "addAvailablePDF"}],
+  "not_retrieved": [{"doi": "...", "journal": "..."}],
+  "institutional_fallback": ["<DOIs needing institutional access / ILL / author contact>"],
+  "title_mismatch_flagged": ["<DOIs whose downloaded PDF title did not match>"]
+}
+```
+
+Also append a short `fulltext` block (counts) to `references/zotero_collection.json`.
+`not_retrieved` DOIs are candidates for institutional access, interlibrary loan, or author
+contact — never bypass paywalls or access controls from this skill.
+
+---
+
 ## Phase 3: Obsidian Literature Notes
 
 ### Step 3.1: Check existing literature notes
@@ -414,16 +481,23 @@ curl -s "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/esummary.fcgi?db=pubmed&i
   | jq -r '.result | to_entries[] | select(.key != "uids") | "\(.value.uid)\t\(.value.elocationid)\t\(.value.title)"'
 ```
 
-For each resolved DOI call `zotero_add_by_doi` (auto-dedup by DOI). For items
-already in the library (returned as "skipped" or detected via `zotero_search_items`
-by DOI), use `zotero_manage_collections` to attach them to the project collection
-**without re-adding** — re-adding by URL/PubMed-URL would bypass DOI dedup and
-create duplicates. Record both `added` and `existing` items in
-`references/zotero_collection.json`.
+For each resolved DOI, search-first with `zotero_search_items`, then call
+`zotero_add_by_doi` — the search is what dedupes (add-by-doi alone does not). For items
+already in the library (detected via `zotero_search_items` by DOI), use
+`zotero_manage_collections` to attach them to the project collection **without re-adding** —
+re-adding by URL/PubMed-URL would bypass the search dedup and create duplicates. Record both
+`added` and `existing` items in `references/zotero_collection.json`.
 
 If a PMID has no DOI in PubMed (rare; older papers, non-indexed), fall back to
 `zotero_add_by_url` with the PubMed URL and mark the entry as `no_doi: true`.
 
+### Worklist ingestion (DOI/PMID/Title; no .bib)
+When the user supplies a worklist file (a `.tsv`/`.csv`/`.md` table with a `DOI` column,
+optional `PMID`/`Title`, or a plain DOI-per-line list — e.g. an SR include set), enter
+Phase 2 directly from it: resolve any PMID-only rows to DOIs (esummary above), then run the
+search-first dedupe + add loop. The same worklist file feeds Phase 2.7 Route A
+(`fetch_oa.py` reads `.tsv`/`.csv`/`.md`/plain natively), so no reformatting is needed.
+
 ---
 
 ## Safety Rules
@@ -439,6 +513,12 @@ If a PMID has no DOI in PubMed (rare; older papers, non-indexed), fall back to
    collection is created.
 6. **Never write `refs.bib` directly.** Only Better BibTeX auto-export may write that file. If auto-export is broken, fix the Zotero setup rather than writing the file from this skill.
 7. **Owner-only execution.** If the current user is a collaborator (no Zotero access per `SSOT.yaml` `reference_manager.required_for`), abort with instructions to flag `[@NEW:topic]` placeholders in the manuscript and notify the owner.
+8. **Fulltext boundary (Phase 2.7).** Retrieve full text only via OA APIs (the
+   `/fulltext-retrieval` engine) and the user-run Zotero "Find Available PDF" snippet
+   (which uses the user's own proxy config). Never automate authenticated browser sessions,
+   never bypass paywalls/access controls, and never hard-code institutional proxies,
+   credentials, or hosts into this skill. `not_retrieved` items are routed to institutional
+   access / ILL / author contact, not worked around.
 
 ## Anti-Hallucination
 

package/skills/lit-sync/skill.yml

@@ -8,6 +8,7 @@ when_to_use:
   - After /search-lit when verified candidates need to flow into Zotero + refs.bib
   - Cross-cutting concept-note extraction from accumulated literature into Obsidian
   - Refreshing manuscript/_src/refs.bib via Better BibTeX auto-export before a build
+  - Opt-in full-text retrieval (Phase 2.7) — disk OA PDFs via /fulltext-retrieval + in-library "Find Available PDF"
 when_NOT_to_use:
   - Pure literature search without sync (use /search-lit)
   - Hand-editing refs.bib (BBT is the sole writer; never touch the file directly)
@@ -17,10 +18,11 @@ inputs:
   - PMID list, DOI list, or Zotero collection name
 outputs:
   - references/zotero_collection.json
+  - references/fulltext_retrieval.json  # SOLE WRITER; opt-in Phase 2.7 retrieval report
   - manuscript/_src/refs.bib  # SOLE WRITER via Better BibTeX auto-export "Keep updated"
   - obsidian_literature_notes
 deterministic_scripts:
-  - none_required  # leverages Zotero MCP + Better BibTeX auto-export GUI
+  - none_required  # leverages Zotero MCP + Better BibTeX auto-export GUI; Phase 2.7 invokes /fulltext-retrieval fetch_oa.py
 side_effects:
   - may_update_zotero
   - may_write_obsidian_notes

package/skills/search-lit/SKILL.md

@@ -263,9 +263,9 @@ Total:       13 references
 
 If a Zotero MCP server is available, integrate search results with the user's library:
 
-1. **Add papers to Zotero**: Use `zotero_add_by_doi` for DOI-based import (auto-downloads OA PDFs).
-2. **Organize into collections**: Use `zotero_manage_collections` to file into the relevant project collection.
-3. **Check for duplicates**: Use `zotero_search_items` to avoid adding papers already in the library.
+1. **Check for duplicates first**: Use `zotero_search_items` (by DOI) to skip papers already in the library — this search-first step is what dedupes; `zotero_add_by_doi` does not dedupe on its own.
+2. **Add papers to Zotero**: Use `zotero_add_by_doi` for DOI-based import (its `attach_mode` argument governs the OA PDF attach attempt at add time).
+3. **Organize into collections**: Use `zotero_manage_collections` to file into the relevant project collection.
 4. **Leverage annotations**: Use `zotero_get_annotations` to reference the user's prior reading notes.
 5. **Write sync audit**: Record collection key, added/skipped/failed counts, and
    unsynced entries in `references/zotero_collection.json` so Zotero status is
@@ -277,77 +277,32 @@ If a Zotero MCP server is available, integrate search results with the user's li
 
 ### Phase 5: Full-Text Retrieval
 
-After identifying relevant papers, retrieve full-text PDFs for detailed review.
-This is especially important for meta-analyses where data extraction requires full text.
+Full-text PDF retrieval is **delegated to `/fulltext-retrieval`** — the single authored
+home of the open-access cascade (arXiv → Unpaywall → PMC → OpenAlex → Crossref → landing
+page, each validated with a `%PDF-` header + ≥10 KB size). Do **not** re-implement OA
+fetching here.
 
-#### Phase 5a: Open Access Auto-Retrieval
+Pass the verified candidate DOIs from `references/library.bib`:
 
-Try sources in order of reliability:
-
-1. **Unpaywall API** (highest quality OA links):
-   ```python
-   import os, requests
-   email = os.environ.get("UNPAYWALL_EMAIL", "user@example.com")
-   url = f"https://api.unpaywall.org/v2/{doi}?email={email}"
-   r = requests.get(url).json()
-   if r.get("best_oa_location", {}).get("url_for_pdf"):
-       pdf_url = r["best_oa_location"]["url_for_pdf"]
-   ```
-
-2. **PubMed Central (PMC)**:
-   - Convert PMID to PMCID via NCBI ID Converter
-   - Download from PMC OA service: `https://www.ncbi.nlm.nih.gov/pmc/articles/PMC{id}/pdf/`
-
-3. **OpenAlex API** (additional OA discovery):
-   ```python
-   url = f"https://api.openalex.org/works/https://doi.org/{doi}"
-   # Requires polite pool: add email in User-Agent header or mailto= param
-   r = requests.get(url, headers={"User-Agent": f"MyApp/1.0 (mailto:{email})"}).json()
-   oa_url = r.get("open_access", {}).get("oa_url")
-   ```
-
-4. **CrossRef landing page**: Follow `https://api.crossref.org/works/{doi}` → publisher link
-   → scrape `<meta name="citation_pdf_url">` tag
-
-#### Phase 5b: Alternative Sources
-
-Some researchers use alternative access methods for paywalled content.
-**Users are responsible for ensuring compliance with their institutional access policies.**
-
-If an environment variable (e.g., `SCIHUB_BASE`) is set, the skill may use it as an
-alternative PDF source. No specific URLs are provided here — users configure this themselves.
-
-Other options:
-- **Institutional proxy/VPN**: Access publisher sites through institutional EZproxy or VPN
-- **Interlibrary loan (ILL)**: Request through library services for papers not otherwise available
-- **Author contact**: Email corresponding authors for preprints
-
-#### PDF Validation
+```bash
+ENGINE="${MEDSCI_SKILLS_ROOT:-$HOME/workspace/medsci-skills}/skills/fulltext-retrieval/fetch_oa.py"
+# extract DOIs from references/library.bib → dois.txt (one per line)
+python3 "$ENGINE" dois.txt -o pdfs/ -e <contact-email> --report pdfs/retrieval_report.json
+```
 
-Always validate downloaded files before use:
+For Zotero-resident PDFs and higher-yield, proxy-aware retrieval, use `/lit-sync` Phase 2.7,
+which also invokes `/fulltext-retrieval` and triggers Zotero's native "Find Available PDF".
 
-```python
-def is_valid_pdf(filepath):
-    """Check that a downloaded file is actually a PDF, not an HTML redirect."""
-    import os
-    if os.path.getsize(filepath) < 10240:  # < 10KB is likely a stub/redirect
-        return False
-    with open(filepath, 'rb') as f:
-        header = f.read(5)
-    return header == b'%PDF-'
-```
+#### Alternative sources (legitimate only)
 
-Additional checks:
-- Verify HTTP `Content-Type: application/pdf` header before saving
-- Files under 10KB are almost always HTML login/redirect pages, not real PDFs
-- Some publishers return CAPTCHA pages — these fail the `%PDF-` check
+For DOIs that open access cannot reach (listed in `pdfs/manual_needed.txt`):
 
-#### Rate Limiting
+- **Institutional access / proxy / VPN** — through your library's own subscriptions.
+- **Interlibrary loan (ILL)** — request via library services.
+- **Author contact** — email the corresponding author for a copy or preprint.
 
-- Unpaywall: Polite pool (no hard limit with email parameter)
-- OpenAlex: Include email in User-Agent for polite pool access
-- NCBI/PMC: 3 requests/sec without API key, 10/sec with `NCBI_API_KEY`
-- General: 2-second minimum interval between requests to any single host
+Never bypass paywalls or publisher access controls, and do not configure unauthorized
+PDF mirrors. Rate limits and PDF validation are handled inside `/fulltext-retrieval`.
 
 ### Phase 6: Gap Analysis