fh-library 0.2.5 → 0.2.7

package/CHANGELOG.md

@@ -1,5 +1,19 @@
 # FH Library — Change Log
 
+### v0.2.7 — 2026-11-08
+
+* **Verdicts are cached** (#26): `library_verify` keeps each judgement as `verify/<2>/<4>/<key>.json` — the file system as the database, laid out like `url/` — and answers the same claim on the same stored text from disk for `FH_LIBRARY_REFRESH_AFTER_DAYS`; a new full text, model or threshold is judged afresh, errors are never kept. The result is unchanged; the log line says `cached`. One atomic file per judgement: several servers on one library need no lock.
+* **Each server publishes its cache counters:** hits and misses of `library_verify` and `library_fetch` since it started, in memory, written to `<library>/.servers/<host>-<pid>.json` on every change and every 5 min as a heartbeat, removed on exit, swept by the next starting server when older than 20 min; `library_info` reports them as `cache`. The server also logs a `startup` line.
+* TUI v79: a `verify` row — stored verdicts ("queries cached"), then hits and misses summed over the servers alive on the library, read from `.servers/`, not from the log; `/status` carries `verdicts`. Resolves `.env` and `library/` from the caller's working directory, as the server does (#31) — `npm run --prefix <repo> tui` from a project root shows that project's library.
+
+### v0.2.6 — 2026-11-08
+
+* **`library_fetch(url, max_length?, start_index?, raw?)`:** the standard MCP `fetch` tool, served from the library. Same answer as the reference server — `Contents of <url>:`, the page as Markdown with its title as the first heading, the truncation note, the raw-content prefix, the `Failed to fetch …` errors — so a model keeps its fetch habits. Underneath, the page is fetched up the download ladder (plain fetch, Playwright and Chrome rungs, Bright Data) when a plain fetch is refused, and stored under `url/`: the same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS`. A PDF comes back as Markdown. `library_info` counts web pages as type `url`; the log line says `cached` or `fetched`.
+* **Script shells climb the ladder:** a page answered 200 whose converted text is under 100 words with a loading/JavaScript phrase (or under 10 words at all) is treated like a wall page — the browser rungs, then Bright Data; a browser rung with so little text waits for the network to go quiet and the text to grow (at most 8 s) before reading the page. When no rung renders more, the shell is served, as fetch would (lifeextension.com's search is such a page: nothing renders it).
+* **Bright Data's 407 is reported:** a token's IP allowlist refusing this host used to surface as a bare "fetch failed" (Node's fetch treats a 407 as a network error); the unlocker now requests over node:https and the log alerts `!! brightdata: unlocker HTTP 407: You are not allowed to access our API via this IP (…)`.
+* TUI v74: a `url` row (web pages, failed fetches); fetch log lines coloured — `cached` green, rung outcomes green/red; `/status` counts `fetches` and `cached`; `(u)pdate` also rescans the library.
+* README: `library_version` returns the package "version", not "library_version" (one more word the prefix rename caught).
+
 ### v0.2.5 — 2026-11-08
 
 * **Tool descriptions** (issue #30): `library_info` names its key `verify`, not `library_verify`; `library_status` answers for one ID of any kind, not "one paper ID"; `library_verify`'s `claim` asks for a complete statement built from the review's words, not a verbatim fragment.

package/README.md

@@ -1,4 +1,4 @@
-![Version 0.1.3](https://img.shields.io/badge/Version-0.2.5-green.svg)
+![Version 0.1.3](https://img.shields.io/badge/Version-0.2.7-green.svg)
 ![License MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
 [![Forever Healthy](https://img.shields.io/badge/(c)_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
 [![npm](https://img.shields.io/badge/npm-fh--library-cb3837.svg)](https://www.npmjs.com/package/fh-library)
@@ -20,10 +20,11 @@ A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a
 
 * `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `coi` and `funding` (PubMed's record, else the paper's own text), `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — files, failed tiers in `attempts`). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `FH_LIBRARY_REFRESH_AFTER_DAYS` gets the record re-read for new notices. One ID per call.
 * `library_status(id)` → read-only check of presence and level; never fetches
-* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
+* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), this server's cache hits and misses since it started (`cache`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
 * `library_verify(claim, id)` → is one finding of a review stated by one paper it cites? `claim` is the finding alone, verbatim, without its citations; `id` one paper (`pmid:`, or any `doi:`), drug label or trial record the review cites for it — a finding cited to three papers is three calls with the same claim; a claim that names a source or carries citations is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters the model's context. `verdict` **pass** (`score` the probability), **fail** (the full text does not state it; or only the title is in the library — `reason` says so), **unverifiable** (only the abstract is in the library and does not state it — `reason`) or **error**; `basis` full_text, abstract or title. See "Verifying claims" below.
 * `library_read(id, section?, query?, max_chars?, max_hits?)` → one small piece of a stored paper's Markdown, never the whole paper: without `section`/`query` the **outline** (headings with word counts, and whether the full text or only the abstract is stored; a heading the paper uses several times is listed as `Participants [3/12]`); `section` → that section's **own text** (heading matched as case-insensitive substring, capped at `max_chars`, default 6000, with `returned_words` against `words` when cut) and, for a parent heading, its `subsections` with word counts to pick from — a duplicated heading named bare lists the candidates instead; `query` — two or three distinctive terms the paper itself would use (a measure and a unit, an outcome and a group, a compound name), not the claim — → **passages** holding every term, each a window of at most 1000 chars around the matches (`clipped: true` when cut; a table row with its header row), with its section, best first (at most `max_hits`, default 5); matching forgives case, accents, hyphens, Markdown markup, plural and -ed/-ing forms; when no passage holds every term the best **partial** matches come back (`partial: true`, the `matched` terms per passage, the rare term first), and `missing_terms` names the terms the text never uses in any form — those are the ones to rephrase. `section` and `query` together search inside that section. `section: "Abstract"` on a full text without one answers from the stored abstract. Every full text converted from JATS XML (PMC, Europe PMC, Elsevier) has a **Declarations** section — funding, conflicts of interest, author notes, gathered from wherever the XML keeps them — and on a text without one `section: "Declarations"` assembles the sections headed funding, conflicts, acknowledgements, ethics, contributions, so it answers who paid and who declares what on every source alike; a paper with neither states nothing. Outline first, then pick. A paper not in the library is added first, as `library_add` would (its lines in the log), then library_read; the answer carries `added: true`. Reads may run in parallel.
-* `library_version()` → the running server's package name, library_version, host and library directory
+* `library_fetch(url, max_length?, start_index?, raw?)` → a web page as Markdown, exactly as the standard MCP `fetch` tool returns it (see "Web pages"): the one tool whose document enters the model's context
+* `library_version()` → the running server's package name, version, host and library directory
 
 IDs follow the `type:value` form: `pmid:`, `doi:`, `dailymed:`, `nct:`; bare digits count as `pmid:`, a bare NCT number as `nct:`.
 
@@ -95,6 +96,8 @@ Every full text is checked before it counts: title present, methods and results
 
 **The call.** `claim` is one finding, verbatim from the review, with markdown links reduced to their text and its citations removed: `IPSS fell 2.3 points more than placebo in men with ED`. Section labels such as `**Magnitude:**` and list bullets may stay. `id` is one paper the review cites for that finding, as `pmid:<PMID>`, already in the library (`library_add` first). One claim, one paper per call: a finding cited to three papers is three calls with the same claim text, and each answer stands on its own — a source that does not carry the finding is named by its own row, so it can be dropped without touching the others. The server never aggregates over papers. A sentence that joins two findings, each with its own citation, is two claims: the text before each citation group, without the group. A sentence without a citation is not a claim.
 
+**Verdicts are kept.** The answer to one claim against one stored text is written to `verify/<2>/<4>/<key>.json` — one file per judgement, laid out like `url/` — and served from there for `FH_LIBRARY_REFRESH_AFTER_DAYS` (30): an audit that runs twice over the same review pays the model once. The key hashes the claim (whitespace aside), the source ID, the model, the threshold and the stored text, so a paper that gains its full text, a changed model or a changed threshold is judged afresh; errors are never kept. The result JSON is the same either way; the log line carries `cached`. Several servers may share the library: a file appears whole or not at all, no lock.
+
 **How to build the claim.** The decision model answers exactly the statement it is given, so the statement must be one the paper could make:
 
 * **A complete statement:** subject, intervention, outcome, number. Splitting a sentence at its citations leaves fragments ("and dementia", "or silodosin", "across long-acting PDE5 inhibitors, 0.40 percentage points lower"); restore the subject from the sentence ("HbA1c was 0.40 percentage points lower with long-acting PDE5 inhibitors than with control"). Resolve "it", "both", "the drug".
@@ -139,6 +142,8 @@ PubMed's book records — National Academies reports, NTP monographs, StatPearls
 
 **Clinical trial records.** `library_add("nct:05173896")` stores the current record of one trial from [ClinicalTrials.gov](https://clinicaltrials.gov/): the ID is the eight digits of the NCT number in the review's link (`…/study/NCT05173896`); `nct:NCT05173896` and a bare `NCT05173896` are accepted and read the same. The API's JSON is converted to Markdown, one section per module: identification with secondary IDs such as EudraCT, status with every date and its actual/estimated type, sponsor and collaborators, description, conditions, design (phase, allocation, masking, enrollment with its type), arms and interventions with doses, primary and secondary outcomes with time frames, eligibility, locations as a country list, references; and, when results are posted, the participant flow, baseline characteristics, every outcome measure with its group values and analyses, and the adverse events, all as tables. Values are written in plain words, "Phase 2", "Active, not recruiting", "100 (estimated)", so a figure the review states can be found as stated. The status carries `nctid`, `publications` (the results publications the record lists, as `pmid:…`, so whether a trial has been published is one status away), the brief title, the lead sponsor as `first_author`, the first-posted year as `year`, the last-update-posted date as `version`, and `publication_types` `["Trial Registration"]`. `library_read` and `library_verify` work on a record as on a paper. An entry older than `FH_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when the record was updated since; a status the review states that has moved on then fails, which is right. Entries live under `nct/<digits 1–2>/<digits 1–4>/<digits>/` with the JSON and the Markdown as `nct-<digits>-fulltext.json` and `-fulltext.md`; the log line reads `library_add nct:05173896 ctgov fetched level=full_text updated 2024-12-09 active, not recruiting ETLAS2`.
 
+**Web pages.** `library_fetch(url, max_length?, start_index?, raw?)` is the standard MCP [fetch](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) tool, served from the library, for every link a review carries that is not a paper, label or trial record: a guideline, a product page, a news item. Its description tells the model to prefer it over a plain fetch or a browser tool, and says why: a page anyone fetched in the last 30 days comes from disk in milliseconds, and a page a plain fetch cannot get climbs the ladder in the same call. The answer is what `fetch` returns, so a model keeps its habits: `Contents of <url>:` and the page as Markdown with the page's title as the first heading, cut at `max_length` (default 5000) with the note `Content truncated. Call the fetch tool with a start_index of N to get more content`; `raw` gives the HTML; a `text/markdown` body comes back as is, any other non-HTML body with the `Content type … cannot be simplified to markdown` prefix; a failure is `Failed to fetch <url> - status code 404` or `Failed to fetch <url>: <reason>`. Two things happen underneath. The page is fetched up the same ladder as a paper — the plain fetch, then the browser rungs on a 403, 429 or bot wall, then Bright Data — so a page a plain fetch cannot reach is still read. And it is stored: `url/<2>/<4>/<16 hex of the URL's SHA-1>/url-<hash>-{meta.md, page.md, page.html|pdf|txt}`, the URL normalised first (host lowercased, fragment and tracking parameters dropped). The same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS` (1) — an audit that runs twice over one review reads its links once. A PDF comes back as Markdown (`fetch` would return its bytes as text). The log says which — `fetch example.com/… cached html … https://example.com/drugs/x` or `fetched 02.3s html retrieved_by=browser hetzner … <url>` (the ID column holds the URL without scheme, cut to a PubMed ID's width; the full URL ends the line); `library_info` counts the pages under `url`, a failed fetch as level `title`.
+
 **Funding and conflicts of interest.** Every status carries `coi`, the paper's conflict-of-interest statement, and `funding`, who paid. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `library_read(id, section: "Declarations")` — the JATS section, or on a PDF- or HTML-sourced text the funding, conflict and acknowledgement sections assembled. In `-meta.md` they are the lines `coi:` (a JSON string) and `funding:` (a JSON array, one item per agency or statement).
 
 **Symbols a PDF font hides.** Some publishers' symbol fonts carry no Unicode for their glyphs, so text extraction gives a control character where the PDF shows a minus, ±, ≤ or β. The library restores the symbol from the glyph's name in the font — the names of the Adobe Pi fonts (`H11350` is ≥ everywhere) and of the Advent symbol font most Elsevier and Wiley PDFs use — each name checked against the glyph's outline in the library's PDFs; a name it does not know is left as it is and counted: the status's `glyphs_unresolved` is the number left in the full text (162 of 1,119 PDF-derived entries have some, down from 277), and `library_verify` returns the same count for the text it judged. A `fail` on a paper with `glyphs_unresolved` above 0 says "check the symbol", not "the paper does not state it". Words a PDF breaks at a line end ("differ- ence") are joined back when the paper uses the word elsewhere or no compound is known at that break; a hyphen the paper uses elsewhere ("time-dependent") and a suspended one ("dose- and time-dependent") are kept.

package/dist/index.js

@@ -11,10 +11,10 @@ function dotenvFile(env = process.env) {
   return join(resolve(lib), ".env");
 }
 var loadedFromDotenv = /* @__PURE__ */ new Set();
-function loadDotenv(file = dotenvFile()) {
-  if (!existsSync(file)) return [];
+function loadDotenv(file2 = dotenvFile()) {
+  if (!existsSync(file2)) return [];
   const loaded = [];
-  for (const raw of readFileSync(file, "utf8").split(/\r?\n/)) {
+  for (const raw of readFileSync(file2, "utf8").split(/\r?\n/)) {
     const line = raw.trim();
     if (!line || line.startsWith("#")) continue;
     const m = /^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/.exec(line);
@@ -60,6 +60,11 @@ var contactEmail = (env = process.env) => str("FH_LIBRARY_CONTACT_EMAIL", env) |
 // src/http.ts
 var TIMEOUT_MS = 3e4;
 var CHROME_UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36";
+var BROWSER_HEADERS = {
+  "User-Agent": CHROME_UA,
+  "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
+  "Accept-Language": "en-US,en;q=0.9"
+};
 var LIBRARY_UA = `fh-library (+https://github.com/forever-healthy/fh-library; ${contactEmail()})`;
 function httpReason(status, body) {
   const m = body.length < 4e3 && /<statusCode>([A-Z_]+)<\/statusCode>|"statusCode"\s*:\s*"([A-Z_]+)"/.exec(body);
@@ -655,12 +660,12 @@ function headingOf(line) {
   if (bare2.length > 60 || !/^[A-Za-z][A-Za-z &'\u2019]*$/.test(bare2)) return null;
   return BY_SQUEEZED.get(squeeze(bare2)) ?? null;
 }
-async function pdfToMarkdown(bytes, title, headLine2, stats) {
+async function pdfToMarkdown(bytes, title, headLine2, stats2) {
   const pdf = await getDocumentProxy(new Uint8Array(bytes), { fontExtraProperties: true });
   const text3 = await extractPages(pdf);
-  if (stats) {
-    stats.pages = pdf.numPages;
-    stats.chars = text3.replace(/\s+/g, " ").trim().length;
+  if (stats2) {
+    stats2.pages = pdf.numPages;
+    stats2.chars = text3.replace(/\s+/g, " ").trim().length;
   }
   const lines = text3.split(/\r?\n/).map((l) => l.replace(/\s+/g, " ").trim());
   const out = [`# ${title}`, "", headLine2, ""];
@@ -1000,6 +1005,8 @@ var NAV_TIMEOUT_MS = 2e4;
 var CHALLENGE_WAIT_MS = 8e3;
 var DOWNLOAD_WAIT_MS = 1e4;
 var IDLE_CLOSE_MS = 12e4;
+var SETTLE_MS = 8e3;
+var SETTLE_WORDS = 100;
 var ctxs = /* @__PURE__ */ new Map();
 var idleTimer = null;
 var LAUNCH_ARGS = ["--disable-blink-features=AutomationControlled", "--window-size=1366,900"];
@@ -1141,6 +1148,12 @@ async function browserFetch(url, endpoint, cfg = browserConfig()) {
       if (!r.ok()) throw new Error(`HTTP ${r.status()} (pdf request)`);
       return { bytes: Buffer.from(await r.body()), contentType: "application/pdf", url: page.url(), via: "request" };
     }
+    const words2 = () => page.evaluate(() => (document.body?.innerText ?? "").split(/\s+/).filter(Boolean).length).catch(() => 0);
+    if (await words2() < SETTLE_WORDS) {
+      await page.waitForLoadState("networkidle", { timeout: SETTLE_MS }).catch(() => void 0);
+      const t2 = Date.now();
+      while (Date.now() - t2 < SETTLE_MS && await words2() < SETTLE_WORDS) await new Promise((r) => setTimeout(r, 1e3));
+    }
     const html = await page.content();
     return { bytes: Buffer.from(html, "utf8"), contentType: contentType || "text/html", url: page.url(), via: "page" };
   } finally {
@@ -1150,14 +1163,39 @@ async function browserFetch(url, endpoint, cfg = browserConfig()) {
 }
 
 // src/unlock.ts
+import { request as httpsRequest } from "https";
 var ENDPOINT = "https://api.brightdata.com/request";
 var TIMEOUT_MS2 = 6e4;
+function post(url, init) {
+  return new Promise((resolve3, reject) => {
+    const req = httpsRequest(url, { method: "POST", headers: { ...init.headers, "Content-Length": String(Buffer.byteLength(init.body)) }, signal: init.signal }, (res) => {
+      const chunks = [];
+      res.on("data", (c) => chunks.push(c));
+      res.on("error", reject);
+      res.on("end", () => {
+        const body = Buffer.concat(chunks);
+        const status = res.statusCode ?? 0;
+        resolve3({
+          ok: status >= 200 && status < 300,
+          status,
+          headers: { get: (name) => {
+            const v = res.headers[name.toLowerCase()];
+            return v == null ? null : Array.isArray(v) ? v.join(", ") : v;
+          } },
+          arrayBuffer: async () => body.buffer.slice(body.byteOffset, body.byteOffset + body.byteLength)
+        });
+      });
+    });
+    req.on("error", (e) => reject(e.name === "AbortError" || e.name === "TimeoutError" ? new Error("timeout") : e));
+    req.end(init.body);
+  });
+}
 var unlockerConfigured = () => !!process.env.FH_LIBRARY_BRIGHTDATA_API_TOKEN?.trim();
-async function unlock(url, fetchImpl = fetch) {
+async function unlock(url, postImpl = post) {
   const token = process.env.FH_LIBRARY_BRIGHTDATA_API_TOKEN?.trim();
   if (!token) throw new Error("unlocker not configured");
   const zone = process.env.FH_LIBRARY_BRIGHTDATA_ZONE?.trim() || "mcp_unlocker";
-  const res = await fetchImpl(ENDPOINT, {
+  const res = await postImpl(ENDPOINT, {
     method: "POST",
     headers: { "Authorization": `Bearer ${token}`, "Content-Type": "application/json" },
     body: JSON.stringify({ zone, url, format: "raw" }),
@@ -1264,13 +1302,13 @@ async function convertBytes(p, bytes, contentType = "") {
   let md;
   if (looksLikePdf(bytes)) {
     format = "pdf";
-    const stats = {};
+    const stats2 = {};
     try {
-      md = await pdfToMarkdown(bytes, p.title, p.headLine, stats);
+      md = await pdfToMarkdown(bytes, p.title, p.headLine, stats2);
     } catch (e) {
       return { ok: false, reason: `pdf: ${errMsg(e)}` };
     }
-    if ((stats.pages ?? 0) >= 1 && (stats.chars ?? 0) < 500 * (stats.pages ?? 1)) return { ok: false, reason: "scanned pdf", format };
+    if ((stats2.pages ?? 0) >= 1 && (stats2.chars ?? 0) < 500 * (stats2.pages ?? 1)) return { ok: false, reason: "scanned pdf", format };
   } else {
     const text3 = bytes.toString("utf8");
     if (isWallPage(text3)) return { ok: false, reason: "wall page" };
@@ -1462,15 +1500,15 @@ function inboxTier(dir = INBOX_DIR) {
     always: true,
     applies: (p) => p.pmid ? true : "no pmid",
     async run(p, ctx) {
-      const file = findInboxFile(dir, p.pmid);
-      if (!file) return { ...miss("inbox", "no file"), skipped: true, ephemeral: true };
-      const name = file.slice(file.lastIndexOf("/") + 1);
+      const file2 = findInboxFile(dir, p.pmid);
+      if (!file2) return { ...miss("inbox", "no file"), skipped: true, ephemeral: true };
+      const name = file2.slice(file2.lastIndexOf("/") + 1);
       const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
-      const bytes = readFileSync3(file);
+      const bytes = readFileSync3(file2);
       const r = await convertBytes(p, bytes, EXT[ext]);
       if (!r.ok) return miss("inbox", `${name}: ${r.reason}`);
       ctx.onTry?.(`${name} ok, removed from the inbox`);
-      unlinkSync(file);
+      unlinkSync(file2);
       return { ok: true, tier: "inbox", format: r.format, raw: r.format === "pdf" ? bytes : bytes.toString("utf8"), md: r.md, version: "published" };
     }
   };
@@ -1743,16 +1781,16 @@ function stamp() {
   return `${d.getFullYear()}-${p(d.getMonth() + 1)}-${p(d.getDate())} ${p(d.getHours())}:${p(d.getMinutes())}:${p(d.getSeconds())}`;
 }
 var LOG_MAX_BYTES = 50 * 1024 * 1024;
-function rotateIfLarge(file, max = LOG_MAX_BYTES) {
+function rotateIfLarge(file2, max = LOG_MAX_BYTES) {
   let size = 0;
   try {
-    size = statSync(file).size;
+    size = statSync(file2).size;
   } catch {
     return false;
   }
   if (size <= max) return false;
   try {
-    renameSync(file, `${file}.1`);
+    renameSync(file2, `${file2}.1`);
     return true;
   } catch {
     return false;
@@ -1770,9 +1808,9 @@ function log(tool, id, message) {
   }
 }
 
-// src/tools.ts
-import { existsSync as existsSync8, readFileSync as readFileSync7 } from "fs";
-import { z } from "zod";
+// src/stats.ts
+import { mkdirSync as mkdirSync4, readdirSync as readdirSync3, readFileSync as readFileSync5, statSync as statSync2, unlinkSync as unlinkSync2 } from "fs";
+import { join as join7 } from "path";
 
 // src/entry.ts
 import { copyFileSync, existsSync as existsSync5, mkdirSync as mkdirSync3, readdirSync as readdirSync2, readFileSync as readFileSync4, renameSync as renameSync2, rmSync, writeFileSync } from "fs";
@@ -2591,10 +2629,10 @@ function readMeta(p) {
     attempts: splitAttempts(fm.attempts)
   };
 }
-function writeAtomic(file, data) {
-  const tmp = `${file}.${process.pid}.${Math.random().toString(36).slice(2, 8)}.tmp`;
+function writeAtomic(file2, data) {
+  const tmp = `${file2}.${process.pid}.${Math.random().toString(36).slice(2, 8)}.tmp`;
   writeFileSync(tmp, data);
-  renameSync2(tmp, file);
+  renameSync2(tmp, file2);
 }
 function writeMeta(p, m) {
   writeAtomic(p.meta, renderMeta(m));
@@ -2873,6 +2911,67 @@ function writeDoiEntry(p, r, json2, converter, abstractFrom = "crossref", record
   return meta;
 }
 
+// src/stats.ts
+var HEARTBEAT_MS = 5 * 6e4;
+var STALE_MS = 20 * 6e4;
+var stats = null;
+var file = null;
+function startStats(pkg2, libraryDir = FH_LIBRARY_DIR) {
+  stats = { host: HOST, pid: process.pid, package: pkg2.name, version: pkg2.version, started: (/* @__PURE__ */ new Date()).toISOString(), verify: { hits: 0, misses: 0 }, fetch: { hits: 0, misses: 0 } };
+  file = join7(libraryDir, ".servers", `${HOST}-${process.pid}.json`);
+  sweep(join7(libraryDir, ".servers"));
+  publish();
+  setInterval(publish, HEARTBEAT_MS).unref();
+  const stop = () => {
+    try {
+      if (file) unlinkSync2(file);
+    } catch {
+    }
+  };
+  process.on("exit", stop);
+  for (const sig of ["SIGINT", "SIGTERM", "SIGHUP"]) process.on(sig, () => process.exit(0));
+  return stats;
+}
+function sweep(dir, staleMs = STALE_MS) {
+  let n = 0;
+  let names;
+  try {
+    names = readdirSync3(dir);
+  } catch {
+    return 0;
+  }
+  for (const name of names) {
+    if (!name.endsWith(".json")) continue;
+    const p = join7(dir, name);
+    try {
+      if (Date.now() - statSync2(p).mtimeMs > staleMs) {
+        unlinkSync2(p);
+        n++;
+      }
+    } catch {
+    }
+  }
+  return n;
+}
+function bump(kind, hit) {
+  if (!stats) return;
+  stats[kind][hit ? "hits" : "misses"]++;
+  publish();
+}
+var currentStats = () => stats;
+function publish() {
+  if (!stats || !file) return;
+  try {
+    mkdirSync4(join7(file, ".."), { recursive: true });
+    writeAtomic(file, JSON.stringify(stats, null, 2) + "\n");
+  } catch {
+  }
+}
+
+// src/tools.ts
+import { existsSync as existsSync10, readFileSync as readFileSync10 } from "fs";
+import { z } from "zod";
+
 // src/cascade.ts
 var PERMANENT = /^(no pmcid|no doi|no pmid|no pmid or doi|no body|closed|no oa copy|no free copy|not in core|no full text|xml without body|template has no placeholder|title missing|title not in text|no methods\/results|no article body|scanned pdf|not a pdf|only copies already tried|only pmc copies|nothing located|unknown content|no text-mining links|needs [a-z/]+ token)/i;
 var BLOCKED = /^(challenge not cleared|empty response)/i;
@@ -2945,11 +3044,11 @@ function parseId(raw) {
 }
 
 // src/info.ts
-import { existsSync as existsSync7, readdirSync as readdirSync3, readFileSync as readFileSync6, statSync as statSync2 } from "fs";
-import { dirname as dirname2, join as join7 } from "path";
+import { existsSync as existsSync7, readdirSync as readdirSync4, readFileSync as readFileSync7, statSync as statSync3 } from "fs";
+import { dirname as dirname2, join as join8 } from "path";
 
 // src/verify.ts
-import { existsSync as existsSync6, readFileSync as readFileSync5 } from "fs";
+import { existsSync as existsSync6, readFileSync as readFileSync6 } from "fs";
 var DEFAULT_MODEL = "pplx-decider-flash-v1.2";
 var QUESTION = "Does the paper state this, in substance? Ignore section labels and definitions of terms in parentheses. Every fact and number of the statement must be stated in the paper or follow from its numbers; a rounded number counts. If any part of the statement is absent from the paper or contradicted by it, answer false.";
 function verifyConfig(env = process.env) {
@@ -3051,9 +3150,9 @@ async function readPaper(raw, libraryDir = FH_LIBRARY_DIR) {
   const paths = entryPaths(libraryDir, p.type, p.id);
   if (!entryExists(paths)) return { id: p.requested, error: "not in library" };
   const full = existsSync6(paths.fulltextMd);
-  if (full) return { id: p.requested, basis: "full_text", text: readFileSync5(paths.fulltextMd, "utf8") };
+  if (full) return { id: p.requested, basis: "full_text", text: readFileSync6(paths.fulltextMd, "utf8") };
   const meta = readMeta(paths);
-  return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text: readFileSync5(paths.md, "utf8") };
+  return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text: readFileSync6(paths.md, "utf8") };
 }
 async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
   const out = { claim, id, verdict: "error", reason: null, score: null, basis: null, windows: 0, glyphs_unresolved: 0, superseded: null, model: cfg?.model ?? null };
@@ -3095,9 +3194,9 @@ async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
 // src/info.ts
 function* metaFiles(dir) {
   if (!existsSync7(dir)) return;
-  for (const name of readdirSync3(dir)) {
-    const p = join7(dir, name);
-    if (statSync2(p).isDirectory()) yield* metaFiles(p);
+  for (const name of readdirSync4(dir)) {
+    const p = join8(dir, name);
+    if (statSync3(p).isDirectory()) yield* metaFiles(p);
     else if (name.endsWith("-meta.md")) yield p;
   }
 }
@@ -3135,11 +3234,11 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
     updates: 0
   };
   const ids = [];
-  for (const type of existsSync7(libraryDir) ? readdirSync3(libraryDir) : []) {
-    const typeDir = join7(libraryDir, type);
-    if (!statSync2(typeDir).isDirectory()) continue;
+  for (const type of existsSync7(libraryDir) ? readdirSync4(libraryDir) : []) {
+    const typeDir = join8(libraryDir, type);
+    if (!statSync3(typeDir).isDirectory()) continue;
     for (const meta of metaFiles(typeDir)) {
-      const md = readFileSync6(meta, "utf8");
+      const md = readFileSync7(meta, "utf8");
       const raw = field(md, "level") || "unknown";
       const level = raw === "abstract" && field(md, "has_abstract") === "no" ? "title" : raw;
       info.entries++;
@@ -3149,9 +3248,9 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
       t.entries++;
       if (level === "full_text" || level === "abstract" || level === "title") t[level]++;
       const dir = dirname2(meta);
-      for (const f of readdirSync3(dir)) {
+      for (const f of readdirSync4(dir)) {
         try {
-          const st = statSync2(join7(dir, f));
+          const st = statSync3(join8(dir, f));
           if (st.isFile()) info.size_bytes += st.size;
         } catch {
         }
@@ -3180,7 +3279,7 @@ function noticeTypes(md, metaPath) {
     return out;
   }
   try {
-    const xml = readFileSync6(metaPath.replace(/-meta\.md$/, "-abstract.xml"), "utf8");
+    const xml = readFileSync7(metaPath.replace(/-meta\.md$/, "-abstract.xml"), "utf8");
     for (const m of xml.matchAll(/<CommentsCorrections RefType="([A-Za-z]+)"/g)) if (XML_TYPES[m[1]]) out.add(XML_TYPES[m[1]]);
   } catch {
   }
@@ -3557,15 +3656,342 @@ function parseCsl(json2) {
   };
 }
 
+// src/urlfetch.ts
+import { createHash as createHash2 } from "crypto";
+import { existsSync as existsSync8, mkdirSync as mkdirSync5, readFileSync as readFileSync8 } from "fs";
+import { join as join9 } from "path";
+var TRACKING = /^(utm_[a-z]+|fbclid|gclid|dclid|msclkid|mc_cid|mc_eid|igshid|ref_src)$/i;
+function normalizeUrl(raw) {
+  let u;
+  try {
+    u = new URL(raw.trim());
+  } catch {
+    return null;
+  }
+  if (u.protocol !== "http:" && u.protocol !== "https:") return null;
+  u.hash = "";
+  u.hostname = u.hostname.toLowerCase();
+  if ([...u.searchParams.keys()].some((k) => TRACKING.test(k))) {
+    for (const k of [...u.searchParams.keys()]) if (TRACKING.test(k)) u.searchParams.delete(k);
+    if (![...u.searchParams.keys()].length) u.search = "";
+  }
+  return u.toString();
+}
+var urlHash = (normalized2) => createHash2("sha1").update(normalized2).digest("hex").slice(0, 16);
+function urlPaths(libraryDir, normalized2) {
+  const hash = urlHash(normalized2);
+  const entry = `url-${hash}`;
+  const entryDir = join9(libraryDir, "url", hash.slice(0, 2), hash.slice(0, 4), hash);
+  return { hash, entry, entryDir, meta: join9(entryDir, `${entry}-meta.md`), md: join9(entryDir, `${entry}-page.md`), raw: (ext) => join9(entryDir, `${entry}-page.${ext}`) };
+}
+function renderUrlMeta(m) {
+  const lines = [
+    ["url", m.url],
+    ["final_url", m.final_url],
+    ["title", m.title === null ? "" : JSON.stringify(m.title)],
+    ["format", m.format],
+    ["content_type", m.content_type],
+    ["level", m.level],
+    ["retrieved_by", m.retrieved_by],
+    ["fetched", m.fetched],
+    ["error", m.error ?? ""],
+    ["files", m.files.join(", ")],
+    ["converted_by", m.converted_by]
+  ];
+  return lines.map(([k, v]) => `${k}: ${v}`).join("\n") + "\n";
+}
+function readUrlMeta(p) {
+  if (!existsSync8(p.meta)) return null;
+  const md = readFileSync8(p.meta, "utf8");
+  const f = (k) => {
+    const m = new RegExp(`^${k}:\\s*(.*)$`, "m").exec(md);
+    return m ? m[1].trim() : "";
+  };
+  let title = null;
+  try {
+    title = f("title") ? JSON.parse(f("title")) : null;
+  } catch {
+    title = f("title") || null;
+  }
+  return {
+    url: f("url"),
+    final_url: f("final_url"),
+    title,
+    format: f("format") || "failed",
+    content_type: f("content_type"),
+    level: f("level") === "full_text" ? "full_text" : "title",
+    retrieved_by: f("retrieved_by"),
+    fetched: f("fetched"),
+    error: f("error") || null,
+    files: f("files") ? f("files").split(",").map((s) => s.trim()).filter(Boolean) : [],
+    converted_by: f("converted_by")
+  };
+}
+function writeUrlMeta(p, m) {
+  mkdirSync5(p.entryDir, { recursive: true });
+  writeAtomic(p.meta, renderUrlMeta(m));
+}
+var SHELL_WORDS = 100;
+var EMPTY_WORDS = 10;
+var SHELL_PHRASE = /\bloading\b|javascript|enable it|connecting to|please wait|redirecting/i;
+function isScriptShell(markdownOrHtml) {
+  const text3 = markdownOrHtml.replace(/<script[\s\S]*?<\/script>|<style[\s\S]*?<\/style>|<noscript[\s\S]*?<\/noscript>/gi, " ").replace(/<[^>]+>/g, " ").replace(/&[a-z#0-9]+;/gi, " ").replace(/\]\([^)]*\)/g, ")");
+  const words2 = text3.split(/\s+/).filter((w) => /[\p{L}\p{N}]/u.test(w)).length;
+  return { shell: words2 < EMPTY_WORDS || words2 < SHELL_WORDS && SHELL_PHRASE.test(text3), words: words2 };
+}
+var viaName = (via) => typeof via === "object" ? `browser ${via.label}` : via;
+async function rung(url, via) {
+  const name = viaName(via);
+  let got;
+  try {
+    got = via === "brightdata" ? await unlock(url) : typeof via === "object" ? await browserFetch(url, via.browser) : await fetchBytes(url, BROWSER_HEADERS);
+  } catch (e) {
+    return { ok: false, reason: errMsg(e), url: e?.url, via: name };
+  }
+  if (via === "brightdata" && !got.bytes.length) return { ok: false, reason: "empty response", via: name };
+  const r = { ok: true, bytes: got.bytes, contentType: got.contentType, url: got.url ?? url, via: name };
+  if (!looksLikePdf(got.bytes)) {
+    const text3 = got.bytes.toString("utf8");
+    if (isHtml(got.contentType, text3)) {
+      if (isWallPage(text3)) return { ok: false, reason: "wall page", url: r.url, via: name };
+      let md = "";
+      try {
+        md = htmlToMarkdown(/<body[^>]*>([\s\S]*)<\/body>/i.exec(text3)?.[1] ?? text3);
+      } catch {
+      }
+      const s = isScriptShell(md || text3);
+      if (s.shell) return { ok: false, reason: `script shell (${s.words} words)`, url: r.url, via: name, shell: r };
+    }
+  }
+  return r;
+}
+var climb = async (url, onTry, onAlert) => {
+  let r = await rung(url, "fetch");
+  const firstReason = r.ok ? void 0 : r.reason;
+  const target = !r.ok && r.url && r.url !== url ? r.url : url;
+  if (target !== url) onTry?.(`\u2192 ${target}`);
+  onTry?.(r.ok ? "ok fetch" : `failed fetch: ${r.reason}`);
+  const escalates = (g) => !g.ok && (g.via !== "fetch" || /^(HTTP 403|HTTP 429|wall page|script shell)/.test(g.reason));
+  let shell = !r.ok ? r.shell : void 0;
+  for (const b of browserRungs()) {
+    if (!escalates(r)) break;
+    r = await rung(target, { browser: b.endpoint, label: b.label });
+    if (!r.ok && r.shell && !shell) shell = r.shell;
+    onTry?.(r.ok ? `ok browser ${b.label}` : `failed browser ${b.label}: ${r.reason}`);
+    if (!r.ok && r.reason.startsWith(MISMATCH)) onAlert?.(`browser ${b.label}: ${r.reason}`);
+  }
+  if (!r.ok && unlockerConfigured() && (r.via !== "fetch" || /^(HTTP 403|wall page|script shell)/.test(r.reason))) {
+    r = await rung(target, "brightdata");
+    onTry?.(r.ok ? "ok brightdata" : `failed brightdata: ${r.reason}`);
+    if (!r.ok && /^unlocker HTTP/.test(r.reason)) onAlert?.(`brightdata: ${r.reason}`);
+  }
+  if (!r.ok && shell) {
+    onTry?.(`script shell kept: no rung rendered the page`);
+    return shell;
+  }
+  return r.ok || r.via === "fetch" ? r : { ...r, first: firstReason };
+};
+var mediaType = (contentType) => contentType.split(";")[0].trim().toLowerCase();
+function isHtml(contentType, text3) {
+  const ct = contentType.toLowerCase();
+  if (/<html/i.test(text3.slice(0, 100))) return true;
+  if (ct.includes("text/html") || ct.includes("application/xhtml+xml")) return true;
+  return !ct && /^(<!doctype html|<html)/.test(text3.replace(/^\uFEFF/, "").trimStart().slice(0, 20).toLowerCase());
+}
+var ENTITIES = { amp: "&", lt: "<", gt: ">", quot: '"', apos: "'", nbsp: " ", ndash: "\u2013", mdash: "\u2014", hellip: "\u2026", copy: "\xA9", reg: "\xAE", trade: "\u2122" };
+var decode2 = (s) => s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
+  if (e[0] === "#") {
+    const n = e[1].toLowerCase() === "x" ? parseInt(e.slice(2), 16) : parseInt(e.slice(1), 10);
+    return Number.isFinite(n) ? String.fromCodePoint(n) : m;
+  }
+  return ENTITIES[e.toLowerCase()] ?? m;
+});
+var visibleText = (html) => decode2(html.replace(/<[^>]+>/g, " ")).replace(/\s+/g, " ").replace(/¶\s*$/, "").trim();
+var normalized = (text3) => {
+  const words2 = text3.toLowerCase().match(/[\p{L}\p{N}]+/gu);
+  return words2 ? ` ${words2.join(" ")} ` : "";
+};
+function pageTitle(html) {
+  const head = html.slice(0, 2e5);
+  const og = /<meta\b[^>]*\bproperty\s*=\s*["']og:title["'][^>]*>/i.exec(head)?.[0] ?? /<meta\b[^>]*\bcontent\s*=\s*["'][^"']*["'][^>]*\bproperty\s*=\s*["']og:title["'][^>]*>/i.exec(head)?.[0];
+  const ogTitle = og ? visibleText(/\bcontent\s*=\s*("([^"]*)"|'([^']*)')/i.exec(og)?.[2] ?? /\bcontent\s*=\s*'([^']*)'/i.exec(og)?.[1] ?? "") : "";
+  const title = visibleText(/<title[^>]*>([\s\S]*?)<\/title>/i.exec(head)?.[1] ?? "");
+  const h1 = visibleText(/<h1\b[^>]*>([\s\S]*?)<\/h1>/i.exec(html)?.[1] ?? "");
+  const headingInTitle = !!normalized(h1) && normalized(title).includes(normalized(h1));
+  if (ogTitle) return ogTitle;
+  if (headingInTitle) return h1;
+  return title || h1 || null;
+}
+var visibleLine = (line) => line.replace(/\]\([^)]*\)/g, " ").replace(/<[a-zA-Z/!?][^>]*>/g, " ");
+function opensWithTitle(content, title) {
+  const first = content.split("\n").find((l) => l.trim()) ?? "";
+  const line = normalized(visibleLine(first)), wanted = normalized(title);
+  if (!line || !wanted) return false;
+  if (line.includes(wanted)) return true;
+  return first.trimStart().startsWith("#") && wanted.includes(line);
+}
+function withPageTitle(html, content) {
+  if (content.startsWith("<error>")) return content;
+  const title = pageTitle(html);
+  if (title === null || opensWithTitle(content, title)) return content;
+  return `# ${title.replace(/([\\`*_\[\]<>#&|~])/g, "\\$1")}
+
+${content.trimStart()}`;
+}
+var SIMPLIFY_FAILED = "<error>Page failed to be simplified from HTML</error>";
+async function convert(bytes, contentType) {
+  if (looksLikePdf(bytes)) {
+    const md = (await pdfToMarkdown(bytes, "", "")).replace(/^# \n\n/, "");
+    return { format: "pdf", md, title: null, ext: "pdf" };
+  }
+  const text3 = bytes.toString("utf8");
+  if (mediaType(contentType) === "text/markdown") return { format: "markdown", md: text3, title: null, ext: "md" };
+  if (isHtml(contentType, text3)) {
+    const body = /<body[^>]*>([\s\S]*)<\/body>/i.exec(text3)?.[1] ?? text3;
+    let md;
+    try {
+      md = htmlToMarkdown(body);
+    } catch {
+      md = "";
+    }
+    md = md.trim() ? withPageTitle(text3, md) : SIMPLIFY_FAILED;
+    return { format: "html", md, title: pageTitle(text3), ext: "html" };
+  }
+  return { format: "text", md: null, title: null, ext: "txt" };
+}
+function fetchAnswer(url, body, prefix, maxLength, startIndex) {
+  const original = body.length;
+  let content;
+  if (startIndex >= original) content = "<error>No more content available.</error>";
+  else {
+    const cut = body.slice(startIndex, startIndex + maxLength);
+    content = cut;
+    const remaining = original - (startIndex + cut.length);
+    if (cut.length === maxLength && remaining > 0) content += `
+
+<error>Content truncated. Call the fetch tool with a start_index of ${startIndex + cut.length} to get more content.</error>`;
+  }
+  return `${prefix}Contents of ${url}:
+${content}`;
+}
+var ID_WIDTH = 13;
+function logId(url) {
+  const bare2 = url.replace(/^https?:\/\//i, "");
+  return bare2.length > ID_WIDTH ? bare2.slice(0, ID_WIDTH - 1) + "\u2026" : bare2;
+}
+var failureText = (url, reason) => {
+  const m = /^HTTP (\d{3})\b/.exec(reason);
+  return m ? `Failed to fetch ${url} - status code ${m[1]}` : `Failed to fetch ${url}: ${reason}`;
+};
+function serve(requested, p, m, args, cached) {
+  const maxLength = args.max_length ?? 5e3, startIndex = args.start_index ?? 0;
+  const read = (file2) => existsSync8(file2) ? readFileSync8(file2, "utf8") : "";
+  const rawFile = m.files.find((f) => !f.endsWith("-meta.md") && !f.endsWith("-page.md"));
+  let body, prefix = "";
+  if (m.format === "markdown") body = read(p.md);
+  else if (m.format === "text" || args.raw) {
+    body = rawFile ? read(join9(p.entryDir, rawFile)) : read(p.md);
+    prefix = `Content type ${m.content_type || "(not declared)"} cannot be simplified to markdown, but here is the raw content:
+`;
+  } else body = read(p.md);
+  return { ok: true, text: fetchAnswer(requested, body, prefix, maxLength, startIndex), cached, meta: m };
+}
+var inflight = /* @__PURE__ */ new Map();
+async function libraryFetch(libraryDir, requested, args, converter, logs = {}, fetchImpl = climb) {
+  const url = normalizeUrl(requested);
+  if (!url) return { ok: false, error: `Failed to fetch ${requested}: invalid URL`, cached: false, meta: null };
+  const p = urlPaths(libraryDir, url);
+  const s = settings();
+  const fresh = (m2) => daysSince(m2.fetched) < (m2.format === "failed" ? s.retry_after_days : s.refresh_after_days);
+  const answer = (m2, cached) => m2.format === "failed" ? { ok: false, error: failureText(requested, m2.error ?? "unknown error"), cached, meta: m2 } : serve(requested, p, m2, args, cached);
+  const stored = readUrlMeta(p);
+  if (stored && fresh(stored)) return answer(stored, true);
+  let run = inflight.get(p.hash);
+  if (!run) {
+    run = (async () => {
+      const r = await fetchImpl(url, logs.onTry, logs.onAlert);
+      const base = { url, fetched: today(), converted_by: converter };
+      if (!r.ok) {
+        const error = r.first && /^HTTP \d{3}/.test(r.first) ? r.first : r.reason;
+        writeUrlMeta(p, { ...base, final_url: r.url ?? url, title: null, format: "failed", content_type: "", level: "title", retrieved_by: r.via, error, files: [`${p.entry}-meta.md`] });
+        return;
+      }
+      const page = await convert(r.bytes, r.contentType);
+      mkdirSync5(p.entryDir, { recursive: true });
+      const files = [`${p.entry}-meta.md`];
+      if (page.md !== null) {
+        writeAtomic(p.md, page.md);
+        files.push(`${p.entry}-page.md`);
+      }
+      if (page.ext !== "md") {
+        writeAtomic(p.raw(page.ext), r.bytes);
+        files.push(`${p.entry}-page.${page.ext}`);
+      }
+      writeUrlMeta(p, { ...base, final_url: r.url, title: page.title, format: page.format, content_type: mediaType(r.contentType), level: "full_text", retrieved_by: r.via, error: null, files });
+    })().finally(() => inflight.delete(p.hash));
+    inflight.set(p.hash, run);
+  }
+  await run;
+  const m = readUrlMeta(p);
+  if (!m) return { ok: false, error: `Failed to fetch ${requested}: nothing stored`, cached: false, meta: null };
+  return answer(m, false);
+}
+
+// src/verifycache.ts
+import { createHash as createHash3 } from "crypto";
+import { existsSync as existsSync9, mkdirSync as mkdirSync6, readFileSync as readFileSync9 } from "fs";
+import { dirname as dirname3, join as join10 } from "path";
+var sha = (s) => createHash3("sha1").update(s).digest("hex");
+var normalizeClaim = (claim) => claim.replace(/\s+/g, " ").trim();
+var textHash = (text3) => sha(text3).slice(0, 16);
+var cacheKey = (claim, id, model, threshold, text3) => sha(`${normalizeClaim(claim)}
+${id}
+${model}
+${threshold}
+${text3}`).slice(0, 16);
+var verdictFile = (libraryDir, key) => join10(libraryDir, "verify", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
+function readVerdict(file2) {
+  if (!existsSync9(file2)) return null;
+  try {
+    return JSON.parse(readFileSync9(file2, "utf8"));
+  } catch {
+    return null;
+  }
+}
+function lookup(file2, maxDays) {
+  const c = readVerdict(file2);
+  return c && daysSince(c.date) < maxDays ? c : null;
+}
+function remember(file2, c) {
+  mkdirSync6(dirname3(file2), { recursive: true });
+  writeAtomic(file2, JSON.stringify(c, null, 2) + "\n");
+}
+async function verifyCached(claim, id, cfg, decide, libraryDir = FH_LIBRARY_DIR, read = readPaper) {
+  const paper = await read(id, libraryDir);
+  const once = () => paper;
+  if ("error" in paper || !cfg) return { ...await verifyClaim(claim, id, cfg, decide, once), cached: false };
+  const text3 = textHash(paper.text);
+  const file2 = verdictFile(libraryDir, cacheKey(claim, paper.id, cfg.model, cfg.threshold, text3));
+  const hit = lookup(file2, settings().refresh_after_days);
+  if (hit) {
+    return { claim, id: paper.id, verdict: hit.verdict, reason: hit.reason, score: hit.score, basis: hit.basis, windows: hit.windows, glyphs_unresolved: hit.glyphs_unresolved, superseded: hit.superseded, model: hit.model, cached: true };
+  }
+  const r = await verifyClaim(claim, id, cfg, decide, once);
+  if (!r.error && r.score !== null) {
+    remember(file2, { key: file2.slice(-21, -5), id: paper.id, claim: normalizeClaim(claim), model: cfg.model, threshold: cfg.threshold, text: text3, date: today(), verdict: r.verdict, reason: r.reason, score: r.score, basis: r.basis, windows: r.windows, glyphs_unresolved: r.glyphs_unresolved, superseded: r.superseded });
+  }
+  return { ...r, cached: false };
+}
+
 // src/tools.ts
 var pkg = (() => {
   try {
-    return JSON.parse(readFileSync7(new URL("../package.json", import.meta.url), "utf8"));
+    return JSON.parse(readFileSync10(new URL("../package.json", import.meta.url), "utf8"));
   } catch {
     return { name: "fh-library", version: "0.0.0" };
   }
 })();
-var INSTRUCTIONS = `fh-library keeps a local library of scientific papers, FDA drug labels and clinical trial records, so you can check what a source actually says. The server downloads each record, converts it to Markdown and stores it on disk. Every tool returns a small JSON answer; the document itself never enters your context. One ID per call.
+var INSTRUCTIONS = `fh-library keeps a local library of scientific papers, FDA drug labels and clinical trial records, so you can check what a source actually says. The server downloads each record, converts it to Markdown and stores it on disk. Every tool returns a small JSON answer; the document itself never enters your context \u2014 except library_fetch, which is the standard fetch tool with a cache. One ID per call.
 
 IDs \u2014 always "type:value"
 - pmid:30955454 \u2014 a paper by PubMed ID. Bare digits are taken as a PMID.
@@ -3575,11 +4001,12 @@ IDs \u2014 always "type:value"
 
 Tools
 - library_version() \u2014 package name, version, library directory.
-- library_info(ids?) \u2014 the whole library: entry counts by type and level, the retrieval sources, browsers, the decision model, the configuration. ids=true lists every entry.
+- library_info(ids?) \u2014 the whole library: entry counts by type and level, the retrieval sources, browsers, the decision model, the configuration, this server's cache hits and misses since it started (cache). ids=true lists every entry.
 - library_add(id) \u2014 download and store one record; returns its status. A stored full text is returned without any download; an abstract-level record retries only the sources that failed transiently; a record older than 30 days gets its correction notices re-read.
 - library_status(id) \u2014 the status of one record without downloading; level "missing" when it is not stored.
 - library_read(id, section?, query?, max_chars?, max_hits?) \u2014 one small piece of a stored document: its outline, one section, or the passages matching a query.
 - library_verify(claim, id) \u2014 does this one source state this one finding? A decision model judges the stored text and returns a verdict.
+- library_fetch(url, max_length?, start_index?, raw?) \u2014 a web page as Markdown, same call and same answer as the standard fetch tool ("Contents of <url>:" and the page, cut at max_length with a note on how to continue). Prefer it for every URL: a cached page answers from disk, a walled or script-rendered page climbs the browser and unlocker ladder in one call. PDFs as Markdown.
 
 Hints
 - library_read and library_verify store a missing record themselves; the library_read answer then carries added: true. No library_add necessary first.
@@ -3587,6 +4014,7 @@ Hints
 - level \u2014 what is stored: "full_text", "abstract" (no full text could be found; attempts names each source tried and why it failed) or "title" (not even an abstract; citable false).
 - error "pmid not found" or "invalid id" \u2014 the ID itself is wrong; the same call gives the same answer.
 - Calls may run in parallel: library_add, library_read and library_verify. Two calls on the same missing record share one add.
+- library_fetch takes any URL that is not a paper, label or trial record; those have IDs, and library_add, library_read and library_verify work on the stored document. A second pass over the same review reads every page from disk; a failed fetch is remembered for a day.
 
 Verifying a claim
 - library_verify judges one claim against one stored document. A finding backed by three papers takes three calls with the same claim.
@@ -3598,6 +4026,7 @@ Verifying a claim
 - verdict "unverifiable" \u2014 only the abstract is stored and it does not state the claim; the full text may (reason says so). A retry gives the same answer until a full text is stored.
 - glyphs_unresolved above 0 \u2014 the PDF font lost symbols (minus, \xB1, \u2264); a fail on a number may be the font. library_read shows the passage.
 - superseded true \u2014 a published correction replaced the figure the claim uses; the status's corrections holds the notice.
+- Verdicts are kept for 30 days: the same claim against the same stored text answers from disk, no model call. A paper that gains its full text is judged afresh.
 
 Reading a document
 - library_read with only the ID returns the outline: every heading with its word count, and whether the full text or only the abstract is stored.
@@ -3737,13 +4166,13 @@ function registerTools(server2) {
     { id: idSchema },
     ({ id }) => addEntry(id)
   );
-  const inflight = /* @__PURE__ */ new Map();
+  const inflight2 = /* @__PURE__ */ new Map();
   function addEntry(id) {
     const key = parseId(id).requested;
-    let p = inflight.get(key);
+    let p = inflight2.get(key);
     if (!p) {
-      p = addEntryNow(id).finally(() => inflight.delete(key));
-      inflight.set(key, p);
+      p = addEntryNow(id).finally(() => inflight2.delete(key));
+      inflight2.set(key, p);
     }
     return p;
   }
@@ -3930,8 +4359,8 @@ function registerTools(server2) {
         }
         added = true;
       }
-      const full = existsSync8(paths.fulltextMd);
-      const md = readFileSync7(full ? paths.fulltextMd : paths.md, "utf8");
+      const full = existsSync10(paths.fulltextMd);
+      const md = readFileSync10(full ? paths.fulltextMd : paths.md, "utf8");
       const text3 = full ? "full_text" : "abstract";
       const base = added ? { requested: parsed.requested, entry: paths.entry, text: text3, added: true } : { requested: parsed.requested, entry: paths.entry, text: text3 };
       const headings2 = () => outline(md, text3).sections;
@@ -3951,8 +4380,8 @@ function registerTools(server2) {
         const cap = max_chars ?? 6e3;
         let sec = readSection(md, section2, cap);
         let via = "";
-        if (!sec && /abstract/i.test(section2) && full && existsSync8(paths.md)) {
-          sec = readSection(readFileSync7(paths.md, "utf8"), "abstract", cap);
+        if (!sec && /abstract/i.test(section2) && full && existsSync10(paths.md)) {
+          sec = readSection(readFileSync10(paths.md, "utf8"), "abstract", cap);
           via = " (from the abstract file)";
         }
         if (!sec && /declaration/i.test(section2)) {
@@ -3996,15 +4425,16 @@ function registerTools(server2) {
         log("library_verify", pid.requested, "missing adding first");
         await addEntry(id);
       }
-      const r = await verifyClaim(claim, id, cfg, cfg ? decider(cfg) : null);
+      const { cached, ...r } = await verifyCached(claim, id, cfg, cfg ? decider(cfg) : null, FH_LIBRARY_DIR);
+      if (!r.error && r.score !== null) bump("verify", cached);
       const secs = `${((Date.now() - t0) / 1e3).toFixed(1).padStart(4, "0")}s`;
-      log("library_verify", r.id, r.error ? `${modelFailure(r.error, cfg) ? "!! " : ""}${secs} verdict=${"error".padEnd(12)} ${r.error} | ${claim}` : `${secs} verdict=${r.verdict.padEnd(12)} basis=${r.basis.padEnd(9)} score=${r.score === null ? "-    " : r.score.toFixed(3)}${r.windows > 1 ? ` windows=${r.windows}` : ""}${r.superseded ? " superseded" : ""}${r.glyphs_unresolved ? ` glyphs=${r.glyphs_unresolved}` : ""} | ${claim}`);
+      log("library_verify", r.id, r.error ? `${modelFailure(r.error, cfg) ? "!! " : ""}${secs} verdict=${"error".padEnd(12)} ${r.error} | ${claim}` : `${secs} verdict=${r.verdict.padEnd(12)} basis=${r.basis.padEnd(9)} score=${r.score === null ? "-    " : r.score.toFixed(3)}${r.windows > 1 ? ` windows=${r.windows}` : ""}${r.superseded ? " superseded" : ""}${r.glyphs_unresolved ? ` glyphs=${r.glyphs_unresolved}` : ""}${cached ? " cached" : ""} | ${claim}`);
       return { content: [{ type: "text", text: JSON.stringify(r, null, 2) }] };
     }
   );
   server2.tool(
     "library_info",
-    "The whole library as JSON: library directory and host; entry count and size on disk; per type (pmid, doi, dailymed, nct) the entries by level (full_text, abstract, title) and retractions; counts of retracted and not-citable entries and of errata, expressions of concern and updates; the retrieval sources in run order and which are off (tiers); the download ladder (download): the Playwright servers with the version they must run, the Chrome rungs, the Bright Data unlocker; the configuration and where each value came from \u2014 environment, .env or default, never the value (config); the decision model behind library_verify and whether it answers right now (verify). ids=true also lists every entry ID.",
+    "The whole library as JSON: library directory and host; entry count and size on disk; per type (pmid, doi, dailymed, nct) the entries by level (full_text, abstract, title) and retractions; counts of retracted and not-citable entries and of errata, expressions of concern and updates; the retrieval sources in run order and which are off (tiers); the download ladder (download): the Playwright servers with the version they must run, the Chrome rungs, the Bright Data unlocker; the configuration and where each value came from \u2014 environment, .env or default, never the value (config); the decision model behind library_verify and whether it answers right now (verify); this server's cache hits and misses for library_verify and library_fetch since it started (cache). ids=true also lists every entry ID.",
     { ids: z.boolean().optional().describe("Also list every entry ID (type:value).") },
     async ({ ids }) => {
       const info = libraryInfo(FH_LIBRARY_DIR, HOST, ids === true);
@@ -4019,7 +4449,35 @@ function registerTools(server2) {
         }
       }
       log("library_info", null, `entries=${info.entries}${info.verify ? ` verify ${info.verify.model} ${info.verify.ok ? `ok ${info.verify.ms} ms` : `down: ${info.verify.error}`}` : " verify not configured"}`);
-      return { content: [{ type: "text", text: JSON.stringify(info, null, 2) }] };
+      const s = currentStats();
+      const cache = s ? { since: s.started, verify: s.verify, fetch: s.fetch } : null;
+      return { content: [{ type: "text", text: JSON.stringify({ ...info, cache }, null, 2) }] };
+    }
+  );
+  server2.tool(
+    "library_fetch",
+    `Fetch a web page as Markdown: the standard fetch tool, cached and hardened. Use it for any URL instead of a plain fetch or a browser. A page fetched in the last ${settings().refresh_after_days} days is answered from disk in milliseconds; a page a plain fetch cannot get (403, rate limit, bot wall, script-rendered) is fetched through the configured browsers and unlocker in this one call. PDFs come back as Markdown.`,
+    {
+      url: z.string().describe("URL to fetch"),
+      max_length: z.number().int().min(1).max(999999).default(5e3).describe("Maximum number of characters to return."),
+      start_index: z.number().int().min(0).default(0).describe("On return output starting at this character index, useful if a previous fetch was truncated and more context is required."),
+      raw: z.boolean().default(false).describe("Get the actual HTML content of the requested page, without simplification.")
+    },
+    async ({ url, max_length, start_index, raw }) => {
+      const t0 = Date.now();
+      const id = logId(url), put = (line) => log("library_fetch", id, `${line} ${url}`);
+      const r = await libraryFetch(FH_LIBRARY_DIR, url, { max_length, start_index, raw }, `${pkg.name} ${pkg.version}`, {
+        onTry: put,
+        onAlert: (line) => put(`!! ${line}`)
+      });
+      const secs = ((Date.now() - t0) / 1e3).toFixed(1).padStart(4, "0") + "s";
+      if (r.meta) bump("fetch", r.cached);
+      if (r.ok) {
+        put(`${r.cached ? "cached" : `fetched ${secs}`} ${r.meta.format} retrieved_by=${r.meta.retrieved_by} fetched=${r.meta.fetched} chars=${r.text.length}${raw ? " raw" : ""}${start_index ? ` start=${start_index}` : ""}`);
+        return { content: [{ type: "text", text: r.text }] };
+      }
+      put(`${r.cached ? "cached failure" : `failed ${secs}`}: ${r.error.replace(url, "<url>")}`);
+      return { content: [{ type: "text", text: r.error }], isError: true };
     }
   );
   server2.tool(
@@ -4039,6 +4497,8 @@ var server = new McpServer(
   { instructions: INSTRUCTIONS }
 );
 registerTools(server);
+startStats(pkg);
+log("startup", null, `${pkg.name} ${pkg.version}`);
 for (const problem of resolveTiers().problems) log("startup", null, `!! ${problem}`);
 var transport = new StdioServerTransport();
 await server.connect(transport);

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "fh-library",
-  "version": "0.2.5",
+  "version": "0.2.7",
   "mcpName": "io.github.forever-healthy/fh-library",
   "description": "MCP Server that builds a local Markdown library of the scientific papers cited by evidence reviews — fetches metadata, abstract and full text per PubMed ID through a configurable cascade of retrieval services",
   "type": "module",