fh-library 0.2.17 → 0.2.20

package/CHANGELOG.md

@@ -1,5 +1,21 @@
 # FH Library — Change Log
 
+### v0.2.20 — 2026-11-09
+
+* **`library_verify` without the reference list:** the decision model no longer gets the paper's References / Bibliography section — other papers' titles, which state nothing this paper found and could pass a claim on a cited title; 22.7% less text per full text (96 full texts measured). The stored text and `library_read` keep it.
+* **Less noise to the decision model:** a paper's citation line (authors, journal, PMID, DOI) is sent as its year only — 9.8% of an abstract.
+
+### v0.2.19 — 2026-11-09
+
+* **Faster full texts:** a browser rung (or Bright Data) that met the same site's challenge twice in a row is skipped for that site for `FH_LIBRARY_RETRY_AFTER_DAYS` — the other rungs still run, a success clears it; the open-access lookups (Europe PMC, Crossref, Unpaywall, OpenAlex) start together instead of one after the other, the downloads stay in tier order.
+* **Smaller status:** without `title`, `first_author` and `doi`.
+
+### v0.2.18 — 2026-11-09
+
+* **Smaller answers per call:** no answer repeats what the call sent. `library_verify` returns `verdict`, `score` (two decimals) and `basis`, plus `reason`, `superseded` or `glyphs_unresolved` only when set, without the claim, the ID or the model; the status drops the ID fields, the refresh dates, `attempts`, `coi` and `funding` (`library_read` section "Declarations" returns the record's statements when the text has none); both searches and the PubMed batch return `total`, `columns` and `rows` only; publication types leave out "Journal Article" and "Research Support, …"; trial enrollment as `48` (actual) or `~100` (estimated), collaborators as the first three and `+N`.
+* **Trial results outcomes first:** `library_clinicaltrials_get_study_results` puts the outcome measures first, then adverse events, and describes each group once instead of in every section — the primary outcomes now sit at the top of the answer instead of after 3,000 characters of arm descriptions.
+* **`library_read` says when there is more:** a query answer carries `more` when further passages match beyond `max_hits`.
+
 ### v0.2.17 — 2026-11-08
 
 * **`library_fetch` answers record URLs from the library:** a PubMed, ClinicalTrials.gov, DailyMed or doi.org link is answered from the library entry (a missing PubMed record is stored at abstract level in one request, no full-text cascade; a DOI PubMed indexes goes through its PMID) in fetch's format — the abstract record for PubMed, the trial record, the label, the full text or abstract for a DOI; `title_only` the title line. No script shell, no browser.

package/README.md

@@ -1,4 +1,4 @@
-![Version 0.1.3](https://img.shields.io/badge/Version-0.2.17-green.svg)
+![Version 0.1.3](https://img.shields.io/badge/Version-0.2.20-green.svg)
 ![License MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
 [![Forever Healthy](https://img.shields.io/badge/(c)_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
 [![npm](https://img.shields.io/badge/npm-fh--library-cb3837.svg)](https://www.npmjs.com/package/fh-library)
@@ -20,11 +20,11 @@ A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a
 
 ## Tools
 
-* `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `coi` and `funding` (PubMed's record, else the paper's own text), `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — files, failed tiers in `attempts`). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `FH_LIBRARY_REFRESH_AFTER_DAYS` gets the record re-read for new notices. One ID per call.
+* `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — the failed tiers are in the log). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `FH_LIBRARY_REFRESH_AFTER_DAYS` gets the record re-read for new notices. One ID per call.
 * `library_status(id)` → read-only check of presence and level; never fetches
 * `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), this server's cache hits and misses since it started (`cache`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
 * `library_verify(claim, id)` → is one finding of a review stated by one paper it cites? `claim` is the finding alone, verbatim, without its citations; `id` one paper (`pmid:`, or any `doi:`), drug label or trial record the review cites for it — a finding cited to three papers is three calls with the same claim; a claim that names a source or carries citations is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters the model's context. `verdict` **pass** (`score` the probability), **fail** (the full text does not state it; or only the title is in the library — `reason` says so), **unverifiable** (only the abstract is in the library and does not state it — `reason`) or **error**; `basis` full_text, abstract or title. See "Verifying claims" below.
-* `library_read(id, section?, query?, max_chars?, max_hits?)` → one small piece of a stored paper's Markdown, never the whole paper: without `section`/`query` the **outline** (headings with word counts, and whether the full text or only the abstract is stored; a heading the paper uses several times is listed as `Participants [3/12]`); `section` → that section's **own text** (heading matched as case-insensitive substring, capped at `max_chars`, default 6000, with `returned_words` against `words` when cut) and, for a parent heading, its `subsections` with word counts to pick from — a duplicated heading named bare lists the candidates instead; `query` — two or three distinctive terms the paper itself would use (a measure and a unit, an outcome and a group, a compound name), not the claim — → **passages** holding every term, each a window of at most 1000 chars around the matches (`clipped: true` when cut; a table row with its header row), with its section, best first (at most `max_hits`, default 5); matching forgives case, accents, hyphens, Markdown markup, plural and -ed/-ing forms; when no passage holds every term the best **partial** matches come back (`partial: true`, the `matched` terms per passage, the rare term first), and `missing_terms` names the terms the text never uses in any form — those are the ones to rephrase. `section` and `query` together search inside that section. `section: "Abstract"` on a full text without one answers from the stored abstract. Every full text converted from JATS XML (PMC, Europe PMC, Elsevier) has a **Declarations** section — funding, conflicts of interest, author notes, gathered from wherever the XML keeps them — and on a text without one `section: "Declarations"` assembles the sections headed funding, conflicts, acknowledgements, ethics, contributions, so it answers who paid and who declares what on every source alike; a paper with neither states nothing. Outline first, then pick. A paper not in the library is added first, as `library_add` would (its lines in the log), then library_read; the answer carries `added: true`. Reads may run in parallel.
+* `library_read(id, section?, query?, max_chars?, max_hits?)` → one small piece of a stored paper's Markdown, never the whole paper: without `section`/`query` the **outline** (headings with word counts, and whether the full text or only the abstract is stored; a heading the paper uses several times is listed as `Participants [3/12]`); `section` → that section's **own text** (heading matched as case-insensitive substring, capped at `max_chars`, default 6000, with `returned_words` against `words` when cut) and, for a parent heading, its `subsections` with word counts to pick from — a duplicated heading named bare lists the candidates instead; `query` — two or three distinctive terms the paper itself would use (a measure and a unit, an outcome and a group, a compound name), not the claim — → **passages** holding every term, each a window of at most 1000 chars around the matches (`clipped: true` when cut; a table row with its header row), with its section, best first (at most `max_hits`, default 5; `more` counts the further passages that match); matching forgives case, accents, hyphens, Markdown markup, plural and -ed/-ing forms; when no passage holds every term the best **partial** matches come back (`partial: true`, the `matched` terms per passage, the rare term first), and `missing_terms` names the terms the text never uses in any form — those are the ones to rephrase. `section` and `query` together search inside that section. `section: "Abstract"` on a full text without one answers from the stored abstract. Every full text converted from JATS XML (PMC, Europe PMC, Elsevier) has a **Declarations** section — funding, conflicts of interest, author notes, gathered from wherever the XML keeps them — and on a text without one `section: "Declarations"` assembles the sections headed funding, conflicts, acknowledgements, ethics, contributions, so it answers who paid and who declares what on every source alike; a paper with neither states nothing. Outline first, then pick. A paper not in the library is added first, as `library_add` would (its lines in the log), then library_read; the answer carries `added: true`. Reads may run in parallel.
 * `library_pubmed_search_articles(query, max_results?, sort?, years?)` → a PubMed search in PubMed's own syntax (see "PubMed in batches"): the total and rows in `columns` order, with the library level of a paper already stored, cached for `FH_LIBRARY_SEARCH_CACHE_DAYS`
 * `library_pubmed_fetch_articles(pmids)` → title, first author, year, publication types, retracted and level for up to 200 PMIDs in one call (see "PubMed in batches"); PMIDs the library lacks are fetched in one PubMed request and stored at abstract level
 * `library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?)` → a ClinicalTrials.gov search (see "Trial search"): the total count and compact rows, cached for a week
@@ -62,7 +62,7 @@ Downloads are plain HTTP with a desktop Chrome User-Agent (API calls identify th
 
 * **Which tiers run, in which order:** `FH_LIBRARY_TIERS=pmc,tdm,unpaywall,openalex` — exactly these, in this order; a tier not listed is off. Unset, every tier runs in the order of the table above, the proxy last.
 * **Keys:** `FH_LIBRARY_ELSEVIER_API_KEY` and `FH_LIBRARY_WILEY_TDM_TOKEN` for `tdm`, `FH_LIBRARY_CORE_API_KEY` for `core`, `FH_LIBRARY_NCBI_API_KEY` (10 instead of 3 requests/s), `FH_LIBRARY_BRIGHTDATA_API_TOKEN` and `FH_LIBRARY_BRIGHTDATA_ZONE` for the unlocker. `FH_LIBRARY_CONTACT_EMAIL` is the one address sent to every service (NCBI `tool`/`email`, Unpaywall and OpenAlex polite pools, Crossref, the User-Agent).
-* **Retries:** `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried, before a page `library_fetch` could not get is fetched again, and before an ID no service knew (a PMID, DOI, set ID or NCT ID that is not found) is asked for again — meanwhile every tool answers "not found" from a marker in the entry's directory, with the date it was checked; `FH_LIBRARY_SEARCH_CACHE_DAYS` (7) how long a PubMed or registry search is answered from disk; `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `library_add` re-reads it for new correction notices and retractions. A tdm link behind a token that is not configured is recorded as `needs elsevier token` and not re-checked; after adding the token, remove those attempts from the entries to give them a go. `FH_LIBRARY_RETRY_BLOCKED` (false) decides whether a browser challenge that did not clear, or a Bright Data refusal, is final: with the same method tomorrow ends the same way, so set it true only after a change of method (a new browser rung, a new service).
+* **Retries:** `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried, before a page `library_fetch` could not get is fetched again, and before an ID no service knew (a PMID, DOI, set ID or NCT ID that is not found) is asked for again — meanwhile every tool answers "not found" from a marker in the entry's directory, with the date it was checked; `FH_LIBRARY_SEARCH_CACHE_DAYS` (7) how long a PubMed or registry search is answered from disk; `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `library_add` re-reads it for new correction notices and retractions. A tdm link behind a token that is not configured is recorded as `needs elsevier token` and not re-checked; after adding the token, remove those attempts from the entries to give them a go. A browser rung or Bright Data that met the same site's challenge twice in a row is skipped for that site (and only that rung) until `FH_LIBRARY_RETRY_AFTER_DAYS` have passed since the last challenge; a success clears it (`library/walls/<host>.json`). `FH_LIBRARY_RETRY_BLOCKED` (false) decides whether a browser challenge that did not clear, or a Bright Data refusal, is final: with the same method tomorrow ends the same way, so set it true only after a change of method (a new browser rung, a new service).
 * **Browsers and verify:** `FH_LIBRARY_PLAYWRIGHT_n` and `FH_LIBRARY_CHROME_n` (hostnames; above) and `FH_LIBRARY_PLAYWRIGHT_HEADLESS`; `FH_LIBRARY_VERIFY_*` ("Verifying claims"). A Playwright server must run the **same `playwright-core` version** as the package — `library_info` shows the required one as `download.playwright.version`, `browser/Dockerfile` builds a server at it, and a server of another version is refused by Playwright itself (the log then carries the alert `!! browser <host>: Playwright version mismatch — server … must run playwright-core <version>`). Moving the version is one release: the pin in `package.json`, the tag in `browser/Dockerfile`, every server rebuilt. The library never launches a browser itself: every rung is a browser somebody runs.
 
 `library_info` reports the tiers in run order and, per variable, whether its value came from the environment, `.env` or the default — never the value.
@@ -160,7 +160,7 @@ PubMed's book records — National Academies reports, NTP monographs, StatPearls
 
 **Web pages.** `library_fetch(url, max_length?, start_index?, raw?)` is the standard MCP [fetch](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) tool, served from the library, for every link a review carries that is not a paper, label or trial record: a guideline, a product page, a news item. Its description tells the model to prefer it over a plain fetch or a browser tool, and says why: a page anyone fetched in the last 30 days comes from disk in milliseconds, and a page a plain fetch cannot get climbs the ladder in the same call. The answer is what `fetch` returns, so a model keeps its habits: `Contents of <url>:` and the page as Markdown with the page's title as the first heading, cut at `max_length` (default 5000) with the note `Content truncated. Call the fetch tool with a start_index of N to get more content`; `raw` gives the HTML; a `text/markdown` body comes back as is, any other non-HTML body with the `Content type … cannot be simplified to markdown` prefix; a failure is `Failed to fetch <url> - status code 404` or `Failed to fetch <url>: <reason>`. Two things happen underneath. The page is fetched up the same ladder as a paper — the plain fetch, then the browser rungs on a 403, 429 or bot wall, then Bright Data — so a page a plain fetch cannot reach is still read. And it is stored: `url/<2>/<4>/<16 hex of the URL's SHA-1>/url-<hash>-{meta.md, page.md, page.html|pdf|txt}`, the URL normalised first (host lowercased, fragment and tracking parameters dropped). The same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS` (1) — an audit that runs twice over one review reads its links once. A PDF comes back as Markdown (`fetch` would return its bytes as text). `title_only` returns `Contents of <url>:` and the title line alone — the page is still fetched and stored in full — so a link check costs a few tokens instead of a page. The log says which — `fetch example.com/… cached html … https://example.com/drugs/x` or `fetched 02.3s html retrieved_by=browser hetzner … <url>` (the ID column holds the URL without scheme, cut to a PubMed ID's width; the full URL ends the line); `library_info` counts the pages under `url`, a failed fetch as level `title`.
 
-**Funding and conflicts of interest.** Every status carries `coi`, the paper's conflict-of-interest statement, and `funding`, who paid. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `library_read(id, section: "Declarations")` — the JATS section, or on a PDF- or HTML-sourced text the funding, conflict and acknowledgement sections assembled. In `-meta.md` they are the lines `coi:` (a JSON string) and `funding:` (a JSON array, one item per agency or statement).
+**Funding and conflicts of interest.** Every entry's meta keeps `coi`, the paper's conflict-of-interest statement, and `funding`, who paid; the status answer leaves them out (hundreds of characters on every add), `library_read(id, section: "Declarations")` returns them. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `library_read(id, section: "Declarations")` — the JATS section, or on a PDF- or HTML-sourced text the funding, conflict and acknowledgement sections assembled. In `-meta.md` they are the lines `coi:` (a JSON string) and `funding:` (a JSON array, one item per agency or statement).
 
 **Symbols a PDF font hides.** Some publishers' symbol fonts carry no Unicode for their glyphs, so text extraction gives a control character where the PDF shows a minus, ±, ≤ or β. The library restores the symbol from the glyph's name in the font — the names of the Adobe Pi fonts (`H11350` is ≥ everywhere) and of the Advent symbol font most Elsevier and Wiley PDFs use — each name checked against the glyph's outline in the library's PDFs; a name it does not know is left as it is and counted: the status's `glyphs_unresolved` is the number left in the full text (162 of 1,119 PDF-derived entries have some, down from 277), and `library_verify` returns the same count for the text it judged. A `fail` on a paper with `glyphs_unresolved` above 0 says "check the symbol", not "the paper does not state it". Words a PDF breaks at a line end ("differ- ence") are joined back when the paper uses the word elsewhere or no compound is known at that break; a hyphen the paper uses elsewhere ("time-dependent") and a suspended one ("dose- and time-dependent") are kept.