fh-library 0.2.5 → 0.2.9

package/CHANGELOG.md

@@ -1,5 +1,33 @@
 # FH Library — Change Log
 
+### v0.2.9 — 2026-11-08
+
+* TUI v80: a `search` row — cached registry searches, hits and misses from the servers' counters; the decision-model row is labelled `decider`; `/status` carries `searches`.
+* ClinicalTrials tool descriptions and server instructions one statement each.
+
+### v0.2.8 — 2026-11-08
+
+* **`library_clinicaltrials_get_study_record(nct)` and `library_clinicaltrials_get_study_results(nct, max_chars?)`:** thin answers over the stored `nct:` entry (added first when missing) — the review's detail of one trial, and its posted results as Markdown with the outcome titles for `library_read`. With the search, the three tools map one to one onto what the AI4L models used of the ClinicalTrials.gov MCP server (#28).
+* `library_verify` cache key folds the claim's typography (#33): case, whitespace and non-breaking spaces, curly quotes, en/em dashes, "+/-" for "±", a trailing period — never a word. The claim is judged and stored as written.
+* `library_verify`: a claim that opens with an odds ratio ("OR 2.15 (95% CI 1.06 to 4.34) for …") is a statement, not a fragment starting with the conjunction "or" (seen in an AI4L audit, pmid:25358826).
+* **Smaller answers (#32):** every tool answers compact JSON (no indentation); `library_add`, `library_status`, `library_read` and `library_clinicaltrials_search_studies` leave out null values, empty lists and the entry's paths and file names (`entry_dir`, `library_dir`, `files`). `library_verify` keeps its nulls (`score`, `reason` null carry meaning). A status answer is about 30% smaller, a 20-row trial search about 25%.
+* **`library_fetch(…, title_only)`:** the title line alone (`Contents of <url>:` and `# Title`); the page is fetched and stored in full as usual. For verifying a link at a few tokens instead of a page (#32).
+* **`library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?)`:** a ClinicalTrials.gov search (API v2, no key) — total count and rows as arrays in `columns` order (nct, title, status with why stopped, phase, enrollment, start, primary_completion, primary_outcome, conditions, interventions, has_results — the names once, not per row); `status` ongoing / completed / all; `sort` relevance / newest / largest / completion. Cached under `search/<2>/<4>/<key>.json` for 7 days. With `library_add("nct:…")` and `library_read` for the record and posted results, this covers what the AI4L models asked the ClinicalTrials.gov MCP server for (#28). `library_info`'s `cache` and the `.servers/` files count `search` hits and misses.
+
+### v0.2.7 — 2026-11-08
+
+* **Verdicts are cached** (#26): `library_verify` keeps each judgement as `verify/<2>/<4>/<key>.json` — the file system as the database, laid out like `url/` — and answers the same claim on the same stored text from disk for `FH_LIBRARY_REFRESH_AFTER_DAYS`; a new full text, model or threshold is judged afresh, errors are never kept. The result is unchanged; the log line says `cached`. One atomic file per judgement: several servers on one library need no lock.
+* **Each server publishes its cache counters:** hits and misses of `library_verify` and `library_fetch` since it started, in memory, written to `<library>/.servers/<host>-<pid>.json` on every change and every 5 min as a heartbeat, removed on exit, swept by the next starting server when older than 20 min; `library_info` reports them as `cache`. The server also logs a `startup` line.
+* TUI v79: a `verify` row — stored verdicts ("queries cached"), then hits and misses summed over the servers alive on the library, read from `.servers/`, not from the log; `/status` carries `verdicts`. Resolves `.env` and `library/` from the caller's working directory, as the server does (#31) — `npm run --prefix <repo> tui` from a project root shows that project's library.
+
+### v0.2.6 — 2026-11-08
+
+* **`library_fetch(url, max_length?, start_index?, raw?)`:** the standard MCP `fetch` tool, served from the library. Same answer as the reference server — `Contents of <url>:`, the page as Markdown with its title as the first heading, the truncation note, the raw-content prefix, the `Failed to fetch …` errors — so a model keeps its fetch habits. Underneath, the page is fetched up the download ladder (plain fetch, Playwright and Chrome rungs, Bright Data) when a plain fetch is refused, and stored under `url/`: the same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS`. A PDF comes back as Markdown. `library_info` counts web pages as type `url`; the log line says `cached` or `fetched`.
+* **Script shells climb the ladder:** a page answered 200 whose converted text is under 100 words with a loading/JavaScript phrase (or under 10 words at all) is treated like a wall page — the browser rungs, then Bright Data; a browser rung with so little text waits for the network to go quiet and the text to grow (at most 8 s) before reading the page. When no rung renders more, the shell is served, as fetch would (lifeextension.com's search is such a page: nothing renders it).
+* **Bright Data's 407 is reported:** a token's IP allowlist refusing this host used to surface as a bare "fetch failed" (Node's fetch treats a 407 as a network error); the unlocker now requests over node:https and the log alerts `!! brightdata: unlocker HTTP 407: You are not allowed to access our API via this IP (…)`.
+* TUI v74: a `url` row (web pages, failed fetches); fetch log lines coloured — `cached` green, rung outcomes green/red; `/status` counts `fetches` and `cached`; `(u)pdate` also rescans the library.
+* README: `library_version` returns the package "version", not "library_version" (one more word the prefix rename caught).
+
 ### v0.2.5 — 2026-11-08
 
 * **Tool descriptions** (issue #30): `library_info` names its key `verify`, not `library_verify`; `library_status` answers for one ID of any kind, not "one paper ID"; `library_verify`'s `claim` asks for a complete statement built from the review's words, not a verbatim fragment.

package/README.md

@@ -1,4 +1,4 @@
-![Version 0.1.3](https://img.shields.io/badge/Version-0.2.5-green.svg)
+![Version 0.1.3](https://img.shields.io/badge/Version-0.2.9-green.svg)
 ![License MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
 [![Forever Healthy](https://img.shields.io/badge/(c)_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
 [![npm](https://img.shields.io/badge/npm-fh--library-cb3837.svg)](https://www.npmjs.com/package/fh-library)
@@ -20,13 +20,19 @@ A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a
 
 * `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `coi` and `funding` (PubMed's record, else the paper's own text), `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — files, failed tiers in `attempts`). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `FH_LIBRARY_REFRESH_AFTER_DAYS` gets the record re-read for new notices. One ID per call.
 * `library_status(id)` → read-only check of presence and level; never fetches
-* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
+* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), this server's cache hits and misses since it started (`cache`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
 * `library_verify(claim, id)` → is one finding of a review stated by one paper it cites? `claim` is the finding alone, verbatim, without its citations; `id` one paper (`pmid:`, or any `doi:`), drug label or trial record the review cites for it — a finding cited to three papers is three calls with the same claim; a claim that names a source or carries citations is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters the model's context. `verdict` **pass** (`score` the probability), **fail** (the full text does not state it; or only the title is in the library — `reason` says so), **unverifiable** (only the abstract is in the library and does not state it — `reason`) or **error**; `basis` full_text, abstract or title. See "Verifying claims" below.
 * `library_read(id, section?, query?, max_chars?, max_hits?)` → one small piece of a stored paper's Markdown, never the whole paper: without `section`/`query` the **outline** (headings with word counts, and whether the full text or only the abstract is stored; a heading the paper uses several times is listed as `Participants [3/12]`); `section` → that section's **own text** (heading matched as case-insensitive substring, capped at `max_chars`, default 6000, with `returned_words` against `words` when cut) and, for a parent heading, its `subsections` with word counts to pick from — a duplicated heading named bare lists the candidates instead; `query` — two or three distinctive terms the paper itself would use (a measure and a unit, an outcome and a group, a compound name), not the claim — → **passages** holding every term, each a window of at most 1000 chars around the matches (`clipped: true` when cut; a table row with its header row), with its section, best first (at most `max_hits`, default 5); matching forgives case, accents, hyphens, Markdown markup, plural and -ed/-ing forms; when no passage holds every term the best **partial** matches come back (`partial: true`, the `matched` terms per passage, the rare term first), and `missing_terms` names the terms the text never uses in any form — those are the ones to rephrase. `section` and `query` together search inside that section. `section: "Abstract"` on a full text without one answers from the stored abstract. Every full text converted from JATS XML (PMC, Europe PMC, Elsevier) has a **Declarations** section — funding, conflicts of interest, author notes, gathered from wherever the XML keeps them — and on a text without one `section: "Declarations"` assembles the sections headed funding, conflicts, acknowledgements, ethics, contributions, so it answers who paid and who declares what on every source alike; a paper with neither states nothing. Outline first, then pick. A paper not in the library is added first, as `library_add` would (its lines in the log), then library_read; the answer carries `added: true`. Reads may run in parallel.
-* `library_version()` → the running server's package name, library_version, host and library directory
+* `library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?)` → a ClinicalTrials.gov search (see "Trial search"): the total count and compact rows, cached for a week
+* `library_clinicaltrials_get_study_record(nct)` → one trial as a review needs it (title, status, phase, design, enrollment, dates, sponsor, conditions, interventions, primary outcomes, eligibility, sites, result publications, has_results, brief summary), from the stored record
+* `library_clinicaltrials_get_study_results(nct, max_chars?)` → a trial's posted results as Markdown, capped, with the outcome measure titles for `library_read`
+* `library_fetch(url, max_length?, start_index?, raw?, title_only?)` → a web page as Markdown, exactly as the standard MCP `fetch` tool returns it (see "Web pages"): the one tool whose document enters the model's context
+* `library_version()` → the running server's package name, version, host and library directory
 
 IDs follow the `type:value` form: `pmid:`, `doi:`, `dailymed:`, `nct:`; bare digits count as `pmid:`, a bare NCT number as `nct:`.
 
+**Context budget.** Every answer is sized to what a model does with it in an evidence review, because every character comes out of the model's context. Answers are compact JSON; a status answer leaves out what carries nothing (null values, empty lists, the entry's paths and file names), while `library_verify` keeps its nulls, because a null `score` or `reason` means something there. A link is verified with `library_fetch(url, title_only: true)`, one line instead of a page; the page is stored in full for a later read. The ClinicalTrials.gov tools follow the same rule: `library_clinicaltrials_search_studies` returns rows as arrays under one `columns` list, so the field names are sent once, not per trial, and carries only what a review's Emerging Research item states about a trial — the rest of a record is one `library_clinicaltrials_get_study_record` call away, and the posted results one `library_clinicaltrials_get_study_results` call, capped by `max_chars`; measured against the ClinicalTrials.gov MCP server on the same calls, the search matches its compact index in size while carrying the primary outcome and interventions it lacks, the record is an eighth of its size, the results half. Searches and verdicts are cached, so a repeated audit pays no second answer. The PubMed tools to come (search, fetch) are built the same way: the parameters the live audits actually use, rows shaped for the review's sections on evidence, benefits, risks and interactions, cached.
+
 ### Retrieval tiers
 
 | Tier | Does | Needs |
@@ -95,6 +101,8 @@ Every full text is checked before it counts: title present, methods and results
 
 **The call.** `claim` is one finding, verbatim from the review, with markdown links reduced to their text and its citations removed: `IPSS fell 2.3 points more than placebo in men with ED`. Section labels such as `**Magnitude:**` and list bullets may stay. `id` is one paper the review cites for that finding, as `pmid:<PMID>`, already in the library (`library_add` first). One claim, one paper per call: a finding cited to three papers is three calls with the same claim text, and each answer stands on its own — a source that does not carry the finding is named by its own row, so it can be dropped without touching the others. The server never aggregates over papers. A sentence that joins two findings, each with its own citation, is two claims: the text before each citation group, without the group. A sentence without a citation is not a claim.
 
+**Verdicts are kept.** The answer to one claim against one stored text is written to `verify/<2>/<4>/<key>.json` — one file per judgement, laid out like `url/` — and served from there for `FH_LIBRARY_REFRESH_AFTER_DAYS` (30): an audit that runs twice over the same review pays the model once. The key hashes the claim with its typography folded (case, whitespace, curly quotes, dashes, "±" forms, a trailing period — never a word: a reworded claim is another statement), the source ID, the model, the threshold and the stored text, so a paper that gains its full text, a changed model or a changed threshold is judged afresh; errors are never kept. The result JSON is the same either way; the log line carries `cached`. Several servers may share the library: a file appears whole or not at all, no lock.
+
 **How to build the claim.** The decision model answers exactly the statement it is given, so the statement must be one the paper could make:
 
 * **A complete statement:** subject, intervention, outcome, number. Splitting a sentence at its citations leaves fragments ("and dementia", "or silodosin", "across long-acting PDE5 inhibitors, 0.40 percentage points lower"); restore the subject from the sentence ("HbA1c was 0.40 percentage points lower with long-acting PDE5 inhibitors than with control"). Resolve "it", "both", "the drug".
@@ -139,6 +147,10 @@ PubMed's book records — National Academies reports, NTP monographs, StatPearls
 
 **Clinical trial records.** `library_add("nct:05173896")` stores the current record of one trial from [ClinicalTrials.gov](https://clinicaltrials.gov/): the ID is the eight digits of the NCT number in the review's link (`…/study/NCT05173896`); `nct:NCT05173896` and a bare `NCT05173896` are accepted and read the same. The API's JSON is converted to Markdown, one section per module: identification with secondary IDs such as EudraCT, status with every date and its actual/estimated type, sponsor and collaborators, description, conditions, design (phase, allocation, masking, enrollment with its type), arms and interventions with doses, primary and secondary outcomes with time frames, eligibility, locations as a country list, references; and, when results are posted, the participant flow, baseline characteristics, every outcome measure with its group values and analyses, and the adverse events, all as tables. Values are written in plain words, "Phase 2", "Active, not recruiting", "100 (estimated)", so a figure the review states can be found as stated. The status carries `nctid`, `publications` (the results publications the record lists, as `pmid:…`, so whether a trial has been published is one status away), the brief title, the lead sponsor as `first_author`, the first-posted year as `year`, the last-update-posted date as `version`, and `publication_types` `["Trial Registration"]`. `library_read` and `library_verify` work on a record as on a paper. An entry older than `FH_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when the record was updated since; a status the review states that has moved on then fails, which is right. Entries live under `nct/<digits 1–2>/<digits 1–4>/<digits>/` with the JSON and the Markdown as `nct-<digits>-fulltext.json` and `-fulltext.md`; the log line reads `library_add nct:05173896 ctgov fetched level=full_text updated 2024-12-09 active, not recruiting ETLAS2`.
 
+**Trial search.** `library_clinicaltrials_search_studies(query?, intervention?, condition?, status?, sort?, max_results?)` asks the ClinicalTrials.gov API (v2, no key) what a review needs from the registry: the trials of an intervention, the ongoing ones (`status: "ongoing"` is recruiting, not yet recruiting, active not recruiting, enrolling by invitation), a free-text or condition search, sorted by relevance, newest, largest or completion date, up to 100 rows. The rows are arrays in the order of `columns` — the field names once, not per row — shaped for the review's Emerging Research item (what AI4L's prompt asks about an ongoing trial; the link is `https://clinicaltrials.gov/study/<NCT ID>`): NCT ID, title, status with why a trial stopped, phase, enrollment as "100 (estimated)", start and primary completion dates (whether "ongoing" still holds), the primary outcome with its time frame, conditions, interventions, `has_results`; the full detail of one row is `library_clinicaltrials_get_study_record`; `total` is the registry's count, for a "no trials found" statement. The same search is answered from `search/<2>/<4>/<key>.json` for 7 days — a registry listing a week old is current enough for a review, and an audit that runs again reads from disk. Two thin tools answer from the stored record of any row (added first when missing): `library_clinicaltrials_get_study_record(nct)` is the review's detail of one trial — official title, design, enrollment, dates, eligibility, sites, result publications as `pmid:` IDs, the brief summary — and `library_clinicaltrials_get_study_results(nct, max_chars?)` is the posted results part of the stored Markdown (participant flow, baseline, outcome measures, adverse events), with the outcome titles listed so `library_read("nct:…", section)` can return one measure whole. Together with the search they cover what the AI4L models asked the ClinicalTrials.gov MCP server for.
+
+**Web pages.** `library_fetch(url, max_length?, start_index?, raw?)` is the standard MCP [fetch](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) tool, served from the library, for every link a review carries that is not a paper, label or trial record: a guideline, a product page, a news item. Its description tells the model to prefer it over a plain fetch or a browser tool, and says why: a page anyone fetched in the last 30 days comes from disk in milliseconds, and a page a plain fetch cannot get climbs the ladder in the same call. The answer is what `fetch` returns, so a model keeps its habits: `Contents of <url>:` and the page as Markdown with the page's title as the first heading, cut at `max_length` (default 5000) with the note `Content truncated. Call the fetch tool with a start_index of N to get more content`; `raw` gives the HTML; a `text/markdown` body comes back as is, any other non-HTML body with the `Content type … cannot be simplified to markdown` prefix; a failure is `Failed to fetch <url> - status code 404` or `Failed to fetch <url>: <reason>`. Two things happen underneath. The page is fetched up the same ladder as a paper — the plain fetch, then the browser rungs on a 403, 429 or bot wall, then Bright Data — so a page a plain fetch cannot reach is still read. And it is stored: `url/<2>/<4>/<16 hex of the URL's SHA-1>/url-<hash>-{meta.md, page.md, page.html|pdf|txt}`, the URL normalised first (host lowercased, fragment and tracking parameters dropped). The same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS` (1) — an audit that runs twice over one review reads its links once. A PDF comes back as Markdown (`fetch` would return its bytes as text). `title_only` returns `Contents of <url>:` and the title line alone — the page is still fetched and stored in full — so a link check costs a few tokens instead of a page. The log says which — `fetch example.com/… cached html … https://example.com/drugs/x` or `fetched 02.3s html retrieved_by=browser hetzner … <url>` (the ID column holds the URL without scheme, cut to a PubMed ID's width; the full URL ends the line); `library_info` counts the pages under `url`, a failed fetch as level `title`.
+
 **Funding and conflicts of interest.** Every status carries `coi`, the paper's conflict-of-interest statement, and `funding`, who paid. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `library_read(id, section: "Declarations")` — the JATS section, or on a PDF- or HTML-sourced text the funding, conflict and acknowledgement sections assembled. In `-meta.md` they are the lines `coi:` (a JSON string) and `funding:` (a JSON array, one item per agency or statement).
 
 **Symbols a PDF font hides.** Some publishers' symbol fonts carry no Unicode for their glyphs, so text extraction gives a control character where the PDF shows a minus, ±, ≤ or β. The library restores the symbol from the glyph's name in the font — the names of the Adobe Pi fonts (`H11350` is ≥ everywhere) and of the Advent symbol font most Elsevier and Wiley PDFs use — each name checked against the glyph's outline in the library's PDFs; a name it does not know is left as it is and counted: the status's `glyphs_unresolved` is the number left in the full text (162 of 1,119 PDF-derived entries have some, down from 277), and `library_verify` returns the same count for the text it judged. A `fail` on a paper with `glyphs_unresolved` above 0 says "check the symbol", not "the paper does not state it". Words a PDF breaks at a line end ("differ- ence") are joined back when the paper uses the word elsewhere or no compound is known at that break; a hyphen the paper uses elsewhere ("time-dependent") and a suspended one ("dose- and time-dependent") are kept.