fh-library 0.2.16 → 0.2.17

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md CHANGED
@@ -1,137 +1,5 @@
1
1
  # FH Library — Change Log
2
2
 
3
- ### v0.2.12 — 2026-11-08
3
+ ### v0.2.17 — 2026-11-08
4
4
 
5
- * **`library_pubmed_search_articles(query, max_results?, sort?, years?)`:** a PubMed search in PubMed's own syntax — esearch for the total and the IDs, one esummary for the rows (PMID, title, first author, year, journal, publication types, PMCID, library level), cached under `search/`. With the batch fetch and `library_read` this covers what the AI4L models used of the PubMed MCP server (#27).
6
- * **`FH_LIBRARY_SEARCH_CACHE_DAYS` (7):** days a PubMed or registry search is cached; was a constant.
7
- * **An ID no service knows is remembered** for `FH_LIBRARY_RETRY_AFTER_DAYS` (1): a PMID PubMed has no record for, a DOI neither Crossref nor doi.org resolves, a set ID without a label, an NCT ID without a study. A marker (`<entry>-missing.md`, date and reason, no meta) in the entry's directory; `library_add`, `library_status`, `library_read`, `library_verify` and the batch tool answer from it ("pmid not found (checked 2026-11-08; asked again after 1 day)"), the service is asked again after the cooldown, a real entry replaces the marker. Network errors are never remembered. The same cooldown a failed page or a transient tier failure already had.
8
- * **`library_pubmed_fetch_articles(pmids)`:** title, first author, year, publication types, retracted and level for up to 200 PMIDs in one call, rows in `columns` order (~120 characters a paper against the PubMed MCP server's ~2,500). Stored PMIDs from disk; the rest in one efetch, stored at abstract level as `library_add`'s first stage, so the cascade runs later as on an untried entry. The first of the PubMed tools (#27); `library_info`'s `cache` and `.servers/` count `pubmed` hits and misses per PMID.
9
-
10
- ### v0.2.9 — 2026-11-08
11
-
12
- * TUI v82: the `url` row shows library_fetch hits and misses from the servers' counters; `/status` carries `fetches`. Counts right-aligned in a 5-character column.
13
- * TUI v80: a `search` row — cached registry searches, hits and misses from the servers' counters; the decision-model row is labelled `decider`; `/status` carries `searches`.
14
- * ClinicalTrials tool descriptions and server instructions one statement each.
15
-
16
- ### v0.2.8 — 2026-11-08
17
-
18
- * **`library_clinicaltrials_get_study_record(nct)` and `library_clinicaltrials_get_study_results(nct, max_chars?)`:** thin answers over the stored `nct:` entry (added first when missing) — the review's detail of one trial, and its posted results as Markdown with the outcome titles for `library_read`. With the search, the three tools map one to one onto what the AI4L models used of the ClinicalTrials.gov MCP server (#28).
19
- * `library_verify` cache key folds the claim's typography (#33): case, whitespace and non-breaking spaces, curly quotes, en/em dashes, "+/-" for "±", a trailing period — never a word. The claim is judged and stored as written.
20
- * `library_verify`: a claim that opens with an odds ratio ("OR 2.15 (95% CI 1.06 to 4.34) for …") is a statement, not a fragment starting with the conjunction "or" (seen in an AI4L audit, pmid:25358826).
21
- * **Smaller answers (#32):** every tool answers compact JSON (no indentation); `library_add`, `library_status`, `library_read` and `library_clinicaltrials_search_studies` leave out null values, empty lists and the entry's paths and file names (`entry_dir`, `library_dir`, `files`). `library_verify` keeps its nulls (`score`, `reason` null carry meaning). A status answer is about 30% smaller, a 20-row trial search about 25%.
22
- * **`library_fetch(…, title_only)`:** the title line alone (`Contents of <url>:` and `# Title`); the page is fetched and stored in full as usual. For verifying a link at a few tokens instead of a page (#32).
23
- * **`library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?)`:** a ClinicalTrials.gov search (API v2, no key) — total count and rows as arrays in `columns` order (nct, title, status with why stopped, phase, enrollment, start, primary_completion, primary_outcome, conditions, interventions, has_results — the names once, not per row); `status` ongoing / completed / all; `sort` relevance / newest / largest / completion. Cached under `search/<2>/<4>/<key>.json` for 7 days. With `library_add("nct:…")` and `library_read` for the record and posted results, this covers what the AI4L models asked the ClinicalTrials.gov MCP server for (#28). `library_info`'s `cache` and the `.servers/` files count `search` hits and misses.
24
-
25
- ### v0.2.7 — 2026-11-08
26
-
27
- * **Verdicts are cached** (#26): `library_verify` keeps each judgement as `verify/<2>/<4>/<key>.json` — the file system as the database, laid out like `url/` — and answers the same claim on the same stored text from disk for `FH_LIBRARY_REFRESH_AFTER_DAYS`; a new full text, model or threshold is judged afresh, errors are never kept. The result is unchanged; the log line says `cached`. One atomic file per judgement: several servers on one library need no lock.
28
- * **Each server publishes its cache counters:** hits and misses of `library_verify` and `library_fetch` since it started, in memory, written to `<library>/.servers/<host>-<pid>.json` on every change and every 5 min as a heartbeat, removed on exit, swept by the next starting server when older than 20 min; `library_info` reports them as `cache`. The server also logs a `startup` line.
29
- * TUI v79: a `verify` row — stored verdicts ("queries cached"), then hits and misses summed over the servers alive on the library, read from `.servers/`, not from the log; `/status` carries `verdicts`. Resolves `.env` and `library/` from the caller's working directory, as the server does (#31) — `npm run --prefix <repo> tui` from a project root shows that project's library.
30
-
31
- ### v0.2.6 — 2026-11-08
32
-
33
- * **`library_fetch(url, max_length?, start_index?, raw?)`:** the standard MCP `fetch` tool, served from the library. Same answer as the reference server — `Contents of <url>:`, the page as Markdown with its title as the first heading, the truncation note, the raw-content prefix, the `Failed to fetch …` errors — so a model keeps its fetch habits. Underneath, the page is fetched up the download ladder (plain fetch, Playwright and Chrome rungs, Bright Data) when a plain fetch is refused, and stored under `url/`: the same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS`. A PDF comes back as Markdown. `library_info` counts web pages as type `url`; the log line says `cached` or `fetched`.
34
- * **Script shells climb the ladder:** a page answered 200 whose converted text is under 100 words with a loading/JavaScript phrase (or under 10 words at all) is treated like a wall page — the browser rungs, then Bright Data; a browser rung with so little text waits for the network to go quiet and the text to grow (at most 8 s) before reading the page. When no rung renders more, the shell is served, as fetch would (lifeextension.com's search is such a page: nothing renders it).
35
- * **Bright Data's 407 is reported:** a token's IP allowlist refusing this host used to surface as a bare "fetch failed" (Node's fetch treats a 407 as a network error); the unlocker now requests over node:https and the log alerts `!! brightdata: unlocker HTTP 407: You are not allowed to access our API via this IP (…)`.
36
- * TUI v74: a `url` row (web pages, failed fetches); fetch log lines coloured — `cached` green, rung outcomes green/red; `/status` counts `fetches` and `cached`; `(u)pdate` also rescans the library.
37
- * README: `library_version` returns the package "version", not "library_version" (one more word the prefix rename caught).
38
-
39
- ### v0.2.5 — 2026-11-08
40
-
41
- * **Tool descriptions** (issue #30): `library_info` names its key `verify`, not `library_verify`; `library_status` answers for one ID of any kind, not "one paper ID"; `library_verify`'s `claim` asks for a complete statement built from the review's words, not a verbatim fragment.
42
-
43
- ### v0.2.4 — 2026-11-08
44
-
45
- * **Fix: `library_status` on a stored entry and `library_info` failed** with "library_status is not defined" / "library_info is not defined" in 0.2.2 and 0.2.3: the rename that restored the `library_` prefix also renamed two local variables. `library_version`'s log line printed "undefined" for the same reason. The Playwright download rung lacked a `readFileSync` import. Prose in the server instructions and READMEs that had "read" or "status" turned into tool names is plain English again.
46
-
47
- ### v0.2.3 — 2026-11-08
48
-
49
- * **Every key is an `FH_LIBRARY_*` variable:** `FH_LIBRARY_NCBI_API_KEY`, `FH_LIBRARY_BRIGHTDATA_API_TOKEN`, `FH_LIBRARY_BRIGHTDATA_ZONE` replace the bare `NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE` the library used to share with other MCP servers. Hard cut: a deployment that still runs those servers keeps the bare lines for them and adds the prefixed ones for the library. The direction: fetching and PubMed move into the library, the separate servers go, and the double configuration with them.
50
-
51
- ### v0.2.2 — 2026-11-08
52
-
53
- * **Tool names carry the `library_` prefix again**, all of them: `library_add`, `library_status`, `library_read`, `library_verify`, `library_info`, `library_version` (0.2.0 and 0.2.1 had the bare names, and `version`). A model that sees many servers' tools should never have to guess what `add` or `read` belongs to. Client permissions and prompts name them with the prefix.
54
- * **Browser rungs by kind:** `FH_LIBRARY_PLAYWRIGHT_1`, `_2`, … (Playwright servers, port 8932) and `FH_LIBRARY_CHROME_1`, `_2`, … (a Chrome with `--remote-debugging-port`, port 9222) replace `FH_LIBRARY_BROWSER_1..3`. Values are hostnames (host or host:port); the kind is the variable's name, the protocol the library's. `FH_LIBRARY_PLAYWRIGHT_HEADLESS` (was `_BROWSER_HEADLESS`) belongs to the Playwright family; a Chrome rung is whatever it is. Order is fixed: Playwright servers, then Chrome rungs. The in-process "local" Chromium rung is gone — the library never launches a browser. `library_info` splits `download` into `playwright: {version, servers}` and `chrome`. Hard cut: every `.env` must move its browser lines.
55
-
56
- ### v0.2.0 — 2026-11-08
57
-
58
- * **Renamed to FH Library.** The npm packages are `fh-library` (public) and `fh-library-dev` (dev), the MCP Registry entry `io.github.forever-healthy/fh-library`, the repos `forever-healthy/fh-library` and `fh-library-dev`; the server announces itself as `fh-library`, and client configs use the keys `fh-library`, `fh-library-dev`, `fh-library-local` (the `d-` prefix is for MCP servers Forever Healthy did not write). The `ai4l-library` packages are deprecated and point here.
59
- * **Tools lose the `library_` prefix:** `add`, `status`, `read`, `verify`, `info`, `version` (was `get_version`). The server name namespaces them in every client.
60
- * **Environment variables are `FH_LIBRARY_*`** (was `AI4L_LIBRARY_*`), hard cut: every `.env` must follow. Keys other servers share (`NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE`) are unchanged. The browser helper is `fh-chrome` (service, window class, `~/.fh-chrome` profile), the browser server container `fh-browser`.
61
- * Entries in the changelog below this line use the old names; they describe the versions that carried them.
62
-
63
- ### v0.1.12 — 2026-11-07
64
-
65
- * Server instructions after the AI4L review (#21): calls may run in parallel — add, read and verify — and two calls on the same missing record share one add (said once, under Hints, instead of "reads may run in parallel"); `corrections` is named as the status's field; the stale `config.json` clause gives way to the proxy URL template from the configuration.
66
-
67
- ### v0.1.11 — 2026-11-07
68
-
69
- * `library_verify`: the verdict `na` is `unverifiable` again, now only for a paper whose abstract is stored and does not state the claim (the full text may). A title-only record — no abstract text, no full text — is a `fail`: nothing to read can back a claim; no model call, `score` null. Both carry `reason`, one fixed sentence each; `reason` is null otherwise. Audit tables map `unverifiable` where they mapped `na`. The log line pads the verdict to 12 characters.
70
- * A record with nothing to read is not citable: `citable` false with `comment` "no abstract and no full text: nothing to verify" (retraction still comes first). `library_info.not_citable` counts them. A full text dropped into the inbox makes the record citable again.
71
- * The server instructions rewritten as one short bullet document (3.9 K chars instead of 11 K): IDs, tools, rules, verify, read, status fields, sources. New guidance from the dev fleet run: no add or status before read or verify; one source, only what that source reports (a two-intervention finding cited to two papers is two claims; a statement about the literature becomes the figure this paper gives); what to do after each verdict (`fail` → read with a query, then reword or drop; `na` → "abstract only", no retry); `partial` → replace the `missing_terms`, keep the rest; a wrong ID is not retried. The `library_read` and `library_verify` descriptions shrunk to what the tool and its parameters do; the guidance lives in the instructions once.
72
-
73
- ### v0.1.10 — 2026-11-07
74
-
75
- * `library_read` and `library_verify` on a paper not in the library add it first, as `library_add` would (its lines in the log), then read or judge it; the read answer carries `added: true`, and a paper the add cannot build is the error. Parallel reads of one missing paper share a single add.
76
- * The `query` parameter says how to search: two or three distinctive terms the paper itself would use — a measure and a unit, an outcome and a group, a compound name — not the claim. The creators were sending 5–8 keyword dumps, which no paragraph holds whole.
77
-
78
- ### v0.1.9 — 2026-11-07
79
-
80
- * `library_info` shows the Playwright version the package pins as `download.playwright`; the fleet TUI's `/status` carries the same as `playwright`. A browser server of another version is logged in words as an alert — `!! browser hetzner: Playwright version mismatch — server ws://hetzner:8932 must run playwright-core 1.65.0` — instead of the raw `428 Precondition Required` (#19).
81
-
82
- ### v0.1.8 — 2026-11-07
83
-
84
- * `playwright-core` pinned to 1.65.0 (was `^1.58.1`), and `browser/Dockerfile` moved to the same version. A Playwright server refuses a client of another version with `428 Precondition Required`; a fresh install of the range had drifted to 1.65.0 while the browser servers ran 1.58.1, so the browser rungs never ran on the evipedia fleet. Package and servers must carry the same exact version.
85
-
86
- ### v0.1.7 — 2026-11-07
87
-
88
- `library_read`, from watching the creators use it (#18):
89
-
90
- * `query` answers windows, not paragraphs: each passage is at most 1000 chars around the matches (`clipped: true` when cut; a table row comes with its header row). A PDF-sourced section used to come back whole — up to 468 KB in one passage.
91
- * When no passage holds every term, the best partial matches come back with `partial: true` and the `matched` terms per passage, the rare term ranked first; `missing_terms` names the terms the text never uses in any form.
92
- * Matching forgives case, accents, hyphen/slash, Markdown markup (`t~max~` is tmax), plural and -ed/-ing/-ation forms.
93
- * `section` and `query` together search inside that section.
94
- * A parent heading answers its own text plus its `subsections` (heading, words) instead of the subtree cut at `max_chars`; a cut section reports `returned_words` against `words`.
95
- * A heading the text uses several times is listed as `Participants [3/12]` and named that way; the bare name lists the candidates.
96
- * `section "Abstract"` on a full text without one answers from the stored abstract; `section "Declarations"` on a text without one assembles the funding / conflict / acknowledgement / ethics sections.
97
- * PDF → Markdown: paragraphs from the page layout (extra leading, a first-line indent, a bullet, a change of type size, a short line ending a sentence), across columns and pages — not only from blank lines, which journal PDFs rarely have. Cochrane's section names are recognised as headings. Stored PDF-sourced texts keep their old structure until re-added.
98
- * JATS: a blank inside `<sub>`/`<sup>` moves outside the markers (`t~max~ around`, not `t~max ~around`).
99
-
100
- ### v0.1.6 — 2026-11-07
101
-
102
- * No change to the server. The fleet TUI (dev repo, not in the package) pulls its checkout with each rescan (`AI4L_LIBRARY_TUI_PULL=1`) instead of on its own timer.
103
-
104
- ### v0.1.5 — 2026-11-07
105
-
106
- * `library.log`: a message starting with `!!` is an alert — a service refused the deployment itself (Bright Data's reason from its `x-brd-err-msg` header, e.g. a token not allowed from this IP; a verify model failure; an unknown tier name at startup).
107
- * Fixed: `AI4L_LIBRARY_DIR` set in `.env` was ignored (the library directory was resolved before `.env` was loaded); only the environment worked.
108
-
109
- ### v0.1.4 — 2026-11-07
110
-
111
- * `.env` is the whole configuration: every value an environment variable, the tier set and the defaults built in. `library/config.json` and `AI4L_LIBRARY_CONFIG` are gone, and with them `${NAME}` references and the per-tier `description`.
112
- * Renamed, no aliases — `CONTACT_EMAIL` → `AI4L_LIBRARY_CONTACT_EMAIL`; `ELSEVIER_API_KEY` → `AI4L_LIBRARY_ELSEVIER_API_KEY`; `WILEY_TDM_TOKEN` → `AI4L_LIBRARY_WILEY_TDM_TOKEN`; `CORE_API_KEY` → `AI4L_LIBRARY_CORE_API_KEY`; `PERPLEXITY_URL` / `PERPLEXITY_API_KEY` / `PERPLEXITY_MODEL` → `AI4L_LIBRARY_VERIFY_URL` / `_API_KEY` / `_MODEL`; config.json's `threshold`, `timeout_seconds`, `retry_after_days`, `retry_blocked`, `refresh_after_days`, `download.headless` → `AI4L_LIBRARY_VERIFY_THRESHOLD`, `AI4L_LIBRARY_VERIFY_TIMEOUT_SECS`, `AI4L_LIBRARY_RETRY_AFTER_DAYS`, `AI4L_LIBRARY_RETRY_BLOCKED`, `AI4L_LIBRARY_REFRESH_AFTER_DAYS`, `AI4L_LIBRARY_BROWSER_HEADLESS`. `NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE` and `AI4L_LIBRARY_BROWSER_n` unchanged.
113
- * `AI4L_LIBRARY_TIERS` is the whole ladder: the listed tiers in exactly that order, a tier not listed is off (an unlisted tier no longer runs after the listed ones).
114
- * One proxy tier, `proxy`, instead of url tiers: `AI4L_LIBRARY_PROXY_URL` (template with `{doi}` `{pmid}` `{pmcid}`) and `AI4L_LIBRARY_PROXY_HEADERS`; `PROXY_TOKEN` and the `my-proxy` example are gone.
115
- * `library_info`: `config` replaces `tiers_config` — the cascade's settings, problems with `AI4L_LIBRARY_TIERS`, and per variable where its value came from (environment, `.env`, default), never the value; `tiers` in run order without descriptions (the README describes them).
116
- * The container's working directory is `/library`; the configuration arrives as its environment.
117
-
118
- ### v0.1.3 — 2026-11-07
119
-
120
- * IDs: `pmid:` (bare digits count as one), `doi:` for papers PubMed does not index (a DOI PubMed indexes is stored under its PMID and copied under the DOI, so every tool takes either), `dailymed:<set ID>` drug labels, `nct:` trial records.
121
- * `library_add(id)`: full text through a configurable cascade (`config.json`: inbox, pmc, europepmc, tdm, unpaywall, openalex, core, url tiers) and a download ladder (plain fetch, browser rungs, Bright Data); every text checked (title, methods and results, no wall page) before it counts; failed tiers recorded in `attempts`, transient ones retried after `retry_after_days`; PubMed record re-read after `refresh_after_days` for correction notices and retractions (`citable`, `corrections`).
122
- * `library_read(id, section?, query?, max_chars?, max_hits?)`: the outline, one section, or the paragraphs matching a query — never the whole paper.
123
- * `library_verify(claim, id)`: is one finding stated by one source, judged on the stored text by the Perplexity Decisions API; `superseded` when a stored correction replaced the figure; `glyphs_unresolved` for symbols a PDF font gave no character for.
124
- * `library_info(ids?)`: the library's directory, tiers, download ladder, verify model health, entries by type and level.
125
- * PDF conversion: symbols restored from the font's glyph names (minus, ±, ≤, Greek letters), words broken at line ends joined back.
126
- * `library.log`: one line per action, host-prefixed.
127
- * Environment variables renamed `LIBRARY_*` → `AI4L_LIBRARY_*` (`AI4L_LIBRARY_DIR`, `AI4L_LIBRARY_CONFIG`, `AI4L_LIBRARY_ENV`, `AI4L_LIBRARY_TIERS`, `AI4L_LIBRARY_BROWSER_n`, `AI4L_LIBRARY_TUI_STATUS_SECS`); the old names are not read.
128
- * `.env` is also looked for in the library directory when neither `AI4L_LIBRARY_ENV` nor `./.env` names one.
129
- * Entry files are written to a temporary name and renamed into place (several servers may share one library); `library.log` rotates to `library.log.1` past 50 MB.
130
-
131
- ### v0.1.1 — 2026-10-26
132
-
133
- * `library_add(id)` and `library_status(id)` for `pmid:` IDs.
134
-
135
- ### v0.0.1 — 2026-10-26
136
-
137
- * Project skeleton.
5
+ * **`library_fetch` answers record URLs from the library:** a PubMed, ClinicalTrials.gov, DailyMed or doi.org link is answered from the library entry (a missing PubMed record is stored at abstract level in one request, no full-text cascade; a DOI PubMed indexes goes through its PMID) in fetch's format — the abstract record for PubMed, the trial record, the label, the full text or abstract for a DOI; `title_only` the title line. No script shell, no browser.
package/README.md CHANGED
@@ -1,4 +1,4 @@
1
- ![Version 0.1.3](https://img.shields.io/badge/Version-0.2.16-green.svg)
1
+ ![Version 0.1.3](https://img.shields.io/badge/Version-0.2.17-green.svg)
2
2
  ![License MIT](https://img.shields.io/badge/License-MIT-yellow.svg)
3
3
  [![Forever Healthy](https://img.shields.io/badge/(c)_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
4
4
  [![npm](https://img.shields.io/badge/npm-fh--library-cb3837.svg)](https://www.npmjs.com/package/fh-library)
@@ -7,7 +7,9 @@
7
7
 
8
8
  ![AI4L – AI for Practical Longevity](https://raw.githubusercontent.com/forever-healthy/fh-library/main/docs/AI4L-Header.png)
9
9
 
10
- **FH Library** - MCP server that builds a local library of the scientific papers an evidence review cites, part of [AI4L](https://github.com/forever-healthy/AI4L)
10
+ # FH Library MCP
11
+
12
+ MCP server that builds a local library of the scientific papers an evidence review cites, part of [AI4L](https://github.com/forever-healthy/AI4L)
11
13
 
12
14
  A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a local library of the scientific papers an evidence review cites. Given a paper ID (`type:value`, e.g. `pmid:30955454`), it fetches the paper's metadata, abstract and — where obtainable — full text through a configurable cascade of retrieval services (PubMed, PMC, Europe PMC, Unpaywall, OpenAlex, …), converts everything to Markdown, and stores it under a mounted `library/` directory. Nothing flows through the model's context: the server writes to disk itself and returns only a small status per call.
13
15
 
@@ -73,8 +75,9 @@ FH_LIBRARY_PROXY_HEADERS=Authorization: Bearer <token>
73
75
  ```
74
76
 
75
77
  * The URL is a template. Placeholders: `{doi}` (`10.1056/NEJMoa042274`), `{pmid}` (`15516696`), `{pmcid}` (`PMC6728809`, with its prefix). Each is URL-encoded when filled — the slash in a DOI becomes `%2F` — so put a placeholder in a path segment or a query value. A template may use several. The tier applies only to a paper for which every placeholder it uses is known; otherwise it is skipped as `no doi` (or `no pmcid`, …), a permanent reason. A template without any placeholder never runs (`template has no placeholder`).
78
+
76
79
  * The headers are optional request headers for credentials (`Authorization`, an API-key header, a cookie), as `Name: value; Name: value`.
77
- * Unset URL: no proxy tier. `FH_LIBRARY_TIERS` places `proxy` in the order like any built-in tier.
80
+
78
81
 
79
82
  **What the proxy tier expects back.** The tier sends one GET to the filled template, with a desktop Chrome User-Agent and the headers; redirects are followed. The service answers the paper as one of two formats, read from the response bytes, not the URL or the Content-Type:
80
83
 
@@ -151,6 +154,8 @@ PubMed's book records — National Academies reports, NTP monographs, StatPearls
151
154
 
152
155
  **PubMed in batches.** `library_pubmed_fetch_articles(pmids)` answers what an auditor asks of a list of PubMed IDs — does each resolve, what is its title, year and type, is it retracted, does the library hold its full text — as rows in `columns` order, about 120 characters per paper. A PMID already in the library is read from disk. The others go to PubMed in one efetch request (up to 200), and each record is stored exactly as `library_add` stores its first stage: the XML, the abstract Markdown, the meta with its correction and retraction notices, no retrieval attempts yet — so a later `library_add` or `library_read` runs the full-text cascade as on an untried entry, and the library fills at abstract level in batches instead of one paper at a time. An entry is checked again right before writing, so a concurrent `library_add` that already stored a full text is never overwritten. Level `not found` marks a PMID PubMed does not know, `invalid` a value that is not a PMID. `library_pubmed_search_articles(query, max_results?, sort?, years?)` is the search: the query goes to PubMed as written, field tags and all; esearch gives the total and the IDs, one esummary the rows — PMID, title, first author, year, journal, publication types (Journal Article left out), PMCID when a free full text exists, and the library's level when the paper is already stored, so the model knows what `library_read` can answer at once. Cached under `search/` like the trial search.
153
156
 
157
+ **Record URLs.** `library_fetch` answers a PubMed, ClinicalTrials.gov, DailyMed or doi.org link from the library instead of the web page, in fetch's format: the abstract record for a PubMed link, the trial record, the label, the full text or abstract for a DOI; `title_only` the title line. A missing PubMed record is stored at abstract level in one request, no full-text cascade (a DOI PubMed indexes goes through its PMID); a trial or label is added as `library_add` adds it. PubMed's own page is a script shell a browser must render; the record answers in half a second, and in milliseconds once stored.
158
+
154
159
  **Trial search.** `library_clinicaltrials_search_studies(query?, intervention?, condition?, status?, sort?, max_results?)` asks the ClinicalTrials.gov API (v2, no key) what a review needs from the registry: the trials of an intervention, the ongoing ones (`status: "ongoing"` is recruiting, not yet recruiting, active not recruiting, enrolling by invitation), a free-text or condition search, sorted by relevance, newest, largest or completion date, up to 100 rows. The rows are arrays in the order of `columns` — the field names once, not per row — shaped for the review's Emerging Research item (what AI4L's prompt asks about an ongoing trial; the link is `https://clinicaltrials.gov/study/<NCT ID>`): NCT ID, title, status with why a trial stopped, phase, enrollment as "100 (estimated)", start and primary completion dates (whether "ongoing" still holds), the primary outcome with its time frame, conditions, interventions, `has_results`; the full detail of one row is `library_clinicaltrials_get_study_record`; `total` is the registry's count, for a "no trials found" statement. The same search is answered from `search/<2>/<4>/<key>.json` for `FH_LIBRARY_SEARCH_CACHE_DAYS` (7) — a registry listing a week old is current enough for a review, and an audit that runs again reads from disk. Two thin tools answer from the stored record of any row (added first when missing): `library_clinicaltrials_get_study_record(nct)` is the review's detail of one trial — official title, design, enrollment, dates, eligibility, sites, result publications as `pmid:` IDs, the brief summary — and `library_clinicaltrials_get_study_results(nct, max_chars?)` is the posted results part of the stored Markdown (participant flow, baseline, outcome measures, adverse events), with the outcome titles listed so `library_read("nct:…", section)` can return one measure whole. Together with the search they cover what the AI4L models asked the ClinicalTrials.gov MCP server for.
155
160
 
156
161
  **Web pages.** `library_fetch(url, max_length?, start_index?, raw?)` is the standard MCP [fetch](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) tool, served from the library, for every link a review carries that is not a paper, label or trial record: a guideline, a product page, a news item. Its description tells the model to prefer it over a plain fetch or a browser tool, and says why: a page anyone fetched in the last 30 days comes from disk in milliseconds, and a page a plain fetch cannot get climbs the ladder in the same call. The answer is what `fetch` returns, so a model keeps its habits: `Contents of <url>:` and the page as Markdown with the page's title as the first heading, cut at `max_length` (default 5000) with the note `Content truncated. Call the fetch tool with a start_index of N to get more content`; `raw` gives the HTML; a `text/markdown` body comes back as is, any other non-HTML body with the `Content type … cannot be simplified to markdown` prefix; a failure is `Failed to fetch <url> - status code 404` or `Failed to fetch <url>: <reason>`. Two things happen underneath. The page is fetched up the same ladder as a paper — the plain fetch, then the browser rungs on a 403, 429 or bot wall, then Bright Data — so a page a plain fetch cannot reach is still read. And it is stored: `url/<2>/<4>/<16 hex of the URL's SHA-1>/url-<hash>-{meta.md, page.md, page.html|pdf|txt}`, the URL normalised first (host lowercased, fragment and tracking parameters dropped). The same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS` (1) — an audit that runs twice over one review reads its links once. A PDF comes back as Markdown (`fetch` would return its bytes as text). `title_only` returns `Contents of <url>:` and the title line alone — the page is still fetched and stored in full — so a link check costs a few tokens instead of a page. The log says which — `fetch example.com/… cached html … https://example.com/drugs/x` or `fetched 02.3s html retrieved_by=browser hetzner … <url>` (the ID column holds the URL without scheme, cut to a PubMed ID's width; the full URL ends the line); `library_info` counts the pages under `url`, a failed fetch as level `title`.
package/dist/index.js CHANGED
@@ -1480,41 +1480,6 @@ function europepmcCandidates(r, pmcid) {
1480
1480
  return out;
1481
1481
  }
1482
1482
 
1483
- // src/tiers/inbox.ts
1484
- import { existsSync as existsSync4, readdirSync, readFileSync as readFileSync3, unlinkSync } from "fs";
1485
- import { join as join4 } from "path";
1486
- var INBOX_DIR = join4(FH_LIBRARY_DIR, "inbox");
1487
- var EXT = { ".pdf": "application/pdf", ".xml": "text/xml", ".html": "text/html", ".htm": "text/html" };
1488
- function findInboxFile(dir, pmid) {
1489
- if (!existsSync4(dir)) return null;
1490
- const re = new RegExp(`(^|[^0-9])${pmid}([^0-9]|$)`);
1491
- for (const name of readdirSync(dir).sort()) {
1492
- const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
1493
- if (ext in EXT && re.test(name.slice(0, -ext.length))) return join4(dir, name);
1494
- }
1495
- return null;
1496
- }
1497
- function inboxTier(dir = INBOX_DIR) {
1498
- return {
1499
- name: "inbox",
1500
- always: true,
1501
- applies: (p) => p.pmid ? true : "no pmid",
1502
- async run(p, ctx) {
1503
- const file2 = findInboxFile(dir, p.pmid);
1504
- if (!file2) return { ...miss("inbox", "no file"), skipped: true, ephemeral: true };
1505
- const name = file2.slice(file2.lastIndexOf("/") + 1);
1506
- const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
1507
- const bytes = readFileSync3(file2);
1508
- const r = await convertBytes(p, bytes, EXT[ext]);
1509
- if (!r.ok) return miss("inbox", `${name}: ${r.reason}`);
1510
- ctx.onTry?.(`${name} ok, removed from the inbox`);
1511
- unlinkSync(file2);
1512
- return { ok: true, tier: "inbox", format: r.format, raw: r.format === "pdf" ? bytes : bytes.toString("utf8"), md: r.md, version: "published" };
1513
- }
1514
- };
1515
- }
1516
- var inbox = inboxTier();
1517
-
1518
1483
  // src/tiers/openalex.ts
1519
1484
  var openalex = {
1520
1485
  name: "openalex",
@@ -1675,7 +1640,6 @@ function unpaywallCandidates(d) {
1675
1640
 
1676
1641
  // src/tiers/index.ts
1677
1642
  var BUILTIN = [
1678
- { name: "inbox", make: () => inbox },
1679
1643
  { name: "pmc", make: () => pmc },
1680
1644
  { name: "europepmc", make: () => europepmc },
1681
1645
  { name: "tdm", make: (o) => tdmTier(o.tokens ?? {}) },
@@ -1774,9 +1738,9 @@ function resolveTiers(env = process.env) {
1774
1738
  // src/log.ts
1775
1739
  import { appendFileSync, mkdirSync as mkdirSync2, renameSync, statSync } from "fs";
1776
1740
  import { hostname } from "os";
1777
- import { join as join5 } from "path";
1741
+ import { join as join4 } from "path";
1778
1742
  var HOST = hostname();
1779
- var LOG_FILE = join5(FH_LIBRARY_DIR, "library.log");
1743
+ var LOG_FILE = join4(FH_LIBRARY_DIR, "library.log");
1780
1744
  function stamp() {
1781
1745
  const d = /* @__PURE__ */ new Date();
1782
1746
  const p = (n) => String(n).padStart(2, "0");
@@ -1811,12 +1775,12 @@ function log(tool, id, message) {
1811
1775
  }
1812
1776
 
1813
1777
  // src/stats.ts
1814
- import { mkdirSync as mkdirSync4, readdirSync as readdirSync3, readFileSync as readFileSync5, statSync as statSync2, unlinkSync as unlinkSync3 } from "fs";
1815
- import { join as join7 } from "path";
1778
+ import { mkdirSync as mkdirSync4, readdirSync as readdirSync2, readFileSync as readFileSync4, statSync as statSync2, unlinkSync as unlinkSync2 } from "fs";
1779
+ import { join as join6 } from "path";
1816
1780
 
1817
1781
  // src/entry.ts
1818
- import { copyFileSync, existsSync as existsSync5, mkdirSync as mkdirSync3, readdirSync as readdirSync2, readFileSync as readFileSync4, renameSync as renameSync2, rmSync, unlinkSync as unlinkSync2, writeFileSync } from "fs";
1819
- import { join as join6 } from "path";
1782
+ import { copyFileSync, existsSync as existsSync4, mkdirSync as mkdirSync3, readdirSync, readFileSync as readFileSync3, renameSync as renameSync2, rmSync, unlinkSync, writeFileSync } from "fs";
1783
+ import { join as join5 } from "path";
1820
1784
  import { basename, dirname } from "path";
1821
1785
 
1822
1786
  // src/pubmed.ts
@@ -2440,7 +2404,7 @@ ${body}
2440
2404
  }
2441
2405
  function readHeadLine(p) {
2442
2406
  try {
2443
- const lines = readFileSync4(p.md, "utf8").split("\n").filter((l) => l.trim());
2407
+ const lines = readFileSync3(p.md, "utf8").split("\n").filter((l) => l.trim());
2444
2408
  return lines[1] ?? "";
2445
2409
  } catch {
2446
2410
  return "";
@@ -2562,28 +2526,28 @@ function splitPublicationTypes(v) {
2562
2526
  }
2563
2527
  function abstractMissing(p) {
2564
2528
  try {
2565
- return readFileSync4(p.md, "utf8").includes("_No abstract available._");
2529
+ return readFileSync3(p.md, "utf8").includes("_No abstract available._");
2566
2530
  } catch {
2567
2531
  return false;
2568
2532
  }
2569
2533
  }
2570
2534
  function correctionsFromXml(p) {
2571
2535
  try {
2572
- return parsePubmedXml(readFileSync4(p.xml, "utf8"))?.corrections ?? [];
2536
+ return parsePubmedXml(readFileSync3(p.xml, "utf8"))?.corrections ?? [];
2573
2537
  } catch {
2574
2538
  return [];
2575
2539
  }
2576
2540
  }
2577
2541
  function declarationsFromXml(p) {
2578
2542
  try {
2579
- const r = parsePubmedXml(readFileSync4(p.xml, "utf8"));
2543
+ const r = parsePubmedXml(readFileSync3(p.xml, "utf8"));
2580
2544
  return { coi: r?.coi ?? null, funding: r?.funding ?? [] };
2581
2545
  } catch {
2582
2546
  return { coi: null, funding: [] };
2583
2547
  }
2584
2548
  }
2585
2549
  function readMeta(p) {
2586
- const { fm, all } = parseFrontmatter(readFileSync4(p.meta, "utf8"));
2550
+ const { fm, all } = parseFrontmatter(readFileSync3(p.meta, "utf8"));
2587
2551
  let title = fm.title ?? "";
2588
2552
  if (title.startsWith('"')) {
2589
2553
  try {
@@ -2639,7 +2603,7 @@ function writeAtomic(file2, data) {
2639
2603
  function writeMeta(p, m) {
2640
2604
  writeAtomic(p.meta, renderMeta(m));
2641
2605
  try {
2642
- unlinkSync2(join6(p.entryDir, `${p.entry}-missing.md`));
2606
+ unlinkSync(join5(p.entryDir, `${p.entry}-missing.md`));
2643
2607
  } catch {
2644
2608
  }
2645
2609
  }
@@ -2648,12 +2612,12 @@ function copyEntry(from, to) {
2648
2612
  rmSync(tmp, { recursive: true, force: true });
2649
2613
  mkdirSync3(tmp, { recursive: true });
2650
2614
  const rename = (n) => n.startsWith(from.entry) ? to.entry + n.slice(from.entry.length) : n;
2651
- for (const n of readdirSync2(from.entryDir)) if (join6(from.entryDir, n) !== from.meta && !n.endsWith(".tmp")) copyFileSync(join6(from.entryDir, n), join6(tmp, rename(n)));
2615
+ for (const n of readdirSync(from.entryDir)) if (join5(from.entryDir, n) !== from.meta && !n.endsWith(".tmp")) copyFileSync(join5(from.entryDir, n), join5(tmp, rename(n)));
2652
2616
  const m = readMeta(from);
2653
2617
  const copy = { ...m, files: m.files.map(rename) };
2654
- writeFileSync(join6(tmp, basename(to.meta)), renderMeta(copy));
2618
+ writeFileSync(join5(tmp, basename(to.meta)), renderMeta(copy));
2655
2619
  mkdirSync3(dirname(to.entryDir), { recursive: true });
2656
- if (existsSync5(to.entryDir)) renameSync2(to.entryDir, old);
2620
+ if (existsSync4(to.entryDir)) renameSync2(to.entryDir, old);
2657
2621
  renameSync2(tmp, to.entryDir);
2658
2622
  rmSync(old, { recursive: true, force: true });
2659
2623
  return copy;
@@ -2675,7 +2639,7 @@ function citability(m) {
2675
2639
  return { citable: true, comment: notes.length ? notes.join("; ") : null };
2676
2640
  }
2677
2641
  function statusFromMeta(p, m, base) {
2678
- const full = existsSync5(p.fulltextMd) ? readFileSync4(p.fulltextMd, "utf8") : null;
2642
+ const full = existsSync4(p.fulltextMd) ? readFileSync3(p.fulltextMd, "utf8") : null;
2679
2643
  const cit = citability(m);
2680
2644
  if (full?.includes(`
2681
2645
  ${CORRECTED_HEADING}
@@ -2717,7 +2681,7 @@ function readStatus(p, base) {
2717
2681
  return statusFromMeta(p, readMeta(p), base);
2718
2682
  }
2719
2683
  function entryExists(p) {
2720
- return existsSync5(p.meta);
2684
+ return existsSync4(p.meta);
2721
2685
  }
2722
2686
  function writeEntry(p, r, xml, converter) {
2723
2687
  mkdirSync3(p.entryDir, { recursive: true });
@@ -2800,15 +2764,15 @@ ${sentences.join("\n\n")}
2800
2764
  `;
2801
2765
  }
2802
2766
  function syncCorrectedBlock(p) {
2803
- if (!existsSync5(p.fulltextMd) || !existsSync5(p.xml)) return;
2767
+ if (!existsSync4(p.fulltextMd) || !existsSync4(p.xml)) return;
2804
2768
  let r;
2805
2769
  try {
2806
- r = parsePubmedXml(readFileSync4(p.xml, "utf8"));
2770
+ r = parsePubmedXml(readFileSync3(p.xml, "utf8"));
2807
2771
  } catch {
2808
2772
  return;
2809
2773
  }
2810
2774
  if (!r) return;
2811
- const md = readFileSync4(p.fulltextMd, "utf8");
2775
+ const md = readFileSync3(p.fulltextMd, "utf8");
2812
2776
  const next = withCorrectedBlock(md, correctedSentences(r));
2813
2777
  if (next !== md) writeAtomic(p.fulltextMd, next);
2814
2778
  }
@@ -2924,13 +2888,13 @@ var stats = null;
2924
2888
  var file = null;
2925
2889
  function startStats(pkg2, libraryDir = FH_LIBRARY_DIR) {
2926
2890
  stats = { host: HOST, pid: process.pid, package: pkg2.name, version: pkg2.version, started: (/* @__PURE__ */ new Date()).toISOString(), verify: { hits: 0, misses: 0 }, fetch: { hits: 0, misses: 0 }, search: { hits: 0, misses: 0 }, pubmed: { hits: 0, misses: 0 } };
2927
- file = join7(libraryDir, ".servers", `${HOST}-${process.pid}.json`);
2928
- sweep(join7(libraryDir, ".servers"));
2891
+ file = join6(libraryDir, ".servers", `${HOST}-${process.pid}.json`);
2892
+ sweep(join6(libraryDir, ".servers"));
2929
2893
  publish();
2930
2894
  setInterval(publish, HEARTBEAT_MS).unref();
2931
2895
  const stop = () => {
2932
2896
  try {
2933
- if (file) unlinkSync3(file);
2897
+ if (file) unlinkSync2(file);
2934
2898
  } catch {
2935
2899
  }
2936
2900
  };
@@ -2942,16 +2906,16 @@ function sweep(dir, staleMs = STALE_MS) {
2942
2906
  let n = 0;
2943
2907
  let names;
2944
2908
  try {
2945
- names = readdirSync3(dir);
2909
+ names = readdirSync2(dir);
2946
2910
  } catch {
2947
2911
  return 0;
2948
2912
  }
2949
2913
  for (const name of names) {
2950
2914
  if (!name.endsWith(".json")) continue;
2951
- const p = join7(dir, name);
2915
+ const p = join6(dir, name);
2952
2916
  try {
2953
2917
  if (Date.now() - statSync2(p).mtimeMs > staleMs) {
2954
- unlinkSync3(p);
2918
+ unlinkSync2(p);
2955
2919
  n++;
2956
2920
  }
2957
2921
  } catch {
@@ -2968,14 +2932,14 @@ var currentStats = () => stats;
2968
2932
  function publish() {
2969
2933
  if (!stats || !file) return;
2970
2934
  try {
2971
- mkdirSync4(join7(file, ".."), { recursive: true });
2935
+ mkdirSync4(join6(file, ".."), { recursive: true });
2972
2936
  writeAtomic(file, JSON.stringify(stats, null, 2) + "\n");
2973
2937
  } catch {
2974
2938
  }
2975
2939
  }
2976
2940
 
2977
2941
  // src/tools.ts
2978
- import { existsSync as existsSync13, readFileSync as readFileSync13 } from "fs";
2942
+ import { existsSync as existsSync12, readFileSync as readFileSync12 } from "fs";
2979
2943
  import { z } from "zod";
2980
2944
 
2981
2945
  // src/cascade.ts
@@ -3008,7 +2972,7 @@ async function fulltextCascade(p, attempts, onEvent, tiers = resolveTiers().tier
3008
2972
  const misses = [];
3009
2973
  const tried = /* @__PURE__ */ new Set();
3010
2974
  for (const tier of tiers) {
3011
- if (!tier.always && shouldSkip(attempts, tier.name, retryAfterDays, /* @__PURE__ */ new Date(), retryBlocked)) continue;
2975
+ if (shouldSkip(attempts, tier.name, retryAfterDays, /* @__PURE__ */ new Date(), retryBlocked)) continue;
3012
2976
  const applies = tier.applies(p);
3013
2977
  const ctx = { tried, onTry: (message) => onEvent?.({ ok: "try", tier: tier.name, message }), onAlert: (message) => onEvent?.({ ok: "alert", tier: tier.name, message }) };
3014
2978
  const r = applies === true ? await tier.run(p, ctx) : { ...miss(tier.name, applies), skipped: true };
@@ -3050,11 +3014,11 @@ function parseId(raw) {
3050
3014
  }
3051
3015
 
3052
3016
  // src/info.ts
3053
- import { existsSync as existsSync7, readdirSync as readdirSync4, readFileSync as readFileSync7, statSync as statSync3 } from "fs";
3054
- import { dirname as dirname2, join as join8 } from "path";
3017
+ import { existsSync as existsSync6, readdirSync as readdirSync3, readFileSync as readFileSync6, statSync as statSync3 } from "fs";
3018
+ import { dirname as dirname2, join as join7 } from "path";
3055
3019
 
3056
3020
  // src/verify.ts
3057
- import { existsSync as existsSync6, readFileSync as readFileSync6 } from "fs";
3021
+ import { existsSync as existsSync5, readFileSync as readFileSync5 } from "fs";
3058
3022
  var DEFAULT_MODEL = "pplx-decider-flash-v1.2";
3059
3023
  var QUESTION = "Does the paper state this, in substance? Ignore section labels and definitions of terms in parentheses. Every fact and number of the statement must be stated in the paper or follow from its numbers; a rounded number counts. If any part of the statement is absent from the paper or contradicted by it, answer false.";
3060
3024
  function verifyConfig(env = process.env) {
@@ -3156,10 +3120,10 @@ async function readPaper(raw, libraryDir = FH_LIBRARY_DIR) {
3156
3120
  if (p.error) return { id: p.requested, error: p.error };
3157
3121
  const paths = entryPaths(libraryDir, p.type, p.id);
3158
3122
  if (!entryExists(paths)) return { id: p.requested, error: "not in library" };
3159
- const full = existsSync6(paths.fulltextMd);
3160
- if (full) return { id: p.requested, basis: "full_text", text: readFileSync6(paths.fulltextMd, "utf8") };
3123
+ const full = existsSync5(paths.fulltextMd);
3124
+ if (full) return { id: p.requested, basis: "full_text", text: readFileSync5(paths.fulltextMd, "utf8") };
3161
3125
  const meta = readMeta(paths);
3162
- return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text: readFileSync6(paths.md, "utf8") };
3126
+ return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text: readFileSync5(paths.md, "utf8") };
3163
3127
  }
3164
3128
  async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
3165
3129
  const out = { claim, id, verdict: "error", reason: null, score: null, basis: null, windows: 0, glyphs_unresolved: 0, superseded: null, model: cfg?.model ?? null };
@@ -3200,9 +3164,9 @@ async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
3200
3164
 
3201
3165
  // src/info.ts
3202
3166
  function* metaFiles(dir) {
3203
- if (!existsSync7(dir)) return;
3204
- for (const name of readdirSync4(dir)) {
3205
- const p = join8(dir, name);
3167
+ if (!existsSync6(dir)) return;
3168
+ for (const name of readdirSync3(dir)) {
3169
+ const p = join7(dir, name);
3206
3170
  if (statSync3(p).isDirectory()) yield* metaFiles(p);
3207
3171
  else if (name.endsWith("-meta.md")) yield p;
3208
3172
  }
@@ -3241,11 +3205,11 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
3241
3205
  updates: 0
3242
3206
  };
3243
3207
  const ids = [];
3244
- for (const type of existsSync7(libraryDir) ? readdirSync4(libraryDir) : []) {
3245
- const typeDir = join8(libraryDir, type);
3208
+ for (const type of existsSync6(libraryDir) ? readdirSync3(libraryDir) : []) {
3209
+ const typeDir = join7(libraryDir, type);
3246
3210
  if (!statSync3(typeDir).isDirectory()) continue;
3247
3211
  for (const meta of metaFiles(typeDir)) {
3248
- const md = readFileSync7(meta, "utf8");
3212
+ const md = readFileSync6(meta, "utf8");
3249
3213
  const raw = field(md, "level") || "unknown";
3250
3214
  const level = raw === "abstract" && field(md, "has_abstract") === "no" ? "title" : raw;
3251
3215
  info.entries++;
@@ -3255,9 +3219,9 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
3255
3219
  t.entries++;
3256
3220
  if (level === "full_text" || level === "abstract" || level === "title") t[level]++;
3257
3221
  const dir = dirname2(meta);
3258
- for (const f of readdirSync4(dir)) {
3222
+ for (const f of readdirSync3(dir)) {
3259
3223
  try {
3260
- const st = statSync3(join8(dir, f));
3224
+ const st = statSync3(join7(dir, f));
3261
3225
  if (st.isFile()) info.size_bytes += st.size;
3262
3226
  } catch {
3263
3227
  }
@@ -3286,7 +3250,7 @@ function noticeTypes(md, metaPath) {
3286
3250
  return out;
3287
3251
  }
3288
3252
  try {
3289
- const xml = readFileSync7(metaPath.replace(/-meta\.md$/, "-abstract.xml"), "utf8");
3253
+ const xml = readFileSync6(metaPath.replace(/-meta\.md$/, "-abstract.xml"), "utf8");
3290
3254
  for (const m of xml.matchAll(/<CommentsCorrections RefType="([A-Za-z]+)"/g)) if (XML_TYPES[m[1]]) out.add(XML_TYPES[m[1]]);
3291
3255
  } catch {
3292
3256
  }
@@ -3665,8 +3629,8 @@ function parseCsl(json2) {
3665
3629
 
3666
3630
  // src/urlfetch.ts
3667
3631
  import { createHash as createHash2 } from "crypto";
3668
- import { existsSync as existsSync8, mkdirSync as mkdirSync5, readFileSync as readFileSync8 } from "fs";
3669
- import { join as join9 } from "path";
3632
+ import { existsSync as existsSync7, mkdirSync as mkdirSync5, readFileSync as readFileSync7 } from "fs";
3633
+ import { join as join8 } from "path";
3670
3634
  var TRACKING = /^(utm_[a-z]+|fbclid|gclid|dclid|msclkid|mc_cid|mc_eid|igshid|ref_src)$/i;
3671
3635
  function normalizeUrl(raw) {
3672
3636
  let u;
@@ -3688,8 +3652,8 @@ var urlHash = (normalized2) => createHash2("sha1").update(normalized2).digest("h
3688
3652
  function urlPaths(libraryDir, normalized2) {
3689
3653
  const hash = urlHash(normalized2);
3690
3654
  const entry = `url-${hash}`;
3691
- const entryDir = join9(libraryDir, "url", hash.slice(0, 2), hash.slice(0, 4), hash);
3692
- return { hash, entry, entryDir, meta: join9(entryDir, `${entry}-meta.md`), md: join9(entryDir, `${entry}-page.md`), raw: (ext) => join9(entryDir, `${entry}-page.${ext}`) };
3655
+ const entryDir = join8(libraryDir, "url", hash.slice(0, 2), hash.slice(0, 4), hash);
3656
+ return { hash, entry, entryDir, meta: join8(entryDir, `${entry}-meta.md`), md: join8(entryDir, `${entry}-page.md`), raw: (ext) => join8(entryDir, `${entry}-page.${ext}`) };
3693
3657
  }
3694
3658
  function renderUrlMeta(m) {
3695
3659
  const lines = [
@@ -3708,8 +3672,8 @@ function renderUrlMeta(m) {
3708
3672
  return lines.map(([k, v]) => `${k}: ${v}`).join("\n") + "\n";
3709
3673
  }
3710
3674
  function readUrlMeta(p) {
3711
- if (!existsSync8(p.meta)) return null;
3712
- const md = readFileSync8(p.meta, "utf8");
3675
+ if (!existsSync7(p.meta)) return null;
3676
+ const md = readFileSync7(p.meta, "utf8");
3713
3677
  const f = (k) => {
3714
3678
  const m = new RegExp(`^${k}:[ \\t]*(.*)$`, "m").exec(md);
3715
3679
  return m ? m[1].trim() : "";
@@ -3793,7 +3757,7 @@ var climb = async (url, onTry, onAlert) => {
3793
3757
  if (!r.ok && /^unlocker HTTP/.test(r.reason)) onAlert?.(`brightdata: ${r.reason}`);
3794
3758
  }
3795
3759
  if (!r.ok && shell) {
3796
- onTry?.(`script shell kept: no rung rendered the page`);
3760
+ onTry?.("shell kept: no rung rendered the page, the script shell is served");
3797
3761
  return shell;
3798
3762
  }
3799
3763
  return r.ok || r.via === "fetch" ? r : { ...r, first: firstReason };
@@ -3886,6 +3850,25 @@ function logId(url) {
3886
3850
  const bare2 = url.replace(/^https?:\/\//i, "");
3887
3851
  return bare2.length > ID_WIDTH ? bare2.slice(0, ID_WIDTH - 1) + "\u2026" : bare2;
3888
3852
  }
3853
+ function libraryIdForUrl(raw) {
3854
+ let u;
3855
+ try {
3856
+ u = new URL(raw.trim());
3857
+ } catch {
3858
+ return null;
3859
+ }
3860
+ const host = u.hostname.toLowerCase().replace(/^www\./, ""), path = u.pathname;
3861
+ let m;
3862
+ if (host === "pubmed.ncbi.nlm.nih.gov" && (m = /^\/(\d{1,8})\/?$/.exec(path))) return `pmid:${m[1]}`;
3863
+ if (host === "ncbi.nlm.nih.gov" && (m = /^\/pubmed\/(\d{1,8})\/?$/.exec(path))) return `pmid:${m[1]}`;
3864
+ if (host === "clinicaltrials.gov" && (m = /^\/(?:study|ct2\/show)\/(NCT\d{8})\/?$/i.exec(path))) return `nct:${m[1].slice(3)}`;
3865
+ if (host === "dailymed.nlm.nih.gov" && /\/(drugInfo|lookup)\.cfm$/i.test(path)) {
3866
+ const id = u.searchParams.get("setid");
3867
+ if (id && /^[0-9a-f-]{36}$/i.test(id)) return `dailymed:${id.toLowerCase()}`;
3868
+ }
3869
+ if ((host === "doi.org" || host === "dx.doi.org") && (m = /^\/(10\.\S+)$/.exec(decodeURIComponent(path)))) return `doi:${m[1]}`;
3870
+ return null;
3871
+ }
3889
3872
  var failureText = (url, reason) => {
3890
3873
  const m = /^HTTP (\d{3})\b/.exec(reason);
3891
3874
  return m ? `Failed to fetch ${url} - status code ${m[1]}` : `Failed to fetch ${url}: ${reason}`;
@@ -3898,14 +3881,14 @@ function titleLine(m, md) {
3898
3881
  }
3899
3882
  function serve(requested, p, m, args, cached) {
3900
3883
  const maxLength = args.max_length ?? 5e3, startIndex = args.start_index ?? 0;
3901
- const read = (file2) => existsSync8(file2) ? readFileSync8(file2, "utf8") : "";
3884
+ const read = (file2) => existsSync7(file2) ? readFileSync7(file2, "utf8") : "";
3902
3885
  if (args.title_only) return { ok: true, text: `Contents of ${requested}:
3903
3886
  ${titleLine(m, read(p.md))}`, cached, meta: m };
3904
3887
  const rawFile = m.files.find((f) => !f.endsWith("-meta.md") && !f.endsWith("-page.md"));
3905
3888
  let body, prefix = "";
3906
3889
  if (m.format === "markdown") body = read(p.md);
3907
3890
  else if (m.format === "text" || args.raw) {
3908
- body = rawFile ? read(join9(p.entryDir, rawFile)) : read(p.md);
3891
+ body = rawFile ? read(join8(p.entryDir, rawFile)) : read(p.md);
3909
3892
  prefix = `Content type ${m.content_type || "(not declared)"} cannot be simplified to markdown, but here is the raw content:
3910
3893
  `;
3911
3894
  } else body = read(p.md);
@@ -3954,8 +3937,8 @@ async function libraryFetch(libraryDir, requested, args, converter, logs = {}, f
3954
3937
 
3955
3938
  // src/verifycache.ts
3956
3939
  import { createHash as createHash3 } from "crypto";
3957
- import { existsSync as existsSync9, mkdirSync as mkdirSync6, readFileSync as readFileSync9 } from "fs";
3958
- import { dirname as dirname3, join as join10 } from "path";
3940
+ import { existsSync as existsSync8, mkdirSync as mkdirSync6, readFileSync as readFileSync8 } from "fs";
3941
+ import { dirname as dirname3, join as join9 } from "path";
3959
3942
  var sha = (s) => createHash3("sha1").update(s).digest("hex");
3960
3943
  var normalizeClaim = (claim) => claim.replace(/\s+/g, " ").trim();
3961
3944
  function foldClaim(claim) {
@@ -3967,11 +3950,11 @@ ${id}
3967
3950
  ${model}
3968
3951
  ${threshold}
3969
3952
  ${text3}`).slice(0, 16);
3970
- var verdictFile = (libraryDir, key) => join10(libraryDir, "verify", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
3953
+ var verdictFile = (libraryDir, key) => join9(libraryDir, "verify", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
3971
3954
  function readVerdict(file2) {
3972
- if (!existsSync9(file2)) return null;
3955
+ if (!existsSync8(file2)) return null;
3973
3956
  try {
3974
- return JSON.parse(readFileSync9(file2, "utf8"));
3957
+ return JSON.parse(readFileSync8(file2, "utf8"));
3975
3958
  } catch {
3976
3959
  return null;
3977
3960
  }
@@ -4021,8 +4004,8 @@ var leanJson = (o) => compact2(lean(o));
4021
4004
 
4022
4005
  // src/trials.ts
4023
4006
  import { createHash as createHash4 } from "crypto";
4024
- import { existsSync as existsSync10, mkdirSync as mkdirSync7, readFileSync as readFileSync10 } from "fs";
4025
- import { dirname as dirname4, join as join11 } from "path";
4007
+ import { existsSync as existsSync9, mkdirSync as mkdirSync7, readFileSync as readFileSync9 } from "fs";
4008
+ import { dirname as dirname4, join as join10 } from "path";
4026
4009
  var searchCacheDays = () => settings().search_cache_days;
4027
4010
  var API2 = "https://clinicaltrials.gov/api/v2/studies";
4028
4011
  var FIELDS = ["NCTId", "BriefTitle", "OverallStatus", "WhyStopped", "Phase", "StudyType", "EnrollmentCount", "EnrollmentType", "StartDate", "PrimaryCompletionDate", "CompletionDate", "LastUpdatePostDate", "PrimaryOutcomeMeasure", "PrimaryOutcomeTimeFrame", "Condition", "InterventionName", "LeadSponsorName", "HasResults"];
@@ -4060,7 +4043,7 @@ function normalizeQuery(q) {
4060
4043
  return out;
4061
4044
  }
4062
4045
  var searchKey = (q) => createHash4("sha1").update(JSON.stringify(["clinicaltrials", q.query ?? "", q.intervention ?? "", q.condition ?? "", q.status, q.sort, q.max_results])).digest("hex").slice(0, 16);
4063
- var searchFile = (libraryDir, key) => join11(libraryDir, "search", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
4046
+ var searchFile = (libraryDir, key) => join10(libraryDir, "search", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
4064
4047
  function searchUrl(q) {
4065
4048
  const p = new URLSearchParams();
4066
4049
  if (q.query) p.set("query.term", q.query);
@@ -4112,9 +4095,9 @@ async function searchTrials(raw, libraryDir = FH_LIBRARY_DIR, fetchImpl = liveFe
4112
4095
  const q = normalizeQuery(raw);
4113
4096
  if ("error" in q) return { ok: false, error: q.error };
4114
4097
  const file2 = searchFile(libraryDir, searchKey(q));
4115
- if (existsSync10(file2)) {
4098
+ if (existsSync9(file2)) {
4116
4099
  try {
4117
- const r = JSON.parse(readFileSync10(file2, "utf8"));
4100
+ const r = JSON.parse(readFileSync9(file2, "utf8"));
4118
4101
  if (daysSince(r.searched) < searchCacheDays()) return { ok: true, result: r, cached: true };
4119
4102
  } catch {
4120
4103
  }
@@ -4146,7 +4129,7 @@ function trialsLogId(q) {
4146
4129
 
4147
4130
  // src/pubmedsearch.ts
4148
4131
  import { createHash as createHash5 } from "crypto";
4149
- import { existsSync as existsSync11, mkdirSync as mkdirSync8, readFileSync as readFileSync11 } from "fs";
4132
+ import { existsSync as existsSync10, mkdirSync as mkdirSync8, readFileSync as readFileSync10 } from "fs";
4150
4133
  import { dirname as dirname5 } from "path";
4151
4134
  var MAX_RESULTS2 = 100;
4152
4135
  var DEFAULT_RESULTS2 = 20;
@@ -4227,9 +4210,9 @@ async function searchArticles(raw, libraryDir = FH_LIBRARY_DIR, fetchImpl = live
4227
4210
  const q = normalizeQuery2(raw);
4228
4211
  if ("error" in q) return { ok: false, error: q.error };
4229
4212
  const file2 = searchFile(libraryDir, searchKey2(q));
4230
- if (existsSync11(file2)) {
4213
+ if (existsSync10(file2)) {
4231
4214
  try {
4232
- const r = JSON.parse(readFileSync11(file2, "utf8"));
4215
+ const r = JSON.parse(readFileSync10(file2, "utf8"));
4233
4216
  if (daysSince(r.searched) < searchCacheDays()) return { ok: true, result: r, cached: true };
4234
4217
  } catch {
4235
4218
  }
@@ -4333,13 +4316,13 @@ function resultsText(nct, md, maxChars) {
4333
4316
  }
4334
4317
 
4335
4318
  // src/missing.ts
4336
- import { existsSync as existsSync12, mkdirSync as mkdirSync9, readFileSync as readFileSync12 } from "fs";
4337
- import { join as join12 } from "path";
4338
- var missingFile = (p) => join12(p.entryDir, `${p.entry}-missing.md`);
4319
+ import { existsSync as existsSync11, mkdirSync as mkdirSync9, readFileSync as readFileSync11 } from "fs";
4320
+ import { join as join11 } from "path";
4321
+ var missingFile = (p) => join11(p.entryDir, `${p.entry}-missing.md`);
4339
4322
  function readMissing(p) {
4340
4323
  const f = missingFile(p);
4341
- if (!existsSync12(f)) return null;
4342
- const md = readFileSync12(f, "utf8");
4324
+ if (!existsSync11(f)) return null;
4325
+ const md = readFileSync11(f, "utf8");
4343
4326
  const get = (k) => /^$/.test(k) ? "" : (new RegExp(`^${k}:[ \\t]*(.*)$`, "m").exec(md)?.[1] ?? "").trim();
4344
4327
  const checked = get("checked"), reason = get("reason");
4345
4328
  return checked ? { checked, reason: reason || "not found" } : null;
@@ -4445,7 +4428,7 @@ async function fetchArticles(ids, libraryDir = FH_LIBRARY_DIR, converter = "fh-l
4445
4428
  // src/tools.ts
4446
4429
  var pkg = (() => {
4447
4430
  try {
4448
- return JSON.parse(readFileSync13(new URL("../package.json", import.meta.url), "utf8"));
4431
+ return JSON.parse(readFileSync12(new URL("../package.json", import.meta.url), "utf8"));
4449
4432
  } catch {
4450
4433
  return { name: "fh-library", version: "0.0.0" };
4451
4434
  }
@@ -4504,7 +4487,7 @@ Reading a document
4504
4487
 
4505
4488
  Status fields: requested, type, id, entry, pmid, pmcid, doi, setid, nctid, publications, title, first_author, year, publication_types, retracted, corrections, coi (conflict-of-interest statement), funding, citable, comment, has_abstract, level, version, retrieved_by, retrieved_date, pubmed_checked, attempts, glyphs_unresolved, error. Answers are compact JSON; a field that is null or an empty list is left out (library_verify keeps its nulls: score and reason null mean something).
4506
4489
 
4507
- Where records come from: PubMed for the record and abstract; the full text from a file you drop into <library>/inbox/ (PMID in the file name), PubMed Central, Europe PMC, publishers' text-mining links, Unpaywall, OpenAlex, CORE, and a proxy URL template from the configuration. Every action is appended as one line to <library>/library.log.`;
4490
+ Where records come from: PubMed for the record and abstract; the full text from PubMed Central, Europe PMC, publishers' text-mining links, Unpaywall, OpenAlex, CORE, and a proxy URL template from the configuration. Every action is appended as one line to <library>/library.log.`;
4508
4491
  var idSchema = z.string().describe('ID as "type:value": a paper "pmid:30955454" (bare digits count as pmid) or "doi:10.5812/asjsm.110753" (a paper PubMed does not index; a DOI PubMed indexes is taken as its PMID), a drug label "dailymed:<set ID>" (the set ID from its DailyMed page URL), or a trial record "nct:05173896" (the eight digits; nct:NCT05173896 and a bare NCT05173896 are accepted).');
4509
4492
  var levelWord = (m) => m.level === "abstract" && !m.has_abstract ? "title" : m.level;
4510
4493
  var json = (s) => ({ content: [{ type: "text", text: leanJson(s) }] });
@@ -4841,8 +4824,8 @@ function registerTools(server2) {
4841
4824
  }
4842
4825
  added = true;
4843
4826
  }
4844
- const full = existsSync13(paths.fulltextMd);
4845
- const md = readFileSync13(full ? paths.fulltextMd : paths.md, "utf8");
4827
+ const full = existsSync12(paths.fulltextMd);
4828
+ const md = readFileSync12(full ? paths.fulltextMd : paths.md, "utf8");
4846
4829
  const text3 = full ? "full_text" : "abstract";
4847
4830
  const base = added ? { requested: parsed.requested, entry: paths.entry, text: text3, added: true } : { requested: parsed.requested, entry: paths.entry, text: text3 };
4848
4831
  const headings2 = () => outline(md, text3).sections;
@@ -4862,8 +4845,8 @@ function registerTools(server2) {
4862
4845
  const cap = max_chars ?? 6e3;
4863
4846
  let sec = readSection(md, section2, cap);
4864
4847
  let via = "";
4865
- if (!sec && /abstract/i.test(section2) && full && existsSync13(paths.md)) {
4866
- sec = readSection(readFileSync13(paths.md, "utf8"), "abstract", cap);
4848
+ if (!sec && /abstract/i.test(section2) && full && existsSync12(paths.md)) {
4849
+ sec = readSection(readFileSync12(paths.md, "utf8"), "abstract", cap);
4867
4850
  via = " (from the abstract file)";
4868
4851
  }
4869
4852
  if (!sec && /declaration/i.test(section2)) {
@@ -4938,7 +4921,7 @@ function registerTools(server2) {
4938
4921
  );
4939
4922
  server2.tool(
4940
4923
  "library_fetch",
4941
- `Fetch a web page as Markdown: the standard fetch tool, cached and hardened. Use it for any URL instead of a plain fetch or a browser. A page fetched in the last ${settings().refresh_after_days} days is answered from disk in milliseconds; a page a plain fetch cannot get (403, rate limit, bot wall, script-rendered) is fetched through the configured browsers and unlocker in this one call. PDFs come back as Markdown. To verify a link, title_only returns just its title line.`,
4924
+ `Fetch a web page as Markdown: the standard fetch tool, cached and hardened. Use it for any URL instead of a plain fetch or a browser. A page fetched in the last ${settings().refresh_after_days} days is answered from disk in milliseconds; a page a plain fetch cannot get (403, rate limit, bot wall, script-rendered) is fetched through the configured browsers and unlocker in this one call. PDFs come back as Markdown. To verify a link, title_only returns just its title line. A PubMed, ClinicalTrials.gov, DailyMed or doi.org URL is answered from the library's record of it.`,
4942
4925
  {
4943
4926
  url: z.string().describe("URL to fetch"),
4944
4927
  max_length: z.number().int().min(1).max(999999).default(5e3).describe("Maximum number of characters to return."),
@@ -4949,6 +4932,39 @@ function registerTools(server2) {
4949
4932
  async ({ url, max_length, start_index, raw, title_only }) => {
4950
4933
  const t0 = Date.now();
4951
4934
  const id = logId(url), put = (line) => log("library_fetch", id, `${line} ${url}`);
4935
+ let rid = raw ? null : libraryIdForUrl(url);
4936
+ if (rid?.startsWith("doi:") && !entryExists(entryPaths(FH_LIBRARY_DIR, "doi", parseId(rid).id))) {
4937
+ try {
4938
+ const pm = await pmidForDoi(parseId(rid).id);
4939
+ if (pm) rid = `pmid:${pm}`;
4940
+ } catch {
4941
+ }
4942
+ }
4943
+ if (rid) {
4944
+ const pid = parseId(rid);
4945
+ const paths = entryPaths(FH_LIBRARY_DIR, pid.type, pid.id);
4946
+ const had = entryExists(paths);
4947
+ if (!had && pid.type === "pmid") {
4948
+ const b = await fetchArticles([pid.id], FH_LIBRARY_DIR, `${pkg.name} ${pkg.version}`);
4949
+ if (b.rows[0]?.[6] === "not found" || !entryExists(paths)) {
4950
+ put(`library ${rid} error: pmid not found`);
4951
+ return { content: [{ type: "text", text: `Failed to fetch ${url}: pmid not found` }], isError: true };
4952
+ }
4953
+ } else if (!had) {
4954
+ const added = JSON.parse((await addEntry(rid)).content[0].text);
4955
+ if (added.error) {
4956
+ put(`library ${rid} error: ${added.error}`);
4957
+ return { content: [{ type: "text", text: `Failed to fetch ${url}: ${added.error}` }], isError: true };
4958
+ }
4959
+ }
4960
+ const file2 = pid.type === "pmid" ? paths.md : existsSync12(paths.fulltextMd) ? paths.fulltextMd : paths.md;
4961
+ const md = existsSync12(file2) ? readFileSync12(file2, "utf8") : "";
4962
+ const text3 = title_only ? `Contents of ${url}:
4963
+ ${(md.split("\n").find((l) => l.trim()) ?? "").trim()}` : fetchAnswer(url, md, "", max_length, start_index);
4964
+ bump("fetch", had);
4965
+ put(`library ${rid} ${had ? "cached" : "added"} chars=${text3.length}${title_only ? " title" : ""}${start_index ? ` start=${start_index}` : ""}`);
4966
+ return { content: [{ type: "text", text: text3 }] };
4967
+ }
4952
4968
  const r = await libraryFetch(FH_LIBRARY_DIR, url, { max_length, start_index, raw, title_only }, `${pkg.name} ${pkg.version}`, {
4953
4969
  onTry: put,
4954
4970
  onAlert: (line) => put(`!! ${line}`)
@@ -5049,7 +5065,7 @@ function registerTools(server2) {
5049
5065
  return { content: [{ type: "text", text: compact2({ error: e.error }) }], isError: true };
5050
5066
  }
5051
5067
  const raw = e.paths.fulltextRaw("json");
5052
- const r = existsSync13(raw) ? recordSummary(readFileSync13(raw, "utf8")) : null;
5068
+ const r = existsSync12(raw) ? recordSummary(readFileSync12(raw, "utf8")) : null;
5053
5069
  if (!r) {
5054
5070
  log("library_trials", e.requested, "record error: no stored record");
5055
5071
  return { content: [{ type: "text", text: compact2({ error: "no stored record" }) }], isError: true };
@@ -5068,7 +5084,7 @@ function registerTools(server2) {
5068
5084
  log("library_trials", nct, `results error: ${e.error}`);
5069
5085
  return { content: [{ type: "text", text: compact2({ error: e.error }) }], isError: true };
5070
5086
  }
5071
- const md = existsSync13(e.paths.fulltextMd) ? readFileSync13(e.paths.fulltextMd, "utf8") : "";
5087
+ const md = existsSync12(e.paths.fulltextMd) ? readFileSync12(e.paths.fulltextMd, "utf8") : "";
5072
5088
  const r = resultsText(e.requested.replace(/^nct:/, "NCT"), md, max_chars);
5073
5089
  log("library_trials", e.requested, `results ${r.has_results ? `${r.outcomes.length} outcomes${r.truncated ? " truncated" : ""}` : "none"}`);
5074
5090
  return { content: [{ type: "text", text: leanJson(r) }] };
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "fh-library",
3
- "version": "0.2.16",
3
+ "version": "0.2.17",
4
4
  "mcpName": "io.github.forever-healthy/fh-library",
5
5
  "description": "MCP Server that builds a local Markdown library of the scientific papers cited by evidence reviews — fetches metadata, abstract and full text per PubMed ID through a configurable cascade of retrieval services",
6
6
  "type": "module",