fh-library 0.2.16 → 0.2.17
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +2 -134
- package/README.md +8 -3
- package/dist/index.js +132 -116
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,137 +1,5 @@
|
|
|
1
1
|
# FH Library — Change Log
|
|
2
2
|
|
|
3
|
-
### v0.2.
|
|
3
|
+
### v0.2.17 — 2026-11-08
|
|
4
4
|
|
|
5
|
-
* **`
|
|
6
|
-
* **`FH_LIBRARY_SEARCH_CACHE_DAYS` (7):** days a PubMed or registry search is cached; was a constant.
|
|
7
|
-
* **An ID no service knows is remembered** for `FH_LIBRARY_RETRY_AFTER_DAYS` (1): a PMID PubMed has no record for, a DOI neither Crossref nor doi.org resolves, a set ID without a label, an NCT ID without a study. A marker (`<entry>-missing.md`, date and reason, no meta) in the entry's directory; `library_add`, `library_status`, `library_read`, `library_verify` and the batch tool answer from it ("pmid not found (checked 2026-11-08; asked again after 1 day)"), the service is asked again after the cooldown, a real entry replaces the marker. Network errors are never remembered. The same cooldown a failed page or a transient tier failure already had.
|
|
8
|
-
* **`library_pubmed_fetch_articles(pmids)`:** title, first author, year, publication types, retracted and level for up to 200 PMIDs in one call, rows in `columns` order (~120 characters a paper against the PubMed MCP server's ~2,500). Stored PMIDs from disk; the rest in one efetch, stored at abstract level as `library_add`'s first stage, so the cascade runs later as on an untried entry. The first of the PubMed tools (#27); `library_info`'s `cache` and `.servers/` count `pubmed` hits and misses per PMID.
|
|
9
|
-
|
|
10
|
-
### v0.2.9 — 2026-11-08
|
|
11
|
-
|
|
12
|
-
* TUI v82: the `url` row shows library_fetch hits and misses from the servers' counters; `/status` carries `fetches`. Counts right-aligned in a 5-character column.
|
|
13
|
-
* TUI v80: a `search` row — cached registry searches, hits and misses from the servers' counters; the decision-model row is labelled `decider`; `/status` carries `searches`.
|
|
14
|
-
* ClinicalTrials tool descriptions and server instructions one statement each.
|
|
15
|
-
|
|
16
|
-
### v0.2.8 — 2026-11-08
|
|
17
|
-
|
|
18
|
-
* **`library_clinicaltrials_get_study_record(nct)` and `library_clinicaltrials_get_study_results(nct, max_chars?)`:** thin answers over the stored `nct:` entry (added first when missing) — the review's detail of one trial, and its posted results as Markdown with the outcome titles for `library_read`. With the search, the three tools map one to one onto what the AI4L models used of the ClinicalTrials.gov MCP server (#28).
|
|
19
|
-
* `library_verify` cache key folds the claim's typography (#33): case, whitespace and non-breaking spaces, curly quotes, en/em dashes, "+/-" for "±", a trailing period — never a word. The claim is judged and stored as written.
|
|
20
|
-
* `library_verify`: a claim that opens with an odds ratio ("OR 2.15 (95% CI 1.06 to 4.34) for …") is a statement, not a fragment starting with the conjunction "or" (seen in an AI4L audit, pmid:25358826).
|
|
21
|
-
* **Smaller answers (#32):** every tool answers compact JSON (no indentation); `library_add`, `library_status`, `library_read` and `library_clinicaltrials_search_studies` leave out null values, empty lists and the entry's paths and file names (`entry_dir`, `library_dir`, `files`). `library_verify` keeps its nulls (`score`, `reason` null carry meaning). A status answer is about 30% smaller, a 20-row trial search about 25%.
|
|
22
|
-
* **`library_fetch(…, title_only)`:** the title line alone (`Contents of <url>:` and `# Title`); the page is fetched and stored in full as usual. For verifying a link at a few tokens instead of a page (#32).
|
|
23
|
-
* **`library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?)`:** a ClinicalTrials.gov search (API v2, no key) — total count and rows as arrays in `columns` order (nct, title, status with why stopped, phase, enrollment, start, primary_completion, primary_outcome, conditions, interventions, has_results — the names once, not per row); `status` ongoing / completed / all; `sort` relevance / newest / largest / completion. Cached under `search/<2>/<4>/<key>.json` for 7 days. With `library_add("nct:…")` and `library_read` for the record and posted results, this covers what the AI4L models asked the ClinicalTrials.gov MCP server for (#28). `library_info`'s `cache` and the `.servers/` files count `search` hits and misses.
|
|
24
|
-
|
|
25
|
-
### v0.2.7 — 2026-11-08
|
|
26
|
-
|
|
27
|
-
* **Verdicts are cached** (#26): `library_verify` keeps each judgement as `verify/<2>/<4>/<key>.json` — the file system as the database, laid out like `url/` — and answers the same claim on the same stored text from disk for `FH_LIBRARY_REFRESH_AFTER_DAYS`; a new full text, model or threshold is judged afresh, errors are never kept. The result is unchanged; the log line says `cached`. One atomic file per judgement: several servers on one library need no lock.
|
|
28
|
-
* **Each server publishes its cache counters:** hits and misses of `library_verify` and `library_fetch` since it started, in memory, written to `<library>/.servers/<host>-<pid>.json` on every change and every 5 min as a heartbeat, removed on exit, swept by the next starting server when older than 20 min; `library_info` reports them as `cache`. The server also logs a `startup` line.
|
|
29
|
-
* TUI v79: a `verify` row — stored verdicts ("queries cached"), then hits and misses summed over the servers alive on the library, read from `.servers/`, not from the log; `/status` carries `verdicts`. Resolves `.env` and `library/` from the caller's working directory, as the server does (#31) — `npm run --prefix <repo> tui` from a project root shows that project's library.
|
|
30
|
-
|
|
31
|
-
### v0.2.6 — 2026-11-08
|
|
32
|
-
|
|
33
|
-
* **`library_fetch(url, max_length?, start_index?, raw?)`:** the standard MCP `fetch` tool, served from the library. Same answer as the reference server — `Contents of <url>:`, the page as Markdown with its title as the first heading, the truncation note, the raw-content prefix, the `Failed to fetch …` errors — so a model keeps its fetch habits. Underneath, the page is fetched up the download ladder (plain fetch, Playwright and Chrome rungs, Bright Data) when a plain fetch is refused, and stored under `url/`: the same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS`. A PDF comes back as Markdown. `library_info` counts web pages as type `url`; the log line says `cached` or `fetched`.
|
|
34
|
-
* **Script shells climb the ladder:** a page answered 200 whose converted text is under 100 words with a loading/JavaScript phrase (or under 10 words at all) is treated like a wall page — the browser rungs, then Bright Data; a browser rung with so little text waits for the network to go quiet and the text to grow (at most 8 s) before reading the page. When no rung renders more, the shell is served, as fetch would (lifeextension.com's search is such a page: nothing renders it).
|
|
35
|
-
* **Bright Data's 407 is reported:** a token's IP allowlist refusing this host used to surface as a bare "fetch failed" (Node's fetch treats a 407 as a network error); the unlocker now requests over node:https and the log alerts `!! brightdata: unlocker HTTP 407: You are not allowed to access our API via this IP (…)`.
|
|
36
|
-
* TUI v74: a `url` row (web pages, failed fetches); fetch log lines coloured — `cached` green, rung outcomes green/red; `/status` counts `fetches` and `cached`; `(u)pdate` also rescans the library.
|
|
37
|
-
* README: `library_version` returns the package "version", not "library_version" (one more word the prefix rename caught).
|
|
38
|
-
|
|
39
|
-
### v0.2.5 — 2026-11-08
|
|
40
|
-
|
|
41
|
-
* **Tool descriptions** (issue #30): `library_info` names its key `verify`, not `library_verify`; `library_status` answers for one ID of any kind, not "one paper ID"; `library_verify`'s `claim` asks for a complete statement built from the review's words, not a verbatim fragment.
|
|
42
|
-
|
|
43
|
-
### v0.2.4 — 2026-11-08
|
|
44
|
-
|
|
45
|
-
* **Fix: `library_status` on a stored entry and `library_info` failed** with "library_status is not defined" / "library_info is not defined" in 0.2.2 and 0.2.3: the rename that restored the `library_` prefix also renamed two local variables. `library_version`'s log line printed "undefined" for the same reason. The Playwright download rung lacked a `readFileSync` import. Prose in the server instructions and READMEs that had "read" or "status" turned into tool names is plain English again.
|
|
46
|
-
|
|
47
|
-
### v0.2.3 — 2026-11-08
|
|
48
|
-
|
|
49
|
-
* **Every key is an `FH_LIBRARY_*` variable:** `FH_LIBRARY_NCBI_API_KEY`, `FH_LIBRARY_BRIGHTDATA_API_TOKEN`, `FH_LIBRARY_BRIGHTDATA_ZONE` replace the bare `NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE` the library used to share with other MCP servers. Hard cut: a deployment that still runs those servers keeps the bare lines for them and adds the prefixed ones for the library. The direction: fetching and PubMed move into the library, the separate servers go, and the double configuration with them.
|
|
50
|
-
|
|
51
|
-
### v0.2.2 — 2026-11-08
|
|
52
|
-
|
|
53
|
-
* **Tool names carry the `library_` prefix again**, all of them: `library_add`, `library_status`, `library_read`, `library_verify`, `library_info`, `library_version` (0.2.0 and 0.2.1 had the bare names, and `version`). A model that sees many servers' tools should never have to guess what `add` or `read` belongs to. Client permissions and prompts name them with the prefix.
|
|
54
|
-
* **Browser rungs by kind:** `FH_LIBRARY_PLAYWRIGHT_1`, `_2`, … (Playwright servers, port 8932) and `FH_LIBRARY_CHROME_1`, `_2`, … (a Chrome with `--remote-debugging-port`, port 9222) replace `FH_LIBRARY_BROWSER_1..3`. Values are hostnames (host or host:port); the kind is the variable's name, the protocol the library's. `FH_LIBRARY_PLAYWRIGHT_HEADLESS` (was `_BROWSER_HEADLESS`) belongs to the Playwright family; a Chrome rung is whatever it is. Order is fixed: Playwright servers, then Chrome rungs. The in-process "local" Chromium rung is gone — the library never launches a browser. `library_info` splits `download` into `playwright: {version, servers}` and `chrome`. Hard cut: every `.env` must move its browser lines.
|
|
55
|
-
|
|
56
|
-
### v0.2.0 — 2026-11-08
|
|
57
|
-
|
|
58
|
-
* **Renamed to FH Library.** The npm packages are `fh-library` (public) and `fh-library-dev` (dev), the MCP Registry entry `io.github.forever-healthy/fh-library`, the repos `forever-healthy/fh-library` and `fh-library-dev`; the server announces itself as `fh-library`, and client configs use the keys `fh-library`, `fh-library-dev`, `fh-library-local` (the `d-` prefix is for MCP servers Forever Healthy did not write). The `ai4l-library` packages are deprecated and point here.
|
|
59
|
-
* **Tools lose the `library_` prefix:** `add`, `status`, `read`, `verify`, `info`, `version` (was `get_version`). The server name namespaces them in every client.
|
|
60
|
-
* **Environment variables are `FH_LIBRARY_*`** (was `AI4L_LIBRARY_*`), hard cut: every `.env` must follow. Keys other servers share (`NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE`) are unchanged. The browser helper is `fh-chrome` (service, window class, `~/.fh-chrome` profile), the browser server container `fh-browser`.
|
|
61
|
-
* Entries in the changelog below this line use the old names; they describe the versions that carried them.
|
|
62
|
-
|
|
63
|
-
### v0.1.12 — 2026-11-07
|
|
64
|
-
|
|
65
|
-
* Server instructions after the AI4L review (#21): calls may run in parallel — add, read and verify — and two calls on the same missing record share one add (said once, under Hints, instead of "reads may run in parallel"); `corrections` is named as the status's field; the stale `config.json` clause gives way to the proxy URL template from the configuration.
|
|
66
|
-
|
|
67
|
-
### v0.1.11 — 2026-11-07
|
|
68
|
-
|
|
69
|
-
* `library_verify`: the verdict `na` is `unverifiable` again, now only for a paper whose abstract is stored and does not state the claim (the full text may). A title-only record — no abstract text, no full text — is a `fail`: nothing to read can back a claim; no model call, `score` null. Both carry `reason`, one fixed sentence each; `reason` is null otherwise. Audit tables map `unverifiable` where they mapped `na`. The log line pads the verdict to 12 characters.
|
|
70
|
-
* A record with nothing to read is not citable: `citable` false with `comment` "no abstract and no full text: nothing to verify" (retraction still comes first). `library_info.not_citable` counts them. A full text dropped into the inbox makes the record citable again.
|
|
71
|
-
* The server instructions rewritten as one short bullet document (3.9 K chars instead of 11 K): IDs, tools, rules, verify, read, status fields, sources. New guidance from the dev fleet run: no add or status before read or verify; one source, only what that source reports (a two-intervention finding cited to two papers is two claims; a statement about the literature becomes the figure this paper gives); what to do after each verdict (`fail` → read with a query, then reword or drop; `na` → "abstract only", no retry); `partial` → replace the `missing_terms`, keep the rest; a wrong ID is not retried. The `library_read` and `library_verify` descriptions shrunk to what the tool and its parameters do; the guidance lives in the instructions once.
|
|
72
|
-
|
|
73
|
-
### v0.1.10 — 2026-11-07
|
|
74
|
-
|
|
75
|
-
* `library_read` and `library_verify` on a paper not in the library add it first, as `library_add` would (its lines in the log), then read or judge it; the read answer carries `added: true`, and a paper the add cannot build is the error. Parallel reads of one missing paper share a single add.
|
|
76
|
-
* The `query` parameter says how to search: two or three distinctive terms the paper itself would use — a measure and a unit, an outcome and a group, a compound name — not the claim. The creators were sending 5–8 keyword dumps, which no paragraph holds whole.
|
|
77
|
-
|
|
78
|
-
### v0.1.9 — 2026-11-07
|
|
79
|
-
|
|
80
|
-
* `library_info` shows the Playwright version the package pins as `download.playwright`; the fleet TUI's `/status` carries the same as `playwright`. A browser server of another version is logged in words as an alert — `!! browser hetzner: Playwright version mismatch — server ws://hetzner:8932 must run playwright-core 1.65.0` — instead of the raw `428 Precondition Required` (#19).
|
|
81
|
-
|
|
82
|
-
### v0.1.8 — 2026-11-07
|
|
83
|
-
|
|
84
|
-
* `playwright-core` pinned to 1.65.0 (was `^1.58.1`), and `browser/Dockerfile` moved to the same version. A Playwright server refuses a client of another version with `428 Precondition Required`; a fresh install of the range had drifted to 1.65.0 while the browser servers ran 1.58.1, so the browser rungs never ran on the evipedia fleet. Package and servers must carry the same exact version.
|
|
85
|
-
|
|
86
|
-
### v0.1.7 — 2026-11-07
|
|
87
|
-
|
|
88
|
-
`library_read`, from watching the creators use it (#18):
|
|
89
|
-
|
|
90
|
-
* `query` answers windows, not paragraphs: each passage is at most 1000 chars around the matches (`clipped: true` when cut; a table row comes with its header row). A PDF-sourced section used to come back whole — up to 468 KB in one passage.
|
|
91
|
-
* When no passage holds every term, the best partial matches come back with `partial: true` and the `matched` terms per passage, the rare term ranked first; `missing_terms` names the terms the text never uses in any form.
|
|
92
|
-
* Matching forgives case, accents, hyphen/slash, Markdown markup (`t~max~` is tmax), plural and -ed/-ing/-ation forms.
|
|
93
|
-
* `section` and `query` together search inside that section.
|
|
94
|
-
* A parent heading answers its own text plus its `subsections` (heading, words) instead of the subtree cut at `max_chars`; a cut section reports `returned_words` against `words`.
|
|
95
|
-
* A heading the text uses several times is listed as `Participants [3/12]` and named that way; the bare name lists the candidates.
|
|
96
|
-
* `section "Abstract"` on a full text without one answers from the stored abstract; `section "Declarations"` on a text without one assembles the funding / conflict / acknowledgement / ethics sections.
|
|
97
|
-
* PDF → Markdown: paragraphs from the page layout (extra leading, a first-line indent, a bullet, a change of type size, a short line ending a sentence), across columns and pages — not only from blank lines, which journal PDFs rarely have. Cochrane's section names are recognised as headings. Stored PDF-sourced texts keep their old structure until re-added.
|
|
98
|
-
* JATS: a blank inside `<sub>`/`<sup>` moves outside the markers (`t~max~ around`, not `t~max ~around`).
|
|
99
|
-
|
|
100
|
-
### v0.1.6 — 2026-11-07
|
|
101
|
-
|
|
102
|
-
* No change to the server. The fleet TUI (dev repo, not in the package) pulls its checkout with each rescan (`AI4L_LIBRARY_TUI_PULL=1`) instead of on its own timer.
|
|
103
|
-
|
|
104
|
-
### v0.1.5 — 2026-11-07
|
|
105
|
-
|
|
106
|
-
* `library.log`: a message starting with `!!` is an alert — a service refused the deployment itself (Bright Data's reason from its `x-brd-err-msg` header, e.g. a token not allowed from this IP; a verify model failure; an unknown tier name at startup).
|
|
107
|
-
* Fixed: `AI4L_LIBRARY_DIR` set in `.env` was ignored (the library directory was resolved before `.env` was loaded); only the environment worked.
|
|
108
|
-
|
|
109
|
-
### v0.1.4 — 2026-11-07
|
|
110
|
-
|
|
111
|
-
* `.env` is the whole configuration: every value an environment variable, the tier set and the defaults built in. `library/config.json` and `AI4L_LIBRARY_CONFIG` are gone, and with them `${NAME}` references and the per-tier `description`.
|
|
112
|
-
* Renamed, no aliases — `CONTACT_EMAIL` → `AI4L_LIBRARY_CONTACT_EMAIL`; `ELSEVIER_API_KEY` → `AI4L_LIBRARY_ELSEVIER_API_KEY`; `WILEY_TDM_TOKEN` → `AI4L_LIBRARY_WILEY_TDM_TOKEN`; `CORE_API_KEY` → `AI4L_LIBRARY_CORE_API_KEY`; `PERPLEXITY_URL` / `PERPLEXITY_API_KEY` / `PERPLEXITY_MODEL` → `AI4L_LIBRARY_VERIFY_URL` / `_API_KEY` / `_MODEL`; config.json's `threshold`, `timeout_seconds`, `retry_after_days`, `retry_blocked`, `refresh_after_days`, `download.headless` → `AI4L_LIBRARY_VERIFY_THRESHOLD`, `AI4L_LIBRARY_VERIFY_TIMEOUT_SECS`, `AI4L_LIBRARY_RETRY_AFTER_DAYS`, `AI4L_LIBRARY_RETRY_BLOCKED`, `AI4L_LIBRARY_REFRESH_AFTER_DAYS`, `AI4L_LIBRARY_BROWSER_HEADLESS`. `NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE` and `AI4L_LIBRARY_BROWSER_n` unchanged.
|
|
113
|
-
* `AI4L_LIBRARY_TIERS` is the whole ladder: the listed tiers in exactly that order, a tier not listed is off (an unlisted tier no longer runs after the listed ones).
|
|
114
|
-
* One proxy tier, `proxy`, instead of url tiers: `AI4L_LIBRARY_PROXY_URL` (template with `{doi}` `{pmid}` `{pmcid}`) and `AI4L_LIBRARY_PROXY_HEADERS`; `PROXY_TOKEN` and the `my-proxy` example are gone.
|
|
115
|
-
* `library_info`: `config` replaces `tiers_config` — the cascade's settings, problems with `AI4L_LIBRARY_TIERS`, and per variable where its value came from (environment, `.env`, default), never the value; `tiers` in run order without descriptions (the README describes them).
|
|
116
|
-
* The container's working directory is `/library`; the configuration arrives as its environment.
|
|
117
|
-
|
|
118
|
-
### v0.1.3 — 2026-11-07
|
|
119
|
-
|
|
120
|
-
* IDs: `pmid:` (bare digits count as one), `doi:` for papers PubMed does not index (a DOI PubMed indexes is stored under its PMID and copied under the DOI, so every tool takes either), `dailymed:<set ID>` drug labels, `nct:` trial records.
|
|
121
|
-
* `library_add(id)`: full text through a configurable cascade (`config.json`: inbox, pmc, europepmc, tdm, unpaywall, openalex, core, url tiers) and a download ladder (plain fetch, browser rungs, Bright Data); every text checked (title, methods and results, no wall page) before it counts; failed tiers recorded in `attempts`, transient ones retried after `retry_after_days`; PubMed record re-read after `refresh_after_days` for correction notices and retractions (`citable`, `corrections`).
|
|
122
|
-
* `library_read(id, section?, query?, max_chars?, max_hits?)`: the outline, one section, or the paragraphs matching a query — never the whole paper.
|
|
123
|
-
* `library_verify(claim, id)`: is one finding stated by one source, judged on the stored text by the Perplexity Decisions API; `superseded` when a stored correction replaced the figure; `glyphs_unresolved` for symbols a PDF font gave no character for.
|
|
124
|
-
* `library_info(ids?)`: the library's directory, tiers, download ladder, verify model health, entries by type and level.
|
|
125
|
-
* PDF conversion: symbols restored from the font's glyph names (minus, ±, ≤, Greek letters), words broken at line ends joined back.
|
|
126
|
-
* `library.log`: one line per action, host-prefixed.
|
|
127
|
-
* Environment variables renamed `LIBRARY_*` → `AI4L_LIBRARY_*` (`AI4L_LIBRARY_DIR`, `AI4L_LIBRARY_CONFIG`, `AI4L_LIBRARY_ENV`, `AI4L_LIBRARY_TIERS`, `AI4L_LIBRARY_BROWSER_n`, `AI4L_LIBRARY_TUI_STATUS_SECS`); the old names are not read.
|
|
128
|
-
* `.env` is also looked for in the library directory when neither `AI4L_LIBRARY_ENV` nor `./.env` names one.
|
|
129
|
-
* Entry files are written to a temporary name and renamed into place (several servers may share one library); `library.log` rotates to `library.log.1` past 50 MB.
|
|
130
|
-
|
|
131
|
-
### v0.1.1 — 2026-10-26
|
|
132
|
-
|
|
133
|
-
* `library_add(id)` and `library_status(id)` for `pmid:` IDs.
|
|
134
|
-
|
|
135
|
-
### v0.0.1 — 2026-10-26
|
|
136
|
-
|
|
137
|
-
* Project skeleton.
|
|
5
|
+
* **`library_fetch` answers record URLs from the library:** a PubMed, ClinicalTrials.gov, DailyMed or doi.org link is answered from the library entry (a missing PubMed record is stored at abstract level in one request, no full-text cascade; a DOI PubMed indexes goes through its PMID) in fetch's format — the abstract record for PubMed, the trial record, the label, the full text or abstract for a DOI; `title_only` the title line. No script shell, no browser.
|
package/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-

|
|
2
2
|

|
|
3
3
|
[_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
|
|
4
4
|
[](https://www.npmjs.com/package/fh-library)
|
|
@@ -7,7 +7,9 @@
|
|
|
7
7
|
|
|
8
8
|

|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
# FH Library MCP
|
|
11
|
+
|
|
12
|
+
MCP server that builds a local library of the scientific papers an evidence review cites, part of [AI4L](https://github.com/forever-healthy/AI4L)
|
|
11
13
|
|
|
12
14
|
A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a local library of the scientific papers an evidence review cites. Given a paper ID (`type:value`, e.g. `pmid:30955454`), it fetches the paper's metadata, abstract and — where obtainable — full text through a configurable cascade of retrieval services (PubMed, PMC, Europe PMC, Unpaywall, OpenAlex, …), converts everything to Markdown, and stores it under a mounted `library/` directory. Nothing flows through the model's context: the server writes to disk itself and returns only a small status per call.
|
|
13
15
|
|
|
@@ -73,8 +75,9 @@ FH_LIBRARY_PROXY_HEADERS=Authorization: Bearer <token>
|
|
|
73
75
|
```
|
|
74
76
|
|
|
75
77
|
* The URL is a template. Placeholders: `{doi}` (`10.1056/NEJMoa042274`), `{pmid}` (`15516696`), `{pmcid}` (`PMC6728809`, with its prefix). Each is URL-encoded when filled — the slash in a DOI becomes `%2F` — so put a placeholder in a path segment or a query value. A template may use several. The tier applies only to a paper for which every placeholder it uses is known; otherwise it is skipped as `no doi` (or `no pmcid`, …), a permanent reason. A template without any placeholder never runs (`template has no placeholder`).
|
|
78
|
+
|
|
76
79
|
* The headers are optional request headers for credentials (`Authorization`, an API-key header, a cookie), as `Name: value; Name: value`.
|
|
77
|
-
|
|
80
|
+
|
|
78
81
|
|
|
79
82
|
**What the proxy tier expects back.** The tier sends one GET to the filled template, with a desktop Chrome User-Agent and the headers; redirects are followed. The service answers the paper as one of two formats, read from the response bytes, not the URL or the Content-Type:
|
|
80
83
|
|
|
@@ -151,6 +154,8 @@ PubMed's book records — National Academies reports, NTP monographs, StatPearls
|
|
|
151
154
|
|
|
152
155
|
**PubMed in batches.** `library_pubmed_fetch_articles(pmids)` answers what an auditor asks of a list of PubMed IDs — does each resolve, what is its title, year and type, is it retracted, does the library hold its full text — as rows in `columns` order, about 120 characters per paper. A PMID already in the library is read from disk. The others go to PubMed in one efetch request (up to 200), and each record is stored exactly as `library_add` stores its first stage: the XML, the abstract Markdown, the meta with its correction and retraction notices, no retrieval attempts yet — so a later `library_add` or `library_read` runs the full-text cascade as on an untried entry, and the library fills at abstract level in batches instead of one paper at a time. An entry is checked again right before writing, so a concurrent `library_add` that already stored a full text is never overwritten. Level `not found` marks a PMID PubMed does not know, `invalid` a value that is not a PMID. `library_pubmed_search_articles(query, max_results?, sort?, years?)` is the search: the query goes to PubMed as written, field tags and all; esearch gives the total and the IDs, one esummary the rows — PMID, title, first author, year, journal, publication types (Journal Article left out), PMCID when a free full text exists, and the library's level when the paper is already stored, so the model knows what `library_read` can answer at once. Cached under `search/` like the trial search.
|
|
153
156
|
|
|
157
|
+
**Record URLs.** `library_fetch` answers a PubMed, ClinicalTrials.gov, DailyMed or doi.org link from the library instead of the web page, in fetch's format: the abstract record for a PubMed link, the trial record, the label, the full text or abstract for a DOI; `title_only` the title line. A missing PubMed record is stored at abstract level in one request, no full-text cascade (a DOI PubMed indexes goes through its PMID); a trial or label is added as `library_add` adds it. PubMed's own page is a script shell a browser must render; the record answers in half a second, and in milliseconds once stored.
|
|
158
|
+
|
|
154
159
|
**Trial search.** `library_clinicaltrials_search_studies(query?, intervention?, condition?, status?, sort?, max_results?)` asks the ClinicalTrials.gov API (v2, no key) what a review needs from the registry: the trials of an intervention, the ongoing ones (`status: "ongoing"` is recruiting, not yet recruiting, active not recruiting, enrolling by invitation), a free-text or condition search, sorted by relevance, newest, largest or completion date, up to 100 rows. The rows are arrays in the order of `columns` — the field names once, not per row — shaped for the review's Emerging Research item (what AI4L's prompt asks about an ongoing trial; the link is `https://clinicaltrials.gov/study/<NCT ID>`): NCT ID, title, status with why a trial stopped, phase, enrollment as "100 (estimated)", start and primary completion dates (whether "ongoing" still holds), the primary outcome with its time frame, conditions, interventions, `has_results`; the full detail of one row is `library_clinicaltrials_get_study_record`; `total` is the registry's count, for a "no trials found" statement. The same search is answered from `search/<2>/<4>/<key>.json` for `FH_LIBRARY_SEARCH_CACHE_DAYS` (7) — a registry listing a week old is current enough for a review, and an audit that runs again reads from disk. Two thin tools answer from the stored record of any row (added first when missing): `library_clinicaltrials_get_study_record(nct)` is the review's detail of one trial — official title, design, enrollment, dates, eligibility, sites, result publications as `pmid:` IDs, the brief summary — and `library_clinicaltrials_get_study_results(nct, max_chars?)` is the posted results part of the stored Markdown (participant flow, baseline, outcome measures, adverse events), with the outcome titles listed so `library_read("nct:…", section)` can return one measure whole. Together with the search they cover what the AI4L models asked the ClinicalTrials.gov MCP server for.
|
|
155
160
|
|
|
156
161
|
**Web pages.** `library_fetch(url, max_length?, start_index?, raw?)` is the standard MCP [fetch](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) tool, served from the library, for every link a review carries that is not a paper, label or trial record: a guideline, a product page, a news item. Its description tells the model to prefer it over a plain fetch or a browser tool, and says why: a page anyone fetched in the last 30 days comes from disk in milliseconds, and a page a plain fetch cannot get climbs the ladder in the same call. The answer is what `fetch` returns, so a model keeps its habits: `Contents of <url>:` and the page as Markdown with the page's title as the first heading, cut at `max_length` (default 5000) with the note `Content truncated. Call the fetch tool with a start_index of N to get more content`; `raw` gives the HTML; a `text/markdown` body comes back as is, any other non-HTML body with the `Content type … cannot be simplified to markdown` prefix; a failure is `Failed to fetch <url> - status code 404` or `Failed to fetch <url>: <reason>`. Two things happen underneath. The page is fetched up the same ladder as a paper — the plain fetch, then the browser rungs on a 403, 429 or bot wall, then Bright Data — so a page a plain fetch cannot reach is still read. And it is stored: `url/<2>/<4>/<16 hex of the URL's SHA-1>/url-<hash>-{meta.md, page.md, page.html|pdf|txt}`, the URL normalised first (host lowercased, fragment and tracking parameters dropped). The same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS` (1) — an audit that runs twice over one review reads its links once. A PDF comes back as Markdown (`fetch` would return its bytes as text). `title_only` returns `Contents of <url>:` and the title line alone — the page is still fetched and stored in full — so a link check costs a few tokens instead of a page. The log says which — `fetch example.com/… cached html … https://example.com/drugs/x` or `fetched 02.3s html retrieved_by=browser hetzner … <url>` (the ID column holds the URL without scheme, cut to a PubMed ID's width; the full URL ends the line); `library_info` counts the pages under `url`, a failed fetch as level `title`.
|
package/dist/index.js
CHANGED
|
@@ -1480,41 +1480,6 @@ function europepmcCandidates(r, pmcid) {
|
|
|
1480
1480
|
return out;
|
|
1481
1481
|
}
|
|
1482
1482
|
|
|
1483
|
-
// src/tiers/inbox.ts
|
|
1484
|
-
import { existsSync as existsSync4, readdirSync, readFileSync as readFileSync3, unlinkSync } from "fs";
|
|
1485
|
-
import { join as join4 } from "path";
|
|
1486
|
-
var INBOX_DIR = join4(FH_LIBRARY_DIR, "inbox");
|
|
1487
|
-
var EXT = { ".pdf": "application/pdf", ".xml": "text/xml", ".html": "text/html", ".htm": "text/html" };
|
|
1488
|
-
function findInboxFile(dir, pmid) {
|
|
1489
|
-
if (!existsSync4(dir)) return null;
|
|
1490
|
-
const re = new RegExp(`(^|[^0-9])${pmid}([^0-9]|$)`);
|
|
1491
|
-
for (const name of readdirSync(dir).sort()) {
|
|
1492
|
-
const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
|
|
1493
|
-
if (ext in EXT && re.test(name.slice(0, -ext.length))) return join4(dir, name);
|
|
1494
|
-
}
|
|
1495
|
-
return null;
|
|
1496
|
-
}
|
|
1497
|
-
function inboxTier(dir = INBOX_DIR) {
|
|
1498
|
-
return {
|
|
1499
|
-
name: "inbox",
|
|
1500
|
-
always: true,
|
|
1501
|
-
applies: (p) => p.pmid ? true : "no pmid",
|
|
1502
|
-
async run(p, ctx) {
|
|
1503
|
-
const file2 = findInboxFile(dir, p.pmid);
|
|
1504
|
-
if (!file2) return { ...miss("inbox", "no file"), skipped: true, ephemeral: true };
|
|
1505
|
-
const name = file2.slice(file2.lastIndexOf("/") + 1);
|
|
1506
|
-
const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
|
|
1507
|
-
const bytes = readFileSync3(file2);
|
|
1508
|
-
const r = await convertBytes(p, bytes, EXT[ext]);
|
|
1509
|
-
if (!r.ok) return miss("inbox", `${name}: ${r.reason}`);
|
|
1510
|
-
ctx.onTry?.(`${name} ok, removed from the inbox`);
|
|
1511
|
-
unlinkSync(file2);
|
|
1512
|
-
return { ok: true, tier: "inbox", format: r.format, raw: r.format === "pdf" ? bytes : bytes.toString("utf8"), md: r.md, version: "published" };
|
|
1513
|
-
}
|
|
1514
|
-
};
|
|
1515
|
-
}
|
|
1516
|
-
var inbox = inboxTier();
|
|
1517
|
-
|
|
1518
1483
|
// src/tiers/openalex.ts
|
|
1519
1484
|
var openalex = {
|
|
1520
1485
|
name: "openalex",
|
|
@@ -1675,7 +1640,6 @@ function unpaywallCandidates(d) {
|
|
|
1675
1640
|
|
|
1676
1641
|
// src/tiers/index.ts
|
|
1677
1642
|
var BUILTIN = [
|
|
1678
|
-
{ name: "inbox", make: () => inbox },
|
|
1679
1643
|
{ name: "pmc", make: () => pmc },
|
|
1680
1644
|
{ name: "europepmc", make: () => europepmc },
|
|
1681
1645
|
{ name: "tdm", make: (o) => tdmTier(o.tokens ?? {}) },
|
|
@@ -1774,9 +1738,9 @@ function resolveTiers(env = process.env) {
|
|
|
1774
1738
|
// src/log.ts
|
|
1775
1739
|
import { appendFileSync, mkdirSync as mkdirSync2, renameSync, statSync } from "fs";
|
|
1776
1740
|
import { hostname } from "os";
|
|
1777
|
-
import { join as
|
|
1741
|
+
import { join as join4 } from "path";
|
|
1778
1742
|
var HOST = hostname();
|
|
1779
|
-
var LOG_FILE =
|
|
1743
|
+
var LOG_FILE = join4(FH_LIBRARY_DIR, "library.log");
|
|
1780
1744
|
function stamp() {
|
|
1781
1745
|
const d = /* @__PURE__ */ new Date();
|
|
1782
1746
|
const p = (n) => String(n).padStart(2, "0");
|
|
@@ -1811,12 +1775,12 @@ function log(tool, id, message) {
|
|
|
1811
1775
|
}
|
|
1812
1776
|
|
|
1813
1777
|
// src/stats.ts
|
|
1814
|
-
import { mkdirSync as mkdirSync4, readdirSync as
|
|
1815
|
-
import { join as
|
|
1778
|
+
import { mkdirSync as mkdirSync4, readdirSync as readdirSync2, readFileSync as readFileSync4, statSync as statSync2, unlinkSync as unlinkSync2 } from "fs";
|
|
1779
|
+
import { join as join6 } from "path";
|
|
1816
1780
|
|
|
1817
1781
|
// src/entry.ts
|
|
1818
|
-
import { copyFileSync, existsSync as
|
|
1819
|
-
import { join as
|
|
1782
|
+
import { copyFileSync, existsSync as existsSync4, mkdirSync as mkdirSync3, readdirSync, readFileSync as readFileSync3, renameSync as renameSync2, rmSync, unlinkSync, writeFileSync } from "fs";
|
|
1783
|
+
import { join as join5 } from "path";
|
|
1820
1784
|
import { basename, dirname } from "path";
|
|
1821
1785
|
|
|
1822
1786
|
// src/pubmed.ts
|
|
@@ -2440,7 +2404,7 @@ ${body}
|
|
|
2440
2404
|
}
|
|
2441
2405
|
function readHeadLine(p) {
|
|
2442
2406
|
try {
|
|
2443
|
-
const lines =
|
|
2407
|
+
const lines = readFileSync3(p.md, "utf8").split("\n").filter((l) => l.trim());
|
|
2444
2408
|
return lines[1] ?? "";
|
|
2445
2409
|
} catch {
|
|
2446
2410
|
return "";
|
|
@@ -2562,28 +2526,28 @@ function splitPublicationTypes(v) {
|
|
|
2562
2526
|
}
|
|
2563
2527
|
function abstractMissing(p) {
|
|
2564
2528
|
try {
|
|
2565
|
-
return
|
|
2529
|
+
return readFileSync3(p.md, "utf8").includes("_No abstract available._");
|
|
2566
2530
|
} catch {
|
|
2567
2531
|
return false;
|
|
2568
2532
|
}
|
|
2569
2533
|
}
|
|
2570
2534
|
function correctionsFromXml(p) {
|
|
2571
2535
|
try {
|
|
2572
|
-
return parsePubmedXml(
|
|
2536
|
+
return parsePubmedXml(readFileSync3(p.xml, "utf8"))?.corrections ?? [];
|
|
2573
2537
|
} catch {
|
|
2574
2538
|
return [];
|
|
2575
2539
|
}
|
|
2576
2540
|
}
|
|
2577
2541
|
function declarationsFromXml(p) {
|
|
2578
2542
|
try {
|
|
2579
|
-
const r = parsePubmedXml(
|
|
2543
|
+
const r = parsePubmedXml(readFileSync3(p.xml, "utf8"));
|
|
2580
2544
|
return { coi: r?.coi ?? null, funding: r?.funding ?? [] };
|
|
2581
2545
|
} catch {
|
|
2582
2546
|
return { coi: null, funding: [] };
|
|
2583
2547
|
}
|
|
2584
2548
|
}
|
|
2585
2549
|
function readMeta(p) {
|
|
2586
|
-
const { fm, all } = parseFrontmatter(
|
|
2550
|
+
const { fm, all } = parseFrontmatter(readFileSync3(p.meta, "utf8"));
|
|
2587
2551
|
let title = fm.title ?? "";
|
|
2588
2552
|
if (title.startsWith('"')) {
|
|
2589
2553
|
try {
|
|
@@ -2639,7 +2603,7 @@ function writeAtomic(file2, data) {
|
|
|
2639
2603
|
function writeMeta(p, m) {
|
|
2640
2604
|
writeAtomic(p.meta, renderMeta(m));
|
|
2641
2605
|
try {
|
|
2642
|
-
|
|
2606
|
+
unlinkSync(join5(p.entryDir, `${p.entry}-missing.md`));
|
|
2643
2607
|
} catch {
|
|
2644
2608
|
}
|
|
2645
2609
|
}
|
|
@@ -2648,12 +2612,12 @@ function copyEntry(from, to) {
|
|
|
2648
2612
|
rmSync(tmp, { recursive: true, force: true });
|
|
2649
2613
|
mkdirSync3(tmp, { recursive: true });
|
|
2650
2614
|
const rename = (n) => n.startsWith(from.entry) ? to.entry + n.slice(from.entry.length) : n;
|
|
2651
|
-
for (const n of
|
|
2615
|
+
for (const n of readdirSync(from.entryDir)) if (join5(from.entryDir, n) !== from.meta && !n.endsWith(".tmp")) copyFileSync(join5(from.entryDir, n), join5(tmp, rename(n)));
|
|
2652
2616
|
const m = readMeta(from);
|
|
2653
2617
|
const copy = { ...m, files: m.files.map(rename) };
|
|
2654
|
-
writeFileSync(
|
|
2618
|
+
writeFileSync(join5(tmp, basename(to.meta)), renderMeta(copy));
|
|
2655
2619
|
mkdirSync3(dirname(to.entryDir), { recursive: true });
|
|
2656
|
-
if (
|
|
2620
|
+
if (existsSync4(to.entryDir)) renameSync2(to.entryDir, old);
|
|
2657
2621
|
renameSync2(tmp, to.entryDir);
|
|
2658
2622
|
rmSync(old, { recursive: true, force: true });
|
|
2659
2623
|
return copy;
|
|
@@ -2675,7 +2639,7 @@ function citability(m) {
|
|
|
2675
2639
|
return { citable: true, comment: notes.length ? notes.join("; ") : null };
|
|
2676
2640
|
}
|
|
2677
2641
|
function statusFromMeta(p, m, base) {
|
|
2678
|
-
const full =
|
|
2642
|
+
const full = existsSync4(p.fulltextMd) ? readFileSync3(p.fulltextMd, "utf8") : null;
|
|
2679
2643
|
const cit = citability(m);
|
|
2680
2644
|
if (full?.includes(`
|
|
2681
2645
|
${CORRECTED_HEADING}
|
|
@@ -2717,7 +2681,7 @@ function readStatus(p, base) {
|
|
|
2717
2681
|
return statusFromMeta(p, readMeta(p), base);
|
|
2718
2682
|
}
|
|
2719
2683
|
function entryExists(p) {
|
|
2720
|
-
return
|
|
2684
|
+
return existsSync4(p.meta);
|
|
2721
2685
|
}
|
|
2722
2686
|
function writeEntry(p, r, xml, converter) {
|
|
2723
2687
|
mkdirSync3(p.entryDir, { recursive: true });
|
|
@@ -2800,15 +2764,15 @@ ${sentences.join("\n\n")}
|
|
|
2800
2764
|
`;
|
|
2801
2765
|
}
|
|
2802
2766
|
function syncCorrectedBlock(p) {
|
|
2803
|
-
if (!
|
|
2767
|
+
if (!existsSync4(p.fulltextMd) || !existsSync4(p.xml)) return;
|
|
2804
2768
|
let r;
|
|
2805
2769
|
try {
|
|
2806
|
-
r = parsePubmedXml(
|
|
2770
|
+
r = parsePubmedXml(readFileSync3(p.xml, "utf8"));
|
|
2807
2771
|
} catch {
|
|
2808
2772
|
return;
|
|
2809
2773
|
}
|
|
2810
2774
|
if (!r) return;
|
|
2811
|
-
const md =
|
|
2775
|
+
const md = readFileSync3(p.fulltextMd, "utf8");
|
|
2812
2776
|
const next = withCorrectedBlock(md, correctedSentences(r));
|
|
2813
2777
|
if (next !== md) writeAtomic(p.fulltextMd, next);
|
|
2814
2778
|
}
|
|
@@ -2924,13 +2888,13 @@ var stats = null;
|
|
|
2924
2888
|
var file = null;
|
|
2925
2889
|
function startStats(pkg2, libraryDir = FH_LIBRARY_DIR) {
|
|
2926
2890
|
stats = { host: HOST, pid: process.pid, package: pkg2.name, version: pkg2.version, started: (/* @__PURE__ */ new Date()).toISOString(), verify: { hits: 0, misses: 0 }, fetch: { hits: 0, misses: 0 }, search: { hits: 0, misses: 0 }, pubmed: { hits: 0, misses: 0 } };
|
|
2927
|
-
file =
|
|
2928
|
-
sweep(
|
|
2891
|
+
file = join6(libraryDir, ".servers", `${HOST}-${process.pid}.json`);
|
|
2892
|
+
sweep(join6(libraryDir, ".servers"));
|
|
2929
2893
|
publish();
|
|
2930
2894
|
setInterval(publish, HEARTBEAT_MS).unref();
|
|
2931
2895
|
const stop = () => {
|
|
2932
2896
|
try {
|
|
2933
|
-
if (file)
|
|
2897
|
+
if (file) unlinkSync2(file);
|
|
2934
2898
|
} catch {
|
|
2935
2899
|
}
|
|
2936
2900
|
};
|
|
@@ -2942,16 +2906,16 @@ function sweep(dir, staleMs = STALE_MS) {
|
|
|
2942
2906
|
let n = 0;
|
|
2943
2907
|
let names;
|
|
2944
2908
|
try {
|
|
2945
|
-
names =
|
|
2909
|
+
names = readdirSync2(dir);
|
|
2946
2910
|
} catch {
|
|
2947
2911
|
return 0;
|
|
2948
2912
|
}
|
|
2949
2913
|
for (const name of names) {
|
|
2950
2914
|
if (!name.endsWith(".json")) continue;
|
|
2951
|
-
const p =
|
|
2915
|
+
const p = join6(dir, name);
|
|
2952
2916
|
try {
|
|
2953
2917
|
if (Date.now() - statSync2(p).mtimeMs > staleMs) {
|
|
2954
|
-
|
|
2918
|
+
unlinkSync2(p);
|
|
2955
2919
|
n++;
|
|
2956
2920
|
}
|
|
2957
2921
|
} catch {
|
|
@@ -2968,14 +2932,14 @@ var currentStats = () => stats;
|
|
|
2968
2932
|
function publish() {
|
|
2969
2933
|
if (!stats || !file) return;
|
|
2970
2934
|
try {
|
|
2971
|
-
mkdirSync4(
|
|
2935
|
+
mkdirSync4(join6(file, ".."), { recursive: true });
|
|
2972
2936
|
writeAtomic(file, JSON.stringify(stats, null, 2) + "\n");
|
|
2973
2937
|
} catch {
|
|
2974
2938
|
}
|
|
2975
2939
|
}
|
|
2976
2940
|
|
|
2977
2941
|
// src/tools.ts
|
|
2978
|
-
import { existsSync as
|
|
2942
|
+
import { existsSync as existsSync12, readFileSync as readFileSync12 } from "fs";
|
|
2979
2943
|
import { z } from "zod";
|
|
2980
2944
|
|
|
2981
2945
|
// src/cascade.ts
|
|
@@ -3008,7 +2972,7 @@ async function fulltextCascade(p, attempts, onEvent, tiers = resolveTiers().tier
|
|
|
3008
2972
|
const misses = [];
|
|
3009
2973
|
const tried = /* @__PURE__ */ new Set();
|
|
3010
2974
|
for (const tier of tiers) {
|
|
3011
|
-
if (
|
|
2975
|
+
if (shouldSkip(attempts, tier.name, retryAfterDays, /* @__PURE__ */ new Date(), retryBlocked)) continue;
|
|
3012
2976
|
const applies = tier.applies(p);
|
|
3013
2977
|
const ctx = { tried, onTry: (message) => onEvent?.({ ok: "try", tier: tier.name, message }), onAlert: (message) => onEvent?.({ ok: "alert", tier: tier.name, message }) };
|
|
3014
2978
|
const r = applies === true ? await tier.run(p, ctx) : { ...miss(tier.name, applies), skipped: true };
|
|
@@ -3050,11 +3014,11 @@ function parseId(raw) {
|
|
|
3050
3014
|
}
|
|
3051
3015
|
|
|
3052
3016
|
// src/info.ts
|
|
3053
|
-
import { existsSync as
|
|
3054
|
-
import { dirname as dirname2, join as
|
|
3017
|
+
import { existsSync as existsSync6, readdirSync as readdirSync3, readFileSync as readFileSync6, statSync as statSync3 } from "fs";
|
|
3018
|
+
import { dirname as dirname2, join as join7 } from "path";
|
|
3055
3019
|
|
|
3056
3020
|
// src/verify.ts
|
|
3057
|
-
import { existsSync as
|
|
3021
|
+
import { existsSync as existsSync5, readFileSync as readFileSync5 } from "fs";
|
|
3058
3022
|
var DEFAULT_MODEL = "pplx-decider-flash-v1.2";
|
|
3059
3023
|
var QUESTION = "Does the paper state this, in substance? Ignore section labels and definitions of terms in parentheses. Every fact and number of the statement must be stated in the paper or follow from its numbers; a rounded number counts. If any part of the statement is absent from the paper or contradicted by it, answer false.";
|
|
3060
3024
|
function verifyConfig(env = process.env) {
|
|
@@ -3156,10 +3120,10 @@ async function readPaper(raw, libraryDir = FH_LIBRARY_DIR) {
|
|
|
3156
3120
|
if (p.error) return { id: p.requested, error: p.error };
|
|
3157
3121
|
const paths = entryPaths(libraryDir, p.type, p.id);
|
|
3158
3122
|
if (!entryExists(paths)) return { id: p.requested, error: "not in library" };
|
|
3159
|
-
const full =
|
|
3160
|
-
if (full) return { id: p.requested, basis: "full_text", text:
|
|
3123
|
+
const full = existsSync5(paths.fulltextMd);
|
|
3124
|
+
if (full) return { id: p.requested, basis: "full_text", text: readFileSync5(paths.fulltextMd, "utf8") };
|
|
3161
3125
|
const meta = readMeta(paths);
|
|
3162
|
-
return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text:
|
|
3126
|
+
return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text: readFileSync5(paths.md, "utf8") };
|
|
3163
3127
|
}
|
|
3164
3128
|
async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
|
|
3165
3129
|
const out = { claim, id, verdict: "error", reason: null, score: null, basis: null, windows: 0, glyphs_unresolved: 0, superseded: null, model: cfg?.model ?? null };
|
|
@@ -3200,9 +3164,9 @@ async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
|
|
|
3200
3164
|
|
|
3201
3165
|
// src/info.ts
|
|
3202
3166
|
function* metaFiles(dir) {
|
|
3203
|
-
if (!
|
|
3204
|
-
for (const name of
|
|
3205
|
-
const p =
|
|
3167
|
+
if (!existsSync6(dir)) return;
|
|
3168
|
+
for (const name of readdirSync3(dir)) {
|
|
3169
|
+
const p = join7(dir, name);
|
|
3206
3170
|
if (statSync3(p).isDirectory()) yield* metaFiles(p);
|
|
3207
3171
|
else if (name.endsWith("-meta.md")) yield p;
|
|
3208
3172
|
}
|
|
@@ -3241,11 +3205,11 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
|
|
|
3241
3205
|
updates: 0
|
|
3242
3206
|
};
|
|
3243
3207
|
const ids = [];
|
|
3244
|
-
for (const type of
|
|
3245
|
-
const typeDir =
|
|
3208
|
+
for (const type of existsSync6(libraryDir) ? readdirSync3(libraryDir) : []) {
|
|
3209
|
+
const typeDir = join7(libraryDir, type);
|
|
3246
3210
|
if (!statSync3(typeDir).isDirectory()) continue;
|
|
3247
3211
|
for (const meta of metaFiles(typeDir)) {
|
|
3248
|
-
const md =
|
|
3212
|
+
const md = readFileSync6(meta, "utf8");
|
|
3249
3213
|
const raw = field(md, "level") || "unknown";
|
|
3250
3214
|
const level = raw === "abstract" && field(md, "has_abstract") === "no" ? "title" : raw;
|
|
3251
3215
|
info.entries++;
|
|
@@ -3255,9 +3219,9 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
|
|
|
3255
3219
|
t.entries++;
|
|
3256
3220
|
if (level === "full_text" || level === "abstract" || level === "title") t[level]++;
|
|
3257
3221
|
const dir = dirname2(meta);
|
|
3258
|
-
for (const f of
|
|
3222
|
+
for (const f of readdirSync3(dir)) {
|
|
3259
3223
|
try {
|
|
3260
|
-
const st = statSync3(
|
|
3224
|
+
const st = statSync3(join7(dir, f));
|
|
3261
3225
|
if (st.isFile()) info.size_bytes += st.size;
|
|
3262
3226
|
} catch {
|
|
3263
3227
|
}
|
|
@@ -3286,7 +3250,7 @@ function noticeTypes(md, metaPath) {
|
|
|
3286
3250
|
return out;
|
|
3287
3251
|
}
|
|
3288
3252
|
try {
|
|
3289
|
-
const xml =
|
|
3253
|
+
const xml = readFileSync6(metaPath.replace(/-meta\.md$/, "-abstract.xml"), "utf8");
|
|
3290
3254
|
for (const m of xml.matchAll(/<CommentsCorrections RefType="([A-Za-z]+)"/g)) if (XML_TYPES[m[1]]) out.add(XML_TYPES[m[1]]);
|
|
3291
3255
|
} catch {
|
|
3292
3256
|
}
|
|
@@ -3665,8 +3629,8 @@ function parseCsl(json2) {
|
|
|
3665
3629
|
|
|
3666
3630
|
// src/urlfetch.ts
|
|
3667
3631
|
import { createHash as createHash2 } from "crypto";
|
|
3668
|
-
import { existsSync as
|
|
3669
|
-
import { join as
|
|
3632
|
+
import { existsSync as existsSync7, mkdirSync as mkdirSync5, readFileSync as readFileSync7 } from "fs";
|
|
3633
|
+
import { join as join8 } from "path";
|
|
3670
3634
|
var TRACKING = /^(utm_[a-z]+|fbclid|gclid|dclid|msclkid|mc_cid|mc_eid|igshid|ref_src)$/i;
|
|
3671
3635
|
function normalizeUrl(raw) {
|
|
3672
3636
|
let u;
|
|
@@ -3688,8 +3652,8 @@ var urlHash = (normalized2) => createHash2("sha1").update(normalized2).digest("h
|
|
|
3688
3652
|
function urlPaths(libraryDir, normalized2) {
|
|
3689
3653
|
const hash = urlHash(normalized2);
|
|
3690
3654
|
const entry = `url-${hash}`;
|
|
3691
|
-
const entryDir =
|
|
3692
|
-
return { hash, entry, entryDir, meta:
|
|
3655
|
+
const entryDir = join8(libraryDir, "url", hash.slice(0, 2), hash.slice(0, 4), hash);
|
|
3656
|
+
return { hash, entry, entryDir, meta: join8(entryDir, `${entry}-meta.md`), md: join8(entryDir, `${entry}-page.md`), raw: (ext) => join8(entryDir, `${entry}-page.${ext}`) };
|
|
3693
3657
|
}
|
|
3694
3658
|
function renderUrlMeta(m) {
|
|
3695
3659
|
const lines = [
|
|
@@ -3708,8 +3672,8 @@ function renderUrlMeta(m) {
|
|
|
3708
3672
|
return lines.map(([k, v]) => `${k}: ${v}`).join("\n") + "\n";
|
|
3709
3673
|
}
|
|
3710
3674
|
function readUrlMeta(p) {
|
|
3711
|
-
if (!
|
|
3712
|
-
const md =
|
|
3675
|
+
if (!existsSync7(p.meta)) return null;
|
|
3676
|
+
const md = readFileSync7(p.meta, "utf8");
|
|
3713
3677
|
const f = (k) => {
|
|
3714
3678
|
const m = new RegExp(`^${k}:[ \\t]*(.*)$`, "m").exec(md);
|
|
3715
3679
|
return m ? m[1].trim() : "";
|
|
@@ -3793,7 +3757,7 @@ var climb = async (url, onTry, onAlert) => {
|
|
|
3793
3757
|
if (!r.ok && /^unlocker HTTP/.test(r.reason)) onAlert?.(`brightdata: ${r.reason}`);
|
|
3794
3758
|
}
|
|
3795
3759
|
if (!r.ok && shell) {
|
|
3796
|
-
onTry?.(
|
|
3760
|
+
onTry?.("shell kept: no rung rendered the page, the script shell is served");
|
|
3797
3761
|
return shell;
|
|
3798
3762
|
}
|
|
3799
3763
|
return r.ok || r.via === "fetch" ? r : { ...r, first: firstReason };
|
|
@@ -3886,6 +3850,25 @@ function logId(url) {
|
|
|
3886
3850
|
const bare2 = url.replace(/^https?:\/\//i, "");
|
|
3887
3851
|
return bare2.length > ID_WIDTH ? bare2.slice(0, ID_WIDTH - 1) + "\u2026" : bare2;
|
|
3888
3852
|
}
|
|
3853
|
+
function libraryIdForUrl(raw) {
|
|
3854
|
+
let u;
|
|
3855
|
+
try {
|
|
3856
|
+
u = new URL(raw.trim());
|
|
3857
|
+
} catch {
|
|
3858
|
+
return null;
|
|
3859
|
+
}
|
|
3860
|
+
const host = u.hostname.toLowerCase().replace(/^www\./, ""), path = u.pathname;
|
|
3861
|
+
let m;
|
|
3862
|
+
if (host === "pubmed.ncbi.nlm.nih.gov" && (m = /^\/(\d{1,8})\/?$/.exec(path))) return `pmid:${m[1]}`;
|
|
3863
|
+
if (host === "ncbi.nlm.nih.gov" && (m = /^\/pubmed\/(\d{1,8})\/?$/.exec(path))) return `pmid:${m[1]}`;
|
|
3864
|
+
if (host === "clinicaltrials.gov" && (m = /^\/(?:study|ct2\/show)\/(NCT\d{8})\/?$/i.exec(path))) return `nct:${m[1].slice(3)}`;
|
|
3865
|
+
if (host === "dailymed.nlm.nih.gov" && /\/(drugInfo|lookup)\.cfm$/i.test(path)) {
|
|
3866
|
+
const id = u.searchParams.get("setid");
|
|
3867
|
+
if (id && /^[0-9a-f-]{36}$/i.test(id)) return `dailymed:${id.toLowerCase()}`;
|
|
3868
|
+
}
|
|
3869
|
+
if ((host === "doi.org" || host === "dx.doi.org") && (m = /^\/(10\.\S+)$/.exec(decodeURIComponent(path)))) return `doi:${m[1]}`;
|
|
3870
|
+
return null;
|
|
3871
|
+
}
|
|
3889
3872
|
var failureText = (url, reason) => {
|
|
3890
3873
|
const m = /^HTTP (\d{3})\b/.exec(reason);
|
|
3891
3874
|
return m ? `Failed to fetch ${url} - status code ${m[1]}` : `Failed to fetch ${url}: ${reason}`;
|
|
@@ -3898,14 +3881,14 @@ function titleLine(m, md) {
|
|
|
3898
3881
|
}
|
|
3899
3882
|
function serve(requested, p, m, args, cached) {
|
|
3900
3883
|
const maxLength = args.max_length ?? 5e3, startIndex = args.start_index ?? 0;
|
|
3901
|
-
const read = (file2) =>
|
|
3884
|
+
const read = (file2) => existsSync7(file2) ? readFileSync7(file2, "utf8") : "";
|
|
3902
3885
|
if (args.title_only) return { ok: true, text: `Contents of ${requested}:
|
|
3903
3886
|
${titleLine(m, read(p.md))}`, cached, meta: m };
|
|
3904
3887
|
const rawFile = m.files.find((f) => !f.endsWith("-meta.md") && !f.endsWith("-page.md"));
|
|
3905
3888
|
let body, prefix = "";
|
|
3906
3889
|
if (m.format === "markdown") body = read(p.md);
|
|
3907
3890
|
else if (m.format === "text" || args.raw) {
|
|
3908
|
-
body = rawFile ? read(
|
|
3891
|
+
body = rawFile ? read(join8(p.entryDir, rawFile)) : read(p.md);
|
|
3909
3892
|
prefix = `Content type ${m.content_type || "(not declared)"} cannot be simplified to markdown, but here is the raw content:
|
|
3910
3893
|
`;
|
|
3911
3894
|
} else body = read(p.md);
|
|
@@ -3954,8 +3937,8 @@ async function libraryFetch(libraryDir, requested, args, converter, logs = {}, f
|
|
|
3954
3937
|
|
|
3955
3938
|
// src/verifycache.ts
|
|
3956
3939
|
import { createHash as createHash3 } from "crypto";
|
|
3957
|
-
import { existsSync as
|
|
3958
|
-
import { dirname as dirname3, join as
|
|
3940
|
+
import { existsSync as existsSync8, mkdirSync as mkdirSync6, readFileSync as readFileSync8 } from "fs";
|
|
3941
|
+
import { dirname as dirname3, join as join9 } from "path";
|
|
3959
3942
|
var sha = (s) => createHash3("sha1").update(s).digest("hex");
|
|
3960
3943
|
var normalizeClaim = (claim) => claim.replace(/\s+/g, " ").trim();
|
|
3961
3944
|
function foldClaim(claim) {
|
|
@@ -3967,11 +3950,11 @@ ${id}
|
|
|
3967
3950
|
${model}
|
|
3968
3951
|
${threshold}
|
|
3969
3952
|
${text3}`).slice(0, 16);
|
|
3970
|
-
var verdictFile = (libraryDir, key) =>
|
|
3953
|
+
var verdictFile = (libraryDir, key) => join9(libraryDir, "verify", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
|
|
3971
3954
|
function readVerdict(file2) {
|
|
3972
|
-
if (!
|
|
3955
|
+
if (!existsSync8(file2)) return null;
|
|
3973
3956
|
try {
|
|
3974
|
-
return JSON.parse(
|
|
3957
|
+
return JSON.parse(readFileSync8(file2, "utf8"));
|
|
3975
3958
|
} catch {
|
|
3976
3959
|
return null;
|
|
3977
3960
|
}
|
|
@@ -4021,8 +4004,8 @@ var leanJson = (o) => compact2(lean(o));
|
|
|
4021
4004
|
|
|
4022
4005
|
// src/trials.ts
|
|
4023
4006
|
import { createHash as createHash4 } from "crypto";
|
|
4024
|
-
import { existsSync as
|
|
4025
|
-
import { dirname as dirname4, join as
|
|
4007
|
+
import { existsSync as existsSync9, mkdirSync as mkdirSync7, readFileSync as readFileSync9 } from "fs";
|
|
4008
|
+
import { dirname as dirname4, join as join10 } from "path";
|
|
4026
4009
|
var searchCacheDays = () => settings().search_cache_days;
|
|
4027
4010
|
var API2 = "https://clinicaltrials.gov/api/v2/studies";
|
|
4028
4011
|
var FIELDS = ["NCTId", "BriefTitle", "OverallStatus", "WhyStopped", "Phase", "StudyType", "EnrollmentCount", "EnrollmentType", "StartDate", "PrimaryCompletionDate", "CompletionDate", "LastUpdatePostDate", "PrimaryOutcomeMeasure", "PrimaryOutcomeTimeFrame", "Condition", "InterventionName", "LeadSponsorName", "HasResults"];
|
|
@@ -4060,7 +4043,7 @@ function normalizeQuery(q) {
|
|
|
4060
4043
|
return out;
|
|
4061
4044
|
}
|
|
4062
4045
|
var searchKey = (q) => createHash4("sha1").update(JSON.stringify(["clinicaltrials", q.query ?? "", q.intervention ?? "", q.condition ?? "", q.status, q.sort, q.max_results])).digest("hex").slice(0, 16);
|
|
4063
|
-
var searchFile = (libraryDir, key) =>
|
|
4046
|
+
var searchFile = (libraryDir, key) => join10(libraryDir, "search", key.slice(0, 2), key.slice(0, 4), `${key}.json`);
|
|
4064
4047
|
function searchUrl(q) {
|
|
4065
4048
|
const p = new URLSearchParams();
|
|
4066
4049
|
if (q.query) p.set("query.term", q.query);
|
|
@@ -4112,9 +4095,9 @@ async function searchTrials(raw, libraryDir = FH_LIBRARY_DIR, fetchImpl = liveFe
|
|
|
4112
4095
|
const q = normalizeQuery(raw);
|
|
4113
4096
|
if ("error" in q) return { ok: false, error: q.error };
|
|
4114
4097
|
const file2 = searchFile(libraryDir, searchKey(q));
|
|
4115
|
-
if (
|
|
4098
|
+
if (existsSync9(file2)) {
|
|
4116
4099
|
try {
|
|
4117
|
-
const r = JSON.parse(
|
|
4100
|
+
const r = JSON.parse(readFileSync9(file2, "utf8"));
|
|
4118
4101
|
if (daysSince(r.searched) < searchCacheDays()) return { ok: true, result: r, cached: true };
|
|
4119
4102
|
} catch {
|
|
4120
4103
|
}
|
|
@@ -4146,7 +4129,7 @@ function trialsLogId(q) {
|
|
|
4146
4129
|
|
|
4147
4130
|
// src/pubmedsearch.ts
|
|
4148
4131
|
import { createHash as createHash5 } from "crypto";
|
|
4149
|
-
import { existsSync as
|
|
4132
|
+
import { existsSync as existsSync10, mkdirSync as mkdirSync8, readFileSync as readFileSync10 } from "fs";
|
|
4150
4133
|
import { dirname as dirname5 } from "path";
|
|
4151
4134
|
var MAX_RESULTS2 = 100;
|
|
4152
4135
|
var DEFAULT_RESULTS2 = 20;
|
|
@@ -4227,9 +4210,9 @@ async function searchArticles(raw, libraryDir = FH_LIBRARY_DIR, fetchImpl = live
|
|
|
4227
4210
|
const q = normalizeQuery2(raw);
|
|
4228
4211
|
if ("error" in q) return { ok: false, error: q.error };
|
|
4229
4212
|
const file2 = searchFile(libraryDir, searchKey2(q));
|
|
4230
|
-
if (
|
|
4213
|
+
if (existsSync10(file2)) {
|
|
4231
4214
|
try {
|
|
4232
|
-
const r = JSON.parse(
|
|
4215
|
+
const r = JSON.parse(readFileSync10(file2, "utf8"));
|
|
4233
4216
|
if (daysSince(r.searched) < searchCacheDays()) return { ok: true, result: r, cached: true };
|
|
4234
4217
|
} catch {
|
|
4235
4218
|
}
|
|
@@ -4333,13 +4316,13 @@ function resultsText(nct, md, maxChars) {
|
|
|
4333
4316
|
}
|
|
4334
4317
|
|
|
4335
4318
|
// src/missing.ts
|
|
4336
|
-
import { existsSync as
|
|
4337
|
-
import { join as
|
|
4338
|
-
var missingFile = (p) =>
|
|
4319
|
+
import { existsSync as existsSync11, mkdirSync as mkdirSync9, readFileSync as readFileSync11 } from "fs";
|
|
4320
|
+
import { join as join11 } from "path";
|
|
4321
|
+
var missingFile = (p) => join11(p.entryDir, `${p.entry}-missing.md`);
|
|
4339
4322
|
function readMissing(p) {
|
|
4340
4323
|
const f = missingFile(p);
|
|
4341
|
-
if (!
|
|
4342
|
-
const md =
|
|
4324
|
+
if (!existsSync11(f)) return null;
|
|
4325
|
+
const md = readFileSync11(f, "utf8");
|
|
4343
4326
|
const get = (k) => /^$/.test(k) ? "" : (new RegExp(`^${k}:[ \\t]*(.*)$`, "m").exec(md)?.[1] ?? "").trim();
|
|
4344
4327
|
const checked = get("checked"), reason = get("reason");
|
|
4345
4328
|
return checked ? { checked, reason: reason || "not found" } : null;
|
|
@@ -4445,7 +4428,7 @@ async function fetchArticles(ids, libraryDir = FH_LIBRARY_DIR, converter = "fh-l
|
|
|
4445
4428
|
// src/tools.ts
|
|
4446
4429
|
var pkg = (() => {
|
|
4447
4430
|
try {
|
|
4448
|
-
return JSON.parse(
|
|
4431
|
+
return JSON.parse(readFileSync12(new URL("../package.json", import.meta.url), "utf8"));
|
|
4449
4432
|
} catch {
|
|
4450
4433
|
return { name: "fh-library", version: "0.0.0" };
|
|
4451
4434
|
}
|
|
@@ -4504,7 +4487,7 @@ Reading a document
|
|
|
4504
4487
|
|
|
4505
4488
|
Status fields: requested, type, id, entry, pmid, pmcid, doi, setid, nctid, publications, title, first_author, year, publication_types, retracted, corrections, coi (conflict-of-interest statement), funding, citable, comment, has_abstract, level, version, retrieved_by, retrieved_date, pubmed_checked, attempts, glyphs_unresolved, error. Answers are compact JSON; a field that is null or an empty list is left out (library_verify keeps its nulls: score and reason null mean something).
|
|
4506
4489
|
|
|
4507
|
-
Where records come from: PubMed for the record and abstract; the full text from
|
|
4490
|
+
Where records come from: PubMed for the record and abstract; the full text from PubMed Central, Europe PMC, publishers' text-mining links, Unpaywall, OpenAlex, CORE, and a proxy URL template from the configuration. Every action is appended as one line to <library>/library.log.`;
|
|
4508
4491
|
var idSchema = z.string().describe('ID as "type:value": a paper "pmid:30955454" (bare digits count as pmid) or "doi:10.5812/asjsm.110753" (a paper PubMed does not index; a DOI PubMed indexes is taken as its PMID), a drug label "dailymed:<set ID>" (the set ID from its DailyMed page URL), or a trial record "nct:05173896" (the eight digits; nct:NCT05173896 and a bare NCT05173896 are accepted).');
|
|
4509
4492
|
var levelWord = (m) => m.level === "abstract" && !m.has_abstract ? "title" : m.level;
|
|
4510
4493
|
var json = (s) => ({ content: [{ type: "text", text: leanJson(s) }] });
|
|
@@ -4841,8 +4824,8 @@ function registerTools(server2) {
|
|
|
4841
4824
|
}
|
|
4842
4825
|
added = true;
|
|
4843
4826
|
}
|
|
4844
|
-
const full =
|
|
4845
|
-
const md =
|
|
4827
|
+
const full = existsSync12(paths.fulltextMd);
|
|
4828
|
+
const md = readFileSync12(full ? paths.fulltextMd : paths.md, "utf8");
|
|
4846
4829
|
const text3 = full ? "full_text" : "abstract";
|
|
4847
4830
|
const base = added ? { requested: parsed.requested, entry: paths.entry, text: text3, added: true } : { requested: parsed.requested, entry: paths.entry, text: text3 };
|
|
4848
4831
|
const headings2 = () => outline(md, text3).sections;
|
|
@@ -4862,8 +4845,8 @@ function registerTools(server2) {
|
|
|
4862
4845
|
const cap = max_chars ?? 6e3;
|
|
4863
4846
|
let sec = readSection(md, section2, cap);
|
|
4864
4847
|
let via = "";
|
|
4865
|
-
if (!sec && /abstract/i.test(section2) && full &&
|
|
4866
|
-
sec = readSection(
|
|
4848
|
+
if (!sec && /abstract/i.test(section2) && full && existsSync12(paths.md)) {
|
|
4849
|
+
sec = readSection(readFileSync12(paths.md, "utf8"), "abstract", cap);
|
|
4867
4850
|
via = " (from the abstract file)";
|
|
4868
4851
|
}
|
|
4869
4852
|
if (!sec && /declaration/i.test(section2)) {
|
|
@@ -4938,7 +4921,7 @@ function registerTools(server2) {
|
|
|
4938
4921
|
);
|
|
4939
4922
|
server2.tool(
|
|
4940
4923
|
"library_fetch",
|
|
4941
|
-
`Fetch a web page as Markdown: the standard fetch tool, cached and hardened. Use it for any URL instead of a plain fetch or a browser. A page fetched in the last ${settings().refresh_after_days} days is answered from disk in milliseconds; a page a plain fetch cannot get (403, rate limit, bot wall, script-rendered) is fetched through the configured browsers and unlocker in this one call. PDFs come back as Markdown. To verify a link, title_only returns just its title line.`,
|
|
4924
|
+
`Fetch a web page as Markdown: the standard fetch tool, cached and hardened. Use it for any URL instead of a plain fetch or a browser. A page fetched in the last ${settings().refresh_after_days} days is answered from disk in milliseconds; a page a plain fetch cannot get (403, rate limit, bot wall, script-rendered) is fetched through the configured browsers and unlocker in this one call. PDFs come back as Markdown. To verify a link, title_only returns just its title line. A PubMed, ClinicalTrials.gov, DailyMed or doi.org URL is answered from the library's record of it.`,
|
|
4942
4925
|
{
|
|
4943
4926
|
url: z.string().describe("URL to fetch"),
|
|
4944
4927
|
max_length: z.number().int().min(1).max(999999).default(5e3).describe("Maximum number of characters to return."),
|
|
@@ -4949,6 +4932,39 @@ function registerTools(server2) {
|
|
|
4949
4932
|
async ({ url, max_length, start_index, raw, title_only }) => {
|
|
4950
4933
|
const t0 = Date.now();
|
|
4951
4934
|
const id = logId(url), put = (line) => log("library_fetch", id, `${line} ${url}`);
|
|
4935
|
+
let rid = raw ? null : libraryIdForUrl(url);
|
|
4936
|
+
if (rid?.startsWith("doi:") && !entryExists(entryPaths(FH_LIBRARY_DIR, "doi", parseId(rid).id))) {
|
|
4937
|
+
try {
|
|
4938
|
+
const pm = await pmidForDoi(parseId(rid).id);
|
|
4939
|
+
if (pm) rid = `pmid:${pm}`;
|
|
4940
|
+
} catch {
|
|
4941
|
+
}
|
|
4942
|
+
}
|
|
4943
|
+
if (rid) {
|
|
4944
|
+
const pid = parseId(rid);
|
|
4945
|
+
const paths = entryPaths(FH_LIBRARY_DIR, pid.type, pid.id);
|
|
4946
|
+
const had = entryExists(paths);
|
|
4947
|
+
if (!had && pid.type === "pmid") {
|
|
4948
|
+
const b = await fetchArticles([pid.id], FH_LIBRARY_DIR, `${pkg.name} ${pkg.version}`);
|
|
4949
|
+
if (b.rows[0]?.[6] === "not found" || !entryExists(paths)) {
|
|
4950
|
+
put(`library ${rid} error: pmid not found`);
|
|
4951
|
+
return { content: [{ type: "text", text: `Failed to fetch ${url}: pmid not found` }], isError: true };
|
|
4952
|
+
}
|
|
4953
|
+
} else if (!had) {
|
|
4954
|
+
const added = JSON.parse((await addEntry(rid)).content[0].text);
|
|
4955
|
+
if (added.error) {
|
|
4956
|
+
put(`library ${rid} error: ${added.error}`);
|
|
4957
|
+
return { content: [{ type: "text", text: `Failed to fetch ${url}: ${added.error}` }], isError: true };
|
|
4958
|
+
}
|
|
4959
|
+
}
|
|
4960
|
+
const file2 = pid.type === "pmid" ? paths.md : existsSync12(paths.fulltextMd) ? paths.fulltextMd : paths.md;
|
|
4961
|
+
const md = existsSync12(file2) ? readFileSync12(file2, "utf8") : "";
|
|
4962
|
+
const text3 = title_only ? `Contents of ${url}:
|
|
4963
|
+
${(md.split("\n").find((l) => l.trim()) ?? "").trim()}` : fetchAnswer(url, md, "", max_length, start_index);
|
|
4964
|
+
bump("fetch", had);
|
|
4965
|
+
put(`library ${rid} ${had ? "cached" : "added"} chars=${text3.length}${title_only ? " title" : ""}${start_index ? ` start=${start_index}` : ""}`);
|
|
4966
|
+
return { content: [{ type: "text", text: text3 }] };
|
|
4967
|
+
}
|
|
4952
4968
|
const r = await libraryFetch(FH_LIBRARY_DIR, url, { max_length, start_index, raw, title_only }, `${pkg.name} ${pkg.version}`, {
|
|
4953
4969
|
onTry: put,
|
|
4954
4970
|
onAlert: (line) => put(`!! ${line}`)
|
|
@@ -5049,7 +5065,7 @@ function registerTools(server2) {
|
|
|
5049
5065
|
return { content: [{ type: "text", text: compact2({ error: e.error }) }], isError: true };
|
|
5050
5066
|
}
|
|
5051
5067
|
const raw = e.paths.fulltextRaw("json");
|
|
5052
|
-
const r =
|
|
5068
|
+
const r = existsSync12(raw) ? recordSummary(readFileSync12(raw, "utf8")) : null;
|
|
5053
5069
|
if (!r) {
|
|
5054
5070
|
log("library_trials", e.requested, "record error: no stored record");
|
|
5055
5071
|
return { content: [{ type: "text", text: compact2({ error: "no stored record" }) }], isError: true };
|
|
@@ -5068,7 +5084,7 @@ function registerTools(server2) {
|
|
|
5068
5084
|
log("library_trials", nct, `results error: ${e.error}`);
|
|
5069
5085
|
return { content: [{ type: "text", text: compact2({ error: e.error }) }], isError: true };
|
|
5070
5086
|
}
|
|
5071
|
-
const md =
|
|
5087
|
+
const md = existsSync12(e.paths.fulltextMd) ? readFileSync12(e.paths.fulltextMd, "utf8") : "";
|
|
5072
5088
|
const r = resultsText(e.requested.replace(/^nct:/, "NCT"), md, max_chars);
|
|
5073
5089
|
log("library_trials", e.requested, `results ${r.has_results ? `${r.outcomes.length} outcomes${r.truncated ? " truncated" : ""}` : "none"}`);
|
|
5074
5090
|
return { content: [{ type: "text", text: leanJson(r) }] };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "fh-library",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.17",
|
|
4
4
|
"mcpName": "io.github.forever-healthy/fh-library",
|
|
5
5
|
"description": "MCP Server that builds a local Markdown library of the scientific papers cited by evidence reviews — fetches metadata, abstract and full text per PubMed ID through a configurable cascade of retrieval services",
|
|
6
6
|
"type": "module",
|