fh-library 0.2.11 → 0.2.13
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +8 -0
- package/README.md +7 -3
- package/dist/index.js +322 -33
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,7 +1,15 @@
|
|
|
1
1
|
# FH Library — Change Log
|
|
2
2
|
|
|
3
|
+
### v0.2.12 — 2026-11-08
|
|
4
|
+
|
|
5
|
+
* **`library_pubmed_search_articles(query, max_results?, sort?, years?)`:** a PubMed search in PubMed's own syntax — esearch for the total and the IDs, one esummary for the rows (PMID, title, first author, year, journal, publication types, PMCID, library level), cached under `search/`. With the batch fetch and `library_read` this covers what the AI4L models used of the PubMed MCP server (#27).
|
|
6
|
+
* **`FH_LIBRARY_SEARCH_CACHE_DAYS` (7):** days a PubMed or registry search is cached; was a constant.
|
|
7
|
+
* **An ID no service knows is remembered** for `FH_LIBRARY_RETRY_AFTER_DAYS` (1): a PMID PubMed has no record for, a DOI neither Crossref nor doi.org resolves, a set ID without a label, an NCT ID without a study. A marker (`<entry>-missing.md`, date and reason, no meta) in the entry's directory; `library_add`, `library_status`, `library_read`, `library_verify` and the batch tool answer from it ("pmid not found (checked 2026-11-08; asked again after 1 day)"), the service is asked again after the cooldown, a real entry replaces the marker. Network errors are never remembered. The same cooldown a failed page or a transient tier failure already had.
|
|
8
|
+
* **`library_pubmed_fetch_articles(pmids)`:** title, first author, year, publication types, retracted and level for up to 200 PMIDs in one call, rows in `columns` order (~120 characters a paper against the PubMed MCP server's ~2,500). Stored PMIDs from disk; the rest in one efetch, stored at abstract level as `library_add`'s first stage, so the cascade runs later as on an untried entry. The first of the PubMed tools (#27); `library_info`'s `cache` and `.servers/` count `pubmed` hits and misses per PMID.
|
|
9
|
+
|
|
3
10
|
### v0.2.9 — 2026-11-08
|
|
4
11
|
|
|
12
|
+
* TUI v82: the `url` row shows library_fetch hits and misses from the servers' counters; `/status` carries `fetches`. Counts right-aligned in a 5-character column.
|
|
5
13
|
* TUI v80: a `search` row — cached registry searches, hits and misses from the servers' counters; the decision-model row is labelled `decider`; `/status` carries `searches`.
|
|
6
14
|
* ClinicalTrials tool descriptions and server instructions one statement each.
|
|
7
15
|
|
package/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-

|
|
2
2
|

|
|
3
3
|
[_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
|
|
4
4
|
[](https://www.npmjs.com/package/fh-library)
|
|
@@ -23,6 +23,8 @@ A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a
|
|
|
23
23
|
* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), this server's cache hits and misses since it started (`cache`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
|
|
24
24
|
* `library_verify(claim, id)` → is one finding of a review stated by one paper it cites? `claim` is the finding alone, verbatim, without its citations; `id` one paper (`pmid:`, or any `doi:`), drug label or trial record the review cites for it — a finding cited to three papers is three calls with the same claim; a claim that names a source or carries citations is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters the model's context. `verdict` **pass** (`score` the probability), **fail** (the full text does not state it; or only the title is in the library — `reason` says so), **unverifiable** (only the abstract is in the library and does not state it — `reason`) or **error**; `basis` full_text, abstract or title. See "Verifying claims" below.
|
|
25
25
|
* `library_read(id, section?, query?, max_chars?, max_hits?)` → one small piece of a stored paper's Markdown, never the whole paper: without `section`/`query` the **outline** (headings with word counts, and whether the full text or only the abstract is stored; a heading the paper uses several times is listed as `Participants [3/12]`); `section` → that section's **own text** (heading matched as case-insensitive substring, capped at `max_chars`, default 6000, with `returned_words` against `words` when cut) and, for a parent heading, its `subsections` with word counts to pick from — a duplicated heading named bare lists the candidates instead; `query` — two or three distinctive terms the paper itself would use (a measure and a unit, an outcome and a group, a compound name), not the claim — → **passages** holding every term, each a window of at most 1000 chars around the matches (`clipped: true` when cut; a table row with its header row), with its section, best first (at most `max_hits`, default 5); matching forgives case, accents, hyphens, Markdown markup, plural and -ed/-ing forms; when no passage holds every term the best **partial** matches come back (`partial: true`, the `matched` terms per passage, the rare term first), and `missing_terms` names the terms the text never uses in any form — those are the ones to rephrase. `section` and `query` together search inside that section. `section: "Abstract"` on a full text without one answers from the stored abstract. Every full text converted from JATS XML (PMC, Europe PMC, Elsevier) has a **Declarations** section — funding, conflicts of interest, author notes, gathered from wherever the XML keeps them — and on a text without one `section: "Declarations"` assembles the sections headed funding, conflicts, acknowledgements, ethics, contributions, so it answers who paid and who declares what on every source alike; a paper with neither states nothing. Outline first, then pick. A paper not in the library is added first, as `library_add` would (its lines in the log), then library_read; the answer carries `added: true`. Reads may run in parallel.
|
|
26
|
+
* `library_pubmed_search_articles(query, max_results?, sort?, years?)` → a PubMed search in PubMed's own syntax (see "PubMed in batches"): the total and rows in `columns` order, with the library level of a paper already stored, cached for `FH_LIBRARY_SEARCH_CACHE_DAYS`
|
|
27
|
+
* `library_pubmed_fetch_articles(pmids)` → title, first author, year, publication types, retracted and level for up to 200 PMIDs in one call (see "PubMed in batches"); PMIDs the library lacks are fetched in one PubMed request and stored at abstract level
|
|
26
28
|
* `library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?)` → a ClinicalTrials.gov search (see "Trial search"): the total count and compact rows, cached for a week
|
|
27
29
|
* `library_clinicaltrials_get_study_record(nct)` → one trial as a review needs it (title, status, phase, design, enrollment, dates, sponsor, conditions, interventions, primary outcomes, eligibility, sites, result publications, has_results, brief summary), from the stored record
|
|
28
30
|
* `library_clinicaltrials_get_study_results(nct, max_chars?)` → a trial's posted results as Markdown, capped, with the outcome measure titles for `library_read`
|
|
@@ -58,7 +60,7 @@ Downloads are plain HTTP with a desktop Chrome User-Agent (API calls identify th
|
|
|
58
60
|
|
|
59
61
|
* **Which tiers run, in which order:** `FH_LIBRARY_TIERS=pmc,tdm,unpaywall,openalex` — exactly these, in this order; a tier not listed is off. Unset, every tier runs in the order of the table above, the proxy last.
|
|
60
62
|
* **Keys:** `FH_LIBRARY_ELSEVIER_API_KEY` and `FH_LIBRARY_WILEY_TDM_TOKEN` for `tdm`, `FH_LIBRARY_CORE_API_KEY` for `core`, `FH_LIBRARY_NCBI_API_KEY` (10 instead of 3 requests/s), `FH_LIBRARY_BRIGHTDATA_API_TOKEN` and `FH_LIBRARY_BRIGHTDATA_ZONE` for the unlocker. `FH_LIBRARY_CONTACT_EMAIL` is the one address sent to every service (NCBI `tool`/`email`, Unpaywall and OpenAlex polite pools, Crossref, the User-Agent).
|
|
61
|
-
* **Retries:** `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried; `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `library_add` re-reads it for new correction notices and retractions. A tdm link behind a token that is not configured is recorded as `needs elsevier token` and not re-checked; after adding the token, remove those attempts from the entries to give them a go. `FH_LIBRARY_RETRY_BLOCKED` (false) decides whether a browser challenge that did not clear, or a Bright Data refusal, is final: with the same method tomorrow ends the same way, so set it true only after a change of method (a new browser rung, a new service).
|
|
63
|
+
* **Retries:** `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried, before a page `library_fetch` could not get is fetched again, and before an ID no service knew (a PMID, DOI, set ID or NCT ID that is not found) is asked for again — meanwhile every tool answers "not found" from a marker in the entry's directory, with the date it was checked; `FH_LIBRARY_SEARCH_CACHE_DAYS` (7) how long a PubMed or registry search is answered from disk; `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `library_add` re-reads it for new correction notices and retractions. A tdm link behind a token that is not configured is recorded as `needs elsevier token` and not re-checked; after adding the token, remove those attempts from the entries to give them a go. `FH_LIBRARY_RETRY_BLOCKED` (false) decides whether a browser challenge that did not clear, or a Bright Data refusal, is final: with the same method tomorrow ends the same way, so set it true only after a change of method (a new browser rung, a new service).
|
|
62
64
|
* **Browsers and verify:** `FH_LIBRARY_PLAYWRIGHT_n` and `FH_LIBRARY_CHROME_n` (hostnames; above) and `FH_LIBRARY_PLAYWRIGHT_HEADLESS`; `FH_LIBRARY_VERIFY_*` ("Verifying claims"). A Playwright server must run the **same `playwright-core` version** as the package — `library_info` shows the required one as `download.playwright.version`, `browser/Dockerfile` builds a server at it, and a server of another version is refused by Playwright itself (the log then carries the alert `!! browser <host>: Playwright version mismatch — server … must run playwright-core <version>`). Moving the version is one release: the pin in `package.json`, the tag in `browser/Dockerfile`, every server rebuilt. The library never launches a browser itself: every rung is a browser somebody runs.
|
|
63
65
|
|
|
64
66
|
`library_info` reports the tiers in run order and, per variable, whether its value came from the environment, `.env` or the default — never the value.
|
|
@@ -147,7 +149,9 @@ PubMed's book records — National Academies reports, NTP monographs, StatPearls
|
|
|
147
149
|
|
|
148
150
|
**Clinical trial records.** `library_add("nct:05173896")` stores the current record of one trial from [ClinicalTrials.gov](https://clinicaltrials.gov/): the ID is the eight digits of the NCT number in the review's link (`…/study/NCT05173896`); `nct:NCT05173896` and a bare `NCT05173896` are accepted and read the same. The API's JSON is converted to Markdown, one section per module: identification with secondary IDs such as EudraCT, status with every date and its actual/estimated type, sponsor and collaborators, description, conditions, design (phase, allocation, masking, enrollment with its type), arms and interventions with doses, primary and secondary outcomes with time frames, eligibility, locations as a country list, references; and, when results are posted, the participant flow, baseline characteristics, every outcome measure with its group values and analyses, and the adverse events, all as tables. Values are written in plain words, "Phase 2", "Active, not recruiting", "100 (estimated)", so a figure the review states can be found as stated. The status carries `nctid`, `publications` (the results publications the record lists, as `pmid:…`, so whether a trial has been published is one status away), the brief title, the lead sponsor as `first_author`, the first-posted year as `year`, the last-update-posted date as `version`, and `publication_types` `["Trial Registration"]`. `library_read` and `library_verify` work on a record as on a paper. An entry older than `FH_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when the record was updated since; a status the review states that has moved on then fails, which is right. Entries live under `nct/<digits 1–2>/<digits 1–4>/<digits>/` with the JSON and the Markdown as `nct-<digits>-fulltext.json` and `-fulltext.md`; the log line reads `library_add nct:05173896 ctgov fetched level=full_text updated 2024-12-09 active, not recruiting ETLAS2`.
|
|
149
151
|
|
|
150
|
-
**
|
|
152
|
+
**PubMed in batches.** `library_pubmed_fetch_articles(pmids)` answers what an auditor asks of a list of PubMed IDs — does each resolve, what is its title, year and type, is it retracted, does the library hold its full text — as rows in `columns` order, about 120 characters per paper. A PMID already in the library is read from disk. The others go to PubMed in one efetch request (up to 200), and each record is stored exactly as `library_add` stores its first stage: the XML, the abstract Markdown, the meta with its correction and retraction notices, no retrieval attempts yet — so a later `library_add` or `library_read` runs the full-text cascade as on an untried entry, and the library fills at abstract level in batches instead of one paper at a time. An entry is checked again right before writing, so a concurrent `library_add` that already stored a full text is never overwritten. Level `not found` marks a PMID PubMed does not know, `invalid` a value that is not a PMID. `library_pubmed_search_articles(query, max_results?, sort?, years?)` is the search: the query goes to PubMed as written, field tags and all; esearch gives the total and the IDs, one esummary the rows — PMID, title, first author, year, journal, publication types (Journal Article left out), PMCID when a free full text exists, and the library's level when the paper is already stored, so the model knows what `library_read` can answer at once. Cached under `search/` like the trial search.
|
|
153
|
+
|
|
154
|
+
**Trial search.** `library_clinicaltrials_search_studies(query?, intervention?, condition?, status?, sort?, max_results?)` asks the ClinicalTrials.gov API (v2, no key) what a review needs from the registry: the trials of an intervention, the ongoing ones (`status: "ongoing"` is recruiting, not yet recruiting, active not recruiting, enrolling by invitation), a free-text or condition search, sorted by relevance, newest, largest or completion date, up to 100 rows. The rows are arrays in the order of `columns` — the field names once, not per row — shaped for the review's Emerging Research item (what AI4L's prompt asks about an ongoing trial; the link is `https://clinicaltrials.gov/study/<NCT ID>`): NCT ID, title, status with why a trial stopped, phase, enrollment as "100 (estimated)", start and primary completion dates (whether "ongoing" still holds), the primary outcome with its time frame, conditions, interventions, `has_results`; the full detail of one row is `library_clinicaltrials_get_study_record`; `total` is the registry's count, for a "no trials found" statement. The same search is answered from `search/<2>/<4>/<key>.json` for `FH_LIBRARY_SEARCH_CACHE_DAYS` (7) — a registry listing a week old is current enough for a review, and an audit that runs again reads from disk. Two thin tools answer from the stored record of any row (added first when missing): `library_clinicaltrials_get_study_record(nct)` is the review's detail of one trial — official title, design, enrollment, dates, eligibility, sites, result publications as `pmid:` IDs, the brief summary — and `library_clinicaltrials_get_study_results(nct, max_chars?)` is the posted results part of the stored Markdown (participant flow, baseline, outcome measures, adverse events), with the outcome titles listed so `library_read("nct:…", section)` can return one measure whole. Together with the search they cover what the AI4L models asked the ClinicalTrials.gov MCP server for.
|
|
151
155
|
|
|
152
156
|
**Web pages.** `library_fetch(url, max_length?, start_index?, raw?)` is the standard MCP [fetch](https://github.com/modelcontextprotocol/servers/tree/main/src/fetch) tool, served from the library, for every link a review carries that is not a paper, label or trial record: a guideline, a product page, a news item. Its description tells the model to prefer it over a plain fetch or a browser tool, and says why: a page anyone fetched in the last 30 days comes from disk in milliseconds, and a page a plain fetch cannot get climbs the ladder in the same call. The answer is what `fetch` returns, so a model keeps its habits: `Contents of <url>:` and the page as Markdown with the page's title as the first heading, cut at `max_length` (default 5000) with the note `Content truncated. Call the fetch tool with a start_index of N to get more content`; `raw` gives the HTML; a `text/markdown` body comes back as is, any other non-HTML body with the `Content type … cannot be simplified to markdown` prefix; a failure is `Failed to fetch <url> - status code 404` or `Failed to fetch <url>: <reason>`. Two things happen underneath. The page is fetched up the same ladder as a paper — the plain fetch, then the browser rungs on a 403, 429 or bot wall, then Bright Data — so a page a plain fetch cannot reach is still read. And it is stored: `url/<2>/<4>/<16 hex of the URL's SHA-1>/url-<hash>-{meta.md, page.md, page.html|pdf|txt}`, the URL normalised first (host lowercased, fragment and tracking parameters dropped). The same URL within `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) is served from disk, a failed fetch is not repeated within `FH_LIBRARY_RETRY_AFTER_DAYS` (1) — an audit that runs twice over one review reads its links once. A PDF comes back as Markdown (`fetch` would return its bytes as text). `title_only` returns `Contents of <url>:` and the title line alone — the page is still fetched and stored in full — so a link check costs a few tokens instead of a page. The log says which — `fetch example.com/… cached html … https://example.com/drugs/x` or `fetched 02.3s html retrieved_by=browser hetzner … <url>` (the ID column holds the URL without scheme, cut to a PubMed ID's width; the full URL ends the line); `library_info` counts the pages under `url`, a failed fetch as level `title`.
|
|
153
157
|
|
package/dist/index.js
CHANGED
|
@@ -1725,6 +1725,7 @@ var VARIABLES = [
|
|
|
1725
1725
|
"FH_LIBRARY_RETRY_AFTER_DAYS",
|
|
1726
1726
|
"FH_LIBRARY_RETRY_BLOCKED",
|
|
1727
1727
|
"FH_LIBRARY_REFRESH_AFTER_DAYS",
|
|
1728
|
+
"FH_LIBRARY_SEARCH_CACHE_DAYS",
|
|
1728
1729
|
"FH_LIBRARY_NCBI_API_KEY",
|
|
1729
1730
|
"FH_LIBRARY_BRIGHTDATA_API_TOKEN",
|
|
1730
1731
|
"FH_LIBRARY_BRIGHTDATA_ZONE"
|
|
@@ -1734,7 +1735,8 @@ function settings(env = process.env) {
|
|
|
1734
1735
|
return {
|
|
1735
1736
|
retry_after_days: num("FH_LIBRARY_RETRY_AFTER_DAYS", 1, env),
|
|
1736
1737
|
retry_blocked: bool("FH_LIBRARY_RETRY_BLOCKED", false, env),
|
|
1737
|
-
refresh_after_days: num("FH_LIBRARY_REFRESH_AFTER_DAYS", 30, env)
|
|
1738
|
+
refresh_after_days: num("FH_LIBRARY_REFRESH_AFTER_DAYS", 30, env),
|
|
1739
|
+
search_cache_days: num("FH_LIBRARY_SEARCH_CACHE_DAYS", 7, env)
|
|
1738
1740
|
};
|
|
1739
1741
|
}
|
|
1740
1742
|
function proxyHeaders(spec) {
|
|
@@ -1809,11 +1811,11 @@ function log(tool, id, message) {
|
|
|
1809
1811
|
}
|
|
1810
1812
|
|
|
1811
1813
|
// src/stats.ts
|
|
1812
|
-
import { mkdirSync as mkdirSync4, readdirSync as readdirSync3, readFileSync as readFileSync5, statSync as statSync2, unlinkSync as
|
|
1814
|
+
import { mkdirSync as mkdirSync4, readdirSync as readdirSync3, readFileSync as readFileSync5, statSync as statSync2, unlinkSync as unlinkSync3 } from "fs";
|
|
1813
1815
|
import { join as join7 } from "path";
|
|
1814
1816
|
|
|
1815
1817
|
// src/entry.ts
|
|
1816
|
-
import { copyFileSync, existsSync as existsSync5, mkdirSync as mkdirSync3, readdirSync as readdirSync2, readFileSync as readFileSync4, renameSync as renameSync2, rmSync, writeFileSync } from "fs";
|
|
1818
|
+
import { copyFileSync, existsSync as existsSync5, mkdirSync as mkdirSync3, readdirSync as readdirSync2, readFileSync as readFileSync4, renameSync as renameSync2, rmSync, unlinkSync as unlinkSync2, writeFileSync } from "fs";
|
|
1817
1819
|
import { join as join6 } from "path";
|
|
1818
1820
|
import { basename, dirname } from "path";
|
|
1819
1821
|
|
|
@@ -2017,11 +2019,11 @@ function inline(nodes) {
|
|
|
2017
2019
|
return out;
|
|
2018
2020
|
}
|
|
2019
2021
|
function table(n) {
|
|
2020
|
-
const
|
|
2022
|
+
const row3 = (x) => kids2(x).filter((c) => ["td", "th"].includes(tagOf2(c))).map((c) => squash2(inline(kids2(c))).replace(/\|/g, "\\|"));
|
|
2021
2023
|
const collect = (nodes, into) => {
|
|
2022
2024
|
for (const x of nodes) {
|
|
2023
2025
|
if ("#text" in x) continue;
|
|
2024
|
-
if (tagOf2(x) === "tr") into.push(
|
|
2026
|
+
if (tagOf2(x) === "tr") into.push(row3(x));
|
|
2025
2027
|
else if (!["thead", "tfoot"].includes(tagOf2(x))) collect(kids2(x), into);
|
|
2026
2028
|
}
|
|
2027
2029
|
};
|
|
@@ -2030,7 +2032,7 @@ function table(n) {
|
|
|
2030
2032
|
const t = tagOf2(x);
|
|
2031
2033
|
if (t === "thead") collect(kids2(x), head);
|
|
2032
2034
|
else if (t === "tfoot") collect(kids2(x), foot);
|
|
2033
|
-
else if (t === "tr") body.push(
|
|
2035
|
+
else if (t === "tr") body.push(row3(x));
|
|
2034
2036
|
else if (!("#text" in x) && t !== "caption") collect(kids2(x), body);
|
|
2035
2037
|
}
|
|
2036
2038
|
const rows = [...head, ...body];
|
|
@@ -2636,6 +2638,10 @@ function writeAtomic(file2, data) {
|
|
|
2636
2638
|
}
|
|
2637
2639
|
function writeMeta(p, m) {
|
|
2638
2640
|
writeAtomic(p.meta, renderMeta(m));
|
|
2641
|
+
try {
|
|
2642
|
+
unlinkSync2(join6(p.entryDir, `${p.entry}-missing.md`));
|
|
2643
|
+
} catch {
|
|
2644
|
+
}
|
|
2639
2645
|
}
|
|
2640
2646
|
function copyEntry(from, to) {
|
|
2641
2647
|
const tmp = `${to.entryDir}.${process.pid}.tmp`, old = `${to.entryDir}.${process.pid}.old`;
|
|
@@ -2917,14 +2923,14 @@ var STALE_MS = 20 * 6e4;
|
|
|
2917
2923
|
var stats = null;
|
|
2918
2924
|
var file = null;
|
|
2919
2925
|
function startStats(pkg2, libraryDir = FH_LIBRARY_DIR) {
|
|
2920
|
-
stats = { host: HOST, pid: process.pid, package: pkg2.name, version: pkg2.version, started: (/* @__PURE__ */ new Date()).toISOString(), verify: { hits: 0, misses: 0 }, fetch: { hits: 0, misses: 0 }, search: { hits: 0, misses: 0 } };
|
|
2926
|
+
stats = { host: HOST, pid: process.pid, package: pkg2.name, version: pkg2.version, started: (/* @__PURE__ */ new Date()).toISOString(), verify: { hits: 0, misses: 0 }, fetch: { hits: 0, misses: 0 }, search: { hits: 0, misses: 0 }, pubmed: { hits: 0, misses: 0 } };
|
|
2921
2927
|
file = join7(libraryDir, ".servers", `${HOST}-${process.pid}.json`);
|
|
2922
2928
|
sweep(join7(libraryDir, ".servers"));
|
|
2923
2929
|
publish();
|
|
2924
2930
|
setInterval(publish, HEARTBEAT_MS).unref();
|
|
2925
2931
|
const stop = () => {
|
|
2926
2932
|
try {
|
|
2927
|
-
if (file)
|
|
2933
|
+
if (file) unlinkSync3(file);
|
|
2928
2934
|
} catch {
|
|
2929
2935
|
}
|
|
2930
2936
|
};
|
|
@@ -2945,7 +2951,7 @@ function sweep(dir, staleMs = STALE_MS) {
|
|
|
2945
2951
|
const p = join7(dir, name);
|
|
2946
2952
|
try {
|
|
2947
2953
|
if (Date.now() - statSync2(p).mtimeMs > staleMs) {
|
|
2948
|
-
|
|
2954
|
+
unlinkSync3(p);
|
|
2949
2955
|
n++;
|
|
2950
2956
|
}
|
|
2951
2957
|
} catch {
|
|
@@ -2953,9 +2959,9 @@ function sweep(dir, staleMs = STALE_MS) {
|
|
|
2953
2959
|
}
|
|
2954
2960
|
return n;
|
|
2955
2961
|
}
|
|
2956
|
-
function bump(kind, hit) {
|
|
2962
|
+
function bump(kind, hit, n = 1) {
|
|
2957
2963
|
if (!stats) return;
|
|
2958
|
-
stats[kind][hit ? "hits" : "misses"]
|
|
2964
|
+
stats[kind][hit ? "hits" : "misses"] += n;
|
|
2959
2965
|
publish();
|
|
2960
2966
|
}
|
|
2961
2967
|
var currentStats = () => stats;
|
|
@@ -2969,7 +2975,7 @@ function publish() {
|
|
|
2969
2975
|
}
|
|
2970
2976
|
|
|
2971
2977
|
// src/tools.ts
|
|
2972
|
-
import { existsSync as
|
|
2978
|
+
import { existsSync as existsSync13, readFileSync as readFileSync13 } from "fs";
|
|
2973
2979
|
import { z } from "zod";
|
|
2974
2980
|
|
|
2975
2981
|
// src/cascade.ts
|
|
@@ -3999,7 +4005,7 @@ async function verifyCached(claim, id, cfg, decide, libraryDir = FH_LIBRARY_DIR,
|
|
|
3999
4005
|
var DROP = /* @__PURE__ */ new Set(["entry_dir", "library_dir", "files"]);
|
|
4000
4006
|
var compact2 = (o) => JSON.stringify(o);
|
|
4001
4007
|
function lean(o) {
|
|
4002
|
-
if (Array.isArray(o)) return o.map((
|
|
4008
|
+
if (Array.isArray(o)) return o.map((row3) => row3 && typeof row3 === "object" && !Array.isArray(row3) ? leanObject(row3) : row3);
|
|
4003
4009
|
if (o && typeof o === "object") return leanObject(o);
|
|
4004
4010
|
return o;
|
|
4005
4011
|
}
|
|
@@ -4017,7 +4023,7 @@ var leanJson = (o) => compact2(lean(o));
|
|
|
4017
4023
|
import { createHash as createHash4 } from "crypto";
|
|
4018
4024
|
import { existsSync as existsSync10, mkdirSync as mkdirSync7, readFileSync as readFileSync10 } from "fs";
|
|
4019
4025
|
import { dirname as dirname4, join as join11 } from "path";
|
|
4020
|
-
var
|
|
4026
|
+
var searchCacheDays = () => settings().search_cache_days;
|
|
4021
4027
|
var API2 = "https://clinicaltrials.gov/api/v2/studies";
|
|
4022
4028
|
var FIELDS = ["NCTId", "BriefTitle", "OverallStatus", "WhyStopped", "Phase", "StudyType", "EnrollmentCount", "EnrollmentType", "StartDate", "PrimaryCompletionDate", "CompletionDate", "LastUpdatePostDate", "PrimaryOutcomeMeasure", "PrimaryOutcomeTimeFrame", "Condition", "InterventionName", "LeadSponsorName", "HasResults"];
|
|
4023
4029
|
var ONGOING = ["RECRUITING", "NOT_YET_RECRUITING", "ACTIVE_NOT_RECRUITING", "ENROLLING_BY_INVITATION"];
|
|
@@ -4109,7 +4115,7 @@ async function searchTrials(raw, libraryDir = FH_LIBRARY_DIR, fetchImpl = liveFe
|
|
|
4109
4115
|
if (existsSync10(file2)) {
|
|
4110
4116
|
try {
|
|
4111
4117
|
const r = JSON.parse(readFileSync10(file2, "utf8"));
|
|
4112
|
-
if (daysSince(r.searched) <
|
|
4118
|
+
if (daysSince(r.searched) < searchCacheDays()) return { ok: true, result: r, cached: true };
|
|
4113
4119
|
} catch {
|
|
4114
4120
|
}
|
|
4115
4121
|
}
|
|
@@ -4138,6 +4144,123 @@ function trialsLogId(q) {
|
|
|
4138
4144
|
return s.length > 10 ? s.slice(0, 9) + "\u2026" : s || "-";
|
|
4139
4145
|
}
|
|
4140
4146
|
|
|
4147
|
+
// src/pubmedsearch.ts
|
|
4148
|
+
import { createHash as createHash5 } from "crypto";
|
|
4149
|
+
import { existsSync as existsSync11, mkdirSync as mkdirSync8, readFileSync as readFileSync11 } from "fs";
|
|
4150
|
+
import { dirname as dirname5 } from "path";
|
|
4151
|
+
var MAX_RESULTS2 = 100;
|
|
4152
|
+
var DEFAULT_RESULTS2 = 20;
|
|
4153
|
+
var COLUMNS2 = ["pmid", "title", "first_author", "year", "journal", "publication_types", "pmcid", "level"];
|
|
4154
|
+
var EUTILS = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils";
|
|
4155
|
+
var ident = (u) => {
|
|
4156
|
+
if (process.env.FH_LIBRARY_NCBI_API_KEY) u.searchParams.set("api_key", process.env.FH_LIBRARY_NCBI_API_KEY);
|
|
4157
|
+
u.searchParams.set("tool", "fh-library");
|
|
4158
|
+
u.searchParams.set("email", contactEmail());
|
|
4159
|
+
return u.toString();
|
|
4160
|
+
};
|
|
4161
|
+
function yearRange(years) {
|
|
4162
|
+
const t = (years ?? "").replace(/\s+/g, "");
|
|
4163
|
+
if (!t) return null;
|
|
4164
|
+
const m = /^(\d{4})?(-)?(\d{4})?$/.exec(t);
|
|
4165
|
+
if (!m || !m[1] && !m[3]) return { error: `years must be "2020-2026", "2022-", "-2019" or "2022", not "${years}"` };
|
|
4166
|
+
const min = m[1] ?? "1000", max = m[3] ?? (m[2] ? String((/* @__PURE__ */ new Date()).getFullYear()) : m[1]);
|
|
4167
|
+
return { min, max };
|
|
4168
|
+
}
|
|
4169
|
+
function normalizeQuery2(q) {
|
|
4170
|
+
const query = (q.query ?? "").replace(/\s+/g, " ").trim();
|
|
4171
|
+
if (!query) return { error: "empty query" };
|
|
4172
|
+
const sort = q.sort ?? "relevance";
|
|
4173
|
+
if (!["relevance", "newest"].includes(sort)) return { error: `sort must be relevance or newest, not "${String(q.sort)}"` };
|
|
4174
|
+
const yr = yearRange(q.years);
|
|
4175
|
+
if (yr && "error" in yr) return yr;
|
|
4176
|
+
const out = { query, sort, max_results: Math.min(MAX_RESULTS2, Math.max(1, Math.floor(q.max_results ?? DEFAULT_RESULTS2))) };
|
|
4177
|
+
if (yr) out.years = `${yr.min === "1000" ? "" : yr.min}-${yr.max}`;
|
|
4178
|
+
return out;
|
|
4179
|
+
}
|
|
4180
|
+
var searchKey2 = (q) => createHash5("sha1").update(JSON.stringify(["pubmed", q.query, q.sort, q.years ?? "", q.max_results])).digest("hex").slice(0, 16);
|
|
4181
|
+
function esearchUrl(q) {
|
|
4182
|
+
const u = new URL(`${EUTILS}/esearch.fcgi`);
|
|
4183
|
+
u.searchParams.set("db", "pubmed");
|
|
4184
|
+
u.searchParams.set("term", q.query);
|
|
4185
|
+
u.searchParams.set("retmax", String(q.max_results));
|
|
4186
|
+
u.searchParams.set("retmode", "json");
|
|
4187
|
+
u.searchParams.set("sort", q.sort === "newest" ? "pub_date" : "relevance");
|
|
4188
|
+
const yr = yearRange(q.years);
|
|
4189
|
+
if (yr && !("error" in yr)) {
|
|
4190
|
+
u.searchParams.set("datetype", "pdat");
|
|
4191
|
+
u.searchParams.set("mindate", yr.min);
|
|
4192
|
+
u.searchParams.set("maxdate", yr.max);
|
|
4193
|
+
}
|
|
4194
|
+
return ident(u);
|
|
4195
|
+
}
|
|
4196
|
+
function esummaryUrl(pmids) {
|
|
4197
|
+
const u = new URL(`${EUTILS}/esummary.fcgi`);
|
|
4198
|
+
u.searchParams.set("db", "pubmed");
|
|
4199
|
+
u.searchParams.set("id", pmids.join(","));
|
|
4200
|
+
u.searchParams.set("retmode", "json");
|
|
4201
|
+
return ident(u);
|
|
4202
|
+
}
|
|
4203
|
+
function parseEsearch(json2) {
|
|
4204
|
+
const d = JSON.parse(json2);
|
|
4205
|
+
const r = d?.esearchresult ?? {};
|
|
4206
|
+
return { total: Number(r.count ?? 0) || 0, ids: Array.isArray(r.idlist) ? r.idlist.map(String) : [] };
|
|
4207
|
+
}
|
|
4208
|
+
function parseEsummary(json2, ids, libraryDir) {
|
|
4209
|
+
const d = JSON.parse(json2);
|
|
4210
|
+
const res = d?.result ?? {};
|
|
4211
|
+
const rows = [];
|
|
4212
|
+
for (const id of ids) {
|
|
4213
|
+
const s = res[id];
|
|
4214
|
+
if (!s || s.error) continue;
|
|
4215
|
+
const year = /\b(\d{4})\b/.exec(String(s.pubdate ?? ""))?.[1] ?? null;
|
|
4216
|
+
const first = String(s.sortfirstauthor ?? (Array.isArray(s.authors) && s.authors[0]?.name) ?? "").replace(/\s+[A-Z]{1,3}$/, "").trim() || null;
|
|
4217
|
+
const pmc2 = (Array.isArray(s.articleids) ? s.articleids : []).find((a) => a.idtype === "pmc")?.value ?? null;
|
|
4218
|
+
const types = Array.isArray(s.pubtype) ? s.pubtype.filter((t) => t !== "Journal Article").join("; ") : "";
|
|
4219
|
+
const paths = entryPaths(libraryDir, "pmid", id);
|
|
4220
|
+
const level = entryExists(paths) ? String(readStatus(paths, emptyStatus(`pmid:${id}`, "pmid", id, libraryDir)).level) : null;
|
|
4221
|
+
rows.push([id, String(s.title ?? "").replace(/\.$/, "") || null, first, year, s.source ? String(s.source) : null, types || null, pmc2 ? String(pmc2) : null, level]);
|
|
4222
|
+
}
|
|
4223
|
+
return rows;
|
|
4224
|
+
}
|
|
4225
|
+
var live = (url) => fetchText(url, { "User-Agent": "fh-library", Accept: "application/json" });
|
|
4226
|
+
async function searchArticles(raw, libraryDir = FH_LIBRARY_DIR, fetchImpl = live) {
|
|
4227
|
+
const q = normalizeQuery2(raw);
|
|
4228
|
+
if ("error" in q) return { ok: false, error: q.error };
|
|
4229
|
+
const file2 = searchFile(libraryDir, searchKey2(q));
|
|
4230
|
+
if (existsSync11(file2)) {
|
|
4231
|
+
try {
|
|
4232
|
+
const r = JSON.parse(readFileSync11(file2, "utf8"));
|
|
4233
|
+
if (daysSince(r.searched) < searchCacheDays()) return { ok: true, result: r, cached: true };
|
|
4234
|
+
} catch {
|
|
4235
|
+
}
|
|
4236
|
+
}
|
|
4237
|
+
let total, ids;
|
|
4238
|
+
try {
|
|
4239
|
+
({ total, ids } = parseEsearch(await fetchImpl(esearchUrl(q))));
|
|
4240
|
+
} catch (e) {
|
|
4241
|
+
return { ok: false, error: `pubmed: ${e instanceof Error ? e.message : String(e)}` };
|
|
4242
|
+
}
|
|
4243
|
+
let rows = [];
|
|
4244
|
+
if (ids.length) {
|
|
4245
|
+
try {
|
|
4246
|
+
rows = parseEsummary(await fetchImpl(esummaryUrl(ids)), ids, libraryDir);
|
|
4247
|
+
} catch (e) {
|
|
4248
|
+
return { ok: false, error: `pubmed: ${e instanceof Error ? e.message : String(e)}` };
|
|
4249
|
+
}
|
|
4250
|
+
}
|
|
4251
|
+
const result = { source: "pubmed", query: q, total, columns: COLUMNS2, rows, searched: today() };
|
|
4252
|
+
try {
|
|
4253
|
+
mkdirSync8(dirname5(file2), { recursive: true });
|
|
4254
|
+
writeAtomic(file2, JSON.stringify(result, null, 2) + "\n");
|
|
4255
|
+
} catch {
|
|
4256
|
+
}
|
|
4257
|
+
return { ok: true, result, cached: false };
|
|
4258
|
+
}
|
|
4259
|
+
function logId2(q) {
|
|
4260
|
+
const s = q.query.replace(/\s+/g, " ");
|
|
4261
|
+
return s.length > 10 ? s.slice(0, 9) + "\u2026" : s;
|
|
4262
|
+
}
|
|
4263
|
+
|
|
4141
4264
|
// src/trialrecord.ts
|
|
4142
4265
|
var list3 = (a) => Array.isArray(a) ? a : [];
|
|
4143
4266
|
var para2 = (s, max = Infinity) => {
|
|
@@ -4209,10 +4332,120 @@ function resultsText(nct, md, maxChars) {
|
|
|
4209
4332
|
return { nct, has_results: true, text: truncated ? part.slice(0, maxChars).trimEnd() + "\n\n[truncated]" : part, truncated, outcomes, note: truncated ? "cut at max_chars; library_read(id, section: <an outcome title>) returns one measure whole" : null };
|
|
4210
4333
|
}
|
|
4211
4334
|
|
|
4335
|
+
// src/missing.ts
|
|
4336
|
+
import { existsSync as existsSync12, mkdirSync as mkdirSync9, readFileSync as readFileSync12 } from "fs";
|
|
4337
|
+
import { join as join12 } from "path";
|
|
4338
|
+
var missingFile = (p) => join12(p.entryDir, `${p.entry}-missing.md`);
|
|
4339
|
+
function readMissing(p) {
|
|
4340
|
+
const f = missingFile(p);
|
|
4341
|
+
if (!existsSync12(f)) return null;
|
|
4342
|
+
const md = readFileSync12(f, "utf8");
|
|
4343
|
+
const get = (k) => /^$/.test(k) ? "" : (new RegExp(`^${k}:[ \\t]*(.*)$`, "m").exec(md)?.[1] ?? "").trim();
|
|
4344
|
+
const checked = get("checked"), reason = get("reason");
|
|
4345
|
+
return checked ? { checked, reason: reason || "not found" } : null;
|
|
4346
|
+
}
|
|
4347
|
+
function freshMissing(p, retryDays) {
|
|
4348
|
+
const m = readMissing(p);
|
|
4349
|
+
return m && daysSince(m.checked) < retryDays ? m : null;
|
|
4350
|
+
}
|
|
4351
|
+
function rememberMissing(p, reason) {
|
|
4352
|
+
const m = { checked: today(), reason };
|
|
4353
|
+
mkdirSync9(p.entryDir, { recursive: true });
|
|
4354
|
+
writeAtomic(missingFile(p), `checked: ${m.checked}
|
|
4355
|
+
reason: ${m.reason}
|
|
4356
|
+
`);
|
|
4357
|
+
return m;
|
|
4358
|
+
}
|
|
4359
|
+
var missingNote = (m, retryDays) => `${m.reason} (checked ${m.checked}; asked again after ${retryDays} day${retryDays === 1 ? "" : "s"})`;
|
|
4360
|
+
|
|
4361
|
+
// src/pubmedbatch.ts
|
|
4362
|
+
var MAX_PMIDS = 200;
|
|
4363
|
+
var COLUMNS3 = ["pmid", "title", "first_author", "year", "publication_types", "retracted", "level"];
|
|
4364
|
+
var EFETCH2 = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi";
|
|
4365
|
+
function efetchBatchUrl(pmids) {
|
|
4366
|
+
const u = new URL(EFETCH2);
|
|
4367
|
+
u.searchParams.set("db", "pubmed");
|
|
4368
|
+
u.searchParams.set("id", pmids.join(","));
|
|
4369
|
+
u.searchParams.set("retmode", "xml");
|
|
4370
|
+
if (process.env.FH_LIBRARY_NCBI_API_KEY) u.searchParams.set("api_key", process.env.FH_LIBRARY_NCBI_API_KEY);
|
|
4371
|
+
u.searchParams.set("tool", "fh-library");
|
|
4372
|
+
u.searchParams.set("email", contactEmail());
|
|
4373
|
+
return u.toString();
|
|
4374
|
+
}
|
|
4375
|
+
function splitArticles(xml) {
|
|
4376
|
+
const out = /* @__PURE__ */ new Map();
|
|
4377
|
+
for (const m of xml.matchAll(/<(PubmedArticle|PubmedBookArticle)\b[\s\S]*?<\/\1>/g)) {
|
|
4378
|
+
const pmid = /<PMID[^>]*>(\d+)<\/PMID>/.exec(m[0])?.[1];
|
|
4379
|
+
if (pmid && !out.has(pmid)) out.set(pmid, `<?xml version="1.0" ?>
|
|
4380
|
+
<PubmedArticleSet>${m[0]}</PubmedArticleSet>
|
|
4381
|
+
`);
|
|
4382
|
+
}
|
|
4383
|
+
return out;
|
|
4384
|
+
}
|
|
4385
|
+
var row2 = (pmid, s, level) => s ? [pmid, s.title, s.first_author, s.year, s.publication_types.length ? s.publication_types.join("; ") : null, s.retracted, level] : [pmid, null, null, null, null, null, level];
|
|
4386
|
+
async function fetchArticles(ids, libraryDir = FH_LIBRARY_DIR, converter = "fh-library", fetchImpl = fetchText) {
|
|
4387
|
+
const result = { columns: COLUMNS3, rows: [], stored: 0, cached: 0, not_found: 0, invalid: 0 };
|
|
4388
|
+
const order2 = [];
|
|
4389
|
+
const invalid = /* @__PURE__ */ new Map();
|
|
4390
|
+
for (const raw of ids.slice(0, MAX_PMIDS)) {
|
|
4391
|
+
const p = parseId(String(raw));
|
|
4392
|
+
if (p.error || p.type !== "pmid" || !/^\d{1,8}$/.test(p.id)) {
|
|
4393
|
+
const k = String(raw).trim();
|
|
4394
|
+
if (!invalid.has(k)) {
|
|
4395
|
+
invalid.set(k, "invalid");
|
|
4396
|
+
order2.push(k);
|
|
4397
|
+
}
|
|
4398
|
+
continue;
|
|
4399
|
+
}
|
|
4400
|
+
if (!order2.includes(p.id)) order2.push(p.id);
|
|
4401
|
+
}
|
|
4402
|
+
const status = (pmid) => {
|
|
4403
|
+
const paths = entryPaths(libraryDir, "pmid", pmid);
|
|
4404
|
+
return entryExists(paths) ? readStatus(paths, emptyStatus(`pmid:${pmid}`, "pmid", pmid, libraryDir)) : null;
|
|
4405
|
+
};
|
|
4406
|
+
const retry = settings().retry_after_days;
|
|
4407
|
+
const known = (pmid) => !!freshMissing(entryPaths(libraryDir, "pmid", pmid), retry);
|
|
4408
|
+
const missing = order2.filter((k) => !invalid.has(k) && !status(k) && !known(k));
|
|
4409
|
+
let records = /* @__PURE__ */ new Map();
|
|
4410
|
+
if (missing.length) records = splitArticles(await fetchImpl(efetchBatchUrl(missing)));
|
|
4411
|
+
for (const k of order2) {
|
|
4412
|
+
if (invalid.has(k)) {
|
|
4413
|
+
result.rows.push(row2(k, null, "invalid"));
|
|
4414
|
+
result.invalid++;
|
|
4415
|
+
continue;
|
|
4416
|
+
}
|
|
4417
|
+
let s = status(k);
|
|
4418
|
+
if (s) {
|
|
4419
|
+
result.rows.push(row2(k, s, String(s.level)));
|
|
4420
|
+
result.cached++;
|
|
4421
|
+
continue;
|
|
4422
|
+
}
|
|
4423
|
+
const paths = entryPaths(libraryDir, "pmid", k);
|
|
4424
|
+
if (known(k)) {
|
|
4425
|
+
result.rows.push(row2(k, null, "not found"));
|
|
4426
|
+
result.not_found++;
|
|
4427
|
+
continue;
|
|
4428
|
+
}
|
|
4429
|
+
const xml = records.get(k);
|
|
4430
|
+
const record = xml ? parsePubmedXml(xml) : null;
|
|
4431
|
+
if (!record) {
|
|
4432
|
+
rememberMissing(paths, "pmid not found");
|
|
4433
|
+
result.rows.push(row2(k, null, "not found"));
|
|
4434
|
+
result.not_found++;
|
|
4435
|
+
continue;
|
|
4436
|
+
}
|
|
4437
|
+
if (!entryExists(paths)) writeEntry(paths, record, xml, converter);
|
|
4438
|
+
s = status(k);
|
|
4439
|
+
result.rows.push(row2(k, s, String(s.level)));
|
|
4440
|
+
result.stored++;
|
|
4441
|
+
}
|
|
4442
|
+
return result;
|
|
4443
|
+
}
|
|
4444
|
+
|
|
4212
4445
|
// src/tools.ts
|
|
4213
4446
|
var pkg = (() => {
|
|
4214
4447
|
try {
|
|
4215
|
-
return JSON.parse(
|
|
4448
|
+
return JSON.parse(readFileSync13(new URL("../package.json", import.meta.url), "utf8"));
|
|
4216
4449
|
} catch {
|
|
4217
4450
|
return { name: "fh-library", version: "0.0.0" };
|
|
4218
4451
|
}
|
|
@@ -4232,6 +4465,8 @@ Tools
|
|
|
4232
4465
|
- library_status(id) \u2014 the status of one record without downloading; level "missing" when it is not stored.
|
|
4233
4466
|
- library_read(id, section?, query?, max_chars?, max_hits?) \u2014 one small piece of a stored document: its outline, one section, or the passages matching a query.
|
|
4234
4467
|
- library_verify(claim, id) \u2014 does this one source state this one finding? A decision model judges the stored text and returns a verdict.
|
|
4468
|
+
- library_pubmed_fetch_articles(pmids) \u2014 title, first author, year, publication types, retracted and level for up to 200 PMIDs at once; faster and far smaller than asking PubMed, known papers from disk, the rest in one request and stored. For abstract or full text: library_read.
|
|
4469
|
+
- library_pubmed_search_articles(query, max_results?, sort?, years?) \u2014 search PubMed in its own syntax: the total and rows in columns order (PMID, title, first author, year, journal, publication types, PMCID, library level); faster and far smaller than asking PubMed. Cached a week.
|
|
4235
4470
|
- library_clinicaltrials_search_studies(query | intervention | condition, status?, sort?, max_results?) \u2014 search ClinicalTrials.gov: the total and rows in columns order (NCT ID, title, status, phase, enrollment, dates, primary outcome, conditions, interventions, has_results). Cached a week.
|
|
4236
4471
|
- library_clinicaltrials_get_study_record(nct) \u2014 one trial as a review needs it (status, design, enrollment, dates, outcomes, eligibility, result publications, summary), from the stored record; added first when missing.
|
|
4237
4472
|
- library_clinicaltrials_get_study_results(nct, max_chars?) \u2014 a trial's posted results as Markdown, capped; the outcome titles listed for library_read.
|
|
@@ -4241,7 +4476,7 @@ Hints
|
|
|
4241
4476
|
- library_read and library_verify store a missing record themselves; the library_read answer then carries added: true. No library_add necessary first.
|
|
4242
4477
|
- citable false \u2014 the paper is retracted (a retraction notice, or PubMed's publication type), or holds nothing to read: no abstract and no full text. comment says which; corrections lists the notices as {type, pmid, source}.
|
|
4243
4478
|
- level \u2014 what is stored: "full_text", "abstract" (no full text could be found; attempts names each source tried and why it failed) or "title" (not even an abstract; citable false).
|
|
4244
|
-
- error "pmid not found"
|
|
4479
|
+
- error "pmid not found", "doi not found", "ctgov: not a study record", "dailymed: not an SPL document" \u2014 no service knows the ID. Remembered for a day (FH_LIBRARY_RETRY_AFTER_DAYS): the same call answers from disk with the date it was checked; after that the service is asked again. "invalid id" \u2014 the ID itself is malformed; the same call gives the same answer.
|
|
4245
4480
|
- Calls may run in parallel: library_add, library_read and library_verify. Two calls on the same missing record share one add.
|
|
4246
4481
|
- library_fetch takes any URL that is not a paper, label or trial record; those have IDs, and library_add, library_read and library_verify work on the stored document. A second pass over the same review reads every page from disk; a failed fetch is remembered for a day.
|
|
4247
4482
|
- To verify that a web page exists and what it is: library_fetch(url, title_only: true). The answer is "Contents of <url>:" and the page's title line \u2014 a few tokens \u2014 or the fetch error; the page itself is stored in full, so a later library_fetch of the same URL reads from disk.
|
|
@@ -4337,7 +4572,8 @@ function registerTools(server2) {
|
|
|
4337
4572
|
const xml = await fetchSpl(setid);
|
|
4338
4573
|
const s = parseSpl(xml);
|
|
4339
4574
|
if (!s) {
|
|
4340
|
-
|
|
4575
|
+
rememberMissing(paths, "dailymed: not an SPL document");
|
|
4576
|
+
log("library_add", requested, "dailymed failed: not an SPL document (remembered)");
|
|
4341
4577
|
return { ...status, error: "dailymed: not an SPL document" };
|
|
4342
4578
|
}
|
|
4343
4579
|
const meta = writeLabelEntry(paths, s, xml, `${pkg.name} ${pkg.version}`);
|
|
@@ -4345,7 +4581,9 @@ function registerTools(server2) {
|
|
|
4345
4581
|
return statusFromMeta(paths, meta, status);
|
|
4346
4582
|
} catch (e) {
|
|
4347
4583
|
const msg = e instanceof Error ? e.message : String(e);
|
|
4348
|
-
|
|
4584
|
+
const gone = /not an SPL document|^HTTP 404\b/.test(msg);
|
|
4585
|
+
if (gone) rememberMissing(paths, msg.startsWith("HTTP 404") ? "dailymed: no label for this set ID (HTTP 404)" : msg);
|
|
4586
|
+
log("library_add", requested, `dailymed failed: ${msg}${gone ? " (remembered)" : ""}`);
|
|
4349
4587
|
return { ...status, error: msg };
|
|
4350
4588
|
}
|
|
4351
4589
|
}
|
|
@@ -4378,7 +4616,8 @@ function registerTools(server2) {
|
|
|
4378
4616
|
const json2 = await fetchStudy(nctid);
|
|
4379
4617
|
const s = parseStudy(json2);
|
|
4380
4618
|
if (!s) {
|
|
4381
|
-
|
|
4619
|
+
rememberMissing(paths, "ctgov: not a study record");
|
|
4620
|
+
log("library_add", requested, "ctgov failed: not a study record (remembered)");
|
|
4382
4621
|
return { ...status, error: "ctgov: not a study record" };
|
|
4383
4622
|
}
|
|
4384
4623
|
const meta = writeStudyEntry(paths, s, json2, `${pkg.name} ${pkg.version}`);
|
|
@@ -4386,7 +4625,9 @@ function registerTools(server2) {
|
|
|
4386
4625
|
return statusFromMeta(paths, meta, status);
|
|
4387
4626
|
} catch (e) {
|
|
4388
4627
|
const msg = e instanceof Error ? e.message : String(e);
|
|
4389
|
-
|
|
4628
|
+
const gone = /not a study record|^HTTP 404\b/.test(msg);
|
|
4629
|
+
if (gone) rememberMissing(paths, msg.startsWith("HTTP 404") ? "ctgov: no study for this NCT ID (HTTP 404)" : msg);
|
|
4630
|
+
log("library_add", requested, `ctgov failed: ${msg}${gone ? " (remembered)" : ""}`);
|
|
4390
4631
|
return { ...status, error: msg };
|
|
4391
4632
|
}
|
|
4392
4633
|
}
|
|
@@ -4409,6 +4650,14 @@ function registerTools(server2) {
|
|
|
4409
4650
|
async function addEntryNow(id) {
|
|
4410
4651
|
const t0 = Date.now();
|
|
4411
4652
|
let parsed = parseId(id);
|
|
4653
|
+
if (!parsed.error) {
|
|
4654
|
+
const mp = entryPaths(FH_LIBRARY_DIR, parsed.type, parsed.id);
|
|
4655
|
+
const miss2 = entryExists(mp) ? null : freshMissing(mp, settings().retry_after_days);
|
|
4656
|
+
if (miss2) {
|
|
4657
|
+
log("library_add", parsed.requested, `missing cached: ${miss2.reason} (${miss2.checked})`);
|
|
4658
|
+
return json({ ...emptyStatus(parsed.requested, parsed.type, parsed.id, FH_LIBRARY_DIR), entry: mp.entry, level: "missing", error: missingNote(miss2, settings().retry_after_days) });
|
|
4659
|
+
}
|
|
4660
|
+
}
|
|
4412
4661
|
if (parsed.error) {
|
|
4413
4662
|
log("library_add", parsed.requested, `error ${parsed.error}`);
|
|
4414
4663
|
return json({ ...emptyStatus(parsed.requested, parsed.type, parsed.id, FH_LIBRARY_DIR), error: parsed.error });
|
|
@@ -4503,7 +4752,8 @@ function registerTools(server2) {
|
|
|
4503
4752
|
}
|
|
4504
4753
|
}
|
|
4505
4754
|
if (!record) {
|
|
4506
|
-
|
|
4755
|
+
rememberMissing(paths, "doi not found");
|
|
4756
|
+
log("library_add", parsed.requested, "crossref failed: doi not found at Crossref or doi.org (remembered)");
|
|
4507
4757
|
return json({ ...status, error: "doi not found" });
|
|
4508
4758
|
}
|
|
4509
4759
|
if (!record.doi) record.doi = parsed.id;
|
|
@@ -4528,7 +4778,8 @@ function registerTools(server2) {
|
|
|
4528
4778
|
}
|
|
4529
4779
|
const record = parsePubmedXml(xml);
|
|
4530
4780
|
if (!record) {
|
|
4531
|
-
|
|
4781
|
+
rememberMissing(paths, "pmid not found");
|
|
4782
|
+
log("library_add", parsed.requested, "pubmed failed: pmid not found (remembered)");
|
|
4532
4783
|
return json({ ...status, error: "pmid not found" });
|
|
4533
4784
|
}
|
|
4534
4785
|
meta = writeEntry(paths, record, xml, `${pkg.name} ${pkg.version}`);
|
|
@@ -4557,8 +4808,9 @@ function registerTools(server2) {
|
|
|
4557
4808
|
log("library_status", parsed.requested, `level=${s.level}`);
|
|
4558
4809
|
return json(s);
|
|
4559
4810
|
}
|
|
4560
|
-
|
|
4561
|
-
|
|
4811
|
+
const miss2 = freshMissing(paths, settings().retry_after_days);
|
|
4812
|
+
log("library_status", parsed.requested, miss2 ? `level=missing (${miss2.reason}, ${miss2.checked})` : "level=missing");
|
|
4813
|
+
return json({ ...status, entry: paths.entry, level: "missing", ...miss2 ? { error: missingNote(miss2, settings().retry_after_days) } : {} });
|
|
4562
4814
|
}
|
|
4563
4815
|
);
|
|
4564
4816
|
server2.tool(
|
|
@@ -4589,8 +4841,8 @@ function registerTools(server2) {
|
|
|
4589
4841
|
}
|
|
4590
4842
|
added = true;
|
|
4591
4843
|
}
|
|
4592
|
-
const full =
|
|
4593
|
-
const md =
|
|
4844
|
+
const full = existsSync13(paths.fulltextMd);
|
|
4845
|
+
const md = readFileSync13(full ? paths.fulltextMd : paths.md, "utf8");
|
|
4594
4846
|
const text3 = full ? "full_text" : "abstract";
|
|
4595
4847
|
const base = added ? { requested: parsed.requested, entry: paths.entry, text: text3, added: true } : { requested: parsed.requested, entry: paths.entry, text: text3 };
|
|
4596
4848
|
const headings2 = () => outline(md, text3).sections;
|
|
@@ -4610,8 +4862,8 @@ function registerTools(server2) {
|
|
|
4610
4862
|
const cap = max_chars ?? 6e3;
|
|
4611
4863
|
let sec = readSection(md, section2, cap);
|
|
4612
4864
|
let via = "";
|
|
4613
|
-
if (!sec && /abstract/i.test(section2) && full &&
|
|
4614
|
-
sec = readSection(
|
|
4865
|
+
if (!sec && /abstract/i.test(section2) && full && existsSync13(paths.md)) {
|
|
4866
|
+
sec = readSection(readFileSync13(paths.md, "utf8"), "abstract", cap);
|
|
4615
4867
|
via = " (from the abstract file)";
|
|
4616
4868
|
}
|
|
4617
4869
|
if (!sec && /declaration/i.test(section2)) {
|
|
@@ -4680,7 +4932,7 @@ function registerTools(server2) {
|
|
|
4680
4932
|
}
|
|
4681
4933
|
log("library_info", null, `entries=${info.entries}${info.verify ? ` verify ${info.verify.model} ${info.verify.ok ? `ok ${info.verify.ms} ms` : `down: ${info.verify.error}`}` : " verify not configured"}`);
|
|
4682
4934
|
const s = currentStats();
|
|
4683
|
-
const cache = s ? { since: s.started, verify: s.verify, fetch: s.fetch, search: s.search } : null;
|
|
4935
|
+
const cache = s ? { since: s.started, verify: s.verify, fetch: s.fetch, search: s.search, pubmed: s.pubmed } : null;
|
|
4684
4936
|
return { content: [{ type: "text", text: compact2({ ...info, cache }) }] };
|
|
4685
4937
|
}
|
|
4686
4938
|
);
|
|
@@ -4711,9 +4963,46 @@ function registerTools(server2) {
|
|
|
4711
4963
|
return { content: [{ type: "text", text: r.error }], isError: true };
|
|
4712
4964
|
}
|
|
4713
4965
|
);
|
|
4966
|
+
server2.tool(
|
|
4967
|
+
"library_pubmed_fetch_articles",
|
|
4968
|
+
`Title, first author, year, publication types, retracted and library level for up to ${MAX_PMIDS} PubMed IDs in one call. Faster and far smaller than asking PubMed: known papers come from disk, the rest in one request, stored for every later call; rows as arrays in columns order, ~120 characters a paper. Level "not found": PubMed has no such record. For abstract or full text: library_read.`,
|
|
4969
|
+
{ pmids: z.array(z.string()).min(1).max(MAX_PMIDS).describe('PubMed IDs, "30955454" or "pmid:30955454", up to 200.') },
|
|
4970
|
+
async ({ pmids }) => {
|
|
4971
|
+
const t0 = Date.now();
|
|
4972
|
+
const r = await fetchArticles(pmids, FH_LIBRARY_DIR, `${pkg.name} ${pkg.version}`);
|
|
4973
|
+
if (r.cached) bump("pubmed", true, r.cached);
|
|
4974
|
+
if (r.stored) bump("pubmed", false, r.stored);
|
|
4975
|
+
const secs = ((Date.now() - t0) / 1e3).toFixed(1).padStart(4, "0") + "s";
|
|
4976
|
+
log("library_pubmed", `pmids:${r.rows.length}`, `fetch_articles ${secs} cached=${r.cached} stored=${r.stored}${r.not_found ? ` not_found=${r.not_found}` : ""}${r.invalid ? ` invalid=${r.invalid}` : ""}`);
|
|
4977
|
+
return { content: [{ type: "text", text: leanJson(r) }] };
|
|
4978
|
+
}
|
|
4979
|
+
);
|
|
4980
|
+
server2.tool(
|
|
4981
|
+
"library_pubmed_search_articles",
|
|
4982
|
+
`Search PubMed with its own query syntax (AND / OR, field tags like [pt], [ti], [mesh]); sort relevance | newest; years "2020-2026". Faster and far smaller than asking PubMed: the total and up to ${MAX_RESULTS2} rows as arrays in columns order \u2014 PMID, title, first author, year, journal, publication types, PMCID when free full text exists, and the library level when the paper is already stored. The same search is answered from disk for ${searchCacheDays()} days. For a batch of PMIDs: library_pubmed_fetch_articles; for abstract or full text: library_read.`,
|
|
4983
|
+
{
|
|
4984
|
+
query: z.string().describe('PubMed query, full syntax: "tadalafil AND (systematic review[pt] OR meta-analysis[pt])".'),
|
|
4985
|
+
max_results: z.number().int().min(1).max(MAX_RESULTS2).default(20).describe("Rows to return (1\u2013100)."),
|
|
4986
|
+
sort: z.enum(["relevance", "newest"]).default("relevance").describe("relevance (PubMed's ranking) or newest (publication date, latest first)."),
|
|
4987
|
+
years: z.string().optional().describe('Publication years: "2020-2026", "2022-", "-2019" or "2022".')
|
|
4988
|
+
},
|
|
4989
|
+
async (args) => {
|
|
4990
|
+
const t0 = Date.now();
|
|
4991
|
+
const r = await searchArticles(args, FH_LIBRARY_DIR);
|
|
4992
|
+
const secs = ((Date.now() - t0) / 1e3).toFixed(1).padStart(4, "0") + "s";
|
|
4993
|
+
if (!r.ok) {
|
|
4994
|
+
log("library_search", "pm:-", `error: ${r.error}`);
|
|
4995
|
+
return { content: [{ type: "text", text: compact2({ error: r.error }) }], isError: true };
|
|
4996
|
+
}
|
|
4997
|
+
bump("search", r.cached);
|
|
4998
|
+
const q = r.result.query;
|
|
4999
|
+
log("library_search", `pm:${logId2(q)}`, `${r.cached ? "cached" : `searched ${secs}`} rows=${r.result.rows.length} total=${r.result.total}${q.sort !== "relevance" ? ` sort=${q.sort}` : ""}${q.years ? ` years=${q.years}` : ""} | ${q.query}`);
|
|
5000
|
+
return { content: [{ type: "text", text: leanJson(r.result) }] };
|
|
5001
|
+
}
|
|
5002
|
+
);
|
|
4714
5003
|
server2.tool(
|
|
4715
5004
|
"library_clinicaltrials_search_studies",
|
|
4716
|
-
`Search ClinicalTrials.gov by free text, intervention or condition; status ongoing | completed | all; sort relevance | newest | largest | completion. Returns the total and up to ${MAX_RESULTS} rows as arrays in columns order: NCT ID, title, status, phase, enrollment, start, primary completion, primary outcome, conditions, interventions, has_results. Link: https://clinicaltrials.gov/study/<NCT ID>. Cached ${
|
|
5005
|
+
`Search ClinicalTrials.gov by free text, intervention or condition; status ongoing | completed | all; sort relevance | newest | largest | completion. Returns the total and up to ${MAX_RESULTS} rows as arrays in columns order: NCT ID, title, status, phase, enrollment, start, primary completion, primary outcome, conditions, interventions, has_results. Link: https://clinicaltrials.gov/study/<NCT ID>. Cached ${searchCacheDays()} days. One trial in detail: library_clinicaltrials_get_study_record.`,
|
|
4717
5006
|
{
|
|
4718
5007
|
query: z.string().optional().describe("Free-text search across all fields (title, conditions, interventions, summary, \u2026); AND / OR / NOT allowed."),
|
|
4719
5008
|
intervention: z.string().optional().describe('Intervention or treatment, e.g. "tadalafil", "cognitive behavioral therapy".'),
|
|
@@ -4760,7 +5049,7 @@ function registerTools(server2) {
|
|
|
4760
5049
|
return { content: [{ type: "text", text: compact2({ error: e.error }) }], isError: true };
|
|
4761
5050
|
}
|
|
4762
5051
|
const raw = e.paths.fulltextRaw("json");
|
|
4763
|
-
const r =
|
|
5052
|
+
const r = existsSync13(raw) ? recordSummary(readFileSync13(raw, "utf8")) : null;
|
|
4764
5053
|
if (!r) {
|
|
4765
5054
|
log("library_trials", e.requested, "record error: no stored record");
|
|
4766
5055
|
return { content: [{ type: "text", text: compact2({ error: "no stored record" }) }], isError: true };
|
|
@@ -4779,7 +5068,7 @@ function registerTools(server2) {
|
|
|
4779
5068
|
log("library_trials", nct, `results error: ${e.error}`);
|
|
4780
5069
|
return { content: [{ type: "text", text: compact2({ error: e.error }) }], isError: true };
|
|
4781
5070
|
}
|
|
4782
|
-
const md =
|
|
5071
|
+
const md = existsSync13(e.paths.fulltextMd) ? readFileSync13(e.paths.fulltextMd, "utf8") : "";
|
|
4783
5072
|
const r = resultsText(e.requested.replace(/^nct:/, "NCT"), md, max_chars);
|
|
4784
5073
|
log("library_trials", e.requested, `results ${r.has_results ? `${r.outcomes.length} outcomes${r.truncated ? " truncated" : ""}` : "none"}`);
|
|
4785
5074
|
return { content: [{ type: "text", text: leanJson(r) }] };
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "fh-library",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.13",
|
|
4
4
|
"mcpName": "io.github.forever-healthy/fh-library",
|
|
5
5
|
"description": "MCP Server that builds a local Markdown library of the scientific papers cited by evidence reviews — fetches metadata, abstract and full text per PubMed ID through a configurable cascade of retrieval services",
|
|
6
6
|
"type": "module",
|