fh-library 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -1
- package/README.md +27 -27
- package/dist/index.js +130 -149
- package/package.json +3 -3
package/CHANGELOG.md
CHANGED
|
@@ -1,8 +1,13 @@
|
|
|
1
1
|
# FH Library — Change Log
|
|
2
2
|
|
|
3
|
+
### v0.2.2 — 2026-11-08
|
|
4
|
+
|
|
5
|
+
* **Tool names carry the `library_` prefix again**, all of them: `library_add`, `library_status`, `library_read`, `library_verify`, `library_info`, `library_version` (0.2.0 and 0.2.1 had the bare names, and `version`). A model that sees many servers' tools should never have to guess what `add` or `read` belongs to. Client permissions and prompts name them with the prefix.
|
|
6
|
+
* **Browser rungs by kind:** `FH_LIBRARY_PLAYWRIGHT_1`, `_2`, … (Playwright servers, port 8932) and `FH_LIBRARY_CHROME_1`, `_2`, … (a Chrome with `--remote-debugging-port`, port 9222) replace `FH_LIBRARY_BROWSER_1..3`. Values are hostnames (host or host:port); the kind is the variable's name, the protocol the library's. `FH_LIBRARY_PLAYWRIGHT_HEADLESS` (was `_BROWSER_HEADLESS`) belongs to the Playwright family; a Chrome rung is whatever it is. Order is fixed: Playwright servers, then Chrome rungs. The in-process "local" Chromium rung is gone — the library never launches a browser. `library_info` splits `download` into `playwright: {version, servers}` and `chrome`. Hard cut: every `.env` must move its browser lines.
|
|
7
|
+
|
|
3
8
|
### v0.2.0 — 2026-11-08
|
|
4
9
|
|
|
5
|
-
* **Renamed to FH Library.** The npm packages are `fh-library` (public) and `fh-library-dev` (dev), the MCP Registry entry `io.github.forever-healthy/fh-library`, the repos `forever-healthy/
|
|
10
|
+
* **Renamed to FH Library.** The npm packages are `fh-library` (public) and `fh-library-dev` (dev), the MCP Registry entry `io.github.forever-healthy/fh-library`, the repos `forever-healthy/fh-library` and `fh-library-dev`; the server announces itself as `fh-library`, and client configs use the keys `fh-library`, `fh-library-dev`, `fh-library-local` (the `d-` prefix is for MCP servers Forever Healthy did not write). The `ai4l-library` packages are deprecated and point here.
|
|
6
11
|
* **Tools lose the `library_` prefix:** `add`, `status`, `read`, `verify`, `info`, `version` (was `get_version`). The server name namespaces them in every client.
|
|
7
12
|
* **Environment variables are `FH_LIBRARY_*`** (was `AI4L_LIBRARY_*`), hard cut: every `.env` must follow. Keys other servers share (`NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE`) are unchanged. The browser helper is `fh-chrome` (service, window class, `~/.fh-chrome` profile), the browser server container `fh-browser`.
|
|
8
13
|
* Entries in the changelog below this line use the old names; they describe the versions that carried them.
|
package/README.md
CHANGED
|
@@ -1,29 +1,29 @@
|
|
|
1
|
-
|
|
2
2
|
|
|
3
3
|
[_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
|
|
4
4
|
[](https://www.npmjs.com/package/fh-library)
|
|
5
5
|
[](https://registry.modelcontextprotocol.io/v0/servers?search=io.github.forever-healthy/fh-library)
|
|
6
|
-
[](https://glama.ai/mcp/servers/forever-healthy/fh-library)
|
|
7
7
|
|
|
8
|
-
|
|
9
9
|
|
|
10
10
|
**FH Library** - MCP server that builds a local library of the scientific papers an evidence review cites, part of [AI4L](https://github.com/forever-healthy/AI4L)
|
|
11
11
|
|
|
12
12
|
A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a local library of the scientific papers an evidence review cites. Given a paper ID (`type:value`, e.g. `pmid:30955454`), it fetches the paper's metadata, abstract and — where obtainable — full text through a configurable cascade of retrieval services (PubMed, PMC, Europe PMC, Unpaywall, OpenAlex, …), converts everything to Markdown, and stores it under a mounted `library/` directory. Nothing flows through the model's context: the server writes to disk itself and returns only a small status per call.
|
|
13
13
|
|
|
14
|
-
**Out of the box, keyless, everywhere:** PubMed, PMC, Europe PMC, Unpaywall and OpenAlex, a plain fetch, and search inside stored papers. Everything stronger is an endpoint or a key the deployment adds in `.env` — browser rungs (Playwright servers, a real Chrome), Bright Data, CORE, the Elsevier and Wiley text-mining tokens, your own proxy — each with an honest fallback when absent; `
|
|
14
|
+
**Out of the box, keyless, everywhere:** PubMed, PMC, Europe PMC, Unpaywall and OpenAlex, a plain fetch, and search inside stored papers. Everything stronger is an endpoint or a key the deployment adds in `.env` — browser rungs (Playwright servers, a real Chrome), Bright Data, CORE, the Elsevier and Wiley text-mining tokens, your own proxy — each with an honest fallback when absent; `library_info` shows what is configured.
|
|
15
15
|
|
|
16
|
-
**Status: v0.2, early development.** Four ID types: papers by `pmid:` and, when PubMed does not index them, by `doi:` (record from Crossref); drug labels by `dailymed:`; trial records by `nct:`. Every paper entry gets metadata and abstract from PubMed (or Crossref and OpenAlex); the full-text cascade then tries PMC (JATS XML), the publishers' text-and-data-mining links (Crossref; Elsevier and Wiley with a token), and open-access copies located via Unpaywall and OpenAlex (PDF, XML or HTML). A full text counts only after checks: the paper's title or DOI in the text, 800+ words of running prose (a publisher's abstract page has 200–600), Methods and Results or a sectioned body, no wall page. `
|
|
16
|
+
**Status: v0.2, early development.** Four ID types: papers by `pmid:` and, when PubMed does not index them, by `doi:` (record from Crossref); drug labels by `dailymed:`; trial records by `nct:`. Every paper entry gets metadata and abstract from PubMed (or Crossref and OpenAlex); the full-text cascade then tries PMC (JATS XML), the publishers' text-and-data-mining links (Crossref; Elsevier and Wiley with a token), and open-access copies located via Unpaywall and OpenAlex (PDF, XML or HTML). A full text counts only after checks: the paper's title or DOI in the text, 800+ words of running prose (a publisher's abstract page has 200–600), Methods and Results or a sectioned body, no wall page. `library_read` serves outline, section or search passages for claim verification. Papers whose PMC XML body is withheld by the publisher are reachable only through open-access copies or a licensed proxy; PMC's web pages are for humans and are not scraped. PDF-sourced texts get their paragraphs from the page layout (leading, indent, a short last line).
|
|
17
17
|
|
|
18
18
|
|
|
19
19
|
## Tools
|
|
20
20
|
|
|
21
|
-
* `
|
|
22
|
-
* `
|
|
23
|
-
* `
|
|
24
|
-
* `
|
|
25
|
-
* `
|
|
26
|
-
* `
|
|
21
|
+
* `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `coi` and `funding` (PubMed's record, else the paper's own text), `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — files, failed tiers in `attempts`). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `FH_LIBRARY_REFRESH_AFTER_DAYS` gets the record re-read for new notices. One ID per call.
|
|
22
|
+
* `library_status(id)` → read-only check of presence and level; never fetches
|
|
23
|
+
* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
|
|
24
|
+
* `library_verify(claim, id)` → is one finding of a review stated by one paper it cites? `claim` is the finding alone, verbatim, without its citations; `id` one paper (`pmid:`, or any `doi:`), drug label or trial record the review cites for it — a finding cited to three papers is three calls with the same claim; a claim that names a source or carries citations is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters the model's context. `verdict` **pass** (`score` the probability), **fail** (the full text does not state it; or only the title is in the library — `reason` says so), **unverifiable** (only the abstract is in the library and does not state it — `reason`) or **error**; `basis` full_text, abstract or title. See "Verifying claims" below.
|
|
25
|
+
* `library_read(id, section?, query?, max_chars?, max_hits?)` → one small piece of a stored paper's Markdown, never the whole paper: without `section`/`query` the **outline** (headings with word counts, and whether the full text or only the abstract is stored; a heading the paper uses several times is listed as `Participants [3/12]`); `section` → that section's **own text** (heading matched as case-insensitive substring, capped at `max_chars`, default 6000, with `returned_words` against `words` when cut) and, for a parent heading, its `subsections` with word counts to pick from — a duplicated heading named bare lists the candidates instead; `query` — two or three distinctive terms the paper itself would use (a measure and a unit, an outcome and a group, a compound name), not the claim — → **passages** holding every term, each a window of at most 1000 chars around the matches (`clipped: true` when cut; a table row with its header row), with its section, best first (at most `max_hits`, default 5); matching forgives case, accents, hyphens, Markdown markup, plural and -ed/-ing forms; when no passage holds every term the best **partial** matches come back (`partial: true`, the `matched` terms per passage, the rare term first), and `missing_terms` names the terms the text never uses in any form — those are the ones to rephrase. `section` and `query` together search inside that section. `section: "Abstract"` on a full text without one answers from the stored abstract. Every full text converted from JATS XML (PMC, Europe PMC, Elsevier) has a **Declarations** section — funding, conflicts of interest, author notes, gathered from wherever the XML keeps them — and on a text without one `section: "Declarations"` assembles the sections headed funding, conflicts, acknowledgements, ethics, contributions, so it answers who paid and who declares what on every source alike; a paper with neither states nothing. Outline first, then pick. A paper not in the library is added first, as `library_add` would (its lines in the log), then library_read; the answer carries `added: true`. Reads may run in parallel.
|
|
26
|
+
* `library_version()` → the running server's package name, library_version, host and library directory
|
|
27
27
|
|
|
28
28
|
IDs follow the `type:value` form: `pmid:`, `doi:`, `dailymed:`, `nct:`; bare digits count as `pmid:`, a bare NCT number as `nct:`.
|
|
29
29
|
|
|
@@ -32,7 +32,7 @@ IDs follow the `type:value` form: `pmid:`, `doi:`, `dailymed:`, `nct:`; bare dig
|
|
|
32
32
|
| Tier | Does | Needs |
|
|
33
33
|
|---|---|---|
|
|
34
34
|
| `pubmed` | metadata, abstract, PMCID, DOI, publication types (retraction) — always | – |
|
|
35
|
-
| `inbox` | a file you downloaded yourself: drop it into `<library>/inbox/` with the PMID in its name (`39025385.pdf`; PDF, XML or HTML) and the next `
|
|
35
|
+
| `inbox` | a file you downloaded yourself: drop it into `<library>/inbox/` with the PMID in its name (`39025385.pdf`; PDF, XML or HTML) and the next `library_add` for that ID converts it into the entry and removes it from the inbox. For publishers that hand a PDF to a browser but nothing to a script | PMID |
|
|
36
36
|
| `pmc` | full text as JATS XML via E-utilities; counts only with a `<body>` | PMCID |
|
|
37
37
|
| `europepmc` | Europe PMC's record: its own full text (JATS XML for the open-access subset, the rendered PDF for everything it holds, author manuscripts included) and the free links it lists | – |
|
|
38
38
|
| `tdm` | the publishers' text-and-data-mining links from Crossref; Elsevier (`X-ELS-APIKey`) and Wiley (`Wiley-TDM-Client-Token`) need a token; full text for open-access and entitled articles by the front door | DOI; `FH_LIBRARY_ELSEVIER_API_KEY`, `FH_LIBRARY_WILEY_TDM_TOKEN` |
|
|
@@ -41,7 +41,7 @@ IDs follow the `type:value` form: `pmid:`, `doi:`, `dailymed:`, `nct:`; bare dig
|
|
|
41
41
|
| `core` | CORE (core.ac.uk), the aggregator of repository copies, by DOI; skipped without a key | `FH_LIBRARY_CORE_API_KEY` (free) |
|
|
42
42
|
| `proxy` | your own service: an institutional proxy, a licensed service, a local mirror — a URL template filled from the paper's IDs (below) | `FH_LIBRARY_PROXY_URL`; the IDs the template uses |
|
|
43
43
|
|
|
44
|
-
A located copy is fetched up a ladder, the way AI4L verifies URLs: a plain **fetch** (no browser) with a desktop Chrome User-Agent; then
|
|
44
|
+
A located copy is fetched up a ladder, the way AI4L verifies URLs: a plain **fetch** (no browser) with a desktop Chrome User-Agent; then the **Playwright servers** in order (`FH_LIBRARY_PLAYWRIGHT_1`, `_2`, …: hostnames, port 8932 unless given as host:port — `browser/compose.yaml` runs one, headed under Xvfb on Microsoft's Playwright image, typically one next to the server and one on a residential line; a server must run the exact `playwright-core` version the package pins, 1.65.0, shown by `library_info` as `download.playwright.version`, since Playwright refuses a client of another version — logged as the alert `!! browser <host>: Playwright version mismatch — server … must run playwright-core 1.65.0`), then the **Chrome rungs** in order (`FH_LIBRARY_CHROME_1`, `_2`, …: hostnames, port 9222 unless given — a real Google Chrome with `--remote-debugging-port`, attached over the DevTools protocol with its own profile and standing, `browser/chrome.md`), each only when the plain fetch met a wall or a 403; then the Bright Data **unlocker** when every browser was refused. No browser ships in the package and none is downloaded at run time. Nothing disguises the automation: a site that blocks automated browsers by policy stays blocked, and the attempt says so.
|
|
45
45
|
|
|
46
46
|
|
|
47
47
|
Downloads are plain HTTP with a desktop Chrome User-Agent (API calls identify the library with the contact address), PDFs first and repository copies before publisher sites; PDF (text only, no tables), JATS XML or HTML are converted to Markdown. NCBI/PMC web URLs are never fetched (PMC content comes through E-utilities). When a publisher answers a located open-access copy with a 403 or a bot wall, the page is requested once more through the [Bright Data](https://brightdata.com) Web Unlocker — the same last-resort tier AI4L's URL verification uses — if `BRIGHTDATA_API_TOKEN` is set (`BRIGHTDATA_ZONE`, default `mcp_unlocker`). Without the token the plain failure stands and is retried later.
|
|
@@ -52,10 +52,10 @@ Downloads are plain HTTP with a desktop Chrome User-Agent (API calls identify th
|
|
|
52
52
|
|
|
53
53
|
* **Which tiers run, in which order:** `FH_LIBRARY_TIERS=pmc,tdm,unpaywall,openalex` — exactly these, in this order; a tier not listed is off. Unset, every tier runs in the order of the table above, the proxy last.
|
|
54
54
|
* **Keys:** `FH_LIBRARY_ELSEVIER_API_KEY` and `FH_LIBRARY_WILEY_TDM_TOKEN` for `tdm`, `FH_LIBRARY_CORE_API_KEY` for `core`, `NCBI_API_KEY` (10 instead of 3 requests/s), `BRIGHTDATA_API_TOKEN` and `BRIGHTDATA_ZONE` for the unlocker. `FH_LIBRARY_CONTACT_EMAIL` is the one address sent to every service (NCBI `tool`/`email`, Unpaywall and OpenAlex polite pools, Crossref, the User-Agent).
|
|
55
|
-
* **Retries:** `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried; `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `
|
|
56
|
-
* **Browsers and verify:** `
|
|
55
|
+
* **Retries:** `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried; `FH_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `library_add` re-reads it for new correction notices and retractions. A tdm link behind a token that is not configured is recorded as `needs elsevier token` and not re-checked; after adding the token, remove those attempts from the entries to give them a go. `FH_LIBRARY_RETRY_BLOCKED` (false) decides whether a browser challenge that did not clear, or a Bright Data refusal, is final: with the same method tomorrow ends the same way, so set it true only after a change of method (a new browser rung, a new service).
|
|
56
|
+
* **Browsers and verify:** `FH_LIBRARY_PLAYWRIGHT_n` and `FH_LIBRARY_CHROME_n` (hostnames; above) and `FH_LIBRARY_PLAYWRIGHT_HEADLESS`; `FH_LIBRARY_VERIFY_*` ("Verifying claims"). A Playwright server must run the **same `playwright-core` version** as the package — `library_info` shows the required one as `download.playwright.version`, `browser/Dockerfile` builds a server at it, and a server of another version is refused by Playwright itself (the log then carries the alert `!! browser <host>: Playwright version mismatch — server … must run playwright-core <version>`). Moving the version is one release: the pin in `package.json`, the tag in `browser/Dockerfile`, every server rebuilt. The library never launches a browser itself: every rung is a browser somebody runs.
|
|
57
57
|
|
|
58
|
-
`
|
|
58
|
+
`library_info` reports the tiers in run order and, per variable, whether its value came from the environment, `.env` or the default — never the value.
|
|
59
59
|
|
|
60
60
|
**The proxy tier.** One service of your own — an institutional proxy, a licensed service, a local mirror — fetched at its position in the ladder like any other tier:
|
|
61
61
|
|
|
@@ -87,13 +87,13 @@ Any other body — an HTML page, JSON, plain text, an empty response — is the
|
|
|
87
87
|
|
|
88
88
|
The proxy tier is not escalated to the browser rungs or Bright Data: it is the service's own answer that counts. NCBI and PMC web URLs are never fetched (the `pmc` tier covers them), and a URL already tried for the paper in the same cascade run is skipped.
|
|
89
89
|
|
|
90
|
-
Every full text is checked before it counts: title present, methods and results present (or a substantial sectioned body), not an error, login, paywall or bot-wall page. The original (XML, HTML or PDF) is stored next to the Markdown. A failed tier is recorded in the entry's `attempts` with reason and date, e.g. `unpaywall (HTTP 403: nature.com) 2026-10-26`. **Retries:** a permanent reason (no PMCID, no body, closed, no open-access copy, content checks failed) is never retried; a transient one (timeout, network error, HTTP 403/429/5xx, wall page) is retried by the next `
|
|
90
|
+
Every full text is checked before it counts: title present, methods and results present (or a substantial sectioned body), not an error, login, paywall or bot-wall page. The original (XML, HTML or PDF) is stored next to the Markdown. A failed tier is recorded in the entry's `attempts` with reason and date, e.g. `unpaywall (HTTP 403: nature.com) 2026-10-26`. **Retries:** a permanent reason (no PMCID, no body, closed, no open-access copy, content checks failed) is never retried; a transient one (timeout, network error, HTTP 403/429/5xx, wall page) is retried by the next `library_add` once `FH_LIBRARY_RETRY_AFTER_DAYS` (default 1) have passed, replacing the old entry. A `full_text` entry is never touched again. A failed PubMed fetch writes no entry at all, so it is simply retried next time.
|
|
91
91
|
|
|
92
92
|
### Verifying claims
|
|
93
93
|
|
|
94
|
-
`
|
|
94
|
+
`library_verify(claim, id)` answers whether one source states one finding of a review — a paper, a drug label (`dailymed:…`) or a trial record (`nct:…`), each judged on its stored full text the same way. This is the contract for an auditor, or any client, built against it.
|
|
95
95
|
|
|
96
|
-
**The call.** `claim` is one finding, verbatim from the review, with markdown links reduced to their text and its citations removed: `IPSS fell 2.3 points more than placebo in men with ED`. Section labels such as `**Magnitude:**` and list bullets may stay. `id` is one paper the review cites for that finding, as `pmid:<PMID>`, already in the library (`
|
|
96
|
+
**The call.** `claim` is one finding, verbatim from the review, with markdown links reduced to their text and its citations removed: `IPSS fell 2.3 points more than placebo in men with ED`. Section labels such as `**Magnitude:**` and list bullets may stay. `id` is one paper the review cites for that finding, as `pmid:<PMID>`, already in the library (`library_add` first). One claim, one paper per call: a finding cited to three papers is three calls with the same claim text, and each answer stands on its own — a source that does not carry the finding is named by its own row, so it can be dropped without touching the others. The server never aggregates over papers. A sentence that joins two findings, each with its own citation, is two claims: the text before each citation group, without the group. A sentence without a citation is not a claim.
|
|
97
97
|
|
|
98
98
|
**How to build the claim.** The decision model answers exactly the statement it is given, so the statement must be one the paper could make:
|
|
99
99
|
|
|
@@ -114,9 +114,9 @@ Every full text is checked before it counts: title present, methods and results
|
|
|
114
114
|
| `unverifiable` | only the paper's abstract is in the library and it does not state the finding — the full text may; `reason` "only the abstract is stored and it does not state the claim; the full text may"; drop the full text into the inbox and verify again |
|
|
115
115
|
| `error` | the call could not run: a refused claim (empty, too short, a fragment, naming a source, compound), paper not in the library, no model configured, the model failing or not answering within the timeout |
|
|
116
116
|
|
|
117
|
-
"In substance": wording may differ, a number may be rounded or derived from the paper's numbers; a changed number, an opposite direction, another population or intervention is not backed. A paper is
|
|
117
|
+
"In substance": wording may differ, a number may be rounded or derived from the paper's numbers; a changed number, an opposite direction, another population or intervention is not backed. A paper is library_read whole; nothing of it enters the caller's context, and the model returns one probability, no quotes and no prose. For a look at the text behind a verdict there is `library_read`; on the hosted model real findings score 0.95 and up and changed numbers 0.20 and under, so a `pass` under 0.7 or a `fail` over 0.3 is the one worth that look.
|
|
118
118
|
|
|
119
|
-
**For an audit table:** one row per call; `pass` → 🟢, `fail` → 🔴, `unverifiable` → N/A, `error` → empty; comment: the score to two decimals and the basis (`title` alone when the basis is `title`, where `score` is null), or the error text. A refused claim (`error` naming a source, a compound claim, a fragment, too short) means the claim cell is wrong: fix it as the error says and call once more. `
|
|
119
|
+
**For an audit table:** one row per call; `pass` → 🟢, `fail` → 🔴, `unverifiable` → N/A, `error` → empty; comment: the score to two decimals and the basis (`title` alone when the basis is `title`, where `score` is null), or the error text. A refused claim (`error` naming a source, a compound claim, a fragment, too short) means the claim cell is wrong: fix it as the error says and call once more. `library_info` reports the model as `verify.model`, its endpoint as `verify.base_url`, whether it is local, and whether it answers right now — `verify.ok` with the answer time in milliseconds in `verify.ms`, or the reason in `verify.error`, from one tiny request per call; write the model once above the table, and stop before the claims when `ok` is false. A finding whose sources split 🟢 / 🔴 keeps the 🟢 rows, and the 🔴 row names the source to remove.
|
|
120
120
|
|
|
121
121
|
#### The judge: Perplexity's Decider
|
|
122
122
|
|
|
@@ -131,19 +131,19 @@ Hosted or local is three lines in `.env`: `FH_LIBRARY_VERIFY_URL`, `FH_LIBRARY_V
|
|
|
131
131
|
curl -s http://<ip>:8080/v1/decisions -H 'Content-Type: application/json' -d '{"model":"pplx-decider-flash-v1.2","state":"I was charged twice for my order.","questions":{"team":{"type":"choice","criteria":{"billing":null,"shipping":null,"technical":null}}}}'
|
|
132
132
|
```
|
|
133
133
|
|
|
134
|
-
**Papers without a PMID.** A review may cite a paper PubMed does not index by its DOI (`https://doi.org/<DOI>`); `
|
|
134
|
+
**Papers without a PMID.** A review may cite a paper PubMed does not index by its DOI (`https://doi.org/<DOI>`); `library_add("doi:10.5812/asjsm.110753")` takes it. If PubMed does index the DOI, the call builds that paper's `pmid:` entry and stores a copy of it under `doi:` — the same files, renamed, with `pmid` set in its meta — so the DOI resolves on disk: every later call answers from the `doi:` entry without asking PubMed (5 of the 101 DOIs evipedia cites). Each `library_add` of such a DOI completes the `pmid:` entry (retries, record refresh) and copies it again, so the copy is as current as the original. A mapped paper counts once under `pmid` and once under `doi` in `library_info`. Otherwise the record comes from [Crossref](https://www.crossref.org/) — or, for a DOI another registry holds (DataCite, mEDRA, …), from doi.org's content negotiation as CSL JSON, stored as `-csl.json`: title, authors, journal, year, the abstract — Crossref's when the publisher deposited one, else OpenAlex's (of the 97 DOI-only papers evipedia cites, Crossref has 55 abstracts, OpenAlex 86; 11 have none anywhere and stay title-only unless a full text is found) — the funders as `funding`, and Crossref's update notices — corrections, retractions, expressions of concern — as `corrections`, so `citable` works as for a PubMed paper. The library_status has `pmid` null, the abstract's head line says "not in PubMed", `retrieved_by` names the source of what the entry holds — `crossref`, `doi.org` or `openalex` for the abstract, `crossref` or `doi.org` for a title alone, the tier once a full text is found; the full-text cascade then runs on the DOI (`unpaywall`, `openalex`, `tdm`, `core`, `proxy`; `pmc`, `europepmc` and `inbox` need a PMID). An entry older than `FH_LIBRARY_REFRESH_AFTER_DAYS` gets its Crossref record re-library_read for new notices. Entries live under `doi/<hash 2>/<hash 4>/<slug>/`, the slug being the DOI in lowercase with every run of other characters than letters, digits, `.` and `-` as `_` (`doi-10.5812_asjsm.110753`), the Crossref record as `-crossref.json`. `library_status`, `library_read` and `library_verify` take any DOI `library_add` took: they library_read its `doi:` entry, the copy or the Crossref one, and never look a DOI up. Measured on evipedia's 915 live reviews (2026-11-06): 87 cite 101 papers by DOI only, against 34,378 PubMed citations.
|
|
135
135
|
|
|
136
136
|
PubMed's book records — National Academies reports, NTP monographs, StatPearls chapters in NCBI Bookshelf — are read as papers too: the chapter or book title, the authors (else the editors), the publisher in place of the journal, publication type `Book`.
|
|
137
137
|
|
|
138
|
-
**Drug labels.** `
|
|
138
|
+
**Drug labels.** `library_add("dailymed:<set ID>")` stores the current FDA prescribing information of one product from [DailyMed](https://dailymed.nlm.nih.gov/), the set ID being the fixed identifier in the label's page URL (`…/drugInfo.cfm?setid=<set ID>`) across all its revisions — the same ID a review links. The SPL XML is converted to Markdown: first a **Highlights** section with the label's header lines — the product line and "Initial U.S. Approval: 2003" (the year is also in the head line) — then every text section in order, a boxed warning headed "Boxed Warning" and the recent major changes headed as such even where the label gives them no title, (indications, dosage, contraindications, warnings, adverse reactions with their tables, drug interactions, clinical pharmacology, clinical studies); the structured product data and package images are left out; there is no abstract. The status carries `setid`, the title as `<brand> (<generic>) <form> — prescribing information`, the labeler as `first_author`, the effective year as `year`, the label version as `version` (`spl 52`), and `publication_types` `["Drug Label"]`. `library_read` and `library_verify` work on a label as on a paper, so a figure the review takes from a label is verified against the label. An entry older than `FH_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when DailyMed has a newer version. Entries live under `dailymed/<first 2>/<first 4>/<set ID>/`, with the SPL XML and the Markdown as `dailymed-<set ID>-fulltext.xml` and `-fulltext.md`, and the log line reads `library_add dailymed:<set ID> dailymed fetched level=full_text spl 52 effective=2026-06-03 Cialis (Tadalafil)`.
|
|
139
139
|
|
|
140
|
-
**Clinical trial records.** `
|
|
140
|
+
**Clinical trial records.** `library_add("nct:05173896")` stores the current record of one trial from [ClinicalTrials.gov](https://clinicaltrials.gov/): the ID is the eight digits of the NCT number in the review's link (`…/study/NCT05173896`); `nct:NCT05173896` and a bare `NCT05173896` are accepted and library_read the same. The API's JSON is converted to Markdown, one section per module: identification with secondary IDs such as EudraCT, status with every date and its actual/estimated type, sponsor and collaborators, description, conditions, design (phase, allocation, masking, enrollment with its type), arms and interventions with doses, primary and secondary outcomes with time frames, eligibility, locations as a country list, references; and, when results are posted, the participant flow, baseline characteristics, every outcome measure with its group values and analyses, and the adverse events, all as tables. Values are written in plain words, "Phase 2", "Active, not recruiting", "100 (estimated)", so a figure the review states can be found as stated. The status carries `nctid`, `publications` (the results publications the record lists, as `pmid:…`, so whether a trial has been published is one status away), the brief title, the lead sponsor as `first_author`, the first-posted year as `year`, the last-update-posted date as `version`, and `publication_types` `["Trial Registration"]`. `library_read` and `library_verify` work on a record as on a paper. An entry older than `FH_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when the record was updated since; a status the review states that has moved on then fails, which is right. Entries live under `nct/<digits 1–2>/<digits 1–4>/<digits>/` with the JSON and the Markdown as `nct-<digits>-fulltext.json` and `-fulltext.md`; the log line reads `library_add nct:05173896 ctgov fetched level=full_text updated 2024-12-09 active, not recruiting ETLAS2`.
|
|
141
141
|
|
|
142
|
-
**Funding and conflicts of interest.** Every status carries `coi`, the paper's conflict-of-interest statement, and `funding`, who paid. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `
|
|
142
|
+
**Funding and conflicts of interest.** Every status carries `coi`, the paper's conflict-of-interest statement, and `funding`, who paid. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `library_read(id, section: "Declarations")` — the JATS section, or on a PDF- or HTML-sourced text the funding, conflict and acknowledgement sections assembled. In `-meta.md` they are the lines `coi:` (a JSON string) and `funding:` (a JSON array, one item per agency or statement).
|
|
143
143
|
|
|
144
|
-
**Symbols a PDF font hides.** Some publishers' symbol fonts carry no Unicode for their glyphs, so text extraction gives a control character where the PDF shows a minus, ±, ≤ or β. The library restores the symbol from the glyph's name in the font — the names of the Adobe Pi fonts (`H11350` is ≥ everywhere) and of the Advent symbol font most Elsevier and Wiley PDFs use — each name checked against the glyph's outline in the library's PDFs; a name it does not know is left as it is and counted: the status's `glyphs_unresolved` is the number left in the full text (162 of 1,119 PDF-derived entries have some, down from 277), and `
|
|
144
|
+
**Symbols a PDF font hides.** Some publishers' symbol fonts carry no Unicode for their glyphs, so text extraction gives a control character where the PDF shows a minus, ±, ≤ or β. The library restores the symbol from the glyph's name in the font — the names of the Adobe Pi fonts (`H11350` is ≥ everywhere) and of the Advent symbol font most Elsevier and Wiley PDFs use — each name checked against the glyph's outline in the library's PDFs; a name it does not know is left as it is and counted: the status's `glyphs_unresolved` is the number left in the full text (162 of 1,119 PDF-derived entries have some, down from 277), and `library_verify` returns the same count for the text it judged. A `fail` on a paper with `glyphs_unresolved` above 0 says "check the symbol", not "the paper does not state it". Words a PDF breaks at a line end ("differ- ence") are joined back when the paper uses the word elsewhere or no compound is known at that break; a hyphen the paper uses elsewhere ("time-dependent") and a suspended one ("dose- and time-dependent") are kept.
|
|
145
145
|
|
|
146
|
-
**Citable, and correction notices.** `citable` is the one flag an auditor checks before a paper counts as evidence: false when PubMed records the paper as retracted — by the "Retracted Publication" type or by a retraction notice — and, once correction notices are assessed (below), when a correction changed its data; false too when the entry holds nothing to read — no abstract text and no full text, `level` title — because a source nothing can be checked against is not evidence (`comment` "no abstract and no full text: nothing to
|
|
146
|
+
**Citable, and correction notices.** `citable` is the one flag an auditor checks before a paper counts as evidence: false when PubMed records the paper as retracted — by the "Retracted Publication" type or by a retraction notice — and, once correction notices are assessed (below), when a correction changed its data; false too when the entry holds nothing to read — no abstract text and no full text, `level` title — because a source nothing can be checked against is not evidence (`comment` "no abstract and no full text: nothing to library_verify"; a full text dropped into the inbox makes it citable). `comment` says why, or, when the paper is citable, names what to be aware of: an erratum (its severity is not assessed yet; `library_add` on the notice's PMID fetches it when it has one), an expression of concern, or a newer version that supersedes this one, typically an updated Cochrane review, which should be cited instead. Behind the flag, `corrections` lists PubMed's notices about the paper as `{type, pmid, source}` — `erratum_in`, `retraction_in`, `partial_retraction_in`, `retracted_and_republished_in`, `expression_of_concern_in` or `update_in`, the notice's own PMID when PubMed has one, and its citation verbatim. Comments are not notices and are not carried: PubMed's CommentIn, SummaryForPatientsIn, ReprintIn and the "Of/For/From" direction (this paper being the notice about another) are ignored. `update_in` is not an erratum: every case seen is a Cochrane review superseded by a newer version. Notices appear after ingest, so `library_add` on an existing entry re-reads the PubMed record once it is older than `FH_LIBRARY_REFRESH_AFTER_DAYS` (default 30; `pubmed_checked` in the entry is the date of the last read) and updates the notices, retraction, publication types and a PMCID that appeared since — never the stored abstract or full text, and without reopening full-text retries. The notice's own text is not fetched or applied. Where NLM has corrected the PubMed abstract after an erratum — it marks each corrected sentence `[corrected]` — those sentences are copied to the end of the stored full text under `## Corrected in PubMed abstract`, at ingest and on every refresh, so a full text that predates the correction still carries the corrected figures and `library_verify` passes a claim that states them — and marks a claim that states a replaced figure as `superseded`. The `comment` then reads "erratum, corrected figures stored" instead of "not assessed". In `-meta.md` each notice is one `corrections:` line, `<type> <pmid or -> "<source>"`; entries written before this feature are read from their stored PubMed XML until they are next rewritten.
|
|
147
147
|
|
|
148
148
|
**Why a notice may have no PMID.** PubMed gives the notice's PMID only when the notice is itself a PubMed record. For about a quarter of the errata in a 6,900-entry library it is not: the journal printed the correction, PubMed recorded the citation in the paper's record, and never indexed the correction as a record of its own (NEJM, JAMA and Lancet errata from 1988 to 2014 are typical). There is nothing to redirect to, so `pmid` is null and `source` — the citation as PubMed wrote it — is the notice's only identifier. Measured on the 79 such errata in the library (2026-11-04): PubMed's citation matcher resolves 18 after all (13 directly, 5 after picking the "Published Erratum" among two candidates), Crossref gives a DOI for 38 more when journal, volume and page match, and 23 have no identifier anywhere. The planned second stage resolves them at ingest (PMID first, then DOI, the citation always kept), fetches the notice's text and classifies it once per paper as major (a number, table, figure, result or conclusion changed) or minor (authors, funding, typos): a major erratum makes the paper not citable, like a retraction, so the audit never has to apply a correction; until then an erratum is a note on a citable paper.
|
|
149
149
|
|
package/dist/index.js
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
#!/usr/bin/env node
|
|
2
2
|
|
|
3
3
|
// src/env.ts
|
|
4
|
-
import { existsSync, readFileSync } from "fs";
|
|
4
|
+
import { existsSync, readFileSync as readFileSync2 } from "fs";
|
|
5
5
|
import { join, resolve } from "path";
|
|
6
6
|
function dotenvFile(env = process.env) {
|
|
7
7
|
const named = env.FH_LIBRARY_ENV?.trim();
|
|
@@ -14,7 +14,7 @@ var loadedFromDotenv = /* @__PURE__ */ new Set();
|
|
|
14
14
|
function loadDotenv(file = dotenvFile()) {
|
|
15
15
|
if (!existsSync(file)) return [];
|
|
16
16
|
const loaded = [];
|
|
17
|
-
for (const raw of
|
|
17
|
+
for (const raw of readFileSync2(file, "utf8").split(/\r?\n/)) {
|
|
18
18
|
const line = raw.trim();
|
|
19
19
|
if (!line || line.startsWith("#")) continue;
|
|
20
20
|
const m = /^(?:export\s+)?([A-Za-z_][A-Za-z0-9_]*)\s*=\s*(.*)$/.exec(line);
|
|
@@ -60,7 +60,7 @@ var contactEmail = (env = process.env) => str("FH_LIBRARY_CONTACT_EMAIL", env) |
|
|
|
60
60
|
// src/http.ts
|
|
61
61
|
var TIMEOUT_MS = 3e4;
|
|
62
62
|
var CHROME_UA = "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/145.0.0.0 Safari/537.36";
|
|
63
|
-
var LIBRARY_UA = `fh-library (+https://github.com/forever-healthy/
|
|
63
|
+
var LIBRARY_UA = `fh-library (+https://github.com/forever-healthy/fh-library; ${contactEmail()})`;
|
|
64
64
|
function httpReason(status, body) {
|
|
65
65
|
const m = body.length < 4e3 && /<statusCode>([A-Z_]+)<\/statusCode>|"statusCode"\s*:\s*"([A-Z_]+)"/.exec(body);
|
|
66
66
|
return m ? `HTTP ${status} ${m[1] ?? m[2]}` : `HTTP ${status}`;
|
|
@@ -889,7 +889,7 @@ var countUnresolved = (text3) => (text3.match(UNRESOLVED_GLYPH) ?? []).length;
|
|
|
889
889
|
var looksLikePdf = (bytes) => bytes.subarray(0, 5).toString("latin1") === "%PDF-";
|
|
890
890
|
|
|
891
891
|
// src/browser.ts
|
|
892
|
-
import { existsSync as existsSync3, mkdirSync
|
|
892
|
+
import { existsSync as existsSync3, mkdirSync } from "fs";
|
|
893
893
|
import { createRequire } from "module";
|
|
894
894
|
import { join as join3 } from "path";
|
|
895
895
|
import { chromium } from "playwright-core";
|
|
@@ -931,17 +931,31 @@ function entryPaths(libraryDir, type, num2) {
|
|
|
931
931
|
}
|
|
932
932
|
|
|
933
933
|
// src/browser.ts
|
|
934
|
+
var PLAYWRIGHT_PORT = 8932;
|
|
935
|
+
var CHROME_PORT = 9222;
|
|
936
|
+
function hostPort(value, defaultPort) {
|
|
937
|
+
const v = value.trim().replace(/^[a-z]+:\/\//i, "").replace(/\/.*$/, "");
|
|
938
|
+
const m = /^(\[[^\]]+\]|[^:]+)(?::(\d{1,5}))?$/.exec(v);
|
|
939
|
+
if (!m) return null;
|
|
940
|
+
return { host: m[1], port: m[2] ? Number(m[2]) : defaultPort };
|
|
941
|
+
}
|
|
934
942
|
function browserConfig(env = process.env) {
|
|
935
943
|
const out = { browsers: [], headless: "auto" };
|
|
944
|
+
const add = (endpoint) => {
|
|
945
|
+
if (!out.browsers.includes(endpoint)) out.browsers.push(endpoint);
|
|
946
|
+
};
|
|
947
|
+
for (let i = 1; i <= 9; i++) {
|
|
948
|
+
const hp = hostPort(str(`FH_LIBRARY_PLAYWRIGHT_${i}`, env), PLAYWRIGHT_PORT);
|
|
949
|
+
if (hp) add(`ws://${hp.host}:${hp.port}`);
|
|
950
|
+
}
|
|
936
951
|
for (let i = 1; i <= 9; i++) {
|
|
937
|
-
const
|
|
938
|
-
if (
|
|
952
|
+
const hp = hostPort(str(`FH_LIBRARY_CHROME_${i}`, env), CHROME_PORT);
|
|
953
|
+
if (hp) add(`cdp://${hp.host}:${hp.port}`);
|
|
939
954
|
}
|
|
940
|
-
const h = str("
|
|
955
|
+
const h = str("FH_LIBRARY_PLAYWRIGHT_HEADLESS", env).toLowerCase();
|
|
941
956
|
if (h === "true" || h === "false") out.headless = h === "true";
|
|
942
957
|
return out;
|
|
943
958
|
}
|
|
944
|
-
var INSTALL_HINT = "npx playwright-core install chromium";
|
|
945
959
|
var PLAYWRIGHT_VERSION = (() => {
|
|
946
960
|
try {
|
|
947
961
|
return createRequire(import.meta.url)("playwright-core/package.json").version;
|
|
@@ -957,37 +971,28 @@ function connectError(e, endpoint) {
|
|
|
957
971
|
var isRemote = (endpoint) => /^wss?:\/\//i.test(endpoint);
|
|
958
972
|
var isCdp = (endpoint) => /^cdp:\/\//i.test(endpoint);
|
|
959
973
|
var cdpHttp = (endpoint) => endpoint.replace(/^cdp:\/\//i, "http://");
|
|
960
|
-
var rungKind = (endpoint) => isCdp(endpoint) ? "chrome" :
|
|
961
|
-
|
|
974
|
+
var rungKind = (endpoint) => isCdp(endpoint) ? "chrome" : "playwright";
|
|
975
|
+
var hostOf = (endpoint) => {
|
|
962
976
|
try {
|
|
963
|
-
|
|
964
|
-
return existsSync3(p) ? p : null;
|
|
977
|
+
return new URL(isCdp(endpoint) ? cdpHttp(endpoint) : endpoint);
|
|
965
978
|
} catch {
|
|
966
979
|
return null;
|
|
967
980
|
}
|
|
968
|
-
}
|
|
981
|
+
};
|
|
969
982
|
function browserLabel(endpoint) {
|
|
970
|
-
|
|
971
|
-
|
|
972
|
-
|
|
973
|
-
return isCdp(endpoint) ? `chrome@${h}` : h;
|
|
974
|
-
} catch {
|
|
975
|
-
return endpoint;
|
|
976
|
-
}
|
|
983
|
+
const u = hostOf(endpoint);
|
|
984
|
+
if (!u) return endpoint;
|
|
985
|
+
return isCdp(endpoint) ? `chrome@${u.hostname}` : u.hostname;
|
|
977
986
|
}
|
|
978
|
-
var hasDisplay = () => !!(process.env.DISPLAY || process.env.WAYLAND_DISPLAY);
|
|
979
987
|
function browserHeadless(endpoint, cfg = browserConfig()) {
|
|
980
988
|
if (isCdp(endpoint)) return false;
|
|
981
|
-
return cfg.headless === "auto" ?
|
|
989
|
+
return cfg.headless === "auto" ? false : cfg.headless;
|
|
982
990
|
}
|
|
983
991
|
function browserRungs(cfg = browserConfig()) {
|
|
984
|
-
return cfg.browsers.map((endpoint) =>
|
|
985
|
-
endpoint
|
|
986
|
-
label: browserLabel(endpoint),
|
|
987
|
-
|
|
988
|
-
available: rungKind(endpoint) !== "local" || localChromium() !== null,
|
|
989
|
-
headless: browserHeadless(endpoint, cfg)
|
|
990
|
-
}));
|
|
992
|
+
return cfg.browsers.map((endpoint) => {
|
|
993
|
+
const u = hostOf(endpoint);
|
|
994
|
+
return { kind: rungKind(endpoint), host: u?.hostname ?? endpoint, port: Number(u?.port) || (isCdp(endpoint) ? CHROME_PORT : PLAYWRIGHT_PORT), label: browserLabel(endpoint), endpoint, headless: browserHeadless(endpoint, cfg) };
|
|
995
|
+
});
|
|
991
996
|
}
|
|
992
997
|
var CHALLENGE = /just a moment|nur einen moment|attention required|verify you are human|human verification|checking your browser|access denied/i;
|
|
993
998
|
var CONNECT_TIMEOUT_MS = 2e4;
|
|
@@ -997,22 +1002,7 @@ var DOWNLOAD_WAIT_MS = 1e4;
|
|
|
997
1002
|
var IDLE_CLOSE_MS = 12e4;
|
|
998
1003
|
var ctxs = /* @__PURE__ */ new Map();
|
|
999
1004
|
var idleTimer = null;
|
|
1000
|
-
function profileDir() {
|
|
1001
|
-
const dir = join3(FH_LIBRARY_DIR, ".browser");
|
|
1002
|
-
mkdirSync(join3(dir, "Default"), { recursive: true });
|
|
1003
|
-
const prefs = join3(dir, "Default", "Preferences");
|
|
1004
|
-
let p = {};
|
|
1005
|
-
try {
|
|
1006
|
-
p = JSON.parse(readFileSync2(prefs, "utf8"));
|
|
1007
|
-
} catch {
|
|
1008
|
-
}
|
|
1009
|
-
p.plugins = { ...p.plugins ?? {}, always_open_pdf_externally: true };
|
|
1010
|
-
p.download = { ...p.download ?? {}, prompt_for_download: false };
|
|
1011
|
-
writeFileSync(prefs, JSON.stringify(p));
|
|
1012
|
-
return dir;
|
|
1013
|
-
}
|
|
1014
1005
|
var LAUNCH_ARGS = ["--disable-blink-features=AutomationControlled", "--window-size=1366,900"];
|
|
1015
|
-
var LOCAL_WINDOW_ARGS = [...LAUNCH_ARGS, "--window-position=4000,4000"];
|
|
1016
1006
|
var TIMEZONE = Intl.DateTimeFormat().resolvedOptions().timeZone || "Europe/Berlin";
|
|
1017
1007
|
var CONTEXT_OPTS = { acceptDownloads: true, viewport: null, locale: "en-US", timezoneId: TIMEZONE };
|
|
1018
1008
|
function stateFile(endpoint) {
|
|
@@ -1031,8 +1021,8 @@ async function context(endpoint, cfg) {
|
|
|
1031
1021
|
if (isCdp(endpoint)) {
|
|
1032
1022
|
const browser = await chromium.connectOverCDP(cdpHttp(endpoint), { timeout: CONNECT_TIMEOUT_MS });
|
|
1033
1023
|
browser.on("disconnected", drop);
|
|
1034
|
-
const
|
|
1035
|
-
return { ctx
|
|
1024
|
+
const ctx = browser.contexts()[0] ?? await browser.newContext(CONTEXT_OPTS);
|
|
1025
|
+
return { ctx, browser };
|
|
1036
1026
|
}
|
|
1037
1027
|
if (isRemote(endpoint)) {
|
|
1038
1028
|
const browser = await chromium.connect(endpoint, {
|
|
@@ -1043,19 +1033,10 @@ async function context(endpoint, cfg) {
|
|
|
1043
1033
|
});
|
|
1044
1034
|
browser.on("disconnected", drop);
|
|
1045
1035
|
const state = stateFile(endpoint);
|
|
1046
|
-
const
|
|
1047
|
-
return { ctx
|
|
1036
|
+
const ctx = await browser.newContext({ ...CONTEXT_OPTS, ...existsSync3(state) ? { storageState: state } : {} });
|
|
1037
|
+
return { ctx, browser };
|
|
1048
1038
|
}
|
|
1049
|
-
|
|
1050
|
-
const ctx = await chromium.launchPersistentContext(profileDir(), {
|
|
1051
|
-
channel: "chromium",
|
|
1052
|
-
headless,
|
|
1053
|
-
...CONTEXT_OPTS,
|
|
1054
|
-
args: headless ? LAUNCH_ARGS : LOCAL_WINDOW_ARGS,
|
|
1055
|
-
ignoreDefaultArgs: ["--enable-automation"]
|
|
1056
|
-
});
|
|
1057
|
-
ctx.on("close", drop);
|
|
1058
|
-
return { ctx, browser: null };
|
|
1039
|
+
throw new Error(`not a browser rung: ${endpoint}`);
|
|
1059
1040
|
})();
|
|
1060
1041
|
ctxs.set(endpoint, p);
|
|
1061
1042
|
p.catch(() => {
|
|
@@ -1138,7 +1119,7 @@ async function browserFetch(url, endpoint, cfg = browserConfig()) {
|
|
|
1138
1119
|
}
|
|
1139
1120
|
if (download2) {
|
|
1140
1121
|
const d = download2;
|
|
1141
|
-
return { bytes:
|
|
1122
|
+
return { bytes: readFileSync(await d.path()), contentType: "application/pdf", url: d.url(), via: "download" };
|
|
1142
1123
|
}
|
|
1143
1124
|
const title = await page.title().catch(() => "");
|
|
1144
1125
|
if (CHALLENGE.test(title)) throw new Error(`challenge not cleared (${title.slice(0, 40)})`);
|
|
@@ -1199,7 +1180,7 @@ var miss = (tier, reason) => ({ ok: false, tier, reason });
|
|
|
1199
1180
|
var errMsg = (e) => (e instanceof Error ? e.message : String(e)).replace(/\x1b\[[0-9;]*m/g, "").split("\n")[0].trim().slice(0, 200);
|
|
1200
1181
|
|
|
1201
1182
|
// src/download.ts
|
|
1202
|
-
var
|
|
1183
|
+
var hostOf2 = (url) => {
|
|
1203
1184
|
try {
|
|
1204
1185
|
return new URL(url).hostname.replace(/^www\./, "");
|
|
1205
1186
|
} catch {
|
|
@@ -1225,7 +1206,7 @@ async function download(tier, p, candidates, ctx) {
|
|
|
1225
1206
|
ctx.onTry?.(r.ok ? `${target.url} ok fetch` : `${target.url} failed fetch: ${r.reason}`);
|
|
1226
1207
|
const why = (reason) => reason.replace(/^not a pdf \((.*)\)$/, "$1");
|
|
1227
1208
|
for (const b of browserRungs()) {
|
|
1228
|
-
if (r.ok || c.headers || c.service ||
|
|
1209
|
+
if (r.ok || c.headers || c.service || !/^(HTTP 403|HTTP 429|wall page|no methods\/results|unknown content|browser )/.test(why(r.reason))) break;
|
|
1229
1210
|
r = await tryOne(p, target, tier, { browser: b.endpoint, label: b.label });
|
|
1230
1211
|
ctx.onTry?.(r.ok ? `${target.url} ok browser ${b.label}` : `${target.url} failed browser ${b.label}: ${bare(r.reason)}`);
|
|
1231
1212
|
if (!r.ok && bare(r.reason).startsWith(MISMATCH)) ctx.onAlert?.(`browser ${b.label}: ${bare(r.reason)}`);
|
|
@@ -1236,7 +1217,7 @@ async function download(tier, p, candidates, ctx) {
|
|
|
1236
1217
|
if (!r.ok && /^unlocker HTTP/.test(bare(r.reason))) ctx.onAlert?.(`brightdata: ${bare(r.reason)}`);
|
|
1237
1218
|
}
|
|
1238
1219
|
if (r.ok) return r;
|
|
1239
|
-
reasons.push(`${r.reason}: ${
|
|
1220
|
+
reasons.push(`${r.reason}: ${hostOf2(c.url)}`);
|
|
1240
1221
|
}
|
|
1241
1222
|
return miss(tier, reasons.join("; "));
|
|
1242
1223
|
}
|
|
@@ -1565,7 +1546,7 @@ var TOKENED = [
|
|
|
1565
1546
|
{ host: /(^|\.)api\.elsevier\.com$/i, key: "elsevier", env: "ELSEVIER_API_KEY", header: "X-ELS-APIKey" },
|
|
1566
1547
|
{ host: /(^|\.)api\.wiley\.com$/i, key: "wiley", env: "WILEY_TDM_TOKEN", header: "Wiley-TDM-Client-Token" }
|
|
1567
1548
|
];
|
|
1568
|
-
var
|
|
1549
|
+
var hostOf3 = (url) => {
|
|
1569
1550
|
try {
|
|
1570
1551
|
return new URL(url).hostname;
|
|
1571
1552
|
} catch {
|
|
@@ -1585,7 +1566,7 @@ function tdmCandidates(work, tokens2 = {}, env = process.env) {
|
|
|
1585
1566
|
continue;
|
|
1586
1567
|
}
|
|
1587
1568
|
seen.add(l.URL);
|
|
1588
|
-
const rule = TOKENED.find((r) => r.host.test(
|
|
1569
|
+
const rule = TOKENED.find((r) => r.host.test(hostOf3(l.URL)));
|
|
1589
1570
|
let headers;
|
|
1590
1571
|
if (rule) {
|
|
1591
1572
|
const token = tokens2[rule.key]?.trim() || env[rule.env]?.trim();
|
|
@@ -1596,7 +1577,7 @@ function tdmCandidates(work, tokens2 = {}, env = process.env) {
|
|
|
1596
1577
|
headers = { [rule.header]: token };
|
|
1597
1578
|
}
|
|
1598
1579
|
if (/text\/plain/.test(ct)) continue;
|
|
1599
|
-
candidates.push({ url: l.URL, version: "published", source: "tdm", host: work.message?.publisher ??
|
|
1580
|
+
candidates.push({ url: l.URL, version: "published", source: "tdm", host: work.message?.publisher ?? hostOf3(l.URL), pdf: /pdf/.test(ct) || /\.pdf(\?|$)/i.test(l.URL), headers });
|
|
1600
1581
|
}
|
|
1601
1582
|
const rank = (c) => /\/xml|text\/xml/.test(c.url) ? 0 : c.pdf ? 1 : 2;
|
|
1602
1583
|
candidates.sort((a, b) => rank(a) - rank(b));
|
|
@@ -1694,10 +1675,10 @@ var VARIABLES = [
|
|
|
1694
1675
|
"FH_LIBRARY_CORE_API_KEY",
|
|
1695
1676
|
"FH_LIBRARY_PROXY_URL",
|
|
1696
1677
|
"FH_LIBRARY_PROXY_HEADERS",
|
|
1697
|
-
"
|
|
1698
|
-
"
|
|
1699
|
-
"
|
|
1700
|
-
"
|
|
1678
|
+
"FH_LIBRARY_PLAYWRIGHT_1",
|
|
1679
|
+
"FH_LIBRARY_PLAYWRIGHT_2",
|
|
1680
|
+
"FH_LIBRARY_CHROME_1",
|
|
1681
|
+
"FH_LIBRARY_PLAYWRIGHT_HEADLESS",
|
|
1701
1682
|
"FH_LIBRARY_VERIFY_URL",
|
|
1702
1683
|
"FH_LIBRARY_VERIFY_API_KEY",
|
|
1703
1684
|
"FH_LIBRARY_VERIFY_MODEL",
|
|
@@ -1794,7 +1775,7 @@ import { existsSync as existsSync8, readFileSync as readFileSync7 } from "fs";
|
|
|
1794
1775
|
import { z } from "zod";
|
|
1795
1776
|
|
|
1796
1777
|
// src/entry.ts
|
|
1797
|
-
import { copyFileSync, existsSync as existsSync5, mkdirSync as mkdirSync3, readdirSync as readdirSync2, readFileSync as readFileSync4, renameSync as renameSync2, rmSync, writeFileSync
|
|
1778
|
+
import { copyFileSync, existsSync as existsSync5, mkdirSync as mkdirSync3, readdirSync as readdirSync2, readFileSync as readFileSync4, renameSync as renameSync2, rmSync, writeFileSync } from "fs";
|
|
1798
1779
|
import { join as join6 } from "path";
|
|
1799
1780
|
import { basename, dirname } from "path";
|
|
1800
1781
|
|
|
@@ -2612,7 +2593,7 @@ function readMeta(p) {
|
|
|
2612
2593
|
}
|
|
2613
2594
|
function writeAtomic(file, data) {
|
|
2614
2595
|
const tmp = `${file}.${process.pid}.${Math.random().toString(36).slice(2, 8)}.tmp`;
|
|
2615
|
-
|
|
2596
|
+
writeFileSync(tmp, data);
|
|
2616
2597
|
renameSync2(tmp, file);
|
|
2617
2598
|
}
|
|
2618
2599
|
function writeMeta(p, m) {
|
|
@@ -2626,7 +2607,7 @@ function copyEntry(from, to) {
|
|
|
2626
2607
|
for (const n of readdirSync2(from.entryDir)) if (join6(from.entryDir, n) !== from.meta && !n.endsWith(".tmp")) copyFileSync(join6(from.entryDir, n), join6(tmp, rename(n)));
|
|
2627
2608
|
const m = readMeta(from);
|
|
2628
2609
|
const copy = { ...m, files: m.files.map(rename) };
|
|
2629
|
-
|
|
2610
|
+
writeFileSync(join6(tmp, basename(to.meta)), renderMeta(copy));
|
|
2630
2611
|
mkdirSync3(dirname(to.entryDir), { recursive: true });
|
|
2631
2612
|
if (existsSync5(to.entryDir)) renameSync2(to.entryDir, old);
|
|
2632
2613
|
renameSync2(tmp, to.entryDir);
|
|
@@ -3210,8 +3191,8 @@ function verifyInfo() {
|
|
|
3210
3191
|
return v ? { model: v.model, base_url: v.base_url, local: isLocal(v.base_url) } : null;
|
|
3211
3192
|
}
|
|
3212
3193
|
function downloadInfo() {
|
|
3213
|
-
const
|
|
3214
|
-
return { fetch: true, playwright: PLAYWRIGHT_VERSION,
|
|
3194
|
+
const rungs = browserRungs();
|
|
3195
|
+
return { fetch: true, playwright: { version: PLAYWRIGHT_VERSION, servers: rungs.filter((b) => b.kind === "playwright") }, chrome: rungs.filter((b) => b.kind === "chrome"), unlocker: unlockerConfigured() };
|
|
3215
3196
|
}
|
|
3216
3197
|
|
|
3217
3198
|
// src/read.ts
|
|
@@ -3593,33 +3574,33 @@ IDs \u2014 always "type:value"
|
|
|
3593
3574
|
- nct:05173896 \u2014 a ClinicalTrials.gov trial record, current version. NCT05173896 is accepted too.
|
|
3594
3575
|
|
|
3595
3576
|
Tools
|
|
3596
|
-
-
|
|
3597
|
-
-
|
|
3598
|
-
-
|
|
3599
|
-
-
|
|
3600
|
-
-
|
|
3601
|
-
-
|
|
3577
|
+
- library_version() \u2014 package name, library_version, library directory.
|
|
3578
|
+
- library_info(ids?) \u2014 the whole library: entry counts by type and level, the retrieval sources, browsers, the decision model, the configuration. ids=true lists every entry.
|
|
3579
|
+
- library_add(id) \u2014 download and store one record; returns its status. A stored full text is returned without any download; an abstract-level record retries only the sources that failed transiently; a record older than 30 days gets its correction notices re-read.
|
|
3580
|
+
- library_status(id) \u2014 the library_status of one record without downloading; level "missing" when it is not stored.
|
|
3581
|
+
- library_read(id, section?, query?, max_chars?, max_hits?) \u2014 one small piece of a stored document: its outline, one section, or the passages matching a query.
|
|
3582
|
+
- library_verify(claim, id) \u2014 does this one source state this one finding? A decision model judges the stored text and returns a verdict.
|
|
3602
3583
|
|
|
3603
3584
|
Hints
|
|
3604
|
-
-
|
|
3585
|
+
- library_read and library_verify store a missing record themselves; the library_read answer then carries added: true. No library_add necessary first.
|
|
3605
3586
|
- citable false \u2014 the paper is retracted (a retraction notice, or PubMed's publication type), or holds nothing to read: no abstract and no full text. comment says which; corrections lists the notices as {type, pmid, source}.
|
|
3606
3587
|
- level \u2014 what is stored: "full_text", "abstract" (no full text could be found; attempts names each source tried and why it failed) or "title" (not even an abstract; citable false).
|
|
3607
3588
|
- error "pmid not found" or "invalid id" \u2014 the ID itself is wrong; the same call gives the same answer.
|
|
3608
|
-
- Calls may run in parallel:
|
|
3589
|
+
- Calls may run in parallel: library_add, library_read and library_verify. Two calls on the same missing record share one add.
|
|
3609
3590
|
|
|
3610
3591
|
Verifying a claim
|
|
3611
|
-
-
|
|
3592
|
+
- library_verify judges one claim against one stored document. A finding backed by three papers takes three calls with the same claim.
|
|
3612
3593
|
- It expects the finding as a complete sentence: who or what, the intervention, the outcome, the number. Citation markers and commentary around the finding are not part of what a document states.
|
|
3613
3594
|
- It can only confirm what this one document states. A finding about two interventions from two papers is two claims, each against its own paper. A summary of several trials ("trials used 60\u2013200 mg") is stated by none of them; the dose this one trial used is.
|
|
3614
3595
|
- It returns error for a claim under four words, a sentence fragment, or a claim still containing citations. A short finding with its setting (in vitro, cell line, dose) is accepted; filler words do not help.
|
|
3615
3596
|
- verdict "pass" \u2014 the document states the claim. score is the model's probability.
|
|
3616
|
-
- verdict "fail" \u2014 the stored full text does not state the claim as worded;
|
|
3597
|
+
- verdict "fail" \u2014 the stored full text does not state the claim as worded; library_read with a query shows what the document does say. Or, with basis "title", nothing is stored to library_read: reason says so, score is null.
|
|
3617
3598
|
- verdict "unverifiable" \u2014 only the abstract is stored and it does not state the claim; the full text may (reason says so). A retry gives the same answer until a full text is stored.
|
|
3618
|
-
- glyphs_unresolved above 0 \u2014 the PDF font lost symbols (minus, \xB1, \u2264); a fail on a number may be the font.
|
|
3599
|
+
- glyphs_unresolved above 0 \u2014 the PDF font lost symbols (minus, \xB1, \u2264); a fail on a number may be the font. library_read shows the passage.
|
|
3619
3600
|
- superseded true \u2014 a published correction replaced the figure the claim uses; the status's corrections holds the notice.
|
|
3620
3601
|
|
|
3621
3602
|
Reading a document
|
|
3622
|
-
-
|
|
3603
|
+
- library_read with only the ID returns the outline: every heading with its word count, and whether the full text or only the abstract is stored.
|
|
3623
3604
|
- section \u2014 a heading from the outline, matched case-insensitively as a substring; the text is capped at max_chars (default 6000). A heading that occurs several times is listed as "Participants [3/12]" and is addressed that way.
|
|
3624
3605
|
- query \u2014 returns the passages holding every term: up to max_hits (default 5), each at most 1000 characters around the matches, with its section, best first. It works best with two or three distinctive terms the document itself would use (a measure and a unit, an outcome and a group, a compound name), not a claim sentence. Case, accents, hyphens, plurals and -ed/-ing forms are matched.
|
|
3625
3606
|
- partial true \u2014 no passage holds every term; missing_terms are the words the document never uses. The same query without them finds the passage if there is one.
|
|
@@ -3651,7 +3632,7 @@ function registerTools(server2) {
|
|
|
3651
3632
|
let ran = false;
|
|
3652
3633
|
const { hit } = await fulltextCascade(paper, meta.attempts, (e) => {
|
|
3653
3634
|
if (e.ok === "alert") {
|
|
3654
|
-
log("
|
|
3635
|
+
log("library_add", requested, `!! ${e.message}`);
|
|
3655
3636
|
return;
|
|
3656
3637
|
}
|
|
3657
3638
|
if (e.ok === "try") tried.add(e.tier);
|
|
@@ -3659,7 +3640,7 @@ function registerTools(server2) {
|
|
|
3659
3640
|
ran = true;
|
|
3660
3641
|
if (e.ok === false) meta = recordAttempts(paths, meta, [attemptLabel(e)]);
|
|
3661
3642
|
log(
|
|
3662
|
-
"
|
|
3643
|
+
"library_add",
|
|
3663
3644
|
requested,
|
|
3664
3645
|
e.ok === true ? `tier ${e.tier} ok (${e.format}, ${e.version})` : e.ok === "try" ? `tier ${e.tier} ${e.message}` : e.skipped ? `tier ${e.tier} skipped: ${e.reason}` : tried.has(e.tier) ? `tier ${e.tier} failed` : `tier ${e.tier} failed: ${e.reason}`
|
|
3665
3646
|
);
|
|
@@ -3671,7 +3652,7 @@ function registerTools(server2) {
|
|
|
3671
3652
|
if (entryExists(paths)) {
|
|
3672
3653
|
const meta = readMeta(paths);
|
|
3673
3654
|
if (daysSince(meta.pubmed_checked) < settings().refresh_after_days) {
|
|
3674
|
-
log("
|
|
3655
|
+
log("library_add", requested, `exists level=full_text retrieved_by=dailymed ${meta.version ?? ""} ${meta.year ?? ""}`);
|
|
3675
3656
|
return statusFromMeta(paths, meta, status);
|
|
3676
3657
|
}
|
|
3677
3658
|
try {
|
|
@@ -3682,14 +3663,14 @@ function registerTools(server2) {
|
|
|
3682
3663
|
if (version === meta.version) {
|
|
3683
3664
|
const next2 = { ...meta, pubmed_checked: today() };
|
|
3684
3665
|
writeMeta(paths, next2);
|
|
3685
|
-
log("
|
|
3666
|
+
log("library_add", requested, `exists level=full_text retrieved_by=dailymed ${meta.version ?? ""} checked`);
|
|
3686
3667
|
return statusFromMeta(paths, next2, status);
|
|
3687
3668
|
}
|
|
3688
3669
|
const next = writeLabelEntry(paths, s, xml, `${pkg.name} ${pkg.version}`);
|
|
3689
|
-
log("
|
|
3670
|
+
log("library_add", requested, `refreshed level=full_text retrieved_by=dailymed ${meta.version ?? "?"} \u2192 ${next.version} effective=${s.effective}`);
|
|
3690
3671
|
return statusFromMeta(paths, next, status);
|
|
3691
3672
|
} catch (e) {
|
|
3692
|
-
log("
|
|
3673
|
+
log("library_add", requested, `refresh failed: ${e instanceof Error ? e.message : String(e)}`);
|
|
3693
3674
|
return statusFromMeta(paths, meta, status);
|
|
3694
3675
|
}
|
|
3695
3676
|
}
|
|
@@ -3697,15 +3678,15 @@ function registerTools(server2) {
|
|
|
3697
3678
|
const xml = await fetchSpl(setid);
|
|
3698
3679
|
const s = parseSpl(xml);
|
|
3699
3680
|
if (!s) {
|
|
3700
|
-
log("
|
|
3681
|
+
log("library_add", requested, "dailymed failed: not an SPL document");
|
|
3701
3682
|
return { ...status, error: "dailymed: not an SPL document" };
|
|
3702
3683
|
}
|
|
3703
3684
|
const meta = writeLabelEntry(paths, s, xml, `${pkg.name} ${pkg.version}`);
|
|
3704
|
-
log("
|
|
3685
|
+
log("library_add", requested, `dailymed fetched level=full_text ${meta.version ?? ""} effective=${s.effective} ${s.brand ?? ""} (${s.generic ?? ""}) entry=${paths.entry}`);
|
|
3705
3686
|
return statusFromMeta(paths, meta, status);
|
|
3706
3687
|
} catch (e) {
|
|
3707
3688
|
const msg = e instanceof Error ? e.message : String(e);
|
|
3708
|
-
log("
|
|
3689
|
+
log("library_add", requested, `dailymed failed: ${msg}`);
|
|
3709
3690
|
return { ...status, error: msg };
|
|
3710
3691
|
}
|
|
3711
3692
|
}
|
|
@@ -3713,7 +3694,7 @@ function registerTools(server2) {
|
|
|
3713
3694
|
if (entryExists(paths)) {
|
|
3714
3695
|
const meta = readMeta(paths);
|
|
3715
3696
|
if (daysSince(meta.pubmed_checked) < settings().refresh_after_days) {
|
|
3716
|
-
log("
|
|
3697
|
+
log("library_add", requested, `exists level=full_text retrieved_by=ctgov updated ${meta.version ?? ""}`);
|
|
3717
3698
|
return statusFromMeta(paths, meta, status);
|
|
3718
3699
|
}
|
|
3719
3700
|
try {
|
|
@@ -3723,14 +3704,14 @@ function registerTools(server2) {
|
|
|
3723
3704
|
if (s.lastUpdate === meta.version) {
|
|
3724
3705
|
const next2 = { ...meta, pubmed_checked: today() };
|
|
3725
3706
|
writeMeta(paths, next2);
|
|
3726
|
-
log("
|
|
3707
|
+
log("library_add", requested, `exists level=full_text retrieved_by=ctgov updated ${meta.version ?? ""} checked`);
|
|
3727
3708
|
return statusFromMeta(paths, next2, status);
|
|
3728
3709
|
}
|
|
3729
3710
|
const next = writeStudyEntry(paths, s, json2, `${pkg.name} ${pkg.version}`);
|
|
3730
|
-
log("
|
|
3711
|
+
log("library_add", requested, `refreshed level=full_text retrieved_by=ctgov updated ${meta.version ?? "?"} \u2192 ${next.version} ${s.status ?? ""}${s.hasResults ? " results" : ""}`);
|
|
3731
3712
|
return statusFromMeta(paths, next, status);
|
|
3732
3713
|
} catch (e) {
|
|
3733
|
-
log("
|
|
3714
|
+
log("library_add", requested, `refresh failed: ${e instanceof Error ? e.message : String(e)}`);
|
|
3734
3715
|
return statusFromMeta(paths, meta, status);
|
|
3735
3716
|
}
|
|
3736
3717
|
}
|
|
@@ -3738,20 +3719,20 @@ function registerTools(server2) {
|
|
|
3738
3719
|
const json2 = await fetchStudy(nctid);
|
|
3739
3720
|
const s = parseStudy(json2);
|
|
3740
3721
|
if (!s) {
|
|
3741
|
-
log("
|
|
3722
|
+
log("library_add", requested, "ctgov failed: not a study record");
|
|
3742
3723
|
return { ...status, error: "ctgov: not a study record" };
|
|
3743
3724
|
}
|
|
3744
3725
|
const meta = writeStudyEntry(paths, s, json2, `${pkg.name} ${pkg.version}`);
|
|
3745
|
-
log("
|
|
3726
|
+
log("library_add", requested, `ctgov fetched level=full_text updated ${meta.version ?? ""} ${s.status ?? ""}${s.hasResults ? " results" : ""} ${s.acronym ?? ""} entry=${paths.entry}`);
|
|
3746
3727
|
return statusFromMeta(paths, meta, status);
|
|
3747
3728
|
} catch (e) {
|
|
3748
3729
|
const msg = e instanceof Error ? e.message : String(e);
|
|
3749
|
-
log("
|
|
3730
|
+
log("library_add", requested, `ctgov failed: ${msg}`);
|
|
3750
3731
|
return { ...status, error: msg };
|
|
3751
3732
|
}
|
|
3752
3733
|
}
|
|
3753
3734
|
server2.tool(
|
|
3754
|
-
"
|
|
3735
|
+
"library_add",
|
|
3755
3736
|
"Download and store one record \u2014 a paper (pmid:, doi:), a drug label (dailymed:<set ID>) or a trial record (nct:<8 digits>) \u2014 and return its status JSON. A paper gets its record, abstract and correction notices from PubMed (Crossref for a DOI PubMed does not index), then the full text from the configured sources; a label or trial record is stored as a full text. A stored full text is returned without any download; an abstract-level record retries only sources that failed transiently; a record older than 30 days gets its notices re-read.",
|
|
3756
3737
|
{ id: idSchema },
|
|
3757
3738
|
({ id }) => addEntry(id)
|
|
@@ -3770,7 +3751,7 @@ function registerTools(server2) {
|
|
|
3770
3751
|
const t0 = Date.now();
|
|
3771
3752
|
let parsed = parseId(id);
|
|
3772
3753
|
if (parsed.error) {
|
|
3773
|
-
log("
|
|
3754
|
+
log("library_add", parsed.requested, `error ${parsed.error}`);
|
|
3774
3755
|
return json({ ...emptyStatus(parsed.requested, parsed.type, parsed.id, FH_LIBRARY_DIR), error: parsed.error });
|
|
3775
3756
|
}
|
|
3776
3757
|
let copyTo = null;
|
|
@@ -3781,18 +3762,18 @@ function registerTools(server2) {
|
|
|
3781
3762
|
try {
|
|
3782
3763
|
pmid = await pmidForDoi(parsed.id);
|
|
3783
3764
|
} catch (e) {
|
|
3784
|
-
log("
|
|
3765
|
+
log("library_add", parsed.requested, `pubmed lookup failed: ${e instanceof Error ? e.message : String(e)}`);
|
|
3785
3766
|
}
|
|
3786
3767
|
if (pmid && !entryExists(entryPaths(FH_LIBRARY_DIR, "pmid", pmid))) {
|
|
3787
3768
|
try {
|
|
3788
3769
|
if (!parsePubmedXml(await fetchPubmedXml(pmid))) {
|
|
3789
|
-
log("
|
|
3770
|
+
log("library_add", parsed.requested, `pmid:${pmid} has no readable PubMed record \u2014 kept as doi:`);
|
|
3790
3771
|
pmid = null;
|
|
3791
3772
|
}
|
|
3792
3773
|
} catch {
|
|
3793
3774
|
}
|
|
3794
3775
|
}
|
|
3795
|
-
if (pmid) log("
|
|
3776
|
+
if (pmid) log("library_add", parsed.requested, `indexed in PubMed as pmid:${pmid}`);
|
|
3796
3777
|
}
|
|
3797
3778
|
if (pmid) {
|
|
3798
3779
|
copyTo = doiPaths;
|
|
@@ -3820,10 +3801,10 @@ function registerTools(server2) {
|
|
|
3820
3801
|
const before = noticeTag(meta);
|
|
3821
3802
|
meta = { ...meta, corrections: r.corrections, retracted: r.corrections.some(isRetractionNotice), funding: meta.funding.length ? meta.funding : r.funding, pubmed_checked: today() };
|
|
3822
3803
|
writeMeta(paths, meta);
|
|
3823
|
-
log("
|
|
3824
|
-
} else log("
|
|
3804
|
+
log("library_add", parsed.requested, `refreshed crossref${noticeTag(meta) !== before ? ` changed: ${before.trim() || "none"} \u2192 ${noticeTag(meta).trim() || "none"}` : ""}`);
|
|
3805
|
+
} else log("library_add", parsed.requested, "refresh failed: doi not found");
|
|
3825
3806
|
} catch (e) {
|
|
3826
|
-
log("
|
|
3807
|
+
log("library_add", parsed.requested, `refresh failed: ${e instanceof Error ? e.message : String(e)}`);
|
|
3827
3808
|
}
|
|
3828
3809
|
} else if (daysSince(meta.pubmed_checked) >= settings().refresh_after_days) {
|
|
3829
3810
|
try {
|
|
@@ -3832,24 +3813,24 @@ function registerTools(server2) {
|
|
|
3832
3813
|
if (record) {
|
|
3833
3814
|
const before = noticeTag(meta);
|
|
3834
3815
|
meta = refreshRecord(paths, meta, record, xml);
|
|
3835
|
-
log("
|
|
3836
|
-
} else log("
|
|
3816
|
+
log("library_add", parsed.requested, `refreshed pubmed_checked=${meta.pubmed_checked}${noticeTag(meta) !== before ? ` changed: ${before.trim() || "none"} \u2192 ${noticeTag(meta).trim() || "none"}` : ""}`);
|
|
3817
|
+
} else log("library_add", parsed.requested, "refresh failed: pmid not found");
|
|
3837
3818
|
} catch (e) {
|
|
3838
|
-
log("
|
|
3819
|
+
log("library_add", parsed.requested, `refresh failed: ${e instanceof Error ? e.message : String(e)}`);
|
|
3839
3820
|
}
|
|
3840
3821
|
}
|
|
3841
3822
|
if (meta.level === "full_text") {
|
|
3842
|
-
log("
|
|
3823
|
+
log("library_add", parsed.requested, `exists level=full_text retrieved_by=${meta.retrieved_by}${noticeTag(meta)} ${meta.doi ? `doi:${meta.doi}` : "no-doi"}`);
|
|
3843
3824
|
return finish(meta);
|
|
3844
3825
|
}
|
|
3845
|
-
log("
|
|
3826
|
+
log("library_add", parsed.requested, `exists level=${levelWord(meta)}${noticeTag(meta)} ${meta.doi ? `doi:${meta.doi}` : "no-doi"}`);
|
|
3846
3827
|
} else if (isDoi) {
|
|
3847
3828
|
let json_;
|
|
3848
3829
|
try {
|
|
3849
3830
|
json_ = await fetchCrossref(parsed.id);
|
|
3850
3831
|
} catch (e) {
|
|
3851
3832
|
const msg = e instanceof Error ? e.message : String(e);
|
|
3852
|
-
log("
|
|
3833
|
+
log("library_add", parsed.requested, `crossref failed: ${msg}`);
|
|
3853
3834
|
return json({ ...status, error: msg });
|
|
3854
3835
|
}
|
|
3855
3836
|
let record = json_ ? parseCrossref(json_) : null;
|
|
@@ -3863,7 +3844,7 @@ function registerTools(server2) {
|
|
|
3863
3844
|
}
|
|
3864
3845
|
}
|
|
3865
3846
|
if (!record) {
|
|
3866
|
-
log("
|
|
3847
|
+
log("library_add", parsed.requested, "crossref failed: doi not found at Crossref or doi.org");
|
|
3867
3848
|
return json({ ...status, error: "doi not found" });
|
|
3868
3849
|
}
|
|
3869
3850
|
if (!record.doi) record.doi = parsed.id;
|
|
@@ -3876,53 +3857,53 @@ function registerTools(server2) {
|
|
|
3876
3857
|
}
|
|
3877
3858
|
}
|
|
3878
3859
|
meta = writeDoiEntry(paths, record, json_, `${pkg.name} ${pkg.version}`, abstractFrom, recordFrom);
|
|
3879
|
-
log("
|
|
3860
|
+
log("library_add", parsed.requested, `${recordFrom === "crossref" ? "crossref" : "doi.org"} fetched level=${levelWord(meta)} \u2014 not in PubMed${noticeTag(meta)}${meta.has_abstract ? ` abstract from ${abstractFrom}` : ""} entry=${paths.entry}`);
|
|
3880
3861
|
} else {
|
|
3881
3862
|
let xml;
|
|
3882
3863
|
try {
|
|
3883
3864
|
xml = await fetchPubmedXml(parsed.id);
|
|
3884
3865
|
} catch (e) {
|
|
3885
3866
|
const msg = e instanceof Error ? e.message : String(e);
|
|
3886
|
-
log("
|
|
3867
|
+
log("library_add", parsed.requested, `pubmed failed: ${msg}`);
|
|
3887
3868
|
return json({ ...status, error: msg });
|
|
3888
3869
|
}
|
|
3889
3870
|
const record = parsePubmedXml(xml);
|
|
3890
3871
|
if (!record) {
|
|
3891
|
-
log("
|
|
3872
|
+
log("library_add", parsed.requested, "pubmed failed: pmid not found");
|
|
3892
3873
|
return json({ ...status, error: "pmid not found" });
|
|
3893
3874
|
}
|
|
3894
3875
|
meta = writeEntry(paths, record, xml, `${pkg.name} ${pkg.version}`);
|
|
3895
|
-
log("
|
|
3876
|
+
log("library_add", parsed.requested, `pubmed fetched level=${levelWord(meta)}${noticeTag(meta)} ${meta.doi ? `doi:${meta.doi}` : "no-doi"}${meta.pmcid ? ` pmcid:${meta.pmcid}` : ""} entry=${paths.entry}`);
|
|
3896
3877
|
}
|
|
3897
3878
|
({ meta, ran } = await completeFulltext(paths, meta, parsed.requested));
|
|
3898
3879
|
if (entryExistedBefore && !ran) return finish(meta);
|
|
3899
3880
|
const secs = `${((Date.now() - t0) / 1e3).toFixed(1).padStart(4, "0")}s`;
|
|
3900
|
-
log("
|
|
3881
|
+
log("library_add", parsed.requested, `done ${secs} level=${levelWord(meta)} retrieved_by=${meta.retrieved_by}`);
|
|
3901
3882
|
return finish(meta);
|
|
3902
3883
|
}
|
|
3903
3884
|
server2.tool(
|
|
3904
|
-
"
|
|
3905
|
-
'Read-only status of the library entry for one paper ID: the same status JSON as
|
|
3885
|
+
"library_status",
|
|
3886
|
+
'Read-only status of the library entry for one paper ID: the same status JSON as library_add, or level "missing" when there is no entry. Never fetches.',
|
|
3906
3887
|
{ id: idSchema },
|
|
3907
3888
|
async ({ id }) => {
|
|
3908
3889
|
const parsed = parseId(id);
|
|
3909
3890
|
const status = emptyStatus(parsed.requested, parsed.type, parsed.id, FH_LIBRARY_DIR);
|
|
3910
3891
|
if (parsed.error) {
|
|
3911
|
-
log("
|
|
3912
|
-
return json({ ...
|
|
3892
|
+
log("library_status", parsed.requested, `error ${parsed.error}`);
|
|
3893
|
+
return json({ ...library_status, error: parsed.error });
|
|
3913
3894
|
}
|
|
3914
3895
|
const paths = entryPaths(FH_LIBRARY_DIR, parsed.type, parsed.id);
|
|
3915
3896
|
if (entryExists(paths)) {
|
|
3916
|
-
const s = readStatus(paths,
|
|
3917
|
-
log("
|
|
3897
|
+
const s = readStatus(paths, library_status);
|
|
3898
|
+
log("library_status", parsed.requested, `level=${s.level}`);
|
|
3918
3899
|
return json(s);
|
|
3919
3900
|
}
|
|
3920
|
-
log("
|
|
3901
|
+
log("library_status", parsed.requested, "level=missing");
|
|
3921
3902
|
return json({ ...status, entry: paths.entry, entry_dir: paths.entryDir, level: "missing" });
|
|
3922
3903
|
}
|
|
3923
3904
|
);
|
|
3924
3905
|
server2.tool(
|
|
3925
|
-
"
|
|
3906
|
+
"library_read",
|
|
3926
3907
|
`One piece of a stored document's Markdown, never the whole. No section/query: the outline (headings with word counts; full text or abstract only). section: that heading's own text (case-insensitive substring; a repeated heading as "Participants [3/12]"), capped at max_chars, its subsections listed. query: passages holding every term, each a window of at most 1000 chars with its section, best first, at most max_hits; partial: true with missing_terms when no passage holds them all. section and query together: inside that section. A missing paper is added first (added: true).`,
|
|
3927
3908
|
{
|
|
3928
3909
|
id: idSchema,
|
|
@@ -3935,17 +3916,17 @@ function registerTools(server2) {
|
|
|
3935
3916
|
const parsed = parseId(id);
|
|
3936
3917
|
const out = (o2) => ({ content: [{ type: "text", text: JSON.stringify(o2, null, 2) }] });
|
|
3937
3918
|
if (parsed.error) {
|
|
3938
|
-
log("
|
|
3919
|
+
log("library_read", parsed.requested, `error ${parsed.error}`);
|
|
3939
3920
|
return out({ requested: parsed.requested, error: parsed.error });
|
|
3940
3921
|
}
|
|
3941
3922
|
const paths = entryPaths(FH_LIBRARY_DIR, parsed.type, parsed.id);
|
|
3942
3923
|
let added = false;
|
|
3943
3924
|
if (!entryExists(paths)) {
|
|
3944
|
-
log("
|
|
3925
|
+
log("library_read", parsed.requested, "missing adding first");
|
|
3945
3926
|
const r = JSON.parse((await addEntry(id)).content[0].text);
|
|
3946
3927
|
if (!entryExists(paths)) {
|
|
3947
|
-
log("
|
|
3948
|
-
return out({ requested: parsed.requested, entry: paths.entry, error: `not in the library and
|
|
3928
|
+
log("library_read", parsed.requested, `error ${r.error ?? "library_add failed"}`);
|
|
3929
|
+
return out({ requested: parsed.requested, entry: paths.entry, error: `not in the library and library_add failed: ${r.error ?? "unknown"}` });
|
|
3949
3930
|
}
|
|
3950
3931
|
added = true;
|
|
3951
3932
|
}
|
|
@@ -3958,12 +3939,12 @@ function registerTools(server2) {
|
|
|
3958
3939
|
if (section2 && query) {
|
|
3959
3940
|
const found = findSections(md, section2);
|
|
3960
3941
|
if (found.length !== 1) {
|
|
3961
|
-
log("
|
|
3942
|
+
log("library_read", parsed.requested, `section "${section2}" query "${query}" \u2192 ${found.length ? `${found.length} sections match` : "section not found"}`);
|
|
3962
3943
|
if (!found.length) return out({ ...base, error: `no section matching "${section2}"`, outline: headings2() });
|
|
3963
3944
|
return out({ ...base, error: `${found.length} sections match "${section2}" \u2014 name one, e.g. "${found[0].label}"`, ambiguous: found.map((s) => ({ heading: s.label, words: s.words })) });
|
|
3964
3945
|
}
|
|
3965
3946
|
const r = search(md, query, max_hits ?? 5, { scope: found[0] });
|
|
3966
|
-
log("
|
|
3947
|
+
log("library_read", parsed.requested, `section "${found[0].label}" query "${query}" \u2192 ${describe2(r)}`);
|
|
3967
3948
|
return out({ ...base, section: found[0].label, query, ...r });
|
|
3968
3949
|
}
|
|
3969
3950
|
if (section2) {
|
|
@@ -3979,29 +3960,29 @@ function registerTools(server2) {
|
|
|
3979
3960
|
via = " (assembled)";
|
|
3980
3961
|
}
|
|
3981
3962
|
if (!sec) {
|
|
3982
|
-
log("
|
|
3963
|
+
log("library_read", parsed.requested, `section "${section2}" not found`);
|
|
3983
3964
|
return out({ ...base, error: `no section matching "${section2}"`, outline: headings2() });
|
|
3984
3965
|
}
|
|
3985
3966
|
if ("ambiguous" in sec) {
|
|
3986
|
-
log("
|
|
3967
|
+
log("library_read", parsed.requested, `section "${section2}" \u2192 ${sec.ambiguous.length} sections match`);
|
|
3987
3968
|
return out({ ...base, error: `${sec.ambiguous.length} sections match "${section2}" \u2014 name one, e.g. "${sec.ambiguous[0].heading}"`, ...sec });
|
|
3988
3969
|
}
|
|
3989
|
-
log("
|
|
3970
|
+
log("library_read", parsed.requested, `section "${section2}" \u2192 "${sec.heading}" ${sec.words} words${sec.truncated ? ` truncated to ${sec.returned_words}` : ""}${sec.subsections ? ` ${sec.subsections.length} subsections` : ""}${via}`);
|
|
3990
3971
|
return out({ ...base, ...sec });
|
|
3991
3972
|
}
|
|
3992
3973
|
if (query) {
|
|
3993
3974
|
const r = search(md, query, max_hits ?? 5);
|
|
3994
|
-
log("
|
|
3975
|
+
log("library_read", parsed.requested, `query "${query}" \u2192 ${describe2(r)}`);
|
|
3995
3976
|
if (!r.passages.length) return out({ ...base, query, ...r, sections: headings2().map((s) => s.heading) });
|
|
3996
3977
|
return out({ ...base, query, ...r });
|
|
3997
3978
|
}
|
|
3998
3979
|
const o = outline(md, text3);
|
|
3999
|
-
log("
|
|
3980
|
+
log("library_read", parsed.requested, `outline ${o.sections.length} sections ${o.words} words`);
|
|
4000
3981
|
return out({ ...base, ...o });
|
|
4001
3982
|
}
|
|
4002
3983
|
);
|
|
4003
3984
|
server2.tool(
|
|
4004
|
-
"
|
|
3985
|
+
"library_verify",
|
|
4005
3986
|
"Does one source state one finding of a review? claim: the finding as a complete statement (subject, intervention, outcome, number), without citations or the review's framing, and only what this source reports. id: one source the review cites for it \u2014 paper, drug label (dailymed:\u2026) or trial record (nct:\u2026); added first when missing. The server reads the stored text and asks a decision model; nothing enters your context. verdict: pass (score is the probability) | fail (the full text does not state it; or only the title is stored \u2014 reason says so) | unverifiable (only the abstract is stored and does not state it \u2014 reason says so) | error (claim refused: under four words, a fragment, carrying citations; add failed; no model; model failure). basis: full_text | abstract | title. glyphs_unresolved: symbols the PDF font lost; above 0 a failed number may be the font. superseded: true when a stored correction replaced the claim's figure.",
|
|
4006
3987
|
{
|
|
4007
3988
|
claim: z.string().describe("One finding, verbatim from the review, without its citations."),
|
|
@@ -4012,18 +3993,18 @@ function registerTools(server2) {
|
|
|
4012
3993
|
const cfg = verifyConfig();
|
|
4013
3994
|
const pid = parseId(id);
|
|
4014
3995
|
if (!pid.error && !entryExists(entryPaths(FH_LIBRARY_DIR, pid.type, pid.id))) {
|
|
4015
|
-
log("
|
|
3996
|
+
log("library_verify", pid.requested, "missing adding first");
|
|
4016
3997
|
await addEntry(id);
|
|
4017
3998
|
}
|
|
4018
3999
|
const r = await verifyClaim(claim, id, cfg, cfg ? decider(cfg) : null);
|
|
4019
4000
|
const secs = `${((Date.now() - t0) / 1e3).toFixed(1).padStart(4, "0")}s`;
|
|
4020
|
-
log("
|
|
4001
|
+
log("library_verify", r.id, r.error ? `${modelFailure(r.error, cfg) ? "!! " : ""}${secs} verdict=${"error".padEnd(12)} ${r.error} | ${claim}` : `${secs} verdict=${r.verdict.padEnd(12)} basis=${r.basis.padEnd(9)} score=${r.score === null ? "- " : r.score.toFixed(3)}${r.windows > 1 ? ` windows=${r.windows}` : ""}${r.superseded ? " superseded" : ""}${r.glyphs_unresolved ? ` glyphs=${r.glyphs_unresolved}` : ""} | ${claim}`);
|
|
4021
4002
|
return { content: [{ type: "text", text: JSON.stringify(r, null, 2) }] };
|
|
4022
4003
|
}
|
|
4023
4004
|
);
|
|
4024
4005
|
server2.tool(
|
|
4025
|
-
"
|
|
4026
|
-
"The whole library as JSON: library directory and host; entry count and size on disk; per type (pmid, doi, dailymed, nct) the entries by level (full_text, abstract, title) and retractions; counts of retracted and not-citable entries and of errata, expressions of concern and updates; the retrieval sources in run order and which are off (tiers); the download ladder
|
|
4006
|
+
"library_info",
|
|
4007
|
+
"The whole library as JSON: library directory and host; entry count and size on disk; per type (pmid, doi, dailymed, nct) the entries by level (full_text, abstract, title) and retractions; counts of retracted and not-citable entries and of errata, expressions of concern and updates; the retrieval sources in run order and which are off (tiers); the download ladder (download): the Playwright servers with the version they must run, the Chrome rungs, the Bright Data unlocker; the configuration and where each value came from \u2014 environment, .env or default, never the value (config); the decision model behind library_verify and whether it answers right now (library_verify). ids=true also lists every entry ID.",
|
|
4027
4008
|
{ ids: z.boolean().optional().describe("Also list every entry ID (type:value).") },
|
|
4028
4009
|
async ({ ids }) => {
|
|
4029
4010
|
const info = libraryInfo(FH_LIBRARY_DIR, HOST, ids === true);
|
|
@@ -4037,16 +4018,16 @@ function registerTools(server2) {
|
|
|
4037
4018
|
info.verify = { ...info.verify, ok: false, ms: Date.now() - t0, error: e instanceof Error ? e.message : String(e) };
|
|
4038
4019
|
}
|
|
4039
4020
|
}
|
|
4040
|
-
log("
|
|
4021
|
+
log("library_info", null, `entries=${library_info.entries}${library_info.verify ? ` verify ${library_info.verify.model} ${library_info.verify.ok ? `ok ${library_info.verify.ms} ms` : `down: ${library_info.verify.error}`}` : " verify not configured"}`);
|
|
4041
4022
|
return { content: [{ type: "text", text: JSON.stringify(info, null, 2) }] };
|
|
4042
4023
|
}
|
|
4043
4024
|
);
|
|
4044
4025
|
server2.tool(
|
|
4045
|
-
"
|
|
4026
|
+
"library_version",
|
|
4046
4027
|
"Get the running fh-library MCP server's package name and version, and the resolved library directory. Useful to confirm which build is loaded.",
|
|
4047
4028
|
{},
|
|
4048
4029
|
async () => {
|
|
4049
|
-
log("
|
|
4030
|
+
log("library_version", null, `${pkg.name} ${pkg.library_version}`);
|
|
4050
4031
|
return { content: [{ type: "text", text: `${pkg.name} v${pkg.version} \xB7 host: ${HOST} \xB7 library_dir: ${FH_LIBRARY_DIR} \xB7 log: ${LOG_FILE}` }] };
|
|
4051
4032
|
}
|
|
4052
4033
|
);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "fh-library",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.2",
|
|
4
4
|
"mcpName": "io.github.forever-healthy/fh-library",
|
|
5
5
|
"description": "MCP Server that builds a local Markdown library of the scientific papers cited by evidence reviews — fetches metadata, abstract and full text per PubMed ID through a configurable cascade of retrieval services",
|
|
6
6
|
"type": "module",
|
|
@@ -33,10 +33,10 @@
|
|
|
33
33
|
],
|
|
34
34
|
"author": "Forever Healthy Foundation",
|
|
35
35
|
"license": "MIT",
|
|
36
|
-
"homepage": "https://github.com/forever-healthy/
|
|
36
|
+
"homepage": "https://github.com/forever-healthy/fh-library",
|
|
37
37
|
"repository": {
|
|
38
38
|
"type": "git",
|
|
39
|
-
"url": "https://github.com/forever-healthy/
|
|
39
|
+
"url": "https://github.com/forever-healthy/fh-library.git"
|
|
40
40
|
},
|
|
41
41
|
"private": false
|
|
42
42
|
}
|