ai4l-library 0.1.3 → 0.1.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +9 -0
- package/README.md +35 -58
- package/dist/index.js +190 -209
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,14 @@
|
|
|
1
1
|
# AI4L Library — Change Log
|
|
2
2
|
|
|
3
|
+
### v0.1.4 — 2026-11-07
|
|
4
|
+
|
|
5
|
+
* `.env` is the whole configuration: every value an environment variable, the tier set and the defaults built in. `library/config.json` and `AI4L_LIBRARY_CONFIG` are gone, and with them `${NAME}` references and the per-tier `description`.
|
|
6
|
+
* Renamed, no aliases — `CONTACT_EMAIL` → `AI4L_LIBRARY_CONTACT_EMAIL`; `ELSEVIER_API_KEY` → `AI4L_LIBRARY_ELSEVIER_API_KEY`; `WILEY_TDM_TOKEN` → `AI4L_LIBRARY_WILEY_TDM_TOKEN`; `CORE_API_KEY` → `AI4L_LIBRARY_CORE_API_KEY`; `PERPLEXITY_URL` / `PERPLEXITY_API_KEY` / `PERPLEXITY_MODEL` → `AI4L_LIBRARY_VERIFY_URL` / `_API_KEY` / `_MODEL`; config.json's `threshold`, `timeout_seconds`, `retry_after_days`, `retry_blocked`, `refresh_after_days`, `download.headless` → `AI4L_LIBRARY_VERIFY_THRESHOLD`, `AI4L_LIBRARY_VERIFY_TIMEOUT_SECS`, `AI4L_LIBRARY_RETRY_AFTER_DAYS`, `AI4L_LIBRARY_RETRY_BLOCKED`, `AI4L_LIBRARY_REFRESH_AFTER_DAYS`, `AI4L_LIBRARY_BROWSER_HEADLESS`. `NCBI_API_KEY`, `BRIGHTDATA_API_TOKEN`, `BRIGHTDATA_ZONE` and `AI4L_LIBRARY_BROWSER_n` unchanged.
|
|
7
|
+
* `AI4L_LIBRARY_TIERS` is the whole ladder: the listed tiers in exactly that order, a tier not listed is off (an unlisted tier no longer runs after the listed ones).
|
|
8
|
+
* One proxy tier, `proxy`, instead of url tiers: `AI4L_LIBRARY_PROXY_URL` (template with `{doi}` `{pmid}` `{pmcid}`) and `AI4L_LIBRARY_PROXY_HEADERS`; `PROXY_TOKEN` and the `my-proxy` example are gone.
|
|
9
|
+
* `library_info`: `config` replaces `tiers_config` — the cascade's settings, problems with `AI4L_LIBRARY_TIERS`, and per variable where its value came from (environment, `.env`, default), never the value; `tiers` in run order without descriptions (the README describes them).
|
|
10
|
+
* The container's working directory is `/library`; the configuration arrives as its environment.
|
|
11
|
+
|
|
3
12
|
### v0.1.3 — 2026-11-07
|
|
4
13
|
|
|
5
14
|
* IDs: `pmid:` (bare digits count as one), `doi:` for papers PubMed does not index (a DOI PubMed indexes is stored under its PMID and copied under the DOI, so every tool takes either), `dailymed:<set ID>` drug labels, `nct:` trial records.
|
package/README.md
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
|
|
2
2
|
|
|
3
3
|
[_2026-Forever_Healthy-573D7D.svg)](https://forever-healthy.org)
|
|
4
4
|
[](https://www.npmjs.com/package/ai4l-library)
|
|
@@ -11,14 +11,14 @@
|
|
|
11
11
|
|
|
12
12
|
A [Model Context Protocol](https://modelcontextprotocol.io) server that builds a local library of the scientific papers an evidence review cites. Given a paper ID (`type:value`, e.g. `pmid:30955454`), it fetches the paper's metadata, abstract and — where obtainable — full text through a configurable cascade of retrieval services (PubMed, PMC, Europe PMC, Unpaywall, OpenAlex, …), converts everything to Markdown, and stores it under a mounted `library/` directory. Nothing flows through the model's context: the server writes to disk itself and returns only a small status per call.
|
|
13
13
|
|
|
14
|
-
**Out of the box, keyless, everywhere:** PubMed, PMC, Europe PMC, Unpaywall and OpenAlex, a plain fetch, and search inside stored papers. Everything stronger is an endpoint or a key the deployment adds in
|
|
14
|
+
**Out of the box, keyless, everywhere:** PubMed, PMC, Europe PMC, Unpaywall and OpenAlex, a plain fetch, and search inside stored papers. Everything stronger is an endpoint or a key the deployment adds in `.env` — browser rungs (Playwright servers, a real Chrome), Bright Data, CORE, the Elsevier and Wiley text-mining tokens, your own proxy — each with an honest fallback when absent; `library_info` shows what is configured.
|
|
15
15
|
|
|
16
|
-
**Status: v0.2, early development.** Four ID types: papers by `pmid:` and, when PubMed does not index them, by `doi:` (record from Crossref); drug labels by `dailymed:`; trial records by `nct:`. Every paper entry gets metadata and abstract from PubMed (or Crossref and OpenAlex); the full-text cascade then tries PMC (JATS XML), the publishers' text-and-data-mining links (Crossref; Elsevier and Wiley with a token), and open-access copies located via Unpaywall and OpenAlex (PDF, XML or HTML). A full text counts only after checks: the paper's title or DOI in the text, 800+ words of running prose (a publisher's abstract page has 200–600), Methods and Results or a sectioned body, no wall page. `library_read` serves outline, section or search passages for claim verification. Papers whose PMC XML body is withheld by the publisher are reachable only through open-access copies or a licensed
|
|
16
|
+
**Status: v0.2, early development.** Four ID types: papers by `pmid:` and, when PubMed does not index them, by `doi:` (record from Crossref); drug labels by `dailymed:`; trial records by `nct:`. Every paper entry gets metadata and abstract from PubMed (or Crossref and OpenAlex); the full-text cascade then tries PMC (JATS XML), the publishers' text-and-data-mining links (Crossref; Elsevier and Wiley with a token), and open-access copies located via Unpaywall and OpenAlex (PDF, XML or HTML). A full text counts only after checks: the paper's title or DOI in the text, 800+ words of running prose (a publisher's abstract page has 200–600), Methods and Results or a sectioned body, no wall page. `library_read` serves outline, section or search passages for claim verification. Papers whose PMC XML body is withheld by the publisher are reachable only through open-access copies or a licensed proxy; PMC's web pages are for humans and are not scraped. What follows: better PDF layout.
|
|
17
17
|
|
|
18
18
|
|
|
19
19
|
## Tools
|
|
20
20
|
|
|
21
|
-
* `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `coi` and `funding` (PubMed's record, else the paper's own text), `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — files, failed tiers in `attempts`). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `
|
|
21
|
+
* `library_add(id)` → build or complete the library entry for one ID — a paper `pmid:<PMID>` or `doi:<DOI>` (see "Papers without a PMID"), a drug label `dailymed:<set ID>` (see "Drug labels"), or a trial record `nct:<8 digits>` (see "Clinical trial records"); returns its status (level `full_text` / `abstract` / `title` — what the entry holds, `title` when there is neither a full text nor abstract text — retrieving tier, `coi` and `funding` (PubMed's record, else the paper's own text), `citable` with `comment` — the one flag to check before citing, from `retracted` and PubMed's correction notices in `corrections` — `has_abstract` — false for title-only PubMed records such as letters and editorials — files, failed tiers in `attempts`). Idempotent: a `full_text` entry is never refetched, an `abstract` or `title` entry retries only tiers whose earlier failure was transient and past the cooldown; an entry whose PubMed record is older than `AI4L_LIBRARY_REFRESH_AFTER_DAYS` gets the record re-read for new notices. One ID per call.
|
|
22
22
|
* `library_status(id)` → read-only check of presence and level; never fetches
|
|
23
23
|
* `library_info(ids?)` → the whole library, everything the TUI shows: the library directory and host, the decision model with its endpoint and whether it answers right now (`verify`), the entries and their size on disk (`size`), per entry type (`types`: pmid, doi, dailymed, nct) the entries with their levels — `full_text`, `abstract`, `title` — and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the configured tiers, library directory and host; `ids: true` also lists every entry ID
|
|
24
24
|
* `library_verify(claim, id)` → is one finding of a review stated by one paper it cites? `claim` is the finding alone, verbatim, without its citations; `id` one paper (`pmid:`, or any `doi:`), drug label or trial record the review cites for it — a finding cited to three papers is three calls with the same claim; a claim that names a source or joins two findings is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters the model's context. `verdict` **pass** (`score` the probability), **fail** (the full text does not state it), **na** (only the abstract is in the library and does not state it) or **error**; `basis` full_text or abstract. See "Verifying claims" below.
|
|
@@ -35,63 +35,40 @@ IDs follow the `type:value` form: `pmid:`, `doi:`, `dailymed:`, `nct:`; bare dig
|
|
|
35
35
|
| `inbox` | a file you downloaded yourself: drop it into `<library>/inbox/` with the PMID in its name (`39025385.pdf`; PDF, XML or HTML) and the next `library_add` for that ID converts it into the entry and removes it from the inbox. For publishers that hand a PDF to a browser but nothing to a script | PMID |
|
|
36
36
|
| `pmc` | full text as JATS XML via E-utilities; counts only with a `<body>` | PMCID |
|
|
37
37
|
| `europepmc` | Europe PMC's record: its own full text (JATS XML for the open-access subset, the rendered PDF for everything it holds, author manuscripts included) and the free links it lists | – |
|
|
38
|
-
| `tdm` | the publishers' text-and-data-mining links from Crossref; Elsevier (`X-ELS-APIKey`) and Wiley (`Wiley-TDM-Client-Token`) need a token
|
|
38
|
+
| `tdm` | the publishers' text-and-data-mining links from Crossref; Elsevier (`X-ELS-APIKey`) and Wiley (`Wiley-TDM-Client-Token`) need a token; full text for open-access and entitled articles by the front door | DOI; `AI4L_LIBRARY_ELSEVIER_API_KEY`, `AI4L_LIBRARY_WILEY_TDM_TOKEN` |
|
|
39
39
|
| `unpaywall` | locates open-access copies (publisher, repositories) and downloads them, best first | DOI |
|
|
40
40
|
| `openalex` | locates open-access copies and downloads them, second opinion; skips URLs the previous tier already tried | PMID or DOI |
|
|
41
|
-
| `core` | CORE (core.ac.uk), the aggregator of repository copies, by DOI; skipped without a key | `
|
|
41
|
+
| `core` | CORE (core.ac.uk), the aggregator of repository copies, by DOI; skipped without a key | `AI4L_LIBRARY_CORE_API_KEY` (free) |
|
|
42
|
+
| `proxy` | your own service: an institutional proxy, a licensed service, a local mirror — a URL template filled from the paper's IDs (below) | `AI4L_LIBRARY_PROXY_URL`; the IDs the template uses |
|
|
42
43
|
|
|
43
|
-
A located copy is fetched up a ladder, the way AI4L verifies URLs: a plain **fetch** (no browser) with a desktop Chrome User-Agent; then each configured **browser** rung in order (a Playwright server, or a real Google Chrome attached over the DevTools protocol, `browser/chrome.md`) (`
|
|
44
|
+
A located copy is fetched up a ladder, the way AI4L verifies URLs: a plain **fetch** (no browser) with a desktop Chrome User-Agent; then each configured **browser** rung in order (a Playwright server, or a real Google Chrome attached over the DevTools protocol, `browser/chrome.md`) (`AI4L_LIBRARY_BROWSER_1`, `_2`, `_3`: Playwright `run-server` endpoints such as `ws://127.0.0.1:8932`, typically one next to the server and one on a residential line; `browser/compose.yaml` runs one, headed under Xvfb on Microsoft's Playwright image; `"local"` uses this host's own Playwright Chromium) when the plain fetch met a 403, a bot wall or a script-rendered landing page; then the Bright Data **unlocker** when every browser was refused. No browser ships in the package and none is downloaded at run time. Nothing disguises the automation: a site that blocks automated browsers by policy stays blocked, and the attempt says so.
|
|
44
45
|
|
|
45
|
-
| *url tiers* | user-configured: a URL template filled from the paper's IDs — an institutional proxy, a licensed service, a local mirror | the IDs the template uses |
|
|
46
46
|
|
|
47
47
|
Downloads are plain HTTP with a desktop Chrome User-Agent (API calls identify the library with the contact address), PDFs first and repository copies before publisher sites; PDF (text only, no tables), JATS XML or HTML are converted to Markdown. NCBI/PMC web URLs are never fetched (PMC content comes through E-utilities). When a publisher answers a located open-access copy with a 403 or a bot wall, the page is requested once more through the [Bright Data](https://brightdata.com) Web Unlocker — the same last-resort tier AI4L's URL verification uses — if `BRIGHTDATA_API_TOKEN` is set (`BRIGHTDATA_ZONE`, default `mcp_unlocker`). Without the token the plain failure stands and is retried later.
|
|
48
48
|
|
|
49
|
-
### Configuring
|
|
49
|
+
### Configuring
|
|
50
50
|
|
|
51
|
-
|
|
51
|
+
`.env` is the whole configuration: every value the library reads is an environment variable, listed and explained in [`.env.example`](.env.example) — copy it to `.env` next to the library and fill in what you use. It is loaded at startup from the working directory (or the file `AI4L_LIBRARY_ENV` names, or `.env` in the library directory); a variable already set in the environment wins, and an unset one takes the built-in default. The library itself is `library/` under the working directory (`/library` when that exists, as in the container); keep it beside `.env`, gitignored.
|
|
52
52
|
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
"refresh_after_days": 30,
|
|
58
|
-
"verify": { "base_url": "${PERPLEXITY_URL}", "api_key": "${PERPLEXITY_API_KEY}", "model": "${PERPLEXITY_MODEL}", "threshold": 0.5, "timeout_seconds": 60 },
|
|
59
|
-
"tiers": [
|
|
60
|
-
{ "name": "inbox", "enabled": true },
|
|
61
|
-
{ "name": "pmc", "enabled": true },
|
|
62
|
-
{ "name": "europepmc", "enabled": true },
|
|
63
|
-
{ "name": "tdm", "enabled": true, "tokens": { "elsevier": "${ELSEVIER_API_KEY}", "wiley": "${WILEY_TDM_TOKEN}" } },
|
|
64
|
-
{ "name": "unpaywall", "enabled": true },
|
|
65
|
-
{ "name": "openalex", "enabled": true },
|
|
66
|
-
{ "name": "core", "enabled": true, "api_key": "${CORE_API_KEY}" },
|
|
67
|
-
{ "name": "my-proxy", "enabled": false, "url": "https://proxy.example.org/login?url=https://doi.org/{doi}",
|
|
68
|
-
"headers": { "Authorization": "Bearer ${PROXY_TOKEN}" } }
|
|
69
|
-
]
|
|
70
|
-
}
|
|
71
|
-
```
|
|
53
|
+
* **Which tiers run, in which order:** `AI4L_LIBRARY_TIERS=pmc,tdm,unpaywall,openalex` — exactly these, in this order; a tier not listed is off. Unset, every tier runs in the order of the table above, the proxy last.
|
|
54
|
+
* **Keys:** `AI4L_LIBRARY_ELSEVIER_API_KEY` and `AI4L_LIBRARY_WILEY_TDM_TOKEN` for `tdm`, `AI4L_LIBRARY_CORE_API_KEY` for `core`, `NCBI_API_KEY` (10 instead of 3 requests/s), `BRIGHTDATA_API_TOKEN` and `BRIGHTDATA_ZONE` for the unlocker. `AI4L_LIBRARY_CONTACT_EMAIL` is the one address sent to every service (NCBI `tool`/`email`, Unpaywall and OpenAlex polite pools, Crossref, the User-Agent).
|
|
55
|
+
* **Retries:** `AI4L_LIBRARY_RETRY_AFTER_DAYS` (default 1) is the cooldown before a transient tier failure is retried; `AI4L_LIBRARY_REFRESH_AFTER_DAYS` (30) the age of an entry's PubMed record after which `library_add` re-reads it for new correction notices and retractions. A tdm link behind a token that is not configured is recorded as `needs elsevier token` and not re-checked; after adding the token, remove those attempts from the entries to give them a go. `AI4L_LIBRARY_RETRY_BLOCKED` (false) decides whether a browser challenge that did not clear, or a Bright Data refusal, is final: with the same method tomorrow ends the same way, so set it true only after a change of method (a new browser rung, a new service).
|
|
56
|
+
* **Browsers and verify:** `AI4L_LIBRARY_BROWSER_1..3` and `AI4L_LIBRARY_BROWSER_HEADLESS` (above); `AI4L_LIBRARY_VERIFY_*` ("Verifying claims").
|
|
72
57
|
|
|
73
|
-
|
|
58
|
+
`library_info` reports the tiers in run order and, per variable, whether its value came from the environment, `.env` or the default — never the value.
|
|
74
59
|
|
|
75
|
-
**
|
|
60
|
+
**The proxy tier.** One service of your own — an institutional proxy, a licensed service, a local mirror — fetched at its position in the ladder like any other tier:
|
|
76
61
|
|
|
77
|
-
```
|
|
78
|
-
{
|
|
79
|
-
|
|
80
|
-
"enabled": false,
|
|
81
|
-
"description": "Example of a url tier: an institutional proxy, by DOI. Placeholders {doi} {pmid} {pmcid}; headers for credentials, ${NAME} from .env.",
|
|
82
|
-
"url": "https://proxy.example.org/login?url=https://doi.org/{doi}",
|
|
83
|
-
"headers": { "Authorization": "Bearer ${PROXY_TOKEN}" }
|
|
84
|
-
}
|
|
62
|
+
```sh
|
|
63
|
+
AI4L_LIBRARY_PROXY_URL=https://proxy.example.org/login?url=https://doi.org/{doi}
|
|
64
|
+
AI4L_LIBRARY_PROXY_HEADERS=Authorization: Bearer <token>
|
|
85
65
|
```
|
|
86
66
|
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
*
|
|
90
|
-
* `headers` — optional request headers, for credentials (`Authorization`, an API-key header, a cookie).
|
|
91
|
-
* `${NAME}` in `url` or a header value is the environment variable `NAME` (from `.env`), so the tracked file never holds a token; an unset variable resolves to an empty string.
|
|
92
|
-
* `enabled` — `false` keeps the entry but skips it; `description` — free text shown by `library_info`.
|
|
67
|
+
* The URL is a template. Placeholders: `{doi}` (`10.1056/NEJMoa042274`), `{pmid}` (`15516696`), `{pmcid}` (`PMC6728809`, with its prefix). Each is URL-encoded when filled — the slash in a DOI becomes `%2F` — so put a placeholder in a path segment or a query value. A template may use several. The tier applies only to a paper for which every placeholder it uses is known; otherwise it is skipped as `no doi` (or `no pmcid`, …), a permanent reason. A template without any placeholder never runs (`template has no placeholder`).
|
|
68
|
+
* The headers are optional request headers for credentials (`Authorization`, an API-key header, a cookie), as `Name: value; Name: value`.
|
|
69
|
+
* Unset URL: no proxy tier. `AI4L_LIBRARY_TIERS` places `proxy` in the order like any built-in tier.
|
|
93
70
|
|
|
94
|
-
**What the
|
|
71
|
+
**What the proxy tier expects back.** The tier sends one GET to the filled template, with a desktop Chrome User-Agent and the headers; redirects are followed. The service answers the paper as one of two formats, read from the response bytes, not the URL or the Content-Type:
|
|
95
72
|
|
|
96
73
|
* **PDF** — the body starts with `%PDF-`. It needs a text layer: under 500 characters per page counts as a scan (`scanned pdf`).
|
|
97
74
|
* **XML** — JATS (as PMC delivers it) or Elsevier's full-text XML, with a `<body>`; recognised by `<?xml` or `<article` at the start.
|
|
@@ -100,17 +77,17 @@ Any other body — an HTML page, JSON, plain text, an empty response — is the
|
|
|
100
77
|
|
|
101
78
|
| response | recorded as | retried |
|
|
102
79
|
|---|---|---|
|
|
103
|
-
| 403, 429, 5xx, timeout, network error | `HTTP 403`, `HTTP 429`, `HTTP 503`, `timeout`, … | after `
|
|
104
|
-
| 404 or another 4xx | `HTTP 404`, … — with the service's message when its body carries one: `HTTP 404 — 404 Not Found GET /10.1056%2FNEJMoa042274` | after `
|
|
105
|
-
| 200, but neither PDF nor XML | `service: <message>` | after `
|
|
80
|
+
| 403, 429, 5xx, timeout, network error | `HTTP 403`, `HTTP 429`, `HTTP 503`, `timeout`, … | after the cooldown (`AI4L_LIBRARY_RETRY_AFTER_DAYS`) |
|
|
81
|
+
| 404 or another 4xx | `HTTP 404`, … — with the service's message when its body carries one: `HTTP 404 — 404 Not Found GET /10.1056%2FNEJMoa042274` | after the cooldown (`AI4L_LIBRARY_RETRY_AFTER_DAYS`) |
|
|
82
|
+
| 200, but neither PDF nor XML | `service: <message>` | after the cooldown (`AI4L_LIBRARY_RETRY_AFTER_DAYS`) |
|
|
106
83
|
| XML without a body | `xml without body` | never |
|
|
107
84
|
| another document (title and DOI absent) | `neither title nor DOI in text (…)` | never |
|
|
108
85
|
| no article body | `no article body (pdf, 342 words, …)` | never |
|
|
109
86
|
| a PDF without a text layer | `scanned pdf` | never |
|
|
110
87
|
|
|
111
|
-
The
|
|
88
|
+
The proxy tier is not escalated to the browser rungs or Bright Data: it is the service's own answer that counts. NCBI and PMC web URLs are never fetched (the `pmc` tier covers them), and a URL already tried for the paper in the same cascade run is skipped.
|
|
112
89
|
|
|
113
|
-
Every full text is checked before it counts: title present, methods and results present (or a substantial sectioned body), not an error, login, paywall or bot-wall page. The original (XML, HTML or PDF) is stored next to the Markdown. A failed tier is recorded in the entry's `attempts` with reason and date, e.g. `unpaywall (HTTP 403: nature.com) 2026-10-26`. **Retries:** a permanent reason (no PMCID, no body, closed, no open-access copy, content checks failed) is never retried; a transient one (timeout, network error, HTTP 403/429/5xx, wall page) is retried by the next `library_add` once `
|
|
90
|
+
Every full text is checked before it counts: title present, methods and results present (or a substantial sectioned body), not an error, login, paywall or bot-wall page. The original (XML, HTML or PDF) is stored next to the Markdown. A failed tier is recorded in the entry's `attempts` with reason and date, e.g. `unpaywall (HTTP 403: nature.com) 2026-10-26`. **Retries:** a permanent reason (no PMCID, no body, closed, no open-access copy, content checks failed) is never retried; a transient one (timeout, network error, HTTP 403/429/5xx, wall page) is retried by the next `library_add` once `AI4L_LIBRARY_RETRY_AFTER_DAYS` (default 1) have passed, replacing the old entry. A `full_text` entry is never touched again. A failed PubMed fetch writes no entry at all, so it is simply retried next time.
|
|
114
91
|
|
|
115
92
|
### Verifying claims
|
|
116
93
|
|
|
@@ -132,10 +109,10 @@ Every full text is checked before it counts: title present, methods and results
|
|
|
132
109
|
|
|
133
110
|
| verdict | meaning |
|
|
134
111
|
|---|---|
|
|
135
|
-
| `pass` | the paper states the finding in substance: `score` reached
|
|
112
|
+
| `pass` | the paper states the finding in substance: `score` reached the threshold (default 0.5) |
|
|
136
113
|
| `fail` | the paper's full text was read and does not state it |
|
|
137
114
|
| `na` | only the paper's abstract is in the library, and that does not state it — the finding may be in the full text (drop it into the inbox and verify again); or only its title is there (PubMed has no abstract for it, no full text was found), in which case nothing is judged and `score` is null |
|
|
138
|
-
| `error` | the call could not run: a refused claim (empty, too short, a fragment, naming a source, compound), paper not in the library, no model configured, the model failing or not answering within
|
|
115
|
+
| `error` | the call could not run: a refused claim (empty, too short, a fragment, naming a source, compound), paper not in the library, no model configured, the model failing or not answering within the timeout |
|
|
139
116
|
|
|
140
117
|
"In substance": wording may differ, a number may be rounded or derived from the paper's numbers; a changed number, an opposite direction, another population or intervention is not backed. A paper is read whole; nothing of it enters the caller's context, and the model returns one probability, no quotes and no prose. For a look at the text behind a verdict there is `library_read`; on the hosted model real findings score 0.95 and up and changed numbers 0.20 and under, so a `pass` under 0.7 or a `fail` over 0.3 is the one worth that look.
|
|
141
118
|
|
|
@@ -148,25 +125,25 @@ The verdict comes from [Perplexity's decision model](https://docs.perplexity.ai/
|
|
|
148
125
|
* **Hosted** at `api.perplexity.ai`: model `pplx-decider-v1.2`, a whole paper in about two seconds, 2 cents per million input tokens, needs a key from [console.perplexity.ai](https://console.perplexity.ai). The paper text leaves the machine.
|
|
149
126
|
* **Local**, the [Decider for Mac](https://docs.perplexity.ai/docs/decisions/decider-for-mac) app on a Mac in the network: model `pplx-decider-flash-v1.2`, no key, text stays local; 32k-token requests, so long papers are judged in windows cut at headings. Use the Mac's IP in the URL, not a host name, with "Allow Network Access" on, and leave its `max_tokens` at 32768 (it serves one request at a time and a larger request blocks it for minutes).
|
|
150
127
|
|
|
151
|
-
|
|
128
|
+
Hosted or local is three lines in `.env`: `AI4L_LIBRARY_VERIFY_URL`, `AI4L_LIBRARY_VERIFY_MODEL`, `AI4L_LIBRARY_VERIFY_API_KEY`. `AI4L_LIBRARY_VERIFY_THRESHOLD` (0.5) is the probability at or above which a paper counts as backing the claim; `AI4L_LIBRARY_VERIFY_TIMEOUT_SECS` (60) caps one model request. A paper longer than 800000 characters hosted (a whole paper), 90000 on the Mac app, is judged in windows. Without the URL the tool answers `error`. Self-test of an endpoint, the app's own example:
|
|
152
129
|
|
|
153
130
|
```sh
|
|
154
131
|
curl -s http://<ip>:8080/v1/decisions -H 'Content-Type: application/json' -d '{"model":"pplx-decider-flash-v1.2","state":"I was charged twice for my order.","questions":{"team":{"type":"choice","criteria":{"billing":null,"shipping":null,"technical":null}}}}'
|
|
155
132
|
```
|
|
156
133
|
|
|
157
|
-
**Papers without a PMID.** A review may cite a paper PubMed does not index by its DOI (`https://doi.org/<DOI>`); `library_add("doi:10.5812/asjsm.110753")` takes it. If PubMed does index the DOI, the call builds that paper's `pmid:` entry and stores a copy of it under `doi:` — the same files, renamed, with `pmid` set in its meta — so the DOI resolves on disk: every later call answers from the `doi:` entry without asking PubMed (5 of the 101 DOIs evipedia cites). Each `library_add` of such a DOI completes the `pmid:` entry (retries, record refresh) and copies it again, so the copy is as current as the original. A mapped paper counts once under `pmid` and once under `doi` in `library_info`. Otherwise the record comes from [Crossref](https://www.crossref.org/) — or, for a DOI another registry holds (DataCite, mEDRA, …), from doi.org's content negotiation as CSL JSON, stored as `-csl.json`: title, authors, journal, year, the abstract — Crossref's when the publisher deposited one, else OpenAlex's (of the 97 DOI-only papers evipedia cites, Crossref has 55 abstracts, OpenAlex 86; 11 have none anywhere and stay title-only unless a full text is found) — the funders as `funding`, and Crossref's update notices — corrections, retractions, expressions of concern — as `corrections`, so `citable` works as for a PubMed paper. The status has `pmid` null, the abstract's head line says "not in PubMed", `retrieved_by` names the source of what the entry holds — `crossref`, `doi.org` or `openalex` for the abstract, `crossref` or `doi.org` for a title alone, the tier once a full text is found; the full-text cascade then runs on the DOI (`unpaywall`, `openalex`, `tdm`, `core`,
|
|
134
|
+
**Papers without a PMID.** A review may cite a paper PubMed does not index by its DOI (`https://doi.org/<DOI>`); `library_add("doi:10.5812/asjsm.110753")` takes it. If PubMed does index the DOI, the call builds that paper's `pmid:` entry and stores a copy of it under `doi:` — the same files, renamed, with `pmid` set in its meta — so the DOI resolves on disk: every later call answers from the `doi:` entry without asking PubMed (5 of the 101 DOIs evipedia cites). Each `library_add` of such a DOI completes the `pmid:` entry (retries, record refresh) and copies it again, so the copy is as current as the original. A mapped paper counts once under `pmid` and once under `doi` in `library_info`. Otherwise the record comes from [Crossref](https://www.crossref.org/) — or, for a DOI another registry holds (DataCite, mEDRA, …), from doi.org's content negotiation as CSL JSON, stored as `-csl.json`: title, authors, journal, year, the abstract — Crossref's when the publisher deposited one, else OpenAlex's (of the 97 DOI-only papers evipedia cites, Crossref has 55 abstracts, OpenAlex 86; 11 have none anywhere and stay title-only unless a full text is found) — the funders as `funding`, and Crossref's update notices — corrections, retractions, expressions of concern — as `corrections`, so `citable` works as for a PubMed paper. The status has `pmid` null, the abstract's head line says "not in PubMed", `retrieved_by` names the source of what the entry holds — `crossref`, `doi.org` or `openalex` for the abstract, `crossref` or `doi.org` for a title alone, the tier once a full text is found; the full-text cascade then runs on the DOI (`unpaywall`, `openalex`, `tdm`, `core`, `proxy`; `pmc`, `europepmc` and `inbox` need a PMID). An entry older than `AI4L_LIBRARY_REFRESH_AFTER_DAYS` gets its Crossref record re-read for new notices. Entries live under `doi/<hash 2>/<hash 4>/<slug>/`, the slug being the DOI in lowercase with every run of other characters than letters, digits, `.` and `-` as `_` (`doi-10.5812_asjsm.110753`), the Crossref record as `-crossref.json`. `library_status`, `library_read` and `library_verify` take any DOI `library_add` took: they read its `doi:` entry, the copy or the Crossref one, and never look a DOI up. Measured on evipedia's 915 live reviews (2026-11-06): 87 cite 101 papers by DOI only, against 34,378 PubMed citations.
|
|
158
135
|
|
|
159
136
|
PubMed's book records — National Academies reports, NTP monographs, StatPearls chapters in NCBI Bookshelf — are read as papers too: the chapter or book title, the authors (else the editors), the publisher in place of the journal, publication type `Book`.
|
|
160
137
|
|
|
161
|
-
**Drug labels.** `library_add("dailymed:<set ID>")` stores the current FDA prescribing information of one product from [DailyMed](https://dailymed.nlm.nih.gov/), the set ID being the fixed identifier in the label's page URL (`…/drugInfo.cfm?setid=<set ID>`) across all its revisions — the same ID a review links. The SPL XML is converted to Markdown: first a **Highlights** section with the label's header lines — the product line and "Initial U.S. Approval: 2003" (the year is also in the head line) — then every text section in order, a boxed warning headed "Boxed Warning" and the recent major changes headed as such even where the label gives them no title, (indications, dosage, contraindications, warnings, adverse reactions with their tables, drug interactions, clinical pharmacology, clinical studies); the structured product data and package images are left out; there is no abstract. The status carries `setid`, the title as `<brand> (<generic>) <form> — prescribing information`, the labeler as `first_author`, the effective year as `year`, the label version as `version` (`spl 52`), and `publication_types` `["Drug Label"]`. `library_read` and `library_verify` work on a label as on a paper, so a figure the review takes from a label is verified against the label. An entry older than `
|
|
138
|
+
**Drug labels.** `library_add("dailymed:<set ID>")` stores the current FDA prescribing information of one product from [DailyMed](https://dailymed.nlm.nih.gov/), the set ID being the fixed identifier in the label's page URL (`…/drugInfo.cfm?setid=<set ID>`) across all its revisions — the same ID a review links. The SPL XML is converted to Markdown: first a **Highlights** section with the label's header lines — the product line and "Initial U.S. Approval: 2003" (the year is also in the head line) — then every text section in order, a boxed warning headed "Boxed Warning" and the recent major changes headed as such even where the label gives them no title, (indications, dosage, contraindications, warnings, adverse reactions with their tables, drug interactions, clinical pharmacology, clinical studies); the structured product data and package images are left out; there is no abstract. The status carries `setid`, the title as `<brand> (<generic>) <form> — prescribing information`, the labeler as `first_author`, the effective year as `year`, the label version as `version` (`spl 52`), and `publication_types` `["Drug Label"]`. `library_read` and `library_verify` work on a label as on a paper, so a figure the review takes from a label is verified against the label. An entry older than `AI4L_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when DailyMed has a newer version. Entries live under `dailymed/<first 2>/<first 4>/<set ID>/`, with the SPL XML and the Markdown as `dailymed-<set ID>-fulltext.xml` and `-fulltext.md`, and the log line reads `add dailymed:<set ID> dailymed fetched level=full_text spl 52 effective=2026-06-03 Cialis (Tadalafil)`.
|
|
162
139
|
|
|
163
|
-
**Clinical trial records.** `library_add("nct:05173896")` stores the current record of one trial from [ClinicalTrials.gov](https://clinicaltrials.gov/): the ID is the eight digits of the NCT number in the review's link (`…/study/NCT05173896`); `nct:NCT05173896` and a bare `NCT05173896` are accepted and read the same. The API's JSON is converted to Markdown, one section per module: identification with secondary IDs such as EudraCT, status with every date and its actual/estimated type, sponsor and collaborators, description, conditions, design (phase, allocation, masking, enrollment with its type), arms and interventions with doses, primary and secondary outcomes with time frames, eligibility, locations as a country list, references; and, when results are posted, the participant flow, baseline characteristics, every outcome measure with its group values and analyses, and the adverse events, all as tables. Values are written in plain words, "Phase 2", "Active, not recruiting", "100 (estimated)", so a figure the review states can be found as stated. The status carries `nctid`, `publications` (the results publications the record lists, as `pmid:…`, so whether a trial has been published is one status away), the brief title, the lead sponsor as `first_author`, the first-posted year as `year`, the last-update-posted date as `version`, and `publication_types` `["Trial Registration"]`. `library_read` and `library_verify` work on a record as on a paper. An entry older than `
|
|
140
|
+
**Clinical trial records.** `library_add("nct:05173896")` stores the current record of one trial from [ClinicalTrials.gov](https://clinicaltrials.gov/): the ID is the eight digits of the NCT number in the review's link (`…/study/NCT05173896`); `nct:NCT05173896` and a bare `NCT05173896` are accepted and read the same. The API's JSON is converted to Markdown, one section per module: identification with secondary IDs such as EudraCT, status with every date and its actual/estimated type, sponsor and collaborators, description, conditions, design (phase, allocation, masking, enrollment with its type), arms and interventions with doses, primary and secondary outcomes with time frames, eligibility, locations as a country list, references; and, when results are posted, the participant flow, baseline characteristics, every outcome measure with its group values and analyses, and the adverse events, all as tables. Values are written in plain words, "Phase 2", "Active, not recruiting", "100 (estimated)", so a figure the review states can be found as stated. The status carries `nctid`, `publications` (the results publications the record lists, as `pmid:…`, so whether a trial has been published is one status away), the brief title, the lead sponsor as `first_author`, the first-posted year as `year`, the last-update-posted date as `version`, and `publication_types` `["Trial Registration"]`. `library_read` and `library_verify` work on a record as on a paper. An entry older than `AI4L_LIBRARY_REFRESH_AFTER_DAYS` is re-fetched and rewritten when the record was updated since; a status the review states that has moved on then fails, which is right. Entries live under `nct/<digits 1–2>/<digits 1–4>/<digits>/` with the JSON and the Markdown as `nct-<digits>-fulltext.json` and `-fulltext.md`; the log line reads `add nct:05173896 ctgov fetched level=full_text updated 2024-12-09 active, not recruiting ETLAS2`.
|
|
164
141
|
|
|
165
142
|
**Funding and conflicts of interest.** Every status carries `coi`, the paper's conflict-of-interest statement, and `funding`, who paid. Both come from PubMed's record at ingest (`CoiStatement`, the grant list's agencies; a quarter of records have each) and are refreshed with it; where PubMed has none and a full text was retrieved, they are taken from the paper's own text — its Declarations section, or a section headed Funding, Conflicts of interest, Competing interests or Disclosure — as the statement verbatim. `coi` null and `funding` empty therefore mean: neither PubMed nor the stored text states anything. For an abstract-only entry that is as far as the library can see. The full statements of a paper with full text are one call away: `library_read(id, section: "Declarations")`. In `-meta.md` they are the lines `coi:` (a JSON string) and `funding:` (a JSON array, one item per agency or statement).
|
|
166
143
|
|
|
167
144
|
**Symbols a PDF font hides.** Some publishers' symbol fonts carry no Unicode for their glyphs, so text extraction gives a control character where the PDF shows a minus, ±, ≤ or β. The library restores the symbol from the glyph's name in the font — the names of the Adobe Pi fonts (`H11350` is ≥ everywhere) and of the Advent symbol font most Elsevier and Wiley PDFs use — each name checked against the glyph's outline in the library's PDFs; a name it does not know is left as it is and counted: the status's `glyphs_unresolved` is the number left in the full text (162 of 1,119 PDF-derived entries have some, down from 277), and `library_verify` returns the same count for the text it judged. A `fail` on a paper with `glyphs_unresolved` above 0 says "check the symbol", not "the paper does not state it". Words a PDF breaks at a line end ("differ- ence") are joined back when the paper uses the word elsewhere or no compound is known at that break; a hyphen the paper uses elsewhere ("time-dependent") and a suspended one ("dose- and time-dependent") are kept.
|
|
168
145
|
|
|
169
|
-
**Citable, and correction notices.** `citable` is the one flag an auditor checks before a paper counts as evidence: false when PubMed records the paper as retracted — by the "Retracted Publication" type or by a retraction notice — and, once correction notices are assessed (below), when a correction changed its data. `comment` says why, or, when the paper is citable, names what to be aware of: an erratum (its severity is not assessed yet; `library_add` on the notice's PMID fetches it when it has one), an expression of concern, or a newer version that supersedes this one, typically an updated Cochrane review, which should be cited instead. Behind the flag, `corrections` lists PubMed's notices about the paper as `{type, pmid, source}` — `erratum_in`, `retraction_in`, `partial_retraction_in`, `retracted_and_republished_in`, `expression_of_concern_in` or `update_in`, the notice's own PMID when PubMed has one, and its citation verbatim. Comments are not notices and are not carried: PubMed's CommentIn, SummaryForPatientsIn, ReprintIn and the "Of/For/From" direction (this paper being the notice about another) are ignored. `update_in` is not an erratum: every case seen is a Cochrane review superseded by a newer version. Notices appear after ingest, so `library_add` on an existing entry re-reads the PubMed record once it is older than `
|
|
146
|
+
**Citable, and correction notices.** `citable` is the one flag an auditor checks before a paper counts as evidence: false when PubMed records the paper as retracted — by the "Retracted Publication" type or by a retraction notice — and, once correction notices are assessed (below), when a correction changed its data. `comment` says why, or, when the paper is citable, names what to be aware of: an erratum (its severity is not assessed yet; `library_add` on the notice's PMID fetches it when it has one), an expression of concern, or a newer version that supersedes this one, typically an updated Cochrane review, which should be cited instead. Behind the flag, `corrections` lists PubMed's notices about the paper as `{type, pmid, source}` — `erratum_in`, `retraction_in`, `partial_retraction_in`, `retracted_and_republished_in`, `expression_of_concern_in` or `update_in`, the notice's own PMID when PubMed has one, and its citation verbatim. Comments are not notices and are not carried: PubMed's CommentIn, SummaryForPatientsIn, ReprintIn and the "Of/For/From" direction (this paper being the notice about another) are ignored. `update_in` is not an erratum: every case seen is a Cochrane review superseded by a newer version. Notices appear after ingest, so `library_add` on an existing entry re-reads the PubMed record once it is older than `AI4L_LIBRARY_REFRESH_AFTER_DAYS` (default 30; `pubmed_checked` in the entry is the date of the last read) and updates the notices, retraction, publication types and a PMCID that appeared since — never the stored abstract or full text, and without reopening full-text retries. The notice's own text is not fetched or applied. Where NLM has corrected the PubMed abstract after an erratum — it marks each corrected sentence `[corrected]` — those sentences are copied to the end of the stored full text under `## Corrected in PubMed abstract`, at ingest and on every refresh, so a full text that predates the correction still carries the corrected figures and `library_verify` passes a claim that states them — and marks a claim that states a replaced figure as `superseded`. The `comment` then reads "erratum, corrected figures stored" instead of "not assessed". In `-meta.md` each notice is one `corrections:` line, `<type> <pmid or -> "<source>"`; entries written before this feature are read from their stored PubMed XML until they are next rewritten.
|
|
170
147
|
|
|
171
148
|
**Why a notice may have no PMID.** PubMed gives the notice's PMID only when the notice is itself a PubMed record. For about a quarter of the errata in a 6,900-entry library it is not: the journal printed the correction, PubMed recorded the citation in the paper's record, and never indexed the correction as a record of its own (NEJM, JAMA and Lancet errata from 1988 to 2014 are typical). There is nothing to redirect to, so `pmid` is null and `source` — the citation as PubMed wrote it — is the notice's only identifier. Measured on the 79 such errata in the library (2026-11-04): PubMed's citation matcher resolves 18 after all (13 directly, 5 after picking the "Published Erratum" among two candidates), Crossref gives a DOI for 38 more when journal, volume and page match, and 23 have no identifier anywhere. The planned second stage resolves them at ingest (PMID first, then DOI, the citation always kept), fetches the notice's text and classifies it once per paper as major (a number, table, figure, result or conclusion changed) or minor (authors, funding, typos): a major erratum makes the paper not citable, like a retraction, so the audit never has to apply a correction; until then an erratum is a note on a citable paper.
|
|
172
149
|
|
package/dist/index.js
CHANGED
|
@@ -10,6 +10,7 @@ function dotenvFile(env = process.env) {
|
|
|
10
10
|
const lib = env.AI4L_LIBRARY_DIR?.trim() || (existsSync("/library") ? "/library" : "library");
|
|
11
11
|
return join(resolve(lib), ".env");
|
|
12
12
|
}
|
|
13
|
+
var loadedFromDotenv = /* @__PURE__ */ new Set();
|
|
13
14
|
function loadDotenv(file = dotenvFile()) {
|
|
14
15
|
if (!existsSync(file)) return [];
|
|
15
16
|
const loaded = [];
|
|
@@ -24,81 +25,46 @@ function loadDotenv(file = dotenvFile()) {
|
|
|
24
25
|
if (process.env[m[1]] === void 0) {
|
|
25
26
|
process.env[m[1]] = v;
|
|
26
27
|
loaded.push(m[1]);
|
|
28
|
+
loadedFromDotenv.add(m[1]);
|
|
27
29
|
}
|
|
28
30
|
}
|
|
29
31
|
return loaded;
|
|
30
32
|
}
|
|
33
|
+
function sourceOf(name, env = process.env) {
|
|
34
|
+
if (!env[name]?.trim()) return "default";
|
|
35
|
+
return loadedFromDotenv.has(name) ? ".env" : "environment";
|
|
36
|
+
}
|
|
37
|
+
var str = (name, env = process.env) => env[name]?.trim() ?? "";
|
|
38
|
+
function num(name, def, env = process.env, ok = (n) => n >= 0) {
|
|
39
|
+
const v = str(name, env);
|
|
40
|
+
if (!v) return def;
|
|
41
|
+
const n = Number(v);
|
|
42
|
+
return Number.isFinite(n) && ok(n) ? n : def;
|
|
43
|
+
}
|
|
44
|
+
function bool(name, def, env = process.env) {
|
|
45
|
+
const v = str(name, env).toLowerCase();
|
|
46
|
+
return ["true", "1", "yes", "on"].includes(v) ? true : ["false", "0", "no", "off"].includes(v) ? false : def;
|
|
47
|
+
}
|
|
31
48
|
|
|
32
49
|
// src/index.ts
|
|
33
50
|
import { McpServer } from "@modelcontextprotocol/sdk/server/mcp.js";
|
|
34
51
|
import { StdioServerTransport } from "@modelcontextprotocol/sdk/server/stdio.js";
|
|
35
52
|
|
|
36
53
|
// src/tools.ts
|
|
37
|
-
import { existsSync as
|
|
54
|
+
import { existsSync as existsSync8, readFileSync as readFileSync7 } from "fs";
|
|
38
55
|
import { z } from "zod";
|
|
39
56
|
|
|
40
57
|
// src/entry.ts
|
|
41
|
-
import { copyFileSync, existsSync as
|
|
42
|
-
import { join as
|
|
58
|
+
import { copyFileSync, existsSync as existsSync2, mkdirSync, readdirSync, readFileSync as readFileSync2, renameSync, rmSync, writeFileSync } from "fs";
|
|
59
|
+
import { join as join2 } from "path";
|
|
43
60
|
import { basename, dirname } from "path";
|
|
44
61
|
|
|
45
62
|
// src/pubmed.ts
|
|
46
63
|
import { XMLParser } from "fast-xml-parser";
|
|
47
64
|
|
|
48
|
-
// src/contact.ts
|
|
49
|
-
import { existsSync as existsSync3, readFileSync as readFileSync2 } from "fs";
|
|
50
|
-
import { join as join3 } from "path";
|
|
51
|
-
|
|
52
|
-
// src/layout.ts
|
|
53
|
-
import { createHash } from "crypto";
|
|
54
|
-
import { existsSync as existsSync2 } from "fs";
|
|
55
|
-
import { join as join2, resolve as resolve2 } from "path";
|
|
56
|
-
var AI4L_LIBRARY_DIR = (() => {
|
|
57
|
-
const env = process.env.AI4L_LIBRARY_DIR;
|
|
58
|
-
if (env && env.trim()) return resolve2(env.trim());
|
|
59
|
-
if (existsSync2("/library")) return "/library";
|
|
60
|
-
return resolve2("library");
|
|
61
|
-
})();
|
|
62
|
-
var doiSlug = (doi) => doi.toLowerCase().replace(/[^a-z0-9.-]+/g, "_");
|
|
63
|
-
function entryDirRel(type, num) {
|
|
64
|
-
if (type === "doi") {
|
|
65
|
-
const h = createHash("sha1").update(num).digest("hex");
|
|
66
|
-
return `${type}/${h.slice(0, 2)}/${h.slice(0, 4)}/${num}`;
|
|
67
|
-
}
|
|
68
|
-
if (!/^\d+$/.test(num)) return `${type}/${num.slice(0, 2)}/${num.slice(0, 4)}/${num}`;
|
|
69
|
-
const l1 = num.length > 6 ? num.slice(0, -6) : "0";
|
|
70
|
-
const l2 = num.length > 4 ? num.slice(0, -4) : "0";
|
|
71
|
-
return `${type}/${l1}/${l2}/${num}`;
|
|
72
|
-
}
|
|
73
|
-
function entryPaths(libraryDir, type, num) {
|
|
74
|
-
if (type === "doi") num = doiSlug(num);
|
|
75
|
-
const entry = `${type}-${num}`;
|
|
76
|
-
const entryDir = join2(libraryDir, entryDirRel(type, num));
|
|
77
|
-
return {
|
|
78
|
-
entry,
|
|
79
|
-
entryDir,
|
|
80
|
-
xml: join2(entryDir, `${entry}-abstract.xml`),
|
|
81
|
-
md: join2(entryDir, `${entry}-abstract.md`),
|
|
82
|
-
meta: join2(entryDir, `${entry}-meta.md`),
|
|
83
|
-
fulltextMd: join2(entryDir, `${entry}-fulltext.md`),
|
|
84
|
-
fulltextRaw: (format) => join2(entryDir, `${entry}-fulltext.${format}`)
|
|
85
|
-
};
|
|
86
|
-
}
|
|
87
|
-
|
|
88
65
|
// src/contact.ts
|
|
89
66
|
var DEFAULT = "hello@forever-healthy.org";
|
|
90
|
-
|
|
91
|
-
const file = process.env.AI4L_LIBRARY_CONFIG?.trim() || join3(AI4L_LIBRARY_DIR, "config.json");
|
|
92
|
-
let v = "";
|
|
93
|
-
if (existsSync3(file)) {
|
|
94
|
-
try {
|
|
95
|
-
v = String(JSON.parse(readFileSync2(file, "utf8")).contact_email ?? "");
|
|
96
|
-
} catch {
|
|
97
|
-
}
|
|
98
|
-
}
|
|
99
|
-
v = v.replace(/\$\{([A-Za-z_][A-Za-z0-9_]*)\}/g, (_, n) => process.env[n] ?? "").trim();
|
|
100
|
-
return v || process.env.CONTACT_EMAIL?.trim() || DEFAULT;
|
|
101
|
-
}
|
|
67
|
+
var contactEmail = (env = process.env) => str("AI4L_LIBRARY_CONTACT_EMAIL", env) || DEFAULT;
|
|
102
68
|
|
|
103
69
|
// src/http.ts
|
|
104
70
|
var TIMEOUT_MS = 3e4;
|
|
@@ -1047,7 +1013,7 @@ ${body}
|
|
|
1047
1013
|
}
|
|
1048
1014
|
function readHeadLine(p) {
|
|
1049
1015
|
try {
|
|
1050
|
-
const lines =
|
|
1016
|
+
const lines = readFileSync2(p.md, "utf8").split("\n").filter((l) => l.trim());
|
|
1051
1017
|
return lines[1] ?? "";
|
|
1052
1018
|
} catch {
|
|
1053
1019
|
return "";
|
|
@@ -1169,28 +1135,28 @@ function splitPublicationTypes(v) {
|
|
|
1169
1135
|
}
|
|
1170
1136
|
function abstractMissing(p) {
|
|
1171
1137
|
try {
|
|
1172
|
-
return
|
|
1138
|
+
return readFileSync2(p.md, "utf8").includes("_No abstract available._");
|
|
1173
1139
|
} catch {
|
|
1174
1140
|
return false;
|
|
1175
1141
|
}
|
|
1176
1142
|
}
|
|
1177
1143
|
function correctionsFromXml(p) {
|
|
1178
1144
|
try {
|
|
1179
|
-
return parsePubmedXml(
|
|
1145
|
+
return parsePubmedXml(readFileSync2(p.xml, "utf8"))?.corrections ?? [];
|
|
1180
1146
|
} catch {
|
|
1181
1147
|
return [];
|
|
1182
1148
|
}
|
|
1183
1149
|
}
|
|
1184
1150
|
function declarationsFromXml(p) {
|
|
1185
1151
|
try {
|
|
1186
|
-
const r = parsePubmedXml(
|
|
1152
|
+
const r = parsePubmedXml(readFileSync2(p.xml, "utf8"));
|
|
1187
1153
|
return { coi: r?.coi ?? null, funding: r?.funding ?? [] };
|
|
1188
1154
|
} catch {
|
|
1189
1155
|
return { coi: null, funding: [] };
|
|
1190
1156
|
}
|
|
1191
1157
|
}
|
|
1192
1158
|
function readMeta(p) {
|
|
1193
|
-
const { fm, all } = parseFrontmatter(
|
|
1159
|
+
const { fm, all } = parseFrontmatter(readFileSync2(p.meta, "utf8"));
|
|
1194
1160
|
let title = fm.title ?? "";
|
|
1195
1161
|
if (title.startsWith('"')) {
|
|
1196
1162
|
try {
|
|
@@ -1251,12 +1217,12 @@ function copyEntry(from, to) {
|
|
|
1251
1217
|
rmSync(tmp, { recursive: true, force: true });
|
|
1252
1218
|
mkdirSync(tmp, { recursive: true });
|
|
1253
1219
|
const rename = (n) => n.startsWith(from.entry) ? to.entry + n.slice(from.entry.length) : n;
|
|
1254
|
-
for (const n of readdirSync(from.entryDir)) if (
|
|
1220
|
+
for (const n of readdirSync(from.entryDir)) if (join2(from.entryDir, n) !== from.meta && !n.endsWith(".tmp")) copyFileSync(join2(from.entryDir, n), join2(tmp, rename(n)));
|
|
1255
1221
|
const m = readMeta(from);
|
|
1256
1222
|
const copy = { ...m, files: m.files.map(rename) };
|
|
1257
|
-
writeFileSync(
|
|
1223
|
+
writeFileSync(join2(tmp, basename(to.meta)), renderMeta(copy));
|
|
1258
1224
|
mkdirSync(dirname(to.entryDir), { recursive: true });
|
|
1259
|
-
if (
|
|
1225
|
+
if (existsSync2(to.entryDir)) renameSync(to.entryDir, old);
|
|
1260
1226
|
renameSync(tmp, to.entryDir);
|
|
1261
1227
|
rmSync(old, { recursive: true, force: true });
|
|
1262
1228
|
return copy;
|
|
@@ -1276,7 +1242,7 @@ function citability(m) {
|
|
|
1276
1242
|
return { citable: true, comment: notes.length ? notes.join("; ") : null };
|
|
1277
1243
|
}
|
|
1278
1244
|
function statusFromMeta(p, m, base) {
|
|
1279
|
-
const full =
|
|
1245
|
+
const full = existsSync2(p.fulltextMd) ? readFileSync2(p.fulltextMd, "utf8") : null;
|
|
1280
1246
|
const cit = citability(m);
|
|
1281
1247
|
if (full?.includes(`
|
|
1282
1248
|
${CORRECTED_HEADING}
|
|
@@ -1318,7 +1284,7 @@ function readStatus(p, base) {
|
|
|
1318
1284
|
return statusFromMeta(p, readMeta(p), base);
|
|
1319
1285
|
}
|
|
1320
1286
|
function entryExists(p) {
|
|
1321
|
-
return
|
|
1287
|
+
return existsSync2(p.meta);
|
|
1322
1288
|
}
|
|
1323
1289
|
function writeEntry(p, r, xml, converter) {
|
|
1324
1290
|
mkdirSync(p.entryDir, { recursive: true });
|
|
@@ -1401,15 +1367,15 @@ ${sentences.join("\n\n")}
|
|
|
1401
1367
|
`;
|
|
1402
1368
|
}
|
|
1403
1369
|
function syncCorrectedBlock(p) {
|
|
1404
|
-
if (!
|
|
1370
|
+
if (!existsSync2(p.fulltextMd) || !existsSync2(p.xml)) return;
|
|
1405
1371
|
let r;
|
|
1406
1372
|
try {
|
|
1407
|
-
r = parsePubmedXml(
|
|
1373
|
+
r = parsePubmedXml(readFileSync2(p.xml, "utf8"));
|
|
1408
1374
|
} catch {
|
|
1409
1375
|
return;
|
|
1410
1376
|
}
|
|
1411
1377
|
if (!r) return;
|
|
1412
|
-
const md =
|
|
1378
|
+
const md = readFileSync2(p.fulltextMd, "utf8");
|
|
1413
1379
|
const next = withCorrectedBlock(md, correctedSentences(r));
|
|
1414
1380
|
if (next !== md) writeAtomic(p.fulltextMd, next);
|
|
1415
1381
|
}
|
|
@@ -1518,10 +1484,6 @@ function writeDoiEntry(p, r, json2, converter, abstractFrom = "crossref", record
|
|
|
1518
1484
|
return meta;
|
|
1519
1485
|
}
|
|
1520
1486
|
|
|
1521
|
-
// src/config.ts
|
|
1522
|
-
import { existsSync as existsSync7, readFileSync as readFileSync6 } from "fs";
|
|
1523
|
-
import { join as join7 } from "path";
|
|
1524
|
-
|
|
1525
1487
|
// src/checks.ts
|
|
1526
1488
|
var GREEK = { "\u03B1": "alpha", "\u03B2": "beta", "\xDF": "beta", "\u03B3": "gamma", "\u03B4": "delta", "\u03B5": "epsilon", "\u03B6": "zeta", "\u03B7": "eta", "\u03B8": "theta", "\u03B9": "iota", "\u03BA": "kappa", "\u03BB": "lambda", "\u03BC": "mu", "\xB5": "mu", "\u03BD": "nu", "\u03BE": "xi", "\u03C0": "pi", "\u03C1": "rho", "\u03C3": "sigma", "\u03C4": "tau", "\u03C5": "upsilon", "\u03C6": "phi", "\u03C7": "chi", "\u03C8": "psi", "\u03C9": "omega", "\u0391": "alpha", "\u0392": "beta", "\u0393": "gamma", "\u0394": "delta", "\u03A9": "omega", "\u03A3": "sigma", "\u039A": "kappa", "\u039B": "lambda", "\u03A0": "pi", "\u03A6": "phi", "\u03A8": "psi" };
|
|
1527
1489
|
var greek = (s) => s.replace(/[\u0370-\u03ff\u00b5\u00df]/g, (c) => GREEK[c] ?? c);
|
|
@@ -1934,24 +1896,55 @@ function renderFulltextMd(doc, headLine2) {
|
|
|
1934
1896
|
}
|
|
1935
1897
|
|
|
1936
1898
|
// src/browser.ts
|
|
1937
|
-
import { existsSync as
|
|
1938
|
-
import { join as
|
|
1899
|
+
import { existsSync as existsSync4, mkdirSync as mkdirSync2, readFileSync as readFileSync3, writeFileSync as writeFileSync2 } from "fs";
|
|
1900
|
+
import { join as join4 } from "path";
|
|
1939
1901
|
import { chromium } from "playwright-core";
|
|
1940
|
-
|
|
1902
|
+
|
|
1903
|
+
// src/layout.ts
|
|
1904
|
+
import { createHash } from "crypto";
|
|
1905
|
+
import { existsSync as existsSync3 } from "fs";
|
|
1906
|
+
import { join as join3, resolve as resolve2 } from "path";
|
|
1907
|
+
var AI4L_LIBRARY_DIR = (() => {
|
|
1908
|
+
const env = process.env.AI4L_LIBRARY_DIR;
|
|
1909
|
+
if (env && env.trim()) return resolve2(env.trim());
|
|
1910
|
+
if (existsSync3("/library")) return "/library";
|
|
1911
|
+
return resolve2("library");
|
|
1912
|
+
})();
|
|
1913
|
+
var doiSlug = (doi) => doi.toLowerCase().replace(/[^a-z0-9.-]+/g, "_");
|
|
1914
|
+
function entryDirRel(type, num2) {
|
|
1915
|
+
if (type === "doi") {
|
|
1916
|
+
const h = createHash("sha1").update(num2).digest("hex");
|
|
1917
|
+
return `${type}/${h.slice(0, 2)}/${h.slice(0, 4)}/${num2}`;
|
|
1918
|
+
}
|
|
1919
|
+
if (!/^\d+$/.test(num2)) return `${type}/${num2.slice(0, 2)}/${num2.slice(0, 4)}/${num2}`;
|
|
1920
|
+
const l1 = num2.length > 6 ? num2.slice(0, -6) : "0";
|
|
1921
|
+
const l2 = num2.length > 4 ? num2.slice(0, -4) : "0";
|
|
1922
|
+
return `${type}/${l1}/${l2}/${num2}`;
|
|
1923
|
+
}
|
|
1924
|
+
function entryPaths(libraryDir, type, num2) {
|
|
1925
|
+
if (type === "doi") num2 = doiSlug(num2);
|
|
1926
|
+
const entry = `${type}-${num2}`;
|
|
1927
|
+
const entryDir = join3(libraryDir, entryDirRel(type, num2));
|
|
1928
|
+
return {
|
|
1929
|
+
entry,
|
|
1930
|
+
entryDir,
|
|
1931
|
+
xml: join3(entryDir, `${entry}-abstract.xml`),
|
|
1932
|
+
md: join3(entryDir, `${entry}-abstract.md`),
|
|
1933
|
+
meta: join3(entryDir, `${entry}-meta.md`),
|
|
1934
|
+
fulltextMd: join3(entryDir, `${entry}-fulltext.md`),
|
|
1935
|
+
fulltextRaw: (format) => join3(entryDir, `${entry}-fulltext.${format}`)
|
|
1936
|
+
};
|
|
1937
|
+
}
|
|
1938
|
+
|
|
1939
|
+
// src/browser.ts
|
|
1940
|
+
function browserConfig(env = process.env) {
|
|
1941
1941
|
const out = { browsers: [], headless: "auto" };
|
|
1942
|
-
|
|
1943
|
-
|
|
1944
|
-
|
|
1945
|
-
if (Array.isArray(d.browsers)) {
|
|
1946
|
-
for (const b of d.browsers) {
|
|
1947
|
-
if (typeof b !== "string") continue;
|
|
1948
|
-
const v = resolveEnv(b).trim();
|
|
1949
|
-
if (v && !/^\$\{[A-Z0-9_]+\}$/.test(v) && !out.browsers.includes(v)) out.browsers.push(v);
|
|
1950
|
-
}
|
|
1951
|
-
}
|
|
1952
|
-
if (d.headless === "auto" || typeof d.headless === "boolean") out.headless = d.headless;
|
|
1953
|
-
} catch {
|
|
1942
|
+
for (let i = 1; i <= 9; i++) {
|
|
1943
|
+
const v = str(`AI4L_LIBRARY_BROWSER_${i}`, env);
|
|
1944
|
+
if (v && !out.browsers.includes(v)) out.browsers.push(v);
|
|
1954
1945
|
}
|
|
1946
|
+
const h = str("AI4L_LIBRARY_BROWSER_HEADLESS", env).toLowerCase();
|
|
1947
|
+
if (h === "true" || h === "false") out.headless = h === "true";
|
|
1955
1948
|
return out;
|
|
1956
1949
|
}
|
|
1957
1950
|
var INSTALL_HINT = "npx playwright-core install chromium";
|
|
@@ -1962,7 +1955,7 @@ var rungKind = (endpoint) => isCdp(endpoint) ? "chrome" : isRemote(endpoint) ? "
|
|
|
1962
1955
|
function localChromium() {
|
|
1963
1956
|
try {
|
|
1964
1957
|
const p = chromium.executablePath();
|
|
1965
|
-
return
|
|
1958
|
+
return existsSync4(p) ? p : null;
|
|
1966
1959
|
} catch {
|
|
1967
1960
|
return null;
|
|
1968
1961
|
}
|
|
@@ -1999,12 +1992,12 @@ var IDLE_CLOSE_MS = 12e4;
|
|
|
1999
1992
|
var ctxs = /* @__PURE__ */ new Map();
|
|
2000
1993
|
var idleTimer = null;
|
|
2001
1994
|
function profileDir() {
|
|
2002
|
-
const dir =
|
|
2003
|
-
mkdirSync2(
|
|
2004
|
-
const prefs =
|
|
1995
|
+
const dir = join4(AI4L_LIBRARY_DIR, ".browser");
|
|
1996
|
+
mkdirSync2(join4(dir, "Default"), { recursive: true });
|
|
1997
|
+
const prefs = join4(dir, "Default", "Preferences");
|
|
2005
1998
|
let p = {};
|
|
2006
1999
|
try {
|
|
2007
|
-
p = JSON.parse(
|
|
2000
|
+
p = JSON.parse(readFileSync3(prefs, "utf8"));
|
|
2008
2001
|
} catch {
|
|
2009
2002
|
}
|
|
2010
2003
|
p.plugins = { ...p.plugins ?? {}, always_open_pdf_externally: true };
|
|
@@ -2017,9 +2010,9 @@ var LOCAL_WINDOW_ARGS = [...LAUNCH_ARGS, "--window-position=4000,4000"];
|
|
|
2017
2010
|
var TIMEZONE = Intl.DateTimeFormat().resolvedOptions().timeZone || "Europe/Berlin";
|
|
2018
2011
|
var CONTEXT_OPTS = { acceptDownloads: true, viewport: null, locale: "en-US", timezoneId: TIMEZONE };
|
|
2019
2012
|
function stateFile(endpoint) {
|
|
2020
|
-
const dir =
|
|
2013
|
+
const dir = join4(AI4L_LIBRARY_DIR, ".browser");
|
|
2021
2014
|
mkdirSync2(dir, { recursive: true });
|
|
2022
|
-
return
|
|
2015
|
+
return join4(dir, `${browserLabel(endpoint).replace(/[^A-Za-z0-9.-]+/g, "_")}.json`);
|
|
2023
2016
|
}
|
|
2024
2017
|
async function context(endpoint, cfg) {
|
|
2025
2018
|
let p = ctxs.get(endpoint);
|
|
@@ -2042,7 +2035,7 @@ async function context(endpoint, cfg) {
|
|
|
2042
2035
|
});
|
|
2043
2036
|
browser.on("disconnected", drop);
|
|
2044
2037
|
const state = stateFile(endpoint);
|
|
2045
|
-
const ctx2 = await browser.newContext({ ...CONTEXT_OPTS, ...
|
|
2038
|
+
const ctx2 = await browser.newContext({ ...CONTEXT_OPTS, ...existsSync4(state) ? { storageState: state } : {} });
|
|
2046
2039
|
return { ctx: ctx2, browser };
|
|
2047
2040
|
}
|
|
2048
2041
|
if (!localChromium()) throw new Error(`no local Chromium (${INSTALL_HINT})`);
|
|
@@ -2137,7 +2130,7 @@ async function browserFetch(url, endpoint, cfg = browserConfig()) {
|
|
|
2137
2130
|
}
|
|
2138
2131
|
if (download2) {
|
|
2139
2132
|
const d = download2;
|
|
2140
|
-
return { bytes:
|
|
2133
|
+
return { bytes: readFileSync3(await d.path()), contentType: "application/pdf", url: d.url(), via: "download" };
|
|
2141
2134
|
}
|
|
2142
2135
|
const title = await page.title().catch(() => "");
|
|
2143
2136
|
if (CHALLENGE.test(title)) throw new Error(`challenge not cleared (${title.slice(0, 40)})`);
|
|
@@ -2456,16 +2449,16 @@ function europepmcCandidates(r, pmcid) {
|
|
|
2456
2449
|
}
|
|
2457
2450
|
|
|
2458
2451
|
// src/tiers/inbox.ts
|
|
2459
|
-
import { existsSync as
|
|
2460
|
-
import { join as
|
|
2461
|
-
var INBOX_DIR =
|
|
2452
|
+
import { existsSync as existsSync5, readdirSync as readdirSync2, readFileSync as readFileSync4, unlinkSync } from "fs";
|
|
2453
|
+
import { join as join5 } from "path";
|
|
2454
|
+
var INBOX_DIR = join5(AI4L_LIBRARY_DIR, "inbox");
|
|
2462
2455
|
var EXT = { ".pdf": "application/pdf", ".xml": "text/xml", ".html": "text/html", ".htm": "text/html" };
|
|
2463
2456
|
function findInboxFile(dir, pmid) {
|
|
2464
|
-
if (!
|
|
2457
|
+
if (!existsSync5(dir)) return null;
|
|
2465
2458
|
const re = new RegExp(`(^|[^0-9])${pmid}([^0-9]|$)`);
|
|
2466
2459
|
for (const name of readdirSync2(dir).sort()) {
|
|
2467
2460
|
const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
|
|
2468
|
-
if (ext in EXT && re.test(name.slice(0, -ext.length))) return
|
|
2461
|
+
if (ext in EXT && re.test(name.slice(0, -ext.length))) return join5(dir, name);
|
|
2469
2462
|
}
|
|
2470
2463
|
return null;
|
|
2471
2464
|
}
|
|
@@ -2479,7 +2472,7 @@ function inboxTier(dir = INBOX_DIR) {
|
|
|
2479
2472
|
if (!file) return { ...miss("inbox", "no file"), skipped: true, ephemeral: true };
|
|
2480
2473
|
const name = file.slice(file.lastIndexOf("/") + 1);
|
|
2481
2474
|
const ext = name.slice(name.lastIndexOf(".")).toLowerCase();
|
|
2482
|
-
const bytes =
|
|
2475
|
+
const bytes = readFileSync4(file);
|
|
2483
2476
|
const r = await convertBytes(p, bytes, EXT[ext]);
|
|
2484
2477
|
if (!r.ok) return miss("inbox", `${name}: ${r.reason}`);
|
|
2485
2478
|
ctx.onTry?.(`${name} ok, removed from the inbox`);
|
|
@@ -2678,74 +2671,70 @@ function urlTier(name, template, headers) {
|
|
|
2678
2671
|
}
|
|
2679
2672
|
|
|
2680
2673
|
// src/config.ts
|
|
2681
|
-
var
|
|
2682
|
-
|
|
2683
|
-
|
|
2684
|
-
|
|
2685
|
-
|
|
2686
|
-
|
|
2687
|
-
|
|
2688
|
-
|
|
2689
|
-
|
|
2690
|
-
|
|
2691
|
-
|
|
2692
|
-
|
|
2693
|
-
|
|
2694
|
-
|
|
2674
|
+
var VARIABLES = [
|
|
2675
|
+
"AI4L_LIBRARY_DIR",
|
|
2676
|
+
"AI4L_LIBRARY_ENV",
|
|
2677
|
+
"AI4L_LIBRARY_TIERS",
|
|
2678
|
+
"AI4L_LIBRARY_CONTACT_EMAIL",
|
|
2679
|
+
"AI4L_LIBRARY_ELSEVIER_API_KEY",
|
|
2680
|
+
"AI4L_LIBRARY_WILEY_TDM_TOKEN",
|
|
2681
|
+
"AI4L_LIBRARY_CORE_API_KEY",
|
|
2682
|
+
"AI4L_LIBRARY_PROXY_URL",
|
|
2683
|
+
"AI4L_LIBRARY_PROXY_HEADERS",
|
|
2684
|
+
"AI4L_LIBRARY_BROWSER_1",
|
|
2685
|
+
"AI4L_LIBRARY_BROWSER_2",
|
|
2686
|
+
"AI4L_LIBRARY_BROWSER_3",
|
|
2687
|
+
"AI4L_LIBRARY_BROWSER_HEADLESS",
|
|
2688
|
+
"AI4L_LIBRARY_VERIFY_URL",
|
|
2689
|
+
"AI4L_LIBRARY_VERIFY_API_KEY",
|
|
2690
|
+
"AI4L_LIBRARY_VERIFY_MODEL",
|
|
2691
|
+
"AI4L_LIBRARY_VERIFY_THRESHOLD",
|
|
2692
|
+
"AI4L_LIBRARY_VERIFY_TIMEOUT_SECS",
|
|
2693
|
+
"AI4L_LIBRARY_RETRY_AFTER_DAYS",
|
|
2694
|
+
"AI4L_LIBRARY_RETRY_BLOCKED",
|
|
2695
|
+
"AI4L_LIBRARY_REFRESH_AFTER_DAYS",
|
|
2696
|
+
"NCBI_API_KEY",
|
|
2697
|
+
"BRIGHTDATA_API_TOKEN",
|
|
2698
|
+
"BRIGHTDATA_ZONE"
|
|
2699
|
+
];
|
|
2700
|
+
var sources = (env = process.env) => Object.fromEntries(VARIABLES.map((v) => [v, sourceOf(v, env)]));
|
|
2701
|
+
function settings(env = process.env) {
|
|
2702
|
+
return {
|
|
2703
|
+
retry_after_days: num("AI4L_LIBRARY_RETRY_AFTER_DAYS", 1, env),
|
|
2704
|
+
retry_blocked: bool("AI4L_LIBRARY_RETRY_BLOCKED", false, env),
|
|
2705
|
+
refresh_after_days: num("AI4L_LIBRARY_REFRESH_AFTER_DAYS", 30, env)
|
|
2706
|
+
};
|
|
2707
|
+
}
|
|
2708
|
+
function proxyHeaders(spec) {
|
|
2709
|
+
const out = {};
|
|
2710
|
+
for (const part of spec.split(";")) {
|
|
2711
|
+
const i = part.indexOf(":");
|
|
2712
|
+
if (i < 0) continue;
|
|
2713
|
+
const name = part.slice(0, i).trim(), value = part.slice(i + 1).trim();
|
|
2714
|
+
if (name && value) out[name] = value;
|
|
2695
2715
|
}
|
|
2696
2716
|
return out;
|
|
2697
2717
|
}
|
|
2698
|
-
function resolveTiers(env = process.env
|
|
2718
|
+
function resolveTiers(env = process.env) {
|
|
2699
2719
|
const problems = [];
|
|
2700
|
-
const
|
|
2701
|
-
const
|
|
2702
|
-
|
|
2703
|
-
|
|
2704
|
-
if (
|
|
2705
|
-
|
|
2706
|
-
|
|
2707
|
-
|
|
2708
|
-
|
|
2709
|
-
|
|
2710
|
-
|
|
2711
|
-
|
|
2712
|
-
if (
|
|
2713
|
-
|
|
2714
|
-
|
|
2715
|
-
|
|
2716
|
-
|
|
2717
|
-
|
|
2718
|
-
continue;
|
|
2719
|
-
}
|
|
2720
|
-
const enabled = e.enabled !== false;
|
|
2721
|
-
const description = typeof e.description === "string" ? e.description : void 0;
|
|
2722
|
-
if (typeof e.url === "string" && e.url.trim()) ordered.set(e.name, { tier: urlTier(e.name, resolveEnv(e.url.trim()), resolveMap(e.headers)), enabled, kind: "url", description });
|
|
2723
|
-
else if (builtin.has(e.name)) ordered.set(e.name, { tier: builtin.get(e.name).make({ tokens: resolveMap(e.tokens), api_key: typeof e.api_key === "string" ? resolveEnv(e.api_key) : void 0 }), enabled, kind: "builtin", description });
|
|
2724
|
-
else problems.push(`${file}: unknown tier "${e.name}" (built-in: ${[...builtin.keys()].join(", ")}; a user tier needs "url")`);
|
|
2725
|
-
}
|
|
2726
|
-
for (const [k, v] of known) if (!ordered.has(k)) ordered.set(k, v);
|
|
2727
|
-
known.clear();
|
|
2728
|
-
for (const [k, v] of ordered) known.set(k, v);
|
|
2729
|
-
}
|
|
2730
|
-
}
|
|
2731
|
-
if (env && env.trim()) {
|
|
2732
|
-
source = "env";
|
|
2733
|
-
for (const v of known.values()) v.enabled = false;
|
|
2734
|
-
const ordered = /* @__PURE__ */ new Map();
|
|
2735
|
-
for (const name of env.split(",").map((s) => s.trim()).filter(Boolean)) {
|
|
2736
|
-
const v = known.get(name);
|
|
2737
|
-
if (!v) {
|
|
2738
|
-
problems.push(`AI4L_LIBRARY_TIERS: unknown tier "${name}"`);
|
|
2739
|
-
continue;
|
|
2740
|
-
}
|
|
2741
|
-
ordered.set(name, { ...v, enabled: true });
|
|
2742
|
-
}
|
|
2743
|
-
for (const [k, v] of known) if (!ordered.has(k)) ordered.set(k, v);
|
|
2744
|
-
known.clear();
|
|
2745
|
-
for (const [k, v] of ordered) known.set(k, v);
|
|
2746
|
-
}
|
|
2747
|
-
const all = [...known.entries()].map(([name, v]) => ({ name, enabled: v.enabled, description: v.description ?? (v.kind === "url" ? "url tier" : "built-in") }));
|
|
2748
|
-
return { tiers: [...known.values()].filter((v) => v.enabled).map((v) => v.tier), all, source, file, problems };
|
|
2720
|
+
const available = /* @__PURE__ */ new Map();
|
|
2721
|
+
const tokens = { elsevier: str("AI4L_LIBRARY_ELSEVIER_API_KEY", env) || void 0, wiley: str("AI4L_LIBRARY_WILEY_TDM_TOKEN", env) || void 0 };
|
|
2722
|
+
for (const t of BUILTIN) available.set(t.name, t.make({ tokens, api_key: str("AI4L_LIBRARY_CORE_API_KEY", env) || void 0 }));
|
|
2723
|
+
const proxyUrl = str("AI4L_LIBRARY_PROXY_URL", env);
|
|
2724
|
+
if (proxyUrl) available.set("proxy", urlTier("proxy", proxyUrl, proxyHeaders(str("AI4L_LIBRARY_PROXY_HEADERS", env))));
|
|
2725
|
+
const listed = str("AI4L_LIBRARY_TIERS", env);
|
|
2726
|
+
const order2 = listed ? listed.split(",").map((s) => s.trim()).filter(Boolean) : [...available.keys()];
|
|
2727
|
+
const tiers = [], seen = /* @__PURE__ */ new Set();
|
|
2728
|
+
for (const name of order2) {
|
|
2729
|
+
if (seen.has(name)) continue;
|
|
2730
|
+
seen.add(name);
|
|
2731
|
+
const t = available.get(name);
|
|
2732
|
+
if (t) tiers.push(t);
|
|
2733
|
+
else if (name === "proxy") problems.push('AI4L_LIBRARY_TIERS: "proxy" listed, but AI4L_LIBRARY_PROXY_URL is unset');
|
|
2734
|
+
else problems.push(`AI4L_LIBRARY_TIERS: unknown tier "${name}" (built-in: ${BUILTIN.map((t2) => t2.name).join(", ")}; proxy when AI4L_LIBRARY_PROXY_URL is set)`);
|
|
2735
|
+
}
|
|
2736
|
+
const all = [...tiers.map((t) => ({ name: t.name, enabled: true })), ...[...available.keys()].filter((n) => !seen.has(n)).map((name) => ({ name, enabled: false }))];
|
|
2737
|
+
return { tiers, all, source: sourceOf("AI4L_LIBRARY_TIERS", env), problems };
|
|
2749
2738
|
}
|
|
2750
2739
|
|
|
2751
2740
|
// src/cascade.ts
|
|
@@ -2820,34 +2809,26 @@ function parseId(raw) {
|
|
|
2820
2809
|
}
|
|
2821
2810
|
|
|
2822
2811
|
// src/info.ts
|
|
2823
|
-
import { existsSync as
|
|
2824
|
-
import { dirname as dirname2, join as
|
|
2812
|
+
import { existsSync as existsSync7, readdirSync as readdirSync3, readFileSync as readFileSync6, statSync } from "fs";
|
|
2813
|
+
import { dirname as dirname2, join as join6 } from "path";
|
|
2825
2814
|
|
|
2826
2815
|
// src/verify.ts
|
|
2827
|
-
import { existsSync as
|
|
2816
|
+
import { existsSync as existsSync6, readFileSync as readFileSync5 } from "fs";
|
|
2828
2817
|
var DEFAULT_MODEL = "pplx-decider-flash-v1.2";
|
|
2829
2818
|
var QUESTION = "Does the paper state this, in substance? Ignore section labels and definitions of terms in parentheses. Every fact and number of the statement must be stated in the paper or follow from its numbers; a rounded number counts. If any part of the statement is absent from the paper or contradicted by it, answer false.";
|
|
2830
|
-
function verifyConfig(
|
|
2831
|
-
|
|
2832
|
-
|
|
2833
|
-
|
|
2834
|
-
|
|
2835
|
-
|
|
2836
|
-
|
|
2837
|
-
|
|
2838
|
-
|
|
2839
|
-
|
|
2840
|
-
|
|
2841
|
-
|
|
2842
|
-
|
|
2843
|
-
// Hosted takes 256k tokens in one request (a 50k-token paper in 2 s); the
|
|
2844
|
-
// Mac app is limited to 32k and slows steeply beyond 25k, so it windows.
|
|
2845
|
-
window_chars: typeof v.window_chars === "number" && v.window_chars >= 1e4 ? v.window_chars : isHosted(base_url) ? 8e5 : 9e4,
|
|
2846
|
-
timeout_seconds: typeof v.timeout_seconds === "number" && v.timeout_seconds > 0 ? v.timeout_seconds : 60
|
|
2847
|
-
};
|
|
2848
|
-
} catch {
|
|
2849
|
-
return null;
|
|
2850
|
-
}
|
|
2819
|
+
function verifyConfig(env = process.env) {
|
|
2820
|
+
const base_url = str("AI4L_LIBRARY_VERIFY_URL", env).replace(/\/+$/, "");
|
|
2821
|
+
if (!base_url) return null;
|
|
2822
|
+
return {
|
|
2823
|
+
base_url,
|
|
2824
|
+
api_key: str("AI4L_LIBRARY_VERIFY_API_KEY", env) || null,
|
|
2825
|
+
model: str("AI4L_LIBRARY_VERIFY_MODEL", env) || DEFAULT_MODEL,
|
|
2826
|
+
threshold: num("AI4L_LIBRARY_VERIFY_THRESHOLD", 0.5, env, (n) => n > 0 && n < 1),
|
|
2827
|
+
// Hosted takes 256k tokens in one request (a 50k-token paper in 2 s); the
|
|
2828
|
+
// Mac app is limited to 32k and slows steeply beyond 25k, so it windows.
|
|
2829
|
+
window_chars: isHosted(base_url) ? 8e5 : 9e4,
|
|
2830
|
+
timeout_seconds: num("AI4L_LIBRARY_VERIFY_TIMEOUT_SECS", 60, env, (n) => n > 0)
|
|
2831
|
+
};
|
|
2851
2832
|
}
|
|
2852
2833
|
var isHosted = (baseUrl) => {
|
|
2853
2834
|
try {
|
|
@@ -2933,10 +2914,10 @@ async function readPaper(raw, libraryDir = AI4L_LIBRARY_DIR) {
|
|
|
2933
2914
|
if (p.error) return { id: p.requested, error: p.error };
|
|
2934
2915
|
const paths = entryPaths(libraryDir, p.type, p.id);
|
|
2935
2916
|
if (!entryExists(paths)) return { id: p.requested, error: "not in library" };
|
|
2936
|
-
const full =
|
|
2937
|
-
if (full) return { id: p.requested, basis: "full_text", text:
|
|
2917
|
+
const full = existsSync6(paths.fulltextMd);
|
|
2918
|
+
if (full) return { id: p.requested, basis: "full_text", text: readFileSync5(paths.fulltextMd, "utf8") };
|
|
2938
2919
|
const meta = readMeta(paths);
|
|
2939
|
-
return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text:
|
|
2920
|
+
return { id: p.requested, basis: meta.has_abstract ? "abstract" : "title", text: readFileSync5(paths.md, "utf8") };
|
|
2940
2921
|
}
|
|
2941
2922
|
async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
|
|
2942
2923
|
const out = { claim, id, verdict: "error", score: null, basis: null, windows: 0, glyphs_unresolved: 0, superseded: null, model: cfg?.model ?? null };
|
|
@@ -2950,7 +2931,7 @@ async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
|
|
|
2950
2931
|
const paper = await read(id);
|
|
2951
2932
|
out.id = paper.id;
|
|
2952
2933
|
if ("error" in paper) return { ...out, error: paper.error };
|
|
2953
|
-
if (!cfg || !decide) return { ...out, error: "no verify model configured (
|
|
2934
|
+
if (!cfg || !decide) return { ...out, error: "no verify model configured (AI4L_LIBRARY_VERIFY_URL)" };
|
|
2954
2935
|
out.basis = paper.basis;
|
|
2955
2936
|
out.glyphs_unresolved = countUnresolved(paper.text);
|
|
2956
2937
|
if (paper.basis === "title") return { ...out, verdict: "na" };
|
|
@@ -2977,9 +2958,9 @@ async function verifyClaim(claim, id, cfg, decide, read = readPaper) {
|
|
|
2977
2958
|
|
|
2978
2959
|
// src/info.ts
|
|
2979
2960
|
function* metaFiles(dir) {
|
|
2980
|
-
if (!
|
|
2961
|
+
if (!existsSync7(dir)) return;
|
|
2981
2962
|
for (const name of readdirSync3(dir)) {
|
|
2982
|
-
const p =
|
|
2963
|
+
const p = join6(dir, name);
|
|
2983
2964
|
if (statSync(p).isDirectory()) yield* metaFiles(p);
|
|
2984
2965
|
else if (name.endsWith("-meta.md")) yield p;
|
|
2985
2966
|
}
|
|
@@ -3002,7 +2983,7 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
|
|
|
3002
2983
|
library_dir: libraryDir,
|
|
3003
2984
|
host,
|
|
3004
2985
|
tiers: tiers.all,
|
|
3005
|
-
|
|
2986
|
+
config: { ...settings(), problems: tiers.problems, sources: sources() },
|
|
3006
2987
|
download: downloadInfo(),
|
|
3007
2988
|
verify: verifyInfo(),
|
|
3008
2989
|
entries: 0,
|
|
@@ -3018,11 +2999,11 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
|
|
|
3018
2999
|
updates: 0
|
|
3019
3000
|
};
|
|
3020
3001
|
const ids = [];
|
|
3021
|
-
for (const type of
|
|
3022
|
-
const typeDir =
|
|
3002
|
+
for (const type of existsSync7(libraryDir) ? readdirSync3(libraryDir) : []) {
|
|
3003
|
+
const typeDir = join6(libraryDir, type);
|
|
3023
3004
|
if (!statSync(typeDir).isDirectory()) continue;
|
|
3024
3005
|
for (const meta of metaFiles(typeDir)) {
|
|
3025
|
-
const md =
|
|
3006
|
+
const md = readFileSync6(meta, "utf8");
|
|
3026
3007
|
const raw = field(md, "level") || "unknown";
|
|
3027
3008
|
const level = raw === "abstract" && field(md, "has_abstract") === "no" ? "title" : raw;
|
|
3028
3009
|
info.entries++;
|
|
@@ -3034,7 +3015,7 @@ function libraryInfo(libraryDir, host, withIds, tiers = resolveTiers()) {
|
|
|
3034
3015
|
const dir = dirname2(meta);
|
|
3035
3016
|
for (const f of readdirSync3(dir)) {
|
|
3036
3017
|
try {
|
|
3037
|
-
const st = statSync(
|
|
3018
|
+
const st = statSync(join6(dir, f));
|
|
3038
3019
|
if (st.isFile()) info.size_bytes += st.size;
|
|
3039
3020
|
} catch {
|
|
3040
3021
|
}
|
|
@@ -3063,7 +3044,7 @@ function noticeTypes(md, metaPath) {
|
|
|
3063
3044
|
return out;
|
|
3064
3045
|
}
|
|
3065
3046
|
try {
|
|
3066
|
-
const xml =
|
|
3047
|
+
const xml = readFileSync6(metaPath.replace(/-meta\.md$/, "-abstract.xml"), "utf8");
|
|
3067
3048
|
for (const m of xml.matchAll(/<CommentsCorrections RefType="([A-Za-z]+)"/g)) if (XML_TYPES[m[1]]) out.add(XML_TYPES[m[1]]);
|
|
3068
3049
|
} catch {
|
|
3069
3050
|
}
|
|
@@ -3165,9 +3146,9 @@ var escape = (s) => s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&");
|
|
|
3165
3146
|
// src/log.ts
|
|
3166
3147
|
import { appendFileSync, mkdirSync as mkdirSync3, renameSync as renameSync2, statSync as statSync2 } from "fs";
|
|
3167
3148
|
import { hostname } from "os";
|
|
3168
|
-
import { join as
|
|
3149
|
+
import { join as join7 } from "path";
|
|
3169
3150
|
var HOST = hostname();
|
|
3170
|
-
var LOG_FILE =
|
|
3151
|
+
var LOG_FILE = join7(AI4L_LIBRARY_DIR, "library.log");
|
|
3171
3152
|
function stamp() {
|
|
3172
3153
|
const d = /* @__PURE__ */ new Date();
|
|
3173
3154
|
const p = (n) => String(n).padStart(2, "0");
|
|
@@ -3355,7 +3336,7 @@ function parseCsl(json2) {
|
|
|
3355
3336
|
// src/tools.ts
|
|
3356
3337
|
var pkg = (() => {
|
|
3357
3338
|
try {
|
|
3358
|
-
return JSON.parse(
|
|
3339
|
+
return JSON.parse(readFileSync7(new URL("../package.json", import.meta.url), "utf8"));
|
|
3359
3340
|
} catch {
|
|
3360
3341
|
return { name: "ai4l-library", version: "0.0.0" };
|
|
3361
3342
|
}
|
|
@@ -3367,7 +3348,7 @@ Tools:
|
|
|
3367
3348
|
- library_status(id) \u2014 read-only: the entry's status, or level "missing".
|
|
3368
3349
|
- library_read(id, section?, query?, max_chars?, max_hits?) \u2014 one small piece of a stored paper's Markdown, never the whole paper. No section/query: the outline (headings with word counts, and whether it is the full text or the abstract only). section: a heading from the outline (case-insensitive substring) \u2192 that section's text, capped at max_chars. query: search terms \u2192 the paragraphs containing all of them, each with its section, best first, at most max_hits; with no hit the answer lists the headings instead. Every full text converted from PMC/JATS XML has a "Declarations" section \u2014 funding, conflicts of interest, author notes \u2014 so section "Declarations" answers who paid and who declares what in one call; an empty answer there means the paper states nothing. Reads are read-only and may run in parallel.
|
|
3369
3350
|
- library_verify(claim, id) \u2014 is one finding of a review stated by one source it cites? claim: the finding alone as a complete statement (subject, intervention, outcome, number), without its citations and without the review's framing or inference; id: one source the review cites for it \u2014 a paper, a drug label (dailymed:\u2026) or a trial record (nct:\u2026), each judged on its stored full text the same way. A finding cited to three papers is three calls with the same claim; a claim that names a source, joins two findings or is a fragment is refused. The server reads the paper (full text, else abstract) and asks a decision model; nothing of the paper enters your context. Result: verdict "pass" (score is the probability), "fail" (the full text does not state it), "na" (only the abstract is in the library and does not state it, or only the title) or "error" (claim empty, too short, a fragment, naming a source or compound; paper not in the library; no model; model failure); basis full_text, abstract or title; glyphs_unresolved, the symbols the paper's PDF font gave no character for (above 0: a fail may be the font, not the paper); superseded, true when the claim uses a figure a stored correction replaced (the verdict is kept), null without a stored correction. Call library_add on the paper first.
|
|
3370
|
-
- library_info(ids?) \u2014 library-wide, everything the TUI shows: the library directory, the decision model and whether it answers, entries and size on disk, per type (pmid, doi, dailymed, nct) the entries with their levels (full_text, abstract, title) and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the
|
|
3351
|
+
- library_info(ids?) \u2014 library-wide, everything the TUI shows: the library directory, the decision model and whether it answers, entries and size on disk, per type (pmid, doi, dailymed, nct) the entries with their levels (full_text, abstract, title) and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the retrieval tiers in run order (and which are off), the configuration (config: retry and refresh settings, problems with AI4L_LIBRARY_TIERS, and per variable whether its value came from the environment, .env or the default \u2014 never the value), the verify model with its endpoint and whether it answers right now (ok, ms, error); ids=true lists every entry ID.
|
|
3371
3352
|
- get_version() \u2014 package name, version and the resolved library directory.
|
|
3372
3353
|
|
|
3373
3354
|
Every action is appended to <library>/library.log, one line per action, prefixed with the host the server runs on.
|
|
@@ -3674,8 +3655,8 @@ function registerTools(server2) {
|
|
|
3674
3655
|
log("library_read", parsed.requested, "error missing");
|
|
3675
3656
|
return out({ requested: parsed.requested, entry: paths.entry, error: "missing \u2014 call library_add first" });
|
|
3676
3657
|
}
|
|
3677
|
-
const full =
|
|
3678
|
-
const md =
|
|
3658
|
+
const full = existsSync8(paths.fulltextMd);
|
|
3659
|
+
const md = readFileSync7(full ? paths.fulltextMd : paths.md, "utf8");
|
|
3679
3660
|
const text3 = full ? "full_text" : "abstract";
|
|
3680
3661
|
const base = { requested: parsed.requested, entry: paths.entry, text: text3 };
|
|
3681
3662
|
if (section2) {
|
|
@@ -3713,7 +3694,7 @@ function registerTools(server2) {
|
|
|
3713
3694
|
);
|
|
3714
3695
|
server2.tool(
|
|
3715
3696
|
"library_info",
|
|
3716
|
-
"Library-wide summary \u2014 everything the TUI shows: the library directory and host; the decision model with its endpoint and whether it answers; the entry count and the size on disk (size, size_bytes); per entry type (types) the entries and their levels (full_text, abstract, title) and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the
|
|
3697
|
+
"Library-wide summary \u2014 everything the TUI shows: the library directory and host; the decision model with its endpoint and whether it answers; the entry count and the size on disk (size, size_bytes); per entry type (types) the entries and their levels (full_text, abstract, title) and retractions; entry counts by type and level, counts of retracted and not-citable entries and of entries with errata, expressions of concern and updates, the retrieval tiers in run order and which are off (tiers), the configuration (config: settings, problems, and per variable the source of its value \u2014 environment, .env or default \u2014 never the value), the decision model behind library_verify (model, endpoint, whether it runs on this machine, and whether it answers right now: ok with the answer time, or the error), the library directory and host. ids=true also lists every entry ID.",
|
|
3717
3698
|
{ ids: z.boolean().optional().describe("Also list every entry ID (type:value).") },
|
|
3718
3699
|
async ({ ids }) => {
|
|
3719
3700
|
const info = libraryInfo(AI4L_LIBRARY_DIR, HOST, ids === true);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "ai4l-library",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.4",
|
|
4
4
|
"mcpName": "io.github.forever-healthy/ai4l-library",
|
|
5
5
|
"description": "MCP Server that builds a local Markdown library of the scientific papers cited by evidence reviews — fetches metadata, abstract and full text per PubMed ID through a configurable cascade of retrieval services",
|
|
6
6
|
"type": "module",
|