@pharmatools/opengate 0.2.1 → 0.4.0

package/ADAPTERS.md

@@ -47,8 +47,10 @@ Every adapter provides two base exports, plus at least one complete **capability
 
 - **`qa`** — `splitClaims(text)` + `analyzeBatch(payload)`: systems that extract claims and verify them against references (scorers: claim-extraction, verdict-accuracy).
 - **`redaction`** — `redact(text)`: systems that remove identifiers from text (scorer: redaction). Returns `{ text, entities: [{ value, type }] }`, where `entities` are the identifiers the system removed.
+- **`simplify`** — `simplify({ text, audience?, tone?, length?, language? })`: systems that rewrite source text for a different audience (scorer: simplification). Returns `{ text }`. Scored on anchor recall (critical facts survive), fabricated numbers (nothing invented), and length contracts — paraphrase is expected, so prose is never checked verbatim.
+- **`retrieval`** — `fetchRecord({ id, type? })`: systems that retrieve records from an authority (scorer: retrieval). Returns `{ record }`. Scored on retrieval fidelity — field presence, hand-verified anchor fields, and structural invariants — catching parser regressions that would silently poison downstream grounding. For deterministic systems (no model), not just AI ones.
 
-Scorers check `adapter.capabilities.<name>` and skip with a reason when a capability is absent — the bundled Redacta adapter (`src/adapters/redacta.mjs`) runs only the redaction scorer, the RefCheckr adapter only the QA scorers.
+Scorers check `adapter.capabilities.<name>` and skip with a reason when a capability is absent — the bundled Redacta adapter runs only redaction, RefCheckr only QA, Patiently only simplification, and PubCrawl (`src/adapters/pubcrawl.mjs`) only retrieval.
 
 ### Base exports (always required)
 

package/README.md

@@ -80,8 +80,8 @@ npm run eval:ci         # exit non-zero on any failure or metric regression
                                         │ adapters
                  ┌──────────────┬───────┴──────┬──────────────┐
                  ▼              ▼              ▼              ▼
-             RefCheckr       Redacta      Patiently AI   your system
-          (first impl., QA) (redaction)    (planned)    (write an adapter)
+          RefCheckr    Redacta    Patiently AI   PubCrawl    your system
+         (QA, first) (redaction)  (simplify)   (retrieval)  (write one)
 ```
 
 Where it sits in the development loop:
@@ -109,6 +109,8 @@ change a prompt, model, or pipeline
 | `claim-extraction` | online | precision / recall / F1 vs gold; non-claim leakage; citation agreement; **fidelity** (extracted claim is verbatim from source) |
 | `verdict-accuracy` | online | exact & adjacency accuracy on a six-point support scale; confusion matrix; **passage hallucination rate**; consistency across repeats; per-claim latency (p50/p95) and token usage for real cost/claim |
 | `redaction` | online | recall on gold identifiers with **leaks as named failures** (verbatim, and word-level for names); over-redaction count; known-gap tracking for documented engine gaps |
+| `simplification` | online | faithfulness of rewritten text: **anchor recall** (critical facts like doses must survive), **fabricated numbers** (nothing invented), length-contract gates, readability grade (info) |
+| `retrieval` | online | fidelity of retrieved records vs the authority: field presence, hand-verified **anchor fields** (author surnames, year, distinctive abstract phrases), and structural invariants that catch parser regressions (collapsed author arrays, `[object Object]` leakage) |
 
 Offline scorers run with no API key — fast enough for every commit. Online scorers exercise a live system through an adapter.
 
@@ -171,6 +173,26 @@ node src/runner.mjs --online --adapter ./src/adapters/redacta.mjs
 
 On its first run against the new gold set, the eval found two real engine bugs — relation phrases like "Next of kin:" swallowed nested name matches, and apostrophe surnames (O'Brien) were dropped from name capture. Both were fixed in `@pharmatools/redacta` 1.2.1 and confirmed by the eval (`knownGap_closed: 2`), then promoted to gold. Street-line address detection followed in 1.3.0, closing the last tracked gap. Current scorecard: **100% recall on 25 gold identifiers, 0 leaks, no open gaps**.
 
+## Third implementation: Patiently AI
+
+[Patiently AI](https://www.pharmatools.ai/patiently-ai) exercises the **simplify capability** — faithfulness scoring for text that is paraphrase by design. On its first production run, the eval found the simplifier **dropping safety-critical specifics**: an antibiotic dose vanished from a Brief discharge summary and a haemoglobin value from a lab letter (anchor recall 86%). Root cause: the composed prompt had no faithfulness rule. A preservation rule added to Patiently's tone prompts (additively — the backend is shared with another product) took the next run to **100% anchor recall, 0 dropped facts, 0 fabricated numbers, 0 contract violations** — with the readability grade slightly *better* than before the fix.
+
+```bash
+node src/runner.mjs --online --adapter ./src/adapters/patiently.mjs
+```
+
+## Fourth implementation: PubCrawl
+
+[PubCrawl](https://www.pharmatools.ai/pubcrawl) is the odd one out — an MCP server wrapping PubMed and ClinicalTrials.gov, with **no model**. It exercises the **retrieval capability**: the deterministic layer everything else grounds on. A silent XML-parser regression (a collapsed author array, a merged abstract) would poison every citation built on the record, so the scorer checks retrieval fidelity against hand-verified anchors and structural invariants. The adapter drives PubCrawl through its real MCP interface, so the full production parse path is under test — and `scripts/capture-retrieval-case.mjs` bootstraps gold cases from live records for you to verify against the source.
+
+```bash
+npm install --no-save @modelcontextprotocol/sdk
+node scripts/capture-retrieval-case.mjs 31904519 > datasets/cases/retrieval-example.json  # then verify anchors
+node src/runner.mjs --online --adapter ./src/adapters/pubcrawl.mjs
+```
+
+That OpenGATE scores a non-AI system at all is the point: **evidence-grounded AI is only as trustworthy as the retrieval beneath it**, so the retrieval belongs in the same regression gate.
+
 ## Layout
 
 ```
@@ -192,7 +214,7 @@ opengate/
 ## Roadmap
 
 
-- **Third adapter** — Patiently AI (faithfulness evaluation for patient-language simplification)
+- **Per-adapter baselines** — `results/baseline.json` is a single reference; runs against different adapters (RefCheckr QA vs Patiently simplification) should baseline separately (workable today via `--results <dir>`, first-class support planned)
 - **Author-year in RefCheckr production** — `detectAuthorYear()` now lands "Smith 2020"-style keys in the reference implementation; adopting them in RefCheckr's numeric-keyed citation mapping is tracked separately
 - **Number-adjacent superscript** — `week 24.1` is genuinely ambiguous with decimals; remains a tracked known gap
 - **Growing gold set** — more domains, all six verdict types, real-world reference material

package/datasets/SCHEMA.md

@@ -7,7 +7,7 @@ One JSON file per case in `datasets/cases/` (files starting with `_` are ignored
 | `id` | yes | all | Unique slug. |
 | `title` / `notes` | no | — | Human context / provenance. |
 | `manuscript` | yes | claim-extraction | The pasted section, with citation markers exactly as authored. |
-| `goldClaims[]` | yes | citation-detection, claim-extraction | The verifiable claims a reviewer should extract. Each has `originalText` (with markers), `text` (clean), and `citations` (number array). |
+| `goldClaims[]` | yes | citation-detection, claim-extraction | The verifiable claims a reviewer should extract. Each has `originalText` (with markers), `text` (clean), and `citations` — an array of numbers and/or author-year string keys (e.g. "Smith 2020", "Meyer 2020a"). Numeric [N] markers are stripped from `text`; author-year mentions are grammatical prose and are never stripped, so for pure author-year claims `text === originalText`. |
 | `goldNonClaims[]` | no | claim-extraction | Sentences that should **not** be extracted (background, aims, transitions). Drives precision/leakage. |
 | `references{}` | online only | verdict-accuracy | Map of citation-number → `{ name, text }`. |
 | `goldVerdicts[]` | online only | verdict-accuracy | `{ claimText, citation, verdict }` where `verdict` ∈ the six-point scale. Mark `_requires: "online"`. |
@@ -28,3 +28,30 @@ Cases for the `redaction` scorer, exercising an adapter's `redact()` capability.
 | `text` | yes | The source clinical note. |
 | `goldEntities[]` | yes | `{ type, value }` — identifiers that MUST be removed. Any that survive (verbatim, or word-level for `*_NAME` types) are **leaks** and fail the run. |
 | `knownGapEntities[]` | no | `{ type, value, comment }` — identifiers the system does not yet catch. Reported as tracked targets (`knownGap_open` / `knownGap_closed`), not failures. Use `comment` to document the reproduction. When a gap closes, promote the entity to `goldEntities`. |
+
+## Simplification cases (`kind: "simplification"`)
+
+Cases for the `simplification` scorer, exercising an adapter's `simplify()` capability. Simplified text is paraphrase by design, so the scorer checks what is deterministic and clinically dangerous to get wrong, not verbatim prose.
+
+| Field | Required | Meaning |
+|---|---|---|
+| `id` / `kind` | yes | `kind` must be `"simplification"`. |
+| `text` | yes | The source clinical text (synthetic). |
+| `audience` / `tone` / `length` | no | Passed to the adapter (defaults: Adult / Informative / Standard). |
+| `anchors[]` | yes | `{ value, aliases? }` — critical facts that MUST survive simplification (drug names, doses, key values, timeframes). Matched case-insensitively, whitespace-tolerant. A missing anchor is a **dropped fact** and fails the run. |
+| `allowedNewNumbers[]` | no | Numbers legitimately introduced by rephrasing (e.g. "twice daily" → "2 times a day"). Any other output number absent from the source is a **fabricated number** and fails the run. |
+| `maxBullets` / `maxWordsPerBullet` | no | Length-contract gates (e.g. Patiently Brief: 3 / 20). Only checked when present. |
+
+## Retrieval cases (`kind: "retrieval"`)
+
+Cases for the `retrieval` scorer, exercising an adapter's `fetchRecord()` capability against a deterministic retrieval system (e.g. PubCrawl). The failure that matters is a parser regression: a field dropped, collapsed, or garbled. Anchors are **independent ground truth** — copied from the source record (the PubMed page, the paper), never from the system's own output. Bootstrap with `scripts/capture-retrieval-case.mjs`, then verify.
+
+| Field | Required | Meaning |
+|---|---|---|
+| `id` / `kind` | yes | `kind` must be `"retrieval"`. |
+| `recordId` | yes | The stable identifier to fetch (PMID, NCT id). |
+| `recordType` | no | `pubmed` (default) or `trial`. |
+| `requireFields[]` | no | Field names that must be present and non-empty in the record (e.g. `title`, `authors`, `year`). |
+| `anchors[]` | no | Per-field ground-truth checks, one of: `{ field, contains }` (verbatim substring survived), `{ field, equals }` (exact value), `{ field, minCount }` (array didn't collapse). At least one of `requireFields`/`anchors` is needed. |
+
+The scorer also always applies structural invariants: `authors` must be a non-empty array of strings (not a collapsed single-author object), `title` a non-empty string, and no field may serialise to `"[object Object]"`.

package/datasets/cases/_retrieval-template.json

@@ -0,0 +1,14 @@
+{
+  "id": "retrieval-EXAMPLE",
+  "kind": "retrieval",
+  "_note": "Template (ignored by the runner — filename starts with _). Copy to a real filename and populate from scripts/capture-retrieval-case.mjs, then VERIFY every anchor against the source paper. Anchors are independent ground truth — copied from the paper, never from PubCrawl's own output.",
+  "recordId": "PMID or NCT id here",
+  "recordType": "pubmed",
+  "requireFields": ["title", "authors", "year"],
+  "anchors": [
+    { "field": "authors", "contains": "FirstAuthorSurname" },
+    { "field": "authors", "minCount": 5 },
+    { "field": "year", "equals": "2020" },
+    { "field": "abstract_sections", "contains": "a distinctive verbatim phrase from the abstract" }
+  ]
+}

package/datasets/cases/case-authoryear-harvard.json

@@ -0,0 +1,56 @@
+{
+  "id": "case-authoryear-harvard",
+  "title": "Author-year (Harvard) citation style — thyroid eye disease",
+  "notes": "SYNTHETIC manuscript exercising pure author-year citations. Unlike numeric [N] markers, author-year mentions are grammatical prose and are never stripped: text === originalText.",
+  "manuscript": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020). Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension. The safety profile was consistent with earlier findings (Kim, 2023). This study aimed to characterise long-term outcomes in routine practice.",
+  "goldClaims": [
+    {
+      "originalText": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020).",
+      "text": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020).",
+      "citations": [
+        "Douglas 2020"
+      ]
+    },
+    {
+      "originalText": "Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension.",
+      "text": "Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension.",
+      "citations": [
+        "Smith 2019"
+      ]
+    },
+    {
+      "originalText": "The safety profile was consistent with earlier findings (Kim, 2023).",
+      "text": "The safety profile was consistent with earlier findings (Kim, 2023).",
+      "citations": [
+        "Kim 2023"
+      ]
+    }
+  ],
+  "goldNonClaims": [
+    "This study aimed to characterise long-term outcomes in routine practice."
+  ],
+  "references": {
+    "Douglas 2020": {
+      "name": "douglas-2020-nejm.txt",
+      "text": "In this randomised trial of teprotumumab for thyroid eye disease, the least-squares mean reduction in proptosis was 2.82 mm greater with teprotumumab than with placebo at week 24."
+    },
+    "Smith 2019": {
+      "name": "smith-2019-extension.txt",
+      "text": "In the 52-week open-label extension, proptosis responses were maintained in the majority of participants, indicating durable treatment effect."
+    }
+  },
+  "goldVerdicts": [
+    {
+      "_requires": "online",
+      "claimText": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020).",
+      "citation": "Douglas 2020",
+      "verdict": "strong_support"
+    },
+    {
+      "_requires": "online",
+      "claimText": "Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension.",
+      "citation": "Smith 2019",
+      "verdict": "strong_support"
+    }
+  ]
+}

package/datasets/cases/case-authoryear-mixed.json

@@ -0,0 +1,48 @@
+{
+  "id": "case-authoryear-mixed",
+  "title": "Mixed numeric + author-year citations in one manuscript",
+  "notes": "SYNTHETIC. Exercises numeric superscript and author-year styles side by side, an apostrophe surname (O'Connor), and a 2020a/2020b suffix pair that must stay distinct.",
+  "manuscript": "Adalimumab improved ACR20 response rates versus placebo.1,2 O'Connor et al. (2021) confirmed the effect in a real-world cohort. Retention was higher with combination therapy (Meyer 2020a; Meyer 2020b). Baseline characteristics are summarised in Table 1.",
+  "goldClaims": [
+    {
+      "originalText": "Adalimumab improved ACR20 response rates versus placebo.1,2",
+      "text": "Adalimumab improved ACR20 response rates versus placebo.",
+      "citations": [
+        1,
+        2
+      ]
+    },
+    {
+      "originalText": "O'Connor et al. (2021) confirmed the effect in a real-world cohort.",
+      "text": "O'Connor et al. (2021) confirmed the effect in a real-world cohort.",
+      "citations": [
+        "O'Connor 2021"
+      ]
+    },
+    {
+      "originalText": "Retention was higher with combination therapy (Meyer 2020a; Meyer 2020b).",
+      "text": "Retention was higher with combination therapy (Meyer 2020a; Meyer 2020b).",
+      "citations": [
+        "Meyer 2020a",
+        "Meyer 2020b"
+      ]
+    }
+  ],
+  "goldNonClaims": [
+    "Baseline characteristics are summarised in Table 1."
+  ],
+  "references": {
+    "O'Connor 2021": {
+      "name": "oconnor-2021-cohort.txt",
+      "text": "In this prospective real-world cohort of 412 patients with rheumatoid arthritis, adalimumab was associated with significantly improved ACR20 response rates compared with historical placebo controls."
+    }
+  },
+  "goldVerdicts": [
+    {
+      "_requires": "online",
+      "claimText": "O'Connor et al. (2021) confirmed the effect in a real-world cohort.",
+      "citation": "O'Connor 2021",
+      "verdict": "strong_support"
+    }
+  ]
+}

package/datasets/cases/retrieval-macra-vascular.json

@@ -0,0 +1,15 @@
+{
+  "id": "retrieval-macra-vascular",
+  "kind": "retrieval",
+  "notes": "PMID 31904519 — Ann Vasc Surg 2020. A four-author paper: anchors are chosen to fail exactly where PubCrawl's known-risky XML paths would regress — author-array collapse (minCount), surname parsing (contains, first token of PubCrawl's 'Surname ForeName' format), and abstract-section truncation (distinctive opening phrase). Verified against the PubMed record.",
+  "recordId": "31904519",
+  "recordType": "pubmed",
+  "requireFields": ["title", "authors", "year"],
+  "anchors": [
+    { "field": "authors", "contains": "Haurani" },
+    { "field": "authors", "contains": "Satiani" },
+    { "field": "authors", "minCount": 4 },
+    { "field": "year", "equals": "2020" },
+    { "field": "abstract_sections", "contains": "The Medicare Access and CHIP Reauthorization Act" }
+  ]
+}

package/datasets/cases/simplify-discharge-brief.json

@@ -0,0 +1,45 @@
+{
+  "id": "simplify-discharge-brief",
+  "kind": "simplification",
+  "description": "SYNTHETIC discharge note. Brief mode exercises Patiently's documented contract: 2-3 bullets, under 20 words each. Anchors are the clinically critical facts a patient must not lose.",
+  "audience": "Adult",
+  "tone": "Informative",
+  "length": "Brief",
+  "text": "Discharge summary: You were admitted with community-acquired pneumonia. You have been started on amoxicillin 500 mg three times daily for 5 days. Please see your GP in 2 weeks for review. Return to hospital if you develop worsening breathlessness or a fever above 38 degrees.",
+  "anchors": [
+    {
+      "value": "amoxicillin"
+    },
+    {
+      "value": "500 mg",
+      "aliases": [
+        "500mg",
+        "500 milligrams"
+      ]
+    },
+    {
+      "value": "5 days",
+      "aliases": [
+        "five days"
+      ]
+    },
+    {
+      "value": "2 weeks",
+      "aliases": [
+        "two weeks"
+      ]
+    },
+    {
+      "value": "38",
+      "aliases": [
+        "38C",
+        "38 degrees"
+      ]
+    }
+  ],
+  "allowedNewNumbers": [
+    "3"
+  ],
+  "maxBullets": 3,
+  "maxWordsPerBullet": 20
+}

package/datasets/cases/simplify-lab-results.json

@@ -0,0 +1,41 @@
+{
+  "id": "simplify-lab-results",
+  "kind": "simplification",
+  "description": "SYNTHETIC lab letter. Standard length, Reassuring tone. Values and the new medication must survive; no invented numbers.",
+  "audience": "Adult",
+  "tone": "Reassuring",
+  "length": "Standard",
+  "text": "Blood test results: your haemoglobin was 9.8 g/dL (below the normal range) and your ferritin was 8 micrograms/L (low), consistent with iron-deficiency anaemia. We recommend starting ferrous sulfate 200 mg twice daily. We will repeat the blood count in 8 weeks.",
+  "anchors": [
+    {
+      "value": "9.8"
+    },
+    {
+      "value": "ferritin"
+    },
+    {
+      "value": "ferrous sulfate",
+      "aliases": [
+        "ferrous sulphate",
+        "iron tablets",
+        "iron supplement"
+      ]
+    },
+    {
+      "value": "200 mg",
+      "aliases": [
+        "200mg",
+        "200 milligrams"
+      ]
+    },
+    {
+      "value": "8 weeks",
+      "aliases": [
+        "eight weeks"
+      ]
+    }
+  ],
+  "allowedNewNumbers": [
+    "2"
+  ]
+}

package/datasets/cases/simplify-medication-change.json

@@ -0,0 +1,34 @@
+{
+  "id": "simplify-medication-change",
+  "kind": "simplification",
+  "description": "SYNTHETIC dose-change letter. The new dose is the fact that must never be lost or altered.",
+  "audience": "Adult",
+  "tone": "Informative",
+  "length": "Standard",
+  "text": "Your levothyroxine dose has been increased from 50 micrograms to 100 micrograms once daily, following your recent TSH result of 8.2 mU/L. Please take the new dose each morning before breakfast. We will repeat your thyroid function tests in 6 to 8 weeks.",
+  "anchors": [
+    {
+      "value": "levothyroxine"
+    },
+    {
+      "value": "100 micrograms",
+      "aliases": [
+        "100 mcg",
+        "100mcg",
+        "100 µg"
+      ]
+    },
+    {
+      "value": "morning"
+    },
+    {
+      "value": "6 to 8 weeks",
+      "aliases": [
+        "6-8 weeks",
+        "six to eight weeks",
+        "6–8 weeks"
+      ]
+    }
+  ],
+  "allowedNewNumbers": []
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pharmatools/opengate",
-  "version": "0.2.1",
+  "version": "0.4.0",
   "type": "module",
   "description": "OpenGATE — Open Grounded AI Testing & Evaluation. An open-source framework for evaluating evidence-grounded AI systems.",
   "license": "MIT",

package/src/adapters/patiently.mjs

@@ -0,0 +1,64 @@
+// Patiently AI adapter — third bundled implementation, exercising the
+// framework's simplify capability. Patiently AI (getpatiently.ai) converts
+// clinical text into patient-friendly language via a Firebase Cloud Function.
+//
+// Config via env (the endpoint is public; no token needed):
+//   PATIENTLY_API_URL     override the translate endpoint
+//   PATIENTLY_EVAL_MODEL  optional label recorded in the scorecard
+//
+// Request contract: POST { text, language, audience, tone, length }
+// Response contract: { optimisedText }
+// Patiently-flavoured keys: audience Child|Teenager|Adult|Carer,
+// tone Friendly|Reassuring|Informative, length Brief|Standard|Detailed.
+
+export const meta = { name: 'patiently' };
+
+const DEFAULT_URL = 'https://us-central1-medicaltextoptimiser.cloudfunctions.net/translate';
+const URL = process.env.PATIENTLY_API_URL || DEFAULT_URL;
+
+export function onlineAvailable() {
+  return Boolean(URL);
+}
+
+export function onlineConfigHint() {
+  return 'Set PATIENTLY_API_URL (or unset it to use the production endpoint).';
+}
+
+export function runModel() {
+  return process.env.PATIENTLY_EVAL_MODEL || null;
+}
+
+// ── Timing capture ──────────────────────────────────────────────────────
+const _calls = [];
+export function resetTiming() { _calls.length = 0; }
+export function callLatencies() { return _calls.map(c => c.ms); }
+
+/**
+ * Simplify capability.
+ * @param {object} req — { text, audience?, tone?, length?, language? }
+ * @returns {Promise<{ text: string }>}
+ */
+export async function simplify(req) {
+  const t0 = performance.now();
+  try {
+    const res = await fetch(URL, {
+      method: 'POST',
+      headers: { 'Content-Type': 'application/json' },
+      body: JSON.stringify({
+        text: req.text,
+        language: req.language || 'en',
+        audience: req.audience || 'Adult',
+        tone: req.tone || 'Informative',
+        length: req.length || 'Standard',
+      }),
+    });
+    const data = await res.json().catch(() => ({}));
+    if (!res.ok) throw new Error(data.error || `translate → HTTP ${res.status}`);
+    if (typeof data.optimisedText !== 'string') {
+      throw new Error('translate response missing optimisedText');
+    }
+    return { text: data.optimisedText };
+  } finally {
+    _calls.push({ ms: performance.now() - t0 });
+  }
+}

package/src/adapters/pubcrawl.mjs

@@ -0,0 +1,107 @@
+// PubCrawl adapter — fourth bundled implementation, exercising the framework's
+// retrieval capability. PubCrawl (@pharmatools/pubcrawl) is an MCP server that
+// gives AI clients access to PubMed, ClinicalTrials.gov, and drug labelling.
+//
+// It is NOT an AI system — it's deterministic wrappers around public APIs. The
+// eval measures RETRIEVAL FIDELITY: does the record a client receives match the
+// authority (right title, all authors, intact abstract)? A silent XML-parser
+// regression here would poison every downstream grounding claim, so this is the
+// foundation the QA/simplify capabilities build on.
+//
+// The adapter talks to PubCrawl through its real MCP interface (no
+// re-implementation of the tool handlers), so the full production parse path is
+// under test. The MCP SDK is imported dynamically — install it alongside
+// PubCrawl; OpenGATE core stays zero-dependency.
+//
+// Config via env:
+//   PUBCRAWL_MCP_URL   HTTP MCP endpoint, e.g. http://localhost:8080/mcp
+//                      (start PubCrawl with `npm run start:http`)
+//   PUBCRAWL_CMD       stdio spawn command (default: "npx")
+//   PUBCRAWL_ARGS      stdio args, space-separated (default: "-y @pharmatools/pubcrawl")
+//   NCBI_API_KEY       forwarded to the server (higher NCBI rate limit)
+// If PUBCRAWL_MCP_URL is set it wins; otherwise the server is spawned over stdio.
+
+export const meta = { name: 'pubcrawl' };
+
+let sdk = null;
+let loadError = null;
+try {
+  const [{ Client }, stdio, http] = await Promise.all([
+    import('@modelcontextprotocol/sdk/client/index.js'),
+    import('@modelcontextprotocol/sdk/client/stdio.js'),
+    import('@modelcontextprotocol/sdk/client/streamableHttp.js').catch(() => ({})),
+  ]);
+  sdk = { Client, StdioClientTransport: stdio.StdioClientTransport, StreamableHTTPClientTransport: http.StreamableHTTPClientTransport };
+} catch (err) {
+  loadError = err;
+}
+
+const HTTP_URL = process.env.PUBCRAWL_MCP_URL;
+
+export function onlineAvailable() {
+  return Boolean(sdk);
+}
+
+export function onlineConfigHint() {
+  if (!sdk) {
+    return 'MCP SDK not installed — run: npm install --no-save @modelcontextprotocol/sdk' +
+      (loadError ? ` (${loadError.code || loadError.message})` : '');
+  }
+  return 'PubCrawl adapter ready (set PUBCRAWL_MCP_URL for HTTP, or it spawns the server over stdio).';
+}
+
+// ── Timing ──────────────────────────────────────────────────────────────
+const _calls = [];
+export function resetTiming() { _calls.length = 0; }
+export function callLatencies() { return _calls.map(c => c.ms); }
+
+function newTransport() {
+  if (HTTP_URL) {
+    if (!sdk.StreamableHTTPClientTransport) throw new Error('HTTP transport unavailable in this SDK build');
+    return new sdk.StreamableHTTPClientTransport(new URL(HTTP_URL));
+  }
+  const command = process.env.PUBCRAWL_CMD || 'npx';
+  const args = (process.env.PUBCRAWL_ARGS || '-y @pharmatools/pubcrawl').split(/\s+/).filter(Boolean);
+  const env = { ...process.env };
+  return new sdk.StdioClientTransport({ command, args, env });
+}
+
+// PubCrawl tool per record type. Extend as retrieval gold cases grow.
+const TOOL_FOR = {
+  pubmed: (id) => ({ name: 'get_abstract', arguments: { pmid: String(id) } }),
+  trial: (id) => ({ name: 'get_trial', arguments: { nctId: String(id) } }),
+};
+
+/**
+ * Retrieval capability. Fetches one record through PubCrawl's MCP interface and
+ * returns its parsed JSON as { record }.
+ * @param {object} req — { id, type? }  (type default: 'pubmed')
+ */
+export async function fetchRecord(req) {
+  const type = req.type || 'pubmed';
+  const build = TOOL_FOR[type];
+  if (!build) throw new Error(`unsupported record type "${type}"`);
+
+  const t0 = performance.now();
+  const client = new sdk.Client({ name: 'opengate', version: '0' }, { capabilities: {} });
+  const transport = newTransport();
+  try {
+    await client.connect(transport);
+    const res = await client.callTool(build(req.id));
+    if (res.isError) {
+      const msg = res.content?.map(c => c.text).join(' ') || 'tool error';
+      throw new Error(msg);
+    }
+    const text = (res.content || []).find(c => c.type === 'text')?.text ?? '';
+    let record;
+    try {
+      record = JSON.parse(text);
+    } catch {
+      throw new Error(`non-JSON tool response: ${text.slice(0, 120)}`);
+    }
+    return { record };
+  } finally {
+    _calls.push({ ms: performance.now() - t0 });
+    await client.close().catch(() => {});
+  }
+}

package/src/lib/adapter.mjs

@@ -24,6 +24,8 @@ const REQUIRED_BASE = ['onlineAvailable', 'onlineConfigHint'];
 const CAPABILITIES = {
   qa: ['splitClaims', 'analyzeBatch'],       // claim extraction + verdicts against references
   redaction: ['redact'],                      // identifier removal from text
+  simplify: ['simplify'],                     // faithful simplification of source text
+  retrieval: ['fetchRecord'],                 // fidelity of retrieved records vs the authority
 };
 
 // Optional: validated if present; no-op defaults are supplied if absent, so

package/src/lib/citations.mjs

@@ -69,7 +69,9 @@ const ETAL = String.raw`(?:et al\.?|and colleagues|and coworkers)`;
 /** Detect author-year citations in raw text; returns sorted unique keys. */
 export function detectAuthorYear(text) {
   const found = new Set();
-  const add = (name, year) => found.add(`${name} ${year.slice(0, 4)}`);
+  // Year suffixes are kept: "Smith 2020a" and "Smith 2020b" are distinct
+  // references in author-year styles.
+  const add = (name, year) => found.add(`${name} ${year}`);
 
   // 1. Narrative with parenthetical year: "Smith et al. (2020)",
   //    "Smith and Jones (2019)", "Smith (2020)".

package/src/runner.mjs

@@ -99,6 +99,8 @@ const SCORERS = [
   './scorers/claim-extraction.mjs',
   './scorers/verdict-accuracy.mjs',
   './scorers/redaction.mjs',
+  './scorers/simplification.mjs',
+  './scorers/retrieval.mjs',
 ];
 
 async function loadJsonDir(dir, { skipPrefix } = {}) {

package/src/scorers/retrieval.mjs

@@ -0,0 +1,143 @@
+// ONLINE scorer — retrieval fidelity.
+//
+// Exercises the adapter's fetchRecord() capability against gold cases of kind
+// "retrieval": a stable record ID plus a few HAND-VERIFIED anchor fields.
+// Unlike the model-scoring scorers, the system under test is deterministic
+// (PubCrawl wraps NCBI / ClinicalTrials.gov). The failure that matters here is
+// a PARSE REGRESSION: a record comes back but a field is dropped, collapsed,
+// or garbled — e.g. an authors array that collapses to a single object when a
+// paper has one author, or an abstract whose sections merge or truncate.
+// Because everything downstream (RefCheckr, any RAG) grounds on these records,
+// a silent parse bug poisons every citation built on it.
+//
+// Ground truth is independent of the system: anchors are copied from the
+// paper/record itself, not from PubCrawl's output (see datasets/SCHEMA.md and
+// scripts/capture-retrieval-case.mjs to bootstrap them). The scorer checks:
+//
+//   • field presence (gate) — every field named in `requireFields` is present
+//     and non-empty (title, authors, year, …). Catches dropped/blank fields.
+//   • anchor fidelity (gate) — each anchor must be satisfied by the record:
+//       { field: "authors",  contains: "Douglas" }   surname survived parsing
+//       { field: "year",     equals: "2020" }
+//       { field: "abstract", contains: "proptosis" } distinctive phrase intact
+//       { field: "authors",  minCount: 5 }            array didn't collapse
+//   • structural invariants (gate) — authors is a non-empty array of strings
+//     (not "[object Object]" from a collapsed single-author record); title is a
+//     non-empty string; no field serialises to "[object Object]".
+
+export const meta = { id: 'retrieval', mode: 'online' };
+
+const norm = (s) => String(s).toLowerCase().replace(/\s+/g, ' ').trim();
+
+/** Read a possibly-nested field from a record; arrays are joined for text ops. */
+function fieldValue(record, field) {
+  const v = record?.[field];
+  if (Array.isArray(v)) return v;
+  return v;
+}
+function asText(v) {
+  return Array.isArray(v) ? v.map(x => (typeof x === 'string' ? x : JSON.stringify(x))).join(' ') : String(v ?? '');
+}
+
+function checkAnchor(record, a) {
+  const v = fieldValue(record, a.field);
+  if (a.minCount != null) {
+    return Array.isArray(v) && v.length >= a.minCount
+      ? null : `${a.field} has ${Array.isArray(v) ? v.length : 0} items, expected ≥ ${a.minCount}`;
+  }
+  if (a.equals != null) {
+    return norm(asText(v)) === norm(a.equals) ? null : `${a.field} = "${asText(v)}", expected "${a.equals}"`;
+  }
+  if (a.contains != null) {
+    return norm(asText(v)).includes(norm(a.contains)) ? null : `${a.field} missing "${a.contains}"`;
+  }
+  return `anchor on ${a.field} has no check (equals/contains/minCount)`;
+}
+
+function structuralProblems(record) {
+  const problems = [];
+  const flat = JSON.stringify(record);
+  if (flat.includes('[object Object]')) problems.push('record contains "[object Object]" (a value failed to serialise — likely a parser shape bug)');
+  if ('authors' in record) {
+    const a = record.authors;
+    if (!Array.isArray(a)) problems.push('authors is not an array (single-author collapse?)');
+    else if (a.some(x => typeof x !== 'string' || !x.trim())) problems.push('authors contains a non-string / empty entry');
+  }
+  if ('title' in record && (typeof record.title !== 'string' || !record.title.trim())) {
+    problems.push('title is missing or not a non-empty string');
+  }
+  return problems;
+}
+
+export async function run({ cases, adapter }) {
+  if (!adapter.capabilities.retrieval) {
+    return { meta, skipped: true, reason: `adapter "${adapter.name}" has no retrieval capability` };
+  }
+  if (!adapter.onlineAvailable()) {
+    return { meta, skipped: true, reason: adapter.onlineConfigHint() };
+  }
+  const goldCases = cases.filter(c => c.kind === 'retrieval' && c.id && (c.anchors || c.requireFields));
+  if (goldCases.length === 0) {
+    return { meta, skipped: true, reason: 'No cases of kind "retrieval" with anchors/requireFields.' };
+  }
+
+  adapter.resetTiming();
+
+  const perCase = [];
+  const failures = [];
+
+  for (const c of goldCases) {
+    let record;
+    try {
+      const res = await adapter.fetchRecord({ id: c.recordId, type: c.recordType });
+      record = res?.record ?? res;
+    } catch (err) {
+      failures.push(`case ${c.id}: ${err.message}`);
+      continue;
+    }
+    if (!record || typeof record !== 'object') {
+      failures.push(`NO RECORD in ${c.id}: fetchRecord returned nothing usable`);
+      continue;
+    }
+
+    const caseFailures = [];
+    for (const f of c.requireFields || []) {
+      const v = fieldValue(record, f);
+      const empty = v == null || (Array.isArray(v) ? v.length === 0 : String(v).trim() === '');
+      if (empty) caseFailures.push(`missing field "${f}"`);
+    }
+    for (const a of c.anchors || []) {
+      const problem = checkAnchor(record, a);
+      if (problem) caseFailures.push(`anchor ${JSON.stringify(a)} — ${problem}`);
+    }
+    for (const s of structuralProblems(record)) caseFailures.push(s);
+
+    for (const cf of caseFailures) failures.push(`FIDELITY ${c.id}: ${cf}`);
+    perCase.push({
+      case: c.id, recordId: c.recordId,
+      anchors: (c.anchors || []).length,
+      requireFields: (c.requireFields || []).length,
+      problems: caseFailures,
+    });
+  }
+
+  const totalChecks = perCase.reduce((a, p) => a + p.anchors + p.requireFields, 0);
+  const failedChecks = perCase.reduce((a, p) => a + p.problems.length, 0);
+  const latencies = adapter.callLatencies();
+
+  const metrics = {
+    n_cases: perCase.length,
+    n_checks: totalChecks,
+    fidelity: round(totalChecks ? 1 - Math.min(failedChecks, totalChecks) / totalChecks : (perCase.length && !failedChecks ? 1 : 0)),
+    failed_checks: failedChecks,
+    ...(latencies.length ? { latency_p50_ms: Math.round(percentileOf(latencies, 50)) } : {}),
+  };
+
+  return { meta, metrics, detail: { perCase }, failures, passed: failures.length === 0 };
+}
+
+function percentileOf(values, p) {
+  const xs = values.filter(Number.isFinite).sort((a, b) => a - b);
+  return xs.length ? xs[Math.min(xs.length - 1, Math.floor((p / 100) * xs.length))] : 0;
+}
+function round(x) { return Math.round(x * 1000) / 1000; }

package/src/scorers/simplification.mjs

@@ -0,0 +1,163 @@
+// ONLINE scorer — simplification faithfulness.
+//
+// Exercises the adapter's simplify() capability against gold cases of kind
+// "simplification": a source clinical text plus the facts that must survive
+// simplification. Simplified text is paraphrase BY DESIGN, so verbatim checks
+// on prose are meaningless — instead the scorer measures what is
+// deterministically checkable and clinically dangerous to get wrong:
+//
+//   • anchor recall (gate) — critical facts (drug names, doses, key values)
+//     must appear in the output, matched case-insensitively with per-anchor
+//     aliases ("5 mg" ≈ "5mg"). A dropped dose is the failure that matters.
+//   • fabricated numbers (gate) — every number in the output must exist in
+//     the source, an anchor/alias, or the case's allowedNewNumbers list.
+//     An invented value in patient-facing text is the worst failure there is.
+//   • length contract (gate, when the case specifies) — e.g. Patiently's
+//     Brief mode: ≤3 bullets, ≤20 words per bullet. Contract drift here
+//     previously hid silent prompt-fallback bugs.
+//   • readability (info) — Flesch-Kincaid grade of the output; reported,
+//     not gated, in v1.
+//
+// Case schema (datasets/SCHEMA.md):
+//   { "id", "kind": "simplification", "text", "audience", "tone", "length",
+//     "anchors": [{ "value", "aliases": [] }],
+//     "allowedNewNumbers": ["2"], "maxBullets": 3, "maxWordsPerBullet": 20 }
+
+import { mean } from '../lib/metrics.mjs';
+
+export const meta = { id: 'simplification', mode: 'online' };
+
+const norm = (s) => String(s).toLowerCase().replace(/\s+/g, ' ');
+/** Whitespace-tolerant, case-insensitive containment. */
+function contains(haystack, needle) {
+  const h = norm(haystack).replace(/\s/g, '');
+  const n = norm(needle).replace(/\s/g, '');
+  return n.length > 0 && h.includes(n);
+}
+
+const NUM_RE = /\d+(?:\.\d+)?/g;
+const numbersIn = (s) => new Set((String(s).match(NUM_RE) || []).map(n => n.replace(/^0+(?=\d)/, '')));
+
+/** Lines that look like bullets: -, •, *, or "1." style. */
+function bulletLines(text) {
+  return String(text).split(/\r?\n/).map(l => l.trim())
+    .filter(l => /^([-•*]|\d+[.)])\s+/.test(l));
+}
+
+// Flesch-Kincaid grade with a vowel-group syllable heuristic. Approximate,
+// which is why readability is reported rather than gated.
+function fleschKincaidGrade(text) {
+  const words = String(text).toLowerCase().match(/[a-z]+/g) || [];
+  const sentences = Math.max(1, (String(text).match(/[.!?]+/g) || []).length);
+  if (!words.length) return null;
+  let syllables = 0;
+  for (const w of words) {
+    const groups = (w.replace(/e$/, '').match(/[aeiouy]+/g) || []).length;
+    syllables += Math.max(1, groups);
+  }
+  return 0.39 * (words.length / sentences) + 11.8 * (syllables / words.length) - 15.59;
+}
+
+export async function run({ cases, adapter }) {
+  if (!adapter.capabilities.simplify) {
+    return { meta, skipped: true, reason: `adapter "${adapter.name}" has no simplify capability` };
+  }
+  if (!adapter.onlineAvailable()) {
+    return { meta, skipped: true, reason: adapter.onlineConfigHint() };
+  }
+  const goldCases = cases.filter(c => c.kind === 'simplification' && (c.anchors || []).length);
+  if (goldCases.length === 0) {
+    return { meta, skipped: true, reason: 'No cases of kind "simplification" with anchors.' };
+  }
+
+  adapter.resetTiming();
+
+  const perCase = [];
+  const failures = [];
+
+  for (const c of goldCases) {
+    let out;
+    try {
+      const res = await adapter.simplify({
+        text: c.text, audience: c.audience, tone: c.tone, length: c.length, language: c.language,
+      });
+      out = res.text ?? '';
+    } catch (err) {
+      failures.push(`case ${c.id}: ${err.message}`);
+      continue;
+    }
+
+    // Anchor recall: value or any alias must survive.
+    const missed = (c.anchors || []).filter(a =>
+      ![a.value, ...(a.aliases || [])].some(v => contains(out, v)));
+    for (const a of missed) {
+      failures.push(`DROPPED FACT in ${c.id}: anchor "${a.value}" absent from simplified output`);
+    }
+
+    // Fabricated numbers: output numbers must come from somewhere legitimate.
+    const legitimate = new Set([
+      ...numbersIn(c.text),
+      ...(c.anchors || []).flatMap(a => [...numbersIn(a.value), ...(a.aliases || []).flatMap(x => [...numbersIn(x)])]),
+      ...(c.allowedNewNumbers || []).map(String),
+    ]);
+    const fabricated = [...numbersIn(out)].filter(n => !legitimate.has(n));
+    for (const n of fabricated) {
+      failures.push(`FABRICATED NUMBER in ${c.id}: "${n}" appears in output but not in source`);
+    }
+
+    // Length contract (only when the case declares one).
+    const bullets = bulletLines(out);
+    const contractViolations = [];
+    if (c.maxBullets != null && bullets.length > c.maxBullets) {
+      contractViolations.push(`${bullets.length} bullets > max ${c.maxBullets}`);
+    }
+    if (c.maxWordsPerBullet != null) {
+      for (const b of bullets) {
+        const words = b.replace(/^([-•*]|\d+[.)])\s+/, '').split(/\s+/).filter(Boolean).length;
+        if (words > c.maxWordsPerBullet) contractViolations.push(`bullet has ${words} words > max ${c.maxWordsPerBullet}`);
+      }
+    }
+    if (c.maxBullets != null && bullets.length === 0) {
+      contractViolations.push('bullet output expected, none found');
+    }
+    for (const v of contractViolations) failures.push(`CONTRACT in ${c.id}: ${v}`);
+
+    perCase.push({
+      case: c.id,
+      anchors: (c.anchors || []).length,
+      anchorsMissed: missed.map(a => a.value),
+      fabricated,
+      contractViolations,
+      grade: round(fleschKincaidGrade(out) ?? -1),
+      outputChars: out.length,
+      bullets: bullets.length,
+      // The output itself, so a dropped-fact failure can be diagnosed from the
+      // snapshot (was it omitted, reworded past the aliases, or replaced?).
+      output: out.slice(0, 600),
+    });
+  }
+
+  const totalAnchors = perCase.reduce((a, p) => a + p.anchors, 0);
+  const totalMissed = perCase.reduce((a, p) => a + p.anchorsMissed.length, 0);
+  const latencies = adapter.callLatencies();
+
+  const metrics = {
+    n_cases: perCase.length,
+    n_anchors: totalAnchors,
+    anchor_recall: round(totalAnchors ? 1 - totalMissed / totalAnchors : 0),
+    dropped_facts: totalMissed,
+    fabricated_numbers: perCase.reduce((a, p) => a + p.fabricated.length, 0),
+    contract_violations: perCase.reduce((a, p) => a + p.contractViolations.length, 0),
+    mean_grade: round(mean(perCase.map(p => p.grade).filter(g => g >= 0))),
+    ...(latencies.length ? { latency_p50_ms: Math.round(percentileOf(latencies, 50)) } : {}),
+    ...(adapter.runModel() ? { run_model: adapter.runModel() } : {}),
+  };
+
+  return { meta, metrics, detail: { perCase }, failures, passed: failures.length === 0 };
+}
+
+function percentileOf(values, p) {
+  const xs = values.filter(Number.isFinite).sort((a, b) => a - b);
+  return xs.length ? xs[Math.min(xs.length - 1, Math.floor((p / 100) * xs.length))] : 0;
+}
+function round(x) { return Math.round(x * 100) / 100; }