@pharmatools/opengate 0.2.0 → 0.3.0

package/README.md

@@ -124,7 +124,7 @@ Scorers never talk to a system directly — they go through an adapter, injected
 OPENGATE_ADAPTER=./adapters/my-system.mjs npm run eval:online
 ```
 
-An adapter is one file with four required exports — `splitClaims(text)`, `analyzeBatch(payload)`, `onlineAvailable()`, `onlineConfigHint()` — plus optional timing/token/model-label hooks that unlock latency and cost columns in the scorecard. Adapters are validated at load: a malformed one fails fast with a message listing every missing export.
+An adapter is one file: two base exports — `onlineAvailable()`, `onlineConfigHint()` — plus at least one complete **capability**: `qa` (`splitClaims` + `analyzeBatch`) or `redaction` (`redact`). Scorers check `adapter.capabilities` and skip cleanly across the boundary. Optional timing/token/model-label hooks unlock latency and cost columns in the scorecard. Adapters are validated at load: a malformed one fails fast with a message naming every missing export and incomplete capability.
 
 For REST-backed systems there's a no-code path: the bundled **generic HTTP adapter** (`src/adapters/http.mjs`) reads endpoint paths and headers from `opengate.http.json` (see `opengate.http.example.json`), with `${ENV}` interpolation and built-in latency/token capture.
 
@@ -137,7 +137,7 @@ Full contract, minimal skeleton, and verdict-mapping notes: **[ADAPTERS.md](ADAP
 Use OpenGATE as a drop-in regression gate in any repository. Keep your gold set and committed `baseline.json` in your own tree; any metric that drops fails the build:
 
 ```yaml
-- uses: nickjlamb/opengate@main
+- uses: nickjlamb/opengate@v0
   with:
     datasets: ./evals/datasets      # your cases/ + fixtures/
     results: ./evals/results        # where baseline.json lives
@@ -169,7 +169,7 @@ npm install --no-save @pharmatools/redacta
 node src/runner.mjs --online --adapter ./src/adapters/redacta.mjs
 ```
 
-On its first run against the new gold set, the eval found two real engine bugs — a relative name directly followed by a parenthesis escapes the name pattern, and apostrophe surnames are dropped from titled-name capture — both now documented as tracked known gaps in the case files. Current scorecard: **100% recall on 23 gold identifiers, 0 leaks, 3 tracked gaps**.
+On its first run against the new gold set, the eval found two real engine bugs — relation phrases like "Next of kin:" swallowed nested name matches, and apostrophe surnames (O'Brien) were dropped from name capture. Both were fixed in `@pharmatools/redacta` 1.2.1 and confirmed by the eval (`knownGap_closed: 2`), then promoted to gold. Street-line address detection followed in 1.3.0, closing the last tracked gap. Current scorecard: **100% recall on 25 gold identifiers, 0 leaks, no open gaps**.
 
 ## Layout
 
@@ -191,7 +191,7 @@ opengate/
 
 ## Roadmap
 
-- **Redacta engine fixes** — two bugs found by the redaction eval (paren-adjacent relative names; apostrophe surnames in titled-name capture) are tracked as known gaps; fixing them in `@pharmatools/redacta` flips `knownGap_closed` and promotes the cases to gold
+
 - **Third adapter** — Patiently AI (faithfulness evaluation for patient-language simplification)
 - **Author-year in RefCheckr production** — `detectAuthorYear()` now lands "Smith 2020"-style keys in the reference implementation; adopting them in RefCheckr's numeric-keyed citation mapping is tracked separately
 - **Number-adjacent superscript** — `week 24.1` is genuinely ambiguous with decimals; remains a tracked known gap

package/datasets/SCHEMA.md

@@ -7,7 +7,7 @@ One JSON file per case in `datasets/cases/` (files starting with `_` are ignored
 | `id` | yes | all | Unique slug. |
 | `title` / `notes` | no | — | Human context / provenance. |
 | `manuscript` | yes | claim-extraction | The pasted section, with citation markers exactly as authored. |
-| `goldClaims[]` | yes | citation-detection, claim-extraction | The verifiable claims a reviewer should extract. Each has `originalText` (with markers), `text` (clean), and `citations` (number array). |
+| `goldClaims[]` | yes | citation-detection, claim-extraction | The verifiable claims a reviewer should extract. Each has `originalText` (with markers), `text` (clean), and `citations` — an array of numbers and/or author-year string keys (e.g. "Smith 2020", "Meyer 2020a"). Numeric [N] markers are stripped from `text`; author-year mentions are grammatical prose and are never stripped, so for pure author-year claims `text === originalText`. |
 | `goldNonClaims[]` | no | claim-extraction | Sentences that should **not** be extracted (background, aims, transitions). Drives precision/leakage. |
 | `references{}` | online only | verdict-accuracy | Map of citation-number → `{ name, text }`. |
 | `goldVerdicts[]` | online only | verdict-accuracy | `{ claimText, citation, verdict }` where `verdict` ∈ the six-point scale. Mark `_requires: "online"`. |

package/datasets/cases/case-authoryear-harvard.json

@@ -0,0 +1,56 @@
+{
+  "id": "case-authoryear-harvard",
+  "title": "Author-year (Harvard) citation style — thyroid eye disease",
+  "notes": "SYNTHETIC manuscript exercising pure author-year citations. Unlike numeric [N] markers, author-year mentions are grammatical prose and are never stripped: text === originalText.",
+  "manuscript": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020). Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension. The safety profile was consistent with earlier findings (Kim, 2023). This study aimed to characterise long-term outcomes in routine practice.",
+  "goldClaims": [
+    {
+      "originalText": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020).",
+      "text": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020).",
+      "citations": [
+        "Douglas 2020"
+      ]
+    },
+    {
+      "originalText": "Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension.",
+      "text": "Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension.",
+      "citations": [
+        "Smith 2019"
+      ]
+    },
+    {
+      "originalText": "The safety profile was consistent with earlier findings (Kim, 2023).",
+      "text": "The safety profile was consistent with earlier findings (Kim, 2023).",
+      "citations": [
+        "Kim 2023"
+      ]
+    }
+  ],
+  "goldNonClaims": [
+    "This study aimed to characterise long-term outcomes in routine practice."
+  ],
+  "references": {
+    "Douglas 2020": {
+      "name": "douglas-2020-nejm.txt",
+      "text": "In this randomised trial of teprotumumab for thyroid eye disease, the least-squares mean reduction in proptosis was 2.82 mm greater with teprotumumab than with placebo at week 24."
+    },
+    "Smith 2019": {
+      "name": "smith-2019-extension.txt",
+      "text": "In the 52-week open-label extension, proptosis responses were maintained in the majority of participants, indicating durable treatment effect."
+    }
+  },
+  "goldVerdicts": [
+    {
+      "_requires": "online",
+      "claimText": "Teprotumumab reduced proptosis by 2.82 mm versus placebo (Douglas et al., 2020).",
+      "citation": "Douglas 2020",
+      "verdict": "strong_support"
+    },
+    {
+      "_requires": "online",
+      "claimText": "Smith and Jones (2019) reported durable responses at 52 weeks in an open-label extension.",
+      "citation": "Smith 2019",
+      "verdict": "strong_support"
+    }
+  ]
+}

package/datasets/cases/case-authoryear-mixed.json

@@ -0,0 +1,48 @@
+{
+  "id": "case-authoryear-mixed",
+  "title": "Mixed numeric + author-year citations in one manuscript",
+  "notes": "SYNTHETIC. Exercises numeric superscript and author-year styles side by side, an apostrophe surname (O'Connor), and a 2020a/2020b suffix pair that must stay distinct.",
+  "manuscript": "Adalimumab improved ACR20 response rates versus placebo.1,2 O'Connor et al. (2021) confirmed the effect in a real-world cohort. Retention was higher with combination therapy (Meyer 2020a; Meyer 2020b). Baseline characteristics are summarised in Table 1.",
+  "goldClaims": [
+    {
+      "originalText": "Adalimumab improved ACR20 response rates versus placebo.1,2",
+      "text": "Adalimumab improved ACR20 response rates versus placebo.",
+      "citations": [
+        1,
+        2
+      ]
+    },
+    {
+      "originalText": "O'Connor et al. (2021) confirmed the effect in a real-world cohort.",
+      "text": "O'Connor et al. (2021) confirmed the effect in a real-world cohort.",
+      "citations": [
+        "O'Connor 2021"
+      ]
+    },
+    {
+      "originalText": "Retention was higher with combination therapy (Meyer 2020a; Meyer 2020b).",
+      "text": "Retention was higher with combination therapy (Meyer 2020a; Meyer 2020b).",
+      "citations": [
+        "Meyer 2020a",
+        "Meyer 2020b"
+      ]
+    }
+  ],
+  "goldNonClaims": [
+    "Baseline characteristics are summarised in Table 1."
+  ],
+  "references": {
+    "O'Connor 2021": {
+      "name": "oconnor-2021-cohort.txt",
+      "text": "In this prospective real-world cohort of 412 patients with rheumatoid arthritis, adalimumab was associated with significantly improved ACR20 response rates compared with historical placebo controls."
+    }
+  },
+  "goldVerdicts": [
+    {
+      "_requires": "online",
+      "claimText": "O'Connor et al. (2021) confirmed the effect in a real-world cohort.",
+      "citation": "O'Connor 2021",
+      "verdict": "strong_support"
+    }
+  ]
+}

package/datasets/cases/redact-cardio-clinic.json

@@ -1,7 +1,7 @@
 {
   "id": "redact-cardio-clinic",
   "kind": "redaction",
-  "description": "Cardiology clinic letter — titled patient name, NHS number, DOB, phone, email, postcode. SYNTHETIC: all identifiers are test-range values; no real persons.",
+  "description": "Cardiology clinic letter — titled patient name, NHS number, DOB, phone, email, postcode. SYNTHETIC: all identifiers are test-range values; no real persons. The street address was a tracked engine gap; street-line detection landed in @pharmatools/redacta 1.3.0 and the entity was promoted to gold.",
   "text": "Mr John Smith (NHS 999 000 0018, DOB 12/03/1958) attended cardiology clinic today. He was seen by Dr Patel for review of atrial fibrillation. Blood pressure was 132/84 mmHg and apixaban was continued at 5 mg BD. Contact him on 07700 900123 or john.smith58@example.com. Address on file: 42 Maple Road, Oxford OX2 6JX. Follow-up in 12 weeks.",
   "goldEntities": [
     {
@@ -27,12 +27,11 @@
     {
       "type": "POSTCODE",
       "value": "OX2 6JX"
-    }
-  ],
-  "knownGapEntities": [
+    },
     {
       "type": "STREET_ADDRESS",
       "value": "42 Maple Road"
     }
-  ]
-}
+  ],
+  "knownGapEntities": []
+}

package/datasets/cases/redact-discharge-summary.json

@@ -1,7 +1,7 @@
 {
   "id": "redact-discharge-summary",
   "kind": "redaction",
-  "description": "Discharge summary with a relative as next of kin. SYNTHETIC.",
+  "description": "Discharge summary with a relative as next of kin. SYNTHETIC. The relative name was a tracked engine gap (a 'Next of kin:' phrase swallowed the inner 'daughter Anita' match — not the parenthesis, as first suspected); fixed in @pharmatools/redacta >1.2.0 and promoted to gold.",
   "text": "Discharge summary for Mrs Priya Sharma, NHS 999 000 0026, DOB 04/11/1947. Admitted with community-acquired pneumonia; completed 5 days of co-amoxiclav and improved steadily. Next of kin: her daughter Anita (07700 900456) was updated by phone. Discharged to home address, postcode LS8 2QT, with district nurse follow-up.",
   "goldEntities": [
     {
@@ -23,13 +23,11 @@
     {
       "type": "POSTCODE",
       "value": "LS8 2QT"
-    }
-  ],
-  "knownGapEntities": [
+    },
     {
       "type": "RELATIVE_NAME",
-      "value": "Anita",
-      "comment": "Engine bug (tracked): a relative name directly followed by an opening parenthesis escapes the relative-name pattern — 'her daughter Anita (07700...' leaks, while 'her daughter Anita was' is caught. The phone match also consumes the opening paren, leaving an orphan ')'."
+      "value": "Anita"
     }
-  ]
+  ],
+  "knownGapEntities": []
 }

package/datasets/cases/redact-ward-round.json

@@ -1,7 +1,7 @@
 {
   "id": "redact-ward-round",
   "kind": "redaction",
-  "description": "Ward round entry — apostrophe surname, NHS number without spaces, carer mention. SYNTHETIC.",
+  "description": "Ward round entry — apostrophe surname, NHS number without spaces, carer mention. SYNTHETIC. The apostrophe surname was a tracked engine gap (NAME pattern required a lowercase letter after the initial capital); fixed in @pharmatools/redacta >1.2.0.",
   "text": "Ward round: Mrs Eileen O'Brien, NHS 9990000034, remains stable post hip hemiarthroplasty. Mobilising with frame. Her son Daniel visited and raised concerns about home support; referred to social work. Plan: continue enoxaparin, orthogeriatric review tomorrow. Contact for family updates: 07700 900321.",
   "goldEntities": [
     {
@@ -21,11 +21,5 @@
       "value": "07700 900321"
     }
   ],
-  "knownGapEntities": [
-    {
-      "type": "PATIENT_SURNAME",
-      "value": "O'Brien",
-      "comment": "Engine bug (tracked): apostrophe surnames are dropped from titled-name capture — 'Mrs Eileen O'Brien' tokenises as 'Mrs Eileen' only, leaving the surname in the text."
-    }
-  ]
+  "knownGapEntities": []
 }

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@pharmatools/opengate",
-  "version": "0.2.0",
+  "version": "0.3.0",
   "type": "module",
   "description": "OpenGATE — Open Grounded AI Testing & Evaluation. An open-source framework for evaluating evidence-grounded AI systems.",
   "license": "MIT",

package/src/lib/citations.mjs

@@ -69,7 +69,9 @@ const ETAL = String.raw`(?:et al\.?|and colleagues|and coworkers)`;
 /** Detect author-year citations in raw text; returns sorted unique keys. */
 export function detectAuthorYear(text) {
   const found = new Set();
-  const add = (name, year) => found.add(`${name} ${year.slice(0, 4)}`);
+  // Year suffixes are kept: "Smith 2020a" and "Smith 2020b" are distinct
+  // references in author-year styles.
+  const add = (name, year) => found.add(`${name} ${year}`);
 
   // 1. Narrative with parenthetical year: "Smith et al. (2020)",
   //    "Smith and Jones (2019)", "Smith (2020)".