medsci-skills 5.1.0 → 5.2.0

package/metadata/distribution_files.json

@@ -2533,8 +2533,13 @@
     },
     {
       "path": "skills/mllm-eval/SKILL.md",
-      "size": 6288,
-      "sha256": "ccf3da2f70b356d432b3d33500f667f9d7858500a9ff325631a9a16f5f300a8b"
+      "size": 6720,
+      "sha256": "afdc167bfccb6bad8a4019e0af5bd7c955129220d360511a89962ecd321bc44d"
+    },
+    {
+      "path": "skills/mllm-eval/references/evaluation_axes.md",
+      "size": 10857,
+      "sha256": "49d77ab63feae5dba5cdae7e47d9de6ea97ee3b9eea4b9cba39ec8b0e185be72"
     },
     {
       "path": "skills/mllm-eval/scripts/check_mllm_eval_completeness.py",
@@ -2623,18 +2628,23 @@
     },
     {
       "path": "skills/model-evaluation/SKILL.md",
-      "size": 5031,
-      "sha256": "17ffde905359e4cffdf747422b7d41c214d2abfcc71e50e1b1e4a689d87fa695"
+      "size": 5780,
+      "sha256": "334b4ca87a2f672446563f3fb6d78c0585386fac12aee29e10dc09465e892193"
     },
     {
       "path": "skills/model-evaluation/references/metric_guide.md",
       "size": 2454,
       "sha256": "8d09ca7ce9fb9f66ee4942689294d9b12ae1d892ac67769cd68fdc38a4e220ee"
     },
+    {
+      "path": "skills/model-evaluation/references/metric_selection_grounding.md",
+      "size": 9588,
+      "sha256": "56723e73b2b74299140d353921d1dba471bedf9aea0b8e769fc69995c4733995"
+    },
     {
       "path": "skills/model-evaluation/scripts/check_metric_reporting.py",
-      "size": 9564,
-      "sha256": "c33f52ee62ae93417027d0bb0b6cf2a95f5747d52403ab99c075bc64a5e2c593"
+      "size": 9562,
+      "sha256": "fd6c0205f652651a5a43910cb415ed8442100fc553ce04aa5d5905d97743a0bf"
     },
     {
       "path": "skills/model-evaluation/scripts/metric_reporting_challenge/fixture/clf_bad.md",
@@ -2646,6 +2656,16 @@
       "size": 186,
       "sha256": "ba44a3b38b4128fa713555c8b332f211e9087e6bc016d80af4c3074c3ca6ef8e"
     },
+    {
+      "path": "skills/model-evaluation/scripts/metric_reporting_challenge/fixture/det_good_wrapped.md",
+      "size": 285,
+      "sha256": "0c1f5ed5167969870602106e7a7c9a2ae53e3a3ddea5fa280c3beb6bf470c454"
+    },
+    {
+      "path": "skills/model-evaluation/scripts/metric_reporting_challenge/fixture/det_no_iou.md",
+      "size": 224,
+      "sha256": "08227fbe46829b8eecdfe15680e1bb32087f1e6c5a353b3df6055c67d8300fc0"
+    },
     {
       "path": "skills/model-evaluation/scripts/metric_reporting_challenge/fixture/seg_bad.md",
       "size": 119,
@@ -2663,8 +2683,8 @@
     },
     {
       "path": "skills/model-evaluation/scripts/metric_reporting_challenge/verify.sh",
-      "size": 1235,
-      "sha256": "f819d1333a7206db6383ccbd41c6eadae015004e61ae335e67a3334936c58cc8"
+      "size": 1663,
+      "sha256": "4aa6db49d3552f01a54ddb70484df5d214806d805b836b24beda81d9aca32bd6"
     },
     {
       "path": "skills/model-evaluation/skill.yml",
@@ -2718,8 +2738,13 @@
     },
     {
       "path": "skills/model-validation/SKILL.md",
-      "size": 9347,
-      "sha256": "ecd48672a03923bf1ace63528fd2dbcf138cd880103cc8c40345b3857d66ad1c"
+      "size": 9810,
+      "sha256": "1a75a5a1f21f5a8778d0b77db2e99574bf37edda2a291d8dde6aafea4a207ff0"
+    },
+    {
+      "path": "skills/model-validation/references/validation_design.md",
+      "size": 11427,
+      "sha256": "16d43b688ea63745174c7ca8fafd78a7342b26c34ad1e10e1fdbc117cceafb2e"
     },
     {
       "path": "skills/model-validation/scripts/check_split_leakage.py",

package/metadata/distribution_manifest.json

@@ -1,6 +1,6 @@
 {
   "schema_version": 1,
-  "version": "5.1.0",
+  "version": "5.2.0",
   "owned_skills": [
     "academic-aio",
     "add-journal",

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "medsci-skills",
-  "version": "5.1.0",
+  "version": "5.2.0",
   "description": "MedSci Skills — a medical/scientific research skill suite for AI coding agents (Claude Code, Codex, Cursor, Copilot). The npm package is a terminal-friendly installer shortcut; the canonical distribution remains the GitHub repository and the Claude Code plugin marketplace.",
   "license": "SEE LICENSE IN LICENSE",
   "homepage": "https://github.com/Aperivue/medsci-skills#readme",

package/skills/mllm-eval/SKILL.md

@@ -106,3 +106,11 @@ mllm-eval (this skill: harness design + completeness gate, model-agnostic)
   ├─ write-paper + check-reporting (TRIPOD-LLM / MI-CLEAR-LLM)
   └─ self-review / peer-review (ME0–ME8 reviewer probe)
 ```
+
+## Reference Files
+
+- `${CLAUDE_SKILL_DIR}/references/evaluation_axes.md` — the *why* behind the ME2–ME7 axes:
+  clinical-efficacy metrics beyond n-gram overlap (e.g. RadGraph-F1 / CheXbert-F1 vs BLEU/ROUGE),
+  faithfulness & hallucination, pretraining/benchmark contamination, prompt-sensitivity &
+  determinism, answer-matching, and the reader study — each mapped to its gate verdict. Load on
+  demand during Phases 2–4.

package/skills/mllm-eval/references/evaluation_axes.md

@@ -0,0 +1,161 @@
+# Evaluation axes (mllm-eval)
+
+Load-on-demand reference behind the ME2–ME7 axes: the clinical-efficacy metrics,
+faithfulness, contamination, prompt-sensitivity, answer-matching, and reader-study
+machinery an LLM/MLLM clinical evaluation must cover. Anchored to the radiology-NLP
+metric literature — **RadGraph** (Jain et al., NeurIPS Datasets & Benchmarks 2021),
+**CheXbert** (Smit et al., 2020), the rule-based **CheXpert** labeler (Irvin et al., 2019),
+and **RadCliQ** (Yu et al., *Patterns* 2023) — and to the reporting standards CLAIM 2024,
+**TRIPOD-LLM**, and **MI-CLEAR-LLM**. This skill **specifies and routes** these metrics to
+their published extractors and to `/analyze-stats`; it does not run the model or compute the
+scores itself. Never report n-gram overlap as clinical correctness, and never fabricate a
+score.
+
+## Clinical-efficacy metrics beyond n-gram overlap (ME2 → `NGRAM_ONLY`)
+
+- **Why n-gram overlap fails.** BLEU / ROUGE / METEOR / CIDEr measure surface lexical
+  overlap. A report can score high while inverting laterality or omitting a pneumothorax, and
+  low while paraphrasing a correct finding. Yu et al. (*Patterns* 2023, RadCliQ) showed these
+  metrics correlate weakly with radiologist-assessed clinical error — so an n-gram score is a
+  **fluency proxy, not a correctness claim**.
+- **RadGraph-F1.** Overlap of the (entity, relation) tuples the RadGraph schema (Jain et al.,
+  2021) extracts from the reference and the candidate report — it rewards getting the *same
+  findings and their relationships*, not the same words.
+- **CheXbert-F1 / CheXpert labeler.** Agreement on the structured CheXpert observation labels
+  extracted by CheXbert (Smit et al., 2020) or by the rule-based CheXpert labeler (Irvin et
+  al., 2019) — a finding-label–level correctness signal.
+- **RadCliQ.** A composite (Yu et al., 2023) that combines metrics to better predict the count
+  of radiologist-judged errors. Report it as a composite **alongside** its components, not as a
+  one-number replacement.
+- **Advise:** report a clinical-efficacy metric (RadGraph-F1 / CheXbert-F1 / RadCliQ)
+  **with bootstrap CIs over reports** and a **per-finding / per-label breakdown**, and present
+  any BLEU/ROUGE explicitly labelled as a surface-overlap measure — never as the headline.
+
+## Faithfulness & hallucination (ME3 → `FAITHFULNESS_MISSING`)
+
+- **Fluency is not faithfulness.** A high-overlap, well-formed report can still assert findings
+  the image does not support. Measure faithfulness directly; do not infer it from an accuracy
+  number.
+- **Atomic-fact decomposition.** Break the generated text into atomic clinical claims and check
+  each against the image / source, then report a **faithfulness (or hallucination) rate** — the
+  fraction of generated claims that are grounded.
+- **Direction matters.** Separate **omission** (a true finding the model missed) from
+  **fabrication** (a false finding the model asserted); they carry different clinical risk and
+  should be reported separately, not folded into one error count.
+- **False-premise / abstention probe.** Ask about an absent finding or an unanswerable
+  question; a faithful model abstains rather than confabulates. Named instruments: **MedVH**,
+  **Med-HALT**.
+- **Advise:** a generation or VQA claim with no faithfulness and no false-premise/abstention
+  evaluation is the central MLLM gap — require both, with rates, before any clinical claim.
+
+## Pretraining / benchmark contamination (ME4 → `CONTAMINATION_UNADDRESSED`)
+
+- **Why public benchmarks are suspect.** VQA-RAD, SLAKE, MIMIC-CXR–derived sets, MedQA,
+  PMC-VQA, PathVQA, OpenI, and PubMedQA may sit inside the model's pretraining corpus, so a
+  high score can be **memorisation, not capability**. For a closed API the corpus is undisclosed,
+  so contamination cannot be excluded — only **bounded** and stated.
+- **Three accepted checks (any one, stated explicitly):**
+  - **Cutoff vs release date** — compare the model's training cutoff against the benchmark's
+    release date; a benchmark that predates the cutoff is at risk.
+  - **Held-out / post-cutoff set** — evaluate on a private, institution-collected, or
+    after-cutoff set the model could not have seen.
+  - **Contamination probe** — canary strings, a perturbed-duplicate performance gap (score on
+    verbatim items vs lightly perturbed copies), or a membership/quiz test (ask the model to
+    reproduce held-out items).
+- **Advise:** never write "no contamination" (or evaluate on a pre-cutoff public benchmark in
+  silence) without one of the checks above; an acknowledged-but-unmitigated risk is a stated
+  limitation, not a clean result.
+
+## Prompt-sensitivity & determinism (ME5 → `PROMPT_PROVENANCE_MISSING`)
+
+- **Outputs move with the prompt and the sampler.** Phrasing, format, the system prompt,
+  temperature, top-p/top-k, and run-to-run sampling all shift results. A single-prompt
+  single-run number overstates stability.
+- **Closed APIs are non-deterministic even at temperature 0** — identical inputs can yield
+  different outputs across calls. Treat any single-run figure as a point estimate of a
+  distribution, not a fixed value.
+- **Disclose (MI-CLEAR-LLM transparency):** the **exact prompt(s)** including the system prompt,
+  the **decoding settings** (temperature / top-p / seed), **≥ 3 runs** with reported variance
+  (e.g., mean ± SD), and a **prompt-robustness** check across **≥ 2 phrasings/formats** for the
+  headline result.
+
+## Answer-matching for VQA / classification (ME6 → `ANSWER_MATCHING_MISSING`)
+
+- **State the matching rule.** Free-text answers must be mapped to the key by a declared rule:
+  **exact** string match, **normalised** match (case/punctuation/synonym folding), or
+  **LLM-as-judge**. An unspecified rule makes the accuracy unreproducible.
+- **An LLM judge is itself a model.** If a model adjudicates correctness, **validate the judge
+  against a human-labelled subset** and report its agreement; route judge validation to
+  `/design-ai-benchmarking`. An unvalidated LLM judge can launder the system's own errors.
+- **Operating discipline.** Report accuracy **at the real clinical prevalence**, not on an
+  artificially balanced QA set, and state **how refusals/abstentions are scored** (counted
+  wrong, excluded, or credited) — the choice can move the headline.
+
+## Reader study for generated reports (ME7 → `READER_STUDY_MISSING`)
+
+- **Automated metrics do not establish clinical acceptability.** Even RadGraph-F1 / CheXbert-F1
+  measure agreement, not whether a clinician would act on the report safely. A deployment or
+  utility claim for generated text needs a **blinded clinical reader study**.
+- **Design elements:** a pre-defined **error taxonomy** (clinically significant vs insignificant;
+  omission vs fabrication), a **severity scale**, and **inter-reader agreement**.
+- **Route:** the rubric and the IRR design → `/design-ai-benchmarking`; the ICC/κ computation →
+  `/analyze-stats`; reader and case **sizing** → `/calc-sample-size`.
+
+## Gate mapping
+
+The deterministic gate (`scripts/check_mllm_eval_completeness.py`) is a presence check on the
+plan text, task-aware. This reference is the *why* behind each verdict:
+
+| Axis (this doc) | Gate verdict | Severity |
+|---|---|---|
+| n-gram only, no clinical metric (report-gen) | `NGRAM_ONLY` | Major |
+| no adjudicated reference standard (report-gen) | `REFERENCE_STANDARD_MISSING` | Major |
+| no faithfulness / false-premise (report-gen, vqa) | `FAITHFULNESS_MISSING` | Major |
+| public benchmark, no contamination handling | `CONTAMINATION_UNADDRESSED` | Major |
+| no blinded reader study (report-gen) | `READER_STUDY_MISSING` | Major (deploy) / Minor |
+| prompt / decoding / multi-run incomplete | `PROMPT_PROVENANCE_MISSING` | Minor |
+| no answer-matching rule (vqa, classification) | `ANSWER_MATCHING_MISSING` | Minor |
+
+A Major verdict is a presence gap, not proof the work is wrong — resolve it by adding the axis
+to the plan (or recording, with a stated reason, why it does not apply).
+
+## Reporting fit & hand-off
+
+Methods / Results stub → `/write-paper`. Item-level compliance with **TRIPOD-LLM**,
+**MI-CLEAR-LLM**, **CLAIM 2024** (and STARD-AI / TRIPOD+AI where a diagnostic/prognostic claim
+is made) → `/check-reporting`. Reviewer-side audit of a finished manuscript uses the
+`mllm_evaluation.md` (ME0–ME8) probe via `/self-review` and `/peer-review`.
+
+## Verification notes
+
+Each claim here is grounded in a named public method/standard or described qualitatively; no
+numbers, thresholds, or dataset contents are invented.
+
+- **n-gram metrics correlate weakly with clinical error; clinical-efficacy metrics needed** —
+  Yu et al., *Patterns* 2023 (RadCliQ). Named public methods paper (matches the citation already
+  vendored in the `mllm_evaluation.md` probe).
+- **RadGraph-F1 (entity-relation overlap)** — Jain et al., NeurIPS Datasets & Benchmarks 2021.
+  Named public methods paper.
+- **CheXbert-F1 / CheXpert observation labels** — Smit et al., 2020 (CheXbert); Irvin et al.,
+  2019 (CheXpert labeler). Named public methods papers; the CheXpert label set is a factual
+  artifact, not invented.
+- **RadCliQ as a composite** — Yu et al., *Patterns* 2023. Named public methods paper.
+- **Atomic-fact faithfulness, omission-vs-fabrication, false-premise/abstention** — described as
+  established evaluation principles; **MedVH** and **Med-HALT** named as instruments only (as in
+  the probe), no scores invented.
+- **Contamination of public benchmarks; closed-corpus unknowability; cutoff/held-out/probe
+  checks (canary, perturbed-duplicate gap, membership test)** — stated as accepted principles and
+  practices, qualitatively; benchmark **names** (VQA-RAD, SLAKE, MIMIC-CXR, MedQA, PMC-VQA,
+  PathVQA, OpenI, PubMedQA) are factual public-dataset names, no contents reproduced.
+- **Closed-API non-determinism even at temperature 0; prompt/format/sampling sensitivity** —
+  described qualitatively as documented behavior; no figure attached.
+- **Prompt + decoding + ≥3 runs + ≥2 phrasings disclosure** — MI-CLEAR-LLM transparency
+  (named standard); the ≥3 / ≥2 conventions are this skill's own house thresholds (carried from
+  SKILL.md / the ME-probe), not literature values.
+- **LLM-as-judge must be validated against human labels** — stated as a methodological principle;
+  judge validation routed to `/design-ai-benchmarking`.
+- **Reader study for deployment/utility claims; error taxonomy, severity, IRR** — consistent with
+  CLAIM 2024 / TRIPOD-LLM reporting expectations (named standards); sizing/IRR routed to
+  `/calc-sample-size` and `/analyze-stats`.
+- **Metrics Reloaded / CLAIM 2024 / TRIPOD-LLM / MI-CLEAR-LLM / Model Cards (Mitchell 2019) /
+  Datasheets (Gebru 2021)** — named public standards, cited by name only.

package/skills/model-evaluation/SKILL.md

@@ -79,6 +79,18 @@ figures → `/make-figures`; the numbers + subgroup performance → `/model-card
 `scripts/check_metric_reporting.py` — flags a task-metric mismatch / missing uncertainty (stdlib,
 network-free). Reproducible challenge: `bash ${CLAUDE_SKILL_DIR}/scripts/metric_reporting_challenge/verify.sh`.
 
+## Reference Files
+
+Load on demand (keep SKILL.md short):
+- `${CLAUDE_SKILL_DIR}/references/metric_guide.md` — operational checklist: the task-correct metric
+  per task (segmentation Dice + HD95/NSD per structure; classification AUROC + AUPRC + sens/spec at
+  deployment prevalence; detection FROC/mAP with a stated IoU), plus calibration, subgroup slices,
+  run-variance, and the per-case CSV hand-off.
+- `${CLAUDE_SKILL_DIR}/references/metric_selection_grounding.md` — the standards grounding behind
+  those choices: the Metrics Reloaded task-fingerprint principle, why each metric pairing is
+  required, calibration vs discrimination, disaggregated reporting, and the CLAIM 2024
+  reporting-fit map (`/check-reporting` owns the item audit).
+
 ## Boundaries
 
 ```

package/skills/model-evaluation/references/metric_selection_grounding.md

@@ -0,0 +1,139 @@
+# Metric-selection grounding and CLAIM 2024 reporting fit (model-evaluation)
+
+The *why* behind the operational checklist in `metric_guide.md`. Where `metric_guide.md` says
+**what** to compute, this doc grounds **why** that pairing is required and where the outputs land
+in the manuscript. Anchored to **Metrics Reloaded** (Maier-Hein & Reinke et al., *Nature Methods*
+2024) and its pitfalls companion (Reinke et al., *Nature Methods* 2024), **CLAIM 2024** (Tejani et
+al., *Radiology: Artificial Intelligence* 2024), **TRIPOD+AI** (the AI extension of TRIPOD; Collins
+et al., *BMJ* 2024), calibration work (Guo et al., *ICML* 2017), and the Model Card **Factors**
+(Mitchell et al., *FAT\** 2019). The deliverable is still the per-case CSV; the deterministic gate
+is `scripts/check_metric_reporting.py`.
+
+> Verify exact **CLAIM 2024 item numbers** and any **NSD tolerance** against the source before
+> quoting them as formal values — the mapping and tolerances below are described qualitatively by
+> design. Do not hand-type a metric value: every number comes from executed code.
+
+## The Metrics Reloaded principle: task fingerprint → metric
+
+- **The metric is derived from the problem, not from habit.** Metrics Reloaded selects metrics from
+  the problem fingerprint — task category (classification / segmentation / detection-localization),
+  structure size and shape, class prevalence, and whether *where* the model is right matters. A
+  metric that ignores a property that matters clinically is the wrong metric.
+- **No single metric is sufficient.** Pair a counting/overlap metric with a complementary one (a
+  boundary metric, a calibration summary) so a blind spot in one is covered by the other. A lone
+  headline number is the recurring pitfall the companion paper warns about.
+- **Report per-class / per-structure with a distribution**, not only a global mean — a mean hides
+  minority-class and small-structure failure.
+- **Define edge-case behaviour explicitly** (empty reference, no positive cases): several metrics
+  are undefined there, and a silent convention changes the score.
+
+## Segmentation — overlap **and** boundary, per structure
+
+- Report an overlap metric (Dice or IoU) **with** a boundary metric (**HD95** or **NSD**). Dice is
+  a volume-overlap measure: it is insensitive to boundary error and unstable on small or thin
+  structures, so it can look high while the contour is clinically wrong.
+- **HD95** = 95th-percentile Hausdorff distance — robust to a few outlier surface points relative to
+  the raw maximum Hausdorff. **NSD** (normalised surface distance / surface Dice) = the fraction of
+  the predicted surface within a **task-specific tolerance τ** of the reference surface; **τ must be
+  stated** and is chosen from clinical acceptability, not invented.
+- Compute **per structure** with bootstrap 95% CIs obtained by resampling **patients, not pixels**
+  (Efron–Tibshirani bootstrap). State the rule for **empty-reference / false-positive-only** cases
+  (a Dice of 0/0 is undefined).
+- Gate: `PIXEL_ACCURACY_SEG` (pixel/voxel accuracy is dominated by background — never the headline)
+  and `NO_BOUNDARY_METRIC`.
+
+## Classification — discrimination, operating point at prevalence, then calibration
+
+- **AUROC and AUPRC.** AUROC summarises ranking across thresholds; under class imbalance the
+  precision–recall view (AUPRC) is more informative, because the ROC's false-positive rate uses the
+  large negative denominator and can look optimistic when negatives dominate (Saito & Rehmsmeier,
+  *PLoS ONE* 2015).
+- **Operating-point metrics at the deployment prevalence.** **PPV/NPV** (and accuracy) move with the
+  base rate, so values read off an artificially balanced test set mislead at deployment — report
+  them at the real prevalence. Sensitivity and specificity are prevalence-independent but depend on
+  the operating threshold, so **fix the threshold on the training/tuning folds, never the test set**
+  (choosing it on the test set is tuning-on-test and inflates the estimate).
+- Bootstrap **95% CIs at the patient level**.
+- Gate: `ACCURACY_ONLY` (a bare-accuracy headline under imbalance is flagged).
+
+## Detection / localization — FROC or mAP with the IoU criterion stated
+
+- Report **FROC** (sensitivity versus mean false positives per image) or **mAP**, and **state the
+  match criterion** — a prediction counts as a true positive only if its overlap with a
+  ground-truth object meets the stated IoU (or centroid/mask) threshold. Per Metrics Reloaded's
+  localization category, the metric is undefined without that criterion.
+- **Patient-level accuracy is not a detection metric**, and a per-lesion result must not be reported
+  as per-patient — respect the analysis unit set in Phase 1.
+- Gate: `DETECTION_METRIC_MISSING`.
+
+## Calibration — a separate axis from discrimination
+
+- A model can **discriminate well (high AUROC) and still be miscalibrated**; report calibration
+  separately (Guo et al., *ICML* 2017). Use a **reliability diagram** plus a summary (**ECE** or the
+  **Brier** score).
+- **ECE is binning-sensitive** — state the binning scheme (or prefer a binning-robust summary) and
+  do not over-read a single ECE value.
+- **TRIPOD+AI** requires reporting **both** discrimination and calibration for a clinical prediction
+  model — calibration is not optional reporting.
+
+## Subgroup slices — disaggregated reporting
+
+- Slice every headline metric by the Model Card **Factors** (scanner/vendor, site, age, sex,
+  disease severity), and report the **per-subgroup n** so the reader can see which estimates are
+  thin.
+- Need enough events per subgroup to estimate the metric; otherwise **say so** rather than report a
+  noisy point estimate.
+- This is disaggregated *reporting*. The formal fairness/equity audit lives in `/model-validation`
+  plus the equity probe — cross-reference, do not duplicate it here.
+
+## CLAIM 2024 reporting fit — where the eval outputs land
+
+CLAIM 2024 organises items under the manuscript sections. The model-evaluation deliverable feeds the
+**Methods** (metric definitions, reference standard, data partition, threshold selection) and the
+**Results** (metrics with uncertainty, calibration, subgroup/failure analysis). `/check-reporting`
+owns the item-by-item CLAIM 2024 / TRIPOD+AI audit; this is the routing map.
+
+| Eval output | CLAIM 2024 area (verify item #) | Note |
+|---|---|---|
+| Metric definitions + how each was computed | Methods | Name the metric and its formula/library; no undefined "accuracy" headline. |
+| Reference / ground-truth standard + how derived | Methods | Reader count, blinding, adjudication — state it. |
+| Held-out, patient-level data partition | Methods | Cross-link `/model-validation` (split leakage). |
+| Performance metrics **with uncertainty (CIs)** | Results | Bootstrap CIs at the analysis unit. |
+| Calibration (reliability diagram + ECE/Brier) | Results | Required alongside discrimination (TRIPOD+AI). |
+| Subgroup + failure-case analysis | Results | Per-subgroup n; flag thin slices. |
+| Threshold + operating point at prevalence | Methods / Results | Threshold fixed on tuning folds, reported prevalence. |
+
+## What the skill checks / advises
+
+1. Run the gate — `PIXEL_ACCURACY_SEG` / `NO_BOUNDARY_METRIC` / `ACCURACY_ONLY` /
+   `DETECTION_METRIC_MISSING` must all be zero.
+2. Advise the author to **state τ** (NSD), **state the IoU criterion** (detection), and **state the
+   deployment prevalence** (operating-point metrics); report **per-structure / per-subgroup with n**;
+   give **patient-level bootstrap CIs**; and report **calibration alongside discrimination**.
+3. Emit `eval/per_case_metrics.csv` for `/analyze-stats` (DeLong / NRI / IDI / decision curves /
+   MRMC) — numbers are never hand-typed, and an uncertain metric or CI method is flagged `[VERIFY]`.
+
+## Verification notes
+
+- **Metrics Reloaded** + pitfalls companion (Maier-Hein, Reinke et al., *Nature Methods* 2024):
+  named public standard — grounds the task-fingerprint principle, single-metric-insufficiency,
+  per-class reporting, edge-case definition, boundary metrics, and the detection/localization
+  category. Cited as a named method, not quoted.
+- **CLAIM 2024** (Tejani et al., *Radiology: Artificial Intelligence* 2024): named reporting
+  checklist. Exact item numbers are **not** quoted — the table maps outputs to manuscript sections;
+  resolve item numbers against the source (`[VERIFY]`). `/check-reporting` owns the audit.
+- **TRIPOD+AI** (Collins et al., *BMJ* 2024): named standard; grounds the calibration-and-
+  discrimination requirement. Written as base TRIPOD + AI extension.
+- **AUPRC under imbalance** (Saito & Rehmsmeier, *PLoS ONE* 2015, **CC-BY**): principle only (ROC vs
+  PR on imbalanced data); no text copied.
+- **Calibration / ECE** (Guo et al., *ICML* 2017): named methods paper; reliability diagram + ECE,
+  with the binning-sensitivity caveat stated qualitatively.
+- **NSD / surface Dice with tolerance**: the tolerance-based surface metric (e.g., Nikolov et al.,
+  head-and-neck OAR segmentation) recommended for boundary error by Metrics Reloaded; **τ described
+  qualitatively, no value invented**.
+- **Bootstrap CIs** (Efron & Tibshirani): canonical resampling method; patient-level resampling
+  principle.
+- **Model Card Factors** (Mitchell et al., *FAT\** 2019): named documentation standard for
+  disaggregated reporting axes.
+- No DOIs, dataset names, numeric thresholds, prevalences, NSD tolerances, or CLAIM item numbers are
+  fabricated; any uncertain specific is flagged `[VERIFY]`.

package/skills/model-evaluation/scripts/check_metric_reporting.py

@@ -62,7 +62,7 @@ P = {
     "spec": r"\b(specificit\w+|true[- ]?negative rate|tnr)\b",
     "detection": r"\b(froc|map\b|mean average precision|sensitivity per (?:false positive|fp)|"
                  r"competition performance metric|cpm)\b",
-    "iou_crit": r"\b(?:iou|intersection over union|overlap)\b[^.\n]{0,40}"
+    "iou_crit": r"\b(?:iou|intersection over union|overlap)\b[^.]{0,40}"
                 r"(?:threshold|criterion|>=|≥|>|\bof\b|above|exceed|\d\.\d)"
                 r"|match(?:ing)? criterion"
                 r"|(?:cent(?:er|re|roid)|distance)[- ]?based"

package/skills/model-evaluation/scripts/metric_reporting_challenge/fixture/det_good_wrapped.md

@@ -0,0 +1,5 @@
+# Results (detection)
+Lesion detection was summarised with FROC. A predicted box was scored as a true
+positive when its IoU
+with the reference lesion exceeded 0.3. Sensitivity at 1 false positive per scan
+was 0.84 (95% CI 0.80-0.88), with the operating point fixed on the tuning fold.

package/skills/model-evaluation/scripts/metric_reporting_challenge/fixture/det_no_iou.md

@@ -0,0 +1,4 @@
+# Results (detection)
+Lesion detection performance was summarised with FROC and mean average precision.
+Sensitivity at 1 false positive per scan was 0.84 (95% CI 0.80-0.88). The operating
+point was fixed on the tuning fold.

package/skills/model-evaluation/scripts/metric_reporting_challenge/verify.sh

@@ -25,5 +25,10 @@ want seg_bad.md segmentation PIXEL_ACCURACY_SEG
 clean seg_good.md segmentation
 want clf_bad.md classification ACCURACY_ONLY
 clean clf_good.md classification
+# Detection branch: a stated IoU match criterion is required; a hard-wrapped criterion
+# (IoU and its threshold on different physical lines) must still be detected — det_good_wrapped
+# locks the iou_crit proximity window against newline-induced false fires.
+want det_no_iou.md detection DETECTION_METRIC_MISSING
+clean det_good_wrapped.md detection
 
-echo "PASS: metric-reporting gate flags Dice-only/pixel-accuracy and accuracy-only, clears task-correct reports."
+echo "PASS: metric-reporting gate flags Dice-only/pixel-accuracy, accuracy-only, and detection without an IoU criterion; clears task-correct reports (including a line-wrapped IoU criterion)."

package/skills/model-validation/SKILL.md

@@ -47,6 +47,12 @@ TorchIO — those produce the model, this validates and publishes it.
 
 ## Workflow
 
+The design/audit rationale behind Phases 2–7 — the full data-leakage taxonomy, the
+internal-vs-genuine-external validation ladder, comparator design, single-run vs multi-seed
+variance, test-set sizing, and the CLAIM 2024 / TRIPOD+AI / STARD-AI reporting map — is in
+`${CLAUDE_SKILL_DIR}/references/validation_design.md` (load on demand). The patient-disjointness
+verdict itself is proven by `scripts/check_split_leakage.py` (Phase 2), not from that prose.
+
 ### Phase 1 — Reconstruct the task, the intended-use horizon, and the analysis unit
 State the model's task (segmentation / classification / detection), its **intended-use horizon**
 (screening, triage, pre-procedure, post-hoc), the **single headline metric** the conclusion leans on,

package/skills/model-validation/references/validation_design.md

@@ -0,0 +1,150 @@
+# Validation-design reference (model-validation)
+
+Load-on-demand backbone for Phases 2–7 — the leakage taxonomy, the internal-vs-external
+tier ladder, comparator design, run variance, test-set sizing, and the reporting map. Anchored
+to **Kapoor & Narayanan** (*Patterns* 2023, leakage taxonomy), **Varoquaux & Cheplygina**
+(*npj Digital Medicine* 2022, medical-imaging ML failure modes), **Metrics Reloaded**
+(Maier-Hein & Reinke et al., *Nature Methods* 2024), **CLAIM 2024**, **TRIPOD+AI** (*BMJ* 2024),
+and **STARD-AI** (*Nature Medicine* 2025). It explains *what to check and advise*; the patient-disjointness
+verdict is proven by `scripts/check_split_leakage.py`, not by this prose.
+
+## 1. The data-leakage taxonomy
+
+Leakage = any information about a test case that could influence training. Organise the audit by
+the three Kapoor-Narayanan (*Patterns* 2023) categories; the deterministic gate covers only the
+first row.
+
+| Category (Kapoor-Narayanan) | Imaging-specific leak | How it inflates the metric | How to catch |
+|---|---|---|---|
+| **No clean train/test separation** | **Patient-level overlap** — the same patient's images straddle train and test | Model memorises patient anatomy, not pathology | `check_split_leakage.py` → `PATIENT_OVERLAP` (set arithmetic on IDs) |
+| ″ | **Near-duplicate / repeated-acquisition** — repeat scans, follow-ups, augmented copies, overlapping patches of one volume across splits | Test cases are not independent of training | Split on the **patient**, not the image/slice/series; dedup by patient before partitioning |
+| ″ | **Preprocessing-before-split** — normalisation stats, intensity windowing, resampling, feature selection, ComBat harmonisation, or foundation-model embeddings fit on the **whole cohort** | Test statistics bleed into the training pipeline | Fit every transform on the **training fold only**; the test set is touched only at scoring time |
+| ″ | **Temporal leakage** — a random split where future and past coexist, under a prognostic/surveillance claim | Model peeks at later-era data | Use a **temporal split** (train on earlier, test on later) when the claim is temporal |
+| **Illegitimate features** | **Site / scanner / burned-in-label shortcut** — the model keys on acquisition site, vendor signature, a laterality token, or a body-part marker rather than the finding | Discrimination collapses off-site | Standalone confound check (shortcut-learning, Geirhos et al. *Nat Mach Intell* 2020; DeGrave et al. *Nat Mach Intell* 2021; Zech et al. *PLOS Med* 2018); subgroup-by-site slice |
+| **Test set ≠ population of interest** | **Spectrum / selection bias** — test cases curated, enriched, or selected on an optional modality | Reported accuracy does not transfer to deployment | Confirm the test set reflects the intended-use population; see §2 / §6 |
+
+The decisive question to ask of every preprocessing and selection step: **could any value used in
+training have been computed only with knowledge of a test case?** If yes, it is leakage even when
+the split table itself looks disjoint.
+
+### Tuning-on-test (the test set must be touched once)
+Architecture search, hyperparameter sweeps, early-stopping, **and operating-point / threshold
+selection** that read the test set are all forms of the first category — the test set has become a
+development set, and the headline metric is optimistic. Fix the threshold and select the model on
+the **training/validation folds**, then evaluate the frozen model on the test set exactly once.
+"Developed with external validation" where the single external set was also used for tuning is no
+longer external validation (§2).
+
+## 2. Internal vs genuine external validation
+
+Classify the evidence honestly and let the tier cap the claim. Cross-validation and bootstrap are
+development-time **optimism corrections** (apparent-performance debiasing), **not** external
+validation — this is the long-standing TRIPOD / prediction-model distinction (Collins et al.,
+TRIPOD 2015; TRIPOD+AI, *BMJ* 2024).
+
+```
+apparent (train=test, never sufficient)
+  → internal random split
+    → k-fold cross-validation / bootstrap   ← still internal (optimism correction)
+      → temporal split (later era held out)
+        → geographic / external (different site, scanner, vendor)
+          → multi-site / prospective external
+```
+
+- A **generalisability or deployment-readiness** claim needs at least a genuine external tier
+  (different site/scanner/vendor), not an internal split.
+- Single-centre external validation supports a narrower claim than multi-site; say which.
+- Reusing the external set for any tuning demotes it back to internal — flag the contradiction.
+
+## 3. Comparator design
+
+A standalone metric rarely answers the clinical question; decide what the model is measured
+*against*, evaluated on the **same** test set. CLAIM 2024 and TRIPOD+AI both ask for comparison to
+current practice / an existing model.
+
+| Comparator | When | Hand-off |
+|---|---|---|
+| **Clinical / no-model baseline** | "does the model beat current standard of care?" | — |
+| **Incremental value over an existing score** | model added on top of an established risk score / radiologist read | added-value statistics (NRI / IDI / decision curve) → `/analyze-stats` |
+| **Reader comparison (standalone or AI-assisted)** | model vs / with radiologists | rubric, reader panel, inter-rater design → `/design-ai-benchmarking` |
+
+Name whether the claim is **standalone** (model alone) or **assistive** (clinician + model); they
+need different comparators and different reporting.
+
+## 4. Single-run vs multi-seed variance
+
+A single training run overstates precision: deep-model metrics move with the random seed
+(initialisation, data order, augmentation), and some GPU ops are non-deterministic even with cuDNN
+deterministic flags set (Varoquaux & Cheplygina, *npj Digit Med* 2022; reproducibility crisis,
+Kapoor & Narayanan 2023). Require the headline metric as **mean ± SD over ≥ 3 seeds / runs**, or a
+**single fixed reported seed with the determinism caveat stated**. A point estimate from one run,
+presented as if exact, is a reporting defect.
+
+## 5. Test-set sizing
+
+Check **events per class in the test set**, not the cohort total. A metric computed on a sparse
+positive set has a confidence interval spanning much of the usable range, so a headline AUROC /
+sensitivity can be statistically uninformative even when the cohort is large.
+
+- Size the test set for the **CI width** of the headline metric and for **per-subgroup** estimates
+  you intend to report.
+- **Calibration** in particular is data-hungry — prediction-model validation guidance uses a rule
+  of thumb of roughly ≥ 100 events and ≥ 100 non-events before a reliability assessment is stable
+  (treat as a rule of thumb, not a hard cutoff; confirm for the specific design).
+- Hand the formal calculation (diagnostic-accuracy precision, AUC precision, agreement, calibration
+  sample size) to `/calc-sample-size`.
+
+Metric **selection** (Dice + boundary metric; AUROC + AUPRC under imbalance; FROC/mAP with the IoU
+criterion) is owned by `/model-evaluation` (`references/metric_guide.md`, anchored to Metrics
+Reloaded); this skill only checks that the chosen metric is task- and prevalence-correct.
+
+## 6. Reporting-guideline fit
+
+Map the study to its standard via `/check-reporting`, and **name both the base instrument and the
+AI extension**, citing each at its actual maturity (published guideline vs protocol-stage), never
+beyond it. The four standards below are cross-checked against the repository's `check-reporting`
+verified checklists.
+
+| Study framing | Primary standard (extension) | Base instrument | Risk of bias |
+|---|---|---|---|
+| Diagnostic / triage **imaging-AI** study (standalone or assistive) | **CLAIM 2024** update (Tejani et al., *Radiology: AI* 2024) | CLAIM 2020 (Mongan, Moy, Kahn, *Radiology: AI* 2020) | PROBAST+AI |
+| **Prediction model** (diagnostic or prognostic; regression or ML) | **TRIPOD+AI** (Collins, Moons et al., *BMJ* 2024) | TRIPOD 2015 (Collins, Reitsma, Altman, Moons) | **PROBAST+AI** (Moons et al., *BMJ* 2025; replaces PROBAST-2019) on base PROBAST (Wolff et al., *Ann Intern Med* 2019) |
+| **Diagnostic accuracy** study (index test vs reference standard; sens/spec) | **STARD-AI** (Sounderajah et al., *Nature Medicine* 2025) | STARD 2015 (Bossuyt et al., *BMJ* 2015) | QUADAS-2 / QUADAS-C |
+
+Tie the partition, leakage controls, validation tier, comparator, run variance, and test-set sizing
+above to the specific items these standards request (data partition, sample size, model evaluation,
+comparison to current practice, reproducibility).
+
+## Hand-offs
+- Patient-disjointness proof → `scripts/check_split_leakage.py` (Phase 2, run first).
+- Test-set / event sizing → `/calc-sample-size`.
+- Reader-comparison rubric + inter-rater design → `/design-ai-benchmarking`.
+- Per-case metric computation + reporting gate → `/model-evaluation` → `/analyze-stats`.
+- Item-by-item compliance → `/check-reporting`; Methods write-up → `/write-paper`; reviewer-side
+  audit of the finished draft → `/self-review` (MD0–MD8 `model_development.md` probe).
+
+## Verification notes (what each claim is grounded on)
+- **Leakage taxonomy / three categories, reproducibility crisis** — Kapoor & Narayanan, "Leakage and
+  the reproducibility crisis in machine-learning-based science," *Patterns* 2023. Imaging-specific
+  failure modes (patient-level split, preprocessing-before-split) — Varoquaux & Cheplygina,
+  *npj Digital Medicine* 2022. Both already cited in `check_split_leakage.py`.
+- **Site/scanner/shortcut leakage** — shortcut learning, Geirhos et al., *Nature Machine
+  Intelligence* 2020; radiographic shortcut example, DeGrave et al., *Nature Machine Intelligence*
+  2021; cross-site generalisation failure, Zech et al., *PLOS Medicine* 2018. Used as named
+  examples of the "illegitimate features" / spectrum-bias rows, not as numeric claims.
+- **Internal vs external, optimism correction, CV ≠ external** — TRIPOD 2015 (Collins, Reitsma,
+  Altman, Moons) and TRIPOD+AI (*BMJ* 2024). The tier ladder mirrors the skill's Phase 3.
+- **Metric selection deferral** — Metrics Reloaded (Maier-Hein & Reinke et al., *Nature Methods*
+  2024); detail lives in `/model-evaluation`.
+- **Reporting map** — CLAIM 2024 update (Tejani et al., *Radiology: AI* 2024) on base CLAIM 2020
+  (Mongan, Moy, Kahn); TRIPOD+AI (*BMJ* 2024); STARD-AI (Sounderajah et al., *Nature Medicine*
+  2025) on base STARD 2015 (Bossuyt et al., *BMJ* 2015); PROBAST+AI (Moons et al., *BMJ* 2025) on
+  base PROBAST (Wolff et al., *Ann Intern Med* 2019). All four are cross-checked against this
+  repository's `check-reporting` verified checklists (CLAIM 2024 = e240300; TRIPOD+AI = e078378;
+  STARD-AI = DOI 10.1038/s41591-025-03953-8, PMID 40954311; PROBAST+AI = e082505).
+- **Numbers deliberately not asserted**: the only quantitative figure is the ~100 events/non-events
+  calibration rule of thumb, flagged as a rule of thumb to confirm per design — no dataset names,
+  thresholds, or performance numbers are invented here. The reporting-standard DOIs/PMIDs above are
+  carried from the repository's verified `check-reporting` checklists; still re-confirm the exact
+  identifier via `/search-lit` before quoting any of them in a manuscript, and mark any uncertain
+  item `[VERIFY]`.