medsci-skills 5.14.0 → 5.16.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +71 -10
- package/metadata/distribution_files.json +169 -14
- package/metadata/distribution_manifest.json +4 -1
- package/metadata/skills_catalog.json +33 -3
- package/package.json +1 -1
- package/skills/explainability/SKILL.md +109 -0
- package/skills/explainability/references/explainability_guide.md +66 -0
- package/skills/explainability/scripts/check_explainability_report.py +239 -0
- package/skills/explainability/scripts/check_explainability_report_challenge/expected/strong.txt +9 -0
- package/skills/explainability/scripts/check_explainability_report_challenge/expected/weak.txt +11 -0
- package/skills/explainability/scripts/check_explainability_report_challenge/fixture/report_strong.json +9 -0
- package/skills/explainability/scripts/check_explainability_report_challenge/fixture/report_weak.json +8 -0
- package/skills/explainability/scripts/check_explainability_report_challenge/problem.md +19 -0
- package/skills/explainability/scripts/check_explainability_report_challenge/verify.sh +37 -0
- package/skills/explainability/skill.yml +45 -0
- package/skills/explainability/tests/test_explainability_report.sh +88 -0
- package/skills/model-scaffold/SKILL.md +31 -15
- package/skills/model-scaffold/references/finetuning_guide.md +91 -0
- package/skills/model-scaffold/scripts/check_training_hygiene.py +38 -2
- package/skills/model-scaffold/scripts/scaffold.py +263 -1
- package/skills/model-scaffold/scripts/scaffold_challenge/problem.md +6 -0
- package/skills/model-scaffold/scripts/scaffold_challenge/verify.sh +20 -2
- package/skills/model-scaffold/skill.yml +4 -1
- package/skills/model-scaffold/tests/fixtures/finetune_no_provenance/train.py +24 -0
- package/skills/model-scaffold/tests/test_training_hygiene.sh +14 -1
- package/skills/preprocess-imaging/SKILL.md +117 -0
- package/skills/preprocess-imaging/references/preprocessing_guide.md +69 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage.py +294 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/expected/clean.txt +9 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/expected/leak.txt +12 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/fixture/manifest_clean.json +16 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/fixture/manifest_leak.json +17 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/problem.md +20 -0
- package/skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/verify.sh +39 -0
- package/skills/preprocess-imaging/skill.yml +45 -0
- package/skills/preprocess-imaging/tests/test_preprocessing_leakage.sh +77 -0
- package/skills/radiomics-ml/SKILL.md +134 -0
- package/skills/radiomics-ml/references/radiomics_ml_guide.md +115 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml.py +229 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml_challenge/expected/strong.txt +9 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml_challenge/expected/weak.txt +14 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml_challenge/fixture/pipeline_strong.json +13 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml_challenge/fixture/pipeline_weak.json +13 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml_challenge/problem.md +20 -0
- package/skills/radiomics-ml/scripts/check_radiomics_ml_challenge/verify.sh +37 -0
- package/skills/radiomics-ml/skill.yml +46 -0
- package/skills/radiomics-ml/tests/test_radiomics_ml.sh +88 -0
- package/skills/self-review/scripts/check_scope_coherence.py +18 -1
- package/skills/self-review/tests/fixtures/scope_antipattern_list.md +13 -0
- package/skills/self-review/tests/test_scope_coherence.sh +14 -0
package/README.md
CHANGED
|
@@ -2,14 +2,14 @@
|
|
|
2
2
|
|
|
3
3
|
# MedSci Skills
|
|
4
4
|
|
|
5
|
-
**
|
|
5
|
+
**54 skills that actually work.** Built by a physician-researcher, tested on real publications.
|
|
6
6
|
|
|
7
7
|
*MedSci Skills is an end-to-end research tool for physician and medical-engineering researchers — design → scaffold → validate → publish — for the clinical manuscript and the medical-AI model behind it. Its moat is the compliance layer — 44 reporting guidelines and risk-of-bias tools, reference/citation verification, and deterministic integrity gates before peer review — now extended by a model-engineering lane that scaffolds reproducible, leakage-safe training repos and audits model validation. Clinical AI model research engineering is in scope; a general AI-scientist platform is not. It competes on clinical submission reliability, not skill count.*
|
|
8
8
|
|
|
9
9
|
[](LICENSE)
|
|
10
10
|
[](https://github.com/Aperivue/medsci-skills/releases/latest)
|
|
11
11
|
[](https://github.com/Aperivue/medsci-skills/actions/workflows/validate.yml)
|
|
12
|
-

|
|
13
13
|
[](https://www.npmjs.com/package/medsci-skills)
|
|
14
14
|
[](https://youtu.be/MclQ_RIofpE)
|
|
15
15
|
[](https://github.com/Aperivue/medsci-skills/contribute)
|
|
@@ -285,6 +285,10 @@ The E2E pipeline (`orchestrate --e2e`) produces everything up to `qc/`. The `sub
|
|
|
285
285
|
|
|
286
286
|
## What's New
|
|
287
287
|
|
|
288
|
+
**v5.16** — model-engineering produce-side depth, clinical fine-tuning focus (Items 3–4 of the [produce-side depth roadmap](docs/roadmap_model_engineering_depth.md)). A new **`radiomics-ml`** skill + `check_radiomics_ml` gate for the most common solo-doable clinical-ML workflow — radiomics / tabular features → any classical learner (LASSO / SVM / random forest / XGBoost / …) → a clinical outcome, no GPU — with a learner-agnostic nested-CV / calibration / stability gate; plus a **`model-scaffold` fine-tuning mode** (`--task finetune` + `--from-pretrained`) that adapts a pretrained backbone on collected clinical data with a frozen→unfrozen schedule, discriminative learning rates, and a pretrained-weight provenance record (`PRETRAINED.md` + a `PRETRAINED_PROVENANCE_MISSING` verdict on the existing `check_training_hygiene`, plus a MedSAM-adaptation + train-only diffusion-augmentation guide). Additive and backward-compatible; **54 skills / 44 guidelines / 49 integrity detectors / 22 domain-probe modules**. (See the [CHANGELOG](CHANGELOG.md) for v5.0–v5.15.)
|
|
289
|
+
|
|
290
|
+
**v5.15** — model-engineering produce-side depth. Two new skills that *produce* what the review lane previously only audited: **`preprocess-imaging`** (DICOM/NIfTI data prep + a `check_preprocessing_leakage` gate that extends the split-leakage moat upstream to the data stage) and **`explainability`** (Grad-CAM / saliency held to the reviewer bar — Adebayo sanity checks, quantitative localisation vs ground truth, attribution-not-validation framing, via `check_explainability_report`). Plus a by-stage skill index, multi-host README/About (Claude Code · Codex · Cursor · Copilot), copy-paste citation ergonomics, and a real-project precision fix. Additive and backward-compatible; **53 skills / 44 guidelines / 48 integrity detectors / 22 domain-probe modules**. (See the [CHANGELOG](CHANGELOG.md) for v5.0–v5.14.)
|
|
291
|
+
|
|
288
292
|
**v4.10** — reviewer-coverage expansion reverse-engineered from high-IF, CC-BY papers (learn-only under the `reverse_engineer/` license firewall), plus a clinician-friendly update path. Additive and backward-compatible; 45 skills / **44 guidelines** / 36 detectors / **15 domain-probe modules** (was 12):
|
|
289
293
|
|
|
290
294
|
- **Three new reviewer domain-probe modules** (`/peer-review` + `/self-review`, vendored byte-identical): **Mendelian randomization** (MR1–MR8 — IV assumptions, pleiotropy-robust sensitivity suite, Steiger, sample overlap, NLMR, drug-target colocalization), **polygenic risk score** (PG1–PG8 — ancestry portability, base/target leakage, incremental value over the clinical model, screening-vs-discrimination, calibration), and **network meta-analysis** (NM1–NM8 — transitivity, incoherence, SUCRA over-interpretation, CINeMA/GRADE-NMA, component-NMA additivity). Plus observational **O17** (agnostic many-exposure-scan multiplicity: ExWAS/EWAS/MWAS).
|
|
@@ -400,6 +404,8 @@ MedSci Skills is **opinionated and narrow on purpose**: a single physician-resea
|
|
|
400
404
|
|
|
401
405
|
📖 **Per-skill reference:** [`docs/skills/`](docs/skills/) — one page per skill (what it does, when it activates, its Quality Card — purpose, safety boundaries, known limitations, validation, evidence — and bundled resources), generated from each `SKILL.md` + `skill.yml`. See [`docs/skills/AUDIT.md`](docs/skills/AUDIT.md) for how the skills are validated.
|
|
402
406
|
|
|
407
|
+
🧠 **ML / DL method coverage:** [`docs/method_coverage_map.md`](docs/method_coverage_map.md) — which machine-learning and deep-learning method families (CNN / transformer / segmentation / detection / foundation / diffusion / SSL for imaging; the full classical family — penalised regression, SVM, k-NN, trees, boosting [XGBoost / LightGBM / CatBoost], MLP, ensembles, clustering — for radiomics/tabular; LLM/MLLM) map to which skills for selection, production, validation, interpretation, and reporting.
|
|
408
|
+
|
|
403
409
|
```
|
|
404
410
|
┌─────────────────────────────────┐
|
|
405
411
|
│ orchestrate: single entry point │
|
|
@@ -447,6 +453,21 @@ ma-scout -> search-lit -> fulltext-retrieval -> design-study ──> write-proto
|
|
|
447
453
|
└─────────────────────────────────────────────┘
|
|
448
454
|
```
|
|
449
455
|
|
|
456
|
+
### By research stage
|
|
457
|
+
|
|
458
|
+
All 54 skills, grouped by where they fit in the clinical-manuscript and medical-AI lifecycle. Full descriptions are in the table below; one page per skill lives in the [per-skill reference](docs/skills/).
|
|
459
|
+
|
|
460
|
+
| Stage | Skills |
|
|
461
|
+
|-------|--------|
|
|
462
|
+
| 🔭 **Discover & scope** | `ma-scout` · `find-cohort-gap` · `search-lit` · `fulltext-retrieval` · `lit-sync` · `author-strategy` |
|
|
463
|
+
| 📐 **Design & plan** | `design-study` · `calc-sample-size` · `define-variables` · `write-protocol` · `fill-protocol` · `design-ai-benchmarking` |
|
|
464
|
+
| 🧹 **Data & analysis** | `deidentify` · `clean-data` · `generate-codebook` · `version-dataset` · `analyze-stats` · `batch-cohort` · `cross-national` · `replicate-study` |
|
|
465
|
+
| 🤖 **Medical-AI model engineering** | `preprocess-imaging` · `architecture-zoo` · `model-scaffold` · `model-validation` · `model-evaluation` · `explainability` · `radiomics-ml` · `model-card` · `mllm-eval` |
|
|
466
|
+
| ✍️ **Write & visualize** | `write-paper` · `make-figures` · `review-paper` · `present-paper` · `humanize` · `polish-language` · `academic-aio` |
|
|
467
|
+
| ✅ **Comply & verify** | `check-reporting` · `self-review` · `verify-refs` · `manage-refs` |
|
|
468
|
+
| 📤 **Submit & respond** | `find-journal` · `add-journal` · `sync-submission` · `revise` · `peer-review` · `fill-icmje-coi` |
|
|
469
|
+
| 🧭 **Orchestrate & manage** | `orchestrate` · `intake-project` · `manage-project` · `meta-analysis` · `grant-builder` · `publish-skill` · `render-pdf-doc` · `setup-medsci` |
|
|
470
|
+
|
|
450
471
|
### Available Now
|
|
451
472
|
|
|
452
473
|
| Skill | What It Does |
|
|
@@ -463,10 +484,13 @@ ma-scout -> search-lit -> fulltext-retrieval -> design-study ──> write-proto
|
|
|
463
484
|
| **design-study** | Study design review: identifies analysis unit, cohort logic, data leakage risks, comparator design, validation strategy, and reporting guideline fit. |
|
|
464
485
|
| **design-ai-benchmarking** | Design and validity review for benchmarking AI system(s) against a human-expert panel: evaluation-question and arm definition, decoupled multi-dimensional rubrics with anchors, planted calibration probes (positive-control / known-bad / instability / mechanism-contradiction), reviewer-panel construction with per-reviewer randomization, inter-rater reliability targets with separate control-item reliability, LLM-as-judge vs human-as-judge adjudication, construct-independence guards, and a structured JSON rating-export schema. Locks the rubric before data collection. |
|
|
465
486
|
| **model-validation** | Design or audit the clinical-validation study for an engineer-built medical-imaging model (segmentation / classification / detection): patient-level split disjointness and the data-leakage taxonomy, tuning-on-test, internal vs genuine external validation, comparator design, single-run vs multi-seed variance, task-correct metric selection (Metrics Reloaded), test-set sizing, and CLAIM 2024 / TRIPOD+AI / STARD-AI reporting fit. Ships a deterministic split-leakage gate that proves patient disjointness by set arithmetic on the emitted split table. Integrates with MONAI / nnU-Net — does not replace them. |
|
|
466
|
-
| **
|
|
487
|
+
| **preprocess-imaging** | Design or audit the data-preparation stage of a medical-imaging model — DICOM/NIfTI intake, resampling, intensity normalization, and the augmentation plan — so the pipeline is leakage-safe before `model-scaffold` builds the training repo. Emits a declarative preprocessing manifest and a deterministic data-stage leakage gate (`check_preprocessing_leakage`) that catches what the split table cannot see: a dataset-level normaliser fit on non-train data (`NORMALIZATION_LEAKAGE`), a data-fitted transform run before the split (`PREPROCESS_BEFORE_SPLIT`), and a patient's slices crossing splits (`PATIENT_CROSS_SPLIT`). Integrates MONAI / TorchIO transforms; never reimplements them or touches real patient data. |
|
|
488
|
+
| **model-scaffold** | Generate a reproducible, runnable PyTorch training repo for a medical-imaging task — segmentation (U-Net), classification, detection, image-to-image synthesis, self-supervised pretraining, or fine-tuning a pretrained backbone (transfer learning) — the missing middle link between choosing an architecture and validating a trained model. Emits a patient-level seed-locked split as an auditable artifact, a task-appropriate model, train/evaluate scripts that seed every RNG and infer under eval mode, a config, requirements, a reproducibility record, and a Methods stub with VERIFY placeholders (no fabricated numbers). Fine-tuning mode (`--task finetune`) adds a frozen→unfrozen schedule, discriminative learning rates, and a pretrained-weight provenance record (`PRETRAINED.md`), with a MedSAM-adaptation + train-only diffusion-augmentation guide. Reproducibility holds by construction; ships a `check_training_hygiene` AST gate (RNG seeding, eval-mode inference, train-split-only loaders, pretrained-provenance) + a network-free build→validate challenge. Integrates with MONAI / nnU-Net / TorchIO / timm / torchvision for production-grade models. |
|
|
467
489
|
| **architecture-zoo** | "Which architecture for which research question" decision tool: maps task (classification / segmentation / detection / transfer), modality, data scale, and class imbalance to a paper-grounded architecture shortlist. Curates the foundational curriculum (ResNet / DenseNet / EfficientNet / ViT / Swin; U-Net / 3-D U-Net / Attention & Residual U-Net / nnU-Net / Mask R-CNN; SAM/MedSAM / TotalSegmentator / BiomedCLIP / DINO / MAE / SimCLR) — each with core idea, when-to-use, medical-imaging use, reference implementation, validation setup, and the matching model-scaffold template. Advisory; teaches archetypes, not a live SOTA leaderboard. |
|
|
468
490
|
| **model-card** | Generate the documentation an engineer-built medical-imaging model must carry — a Model Card (Mitchell et al. 2019), a Datasheet for its dataset (Gebru et al. 2021), and a METRIC-informed data-quality pass — filled from user-supplied facts (never fabricated), then verify every required section is present and non-empty with a deterministic completeness gate (`check_model_card_complete`). Model Card / Datasheet are documentation standards vendored as templates, not counted reporting checklists. |
|
|
469
491
|
| **model-evaluation** | Compute and report task-correct held-out metrics for a trained medical-imaging model — segmentation (Dice + a boundary metric HD95/NSD, per structure), classification (AUROC + AUPRC + sensitivity/specificity with bootstrap CIs at the deployment prevalence), or detection (FROC/mAP with a stated IoU criterion) — plus calibration and subgroup slices. Emits a per-case table for analyze-stats and gates the metric choice against Metrics Reloaded / CLAIM 2024 (`check_metric_reporting`). Numbers come only from executed code. |
|
|
492
|
+
| **explainability** | Produce or audit the interpretability/explainability analysis of a medical-imaging model — Grad-CAM / Grad-CAM++ / attention-rollout / saliency / integrated-gradients — so it clears the rigor bar: mandatory Adebayo sanity checks (model- and data-randomisation), a quantitative localisation metric against ground truth (IoU / pointing game / Dice) instead of eyeballed examples, a cohort-level result, and attribution framing rather than "proof the model is correct". Emits an explainability-report manifest + a deterministic gate (`check_explainability_report`): `SALIENCY_AS_VALIDATION`, `NO_SANITY_CHECK`, `NO_LOCALIZATION_METRIC` (Major); `INSUFFICIENT_SANITY`, `CHERRY_PICKED_EXAMPLES`, `MISSING_METHOD` (Minor). Integrates captum / pytorch-grad-cam; never reimplements them or touches real patient data. |
|
|
493
|
+
| **radiomics-ml** | Produce or audit a radiomics / tabular clinical-ML study — features → random forest / XGBoost / regularised logistic → clinical outcome — the most common solo-doable clinical-ML workflow (no GPU, no engineer). Emits a pipeline manifest + a deterministic rigor gate (`check_radiomics_ml`): `NO_NESTED_CV`, `HIGH_DIM_LOW_EVENTS`, `SELECTION_OUTSIDE_CV` (Major); `NO_FEATURE_STABILITY`, `NO_CALIBRATION`, `NO_EXTERNAL_VALIDATION` (Minor). pyradiomics/IBSI settings, nested CV with in-fold selection, ICC stability, SHAP, calibration + decision curve, CLEAR/TRIPOD+AI/PROBAST-AI reporting. Integrates scikit-learn / xgboost / pyradiomics; never reimplements them. |
|
|
470
494
|
| **mllm-eval** | Model-agnostic evaluation harness (closed API or open weights) for an LLM/MLLM on a clinical task — radiology report generation, VQA, clinical text extraction — covering the adjudicated reference standard, clinical-efficacy metrics (RadGraph-F1 / CheXbert-F1 beyond BLEU/ROUGE), faithfulness/hallucination, pretraining-contamination, prompt sensitivity, and a reader study; gates the plan with `check_mllm_eval_completeness` and routes the reviewer audit to the MLLM probe. |
|
|
471
495
|
| **intake-project** | Classifies new research projects, summarizes current state, identifies missing inputs, and recommends next steps. |
|
|
472
496
|
| **grant-builder** | Structures grant proposals: significance, innovation, approach, milestones, and consortium roles. |
|
|
@@ -787,15 +811,44 @@ It helps other researchers find the toolkit — and we list it (with your permis
|
|
|
787
811
|
|
|
788
812
|
## Citation
|
|
789
813
|
|
|
790
|
-
If
|
|
791
|
-
|
|
792
|
-
|
|
814
|
+
If MedSci Skills helped produce your manuscript, protocol, or analysis, please cite it —
|
|
815
|
+
software citation is how a tool like this earns academic recognition, and it takes one line.
|
|
816
|
+
|
|
817
|
+
**In your manuscript** (Methods or Acknowledgements — cite the version you actually used):
|
|
818
|
+
|
|
819
|
+
> Reporting-guideline compliance, reference verification, and pre-submission integrity checks
|
|
820
|
+
> were assisted by MedSci Skills (version X.Y.Z; https://github.com/Aperivue/medsci-skills;
|
|
821
|
+
> archived at Zenodo, https://doi.org/10.5281/zenodo.20155321).
|
|
793
822
|
|
|
794
|
-
|
|
823
|
+
**BibTeX** (the software, and the preprint describing its design):
|
|
795
824
|
|
|
796
|
-
|
|
797
|
-
|
|
798
|
-
|
|
825
|
+
```bibtex
|
|
826
|
+
@software{nam_medsci_skills,
|
|
827
|
+
author = {Nam, Yoojin},
|
|
828
|
+
title = {{MedSci Skills: Claude Code Skills for the Medical Research Lifecycle}},
|
|
829
|
+
year = {2026},
|
|
830
|
+
publisher = {Zenodo},
|
|
831
|
+
doi = {10.5281/zenodo.20155321},
|
|
832
|
+
url = {https://github.com/Aperivue/medsci-skills}
|
|
833
|
+
}
|
|
834
|
+
|
|
835
|
+
@article{nam2026agentic,
|
|
836
|
+
author = {Nam, Yoojin and Kim, Namkug},
|
|
837
|
+
title = {{Agentic Skills for Auditable and Reproducible Medical Research Writing:
|
|
838
|
+
An Integrity-gated Architecture for LLM-Assisted Clinical Manuscripts}},
|
|
839
|
+
year = {2026},
|
|
840
|
+
journal = {arXiv preprint arXiv:2606.09500},
|
|
841
|
+
url = {https://arxiv.org/abs/2606.09500}
|
|
842
|
+
}
|
|
843
|
+
```
|
|
844
|
+
|
|
845
|
+
The Zenodo **concept DOI** [10.5281/zenodo.20155321](https://doi.org/10.5281/zenodo.20155321)
|
|
846
|
+
always resolves to the latest release; [`CITATION.cff`](CITATION.cff) carries the machine-readable
|
|
847
|
+
metadata (GitHub's "Cite this repository" button reads it).
|
|
848
|
+
|
|
849
|
+
**Used it in published or in-review work?** Tell us via the
|
|
850
|
+
["Used in research" issue template](https://github.com/Aperivue/medsci-skills/issues/new?template=used-in-research.yml)
|
|
851
|
+
— with your permission it is added to [`docs/citations.md`](docs/citations.md).
|
|
799
852
|
|
|
800
853
|
## Disclaimer
|
|
801
854
|
|
|
@@ -815,8 +868,16 @@ Bundled reporting guideline checklists retain their original Creative Commons li
|
|
|
815
868
|
|
|
816
869
|
Optional dependency: `pdf_to_md.py` uses [pymupdf4llm](https://pymupdf.readthedocs.io) (AGPL-3.0). Not bundled -- installed separately by the user via `pip install pymupdf4llm`.
|
|
817
870
|
|
|
871
|
+
## Star History
|
|
872
|
+
|
|
873
|
+
<a href="https://star-history.com/#Aperivue/medsci-skills&Date">
|
|
874
|
+
<img src="https://api.star-history.com/svg?repos=Aperivue/medsci-skills&type=Date" alt="MedSci Skills star history" width="640">
|
|
875
|
+
</a>
|
|
876
|
+
|
|
818
877
|
## About
|
|
819
878
|
|
|
820
879
|
Built by [Aperivue](https://aperivue.com) -- tools for medical AI research and education.
|
|
821
880
|
|
|
881
|
+
**Runs anywhere the [Agent Skills standard](https://agentskills.io) is supported** -- not Claude Code alone. Claude Code, OpenAI Codex, Cursor, and GitHub Copilot are all verified against each host's own docs: one install, four hosts, no per-host fork ([host compatibility](docs/host_compatibility.md)).
|
|
882
|
+
|
|
822
883
|
If you find this useful, consider giving it a star. It helps other researchers discover these tools.
|
|
@@ -1081,6 +1081,56 @@
|
|
|
1081
1081
|
"size": 1411,
|
|
1082
1082
|
"sha256": "3981d5da7734b90e6905ac329e996b145af82bd6a28ec2a8d42918e9ce077ee5"
|
|
1083
1083
|
},
|
|
1084
|
+
{
|
|
1085
|
+
"path": "skills/explainability/SKILL.md",
|
|
1086
|
+
"size": 6474,
|
|
1087
|
+
"sha256": "4d1c082ad48393adb65203ffbedf63f772fdc521d873f657cbfc81150c7b36de"
|
|
1088
|
+
},
|
|
1089
|
+
{
|
|
1090
|
+
"path": "skills/explainability/references/explainability_guide.md",
|
|
1091
|
+
"size": 4203,
|
|
1092
|
+
"sha256": "24bc06181e54801b148f142f9312fe0eede9f540a4e57810701506681e254b4b"
|
|
1093
|
+
},
|
|
1094
|
+
{
|
|
1095
|
+
"path": "skills/explainability/scripts/check_explainability_report.py",
|
|
1096
|
+
"size": 10783,
|
|
1097
|
+
"sha256": "4071a93aacf509f73a44b2c7f6423b5722aab3d44c79e9a8680845454261d47d"
|
|
1098
|
+
},
|
|
1099
|
+
{
|
|
1100
|
+
"path": "skills/explainability/scripts/check_explainability_report_challenge/expected/strong.txt",
|
|
1101
|
+
"size": 391,
|
|
1102
|
+
"sha256": "317018eb7546f77b2de39ff1b73c9e82584a9d65c25eb91a940de76a4c7d2ec5"
|
|
1103
|
+
},
|
|
1104
|
+
{
|
|
1105
|
+
"path": "skills/explainability/scripts/check_explainability_report_challenge/expected/weak.txt",
|
|
1106
|
+
"size": 876,
|
|
1107
|
+
"sha256": "de5c090c1ee4c632d7d59bc0d5f20fac8d28342b74b68eb51066677658245451"
|
|
1108
|
+
},
|
|
1109
|
+
{
|
|
1110
|
+
"path": "skills/explainability/scripts/check_explainability_report_challenge/fixture/report_strong.json",
|
|
1111
|
+
"size": 238,
|
|
1112
|
+
"sha256": "b835871c1d98313f736575faf7f6adbb5cfabe4e3dfd1abe1279976c8c5f1867"
|
|
1113
|
+
},
|
|
1114
|
+
{
|
|
1115
|
+
"path": "skills/explainability/scripts/check_explainability_report_challenge/fixture/report_weak.json",
|
|
1116
|
+
"size": 163,
|
|
1117
|
+
"sha256": "dfde5db357a83660f941fdde957bb3a9af25f60d26300322f04b3981b13c429a"
|
|
1118
|
+
},
|
|
1119
|
+
{
|
|
1120
|
+
"path": "skills/explainability/scripts/check_explainability_report_challenge/problem.md",
|
|
1121
|
+
"size": 805,
|
|
1122
|
+
"sha256": "1b667c138ac54e38614e438efdb995adba208588007e95eed6f374233ff6dcdc"
|
|
1123
|
+
},
|
|
1124
|
+
{
|
|
1125
|
+
"path": "skills/explainability/scripts/check_explainability_report_challenge/verify.sh",
|
|
1126
|
+
"size": 2005,
|
|
1127
|
+
"sha256": "5246897dc616e04fa1260e9b20d9970e1499f95ebb52034bba59482d401c1156"
|
|
1128
|
+
},
|
|
1129
|
+
{
|
|
1130
|
+
"path": "skills/explainability/skill.yml",
|
|
1131
|
+
"size": 3569,
|
|
1132
|
+
"sha256": "5b3e135825470392867dc851627d9899bc512d9a5ee52556c205eea8fae4edb1"
|
|
1133
|
+
},
|
|
1084
1134
|
{
|
|
1085
1135
|
"path": "skills/fill-icmje-coi/SKILL.md",
|
|
1086
1136
|
"size": 9997,
|
|
@@ -2818,8 +2868,13 @@
|
|
|
2818
2868
|
},
|
|
2819
2869
|
{
|
|
2820
2870
|
"path": "skills/model-scaffold/SKILL.md",
|
|
2821
|
-
"size":
|
|
2822
|
-
"sha256": "
|
|
2871
|
+
"size": 9237,
|
|
2872
|
+
"sha256": "d7e736178cb60d2d13cef9174474297bd54bc6d4d1f3862023d802ff0c034a95"
|
|
2873
|
+
},
|
|
2874
|
+
{
|
|
2875
|
+
"path": "skills/model-scaffold/references/finetuning_guide.md",
|
|
2876
|
+
"size": 6208,
|
|
2877
|
+
"sha256": "b7817d53eeeb506b573beaae1889fa863dd81286d474ab99b46b38ad046b32e7"
|
|
2823
2878
|
},
|
|
2824
2879
|
{
|
|
2825
2880
|
"path": "skills/model-scaffold/references/training_guide.md",
|
|
@@ -2828,13 +2883,13 @@
|
|
|
2828
2883
|
},
|
|
2829
2884
|
{
|
|
2830
2885
|
"path": "skills/model-scaffold/scripts/check_training_hygiene.py",
|
|
2831
|
-
"size":
|
|
2832
|
-
"sha256": "
|
|
2886
|
+
"size": 14095,
|
|
2887
|
+
"sha256": "57542a54dac607e89e7f13e5025bd2ce6b32e629064a18cbb0f0c15fdfc6181c"
|
|
2833
2888
|
},
|
|
2834
2889
|
{
|
|
2835
2890
|
"path": "skills/model-scaffold/scripts/scaffold.py",
|
|
2836
|
-
"size":
|
|
2837
|
-
"sha256": "
|
|
2891
|
+
"size": 51482,
|
|
2892
|
+
"sha256": "150abbc4864a92913b202893962c29cda8ec6ce1918f35558aa5b02514b6a0cb"
|
|
2838
2893
|
},
|
|
2839
2894
|
{
|
|
2840
2895
|
"path": "skills/model-scaffold/scripts/scaffold_challenge/expected/split_assignment.csv",
|
|
@@ -2848,18 +2903,18 @@
|
|
|
2848
2903
|
},
|
|
2849
2904
|
{
|
|
2850
2905
|
"path": "skills/model-scaffold/scripts/scaffold_challenge/problem.md",
|
|
2851
|
-
"size":
|
|
2852
|
-
"sha256": "
|
|
2906
|
+
"size": 3226,
|
|
2907
|
+
"sha256": "950af6c956f73b491b2bee102d884765aaeb783b5842eb428ac2ad81b49f67e8"
|
|
2853
2908
|
},
|
|
2854
2909
|
{
|
|
2855
2910
|
"path": "skills/model-scaffold/scripts/scaffold_challenge/verify.sh",
|
|
2856
|
-
"size":
|
|
2857
|
-
"sha256": "
|
|
2911
|
+
"size": 6396,
|
|
2912
|
+
"sha256": "77d81387647abf00de4b03f91a7c7bf9bef38d9679b148762c906eaccfee9f73"
|
|
2858
2913
|
},
|
|
2859
2914
|
{
|
|
2860
2915
|
"path": "skills/model-scaffold/skill.yml",
|
|
2861
|
-
"size":
|
|
2862
|
-
"sha256": "
|
|
2916
|
+
"size": 3772,
|
|
2917
|
+
"sha256": "aaae59edb50c50b01be2aaca3c94b63ce76af067ff25750248517c5ce91ccf41"
|
|
2863
2918
|
},
|
|
2864
2919
|
{
|
|
2865
2920
|
"path": "skills/model-validation/SKILL.md",
|
|
@@ -3181,6 +3236,56 @@
|
|
|
3181
3236
|
"size": 1753,
|
|
3182
3237
|
"sha256": "ce4996820b24c230584890c501860c8d37dfd07521195954077d2fdd1a26f419"
|
|
3183
3238
|
},
|
|
3239
|
+
{
|
|
3240
|
+
"path": "skills/preprocess-imaging/SKILL.md",
|
|
3241
|
+
"size": 7068,
|
|
3242
|
+
"sha256": "e17b73b0760194d4e6e3924ec2a0452601adc27122e7455cc5f2993d5af1d0d0"
|
|
3243
|
+
},
|
|
3244
|
+
{
|
|
3245
|
+
"path": "skills/preprocess-imaging/references/preprocessing_guide.md",
|
|
3246
|
+
"size": 5023,
|
|
3247
|
+
"sha256": "fdab6d435ff6bce7b6787319acda9b0b160411bc2e1109e15154bbb8a68c8282"
|
|
3248
|
+
},
|
|
3249
|
+
{
|
|
3250
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage.py",
|
|
3251
|
+
"size": 12992,
|
|
3252
|
+
"sha256": "dcce1493a29122ed4259adc116ed7ad239b11ee1f45a10f6f7eb94698b3efa67"
|
|
3253
|
+
},
|
|
3254
|
+
{
|
|
3255
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/expected/clean.txt",
|
|
3256
|
+
"size": 370,
|
|
3257
|
+
"sha256": "bc1efcfc1d8575d78a0448752e60a3b58d654a5bc1a7b11360e4d9ba690665ac"
|
|
3258
|
+
},
|
|
3259
|
+
{
|
|
3260
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/expected/leak.txt",
|
|
3261
|
+
"size": 1030,
|
|
3262
|
+
"sha256": "fe046c81f20b34a6ff3fd6858f7832e786106ed56635b39f7e413b4853d4b7e8"
|
|
3263
|
+
},
|
|
3264
|
+
{
|
|
3265
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/fixture/manifest_clean.json",
|
|
3266
|
+
"size": 800,
|
|
3267
|
+
"sha256": "e01ff6d09aae6f76d3b2b1cd05c333d66fd53499a2a2d1296380eefde720c515"
|
|
3268
|
+
},
|
|
3269
|
+
{
|
|
3270
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/fixture/manifest_leak.json",
|
|
3271
|
+
"size": 881,
|
|
3272
|
+
"sha256": "a55389654128c577e16eefb3c6b8f85d8a47bdc06a817619f13036a1e1ebe4b4"
|
|
3273
|
+
},
|
|
3274
|
+
{
|
|
3275
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/problem.md",
|
|
3276
|
+
"size": 903,
|
|
3277
|
+
"sha256": "042ddd4736133ccf63192ea65c4faa714b2854ec9bb704f29ca2f401cd4658ee"
|
|
3278
|
+
},
|
|
3279
|
+
{
|
|
3280
|
+
"path": "skills/preprocess-imaging/scripts/check_preprocessing_leakage_challenge/verify.sh",
|
|
3281
|
+
"size": 2188,
|
|
3282
|
+
"sha256": "9201d264720627f7281f43bcf3b1093bf6940f67acd7c299c8b3ac64fb58b564"
|
|
3283
|
+
},
|
|
3284
|
+
{
|
|
3285
|
+
"path": "skills/preprocess-imaging/skill.yml",
|
|
3286
|
+
"size": 3518,
|
|
3287
|
+
"sha256": "c6edc7ac6421438d4e0b64cd416202ac9f91092870ea1cb8531ba15fda7d5c72"
|
|
3288
|
+
},
|
|
3184
3289
|
{
|
|
3185
3290
|
"path": "skills/present-paper/SKILL.md",
|
|
3186
3291
|
"size": 33518,
|
|
@@ -3311,6 +3416,56 @@
|
|
|
3311
3416
|
"size": 1372,
|
|
3312
3417
|
"sha256": "81caa10775246f2db3f5e08948292f63076d44ffc039d8eec764f1d46aab275f"
|
|
3313
3418
|
},
|
|
3419
|
+
{
|
|
3420
|
+
"path": "skills/radiomics-ml/SKILL.md",
|
|
3421
|
+
"size": 8273,
|
|
3422
|
+
"sha256": "cc65bfd50db2b6e9b6681e1c9649b306490cd5dd74192bfdf5316362b71f141b"
|
|
3423
|
+
},
|
|
3424
|
+
{
|
|
3425
|
+
"path": "skills/radiomics-ml/references/radiomics_ml_guide.md",
|
|
3426
|
+
"size": 6707,
|
|
3427
|
+
"sha256": "29bf2b2b847c488ddb33e50cd21ea0e2cd4937a1a62ab2655f31227a0e42215a"
|
|
3428
|
+
},
|
|
3429
|
+
{
|
|
3430
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml.py",
|
|
3431
|
+
"size": 10179,
|
|
3432
|
+
"sha256": "b1cfbd9991aa8871d5aabc528c9aa22eecf8e46cb5983872bca2359229223840"
|
|
3433
|
+
},
|
|
3434
|
+
{
|
|
3435
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml_challenge/expected/strong.txt",
|
|
3436
|
+
"size": 369,
|
|
3437
|
+
"sha256": "c73c776e6558d9a78dacd3c63ae1cc3dee183d9e1a2f27d99b1e0db70f44997e"
|
|
3438
|
+
},
|
|
3439
|
+
{
|
|
3440
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml_challenge/expected/weak.txt",
|
|
3441
|
+
"size": 1408,
|
|
3442
|
+
"sha256": "3bb90f31587e94f3f2b267874a9ac58cce5adf3436254bc206092e0acfb86cd7"
|
|
3443
|
+
},
|
|
3444
|
+
{
|
|
3445
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml_challenge/fixture/pipeline_strong.json",
|
|
3446
|
+
"size": 322,
|
|
3447
|
+
"sha256": "83d128fe0b728ef52a1fd98ef5b5bd478c3e48455cdb07d8040f9d130df81cee"
|
|
3448
|
+
},
|
|
3449
|
+
{
|
|
3450
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml_challenge/fixture/pipeline_weak.json",
|
|
3451
|
+
"size": 313,
|
|
3452
|
+
"sha256": "2fa1a3f98f952faf2e6e1faf1bdbf4675a44dc88679dcfaf18a4f32eb47ba9e7"
|
|
3453
|
+
},
|
|
3454
|
+
{
|
|
3455
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml_challenge/problem.md",
|
|
3456
|
+
"size": 894,
|
|
3457
|
+
"sha256": "9a6abad4e12f9b4e9290eb305fb9ad04127fa1cde03961a38ac4ccb5431a2100"
|
|
3458
|
+
},
|
|
3459
|
+
{
|
|
3460
|
+
"path": "skills/radiomics-ml/scripts/check_radiomics_ml_challenge/verify.sh",
|
|
3461
|
+
"size": 1988,
|
|
3462
|
+
"sha256": "b9e55aaf9b29f3e810b837fc373c7a212dc5f778a49bac11854bdb50ed84a184"
|
|
3463
|
+
},
|
|
3464
|
+
{
|
|
3465
|
+
"path": "skills/radiomics-ml/skill.yml",
|
|
3466
|
+
"size": 3790,
|
|
3467
|
+
"sha256": "9b99b87add656873d3f7277ce12490ffea36d919e9538bc4f4fbe1ebbfedc18e"
|
|
3468
|
+
},
|
|
3314
3469
|
{
|
|
3315
3470
|
"path": "skills/render-pdf-doc/SKILL.md",
|
|
3316
3471
|
"size": 7490,
|
|
@@ -3758,8 +3913,8 @@
|
|
|
3758
3913
|
},
|
|
3759
3914
|
{
|
|
3760
3915
|
"path": "skills/self-review/scripts/check_scope_coherence.py",
|
|
3761
|
-
"size":
|
|
3762
|
-
"sha256": "
|
|
3916
|
+
"size": 12989,
|
|
3917
|
+
"sha256": "2444bce9c5bd2fc088c43eae97c1fc5791ae4da44f428a2caa0559b976dbe622"
|
|
3763
3918
|
},
|
|
3764
3919
|
{
|
|
3765
3920
|
"path": "skills/self-review/scripts/check_supplement_hygiene.py",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"schema_version": 1,
|
|
3
|
-
"version": "5.
|
|
3
|
+
"version": "5.16.0",
|
|
4
4
|
"owned_skills": [
|
|
5
5
|
"academic-aio",
|
|
6
6
|
"add-journal",
|
|
@@ -16,6 +16,7 @@
|
|
|
16
16
|
"deidentify",
|
|
17
17
|
"design-ai-benchmarking",
|
|
18
18
|
"design-study",
|
|
19
|
+
"explainability",
|
|
19
20
|
"fill-icmje-coi",
|
|
20
21
|
"fill-protocol",
|
|
21
22
|
"find-cohort-gap",
|
|
@@ -39,8 +40,10 @@
|
|
|
39
40
|
"orchestrate",
|
|
40
41
|
"peer-review",
|
|
41
42
|
"polish-language",
|
|
43
|
+
"preprocess-imaging",
|
|
42
44
|
"present-paper",
|
|
43
45
|
"publish-skill",
|
|
46
|
+
"radiomics-ml",
|
|
44
47
|
"render-pdf-doc",
|
|
45
48
|
"replicate-study",
|
|
46
49
|
"review-paper",
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"_comment": "AUTO-GENERATED by scripts/gen_skills_catalog_json.py from each skills/<slug>/SKILL.md + skill.yml. Machine-readable skill catalog (single source of truth) consumed by external surfaces such as the aperivue.com storefront to gate skill-list completeness. Do not hand-edit; CI gate: python3 scripts/gen_skills_catalog_json.py --check.",
|
|
3
|
-
"skill_count":
|
|
3
|
+
"skill_count": 54,
|
|
4
4
|
"categories": [
|
|
5
5
|
{
|
|
6
6
|
"key": "literature_references",
|
|
@@ -32,11 +32,14 @@
|
|
|
32
32
|
"label": "Model Engineering & Validation",
|
|
33
33
|
"slugs": [
|
|
34
34
|
"architecture-zoo",
|
|
35
|
+
"explainability",
|
|
35
36
|
"mllm-eval",
|
|
36
37
|
"model-card",
|
|
37
38
|
"model-evaluation",
|
|
38
39
|
"model-scaffold",
|
|
39
|
-
"model-validation"
|
|
40
|
+
"model-validation",
|
|
41
|
+
"preprocess-imaging",
|
|
42
|
+
"radiomics-ml"
|
|
40
43
|
]
|
|
41
44
|
},
|
|
42
45
|
{
|
|
@@ -235,6 +238,15 @@
|
|
|
235
238
|
"maturity": "official",
|
|
236
239
|
"description": "Study design and validity review for radiology and medical AI research."
|
|
237
240
|
},
|
|
241
|
+
{
|
|
242
|
+
"slug": "explainability",
|
|
243
|
+
"category": "model_engineering",
|
|
244
|
+
"category_label": "Model Engineering & Validation",
|
|
245
|
+
"layer": "D",
|
|
246
|
+
"owner_domain": "model_validation",
|
|
247
|
+
"maturity": "official",
|
|
248
|
+
"description": "Produce or audit the interpretability/explainability analysis of a medical-imaging model — Grad-CAM / Grad-CAM++ / attention-rollout / saliency / integrated-gradients — so it clears the rigor bar a re…"
|
|
249
|
+
},
|
|
238
250
|
{
|
|
239
251
|
"slug": "fill-icmje-coi",
|
|
240
252
|
"category": "submission_journals",
|
|
@@ -404,7 +416,7 @@
|
|
|
404
416
|
"layer": "B",
|
|
405
417
|
"owner_domain": "model_development",
|
|
406
418
|
"maturity": "official",
|
|
407
|
-
"description": "Generate a reproducible, runnable PyTorch training repo for a medical-imaging task — segmentation, classification, detection, image-to-image synthesis,
|
|
419
|
+
"description": "Generate a reproducible, runnable PyTorch training repo for a medical-imaging task — segmentation, classification, detection, image-to-image synthesis, self-supervised pretraining, or fine-tuning a pr…"
|
|
408
420
|
},
|
|
409
421
|
{
|
|
410
422
|
"slug": "model-validation",
|
|
@@ -442,6 +454,15 @@
|
|
|
442
454
|
"maturity": "official",
|
|
443
455
|
"description": "Academic English consistency linting and non-native (ESL) language polish for medical manuscripts."
|
|
444
456
|
},
|
|
457
|
+
{
|
|
458
|
+
"slug": "preprocess-imaging",
|
|
459
|
+
"category": "model_engineering",
|
|
460
|
+
"category_label": "Model Engineering & Validation",
|
|
461
|
+
"layer": "D",
|
|
462
|
+
"owner_domain": "model_validation",
|
|
463
|
+
"maturity": "official",
|
|
464
|
+
"description": "Design or audit the data-preparation stage of a medical-imaging model — DICOM/NIfTI intake, resampling and intensity normalisation, and the augmentation plan — so the pipeline is leakage-safe before m…"
|
|
465
|
+
},
|
|
445
466
|
{
|
|
446
467
|
"slug": "present-paper",
|
|
447
468
|
"category": "presentation_tooling",
|
|
@@ -460,6 +481,15 @@
|
|
|
460
481
|
"maturity": "official",
|
|
461
482
|
"description": "Convert a personal agent skill into a distributable, open-source-ready skill."
|
|
462
483
|
},
|
|
484
|
+
{
|
|
485
|
+
"slug": "radiomics-ml",
|
|
486
|
+
"category": "model_engineering",
|
|
487
|
+
"category_label": "Model Engineering & Validation",
|
|
488
|
+
"layer": "D",
|
|
489
|
+
"owner_domain": "model_validation",
|
|
490
|
+
"maturity": "official",
|
|
491
|
+
"description": "Produce or audit a radiomics / tabular clinical-ML study — imaging or clinical features → any classical learner (penalised logistic [LASSO / ridge / elastic-net], SVM, k-NN, naive Bayes, LDA/QDA, deci…"
|
|
492
|
+
},
|
|
463
493
|
{
|
|
464
494
|
"slug": "render-pdf-doc",
|
|
465
495
|
"category": "presentation_tooling",
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "medsci-skills",
|
|
3
|
-
"version": "5.
|
|
3
|
+
"version": "5.16.0",
|
|
4
4
|
"description": "MedSci Skills — a medical/scientific research skill suite for AI coding agents (Claude Code, Codex, Cursor, Copilot). The npm package is a terminal-friendly installer shortcut; the canonical distribution remains the GitHub repository and the Claude Code plugin marketplace.",
|
|
5
5
|
"license": "SEE LICENSE IN LICENSE",
|
|
6
6
|
"homepage": "https://github.com/Aperivue/medsci-skills#readme",
|
|
@@ -0,0 +1,109 @@
|
|
|
1
|
+
---
|
|
2
|
+
name: explainability
|
|
3
|
+
description: >
|
|
4
|
+
Produce or audit the interpretability/explainability analysis of a medical-imaging model —
|
|
5
|
+
Grad-CAM / Grad-CAM++ / attention-rollout / saliency / integrated-gradients — so it clears the rigor
|
|
6
|
+
bar a reviewer expects: mandatory Adebayo sanity checks (model- and data-randomisation), a
|
|
7
|
+
quantitative localisation metric against ground truth (IoU / pointing game / Dice) instead of eyeballed
|
|
8
|
+
examples, a cohort-level result rather than cherry-picked cases, and attribution framing rather than
|
|
9
|
+
"proof the model is correct". Emits an explainability-report manifest and a deterministic rigor gate.
|
|
10
|
+
Integrates captum / pytorch-grad-cam; it does not reimplement them, and never runs a model on real
|
|
11
|
+
patient data.
|
|
12
|
+
triggers: explainability, interpretability, saliency, saliency map, grad-cam, gradcam, grad-cam++, attention map, attention rollout, integrated gradients, captum, pytorch-grad-cam, heatmap, class activation map, CAM, feature attribution, sanity check, Adebayo, model randomization, localization metric, pointing game, IoU with ground truth, XAI, explainable AI, model looks at, faithfulness
|
|
13
|
+
tools: Read, Write, Edit, Bash, Grep, Glob
|
|
14
|
+
model: inherit
|
|
15
|
+
---
|
|
16
|
+
|
|
17
|
+
# Explainability Skill
|
|
18
|
+
|
|
19
|
+
## Purpose
|
|
20
|
+
|
|
21
|
+
A saliency / Grad-CAM heat-map is the **most over-interpreted artifact** in medical-imaging AI: a
|
|
22
|
+
colourful map over the lesion is routinely presented as proof the model "looks at the right thing."
|
|
23
|
+
Adebayo et al. (*NeurIPS* 2018) showed many saliency methods produce visually convincing maps that are
|
|
24
|
+
**independent of the model's learned weights and of the labels** — so they explain nothing. This skill
|
|
25
|
+
produces an explainability analysis that clears the rigor bar, and audits an existing one, so the map
|
|
26
|
+
is trustworthy before it reaches a manuscript (CLAIM 2024 / TRIPOD+AI interpretability items).
|
|
27
|
+
|
|
28
|
+
It sits alongside evaluation in the lane: `/architecture-zoo` → `/preprocess-imaging` →
|
|
29
|
+
`/model-scaffold` → `/model-validation` → `/model-evaluation` + **explainability** →
|
|
30
|
+
`/write-paper` + `/check-reporting`. It **integrates** captum / pytorch-grad-cam (referenced in the
|
|
31
|
+
plan); it does not reimplement them and never runs a model on real patient data.
|
|
32
|
+
|
|
33
|
+
## When to use
|
|
34
|
+
- You produced (or are about to produce) saliency / Grad-CAM / attention maps and want them reported
|
|
35
|
+
to the standard a reviewer expects.
|
|
36
|
+
- You want to audit an explainability analysis for the four failure modes below.
|
|
37
|
+
|
|
38
|
+
## When NOT to use
|
|
39
|
+
- Discrimination / calibration metrics → `/model-evaluation` then `/analyze-stats`.
|
|
40
|
+
- Split or preprocessing leakage → `/model-validation` / `/preprocess-imaging`.
|
|
41
|
+
- LLM/MLLM faithfulness & hallucination → `/mllm-eval`.
|
|
42
|
+
- Reimplementing captum / pytorch-grad-cam → out of scope (this skill wires and audits them).
|
|
43
|
+
|
|
44
|
+
## The four failure modes (what the gate enforces)
|
|
45
|
+
1. **Saliency as validation.** A map is *attribution*, not proof the model is correct or that the
|
|
46
|
+
relationship is causal. Frame it as "where signal is attributed", never as "the model is right".
|
|
47
|
+
2. **No sanity check.** Run the Adebayo **model-randomisation** and **data-randomisation** tests. A map
|
|
48
|
+
that survives neither is uninterpretable; both axes are the minimum bar.
|
|
49
|
+
3. **No quantitative localisation.** If you claim the map localises the finding, measure it — IoU /
|
|
50
|
+
pointing game / Dice against ground-truth masks — do not eyeball a few examples.
|
|
51
|
+
4. **Cherry-picked examples.** Report a cohort-level result, not a handful of hand-picked cases.
|
|
52
|
+
|
|
53
|
+
## Workflow
|
|
54
|
+
|
|
55
|
+
### Phase 1 — Produce the maps (integrate, don't reimplement)
|
|
56
|
+
Choose the method for the architecture (`references/explainability_guide.md`): Grad-CAM / Grad-CAM++
|
|
57
|
+
for CNNs, attention-rollout for ViTs, integrated-gradients / SHAP for attribution. Wire captum or
|
|
58
|
+
pytorch-grad-cam; do not write a new CAM implementation.
|
|
59
|
+
|
|
60
|
+
### Phase 2 — Sanity-check and quantify
|
|
61
|
+
- Run the **model-parameter randomisation** and **data (label) randomisation** tests (Adebayo 2018);
|
|
62
|
+
a faithful map degrades when the model/labels are randomised.
|
|
63
|
+
- Compute a **quantitative localisation metric** against ground-truth masks (IoU / pointing game /
|
|
64
|
+
Dice) over the cohort — not a visual impression.
|
|
65
|
+
|
|
66
|
+
### Phase 3 — Emit the explainability-report manifest
|
|
67
|
+
```json
|
|
68
|
+
{
|
|
69
|
+
"method": "grad-cam++",
|
|
70
|
+
"n_examples": 200,
|
|
71
|
+
"cohort_level": true,
|
|
72
|
+
"localization_metric": "iou",
|
|
73
|
+
"localization_value": 0.63,
|
|
74
|
+
"sanity_checks": ["model_randomization", "data_randomization"],
|
|
75
|
+
"interpretation": "localization"
|
|
76
|
+
}
|
|
77
|
+
```
|
|
78
|
+
`interpretation`: `attribution` / `localization` / `faithfulness` (descriptive) — never
|
|
79
|
+
`validation` / `causal` (overclaim).
|
|
80
|
+
|
|
81
|
+
### Phase 4 — Gate the report (deterministic)
|
|
82
|
+
```bash
|
|
83
|
+
python3 scripts/check_explainability_report.py --manifest explainability_report.json --strict
|
|
84
|
+
```
|
|
85
|
+
Verdicts: `SALIENCY_AS_VALIDATION`, `NO_SANITY_CHECK`, `NO_LOCALIZATION_METRIC` (Major);
|
|
86
|
+
`INSUFFICIENT_SANITY`, `CHERRY_PICKED_EXAMPLES`, `MISSING_METHOD` (Minor). The verdict is reproduced
|
|
87
|
+
by rule on the manifest, never asserted from prose.
|
|
88
|
+
|
|
89
|
+
## Integration
|
|
90
|
+
- **`/model-evaluation`** — explainability accompanies the held-out metrics as a secondary analysis.
|
|
91
|
+
- **`/self-review`** `ai_overclaiming` / `image_synthesis` probes audit saliency overclaiming in a
|
|
92
|
+
finished manuscript; this skill *produces* the rigorous analysis they look for.
|
|
93
|
+
- **`/check-reporting`** — the manifest documents the CLAIM 2024 / TRIPOD+AI interpretability items.
|
|
94
|
+
|
|
95
|
+
## Anti-Hallucination
|
|
96
|
+
|
|
97
|
+
- **Never fabricate saliency maps, localisation metrics, or sanity-check results.** Every value in the
|
|
98
|
+
manifest comes from the researcher's executed XAI code — never invented. This skill designs and
|
|
99
|
+
audits the analysis; it does not run a model on real patient data.
|
|
100
|
+
- **Never present a saliency map as proof of model correctness or causation.** A map is attribution;
|
|
101
|
+
claiming it validates the model is the overclaim this skill exists to prevent (`SALIENCY_AS_VALIDATION`).
|
|
102
|
+
- **Never report an explainability-audit "pass" without running `check_explainability_report.py`.** The
|
|
103
|
+
rigor verdict is reproduced deterministically, never asserted from prose.
|
|
104
|
+
- **Integrate, don't reimplement.** Reference captum / pytorch-grad-cam; do not write a new CAM /
|
|
105
|
+
attribution implementation or claim results for one.
|
|
106
|
+
|
|
107
|
+
## Reproducible challenge
|
|
108
|
+
`scripts/check_explainability_report_challenge/` ships a synthetic weak/strong report pair with a
|
|
109
|
+
network-free `verify.sh` wired into the skill's validation commands.
|