medsci-skills 6.2.0 → 6.3.0

package/skills/peer-review/SKILL.md

@@ -91,7 +91,7 @@ not installed.
 
 ### Phase 2: Manuscript Analysis
 
-1. **Read the manuscript PDF** thoroughly — Abstract, Methods, Results, Discussion, Tables, Figures.
+1. **Read the manuscript PDF** thoroughly (all sections, Tables, Figures); then, before the detailed checks, follow the provisional-record instructions at the top of `references/review_draft_template.md` and save the record in the review's `work/` folder (finalize it in Phase 3).
 2. **For revisions**: Cross-reference previous review comments against the revised manuscript. Do
    **not** trust the response letter's "we added / we changed X" at face value — the source of truth is
    the revised body. When you have both the author response and the revised manuscript as text/`.docx`,
@@ -179,14 +179,14 @@ output (Major / Minor comments, Confidential Comments to the Editor, Major #1).
 
 | | Fires when | Module (`references/domain-probes/`) |
 |---|---|---|
-| **2A** Systematic Review / Meta-Analysis | (P0) plus 19-probe checklist (P1–P19) **only when manuscript type is "Systematic Review", "Meta-Analysis", or "Systematic Review and Meta-Analysis"** | `sr_ma.md` |
+| **2A** Systematic Review / Meta-Analysis | (P0) plus 20-probe checklist (P1–P20) **only when manuscript type is "Systematic Review", "Meta-Analysis", or "Systematic Review and Meta-Analysis"** | `sr_ma.md` |
 | **2B** Survival / Prognostic Model | **only when manuscript involves time-to-event outcomes** (OS, DFS, LRFS, DMFS, RFS, PFS, time-to-recurrence) **or prognostic model development** (Cox proportional hazards, DeepSurv, DeepHit, Random Survival Forest, nomogram development/validation, multi-state or multi-outcome survival cascade, risk-stratification with cutoff-based phenotyping) | `survival_prognostic.md` |
 | **2C** Radiomics / Feature-Reproducibility | **only when the manuscript maps radiomic feature reliability/reproducibility or feature stability** (test-retest, noise sensitivity, ICC-based reproducibility), runs an **acquisition–reconstruction parameter sweep** (tube voltage, tube current, bin width, reconstruction kernel, slice thickness, iterative reconstruction), or claims that **reliability/robustness/harmonization-based feature filtering** (e.g., ComBat, ICC thresholding) improves a downstream clinical task or transports across scanners/centers/vendors | `radiomics.md` |
 | **2D** Narrative / Review-Article | (RV1–RV9) **only when the manuscript is a Review / narrative review / primer / state-of-the-art / educational review** — i.e., a non-systematic synthesis rather than original research | `narrative_review.md` |
 | **2E** Observational / Confounding | (O1–O18) **only when the manuscript is an observational study** (cohort, case-control, cross-sectional, health-screening / registry) **whose central claim is an adjusted exposure–outcome association** estimated by covariate adjustment rather than randomization | `observational_confounding.md` |
 | **2G** AI / ML Overclaiming | an AI/ML **primary study** (diagnostic, prognostic, triage, detection) makes a clinical claim in the Title/Abstract/Conclusion — generalizable, outperforms clinicians, deployment-ready, can replace a reader | `ai_overclaiming.md` |
 | **2H** RCT / Intervention-Trial | (RC0–RC7) **only when the manuscript is a randomised controlled trial** (parallel-group, crossover, cluster, stepped-wedge) whose claim is that an intervention *causes* an outcome difference | `rct_trial.md` |
-| **2I** Diagnostic-Accuracy / Reader-Study | (D1–D12) **only when the manuscript is a diagnostic test accuracy (DTA) primary study** — an index test against a reference standard — including **multi-reader multi-case (MRMC)** reader studies (AI-vs-reader or modality comparison) | `diagnostic_accuracy.md` |
+| **2I** Diagnostic-Accuracy / Reader-Study | (D1–D15) **only when the manuscript is a diagnostic test accuracy (DTA) primary study** — an index test against a reference standard — including **multi-reader multi-case (MRMC)** reader studies (AI-vs-reader or modality comparison) | `diagnostic_accuracy.md` |
 | **2J** Case-Report | (CR1–CR9) **only when the manuscript is a case report, a case series, or a small single-patient clinical narrative** | `case_report.md` |
 | **2K** Image-Synthesis / Cross-Modality Generation | (IS1–IS4) **only when the manuscript synthesizes one imaging modality from another** (MRI→PET, MRI→CT, CT→MRI, non-contrast→contrast, low-dose→full-dose) using a generative model (GAN/PatchGAN, diffusion, U-Net/Swin-UNet, CycleGAN) **and** frames the synthetic image as carrying functional/molecular information or as a substitute for the unavailable real target modality | `image_synthesis.md` |
 | **2L** Fairness / Equity / Subgroup-performance | (EQ0–EQ6) **only when the manuscript makes (or implies) a claim that an AI/ML model, score, or test performs adequately across a heterogeneous population** (generalizable / deployment-ready / "works for patients") **or presents subgroup analyses as evidence of fairness/equity** | `equity_fairness.md` |
@@ -322,7 +322,7 @@ After drafting, verify mechanically:
    - ≥50% of Minor requests use hedged forms ("I'd suggest," "could," "would help") rather than imperative ("must," bare "Please [verb]")
    - General Comments names ≥2 specific strengths before listing concerns
    - At most 1 typo/grammar Minor Comment, only if in formal section or systematic
-10. **SR-MA-specific QC** (if Phase 2A applied): Confirm the P0 internal-consistency gate was run before any fabrication claim. For each P1–P19 probe used, verify the corresponding Major comment cites source PMID + source page/table reference + verbatim quote, and that no probe lead was promoted to a finding without source confirmation (leads-vs-findings discipline). Reviews citing extraction errors without source-page reference are not actionable for authors.
+10. **SR-MA-specific QC** (if Phase 2A applied): Confirm the P0 internal-consistency gate was run before any fabrication claim. For each probe used that makes a study-level factual claim (extraction, eligibility, an included study's design), verify the corresponding Major comment cites source PMID + source page/table reference + verbatim quote; a synthesis-method finding (e.g. P20) cites the manuscript location and the methodological reason instead. Check that no probe lead was promoted to a finding without source confirmation (leads-vs-findings discipline). Reviews citing extraction errors without source-page reference are not actionable for authors.
 11. **Radiomics-reproducibility QC** (if Phase 2C applied): If an acquisition-parameter sweep predicts an outcome from its own grid axes (R1 design-grid circularity) or the substantive result is a cross-domain failure framed as success (R3), confirm the recommendation reflects design-level severity and is not softened to a reporting fix. Where a model × threshold/cohort grid yields a few p < 0.05, confirm the multiplicity / expected-false-positive count is named (R4), not deferred to "statistical review needed."
 12. **Review-article QC** (if Phase 2D applied): Confirm RV1–RV9 are reflected — in particular that novelty/value-add (RV1) is raised for a saturated topic and that gap-filling (RV8) is present, not just error-spotting. Verify SANRA is used as an appraisal aid, not over-enforced as a reporting guideline (no PRISMA demand on a narrative review; only RV3 is SANRA-aligned and phrased as a suggestion). Verify every suggested addition uses "consider adding" phrasing (no "must cite"), is source-confirmed, and that preprints are labeled as preprints (not equated with peer-reviewed guidelines). Confirm Phase 2F was run for the recommendation: when RV1 novelty is a Major in a saturated space with no distinct contribution, the recommendation is escalated toward Reject (the contribution IS the product — weak novelty is unfixable-in-current-form), not defaulted to the revision/Reconsider tier.
 13. **AI/method/review priority QC**: Before a Major Revision (or reconsider-tier) recommendation, confirm Phase 2F

package/skills/peer-review/references/domain-probes/ai_overclaiming.md

@@ -22,9 +22,9 @@ A 7-probe checklist (AO1–AO7, with AO0 as a gate) for medical-AI/ML primary st
 - If the generalizability claim outruns the external evidence → recommend softening to the evidence ("validated at one external site") and moving multi-setting generalizability to a stated limitation + next step. MAJOR candidate when it is a headline claim; MINOR when it is a single qualifier in the Discussion.
 
 **AO2 — Superiority language against overlapping or under-powered comparison**:
-- Flag "outperforms", "superior to", "beats", "can replace [clinician/radiologist]" when (a) the model vs comparator 95% CIs overlap, (b) no test of the *difference* is reported (two separate AUCs are not a comparison), or (c) the comparison rests on a small test set / few readers.
+- Flag "outperforms", "superior to", "beats", "can replace [clinician/radiologist]" when (a) no estimate of the *difference* with its CI, or test of it, is reported (two separate AUCs are not a comparison, and overlapping or non-overlapping marginal CIs do not settle it, because paired estimates can differ clearly despite overlap), (b) for a superiority claim, the CI of the difference includes no difference (then statistical superiority is unsupported), or the claim is of clinically meaningful superiority while the CI shows at most a trivial gain (then "statistically higher, with limited clinical importance" is the supportable wording), or (c) the claim goes beyond the comparison made (a replacement claim from a small test set or a few readers, or from a task that differs from clinical reading). A replacement claim is judged by its own justification (superiority, non-inferiority within a pre-specified margin, or equivalence) and its clinical trade-offs; a non-inferiority result can support it with a CI that includes zero.
 - Ask for the difference in the metric with its CI and a paired test of that difference, not two standalone estimates.
-- If the difference is not statistically supported → recommend reframing from "outperforms" to "comparable to" (still a meaningful result). MAJOR when a superiority/replacement claim is the headline; otherwise MINOR.
+- If the difference is not statistically supported → recommend dropping the superiority language and reporting the difference with its CI. Do **not** suggest "comparable to" or "equivalent" in its place: a non-significant difference is not evidence of similarity unless the CI lies within a pre-specified, clinically justified margin (`diagnostic_accuracy.md` D6). The supportable wording is that no difference was detected and that a difference of the size the CI still admits cannot be excluded. MAJOR when a superiority/replacement claim is the headline; otherwise MINOR.
 
 **AO3 — Comparison-frame mismatch (model task ≠ human task)**:
 - When a model-vs-clinician comparison drives a claim, verify the two performed the **same task on the same inputs under the same constraints**: same images/inputs available, same time budget, same question asked, same decision point.
@@ -78,7 +78,7 @@ Co-apply when a study claims **clinical utility, deployment, or decision impact*
 > "The Conclusion states the model 'generalizes across institutions,' but external validation appears limited to a single site ([Methods, External validation]). I'd suggest softening this to the evidence — e.g., 'validated at one external site' — and framing multi-institution generalizability as a stated limitation and a next step. If a broader claim is intended, an external set spanning multiple sites/vendors would be needed to support it."
 
 **Output template (AO2 / AO3 example)**:
-> "The 'outperforms radiologists' claim rests on a comparison whose 95% CIs for model and reader [metric] overlap ([Figure/Table]), and no test of the difference is reported; the reader task also differs from the model's in [inputs/time] ([Methods/Table]). I'd suggest (a) reporting the difference in [metric] with its CI and a paired test rather than two separate estimates, and (b) stating explicitly which clinical task the comparison establishes. If the difference is not statistically supported, reframing from 'outperforms' to 'comparable to' would be both defensible and still a meaningful result."
+> "The 'outperforms radiologists' claim rests on a comparison whose 95% CIs for model and reader [metric] overlap ([Figure/Table]), and no test of the difference is reported; the reader task also differs from the model's in [inputs/time] ([Methods/Table]). I'd suggest (a) reporting the difference in [metric] with its CI and a paired test rather than two separate estimates, and (b) stating explicitly which clinical task the comparison establishes. If the difference is not statistically supported, I'd suggest removing 'outperforms' and reporting the difference with its CI; 'comparable to' would need a pre-specified margin that the CI stays within."
 
 **Output template (AO5 example)**:
 > "Table 2 appears to report the single best cross-validation fold; because best-fold figures are optimistically biased, I'd suggest reporting the mean across folds with its SD (or a 95% CI). Relatedly, the sensitivity/specificity depend on a classification threshold I could not find stated — reporting the operating point and how it was chosen (e.g., Youden's J on the training folds only, to avoid tuning on the test data) would make these numbers reproducible. Finally, since the training set was balanced to 1:1 while the clinical prevalence is much lower, accuracy (and PPV/NPV) on a re-balanced set may not reflect performance at the true base rate; stating the evaluation class distribution and adding AUROC/AUPRC with CIs alongside sensitivity/FNR at the stated threshold, with PPV/NPV on a prevalence-representative holdout, would make the clinical cost of a miss visible."

package/skills/peer-review/references/domain-probes/clinical_prediction_model.md

@@ -6,25 +6,26 @@
        - self-review: Anticipated Major / Minor Comments (Fatal / Fixable) mapped to category letters.
      Do NOT edit one copy only — run `python3 scripts/check_domain_probe_sync.py --sync`. -->
 
-# Clinical Prediction-Model probes (CP1–CP6)
+# Clinical Prediction-Model probes (CP1–CP8)
 
-A 6-probe checklist for **cross-sectional / observational clinical prediction models** — a binary or multinomial outcome predicted from a covariate set, reported under TRIPOD / TRIPOD+AI, usually as a **nested predictor-set comparison** ("does adding marker X to a base model improve prediction of Y?"). This is the diagnostic/prognostic *prediction* counterpart to `survival_prognostic.md` (time-to-event) and complements the `observational_confounding.md` probes (a prediction model still has an analysis unit (O8) and can be over-adjusted (O7)). Route here when the manuscript develops or compares prediction models and reports discrimination (AUC/C-statistic), calibration, or decision-curve / net-benefit analysis.
+An 8-probe checklist for **cross-sectional / observational clinical prediction models** — a binary or multinomial outcome predicted from a covariate set, reported under TRIPOD / TRIPOD+AI, usually as a **nested predictor-set comparison** ("does adding marker X to a base model improve prediction of Y?"). This is the diagnostic/prognostic *prediction* counterpart to `survival_prognostic.md` (time-to-event) and complements the `observational_confounding.md` probes (a prediction model still has an analysis unit (O8) and can be over-adjusted (O7)). Route here when the manuscript develops or compares prediction models and reports discrimination (AUC/C-statistic), calibration, or decision-curve / net-benefit analysis.
 
 **CP1 — Apparent vs optimism-corrected performance**:
 - Discrimination, **calibration** (slope / intercept / calibration plot), and **decision-curve / net-benefit** all suffer optimism when computed in the same sample used to fit the model. A manuscript that optimism-corrects *only* the C-statistic (bootstrap / cross-validation) but reports calibration and net benefit **in-sample** has corrected one axis and left two apparent.
 - An apparent calibration slope near unity (≈ 1.0) does **not** establish out-of-sample calibration — it is the expected in-sample result and says little about transportability.
 - Calibration and DCA computed in-sample must be **labelled "apparent"**, or the optimism correction must be extended to them (e.g. bootstrap the calibration slope/intercept and the net-benefit curves). In-sample calibration/DCA described as "well calibrated" / "clinically useful" without the apparent caveat → MAJOR.
+- A non-significant Hosmer–Lemeshow test does not establish calibration: it depends on sample size and grouping and says nothing about the direction or size of miscalibration. Ask for a calibration curve with its uncertainty, plus calibration intercept and slope, each labelled apparent or validated. In a small cohort a smoothed curve with wide uncertainty and an honest caveat is enough; do not demand every measure. Cross-link `survival_prognostic.md` S7 / T2.
 
 **CP2 — The two-null distinction (incremental value vs marginal effect)**:
 - A "negative" prediction study conflates two different nulls that have different evidentiary status:
-  - **Incremental value negligible** — a *well-powered* statement: ΔAUC ≈ 0, optimism correction does not favour the larger model (and may reverse sign toward the parsimonious one), net benefit does not exceed the base model. This is a genuine "adding X does not improve prediction."
+  - **Incremental value negligible** — a *well-powered* statement: ΔAUC ≈ 0, optimism correction does not favour the larger model (and may reverse sign toward the parsimonious one), net benefit does not exceed the base model. It supports "adding X does not improve prediction" only when the uncertainty (the CI of ΔAUC or of the net-benefit difference) excludes a clinically meaningful gain; otherwise the supportable wording is "no improvement was detected".
   - **Marginal predictor effect null** — an *underpowered* statement: the predictor's adjusted OR/coefficient has a confidence interval that still admits a clinically relevant effect (e.g. OR up to ~1.7). This is "we could not exclude an effect," not "there is no effect."
 - A manuscript that collapses both into a blanket "X did not predict Y" overstates the marginal-effect arm. Flag when a negative conclusion rests on a non-significant marginal OR whose CI admits a relevant effect, without separating it from the (better-powered) incremental-value finding → MAJOR. The honest reframe reports both explicitly.
 
 **CP3 — Events-per-variable (EPV) per nested model**:
-- Report EPV for **each** nested model, not just the full cohort. A model with 6 covariates and 60 events (EPV 10) is adequately supported; the same 6 covariates in an age≥60 subgroup with 13 events (EPV ≈ 2) is separation-prone and unstable.
-- Penalization (Firth / ridge / LASSO) is justified and should be stated when EPV < 10; subgroup or interaction models with EPV in the low single digits are **separation-affected and must be descriptive-only**, not presented as estimated effects.
-- Subgroup/interaction effect estimates from a model with EPV ≲ 5, presented as if reliably estimated, or no EPV disclosure for the nested models → MAJOR (subgroup) / MINOR (EPV simply not reported but adequate).
+- Report events per candidate parameter for **each** nested model, not just the full cohort, counting every candidate parameter considered (dummy levels, non-linear terms, variables screened out), not only those retained. EPV is a rough screen, not a sufficiency rule: whether the sample supports a model depends on the number of candidate parameters, the outcome proportion and the expected model performance (sample-size criteria for model development: Riley et al., *BMJ* 2020;368:m441). The same 6 covariates fitted in an age≥60 subgroup with 13 events (EPV ≈ 2) remain separation-prone and unstable.
+- Penalization (Firth / ridge / LASSO) reduces overfitting and should be stated, but it does not rescue a sample that is too small for the candidate set, and the amount of shrinkage is itself unstable in small samples. For subgroup or interaction models with EPV in the low single digits, assess separation, the estimation method, the uncertainty and the model complexity; unstable estimates should be reported as exploratory with their intervals, and no reliable effect should be inferred from them.
+- Low EPV triggers an assessment of the sample-size justification, separation, uncertainty and validation. MAJOR when demonstrated instability undermines the claim (for example, subgroup or interaction effects presented as reliably estimated); request clarification when adequacy cannot be assessed from what is reported.
 
 **CP4 — Net benefit is a model comparison, not a policy endorsement**:
 - A decision curve showing the model exceeds treat-all / treat-none across a threshold range is a **model-vs-default** statement about ranking, not evidence that the underlying **imaging / test / screening strategy** should be adopted in practice. Net benefit is computed on the same outcomes and says nothing about cost, downstream harm, or whether the test should be ordered.
@@ -37,8 +38,20 @@ A 6-probe checklist for **cross-sectional / observational clinical prediction mo
 
 **CP6 — Validation-nomenclature conflation (development step vs test step)**:
 - Distinguish the **development** step (including any internal cross-validation or bootstrap optimism correction) from a **held-out / external test** step. Flag phrasing that merges them: "developed with external validation" where a single external set is used *as* development data; "trained and validated on the external cohort"; an "external validation set" that was used for tuning/feature selection.
-- Cross-validation and bootstrapping are development-time optimism corrections, **not** external validation; calling them "validation" overclaims out-of-sample evidence. Require an explicit statement of which data trained, which tuned, and which was touched only once for the final estimate.
+- Bootstrap optimism correction and ordinary within-population cross-validation are legitimate **internal validation** when every development step (variable selection, tuning, threshold choice) is repeated inside each resample; they estimate optimism in the development population and are **not** evidence of transportability. Internal–external cross-validation that holds out whole centres or studies does speak to performance across settings, limited to the settings represented. Proper internal validation does support qualified conclusions about new patients from the development population. The overclaim is presenting it as external validation or as evidence of transportability to untested populations or settings ("externally validated", "generalisable across centres", "ready for clinical use" with no external data); ask for "internally validated" and a clear description of the resampling workflow. Resampling only to count how often each variable is selected measures selection **stability**, not performance, and is not validation. Require an explicit statement of which data trained and which tuned, and, when a held-out or external set is claimed, that it did not influence model fitting, selection, tuning or thresholds (pre-specified evaluation analyses and uncertainty estimates on the frozen model are fine).
 - → MAJOR when the overclaim props up a generalisability/deployment-readiness conclusion; a clarify-request when the data flow is actually clean but the wording is loose.
 
+**CP7 — Prediction claim, association evidence (entry check)**:
+- Decide first whether the paper asks an **association** question (is X independently related to Y after adjustment?) or a **prediction** question (how accurately can risk be estimated for a new patient?). A significant adjusted OR or HR does not show useful prediction, and a model built for prediction does not license independent-effect or causal language for its coefficients.
+- Flag a prediction claim ("predicts", "can be used to predict", "a predictive marker") supported only by association statistics with no discrimination, calibration or validation, and the reverse: causal interpretations or inferential claims about predictor effects that the design does not support or that ignore selection and shrinkage (describing how a predictor moves the fitted prediction is fine). A paper that states both objectives separately and analyses each appropriately is fine.
+- Severity: MAJOR when the headline prediction claim rests on association statistics alone. Cross-link CP2 (two nulls), CP4, and the task-formulation audit.
+
+**CP8 — How the predictors were chosen and handled, and whether the reported model is the fitted one**:
+- The common shortcuts fail in different ways. **Univariable P-value screening** discards predictors that matter only jointly and keeps chance associations. **Stepwise or other automated selection** gives unstable models and biased coefficients; repeating the whole selection procedure inside internal validation is needed for an honest performance estimate, though it does not remove the instability or the coefficient bias. **Dichotomising a continuous predictor** at a data-derived "optimal" cut-off loses information and inflates apparent performance. **Complete-case analysis**, or excluding patients whose outcome or predictor could not be obtained, reduces the sample and may introduce selection bias or restrict applicability, depending on the missingness mechanism, who was excluded and the intended deployment population. Ask what was done and whether internal validation repeated it.
+- Severity follows the consequence, not the label; disclosure or penalization does not by itself make the finding MINOR. MAJOR when selection or categorisation is data-driven, is not repeated within the internal-validation resamples, the frozen pipeline has no independent evaluation, and the performance claim depends on it; MINOR when it is disclosed and its effect is shown to be small. Evaluating the frozen model on untouched external data is valid without repeating the selection there.
+- Check that the model in the paper is the model that was fitted: the variables and coefficients of the final model (or its equation or score) match the multivariable results table, and any re-fitting step between the two is described. Keep an explanatory adjusted-OR table distinct from the predictive equation. A table–model mismatch that changes which predictors the model uses → MAJOR.
+- Reporting completeness is not low risk of bias: a fully ticked TRIPOD or TRIPOD+AI checklist can describe a high-risk design. Judge risk of bias separately (PROBAST or its AI extension), and check that the checklist's page references point to text that exists.
+- Cross-link `observational_confounding.md` O2 (adjustment-set selection in causal models, a different reason), O5 (missing data), O12 (data-driven thresholds), `model_development.md` MD1 / MD2 / MD7, `survival_prognostic.md` T4, CP3.
+
 **Output template (CP1 / CP2 example)**:
-> "Discrimination is bootstrap optimism-corrected, but the calibration slope (0.99) and the net-benefit curves are computed in the development sample and described as 'well calibrated' and 'clinically useful' without the apparent caveat — an in-sample slope near 1.0 is expected and does not establish out-of-sample calibration. I'd suggest labelling the calibration and decision-curve results 'apparent' or extending the optimism correction to them. Relatedly, the Conclusion states that the marker 'did not predict' the outcome, but two distinct results are merged: the incremental-value analysis is well powered (ΔAUC ≈ 0, optimism correction favours the parsimonious model), whereas the marginal adjusted OR has a confidence interval that still admits an effect up to ~1.7 (underpowered). I'd report these separately — 'adding the marker did not improve prediction' is supported; 'the marker has no association' is not."
+> "Discrimination is bootstrap optimism-corrected, but the calibration slope (0.99) and the net-benefit curves are computed in the development sample and described as 'well calibrated' and 'clinically useful' without the apparent caveat — an in-sample slope near 1.0 is expected and does not establish out-of-sample calibration. I'd suggest labelling the calibration and decision-curve results 'apparent' or extending the optimism correction to them. Relatedly, the Conclusion states that the marker 'did not predict' the outcome, but two distinct results are merged: the incremental-value estimate is near zero (ΔAUC ≈ 0, optimism correction favours the parsimonious model), but its CI must be examined before a meaningful gain can be excluded, whereas the marginal adjusted OR has a confidence interval that still admits an effect up to ~1.7 (inconclusive). I'd report these separately — 'adding the marker did not meaningfully improve discrimination as measured by AUC' is supported if the CI of ΔAUC excludes a meaningful gain (calibration and net benefit need their own assessment before a broader claim); 'the marker has no association' is not."

package/skills/peer-review/references/domain-probes/diagnostic_accuracy.md

@@ -6,7 +6,7 @@
        - self-review: Anticipated Major / Minor Comments (Fatal / Fixable) mapped to category letters.
      Do NOT edit one copy only — run `python3 scripts/check_domain_probe_sync.py --sync`. -->
 
-# Diagnostic-accuracy / reader-study probes (D1–D12)
+# Diagnostic-accuracy / reader-study probes (D1–D15)
 
 A checklist for **diagnostic test accuracy (DTA) primary studies** — an index test against a reference standard, including **multi-reader multi-case (MRMC)** reader studies (e.g., AI-vs-reader or modality-comparison). These probes complement (do not replace) the generic Phase 2 issue checklist and the STARD / QUADAS-2 items; they target the biases QUADAS-2 names and the MRMC design/variance issues a reader study adds. Pairs the `analyze-stats` `table-standards/table-types/reader_study.md` table and the `make-figures` `exemplar_plots/mrmc_roc.md` figure. (For a DTA **meta-analysis**, use `sr_ma.md`.)
 
@@ -21,7 +21,8 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 
 **D3 — Blinding of index and reference interpretation**:
 - Was the **index test** interpreted blind to the reference standard and to clinical information that would not be available at the point of use, and vice versa? Unblinded interpretation (review bias) inflates agreement.
-- For an AI index test, was the **operating threshold pre-specified** (not tuned on the test set)? A test-set-tuned threshold reported as the result is an optimistic-validation finding (pair with `exemplar_reviews/optimistic_validation_reporting.md`).
+- For any index test read against a threshold (an AI score, or a quantitative imaging measure such as ADC, SUV or a histogram percentile), was the **operating threshold pre-specified**, or chosen on the same data that report its accuracy (an "optimal" cut-off by Youden index or similar)? A data-derived cut-off flatters the data it came from. A test-set-tuned threshold reported as the result is an optimistic-validation finding (pair with `exemplar_reviews/optimistic_validation_reporting.md`).
+- When the study develops the threshold itself, internal resampling must repeat the threshold selection, and an external or temporal confirmation is needed only for a transportability claim. A threshold-development study that labels its cut-off exploratory is not a defect. Cross-link `model_development.md` MD2, `ai_overclaiming.md` AO5(b), `survival_prognostic.md` S4.
 
 **D4 — Indeterminate / uninterpretable results**:
 - How were **indeterminate, uninterpretable, or intermediate** index results handled — excluded, or counted (intention-to-diagnose)? Silently dropping them inflates accuracy. State the rule and report an **intention-to-diagnose** sensitivity analysis (non-diagnostic counted as wrong). Undisclosed exclusion of indeterminates → MAJOR.
@@ -33,6 +34,7 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 **D6 — MRMC analysis + estimand**:
 - Does the analysis account for **both reader and case variability** (Obuchowski–Rockette / Dorfman–Berbaum–Metz or an equivalent multi-reader model), or does it treat readers as fixed / pool reads as if independent? Ignoring reader variance understates uncertainty and is a MAJOR statistical flaw for a generalising claim.
 - Is the **estimand** clear: reader-averaged vs a fixed specific reader; per-**patient** vs per-**lesion** (clustered) unit; superiority vs **non-inferiority** with a pre-specified margin? Are **per-reader** results shown alongside the reader-averaged estimate (a single averaged AUC can hide one weak reader)? A non-inferiority claim with no pre-specified margin, or a clustered design analysed as independent, → MAJOR.
+- **Not significant is not equivalent.** A non-significant difference between two tests (or a test and its reduced-dose, faster or cheaper version) does not show they perform alike; in a small sample the CI of the difference usually still admits a clinically important loss. "Comparable", "non-inferior", or a replacement claim that rests on similarity needs a clinically justified margin and a CI that excludes the unacceptable difference: one bound for non-inferiority, both bounds for equivalence. A replacement claim that rests on demonstrated superiority needs no margin, only evidence for the advantage and an account of the clinical trade-offs (D13). Without a margin, the supportable wording is that no difference was detected and a loss of the size the CI admits cannot be excluded. A missing prospective sample-size statement does not by itself make a precise estimate inconclusive; the CI decides. Post hoc (observed) power computed from the observed effect is a transformation of the P value and adds nothing, so ask for the difference with its CI instead. Severity: a replacement or equivalence headline resting only on a non-significant P → MAJOR; loose "similar" wording for a secondary finding → MINOR. Cross-link `ai_overclaiming.md` AO2, `sr_ma.md` P9, `clinical_prediction_model.md` CP2.
 
 **D7 — Index-test-as-enrollment-criterion circularity**:
 - Cross-check the **inclusion criteria** against the **index / proposed test**. When a study proposes or validates a diagnostic or classification instrument *and* uses a threshold of that **same** instrument (or a component of it) as an enrollment criterion, the validation is circular: the spectrum is built into the design, and sensitivity/specificity (or the instrument's apparent discrimination) are inflated by construction.
@@ -43,6 +45,7 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 - Cross-check the **exclusion criteria drawn in the participant flow diagram** (STROBE/STARD flow) against the **exclusion list in the Methods prose**. A criterion that appears in one but not the other (a count in the flow with no prose rationale, or a prose exclusion not reflected in the flow boxes) is a reporting inconsistency a co-reviewer catches by reading the two side by side.
 - For an **imaging** study, check that **modality-specific safety contraindications** and **device/artifact exclusions** are enumerated where applicable: MR safety (pacemaker/implant, claustrophobia), iodinated/gadolinium **contrast** contraindications (renal function, allergy, pregnancy), and **image-quality/artifact** exclusions (motion, metal artifact, incomplete coverage). Silent omission of these categories in a prospective imaging cohort understates the selected spectrum.
 - Severity: a flow-vs-prose exclusion mismatch is MAJOR when it changes the analytic-N or the eligible spectrum; missing modality-safety/artifact exclusion categories is MINOR–MAJOR depending on how much of the source population they remove. The fix is a reconciled exclusion list (flow == prose) plus an explicit modality-safety/artifact exclusion enumeration.
+- Beyond exclusions, check that the Results report what the Methods promised: the pre-specified primary metric first, with its estimate and 95% CI (numerator and denominator for proportions; sample size and analysis unit for AUC and other metrics); results at the pre-specified threshold, not only at a new "optimal" one; planned versus achieved sample size; the number verified by each reference-standard method; the stated unit of analysis. An undisclosed outcome-dependent switch of the primary metric or threshold → MAJOR. For a documented amendment, ask when it was made relative to access to the results and why: a change made before the relevant results were seen is a clarify-request, while an outcome-informed change makes the affected analyses exploratory, however well it is documented.
 
 **D9 — Confidence-weighted / rating reader study: unweighted baseline AUC must accompany the weighted primary**:
 - When a reader study's novelty is a **confidence-weighted** (or rating-collapsed) score used as the ROC/AUC predictor, the **unweighted binary-call AUC** must be reported side-by-side (as a sensitivity analysis). Without it, you cannot tell whether the weighting *created* the result or hid an estimator fragility (e.g. a folded/non-monotonic encoding that collapses `real/5` with `ai/1`).
@@ -65,9 +68,25 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 - Confirm the **assistance paradigm** is stated and matches the claim — AI shown **concurrently**, as a **second reader** after an independent read, or as **triage / pre-selection**. These measure different quantities and are not interchangeable; a concurrent-display study cannot support a second-reader deployment claim. Where the same readers read both arms, the washout / ordering controls of D5 apply to the arm comparison too.
 - Severity: an adoption/workflow claim with no assisted arm, or a pooled assisted delta with no AI-correct / AI-incorrect decomposition, → MAJOR (the pooled number cannot exclude automation-bias harm); an unstated assistance paradigm or a missing experience-stratified breakdown → MINOR pending clarification. Boundary: D5/D6 own the MRMC design and variance mechanics; D12 owns what the arms *mean* and the assisted-arm harm decomposition.
 
+**D13 — Intended use and role of the test → the metric and design that can carry the claim**:
+- Read the intended use off the Introduction and the Conclusion: where the test sits in the clinical pathway (**replacement** of an existing test, **triage** before it, **add-on** after it, or surveillance / staging) and which decision it changes. The role decides which error matters more, which population must be sampled, and which comparison answers the question.
+- Check that the headline metric fits the role and that both error directions are reported. A rule-out claim rests on sensitivity and NPV, a rule-in claim on specificity and PPV, with predictive values at a prevalence that matches the intended setting. A replacement or add-on claim needs a comparison with the existing pathway, in the same patients where feasible, not two standalone accuracies. Replacement does not always require a formal non-inferiority design, but the comparison must exist (D6 covers "not significant ≠ equivalent").
+- Check the conclusion against the role: per-lesion detection in a surgical cohort used to claim replacement of surveillance imaging, or a high-sensitivity result used to claim rule-in, overreaches.
+- Severity: MAJOR only when the headline role claim (replace, triage, add-on, rule out, rule in) is not supported by the design and the metric; an unstated role with an otherwise coherent design is a clarify-request. Cross-link D2 (spectrum), D6 (estimand), D12 (which arm carries the claim), `clinical_prediction_model.md` CP5 (decision horizon), `ai_overclaiming.md` AO4.
+
+**D14 — A significant group difference is not diagnostic performance, and a separable difference is not necessarily a meaningful one**:
+- A P value for the difference in a quantitative marker between diseased and non-diseased groups (mean ADC, SUV, a texture feature) tests the specified between-group hypothesis, not the classification of an individual patient; with enough patients a heavily overlapping distribution is "highly significant". When a diagnostic claim rests on a group comparison, ask for the measures the claim needs (the distributions, discrimination as AUC with CI, accuracy at a stated threshold with the 2×2 table) or for the claim to be narrowed to an association. An exploratory biomarker study can resolve this by narrowing its claim rather than by adding analyses.
+- For a quantitative measure, put differences in the context of measurement error in the same units: within-subject SD, repeatability coefficient or limits of agreement from a repeatability or reproducibility setting that matches use (same scanner, different scanner, different reader). A representative accuracy study of a single-measurement protocol already carries that protocol's measurement variability, so it does not need a separate repeatability study. Ask for repeatability evidence when the claim depends on it: consistency of classification on repeat measurement, robustness across scanners, sites or readers, or a change within one patient, where the change must exceed the repeatability of that change for the measurement scheme used. ICC, in either its consistency or its absolute-agreement form, is a ratio driven by between-subject spread and does not express error in measurement units; a high ICC from re-reading the same images says nothing about scan–rescan or cross-scanner variation.
+- Severity: MAJOR when a diagnostic or individual-level claim rests only on a significant group difference, or when the paper claims that change in an individual is reliably distinguishable from measurement error without evidence for it (a decision threshold below the repeatability coefficient can still be valid if its error rates are stated and validated); MINOR when the claim depends on measurement error that is not reported. Cross-link `ai_overclaiming.md` AO7 (benchmark variability never quantified), `observational_confounding.md` O18 (pseudoreplication in agreement studies).
+
+**D15 — One metric chosen from many (histogram percentiles × sequences × readers)**:
+- When the index test is one of many candidate measures (histogram percentiles, several sequences or parameters, several readers or ROIs), ask which metric was primary and whether it was fixed in advance, whether every tested candidate is reported (a supplementary table suffices), how multiplicity was handled for the claim being made, what rationale supports the chosen metric, and whether it was confirmed in independent data. Under the global null, 20 independent tests each at α = 0.05 have a 1 − 0.95^20 ≈ 64% probability of at least one false positive.
+- When candidate measures are combined through a fitted model, co-apply `clinical_prediction_model.md` and distinguish development from validation. A fixed, pre-specified ratio or composite index stays in this diagnostic-accuracy framework; assess any data-driven construction or threshold choice under D3.
+- Severity: MAJOR when a post hoc selected metric is the headline and the selection is not disclosed; a transparently exploratory screen with all candidates reported is not a defect. Cross-link `model_development.md` MD2, `ai_overclaiming.md` AO5, `observational_confounding.md` O17.
+
 **Output template (D2 / D6 example)**:
 > "The study uses a case-control (two-gate) design — confirmed cases versus healthy controls — rather than a consecutive series of patients in whom the diagnosis was suspected. This typically overestimates accuracy and does not reflect the intended-use spectrum, so I'd read the reported sensitivity/specificity as proof-of-concept rather than clinical accuracy, and suggest tempering the Abstract accordingly. Separately, the reader study reports a single reader-averaged AUC; because readers are a sample, I'd suggest an MRMC analysis (e.g., Obuchowski–Rockette) that accounts for both reader and case variance, with per-reader estimates shown and the unit of analysis (per-patient vs per-lesion) stated."
 
-**Discipline — leads vs findings (applies to D1–D12)**:
+**Discipline — leads vs findings (applies to D1–D15)**:
 - A verification/blinding/spectrum concern from a quick scan is a **lead until Methods and the participant flow are read together** — distinguish under-reporting (ask to clarify) from a true design bias (MAJOR).
 - Anchor each comment to the exact bias (partial vs differential verification; single- vs two-gate; reader-averaged vs fixed-reader; per-patient vs per-lesion) and the location. Keep severity tied to what the flaw does: two-gate sampling, incorporation bias, or ignoring reader variance is design/analysis-level (MAJOR, often Major #1); an unreported reading-order detail is a clarify-request.

package/skills/peer-review/references/domain-probes/sr_ma.md

@@ -6,9 +6,9 @@
        - self-review: Anticipated Major / Minor Comments (Fatal / Fixable) mapped to category letters.
      Do NOT edit one copy only — run `python3 scripts/check_domain_probe_sync.py --sync`. -->
 
-# Systematic Review / Meta-Analysis probes (P0–P19)
+# Systematic Review / Meta-Analysis probes (P0–P20)
 
-Internal-consistency-first gate (P0) plus a 19-probe checklist (P1–P19). These probes complement (do not replace) the generic Phase 2 issue checklist.
+Internal-consistency-first gate (P0) plus a 20-probe checklist (P1–P20). These probes complement (do not replace) the generic Phase 2 issue checklist.
 
 **P0 — Internal-consistency-first gate (run before P1; gates any fabrication claim)**:
 - Before alleging fabrication on a manuscript that "feels AI-generated", reproduce the headline pooled statistics, paired study counts (k), and subgroup counts directly from the extracted data table (or supplement included-studies table).
@@ -83,6 +83,7 @@ Internal-consistency-first gate (P0) plus a 19-probe checklist (P1–P19). These
 - Cross-check the **figure-generating data matrix** (the R/Python script that draws the traffic-light) against the **supplementary RoB table** cell-by-cell. A transcription drift between the two surfaces as a figure-vs-table mismatch even when each row's printed total looks fine.
 - The single source of truth is the **primary reviewer assessment form** (the filled extraction/appraisal sheet), not a value hardcoded in a plotting script. When they disagree, regenerate the figure from the assessment form and re-derive the table; then re-sync the table, caption, and any prose sentence that names per-item losses.
 - Severity: a row-sum vs total mismatch (or figure vs table mismatch) in a submitted artifact is MAJOR (it reads as a data-handling error). Motivating case: an NOS figure drew one study with seven stars but labeled it 6/9 because the plotting script's comparability cell contradicted the primary appraisal form.
+- Beyond reconciliation, the text should say which risk-of-bias and applicability domains were most often high or unclear and what that means for the pooled estimates; a traffic-light figure alone leaves the reader to infer it. Missing interpretation alone is MINOR. Do not sum QUADAS-2 domains into a score.
 
 **P13 — Included-study ↔ reference-list completeness**:
 - In an SR/MA, **every study in the characteristics table (Table 1) must be cited as a numbered reference.** A manuscript that lists included studies only as "First-author Year" labels in Table 1, while the reference list holds only methodology/guideline citations, will draw an "included studies not cited" reviewer flag.
@@ -122,6 +123,12 @@ Internal-consistency-first gate (P0) plus a 19-probe checklist (P1–P19). These
 - Also flag validation-type / analysis-unit mislabels (5-fold CV recorded as train/validation; N scans counted as N patients). Absence of a per-study 2×2 / sens–spec table is itself a PRISMA-DTA reporting MAJOR.
 - Lead: sample K OA included studies; per study confirm metric TYPE + value + that the extracted design satisfies the paper's own eligibility + that any tabled CI appears in the source. Severity scales with the sampled error rate.
 
+**P20 — DTA pooling model and threshold effect**:
+- Sensitivity and specificity from the same studies are correlated and may trade off across thresholds. The default synthesis is a bivariate random-effects model (or the equivalent HSROC model), which also yields the summary ROC curve; separate univariate pooling of sensitivity and specificity ignores that correlation. A univariate logistic random-effects fallback is acceptable when the bivariate model is not estimable (few studies, sparse cells, non-convergence), provided the paper says so and why, as `/meta-analysis` also allows. Do not assert that univariate pooling always gives narrower intervals.
+- Heterogeneity in a DTA review is not captured by I² for each margin alone. Judge a threshold effect from what the studies actually used (the explicit thresholds, and whether the study points fall along an ROC-shaped curve in SROC space), not from a significance test. A Spearman correlation between sensitivity and false-positive rate (positive under a threshold effect; equivalently, negative between sensitivity and specificity) is descriptive: its absence does not exclude a threshold effect and its presence does not prove one.
+- Check that within-study choices were pre-specified: which threshold, reader or analysis unit was taken when a study reported several, and how indeterminate results were handled (cross-link P2, P11, `diagnostic_accuracy.md` D4).
+- Severity: first ask for the rationale and model diagnostics. MAJOR when the chosen synthesis is inappropriate for the data or the undescribed threshold or reader selection could materially change the conclusions; MINOR when an appropriate fallback or selection rule is simply unexplained.
+
 **Output template (P1 cell-swap example)**:
 > "I spot-checked [Author Year] (PMID [...]) against the source paper and found that the values in Figure X are swapped. The source paper reports external-test sensitivity A% / specificity B% (n=N); the manuscript forest entries place [num1/denom1] in the sensitivity slot (which is the source's specificity numerator/denominator) and [num2/denom2] in the specificity slot (which is the source's sensitivity)."
 

package/skills/peer-review/references/review_draft_template.md

@@ -11,13 +11,20 @@ Confidential-to-Editor block carries the recommendation and must never reach the
 Verify the split on the rendered proof, not on this draft — the portal's two free-text
 boxes are adjacent and unvalidated.
 
+**Provisional editor record** (Phase 2 step 1 sends you here before the detailed checks). After
+the first full read, write elements (1), (2) and (4) of the editor block below as a provisional record in
+`work/` (a fixability call may be "unclear" at this point). Finalize the block after
+verification. If the final recommendation differs from the provisional one, in either direction,
+add one line to the record naming the new fact or the corrected reasoning. Keep both versions:
+together they are the reviewer's calibration record.
+
 ```markdown
 # {manuscript_id} — Review Draft
 
 **Manuscript**: {title}
 **Journal**: {journal}
 **Type**: {Original Research | Review | Technical Note | ...}
-**Recommendation**: {Major Revision | Minor Revision}
+**Recommendation**: {the journal's own options, e.g. Accept | Minor Revision | Major Revision | Reject & resubmit | Reject}
 
 ---
 
@@ -27,7 +34,15 @@ boxes are adjacent and unvalidated.
 
 ## CONFIDENTIAL COMMENTS TO THE EDITOR
 
-{100-150 words: summary + strengths + key concerns + fatal flaw hierarchy if applicable + recommendation}
+{100-150 words, in this order:
+ (1) the manuscript's main claim, in one sentence;
+ (2) the major issues ranked by their effect on that claim, each graded as in
+     `reviewer_calibration/recommendation_calibration.md`: fixable with existing data / fixable
+     with additional work on the existing material (state the scope) / not fixable in this
+     submission / unclear;
+ (3) the balance of contribution against limitations, including strengths;
+ (4) the recommendation, consistent with (2): an unfixable issue that undercuts the main claim
+     governs it, unless a narrower contribution still stands on its own.}
 **Clinical Impact**: {High/Moderate/Low} — {1 sentence on implications}
 
 ---
@@ -43,7 +58,7 @@ Major strengths:
 2. {Specific strength}
 3. {Specific strength (optional)}
 
-{Scope + feasibility: 1-2 sentences — "I have suggestions focused on [areas]. Achievable within existing data."}
+{Scope + feasibility: 1-2 sentences — "I have suggestions focused on [areas]." Describe feasibility from the fixability grades in the editor block (for example, "most can be addressed with the existing data; point 2 would need [additional work]"), not as a stock phrase.}
 
 (80-150 words total)
 

package/skills/peer-review/references/reviewer_calibration/recommendation_calibration.md

@@ -64,6 +64,16 @@ missing supplementary, a mislabeled table, an over-claiming sentence) from defec
 within the current submission (poolability of incommensurable studies, a broken construct, an invalid
 evaluation instrument). When both classes are present, the **unfixable** class governs the recommendation —
 do not let a long list of fixable items reframe an unfixable core as "addressable in revision."
+Grade fixability in three levels, and state the scope of any work you ask for. (1) **Fixable with the data
+the authors already hold**: re-analysis, disclosure, rewording, an added table. (2) **Fixable with additional
+work on the existing material**: added readers re-reading the same cases, reference-standard adjudication
+from existing records; whether this fits a revision depends on its scope, access and the journal, so say
+which and how much. (3) **Not fixable in this submission**: a repair that requires a new study or data collection that is
+infeasible within this revision, such as a different design or a reference standard that cannot be
+obtained for this cohort. Adding an accessible cohort or completing recruitment is judged by scope,
+access and timing like level (2). Seeing how a defect could be fixed is not the same as the
+authors being able to fix it in a revision; when feasibility is unknown, say "unclear" rather than default
+to (1) or (2).
 
 **Salvage-reframe that shrinks the contribution is NOT a fixable major revision.** When your proposed fix
 for a construct/validity flaw is to *narrow the claim* (a clinical claim reframed as a weaker technical

package/skills/self-review/SKILL.md

@@ -158,7 +158,7 @@ whose row matches:
 
 | Manuscript type / signal | Probe module |
 |---|---|
-| Systematic Review / Meta-Analysis | `references/domain-probes/sr_ma.md` (P0–P19) |
+| Systematic Review / Meta-Analysis | `references/domain-probes/sr_ma.md` (P0–P20) |
 | Time-to-event / survival / prognostic model (Cox, Fine-Gray, DeepSurv, nomogram, risk-stratification cutoff) | `references/domain-probes/survival_prognostic.md` (S1–S9) |
 | Radiomic feature reproducibility / acquisition-parameter sweep / reliability-based feature filtering | `references/domain-probes/radiomics.md` (R1–R4) |
 | Cross-modality image synthesis (MRI→PET / MRI→CT / non-contrast→contrast / low-dose→full-dose) claiming functional/molecular information or target-modality substitution | `references/domain-probes/image_synthesis.md` (IS1–IS4) |
@@ -168,7 +168,7 @@ whose row matches:
 | Engineer-built medical-imaging model (segmentation / classification / detection) being validated — partition/leakage, seed & run variance, metric selection, reproducibility, reference-standard quality; plus saliency faithfulness, uncertainty/OOD/abstention and deployment feasibility when a clinical-use claim is made | `references/domain-probes/model_development.md` (MD0–MD11) |
 | LLM / MLLM evaluated on a clinical task (report generation, VQA, clinical text extraction/classification; closed API or open weights) | `references/domain-probes/mllm_evaluation.md` (ME0–ME8) |
 | Randomised controlled trial (parallel / crossover / cluster / stepped-wedge) | `references/domain-probes/rct_trial.md` (RC0–RC7) |
-| Diagnostic test accuracy (DTA) primary study / multi-reader multi-case (MRMC) reader study (AI-vs-reader, AI-assisted reading, modality comparison) | `references/domain-probes/diagnostic_accuracy.md` (D1–D12) |
+| Diagnostic test accuracy (DTA) primary study / multi-reader multi-case (MRMC) reader study (AI-vs-reader, AI-assisted reading, modality comparison) | `references/domain-probes/diagnostic_accuracy.md` (D1–D15) |
 | Case report / case series (incl. adverse-event/pharmacovigilance and imaging-led radiology/nuclear-medicine/IR reports) | `references/domain-probes/case_report.md` (CR1–CR9) |
 | AI/ML, prediction, or diagnostic study claiming cross-population performance, or presenting subgroup analyses as a fairness/equity argument | `references/domain-probes/equity_fairness.md` (EQ0–EQ6) |
 | Mendelian randomization (two-sample, one-sample, multivariable, drug-target / cis-MR, non-linear MR) | `references/domain-probes/mendelian_randomization.md` (MR1–MR8) |

package/skills/self-review/references/domain-probes/ai_overclaiming.md

@@ -22,9 +22,9 @@ A 7-probe checklist (AO1–AO7, with AO0 as a gate) for medical-AI/ML primary st
 - If the generalizability claim outruns the external evidence → recommend softening to the evidence ("validated at one external site") and moving multi-setting generalizability to a stated limitation + next step. MAJOR candidate when it is a headline claim; MINOR when it is a single qualifier in the Discussion.
 
 **AO2 — Superiority language against overlapping or under-powered comparison**:
-- Flag "outperforms", "superior to", "beats", "can replace [clinician/radiologist]" when (a) the model vs comparator 95% CIs overlap, (b) no test of the *difference* is reported (two separate AUCs are not a comparison), or (c) the comparison rests on a small test set / few readers.
+- Flag "outperforms", "superior to", "beats", "can replace [clinician/radiologist]" when (a) no estimate of the *difference* with its CI, or test of it, is reported (two separate AUCs are not a comparison, and overlapping or non-overlapping marginal CIs do not settle it, because paired estimates can differ clearly despite overlap), (b) for a superiority claim, the CI of the difference includes no difference (then statistical superiority is unsupported), or the claim is of clinically meaningful superiority while the CI shows at most a trivial gain (then "statistically higher, with limited clinical importance" is the supportable wording), or (c) the claim goes beyond the comparison made (a replacement claim from a small test set or a few readers, or from a task that differs from clinical reading). A replacement claim is judged by its own justification (superiority, non-inferiority within a pre-specified margin, or equivalence) and its clinical trade-offs; a non-inferiority result can support it with a CI that includes zero.
 - Ask for the difference in the metric with its CI and a paired test of that difference, not two standalone estimates.
-- If the difference is not statistically supported → recommend reframing from "outperforms" to "comparable to" (still a meaningful result). MAJOR when a superiority/replacement claim is the headline; otherwise MINOR.
+- If the difference is not statistically supported → recommend dropping the superiority language and reporting the difference with its CI. Do **not** suggest "comparable to" or "equivalent" in its place: a non-significant difference is not evidence of similarity unless the CI lies within a pre-specified, clinically justified margin (`diagnostic_accuracy.md` D6). The supportable wording is that no difference was detected and that a difference of the size the CI still admits cannot be excluded. MAJOR when a superiority/replacement claim is the headline; otherwise MINOR.
 
 **AO3 — Comparison-frame mismatch (model task ≠ human task)**:
 - When a model-vs-clinician comparison drives a claim, verify the two performed the **same task on the same inputs under the same constraints**: same images/inputs available, same time budget, same question asked, same decision point.
@@ -78,7 +78,7 @@ Co-apply when a study claims **clinical utility, deployment, or decision impact*
 > "The Conclusion states the model 'generalizes across institutions,' but external validation appears limited to a single site ([Methods, External validation]). I'd suggest softening this to the evidence — e.g., 'validated at one external site' — and framing multi-institution generalizability as a stated limitation and a next step. If a broader claim is intended, an external set spanning multiple sites/vendors would be needed to support it."
 
 **Output template (AO2 / AO3 example)**:
-> "The 'outperforms radiologists' claim rests on a comparison whose 95% CIs for model and reader [metric] overlap ([Figure/Table]), and no test of the difference is reported; the reader task also differs from the model's in [inputs/time] ([Methods/Table]). I'd suggest (a) reporting the difference in [metric] with its CI and a paired test rather than two separate estimates, and (b) stating explicitly which clinical task the comparison establishes. If the difference is not statistically supported, reframing from 'outperforms' to 'comparable to' would be both defensible and still a meaningful result."
+> "The 'outperforms radiologists' claim rests on a comparison whose 95% CIs for model and reader [metric] overlap ([Figure/Table]), and no test of the difference is reported; the reader task also differs from the model's in [inputs/time] ([Methods/Table]). I'd suggest (a) reporting the difference in [metric] with its CI and a paired test rather than two separate estimates, and (b) stating explicitly which clinical task the comparison establishes. If the difference is not statistically supported, I'd suggest removing 'outperforms' and reporting the difference with its CI; 'comparable to' would need a pre-specified margin that the CI stays within."
 
 **Output template (AO5 example)**:
 > "Table 2 appears to report the single best cross-validation fold; because best-fold figures are optimistically biased, I'd suggest reporting the mean across folds with its SD (or a 95% CI). Relatedly, the sensitivity/specificity depend on a classification threshold I could not find stated — reporting the operating point and how it was chosen (e.g., Youden's J on the training folds only, to avoid tuning on the test data) would make these numbers reproducible. Finally, since the training set was balanced to 1:1 while the clinical prevalence is much lower, accuracy (and PPV/NPV) on a re-balanced set may not reflect performance at the true base rate; stating the evaluation class distribution and adding AUROC/AUPRC with CIs alongside sensitivity/FNR at the stated threshold, with PPV/NPV on a prevalence-representative holdout, would make the clinical cost of a miss visible."

package/skills/self-review/references/domain-probes/clinical_prediction_model.md

@@ -6,25 +6,26 @@
        - self-review: Anticipated Major / Minor Comments (Fatal / Fixable) mapped to category letters.
      Do NOT edit one copy only — run `python3 scripts/check_domain_probe_sync.py --sync`. -->
 
-# Clinical Prediction-Model probes (CP1–CP6)
+# Clinical Prediction-Model probes (CP1–CP8)
 
-A 6-probe checklist for **cross-sectional / observational clinical prediction models** — a binary or multinomial outcome predicted from a covariate set, reported under TRIPOD / TRIPOD+AI, usually as a **nested predictor-set comparison** ("does adding marker X to a base model improve prediction of Y?"). This is the diagnostic/prognostic *prediction* counterpart to `survival_prognostic.md` (time-to-event) and complements the `observational_confounding.md` probes (a prediction model still has an analysis unit (O8) and can be over-adjusted (O7)). Route here when the manuscript develops or compares prediction models and reports discrimination (AUC/C-statistic), calibration, or decision-curve / net-benefit analysis.
+An 8-probe checklist for **cross-sectional / observational clinical prediction models** — a binary or multinomial outcome predicted from a covariate set, reported under TRIPOD / TRIPOD+AI, usually as a **nested predictor-set comparison** ("does adding marker X to a base model improve prediction of Y?"). This is the diagnostic/prognostic *prediction* counterpart to `survival_prognostic.md` (time-to-event) and complements the `observational_confounding.md` probes (a prediction model still has an analysis unit (O8) and can be over-adjusted (O7)). Route here when the manuscript develops or compares prediction models and reports discrimination (AUC/C-statistic), calibration, or decision-curve / net-benefit analysis.
 
 **CP1 — Apparent vs optimism-corrected performance**:
 - Discrimination, **calibration** (slope / intercept / calibration plot), and **decision-curve / net-benefit** all suffer optimism when computed in the same sample used to fit the model. A manuscript that optimism-corrects *only* the C-statistic (bootstrap / cross-validation) but reports calibration and net benefit **in-sample** has corrected one axis and left two apparent.
 - An apparent calibration slope near unity (≈ 1.0) does **not** establish out-of-sample calibration — it is the expected in-sample result and says little about transportability.
 - Calibration and DCA computed in-sample must be **labelled "apparent"**, or the optimism correction must be extended to them (e.g. bootstrap the calibration slope/intercept and the net-benefit curves). In-sample calibration/DCA described as "well calibrated" / "clinically useful" without the apparent caveat → MAJOR.
+- A non-significant Hosmer–Lemeshow test does not establish calibration: it depends on sample size and grouping and says nothing about the direction or size of miscalibration. Ask for a calibration curve with its uncertainty, plus calibration intercept and slope, each labelled apparent or validated. In a small cohort a smoothed curve with wide uncertainty and an honest caveat is enough; do not demand every measure. Cross-link `survival_prognostic.md` S7 / T2.
 
 **CP2 — The two-null distinction (incremental value vs marginal effect)**:
 - A "negative" prediction study conflates two different nulls that have different evidentiary status:
-  - **Incremental value negligible** — a *well-powered* statement: ΔAUC ≈ 0, optimism correction does not favour the larger model (and may reverse sign toward the parsimonious one), net benefit does not exceed the base model. This is a genuine "adding X does not improve prediction."
+  - **Incremental value negligible** — a *well-powered* statement: ΔAUC ≈ 0, optimism correction does not favour the larger model (and may reverse sign toward the parsimonious one), net benefit does not exceed the base model. It supports "adding X does not improve prediction" only when the uncertainty (the CI of ΔAUC or of the net-benefit difference) excludes a clinically meaningful gain; otherwise the supportable wording is "no improvement was detected".
   - **Marginal predictor effect null** — an *underpowered* statement: the predictor's adjusted OR/coefficient has a confidence interval that still admits a clinically relevant effect (e.g. OR up to ~1.7). This is "we could not exclude an effect," not "there is no effect."
 - A manuscript that collapses both into a blanket "X did not predict Y" overstates the marginal-effect arm. Flag when a negative conclusion rests on a non-significant marginal OR whose CI admits a relevant effect, without separating it from the (better-powered) incremental-value finding → MAJOR. The honest reframe reports both explicitly.
 
 **CP3 — Events-per-variable (EPV) per nested model**:
-- Report EPV for **each** nested model, not just the full cohort. A model with 6 covariates and 60 events (EPV 10) is adequately supported; the same 6 covariates in an age≥60 subgroup with 13 events (EPV ≈ 2) is separation-prone and unstable.
-- Penalization (Firth / ridge / LASSO) is justified and should be stated when EPV < 10; subgroup or interaction models with EPV in the low single digits are **separation-affected and must be descriptive-only**, not presented as estimated effects.
-- Subgroup/interaction effect estimates from a model with EPV ≲ 5, presented as if reliably estimated, or no EPV disclosure for the nested models → MAJOR (subgroup) / MINOR (EPV simply not reported but adequate).
+- Report events per candidate parameter for **each** nested model, not just the full cohort, counting every candidate parameter considered (dummy levels, non-linear terms, variables screened out), not only those retained. EPV is a rough screen, not a sufficiency rule: whether the sample supports a model depends on the number of candidate parameters, the outcome proportion and the expected model performance (sample-size criteria for model development: Riley et al., *BMJ* 2020;368:m441). The same 6 covariates fitted in an age≥60 subgroup with 13 events (EPV ≈ 2) remain separation-prone and unstable.
+- Penalization (Firth / ridge / LASSO) reduces overfitting and should be stated, but it does not rescue a sample that is too small for the candidate set, and the amount of shrinkage is itself unstable in small samples. For subgroup or interaction models with EPV in the low single digits, assess separation, the estimation method, the uncertainty and the model complexity; unstable estimates should be reported as exploratory with their intervals, and no reliable effect should be inferred from them.
+- Low EPV triggers an assessment of the sample-size justification, separation, uncertainty and validation. MAJOR when demonstrated instability undermines the claim (for example, subgroup or interaction effects presented as reliably estimated); request clarification when adequacy cannot be assessed from what is reported.
 
 **CP4 — Net benefit is a model comparison, not a policy endorsement**:
 - A decision curve showing the model exceeds treat-all / treat-none across a threshold range is a **model-vs-default** statement about ranking, not evidence that the underlying **imaging / test / screening strategy** should be adopted in practice. Net benefit is computed on the same outcomes and says nothing about cost, downstream harm, or whether the test should be ordered.
@@ -37,8 +38,20 @@ A 6-probe checklist for **cross-sectional / observational clinical prediction mo
 
 **CP6 — Validation-nomenclature conflation (development step vs test step)**:
 - Distinguish the **development** step (including any internal cross-validation or bootstrap optimism correction) from a **held-out / external test** step. Flag phrasing that merges them: "developed with external validation" where a single external set is used *as* development data; "trained and validated on the external cohort"; an "external validation set" that was used for tuning/feature selection.
-- Cross-validation and bootstrapping are development-time optimism corrections, **not** external validation; calling them "validation" overclaims out-of-sample evidence. Require an explicit statement of which data trained, which tuned, and which was touched only once for the final estimate.
+- Bootstrap optimism correction and ordinary within-population cross-validation are legitimate **internal validation** when every development step (variable selection, tuning, threshold choice) is repeated inside each resample; they estimate optimism in the development population and are **not** evidence of transportability. Internal–external cross-validation that holds out whole centres or studies does speak to performance across settings, limited to the settings represented. Proper internal validation does support qualified conclusions about new patients from the development population. The overclaim is presenting it as external validation or as evidence of transportability to untested populations or settings ("externally validated", "generalisable across centres", "ready for clinical use" with no external data); ask for "internally validated" and a clear description of the resampling workflow. Resampling only to count how often each variable is selected measures selection **stability**, not performance, and is not validation. Require an explicit statement of which data trained and which tuned, and, when a held-out or external set is claimed, that it did not influence model fitting, selection, tuning or thresholds (pre-specified evaluation analyses and uncertainty estimates on the frozen model are fine).
 - → MAJOR when the overclaim props up a generalisability/deployment-readiness conclusion; a clarify-request when the data flow is actually clean but the wording is loose.
 
+**CP7 — Prediction claim, association evidence (entry check)**:
+- Decide first whether the paper asks an **association** question (is X independently related to Y after adjustment?) or a **prediction** question (how accurately can risk be estimated for a new patient?). A significant adjusted OR or HR does not show useful prediction, and a model built for prediction does not license independent-effect or causal language for its coefficients.
+- Flag a prediction claim ("predicts", "can be used to predict", "a predictive marker") supported only by association statistics with no discrimination, calibration or validation, and the reverse: causal interpretations or inferential claims about predictor effects that the design does not support or that ignore selection and shrinkage (describing how a predictor moves the fitted prediction is fine). A paper that states both objectives separately and analyses each appropriately is fine.
+- Severity: MAJOR when the headline prediction claim rests on association statistics alone. Cross-link CP2 (two nulls), CP4, and the task-formulation audit.
+
+**CP8 — How the predictors were chosen and handled, and whether the reported model is the fitted one**:
+- The common shortcuts fail in different ways. **Univariable P-value screening** discards predictors that matter only jointly and keeps chance associations. **Stepwise or other automated selection** gives unstable models and biased coefficients; repeating the whole selection procedure inside internal validation is needed for an honest performance estimate, though it does not remove the instability or the coefficient bias. **Dichotomising a continuous predictor** at a data-derived "optimal" cut-off loses information and inflates apparent performance. **Complete-case analysis**, or excluding patients whose outcome or predictor could not be obtained, reduces the sample and may introduce selection bias or restrict applicability, depending on the missingness mechanism, who was excluded and the intended deployment population. Ask what was done and whether internal validation repeated it.
+- Severity follows the consequence, not the label; disclosure or penalization does not by itself make the finding MINOR. MAJOR when selection or categorisation is data-driven, is not repeated within the internal-validation resamples, the frozen pipeline has no independent evaluation, and the performance claim depends on it; MINOR when it is disclosed and its effect is shown to be small. Evaluating the frozen model on untouched external data is valid without repeating the selection there.
+- Check that the model in the paper is the model that was fitted: the variables and coefficients of the final model (or its equation or score) match the multivariable results table, and any re-fitting step between the two is described. Keep an explanatory adjusted-OR table distinct from the predictive equation. A table–model mismatch that changes which predictors the model uses → MAJOR.
+- Reporting completeness is not low risk of bias: a fully ticked TRIPOD or TRIPOD+AI checklist can describe a high-risk design. Judge risk of bias separately (PROBAST or its AI extension), and check that the checklist's page references point to text that exists.
+- Cross-link `observational_confounding.md` O2 (adjustment-set selection in causal models, a different reason), O5 (missing data), O12 (data-driven thresholds), `model_development.md` MD1 / MD2 / MD7, `survival_prognostic.md` T4, CP3.
+
 **Output template (CP1 / CP2 example)**:
-> "Discrimination is bootstrap optimism-corrected, but the calibration slope (0.99) and the net-benefit curves are computed in the development sample and described as 'well calibrated' and 'clinically useful' without the apparent caveat — an in-sample slope near 1.0 is expected and does not establish out-of-sample calibration. I'd suggest labelling the calibration and decision-curve results 'apparent' or extending the optimism correction to them. Relatedly, the Conclusion states that the marker 'did not predict' the outcome, but two distinct results are merged: the incremental-value analysis is well powered (ΔAUC ≈ 0, optimism correction favours the parsimonious model), whereas the marginal adjusted OR has a confidence interval that still admits an effect up to ~1.7 (underpowered). I'd report these separately — 'adding the marker did not improve prediction' is supported; 'the marker has no association' is not."
+> "Discrimination is bootstrap optimism-corrected, but the calibration slope (0.99) and the net-benefit curves are computed in the development sample and described as 'well calibrated' and 'clinically useful' without the apparent caveat — an in-sample slope near 1.0 is expected and does not establish out-of-sample calibration. I'd suggest labelling the calibration and decision-curve results 'apparent' or extending the optimism correction to them. Relatedly, the Conclusion states that the marker 'did not predict' the outcome, but two distinct results are merged: the incremental-value estimate is near zero (ΔAUC ≈ 0, optimism correction favours the parsimonious model), but its CI must be examined before a meaningful gain can be excluded, whereas the marginal adjusted OR has a confidence interval that still admits an effect up to ~1.7 (inconclusive). I'd report these separately — 'adding the marker did not meaningfully improve discrimination as measured by AUC' is supported if the CI of ΔAUC excludes a meaningful gain (calibration and net benefit need their own assessment before a broader claim); 'the marker has no association' is not."

package/skills/self-review/references/domain-probes/diagnostic_accuracy.md

@@ -6,7 +6,7 @@
        - self-review: Anticipated Major / Minor Comments (Fatal / Fixable) mapped to category letters.
      Do NOT edit one copy only — run `python3 scripts/check_domain_probe_sync.py --sync`. -->
 
-# Diagnostic-accuracy / reader-study probes (D1–D12)
+# Diagnostic-accuracy / reader-study probes (D1–D15)
 
 A checklist for **diagnostic test accuracy (DTA) primary studies** — an index test against a reference standard, including **multi-reader multi-case (MRMC)** reader studies (e.g., AI-vs-reader or modality-comparison). These probes complement (do not replace) the generic Phase 2 issue checklist and the STARD / QUADAS-2 items; they target the biases QUADAS-2 names and the MRMC design/variance issues a reader study adds. Pairs the `analyze-stats` `table-standards/table-types/reader_study.md` table and the `make-figures` `exemplar_plots/mrmc_roc.md` figure. (For a DTA **meta-analysis**, use `sr_ma.md`.)
 
@@ -21,7 +21,8 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 
 **D3 — Blinding of index and reference interpretation**:
 - Was the **index test** interpreted blind to the reference standard and to clinical information that would not be available at the point of use, and vice versa? Unblinded interpretation (review bias) inflates agreement.
-- For an AI index test, was the **operating threshold pre-specified** (not tuned on the test set)? A test-set-tuned threshold reported as the result is an optimistic-validation finding (pair with `exemplar_reviews/optimistic_validation_reporting.md`).
+- For any index test read against a threshold (an AI score, or a quantitative imaging measure such as ADC, SUV or a histogram percentile), was the **operating threshold pre-specified**, or chosen on the same data that report its accuracy (an "optimal" cut-off by Youden index or similar)? A data-derived cut-off flatters the data it came from. A test-set-tuned threshold reported as the result is an optimistic-validation finding (pair with `exemplar_reviews/optimistic_validation_reporting.md`).
+- When the study develops the threshold itself, internal resampling must repeat the threshold selection, and an external or temporal confirmation is needed only for a transportability claim. A threshold-development study that labels its cut-off exploratory is not a defect. Cross-link `model_development.md` MD2, `ai_overclaiming.md` AO5(b), `survival_prognostic.md` S4.
 
 **D4 — Indeterminate / uninterpretable results**:
 - How were **indeterminate, uninterpretable, or intermediate** index results handled — excluded, or counted (intention-to-diagnose)? Silently dropping them inflates accuracy. State the rule and report an **intention-to-diagnose** sensitivity analysis (non-diagnostic counted as wrong). Undisclosed exclusion of indeterminates → MAJOR.
@@ -33,6 +34,7 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 **D6 — MRMC analysis + estimand**:
 - Does the analysis account for **both reader and case variability** (Obuchowski–Rockette / Dorfman–Berbaum–Metz or an equivalent multi-reader model), or does it treat readers as fixed / pool reads as if independent? Ignoring reader variance understates uncertainty and is a MAJOR statistical flaw for a generalising claim.
 - Is the **estimand** clear: reader-averaged vs a fixed specific reader; per-**patient** vs per-**lesion** (clustered) unit; superiority vs **non-inferiority** with a pre-specified margin? Are **per-reader** results shown alongside the reader-averaged estimate (a single averaged AUC can hide one weak reader)? A non-inferiority claim with no pre-specified margin, or a clustered design analysed as independent, → MAJOR.
+- **Not significant is not equivalent.** A non-significant difference between two tests (or a test and its reduced-dose, faster or cheaper version) does not show they perform alike; in a small sample the CI of the difference usually still admits a clinically important loss. "Comparable", "non-inferior", or a replacement claim that rests on similarity needs a clinically justified margin and a CI that excludes the unacceptable difference: one bound for non-inferiority, both bounds for equivalence. A replacement claim that rests on demonstrated superiority needs no margin, only evidence for the advantage and an account of the clinical trade-offs (D13). Without a margin, the supportable wording is that no difference was detected and a loss of the size the CI admits cannot be excluded. A missing prospective sample-size statement does not by itself make a precise estimate inconclusive; the CI decides. Post hoc (observed) power computed from the observed effect is a transformation of the P value and adds nothing, so ask for the difference with its CI instead. Severity: a replacement or equivalence headline resting only on a non-significant P → MAJOR; loose "similar" wording for a secondary finding → MINOR. Cross-link `ai_overclaiming.md` AO2, `sr_ma.md` P9, `clinical_prediction_model.md` CP2.
 
 **D7 — Index-test-as-enrollment-criterion circularity**:
 - Cross-check the **inclusion criteria** against the **index / proposed test**. When a study proposes or validates a diagnostic or classification instrument *and* uses a threshold of that **same** instrument (or a component of it) as an enrollment criterion, the validation is circular: the spectrum is built into the design, and sensitivity/specificity (or the instrument's apparent discrimination) are inflated by construction.
@@ -43,6 +45,7 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 - Cross-check the **exclusion criteria drawn in the participant flow diagram** (STROBE/STARD flow) against the **exclusion list in the Methods prose**. A criterion that appears in one but not the other (a count in the flow with no prose rationale, or a prose exclusion not reflected in the flow boxes) is a reporting inconsistency a co-reviewer catches by reading the two side by side.
 - For an **imaging** study, check that **modality-specific safety contraindications** and **device/artifact exclusions** are enumerated where applicable: MR safety (pacemaker/implant, claustrophobia), iodinated/gadolinium **contrast** contraindications (renal function, allergy, pregnancy), and **image-quality/artifact** exclusions (motion, metal artifact, incomplete coverage). Silent omission of these categories in a prospective imaging cohort understates the selected spectrum.
 - Severity: a flow-vs-prose exclusion mismatch is MAJOR when it changes the analytic-N or the eligible spectrum; missing modality-safety/artifact exclusion categories is MINOR–MAJOR depending on how much of the source population they remove. The fix is a reconciled exclusion list (flow == prose) plus an explicit modality-safety/artifact exclusion enumeration.
+- Beyond exclusions, check that the Results report what the Methods promised: the pre-specified primary metric first, with its estimate and 95% CI (numerator and denominator for proportions; sample size and analysis unit for AUC and other metrics); results at the pre-specified threshold, not only at a new "optimal" one; planned versus achieved sample size; the number verified by each reference-standard method; the stated unit of analysis. An undisclosed outcome-dependent switch of the primary metric or threshold → MAJOR. For a documented amendment, ask when it was made relative to access to the results and why: a change made before the relevant results were seen is a clarify-request, while an outcome-informed change makes the affected analyses exploratory, however well it is documented.
 
 **D9 — Confidence-weighted / rating reader study: unweighted baseline AUC must accompany the weighted primary**:
 - When a reader study's novelty is a **confidence-weighted** (or rating-collapsed) score used as the ROC/AUC predictor, the **unweighted binary-call AUC** must be reported side-by-side (as a sensitivity analysis). Without it, you cannot tell whether the weighting *created* the result or hid an estimator fragility (e.g. a folded/non-monotonic encoding that collapses `real/5` with `ai/1`).
@@ -65,9 +68,25 @@ A checklist for **diagnostic test accuracy (DTA) primary studies** — an index
 - Confirm the **assistance paradigm** is stated and matches the claim — AI shown **concurrently**, as a **second reader** after an independent read, or as **triage / pre-selection**. These measure different quantities and are not interchangeable; a concurrent-display study cannot support a second-reader deployment claim. Where the same readers read both arms, the washout / ordering controls of D5 apply to the arm comparison too.
 - Severity: an adoption/workflow claim with no assisted arm, or a pooled assisted delta with no AI-correct / AI-incorrect decomposition, → MAJOR (the pooled number cannot exclude automation-bias harm); an unstated assistance paradigm or a missing experience-stratified breakdown → MINOR pending clarification. Boundary: D5/D6 own the MRMC design and variance mechanics; D12 owns what the arms *mean* and the assisted-arm harm decomposition.
 
+**D13 — Intended use and role of the test → the metric and design that can carry the claim**:
+- Read the intended use off the Introduction and the Conclusion: where the test sits in the clinical pathway (**replacement** of an existing test, **triage** before it, **add-on** after it, or surveillance / staging) and which decision it changes. The role decides which error matters more, which population must be sampled, and which comparison answers the question.
+- Check that the headline metric fits the role and that both error directions are reported. A rule-out claim rests on sensitivity and NPV, a rule-in claim on specificity and PPV, with predictive values at a prevalence that matches the intended setting. A replacement or add-on claim needs a comparison with the existing pathway, in the same patients where feasible, not two standalone accuracies. Replacement does not always require a formal non-inferiority design, but the comparison must exist (D6 covers "not significant ≠ equivalent").
+- Check the conclusion against the role: per-lesion detection in a surgical cohort used to claim replacement of surveillance imaging, or a high-sensitivity result used to claim rule-in, overreaches.
+- Severity: MAJOR only when the headline role claim (replace, triage, add-on, rule out, rule in) is not supported by the design and the metric; an unstated role with an otherwise coherent design is a clarify-request. Cross-link D2 (spectrum), D6 (estimand), D12 (which arm carries the claim), `clinical_prediction_model.md` CP5 (decision horizon), `ai_overclaiming.md` AO4.
+
+**D14 — A significant group difference is not diagnostic performance, and a separable difference is not necessarily a meaningful one**:
+- A P value for the difference in a quantitative marker between diseased and non-diseased groups (mean ADC, SUV, a texture feature) tests the specified between-group hypothesis, not the classification of an individual patient; with enough patients a heavily overlapping distribution is "highly significant". When a diagnostic claim rests on a group comparison, ask for the measures the claim needs (the distributions, discrimination as AUC with CI, accuracy at a stated threshold with the 2×2 table) or for the claim to be narrowed to an association. An exploratory biomarker study can resolve this by narrowing its claim rather than by adding analyses.
+- For a quantitative measure, put differences in the context of measurement error in the same units: within-subject SD, repeatability coefficient or limits of agreement from a repeatability or reproducibility setting that matches use (same scanner, different scanner, different reader). A representative accuracy study of a single-measurement protocol already carries that protocol's measurement variability, so it does not need a separate repeatability study. Ask for repeatability evidence when the claim depends on it: consistency of classification on repeat measurement, robustness across scanners, sites or readers, or a change within one patient, where the change must exceed the repeatability of that change for the measurement scheme used. ICC, in either its consistency or its absolute-agreement form, is a ratio driven by between-subject spread and does not express error in measurement units; a high ICC from re-reading the same images says nothing about scan–rescan or cross-scanner variation.
+- Severity: MAJOR when a diagnostic or individual-level claim rests only on a significant group difference, or when the paper claims that change in an individual is reliably distinguishable from measurement error without evidence for it (a decision threshold below the repeatability coefficient can still be valid if its error rates are stated and validated); MINOR when the claim depends on measurement error that is not reported. Cross-link `ai_overclaiming.md` AO7 (benchmark variability never quantified), `observational_confounding.md` O18 (pseudoreplication in agreement studies).
+
+**D15 — One metric chosen from many (histogram percentiles × sequences × readers)**:
+- When the index test is one of many candidate measures (histogram percentiles, several sequences or parameters, several readers or ROIs), ask which metric was primary and whether it was fixed in advance, whether every tested candidate is reported (a supplementary table suffices), how multiplicity was handled for the claim being made, what rationale supports the chosen metric, and whether it was confirmed in independent data. Under the global null, 20 independent tests each at α = 0.05 have a 1 − 0.95^20 ≈ 64% probability of at least one false positive.
+- When candidate measures are combined through a fitted model, co-apply `clinical_prediction_model.md` and distinguish development from validation. A fixed, pre-specified ratio or composite index stays in this diagnostic-accuracy framework; assess any data-driven construction or threshold choice under D3.
+- Severity: MAJOR when a post hoc selected metric is the headline and the selection is not disclosed; a transparently exploratory screen with all candidates reported is not a defect. Cross-link `model_development.md` MD2, `ai_overclaiming.md` AO5, `observational_confounding.md` O17.
+
 **Output template (D2 / D6 example)**:
 > "The study uses a case-control (two-gate) design — confirmed cases versus healthy controls — rather than a consecutive series of patients in whom the diagnosis was suspected. This typically overestimates accuracy and does not reflect the intended-use spectrum, so I'd read the reported sensitivity/specificity as proof-of-concept rather than clinical accuracy, and suggest tempering the Abstract accordingly. Separately, the reader study reports a single reader-averaged AUC; because readers are a sample, I'd suggest an MRMC analysis (e.g., Obuchowski–Rockette) that accounts for both reader and case variance, with per-reader estimates shown and the unit of analysis (per-patient vs per-lesion) stated."
 
-**Discipline — leads vs findings (applies to D1–D12)**:
+**Discipline — leads vs findings (applies to D1–D15)**:
 - A verification/blinding/spectrum concern from a quick scan is a **lead until Methods and the participant flow are read together** — distinguish under-reporting (ask to clarify) from a true design bias (MAJOR).
 - Anchor each comment to the exact bias (partial vs differential verification; single- vs two-gate; reader-averaged vs fixed-reader; per-patient vs per-lesion) and the location. Keep severity tied to what the flaw does: two-gate sampling, incorporation bias, or ignoring reader variance is design/analysis-level (MAJOR, often Major #1); an unreported reading-order detail is a clarify-request.