medsci-skills 4.1.0

package/skills/self-review/references/panel_review_template.md

@@ -0,0 +1,177 @@
+# Panel Review Template (Phase 2.6)
+
+Reusable scaffolding for the multi-agent panel: a reviewer output schema, a
+generic reviewer prompt skeleton with per-domain focus checklists, and an
+editor synthesis prompt skeleton. This is a template, not a runtime program —
+it carries no manuscript-specific content. Fill the `{...}` placeholders from
+the manuscript under review and from the reviewer-set mapping in Phase 2.6.
+
+The per-domain focus checklists below name *categories of concern*; the precise
+probes (and their output templates) live in the vendored domain-probe modules
+(`references/domain-probes/*.md`), which each reviewer should load.
+
+---
+
+## Reviewer output schema
+
+Each reviewer returns one object with these fields:
+
+```json
+{
+  "reviewer_id": "R1",
+  "expertise_area": "Biostatistics & Study Design",
+  "overall_assessment": "3-5 sentences. State explicitly whether a fatal flaw exists and name the single biggest threat to the conclusions.",
+  "strengths": ["...", "..."],
+  "major": [
+    {
+      "number": 1,
+      "heading": "Short title of the issue",
+      "comment": "Detailed critique. Quote the manuscript where relevant. Explain why it threatens validity.",
+      "location": "section / table / figure / specific claim",
+      "severity": "Fatal | Fixable",
+      "suggested_fix": "What the authors should do to address it"
+    }
+  ],
+  "minor": [
+    { "number": 1, "comment": "...", "location": "..." }
+  ]
+}
+```
+
+`severity` uses this skill's own scale: **Fatal** for a conclusion-threatening /
+design-level finding, **Fixable** for a reporting-level finding.
+
+---
+
+## Reviewer prompt skeleton
+
+> You are an expert peer reviewer for a competitive journal in {field}, performing a
+> blinded pre-submission review of the author's own manuscript. Read the full
+> manuscript (and any supplement) first.
+>
+> TONE: rigorous and skeptical, but fair and constructive. Hunt for the issues that
+> threaten the manuscript's conclusions. Keep strengths to 2–3 genuine items. Every
+> major comment must threaten an actual conclusion or a reporting requirement; quote
+> the manuscript when you criticize a specific claim, and cite the location.
+>
+> Stay strictly within YOUR assigned area of expertise: {expertise_area}. Do not
+> stray into the other reviewers' domains. Load and apply the probes in
+> {domain_probe_module} where it applies. Produce 4–8 major comments and 4–10 minor
+> comments, and return them in the reviewer output schema above. Set `reviewer_id`
+> to "{reviewer_id}" and `expertise_area` to "{expertise_area}".
+>
+> YOUR FOCUS:
+> {focus_checklist}
+
+### Per-domain focus checklists (generic)
+
+**Biostatistics & Study Design**
+- Model specification: is the primary model pre-specified, or chosen post hoc in a results-favorable direction? Over-adjustment / conditioning on mediators on the causal pathway?
+- Assumption checks appropriate to the model (e.g., proportional hazards for Cox), and adequacy of any corrections.
+- Missing data: extent, plausibility of the missingness mechanism, whether the headline estimate survives a principled imputation, and whether imputation is primary or relegated to sensitivity.
+- Competing risks / informative censoring where multiple event types exist.
+- Sparse events: events-per-variable, penalization adequacy, CI stability.
+- Multiplicity across multiple outcomes, subgroups, and exploratory analyses.
+- Sensitivity / bias analyses (e.g., E-value) computed and interpreted correctly.
+- Time-zero / immortal-time / left-truncation; power for any null finding.
+
+**Clinical (domain)**
+- Clinical actionability: do the stated recommendations follow from the actual (possibly attenuated) results, or do they overreach?
+- Residual confounding by the dominant clinical driver of the outcome; can the exposure be disentangled from it?
+- Screening vs symptomatic population framing; external validity to the target readership.
+- Plausibility of the pattern of findings (e.g., a selective association without an expected concomitant one) — biology vs confounding vs chance.
+- Whether the findings would change management for a real patient.
+- Missing clinical variables and their interpretive cost.
+
+**Imaging / Radiology**
+- Exposure / measurement validity: how the imaging variable was defined and measured; visual/binary vs quantitative; threshold dependence.
+- Interobserver reliability measured in THIS cohort vs cited from the literature; defensibility of any non-differential-misclassification "bias toward the null" claim.
+- Protocol heterogeneity over time (scanner generation, slice thickness, reconstruction, dose) and its effect on the measurement.
+- Unavailable / non-retrievable images and selection implications.
+- Subtype/severity collapsed into a binary; loss of dose-response.
+- Reliability of routine clinical reports used as a research-grade variable.
+
+**Methodology (SR/MA)**
+- Search comprehensiveness and reproducibility; screening reviewer count.
+- Extraction fidelity vs source; comparator existence and consistent definition.
+- Non-independence (overlapping cohorts / shared public benchmarks).
+- Risk-of-bias instrument and per-study application; supplementary completeness.
+- Registration (PROSPERO) format and amendment discipline.
+
+**ML / Statistics (radiomics, AI)**
+- Design-grid circularity: is an outcome predicted from the very axes used to construct the dataset?
+- Construct validity: reliability ≠ predictiveness; orthogonality of proxy and target.
+- Transportability: cross-domain failure framed as success; negative R² read as a weak metric.
+- Multiplicity across model × threshold / cohort grids; small-cohort bootstrap intervals.
+- Leakage (patient-level vs image-level splits); calibration beyond discrimination.
+
+**Clinical translation / reference standard**
+- Reference-standard validity and verification bias.
+- Whether reported performance reflects the intended-use population and decision point.
+- Incremental value over the comparator already in routine use.
+
+**Methodology / SANRA (narrative review)**
+- Novelty / value-add against recent reviews; scope and aims clarity.
+- Evidence-gathering transparency (suggestion-level; not PRISMA).
+- Taxonomy / synthesis coherence; balance, currency, citation accuracy.
+- Load-bearing figures/tables; proportionate gap-filling ("consider adding", never "must cite").
+
+**Technical accuracy (narrative review)**
+- Engineering and domain correctness of specific claims; itemize errors with location.
+- Verify-your-own-criticism: cross-check each asserted inaccuracy against a current authoritative source before raising it.
+
+---
+
+## Editor synthesis prompt skeleton
+
+> You are the handling editor. {N} expert reviewers (areas: {areas}) have returned
+> independent blinded reviews of this pre-submission manuscript. Here are their
+> structured reviews as JSON:
+>
+> {reviews_json}
+>
+> Read enough of the manuscript to adjudicate conflicts and weigh severity yourself.
+> Then:
+> 1. Reach an internal readiness decision and state the rationale honestly. (This sets
+>    the Phase 3c verdict / score; it is not a journal recommendation to print.)
+> 2. De-duplicate and consolidate the major comments by theme. For each consolidated
+>    point, flag CONSENSUS (raised by ≥2 reviewers) or single-reviewer, and attribute
+>    (R1/R2/R3).
+> 3. List the top priority pre-submission actions, ranked and concrete.
+> 4. Give an honest readiness verdict: ready for the target tier now, fix specific
+>    items first, or consider a different tier.
+>
+> Map every finding onto the self-review framing (Fatal / Fixable, category letters
+> A–K) and emit it through the Phase 3 report, Phase 3b R0 numbering, and Phase 3c
+> JSON, adding the optional `consensus` field where ≥2 reviewers agreed. Follow the
+> manuscript-style rules: no "§" symbols, minimal em-dashes, full prose, cite specific
+> locations.
+
+---
+
+## Lens-diversity gate (Step 3.5)
+
+Before finalizing, the editor runs `scripts/check_panel_diversity.py` on the collected
+reviewer JSON. The gate classifies each major finding into a concern family and checks
+that the panel spans the axes its research type is expected to probe — the deterministic
+backstop against a panel converging on one easy theme while a high-risk axis goes unprobed.
+
+**Expected high-risk axes per research type** (each should yield ≥1 major; mirrors the
+Phase 2.6 reviewer-set table). Optional axes — for example imaging when the exposure is
+non-imaging — are not required:
+
+| Research type | Expected axes (families) |
+|---|---|
+| Survival / prognostic | statistics, clinical |
+| Systematic review / meta-analysis | search_screening, clinical, statistics |
+| Radiomics | imaging, statistics, clinical |
+| Diagnostic-accuracy / AI model | design_leakage, statistics, clinical |
+| Observational (STROBE) | confounding, clinical, statistics |
+| Narrative / review article | clinical, reporting |
+
+Concern families the classifier recognizes: `search_screening`, `design_leakage`,
+`confounding`, `imaging`, `reporting`, `reproducibility`, `statistics`, `clinical`
+(everything else falls to `other` and does not count toward coverage). When the research
+type is unknown, the axis-coverage check is skipped (the monoculture and lens-collapse
+checks still run). The gate never penalizes genuine consensus — only full reviewer
+redundancy (`LENS_COLLAPSE`) or panel-level concentration (`FAMILY_MONOCULTURE`).

package/skills/self-review/scripts/check_artifact_coverage.py

@@ -0,0 +1,301 @@
+#!/usr/bin/env python3
+"""Methods <-> Results <-> disk artifact coverage gate (self-review Phase 2.5f).
+
+Two directional failures survive a single prose pass because the manuscript is
+internally consistent:
+
+  FORWARD  PROMISED_ABSENT   an analysis named in the Methods / Statistical
+                            Analysis subsection (a sensitivity analysis, multiple
+                            imputation, an interaction test, a subgroup, mediation,
+                            a competing-risk or landmark model, an E-value) never
+                            reaches Results. Methods promised it; the paper never
+                            delivered it.
+  REVERSE  DISK_UNREPORTED   an analysis output that exists on disk (an
+                            added-value DeLong CSV, a calibration table) is never
+                            mentioned in the manuscript. The work was done and run
+                            but its result — which may contradict the headline —
+                            is silently absent.
+
+The reverse direction is the false-positive-prone one, so it is calibrated: when
+an `_analysis_outputs.md` manifest exists (written by /analyze-stats) it is the
+source of truth; otherwise the analysis directory is globbed and a finding is only
+a Major when the file stem carries an analysis-bearing token (delong, nested,
+added-value, interaction, sensitivity, subgroup, mediation, imputation, landmark,
+calibration, dca, nri, idi). A cryptic stem with no such token is a Minor flag.
+
+INPUTS
+  --manuscript    manuscript markdown/text (required).
+  --analysis-dir  directory of analysis outputs. If omitted, the first existing of
+                  output/analysis/, analysis/, results/ is used. An
+                  `_analysis_outputs.md` manifest in that dir (or alongside the
+                  manuscript) takes precedence as the output source of truth.
+
+OUTPUT
+  A reconciliation table (stdout) and, with --out, a JSON artifact:
+    {manuscript, analysis_dir, manifest, claims[{verdict, severity, detail, where}], summary}
+  PROMISED_ABSENT is always Major; DISK_UNREPORTED is Major or Minor per the rule
+  above. Exit 1 (with --strict) when any Major-severity claim exists.
+
+Stdlib-only (json / re / argparse / pathlib). Exit codes: 0 clean (or report-only),
+1 Major claim(s) found (with --strict), 2 input/usage error.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import re
+import sys
+from pathlib import Path
+
+# Promised analyses: key -> (detect-in-Methods regex, appears-in-Results regex).
+# The Results regex is intentionally looser (the concept can be phrased differently
+# downstream) but still anchored to the analysis name.
+PROMISED = {
+    "multiple imputation": (r"multiple imputation|\bm\s*=\s*\d+\s*imput|imputed using",
+                            r"imput-|imputation|imputed"),
+    "sensitivity analysis": (r"sensitivity analys[ie]s", r"sensitivity analys[ie]s"),
+    "leave-one-out": (r"leave[-\s]one[-\s]out", r"leave[-\s]one[-\s]out"),
+    "interaction": (r"interaction (?:term|test|analys|effect)|tested? for interaction|"
+                    r"p[-\s]?(?:for[-\s])?interaction",
+                    r"interaction|p[-\s]?(?:for[-\s])?interaction|effect modif"),
+    "subgroup": (r"subgroup analys", r"subgroup"),
+    "mediation": (r"mediation (?:analys|model)", r"mediation|mediat"),
+    "competing risk": (r"competing[-\s]risk|fine[-\s]?gray|subdistribution",
+                       r"competing[-\s]risk|fine[-\s]?gray|subdistribution|cumulative incidence"),
+    "landmark": (r"landmark analys", r"landmark"),
+    "E-value": (r"e[-\s]?value", r"e[-\s]?value"),
+}
+
+ANALYSIS_TOKENS = (
+    "delong", "nested", "added", "addedvalue", "incremental", "interaction",
+    "sensitivity", "subgroup", "mediation", "imputation", "imputed", "landmark",
+    "calibration", "dca", "netbenefit", "nri", "idi", "competing", "finegray",
+    "tipping", "leaveone", "bootstrap",
+)
+OUTPUT_EXT = (".csv", ".tsv", ".r", ".py", ".rds")
+MANIFEST_NAME = "_analysis_outputs.md"
+
+
+def _norm(s: str) -> str:
+    return re.sub(r"[^a-z0-9]+", "", s.lower())
+
+
+def split_sections(text: str) -> list[tuple[str, str]]:
+    """Return (heading, body) for each markdown heading region."""
+    sections, heading, buf = [], "(preamble)", []
+    for line in text.splitlines():
+        m = re.match(r"^#{1,4}\s+(.*)", line)
+        if m:
+            sections.append((heading, "\n".join(buf)))
+            heading = re.sub(r"[*_`]", "", m.group(1)).strip()
+            buf = []
+        else:
+            buf.append(line)
+    sections.append((heading, "\n".join(buf)))
+    return sections
+
+
+def section_text(sections: list[tuple[str, str]], names: tuple[str, ...]) -> str:
+    out = []
+    for heading, body in sections:
+        h = heading.lower()
+        if any(n in h for n in names):
+            out.append(body)
+    return "\n".join(out)
+
+
+# --- FORWARD: promised-but-absent ------------------------------------------
+
+def check_forward(text: str) -> list[dict]:
+    sections = split_sections(text)
+    methods = section_text(sections, ("method", "statistical analys", "analysis plan"))
+    results = section_text(sections, ("result", "finding"))
+    if not methods.strip():
+        return []  # no Methods section to read promises from
+    # If there is no separate Results section, compare against the whole document
+    # minus the Methods text (conservative: avoids matching a promise to itself).
+    haystack = results if results.strip() else text
+    claims = []
+    for key, (mre, rre) in PROMISED.items():
+        if re.search(mre, methods, re.I):
+            if not re.search(rre, haystack, re.I):
+                claims.append({
+                    "verdict": "PROMISED_ABSENT",
+                    "severity": "Major",
+                    "detail": (f"Methods promises a '{key}' analysis, but it does not "
+                               f"appear in Results"),
+                    "where": "Methods → Results",
+                })
+    return claims
+
+
+# --- REVERSE: disk-present-but-unreported -----------------------------------
+
+def find_analysis_dir(manuscript: Path, override: str | None) -> Path | None:
+    if override:
+        p = Path(override)
+        return p if p.is_dir() else None
+    base = manuscript.resolve().parent
+    for cand in ("output/analysis", "analysis", "results", "output"):
+        for root in (base, base.parent):
+            p = root / cand
+            if p.is_dir():
+                return p
+    return None
+
+
+def parse_manifest(path: Path) -> list[str]:
+    """Return declared output basenames from an _analysis_outputs.md manifest.
+    Lines like '- `table1.csv` -- desc' or '* roc_curve.pdf'."""
+    items = []
+    for line in path.read_text(encoding="utf-8").splitlines():
+        for m in re.finditer(r"`([^`]+\.[A-Za-z0-9]{1,5})`", line):
+            items.append(m.group(1))
+        if not re.search(r"`", line):
+            m = re.search(r"[-*]\s+([\w./-]+\.[A-Za-z0-9]{1,5})", line)
+            if m:
+                items.append(m.group(1))
+    return items
+
+
+def stem_tokens(name: str) -> list[str]:
+    stem = Path(name).stem
+    toks = re.split(r"[_\-.\s]+", stem)
+    return [t for t in toks if len(t) >= 4 and re.search(r"[a-z]", t.lower())]
+
+
+def mentioned(name: str, body_norm: str) -> bool:
+    toks = stem_tokens(name)
+    if not toks:
+        return True  # nothing distinctive to look for -> do not flag
+    return any(_norm(t) in body_norm for t in toks)
+
+
+def check_reverse(text: str, manuscript: Path, analysis_dir: str | None) -> tuple[list[dict], dict]:
+    body_norm = _norm(text)
+    meta = {"analysis_dir": None, "manifest": None}
+    # 1) manifest precedence
+    manifest_files: list[str] = []
+    for cand in (manuscript.resolve().parent / MANIFEST_NAME,):
+        if cand.is_file():
+            manifest_files = parse_manifest(cand)
+            meta["manifest"] = str(cand)
+            break
+    adir = find_analysis_dir(manuscript, analysis_dir)
+    if adir is not None:
+        meta["analysis_dir"] = str(adir)
+        mpath = adir / MANIFEST_NAME
+        if not manifest_files and mpath.is_file():
+            manifest_files = parse_manifest(mpath)
+            meta["manifest"] = str(mpath)
+
+    claims = []
+    if manifest_files:
+        for name in sorted(set(manifest_files)):
+            if Path(name).suffix.lower() not in OUTPUT_EXT:
+                continue  # figures (.pdf/.png) are checked by /make-figures legends
+            if not mentioned(name, body_norm):
+                claims.append({
+                    "verdict": "DISK_UNREPORTED",
+                    "severity": "Major",
+                    "detail": (f"manifest output '{name}' is not mentioned anywhere in "
+                               f"the manuscript"),
+                    "where": meta["manifest"],
+                })
+        return claims, meta
+
+    # 2) glob fallback (calibrated severity)
+    if adir is None:
+        return claims, meta
+    for f in sorted(adir.rglob("*")):
+        if not f.is_file() or f.suffix.lower() not in OUTPUT_EXT:
+            continue
+        if mentioned(f.name, body_norm):
+            continue
+        analysis_bearing = any(_norm(t) in ANALYSIS_TOKENS for t in stem_tokens(f.name))
+        claims.append({
+            "verdict": "DISK_UNREPORTED",
+            "severity": "Major" if analysis_bearing else "Minor",
+            "detail": (f"analysis output '{f.name}' exists on disk but is not mentioned "
+                       f"in the manuscript" + (" (analysis-bearing name)" if analysis_bearing else "")),
+            "where": str(f.relative_to(adir.parent) if adir.parent in f.parents else f),
+        })
+    return claims, meta
+
+
+# --- driver ----------------------------------------------------------------
+
+def analyze(manuscript: str, analysis_dir: str | None) -> dict:
+    p = Path(manuscript)
+    if not p.is_file():
+        sys.stderr.write(f"ERROR: manuscript not found: {manuscript}\n")
+        sys.exit(2)
+    text = p.read_text(encoding="utf-8")
+
+    claims = check_forward(text)
+    rev, meta = check_reverse(text, p, analysis_dir)
+    claims += rev
+
+    n_major = sum(1 for c in claims if c["severity"] == "Major")
+    return {
+        "manuscript": str(p),
+        "analysis_dir": meta["analysis_dir"],
+        "manifest": meta["manifest"],
+        "claims": claims,
+        "summary": {
+            "n_claims": len(claims),
+            "n_major": n_major,
+            "n_flag": len(claims) - n_major,
+            "verdict": "MAJOR_CANDIDATE" if n_major else "OK",
+        },
+    }
+
+
+def render(result: dict) -> str:
+    lines = ["| Direction | Severity | Detail |", "|---|---|---|"]
+    for c in result["claims"]:
+        lines.append(f"| {c['verdict']} | {c['severity']} | {c['detail']} |")
+    if len(lines) == 2:
+        lines.append("| (none) | — | Methods↔Results↔disk all reconciled |")
+    return "\n".join(lines)
+
+
+def main() -> int:
+    ap = argparse.ArgumentParser(description="Methods<->Results<->disk coverage gate (Phase 2.5f).")
+    ap.add_argument("--manuscript", required=True, help="manuscript markdown/text")
+    ap.add_argument("--analysis-dir", help="analysis-output dir (default: output/analysis, analysis, results)")
+    ap.add_argument("--out", help="write JSON artifact to this path")
+    ap.add_argument("--strict", action="store_true", help="exit 1 if any Major claim exists")
+    ap.add_argument("--quiet", action="store_true", help="suppress stdout table")
+    args = ap.parse_args()
+
+    result = analyze(args.manuscript, args.analysis_dir)
+
+    if not args.quiet:
+        print("=" * 41)
+        print(" Artifact Coverage (Phase 2.5f)")
+        print("=" * 41)
+        if result["manifest"]:
+            print(f"manifest: {result['manifest']}")
+        elif result["analysis_dir"]:
+            print(f"analysis dir (globbed): {result['analysis_dir']}")
+        print(render(result))
+        print()
+        s = result["summary"]
+        if s["n_major"]:
+            print(f"MAJOR candidate: {s['n_major']} coverage gap(s).")
+        else:
+            print("OK: Methods/Results/disk reconciled.")
+
+    if args.out:
+        Path(args.out).parent.mkdir(parents=True, exist_ok=True)
+        Path(args.out).write_text(json.dumps(result, indent=2), encoding="utf-8")
+        if not args.quiet:
+            print(f"\nwrote {args.out}")
+
+    return 1 if (args.strict and result["summary"]["n_major"]) else 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())