@brainpilot/skills 0.0.6 → 0.0.8

package/skills/06_fMRI_Neuroimaging/dmt-har-med-skill/SKILL.md

@@ -0,0 +1,183 @@
+---
+name: dmt-har-med-skill
+description: "Use this skill whenever the user wants an end-to-end workflow for the DMT-HAR-MED dataset (ds006644), including download, BIDS organization, and processing of rs-fMRI data from a psychedelic intervention study. Triggers include: 'DMT-HAR-MED', 'DMT HAR MED', 'ds006644', 'process DMT data', 'psychedelic fMRI', or any request to run the DMT-HAR-MED pipeline. This is the NeuroClaw dataset-orchestration layer for DMT-HAR-MED."
+license: MIT License (NeuroClaw custom skill - freely modifiable within the project)
+layer: subagent
+skill_type: dataset
+dependencies:
+  - fmri-skill
+  - bids-organizer
+  - claw-shell
+---
+# DMT-HAR-MED Skill (Dataset-Orchestration Layer)
+
+## Overview
+`dmt-har-med-skill` is the NeuroClaw orchestration skill for the **DMT-HAR-MED** dataset (OpenNeuro ds006644).
+
+DMT-HAR-MED contains rs-fMRI data from 40 participants in a psychedelic intervention study investigating the effects of N,N-Dimethyltryptamine (DMT) on brain function. The dataset includes multiple experimental conditions (DMT, placebo) and behavioral/physiological measurements.
+
+It coordinates a fixed three-phase workflow:
+1. Download DMT-HAR-MED data from OpenNeuro.
+2. Prepare and validate BIDS-style data organization.
+3. Delegate fMRI processing to `fmri-skill`.
+
+It also provides **phenotype extraction** and **QC integration** paths:
+- Extract DMT-HAR-MED phenotype data (intervention conditions, behavioral measures).
+- Generate per-subject QC summaries with exclusion lists.
+
+This skill follows NeuroClaw hierarchy:
+- Defines **WHAT to do**, not low-level implementation details.
+- Does **not** execute direct shell commands itself.
+- Delegates all execution via `claw-shell` to base/tool skills.
+
+**Research use only.**
+
+---
+
+## Download Stage (Mandatory First Step)
+
+### Source
+DMT-HAR-MED data is available on OpenNeuro:
+- Website: https://openneuro.org/datasets/ds006644/versions/1.0.1
+
+### Supported DMT-HAR-MED Data Packages
+- **Imaging data**: rs-fMRI (NIfTI format)
+- **Phenotype data**: intervention conditions, behavioral and physiological measurements
+- **Participants**: 40 participants with psychedelic intervention
+
+### Delegation Rules for Download
+- Environment/setup checks: `dependency-planner` + `conda-env-manager`
+- OpenNeuro dataset download: `claw-shell` (via `openneuro` CLI or `datalad`)
+- Optional raw-data organization to BIDS-style staging: `bids-organizer`
+
+### Download Inputs to Confirm in Plan
+- Subject list scope (full or custom subset)
+- Destination directory with sufficient disk space
+
+---
+
+## Core Workflow (Never Bypassed)
+1. Identify user target: download, BIDS staging, or full preprocessing.
+2. Generate a numbered plan with tools, outputs, runtime, storage, and risks.
+3. Wait for explicit confirmation (`YES` / `execute` / `proceed`).
+4. On confirmation, run download stage first (if needed).
+5. After download success, verify/prepare BIDS staging using `scripts/reorganize_dmt_har_med.py`.
+6. Delegate to `fmri-skill` for rs-fMRI processing.
+7. If phenotype extraction is requested, run `scripts/extract_dmt_har_med_phenotype.py`.
+8. If QC summary is requested, run `scripts/dmt_har_med_qc_summary.py`.
+9. Save outputs into a DMT-HAR-MED-centered structure under `dmt_har_med_output/`.
+
+---
+
+## BIDS Preparation
+
+### Script: `scripts/reorganize_dmt_har_med.py`
+
+Verifies and optionally reorganizes DMT-HAR-MED data to BIDS-compliant layout.
+
+```bash
+python skills/dmt-har-med-skill/scripts/reorganize_dmt_har_med.py \
+  --input /path/to/dmt_har_med_raw \
+  --output /path/to/dmt_har_med_bids
+```
+
+Features:
+- BIDS validation and verification
+- Subject ID normalization
+- Modality routing: rs-fMRI
+- `dataset_description.json` and `participants.tsv` generation
+- Dry-run mode: `--dry-run` to preview
+
+---
+
+## Phenotype Extraction
+
+### Script: `scripts/extract_dmt_har_med_phenotype.py`
+
+```bash
+python skills/dmt-har-med-skill/scripts/extract_dmt_har_med_phenotype.py \
+  --phenotype-dir /path/to/dmt_har_med_raw/phenotype \
+  --output /path/to/dmt_har_med_output/phenotype/merged_phenotype.csv \
+  --imaging-ids /path/to/dmt_har_med_output/bids/participants.tsv
+```
+
+---
+
+## QC Integration
+
+### Script: `scripts/dmt_har_med_qc_summary.py`
+
+```bash
+python skills/dmt-har-med-skill/scripts/dmt_har_med_qc_summary.py \
+  --fmriprep-dir /path/to/dmt_har_med_output/fmriprep \
+  --output /path/to/dmt_har_med_output/qc/qc_summary.csv \
+  --exclude-output /path/to/dmt_har_med_output/qc/exclude_list.csv \
+  --fd-threshold 0.3
+```
+
+---
+
+## Recommended Output Layout
+All assets should be organized under `./dmt_har_med_output/`:
+- `dmt_har_med_output/raw/` (downloaded original files)
+- `dmt_har_med_output/bids/` (BIDS data)
+- `dmt_har_med_output/fmri/` (links or copies from `fmri_output/`)
+- `dmt_har_med_output/phenotype/` (merged phenotype tables)
+- `dmt_har_med_output/qc/` (QC summaries and exclusion lists)
+- `dmt_har_med_output/logs/` (download + orchestration logs)
+
+---
+
+## Benchmark Adapter Guidance
+
+For benchmark-style prompts, do not force the full `download -> staging -> multimodal processing` orchestration when the task is only asking for local DMT-HAR-MED data staging or organization.
+
+- If the task starts from raw DMT-HAR-MED data already present on disk and only asks for BIDS-style staging / validation:
+  - skip the mandatory download stage
+  - default to the narrow path `local raw DMT-HAR-MED discovery -> BIDS validation -> minimal metadata -> report`
+- In benchmark mode, do not require explicit confirmation before presenting the direct staging solution.
+
+---
+
+## Safety and Execution Policy
+- No execution before explicit plan confirmation.
+- All execution must be routed via `claw-shell`.
+- Missing dependencies must be resolved by `dependency-planner` before running.
+
+---
+
+## Important Notes and Limitations
+- DMT-HAR-MED is a specialized psychedelic intervention dataset; analysis requires domain expertise.
+- DMT-HAR-MED data from OpenNeuro is already in BIDS format; re-staging may not be needed.
+- The dataset includes multiple experimental conditions; event files define condition timing.
+- DMT-HAR-MED includes only rs-fMRI; no structural (T1w) or diffusion data is available.
+- `dmt-har-med-skill` is orchestration-only; detailed preprocessing logic remains in `fmri-skill`.
+
+---
+
+## When to Call This Skill
+- User asks for end-to-end DMT-HAR-MED workflow.
+- User asks to download DMT-HAR-MED data and then run rs-fMRI processing.
+- User needs BIDS validation for DMT-HAR-MED data.
+- User asks to extract DMT-HAR-MED phenotype data.
+- User needs DMT-HAR-MED-specific QC summaries.
+
+---
+
+## Complementary / Related Skills
+- `fmri-skill`
+- `bids-organizer`
+- `fmriprep-tool`
+- `dependency-planner`
+- `conda-env-manager`
+- `claw-shell`
+
+---
+
+## Reference
+- DMT-HAR-MED: https://openneuro.org/datasets/ds006644/versions/1.0.1
+- BIDS spec: https://bids.neuroimaging.io/
+
+Created At: 2026-05-06 01:56 HKT
+Last Updated At: 2026-05-06 01:56 HKT
+Author: chengwang96

package/skills/06_fMRI_Neuroimaging/dmt-har-med-skill/scripts/dmt_har_med_qc_summary.py

@@ -0,0 +1,96 @@
+#!/usr/bin/env python3
+"""Generate per-subject QC summaries and exclusion lists for DMT-HAR-MED."""
+import argparse
+import sys
+from pathlib import Path
+from typing import Dict, List, Tuple
+
+try:
+    import pandas as pd
+except ImportError:
+    print("Error: pandas is required.", file=sys.stderr)
+    sys.exit(1)
+
+
+def detect_delimiter(file_path: Path) -> str:
+    with open(file_path, "r", encoding="utf-8") as f:
+        return "\t" if "\t" in f.readline() else ","
+
+
+def collect_fmriprep_qc(fmriprep_dir: Path) -> Dict[str, Dict[str, float]]:
+    qc = {}
+    for confounds_file in fmriprep_dir.rglob("**/*_desc-confounds_timeseries.tsv"):
+        subject_id = next((p for p in confounds_file.parts if p.startswith("sub-")), None)
+        if not subject_id:
+            continue
+        try:
+            df = pd.read_csv(confounds_file, sep=detect_delimiter(confounds_file), low_memory=False)
+        except Exception:
+            continue
+        fd_col = next((c for c in ["framewise_displacement", "fd", "FD"] if c in df.columns), None)
+        if fd_col:
+            fd = pd.to_numeric(df[fd_col], errors="coerce").dropna()
+            if len(fd) > 0:
+                metrics = {"mean_fd": float(fd.mean()), "max_fd": float(fd.max()), "n_volumes": len(df)}
+                if subject_id in qc:
+                    qc[subject_id]["mean_fd"] = max(qc[subject_id].get("mean_fd", 0), metrics["mean_fd"])
+                    qc[subject_id]["max_fd"] = max(qc[subject_id].get("max_fd", 0), metrics["max_fd"])
+                    qc[subject_id]["n_volumes"] = qc[subject_id].get("n_volumes", 0) + metrics["n_volumes"]
+                else:
+                    qc[subject_id] = metrics
+    return qc
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Generate DMT-HAR-MED QC summaries.")
+    parser.add_argument("--fmriprep-dir", help="Path to fMRIPrep output directory")
+    parser.add_argument("--output", required=True, help="Output path for QC summary CSV")
+    parser.add_argument("--exclude-output", help="Output path for exclusion list CSV")
+    parser.add_argument("--fd-threshold", type=float, default=0.3, help="Mean FD threshold (default: 0.3)")
+    parser.add_argument("--max-fd-threshold", type=float, default=5.0, help="Max FD threshold (default: 5.0)")
+    args = parser.parse_args()
+
+    fmriprep_qc = {}
+    if args.fmriprep_dir:
+        fp_dir = Path(args.fmriprep_dir).resolve()
+        if fp_dir.exists():
+            print(f"Collecting fMRIPrep QC from {fp_dir}...")
+            fmriprep_qc = collect_fmriprep_qc(fp_dir)
+            print(f"  Found {len(fmriprep_qc)} subjects")
+
+    if not fmriprep_qc:
+        print("[ERROR] No QC data collected.", file=sys.stderr)
+        return 1
+
+    rows = []
+    excluded = []
+    for sub_id in sorted(fmriprep_qc.keys()):
+        fp = fmriprep_qc[sub_id]
+        reasons = []
+        if fp.get("mean_fd", 0) > args.fd_threshold:
+            reasons.append(f"mean_fd={fp['mean_fd']:.3f}>{args.fd_threshold}")
+        if fp.get("max_fd", 0) > args.max_fd_threshold:
+            reasons.append(f"max_fd={fp['max_fd']:.3f}>{args.max_fd_threshold}")
+        row = {"subject_id": sub_id, "mean_fd": fp.get("mean_fd"), "max_fd": fp.get("max_fd"), "n_volumes": fp.get("n_volumes"), "exclude": bool(reasons), "exclude_reasons": "; ".join(reasons)}
+        if reasons:
+            excluded.append(sub_id)
+        rows.append(row)
+
+    summary_df = pd.DataFrame(rows)
+    output_path = Path(args.output).resolve()
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    summary_df.to_csv(output_path, index=False)
+    print(f"\nQC Summary: {len(summary_df)} subjects -> {output_path}")
+    print(f"  Excluded: {len(excluded)} / {len(summary_df)} ({100*len(excluded)/max(len(summary_df),1):.1f}%)")
+
+    if args.exclude_output:
+        exclude_path = Path(args.exclude_output).resolve()
+        exclude_path.parent.mkdir(parents=True, exist_ok=True)
+        summary_df[summary_df["exclude"] == True][["subject_id", "exclude_reasons"]].to_csv(exclude_path, index=False)
+        print(f"  Exclusion list: {exclude_path}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/06_fMRI_Neuroimaging/dmt-har-med-skill/scripts/extract_dmt_har_med_phenotype.py

@@ -0,0 +1,121 @@
+#!/usr/bin/env python3
+"""Extract and merge DMT-HAR-MED phenotype data for downstream analysis."""
+import argparse
+import csv
+import re
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional
+
+try:
+    import pandas as pd
+except ImportError:
+    print("Error: pandas is required.", file=sys.stderr)
+    sys.exit(1)
+
+COLUMN_MAP = {
+    "participant_id": "subject_id",
+    "subject": "subject_id",
+    "SUBJECT_ID": "subject_id",
+    "group": "condition",
+    "GROUP": "condition",
+    "condition": "condition",
+    "age": "age",
+    "AGE": "age",
+    "sex": "sex",
+    "SEX": "sex",
+    "gender": "sex",
+    "session": "session",
+    "SESSION": "session",
+    "visit": "session",
+}
+
+CONDITION_LABELS = {
+    "DMT": "dmt",
+    "dmt": "dmt",
+    "placebo": "placebo",
+    "Placebo": "placebo",
+    "control": "placebo",
+    "Control": "placebo",
+}
+
+
+def normalize_subject_id(raw_id) -> str:
+    clean = re.sub(r"[^a-zA-Z0-9]", "", str(raw_id).strip())
+    return f"sub-{clean}"
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Extract DMT-HAR-MED phenotype data.")
+    parser.add_argument("--phenotype-dir", help="Directory containing phenotype CSV/TSV files")
+    parser.add_argument("--phenotype-file", help="Direct path to phenotype CSV/TSV file")
+    parser.add_argument("--output", required=True, help="Output path for phenotype CSV")
+    parser.add_argument("--imaging-ids", help="Path to BIDS participants.tsv for filtering")
+    args = parser.parse_args()
+
+    phenotype_files = []
+    if args.phenotype_file:
+        p = Path(args.phenotype_file).resolve()
+        if p.exists():
+            phenotype_files.append(p)
+    elif args.phenotype_dir:
+        phenotype_dir = Path(args.phenotype_dir).resolve()
+        if phenotype_dir.exists():
+            phenotype_files = sorted(f for f in phenotype_dir.iterdir() if f.is_file() and f.suffix in (".csv", ".tsv"))
+
+    if not phenotype_files:
+        print("[ERROR] No phenotype files found.", file=sys.stderr)
+        return 1
+
+    print(f"Found {len(phenotype_files)} phenotype file(s)")
+
+    dataframes = []
+    for f in phenotype_files:
+        try:
+            df = pd.read_csv(f, sep=None, engine="python", low_memory=False)
+            rename = {k: v for k, v in COLUMN_MAP.items() if k in df.columns and v not in df.columns}
+            df = df.rename(columns=rename)
+            if "condition" in df.columns:
+                df["condition"] = df["condition"].map(lambda x: CONDITION_LABELS.get(str(x), str(x).lower()))
+            dataframes.append(df)
+        except Exception as e:
+            print(f"[WARN] Failed to read {f}: {e}")
+
+    if not dataframes:
+        print("[ERROR] No data loaded.", file=sys.stderr)
+        return 1
+
+    merged = dataframes[0]
+    for df in dataframes[1:]:
+        if "subject_id" in merged.columns and "subject_id" in df.columns:
+            new_cols = [c for c in df.columns if c not in merged.columns or c == "subject_id"]
+            merged = pd.merge(merged, df[new_cols], on="subject_id", how="outer")
+        else:
+            merged = pd.concat([merged, df], ignore_index=True)
+
+    if args.imaging_ids:
+        imaging_path = Path(args.imaging_ids).resolve()
+        if imaging_path.exists() and "subject_id" in merged.columns:
+            id_df = pd.read_csv(imaging_path, sep="\t")
+            for col in ["participant_id", "subject_id"]:
+                if col in id_df.columns:
+                    imaging_ids = set(id_df[col].astype(str).str.strip())
+                    normalized = {normalize_subject_id(sid) for sid in imaging_ids} | imaging_ids
+                    merged["_norm"] = merged["subject_id"].apply(normalize_subject_id)
+                    merged = merged[merged["_norm"].isin(normalized) | merged["subject_id"].astype(str).isin(normalized)]
+                    merged = merged.drop(columns=["_norm"])
+                    break
+
+    output_path = Path(args.output).resolve()
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    merged.to_csv(output_path, index=False)
+    print(f"\nWrote {len(merged)} rows x {len(merged.columns)} columns to {output_path}")
+
+    if "condition" in merged.columns:
+        print(f"\nCondition distribution:\n{merged['condition'].value_counts().to_string()}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

package/skills/06_fMRI_Neuroimaging/dmt-har-med-skill/scripts/reorganize_dmt_har_med.py

@@ -0,0 +1,125 @@
+#!/usr/bin/env python3
+"""Validate and optionally reorganize DMT-HAR-MED OpenNeuro BIDS data."""
+import argparse
+import json
+import shutil
+import sys
+from pathlib import Path
+from typing import Dict, List, Optional
+
+BIDS_MODALITIES = {"T1w": "anat", "T2w": "anat", "FLAIR": "anat",
+                   "task-rest_bold": "func", "task-*_bold": "func",
+                   "dwi": "dwi", "bold": "func"}
+
+
+def validate_bids(bids_root: Path) -> Dict[str, List[str]]:
+    """Validate BIDS structure and return report."""
+    issues: Dict[str, List[str]] = {"errors": [], "warnings": []}
+    if not (bids_root / "dataset_description.json").exists():
+        issues["errors"].append("Missing dataset_description.json")
+    participants_tsv = bids_root / "participants.tsv"
+    if not participants_tsv.exists():
+        issues["warnings"].append("Missing participants.tsv")
+    subject_dirs = sorted(d for d in bids_root.iterdir() if d.is_dir() and d.name.startswith("sub-"))
+    if not subject_dirs:
+        issues["errors"].append("No subject directories found")
+    for sub_dir in subject_dirs:
+        session_dirs = [d for d in sub_dir.iterdir() if d.is_dir() and d.name.startswith("ses-")]
+        if not session_dirs:
+            session_dirs = [sub_dir]
+        for ses_dir in session_dirs:
+            func_dir = ses_dir / "func"
+            anat_dir = ses_dir / "anat"
+            if not func_dir.exists() and not anat_dir.exists():
+                issues["warnings"].append(f"{sub_dir.name}/{ses_dir.name}: no func/ or anat/ directory")
+    return issues
+
+
+def collect_nifti_manifest(bids_root: Path) -> List[Dict[str, str]]:
+    """Collect all NIfTI files in BIDS directory."""
+    manifest = []
+    for nifti in sorted(bids_root.rglob("*.nii.gz")):
+        rel = nifti.relative_to(bids_root)
+        parts = rel.parts
+        subject = next((p for p in parts if p.startswith("sub-")), "unknown")
+        session = next((p for p in parts if p.startswith("ses-")), "")
+        modality_dir = next((p for p in parts if p in ("anat", "func", "dwi", "fmap")), "unknown")
+        manifest.append({
+            "subject": subject,
+            "session": session,
+            "modality": modality_dir,
+            "filename": nifti.name,
+            "path": str(rel),
+        })
+    return manifest
+
+
+def copy_to_output(bids_root: Path, output_dir: Path, manifest: List[Dict[str, str]]) -> int:
+    """Copy BIDS files to output directory."""
+    output_dir.mkdir(parents=True, exist_ok=True)
+    copied = 0
+    for entry in manifest:
+        src = bids_root / entry["path"]
+        dst = output_dir / entry["path"]
+        dst.parent.mkdir(parents=True, exist_ok=True)
+        if not dst.exists():
+            shutil.copy2(str(src), str(dst))
+            copied += 1
+    return copied
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(description="Validate/reorganize DMT-HAR-MED BIDS data.")
+    parser.add_argument("--input", required=True, help="Path to DMT-HAR-MED BIDS directory")
+    parser.add_argument("--output", help="Output path for reorganized BIDS directory")
+    parser.add_argument("--validate-only", action="store_true", help="Only validate, don't copy")
+    parser.add_argument("--manifest-output", help="Output path for NIfTI manifest TSV")
+    args = parser.parse_args()
+
+    input_dir = Path(args.input).resolve()
+    if not input_dir.exists():
+        print(f"Input directory not found: {input_dir}", file=sys.stderr)
+        return 1
+
+    print(f"Validating BIDS structure at {input_dir}...")
+    issues = validate_bids(input_dir)
+
+    if issues["errors"]:
+        print(f"\n[ERRORS] ({len(issues['errors'])}):")
+        for e in issues["errors"]:
+            print(f"  - {e}")
+    if issues["warnings"]:
+        print(f"\n[WARNINGS] ({len(issues['warnings'])}):")
+        for w in issues["warnings"]:
+            print(f"  - {w}")
+    if not issues["errors"] and not issues["warnings"]:
+        print("  BIDS structure looks valid.")
+
+    manifest = collect_nifti_manifest(input_dir)
+    print(f"\nFound {len(manifest)} NIfTI files")
+
+    subjects = set(e["subject"] for e in manifest)
+    modalities = set(e["modality"] for e in manifest)
+    print(f"  Subjects: {len(subjects)}")
+    print(f"  Modalities: {', '.join(sorted(modalities))}")
+
+    if args.manifest_output:
+        import csv
+        manifest_path = Path(args.manifest_output).resolve()
+        manifest_path.parent.mkdir(parents=True, exist_ok=True)
+        with open(manifest_path, "w", encoding="utf-8", newline="") as f:
+            writer = csv.DictWriter(f, fieldnames=manifest[0].keys(), delimiter="\t")
+            writer.writeheader()
+            writer.writerows(manifest)
+        print(f"  Manifest: {manifest_path}")
+
+    if not args.validate_only and args.output:
+        output_dir = Path(args.output).resolve()
+        copied = copy_to_output(input_dir, output_dir, manifest)
+        print(f"\nCopied {copied} files to {output_dir}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())