@brainpilot/skills 0.0.6 → 0.0.7

package/skills/06_fMRI_Neuroimaging/aibl-skill/SKILL.md

@@ -0,0 +1,295 @@
+---
+name: aibl-skill
+description: "Use this skill whenever the user wants an end-to-end workflow for the AIBL (Australian Imaging, Biomarkers and Lifestyle) dataset, including data access guidance, BIDS organization, and multimodal processing of sMRI and PET (PiB, FDG, tau). Triggers include: 'AIBL', 'AIBL data', 'process AIBL', 'AIBL PET', 'AIBL MRI', or any request to run the AIBL multimodal pipeline. This is the NeuroClaw dataset-orchestration layer for AIBL."
+license: MIT License (NeuroClaw custom skill - freely modifiable within the project)
+layer: subagent
+skill_type: dataset
+dependencies:
+  - smri-skill
+  - bids-organizer
+  - claw-shell
+complementary_skills:
+  - pet-skill
+---
+# AIBL Skill (Dataset-Orchestration Layer)
+
+## Overview
+`aibl-skill` is the NeuroClaw orchestration skill for the **AIBL (Australian Imaging, Biomarkers and Lifestyle)** dataset.
+
+It coordinates a fixed three-phase workflow:
+1. Guide AIBL data access and download from the AIBL research portal.
+2. Prepare and validate BIDS-style data organization for downstream processing.
+3. Delegate modality pipelines to `smri-skill` for structural MRI and PET processing.
+
+It also provides **phenotype extraction** and **QC integration** paths:
+- Extract and merge AIBL phenotype tables (cognitive assessments, blood biomarkers, APOE genotype, lifestyle data).
+- Generate per-subject QC summaries with exclusion lists.
+
+This skill follows NeuroClaw hierarchy:
+- Defines **WHAT to do**, not low-level implementation details.
+- Does **not** execute direct shell commands itself.
+- Delegates all execution via `claw-shell` to base/tool skills.
+
+**Research use only.**
+
+---
+
+## Download Stage (Mandatory First Step)
+
+### Source
+AIBL data is distributed through the **AIBL research portal**:
+- Website: https://aibl.csiro.au/
+- Data access: requires registration and data use agreement
+- Imaging data available via LONI Image Data Archive (IDA): https://ida.loni.usc.edu/
+
+### Supported AIBL Data Packages
+- **Imaging data**: T1w MRI, PET (PiB amyloid, FDG metabolism, tau)
+- **Phenotype data**: cognitive assessments (CDR, MMSE, MoCA, ADAS-Cog), blood biomarkers (Aβ42, Aβ40, p-tau, NfL), APOE genotype, lifestyle and demographic data
+- **Derived imaging data**: FreeSurfer outputs (if available)
+
+### Delegation Rules for Download
+- Environment/setup checks: `dependency-planner` + `conda-env-manager`
+- LONI IDA download tool installation and execution: `claw-shell`
+- Optional raw-data organization to BIDS-style staging: `bids-organizer`
+
+### Download Inputs to Confirm in Plan
+- LONI IDA credentials/authorized access
+- Target data package (imaging only, phenotype only, or both)
+- Subject list scope (full cohort or custom subset)
+- AIBL phase (screening, baseline, 18-month, 36-month, 54-month, etc.)
+- Destination directory with sufficient disk space
+
+---
+
+## Narrow Path: AIBL Raw Data -> BIDS Staging
+
+Use this path when the task only asks to reorganize raw AIBL files into a BIDS-style dataset and does not require preprocessing, ROI extraction, phenotype merging, or downstream analysis.
+
+### When this narrow path should dominate
+- The task objective is limited to AIBL data staging, BIDS renaming, sidecar handling, and dataset-level metadata.
+- Inputs are already local AIBL files or AIBL-style subject/date folders.
+- The required deliverable is a direct staging script or command sequence, not a plan for preprocessing or downstream analysis.
+
+### Narrow-path contract
+- Do not widen the solution to preprocessing, ROI extraction, phenotype merging, or downstream analysis unless the task explicitly requires them.
+- Treat this as a direct file-organization problem: scan AIBL subject/session layout, normalize subject labels, map modalities to BIDS names, copy or symlink files plus matching sidecars, and write dataset-level metadata plus staging logs.
+- If the task is benchmark-style, prefer a single direct end-to-end staging script over a confirmation-first orchestration plan.
+
+### Expected narrow-path behavior
+1. Detect AIBL-style subject IDs (e.g., `002_S_0295`) and normalize to BIDS labels such as `sub-002S0295`.
+2. Detect visit/timepoint information and normalize to session labels such as `ses-screening`, `ses-baseline`, `ses-18month`, etc.
+3. Route modalities:
+   - T1w/MPRAGE -> `anat/*_T1w`
+   - PET PiB -> `pet/*_pet` (tracer: PiB)
+   - PET FDG -> `pet/*_pet` (tracer: FDG)
+   - PET tau -> `pet/*_pet` (tracer: tau)
+4. Preserve or rename matching JSON sidecars when available; if metadata is absent, create only the minimal dataset files required by the task and log the limitation.
+5. Emit dataset-level outputs such as `dataset_description.json`, `participants.tsv`, `README`, and a manifest or skipped-file report.
+
+---
+
+## Core Workflow (Never Bypassed)
+1. Identify user target: full AIBL processing, imaging subset, phenotype extraction, or BIDS staging only.
+2. Generate a numbered plan with tools, outputs, runtime, storage, and risks.
+3. Wait for explicit confirmation (`YES` / `execute` / `proceed`).
+4. On confirmation, run download stage first (if needed).
+5. After download success, run BIDS preparation using `scripts/reorganize_aibl.py`.
+6. Delegate to `smri-skill` for structural MRI processing (brain extraction, tissue segmentation, cortical reconstruction).
+7. If PET processing is requested, handle PET-specific preprocessing (spatial normalization, SUVR computation).
+8. If phenotype extraction is requested, run `scripts/extract_aibl_phenotype.py`.
+9. If QC summary is requested, run `scripts/aibl_qc_summary.py`.
+10. Save outputs into an AIBL-centered structure under `aibl_output/`.
+
+---
+
+## Input Layout (Example)
+
+Subject `002_S_0295` (T1w + PET):
+
+```
+aibl_raw/
+  002_S_0295/
+    screening/
+      T1/
+        002_S_0295_T1.nii.gz
+        002_S_0295_T1.json
+      PET_PiB/
+        002_S_0295_PET_PiB.nii.gz
+        002_S_0295_PET_PiB.json
+      PET_FDG/
+        002_S_0295_PET_FDG.nii.gz
+        002_S_0295_PET_FDG.json
+  phenotype/
+    cognitive_assessments.csv
+    blood_biomarkers.csv
+    apoe_genotype.csv
+    demographics.csv
+```
+
+---
+
+## BIDS Preparation
+
+### Script: `scripts/reorganize_aibl.py`
+
+Converts AIBL raw directory structure to BIDS-compliant layout.
+
+```bash
+python skills/aibl-skill/scripts/reorganize_aibl.py \
+  --input /path/to/aibl_raw \
+  --output /path/to/aibl_bids \
+  --participants-file /path/to/aibl_raw/phenotype/demographics.csv
+```
+
+Features:
+- Subject ID normalization: AIBL format to BIDS `sub-002S0295`
+- Session mapping: AIBL visit names to BIDS `ses-` labels
+- Modality routing: T1w, PET (PiB, FDG, tau)
+- Sidecar JSON preservation and validation
+- `dataset_description.json` and `participants.tsv` generation
+- Dry-run mode: `--dry-run` to preview without copying
+
+---
+
+## Modality Processing Delegation
+
+After BIDS staging completes, `aibl-skill` delegates by modality:
+
+| Modality | Delegated skill | Typical tasks | Main outputs |
+|---|---|---|---|
+| sMRI (T1w) | `smri-skill` | brain extraction, tissue segmentation, cortical reconstruction, ROI morphometry | `smri_output/` derivatives and stats |
+| PET (PiB/FDG/tau) | `pet-skill` | spatial normalization to template, SUVR computation, reference region quantification | `pet_output/` SUVR maps and ROI values |
+
+### Delegation Strategy
+- If user asks for full multimodal AIBL analysis: run sMRI -> PET in ordered phases.
+- If user asks for one modality only: call only the corresponding modality skill.
+- PET processing typically requires co-registration to T1w first, then normalization to standard space.
+
+---
+
+## Phenotype Extraction
+
+### Script: `scripts/extract_aibl_phenotype.py`
+
+Extracts and merges AIBL phenotype tables for downstream analysis.
+
+```bash
+python skills/aibl-skill/scripts/extract_aibl_phenotype.py \
+  --phenotype-dir /path/to/aibl_raw/phenotype \
+  --output /path/to/aibl_output/phenotype/merged_phenotype.csv \
+  --columns subject_id,visit,diagnosis,age,sex,mmse,cdr,apoe \
+  --imaging-ids /path/to/aibl_output/bids/participants.tsv
+```
+
+Features:
+- Reads AIBL phenotype CSV/TSV files (cognitive, biomarker, genetic, demographic)
+- Column selection and renaming
+- Visit alignment (screening, baseline, 18-month, 36-month, 54-month)
+- Missing value handling (filter or impute)
+- Cross-reference with imaging subject list to keep only subjects with both imaging and phenotype data
+- Diagnostic group classification: healthy controls (HC), mild cognitive impairment (MCI), Alzheimer's disease (AD)
+- Outputs merged CSV ready for statistical analysis or model training
+
+---
+
+## QC Integration
+
+### Script: `scripts/aibl_qc_summary.py`
+
+Generates per-subject QC summaries and exclusion lists.
+
+```bash
+python skills/aibl-skill/scripts/aibl_qc_summary.py \
+  --fmriprep-dir /path/to/aibl_output/fmriprep \
+  --freesurfer-dir /path/to/aibl_output/smri/freesurfer \
+  --output /path/to/aibl_output/qc/qc_summary.csv \
+  --exclude-output /path/to/aibl_output/qc/exclude_list.csv \
+  --fd-threshold 0.3
+```
+
+Features:
+- Reads fMRIPrep confounds (if fMRI data available)
+- Reads FreeSurfer recon-all QC metrics
+- Structural quality assessment (motion artifacts, coverage)
+- Applies exclusion criteria: motion threshold (FD), structural quality
+- Outputs per-subject QC summary CSV and exclusion list CSV
+
+---
+
+## Recommended Output Layout
+All assets should be organized under `./aibl_output/`:
+- `aibl_output/raw/` (downloaded original AIBL files)
+- `aibl_output/bids/` (staged BIDS data)
+- `aibl_output/smri/` (links or copies from `smri_output/`)
+- `aibl_output/pet/` (PET processing outputs)
+- `aibl_output/phenotype/` (merged phenotype tables)
+- `aibl_output/qc/` (QC summaries and exclusion lists)
+- `aibl_output/logs/` (download + orchestration logs)
+
+---
+
+## Benchmark Adapter Guidance
+
+For benchmark-style prompts, do not force the full `download -> staging -> multimodal processing` orchestration when the task is only asking for local AIBL data staging or organization.
+
+- If the task starts from raw AIBL data already present on disk and only asks for BIDS-style staging / organization:
+  - skip the mandatory download stage
+  - do not automatically delegate to `smri-skill`
+  - default to the narrow path `local raw AIBL discovery -> BIDS-style staging -> minimal metadata -> validation/report`
+- In benchmark mode, do not require explicit confirmation before presenting the direct staging solution.
+- Preserve the AIBL-centered output contract under `aibl_output/bids/` when the task is specifically a staging benchmark.
+- Only use the full multimodal orchestration and confirmation-heavy workflow when the prompt explicitly asks for download, end-to-end multimodal AIBL processing, or post-staging structural / PET analysis.
+
+---
+
+## Safety and Execution Policy
+- No execution before explicit plan confirmation.
+- All execution must be routed via `claw-shell`.
+- Missing dependencies must be resolved by `dependency-planner` before running.
+- If download fails for partial subjects, continue batch with clear failure report and retry list.
+
+---
+
+## Important Notes and Limitations
+- AIBL imaging data includes both structural MRI and multiple PET tracers (PiB, FDG, tau). PET processing requires tracer-specific workflows.
+- LONI IDA download requires authenticated access and compliance with the AIBL Data Use Agreement.
+- AIBL subject IDs follow the format `XXX_S_XXXX`; normalization to BIDS labels must be consistent across all stages.
+- AIBL has multiple follow-up timepoints (screening through 54-month); session handling must account for longitudinal structure.
+- AIBL phenotype tables may use different delimiter formats across releases; auto-detection is recommended.
+- PET SUVR computation requires reference region definition (e.g., cerebellar cortex for PiB, pons for FDG).
+- `aibl-skill` is orchestration-only; detailed preprocessing logic remains in `smri-skill` and PET processing tools.
+
+---
+
+## When to Call This Skill
+- User asks for end-to-end AIBL workflow.
+- User asks to process AIBL MRI and/or PET data.
+- User needs BIDS staging for raw AIBL files.
+- User asks to extract and merge AIBL phenotype tables (cognitive, biomarker, genetic).
+- User asks for AIBL-specific QC summaries and exclusion lists.
+- User needs a single entry point for AIBL multimodal orchestration.
+
+---
+
+## Complementary / Related Skills
+- `smri-skill`
+- `pet-skill` → PET processing (SUVR computation, tracer-specific reference regions, partial volume correction)
+- `bids-organizer`
+- `freesurfer-tool`
+- `nibabel-skill`
+- `brain-visualization`
+- `dependency-planner`
+- `conda-env-manager`
+- `claw-shell`
+
+---
+
+## Reference
+- AIBL Study: https://aibl.csiro.au/
+- LONI IDA: https://ida.loni.usc.edu/
+- BIDS spec: https://bids.neuroimaging.io/
+- BIDS PET extension: https://bids-specification.readthedocs.io/en/stable/04-modality-specific-files/09-positron-emission-tomography.html
+
+Created At: 2026-05-06 11:24 HKT
+Last Updated At: 2026-05-06 12:19 HKT
+Author: chengwang96

package/skills/06_fMRI_Neuroimaging/aibl-skill/scripts/aibl_qc_summary.py

@@ -0,0 +1,260 @@
+#!/usr/bin/env python3
+"""Generate per-subject QC summaries and exclusion lists for AIBL.
+
+Combines fMRIPrep confounds (if available), FreeSurfer recon-all metrics,
+and structural quality assessment to produce a unified QC report.
+"""
+import argparse
+import sys
+from pathlib import Path
+from typing import Dict, List, Tuple
+
+try:
+    import pandas as pd
+except ImportError:
+    print("Error: pandas is required. Install with: pip install pandas", file=sys.stderr)
+    sys.exit(1)
+
+
+def detect_delimiter(file_path: Path) -> str:
+    """Detect whether file uses tab or comma delimiter."""
+    with open(file_path, "r", encoding="utf-8") as f:
+        first_line = f.readline()
+    if "\t" in first_line:
+        return "\t"
+    return ","
+
+
+def collect_fmriprep_qc(fmriprep_dir: Path) -> Dict[str, Dict[str, float]]:
+    """Collect QC metrics from fMRIPrep outputs (if fMRI data available)."""
+    qc = {}
+
+    confounds_patterns = [
+        "**/desc-confounds_timeseries.tsv",
+        "**/*_desc-confounds_timeseries.tsv",
+        "**/confounds.tsv",
+    ]
+
+    confounds_files = []
+    for pattern in confounds_patterns:
+        confounds_files.extend(fmriprep_dir.rglob(pattern))
+
+    for confounds_file in confounds_files:
+        subject_id = None
+        for part in confounds_file.parts:
+            if part.startswith("sub-"):
+                subject_id = part
+                break
+
+        if not subject_id:
+            continue
+
+        delimiter = detect_delimiter(confounds_file)
+        try:
+            df = pd.read_csv(confounds_file, sep=delimiter, low_memory=False)
+        except Exception:
+            continue
+
+        fd_col = None
+        for col_name in ["framewise_displacement", "fd", "FD"]:
+            if col_name in df.columns:
+                fd_col = col_name
+                break
+
+        metrics = {"n_volumes": len(df)}
+
+        if fd_col:
+            fd_values = pd.to_numeric(df[fd_col], errors="coerce").dropna()
+            if len(fd_values) > 0:
+                metrics["mean_fd"] = float(fd_values.mean())
+                metrics["max_fd"] = float(fd_values.max())
+            else:
+                metrics["mean_fd"] = 0.0
+                metrics["max_fd"] = 0.0
+
+        if subject_id in qc:
+            existing = qc[subject_id]
+            existing["mean_fd"] = max(existing.get("mean_fd", 0), metrics.get("mean_fd", 0))
+            existing["max_fd"] = max(existing.get("max_fd", 0), metrics.get("max_fd", 0))
+            existing["n_volumes"] = existing.get("n_volumes", 0) + metrics.get("n_volumes", 0)
+        else:
+            qc[subject_id] = metrics
+
+    return qc
+
+
+def collect_freesurfer_qc(freesurfer_dir: Path) -> Dict[str, Dict[str, float]]:
+    """Collect QC metrics from FreeSurfer recon-all outputs."""
+    qc = {}
+
+    for subject_dir in sorted(freesurfer_dir.rglob("sub-*")):
+        if not subject_dir.is_dir():
+            continue
+
+        subject_id = None
+        for part in subject_dir.parts:
+            if part.startswith("sub-"):
+                subject_id = part
+                break
+        if not subject_id:
+            subject_id = subject_dir.name
+
+        # Check for recon-all log
+        recon_log = None
+        matches = list(freesurfer_dir.rglob(f"{subject_id}/**/recon-all.log"))
+        if matches:
+            recon_log = matches[0]
+
+        completed = False
+        if recon_log and recon_log.exists():
+            try:
+                with open(recon_log, "r", encoding="utf-8", errors="ignore") as f:
+                    content = f.read()
+                    completed = "finished without error" in content.lower() or "recon-all -done" in content.lower()
+            except Exception:
+                pass
+
+        # Read aseg.stats for volume metrics
+        aseg_matches = list(freesurfer_dir.rglob(f"{subject_id}**/aseg.stats"))
+        total_brain_vol = None
+        etiv = None
+
+        if aseg_matches:
+            aseg_file = aseg_matches[0]
+            try:
+                with open(aseg_file, "r", encoding="utf-8", errors="ignore") as f:
+                    for line in f:
+                        if "EstimatedTotalIntraCranialVol" in line:
+                            parts = line.split()
+                            if len(parts) >= 4:
+                                try:
+                                    etiv = float(parts[3])
+                                except ValueError:
+                                    pass
+                        elif "BrainSegVol" in line and "Not" not in line:
+                            parts = line.split()
+                            if len(parts) >= 4:
+                                try:
+                                    total_brain_vol = float(parts[3])
+                                except ValueError:
+                                    pass
+            except Exception:
+                pass
+
+        metrics = {"completed": completed}
+        if total_brain_vol is not None:
+            metrics["total_brain_volume"] = total_brain_vol
+        if etiv is not None:
+            metrics["estimated_total_intracranial_volume"] = etiv
+
+        qc[subject_id] = metrics
+
+    return qc
+
+
+def generate_qc_summary(
+    fmriprep_qc: Dict[str, Dict[str, float]],
+    freesurfer_qc: Dict[str, Dict[str, float]],
+    fd_threshold: float = 0.3,
+    max_fd_threshold: float = 5.0,
+) -> Tuple["pd.DataFrame", List[str]]:
+    """Generate QC summary DataFrame and exclusion list."""
+    all_subjects = set()
+    all_subjects.update(fmriprep_qc.keys())
+    all_subjects.update(freesurfer_qc.keys())
+
+    rows = []
+    excluded = []
+
+    for sub_id in sorted(all_subjects):
+        row = {"subject_id": sub_id}
+        exclude_reasons = []
+
+        fp = fmriprep_qc.get(sub_id, {})
+        row["mean_fd"] = fp.get("mean_fd", None)
+        row["max_fd"] = fp.get("max_fd", None)
+        row["n_volumes"] = fp.get("n_volumes", None)
+
+        fs = freesurfer_qc.get(sub_id, {})
+        row["fs_completed"] = fs.get("completed", None)
+        row["total_brain_volume"] = fs.get("total_brain_volume", None)
+        row["etiv"] = fs.get("estimated_total_intracranial_volume", None)
+
+        if row["mean_fd"] is not None and row["mean_fd"] > fd_threshold:
+            exclude_reasons.append(f"mean_fd={row['mean_fd']:.3f}>{fd_threshold}")
+
+        if row["max_fd"] is not None and row["max_fd"] > max_fd_threshold:
+            exclude_reasons.append(f"max_fd={row['max_fd']:.3f}>{max_fd_threshold}")
+
+        if row["fs_completed"] is False:
+            exclude_reasons.append("FreeSurfer recon-all incomplete")
+
+        row["exclude"] = len(exclude_reasons) > 0
+        row["exclude_reasons"] = "; ".join(exclude_reasons) if exclude_reasons else ""
+
+        if exclude_reasons:
+            excluded.append(sub_id)
+
+        rows.append(row)
+
+    df = pd.DataFrame(rows)
+    return df, excluded
+
+
+def main() -> int:
+    parser = argparse.ArgumentParser(
+        description="Generate per-subject QC summaries and exclusion lists for AIBL."
+    )
+    parser.add_argument("--fmriprep-dir", help="Path to fMRIPrep output directory")
+    parser.add_argument("--freesurfer-dir", help="Path to FreeSurfer output directory")
+    parser.add_argument("--output", required=True, help="Output path for QC summary CSV")
+    parser.add_argument("--exclude-output", help="Output path for exclusion list CSV")
+    parser.add_argument("--fd-threshold", type=float, default=0.3, help="Mean FD threshold (default: 0.3)")
+    parser.add_argument("--max-fd-threshold", type=float, default=5.0, help="Max FD threshold (default: 5.0)")
+    args = parser.parse_args()
+
+    fmriprep_qc = {}
+    freesurfer_qc = {}
+
+    if args.fmriprep_dir:
+        fp_dir = Path(args.fmriprep_dir).resolve()
+        if fp_dir.exists():
+            print(f"Collecting fMRIPrep QC from {fp_dir}...")
+            fmriprep_qc = collect_fmriprep_qc(fp_dir)
+            print(f"  Found {len(fmriprep_qc)} subjects")
+
+    if args.freesurfer_dir:
+        fs_dir = Path(args.freesurfer_dir).resolve()
+        if fs_dir.exists():
+            print(f"Collecting FreeSurfer QC from {fs_dir}...")
+            freesurfer_qc = collect_freesurfer_qc(fs_dir)
+            print(f"  Found {len(freesurfer_qc)} subjects")
+
+    if not fmriprep_qc and not freesurfer_qc:
+        print("[ERROR] No QC data collected. Check input paths.", file=sys.stderr)
+        return 1
+
+    summary_df, excluded = generate_qc_summary(
+        fmriprep_qc=fmriprep_qc,
+        freesurfer_qc=freesurfer_qc,
+        fd_threshold=args.fd_threshold,
+        max_fd_threshold=args.max_fd_threshold,
+    )
+
+    output_path = Path(args.output).resolve()
+    output_path.parent.mkdir(parents=True, exist_ok=True)
+    summary_df.to_csv(output_path, index=False)
+    print(f"\nQC Summary: {len(summary_df)} subjects -> {output_path}")
+    print(f"  Excluded: {len(excluded)} / {len(summary_df)} ({100*len(excluded)/max(len(summary_df),1):.1f}%)")
+
+    if args.exclude_output:
+        exclude_path = Path(args.exclude_output).resolve()
+        exclude_path.parent.mkdir(parents=True, exist_ok=True)
+        summary_df[summary_df["exclude"] == True][["subject_id", "exclude_reasons"]].to_csv(exclude_path, index=False)
+        print(f"  Exclusion list: {exclude_path}")
+
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())