@bgicli/bgicli 2.1.1 → 2.2.0

package/data/skills/exosome-ev-analysis-agent/SKILL.md

@@ -0,0 +1,171 @@
+<!--
+# COPYRIGHT NOTICE
+# This file is part of the "Universal Biomedical Skills" project.
+# Copyright (c) 2026 MD BABU MIA, PhD <md.babu.mia@mssm.edu>
+# All Rights Reserved.
+#
+# This code is proprietary and confidential.
+# Unauthorized copying of this file, via any medium is strictly prohibited.
+#
+# Provenance: Authenticated by MD BABU MIA
+
+-->
+
+---
+name: 'exosome-ev-analysis-agent'
+description: 'AI-powered extracellular vesicle and exosome analysis for cancer biomarker discovery, liquid biopsy applications, and intercellular communication profiling.'
+measurable_outcome: Execute skill workflow successfully with valid output within 15 minutes.
+allowed-tools:
+  - read_file
+  - run_shell_command
+---
+
+
+# Exosome/EV Analysis Agent
+
+The **Exosome/EV Analysis Agent** provides comprehensive AI-driven analysis of extracellular vesicles for cancer biomarker discovery, liquid biopsy applications, and tumor-microenvironment communication profiling.
+
+## When to Use This Skill
+
+* When analyzing exosome cargo (RNA, protein, lipids) for biomarker discovery.
+* To identify tumor-derived EVs in liquid biopsy samples.
+* For profiling EV-mediated intercellular communication in cancer.
+* When predicting EV uptake and functional effects on recipient cells.
+* To design EV-based diagnostic or therapeutic applications.
+
+## Core Capabilities
+
+1. **EV Cargo Profiling**: Analyze exosomal RNA (miRNA, lncRNA, circRNA), proteins, and lipids.
+
+2. **Tumor EV Identification**: Distinguish tumor-derived EVs from normal EVs using surface markers and cargo.
+
+3. **Biomarker Discovery**: ML-driven identification of cancer-specific EV signatures.
+
+4. **Communication Network**: Map EV-mediated signaling between tumor and TME cells.
+
+5. **Functional Prediction**: Predict downstream effects of EV cargo on recipient cells.
+
+6. **Diagnostic Development**: Support EV-based diagnostic assay design.
+
+## EV Classification
+
+| Type | Size | Origin | Markers |
+|------|------|--------|---------|
+| Exosomes | 30-150 nm | MVB fusion | CD9, CD63, CD81 |
+| Microvesicles | 100-1000 nm | Membrane budding | Annexin V, ARF6 |
+| Apoptotic bodies | 500-5000 nm | Cell death | Annexin V, PS |
+| Large oncosomes | 1-10 μm | Tumor-specific | Variable |
+
+## Workflow
+
+1. **Input**: EV isolation method, cargo profiling data (RNA-seq, proteomics), characterization data.
+
+2. **Quality Assessment**: Evaluate EV purity and characterization (NTA, TEM, markers).
+
+3. **Cargo Analysis**: Profile RNA, protein, and lipid content.
+
+4. **Source Deconvolution**: Identify tumor vs stromal EV origin.
+
+5. **Biomarker Selection**: Identify cancer-specific signatures.
+
+6. **Functional Prediction**: Predict effects on recipient cells.
+
+7. **Output**: EV profile, biomarker candidates, functional predictions.
+
+## Example Usage
+
+**User**: "Analyze exosomal miRNA profiles from plasma samples to identify pancreatic cancer biomarkers."
+
+**Agent Action**:
+```bash
+python3 Skills/Oncology/Exosome_EV_Analysis_Agent/ev_analyzer.py \
+    --ev_mirna exosome_smallrna.tsv \
+    --ev_protein exosome_proteome.tsv \
+    --sample_groups pancreatic_cancer,healthy \
+    --normalization spike_in \
+    --biomarker_discovery true \
+    --output ev_biomarker_report/
+```
+
+## Exosomal miRNA Cancer Biomarkers
+
+| Cancer Type | Elevated miRNAs | Clinical Use |
+|-------------|-----------------|--------------|
+| Pancreatic | miR-21, miR-17-5p, miR-155 | Early detection |
+| Lung | miR-21, miR-126, miR-210 | Screening |
+| Colorectal | miR-21, miR-92a, miR-29a | Detection |
+| Prostate | miR-141, miR-375, miR-1290 | Prognosis |
+| Ovarian | miR-21, miR-141, miR-200 family | Detection |
+| Breast | miR-21, miR-155, miR-10b | Metastasis |
+
+## EV Isolation Methods
+
+| Method | Principle | Purity | Yield | Scalability |
+|--------|-----------|--------|-------|-------------|
+| Ultracentrifugation | Density | Moderate | High | Low |
+| Size exclusion | Size | High | Moderate | Moderate |
+| Immunocapture | Surface markers | Very high | Low | Low |
+| Precipitation | Polymer | Low | Very high | High |
+| Microfluidics | Various | Variable | Low | Low |
+
+## AI/ML Components
+
+**Biomarker Discovery**:
+- Differential expression analysis
+- Machine learning feature selection
+- Multi-marker panel optimization
+- Cross-validation and independent validation
+
+**Source Deconvolution**:
+- Marker-based classification
+- ML models for tumor vs normal EVs
+- Cell-type specific cargo signatures
+
+**Functional Prediction**:
+- miRNA target prediction
+- Pathway enrichment
+- Recipient cell effect modeling
+
+## EV Characterization Quality
+
+**MISEV Guidelines Requirements**:
+- Particle concentration (NTA/TRPS)
+- Size distribution (NTA/DLS/TEM)
+- Protein markers (CD9/63/81, TSG101, ALIX)
+- Negative markers (calnexin, albumin)
+- Morphology (TEM)
+
+## Clinical Applications
+
+1. **Early Detection**: Cancer screening from blood EVs
+2. **Prognosis**: EV signatures predicting outcomes
+3. **Therapy Response**: Monitor treatment effect
+4. **Metastasis**: Predict metastatic potential
+5. **Resistance**: Identify resistance mechanisms
+
+## Prerequisites
+
+* Python 3.10+
+* Small RNA analysis tools
+* Proteomics analysis packages
+* ML frameworks (scikit-learn, XGBoost)
+
+## Related Skills
+
+* Liquid_Biopsy_Analytics_Agent - For other liquid biopsy analytes
+* Tumor_Microenvironment - For TME communication
+* Cell-Free RNA Analysis - For plasma RNA
+
+## Emerging Applications
+
+1. **EV-based Drug Delivery**: Therapeutic cargo loading
+2. **EV Engineering**: Surface modification for targeting
+3. **Tumor Vaccines**: EV-based immunotherapy
+4. **Companion Diagnostics**: Treatment selection markers
+
+## Author
+
+AI Group - Biomedical AI Platform
+
+
+<!-- AUTHOR_SIGNATURE: 9a7f3c2e-MD-BABU-MIA-2026-MSSM-SECURE -->

package/data/skills/exploratory-data-analysis/SKILL.md

@@ -0,0 +1,440 @@
+---
+name: exploratory-data-analysis
+description: Perform comprehensive exploratory data analysis on scientific data files across 200+ file formats. This skill should be used when analyzing any scientific data file to understand its structure, content, quality, and characteristics. Automatically detects file type and generates detailed markdown reports with format-specific analysis, quality metrics, and downstream analysis recommendations. Covers chemistry, bioinformatics, microscopy, spectroscopy, proteomics, metabolomics, and general scientific data formats.
+---
+
+# Exploratory Data Analysis
+
+## Overview
+
+Perform comprehensive exploratory data analysis (EDA) on scientific data files across multiple domains. This skill provides automated file type detection, format-specific analysis, data quality assessment, and generates detailed markdown reports suitable for documentation and downstream analysis planning.
+
+**Key Capabilities:**
+- Automatic detection and analysis of 200+ scientific file formats
+- Comprehensive format-specific metadata extraction
+- Data quality and integrity assessment
+- Statistical summaries and distributions
+- Visualization recommendations
+- Downstream analysis suggestions
+- Markdown report generation
+
+## When to Use This Skill
+
+Use this skill when:
+- User provides a path to a scientific data file for analysis
+- User asks to "explore", "analyze", or "summarize" a data file
+- User wants to understand the structure and content of scientific data
+- User needs a comprehensive report of a dataset before analysis
+- User wants to assess data quality or completeness
+- User asks what type of analysis is appropriate for a file
+
+## Supported File Categories
+
+The skill has comprehensive coverage of scientific file formats organized into six major categories:
+
+### 1. Chemistry and Molecular Formats (60+ extensions)
+Structure files, computational chemistry outputs, molecular dynamics trajectories, and chemical databases.
+
+**File types include:** `.pdb`, `.cif`, `.mol`, `.mol2`, `.sdf`, `.xyz`, `.smi`, `.gro`, `.log`, `.fchk`, `.cube`, `.dcd`, `.xtc`, `.trr`, `.prmtop`, `.psf`, and more.
+
+**Reference file:** `references/chemistry_molecular_formats.md`
+
+### 2. Bioinformatics and Genomics Formats (50+ extensions)
+Sequence data, alignments, annotations, variants, and expression data.
+
+**File types include:** `.fasta`, `.fastq`, `.sam`, `.bam`, `.vcf`, `.bed`, `.gff`, `.gtf`, `.bigwig`, `.h5ad`, `.loom`, `.counts`, `.mtx`, and more.
+
+**Reference file:** `references/bioinformatics_genomics_formats.md`
+
+### 3. Microscopy and Imaging Formats (45+ extensions)
+Microscopy images, medical imaging, whole slide imaging, and electron microscopy.
+
+**File types include:** `.tif`, `.nd2`, `.lif`, `.czi`, `.ims`, `.dcm`, `.nii`, `.mrc`, `.dm3`, `.vsi`, `.svs`, `.ome.tiff`, and more.
+
+**Reference file:** `references/microscopy_imaging_formats.md`
+
+### 4. Spectroscopy and Analytical Chemistry Formats (35+ extensions)
+NMR, mass spectrometry, IR/Raman, UV-Vis, X-ray, chromatography, and other analytical techniques.
+
+**File types include:** `.fid`, `.mzML`, `.mzXML`, `.raw`, `.mgf`, `.spc`, `.jdx`, `.xy`, `.cif` (crystallography), `.wdf`, and more.
+
+**Reference file:** `references/spectroscopy_analytical_formats.md`
+
+### 5. Proteomics and Metabolomics Formats (30+ extensions)
+Mass spec proteomics, metabolomics, lipidomics, and multi-omics data.
+
+**File types include:** `.mzML`, `.pepXML`, `.protXML`, `.mzid`, `.mzTab`, `.sky`, `.mgf`, `.msp`, `.h5ad`, and more.
+
+**Reference file:** `references/proteomics_metabolomics_formats.md`
+
+### 6. General Scientific Data Formats (30+ extensions)
+Arrays, tables, hierarchical data, compressed archives, and common scientific formats.
+
+**File types include:** `.npy`, `.npz`, `.csv`, `.xlsx`, `.json`, `.hdf5`, `.zarr`, `.parquet`, `.mat`, `.fits`, `.nc`, `.xml`, and more.
+
+**Reference file:** `references/general_scientific_formats.md`
+
+## Workflow
+
+### Step 1: File Type Detection
+
+When a user provides a file path, first identify the file type:
+
+1. Extract the file extension
+2. Look up the extension in the appropriate reference file
+3. Identify the file category and format description
+4. Load format-specific information
+
+**Example:**
+```
+User: "Analyze data.fastq"
+→ Extension: .fastq
+→ Category: bioinformatics_genomics
+→ Format: FASTQ Format (sequence data with quality scores)
+→ Reference: references/bioinformatics_genomics_formats.md
+```
+
+### Step 2: Load Format-Specific Information
+
+Based on the file type, read the corresponding reference file to understand:
+- **Typical Data:** What kind of data this format contains
+- **Use Cases:** Common applications for this format
+- **Python Libraries:** How to read the file in Python
+- **EDA Approach:** What analyses are appropriate for this data type
+
+Search the reference file for the specific extension (e.g., search for "### .fastq" in `bioinformatics_genomics_formats.md`).
+
+### Step 3: Perform Data Analysis
+
+Use the `scripts/eda_analyzer.py` script OR implement custom analysis:
+
+**Option A: Use the analyzer script**
+```python
+# The script automatically:
+# 1. Detects file type
+# 2. Loads reference information
+# 3. Performs format-specific analysis
+# 4. Generates markdown report
+
+python scripts/eda_analyzer.py <filepath> [output.md]
+```
+
+**Option B: Custom analysis in the conversation**
+Based on the format information from the reference file, perform appropriate analysis:
+
+For tabular data (CSV, TSV, Excel):
+- Load with pandas
+- Check dimensions, data types
+- Analyze missing values
+- Calculate summary statistics
+- Identify outliers
+- Check for duplicates
+
+For sequence data (FASTA, FASTQ):
+- Count sequences
+- Analyze length distributions
+- Calculate GC content
+- Assess quality scores (FASTQ)
+
+For images (TIFF, ND2, CZI):
+- Check dimensions (X, Y, Z, C, T)
+- Analyze bit depth and value range
+- Extract metadata (channels, timestamps, spatial calibration)
+- Calculate intensity statistics
+
+For arrays (NPY, HDF5):
+- Check shape and dimensions
+- Analyze data type
+- Calculate statistical summaries
+- Check for missing/invalid values
+
+### Step 4: Generate Comprehensive Report
+
+Create a markdown report with the following sections:
+
+#### Required Sections:
+1. **Title and Metadata**
+   - Filename and timestamp
+   - File size and location
+
+2. **Basic Information**
+   - File properties
+   - Format identification
+
+3. **File Type Details**
+   - Format description from reference
+   - Typical data content
+   - Common use cases
+   - Python libraries for reading
+
+4. **Data Analysis**
+   - Structure and dimensions
+   - Statistical summaries
+   - Quality assessment
+   - Data characteristics
+
+5. **Key Findings**
+   - Notable patterns
+   - Potential issues
+   - Quality metrics
+
+6. **Recommendations**
+   - Preprocessing steps
+   - Appropriate analyses
+   - Tools and methods
+   - Visualization approaches
+
+#### Template Location
+Use `assets/report_template.md` as a guide for report structure.
+
+### Step 5: Save Report
+
+Save the markdown report with a descriptive filename:
+- Pattern: `{original_filename}_eda_report.md`
+- Example: `experiment_data.fastq` → `experiment_data_eda_report.md`
+
+## Detailed Format References
+
+Each reference file contains comprehensive information for dozens of file types. To find information about a specific format:
+
+1. Identify the category from the extension
+2. Read the appropriate reference file
+3. Search for the section heading matching the extension (e.g., "### .pdb")
+4. Extract the format information
+
+### Reference File Structure
+
+Each format entry includes:
+- **Description:** What the format is
+- **Typical Data:** What it contains
+- **Use Cases:** Common applications
+- **Python Libraries:** How to read it (with code examples)
+- **EDA Approach:** Specific analyses to perform
+
+**Example lookup:**
+```markdown
+### .pdb - Protein Data Bank
+**Description:** Standard format for 3D structures of biological macromolecules
+**Typical Data:** Atomic coordinates, residue information, secondary structure
+**Use Cases:** Protein structure analysis, molecular visualization, docking
+**Python Libraries:**
+- `Biopython`: `Bio.PDB`
+- `MDAnalysis`: `MDAnalysis.Universe('file.pdb')`
+**EDA Approach:**
+- Structure validation (bond lengths, angles)
+- B-factor distribution
+- Missing residues detection
+- Ramachandran plots
+```
+
+## Best Practices
+
+### Reading Reference Files
+
+Reference files are large (10,000+ words each). To efficiently use them:
+
+1. **Search by extension:** Use grep to find the specific format
+   ```python
+   import re
+   with open('references/chemistry_molecular_formats.md', 'r') as f:
+       content = f.read()
+       pattern = r'### \.pdb[^#]*?(?=###|\Z)'
+       match = re.search(pattern, content, re.IGNORECASE | re.DOTALL)
+   ```
+
+2. **Extract relevant sections:** Don't load entire reference files into context unnecessarily
+
+3. **Cache format info:** If analyzing multiple files of the same type, reuse the format information
+
+### Data Analysis
+
+1. **Sample large files:** For files with millions of records, analyze a representative sample
+2. **Handle errors gracefully:** Many scientific formats require specific libraries; provide clear installation instructions
+3. **Validate metadata:** Cross-check metadata consistency (e.g., stated dimensions vs actual data)
+4. **Consider data provenance:** Note instrument, software versions, processing steps
+
+### Report Generation
+
+1. **Be comprehensive:** Include all relevant information for downstream analysis
+2. **Be specific:** Provide concrete recommendations based on the file type
+3. **Be actionable:** Suggest specific next steps and tools
+4. **Include code examples:** Show how to load and work with the data
+
+## Examples
+
+### Example 1: Analyzing a FASTQ file
+
+```python
+# User provides: "Analyze reads.fastq"
+
+# 1. Detect file type
+extension = '.fastq'
+category = 'bioinformatics_genomics'
+
+# 2. Read reference info
+# Search references/bioinformatics_genomics_formats.md for "### .fastq"
+
+# 3. Perform analysis
+from Bio import SeqIO
+sequences = list(SeqIO.parse('reads.fastq', 'fastq'))
+# Calculate: read count, length distribution, quality scores, GC content
+
+# 4. Generate report
+# Include: format description, analysis results, QC recommendations
+
+# 5. Save as: reads_eda_report.md
+```
+
+### Example 2: Analyzing a CSV dataset
+
+```python
+# User provides: "Explore experiment_results.csv"
+
+# 1. Detect: .csv → general_scientific
+
+# 2. Load reference for CSV format
+
+# 3. Analyze
+import pandas as pd
+df = pd.read_csv('experiment_results.csv')
+# Dimensions, dtypes, missing values, statistics, correlations
+
+# 4. Generate report with:
+# - Data structure
+# - Missing value patterns
+# - Statistical summaries
+# - Correlation matrix
+# - Outlier detection results
+
+# 5. Save report
+```
+
+### Example 3: Analyzing microscopy data
+
+```python
+# User provides: "Analyze cells.nd2"
+
+# 1. Detect: .nd2 → microscopy_imaging (Nikon format)
+
+# 2. Read reference for ND2 format
+# Learn: multi-dimensional (XYZCT), requires nd2reader
+
+# 3. Analyze
+from nd2reader import ND2Reader
+with ND2Reader('cells.nd2') as images:
+    # Extract: dimensions, channels, timepoints, metadata
+    # Calculate: intensity statistics, frame info
+
+# 4. Generate report with:
+# - Image dimensions (XY, Z-stacks, time, channels)
+# - Channel wavelengths
+# - Pixel size and calibration
+# - Recommendations for image analysis
+
+# 5. Save report
+```
+
+## Troubleshooting
+
+### Missing Libraries
+
+Many scientific formats require specialized libraries:
+
+**Problem:** Import error when trying to read a file
+
+**Solution:** Provide clear installation instructions
+```python
+try:
+    from Bio import SeqIO
+except ImportError:
+    print("Install Biopython: uv pip install biopython")
+```
+
+Common requirements by category:
+- **Bioinformatics:** `biopython`, `pysam`, `pyBigWig`
+- **Chemistry:** `rdkit`, `mdanalysis`, `cclib`
+- **Microscopy:** `tifffile`, `nd2reader`, `aicsimageio`, `pydicom`
+- **Spectroscopy:** `nmrglue`, `pymzml`, `pyteomics`
+- **General:** `pandas`, `numpy`, `h5py`, `scipy`
+
+### Unknown File Types
+
+If a file extension is not in the references:
+
+1. Ask the user about the file format
+2. Check if it's a vendor-specific variant
+3. Attempt generic analysis based on file structure (text vs binary)
+4. Provide general recommendations
+
+### Large Files
+
+For very large files:
+
+1. Use sampling strategies (first N records)
+2. Use memory-mapped access (for HDF5, NPY)
+3. Process in chunks (for CSV, FASTQ)
+4. Provide estimates based on samples
+
+## Script Usage
+
+The `scripts/eda_analyzer.py` can be used directly:
+
+```bash
+# Basic usage
+python scripts/eda_analyzer.py data.csv
+
+# Specify output file
+python scripts/eda_analyzer.py data.csv output_report.md
+
+# The script will:
+# 1. Auto-detect file type
+# 2. Load format references
+# 3. Perform appropriate analysis
+# 4. Generate markdown report
+```
+
+The script supports automatic analysis for many common formats, but custom analysis in the conversation provides more flexibility and domain-specific insights.
+
+## Advanced Usage
+
+### Multi-File Analysis
+
+When analyzing multiple related files:
+1. Perform individual EDA on each file
+2. Create a summary comparison report
+3. Identify relationships and dependencies
+4. Suggest integration strategies
+
+### Quality Control
+
+For data quality assessment:
+1. Check format compliance
+2. Validate metadata consistency
+3. Assess completeness
+4. Identify outliers and anomalies
+5. Compare to expected ranges/distributions
+
+### Preprocessing Recommendations
+
+Based on data characteristics, recommend:
+1. Normalization strategies
+2. Missing value imputation
+3. Outlier handling
+4. Batch correction
+5. Format conversions
+
+## Resources
+
+### scripts/
+- `eda_analyzer.py`: Comprehensive analysis script that can be run directly or imported
+
+### references/
+- `chemistry_molecular_formats.md`: 60+ chemistry/molecular file formats
+- `bioinformatics_genomics_formats.md`: 50+ bioinformatics formats
+- `microscopy_imaging_formats.md`: 45+ imaging formats
+- `spectroscopy_analytical_formats.md`: 35+ spectroscopy formats
+- `proteomics_metabolomics_formats.md`: 30+ omics formats
+- `general_scientific_formats.md`: 30+ general formats
+
+### assets/
+- `report_template.md`: Comprehensive markdown template for EDA reports