@pikaa-ai/pikaa 0.2.5 → 0.3.1

package/skills/arboreto/SKILL.md

@@ -0,0 +1,267 @@
+---
+name: arboreto
+description: Infer gene regulatory networks (GRNs) from gene expression data using scalable algorithms (GRNBoost2, GENIE3). Use when analyzing transcriptomics data (bulk RNA-seq, single-cell RNA-seq) to identify transcription factor-target gene relationships and regulatory interactions. Supports distributed computation for large-scale datasets.
+license: BSD-3-Clause license
+metadata:
+  version: "1.0"
+  skill-author: K-Dense Inc.
+---
+
+# Arboreto
+
+## Overview
+
+Arboreto is a Python library from [Aerts Lab](https://github.com/aertslab/arboreto) for inferring gene regulatory networks (GRNs) from gene expression data. It parallelizes tree-based ensemble regression (GRNBoost2, GENIE3) with [Dask](https://distributed.dask.org/) across local cores or remote clusters.
+
+**Core capability**: Identify which transcription factors (TFs) regulate which target genes based on expression patterns across observations (cells, samples, conditions).
+
+**Upstream**: PyPI **0.1.6** (2021-02-09, latest). Docs: [arboreto.readthedocs.io](https://arboreto.readthedocs.io/en/latest/). Primary downstream consumer: [pySCENIC](https://github.com/aertslab/pySCENIC).
+
+## Quick Start
+
+Install arboreto:
+```bash
+uv pip install arboreto
+```
+
+Basic GRN inference:
+```python
+import pandas as pd
+from arboreto.algo import grnboost2
+
+if __name__ == '__main__':
+    # Load expression data (genes as columns)
+    expression_matrix = pd.read_csv('expression_data.tsv', sep='\t')
+
+    # Infer regulatory network
+    network = grnboost2(expression_data=expression_matrix)
+
+    # Save results (TF, target, importance)
+    network.to_csv('network.tsv', sep='\t', index=False, header=False)
+```
+
+**Critical**: Always use `if __name__ == '__main__':` guard because Dask spawns new processes.
+
+## Core Capabilities
+
+### 1. Basic GRN Inference
+
+For standard GRN inference workflows including:
+- Input data preparation (Pandas DataFrame or NumPy array)
+- Running inference with GRNBoost2 or GENIE3
+- Filtering by transcription factors
+- Output format and interpretation
+
+**See**: `references/basic_inference.md`
+
+**Use the ready-to-run script**: `scripts/basic_grn_inference.py` for standard inference tasks:
+```bash
+python scripts/basic_grn_inference.py expression_data.tsv output_network.tsv --tf-file tfs.txt --seed 777 --limit 5000
+```
+
+### 2. Algorithm Selection
+
+Arboreto provides two algorithms:
+
+**GRNBoost2 (Recommended)**:
+- Fast gradient boosting-based inference
+- Optimized for large datasets (10k+ observations)
+- Default choice for most analyses
+
+**GENIE3**:
+- Random Forest-based inference
+- Original multiple regression approach
+- Use for comparison or validation
+
+Quick comparison:
+```python
+from arboreto.algo import grnboost2, genie3
+
+# Fast, recommended
+network_grnboost = grnboost2(expression_data=matrix)
+
+# Classic algorithm
+network_genie3 = genie3(expression_data=matrix)
+```
+
+**For detailed algorithm comparison, parameters, and selection guidance**: `references/algorithms.md`
+
+### 3. Distributed Computing
+
+Scale inference from local multi-core to cluster environments:
+
+**Local (default)** - Uses all available cores automatically:
+```python
+network = grnboost2(expression_data=matrix)
+```
+
+**Custom local client** - Control resources:
+```python
+from distributed import LocalCluster, Client
+
+local_cluster = LocalCluster(n_workers=10, memory_limit='8GB')
+client = Client(local_cluster)
+
+network = grnboost2(expression_data=matrix, client_or_address=client)
+
+client.close()
+local_cluster.close()
+```
+
+**Cluster computing** - Connect to remote Dask scheduler:
+```python
+from distributed import Client
+
+client = Client('tcp://scheduler:8786')
+network = grnboost2(expression_data=matrix, client_or_address=client)
+```
+
+**For cluster setup, performance optimization, and large-scale workflows**: `references/distributed_computing.md`
+
+## Installation
+
+```bash
+uv pip install arboreto
+```
+
+Conda (Bioconda):
+
+```bash
+conda install -c bioconda arboreto
+```
+
+**Dependencies** (from upstream `requirements.txt`): `dask[complete]`, `distributed`, `numpy`, `pandas`, `scikit-learn`, `scipy`
+
+**Input formats**: pandas DataFrame, dense `numpy.ndarray`, or sparse `scipy.sparse.csc_matrix` (rows = observations, columns = genes). For array/matrix inputs, pass `gene_names` explicitly.
+
+## Common Use Cases
+
+### Single-Cell RNA-seq Analysis
+```python
+import pandas as pd
+from arboreto.algo import grnboost2
+
+if __name__ == '__main__':
+    # Load single-cell expression matrix (cells x genes)
+    sc_data = pd.read_csv('scrna_counts.tsv', sep='\t')
+
+    # Infer cell-type-specific regulatory network
+    network = grnboost2(expression_data=sc_data, seed=42)
+
+    # Filter high-confidence links
+    high_confidence = network[network['importance'] > 0.5]
+    high_confidence.to_csv('grn_high_confidence.tsv', sep='\t', index=False)
+```
+
+### Bulk RNA-seq with TF Filtering
+```python
+from arboreto.utils import load_tf_names
+from arboreto.algo import grnboost2
+
+if __name__ == '__main__':
+    # Load data
+    expression_data = pd.read_csv('rnaseq_tpm.tsv', sep='\t')
+    tf_names = load_tf_names('human_tfs.txt')
+
+    # Infer with TF restriction
+    network = grnboost2(
+        expression_data=expression_data,
+        tf_names=tf_names,
+        seed=123
+    )
+
+    network.to_csv('tf_target_network.tsv', sep='\t', index=False)
+```
+
+### Comparative Analysis (Multiple Conditions)
+```python
+from arboreto.algo import grnboost2
+
+if __name__ == '__main__':
+    # Infer networks for different conditions
+    conditions = ['control', 'treatment_24h', 'treatment_48h']
+
+    for condition in conditions:
+        data = pd.read_csv(f'{condition}_expression.tsv', sep='\t')
+        network = grnboost2(expression_data=data, seed=42)
+        network.to_csv(f'{condition}_network.tsv', sep='\t', index=False)
+```
+
+## Output Interpretation
+
+Arboreto returns a DataFrame with regulatory links:
+
+| Column | Description |
+|--------|-------------|
+| `TF` | Transcription factor (regulator) |
+| `target` | Target gene |
+| `importance` | Regulatory importance score (higher = stronger) |
+
+**Filtering strategy**:
+- `limit=N` at inference time (return top N links globally)
+- Post-hoc importance threshold (e.g., > 0.5)
+- Top links per target via `groupby('target')`
+- Statistical significance testing (permutation tests, external tools)
+
+## Integration with pySCENIC
+
+Arboreto powers the GRN inference step in [pySCENIC](https://github.com/aertslab/pySCENIC). pySCENIC 0.11+ passes sparse expression matrices to `grnboost2` / `genie3`; pySCENIC 0.12+ defaults to `arboreto_with_multiprocessing.py` (no Dask) for compatibility — use standalone arboreto when you need Dask scaling.
+
+```python
+# Standalone: infer co-expression modules before pySCENIC cisTarget pruning
+from arboreto.algo import grnboost2
+
+network = grnboost2(expression_data=expression_df, tf_names=tf_list, limit=5000)
+
+# Downstream: pySCENIC ctx pruning, regulon definition, AUCell (see pySCENIC docs)
+```
+
+Convert AnnData to a DataFrame for arboreto directly:
+
+```python
+expression_df = adata.to_df()  # cells x genes
+```
+
+## Reproducibility
+
+Always set a seed for reproducible results:
+```python
+network = grnboost2(expression_data=matrix, seed=777)
+```
+
+Run multiple seeds for robustness analysis:
+```python
+from distributed import LocalCluster, Client
+
+if __name__ == '__main__':
+    client = Client(LocalCluster())
+
+    seeds = [42, 123, 777]
+    networks = []
+
+    for seed in seeds:
+        net = grnboost2(expression_data=matrix, client_or_address=client, seed=seed)
+        networks.append(net)
+
+    # Consensus: links recurring across runs (example: mean importance per TF-target pair)
+    import pandas as pd
+    combined = pd.concat(networks)
+    consensus = (
+        combined.groupby(['TF', 'target'], as_index=False)['importance']
+        .mean()
+        .query('importance > 0.5')
+    )
+```
+
+## Troubleshooting
+
+**Memory errors**: Reduce dataset size by filtering low-variance genes or use distributed computing
+
+**Slow performance**: Use GRNBoost2 instead of GENIE3, enable distributed client, filter TF list
+
+**Dask errors**: Ensure `if __name__ == '__main__':` guard is present in scripts (required on Windows/macOS with spawn-based multiprocessing)
+
+**Empty results**: Check data format (genes as columns), verify TF names match column names in the expression matrix
+
+**Sparse data**: Use `scipy.sparse.csc_matrix` and pass matching `gene_names`; supported since arboreto 0.1.6 / pySCENIC 0.11
+

package/skills/arboreto/references/algorithms.md

@@ -0,0 +1,152 @@
+# GRN Inference Algorithms
+
+Arboreto provides two high-level algorithms for gene regulatory network (GRN) inference, both based on the multiple regression approach.
+
+## Algorithm Overview
+
+Both algorithms follow the same inference strategy:
+1. For each target gene in the dataset, train a regression model
+2. Identify the most important features (potential regulators) from the model
+3. Emit these features as candidate regulators with importance scores
+
+The key difference is **computational efficiency** and the underlying regression method.
+
+## GRNBoost2 (Recommended)
+
+**Purpose**: Fast GRN inference for large-scale datasets using gradient boosting.
+
+### When to Use
+- **Large datasets**: Tens of thousands of observations (e.g., single-cell RNA-seq)
+- **Time-constrained analysis**: Need faster results than GENIE3
+- **Default choice**: GRNBoost2 is the flagship algorithm and recommended for most use cases
+
+### Technical Details
+- **Method**: Stochastic gradient boosting with early-stopping regularization
+- **Performance**: Significantly faster than GENIE3 on large datasets
+- **Output**: Same format as GENIE3 (TF-target-importance triplets)
+
+### Usage
+```python
+from arboreto.algo import grnboost2
+
+network = grnboost2(
+    expression_data=expression_matrix,
+    tf_names=tf_names,
+    seed=42,
+    limit=5000,
+)
+```
+
+### Parameters (`grnboost2`)
+```python
+grnboost2(
+    expression_data,              # DataFrame, ndarray, or scipy.sparse.csc_matrix
+    gene_names=None,              # Required for ndarray/sparse inputs
+    tf_names='all',                 # TF list, None/'all' → all genes as regulators
+    client_or_address='local',      # 'local', scheduler address, or Dask Client
+    early_stop_window_length=25,    # Early-stopping window (GRNBoost2 only)
+    limit=None,                     # Return top N links globally
+    seed=None,                      # Random seed; None = non-deterministic
+    verbose=False,
+)
+```
+
+## GENIE3
+
+**Purpose**: Classic Random Forest-based GRN inference, serving as the conceptual blueprint.
+
+### When to Use
+- **Smaller datasets**: When dataset size allows for longer computation
+- **Comparison studies**: When comparing with published GENIE3 results
+- **Validation**: To validate GRNBoost2 results
+
+### Technical Details
+- **Method**: Random Forest regression (ExtraTrees available via `diy`)
+- **Foundation**: Original multiple regression GRN inference strategy
+- **Trade-off**: More computationally expensive but well-established
+
+### Usage
+```python
+from arboreto.algo import genie3
+
+network = genie3(
+    expression_data=expression_matrix,
+    tf_names=tf_names,
+    seed=42,
+)
+```
+
+### Parameters (`genie3`)
+```python
+genie3(
+    expression_data,
+    gene_names=None,
+    tf_names='all',
+    client_or_address='local',
+    limit=None,
+    seed=None,
+    verbose=False,
+)
+```
+
+## Algorithm Comparison
+
+| Feature | GRNBoost2 | GENIE3 |
+|---------|-----------|--------|
+| **Speed** | Fast (optimized for large data) | Slower |
+| **Method** | Gradient boosting (GBM) | Random Forest |
+| **Best for** | Large-scale data (10k+ observations) | Small-medium datasets |
+| **Output format** | Same | Same |
+| **Inference strategy** | Multiple regression | Multiple regression |
+| **Recommended** | Yes (default choice) | For comparison/validation |
+| **Early stopping** | Yes (`early_stop_window_length`) | No |
+
+## Advanced: Custom Regressors with `diy`
+
+For custom scikit-learn regressor settings, use `diy()` (not `grnboost2`/`genie3` kwargs):
+
+```python
+from arboreto.algo import diy
+from arboreto.core import SGBM_KWARGS, RF_KWARGS
+
+# Custom GRNBoost2-style run
+custom_gbm = diy(
+    expression_data=expression_matrix,
+    regressor_type='GBM',  # 'RF', 'GBM', or 'ET'
+    regressor_kwargs={
+        **SGBM_KWARGS,
+        'n_estimators': 100,
+        'max_depth': 5,
+        'learning_rate': 0.1,
+    },
+    tf_names=tf_names,
+    seed=42,
+)
+
+# Custom GENIE3-style run
+custom_rf = diy(
+    expression_data=expression_matrix,
+    regressor_type='RF',
+    regressor_kwargs={
+        **RF_KWARGS,
+        'n_estimators': 1000,
+        'max_features': 'sqrt',
+    },
+    tf_names=tf_names,
+)
+```
+
+Import default kwargs from `arboreto.core` and override only the keys you need.
+
+## Choosing the Right Algorithm
+
+**Decision guide**:
+
+1. **Start with GRNBoost2** — faster and better suited to large single-cell datasets
+2. **Use GENIE3 if**:
+   - Comparing with existing GENIE3 publications
+   - Dataset is small-medium sized
+   - Validating GRNBoost2 results
+3. **Use `diy()` if** you need non-default regressor hyperparameters
+
+Both algorithms produce comparable regulatory networks with the same output format.

package/skills/arboreto/references/basic_inference.md

@@ -0,0 +1,181 @@
+# Basic GRN Inference with Arboreto
+
+## Input Data Requirements
+
+Arboreto requires gene expression data in one of two formats:
+
+### Pandas DataFrame (Recommended)
+- **Rows**: Observations (cells, samples, conditions)
+- **Columns**: Genes (with gene names as column headers)
+- **Format**: Numeric expression values
+
+Example:
+```python
+import pandas as pd
+
+# Load expression matrix with genes as columns
+expression_matrix = pd.read_csv('expression_data.tsv', sep='\t')
+# Columns: ['gene1', 'gene2', 'gene3', ...]
+# Rows: observation data
+```
+
+### NumPy Array
+- **Shape**: (observations, genes)
+- **Requirement**: Separately provide gene names list matching column order
+
+Example:
+```python
+import numpy as np
+
+expression_matrix = np.genfromtxt('expression_data.tsv', delimiter='\t', skip_header=1)
+with open('expression_data.tsv') as f:
+    gene_names = [gene.strip() for gene in f.readline().split('\t')]
+
+assert expression_matrix.shape[1] == len(gene_names)
+```
+
+### Sparse CSC Matrix (arboreto 0.1.6+)
+- **Format**: `scipy.sparse.csc_matrix` with shape (observations, genes)
+- **Requirement**: Provide `gene_names` matching column order (same as NumPy)
+- **Use case**: Large single-cell matrices; also used by pySCENIC 0.11+ when `--sparse` is enabled
+
+Example:
+```python
+import scipy.sparse as sp
+from arboreto.algo import grnboost2
+
+# expression_sparse: csc_matrix, cells x genes
+network = grnboost2(
+    expression_data=expression_sparse,
+    gene_names=gene_names,
+    tf_names=tf_names,
+)
+```
+
+## Transcription Factors (TFs)
+
+Optionally provide a list of transcription factor names to restrict regulatory inference:
+
+```python
+from arboreto.utils import load_tf_names
+
+# Load from file (one TF per line)
+tf_names = load_tf_names('transcription_factors.txt')
+
+# Or define directly
+tf_names = ['TF1', 'TF2', 'TF3']
+```
+
+If `tf_names` is `None` or `'all'`, all `gene_names` are treated as potential regulators.
+
+## Basic Inference Workflow
+
+### Using Pandas DataFrame
+
+```python
+import pandas as pd
+from arboreto.utils import load_tf_names
+from arboreto.algo import grnboost2
+
+if __name__ == '__main__':
+    # Load expression data
+    expression_matrix = pd.read_csv('expression_data.tsv', sep='\t')
+
+    # Load transcription factors (optional)
+    tf_names = load_tf_names('tf_list.txt')
+
+    # Run GRN inference
+    network = grnboost2(
+        expression_data=expression_matrix,
+        tf_names=tf_names  # Optional
+    )
+
+    # Save results
+    network.to_csv('network_output.tsv', sep='\t', index=False, header=False)
+```
+
+**Critical**: The `if __name__ == '__main__':` guard is required because Dask spawns new processes internally.
+
+### Using NumPy Array
+
+```python
+import numpy as np
+from arboreto.algo import grnboost2
+
+if __name__ == '__main__':
+    # Load expression matrix
+    expression_matrix = np.genfromtxt('expression_data.tsv', delimiter='\t', skip_header=1)
+
+    # Extract gene names from header
+    with open('expression_data.tsv') as f:
+        gene_names = [gene.strip() for gene in f.readline().split('\t')]
+
+    # Verify dimensions match
+    assert expression_matrix.shape[1] == len(gene_names)
+
+    # Run inference with explicit gene names
+    network = grnboost2(
+        expression_data=expression_matrix,
+        gene_names=gene_names,
+        tf_names=tf_names
+    )
+
+    network.to_csv('network_output.tsv', sep='\t', index=False, header=False)
+```
+
+## Output Format
+
+Arboreto returns a Pandas DataFrame with three columns:
+
+| Column | Description |
+|--------|-------------|
+| `TF` | Transcription factor (regulator) gene name |
+| `target` | Target gene name |
+| `importance` | Regulatory importance score (higher = stronger regulation) |
+
+Example output:
+```
+TF1    gene5    0.856
+TF2    gene12   0.743
+TF1    gene8    0.621
+```
+
+## Setting Random Seed
+
+For reproducible results, pass an explicit `seed` (`None` uses random seeds per regressor):
+
+```python
+network = grnboost2(
+    expression_data=expression_matrix,
+    tf_names=tf_names,
+    seed=777
+)
+```
+
+## Limiting Output Size
+
+Return only the top N regulatory links globally:
+
+```python
+network = grnboost2(
+    expression_data=expression_matrix,
+    tf_names=tf_names,
+    limit=5000,
+)
+```
+
+## Algorithm Selection
+
+Use `grnboost2()` for most cases (faster, handles large datasets):
+```python
+from arboreto.algo import grnboost2
+network = grnboost2(expression_data=expression_matrix)
+```
+
+Use `genie3()` for comparison or specific requirements:
+```python
+from arboreto.algo import genie3
+network = genie3(expression_data=expression_matrix)
+```
+
+See `references/algorithms.md` for detailed algorithm comparison.