@pikaa-ai/pikaa 0.3.22 → 0.3.24

package/skills/arboreto/SKILL.md

@@ -1,267 +0,0 @@
----
-name: arboreto
-description: Infer gene regulatory networks (GRNs) from gene expression data using scalable algorithms (GRNBoost2, GENIE3). Use when analyzing transcriptomics data (bulk RNA-seq, single-cell RNA-seq) to identify transcription factor-target gene relationships and regulatory interactions. Supports distributed computation for large-scale datasets.
-license: BSD-3-Clause license
-metadata:
-  version: "1.0"
-  skill-author: K-Dense Inc.
----
-
-# Arboreto
-
-## Overview
-
-Arboreto is a Python library from [Aerts Lab](https://github.com/aertslab/arboreto) for inferring gene regulatory networks (GRNs) from gene expression data. It parallelizes tree-based ensemble regression (GRNBoost2, GENIE3) with [Dask](https://distributed.dask.org/) across local cores or remote clusters.
-
-**Core capability**: Identify which transcription factors (TFs) regulate which target genes based on expression patterns across observations (cells, samples, conditions).
-
-**Upstream**: PyPI **0.1.6** (2021-02-09, latest). Docs: [arboreto.readthedocs.io](https://arboreto.readthedocs.io/en/latest/). Primary downstream consumer: [pySCENIC](https://github.com/aertslab/pySCENIC).
-
-## Quick Start
-
-Install arboreto:
-```bash
-uv pip install arboreto
-```
-
-Basic GRN inference:
-```python
-import pandas as pd
-from arboreto.algo import grnboost2
-
-if __name__ == '__main__':
-    # Load expression data (genes as columns)
-    expression_matrix = pd.read_csv('expression_data.tsv', sep='\t')
-
-    # Infer regulatory network
-    network = grnboost2(expression_data=expression_matrix)
-
-    # Save results (TF, target, importance)
-    network.to_csv('network.tsv', sep='\t', index=False, header=False)
-```
-
-**Critical**: Always use `if __name__ == '__main__':` guard because Dask spawns new processes.
-
-## Core Capabilities
-
-### 1. Basic GRN Inference
-
-For standard GRN inference workflows including:
-- Input data preparation (Pandas DataFrame or NumPy array)
-- Running inference with GRNBoost2 or GENIE3
-- Filtering by transcription factors
-- Output format and interpretation
-
-**See**: `references/basic_inference.md`
-
-**Use the ready-to-run script**: `scripts/basic_grn_inference.py` for standard inference tasks:
-```bash
-python scripts/basic_grn_inference.py expression_data.tsv output_network.tsv --tf-file tfs.txt --seed 777 --limit 5000
-```
-
-### 2. Algorithm Selection
-
-Arboreto provides two algorithms:
-
-**GRNBoost2 (Recommended)**:
-- Fast gradient boosting-based inference
-- Optimized for large datasets (10k+ observations)
-- Default choice for most analyses
-
-**GENIE3**:
-- Random Forest-based inference
-- Original multiple regression approach
-- Use for comparison or validation
-
-Quick comparison:
-```python
-from arboreto.algo import grnboost2, genie3
-
-# Fast, recommended
-network_grnboost = grnboost2(expression_data=matrix)
-
-# Classic algorithm
-network_genie3 = genie3(expression_data=matrix)
-```
-
-**For detailed algorithm comparison, parameters, and selection guidance**: `references/algorithms.md`
-
-### 3. Distributed Computing
-
-Scale inference from local multi-core to cluster environments:
-
-**Local (default)** - Uses all available cores automatically:
-```python
-network = grnboost2(expression_data=matrix)
-```
-
-**Custom local client** - Control resources:
-```python
-from distributed import LocalCluster, Client
-
-local_cluster = LocalCluster(n_workers=10, memory_limit='8GB')
-client = Client(local_cluster)
-
-network = grnboost2(expression_data=matrix, client_or_address=client)
-
-client.close()
-local_cluster.close()
-```
-
-**Cluster computing** - Connect to remote Dask scheduler:
-```python
-from distributed import Client
-
-client = Client('tcp://scheduler:8786')
-network = grnboost2(expression_data=matrix, client_or_address=client)
-```
-
-**For cluster setup, performance optimization, and large-scale workflows**: `references/distributed_computing.md`
-
-## Installation
-
-```bash
-uv pip install arboreto
-```
-
-Conda (Bioconda):
-
-```bash
-conda install -c bioconda arboreto
-```
-
-**Dependencies** (from upstream `requirements.txt`): `dask[complete]`, `distributed`, `numpy`, `pandas`, `scikit-learn`, `scipy`
-
-**Input formats**: pandas DataFrame, dense `numpy.ndarray`, or sparse `scipy.sparse.csc_matrix` (rows = observations, columns = genes). For array/matrix inputs, pass `gene_names` explicitly.
-
-## Common Use Cases
-
-### Single-Cell RNA-seq Analysis
-```python
-import pandas as pd
-from arboreto.algo import grnboost2
-
-if __name__ == '__main__':
-    # Load single-cell expression matrix (cells x genes)
-    sc_data = pd.read_csv('scrna_counts.tsv', sep='\t')
-
-    # Infer cell-type-specific regulatory network
-    network = grnboost2(expression_data=sc_data, seed=42)
-
-    # Filter high-confidence links
-    high_confidence = network[network['importance'] > 0.5]
-    high_confidence.to_csv('grn_high_confidence.tsv', sep='\t', index=False)
-```
-
-### Bulk RNA-seq with TF Filtering
-```python
-from arboreto.utils import load_tf_names
-from arboreto.algo import grnboost2
-
-if __name__ == '__main__':
-    # Load data
-    expression_data = pd.read_csv('rnaseq_tpm.tsv', sep='\t')
-    tf_names = load_tf_names('human_tfs.txt')
-
-    # Infer with TF restriction
-    network = grnboost2(
-        expression_data=expression_data,
-        tf_names=tf_names,
-        seed=123
-    )
-
-    network.to_csv('tf_target_network.tsv', sep='\t', index=False)
-```
-
-### Comparative Analysis (Multiple Conditions)
-```python
-from arboreto.algo import grnboost2
-
-if __name__ == '__main__':
-    # Infer networks for different conditions
-    conditions = ['control', 'treatment_24h', 'treatment_48h']
-
-    for condition in conditions:
-        data = pd.read_csv(f'{condition}_expression.tsv', sep='\t')
-        network = grnboost2(expression_data=data, seed=42)
-        network.to_csv(f'{condition}_network.tsv', sep='\t', index=False)
-```
-
-## Output Interpretation
-
-Arboreto returns a DataFrame with regulatory links:
-
-| Column | Description |
-|--------|-------------|
-| `TF` | Transcription factor (regulator) |
-| `target` | Target gene |
-| `importance` | Regulatory importance score (higher = stronger) |
-
-**Filtering strategy**:
-- `limit=N` at inference time (return top N links globally)
-- Post-hoc importance threshold (e.g., > 0.5)
-- Top links per target via `groupby('target')`
-- Statistical significance testing (permutation tests, external tools)
-
-## Integration with pySCENIC
-
-Arboreto powers the GRN inference step in [pySCENIC](https://github.com/aertslab/pySCENIC). pySCENIC 0.11+ passes sparse expression matrices to `grnboost2` / `genie3`; pySCENIC 0.12+ defaults to `arboreto_with_multiprocessing.py` (no Dask) for compatibility — use standalone arboreto when you need Dask scaling.
-
-```python
-# Standalone: infer co-expression modules before pySCENIC cisTarget pruning
-from arboreto.algo import grnboost2
-
-network = grnboost2(expression_data=expression_df, tf_names=tf_list, limit=5000)
-
-# Downstream: pySCENIC ctx pruning, regulon definition, AUCell (see pySCENIC docs)
-```
-
-Convert AnnData to a DataFrame for arboreto directly:
-
-```python
-expression_df = adata.to_df()  # cells x genes
-```
-
-## Reproducibility
-
-Always set a seed for reproducible results:
-```python
-network = grnboost2(expression_data=matrix, seed=777)
-```
-
-Run multiple seeds for robustness analysis:
-```python
-from distributed import LocalCluster, Client
-
-if __name__ == '__main__':
-    client = Client(LocalCluster())
-
-    seeds = [42, 123, 777]
-    networks = []
-
-    for seed in seeds:
-        net = grnboost2(expression_data=matrix, client_or_address=client, seed=seed)
-        networks.append(net)
-
-    # Consensus: links recurring across runs (example: mean importance per TF-target pair)
-    import pandas as pd
-    combined = pd.concat(networks)
-    consensus = (
-        combined.groupby(['TF', 'target'], as_index=False)['importance']
-        .mean()
-        .query('importance > 0.5')
-    )
-```
-
-## Troubleshooting
-
-**Memory errors**: Reduce dataset size by filtering low-variance genes or use distributed computing
-
-**Slow performance**: Use GRNBoost2 instead of GENIE3, enable distributed client, filter TF list
-
-**Dask errors**: Ensure `if __name__ == '__main__':` guard is present in scripts (required on Windows/macOS with spawn-based multiprocessing)
-
-**Empty results**: Check data format (genes as columns), verify TF names match column names in the expression matrix
-
-**Sparse data**: Use `scipy.sparse.csc_matrix` and pass matching `gene_names`; supported since arboreto 0.1.6 / pySCENIC 0.11
-

package/skills/astropy/SKILL.md

@@ -1,353 +0,0 @@
----
-name: astropy
-description: Core Python library for astronomy and astrophysics workflows that need Astropy APIs, including units/quantities, coordinates, FITS I/O, tables, time systems, WCS, and cosmology. Use when implementing or debugging astronomical data analysis code with Astropy.
-license: BSD-3-Clause license
-compatibility: Requires Python 3.11+ with astropy installed (uv for package installation). Some features (object name resolution, site lookups, remote FITS reads, IERS updates) need network access.
-metadata:
-  version: "1.2"
-  skill-author: K-Dense Inc.
----
-
-# Astropy
-
-## Overview
-
-Astropy is the core Python package for astronomy, providing essential functionality for astronomical research and data analysis. Use astropy for coordinate transformations, unit and quantity calculations, FITS file operations, cosmological calculations, precise time handling, tabular data manipulation, and astronomical image processing.
-
-## When to Use This Skill
-
-Use astropy when tasks involve:
-- Converting between celestial coordinate systems (ICRS, Galactic, FK5, AltAz, etc.)
-- Working with physical units and quantities (converting Jy to mJy, parsecs to km, etc.)
-- Reading, writing, or manipulating FITS files (images or tables)
-- Cosmological calculations (luminosity distance, lookback time, Hubble parameter)
-- Precise time handling with different time scales (UTC, TAI, TT, TDB) and formats (JD, MJD, ISO)
-- Table operations (reading catalogs, cross-matching, filtering, joining)
-- WCS transformations between pixel and world coordinates
-- Astronomical constants and calculations
-
-## Quick Start
-
-```python
-import astropy.units as u
-from astropy.coordinates import SkyCoord
-from astropy.time import Time
-from astropy.io import fits
-from astropy.table import Table
-from astropy.cosmology import Planck18
-
-# Units and quantities
-distance = 100 * u.pc
-distance_km = distance.to(u.km)
-
-# Coordinates
-coord = SkyCoord(ra=10.5*u.degree, dec=41.2*u.degree, frame='icrs')
-coord_galactic = coord.galactic
-
-# Time
-t = Time('2023-01-15 12:30:00')
-jd = t.jd  # Julian Date
-
-# FITS files
-data = fits.getdata('image.fits')
-header = fits.getheader('image.fits')
-
-# Tables
-table = Table.read('catalog.fits')
-
-# Cosmology
-d_L = Planck18.luminosity_distance(z=1.0)
-```
-
-## Core Capabilities
-
-### 1. Units and Quantities (`astropy.units`)
-
-Handle physical quantities with units, perform unit conversions, and ensure dimensional consistency in calculations.
-
-**Key operations:**
-- Create quantities by multiplying values with units
-- Convert between units using `.to()` method
-- Perform arithmetic with automatic unit handling
-- Use equivalencies for domain-specific conversions (spectral, doppler, parallax)
-- Work with logarithmic units (magnitudes, decibels)
-
-**See:** `references/units.md` for comprehensive documentation, unit systems, equivalencies, performance optimization, and unit arithmetic.
-
-### 2. Coordinate Systems (`astropy.coordinates`)
-
-Represent celestial positions and transform between different coordinate frames.
-
-**Key operations:**
-- Create coordinates with `SkyCoord` in any frame (ICRS, Galactic, FK5, AltAz, etc.)
-- Transform between coordinate systems
-- Calculate angular separations and position angles
-- Match coordinates to catalogs
-- Include distance for 3D coordinate operations
-- Handle proper motions and radial velocities
-- Query named objects from online databases
-
-**See:** `references/coordinates.md` for detailed coordinate frame descriptions, transformations, observer-dependent frames (AltAz), catalog matching, and performance tips.
-
-### 3. Cosmological Calculations (`astropy.cosmology`)
-
-Perform cosmological calculations using standard cosmological models.
-
-**Key operations:**
-- Use built-in cosmologies (Planck18, WMAP9, etc.)
-- Create custom cosmological models
-- Calculate distances (luminosity, comoving, angular diameter)
-- Compute ages and lookback times
-- Determine Hubble parameter at any redshift
-- Calculate density parameters and volumes
-- Perform inverse calculations (find z for given distance)
-
-**See:** `references/cosmology.md` for available models, distance calculations, time calculations, density parameters, and neutrino effects.
-
-### 4. FITS File Handling (`astropy.io.fits`)
-
-Read, write, and manipulate FITS (Flexible Image Transport System) files.
-
-**Key operations:**
-- Open FITS files with context managers
-- Access HDUs (Header Data Units) by index or name
-- Read and modify headers (keywords, comments, history)
-- Work with image data (NumPy arrays)
-- Handle table data (binary and ASCII tables)
-- Create new FITS files (single or multi-extension)
-- Use memory mapping for large files
-- Access remote FITS files (S3, HTTP)
-
-**See:** `references/fits.md` for comprehensive file operations, header manipulation, image and table handling, multi-extension files, and performance considerations.
-
-### 5. Table Operations (`astropy.table`)
-
-Work with tabular data with support for units, metadata, and various file formats.
-
-**Key operations:**
-- Create tables from arrays, lists, or dictionaries
-- Read/write tables in multiple formats (FITS, CSV, HDF5, VOTable)
-- Access and modify columns and rows
-- Sort, filter, and index tables
-- Perform database-style operations (join, group, aggregate)
-- Stack and concatenate tables
-- Work with unit-aware columns (QTable)
-- Handle missing data with masking
-
-**See:** `references/tables.md` for table creation, I/O operations, data manipulation, sorting, filtering, joins, grouping, and performance tips.
-
-### 6. Time Handling (`astropy.time`)
-
-Precise time representation and conversion between time scales and formats.
-
-**Key operations:**
-- Create Time objects in various formats (ISO, JD, MJD, Unix, etc.)
-- Convert between time scales (UTC, TAI, TT, TDB, etc.)
-- Perform time arithmetic with TimeDelta
-- Calculate sidereal time for observers
-- Compute light travel time corrections (barycentric, heliocentric)
-- Work with time arrays efficiently
-- Handle masked (missing) times
-
-**See:** `references/time.md` for time formats, time scales, conversions, arithmetic, observing features, and precision handling.
-
-### 7. World Coordinate System (`astropy.wcs`)
-
-Transform between pixel coordinates in images and world coordinates.
-
-**Key operations:**
-- Read WCS from FITS headers
-- Convert pixel coordinates to world coordinates (and vice versa)
-- Calculate image footprints
-- Access WCS parameters (reference pixel, projection, scale)
-- Create custom WCS objects
-
-**See:** `references/wcs_and_other_modules.md` for WCS operations and transformations.
-
-## Additional Capabilities
-
-The `references/wcs_and_other_modules.md` file also covers:
-
-### NDData and CCDData
-Containers for n-dimensional datasets with metadata, uncertainty, masking, and WCS information.
-
-### Modeling
-Framework for creating and fitting mathematical models to astronomical data.
-
-### Visualization
-Tools for astronomical image display with appropriate stretching and scaling.
-
-### Constants
-Physical and astronomical constants with proper units (speed of light, solar mass, Planck constant, etc.).
-
-### Convolution
-Image processing kernels for smoothing and filtering.
-
-### Statistics
-Robust statistical functions including sigma clipping and outlier rejection.
-
-## Installation
-
-```bash
-# Reproducible install against the current stable release
-uv pip install "astropy==7.2.0"
-
-# Recommended optional dependencies for plotting and common workflows
-uv pip install "astropy[recommended]==7.2.0"
-
-# Full optional dependency set for broad astronomy workflows
-uv pip install "astropy[all]==7.2.0"
-```
-
-Astropy 7.2.0 requires Python 3.11+ and depends on NumPy, PyERFA, PyYAML, and packaging. Use an isolated virtual environment; do not install Astropy with elevated privileges.
-
-Note that the `[recommended]` and `[all]` extras pull in transitive dependencies (matplotlib, scipy, etc.) at unpinned versions. For reproducible production environments, pin the full dependency tree with a lockfile (`uv lock` in a project, or `uv pip compile` for requirements files) and review the resolved versions before deploying.
-
-## Common Workflows
-
-### Converting Coordinates Between Systems
-
-```python
-from astropy.coordinates import SkyCoord
-import astropy.units as u
-
-# Create coordinate
-c = SkyCoord(ra='05h23m34.5s', dec='-69d45m22s', frame='icrs')
-
-# Transform to galactic
-c_gal = c.galactic
-print(f"l={c_gal.l.deg}, b={c_gal.b.deg}")
-
-# Transform to alt-az (requires time and location)
-from astropy.time import Time
-from astropy.coordinates import EarthLocation, AltAz
-
-observing_time = Time('2023-06-15 23:00:00')
-observing_location = EarthLocation(lat=40*u.deg, lon=-120*u.deg)
-aa_frame = AltAz(obstime=observing_time, location=observing_location)
-c_altaz = c.transform_to(aa_frame)
-print(f"Alt={c_altaz.alt.deg}, Az={c_altaz.az.deg}")
-```
-
-### Reading and Analyzing FITS Files
-
-```python
-from astropy.io import fits
-import numpy as np
-
-# Open FITS file
-with fits.open('observation.fits') as hdul:
-    # Display structure
-    hdul.info()
-
-    # Get image data and header
-    data = hdul[1].data
-    header = hdul[1].header
-
-    # Access header values
-    exptime = header['EXPTIME']
-    filter_name = header['FILTER']
-
-    # Analyze data
-    mean = np.mean(data)
-    median = np.median(data)
-    print(f"Mean: {mean}, Median: {median}")
-```
-
-### Cosmological Distance Calculations
-
-```python
-from astropy.cosmology import Planck18
-import astropy.units as u
-import numpy as np
-
-# Calculate distances at z=1.5
-z = 1.5
-d_L = Planck18.luminosity_distance(z)
-d_A = Planck18.angular_diameter_distance(z)
-
-print(f"Luminosity distance: {d_L}")
-print(f"Angular diameter distance: {d_A}")
-
-# Age of universe at that redshift
-age = Planck18.age(z)
-print(f"Age at z={z}: {age.to(u.Gyr)}")
-
-# Lookback time
-t_lookback = Planck18.lookback_time(z)
-print(f"Lookback time: {t_lookback.to(u.Gyr)}")
-```
-
-### Cross-Matching Catalogs
-
-```python
-from astropy.table import Table
-from astropy.coordinates import SkyCoord, match_coordinates_sky
-import astropy.units as u
-
-# Read catalogs
-cat1 = Table.read('catalog1.fits')
-cat2 = Table.read('catalog2.fits')
-
-# Create coordinate objects
-coords1 = SkyCoord(ra=cat1['RA']*u.degree, dec=cat1['DEC']*u.degree)
-coords2 = SkyCoord(ra=cat2['RA']*u.degree, dec=cat2['DEC']*u.degree)
-
-# Find matches
-idx, sep, _ = coords1.match_to_catalog_sky(coords2)
-
-# Filter by separation threshold
-max_sep = 1 * u.arcsec
-matches = sep < max_sep
-
-# Create matched catalogs
-cat1_matched = cat1[matches]
-cat2_matched = cat2[idx[matches]]
-print(f"Found {len(cat1_matched)} matches")
-```
-
-## Best Practices
-
-1. **Always use units**: Attach units to quantities to avoid errors and ensure dimensional consistency
-2. **Use context managers for FITS files**: Ensures proper file closing
-3. **Prefer arrays over loops**: Process multiple coordinates/times as arrays for better performance
-4. **Check coordinate frames**: Verify the frame before transformations
-5. **Use appropriate cosmology**: Choose the right cosmological model for your analysis
-6. **Handle missing data**: Use masked columns for tables with missing values
-7. **Specify time scales**: Be explicit about time scales (UTC, TT, TDB) for precise timing
-8. **Use QTable for unit-aware tables**: When table columns have units
-9. **Check WCS validity**: Verify WCS before using transformations
-10. **Cache frequently used values**: Expensive calculations (e.g., cosmological distances) can be cached
-11. **Be explicit about network access**: `SkyCoord.from_name()`, `EarthLocation.of_site(refresh_cache=True)`, `EarthLocation.of_address()`, `download_file()`, remote FITS reads, and some IERS time/coordinate transforms can contact external services or update local caches. Avoid sending sensitive target names, addresses, URLs, or proprietary file locations to third-party services. When working with potentially sensitive targets or data locations, confirm with the user before making these network calls.
-12. **Pin for reproducibility**: Use pinned versions such as `astropy==7.2.0` for shared environments; update pins intentionally after reviewing release notes.
-
-## Current-Version Notes
-
-- Current stable release researched: Astropy 7.2.0 (released 2025-11-25; verified current as of 2026-06-10)
-- Python requirement: 3.11+
-- **Astropy 8.0 is at release-candidate stage** (8.0.0rc1, 2026-05-26). Key changes to anticipate:
-  - The deprecated `astropy.cosmology` submodule shims (`astropy.cosmology.flrw`, `.core`, `.funcs`, `.connect`, `.parameter`) are removed — import everything directly from `astropy.cosmology` (e.g., `from astropy.cosmology import FlatLambdaCDM, z_at_value`)
-  - `astropy.constants` defaults change from CODATA 2018 to CODATA 2022; pin a constants version via the `astropyconst` science states if reproducibility matters
-  - NumPy 2.0 becomes the minimum supported version; the 7.2.x LTS branch retains NumPy 1.x support for six months after the 8.0 release
-  - The built-in test runner (`astropy.test()`, `TestRunner`) is formally deprecated — invoke `pytest` directly
-- Recent 7.x deprecations to avoid in new code: passing a table index identifier as the first `.loc` element (`t.loc["b", 2]`) — use `t.loc.with_index("b")[2]` instead (removal planned for 9.0); `astropy.utils.isiterable()` — use `numpy.iterable()`
-- Recent 7.0 removals: older deprecated FITS APIs such as `(Bin)Table.update`, `_ExtensionHDU`, `_NonstandardExtHDU`, and the `tile_size` argument for `CompImageHDU`; `CompImageHeader` is deprecated. Avoid those legacy patterns in new examples.
-- The recommended optional extras are `recommended` for common plotting/scientific dependencies and `all` only when a broad optional feature set is needed.
-
-## Documentation and Resources
-
-- Official Astropy Documentation: https://docs.astropy.org/en/stable/
-- Tutorials: https://learn.astropy.org/
-- GitHub: https://github.com/astropy/astropy
-
-## Reference Files
-
-For detailed information on specific modules:
-- `references/units.md` - Units, quantities, conversions, and equivalencies
-- `references/coordinates.md` - Coordinate systems, transformations, and catalog matching
-- `references/cosmology.md` - Cosmological models and calculations
-- `references/fits.md` - FITS file operations and manipulation
-- `references/tables.md` - Table creation, I/O, and operations
-- `references/time.md` - Time formats, scales, and calculations
-- `references/wcs_and_other_modules.md` - WCS, NDData, modeling, visualization, constants, and utilities
-