@saibolla/ada 0.1.2

package/skills/pyopenms/references/signal_processing.md

@@ -0,0 +1,433 @@
+# Signal Processing
+
+## Overview
+
+PyOpenMS provides algorithms for processing raw mass spectrometry data including smoothing, filtering, peak picking, centroiding, normalization, and deconvolution.
+
+## Algorithm Pattern
+
+Most signal processing algorithms follow a standard pattern:
+
+```python
+import pyopenms as ms
+
+# 1. Create algorithm instance
+algo = ms.AlgorithmName()
+
+# 2. Get and modify parameters
+params = algo.getParameters()
+params.setValue("parameter_name", value)
+algo.setParameters(params)
+
+# 3. Apply to data
+algo.filterExperiment(exp)  # or filterSpectrum(spec)
+```
+
+## Smoothing
+
+### Gaussian Filter
+
+Apply Gaussian smoothing to reduce noise:
+
+```python
+# Create Gaussian filter
+gaussian = ms.GaussFilter()
+
+# Configure parameters
+params = gaussian.getParameters()
+params.setValue("gaussian_width", 0.2)  # Width in m/z or RT units
+params.setValue("ppm_tolerance", 10.0)  # For m/z dimension
+params.setValue("use_ppm_tolerance", "true")
+gaussian.setParameters(params)
+
+# Apply to experiment
+gaussian.filterExperiment(exp)
+
+# Or apply to single spectrum
+spec = exp.getSpectrum(0)
+gaussian.filterSpectrum(spec)
+```
+
+### Savitzky-Golay Filter
+
+Polynomial smoothing that preserves peak shapes:
+
+```python
+# Create Savitzky-Golay filter
+sg_filter = ms.SavitzkyGolayFilter()
+
+# Configure parameters
+params = sg_filter.getParameters()
+params.setValue("frame_length", 11)  # Window size (must be odd)
+params.setValue("polynomial_order", 4)  # Polynomial degree
+sg_filter.setParameters(params)
+
+# Apply smoothing
+sg_filter.filterExperiment(exp)
+```
+
+## Peak Picking and Centroiding
+
+### Peak Picker High Resolution
+
+Detect peaks in high-resolution data:
+
+```python
+# Create peak picker
+peak_picker = ms.PeakPickerHiRes()
+
+# Configure parameters
+params = peak_picker.getParameters()
+params.setValue("signal_to_noise", 3.0)  # S/N threshold
+params.setValue("spacing_difference", 1.5)  # Minimum peak spacing
+peak_picker.setParameters(params)
+
+# Pick peaks
+exp_picked = ms.MSExperiment()
+peak_picker.pickExperiment(exp, exp_picked)
+```
+
+### Peak Picker for CWT
+
+Continuous wavelet transform-based peak picking:
+
+```python
+# Create CWT peak picker
+cwt_picker = ms.PeakPickerCWT()
+
+# Configure parameters
+params = cwt_picker.getParameters()
+params.setValue("signal_to_noise", 1.0)
+params.setValue("peak_width", 0.15)  # Expected peak width
+cwt_picker.setParameters(params)
+
+# Pick peaks
+cwt_picker.pickExperiment(exp, exp_picked)
+```
+
+## Normalization
+
+### Normalizer
+
+Normalize peak intensities within spectra:
+
+```python
+# Create normalizer
+normalizer = ms.Normalizer()
+
+# Configure normalization method
+params = normalizer.getParameters()
+params.setValue("method", "to_one")  # Options: "to_one", "to_TIC"
+normalizer.setParameters(params)
+
+# Apply normalization
+normalizer.filterExperiment(exp)
+```
+
+## Peak Filtering
+
+### Threshold Mower
+
+Remove peaks below intensity threshold:
+
+```python
+# Create threshold filter
+mower = ms.ThresholdMower()
+
+# Configure threshold
+params = mower.getParameters()
+params.setValue("threshold", 1000.0)  # Absolute intensity threshold
+mower.setParameters(params)
+
+# Apply filter
+mower.filterExperiment(exp)
+```
+
+### Window Mower
+
+Keep only highest peaks in sliding windows:
+
+```python
+# Create window mower
+window_mower = ms.WindowMower()
+
+# Configure parameters
+params = window_mower.getParameters()
+params.setValue("windowsize", 50.0)  # Window size in m/z
+params.setValue("peakcount", 2)  # Keep top N peaks per window
+window_mower.setParameters(params)
+
+# Apply filter
+window_mower.filterExperiment(exp)
+```
+
+### N Largest Peaks
+
+Keep only the N most intense peaks:
+
+```python
+# Create N largest filter
+n_largest = ms.NLargest()
+
+# Configure parameters
+params = n_largest.getParameters()
+params.setValue("n", 200)  # Keep 200 most intense peaks
+n_largest.setParameters(params)
+
+# Apply filter
+n_largest.filterExperiment(exp)
+```
+
+## Baseline Reduction
+
+### Morphological Filter
+
+Remove baseline using morphological operations:
+
+```python
+# Create morphological filter
+morph_filter = ms.MorphologicalFilter()
+
+# Configure parameters
+params = morph_filter.getParameters()
+params.setValue("struc_elem_length", 3.0)  # Structuring element size
+params.setValue("method", "tophat")  # Method: "tophat", "bothat", "erosion", "dilation"
+morph_filter.setParameters(params)
+
+# Apply filter
+morph_filter.filterExperiment(exp)
+```
+
+## Spectrum Merging
+
+### Spectra Merger
+
+Combine multiple spectra into one:
+
+```python
+# Create merger
+merger = ms.SpectraMerger()
+
+# Configure parameters
+params = merger.getParameters()
+params.setValue("average_gaussian:spectrum_type", "profile")
+params.setValue("average_gaussian:rt_FWHM", 5.0)  # RT window
+merger.setParameters(params)
+
+# Merge spectra
+merger.mergeSpectraBlockWise(exp)
+```
+
+## Deconvolution
+
+### Charge Deconvolution
+
+Determine charge states and convert to neutral masses:
+
+```python
+# Create feature deconvoluter
+deconvoluter = ms.FeatureDeconvolution()
+
+# Configure parameters
+params = deconvoluter.getParameters()
+params.setValue("charge_min", 1)
+params.setValue("charge_max", 4)
+params.setValue("potential_charge_states", "1,2,3,4")
+deconvoluter.setParameters(params)
+
+# Apply deconvolution
+feature_map_out = ms.FeatureMap()
+deconvoluter.compute(exp, feature_map, feature_map_out, ms.ConsensusMap())
+```
+
+### Isotope Deconvolution
+
+Remove isotopic patterns:
+
+```python
+# Create isotope wavelet transform
+isotope_wavelet = ms.IsotopeWaveletTransform()
+
+# Configure parameters
+params = isotope_wavelet.getParameters()
+params.setValue("max_charge", 3)
+params.setValue("intensity_threshold", 10.0)
+isotope_wavelet.setParameters(params)
+
+# Apply transformation
+isotope_wavelet.transform(exp)
+```
+
+## Retention Time Alignment
+
+### Map Alignment
+
+Align retention times across multiple runs:
+
+```python
+# Create map aligner
+aligner = ms.MapAlignmentAlgorithmPoseClustering()
+
+# Load multiple experiments
+exp1 = ms.MSExperiment()
+exp2 = ms.MSExperiment()
+ms.MzMLFile().load("run1.mzML", exp1)
+ms.MzMLFile().load("run2.mzML", exp2)
+
+# Create reference
+reference = ms.MSExperiment()
+
+# Align experiments
+transformations = []
+aligner.align(exp1, exp2, transformations)
+
+# Apply transformation
+transformer = ms.MapAlignmentTransformer()
+transformer.transformRetentionTimes(exp2, transformations[0])
+```
+
+## Mass Calibration
+
+### Internal Calibration
+
+Calibrate mass axis using known reference masses:
+
+```python
+# Create internal calibration
+calibration = ms.InternalCalibration()
+
+# Set reference masses
+reference_masses = [500.0, 1000.0, 1500.0]  # Known m/z values
+
+# Calibrate
+calibration.calibrate(exp, reference_masses)
+```
+
+## Quality Control
+
+### Spectrum Statistics
+
+Calculate quality metrics:
+
+```python
+# Get spectrum
+spec = exp.getSpectrum(0)
+
+# Calculate statistics
+mz, intensity = spec.get_peaks()
+
+# Total ion current
+tic = sum(intensity)
+
+# Base peak
+base_peak_intensity = max(intensity)
+base_peak_mz = mz[intensity.argmax()]
+
+print(f"TIC: {tic}")
+print(f"Base peak: {base_peak_mz} m/z at {base_peak_intensity}")
+```
+
+## Spectrum Preprocessing Pipeline
+
+### Complete Preprocessing Example
+
+```python
+import pyopenms as ms
+
+def preprocess_experiment(input_file, output_file):
+    """Complete preprocessing pipeline."""
+
+    # Load data
+    exp = ms.MSExperiment()
+    ms.MzMLFile().load(input_file, exp)
+
+    # 1. Smooth with Gaussian filter
+    gaussian = ms.GaussFilter()
+    gaussian.filterExperiment(exp)
+
+    # 2. Pick peaks
+    picker = ms.PeakPickerHiRes()
+    exp_picked = ms.MSExperiment()
+    picker.pickExperiment(exp, exp_picked)
+
+    # 3. Normalize intensities
+    normalizer = ms.Normalizer()
+    params = normalizer.getParameters()
+    params.setValue("method", "to_TIC")
+    normalizer.setParameters(params)
+    normalizer.filterExperiment(exp_picked)
+
+    # 4. Filter low-intensity peaks
+    mower = ms.ThresholdMower()
+    params = mower.getParameters()
+    params.setValue("threshold", 10.0)
+    mower.setParameters(params)
+    mower.filterExperiment(exp_picked)
+
+    # Save processed data
+    ms.MzMLFile().store(output_file, exp_picked)
+
+    return exp_picked
+
+# Run pipeline
+exp_processed = preprocess_experiment("raw_data.mzML", "processed_data.mzML")
+```
+
+## Best Practices
+
+### Parameter Optimization
+
+Test parameters on representative data:
+
+```python
+# Try different Gaussian widths
+widths = [0.1, 0.2, 0.5]
+
+for width in widths:
+    exp_test = ms.MSExperiment()
+    ms.MzMLFile().load("test_data.mzML", exp_test)
+
+    gaussian = ms.GaussFilter()
+    params = gaussian.getParameters()
+    params.setValue("gaussian_width", width)
+    gaussian.setParameters(params)
+    gaussian.filterExperiment(exp_test)
+
+    # Evaluate quality
+    # ... add evaluation code ...
+```
+
+### Preserve Original Data
+
+Keep original data for comparison:
+
+```python
+# Load original
+exp_original = ms.MSExperiment()
+ms.MzMLFile().load("data.mzML", exp_original)
+
+# Create copy for processing
+exp_processed = ms.MSExperiment(exp_original)
+
+# Process copy
+gaussian = ms.GaussFilter()
+gaussian.filterExperiment(exp_processed)
+
+# Original remains unchanged
+```
+
+### Profile vs Centroid Data
+
+Check data type before processing:
+
+```python
+# Check if spectrum is centroided
+spec = exp.getSpectrum(0)
+
+if spec.isSorted():
+    # Likely centroided
+    print("Centroid data")
+else:
+    # Likely profile
+    print("Profile data - apply peak picking")
+```

package/skills/pysam/SKILL.md

@@ -0,0 +1,263 @@
+---
+name: pysam
+description: Genomic file toolkit. Read/write SAM/BAM/CRAM alignments, VCF/BCF variants, FASTA/FASTQ sequences, extract regions, calculate coverage, for NGS data processing pipelines.
+license: MIT license
+metadata:
+    skill-author: K-Dense Inc.
+---
+
+# Pysam
+
+## Overview
+
+Pysam is a Python module for reading, manipulating, and writing genomic datasets. Read/write SAM/BAM/CRAM alignment files, VCF/BCF variant files, and FASTA/FASTQ sequences with a Pythonic interface to htslib. Query tabix-indexed files, perform pileup analysis for coverage, and execute samtools/bcftools commands.
+
+## When to Use This Skill
+
+This skill should be used when:
+- Working with sequencing alignment files (BAM/CRAM)
+- Analyzing genetic variants (VCF/BCF)
+- Extracting reference sequences or gene regions
+- Processing raw sequencing data (FASTQ)
+- Calculating coverage or read depth
+- Implementing bioinformatics analysis pipelines
+- Quality control of sequencing data
+- Variant calling and annotation workflows
+
+## Quick Start
+
+### Installation
+```bash
+uv pip install pysam
+```
+
+### Basic Examples
+
+**Read alignment file:**
+```python
+import pysam
+
+# Open BAM file and fetch reads in region
+samfile = pysam.AlignmentFile("example.bam", "rb")
+for read in samfile.fetch("chr1", 1000, 2000):
+    print(f"{read.query_name}: {read.reference_start}")
+samfile.close()
+```
+
+**Read variant file:**
+```python
+# Open VCF file and iterate variants
+vcf = pysam.VariantFile("variants.vcf")
+for variant in vcf:
+    print(f"{variant.chrom}:{variant.pos} {variant.ref}>{variant.alts}")
+vcf.close()
+```
+
+**Query reference sequence:**
+```python
+# Open FASTA and extract sequence
+fasta = pysam.FastaFile("reference.fasta")
+sequence = fasta.fetch("chr1", 1000, 2000)
+print(sequence)
+fasta.close()
+```
+
+## Core Capabilities
+
+### 1. Alignment File Operations (SAM/BAM/CRAM)
+
+Use the `AlignmentFile` class to work with aligned sequencing reads. This is appropriate for analyzing mapping results, calculating coverage, extracting reads, or quality control.
+
+**Common operations:**
+- Open and read BAM/SAM/CRAM files
+- Fetch reads from specific genomic regions
+- Filter reads by mapping quality, flags, or other criteria
+- Write filtered or modified alignments
+- Calculate coverage statistics
+- Perform pileup analysis (base-by-base coverage)
+- Access read sequences, quality scores, and alignment information
+
+**Reference:** See `references/alignment_files.md` for detailed documentation on:
+- Opening and reading alignment files
+- AlignedSegment attributes and methods
+- Region-based fetching with `fetch()`
+- Pileup analysis for coverage
+- Writing and creating BAM files
+- Coordinate systems and indexing
+- Performance optimization tips
+
+### 2. Variant File Operations (VCF/BCF)
+
+Use the `VariantFile` class to work with genetic variants from variant calling pipelines. This is appropriate for variant analysis, filtering, annotation, or population genetics.
+
+**Common operations:**
+- Read and write VCF/BCF files
+- Query variants in specific regions
+- Access variant information (position, alleles, quality)
+- Extract genotype data for samples
+- Filter variants by quality, allele frequency, or other criteria
+- Annotate variants with additional information
+- Subset samples or regions
+
+**Reference:** See `references/variant_files.md` for detailed documentation on:
+- Opening and reading variant files
+- VariantRecord attributes and methods
+- Accessing INFO and FORMAT fields
+- Working with genotypes and samples
+- Creating and writing VCF files
+- Filtering and subsetting variants
+- Multi-sample VCF operations
+
+### 3. Sequence File Operations (FASTA/FASTQ)
+
+Use `FastaFile` for random access to reference sequences and `FastxFile` for reading raw sequencing data. This is appropriate for extracting gene sequences, validating variants against reference, or processing raw reads.
+
+**Common operations:**
+- Query reference sequences by genomic coordinates
+- Extract sequences for genes or regions of interest
+- Read FASTQ files with quality scores
+- Validate variant reference alleles
+- Calculate sequence statistics
+- Filter reads by quality or length
+- Convert between FASTA and FASTQ formats
+
+**Reference:** See `references/sequence_files.md` for detailed documentation on:
+- FASTA file access and indexing
+- Extracting sequences by region
+- Handling reverse complement for genes
+- Reading FASTQ files sequentially
+- Quality score conversion and filtering
+- Working with tabix-indexed files (BED, GTF, GFF)
+- Common sequence processing patterns
+
+### 4. Integrated Bioinformatics Workflows
+
+Pysam excels at integrating multiple file types for comprehensive genomic analyses. Common workflows combine alignment files, variant files, and reference sequences.
+
+**Common workflows:**
+- Calculate coverage statistics for specific regions
+- Validate variants against aligned reads
+- Annotate variants with coverage information
+- Extract sequences around variant positions
+- Filter alignments or variants based on multiple criteria
+- Generate coverage tracks for visualization
+- Quality control across multiple data types
+
+**Reference:** See `references/common_workflows.md` for detailed examples of:
+- Quality control workflows (BAM statistics, reference consistency)
+- Coverage analysis (per-base coverage, low coverage detection)
+- Variant analysis (annotation, filtering by read support)
+- Sequence extraction (variant contexts, gene sequences)
+- Read filtering and subsetting
+- Integration patterns (BAM+VCF, VCF+BED, etc.)
+- Performance optimization for complex workflows
+
+## Key Concepts
+
+### Coordinate Systems
+
+**Critical:** Pysam uses **0-based, half-open** coordinates (Python convention):
+- Start positions are 0-based (first base is position 0)
+- End positions are exclusive (not included in the range)
+- Region 1000-2000 includes bases 1000-1999 (1000 bases total)
+
+**Exception:** Region strings in `fetch()` follow samtools convention (1-based):
+```python
+samfile.fetch("chr1", 999, 2000)      # 0-based: positions 999-1999
+samfile.fetch("chr1:1000-2000")       # 1-based string: positions 1000-2000
+```
+
+**VCF files:** Use 1-based coordinates in the file format, but `VariantRecord.start` is 0-based.
+
+### Indexing Requirements
+
+Random access to specific genomic regions requires index files:
+- **BAM files**: Require `.bai` index (create with `pysam.index()`)
+- **CRAM files**: Require `.crai` index
+- **FASTA files**: Require `.fai` index (create with `pysam.faidx()`)
+- **VCF.gz files**: Require `.tbi` tabix index (create with `pysam.tabix_index()`)
+- **BCF files**: Require `.csi` index
+
+Without an index, use `fetch(until_eof=True)` for sequential reading.
+
+### File Modes
+
+Specify format when opening files:
+- `"rb"` - Read BAM (binary)
+- `"r"` - Read SAM (text)
+- `"rc"` - Read CRAM
+- `"wb"` - Write BAM
+- `"w"` - Write SAM
+- `"wc"` - Write CRAM
+
+### Performance Considerations
+
+1. **Always use indexed files** for random access operations
+2. **Use `pileup()` for column-wise analysis** instead of repeated fetch operations
+3. **Use `count()` for counting** instead of iterating and counting manually
+4. **Process regions in parallel** when analyzing independent genomic regions
+5. **Close files explicitly** to free resources
+6. **Use `until_eof=True`** for sequential processing without index
+7. **Avoid multiple iterators** unless necessary (use `multiple_iterators=True` if needed)
+
+## Common Pitfalls
+
+1. **Coordinate confusion:** Remember 0-based vs 1-based systems in different contexts
+2. **Missing indices:** Many operations require index files—create them first
+3. **Partial overlaps:** `fetch()` returns reads overlapping region boundaries, not just those fully contained
+4. **Iterator scope:** Keep pileup iterator references alive to avoid "PileupProxy accessed after iterator finished" errors
+5. **Quality score editing:** Cannot modify `query_qualities` in place after changing `query_sequence`—create a copy first
+6. **Stream limitations:** Only stdin/stdout are supported for streaming, not arbitrary Python file objects
+7. **Thread safety:** While GIL is released during I/O, comprehensive thread-safety hasn't been fully validated
+
+## Command-Line Tools
+
+Pysam provides access to samtools and bcftools commands:
+
+```python
+# Sort BAM file
+pysam.samtools.sort("-o", "sorted.bam", "input.bam")
+
+# Index BAM
+pysam.samtools.index("sorted.bam")
+
+# View specific region
+pysam.samtools.view("-b", "-o", "region.bam", "input.bam", "chr1:1000-2000")
+
+# BCF tools
+pysam.bcftools.view("-O", "z", "-o", "output.vcf.gz", "input.vcf")
+```
+
+**Error handling:**
+```python
+try:
+    pysam.samtools.sort("-o", "output.bam", "input.bam")
+except pysam.SamtoolsError as e:
+    print(f"Error: {e}")
+```
+
+## Resources
+
+### references/
+
+Detailed documentation for each major capability:
+
+- **alignment_files.md** - Complete guide to SAM/BAM/CRAM operations, including AlignmentFile class, AlignedSegment attributes, fetch operations, pileup analysis, and writing alignments
+
+- **variant_files.md** - Complete guide to VCF/BCF operations, including VariantFile class, VariantRecord attributes, genotype handling, INFO/FORMAT fields, and multi-sample operations
+
+- **sequence_files.md** - Complete guide to FASTA/FASTQ operations, including FastaFile and FastxFile classes, sequence extraction, quality score handling, and tabix-indexed file access
+
+- **common_workflows.md** - Practical examples of integrated bioinformatics workflows combining multiple file types, including quality control, coverage analysis, variant validation, and sequence extraction
+
+## Getting Help
+
+For detailed information on specific operations, refer to the appropriate reference document:
+
+- Working with BAM files or calculating coverage → `alignment_files.md`
+- Analyzing variants or genotypes → `variant_files.md`
+- Extracting sequences or processing FASTQ → `sequence_files.md`
+- Complex workflows integrating multiple file types → `common_workflows.md`
+
+Official documentation: https://pysam.readthedocs.io/
+