py-gbcms 2.0.0__tar.gz → 2.1.2__tar.gz

{py_gbcms-2.0.0 → py_gbcms-2.1.2}/.gitignore

@@ -33,6 +33,9 @@ htmlcov/
 .tox/
 coverage.xml
 *.cover
+.mypy_cache/
+.ruff_cache/
+.benchmarks/
 
 # IDEs
 .vscode/
@@ -56,6 +59,16 @@ test_data/
 *.maf
 output/
 results/
+test_output*/
+*.txt
+
+# Exception: Allow test data files
+!tests/testdata/*.bam
+!tests/testdata/*.bam.bai
+!tests/testdata/*.fa
+!tests/testdata/*.fa.fai
+!tests/testdata/*.vcf
+!tests/testdata/*.maf
 
 # Logs
 *.log
@@ -63,3 +76,9 @@ results/
 # Temporary files
 tmp/
 temp/
+
+# Rust
+target/
+
+# Documentation
+site/

py_gbcms-2.1.2/CHANGELOG.md

@@ -0,0 +1,192 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [2.1.2] - 2025-11-25
+
+### 🔧 Fixed
+- **PyPI Distribution**: Fixed source distribution size issue by correctly excluding large files (tests, docs, etc.) via `pyproject.toml` configuration.
+
+## [2.1.1] - 2025-11-25 [YANKED]
+
+> [!WARNING]
+> This release was yanked from PyPI due to a source distribution size limit error. Use 2.1.2 instead.
+
+### 🔧 Fixed
+- **PyPI Distribution**: Added MANIFEST.in (failed to work with Hatchling) to reduce source distribution size
+- **Documentation**: Added comprehensive Installation guide
+- **Documentation**: Unified Contributing guide (merged code + docs contributions)
+- **Documentation**: Added Changelog to documentation navigation
+
+## [2.1.0] - 2025-11-25
+
+### ✨ Added
+
+#### Nextflow Workflow
+- **Production-ready Nextflow workflow** for processing multiple samples in parallel
+- **SLURM cluster support** with customizable queue configuration
+- **Per-sample suffix support** via optional `suffix` column in samplesheet
+- **Docker and Singularity profiles** for containerized execution
+- **Automatic BAI index discovery** with validation
+- **Resume capability** for failed workflow runs
+- **Resource management** with automatic retry and scaling
+- **Comprehensive documentation** in `docs/NEXTFLOW.md` and `nextflow/README.md`
+
+#### Documentation
+- **Usage pattern comparison** guide (`docs/WORKFLOWS.md`) for choosing between CLI and Nextflow
+- **MkDocs integration** for beautiful GitHub Pages documentation
+- **Local documentation preview** with live reload (`mkdocs serve`)
+- **Staging deployment** from `develop` branch for testing docs
+- **Production deployment** from `main` branch
+- **Reorganized documentation structure** with clear CLI vs Nextflow separation
+- **CLI Quick Start guide** (`docs/quick-start.md`)
+
+### 🔧 Changed
+- **Documentation workflow**: docs now live on `main` branch with automated deployment
+- **GitBook integration**: configured to read from `main` branch
+- **Nextflow module**: improved parameter passing with meta.suffix support
+
+### 📝 Documentation
+- Complete Nextflow workflow guide with SLURM examples
+- Per-sample suffix usage examples
+- Git-flow documentation workflow guide
+- Local preview instructions
+- Updated README with clear usage pattern separation
+
+## [2.0.0] - 2025-11-21
+
+### 🚀 Major Rewrite
+
+Version 2.0.0 represents a complete rewrite of py-gbcms with a focus on performance, correctness, and modern architecture.
+
+### ✨ Added
+
+#### Core Features
+- **Rust-based Counting Engine**: Hybrid Python/Rust architecture for 20x+ performance improvement
+- **Strand Bias Statistics**: Fisher's exact test p-values and odds ratios for both reads (`SB_PVAL`, `SB_OR`) and fragments (`FSB_PVAL`, `FSB_OR`)
+- **Fragment-Level Counting**: Majority-rule fragment counting with strand-specific counts (`RDF`, `ADF`)
+- **Variant Allele Fractions**: Read-level (`VAF`) and fragment-level (`FAF`) allele fraction calculations
+- **Thread Control**: Explicit control over parallelism via `--threads` argument (default: 1)
+
+#### Input/Output
+- **VCF Output Format**: Standard VCF with comprehensive INFO and FORMAT fields
+- **MAF Output Format**: Extended MAF with custom columns for strand counts and statistics
+- **Column Preservation**: Input MAF columns are preserved in output
+- **Multiple BAM Support**: Process multiple samples via `--bam-list` or repeated `--bam` arguments
+- **Sample ID Override**: Explicit sample naming via `--bam sample_id:path` syntax
+
+#### Filters
+- `--filter-duplicates`: Filter duplicate reads (default: enabled)
+- `--filter-secondary`: Filter secondary alignments
+- `--filter-supplementary`: Filter supplementary alignments
+- `--filter-qc-failed`: Filter reads that failed QC
+- `--filter-improper-pair`: Filter improperly paired reads
+- `--filter-indel`: Filter reads with indels in CIGAR
+
+#### CLI & Usability
+- **Modern CLI**: Built with Typer and Rich for beautiful terminal output
+- **Progress Tracking**: Real-time progress bars and status indicators
+- **Direct Invocation**: Use `gbcms run` instead of `python -m gbcms.cli`
+- **Output Customization**: `--suffix` flag for output filename customization
+- **Flexible Input**: Support for both VCF and MAF input formats
+
+#### Infrastructure
+- **Docker Support**: Production-ready multi-stage Dockerfile with optimized layers
+- **Type Safety**: Full type annotations with mypy support
+- **Type Stubs**: Provided `.pyi` stub file for Rust extension
+- **Comprehensive Tests**: Extended test suite with accuracy and filter validation
+- **CI/CD**: GitHub Actions workflows for testing, linting, and releases
+
+### 🔄 Changed
+
+#### Architecture
+- Migrated from pure Python to hybrid Python/Rust architecture
+- Core counting logic implemented in Rust using `rust-htslib`
+- Data parallelism over variants with per-thread BAM readers
+
+#### Output Formats
+- **VCF FORMAT fields**: Strand-specific counts now use comma-separated values (e.g., `RD=5,3` for forward,reverse)
+- **MAF columns**: Standardized column names (`t_ref_count_forward`, `t_alt_count_reverse`, etc.)
+- **Coordinate System**: Internal 0-based indexing with correct conversion for VCF (1-based) and MAF output
+
+#### Performance
+- **Speed**: 20x+ faster than v1.x on typical datasets
+- **Memory**: Efficient per-thread BAM readers with minimal overhead
+- **Scalability**: Configurable thread pool for optimal resource usage
+
+#### Dependencies
+- **Python**: Updated to require Python ≥3.10
+- **Rust**: pyo3 0.27.1, rust-htslib 0.51.0, statrs 0.18.0
+- **Python Packages**: pysam ≥0.21.0, typer ≥0.9.0, rich ≥13.0.0, pydantic ≥2.0.0
+
+### 🗑️ Removed
+
+- **Legacy Python Counting**: Pure Python implementation removed in favor of Rust
+- **Old CLI**: Deprecated `python -m gbcms.cli` entry point
+- **Unused Dependencies**: Removed `cyvcf2` and `numba` (no longer needed)
+- **Pre-commit Hooks**: Removed in favor of explicit linting in CI
+
+### 🐛 Fixed
+
+- Correct handling of complex variants (MNPs, DelIns)
+- Proper strand assignment for fragment counting
+- Reference validation against FASTA for all variant types
+- Thread-safe BAM access with per-thread readers
+
+### 📚 Documentation
+
+- Complete rewrite of all documentation
+- New guides: `INSTALLATION.md`, `CLI_FEATURES.md`, `INPUT_OUTPUT.md`
+- Comprehensive API documentation
+- Docker usage examples
+- Contributing guidelines updated
+
+### 🔧 Technical Details
+
+#### Rust Components
+- `gbcms_rs`: PyO3-based extension module
+- Fisher's exact test via `statrs` crate
+- Rayon-based parallelism with configurable thread pools
+- Safe memory management with Rust's ownership model
+
+#### Testing
+- 16 comprehensive test cases
+- Accuracy validation with synthetic BAM files
+- Filter validation for all read flag combinations
+- Integration tests with real-world data
+
+### ⚠️ Breaking Changes
+
+Version 2.0.0 is **not backward compatible** with 1.x. Key breaking changes:
+
+1. **CLI syntax**: Use `gbcms run` instead of `python -m gbcms.cli`
+2. **Output format**: VCF/MAF column structures have changed
+3. **Default behavior**: Only duplicate filtering enabled by default (was: all filters)
+4. **Dependencies**: Requires Rust toolchain for installation from source
+5. **Python version**: Minimum Python 3.10 (was: 3.8)
+
+### 📦 Installation
+
+```bash
+# From PyPI (includes pre-built wheels)
+pip install py-gbcms
+
+# From source (requires Rust)
+pip install git+https://github.com/msk-access/py-gbcms.git
+
+# Docker
+docker pull ghcr.io/msk-access/py-gbcms:2.0.0
+```
+
+### 🙏 Acknowledgments
+
+This rewrite was designed and implemented with a focus on correctness, performance, and modern best practices in bioinformatics software development.
+
+---
+
+## [1.x] - Legacy
+
+Previous versions (1.x) used a pure Python implementation. See git history for details.

{py_gbcms-2.0.0 → py_gbcms-2.1.2}/CONTRIBUTING.md

@@ -20,10 +20,7 @@ Thank you for your interest in contributing to GetBaseCounts! This document prov
    uv pip install -e ".[dev]"
    ```
 
-4. **Install pre-commit hooks**
-   ```bash
-   pre-commit install
-   ```
+
 
 ## Development Workflow
 
@@ -64,10 +61,6 @@ mypy src/
 ```bash
 # Build Docker image
 docker build -t gbcms:latest .
-
-# Run tests in Docker
-docker build -f Dockerfile.test -t gbcms:test .
-docker run --rm gbcms:test
 ```
 
 ## Code Style

py_gbcms-2.1.2/PKG-INFO

@@ -0,0 +1,216 @@
+Metadata-Version: 2.4
+Name: py-gbcms
+Version: 2.1.2
+Summary: Python implementation of GetBaseCountsMultiSample (gbcms) for calculating base counts in BAM files
+Project-URL: Homepage, https://github.com/msk-access/py-gbcms
+Project-URL: Repository, https://github.com/msk-access/py-gbcms
+Project-URL: Documentation, https://github.com/msk-access/py-gbcms#readme
+Project-URL: Bug Tracker, https://github.com/msk-access/py-gbcms/issues
+Author-email: MSK-ACCESS <shahr2@mskcc.org>
+License: AGPL-3.0
+License-File: LICENSE
+Keywords: bam,base-counts,bioinformatics,gbcms,genomics,maf,vcf
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: GNU Affero General Public License v3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Requires-Python: >=3.10
+Requires-Dist: pydantic>=2.0.0
+Requires-Dist: pysam>=0.21.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: typer>=0.9.0
+Provides-Extra: all
+Provides-Extra: dev
+Requires-Dist: black>=23.0.0; extra == 'dev'
+Requires-Dist: mkdocs-material>=9.0.0; extra == 'dev'
+Requires-Dist: mypy>=1.5.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
+Requires-Dist: pytest-mock>=3.11.0; extra == 'dev'
+Requires-Dist: pytest>=7.4.0; extra == 'dev'
+Requires-Dist: ruff>=0.1.0; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0.0; extra == 'dev'
+Provides-Extra: fast
+Description-Content-Type: text/markdown
+
+# py-gbcms
+
+**Complete orientation-aware counting system for genomic variants**
+
+[![Tests](https://github.com/msk-access/py-gbcms/workflows/Tests/badge.svg)](https://github.com/msk-access/py-gbcms/actions)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+
+## Features
+
+- 🚀 **High Performance**: Rust-powered core engine with multi-threading
+- 🧬 **Complete Variant Support**: SNP, MNP, insertion, deletion, and complex variants (DelIns, SNP+Indel)
+- 📊 **Orientation-Aware**: Forward and reverse strand analysis with fragment counting
+- 🔬 **Statistical Analysis**: Fisher's exact test for strand bias
+- 📁 **Flexible I/O**: VCF and MAF input/output formats
+- 🎯 **Quality Filters**: 7 configurable read filtering options
+
+## Installation
+
+**Quick install:**
+```bash
+pip install py-gbcms
+```
+
+**From source (requires Rust):**
+```bash
+git clone https://github.com/msk-access/py-gbcms.git
+cd py-gbcms
+pip install .
+```
+
+**Docker:**
+```bash
+docker pull ghcr.io/msk-access/py-gbcms:2.1.0
+```
+
+📖 **Full documentation:** https://msk-access.github.io/py-gbcms/
+
+---
+
+## Usage
+
+`py-gbcms` can be used in two ways:
+
+### 🔧 Option 1: Standalone CLI (1-10 samples)
+
+**Best for:** Quick analysis, local processing, direct control
+
+```bash
+gbcms run \
+    --variants variants.vcf \
+    --bam sample1.bam \
+    --fasta reference.fa \
+    --output-dir results/
+```
+
+**Output:** `results/sample1.vcf`
+
+**Learn more:**
+- 📘 [CLI Quick Start](https://cmo-ci.gitbook.io/py-gbcms/quick-start)
+- 📖 [CLI Reference](https://cmo-ci.gitbook.io/py-gbcms/cli_features)
+
+---
+
+### 🔄 Option 2: Nextflow Workflow (10+ samples, HPC)
+
+**Best for:** Many samples, HPC clusters (SLURM), reproducible pipelines
+
+```bash
+nextflow run nextflow/main.nf \
+    --input samplesheet.csv \
+    --variants variants.vcf \
+    --fasta reference.fa \
+    -profile slurm
+```
+
+**Features:**
+- ✅ Automatic parallelization across samples
+- ✅ SLURM/HPC integration
+- ✅ Container support (Docker/Singularity)
+- ✅ Resume failed runs
+
+**Learn more:**
+- 🔄 [Nextflow Workflow Guide](https://cmo-ci.gitbook.io/py-gbcms/nextflow)
+- 📋 [Usage Patterns Comparison](https://cmo-ci.gitbook.io/py-gbcms/workflows)
+
+---
+
+## Which Should I Use?
+
+| Scenario | Recommendation |
+|----------|----------------|
+| 1-10 samples, local machine | **CLI** |
+| 10+ samples, HPC cluster | **Nextflow** |
+| Quick ad-hoc analysis | **CLI** |
+| Production pipeline | **Nextflow** |
+| Need auto-parallelization | **Nextflow** |
+| Full manual control | **CLI** |
+
+---
+
+## Quick Examples
+
+### CLI: Single Sample
+```bash
+gbcms run \
+    --variants variants.vcf \
+    --bam tumor.bam \
+    --fasta hg19.fa \
+    --output-dir results/ \
+    --threads 4
+```
+
+### CLI: Multiple Samples (Sequential)
+```bash
+gbcms run \
+    --variants variants.vcf \
+    --bam-list samples.txt \
+    --fasta hg19.fa \
+    --output-dir results/
+```
+
+### Nextflow: Many Samples (Parallel)
+```bash
+# samplesheet.csv:
+# sample,bam,bai
+# tumor1,/path/to/tumor1.bam,
+# tumor2,/path/to/tumor2.bam,
+
+nextflow run nextflow/main.nf \
+    --input samplesheet.csv \
+    --variants variants.vcf \
+    --fasta hg19.fa \
+    --outdir results \
+    -profile slurm
+```
+
+---
+
+## Documentation
+
+📚 **Full Documentation:** https://cmo-ci.gitbook.io/py-gbcms/
+
+**Quick Links:**
+- [Installation](https://cmo-ci.gitbook.io/py-gbcms/installation)
+- [CLI Quick Start](https://cmo-ci.gitbook.io/py-gbcms/quick-start)
+- [Nextflow Workflow](https://cmo-ci.gitbook.io/py-gbcms/nextflow)
+- [CLI Reference](https://cmo-ci.gitbook.io/py-gbcms/cli_features)
+- [Input & Output Formats](https://cmo-ci.gitbook.io/py-gbcms/input_output)
+- [Architecture](https://cmo-ci.gitbook.io/py-gbcms/architecture)
+
+---
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development guidelines.
+
+To contribute to documentation, see the [`gh-pages` branch](https://github.com/msk-access/py-gbcms/tree/gh-pages).
+
+---
+
+## Citation
+
+If you use `py-gbcms` in your research, please cite:
+
+```
+[Citation to be added]
+```
+
+---
+
+## License
+
+AGPL-3.0 - see [LICENSE](LICENSE) for details.
+
+---
+
+## Support
+
+- 🐛 **Issues:** https://github.com/msk-access/py-gbcms/issues
+- 💬 **Discussions:** https://github.com/msk-access/py-gbcms/discussions

py_gbcms-2.1.2/README.md

@@ -0,0 +1,180 @@
+# py-gbcms
+
+**Complete orientation-aware counting system for genomic variants**
+
+[![Tests](https://github.com/msk-access/py-gbcms/workflows/Tests/badge.svg)](https://github.com/msk-access/py-gbcms/actions)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10+-blue.svg)](https://www.python.org/downloads/)
+
+## Features
+
+- 🚀 **High Performance**: Rust-powered core engine with multi-threading
+- 🧬 **Complete Variant Support**: SNP, MNP, insertion, deletion, and complex variants (DelIns, SNP+Indel)
+- 📊 **Orientation-Aware**: Forward and reverse strand analysis with fragment counting
+- 🔬 **Statistical Analysis**: Fisher's exact test for strand bias
+- 📁 **Flexible I/O**: VCF and MAF input/output formats
+- 🎯 **Quality Filters**: 7 configurable read filtering options
+
+## Installation
+
+**Quick install:**
+```bash
+pip install py-gbcms
+```
+
+**From source (requires Rust):**
+```bash
+git clone https://github.com/msk-access/py-gbcms.git
+cd py-gbcms
+pip install .
+```
+
+**Docker:**
+```bash
+docker pull ghcr.io/msk-access/py-gbcms:2.1.0
+```
+
+📖 **Full documentation:** https://msk-access.github.io/py-gbcms/
+
+---
+
+## Usage
+
+`py-gbcms` can be used in two ways:
+
+### 🔧 Option 1: Standalone CLI (1-10 samples)
+
+**Best for:** Quick analysis, local processing, direct control
+
+```bash
+gbcms run \
+    --variants variants.vcf \
+    --bam sample1.bam \
+    --fasta reference.fa \
+    --output-dir results/
+```
+
+**Output:** `results/sample1.vcf`
+
+**Learn more:**
+- 📘 [CLI Quick Start](https://cmo-ci.gitbook.io/py-gbcms/quick-start)
+- 📖 [CLI Reference](https://cmo-ci.gitbook.io/py-gbcms/cli_features)
+
+---
+
+### 🔄 Option 2: Nextflow Workflow (10+ samples, HPC)
+
+**Best for:** Many samples, HPC clusters (SLURM), reproducible pipelines
+
+```bash
+nextflow run nextflow/main.nf \
+    --input samplesheet.csv \
+    --variants variants.vcf \
+    --fasta reference.fa \
+    -profile slurm
+```
+
+**Features:**
+- ✅ Automatic parallelization across samples
+- ✅ SLURM/HPC integration
+- ✅ Container support (Docker/Singularity)
+- ✅ Resume failed runs
+
+**Learn more:**
+- 🔄 [Nextflow Workflow Guide](https://cmo-ci.gitbook.io/py-gbcms/nextflow)
+- 📋 [Usage Patterns Comparison](https://cmo-ci.gitbook.io/py-gbcms/workflows)
+
+---
+
+## Which Should I Use?
+
+| Scenario | Recommendation |
+|----------|----------------|
+| 1-10 samples, local machine | **CLI** |
+| 10+ samples, HPC cluster | **Nextflow** |
+| Quick ad-hoc analysis | **CLI** |
+| Production pipeline | **Nextflow** |
+| Need auto-parallelization | **Nextflow** |
+| Full manual control | **CLI** |
+
+---
+
+## Quick Examples
+
+### CLI: Single Sample
+```bash
+gbcms run \
+    --variants variants.vcf \
+    --bam tumor.bam \
+    --fasta hg19.fa \
+    --output-dir results/ \
+    --threads 4
+```
+
+### CLI: Multiple Samples (Sequential)
+```bash
+gbcms run \
+    --variants variants.vcf \
+    --bam-list samples.txt \
+    --fasta hg19.fa \
+    --output-dir results/
+```
+
+### Nextflow: Many Samples (Parallel)
+```bash
+# samplesheet.csv:
+# sample,bam,bai
+# tumor1,/path/to/tumor1.bam,
+# tumor2,/path/to/tumor2.bam,
+
+nextflow run nextflow/main.nf \
+    --input samplesheet.csv \
+    --variants variants.vcf \
+    --fasta hg19.fa \
+    --outdir results \
+    -profile slurm
+```
+
+---
+
+## Documentation
+
+📚 **Full Documentation:** https://cmo-ci.gitbook.io/py-gbcms/
+
+**Quick Links:**
+- [Installation](https://cmo-ci.gitbook.io/py-gbcms/installation)
+- [CLI Quick Start](https://cmo-ci.gitbook.io/py-gbcms/quick-start)
+- [Nextflow Workflow](https://cmo-ci.gitbook.io/py-gbcms/nextflow)
+- [CLI Reference](https://cmo-ci.gitbook.io/py-gbcms/cli_features)
+- [Input & Output Formats](https://cmo-ci.gitbook.io/py-gbcms/input_output)
+- [Architecture](https://cmo-ci.gitbook.io/py-gbcms/architecture)
+
+---
+
+## Contributing
+
+See [CONTRIBUTING.md](CONTRIBUTING.md) for development guidelines.
+
+To contribute to documentation, see the [`gh-pages` branch](https://github.com/msk-access/py-gbcms/tree/gh-pages).
+
+---
+
+## Citation
+
+If you use `py-gbcms` in your research, please cite:
+
+```
+[Citation to be added]
+```
+
+---
+
+## License
+
+AGPL-3.0 - see [LICENSE](LICENSE) for details.
+
+---
+
+## Support
+
+- 🐛 **Issues:** https://github.com/msk-access/py-gbcms/issues
+- 💬 **Discussions:** https://github.com/msk-access/py-gbcms/discussions