pywombat 0.5.0__tar.gz → 1.0.1__tar.gz

pywombat-1.0.1/.github/copilot-instructions.md

@@ -0,0 +1,130 @@
+# PyWombat Development Guide
+
+## Project Overview
+
+PyWombat is a bioinformatics CLI tool that transforms bcftools tabulated TSV files into analysis-ready long-format data. It specializes in:
+- Expanding VCF INFO fields from `(null)` column into individual columns
+- Converting wide-format sample data to long-format (melting)
+- Extracting and calculating genotype metrics (GT, DP, GQ, AD, VAF)
+- De novo mutation (DNM) detection with sex-chromosome awareness
+- Trio/family analysis with pedigree-based parent genotype joining
+
+## Architecture
+
+### Single-File Monolith
+All functionality lives in [src/pywombat/cli.py](src/pywombat/cli.py) (~2100 lines). This is intentional for deployment simplicity with `uvx` (one-command installation-free execution).
+
+### Core Data Flow
+1. **Input**: bcftools tabulated TSV (wide format, one row per variant)
+2. **Transform Pipeline**:
+   - `format_bcftools_tsv_lazy()` → Expand INFO, melt samples, parse genotypes
+   - `apply_filters_lazy()` → Apply quality/expression filters OR DNM detection
+3. **Output**: Long-format TSV/Parquet (one row per variant-sample)
+
+### Key Functions
+- **`format_bcftools_tsv()`**: Eager transformation (collects data)
+- **`format_bcftools_tsv_lazy()`**: Streaming wrapper using Polars LazyFrame
+- **`apply_filters_lazy()`**: Conditional filter dispatcher (quality/expression OR DNM)
+- **`apply_de_novo_filter()`**: Complex vectorized DNM logic with PAR region handling
+- **`parse_impact_filter_expression()`**: Expression parser for flexible YAML-based filtering
+
+## Development Workflow
+
+### Setup & Running
+```bash
+# Install dependencies (uses uv package manager)
+uv sync
+
+# Run CLI locally
+uv run wombat input.tsv -o output
+
+# Production usage (no installation)
+uvx pywombat input.tsv -o output
+```
+
+### Testing
+- Test data: `tests/C0733-011-068.*` files (real variant data)
+- Compare script: [tests/compare_dnm_results.py](tests/compare_dnm_results.py) validates DNM detection
+- No formal test framework yet—manual validation against reference outputs in `tests/check/`
+
+## Critical Domain Logic
+
+### Genotype Parsing
+- Input format: `GT:DP:GQ:AD` (e.g., `0/1:25:99:10,15`)
+- Extracted fields:
+  - `sample_gt`: Genotype (0/1, 1/1, etc.)
+  - `sample_dp`: Total depth (25)
+  - `sample_gq`: Quality (99)
+  - `sample_ad`: **Second value** from AD (15 = alternate allele depth)
+  - `sample_vaf`: Calculated as `sample_ad / sample_dp` (0.6)
+
+### De Novo Mutation Detection
+Sex-chromosome-aware logic in `apply_de_novo_filter()`:
+- **Autosomes**: Both parents must have VAF < 2% (reference genotype)
+- **X chromosome (male proband)**: Mother must be reference; father is hemizygous (ignored)
+- **Y chromosome**: Only father checked (mother doesn't have Y)
+- **PAR regions**: Treated as autosomal (both parents checked)
+- **Hemizygous variants** (X/Y in males): Require VAF ≥ 85% (not ~50% like het)
+
+PAR regions defined in [examples/de_novo_mutations.yml](examples/de_novo_mutations.yml):
+```yaml
+par_regions:
+  GRCh38:
+    PAR1: { chrom: "X", start: 10001, end: 2781479 }
+    PAR2: { chrom: "X", start: 155701383, end: 156030895 }
+```
+
+## Conventions & Patterns
+
+### Polars Usage
+- **Lazy evaluation**: Use `scan_csv()` → `LazyFrame` → `sink_csv()` for streaming
+- **Streaming mode**: `collect(streaming=True)` for memory-efficient processing
+- **Schema overrides**: Force string type for sample IDs to prevent numeric inference:
+  ```python
+  schema_overrides = {col: pl.Utf8 for col in ["sample", "FatherBarcode", ...]}
+  ```
+
+### YAML Configuration Files
+Filter configs in [examples/](examples/) define:
+- `quality`: Thresholds (DP, GQ, VAF ranges for het/hom)
+- `expression`: Polars-compatible filter expressions (see `parse_impact_filter_expression()`)
+- `dnm`: De novo detection parameters (parent thresholds, PAR regions)
+
+Example expression syntax:
+```yaml
+expression: "VEP_IMPACT = HIGH & fafmax_faf95_max_genomes <= 0.001"
+```
+Supports: `=`, `!=`, `<=`, `>=`, `<`, `>`, `&`, `|`, `()`, NULL, NaN
+
+### Error Handling
+- Use `click.echo(..., err=True)` for verbose messages (stderr)
+- Raise `click.Abort()` instead of generic exceptions for CLI errors
+- Validate pedigree/config requirements early (e.g., DNM mode requires pedigree)
+
+## Common Pitfalls
+
+1. **VAF Calculation**: Always use `sample_ad / sample_dp`, not first AD value
+2. **Genotype Filtering**: Use `str.contains("1")` not regex—handles 0/1, 1/0, 1/1, 1/2
+3. **Parent Joining**: Pedigree must have `sample_id`, `FatherBarcode`, `MotherBarcode` columns
+4. **Sex Normalization**: DNM filter accepts "1"/"2" or "M"/"F" for sex; normalizes to uppercase
+5. **Categorical Types**: Convert `#CHROM` from Categorical to Utf8 before string operations
+
+## File References
+
+- Main CLI: [src/pywombat/cli.py](src/pywombat/cli.py)
+- Config examples: [examples/de_novo_mutations.yml](examples/de_novo_mutations.yml), [examples/rare_variants_high_impact.yml](examples/rare_variants_high_impact.yml)
+- Test data: [tests/C0733-011-068.pedigree.tsv](tests/C0733-011-068.pedigree.tsv)
+- Documentation: [README.md](README.md), [QUICKSTART.md](QUICKSTART.md)
+
+## Adding New Features
+
+When adding filters:
+1. Update YAML schema in config examples
+2. Extend `apply_filters_lazy()` or create specialized function (like `apply_de_novo_filter()`)
+3. Use lazy operations when possible; collect only if vectorization requires it
+4. Add verbose logging with `--verbose` flag support
+
+When modifying transforms:
+- Keep `format_bcftools_tsv()` and `format_bcftools_tsv_lazy()` in sync
+- Test with real data from `tests/` directory
+- Verify output with `compare_dnm_results.py` for DNM changes

pywombat-1.0.1/CHANGELOG.md

@@ -0,0 +1,135 @@
+# Changelog
+
+All notable changes to PyWombat will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.0.1] - 2026-01-24
+
+### Added
+
+- **Boolean Flag Support in INFO Fields**: INFO field entries without `=` signs (e.g., `PASS`, `DB`, `SOMATIC`) are now extracted as boolean columns with `True`/`False` values instead of being ignored. This enables filtering on VCF flag fields.
+
+### Fixed
+
+- INFO field parsing now handles all field types correctly, including standalone boolean flags commonly used in VCF files.
+
+## [1.0.0] - 2026-01-23
+
+First stable release of PyWombat! 🎉
+
+### Added
+
+#### Core Features
+
+- **Fast TSV Processing**: Efficient processing of bcftools tabulated TSV files using Polars
+- **Flexible Output Formats**: Support for TSV, compressed TSV (`.gz`), and Parquet formats
+- **Streaming Mode**: Memory-efficient processing for large files
+- **Pedigree Support**: Trio and family analysis with automatic parent genotype joining
+- **Multiple Sample Formats**: Handles various genotype formats (GT:DP:GQ:AD and variants)
+
+#### Filtering Capabilities
+
+- **Quality Filters**: Configurable thresholds for depth (DP), genotype quality (GQ), and variant allele frequency (VAF)
+- **Genotype-Specific VAF Filters**: Separate thresholds for heterozygous, homozygous alternate, and homozygous reference calls
+- **Expression-Based Filtering**: Complex logical expressions with comparison operators (`=`, `!=`, `<`, `>`, `<=`, `>=`) and logical operators (`&`, `|`)
+- **Parent Quality Filtering**: Optional quality filter application to parent genotypes
+
+#### De Novo Mutation Detection
+
+- **Sex-Chromosome Aware Logic**: Proper handling of X and Y chromosomes in males
+- **PAR Region Support**: Configurable pseudo-autosomal region (PAR) coordinates for GRCh37 and GRCh38
+- **Hemizygous Variant Detection**: Specialized VAF thresholds for X chromosome in males (non-PAR) and Y chromosome
+- **Homozygous VAF Thresholds**: Higher VAF requirements (≥85%) for homozygous variants
+- **Parent Genotype Validation**: Ensures parents are homozygous reference with low VAF (<2%)
+- **Missing Genotype Filtering**: Removes variants with partial/missing genotypes (`./.`, `0/.`, etc.)
+- **Population Frequency Filtering**: Maximum allele frequency thresholds (gnomAD fafmax_faf95_max_genomes)
+- **Quality Filter Support**: gnomAD genomes_filters PASS-only option
+
+#### User Experience
+
+- **Debug Mode**: Inspect specific variants by chromosome:position for troubleshooting
+- **Verbose Mode**: Detailed filtering step information with variant counts
+- **Automatic Output Naming**: Intelligent output file naming based on input and filter config
+- **Configuration Examples**: Two comprehensive example configurations with extensive documentation
+  - `rare_variants_high_impact.yml`: Ultra-rare, high-impact variant filtering
+  - `de_novo_mutations.yml`: De novo mutation detection with full documentation
+
+#### Documentation
+
+- **Comprehensive README**: Complete usage guide with examples for all features
+- **Example Workflows**: Real-world usage scenarios (rare disease, autism trios, etc.)
+- **Input Requirements**: Detailed bcftools command examples for generating input files
+- **VEP Annotation Guide**: Complete workflow from VEP annotation to PyWombat processing
+- **Examples Directory**: Dedicated directory with configuration files and detailed README
+- **Troubleshooting Section**: Common issues and solutions
+
+#### Installation Methods
+
+- **uvx Support**: One-line execution without installation (`uvx pywombat`)
+- **uv Development Mode**: Local installation for repeated use (`uv sync`, `uv run wombat`)
+
+### Changed
+
+- Improved performance with streaming lazy operations
+- Optimized parent genotype lookup (excludes 0/0 genotypes from storage)
+- Enhanced error messages for better user experience
+- Normalized chromosome names for PAR region matching (handles both 'X' and 'chrX')
+
+### Fixed
+
+- Sex column reading from pedigree file
+- Parent genotype column naming consistency (father_id/mother_id)
+- Genotype filtering to catch all partial genotypes (`./.`, `0/.`, `1/.`)
+- PAR region matching for different chromosome naming conventions
+- Empty chunk handling in output to avoid blank lines
+
+### Performance Optimizations
+
+- Delayed annotation expansion (filter before expanding `(null)` field)
+- Vectorized filtering operations (no Python loops)
+- Early genotype filtering (skip 0/0 before parent lookup)
+- Optimized parent lookup (stores only non-reference genotypes)
+- Streaming mode by default for memory efficiency
+
+### Removed
+
+- **Progress bar options**: Removed `--progress`/`--no-progress` and `--chunk-size` options for simplicity
+- **Chunked processing mode**: Simplified to use only efficient streaming mode
+
+## [0.5.0] - 2026-01-20
+
+### Added
+
+- Initial de novo mutation detection implementation
+- Pedigree file support
+- Basic quality filtering
+- Expression-based filtering
+
+### Known Issues
+
+- Progress bar had reliability issues (removed in 1.0.0)
+- Chunked processing was complex (simplified in 1.0.0)
+
+---
+
+## Release Notes
+
+### v1.0.0 - Production Ready
+
+This release marks PyWombat as production-ready for:
+
+- Rare disease gene discovery
+- De novo mutation detection in autism and developmental disorders
+- Trio and family-based variant analysis
+- High-throughput variant filtering workflows
+
+**Recommended for**: Research groups working with rare variants, de novo mutations, and family-based genomic studies.
+
+**Breaking Changes**: None from 0.5.0, but removed progress bar options for cleaner interface.
+
+---
+
+[1.0.0]: https://github.com/bourgeron-lab/pywombat/releases/tag/v1.0.0
+[0.5.0]: https://github.com/bourgeron-lab/pywombat/releases/tag/v0.5.0