pywombat 1.0.0__tar.gz → 1.0.2__tar.gz

pywombat-1.0.2/.github/copilot-instructions.md

@@ -0,0 +1,130 @@
+# PyWombat Development Guide
+
+## Project Overview
+
+PyWombat is a bioinformatics CLI tool that transforms bcftools tabulated TSV files into analysis-ready long-format data. It specializes in:
+- Expanding VCF INFO fields from `(null)` column into individual columns
+- Converting wide-format sample data to long-format (melting)
+- Extracting and calculating genotype metrics (GT, DP, GQ, AD, VAF)
+- De novo mutation (DNM) detection with sex-chromosome awareness
+- Trio/family analysis with pedigree-based parent genotype joining
+
+## Architecture
+
+### Single-File Monolith
+All functionality lives in [src/pywombat/cli.py](src/pywombat/cli.py) (~2100 lines). This is intentional for deployment simplicity with `uvx` (one-command installation-free execution).
+
+### Core Data Flow
+1. **Input**: bcftools tabulated TSV (wide format, one row per variant)
+2. **Transform Pipeline**:
+   - `format_bcftools_tsv_lazy()` → Expand INFO, melt samples, parse genotypes
+   - `apply_filters_lazy()` → Apply quality/expression filters OR DNM detection
+3. **Output**: Long-format TSV/Parquet (one row per variant-sample)
+
+### Key Functions
+- **`format_bcftools_tsv()`**: Eager transformation (collects data)
+- **`format_bcftools_tsv_lazy()`**: Streaming wrapper using Polars LazyFrame
+- **`apply_filters_lazy()`**: Conditional filter dispatcher (quality/expression OR DNM)
+- **`apply_de_novo_filter()`**: Complex vectorized DNM logic with PAR region handling
+- **`parse_impact_filter_expression()`**: Expression parser for flexible YAML-based filtering
+
+## Development Workflow
+
+### Setup & Running
+```bash
+# Install dependencies (uses uv package manager)
+uv sync
+
+# Run CLI locally
+uv run wombat input.tsv -o output
+
+# Production usage (no installation)
+uvx pywombat input.tsv -o output
+```
+
+### Testing
+- Test data: `tests/C0733-011-068.*` files (real variant data)
+- Compare script: [tests/compare_dnm_results.py](tests/compare_dnm_results.py) validates DNM detection
+- No formal test framework yet—manual validation against reference outputs in `tests/check/`
+
+## Critical Domain Logic
+
+### Genotype Parsing
+- Input format: `GT:DP:GQ:AD` (e.g., `0/1:25:99:10,15`)
+- Extracted fields:
+  - `sample_gt`: Genotype (0/1, 1/1, etc.)
+  - `sample_dp`: Total depth (25)
+  - `sample_gq`: Quality (99)
+  - `sample_ad`: **Second value** from AD (15 = alternate allele depth)
+  - `sample_vaf`: Calculated as `sample_ad / sample_dp` (0.6)
+
+### De Novo Mutation Detection
+Sex-chromosome-aware logic in `apply_de_novo_filter()`:
+- **Autosomes**: Both parents must have VAF < 2% (reference genotype)
+- **X chromosome (male proband)**: Mother must be reference; father is hemizygous (ignored)
+- **Y chromosome**: Only father checked (mother doesn't have Y)
+- **PAR regions**: Treated as autosomal (both parents checked)
+- **Hemizygous variants** (X/Y in males): Require VAF ≥ 85% (not ~50% like het)
+
+PAR regions defined in [examples/de_novo_mutations.yml](examples/de_novo_mutations.yml):
+```yaml
+par_regions:
+  GRCh38:
+    PAR1: { chrom: "X", start: 10001, end: 2781479 }
+    PAR2: { chrom: "X", start: 155701383, end: 156030895 }
+```
+
+## Conventions & Patterns
+
+### Polars Usage
+- **Lazy evaluation**: Use `scan_csv()` → `LazyFrame` → `sink_csv()` for streaming
+- **Streaming mode**: `collect(streaming=True)` for memory-efficient processing
+- **Schema overrides**: Force string type for sample IDs to prevent numeric inference:
+  ```python
+  schema_overrides = {col: pl.Utf8 for col in ["sample", "FatherBarcode", ...]}
+  ```
+
+### YAML Configuration Files
+Filter configs in [examples/](examples/) define:
+- `quality`: Thresholds (DP, GQ, VAF ranges for het/hom)
+- `expression`: Polars-compatible filter expressions (see `parse_impact_filter_expression()`)
+- `dnm`: De novo detection parameters (parent thresholds, PAR regions)
+
+Example expression syntax:
+```yaml
+expression: "VEP_IMPACT = HIGH & fafmax_faf95_max_genomes <= 0.001"
+```
+Supports: `=`, `!=`, `<=`, `>=`, `<`, `>`, `&`, `|`, `()`, NULL, NaN
+
+### Error Handling
+- Use `click.echo(..., err=True)` for verbose messages (stderr)
+- Raise `click.Abort()` instead of generic exceptions for CLI errors
+- Validate pedigree/config requirements early (e.g., DNM mode requires pedigree)
+
+## Common Pitfalls
+
+1. **VAF Calculation**: Always use `sample_ad / sample_dp`, not first AD value
+2. **Genotype Filtering**: Use `str.contains("1")` not regex—handles 0/1, 1/0, 1/1, 1/2
+3. **Parent Joining**: Pedigree must have `sample_id`, `FatherBarcode`, `MotherBarcode` columns
+4. **Sex Normalization**: DNM filter accepts "1"/"2" or "M"/"F" for sex; normalizes to uppercase
+5. **Categorical Types**: Convert `#CHROM` from Categorical to Utf8 before string operations
+
+## File References
+
+- Main CLI: [src/pywombat/cli.py](src/pywombat/cli.py)
+- Config examples: [examples/de_novo_mutations.yml](examples/de_novo_mutations.yml), [examples/rare_variants_high_impact.yml](examples/rare_variants_high_impact.yml)
+- Test data: [tests/C0733-011-068.pedigree.tsv](tests/C0733-011-068.pedigree.tsv)
+- Documentation: [README.md](README.md), [QUICKSTART.md](QUICKSTART.md)
+
+## Adding New Features
+
+When adding filters:
+1. Update YAML schema in config examples
+2. Extend `apply_filters_lazy()` or create specialized function (like `apply_de_novo_filter()`)
+3. Use lazy operations when possible; collect only if vectorization requires it
+4. Add verbose logging with `--verbose` flag support
+
+When modifying transforms:
+- Keep `format_bcftools_tsv()` and `format_bcftools_tsv_lazy()` in sync
+- Test with real data from `tests/` directory
+- Verify output with `compare_dnm_results.py` for DNM changes

{pywombat-1.0.0 → pywombat-1.0.2}/CHANGELOG.md

@@ -5,6 +5,16 @@ All notable changes to PyWombat will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [1.0.1] - 2026-01-24
+
+### Added
+
+- **Boolean Flag Support in INFO Fields**: INFO field entries without `=` signs (e.g., `PASS`, `DB`, `SOMATIC`) are now extracted as boolean columns with `True`/`False` values instead of being ignored. This enables filtering on VCF flag fields.
+
+### Fixed
+
+- INFO field parsing now handles all field types correctly, including standalone boolean flags commonly used in VCF files.
+
 ## [1.0.0] - 2026-01-23
 
 First stable release of PyWombat! 🎉

{pywombat-1.0.0 → pywombat-1.0.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: pywombat
-Version: 1.0.0
+Version: 1.0.2
 Summary: A CLI tool for processing and filtering bcftools tabulated TSV files with pedigree support
 Project-URL: Homepage, https://github.com/bourgeron-lab/pywombat
 Project-URL: Repository, https://github.com/bourgeron-lab/pywombat
@@ -35,6 +35,7 @@ A high-performance CLI tool for processing and filtering bcftools tabulated TSV
 🧬 **De Novo Detection**: Sex-chromosome-aware DNM identification  
 📊 **Flexible Output**: TSV, compressed TSV, or Parquet formats  
 🎯 **Expression Filters**: Complex filtering with logical expressions  
+🏷️ **Boolean Flag Support**: INFO field flags (PASS, DB, etc.) extracted as True/False columns  
 ⚡ **Streaming Mode**: Memory-efficient processing of large files
 
 ---
@@ -77,7 +78,7 @@ uv run wombat input.tsv -o output
 
 PyWombat transforms bcftools tabulated TSV files into analysis-ready formats by:
 
-1. **Expanding the `(null)` INFO column**: Extracts all `NAME=value` fields (e.g., `DP=30;AF=0.5;AC=2`) into separate columns
+1. **Expanding the `(null)` INFO column**: Extracts all `NAME=value` fields (e.g., `DP=30;AF=0.5;AC=2`) and boolean flags (e.g., `PASS`, `DB`) into separate columns
 2. **Melting sample columns**: Converts wide-format sample data into long format with one row per variant-sample combination
 3. **Extracting genotype data**: Parses `GT:DP:GQ:AD` format into separate columns with calculated VAF
 4. **Adding parent data**: Joins father/mother genotypes when pedigree is provided
@@ -88,20 +89,22 @@ PyWombat transforms bcftools tabulated TSV files into analysis-ready formats by:
 **Input (Wide Format):**
 
 ```tsv
-#CHROM  POS  REF  ALT  (null)              Sample1:GT:DP:GQ:AD  Sample2:GT:DP:GQ:AD
-chr1    100  A    T    DP=30;AF=0.5;AC=2   0/1:15:99:5,10      1/1:18:99:0,18
+#CHROM  POS  REF  ALT  (null)                      Sample1:GT:DP:GQ:AD  Sample2:GT:DP:GQ:AD
+chr1    100  A    T    DP=30;AF=0.5;PASS;AC=2      0/1:15:99:5,10      1/1:18:99:0,18
 ```
 
 **Output (Long Format):**
 
 ```tsv
-#CHROM  POS  REF  ALT  AC  AF   DP  sample   sample_gt  sample_dp  sample_gq  sample_ad  sample_vaf
-chr1    100  A    T    2   0.5  30  Sample1  0/1        15         99         10         0.6667
-chr1    100  A    T    2   0.5  30  Sample2  1/1        18         99         18         1.0
+#CHROM  POS  REF  ALT  AC  AF   DP  PASS  sample   sample_gt  sample_dp  sample_gq  sample_ad  sample_vaf
+chr1    100  A    T    2   0.5  30  true  Sample1  0/1        15         99         10         0.6667
+chr1    100  A    T    2   0.5  30  true  Sample2  1/1        18         99         18         1.0
 ```
 
 **Generated Columns:**
 
+- INFO fields with `=`: Extracted as separate columns (e.g., `DP`, `AF`, `AC`)
+- INFO boolean flags: Extracted as True/False columns (e.g., `PASS`, `DB`, `SOMATIC`)
 - `sample`: Sample identifier
 - `sample_gt`: Genotype (e.g., 0/1, 1/1)
 - `sample_dp`: Read depth (total coverage)

{pywombat-1.0.0 → pywombat-1.0.2}/README.md

@@ -13,6 +13,7 @@ A high-performance CLI tool for processing and filtering bcftools tabulated TSV
 🧬 **De Novo Detection**: Sex-chromosome-aware DNM identification  
 📊 **Flexible Output**: TSV, compressed TSV, or Parquet formats  
 🎯 **Expression Filters**: Complex filtering with logical expressions  
+🏷️ **Boolean Flag Support**: INFO field flags (PASS, DB, etc.) extracted as True/False columns  
 ⚡ **Streaming Mode**: Memory-efficient processing of large files
 
 ---
@@ -55,7 +56,7 @@ uv run wombat input.tsv -o output
 
 PyWombat transforms bcftools tabulated TSV files into analysis-ready formats by:
 
-1. **Expanding the `(null)` INFO column**: Extracts all `NAME=value` fields (e.g., `DP=30;AF=0.5;AC=2`) into separate columns
+1. **Expanding the `(null)` INFO column**: Extracts all `NAME=value` fields (e.g., `DP=30;AF=0.5;AC=2`) and boolean flags (e.g., `PASS`, `DB`) into separate columns
 2. **Melting sample columns**: Converts wide-format sample data into long format with one row per variant-sample combination
 3. **Extracting genotype data**: Parses `GT:DP:GQ:AD` format into separate columns with calculated VAF
 4. **Adding parent data**: Joins father/mother genotypes when pedigree is provided
@@ -66,20 +67,22 @@ PyWombat transforms bcftools tabulated TSV files into analysis-ready formats by:
 **Input (Wide Format):**
 
 ```tsv
-#CHROM  POS  REF  ALT  (null)              Sample1:GT:DP:GQ:AD  Sample2:GT:DP:GQ:AD
-chr1    100  A    T    DP=30;AF=0.5;AC=2   0/1:15:99:5,10      1/1:18:99:0,18
+#CHROM  POS  REF  ALT  (null)                      Sample1:GT:DP:GQ:AD  Sample2:GT:DP:GQ:AD
+chr1    100  A    T    DP=30;AF=0.5;PASS;AC=2      0/1:15:99:5,10      1/1:18:99:0,18
 ```
 
 **Output (Long Format):**
 
 ```tsv
-#CHROM  POS  REF  ALT  AC  AF   DP  sample   sample_gt  sample_dp  sample_gq  sample_ad  sample_vaf
-chr1    100  A    T    2   0.5  30  Sample1  0/1        15         99         10         0.6667
-chr1    100  A    T    2   0.5  30  Sample2  1/1        18         99         18         1.0
+#CHROM  POS  REF  ALT  AC  AF   DP  PASS  sample   sample_gt  sample_dp  sample_gq  sample_ad  sample_vaf
+chr1    100  A    T    2   0.5  30  true  Sample1  0/1        15         99         10         0.6667
+chr1    100  A    T    2   0.5  30  true  Sample2  1/1        18         99         18         1.0
 ```
 
 **Generated Columns:**
 
+- INFO fields with `=`: Extracted as separate columns (e.g., `DP`, `AF`, `AC`)
+- INFO boolean flags: Extracted as True/False columns (e.g., `PASS`, `DB`, `SOMATIC`)
 - `sample`: Sample identifier
 - `sample_gt`: Genotype (e.g., 0/1, 1/1)
 - `sample_dp`: Read depth (total coverage)

{pywombat-1.0.0 → pywombat-1.0.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "pywombat"
-version = "1.0.0"
+version = "1.0.2"
 description = "A CLI tool for processing and filtering bcftools tabulated TSV files with pedigree support"
 readme = "README.md"
 authors = [{ name = "Freddy Cliquet", email = "fcliquet@pasteur.fr" }]

{pywombat-1.0.0 → pywombat-1.0.2}/src/pywombat/cli.py

@@ -1198,15 +1198,18 @@ def read_pedigree(pedigree_path: Path) -> pl.DataFrame:
     pedigree_df = df.select(select_cols)
 
     # Replace 0 and -9 with null (indicating no parent)
+    # Explicit cast to Utf8 ensures type is preserved even when all values become null
     pedigree_df = pedigree_df.with_columns(
         [
             pl.when(pl.col("father_id").cast(pl.Utf8).is_in(["0", "-9"]))
             .then(None)
             .otherwise(pl.col("father_id"))
+            .cast(pl.Utf8)
             .alias("father_id"),
             pl.when(pl.col("mother_id").cast(pl.Utf8).is_in(["0", "-9"]))
             .then(None)
             .otherwise(pl.col("mother_id"))
+            .cast(pl.Utf8)
             .alias("mother_id"),
         ]
     )
@@ -1313,6 +1316,10 @@ def format_expand_annotations(df: pl.DataFrame) -> pl.DataFrame:
     This is a separate step that can be applied after filtering to avoid
     expensive annotation expansion on variants that will be filtered out.
 
+    Handles two types of INFO fields:
+    - Key-value pairs (e.g., "DP=30") -> extracted as string values
+    - Boolean flags (e.g., "PASS", "DB") -> created as True/False columns
+
     Args:
         df: DataFrame with (null) column
 
@@ -1324,9 +1331,10 @@ def format_expand_annotations(df: pl.DataFrame) -> pl.DataFrame:
         # Already expanded or missing - return as-is
         return df
 
-    # Extract all unique field names from the (null) column
+    # Extract all unique field names and flags from the (null) column
     null_values = df.select("(null)").to_series()
     all_fields = set()
+    all_flags = set()
 
     for value in null_values:
         if value and not (isinstance(value, float)):  # Skip null/NaN values
@@ -1335,8 +1343,10 @@ def format_expand_annotations(df: pl.DataFrame) -> pl.DataFrame:
                 if "=" in pair:
                     field_name = pair.split("=", 1)[0]
                     all_fields.add(field_name)
+                elif pair.strip():  # Boolean flag (no '=')
+                    all_flags.add(pair.strip())
 
-    # Create expressions to extract each field
+    # Create expressions to extract each key-value field
     for field in sorted(all_fields):
         # Extract the field value from the (null) column
         # Pattern: extract value after "field=" and before ";" or end of string
@@ -1344,6 +1354,14 @@ def format_expand_annotations(df: pl.DataFrame) -> pl.DataFrame:
             pl.col("(null)").str.extract(f"{field}=([^;]+)").alias(field)
         )
 
+    # Create boolean columns for flags
+    for flag in sorted(all_flags):
+        # Check if flag appears in the (null) column (as whole word)
+        # Use regex to match flag as a separate field (not part of another field name)
+        df = df.with_columns(
+            pl.col("(null)").str.contains(f"(^|;){flag}(;|$)").alias(flag)
+        )
+
     # Drop the original (null) column
     df = df.drop("(null)")