exomeflow 1.0.0__tar.gz

exomeflow-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Robin Tomar, AIIMS New Delhi
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

exomeflow-1.0.0/PKG-INFO

@@ -0,0 +1,287 @@
+Metadata-Version: 2.4
+Name: exomeflow
+Version: 1.0.0
+Summary: Production-quality Whole Exome Sequencing analysis pipeline
+Author-email: Robin Tomar <robin@aiims.ac.in>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/robintomar/exomeflow
+Project-URL: Repository, https://github.com/robintomar/exomeflow
+Project-URL: Bug Tracker, https://github.com/robintomar/exomeflow/issues
+Keywords: bioinformatics,WES,NGS,genomics,exome,variant-calling
+Classifier: Development Status :: 5 - Production/Stable
+Classifier: Intended Audience :: Science/Research
+Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Operating System :: POSIX :: Linux
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer>=0.12.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: pandas>=2.0.0
+Requires-Dist: pysam>=0.22.0
+Dynamic: license-file
+
+# ExomeFlow
+
+**Production-quality Whole Exome Sequencing (WES) analysis pipeline**
+
+> Author: Robin Tomar, AIIMS New Delhi  
+> License: MIT
+
+---
+
+## Overview
+
+ExomeFlow is a Python package that wraps a complete WES analysis workflow into a single, reproducible CLI command. It handles cohort-level processing (multiple samples), checkpointing for resumable runs, structured logging, and parallel execution.
+
+```
+FASTQ
+ └─ fastp (QC + trimming)
+     └─ BWA MEM (alignment)
+         └─ GATK SortSam (coordinate sort)
+             └─ samtools flagstat (alignment QC)
+                 └─ GATK MarkDuplicates
+                     └─ GATK BuildBamIndex
+                         └─ GATK BQSR (BaseRecalibrator + ApplyBQSR)
+                             └─ GATK HaplotypeCaller (variant calling)
+                                 └─ GATK VariantFiltration (hard filters)
+                                     └─ ANNOVAR (functional annotation)
+```
+
+---
+
+## Requirements
+
+### System dependencies (must be on `PATH`)
+
+| Tool | Version tested |
+|------|---------------|
+| `bwa` | ≥ 0.7.17 |
+| `samtools` | ≥ 1.17 |
+| `gatk` | 4.6.x |
+| `fastp` | ≥ 0.23 |
+| Perl + ANNOVAR | `table_annovar.pl` |
+
+### Python
+
+- Python ≥ 3.9
+- See `requirements.txt` for Python dependencies
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install exomeflow
+```
+
+### From source
+
+```bash
+git clone https://github.com/robintomar/exomeflow.git
+cd exomeflow
+pip install -e .
+```
+
+---
+
+## Reference files required
+
+| File | Description |
+|------|-------------|
+| `hg38.fa` | BWA-indexed reference genome |
+| `dbsnp.vcf.gz` | dbSNP (bgzipped + tabix-indexed) |
+| `Mills_and_1000G_gold_standard.indels.hg38.vcf.gz` | Mills indels |
+| `Homo_sapiens_assembly38.known_indels.vcf.gz` | Known indels |
+| Exome capture BED | e.g. `Illumina_Exome_TargetedRegions_v1.2.hg38.bed` |
+| ANNOVAR humandb | `hg38` annotation databases |
+
+---
+
+## Input FASTQ naming convention
+
+ExomeFlow automatically detects samples from paired-end FASTQ files:
+
+```
+fastq/
+├── sample1_1.fastq.gz
+├── sample1_2.fastq.gz
+├── sample2_1.fastq.gz
+└── sample2_2.fastq.gz
+```
+
+Pattern: `<sample_id>_1.fastq.gz` / `<sample_id>_2.fastq.gz`
+
+---
+
+## Usage
+
+### Minimal example
+
+```bash
+exomeflow run \
+  --input-dir fastq/ \
+  --output results/ \
+  --reference /refs/hg38.fa \
+  --dbsnp /refs/dbsnp.vcf.gz \
+  --mills /refs/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \
+  --known-indels /refs/Homo_sapiens_assembly38.known_indels.vcf.gz \
+  --annovar-bin /tools/annovar \
+  --annovar-db /tools/annovar/humandb
+```
+
+### Full example with all options
+
+```bash
+exomeflow run \
+  --input-dir fastq/ \
+  --output results/ \
+  --reference /refs/hg38.fa \
+  --dbsnp /refs/dbsnp.vcf.gz \
+  --mills /refs/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \
+  --known-indels /refs/Homo_sapiens_assembly38.known_indels.vcf.gz \
+  --intervals /refs/Illumina_Exome_TargetedRegions_v1.2.hg38.bed \
+  --interval-padding 100 \
+  --annovar-bin /tools/annovar \
+  --annovar-db /tools/annovar/humandb \
+  --threads 32 \
+  --fastp-threads 8 \
+  --annovar-threads 24 \
+  --max-workers 2 \
+  --java-opts "-Xmx80g"
+```
+
+### Check version
+
+```bash
+exomeflow --version
+```
+
+### Help
+
+```bash
+exomeflow run --help
+```
+
+---
+
+## Output files
+
+After a successful run the `results/` directory contains:
+
+```
+results/
+├── QC/                          # fastp HTML/JSON reports (reserved)
+├── filtered_fastp/
+│   ├── <sample>_1_filtered.fastq.gz
+│   ├── <sample>_2_filtered.fastq.gz
+│   ├── <sample>_fastp.html
+│   └── <sample>_fastp.json
+├── Mapsam/
+│   ├── <sample>_recalibrated.bam   ← use in IGV for variant validation
+│   └── <sample>_recalibrated.bam.bai
+├── VCF/
+│   ├── <sample>.vcf                          ← raw HaplotypeCaller output
+│   ├── <sample>_PASS.vcf                     ← PASS-only hard-filtered variants
+│   ├── <sample>.annovar.hg38_multianno.vcf   ← annotated VCF
+│   └── <sample>.annovar.hg38_multianno.txt   ← annotated tab-delimited table
+├── logs/
+│   ├── analysis_<timestamp>.log   ← full pipeline log
+│   ├── errors_<timestamp>.log     ← errors only
+│   └── <sample>_<timestamp>.log   ← per-sample log
+└── .checkpoints/                  ← resume state (do not delete during a run)
+```
+
+---
+
+## Checkpointing & resuming
+
+ExomeFlow writes a checkpoint file for every completed step. If the pipeline
+is interrupted (power failure, wall-time limit, etc.) simply re-run the
+**exact same command** — completed steps are skipped automatically.
+
+---
+
+## GATK hard-filter thresholds
+
+### SNPs
+
+| Filter name | Expression |
+|-------------|-----------|
+| `SNP_LowQD` | `QD < 2.0` |
+| `SNP_StrandBias` | `FS > 60.0` |
+| `SNP_StrandOddsRatio` | `SOR > 3.0` |
+| `SNP_LowMQ` | `MQ < 40.0` |
+| `SNP_MQRankSum` | `MQRankSum < -12.5` |
+| `SNP_ReadPosRankSum` | `ReadPosRankSum < -8.0` |
+| `LowDepth` | `DP < 10` |
+| `LowGQ` *(genotype)* | `GQ < 20` |
+
+### INDELs
+
+| Filter name | Expression |
+|-------------|-----------|
+| `INDEL_LowQD` | `QD < 2.0` |
+| `INDEL_StrandBias` | `FS > 200.0` |
+| `INDEL_StrandOddsRatio` | `SOR > 10.0` |
+| `INDEL_ReadPosRankSum` | `ReadPosRankSum < -20.0` |
+| `LowDepth` | `DP < 10` |
+| `LowGQ` *(genotype)* | `GQ < 20` |
+
+---
+
+## ANNOVAR annotation databases (default)
+
+```
+refGene, dbnsfp47a, clinvar_20240416, gnomad41_exome,
+gnomad41_genome, avsnp150, cosmic84_coding, exac03
+```
+
+---
+
+## Publishing to PyPI
+
+```bash
+pip install build twine
+
+# Build source + wheel distributions
+python -m build
+
+# Upload to PyPI (requires ~/.pypirc or TWINE_USERNAME / TWINE_PASSWORD env vars)
+twine upload dist/*
+```
+
+To publish to TestPyPI first:
+
+```bash
+twine upload --repository testpypi dist/*
+pip install --index-url https://test.pypi.org/simple/ exomeflow
+```
+
+---
+
+## Development
+
+```bash
+# Install in editable mode with dev extras
+pip install -e ".[dev]"
+
+# Lint
+flake8 exomeflow/
+mypy exomeflow/
+```
+
+---
+
+## Citation
+
+If you use ExomeFlow in your research, please cite:
+
+> Robin Tomar. *ExomeFlow: a production-quality whole exome sequencing pipeline*. AIIMS New Delhi, 2025.

exomeflow-1.0.0/README.md

@@ -0,0 +1,259 @@
+# ExomeFlow
+
+**Production-quality Whole Exome Sequencing (WES) analysis pipeline**
+
+> Author: Robin Tomar, AIIMS New Delhi  
+> License: MIT
+
+---
+
+## Overview
+
+ExomeFlow is a Python package that wraps a complete WES analysis workflow into a single, reproducible CLI command. It handles cohort-level processing (multiple samples), checkpointing for resumable runs, structured logging, and parallel execution.
+
+```
+FASTQ
+ └─ fastp (QC + trimming)
+     └─ BWA MEM (alignment)
+         └─ GATK SortSam (coordinate sort)
+             └─ samtools flagstat (alignment QC)
+                 └─ GATK MarkDuplicates
+                     └─ GATK BuildBamIndex
+                         └─ GATK BQSR (BaseRecalibrator + ApplyBQSR)
+                             └─ GATK HaplotypeCaller (variant calling)
+                                 └─ GATK VariantFiltration (hard filters)
+                                     └─ ANNOVAR (functional annotation)
+```
+
+---
+
+## Requirements
+
+### System dependencies (must be on `PATH`)
+
+| Tool | Version tested |
+|------|---------------|
+| `bwa` | ≥ 0.7.17 |
+| `samtools` | ≥ 1.17 |
+| `gatk` | 4.6.x |
+| `fastp` | ≥ 0.23 |
+| Perl + ANNOVAR | `table_annovar.pl` |
+
+### Python
+
+- Python ≥ 3.9
+- See `requirements.txt` for Python dependencies
+
+---
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install exomeflow
+```
+
+### From source
+
+```bash
+git clone https://github.com/robintomar/exomeflow.git
+cd exomeflow
+pip install -e .
+```
+
+---
+
+## Reference files required
+
+| File | Description |
+|------|-------------|
+| `hg38.fa` | BWA-indexed reference genome |
+| `dbsnp.vcf.gz` | dbSNP (bgzipped + tabix-indexed) |
+| `Mills_and_1000G_gold_standard.indels.hg38.vcf.gz` | Mills indels |
+| `Homo_sapiens_assembly38.known_indels.vcf.gz` | Known indels |
+| Exome capture BED | e.g. `Illumina_Exome_TargetedRegions_v1.2.hg38.bed` |
+| ANNOVAR humandb | `hg38` annotation databases |
+
+---
+
+## Input FASTQ naming convention
+
+ExomeFlow automatically detects samples from paired-end FASTQ files:
+
+```
+fastq/
+├── sample1_1.fastq.gz
+├── sample1_2.fastq.gz
+├── sample2_1.fastq.gz
+└── sample2_2.fastq.gz
+```
+
+Pattern: `<sample_id>_1.fastq.gz` / `<sample_id>_2.fastq.gz`
+
+---
+
+## Usage
+
+### Minimal example
+
+```bash
+exomeflow run \
+  --input-dir fastq/ \
+  --output results/ \
+  --reference /refs/hg38.fa \
+  --dbsnp /refs/dbsnp.vcf.gz \
+  --mills /refs/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \
+  --known-indels /refs/Homo_sapiens_assembly38.known_indels.vcf.gz \
+  --annovar-bin /tools/annovar \
+  --annovar-db /tools/annovar/humandb
+```
+
+### Full example with all options
+
+```bash
+exomeflow run \
+  --input-dir fastq/ \
+  --output results/ \
+  --reference /refs/hg38.fa \
+  --dbsnp /refs/dbsnp.vcf.gz \
+  --mills /refs/Mills_and_1000G_gold_standard.indels.hg38.vcf.gz \
+  --known-indels /refs/Homo_sapiens_assembly38.known_indels.vcf.gz \
+  --intervals /refs/Illumina_Exome_TargetedRegions_v1.2.hg38.bed \
+  --interval-padding 100 \
+  --annovar-bin /tools/annovar \
+  --annovar-db /tools/annovar/humandb \
+  --threads 32 \
+  --fastp-threads 8 \
+  --annovar-threads 24 \
+  --max-workers 2 \
+  --java-opts "-Xmx80g"
+```
+
+### Check version
+
+```bash
+exomeflow --version
+```
+
+### Help
+
+```bash
+exomeflow run --help
+```
+
+---
+
+## Output files
+
+After a successful run the `results/` directory contains:
+
+```
+results/
+├── QC/                          # fastp HTML/JSON reports (reserved)
+├── filtered_fastp/
+│   ├── <sample>_1_filtered.fastq.gz
+│   ├── <sample>_2_filtered.fastq.gz
+│   ├── <sample>_fastp.html
+│   └── <sample>_fastp.json
+├── Mapsam/
+│   ├── <sample>_recalibrated.bam   ← use in IGV for variant validation
+│   └── <sample>_recalibrated.bam.bai
+├── VCF/
+│   ├── <sample>.vcf                          ← raw HaplotypeCaller output
+│   ├── <sample>_PASS.vcf                     ← PASS-only hard-filtered variants
+│   ├── <sample>.annovar.hg38_multianno.vcf   ← annotated VCF
+│   └── <sample>.annovar.hg38_multianno.txt   ← annotated tab-delimited table
+├── logs/
+│   ├── analysis_<timestamp>.log   ← full pipeline log
+│   ├── errors_<timestamp>.log     ← errors only
+│   └── <sample>_<timestamp>.log   ← per-sample log
+└── .checkpoints/                  ← resume state (do not delete during a run)
+```
+
+---
+
+## Checkpointing & resuming
+
+ExomeFlow writes a checkpoint file for every completed step. If the pipeline
+is interrupted (power failure, wall-time limit, etc.) simply re-run the
+**exact same command** — completed steps are skipped automatically.
+
+---
+
+## GATK hard-filter thresholds
+
+### SNPs
+
+| Filter name | Expression |
+|-------------|-----------|
+| `SNP_LowQD` | `QD < 2.0` |
+| `SNP_StrandBias` | `FS > 60.0` |
+| `SNP_StrandOddsRatio` | `SOR > 3.0` |
+| `SNP_LowMQ` | `MQ < 40.0` |
+| `SNP_MQRankSum` | `MQRankSum < -12.5` |
+| `SNP_ReadPosRankSum` | `ReadPosRankSum < -8.0` |
+| `LowDepth` | `DP < 10` |
+| `LowGQ` *(genotype)* | `GQ < 20` |
+
+### INDELs
+
+| Filter name | Expression |
+|-------------|-----------|
+| `INDEL_LowQD` | `QD < 2.0` |
+| `INDEL_StrandBias` | `FS > 200.0` |
+| `INDEL_StrandOddsRatio` | `SOR > 10.0` |
+| `INDEL_ReadPosRankSum` | `ReadPosRankSum < -20.0` |
+| `LowDepth` | `DP < 10` |
+| `LowGQ` *(genotype)* | `GQ < 20` |
+
+---
+
+## ANNOVAR annotation databases (default)
+
+```
+refGene, dbnsfp47a, clinvar_20240416, gnomad41_exome,
+gnomad41_genome, avsnp150, cosmic84_coding, exac03
+```
+
+---
+
+## Publishing to PyPI
+
+```bash
+pip install build twine
+
+# Build source + wheel distributions
+python -m build
+
+# Upload to PyPI (requires ~/.pypirc or TWINE_USERNAME / TWINE_PASSWORD env vars)
+twine upload dist/*
+```
+
+To publish to TestPyPI first:
+
+```bash
+twine upload --repository testpypi dist/*
+pip install --index-url https://test.pypi.org/simple/ exomeflow
+```
+
+---
+
+## Development
+
+```bash
+# Install in editable mode with dev extras
+pip install -e ".[dev]"
+
+# Lint
+flake8 exomeflow/
+mypy exomeflow/
+```
+
+---
+
+## Citation
+
+If you use ExomeFlow in your research, please cite:
+
+> Robin Tomar. *ExomeFlow: a production-quality whole exome sequencing pipeline*. AIIMS New Delhi, 2025.

exomeflow-1.0.0/exomeflow/__init__.py

@@ -0,0 +1,9 @@
+"""
+ExomeFlow — Whole Exome Sequencing analysis pipeline.
+
+Author: Robin Tomar, AIIMS New Delhi
+"""
+
+__version__ = "1.0.0"
+__author__ = "Robin Tomar"
+__email__ = "robin@aiims.ac.in"

exomeflow-1.0.0/exomeflow/alignment.py

@@ -0,0 +1,103 @@
+"""
+Step 2 — Read alignment with BWA MEM.
+
+Mirrors the Bash ``run_bwa_mem`` function exactly:
+  - BWA MEM flags: -Y -K 100000000
+  - Read-group tag set to sample name (ID, PU, SM, LB = sample; PL = illumina)
+  - Output piped through samtools view -Shb to produce a raw BAM
+"""
+
+from __future__ import annotations
+
+import logging
+import subprocess
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from exomeflow.utils import Checkpoint, PipelineStepError
+
+if TYPE_CHECKING:
+    from exomeflow.config import Config
+
+logger = logging.getLogger("exomeflow")
+
+STEP = "bwa"
+
+
+def run_bwa_mem(sample: str, cfg: "Config", checkpoint: Checkpoint) -> None:
+    """
+    Align filtered reads for *sample* with BWA MEM and convert to BAM.
+
+    Input
+    -----
+    <fastp_dir>/<sample>_1_filtered.fastq.gz
+    <fastp_dir>/<sample>_2_filtered.fastq.gz
+
+    Output
+    ------
+    <map_dir>/<sample>.bam
+    """
+    if checkpoint.done(sample, STEP):
+        logger.info("[%s] BWA MEM already completed, skipping.", sample)
+        return
+
+    r1 = cfg.fastp_dir / f"{sample}_1_filtered.fastq.gz"
+    r2 = cfg.fastp_dir / f"{sample}_2_filtered.fastq.gz"
+    output = cfg.map_dir / f"{sample}.bam"
+
+    read_group = (
+        f"@RG\\tID:{sample}\\tPU:{sample}\\tSM:{sample}"
+        f"\\tLB:{sample}\\tPL:illumina"
+    )
+
+    logger.info("[%s] Running BWA MEM ...", sample)
+
+    # BWA MEM → samtools view pipe (mirrors the Bash pipe)
+    bwa_cmd = [
+        "bwa", "mem",
+        "-Y",
+        "-K", "100000000",
+        "-t", str(cfg.threads),
+        "-R", read_group,
+        str(cfg.reference),
+        str(r1),
+        str(r2),
+    ]
+
+    samtools_cmd = [
+        "samtools", "view",
+        "-Shb",
+        "-o", str(output),
+        "-",
+    ]
+
+    env = cfg.env()
+
+    bwa_proc = subprocess.Popen(
+        bwa_cmd,
+        stdout=subprocess.PIPE,
+        stderr=None,
+        env=env,
+    )
+
+    samtools_proc = subprocess.Popen(
+        samtools_cmd,
+        stdin=bwa_proc.stdout,
+        env=env,
+    )
+
+    # Allow bwa to receive SIGPIPE if samtools exits early
+    if bwa_proc.stdout:
+        bwa_proc.stdout.close()
+
+    samtools_rc = samtools_proc.wait()
+    bwa_rc = bwa_proc.wait()
+
+    if bwa_rc != 0 or samtools_rc != 0:
+        raise PipelineStepError(
+            f"[{sample}] BWA MEM / samtools pipe failed "
+            f"(bwa={bwa_rc}, samtools={samtools_rc})"
+        )
+
+    checkpoint.mark(sample, STEP)
+    logger.log(25, "[%s] BWA MEM and conversion to BAM completed.", sample)