densecall 0.0.1__tar.gz

densecall-0.0.1/MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md
+include LICENCE.txt
+include requirements.txt
+recursive-include densecall/models *

densecall-0.0.1/PKG-INFO

@@ -0,0 +1,175 @@
+Metadata-Version: 2.1
+Name: densecall
+Version: 0.0.1
+Home-page: https://github.com/n-damo/densecall
+Author: linlian
+Author-email: 21620151153308@stu.xmu.edu.cn
+Description-Content-Type: text/markdown
+License-File: LICENCE.txt
+
+# DenseCall2
+
+**DenseCall2: De Novo Base-Calling of DNA Modifications Using Nanopore Sequencing**
+
+## Contents
+- [DenseCall2](#densecall2)
+  - [Contents](#contents)
+  - [Requirements](#requirements)
+    - [Hardware](#hardware)
+    - [Software](#software)
+  - [Installation](#installation)
+    - [DenseCall2](#densecall2-1)
+    - [Basecalling of FAST5/Pod5 Files](#basecalling-of-fast5pod5-files)
+  - [Usage Notes](#usage-notes)
+    - [Training Your Own Basecalling Model (Optional)](#training-your-own-basecalling-model-optional)
+  - [Knowledge distilation](#knowledge-distilation)
+  - [Downstream Analysis](#downstream-analysis)
+  - [Citing](#citing)
+  - [License](#license)
+  - [Acknowledgement](#acknowledgement)
+
+
+DenseCall2 is an updated base-caller built on an optimized Conformer architecture for nanopore signal processing, enabling simultaneous base-calling and modification detection.
+
+![image1](./doc/image1.png)
+
+## Requirements
+
+### Hardware
+- **RAM**: 2 GB minimum; 16 GB or more recommended
+- **CPU**: 4 cores minimum, ≥ 2.3 GHz per core
+- **GPU**: NVIDIA RTX 4090 or newer (required for DenseCall2)
+
+Benchmarks were collected on an ASUSTeK SVR TS700-E9-RS8 workstation  
+(Xeon Silver 4214 @ 2.20 GHz, 64 GB RAM, RTX 4090 24 GB).
+
+### Software
+
+**Supported Operating Systems**
+- Linux: Ubuntu 22.04 or newer  
+- Windows and macOS are not yet supported
+
+**Python**
+- Version 3.10 or higher is required  
+  Install on Ubuntu with:
+  ```bash
+  sudo apt update
+  sudo apt install python3 python3-pip
+  ```
+
+## Installation
+
+### DenseCall2
+
+First, set up a new environment and install the necessary Python packages using conda and pip:
+
+```shell
+# Create a new conda environment
+conda create -n densecall python=3.10 -y
+conda activate densecall
+
+# Upgrade pip
+pip install --upgrade pip
+
+# Download and install 
+git clone https://github.com/LuChenLab/DENSECALL2.git
+cd DENSECALL2
+pip install -r requirements.txt
+pip install flash-attn==2.8.3 --no-build-isolation --no-cache-dir
+python setup.py develop
+```
+
+DenseCall2 is compatible with the basecaller of ont-bonito, allowing our trained models to be used for the basecalling process. Install ont-bonito as follows:
+
+```shell
+cd ont-bonito-0.7.3
+python setup.py develop
+```
+
+### Basecalling of FAST5\/Pod5 Files
+
+After installing DenseCall2, download the pre-trained models for human-specific models from [Pre-trained Basecalling Models](https://figshare.com/articles/dataset/Densecall_models/25712856). Available models include `dna_r9.4.1_hac_m5C@v1.0.tar.gz` for r9.4.1 data and `dna_r10.4.1_hac_m5C@v1.0.tar.gz` for r10.4.1 data.
+
+DenseCall2 provides a method for transforming `.fast5` files into `.fastq` format or `.sam` format. Follow the commands below to perform basecalling:
+
+```shell
+# Activate the DenseCall2 conda environment
+conda activate densecall2
+
+# Navigate to the directory where you want to download the models
+cd /path/to/Densecall2/densecall/models/
+
+# Download and extract the models
+# Note: Ensure you have already downloaded the .tar.gz files to this directory
+tar -xzvf dna_r9.4.1_hac_m5C@v1.0.tar.gz 
+
+# Perform basecalling on the .fast5 files to generate .fastq files 
+densecall basecaller dna_r9.4.1_hac_m5C@v1.0 /path/to/fast5_data/ --mod --chunksize 12000 --overlap 600 --reference chr22.mmi --recursive --alignment-threads 12 > mod.sam
+```
+
+If you are using the tool solely for standard basecalling, you can omit the --mod flag.
+```
+densecall basecaller dna_r9.4.1_hac_m5C@v1.0 /path/to/fast5_data/ --chunksize 12000 --overlap 600  > mod.fastq
+```
+
+## Usage Notes
+
+- **Modified-base calling**  
+  Add `--mod` together with `--reference` and ensure the output file has a `.sam` extension.  
+  DenseCall2 will perform Viterbi decoding and append MM/ML tags to the SAM output.
+
+- **Standard base calling**  
+  Omit `--mod` and set the output extension to `.fastq` or `.fq`.  
+  DenseCall2 will use beam-search decoding.
+
+### Training Your Own Basecalling Model (Optional)
+
+`densecall train` - train a DenseCall2 model
+
+To train a model using your own reads, first get a trained model from [Remora](https://github.com/nanoporetech/remora).
+
+```shell
+remora model download 
+```
+
+```shell
+densecall basecaller dna_r10.4.1_e8.2_400bps_hac@v3.5.2 ./chr1_fast5 --batchsize 64 --chunksize 5000  --overlap 100 \
+--reference chr1.mmi --recursive --save-ctc --min-accuracy-save-ctc 0.9 \
+--alphabet NACZGT \
+--modified-codes Z \
+--modified-base-model /path/to/dna_r10.4.1_e8.2_400bps_hac_v3.5.1_5mc_CG_v2.pt \
+--max-reads 100000 > r10_train_data/test.sam
+```
+
+Training a new model from scratch:
+
+```bash
+densecall train test --directory r10_train_data/ -f --batch 64 --epochs 30 \
+--no-quantile-grad-clip --lr 0.002 --alphabet NACZGT \
+--config conformer.toml --new --compile
+```
+
+
+## Knowledge distilation 
+To train a model using knowledge distilation, add the `--teacher` flag to the training command.
+
+```bash
+densecall train r10_student  --directory r10_train_data/ -f --batch 64  --epochs 20  --grad-accum-split 2 --no-quantile-grad-clip --lr 0.002    --alphabet NACZGT --config conformer_fast.toml   --new --compile --teacher r10_teacher/
+```
+
+All training calls use Automatic Mixed Precision to speed up training.
+
+## Downstream Analysis
+
+The results were analyzed using the ONT tool [modkit](https://github.com/nanoporetech/modkit), which processes BAM files containing MM/ML tags to generate comprehensive statistical reports. This study specifically employed modkit's "validate" and "pileup" functions.
+
+## Citing
+
+A pre-print will be uploaded soon.
+
+## License
+
+GNU General Public License v3.0
+
+## Acknowledgement
+We thank Bonito for making its source code available. DenseCall2 was built on Bonito’s framework: the save-CTC module, the training pipeline and the conversion of Conformer-based basecall outputs to base sequences have all been modified from Bonito’s original implementation. Bonito’s licence can be found [here](https://github.com/nanoporetech/bonito/blob/master/LICENCE.txt).

densecall-0.0.1/README.md

@@ -0,0 +1,166 @@
+# DenseCall2
+
+**DenseCall2: De Novo Base-Calling of DNA Modifications Using Nanopore Sequencing**
+
+## Contents
+- [DenseCall2](#densecall2)
+  - [Contents](#contents)
+  - [Requirements](#requirements)
+    - [Hardware](#hardware)
+    - [Software](#software)
+  - [Installation](#installation)
+    - [DenseCall2](#densecall2-1)
+    - [Basecalling of FAST5/Pod5 Files](#basecalling-of-fast5pod5-files)
+  - [Usage Notes](#usage-notes)
+    - [Training Your Own Basecalling Model (Optional)](#training-your-own-basecalling-model-optional)
+  - [Knowledge distilation](#knowledge-distilation)
+  - [Downstream Analysis](#downstream-analysis)
+  - [Citing](#citing)
+  - [License](#license)
+  - [Acknowledgement](#acknowledgement)
+
+
+DenseCall2 is an updated base-caller built on an optimized Conformer architecture for nanopore signal processing, enabling simultaneous base-calling and modification detection.
+
+![image1](./doc/image1.png)
+
+## Requirements
+
+### Hardware
+- **RAM**: 2 GB minimum; 16 GB or more recommended
+- **CPU**: 4 cores minimum, ≥ 2.3 GHz per core
+- **GPU**: NVIDIA RTX 4090 or newer (required for DenseCall2)
+
+Benchmarks were collected on an ASUSTeK SVR TS700-E9-RS8 workstation  
+(Xeon Silver 4214 @ 2.20 GHz, 64 GB RAM, RTX 4090 24 GB).
+
+### Software
+
+**Supported Operating Systems**
+- Linux: Ubuntu 22.04 or newer  
+- Windows and macOS are not yet supported
+
+**Python**
+- Version 3.10 or higher is required  
+  Install on Ubuntu with:
+  ```bash
+  sudo apt update
+  sudo apt install python3 python3-pip
+  ```
+
+## Installation
+
+### DenseCall2
+
+First, set up a new environment and install the necessary Python packages using conda and pip:
+
+```shell
+# Create a new conda environment
+conda create -n densecall python=3.10 -y
+conda activate densecall
+
+# Upgrade pip
+pip install --upgrade pip
+
+# Download and install 
+git clone https://github.com/LuChenLab/DENSECALL2.git
+cd DENSECALL2
+pip install -r requirements.txt
+pip install flash-attn==2.8.3 --no-build-isolation --no-cache-dir
+python setup.py develop
+```
+
+DenseCall2 is compatible with the basecaller of ont-bonito, allowing our trained models to be used for the basecalling process. Install ont-bonito as follows:
+
+```shell
+cd ont-bonito-0.7.3
+python setup.py develop
+```
+
+### Basecalling of FAST5\/Pod5 Files
+
+After installing DenseCall2, download the pre-trained models for human-specific models from [Pre-trained Basecalling Models](https://figshare.com/articles/dataset/Densecall_models/25712856). Available models include `dna_r9.4.1_hac_m5C@v1.0.tar.gz` for r9.4.1 data and `dna_r10.4.1_hac_m5C@v1.0.tar.gz` for r10.4.1 data.
+
+DenseCall2 provides a method for transforming `.fast5` files into `.fastq` format or `.sam` format. Follow the commands below to perform basecalling:
+
+```shell
+# Activate the DenseCall2 conda environment
+conda activate densecall2
+
+# Navigate to the directory where you want to download the models
+cd /path/to/Densecall2/densecall/models/
+
+# Download and extract the models
+# Note: Ensure you have already downloaded the .tar.gz files to this directory
+tar -xzvf dna_r9.4.1_hac_m5C@v1.0.tar.gz 
+
+# Perform basecalling on the .fast5 files to generate .fastq files 
+densecall basecaller dna_r9.4.1_hac_m5C@v1.0 /path/to/fast5_data/ --mod --chunksize 12000 --overlap 600 --reference chr22.mmi --recursive --alignment-threads 12 > mod.sam
+```
+
+If you are using the tool solely for standard basecalling, you can omit the --mod flag.
+```
+densecall basecaller dna_r9.4.1_hac_m5C@v1.0 /path/to/fast5_data/ --chunksize 12000 --overlap 600  > mod.fastq
+```
+
+## Usage Notes
+
+- **Modified-base calling**  
+  Add `--mod` together with `--reference` and ensure the output file has a `.sam` extension.  
+  DenseCall2 will perform Viterbi decoding and append MM/ML tags to the SAM output.
+
+- **Standard base calling**  
+  Omit `--mod` and set the output extension to `.fastq` or `.fq`.  
+  DenseCall2 will use beam-search decoding.
+
+### Training Your Own Basecalling Model (Optional)
+
+`densecall train` - train a DenseCall2 model
+
+To train a model using your own reads, first get a trained model from [Remora](https://github.com/nanoporetech/remora).
+
+```shell
+remora model download 
+```
+
+```shell
+densecall basecaller dna_r10.4.1_e8.2_400bps_hac@v3.5.2 ./chr1_fast5 --batchsize 64 --chunksize 5000  --overlap 100 \
+--reference chr1.mmi --recursive --save-ctc --min-accuracy-save-ctc 0.9 \
+--alphabet NACZGT \
+--modified-codes Z \
+--modified-base-model /path/to/dna_r10.4.1_e8.2_400bps_hac_v3.5.1_5mc_CG_v2.pt \
+--max-reads 100000 > r10_train_data/test.sam
+```
+
+Training a new model from scratch:
+
+```bash
+densecall train test --directory r10_train_data/ -f --batch 64 --epochs 30 \
+--no-quantile-grad-clip --lr 0.002 --alphabet NACZGT \
+--config conformer.toml --new --compile
+```
+
+
+## Knowledge distilation 
+To train a model using knowledge distilation, add the `--teacher` flag to the training command.
+
+```bash
+densecall train r10_student  --directory r10_train_data/ -f --batch 64  --epochs 20  --grad-accum-split 2 --no-quantile-grad-clip --lr 0.002    --alphabet NACZGT --config conformer_fast.toml   --new --compile --teacher r10_teacher/
+```
+
+All training calls use Automatic Mixed Precision to speed up training.
+
+## Downstream Analysis
+
+The results were analyzed using the ONT tool [modkit](https://github.com/nanoporetech/modkit), which processes BAM files containing MM/ML tags to generate comprehensive statistical reports. This study specifically employed modkit's "validate" and "pileup" functions.
+
+## Citing
+
+A pre-print will be uploaded soon.
+
+## License
+
+GNU General Public License v3.0
+
+## Acknowledgement
+We thank Bonito for making its source code available. DenseCall2 was built on Bonito’s framework: the save-CTC module, the training pipeline and the conversion of Conformer-based basecall outputs to base sequences have all been modified from Bonito’s original implementation. Bonito’s licence can be found [here](https://github.com/nanoporetech/bonito/blob/master/LICENCE.txt).

densecall-0.0.1/densecall/__init__.py

@@ -0,0 +1,37 @@
+from argparse import ArgumentDefaultsHelpFormatter, ArgumentParser
+from densecall.cli import train, basecaller, accuracy
+import os
+
+modules = [
+    'accuracy', 'train','basecaller'
+]
+
+__version__ = '0.0.1'
+
+
+
+
+def main():
+    parser = ArgumentParser(
+        'densecall',
+        formatter_class=ArgumentDefaultsHelpFormatter
+    )
+
+    parser.add_argument(
+        '-v', '--version', action='version',
+        version='%(prog)s {}'.format(__version__)
+    )
+
+    subparsers = parser.add_subparsers(
+        title='subcommands', description='valid commands',
+        help='additional help', dest='command'
+    )
+    subparsers.required = True
+
+    for module in modules:
+        mod = globals()[module]
+        p = subparsers.add_parser(module, parents=[mod.argparser()])
+        p.set_defaults(func=mod.main)
+
+    args = parser.parse_args()
+    args.func(args)

densecall-0.0.1/densecall/__main__.py

@@ -0,0 +1,2 @@
+from densecall import main
+main()

densecall-0.0.1/densecall/aligner.py

@@ -0,0 +1,83 @@
+"""
+modified based on Bonito Aligner
+"""
+
+from threading import Thread
+from functools import partial
+import mappy 
+import numpy as np
+from mappy import Aligner, ThreadBuffer
+import sys 
+from densecall.multiprocessing import ThreadMap
+from densecall.mod_util import call_mods, load_mods_model
+
+def align_map(aligner, sequences, n_thread=4, mod_model=None, use_reference_anchored=False):
+    """
+    Align `sequences` with minimap using `n_thread` threads.
+    """
+    return ThreadMap(partial(MappyWorker, aligner, mod_model, use_reference_anchored), sequences, n_thread)
+
+
+class ManagedThreadBuffer:
+    """
+    Minimap2 ThreadBuffer that is periodically reallocated.
+    """
+    def __init__(self, max_uses=20):
+        self.max_uses = max_uses
+        self.uses = 0
+        self._b = ThreadBuffer()
+
+    @property
+    def buffer(self):
+        if self.uses > self.max_uses:
+            self._b = ThreadBuffer()
+            self.uses = 0
+        self.uses += 1
+        return self._b
+
+
+class MappyWorker(Thread):
+    """
+    Process that reads items from an input_queue, applies a func to them and puts them on an output_queue
+    """
+    def __init__(self, aligner, mod_model, use_reference_anchored, input_queue=None, output_queue=None):
+        super().__init__()
+        self.aligner = aligner
+        self.input_queue = input_queue
+        self.output_queue = output_queue
+        self.mod_model = mod_model
+        self.use_reference_anchored = use_reference_anchored
+
+    def run(self):
+        thrbuf = ManagedThreadBuffer()
+        while True:
+            item = self.input_queue.get()
+            if item is StopIteration:
+                self.output_queue.put(item)
+                break
+            k, v = item
+
+            mapping = next(self.aligner.map(v['sequence'], buf=thrbuf.buffer, MD=True), None)
+            
+            if self.mod_model and mapping:
+                
+                refseq = self.aligner.seq(mapping.ctg, mapping.r_st, mapping.r_en)
+                cigar = np.array(mapping.cigar)
+                cigar = cigar[:, [1, 0]].tolist()
+                if mapping.strand == -1:
+                    refseq = mappy.revcomp(refseq)
+                    cigar = cigar[::-1]
+                    
+                v['refseq'] = refseq
+                v['cigar'] = cigar
+                v['modified_results'] = []
+                for mod_model, code in self.mod_model:
+                    remora_model, remora_metadata = mod_model
+                    can_base = remora_metadata['can_base']
+                    v = call_mods(mod_model, k, v, use_reference_anchored=self.use_reference_anchored)
+                    v['modified_results'].append((can_base, code, v['modified_result']))
+
+            
+            self.output_queue.put((k, {**v, 'mapping': mapping}))
+
+