xengsort 2.2.1__tar.gz → 2.2.2__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {xengsort-2.2.1/xengsort.egg-info → xengsort-2.2.2}/PKG-INFO +32 -21
- {xengsort-2.2.1 → xengsort-2.2.2}/README.md +31 -20
- {xengsort-2.2.1 → xengsort-2.2.2}/pyproject.toml +1 -1
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/cptasks_kmers.py +25 -4
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/hash_s3c_fbcbvb.py +16 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/fastaio.py +7 -3
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/fastqio.py +10 -7
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/generalio.py +12 -1
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/hashio.py +7 -1
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/kmers.py +5 -4
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/libc.py +16 -2
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/srhash.py +3 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_classify.py +7 -7
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_index.py +6 -3
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_info.py +3 -3
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_main.py +9 -2
- {xengsort-2.2.1 → xengsort-2.2.2/xengsort.egg-info}/PKG-INFO +32 -21
- {xengsort-2.2.1 → xengsort-2.2.2}/LICENSE +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/MANIFEST.in +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/setup.cfg +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/tests/test_trimmed_reads.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/dnaencode.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/dnaencode_fast.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/fastcash_info.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/fastcash_main.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/fastcash_optimize.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/hash_minimizer.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/hash_srh_fbcbvb.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/binaryio.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/seqio.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/textio.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/xorio.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/aligned_arrays.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/bitarray.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/conpro.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/debug.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/intbitarray.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/llvm.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/numbautils.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/packedarray.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/mask.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/mathutils.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/parameters.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/subtable_hashfunctions.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/values/xenograft.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/config/classify.yaml +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/config/index.yaml +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_load.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_remove.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_weak.py +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/SOURCES.txt +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/dependency_links.txt +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/entry_points.txt +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/requires.txt +0 -0
- {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: xengsort
|
|
3
|
-
Version: 2.2.
|
|
3
|
+
Version: 2.2.2
|
|
4
4
|
Summary: xenograft sorting of FASTQ reads into five categories: host, graft, both, neither, ambiguous.
|
|
5
5
|
Author-email: Jens Zentgraf <zentgraf@cs.uni-saarland.de>, Sven Rahmann <rahmann@cs.uni-saarland.de>
|
|
6
6
|
License: MIT License
|
|
@@ -52,6 +52,7 @@ This tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the
|
|
|
52
52
|
For a description of the method an evaluation on several datasets, please see our
|
|
53
53
|
[article in "Algorithms for Molecular Biology"](https://almob.biomedcentral.com/articles/10.1186/s13015-021-00181-w), or the initial [WABI 2020 publication](https://drops.dagstuhl.de/opus/volltexte/2020/12793/).
|
|
54
54
|
(There was also a [preprint on bioRxiv](https://www.biorxiv.org/content/10.1101/2020.05.14.095604v2).)
|
|
55
|
+
If you use xengsort to sort converted reads (RRBS or WGBS) there is a [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2026.10.06.756952v1.abstract) available.
|
|
55
56
|
BibTeX entrys for citation can be found at the end of this README file.
|
|
56
57
|
|
|
57
58
|
In case of problems,
|
|
@@ -71,24 +72,23 @@ Thank you!
|
|
|
71
72
|
|
|
72
73
|
We here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
|
|
73
74
|
This tutorial uses the workflow management system `Snakemake` to run the example.
|
|
74
|
-
This is, however, not necessary to use `xengsort`.
|
|
75
|
+
This is, however, not necessary to use `xengsort`.
|
|
75
76
|
Please refer to the Usage Guide below to learn how to run `xengsort` on your own data manually.
|
|
76
77
|
|
|
77
78
|
|
|
78
79
|
Our software `xengsort` is provided as a Python package.
|
|
79
|
-
For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
|
|
80
|
+
For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
|
|
80
81
|
We here use the package manager [conda](https://docs.conda.io/en/latest/) to manage Python packages and to use a separate environment for `xengsort`.
|
|
81
82
|
|
|
82
83
|
|
|
83
|
-
### Install the conda package manager (
|
|
84
|
-
Go to https://
|
|
85
|
-
Choose
|
|
86
|
-
Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
|
|
84
|
+
### Install the conda package manager (conda-forge)
|
|
85
|
+
Go to https://conda-forge.org/download/ and download the conda-forge installer:
|
|
86
|
+
Choose your operating system, follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
|
|
87
87
|
You can let the installer do it, or do it manually by editing your ``.bashrc`` or similar file under Linux or MacOS, or by editing the environment variables under Windows.
|
|
88
88
|
To verify that the installation works, open a new terminal and execute
|
|
89
89
|
```
|
|
90
|
-
conda --version # ideally
|
|
91
|
-
python --version # ideally 3.
|
|
90
|
+
conda --version # ideally 26.9.xx or higher
|
|
91
|
+
python --version # ideally 3.12.xx or higher
|
|
92
92
|
```
|
|
93
93
|
|
|
94
94
|
|
|
@@ -122,7 +122,7 @@ within the cloned directory tree.
|
|
|
122
122
|
|
|
123
123
|
|
|
124
124
|
### Create and activate a conda environment
|
|
125
|
-
To run our software, a [conda](https://
|
|
125
|
+
To run our software, a [conda](https://conda-forge.org/docs/) environment with the required libraries needs to be created.
|
|
126
126
|
A list of needed libraries is provided in the ``environment.yml`` file in the cloned repository;
|
|
127
127
|
it can be used to create a new environment:
|
|
128
128
|
|
|
@@ -133,7 +133,7 @@ conda env create
|
|
|
133
133
|
which will create an environment named ``xengsort`` with the required dependencies,
|
|
134
134
|
using the provided ``environment.yml`` file in the same directory.
|
|
135
135
|
|
|
136
|
-
A more explicit alternative is to run
|
|
136
|
+
A more explicit alternative is to run
|
|
137
137
|
```
|
|
138
138
|
conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
|
|
139
139
|
```
|
|
@@ -217,11 +217,11 @@ To build an index for xengsort, several parameters must be provided, which are d
|
|
|
217
217
|
First, a file name and a path for the index must be chosen.
|
|
218
218
|
The index is stored in two files. We will use `myindex` to store the index in the current folder.
|
|
219
219
|
|
|
220
|
-
Second, two reference genomes (host and graft) must be provided (in FASTA files).
|
|
221
|
-
We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
|
|
220
|
+
Second, two reference genomes (host and graft) must be provided (in FASTA files).
|
|
221
|
+
We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
|
|
222
222
|
The files can be provided as an uncompressed file or compressed using `gzip`, `bzip2` or `xz`.
|
|
223
|
-
The corresponding options are `-H` or `--host` and `-G` or `--graft`.
|
|
224
|
-
Each option can take several arguments as files.
|
|
223
|
+
The corresponding options are `-H` or `--host` and `-G` or `--graft`.
|
|
224
|
+
Each option can take several arguments as files.
|
|
225
225
|
```
|
|
226
226
|
xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000 [OPTIONS]
|
|
227
227
|
```
|
|
@@ -249,7 +249,7 @@ The final important parameter is about parallelization:
|
|
|
249
249
|
|
|
250
250
|
Most of the parameters can also be provided in a config file (`.yaml`):
|
|
251
251
|
- `--cfg` or `--config` defines the path the the config file.
|
|
252
|
-
|
|
252
|
+
|
|
253
253
|
### How to classify
|
|
254
254
|
|
|
255
255
|
To classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
|
|
@@ -257,7 +257,7 @@ Then run the `xengsort classify` command with a previously built index, such as
|
|
|
257
257
|
```
|
|
258
258
|
xengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode count
|
|
259
259
|
```
|
|
260
|
-
for single-end reads, or
|
|
260
|
+
for single-end reads, or
|
|
261
261
|
```
|
|
262
262
|
xengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode count
|
|
263
263
|
```
|
|
@@ -275,7 +275,7 @@ The parameter `--prefix` or equivalently `--out` is required and defines the pre
|
|
|
275
275
|
- `{prefix}.both.1.fq.gz`: reads that could originate from both
|
|
276
276
|
- `{prefix}.neither.1.fq.gz`: reads that originate from neither host nor graft
|
|
277
277
|
- `{prefix}.ambiguous.1.fq.gz`: (few) ambiguous reads that cannot be classified,
|
|
278
|
-
|
|
278
|
+
|
|
279
279
|
and similarly with `.2.fq.gz`. For single-end reads, there is only `.fq.gz` (no numbers).
|
|
280
280
|
|
|
281
281
|
The compression type can be specified using the `--compression` parameter.
|
|
@@ -301,7 +301,7 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 9_000_000_000 --b
|
|
|
301
301
|
If an index is build using the bisulfite parameter, also the read is converted in the classification step.
|
|
302
302
|
It is not necessary to change the parameters of the classification step.
|
|
303
303
|
Currently, it only supports the count-based classification.
|
|
304
|
-
|
|
304
|
+
A detailed description on how the benchmarks on bisulfite sequencing data was performed can be found [here](https://github.com/S-Altend/MethSortBench/tree/main/03_read_sorting).
|
|
305
305
|
|
|
306
306
|
Please report a [GitLab issue](https://gitlab.com/genomeinformatics/xengsort/-/issues) in case of problems.
|
|
307
307
|
|
|
@@ -315,13 +315,13 @@ Happy sorting!
|
|
|
315
315
|
* **Snakemake throws errors when attempting to download the example dataset from SRA**.
|
|
316
316
|
|
|
317
317
|
This may happen for several reasons:
|
|
318
|
-
1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
|
|
318
|
+
1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
|
|
319
319
|
1. Another source of trouble may be that for some reason and older version of `sra-tools` gets called. We use the `fasterq-dump` tool to download the data, which did not exist in older verisons. To use a clean environment with a recent version of `sra-tools`, you can run snakemake as `snakemake -j 16 --use-conda`, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).
|
|
320
320
|
1. In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (`xengsort --version`). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and run `pip install -e .`, as explained above in the tutorial.
|
|
321
321
|
|
|
322
322
|
* **Indexing stops after a few seconds with a failure.**
|
|
323
323
|
|
|
324
|
-
The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
|
|
324
|
+
The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
|
|
325
325
|
(From version 1.0.1, the `-n` parameter is required, and the error cannot occur anymore. Please update to the latest version!)
|
|
326
326
|
For the human and mouse genomes this is approximately `-n 4_500_000_000` (4.5 billion).
|
|
327
327
|
It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
|
|
@@ -377,3 +377,14 @@ In addition, you may also cite the WABI 2020 proceedigs paper:
|
|
|
377
377
|
annote = {Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
|
|
378
378
|
}
|
|
379
379
|
```
|
|
380
|
+
|
|
381
|
+
If you use xengsort to sort converted reads, please also cite our corresponding bioRxiv preprint:
|
|
382
|
+
```
|
|
383
|
+
@article {Altendorfer2026.10.06.756952,
|
|
384
|
+
author = {Altendorfer, Stefan and Zentgraf, Jens and Schmitz, Johanna Elena and Plass, Christoph and Gerh{\"a}user, Clarissa and Rahmann, Sven},
|
|
385
|
+
title = {High-performance xenograft read-sorting for DNA methylation analysis},
|
|
386
|
+
year = {2026},
|
|
387
|
+
doi = {10.64898/2026.10.06.756952},
|
|
388
|
+
journal = {bioRxiv}
|
|
389
|
+
}
|
|
390
|
+
```
|
|
@@ -5,6 +5,7 @@ This tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the
|
|
|
5
5
|
For a description of the method an evaluation on several datasets, please see our
|
|
6
6
|
[article in "Algorithms for Molecular Biology"](https://almob.biomedcentral.com/articles/10.1186/s13015-021-00181-w), or the initial [WABI 2020 publication](https://drops.dagstuhl.de/opus/volltexte/2020/12793/).
|
|
7
7
|
(There was also a [preprint on bioRxiv](https://www.biorxiv.org/content/10.1101/2020.05.14.095604v2).)
|
|
8
|
+
If you use xengsort to sort converted reads (RRBS or WGBS) there is a [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2026.10.06.756952v1.abstract) available.
|
|
8
9
|
BibTeX entrys for citation can be found at the end of this README file.
|
|
9
10
|
|
|
10
11
|
In case of problems,
|
|
@@ -24,24 +25,23 @@ Thank you!
|
|
|
24
25
|
|
|
25
26
|
We here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
|
|
26
27
|
This tutorial uses the workflow management system `Snakemake` to run the example.
|
|
27
|
-
This is, however, not necessary to use `xengsort`.
|
|
28
|
+
This is, however, not necessary to use `xengsort`.
|
|
28
29
|
Please refer to the Usage Guide below to learn how to run `xengsort` on your own data manually.
|
|
29
30
|
|
|
30
31
|
|
|
31
32
|
Our software `xengsort` is provided as a Python package.
|
|
32
|
-
For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
|
|
33
|
+
For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
|
|
33
34
|
We here use the package manager [conda](https://docs.conda.io/en/latest/) to manage Python packages and to use a separate environment for `xengsort`.
|
|
34
35
|
|
|
35
36
|
|
|
36
|
-
### Install the conda package manager (
|
|
37
|
-
Go to https://
|
|
38
|
-
Choose
|
|
39
|
-
Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
|
|
37
|
+
### Install the conda package manager (conda-forge)
|
|
38
|
+
Go to https://conda-forge.org/download/ and download the conda-forge installer:
|
|
39
|
+
Choose your operating system, follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
|
|
40
40
|
You can let the installer do it, or do it manually by editing your ``.bashrc`` or similar file under Linux or MacOS, or by editing the environment variables under Windows.
|
|
41
41
|
To verify that the installation works, open a new terminal and execute
|
|
42
42
|
```
|
|
43
|
-
conda --version # ideally
|
|
44
|
-
python --version # ideally 3.
|
|
43
|
+
conda --version # ideally 26.9.xx or higher
|
|
44
|
+
python --version # ideally 3.12.xx or higher
|
|
45
45
|
```
|
|
46
46
|
|
|
47
47
|
|
|
@@ -75,7 +75,7 @@ within the cloned directory tree.
|
|
|
75
75
|
|
|
76
76
|
|
|
77
77
|
### Create and activate a conda environment
|
|
78
|
-
To run our software, a [conda](https://
|
|
78
|
+
To run our software, a [conda](https://conda-forge.org/docs/) environment with the required libraries needs to be created.
|
|
79
79
|
A list of needed libraries is provided in the ``environment.yml`` file in the cloned repository;
|
|
80
80
|
it can be used to create a new environment:
|
|
81
81
|
|
|
@@ -86,7 +86,7 @@ conda env create
|
|
|
86
86
|
which will create an environment named ``xengsort`` with the required dependencies,
|
|
87
87
|
using the provided ``environment.yml`` file in the same directory.
|
|
88
88
|
|
|
89
|
-
A more explicit alternative is to run
|
|
89
|
+
A more explicit alternative is to run
|
|
90
90
|
```
|
|
91
91
|
conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
|
|
92
92
|
```
|
|
@@ -170,11 +170,11 @@ To build an index for xengsort, several parameters must be provided, which are d
|
|
|
170
170
|
First, a file name and a path for the index must be chosen.
|
|
171
171
|
The index is stored in two files. We will use `myindex` to store the index in the current folder.
|
|
172
172
|
|
|
173
|
-
Second, two reference genomes (host and graft) must be provided (in FASTA files).
|
|
174
|
-
We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
|
|
173
|
+
Second, two reference genomes (host and graft) must be provided (in FASTA files).
|
|
174
|
+
We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
|
|
175
175
|
The files can be provided as an uncompressed file or compressed using `gzip`, `bzip2` or `xz`.
|
|
176
|
-
The corresponding options are `-H` or `--host` and `-G` or `--graft`.
|
|
177
|
-
Each option can take several arguments as files.
|
|
176
|
+
The corresponding options are `-H` or `--host` and `-G` or `--graft`.
|
|
177
|
+
Each option can take several arguments as files.
|
|
178
178
|
```
|
|
179
179
|
xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000 [OPTIONS]
|
|
180
180
|
```
|
|
@@ -202,7 +202,7 @@ The final important parameter is about parallelization:
|
|
|
202
202
|
|
|
203
203
|
Most of the parameters can also be provided in a config file (`.yaml`):
|
|
204
204
|
- `--cfg` or `--config` defines the path the the config file.
|
|
205
|
-
|
|
205
|
+
|
|
206
206
|
### How to classify
|
|
207
207
|
|
|
208
208
|
To classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
|
|
@@ -210,7 +210,7 @@ Then run the `xengsort classify` command with a previously built index, such as
|
|
|
210
210
|
```
|
|
211
211
|
xengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode count
|
|
212
212
|
```
|
|
213
|
-
for single-end reads, or
|
|
213
|
+
for single-end reads, or
|
|
214
214
|
```
|
|
215
215
|
xengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode count
|
|
216
216
|
```
|
|
@@ -228,7 +228,7 @@ The parameter `--prefix` or equivalently `--out` is required and defines the pre
|
|
|
228
228
|
- `{prefix}.both.1.fq.gz`: reads that could originate from both
|
|
229
229
|
- `{prefix}.neither.1.fq.gz`: reads that originate from neither host nor graft
|
|
230
230
|
- `{prefix}.ambiguous.1.fq.gz`: (few) ambiguous reads that cannot be classified,
|
|
231
|
-
|
|
231
|
+
|
|
232
232
|
and similarly with `.2.fq.gz`. For single-end reads, there is only `.fq.gz` (no numbers).
|
|
233
233
|
|
|
234
234
|
The compression type can be specified using the `--compression` parameter.
|
|
@@ -254,7 +254,7 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 9_000_000_000 --b
|
|
|
254
254
|
If an index is build using the bisulfite parameter, also the read is converted in the classification step.
|
|
255
255
|
It is not necessary to change the parameters of the classification step.
|
|
256
256
|
Currently, it only supports the count-based classification.
|
|
257
|
-
|
|
257
|
+
A detailed description on how the benchmarks on bisulfite sequencing data was performed can be found [here](https://github.com/S-Altend/MethSortBench/tree/main/03_read_sorting).
|
|
258
258
|
|
|
259
259
|
Please report a [GitLab issue](https://gitlab.com/genomeinformatics/xengsort/-/issues) in case of problems.
|
|
260
260
|
|
|
@@ -268,13 +268,13 @@ Happy sorting!
|
|
|
268
268
|
* **Snakemake throws errors when attempting to download the example dataset from SRA**.
|
|
269
269
|
|
|
270
270
|
This may happen for several reasons:
|
|
271
|
-
1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
|
|
271
|
+
1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
|
|
272
272
|
1. Another source of trouble may be that for some reason and older version of `sra-tools` gets called. We use the `fasterq-dump` tool to download the data, which did not exist in older verisons. To use a clean environment with a recent version of `sra-tools`, you can run snakemake as `snakemake -j 16 --use-conda`, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).
|
|
273
273
|
1. In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (`xengsort --version`). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and run `pip install -e .`, as explained above in the tutorial.
|
|
274
274
|
|
|
275
275
|
* **Indexing stops after a few seconds with a failure.**
|
|
276
276
|
|
|
277
|
-
The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
|
|
277
|
+
The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
|
|
278
278
|
(From version 1.0.1, the `-n` parameter is required, and the error cannot occur anymore. Please update to the latest version!)
|
|
279
279
|
For the human and mouse genomes this is approximately `-n 4_500_000_000` (4.5 billion).
|
|
280
280
|
It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
|
|
@@ -330,3 +330,14 @@ In addition, you may also cite the WABI 2020 proceedigs paper:
|
|
|
330
330
|
annote = {Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
|
|
331
331
|
}
|
|
332
332
|
```
|
|
333
|
+
|
|
334
|
+
If you use xengsort to sort converted reads, please also cite our corresponding bioRxiv preprint:
|
|
335
|
+
```
|
|
336
|
+
@article {Altendorfer2026.10.06.756952,
|
|
337
|
+
author = {Altendorfer, Stefan and Zentgraf, Jens and Schmitz, Johanna Elena and Plass, Christoph and Gerh{\"a}user, Clarissa and Rahmann, Sven},
|
|
338
|
+
title = {High-performance xenograft read-sorting for DNA methylation analysis},
|
|
339
|
+
year = {2026},
|
|
340
|
+
doi = {10.64898/2026.10.06.756952},
|
|
341
|
+
journal = {bioRxiv}
|
|
342
|
+
}
|
|
343
|
+
```
|
|
@@ -544,15 +544,27 @@ def compile_cptask_insert_filtered_subkeys(
|
|
|
544
544
|
myfilter,
|
|
545
545
|
constant_value=0,
|
|
546
546
|
maxfailures=0,
|
|
547
|
-
maxwalk=1000
|
|
547
|
+
maxwalk=1000,
|
|
548
|
+
prefetch_distance=8):
|
|
548
549
|
debugprint0, debugprint1, debugprint2 = debug.debugprint
|
|
549
550
|
dassert = debug.dassert
|
|
551
|
+
|
|
552
|
+
@njit(nogil=True)
|
|
553
|
+
def dummy_prefetch(*_):
|
|
554
|
+
return
|
|
555
|
+
|
|
556
|
+
prefetch_filter = prefetch_table = dummy_prefetch
|
|
550
557
|
if myfilter is not None:
|
|
551
558
|
wwo = "with"
|
|
552
559
|
lookup_and_insert_in_subfilter = myfilter.private.lookup_and_insert_in_subfilter
|
|
560
|
+
table_prefetch_distance = 0
|
|
561
|
+
filter_prefetch_distance = prefetch_distance
|
|
562
|
+
prefetch_filter = myfilter.private.prefetch_block_by_subkey_subfilter
|
|
553
563
|
else:
|
|
554
564
|
wwo = "without"
|
|
555
|
-
|
|
565
|
+
table_prefetch_distance = prefetch_distance
|
|
566
|
+
filter_prefetch_distance = 0
|
|
567
|
+
prefetch_table = myhashtable.private.prefetch_subkey
|
|
556
568
|
@njit(nogil=True)
|
|
557
569
|
def lookup_and_insert_in_subfilter(ft, st, subkey):
|
|
558
570
|
return True
|
|
@@ -599,7 +611,12 @@ def compile_cptask_insert_filtered_subkeys(
|
|
|
599
611
|
# dassert(buffer_type == 1, __name__, 'buffer_type == 1', buffer_type, message='0 means sequence buffer')
|
|
600
612
|
|
|
601
613
|
# consume the buffer
|
|
602
|
-
for
|
|
614
|
+
for ii in range(nsubkeys):
|
|
615
|
+
subkey = active_buffer[ii]
|
|
616
|
+
if filter_prefetch_distance and ii + filter_prefetch_distance < nsubkeys:
|
|
617
|
+
prefetch_filter(ft, active_buffer[ii + prefetch_distance], st)
|
|
618
|
+
if table_prefetch_distance and ii + table_prefetch_distance < nsubkeys:
|
|
619
|
+
prefetch_table(ht, active_buffer[ii + prefetch_distance], st)
|
|
603
620
|
if lookup_and_insert_in_subfilter(ft, st, subkey):
|
|
604
621
|
status, result = update(ht, st, subkey, value) # see docstring below
|
|
605
622
|
if status & 128 == 0:
|
|
@@ -618,9 +635,11 @@ def compile_cptask_insert_filtered_subkeys(
|
|
|
618
635
|
|
|
619
636
|
def compile_cptask_update_existing_subkeys(
|
|
620
637
|
myhashtable,
|
|
621
|
-
constant_value=1
|
|
638
|
+
constant_value=1,
|
|
639
|
+
prefetch_distance=8):
|
|
622
640
|
debugprint0, debugprint1, debugprint2 = debug.debugprint
|
|
623
641
|
update_existing = myhashtable.private.update_existing_ssk
|
|
642
|
+
prefetch = myhashtable.private.prefetch_subkey
|
|
624
643
|
value = uint64(constant_value)
|
|
625
644
|
debugprint1(f"- compiling cptask_update_existing_subkeys: {constant_value=}")
|
|
626
645
|
|
|
@@ -652,6 +671,8 @@ def compile_cptask_update_existing_subkeys(
|
|
|
652
671
|
|
|
653
672
|
# consume the buffer
|
|
654
673
|
for ii in range(nsubkeys):
|
|
674
|
+
if prefetch_distance and ii + prefetch_distance < nsubkeys:
|
|
675
|
+
prefetch(ht, active_buffer[ii + prefetch_distance], st)
|
|
655
676
|
subkey = active_buffer[ii]
|
|
656
677
|
_, _ = update_existing(ht, st, subkey, value) # see docstring below
|
|
657
678
|
# status, result = update_existing(ht, st, subkey, value) # see docstring below
|
|
@@ -134,6 +134,22 @@ def build_hash(universe, n, subtables, bucketsize,
|
|
|
134
134
|
def prefetch_bucket(table, subtable, bucket):
|
|
135
135
|
startbit = subtable * subtablebits + bucket * bucketsizebits
|
|
136
136
|
hprefetch(table, startbit)
|
|
137
|
+
hprefetch(table, startbit + uint64(bucketsizebits - 1))
|
|
138
|
+
|
|
139
|
+
@njit(nogil=True, locals=dict(
|
|
140
|
+
bucket=int64, startbit=uint64))
|
|
141
|
+
def prefetch_subkey(table, subkey, subtable):
|
|
142
|
+
bucket, _ = get_bs1(subkey)
|
|
143
|
+
startbit = subtable * subtablebits + bucket * bucketsizebits
|
|
144
|
+
hprefetch(table, startbit)
|
|
145
|
+
|
|
146
|
+
bucket, _ = get_bs2(subkey)
|
|
147
|
+
startbit = subtable * subtablebits + bucket * bucketsizebits
|
|
148
|
+
hprefetch(table, startbit)
|
|
149
|
+
|
|
150
|
+
bucket, _ = get_bs3(subkey)
|
|
151
|
+
startbit = subtable * subtablebits + bucket * bucketsizebits
|
|
152
|
+
hprefetch(table, startbit)
|
|
137
153
|
|
|
138
154
|
@njit(nogil=True, locals=dict(
|
|
139
155
|
bucket=int64, startbit=uint64))
|
|
@@ -103,6 +103,7 @@ def cptask_read_fasta_into_buffers(fd, W, inbuffers, incontrol, ininfos, outbuff
|
|
|
103
103
|
inside_seq = True
|
|
104
104
|
header_seq = False
|
|
105
105
|
nseq = 0
|
|
106
|
+
last_seq_length = 2 * W
|
|
106
107
|
while True:
|
|
107
108
|
old = nactive
|
|
108
109
|
old_buffer = active_buffer
|
|
@@ -134,7 +135,11 @@ def cptask_read_fasta_into_buffers(fd, W, inbuffers, incontrol, ininfos, outbuff
|
|
|
134
135
|
if ntodo < 0:
|
|
135
136
|
errorcode = ntodo
|
|
136
137
|
break
|
|
138
|
+
# If we have a long header but a seq smaller W, the offset can be bigger than W
|
|
139
|
+
if header_seq and last_seq_length < W:
|
|
140
|
+
break
|
|
137
141
|
# Nothing read and the offset is not a full (gapped) k-mer
|
|
142
|
+
# I think this should trigger
|
|
138
143
|
if offset < W:
|
|
139
144
|
break
|
|
140
145
|
# Nothing read but we still have some data in the buffer
|
|
@@ -157,7 +162,6 @@ def cptask_read_fasta_into_buffers(fd, W, inbuffers, incontrol, ininfos, outbuff
|
|
|
157
162
|
offset = (W - 1)
|
|
158
163
|
else:
|
|
159
164
|
offset = end_buffer - linemarkbuffers[nactive, nseq - 1, 3]
|
|
160
|
-
|
|
161
165
|
outcontrol[nactive, 7] = nseq
|
|
162
166
|
outcontrol[nactive, 6] = 0 # buffer type sequence data
|
|
163
167
|
|
|
@@ -417,7 +421,7 @@ def fastaextract(args):
|
|
|
417
421
|
seps = args.separators
|
|
418
422
|
sfx = [args.suffix] if args.suffix else []
|
|
419
423
|
seps = [_SEPARATORS.get(sep.upper(), sep) for sep in seps]
|
|
420
|
-
if items is None:
|
|
424
|
+
if items is None:
|
|
421
425
|
items = list()
|
|
422
426
|
seps = list()
|
|
423
427
|
if len(seps) == 1:
|
|
@@ -436,7 +440,7 @@ def fastaextract(args):
|
|
|
436
440
|
def get_header_info(header, items, assigner, seps):
|
|
437
441
|
fields = header.decode("ascii").split()
|
|
438
442
|
assigners = [i for (i, field) in enumerate(fields) if assigner in field]
|
|
439
|
-
if 0 in assigners:
|
|
443
|
+
if 0 in assigners:
|
|
440
444
|
assigners.remove(0)
|
|
441
445
|
D = dict()
|
|
442
446
|
if items is None:
|
|
@@ -4,7 +4,7 @@ import numpy as np
|
|
|
4
4
|
from numba import njit, int32, int64, uint64
|
|
5
5
|
|
|
6
6
|
from .seqio import FormatError, _universal_reads
|
|
7
|
-
from ..lowlevel.libc import read_block
|
|
7
|
+
from ..lowlevel.libc import read_block, _memchr
|
|
8
8
|
from ..lowlevel.conpro import \
|
|
9
9
|
find_buffer_for_writing, \
|
|
10
10
|
mark_buffer_for_reading, mark_buffer_for_writing
|
|
@@ -604,6 +604,7 @@ def _find_fastq_seqmarks(buf, linemarks, sequence_id=-1):
|
|
|
604
604
|
if n == 0:
|
|
605
605
|
return (0, 0)
|
|
606
606
|
M = linemarks.shape[0]
|
|
607
|
+
start_ptr = buf.ctypes.data
|
|
607
608
|
i = 0
|
|
608
609
|
m = -1 # number of current record we're in
|
|
609
610
|
nxt = 0 # byte of beginning of last record
|
|
@@ -633,13 +634,15 @@ def _find_fastq_seqmarks(buf, linemarks, sequence_id=-1):
|
|
|
633
634
|
linemarks[m, 2] = i
|
|
634
635
|
elif line == 1:
|
|
635
636
|
linemarks[m, 0] = i
|
|
637
|
+
|
|
636
638
|
# find end of current line
|
|
637
|
-
|
|
638
|
-
|
|
639
|
-
|
|
640
|
-
|
|
641
|
-
|
|
642
|
-
|
|
639
|
+
next_ptr = _memchr(start_ptr + i, 10, n - i)
|
|
640
|
+
if next_ptr == 0:
|
|
641
|
+
# we did not find the end of the line before the buffer was exhausted
|
|
642
|
+
# we cannot set linemarks[m][1]
|
|
643
|
+
return (m, nxt)
|
|
644
|
+
i = next_ptr - start_ptr
|
|
645
|
+
|
|
643
646
|
if line == 1:
|
|
644
647
|
linemarks[m, 1] = i
|
|
645
648
|
line = (line + 1) % 4
|
|
@@ -19,9 +19,18 @@ from .fastaio import \
|
|
|
19
19
|
from .binaryio import \
|
|
20
20
|
compile_cptask_read_data_into_buffers
|
|
21
21
|
|
|
22
|
+
igzip_installed = False
|
|
23
|
+
if which("igzip") is not None:
|
|
24
|
+
igzip_installed = True
|
|
25
|
+
|
|
26
|
+
_gz_decompress = "igzip -d -c" if igzip_installed else "bgzip -cd"
|
|
22
27
|
|
|
23
28
|
_in_suffixes = {
|
|
24
|
-
".gz": "bgzip -cd",
|
|
29
|
+
# ".gz": "bgzip -cd",
|
|
30
|
+
# ".gz": "rapidgzip -d -c",
|
|
31
|
+
# ".gz": "rapidgzip-rust",
|
|
32
|
+
# ".gz": "igzip -d -c",
|
|
33
|
+
".gz": _gz_decompress,
|
|
25
34
|
".bz2": "bzip2 -cd",
|
|
26
35
|
".xz": "xz -cd",
|
|
27
36
|
".zst": "zstd -cd",
|
|
@@ -34,6 +43,8 @@ _out_suffixes = {
|
|
|
34
43
|
}
|
|
35
44
|
_threads = {
|
|
36
45
|
"bgzip": "@",
|
|
46
|
+
"rapidgzip": "P",
|
|
47
|
+
"rapidgzip-rust": "P",
|
|
37
48
|
"xz": "T",
|
|
38
49
|
"zstd": "T",
|
|
39
50
|
}
|
|
@@ -54,7 +54,7 @@ from ..lowlevel.intbitarray import intbitarray
|
|
|
54
54
|
from ..lowlevel.libc import write_block, write_uint64, write_dna
|
|
55
55
|
|
|
56
56
|
|
|
57
|
-
_ATTRS_TO_SAVE = "hashtype choices nvalues subtables aligned universe nslots=n nbuckets bucketsize nfingerprints mem_bytes shortcutbits hashfuncs".split()
|
|
57
|
+
_ATTRS_TO_SAVE = "hashtype choices nvalues subtables aligned universe nslots=n nbuckets bucketsize windowsize nfingerprints mem_bytes shortcutbits hashfuncs".split()
|
|
58
58
|
_ATTR_PAIRS = [((name if '=' not in name else name.split("=")[0]), (name if '=' not in name else name.split("=")[1])) for name in _ATTRS_TO_SAVE]
|
|
59
59
|
# TODO: see comment in save_hash on how to get rid of these
|
|
60
60
|
|
|
@@ -219,6 +219,12 @@ def load_hash(filename, *, shared_memory=False, info_only=False, quiet=False):
|
|
|
219
219
|
bucketsize, hashfuncs, nvalues, update_value,
|
|
220
220
|
aligned=aligned, nfingerprints=nfingerprints,
|
|
221
221
|
init=init, maxwalk=maxwalk, shm=shm)
|
|
222
|
+
elif hashtype == 'ccrh':
|
|
223
|
+
h = m.build_hash(universe, n, subtables,
|
|
224
|
+
bucketsize, hashfuncs, nvalues, update_value,
|
|
225
|
+
aligned=aligned, nfingerprints=nfingerprints,
|
|
226
|
+
init=init, maxwalk=maxwalk, shortcutbits=shortcutbits,
|
|
227
|
+
windowsize=hashinfo['windowsize'], shm=shm)
|
|
222
228
|
else:
|
|
223
229
|
h = m.build_hash(universe, n, subtables,
|
|
224
230
|
bucketsize, hashfuncs, nvalues, update_value,
|
|
@@ -605,6 +605,7 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
|
|
|
605
605
|
w = mask.w
|
|
606
606
|
if k < 1 or k > 32:
|
|
607
607
|
raise ValueError(f"only 1<=k<=32 is supported, but k={k}.")
|
|
608
|
+
|
|
608
609
|
revcomp, ccode = compile_revcomp_and_canonical_code_new(k, rcmode) if new_encoding else compile_revcomp_and_canonical_code_old(k, rcmode)
|
|
609
610
|
invalid = uint64(np.iinfo(np.uint64).max)
|
|
610
611
|
popcnt = compile_popcount('uint64')
|
|
@@ -686,7 +687,7 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
|
|
|
686
687
|
contained_ns = contained_ns >> 1
|
|
687
688
|
|
|
688
689
|
# compute gapped k-mers iteratively based on previous gapped k-mers
|
|
689
|
-
for i in range(
|
|
690
|
+
for i in range(end_naive, (end - start) - (width - 1)):
|
|
690
691
|
contained_ns = contained_ns << 1
|
|
691
692
|
code = 0
|
|
692
693
|
for k, shift, mask in steps:
|
|
@@ -701,13 +702,13 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
|
|
|
701
702
|
@njit(nogil=True)
|
|
702
703
|
def canonical_codes_inplace(arr, nhashes):
|
|
703
704
|
for i in range(nhashes):
|
|
704
|
-
arr[i] = ccode(arr[i])
|
|
705
|
+
arr[i] = ccode(arr[i]) if arr[i] != invalid else invalid
|
|
705
706
|
return nhashes
|
|
706
707
|
|
|
707
708
|
@njit(nogil=True)
|
|
708
709
|
def canonical_codes_sampled(arr, nhashes):
|
|
709
710
|
for idx, hash_idx in enumerate(range(0, nhashes, w // 2)):
|
|
710
|
-
arr[idx] = ccode(arr[
|
|
711
|
+
arr[idx] = ccode(arr[i]) if arr[i] != invalid else invalid
|
|
711
712
|
return idx + 1
|
|
712
713
|
|
|
713
714
|
canonical_codes = canonical_codes_sampled if sampled else canonical_codes_inplace
|
|
@@ -716,7 +717,7 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
|
|
|
716
717
|
def get_gapped_kmers(seq, hashes):
|
|
717
718
|
hashes[:] = 0
|
|
718
719
|
issh_processor_fast(hashes, seq, 0, len(seq))
|
|
719
|
-
return canonical_codes(hashes, len(seq) - w + 1)
|
|
720
|
+
return canonical_codes(hashes, max(0, len(seq) - w + 1))
|
|
720
721
|
|
|
721
722
|
@njit(nogil=True)
|
|
722
723
|
def get_gapped_kmers_forward(seq, hashes):
|
|
@@ -49,6 +49,10 @@ def _setup_libc():
|
|
|
49
49
|
_libc.write.argtypes = (ctypes.c_int, ctypes.c_void_p, ctypes.c_size_t)
|
|
50
50
|
_write = _libc.write
|
|
51
51
|
|
|
52
|
+
_libc.memchr.restype = ctypes.c_ssize_t
|
|
53
|
+
_libc.memchr.argtypes = (ctypes.c_size_t, ctypes.c_int, ctypes.c_size_t)
|
|
54
|
+
_memchr = _libc.memchr
|
|
55
|
+
|
|
52
56
|
if 'linux' in platform:
|
|
53
57
|
_libc.__errno_location.argtypes = tuple()
|
|
54
58
|
_libc.__errno_location.restype = ctypes.POINTER(ctypes.c_int)
|
|
@@ -61,7 +65,7 @@ def _setup_libc():
|
|
|
61
65
|
raise NotImplementedError(msg)
|
|
62
66
|
|
|
63
67
|
_errno = njit(nogil=True)(lambda: _errno_location()[0])
|
|
64
|
-
return _libc, _read, _write, _errno
|
|
68
|
+
return _libc, _read, _write, _memchr, _errno
|
|
65
69
|
|
|
66
70
|
else:
|
|
67
71
|
"""
|
|
@@ -70,7 +74,17 @@ def _setup_libc():
|
|
|
70
74
|
raise NotImplementedError(msg)
|
|
71
75
|
|
|
72
76
|
|
|
73
|
-
libc, _read, _write, errno = _setup_libc()
|
|
77
|
+
libc, _read, _write, _memchr, errno = _setup_libc()
|
|
78
|
+
|
|
79
|
+
|
|
80
|
+
@njit(nogil=True)
|
|
81
|
+
def next_char_at(buf, c):
|
|
82
|
+
start_ptr = buf.ctypes.data
|
|
83
|
+
nbytes = buf.shape[0]
|
|
84
|
+
res = _memchr(start_ptr, c, nbytes)
|
|
85
|
+
if res == 0:
|
|
86
|
+
return -1
|
|
87
|
+
return res - start_ptr
|
|
74
88
|
|
|
75
89
|
|
|
76
90
|
@njit(nogil=True, locals=dict()) # offset=int32, buflen=int32, space=int32, res=int32, fill=int32, err=int32))
|
|
@@ -34,6 +34,7 @@ SRHash = namedtuple("SRHash", [
|
|
|
34
34
|
"subtables",
|
|
35
35
|
"nbuckets",
|
|
36
36
|
"bucketsize",
|
|
37
|
+
"windowsize", # window size for windowed hash types (hash_ccrh); 0 if unused
|
|
37
38
|
"nfingerprints",
|
|
38
39
|
"nvalues",
|
|
39
40
|
"mem_bytes",
|
|
@@ -93,6 +94,7 @@ SRHash_private = namedtuple("SRHash_private", [
|
|
|
93
94
|
"get_subkey_choice_from_bucket_signature",
|
|
94
95
|
"prefetch_bucket",
|
|
95
96
|
"prefetch_key",
|
|
97
|
+
"prefetch_subkey",
|
|
96
98
|
|
|
97
99
|
# Copies of the public functions but with subtable/subkey interface
|
|
98
100
|
# (only defined for hashes with subtables; otherwise None)
|
|
@@ -111,6 +113,7 @@ def create_SRHash(d):
|
|
|
111
113
|
# The given d is copied and reduced to the required fields.
|
|
112
114
|
# The hashfuncs tuple is reduced to a single ASCII bytestring.
|
|
113
115
|
d0 = dict(d)
|
|
116
|
+
d0.setdefault('windowsize', 0) # only windowed hash types (hash_ccrh) set this
|
|
114
117
|
d0['mem_bytes'] = d0['hashtable'].nbytes
|
|
115
118
|
d0['slots_nonempty'] = compile_slots_nonempty(d)
|
|
116
119
|
d0['slots_with_value'] = compile_slots_with_value(d)
|
|
@@ -821,10 +821,12 @@ def compile_classify_read_from_fastq(
|
|
|
821
821
|
pass
|
|
822
822
|
|
|
823
823
|
@contextmanager
|
|
824
|
-
def cond_contextmgr(name, suffix, count, filt):
|
|
824
|
+
def cond_contextmgr(name, suffix, count, filt, enabled=True):
|
|
825
825
|
if count and filt:
|
|
826
826
|
raise ValueError("ERROR: cannot use both --count and --filter option at the same time.")
|
|
827
|
-
if
|
|
827
|
+
if not enabled:
|
|
828
|
+
yield dummy_contextmgr()
|
|
829
|
+
elif count:
|
|
828
830
|
yield dummy_contextmgr()
|
|
829
831
|
elif filt:
|
|
830
832
|
if "graft" in suffix:
|
|
@@ -840,7 +842,6 @@ def compile_classify_read_from_fastq(
|
|
|
840
842
|
counts = [0, 0, 0, 0, 0, 0] # host, graft, amb., both, neither
|
|
841
843
|
nprocessed = 0
|
|
842
844
|
|
|
843
|
-
site_count = count and not use_start_sites
|
|
844
845
|
with cond_contextmgr(path, f"-host.1.fq{compression}", count, filt) as host1, \
|
|
845
846
|
cond_contextmgr(path, f"-host.2.fq{compression}", count, filt) as host2, \
|
|
846
847
|
cond_contextmgr(path, f"-graft.1.fq{compression}", count, filt) as graft1, \
|
|
@@ -851,8 +852,8 @@ def compile_classify_read_from_fastq(
|
|
|
851
852
|
cond_contextmgr(path, f"-both.2.fq{compression}", count, filt) as both2, \
|
|
852
853
|
cond_contextmgr(path, f"-neither.1.fq{compression}", count, filt) as neither1, \
|
|
853
854
|
cond_contextmgr(path, f"-neither.2.fq{compression}", count, filt) as neither2, \
|
|
854
|
-
cond_contextmgr(path, f"-sites.1.fq{compression}",
|
|
855
|
-
cond_contextmgr(path, f"-sites.2.fq{compression}",
|
|
855
|
+
cond_contextmgr(path, f"-sites.1.fq{compression}", count, filt, use_start_sites) as sites1, \
|
|
856
|
+
cond_contextmgr(path, f"-sites.2.fq{compression}", count, filt, use_start_sites) as sites2:
|
|
856
857
|
streams = ((host1, host2), (graft1, graft2), (ambiguous1, ambiguous2), (both1, both2), (neither1, neither2), (sites1, sites2))
|
|
857
858
|
|
|
858
859
|
with ThreadPoolExecutor(max_workers=threads) as executor:
|
|
@@ -910,13 +911,12 @@ def compile_classify_read_from_fastq(
|
|
|
910
911
|
def classify_read_from_fastq_single(fastqs, ht):
|
|
911
912
|
counts = [0, 0, 0, 0, 0, 0]
|
|
912
913
|
nprocessed = 0
|
|
913
|
-
site_count = count and not use_start_sites
|
|
914
914
|
with cond_contextmgr(path, f"-host.fq{compression}", count, filt) as host, \
|
|
915
915
|
cond_contextmgr(path, f"-graft.fq{compression}", count, filt) as graft, \
|
|
916
916
|
cond_contextmgr(path, f"-ambiguous.fq{compression}", count, filt) as ambiguous, \
|
|
917
917
|
cond_contextmgr(path, f"-both.fq{compression}", count, filt) as both, \
|
|
918
918
|
cond_contextmgr(path, f"-neither.fq{compression}", count, filt) as neither, \
|
|
919
|
-
cond_contextmgr(path, f"-sites.fq{compression}",
|
|
919
|
+
cond_contextmgr(path, f"-sites.fq{compression}", count, filt, use_start_sites) as sites:
|
|
920
920
|
streams = (host, graft, ambiguous, both, neither, sites)
|
|
921
921
|
|
|
922
922
|
running_jobs = []
|
|
@@ -211,9 +211,12 @@ def main(args):
|
|
|
211
211
|
|
|
212
212
|
startweak = timestamp0(msg="- Computing weak k-mers...")
|
|
213
213
|
nextchars = 3 # TODO:? parameterize via args?
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
214
|
+
if args.weakthreads > 0:
|
|
215
|
+
calculate_weak_set(
|
|
216
|
+
h, mask.k, args.groupprefixlength, nextchars,
|
|
217
|
+
rcmode=rcmode, threads=args.weakthreads)
|
|
218
|
+
else:
|
|
219
|
+
debugprint0('- WARNING: 0 weak threads provided. No weak k-mers computed.')
|
|
217
220
|
timestamp0(startweak, msg="- weak k-mers: wall time")
|
|
218
221
|
timestamp0(startweak, msg="- weak k-mers: wall time", minutes=True)
|
|
219
222
|
|
|
@@ -7,7 +7,7 @@ from numba import njit
|
|
|
7
7
|
|
|
8
8
|
from ..lowlevel import debug
|
|
9
9
|
from ..srhash import print_statistics
|
|
10
|
-
from ..io.hashio import load_hash,
|
|
10
|
+
from ..io.hashio import load_hash, _export_to_datafiles
|
|
11
11
|
|
|
12
12
|
|
|
13
13
|
def print_dict(d, title=None):
|
|
@@ -63,7 +63,7 @@ def main(args):
|
|
|
63
63
|
anyformat = (fmt is not None)
|
|
64
64
|
prefix = args.outprefix
|
|
65
65
|
nvalues = values.NVALUES
|
|
66
|
-
filterfunc = compile_filter_function(args
|
|
66
|
+
filterfunc = compile_filter_function(getattr(args, "filter", None))
|
|
67
67
|
if (prefix is None) and anyformat:
|
|
68
68
|
prefix = args.hashname
|
|
69
69
|
if prefix is not None:
|
|
@@ -75,7 +75,7 @@ def main(args):
|
|
|
75
75
|
fmt = "native"
|
|
76
76
|
elif fmt == "txt":
|
|
77
77
|
fmt = "text"
|
|
78
|
-
written =
|
|
78
|
+
written = _export_to_datafiles(h, nvalues, infotup, prefix, fmt, filterfunc)
|
|
79
79
|
debugprint0(f"- wrote {written} items")
|
|
80
80
|
timestamp0(startout, msg="- done writing data; writing time")
|
|
81
81
|
|
|
@@ -53,7 +53,9 @@ def classify(p):
|
|
|
53
53
|
help="only count reads or read pairs for each class, do not output any FASTQ")
|
|
54
54
|
p.add_argument("--prefetchlevel", metavar="INT",
|
|
55
55
|
type=restricted_number_type("from_0_to_2", int, [(">=", 0), ("<=", 2)]),
|
|
56
|
-
help="amount of prefetching: none (0), second bucket (1), all buckets (2)")
|
|
56
|
+
help="amount of prefetching: none (0), second bucket (1), all buckets (2) [classify only]")
|
|
57
|
+
p.add_argument("--lookahead", metavar="INT", type=int, default=16,
|
|
58
|
+
help="classify2 only: number of k-mers to prefetch ahead during lookup [16]")
|
|
57
59
|
p.add_argument("--chunksize", "-C", metavar="FLOAT_SIZE_MB", type=float, default=8.0,
|
|
58
60
|
help="chunk size in MB; one chunk is allocated per thread.")
|
|
59
61
|
p.add_argument("--chunkreads", "-R", metavar="INT", type=int,
|
|
@@ -146,6 +148,11 @@ SUBCOMMANDS = [
|
|
|
146
148
|
classify,
|
|
147
149
|
"xengsort_classify", "main",
|
|
148
150
|
[f"{CFGPATH}/classify.yaml", 'config/classify.yaml', 'classify.yaml']),
|
|
151
|
+
("classify2",
|
|
152
|
+
"sort/filter/count FASTQ reads (new producer/consumer pipeline; for comparison)",
|
|
153
|
+
classify,
|
|
154
|
+
"classify", "main",
|
|
155
|
+
[f"{CFGPATH}/classify.yaml", 'config/classify.yaml', 'classify.yaml']),
|
|
149
156
|
("info",
|
|
150
157
|
"get information about a hash table and dump its data",
|
|
151
158
|
info,
|
|
@@ -193,7 +200,7 @@ def get_argument_parser():
|
|
|
193
200
|
chandler = 'error'
|
|
194
201
|
sp = ArgumentParser(prog=name, description=helptext,
|
|
195
202
|
default_config_files=default_configs)
|
|
196
|
-
if name in ["classify", "index"]:
|
|
203
|
+
if name in ["classify", "classify2", "index"]:
|
|
197
204
|
sp.add_argument('--cfg', "--config", action=ActionConfigFile)
|
|
198
205
|
sp.add_argument("--func", default=(module, f_main), help=SUPPRESS)
|
|
199
206
|
f_parser(sp)
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: xengsort
|
|
3
|
-
Version: 2.2.
|
|
3
|
+
Version: 2.2.2
|
|
4
4
|
Summary: xenograft sorting of FASTQ reads into five categories: host, graft, both, neither, ambiguous.
|
|
5
5
|
Author-email: Jens Zentgraf <zentgraf@cs.uni-saarland.de>, Sven Rahmann <rahmann@cs.uni-saarland.de>
|
|
6
6
|
License: MIT License
|
|
@@ -52,6 +52,7 @@ This tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the
|
|
|
52
52
|
For a description of the method an evaluation on several datasets, please see our
|
|
53
53
|
[article in "Algorithms for Molecular Biology"](https://almob.biomedcentral.com/articles/10.1186/s13015-021-00181-w), or the initial [WABI 2020 publication](https://drops.dagstuhl.de/opus/volltexte/2020/12793/).
|
|
54
54
|
(There was also a [preprint on bioRxiv](https://www.biorxiv.org/content/10.1101/2020.05.14.095604v2).)
|
|
55
|
+
If you use xengsort to sort converted reads (RRBS or WGBS) there is a [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2026.10.06.756952v1.abstract) available.
|
|
55
56
|
BibTeX entrys for citation can be found at the end of this README file.
|
|
56
57
|
|
|
57
58
|
In case of problems,
|
|
@@ -71,24 +72,23 @@ Thank you!
|
|
|
71
72
|
|
|
72
73
|
We here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
|
|
73
74
|
This tutorial uses the workflow management system `Snakemake` to run the example.
|
|
74
|
-
This is, however, not necessary to use `xengsort`.
|
|
75
|
+
This is, however, not necessary to use `xengsort`.
|
|
75
76
|
Please refer to the Usage Guide below to learn how to run `xengsort` on your own data manually.
|
|
76
77
|
|
|
77
78
|
|
|
78
79
|
Our software `xengsort` is provided as a Python package.
|
|
79
|
-
For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
|
|
80
|
+
For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
|
|
80
81
|
We here use the package manager [conda](https://docs.conda.io/en/latest/) to manage Python packages and to use a separate environment for `xengsort`.
|
|
81
82
|
|
|
82
83
|
|
|
83
|
-
### Install the conda package manager (
|
|
84
|
-
Go to https://
|
|
85
|
-
Choose
|
|
86
|
-
Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
|
|
84
|
+
### Install the conda package manager (conda-forge)
|
|
85
|
+
Go to https://conda-forge.org/download/ and download the conda-forge installer:
|
|
86
|
+
Choose your operating system, follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
|
|
87
87
|
You can let the installer do it, or do it manually by editing your ``.bashrc`` or similar file under Linux or MacOS, or by editing the environment variables under Windows.
|
|
88
88
|
To verify that the installation works, open a new terminal and execute
|
|
89
89
|
```
|
|
90
|
-
conda --version # ideally
|
|
91
|
-
python --version # ideally 3.
|
|
90
|
+
conda --version # ideally 26.9.xx or higher
|
|
91
|
+
python --version # ideally 3.12.xx or higher
|
|
92
92
|
```
|
|
93
93
|
|
|
94
94
|
|
|
@@ -122,7 +122,7 @@ within the cloned directory tree.
|
|
|
122
122
|
|
|
123
123
|
|
|
124
124
|
### Create and activate a conda environment
|
|
125
|
-
To run our software, a [conda](https://
|
|
125
|
+
To run our software, a [conda](https://conda-forge.org/docs/) environment with the required libraries needs to be created.
|
|
126
126
|
A list of needed libraries is provided in the ``environment.yml`` file in the cloned repository;
|
|
127
127
|
it can be used to create a new environment:
|
|
128
128
|
|
|
@@ -133,7 +133,7 @@ conda env create
|
|
|
133
133
|
which will create an environment named ``xengsort`` with the required dependencies,
|
|
134
134
|
using the provided ``environment.yml`` file in the same directory.
|
|
135
135
|
|
|
136
|
-
A more explicit alternative is to run
|
|
136
|
+
A more explicit alternative is to run
|
|
137
137
|
```
|
|
138
138
|
conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
|
|
139
139
|
```
|
|
@@ -217,11 +217,11 @@ To build an index for xengsort, several parameters must be provided, which are d
|
|
|
217
217
|
First, a file name and a path for the index must be chosen.
|
|
218
218
|
The index is stored in two files. We will use `myindex` to store the index in the current folder.
|
|
219
219
|
|
|
220
|
-
Second, two reference genomes (host and graft) must be provided (in FASTA files).
|
|
221
|
-
We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
|
|
220
|
+
Second, two reference genomes (host and graft) must be provided (in FASTA files).
|
|
221
|
+
We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
|
|
222
222
|
The files can be provided as an uncompressed file or compressed using `gzip`, `bzip2` or `xz`.
|
|
223
|
-
The corresponding options are `-H` or `--host` and `-G` or `--graft`.
|
|
224
|
-
Each option can take several arguments as files.
|
|
223
|
+
The corresponding options are `-H` or `--host` and `-G` or `--graft`.
|
|
224
|
+
Each option can take several arguments as files.
|
|
225
225
|
```
|
|
226
226
|
xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000 [OPTIONS]
|
|
227
227
|
```
|
|
@@ -249,7 +249,7 @@ The final important parameter is about parallelization:
|
|
|
249
249
|
|
|
250
250
|
Most of the parameters can also be provided in a config file (`.yaml`):
|
|
251
251
|
- `--cfg` or `--config` defines the path the the config file.
|
|
252
|
-
|
|
252
|
+
|
|
253
253
|
### How to classify
|
|
254
254
|
|
|
255
255
|
To classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
|
|
@@ -257,7 +257,7 @@ Then run the `xengsort classify` command with a previously built index, such as
|
|
|
257
257
|
```
|
|
258
258
|
xengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode count
|
|
259
259
|
```
|
|
260
|
-
for single-end reads, or
|
|
260
|
+
for single-end reads, or
|
|
261
261
|
```
|
|
262
262
|
xengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode count
|
|
263
263
|
```
|
|
@@ -275,7 +275,7 @@ The parameter `--prefix` or equivalently `--out` is required and defines the pre
|
|
|
275
275
|
- `{prefix}.both.1.fq.gz`: reads that could originate from both
|
|
276
276
|
- `{prefix}.neither.1.fq.gz`: reads that originate from neither host nor graft
|
|
277
277
|
- `{prefix}.ambiguous.1.fq.gz`: (few) ambiguous reads that cannot be classified,
|
|
278
|
-
|
|
278
|
+
|
|
279
279
|
and similarly with `.2.fq.gz`. For single-end reads, there is only `.fq.gz` (no numbers).
|
|
280
280
|
|
|
281
281
|
The compression type can be specified using the `--compression` parameter.
|
|
@@ -301,7 +301,7 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 9_000_000_000 --b
|
|
|
301
301
|
If an index is build using the bisulfite parameter, also the read is converted in the classification step.
|
|
302
302
|
It is not necessary to change the parameters of the classification step.
|
|
303
303
|
Currently, it only supports the count-based classification.
|
|
304
|
-
|
|
304
|
+
A detailed description on how the benchmarks on bisulfite sequencing data was performed can be found [here](https://github.com/S-Altend/MethSortBench/tree/main/03_read_sorting).
|
|
305
305
|
|
|
306
306
|
Please report a [GitLab issue](https://gitlab.com/genomeinformatics/xengsort/-/issues) in case of problems.
|
|
307
307
|
|
|
@@ -315,13 +315,13 @@ Happy sorting!
|
|
|
315
315
|
* **Snakemake throws errors when attempting to download the example dataset from SRA**.
|
|
316
316
|
|
|
317
317
|
This may happen for several reasons:
|
|
318
|
-
1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
|
|
318
|
+
1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
|
|
319
319
|
1. Another source of trouble may be that for some reason and older version of `sra-tools` gets called. We use the `fasterq-dump` tool to download the data, which did not exist in older verisons. To use a clean environment with a recent version of `sra-tools`, you can run snakemake as `snakemake -j 16 --use-conda`, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).
|
|
320
320
|
1. In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (`xengsort --version`). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and run `pip install -e .`, as explained above in the tutorial.
|
|
321
321
|
|
|
322
322
|
* **Indexing stops after a few seconds with a failure.**
|
|
323
323
|
|
|
324
|
-
The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
|
|
324
|
+
The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
|
|
325
325
|
(From version 1.0.1, the `-n` parameter is required, and the error cannot occur anymore. Please update to the latest version!)
|
|
326
326
|
For the human and mouse genomes this is approximately `-n 4_500_000_000` (4.5 billion).
|
|
327
327
|
It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
|
|
@@ -377,3 +377,14 @@ In addition, you may also cite the WABI 2020 proceedigs paper:
|
|
|
377
377
|
annote = {Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
|
|
378
378
|
}
|
|
379
379
|
```
|
|
380
|
+
|
|
381
|
+
If you use xengsort to sort converted reads, please also cite our corresponding bioRxiv preprint:
|
|
382
|
+
```
|
|
383
|
+
@article {Altendorfer2026.10.06.756952,
|
|
384
|
+
author = {Altendorfer, Stefan and Zentgraf, Jens and Schmitz, Johanna Elena and Plass, Christoph and Gerh{\"a}user, Clarissa and Rahmann, Sven},
|
|
385
|
+
title = {High-performance xenograft read-sorting for DNA methylation analysis},
|
|
386
|
+
year = {2026},
|
|
387
|
+
doi = {10.64898/2026.10.06.756952},
|
|
388
|
+
journal = {bioRxiv}
|
|
389
|
+
}
|
|
390
|
+
```
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|