xengsort 2.2.1__tar.gz → 2.2.2__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (55) hide show
  1. {xengsort-2.2.1/xengsort.egg-info → xengsort-2.2.2}/PKG-INFO +32 -21
  2. {xengsort-2.2.1 → xengsort-2.2.2}/README.md +31 -20
  3. {xengsort-2.2.1 → xengsort-2.2.2}/pyproject.toml +1 -1
  4. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/cptasks_kmers.py +25 -4
  5. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/hash_s3c_fbcbvb.py +16 -0
  6. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/fastaio.py +7 -3
  7. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/fastqio.py +10 -7
  8. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/generalio.py +12 -1
  9. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/hashio.py +7 -1
  10. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/kmers.py +5 -4
  11. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/libc.py +16 -2
  12. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/srhash.py +3 -0
  13. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_classify.py +7 -7
  14. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_index.py +6 -3
  15. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_info.py +3 -3
  16. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_main.py +9 -2
  17. {xengsort-2.2.1 → xengsort-2.2.2/xengsort.egg-info}/PKG-INFO +32 -21
  18. {xengsort-2.2.1 → xengsort-2.2.2}/LICENSE +0 -0
  19. {xengsort-2.2.1 → xengsort-2.2.2}/MANIFEST.in +0 -0
  20. {xengsort-2.2.1 → xengsort-2.2.2}/setup.cfg +0 -0
  21. {xengsort-2.2.1 → xengsort-2.2.2}/tests/test_trimmed_reads.py +0 -0
  22. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/dnaencode.py +0 -0
  23. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/dnaencode_fast.py +0 -0
  24. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/fastcash_info.py +0 -0
  25. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/fastcash_main.py +0 -0
  26. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/fastcash_optimize.py +0 -0
  27. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/hash_minimizer.py +0 -0
  28. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/hash_srh_fbcbvb.py +0 -0
  29. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/binaryio.py +0 -0
  30. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/seqio.py +0 -0
  31. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/textio.py +0 -0
  32. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/io/xorio.py +0 -0
  33. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/aligned_arrays.py +0 -0
  34. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/bitarray.py +0 -0
  35. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/conpro.py +0 -0
  36. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/debug.py +0 -0
  37. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/intbitarray.py +0 -0
  38. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/llvm.py +0 -0
  39. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/numbautils.py +0 -0
  40. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/lowlevel/packedarray.py +0 -0
  41. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/mask.py +0 -0
  42. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/mathutils.py +0 -0
  43. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/parameters.py +0 -0
  44. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/subtable_hashfunctions.py +0 -0
  45. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/values/xenograft.py +0 -0
  46. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/config/classify.yaml +0 -0
  47. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/config/index.yaml +0 -0
  48. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_load.py +0 -0
  49. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_remove.py +0 -0
  50. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort/xengsort/xengsort_weak.py +0 -0
  51. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/SOURCES.txt +0 -0
  52. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/dependency_links.txt +0 -0
  53. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/entry_points.txt +0 -0
  54. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/requires.txt +0 -0
  55. {xengsort-2.2.1 → xengsort-2.2.2}/xengsort.egg-info/top_level.txt +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: xengsort
3
- Version: 2.2.1
3
+ Version: 2.2.2
4
4
  Summary: xenograft sorting of FASTQ reads into five categories: host, graft, both, neither, ambiguous.
5
5
  Author-email: Jens Zentgraf <zentgraf@cs.uni-saarland.de>, Sven Rahmann <rahmann@cs.uni-saarland.de>
6
6
  License: MIT License
@@ -52,6 +52,7 @@ This tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the
52
52
  For a description of the method an evaluation on several datasets, please see our
53
53
  [article in "Algorithms for Molecular Biology"](https://almob.biomedcentral.com/articles/10.1186/s13015-021-00181-w), or the initial [WABI 2020 publication](https://drops.dagstuhl.de/opus/volltexte/2020/12793/).
54
54
  (There was also a [preprint on bioRxiv](https://www.biorxiv.org/content/10.1101/2020.05.14.095604v2).)
55
+ If you use xengsort to sort converted reads (RRBS or WGBS) there is a [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2026.10.06.756952v1.abstract) available.
55
56
  BibTeX entrys for citation can be found at the end of this README file.
56
57
 
57
58
  In case of problems,
@@ -71,24 +72,23 @@ Thank you!
71
72
 
72
73
  We here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
73
74
  This tutorial uses the workflow management system `Snakemake` to run the example.
74
- This is, however, not necessary to use `xengsort`.
75
+ This is, however, not necessary to use `xengsort`.
75
76
  Please refer to the Usage Guide below to learn how to run `xengsort` on your own data manually.
76
77
 
77
78
 
78
79
  Our software `xengsort` is provided as a Python package.
79
- For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
80
+ For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
80
81
  We here use the package manager [conda](https://docs.conda.io/en/latest/) to manage Python packages and to use a separate environment for `xengsort`.
81
82
 
82
83
 
83
- ### Install the conda package manager (miniconda)
84
- Go to https://docs.conda.io/en/latest/miniconda.html and download the Miniconda installer:
85
- Choose Python 3.10 (or higher), your operating system, and preferably the 64 bit version.
86
- Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
84
+ ### Install the conda package manager (conda-forge)
85
+ Go to https://conda-forge.org/download/ and download the conda-forge installer:
86
+ Choose your operating system, follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
87
87
  You can let the installer do it, or do it manually by editing your ``.bashrc`` or similar file under Linux or MacOS, or by editing the environment variables under Windows.
88
88
  To verify that the installation works, open a new terminal and execute
89
89
  ```
90
- conda --version # ideally 22.9.xx or higher
91
- python --version # ideally 3.10.xx or higher
90
+ conda --version # ideally 26.9.xx or higher
91
+ python --version # ideally 3.12.xx or higher
92
92
  ```
93
93
 
94
94
 
@@ -122,7 +122,7 @@ within the cloned directory tree.
122
122
 
123
123
 
124
124
  ### Create and activate a conda environment
125
- To run our software, a [conda](https://docs.conda.io/en/latest/) environment with the required libraries needs to be created.
125
+ To run our software, a [conda](https://conda-forge.org/docs/) environment with the required libraries needs to be created.
126
126
  A list of needed libraries is provided in the ``environment.yml`` file in the cloned repository;
127
127
  it can be used to create a new environment:
128
128
 
@@ -133,7 +133,7 @@ conda env create
133
133
  which will create an environment named ``xengsort`` with the required dependencies,
134
134
  using the provided ``environment.yml`` file in the same directory.
135
135
 
136
- A more explicit alternative is to run
136
+ A more explicit alternative is to run
137
137
  ```
138
138
  conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
139
139
  ```
@@ -217,11 +217,11 @@ To build an index for xengsort, several parameters must be provided, which are d
217
217
  First, a file name and a path for the index must be chosen.
218
218
  The index is stored in two files. We will use `myindex` to store the index in the current folder.
219
219
 
220
- Second, two reference genomes (host and graft) must be provided (in FASTA files).
221
- We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
220
+ Second, two reference genomes (host and graft) must be provided (in FASTA files).
221
+ We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
222
222
  The files can be provided as an uncompressed file or compressed using `gzip`, `bzip2` or `xz`.
223
- The corresponding options are `-H` or `--host` and `-G` or `--graft`.
224
- Each option can take several arguments as files.
223
+ The corresponding options are `-H` or `--host` and `-G` or `--graft`.
224
+ Each option can take several arguments as files.
225
225
  ```
226
226
  xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000 [OPTIONS]
227
227
  ```
@@ -249,7 +249,7 @@ The final important parameter is about parallelization:
249
249
 
250
250
  Most of the parameters can also be provided in a config file (`.yaml`):
251
251
  - `--cfg` or `--config` defines the path the the config file.
252
-
252
+
253
253
  ### How to classify
254
254
 
255
255
  To classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
@@ -257,7 +257,7 @@ Then run the `xengsort classify` command with a previously built index, such as
257
257
  ```
258
258
  xengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode count
259
259
  ```
260
- for single-end reads, or
260
+ for single-end reads, or
261
261
  ```
262
262
  xengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode count
263
263
  ```
@@ -275,7 +275,7 @@ The parameter `--prefix` or equivalently `--out` is required and defines the pre
275
275
  - `{prefix}.both.1.fq.gz`: reads that could originate from both
276
276
  - `{prefix}.neither.1.fq.gz`: reads that originate from neither host nor graft
277
277
  - `{prefix}.ambiguous.1.fq.gz`: (few) ambiguous reads that cannot be classified,
278
-
278
+
279
279
  and similarly with `.2.fq.gz`. For single-end reads, there is only `.fq.gz` (no numbers).
280
280
 
281
281
  The compression type can be specified using the `--compression` parameter.
@@ -301,7 +301,7 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 9_000_000_000 --b
301
301
  If an index is build using the bisulfite parameter, also the read is converted in the classification step.
302
302
  It is not necessary to change the parameters of the classification step.
303
303
  Currently, it only supports the count-based classification.
304
-
304
+ A detailed description on how the benchmarks on bisulfite sequencing data was performed can be found [here](https://github.com/S-Altend/MethSortBench/tree/main/03_read_sorting).
305
305
 
306
306
  Please report a [GitLab issue](https://gitlab.com/genomeinformatics/xengsort/-/issues) in case of problems.
307
307
 
@@ -315,13 +315,13 @@ Happy sorting!
315
315
  * **Snakemake throws errors when attempting to download the example dataset from SRA**.
316
316
 
317
317
  This may happen for several reasons:
318
- 1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
318
+ 1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
319
319
  1. Another source of trouble may be that for some reason and older version of `sra-tools` gets called. We use the `fasterq-dump` tool to download the data, which did not exist in older verisons. To use a clean environment with a recent version of `sra-tools`, you can run snakemake as `snakemake -j 16 --use-conda`, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).
320
320
  1. In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (`xengsort --version`). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and run `pip install -e .`, as explained above in the tutorial.
321
321
 
322
322
  * **Indexing stops after a few seconds with a failure.**
323
323
 
324
- The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
324
+ The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
325
325
  (From version 1.0.1, the `-n` parameter is required, and the error cannot occur anymore. Please update to the latest version!)
326
326
  For the human and mouse genomes this is approximately `-n 4_500_000_000` (4.5 billion).
327
327
  It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
@@ -377,3 +377,14 @@ In addition, you may also cite the WABI 2020 proceedigs paper:
377
377
  annote = {Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
378
378
  }
379
379
  ```
380
+
381
+ If you use xengsort to sort converted reads, please also cite our corresponding bioRxiv preprint:
382
+ ```
383
+ @article {Altendorfer2026.10.06.756952,
384
+ author = {Altendorfer, Stefan and Zentgraf, Jens and Schmitz, Johanna Elena and Plass, Christoph and Gerh{\"a}user, Clarissa and Rahmann, Sven},
385
+ title = {High-performance xenograft read-sorting for DNA methylation analysis},
386
+ year = {2026},
387
+ doi = {10.64898/2026.10.06.756952},
388
+ journal = {bioRxiv}
389
+ }
390
+ ```
@@ -5,6 +5,7 @@ This tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the
5
5
  For a description of the method an evaluation on several datasets, please see our
6
6
  [article in "Algorithms for Molecular Biology"](https://almob.biomedcentral.com/articles/10.1186/s13015-021-00181-w), or the initial [WABI 2020 publication](https://drops.dagstuhl.de/opus/volltexte/2020/12793/).
7
7
  (There was also a [preprint on bioRxiv](https://www.biorxiv.org/content/10.1101/2020.05.14.095604v2).)
8
+ If you use xengsort to sort converted reads (RRBS or WGBS) there is a [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2026.10.06.756952v1.abstract) available.
8
9
  BibTeX entrys for citation can be found at the end of this README file.
9
10
 
10
11
  In case of problems,
@@ -24,24 +25,23 @@ Thank you!
24
25
 
25
26
  We here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
26
27
  This tutorial uses the workflow management system `Snakemake` to run the example.
27
- This is, however, not necessary to use `xengsort`.
28
+ This is, however, not necessary to use `xengsort`.
28
29
  Please refer to the Usage Guide below to learn how to run `xengsort` on your own data manually.
29
30
 
30
31
 
31
32
  Our software `xengsort` is provided as a Python package.
32
- For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
33
+ For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
33
34
  We here use the package manager [conda](https://docs.conda.io/en/latest/) to manage Python packages and to use a separate environment for `xengsort`.
34
35
 
35
36
 
36
- ### Install the conda package manager (miniconda)
37
- Go to https://docs.conda.io/en/latest/miniconda.html and download the Miniconda installer:
38
- Choose Python 3.10 (or higher), your operating system, and preferably the 64 bit version.
39
- Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
37
+ ### Install the conda package manager (conda-forge)
38
+ Go to https://conda-forge.org/download/ and download the conda-forge installer:
39
+ Choose your operating system, follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
40
40
  You can let the installer do it, or do it manually by editing your ``.bashrc`` or similar file under Linux or MacOS, or by editing the environment variables under Windows.
41
41
  To verify that the installation works, open a new terminal and execute
42
42
  ```
43
- conda --version # ideally 22.9.xx or higher
44
- python --version # ideally 3.10.xx or higher
43
+ conda --version # ideally 26.9.xx or higher
44
+ python --version # ideally 3.12.xx or higher
45
45
  ```
46
46
 
47
47
 
@@ -75,7 +75,7 @@ within the cloned directory tree.
75
75
 
76
76
 
77
77
  ### Create and activate a conda environment
78
- To run our software, a [conda](https://docs.conda.io/en/latest/) environment with the required libraries needs to be created.
78
+ To run our software, a [conda](https://conda-forge.org/docs/) environment with the required libraries needs to be created.
79
79
  A list of needed libraries is provided in the ``environment.yml`` file in the cloned repository;
80
80
  it can be used to create a new environment:
81
81
 
@@ -86,7 +86,7 @@ conda env create
86
86
  which will create an environment named ``xengsort`` with the required dependencies,
87
87
  using the provided ``environment.yml`` file in the same directory.
88
88
 
89
- A more explicit alternative is to run
89
+ A more explicit alternative is to run
90
90
  ```
91
91
  conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
92
92
  ```
@@ -170,11 +170,11 @@ To build an index for xengsort, several parameters must be provided, which are d
170
170
  First, a file name and a path for the index must be chosen.
171
171
  The index is stored in two files. We will use `myindex` to store the index in the current folder.
172
172
 
173
- Second, two reference genomes (host and graft) must be provided (in FASTA files).
174
- We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
173
+ Second, two reference genomes (host and graft) must be provided (in FASTA files).
174
+ We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
175
175
  The files can be provided as an uncompressed file or compressed using `gzip`, `bzip2` or `xz`.
176
- The corresponding options are `-H` or `--host` and `-G` or `--graft`.
177
- Each option can take several arguments as files.
176
+ The corresponding options are `-H` or `--host` and `-G` or `--graft`.
177
+ Each option can take several arguments as files.
178
178
  ```
179
179
  xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000 [OPTIONS]
180
180
  ```
@@ -202,7 +202,7 @@ The final important parameter is about parallelization:
202
202
 
203
203
  Most of the parameters can also be provided in a config file (`.yaml`):
204
204
  - `--cfg` or `--config` defines the path the the config file.
205
-
205
+
206
206
  ### How to classify
207
207
 
208
208
  To classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
@@ -210,7 +210,7 @@ Then run the `xengsort classify` command with a previously built index, such as
210
210
  ```
211
211
  xengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode count
212
212
  ```
213
- for single-end reads, or
213
+ for single-end reads, or
214
214
  ```
215
215
  xengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode count
216
216
  ```
@@ -228,7 +228,7 @@ The parameter `--prefix` or equivalently `--out` is required and defines the pre
228
228
  - `{prefix}.both.1.fq.gz`: reads that could originate from both
229
229
  - `{prefix}.neither.1.fq.gz`: reads that originate from neither host nor graft
230
230
  - `{prefix}.ambiguous.1.fq.gz`: (few) ambiguous reads that cannot be classified,
231
-
231
+
232
232
  and similarly with `.2.fq.gz`. For single-end reads, there is only `.fq.gz` (no numbers).
233
233
 
234
234
  The compression type can be specified using the `--compression` parameter.
@@ -254,7 +254,7 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 9_000_000_000 --b
254
254
  If an index is build using the bisulfite parameter, also the read is converted in the classification step.
255
255
  It is not necessary to change the parameters of the classification step.
256
256
  Currently, it only supports the count-based classification.
257
-
257
+ A detailed description on how the benchmarks on bisulfite sequencing data was performed can be found [here](https://github.com/S-Altend/MethSortBench/tree/main/03_read_sorting).
258
258
 
259
259
  Please report a [GitLab issue](https://gitlab.com/genomeinformatics/xengsort/-/issues) in case of problems.
260
260
 
@@ -268,13 +268,13 @@ Happy sorting!
268
268
  * **Snakemake throws errors when attempting to download the example dataset from SRA**.
269
269
 
270
270
  This may happen for several reasons:
271
- 1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
271
+ 1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
272
272
  1. Another source of trouble may be that for some reason and older version of `sra-tools` gets called. We use the `fasterq-dump` tool to download the data, which did not exist in older verisons. To use a clean environment with a recent version of `sra-tools`, you can run snakemake as `snakemake -j 16 --use-conda`, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).
273
273
  1. In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (`xengsort --version`). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and run `pip install -e .`, as explained above in the tutorial.
274
274
 
275
275
  * **Indexing stops after a few seconds with a failure.**
276
276
 
277
- The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
277
+ The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
278
278
  (From version 1.0.1, the `-n` parameter is required, and the error cannot occur anymore. Please update to the latest version!)
279
279
  For the human and mouse genomes this is approximately `-n 4_500_000_000` (4.5 billion).
280
280
  It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
@@ -330,3 +330,14 @@ In addition, you may also cite the WABI 2020 proceedigs paper:
330
330
  annote = {Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
331
331
  }
332
332
  ```
333
+
334
+ If you use xengsort to sort converted reads, please also cite our corresponding bioRxiv preprint:
335
+ ```
336
+ @article {Altendorfer2026.10.06.756952,
337
+ author = {Altendorfer, Stefan and Zentgraf, Jens and Schmitz, Johanna Elena and Plass, Christoph and Gerh{\"a}user, Clarissa and Rahmann, Sven},
338
+ title = {High-performance xenograft read-sorting for DNA methylation analysis},
339
+ year = {2026},
340
+ doi = {10.64898/2026.10.06.756952},
341
+ journal = {bioRxiv}
342
+ }
343
+ ```
@@ -5,7 +5,7 @@ build-backend = "setuptools.build_meta"
5
5
 
6
6
  [project]
7
7
  name = "xengsort"
8
- version = "2.2.1"
8
+ version = "2.2.2"
9
9
 
10
10
  license = {file = "LICENSE"}
11
11
 
@@ -544,15 +544,27 @@ def compile_cptask_insert_filtered_subkeys(
544
544
  myfilter,
545
545
  constant_value=0,
546
546
  maxfailures=0,
547
- maxwalk=1000):
547
+ maxwalk=1000,
548
+ prefetch_distance=8):
548
549
  debugprint0, debugprint1, debugprint2 = debug.debugprint
549
550
  dassert = debug.dassert
551
+
552
+ @njit(nogil=True)
553
+ def dummy_prefetch(*_):
554
+ return
555
+
556
+ prefetch_filter = prefetch_table = dummy_prefetch
550
557
  if myfilter is not None:
551
558
  wwo = "with"
552
559
  lookup_and_insert_in_subfilter = myfilter.private.lookup_and_insert_in_subfilter
560
+ table_prefetch_distance = 0
561
+ filter_prefetch_distance = prefetch_distance
562
+ prefetch_filter = myfilter.private.prefetch_block_by_subkey_subfilter
553
563
  else:
554
564
  wwo = "without"
555
-
565
+ table_prefetch_distance = prefetch_distance
566
+ filter_prefetch_distance = 0
567
+ prefetch_table = myhashtable.private.prefetch_subkey
556
568
  @njit(nogil=True)
557
569
  def lookup_and_insert_in_subfilter(ft, st, subkey):
558
570
  return True
@@ -599,7 +611,12 @@ def compile_cptask_insert_filtered_subkeys(
599
611
  # dassert(buffer_type == 1, __name__, 'buffer_type == 1', buffer_type, message='0 means sequence buffer')
600
612
 
601
613
  # consume the buffer
602
- for subkey in active_buffer[:nsubkeys]:
614
+ for ii in range(nsubkeys):
615
+ subkey = active_buffer[ii]
616
+ if filter_prefetch_distance and ii + filter_prefetch_distance < nsubkeys:
617
+ prefetch_filter(ft, active_buffer[ii + prefetch_distance], st)
618
+ if table_prefetch_distance and ii + table_prefetch_distance < nsubkeys:
619
+ prefetch_table(ht, active_buffer[ii + prefetch_distance], st)
603
620
  if lookup_and_insert_in_subfilter(ft, st, subkey):
604
621
  status, result = update(ht, st, subkey, value) # see docstring below
605
622
  if status & 128 == 0:
@@ -618,9 +635,11 @@ def compile_cptask_insert_filtered_subkeys(
618
635
 
619
636
  def compile_cptask_update_existing_subkeys(
620
637
  myhashtable,
621
- constant_value=1):
638
+ constant_value=1,
639
+ prefetch_distance=8):
622
640
  debugprint0, debugprint1, debugprint2 = debug.debugprint
623
641
  update_existing = myhashtable.private.update_existing_ssk
642
+ prefetch = myhashtable.private.prefetch_subkey
624
643
  value = uint64(constant_value)
625
644
  debugprint1(f"- compiling cptask_update_existing_subkeys: {constant_value=}")
626
645
 
@@ -652,6 +671,8 @@ def compile_cptask_update_existing_subkeys(
652
671
 
653
672
  # consume the buffer
654
673
  for ii in range(nsubkeys):
674
+ if prefetch_distance and ii + prefetch_distance < nsubkeys:
675
+ prefetch(ht, active_buffer[ii + prefetch_distance], st)
655
676
  subkey = active_buffer[ii]
656
677
  _, _ = update_existing(ht, st, subkey, value) # see docstring below
657
678
  # status, result = update_existing(ht, st, subkey, value) # see docstring below
@@ -134,6 +134,22 @@ def build_hash(universe, n, subtables, bucketsize,
134
134
  def prefetch_bucket(table, subtable, bucket):
135
135
  startbit = subtable * subtablebits + bucket * bucketsizebits
136
136
  hprefetch(table, startbit)
137
+ hprefetch(table, startbit + uint64(bucketsizebits - 1))
138
+
139
+ @njit(nogil=True, locals=dict(
140
+ bucket=int64, startbit=uint64))
141
+ def prefetch_subkey(table, subkey, subtable):
142
+ bucket, _ = get_bs1(subkey)
143
+ startbit = subtable * subtablebits + bucket * bucketsizebits
144
+ hprefetch(table, startbit)
145
+
146
+ bucket, _ = get_bs2(subkey)
147
+ startbit = subtable * subtablebits + bucket * bucketsizebits
148
+ hprefetch(table, startbit)
149
+
150
+ bucket, _ = get_bs3(subkey)
151
+ startbit = subtable * subtablebits + bucket * bucketsizebits
152
+ hprefetch(table, startbit)
137
153
 
138
154
  @njit(nogil=True, locals=dict(
139
155
  bucket=int64, startbit=uint64))
@@ -103,6 +103,7 @@ def cptask_read_fasta_into_buffers(fd, W, inbuffers, incontrol, ininfos, outbuff
103
103
  inside_seq = True
104
104
  header_seq = False
105
105
  nseq = 0
106
+ last_seq_length = 2 * W
106
107
  while True:
107
108
  old = nactive
108
109
  old_buffer = active_buffer
@@ -134,7 +135,11 @@ def cptask_read_fasta_into_buffers(fd, W, inbuffers, incontrol, ininfos, outbuff
134
135
  if ntodo < 0:
135
136
  errorcode = ntodo
136
137
  break
138
+ # If we have a long header but a seq smaller W, the offset can be bigger than W
139
+ if header_seq and last_seq_length < W:
140
+ break
137
141
  # Nothing read and the offset is not a full (gapped) k-mer
142
+ # I think this should trigger
138
143
  if offset < W:
139
144
  break
140
145
  # Nothing read but we still have some data in the buffer
@@ -157,7 +162,6 @@ def cptask_read_fasta_into_buffers(fd, W, inbuffers, incontrol, ininfos, outbuff
157
162
  offset = (W - 1)
158
163
  else:
159
164
  offset = end_buffer - linemarkbuffers[nactive, nseq - 1, 3]
160
-
161
165
  outcontrol[nactive, 7] = nseq
162
166
  outcontrol[nactive, 6] = 0 # buffer type sequence data
163
167
 
@@ -417,7 +421,7 @@ def fastaextract(args):
417
421
  seps = args.separators
418
422
  sfx = [args.suffix] if args.suffix else []
419
423
  seps = [_SEPARATORS.get(sep.upper(), sep) for sep in seps]
420
- if items is None:
424
+ if items is None:
421
425
  items = list()
422
426
  seps = list()
423
427
  if len(seps) == 1:
@@ -436,7 +440,7 @@ def fastaextract(args):
436
440
  def get_header_info(header, items, assigner, seps):
437
441
  fields = header.decode("ascii").split()
438
442
  assigners = [i for (i, field) in enumerate(fields) if assigner in field]
439
- if 0 in assigners:
443
+ if 0 in assigners:
440
444
  assigners.remove(0)
441
445
  D = dict()
442
446
  if items is None:
@@ -4,7 +4,7 @@ import numpy as np
4
4
  from numba import njit, int32, int64, uint64
5
5
 
6
6
  from .seqio import FormatError, _universal_reads
7
- from ..lowlevel.libc import read_block
7
+ from ..lowlevel.libc import read_block, _memchr
8
8
  from ..lowlevel.conpro import \
9
9
  find_buffer_for_writing, \
10
10
  mark_buffer_for_reading, mark_buffer_for_writing
@@ -604,6 +604,7 @@ def _find_fastq_seqmarks(buf, linemarks, sequence_id=-1):
604
604
  if n == 0:
605
605
  return (0, 0)
606
606
  M = linemarks.shape[0]
607
+ start_ptr = buf.ctypes.data
607
608
  i = 0
608
609
  m = -1 # number of current record we're in
609
610
  nxt = 0 # byte of beginning of last record
@@ -633,13 +634,15 @@ def _find_fastq_seqmarks(buf, linemarks, sequence_id=-1):
633
634
  linemarks[m, 2] = i
634
635
  elif line == 1:
635
636
  linemarks[m, 0] = i
637
+
636
638
  # find end of current line
637
- while buf[i] != 10:
638
- i += 1
639
- if i >= n:
640
- # we did not find the end of the line before the buffer was exhausted
641
- # we cannot set linemarks[m][1]
642
- return (m, nxt)
639
+ next_ptr = _memchr(start_ptr + i, 10, n - i)
640
+ if next_ptr == 0:
641
+ # we did not find the end of the line before the buffer was exhausted
642
+ # we cannot set linemarks[m][1]
643
+ return (m, nxt)
644
+ i = next_ptr - start_ptr
645
+
643
646
  if line == 1:
644
647
  linemarks[m, 1] = i
645
648
  line = (line + 1) % 4
@@ -19,9 +19,18 @@ from .fastaio import \
19
19
  from .binaryio import \
20
20
  compile_cptask_read_data_into_buffers
21
21
 
22
+ igzip_installed = False
23
+ if which("igzip") is not None:
24
+ igzip_installed = True
25
+
26
+ _gz_decompress = "igzip -d -c" if igzip_installed else "bgzip -cd"
22
27
 
23
28
  _in_suffixes = {
24
- ".gz": "bgzip -cd",
29
+ # ".gz": "bgzip -cd",
30
+ # ".gz": "rapidgzip -d -c",
31
+ # ".gz": "rapidgzip-rust",
32
+ # ".gz": "igzip -d -c",
33
+ ".gz": _gz_decompress,
25
34
  ".bz2": "bzip2 -cd",
26
35
  ".xz": "xz -cd",
27
36
  ".zst": "zstd -cd",
@@ -34,6 +43,8 @@ _out_suffixes = {
34
43
  }
35
44
  _threads = {
36
45
  "bgzip": "@",
46
+ "rapidgzip": "P",
47
+ "rapidgzip-rust": "P",
37
48
  "xz": "T",
38
49
  "zstd": "T",
39
50
  }
@@ -54,7 +54,7 @@ from ..lowlevel.intbitarray import intbitarray
54
54
  from ..lowlevel.libc import write_block, write_uint64, write_dna
55
55
 
56
56
 
57
- _ATTRS_TO_SAVE = "hashtype choices nvalues subtables aligned universe nslots=n nbuckets bucketsize nfingerprints mem_bytes shortcutbits hashfuncs".split()
57
+ _ATTRS_TO_SAVE = "hashtype choices nvalues subtables aligned universe nslots=n nbuckets bucketsize windowsize nfingerprints mem_bytes shortcutbits hashfuncs".split()
58
58
  _ATTR_PAIRS = [((name if '=' not in name else name.split("=")[0]), (name if '=' not in name else name.split("=")[1])) for name in _ATTRS_TO_SAVE]
59
59
  # TODO: see comment in save_hash on how to get rid of these
60
60
 
@@ -219,6 +219,12 @@ def load_hash(filename, *, shared_memory=False, info_only=False, quiet=False):
219
219
  bucketsize, hashfuncs, nvalues, update_value,
220
220
  aligned=aligned, nfingerprints=nfingerprints,
221
221
  init=init, maxwalk=maxwalk, shm=shm)
222
+ elif hashtype == 'ccrh':
223
+ h = m.build_hash(universe, n, subtables,
224
+ bucketsize, hashfuncs, nvalues, update_value,
225
+ aligned=aligned, nfingerprints=nfingerprints,
226
+ init=init, maxwalk=maxwalk, shortcutbits=shortcutbits,
227
+ windowsize=hashinfo['windowsize'], shm=shm)
222
228
  else:
223
229
  h = m.build_hash(universe, n, subtables,
224
230
  bucketsize, hashfuncs, nvalues, update_value,
@@ -605,6 +605,7 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
605
605
  w = mask.w
606
606
  if k < 1 or k > 32:
607
607
  raise ValueError(f"only 1<=k<=32 is supported, but k={k}.")
608
+
608
609
  revcomp, ccode = compile_revcomp_and_canonical_code_new(k, rcmode) if new_encoding else compile_revcomp_and_canonical_code_old(k, rcmode)
609
610
  invalid = uint64(np.iinfo(np.uint64).max)
610
611
  popcnt = compile_popcount('uint64')
@@ -686,7 +687,7 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
686
687
  contained_ns = contained_ns >> 1
687
688
 
688
689
  # compute gapped k-mers iteratively based on previous gapped k-mers
689
- for i in range(start + last, (end - start) - (width - 1)):
690
+ for i in range(end_naive, (end - start) - (width - 1)):
690
691
  contained_ns = contained_ns << 1
691
692
  code = 0
692
693
  for k, shift, mask in steps:
@@ -701,13 +702,13 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
701
702
  @njit(nogil=True)
702
703
  def canonical_codes_inplace(arr, nhashes):
703
704
  for i in range(nhashes):
704
- arr[i] = ccode(arr[i])
705
+ arr[i] = ccode(arr[i]) if arr[i] != invalid else invalid
705
706
  return nhashes
706
707
 
707
708
  @njit(nogil=True)
708
709
  def canonical_codes_sampled(arr, nhashes):
709
710
  for idx, hash_idx in enumerate(range(0, nhashes, w // 2)):
710
- arr[idx] = ccode(arr[hash_idx])
711
+ arr[idx] = ccode(arr[i]) if arr[i] != invalid else invalid
711
712
  return idx + 1
712
713
 
713
714
  canonical_codes = canonical_codes_sampled if sampled else canonical_codes_inplace
@@ -716,7 +717,7 @@ def compile_get_gapped_kmers_issh(mask, rcmode='max', new_encoding=True, sampled
716
717
  def get_gapped_kmers(seq, hashes):
717
718
  hashes[:] = 0
718
719
  issh_processor_fast(hashes, seq, 0, len(seq))
719
- return canonical_codes(hashes, len(seq) - w + 1)
720
+ return canonical_codes(hashes, max(0, len(seq) - w + 1))
720
721
 
721
722
  @njit(nogil=True)
722
723
  def get_gapped_kmers_forward(seq, hashes):
@@ -49,6 +49,10 @@ def _setup_libc():
49
49
  _libc.write.argtypes = (ctypes.c_int, ctypes.c_void_p, ctypes.c_size_t)
50
50
  _write = _libc.write
51
51
 
52
+ _libc.memchr.restype = ctypes.c_ssize_t
53
+ _libc.memchr.argtypes = (ctypes.c_size_t, ctypes.c_int, ctypes.c_size_t)
54
+ _memchr = _libc.memchr
55
+
52
56
  if 'linux' in platform:
53
57
  _libc.__errno_location.argtypes = tuple()
54
58
  _libc.__errno_location.restype = ctypes.POINTER(ctypes.c_int)
@@ -61,7 +65,7 @@ def _setup_libc():
61
65
  raise NotImplementedError(msg)
62
66
 
63
67
  _errno = njit(nogil=True)(lambda: _errno_location()[0])
64
- return _libc, _read, _write, _errno
68
+ return _libc, _read, _write, _memchr, _errno
65
69
 
66
70
  else:
67
71
  """
@@ -70,7 +74,17 @@ def _setup_libc():
70
74
  raise NotImplementedError(msg)
71
75
 
72
76
 
73
- libc, _read, _write, errno = _setup_libc()
77
+ libc, _read, _write, _memchr, errno = _setup_libc()
78
+
79
+
80
+ @njit(nogil=True)
81
+ def next_char_at(buf, c):
82
+ start_ptr = buf.ctypes.data
83
+ nbytes = buf.shape[0]
84
+ res = _memchr(start_ptr, c, nbytes)
85
+ if res == 0:
86
+ return -1
87
+ return res - start_ptr
74
88
 
75
89
 
76
90
  @njit(nogil=True, locals=dict()) # offset=int32, buflen=int32, space=int32, res=int32, fill=int32, err=int32))
@@ -34,6 +34,7 @@ SRHash = namedtuple("SRHash", [
34
34
  "subtables",
35
35
  "nbuckets",
36
36
  "bucketsize",
37
+ "windowsize", # window size for windowed hash types (hash_ccrh); 0 if unused
37
38
  "nfingerprints",
38
39
  "nvalues",
39
40
  "mem_bytes",
@@ -93,6 +94,7 @@ SRHash_private = namedtuple("SRHash_private", [
93
94
  "get_subkey_choice_from_bucket_signature",
94
95
  "prefetch_bucket",
95
96
  "prefetch_key",
97
+ "prefetch_subkey",
96
98
 
97
99
  # Copies of the public functions but with subtable/subkey interface
98
100
  # (only defined for hashes with subtables; otherwise None)
@@ -111,6 +113,7 @@ def create_SRHash(d):
111
113
  # The given d is copied and reduced to the required fields.
112
114
  # The hashfuncs tuple is reduced to a single ASCII bytestring.
113
115
  d0 = dict(d)
116
+ d0.setdefault('windowsize', 0) # only windowed hash types (hash_ccrh) set this
114
117
  d0['mem_bytes'] = d0['hashtable'].nbytes
115
118
  d0['slots_nonempty'] = compile_slots_nonempty(d)
116
119
  d0['slots_with_value'] = compile_slots_with_value(d)
@@ -821,10 +821,12 @@ def compile_classify_read_from_fastq(
821
821
  pass
822
822
 
823
823
  @contextmanager
824
- def cond_contextmgr(name, suffix, count, filt):
824
+ def cond_contextmgr(name, suffix, count, filt, enabled=True):
825
825
  if count and filt:
826
826
  raise ValueError("ERROR: cannot use both --count and --filter option at the same time.")
827
- if count:
827
+ if not enabled:
828
+ yield dummy_contextmgr()
829
+ elif count:
828
830
  yield dummy_contextmgr()
829
831
  elif filt:
830
832
  if "graft" in suffix:
@@ -840,7 +842,6 @@ def compile_classify_read_from_fastq(
840
842
  counts = [0, 0, 0, 0, 0, 0] # host, graft, amb., both, neither
841
843
  nprocessed = 0
842
844
 
843
- site_count = count and not use_start_sites
844
845
  with cond_contextmgr(path, f"-host.1.fq{compression}", count, filt) as host1, \
845
846
  cond_contextmgr(path, f"-host.2.fq{compression}", count, filt) as host2, \
846
847
  cond_contextmgr(path, f"-graft.1.fq{compression}", count, filt) as graft1, \
@@ -851,8 +852,8 @@ def compile_classify_read_from_fastq(
851
852
  cond_contextmgr(path, f"-both.2.fq{compression}", count, filt) as both2, \
852
853
  cond_contextmgr(path, f"-neither.1.fq{compression}", count, filt) as neither1, \
853
854
  cond_contextmgr(path, f"-neither.2.fq{compression}", count, filt) as neither2, \
854
- cond_contextmgr(path, f"-sites.1.fq{compression}", site_count, filt) as sites1, \
855
- cond_contextmgr(path, f"-sites.2.fq{compression}", site_count, filt) as sites2:
855
+ cond_contextmgr(path, f"-sites.1.fq{compression}", count, filt, use_start_sites) as sites1, \
856
+ cond_contextmgr(path, f"-sites.2.fq{compression}", count, filt, use_start_sites) as sites2:
856
857
  streams = ((host1, host2), (graft1, graft2), (ambiguous1, ambiguous2), (both1, both2), (neither1, neither2), (sites1, sites2))
857
858
 
858
859
  with ThreadPoolExecutor(max_workers=threads) as executor:
@@ -910,13 +911,12 @@ def compile_classify_read_from_fastq(
910
911
  def classify_read_from_fastq_single(fastqs, ht):
911
912
  counts = [0, 0, 0, 0, 0, 0]
912
913
  nprocessed = 0
913
- site_count = count and not use_start_sites
914
914
  with cond_contextmgr(path, f"-host.fq{compression}", count, filt) as host, \
915
915
  cond_contextmgr(path, f"-graft.fq{compression}", count, filt) as graft, \
916
916
  cond_contextmgr(path, f"-ambiguous.fq{compression}", count, filt) as ambiguous, \
917
917
  cond_contextmgr(path, f"-both.fq{compression}", count, filt) as both, \
918
918
  cond_contextmgr(path, f"-neither.fq{compression}", count, filt) as neither, \
919
- cond_contextmgr(path, f"-sites.fq{compression}", site_count, filt) as sites:
919
+ cond_contextmgr(path, f"-sites.fq{compression}", count, filt, use_start_sites) as sites:
920
920
  streams = (host, graft, ambiguous, both, neither, sites)
921
921
 
922
922
  running_jobs = []
@@ -211,9 +211,12 @@ def main(args):
211
211
 
212
212
  startweak = timestamp0(msg="- Computing weak k-mers...")
213
213
  nextchars = 3 # TODO:? parameterize via args?
214
- calculate_weak_set(
215
- h, mask.k, args.groupprefixlength, nextchars,
216
- rcmode=rcmode, threads=args.weakthreads)
214
+ if args.weakthreads > 0:
215
+ calculate_weak_set(
216
+ h, mask.k, args.groupprefixlength, nextchars,
217
+ rcmode=rcmode, threads=args.weakthreads)
218
+ else:
219
+ debugprint0('- WARNING: 0 weak threads provided. No weak k-mers computed.')
217
220
  timestamp0(startweak, msg="- weak k-mers: wall time")
218
221
  timestamp0(startweak, msg="- weak k-mers: wall time", minutes=True)
219
222
 
@@ -7,7 +7,7 @@ from numba import njit
7
7
 
8
8
  from ..lowlevel import debug
9
9
  from ..srhash import print_statistics
10
- from ..io.hashio import load_hash, dump_to_datafiles
10
+ from ..io.hashio import load_hash, _export_to_datafiles
11
11
 
12
12
 
13
13
  def print_dict(d, title=None):
@@ -63,7 +63,7 @@ def main(args):
63
63
  anyformat = (fmt is not None)
64
64
  prefix = args.outprefix
65
65
  nvalues = values.NVALUES
66
- filterfunc = compile_filter_function(args.filter)
66
+ filterfunc = compile_filter_function(getattr(args, "filter", None))
67
67
  if (prefix is None) and anyformat:
68
68
  prefix = args.hashname
69
69
  if prefix is not None:
@@ -75,7 +75,7 @@ def main(args):
75
75
  fmt = "native"
76
76
  elif fmt == "txt":
77
77
  fmt = "text"
78
- written = dump_to_datafiles(h, nvalues, infotup, prefix, fmt, filterfunc)
78
+ written = _export_to_datafiles(h, nvalues, infotup, prefix, fmt, filterfunc)
79
79
  debugprint0(f"- wrote {written} items")
80
80
  timestamp0(startout, msg="- done writing data; writing time")
81
81
 
@@ -53,7 +53,9 @@ def classify(p):
53
53
  help="only count reads or read pairs for each class, do not output any FASTQ")
54
54
  p.add_argument("--prefetchlevel", metavar="INT",
55
55
  type=restricted_number_type("from_0_to_2", int, [(">=", 0), ("<=", 2)]),
56
- help="amount of prefetching: none (0), second bucket (1), all buckets (2)")
56
+ help="amount of prefetching: none (0), second bucket (1), all buckets (2) [classify only]")
57
+ p.add_argument("--lookahead", metavar="INT", type=int, default=16,
58
+ help="classify2 only: number of k-mers to prefetch ahead during lookup [16]")
57
59
  p.add_argument("--chunksize", "-C", metavar="FLOAT_SIZE_MB", type=float, default=8.0,
58
60
  help="chunk size in MB; one chunk is allocated per thread.")
59
61
  p.add_argument("--chunkreads", "-R", metavar="INT", type=int,
@@ -146,6 +148,11 @@ SUBCOMMANDS = [
146
148
  classify,
147
149
  "xengsort_classify", "main",
148
150
  [f"{CFGPATH}/classify.yaml", 'config/classify.yaml', 'classify.yaml']),
151
+ ("classify2",
152
+ "sort/filter/count FASTQ reads (new producer/consumer pipeline; for comparison)",
153
+ classify,
154
+ "classify", "main",
155
+ [f"{CFGPATH}/classify.yaml", 'config/classify.yaml', 'classify.yaml']),
149
156
  ("info",
150
157
  "get information about a hash table and dump its data",
151
158
  info,
@@ -193,7 +200,7 @@ def get_argument_parser():
193
200
  chandler = 'error'
194
201
  sp = ArgumentParser(prog=name, description=helptext,
195
202
  default_config_files=default_configs)
196
- if name in ["classify", "index"]:
203
+ if name in ["classify", "classify2", "index"]:
197
204
  sp.add_argument('--cfg', "--config", action=ActionConfigFile)
198
205
  sp.add_argument("--func", default=(module, f_main), help=SUPPRESS)
199
206
  f_parser(sp)
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: xengsort
3
- Version: 2.2.1
3
+ Version: 2.2.2
4
4
  Summary: xenograft sorting of FASTQ reads into five categories: host, graft, both, neither, ambiguous.
5
5
  Author-email: Jens Zentgraf <zentgraf@cs.uni-saarland.de>, Sven Rahmann <rahmann@cs.uni-saarland.de>
6
6
  License: MIT License
@@ -52,6 +52,7 @@ This tool, xengsort, uses 3-way bucketed Cuckoo hashing to efficiently solve the
52
52
  For a description of the method an evaluation on several datasets, please see our
53
53
  [article in "Algorithms for Molecular Biology"](https://almob.biomedcentral.com/articles/10.1186/s13015-021-00181-w), or the initial [WABI 2020 publication](https://drops.dagstuhl.de/opus/volltexte/2020/12793/).
54
54
  (There was also a [preprint on bioRxiv](https://www.biorxiv.org/content/10.1101/2020.05.14.095604v2).)
55
+ If you use xengsort to sort converted reads (RRBS or WGBS) there is a [bioRxiv preprint](https://www.biorxiv.org/content/10.64898/2026.10.06.756952v1.abstract) available.
55
56
  BibTeX entrys for citation can be found at the end of this README file.
56
57
 
57
58
  In case of problems,
@@ -71,24 +72,23 @@ Thank you!
71
72
 
72
73
  We here provide an example showing how to run xengsort on a human-captured mouse exome (part of one of the datasets described in the paper).
73
74
  This tutorial uses the workflow management system `Snakemake` to run the example.
74
- This is, however, not necessary to use `xengsort`.
75
+ This is, however, not necessary to use `xengsort`.
75
76
  Please refer to the Usage Guide below to learn how to run `xengsort` on your own data manually.
76
77
 
77
78
 
78
79
  Our software `xengsort` is provided as a Python package.
79
- For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
80
+ For efficiency, it uses just-in-time compilation provided by the [numba](http://numba.pydata.org/) package.
80
81
  We here use the package manager [conda](https://docs.conda.io/en/latest/) to manage Python packages and to use a separate environment for `xengsort`.
81
82
 
82
83
 
83
- ### Install the conda package manager (miniconda)
84
- Go to https://docs.conda.io/en/latest/miniconda.html and download the Miniconda installer:
85
- Choose Python 3.10 (or higher), your operating system, and preferably the 64 bit version.
86
- Follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
84
+ ### Install the conda package manager (conda-forge)
85
+ Go to https://conda-forge.org/download/ and download the conda-forge installer:
86
+ Choose your operating system, follow the instructions of the installer and append the conda executable to your PATH (even if the installer does not recommend it).
87
87
  You can let the installer do it, or do it manually by editing your ``.bashrc`` or similar file under Linux or MacOS, or by editing the environment variables under Windows.
88
88
  To verify that the installation works, open a new terminal and execute
89
89
  ```
90
- conda --version # ideally 22.9.xx or higher
91
- python --version # ideally 3.10.xx or higher
90
+ conda --version # ideally 26.9.xx or higher
91
+ python --version # ideally 3.12.xx or higher
92
92
  ```
93
93
 
94
94
 
@@ -122,7 +122,7 @@ within the cloned directory tree.
122
122
 
123
123
 
124
124
  ### Create and activate a conda environment
125
- To run our software, a [conda](https://docs.conda.io/en/latest/) environment with the required libraries needs to be created.
125
+ To run our software, a [conda](https://conda-forge.org/docs/) environment with the required libraries needs to be created.
126
126
  A list of needed libraries is provided in the ``environment.yml`` file in the cloned repository;
127
127
  it can be used to create a new environment:
128
128
 
@@ -133,7 +133,7 @@ conda env create
133
133
  which will create an environment named ``xengsort`` with the required dependencies,
134
134
  using the provided ``environment.yml`` file in the same directory.
135
135
 
136
- A more explicit alternative is to run
136
+ A more explicit alternative is to run
137
137
  ```
138
138
  conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
139
139
  ```
@@ -217,11 +217,11 @@ To build an index for xengsort, several parameters must be provided, which are d
217
217
  First, a file name and a path for the index must be chosen.
218
218
  The index is stored in two files. We will use `myindex` to store the index in the current folder.
219
219
 
220
- Second, two reference genomes (host and graft) must be provided (in FASTA files).
221
- We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
220
+ Second, two reference genomes (host and graft) must be provided (in FASTA files).
221
+ We assume that they are given as `host.fa.gz` and `graft.fa.gz`.
222
222
  The files can be provided as an uncompressed file or compressed using `gzip`, `bzip2` or `xz`.
223
- The corresponding options are `-H` or `--host` and `-G` or `--graft`.
224
- Each option can take several arguments as files.
223
+ The corresponding options are `-H` or `--host` and `-G` or `--graft`.
224
+ Each option can take several arguments as files.
225
225
  ```
226
226
  xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000 [OPTIONS]
227
227
  ```
@@ -249,7 +249,7 @@ The final important parameter is about parallelization:
249
249
 
250
250
  Most of the parameters can also be provided in a config file (`.yaml`):
251
251
  - `--cfg` or `--config` defines the path the the config file.
252
-
252
+
253
253
  ### How to classify
254
254
 
255
255
  To classify a FASTQ sample (one or several single-end or paired-end files), make sure you are in an environment where xengsort and its dependencies are installed.
@@ -257,7 +257,7 @@ Then run the `xengsort classify` command with a previously built index, such as
257
257
  ```
258
258
  xengsort classify --index myindex --fastq single.fq.gz --prefix myresults --mode count
259
259
  ```
260
- for single-end reads, or
260
+ for single-end reads, or
261
261
  ```
262
262
  xengsort classify --index myindex --fastq paired.1.fq.gz --pairs paired.2.fq.gz --prefix myresults --mode count
263
263
  ```
@@ -275,7 +275,7 @@ The parameter `--prefix` or equivalently `--out` is required and defines the pre
275
275
  - `{prefix}.both.1.fq.gz`: reads that could originate from both
276
276
  - `{prefix}.neither.1.fq.gz`: reads that originate from neither host nor graft
277
277
  - `{prefix}.ambiguous.1.fq.gz`: (few) ambiguous reads that cannot be classified,
278
-
278
+
279
279
  and similarly with `.2.fq.gz`. For single-end reads, there is only `.fq.gz` (no numbers).
280
280
 
281
281
  The compression type can be specified using the `--compression` parameter.
@@ -301,7 +301,7 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 9_000_000_000 --b
301
301
  If an index is build using the bisulfite parameter, also the read is converted in the classification step.
302
302
  It is not necessary to change the parameters of the classification step.
303
303
  Currently, it only supports the count-based classification.
304
-
304
+ A detailed description on how the benchmarks on bisulfite sequencing data was performed can be found [here](https://github.com/S-Altend/MethSortBench/tree/main/03_read_sorting).
305
305
 
306
306
  Please report a [GitLab issue](https://gitlab.com/genomeinformatics/xengsort/-/issues) in case of problems.
307
307
 
@@ -315,13 +315,13 @@ Happy sorting!
315
315
  * **Snakemake throws errors when attempting to download the example dataset from SRA**.
316
316
 
317
317
  This may happen for several reasons:
318
- 1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
318
+ 1. The dataset is quite large, over 20 GB. Make sure you have sufficient space (also temporary space) and sufficient disk quota (on a shared system) to store all of the files.
319
319
  1. Another source of trouble may be that for some reason and older version of `sra-tools` gets called. We use the `fasterq-dump` tool to download the data, which did not exist in older verisons. To use a clean environment with a recent version of `sra-tools`, you can run snakemake as `snakemake -j 16 --use-conda`, which will generate a separate environment only for the download (this takes additional time, but may resolve the problem).
320
320
  1. In earlier versions, the dataset was downloaded from a different location which is no longer available. Please check that you are using at least v1.1.0 (`xengsort --version`). If not, please do a fresh clone and follow the tutorial to set up a new environment (delete the old one first) and run `pip install -e .`, as explained above in the tutorial.
321
321
 
322
322
  * **Indexing stops after a few seconds with a failure.**
323
323
 
324
- The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
324
+ The most likely cause of this is that you have not specified the size of the hash table (`-n` parameter).
325
325
  (From version 1.0.1, the `-n` parameter is required, and the error cannot occur anymore. Please update to the latest version!)
326
326
  For the human and mouse genomes this is approximately `-n 4_500_000_000` (4.5 billion).
327
327
  It is unfortunately a limitation of the implementation that the hash table size has to be specified in advance.
@@ -377,3 +377,14 @@ In addition, you may also cite the WABI 2020 proceedigs paper:
377
377
  annote = {Keywords: xenograft sorting, alignment-free method, Cuckoo hashing, k-mer}
378
378
  }
379
379
  ```
380
+
381
+ If you use xengsort to sort converted reads, please also cite our corresponding bioRxiv preprint:
382
+ ```
383
+ @article {Altendorfer2026.10.06.756952,
384
+ author = {Altendorfer, Stefan and Zentgraf, Jens and Schmitz, Johanna Elena and Plass, Christoph and Gerh{\"a}user, Clarissa and Rahmann, Sven},
385
+ title = {High-performance xenograft read-sorting for DNA methylation analysis},
386
+ year = {2026},
387
+ doi = {10.64898/2026.10.06.756952},
388
+ journal = {bioRxiv}
389
+ }
390
+ ```
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes
File without changes