xengsort 2.0.4__tar.gz → 2.0.7__tar.gz

{xengsort-2.0.4 → xengsort-2.0.7}/MANIFEST.in

@@ -1,4 +1,5 @@
-include xengsort/xengsort/*/*
+include xengsort/xengsort/*
 include xengsort/io/*
 include xengsort/lowlevel/*
 include xengsort/values/*
+include xengsort/config/*

{xengsort-2.0.4/xengsort.egg-info → xengsort-2.0.7}/PKG-INFO

@@ -1,9 +1,30 @@
 Metadata-Version: 2.1
 Name: xengsort
-Version: 2.0.4
-Summary: A fast xenograft read sorter based on space-efficient k-mer hashing
+Version: 2.0.7
+Summary: xenograft sorting of FASTQ reads into five categories: host, graft, both, neither, ambiguous.
 Author-email: Jens Zentgraf <zentgraf@cs.uni-saarland.de>, Sven Rahmann <rahmann@cs.uni-saarland.de>
-License: MIT
+License: MIT License
+        
+        Copyright (c) 2019-2022 Jens Zentgraf & Sven Rahmann
+        
+        Permission is hereby granted, free of charge, to any person obtaining a copy
+        of this software and associated documentation files (the "Software"), to deal
+        in the Software without restriction, including without limitation the rights
+        to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+        copies of the Software, and to permit persons to whom the Software is
+        furnished to do so, subject to the following conditions:
+        
+        The above copyright notice and this permission notice shall be included in all
+        copies or substantial portions of the Software.
+        
+        THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+        IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+        FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+        AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+        LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+        OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+        SOFTWARE.
+        
 Project-URL: Homepage, https://gitlab.com/genomeinformatics/xengsort
 Project-URL: Bug Tracker, https://gitlab.com/genomeinformatics/xengsort/issues
 Classifier: Programming Language :: Python :: 3
@@ -14,13 +35,14 @@ Classifier: Environment :: Console
 Classifier: Intended Audience :: Science/Research
 Classifier: Natural Language :: English
 Classifier: Topic :: Scientific/Engineering :: Bio-Informatics
-Requires-Python: ==3.11
+Requires-Python: >=3.12
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: numpy>=1.24
-Requires-Dist: numba>=0.57
+Requires-Dist: numpy>=2
+Requires-Dist: numba>=0.60
 Requires-Dist: pytest
-Requires-Dist: jsonargparse
+Requires-Dist: jsonargparse>=4.29.0
+Requires-Dist: pip
 
 # xengsort: Fast lightweight accurate xenograft sorting
 
@@ -116,7 +138,7 @@ conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
 ```
 which will do the same thing using the packages mentioned in `requirements.txt`.
 
-**Note:** While xengsort works on Windows/Mac without problems, some of the required packages may currently not exist for Windows/Mac.
+**Note:** While xengsort works on Linux without problems, some of the required packages may currently not exist for Windows/Mac.
 
 Setting up the environment may take some time, as conda searches and downloads packages.
 After all dependencies are resolved, you activate the environment and install the package from the repository into this environment.
@@ -127,7 +149,7 @@ pip install -e .  # install xengsort package using pip
 ```
 
 
-### Run the Snakemake example workflow
+## Run the Snakemake example workflow
 
 You can in principle remove `snakemake-minimal` and `sra-toolkit` from the list of packages.
 However, you will not be able to automatically run the provided example workflow or download sequence datasets from SRA.
@@ -170,7 +192,7 @@ This will run for quite some time (especially downloads and index creation), bes
 The results will appear in the `results/` directory.
 
 To run your own human/mouse xenograft analysis, you can continue to use the same index.
-All reference files, including the index in ZARR format (`.zarr`) are in the `ref/` directory.
+All reference files, including the index (`.hash` and `.info`) are in the `ref/` directory.
 You can follow the structure of the Snakefile to create your own custom workflow.
 Additional information on how to use `xengsort` is given in the usage guide below.
 
@@ -205,29 +227,27 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000  [O
 
 We must specify the size of the hash table:
 
-- `-n` or  `--nobjects`: number of k-mers that will be stored in the hash table. This depends on the used refernece genomes and must be estimated beforehand! As a precise estimate of the number of different k-mers can be difficult, you can err on the safe side and provide a generously large estimate, examine the final (low) load factor and then rebuild the index with a smaller `-n` parameter to aechieve the desired load. There are also some tools that quickly estimate the number of distinct k-mers in large files, such as [ntCard](https://github.com/bcgsc/ntCard) or [KmerEstimate](https://github.com/srbehera11/KmerEstimate). As a guide: Human and mouse genome and transcriptome together comprise around 4.5 billion 25-mers, as shown in the examples above.
+- `-n` or  `--nobjects`: number of k-mers that will be stored in the hash table. This depends on the used reference genomes and must be estimated beforehand! As a precise estimate of the number of different k-mers can be difficult, you can err on the safe side and provide a generously large estimate, examine the final (low) load factor and then rebuild the index with a smaller `-n` parameter to achieve the desired load. There are also some tools that quickly estimate the number of distinct k-mers in large files, such as [ntCard](https://github.com/bcgsc/ntCard) or [KmerEstimate](https://github.com/srbehera11/KmerEstimate). As a guide: Human and mouse genome and transcriptome together comprise around 4.5 billion 25-mers, as shown in the examples above.
 **This option must be specified; there is no default!**
 
 
 We may further specify additional properties of the hash table:
 
-- `-p` or `--pagesize` indicates how many elements can be stored in one bucket (or page). This is 4 by default.
+- `-b` or `--bucketsize` indicates how many elements can be stored in one bucket (or page). This is 4 by default.
 
 - `--fill` between 0.0 and 1.0 describes the desired fill rate or load factor of the hash table.
 Together with `-n`, the number of slots in the table is calculated as `ceil(n/fill)`. In our experiments we used 0.88. (The number of buckets is then the smallest odd integer that is at least `ceil(ceil(n/fill)/p)`.)
 
-- `--aligned` or `--unaligned`: indicates whether each bucket should consume a number of bits that is a power of 2. Using `--aligned` ensures that each bucket stays within the same cache line, but may waste space (padding bits), yielding faster speed but possibliy (much!) larger space requirements. With `--unaligned`, no bits are used for padding and buckets may cross cache line boundaries. This is slightly slower, but may save a little or a lot of space (depending on the bucket size in bits). The default is `--unaligned`, because the speed decrease is small and the memory savings can be significant.
+- `--aligned` or `--unaligned`: indicates whether each bucket should consume a number of bits that is a power of 2. Using `--aligned` ensures that each bucket stays within the same cache line, but may waste space (padding bits), yielding faster speed but possibly (much!) larger space requirements. With `--unaligned`, no bits are used for padding and buckets may cross cache line boundaries. This is slightly slower, but may save a little or a lot of space (depending on the bucket size in bits). The default is `--unaligned`, because the speed decrease is small and the memory savings can be significant.
 
 - `--hashfunctions` defines the parameters for the hash functions used to store the key-value pairs. If the parameter is unspecified, different random functions are chosen each time. The hash functions can be specified using a colon separated list: `--hashfunctions linear945:linear9123641:linear349341847`. It is recommended to have them chosen randomly unless you need strictly reproducible behavior, in which case the example given here is recommended.
 
-The final important parameter is about paralellization:
+The final important parameter is about parallelization:
 
-- `-T` defines how many threads are used to calculate weak *k*-mers.
-
-The following hash table options should not be modified from their default values and are for internal development only:
-
-- `-P` or `--parameters` can be used to combine all parameters stated above into one string (see the `--help` text)
+- `-W` or `--weakthreads` defines how many threads are used to calculate weak *k*-mers.
 
+Most of the parameters can also be provided in a config file (`.yaml`):
+- `--cfg` or `--config` defines the path the the config file.
  
 ### How to classify
 

{xengsort-2.0.4 → xengsort-2.0.7}/README.md

@@ -92,7 +92,7 @@ conda create --name xengsort -c conda-forge -c bioconda --file requirements.txt
 ```
 which will do the same thing using the packages mentioned in `requirements.txt`.
 
-**Note:** While xengsort works on Windows/Mac without problems, some of the required packages may currently not exist for Windows/Mac.
+**Note:** While xengsort works on Linux without problems, some of the required packages may currently not exist for Windows/Mac.
 
 Setting up the environment may take some time, as conda searches and downloads packages.
 After all dependencies are resolved, you activate the environment and install the package from the repository into this environment.
@@ -103,7 +103,7 @@ pip install -e .  # install xengsort package using pip
 ```
 
 
-### Run the Snakemake example workflow
+## Run the Snakemake example workflow
 
 You can in principle remove `snakemake-minimal` and `sra-toolkit` from the list of packages.
 However, you will not be able to automatically run the provided example workflow or download sequence datasets from SRA.
@@ -146,7 +146,7 @@ This will run for quite some time (especially downloads and index creation), bes
 The results will appear in the `results/` directory.
 
 To run your own human/mouse xenograft analysis, you can continue to use the same index.
-All reference files, including the index in ZARR format (`.zarr`) are in the `ref/` directory.
+All reference files, including the index (`.hash` and `.info`) are in the `ref/` directory.
 You can follow the structure of the Snakefile to create your own custom workflow.
 Additional information on how to use `xengsort` is given in the usage guide below.
 
@@ -181,29 +181,27 @@ xengsort index --index myindex -H host.fa.gz -G graft.fa.gz -n 4_500_000_000  [O
 
 We must specify the size of the hash table:
 
-- `-n` or  `--nobjects`: number of k-mers that will be stored in the hash table. This depends on the used refernece genomes and must be estimated beforehand! As a precise estimate of the number of different k-mers can be difficult, you can err on the safe side and provide a generously large estimate, examine the final (low) load factor and then rebuild the index with a smaller `-n` parameter to aechieve the desired load. There are also some tools that quickly estimate the number of distinct k-mers in large files, such as [ntCard](https://github.com/bcgsc/ntCard) or [KmerEstimate](https://github.com/srbehera11/KmerEstimate). As a guide: Human and mouse genome and transcriptome together comprise around 4.5 billion 25-mers, as shown in the examples above.
+- `-n` or  `--nobjects`: number of k-mers that will be stored in the hash table. This depends on the used reference genomes and must be estimated beforehand! As a precise estimate of the number of different k-mers can be difficult, you can err on the safe side and provide a generously large estimate, examine the final (low) load factor and then rebuild the index with a smaller `-n` parameter to achieve the desired load. There are also some tools that quickly estimate the number of distinct k-mers in large files, such as [ntCard](https://github.com/bcgsc/ntCard) or [KmerEstimate](https://github.com/srbehera11/KmerEstimate). As a guide: Human and mouse genome and transcriptome together comprise around 4.5 billion 25-mers, as shown in the examples above.
 **This option must be specified; there is no default!**
 
 
 We may further specify additional properties of the hash table:
 
-- `-p` or `--pagesize` indicates how many elements can be stored in one bucket (or page). This is 4 by default.
+- `-b` or `--bucketsize` indicates how many elements can be stored in one bucket (or page). This is 4 by default.
 
 - `--fill` between 0.0 and 1.0 describes the desired fill rate or load factor of the hash table.
 Together with `-n`, the number of slots in the table is calculated as `ceil(n/fill)`. In our experiments we used 0.88. (The number of buckets is then the smallest odd integer that is at least `ceil(ceil(n/fill)/p)`.)
 
-- `--aligned` or `--unaligned`: indicates whether each bucket should consume a number of bits that is a power of 2. Using `--aligned` ensures that each bucket stays within the same cache line, but may waste space (padding bits), yielding faster speed but possibliy (much!) larger space requirements. With `--unaligned`, no bits are used for padding and buckets may cross cache line boundaries. This is slightly slower, but may save a little or a lot of space (depending on the bucket size in bits). The default is `--unaligned`, because the speed decrease is small and the memory savings can be significant.
+- `--aligned` or `--unaligned`: indicates whether each bucket should consume a number of bits that is a power of 2. Using `--aligned` ensures that each bucket stays within the same cache line, but may waste space (padding bits), yielding faster speed but possibly (much!) larger space requirements. With `--unaligned`, no bits are used for padding and buckets may cross cache line boundaries. This is slightly slower, but may save a little or a lot of space (depending on the bucket size in bits). The default is `--unaligned`, because the speed decrease is small and the memory savings can be significant.
 
 - `--hashfunctions` defines the parameters for the hash functions used to store the key-value pairs. If the parameter is unspecified, different random functions are chosen each time. The hash functions can be specified using a colon separated list: `--hashfunctions linear945:linear9123641:linear349341847`. It is recommended to have them chosen randomly unless you need strictly reproducible behavior, in which case the example given here is recommended.
 
-The final important parameter is about paralellization:
+The final important parameter is about parallelization:
 
-- `-T` defines how many threads are used to calculate weak *k*-mers.
-
-The following hash table options should not be modified from their default values and are for internal development only:
-
-- `-P` or `--parameters` can be used to combine all parameters stated above into one string (see the `--help` text)
+- `-W` or `--weakthreads` defines how many threads are used to calculate weak *k*-mers.
 
+Most of the parameters can also be provided in a config file (`.yaml`):
+- `--cfg` or `--config` defines the path the the config file.
  
 ### How to classify
 

{xengsort-2.0.4 → xengsort-2.0.7}/pyproject.toml

@@ -1,17 +1,35 @@
 [build-system]
-requires = ["setuptools==68.2.2"]
+requires = ["setuptools>=69.0.3"]
 build-backend = "setuptools.build_meta"
 
+
 [project]
 name = "xengsort"
-version = "2.0.4"
+version = "2.0.7"
+
+license = {file = "LICENSE"}
+
 authors = [
   { name="Jens Zentgraf", email="zentgraf@cs.uni-saarland.de" },
   { name="Sven Rahmann", email="rahmann@cs.uni-saarland.de" }
 ]
-description = "A fast xenograft read sorter based on space-efficient k-mer hashing"
+
+description = "xenograft sorting of FASTQ reads into five categories: host, graft, both, neither, ambiguous."
+
 readme = "README.md"
-requires-python = "==3.11"
+
+requires-python = ">=3.12"
+
+dependencies = [
+  "numpy>=2",
+  "numba>=0.60",
+  "pytest",
+  "jsonargparse>=4.29.0",
+  "pip",
+]
+
+scripts = {xengsort = "xengsort.xengsort.xengsort_main:main"}
+
 classifiers = [
     "Programming Language :: Python :: 3",
     "License :: OSI Approved :: MIT License",
@@ -22,13 +40,10 @@ classifiers = [
     "Natural Language :: English",
     "Topic :: Scientific/Engineering :: Bio-Informatics"
 ]
-dependencies = [
-  "numpy>=1.24",
-  "numba>=0.57",
-  "pytest",
-  "jsonargparse",
-]
 
 [project.urls]
 "Homepage" = "https://gitlab.com/genomeinformatics/xengsort"
-"Bug Tracker" = "https://gitlab.com/genomeinformatics/xengsort/issues"
+"Bug Tracker" = "https://gitlab.com/genomeinformatics/xengsort/issues"
+
+[tool.setuptools]
+packages=["xengsort"]

xengsort-2.0.7/tests/test_trimmed_reads.py

@@ -0,0 +1,28 @@
+from fastcash.io.fastqio import fastq_chunks_paired
+
+def test_fastq_chunks_paired_file2_trimmed():
+    pair = ("tests/data/test_chunks1.fq", "tests/data/test_chunks2.fq")
+    reads1 = 0
+    reads2 = 0
+    for buf1, linemarks1, buf2, linemarks2 in fastq_chunks_paired(pair, bufsize=4000, maxreads=40):
+        assert len(linemarks1) == len(linemarks2)
+        for read in range(len(linemarks1)):
+            for nuc in range(linemarks2[read][1]-linemarks2[read][0]-1):
+                assert buf2[linemarks2[read][0]:linemarks2[read][1]][nuc] == buf1[linemarks1[read][0]:linemarks1[read][1]][nuc]
+        reads1 += len(linemarks1)
+        reads2 += len(linemarks2)
+        assert reads1 == reads2
+    assert reads1 == 60
+
+def test_fastq_chunks_paired_file1_trimmed():
+    pair = ("tests/data/test_chunks2.fq", "tests/data/test_chunks1.fq")
+    reads1 = 0
+    reads2 = 0
+    for buf1, linemarks1, buf2, linemarks2 in fastq_chunks_paired(pair, bufsize=4000, maxreads=40):
+        assert len(linemarks1) == len(linemarks2)
+        for read in range(len(linemarks1)):
+            for nuc in range(linemarks1[read][1]-linemarks1[read][0]-1):
+                assert buf2[linemarks2[read][0]:linemarks2[read][1]][nuc] == buf1[linemarks1[read][0]:linemarks1[read][1]][nuc]
+        reads1 += len(linemarks1)
+        reads2 += len(linemarks2)
+    assert reads1 == reads2 == 60

{xengsort-2.0.4 → xengsort-2.0.7}/xengsort/cptasks_kmers.py

@@ -1,5 +1,5 @@
 """
-xengsort/cptasks_kmers.py
+fastcash/cptasks_kmers.py
 Functions for compiling ConsumerProducerTasks for k-mers
 """
 
@@ -65,6 +65,7 @@ def compile_cptask_scatter_kmers_from_linemarked(
         if p >= outbufsize:
             if j >= 0:
                 outcontrol[j, 7] = outbufsize
+                outcontrol[j, 6] = 1  # buffer type k-mer codes
                 mark_buffer_for_reading(outcontrol, j)
                 stj = j - st * noutbuffers_per_subtable  # the "local" j for subtable st
                 assert 0 <= stj < noutbuffers_per_subtable
@@ -175,6 +176,221 @@ def compile_cptask_scatter_kmers_from_linemarked(
         assert outbuffers.shape[1] >= outbufsize
         assert outinfos.shape[1] == 0
 
+        wait_read = wait_write = 0
+        nactive = -1  # active input buffer
+        outidx = np.full(nsubtables, (-1), dtype=np.int32)
+        outpos = np.full(nsubtables, outbufsize, dtype=np.int32)
+        while True:
+            nactive, wait = find_buffer_for_reading(incontrol, nactive)
+            wait_read += wait
+            if nactive < 0:  # all finished
+                errorcode = int(nactive <= -2)
+                break
+            active_buffer = inbuffers[nactive]
+            buffer_type = incontrol[nactive, 6]
+            if buffer_type == 0:
+                nseqs = incontrol[nactive, 7]
+                errorcode = 0
+                linemarks = linemarkbuffers[nactive]
+                for i in range(nseqs):
+                    sq = active_buffer[linemarks[i, 0]:linemarks[i, 1]]
+                    quick_dna_to_2bits(sq)
+                    (wait, _errorcode) = _process(
+                        sq, outidx, outpos, outbuffers, outcontrol)
+                    wait_write += wait
+                    errorcode |= _errorcode
+                if errorcode:
+                    mark_my_buffers_failed(incontrol)
+                    debugprint2("- FAILED: cptask_scatter_kmers", errorcode)
+                    break
+            elif buffer_type == 1:
+                nkeys = incontrol[nactive, 7]
+                errorcode = 0
+                active_buffer = active_buffer.view(np.uint64)
+                for i in range(nkeys):
+                    key = active_buffer[i]
+                    (wait, _errorcode) = _store(
+                        key, outidx, outpos, outbuffers, outcontrol)
+                    wait_write += wait
+                    errorcode |= _errorcode
+                if errorcode:
+                    mark_my_buffers_failed(incontrol)
+                    debugprint2("- FAILED: cptask_scatter_kmers", errorcode)
+                    break
+            else:
+                raise ValueError("- buffer type {buffer_type} not supported in cptask_scatter_kmers")
+
+            mark_buffer_for_writing(incontrol, nactive)
+        # We break out of while True loop when all bufferes were marked as finished by FASTQ reader processes
+        # We then let the next task read our output buffers one final time.
+        if not errorcode:
+            for st in range(nsubtables):
+                j = outidx[st]
+                if j >= 0:
+                    # assert outpos[st] > 0
+                    outcontrol[j, 7] = outpos[st]
+                    mark_buffer_for_reading(outcontrol, j)
+            debugprint2("- ending: cptask_scatter_kmers", errorcode)
+        return (wait_read, wait_write, -errorcode)
+
+    return _cptask_scatter_kmers
+
+
+def compile_cptask_scatter_kmers_position_from_linemarked(
+        mask, values, rcmode, hf0,
+        nsubtables, noutbuffers_per_subtable, outbufsize):
+    """
+    Compile a cptask function that
+    creates (gapped) k-mers and value from linemarked buffers.
+    The Value is a compressed representation of chromosome and position.
+    - mask: a Mask object
+    - rcmode: how to treat reverse complements; f, r, b, max, min
+    - hf0: the 0-th hash function that determines the subtable
+    - nsubtables: the number of subtables
+    - noutbuffers_per_subtable: number of output buffers per subtable
+    - outbufsize: the output buffer size for which to compile this function
+    """
+    debugprint0, debugprint1, debugprint2 = debug.debugprint
+    both = (rcmode[0].lower() == "b")
+
+    k, w, tpl = mask.k, mask.w, mask.tuple
+    assert len(tpl) == k
+    if k < 1 or k > 32:
+        raise ValueError(f"Only 1 <= k <= 32 is supported, but {k=}.")
+    codemask = uint64(4**(k - 1) - 1)
+    revcomp, ccode = compile_revcomp_and_canonical_code(k, rcmode)
+    if mask.is_contiguous:
+        debugprint1(f"- compiling cptask_scatter_kmers for contiguous {k}-mers.")
+    else:
+        debugprint1(f"- compiling cptask_scatter_kmers for gapped ({k},{w})-mers: {mask.mask}.")
+
+    merge_seq_pos = values.merge_seq_pos
+
+    @njit(nogil=True, locals=dict(
+        code=uint64, st=uint64, sk=uint64,
+        j=int32, stj=int32, p=int32, wait=int64))
+    def _store(code, chrom, position, outidx, outpos, outbuffers, outcontrol):
+        st, sk = hf0(code)
+        j = outidx[st]
+        p = outpos[st]
+        error = 0
+        wait_write = 0  # TODO: do we count wait time correctly?
+        if p >= outbufsize:
+            if j >= 0:
+                outcontrol[j, 7] = outbufsize
+                outcontrol[j, 6] = 1  # buffer type k-mer codes
+                mark_buffer_for_reading(outcontrol, j)
+                stj = j - st * noutbuffers_per_subtable  # the "local" j for subtable st
+                assert 0 <= stj < noutbuffers_per_subtable
+            else:
+                assert j == -1
+                stj = -1
+            stoutcontrol = outcontrol[(st * noutbuffers_per_subtable):(st + 1) * noutbuffers_per_subtable]
+            stj, wait = find_buffer_for_writing(stoutcontrol, stj)
+            outidx[st] = j = (stj + st * noutbuffers_per_subtable)
+            outpos[st] = p = 0
+            error = outcontrol[j, 1]
+        outbuffers[j, p] = sk  # put subkey (not code!) into outbuffer
+        value = merge_seq_pos(chrom, position)
+        outbuffers[j, p + 1] = value
+        outpos[st] += 2
+        return (wait, error)
+
+    @njit(nogil=True, locals=dict(
+        code=uint64, i=int64, j=int64, c=uint64,
+        errorcode=int32, _errorcode=int32,
+        wait=int64, _wait=int64))
+    def _process_gapped(seq, chrom, offset, *allout):  # (outidx, outpos, outbuffers, outcontrol):
+        startpoints = seq.size - tpl[k - 1]
+        errorcode = 0
+        wait = 0
+        for i in range(startpoints):
+            code = 0
+            for j in tpl:
+                c = seq[i + j]
+                if c > 3:
+                    break
+                code = uint64(code << 2) | uint64(c)
+            else:  # no break
+                if both:
+                    _wait, errorcode = _store(code, chrom, offset + i, *allout)  # outidx, outpos, outbuffers, outcontrol)
+                    wait += wait
+                    _wait, _errorcode = _store(revcomp(code), chrom, offset + i, *allout)  # outidx, outpos, outbuffers, outcontrol)
+                    wait += _wait
+                    errorcode |= _errorcode
+                else:
+                    _wait, errorcode = _store(ccode(code), chrom, offset + i, *allout)  # outidx, outpos, outbuffers, outcontrol)
+                    wait += _wait
+            if errorcode != 0:
+                break
+        return (wait, errorcode)
+
+    @njit(nogil=True, locals=dict(
+        code=uint64, i=int64, j=int64, c=uint64, errorcode=int32,
+        wait=int64, _wait=int64))
+    def _process_contiguous(seq, chrom, offset, *allout):  # (outidx, outpos, outbuffers, outcontrol):
+        endpoint = seq.size - (k - 1)
+        valid = False
+        errorcode = 0
+        wait = 0
+        i = 0
+        while i < endpoint:
+            if not valid:
+                code = 0
+                for j in range(k):
+                    c = seq[i + j]
+                    if c > 3:
+                        i += j + 1  # skip invalid
+                        break
+                    code = (code << 2) | c
+                else:  # no break
+                    valid = True
+                if not valid:
+                    continue  # with while
+            else:  # was valid, we have an old code
+                c = seq[i + k - 1]
+                if c > 3:
+                    valid = False
+                    i += k  # skip invalid
+                    continue  # with while
+                code = ((code & codemask) << 2) | c
+            # at this point, we have a valid code
+            if both:
+                _wait, errorcode = _store(code, chrom, offset + i, *allout)  # outidx, outpos, outbuffers, outcontrol)
+                wait += _wait
+                _wait, _errorcode = _store(revcomp(code), chrom, offset + i, *allout)  # outidx, outpos, outbuffers, outcontrol)
+                wait += _wait
+                errorcode |= _errorcode
+            else:
+                _wait, errorcode = _store(ccode(code), chrom, offset + i, *allout)  # outidx, outpos, outbuffers, outcontrol)
+                wait += _wait
+            i += 1
+            if errorcode != 0:
+                break
+        return (wait, errorcode)
+
+    _process = _process_contiguous if mask.is_contiguous else _process_gapped
+
+    @njit(nogil=True, locals=dict(
+        nactive=int32, nseqs=int32, i=int32, j=int32, outj=int32,
+        wait=int64, wait_read=int64, wait_write=int64,
+        errorcode=int32, _errorcode=int32))
+    def _cptask_scatter_kmers(inbuffers, incontrol, ininfos, outbuffers, outcontrol, outinfos):
+        # find a linemarked buffer to read (among inbuffers)
+        # find and reserve nsubtables output buffers among the output buffers [outinfos]
+        # incontrol[i, 7]: number of FASTQ reads in input buffer i
+        # ininfos[i]: linemarks for input buffer i
+        # outcontrol[B, 7]: k-mers in buffer B
+        debugprint2("- running: cptask_scatter_kmers; outbuffers shape =", outbuffers.shape)
+        assert inbuffers.ndim == 2
+        assert outbuffers.ndim == 2
+        M, N = ininfos.shape
+        assert N % 6 == 0  # We have 6 entries per linemark. Start and end of record and read and chromosome and positions
+        linemarkbuffers = ininfos.reshape(M, N // 6, 6)
+        assert outbuffers.shape[0] == nsubtables * noutbuffers_per_subtable
+        assert outbuffers.shape[1] >= outbufsize
+        assert outinfos.shape[1] == 0
+
         wait_read = wait_write = 0
         nactive = -1  # active input buffer
         outidx = np.full(nsubtables, (-1), dtype=np.int32)
@@ -187,13 +403,18 @@ def compile_cptask_scatter_kmers_from_linemarked(
                 break
             active_buffer = inbuffers[nactive]
             nseqs = incontrol[nactive, 7]
+            buffer_type = incontrol[nactive, 6]
+            assert buffer_type == 0  # 0 means sequence buffer
+
             errorcode = 0
             linemarks = linemarkbuffers[nactive]
             for i in range(nseqs):
                 sq = active_buffer[linemarks[i, 0]:linemarks[i, 1]]
+                chromosome = linemarks[i, 4]
+                position = linemarks[i, 5]
                 quick_dna_to_2bits(sq)
                 (wait, _errorcode) = _process(
-                    sq, outidx, outpos, outbuffers, outcontrol)
+                    sq, chromosome, position, outidx, outpos, outbuffers, outcontrol)
                 wait_write += wait
                 errorcode |= _errorcode
             if errorcode:
@@ -201,7 +422,7 @@ def compile_cptask_scatter_kmers_from_linemarked(
                 debugprint2("- FAILED: cptask_scatter_kmers", errorcode)
                 break
             mark_buffer_for_writing(incontrol, nactive)
-        # We break out of while True loop when all bufferes were marked as finished by FASTQ reader processes
+        # We break out of while True loop when all buffers were marked as finished by FASTQ reader processes
         # We then let the next task read our output buffers one final time.
         if not errorcode:
             for st in range(nsubtables):
@@ -216,6 +437,68 @@ def compile_cptask_scatter_kmers_from_linemarked(
     return _cptask_scatter_kmers
 
 
+def compile_cptask_insert_subkey_value(
+        myhashtable,
+        maxfailures=0,
+        maxwalk=1000):
+    debugprint0, debugprint1, debugprint2 = debug.debugprint
+    update = myhashtable.private.update_ssk
+
+    @njit(nogil=True, locals=dict(
+        nactive=int32, value=uint64, mytotal=int64, failed=int64,
+        nsubkeys=int32, subkey=uint64, status=int32, result=uint64,
+        wait=int64, wait_read=int64, wait_write=int64))
+    def _cptask_insert_subkey_value(st, ht, ft, inbuffers, incontrol, ininfos, outbuffers, outcontrol, outinfos):
+        """
+        Insert subkeys and values of k-mers into subtables of a hash table.
+        """
+        # incontrol[i, 0]: buffer status
+        # incontrol[i, 1]: set to nonzero on error
+        # incontrol[i, 7]: k-mers (or subkeys) in input buffer i
+        assert inbuffers.ndim == 2
+        assert outbuffers.ndim == 2
+        assert outbuffers.shape[0] == 1
+        assert outbuffers.shape[1] >= maxwalk + 1
+        assert outinfos.shape[1] == 0
+        debugprint2("- running: cptask_insert_subkey_value; subtable", st)
+
+        wait_read = wait_write = 0
+        nactive = -1  # active input buffer
+        active_buffer = inbuffers[0]  # irrelevant
+        out = outbuffers[0]
+        failed = mytotal = 0
+        while True:
+            nactive, wait = find_buffer_for_reading(incontrol, nactive)
+            wait_read += wait
+            if nactive < 0:  # all finished
+                debugprint2("- ending: cptask_insert_subkey_value; subtable", st, "with", mytotal, "total and", failed, "failed k-mers.")
+                break
+            active_buffer = inbuffers[nactive]
+            nsubkeys = incontrol[nactive, 7]
+            buffer_type = incontrol[nactive, 6]
+            assert buffer_type == 1  # 1 means k-mer/subkey buffer
+            # TODO: NOTE: This is an interleaved subkey/value buffer.
+            # Is this a different type?
+
+            # consume the buffer
+            for i in range(0, nsubkeys, 2):
+                subkey = active_buffer[i]
+                value = active_buffer[i + 1]
+                status, result = update(ht, st, subkey, value)  # see docstring below
+                if status & 128 == 0:
+                    out[result] += 1
+                    failed += (status == 0)
+            mytotal += nsubkeys
+            mark_buffer_for_writing(incontrol, nactive)
+            if failed > maxfailures:
+                mark_my_buffers_failed(incontrol)
+                debugprint2("- FAILED: cptask_insert_subkey_value; subtable", st, "with", mytotal, "total and", failed, "failed k-mers.")
+                break
+        return (mytotal, failed, wait_read, wait_write, -(failed > 0))
+
+    return _cptask_insert_subkey_value
+
+
 def compile_cptask_insert_filtered_subkeys(
         myhashtable,
         myfilter,
@@ -229,7 +512,7 @@ def compile_cptask_insert_filtered_subkeys(
     else:
         wwo = "without"
 
-        @njit(nogil=True, inline="always")
+        @njit(nogil=True)
         def lookup_and_insert_in_subfilter(ft, st, subkey):
             return True
     update = myhashtable.private.update_ssk
@@ -268,6 +551,9 @@ def compile_cptask_insert_filtered_subkeys(
                 break
             active_buffer = inbuffers[nactive]
             nsubkeys = incontrol[nactive, 7]
+            buffer_type = incontrol[nactive, 6]
+            assert buffer_type == 1  # 1 means k-mer buffer (or subkeys)
+
             # consume the buffer
             for subkey in active_buffer[:nsubkeys]:
                 if lookup_and_insert_in_subfilter(ft, st, subkey):
@@ -316,11 +602,22 @@ def compile_cptask_update_existing_subkeys(
                 break
             active_buffer = inbuffers[nactive]
             nsubkeys = incontrol[nactive, 7]
+            buffer_type = incontrol[nactive, 6]
+            # print("&&& buffer_type", buffer_type)
+            assert buffer_type == 1  # 1 means k-mer buffer (0 would be sequence buffer)
+
             # consume the buffer
-            for subkey in active_buffer[:nsubkeys]:
-                status, result = update_existing(ht, st, subkey, value)  # see docstring below
-                if status & 128 == 0:
-                    failed += (status == 0)
+            for ii in range(nsubkeys):
+                subkey = active_buffer[ii]
+                _, _ = update_existing(ht, st, subkey, value)  # see docstring below
+                # status, result = update_existing(ht, st, subkey, value)  # see docstring below
+                # if status & 128:
+                #     print("update ->", result)
+                # else:
+                #     print("ignore unknown")
+                # if status & 128 == 0:
+                #     failed += (status == 0)  # TODO: should we not just ignore new stuff?
+                # The current implementation will count not directly insertable stuff as failures.
             mytotal += nsubkeys
             mark_buffer_for_writing(incontrol, nactive)
         debugprint2("- ending: cptask_update_existing_subkeys; subtable", st, "with", mytotal, "total and", failed, "failed k-mers.")