XspecT 0.5.3__tar.gz → 0.6.0__tar.gz

{xspect-0.5.3 → xspect-0.6.0}/.gitignore

@@ -177,4 +177,14 @@ out.png
 
 xspect-data/
 
-.devcontainer/
+.devcontainer/
+
+# Nextflow
+.nextflow.log*
+.nextflow/
+work/
+data/
+results/
+
+# Slurm
+slurm-*

{xspect-0.5.3/src/XspecT.egg-info → xspect-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: XspecT
-Version: 0.5.3
+Version: 0.6.0
 Summary: Tool to monitor and characterize pathogens using Bloom filters.
 License: MIT License
         
@@ -45,6 +45,9 @@ Requires-Dist: xxhash
 Requires-Dist: fastapi
 Requires-Dist: uvicorn
 Requires-Dist: python-multipart
+Requires-Dist: mappy
+Requires-Dist: pysam
+Requires-Dist: numpy
 Provides-Extra: docs
 Requires-Dist: mkdocs-material; extra == "docs"
 Requires-Dist: mkdocs-include-markdown-plugin; extra == "docs"

xspect-0.6.0/docs/benchmark.md

@@ -0,0 +1,34 @@
+# Benchmark
+
+XspecT is a tool designed for fast and accurate species classification of genome assemblies and simulated reads. To evaluate its classification accuracy, we conducted a benchmark using a set of Acinetobacter genomes.
+
+The benchmark was performed by first download all available Acinetobacter genomes from Genbank, filtered on a passed ("OK") taxonomy check status. Genomes assigned to strain IDs were remapped to their respective species IDs, after which genomes with species IDs not contained in XspecT's Acinetobacter model were removed. The remaining genomes were then used to classify both assemblies and simulated reads generated from them. Simulated reads were generated by first filtering on genomes that were not part of the training data and that were categorized as "complete" by NCBI. The reads were then simulated from the longest contig of each genome (assumed to be the chromosome) using a custom Python script. Up to three genomes were selected per species. 100 000 reads were simulated for each genome, with a read length of 100 bp and no simulated sequencing errors. The reads were then classified using XspecT with predictions based on the maximum-scoring species.
+
+## Benchmark Results
+
+The benchmark results show that XspecT achieves high classification accuracy, with an overall accuracy of nearly 100% for whole genomes and 82% for simulated reads. However, the low macro-average F1 score (0.41) for the read dataset highlights a substantial class imbalance.
+
+| Dataset   | Total Samples | Matches   | Mismatches | Match Rate | Mismatch Rate | Accuracy | Macro Avg F1 | Weighted Avg F1 |
+|-----------|--------------:|----------:|-----------:|-----------:|--------------:|---------:|-------------:|----------------:|
+| Assembly  | 44,905        | 44,879    | 26         | 99.94%     | 0.06%         | ≈1.00    | 0.95         | 1.00            |
+| Reads     | 9,200,000     | 7,526,902 | 1,673,098  | 81.81%     | 18.19%        | 0.82     | 0.41         | 0.87            |
+
+## Running the benchmark yourself
+
+To benchmark XspecT performance yourself, you can use the Nextflow workflow provided in the `scripts/benchmark` directory. This workflow allows you to run XspecT on a set of samples and measure species classification accuracy on both genome assemblies, as well as on simulated reads.
+
+Before you run the benchmark, you first need to download benchmarking data to the `data` directory, for example from NCBI. To do so, you can use the bash script in `scripts/benchmark-data` to download the data using the [NCBI Datasets CLI](https://www.ncbi.nlm.nih.gov/datasets/docs/v2/command-line-tools/download-and-install/), which needs to be installed first. The script will download all available Acinetobacter genomes, as well as taxonomic data.
+
+To run the benchmark, install [Nextflow](https://www.nextflow.io/docs/latest/install.html) and run the following command:
+
+```bash
+nextflow run scripts/benchmark
+```
+
+This will execute the benchmark workflow, which will classify the samples, as well as reads generated from them, using XspecT. The results will be saved in the `results` directory:
+
+- `results/classifications.tsv` for the classifications of the assemblies
+- `results/read_classifications.tsv` for the classifications of the simulated reads
+- `results/confusion_matrix.png` for the confusion matrix of genome assembly classifications
+- `results/mismatches_confusion_matrix.png` for a confusion matrix filtered on mismatches of genome assembly classifications
+- `results/stats.txt` for the statistics of the benchmark run

{xspect-0.5.3 → xspect-0.6.0}/docs/cli.md

@@ -12,7 +12,7 @@ In general, XspecT commands will prompt you for parameters if they are not provi
 
 ## Model Management
 
-At its core, XspecT uses models to classify and filter samples. These models are based on kmer indices trained on publicly availabel genomes as well as, possibly, a support vector machine (SVM) classifier.
+At its core, XspecT uses models to classify and filter samples. These models are based on kmer indices trained on publicly available genomes as well as, possibly, a support vector machine (SVM) classifier.
 
 To manage models, the `xspect models` command can be used. This command allows you to download, train, and view available models.
 
@@ -43,6 +43,8 @@ To train a model with NCBI data, run the following command:
 xspect models train ncbi
 ```
 
+By default, XspecT filters out NCBI accessions that do not meed minimum N50 thresholds, have an inconclusive taxonomy check status, or are deemed atypical by NCBI. Furthermore, species with "Candidatus" and "sp." in their species names are filtered out. To disable filtering behavior, use the respective flag (see `xspect models train ncbi --help`).
+
 If you would like to train models with manually curated data from a directory, you can use:
 
 ```bash
@@ -114,16 +116,24 @@ xspect classify species --sparse-sampling-step 10 Acinetobacter path
 
 This will only consider every 10th kmer in the sample.
 
+### Inclusion of display names
+By default, the classification results show only the taxonomy ID of each species along with its corresponding score for better readability. To display the full names associated with each taxonomy ID, you can use the `--display-names` (or `-n`) option:
+
+```bash
+xspect classify species --display-names Acinetobacter path
+```
+The output will then be formatted as: `Taxonomy_ID - Display_Name: Score` for each species.
+
 ### MLST Classification
 
 Samples can also be classified based on Multi-locus sequence type schemas. To MLST-classify a sample, run:
 
 ```bash
-xspect classify-mlst -p path
+xspect classify mlst
 ```
 
 ## Filtering
-XspecT can also be used to filter samples based on their classification results. This is useful when analyzing metagenome samples, for example when looking at genomic bycatch.
+XspecT can also be used to filter samples based on their classification results. This is useful when analyzing metagenomic samples, for example when looking at genomic bycatch.
 
 To filter samples, the command `xspect filter` can be used. This command will filter the samples based on the specified criteria.
 

{xspect-0.5.3 → xspect-0.6.0}/docs/contributing.md

@@ -20,11 +20,14 @@ Get started by cloning the repository:
 git clone https://github.com/BIONF/XspecT2.git
 ```
 
-You then need to build the web application using Vite. Navigate to the `xspect-web` directory and run the build command, which will also watch for changes:
+You then need to build the web application using Vite. Navigate to the `xspect-web` directory, install dependencies, and run the build command, which will also watch for changes:
 ```bash
 cd XspecT2/src/xspect/xspect-web
 ```
 ```bash
+npm i
+```
+```bash
 npx vite build --watch
 ```
 

{xspect-0.5.3 → xspect-0.6.0}/mkdocs.yml

@@ -15,5 +15,7 @@ nav:
   - Quickstart: quickstart.md
   - CLI: cli.md
   - "Web App": web.md
-  - "Understanding XspecT": understanding.md
+  - "Understanding XspecT":
+    - understanding.md
+    - benchmark.md
   - Contributing: contributing.md

{xspect-0.5.3 → xspect-0.6.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "XspecT"
-version = "0.5.3"
+version = "0.6.0"
 description = "Tool to monitor and characterize pathogens using Bloom filters."
 readme = {file = "README.md", content-type = "text/markdown"}
 license = {file = "LICENSE"}
@@ -18,7 +18,10 @@ dependencies = [
   "xxhash",
   "fastapi",
   "uvicorn",
-  "python-multipart"
+  "python-multipart",
+  "mappy",
+  "pysam",
+  "numpy"
   ]
 classifiers = [
   "Intended Audience :: Developers",

xspect-0.6.0/scripts/benchmark/classify/main.nf

@@ -0,0 +1,22 @@
+process classifySample {
+  conda "./scripts/benchmark/environment.yml"
+  cpus 4
+  memory '32 GB'
+
+  input:
+  path sample
+
+  output:
+  path "${sample.baseName}.json"
+
+  script:
+  """
+  xspect classify species -g Acinetobacter -i ${sample} -o ${sample.baseName}.json
+  """
+
+  stub:
+  """
+  mkdir -p results
+  touch results/${sample.baseName}.json
+  """
+}

xspect-0.6.0/scripts/benchmark/environment.yml

@@ -0,0 +1,7 @@
+name: xspect-benchmark
+channels:
+  - conda-forge
+dependencies:
+  - pip
+  - pip:
+      - XspecT