XspecT 0.2.5__tar.gz → 0.2.7__tar.gz

{xspect-0.2.5 → xspect-0.2.7}/.github/workflows/test.yml

@@ -13,9 +13,9 @@ jobs:
           run: |
             python -m pip install --upgrade pip
             pip install '.[test]'
-        - name: Download filters
+        - name: Download models
           run: |
-            xspect download-filters
+            xspect download-models
         - name: Test with pytest
           run: |
             pytest --cov

{xspect-0.2.5/src/XspecT.egg-info → xspect-0.2.7}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: XspecT
-Version: 0.2.5
+Version: 0.2.7
 Summary: Tool to monitor and characterize pathogens using Bloom filters.
 License: MIT License
         
@@ -63,7 +63,7 @@ Requires-Dist: pytest-cov; extra == "test"
 <img src="/docs/img/logo.png" height="50%" width="50%">
 
 <!-- start intro -->
-XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or sub-type level using [Bloom Filters] and a [Support Vector Machine]. It also identifies existing [blaOxa-genes] and provides a list of relevant research papers for further information.
+XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [Bloom Filters] and a [Support Vector Machine].
 <br/><br/>
 
 XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction the results are classified using a Support Vector Machine. 
@@ -88,14 +88,14 @@ pip install xspect
 Please note that Windows and Alpine Linux is currently not supported.
 
 ## Usage
-### Get the Bloomfilters
-To download basic pre-trained filters, you can use the built-in command:
+### Get the models
+To download basic pre-trained models, you can use the built-in command:
 ```
-xspect download-filters
+xspect download-models
 ```
-Additional species filters can be trained using:
+Additional species models can be trained using:
 ```
-xspect train you-ncbi-genus-name
+xspect train-species you-ncbi-genus-name
 ```
 
 ### How to run the web app
@@ -107,7 +107,7 @@ xspect api
 ### How to use the XspecT command line interface
 Run xspect with the configuration you want to run it with as arguments.
 ```
-xspect classify your-genus path/to/your/input-set
+xspect classify-species your-genus path/to/your/input-set
 ```
 For further instructions on how to use the command line interface, please refer to the [documentation] or execute:
 ```

{xspect-0.2.5 → xspect-0.2.7}/README.md

@@ -6,7 +6,7 @@
 <img src="/docs/img/logo.png" height="50%" width="50%">
 
 <!-- start intro -->
-XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or sub-type level using [Bloom Filters] and a [Support Vector Machine]. It also identifies existing [blaOxa-genes] and provides a list of relevant research papers for further information.
+XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [Bloom Filters] and a [Support Vector Machine].
 <br/><br/>
 
 XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction the results are classified using a Support Vector Machine. 
@@ -31,14 +31,14 @@ pip install xspect
 Please note that Windows and Alpine Linux is currently not supported.
 
 ## Usage
-### Get the Bloomfilters
-To download basic pre-trained filters, you can use the built-in command:
+### Get the models
+To download basic pre-trained models, you can use the built-in command:
 ```
-xspect download-filters
+xspect download-models
 ```
-Additional species filters can be trained using:
+Additional species models can be trained using:
 ```
-xspect train you-ncbi-genus-name
+xspect train-species you-ncbi-genus-name
 ```
 
 ### How to run the web app
@@ -50,7 +50,7 @@ xspect api
 ### How to use the XspecT command line interface
 Run xspect with the configuration you want to run it with as arguments.
 ```
-xspect classify your-genus path/to/your/input-set
+xspect classify-species your-genus path/to/your/input-set
 ```
 For further instructions on how to use the command line interface, please refer to the [documentation] or execute:
 ```

xspect-0.2.7/docs/cli.md

@@ -0,0 +1,80 @@
+# How to use the CLI
+
+XspecT comes with a built-in command line interface (CLI), which enables quick classifications without the need to use the web interface. The command line interface can also be used to download and train models.
+
+After installing XspecT, a list of available commands can be viewed by running:
+
+```bash
+xspect --help
+```
+
+## Model downloads
+
+A basic set of pre-trained models (Acinetobacter and Salonella) can be downloaded using the following command:
+
+```bash
+xspect download-models
+```
+
+For the moment, it is not possible to specify exactly which models should be downloaded.
+
+## Classification
+
+To classify samples, the command
+
+```bash
+xspect classify-species GENUS PATH
+```
+
+can be used, when `GENUS` refers to the NCBI genus name of your sample and `PATH` refers to the path to your sample *directory*. This command will classify the species of your sample within the given genus.
+
+The following options are available:
+
+```bash
+-m, --meta / --no-meta  Metagenome classification.
+-s, --step INTEGER      Sparse sampling step size (e. g. only every 500th
+                        kmer for step=500).
+--help                  Show this message and exit.
+```
+
+To speed up the analysis, only every nth kmer can be considered ("sparse sampling"). For example, to only consider every 10th kmer, run:
+
+```bash
+xspect classify-species -s 10 Acinetobacter path
+```
+
+### Metagenome Mode
+
+To analyze a sample in metagenome mode, the `-m`/`--meta` (`--no-meta`) option can be used:
+
+```bash
+xspect classify-species -m Acinetobacter path
+```
+
+Compared to normal XspecT species classification, this mode first identifies reads belonging to the given genus and continues classification only with the resulting reads, It is thus more suitable for metagenomic samples as the resulting runtime is decreased.
+
+### MLST Classification
+
+Samples can also be classified based on Multi-locus sequence type schemas. To MLST-classify a sample, run:
+
+```bash
+xspect classify-mlst -p path
+```
+
+## Model Training
+
+Models can be trained based on data from NCBI, which is automatically downloaded and processed by XspecT.
+
+To train a model, run the following command:
+
+```bash
+xspect train-species your-ncbi-genus
+```
+
+`you-ncbi-genus` can be a genus name from NCBI or an NCBI taxonomy ID.
+
+To train models for MLST classifications, run:
+
+```bash
+xspect train-mlst
+```

xspect-0.2.7/docs/input_data.md

@@ -0,0 +1,2 @@
+# Input Data
+XspecT is able to use either raw sequence-reads (FASTQ-format .fq/.fastq) or already assembled genomes (FASTA-format .fasta/.fna). Using sequence-reads saves up the assembly process but high-quality reads with a low error-rate are needed (e.g. Illumina-reads).

{xspect-0.2.5 → xspect-0.2.7}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "XspecT"
-version = "0.2.5"
+version = "0.2.7"
 description = "Tool to monitor and characterize pathogens using Bloom filters."
 readme = {file = "README.md", content-type = "text/markdown"}
 license = {file = "LICENSE"}

{xspect-0.2.5 → xspect-0.2.7/src/XspecT.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.2
 Name: XspecT
-Version: 0.2.5
+Version: 0.2.7
 Summary: Tool to monitor and characterize pathogens using Bloom filters.
 License: MIT License
         
@@ -63,7 +63,7 @@ Requires-Dist: pytest-cov; extra == "test"
 <img src="/docs/img/logo.png" height="50%" width="50%">
 
 <!-- start intro -->
-XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or sub-type level using [Bloom Filters] and a [Support Vector Machine]. It also identifies existing [blaOxa-genes] and provides a list of relevant research papers for further information.
+XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [Bloom Filters] and a [Support Vector Machine].
 <br/><br/>
 
 XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction the results are classified using a Support Vector Machine. 
@@ -88,14 +88,14 @@ pip install xspect
 Please note that Windows and Alpine Linux is currently not supported.
 
 ## Usage
-### Get the Bloomfilters
-To download basic pre-trained filters, you can use the built-in command:
+### Get the models
+To download basic pre-trained models, you can use the built-in command:
 ```
-xspect download-filters
+xspect download-models
 ```
-Additional species filters can be trained using:
+Additional species models can be trained using:
 ```
-xspect train you-ncbi-genus-name
+xspect train-species you-ncbi-genus-name
 ```
 
 ### How to run the web app
@@ -107,7 +107,7 @@ xspect api
 ### How to use the XspecT command line interface
 Run xspect with the configuration you want to run it with as arguments.
 ```
-xspect classify your-genus path/to/your/input-set
+xspect classify-species your-genus path/to/your/input-set
 ```
 For further instructions on how to use the command line interface, please refer to the [documentation] or execute:
 ```

{xspect-0.2.5 → xspect-0.2.7}/src/XspecT.egg-info/SOURCES.txt

@@ -51,7 +51,7 @@ src/XspecT.egg-info/requires.txt
 src/XspecT.egg-info/top_level.txt
 src/xspect/__init__.py
 src/xspect/definitions.py
-src/xspect/download_filters.py
+src/xspect/download_models.py
 src/xspect/fastapi.py
 src/xspect/file_io.py
 src/xspect/main.py
@@ -59,7 +59,11 @@ src/xspect/model_management.py
 src/xspect/pipeline.py
 src/xspect/run.py
 src/xspect/train.py
+src/xspect/mlst_feature/__init__.py
+src/xspect/mlst_feature/mlst_helper.py
+src/xspect/mlst_feature/pub_mlst_handler.py
 src/xspect/models/__init__.py
+src/xspect/models/probabilistic_filter_mlst_model.py
 src/xspect/models/probabilistic_filter_model.py
 src/xspect/models/probabilistic_filter_svm_model.py
 src/xspect/models/probabilistic_single_filter_model.py
@@ -67,7 +71,6 @@ src/xspect/models/result.py
 src/xspect/train_filter/__init__.py
 src/xspect/train_filter/create_svm.py
 src/xspect/train_filter/extract_and_concatenate.py
-src/xspect/train_filter/html_scrap.py
 src/xspect/train_filter/ncbi_api/__init__.py
 src/xspect/train_filter/ncbi_api/download_assemblies.py
 src/xspect/train_filter/ncbi_api/ncbi_assembly_metadata.py
@@ -80,7 +83,9 @@ tests/test_file_io.py
 tests/test_model_management.py
 tests/test_model_result.py
 tests/test_pipeline.py
+tests/test_probabilisitc_filter_mlst_model.py
 tests/test_probabilistic_filter_model.py
 tests/test_probabilistic_filter_svm_model.py
 tests/test_probabilistic_single_filter_model.py
+tests/test_pub_mlst_handler.py
 tests/test_train.py

{xspect-0.2.5 → xspect-0.2.7}/src/xspect/definitions.py

@@ -40,3 +40,10 @@ def get_xspect_runs_path():
     runs_path = get_xspect_root_path() / "runs"
     runs_path.mkdir(exist_ok=True, parents=True)
     return runs_path
+
+
+def get_xspect_mlst_path():
+    """Return the path to the XspecT runs directory."""
+    mlst_path = get_xspect_root_path() / "mlst"
+    mlst_path.mkdir(exist_ok=True, parents=True)
+    return mlst_path

xspect-0.2.5/src/xspect/download_filters.py → xspect-0.2.7/src/xspect/download_models.py

@@ -7,8 +7,8 @@ import requests
 from xspect.definitions import get_xspect_model_path, get_xspect_tmp_path
 
 
-def download_test_filters(url):
-    """Download filters."""
+def download_test_models(url):
+    """Download models."""
 
     download_path = get_xspect_tmp_path() / "models.zip"
     extract_path = get_xspect_tmp_path() / "extracted_models"

{xspect-0.2.5 → xspect-0.2.7}/src/xspect/fastapi.py

@@ -5,7 +5,7 @@ from pathlib import Path
 from shutil import copyfileobj
 from fastapi import FastAPI, UploadFile, BackgroundTasks
 from xspect.definitions import get_xspect_runs_path, get_xspect_upload_path
-from xspect.download_filters import download_test_filters
+from xspect.download_models import download_test_models
 import xspect.model_management as mm
 from xspect.models.result import StepType
 from xspect.pipeline import ModelExecution, Pipeline, PipelineStep
@@ -17,7 +17,7 @@ app = FastAPI()
 @app.get("/download-filters")
 def download_filters():
     """Download filters."""
-    download_test_filters("https://xspect2.s3.eu-central-1.amazonaws.com/models.zip")
+    download_test_models("https://xspect2.s3.eu-central-1.amazonaws.com/models.zip")
 
 
 @app.get("/classify")

{xspect-0.2.5 → xspect-0.2.7}/src/xspect/main.py

@@ -6,13 +6,23 @@ import uuid
 import click
 import uvicorn
 from xspect import fastapi
-from xspect.download_filters import download_test_filters
+from xspect.download_models import download_test_models
 from xspect.train import train_ncbi
 from xspect.models.result import (
     StepType,
 )
-from xspect.definitions import get_xspect_runs_path, fasta_endings, fastq_endings
+from xspect.definitions import (
+    get_xspect_runs_path,
+    fasta_endings,
+    fastq_endings,
+    get_xspect_model_path,
+)
 from xspect.pipeline import ModelExecution, Pipeline, PipelineStep
+from xspect.mlst_feature.mlst_helper import pick_scheme, pick_scheme_from_models_dir
+from xspect.mlst_feature.pub_mlst_handler import PubMLSTHandler
+from xspect.models.probabilistic_filter_mlst_model import (
+    ProbabilisticFilterMlstSchemeModel,
+)
 
 
 @click.group()
@@ -22,10 +32,10 @@ def cli():
 
 
 @cli.command()
-def download_filters():
-    """Download filters."""
-    click.echo("Downloading filters, this may take a while...")
-    download_test_filters("https://xspect2.s3.eu-central-1.amazonaws.com/models.zip")
+def download_models():
+    """Download models."""
+    click.echo("Downloading models, this may take a while...")
+    download_test_models("https://xspect2.s3.eu-central-1.amazonaws.com/models.zip")
 
 
 @cli.command()
@@ -43,7 +53,7 @@ def download_filters():
     help="Sparse sampling step size (e. g. only every 500th kmer for step=500).",
     default=1,
 )
-def classify(genus, path, meta, step):
+def classify_species(genus, path, meta, step):
     """Classify sample(s) from file or directory PATH."""
     click.echo("Classifying...")
     click.echo(f"Step: {step}")
@@ -105,7 +115,7 @@ def classify(genus, path, meta, step):
     help="SVM Sparse sampling step size (e. g. only every 500th kmer for step=500).",
     default=1,
 )
-def train(genus, bf_assembly_path, svm_assembly_path, svm_step):
+def train_species(genus, bf_assembly_path, svm_assembly_path, svm_step):
     """Train model."""
 
     if bf_assembly_path or svm_assembly_path:
@@ -118,6 +128,49 @@ def train(genus, bf_assembly_path, svm_assembly_path, svm_step):
         raise click.ClickException(str(e)) from e
 
 
+@cli.command()
+@click.option(
+    "-c",
+    "--choose_schemes",
+    is_flag=True,
+    help="Choose your own schemes."
+    "Default setting is Oxford and Pasteur scheme of A.baumannii.",
+)
+def train_mlst(choose_schemes):
+    """Download alleles and train bloom filters."""
+    click.echo("Updating alleles")
+    handler = PubMLSTHandler()
+    handler.download_alleles(choose_schemes)
+    click.echo("Download finished")
+    scheme_path = pick_scheme(handler.get_scheme_paths())
+    species_name = str(scheme_path).split("/")[-2]
+    scheme_name = str(scheme_path).split("/")[-1]
+    model = ProbabilisticFilterMlstSchemeModel(
+        31, f"{species_name}:{scheme_name}", get_xspect_model_path()
+    )
+    click.echo("Creating mlst model")
+    model.fit(scheme_path)
+    model.save()
+    click.echo(f"Saved at {model.cobs_path}")
+
+
+@cli.command()
+@click.option(
+    "-p",
+    "--path",
+    help="Path to FASTA-file for mlst identification.",
+    type=click.Path(exists=True, dir_okay=True, file_okay=True),
+)
+def classify_mlst(path):
+    """MLST classify a sample."""
+    click.echo("Classifying...")
+    path = Path(path)
+    scheme_path = pick_scheme_from_models_dir()
+    model = ProbabilisticFilterMlstSchemeModel.load(scheme_path)
+    model.predict(scheme_path, path).save(model.model_display_name, path)
+    click.echo(f"Run saved at {get_xspect_runs_path()}.")
+
+
 @cli.command()
 def api():
     """Open the XspecT FastAPI."""

xspect-0.2.7/src/xspect/mlst_feature/mlst_helper.py

@@ -0,0 +1,155 @@
+"""Module for utility functions used in other modules regarding MLST."""
+
+__author__ = "Cetin, Oemer"
+
+import requests
+import json
+from io import StringIO
+from pathlib import Path
+from Bio import SeqIO
+from xspect.definitions import get_xspect_model_path, get_xspect_runs_path
+
+
+def create_fasta_files(locus_path: Path, fasta_batch: str):
+    """Create Fasta-Files for every allele of a locus."""
+    # fasta_batch = full string of a fasta file containing every allele sequence of a locus
+    for record in SeqIO.parse(StringIO(fasta_batch), "fasta"):
+        number = record.id.split("_")[-1]  # example id = Oxf_cpn60_263
+        output_fasta_file = locus_path / f"Allele_ID_{number}.fasta"
+        if output_fasta_file.exists():
+            continue  # Ignore existing ones
+        with open(output_fasta_file, "w") as allele:
+            SeqIO.write(record, allele, "fasta")
+
+
+def pick_species_number_from_db(available_species: dict) -> str:
+    """Returns the chosen species from all available ones in the database."""
+    # The "database" string can look like this: pubmlst_abaumannii_seqdef
+    for counter, database in available_species.items():
+        print(str(counter) + ":" + database.split("_")[1])
+    print("\nPick one of the above databases")
+    while True:
+        try:
+            choice = input("Choose a species by selecting the corresponding number:")
+            if int(choice) in available_species.keys():
+                chosen_species = available_species.get(int(choice))
+                return chosen_species
+            else:
+                print(
+                    "Wrong input! Try again with a number that is available in the list above."
+                )
+        except ValueError:
+            print(
+                "Wrong input! Try again with a number that is available in the list above."
+            )
+
+
+def pick_scheme_number_from_db(available_schemes: dict) -> str:
+    """Returns the chosen schemes from all available ones of a species."""
+    # List all available schemes of a species database
+    for counter, scheme in available_schemes.items():
+        print(str(counter) + ":" + scheme[0])
+    print("\nPick any available scheme that is listed for download")
+    while True:
+        try:
+            choice = input("Choose a scheme by selecting the corresponding number:")
+            if int(choice) in available_schemes.keys():
+                chosen_scheme = available_schemes.get(int(choice))[1]
+                return chosen_scheme
+            else:
+                print(
+                    "Wrong input! Try again with a number that is available in the above list."
+                )
+        except ValueError:
+            print(
+                "Wrong input! Try again with a number that is available in the above list."
+            )
+
+
+def scheme_list_to_dict(scheme_list: list[str]):
+    """Converts the scheme list attribute into a dictionary with a number as the key."""
+    return dict(zip(range(1, len(scheme_list) + 1), scheme_list))
+
+
+def pick_scheme_from_models_dir() -> Path:
+    """Returns the chosen scheme from models that have been fitted prior."""
+    schemes = {}
+    counter = 1
+    for entry in sorted((get_xspect_model_path() / "MLST").iterdir()):
+        schemes[counter] = entry
+        counter += 1
+    return pick_scheme(schemes)
+
+
+def pick_scheme(available_schemes: dict) -> Path:
+    """Returns the chosen scheme from the scheme list."""
+    if not available_schemes:
+        raise ValueError("No scheme has been chosen for download yet!")
+
+    if len(available_schemes.items()) == 1:
+        return next(iter(available_schemes.values()))
+
+    # List available schemes
+    for counter, scheme in available_schemes.items():
+        # For Strain Typing with an API-POST Request to the db
+        if str(scheme).startswith("http"):
+            scheme_json = requests.get(scheme).json()
+            print(str(counter) + ":" + scheme_json["description"])
+
+        # To pick a scheme after download for fitting
+        else:
+            print(str(counter) + ":" + str(scheme).split("/")[-1])
+
+    print("\nPick a scheme for strain type prediction")
+    while True:
+        try:
+            choice = input("Choose a scheme by selecting the corresponding number:")
+            if int(choice) in available_schemes.keys():
+                chosen_scheme = available_schemes.get(int(choice))
+                return chosen_scheme
+            else:
+                print(
+                    "Wrong input! Try again with a number that is available in the above list."
+                )
+        except ValueError:
+            print(
+                "Wrong input! Try again with a number that is available in the above list."
+            )
+
+
+class MlstResult:
+    """Class for storing mlst results."""
+
+    def __init__(
+        self,
+        scheme_model: str,
+        steps: int,
+        hits: dict[str, list[dict]],
+    ):
+        self.scheme_model = scheme_model
+        self.steps = steps
+        self.hits = hits
+
+    def get_results(self) -> dict:
+        """Stores the result of a prediction in a dictionary."""
+        results = {seq_id: result for seq_id, result in self.hits.items()}
+        return results
+
+    def to_dict(self) -> dict:
+        """Converts all attributes into one dictionary."""
+        result = {
+            "Scheme": self.scheme_model,
+            "Steps": self.steps,
+            "Results": self.get_results(),
+        }
+        return result
+
+    def save(self, display: str, file_path: Path) -> None:
+        """Saves the result inside the "runs" directory"""
+        file_name = str(file_path).split("/")[-1]
+        json_path = get_xspect_runs_path() / "MLST" / f"{file_name}-{display}.json"
+        json_path.parent.mkdir(exist_ok=True, parents=True)
+        json_object = json.dumps(self.to_dict(), indent=4)
+
+        with open(json_path, "w", encoding="utf-8") as file:
+            file.write(json_object)