XspecT 0.2.7__tar.gz → 0.4.0__tar.gz

{xspect-0.2.7 → xspect-0.4.0}/.github/workflows/test.yml

@@ -15,7 +15,7 @@ jobs:
             pip install '.[test]'
         - name: Download models
           run: |
-            xspect download-models
+            xspect models download
         - name: Test with pytest
           run: |
             pytest --cov

{xspect-0.2.7/src/XspecT.egg-info → xspect-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: XspecT
-Version: 0.2.7
+Version: 0.4.0
 Summary: Tool to monitor and characterize pathogens using Bloom filters.
 License: MIT License
         
@@ -54,34 +54,27 @@ Requires-Dist: sphinx-autobuild; extra == "docs"
 Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
 Requires-Dist: pytest-cov; extra == "test"
+Dynamic: license-file
 
 # XspecT - Acinetobacter Species Assignment Tool
+<!-- start intro -->
 ![Test](https://github.com/bionf/xspect2/actions/workflows/test.yml/badge.svg)
 [![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-<img src="/docs/img/logo.png" height="50%" width="50%">
-
-<!-- start intro -->
-XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [Bloom Filters] and a [Support Vector Machine].
-<br/><br/>
-
-XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction the results are classified using a Support Vector Machine. 
-<br/>
+XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [kmer indices] and a [Support Vector Machine].
 
-Local extensions of the reference database are supported.
-<br/>
+XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction, the results are classified using a Support Vector Machine.
 
-The tool is available as a web-based application and a smaller command line interface.
+The tool is available as a web-based application and as a command line interface.
 
-[Bloom Filters]: https://en.wikipedia.org/wiki/Bloom_filter
+[kmer indices]: https://arxiv.org/abs/1905.09624
 [Support Vector Machine]: https://en.wikipedia.org/wiki/Support-vector_machine
-[blaOxa-genes]: https://en.wikipedia.org/wiki/Beta-lactamase#OXA_beta-lactamases_(class_D)
 <!-- end intro -->
 
 <!-- start quickstart -->
 ## Installation
-To install Xspect, please download the lastest 64 bit Python version and install the package using pip:
+To install XspecT, please download the lastest 64 bit Python version and install the package using pip:
 ```
 pip install xspect
 ```
@@ -91,23 +84,23 @@ Please note that Windows and Alpine Linux is currently not supported.
 ### Get the models
 To download basic pre-trained models, you can use the built-in command:
 ```
-xspect download-models
+xspect models download
 ```
 Additional species models can be trained using:
 ```
-xspect train-species you-ncbi-genus-name
+xspect models train ncbi
 ```
 
 ### How to run the web app
 To run the web app, install and run [XspecT Web](https://github.com/aromberg/xspect-web). Additionally, run XspecT in API mode:
 ```
-xspect api
+xspect web
 ```
 
 ### How to use the XspecT command line interface
-Run xspect with the configuration you want to run it with as arguments.
+Run XspecT with the configuration you want to run it with as arguments.
 ```
-xspect classify-species your-genus path/to/your/input-set
+xspect classify species
 ```
 For further instructions on how to use the command line interface, please refer to the [documentation] or execute:
 ```

{xspect-0.2.7 → xspect-0.4.0}/README.md

@@ -1,30 +1,22 @@
 # XspecT - Acinetobacter Species Assignment Tool
+<!-- start intro -->
 ![Test](https://github.com/bionf/xspect2/actions/workflows/test.yml/badge.svg)
 [![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-<img src="/docs/img/logo.png" height="50%" width="50%">
-
-<!-- start intro -->
-XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [Bloom Filters] and a [Support Vector Machine].
-<br/><br/>
-
-XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction the results are classified using a Support Vector Machine. 
-<br/>
+XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [kmer indices] and a [Support Vector Machine].
 
-Local extensions of the reference database are supported.
-<br/>
+XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction, the results are classified using a Support Vector Machine.
 
-The tool is available as a web-based application and a smaller command line interface.
+The tool is available as a web-based application and as a command line interface.
 
-[Bloom Filters]: https://en.wikipedia.org/wiki/Bloom_filter
+[kmer indices]: https://arxiv.org/abs/1905.09624
 [Support Vector Machine]: https://en.wikipedia.org/wiki/Support-vector_machine
-[blaOxa-genes]: https://en.wikipedia.org/wiki/Beta-lactamase#OXA_beta-lactamases_(class_D)
 <!-- end intro -->
 
 <!-- start quickstart -->
 ## Installation
-To install Xspect, please download the lastest 64 bit Python version and install the package using pip:
+To install XspecT, please download the lastest 64 bit Python version and install the package using pip:
 ```
 pip install xspect
 ```
@@ -34,23 +26,23 @@ Please note that Windows and Alpine Linux is currently not supported.
 ### Get the models
 To download basic pre-trained models, you can use the built-in command:
 ```
-xspect download-models
+xspect models download
 ```
 Additional species models can be trained using:
 ```
-xspect train-species you-ncbi-genus-name
+xspect models train ncbi
 ```
 
 ### How to run the web app
 To run the web app, install and run [XspecT Web](https://github.com/aromberg/xspect-web). Additionally, run XspecT in API mode:
 ```
-xspect api
+xspect web
 ```
 
 ### How to use the XspecT command line interface
-Run xspect with the configuration you want to run it with as arguments.
+Run XspecT with the configuration you want to run it with as arguments.
 ```
-xspect classify-species your-genus path/to/your/input-set
+xspect classify species
 ```
 For further instructions on how to use the command line interface, please refer to the [documentation] or execute:
 ```

{xspect-0.2.7 → xspect-0.4.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "XspecT"
-version = "0.2.7"
+version = "0.4.0"
 description = "Tool to monitor and characterize pathogens using Bloom filters."
 readme = {file = "README.md", content-type = "text/markdown"}
 license = {file = "LICENSE"}

{xspect-0.2.7 → xspect-0.4.0/src/XspecT.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: XspecT
-Version: 0.2.7
+Version: 0.4.0
 Summary: Tool to monitor and characterize pathogens using Bloom filters.
 License: MIT License
         
@@ -54,34 +54,27 @@ Requires-Dist: sphinx-autobuild; extra == "docs"
 Provides-Extra: test
 Requires-Dist: pytest; extra == "test"
 Requires-Dist: pytest-cov; extra == "test"
+Dynamic: license-file
 
 # XspecT - Acinetobacter Species Assignment Tool
+<!-- start intro -->
 ![Test](https://github.com/bionf/xspect2/actions/workflows/test.yml/badge.svg)
 [![linting: pylint](https://img.shields.io/badge/linting-pylint-yellowgreen)](https://github.com/pylint-dev/pylint)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 
-<img src="/docs/img/logo.png" height="50%" width="50%">
-
-<!-- start intro -->
-XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [Bloom Filters] and a [Support Vector Machine].
-<br/><br/>
-
-XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction the results are classified using a Support Vector Machine. 
-<br/>
+XspecT is a Python-based tool to taxonomically classify sequence-reads (or assembled genomes) on the species and/or MLST level using [kmer indices] and a [Support Vector Machine].
 
-Local extensions of the reference database are supported.
-<br/>
+XspecT utilizes the uniqueness of kmers and compares extracted kmers from the input-data to a reference database. Bloom Filter ensure a fast lookup in this process. For a final prediction, the results are classified using a Support Vector Machine.
 
-The tool is available as a web-based application and a smaller command line interface.
+The tool is available as a web-based application and as a command line interface.
 
-[Bloom Filters]: https://en.wikipedia.org/wiki/Bloom_filter
+[kmer indices]: https://arxiv.org/abs/1905.09624
 [Support Vector Machine]: https://en.wikipedia.org/wiki/Support-vector_machine
-[blaOxa-genes]: https://en.wikipedia.org/wiki/Beta-lactamase#OXA_beta-lactamases_(class_D)
 <!-- end intro -->
 
 <!-- start quickstart -->
 ## Installation
-To install Xspect, please download the lastest 64 bit Python version and install the package using pip:
+To install XspecT, please download the lastest 64 bit Python version and install the package using pip:
 ```
 pip install xspect
 ```
@@ -91,23 +84,23 @@ Please note that Windows and Alpine Linux is currently not supported.
 ### Get the models
 To download basic pre-trained models, you can use the built-in command:
 ```
-xspect download-models
+xspect models download
 ```
 Additional species models can be trained using:
 ```
-xspect train-species you-ncbi-genus-name
+xspect models train ncbi
 ```
 
 ### How to run the web app
 To run the web app, install and run [XspecT Web](https://github.com/aromberg/xspect-web). Additionally, run XspecT in API mode:
 ```
-xspect api
+xspect web
 ```
 
 ### How to use the XspecT command line interface
-Run xspect with the configuration you want to run it with as arguments.
+Run XspecT with the configuration you want to run it with as arguments.
 ```
-xspect classify-species your-genus path/to/your/input-set
+xspect classify species
 ```
 For further instructions on how to use the command line interface, please refer to the [documentation] or execute:
 ```

{xspect-0.2.7 → xspect-0.4.0}/src/XspecT.egg-info/SOURCES.txt

@@ -56,8 +56,7 @@ src/xspect/fastapi.py
 src/xspect/file_io.py
 src/xspect/main.py
 src/xspect/model_management.py
-src/xspect/pipeline.py
-src/xspect/run.py
+src/xspect/ncbi.py
 src/xspect/train.py
 src/xspect/mlst_feature/__init__.py
 src/xspect/mlst_feature/mlst_helper.py
@@ -68,21 +67,13 @@ src/xspect/models/probabilistic_filter_model.py
 src/xspect/models/probabilistic_filter_svm_model.py
 src/xspect/models/probabilistic_single_filter_model.py
 src/xspect/models/result.py
-src/xspect/train_filter/__init__.py
-src/xspect/train_filter/create_svm.py
-src/xspect/train_filter/extract_and_concatenate.py
-src/xspect/train_filter/ncbi_api/__init__.py
-src/xspect/train_filter/ncbi_api/download_assemblies.py
-src/xspect/train_filter/ncbi_api/ncbi_assembly_metadata.py
-src/xspect/train_filter/ncbi_api/ncbi_children_tree.py
-src/xspect/train_filter/ncbi_api/ncbi_taxon_metadata.py
 tests/__init__.py
 tests/conftest.py
 tests/test_cli.py
 tests/test_file_io.py
 tests/test_model_management.py
 tests/test_model_result.py
-tests/test_pipeline.py
+tests/test_ncbi.py
 tests/test_probabilisitc_filter_mlst_model.py
 tests/test_probabilistic_filter_model.py
 tests/test_probabilistic_filter_svm_model.py

{xspect-0.2.7 → xspect-0.4.0}/src/xspect/definitions.py

@@ -21,13 +21,6 @@ def get_xspect_model_path():
     return model_path
 
 
-def get_xspect_tmp_path():
-    """Return the path to the XspecT temporary files."""
-    tmp_path = get_xspect_root_path() / "tmp"
-    tmp_path.mkdir(exist_ok=True, parents=True)
-    return tmp_path
-
-
 def get_xspect_upload_path():
     """Return the path to the XspecT upload directory."""
     upload_path = get_xspect_root_path() / "uploads"

xspect-0.4.0/src/xspect/download_models.py

@@ -0,0 +1,34 @@
+"""Download filters from public repository."""
+
+import shutil
+from tempfile import TemporaryDirectory
+from pathlib import Path
+import requests
+
+from xspect.definitions import get_xspect_model_path
+
+
+def download_test_models(url):
+    """Download models."""
+    with TemporaryDirectory() as tmp_dir:
+        tmp_dir = Path(tmp_dir)
+        download_path = tmp_dir / "models.zip"
+        extract_path = tmp_dir / "extracted_models"
+
+        r = requests.get(url, allow_redirects=True, timeout=10)
+        with open(download_path, "wb") as f:
+            f.write(r.content)
+
+        shutil.unpack_archive(
+            download_path,
+            extract_path,
+            "zip",
+        )
+
+        shutil.copytree(
+            extract_path,
+            get_xspect_model_path(),
+            dirs_exist_ok=True,
+        )
+
+        shutil.rmtree(extract_path)

{xspect-0.2.7 → xspect-0.4.0}/src/xspect/fastapi.py

@@ -1,15 +1,14 @@
 """FastAPI application for XspecT."""
 
-import datetime
+from uuid import uuid4
 from pathlib import Path
 from shutil import copyfileobj
 from fastapi import FastAPI, UploadFile, BackgroundTasks
 from xspect.definitions import get_xspect_runs_path, get_xspect_upload_path
 from xspect.download_models import download_test_models
+from xspect.file_io import filter_sequences
 import xspect.model_management as mm
-from xspect.models.result import StepType
-from xspect.pipeline import ModelExecution, Pipeline, PipelineStep
-from xspect.train import train_ncbi
+from xspect.train import train_from_ncbi
 
 app = FastAPI()
 
@@ -17,43 +16,41 @@ app = FastAPI()
 @app.get("/download-filters")
 def download_filters():
     """Download filters."""
-    download_test_models("https://xspect2.s3.eu-central-1.amazonaws.com/models.zip")
+    download_test_models("http://assets.adrianromberg.com/xspect-models.zip")
 
 
 @app.get("/classify")
 def classify(genus: str, file: str, meta: bool = False, step: int = 500):
     """Classify uploaded sample."""
 
-    path = get_xspect_upload_path() / file
+    input_path = get_xspect_upload_path() / file
+
+    uuid = str(uuid4())
 
-    pipeline = Pipeline(genus + " classification", "Test Author", "test@example.com")
-    species_execution = ModelExecution(
-        genus.lower() + "-species", sparse_sampling_step=step
-    )
     if meta:
-        species_filtering_step = PipelineStep(
-            StepType.FILTERING, genus, 0.7, species_execution
-        )
-        genus_execution = ModelExecution(
-            genus.lower() + "-genus", sparse_sampling_step=step
+        genus_model = mm.get_genus_model(genus)
+        genus_result = genus_model.predict(input_path, step=step)
+        included_ids = genus_result.get_filtered_subsequence_labels(genus)
+        if not included_ids:
+            return {"message": "No sequences found for the given genus."}
+        filtered_path = get_xspect_runs_path() / f"filtered_{uuid}.fasta"
+        filter_sequences(
+            Path(input_path),
+            Path(filtered_path),
+            included_ids=included_ids,
         )
-        genus_execution.add_pipeline_step(species_filtering_step)
-        pipeline.add_pipeline_step(genus_execution)
-    else:
-        pipeline.add_pipeline_step(species_execution)
-
-    run = pipeline.run(Path(path))
-    time_str = datetime.datetime.now().strftime("%Y-%m-%d-%H-%M-%S")
-    save_path = get_xspect_runs_path() / f"run_{time_str}.json"
-    run.save(save_path)
+        input_path = filtered_path
 
-    return run.to_dict()
+    species_model = mm.get_species_model(genus)
+    species_result = species_model.predict(input_path, step=step)
+    species_result.save(get_xspect_runs_path() / f"result_{uuid}.json")
+    return species_result.to_dict()
 
 
 @app.post("/train")
 def train(genus: str, background_tasks: BackgroundTasks, svm_steps: int = 1):
     """Train NCBI model."""
-    background_tasks.add_task(train_ncbi, genus, svm_steps)
+    background_tasks.add_task(train_from_ncbi, genus, svm_steps)
 
     return {"message": "Training started."}
 

xspect-0.4.0/src/xspect/file_io.py

@@ -0,0 +1,171 @@
+"""
+File IO module.
+"""
+
+from json import loads
+import os
+from pathlib import Path
+import zipfile
+from Bio import SeqIO
+from xspect.definitions import fasta_endings, fastq_endings
+
+
+def delete_zip_files(dir_path):
+    """Delete all zip files in the given directory."""
+    files = os.listdir(dir_path)
+    for file in files:
+        if zipfile.is_zipfile(file):
+            file_path = dir_path / str(file)
+            os.remove(file_path)
+
+
+def extract_zip(zip_path: Path, unzipped_path: Path):
+    """Extracts all files from a directory with zip files."""
+    # Make new directory.
+    unzipped_path.mkdir(parents=True, exist_ok=True)
+
+    file_names = os.listdir(zip_path)
+    for file in file_names:
+        file_path = zip_path / file
+        if zipfile.is_zipfile(file_path):
+            with zipfile.ZipFile(file_path) as item:
+                directory = unzipped_path / file.replace(".zip", "")
+                item.extractall(directory)
+
+
+def concatenate_meta(path: Path, genus: str):
+    """Concatenates all species files to one fasta file.
+
+    :param path: Path to the directory with the concatenated fasta files.
+    :type path: Path
+    :param genus: Genus name.
+    :type genus: str
+    """
+    files_path = path / "concatenate"
+    meta_path = path / (genus + ".fasta")
+    files = os.listdir(files_path)
+
+    with open(meta_path, "w", encoding="utf-8") as meta_file:
+        # Write the header.
+        meta_header = f">{genus} metagenome\n"
+        meta_file.write(meta_header)
+
+        # Open each concatenated species file and write the sequence in the meta file.
+        for file in files:
+            file_ending = str(file).rsplit(".", maxsplit=1)[-1]
+            if file_ending in fasta_endings:
+                with open(
+                    (files_path / str(file)), "r", encoding="utf-8"
+                ) as species_file:
+                    for line in species_file:
+                        if line[0] != ">":
+                            meta_file.write(line.replace("\n", ""))
+
+
+def get_record_iterator(file_path: Path):
+    """Returns a record iterator for a fasta or fastq file."""
+    if not isinstance(file_path, Path):
+        raise ValueError("Path must be a Path object")
+
+    if not file_path.exists():
+        raise ValueError("File does not exist")
+
+    if not file_path.is_file():
+        raise ValueError("Path must be a file")
+
+    if file_path.suffix[1:] in fasta_endings:
+        return SeqIO.parse(file_path, "fasta")
+
+    if file_path.suffix[1:] in fastq_endings:
+        return SeqIO.parse(file_path, "fastq")
+
+    raise ValueError("Invalid file format, must be a fasta or fastq file")
+
+
+def get_records_by_id(file: Path, ids: list[str]):
+    """Return records with the specified ids."""
+    records = get_record_iterator(file)
+    return [record for record in records if record.id in ids]
+
+
+def concatenate_species_fasta_files(input_folders: list[Path], output_directory: Path):
+    """Concatenate fasta files from different species into one file per species.
+
+    Args:
+        input_species_folders (list[Path]): List of paths to species folders.
+        output_directory (Path): Path to the output directory.
+    """
+    for species_folder in input_folders:
+        species_name = species_folder.name
+        fasta_files = [
+            f for ending in fasta_endings for f in species_folder.glob(f"*.{ending}")
+        ]
+        if len(fasta_files) == 0:
+            raise ValueError(f"no fasta files found in {species_folder}")
+
+        # concatenate fasta files
+        concatenated_fasta = output_directory / f"{species_name}.fasta"
+        with open(concatenated_fasta, "w", encoding="utf-8") as f:
+            for fasta_file in fasta_files:
+                with open(fasta_file, "r", encoding="utf-8") as f_in:
+                    f.write(f_in.read())
+
+
+def concatenate_metagenome(fasta_dir: Path, meta_path: Path):
+    """Concatenate all fasta files in a directory into one file.
+
+    Args:
+        fasta_dir (Path): Path to the directory with the fasta files.
+        meta_path (Path): Path to the output file.
+    """
+    with open(meta_path, "w", encoding="utf-8") as meta_file:
+        for fasta_file in fasta_dir.glob("*.fasta"):
+            with open(fasta_file, "r", encoding="utf-8") as f_in:
+                meta_file.write(f_in.read())
+
+
+def get_ncbi_dataset_accession_paths(
+    ncbi_dataset_path: Path,
+) -> dict[str, Path]:
+    """Get the paths of the NCBI dataset accessions.
+
+    Args:
+        ncbi_dataset_path (Path): Path to the NCBI dataset directory.
+
+    Returns:
+        dict[str, Path]: Dictionary with the accession as key and the path as value.
+    """
+    data_path = ncbi_dataset_path / "ncbi_dataset" / "data"
+    if not data_path.exists():
+        raise ValueError(f"Path {data_path} does not exist.")
+
+    accession_paths = {}
+    with open(data_path / "dataset_catalog.json", "r", encoding="utf-8") as f:
+        res = loads(f.read())
+        for assembly in res["assemblies"][1:]:  # the first item is the data report
+            accession = assembly["accession"]
+            assembly_path = data_path / assembly["files"][0]["filePath"]
+            accession_paths[accession] = assembly_path
+    return accession_paths
+
+
+def filter_sequences(
+    input_file: Path,
+    output_file: Path,
+    included_ids: list[str],
+):
+    """Filter sequences by IDs from an input file and save them to an output file.
+
+    Args:
+        input_file (Path): Path to the input file.
+        output_file (Path): Path to the output file.
+        included_ids (list[str], optional): List of IDs to include. If None, no output file is created.
+    """
+    if not included_ids:
+        print("No IDs provided, no output file will be created.")
+        return
+
+    with open(output_file, "w", encoding="utf-8") as out_f:
+        for record in get_record_iterator(input_file):
+            if record.id in included_ids:
+                SeqIO.write(record, out_f, "fasta")