boldigger3 1.0.0__tar.gz

boldigger3-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 DominikBuchner
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

boldigger3-1.0.0/PKG-INFO

@@ -0,0 +1,190 @@
+Metadata-Version: 2.1
+Name: boldigger3
+Version: 1.0.0
+Summary: A python package to query different databases of boldsystems.org v5!
+Home-page: https://github.com/DominikBuchner/BOLDigger3
+Author: Dominik Buchner
+Author-email: dominik.buchner@uni-due.de
+License: MIT
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: beautifulsoup4>=4.12.3
+Requires-Dist: Bio>=1.7.1
+Requires-Dist: biopython>=1.84
+Requires-Dist: get_pypi_latest_version>=0.0.12
+Requires-Dist: more_itertools>=10.5.0
+Requires-Dist: numpy>=2.1.2
+Requires-Dist: pandas>=2.2.3
+Requires-Dist: playwright>=1.48.0
+Requires-Dist: Requests>=2.32.3
+Requires-Dist: requests_html_playwright>=0.12.3
+Requires-Dist: setuptools>=65.5.0
+Requires-Dist: tqdm>=4.66.4
+Requires-Dist: urllib3>=1.26.14
+Requires-Dist: tables>=3.9.2
+Requires-Dist: html5lib>=1.1
+Requires-Dist: soupsieve>=2.5
+Requires-Dist: openpyxl>=3.1.1
+Requires-Dist: pyarrow>=11.0.0
+Requires-Dist: lxml_html_clean>=0.1.1
+
+# BOLDigger3
+ 
+![boldigger_logo]()
+
+[![Downloads](https://pepy.tech/badge/boldigger3)](https://pepy.tech/project/boldigger3)
+
+A Python program to query .fasta files against the databases of www.boldsystems.org v5!
+
+## Introduction
+DNA metabarcoding datasets often comprise hundreds of Operational Taxonomic Units (OTUs), requiring querying against databases for taxonomic assignment. The Barcode of Life Data system (BOLD) is a widely used database for this purpose among biologists. However, BOLD's online platform limits users to identifying batches of only 1000, 200 or 100  (depending on operating mode) sequences at a time.
+
+BOLDigger3, the successor to BOLDigger2 and BOLDigger, aims to overcome these limitations. As a pure Python program, BOLDigger3 offers:
+
+- Automated access to BOLD's identification engine
+- Downloading of additional metadata for each hit
+- Selection of the best-fitting hit from the returned results
+
+## Overview
+
+**BOLDigger3** is an automated tool designed for DNA sequence identification through **BOLDSystems v5**, supporting integration into bioinformatics pipelines with enhanced functionality and performance. With BOLDigger3, users can identify up to **10,000 sequences per hour** without the need for credentials, using an optimized data storage and queuing system that improves speed and process safety.
+
+## Key Differences Between BOLDigger3 and BOLDigger2
+
+- **Unified Function**: BOLDigger3 consolidates all actions into a single function, `identify`, which automatically performs identification, additional data downloading, and top-hit selection, making it easier to integrate into pipelines.
+- **Enhanced Database Accessibility**: Users have access to all databases offered by **BOLDSystems v5** and can select from three different operating modes.
+- **Improved Speeds**: Depending on the operating mode, BOLDigger3 can identify up to **10,000 sequences per hour**, significantly faster than BOLDigger2.
+- **No Password Required**: Users no longer need credentials to perform identifications—just select the FASTA file, database, and operating mode to start.
+- **Streamlined Data Storage**: Data is stored in an **HDF Store** for faster processing, with final outputs available in `.xlsx` and `.parquet` formats.
+- **Process Safety**: BOLDigger3 can resume interrupted executions, continuing exactly where it left off.
+- **Dynamic Queuing**: The tool automatically manages request queuing based on the selected operating mode.
+
+## Features
+
+- **Identify Sequences Automatically**: Run DNA sequence identifications with a single command.
+- **Flexible Database Options**: Access to all BOLDSystems v5 databases with user-selected operating modes.
+- **High-Performance Processing**: Up to 10,000 identifications per hour, depending on settings.
+- **Robust Storage**: Data stored in HDF format for efficient processing; results in `.xlsx` and `.parquet`.
+- **User-Friendly**: No credentials needed for use.
+
+## Installation and Usage
+
+BOLDigger3 requires Python version 3.10 or higher and can be easily installed using pip in any command line:
+
+`pip install boldigger3`
+
+This command will install BOLDigger3 along with all its dependencies.
+BOLDigger3 uses the Python package ```playwright```, which needs a separate installation prior to first execution:
+
+`playwright install`
+
+**FOR BOLDigger2 USERS:**
+BOLDigger2 used requests-html which relied on an old version of pyppeteer. This may lead to conflicts with playwright. To solve:
+
+`
+pip uninstall pyppeteer
+pip install --upgrade pyee
+`
+
+To run the ```identify``` function, use the following command:
+
+`boldigger3 identify PATH_TO_FASTA --db DATABASE_NR --mode OPERATING MODE`
+
+# Databases
+
+The ```--db``` is a number between 1 and 7 corresponding to the seven databases BOLD v5 currently offers:
+
+1: **ANIMAL LIBRARY (PUBLIC)**   
+2: **ANIMAL SPECIES-LEVEL LIBRARY (PUBLIC + PRIVATE)**   
+3: **ANIMAL LIBRARY (PUBLIC+PRIVATE)**    
+4: **VALIDATED CANADIAN ARTHROPOD LIBRARY**   
+5: **PLANT LIBRARY (PUBLIC)**   
+6: **FUNGI LIBRARY (PUBLIC)**   
+7: **ANIMAL SECONDARY MARKERS (PUBLIC)**   
+
+# Operating modes
+
+The ```--mode``` is a number between 1 and the corresponding to the 3 operating modes BOLD v5 currently offers:
+
+1: **Rapid Species Search**   
+2: **Genus and Species Search**   
+3: **Exhaustive Search**   
+
+To customize the implemented thresholds for user-specific needs, the thresholds can be passed as an additional (ordered) argument. Up to five different thresholds can be passed for the different taxonomic levels (Species, Genus, Family, Order, Class). Thresholds not passed will be replaced by default, but BOLDigger3 will also inform you about this:
+
+`boldigger3 identify PATH_TO_FASTA --db DATABASE_NR --mode OPERATING MODE --thresholds 99 97`
+
+Output:
+
+```
+19:16:16: Default thresholds changed!
+19:16:16: Species: 99, Genus: 97, Family: 90, Order: 85
+```
+
+When a new version is released, you can update BOLDigger3 by typing:
+
+`pip install --upgrade boldigger3`
+
+## How to cite
+
+Buchner D, Leese F (2020) BOLDigger – a Python package to identify and organise sequences with the Barcode of Life Data systems. Metabarcoding and Metagenomics 4: e53535. https://doi.org/10.3897/mbmg.4.53535
+
+
+## The BOLDigger3 Algorithm
+
+The BOLDigger3 algorithm operates as follows:
+
+1. **Split the FASTA**: The input FASTA file is divided into chunks that fit the limits of the selected operating mode of the identification engine.
+
+2. **Queue the Chunks**: These chunks are then queued in the identification engine for processing.
+
+3. **Check for Results**: The algorithm periodically checks if any results can be downloaded.
+
+4. **Data Download**: Once results are available, the data is downloaded.
+
+5. **Data Validation**: The algorithm ensures that all data has been correctly downloaded.
+
+6. **Retrieve Additional Data**: Additional data is obtained via the API.
+
+7. **Select Top Hit**: Finally, the algorithm selects the top hit backed by the most database entries for the final output.
+
+### Top hit selection
+
+Different thresholds (97%: species level, 95%: genus level, 90%: family level, 85%: order level) for the taxonomic levels are used to find the best fitting hit. After determining the threshold for all hits the most common hit above the threshold will be selected. Note that for all hits below the threshold, the taxonomic resolution will be adjusted accordingly (e.g. for a 96% hit the species-level information will be discarded, and genus-level information will be used as the lowest taxonomic level).
+
+The BOLDigger3 algorithm functions as follows:
+
+1. **Identify Maximum Similarity**: Find the maximum similarity value among the top 100 hits currently under consideration.
+   
+2. **Set Threshold**: Set the threshold to this maximum similarity level. Remove all hits with a similarity below this threshold. For example, if the highest hit has a similarity of 100%, the threshold will be set to 97%, and all hits below this threshold will be removed temporarily.
+
+3. **Classification and Sorting**: Count all individual classifications and sort them by abundance.
+
+4. **Filter Missing Data**: Drop all classifications that contain missing data. For instance, if the most common hit is "Arthropoda --> Insecta" with a similarity of 100% but missing values for Order, Family, Genus, and Species.
+
+5. **Identify Common Hit**: Look for the most common hit that has no missing values.
+
+6. **Return Hit**: If a hit with no missing values is found, return that hit.
+
+7. **Threshold Adjustment**: If no hit with no missing values is found, increase the threshold to the next higher level and repeat the process until a hit is found.
+
+
+### BOLDigger3 Flagging System
+
+BOLDigger3 employs a flagging system to highlight certain conditions, indicating a degree of uncertainty in the selected hit. Currently, there are five flags implemented, which may be updated as needed:
+
+1. **Reverse BIN Taxonomy**: This flag is raised if all of the top 100 hits representing the selected match utilize reverse BIN taxonomy. Reverse BIN taxonomy assigns species names to deposited sequences on BOLD that lack species information, potentially introducing uncertainty.
+
+2. **Differing Taxonomic Information**: If there are two or more entries with differing taxonomic information above the selected threshold (e.g., two species above 97%), this flag is triggered, suggesting potential discrepancies.
+
+3. **Private Data**: If all of the top 100 hits representing the top hit are private hits, this flag is raised, indicating limited accessibility to data.
+
+4. **Unique Hit**: This flag indicates that the top hit result represents a unique hit among the top 100 hits, potentially requiring further scrutiny.
+
+5. **Multiple BINs**: If the selected species-level hit is composed of more than one BIN, this flag is raised, suggesting potential complexities in taxonomic assignment.
+
+Given the presence of these flags, it is advisable to conduct a closer examination of all flagged hits to better understand and address any uncertainties in the selected hit.

boldigger3-1.0.0/README.md

@@ -0,0 +1,156 @@
+# BOLDigger3
+ 
+![boldigger_logo]()
+
+[![Downloads](https://pepy.tech/badge/boldigger3)](https://pepy.tech/project/boldigger3)
+
+A Python program to query .fasta files against the databases of www.boldsystems.org v5!
+
+## Introduction
+DNA metabarcoding datasets often comprise hundreds of Operational Taxonomic Units (OTUs), requiring querying against databases for taxonomic assignment. The Barcode of Life Data system (BOLD) is a widely used database for this purpose among biologists. However, BOLD's online platform limits users to identifying batches of only 1000, 200 or 100  (depending on operating mode) sequences at a time.
+
+BOLDigger3, the successor to BOLDigger2 and BOLDigger, aims to overcome these limitations. As a pure Python program, BOLDigger3 offers:
+
+- Automated access to BOLD's identification engine
+- Downloading of additional metadata for each hit
+- Selection of the best-fitting hit from the returned results
+
+## Overview
+
+**BOLDigger3** is an automated tool designed for DNA sequence identification through **BOLDSystems v5**, supporting integration into bioinformatics pipelines with enhanced functionality and performance. With BOLDigger3, users can identify up to **10,000 sequences per hour** without the need for credentials, using an optimized data storage and queuing system that improves speed and process safety.
+
+## Key Differences Between BOLDigger3 and BOLDigger2
+
+- **Unified Function**: BOLDigger3 consolidates all actions into a single function, `identify`, which automatically performs identification, additional data downloading, and top-hit selection, making it easier to integrate into pipelines.
+- **Enhanced Database Accessibility**: Users have access to all databases offered by **BOLDSystems v5** and can select from three different operating modes.
+- **Improved Speeds**: Depending on the operating mode, BOLDigger3 can identify up to **10,000 sequences per hour**, significantly faster than BOLDigger2.
+- **No Password Required**: Users no longer need credentials to perform identifications—just select the FASTA file, database, and operating mode to start.
+- **Streamlined Data Storage**: Data is stored in an **HDF Store** for faster processing, with final outputs available in `.xlsx` and `.parquet` formats.
+- **Process Safety**: BOLDigger3 can resume interrupted executions, continuing exactly where it left off.
+- **Dynamic Queuing**: The tool automatically manages request queuing based on the selected operating mode.
+
+## Features
+
+- **Identify Sequences Automatically**: Run DNA sequence identifications with a single command.
+- **Flexible Database Options**: Access to all BOLDSystems v5 databases with user-selected operating modes.
+- **High-Performance Processing**: Up to 10,000 identifications per hour, depending on settings.
+- **Robust Storage**: Data stored in HDF format for efficient processing; results in `.xlsx` and `.parquet`.
+- **User-Friendly**: No credentials needed for use.
+
+## Installation and Usage
+
+BOLDigger3 requires Python version 3.10 or higher and can be easily installed using pip in any command line:
+
+`pip install boldigger3`
+
+This command will install BOLDigger3 along with all its dependencies.
+BOLDigger3 uses the Python package ```playwright```, which needs a separate installation prior to first execution:
+
+`playwright install`
+
+**FOR BOLDigger2 USERS:**
+BOLDigger2 used requests-html which relied on an old version of pyppeteer. This may lead to conflicts with playwright. To solve:
+
+`
+pip uninstall pyppeteer
+pip install --upgrade pyee
+`
+
+To run the ```identify``` function, use the following command:
+
+`boldigger3 identify PATH_TO_FASTA --db DATABASE_NR --mode OPERATING MODE`
+
+# Databases
+
+The ```--db``` is a number between 1 and 7 corresponding to the seven databases BOLD v5 currently offers:
+
+1: **ANIMAL LIBRARY (PUBLIC)**   
+2: **ANIMAL SPECIES-LEVEL LIBRARY (PUBLIC + PRIVATE)**   
+3: **ANIMAL LIBRARY (PUBLIC+PRIVATE)**    
+4: **VALIDATED CANADIAN ARTHROPOD LIBRARY**   
+5: **PLANT LIBRARY (PUBLIC)**   
+6: **FUNGI LIBRARY (PUBLIC)**   
+7: **ANIMAL SECONDARY MARKERS (PUBLIC)**   
+
+# Operating modes
+
+The ```--mode``` is a number between 1 and the corresponding to the 3 operating modes BOLD v5 currently offers:
+
+1: **Rapid Species Search**   
+2: **Genus and Species Search**   
+3: **Exhaustive Search**   
+
+To customize the implemented thresholds for user-specific needs, the thresholds can be passed as an additional (ordered) argument. Up to five different thresholds can be passed for the different taxonomic levels (Species, Genus, Family, Order, Class). Thresholds not passed will be replaced by default, but BOLDigger3 will also inform you about this:
+
+`boldigger3 identify PATH_TO_FASTA --db DATABASE_NR --mode OPERATING MODE --thresholds 99 97`
+
+Output:
+
+```
+19:16:16: Default thresholds changed!
+19:16:16: Species: 99, Genus: 97, Family: 90, Order: 85
+```
+
+When a new version is released, you can update BOLDigger3 by typing:
+
+`pip install --upgrade boldigger3`
+
+## How to cite
+
+Buchner D, Leese F (2020) BOLDigger – a Python package to identify and organise sequences with the Barcode of Life Data systems. Metabarcoding and Metagenomics 4: e53535. https://doi.org/10.3897/mbmg.4.53535
+
+
+## The BOLDigger3 Algorithm
+
+The BOLDigger3 algorithm operates as follows:
+
+1. **Split the FASTA**: The input FASTA file is divided into chunks that fit the limits of the selected operating mode of the identification engine.
+
+2. **Queue the Chunks**: These chunks are then queued in the identification engine for processing.
+
+3. **Check for Results**: The algorithm periodically checks if any results can be downloaded.
+
+4. **Data Download**: Once results are available, the data is downloaded.
+
+5. **Data Validation**: The algorithm ensures that all data has been correctly downloaded.
+
+6. **Retrieve Additional Data**: Additional data is obtained via the API.
+
+7. **Select Top Hit**: Finally, the algorithm selects the top hit backed by the most database entries for the final output.
+
+### Top hit selection
+
+Different thresholds (97%: species level, 95%: genus level, 90%: family level, 85%: order level) for the taxonomic levels are used to find the best fitting hit. After determining the threshold for all hits the most common hit above the threshold will be selected. Note that for all hits below the threshold, the taxonomic resolution will be adjusted accordingly (e.g. for a 96% hit the species-level information will be discarded, and genus-level information will be used as the lowest taxonomic level).
+
+The BOLDigger3 algorithm functions as follows:
+
+1. **Identify Maximum Similarity**: Find the maximum similarity value among the top 100 hits currently under consideration.
+   
+2. **Set Threshold**: Set the threshold to this maximum similarity level. Remove all hits with a similarity below this threshold. For example, if the highest hit has a similarity of 100%, the threshold will be set to 97%, and all hits below this threshold will be removed temporarily.
+
+3. **Classification and Sorting**: Count all individual classifications and sort them by abundance.
+
+4. **Filter Missing Data**: Drop all classifications that contain missing data. For instance, if the most common hit is "Arthropoda --> Insecta" with a similarity of 100% but missing values for Order, Family, Genus, and Species.
+
+5. **Identify Common Hit**: Look for the most common hit that has no missing values.
+
+6. **Return Hit**: If a hit with no missing values is found, return that hit.
+
+7. **Threshold Adjustment**: If no hit with no missing values is found, increase the threshold to the next higher level and repeat the process until a hit is found.
+
+
+### BOLDigger3 Flagging System
+
+BOLDigger3 employs a flagging system to highlight certain conditions, indicating a degree of uncertainty in the selected hit. Currently, there are five flags implemented, which may be updated as needed:
+
+1. **Reverse BIN Taxonomy**: This flag is raised if all of the top 100 hits representing the selected match utilize reverse BIN taxonomy. Reverse BIN taxonomy assigns species names to deposited sequences on BOLD that lack species information, potentially introducing uncertainty.
+
+2. **Differing Taxonomic Information**: If there are two or more entries with differing taxonomic information above the selected threshold (e.g., two species above 97%), this flag is triggered, suggesting potential discrepancies.
+
+3. **Private Data**: If all of the top 100 hits representing the top hit are private hits, this flag is raised, indicating limited accessibility to data.
+
+4. **Unique Hit**: This flag indicates that the top hit result represents a unique hit among the top 100 hits, potentially requiring further scrutiny.
+
+5. **Multiple BINs**: If the selected species-level hit is composed of more than one BIN, this flag is raised, suggesting potential complexities in taxonomic assignment.
+
+Given the presence of these flags, it is advisable to conduct a closer examination of all flagged hits to better understand and address any uncertainties in the selected hit.

boldigger3-1.0.0/boldigger3/__main__.py

@@ -0,0 +1,125 @@
+import argparse, sys, datetime, time
+from boldigger3 import id_engine, additional_data_download, select_top_hit
+from importlib.metadata import version
+from get_pypi_latest_version import GetPyPiLatestVersion
+
+
+# main function to program the commandline interface
+def main() -> None:
+    """Function to define the commandline interface."""
+    # initialize the default behaviour if boldigger3 is called without any argument
+    formatter = lambda prog: argparse.HelpFormatter(prog, max_help_position=35)
+
+    # define the parser
+    parser = argparse.ArgumentParser(
+        prog="boldigger3",
+        description="A Python package to identify and organise sequences with the Barcode of Life Data systems.",
+        formatter_class=formatter,
+    )
+
+    # display help when no argument is called
+    parser.set_defaults(func=lambda x: parser.print_help())
+
+    # add the subparsers
+    subparsers = parser.add_subparsers(dest="function")
+
+    # add the identify parser
+    parser_identify = subparsers.add_parser(
+        "identify", help="Run the BOLD v5 identification engine"
+    )
+
+    # add the fasta path argument
+    parser_identify.add_argument(
+        "fasta_file",
+        help="Path to the fasta file or fasta file in the current working directory to be identified.",
+        type=str,
+    )
+
+    # add the database argument
+    parser_identify.add_argument(
+        "--db",
+        required=True,
+        help="Integer that defines which database to use (1 to 7). See readme for details",
+        type=int,
+        choices=range(1, 8),
+    )
+
+    # add the operating mode argument
+    parser_identify.add_argument(
+        "--mode",
+        required=True,
+        help="Integer that defines which operating mode to use (1 to 3). See readme for details.",
+        type=int,
+        choices=range(1, 4),
+    )
+
+    # add the optional argument thresholds
+    parser_identify.add_argument(
+        "--thresholds",
+        nargs="+",
+        type=int,
+        help="Thresholds to use for the selection of the top hit.",
+    )
+
+    # add version control
+    # get the installed version
+    current_version = version("boldigger2")
+    obtainer = GetPyPiLatestVersion()
+    latest_version = obtainer("boldigger2")
+
+    # give a user warning if the latest version is not installed
+    if current_version != latest_version:
+        print(
+            "{}: Your boldigger3 version is outdated. Consider updating to the latest version.".format(
+                datetime.datetime.now().strftime("%H:%M:%S")
+            )
+        )
+
+    # add the version argument
+    parser.add_argument("--version", action="version", version=version("boldigger2"))
+
+    # parse the arguments
+    arguments = parser.parse_args()
+
+    # print help if no argument is provided
+    if len(sys.argv) == 1:
+        arguments.func(arguments)
+        sys.exit()
+
+    # only use the threshold provided by the user replace the rest with defaults
+    default_thresholds = [97, 95, 90, 85]
+    thresholds = []
+
+    for i in range(4):
+        try:
+            thresholds.append(arguments.thresholds[i])
+        except (IndexError, TypeError):
+            thresholds.append(default_thresholds[i])
+
+    if arguments.thresholds:
+        # give user output
+        print(
+            "{}: Default thresholds changed!\n{}: Species: {}, Genus: {}, Family: {}, Order: {}".format(
+                datetime.datetime.now().strftime("%H:%M:%S"),
+                datetime.datetime.now().strftime("%H:%M:%S"),
+                *thresholds
+            )
+        )
+
+    # run the identification engine
+    if arguments.function == "identify":
+        # run the id engine
+        id_engine.main(
+            arguments.fasta_file,
+            database=arguments.db,
+            operating_mode=arguments.mode,
+        )
+        # download the additional data
+        additional_data_download.main(arguments.fasta_file)
+        # select the top hit
+        select_top_hit.main(arguments.fasta_file, thresholds=thresholds)
+
+
+# run only if called as a top level script
+if __name__ == "__main__":
+    main()