genbenchQC 1.0.0__tar.gz

genbenchqc-1.0.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Katarina Gresova
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

genbenchqc-1.0.0/PKG-INFO

@@ -0,0 +1,25 @@
+Metadata-Version: 2.4
+Name: genbenchQC
+Version: 1.0.0
+Summary: Genomic Benchmarks QC: Automated Quality Control for Genomic Machine Learning Datasets
+Author: Katarina Gresova
+Author-email: gresova11@gmail.com
+Keywords: genomic benchmarks,deep learning,machine learning,computational biology,bioinformatics,genomics,quality control
+License-File: LICENSE
+Requires-Dist: numpy>=1.23
+Requires-Dist: pandas>=1.5
+Requires-Dist: matplotlib>=3.6
+Requires-Dist: seaborn>=0.12
+Requires-Dist: biopython>=1.8
+Requires-Dist: scikit-learn>=1.2
+Requires-Dist: cdhit-reader==0.2.0
+Requires-Dist: statsmodels>=0.13
+Provides-Extra: develop
+Requires-Dist: pytest>=3; extra == "develop"
+Dynamic: author
+Dynamic: author-email
+Dynamic: keywords
+Dynamic: license-file
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: summary

genbenchqc-1.0.0/README.md

@@ -0,0 +1,147 @@
+# Genomic Benchmarks QC: Automated Quality Control for Genomic Machine Learning Datasets
+
+Genomic Benchmarks QC is a Python package and CLI toolkit for automated quality control of genomic datasets used in machine learning.
+It helps detect biases, inconsistencies, and potential data leakage across sequences, dataset classes, and train-test splits — ensuring your datasets are reliable before model training.
+
+## Features
+
+### Provided Tools
+- **evaluate_sequences** – QC of a single dataset or dataset subset.
+- **evaluate_dataset** – QC across multiple dataset classes.
+- **evaluate_split** – Train–test split leakage detection.
+
+### General Features
+- [**Sequence-level QC**](#evaluate-sequences) – Evaluate nucleotide composition, sequence length distribution, GC content, and more.
+- [**Class-level QC**](#evaluate-dataset) – Compare multiple classes for feature similarity or bias.
+- [**Train–test split validation**](#evaluate-split) – Detect potential data leakage through sequence similarity and clustering.
+- [**Multiple input formats**](#supported-input-file-formats) – Supports FASTA, CSV, and TSV datasets.
+- **Customizable reporting** – Generate JSON, HTML, or simple text summaries.
+- **Integration-ready** – Available as both CLI tools and a Python API.
+- **Flexible sequence handling** – Works with single or multiple sequence columns.
+
+## Installation
+
+Install Genomic Benchmarks QC using pip:
+
+```bash
+pip install genbenchQC
+```
+
+If you plan to use `evaluate_split`, install [cd-hit](https://www.bioinformatics.org/cd-hit/cd-hit-user-guide):
+
+```bash
+conda install -c bioconda cd-hit
+# or follow: https://github.com/weizhongli/cdhit/wiki/2.-Installation
+```
+
+## Quick Start
+
+Clone the repository to access example datasets:
+
+```bash
+git clone https://github.com/katarinagresova/GenBenchQC.git
+cd GenBenchQC
+```
+
+### Evaluate Sequences
+
+```bash
+evaluate_sequences \
+  --input example_datasets/G4_positives.fasta \
+  --format fasta \
+  --out_folder example_outputs/G4_dataset_positives
+```
+
+The same evaluation would be executed from Python as follows:
+
+```python
+from genbenchQC import evaluate_sequences
+
+evaluate_sequences.run(
+  input='example_datasets/G4_positives.fasta', 
+  format='fasta',
+  out_folder='example_outputs/G4_dataset_positives'
+)
+```
+
+Outputs with their description are in [example_outputs/G4_dataset_positives](example_outputs/G4_dataset_positives).
+
+### Evaluate Dataset
+
+Running from CLI with fasta file:
+
+```bash
+evaluate_dataset \
+  --input example_datasets/G4_positives.fasta example_datasets/G4_negatives.fasta \
+  --format fasta \
+  --out_folder example_outputs/G4_dataset
+```
+
+Outputs with their description are in [example_outputs/G4_dataset](example_outputs/G4_dataset).
+
+Running from Python with CSV file with multiple sequence columns:
+
+```python
+from genbenchQC import evaluate_dataset
+
+evaluate_dataset.run(
+  input=['example_datasets/miRNA_mRNA_pairs_dataset.tsv'], 
+  format='tsv', 
+  out_folder='example_outputs/miRNA_mRNA_dataset', 
+  sequence_column=['gene', 'noncodingRNA'], 
+  label_column='label', 
+  label_list=['0', '1']
+)
+```
+
+Outputs with their description are in [example_outputs/miRNA_mRNA_dataset](example_outputs/miRNA_mRNA_dataset).
+
+### Evaluate Split
+
+```bash
+evaluate_split \
+  --train_input example_datasets/enhancers_train.csv \
+  --test_input example_datasets/enhancers_test.csv \
+  --format csv \
+  --sequence_column sequence \
+  --out_folder example_outputs/enhancers_dataset
+```
+
+The same evaluation would be executed from Python as follows:
+
+```python
+from genbenchQC import evaluate_split
+
+evaluate_split.run(
+  train_files=['example_datasets/enhancers_train.csv'],
+  test_files=['example_datasets/enhancers_test.csv'],
+  format='csv',
+  sequence_column='sequence',
+  out_folder='example_outputs/enhancers_dataset'
+)
+```
+
+Outputs with their description are in [example_outputs/enhancers_dataset](example_outputs/enhancers_dataset).
+
+## Supported input file formats
+
+You can choose to run the tool while having different dataset formats:
+- **FASTA**: The input is a FASTA file / list of FASTA files. One file needs to contain sequences of one class if running *evaluate_sequences* mode.
+- **CSV/TSV**: The input is a CSV/TSV file, and you provide the name of the column containing sequences. You can have either:
+  - **multiple files**, each one containing sequences from one class (similar as with FASTA input)
+  - **one file** containing sequences from multiple classes. In this case, when running *evaluate_sequences* mode, you need to provide the name of the column containing class labels so the tool can split the dataset into parts. The label classes can then be inferred, or you can specify their list by yourself. The dataset will then be split into pieces containing sequences with corresponding labels and analysis will be performed similarly as with multiple files.
+- **CSV.GZ/TSV.GZ**: Functionality is the same as CSV/TSV files
+
+When having CSV/TSV/CSV.GZ/TSV.GZ input, you can also decide to provide multiple sequence columns to analyze. In this case, the analysis in modes *evaluate_sequences* and *evaluate_dataset* will be performed for each column separately and lastly for sequences made by concatenating sequences throughout all the columns. 
+*evaluate_split* mode will run only the concatenated sequences.
+
+
+## Development
+
+If you want to help with the development of Genomic Benchmarks QC, you are more than welcome to join in!
+
+For a guidance, have a look at [CONTRIBUTING.md](CONTRIBUTING.md)
+
+## License
+
+Genomic Benchmarks QC is MIT-style licensed, as found in the [LICENSE](LICENSE) file.

genbenchqc-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

genbenchqc-1.0.0/setup.py

@@ -0,0 +1,40 @@
+from setuptools import find_packages, setup
+
+requirements = [
+    'numpy>=1.23',
+    'pandas>=1.5',
+    'matplotlib>=3.6',
+    'seaborn>=0.12',
+    'biopython>=1.8',
+    'scikit-learn>=1.2',
+    'cdhit-reader==0.2.0',
+    'statsmodels>=0.13',
+]
+
+test_requirements = [
+    'pytest>=3',
+]
+
+setup(
+    name='genbenchQC',
+    version='1.0.0',
+    description='Genomic Benchmarks QC: Automated Quality Control for Genomic Machine Learning Datasets',
+    author="Katarina Gresova",
+    author_email='gresova11@gmail.com',
+    packages=find_packages("src"),
+    package_dir={"": "src"},
+    install_requires=requirements,
+    extras_require={
+        "develop": test_requirements,
+    },
+    tests_require=["pytest"],
+    test_suite='tests',
+    entry_points='''
+      [console_scripts]
+      evaluate_sequences=genbenchQC.evaluate_sequences:main
+      evaluate_dataset=genbenchQC.evaluate_dataset:main
+      evaluate_split=genbenchQC.evaluate_split:main
+      ''',
+    keywords=["genomic benchmarks", "deep learning", "machine learning",
+      "computational biology", "bioinformatics", "genomics", "quality control"],
+)

genbenchqc-1.0.0/src/genbenchQC/evaluate_dataset.py

@@ -0,0 +1,287 @@
+import argparse
+import logging
+from pathlib import Path
+from itertools import combinations
+from typing import Optional
+import pandas as pd
+
+from genbenchQC.utils.statistics import SequenceStatistics
+from genbenchQC.utils.testing import flag_significant_differences
+from genbenchQC.report.report_generator import generate_json_report, generate_sequence_html_report, generate_simple_report, generate_dataset_html_report
+from genbenchQC.utils.input_utils import read_fasta, read_sequences_from_df, read_multisequence_df, read_csv_file, setup_logger
+
+def run_analysis(input_statistics, out_folder, report_types, seq_report_types, plot_type, flag_threshold):
+   
+    out_folder = Path(out_folder)
+
+    # run individual analysis
+    for s in input_statistics:
+        stats, end_position = s.compute()
+
+        if seq_report_types:
+
+            filename = Path(s.filename).stem
+            if s.seq_column is not None:
+                filename += f'_{s.seq_column}'
+            if s.label is not None:
+                filename += f'_{s.label}'
+
+            if 'json' in seq_report_types:
+                json_report_path = out_folder / Path(filename + '_report.json')
+                generate_json_report(stats, json_report_path)
+
+            if 'html' in seq_report_types:
+                html_report_path = out_folder / Path(filename + '_report.html')
+                plots_path = out_folder / Path(filename + '_plots')
+                generate_sequence_html_report(stats, html_report_path, plots_path, end_position, plot_type)
+
+    if len(input_statistics) < 2:
+        return
+
+    # run pair comparison analysis with all combinations
+    for stat1, stat2 in combinations(input_statistics, 2):
+        filename = "dataset_report"
+        if stat1.seq_column is not None:
+            filename += f'_{stat1.seq_column}'
+        if stat1.label is not None and stat2.label is not None:
+            filename += f'_label_{stat1.label}_vs_{stat2.label}'
+            logging.debug(f"Comparing datasets label: {stat1.label} vs {stat2.label}")
+        else:
+            filename += f'_{Path(stat1.filename).stem}_{Path(stat2.filename).stem}'
+            logging.debug(
+                f"Comparing datasets: {stat1.filename} vs {stat2.filename}")
+
+        results = flag_significant_differences(
+            stat1.sequences, stat1.stats, 
+            stat2.sequences, stat2.stats, 
+            threshold=flag_threshold, 
+            end_position=min(stat1.end_position, stat2.end_position)
+        )
+        
+        if 'simple' in report_types:
+            simple_report_path = out_folder / Path(f'{filename}.csv')
+            generate_simple_report(results, simple_report_path)
+
+        if 'html' in report_types:
+            html_report_path = out_folder / Path(f'{filename}.html')
+            plots_path = out_folder / Path(f'{filename}_plots')
+            generate_dataset_html_report(
+                stat1, stat2, results, 
+                html_report_path, 
+                plots_path=plots_path, 
+                threshold=flag_threshold,
+                end_position=min(stat1.end_position, stat2.end_position),
+                plot_type=plot_type
+            )
+
+def run(input, 
+        format, 
+        out_folder='.', 
+        sequence_column: Optional[list[str]] = ['sequences'], 
+        label_column='label', 
+        label_list: Optional[list[str]] = ['infer'],
+        regression: Optional[bool] = False,
+        report_types: Optional[list[str]] = ['html', 'simple'],
+        seq_report_types: Optional[list[str]] = None,
+        end_position: Optional[int] = None,
+        plot_type: Optional[str] = 'boxen',
+        flag_threshold: Optional[float] = 0.015,
+        log_level: Optional[str] = 'INFO',
+        log_file: Optional[str] = None
+    ):
+    """Run the dataset evaluation.
+
+    This function reads sequences from the provided input files, performs analysis, and generates reports about the sequences.
+
+    @param input: List of paths to input files. Can be a list of files, each containing sequences from one class.
+    @param format: Format of the input files (fasta, csv, csv.gz, tsv, tsv.gz).
+    @param out_folder: Path to the output folder. Default: '.'.
+    @param sequence_column: Name of the columns with sequences to analyze for datasets in CSV/TSV format. 
+                            Either one column or list of columns. Default: ['sequences']
+    @param label_column: Name of the label column for datasets in CSV/TSV format. Default: 'label'.
+    @param label_list: List of label classes to consider or "infer" to parse different labels automatically from label column.
+                      For datasets in CSV/TSV format.
+    @param regression: If True, label column is considered as a regression target and values are split into 2 classes.
+    @param report_types: Types of reports to generate. Default: ['html', 'simple'].
+    @param seq_report_types: Types of reports to generate for individual groups of sequences. Default: None.
+    @param end_position: End position of the sequences to consider in per position statistics. 
+                         If not provided, 75th percentile of sequence lengths will be used. Default: None.
+    @param plot_type: Type of plot to use for visualizations. For bigger datasets, "boxen" is recommended. Default: 'boxen'.
+    @param flag_threshold: Threshold for flagging significant differences in sequence statistics. Default: 0.015
+    @param log_level: Logging level, default to INFO.
+    @param log_file: Path to the log file. If provided, logs will be written to this file as well as to the console.
+    @return: None
+    """
+
+    setup_logger(log_level, log_file)
+    logging.info("Starting dataset evaluation.")
+
+    if not Path(out_folder).exists():
+        logging.info(f"Output folder {out_folder} does not exist. Creating it.")
+        Path(out_folder).mkdir(parents=True, exist_ok=True)
+
+    # we have multiple fasta files with one label each
+    if format == 'fasta':
+        seq_stats = []
+        for input_file in input:
+            sequences = read_fasta(input_file)
+            logging.debug(f"Read {len(sequences)} sequences from FASTA file {input_file}.")
+            seq_stats += [SequenceStatistics(sequences, filename=Path(input_file).name, 
+                                             label=Path(input_file).stem, end_position=end_position)]
+        run_analysis(
+            input_statistics = seq_stats, 
+            out_folder = out_folder, 
+            report_types = report_types, 
+            seq_report_types = seq_report_types, 
+            plot_type = plot_type, 
+            flag_threshold = flag_threshold
+        )
+
+    # we have CSV/TSV
+    else:
+        # we have one file with multiple labels or regression target
+        if len(input) == 1:
+            df = read_csv_file(input[0], format, sequence_column, label_column)
+
+            # if regression is True, we split the label column into two classes
+            if regression:
+                # convert the label column to numeric if it is not already
+                if not pd.api.types.is_numeric_dtype(df[label_column]):
+                    logging.debug(f"Converting label column '{label_column}' to numeric type for regression.")
+                    df[label_column] = pd.to_numeric(df[label_column], errors='coerce')
+                # infer the threshold as the median of the label column
+                threshold = df[label_column].median()
+                logging.debug(f"Inferred threshold for regression: {threshold}")
+                df[label_column] = df[label_column].apply(lambda x: 'high' if x >= threshold else 'low')
+                labels = ['high', 'low']
+
+            # get the list of labels to consider
+            elif len(label_list) == 1 and label_list[0] == 'infer':
+                labels = df[label_column].unique().tolist()
+                logging.debug(f"Inferred labels: {labels}")
+            else:
+                labels = label_list
+
+            # loop over sequences with specific label and run statistics
+            for seq_col in sequence_column:
+                seq_stats = []
+                for label in labels:
+                    sequences = read_sequences_from_df(df, seq_col, label_column, label)
+                    logging.debug(f"Read {len(sequences)} sequences for label '{label}' from column '{seq_col}'.")
+                    seq_stats += [SequenceStatistics(sequences, filename=Path(input[0]).name, label=label, 
+                                                     seq_column=seq_col, end_position=end_position)]
+                run_analysis(
+                    input_statistics = seq_stats, 
+                    out_folder = out_folder, 
+                    report_types = report_types, 
+                    seq_report_types = seq_report_types, 
+                    plot_type = plot_type, 
+                    flag_threshold = flag_threshold
+                )
+
+            # handle multiple sequence columns by concatenating sequences and running statistics on them
+            if len(sequence_column) > 1:
+                seq_stats = []
+                for label in labels:
+                    sequences = read_multisequence_df(df, sequence_column, label_column, label)
+                    seq_stats += [SequenceStatistics(sequences, filename=Path(input[0]).name, label=label,
+                                                     seq_column='_'.join(sequence_column))]
+                run_analysis(
+                    input_statistics = seq_stats, 
+                    out_folder = out_folder, 
+                    report_types = report_types, 
+                    seq_report_types = seq_report_types, 
+                    plot_type = plot_type, 
+                    flag_threshold = flag_threshold
+                )
+
+        # we have multiple files with one label each
+        else:
+            # run statistics across input files
+            for seq_col in sequence_column:
+                seq_stats = []
+                for input_file in input:
+                    sequences = read_sequences_from_df(read_csv_file(input_file, format, seq_col), seq_col)
+                    logging.debug(f"Read {len(sequences)} sequences from file {input_file} in column '{seq_col}'.")
+                    seq_stats += [SequenceStatistics(sequences, filename=Path(input_file).name, 
+                                                     label=Path(input_file).stem, seq_column=seq_col,
+                                                     end_position=end_position)]
+                run_analysis(
+                    input_statistics = seq_stats, 
+                    out_folder = out_folder, 
+                    report_types = report_types, 
+                    seq_report_types = seq_report_types, 
+                    plot_type = plot_type, 
+                    flag_threshold = flag_threshold
+                )
+
+            # handle multiple sequence columns
+            if len(sequence_column) > 1:
+                seq_stats = []
+                for input_file in input:
+                    sequences = read_multisequence_df(read_csv_file(input_file, format, sequence_column), sequence_column)
+                    seq_stats += [SequenceStatistics(sequences, filename=Path(input_file).name, label=Path(input_file).stem,
+                                                     seq_column='_'.join(sequence_column), end_position=end_position)]
+                run_analysis(
+                    input_statistics = seq_stats, 
+                    out_folder = out_folder, 
+                    report_types = report_types, 
+                    seq_report_types = seq_report_types, 
+                    plot_type = plot_type, 
+                    flag_threshold = flag_threshold
+                )
+
+    logging.info("Dataset evaluation successfully completed.")
+
+
+def parse_args():
+    parser = argparse.ArgumentParser(description='A tool for evaluating sequence datasets.')
+    parser.add_argument('--input', type=str, help='Path to the dataset file. '
+                                                  'Can be a list of files, each containing sequences from one class.', nargs='+', required=True)
+    parser.add_argument('--format', help="Format of the input files.", choices=['fasta', 'csv', 'csv.gz', 'tsv', 'tsv.gz'], required=True) # potentially add HF support
+    parser.add_argument('--sequence_column', type=str, help='Name of the columns with sequences to analyze for datasets in CSV/TSV format. '
+                                                            'Either one column or list of columns.', nargs='+', default=['sequence'])
+    parser.add_argument('--label_column', type=str, help='Name with the label column for datasets in CSV/TSV format.', default='label')
+    parser.add_argument('--label_list', type=str, nargs='+', help='List of label classes to consider or "infer" to parse different labels automatically from label column.'
+                                                       ' For datasets in CSV/TSV format.', default=['infer'])
+    parser.add_argument('--regression', action='store_true', help='If True, label column is considered as a regression target and values are split into 2 classes')
+    parser.add_argument('--out_folder', type=str, help='Path to the output folder.', default='.')
+    parser.add_argument('--report_types', type=str, nargs='+', choices=['json', 'html', 'simple'], default=['html', 'simple'],
+                        help='Types of reports to generate. Default: [html, simple].')
+    parser.add_argument('--seq_report_types', type=str, nargs='+', choices=['json', 'html'], default=[],
+                        help='Types of reports to generate for individual groups of sequences. Default: [].')
+    parser.add_argument('--end_position', type=int, default=None,
+                        help='End position of the sequences to consider in per position statistics. If not provided, 75th percentile of sequence lengths will be used.')
+    parser.add_argument('--plot_type', type=str, help='Type of plot to use for visualizations. For bigger datasets, "boxen" in recommended. Default: boxen.',
+                        choices=['boxen', 'violin'], default='boxen')
+    parser.add_argument('--flag_threshold', type=float, default=0.015,
+                        help='Threshold for flagging significant differences in sequence statistics. Default: 0.015')
+    parser.add_argument('--log_level', type=str, help='Logging level, default to INFO.', choices=['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL'], default='INFO')
+    parser.add_argument('--log_file', type=str, help='Path to the log file. If provided, logs will be written to this file as well as to the console.', default=None)
+    args = parser.parse_args()
+
+    if args.format == 'fasta' and len(args.input) < 2:
+        parser.error("When format is 'fasta', the input must contain individual files for each class.")
+
+    return args
+
+def main():
+    args = parse_args()
+    run(input = args.input, 
+        format = args.format, 
+        out_folder = args.out_folder, 
+        sequence_column = args.sequence_column, 
+        label_column = args.label_column, 
+        label_list = args.label_list,
+        regression = args.regression,
+        report_types = args.report_types,
+        seq_report_types = args.seq_report_types,
+        end_position = args.end_position,
+        plot_type = args.plot_type,
+        flag_threshold = args.flag_threshold,
+        log_level = args.log_level,
+        log_file = args.log_file
+    )
+
+if __name__ == '__main__':
+    main()