eva-sub-cli 0.4.4__tar.gz → 0.4.6__tar.gz

eva_sub_cli-0.4.6/.github/workflows/publish.yml

@@ -0,0 +1,51 @@
+name: Publish Python Package
+
+on:
+  push:
+    tags:
+      - '**'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.9"]
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python ${{ matrix.python-version }}
+      uses: actions/setup-python@v5
+      with:
+        python-version: ${{ matrix.python-version }}
+        cache: pip
+        cache-dependency-path: '**/pyproject.toml'
+    - name: Install dependencies
+      run: |
+        python -m pip install --upgrade pip
+        pip install pytest
+        python -m pip install .
+    - name: Run tests
+      run: |
+        PYTHONPATH=. pytest tests -k "not integration"
+  deploy:
+    runs-on: ubuntu-latest
+    needs: [test]
+    environment: release
+    permissions:
+      id-token: write
+    steps:
+    - uses: actions/checkout@v4
+    - name: Set up Python
+      uses: actions/setup-python@v5
+      with:
+        python-version: '3.9'
+        cache: pip
+        cache-dependency-path: '**/pyproject.toml'
+    - name: Install dependencies
+      run: |
+        pip install setuptools wheel build
+    - name: Build
+      run: |
+        python -m build
+    - name: Publish
+      uses: pypa/gh-action-pypi-publish@release/v1

{eva_sub_cli-0.4.4/eva_sub_cli.egg-info → eva_sub_cli-0.4.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.4.4
+Version: 0.4.6
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
@@ -19,13 +19,12 @@ Classifier: License :: OSI Approved :: Apache Software License
 Requires-Python: >=3.6
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ebi_eva_common_pyutils==0.6.11
-Requires-Dist: jinja2
-Requires-Dist: jsonschema
-Requires-Dist: openpyxl
-Requires-Dist: pyyaml
-Requires-Dist: requests
-Requires-Dist: retry
+Requires-Dist: ebi_eva_common_pyutils==0.*,>0.6.11
+Requires-Dist: jinja2==3.*,>=3.0.0
+Requires-Dist: jsonschema==4.*
+Requires-Dist: openpyxl==3.*
+Requires-Dist: pyyaml==6.*
+Requires-Dist: requests==2.*
 Dynamic: license-file
 
 # eva-sub-cli
@@ -76,13 +75,13 @@ Install each of these and ensure they are included in your PATH. Then install th
 
 ## Getting started with the eva-sub-cli tool 
 
-The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
+The ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
 
 ## Options and parameters guide
 
 The eva-sub-cli tool provides several options and parameters that you can use to tailor its functionality to your needs.
 You can view all the available parameters with the command `eva-sub-cli.py -h` and view detailed explanations for the
-input file requirements in the ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md).
+input file requirements in the ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md).
 Below is an overview of the key parameters.
 
 ### Submission directory
@@ -128,7 +127,7 @@ eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submis
 
 Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
 For each of the below commands, add the command line option `--executor docker`, which will
-fetch and manage the docker container for you. 
+fetch and manage the Docker container for you. 
 
 ```shell
 eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker 
@@ -162,3 +161,53 @@ This will only submit the data and not validate.
 If you are working with large VCF files and find that validation takes a very long time, you can add the
 argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
 shallow validation will **not** be sufficient for actual submission.
+
+
+## Leveraging Nextflow to parallelize the validation process  
+
+Nextflow is a common workflow management system that helps orchestrate tasks and interface with the execution engine (like HPC or cloud). When running natively (i.e. not using Docker), eva-sub-cli will use Nextflow to run all the validation steps. In this section we'll see how it can be parameterised to work with your compute infrastructure.
+
+When no options are provided, Nextflow will run as many tasks as there are available CPUs on the machine executing it. To modify how many tasks can start and how Nextflow will process each one, you can provide a Nextflow configuration file in several ways.
+
+From the command line you can use `--nextflow_config <path>` to specify the Nextflow config file you want to apply. The configuration can also be picked up from other places directly by Nextflow. Please refer to [the nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more details.
+
+### Basic Nextflow configuration.
+
+There are many options to configure Nextflow so we will not provide them all. Please refer to [the documentation](https://www.nextflow.io/docs/latest/reference/config.html) for advanced features.
+Below is a very basic Nextflow configuration file that will request 2 cpus for each process, essentially limiting the number of process to half the number of available CPUs 
+```
+process {
+    executor="local"
+    cpus=2
+}
+```
+In this configuration, all the process will be running on the same machine where eva-sub-cli was started as described in the schema below.
+```
+(Local machine)
+eva-sub-cli
+  |_ nextflow
+      |_ task1
+      |_ task2
+```
+
+### Basic Nextflow configuration for HPC use.
+
+If you have access to High Performance Compute (HPC) environment, Nextflow supports the main resource managers such as [SLURM](https://www.nextflow.io/docs/latest/executor.html#slurm), [SGE](https://www.nextflow.io/docs/latest/executor.html#sge), [LSF](https://www.nextflow.io/docs/latest/executor.html#lsf) and others.
+In the configuration below, we're assuming that you are using SLURM. It would work similarly with other resource managers.
+```
+process {
+    executor="slurm"
+    queue="my_production_queue"
+}
+```
+
+In this configuration, the subtasks will be performed in other machines as specified by your SLURM resource manager as described in the schema below.
+```
+(Local machine)
+eva-sub-cli
+  |_ nextflow
+(Other compute node)
+task1
+(Other compute node)
+task2
+```

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/README.md

@@ -46,13 +46,13 @@ Install each of these and ensure they are included in your PATH. Then install th
 
 ## Getting started with the eva-sub-cli tool 
 
-The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
+The ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
 
 ## Options and parameters guide
 
 The eva-sub-cli tool provides several options and parameters that you can use to tailor its functionality to your needs.
 You can view all the available parameters with the command `eva-sub-cli.py -h` and view detailed explanations for the
-input file requirements in the ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md).
+input file requirements in the ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md).
 Below is an overview of the key parameters.
 
 ### Submission directory
@@ -98,7 +98,7 @@ eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submis
 
 Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
 For each of the below commands, add the command line option `--executor docker`, which will
-fetch and manage the docker container for you. 
+fetch and manage the Docker container for you. 
 
 ```shell
 eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker 
@@ -132,3 +132,53 @@ This will only submit the data and not validate.
 If you are working with large VCF files and find that validation takes a very long time, you can add the
 argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
 shallow validation will **not** be sufficient for actual submission.
+
+
+## Leveraging Nextflow to parallelize the validation process  
+
+Nextflow is a common workflow management system that helps orchestrate tasks and interface with the execution engine (like HPC or cloud). When running natively (i.e. not using Docker), eva-sub-cli will use Nextflow to run all the validation steps. In this section we'll see how it can be parameterised to work with your compute infrastructure.
+
+When no options are provided, Nextflow will run as many tasks as there are available CPUs on the machine executing it. To modify how many tasks can start and how Nextflow will process each one, you can provide a Nextflow configuration file in several ways.
+
+From the command line you can use `--nextflow_config <path>` to specify the Nextflow config file you want to apply. The configuration can also be picked up from other places directly by Nextflow. Please refer to [the nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more details.
+
+### Basic Nextflow configuration.
+
+There are many options to configure Nextflow so we will not provide them all. Please refer to [the documentation](https://www.nextflow.io/docs/latest/reference/config.html) for advanced features.
+Below is a very basic Nextflow configuration file that will request 2 cpus for each process, essentially limiting the number of process to half the number of available CPUs 
+```
+process {
+    executor="local"
+    cpus=2
+}
+```
+In this configuration, all the process will be running on the same machine where eva-sub-cli was started as described in the schema below.
+```
+(Local machine)
+eva-sub-cli
+  |_ nextflow
+      |_ task1
+      |_ task2
+```
+
+### Basic Nextflow configuration for HPC use.
+
+If you have access to High Performance Compute (HPC) environment, Nextflow supports the main resource managers such as [SLURM](https://www.nextflow.io/docs/latest/executor.html#slurm), [SGE](https://www.nextflow.io/docs/latest/executor.html#sge), [LSF](https://www.nextflow.io/docs/latest/executor.html#lsf) and others.
+In the configuration below, we're assuming that you are using SLURM. It would work similarly with other resource managers.
+```
+process {
+    executor="slurm"
+    queue="my_production_queue"
+}
+```
+
+In this configuration, the subtasks will be performed in other machines as specified by your SLURM resource manager as described in the schema below.
+```
+(Local machine)
+eva-sub-cli
+  |_ nextflow
+(Other compute node)
+task1
+(Other compute node)
+task2
+```

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/__init__.py

@@ -6,6 +6,7 @@ PACKAGE_DIR = os.path.dirname(os.path.abspath(__file__))
 NEXTFLOW_DIR = os.path.join(PACKAGE_DIR, 'nextflow')
 ETC_DIR = os.path.join(PACKAGE_DIR, 'etc')
 LSRI_CLIENT_ID = "aa0fcc42-096a-4f9d-b871-aceb1a97d174"
+DEFAULT_METADATA_XLSX_TEMPLATE_VERSION = '1.1.6'
 
 SUB_CLI_CONFIG_FILE = ".eva_sub_cli_config.yml"
 

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/_version.py

@@ -17,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.4.4'
-__version_tuple__ = version_tuple = (0, 4, 4)
+__version__ = version = '0.4.6'
+__version_tuple__ = version_tuple = (0, 4, 6)

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/etc/spreadsheet2json_conf.yaml

@@ -32,7 +32,7 @@ Project:
     Project Title: title
     Description: description
     Center: centre
-    Tax ID: taxId
+    Taxonomy ID: taxId
   optional:
     Publication(s): publications
     Parent Project: parentProject
@@ -74,6 +74,7 @@ Analysis:
     Imputation: boolean
     Phasing: boolean
     Date: date
+    Software: list
 
 Sample:
   header_row: 3
@@ -89,7 +90,7 @@ Sample:
     Prefix: prefix
     Subject: subject
     Derived From: derivedFrom
-    Tax Id: taxId
+    Taxonomy ID: taxId
     Scientific Name: scientificName
     Common Name: commonName
     mating_type: matingType

eva_sub_cli-0.4.6/eva_sub_cli/exceptions/metadata_template_version_exception.py

@@ -0,0 +1,10 @@
+class MetadataTemplateVersionException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class MetadataTemplateVersionNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/executables/cli.py

@@ -1,11 +1,14 @@
 import sys
 
 import eva_sub_cli
+from eva_sub_cli.exceptions.metadata_template_version_exception import MetadataTemplateVersionException, \
+    MetadataTemplateVersionNotFoundException
 from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
 from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
 
 if not sys.warnoptions:
     import warnings
+
     warnings.simplefilter("ignore")
 
 import logging
@@ -35,11 +38,16 @@ def validate_command_line_arguments(args, argparser):
         print(f"'{args.submission_dir}' does not have write permissions or is not a directory.")
         sys.exit(1)
 
+    if args.nextflow_config and not os.path.isfile(args.nextflow_config):
+        print(f"'{args.nextflow_config}' is not a file or does not exist.")
+        argparser.print_usage()
+        sys.exit(1)
+
 
 def parse_args(cmd_line_args):
     argparser = ArgumentParser(prog='eva-sub-cli',
                                description='EVA Submission CLI - validate and submit data to EVA. '
-                               'For full details, please see https://github.com/EBIvariation/eva-sub-cli')
+                                           'For full details, please see https://github.com/EBIvariation/eva-sub-cli')
     argparser.add_argument('--version', action='version', version=f'%(prog)s {eva_sub_cli.__version__}')
     argparser.add_argument('--submission_dir', required=True, type=str,
                            help='Path to the directory where all processing is done and submission info is stored')
@@ -71,6 +79,9 @@ def parse_args(cmd_line_args):
     argparser.add_argument('--shallow', action='store_true', default=False, dest='shallow_validation',
                            help='Set the validation to be performed on the first 10000 records of the VCF. '
                                 'Only applies if the number of records exceed 10000')
+    argparser.add_argument('--nextflow_config', type=str,
+                           help='Path to the configuration file that will be applied to the Nextflow process. '
+                                'This will override other nextflow configuration files you might have on your filesystem')
     argparser.add_argument('--debug', action='store_true', default=False,
                            help='Set the script to output debug messages')
     args = argparser.parse_args(cmd_line_args)
@@ -111,5 +122,10 @@ def main():
     except SubmissionStatusException as sse:
         print(f'{sse}. Please try again later. If the problem persists, please contact EVA Helpdesk')
         exit_status = 68
+    except MetadataTemplateVersionException as mte:
+        print(mte)
+        exit_status = 69
+    except MetadataTemplateVersionNotFoundException as mte:
+        print(mte)
+        exit_status = 70
     return exit_status
-

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/orchestrator.py

@@ -2,15 +2,23 @@
 import csv
 import json
 import os
+import re
 from collections import defaultdict
 
 import requests
 from ebi_eva_common_pyutils.config import WritableConfig
 from ebi_eva_common_pyutils.logger import logging_config
 from openpyxl.reader.excel import load_workbook
+from packaging import version
+from requests import HTTPError
+from retry import retry
 
+import eva_sub_cli
+from eva_sub_cli import DEFAULT_METADATA_XLSX_TEMPLATE_VERSION
 from eva_sub_cli import SUB_CLI_CONFIG_FILE, __version__
 from eva_sub_cli.exceptions.invalid_file_type_exception import InvalidFileTypeError
+from eva_sub_cli.exceptions.metadata_template_version_exception import MetadataTemplateVersionException, \
+    MetadataTemplateVersionNotFoundException
 from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
 from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
 from eva_sub_cli.submission_ws import SubmissionWSClient
@@ -120,6 +128,60 @@ def get_project_and_vcf_fasta_mapping_from_metadata_json(metadata_json, mapping_
     return project_title, vcf_fasta_report_mapping
 
 
+def get_sub_cli_version():
+    if version.parse(eva_sub_cli.__version__).is_devrelease:
+        major, minor, patch = map(int, version.parse(eva_sub_cli.__version__).base_version.split('.'))
+        if patch > 0:
+            patch -= 1
+        elif minor > 0:
+            minor -= 1
+            patch = 0
+        elif major > 0:
+            major -= 1
+            minor = 0
+            patch = 0
+        return f"{major}.{minor}.{patch}"
+    else:
+        return version.parse(eva_sub_cli.__version__).base_version
+
+
+@retry(exceptions=(HTTPError,), tries=3, delay=2, backoff=1.2, jitter=(1, 3))
+def get_sub_cli_github_tags():
+    url = f"https://api.github.com/repos/EBIvariation/eva-sub-cli/tags"
+    response = requests.get(url)
+    if response.status_code == 200:
+        tags = [tag["name"][1:] for tag in response.json()]
+        return tags
+    else:
+        return []
+
+
+def get_metadata_xlsx_template_link():
+    sub_cli_version = get_sub_cli_version()
+    sub_cli_tags = get_sub_cli_github_tags()
+    if sub_cli_version in sub_cli_tags:
+        return f'https://raw.githubusercontent.com/EBIvariation/eva-sub-cli/v{sub_cli_version}/eva-sub-cli/eva_sub_cli/etc/EVA_Submission_template.xlsx'
+    else:
+        return 'https://raw.githubusercontent.com/EBIvariation/eva-sub-cli/main/eva_sub_cli/etc/EVA_Submission_template.xlsx'
+
+
+def verify_metadata_xlsx_version(metadata_xlsx, min_req_version):
+    workbook = load_workbook(metadata_xlsx)
+    instructions_sheet = workbook['PLEASE READ FIRST']
+    xlsx_sheet_version_value = instructions_sheet[3][0].value
+    match = re.search(r'(\d+\.\d+\.\d+)', '' if xlsx_sheet_version_value is None else xlsx_sheet_version_value)
+    xlsx_version = match.group(1) if match else None
+    if xlsx_version:
+        if version.parse(xlsx_version) < version.parse(min_req_version):
+            raise MetadataTemplateVersionException(
+                f"Metadata template version {xlsx_version} is lower than min required {min_req_version}. "
+                f"Please download the correct template from EVA github project {get_metadata_xlsx_template_link()}")
+    else:
+        raise MetadataTemplateVersionNotFoundException(
+            f"No version information found in metadata xlsx sheet {metadata_xlsx}. "
+            f"Please download the correct template from EVA github project {get_metadata_xlsx_template_link()}")
+
+
 def get_project_and_vcf_fasta_mapping_from_metadata_xlsx(metadata_xlsx, mapping_req=False):
     workbook = load_workbook(metadata_xlsx)
 
@@ -148,10 +210,15 @@ def get_project_and_vcf_fasta_mapping_from_metadata_xlsx(metadata_xlsx, mapping_
             files_headers[cell.value] = cell.column - 1
 
         for row in files_sheet.iter_rows(min_row=2, values_only=True):
-            file_name = os.path.abspath(row[files_headers['File Name']])
+            file_name = row[files_headers['File Name']]
+            if file_name:
+                file_name = os.path.abspath(file_name)
             analysis_alias = row[files_headers['Analysis Alias']]
-            reference_fasta = os.path.abspath(analysis_alias_dict[analysis_alias])
-            vcf_fasta_report_mapping.append([os.path.abspath(file_name), os.path.abspath(reference_fasta), ''])
+            reference_fasta = analysis_alias_dict[analysis_alias]
+            if reference_fasta:
+                reference_fasta = os.path.abspath(reference_fasta)
+            if file_name and reference_fasta:
+                vcf_fasta_report_mapping.append([file_name, reference_fasta, ''])
 
     return project_title, vcf_fasta_report_mapping
 
@@ -190,7 +257,8 @@ def check_validation_required(tasks, sub_config, username=None, password=None):
 
 
 def orchestrate_process(submission_dir, vcf_files, reference_fasta, metadata_json, metadata_xlsx,
-                        tasks, executor, username=None, password=None, shallow_validation=False, **kwargs):
+                        tasks, executor, username=None, password=None, shallow_validation=False, nextflow_config=None,
+                        **kwargs):
     # load config
     config_file_path = os.path.join(submission_dir, SUB_CLI_CONFIG_FILE)
     sub_config = WritableConfig(config_file_path, version=__version__)
@@ -203,6 +271,8 @@ def orchestrate_process(submission_dir, vcf_files, reference_fasta, metadata_jso
         metadata_json = os.path.abspath(metadata_json)
     if metadata_xlsx:
         metadata_xlsx = os.path.abspath(metadata_xlsx)
+        # check metadata xlsx version is not lower than the required min metadata template version
+        verify_metadata_xlsx_version(metadata_xlsx, DEFAULT_METADATA_XLSX_TEMPLATE_VERSION)
 
     # Get the provided Project Title and VCF files mapping (VCF, Fasta and Report)
     project_title, vcf_files_mapping = get_project_title_and_create_vcf_files_mapping(
@@ -220,7 +290,8 @@ def orchestrate_process(submission_dir, vcf_files, reference_fasta, metadata_jso
         # default to native execution
         else:
             validator = NativeValidator(vcf_files_mapping, submission_dir, project_title, metadata_json, metadata_xlsx,
-                                        shallow_validation=shallow_validation, submission_config=sub_config)
+                                        shallow_validation=shallow_validation, submission_config=sub_config,
+                                        nextflow_config=nextflow_config)
         with validator:
             validator.validate_and_report()
             if not metadata_json:
@@ -229,5 +300,6 @@ def orchestrate_process(submission_dir, vcf_files, reference_fasta, metadata_jso
             sub_config.set('vcf_files', value=vcf_files)
 
     if SUBMIT in tasks:
-        with StudySubmitter(submission_dir, submission_config=sub_config, username=username, password=password) as submitter:
+        with StudySubmitter(submission_dir, submission_config=sub_config, username=username,
+                            password=password) as submitter:
             submitter.submit()

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/validators/docker_validator.py

@@ -12,7 +12,7 @@ from eva_sub_cli.validators.validator import Validator
 logger = logging_config.get_logger(__name__)
 
 container_image = 'ebivariation/eva-sub-cli'
-container_tag = 'v0.0.2'
+container_tag = 'v0.0.3'
 container_validation_dir = '/opt/vcf_validation'
 container_validation_output_dir = 'vcf_validation_output'
 

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6}/eva_sub_cli/validators/native_validator.py

@@ -13,10 +13,11 @@ class NativeValidator(Validator):
     def __init__(self, mapping_file, submission_dir, project_title, metadata_json=None, metadata_xlsx=None,
                  shallow_validation=False, vcf_validator_path='vcf_validator',
                  assembly_checker_path='vcf_assembly_checker', biovalidator_path='biovalidator',
-                 submission_config=None):
+                 submission_config=None, nextflow_config=None):
         super().__init__(mapping_file, submission_dir, project_title, metadata_json=metadata_json,
                          metadata_xlsx=metadata_xlsx, shallow_validation=shallow_validation,
                          submission_config=submission_config)
+        self.nextflow_config = nextflow_config
         self.vcf_validator_path = vcf_validator_path
         self.assembly_checker_path = assembly_checker_path
         self.biovalidator_path = biovalidator_path
@@ -44,14 +45,15 @@ class NativeValidator(Validator):
         path_to_workflow = os.path.join(os.path.dirname(os.path.dirname(os.path.abspath(__file__))),
                                         'nextflow/validation.nf')
         return ''.join([
-            f"nextflow run {path_to_workflow} ",
-            f"--vcf_files_mapping {self.mapping_file} ",
-            f"{metadata_flag} ",
-            f"--output_dir {self.output_dir} ",
-            f"--shallow_validation true " if self.shallow_validation else "",
-            f"--executable.vcf_validator {self.vcf_validator_path} ",
-            f"--executable.vcf_assembly_checker {self.assembly_checker_path} ",
-            f"--executable.biovalidator {self.biovalidator_path}"
+            f"nextflow run {path_to_workflow}",
+            f" --vcf_files_mapping {self.mapping_file}",
+            f" {metadata_flag}",
+            f" --output_dir {self.output_dir}",
+            f" --shallow_validation true " if self.shallow_validation else "",
+            f" --executable.vcf_validator {self.vcf_validator_path}",
+            f" --executable.vcf_assembly_checker {self.assembly_checker_path}",
+            f" --executable.biovalidator {self.biovalidator_path}",
+            f" -c {self.nextflow_config} " if self.nextflow_config else ""
         ])
 
     def verify_executables_installed(self):

{eva_sub_cli-0.4.4 → eva_sub_cli-0.4.6/eva_sub_cli.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.4.4
+Version: 0.4.6
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
@@ -19,13 +19,12 @@ Classifier: License :: OSI Approved :: Apache Software License
 Requires-Python: >=3.6
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ebi_eva_common_pyutils==0.6.11
-Requires-Dist: jinja2
-Requires-Dist: jsonschema
-Requires-Dist: openpyxl
-Requires-Dist: pyyaml
-Requires-Dist: requests
-Requires-Dist: retry
+Requires-Dist: ebi_eva_common_pyutils==0.*,>0.6.11
+Requires-Dist: jinja2==3.*,>=3.0.0
+Requires-Dist: jsonschema==4.*
+Requires-Dist: openpyxl==3.*
+Requires-Dist: pyyaml==6.*
+Requires-Dist: requests==2.*
 Dynamic: license-file
 
 # eva-sub-cli
@@ -76,13 +75,13 @@ Install each of these and ensure they are included in your PATH. Then install th
 
 ## Getting started with the eva-sub-cli tool 
 
-The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
+The ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
 
 ## Options and parameters guide
 
 The eva-sub-cli tool provides several options and parameters that you can use to tailor its functionality to your needs.
 You can view all the available parameters with the command `eva-sub-cli.py -h` and view detailed explanations for the
-input file requirements in the ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md).
+input file requirements in the ["Getting Started" guide](docs/Getting_Started_with_eva_sub_cli.md).
 Below is an overview of the key parameters.
 
 ### Submission directory
@@ -128,7 +127,7 @@ eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submis
 
 Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
 For each of the below commands, add the command line option `--executor docker`, which will
-fetch and manage the docker container for you. 
+fetch and manage the Docker container for you. 
 
 ```shell
 eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker 
@@ -162,3 +161,53 @@ This will only submit the data and not validate.
 If you are working with large VCF files and find that validation takes a very long time, you can add the
 argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
 shallow validation will **not** be sufficient for actual submission.
+
+
+## Leveraging Nextflow to parallelize the validation process  
+
+Nextflow is a common workflow management system that helps orchestrate tasks and interface with the execution engine (like HPC or cloud). When running natively (i.e. not using Docker), eva-sub-cli will use Nextflow to run all the validation steps. In this section we'll see how it can be parameterised to work with your compute infrastructure.
+
+When no options are provided, Nextflow will run as many tasks as there are available CPUs on the machine executing it. To modify how many tasks can start and how Nextflow will process each one, you can provide a Nextflow configuration file in several ways.
+
+From the command line you can use `--nextflow_config <path>` to specify the Nextflow config file you want to apply. The configuration can also be picked up from other places directly by Nextflow. Please refer to [the nextflow documentation](https://www.nextflow.io/docs/latest/config.html) for more details.
+
+### Basic Nextflow configuration.
+
+There are many options to configure Nextflow so we will not provide them all. Please refer to [the documentation](https://www.nextflow.io/docs/latest/reference/config.html) for advanced features.
+Below is a very basic Nextflow configuration file that will request 2 cpus for each process, essentially limiting the number of process to half the number of available CPUs 
+```
+process {
+    executor="local"
+    cpus=2
+}
+```
+In this configuration, all the process will be running on the same machine where eva-sub-cli was started as described in the schema below.
+```
+(Local machine)
+eva-sub-cli
+  |_ nextflow
+      |_ task1
+      |_ task2
+```
+
+### Basic Nextflow configuration for HPC use.
+
+If you have access to High Performance Compute (HPC) environment, Nextflow supports the main resource managers such as [SLURM](https://www.nextflow.io/docs/latest/executor.html#slurm), [SGE](https://www.nextflow.io/docs/latest/executor.html#sge), [LSF](https://www.nextflow.io/docs/latest/executor.html#lsf) and others.
+In the configuration below, we're assuming that you are using SLURM. It would work similarly with other resource managers.
+```
+process {
+    executor="slurm"
+    queue="my_production_queue"
+}
+```
+
+In this configuration, the subtasks will be performed in other machines as specified by your SLURM resource manager as described in the schema below.
+```
+(Local machine)
+eva-sub-cli
+  |_ nextflow
+(Other compute node)
+task1
+(Other compute node)
+task2
+```