eva-sub-cli 0.4__tar.gz → 0.4.4__tar.gz

{eva_sub_cli-0.4/eva_sub_cli.egg-info → eva_sub_cli-0.4.4}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.1
+Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.4
+Version: 0.4.4
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console
@@ -19,13 +19,14 @@ Classifier: License :: OSI Approved :: Apache Software License
 Requires-Python: >=3.6
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: ebi_eva_common_pyutils==0.6.10
+Requires-Dist: ebi_eva_common_pyutils==0.6.11
 Requires-Dist: jinja2
 Requires-Dist: jsonschema
 Requires-Dist: openpyxl
 Requires-Dist: pyyaml
 Requires-Dist: requests
 Requires-Dist: retry
+Dynamic: license-file
 
 # eva-sub-cli
 The eva-sub-cli tool is a command line interface tool for data validation and upload. The tool transforms the submission process at EVA by enabling users to take control of data validation process. Previously handled by our helpdesk team, validation can now be performed directly by users, streamlining and improving the overall submission workflow at the EVA. 
@@ -53,21 +54,9 @@ eva-sub-cli.py --help
 
 Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
 This method requires just Python 3.8+ and [Docker](https://docs.docker.com/engine/install/) to be installed.
-Once it is set up, you can either clone the git repository, or download the newest tagged release from [here](https://github.com/EBIvariation/eva-sub-cli/tags):
+Then you can install the most recent version from [PyPI](https://pypi.org/project/eva-sub-cli/) in a virtual environment:
 ```bash
-git clone https://github.com/EBIvariation/eva-sub-cli.git
-
-# OR (replace "v0.2" with the newest version)
-
-wget -O eva-sub-cli.zip https://github.com/EBIvariation/eva-sub-cli/archive/refs/tags/v0.2.zip
-unzip eva-sub-cli.zip && mv eva-sub-cli-* eva-sub-cli
-```
-
-Then install the library and its dependencies as follows (e.g. in a virtual environment):
-```bash
-cd eva-sub-cli
-# Activate your virtual environment 
-python -m pip install .
+pip install eva-sub-cli
 ```
 
 To verify that the cli tool is installed correctly, run the following command, and you should see the help message displayed : 
@@ -83,41 +72,47 @@ This installation method requires the following :
 * [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
 * [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.7+
 
-Install each of these and ensure they are included in your PATH. Then, either clone the repository using Git or install the latest release as previously described.
+Install each of these and ensure they are included in your PATH. Then install the latest release as previously described.
 
 ## Getting started with the eva-sub-cli tool 
 
-
 The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
 
-## eva-sub-cli tool: Options and parameters guide
+## Options and parameters guide
+
+The eva-sub-cli tool provides several options and parameters that you can use to tailor its functionality to your needs.
+You can view all the available parameters with the command `eva-sub-cli.py -h` and view detailed explanations for the
+input file requirements in the ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md).
+Below is an overview of the key parameters.
+
+### Submission directory
 
-The eva-sub-cli tool provides several options/parameters that you can use to tailor its functionality to your needs. Understanding these parameters is crucial for configuring the tool correctly. Below is an overview of the key parameters and options:
+This is the directory where all processing will take place, and where configuration and reports will be saved.
+Crucially, the eva-sub-cli tool requires that there be **only one submission per directory** and that the submission directory not be reused.
+Running multiple submissions from a single directory can result in data loss during validation and submission.
 
-| OPTIONS/PARAMETERS         | DESCRIPTION                                                                                                                                                                                                        |
-|----------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| --version                  | Shows version number of the program and exit                                                                                                                                                                       |
-| --metadata_xlsx            | Excel spreadsheet that describe the project, analysis, samples and files                                                                                                                                           |
-| --metadata_json            | Json file that describe the project, analysis, samples and files                                                                                                                                                   |
-| --vcf_files                | One or several vcf files to validate.This allows you to provide multiple VCF files to validate and a single associated reference genome file. The VCF files and the associated reference genome file must use the same chromosome naming convention                                                                                                                                                                              |
-| --reference_fasta          | The fasta file containing the reference genome from which the variants were derived                                                                                                                                |
-| --submission_dir           | Path to the directory where all processing will be done and submission data is/will be stored                                                                                                                      |
-| --tasks {validate,submit}  | Selecting VALIDATE will run the validation regardless of the outcome of previous runs. Selecting SUBMIT will run validate only if the validation was not performed successfully before and then run the submission |
-| --executor {docker,native} | Select an execution type for running validation (default native)                                                                                                                                                   |
-| --shallow                  | Set the validation to be performed on the first 10000 records of the VCF. Only applies if the number of record exceed 10000                                                                                        |
-| --username                 | Username used for connecting to the ENA webin account                                                                                                                                                              |
-| --password                 | Password used for connecting to the ENA webin account                                                                                                                                                              |
+### Metadata file
 
+Metadata can be provided in one of two files.
 
 #### The metadata spreadsheet
 
-The metadata template can be found within the etc folder at `eva_sub_cli/etc/EVA_Submission_template.xlsx`
-It should be populated following the instruction provided within the template
+The metadata template can be found within the [etc folder](eva_sub_cli/etc/EVA_Submission_template.xlsx). It should be populated following the instructions provided within the template.
+This is passed using the option `--metadata_xlsx`.
 
 #### The metadata JSON
 
 The metadata can also be provided via a JSON file, which should conform to the schema located [here](eva_sub_cli/etc/eva_schema.json).
+This is passed using the option `--metadata_json`.
+
+### VCF files and Reference FASTA
+
+These can be provided either in the metadata file directly, or on the command line using the `--vcf_files` and `--reference_fata` options.
+Note that if you are using more than one reference FASTA, you **cannot** use the command line options; you must specify which VCF files use which FASTA files in the metadata.
 
+VCF files can be either uncompressed or compressed using bgzip.
+Other types of compression are not allowed and will result in errors during validation.
+FASTA files must be uncompressed.
 
 ## Execution
 
@@ -141,7 +136,7 @@ eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submis
 
 ### Validate and submit your dataset
 
-To validate and submit run the following command
+To validate and submit, run the following command:
 
 ```shell
 eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx \
@@ -160,7 +155,7 @@ or
 ```shell
 eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks SUBMIT
 ```
-Will only submit the data and not validate.
+This will only submit the data and not validate.
 
 ### Shallow validation
 

eva_sub_cli-0.4.4/README.md

@@ -0,0 +1,134 @@
+# eva-sub-cli
+The eva-sub-cli tool is a command line interface tool for data validation and upload. The tool transforms the submission process at EVA by enabling users to take control of data validation process. Previously handled by our helpdesk team, validation can now be performed directly by users, streamlining and improving the overall submission workflow at the EVA. 
+
+
+## Installation
+
+There are currently three ways to install and run the tool : 
+- Using conda
+- From source using Docker
+- From source natively (i.e. installing dependencies yourself)
+
+### 1. Conda
+
+The most straightforward way to install eva-sub-cli and its dependencies is through conda.
+For instance, the following commands install eva-sub-cli in a new environment called `eva`, activate the environment, and print
+the help message:
+```bash
+conda create -n eva -c conda-forge -c bioconda eva-sub-cli
+conda activate eva
+eva-sub-cli.py --help
+````
+
+### 2. From source using Docker
+
+Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
+This method requires just Python 3.8+ and [Docker](https://docs.docker.com/engine/install/) to be installed.
+Then you can install the most recent version from [PyPI](https://pypi.org/project/eva-sub-cli/) in a virtual environment:
+```bash
+pip install eva-sub-cli
+```
+
+To verify that the cli tool is installed correctly, run the following command, and you should see the help message displayed : 
+```bash
+eva-sub-cli.py -h
+```
+
+### 3. From source natively
+
+This installation method requires the following :
+* Python 3.8+
+* [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html) 21.10+
+* [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
+* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.7+
+
+Install each of these and ensure they are included in your PATH. Then install the latest release as previously described.
+
+## Getting started with the eva-sub-cli tool 
+
+The ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md) serves as an introduction for users of the eva-sub-cli tool. It includes instructions on how to prepare your data and metadata, ensuring that users are equipped with the necessary information to successfully submit variant data. This guide is essential for new users, offering practical advice and tips for a smooth onboarding experience with the eva-sub-cli tool.
+
+## Options and parameters guide
+
+The eva-sub-cli tool provides several options and parameters that you can use to tailor its functionality to your needs.
+You can view all the available parameters with the command `eva-sub-cli.py -h` and view detailed explanations for the
+input file requirements in the ["Getting Started" guide](Getting_Started_with_eva_sub_cli.md).
+Below is an overview of the key parameters.
+
+### Submission directory
+
+This is the directory where all processing will take place, and where configuration and reports will be saved.
+Crucially, the eva-sub-cli tool requires that there be **only one submission per directory** and that the submission directory not be reused.
+Running multiple submissions from a single directory can result in data loss during validation and submission.
+
+### Metadata file
+
+Metadata can be provided in one of two files.
+
+#### The metadata spreadsheet
+
+The metadata template can be found within the [etc folder](eva_sub_cli/etc/EVA_Submission_template.xlsx). It should be populated following the instructions provided within the template.
+This is passed using the option `--metadata_xlsx`.
+
+#### The metadata JSON
+
+The metadata can also be provided via a JSON file, which should conform to the schema located [here](eva_sub_cli/etc/eva_schema.json).
+This is passed using the option `--metadata_json`.
+
+### VCF files and Reference FASTA
+
+These can be provided either in the metadata file directly, or on the command line using the `--vcf_files` and `--reference_fata` options.
+Note that if you are using more than one reference FASTA, you **cannot** use the command line options; you must specify which VCF files use which FASTA files in the metadata.
+
+VCF files can be either uncompressed or compressed using bgzip.
+Other types of compression are not allowed and will result in errors during validation.
+FASTA files must be uncompressed.
+
+## Execution
+
+### Validate only
+
+To validate and not submit, run the following command:
+
+```shell
+eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE
+```
+
+**Note for Docker users:** 
+
+Make sure that Docker is running in the background, e.g. by opening Docker Desktop.
+For each of the below commands, add the command line option `--executor docker`, which will
+fetch and manage the docker container for you. 
+
+```shell
+eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks VALIDATE --executor docker 
+```
+
+### Validate and submit your dataset
+
+To validate and submit, run the following command:
+
+```shell
+eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx \
+               --vcf_files vcf_file1.vcf vcf_file2.vcf --reference_fasta assembly.fa --submission_dir submission_dir
+```
+
+
+### Submit only
+
+All submissions must have been validated. You cannot run the submission without validation. Once validated, execute the following command:
+
+```shell
+eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir
+```
+or 
+```shell
+eva-sub-cli.py --metadata_xlsx metadata_spreadsheet.xlsx --submission_dir submission_dir --tasks SUBMIT
+```
+This will only submit the data and not validate.
+
+### Shallow validation
+
+If you are working with large VCF files and find that validation takes a very long time, you can add the
+argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
+shallow validation will **not** be sufficient for actual submission.

eva_sub_cli-0.4.4/conda/build.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+
+BIOVALIDATOR_VERSION=2.2.1
+
+EVA_SUB_CLI="${PREFIX}/share/${PKG_NAME}-${PKG_VERSION}"
+mkdir -p ${EVA_SUB_CLI}
+
+# Install eva-sub-cli
+$PYTHON -m pip install .
+echo "Done with eva-sub-cli"
+
+cd ${EVA_SUB_CLI}
+
+# Install biovalidator from source
+curl -Lo biovalidator.zip https://github.com/elixir-europe/biovalidator/archive/refs/tags/v${BIOVALIDATOR_VERSION}.zip \
+    && unzip -q biovalidator.zip && rm biovalidator.zip \
+    && cd biovalidator-${BIOVALIDATOR_VERSION} \
+    && npm install && npm install -g \
+    && cd ..
+echo "Done with biovalidator"

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/conda/meta.yaml

@@ -1,37 +1,42 @@
 {% set name = "eva-sub-cli" %}
-{% set version = "0.2" %}
+{% set version = "0.4" %}
 
 package:
   name: {{ name }}
   version: {{ version }}
 
 source:
-  path: ../
-# For submission to bioconda recipes, replace 'path' with the following (filling in the appropriate sha256):
-#  url: https://github.com/EBIvariation/eva-sub-cli/archive/v{{version}}.tar.gz
-#  sha256:
+  url: https://github.com/EBIvariation/eva-sub-cli/archive/v{{version}}.tar.gz
+  sha256: be4d4109d9879ad94ccc01a3bd534ff6077775f126977dfb3f2d6fd9cfb251ba
 
 build:
   number: 0
   noarch: generic
-  preserve_egg_dir: True
-# For submission to bioconda recipes, replace 'preserve_egg_dir' with the following:
-#  run_exports:
-#    - {{ pin_subpackage(name, max_pin="x.x") }}
+  script_env:
+    - SETUPTOOLS_SCM_PRETEND_VERSION={{ version }}
+  run_exports:
+    - {{ pin_subpackage(name, max_pin="x.x") }}
 
 requirements:
   host:
     - nextflow >=21.10.0
     - python >=3.8
     - nodejs >=10.19.1
+    - vcf-validator >=0.10.0
+    - ebi-eva-common-pyutils >=0.6.1
     - pyyaml
     - jinja2
     - openpyxl
     - requests
     - jsonschema
+    - unzip
+    - setuptools-scm
   run:
     - nextflow >=21.10.0
     - python >=3.8
+    - nodejs >=10.19.1
+    - vcf-validator >=0.9.7
+    - ebi-eva-common-pyutils >=0.6.1
     - pyyaml
     - jinja2
     - openpyxl
@@ -43,7 +48,6 @@ test:
     - eva_sub_cli
   commands:
     - biovalidator --help
-    - vcf_validator --help
     - eva-sub-cli.py --help
 
 about:
@@ -54,4 +58,6 @@ about:
 
 extra:
   recipe-maintainers:
-    - apriltuesday
+    - apriltuesday
+    - tcezard
+    - ebi-variation

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/docker/Dockerfile

@@ -1,6 +1,6 @@
 FROM python:3.10
 
-ENV vcf_validator_version=0.9.7
+ENV vcf_validator_version=0.10.0
 ENV NXF_VER=22.10.6
 
 WORKDIR /opt

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/eva_sub_cli/_version.py

@@ -1,8 +1,13 @@
-# file generated by setuptools_scm
+# file generated by setuptools-scm
 # don't change, don't track in version control
+
+__all__ = ["__version__", "__version_tuple__", "version", "version_tuple"]
+
 TYPE_CHECKING = False
 if TYPE_CHECKING:
-    from typing import Tuple, Union
+    from typing import Tuple
+    from typing import Union
+
     VERSION_TUPLE = Tuple[Union[int, str], ...]
 else:
     VERSION_TUPLE = object
@@ -12,5 +17,5 @@ __version__: str
 __version_tuple__: VERSION_TUPLE
 version_tuple: VERSION_TUPLE
 
-__version__ = version = '0.4'
-__version_tuple__ = version_tuple = (0, 4)
+__version__ = version = '0.4.4'
+__version_tuple__ = version_tuple = (0, 4, 4)

eva_sub_cli-0.4.4/eva_sub_cli/exceptions/invalid_file_type_exception.py

@@ -0,0 +1,4 @@
+class InvalidFileTypeError(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/eva_sub_cli/executables/cli.py

@@ -15,7 +15,7 @@ from ebi_eva_common_pyutils.logger import logging_config
 
 from eva_sub_cli import orchestrator
 from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
-from eva_sub_cli.file_utils import is_submission_dir_writable
+from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, DirLock
 
 
 def validate_command_line_arguments(args, argparser):
@@ -68,7 +68,7 @@ def parse_args(cmd_line_args):
                                                                    'to submit to the EVA')
     credential_group.add_argument("--username", help="Username for your ENA Webin account")
     credential_group.add_argument("--password", help="Password for your ENA Webin account")
-    argparser.add_argument('--shallow', action='store_true', default=False,
+    argparser.add_argument('--shallow', action='store_true', default=False, dest='shallow_validation',
                            help='Set the validation to be performed on the first 10000 records of the VCF. '
                                 'Only applies if the number of records exceed 10000')
     argparser.add_argument('--debug', action='store_true', default=False,
@@ -79,6 +79,7 @@ def parse_args(cmd_line_args):
 
 
 def main():
+    exit_status = 0
     args = parse_args(sys.argv[1:])
 
     args.submission_dir = os.path.abspath(args.submission_dir)
@@ -87,14 +88,28 @@ def main():
         logging_config.add_stdout_handler(logging.DEBUG)
     else:
         logging_config.add_stdout_handler(logging.INFO)
-    logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
 
     try:
-        # Pass on all the arguments
-        orchestrator.orchestrate_process(**args.__dict__)
+        # lock the submission directory
+
+        with DirLock(os.path.join(args.submission_dir)) as lock:
+            # Create the log file
+            logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
+            # Pass on all the arguments to the orchestrator
+            orchestrator.orchestrate_process(**args.__dict__)
+    except DirLockError:
+        print(f'Could not acquire the lock file for {args.submission_dir} because another process is using this '
+              f'directory or a previous process did not terminate correctly. '
+              f'If the problem persists, remove the lock file manually.')
+        exit_status = 65
     except FileNotFoundError as fne:
         print(fne)
+        exit_status = 66
     except SubmissionNotFoundException as snfe:
         print(f'{snfe}. Please contact EVA Helpdesk')
+        exit_status = 67
     except SubmissionStatusException as sse:
         print(f'{sse}. Please try again later. If the problem persists, please contact EVA Helpdesk')
+        exit_status = 68
+    return exit_status
+

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/eva_sub_cli/file_utils.py

@@ -2,6 +2,7 @@ import glob
 import gzip
 import os
 import shutil
+import time
 from itertools import groupby
 
 
@@ -70,3 +71,47 @@ def fasta_iter(input_fasta):
             # join all sequence lines to one.
             seq = "".join(s.strip() for s in faiter.__next__())
             yield (headerStr, seq)
+
+
+class DirLockError(Exception):
+    pass
+
+
+class DirLock(object):
+
+    _SPIN_PERIOD_SECONDS = 0.05
+
+    def __init__(self, dirname, timeout=3):
+        """Prepare a file lock to protect access to dirname. timeout is the
+        period (in seconds) after an acquisition attempt is aborted.
+        The directory must exist, otherwise aquire() will timeout.
+        """
+        self._lockfilename = os.path.join(dirname, ".lock")
+        self._timeout = timeout
+
+    def acquire(self):
+        start_time = time.time()
+        while True:
+            try:
+                # O_EXCL: fail if file exists or create it (atomically)
+                os.close(os.open(self._lockfilename,
+                                 os.O_CREAT | os.O_EXCL | os.O_RDWR))
+                break
+            except OSError:
+                if (time.time() - start_time) > self._timeout:
+                    raise DirLockError(f"could not create {self._lockfilename} after {self._timeout} seconds")
+                else:
+                    time.sleep(self._SPIN_PERIOD_SECONDS)
+
+    def release(self):
+        try:
+            os.remove(self._lockfilename)
+        except OSError:
+            pass
+
+    def __enter__(self):
+        self.acquire()
+        return self
+
+    def __exit__(self, type_, value, traceback):
+        self.release()

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/eva_sub_cli/jinja_templates/html_report.html

@@ -73,8 +73,10 @@
         Also checks whether the variants' reference alleles match against the reference assembly.
     </div>
     {% for file_name in vcf_files %}
+        {% if file_name!= "pass"%}
         <h3>{{ file_name }}</h3>
         {{ file_validation_report(validation_results, file_name) }}
+        {% endif %}
     {% endfor %}
 </section>
 
@@ -93,8 +95,10 @@
         Also checks if the reference assembly accession in the metadata matches the one determined from the FASTA file.
     </div>
     {% for file_name in fasta_files %}
+        {% if file_name!= "pass"%}
         <h3>{{ file_name }}</h3>
         {{ fasta_check_report(validation_results, file_name) }}
+        {% endif %}
     {% endfor %}
 </section>
 

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/eva_sub_cli/nextflow/validation.nf

@@ -137,7 +137,6 @@ process check_vcf_valid {
     tuple path(vcf), path(fasta), path(report)
 
     output:
-    path "vcf_format/*.errors.*.db", emit: vcf_validation_db
     path "vcf_format/*.errors.*.txt", emit: vcf_validation_txt
     path "vcf_format/*.vcf_format.log", emit: vcf_validation_log
 
@@ -145,7 +144,7 @@ process check_vcf_valid {
     trap 'if [[ \$? == 1 ]]; then exit 0; fi' EXIT
 
     mkdir -p vcf_format
-    $params.executable.vcf_validator -i $vcf -r database,text -o vcf_format --require-evidence > vcf_format/${vcf}.vcf_format.log 2>&1
+    $params.executable.vcf_validator -i $vcf -r text -o vcf_format --require-evidence > vcf_format/${vcf}.vcf_format.log 2>&1
     """
 }
 

{eva_sub_cli-0.4 → eva_sub_cli-0.4.4}/eva_sub_cli/orchestrator.py

@@ -10,10 +10,12 @@ from ebi_eva_common_pyutils.logger import logging_config
 from openpyxl.reader.excel import load_workbook
 
 from eva_sub_cli import SUB_CLI_CONFIG_FILE, __version__
+from eva_sub_cli.exceptions.invalid_file_type_exception import InvalidFileTypeError
 from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
 from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
 from eva_sub_cli.submission_ws import SubmissionWSClient
-from eva_sub_cli.submit import StudySubmitter, SUB_CLI_CONFIG_KEY_SUBMISSION_ID
+from eva_sub_cli.submit import StudySubmitter, SUB_CLI_CONFIG_KEY_SUBMISSION_ID, \
+    SUB_CLI_CONFIG_KEY_SUBMISSION_UPLOAD_URL
 from eva_sub_cli.validators.docker_validator import DockerValidator
 from eva_sub_cli.validators.native_validator import NativeValidator
 from eva_sub_cli.validators.validator import READY_FOR_SUBMISSION_TO_EVA
@@ -35,7 +37,19 @@ def get_vcf_files(mapping_file):
     return vcf_files
 
 
-def get_project_title_and_create_vcf_files_mapping(submission_dir, vcf_files, reference_fasta, metadata_json, metadata_xlsx):
+def get_project_title_and_create_vcf_files_mapping(submission_dir, vcf_files, reference_fasta,
+                                                   metadata_json, metadata_xlsx):
+    """
+    Get project title and mapping between VCF files and reference FASTA files, from three sources: command line
+        arguments, metadata JSON file, or metadata XLSX file.
+
+    :param submission_dir: Directory where mapping file will be saved
+    :param vcf_files: VCF files from command line, if present
+    :param reference_fasta: Reference FASTA from command line, if present
+    :param metadata_json: Metadata JSON from command line, if present
+    :param metadata_xlsx: Metadata XLSX from command line, if present
+    :return: Project title and path to the mapping file
+    """
     mapping_file = os.path.join(submission_dir, 'vcf_mapping_file.csv')
     with open(mapping_file, 'w') as open_file:
         writer = csv.writer(open_file, delimiter=',')
@@ -44,7 +58,7 @@ def get_project_title_and_create_vcf_files_mapping(submission_dir, vcf_files, re
         vcf_files_mapping = []
         if vcf_files and reference_fasta:
             for vcf_file in vcf_files:
-                vcf_files_mapping.append([os.path.abspath(vcf_file), os.path.abspath(reference_fasta)])
+                vcf_files_mapping.append([os.path.abspath(vcf_file), os.path.abspath(reference_fasta), ''])
             if metadata_json:
                 project_title, _ = get_project_and_vcf_fasta_mapping_from_metadata_json(metadata_json, False)
             elif metadata_xlsx:
@@ -54,12 +68,32 @@ def get_project_title_and_create_vcf_files_mapping(submission_dir, vcf_files, re
         elif metadata_xlsx:
             project_title, vcf_files_mapping = get_project_and_vcf_fasta_mapping_from_metadata_xlsx(metadata_xlsx, True)
 
+        validate_vcf_mapping(vcf_files_mapping)
         for mapping in vcf_files_mapping:
             writer.writerow(mapping)
 
     return project_title, mapping_file
 
 
+def validate_vcf_mapping(vcf_mapping):
+    """
+    Validate that VCF files and FASTA files in the mapping are present and FASTA files are not compressed.
+
+    :param vcf_mapping: iterable of triples (VCF file path, reference FASTA path, optional assembly report path)
+    :return:
+    """
+    for vcf_file, fasta_file, report_file in vcf_mapping:
+        if not (vcf_file and os.path.isfile(vcf_file)):
+            raise FileNotFoundError(f'The variant file {vcf_file} does not exist, please check the file path.')
+        if not (fasta_file and os.path.isfile(fasta_file)):
+            raise FileNotFoundError(f'The reference fasta {fasta_file} does not exist, please check the file path.')
+        if fasta_file.lower().endswith('gz'):
+            raise InvalidFileTypeError(f'The reference fasta {fasta_file} is compressed, please uncompress the file.')
+        if report_file and not os.path.isfile(report_file):
+            raise FileNotFoundError(f'The assembly report file {report_file} does not exist, please check the file '
+                                    f'path.')
+
+
 def get_project_and_vcf_fasta_mapping_from_metadata_json(metadata_json, mapping_req=False):
     with open(metadata_json) as file:
         json_metadata = json.load(file)
@@ -117,10 +151,6 @@ def get_project_and_vcf_fasta_mapping_from_metadata_xlsx(metadata_xlsx, mapping_
             file_name = os.path.abspath(row[files_headers['File Name']])
             analysis_alias = row[files_headers['Analysis Alias']]
             reference_fasta = os.path.abspath(analysis_alias_dict[analysis_alias])
-            if not (file_name and os.path.isfile(file_name)):
-                raise FileNotFoundError(f'The variant file {file_name} provided in spreadsheet {metadata_xlsx} does not exist')
-            if not (reference_fasta and os.path.isfile(reference_fasta)):
-                raise FileNotFoundError(f'The reference fasta {reference_fasta} in spreadsheet {metadata_xlsx} does not exist')
             vcf_fasta_report_mapping.append([os.path.abspath(file_name), os.path.abspath(reference_fasta), ''])
 
     return project_title, vcf_fasta_report_mapping
@@ -131,11 +161,15 @@ def check_validation_required(tasks, sub_config, username=None, password=None):
     if SUBMIT in tasks:
         if not sub_config.get(READY_FOR_SUBMISSION_TO_EVA, False):
             return True
+        # If we are working with an existing submission check its status to see if it was submitted and failed before.
         submission_id = sub_config.get(SUB_CLI_CONFIG_KEY_SUBMISSION_ID, None)
         if submission_id:
             try:
                 submission_status = SubmissionWSClient(username, password).get_submission_status(submission_id)
                 if submission_status == 'FAILED':
+                    # Reset the submission_id which will force the creation of a new one
+                    sub_config.set(SUB_CLI_CONFIG_KEY_SUBMISSION_ID, value=None)
+                    sub_config.set(SUB_CLI_CONFIG_KEY_SUBMISSION_UPLOAD_URL, value=None)
                     return True
                 else:
                     return False
@@ -151,7 +185,7 @@ def check_validation_required(tasks, sub_config, username=None, password=None):
                     raise SubmissionStatusException(f'Error occurred while getting status of the submission '
                                                     f'with Id {submission_id}')
 
-        logger.info(f'submission id not found in config. This might be the first time user is submitting')
+        logger.debug(f'submission id not found in config. This might be the first time user is submitting')
         return False