eva-sub-cli 0.5.0__tar.gz → 0.5.1__tar.gz

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/.github/workflows/tests.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches: [ main ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, add-validation-tasks ]
 
 jobs:
   build:

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.5.0
+Version: 0.5.1
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/docs/how_to_submit.md

@@ -65,6 +65,21 @@ If you are working with large VCF files and find that validation takes a very lo
 argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
 shallow validation will **not** be sufficient for actual submission but will allow you to identify most validation concerns without long run time.
 
+### Validation tasks
+To run only a subset of the validation steps, you can use the `--validation_tasks` argument.
+This can be useful if you want to avoid re-running long-running validations that have already passed.
+Note that all validation tasks must pass in order to submit. The report will aggregate the results of previous runs and new ones.
+The possible tasks are:
+* `vcf_check` - includes syntax validation and other checks on VCF files
+* `assembly_check` - includes all checks involving the FASTA file
+* `metadata_check` - includes syntactic and semantic checks on metadata
+* `sample_check` - includes sample coherence checks between VCF files and metadata
+
+For example, to run only `vcf_check` and `sample_check`:
+```shell
+eva-sub-cli.py --metadata_xlsx <metadata file> --submission_dir <submission directory> --validation_tasks vcf_check sample_check
+```
+
 ### VCF files and reference FASTA
 These can be provided either in the metadata file directly, or on the command line using the `--vcf_files` and
 `--reference_fasta` options. Note that if you are using more than one reference FASTA, you **cannot** use the command 

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/docs/input_file_overview.md

@@ -34,12 +34,12 @@ Body: The body of the VCF file contains the actual variant data, with each row r
 # Metadata Spreadsheet
 
 The spreadsheet provides comprehensive contextual information about the dataset, ensuring that each submission is accompanied by detailed descriptions that facilitate proper understanding and use of the data. Key elements included in the metadata spreadsheet are analysis and project information, sample information, sequencing methodologies, experimental details.
-The spreadsheet is composed of editable tabs that will receive the metadata and non-editable helper tabs that provide detailed description of each column.
-
-| WORKSHEET         | EXPLANATION                                                                                                                                                                                                                                                                                                                                                       |
-|-------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Submitter Details | This sheet captures the details of the submitter                                                                                                                                                                                                                                                                                                                  |
-| Project           | The objective of this sheet is to gather general information about the Project including submitter, submitting centre, collaborators, project title, description and publications.                                                                                                                                                                                |
-| Sample            | Projects consist of analyses that are run on samples. We accept sample information in the form of BioSample, ENA or EGA accession(s). We also accept BioSamples sampleset accessions. If your samples are not yet accessioned, and are therefore novel, please use the "Novel sample(s)" sections of the Sample(s) worksheet to have them registered at BioSample |
-| Analysis          | For EVA, each analysis is one vcf file, plus an unlimited number of ancillary files. This sheet allows EVA to link vcf files to a project and to other EVA analyses. Additionally, this worksheet contains experimental meta-data detailing the methodology of each analysis. Important to note; one project can have multiple associated analyses                |
-| Files             | Filenames and associated checking data associated with this EVA submission should be entered into this worksheet. Each file should be linked to exactly one analysis.                                                                                                                                                                                             |
+The spreadsheet is organized into editable tabs, designed for metadata entry, and non-editable helper tabs, which offer detailed explanations and guidance for each column. Users are required to complete all relevant sections within the editable tabs. Mandatory fields in each section are indicated in bold to highlight essential information that must be provided for a valid submission. However, users are strongly encouraged to provide as much additional information as possible to enhance the completeness and usefulness of the metadata.
+
+| WORKSHEET         | EXPLANATION                                                                                                                                                                                                                                                                                                                                                                                                        |
+|-------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Submitter Details | This sheet captures the details of the submitter                                                                                                                                                                                                                                                                                                                                                                   |
+| Project           | The objective of this sheet is to gather general information about the Project. If you are submitting to an existing project, you can skip the other details and just provide the project accession and analyses will be linked to that project. In case of a new project, please provide the relevant details including submitter, submitting centre, collaborators, project title, description and publications. |
+| Sample            | Projects consist of analyses that are run on samples. We accept sample information in the form of BioSample, ENA or EGA accession(s). We also accept BioSamples sampleset accessions. If your samples are not yet accessioned, and are therefore novel, please use the "Novel sample(s)" sections of the Sample(s) worksheet to have them registered at BioSample                                                  |
+| Analysis          | For EVA, each analysis is one vcf file, plus an unlimited number of ancillary files. This sheet allows EVA to link vcf files to a project and to other EVA analyses. Additionally, this worksheet contains experimental meta-data detailing the methodology of each analysis. Important to note; one project can have multiple associated analyses                                                                 |
+| Files             | Filenames and associated checking data associated with this EVA submission should be entered into this worksheet. Each file should be linked to exactly one analysis.                                                                                                                                                                                                                                              |

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/docs/validation_overview.md

@@ -17,7 +17,7 @@ Key points to note before validating your metadata spreadsheet with the eva-sub-
 
 - Please do not change the existing structure of the spreadsheet
 - Ensure all mandatory columns (marked in bold) are filled.
-- Pre-registered samples must be released and not kept in private status
+- Pre-registered project and samples must be released and not kept in private status
 - Sample names in the spreadsheet must match those in the VCF file.
 - Analysis aliases must match across the sheets (Analysis, Sample, and File sheets).
 

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.5.0'
-__version_tuple__ = version_tuple = (0, 5, 0)
+__version__ = version = '0.5.1'
+__version_tuple__ = version_tuple = (0, 5, 1)
 
-__commit_id__ = commit_id = 'g1b663fd41'
+__commit_id__ = commit_id = 'g8a5f488cb'

eva_sub_cli-0.5.1/eva_sub_cli/exceptions/submission_upload_exception.py

@@ -0,0 +1,5 @@
+
+class SubmissionUploadException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/executables/cli.py

@@ -1,14 +1,11 @@
 import sys
 
-import eva_sub_cli
-from eva_sub_cli.exceptions.metadata_template_version_exception import MetadataTemplateVersionException, \
-    MetadataTemplateVersionNotFoundException
-from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
-from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
+from requests import HTTPError
+
+from eva_sub_cli.exceptions.submission_upload_exception import SubmissionUploadException
 
 if not sys.warnoptions:
     import warnings
-
     warnings.simplefilter("ignore")
 
 import logging
@@ -16,9 +13,15 @@ import os
 from argparse import ArgumentParser
 from ebi_eva_common_pyutils.logger import logging_config
 
+import eva_sub_cli
 from eva_sub_cli import orchestrator
-from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
+from eva_sub_cli.exceptions.metadata_template_version_exception import MetadataTemplateVersionException, \
+    MetadataTemplateVersionNotFoundException
+from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
+from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
 from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, DirLock
+from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
+from eva_sub_cli.validators.validator import ALL_VALIDATION_TASKS
 
 
 def validate_command_line_arguments(args, argparser):
@@ -93,6 +96,9 @@ def parse_args(cmd_line_args):
                            help='Select a task to perform (default SUBMIT). VALIDATE will run the validation'
                                 ' regardless of the outcome of previous runs. SUBMIT will run validate only if'
                                 ' the validation was not performed successfully before and then run the submission.')
+    argparser.add_argument('--validation_tasks', nargs='+', choices=ALL_VALIDATION_TASKS, default=ALL_VALIDATION_TASKS,
+                           type=str.lower, help='Select only a subset of the validation tasks to run. Note that all '
+                                                'tasks need to be successful for the validation to pass')
     argparser.add_argument('--executor', choices=[DOCKER, NATIVE], default=NATIVE, type=str.lower,
                            help='Select the execution type for running validation (default native)')
     credential_group = argparser.add_argument_group('Credentials', 'Specify the ENA Webin credentials you want to use '
@@ -125,7 +131,6 @@ def main():
 
     try:
         # lock the submission directory
-
         with DirLock(os.path.join(args.submission_dir)) as lock:
             # Create the log file
             logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
@@ -151,7 +156,15 @@ def main():
     except MetadataTemplateVersionNotFoundException as mte:
         print(mte)
         exit_status = 70
+    except SubmissionUploadException as sue:
+        print(sue)
+        exit_status = 71
+    except HTTPError as http_err:
+        print(http_err)
+        if http_err.response is not None and http_err.response.text:
+            print(http_err.response.text)
+        exit_status = 72
     except Exception as ex:
         print(ex)
-        exit_status = 71
+        exit_status = 73
     return exit_status

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/jinja_templates/html/file_validation.html

@@ -1,6 +1,6 @@
 
 {% macro file_validation_report(validation_results, file_name) -%}
-    {% for check_type, check_per_file in validation_results.items() %}
+    {% for check_type, check_per_file in validation_results.items() if check_type not in ["trim_down", "version"] %}
         {% set result = check_per_file.get(file_name, {}) %}
         {% if check_type == "assembly_check" %}
             {{ assembly_check(result) }}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/jinja_templates/html/report.html

@@ -6,6 +6,13 @@
 {% from 'metadata_validation.html' import metadata_validation_report %}
 {% from 'shallow_validation.html' import optional_shallow_validation_report %}
 
+{% macro validation_not_run_yet_message(text="Process not run yet") %}
+    {% set icon = "⏸" %}   {# pause symbol ⏸ #}
+    {% set row_class = "report-section info" %}
+    {% set expand_icon = "" %}
+    <div class='{{ row_class }}'><span class="expand_icon">{{ expand_icon }}</span> {{ icon }} {{ text }}</div>
+{% endmacro %}
+
 <html lang="EN">
 <head>
     <meta charset="UTF-8">
@@ -63,7 +70,12 @@
         Ensures that required fields are present and values are formatted correctly.
         For requirements, please refer to the <a href="https://www.ebi.ac.uk/eva/?Submit-Data" target=”_blank”>EVA website</a>.
     </div>
-    {{ metadata_validation_report(validation_results) }}
+    {% set run_status = validation_results.get('metadata_check', {}).get('run_status', '') %}
+    {% if run_status %}
+        {{ metadata_validation_report(validation_results) }}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+    {% endif %}
 </section>
 
 <section>
@@ -72,12 +84,17 @@
         Checks whether each file is compliant with the <a href="http://samtools.github.io/hts-specs/VCFv4.4.pdf" target=”_blank”>VCF specification</a>.
         Also checks whether the variants' reference alleles match against the reference assembly.
     </div>
-    {% for file_name in vcf_files %}
-        {% if file_name != "pass"%}
-        <h3>{{ file_name }}</h3>
-        {{ file_validation_report(validation_results, file_name) }}
-        {% endif %}
-    {% endfor %}
+    {% set run_status = validation_results.get('vcf_check', {}).get('run_status', '') %}
+    {% if run_status %}
+        {% for file_name in vcf_files %}
+            {% if file_name != "pass"%}
+            <h3>{{ file_name }}</h3>
+            {{ file_validation_report(validation_results, file_name) }}
+            {% endif %}
+        {% endfor %}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+    {% endif %}
 </section>
 
 <section>
@@ -85,7 +102,13 @@
     <div class="description">
         Checks whether information in the metadata is concordant with that contained in the VCF files, in particular sample names.
     </div>
-    {{ sample_name_check_report(validation_results)}}
+    {% set run_status = validation_results.get('sample_check', {}).get('run_status', '') %}
+    {% if run_status %}
+        {{ sample_name_check_report(validation_results)}}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+
+    {% endif %}
 </section>
 
 <section>
@@ -94,12 +117,17 @@
         Checks that the reference sequences in the FASTA file used to call the variants are accessioned in INSDC.
         Also checks if the reference assembly accession in the metadata matches the one determined from the FASTA file.
     </div>
-    {% for file_name in fasta_files %}
-        {% if file_name != "pass"%}
-        <h3>{{ file_name }}</h3>
-        {{ fasta_check_report(validation_results, file_name) }}
-        {% endif %}
-    {% endfor %}
+    {% set run_status = validation_results.get('fasta_check', {}).get('run_status', '') %}
+    {% if run_status %}
+       {% for file_name in fasta_files %}
+            {% if file_name != "pass"%}
+            <h3>{{ file_name }}</h3>
+            {{ fasta_check_report(validation_results, file_name) }}
+            {% endif %}
+        {% endfor %}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+    {% endif %}
 </section>
 
 <script>

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/jinja_templates/html/shallow_validation.html

@@ -2,7 +2,7 @@
 {% macro optional_shallow_validation_report(validation_results) -%}
     {% set results = validation_results.get('shallow_validation', {}) %}
 
-    {% if results.get('required') %}
+    {% if validation_results.get('trim_down') %}
     <section>
         <div class="report-section fail collapsible"> <span class="expand_icon">&#9654;</span>
             &#10060; <b>You requested to run the shallow validation, please run full validation before submitting the data</b>

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/jinja_templates/text/file_validation.txt

@@ -1,6 +1,6 @@
 
 {% macro file_validation_report(validation_results, file_name) -%}
-    {% for check_type, check_per_file in validation_results.items() %}
+    {% for check_type, check_per_file in validation_results.items() if check_type not in ["trim_down", "version"] %}
         {% set result = check_per_file.get(file_name, {}) %}
         {% if check_type == "assembly_check" %}
             {{ assembly_check(result) }}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/jinja_templates/text/report.txt

@@ -5,6 +5,11 @@
 {% from 'metadata_validation.txt' import metadata_validation_report %}
 {% from 'shallow_validation.txt' import optional_shallow_validation_report %}
 
+{% macro validation_not_run_yet_message(text="Process not run yet") -%}
+{% set icon = "\u23F8" %}
+{{ icon }} {{ text }}
+{%- endmacro %}
+
 VALIDATION REPORT
 eva-sub-cli v{{cli_version}}
 
@@ -23,27 +28,41 @@ METADATA VALIDATION RESULTS
 Ensures that required fields are present and values are formatted correctly.
 For requirements, please refer to the EVA website (https://www.ebi.ac.uk/eva/?Submit-Data).
 
-{{ metadata_validation_report(validation_results) }}
-
+{% set run_status = validation_results.get('metadata_check', {}).get('run_status', '') %}
+{% if run_status %}
+    {{ metadata_validation_report(validation_results) }}
+{% else %}
+    {{ validation_not_run_yet_message() }}
+{% endif %}
 -
 
 VCF VALIDATION RESULTS
 Checks whether each file is compliant with the VCF specification (http://samtools.github.io/hts-specs/VCFv4.4.pdf).
 Also checks whether the variants' reference alleles match against the reference assembly.
 
+{% set run_status = validation_results.get('vcf_check', {}).get('run_status', '') %}
+{% if run_status %}
 {% for file_name in vcf_files %}
 {% if file_name != "pass"%}
-	{{ file_name }}
-	{{ file_validation_report(validation_results, file_name) }}
+    {{ file_name }}
+    {{ file_validation_report(validation_results, file_name) }}
 {% endif %}
 {% endfor %}
+{% else %}
+    {{ validation_not_run_yet_message() }}
+{% endif %}
 
 -
 
 SAMPLE NAME CONCORDANCE CHECK
 Checks whether information in the metadata is concordant with that contained in the VCF files, in particular sample names.
 
-{{ sample_name_check_report(validation_results)}}
+{% set run_status = validation_results.get('sample_check', {}).get('run_status', '') %}
+{% if run_status %}
+    {{ sample_name_check_report(validation_results) }}
+{% else %}
+    {{ validation_not_run_yet_message() }}
+{% endif %}
 
 -
 
@@ -51,9 +70,14 @@ REFERENCE GENOME INSDC CHECK
 Checks that the reference sequences in the FASTA file used to call the variants are accessioned in INSDC.
 Also checks if the reference assembly accession in the metadata matches the one determined from the FASTA file.
 
+{% set run_status = validation_results.get('fasta_check', {}).get('run_status', '') %}
+{% if run_status %}
 {% for file_name in fasta_files %}
 {% if file_name != "pass"%}
-	{{ file_name }}
-	{{ fasta_check_report(validation_results, file_name) }}
+    {{ file_name }}
+    {{ fasta_check_report(validation_results, file_name) }}
 {% endif %}
 {% endfor %}
+{% else %}
+    {{ validation_not_run_yet_message() }}
+{% endif %}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/jinja_templates/text/shallow_validation.txt

@@ -1,7 +1,7 @@
 {% macro optional_shallow_validation_report(validation_results) -%}
 
 {% set results = validation_results.get('shallow_validation', {}) %}
-{% if results.get('required') %}
+{% if validation_results.get('trim_down') %}
 
 {{ "\u274C" }} You requested to run the shallow validation, please run full validation before submitting the data
 {% for vcf_file in results.get('metrics') %}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/nextflow/validation.nf

@@ -7,6 +7,7 @@ def helpMessage() {
     Validate a set of VCF files and metadata to check if they are valid to be submitted to EVA.
 
     Inputs:
+            --tasks					    Which validation tasks to run
             --vcf_files_mapping         csv file with the mappings for vcf files, fasta and assembly report
             --output_dir                output_directory where the reports will be output
             --metadata_json             Json file describing the project, analysis, samples and files
@@ -15,6 +16,14 @@ def helpMessage() {
     """
 }
 
+// Values from validators.validator.ALL_VALIDATION_TASKS
+VCF_CHECK = 'vcf_check'
+ASSEMBLY_CHECK = 'assembly_check'
+METADATA_CHECK = 'metadata_check'
+SAMPLE_CHECK = 'sample_check'
+
+// Default to running all tasks
+params.tasks = [VCF_CHECK, ASSEMBLY_CHECK, METADATA_CHECK, SAMPLE_CHECK]
 params.vcf_files_mapping = null
 params.output_dir = null
 params.metadata_json = null
@@ -59,7 +68,7 @@ conversion_configuration = "${schema_dir}/$params.conversion_configuration_name"
 
 
 def joinBasePath(path) {
-    if (path){
+    if (path) {
         return params.base_dir + '/' + path
     }
     return 'NO_FILE'
@@ -76,36 +85,41 @@ workflow {
             file(joinBasePath(row.fasta)),
             file(joinBasePath(row.report))
         )}
-    if (params.shallow_validation){
+    if (params.shallow_validation) {
         // create a smaller vcf and fasta then replace the channel
         trim_down_vcf(vcf_and_ref_ch)
         vcf_and_ref_ch = trim_down_vcf.out.vcf_and_ref
     }
     vcf_files = vcf_and_ref_ch.map{row -> row[0]}
     fasta_to_vcfs = vcf_and_ref_ch.map{row -> tuple(row[1], row[0])}.groupTuple(by:0)
-    // VCF checks
-    check_vcf_valid(vcf_and_ref_ch)
-    check_vcf_reference(vcf_and_ref_ch)
-
-    generate_file_size_and_md5_digests(vcf_files)
-    collect_file_size_and_md5(generate_file_size_and_md5_digests.out.file_size_and_digest_info.collect())
-
 
     // Metadata conversion
     if (params.metadata_xlsx && !params.metadata_json){
-        convert_xlsx_2_json(joinBasePath(params.metadata_xlsx))
-        metadata_json = convert_xlsx_2_json.out.metadata_json
-    } else {
-        metadata_json = joinBasePath(params.metadata_json)
-    }
-    if (metadata_json) {
-        // Metadata checks and concordance checks
-        metadata_json_validation(metadata_json)
-        metadata_semantic_check(metadata_json)
-        sample_name_concordance(metadata_json, vcf_files.collect())
+		convert_xlsx_2_json(joinBasePath(params.metadata_xlsx))
+		metadata_json = convert_xlsx_2_json.out.metadata_json
+	} else {
+		metadata_json = joinBasePath(params.metadata_json)
+	}
+	// File size and MD5
+	generate_file_size_and_md5_digests(vcf_files)
+	collect_file_size_and_md5(generate_file_size_and_md5_digests.out.file_size_and_digest_info.collect())
+
+	// Task-specific processing
+    if (params.tasks.contains(VCF_CHECK)) {
+        check_vcf_valid(vcf_and_ref_ch)
         evidence_type_check(metadata_json, vcf_files.collect())
-        insdc_checker(metadata_json, fasta_to_vcfs)
-    }
+	}
+	if (params.tasks.contains(ASSEMBLY_CHECK)) {
+		check_vcf_reference(vcf_and_ref_ch)
+		insdc_checker(metadata_json, fasta_to_vcfs)
+	}
+	if (params.tasks.contains(METADATA_CHECK)) {
+		metadata_json_validation(metadata_json)
+		metadata_semantic_check(metadata_json)
+	}
+	if (params.tasks.contains(SAMPLE_CHECK)) {
+		sample_name_concordance(metadata_json, vcf_files.collect())
+	}
 }
 
 

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/orchestrator.py

@@ -28,7 +28,7 @@ from eva_sub_cli.submit import StudySubmitter, SUB_CLI_CONFIG_KEY_SUBMISSION_ID,
 from eva_sub_cli.utils import get_project_title_from_ena
 from eva_sub_cli.validators.docker_validator import DockerValidator
 from eva_sub_cli.validators.native_validator import NativeValidator
-from eva_sub_cli.validators.validator import READY_FOR_SUBMISSION_TO_EVA
+from eva_sub_cli.validators.validator import READY_FOR_SUBMISSION_TO_EVA, ALL_VALIDATION_TASKS
 
 VALIDATE = 'validate'
 SUBMIT = 'submit'
@@ -279,8 +279,8 @@ def check_validation_required(tasks, sub_config, username=None, password=None):
 
 
 def orchestrate_process(submission_dir, vcf_files, reference_fasta, metadata_json, metadata_xlsx,
-                        tasks, executor, username=None, password=None, shallow_validation=False, nextflow_config=None,
-                        **kwargs):
+                        tasks, executor, validation_tasks=ALL_VALIDATION_TASKS, username=None, password=None,
+                        shallow_validation=False, nextflow_config=None, **kwargs):
     # load config
     config_file_path = os.path.join(submission_dir, SUB_CLI_CONFIG_FILE)
     sub_config = WritableConfig(config_file_path, version=__version__)
@@ -309,13 +309,14 @@ def orchestrate_process(submission_dir, vcf_files, reference_fasta, metadata_jso
     if VALIDATE in tasks:
         if executor == DOCKER:
             validator = DockerValidator(vcf_files_mapping, submission_dir, project_title, metadata_json, metadata_xlsx,
-                                        metadata_xlsx_version, shallow_validation=shallow_validation,
-                                        submission_config=sub_config)
+                                        metadata_xlsx_version, validation_tasks=validation_tasks,
+                                        shallow_validation=shallow_validation, submission_config=sub_config)
         # default to native execution
         else:
             validator = NativeValidator(vcf_files_mapping, submission_dir, project_title, metadata_json, metadata_xlsx,
-                                        metadata_xlsx_version, shallow_validation=shallow_validation,
-                                        submission_config=sub_config, nextflow_config=nextflow_config)
+                                        metadata_xlsx_version, validation_tasks=validation_tasks,
+                                        shallow_validation=shallow_validation, submission_config=sub_config,
+                                        nextflow_config=nextflow_config)
         with validator:
             validator.validate_and_report()
             if not metadata_json:

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/report.py

@@ -17,12 +17,16 @@ def get_logo_data():
 
 def generate_report(validation_results, validation_date, submission_dir, vcf_fasta_analysis_mapping, project_title,
                     consent_statement_required, subdir, template_file):
+
+    from eva_sub_cli.validators.validator import RUN_STATUS_KEY, PASS
+
     results_for_report = {k: v for k, v in validation_results.items() if k != 'ready_for_submission_to_eva'}
     vcf_files = sorted(set([file_name
                             for check in results_for_report if check in ["vcf_check", "assembly_check"]
                             for file_name in results_for_report[check]
+                            if file_name not in {RUN_STATUS_KEY, PASS}
                             ]))
-    fasta_files = sorted([file_name for file_name in results_for_report['fasta_check']])
+    fasta_files = sorted([file_name for file_name in results_for_report['fasta_check'] if file_name not in {RUN_STATUS_KEY, PASS}])
     template = Environment(
         loader=FileSystemLoader(os.path.join(current_dir, 'jinja_templates', subdir))
     ).get_template(template_file)

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/submission_ws.py

@@ -7,6 +7,7 @@ from retry import retry
 
 from eva_sub_cli import SUBMISSION_WS_VAR
 from eva_sub_cli.auth import get_auth
+from eva_sub_cli.exceptions.submission_upload_exception import SubmissionUploadException
 
 
 class SubmissionWSClient(AppLogger):
@@ -43,10 +44,14 @@ class SubmissionWSClient(AppLogger):
 
     def mark_submission_uploaded(self, submission_id, metadata_json):
         response = requests.put(self._submission_uploaded_url(submission_id),
-                                headers={'Accept': 'application/json', 'Authorization': 'Bearer ' + self.auth.token},
-                                json=metadata_json)
-        response.raise_for_status()
-        return response.json()
+                            headers={'Accept': 'application/json', 'Authorization': 'Bearer ' + self.auth.token},
+                            json=metadata_json)
+        if 400 <= response.status_code < 500:
+            raise SubmissionUploadException(f"There was an error in uploading the submission {submission_id}."
+                                            f"Status: {response.status_code}. Error: {response.text}")
+        else:
+            response.raise_for_status()
+            return response.json()
 
     def initiate_submission(self):
         response = requests.post(self._submission_initiate_url(), headers={'Accept': 'application/json',

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/submit.py

@@ -57,6 +57,8 @@ class StudySubmitter(AppLogger):
 
         submission_upload_url = self.sub_config[SUB_CLI_CONFIG_KEY_SUBMISSION_UPLOAD_URL]
 
+        self._upload_file(submission_upload_url, self.metadata_json_file)
+
         for f in self.vcf_files:
             self._upload_file(submission_upload_url, f)
 

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.1}/eva_sub_cli/validators/docker_validator.py

@@ -7,12 +7,12 @@ import time
 from ebi_eva_common_pyutils.logger import logging_config
 from retry import retry
 
-from eva_sub_cli.validators.validator import Validator
+from eva_sub_cli.validators.validator import Validator, ALL_VALIDATION_TASKS
 
 logger = logging_config.get_logger(__name__)
 
 default_container_image = 'ebivariation/eva-sub-cli'
-default_container_tag = 'v0.0.7'
+default_container_tag = 'v0.0.8'
 container_validation_dir = '/opt/vcf_validation'
 container_validation_output_dir = 'vcf_validation_output'
 
@@ -20,11 +20,13 @@ container_validation_output_dir = 'vcf_validation_output'
 class DockerValidator(Validator):
 
     def __init__(self, mapping_file, submission_dir, project_title, metadata_json=None,
-                 metadata_xlsx=None, metadata_xlsx_version=None, shallow_validation=False, docker_path='docker',
-                 submission_config=None, container_image=None, container_tag=None, container_name=None):
+                 metadata_xlsx=None, metadata_xlsx_version=None, validation_tasks=ALL_VALIDATION_TASKS,
+                 shallow_validation=False, docker_path='docker', submission_config=None, container_image=None,
+                 container_tag=None, container_name=None):
         super().__init__(mapping_file, submission_dir, project_title, metadata_json=metadata_json,
                          metadata_xlsx=metadata_xlsx, metadata_xlsx_version=metadata_xlsx_version,
-                         shallow_validation=shallow_validation, submission_config=submission_config)
+                         validation_tasks=validation_tasks, shallow_validation=shallow_validation,
+                         submission_config=submission_config)
         self.docker_path = docker_path
         submission_basename = re.sub('[^a-zA-Z0-9]', '', os.path.basename(submission_dir))
         self.container_image = container_image or default_container_image
@@ -42,6 +44,7 @@ class DockerValidator(Validator):
         docker_cmd = ''.join([
             f"{self.docker_path} exec {self.container_name} nextflow run eva_sub_cli/nextflow/validation.nf ",
             f"--base_dir {container_validation_dir} ",
+            f"--tasks {','.join(self.tasks)} "
             f"--vcf_files_mapping {self.mapping_file} ",
             f"--metadata_xlsx {self.metadata_xlsx} ",
             f"--conversion_configuration_name {self._get_xlsx_conversion_configuration_name()} "