eva-sub-cli 0.5.0__tar.gz → 0.5.2__tar.gz

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/.github/workflows/tests.yml

@@ -4,7 +4,7 @@ on:
   push:
     branches: [ main ]
   pull_request:
-    branches: [ main ]
+    branches: [ main, add-validation-tasks ]
 
 jobs:
   build:

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.5.0
+Version: 0.5.2
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/docs/how_to_submit.md

@@ -65,14 +65,21 @@ If you are working with large VCF files and find that validation takes a very lo
 argument `--shallow` to the command, which will validate only the first 10,000 lines in each VCF. Note that running
 shallow validation will **not** be sufficient for actual submission but will allow you to identify most validation concerns without long run time.
 
-### VCF files and reference FASTA
-These can be provided either in the metadata file directly, or on the command line using the `--vcf_files` and
-`--reference_fasta` options. Note that if you are using more than one reference FASTA, you **cannot** use the command 
-line options; you must specify which VCF files use which FASTA files in the metadata.
-
-VCF files can be either uncompressed or compressed using bgzip.
-Other types of compression are not allowed and will result in errors during validation.
-FASTA files must be uncompressed.
+### Validation tasks
+To run only a subset of the validation steps, you can use the `--validation_tasks` argument.
+This can be useful if you want to avoid re-running long-running validations that have already passed.
+Note that all validation tasks must pass in order to submit. The report will aggregate the results of previous runs and new ones.
+The possible tasks are:
+* [`vcf_check`](validation_overview.md#vcf-checks) - includes syntax validation and other checks on VCF files
+${CODE_ROOT}/env/bin/upload_to_gcloud.py --input-file evidence.json.gz --destination-folder pharmacogenomics
+* [`assembly_check`](validation_overview.md#assembly-checks) - includes all checks involving the FASTA file
+* [`metadata_check`](validation_overview.md#metadata-check) - includes syntactic and semantic checks on metadata
+* [`sample_check`](validation_overview.md#sample-name-concordance-check) - includes sample coherence checks between VCF files and metadata
+
+For example, to run only `vcf_check` and `sample_check`:
+```shell
+eva-sub-cli.py --metadata_xlsx <metadata file> --submission_dir <submission directory> --validation_tasks vcf_check sample_check
+```
 
 ### Metadata JSON
 Frequent submitters may be interested in using our [metadata JSON schema](https://github.com/EBIvariation/eva-sub-cli/blob/main/eva_sub_cli/etc/eva_schema.json)

eva_sub_cli-0.5.2/docs/input_file_overview.md

@@ -0,0 +1,115 @@
+# Overview of Input Files
+
+View our [video tutorial on input files](https://embl-ebi.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=9284869a-9f57-43c6-8222-b3aa0158941e).
+
+The eva-sub-cli tool requires the following inputs:
+
+- One or several valid VCF files
+- Reference genome in FASTA format
+- Completed metadata spreadsheet
+
+Only the metadata file is passed directly to the CLI tool. The VCF and FASTA files should be referenced from the 
+metadata file.
+
+In the following sections, we will examine each of these inputs in detail.
+
+## VCF File
+
+A VCF (Variant Call Format) file is a type of file used in bioinformatics to store information about genetic variants.
+The EVA requires data files to conform to the official VCF specifications, so that the data can be interpreted 
+consistently by other databases and researchers looking to reuse the data. Many tools can be used to generate VCF files,
+including [BCFtools](https://samtools.github.io/bcftools/), [PLINK](https://www.cog-genomics.org/plink/2.0/), and 
+[GATK](https://gatk.broadinstitute.org).
+
+Besides being compliant with the VCF specification, we also require that each VCF file contains the necessary 
+evidence linking the variant to its biological source. This evidence can be in the form of sample genotypes or allele frequencies.
+
+Here is an example of how sample genotypes might look like in a VCF file:
+```
+##fileformat=VCFv4.2
+##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">
+#CHROM  POS ID  REF ALT QUAL    FILTER  INFO    FORMAT  SAMPLE1 SAMPLE2
+1   10583   rs58108140  G   A   100 PASS    .   GT  0/1 1/1
+```
+
+And here is how allele frequencies would look:
+```
+##fileformat=VCFv4.2
+##INFO=<ID=AF,Number=A,Type=Float,Description="Allele Frequency">
+#CHROM  POS ID  REF ALT QUAL    FILTER  INFO
+1   10583   rs58108140  G   A   100 PASS    AF=0.25
+```
+
+When using the eva-sub-cli tool, VCF files can be either uncompressed or compressed using bgzip or gzip.
+
+## FASTA File
+
+This is the reference genome that your variants were called against, in uncompressed FASTA format. The EVA uses this to
+check that the reference alleles are set correctly in the VCF files.
+
+The sequences of the reference genome must be INSDC-registered, which means they are publicly available in one of the archives within the
+INSDC consortium: [Genbank](https://www.ncbi.nlm.nih.gov/genbank/), [ENA](https://www.ebi.ac.uk/ena/browser/home), or
+[DDBJ](https://www.ddbj.nig.ac.jp/index-e.html). This ensures the long-term availability of the genome and the 
+reusability of your variation data.
+
+## Metadata Spreadsheet
+
+The metadata spreadsheet provides comprehensive contextual information about the dataset, ensuring that each submission
+is accompanied by detailed descriptions that facilitate proper understanding and use of the data. Key elements included
+in the metadata spreadsheet are analysis and project information, sample information, sequencing methodologies, and
+experimental details. While not all fields are required, users are strongly encouraged to provide as much information
+as possible to enhance the completeness and usefulness of the metadata.
+
+The spreadsheet is organized into editable tabs, designed for metadata entry, and non-editable helper tabs, which offer
+detailed explanations and guidance for each column. Users are required to complete all relevant sections within the
+editable tabs. Mandatory fields in each section are indicated in bold to highlight essential information that must be 
+provided for a valid submission. Fields highlighted in green indicate an either/or choice. This means you should fill
+in the choice most relevant for your submission, but not both.
+
+Below we go through some important details for each of the tabs.
+
+### Submitter Details
+
+This sheet captures basic information about the person or team submitting the data. This includes the lab name and
+center, which is the name of the submitting institution or organization and is the name that will be visible once the
+project is live on the EVA website.
+
+### Project
+
+The objective of this sheet is to gather general information about the Project. If you are submitting to an existing
+project, you can skip the other details and just provide the project accession; the data in this submission will be
+added as analyses linked to that project. In case of a new project, please provide the relevant details including 
+submitter, submitting center, collaborators, project title, description and publications.
+
+One important column to note is the Hold Date, which is the date until which the data should be kept private. EVA will
+release the data automatically after this date. If it is missing, the default value is three days after the date of
+submission.
+
+### Analysis
+
+For EVA, an analysis is a grouping of samples and data files. This sheet allows us to link VCF files to a project and
+to other EVA analyses. Additionally, this worksheet contains experimental metadata detailing the methodology of each
+analysis. This includes a local path to your reference FASTA file, as described [above](#FASTA-file).
+
+One project can have multiple associated analyses. EVA links analyses to samples and VCF files through the analysis 
+alias, which is a shortened identifier you must provide for each analysis.
+
+### Sample
+
+This is where you describe the biological samples used for your analyses. Each row describes one sample and must include
+the Analysis Alias to indicate which analysis it belongs to, and "Sample Name in VCF" which is the exact name of the 
+sample as it appears in the VCF file.
+
+We accept preregistered samples, which should be provided using BioSamples sample or sampleset accessions. Please
+ensure these are publicly accessible, as otherwise EVA will not be able to validate them.
+
+If your samples are not yet accessioned, and are therefore novel, please use the "Novel sample(s)" sections of the
+Sample worksheet to have them registered at BioSamples. Make sure to fill in the required fields in bold, including the
+BioSample name, title, taxonomy ID, geographic location (chosen from the controlled vocabulary in the drop-down menu),
+and collection date (in YYYY-MM-DD format).
+
+### Files
+
+This sheet lists the VCF files in your submission. These should be provided as local paths within your system, so that
+the CLI tool can locate them, along with the Analysis Alias which should be associated with each file. Each file should 
+be linked to exactly one analysis.

eva_sub_cli-0.5.2/docs/validation_overview.md

@@ -0,0 +1,103 @@
+# Overview of Validation Checks 
+
+The CLI tool performs the following validation checks and generates corresponding reports:
+
+- Metadata check to ensure that the metadata fields have been correctly filled in
+- VCF check to ensure that the VCF file follows the VCF format specification
+- Assembly check to ensure that the genome and the VCF match
+- Sample name check to ensure that the samples in the metadata can be associated with the samples in the VCF
+
+In the following sections, we will examine each of these checks in detail.
+
+## Metadata Check
+
+Once the user passes the metadata spreadsheet for validation checks, the eva-sub-cli tool verifies that all mandatory columns, marked in bold in the spreadsheet, are filled in.
+This data is crucial for further validation processes, such as retrieving the INDSC accession of the reference genome used to call the variants, and for sample and project metadata.
+
+Each tab in the spreadsheet has an associated help tab, which provides detailed instructions on how to fill in your metadata correctly.
+If any mandatory columns or sheets are missing, the CLI tool will raise errors.
+
+Key points to note before validating your metadata spreadsheet with the eva-sub-cli tool:
+
+- Please do not change the existing structure of the spreadsheet.
+- Ensure all mandatory columns (marked in bold) are filled.
+- Columns marked in green indicate an either/or requirement, so only one of the sections should be filled in.
+- Any pre-registered projects and samples must be released and not kept in private status.
+- Sample names in the spreadsheet must match those in the VCF file.
+- Analysis aliases must match across the sheets (Analysis, Sample, and File sheets).
+- Use the Hold Date field in the Project sheet for data that needs to be kept under embargo.
+
+Common errors seen with metadata checks:
+
+- Analysis alias is not filled in for the respective samples in the Sample tab.
+- Reference field in the Analysis tab is not filled with an INSDC accession. Submitters should not use a non-GCA accession or generic assembly name as their reference genome.
+- Taxonomy ID and the scientific name of the organism do not match for novel samples.
+- Collection date and geographic location of the samples are not filled in for novel samples.
+- Date fields do not follow the YYYY-MM-DD format.
+- Custom values are used for controlled vocabulary fields (indicated with a drop-down menu). Submitters should select from the values provided or contact us if the required value is not present.
+
+Most issues around metadata will be reported in the "Metadata validation results" section of the validation report.
+However, note that other validation failures may also require you to modify your metadata file.
+
+## VCF Checks
+
+Ensuring data consistency upon submission is crucial for interoperability and supporting cross-study comparative genomics.
+Before accepting a VCF submission, the CLI tool verifies that the submitted information adheres to the official VCF specifications.
+Additionally, submitted variants must be supported by either experimentally determined sample genotypes or population allele frequencies.
+
+Key points to note before validating your VCF file with the eva-sub-cli tool:
+
+- File Format Version: Always start the header with the VCF version number (versions 4.1-4 are accepted).
+- Header Metadata: Should include the reference genome, information fields (INFO), filters (FILTER), AF and  genotype metadata
+- Variant Information: VCF files must provide either sample genotypes and/or aggregated sample summary-level allele frequencies.
+- Unique Variants: Variant lines should be unique and not specify duplicate loci.
+- Reference Genome: All variants must be submitted with positions on a reference genome accessioned by a member of the INSDC consortium: [Genbank](https://www.ncbi.nlm.nih.gov/genbank/), [ENA](https://www.ebi.ac.uk/ena/browser/home), or [DDBJ](https://www.ddbj.nig.ac.jp/index-e.html).
+
+Common errors seen with VCF checks:
+
+- The VCF version is not one of 4.1, 4.2, 4.3, or 4.4.
+- The VCF file contains extra spaces, blanks, or extra quotations causing validation to fail. Tools like bcftools can help verify the header before validating the file.
+- GT and AF fields are not defined in the header section.
+- The fields used do not conform to the official VCF specifications.
+
+Issues in VCF format validation will be reported in the "VCF validation results" section of the validation report.
+Issues in determining evidence type (genotypes or allele frequencies) are reported per analysis at the top of the report.
+
+## Assembly Checks
+
+The EVA requires that all variants be submitted with an asserted position on an INSDC sequence.
+This means that the reference allele for every variant must match a position in a sequence that has been accessioned in the GenBank, ENA, or DDBJ databases.
+Aligning all submitted data with INSDC sequences enables integration with other EMBL-EBI resources, including Ensembl, and is crucial for maintaining standardisation at the EVA.
+This also means that all sequence identifiers in your VCF must match those in the reference FASTA file.
+
+Key points to note before validating your data with the eva-sub-cli tool:
+
+- Ensure that the reference sequences in the FASTA file used to call the variants are accessioned in INSDC.
+- Verify that the contig names in the VCF file match those in the FASTA file.
+
+ Common errors seen with assembly checks:
+ 
+- VCF file uses contig name not found in the FASTA file, causing the assembly check to fail.
+- Major Allele Used as REF Allele: This typically occurs when a specific version of Plink or Tassel is used to create VCF files, causing the tool to use the major allele as the reference allele. In such cases, submitters should use the GCA FASTA sequence to create corrected files.
+
+Issues around reference allele matching in VCF files will be reported in the "VCF validation results" section of the validation report,
+while issues around INSDC accessioning of the assembly will be reported in the "Reference genome INSDC check" section.
+
+## Sample Name Concordance Check
+
+The sample name concordance check ensures that the sample names in the metadata spreadsheet match those in the VCF file.
+This is achieved by cross-checking the "Sample name in VCF" column in the spreadsheet with the sample names registered in the VCF file.
+Any discrepancies must be addressed by the submitter when the CLI tool generates a report of the mismatches found.
+
+Key points to note before validating your data with the eva-sub-cli tool:
+
+- Ensure that sample names between the VCF file and the metadata spreadsheet match. This comparison is case-sensitive.
+- Ensure there are no extra spaces in the sample names.
+
+Common errors seen with sample concordance checks:
+
+- Link between "Sample" and "File" provided via the Analysis alias is not correctly defined in the metadata which causes the sample name concordance check to fail.
+- Extra white spaces in the sample names can lead to mismatches.
+- Case sensitivity issues between the sample names in the VCF file and the metadata spreadsheet.
+
+Issues in sample name concordance will be reported in the "Sample name concordance check" section of the validation report.

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.5.0'
-__version_tuple__ = version_tuple = (0, 5, 0)
+__version__ = version = '0.5.2'
+__version_tuple__ = version_tuple = (0, 5, 2)
 
-__commit_id__ = commit_id = 'g1b663fd41'
+__commit_id__ = commit_id = 'g31e734672'

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/etc/eva_schema.json

@@ -106,7 +106,10 @@
             },
             "publications": {
               "type": "array",
-              "items": true
+              "items": {
+                "type": "string",
+                "pattern": "^[^:,]+?:[^:,]+?$"
+              }
             },
             "parentProject": {
               "$ref": "#/definitions/projectAccession"

eva_sub_cli-0.5.2/eva_sub_cli/exceptions/submission_upload_exception.py

@@ -0,0 +1,5 @@
+
+class SubmissionUploadException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/executables/check_fasta_insdc.py

@@ -54,6 +54,11 @@ def _get_containing_assemblies_paged(url):
 
 
 def get_containing_assemblies(md5_digest):
+    """
+    Return the INSDC accessions of assemblies in contig alias containing the sequence with the provided md5 digest.
+    :param md5_digest: md5 digest of the sequence to query
+    :return: set of INSDC accessions
+    """
     # Wrapper method to handle pagination
     url = f'{CONTIG_ALIAS_SERVER}/{md5_digest}'
     return _get_containing_assemblies_paged(url)
@@ -89,7 +94,7 @@ def assess_fasta(input_fasta, analyses, assembly_in_metadata):
             elif len(possible_assemblies) == 0:
                 possible_assemblies = containing_assemblies
             else:
-                possible_assemblies &= containing_assemblies
+                possible_assemblies = possible_assemblies.intersection(containing_assemblies)
     except Exception as e:
         # Server errors from either ENA refget or EVA contig alias will halt the check prematurely.
         # Report the error but do not return from the method, so that incomplete results can be reported
@@ -134,6 +139,10 @@ def get_analyses_and_reference_genome_from_metadata(vcf_files_for_fasta, json_fi
     return all_analyses, assemblies[0]
 
 
+def check_assembly_in_metadata(assembly_in_metadata):
+    return assembly_in_metadata and assembly_in_metadata.upper().startswith('GCA')
+
+
 def main():
     arg_parser = argparse.ArgumentParser(
         description="Calculate each sequence's Refget MD5 digest and compare these against INSDC Refget server.")
@@ -147,6 +156,7 @@ def main():
                             help='Path to the location of the results')
     args = arg_parser.parse_args()
     logging_config.add_stdout_handler()
-    analyses, metadata_insdc = get_analyses_and_reference_genome_from_metadata(args.vcf_files, args.metadata_json)
-    results = assess_fasta(args.input_fasta, analyses, metadata_insdc)
+    analyses, assembly_in_metadata = get_analyses_and_reference_genome_from_metadata(args.vcf_files, args.metadata_json)
+    results = assess_fasta(args.input_fasta, analyses, assembly_in_metadata)
+    results['metadata_assembly_gca'] = check_assembly_in_metadata(assembly_in_metadata)
     write_result_yaml(args.output_yaml, results)

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/executables/cli.py

@@ -1,14 +1,11 @@
 import sys
 
-import eva_sub_cli
-from eva_sub_cli.exceptions.metadata_template_version_exception import MetadataTemplateVersionException, \
-    MetadataTemplateVersionNotFoundException
-from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
-from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
+from requests import HTTPError
+
+from eva_sub_cli.exceptions.submission_upload_exception import SubmissionUploadException
 
 if not sys.warnoptions:
     import warnings
-
     warnings.simplefilter("ignore")
 
 import logging
@@ -16,27 +13,19 @@ import os
 from argparse import ArgumentParser
 from ebi_eva_common_pyutils.logger import logging_config
 
+import eva_sub_cli
 from eva_sub_cli import orchestrator
-from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
+from eva_sub_cli.exceptions.metadata_template_version_exception import MetadataTemplateVersionException, \
+    MetadataTemplateVersionNotFoundException
+from eva_sub_cli.exceptions.submission_status_exception import SubmissionStatusException
+from eva_sub_cli.exceptions.submission_not_found_exception import SubmissionNotFoundException
 from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, DirLock
+from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
+from eva_sub_cli.validators.validator import ALL_VALIDATION_TASKS
 
 
 def validate_command_line_arguments(args, argparser):
     fail = False
-    if (args.vcf_files and not args.reference_fasta) or (not args.vcf_files and args.reference_fasta):
-        print("When using --vcf_files and --reference_fasta, both need to be specified")
-        fail = True
-
-    if args.vcf_files:
-        for vcf_file in args.vcf_files:
-            if not os.path.isfile(vcf_file):
-                print(f"VCF file {vcf_file} is not a file")
-                fail = True
-
-    if args.reference_fasta:
-        if not os.path.isfile(args.reference_fasta):
-            print(f"Fasta file {args.reference_fasta} is not a file")
-            fail = True
 
     if args.metadata_xlsx:
         if not os.path.isfile(args.metadata_xlsx):
@@ -74,14 +63,6 @@ def parse_args(cmd_line_args):
     argparser.add_argument('--version', action='version', version=f'%(prog)s {eva_sub_cli.__version__}')
     argparser.add_argument('--submission_dir', required=True, type=str,
                            help='Path to the directory where all processing is done and submission info is stored')
-    vcf_group = argparser.add_argument_group(
-        'Input VCF and assembly',
-        "Specify the VCF files and associated assembly with the following options. If you used different assemblies "
-        "for different VCF files, then you must include these in the metadata file rather than specifying them here."
-    )
-    vcf_group.add_argument('--vcf_files', nargs='+', help="One or more VCF files to validate")
-    vcf_group.add_argument('--reference_fasta',
-                           help="The FASTA file containing the reference genome from which the variants were derived")
 
     metadata_group = argparser.add_argument_group('Metadata', 'Specify the metadata in a spreadsheet or in a JSON file')
     metadata_group = metadata_group.add_mutually_exclusive_group(required=True)
@@ -93,6 +74,9 @@ def parse_args(cmd_line_args):
                            help='Select a task to perform (default SUBMIT). VALIDATE will run the validation'
                                 ' regardless of the outcome of previous runs. SUBMIT will run validate only if'
                                 ' the validation was not performed successfully before and then run the submission.')
+    argparser.add_argument('--validation_tasks', nargs='+', choices=ALL_VALIDATION_TASKS, default=ALL_VALIDATION_TASKS,
+                           type=str.lower, help='Select only a subset of the validation tasks to run. Note that all '
+                                                'tasks need to be successful for the validation to pass')
     argparser.add_argument('--executor', choices=[DOCKER, NATIVE], default=NATIVE, type=str.lower,
                            help='Select the execution type for running validation (default native)')
     credential_group = argparser.add_argument_group('Credentials', 'Specify the ENA Webin credentials you want to use '
@@ -125,7 +109,6 @@ def main():
 
     try:
         # lock the submission directory
-
         with DirLock(os.path.join(args.submission_dir)) as lock:
             # Create the log file
             logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
@@ -151,7 +134,15 @@ def main():
     except MetadataTemplateVersionNotFoundException as mte:
         print(mte)
         exit_status = 70
+    except SubmissionUploadException as sue:
+        print(sue)
+        exit_status = 71
+    except HTTPError as http_err:
+        print(http_err)
+        if http_err.response is not None and http_err.response.text:
+            print(http_err.response.text)
+        exit_status = 72
     except Exception as ex:
         print(ex)
-        exit_status = 71
+        exit_status = 73
     return exit_status

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/jinja_templates/html/fasta_check.html

@@ -9,6 +9,12 @@
             <br><strong>Error message:</strong> {{ results_for_fasta.get('connection_error') }}
         </div>
     {% endif %}
+	{% if 'metadata_assembly_compatible' in results_for_fasta and not results_for_fasta.get('metadata_assembly_gca') %}
+        <div class="report-section fail">
+            <strong>Warning:</strong> Non-GCA reference found in metadata. Please provide the INSDC accession for your reference assembly.
+            If you would like to submit using a non-GCA reference sequence, contact <a href="mailto:eva-helpdesk@ebi.ac.uk">eva-helpdesk@ebi.ac.uk</a>.
+        </div>
+	{% endif %}
 
     <!-- All INSDC check results -->
     {% set expand_icon = "" %}
@@ -52,8 +58,8 @@
                 {% set text = analysis_text + ": Assembly accession in metadata is compatible" %}
             {% else %}
                 {% set expand_icon = "&#9654;" %}
-                {% set icon = "&#10060;" %}
-                {% set row_class = "report-section fail collapsible" %}
+                {% set icon = "&#10071;" %}
+                {% set row_class = "report-section warn collapsible" %}
                 {% set text = analysis_text + ": Assembly accession in metadata is not compatible" %}
             {% endif %}
         {% else %}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/jinja_templates/html/file_validation.html

@@ -1,6 +1,6 @@
 
 {% macro file_validation_report(validation_results, file_name) -%}
-    {% for check_type, check_per_file in validation_results.items() %}
+    {% for check_type, check_per_file in validation_results.items() if check_type not in ["trim_down", "version"] %}
         {% set result = check_per_file.get(file_name, {}) %}
         {% if check_type == "assembly_check" %}
             {{ assembly_check(result) }}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/jinja_templates/html/report.html

@@ -6,6 +6,13 @@
 {% from 'metadata_validation.html' import metadata_validation_report %}
 {% from 'shallow_validation.html' import optional_shallow_validation_report %}
 
+{% macro validation_not_run_yet_message(text="Process not run yet") %}
+    {% set icon = "⏸" %}   {# pause symbol ⏸ #}
+    {% set row_class = "report-section info" %}
+    {% set expand_icon = "" %}
+    <div class='{{ row_class }}'><span class="expand_icon">{{ expand_icon }}</span> {{ icon }} {{ text }}</div>
+{% endmacro %}
+
 <html lang="EN">
 <head>
     <meta charset="UTF-8">
@@ -24,12 +31,14 @@
 
         .report-section { padding: 12px; border: 1px solid white; margin-bottom: 5px}
         .active, .collapsible:hover.fail { background-color: #e6a3ae; }
+        .active, .collapsible:hover.warn { background-color: #e6de7c; }
         .active, .collapsible:hover.info { background-color: #c0c0c0; }
 
         table { border-collapse: collapse; padding: 10px; margin-bottom: 20px}
         th, td { border: 1px solid black;  padding: 8px;  text-align: left; }
         th {  background-color: lightgrey;  }
         .fail { background-color: #FFB6C1; }
+        .warn { background-color: #f5ed7f; }
         .pass { background-color: #90EE90; }
         .info { background-color: #dadada; }
         .error-list, .no-show { display: none; }
@@ -63,7 +72,12 @@
         Ensures that required fields are present and values are formatted correctly.
         For requirements, please refer to the <a href="https://www.ebi.ac.uk/eva/?Submit-Data" target=”_blank”>EVA website</a>.
     </div>
-    {{ metadata_validation_report(validation_results) }}
+    {% set run_status = validation_results.get('metadata_check', {}).get('run_status', '') %}
+    {% if run_status %}
+        {{ metadata_validation_report(validation_results) }}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+    {% endif %}
 </section>
 
 <section>
@@ -72,12 +86,17 @@
         Checks whether each file is compliant with the <a href="http://samtools.github.io/hts-specs/VCFv4.4.pdf" target=”_blank”>VCF specification</a>.
         Also checks whether the variants' reference alleles match against the reference assembly.
     </div>
-    {% for file_name in vcf_files %}
-        {% if file_name != "pass"%}
-        <h3>{{ file_name }}</h3>
-        {{ file_validation_report(validation_results, file_name) }}
-        {% endif %}
-    {% endfor %}
+    {% set run_status = validation_results.get('vcf_check', {}).get('run_status', '') %}
+    {% if run_status %}
+        {% for file_name in vcf_files %}
+            {% if file_name != "pass"%}
+            <h3>{{ file_name }}</h3>
+            {{ file_validation_report(validation_results, file_name) }}
+            {% endif %}
+        {% endfor %}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+    {% endif %}
 </section>
 
 <section>
@@ -85,7 +104,13 @@
     <div class="description">
         Checks whether information in the metadata is concordant with that contained in the VCF files, in particular sample names.
     </div>
-    {{ sample_name_check_report(validation_results)}}
+    {% set run_status = validation_results.get('sample_check', {}).get('run_status', '') %}
+    {% if run_status %}
+        {{ sample_name_check_report(validation_results)}}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+
+    {% endif %}
 </section>
 
 <section>
@@ -94,12 +119,17 @@
         Checks that the reference sequences in the FASTA file used to call the variants are accessioned in INSDC.
         Also checks if the reference assembly accession in the metadata matches the one determined from the FASTA file.
     </div>
-    {% for file_name in fasta_files %}
-        {% if file_name != "pass"%}
-        <h3>{{ file_name }}</h3>
-        {{ fasta_check_report(validation_results, file_name) }}
-        {% endif %}
-    {% endfor %}
+    {% set run_status = validation_results.get('fasta_check', {}).get('run_status', '') %}
+    {% if run_status %}
+       {% for file_name in fasta_files %}
+            {% if file_name != "pass"%}
+            <h3>{{ file_name }}</h3>
+            {{ fasta_check_report(validation_results, file_name) }}
+            {% endif %}
+        {% endfor %}
+    {% else %}
+        {{ validation_not_run_yet_message() }}
+    {% endif %}
 </section>
 
 <script>

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/jinja_templates/html/shallow_validation.html

@@ -2,7 +2,7 @@
 {% macro optional_shallow_validation_report(validation_results) -%}
     {% set results = validation_results.get('shallow_validation', {}) %}
 
-    {% if results.get('required') %}
+    {% if validation_results.get('trim_down') %}
     <section>
         <div class="report-section fail collapsible"> <span class="expand_icon">&#9654;</span>
             &#10060; <b>You requested to run the shallow validation, please run full validation before submitting the data</b>

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/jinja_templates/text/fasta_check.txt

@@ -7,6 +7,10 @@
 		Please try again later for complete results.
 		Error message: {{ results_for_fasta.get('connection_error') }}
 	{% endif %}
+	{% if 'metadata_assembly_compatible' in results_for_fasta and not results_for_fasta.get('metadata_assembly_gca') %}
+        Warning: Non-GCA reference found in metadata. Please provide the INSDC accession for your reference assembly.
+        If you would like to submit using a non-GCA reference sequence, contact eva-helpdesk@ebi.ac.uk.
+	{% endif %}
 
 	{% if results_for_fasta.get('all_insdc') %}
 		{{ "\u2714" }} All sequences are INSDC accessioned.
@@ -28,7 +32,7 @@
 				{% set icon = "\u2714" %}
 				{% set text = analysis_text + ": Assembly accession in metadata is compatible" %}
 			{% else %}
-				{% set icon = "\u274C" %}
+				{% set icon = "\u2757" %}
 				{% set text = analysis_text + ": Assembly accession in metadata is not compatible" %}
 			{% endif %}
 		{% else %}

{eva_sub_cli-0.5.0 → eva_sub_cli-0.5.2}/eva_sub_cli/jinja_templates/text/file_validation.txt

@@ -1,6 +1,6 @@
 
 {% macro file_validation_report(validation_results, file_name) -%}
-    {% for check_type, check_per_file in validation_results.items() %}
+    {% for check_type, check_per_file in validation_results.items() if check_type not in ["trim_down", "version"] %}
         {% set result = check_per_file.get(file_name, {}) %}
         {% if check_type == "assembly_check" %}
             {{ assembly_check(result) }}