eva-sub-cli 0.6.5__tar.gz → 0.6.7__tar.gz

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.6.5
+Version: 0.6.7
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

eva_sub_cli-0.6.7/docs/common_metadata_errors.md

@@ -0,0 +1,116 @@
+# **Frequently encountered metadata errors** 
+
+This section provides a curated overview of some of the common metadata errors. Each error is illustrated with a real validation output example, followed by a brief description of the error and the expected resolution for EVA submissions.
+
+## 1. Missing mandatory metadata
+
+**Example Validation Error** 
+
+| **Sheet** | **Row** | **Column** | **Description** |
+|---|---:|---|---|
+| **Submitter Details** | **2** | **Center** | Column "Center" is not populated |
+
+**Description**
+
+The Center field is required but has been left empty in the Submitter Details sheet.
+
+**Resolution** 
+
+Populate all mandatory fields indicated in bold. EVA expects these fields to contain a valid value and not be left blank. 
+
+For the Center field, provide the name of the submitting center/institution.
+
+## 2. Invalid metadata values
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+|---|---:|---|---|
+| **Sample** | **3** | **Scientific Name** | Species *Homo sapens* does not match taxonomy 9606 (*Homo sapiens*). |
+
+**Description**
+
+The Scientific Name does not match the species associated with the submitted Taxonomy ID. In the above example, taxonomy ID 9606 corresponds to Homo sapiens and not Homo sapens.
+
+**Resolution** 
+
+Check that the metadata value is valid and follows the expected format or controlled vocabulary. 
+
+## 3. Invalid metadata format 
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+| --- | --- | --- | --- |
+| **Submitter Details** | **2** | **Email Address** | Must match the `email` format. |
+
+**Description**
+
+The email address provided in the Submitter Details sheet does not match the expected email address format.
+
+**Resolution**
+
+Enter a valid email address in the Email Address field, for example, name@example.com.
+
+For similar metadata validation errors, check the value against the format or allowed values expected by EVA and correct the metadata accordingly.
+
+## 4. Project or Sample Not Publicly Accessible
+
+**Example Validation Error** 
+
+```text
+Exception: PRJEBXXXXXX does not exist in ENA or is private
+
+SAMNXXXXXXXX does not exist or is private
+
+```
+
+**Description**
+
+The submitted project or sample accession cannot be found in ENA or is not publicly accessible. This may occur when the accession is incorrect, does not exist, or the corresponding object is still private.
+
+**Resolution**
+
+EVA cannot access or validate private objects. Make sure the project or sample has been released publicly before submitting it to EVA for validation.
+
+## 5. BioSamples issued by NCBI : geographic location/collection date issue 
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+|---|---:|---|---|
+| **Sample** | **4** | **Sample Accession** | Error validating existing sample SAMN47377199: must have required property **'collection date'**. |
+| **Sample** | **4** | **Sample Accession** | Error validating existing sample SAMN47377199: must have required property **'geographic location (country and/or sea)'**. |
+
+**Description**
+
+This error is due to a difference in reporting geographical location and collection date between NCBI BioSamples and ENA BioSamples. NCBI BioSamples uses property names "geo_loc_name" and "collection_date", whereas ENA uses "geographic location (country and/or sea)" and "collection date". EVA validates BioSamples against ENA's definition, which makes reusing NCBI BioSamples in a submission to EVA difficult to automate at the moment.
+
+**Resolution** 
+
+Once you have validated the rest of the data, contact eva-helpdesk@ebi.ac.uk to submit the data, as this will require manual intervention. Both NCBI and ENA are aware of this issue and we are working towards fixing this.
+
+## 6. Either/Or Fields Completed Incorrectly
+
+**Description**
+
+Some metadata fields are highlighted in green to indicate an either/or requirement. Only one of the specified options should be completed. 
+
+For example, in the Project tab, the submitter should complete either the Pre-registered Project section or the New Project section. Both sections should not be completed.
+
+**Resolution** 
+
+When fields are highlighted in green, check the available options and complete only the section that applies to the submission. Leave the other option blank.
+
+
+## 7. Publication Field Not Completed Correctly 
+
+**Description**
+
+The Publication(s) field is optional, but when provided, the publication must be and given in the expected DB format. An example valid entry would be: PubMed:23128226
+
+**Resolution** 
+
+If publications related to the project are available, enter them using the required DB format, for example, PubMed:23128226. Multiple publications should be separated by commas. For example: PubMed:23128226, PubMed:223445.
+
+If there are no relevant publications, the field can be left blank and can be added later by contacting the EVA helpdesk team.

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.6.5'
-__version_tuple__ = version_tuple = (0, 6, 5)
+__version__ = version = '0.6.7'
+__version_tuple__ = version_tuple = (0, 6, 7)
 
-__commit_id__ = commit_id = 'ge5181ff56'
+__commit_id__ = commit_id = 'g1607240b0'

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/auth.py

@@ -8,6 +8,7 @@ from ebi_eva_common_pyutils.logger import AppLogger
 from urllib3.exceptions import ResponseError
 
 from eva_sub_cli import LSRI_CLIENT_ID, ENA_WEBIN_ACCOUNT_VAR, ENA_WEBIN_PASSWORD_VAR
+from eva_sub_cli.exceptions import WebinBadCredentialsException
 
 ENA_AUTH_URL = "https://www.ebi.ac.uk/ena/submit/webin/auth/token"
 LSRI_AUTH_URL = "https://www.ebi.ac.uk/eva/v1/submission/auth/lsri"
@@ -69,8 +70,11 @@ class WebinAuth(AppLogger):
         if response.status_code == 200:
             self.info("Webin authentication successful!")
             return response.text
+        elif response.status_code == 401:
+            raise WebinBadCredentialsException('Could not authenticate with ENA Webin. Please check your credentials '
+                                               'and try again.')
         else:
-            raise ResponseError('Webin Authentication Error')
+            raise ResponseError(f'Could not authenticate with ENA Webin: {response.status_code}, {response.text}')
 
     def _get_webin_username_password(self):
         username = self.cmd_line_username

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/etc/eva_schema.json

@@ -64,84 +64,90 @@
       }
     },
     "project": {
-      "oneOf": [
-        {
-          "type": "object",
-          "required": [
-            "projectAccession"
-          ],
-          "properties": {
-            "projectAccession": {
-              "$ref": "#/definitions/projectAccession"
-            }
+      "type": "object",
+      "if": {
+        "required": [
+          "projectAccession"
+        ]
+      },
+      "then": {
+        "properties": {
+          "projectAccession": {
+            "$ref": "#/definitions/projectAccession"
           }
-        },
-        {
-          "type": "object",
-          "description": "General information about the Project",
-          "required": [
-            "title",
-            "description",
-            "taxId",
-            "centre"
-          ],
-          "properties": {
-            "title": {
-              "type": "string",
-              "description": "Title of the project",
-              "maxLength": 500
-            },
-            "description": {
-              "type": "string",
-              "description": "Description of the project",
-              "maxLength": 5000
-            },
-            "centre": {
-              "type": "string",
-              "description": "Centre, institute or company submitting the project"
-            },
-            "taxId": {
-              "type": "number",
-              "description": "Taxonomy ID of the species associated with the project (found here: http://www.ncbi.nlm.nih.gov/taxonomy)"
-            },
-            "publications": {
-              "type": "array",
-              "items": {
-                "type": "string",
-                "pattern": "^[^:,]+?:[^:,]+?$"
-              }
-            },
-            "parentProject": {
-              "$ref": "#/definitions/projectAccession"
-            },
-            "childProjects": {
-              "type": "array",
-              "items": {
-                "$ref": "#/definitions/projectAccession"
-              }
-            },
-            "peerProjects": {
-              "type": "array",
-              "items": {
-                "$ref": "#/definitions/projectAccession"
-              }
-            },
-            "links": {
-              "type": "array",
-              "items": {
-                "type": "string"
-              }
-            },
-            "holdDate": {
+        }
+      },
+      "else": {
+        "description": "General information about the Project",
+        "required": [
+          "title",
+          "description",
+          "taxId",
+          "centre"
+        ],
+        "properties": {
+          "title": {
+            "type": "string",
+            "description": "Title of the project",
+            "minLength": 20,
+            "maxLength": 250
+          },
+          "description": {
+            "type": "string",
+            "description": "Description of the project",
+            "minLength": 20,
+            "maxLength": 4000
+          },
+          "centre": {
+            "type": "string",
+            "description": "Centre, institute or company submitting the project"
+          },
+          "taxId": {
+            "type": "number",
+            "description": "Taxonomy ID of the species associated with the project (found here: http://www.ncbi.nlm.nih.gov/taxonomy)"
+          },
+          "publications": {
+            "type": "array",
+            "description": "Publications associated with the project, each given as <database>:<identifier>",
+            "items": {
               "type": "string",
-              "format": "date",
+              "pattern": "^[^:,]+?:[^:,]+?$",
               "examples": [
-                "2018-11-13"
+                "DOI:10.1093/nar/gkw1121",
+                "PMID:24565421"
               ]
             }
+          },
+          "parentProject": {
+            "$ref": "#/definitions/projectAccession"
+          },
+          "childProjects": {
+            "type": "array",
+            "items": {
+              "$ref": "#/definitions/projectAccession"
+            }
+          },
+          "peerProjects": {
+            "type": "array",
+            "items": {
+              "$ref": "#/definitions/projectAccession"
+            }
+          },
+          "links": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "holdDate": {
+            "type": "string",
+            "format": "date",
+            "examples": [
+              "2018-11-13"
+            ]
           }
         }
-      ]
+      }
     },
     "analysis": {
       "type": "array",
@@ -307,21 +313,24 @@
       "minItems": 1,
       "items": {
         "type": "object",
-        "oneOf": [
-          {
-            "required": [
-              "analysisAlias",
-              "sampleInVCF",
-              "bioSampleAccession"
-            ]
-          },
-          {
-            "required": [
-              "analysisAlias",
-              "bioSampleObject"
-            ]
-          }
-        ],
+        "if": {
+          "required": [
+            "bioSampleAccession"
+          ]
+        },
+        "then": {
+          "required": [
+            "analysisAlias",
+            "sampleInVCF",
+            "bioSampleAccession"
+          ]
+        },
+        "else": {
+          "required": [
+            "analysisAlias",
+            "bioSampleObject"
+          ]
+        },
         "properties": {
           "analysisAlias": {
             "type": "array",

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/exceptions.py

@@ -38,3 +38,39 @@ class SubmissionUploadException(Exception):
     def __init__(self, message):
         self.message = message
         super().__init__(self.message)
+
+
+class DirLockError(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class UserFileNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class DependencyNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class WebinBadCredentialsException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class InvalidSubmissionException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class DockerValidatorException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/executables/cli.py

@@ -15,7 +15,9 @@ import eva_sub_cli
 from eva_sub_cli import ENA_WEBIN_ACCOUNT_VAR, ENA_WEBIN_PASSWORD_VAR
 from eva_sub_cli import orchestrator
 from eva_sub_cli.exceptions import MetadataTemplateVersionException, MetadataTemplateVersionNotFoundException, \
-    SubmissionStatusException, SubmissionNotFoundException, SubmissionUploadException, NoVcfsFoundException
+    SubmissionStatusException, SubmissionNotFoundException, SubmissionUploadException, NoVcfsFoundException, \
+    UserFileNotFoundException, DependencyNotFoundException, WebinBadCredentialsException, InvalidFileTypeError, \
+    InvalidSubmissionException, DockerValidatorException
 from eva_sub_cli.call_home import CallHomeClient
 from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, DirLock
 from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
@@ -126,51 +128,69 @@ def main():
             logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
             # Pass on all the arguments to the orchestrator
             orchestrator.orchestrate_process(call_home=call_home, **args.__dict__)
+
+    # User errors: not displayed as exceptions, call-home records as failure with no exception
     except DirLockError as dle:
-        logger.exception(f'Could not acquire the lock file for {args.submission_dir} because another process is using '
-                         f'this directory or a previous process did not terminate correctly. '
-                         f'If the problem persists, remove the lock file manually.')
-        caught_exception = dle
-        exit_status = 65
+        print(f'Could not acquire the lock file for {args.submission_dir} because another process is using this '
+              f'directory or a previous process did not terminate correctly. If the problem persists, remove the lock '
+              f'file manually.')
+        exit_status = 101
+    except NoVcfsFoundException as nvfe:
+        print(nvfe.message)
+        exit_status = 102
+    except MetadataTemplateVersionException as mte:
+        print(mte.message)
+        exit_status = 103
+    except MetadataTemplateVersionNotFoundException as mte:
+        print(mte.message)
+        exit_status = 104
+    except UserFileNotFoundException as ufne:
+        print(ufne.message)
+        exit_status = 105
+    except InvalidFileTypeError as ife:
+        print(ife.message)
+        exit_status = 106
+    except DependencyNotFoundException as dnfe:
+        print(dnfe.message)
+        exit_status = 107
+    except WebinBadCredentialsException as wae:
+        print(wae.message)
+        exit_status = 108
+    except InvalidSubmissionException as ise:
+        print(ise.message)
+        exit_status = 109
+
+    # Process errors: displayed as exceptions and sent to call-home
     except FileNotFoundError as fne:
         logger.exception(fne)
         caught_exception = fne
-        exit_status = 66
-    except NoVcfsFoundException as nvfe:
-        logger.exception(f'{nvfe}. Please ensure that you have listed VCF files in the metadata (.vcf or .vcf.gz file '
-                         f'extension), and that you have not modified the metadata template in any way.')
-        caught_exception = nvfe
-        exit_status = 67
+        exit_status = 201
     except SubmissionNotFoundException as snfe:
-        logger.exception(f'{snfe}. Please contact EVA Helpdesk')
+        logger.exception(snfe)
         caught_exception = snfe
-        exit_status = 68
+        exit_status = 202
     except SubmissionStatusException as sse:
-        logger.exception(f'{sse}. Please try again later. If the problem persists, please contact EVA Helpdesk')
+        logger.exception(sse)
         caught_exception = sse
-        exit_status = 69
-    except MetadataTemplateVersionException as mte:
-        logger.exception(mte)
-        caught_exception = mte
-        exit_status = 70
-    except MetadataTemplateVersionNotFoundException as mte:
-        logger.exception(mte)
-        caught_exception = mte
-        exit_status = 71
+        exit_status = 203
     except SubmissionUploadException as sue:
         logger.exception(sue)
         caught_exception = sue
-        exit_status = 72
+        exit_status = 204
     except HTTPError as http_err:
         logger.exception(http_err)
         if http_err.response is not None and http_err.response.text:
             print(http_err.response.text)
         caught_exception = http_err
-        exit_status = 73
+        exit_status = 205
+    except DockerValidatorException as dve:
+        logger.exception(dve)
+        caught_exception = dve
+        exit_status = 206
     except Exception as ex:
         logger.exception(ex)
         caught_exception = ex
-        exit_status = 74
+        exit_status = 299
 
     if call_home is not None:
         if exit_status == 0:

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/executables/xlsx2json.py

@@ -51,7 +51,7 @@ class XlsxParser:
         with open(conf_filename, 'r') as conf_file:
             self.xlsx_conf = yaml.safe_load(conf_file)
         try:
-            self.workbook = load_workbook(xlsx_filename, read_only=True)
+            self.workbook = load_workbook(xlsx_filename, read_only=True, data_only=True)
         except Exception as e:
             self.add_error(f'Error loading {xlsx_filename}: {repr(e)}')
             self.file_loaded = False
@@ -165,9 +165,13 @@ class XlsxParser:
                 cell = row[header_index]
                 if cell.value is not None:
                     has_notnull = True
-                data[header] = self.trim_value(self.cast_value(
-                    cell.value, self.xlsx_conf[worksheet].get(CAST_KEY_NAME, {}).get(header)
-                ))
+                try:
+                    value = self.cast_value(
+                        cell.value, self.xlsx_conf[worksheet].get(CAST_KEY_NAME, {}).get(header)
+                    )
+                except Exception as e:
+                    value = cell.value
+                data[header] = self.trim_value(value)
 
             if has_notnull:
                 data['row_num'] = self.row_offset[worksheet]

{eva_sub_cli-0.6.5 → eva_sub_cli-0.6.7}/eva_sub_cli/file_utils.py

@@ -7,6 +7,8 @@ from itertools import groupby
 import pysam
 from ebi_eva_common_pyutils.logger import logging_config
 
+from eva_sub_cli.exceptions import DirLockError
+
 logger = logging_config.get_logger(__name__)
 
 
@@ -158,10 +160,6 @@ def fasta_iter(input_fasta):
             yield (headerStr, seq)
 
 
-class DirLockError(Exception):
-    pass
-
-
 class DirLock(object):
     _SPIN_PERIOD_SECONDS = 0.05
 
@@ -183,7 +181,7 @@ class DirLock(object):
                 break
             except OSError:
                 if (time.time() - start_time) > self._timeout:
-                    raise DirLockError(f"could not create {self._lockfilename} after {self._timeout} seconds")
+                    raise DirLockError(f"Could not create {self._lockfilename} after {self._timeout} seconds")
                 else:
                     time.sleep(self._SPIN_PERIOD_SECONDS)
 

eva_sub_cli-0.6.7/eva_sub_cli/jinja_templates/html/file_validation.html

@@ -0,0 +1,89 @@
+
+
+{% macro vcf_check_report(vcf_check_result_per_file, vcf_file_names) -%}
+    {% for file_name in vcf_file_names %}
+        {% set vcf_check_result = vcf_check_result_per_file.get(file_name, {}) %}
+        {% set critical_count = vcf_check_result.get("critical_count", 0) %}
+        {% set error_count = vcf_check_result.get("error_count", 0) %}
+        {% set expand_icon = "" %}
+        {% if critical_count > 0 %}
+            {% set expand_icon = "▶" %}
+            {% set icon = "❌" %}
+            {% set row_class = "report-section fail collapsible" %}
+        {% elif error_count > 0 %}
+            {% set expand_icon = "▶" %}
+            {% set icon = "❗" %}
+            {% set row_class = "report-section warn collapsible" %}
+        {% else %}
+            {% set icon = "✔" %}
+            {% set row_class = "report-section pass" %}
+        {% endif %}
+        <div class='{{ row_class }}'><span class="expand_icon">{{ expand_icon }}</span> {{ icon }} {{ file_name }}: {{ critical_count }} critical errors, {{ error_count }} non-critical errors </div>
+        {% set critical_list = vcf_check_result.get("critical_list") %}
+        {% set error_list = vcf_check_result.get("error_list") %}
+
+        {% if critical_list or error_list%}
+            <div class="error-list">
+                <div class="error-description">First 10 errors per category are below. <strong>Full report:</strong> {{ vcf_check_result.get('report_path', '') }}</div>
+                <table>
+                    <tr>
+                        <th>Category</th><th>Error</th>
+                    </tr>
+                    {% for error in critical_list[:10] %}
+                        <tr>
+                            <td><strong>critical error</strong></td><td> {{ error }}</td>
+                        </tr>
+                    {% endfor %}
+                    {% for error in error_list[:10] %}
+                        <tr>
+                            <td><strong>non-critical error</strong></td><td> {{ error }}</td>
+                        </tr>
+                    {% endfor %}
+                </table>
+            </div>
+        {% endif %}
+    {% endfor %}
+{%- endmacro %}
+
+{% macro assembly_check_report(assembly_check_result_per_file, vcf_file_names) -%}
+    {% for file_name in vcf_file_names %}
+        {% set assembly_check_result = assembly_check_result_per_file.get(file_name, {}) %}
+        {% set nb_match = assembly_check_result.get("match", 0) %}
+        {% set nb_total = assembly_check_result.get("total", 0) %}
+        {% set match_percentage = nb_match / nb_total * 100 if nb_total else 0 %}
+        {% set expand_icon = "" %}
+        {% if assembly_check_result.get("nb_mismatch", 0) > 0 or nb_total == 0 %}
+            {% set expand_icon = "&#9654;" %}
+            {% set icon = "&#10060;" %}
+            {% set row_class = "report-section fail collapsible" %}
+        {% else %}
+            {% set icon = "&#10004;" %}
+            {% set row_class = "report-section pass" %}
+        {% endif %}
+        <div class='{{ row_class }}'><span class="expand_icon">{{ expand_icon }}</span> {{ icon }} {{ file_name }}: {{ nb_match }}/{{ nb_total }} ({{ match_percentage|round(2) }}%)</div>
+        {% set mismatch_list = assembly_check_result.get("mismatch_list") %}
+        {% set error_list = assembly_check_result.get("error_list") %}
+        {% if mismatch_list or error_list %}
+            <div class="error-list">
+            <div class="error-description">First 10 errors per category are below. <strong>Full report:</strong> {{ assembly_check_result.get('report_path', '') }}</div>
+            <table>
+                <tr>
+                    <th>Category</th><th>Error</th>
+                </tr>
+                {% for error in error_list[:10] %}
+                    <tr>
+                        <td><strong>Parsing Error</strong></td><td> {{ error }}</td>
+                    </tr>
+                {% endfor %}
+                {% for error in mismatch_list[:10] %}
+                    <tr>
+                        <td><strong>mismatch error</strong></td><td> {{ error }}</td>
+                    </tr>
+                {% endfor %}
+            </table>
+            </div>
+        {% endif %}
+    {% endfor %}
+{%- endmacro %}
+
+