eva-sub-cli 0.6.6__tar.gz → 0.6.7__tar.gz

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.6.6
+Version: 0.6.7
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

eva_sub_cli-0.6.7/docs/common_metadata_errors.md

@@ -0,0 +1,116 @@
+# **Frequently encountered metadata errors** 
+
+This section provides a curated overview of some of the common metadata errors. Each error is illustrated with a real validation output example, followed by a brief description of the error and the expected resolution for EVA submissions.
+
+## 1. Missing mandatory metadata
+
+**Example Validation Error** 
+
+| **Sheet** | **Row** | **Column** | **Description** |
+|---|---:|---|---|
+| **Submitter Details** | **2** | **Center** | Column "Center" is not populated |
+
+**Description**
+
+The Center field is required but has been left empty in the Submitter Details sheet.
+
+**Resolution** 
+
+Populate all mandatory fields indicated in bold. EVA expects these fields to contain a valid value and not be left blank. 
+
+For the Center field, provide the name of the submitting center/institution.
+
+## 2. Invalid metadata values
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+|---|---:|---|---|
+| **Sample** | **3** | **Scientific Name** | Species *Homo sapens* does not match taxonomy 9606 (*Homo sapiens*). |
+
+**Description**
+
+The Scientific Name does not match the species associated with the submitted Taxonomy ID. In the above example, taxonomy ID 9606 corresponds to Homo sapiens and not Homo sapens.
+
+**Resolution** 
+
+Check that the metadata value is valid and follows the expected format or controlled vocabulary. 
+
+## 3. Invalid metadata format 
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+| --- | --- | --- | --- |
+| **Submitter Details** | **2** | **Email Address** | Must match the `email` format. |
+
+**Description**
+
+The email address provided in the Submitter Details sheet does not match the expected email address format.
+
+**Resolution**
+
+Enter a valid email address in the Email Address field, for example, name@example.com.
+
+For similar metadata validation errors, check the value against the format or allowed values expected by EVA and correct the metadata accordingly.
+
+## 4. Project or Sample Not Publicly Accessible
+
+**Example Validation Error** 
+
+```text
+Exception: PRJEBXXXXXX does not exist in ENA or is private
+
+SAMNXXXXXXXX does not exist or is private
+
+```
+
+**Description**
+
+The submitted project or sample accession cannot be found in ENA or is not publicly accessible. This may occur when the accession is incorrect, does not exist, or the corresponding object is still private.
+
+**Resolution**
+
+EVA cannot access or validate private objects. Make sure the project or sample has been released publicly before submitting it to EVA for validation.
+
+## 5. BioSamples issued by NCBI : geographic location/collection date issue 
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+|---|---:|---|---|
+| **Sample** | **4** | **Sample Accession** | Error validating existing sample SAMN47377199: must have required property **'collection date'**. |
+| **Sample** | **4** | **Sample Accession** | Error validating existing sample SAMN47377199: must have required property **'geographic location (country and/or sea)'**. |
+
+**Description**
+
+This error is due to a difference in reporting geographical location and collection date between NCBI BioSamples and ENA BioSamples. NCBI BioSamples uses property names "geo_loc_name" and "collection_date", whereas ENA uses "geographic location (country and/or sea)" and "collection date". EVA validates BioSamples against ENA's definition, which makes reusing NCBI BioSamples in a submission to EVA difficult to automate at the moment.
+
+**Resolution** 
+
+Once you have validated the rest of the data, contact eva-helpdesk@ebi.ac.uk to submit the data, as this will require manual intervention. Both NCBI and ENA are aware of this issue and we are working towards fixing this.
+
+## 6. Either/Or Fields Completed Incorrectly
+
+**Description**
+
+Some metadata fields are highlighted in green to indicate an either/or requirement. Only one of the specified options should be completed. 
+
+For example, in the Project tab, the submitter should complete either the Pre-registered Project section or the New Project section. Both sections should not be completed.
+
+**Resolution** 
+
+When fields are highlighted in green, check the available options and complete only the section that applies to the submission. Leave the other option blank.
+
+
+## 7. Publication Field Not Completed Correctly 
+
+**Description**
+
+The Publication(s) field is optional, but when provided, the publication must be and given in the expected DB format. An example valid entry would be: PubMed:23128226
+
+**Resolution** 
+
+If publications related to the project are available, enter them using the required DB format, for example, PubMed:23128226. Multiple publications should be separated by commas. For example: PubMed:23128226, PubMed:223445.
+
+If there are no relevant publications, the field can be left blank and can be added later by contacting the EVA helpdesk team.

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.6.6'
-__version_tuple__ = version_tuple = (0, 6, 6)
+__version__ = version = '0.6.7'
+__version_tuple__ = version_tuple = (0, 6, 7)
 
-__commit_id__ = commit_id = 'gdb6448e77'
+__commit_id__ = commit_id = 'g1607240b0'

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/auth.py

@@ -8,6 +8,7 @@ from ebi_eva_common_pyutils.logger import AppLogger
 from urllib3.exceptions import ResponseError
 
 from eva_sub_cli import LSRI_CLIENT_ID, ENA_WEBIN_ACCOUNT_VAR, ENA_WEBIN_PASSWORD_VAR
+from eva_sub_cli.exceptions import WebinBadCredentialsException
 
 ENA_AUTH_URL = "https://www.ebi.ac.uk/ena/submit/webin/auth/token"
 LSRI_AUTH_URL = "https://www.ebi.ac.uk/eva/v1/submission/auth/lsri"
@@ -69,8 +70,11 @@ class WebinAuth(AppLogger):
         if response.status_code == 200:
             self.info("Webin authentication successful!")
             return response.text
+        elif response.status_code == 401:
+            raise WebinBadCredentialsException('Could not authenticate with ENA Webin. Please check your credentials '
+                                               'and try again.')
         else:
-            raise ResponseError('Webin Authentication Error')
+            raise ResponseError(f'Could not authenticate with ENA Webin: {response.status_code}, {response.text}')
 
     def _get_webin_username_password(self):
         username = self.cmd_line_username

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/etc/eva_schema.json

@@ -64,86 +64,90 @@
       }
     },
     "project": {
-      "oneOf": [
-        {
-          "type": "object",
-          "required": [
-            "projectAccession"
-          ],
-          "properties": {
-            "projectAccession": {
-              "$ref": "#/definitions/projectAccession"
-            }
+      "type": "object",
+      "if": {
+        "required": [
+          "projectAccession"
+        ]
+      },
+      "then": {
+        "properties": {
+          "projectAccession": {
+            "$ref": "#/definitions/projectAccession"
           }
-        },
-        {
-          "type": "object",
-          "description": "General information about the Project",
-          "required": [
-            "title",
-            "description",
-            "taxId",
-            "centre"
-          ],
-          "properties": {
-            "title": {
-              "type": "string",
-              "description": "Title of the project",
-              "minLength": 20,
-              "maxLength": 250
-            },
-            "description": {
-              "type": "string",
-              "description": "Description of the project",
-              "minLength": 20,
-              "maxLength": 4000
-            },
-            "centre": {
-              "type": "string",
-              "description": "Centre, institute or company submitting the project"
-            },
-            "taxId": {
-              "type": "number",
-              "description": "Taxonomy ID of the species associated with the project (found here: http://www.ncbi.nlm.nih.gov/taxonomy)"
-            },
-            "publications": {
-              "type": "array",
-              "items": {
-                "type": "string",
-                "pattern": "^[^:,]+?:[^:,]+?$"
-              }
-            },
-            "parentProject": {
-              "$ref": "#/definitions/projectAccession"
-            },
-            "childProjects": {
-              "type": "array",
-              "items": {
-                "$ref": "#/definitions/projectAccession"
-              }
-            },
-            "peerProjects": {
-              "type": "array",
-              "items": {
-                "$ref": "#/definitions/projectAccession"
-              }
-            },
-            "links": {
-              "type": "array",
-              "items": {
-                "type": "string"
-              }
-            },
-            "holdDate": {
+        }
+      },
+      "else": {
+        "description": "General information about the Project",
+        "required": [
+          "title",
+          "description",
+          "taxId",
+          "centre"
+        ],
+        "properties": {
+          "title": {
+            "type": "string",
+            "description": "Title of the project",
+            "minLength": 20,
+            "maxLength": 250
+          },
+          "description": {
+            "type": "string",
+            "description": "Description of the project",
+            "minLength": 20,
+            "maxLength": 4000
+          },
+          "centre": {
+            "type": "string",
+            "description": "Centre, institute or company submitting the project"
+          },
+          "taxId": {
+            "type": "number",
+            "description": "Taxonomy ID of the species associated with the project (found here: http://www.ncbi.nlm.nih.gov/taxonomy)"
+          },
+          "publications": {
+            "type": "array",
+            "description": "Publications associated with the project, each given as <database>:<identifier>",
+            "items": {
               "type": "string",
-              "format": "date",
+              "pattern": "^[^:,]+?:[^:,]+?$",
               "examples": [
-                "2018-11-13"
+                "DOI:10.1093/nar/gkw1121",
+                "PMID:24565421"
               ]
             }
+          },
+          "parentProject": {
+            "$ref": "#/definitions/projectAccession"
+          },
+          "childProjects": {
+            "type": "array",
+            "items": {
+              "$ref": "#/definitions/projectAccession"
+            }
+          },
+          "peerProjects": {
+            "type": "array",
+            "items": {
+              "$ref": "#/definitions/projectAccession"
+            }
+          },
+          "links": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "holdDate": {
+            "type": "string",
+            "format": "date",
+            "examples": [
+              "2018-11-13"
+            ]
           }
         }
-      ]
+      }
     },
     "analysis": {
       "type": "array",
@@ -309,21 +313,24 @@
       "minItems": 1,
       "items": {
         "type": "object",
-        "oneOf": [
-          {
-            "required": [
-              "analysisAlias",
-              "sampleInVCF",
-              "bioSampleAccession"
-            ]
-          },
-          {
-            "required": [
-              "analysisAlias",
-              "bioSampleObject"
-            ]
-          }
-        ],
+        "if": {
+          "required": [
+            "bioSampleAccession"
+          ]
+        },
+        "then": {
+          "required": [
+            "analysisAlias",
+            "sampleInVCF",
+            "bioSampleAccession"
+          ]
+        },
+        "else": {
+          "required": [
+            "analysisAlias",
+            "bioSampleObject"
+          ]
+        },
         "properties": {
           "analysisAlias": {
             "type": "array",

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/exceptions.py

@@ -38,3 +38,39 @@ class SubmissionUploadException(Exception):
     def __init__(self, message):
         self.message = message
         super().__init__(self.message)
+
+
+class DirLockError(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class UserFileNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class DependencyNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class WebinBadCredentialsException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class InvalidSubmissionException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class DockerValidatorException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/executables/cli.py

@@ -15,7 +15,9 @@ import eva_sub_cli
 from eva_sub_cli import ENA_WEBIN_ACCOUNT_VAR, ENA_WEBIN_PASSWORD_VAR
 from eva_sub_cli import orchestrator
 from eva_sub_cli.exceptions import MetadataTemplateVersionException, MetadataTemplateVersionNotFoundException, \
-    SubmissionStatusException, SubmissionNotFoundException, SubmissionUploadException, NoVcfsFoundException
+    SubmissionStatusException, SubmissionNotFoundException, SubmissionUploadException, NoVcfsFoundException, \
+    UserFileNotFoundException, DependencyNotFoundException, WebinBadCredentialsException, InvalidFileTypeError, \
+    InvalidSubmissionException, DockerValidatorException
 from eva_sub_cli.call_home import CallHomeClient
 from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, DirLock
 from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
@@ -126,51 +128,69 @@ def main():
             logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
             # Pass on all the arguments to the orchestrator
             orchestrator.orchestrate_process(call_home=call_home, **args.__dict__)
+
+    # User errors: not displayed as exceptions, call-home records as failure with no exception
     except DirLockError as dle:
-        logger.exception(f'Could not acquire the lock file for {args.submission_dir} because another process is using '
-                         f'this directory or a previous process did not terminate correctly. '
-                         f'If the problem persists, remove the lock file manually.')
-        caught_exception = dle
-        exit_status = 65
+        print(f'Could not acquire the lock file for {args.submission_dir} because another process is using this '
+              f'directory or a previous process did not terminate correctly. If the problem persists, remove the lock '
+              f'file manually.')
+        exit_status = 101
+    except NoVcfsFoundException as nvfe:
+        print(nvfe.message)
+        exit_status = 102
+    except MetadataTemplateVersionException as mte:
+        print(mte.message)
+        exit_status = 103
+    except MetadataTemplateVersionNotFoundException as mte:
+        print(mte.message)
+        exit_status = 104
+    except UserFileNotFoundException as ufne:
+        print(ufne.message)
+        exit_status = 105
+    except InvalidFileTypeError as ife:
+        print(ife.message)
+        exit_status = 106
+    except DependencyNotFoundException as dnfe:
+        print(dnfe.message)
+        exit_status = 107
+    except WebinBadCredentialsException as wae:
+        print(wae.message)
+        exit_status = 108
+    except InvalidSubmissionException as ise:
+        print(ise.message)
+        exit_status = 109
+
+    # Process errors: displayed as exceptions and sent to call-home
     except FileNotFoundError as fne:
         logger.exception(fne)
         caught_exception = fne
-        exit_status = 66
-    except NoVcfsFoundException as nvfe:
-        logger.exception(f'{nvfe}. Please ensure that you have listed VCF files in the metadata (.vcf or .vcf.gz file '
-                         f'extension), and that you have not modified the metadata template in any way.')
-        caught_exception = nvfe
-        exit_status = 67
+        exit_status = 201
     except SubmissionNotFoundException as snfe:
-        logger.exception(f'{snfe}. Please contact EVA Helpdesk')
+        logger.exception(snfe)
         caught_exception = snfe
-        exit_status = 68
+        exit_status = 202
     except SubmissionStatusException as sse:
-        logger.exception(f'{sse}. Please try again later. If the problem persists, please contact EVA Helpdesk')
+        logger.exception(sse)
         caught_exception = sse
-        exit_status = 69
-    except MetadataTemplateVersionException as mte:
-        logger.exception(mte)
-        caught_exception = mte
-        exit_status = 70
-    except MetadataTemplateVersionNotFoundException as mte:
-        logger.exception(mte)
-        caught_exception = mte
-        exit_status = 71
+        exit_status = 203
     except SubmissionUploadException as sue:
         logger.exception(sue)
         caught_exception = sue
-        exit_status = 72
+        exit_status = 204
     except HTTPError as http_err:
         logger.exception(http_err)
         if http_err.response is not None and http_err.response.text:
             print(http_err.response.text)
         caught_exception = http_err
-        exit_status = 73
+        exit_status = 205
+    except DockerValidatorException as dve:
+        logger.exception(dve)
+        caught_exception = dve
+        exit_status = 206
     except Exception as ex:
         logger.exception(ex)
         caught_exception = ex
-        exit_status = 74
+        exit_status = 299
 
     if call_home is not None:
         if exit_status == 0:

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/executables/xlsx2json.py

@@ -51,7 +51,7 @@ class XlsxParser:
         with open(conf_filename, 'r') as conf_file:
             self.xlsx_conf = yaml.safe_load(conf_file)
         try:
-            self.workbook = load_workbook(xlsx_filename, read_only=True)
+            self.workbook = load_workbook(xlsx_filename, read_only=True, data_only=True)
         except Exception as e:
             self.add_error(f'Error loading {xlsx_filename}: {repr(e)}')
             self.file_loaded = False
@@ -165,9 +165,13 @@ class XlsxParser:
                 cell = row[header_index]
                 if cell.value is not None:
                     has_notnull = True
-                data[header] = self.trim_value(self.cast_value(
-                    cell.value, self.xlsx_conf[worksheet].get(CAST_KEY_NAME, {}).get(header)
-                ))
+                try:
+                    value = self.cast_value(
+                        cell.value, self.xlsx_conf[worksheet].get(CAST_KEY_NAME, {}).get(header)
+                    )
+                except Exception as e:
+                    value = cell.value
+                data[header] = self.trim_value(value)
 
             if has_notnull:
                 data['row_num'] = self.row_offset[worksheet]

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/file_utils.py

@@ -7,6 +7,8 @@ from itertools import groupby
 import pysam
 from ebi_eva_common_pyutils.logger import logging_config
 
+from eva_sub_cli.exceptions import DirLockError
+
 logger = logging_config.get_logger(__name__)
 
 
@@ -158,10 +160,6 @@ def fasta_iter(input_fasta):
             yield (headerStr, seq)
 
 
-class DirLockError(Exception):
-    pass
-
-
 class DirLock(object):
     _SPIN_PERIOD_SECONDS = 0.05
 
@@ -183,7 +181,7 @@ class DirLock(object):
                 break
             except OSError:
                 if (time.time() - start_time) > self._timeout:
-                    raise DirLockError(f"could not create {self._lockfilename} after {self._timeout} seconds")
+                    raise DirLockError(f"Could not create {self._lockfilename} after {self._timeout} seconds")
                 else:
                     time.sleep(self._SPIN_PERIOD_SECONDS)
 

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/metadata.py

@@ -6,6 +6,8 @@ from functools import cached_property
 
 from ebi_eva_common_pyutils.logger import AppLogger
 
+from eva_sub_cli.exceptions import UserFileNotFoundException
+
 
 class EvaMetadataJson(AppLogger):
 
@@ -86,7 +88,7 @@ class EvaMetadataJson(AppLogger):
     def get_analysis_for_vcf_file(self, vcf_file):
         """Returns list of analysis aliases associated with the vcf file path."""
         if not os.path.exists(vcf_file):
-            raise FileNotFoundError(f'{vcf_file} cannot be resolved')
+            raise UserFileNotFoundException(f'{vcf_file} cannot be resolved, please check the file path.')
         analysis_aliases = [analysis_alias for analysis_alias in self.files_per_analysis
                             if vcf_file in self.files_per_analysis[analysis_alias]
                             or os.path.basename(vcf_file) in [

{eva_sub_cli-0.6.6 → eva_sub_cli-0.6.7}/eva_sub_cli/orchestrator.py

@@ -14,7 +14,7 @@ from eva_sub_cli import MINIMUM_METADATA_XLSX_TEMPLATE_VERSION
 from eva_sub_cli import SUB_CLI_CONFIG_FILE, __version__
 from eva_sub_cli.exceptions import InvalidFileTypeError, MetadataTemplateVersionException, \
     MetadataTemplateVersionNotFoundException, SubmissionNotFoundException, SubmissionStatusException, \
-    NoVcfsFoundException
+    NoVcfsFoundException, UserFileNotFoundException
 from eva_sub_cli.file_utils import is_vcf_file
 from eva_sub_cli.metadata import EvaMetadataJson
 from eva_sub_cli.submission_ws import SubmissionWSClient
@@ -111,17 +111,20 @@ def validate_vcf_mapping(vcf_mapping):
     :return:
     """
     if len(vcf_mapping) == 0:
-        raise NoVcfsFoundException('No VCF files detected in metadata')
+        raise NoVcfsFoundException('No VCF files detected in metadata. Please ensure that you have listed at least one '
+                                   'VCF file in the metadata Files section (.vcf or .vcf.gz file extension), and that '
+                                   'you have not modified the metadata template in any way.')
     for vcf_file, fasta_file, report_file in vcf_mapping:
         if not (vcf_file and os.path.isfile(vcf_file)):
-            raise FileNotFoundError(f'The variant file {vcf_file} does not exist, please check the file path.')
+            raise UserFileNotFoundException(f'The variant file {vcf_file} does not exist, please check the file path.')
         if not (fasta_file and os.path.isfile(fasta_file)):
-            raise FileNotFoundError(f'The reference fasta {fasta_file} does not exist, please check the file path.')
+            raise UserFileNotFoundException(f'The reference fasta {fasta_file} does not exist, please check the file '
+                                            f'path.')
         if fasta_file.lower().endswith('gz'):
             raise InvalidFileTypeError(f'The reference fasta {fasta_file} is compressed, please uncompress the file.')
         if report_file and not os.path.isfile(report_file):
-            raise FileNotFoundError(f'The assembly report file {report_file} does not exist, please check the file '
-                                    f'path.')
+            raise UserFileNotFoundException(f'The assembly report file {report_file} does not exist, please check the '
+                                            f'file path.')
 
 
 def get_project_and_vcf_fasta_mapping_from_metadata_json(metadata_json):
@@ -243,12 +246,14 @@ def check_validation_required(tasks, sub_config, username=None, password=None):
                 logger.error(
                     f'Submission with id {submission_id} could not be found: '
                     f'status code: {ex.response.status_code} response: {ex.response.text}')
-                raise SubmissionNotFoundException(f'Submission with id {submission_id} could not be found')
+                raise SubmissionNotFoundException(f'Submission with id {submission_id} could not be found. Please '
+                                                  f'contact EVA Helpdesk.')
             else:
-                logger.error(f'Error occurred while getting status of the submission with Id {submission_id}: '
+                logger.error(f'Error occurred while getting status of the submission with ID {submission_id}: '
                              f'status code: {ex.response.status_code} response: {ex.response.text}')
                 raise SubmissionStatusException(f'Error occurred while getting status of the submission '
-                                                f'with Id {submission_id}')
+                                                f'with ID {submission_id}. Please try again later. If the problem '
+                                                f'persists, please contact EVA Helpdesk.')
 
     logger.debug(f'submission id not found in config. This might be the first time user is submitting')
     return False
@@ -263,7 +268,8 @@ def orchestrate_process(submission_dir, metadata_json, metadata_xlsx,
 
     metadata_file = metadata_json or metadata_xlsx
     if not os.path.exists(os.path.abspath(metadata_file)):
-        raise FileNotFoundError(f'The provided metadata file {os.path.abspath(metadata_file)} does not exist')
+        raise UserFileNotFoundException(f'The provided metadata file {os.path.abspath(metadata_file)} does not exist, '
+                                        f'please check the file path.')
 
     if metadata_json:
         metadata_json = os.path.abspath(metadata_json)