eva-sub-cli 0.6.6__tar.gz → 0.7.0__tar.gz

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/.github/workflows/trigger-eva-sub-cli-integration-tests.yml

@@ -12,7 +12,7 @@ jobs:
     runs-on: ubuntu-latest
     steps:
       - name: Trigger eva sub cli integration tests
-        uses: peter-evans/repository-dispatch@v2
+        uses: peter-evans/repository-dispatch@v4
         with:
           token: ${{ secrets.WORKFLOW_ACCESS_TOKEN }}
           repository: EBIvariation/eva-integration-tests

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.6.6
+Version: 0.7.0
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

eva_sub_cli-0.7.0/docs/common_metadata_errors.md

@@ -0,0 +1,116 @@
+# **Frequently encountered metadata errors** 
+
+This section provides a curated overview of some of the common metadata errors. Each error is illustrated with a real validation output example, followed by a brief description of the error and the expected resolution for EVA submissions.
+
+## 1. Missing mandatory metadata
+
+**Example Validation Error** 
+
+| **Sheet** | **Row** | **Column** | **Description** |
+|---|---:|---|---|
+| **Submitter Details** | **2** | **Center** | Column "Center" is not populated |
+
+**Description**
+
+The Center field is required but has been left empty in the Submitter Details sheet.
+
+**Resolution** 
+
+Populate all mandatory fields indicated in bold. EVA expects these fields to contain a valid value and not be left blank. 
+
+For the Center field, provide the name of the submitting center/institution.
+
+## 2. Invalid metadata values
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+|---|---:|---|---|
+| **Sample** | **3** | **Scientific Name** | Species *Homo sapens* does not match taxonomy 9606 (*Homo sapiens*). |
+
+**Description**
+
+The Scientific Name does not match the species associated with the submitted Taxonomy ID. In the above example, taxonomy ID 9606 corresponds to Homo sapiens and not Homo sapens.
+
+**Resolution** 
+
+Check that the metadata value is valid and follows the expected format or controlled vocabulary. 
+
+## 3. Invalid metadata format 
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+| --- | --- | --- | --- |
+| **Submitter Details** | **2** | **Email Address** | Must match the `email` format. |
+
+**Description**
+
+The email address provided in the Submitter Details sheet does not match the expected email address format.
+
+**Resolution**
+
+Enter a valid email address in the Email Address field, for example, name@example.com.
+
+For similar metadata validation errors, check the value against the format or allowed values expected by EVA and correct the metadata accordingly.
+
+## 4. Project or Sample Not Publicly Accessible
+
+**Example Validation Error** 
+
+```text
+Exception: PRJEBXXXXXX does not exist in ENA or is private
+
+SAMNXXXXXXXX does not exist or is private
+
+```
+
+**Description**
+
+The submitted project or sample accession cannot be found in ENA or is not publicly accessible. This may occur when the accession is incorrect, does not exist, or the corresponding object is still private.
+
+**Resolution**
+
+EVA cannot add to, validate, or access private objects. Make sure the project or sample has been released publicly before submitting it to EVA for validation.
+
+## 5. BioSamples issued by NCBI : geographic location/collection date issue 
+
+**Example Validation Error** 
+
+| Sheet | Row | Column | Description |
+|---|---:|---|---|
+| **Sample** | **4** | **Sample Accession** | Error validating existing sample SAMN47377199: must have required property **'collection date'**. |
+| **Sample** | **4** | **Sample Accession** | Error validating existing sample SAMN47377199: must have required property **'geographic location (country and/or sea)'**. |
+
+**Description**
+
+This error is due to a difference in reporting geographical location and collection date between NCBI BioSamples and ENA BioSamples. NCBI BioSamples uses property names "geo_loc_name" and "collection_date", whereas ENA uses "geographic location (country and/or sea)" and "collection date". EVA validates BioSamples against ENA's definition, which makes reusing NCBI BioSamples in a submission to EVA difficult to automate at the moment.
+
+**Resolution** 
+
+Once you have validated the rest of the data, contact eva-helpdesk@ebi.ac.uk to submit the data, as this will require manual intervention. Both NCBI and ENA are aware of this issue and we are working towards fixing this.
+
+## 6. Either/Or Fields Completed Incorrectly
+
+**Description**
+
+Some metadata fields are highlighted in green to indicate an either/or requirement. Only one of the specified options should be completed. 
+
+For example, in the Project tab, the submitter should complete either the Pre-registered Project section or the New Project section. Both sections should not be completed.
+
+**Resolution** 
+
+When fields are highlighted in green, check the available options and complete only the section that applies to the submission. Leave the other option blank.
+
+
+## 7. Publication Field Not Completed Correctly 
+
+**Description**
+
+The Publication(s) field is optional, but when provided, the publication must be and given in the expected DB format. An example valid entry would be: PubMed:23128226
+
+**Resolution** 
+
+If publications related to the project are available, enter them using the required DB format, for example, PubMed:23128226. Multiple publications should be separated by commas. For example: PubMed:23128226, PubMed:223445.
+
+If there are no relevant publications, the field can be left blank and can be added later by contacting the EVA helpdesk team.

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/docs/how_to_submit.md

@@ -71,7 +71,6 @@ This can be useful if you want to avoid re-running long-running validations that
 Note that all validation tasks must pass in order to submit. The report will aggregate the results of previous runs and new ones.
 The possible tasks are:
 * [`vcf_check`](validation_overview.md#vcf-checks) - includes syntax validation and other checks on VCF files
-${CODE_ROOT}/env/bin/upload_to_gcloud.py --input-file evidence.json.gz --destination-folder pharmacogenomics
 * [`assembly_check`](validation_overview.md#assembly-checks) - includes all checks involving the FASTA file
 * [`metadata_check`](validation_overview.md#metadata-check) - includes syntactic and semantic checks on metadata
 * [`sample_check`](validation_overview.md#sample-name-concordance-check) - includes sample coherence checks between VCF files and metadata

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/docs/input_file_overview.md

@@ -87,6 +87,17 @@ One important column to note is the Hold Date, which is the date until which the
 release the data automatically after this date. If it is missing, the default value is three days after the date of
 submission.
 
+### Statements
+
+This sheet contains statements that the submitter must agree to before submitting data. Please read each
+statement carefully and confirm your agreement before submitting.
+
+Additional information about the Access and Benefit-Sharing statement is available in our
+[submission FAQs](https://www.ebi.ac.uk/eva/?Help#submissionPanel&link=confirm-data-shared-openly).
+
+NOTE: To use your Webin account, you must also log in to your account and do a one time confirmation of the Access and
+Benefit-Sharing acknowledgement there. [Webin login](https://www.ebi.ac.uk/ena/submit/webin/login) => Manage Account => Tick the box => Save.
+
 ### Analysis
 
 For EVA, an analysis is a grouping of samples and data files. This sheet allows us to link VCF files to a project and

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/eva_sub_cli/_version.py

@@ -18,7 +18,7 @@ version_tuple: tuple[int | str, ...]
 commit_id: str | None
 __commit_id__: str | None
 
-__version__ = version = '0.6.6'
-__version_tuple__ = version_tuple = (0, 6, 6)
+__version__ = version = '0.7.0'
+__version_tuple__ = version_tuple = (0, 7, 0)
 
-__commit_id__ = commit_id = 'gdb6448e77'
+__commit_id__ = commit_id = 'g3d76f3cd1'

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/eva_sub_cli/auth.py

@@ -8,6 +8,7 @@ from ebi_eva_common_pyutils.logger import AppLogger
 from urllib3.exceptions import ResponseError
 
 from eva_sub_cli import LSRI_CLIENT_ID, ENA_WEBIN_ACCOUNT_VAR, ENA_WEBIN_PASSWORD_VAR
+from eva_sub_cli.exceptions import WebinBadCredentialsException
 
 ENA_AUTH_URL = "https://www.ebi.ac.uk/ena/submit/webin/auth/token"
 LSRI_AUTH_URL = "https://www.ebi.ac.uk/eva/v1/submission/auth/lsri"
@@ -69,8 +70,11 @@ class WebinAuth(AppLogger):
         if response.status_code == 200:
             self.info("Webin authentication successful!")
             return response.text
+        elif response.status_code == 401:
+            raise WebinBadCredentialsException('Could not authenticate with ENA Webin. Please check your credentials '
+                                               'and try again.')
         else:
-            raise ResponseError('Webin Authentication Error')
+            raise ResponseError(f'Could not authenticate with ENA Webin: {response.status_code}, {response.text}')
 
     def _get_webin_username_password(self):
         username = self.cmd_line_username

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/eva_sub_cli/etc/eva_schema.json

@@ -63,87 +63,110 @@
         }
       }
     },
+    "statements": {
+      "type": "array",
+      "description": "Submitter acknowledgements of various EVA data statements",
+      "items": {
+        "type": "object",
+        "required": ["acknowledgement", "statement"],
+        "additionalProperties": false,
+        "properties": {
+          "acknowledgement": {
+            "type": "boolean",
+            "description": "Whether the submitter confirmed this statement"
+          },
+          "statement": {
+            "type": "string",
+            "description": "Full text of the statement as presented to the submitter"
+          }
+        }
+      }
+    },
     "project": {
-      "oneOf": [
-        {
-          "type": "object",
-          "required": [
-            "projectAccession"
-          ],
-          "properties": {
-            "projectAccession": {
-              "$ref": "#/definitions/projectAccession"
-            }
+      "type": "object",
+      "if": {
+        "required": [
+          "projectAccession"
+        ]
+      },
+      "then": {
+        "properties": {
+          "projectAccession": {
+            "$ref": "#/definitions/projectAccession"
           }
-        },
-        {
-          "type": "object",
-          "description": "General information about the Project",
-          "required": [
-            "title",
-            "description",
-            "taxId",
-            "centre"
-          ],
-          "properties": {
-            "title": {
-              "type": "string",
-              "description": "Title of the project",
-              "minLength": 20,
-              "maxLength": 250
-            },
-            "description": {
-              "type": "string",
-              "description": "Description of the project",
-              "minLength": 20,
-              "maxLength": 4000
-            },
-            "centre": {
-              "type": "string",
-              "description": "Centre, institute or company submitting the project"
-            },
-            "taxId": {
-              "type": "number",
-              "description": "Taxonomy ID of the species associated with the project (found here: http://www.ncbi.nlm.nih.gov/taxonomy)"
-            },
-            "publications": {
-              "type": "array",
-              "items": {
-                "type": "string",
-                "pattern": "^[^:,]+?:[^:,]+?$"
-              }
-            },
-            "parentProject": {
-              "$ref": "#/definitions/projectAccession"
-            },
-            "childProjects": {
-              "type": "array",
-              "items": {
-                "$ref": "#/definitions/projectAccession"
-              }
-            },
-            "peerProjects": {
-              "type": "array",
-              "items": {
-                "$ref": "#/definitions/projectAccession"
-              }
-            },
-            "links": {
-              "type": "array",
-              "items": {
-                "type": "string"
-              }
-            },
-            "holdDate": {
+        }
+      },
+      "else": {
+        "description": "General information about the Project",
+        "required": [
+          "title",
+          "description",
+          "taxId",
+          "centre"
+        ],
+        "properties": {
+          "title": {
+            "type": "string",
+            "description": "Title of the project",
+            "minLength": 20,
+            "maxLength": 250
+          },
+          "description": {
+            "type": "string",
+            "description": "Description of the project",
+            "minLength": 20,
+            "maxLength": 4000
+          },
+          "centre": {
+            "type": "string",
+            "description": "Centre, institute or company submitting the project"
+          },
+          "taxId": {
+            "type": "number",
+            "description": "Taxonomy ID of the species associated with the project (found here: http://www.ncbi.nlm.nih.gov/taxonomy)"
+          },
+          "publications": {
+            "type": "array",
+            "description": "Publications associated with the project, each given as <database>:<identifier>",
+            "items": {
               "type": "string",
-              "format": "date",
+              "pattern": "^[^:,]+?:[^:,]+?$",
               "examples": [
-                "2018-11-13"
+                "DOI:10.1093/nar/gkw1121",
+                "PMID:24565421"
               ]
             }
+          },
+          "parentProject": {
+            "$ref": "#/definitions/projectAccession"
+          },
+          "childProjects": {
+            "type": "array",
+            "items": {
+              "$ref": "#/definitions/projectAccession"
+            }
+          },
+          "peerProjects": {
+            "type": "array",
+            "items": {
+              "$ref": "#/definitions/projectAccession"
+            }
+          },
+          "links": {
+            "type": "array",
+            "items": {
+              "type": "string"
+            }
+          },
+          "holdDate": {
+            "type": "string",
+            "format": "date",
+            "examples": [
+              "2018-11-13"
+            ]
           }
         }
-      ]
+      }
     },
     "analysis": {
       "type": "array",
@@ -309,21 +332,24 @@
       "minItems": 1,
       "items": {
         "type": "object",
-        "oneOf": [
-          {
-            "required": [
-              "analysisAlias",
-              "sampleInVCF",
-              "bioSampleAccession"
-            ]
-          },
-          {
-            "required": [
-              "analysisAlias",
-              "bioSampleObject"
-            ]
-          }
-        ],
+        "if": {
+          "required": [
+            "bioSampleAccession"
+          ]
+        },
+        "then": {
+          "required": [
+            "analysisAlias",
+            "sampleInVCF",
+            "bioSampleAccession"
+          ]
+        },
+        "else": {
+          "required": [
+            "analysisAlias",
+            "bioSampleObject"
+          ]
+        },
         "properties": {
           "analysisAlias": {
             "type": "array",

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/eva_sub_cli/etc/spreadsheet2json_conf.yaml

@@ -11,6 +11,7 @@
 # List of worksheets to be parsed.
 worksheets:
   Submitter Details: submitterDetails
+  Statements: statements
   Project: project
   Analysis: analysis
   Sample: sample
@@ -26,6 +27,14 @@ Submitter Details:
   optional:
     Address: address
 
+Statements:
+  header_row: 2
+  required:
+    Acknowledgement: acknowledgement
+    Statement: statement
+  cast:
+    Acknowledgement: boolean
+
 # List of field for Projects
 Project:
   header_row: 3

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/eva_sub_cli/exceptions.py

@@ -38,3 +38,39 @@ class SubmissionUploadException(Exception):
     def __init__(self, message):
         self.message = message
         super().__init__(self.message)
+
+
+class DirLockError(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class UserFileNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class DependencyNotFoundException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class WebinBadCredentialsException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class InvalidSubmissionException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)
+
+
+class DockerValidatorException(Exception):
+    def __init__(self, message):
+        self.message = message
+        super().__init__(self.message)

{eva_sub_cli-0.6.6 → eva_sub_cli-0.7.0}/eva_sub_cli/executables/cli.py

@@ -15,7 +15,9 @@ import eva_sub_cli
 from eva_sub_cli import ENA_WEBIN_ACCOUNT_VAR, ENA_WEBIN_PASSWORD_VAR
 from eva_sub_cli import orchestrator
 from eva_sub_cli.exceptions import MetadataTemplateVersionException, MetadataTemplateVersionNotFoundException, \
-    SubmissionStatusException, SubmissionNotFoundException, SubmissionUploadException, NoVcfsFoundException
+    SubmissionStatusException, SubmissionNotFoundException, SubmissionUploadException, NoVcfsFoundException, \
+    UserFileNotFoundException, DependencyNotFoundException, WebinBadCredentialsException, InvalidFileTypeError, \
+    InvalidSubmissionException, DockerValidatorException
 from eva_sub_cli.call_home import CallHomeClient
 from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, DirLock
 from eva_sub_cli.orchestrator import VALIDATE, SUBMIT, DOCKER, NATIVE
@@ -126,51 +128,69 @@ def main():
             logging_config.add_file_handler(os.path.join(args.submission_dir, 'eva_submission.log'), logging.DEBUG)
             # Pass on all the arguments to the orchestrator
             orchestrator.orchestrate_process(call_home=call_home, **args.__dict__)
+
+    # User errors: not displayed as exceptions, call-home records as failure with no exception
     except DirLockError as dle:
-        logger.exception(f'Could not acquire the lock file for {args.submission_dir} because another process is using '
-                         f'this directory or a previous process did not terminate correctly. '
-                         f'If the problem persists, remove the lock file manually.')
-        caught_exception = dle
-        exit_status = 65
+        print(f'Could not acquire the lock file for {args.submission_dir} because another process is using this '
+              f'directory or a previous process did not terminate correctly. If the problem persists, remove the lock '
+              f'file manually.')
+        exit_status = 101
+    except NoVcfsFoundException as nvfe:
+        print(nvfe.message)
+        exit_status = 102
+    except MetadataTemplateVersionException as mte:
+        print(mte.message)
+        exit_status = 103
+    except MetadataTemplateVersionNotFoundException as mte:
+        print(mte.message)
+        exit_status = 104
+    except UserFileNotFoundException as ufne:
+        print(ufne.message)
+        exit_status = 105
+    except InvalidFileTypeError as ife:
+        print(ife.message)
+        exit_status = 106
+    except DependencyNotFoundException as dnfe:
+        print(dnfe.message)
+        exit_status = 107
+    except WebinBadCredentialsException as wae:
+        print(wae.message)
+        exit_status = 108
+    except InvalidSubmissionException as ise:
+        print(ise.message)
+        exit_status = 109
+
+    # Process errors: displayed as exceptions and sent to call-home
     except FileNotFoundError as fne:
         logger.exception(fne)
         caught_exception = fne
-        exit_status = 66
-    except NoVcfsFoundException as nvfe:
-        logger.exception(f'{nvfe}. Please ensure that you have listed VCF files in the metadata (.vcf or .vcf.gz file '
-                         f'extension), and that you have not modified the metadata template in any way.')
-        caught_exception = nvfe
-        exit_status = 67
+        exit_status = 201
     except SubmissionNotFoundException as snfe:
-        logger.exception(f'{snfe}. Please contact EVA Helpdesk')
+        logger.exception(snfe)
         caught_exception = snfe
-        exit_status = 68
+        exit_status = 202
     except SubmissionStatusException as sse:
-        logger.exception(f'{sse}. Please try again later. If the problem persists, please contact EVA Helpdesk')
+        logger.exception(sse)
         caught_exception = sse
-        exit_status = 69
-    except MetadataTemplateVersionException as mte:
-        logger.exception(mte)
-        caught_exception = mte
-        exit_status = 70
-    except MetadataTemplateVersionNotFoundException as mte:
-        logger.exception(mte)
-        caught_exception = mte
-        exit_status = 71
+        exit_status = 203
     except SubmissionUploadException as sue:
         logger.exception(sue)
         caught_exception = sue
-        exit_status = 72
+        exit_status = 204
     except HTTPError as http_err:
         logger.exception(http_err)
         if http_err.response is not None and http_err.response.text:
             print(http_err.response.text)
         caught_exception = http_err
-        exit_status = 73
+        exit_status = 205
+    except DockerValidatorException as dve:
+        logger.exception(dve)
+        caught_exception = dve
+        exit_status = 206
     except Exception as ex:
         logger.exception(ex)
         caught_exception = ex
-        exit_status = 74
+        exit_status = 299
 
     if call_home is not None:
         if exit_status == 0: