eva-sub-cli 0.4.13__tar.gz → 0.4.14__tar.gz

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.4.13
+Version: 0.4.14
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

eva_sub_cli-0.4.14/docs/installation.md

@@ -0,0 +1,70 @@
+# Installation
+
+There are two main ways to install eva-sub-cli, either through [conda](#1-conda) or [pip](#2-pip).
+Conda is the most straightforward and recommended method, as it will install all dependencies for you.
+
+Otherwise you can install using pip, in which case you will need to handle non-Python dependencies separately.
+You can do this either using Docker or natively (i.e. installing by yourself).
+Note that pip using Docker is currently the only installation method available for Windows users.
+
+We encourage users to install the most recent version, as we are constantly improving the tool. You can check that you
+have the latest version installed by running `eva-sub-cli.py --version` and comparing against the
+[release page](https://github.com/EBIvariation/eva-sub-cli/releases).
+
+All installation methods require Python 3.8 or above.
+
+## 1. Conda
+
+View our [video tutorial on conda installation](https://embl-ebi.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=aa82710e-4401-4074-b6b0-b346016bb14e).
+
+The most straightforward way to install eva-sub-cli and its dependencies is through conda.
+For instance, the following commands install eva-sub-cli in a new environment called `eva`, activate the environment, and print
+the help message:
+```bash
+conda create -n eva -c conda-forge -c bioconda eva-sub-cli
+conda activate eva
+eva-sub-cli.py --help
+````
+
+To upgrade to the newest version, run `conda update eva-sub-cli`.
+
+## 2. Pip
+
+View our [video tutorial on pip installation](https://embl-ebi.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=2e6828ed-31c2-4621-8eb5-b34701484aae).
+
+You can also install eva-sub-cli using pip. 
+The following commands will create a new virtual environment called `eva`, activate the environment, and install eva-sub-cli:
+```bash
+python3 -m venv eva
+source ./eva/bin/activate
+pip install eva-sub-cli
+```
+
+You can verify the installation by printing the help message:
+```bash
+eva-sub-cli.py --help
+```
+
+For external (non-Python) dependencies, you will need to install them either [via Docker](#docker) or [natively](#native).
+These methods are described below.
+
+To upgrade to the newest version of the tool, run `pip install --upgrade eva-sub-cli`.
+
+### Docker
+
+Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
+For this method, all you need is for [Docker](https://docs.docker.com/engine/install/) to be installed and running in the background.
+
+When you run eva-sub-cli, you will need to [specify execution using Docker](how_to_submit.md#running-using-docker).
+The tool will then automatically run using a Docker container with all dependencies pre-installed.
+
+### Native
+
+For those requiring more flexibility or working on systems where Docker is not available, you can also install the 
+external dependencies on your own. These are:
+* [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html) 21.10+
+* [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
+* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.7+
+
+Install each of these and ensure they are included in your PATH.
+This will allow eva-sub-cli to be executed natively, i.e. directly on your machine.

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/eva_sub_cli/_version.py

@@ -28,7 +28,7 @@ version_tuple: VERSION_TUPLE
 commit_id: COMMIT_ID
 __commit_id__: COMMIT_ID
 
-__version__ = version = '0.4.13'
-__version_tuple__ = version_tuple = (0, 4, 13)
+__version__ = version = '0.4.14'
+__version_tuple__ = version_tuple = (0, 4, 14)
 
-__commit_id__ = commit_id = 'gc6e52eaa9'
+__commit_id__ = commit_id = 'g1808b05b1'

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/eva_sub_cli/etc/eva_schema.json

@@ -324,18 +324,7 @@
             "description": "BioSample accession of pre-existing sample. Only use for preregistered samples"
           },
           "bioSampleObject": {
-            "type": "object",
-            "anyOf": [
-              {
-                "$ref": "eva-biosamples.json"
-              },
-              {
-                "$ref": "https://www.ebi.ac.uk/biosamples/schema-store/registry/schemas/ERC000011"
-              },
-              {
-                "$ref": "https://www.ebi.ac.uk/biosamples/schemas/certification/plant-miappe.json"
-              }
-            ]
+            "$ref": "https://www.ebi.ac.uk/biosamples/schema-store/registry/schemas/ERC000011"
           }
         }
       }

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/eva_sub_cli/executables/cli.py

@@ -22,24 +22,47 @@ from eva_sub_cli.file_utils import is_submission_dir_writable, DirLockError, Dir
 
 
 def validate_command_line_arguments(args, argparser):
+    fail = False
     if (args.vcf_files and not args.reference_fasta) or (not args.vcf_files and args.reference_fasta):
         print("When using --vcf_files and --reference_fasta, both need to be specified")
-        argparser.print_usage()
-        sys.exit(1)
+        fail = True
+
+    if args.vcf_files:
+        for vcf_file in args.vcf_files:
+            if not os.path.isfile(vcf_file):
+                print(f"VCF file {vcf_file} is not a file")
+                fail = True
+
+    if args.reference_fasta:
+        if not os.path.isfile(args.reference_fasta):
+            print(f"Fasta file {args.reference_fasta} is not a file")
+            fail = True
+
+    if args.metadata_xlsx:
+        if not os.path.isfile(args.metadata_xlsx):
+            print(f"Spreadsheet file {args.metadata_xlsx} is not a file")
+            fail = True
+
+    if args.metadata_json:
+        if not os.path.isfile(args.metadata_json):
+            print(f"JSON file {args.metadata_json} is not a file")
+            fail = True
 
     if SUBMIT in args.tasks and (
             not (args.username or os.environ.get('ENAWEBINACCOUNT')) or
             not (args.password or os.environ.get('ENAWEBINPASSWORD'))):
         print("To submit your data, you need to provide a Webin username and password")
-        argparser.print_usage()
-        sys.exit(1)
+        fail = True
 
     if not is_submission_dir_writable(args.submission_dir):
         print(f"'{args.submission_dir}' does not have write permissions or is not a directory.")
-        sys.exit(1)
+        fail = True
 
     if args.nextflow_config and not os.path.isfile(args.nextflow_config):
         print(f"'{args.nextflow_config}' is not a file or does not exist.")
+        fail = True
+
+    if fail:
         argparser.print_usage()
         sys.exit(1)
 

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/eva_sub_cli/nextflow/validation.nf

@@ -132,6 +132,9 @@ process trim_down_vcf {
 * Validate the VCF file format
 */
 process check_vcf_valid {
+
+    label 'long_time', 'med_mem'
+
     publishDir output_dir,
             overwrite: false,
             mode: "copy"
@@ -155,6 +158,9 @@ process check_vcf_valid {
 * Validate the VCF reference allele
 */
 process check_vcf_reference {
+
+    label 'long_time', 'med_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -178,6 +184,9 @@ process check_vcf_reference {
 }
 
 process generate_file_size_and_md5_digests {
+
+    label 'med_time', 'med_mem'
+
     input:
     path(vcf_file)
 
@@ -192,6 +201,9 @@ process generate_file_size_and_md5_digests {
 }
 
 process collect_file_size_and_md5 {
+
+    label 'med_time', 'med_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -210,6 +222,10 @@ process collect_file_size_and_md5 {
 
 
 process convert_xlsx_2_json {
+
+    label 'short_time', 'small_mem'
+
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -231,6 +247,9 @@ process convert_xlsx_2_json {
 }
 
 process metadata_json_validation {
+
+    label 'short_time', 'small_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -248,6 +267,9 @@ process metadata_json_validation {
 }
 
 process sample_name_concordance {
+
+    label 'default_time', 'default_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -268,6 +290,9 @@ process sample_name_concordance {
 
 
 process evidence_type_check {
+
+    label 'default_time', 'default_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -288,6 +313,9 @@ process evidence_type_check {
 
 
 process insdc_checker {
+
+    label 'long_time', 'default_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"
@@ -308,6 +336,9 @@ process insdc_checker {
 
 
 process metadata_semantic_check {
+
+    label 'default_time', 'default_mem'
+
     publishDir output_dir,
             overwrite: true,
             mode: "copy"

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/eva_sub_cli/validators/validator.py

@@ -335,11 +335,8 @@ class Validator(AppLogger):
             with open(self._evidence_type_check_yaml) as open_yaml:
                 self.results['evidence_type_check'] = yaml.safe_load(open_yaml)
             self.results['evidence_type_check']['report_path'] = self._evidence_type_check_yaml
-
         self._update_metadata_with_evidence_type()
 
-
-
     def _collect_metadata_results(self):
         self.results['metadata_check'] = {}
         self._load_spreadsheet_conversion_errors()
@@ -478,7 +475,6 @@ class Validator(AppLogger):
         else:
             self.error(f'Cannot locate the metadata in JSON format in {os.path.join(self.output_dir, "metadata.json")}')
 
-
     def _update_metadata_with_evidence_type(self):
         if self.metadata_json_post_validation:
             metadata = EvaMetadataJson(self.metadata_json_post_validation)
@@ -487,8 +483,10 @@ class Validator(AppLogger):
                 if metadata.analyses:
                     for analysis in metadata.analyses:
                         analysis_alias = analysis['analysisAlias']
-                        analysis['evidenceType'] = self.results['evidence_type_check'][analysis_alias][
-                            'evidence_type'] if analysis_alias in self.results['evidence_type_check'] else None
+                        if (analysis_alias in self.results['evidence_type_check']
+                                and self.results['evidence_type_check'][analysis_alias]['evidence_type'] is not None):
+                            analysis['evidenceType'] = self.results['evidence_type_check'][analysis_alias][
+                                'evidence_type']
                         analysis_data.append(analysis)
                 else:
                     self.error('No analyses found in metadata')

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/eva_sub_cli.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: eva_sub_cli
-Version: 0.4.13
+Version: 0.4.14
 Summary: EBI EVA - validation and submission command line tool
 Classifier: Development Status :: 5 - Production/Stable
 Classifier: Environment :: Console

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/tests/test_cli.py

@@ -9,6 +9,7 @@ from unittest.mock import patch, Mock
 
 from eva_sub_cli import orchestrator
 from eva_sub_cli.executables import cli
+from tests.test_utils import touch
 
 
 class TestCli(TestCase):
@@ -41,11 +42,17 @@ class TestCli(TestCase):
                 all_lines[0].endswith('[eva_sub_cli.orchestrator][DEBUG] test\n')
 
     def test_validate_args(self):
+        vcf_file = os.path.join(self.submission_dir,'test.vcf')
+        fasta_file = os.path.join(self.submission_dir, 'test.fasta')
+        json_file = os.path.join(self.submission_dir, 'test.json')
+        touch(vcf_file)
+        touch(fasta_file)
+        touch(json_file)
         cmd_args = [
             '--submission_dir', self.submission_dir,
-            '--vcf_files', 'test.vcf',
-            '--reference_fasta', 'test.fasta',
-            '--metadata_json', 'test.json',
+            '--vcf_files', vcf_file,
+            '--reference_fasta', fasta_file,
+            '--metadata_json', json_file,
             '--tasks', 'validate',
             '--executor', 'native',
             '--debug'

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/tests/test_validator.py

@@ -1,4 +1,5 @@
 import os.path
+import shutil
 from copy import deepcopy
 from unittest import TestCase
 
@@ -93,6 +94,7 @@ class TestValidator(TestCase):
     output_dir = os.path.join(resource_dir, 'validation_reports')
     mapping_file = os.path.join(output_dir, 'vcf_files_mapping.csv')
     metadata_xlsx_file = os.path.join(resource_dir, 'EVA_Submission_test.xlsx')
+    metadata_json_file = os.path.join(resource_dir, 'metadata_with_filename.json')
 
     def setUp(self) -> None:
         # create vcf mapping file
@@ -102,7 +104,10 @@ class TestValidator(TestCase):
                             [os.path.join(self.fasta_files, 'input_passed.fa')],
                             [os.path.join(self.assembly_reports, 'input_passed.txt')])
         self.validator = Validator(self.mapping_file, self.output_dir, metadata_xlsx=self.metadata_xlsx_file)
-        self.validator_json = Validator(self.mapping_file, self.output_dir)
+        self.validator_json = Validator(self.mapping_file, self.output_dir, metadata_json=self.metadata_json_file)
+        # Backup metadata json file so can restore after tests
+        self.backup_metadata_json = f'{self.metadata_json_file}.backup'
+        shutil.copy(self.metadata_json_file, self.backup_metadata_json)
 
     def tearDown(self) -> None:
         files_from_tests = [
@@ -115,6 +120,8 @@ class TestValidator(TestCase):
         for f in files_from_tests:
             if os.path.exists(f):
                 os.remove(f)
+        # Restore metadata json file
+        shutil.move(self.backup_metadata_json, self.metadata_json_file)
 
     def format_data_structure(self, source):
         if isinstance(source, dict):
@@ -329,3 +336,39 @@ class TestValidator(TestCase):
             os.remove(updated_metadata)
         # Reset metadata_json in case other tests need it
         self.validator_json.metadata_json = prev_metadata_json_value
+
+    def test__update_metadata_with_evidence_type_success(self):
+        self.validator_json.results['evidence_type_check'] = {
+            'AA': {
+                'errors': None,
+                'evidence_type': 'allele_frequency'
+            },
+            'report_path': '{resource_dir}/validation_reports/validation_output/other_validations/evidence_type_checker.yml'
+        }
+        self.validator_json._update_metadata_with_evidence_type()
+
+        # Analysis updated with evidence type
+        updated_metadata = EvaMetadataJson(self.validator_json.metadata_json_post_validation)
+        assert updated_metadata.analyses[0]['evidenceType'] == 'allele_frequency'
+
+    def test__update_metadata_with_evidence_type_failure(self):
+        self.validator_json.results['evidence_type_check'] = {
+            'AA': {
+                'errors': ['VCF file evidence type could not be determined'],
+                'evidence_type': None
+            },
+            'report_path': '{resource_dir}/validation_reports/validation_output/other_validations/evidence_type_checker.yml'
+        }
+        self.validator_json._update_metadata_with_evidence_type()
+
+        # Nothing added to analysis
+        updated_metadata = EvaMetadataJson(self.validator_json.metadata_json_post_validation)
+        assert 'evidenceType' not in updated_metadata.analyses[0]
+
+    def test__update_metadata_with_evidence_type_did_not_run(self):
+        self.validator_json.results['evidence_type_check'] = {}
+        self.validator_json._update_metadata_with_evidence_type()
+
+        # Nothing added to analysis
+        updated_metadata = EvaMetadataJson(self.validator_json.metadata_json_post_validation)
+        assert 'evidenceType' not in updated_metadata.analyses[0]

{eva_sub_cli-0.4.13 → eva_sub_cli-0.4.14}/tests/test_xlsx2json.py

@@ -14,7 +14,6 @@ class TestXlsReader(TestCase):
     resource_dir = os.path.join(os.path.dirname(__file__), 'resources')
     conf_filename = os.path.join(ETC_DIR, 'spreadsheet2json_conf.yaml')
     eva_schema = os.path.abspath(os.path.join(__file__, "../../eva_sub_cli/etc/eva_schema.json", ))
-    biosample_schema = os.path.abspath(os.path.join(__file__, "../../eva_sub_cli/etc/eva-biosamples.json", ))
 
     def tearDown(self):
         files_from_tests = [
@@ -62,16 +61,9 @@ class TestXlsReader(TestCase):
         # assert json schema
         with open(self.eva_schema) as eva_schema_file:
             eva_json_schema = json.load(eva_schema_file)
-        with open(self.biosample_schema) as biosample_schema_file:
-            biosample_json_schema = json.load(biosample_schema_file)
-
-        # assert created json file sample field conforms to eva-biosamples schema
-        jsonschema.validate(json_data['sample'][3]['bioSampleObject'], biosample_json_schema)
 
         # assert created json file conform to eva_schema
-        resolver = jsonschema.RefResolver.from_schema(eva_json_schema)
-        resolver.store['eva-biosamples.json'] = biosample_json_schema
-        jsonschema.validate(json_data, eva_json_schema, resolver=resolver)
+        jsonschema.validate(json_data, eva_json_schema)
 
     def test_create_xls_template(self):
         metadata_file = os.path.join(self.resource_dir, 'metadata_not_existing.xlsx')
@@ -106,10 +98,6 @@ class TestXlsReader(TestCase):
     def test_json_validation_fails_for_large_project_title_and_project_description(self) -> None:
         with open(self.eva_schema) as eva_schema_file:
             eva_json_schema = json.load(eva_schema_file)
-        with open(self.biosample_schema) as biosample_schema_file:
-            biosample_json_schema = json.load(biosample_schema_file)
-        resolver = jsonschema.RefResolver.from_schema(eva_json_schema)
-        resolver.store['eva-biosamples.json'] = biosample_json_schema
 
         # test fails for title
         json_data = self.get_expected_json()
@@ -117,7 +105,7 @@ class TestXlsReader(TestCase):
 
         assert len(json_data['project']['title']) == 600
         with self.assertRaises(jsonschema.ValidationError):
-            jsonschema.validate(json_data, eva_json_schema, resolver=resolver)
+            jsonschema.validate(json_data, eva_json_schema)
 
         # test fails for description
         json_data = self.get_expected_json()
@@ -125,7 +113,7 @@ class TestXlsReader(TestCase):
 
         assert len(json_data['project']['description']) == 6000
         with self.assertRaises(jsonschema.ValidationError):
-            jsonschema.validate(json_data, eva_json_schema, resolver=resolver)
+            jsonschema.validate(json_data, eva_json_schema)
 
     def build_large_string_of_length(self, length):
         return "A" * length
@@ -235,6 +223,9 @@ class TestXlsReader(TestCase):
                             ],
                             'collection date': [
                                 {'text': '2021-03-12'}
+                            ],
+                            'geographic location (country and/or sea)': [
+                                {'text': 'Afghanistan'}
                             ]
                         }
                     }

eva_sub_cli-0.4.13/docs/installation.md

@@ -1,49 +0,0 @@
-# Installation
-
-There are currently three ways to install and run eva-sub-cli:
-- Using conda
-- From source using Docker
-- From source natively (i.e. installing dependencies yourself)
-
-We encourage users to install the most recent version, as we are constantly improving the tool. You can check that you
-have the latest version installed by running `eva-sub-cli.py --version` and comparing against the
-[release page](https://github.com/EBIvariation/eva-sub-cli/releases).
-
-## 1. Conda
-
-The most straightforward way to install eva-sub-cli and its dependencies is through conda.
-For instance, the following commands install eva-sub-cli in a new environment called `eva`, activate the environment, and print
-the help message:
-```bash
-conda create -n eva -c conda-forge -c bioconda eva-sub-cli
-conda activate eva
-eva-sub-cli.py --help
-````
-
-To upgrade to the newest version, run `conda update eva-sub-cli`.
-
-## 2. From source using Docker
-
-Docker provides an easy way to run eva-sub-cli without installing dependencies separately.
-This method requires just Python 3.8+ and [Docker](https://docs.docker.com/engine/install/) to be installed.
-Then you can install the most recent version from [PyPI](https://pypi.org/project/eva-sub-cli/) in a virtual environment:
-```bash
-pip install eva-sub-cli
-```
-
-To verify that the cli tool is installed correctly, run the following command, and you should see the help message displayed: 
-```bash
-eva-sub-cli.py --help
-```
-
-To upgrade to the newest version, run `pip install --upgrade eva-sub-cli`.
-
-## 3. From source natively
-
-This installation method requires the following:
-* Python 3.8+
-* [Nextflow](https://www.nextflow.io/docs/latest/getstarted.html) 21.10+
-* [biovalidator](https://github.com/elixir-europe/biovalidator) 2.1.0+
-* [vcf-validator](https://github.com/EBIvariation/vcf-validator) 0.9.7+
-
-Install each of these and ensure they are included in your PATH. Then install the latest eva-sub-cli release as previously described.