deep-code 0.0.1.dev0__tar.gz → 0.1.1__tar.gz

{deep_code-0.0.1.dev0 → deep_code-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: deep_code
-Version: 0.0.1.dev0
+Version: 0.1.1
 Summary: deepesdl earthcode integration utility tool
 Author-email: Tejas Morbagal Harish <tejas.morbagalharish@brockmann-consult.de>
 License: MIT
@@ -28,6 +28,7 @@ Requires-Dist: ruff; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-recording; extra == "dev"
+Dynamic: license-file
 
 # deep-code
 
@@ -87,14 +88,9 @@ pytest --cov-report html --cov=deep-code
 providing different utility functions.
 Use the --help option with these subcommands to get more details on usage.
 
-###  deep-code publish-product
-
-Publish a dataset which is a result of an experiment to the EarthCODE 
-open-science catalog.
-
-```commandline
- deep-code publish-dataset /path/to/dataset-config.yaml
- ```
+The CLI retrieves the Git username and personal access token from a hidden file named 
+.gitaccess. Ensure this file is located in the same directory where you execute the CLI
+command.
 
 #### .gitaccess example
 
@@ -103,59 +99,49 @@ github-username: your-git-user
 github-token: personal access token
 ```
 
+###  deep-code publish
+
+Publish the experiment, workflow and dataset which is a result of an experiment to 
+the EarthCODE open-science catalog.
+
+```commandline
+ deep-code publish /path/to/dataset-config.yaml /path/to/workflow-config.yaml
+ ```
+
 #### dataset-config.yaml example
 
 ```
-dataset_id: hydrology-1D-0.009deg-100x60x60-3.0.2.zarr
-collection_id: hydrology
+dataset_id: esa-cci-permafrost-1x1151x1641-1.0.0.zarr
+collection_id: esa-cci-permafrost
 osc_themes:
-  - Land
-  - Oceans
+  - cryosphere
+osc_region: global
 # non-mandatory
-documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/hydrology-1D-0.009deg-100x60x60-3.0.2.zarr/
-access_link: s3://test
+documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/esa-cci-permafrost-1x1151x1641-0-0-2-zarr
+access_link: s3://deep-esdl-public/esa-cci-permafrost-1x1151x1641-1.0.0.zarr
 dataset_status: completed
-osc_region: global
-cf_parameter:
-  - name: hydrology
 ```
 
-dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 or your team bucket.
-
-### deep-code publish-workflow
+dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 bucket or your team 
+bucket.
 
-Publish a workflow/experiment to the EarthCODE open-science catalog.
-
-```commandline
-deep-code publish-workflow /path/to/workflow-config.yaml
- ```
 #### workflow-config.yaml example
 
 ```
-workflow_id: "4D Med hydrology cube generation"
+workflow_id: "esa-cci-permafrost"
 properties:
-  title: "Hydrology cube generation recipe"
-  description: "4D Med cube generation"
+  title: "ESA CCI permafrost"
+  description: "cube generation workflow for esa-cci-permafrost"
   keywords:
     - Earth Science
   themes:
-      - Atmosphere
-      - Ocean
-      - Evaporation
+      - cryosphere
   license: proprietary
   jupyter_kernel_info:
-    name: deepesdl-xcube-1.7.1
+    name: deepesdl-xcube-1.8.3
     python_version: 3.11
-    env_file: https://git/env.yml
-links:
-  - rel: "documentation"
-    type: "application/json"
-    title: "4DMed Hydrology Cube Generation Recipe"
-    href: "https://github.com/deepesdl/cube-gen/tree/main/hydrology/README.md"
-  - rel: "jupyter-notebook"
-    type: "application/json"
-    title: "Workflow Jupyter Notebook"
-    href: "https://github.com/deepesdl/cube-gen/blob/main/hydrology/notebooks/reading_hydrology.ipynb"
+    env_file: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/environment.yml"
+jupyter_notebook_url: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/Create-CCI-Permafrost-cube-EarthCODE.ipynb"
 contact:
   - name: Tejas Morbagal Harish
     organization: Brockmann Consult GmbH

{deep_code-0.0.1.dev0 → deep_code-0.1.1}/README.md

@@ -56,14 +56,9 @@ pytest --cov-report html --cov=deep-code
 providing different utility functions.
 Use the --help option with these subcommands to get more details on usage.
 
-###  deep-code publish-product
-
-Publish a dataset which is a result of an experiment to the EarthCODE 
-open-science catalog.
-
-```commandline
- deep-code publish-dataset /path/to/dataset-config.yaml
- ```
+The CLI retrieves the Git username and personal access token from a hidden file named 
+.gitaccess. Ensure this file is located in the same directory where you execute the CLI
+command.
 
 #### .gitaccess example
 
@@ -72,59 +67,49 @@ github-username: your-git-user
 github-token: personal access token
 ```
 
+###  deep-code publish
+
+Publish the experiment, workflow and dataset which is a result of an experiment to 
+the EarthCODE open-science catalog.
+
+```commandline
+ deep-code publish /path/to/dataset-config.yaml /path/to/workflow-config.yaml
+ ```
+
 #### dataset-config.yaml example
 
 ```
-dataset_id: hydrology-1D-0.009deg-100x60x60-3.0.2.zarr
-collection_id: hydrology
+dataset_id: esa-cci-permafrost-1x1151x1641-1.0.0.zarr
+collection_id: esa-cci-permafrost
 osc_themes:
-  - Land
-  - Oceans
+  - cryosphere
+osc_region: global
 # non-mandatory
-documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/hydrology-1D-0.009deg-100x60x60-3.0.2.zarr/
-access_link: s3://test
+documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/esa-cci-permafrost-1x1151x1641-0-0-2-zarr
+access_link: s3://deep-esdl-public/esa-cci-permafrost-1x1151x1641-1.0.0.zarr
 dataset_status: completed
-osc_region: global
-cf_parameter:
-  - name: hydrology
 ```
 
-dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 or your team bucket.
-
-### deep-code publish-workflow
+dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 bucket or your team 
+bucket.
 
-Publish a workflow/experiment to the EarthCODE open-science catalog.
-
-```commandline
-deep-code publish-workflow /path/to/workflow-config.yaml
- ```
 #### workflow-config.yaml example
 
 ```
-workflow_id: "4D Med hydrology cube generation"
+workflow_id: "esa-cci-permafrost"
 properties:
-  title: "Hydrology cube generation recipe"
-  description: "4D Med cube generation"
+  title: "ESA CCI permafrost"
+  description: "cube generation workflow for esa-cci-permafrost"
   keywords:
     - Earth Science
   themes:
-      - Atmosphere
-      - Ocean
-      - Evaporation
+      - cryosphere
   license: proprietary
   jupyter_kernel_info:
-    name: deepesdl-xcube-1.7.1
+    name: deepesdl-xcube-1.8.3
     python_version: 3.11
-    env_file: https://git/env.yml
-links:
-  - rel: "documentation"
-    type: "application/json"
-    title: "4DMed Hydrology Cube Generation Recipe"
-    href: "https://github.com/deepesdl/cube-gen/tree/main/hydrology/README.md"
-  - rel: "jupyter-notebook"
-    type: "application/json"
-    title: "Workflow Jupyter Notebook"
-    href: "https://github.com/deepesdl/cube-gen/blob/main/hydrology/notebooks/reading_hydrology.ipynb"
+    env_file: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/environment.yml"
+jupyter_notebook_url: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/Create-CCI-Permafrost-cube-EarthCODE.ipynb"
 contact:
   - name: Tejas Morbagal Harish
     organization: Brockmann Consult GmbH

{deep_code-0.0.1.dev0 → deep_code-0.1.1}/deep_code/cli/main.py

@@ -6,7 +6,7 @@
 
 import click
 
-from deep_code.cli.publish import publish_dataset, publish_workflow
+from deep_code.cli.publish import publish
 
 
 @click.group()
@@ -15,8 +15,7 @@ def main():
     pass
 
 
-main.add_command(publish_dataset)
-main.add_command(publish_workflow)
+main.add_command(publish)
 
 if __name__ == "__main__":
     main()

deep_code-0.1.1/deep_code/cli/publish.py

@@ -0,0 +1,21 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import click
+
+from deep_code.tools.publish import Publisher
+
+
+@click.command(name="publish")
+@click.argument("dataset_config", type=click.Path(exists=True))
+@click.argument("workflow_config", type=click.Path(exists=True))
+def publish(dataset_config, workflow_config):
+    """Request publishing a dataset to the open science catalogue.
+    """
+    publisher = Publisher(
+        dataset_config_path=dataset_config, workflow_config_path=workflow_config
+    )
+    publisher.publish_all()

deep_code-0.1.1/deep_code/constants.py

@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2024 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+OSC_SCHEMA_URI = "https://stac-extensions.github.io/osc/v1.0.0/schema.json"
+CF_SCHEMA_URI = "https://stac-extensions.github.io/cf/v0.2.0/schema.json"
+THEMES_SCHEMA_URI = "https://stac-extensions.github.io/themes/v1.0.0/schema.json"
+OSC_THEME_SCHEME = "https://github.com/stac-extensions/osc#theme"
+OSC_REPO_OWNER = "ESA-EarthCODE"
+OSC_REPO_NAME = "open-science-catalog-metadata"
+OSC_BRANCH_NAME = "add-new-collection"
+DEFAULT_THEME_SCHEME = (
+    "https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/sciencekeywords"
+)
+OGC_API_RECORD_SPEC = "http://www.opengis.net/spec/ogcapi-records-1/1.0/req/record-core"
+WF_BRANCH_NAME = "add-new-workflow-from-deepesdl"
+VARIABLE_BASE_CATALOG_SELF_HREF = "https://esa-earthcode.github.io/open-science-catalog-metadata/variables/catalog.json"
+PRODUCT_BASE_CATALOG_SELF_HREF = "https://esa-earthcode.github.io/open-science-catalog-metadata/products/catalog.json"
+DEEPESDL_COLLECTION_SELF_HREF = (
+    "https://esa-earthcode.github.io/open-science-catalog-metadata/projects/deepesdl"
+    "/collection.json"
+)
+BASE_URL_OSC = "https://esa-earthcode.github.io/open-science-catalog-metadata"
+EXPERIMENT_BASE_CATALOG_SELF_HREF = "https://esa-earthcode.github.io/open-science-catalog-metadata/experiments/catalog.json"
+WORKFLOW_BASE_CATALOG_SELF_HREF = (
+    "https://esa-earthcode.github.io/open-science-catalog-metadata/workflows/catalog"
+    ".json"
+)
+PROJECT_COLLECTION_NAME = "deep-earth-system-data-lab"

deep_code-0.1.1/deep_code/tests/tools/test_publish.py

@@ -0,0 +1,108 @@
+import unittest
+from unittest.mock import patch, mock_open, MagicMock
+import json
+import yaml
+from pathlib import Path
+import tempfile
+from pystac import Catalog
+
+from deep_code.tools.publish import Publisher
+
+
+class TestPublisher(unittest.TestCase):
+    @patch("fsspec.open")
+    @patch("deep_code.tools.publish.GitHubPublisher")
+    def setUp(self, mock_github_publisher, mock_fsspec_open):
+        # Mock GitHubPublisher to avoid reading .gitaccess
+        self.mock_github_publisher_instance = MagicMock()
+        mock_github_publisher.return_value = self.mock_github_publisher_instance
+
+        # Mock dataset and workflow config files
+        self.dataset_config = {
+            "collection_id": "test-collection",
+            "dataset_id": "test-dataset",
+        }
+        self.workflow_config = {
+            "properties": {"title": "Test Workflow"},
+            "workflow_id": "test-workflow",
+        }
+
+        # Mock fsspec.open for config files
+        self.mock_fsspec_open = mock_fsspec_open
+        self.mock_fsspec_open.side_effect = [
+            mock_open(read_data=yaml.dump(self.dataset_config)).return_value,
+            mock_open(read_data=yaml.dump(self.workflow_config)).return_value,
+        ]
+
+        # Initialize Publisher
+        self.publisher = Publisher(
+            dataset_config_path="test-dataset-config.yaml",
+            workflow_config_path="test-workflow-config.yaml",
+        )
+
+    def test_normalize_name(self):
+        self.assertEqual(Publisher._normalize_name("Test Name"), "test-name")
+        self.assertEqual(Publisher._normalize_name("Test   Name"), "test---name")
+        self.assertIsNone(Publisher._normalize_name(""))
+        self.assertIsNone(Publisher._normalize_name(None))
+
+    def test_write_to_file(self):
+        # Create a temporary file
+        with tempfile.NamedTemporaryFile(delete=False) as temp_file:
+            file_path = temp_file.name
+
+        # Test data
+        data = {"key": "value"}
+
+        # Call the method
+        Publisher._write_to_file(file_path, data)
+
+        # Read the file and verify its content
+        with open(file_path, "r") as f:
+            content = json.load(f)
+            self.assertEqual(content, data)
+
+        # Clean up
+        Path(file_path).unlink()
+
+    def test_update_base_catalog(self):
+        # Create a mock Catalog
+        catalog = Catalog(id="test-catalog", description="Test Catalog")
+
+        # Mock file path and item ID
+        catalog_path = "test-catalog.json"
+        item_id = "test-item"
+        self_href = "https://example.com/catalog.json"
+
+        self.publisher.workflow_title = "Test Workflow"
+
+        # Mock the Catalog.from_file method
+        with patch("pystac.Catalog.from_file", return_value=catalog):
+            updated_catalog = self.publisher._update_base_catalog(
+                catalog_path, item_id, self_href
+            )
+
+        # Assertions
+        self.assertEqual(updated_catalog.get_self_href(), self_href)
+        self.assertIsInstance(updated_catalog, Catalog)
+
+    def test_read_config_files(self):
+        # Mock dataset and workflow config files
+        dataset_config = {
+            "collection_id": "test-collection",
+            "dataset_id": "test-dataset",
+        }
+        workflow_config = {
+            "properties": {"title": "Test Workflow"},
+            "workflow_id": "test-workflow",
+        }
+
+        # Mock fsspec.open for config files
+        self.mock_fsspec_open.side_effect = [
+            mock_open(read_data=yaml.dump(dataset_config)).return_value,
+            mock_open(read_data=yaml.dump(workflow_config)).return_value,
+        ]
+
+        # Assertions
+        self.assertEqual(self.publisher.dataset_config, dataset_config)
+        self.assertEqual(self.publisher.workflow_config, workflow_config)

{deep_code-0.0.1.dev0 → deep_code-0.1.1}/deep_code/tests/utils/test_dataset_stac_generator.py

@@ -217,3 +217,61 @@ class TestOSCProductSTACGenerator(unittest.TestCase):
         )
         self.assertIn("Public store, Authenticated store", str(context.exception))
         self.assertEqual(mock_new_data_store.call_count, 2)
+
+
+class TestFormatString(unittest.TestCase):
+    def test_single_word(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("temperature"), "Temperature"
+        )
+        self.assertEqual(OscDatasetStacGenerator.format_string("temp"), "Temp")
+        self.assertEqual(OscDatasetStacGenerator.format_string("hello"), "Hello")
+
+    def test_multiple_words_with_spaces(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("surface temp"), "Surface Temp"
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("this is a test"), "This Is A Test"
+        )
+
+    def test_multiple_words_with_underscores(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("surface_temp"), "Surface Temp"
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("this_is_a_test"), "This Is A Test"
+        )
+
+    def test_mixed_spaces_and_underscores(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("surface_temp and_more"),
+            "Surface Temp And More",
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string(
+                "mixed_case_with_underscores_and spaces"
+            ),
+            "Mixed Case With Underscores And Spaces",
+        )
+
+    def test_edge_cases(self):
+        # Empty string
+        self.assertEqual(OscDatasetStacGenerator.format_string(""), "")
+        # Single word with trailing underscore
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("temperature_"), "Temperature"
+        )
+        # Single word with leading underscore
+        self.assertEqual(OscDatasetStacGenerator.format_string("_temp"), "Temp")
+        # Single word with leading/trailing spaces
+        self.assertEqual(OscDatasetStacGenerator.format_string("  hello  "), "Hello")
+        # Multiple spaces or underscores
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("too___many___underscores"),
+            "Too Many Underscores",
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("too   many   spaces"),
+            "Too Many Spaces",
+        )

deep_code-0.1.1/deep_code/tests/utils/test_github_automation.py

@@ -0,0 +1,171 @@
+import logging
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock, patch
+
+from deep_code.utils.github_automation import GitHubAutomation
+
+
+class TestGitHubAutomation(unittest.TestCase):
+    def setUp(self):
+        # Set up test data
+        self.username = "testuser"
+        self.token = "testtoken"
+        self.repo_owner = "testowner"
+        self.repo_name = "testrepo"
+        self.github_automation = GitHubAutomation(
+            self.username, self.token, self.repo_owner, self.repo_name
+        )
+        logging.disable(logging.CRITICAL)  # Disable logging during tests
+
+    def tearDown(self):
+        logging.disable(logging.NOTSET)  # Re-enable logging after tests
+
+    @patch("requests.post")
+    def test_fork_repository(self, mock_post):
+        # Mock the response from GitHub API
+        mock_response = MagicMock()
+        mock_response.raise_for_status.return_value = None
+        mock_post.return_value = mock_response
+
+        # Call the method
+        self.github_automation.fork_repository()
+
+        # Assertions
+        mock_post.assert_called_once_with(
+            f"https://api.github.com/repos/{self.repo_owner}/{self.repo_name}/forks",
+            headers={"Authorization": f"token {self.token}"},
+        )
+
+    @patch("subprocess.run")
+    def test_clone_repository_new(self, mock_run):
+        # Mock the subprocess.run method
+        mock_run.return_value = MagicMock()
+
+        # Mock os.path.exists to return False (directory does not exist)
+        with patch("os.path.exists", return_value=False):
+            self.github_automation.clone_sync_repository()
+
+        # Assertions
+        mock_run.assert_called_once_with(
+            [
+                "git",
+                "clone",
+                f"https://{self.username}:{self.token}@github.com/{self.username}/{self.repo_name}.git",
+                self.github_automation.local_clone_dir,
+            ],
+            check=True,
+        )
+
+    @patch("subprocess.run")
+    def test_clone_repository_existing(self, mock_run):
+        # Mock the subprocess.run method
+        mock_run.return_value = MagicMock()
+
+        # Mock os.path.exists to return True (directory exists)
+        with patch("os.path.exists", return_value=True):
+            with patch("os.chdir"):
+                self.github_automation.clone_sync_repository()
+
+        # Assertions
+        mock_run.assert_called_once_with(["git", "pull"], check=True)
+
+    @patch("subprocess.run")
+    def test_create_branch(self, mock_run):
+        # Mock the subprocess.run method
+        mock_run.return_value = MagicMock()
+
+        # Mock os.chdir
+        with patch("os.chdir"):
+            self.github_automation.create_branch("test-branch")
+
+        # Assertions
+        mock_run.assert_called_once_with(
+            ["git", "checkout", "-b", "test-branch"], check=True
+        )
+
+    @patch("subprocess.run")
+    def test_add_file(self, mock_run):
+        # Mock the subprocess.run method
+        mock_run.return_value = MagicMock()
+
+        # Mock os.chdir and Path
+        with patch("os.chdir"), patch("pathlib.Path.mkdir"), patch(
+            "builtins.open", unittest.mock.mock_open()
+        ):
+            self.github_automation.add_file("test/file.json", {"key": "value"})
+
+        # Assertions
+        mock_run.assert_called_once_with(
+            [
+                "git",
+                "add",
+                str(Path(self.github_automation.local_clone_dir) / "test/file.json"),
+            ],
+            check=True,
+        )
+
+    @patch("subprocess.run")
+    def test_commit_and_push(self, mock_run):
+        # Mock the subprocess.run method
+        mock_run.return_value = MagicMock()
+
+        # Mock os.chdir
+        with patch("os.chdir"):
+            self.github_automation.commit_and_push("test-branch", "Test commit message")
+
+        # Assertions
+        mock_run.assert_any_call(
+            ["git", "commit", "-m", "Test commit message"], check=True
+        )
+        mock_run.assert_any_call(
+            ["git", "push", "-u", "origin", "test-branch"], check=True
+        )
+
+    @patch("requests.post")
+    def test_create_pull_request(self, mock_post):
+        # Mock the response from GitHub API
+        mock_response = MagicMock()
+        mock_response.raise_for_status.return_value = None
+        mock_response.json.return_value = {"html_url": "https://github.com/test/pull/1"}
+        mock_post.return_value = mock_response
+
+        # Mock os.chdir
+        with patch("os.chdir"):
+            self.github_automation.create_pull_request(
+                "test-branch", "Test PR", "Test body"
+            )
+
+        # Assertions
+        mock_post.assert_called_once_with(
+            f"https://api.github.com/repos/{self.repo_owner}/{self.repo_name}/pulls",
+            headers={"Authorization": f"token {self.token}"},
+            json={
+                "title": "Test PR",
+                "head": f"{self.username}:test-branch",
+                "base": "main",
+                "body": "Test body",
+            },
+        )
+
+    @patch("subprocess.run")
+    def test_clean_up(self, mock_run):
+        # Mock the subprocess.run method
+        mock_run.return_value = MagicMock()
+
+        # Mock os.chdir
+        with patch("os.chdir"):
+            self.github_automation.clean_up()
+
+        # Assertions
+        mock_run.assert_called_once_with(
+            ["rm", "-rf", self.github_automation.local_clone_dir]
+        )
+
+    def test_file_exists(self):
+        # Mock os.path.isfile
+        with patch("os.path.isfile", return_value=True):
+            result = self.github_automation.file_exists("test/file.json")
+
+        # Assertions
+        self.assertTrue(result)