deep-code 0.0.1.dev0__tar.gz → 0.1.2__tar.gz

{deep_code-0.0.1.dev0 → deep_code-0.1.2}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.2
+Metadata-Version: 2.4
 Name: deep_code
-Version: 0.0.1.dev0
+Version: 0.1.2
 Summary: deepesdl earthcode integration utility tool
 Author-email: Tejas Morbagal Harish <tejas.morbagalharish@brockmann-consult.de>
 License: MIT
@@ -28,6 +28,7 @@ Requires-Dist: ruff; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-cov; extra == "dev"
 Requires-Dist: pytest-recording; extra == "dev"
+Dynamic: license-file
 
 # deep-code
 
@@ -46,7 +47,8 @@ experiments/workflow as OGC API record and Datasets as an OSC stac collection.
 ## Setup
 
 ## Install
-`deep-code` will be available in PyPI and conda-forge. Till the stable release,
+`deep-code` will be available in PyPI for now and will be available in conda-forge 
+in the near future. Till the stable release,
 developers/contributors can follow the below steps to install deep-code.
 
 ## Installing from the repository for Developers/Contributors
@@ -87,14 +89,9 @@ pytest --cov-report html --cov=deep-code
 providing different utility functions.
 Use the --help option with these subcommands to get more details on usage.
 
-###  deep-code publish-product
-
-Publish a dataset which is a result of an experiment to the EarthCODE 
-open-science catalog.
-
-```commandline
- deep-code publish-dataset /path/to/dataset-config.yaml
- ```
+The CLI retrieves the Git username and personal access token from a hidden file named 
+.gitaccess. Ensure this file is located in the same directory where you execute the CLI
+command.
 
 #### .gitaccess example
 
@@ -102,60 +99,93 @@ open-science catalog.
 github-username: your-git-user
 github-token: personal access token
 ```
+### deep-code generate-config
+
+Generates starter configuration templates for publishing to EarthCODE openscience 
+catalog.
+
+#### Usage
+```
+deep-code generate-config [OPTIONS]
+```
+
+#### Options
+     --output-dir, -o : Output directory (default: current)
+
+#### Examples:
+```
+deep-code generate-config
+deep-code generate-config -o ./configs
+```
+
+###  deep-code publish
+
+Publishes metadata of experiment, workflow and dataset to the EarthCODE open-science 
+catalog
+
+### Usage
+```
+deep-code publish DATASET_CONFIG WORKFLOW_CONFIG [--environment ENVIRONMENT]
+ ```
+
+#### Arguments
+    DATASET_CONFIG - Path to the dataset configuration YAML file
+    (e.g., dataset-config.yaml)
+
+    WORKFLOW_CONFIG - Path to the workflow configuration YAML file
+    (e.g., workflow-config.yaml)
+
+#### Options
+    --environment, -e - Target catalog environment:
+    production (default) | staging | testing
 
+#### Examples:
+1. Publish to staging catalog
+```
+deep-code publish dataset-config.yaml workflow-config.yaml --environment=staging
+```
+2. Publish to testing catalog
+```
+deep-code publish dataset-config.yaml workflow-config.yaml -e testing
+```
+3. Publish to production catalog
+```
+deep-code publish dataset-config.yaml workflow-config.yaml
+```
 #### dataset-config.yaml example
 
 ```
-dataset_id: hydrology-1D-0.009deg-100x60x60-3.0.2.zarr
-collection_id: hydrology
+dataset_id: esa-cci-permafrost-1x1151x1641-1.0.0.zarr
+collection_id: esa-cci-permafrost
 osc_themes:
-  - Land
-  - Oceans
+  - cryosphere
+osc_region: global
 # non-mandatory
-documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/hydrology-1D-0.009deg-100x60x60-3.0.2.zarr/
-access_link: s3://test
+documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/esa-cci-permafrost-1x1151x1641-0-0-2-zarr
+access_link: s3://deep-esdl-public/esa-cci-permafrost-1x1151x1641-1.0.0.zarr
 dataset_status: completed
-osc_region: global
-cf_parameter:
-  - name: hydrology
 ```
 
-dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 or your team bucket.
+dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 bucket or your team 
+bucket.
 
-### deep-code publish-workflow
-
-Publish a workflow/experiment to the EarthCODE open-science catalog.
-
-```commandline
-deep-code publish-workflow /path/to/workflow-config.yaml
- ```
 #### workflow-config.yaml example
 
 ```
-workflow_id: "4D Med hydrology cube generation"
+workflow_id: "esa-cci-permafrost"
 properties:
-  title: "Hydrology cube generation recipe"
-  description: "4D Med cube generation"
+  title: "ESA CCI permafrost"
+  description: "cube generation workflow for esa-cci-permafrost"
   keywords:
     - Earth Science
   themes:
-      - Atmosphere
-      - Ocean
-      - Evaporation
+      - cryosphere
   license: proprietary
   jupyter_kernel_info:
-    name: deepesdl-xcube-1.7.1
+    name: deepesdl-xcube-1.8.3
     python_version: 3.11
-    env_file: https://git/env.yml
-links:
-  - rel: "documentation"
-    type: "application/json"
-    title: "4DMed Hydrology Cube Generation Recipe"
-    href: "https://github.com/deepesdl/cube-gen/tree/main/hydrology/README.md"
-  - rel: "jupyter-notebook"
-    type: "application/json"
-    title: "Workflow Jupyter Notebook"
-    href: "https://github.com/deepesdl/cube-gen/blob/main/hydrology/notebooks/reading_hydrology.ipynb"
+    env_file: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/environment.yml"
+jupyter_notebook_url: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/Create-CCI-Permafrost-cube-EarthCODE.ipynb"
 contact:
   - name: Tejas Morbagal Harish
     organization: Brockmann Consult GmbH

{deep_code-0.0.1.dev0 → deep_code-0.1.2}/README.md

@@ -15,7 +15,8 @@ experiments/workflow as OGC API record and Datasets as an OSC stac collection.
 ## Setup
 
 ## Install
-`deep-code` will be available in PyPI and conda-forge. Till the stable release,
+`deep-code` will be available in PyPI for now and will be available in conda-forge 
+in the near future. Till the stable release,
 developers/contributors can follow the below steps to install deep-code.
 
 ## Installing from the repository for Developers/Contributors
@@ -56,14 +57,9 @@ pytest --cov-report html --cov=deep-code
 providing different utility functions.
 Use the --help option with these subcommands to get more details on usage.
 
-###  deep-code publish-product
-
-Publish a dataset which is a result of an experiment to the EarthCODE 
-open-science catalog.
-
-```commandline
- deep-code publish-dataset /path/to/dataset-config.yaml
- ```
+The CLI retrieves the Git username and personal access token from a hidden file named 
+.gitaccess. Ensure this file is located in the same directory where you execute the CLI
+command.
 
 #### .gitaccess example
 
@@ -71,60 +67,93 @@ open-science catalog.
 github-username: your-git-user
 github-token: personal access token
 ```
+### deep-code generate-config
+
+Generates starter configuration templates for publishing to EarthCODE openscience 
+catalog.
+
+#### Usage
+```
+deep-code generate-config [OPTIONS]
+```
+
+#### Options
+     --output-dir, -o : Output directory (default: current)
+
+#### Examples:
+```
+deep-code generate-config
+deep-code generate-config -o ./configs
+```
+
+###  deep-code publish
+
+Publishes metadata of experiment, workflow and dataset to the EarthCODE open-science 
+catalog
+
+### Usage
+```
+deep-code publish DATASET_CONFIG WORKFLOW_CONFIG [--environment ENVIRONMENT]
+ ```
+
+#### Arguments
+    DATASET_CONFIG - Path to the dataset configuration YAML file
+    (e.g., dataset-config.yaml)
+
+    WORKFLOW_CONFIG - Path to the workflow configuration YAML file
+    (e.g., workflow-config.yaml)
+
+#### Options
+    --environment, -e - Target catalog environment:
+    production (default) | staging | testing
 
+#### Examples:
+1. Publish to staging catalog
+```
+deep-code publish dataset-config.yaml workflow-config.yaml --environment=staging
+```
+2. Publish to testing catalog
+```
+deep-code publish dataset-config.yaml workflow-config.yaml -e testing
+```
+3. Publish to production catalog
+```
+deep-code publish dataset-config.yaml workflow-config.yaml
+```
 #### dataset-config.yaml example
 
 ```
-dataset_id: hydrology-1D-0.009deg-100x60x60-3.0.2.zarr
-collection_id: hydrology
+dataset_id: esa-cci-permafrost-1x1151x1641-1.0.0.zarr
+collection_id: esa-cci-permafrost
 osc_themes:
-  - Land
-  - Oceans
+  - cryosphere
+osc_region: global
 # non-mandatory
-documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/hydrology-1D-0.009deg-100x60x60-3.0.2.zarr/
-access_link: s3://test
+documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/esa-cci-permafrost-1x1151x1641-0-0-2-zarr
+access_link: s3://deep-esdl-public/esa-cci-permafrost-1x1151x1641-1.0.0.zarr
 dataset_status: completed
-osc_region: global
-cf_parameter:
-  - name: hydrology
 ```
 
-dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 or your team bucket.
+dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 bucket or your team 
+bucket.
 
-### deep-code publish-workflow
-
-Publish a workflow/experiment to the EarthCODE open-science catalog.
-
-```commandline
-deep-code publish-workflow /path/to/workflow-config.yaml
- ```
 #### workflow-config.yaml example
 
 ```
-workflow_id: "4D Med hydrology cube generation"
+workflow_id: "esa-cci-permafrost"
 properties:
-  title: "Hydrology cube generation recipe"
-  description: "4D Med cube generation"
+  title: "ESA CCI permafrost"
+  description: "cube generation workflow for esa-cci-permafrost"
   keywords:
     - Earth Science
   themes:
-      - Atmosphere
-      - Ocean
-      - Evaporation
+      - cryosphere
   license: proprietary
   jupyter_kernel_info:
-    name: deepesdl-xcube-1.7.1
+    name: deepesdl-xcube-1.8.3
     python_version: 3.11
-    env_file: https://git/env.yml
-links:
-  - rel: "documentation"
-    type: "application/json"
-    title: "4DMed Hydrology Cube Generation Recipe"
-    href: "https://github.com/deepesdl/cube-gen/tree/main/hydrology/README.md"
-  - rel: "jupyter-notebook"
-    type: "application/json"
-    title: "Workflow Jupyter Notebook"
-    href: "https://github.com/deepesdl/cube-gen/blob/main/hydrology/notebooks/reading_hydrology.ipynb"
+    env_file: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/environment.yml"
+jupyter_notebook_url: "https://github.com/deepesdl/cube-gen/blob/main/Permafrost/Create-CCI-Permafrost-cube-EarthCODE.ipynb"
 contact:
   - name: Tejas Morbagal Harish
     organization: Brockmann Consult GmbH

deep_code-0.1.2/deep_code/cli/generate_config.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import click
+
+from deep_code.tools.new import TemplateGenerator
+
+
+@click.command(name="generate-config")
+@click.option(
+    "--output-dir",
+    "-o",
+    type=click.Path(exists=True, file_okay=False, writable=True),
+    default=".",
+    help="Output directory for templates",
+)
+def generate_config(output_dir):
+    TemplateGenerator.generate_workflow_template(f"{output_dir}/workflow_config.yaml")
+    TemplateGenerator.generate_dataset_template(f"{output_dir}/dataset_config.yaml")

{deep_code-0.0.1.dev0 → deep_code-0.1.2}/deep_code/cli/main.py

@@ -6,7 +6,8 @@
 
 import click
 
-from deep_code.cli.publish import publish_dataset, publish_workflow
+from deep_code.cli.generate_config import generate_config
+from deep_code.cli.publish import publish
 
 
 @click.group()
@@ -15,8 +16,8 @@ def main():
     pass
 
 
-main.add_command(publish_dataset)
-main.add_command(publish_workflow)
+main.add_command(publish)
+main.add_command(generate_config)
 
 if __name__ == "__main__":
     main()

deep_code-0.1.2/deep_code/cli/publish.py

@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import click
+
+from deep_code.tools.publish import Publisher
+
+
+@click.command(name="publish")
+@click.argument("dataset_config", type=click.Path(exists=True))
+@click.argument("workflow_config", type=click.Path(exists=True))
+@click.option(
+    "--environment",
+    "-e",
+    type=click.Choice(["production", "staging", "testing"], case_sensitive=False),
+    default="production",
+    help="Target environment for publishing (production, staging, testing)",
+)
+def publish(dataset_config, workflow_config, environment):
+    """Request publishing a dataset along with experiment and workflow metadata to the
+    open science catalogue.
+    """
+    publisher = Publisher(
+        dataset_config_path=dataset_config,
+        workflow_config_path=workflow_config,
+        environment=environment.lower(),
+    )
+    publisher.publish_all()

deep_code-0.1.2/deep_code/constants.py

@@ -0,0 +1,31 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2024 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+OSC_SCHEMA_URI = "https://stac-extensions.github.io/osc/v1.0.0/schema.json"
+CF_SCHEMA_URI = "https://stac-extensions.github.io/cf/v0.2.0/schema.json"
+THEMES_SCHEMA_URI = "https://stac-extensions.github.io/themes/v1.0.0/schema.json"
+OSC_THEME_SCHEME = "https://github.com/stac-extensions/osc#theme"
+OSC_REPO_OWNER = "ESA-EarthCODE"
+OSC_REPO_NAME = "open-science-catalog-metadata"
+OSC_BRANCH_NAME = "add-new-collection"
+DEFAULT_THEME_SCHEME = (
+    "https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/sciencekeywords"
+)
+OGC_API_RECORD_SPEC = "http://www.opengis.net/spec/ogcapi-records-1/1.0/req/record-core"
+WF_BRANCH_NAME = "add-new-workflow-from-deepesdl"
+VARIABLE_BASE_CATALOG_SELF_HREF = "https://esa-earthcode.github.io/open-science-catalog-metadata/variables/catalog.json"
+PRODUCT_BASE_CATALOG_SELF_HREF = "https://esa-earthcode.github.io/open-science-catalog-metadata/products/catalog.json"
+DEEPESDL_COLLECTION_SELF_HREF = (
+    "https://esa-earthcode.github.io/open-science-catalog-metadata/projects/deepesdl"
+    "/collection.json"
+)
+BASE_URL_OSC = "https://esa-earthcode.github.io/open-science-catalog-metadata"
+EXPERIMENT_BASE_CATALOG_SELF_HREF = "https://esa-earthcode.github.io/open-science-catalog-metadata/experiments/catalog.json"
+WORKFLOW_BASE_CATALOG_SELF_HREF = (
+    "https://esa-earthcode.github.io/open-science-catalog-metadata/workflows/catalog"
+    ".json"
+)
+PROJECT_COLLECTION_NAME = "deep-earth-system-data-lab"

deep_code-0.1.2/deep_code/tests/tools/test_publish.py

@@ -0,0 +1,109 @@
+import json
+import tempfile
+import unittest
+from pathlib import Path
+from unittest.mock import MagicMock, mock_open, patch
+
+import yaml
+from pystac import Catalog
+
+from deep_code.tools.publish import Publisher
+
+
+class TestPublisher(unittest.TestCase):
+    @patch("fsspec.open")
+    @patch("deep_code.tools.publish.GitHubPublisher")
+    def setUp(self, mock_github_publisher, mock_fsspec_open):
+        # Mock GitHubPublisher to avoid reading .gitaccess
+        self.mock_github_publisher_instance = MagicMock()
+        mock_github_publisher.return_value = self.mock_github_publisher_instance
+
+        # Mock dataset and workflow config files
+        self.dataset_config = {
+            "collection_id": "test-collection",
+            "dataset_id": "test-dataset",
+        }
+        self.workflow_config = {
+            "properties": {"title": "Test Workflow"},
+            "workflow_id": "test-workflow",
+        }
+
+        # Mock fsspec.open for config files
+        self.mock_fsspec_open = mock_fsspec_open
+        self.mock_fsspec_open.side_effect = [
+            mock_open(read_data=yaml.dump(self.dataset_config)).return_value,
+            mock_open(read_data=yaml.dump(self.workflow_config)).return_value,
+        ]
+
+        # Initialize Publisher
+        self.publisher = Publisher(
+            dataset_config_path="test-dataset-config.yaml",
+            workflow_config_path="test-workflow-config.yaml",
+        )
+
+    def test_normalize_name(self):
+        self.assertEqual(Publisher._normalize_name("Test Name"), "test-name")
+        self.assertEqual(Publisher._normalize_name("Test   Name"), "test---name")
+        self.assertIsNone(Publisher._normalize_name(""))
+        self.assertIsNone(Publisher._normalize_name(None))
+
+    def test_write_to_file(self):
+        # Create a temporary file
+        with tempfile.NamedTemporaryFile(delete=False) as temp_file:
+            file_path = temp_file.name
+
+        # Test data
+        data = {"key": "value"}
+
+        # Call the method
+        Publisher._write_to_file(file_path, data)
+
+        # Read the file and verify its content
+        with open(file_path, "r") as f:
+            content = json.load(f)
+            self.assertEqual(content, data)
+
+        # Clean up
+        Path(file_path).unlink()
+
+    def test_update_base_catalog(self):
+        # Create a mock Catalog
+        catalog = Catalog(id="test-catalog", description="Test Catalog")
+
+        # Mock file path and item ID
+        catalog_path = "test-catalog.json"
+        item_id = "test-item"
+        self_href = "https://example.com/catalog.json"
+
+        self.publisher.workflow_title = "Test Workflow"
+
+        # Mock the Catalog.from_file method
+        with patch("pystac.Catalog.from_file", return_value=catalog):
+            updated_catalog = self.publisher._update_base_catalog(
+                catalog_path, item_id, self_href
+            )
+
+        # Assertions
+        self.assertEqual(updated_catalog.get_self_href(), self_href)
+        self.assertIsInstance(updated_catalog, Catalog)
+
+    def test_read_config_files(self):
+        # Mock dataset and workflow config files
+        dataset_config = {
+            "collection_id": "test-collection",
+            "dataset_id": "test-dataset",
+        }
+        workflow_config = {
+            "properties": {"title": "Test Workflow"},
+            "workflow_id": "test-workflow",
+        }
+
+        # Mock fsspec.open for config files
+        self.mock_fsspec_open.side_effect = [
+            mock_open(read_data=yaml.dump(dataset_config)).return_value,
+            mock_open(read_data=yaml.dump(workflow_config)).return_value,
+        ]
+
+        # Assertions
+        self.assertEqual(self.publisher.dataset_config, dataset_config)
+        self.assertEqual(self.publisher.workflow_config, workflow_config)

{deep_code-0.0.1.dev0 → deep_code-0.1.2}/deep_code/tests/utils/test_dataset_stac_generator.py

@@ -217,3 +217,61 @@ class TestOSCProductSTACGenerator(unittest.TestCase):
         )
         self.assertIn("Public store, Authenticated store", str(context.exception))
         self.assertEqual(mock_new_data_store.call_count, 2)
+
+
+class TestFormatString(unittest.TestCase):
+    def test_single_word(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("temperature"), "Temperature"
+        )
+        self.assertEqual(OscDatasetStacGenerator.format_string("temp"), "Temp")
+        self.assertEqual(OscDatasetStacGenerator.format_string("hello"), "Hello")
+
+    def test_multiple_words_with_spaces(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("surface temp"), "Surface Temp"
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("this is a test"), "This Is A Test"
+        )
+
+    def test_multiple_words_with_underscores(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("surface_temp"), "Surface Temp"
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("this_is_a_test"), "This Is A Test"
+        )
+
+    def test_mixed_spaces_and_underscores(self):
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("surface_temp and_more"),
+            "Surface Temp And More",
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string(
+                "mixed_case_with_underscores_and spaces"
+            ),
+            "Mixed Case With Underscores And Spaces",
+        )
+
+    def test_edge_cases(self):
+        # Empty string
+        self.assertEqual(OscDatasetStacGenerator.format_string(""), "")
+        # Single word with trailing underscore
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("temperature_"), "Temperature"
+        )
+        # Single word with leading underscore
+        self.assertEqual(OscDatasetStacGenerator.format_string("_temp"), "Temp")
+        # Single word with leading/trailing spaces
+        self.assertEqual(OscDatasetStacGenerator.format_string("  hello  "), "Hello")
+        # Multiple spaces or underscores
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("too___many___underscores"),
+            "Too Many Underscores",
+        )
+        self.assertEqual(
+            OscDatasetStacGenerator.format_string("too   many   spaces"),
+            "Too Many Spaces",
+        )