deep-code 0.0.1.dev0__tar.gz

deep_code-0.0.1.dev0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 DeepESDL and Brockmann Consult GmbH
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

deep_code-0.0.1.dev0/PKG-INFO

@@ -0,0 +1,166 @@
+Metadata-Version: 2.2
+Name: deep_code
+Version: 0.0.1.dev0
+Summary: deepesdl earthcode integration utility tool
+Author-email: Tejas Morbagal Harish <tejas.morbagalharish@brockmann-consult.de>
+License: MIT
+Project-URL: Repository, https://github.com/deepesdl/deep-code
+Project-URL: Issues, https://github.com/deepesdl/deep-code/issues
+Project-URL: Changelog, https://github.com/deepesdl/deep-code/blob/main/CHANGES.md
+Keywords: analysis ready data,data science,datacube,xarray,zarr,xcube,stac,FAIR,reproducible workflow,DeepESDL
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: click
+Requires-Dist: fsspec
+Requires-Dist: jsonschema
+Requires-Dist: requests
+Requires-Dist: pandas
+Requires-Dist: pystac
+Requires-Dist: pyyaml
+Requires-Dist: xcube-core
+Requires-Dist: xrlint
+Provides-Extra: dev
+Requires-Dist: black; extra == "dev"
+Requires-Dist: flake8; extra == "dev"
+Requires-Dist: numpy; extra == "dev"
+Requires-Dist: ruff; extra == "dev"
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-cov; extra == "dev"
+Requires-Dist: pytest-recording; extra == "dev"
+
+# deep-code
+
+[![Build Status](https://github.com/deepesdl/deep-code/actions/workflows/unittest-workflow.yaml/badge.svg)](https://github.com/deepesdl/deep-code/actions/workflows/unittest-workflow.yaml)
+[![codecov](https://codecov.io/gh/deepesdl/deep-code/graph/badge.svg?token=47MQXOXWOK)](https://codecov.io/gh/deepesdl/deep-code)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![License](https://img.shields.io/github/license/dcs4cop/xcube-smos)](https://github.com/deepesdl/deep-code/blob/main/LICENSE)
+
+`deep-code` is a lightweight python tool that comprises a command line interface(CLI) 
+and Python API providing utilities that aid integration of DeepESDL datasets, 
+experiments with EarthCODE.
+
+The first release will focus on implementing the publish feature of DeepESDL 
+experiments/workflow as OGC API record and Datasets as an OSC stac collection.
+
+## Setup
+
+## Install
+`deep-code` will be available in PyPI and conda-forge. Till the stable release,
+developers/contributors can follow the below steps to install deep-code.
+
+## Installing from the repository for Developers/Contributors
+
+To install deep-code directly from the git repository, clone the repository, and execute the steps below:
+
+```commandline
+conda env create
+conda activate deep-code
+pip install -e .
+```
+
+This installs all the dependencies of `deep-code` into a fresh conda environment, 
+and installs deep-code from the repository into the same environment.
+
+## Testing
+
+To run the unit test suite:
+
+```commandline
+pytest
+```
+
+To analyze test coverage
+```shell
+pytest --cov=deep-code
+```
+
+To produce an HTML coverage report
+
+```commandline
+pytest --cov-report html --cov=deep-code
+```
+
+## deep_code usage
+
+`deep_code` provides a command-line tool called deep-code, which has several subcommands 
+providing different utility functions.
+Use the --help option with these subcommands to get more details on usage.
+
+###  deep-code publish-product
+
+Publish a dataset which is a result of an experiment to the EarthCODE 
+open-science catalog.
+
+```commandline
+ deep-code publish-dataset /path/to/dataset-config.yaml
+ ```
+
+#### .gitaccess example
+
+```
+github-username: your-git-user
+github-token: personal access token
+```
+
+#### dataset-config.yaml example
+
+```
+dataset_id: hydrology-1D-0.009deg-100x60x60-3.0.2.zarr
+collection_id: hydrology
+osc_themes:
+  - Land
+  - Oceans
+# non-mandatory
+documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/hydrology-1D-0.009deg-100x60x60-3.0.2.zarr/
+access_link: s3://test
+dataset_status: completed
+osc_region: global
+cf_parameter:
+  - name: hydrology
+```
+
+dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 or your team bucket.
+
+### deep-code publish-workflow
+
+Publish a workflow/experiment to the EarthCODE open-science catalog.
+
+```commandline
+deep-code publish-workflow /path/to/workflow-config.yaml
+ ```
+#### workflow-config.yaml example
+
+```
+workflow_id: "4D Med hydrology cube generation"
+properties:
+  title: "Hydrology cube generation recipe"
+  description: "4D Med cube generation"
+  keywords:
+    - Earth Science
+  themes:
+      - Atmosphere
+      - Ocean
+      - Evaporation
+  license: proprietary
+  jupyter_kernel_info:
+    name: deepesdl-xcube-1.7.1
+    python_version: 3.11
+    env_file: https://git/env.yml
+links:
+  - rel: "documentation"
+    type: "application/json"
+    title: "4DMed Hydrology Cube Generation Recipe"
+    href: "https://github.com/deepesdl/cube-gen/tree/main/hydrology/README.md"
+  - rel: "jupyter-notebook"
+    type: "application/json"
+    title: "Workflow Jupyter Notebook"
+    href: "https://github.com/deepesdl/cube-gen/blob/main/hydrology/notebooks/reading_hydrology.ipynb"
+contact:
+  - name: Tejas Morbagal Harish
+    organization: Brockmann Consult GmbH
+    links:
+      - rel: "about"
+        type: "text/html"
+        href: "https://www.brockmann-consult.de/"
+```

deep_code-0.0.1.dev0/README.md

@@ -0,0 +1,135 @@
+# deep-code
+
+[![Build Status](https://github.com/deepesdl/deep-code/actions/workflows/unittest-workflow.yaml/badge.svg)](https://github.com/deepesdl/deep-code/actions/workflows/unittest-workflow.yaml)
+[![codecov](https://codecov.io/gh/deepesdl/deep-code/graph/badge.svg?token=47MQXOXWOK)](https://codecov.io/gh/deepesdl/deep-code)
+[![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
+[![License](https://img.shields.io/github/license/dcs4cop/xcube-smos)](https://github.com/deepesdl/deep-code/blob/main/LICENSE)
+
+`deep-code` is a lightweight python tool that comprises a command line interface(CLI) 
+and Python API providing utilities that aid integration of DeepESDL datasets, 
+experiments with EarthCODE.
+
+The first release will focus on implementing the publish feature of DeepESDL 
+experiments/workflow as OGC API record and Datasets as an OSC stac collection.
+
+## Setup
+
+## Install
+`deep-code` will be available in PyPI and conda-forge. Till the stable release,
+developers/contributors can follow the below steps to install deep-code.
+
+## Installing from the repository for Developers/Contributors
+
+To install deep-code directly from the git repository, clone the repository, and execute the steps below:
+
+```commandline
+conda env create
+conda activate deep-code
+pip install -e .
+```
+
+This installs all the dependencies of `deep-code` into a fresh conda environment, 
+and installs deep-code from the repository into the same environment.
+
+## Testing
+
+To run the unit test suite:
+
+```commandline
+pytest
+```
+
+To analyze test coverage
+```shell
+pytest --cov=deep-code
+```
+
+To produce an HTML coverage report
+
+```commandline
+pytest --cov-report html --cov=deep-code
+```
+
+## deep_code usage
+
+`deep_code` provides a command-line tool called deep-code, which has several subcommands 
+providing different utility functions.
+Use the --help option with these subcommands to get more details on usage.
+
+###  deep-code publish-product
+
+Publish a dataset which is a result of an experiment to the EarthCODE 
+open-science catalog.
+
+```commandline
+ deep-code publish-dataset /path/to/dataset-config.yaml
+ ```
+
+#### .gitaccess example
+
+```
+github-username: your-git-user
+github-token: personal access token
+```
+
+#### dataset-config.yaml example
+
+```
+dataset_id: hydrology-1D-0.009deg-100x60x60-3.0.2.zarr
+collection_id: hydrology
+osc_themes:
+  - Land
+  - Oceans
+# non-mandatory
+documentation_link: https://deepesdl.readthedocs.io/en/latest/datasets/hydrology-1D-0.009deg-100x60x60-3.0.2.zarr/
+access_link: s3://test
+dataset_status: completed
+osc_region: global
+cf_parameter:
+  - name: hydrology
+```
+
+dataset-id has to be a valid dataset-id from `deep-esdl-public` s3 or your team bucket.
+
+### deep-code publish-workflow
+
+Publish a workflow/experiment to the EarthCODE open-science catalog.
+
+```commandline
+deep-code publish-workflow /path/to/workflow-config.yaml
+ ```
+#### workflow-config.yaml example
+
+```
+workflow_id: "4D Med hydrology cube generation"
+properties:
+  title: "Hydrology cube generation recipe"
+  description: "4D Med cube generation"
+  keywords:
+    - Earth Science
+  themes:
+      - Atmosphere
+      - Ocean
+      - Evaporation
+  license: proprietary
+  jupyter_kernel_info:
+    name: deepesdl-xcube-1.7.1
+    python_version: 3.11
+    env_file: https://git/env.yml
+links:
+  - rel: "documentation"
+    type: "application/json"
+    title: "4DMed Hydrology Cube Generation Recipe"
+    href: "https://github.com/deepesdl/cube-gen/tree/main/hydrology/README.md"
+  - rel: "jupyter-notebook"
+    type: "application/json"
+    title: "Workflow Jupyter Notebook"
+    href: "https://github.com/deepesdl/cube-gen/blob/main/hydrology/notebooks/reading_hydrology.ipynb"
+contact:
+  - name: Tejas Morbagal Harish
+    organization: Brockmann Consult GmbH
+    links:
+      - rel: "about"
+        type: "text/html"
+        href: "https://www.brockmann-consult.de/"
+```

deep_code-0.0.1.dev0/deep_code/__init__.py

@@ -0,0 +1,24 @@
+# The MIT License (MIT)
+# Copyright (c) 2024 by DeepESDL and Brockmann Consult GmbH
+#
+# Permission is hereby granted, free of charge, to any person obtaining a
+# copy of this software and associated documentation files (the "Software"),
+# to deal in the Software without restriction, including without limitation
+# the rights to use, copy, modify, merge, publish, distribute, sublicense,
+# and/or sell copies of the Software, and to permit persons to whom the
+# Software is furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+# FITNESS FOR A PARTICULAR PURPOSE AND NON INFRINGEMENT. IN NO EVENT SHALL THE
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
+# DEALINGS IN THE SOFTWARE.
+
+from .version import version
+
+__version__ = version

deep_code-0.0.1.dev0/deep_code/cli/__init__.py

@@ -0,0 +1,3 @@
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.

deep_code-0.0.1.dev0/deep_code/cli/main.py

@@ -0,0 +1,22 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import click
+
+from deep_code.cli.publish import publish_dataset, publish_workflow
+
+
+@click.group()
+def main():
+    """Deep Code CLI."""
+    pass
+
+
+main.add_command(publish_dataset)
+main.add_command(publish_workflow)
+
+if __name__ == "__main__":
+    main()

deep_code-0.0.1.dev0/deep_code/cli/publish.py

@@ -0,0 +1,26 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+import click
+
+from deep_code.tools.publish import DatasetPublisher, WorkflowPublisher
+
+
+@click.command(name="publish-dataset")
+@click.argument("dataset_config", type=click.Path(exists=True))
+def publish_dataset(dataset_config):
+    """Request publishing a dataset to the open science catalogue.
+    """
+    publisher = DatasetPublisher()
+    publisher.publish_dataset(dataset_config_path=dataset_config)
+
+
+@click.command(name="publish-workflow")
+@click.argument("workflow_metadata", type=click.Path(exists=True))
+def publish_workflow(workflow_metadata):
+
+    workflow_publisher = WorkflowPublisher()
+    workflow_publisher.publish_workflow(workflow_config_path=workflow_metadata)

deep_code-0.0.1.dev0/deep_code/constants.py

@@ -0,0 +1,16 @@
+#!/usr/bin/env python3
+
+# Copyright (c) 2024 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.
+
+OSC_SCHEMA_URI = "https://stac-extensions.github.io/osc/v1.0.0-rc.3/schema.json"
+CF_SCHEMA_URI = "https://stac-extensions.github.io/cf/v0.2.0/schema.json"
+OSC_REPO_OWNER = "ESA-EarthCODE"
+OSC_REPO_NAME = "open-science-catalog-metadata-testing"
+OSC_BRANCH_NAME = "add-new-collection"
+DEFAULT_THEME_SCHEME = (
+    "https://gcmd.earthdata.nasa.gov/kms/concepts/concept_scheme/sciencekeywords"
+)
+OGC_API_RECORD_SPEC = "http://www.opengis.net/spec/ogcapi-records-1/1.0/req/record-core"
+WF_BRANCH_NAME = "add-new-workflow-from-deepesdl"

deep_code-0.0.1.dev0/deep_code/tests/tools/__init__.py

@@ -0,0 +1,3 @@
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.

deep_code-0.0.1.dev0/deep_code/tests/tools/test_publish.py

@@ -0,0 +1,120 @@
+from unittest.mock import MagicMock, mock_open, patch
+
+import pytest
+
+from deep_code.tools.publish import DatasetPublisher
+
+
+class TestDatasetPublisher:
+    @patch("deep_code.tools.publish.fsspec.open")
+    def test_init_missing_credentials(self, mock_fsspec_open):
+        mock_fsspec_open.return_value.__enter__.return_value = mock_open(
+            read_data="{}"
+        )()
+
+        with pytest.raises(
+            ValueError, match="GitHub credentials are missing in the `.gitaccess` file."
+        ):
+            DatasetPublisher()
+
+    @patch("deep_code.tools.publish.fsspec.open")
+    def test_publish_dataset_missing_ids(self, mock_fsspec_open):
+        git_yaml_content = """
+        github-username: test-user
+        github-token: test-token
+        """
+        dataset_yaml_content = """
+        collection-id: test-collection
+        """
+        mock_fsspec_open.side_effect = [
+            mock_open(read_data=git_yaml_content)(),
+            mock_open(read_data=dataset_yaml_content)(),
+        ]
+
+        publisher = DatasetPublisher()
+
+        with pytest.raises(
+            ValueError, match="Dataset ID or Collection ID missing in the config."
+        ):
+            publisher.publish_dataset("/path/to/dataset-config.yaml")
+
+    @patch("deep_code.utils.github_automation.os.chdir")
+    @patch("deep_code.utils.github_automation.subprocess.run")
+    @patch("deep_code.utils.github_automation.os.path.expanduser", return_value="/tmp")
+    @patch("requests.post")
+    @patch("deep_code.utils.github_automation.GitHubAutomation")
+    @patch("deep_code.tools.publish.fsspec.open")
+    def test_publish_dataset_success(
+        self,
+        mock_fsspec_open,
+        mock_github_automation,
+        mock_requests_post,
+        mock_expanduser,
+        mock_subprocess_run,
+        mock_chdir,
+    ):
+        #  Mock the YAML reads
+        git_yaml_content = """
+             github-username: test-user
+             github-token: test-token
+             """
+        dataset_yaml_content = """
+             dataset_id: test-dataset
+             collection_id: test-collection
+             documentation_link: http://example.com/doc
+             access_link: http://example.com/access
+             dataset_status: ongoing
+             dataset_region: Global
+             osc_theme: ["climate"]
+             cf_parameter: []
+             """
+        mock_fsspec_open.side_effect = [
+            mock_open(read_data=git_yaml_content)(),
+            mock_open(read_data=dataset_yaml_content)(),
+        ]
+
+        # Mock GitHubAutomation methods
+        mock_git = mock_github_automation.return_value
+        mock_git.fork_repository.return_value = None
+        mock_git.clone_repository.return_value = None
+        mock_git.create_branch.return_value = None
+        mock_git.add_file.return_value = None
+        mock_git.commit_and_push.return_value = None
+        mock_git.create_pull_request.return_value = "http://example.com/pr"
+        mock_git.clean_up.return_value = None
+
+        # Mock subprocess.run & os.chdir
+        mock_subprocess_run.return_value = None
+        mock_chdir.return_value = None
+
+        # Mock STAC generator
+        mock_collection = MagicMock()
+        mock_collection.to_dict.return_value = {
+            "type": "Collection",
+            "id": "test-collection",
+            "description": "A test STAC collection",
+            "extent": {
+                "spatial": {"bbox": [[-180.0, -90.0, 180.0, 90.0]]},
+                "temporal": {"interval": [["2023-01-01T00:00:00Z", None]]},
+            },
+            "links": [],
+            "stac_version": "1.0.0",
+        }
+        with patch("deep_code.tools.publish.OscDatasetStacGenerator") as mock_generator:
+            mock_generator.return_value.build_dataset_stac_collection.return_value = (
+                mock_collection
+            )
+
+            # Instantiate & publish
+            publisher = DatasetPublisher()
+            publisher.publish_dataset("/fake/path/to/dataset-config.yaml")
+
+        # Assert that we called git clone with /tmp/temp_repo
+        # Because expanduser("~") is now patched to /tmp, the actual path is /tmp/temp_repo
+        auth_url = "https://test-user:test-token@github.com/test-user/open-science-catalog-metadata-testing.git"
+        mock_subprocess_run.assert_any_call(
+            ["git", "clone", auth_url, "/tmp/temp_repo"], check=True
+        )
+
+        # Also confirm we changed directories to /tmp/temp_repo
+        mock_chdir.assert_any_call("/tmp/temp_repo")

deep_code-0.0.1.dev0/deep_code/tests/utils/__init__.py

@@ -0,0 +1,3 @@
+# Copyright (c) 2025 by Brockmann Consult GmbH
+# Permissions are hereby granted under the terms of the MIT License:
+# https://opensource.org/licenses/MIT.