dataproduct-cli 0.0.1__tar.gz

dataproduct_cli-0.0.1/CHANGELOG.md

@@ -0,0 +1,16 @@
+# Changelog
+
+All notable changes to this project are documented here. Each entry is one line:
+what changed (user-facing).
+
+## [Unreleased]
+
+## [0.0.1]
+
+- Test release to validate the PyPI publishing pipeline
+
+## [0.1.0]
+
+- `init` command: create a valid `dataproduct.odps.yaml` from a bundled ODPS v1.0.0 template
+- `lint` command: validate a data product against the ODPS v1.0.0 JSON Schema (json/junit output)
+- `publish` command: publish a data product to Entropy Data (`PUT /api/dataproducts/{id}`)

dataproduct_cli-0.0.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Data Contract
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

dataproduct_cli-0.0.1/MANIFEST.in

@@ -0,0 +1,3 @@
+include README.md LICENSE CHANGELOG.md
+recursive-include dataproduct/schemas *.json *.yaml
+include dataproduct/py.typed

dataproduct_cli-0.0.1/PKG-INFO

@@ -0,0 +1,126 @@
+Metadata-Version: 2.4
+Name: dataproduct-cli
+Version: 0.0.1
+Summary: The dataproduct CLI is an open source command-line tool for working with data products. It uses data product YAML files following the Open Data Product Standard (ODPS) to lint the definition against the standard and publish it to Entropy Data. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
+Author-email: Michael Kutz <michael.kutz@entropy-data.com>
+License-Expression: MIT
+Project-URL: Homepage, https://github.com/datacontract/dataproduct-cli
+Project-URL: Repository, https://github.com/datacontract/dataproduct-cli
+Project-URL: Issues, https://github.com/datacontract/dataproduct-cli/issues
+Keywords: data-product,data-products,odps,open-data-product-standard,data-governance,data-mesh,data-engineering
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Information Technology
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Database
+Classifier: Topic :: Software Development :: Quality Assurance
+Requires-Python: <3.15,>=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: typer<0.28,>=0.18.0
+Requires-Dist: click<9.0.0,>=8.1.0
+Requires-Dist: pydantic<2.14.0,>=2.8.2
+Requires-Dist: pyyaml~=6.0.1
+Requires-Dist: requests<2.35,>=2.31
+Requires-Dist: jsonschema<5.0.0,>=4.23.0
+Requires-Dist: rich<16.0,>=13.7
+Requires-Dist: python-dotenv<2.0.0,>=1.0.0
+Requires-Dist: typing-extensions>=4.7.0
+Provides-Extra: dev
+Requires-Dist: pytest; extra == "dev"
+Requires-Dist: pytest-xdist; extra == "dev"
+Requires-Dist: ruff==0.16.1; extra == "dev"
+Requires-Dist: pre-commit<4.7.0,>=3.7.1; extra == "dev"
+Dynamic: license-file
+
+# Data Product CLI
+
+The `dataproduct` CLI is an open-source command-line tool for working with
+**data products** defined with the
+[Open Data Product Standard (ODPS)](https://bitol-io.github.io/open-data-product-standard/v1.0.0/).
+
+It is the data-product sibling of
+[`datacontract-cli`](https://github.com/datacontract/datacontract-cli) (which
+targets the Open Data **Contract** Standard) and mirrors its technology and
+release mechanism. The tool is written in Python and can be used standalone, in
+CI/CD, or as a Python library.
+
+> The behavior of each command is specified in [`specs/`](specs/).
+
+## Install
+
+```bash
+uv tool install --python python3.11 dataproduct-cli
+# or
+pip install dataproduct-cli
+```
+
+## Commands
+
+### `init` — create a data product
+
+```bash
+dataproduct init                       # writes ./dataproduct.odps.yaml
+dataproduct init my.odps.yaml          # custom path
+dataproduct init --overwrite           # replace an existing file
+dataproduct init --template <url|path> # seed from a template or existing product
+```
+
+### `lint` — validate against the ODPS standard
+
+```bash
+dataproduct lint                                   # lints ./dataproduct.odps.yaml
+dataproduct lint my.odps.yaml
+dataproduct lint --all-errors                      # report every schema violation
+dataproduct lint --output-format junit --output TEST-dataproduct.xml
+dataproduct lint --json-schema ./odps.schema.json  # validate against a custom schema
+```
+
+Validation is schema-only in 0.1: the data product is checked against the
+bundled ODPS v1.0.0 JSON Schema. Exit code is `0` when valid, `1` otherwise.
+
+### `publish` — publish to Entropy Data
+
+```bash
+export ENTROPY_DATA_API_KEY=...        # from Entropy Data → Settings → API Keys
+dataproduct publish my.odps.yaml
+```
+
+`publish` sends `PUT {host}/api/dataproducts/{id}` with an `x-api-key` header.
+The host defaults to `https://api.entropy-data.com` and can be overridden with
+`ENTROPY_DATA_HOST`. The referenced `team` must already exist in your org.
+
+## Configuration
+
+| Purpose | Variable (with fallbacks) |
+|---|---|
+| API host | `ENTROPY_DATA_HOST` → `DATAMESH_MANAGER_HOST` → `DATACONTRACT_MANAGER_HOST` → `https://api.entropy-data.com` |
+| API key | `ENTROPY_DATA_API_KEY` → `DATAMESH_MANAGER_API_KEY` → `DATACONTRACT_MANAGER_API_KEY` |
+
+`.env` files are loaded automatically.
+
+## Use as a library
+
+```python
+from dataproduct.data_product import DataProduct
+
+run = DataProduct(data_product_file="dataproduct.odps.yaml").lint()
+assert run.result == "passed"
+
+DataProduct(data_product_file="dataproduct.odps.yaml").publish()
+```
+
+## Development
+
+See [AGENTS.md](AGENTS.md).
+
+## License
+
+MIT

dataproduct_cli-0.0.1/README.md

@@ -0,0 +1,84 @@
+# Data Product CLI
+
+The `dataproduct` CLI is an open-source command-line tool for working with
+**data products** defined with the
+[Open Data Product Standard (ODPS)](https://bitol-io.github.io/open-data-product-standard/v1.0.0/).
+
+It is the data-product sibling of
+[`datacontract-cli`](https://github.com/datacontract/datacontract-cli) (which
+targets the Open Data **Contract** Standard) and mirrors its technology and
+release mechanism. The tool is written in Python and can be used standalone, in
+CI/CD, or as a Python library.
+
+> The behavior of each command is specified in [`specs/`](specs/).
+
+## Install
+
+```bash
+uv tool install --python python3.11 dataproduct-cli
+# or
+pip install dataproduct-cli
+```
+
+## Commands
+
+### `init` — create a data product
+
+```bash
+dataproduct init                       # writes ./dataproduct.odps.yaml
+dataproduct init my.odps.yaml          # custom path
+dataproduct init --overwrite           # replace an existing file
+dataproduct init --template <url|path> # seed from a template or existing product
+```
+
+### `lint` — validate against the ODPS standard
+
+```bash
+dataproduct lint                                   # lints ./dataproduct.odps.yaml
+dataproduct lint my.odps.yaml
+dataproduct lint --all-errors                      # report every schema violation
+dataproduct lint --output-format junit --output TEST-dataproduct.xml
+dataproduct lint --json-schema ./odps.schema.json  # validate against a custom schema
+```
+
+Validation is schema-only in 0.1: the data product is checked against the
+bundled ODPS v1.0.0 JSON Schema. Exit code is `0` when valid, `1` otherwise.
+
+### `publish` — publish to Entropy Data
+
+```bash
+export ENTROPY_DATA_API_KEY=...        # from Entropy Data → Settings → API Keys
+dataproduct publish my.odps.yaml
+```
+
+`publish` sends `PUT {host}/api/dataproducts/{id}` with an `x-api-key` header.
+The host defaults to `https://api.entropy-data.com` and can be overridden with
+`ENTROPY_DATA_HOST`. The referenced `team` must already exist in your org.
+
+## Configuration
+
+| Purpose | Variable (with fallbacks) |
+|---|---|
+| API host | `ENTROPY_DATA_HOST` → `DATAMESH_MANAGER_HOST` → `DATACONTRACT_MANAGER_HOST` → `https://api.entropy-data.com` |
+| API key | `ENTROPY_DATA_API_KEY` → `DATAMESH_MANAGER_API_KEY` → `DATACONTRACT_MANAGER_API_KEY` |
+
+`.env` files are loaded automatically.
+
+## Use as a library
+
+```python
+from dataproduct.data_product import DataProduct
+
+run = DataProduct(data_product_file="dataproduct.odps.yaml").lint()
+assert run.result == "passed"
+
+DataProduct(data_product_file="dataproduct.odps.yaml").publish()
+```
+
+## Development
+
+See [AGENTS.md](AGENTS.md).
+
+## License
+
+MIT

dataproduct_cli-0.0.1/dataproduct/__init__.py

@@ -0,0 +1,5 @@
+"""dataproduct-cli — a CLI for the Open Data Product Standard (ODPS).
+
+Sister project of datacontract-cli. See the ``specs/`` directory for the
+behavioral specification of each command.
+"""

dataproduct_cli-0.0.1/dataproduct/cli.py

@@ -0,0 +1,81 @@
+import logging
+import sys
+from importlib import metadata
+from pathlib import Path
+from typing import Iterable, Optional
+
+import typer
+from click import Context
+from dotenv import find_dotenv, load_dotenv
+from rich.console import Console
+from typer.core import TyperGroup
+from typing_extensions import Annotated
+
+from dataproduct.output.output_format import OutputFormat
+
+console = Console()
+
+debug_option = Annotated[bool, typer.Option(help="Enable debug logging")]
+
+# Order in which top-level commands appear in `dataproduct --help`.
+COMMAND_ORDER = ["init", "lint", "publish"]
+
+
+class OrderedCommands(TyperGroup):
+    def list_commands(self, ctx: Context) -> Iterable[str]:
+        known = set(COMMAND_ORDER)
+        return [c for c in COMMAND_ORDER if c in self.commands] + [c for c in self.commands if c not in known]
+
+
+app = typer.Typer(
+    cls=OrderedCommands,
+    no_args_is_help=True,
+    add_completion=False,
+    pretty_exceptions_show_locals=False,
+    help="CLI for data products following the Open Data Product Standard (ODPS).",
+)
+
+
+def version_callback(value: bool) -> None:
+    if value:
+        try:
+            version = metadata.version("dataproduct-cli")
+        except metadata.PackageNotFoundError:
+            version = "0.0.0"
+        console.print(version)
+        raise typer.Exit()
+
+
+@app.callback()
+def common(
+    ctx: typer.Context,
+    version: Annotated[
+        Optional[bool],
+        typer.Option("--version", callback=version_callback, is_eager=True, help="Print the version and exit."),
+    ] = None,
+) -> None:
+    """dataproduct CLI."""
+    load_dotenv(find_dotenv(usecwd=True))
+
+
+def enable_debug_logging(debug: bool, otherwise_disable_stderr: bool = False) -> None:
+    if debug:
+        logging.basicConfig(level=logging.DEBUG, stream=sys.stderr, format="%(asctime)s %(levelname)s %(message)s")
+    elif otherwise_disable_stderr:
+        logging.disable(logging.CRITICAL)
+
+
+def resolve_output_format(output_format: Optional[OutputFormat], output: Optional[Path]) -> Optional[OutputFormat]:
+    if output_format is not None:
+        return output_format
+    if output is not None:
+        return OutputFormat.junit if str(output).endswith(".xml") else OutputFormat.json
+    return None
+
+
+# Register the commands (each module attaches itself to `app`).
+from dataproduct import command_init, command_lint, command_publish  # noqa: E402, F401
+
+
+def main() -> None:
+    app()

dataproduct_cli-0.0.1/dataproduct/command_init.py

@@ -0,0 +1,33 @@
+import os
+
+import typer
+from typing_extensions import Annotated
+
+from dataproduct.cli import app, console, debug_option, enable_debug_logging
+from dataproduct.init.init_template import get_init_template
+
+
+@app.command(
+    name="init",
+    epilog="Example: dataproduct init dataproduct.odps.yaml",
+)
+def init(
+    location: Annotated[
+        str, typer.Argument(help="The location of the data product file to create.")
+    ] = "dataproduct.odps.yaml",
+    template: Annotated[str, typer.Option(help="URL or path of a template or data product")] = None,
+    overwrite: Annotated[bool, typer.Option(help="Replace the existing data product file")] = False,
+    debug: debug_option = None,
+):
+    """
+    Create a new data product file.
+    """
+    enable_debug_logging(debug)
+
+    if not overwrite and os.path.exists(location):
+        console.print("File already exists, use --overwrite to overwrite")
+        raise typer.Exit(code=1)
+    template_str = get_init_template(template)
+    with open(location, "w") as f:
+        f.write(template_str)
+    console.print("📄 data product written to " + location)

dataproduct_cli-0.0.1/dataproduct/command_lint.py

@@ -0,0 +1,58 @@
+from pathlib import Path
+
+import typer
+from typing_extensions import Annotated
+
+from dataproduct.cli import app, console, debug_option, enable_debug_logging, resolve_output_format
+from dataproduct.config import cli_config
+from dataproduct.data_product import DataProduct
+from dataproduct.output.output_format import OutputFormat
+from dataproduct.output.result_writer import write_result
+
+
+@app.command(
+    name="lint",
+    epilog="Example: dataproduct lint dataproduct.odps.yaml",
+)
+def lint(
+    location: Annotated[
+        str,
+        typer.Argument(help="The location (url or local path) of the data product yaml."),
+    ] = "dataproduct.odps.yaml",
+    schema: Annotated[
+        str,
+        typer.Option("--json-schema", help="The location (url or path) of the ODPS JSON Schema"),
+    ] = None,
+    output: Annotated[
+        Path,
+        typer.Option(
+            help="File path to write the results to (e.g. './TEST-dataproduct.xml'). "
+            "If omitted, results are printed to stdout."
+        ),
+    ] = None,
+    output_format: Annotated[
+        OutputFormat,
+        typer.Option(help="The result format. Accepted values: json, junit."),
+    ] = None,
+    all_errors: Annotated[
+        bool,
+        typer.Option(
+            "--all-errors",
+            help="Report all JSON Schema validation errors instead of stopping after the first one.",
+        ),
+    ] = False,
+    debug: debug_option = None,
+):
+    """
+    Validate that the data product is correctly formatted (against the ODPS JSON Schema).
+    """
+    enable_debug_logging(debug, otherwise_disable_stderr=True)
+
+    output_format = resolve_output_format(output_format, output)
+    run = DataProduct(
+        config=cli_config(),
+        data_product_file=location,
+        schema_location=schema,
+        all_errors=all_errors,
+    ).lint()
+    write_result(run, console, output_format, output)

dataproduct_cli-0.0.1/dataproduct/command_publish.py

@@ -0,0 +1,47 @@
+import typer
+from typing_extensions import Annotated
+
+from dataproduct.cli import app, console, debug_option, enable_debug_logging
+from dataproduct.config import cli_config
+from dataproduct.integration.entropy_data import (
+    DataProductPublishError,
+    publish_data_product_to_entropy_data,
+)
+from dataproduct.lint.resolve import resolve_data_product_dict
+from dataproduct.model.exceptions import DataProductException
+
+
+@app.command(
+    name="publish",
+    epilog="Example: dataproduct publish dataproduct.odps.yaml",
+)
+def publish(
+    location: Annotated[
+        str,
+        typer.Argument(help="The location (url or local path) of the data product yaml."),
+    ] = "dataproduct.odps.yaml",
+    schema: Annotated[
+        str,
+        typer.Option("--json-schema", help="The location (url or path) of the ODPS JSON Schema"),
+    ] = None,
+    ssl_verification: Annotated[
+        bool,
+        typer.Option(help="SSL verification when publishing the data product."),
+    ] = True,
+    debug: debug_option = None,
+):
+    """
+    Publish the data product to Entropy Data.
+    """
+    enable_debug_logging(debug)
+
+    try:
+        data_product_dict = resolve_data_product_dict(location, config=cli_config())
+        publish_data_product_to_entropy_data(
+            data_product_dict=data_product_dict,
+            ssl_verification=ssl_verification,
+            config=cli_config(),
+        )
+    except (DataProductPublishError, DataProductException) as e:
+        console.print(f"[red]Failed publishing data product. Error: {e}[/red]")
+        raise typer.Exit(code=1)

dataproduct_cli-0.0.1/dataproduct/config/__init__.py

@@ -0,0 +1,67 @@
+"""Programmatic configuration for hosts and API keys.
+
+Every value the CLI reads from ``ENTROPY_DATA_*`` (and the legacy
+``DATAMESH_MANAGER_*`` / ``DATACONTRACT_MANAGER_*``) environment variables can
+also be provided programmatically via :class:`Config` (or a plain dict keyed by
+the env var names). Reads fall back to the process environment.
+"""
+
+import os
+from typing import Optional, Union
+
+
+class Config:
+    def __init__(self, values: Optional[dict] = None):
+        self._values = dict(values or {})
+
+    def getenv(self, key: str) -> Optional[str]:
+        value = self._values.get(key)
+        if value is not None:
+            return value
+        return os.environ.get(key)
+
+    @classmethod
+    def resolve(cls, config: "Optional[Union[Config, dict]]" = None) -> "Config":
+        if config is None:
+            return cls()
+        if isinstance(config, Config):
+            return config
+        if isinstance(config, dict):
+            return cls(config)
+        raise TypeError(f"Unsupported config type: {type(config)!r}")
+
+    # Hosts
+    def get_entropy_data_host(self) -> Optional[str]:
+        return self.getenv("ENTROPY_DATA_HOST")
+
+    def get_datamesh_manager_host(self) -> Optional[str]:
+        return self.getenv("DATAMESH_MANAGER_HOST")
+
+    def get_datacontract_manager_host(self) -> Optional[str]:
+        return self.getenv("DATACONTRACT_MANAGER_HOST")
+
+    # API keys
+    def get_entropy_data_api_key(self) -> Optional[str]:
+        return self.getenv("ENTROPY_DATA_API_KEY")
+
+    def get_datamesh_manager_api_key(self) -> Optional[str]:
+        return self.getenv("DATAMESH_MANAGER_API_KEY")
+
+    def get_datacontract_manager_api_key(self) -> Optional[str]:
+        return self.getenv("DATACONTRACT_MANAGER_API_KEY")
+
+
+# Simple process-wide "current" config set from the CLI entry point.
+_cli_config = Config()
+
+
+def set_cli_config(config: Config) -> None:
+    global _cli_config
+    _cli_config = config
+
+
+def cli_config() -> Config:
+    return _cli_config
+
+
+__all__ = ["Config", "cli_config", "set_cli_config"]

dataproduct_cli-0.0.1/dataproduct/data_product.py

@@ -0,0 +1,93 @@
+"""Core library entry point, parallel to datacontract-cli's ``DataContract``.
+
+from dataproduct.data_product import DataProduct
+
+run = DataProduct(data_product_file="dataproduct.odps.yaml").lint()
+DataProduct(data_product_file="dataproduct.odps.yaml").publish()
+"""
+
+from typing import Optional, Union
+
+from dataproduct.config import Config
+from dataproduct.integration.entropy_data import publish_data_product_to_entropy_data
+from dataproduct.lint.files import read_resource
+from dataproduct.lint.schema import fetch_schema
+from dataproduct.lint.validate import parse_yaml, validate_against_schema
+from dataproduct.model.exceptions import DataProductException
+from dataproduct.model.run import Check, ResultEnum, Run
+
+
+class DataProduct:
+    def __init__(
+        self,
+        data_product_file: Optional[str] = None,
+        data_product_str: Optional[str] = None,
+        schema_location: Optional[str] = None,
+        all_errors: bool = False,
+        config: "Optional[Union[Config, dict]]" = None,
+    ):
+        self._data_product_file = data_product_file
+        self._data_product_str = data_product_str
+        self._schema_location = schema_location
+        self._all_errors = all_errors
+        self._config = Config.resolve(config)
+
+    def _load_dict(self) -> dict:
+        if self._data_product_file is not None:
+            content = read_resource(self._data_product_file, self._config)
+        elif self._data_product_str is not None:
+            content = self._data_product_str
+        else:
+            raise DataProductException(
+                type="lint",
+                name="Load data product",
+                reason="No data product provided (file or string).",
+            )
+        return parse_yaml(content)
+
+    def lint(self) -> Run:
+        """Validate the data product against the ODPS JSON Schema (schema-only)."""
+        run = Run.create_run()
+        run.log_info("Linting data product")
+        try:
+            data = self._load_dict()
+            run.dataProductId = data.get("id")
+            run.dataProductVersion = data.get("version")
+            schema = fetch_schema(self._schema_location)
+            checks = validate_against_schema(data, schema, self._all_errors)
+            if checks:
+                run.checks.extend(checks)
+                for check in checks:
+                    run.log_error(str(check.reason))
+            else:
+                run.checks.append(
+                    Check(
+                        type="lint",
+                        result=ResultEnum.passed,
+                        name="Data product is syntactically valid",
+                    )
+                )
+        except DataProductException as e:
+            run.checks.append(Check(type=e.type, result=e.result, name=e.name, reason=e.reason, engine=e.engine))
+            run.log_error(str(e))
+        except Exception as e:
+            run.checks.append(
+                Check(
+                    type="general",
+                    result=ResultEnum.error,
+                    name="Check Data Product",
+                    reason=str(e),
+                )
+            )
+            run.log_error(str(e))
+        run.finish()
+        return run
+
+    def publish(self, ssl_verification: bool = True) -> None:
+        """Publish the data product to Entropy Data (no client-side lint in 0.1)."""
+        data = self._load_dict()
+        publish_data_product_to_entropy_data(
+            data_product_dict=data,
+            ssl_verification=ssl_verification,
+            config=self._config,
+        )

dataproduct_cli-0.0.1/dataproduct/init/init_template.py

@@ -0,0 +1,26 @@
+import importlib.resources as resources
+import logging
+
+import requests
+
+DEFAULT_DATA_PRODUCT_INIT_TEMPLATE = "odps-1.0.0.init.yaml"
+
+
+def get_init_template(location: str = None) -> str:
+    """Return the contents of an init template.
+
+    - ``None`` -> the bundled default template.
+    - an ``http(s)://`` URL -> the fetched body.
+    - anything else -> read as a local file path.
+    """
+    if location is None:
+        logging.info("Use default bundled template " + DEFAULT_DATA_PRODUCT_INIT_TEMPLATE)
+        schemas = resources.files("dataproduct")
+        template = schemas.joinpath("schemas", DEFAULT_DATA_PRODUCT_INIT_TEMPLATE)
+        with template.open("r") as file:
+            return file.read()
+    elif location.startswith("http://") or location.startswith("https://"):
+        return requests.get(location).text
+    else:
+        with open(location, "r") as file:
+            return file.read()