datacontract-cli 0.10.19__tar.gz → 0.10.21__tar.gz

{datacontract_cli-0.10.19/datacontract_cli.egg-info → datacontract_cli-0.10.21}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: datacontract-cli
-Version: 0.10.19
+Version: 0.10.21
 Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
 Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 Project-URL: Homepage, https://cli.datacontract.com
@@ -15,8 +15,6 @@ Requires-Dist: typer<0.16,>=0.15.1
 Requires-Dist: pydantic<2.11.0,>=2.8.2
 Requires-Dist: pyyaml~=6.0.1
 Requires-Dist: requests<2.33,>=2.31
-Requires-Dist: fastapi==0.115.6
-Requires-Dist: uvicorn==0.34.0
 Requires-Dist: fastjsonschema<2.22.0,>=2.19.1
 Requires-Dist: fastparquet==2024.11.0
 Requires-Dist: numpy<2.0.0,>=1.26.4
@@ -51,7 +49,7 @@ Provides-Extra: postgres
 Requires-Dist: soda-core-postgres<3.4.0,>=3.3.20; extra == "postgres"
 Provides-Extra: s3
 Requires-Dist: s3fs==2024.12.0; extra == "s3"
-Requires-Dist: aiobotocore<2.18.0,>=2.17.0; extra == "s3"
+Requires-Dist: aiobotocore<2.20.0,>=2.17.0; extra == "s3"
 Provides-Extra: snowflake
 Requires-Dist: snowflake-connector-python[pandas]<3.13,>=3.6; extra == "snowflake"
 Requires-Dist: soda-core-snowflake<3.4.0,>=3.3.20; extra == "snowflake"
@@ -65,13 +63,18 @@ Provides-Extra: dbml
 Requires-Dist: pydbml>=1.1.1; extra == "dbml"
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=18.1.0; extra == "parquet"
+Provides-Extra: api
+Requires-Dist: fastapi==0.115.6; extra == "api"
+Requires-Dist: uvicorn==0.34.0; extra == "api"
+Provides-Extra: custom
+Requires-Dist: Jinja2>=3.1.5; extra == "custom"
 Provides-Extra: all
-Requires-Dist: datacontract-cli[bigquery,csv,databricks,dbml,dbt,iceberg,kafka,parquet,postgres,s3,snowflake,sqlserver,trino]; extra == "all"
+Requires-Dist: datacontract-cli[api,bigquery,csv,custom,databricks,dbml,dbt,iceberg,kafka,parquet,postgres,s3,snowflake,sqlserver,trino]; extra == "all"
 Provides-Extra: dev
 Requires-Dist: datacontract-cli[all]; extra == "dev"
 Requires-Dist: httpx==0.28.1; extra == "dev"
 Requires-Dist: kafka-python; extra == "dev"
-Requires-Dist: moto==5.0.26; extra == "dev"
+Requires-Dist: moto==5.0.27; extra == "dev"
 Requires-Dist: pandas>=2.1.0; extra == "dev"
 Requires-Dist: pre-commit<4.1.0,>=3.7.1; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
@@ -304,7 +307,7 @@ Commands
 - [diff](#diff)
 - [catalog](#catalog)
 - [publish](#publish)
-- [serve](#serve)
+- [api](#api)
 
 ### init
 ```
@@ -880,7 +883,7 @@ models:
 │                    vro-idl|sql|sql-query|html|                               │
 │                    go|bigquery|dbml|spark|sqla                               │
 │                    lchemy|data-caterer|dcs|mar                               │
-│                    kdown|iceberg]                                            │
+│                    kdown|iceberg|custom]                                     │
 │    --output        PATH                         Specify the file path where  │
 │                                                 the exported data will be    │
 │                                                 saved. If no path is         │
@@ -903,6 +906,9 @@ models:
 │    --engine        TEXT                         [engine] The engine used for │
 │                                                 great expection run.         │
 │                                                 [default: None]              │
+│    --template      PATH                         [custom] The file path of    │
+│                                                 Jinja template.              │
+│                                                 [default: None]              │
 │    --help                                       Show this message and exit.  │
 ╰──────────────────────────────────────────────────────────────────────────────╯
 ╭─ RDF Options ────────────────────────────────────────────────────────────────╮
@@ -954,6 +960,7 @@ Available export options:
 | `dcs`                | Export to Data Contract Specification in YAML format    | ✅      |
 | `markdown`           | Export to Markdown                                      | ✅      |
 | `iceberg`            | Export to an Iceberg JSON Schema Definition             | partial |
+| `custom`             | Export to Custom format with Jinja                      | ✅      |
 | Missing something?   | Please create an issue on GitHub                        | TBD    |
 
 
@@ -1135,6 +1142,76 @@ to limit your contract export to a single model.
 }
 ```
 
+#### Custom
+
+The export function converts the data contract specification into the custom format with Jinja. You can specify the path to a Jinja template with the `--template` argument, allowing you to output files in any format.
+
+```shell
+datacontract export --format custom --template template.txt datacontract.yaml
+```
+
+##### Jinja variables
+
+You can directly use the Data Contract Specification as template variables.
+
+```shell
+$ cat template.txt
+title: {{ data_contract.info.title }}
+
+$ datacontract export --format custom --template template.txt datacontract.yaml
+title: Orders Latest
+```
+
+##### Example Jinja Templates
+
+###### Customized dbt model
+
+You can export the dbt models containing any logic.
+
+Below is an example of a dbt staging layer that converts a field of `type: timestamp` to a `DATETIME` type with time zone conversion.
+
+template.sql
+
+{% raw %}
+```sql
+{%- for model_name, model in data_contract.models.items() %}
+{#- Export only the first model #}
+{%- if loop.first -%}
+SELECT
+{%- for field_name, field in model.fields.items() %}
+  {%- if field.type == "timestamp" %}
+  DATETIME({{ field_name }}, "Asia/Tokyo") AS {{ field_name }},
+  {%- else %}
+  {{ field_name }} AS {{ field_name }},
+  {%- endif %}
+{%- endfor %}
+FROM
+  {{ "{{" }} ref('{{ model_name }}') {{ "}}" }} 
+{%- endif %}
+{%- endfor %}
+```
+{% endraw %}
+
+command
+
+```shell
+datacontract export --format custom --template template.sql --output output.sql datacontract.yaml
+```
+
+output.sql
+
+```sql
+SELECT
+  order_id AS order_id,
+  DATETIME(order_timestamp, "Asia/Tokyo") AS order_timestamp,
+  order_total AS order_total,
+  customer_id AS customer_id,
+  customer_email_address AS customer_email_address,
+  DATETIME(processed_timestamp, "Asia/Tokyo") AS processed_timestamp,
+FROM
+  {{ ref('orders') }}
+```
+
 ### import
 ```
                                                                                 
@@ -1539,18 +1616,23 @@ datacontract catalog --files "*.odcs.yaml"
 
 ```
 
-### serve
+### api
 ```
-                                                                                
- Usage: datacontract serve [OPTIONS]                                            
-                                                                                
- Start the datacontract web server.                                             
-                                                                                
-╭─ Options ────────────────────────────────────────────────────────────────────╮
-│ --port        INTEGER  Bind socket to this port. [default: 4242]             │
-│ --host        TEXT     Bind socket to this host. [default: 127.0.0.1]        │
-│ --help                 Show this message and exit.                           │
-╰──────────────────────────────────────────────────────────────────────────────╯
+
+ Usage: datacontract api [OPTIONS]   
+                                                                                 
+ Start the datacontract CLI as server application with REST API.                                                                                                                  
+ The OpenAPI documentation as Swagger UI is available on http://localhost:4242. You can execute the commands directly from the Swagger UI.                                        
+ To protect the API, you can set the environment variable DATACONTRACT_CLI_API_KEY to a secret API key. To authenticate, requests must include the header 'x-api-key' with the    
+ correct API key. This is highly recommended, as data contract tests may be subject to SQL injections or leak sensitive information.                                              
+ To connect to servers (such as a Snowflake data source), set the credentials as environment variables as documented in https://cli.datacontract.com/#test                        
+                                                                                                                                                                                  
+╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --port        INTEGER  Bind socket to this port. [default: 4242]                                                                                                               │
+│ --host        TEXT     Bind socket to this host. Hint: For running in docker, set it to 0.0.0.0 [default: 127.0.0.1]                                                           │
+│ --help                 Show this message and exit.                                                                                                                             │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+
 
 ```
 
@@ -1708,7 +1790,7 @@ class CustomExporter(Exporter):
 
 
 # Register the new custom class into factory
-exporter_factory.register_exporter("custom", CustomExporter)
+exporter_factory.register_exporter("custom_exporter", CustomExporter)
 
 
 if __name__ == "__main__":
@@ -1718,7 +1800,7 @@ if __name__ == "__main__":
     )
     # Call export
     result = data_contract.export(
-        export_format="custom", model="orders", server="production", custom_arg="my_custom_arg"
+        export_format="custom_exporter", model="orders", server="production", custom_arg="my_custom_arg"
     )
     print(result)
 

{datacontract_cli-0.10.19 → datacontract_cli-0.10.21}/README.md

@@ -221,7 +221,7 @@ Commands
 - [diff](#diff)
 - [catalog](#catalog)
 - [publish](#publish)
-- [serve](#serve)
+- [api](#api)
 
 ### init
 ```
@@ -797,7 +797,7 @@ models:
 │                    vro-idl|sql|sql-query|html|                               │
 │                    go|bigquery|dbml|spark|sqla                               │
 │                    lchemy|data-caterer|dcs|mar                               │
-│                    kdown|iceberg]                                            │
+│                    kdown|iceberg|custom]                                     │
 │    --output        PATH                         Specify the file path where  │
 │                                                 the exported data will be    │
 │                                                 saved. If no path is         │
@@ -820,6 +820,9 @@ models:
 │    --engine        TEXT                         [engine] The engine used for │
 │                                                 great expection run.         │
 │                                                 [default: None]              │
+│    --template      PATH                         [custom] The file path of    │
+│                                                 Jinja template.              │
+│                                                 [default: None]              │
 │    --help                                       Show this message and exit.  │
 ╰──────────────────────────────────────────────────────────────────────────────╯
 ╭─ RDF Options ────────────────────────────────────────────────────────────────╮
@@ -871,6 +874,7 @@ Available export options:
 | `dcs`                | Export to Data Contract Specification in YAML format    | ✅      |
 | `markdown`           | Export to Markdown                                      | ✅      |
 | `iceberg`            | Export to an Iceberg JSON Schema Definition             | partial |
+| `custom`             | Export to Custom format with Jinja                      | ✅      |
 | Missing something?   | Please create an issue on GitHub                        | TBD    |
 
 
@@ -1052,6 +1056,76 @@ to limit your contract export to a single model.
 }
 ```
 
+#### Custom
+
+The export function converts the data contract specification into the custom format with Jinja. You can specify the path to a Jinja template with the `--template` argument, allowing you to output files in any format.
+
+```shell
+datacontract export --format custom --template template.txt datacontract.yaml
+```
+
+##### Jinja variables
+
+You can directly use the Data Contract Specification as template variables.
+
+```shell
+$ cat template.txt
+title: {{ data_contract.info.title }}
+
+$ datacontract export --format custom --template template.txt datacontract.yaml
+title: Orders Latest
+```
+
+##### Example Jinja Templates
+
+###### Customized dbt model
+
+You can export the dbt models containing any logic.
+
+Below is an example of a dbt staging layer that converts a field of `type: timestamp` to a `DATETIME` type with time zone conversion.
+
+template.sql
+
+{% raw %}
+```sql
+{%- for model_name, model in data_contract.models.items() %}
+{#- Export only the first model #}
+{%- if loop.first -%}
+SELECT
+{%- for field_name, field in model.fields.items() %}
+  {%- if field.type == "timestamp" %}
+  DATETIME({{ field_name }}, "Asia/Tokyo") AS {{ field_name }},
+  {%- else %}
+  {{ field_name }} AS {{ field_name }},
+  {%- endif %}
+{%- endfor %}
+FROM
+  {{ "{{" }} ref('{{ model_name }}') {{ "}}" }} 
+{%- endif %}
+{%- endfor %}
+```
+{% endraw %}
+
+command
+
+```shell
+datacontract export --format custom --template template.sql --output output.sql datacontract.yaml
+```
+
+output.sql
+
+```sql
+SELECT
+  order_id AS order_id,
+  DATETIME(order_timestamp, "Asia/Tokyo") AS order_timestamp,
+  order_total AS order_total,
+  customer_id AS customer_id,
+  customer_email_address AS customer_email_address,
+  DATETIME(processed_timestamp, "Asia/Tokyo") AS processed_timestamp,
+FROM
+  {{ ref('orders') }}
+```
+
 ### import
 ```
                                                                                 
@@ -1456,18 +1530,23 @@ datacontract catalog --files "*.odcs.yaml"
 
 ```
 
-### serve
+### api
 ```
-                                                                                
- Usage: datacontract serve [OPTIONS]                                            
-                                                                                
- Start the datacontract web server.                                             
-                                                                                
-╭─ Options ────────────────────────────────────────────────────────────────────╮
-│ --port        INTEGER  Bind socket to this port. [default: 4242]             │
-│ --host        TEXT     Bind socket to this host. [default: 127.0.0.1]        │
-│ --help                 Show this message and exit.                           │
-╰──────────────────────────────────────────────────────────────────────────────╯
+
+ Usage: datacontract api [OPTIONS]   
+                                                                                 
+ Start the datacontract CLI as server application with REST API.                                                                                                                  
+ The OpenAPI documentation as Swagger UI is available on http://localhost:4242. You can execute the commands directly from the Swagger UI.                                        
+ To protect the API, you can set the environment variable DATACONTRACT_CLI_API_KEY to a secret API key. To authenticate, requests must include the header 'x-api-key' with the    
+ correct API key. This is highly recommended, as data contract tests may be subject to SQL injections or leak sensitive information.                                              
+ To connect to servers (such as a Snowflake data source), set the credentials as environment variables as documented in https://cli.datacontract.com/#test                        
+                                                                                                                                                                                  
+╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --port        INTEGER  Bind socket to this port. [default: 4242]                                                                                                               │
+│ --host        TEXT     Bind socket to this host. Hint: For running in docker, set it to 0.0.0.0 [default: 127.0.0.1]                                                           │
+│ --help                 Show this message and exit.                                                                                                                             │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+
 
 ```
 
@@ -1625,7 +1704,7 @@ class CustomExporter(Exporter):
 
 
 # Register the new custom class into factory
-exporter_factory.register_exporter("custom", CustomExporter)
+exporter_factory.register_exporter("custom_exporter", CustomExporter)
 
 
 if __name__ == "__main__":
@@ -1635,7 +1714,7 @@ if __name__ == "__main__":
     )
     # Call export
     result = data_contract.export(
-        export_format="custom", model="orders", server="production", custom_arg="my_custom_arg"
+        export_format="custom_exporter", model="orders", server="production", custom_arg="my_custom_arg"
     )
     print(result)
 

datacontract_cli-0.10.21/datacontract/api.py

@@ -0,0 +1,253 @@
+import logging
+import os
+from typing import Annotated, Optional
+
+import typer
+from fastapi import Body, Depends, FastAPI, HTTPException, Query, status
+from fastapi.responses import PlainTextResponse
+from fastapi.security.api_key import APIKeyHeader
+
+from datacontract.data_contract import DataContract, ExportFormat
+from datacontract.model.run import Run
+
+DATA_CONTRACT_EXAMPLE_PAYLOAD = """dataContractSpecification: 1.1.0
+id: urn:datacontract:checkout:orders-latest
+info:
+  title: Orders Latest
+  version: 2.0.0
+  owner: Sales Team
+servers:
+  production:
+    type: s3
+    location: s3://datacontract-example-orders-latest/v2/{model}/*.json
+    format: json
+    delimiter: new_line
+models:
+  orders:
+    description: One record per order. Includes cancelled and deleted orders.
+    type: table
+    fields:
+      order_id:
+        type: string
+        primaryKey: true
+      order_timestamp:
+        description: The business timestamp in UTC when the order was successfully registered in the source system and the payment was successful.
+        type: timestamp
+        required: true
+        examples:
+          - "2024-09-09T08:30:00Z"
+      order_total:
+        description: Total amount the smallest monetary unit (e.g., cents).
+        type: long
+        required: true
+        examples:
+          - 9999
+        quality:
+          - type: sql
+            description: 95% of all order total values are expected to be between 10 and 499 EUR.
+            query: |
+              SELECT quantile_cont(order_total, 0.95) AS percentile_95
+              FROM orders
+            mustBeBetween: [1000, 99900]
+      customer_id:
+        description: Unique identifier for the customer.
+        type: text
+        minLength: 10
+        maxLength: 20
+"""
+
+app = FastAPI(
+    docs_url="/",
+    title="Data Contract CLI API",
+    summary="You can use the API to test, export, and lint your data contracts.",
+    license_info={
+        "name": "MIT License",
+        "identifier": "MIT",
+    },
+    contact={"name": "Data Contract CLI", "url": "https://cli.datacontract.com/"},
+    openapi_tags=[
+        {
+            "name": "test",
+            "externalDocs": {
+                "description": "Documentation",
+                "url": "https://cli.datacontract.com/#test",
+            },
+        },
+        {
+            "name": "lint",
+            "externalDocs": {
+                "description": "Documentation",
+                "url": "https://cli.datacontract.com/#lint",
+            },
+        },
+        {
+            "name": "export",
+            "externalDocs": {
+                "description": "Documentation",
+                "url": "https://cli.datacontract.com/#export",
+            },
+        },
+    ],
+)
+
+api_key_header = APIKeyHeader(
+    name="x-api-key",
+    auto_error=False,  # this makes authentication optional
+)
+
+
+def check_api_key(api_key_header: str | None):
+    correct_api_key = os.getenv("DATACONTRACT_CLI_API_KEY")
+    if correct_api_key is None or correct_api_key == "":
+        logging.info("Environment variable DATACONTRACT_CLI_API_KEY is not set. Skip API key check.")
+        return
+    if api_key_header is None or api_key_header == "":
+        logging.info("The API key is missing.")
+        raise HTTPException(
+            status_code=status.HTTP_401_UNAUTHORIZED,
+            detail="Missing API key. Use Header 'x-api-key' to provide the API key.",
+        )
+    if api_key_header != correct_api_key:
+        logging.info("The provided API key is not correct.")
+        raise HTTPException(
+            status_code=status.HTTP_403_FORBIDDEN,
+            detail="The provided API key is not correct.",
+        )
+    logging.info("Request authenticated with API key.")
+    pass
+
+
+@app.post(
+    "/test",
+    tags=["test"],
+    summary="Run data contract tests",
+    description="""
+              Run schema and quality tests. Data Contract CLI connects to the data sources configured in the server section.
+              This usually requires credentials to access the data sources.
+              Credentials must be provided via environment variables when running the web server.
+              POST the data contract YAML as payload.
+            """,
+    responses={
+        401: {
+            "description": "Unauthorized (when an environment variable DATACONTRACT_CLI_API_KEY is configured).",
+            "content": {
+                "application/json": {
+                    "examples": {
+                        "api_key_missing": {
+                            "summary": "API key Missing",
+                            "value": {"detail": "Missing API key. Use Header 'x-api-key' to provide the API key."},
+                        },
+                        "api_key_wrong": {
+                            "summary": "API key Wrong",
+                            "value": {"detail": "The provided API key is not correct."},
+                        },
+                    }
+                }
+            },
+        },
+    },
+    response_model_exclude_none=True,
+    response_model_exclude_unset=True,
+)
+async def test(
+    body: Annotated[
+        str,
+        Body(
+            title="Data Contract YAML",
+            media_type="application/yaml",
+            examples=[DATA_CONTRACT_EXAMPLE_PAYLOAD],
+        ),
+    ],
+    api_key: Annotated[str | None, Depends(api_key_header)] = None,
+    server: Annotated[
+        str | None,
+        Query(
+            example="production",
+            description="The server name to test. Optional, if there is only one server.",
+        ),
+    ] = None,
+) -> Run:
+    check_api_key(api_key)
+    logging.info("Testing data contract...")
+    logging.info(body)
+    return DataContract(data_contract_str=body, server=server).test()
+
+
+@app.post(
+    "/lint",
+    tags=["lint"],
+    summary="Validate that the datacontract.yaml is correctly formatted.",
+    description="""Validate that the datacontract.yaml is correctly formatted.""",
+)
+async def lint(
+    body: Annotated[
+        str,
+        Body(
+            title="Data Contract YAML",
+            media_type="application/yaml",
+            examples=[DATA_CONTRACT_EXAMPLE_PAYLOAD],
+        ),
+    ],
+    schema: Annotated[
+        str | None,
+        Query(
+            example="https://datacontract.com/datacontract.schema.json",
+            description="The schema to use for validation. This must be a URL.",
+        ),
+    ] = None,
+):
+    data_contract = DataContract(data_contract_str=body, schema_location=schema)
+    lint_result = data_contract.lint()
+    return {"result": lint_result.result, "checks": lint_result.checks}
+
+
+@app.post(
+    "/export",
+    tags=["export"],
+    summary="Convert data contract to a specific format.",
+    response_class=PlainTextResponse,
+)
+def export(
+    body: Annotated[
+        str,
+        Body(
+            title="Data Contract YAML",
+            media_type="application/yaml",
+            examples=[DATA_CONTRACT_EXAMPLE_PAYLOAD],
+        ),
+    ],
+    format: Annotated[ExportFormat, typer.Option(help="The export format.")],
+    server: Annotated[
+        str | None,
+        Query(
+            example="production",
+            description="The server name to export. Optional, if there is only one server.",
+        ),
+    ] = None,
+    model: Annotated[
+        str | None,
+        Query(
+            description="Use the key of the model in the data contract yaml file "
+            "to refer to a model, e.g., `orders`, or `all` for all "
+            "models (default).",
+        ),
+    ] = "all",
+    rdf_base: Annotated[
+        Optional[str],
+        typer.Option(help="[rdf] The base URI used to generate the RDF graph.", rich_help_panel="RDF Options"),
+    ] = None,
+    sql_server_type: Annotated[
+        Optional[str],
+        Query(
+            description="[sql] The server type to determine the sql dialect. By default, it uses 'auto' to automatically detect the sql dialect via the specified servers in the data contract.",
+        ),
+    ] = None,
+):
+    result = DataContract(data_contract_str=body, server=server).export(
+        export_format=format,
+        model=model,
+        rdf_base=rdf_base,
+        sql_server_type=sql_server_type,
+    )
+
+    return result