datacontract-cli 0.10.10__tar.gz → 0.10.11__tar.gz

{datacontract_cli-0.10.10 → datacontract_cli-0.10.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: datacontract-cli
-Version: 0.10.10
+Version: 0.10.11
 Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
 Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 Project-URL: Homepage, https://cli.datacontract.com
@@ -19,7 +19,7 @@ Requires-Dist: fastapi==0.111.1
 Requires-Dist: fastparquet==2024.5.0
 Requires-Dist: python-multipart==0.0.9
 Requires-Dist: rich~=13.7.0
-Requires-Dist: simple-ddl-parser==1.5.1
+Requires-Dist: simple-ddl-parser==1.5.2
 Requires-Dist: soda-core-duckdb<3.4.0,>=3.3.1
 Requires-Dist: setuptools>=60
 Requires-Dist: duckdb==1.0.0
@@ -37,7 +37,7 @@ Provides-Extra: bigquery
 Requires-Dist: soda-core-bigquery<3.4.0,>=3.3.1; extra == "bigquery"
 Provides-Extra: databricks
 Requires-Dist: soda-core-spark-df<3.4.0,>=3.3.1; extra == "databricks"
-Requires-Dist: databricks-sql-connector<3.3.0,>=3.1.2; extra == "databricks"
+Requires-Dist: databricks-sql-connector<3.4.0,>=3.1.2; extra == "databricks"
 Requires-Dist: soda-core-spark[databricks]<3.4.0,>=3.3.1; extra == "databricks"
 Provides-Extra: deltalake
 Requires-Dist: deltalake<0.19,>=0.17; extra == "deltalake"
@@ -61,19 +61,19 @@ Provides-Extra: dev
 Requires-Dist: datacontract-cli[all]; extra == "dev"
 Requires-Dist: httpx==0.27.0; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
-Requires-Dist: pre-commit~=3.7.1; extra == "dev"
+Requires-Dist: pre-commit<3.9.0,>=3.7.1; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
 Requires-Dist: moto==5.0.11; extra == "dev"
 Requires-Dist: pymssql==2.3.0; extra == "dev"
 Requires-Dist: kafka-python; extra == "dev"
 Requires-Dist: trino==0.329.0; extra == "dev"
-Requires-Dist: testcontainers<4.8,>=4.5; extra == "dev"
-Requires-Dist: testcontainers[core]; extra == "dev"
-Requires-Dist: testcontainers[minio]; extra == "dev"
-Requires-Dist: testcontainers[postgres]; extra == "dev"
-Requires-Dist: testcontainers[kafka]; extra == "dev"
-Requires-Dist: testcontainers[mssql]; extra == "dev"
+Requires-Dist: testcontainers==4.7.2; extra == "dev"
+Requires-Dist: testcontainers[core]==4.7.2; extra == "dev"
+Requires-Dist: testcontainers[minio]==4.7.2; extra == "dev"
+Requires-Dist: testcontainers[postgres]==4.7.2; extra == "dev"
+Requires-Dist: testcontainers[kafka]==4.7.2; extra == "dev"
+Requires-Dist: testcontainers[mssql]==4.7.2; extra == "dev"
 
 # Data Contract CLI
 
@@ -197,10 +197,10 @@ $ datacontract export --format html datacontract.yaml > datacontract.html
 # import avro (other formats: sql, glue, bigquery...)
 $ datacontract import --format avro --source avro_schema.avsc
 
-# find differences between to data contracts
+# find differences between two data contracts
 $ datacontract diff datacontract-v1.yaml datacontract-v2.yaml
 
-# find differences between to data contracts categorized into error, warning, and info.
+# find differences between two data contracts categorized into error, warning, and info.
 $ datacontract changelog datacontract-v1.yaml datacontract-v2.yaml
 
 # fail pipeline on breaking changes. Uses changelog internally and showing only error and warning.
@@ -385,7 +385,7 @@ Supported server types:
 - [sqlserver](#sqlserver)
 - [databricks](#databricks)
 - [databricks (programmatic)](#databricks-programmatic)
-- [dataframr (programmatic)](#dataframe-programmatic)
+- [dataframe (programmatic)](#dataframe-programmatic)
 - [snowflake](#snowflake)
 - [kafka](#kafka)
 - [postgres](#postgres)
@@ -665,14 +665,31 @@ models:
 ```
 
 #### Environment Variables
-
-| Environment Variable               | Example            | Description                                         |
-|------------------------------------|--------------------|-----------------------------------------------------|
-| `DATACONTRACT_SNOWFLAKE_USERNAME`  | `datacontract`     | Username                                            |
-| `DATACONTRACT_SNOWFLAKE_PASSWORD`  | `mysecretpassword` | Password                                            |
-| `DATACONTRACT_SNOWFLAKE_ROLE`      | `DATAVALIDATION`   | The snowflake role to use.                          |
-| `DATACONTRACT_SNOWFLAKE_WAREHOUSE` | `COMPUTE_WH`       | The Snowflake Warehouse to use executing the tests. |
-
+All [parameters supported by Soda](https://docs.soda.io/soda/connect-snowflake.html), uppercased and prepended by `DATACONTRACT_SNOWFLAKE_` prefix.  
+For example:
+
+| Soda parameter       | Environment Variable                        | 
+|----------------------|---------------------------------------------|
+| `username`           | `DATACONTRACT_SNOWFLAKE_USERNAME`           |
+| `password`           | `DATACONTRACT_SNOWFLAKE_PASSWORD`           |
+| `warehouse`          | `DATACONTRACT_SNOWFLAKE_WAREHOUSE`          |
+| `role`               | `DATACONTRACT_SNOWFLAKE_ROLE`               |
+| `connection_timeout` | `DATACONTRACT_SNOWFLAKE_CONNECTION_TIMEOUT` |
+
+Beware, that parameters:
+* `account`
+* `database`
+* `schema`
+
+are obtained from the `servers` section of the YAML-file.  
+E.g. from the example above:
+```yaml
+servers:
+  snowflake:
+    account: abcdefg-xn12345
+    database: ORDER_DB
+    schema: ORDERS_PII_V2
+```
 
 
 ### Kafka
@@ -925,23 +942,34 @@ models:
 ```
  Usage: datacontract import [OPTIONS]
 
- Create a data contract from the given source location. Prints to stdout.
-
-╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *  --format                  [sql|avro|glue|bigquery|jsonschema|  The format of the source file. [default: None] [required]         |
-│                               unity|spark]                                                                                          |
-│    --source                  TEXT                                 The path to the file or Glue Database that should be imported.    │
-│                                                                   [default: None]                                                   │
-│    --glue-table              TEXT                                 List of table ids to import from the Glue Database (repeat for    │
-│                                                                   multiple table ids, leave empty for all tables in the dataset).   │
-│                                                                   [default: None]                                                   │
-│    --bigquery-project        TEXT                                 The bigquery project id. [default: None]                          │
-│    --bigquery-dataset        TEXT                                 The bigquery dataset id. [default: None]                          │
-│    --bigquery-table          TEXT                                 List of table ids to import from the bigquery API (repeat for     │
-│                                                                   multiple table ids, leave empty for all tables in the dataset).   │
-│                                                                   [default: None]                                                   │
-│    --help                                                         Show this message and exit.                                       │
-╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+ Create a data contract from the given source location. Prints to stdout.                                                      
+                                                                                                                               
+╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *  --format                       [sql|avro|dbt|glue|jsonschema|bigquery|odcs  The format of the source file.               │
+│                                   |unity|spark]                                [default: None]                              │
+│                                                                                [required]                                   │
+│    --source                       TEXT                                         The path to the file or Glue Database that   │
+│                                                                                should be imported.                          │
+│                                                                                [default: None]                              │
+│    --glue-table                   TEXT                                         List of table ids to import from the Glue    │
+│                                                                                Database (repeat for multiple table ids,     │
+│                                                                                leave empty for all tables in the dataset).  │
+│                                                                                [default: None]                              │
+│    --bigquery-project             TEXT                                         The bigquery project id. [default: None]     │
+│    --bigquery-dataset             TEXT                                         The bigquery dataset id. [default: None]     │
+│    --bigquery-table               TEXT                                         List of table ids to import from the         │
+│                                                                                bigquery API (repeat for multiple table ids, │
+│                                                                                leave empty for all tables in the dataset).  │
+│                                                                                [default: None]                              │
+│    --unity-table-full-name        TEXT                                         Full name of a table in the unity catalog    │
+│                                                                                [default: None]                              │
+│    --dbt-model                    TEXT                                         List of models names to import from the dbt  │
+│                                                                                manifest file (repeat for multiple models    │
+│                                                                                names, leave empty for all models in the     │
+│                                                                                dataset).                                    │
+│                                                                                [default: None]                              │
+│    --help                                                                      Show this message and exit.                  │
+╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
 Example:
@@ -952,18 +980,19 @@ datacontract import --format sql --source my_ddl.sql
 
 Available import options:
 
-| Type               | Description                                    | Status  |
-|--------------------|------------------------------------------------|---------|
+| Type               | Description                                    | Status |
+|--------------------|------------------------------------------------|--------|
 | `sql`              | Import from SQL DDL                            | ✅      |
 | `avro`             | Import from AVRO schemas                       | ✅      |
 | `glue`             | Import from AWS Glue DataCatalog               | ✅      |
-| `protobuf`         | Import from Protobuf schemas                   | TBD     |
 | `jsonschema`       | Import from JSON Schemas                       | ✅      |
 | `bigquery`         | Import from BigQuery Schemas                   | ✅      |
 | `unity`            | Import from Databricks Unity Catalog           | partial |
-| `dbt`              | Import from dbt models                         | TBD     |
+| `dbt`              | Import from dbt models                         | ✅      |
 | `odcs`             | Import from Open Data Contract Standard (ODCS) | ✅      |
-| Missing something? | Please create an issue on GitHub               | TBD     |
+| `spark`            | Import from Spark StructTypes                  | ✅      |
+| `protobuf`         | Import from Protobuf schemas                   | TBD    |
+| Missing something? | Please create an issue on GitHub               | TBD    |
 
 
 #### BigQuery
@@ -1005,6 +1034,23 @@ export DATABRICKS_IMPORT_ACCESS_TOKEN=<token>
 datacontract import --format unity --unity-table-full-name <table_full_name>
 ```
 
+#### dbt
+
+Importing from dbt manifest file.
+You may give the `dbt-model` parameter to enumerate the tables that should be imported. If no tables are given, _all_ available tables of the database will be imported.
+
+Examples:
+
+```bash
+# Example import from dbt manifest with specifying the tables to import
+datacontract import --format dbt --source <manifest_path> --dbt-model <model_name_1> --dbt-model <model_name_2> --dbt-model <model_name_3>
+```
+
+```bash
+# Example import from dbt manifest importing all tables in the database
+datacontract import --format dbt --source <manifest_path>
+```
+
 #### Glue
 
 Importing from Glue reads the necessary Data directly off of the AWS API.
@@ -1304,7 +1350,7 @@ if __name__ == "__main__":
     data_contract = DataContract(
         data_contract_file="/path/datacontract.yaml"
     )
-    # call export
+    # Call export
     result = data_contract.export(
         export_format="custom", model="orders", server="production", custom_arg="my_custom_arg"
     )
@@ -1330,10 +1376,11 @@ Output
 Using the importer factory to add a new custom importer
 ```python
 
-from datacontract.model.data_contract_specification import DataContractSpecification
+from datacontract.model.data_contract_specification import DataContractSpecification, Field, Model
 from datacontract.data_contract import DataContract
 from datacontract.imports.importer import Importer
 from datacontract.imports.importer_factory import importer_factory
+
 import json
 
 # Create a custom class that implements import_source method
@@ -1344,43 +1391,89 @@ class CustomImporter(Importer):
         source_dict = json.loads(source)
         data_contract_specification.id = source_dict.get("id_custom")
         data_contract_specification.info.title = source_dict.get("title")
+        data_contract_specification.info.version = source_dict.get("version")
         data_contract_specification.info.description = source_dict.get("description_from_app")
-
+        
+        for model in source_dict.get("models", []):
+            fields = {}
+            for column in model.get('columns'):
+                field = Field(
+                    description=column.get('column_description'), 
+                    type=column.get('type') 
+                )
+                fields[column.get('name')] = field               
+                   
+            dc_model = Model(
+                description=model.get('description'), 
+                fields= fields
+            )
+
+            data_contract_specification.models[model.get('name')] = dc_model
         return data_contract_specification
-
+ 
 
 # Register the new custom class into factory
 importer_factory.register_importer("custom_company_importer", CustomImporter)
 
 
 if __name__ == "__main__":
-    # get a custom da
-    json_from_custom_app = '{"id_custom":"uuid-custom","version":"0.0.2", "title":"my_custom_imported_data", "description_from_app": "Custom contract description"}'
+    # Get a custom data from other app 
+    json_from_custom_app = '''
+    {
+        "id_custom": "uuid-custom",
+        "version": "0.0.2",
+        "title": "my_custom_imported_data",
+        "description_from_app": "Custom contract description",
+        "models": [
+            {
+            "name": "model1",
+            "desctiption": "model description from app",
+            "columns": [
+                {
+                "name": "columnA",
+                "type": "varchar",
+                "column_description": "my_column description"
+                },
+                {
+                "name": "columnB",
+                "type": "varchar",
+                "column_description": "my_columnB description"
+                }
+            ]
+            }
+        ]
+        }
+    '''
     # Create a DataContract instance
     data_contract = DataContract()
 
-    # call import_from
+    # Call import_from_source
     result = data_contract.import_from_source(
-        format="custom_company_importer", data_contract_specification=DataContract.init(), source=json_from_custom_app
-    )
-    print(dict(result))
-
+        format="custom_company_importer", 
+        data_contract_specification=DataContract.init(), 
+        source=json_from_custom_app
+    ) 
+    print(result.to_yaml() )
 ```
 Output
+  
+```yaml
+dataContractSpecification: 0.9.3
+id: uuid-custom
+info:
+  title: my_custom_imported_data
+  version: 0.0.2
+  description: Custom contract description
+models:
+  model1:
+    fields:
+      columnA:
+        type: varchar
+        description: my_column description
+      columnB:
+        type: varchar
+        description: my_columnB description
 
-```python
-{
-  'dataContractSpecification': '0.9.3', 
-  'id': 'uuid-custom', 
-  'info': Info(title='my_custom_imported_data', version='0.0.1', status=None, description='Custom contract description', owner=None, contact=None), 
-  'servers': {}, 
-  'terms': None, 
-  'models': {}, 
-  'definitions': {}, 
-  'examples': [], 
-  'quality': None, 
-  'servicelevels': None
-}
 ```
 ## Development Setup
 
@@ -1469,6 +1562,7 @@ We are happy to receive your contributions. Propose your change in an issue or d
 ## Companies using this tool
 
 - [INNOQ](https://innoq.com)
+- [Data Catering](https://data.catering/)
 - And many more. To add your company, please create a pull request.
 
 ## Related Tools
@@ -1476,6 +1570,7 @@ We are happy to receive your contributions. Propose your change in an issue or d
 - [Data Contract Manager](https://www.datacontract-manager.com/) is a commercial tool to manage data contracts. It contains a web UI, access management, and data governance for a full enterprise data marketplace.
 - [Data Contract GPT](https://gpt.datacontract.com) is a custom GPT that can help you write data contracts.
 - [Data Contract Editor](https://editor.datacontract.com) is an editor for Data Contracts, including a live html preview.
+- [Data Contract Playground](https://data-catering.github.io/data-contract-playground/) allows you to validate and export your data contract to different formats within your browser.
 
 ## License