datacontract-cli 0.10.21__tar.gz → 0.10.22__tar.gz

{datacontract_cli-0.10.21/datacontract_cli.egg-info → datacontract_cli-0.10.22}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.2
 Name: datacontract-cli
-Version: 0.10.21
+Version: 0.10.22
 Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
 Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 Project-URL: Homepage, https://cli.datacontract.com
@@ -20,13 +20,13 @@ Requires-Dist: fastparquet==2024.11.0
 Requires-Dist: numpy<2.0.0,>=1.26.4
 Requires-Dist: python-multipart==0.0.20
 Requires-Dist: rich<13.10,>=13.7
-Requires-Dist: simple-ddl-parser==1.7.1
+Requires-Dist: sqlglot<27.0.0,>=26.6.0
 Requires-Dist: duckdb==1.1.2
-Requires-Dist: soda-core-duckdb<3.4.0,>=3.3.20
+Requires-Dist: soda-core-duckdb<3.5.0,>=3.3.20
 Requires-Dist: setuptools>=60
 Requires-Dist: python-dotenv~=1.0.0
-Requires-Dist: rdflib==7.0.0
-Requires-Dist: boto3<1.35.98,>=1.34.41
+Requires-Dist: boto3<1.36.12,>=1.34.41
+Requires-Dist: Jinja2>=3.1.5
 Requires-Dist: jinja_partials>=0.2.1
 Provides-Extra: avro
 Requires-Dist: avro==1.12.0; extra == "avro"
@@ -51,8 +51,8 @@ Provides-Extra: s3
 Requires-Dist: s3fs==2024.12.0; extra == "s3"
 Requires-Dist: aiobotocore<2.20.0,>=2.17.0; extra == "s3"
 Provides-Extra: snowflake
-Requires-Dist: snowflake-connector-python[pandas]<3.13,>=3.6; extra == "snowflake"
-Requires-Dist: soda-core-snowflake<3.4.0,>=3.3.20; extra == "snowflake"
+Requires-Dist: snowflake-connector-python[pandas]<3.14,>=3.6; extra == "snowflake"
+Requires-Dist: soda-core-snowflake<3.5.0,>=3.3.20; extra == "snowflake"
 Provides-Extra: sqlserver
 Requires-Dist: soda-core-sqlserver<3.4.0,>=3.3.20; extra == "sqlserver"
 Provides-Extra: trino
@@ -63,13 +63,13 @@ Provides-Extra: dbml
 Requires-Dist: pydbml>=1.1.1; extra == "dbml"
 Provides-Extra: parquet
 Requires-Dist: pyarrow>=18.1.0; extra == "parquet"
+Provides-Extra: rdf
+Requires-Dist: rdflib==7.0.0; extra == "rdf"
 Provides-Extra: api
 Requires-Dist: fastapi==0.115.6; extra == "api"
 Requires-Dist: uvicorn==0.34.0; extra == "api"
-Provides-Extra: custom
-Requires-Dist: Jinja2>=3.1.5; extra == "custom"
 Provides-Extra: all
-Requires-Dist: datacontract-cli[api,bigquery,csv,custom,databricks,dbml,dbt,iceberg,kafka,parquet,postgres,s3,snowflake,sqlserver,trino]; extra == "all"
+Requires-Dist: datacontract-cli[api,bigquery,csv,databricks,dbml,dbt,iceberg,kafka,parquet,postgres,rdf,s3,snowflake,sqlserver,trino]; extra == "all"
 Provides-Extra: dev
 Requires-Dist: datacontract-cli[all]; extra == "dev"
 Requires-Dist: httpx==0.28.1; extra == "dev"
@@ -194,17 +194,20 @@ $ datacontract init datacontract.yaml
 # lint the datacontract.yaml
 $ datacontract lint datacontract.yaml
 
-# execute schema and quality checks
+# execute schema and quality checks (define credentials as environment variables)
 $ datacontract test datacontract.yaml
 
-# execute schema and quality checks on the examples within the contract
-$ datacontract test --examples datacontract.yaml
+# export data contract as html (other formats: avro, dbt, dbt-sources, dbt-staging-sql, jsonschema, odcs, rdf, sql, sodacl, terraform, ...)
+$ datacontract export --format html datacontract.yaml --output datacontract.html
 
-# export data contract as html (other formats: avro, dbt, dbt-sources, dbt-staging-sql, jsonschema, odcs_v2, odcs_v3, rdf, sql, sodacl, terraform, ...)
-$ datacontract export --format html datacontract.yaml > datacontract.html
+# export data contract to ODCS
+$ datacontract export --format odcs datacontract.yaml --output odcs.yaml
 
-# import avro (other formats: sql, glue, bigquery...)
-$ datacontract import --format avro --source avro_schema.avsc
+# import ODCS to data contract
+$ datacontract import --format odcs odcs.yaml --output datacontract.yaml
+
+# import sql (other formats: avro, glue, bigquery, jsonschema ...)
+$ datacontract import --format sql --source my-ddl.sql --dialect postgres --output datacontract.yaml
 
 # find differences between two data contracts
 $ datacontract diff datacontract-v1.yaml datacontract-v2.yaml
@@ -237,12 +240,28 @@ Python 3.10, 3.11, and 3.12 are supported. We recommend to use Python 3.11.
 
 ```bash
 python3 -m pip install 'datacontract-cli[all]'
+datacontract --version
+```
+
+### pip with venv
+
+Typically it is better to install the application in a virtual environment for your projects:
+
+```bash
+cd my-project
+python3.11 -m venv venv
+source venv/bin/activate
+pip install 'datacontract-cli[all]'
+datacontract --version
 ```
 
 ### pipx
+
 pipx installs into an isolated environment.
+
 ```bash
 pipx install 'datacontract-cli[all]'
+datacontract --version
 ```
 
 ### Docker
@@ -264,7 +283,7 @@ _Note:_ The output of Docker command line messages is limited to 80 columns and
 
 
 
-## Optional Dependencies
+## Optional Dependencies (Extras)
 
 The CLI tool defines several optional dependencies (also known as extras) that can be installed for using with specific servers types.
 With _all_, all server dependencies are included.
@@ -275,21 +294,23 @@ pip install datacontract-cli[all]
 
 A list of available extras:
 
-| Dependency             | Installation Command                       |
-|------------------------|--------------------------------------------|
-| Avro Support           | `pip install datacontract-cli[avro]`       |
-| Google BigQuery        | `pip install datacontract-cli[bigquery]`   |
-| Databricks Integration | `pip install datacontract-cli[databricks]` |
-| Iceberg                | `pip install datacontract-cli[iceberg]`    |
-| Kafka Integration      | `pip install datacontract-cli[kafka]`      |
-| PostgreSQL Integration | `pip install datacontract-cli[postgres]`   |
-| S3 Integration         | `pip install datacontract-cli[s3]`         |
-| Snowflake Integration  | `pip install datacontract-cli[snowflake]`  |
-| Microsoft SQL Server   | `pip install datacontract-cli[sqlserver]`  |
-| Trino                  | `pip install datacontract-cli[trino]`      |
-| Dbt                    | `pip install datacontract-cli[dbt]`        |
-| Dbml                   | `pip install datacontract-cli[dbml]`       |
-| Parquet                | `pip install datacontract-cli[parquet]`    |
+| Dependency              | Installation Command                       |
+|-------------------------|--------------------------------------------|
+| Avro Support            | `pip install datacontract-cli[avro]`       |
+| Google BigQuery         | `pip install datacontract-cli[bigquery]`   |
+| Databricks Integration  | `pip install datacontract-cli[databricks]` |
+| Iceberg                 | `pip install datacontract-cli[iceberg]`    |
+| Kafka Integration       | `pip install datacontract-cli[kafka]`      |
+| PostgreSQL Integration  | `pip install datacontract-cli[postgres]`   |
+| S3 Integration          | `pip install datacontract-cli[s3]`         |
+| Snowflake Integration   | `pip install datacontract-cli[snowflake]`  |
+| Microsoft SQL Server    | `pip install datacontract-cli[sqlserver]`  |
+| Trino                   | `pip install datacontract-cli[trino]`      |
+| dbt                     | `pip install datacontract-cli[dbt]`        |
+| DBML                    | `pip install datacontract-cli[dbml]`       |
+| Parquet                 | `pip install datacontract-cli[parquet]`    |
+| RDF                     | `pip install datacontract-cli[rdf]`        |
+| API (run as web server) | `pip install datacontract-cli[api]`        |
 
 
 
@@ -386,11 +407,6 @@ Commands
 │                                                      for all servers         │
 │                                                      (default).              │
 │                                                      [default: all]          │
-│ --examples            --no-examples                  Run the schema and      │
-│                                                      quality tests on the    │
-│                                                      example data within the │
-│                                                      data contract.          │
-│                                                      [default: no-examples]  │
 │ --publish                                      TEXT  The url to publish the  │
 │                                                      results after the test  │
 │                                                      [default: None]         │
@@ -877,8 +893,8 @@ models:
 ╭─ Options ────────────────────────────────────────────────────────────────────╮
 │ *  --format        [jsonschema|pydantic-model|  The export format.           │
 │                    sodacl|dbt|dbt-sources|dbt-  [default: None]              │
-│                    staging-sql|odcs|odcs_v2|od  [required]                   │
-│                    cs_v3|rdf|avro|protobuf|gre                               │
+│                    staging-sql|odcs|            [required]                   │
+│                    rdf|avro|protobuf|gre                                     │
 │                    at-expectations|terraform|a                               │
 │                    vro-idl|sql|sql-query|html|                               │
 │                    go|bigquery|dbml|spark|sqla                               │
@@ -936,8 +952,6 @@ Available export options:
 |----------------------|---------------------------------------------------------|--------|
 | `html`               | Export to HTML                                          | ✅      |
 | `jsonschema`         | Export to JSON Schema                                   | ✅      |
-| `odcs_v2`            | Export to Open Data Contract Standard (ODCS) V2         | ✅      |
-| `odcs_v3`            | Export to Open Data Contract Standard (ODCS) V3         | ✅      |
 | `odcs`               | Export to Open Data Contract Standard (ODCS) V3         | ✅      |
 | `sodacl`             | Export to SodaCL quality checks in YAML format          | ✅      |
 | `dbt`                | Export to dbt models in YAML format                     | ✅      |
@@ -1214,107 +1228,80 @@ FROM
 
 ### import
 ```
-                                                                                
- Usage: datacontract import [OPTIONS]                                           
-                                                                                
- Create a data contract from the given source location. Saves to file specified 
- by `output` option if present, otherwise prints to stdout.                     
-                                                                                
-╭─ Options ────────────────────────────────────────────────────────────────────╮
-│ *  --format                     [sql|avro|dbt|dbml|gl  The format of the     │
-│                                 ue|jsonschema|bigquer  source file.          │
-│                                 y|odcs|unity|spark|ic  [default: None]       │
-│                                 eberg|parquet|csv]     [required]            │
-│    --output                     PATH                   Specify the file path │
-│                                                        where the Data        │
-│                                                        Contract will be      │
-│                                                        saved. If no path is  │
-│                                                        provided, the output  │
-│                                                        will be printed to    │
-│                                                        stdout.               │
-│                                                        [default: None]       │
-│    --source                     TEXT                   The path to the file  │
-│                                                        or Glue Database that │
-│                                                        should be imported.   │
-│                                                        [default: None]       │
-│    --glue-table                 TEXT                   List of table ids to  │
-│                                                        import from the Glue  │
-│                                                        Database (repeat for  │
-│                                                        multiple table ids,   │
-│                                                        leave empty for all   │
-│                                                        tables in the         │
-│                                                        dataset).             │
-│                                                        [default: None]       │
-│    --bigquery-project           TEXT                   The bigquery project  │
-│                                                        id.                   │
-│                                                        [default: None]       │
-│    --bigquery-dataset           TEXT                   The bigquery dataset  │
-│                                                        id.                   │
-│                                                        [default: None]       │
-│    --bigquery-table             TEXT                   List of table ids to  │
-│                                                        import from the       │
-│                                                        bigquery API (repeat  │
-│                                                        for multiple table    │
-│                                                        ids, leave empty for  │
-│                                                        all tables in the     │
-│                                                        dataset).             │
-│                                                        [default: None]       │
-│    --unity-table-full-n…        TEXT                   Full name of a table  │
-│                                                        in the unity catalog  │
-│                                                        [default: None]       │
-│    --dbt-model                  TEXT                   List of models names  │
-│                                                        to import from the    │
-│                                                        dbt manifest file     │
-│                                                        (repeat for multiple  │
-│                                                        models names, leave   │
-│                                                        empty for all models  │
-│                                                        in the dataset).      │
-│                                                        [default: None]       │
-│    --dbml-schema                TEXT                   List of schema names  │
-│                                                        to import from the    │
-│                                                        DBML file (repeat for │
-│                                                        multiple schema       │
-│                                                        names, leave empty    │
-│                                                        for all tables in the │
-│                                                        file).                │
-│                                                        [default: None]       │
-│    --dbml-table                 TEXT                   List of table names   │
-│                                                        to import from the    │
-│                                                        DBML file (repeat for │
-│                                                        multiple table names, │
-│                                                        leave empty for all   │
-│                                                        tables in the file).  │
-│                                                        [default: None]       │
-│    --iceberg-table              TEXT                   Table name to assign  │
-│                                                        to the model created  │
-│                                                        from the Iceberg      │
-│                                                        schema.               │
-│                                                        [default: None]       │
-│    --template                   TEXT                   The location (url or  │
-│                                                        path) of the Data     │
-│                                                        Contract              │
-│                                                        Specification         │
-│                                                        Template              │
-│                                                        [default: None]       │
-│    --schema                     TEXT                   The location (url or  │
-│                                                        path) of the Data     │
-│                                                        Contract              │
-│                                                        Specification JSON    │
-│                                                        Schema                │
-│                                                        [default:             │
-│                                                        https://datacontract… │
-│    --help                                              Show this message and │
-│                                                        exit.                 │
-╰──────────────────────────────────────────────────────────────────────────────╯
+ Usage: datacontract import [OPTIONS]                                                                             
+                                                                                                                  
+ Create a data contract from the given source location. Saves to file specified by `output` option if present,    
+ otherwise prints to stdout.                                                                                      
+                                                                                                                  
+╭─ Options ──────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *  --format                       [sql|avro|dbt|dbml|glue|jsonschema|bi  The format of the source file.        │
+│                                   gquery|odcs|unity|spark|iceberg|parqu  [default: None]                       │
+│                                   et|csv]                                [required]                            │
+│    --output                       PATH                                   Specify the file path where the Data  │
+│                                                                          Contract will be saved. If no path is │
+│                                                                          provided, the output will be printed  │
+│                                                                          to stdout.                            │
+│                                                                          [default: None]                       │
+│    --source                       TEXT                                   The path to the file or Glue Database │
+│                                                                          that should be imported.              │
+│                                                                          [default: None]                       │
+│    --dialect                      TEXT                                   The SQL dialect to use when importing │
+│                                                                          SQL files, e.g., postgres, tsql,      │
+│                                                                          bigquery.                             │
+│                                                                          [default: None]                       │
+│    --glue-table                   TEXT                                   List of table ids to import from the  │
+│                                                                          Glue Database (repeat for multiple    │
+│                                                                          table ids, leave empty for all tables │
+│                                                                          in the dataset).                      │
+│                                                                          [default: None]                       │
+│    --bigquery-project             TEXT                                   The bigquery project id.              │
+│                                                                          [default: None]                       │
+│    --bigquery-dataset             TEXT                                   The bigquery dataset id.              │
+│                                                                          [default: None]                       │
+│    --bigquery-table               TEXT                                   List of table ids to import from the  │
+│                                                                          bigquery API (repeat for multiple     │
+│                                                                          table ids, leave empty for all tables │
+│                                                                          in the dataset).                      │
+│                                                                          [default: None]                       │
+│    --unity-table-full-name        TEXT                                   Full name of a table in the unity     │
+│                                                                          catalog                               │
+│                                                                          [default: None]                       │
+│    --dbt-model                    TEXT                                   List of models names to import from   │
+│                                                                          the dbt manifest file (repeat for     │
+│                                                                          multiple models names, leave empty    │
+│                                                                          for all models in the dataset).       │
+│                                                                          [default: None]                       │
+│    --dbml-schema                  TEXT                                   List of schema names to import from   │
+│                                                                          the DBML file (repeat for multiple    │
+│                                                                          schema names, leave empty for all     │
+│                                                                          tables in the file).                  │
+│                                                                          [default: None]                       │
+│    --dbml-table                   TEXT                                   List of table names to import from    │
+│                                                                          the DBML file (repeat for multiple    │
+│                                                                          table names, leave empty for all      │
+│                                                                          tables in the file).                  │
+│                                                                          [default: None]                       │
+│    --iceberg-table                TEXT                                   Table name to assign to the model     │
+│                                                                          created from the Iceberg schema.      │
+│                                                                          [default: None]                       │
+│    --template                     TEXT                                   The location (url or path) of the     │
+│                                                                          Data Contract Specification Template  │
+│                                                                          [default: None]                       │
+│    --schema                       TEXT                                   The location (url or path) of the     │
+│                                                                          Data Contract Specification JSON      │
+│                                                                          Schema                                │
+│                                                                          [default: None]                       │
+│    --help                                                                Show this message and exit.           │
+╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 
 ```
 
 Example:
 ```bash
 # Example import from SQL DDL
-datacontract import --format sql --source my_ddl.sql
+datacontract import --format sql --source my_ddl.sql --dialect postgres
 # To save to file
-datacontract import --format sql --source my_ddl.sql --output datacontract.yaml
+datacontract import --format sql --source my_ddl.sql --dialect postgres --output datacontract.yaml
 ```
 
 Available import options:
@@ -1669,24 +1656,20 @@ Create a data contract based on the actual data. This is the fastest way to get
     $ datacontract test
     ```
 
-2. Add examples to the `datacontract.yaml`. If you can, use actual data and anonymize. Make sure that the examples match the imported logical data model.
+2. Add quality checks and additional type constraints one by one to the contract and make sure the
+   data still adheres to the contract.
    ```bash
-   $ datacontract test --examples
-   ```
-
-
-3. Add quality checks and additional type constraints one by one to the contract and make sure the examples and the actual data still adheres to the contract. Check against examples for a very fast feedback loop.
-   ```bash
-   $ datacontract test --examples
    $ datacontract test
    ```
 
-4. Make sure that all the best practices for a `datacontract.yaml` are met using the linter. You probably forgot to document some fields and add the terms and conditions.
+3. Make sure that all the best practices for a `datacontract.yaml` are met using the linter. You
+   probably forgot to document some fields and add the terms and conditions.
    ```bash
    $ datacontract lint
    ```
 
-5. Set up a CI pipeline that executes daily for continuous quality checks. You can also report the test results to tools like [Data Mesh Manager](https://datamesh-manager.com) 
+4. Set up a CI pipeline that executes daily for continuous quality checks. You can also report the
+   test results to tools like [Data Mesh Manager](https://datamesh-manager.com)
    ```bash
    $ datacontract test --publish https://api.datamesh-manager.com/api/test-results
    ```
@@ -1700,30 +1683,15 @@ Create a data contract based on the requirements from use cases.
    $ datacontract init
    ```
 
-2. Add examples to the `datacontract.yaml`. Do not start with the data model, although you are probably tempted to do that. Examples are the fastest way to get feedback from everybody and not loose someone in the discussion.
-
-3. Create the model based on the examples. Test the model against the examples to double-check whether the model matches the examples.
-    ```bash
-    $ datacontract test --examples
-    ```
-
-4. Add quality checks and additional type constraints one by one to the contract and make sure the examples and the actual data still adheres to the contract. Check against examples for a very fast feedback loop.
-    ```bash
-    $ datacontract test --examples
-    ```
-
-5. Fill in the terms, descriptions, etc. Make sure you follow all best practices for a `datacontract.yaml` using the linter.
+2. Create the model and quality guarantees based on your business requirements. Fill in the terms,
+   descriptions, etc. Make sure you follow all best practices for a `datacontract.yaml` using the
+   linter.
     ```bash
     $ datacontract lint
     ```
 
-6. Set up a CI pipeline that lints and tests the examples so you make sure that any changes later do not decrease the quality of the contract.
-    ```bash
-    $ datacontract lint
-    $ datacontract test --examples
-    ```
-
-7. Use the export function to start building the providing data product as well as the integration into the consuming data products.
+3. Use the export function to start building the providing data product as well as the integration
+   into the consuming data products.
     ```bash
     # data provider
     $ datacontract export --format dbt
@@ -1732,6 +1700,11 @@ Create a data contract based on the requirements from use cases.
     $ datacontract export --format dbt-staging-sql
     ```
 
+4. Test that your data product implementation adheres to the contract.
+    ```bash
+    $ datacontract test
+    ```
+
 ### Schema Evolution
 
 #### Non-breaking Changes