datacontract-cli 0.10.1__tar.gz → 0.10.2__tar.gz

{datacontract_cli-0.10.1/datacontract_cli.egg-info → datacontract_cli-0.10.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: datacontract-cli
-Version: 0.10.1
+Version: 0.10.2
 Summary: Test data contracts
 Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>
 Project-URL: Homepage, https://cli.datacontract.com
@@ -19,7 +19,7 @@ Requires-Dist: fastapi==0.110.1
 Requires-Dist: fastparquet==2024.2.0
 Requires-Dist: python-multipart==0.0.9
 Requires-Dist: rich~=13.7.0
-Requires-Dist: simple-ddl-parser==1.0.4
+Requires-Dist: simple-ddl-parser==1.1.0
 Requires-Dist: soda-core-bigquery<3.4.0,>=3.3.1
 Requires-Dist: soda-core-duckdb<3.4.0,>=3.3.1
 Requires-Dist: soda-core-postgres<3.4.0,>=3.3.1
@@ -27,7 +27,7 @@ Requires-Dist: soda-core-snowflake<3.4.0,>=3.3.1
 Requires-Dist: soda-core-spark[databricks]<3.4.0,>=3.3.1
 Requires-Dist: soda-core-spark-df<3.4.0,>=3.3.1
 Requires-Dist: snowflake-connector-python[pandas]<3.8,>=3.6
-Requires-Dist: duckdb==0.10.1
+Requires-Dist: duckdb==0.10.2
 Requires-Dist: fastjsonschema~=2.19.1
 Requires-Dist: python-dotenv~=1.0.0
 Requires-Dist: s3fs==2024.3.1
@@ -35,12 +35,16 @@ Requires-Dist: rdflib==7.0.0
 Requires-Dist: avro==1.11.3
 Requires-Dist: opentelemetry-exporter-otlp-proto-grpc~=1.16.0
 Requires-Dist: opentelemetry-exporter-otlp-proto-http~=1.16.0
+Requires-Dist: deltalake~=0.17.0
+Requires-Dist: boto3<1.34.70,>=1.34.41
+Requires-Dist: botocore<1.34.70,>=1.34.41
 Provides-Extra: dev
 Requires-Dist: httpx==0.27.0; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: testcontainers<4.0; extra == "dev"
+Requires-Dist: moto; extra == "dev"
+Requires-Dist: testcontainers<4.0.0; extra == "dev"
 Requires-Dist: testcontainers-minio; extra == "dev"
 Requires-Dist: testcontainers-postgres; extra == "dev"
 Requires-Dist: testcontainers-kafka; extra == "dev"
@@ -136,7 +140,13 @@ CREATE TABLE line_items (
 );
 ```
 
-Or generate this [HTML page](https://datacontract.com/examples/orders-latest/datacontract.html).
+Or generate an HTML export:
+
+```bash
+$ datacontract export --format html https://datacontract.com/examples/orders-latest/datacontract.yaml > datacontract.html
+```
+
+which will create this [HTML export](https://datacontract.com/examples/orders-latest/datacontract.html).
 
 ## Usage
 
@@ -224,6 +234,7 @@ Commands
 - [breaking](#breaking)
 - [changelog](#changelog)
 - [diff](#diff)
+- [catalog](#catalog)
 
 ### init
 
@@ -296,46 +307,49 @@ Commands
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
-Data Contract CLI can connect to data sources and run schema and quality tests to verify that the data contract is valid.
+Data Contract CLI connects to a data source and runs schema and quality tests to verify that the data contract is valid.
 
 ```bash
 $ datacontract test --server production datacontract.yaml
 ```
 
-To connect to the databases the `server` block in the datacontract.yaml is used to set up the connection. In addition, credentials, such as username and passwords, may be defined with environment variables.
+To connect to the databases the `server` block in the datacontract.yaml is used to set up the connection. 
+In addition, credentials, such as username and passwords, may be defined with environment variables.
 
 The application uses different engines, based on the server `type`.
-Internally, it connects with DuckDB, Spark, or a native connection and executes the most tests with soda-core and fastjsonschema. 
-Credentials are read from the environment variables.
+Internally, it connects with DuckDB, Spark, or a native connection and executes the most tests with _soda-core_ and _fastjsonschema_. 
+
+Credentials are provided with environment variables.
 
 Supported server types:
 
-| Type         | Format     | Status                                                                          |
-|--------------|------------|---------------------------------------------------------------------------------|
-| `s3`         | `parquet`  | ✅                                                                               |
-| `s3`         | `json`     | ✅                                                                               |
-| `s3`         | `csv`      | ✅                                                                               |
-| `s3`         | `delta`    | Coming soon ([#24](https://github.com/datacontract/datacontract-cli/issues/24)) |
-| `s3`         | `iceberg`  | Coming soon                                                                     |
-| `postgres`   | n/a        | ✅                                                                               |
-| `snowflake`  | n/a        | ✅                                                                               |
-| `bigquery`   | n/a        | ✅                                                                               |
-| `redshift`   | n/a        | Coming soon                                                                     |
-| `databricks` | n/a        | ✅                                                                               |
-| `kafka`      | `json`     | ✅                                                                               |
-| `kafka`      | `avro`     | Coming soon                                                                     |
-| `kafka`      | `protobuf` | Coming soon                                                                     |
-| `local`      | `parquet`  | ✅                                                                               |
-| `local`      | `json`     | ✅                                                                               |
-| `local`      | `csv`      | ✅                                                                               |
-
-Feel free to create an issue, if you need support for an additional type.
+- [s3](#S3)
+- [bigquery](#bigquery)
+- [azure](#azure)
+- [databricks](#databricks)
+- [databricks (programmatic)](#databricks-programmatic)
+- [snowflake](#snowflake)
+- [kafka](#kafka)
+- [postgres](#postgres)
+- [local](#local)
+
+Supported formats:
+
+- parquet
+- json
+- csv
+- delta
+- iceberg (coming soon)
+
+Feel free to create an [issue](https://github.com/datacontract/datacontract-cli/issues), if you need support for an additional type and formats.
 
 ### S3
 
 Data Contract CLI can test data that is stored in S3 buckets or any S3-compliant endpoints in various formats.
 
-#### Example
+#### Examples
+
+##### JSON
 
 datacontract.yaml
 ```yaml
@@ -348,6 +362,18 @@ servers:
     delimiter: new_line # new_line, array, or none
 ```
 
+##### Delta Tables
+
+datacontract.yaml
+```yaml
+servers:
+  production:
+    type: s3
+    endpointUrl: https://minio.example.com # not needed with AWS S3
+    location: s3://bucket-name/path/table.delta # path to the Delta table folder containing parquet data files and the _delta_log
+    format: delta
+```
+
 #### Environment Variables
 
 | Environment Variable              | Example                       | Description           |
@@ -357,76 +383,40 @@ servers:
 | `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key |
 
 
-### Postgres
-
-Data Contract CLI can test data in Postgres or Postgres-compliant databases (e.g., RisingWave).
-
-#### Example
-
-datacontract.yaml
-```yaml
-servers:
-  postgres:
-    type: postgres
-    host: localhost
-    port: 5432
-    database: postgres
-    schema: public
-models:
-  my_table_1: # corresponds to a table
-    type: table
-    fields: 
-      my_column_1: # corresponds to a column
-        type: varchar
-```
-
-#### Environment Variables
-
-| Environment Variable             | Example            | Description |
-|----------------------------------|--------------------|-------------|
-| `DATACONTRACT_POSTGRES_USERNAME` | `postgres`         | Username    |
-| `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
 
+### BigQuery
 
-### Snowflake
+We support authentication to BigQuery using Service Account Key. The used Service Account should include the roles:
+* BigQuery Job User
+* BigQuery Data Viewer
 
-Data Contract CLI can test data in Snowflake.
 
 #### Example
 
 datacontract.yaml
 ```yaml
-
 servers:
-  snowflake:
-    type: snowflake
-    account: abcdefg-xn12345
-    database: ORDER_DB
-    schema: ORDERS_PII_V2
+  production:
+    type: bigquery
+    project: datameshexample-product
+    dataset: datacontract_cli_test_dataset
 models:
-  my_table_1: # corresponds to a table
+  datacontract_cli_test_table: # corresponds to a BigQuery table
     type: table
-    fields: 
-      my_column_1: # corresponds to a column
-        type: varchar
+    fields: ...
 ```
 
 #### Environment Variables
 
-| Environment Variable               | Example            | Description                                         |
-|------------------------------------|--------------------|-----------------------------------------------------|
-| `DATACONTRACT_SNOWFLAKE_USERNAME`  | `datacontract`     | Username                                            |
-| `DATACONTRACT_SNOWFLAKE_PASSWORD`  | `mysecretpassword` | Password                                            |
-| `DATACONTRACT_SNOWFLAKE_ROLE`      | `DATAVALIDATION`   | The snowflake role to use.                          |
-| `DATACONTRACT_SNOWFLAKE_WAREHOUSE` | `COMPUTE_WH`       | The Snowflake Warehouse to use executing the tests. |
+| Environment Variable                         | Example                   | Description                                             |
+|----------------------------------------------|---------------------------|---------------------------------------------------------|
+| `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Access key as saved on key creation by BigQuery |
 
 
-### BigQuery
 
-We support authentication to BigQuery using Service Account Key. The used Service Account should include the roles:
-* BigQuery Job User
-* BigQuery Data Viewer
+### Azure
 
+Data Contract CLI can test data that is stored in Azure Blob storage or Azure Data Lake Storage (Gen2) (ADLS) in various formats.
 
 #### Example
 
@@ -434,20 +424,21 @@ datacontract.yaml
 ```yaml
 servers:
   production:
-    type: bigquery
-    project: datameshexample-product
-    dataset: datacontract_cli_test_dataset
-models:
-  datacontract_cli_test_table: # corresponds to a BigQuery table
-    type: table
-    fields: ...
+    type: azure
+    location: abfss://datameshdatabricksdemo.dfs.core.windows.net/dataproducts/inventory_events/*.parquet
+    format: parquet
 ```
 
 #### Environment Variables
 
-| Environment Variable                         | Example                   | Description                                             |
-|----------------------------------------------|---------------------------|---------------------------------------------------------|
-| `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Access key as saved on key creation by BigQuery |
+Authentication works with an Azure Service Principal (SPN) aka App Registration with a secret.
+
+| Environment Variable              | Example                       | Description                                          |
+|-----------------------------------|-------------------------------|------------------------------------------------------|
+| `DATACONTRACT_AZURE_TENANT_ID`            | `79f5b80f-10ff-40b9-9d1f-774b42d605fc`                  | The Azure Tenant ID                                  |
+| `DATACONTRACT_AZURE_CLIENT_ID`     | `3cf7ce49-e2e9-4cbc-a922-4328d4a58622`            | The ApplicationID / ClientID of the app registration |
+| `DATACONTRACT_AZURE_CLIENT_SECRET` | `yZK8Q~GWO1MMXXXXXXXXXXXXX` | The Client Secret value                              |
+
 
 
 ### Databricks
@@ -518,6 +509,41 @@ run = data_contract.test()
 run.result
 ```
 
+
+### Snowflake
+
+Data Contract CLI can test data in Snowflake.
+
+#### Example
+
+datacontract.yaml
+```yaml
+
+servers:
+  snowflake:
+    type: snowflake
+    account: abcdefg-xn12345
+    database: ORDER_DB
+    schema: ORDERS_PII_V2
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields: 
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+#### Environment Variables
+
+| Environment Variable               | Example            | Description                                         |
+|------------------------------------|--------------------|-----------------------------------------------------|
+| `DATACONTRACT_SNOWFLAKE_USERNAME`  | `datacontract`     | Username                                            |
+| `DATACONTRACT_SNOWFLAKE_PASSWORD`  | `mysecretpassword` | Password                                            |
+| `DATACONTRACT_SNOWFLAKE_ROLE`      | `DATAVALIDATION`   | The snowflake role to use.                          |
+| `DATACONTRACT_SNOWFLAKE_WAREHOUSE` | `COMPUTE_WH`       | The Snowflake Warehouse to use executing the tests. |
+
+
+
 ### Kafka
 
 Kafka support is currently considered experimental.
@@ -542,13 +568,44 @@ servers:
 | `DATACONTRACT_KAFKA_SASL_PASSWORD` | `xxx`   | The SASL password (secret). |
 
 
+### Postgres
+
+Data Contract CLI can test data in Postgres or Postgres-compliant databases (e.g., RisingWave).
+
+#### Example
+
+datacontract.yaml
+```yaml
+servers:
+  postgres:
+    type: postgres
+    host: localhost
+    port: 5432
+    database: postgres
+    schema: public
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields: 
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+#### Environment Variables
+
+| Environment Variable             | Example            | Description |
+|----------------------------------|--------------------|-------------|
+| `DATACONTRACT_POSTGRES_USERNAME` | `postgres`         | Username    |
+| `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
+
+
 
 ### export
 
 ```
  Usage: datacontract export [OPTIONS] [LOCATION]                                                                                                                           
                                                                                                                                                                            
- Convert data contract to a specific format. console.prints to stdout.                                                                                                     
+ Convert data contract to a specific format. Prints to stdout or to the specified output file.                                                                                                     
                                                                                                                                                                            
 ╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
 │   location      [LOCATION]  The location (url or path) of the data contract yaml. [default: datacontract.yaml]                                                          │
@@ -641,15 +698,15 @@ data products, find the true domain owner of a field attribute)
 ### import
 
 ```
- Usage: datacontract import [OPTIONS]                                                                              
-                                                                                                                   
- Create a data contract from the given source file. Prints to stdout.                                              
-                                                                                                                   
-╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *  --format        [sql|avro]  The format of the source file. [default: None] [required]                        │
-│ *  --source        TEXT        The path to the file that should be imported. [default: None] [required]         │
-│    --help                      Show this message and exit.                                                      │
-╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+ Usage: datacontract import [OPTIONS]
+
+ Create a data contract from the given source location. Prints to stdout.
+
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ *  --format        [sql|avro|glue]  The format of the source file. [default: None] [required]                                    │
+│ *  --source        TEXT             The path to the file or Glue Database that should be imported. [default: None] [required]    │
+│    --help                           Show this message and exit.                                                                  │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
 Example: 
@@ -664,6 +721,7 @@ Available import options:
 |--------------------|------------------------------------------------|---------|
 | `sql`              | Import from SQL DDL                            | ✅       | 
 | `avro`             | Import from AVRO schemas                       | ✅     |
+| `glue`             | Import from AWS Glue DataCatalog               | ✅     |
 | `protobuf`         | Import from Protobuf schemas                   | TBD     |
 | `jsonschema`       | Import from JSON Schemas                       | TBD     |
 | `bigquery`         | Import from BigQuery Schemas                   | TBD     |
@@ -720,6 +778,21 @@ Available import options:
 ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
+### catalog
+
+```
+
+ Usage: datacontract catalog [OPTIONS]
+
+ Create an html catalog of data contracts.
+
+╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+│ --files         TEXT  Glob pattern for the data contract files to include in the catalog. [default: *.yaml]              │
+│ --output        TEXT  Output directory for the catalog html files. [default: catalog/]                                   │
+│ --help                Show this message and exit.                                                                        │
+╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+```
+
 
 ## Integrations
 
@@ -887,7 +960,7 @@ source venv/bin/activate
 pip install --upgrade pip setuptools wheel
 pip install -e '.[dev]'
 ruff check --fix
-ruff format --check
+ruff format
 pytest
 ```
 
@@ -938,6 +1011,11 @@ This command runs the container momentarily to check the version of the `datacon
 
 We are happy to receive your contributions. Propose your change in an issue or directly create a pull request with your improvements.
 
+## Companies using this tool
+
+- [INNOQ](https://innoq.com)
+- And many more. To add your company, please create a pull request.
+
 ## License
 
 [MIT License](LICENSE)