datacontract-cli 0.10.8__py3-none-any.whl → 0.10.10__py3-none-any.whl

{datacontract_cli-0.10.8.dist-info → datacontract_cli-0.10.10.dist-info}/METADATA

@@ -1,10 +1,10 @@
 Metadata-Version: 2.1
 Name: datacontract-cli
-Version: 0.10.8
-Summary: Test data contracts
-Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>
+Version: 0.10.10
+Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
+Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 Project-URL: Homepage, https://cli.datacontract.com
-Project-URL: Issues, https://github.com/datacontract/cli/issues
+Project-URL: Issues, https://github.com/datacontract/datacontract-cli/issues
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
@@ -12,10 +12,10 @@ Requires-Python: >=3.10
 Description-Content-Type: text/markdown
 License-File: LICENSE
 Requires-Dist: typer[all] <0.13,>=0.9
-Requires-Dist: pydantic <2.8.0,>=2.5.3
+Requires-Dist: pydantic <2.9.0,>=2.8.2
 Requires-Dist: pyyaml ~=6.0.1
 Requires-Dist: requests <2.33,>=2.31
-Requires-Dist: fastapi ==0.111.0
+Requires-Dist: fastapi ==0.111.1
 Requires-Dist: fastparquet ==2024.5.0
 Requires-Dist: python-multipart ==0.0.9
 Requires-Dist: rich ~=13.7.0
@@ -28,18 +28,18 @@ Requires-Dist: python-dotenv ~=1.0.0
 Requires-Dist: rdflib ==7.0.0
 Requires-Dist: opentelemetry-exporter-otlp-proto-grpc ~=1.16
 Requires-Dist: opentelemetry-exporter-otlp-proto-http ~=1.16
-Requires-Dist: boto3 <1.34.130,>=1.34.41
-Requires-Dist: botocore <1.34.128,>=1.34.41
+Requires-Dist: boto3 <1.34.137,>=1.34.41
+Requires-Dist: botocore <1.34.137,>=1.34.41
 Requires-Dist: jinja-partials >=0.2.1
 Provides-Extra: all
-Requires-Dist: datacontract-cli[bigquery,databricks,deltalake,kafka,postgres,s3,snowflake,sqlserver] ; extra == 'all'
+Requires-Dist: datacontract-cli[bigquery,databricks,deltalake,kafka,postgres,s3,snowflake,sqlserver,trino] ; extra == 'all'
 Provides-Extra: avro
 Requires-Dist: avro ==1.11.3 ; extra == 'avro'
 Provides-Extra: bigquery
 Requires-Dist: soda-core-bigquery <3.4.0,>=3.3.1 ; extra == 'bigquery'
 Provides-Extra: databricks
 Requires-Dist: soda-core-spark-df <3.4.0,>=3.3.1 ; extra == 'databricks'
-Requires-Dist: databricks-sql-connector <3.2.0,>=3.1.2 ; extra == 'databricks'
+Requires-Dist: databricks-sql-connector <3.3.0,>=3.1.2 ; extra == 'databricks'
 Requires-Dist: soda-core-spark[databricks] <3.4.0,>=3.3.1 ; extra == 'databricks'
 Provides-Extra: deltalake
 Requires-Dist: deltalake <0.19,>=0.17 ; extra == 'deltalake'
@@ -50,10 +50,12 @@ Requires-Dist: ruff ; extra == 'dev'
 Requires-Dist: pre-commit ~=3.7.1 ; extra == 'dev'
 Requires-Dist: pytest ; extra == 'dev'
 Requires-Dist: pytest-xdist ; extra == 'dev'
-Requires-Dist: moto ; extra == 'dev'
+Requires-Dist: moto ==5.0.11 ; extra == 'dev'
 Requires-Dist: pymssql ==2.3.0 ; extra == 'dev'
 Requires-Dist: kafka-python ; extra == 'dev'
-Requires-Dist: testcontainers ~=4.5.0 ; extra == 'dev'
+Requires-Dist: trino ==0.329.0 ; extra == 'dev'
+Requires-Dist: testcontainers <4.8,>=4.5 ; extra == 'dev'
+Requires-Dist: testcontainers[core] ; extra == 'dev'
 Requires-Dist: testcontainers[minio] ; extra == 'dev'
 Requires-Dist: testcontainers[postgres] ; extra == 'dev'
 Requires-Dist: testcontainers[kafka] ; extra == 'dev'
@@ -64,12 +66,14 @@ Requires-Dist: soda-core-spark-df <3.4.0,>=3.3.1 ; extra == 'kafka'
 Provides-Extra: postgres
 Requires-Dist: soda-core-postgres <3.4.0,>=3.3.1 ; extra == 'postgres'
 Provides-Extra: s3
-Requires-Dist: s3fs ==2024.6.0 ; extra == 's3'
+Requires-Dist: s3fs ==2024.6.1 ; extra == 's3'
 Provides-Extra: snowflake
-Requires-Dist: snowflake-connector-python[pandas] <3.11,>=3.6 ; extra == 'snowflake'
+Requires-Dist: snowflake-connector-python[pandas] <3.12,>=3.6 ; extra == 'snowflake'
 Requires-Dist: soda-core-snowflake <3.4.0,>=3.3.1 ; extra == 'snowflake'
 Provides-Extra: sqlserver
 Requires-Dist: soda-core-sqlserver <3.4.0,>=3.3.1 ; extra == 'sqlserver'
+Provides-Extra: trino
+Requires-Dist: soda-core-trino <3.4.0,>=3.3.1 ; extra == 'trino'
 
 # Data Contract CLI
 
@@ -258,17 +262,18 @@ pip install datacontract-cli[all]
 
 A list of available extras:
 
-| Dependency              | Installation Command                                        |
-|-------------------------|-------------------------------------------------------------|
-| Avro Support            | `pip install datacontract-cli[avro]`                        |
-| Google BigQuery         | `pip install datacontract-cli[bigquery]`                    |
-| Databricks Integration  | `pip install datacontract-cli[databricks]`                  |
-| Deltalake Integration   | `pip install datacontract-cli[deltalake]`                  |
-| Kafka Integration       | `pip install datacontract-cli[kafka]`                       |
-| PostgreSQL Integration  | `pip install datacontract-cli[postgres]`                    |
-| S3 Integration          | `pip install datacontract-cli[s3]`                          |
-| Snowflake Integration   | `pip install datacontract-cli[snowflake]`                   |
-| Microsoft SQL Server    | `pip install datacontract-cli[sqlserver]`                   |
+| Dependency             | Installation Command                       |
+|------------------------|--------------------------------------------|
+| Avro Support           | `pip install datacontract-cli[avro]`       |
+| Google BigQuery        | `pip install datacontract-cli[bigquery]`   |
+| Databricks Integration | `pip install datacontract-cli[databricks]` |
+| Deltalake Integration  | `pip install datacontract-cli[deltalake]`  |
+| Kafka Integration      | `pip install datacontract-cli[kafka]`      |
+| PostgreSQL Integration | `pip install datacontract-cli[postgres]`   |
+| S3 Integration         | `pip install datacontract-cli[s3]`         |
+| Snowflake Integration  | `pip install datacontract-cli[snowflake]`  |
+| Microsoft SQL Server   | `pip install datacontract-cli[sqlserver]`  |
+| Trino                  | `pip install datacontract-cli[trino]`      |
 
 
 
@@ -295,16 +300,16 @@ Commands
  Download a datacontract.yaml template and write it to file.
 
 ╭─ Arguments ──────────────────────────────────────────────────────────────────────────────────╮
-│   location      [LOCATION]  The location (url or path) of the data contract yaml to create. │
-│                             [default: datacontract.yaml]                                    │
+│   location      [LOCATION]  The location (url or path) of the data contract yaml to create.  │
+│                             [default: datacontract.yaml]                                     │
 ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
 ╭─ Options ────────────────────────────────────────────────────────────────────────────────────╮
-│ --template                       TEXT  URL of a template or data contract                   │
-│                                        [default:                                            │
-│                                        https://datacontract.com/datacontract.init.yaml]     │
-│ --overwrite    --no-overwrite          Replace the existing datacontract.yaml               │
-│                                        [default: no-overwrite]                              │
-│ --help                                 Show this message and exit.                          │
+│ --template                       TEXT  URL of a template or data contract                    │
+│                                        [default:                                             │
+│                                        https://datacontract.com/datacontract.init.yaml]      │
+│ --overwrite    --no-overwrite          Replace the existing datacontract.yaml                │
+│                                        [default: no-overwrite]                               │
+│ --help                                 Show this message and exit.                           │
 ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -316,12 +321,12 @@ Commands
  Validate that the datacontract.yaml is correctly formatted.
 
 ╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│   location      [LOCATION]  The location (url or path) of the data contract yaml. [default: datacontract.yaml]                 │
+│   location      [LOCATION]  The location (url or path) of the data contract yaml. [default: datacontract.yaml]                  │
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --schema        TEXT  The location (url or path) of the Data Contract Specification JSON Schema                                │
-│                       [default: https://datacontract.com/datacontract.schema.json]                                             │
-│ --help                Show this message and exit.                                                                              │
+│ --schema        TEXT  The location (url or path) of the Data Contract Specification JSON Schema                                 │
+│                       [default: https://datacontract.com/datacontract.schema.json]                                              │
+│ --help                Show this message and exit.                                                                               │
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -333,28 +338,28 @@ Commands
  Run schema and quality tests on configured servers.
 
 ╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│   location      [LOCATION]  The location (url or path) of the data contract yaml. [default: datacontract.yaml]                 │
+│   location      [LOCATION]  The location (url or path) of the data contract yaml. [default: datacontract.yaml]                  │
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --schema                                                       TEXT  The location (url or path) of the Data Contract           │
-│                                                                      Specification JSON Schema                                 │
-│                                                                      [default:                                                 │
-│                                                                      https://datacontract.com/datacontract.schema.json]        │
-│ --server                                                       TEXT  The server configuration to run the schema and quality    │
-│                                                                      tests. Use the key of the server object in the data       │
-│                                                                      contract yaml file to refer to a server, e.g.,            │
-│                                                                      `production`, or `all` for all servers (default).         │
-│                                                                      [default: all]                                            │
-│ --examples                    --no-examples                          Run the schema and quality tests on the example data      │
-│                                                                      within the data contract.                                 │
-│                                                                      [default: no-examples]                                    │
-│ --publish                                                      TEXT  The url to publish the results after the test             │
-│                                                                      [default: None]                                           │
-│ --publish-to-opentelemetry    --no-publish-to-opentelemetry          Publish the results to opentelemetry. Use environment     │
-│                                                                      variables to configure the OTLP endpoint, headers, etc.   │
-│                                                                      [default: no-publish-to-opentelemetry]                    │
-│ --logs                        --no-logs                              Print logs [default: no-logs]                             │
-│ --help                                                               Show this message and exit.                               │
+│ --schema                                                       TEXT  The location (url or path) of the Data Contract            │
+│                                                                      Specification JSON Schema                                  │
+│                                                                      [default:                                                  │
+│                                                                      https://datacontract.com/datacontract.schema.json]         │
+│ --server                                                       TEXT  The server configuration to run the schema and quality     │
+│                                                                      tests. Use the key of the server object in the data        │
+│                                                                      contract yaml file to refer to a server, e.g.,             │
+│                                                                      `production`, or `all` for all servers (default).          │
+│                                                                      [default: all]                                             │
+│ --examples                    --no-examples                          Run the schema and quality tests on the example data       │
+│                                                                      within the data contract.                                  │
+│                                                                      [default: no-examples]                                     │
+│ --publish                                                      TEXT  The url to publish the results after the test              │
+│                                                                      [default: None]                                            │
+│ --publish-to-opentelemetry    --no-publish-to-opentelemetry          Publish the results to opentelemetry. Use environment      │
+│                                                                      variables to configure the OTLP endpoint, headers, etc.    │
+│                                                                      [default: no-publish-to-opentelemetry]                     │
+│ --logs                        --no-logs                              Print logs [default: no-logs]                              │
+│ --help                                                               Show this message and exit.                                │
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -384,6 +389,7 @@ Supported server types:
 - [snowflake](#snowflake)
 - [kafka](#kafka)
 - [postgres](#postgres)
+- [trino](#trino)
 - [local](#local)
 
 Supported formats:
@@ -429,11 +435,12 @@ servers:
 
 #### Environment Variables
 
-| Environment Variable              | Example                       | Description           |
-|-----------------------------------|-------------------------------|-----------------------|
-| `DATACONTRACT_S3_REGION`            | `eu-central-1`                  | Region of S3 bucket   |
-| `DATACONTRACT_S3_ACCESS_KEY_ID`     | `AKIAXV5Q5QABCDEFGH`            | AWS Access Key ID     |
-| `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key |
+| Environment Variable                | Example                         | Description                            |
+|-------------------------------------|---------------------------------|----------------------------------------|
+| `DATACONTRACT_S3_REGION`            | `eu-central-1`                  | Region of S3 bucket                    |
+| `DATACONTRACT_S3_ACCESS_KEY_ID`     | `AKIAXV5Q5QABCDEFGH`            | AWS Access Key ID                      |
+| `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key                  |
+| `DATACONTRACT_S3_SESSION_TOKEN`     | `AQoDYXdzEJr...`                | AWS temporary session token (optional) |
 
 
 
@@ -466,7 +473,6 @@ models:
 | `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Access key as saved on key creation by BigQuery. If this environment variable isn't set, the cli tries to use `GOOGLE_APPLICATION_CREDENTIALS` as a fallback, so if you have that set for using their Python library anyway, it should work seamlessly. |
 
 
-
 ### Azure
 
 Data Contract CLI can test data that is stored in Azure Blob storage or Azure Data Lake Storage (Gen2) (ADLS) in various formats.
@@ -486,11 +492,11 @@ servers:
 
 Authentication works with an Azure Service Principal (SPN) aka App Registration with a secret.
 
-| Environment Variable              | Example                       | Description                                          |
-|-----------------------------------|-------------------------------|------------------------------------------------------|
-| `DATACONTRACT_AZURE_TENANT_ID`            | `79f5b80f-10ff-40b9-9d1f-774b42d605fc`                  | The Azure Tenant ID                                  |
-| `DATACONTRACT_AZURE_CLIENT_ID`     | `3cf7ce49-e2e9-4cbc-a922-4328d4a58622`            | The ApplicationID / ClientID of the app registration |
-| `DATACONTRACT_AZURE_CLIENT_SECRET` | `yZK8Q~GWO1MMXXXXXXXXXXXXX` | The Client Secret value                              |
+| Environment Variable               | Example                                | Description                                          |
+|------------------------------------|----------------------------------------|------------------------------------------------------|
+| `DATACONTRACT_AZURE_TENANT_ID`     | `79f5b80f-10ff-40b9-9d1f-774b42d605fc` | The Azure Tenant ID                                  |
+| `DATACONTRACT_AZURE_CLIENT_ID`     | `3cf7ce49-e2e9-4cbc-a922-4328d4a58622` | The ApplicationID / ClientID of the app registration |
+| `DATACONTRACT_AZURE_CLIENT_SECRET` | `yZK8Q~GWO1MMXXXXXXXXXXXXX`            | The Client Secret value                              |
 
 
 
@@ -520,13 +526,13 @@ models:
 
 #### Environment Variables
 
-| Environment Variable             | Example            | Description |
-|----------------------------------|--------------------|-------------|
-| `DATACONTRACT_SQLSERVER_USERNAME` | `root`         | Username    |
-| `DATACONTRACT_SQLSERVER_PASSWORD` | `toor` | Password    |
-| `DATACONTRACT_SQLSERVER_TRUSTED_CONNECTION` | `True` | Use windows authentication, instead of login    |
-| `DATACONTRACT_SQLSERVER_TRUST_SERVER_CERTIFICATE` | `True` | Trust self-signed certificate    |
-| `DATACONTRACT_SQLSERVER_ENCRYPTED_CONNECTION` | `True` | Use SSL    |
+| Environment Variable                              | Example| Description                                  |
+|---------------------------------------------------|--------|----------------------------------------------|
+| `DATACONTRACT_SQLSERVER_USERNAME`                 | `root` | Username                                     |
+| `DATACONTRACT_SQLSERVER_PASSWORD`                 | `toor` | Password                                     |
+| `DATACONTRACT_SQLSERVER_TRUSTED_CONNECTION`       | `True` | Use windows authentication, instead of login |
+| `DATACONTRACT_SQLSERVER_TRUST_SERVER_CERTIFICATE` | `True` | Trust self-signed certificate                |
+| `DATACONTRACT_SQLSERVER_ENCRYPTED_CONNECTION`     | `True` | Use SSL                                      |
 
 
 
@@ -557,8 +563,8 @@ models:
 
 | Environment Variable                         | Example                              | Description                                           |
 |----------------------------------------------|--------------------------------------|-------------------------------------------------------|
-| `DATACONTRACT_DATABRICKS_TOKEN` | `dapia00000000000000000000000000000` | The personal access token to authenticate             |
-| `DATACONTRACT_DATABRICKS_HTTP_PATH` | `/sql/1.0/warehouses/b053a3ffffffff` | The HTTP path to the SQL warehouse or compute cluster |
+| `DATACONTRACT_DATABRICKS_TOKEN`              | `dapia00000000000000000000000000000` | The personal access token to authenticate             |
+| `DATACONTRACT_DATABRICKS_HTTP_PATH`          | `/sql/1.0/warehouses/b053a3ffffffff` | The HTTP path to the SQL warehouse or compute cluster |
 
 
 ### Databricks (programmatic)
@@ -603,7 +609,7 @@ run.result
 
 Works with Spark DataFrames.
 DataFrames need to be created as named temporary views.
-Multiple temporary views are suppored if your data contract contains multiple models.
+Multiple temporary views are supported if your data contract contains multiple models.
 
 Testing DataFrames is useful to test your datasets in a pipeline before writing them to a data source.
 
@@ -724,6 +730,35 @@ models:
 | `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
 
 
+### Trino
+
+Data Contract CLI can test data in Trino.
+
+#### Example
+
+datacontract.yaml
+```yaml
+servers:
+  trino:
+    type: trino
+    host: localhost
+    port: 8080
+    catalog: my_catalog
+    schema: my_schema
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields:
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+#### Environment Variables
+
+| Environment Variable          | Example            | Description |
+|-------------------------------|--------------------|-------------|
+| `DATACONTRACT_TRINO_USERNAME` | `trino`            | Username    |
+| `DATACONTRACT_TRINO_PASSWORD` | `mysecretpassword` | Password    |
 
 
 
@@ -742,7 +777,7 @@ models:
 │ *  --format        [jsonschema|pydantic-model|sodacl|dbt|dbt-sources|db  The export format. [default: None] [required]         │
 │                    t-staging-sql|odcs|rdf|avro|protobuf|great-expectati                                                        │
 │                    ons|terraform|avro-idl|sql|sql-query|html|go|bigquer                                                        │
-│                    y|dbml]                                                                                                     │
+│                    y|dbml|spark]                                                                                               │
 │    --output        PATH                                                  Specify the file path where the exported data will be │
 │                                                                          saved. If no path is provided, the output will be     │
 │                                                                          printed to stdout.                                    │
@@ -792,6 +827,7 @@ Available export options:
 | `go`                 | Export to Go types                                      | ✅     |
 | `pydantic-model`     | Export to pydantic models                               | ✅     |
 | `DBML`               | Export to a DBML Diagram description                    | ✅     |
+| `spark`              | Export to a Spark StructType                            | ✅     |
 | Missing something?   | Please create an issue on GitHub                        | TBD    |
 
 #### Great Expectations
@@ -838,6 +874,10 @@ The export function converts the logical data types of the datacontract into the
 if a server is selected via the `--server` option (based on the `type` of that server). If no server is selected, the
 logical data types are exported.
 
+#### Spark
+
+The export function converts the data contract specification into a StructType Spark schema. The returned value is a Python code picture of the model schemas.  
+Spark DataFrame schema is defined as StructType. For more details about Spark Data Types please see [the spark documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) 
 
 #### Avro
 
@@ -888,18 +928,19 @@ models:
  Create a data contract from the given source location. Prints to stdout.
 
 ╭─ Options ───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *  --format                  [sql|avro|glue|bigquery|jsonschema|unity]  The format of the source file. [default: None] [required]         │
-│    --source                  TEXT                                       The path to the file or Glue Database that should be imported.    │
-│                                                                         [default: None]                                                   │
-│    --glue-table              TEXT                                       List of table ids to import from the Glue Database (repeat for    │
-│                                                                         multiple table ids, leave empty for all tables in the dataset).   │
-│                                                                         [default: None]                                                   │
-│    --bigquery-project        TEXT                                       The bigquery project id. [default: None]                          │
-│    --bigquery-dataset        TEXT                                       The bigquery dataset id. [default: None]                          │
-│    --bigquery-table          TEXT                                       List of table ids to import from the bigquery API (repeat for     │
-│                                                                         multiple table ids, leave empty for all tables in the dataset).   │
-│                                                                         [default: None]                                                   │
-│    --help                                                               Show this message and exit.                                       │
+│ *  --format                  [sql|avro|glue|bigquery|jsonschema|  The format of the source file. [default: None] [required]         |
+│                               unity|spark]                                                                                          |
+│    --source                  TEXT                                 The path to the file or Glue Database that should be imported.    │
+│                                                                   [default: None]                                                   │
+│    --glue-table              TEXT                                 List of table ids to import from the Glue Database (repeat for    │
+│                                                                   multiple table ids, leave empty for all tables in the dataset).   │
+│                                                                   [default: None]                                                   │
+│    --bigquery-project        TEXT                                 The bigquery project id. [default: None]                          │
+│    --bigquery-dataset        TEXT                                 The bigquery dataset id. [default: None]                          │
+│    --bigquery-table          TEXT                                 List of table ids to import from the bigquery API (repeat for     │
+│                                                                   multiple table ids, leave empty for all tables in the dataset).   │
+│                                                                   [default: None]                                                   │
+│    --help                                                         Show this message and exit.                                       │
 ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -917,11 +958,11 @@ Available import options:
 | `avro`             | Import from AVRO schemas                       | ✅      |
 | `glue`             | Import from AWS Glue DataCatalog               | ✅      |
 | `protobuf`         | Import from Protobuf schemas                   | TBD     |
-| `jsonschema`       | Import from JSON Schemas                       | ✅       |
-| `bigquery`         | Import from BigQuery Schemas                   | ✅       |
+| `jsonschema`       | Import from JSON Schemas                       | ✅      |
+| `bigquery`         | Import from BigQuery Schemas                   | ✅      |
 | `unity`            | Import from Databricks Unity Catalog           | partial |
 | `dbt`              | Import from dbt models                         | TBD     |
-| `odcs`             | Import from Open Data Contract Standard (ODCS) | ✅       |
+| `odcs`             | Import from Open Data Contract Standard (ODCS) | ✅      |
 | Missing something? | Please create an issue on GitHub               | TBD     |
 
 
@@ -964,7 +1005,7 @@ export DATABRICKS_IMPORT_ACCESS_TOKEN=<token>
 datacontract import --format unity --unity-table-full-name <table_full_name>
 ```
 
-### Glue
+#### Glue
 
 Importing from Glue reads the necessary Data directly off of the AWS API.
 You may give the `glue-table` parameter to enumerate the tables that should be imported. If no tables are given, _all_ available tables of the database will be imported.
@@ -981,6 +1022,15 @@ datacontract import --format glue --source <database_name> --glue-table <table_n
 datacontract import --format glue --source <database_name>
 ```
 
+#### Spark
+
+Importing from Spark table or view these must be created or accessible in the Spark context. Specify tables list in `source` parameter.
+
+Example:
+
+```bash
+datacontract import --format spark --source "users,orders"
+```
 
 ### breaking
 
@@ -990,11 +1040,11 @@ datacontract import --format glue --source <database_name>
  Identifies breaking changes between data contracts. Prints to stdout.
 
 ╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *    location_old      TEXT  The location (url or path) of the old data contract yaml. [default: None] [required]        │
-│ *    location_new      TEXT  The location (url or path) of the new data contract yaml. [default: None] [required]        │
+│ *    location_old      TEXT  The location (url or path) of the old data contract yaml. [default: None] [required]         │
+│ *    location_new      TEXT  The location (url or path) of the new data contract yaml. [default: None] [required]         │
 ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --help          Show this message and exit.                                                                              │
+│ --help          Show this message and exit.                                                                               │
 ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -1006,11 +1056,11 @@ datacontract import --format glue --source <database_name>
  Generate a changelog between data contracts. Prints to stdout.
 
 ╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *    location_old      TEXT  The location (url or path) of the old data contract yaml. [default: None] [required]        │
-│ *    location_new      TEXT  The location (url or path) of the new data contract yaml. [default: None] [required]        │
+│ *    location_old      TEXT  The location (url or path) of the old data contract yaml. [default: None] [required]         │
+│ *    location_new      TEXT  The location (url or path) of the new data contract yaml. [default: None] [required]         │
 ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --help          Show this message and exit.                                                                              │
+│ --help          Show this message and exit.                                                                               │
 ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -1022,8 +1072,8 @@ datacontract import --format glue --source <database_name>
  PLACEHOLDER. Currently works as 'changelog' does.
 
 ╭─ Arguments ───────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ *    location_old      TEXT  The location (url or path) of the old data contract yaml. [default: None] [required]        │
-│ *    location_new      TEXT  The location (url or path) of the new data contract yaml. [default: None] [required]        │
+│ *    location_old      TEXT  The location (url or path) of the old data contract yaml. [default: None] [required]         │
+│ *    location_new      TEXT  The location (url or path) of the new data contract yaml. [default: None] [required]         │
 ╰───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ╭─ Options ─────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
 │ --help          Show this message and exit.                                                                               │
@@ -1039,9 +1089,9 @@ datacontract import --format glue --source <database_name>
  Create an html catalog of data contracts.
 
 ╭─ Options ────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-│ --files         TEXT  Glob pattern for the data contract files to include in the catalog. [default: *.yaml]             │
-│ --output        TEXT  Output directory for the catalog html files. [default: catalog/]                                  │
-│ --help                Show this message and exit.                                                                       │
+│ --files         TEXT  Glob pattern for the data contract files to include in the catalog. [default: *.yaml]              │
+│ --output        TEXT  Output directory for the catalog html files. [default: catalog/]                                   │
+│ --help                Show this message and exit.                                                                        │
 ╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
 ```
 
@@ -1063,19 +1113,22 @@ datacontract import --format glue --source <database_name>
 
 ## Integrations
 
-| Integration       | Option                       | Description                                                                                           |
-|-------------------|------------------------------|-------------------------------------------------------------------------------------------------------|
-| Data Mesh Manager | `--publish`                  | Push full results to the [Data Mesh Manager API](https://api.datamesh-manager.com/swagger/index.html) |
-| OpenTelemetry     | `--publish-to-opentelemetry` | Push result as gauge metrics                                                                          |
+| Integration           | Option                       | Description                                                                                                   |
+|-----------------------|------------------------------|---------------------------------------------------------------------------------------------------------------|
+| Data Mesh Manager     | `--publish`                  | Push full results to the [Data Mesh Manager API](https://api.datamesh-manager.com/swagger/index.html)         |
+| Data Contract Manager | `--publish`                  | Push full results to the [Data Contract Manager API](https://api.datacontract-manager.com/swagger/index.html) |
+| OpenTelemetry         | `--publish-to-opentelemetry` | Push result as gauge metrics                                                                                  |
 
 ### Integration with Data Mesh Manager
 
-If you use [Data Mesh Manager](https://datamesh-manager.com/), you can use the data contract URL and append the `--publish` option to send and display the test results. Set an environment variable for your API key.
+If you use [Data Mesh Manager](https://datamesh-manager.com/) or [Data Contract Manager](https://datacontract-manager.com/), you can use the data contract URL and append the `--publish` option to send and display the test results. Set an environment variable for your API key.
 
 ```bash
 # Fetch current data contract, execute tests on production, and publish result to data mesh manager
 $ EXPORT DATAMESH_MANAGER_API_KEY=xxx
-$ datacontract test https://demo.datamesh-manager.com/demo279750347121/datacontracts/4df9d6ee-e55d-4088-9598-b635b2fdcbbc/datacontract.yaml --server production --publish
+$ datacontract test https://demo.datamesh-manager.com/demo279750347121/datacontracts/4df9d6ee-e55d-4088-9598-b635b2fdcbbc/datacontract.yaml \ 
+ --server production \
+ --publish https://api.datamesh-manager.com/api/test-results
 ```
 
 ### Integration with OpenTelemetry
@@ -1085,12 +1138,12 @@ If you use OpenTelemetry, you can use the data contract URL and append the `--pu
 The metric name is "datacontract.cli.test.result" and it uses the following encoding for the result:
 
 | datacontract.cli.test.result | Description                           |
-|-------|---------------------------------------|
-| 0     | test run passed, no warnings          |
-| 1     | test run has warnings                 |
-| 2     | test run failed                       |
-| 3     | test run not possible due to an error |
-| 4     | test status unknown                   |
+|------------------------------|---------------------------------------|
+| 0                            | test run passed, no warnings          |
+| 1                            | test run has warnings                 |
+| 2                            | test run failed                       |
+| 3                            | test run not possible due to an error |
+| 4                            | test status unknown                   |
 
 
 ```bash
@@ -1118,7 +1171,7 @@ Create a data contract based on the actual data. This is the fastest way to get
 
 1. Use an existing physical schema (e.g., SQL DDL) as a starting point to define your logical data model in the contract. Double check right after the import whether the actual data meets the imported logical data model. Just to be sure.
     ```bash
-    $ datacontract import --format sql ddl.sql
+    $ datacontract import --format sql --source ddl.sql
     $ datacontract test
     ```
 
@@ -1141,7 +1194,7 @@ Create a data contract based on the actual data. This is the fastest way to get
 
 5. Set up a CI pipeline that executes daily and reports the results to the [Data Mesh Manager](https://datamesh-manager.com). Or to some place else. You can even publish to any opentelemetry compatible system.
    ```bash
-   $ datacontract test --publish https://api.datamesh-manager.com/api/runs
+   $ datacontract test --publish https://api.datamesh-manager.com/api/test-results
    ```
 
 ### Contract-First
@@ -1193,7 +1246,7 @@ Examples: adding models or fields
 - Add the models or fields in the datacontract.yaml
 - Increment the minor version of the datacontract.yaml on any change. Simply edit the datacontract.yaml for this.
 - You need a policy that these changes are non-breaking. That means that one cannot use the star expression in SQL to query a table under contract. Make the consequences known.
-- Fail the build in the Pull Request if a datacontract.yaml accidentially adds a breaking change even despite only a minor version change
+- Fail the build in the Pull Request if a datacontract.yaml accidentally adds a breaking change even despite only a minor version change
    ```bash
   $ datacontract breaking datacontract-from-pr.yaml datacontract-from-main.yaml
   ```
@@ -1214,6 +1267,121 @@ Examples: Removing or renaming models and fields.
   $ datacontract changelog datacontract-from-pr.yaml datacontract-from-main.yaml
   ```
 
+## Customizing Exporters and Importers
+ 
+### Custom Exporter
+Using the exporter factory to add a new custom exporter
+```python
+
+from datacontract.data_contract import DataContract
+from datacontract.export.exporter import Exporter
+from datacontract.export.exporter_factory import exporter_factory
+
+
+# Create a custom class that implements export method
+class CustomExporter(Exporter):
+    def export(self, data_contract, model, server, sql_server_type, export_args) -> dict:
+        result = {
+            "title": data_contract.info.title,
+            "version": data_contract.info.version,
+            "description": data_contract.info.description,
+            "email": data_contract.info.contact.email,
+            "url": data_contract.info.contact.url,
+            "model": model,
+            "model_columns": ", ".join(list(data_contract.models.get(model).fields.keys())),
+            "export_args": export_args,
+            "custom_args": export_args.get("custom_arg", ""),
+        }
+        return result
+
+
+# Register the new custom class into factory
+exporter_factory.register_exporter("custom", CustomExporter)
+
+
+if __name__ == "__main__":
+    # Create a DataContract instance
+    data_contract = DataContract(
+        data_contract_file="/path/datacontract.yaml"
+    )
+    # call export
+    result = data_contract.export(
+        export_format="custom", model="orders", server="production", custom_arg="my_custom_arg"
+    )
+    print(result)
+
+```
+Output
+```python
+{
+ 'title': 'Orders Unit Test', 
+ 'version': '1.0.0', 
+ 'description': 'The orders data contract', 
+ 'email': 'team-orders@example.com', 
+ 'url': 'https://wiki.example.com/teams/checkout', 
+ 'model': 'orders', 
+ 'model_columns': 'order_id, order_total, order_status', 
+ 'export_args': {'server': 'production', 'custom_arg': 'my_custom_arg'}, 
+ 'custom_args': 'my_custom_arg'
+}
+```
+  
+### Custom Importer
+Using the importer factory to add a new custom importer
+```python
+
+from datacontract.model.data_contract_specification import DataContractSpecification
+from datacontract.data_contract import DataContract
+from datacontract.imports.importer import Importer
+from datacontract.imports.importer_factory import importer_factory
+import json
+
+# Create a custom class that implements import_source method
+class CustomImporter(Importer):
+    def import_source(
+        self, data_contract_specification: DataContractSpecification, source: str, import_args: dict
+    ) -> dict:
+        source_dict = json.loads(source)
+        data_contract_specification.id = source_dict.get("id_custom")
+        data_contract_specification.info.title = source_dict.get("title")
+        data_contract_specification.info.description = source_dict.get("description_from_app")
+
+        return data_contract_specification
+
+
+# Register the new custom class into factory
+importer_factory.register_importer("custom_company_importer", CustomImporter)
+
+
+if __name__ == "__main__":
+    # get a custom da
+    json_from_custom_app = '{"id_custom":"uuid-custom","version":"0.0.2", "title":"my_custom_imported_data", "description_from_app": "Custom contract description"}'
+    # Create a DataContract instance
+    data_contract = DataContract()
+
+    # call import_from
+    result = data_contract.import_from_source(
+        format="custom_company_importer", data_contract_specification=DataContract.init(), source=json_from_custom_app
+    )
+    print(dict(result))
+
+```
+Output
+
+```python
+{
+  'dataContractSpecification': '0.9.3', 
+  'id': 'uuid-custom', 
+  'info': Info(title='my_custom_imported_data', version='0.0.1', status=None, description='Custom contract description', owner=None, contact=None), 
+  'servers': {}, 
+  'terms': None, 
+  'models': {}, 
+  'definitions': {}, 
+  'examples': [], 
+  'quality': None, 
+  'servicelevels': None
+}
+```
 ## Development Setup
 
 Python base interpreter should be 3.11.x (unless working on 3.12 release candidate).
@@ -1263,7 +1431,27 @@ docker compose run --rm datacontract --version
 
 This command runs the container momentarily to check the version of the `datacontract` CLI. The `--rm` flag ensures that the container is automatically removed after the command executes, keeping your environment clean.
 
+## Use with pre-commit
+
+To run `datacontract-cli` as part of a [pre-commit](https://pre-commit.com/) workflow, add something like the below to the `repos` list in the project's `.pre-commit-config.yaml`:
+
+```yaml
+repos:
+  - repo: https://github.com/datacontract/datacontract-cli
+    rev: "v0.10.9"
+    hooks:
+      - id: datacontract-lint
+      - id: datacontract-test
+        args: ["--server", "production"]
+```
+
+### Available Hook IDs
 
+| Hook ID           | Description                                        | Dependency |
+| ----------------- | -------------------------------------------------- | ---------- |
+| datacontract-lint | Runs the lint subcommand.                          | Python3    |
+| datacontract-test | Runs the test subcommand. Please look at           | Python3    |
+|                   | [test](#test) section for all available arguments. |            |
 
 ## Release Steps
 
@@ -1285,7 +1473,7 @@ We are happy to receive your contributions. Propose your change in an issue or d
 
 ## Related Tools
 
-- [Data Mesh Manager](https://www.datamesh-manager.com/) is a commercial tool to manage data products and data contracts. It supports the data contract specification and allows the user to import or export data contracts using this specification.
+- [Data Contract Manager](https://www.datacontract-manager.com/) is a commercial tool to manage data contracts. It contains a web UI, access management, and data governance for a full enterprise data marketplace.
 - [Data Contract GPT](https://gpt.datacontract.com) is a custom GPT that can help you write data contracts.
 - [Data Contract Editor](https://editor.datacontract.com) is an editor for Data Contracts, including a live html preview.