datacontract-cli 0.10.8__tar.gz → 0.10.9__tar.gz

{datacontract_cli-0.10.8 → datacontract_cli-0.10.9}/PKG-INFO

@@ -1,10 +1,10 @@
 Metadata-Version: 2.1
 Name: datacontract-cli
-Version: 0.10.8
-Summary: Test data contracts
-Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>
+Version: 0.10.9
+Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
+Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 Project-URL: Homepage, https://cli.datacontract.com
-Project-URL: Issues, https://github.com/datacontract/cli/issues
+Project-URL: Issues, https://github.com/datacontract/datacontract-cli/issues
 Classifier: Programming Language :: Python :: 3
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Operating System :: OS Independent
@@ -28,8 +28,8 @@ Requires-Dist: python-dotenv~=1.0.0
 Requires-Dist: rdflib==7.0.0
 Requires-Dist: opentelemetry-exporter-otlp-proto-grpc~=1.16
 Requires-Dist: opentelemetry-exporter-otlp-proto-http~=1.16
-Requires-Dist: boto3<1.34.130,>=1.34.41
-Requires-Dist: botocore<1.34.128,>=1.34.41
+Requires-Dist: boto3<1.34.137,>=1.34.41
+Requires-Dist: botocore<1.34.137,>=1.34.41
 Requires-Dist: jinja_partials>=0.2.1
 Provides-Extra: avro
 Requires-Dist: avro==1.11.3; extra == "avro"
@@ -37,7 +37,7 @@ Provides-Extra: bigquery
 Requires-Dist: soda-core-bigquery<3.4.0,>=3.3.1; extra == "bigquery"
 Provides-Extra: databricks
 Requires-Dist: soda-core-spark-df<3.4.0,>=3.3.1; extra == "databricks"
-Requires-Dist: databricks-sql-connector<3.2.0,>=3.1.2; extra == "databricks"
+Requires-Dist: databricks-sql-connector<3.3.0,>=3.1.2; extra == "databricks"
 Requires-Dist: soda-core-spark[databricks]<3.4.0,>=3.3.1; extra == "databricks"
 Provides-Extra: deltalake
 Requires-Dist: deltalake<0.19,>=0.17; extra == "deltalake"
@@ -47,14 +47,16 @@ Requires-Dist: soda-core-spark-df<3.4.0,>=3.3.1; extra == "kafka"
 Provides-Extra: postgres
 Requires-Dist: soda-core-postgres<3.4.0,>=3.3.1; extra == "postgres"
 Provides-Extra: s3
-Requires-Dist: s3fs==2024.6.0; extra == "s3"
+Requires-Dist: s3fs==2024.6.1; extra == "s3"
 Provides-Extra: snowflake
-Requires-Dist: snowflake-connector-python[pandas]<3.11,>=3.6; extra == "snowflake"
+Requires-Dist: snowflake-connector-python[pandas]<3.12,>=3.6; extra == "snowflake"
 Requires-Dist: soda-core-snowflake<3.4.0,>=3.3.1; extra == "snowflake"
 Provides-Extra: sqlserver
 Requires-Dist: soda-core-sqlserver<3.4.0,>=3.3.1; extra == "sqlserver"
+Provides-Extra: trino
+Requires-Dist: soda-core-trino<3.4.0,>=3.3.1; extra == "trino"
 Provides-Extra: all
-Requires-Dist: datacontract-cli[bigquery,databricks,deltalake,kafka,postgres,s3,snowflake,sqlserver]; extra == "all"
+Requires-Dist: datacontract-cli[bigquery,databricks,deltalake,kafka,postgres,s3,snowflake,sqlserver,trino]; extra == "all"
 Provides-Extra: dev
 Requires-Dist: datacontract-cli[all]; extra == "dev"
 Requires-Dist: httpx==0.27.0; extra == "dev"
@@ -65,7 +67,9 @@ Requires-Dist: pytest-xdist; extra == "dev"
 Requires-Dist: moto; extra == "dev"
 Requires-Dist: pymssql==2.3.0; extra == "dev"
 Requires-Dist: kafka-python; extra == "dev"
-Requires-Dist: testcontainers~=4.5.0; extra == "dev"
+Requires-Dist: trino==0.328.0; extra == "dev"
+Requires-Dist: testcontainers<4.8,>=4.5; extra == "dev"
+Requires-Dist: testcontainers[core]; extra == "dev"
 Requires-Dist: testcontainers[minio]; extra == "dev"
 Requires-Dist: testcontainers[postgres]; extra == "dev"
 Requires-Dist: testcontainers[kafka]; extra == "dev"
@@ -258,17 +262,18 @@ pip install datacontract-cli[all]
 
 A list of available extras:
 
-| Dependency              | Installation Command                                        |
-|-------------------------|-------------------------------------------------------------|
-| Avro Support            | `pip install datacontract-cli[avro]`                        |
-| Google BigQuery         | `pip install datacontract-cli[bigquery]`                    |
-| Databricks Integration  | `pip install datacontract-cli[databricks]`                  |
-| Deltalake Integration   | `pip install datacontract-cli[deltalake]`                  |
-| Kafka Integration       | `pip install datacontract-cli[kafka]`                       |
-| PostgreSQL Integration  | `pip install datacontract-cli[postgres]`                    |
-| S3 Integration          | `pip install datacontract-cli[s3]`                          |
-| Snowflake Integration   | `pip install datacontract-cli[snowflake]`                   |
-| Microsoft SQL Server    | `pip install datacontract-cli[sqlserver]`                   |
+| Dependency             | Installation Command                       |
+|------------------------|--------------------------------------------|
+| Avro Support           | `pip install datacontract-cli[avro]`       |
+| Google BigQuery        | `pip install datacontract-cli[bigquery]`   |
+| Databricks Integration | `pip install datacontract-cli[databricks]` |
+| Deltalake Integration  | `pip install datacontract-cli[deltalake]`  |
+| Kafka Integration      | `pip install datacontract-cli[kafka]`      |
+| PostgreSQL Integration | `pip install datacontract-cli[postgres]`   |
+| S3 Integration         | `pip install datacontract-cli[s3]`         |
+| Snowflake Integration  | `pip install datacontract-cli[snowflake]`  |
+| Microsoft SQL Server   | `pip install datacontract-cli[sqlserver]`  |
+| Trino                  | `pip install datacontract-cli[trino]`      |
 
 
 
@@ -384,6 +389,7 @@ Supported server types:
 - [snowflake](#snowflake)
 - [kafka](#kafka)
 - [postgres](#postgres)
+- [trino](#trino)
 - [local](#local)
 
 Supported formats:
@@ -429,11 +435,12 @@ servers:
 
 #### Environment Variables
 
-| Environment Variable              | Example                       | Description           |
-|-----------------------------------|-------------------------------|-----------------------|
-| `DATACONTRACT_S3_REGION`            | `eu-central-1`                  | Region of S3 bucket   |
-| `DATACONTRACT_S3_ACCESS_KEY_ID`     | `AKIAXV5Q5QABCDEFGH`            | AWS Access Key ID     |
-| `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key |
+| Environment Variable                | Example                         | Description                            |
+|-------------------------------------|---------------------------------|----------------------------------------|
+| `DATACONTRACT_S3_REGION`            | `eu-central-1`                  | Region of S3 bucket                    |
+| `DATACONTRACT_S3_ACCESS_KEY_ID`     | `AKIAXV5Q5QABCDEFGH`            | AWS Access Key ID                      |
+| `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key                  |
+| `DATACONTRACT_S3_SESSION_TOKEN`     | `AQoDYXdzEJr...`                | AWS temporary session token (optional) |
 
 
 
@@ -724,6 +731,35 @@ models:
 | `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
 
 
+### Trino
+
+Data Contract CLI can test data in Trino.
+
+#### Example
+
+datacontract.yaml
+```yaml
+servers:
+  trino:
+    type: trino
+    host: localhost
+    port: 8080
+    catalog: my_catalog
+    schema: my_schema
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields:
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+#### Environment Variables
+
+| Environment Variable          | Example            | Description |
+|-------------------------------|--------------------|-------------|
+| `DATACONTRACT_TRINO_USERNAME` | `trino`            | Username    |
+| `DATACONTRACT_TRINO_PASSWORD` | `mysecretpassword` | Password    |
 
 
 
@@ -742,7 +778,7 @@ models:
 │ *  --format        [jsonschema|pydantic-model|sodacl|dbt|dbt-sources|db  The export format. [default: None] [required]         │
 │                    t-staging-sql|odcs|rdf|avro|protobuf|great-expectati                                                        │
 │                    ons|terraform|avro-idl|sql|sql-query|html|go|bigquer                                                        │
-│                    y|dbml]                                                                                                     │
+│                    y|dbml|spark]                                                                                                     │
 │    --output        PATH                                                  Specify the file path where the exported data will be │
 │                                                                          saved. If no path is provided, the output will be     │
 │                                                                          printed to stdout.                                    │
@@ -792,6 +828,7 @@ Available export options:
 | `go`                 | Export to Go types                                      | ✅     |
 | `pydantic-model`     | Export to pydantic models                               | ✅     |
 | `DBML`               | Export to a DBML Diagram description                    | ✅     |
+| `spark`              | Export to a Spark StructType                            | ✅     |
 | Missing something?   | Please create an issue on GitHub                        | TBD    |
 
 #### Great Expectations
@@ -838,6 +875,10 @@ The export function converts the logical data types of the datacontract into the
 if a server is selected via the `--server` option (based on the `type` of that server). If no server is selected, the
 logical data types are exported.
 
+#### Spark
+
+The export function converts the data contract specification into a StructType Spark schema. The returned value is a Python code picture of the model schemas.  
+Spark DataFrame schema is defined as StructType. For more details about Spark Data Types please see [the spark documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) 
 
 #### Avro
 
@@ -1214,6 +1255,121 @@ Examples: Removing or renaming models and fields.
   $ datacontract changelog datacontract-from-pr.yaml datacontract-from-main.yaml
   ```
 
+## Customizing Exporters and Importers
+ 
+### Custom Exporter
+Using the exporter factory to add a new custom exporter
+```python
+
+from datacontract.data_contract import DataContract
+from datacontract.export.exporter import Exporter
+from datacontract.export.exporter_factory import exporter_factory
+
+
+# Create a custom class that implements export method
+class CustomExporter(Exporter):
+    def export(self, data_contract, model, server, sql_server_type, export_args) -> dict:
+        result = {
+            "title": data_contract.info.title,
+            "version": data_contract.info.version,
+            "description": data_contract.info.description,
+            "email": data_contract.info.contact.email,
+            "url": data_contract.info.contact.url,
+            "model": model,
+            "model_columns": ", ".join(list(data_contract.models.get(model).fields.keys())),
+            "export_args": export_args,
+            "custom_args": export_args.get("custom_arg", ""),
+        }
+        return result
+
+
+# Register the new custom class into factory
+exporter_factory.register_exporter("custom", CustomExporter)
+
+
+if __name__ == "__main__":
+    # Create a DataContract instance
+    data_contract = DataContract(
+        data_contract_file="/path/datacontract.yaml"
+    )
+    # call export
+    result = data_contract.export(
+        export_format="custom", model="orders", server="production", custom_arg="my_custom_arg"
+    )
+    print(result)
+
+```
+Output
+```python
+{
+ 'title': 'Orders Unit Test', 
+ 'version': '1.0.0', 
+ 'description': 'The orders data contract', 
+ 'email': 'team-orders@example.com', 
+ 'url': 'https://wiki.example.com/teams/checkout', 
+ 'model': 'orders', 
+ 'model_columns': 'order_id, order_total, order_status', 
+ 'export_args': {'server': 'production', 'custom_arg': 'my_custom_arg'}, 
+ 'custom_args': 'my_custom_arg'
+}
+```
+  
+### Custom Importer
+Using the importer factory to add a new custom importer
+```python
+
+from datacontract.model.data_contract_specification import DataContractSpecification
+from datacontract.data_contract import DataContract
+from datacontract.imports.importer import Importer
+from datacontract.imports.importer_factory import importer_factory
+import json
+
+# Create a custom class that implements import_source method
+class CustomImporter(Importer):
+    def import_source(
+        self, data_contract_specification: DataContractSpecification, source: str, import_args: dict
+    ) -> dict:
+        source_dict = json.loads(source)
+        data_contract_specification.id = source_dict.get("id_custom")
+        data_contract_specification.info.title = source_dict.get("title")
+        data_contract_specification.info.description = source_dict.get("description_from_app")
+
+        return data_contract_specification
+
+
+# Register the new custom class into factory
+importer_factory.register_importer("custom_company_importer", CustomImporter)
+
+
+if __name__ == "__main__":
+    # get a custom da
+    json_from_custom_app = '{"id_custom":"uuid-custom","version":"0.0.2", "title":"my_custom_imported_data", "description_from_app": "Custom contract description"}'
+    # Create a DataContract instance
+    data_contract = DataContract()
+
+    # call import_from
+    result = data_contract.import_from_source(
+        format="custom_company_importer", data_contract_specification=DataContract.init(), source=json_from_custom_app
+    )
+    print(dict(result))
+
+```
+Output
+
+```python
+{
+  'dataContractSpecification': '0.9.3', 
+  'id': 'uuid-custom', 
+  'info': Info(title='my_custom_imported_data', version='0.0.1', status=None, description='Custom contract description', owner=None, contact=None), 
+  'servers': {}, 
+  'terms': None, 
+  'models': {}, 
+  'definitions': {}, 
+  'examples': [], 
+  'quality': None, 
+  'servicelevels': None
+}
+```
 ## Development Setup
 
 Python base interpreter should be 3.11.x (unless working on 3.12 release candidate).
@@ -1263,7 +1419,26 @@ docker compose run --rm datacontract --version
 
 This command runs the container momentarily to check the version of the `datacontract` CLI. The `--rm` flag ensures that the container is automatically removed after the command executes, keeping your environment clean.
 
+## Use with pre-commit
+
+To run `datacontract-cli` as part of a [pre-commit](https://pre-commit.com/) workflow, add something like the below to the `repos` list in the project's `.pre-commit-config.yaml`:
+
+```yaml
+repos:
+  - repo: https://github.com/datacontract/datacontract-cli
+    rev: "v0.10.9"
+    hooks:
+      - id: datacontract-lint
+      - id: datacontract-test
+        args: ["--server", "production"]
+```
+
+### Available Hook IDs
 
+| Hook ID           | Description              | Dependency |
+| ----------------- | ------------------------ | ---------- |
+| datacontract-lint | Runs the lint subcommand. | Python3    |
+| datacontract-test | Runs the test subcommand. Please look at [test](#test) section for all available arguments. | Python3 |
 
 ## Release Steps
 

datacontract_cli-0.10.8/datacontract_cli.egg-info/PKG-INFO → datacontract_cli-0.10.9/README.md

@@ -1,76 +1,3 @@
-Metadata-Version: 2.1
-Name: datacontract-cli
-Version: 0.10.8
-Summary: Test data contracts
-Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>
-Project-URL: Homepage, https://cli.datacontract.com
-Project-URL: Issues, https://github.com/datacontract/cli/issues
-Classifier: Programming Language :: Python :: 3
-Classifier: License :: OSI Approved :: MIT License
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.10
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: typer[all]<0.13,>=0.9
-Requires-Dist: pydantic<2.8.0,>=2.5.3
-Requires-Dist: pyyaml~=6.0.1
-Requires-Dist: requests<2.33,>=2.31
-Requires-Dist: fastapi==0.111.0
-Requires-Dist: fastparquet==2024.5.0
-Requires-Dist: python-multipart==0.0.9
-Requires-Dist: rich~=13.7.0
-Requires-Dist: simple-ddl-parser==1.5.1
-Requires-Dist: soda-core-duckdb<3.4.0,>=3.3.1
-Requires-Dist: setuptools>=60
-Requires-Dist: duckdb==1.0.0
-Requires-Dist: fastjsonschema<2.21.0,>=2.19.1
-Requires-Dist: python-dotenv~=1.0.0
-Requires-Dist: rdflib==7.0.0
-Requires-Dist: opentelemetry-exporter-otlp-proto-grpc~=1.16
-Requires-Dist: opentelemetry-exporter-otlp-proto-http~=1.16
-Requires-Dist: boto3<1.34.130,>=1.34.41
-Requires-Dist: botocore<1.34.128,>=1.34.41
-Requires-Dist: jinja_partials>=0.2.1
-Provides-Extra: avro
-Requires-Dist: avro==1.11.3; extra == "avro"
-Provides-Extra: bigquery
-Requires-Dist: soda-core-bigquery<3.4.0,>=3.3.1; extra == "bigquery"
-Provides-Extra: databricks
-Requires-Dist: soda-core-spark-df<3.4.0,>=3.3.1; extra == "databricks"
-Requires-Dist: databricks-sql-connector<3.2.0,>=3.1.2; extra == "databricks"
-Requires-Dist: soda-core-spark[databricks]<3.4.0,>=3.3.1; extra == "databricks"
-Provides-Extra: deltalake
-Requires-Dist: deltalake<0.19,>=0.17; extra == "deltalake"
-Provides-Extra: kafka
-Requires-Dist: datacontract-cli[avro]; extra == "kafka"
-Requires-Dist: soda-core-spark-df<3.4.0,>=3.3.1; extra == "kafka"
-Provides-Extra: postgres
-Requires-Dist: soda-core-postgres<3.4.0,>=3.3.1; extra == "postgres"
-Provides-Extra: s3
-Requires-Dist: s3fs==2024.6.0; extra == "s3"
-Provides-Extra: snowflake
-Requires-Dist: snowflake-connector-python[pandas]<3.11,>=3.6; extra == "snowflake"
-Requires-Dist: soda-core-snowflake<3.4.0,>=3.3.1; extra == "snowflake"
-Provides-Extra: sqlserver
-Requires-Dist: soda-core-sqlserver<3.4.0,>=3.3.1; extra == "sqlserver"
-Provides-Extra: all
-Requires-Dist: datacontract-cli[bigquery,databricks,deltalake,kafka,postgres,s3,snowflake,sqlserver]; extra == "all"
-Provides-Extra: dev
-Requires-Dist: datacontract-cli[all]; extra == "dev"
-Requires-Dist: httpx==0.27.0; extra == "dev"
-Requires-Dist: ruff; extra == "dev"
-Requires-Dist: pre-commit~=3.7.1; extra == "dev"
-Requires-Dist: pytest; extra == "dev"
-Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: moto; extra == "dev"
-Requires-Dist: pymssql==2.3.0; extra == "dev"
-Requires-Dist: kafka-python; extra == "dev"
-Requires-Dist: testcontainers~=4.5.0; extra == "dev"
-Requires-Dist: testcontainers[minio]; extra == "dev"
-Requires-Dist: testcontainers[postgres]; extra == "dev"
-Requires-Dist: testcontainers[kafka]; extra == "dev"
-Requires-Dist: testcontainers[mssql]; extra == "dev"
-
 # Data Contract CLI
 
 <p>
@@ -258,17 +185,18 @@ pip install datacontract-cli[all]
 
 A list of available extras:
 
-| Dependency              | Installation Command                                        |
-|-------------------------|-------------------------------------------------------------|
-| Avro Support            | `pip install datacontract-cli[avro]`                        |
-| Google BigQuery         | `pip install datacontract-cli[bigquery]`                    |
-| Databricks Integration  | `pip install datacontract-cli[databricks]`                  |
-| Deltalake Integration   | `pip install datacontract-cli[deltalake]`                  |
-| Kafka Integration       | `pip install datacontract-cli[kafka]`                       |
-| PostgreSQL Integration  | `pip install datacontract-cli[postgres]`                    |
-| S3 Integration          | `pip install datacontract-cli[s3]`                          |
-| Snowflake Integration   | `pip install datacontract-cli[snowflake]`                   |
-| Microsoft SQL Server    | `pip install datacontract-cli[sqlserver]`                   |
+| Dependency             | Installation Command                       |
+|------------------------|--------------------------------------------|
+| Avro Support           | `pip install datacontract-cli[avro]`       |
+| Google BigQuery        | `pip install datacontract-cli[bigquery]`   |
+| Databricks Integration | `pip install datacontract-cli[databricks]` |
+| Deltalake Integration  | `pip install datacontract-cli[deltalake]`  |
+| Kafka Integration      | `pip install datacontract-cli[kafka]`      |
+| PostgreSQL Integration | `pip install datacontract-cli[postgres]`   |
+| S3 Integration         | `pip install datacontract-cli[s3]`         |
+| Snowflake Integration  | `pip install datacontract-cli[snowflake]`  |
+| Microsoft SQL Server   | `pip install datacontract-cli[sqlserver]`  |
+| Trino                  | `pip install datacontract-cli[trino]`      |
 
 
 
@@ -384,6 +312,7 @@ Supported server types:
 - [snowflake](#snowflake)
 - [kafka](#kafka)
 - [postgres](#postgres)
+- [trino](#trino)
 - [local](#local)
 
 Supported formats:
@@ -429,11 +358,12 @@ servers:
 
 #### Environment Variables
 
-| Environment Variable              | Example                       | Description           |
-|-----------------------------------|-------------------------------|-----------------------|
-| `DATACONTRACT_S3_REGION`            | `eu-central-1`                  | Region of S3 bucket   |
-| `DATACONTRACT_S3_ACCESS_KEY_ID`     | `AKIAXV5Q5QABCDEFGH`            | AWS Access Key ID     |
-| `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key |
+| Environment Variable                | Example                         | Description                            |
+|-------------------------------------|---------------------------------|----------------------------------------|
+| `DATACONTRACT_S3_REGION`            | `eu-central-1`                  | Region of S3 bucket                    |
+| `DATACONTRACT_S3_ACCESS_KEY_ID`     | `AKIAXV5Q5QABCDEFGH`            | AWS Access Key ID                      |
+| `DATACONTRACT_S3_SECRET_ACCESS_KEY` | `93S7LRrJcqLaaaa/XXXXXXXXXXXXX` | AWS Secret Access Key                  |
+| `DATACONTRACT_S3_SESSION_TOKEN`     | `AQoDYXdzEJr...`                | AWS temporary session token (optional) |
 
 
 
@@ -724,6 +654,35 @@ models:
 | `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
 
 
+### Trino
+
+Data Contract CLI can test data in Trino.
+
+#### Example
+
+datacontract.yaml
+```yaml
+servers:
+  trino:
+    type: trino
+    host: localhost
+    port: 8080
+    catalog: my_catalog
+    schema: my_schema
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields:
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+#### Environment Variables
+
+| Environment Variable          | Example            | Description |
+|-------------------------------|--------------------|-------------|
+| `DATACONTRACT_TRINO_USERNAME` | `trino`            | Username    |
+| `DATACONTRACT_TRINO_PASSWORD` | `mysecretpassword` | Password    |
 
 
 
@@ -742,7 +701,7 @@ models:
 │ *  --format        [jsonschema|pydantic-model|sodacl|dbt|dbt-sources|db  The export format. [default: None] [required]         │
 │                    t-staging-sql|odcs|rdf|avro|protobuf|great-expectati                                                        │
 │                    ons|terraform|avro-idl|sql|sql-query|html|go|bigquer                                                        │
-│                    y|dbml]                                                                                                     │
+│                    y|dbml|spark]                                                                                                     │
 │    --output        PATH                                                  Specify the file path where the exported data will be │
 │                                                                          saved. If no path is provided, the output will be     │
 │                                                                          printed to stdout.                                    │
@@ -792,6 +751,7 @@ Available export options:
 | `go`                 | Export to Go types                                      | ✅     |
 | `pydantic-model`     | Export to pydantic models                               | ✅     |
 | `DBML`               | Export to a DBML Diagram description                    | ✅     |
+| `spark`              | Export to a Spark StructType                            | ✅     |
 | Missing something?   | Please create an issue on GitHub                        | TBD    |
 
 #### Great Expectations
@@ -838,6 +798,10 @@ The export function converts the logical data types of the datacontract into the
 if a server is selected via the `--server` option (based on the `type` of that server). If no server is selected, the
 logical data types are exported.
 
+#### Spark
+
+The export function converts the data contract specification into a StructType Spark schema. The returned value is a Python code picture of the model schemas.  
+Spark DataFrame schema is defined as StructType. For more details about Spark Data Types please see [the spark documentation](https://spark.apache.org/docs/latest/sql-ref-datatypes.html) 
 
 #### Avro
 
@@ -1214,6 +1178,121 @@ Examples: Removing or renaming models and fields.
   $ datacontract changelog datacontract-from-pr.yaml datacontract-from-main.yaml
   ```
 
+## Customizing Exporters and Importers
+ 
+### Custom Exporter
+Using the exporter factory to add a new custom exporter
+```python
+
+from datacontract.data_contract import DataContract
+from datacontract.export.exporter import Exporter
+from datacontract.export.exporter_factory import exporter_factory
+
+
+# Create a custom class that implements export method
+class CustomExporter(Exporter):
+    def export(self, data_contract, model, server, sql_server_type, export_args) -> dict:
+        result = {
+            "title": data_contract.info.title,
+            "version": data_contract.info.version,
+            "description": data_contract.info.description,
+            "email": data_contract.info.contact.email,
+            "url": data_contract.info.contact.url,
+            "model": model,
+            "model_columns": ", ".join(list(data_contract.models.get(model).fields.keys())),
+            "export_args": export_args,
+            "custom_args": export_args.get("custom_arg", ""),
+        }
+        return result
+
+
+# Register the new custom class into factory
+exporter_factory.register_exporter("custom", CustomExporter)
+
+
+if __name__ == "__main__":
+    # Create a DataContract instance
+    data_contract = DataContract(
+        data_contract_file="/path/datacontract.yaml"
+    )
+    # call export
+    result = data_contract.export(
+        export_format="custom", model="orders", server="production", custom_arg="my_custom_arg"
+    )
+    print(result)
+
+```
+Output
+```python
+{
+ 'title': 'Orders Unit Test', 
+ 'version': '1.0.0', 
+ 'description': 'The orders data contract', 
+ 'email': 'team-orders@example.com', 
+ 'url': 'https://wiki.example.com/teams/checkout', 
+ 'model': 'orders', 
+ 'model_columns': 'order_id, order_total, order_status', 
+ 'export_args': {'server': 'production', 'custom_arg': 'my_custom_arg'}, 
+ 'custom_args': 'my_custom_arg'
+}
+```
+  
+### Custom Importer
+Using the importer factory to add a new custom importer
+```python
+
+from datacontract.model.data_contract_specification import DataContractSpecification
+from datacontract.data_contract import DataContract
+from datacontract.imports.importer import Importer
+from datacontract.imports.importer_factory import importer_factory
+import json
+
+# Create a custom class that implements import_source method
+class CustomImporter(Importer):
+    def import_source(
+        self, data_contract_specification: DataContractSpecification, source: str, import_args: dict
+    ) -> dict:
+        source_dict = json.loads(source)
+        data_contract_specification.id = source_dict.get("id_custom")
+        data_contract_specification.info.title = source_dict.get("title")
+        data_contract_specification.info.description = source_dict.get("description_from_app")
+
+        return data_contract_specification
+
+
+# Register the new custom class into factory
+importer_factory.register_importer("custom_company_importer", CustomImporter)
+
+
+if __name__ == "__main__":
+    # get a custom da
+    json_from_custom_app = '{"id_custom":"uuid-custom","version":"0.0.2", "title":"my_custom_imported_data", "description_from_app": "Custom contract description"}'
+    # Create a DataContract instance
+    data_contract = DataContract()
+
+    # call import_from
+    result = data_contract.import_from_source(
+        format="custom_company_importer", data_contract_specification=DataContract.init(), source=json_from_custom_app
+    )
+    print(dict(result))
+
+```
+Output
+
+```python
+{
+  'dataContractSpecification': '0.9.3', 
+  'id': 'uuid-custom', 
+  'info': Info(title='my_custom_imported_data', version='0.0.1', status=None, description='Custom contract description', owner=None, contact=None), 
+  'servers': {}, 
+  'terms': None, 
+  'models': {}, 
+  'definitions': {}, 
+  'examples': [], 
+  'quality': None, 
+  'servicelevels': None
+}
+```
 ## Development Setup
 
 Python base interpreter should be 3.11.x (unless working on 3.12 release candidate).
@@ -1263,7 +1342,26 @@ docker compose run --rm datacontract --version
 
 This command runs the container momentarily to check the version of the `datacontract` CLI. The `--rm` flag ensures that the container is automatically removed after the command executes, keeping your environment clean.
 
+## Use with pre-commit
+
+To run `datacontract-cli` as part of a [pre-commit](https://pre-commit.com/) workflow, add something like the below to the `repos` list in the project's `.pre-commit-config.yaml`:
+
+```yaml
+repos:
+  - repo: https://github.com/datacontract/datacontract-cli
+    rev: "v0.10.9"
+    hooks:
+      - id: datacontract-lint
+      - id: datacontract-test
+        args: ["--server", "production"]
+```
+
+### Available Hook IDs
 
+| Hook ID           | Description              | Dependency |
+| ----------------- | ------------------------ | ---------- |
+| datacontract-lint | Runs the lint subcommand. | Python3    |
+| datacontract-test | Runs the test subcommand. Please look at [test](#test) section for all available arguments. | Python3 |
 
 ## Release Steps