datacontract-cli 0.11.6__tar.gz → 0.11.7__tar.gz

{datacontract_cli-0.11.6/datacontract_cli.egg-info → datacontract_cli-0.11.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datacontract-cli
-Version: 0.11.6
+Version: 0.11.7
 Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
 Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 License-Expression: MIT
@@ -43,19 +43,22 @@ Requires-Dist: databricks-sdk<0.86.0; extra == "databricks"
 Requires-Dist: pyspark<5.0.0,>=3.5.0; extra == "databricks"
 Requires-Dist: numpy<2.0.0,>=1.26.4; extra == "databricks"
 Provides-Extra: iceberg
-Requires-Dist: pyiceberg==0.10.0; extra == "iceberg"
+Requires-Dist: pyiceberg==0.11.0; extra == "iceberg"
 Provides-Extra: kafka
 Requires-Dist: datacontract-cli[avro]; extra == "kafka"
 Requires-Dist: soda-core-spark-df<3.6.0,>=3.3.20; extra == "kafka"
 Requires-Dist: pyspark<5.0.0,>=3.5.0; extra == "kafka"
 Requires-Dist: numpy<2.0.0,>=1.26.4; extra == "kafka"
+Provides-Extra: mysql
+Requires-Dist: soda-core-mysql<3.6.0,>=3.3.20; extra == "mysql"
+Requires-Dist: mysql-connector-python<10.0.0,>=8.0.30; extra == "mysql"
 Provides-Extra: postgres
 Requires-Dist: soda-core-postgres<3.6.0,>=3.3.20; extra == "postgres"
 Provides-Extra: s3
 Requires-Dist: s3fs<2027.0.0,>=2025.2.0; extra == "s3"
 Requires-Dist: aiobotocore<3.2.0,>=2.17.0; extra == "s3"
 Provides-Extra: snowflake
-Requires-Dist: snowflake-connector-python[pandas]<4.3,>=3.6; extra == "snowflake"
+Requires-Dist: snowflake-connector-python[pandas]<4.4,>=3.6; extra == "snowflake"
 Requires-Dist: soda-core-snowflake<3.6.0,>=3.3.20; extra == "snowflake"
 Provides-Extra: sqlserver
 Requires-Dist: soda-core-sqlserver<3.6.0,>=3.3.20; extra == "sqlserver"
@@ -82,7 +85,7 @@ Requires-Dist: uvicorn==0.40.0; extra == "api"
 Provides-Extra: protobuf
 Requires-Dist: grpcio-tools>=1.53; extra == "protobuf"
 Provides-Extra: all
-Requires-Dist: datacontract-cli[api,athena,bigquery,csv,databricks,dbml,dbt,duckdb,excel,iceberg,kafka,oracle,parquet,postgres,protobuf,rdf,s3,snowflake,sqlserver,trino]; extra == "all"
+Requires-Dist: datacontract-cli[api,athena,bigquery,csv,databricks,dbml,dbt,duckdb,excel,iceberg,kafka,mysql,oracle,parquet,postgres,protobuf,rdf,s3,snowflake,sqlserver,trino]; extra == "all"
 Provides-Extra: dev
 Requires-Dist: datacontract-cli[all]; extra == "dev"
 Requires-Dist: httpx==0.28.1; extra == "dev"
@@ -94,8 +97,8 @@ Requires-Dist: pre-commit<4.6.0,>=3.7.1; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
 Requires-Dist: pymssql==2.3.13; extra == "dev"
-Requires-Dist: ruff==0.14.3; extra == "dev"
-Requires-Dist: testcontainers[kafka,minio,mssql,postgres]==4.14.1; extra == "dev"
+Requires-Dist: ruff==0.15.2; extra == "dev"
+Requires-Dist: testcontainers[kafka,minio,mssql,mysql,postgres]==4.14.1; extra == "dev"
 Requires-Dist: trino==0.336.0; extra == "dev"
 Dynamic: license-file
 
@@ -340,6 +343,7 @@ A list of available extras:
 | DuckDB (local/S3/GCS/Azure file testing) | `pip install datacontract-cli[duckdb]` |
 | Iceberg                 | `pip install datacontract-cli[iceberg]`    |
 | Kafka Integration       | `pip install datacontract-cli[kafka]`      |
+| MySQL Integration       | `pip install datacontract-cli[mysql]`      |
 | PostgreSQL Integration  | `pip install datacontract-cli[postgres]`   |
 | S3 Integration          | `pip install datacontract-cli[s3]`         |
 | Snowflake Integration   | `pip install datacontract-cli[snowflake]`  |
@@ -496,6 +500,7 @@ Supported server types:
 - [snowflake](#snowflake)
 - [kafka](#kafka)
 - [postgres](#postgres)
+- [mysql](#mysql)
 - [trino](#trino)
 - [impala](#impala)
 - [api](#api)
@@ -997,6 +1002,36 @@ models:
 | `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
 
 
+#### MySQL
+
+Data Contract CLI can test data in MySQL or MySQL-compliant databases (e.g., MariaDB).
+
+##### Example
+
+datacontract.yaml
+```yaml
+servers:
+  mysql:
+    type: mysql
+    host: localhost
+    port: 3306
+    database: mydb
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields:
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+##### Environment Variables
+
+| Environment Variable          | Example            | Description |
+|-------------------------------|--------------------|-------------|
+| `DATACONTRACT_MYSQL_USERNAME` | `root`             | Username    |
+| `DATACONTRACT_MYSQL_PASSWORD` | `mysecretpassword` | Password    |
+
+
 #### Trino
 
 Data Contract CLI can test data in Trino.

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/README.md

@@ -239,6 +239,7 @@ A list of available extras:
 | DuckDB (local/S3/GCS/Azure file testing) | `pip install datacontract-cli[duckdb]` |
 | Iceberg                 | `pip install datacontract-cli[iceberg]`    |
 | Kafka Integration       | `pip install datacontract-cli[kafka]`      |
+| MySQL Integration       | `pip install datacontract-cli[mysql]`      |
 | PostgreSQL Integration  | `pip install datacontract-cli[postgres]`   |
 | S3 Integration          | `pip install datacontract-cli[s3]`         |
 | Snowflake Integration   | `pip install datacontract-cli[snowflake]`  |
@@ -395,6 +396,7 @@ Supported server types:
 - [snowflake](#snowflake)
 - [kafka](#kafka)
 - [postgres](#postgres)
+- [mysql](#mysql)
 - [trino](#trino)
 - [impala](#impala)
 - [api](#api)
@@ -896,6 +898,36 @@ models:
 | `DATACONTRACT_POSTGRES_PASSWORD` | `mysecretpassword` | Password    |
 
 
+#### MySQL
+
+Data Contract CLI can test data in MySQL or MySQL-compliant databases (e.g., MariaDB).
+
+##### Example
+
+datacontract.yaml
+```yaml
+servers:
+  mysql:
+    type: mysql
+    host: localhost
+    port: 3306
+    database: mydb
+models:
+  my_table_1: # corresponds to a table
+    type: table
+    fields:
+      my_column_1: # corresponds to a column
+        type: varchar
+```
+
+##### Environment Variables
+
+| Environment Variable          | Example            | Description |
+|-------------------------------|--------------------|-------------|
+| `DATACONTRACT_MYSQL_USERNAME` | `root`             | Username    |
+| `DATACONTRACT_MYSQL_PASSWORD` | `mysecretpassword` | Password    |
+
+
 #### Trino
 
 Data Contract CLI can test data in Trino.

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/catalog/catalog.py

@@ -65,6 +65,7 @@ class _SpecView:
 
     info: _InfoView
     models: dict
+    tags: list[str] | None = None
 
 
 @dataclass
@@ -109,6 +110,7 @@ class DataContractView:
                 description=description,
             ),
             models=models,
+            tags=self.odcs.tags,
         )
 
 

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/engines/data_contract_checks.py

@@ -25,6 +25,18 @@ class QuotingConfig:
     quote_model_name_with_backticks: bool = False
 
 
+def _escape_sql_string_values(values):
+    """Escape single quotes in string values for use in SodaCL checks.
+
+    SodaCL renders valid/invalid values directly into SQL IN clauses, so single
+    quotes inside string values (e.g. "peter's") must be escaped as two single
+    quotes (e.g. "peter''s") to produce valid SQL.
+    """
+    if values is None:
+        return None
+    return [v.replace("'", "''") if isinstance(v, str) else v for v in values]
+
+
 def _quote_field_name(field_name: str, quoting_config: QuotingConfig) -> str:
     """Quote a field name according to the quoting configuration."""
     if field_name is None:
@@ -85,9 +97,9 @@ def to_schema_checks(schema_object: SchemaObject, server: Server) -> List[Check]
     type1 = server.type if server and server.type else None
     config = QuotingConfig(
         quote_field_name=type1 in ["postgres", "sqlserver", "snowflake", "azure", "s3", "gcs", "local"],
-        quote_field_name_with_backticks=type1 in ["databricks"],
+        quote_field_name_with_backticks=type1 in ["databricks", "bigquery", "mysql"],
         quote_model_name=type1 in ["postgres", "sqlserver", "snowflake", "azure", "s3", "gcs", "local"],
-        quote_model_name_with_backticks=type1 == "bigquery",
+        quote_model_name_with_backticks=type1 in ["databricks", "bigquery", "mysql"],
     )
     quoting_config = config
 
@@ -432,7 +444,7 @@ def check_property_not_equal(
             {
                 f"invalid_count({field_name_for_soda}) = 0": {
                     "name": check_key,
-                    "invalid values": [value],
+                    "invalid values": _escape_sql_string_values([value]),
                 },
             }
         ],
@@ -461,7 +473,7 @@ def check_property_enum(model_name: str, field_name: str, enum: list, quoting_co
             {
                 f"invalid_count({field_name_for_soda}) = 0": {
                     "name": check_key,
-                    "valid values": enum,
+                    "valid values": _escape_sql_string_values(enum),
                 },
             }
         ],
@@ -639,7 +651,7 @@ def check_property_invalid_values(
     }
 
     if valid_values is not None:
-        sodacl_check_config["valid values"] = valid_values
+        sodacl_check_config["valid values"] = _escape_sql_string_values(valid_values)
 
     sodacl_check_dict = {
         checks_for(model_name, quoting_config, check_type): [
@@ -681,7 +693,7 @@ def check_property_missing_values(
     if missing_values is not None:
         filtered_missing_values = [v for v in missing_values if v is not None]
         if filtered_missing_values:
-            sodacl_check_config["missing values"] = filtered_missing_values
+            sodacl_check_config["missing values"] = _escape_sql_string_values(filtered_missing_values)
 
     sodacl_check_dict = {
         checks_for(model_name, quoting_config, check_type): [

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/engines/soda/check_soda_execute.py

@@ -16,6 +16,7 @@ from datacontract.engines.soda.connections.databricks import to_databricks_soda_
 from datacontract.engines.soda.connections.duckdb_connection import get_duckdb_connection
 from datacontract.engines.soda.connections.impala import to_impala_soda_configuration
 from datacontract.engines.soda.connections.kafka import create_spark_session, read_kafka_topic
+from datacontract.engines.soda.connections.mysql import to_mysql_soda_configuration
 from datacontract.engines.soda.connections.postgres import to_postgres_soda_configuration
 from datacontract.engines.soda.connections.snowflake import to_snowflake_soda_configuration
 from datacontract.engines.soda.connections.sqlserver import to_sqlserver_soda_configuration
@@ -73,6 +74,10 @@ def check_soda_execute(
         soda_configuration_str = to_postgres_soda_configuration(server)
         scan.add_configuration_yaml_str(soda_configuration_str)
         scan.set_data_source_name(server.type)
+    elif server.type == "mysql":
+        soda_configuration_str = to_mysql_soda_configuration(server)
+        scan.add_configuration_yaml_str(soda_configuration_str)
+        scan.set_data_source_name(server.type)
     elif server.type == "databricks":
         if spark is not None:
             run.log_info("Connecting to databricks via spark")

datacontract_cli-0.11.7/datacontract/engines/soda/connections/mysql.py

@@ -0,0 +1,19 @@
+import os
+
+import yaml
+
+
+def to_mysql_soda_configuration(server):
+    soda_configuration = {
+        f"data_source {server.type}": {
+            "type": "mysql",
+            "host": server.host,
+            "port": str(server.port),
+            "username": os.getenv("DATACONTRACT_MYSQL_USERNAME"),
+            "password": os.getenv("DATACONTRACT_MYSQL_PASSWORD"),
+            "database": server.database,
+        }
+    }
+
+    soda_configuration_str = yaml.dump(soda_configuration)
+    return soda_configuration_str

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/export/exporter.py

@@ -115,6 +115,8 @@ def _determine_sql_server_type(
             return "snowflake"
         elif "postgres" in server_types:
             return "postgres"
+        elif "mysql" in server_types:
+            return "mysql"
         elif "databricks" in server_types:
             return "databricks"
         else:

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/export/sql_exporter.py

@@ -46,7 +46,7 @@ def _to_sql_query(model_name: str, model_value: SchemaObject, server_type: str)
     if model_value.properties:
         for prop in model_value.properties:
             # TODO escape SQL reserved key words, probably dependent on server type
-            columns.append(prop.name)
+            columns.append(prop.physicalName or prop.name)
 
     result = "select\n"
     current_column_index = 1
@@ -82,6 +82,9 @@ def to_sql_ddl(data_contract: OpenDataContractStandard, server_type: str = "snow
         if srv.type == "postgres":
             server_type = "postgres"
             break
+        if srv.type == "mysql":
+            server_type = "mysql"
+            break
         if srv.type == "databricks":
             server_type = "databricks"
             if srv.catalog is not None and srv.schema_ is not None:
@@ -114,7 +117,8 @@ def _to_sql_table(model_name: str, model: SchemaObject, server_type: str = "snow
 
     for prop in properties:
         type_str = convert_to_sql_type(prop, server_type)
-        result += f"  {prop.name} {type_str}"
+        column_name = prop.physicalName or prop.name
+        result += f"  {column_name} {type_str}"
         if prop.required:
             result += " not null"
         if prop.primaryKey:

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/export/sql_type_converter.py

@@ -102,6 +102,8 @@ def convert_to_sql_type(field: Union[SchemaProperty, FieldLike], server_type: st
         return convert_to_snowflake(field)
     elif server_type == "postgres":
         return convert_type_to_postgres(field)
+    elif server_type == "mysql":
+        return convert_type_to_mysql(field)
     elif server_type == "dataframe":
         return convert_to_dataframe(field)
     elif server_type == "databricks":
@@ -214,6 +216,55 @@ def convert_type_to_postgres(field: Union[SchemaProperty, FieldLike]) -> None |
     return None
 
 
+# https://dev.mysql.com/doc/refman/8.0/en/data-types.html
+def convert_type_to_mysql(field: Union[SchemaProperty, FieldLike]) -> None | str:
+    mysql_type = _get_config_value(field, "mysqlType")
+    if mysql_type:
+        return mysql_type
+
+    type = _get_type(field)
+    if type is None:
+        return None
+    format = _get_format(field)
+    if type.lower() in ["string", "varchar", "text"]:
+        if format == "uuid":
+            return "varchar(36)"
+        if type.lower() == "varchar":
+            return "varchar"
+        if type.lower() == "text":
+            return "text"
+        return "varchar"
+    if type.lower() in ["timestamp", "timestamp_tz"]:
+        return "timestamp"
+    if type.lower() in ["timestamp_ntz"]:
+        return "datetime"
+    if type.lower() in ["date"]:
+        return "date"
+    if type.lower() in ["time"]:
+        return "time"
+    if type.lower() in ["number", "decimal", "numeric"]:
+        if type.lower() == "number":
+            return "decimal"
+        return type.lower()
+    if type.lower() in ["float"]:
+        return "float"
+    if type.lower() in ["double"]:
+        return "double"
+    if type.lower() in ["integer", "int"]:
+        return "int"
+    if type.lower() in ["long", "bigint"]:
+        return "bigint"
+    if type.lower() in ["boolean"]:
+        return "boolean"
+    if type.lower() in ["object", "record", "struct"]:
+        return "json"
+    if type.lower() in ["bytes"]:
+        return "blob"
+    if type.lower() in ["array"]:
+        return "json"
+    return None
+
+
 # dataframe data types:
 # https://spark.apache.org/docs/latest/sql-ref-datatypes.html
 def convert_to_dataframe(field: Union[SchemaProperty, FieldLike]) -> None | str:

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/imports/dbml_importer.py

@@ -97,7 +97,7 @@ def import_table_fields(table, references) -> List[SchemaProperty]:
         description = field.note.text if field.note else None
         is_primary_key = field.pk
         is_unique = field.unique
-        logical_type = map_type_from_sql(field.type)
+        logical_type, format = map_type_from_sql(field.type)
 
         ref = get_reference(field, references)
 
@@ -109,6 +109,7 @@ def import_table_fields(table, references) -> List[SchemaProperty]:
             name=field_name,
             logical_type=logical_type if logical_type else "string",
             physical_type=field.type,
+            format=format,
             description=description,
             required=required if required else None,
             primary_key=is_primary_key if is_primary_key else None,

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/imports/odcs_helper.py

@@ -34,6 +34,7 @@ def create_schema_object(
     description: str = None,
     business_name: str = None,
     properties: List[SchemaProperty] = None,
+    tags: List[str] = None,
 ) -> SchemaObject:
     """Create a SchemaObject (equivalent to DCS Model)."""
     schema = SchemaObject(
@@ -48,6 +49,8 @@ def create_schema_object(
         schema.businessName = business_name
     if properties:
         schema.properties = properties
+    if tags:
+        schema.tags = tags
     return schema
 
 

{datacontract_cli-0.11.6 → datacontract_cli-0.11.7}/datacontract/imports/parquet_importer.py

@@ -85,6 +85,26 @@ def map_pyarrow_field_to_property(pyarrow_field: pyarrow.Field, field_name: str)
         return create_property(name=field_name, logical_type="object", physical_type="STRUCT")
     if pyarrow.types.is_list(pyarrow_field.type):
         return create_property(name=field_name, logical_type="array", physical_type="LIST")
+    if pyarrow.types.is_large_string(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="string", physical_type="LARGE_STRING")
+    if pyarrow.types.is_large_binary(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="bytes", physical_type="LARGE_BINARY")
+    if pyarrow.types.is_large_list(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="array", physical_type="LARGE_LIST")
+    if pyarrow.types.is_float16(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="number", physical_type="HALF_FLOAT")
+    if pyarrow.types.is_time(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="time", physical_type="TIME")
+    if pyarrow.types.is_duration(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="string", physical_type="DURATION")
+    if pyarrow.types.is_fixed_size_binary(pyarrow_field.type):
+        return create_property(
+            name=field_name,
+            logical_type="bytes",
+            physical_type=f"FIXED_SIZE_BINARY({pyarrow_field.type.byte_width})",
+        )
+    if pyarrow.types.is_fixed_size_list(pyarrow_field.type):
+        return create_property(name=field_name, logical_type="array", physical_type="FIXED_SIZE_LIST")
 
     raise DataContractException(
         type="schema",