datacontract-cli 0.11.4__tar.gz → 0.11.5__tar.gz

{datacontract_cli-0.11.4/datacontract_cli.egg-info → datacontract_cli-0.11.5}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: datacontract-cli
-Version: 0.11.4
+Version: 0.11.5
 Summary: The datacontract CLI is an open source command-line tool for working with Data Contracts. It uses data contract YAML files to lint the data contract, connect to data sources and execute schema and quality tests, detect breaking changes, and export to different formats. The tool is written in Python. It can be used as a standalone CLI tool, in a CI/CD pipeline, or directly as a Python library.
 Author-email: Jochen Christ <jochen.christ@innoq.com>, Stefan Negele <stefan.negele@innoq.com>, Simon Harrer <simon.harrer@innoq.com>
 License-Expression: MIT
@@ -39,7 +39,7 @@ Provides-Extra: databricks
 Requires-Dist: soda-core-spark-df<3.6.0,>=3.3.20; extra == "databricks"
 Requires-Dist: soda-core-spark[databricks]<3.6.0,>=3.3.20; extra == "databricks"
 Requires-Dist: databricks-sql-connector<4.3.0,>=3.7.0; extra == "databricks"
-Requires-Dist: databricks-sdk<0.78.0; extra == "databricks"
+Requires-Dist: databricks-sdk<0.86.0; extra == "databricks"
 Requires-Dist: pyspark<5.0.0,>=3.5.0; extra == "databricks"
 Requires-Dist: numpy<2.0.0,>=1.26.4; extra == "databricks"
 Provides-Extra: iceberg
@@ -52,10 +52,10 @@ Requires-Dist: numpy<2.0.0,>=1.26.4; extra == "kafka"
 Provides-Extra: postgres
 Requires-Dist: soda-core-postgres<3.6.0,>=3.3.20; extra == "postgres"
 Provides-Extra: s3
-Requires-Dist: s3fs<2026.0.0,>=2025.2.0; extra == "s3"
+Requires-Dist: s3fs<2027.0.0,>=2025.2.0; extra == "s3"
 Requires-Dist: aiobotocore<3.2.0,>=2.17.0; extra == "s3"
 Provides-Extra: snowflake
-Requires-Dist: snowflake-connector-python[pandas]<4.2,>=3.6; extra == "snowflake"
+Requires-Dist: snowflake-connector-python[pandas]<4.3,>=3.6; extra == "snowflake"
 Requires-Dist: soda-core-snowflake<3.6.0,>=3.3.20; extra == "snowflake"
 Provides-Extra: sqlserver
 Requires-Dist: soda-core-sqlserver<3.6.0,>=3.3.20; extra == "sqlserver"
@@ -77,7 +77,7 @@ Requires-Dist: pyarrow>=18.1.0; extra == "parquet"
 Provides-Extra: rdf
 Requires-Dist: rdflib==7.5.0; extra == "rdf"
 Provides-Extra: api
-Requires-Dist: fastapi==0.121.2; extra == "api"
+Requires-Dist: fastapi==0.128.0; extra == "api"
 Requires-Dist: uvicorn==0.40.0; extra == "api"
 Provides-Extra: protobuf
 Requires-Dist: grpcio-tools>=1.53; extra == "protobuf"
@@ -87,15 +87,15 @@ Provides-Extra: dev
 Requires-Dist: datacontract-cli[all]; extra == "dev"
 Requires-Dist: httpx==0.28.1; extra == "dev"
 Requires-Dist: kafka-python; extra == "dev"
-Requires-Dist: minio==7.2.17; extra == "dev"
-Requires-Dist: moto==5.1.18; extra == "dev"
+Requires-Dist: minio==7.2.20; extra == "dev"
+Requires-Dist: moto==5.1.21; extra == "dev"
 Requires-Dist: pandas>=2.1.0; extra == "dev"
 Requires-Dist: pre-commit<4.6.0,>=3.7.1; extra == "dev"
 Requires-Dist: pytest; extra == "dev"
 Requires-Dist: pytest-xdist; extra == "dev"
-Requires-Dist: pymssql==2.3.9; extra == "dev"
+Requires-Dist: pymssql==2.3.13; extra == "dev"
 Requires-Dist: ruff; extra == "dev"
-Requires-Dist: testcontainers[kafka,minio,mssql,postgres]==4.13.3; extra == "dev"
+Requires-Dist: testcontainers[kafka,minio,mssql,postgres]==4.14.1; extra == "dev"
 Requires-Dist: trino==0.336.0; extra == "dev"
 Dynamic: license-file
 
@@ -622,10 +622,11 @@ servers:
 
 #### BigQuery
 
-We support authentication to BigQuery using Service Account Key. The used Service Account should include the roles:
+We support authentication to BigQuery using Service Account Key or Application Default Credentials (ADC). ADC supports Workload Identity Federation (WIF), GCE metadata server, and `gcloud auth application-default login`. The used Service Account should include the roles:
 * BigQuery Job User
 * BigQuery Data Viewer
 
+When no `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` is set, the CLI falls back to ADC/WIF automatically via Soda's `use_context_auth`.
 
 ##### Example
 
@@ -646,7 +647,8 @@ models:
 
 | Environment Variable                         | Example                   | Description                                             |
 |----------------------------------------------|---------------------------|---------------------------------------------------------|
-| `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Access key as saved on key creation by BigQuery. If this environment variable isn't set, the cli tries to use `GOOGLE_APPLICATION_CREDENTIALS` as a fallback, so if you have that set for using their Python library anyway, it should work seamlessly. |
+| `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Account key JSON file. If not set, ADC/WIF is used automatically. |
+| `DATACONTRACT_BIGQUERY_IMPERSONATION_ACCOUNT` | `sa@project.iam.gserviceaccount.com` | Optional. Service account to impersonate. Works with both key file and ADC auth. |
 
 
 #### Azure

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/README.md

@@ -521,10 +521,11 @@ servers:
 
 #### BigQuery
 
-We support authentication to BigQuery using Service Account Key. The used Service Account should include the roles:
+We support authentication to BigQuery using Service Account Key or Application Default Credentials (ADC). ADC supports Workload Identity Federation (WIF), GCE metadata server, and `gcloud auth application-default login`. The used Service Account should include the roles:
 * BigQuery Job User
 * BigQuery Data Viewer
 
+When no `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` is set, the CLI falls back to ADC/WIF automatically via Soda's `use_context_auth`.
 
 ##### Example
 
@@ -545,7 +546,8 @@ models:
 
 | Environment Variable                         | Example                   | Description                                             |
 |----------------------------------------------|---------------------------|---------------------------------------------------------|
-| `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Access key as saved on key creation by BigQuery. If this environment variable isn't set, the cli tries to use `GOOGLE_APPLICATION_CREDENTIALS` as a fallback, so if you have that set for using their Python library anyway, it should work seamlessly. |
+| `DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH` | `~/service-access-key.json` | Service Account key JSON file. If not set, ADC/WIF is used automatically. |
+| `DATACONTRACT_BIGQUERY_IMPERSONATION_ACCOUNT` | `sa@project.iam.gserviceaccount.com` | Optional. Service account to impersonate. Works with both key file and ADC auth. |
 
 
 #### Azure

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/datacontract/catalog/catalog.py

@@ -32,7 +32,11 @@ def create_data_contract_html(contracts, file: Path, path: Path, schema: str):
     odcs = data_contract.get_data_contract()
     file_without_suffix = file.with_suffix(".html")
     html_filepath = path / file_without_suffix
-    html_filepath.parent.mkdir(parents=True, exist_ok=True)
+    try:
+        html_filepath.parent.mkdir(parents=True, exist_ok=True)
+    except FileExistsError:
+        if not html_filepath.parent.is_dir():
+            raise
     with open(html_filepath, "w", encoding="utf-8") as f:
         f.write(html)
     contracts.append(

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/datacontract/cli.py

@@ -431,7 +431,11 @@ def catalog(
     enable_debug_logging(debug)
 
     path = Path(output)
-    path.mkdir(parents=True, exist_ok=True)
+    try:
+        path.mkdir(parents=True, exist_ok=True)
+    except FileExistsError:
+        if not path.is_dir():
+            raise
     console.print(f"Created {output}")
 
     contracts = []

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/datacontract/engines/data_contract_checks.py

@@ -20,10 +20,22 @@ from datacontract.model.run import Check
 @dataclass
 class QuotingConfig:
     quote_field_name: bool = False
+    quote_field_name_with_backticks: bool = False
     quote_model_name: bool = False
     quote_model_name_with_backticks: bool = False
 
 
+def _quote_field_name(field_name: str, quoting_config: QuotingConfig) -> str:
+    """Quote a field name according to the quoting configuration."""
+    if field_name is None:
+        return field_name
+    if quoting_config.quote_field_name:
+        return f'"{field_name}"'
+    elif quoting_config.quote_field_name_with_backticks:
+        return f"`{field_name}`"
+    return field_name
+
+
 def _get_logical_type_option(prop: SchemaProperty, key: str):
     """Get a logical type option value."""
     if prop.logicalTypeOptions is None:
@@ -72,8 +84,9 @@ def to_schema_checks(schema_object: SchemaObject, server: Server) -> List[Check]
 
     type1 = server.type if server and server.type else None
     config = QuotingConfig(
-        quote_field_name=type1 in ["postgres", "sqlserver"],
-        quote_model_name=type1 in ["postgres", "sqlserver"],
+        quote_field_name=type1 in ["postgres", "sqlserver", "azure", "snowflake"],
+        quote_field_name_with_backticks=type1 in ["databricks"],
+        quote_model_name=type1 in ["postgres", "sqlserver", "snowflake"],
         quote_model_name_with_backticks=type1 == "bigquery",
     )
     quoting_config = config
@@ -229,10 +242,7 @@ def check_property_type(
 
 
 def check_property_required(model_name: str, field_name: str, quoting_config: QuotingConfig = QuotingConfig()):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_required"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -260,10 +270,7 @@ def check_property_required(model_name: str, field_name: str, quoting_config: Qu
 
 
 def check_property_unique(model_name: str, field_name: str, quoting_config: QuotingConfig = QuotingConfig()):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_unique"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -293,10 +300,7 @@ def check_property_unique(model_name: str, field_name: str, quoting_config: Quot
 def check_property_min_length(
     model_name: str, field_name: str, min_length: int, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_min_length"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -327,10 +331,7 @@ def check_property_min_length(
 def check_property_max_length(
     model_name: str, field_name: str, max_length: int, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_max_length"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -361,10 +362,7 @@ def check_property_max_length(
 def check_property_minimum(
     model_name: str, field_name: str, minimum: int, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_minimum"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -395,10 +393,7 @@ def check_property_minimum(
 def check_property_maximum(
     model_name: str, field_name: str, maximum: int, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_maximum"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -429,10 +424,7 @@ def check_property_maximum(
 def check_property_not_equal(
     model_name: str, field_name: str, value: int, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_not_equal"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -461,10 +453,7 @@ def check_property_not_equal(
 
 
 def check_property_enum(model_name: str, field_name: str, enum: list, quoting_config: QuotingConfig = QuotingConfig()):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_enum"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -493,10 +482,7 @@ def check_property_enum(model_name: str, field_name: str, enum: list, quoting_co
 
 
 def check_property_regex(model_name: str, field_name: str, pattern: str, quoting_config: QuotingConfig = QuotingConfig()):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_regex"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -553,7 +539,7 @@ def check_model_duplicate_values(
 ):
     check_type = "model_duplicate_values"
     check_key = f"{model_name}__{check_type}"
-    col_joined = ", ".join(cols)
+    col_joined = ", ".join(_quote_field_name(col, quoting_config) for col in cols)
     sodacl_check_dict = {
         checks_for(model_name, quoting_config, check_type): [
             {
@@ -578,10 +564,7 @@ def check_model_duplicate_values(
 def check_property_duplicate_values(
     model_name: str, field_name: str, threshold: str, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_duplicate_values"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -611,10 +594,7 @@ def check_property_duplicate_values(
 def check_property_null_values(
     model_name: str, field_name: str, threshold: str, quoting_config: QuotingConfig = QuotingConfig()
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_null_values"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -648,10 +628,7 @@ def check_property_invalid_values(
     valid_values: list = None,
     quoting_config: QuotingConfig = QuotingConfig(),
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_invalid_values"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -691,10 +668,7 @@ def check_property_missing_values(
     missing_values: list = None,
     quoting_config: QuotingConfig = QuotingConfig(),
 ):
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     check_type = "field_missing_values"
     check_key = f"{model_name}__{field_name}__{check_type}"
@@ -863,10 +837,7 @@ def prepare_query(
 
     query = quality.query
 
-    if quoting_config.quote_field_name:
-        field_name_for_soda = f'"{field_name}"'
-    else:
-        field_name_for_soda = field_name
+    field_name_for_soda = _quote_field_name(field_name, quoting_config)
 
     if quoting_config.quote_model_name:
         model_name_for_soda = f'"{model_name}"'
@@ -1050,11 +1021,12 @@ def to_servicelevel_retention_check(data_contract: OpenDataContractStandard, sla
         logger.info(f"Model {model_name} not found in schema, skipping retention check")
         return None
 
-    # Parse ISO 8601 duration to seconds
-    retention_period = sla.value
-    seconds = _parse_iso8601_to_seconds(retention_period)
+    # Convert retention value to seconds
+    # Supports both numeric value + unit (ODCS style: value=3, unit=y)
+    # and ISO 8601 duration strings (e.g., "P1Y")
+    seconds = _retention_value_to_seconds(sla.value, sla.unit)
     if seconds is None:
-        logger.info(f"Could not parse retention period {retention_period}, skipping retention check")
+        logger.info(f"Could not parse retention period (value={sla.value}, unit={sla.unit}), skipping retention check")
         return None
 
     check_type = "servicelevel_retention"
@@ -1083,6 +1055,43 @@ def to_servicelevel_retention_check(data_contract: OpenDataContractStandard, sla
     )
 
 
+def _retention_value_to_seconds(value, unit: str | None) -> int | None:
+    """Convert a retention value to seconds.
+
+    Supports:
+    - Numeric value with unit (ODCS style): value=3, unit="y"
+    - ISO 8601 duration string: value="P1Y"
+    """
+    if value is None:
+        return None
+
+    # If value is numeric, use the unit to convert to seconds
+    if isinstance(value, (int, float)):
+        numeric_value = int(value)
+        unit_lower = unit.lower() if unit else "d"
+        if unit_lower in ("y", "yr", "year", "years"):
+            return numeric_value * 365 * 24 * 60 * 60
+        elif unit_lower in ("m", "mo", "month", "months"):
+            return numeric_value * 30 * 24 * 60 * 60
+        elif unit_lower in ("d", "day", "days"):
+            return numeric_value * 24 * 60 * 60
+        elif unit_lower in ("h", "hr", "hour", "hours"):
+            return numeric_value * 60 * 60
+        elif unit_lower in ("min", "minute", "minutes"):
+            return numeric_value * 60
+        elif unit_lower in ("s", "sec", "second", "seconds"):
+            return numeric_value
+        else:
+            logger.info(f"Unsupported retention unit: {unit}")
+            return None
+
+    # If value is a string, try ISO 8601 parsing
+    if isinstance(value, str):
+        return _parse_iso8601_to_seconds(value)
+
+    return None
+
+
 def _parse_iso8601_to_seconds(duration: str) -> int | None:
     """Parse ISO 8601 duration to seconds."""
     if not duration:

datacontract_cli-0.11.5/datacontract/engines/soda/connections/bigquery.py

@@ -0,0 +1,24 @@
+import os
+
+import yaml
+
+
+# https://docs.soda.io/soda/connect-bigquery.html#authentication-methods
+def to_bigquery_soda_configuration(server):
+    data_source = {
+        "type": "bigquery",
+        "project_id": server.project,
+        "dataset": server.dataset,
+    }
+
+    if "DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH" in os.environ:
+        data_source["account_info_json_path"] = os.environ["DATACONTRACT_BIGQUERY_ACCOUNT_INFO_JSON_PATH"]
+        data_source["auth_scopes"] = ["https://www.googleapis.com/auth/bigquery"]
+    else:
+        data_source["use_context_auth"] = True
+
+    if "DATACONTRACT_BIGQUERY_IMPERSONATION_ACCOUNT" in os.environ:
+        data_source["impersonation_account"] = os.environ["DATACONTRACT_BIGQUERY_IMPERSONATION_ACCOUNT"]
+
+    soda_configuration_str = yaml.dump({f"data_source {server.type}": data_source})
+    return soda_configuration_str

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/datacontract/export/markdown_exporter.py

@@ -105,7 +105,7 @@ def obj_attributes_to_markdown(obj: BaseModel, excluded_fields: set = set(), is_
         return ""
     if is_in_table_cell:
         bullet_char = "•"
-        newline_char = "<br>"
+        newline_char = "<br />"
     else:
         bullet_char = "-"
         newline_char = "\n"
@@ -247,12 +247,12 @@ def description_to_markdown(description) -> str:
     if description is None:
         return "No description."
     if isinstance(description, str):
-        return description.replace("\n", "<br>")
+        return description.replace("\n", "<br />")
     # Handle Description object - use purpose as the primary description
     if hasattr(description, "purpose") and description.purpose:
-        return description.purpose.replace("\n", "<br>")
+        return description.purpose.replace("\n", "<br />")
     if hasattr(description, "usage") and description.usage:
-        return description.usage.replace("\n", "<br>")
+        return description.usage.replace("\n", "<br />")
     return "No description."
 
 
@@ -285,7 +285,7 @@ def array_of_dict_to_markdown(array: List[Dict[str, str]]) -> str:
         markdown_parts.append(
             "| "
             + " | ".join(
-                f"{str(element.get(header, ''))}".replace("\n", "<br>").replace("\t", TAB) for header in headers
+                f"{str(element.get(header, ''))}".replace("\n", "<br />").replace("\t", TAB) for header in headers
             )
             + " |"
         )
@@ -352,7 +352,7 @@ def extra_to_markdown(obj: BaseModel, is_in_table_cell: bool = False) -> str:
 
     bullet_char = "•"
     value_line_ending = "" if is_in_table_cell else "\n"
-    row_suffix = "<br>" if is_in_table_cell else ""
+    row_suffix = "<br />" if is_in_table_cell else ""
 
     def render_header(key: str) -> str:
         return f"{bullet_char} **{key}:** " if is_in_table_cell else f"\n### {key.capitalize()}\n"

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/datacontract/export/sql_type_converter.py

@@ -5,6 +5,7 @@ from open_data_contract_standard.model import SchemaProperty
 
 class FieldLike(Protocol):
     """Protocol for field-like objects (DCS Field or PropertyAdapter)."""
+
     type: Optional[str]
     config: Optional[Dict[str, Any]]
     precision: Optional[int]
@@ -270,11 +271,7 @@ def convert_to_dataframe(field: Union[SchemaProperty, FieldLike]) -> None | str:
 def convert_to_databricks(field: Union[SchemaProperty, FieldLike]) -> None | str:
     type = _get_type(field)
     databricks_type = _get_config_value(field, "databricksType")
-    if (
-        databricks_type
-        and type
-        and type.lower() not in ["array", "object", "record", "struct"]
-    ):
+    if databricks_type and type and type.lower() not in ["array", "object", "record", "struct"]:
         return databricks_type
     if type is None:
         return None
@@ -343,6 +340,7 @@ def convert_to_duckdb(field: Union[SchemaProperty, FieldLike]) -> None | str:
 
     # Prepare
     type_mapping = {
+        "nvarchar": "VARCHAR",
         "varchar": "VARCHAR",
         "string": "VARCHAR",
         "text": "VARCHAR",
@@ -370,10 +368,15 @@ def convert_to_duckdb(field: Union[SchemaProperty, FieldLike]) -> None | str:
         return type_mapping[type_lower]
 
     # convert decimal numbers with precision and scale
-    if type_lower == "decimal" or type_lower == "number" or type_lower == "numeric":
+    if "decimal" in type_lower or "number" in type_lower or "numeric" in type_lower:
+        # try CustomProperties
         precision = _get_precision(field)
         scale = _get_scale(field)
-        return f"DECIMAL({precision},{scale})"
+        if precision and scale:
+            return f"DECIMAL({precision},{scale})"
+        else:
+            # force physicalType as is
+            return type
 
     # Check list and map
     if type_lower == "list" or type_lower == "array":
@@ -576,6 +579,7 @@ def convert_type_to_impala(field: Union[SchemaProperty, FieldLike]) -> None | st
     # (returning None means no "has type" check is generated)
     return None
 
+
 def convert_type_to_oracle(schema_property: SchemaProperty) -> None | str:
     """Convert ODCS logical types to Oracle types.
 
@@ -605,4 +609,4 @@ def convert_type_to_oracle(schema_property: SchemaProperty) -> None | str:
         "array": "CLOB",
     }
 
-    return mapping.get(logical_type)
+    return mapping.get(logical_type)

{datacontract_cli-0.11.4 → datacontract_cli-0.11.5}/datacontract/imports/bigquery_importer.py

@@ -143,27 +143,58 @@ def import_table_fields(table_fields) -> List[SchemaProperty]:
         required = field.get("mode") == "REQUIRED"
         description = field.get("description")
         field_type = field.get("type")
+        repeated = field.get("mode") == "REPEATED"
 
         if field_type == "RECORD":
             nested_properties = import_table_fields(field.get("fields", []))
-            prop = create_property(
-                name=field_name,
-                logical_type="object",
-                physical_type="RECORD",
-                description=description,
-                required=required if required else None,
-                properties=nested_properties,
-            )
+            if repeated:
+                items_prop = create_property(
+                    name="items",
+                    logical_type="object",
+                    physical_type="RECORD",
+                    properties=nested_properties,
+                )
+                prop = create_property(
+                    name=field_name,
+                    logical_type="array",
+                    description=description,
+                    required=None,
+                    items=items_prop,
+                )
+            else:
+                prop = create_property(
+                    name=field_name,
+                    logical_type="object",
+                    physical_type="RECORD",
+                    description=description,
+                    required=required if required else None,
+                    properties=nested_properties,
+                )
         elif field_type == "STRUCT":
             nested_properties = import_table_fields(field.get("fields", []))
-            prop = create_property(
-                name=field_name,
-                logical_type="object",
-                physical_type="STRUCT",
-                description=description,
-                required=required if required else None,
-                properties=nested_properties,
-            )
+            if repeated:
+                items_prop = create_property(
+                    name="items",
+                    logical_type="object",
+                    physical_type="STRUCT",
+                    properties=nested_properties,
+                )
+                prop = create_property(
+                    name=field_name,
+                    logical_type="array",
+                    description=description,
+                    required=None,
+                    items=items_prop,
+                )
+            else:
+                prop = create_property(
+                    name=field_name,
+                    logical_type="object",
+                    physical_type="STRUCT",
+                    description=description,
+                    required=required if required else None,
+                    properties=nested_properties,
+                )
         elif field_type == "RANGE":
             # Range of date/datetime/timestamp - multiple values, map to array
             items_prop = create_property(
@@ -194,17 +225,33 @@ def import_table_fields(table_fields) -> List[SchemaProperty]:
                 if field.get("scale") is not None:
                     scale = int(field.get("scale"))
 
-            prop = create_property(
-                name=field_name,
-                logical_type=logical_type,
-                physical_type=field_type,
-                description=description,
-                required=required if required else None,
-                max_length=max_length,
-                precision=precision,
-                scale=scale,
-            )
-
+            if repeated:
+                items_prop = create_property(
+                    name="items",
+                    logical_type=logical_type,
+                    physical_type=field_type,
+                    max_length=max_length,
+                    precision=precision,
+                    scale=scale,
+                )
+                prop = create_property(
+                    name=field_name,
+                    logical_type="array",
+                    description=description,
+                    required=None,
+                    items=items_prop,
+                )
+            else:
+                prop = create_property(
+                    name=field_name,
+                    logical_type=logical_type,
+                    physical_type=field_type,
+                    description=description,
+                    required=required if required else None,
+                    max_length=max_length,
+                    precision=precision,
+                    scale=scale,
+                )
         properties.append(prop)
 
     return properties