snowflake-data-validation 1.46.2__tar.gz → 1.47.0__tar.gz

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: snowflake-data-validation
-Version: 1.46.2
+Version: 1.47.0
 Summary: Snowflake Data Validation
 Project-URL: Bug Tracker, https://github.com/snowflake-eng/migrations-data-validation/issues
 Project-URL: Source code, https://github.com/snowflake-eng/migrations-data-validation/
@@ -38,7 +38,7 @@ Requires-Dist: pydantic-yaml>=1.4.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: ruamel-yaml>=0.18.0
-Requires-Dist: snowflake-connector-python<5,>=4.7.3
+Requires-Dist: snowflake-connector-python<6,>=5
 Requires-Dist: toml>=0.10.2
 Requires-Dist: typer>=0.15.3
 Requires-Dist: typing-extensions>=4.13.2
@@ -79,7 +79,7 @@ Requires-Dist: psycopg[binary]>=3.2.0; extra == 'postgresql'
 Provides-Extra: redshift
 Requires-Dist: redshift-connector>=2.1.14; extra == 'redshift'
 Provides-Extra: scai
-Requires-Dist: snowflake-code-unit-registry>=0.9.10; extra == 'scai'
+Requires-Dist: snowflake-code-unit-registry>=0.9.15; extra == 'scai'
 Provides-Extra: sqlite
 Requires-Dist: sqlean-py>=3.45.0; extra == 'sqlite'
 Provides-Extra: sqlserver
@@ -182,6 +182,14 @@ L1 and L2 read each column's catalog spelling rather than an upper-cased name, s
 
 ### Views
 
+SQL Server and Azure Synapse source scans delimit each configured object-name part
+through the shared identifier manager. Reserved words, dots inside quoted parts,
+escaped closing brackets, and omitted schema segments remain valid in generated
+SQL, including whitespace between a separator and a quoted part.
+The platform-aware table-context configuration boundary parses T-SQL references
+into actual name components once; repeated queries write those components without
+reinterpreting quote characters or reparsing the original configuration.
+
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
 
 ### Row Validation
@@ -190,6 +198,26 @@ Performs row-by-row comparison:
 - Field-level value comparison
 - Mismatch reporting
 
+Row-MD5 results preserve the exact key aliases projected from actual catalog names; joins and report predicates do not uppercase or case-fold those aliases. Distinct keys such as `MixedCase` and `MIXEDCASE` keep their own values. Join alignment accepts an exact `_IDX` projection alias when the key's direct alias is absent; reports read the resulting canonical aliases directly. Each report side takes its identifier and literal rules from the same query generator. Only the generated hash alias is recognized by its dialect spelling (`row_md5` on PostgreSQL/Redshift). An index named `ROW_MD5` conflicts with that reserved result column and is rejected rather than used as a hash. Empty results retain cursor headers and complete missing join/hash columns; populated results fail explicitly for a missing key alias (including a known exclusion) or a missing row-MD5 column rather than guessing another key by casing.
+
+For each mismatched row, the report holds a source and a target `SELECT` that find that row, and the run also saves a diff query per side. Each side's queries write names by that side's identifier rules: every part of the object name is read as that platform reads a configured name (an unquoted part folds to its storage case, a quoted part is kept) and written delimited, and every key column is written delimited too. So a SQL Server or Azure Synapse `Load.dbo.File` is queried as `[Load].[dbo].[File]`, the Snowflake target `LOAD.DBO.FILE` as `"LOAD"."DBO"."FILE"`, and a BigQuery key column as `` `Order Id` ``; a reserved word such as `File` stays valid. The report's table name and the saved query file names keep the configured name (`Load.dbo.File`). A string key is escaped by the query generator of the platform that runs that side (`escape_string_literal`). Most dialects use standard SQL and double an apostrophe. Snowflake also doubles backslashes, because it reads a backslash in a string literal as an escape, and BigQuery backslash-escapes both backslashes and apostrophes. A platform whose literals differ overrides `escape_string_literal` on its query generator.
+
+Redshift also doubles backslashes and apostrophes. PostgreSQL's standard-SQL escaping assumes `standard_conforming_strings=on` (the server default since PostgreSQL 9.1); the connector does not set this session option. The unit tests check generated literal text for all ten engine dialects, not live database parsing.
+
+Qualified source references retain parsed actual components until the identifier
+manager writes SQL. At the table-context boundary, an explicitly supplied actual
+schema fills an omitted schema position; it is not reparsed or replaced with
+`dbo`. Without a resolved schema, valid omitted-position syntax such as
+`tempdb..#snapshot` remains omitted. Separate internal temporary references do
+not inherit the source table's schema. Catalog predicates likewise escape actual
+names as string values rather than stripping literal brackets, quotes or padding.
+
+Row-query result labels retain their exact aliases. Catalog-metadata boundaries
+normalize only field labels (such as `DATA_TYPE` and `ROW_COUNT`) for metadata
+consumers; actual column-name values remain unchanged, including mixed case.
+
+A key present once on each side is a `MISMATCH` when its row MD5s differ. A NULL row MD5 on one side against a digest on the other is a difference too; NULL on both sides is a match. This is the `IS DISTINCT FROM` rule the Snowflake pushdown compare uses (`docs/common/l3-signature-pushdown-compare.md`).
+
 ---
 
 ## 📊 Reports
@@ -222,6 +250,18 @@ Raising `requires-python` here above the pinned version fails the audit until th
 
 ---
 
+## 🧩 Custom result processors
+
+`execute_query` and `execute_query_from_table_configuration` take an optional
+`result_processor(raw_result, apply_column_validated_uppercase)` that turns the
+connector's raw result into a DataFrame. A processor you pass is called for every
+result, an empty one included, with the `apply_column_validated_uppercase` flag,
+and the DataFrame it returns is the result, so the columns it gives an empty
+frame are kept. Without one, the platform's default processor is used and an
+empty result becomes an empty DataFrame with no columns.
+
+---
+
 ## 📚 Documentation
 
 For configuration options and examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
@@ -230,6 +270,27 @@ For configuration options and examples, see the [official documentation](https:/
 
 ## Changelog
 
+### v1.47.0
+
+**New features**
+
+- Added L1, L2, and L3 validation of BigQuery Iceberg tables.
+- Added L2 and L3 comparison of Teradata `BLOB` and `CLOB` columns.
+
+**Improvements**
+
+- Improved Teradata L2 validation to compute the standard deviation of integer columns exactly.
+- Updated the Snowflake connector to `snowflake-connector-python` 5.x.
+
+**Bug fixes**
+
+- Fixed row validation losing join keys when their casing differed or the key column was excluded.
+- Fixed row validation of tables with duplicate keys by matching duplicate-key rows as multisets, and quoted SQL Server identifiers.
+- Fixed custom processing being dropped when a query returned no rows.
+- Fixed the saved row-diff SQL ending in an empty `WHERE` when every mismatched row was missing on one side.
+- Fixed the row drill-down treating a `NULL` `ROW_MD5` on one side as equal to a digest on the other.
+- Fixed Oracle `TIMESTAMP WITH LOCAL TIME ZONE` and `XMLTYPE` comparison by reading them as UTC text and compact `XMLSERIALIZE` text.
+
 ### v1.46.2
 
 **Improvements**

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/README.md

@@ -91,6 +91,14 @@ L1 and L2 read each column's catalog spelling rather than an upper-cased name, s
 
 ### Views
 
+SQL Server and Azure Synapse source scans delimit each configured object-name part
+through the shared identifier manager. Reserved words, dots inside quoted parts,
+escaped closing brackets, and omitted schema segments remain valid in generated
+SQL, including whitespace between a separator and a quoted part.
+The platform-aware table-context configuration boundary parses T-SQL references
+into actual name components once; repeated queries write those components without
+reinterpreting quote characters or reparsing the original configuration.
+
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
 
 ### Row Validation
@@ -99,6 +107,26 @@ Performs row-by-row comparison:
 - Field-level value comparison
 - Mismatch reporting
 
+Row-MD5 results preserve the exact key aliases projected from actual catalog names; joins and report predicates do not uppercase or case-fold those aliases. Distinct keys such as `MixedCase` and `MIXEDCASE` keep their own values. Join alignment accepts an exact `_IDX` projection alias when the key's direct alias is absent; reports read the resulting canonical aliases directly. Each report side takes its identifier and literal rules from the same query generator. Only the generated hash alias is recognized by its dialect spelling (`row_md5` on PostgreSQL/Redshift). An index named `ROW_MD5` conflicts with that reserved result column and is rejected rather than used as a hash. Empty results retain cursor headers and complete missing join/hash columns; populated results fail explicitly for a missing key alias (including a known exclusion) or a missing row-MD5 column rather than guessing another key by casing.
+
+For each mismatched row, the report holds a source and a target `SELECT` that find that row, and the run also saves a diff query per side. Each side's queries write names by that side's identifier rules: every part of the object name is read as that platform reads a configured name (an unquoted part folds to its storage case, a quoted part is kept) and written delimited, and every key column is written delimited too. So a SQL Server or Azure Synapse `Load.dbo.File` is queried as `[Load].[dbo].[File]`, the Snowflake target `LOAD.DBO.FILE` as `"LOAD"."DBO"."FILE"`, and a BigQuery key column as `` `Order Id` ``; a reserved word such as `File` stays valid. The report's table name and the saved query file names keep the configured name (`Load.dbo.File`). A string key is escaped by the query generator of the platform that runs that side (`escape_string_literal`). Most dialects use standard SQL and double an apostrophe. Snowflake also doubles backslashes, because it reads a backslash in a string literal as an escape, and BigQuery backslash-escapes both backslashes and apostrophes. A platform whose literals differ overrides `escape_string_literal` on its query generator.
+
+Redshift also doubles backslashes and apostrophes. PostgreSQL's standard-SQL escaping assumes `standard_conforming_strings=on` (the server default since PostgreSQL 9.1); the connector does not set this session option. The unit tests check generated literal text for all ten engine dialects, not live database parsing.
+
+Qualified source references retain parsed actual components until the identifier
+manager writes SQL. At the table-context boundary, an explicitly supplied actual
+schema fills an omitted schema position; it is not reparsed or replaced with
+`dbo`. Without a resolved schema, valid omitted-position syntax such as
+`tempdb..#snapshot` remains omitted. Separate internal temporary references do
+not inherit the source table's schema. Catalog predicates likewise escape actual
+names as string values rather than stripping literal brackets, quotes or padding.
+
+Row-query result labels retain their exact aliases. Catalog-metadata boundaries
+normalize only field labels (such as `DATA_TYPE` and `ROW_COUNT`) for metadata
+consumers; actual column-name values remain unchanged, including mixed case.
+
+A key present once on each side is a `MISMATCH` when its row MD5s differ. A NULL row MD5 on one side against a digest on the other is a difference too; NULL on both sides is a match. This is the `IS DISTINCT FROM` rule the Snowflake pushdown compare uses (`docs/common/l3-signature-pushdown-compare.md`).
+
 ---
 
 ## 📊 Reports
@@ -131,6 +159,18 @@ Raising `requires-python` here above the pinned version fails the audit until th
 
 ---
 
+## 🧩 Custom result processors
+
+`execute_query` and `execute_query_from_table_configuration` take an optional
+`result_processor(raw_result, apply_column_validated_uppercase)` that turns the
+connector's raw result into a DataFrame. A processor you pass is called for every
+result, an empty one included, with the `apply_column_validated_uppercase` flag,
+and the DataFrame it returns is the result, so the columns it gives an empty
+frame are kept. Without one, the platform's default processor is used and an
+empty result becomes an empty DataFrame with no columns.
+
+---
+
 ## 📚 Documentation
 
 For configuration options and examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
@@ -139,6 +179,27 @@ For configuration options and examples, see the [official documentation](https:/
 
 ## Changelog
 
+### v1.47.0
+
+**New features**
+
+- Added L1, L2, and L3 validation of BigQuery Iceberg tables.
+- Added L2 and L3 comparison of Teradata `BLOB` and `CLOB` columns.
+
+**Improvements**
+
+- Improved Teradata L2 validation to compute the standard deviation of integer columns exactly.
+- Updated the Snowflake connector to `snowflake-connector-python` 5.x.
+
+**Bug fixes**
+
+- Fixed row validation losing join keys when their casing differed or the key column was excluded.
+- Fixed row validation of tables with duplicate keys by matching duplicate-key rows as multisets, and quoted SQL Server identifiers.
+- Fixed custom processing being dropped when a query returned no rows.
+- Fixed the saved row-diff SQL ending in an empty `WHERE` when every mismatched row was missing on one side.
+- Fixed the row drill-down treating a `NULL` `ROW_MD5` on one side as equal to a digest on the other.
+- Fixed Oracle `TIMESTAMP WITH LOCAL TIME ZONE` and `XMLTYPE` comparison by reading them as UTC text and compact `XMLSERIALIZE` text.
+
 ### v1.46.2
 
 **Improvements**

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/pyproject.toml

@@ -36,7 +36,7 @@ dependencies = [
   "connectorx>=0.3.0",
   "cryptography>=50.0.0",
   "deepdiff>=8.0.0",
-  "snowflake-connector-python>=4.7.3,<5",
+  "snowflake-connector-python>=5,<6",
   "pandas>=2.2.3",
   "polars>=1.0.0",
   "pyarrow>=14.0.0",
@@ -106,7 +106,7 @@ db2 = [
 # musllinux build, and SDV is published to public PyPI. Inside DMO/DEA the
 # dependency is already declared, so SDV's logger tree is covered there.
 scai = [
-  "snowflake-code-unit-registry>=0.9.10",
+  "snowflake-code-unit-registry>=0.9.15",
 ]
 
 [project.urls]
@@ -192,6 +192,14 @@ exclude_lines = [
   "if __name__ == .__main__.:",
 ]
 
+# Ruff only reads the sources, so the linter needs neither the package nor its drivers.
+[tool.hatch.envs.linter]
+detached = true
+installer = "uv"
+dependencies = [
+  "ruff>=0.15",
+]
+
 [tool.hatch.envs.linter.scripts]
 check = [
   "echo 'Running linting checks...' && ruff check --config=../ruff.toml --statistics --verbose . || (echo '❌ LINTING FAILED: Please fix the above linting issues before proceeding. Use \"ruff check --config=../ruff.toml --fix .\" to auto-fix some issues, or fix them manually.' && exit 1)",

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/src/snowflake/snowflake_data_validation/bigquery/extractor/templates/bigquery_datatypes_normalization_templates.yaml

@@ -96,4 +96,16 @@ GEOGRAPHY: CASE WHEN `{{ col_name }}` IS NULL THEN NULL ELSE TRIM(REPLACE(REPLAC
 INTERVAL: CAST(`{{ col_name }}` AS STRING)
 RANGE: CAST(`{{ col_name }}` AS STRING)
 
+# Iceberg EXPORT DATA text. Compatible clocks are the export format (no offset,
+# six fractional digits). Native TIMESTAMP_NTZ pads those six digits to the nine
+# Snowflake TO_CHAR prints for Iceberg v3. JSON/STRUCT/ARRAY hash TO_JSON_STRING,
+# the same text the extract wrote, including the size guard the VARCHAR target uses.
+TIMESTAMP_ICEBERG: FORMAT_TIMESTAMP('%Y-%m-%dT%H:%M:%E6S', `{{ col_name }}`)
+DATETIME_ICEBERG: FORMAT_DATETIME('%Y-%m-%dT%H:%M:%E6S', `{{ col_name }}`)
+TIME_ICEBERG: CAST(`{{ col_name }}` AS STRING)
+TIMESTAMP_ICEBERG_NTZ: FORMAT_TIMESTAMP('%E4Y-%m-%dT%H:%M:%E6S', `{{ col_name }}`) || '000'
+DATETIME_ICEBERG_NTZ: FORMAT_DATETIME('%E4Y-%m-%dT%H:%M:%E6S', `{{ col_name }}`) || '000'
+BIGNUMERIC_ICEBERG: CAST(`{{ col_name }}` AS STRING)
+JSON_ICEBERG: CASE WHEN `{{ col_name }}` IS NULL THEN NULL WHEN BYTE_LENGTH(TO_JSON_STRING(`{{ col_name }}`)) > {{ semi_structured_size_limit_bytes }} THEN '{{ semi_structured_size_excluded_sentinel }}' ELSE NULLIF(TO_JSON_STRING(`{{ col_name }}`), 'null') END
+
 DEFAULT: CAST(`{{ col_name }}` AS STRING)

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/src/snowflake/snowflake_data_validation/bigquery/query/query_generator_bigquery.py

@@ -38,8 +38,15 @@ from snowflake.snowflake_data_validation.query.query_generator_base import (
 from snowflake.snowflake_data_validation.utils.constants import (
     COL_NAME_NO_QUOTES_PLACEHOLDER,
     METRIC_COLUMN_MODIFIER_PLACEHOLDER,
+    SEMI_STRUCTURED_EXPORT_SIZE_LIMIT_BYTES,
+    SEMI_STRUCTURED_SIZE_EXCLUDED_SENTINEL,
     Platform,
 )
+from snowflake.snowflake_data_validation.utils.iceberg_dv_common import (
+    ICEBERG_BIGQUERY_NUMERIC_TYPE,
+    IcebergProfile,
+    resolve_iceberg_bigquery_source_normalization_key,
+)
 from snowflake.snowflake_data_validation.utils.iso_dv_common import (
     get_paired_target_column_types,
     normalize_snowflake_base_type,
@@ -60,6 +67,7 @@ if TYPE_CHECKING:  # pragma: no cover - used only for type hints.
 LOGGER = logging.getLogger(__name__)
 
 _WIDE_NUMERIC_BASE_TYPES = frozenset({BigQueryType.BIGNUMERIC, BigQueryType.BIGDECIMAL})
+_ICEBERG_DECIMAL128_SOURCE_TYPES = frozenset({CommonType.NUMERIC, CommonType.DECIMAL})
 # Normalization key for a BIGNUMERIC / BIGDECIMAL whose Snowflake mirror is DECFLOAT.
 BIGNUMERIC_DECFLOAT_NORMALIZATION_KEY = "BIGNUMERIC_DECFLOAT"
 # Normalization key for an ARRAY / STRUCT holding no NUMERIC / BIGNUMERIC, which data
@@ -100,6 +108,10 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
         """Initialize the BigQuery query generator."""
         super().__init__(platform=Platform.BIGQUERY)
 
+    def escape_string_literal(self, value: str) -> str:
+        r"""Backslash-escape backslashes and quotes: BigQuery reads ``\\`` and ``\'`` in a literal, not ``''``."""
+        return value.replace("\\", "\\\\").replace("'", "\\'")
+
     def _measure_collation_tokens(self, column, metric_name: str) -> tuple[str, str]:
         """Force binary DISTINCT semantics only for metadata-gated STRING columns."""
         if (
@@ -247,7 +259,21 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
         table_context: TableContext,
     ) -> str:
         base_type = self.normalize_bigquery_type(column.data_type)
-        if self._lands_in_decfloat(column, table_context):
+        profile = IcebergProfile.from_table_context(table_context)
+        paired = get_paired_target_column_types(table_context).get(column.name)
+        iceberg_key = resolve_iceberg_bigquery_source_normalization_key(
+            base_type,
+            is_iceberg=profile.is_active,
+            paired_target_type=paired if isinstance(paired, str) else None,
+        )
+        if iceberg_key != base_type and not has_custom_column_normalization(
+            column.name,
+            base_type,
+            column_normalization_map=None,
+            templates_loader_manager=table_context.templates_loader_manager,
+        ):
+            base_type = iceberg_key
+        elif self._lands_in_decfloat(column, table_context):
             base_type = BIGNUMERIC_DECFLOAT_NORMALIZATION_KEY
         elif serializes_exactly_through_to_json(column.data_type) and not has_custom_column_normalization(
             column.name,
@@ -284,6 +310,31 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
             return None
         return f"NULLIF(TO_JSON_STRING(PARSE_JSON({field_wise})), 'null')"
 
+    def _iceberg_field_wise_normalization(self, column: ColumnMetadata) -> str | None:
+        """Hash the field-wise JSON text the Iceberg extract wrote, without PARSE_JSON."""
+        column_ref = f"`{self._normalization_column_name(column)}`"
+        field_wise = field_wise_json_text(column.data_type, column_ref)
+        if field_wise is None:
+            return None
+        limit = SEMI_STRUCTURED_EXPORT_SIZE_LIMIT_BYTES
+        sentinel = SEMI_STRUCTURED_SIZE_EXCLUDED_SENTINEL
+        return (
+            f"CASE WHEN {column_ref} IS NULL THEN NULL "
+            f"WHEN BYTE_LENGTH({field_wise}) > {limit} THEN '{sentinel}' "
+            f"ELSE NULLIF({field_wise}, 'null') END"
+        )
+
+    def _column_with_iceberg_numeric_scale(self, column: ColumnMetadata, table_context: TableContext) -> ColumnMetadata:
+        """Force NUMERIC/DECIMAL scale to 9 so the source mask matches NUMBER(38,9)."""
+        if self.normalize_bigquery_type(column.data_type) not in _ICEBERG_DECIMAL128_SOURCE_TYPES:
+            return column
+        paired = get_paired_target_column_types(table_context).get(column.name)
+        if not isinstance(paired, str) or normalize_snowflake_base_type(paired) != CommonType.NUMBER:
+            return column
+        rendered = column.copy()
+        rendered.data_type = ICEBERG_BIGQUERY_NUMERIC_TYPE
+        return rendered
+
     def _column_normalization(
         self,
         column: ColumnMetadata,
@@ -295,10 +346,21 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
         A user's normalization, a per-column rule or an override of the column's base type
         (``STRUCT``, ``ARRAY``), wins over the built-in field-wise text.
         """
+        base_type = self.normalize_bigquery_type(column.data_type)
+        profile = IcebergProfile.from_table_context(table_context)
+        if profile.is_active:
+            iceberg_field_wise = self._iceberg_field_wise_normalization(column)
+            if iceberg_field_wise is not None and not has_custom_column_normalization(
+                column.name,
+                base_type,
+                column_normalization_map,
+                table_context.templates_loader_manager,
+            ):
+                return iceberg_field_wise
         field_wise = self._field_wise_normalization(column)
         if field_wise is not None and not has_custom_column_normalization(
             column.name,
-            self.normalize_bigquery_type(column.data_type),
+            base_type,
             column_normalization_map,
             table_context.templates_loader_manager,
         ):
@@ -315,7 +377,8 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
         )
         # Resolve ``{{ col_name }}`` and, for numeric types, the scale-aware
         # ``{{ numeric_format_mask }}`` (BigQuery emits a printf ``%.Nf``).
-        return self._render_normalization_template(normalization_template, column)
+        rendered_column = self._column_with_iceberg_numeric_scale(column, table_context) if profile.is_active else column
+        return self._render_normalization_template(normalization_template, rendered_column)
 
     def _build_datatypes_normalization_renderer_templates(
         self,

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/src/snowflake/snowflake_data_validation/connector/connector_base.py

@@ -14,10 +14,14 @@
 # limitations under the License.
 
 
+from __future__ import annotations
+
 import logging
 import time
 
 from abc import ABC, abstractmethod
+from collections.abc import Callable
+from typing import TYPE_CHECKING
 
 from snowflake.snowflake_data_validation.connector.connector_failure_tracker import (
     ConnectionFailureTracker,
@@ -27,6 +31,10 @@ from snowflake.snowflake_data_validation.utils.constants import (
 )
 
 
+if TYPE_CHECKING:
+    import pandas as pd
+
+
 class ConnectorBase(ABC):
     """Abstract base class for database connectors."""
 
@@ -162,6 +170,10 @@ class ConnectorBase(ABC):
         """Execute a query on the database."""
         pass
 
+    def get_result_processor(self) -> Callable[[list, bool], pd.DataFrame] | None:
+        """Return a connector-specific normalizer, or use the platform default."""
+        return None
+
     @abstractmethod
     def execute_statement(self, statement: str) -> None:
         """Execute a statement on the database."""

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.47.0}/src/snowflake/snowflake_data_validation/executer/sync_validation_executor.py

@@ -27,6 +27,7 @@ public workflow API module:
 """
 
 import hashlib
+import logging
 
 from collections.abc import Callable
 from concurrent.futures import ThreadPoolExecutor
@@ -34,6 +35,7 @@ from typing import Any
 
 import pandas as pd
 
+from shared.identifiers.managers import identifier_manager
 from snowflake.snowflake_data_validation.connector.connector_base import (
     ConnectorBase,
 )
@@ -80,9 +82,12 @@ from snowflake.snowflake_data_validation.utils.telemetry import (
     report_telemetry,
 )
 from snowflake.snowflake_data_validation.validation.row_data_validator import (
+    ROW_MD5_KEY,
     SOURCE_SUFFIX,
     TARGET_SUFFIX,
+    MissingRowJoinColumnError,
     RowDataValidator,
+    RowReportSide,
 )
 from snowflake.snowflake_data_validation.validation.validation_report_buffer import (
     ValidationReportBuffer,
@@ -91,6 +96,23 @@ from snowflake.snowflake_data_validation.validation.validation_report_buffer imp
 
 # ``execute_query``'s ``result_processor(raw_result, apply_column_validated_uppercase)``.
 SchemaResultProcessor = Callable[[list[Any], bool], pd.DataFrame]
+LOGGER = logging.getLogger(__name__)
+
+
+def _suffix_row_result(frame: pd.DataFrame, suffix: str, table_context: TableContext) -> pd.DataFrame:
+    """Recognize the generated unquoted hash alias; preserve every actual key alias."""
+    if any(column.name == ROW_MD5_KEY for column in table_context.index_column_collection):
+        raise MissingRowJoinColumnError(f"index column {ROW_MD5_KEY} conflicts with the reserved row MD5 alias")
+    try:
+        manager = identifier_manager(table_context.platform)
+    except ValueError:
+        LOGGER.debug(f"Dialect alias rules are unavailable for {table_context.platform}; retaining the generated hash alias")
+        hash_alias = ROW_MD5_KEY
+    else:
+        hash_alias = manager.parse_identifier(ROW_MD5_KEY)
+    if hash_alias != ROW_MD5_KEY and hash_alias in frame.columns and ROW_MD5_KEY not in frame.columns:
+        frame = frame.rename(columns={hash_alias: ROW_MD5_KEY})
+    return frame.convert_dtypes().add_suffix(suffix)
 
 
 def _ignoring_case_flag(process: Callable[[list], pd.DataFrame]) -> Callable[[list, bool], pd.DataFrame]:
@@ -441,8 +463,8 @@ class SyncValidationExecutor(BaseValidationExecutor):
 
         source_index_column_collection = [column.name for column in source_table_context.index_column_collection]
         target_index_column_collection = [column.name for column in target_table_context.index_column_collection]
-        source_index_column_suffix_collection = [col.upper() + SOURCE_SUFFIX for col in source_index_column_collection]
-        target_index_column_suffix_collection = [col.upper() + TARGET_SUFFIX for col in target_index_column_collection]
+        source_index_column_suffix_collection = [col + SOURCE_SUFFIX for col in source_index_column_collection]
+        target_index_column_suffix_collection = [col + TARGET_SUFFIX for col in target_index_column_collection]
 
         source_query = data_validation_api.generate_row_md5_select_query(
             table_context=source_table_context,
@@ -471,8 +493,8 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 executor.shutdown(wait=False, cancel_futures=True)
                 raise
 
-        source_md5_df = source_rows_result.dataframe.convert_dtypes().rename(columns=lambda x: x.upper() + SOURCE_SUFFIX)
-        target_md5_df = target_rows_result.dataframe.convert_dtypes().rename(columns=lambda x: x.upper() + TARGET_SUFFIX)
+        source_md5_df = _suffix_row_result(source_rows_result.dataframe, SOURCE_SUFFIX, source_table_context)
+        target_md5_df = _suffix_row_result(target_rows_result.dataframe, TARGET_SUFFIX, target_table_context)
 
         # SQLite has no native MD5: the row-md5-select query returns CONCAT_TEXT
         # instead of ROW_MD5. Compute hashlib MD5 here so the comparison columns match.
@@ -486,12 +508,18 @@ class SyncValidationExecutor(BaseValidationExecutor):
         source_transformed = QueryExecutionResult(dataframe=source_md5_df)
         target_transformed = QueryExecutionResult(dataframe=target_md5_df)
 
+        excluded_column_names: list[str] = []
+        for side_context in (source_table_context, target_table_context):
+            if side_context.is_exclusion_mode and side_context.column_selection_list is not None:
+                excluded_column_names.extend(side_context.column_selection_list)
+
         rows_comparison_result = data_validation_api.compare_md5_rows_chunk(
             source_rows_result=source_transformed,
             target_rows_result=target_transformed,
             source_index_column_suffix_collection=source_index_column_suffix_collection,
             target_index_column_suffix_collection=target_index_column_suffix_collection,
             table_context=source_table_context,
+            excluded_column_names=excluded_column_names,
         )
 
         rows_compared_df = rows_comparison_result.results_dataframe
@@ -499,6 +527,16 @@ class SyncValidationExecutor(BaseValidationExecutor):
         has_differences = len(rows_compared_df) > 0
 
         if has_differences:
+            source_generator = self.source_extractor.query_generator
+            target_generator = self.target_extractor.query_generator
+            source_side = RowReportSide(
+                escape_literal=source_generator.escape_string_literal,
+                platform=source_generator.platform,
+            )
+            target_side = RowReportSide(
+                escape_literal=target_generator.escape_string_literal,
+                platform=target_generator.platform,
+            )
             data_validator = RowDataValidator()
             data_validator.generate_row_validation_report(
                 compared_df=rows_compared_df,
@@ -512,6 +550,8 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 context=self.context,
                 db_manager=self.db_manager,
                 object_type=source_table_context.object_type,
+                source_side=source_side,
+                target_side=target_side,
             )
 
             data_validator.generate_row_validation_queries(
@@ -522,6 +562,8 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 target_index_column_collection=target_index_column_collection,
                 object_id=source_table_context.id,
                 context=self.context,
+                source_side=source_side,
+                target_side=target_side,
             )
 
             self._register_row_validation_execution_status(