snowflake-data-validation 1.44.1__tar.gz → 1.45.1__tar.gz

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: snowflake-data-validation
-Version: 1.44.1
+Version: 1.45.1
 Summary: Snowflake Data Validation
 Project-URL: Bug Tracker, https://github.com/snowflake-eng/migrations-data-validation/issues
 Project-URL: Source code, https://github.com/snowflake-eng/migrations-data-validation/
@@ -154,6 +154,14 @@ Compares table structure between source and target:
 - Precision, scale, and length
 - Nullable constraints
 
+Some differences are reported as `WARNING` and do not fail L1:
+- Snowflake still holds every source value: a different column order, a Snowflake precision or scale that can hold the source's, and a `NOT NULL` source column whose Snowflake column is nullable (a Snowflake virtual column is always nullable).
+- Snowflake may truncate or reject source values, so review it: a source character length longer than Snowflake's.
+
+A nullable source column on a `NOT NULL` Snowflake column still fails, since Snowflake can reject the source's NULLs.
+
+A PostgreSQL user-defined `ENUM` column is validated as `ENUM` against a Snowflake `VARCHAR`. `information_schema` reports every enum as `USER-DEFINED`, so the metadata queries read `pg_type.typtype` to tell enums apart, and report the longest label as the column's length. L2 and L3 compare the label text.
+
 ### Metrics Validation
 Compares statistical metrics for each column:
 - Row count
@@ -164,6 +172,10 @@ Compares statistical metrics for each column:
 
 For some types these metrics do not depend on the values themselves (for SQL Server: `VECTOR`, `XML`, `IMAGE`, `SQL_VARIANT`, `JSON`, `GEOGRAPHY`, `GEOMETRY`, `UNIQUEIDENTIFIER`), so a changed value can pass L2. Use L3 row validation to compare those columns' values. The per-type list is in `docs/data-migration-orchestrator/sqlserver-data-type-coverage.md` in this repo.
 
+### Column pairing (L1 and L2)
+
+L1 and L2 read each column's catalog spelling rather than an upper-cased name, so the report shows PostgreSQL `order_id` and SQL Server `OrderId` as the catalog spells them. They pair a source column with its target column by the source platform's identifier rule, the same rule data migration uses: the exact name; on a case-insensitive source (SQL Server, Teradata, Redshift, Azure Synapse, BigQuery, SQLite) a name differing only by case; else the name as Snowflake reads it unquoted, so PostgreSQL `order_id` pairs with `ORDER_ID`. On Oracle, PostgreSQL and DB2, `MixedCase` and `mixedcase` are two columns, and a mixed-case source column whose target exists only as `MIXEDCASE` is reported missing from the target. A `column_mappings` key names a source column by the same rule: on Oracle the key `MixedCase` names `MIXEDCASE`, and `"MixedCase"` names `MixedCase`. Result-set and query inputs pair names exactly, else ignoring case.
+
 ### Views
 
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
@@ -214,6 +226,48 @@ For configuration options and examples, see the [official documentation](https:/
 
 ## Changelog
 
+### v1.45.1
+
+**Improvements**
+
+- Improved schema-level metadata retrieval by using `SHOW COLUMNS`.
+- Improved Iceberg timestamp normalization to follow the target physical type.
+
+**Bug fixes**
+
+- Fixed schema validation failing for source columns whose names end in a space.
+- Fixed BigQuery `ARRAY<INT64>` elements beyond 2^53 being compared as quoted JSON strings.
+- Fixed validation never passing for a SQL Server `NOT NULL` computed column migrated as a Snowflake virtual column.
+- Fixed L1 and L2 pairing of source and target columns to ignore case for case-sensitive sources when no target names are given.
+- Fixed SQL Server Iceberg L1 schema validation reporting a missing Snowflake character length.
+- Fixed Redshift `DATE`, `TIMESTAMP` and `TIMESTAMPTZ` values on Iceberg losing signed-year values.
+- Fixed Teradata numeric L3 text exceeding the `TO_CHAR` digit limit.
+- Fixed PostgreSQL `ENUM` columns surfacing as `USER-DEFINED`, so `ENUM` type rules never matched.
+- Fixed Teradata column metadata not reporting precision and scale, which made `TIME` pairing fall back to 6.
+- Fixed L1 failing `CHARACTER_MAXIMUM_LENGTH` for a SQL Server `xml` column migrated as `VARIANT`.
+
+### v1.45.0
+
+**New features**
+
+- Added support for applying custom L3 normalization per side, including `targetDataType: "*"` to target the mapped type.
+- Added the `includeCountDistinct` option; L2 `COUNT(DISTINCT)` is now off by default and opt-in globally or per table.
+- Added support for additional table and view types in data validation, including materialized views, temporary tables, BigQuery clones and snapshots, and DB2 materialized query tables.
+
+**Improvements**
+
+- Improved row comparison with positional column identity, so duplicate and case-variant column names no longer collapse.
+- Improved Redshift connection resiliency with TCP keepalive defaults.
+- Improved L1 validation to treat native integer bits-versus-digits precision differences as compatible, reported as a warning.
+
+**Bug fixes**
+
+- Fixed Oracle L2 metric rows and L3 cell frames upper-casing column names, which made mixed-case columns collide.
+- Fixed L3 failing rows over a lowercase SQL Server `UNIQUEIDENTIFIER` or an ignored `FLOAT` `targetExpression`.
+- Fixed a possible deadlock in expression casts when validators run on several threads.
+- Fixed Redshift and PostgreSQL `REAL` values being compared against a non-exact widened value.
+- Fixed Iceberg `FULL_INGEST` validation to follow the lossless-only temporal promotion of the load.
+
 ### v1.44.1
 
 **New features**

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/README.md

@@ -63,6 +63,14 @@ Compares table structure between source and target:
 - Precision, scale, and length
 - Nullable constraints
 
+Some differences are reported as `WARNING` and do not fail L1:
+- Snowflake still holds every source value: a different column order, a Snowflake precision or scale that can hold the source's, and a `NOT NULL` source column whose Snowflake column is nullable (a Snowflake virtual column is always nullable).
+- Snowflake may truncate or reject source values, so review it: a source character length longer than Snowflake's.
+
+A nullable source column on a `NOT NULL` Snowflake column still fails, since Snowflake can reject the source's NULLs.
+
+A PostgreSQL user-defined `ENUM` column is validated as `ENUM` against a Snowflake `VARCHAR`. `information_schema` reports every enum as `USER-DEFINED`, so the metadata queries read `pg_type.typtype` to tell enums apart, and report the longest label as the column's length. L2 and L3 compare the label text.
+
 ### Metrics Validation
 Compares statistical metrics for each column:
 - Row count
@@ -73,6 +81,10 @@ Compares statistical metrics for each column:
 
 For some types these metrics do not depend on the values themselves (for SQL Server: `VECTOR`, `XML`, `IMAGE`, `SQL_VARIANT`, `JSON`, `GEOGRAPHY`, `GEOMETRY`, `UNIQUEIDENTIFIER`), so a changed value can pass L2. Use L3 row validation to compare those columns' values. The per-type list is in `docs/data-migration-orchestrator/sqlserver-data-type-coverage.md` in this repo.
 
+### Column pairing (L1 and L2)
+
+L1 and L2 read each column's catalog spelling rather than an upper-cased name, so the report shows PostgreSQL `order_id` and SQL Server `OrderId` as the catalog spells them. They pair a source column with its target column by the source platform's identifier rule, the same rule data migration uses: the exact name; on a case-insensitive source (SQL Server, Teradata, Redshift, Azure Synapse, BigQuery, SQLite) a name differing only by case; else the name as Snowflake reads it unquoted, so PostgreSQL `order_id` pairs with `ORDER_ID`. On Oracle, PostgreSQL and DB2, `MixedCase` and `mixedcase` are two columns, and a mixed-case source column whose target exists only as `MIXEDCASE` is reported missing from the target. A `column_mappings` key names a source column by the same rule: on Oracle the key `MixedCase` names `MIXEDCASE`, and `"MixedCase"` names `MixedCase`. Result-set and query inputs pair names exactly, else ignoring case.
+
 ### Views
 
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
@@ -123,6 +135,48 @@ For configuration options and examples, see the [official documentation](https:/
 
 ## Changelog
 
+### v1.45.1
+
+**Improvements**
+
+- Improved schema-level metadata retrieval by using `SHOW COLUMNS`.
+- Improved Iceberg timestamp normalization to follow the target physical type.
+
+**Bug fixes**
+
+- Fixed schema validation failing for source columns whose names end in a space.
+- Fixed BigQuery `ARRAY<INT64>` elements beyond 2^53 being compared as quoted JSON strings.
+- Fixed validation never passing for a SQL Server `NOT NULL` computed column migrated as a Snowflake virtual column.
+- Fixed L1 and L2 pairing of source and target columns to ignore case for case-sensitive sources when no target names are given.
+- Fixed SQL Server Iceberg L1 schema validation reporting a missing Snowflake character length.
+- Fixed Redshift `DATE`, `TIMESTAMP` and `TIMESTAMPTZ` values on Iceberg losing signed-year values.
+- Fixed Teradata numeric L3 text exceeding the `TO_CHAR` digit limit.
+- Fixed PostgreSQL `ENUM` columns surfacing as `USER-DEFINED`, so `ENUM` type rules never matched.
+- Fixed Teradata column metadata not reporting precision and scale, which made `TIME` pairing fall back to 6.
+- Fixed L1 failing `CHARACTER_MAXIMUM_LENGTH` for a SQL Server `xml` column migrated as `VARIANT`.
+
+### v1.45.0
+
+**New features**
+
+- Added support for applying custom L3 normalization per side, including `targetDataType: "*"` to target the mapped type.
+- Added the `includeCountDistinct` option; L2 `COUNT(DISTINCT)` is now off by default and opt-in globally or per table.
+- Added support for additional table and view types in data validation, including materialized views, temporary tables, BigQuery clones and snapshots, and DB2 materialized query tables.
+
+**Improvements**
+
+- Improved row comparison with positional column identity, so duplicate and case-variant column names no longer collapse.
+- Improved Redshift connection resiliency with TCP keepalive defaults.
+- Improved L1 validation to treat native integer bits-versus-digits precision differences as compatible, reported as a warning.
+
+**Bug fixes**
+
+- Fixed Oracle L2 metric rows and L3 cell frames upper-casing column names, which made mixed-case columns collide.
+- Fixed L3 failing rows over a lowercase SQL Server `UNIQUEIDENTIFIER` or an ignored `FLOAT` `targetExpression`.
+- Fixed a possible deadlock in expression casts when validators run on several threads.
+- Fixed Redshift and PostgreSQL `REAL` values being compared against a non-exact widened value.
+- Fixed Iceberg `FULL_INGEST` validation to follow the lossless-only temporal promotion of the load.
+
 ### v1.44.1
 
 **New features**

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/extractor/metadata_extractor_azure_synapse.py

@@ -78,13 +78,11 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
     def process_schema_query_result(
         self,
         result: list,
-        apply_column_validated_uppercase: bool = True,
     ) -> pd.DataFrame:
         """Process schema query results into a DataFrame for Azure Synapse.
 
         Args:
             result: Query result containing (columns_names, data_rows).
-            apply_column_validated_uppercase: Whether to uppercase column names.
 
         Returns:
             pd.DataFrame: Processed DataFrame with schema metadata.
@@ -94,7 +92,6 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
         df = self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(columns_names),
             data_rows=list(metadata_info),
-            apply_column_validated_uppercase=apply_column_validated_uppercase,
             sort_and_reset_index=True,
         )
         if COLUMN_DATATYPE in df.columns:
@@ -115,7 +112,6 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result_data),
-            apply_column_validated_uppercase=True,
             sort_and_reset_index=True,
         )
 
@@ -173,6 +169,5 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/extractor/templates/azure_synapse_datatypes_normalization_templates.yaml

@@ -52,6 +52,9 @@ NVARCHAR: '"{{ col_name }}"'
 DATE: CONVERT(VARCHAR(10), "{{ col_name }}", 23)
 # Same text as SQL Server TIME: hh:mi:ss.fffffff + '00', cut to {{ time_text_length }}.
 TIME: LEFT(SUBSTRING(CONVERT(VARCHAR(27), CAST("{{ col_name }}" AS DATETIME2(7)), 121), 12, 16) + '00', {{ time_text_length }})
+# Iceberg compatible TIME stored as VARCHAR (CETAS): the extract's own text, TIME(7)
+# widened then '00', nine digits whatever the scale. '+' keeps NULL as NULL.
+TIME_ICEBERG: CAST(CAST("{{ col_name }}" AS TIME(7)) AS VARCHAR(16)) + '00'
 DATETIME: STUFF(CONVERT(VARCHAR(27), CAST(CAST("{{ col_name }}" AS DATETIME2(3)) AS DATETIME2(7)), 121), 11, 1, 'T') + '00'
 DATETIME2: LEFT(STUFF(CONVERT(VARCHAR(27), CAST("{{ col_name }}" AS DATETIME2(7)), 121), 11, 1, 'T'), 26) + '000'
 SMALLDATETIME: CONVERT(VARCHAR(19), "{{ col_name }}", 126) + '.000000000'

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/query/query_generator_azure_synapse.py

@@ -53,6 +53,11 @@ from snowflake.snowflake_data_validation.sqlserver.query.charset_normalization i
 from snowflake.snowflake_data_validation.utils.constants import (
     Platform,
 )
+from snowflake.snowflake_data_validation.utils.iceberg_dv_common import (
+    IcebergProfile,
+    resolve_iceberg_compatible_tsql_source_normalization_key,
+)
+from snowflake.snowflake_data_validation.utils.iso_dv_common import get_paired_target_column_types
 from snowflake.snowflake_data_validation.utils.measure_collation import (
     TSQL_ROW_HASH_COLLATION_SUFFIX,
     tsql_comparison_collation_tokens,
@@ -141,8 +146,14 @@ class QueryGeneratorAzureSynapse(TsqlFloatMetricModifierMixin, QueryGeneratorBas
         datatypes_normalization_templates = table_context.templates_loader_manager.datatypes_normalization_templates
         rendered: dict[str, str] = {}
         renderer_columns = self._row_md5_normalization_columns(table_context)
+        paired_target_types = get_paired_target_column_types(table_context)
+        is_iceberg = IcebergProfile.from_table_context(table_context).is_active
         for column in renderer_columns:
-            lookup_key = (column.data_type or "").strip().upper()
+            lookup_key = resolve_iceberg_compatible_tsql_source_normalization_key(
+                (column.data_type or "").strip().upper(),
+                paired_target_type=paired_target_types.get(column.name),
+                is_iceberg=is_iceberg,
+            )
             normalization_template = self._lookup_normalization_template(
                 datatypes_normalization_templates,
                 lookup_key,

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/script_writer/script_writer_azure_synapse.py

@@ -99,7 +99,6 @@ class ScriptWriterAzureSynapse(ScriptWriterBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=result_columns,
             data_rows=result,
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )
 
@@ -137,6 +136,5 @@ class ScriptWriterAzureSynapse(ScriptWriterBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/bigquery/extractor/templates/bigquery_datatypes_normalization_templates.yaml

@@ -82,6 +82,12 @@ TIMESTAMP: FORMAT_TIMESTAMP('%E4Y-%m-%dT%H:%M:%E9S%Ez', `{{ col_name }}`, 'UTC')
 ARRAY: NULLIF(TO_JSON_STRING(PARSE_JSON(TO_JSON_STRING(`{{ col_name }}`))), 'null')
 STRUCT: NULLIF(TO_JSON_STRING(PARSE_JSON(TO_JSON_STRING(`{{ col_name }}`))), 'null')
 JSON: NULLIF(TO_JSON_STRING(PARSE_JSON(TO_JSON_STRING(`{{ col_name }}`))), 'null')
+# An ARRAY / STRUCT holding no NUMERIC / BIGNUMERIC (QueryGeneratorBigQuery picks this key)
+# goes through TO_JSON, as data migration extracts it: TO_JSON_STRING quotes an INT64 beyond
+# +/-2^53, while TO_JSON keeps it the number Snowflake holds. TO_JSON sorts object keys
+# itself, so no PARSE_JSON round trip. One holding a decimal keeps the keys above, since
+# TO_JSON rounds a wide NUMERIC to FLOAT64.
+NESTED_TO_JSON: NULLIF(TO_JSON_STRING(TO_JSON(`{{ col_name }}`)), 'null')
 # Spatial → WKT for L3 hashing. BigQuery ST_ASTEXT may emit "POINT (x y)" (space after
 # the type keyword) and ", " between vertices; Snowflake ST_ASWKT does not. Canonicalize
 # with chained REPLACE — preserve SQL NULL so null rows use the shared L3 sentinel.

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/bigquery/extractor/templates/bigquery_table_metadata_query.sql.j2

@@ -48,7 +48,10 @@ WITH rc_cte AS (
         (
             SELECT CASE table_type
                 WHEN 'BASE TABLE' THEN 'TABLE'
+                WHEN 'CLONE' THEN 'TABLE'
+                WHEN 'SNAPSHOT' THEN 'TABLE'
                 WHEN 'VIEW' THEN 'VIEW'
+                WHEN 'MATERIALIZED VIEW' THEN 'VIEW'
                 ELSE 'UNKNOWN'
             END
             FROM {{ object_database_sql }}.{{ object_schema_sql }}.INFORMATION_SCHEMA.TABLES

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/bigquery/query/query_generator_bigquery.py

@@ -17,7 +17,7 @@ from __future__ import annotations
 
 import logging
 
-from shared.datatypes.bigquery import BigQueryType
+from shared.datatypes.bigquery import BigQueryType, serializes_exactly_through_to_json
 from shared.datatypes.common import CommonType
 from shared.datatypes.snowflake import SnowflakeType
 from snowflake.snowflake_data_validation.query.decimal_text_normalization import (
@@ -46,6 +46,7 @@ from snowflake.snowflake_data_validation.utils.model.table_context import TableC
 from snowflake.snowflake_data_validation.validation.comparison_codec import (
     ComparisonCodec,
     apply_text_codec_to_renderer_templates,
+    has_custom_column_normalization,
     resolve_column_normalization_template,
 )
 
@@ -55,6 +56,9 @@ LOGGER = logging.getLogger(__name__)
 _WIDE_NUMERIC_BASE_TYPES = frozenset({BigQueryType.BIGNUMERIC, BigQueryType.BIGDECIMAL})
 # Normalization key for a BIGNUMERIC / BIGDECIMAL whose Snowflake mirror is DECFLOAT.
 BIGNUMERIC_DECFLOAT_NORMALIZATION_KEY = "BIGNUMERIC_DECFLOAT"
+# Normalization key for an ARRAY / STRUCT holding no NUMERIC / BIGNUMERIC, which data
+# migration serializes through TO_JSON so a wide INT64 lands as a number.
+NESTED_TO_JSON_NORMALIZATION_KEY = "NESTED_TO_JSON"
 
 
 def build_bigquery_float_bit_canonical_expr(col_ref: str) -> str:
@@ -239,6 +243,15 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
         base_type = self.normalize_bigquery_type(column.data_type)
         if self._lands_in_decfloat(column, table_context):
             base_type = BIGNUMERIC_DECFLOAT_NORMALIZATION_KEY
+        elif serializes_exactly_through_to_json(column.data_type) and not has_custom_column_normalization(
+            column.name,
+            base_type,
+            column_normalization_map=None,
+            templates_loader_manager=table_context.templates_loader_manager,
+        ):
+            # A workflow or custom-YAML override of ``ARRAY`` / ``STRUCT`` wins over the
+            # built-in TO_JSON text; a per-column rule replaces the template afterwards.
+            base_type = NESTED_TO_JSON_NORMALIZATION_KEY
         normalization_template = datatypes_normalization_templates.get(base_type)
         if normalization_template is None:
             LOGGER.debug(

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/configuration/model/configuration_model.py

@@ -96,6 +96,7 @@ class ConfigurationModel(BaseModel):
             self._check_max_failed_rows_number(table)
             self._check_chunk_number(table)
             self.set_exclude_metrics(table)
+            self.set_include_count_distinct(table)
             self.set_apply_metric_column_modifier(table)
             self.set_include_lob_columns_in_row_validation(table)
 
@@ -113,6 +114,7 @@ class ConfigurationModel(BaseModel):
             self._check_max_failed_rows_number(view)
             self._check_chunk_number(view)
             self.set_exclude_metrics(view)
+            self.set_include_count_distinct(view)
             self.set_apply_metric_column_modifier(view)
             self.set_include_lob_columns_in_row_validation(view)
 
@@ -198,6 +200,10 @@ class ConfigurationModel(BaseModel):
         if table.exclude_metrics is None:
             table.exclude_metrics = self.validation_configuration.exclude_metrics
 
+    def set_include_count_distinct(self, table: TableConfiguration) -> None:
+        if table.include_count_distinct is None:
+            table.include_count_distinct = self.validation_configuration.include_count_distinct
+
     def set_include_lob_columns_in_row_validation(self, table: TableConfiguration) -> None:
         if table.include_lob_columns_in_row_validation is None:
             table.include_lob_columns_in_row_validation = (

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/configuration/model/table_configuration.py

@@ -32,6 +32,7 @@ VISIBLE_PROPERTIES: set[str] = {
     "chunk_number",
     "column_mappings",
     "exclude_metrics",
+    "include_count_distinct",
     "include_lob_columns_in_row_validation",
     "index_column_list",
     "max_failed_rows_number",
@@ -83,6 +84,7 @@ class TableConfiguration(BaseModel):
     column_mappings: dict[str, str] = {}
     max_failed_rows_number: int | None = None
     exclude_metrics: bool | None = None
+    include_count_distinct: bool | None = None
     apply_metric_column_modifier: bool | None = None
     include_lob_columns_in_row_validation: bool | None = None
     object_type: ObjectType = ObjectType.TABLE

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/configuration/model/validation_configuration.py

@@ -41,6 +41,9 @@ class ValidationConfiguration(BaseModel):
     custom_templates_path: Path | None = None
     max_failed_rows_number: int = MAX_FAILED_ROWS_NUMBER_DEFAULT_VALUE
     exclude_metrics: bool = False
+    # L2 COUNT(DISTINCT) is opt-in (SNOW-4338258): it dominates L2 cost on the source
+    # (one SQL Server batch 26.4 s with it, 1.4 s without) and L2 is the fast metrics level.
+    include_count_distinct: bool = False
     apply_metric_column_modifier: bool = True
     include_lob_columns_in_row_validation: bool = False
 

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/db2/extractor/templates/db2_table_metadata_query.sql.j2

@@ -49,13 +49,15 @@ SELECT
     (
         SELECT CASE TYPE
             WHEN 'T' THEN 'TABLE'
+            WHEN 'G' THEN 'TABLE'
             WHEN 'V' THEN 'VIEW'
+            WHEN 'S' THEN 'VIEW'
             ELSE 'UNKNOWN'
         END
         FROM SYSCAT.TABLES
         WHERE TABSCHEMA = '{{ object_schema | escape_sql_literal }}'
           AND TABNAME = '{{ object_name | escape_sql_literal }}'
-          AND TYPE IN ('T', 'V')
+          AND TYPE IN ('T', 'G', 'V', 'S')
     ) AS OBJECT_TYPE,
     rc.ROW_COUNT
 FROM SYSCAT.COLUMNS C

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/db2/script_writer/script_writer_db2.py

@@ -88,6 +88,5 @@ class ScriptWriterDb2(ScriptWriterOracle):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=result_columns,
             data_rows=result,
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/async_generation_executor.py

@@ -66,19 +66,28 @@ class AsyncGenerationExecutor(BaseValidationExecutor):
 
         """
         try:
-            source_query = data_validation_api.generate_schema_query(source_table_context)
-            target_query = data_validation_api.generate_schema_query(target_table_context)
+            from snowflake.snowflake_data_validation.public.model import GeneratedQuery
+            from snowflake.snowflake_data_validation.public.query_generation.schema_metrics_queries import (
+                schema_queries_as_script_sql,
+            )
+
+            source_script = schema_queries_as_script_sql(
+                data_validation_api.generate_schema_queries(source_table_context)
+            )
+            target_script = schema_queries_as_script_sql(
+                data_validation_api.generate_schema_queries(target_table_context)
+            )
 
             data_validation_api.write_schema_query_to_script(
                 table_context=source_table_context,
-                generated_query=source_query,
+                generated_query=GeneratedQuery(query_string=source_script),
                 output_directory=self.source_extractor.report_path,
                 timestamp=self.context.run_start_time,
             )
 
             data_validation_api.write_schema_query_to_script(
                 table_context=target_table_context,
-                generated_query=target_query,
+                generated_query=GeneratedQuery(query_string=target_script),
                 output_directory=self.source_extractor.report_path,
                 timestamp=self.context.run_start_time,
             )

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/base_validation_executor.py

@@ -234,6 +234,7 @@ class BaseValidationExecutor(ABC):
             columns=source_table_column_metadata.columns,
             database_name=configured_object_name(self.context.source_platform, table_configuration.source_database),
             exclude_metrics=table_configuration.exclude_metrics,
+            include_count_distinct=bool(table_configuration.include_count_distinct),
             fully_qualified_name=table_configuration.fully_qualified_name,
             has_where_clause=table_configuration.has_where_clause,
             max_failed_rows_number=table_configuration.max_failed_rows_number,
@@ -279,6 +280,7 @@ class BaseValidationExecutor(ABC):
             columns=target_table_column_metadata.columns,
             database_name=configured_object_name(self.context.target_platform, table_configuration.target_database),
             exclude_metrics=table_configuration.exclude_metrics,
+            include_count_distinct=bool(table_configuration.include_count_distinct),
             fully_qualified_name=table_configuration.target_fully_qualified_name,
             has_where_clause=table_configuration.has_target_where_clause,
             max_failed_rows_number=table_configuration.max_failed_rows_number,

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/source_validation_executor.py

@@ -128,6 +128,7 @@ class SourceValidationExecutor:
             columns=source_table_column_metadata.columns,
             database_name=configured_object_name(self.context.source_platform, table_configuration.source_database),
             exclude_metrics=table_configuration.exclude_metrics,
+            include_count_distinct=bool(table_configuration.include_count_distinct),
             fully_qualified_name=table_configuration.fully_qualified_name,
             has_where_clause=table_configuration.has_where_clause,
             max_failed_rows_number=table_configuration.max_failed_rows_number,
@@ -159,13 +160,23 @@ class SourceValidationExecutor:
             source_table_context: Source table context
 
         """
-        generated_query = data_validation_api.generate_schema_query(source_table_context)
-        query_result = data_validation_api.execute_query(
-            table_context=source_table_context,
-            connector=self.source_extractor.connector,
-            generated_query=generated_query,
+        from snowflake.snowflake_data_validation.public.model import GeneratedQuery
+        from snowflake.snowflake_data_validation.public.model.query_generator_factory import (
+            QueryGeneratorFactory,
+        )
+
+        queries = data_validation_api.generate_schema_queries(source_table_context)
+        generator = QueryGeneratorFactory.create(source_table_context.platform)
+        schema_df = data_validation_api.fetch_schema_metadata(
+            queries,
+            lambda sql: data_validation_api.execute_query(
+                table_context=source_table_context,
+                connector=self.source_extractor.connector,
+                generated_query=GeneratedQuery(query_string=sql),
+            ).dataframe,
+            generator,
+            source_table_context,
         )
-        schema_df = query_result.dataframe
 
         output_path = self._get_output_path("schema", source_table_context)
         self._save_dataframe_as_parquet(schema_df, output_path)

{snowflake_data_validation-1.44.1 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/sync_validation_executor.py

@@ -28,10 +28,15 @@ public workflow API module:
 
 import hashlib
 
+from collections.abc import Callable
 from concurrent.futures import ThreadPoolExecutor
+from typing import Any
 
 import pandas as pd
 
+from snowflake.snowflake_data_validation.connector.connector_base import (
+    ConnectorBase,
+)
 from snowflake.snowflake_data_validation.executer.base_validation_executor import (
     BaseValidationExecutor,
 )
@@ -41,6 +46,7 @@ from snowflake.snowflake_data_validation.public import (
 from snowflake.snowflake_data_validation.public.model import (
     ConfigurationError,
     DataValidationExceptionError,
+    GeneratedQuery,
     MetricsValidationError,
     QueryExecutionResult,
     SchemaValidationError,
@@ -83,6 +89,20 @@ from snowflake.snowflake_data_validation.validation.validation_report_buffer imp
 )
 
 
+# ``execute_query``'s ``result_processor(raw_result, apply_column_validated_uppercase)``.
+SchemaResultProcessor = Callable[[list[Any], bool], pd.DataFrame]
+
+
+def _ignoring_case_flag(process: Callable[[list], pd.DataFrame]) -> Callable[[list, bool], pd.DataFrame]:
+    """Adapt an extractor's one-argument result callback to ``execute_query``'s processor contract.
+
+    ``execute_query`` calls ``result_processor(raw_result, apply_column_validated_uppercase)``.
+    The extractors take the raw result alone: they keep each column's catalog
+    spelling, and the validators pair names by the source's identifier rule.
+    """
+    return lambda result, _apply_column_validated_uppercase: process(result)
+
+
 class SyncValidationExecutor(BaseValidationExecutor):
     """Executor for synchronous validation operations.
 
@@ -99,6 +119,24 @@ class SyncValidationExecutor(BaseValidationExecutor):
     parallelization of source and target operations.
     """
 
+    def _schema_side_executor(
+        self,
+        table_context: TableContext,
+        connector: ConnectorBase,
+        processor: SchemaResultProcessor,
+    ) -> Callable[[str], pd.DataFrame]:
+        """Return the runner for one side's schema statements."""
+
+        def execute(sql: str) -> pd.DataFrame:
+            return data_validation_api.execute_query(
+                table_context=table_context,
+                connector=connector,
+                generated_query=GeneratedQuery(query_string=sql),
+                result_processor=processor,
+            ).dataframe
+
+        return execute
+
     @report_telemetry(params_list=["source_table_context", "target_table_context"])
     def execute_schema_validation(
         self,
@@ -121,11 +159,13 @@ class SyncValidationExecutor(BaseValidationExecutor):
 
         """
         try:
-            source_query = data_validation_api.generate_schema_query(source_table_context)
-            target_query = data_validation_api.generate_schema_query(target_table_context)
+            source_queries = data_validation_api.generate_schema_queries(source_table_context)
+            target_queries = data_validation_api.generate_schema_queries(target_table_context)
+            source_generator = QueryGeneratorFactory.create(source_table_context.platform)
+            target_generator = QueryGeneratorFactory.create(target_table_context.platform)
 
             self.source_extractor.prepare_for_table(source_table_context)
-            source_processor = self.source_extractor.process_schema_query_result
+            source_processor = _ignoring_case_flag(self.source_extractor.process_schema_query_result)
 
             # SNOW-3981941: Teradata view L1 now hits DBC.ColumnsQV. If that
             # catalog query fails for any reason (9719, access, wrapped/masked
@@ -136,12 +176,17 @@ class SyncValidationExecutor(BaseValidationExecutor):
             is_teradata_view = source_table_context.object_type == ObjectType.VIEW and self.context.source_platform == Platform.TERADATA
 
             try:
-                source_result = data_validation_api.execute_query(
-                    table_context=source_table_context,
-                    connector=self.source_extractor.connector,
-                    generated_query=source_query,
-                    result_processor=source_processor,
+                source_df = data_validation_api.fetch_schema_metadata(
+                    source_queries,
+                    self._schema_side_executor(
+                        source_table_context,
+                        self.source_extractor.connector,
+                        source_processor,
+                    ),
+                    source_generator,
+                    source_table_context,
                 )
+                source_result = QueryExecutionResult(dataframe=source_df)
             except Exception as exc:
                 if not is_teradata_view:
                     raise
@@ -176,12 +221,17 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 object_name=source_table_context.fully_qualified_name,
             )
 
-            target_result = data_validation_api.execute_query(
-                table_context=target_table_context,
-                connector=self.target_extractor.connector,
-                generated_query=target_query,
-                result_processor=self.target_extractor.process_schema_query_result,
+            target_df = data_validation_api.fetch_schema_metadata(
+                target_queries,
+                self._schema_side_executor(
+                    target_table_context,
+                    self.target_extractor.connector,
+                    _ignoring_case_flag(self.target_extractor.process_schema_query_result),
+                ),
+                target_generator,
+                target_table_context,
             )
+            target_result = QueryExecutionResult(dataframe=target_df)
 
             # SNOW-3981941: Teradata views used to be forced through
             # ``normalize_table_metadata`` + ``basic_validation=True``, stripping
@@ -308,8 +358,7 @@ class SyncValidationExecutor(BaseValidationExecutor):
             if not source_queries or not target_queries:
                 raise RuntimeError(f"generate_batched_metrics_queries returned no queries for {source_table_context.fully_qualified_name}")
 
-            def source_processor(result, _):
-                return self.source_extractor.process_metrics_query_result(result)
+            source_processor = _ignoring_case_flag(self.source_extractor.process_metrics_query_result)
 
             source_dfs: list[pd.DataFrame] = []
             for source_batch in source_queries:
@@ -322,8 +371,7 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 source_dfs.append(reshape_wide_metrics_result(batch_result.dataframe, source_batch.result_shape))
             source_result = QueryExecutionResult(dataframe=pd.concat(source_dfs, ignore_index=True))
 
-            def target_processor(result, _):
-                return self.target_extractor.process_metrics_query_result(result)
+            target_processor = _ignoring_case_flag(self.target_extractor.process_metrics_query_result)
 
             target_dfs: list[pd.DataFrame] = []
             for target_batch in target_queries: