snowflake-data-validation 1.45.0__tar.gz → 1.45.1__tar.gz

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: snowflake-data-validation
-Version: 1.45.0
+Version: 1.45.1
 Summary: Snowflake Data Validation
 Project-URL: Bug Tracker, https://github.com/snowflake-eng/migrations-data-validation/issues
 Project-URL: Source code, https://github.com/snowflake-eng/migrations-data-validation/
@@ -154,6 +154,14 @@ Compares table structure between source and target:
 - Precision, scale, and length
 - Nullable constraints
 
+Some differences are reported as `WARNING` and do not fail L1:
+- Snowflake still holds every source value: a different column order, a Snowflake precision or scale that can hold the source's, and a `NOT NULL` source column whose Snowflake column is nullable (a Snowflake virtual column is always nullable).
+- Snowflake may truncate or reject source values, so review it: a source character length longer than Snowflake's.
+
+A nullable source column on a `NOT NULL` Snowflake column still fails, since Snowflake can reject the source's NULLs.
+
+A PostgreSQL user-defined `ENUM` column is validated as `ENUM` against a Snowflake `VARCHAR`. `information_schema` reports every enum as `USER-DEFINED`, so the metadata queries read `pg_type.typtype` to tell enums apart, and report the longest label as the column's length. L2 and L3 compare the label text.
+
 ### Metrics Validation
 Compares statistical metrics for each column:
 - Row count
@@ -164,6 +172,10 @@ Compares statistical metrics for each column:
 
 For some types these metrics do not depend on the values themselves (for SQL Server: `VECTOR`, `XML`, `IMAGE`, `SQL_VARIANT`, `JSON`, `GEOGRAPHY`, `GEOMETRY`, `UNIQUEIDENTIFIER`), so a changed value can pass L2. Use L3 row validation to compare those columns' values. The per-type list is in `docs/data-migration-orchestrator/sqlserver-data-type-coverage.md` in this repo.
 
+### Column pairing (L1 and L2)
+
+L1 and L2 read each column's catalog spelling rather than an upper-cased name, so the report shows PostgreSQL `order_id` and SQL Server `OrderId` as the catalog spells them. They pair a source column with its target column by the source platform's identifier rule, the same rule data migration uses: the exact name; on a case-insensitive source (SQL Server, Teradata, Redshift, Azure Synapse, BigQuery, SQLite) a name differing only by case; else the name as Snowflake reads it unquoted, so PostgreSQL `order_id` pairs with `ORDER_ID`. On Oracle, PostgreSQL and DB2, `MixedCase` and `mixedcase` are two columns, and a mixed-case source column whose target exists only as `MIXEDCASE` is reported missing from the target. A `column_mappings` key names a source column by the same rule: on Oracle the key `MixedCase` names `MIXEDCASE`, and `"MixedCase"` names `MixedCase`. Result-set and query inputs pair names exactly, else ignoring case.
+
 ### Views
 
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
@@ -214,6 +226,26 @@ For configuration options and examples, see the [official documentation](https:/
 
 ## Changelog
 
+### v1.45.1
+
+**Improvements**
+
+- Improved schema-level metadata retrieval by using `SHOW COLUMNS`.
+- Improved Iceberg timestamp normalization to follow the target physical type.
+
+**Bug fixes**
+
+- Fixed schema validation failing for source columns whose names end in a space.
+- Fixed BigQuery `ARRAY<INT64>` elements beyond 2^53 being compared as quoted JSON strings.
+- Fixed validation never passing for a SQL Server `NOT NULL` computed column migrated as a Snowflake virtual column.
+- Fixed L1 and L2 pairing of source and target columns to ignore case for case-sensitive sources when no target names are given.
+- Fixed SQL Server Iceberg L1 schema validation reporting a missing Snowflake character length.
+- Fixed Redshift `DATE`, `TIMESTAMP` and `TIMESTAMPTZ` values on Iceberg losing signed-year values.
+- Fixed Teradata numeric L3 text exceeding the `TO_CHAR` digit limit.
+- Fixed PostgreSQL `ENUM` columns surfacing as `USER-DEFINED`, so `ENUM` type rules never matched.
+- Fixed Teradata column metadata not reporting precision and scale, which made `TIME` pairing fall back to 6.
+- Fixed L1 failing `CHARACTER_MAXIMUM_LENGTH` for a SQL Server `xml` column migrated as `VARIANT`.
+
 ### v1.45.0
 
 **New features**

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/README.md

@@ -63,6 +63,14 @@ Compares table structure between source and target:
 - Precision, scale, and length
 - Nullable constraints
 
+Some differences are reported as `WARNING` and do not fail L1:
+- Snowflake still holds every source value: a different column order, a Snowflake precision or scale that can hold the source's, and a `NOT NULL` source column whose Snowflake column is nullable (a Snowflake virtual column is always nullable).
+- Snowflake may truncate or reject source values, so review it: a source character length longer than Snowflake's.
+
+A nullable source column on a `NOT NULL` Snowflake column still fails, since Snowflake can reject the source's NULLs.
+
+A PostgreSQL user-defined `ENUM` column is validated as `ENUM` against a Snowflake `VARCHAR`. `information_schema` reports every enum as `USER-DEFINED`, so the metadata queries read `pg_type.typtype` to tell enums apart, and report the longest label as the column's length. L2 and L3 compare the label text.
+
 ### Metrics Validation
 Compares statistical metrics for each column:
 - Row count
@@ -73,6 +81,10 @@ Compares statistical metrics for each column:
 
 For some types these metrics do not depend on the values themselves (for SQL Server: `VECTOR`, `XML`, `IMAGE`, `SQL_VARIANT`, `JSON`, `GEOGRAPHY`, `GEOMETRY`, `UNIQUEIDENTIFIER`), so a changed value can pass L2. Use L3 row validation to compare those columns' values. The per-type list is in `docs/data-migration-orchestrator/sqlserver-data-type-coverage.md` in this repo.
 
+### Column pairing (L1 and L2)
+
+L1 and L2 read each column's catalog spelling rather than an upper-cased name, so the report shows PostgreSQL `order_id` and SQL Server `OrderId` as the catalog spells them. They pair a source column with its target column by the source platform's identifier rule, the same rule data migration uses: the exact name; on a case-insensitive source (SQL Server, Teradata, Redshift, Azure Synapse, BigQuery, SQLite) a name differing only by case; else the name as Snowflake reads it unquoted, so PostgreSQL `order_id` pairs with `ORDER_ID`. On Oracle, PostgreSQL and DB2, `MixedCase` and `mixedcase` are two columns, and a mixed-case source column whose target exists only as `MIXEDCASE` is reported missing from the target. A `column_mappings` key names a source column by the same rule: on Oracle the key `MixedCase` names `MIXEDCASE`, and `"MixedCase"` names `MixedCase`. Result-set and query inputs pair names exactly, else ignoring case.
+
 ### Views
 
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
@@ -123,6 +135,26 @@ For configuration options and examples, see the [official documentation](https:/
 
 ## Changelog
 
+### v1.45.1
+
+**Improvements**
+
+- Improved schema-level metadata retrieval by using `SHOW COLUMNS`.
+- Improved Iceberg timestamp normalization to follow the target physical type.
+
+**Bug fixes**
+
+- Fixed schema validation failing for source columns whose names end in a space.
+- Fixed BigQuery `ARRAY<INT64>` elements beyond 2^53 being compared as quoted JSON strings.
+- Fixed validation never passing for a SQL Server `NOT NULL` computed column migrated as a Snowflake virtual column.
+- Fixed L1 and L2 pairing of source and target columns to ignore case for case-sensitive sources when no target names are given.
+- Fixed SQL Server Iceberg L1 schema validation reporting a missing Snowflake character length.
+- Fixed Redshift `DATE`, `TIMESTAMP` and `TIMESTAMPTZ` values on Iceberg losing signed-year values.
+- Fixed Teradata numeric L3 text exceeding the `TO_CHAR` digit limit.
+- Fixed PostgreSQL `ENUM` columns surfacing as `USER-DEFINED`, so `ENUM` type rules never matched.
+- Fixed Teradata column metadata not reporting precision and scale, which made `TIME` pairing fall back to 6.
+- Fixed L1 failing `CHARACTER_MAXIMUM_LENGTH` for a SQL Server `xml` column migrated as `VARIANT`.
+
 ### v1.45.0
 
 **New features**

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/extractor/metadata_extractor_azure_synapse.py

@@ -78,13 +78,11 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
     def process_schema_query_result(
         self,
         result: list,
-        apply_column_validated_uppercase: bool = True,
     ) -> pd.DataFrame:
         """Process schema query results into a DataFrame for Azure Synapse.
 
         Args:
             result: Query result containing (columns_names, data_rows).
-            apply_column_validated_uppercase: Whether to uppercase column names.
 
         Returns:
             pd.DataFrame: Processed DataFrame with schema metadata.
@@ -94,7 +92,6 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
         df = self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(columns_names),
             data_rows=list(metadata_info),
-            apply_column_validated_uppercase=apply_column_validated_uppercase,
             sort_and_reset_index=True,
         )
         if COLUMN_DATATYPE in df.columns:
@@ -115,7 +112,6 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result_data),
-            apply_column_validated_uppercase=True,
             sort_and_reset_index=True,
         )
 
@@ -173,6 +169,5 @@ class MetadataExtractorAzureSynapse(MetadataExtractorBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/query/query_generator_azure_synapse.py

@@ -54,6 +54,7 @@ from snowflake.snowflake_data_validation.utils.constants import (
     Platform,
 )
 from snowflake.snowflake_data_validation.utils.iceberg_dv_common import (
+    IcebergProfile,
     resolve_iceberg_compatible_tsql_source_normalization_key,
 )
 from snowflake.snowflake_data_validation.utils.iso_dv_common import get_paired_target_column_types
@@ -146,14 +147,12 @@ class QueryGeneratorAzureSynapse(TsqlFloatMetricModifierMixin, QueryGeneratorBas
         rendered: dict[str, str] = {}
         renderer_columns = self._row_md5_normalization_columns(table_context)
         paired_target_types = get_paired_target_column_types(table_context)
-        iceberg_spec = table_context.iceberg_spec
-        storage_mode = table_context.storage_mode
+        is_iceberg = IcebergProfile.from_table_context(table_context).is_active
         for column in renderer_columns:
             lookup_key = resolve_iceberg_compatible_tsql_source_normalization_key(
                 (column.data_type or "").strip().upper(),
                 paired_target_type=paired_target_types.get(column.name),
-                iceberg_spec=iceberg_spec,
-                storage_mode=storage_mode,
+                is_iceberg=is_iceberg,
             )
             normalization_template = self._lookup_normalization_template(
                 datatypes_normalization_templates,

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/azure_synapse/script_writer/script_writer_azure_synapse.py

@@ -99,7 +99,6 @@ class ScriptWriterAzureSynapse(ScriptWriterBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=result_columns,
             data_rows=result,
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )
 
@@ -137,6 +136,5 @@ class ScriptWriterAzureSynapse(ScriptWriterBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/bigquery/extractor/templates/bigquery_datatypes_normalization_templates.yaml

@@ -82,6 +82,12 @@ TIMESTAMP: FORMAT_TIMESTAMP('%E4Y-%m-%dT%H:%M:%E9S%Ez', `{{ col_name }}`, 'UTC')
 ARRAY: NULLIF(TO_JSON_STRING(PARSE_JSON(TO_JSON_STRING(`{{ col_name }}`))), 'null')
 STRUCT: NULLIF(TO_JSON_STRING(PARSE_JSON(TO_JSON_STRING(`{{ col_name }}`))), 'null')
 JSON: NULLIF(TO_JSON_STRING(PARSE_JSON(TO_JSON_STRING(`{{ col_name }}`))), 'null')
+# An ARRAY / STRUCT holding no NUMERIC / BIGNUMERIC (QueryGeneratorBigQuery picks this key)
+# goes through TO_JSON, as data migration extracts it: TO_JSON_STRING quotes an INT64 beyond
+# +/-2^53, while TO_JSON keeps it the number Snowflake holds. TO_JSON sorts object keys
+# itself, so no PARSE_JSON round trip. One holding a decimal keeps the keys above, since
+# TO_JSON rounds a wide NUMERIC to FLOAT64.
+NESTED_TO_JSON: NULLIF(TO_JSON_STRING(TO_JSON(`{{ col_name }}`)), 'null')
 # Spatial → WKT for L3 hashing. BigQuery ST_ASTEXT may emit "POINT (x y)" (space after
 # the type keyword) and ", " between vertices; Snowflake ST_ASWKT does not. Canonicalize
 # with chained REPLACE — preserve SQL NULL so null rows use the shared L3 sentinel.

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/bigquery/query/query_generator_bigquery.py

@@ -17,7 +17,7 @@ from __future__ import annotations
 
 import logging
 
-from shared.datatypes.bigquery import BigQueryType
+from shared.datatypes.bigquery import BigQueryType, serializes_exactly_through_to_json
 from shared.datatypes.common import CommonType
 from shared.datatypes.snowflake import SnowflakeType
 from snowflake.snowflake_data_validation.query.decimal_text_normalization import (
@@ -46,6 +46,7 @@ from snowflake.snowflake_data_validation.utils.model.table_context import TableC
 from snowflake.snowflake_data_validation.validation.comparison_codec import (
     ComparisonCodec,
     apply_text_codec_to_renderer_templates,
+    has_custom_column_normalization,
     resolve_column_normalization_template,
 )
 
@@ -55,6 +56,9 @@ LOGGER = logging.getLogger(__name__)
 _WIDE_NUMERIC_BASE_TYPES = frozenset({BigQueryType.BIGNUMERIC, BigQueryType.BIGDECIMAL})
 # Normalization key for a BIGNUMERIC / BIGDECIMAL whose Snowflake mirror is DECFLOAT.
 BIGNUMERIC_DECFLOAT_NORMALIZATION_KEY = "BIGNUMERIC_DECFLOAT"
+# Normalization key for an ARRAY / STRUCT holding no NUMERIC / BIGNUMERIC, which data
+# migration serializes through TO_JSON so a wide INT64 lands as a number.
+NESTED_TO_JSON_NORMALIZATION_KEY = "NESTED_TO_JSON"
 
 
 def build_bigquery_float_bit_canonical_expr(col_ref: str) -> str:
@@ -239,6 +243,15 @@ class QueryGeneratorBigQuery(QueryGeneratorBase):
         base_type = self.normalize_bigquery_type(column.data_type)
         if self._lands_in_decfloat(column, table_context):
             base_type = BIGNUMERIC_DECFLOAT_NORMALIZATION_KEY
+        elif serializes_exactly_through_to_json(column.data_type) and not has_custom_column_normalization(
+            column.name,
+            base_type,
+            column_normalization_map=None,
+            templates_loader_manager=table_context.templates_loader_manager,
+        ):
+            # A workflow or custom-YAML override of ``ARRAY`` / ``STRUCT`` wins over the
+            # built-in TO_JSON text; a per-column rule replaces the template afterwards.
+            base_type = NESTED_TO_JSON_NORMALIZATION_KEY
         normalization_template = datatypes_normalization_templates.get(base_type)
         if normalization_template is None:
             LOGGER.debug(

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/db2/script_writer/script_writer_db2.py

@@ -88,6 +88,5 @@ class ScriptWriterDb2(ScriptWriterOracle):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=result_columns,
             data_rows=result,
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/async_generation_executor.py

@@ -66,19 +66,28 @@ class AsyncGenerationExecutor(BaseValidationExecutor):
 
         """
         try:
-            source_query = data_validation_api.generate_schema_query(source_table_context)
-            target_query = data_validation_api.generate_schema_query(target_table_context)
+            from snowflake.snowflake_data_validation.public.model import GeneratedQuery
+            from snowflake.snowflake_data_validation.public.query_generation.schema_metrics_queries import (
+                schema_queries_as_script_sql,
+            )
+
+            source_script = schema_queries_as_script_sql(
+                data_validation_api.generate_schema_queries(source_table_context)
+            )
+            target_script = schema_queries_as_script_sql(
+                data_validation_api.generate_schema_queries(target_table_context)
+            )
 
             data_validation_api.write_schema_query_to_script(
                 table_context=source_table_context,
-                generated_query=source_query,
+                generated_query=GeneratedQuery(query_string=source_script),
                 output_directory=self.source_extractor.report_path,
                 timestamp=self.context.run_start_time,
             )
 
             data_validation_api.write_schema_query_to_script(
                 table_context=target_table_context,
-                generated_query=target_query,
+                generated_query=GeneratedQuery(query_string=target_script),
                 output_directory=self.source_extractor.report_path,
                 timestamp=self.context.run_start_time,
             )

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/source_validation_executor.py

@@ -160,13 +160,23 @@ class SourceValidationExecutor:
             source_table_context: Source table context
 
         """
-        generated_query = data_validation_api.generate_schema_query(source_table_context)
-        query_result = data_validation_api.execute_query(
-            table_context=source_table_context,
-            connector=self.source_extractor.connector,
-            generated_query=generated_query,
+        from snowflake.snowflake_data_validation.public.model import GeneratedQuery
+        from snowflake.snowflake_data_validation.public.model.query_generator_factory import (
+            QueryGeneratorFactory,
+        )
+
+        queries = data_validation_api.generate_schema_queries(source_table_context)
+        generator = QueryGeneratorFactory.create(source_table_context.platform)
+        schema_df = data_validation_api.fetch_schema_metadata(
+            queries,
+            lambda sql: data_validation_api.execute_query(
+                table_context=source_table_context,
+                connector=self.source_extractor.connector,
+                generated_query=GeneratedQuery(query_string=sql),
+            ).dataframe,
+            generator,
+            source_table_context,
         )
-        schema_df = query_result.dataframe
 
         output_path = self._get_output_path("schema", source_table_context)
         self._save_dataframe_as_parquet(schema_df, output_path)

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/executer/sync_validation_executor.py

@@ -28,10 +28,15 @@ public workflow API module:
 
 import hashlib
 
+from collections.abc import Callable
 from concurrent.futures import ThreadPoolExecutor
+from typing import Any
 
 import pandas as pd
 
+from snowflake.snowflake_data_validation.connector.connector_base import (
+    ConnectorBase,
+)
 from snowflake.snowflake_data_validation.executer.base_validation_executor import (
     BaseValidationExecutor,
 )
@@ -41,6 +46,7 @@ from snowflake.snowflake_data_validation.public import (
 from snowflake.snowflake_data_validation.public.model import (
     ConfigurationError,
     DataValidationExceptionError,
+    GeneratedQuery,
     MetricsValidationError,
     QueryExecutionResult,
     SchemaValidationError,
@@ -83,6 +89,20 @@ from snowflake.snowflake_data_validation.validation.validation_report_buffer imp
 )
 
 
+# ``execute_query``'s ``result_processor(raw_result, apply_column_validated_uppercase)``.
+SchemaResultProcessor = Callable[[list[Any], bool], pd.DataFrame]
+
+
+def _ignoring_case_flag(process: Callable[[list], pd.DataFrame]) -> Callable[[list, bool], pd.DataFrame]:
+    """Adapt an extractor's one-argument result callback to ``execute_query``'s processor contract.
+
+    ``execute_query`` calls ``result_processor(raw_result, apply_column_validated_uppercase)``.
+    The extractors take the raw result alone: they keep each column's catalog
+    spelling, and the validators pair names by the source's identifier rule.
+    """
+    return lambda result, _apply_column_validated_uppercase: process(result)
+
+
 class SyncValidationExecutor(BaseValidationExecutor):
     """Executor for synchronous validation operations.
 
@@ -99,6 +119,24 @@ class SyncValidationExecutor(BaseValidationExecutor):
     parallelization of source and target operations.
     """
 
+    def _schema_side_executor(
+        self,
+        table_context: TableContext,
+        connector: ConnectorBase,
+        processor: SchemaResultProcessor,
+    ) -> Callable[[str], pd.DataFrame]:
+        """Return the runner for one side's schema statements."""
+
+        def execute(sql: str) -> pd.DataFrame:
+            return data_validation_api.execute_query(
+                table_context=table_context,
+                connector=connector,
+                generated_query=GeneratedQuery(query_string=sql),
+                result_processor=processor,
+            ).dataframe
+
+        return execute
+
     @report_telemetry(params_list=["source_table_context", "target_table_context"])
     def execute_schema_validation(
         self,
@@ -121,11 +159,13 @@ class SyncValidationExecutor(BaseValidationExecutor):
 
         """
         try:
-            source_query = data_validation_api.generate_schema_query(source_table_context)
-            target_query = data_validation_api.generate_schema_query(target_table_context)
+            source_queries = data_validation_api.generate_schema_queries(source_table_context)
+            target_queries = data_validation_api.generate_schema_queries(target_table_context)
+            source_generator = QueryGeneratorFactory.create(source_table_context.platform)
+            target_generator = QueryGeneratorFactory.create(target_table_context.platform)
 
             self.source_extractor.prepare_for_table(source_table_context)
-            source_processor = self.source_extractor.process_schema_query_result
+            source_processor = _ignoring_case_flag(self.source_extractor.process_schema_query_result)
 
             # SNOW-3981941: Teradata view L1 now hits DBC.ColumnsQV. If that
             # catalog query fails for any reason (9719, access, wrapped/masked
@@ -136,12 +176,17 @@ class SyncValidationExecutor(BaseValidationExecutor):
             is_teradata_view = source_table_context.object_type == ObjectType.VIEW and self.context.source_platform == Platform.TERADATA
 
             try:
-                source_result = data_validation_api.execute_query(
-                    table_context=source_table_context,
-                    connector=self.source_extractor.connector,
-                    generated_query=source_query,
-                    result_processor=source_processor,
+                source_df = data_validation_api.fetch_schema_metadata(
+                    source_queries,
+                    self._schema_side_executor(
+                        source_table_context,
+                        self.source_extractor.connector,
+                        source_processor,
+                    ),
+                    source_generator,
+                    source_table_context,
                 )
+                source_result = QueryExecutionResult(dataframe=source_df)
             except Exception as exc:
                 if not is_teradata_view:
                     raise
@@ -176,12 +221,17 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 object_name=source_table_context.fully_qualified_name,
             )
 
-            target_result = data_validation_api.execute_query(
-                table_context=target_table_context,
-                connector=self.target_extractor.connector,
-                generated_query=target_query,
-                result_processor=self.target_extractor.process_schema_query_result,
+            target_df = data_validation_api.fetch_schema_metadata(
+                target_queries,
+                self._schema_side_executor(
+                    target_table_context,
+                    self.target_extractor.connector,
+                    _ignoring_case_flag(self.target_extractor.process_schema_query_result),
+                ),
+                target_generator,
+                target_table_context,
             )
+            target_result = QueryExecutionResult(dataframe=target_df)
 
             # SNOW-3981941: Teradata views used to be forced through
             # ``normalize_table_metadata`` + ``basic_validation=True``, stripping
@@ -308,8 +358,7 @@ class SyncValidationExecutor(BaseValidationExecutor):
             if not source_queries or not target_queries:
                 raise RuntimeError(f"generate_batched_metrics_queries returned no queries for {source_table_context.fully_qualified_name}")
 
-            def source_processor(result, _):
-                return self.source_extractor.process_metrics_query_result(result)
+            source_processor = _ignoring_case_flag(self.source_extractor.process_metrics_query_result)
 
             source_dfs: list[pd.DataFrame] = []
             for source_batch in source_queries:
@@ -322,8 +371,7 @@ class SyncValidationExecutor(BaseValidationExecutor):
                 source_dfs.append(reshape_wide_metrics_result(batch_result.dataframe, source_batch.result_shape))
             source_result = QueryExecutionResult(dataframe=pd.concat(source_dfs, ignore_index=True))
 
-            def target_processor(result, _):
-                return self.target_extractor.process_metrics_query_result(result)
+            target_processor = _ignoring_case_flag(self.target_extractor.process_metrics_query_result)
 
             target_dfs: list[pd.DataFrame] = []
             for target_batch in target_queries:

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/extractor/metadata_extractor_base.py

@@ -109,10 +109,26 @@ class MetadataExtractorBase(ABC):
             table_context.fully_qualified_name,
         )
 
-        sql_query = self.query_generator.generate_schema_query(table_context)
+        from snowflake.snowflake_data_validation.public.query_generation.schema_metrics_queries import (
+            fetch_schema_metadata,
+            generate_schema_queries,
+        )
+
+        queries = generate_schema_queries(table_context, self.query_generator)
+
+        def _execute(sql: str) -> pd.DataFrame:
+            result = self.connector.execute_query(sql)
+            if not result:
+                return pd.DataFrame()
+            return self.process_schema_query_result(result)
 
         try:
-            result = self.connector.execute_query(sql_query)
+            dataframe = fetch_schema_metadata(
+                queries,
+                _execute,
+                self.query_generator,
+                table_context,
+            )
         except Exception:
             error_message = self._get_error_message(
                 "Schema validation query failed",
@@ -121,17 +137,12 @@ class MetadataExtractorBase(ABC):
             LOGGER.critical(error_message)
             raise
 
-        if not result:
+        if dataframe.empty:
             output_handler.handle_message(
                 message=f"No metadata found for table: {table_context.fully_qualified_name}",
                 level=OutputMessageLevel.WARNING,
             )
-            return pd.DataFrame()
-
-        return self.process_schema_query_result(
-            result,
-            True,
-        )
+        return dataframe
 
     @log
     def extract_metrics_metadata(
@@ -232,7 +243,6 @@ class MetadataExtractorBase(ABC):
         origin: Origin,
         result: list,
         output_handler: OutputHandlerBase,
-        apply_column_validated_uppercase: bool = True,
     ) -> pd.DataFrame:
         """Process schema query results into a DataFrame.
 
@@ -240,7 +250,6 @@ class MetadataExtractorBase(ABC):
             origin: The origin enum value.
             result: Query result list containing columns and data.
             output_handler: Output handler for logging and reporting.
-            apply_column_validated_uppercase: whether returned dataframe should have columns in uppercase.
 
         Returns:
             pd.DataFrame: Processed DataFrame with schema metadata.

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/extractor/sql_queries_template_generator.py

@@ -742,6 +742,7 @@ class SQLQueriesTemplateGenerator:
         hash_md5_function: str | None = None,
         row_hash_collation_suffix: str = "",
         write_identifier: IdentifierWriter | None = None,
+        extra_from_joins: list[str] | None = None,
     ) -> str:
         """Render an inline SELECT that returns (index_cols, ROW_MD5) directly.
 
@@ -749,7 +750,9 @@ class SQLQueriesTemplateGenerator:
         no temp tables and no OFFSET/FETCH.  When *order_by_columns* is
         provided the result set is sorted by those columns. ``write_identifier``
         writes a column name as the dialect's identifier (see
-        ``column_sql_names``).
+        ``column_sql_names``). ``extra_from_joins`` (Redshift SUPER sorted-object
+        joins) is appended after the base ``FROM``; templates that omit it are
+        unaffected.
         """
         template_file = f"{platform}_row_md5_select.sql.j2"
         try:
@@ -765,6 +768,7 @@ class SQLQueriesTemplateGenerator:
                 order_by_columns=order_by_columns or [],
                 order_by_collation_expressions=order_by_collation_expressions or {},
                 row_hash_collation_suffix=row_hash_collation_suffix,
+                extra_from_joins=extra_from_joins or [],
                 **column_sql_names(platform, write_identifier),
             )
             if column_groups is not None:

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/oracle/extractor/metadata_extractor_oracle.py

@@ -151,14 +151,12 @@ class MetadataExtractorOracle(MetadataExtractorBase):
 
         return self.process_schema_query_result(
             result,
-            True,
             table_context=table_context,
         )
 
     def process_schema_query_result(
         self,
         result: list,
-        apply_column_validated_uppercase: bool = True,
         *,
         table_context: TableContext | None = None,
     ) -> pd.DataFrame:
@@ -166,9 +164,6 @@ class MetadataExtractorOracle(MetadataExtractorBase):
 
         Args:
             result: Query result containing (columns_names, data_rows).
-            apply_column_validated_uppercase: Whether to uppercase the
-                COLUMN_VALIDATED values. Defaults to True; Oracle returns
-                column names in uppercase so this is normally a no-op.
             table_context: When provided, run a VECTOR_INFO follow-up if
                 phase-1 schema shows VECTOR columns without dimension.
 
@@ -180,7 +175,6 @@ class MetadataExtractorOracle(MetadataExtractorBase):
         df = self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(columns_names),
             data_rows=list(metadata_info),
-            apply_column_validated_uppercase=apply_column_validated_uppercase,
             sort_and_reset_index=True,
         )
 
@@ -240,7 +234,6 @@ class MetadataExtractorOracle(MetadataExtractorBase):
         vector_df = self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(vector_rows),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )
         merge_vector_metadata(rows, vector_df.to_dict(orient="records"))
@@ -260,7 +253,6 @@ class MetadataExtractorOracle(MetadataExtractorBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result_data),
-            apply_column_validated_uppercase=True,
             sort_and_reset_index=True,
         )
 
@@ -327,6 +319,5 @@ class MetadataExtractorOracle(MetadataExtractorBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )

{snowflake_data_validation-1.45.0 → snowflake_data_validation-1.45.1}/src/snowflake/snowflake_data_validation/oracle/script_writer/script_writer_oracle.py

@@ -95,7 +95,6 @@ class ScriptWriterOracle(ScriptWriterBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=result_columns,
             data_rows=result,
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )
 
@@ -131,6 +130,5 @@ class ScriptWriterOracle(ScriptWriterBase):
         return self.helper_dataframe.process_query_result_to_dataframe(
             columns_names=list(result_columns),
             data_rows=list(result),
-            apply_column_validated_uppercase=False,
             sort_and_reset_index=False,
         )