snowflake-data-validation 1.46.2__tar.gz → 1.48.0__tar.gz

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.48.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: snowflake-data-validation
-Version: 1.46.2
+Version: 1.48.0
 Summary: Snowflake Data Validation
 Project-URL: Bug Tracker, https://github.com/snowflake-eng/migrations-data-validation/issues
 Project-URL: Source code, https://github.com/snowflake-eng/migrations-data-validation/
@@ -38,7 +38,7 @@ Requires-Dist: pydantic-yaml>=1.4.0
 Requires-Dist: pydantic>=2.0
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: ruamel-yaml>=0.18.0
-Requires-Dist: snowflake-connector-python<5,>=4.7.3
+Requires-Dist: snowflake-connector-python<6,>=5
 Requires-Dist: toml>=0.10.2
 Requires-Dist: typer>=0.15.3
 Requires-Dist: typing-extensions>=4.13.2
@@ -79,7 +79,7 @@ Requires-Dist: psycopg[binary]>=3.2.0; extra == 'postgresql'
 Provides-Extra: redshift
 Requires-Dist: redshift-connector>=2.1.14; extra == 'redshift'
 Provides-Extra: scai
-Requires-Dist: snowflake-code-unit-registry>=0.9.10; extra == 'scai'
+Requires-Dist: snowflake-code-unit-registry>=0.9.19; extra == 'scai'
 Provides-Extra: sqlite
 Requires-Dist: sqlean-py>=3.45.0; extra == 'sqlite'
 Provides-Extra: sqlserver
@@ -164,7 +164,7 @@ An integer column mapped to a `NUMBER` with scale 0 is the exception to the prec
 
 A PostgreSQL user-defined `ENUM` column is validated as `ENUM` against a Snowflake `VARCHAR`. `information_schema` reports every enum as `USER-DEFINED`, so the metadata queries read `pg_type.typtype` to tell enums apart, and report the longest label as the column's length. L2 and L3 compare the label text.
 
-Snowflake database, schema and table names may contain a backslash or an apostrophe, for example `NAFMAIN\TUSDB102`. Snowflake reads backslash escapes inside a string literal, so the Snowflake metadata queries (table metadata, column metadata, case-sensitive columns and object-type lookups) write each name through `escape_snowflake_sql_literal` / `quote_snowflake_sql_literal` in `shared/queries/sql_literals.py`, which double backslashes as well as single quotes. Use these helpers, not `escape_sql_literal`, for any name embedded in a Snowflake string literal.
+Snowflake database, schema and table names may contain a backslash or an apostrophe, for example `NAFMAIN\TUSDB102`. Snowflake reads backslash escapes inside a string literal, so the Snowflake metadata queries (table metadata, column metadata, case-sensitive columns, object-type lookups, and the column list and table list that column partitioning and the configuration generator read through `utils/catalog_literals.py`) write each name through `escape_snowflake_sql_literal` / `quote_snowflake_sql_literal` in `shared/queries/sql_literals.py`, which double backslashes as well as single quotes. Use these helpers, not `escape_sql_literal`, for any name embedded in a Snowflake string literal.
 
 ### Metrics Validation
 Compares statistical metrics for each column:
@@ -182,6 +182,14 @@ L1 and L2 read each column's catalog spelling rather than an upper-cased name, s
 
 ### Views
 
+SQL Server and Azure Synapse source scans delimit each configured object-name part
+through the shared identifier manager. Reserved words, dots inside quoted parts,
+escaped closing brackets, and omitted schema segments remain valid in generated
+SQL, including whitespace between a separator and a quoted part.
+The platform-aware table-context configuration boundary parses T-SQL references
+into actual name components once; repeated queries write those components without
+reinterpreting quote characters or reparsing the original configuration.
+
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
 
 ### Row Validation
@@ -190,6 +198,26 @@ Performs row-by-row comparison:
 - Field-level value comparison
 - Mismatch reporting
 
+Row-MD5 results preserve the exact key aliases projected from actual catalog names; joins and report predicates do not uppercase or case-fold those aliases. Distinct keys such as `MixedCase` and `MIXEDCASE` keep their own values. Join alignment accepts an exact `_IDX` projection alias when the key's direct alias is absent; reports read the resulting canonical aliases directly. Each report side takes its identifier and literal rules from the same query generator. Only the generated hash alias is recognized by its dialect spelling (`row_md5` on PostgreSQL/Redshift). An index named `ROW_MD5` conflicts with that reserved result column and is rejected rather than used as a hash. Empty results retain cursor headers and complete missing join/hash columns; populated results fail explicitly for a missing key alias (including a known exclusion) or a missing row-MD5 column rather than guessing another key by casing.
+
+For each mismatched row, the report holds a source and a target `SELECT` that find that row, and the run also saves a diff query per side. Each side's queries write names by that side's identifier rules: every part of the object name is read as that platform reads a configured name (an unquoted part folds to its storage case, a quoted part is kept) and written delimited, and every key column is written delimited too. So a SQL Server or Azure Synapse `Load.dbo.File` is queried as `[Load].[dbo].[File]`, the Snowflake target `LOAD.DBO.FILE` as `"LOAD"."DBO"."FILE"`, and a BigQuery key column as `` `Order Id` ``; a reserved word such as `File` stays valid. The report's table name and the saved query file names keep the configured name (`Load.dbo.File`). A string key is escaped by the query generator of the platform that runs that side (`escape_string_literal`). Most dialects use standard SQL and double an apostrophe. Snowflake also doubles backslashes, because it reads a backslash in a string literal as an escape, and BigQuery backslash-escapes both backslashes and apostrophes. A platform whose literals differ overrides `escape_string_literal` on its query generator.
+
+Redshift also doubles backslashes and apostrophes. PostgreSQL's standard-SQL escaping assumes `standard_conforming_strings=on` (the server default since PostgreSQL 9.1); the connector does not set this session option. The unit tests check generated literal text for all ten engine dialects, not live database parsing.
+
+Qualified source references retain parsed actual components until the identifier
+manager writes SQL. At the table-context boundary, an explicitly supplied actual
+schema fills an omitted schema position; it is not reparsed or replaced with
+`dbo`. Without a resolved schema, valid omitted-position syntax such as
+`tempdb..#snapshot` remains omitted. Separate internal temporary references do
+not inherit the source table's schema. Catalog predicates likewise escape actual
+names as string values rather than stripping literal brackets, quotes or padding.
+
+Row-query result labels retain their exact aliases. Catalog-metadata boundaries
+normalize only field labels (such as `DATA_TYPE` and `ROW_COUNT`) for metadata
+consumers; actual column-name values remain unchanged, including mixed case.
+
+A key present once on each side is a `MISMATCH` when its row MD5s differ. A NULL row MD5 on one side against a digest on the other is a difference too; NULL on both sides is a match. This is the `IS DISTINCT FROM` rule the Snowflake pushdown compare uses (`docs/common/l3-signature-pushdown-compare.md`).
+
 ---
 
 ## 📊 Reports
@@ -222,14 +250,74 @@ Raising `requires-python` here above the pinned version fails the audit until th
 
 ---
 
+## 🧩 Custom result processors
+
+`execute_query` and `execute_query_from_table_configuration` take an optional
+`result_processor(raw_result, apply_column_validated_uppercase)` that turns the
+connector's raw result into a DataFrame. A processor you pass is called for every
+result, an empty one included, with the `apply_column_validated_uppercase` flag,
+and the DataFrame it returns is the result, so the columns it gives an empty
+frame are kept. Without one, the platform's default processor is used and an
+empty result becomes an empty DataFrame with no columns.
+
+---
+
 ## 📚 Documentation
 
 For configuration options and examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
 
+Async validation reads extracted schema and metrics files laid out as described in [Async validation files](../docs/snowflake-data-validation/async-validation-files.md).
+
 ---
 
 ## Changelog
 
+### v1.48.0
+
+**New features**
+
+- Added a row checksum gate that validates a large matching table with one aggregate query per side before extracting row signatures.
+- Added support for Snowflake `IPADDRESS` as an opt-in target for PostgreSQL `INET`.
+- Added L1, L2, and L3 validation parity for Oracle Iceberg tables.
+- Added validation of `INTERVAL` and `NUMBER(*)` target encodings.
+- Added a validation config setting that names one `HASH_MD5` UDF for every Teradata source database.
+- Added the adoption row-hash check for partially migrated Snowflake targets.
+
+**Improvements**
+
+- Improved L2 metrics by dropping the `FLOAT` sum, adding exact sign counts, and aligning the text and binary metric sets.
+- Improved Oracle `SDO_GEOMETRY` validation by comparing the whole WKT by MD5 and keeping the SRID as EWKT on `GEOMETRY` targets.
+
+**Bug fixes**
+
+- Fixed Snowflake catalog and `SHOW` lookups not doubling backslashes in name literals.
+- Fixed Azure Synapse `VARCHAR` time values not being compared as the text CETAS stored.
+- Fixed `etl-validate` row validation by pairing each target column with its source type.
+- Fixed Oracle dates and timestamps before year 1 losing their sign.
+- Fixed Oracle integer spatial coordinates not being compared as Snowflake writes them.
+- Fixed Teradata L3 false mismatches on non-ASCII values in `LATIN` columns.
+
+### v1.47.0
+
+**New features**
+
+- Added L1, L2, and L3 validation of BigQuery Iceberg tables.
+- Added L2 and L3 comparison of Teradata `BLOB` and `CLOB` columns.
+
+**Improvements**
+
+- Improved Teradata L2 validation to compute the standard deviation of integer columns exactly.
+- Updated the Snowflake connector to `snowflake-connector-python` 5.x.
+
+**Bug fixes**
+
+- Fixed row validation losing join keys when their casing differed or the key column was excluded.
+- Fixed row validation of tables with duplicate keys by matching duplicate-key rows as multisets, and quoted SQL Server identifiers.
+- Fixed custom processing being dropped when a query returned no rows.
+- Fixed the saved row-diff SQL ending in an empty `WHERE` when every mismatched row was missing on one side.
+- Fixed the row drill-down treating a `NULL` `ROW_MD5` on one side as equal to a digest on the other.
+- Fixed Oracle `TIMESTAMP WITH LOCAL TIME ZONE` and `XMLTYPE` comparison by reading them as UTC text and compact `XMLSERIALIZE` text.
+
 ### v1.46.2
 
 **Improvements**

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.48.0}/README.md

@@ -73,7 +73,7 @@ An integer column mapped to a `NUMBER` with scale 0 is the exception to the prec
 
 A PostgreSQL user-defined `ENUM` column is validated as `ENUM` against a Snowflake `VARCHAR`. `information_schema` reports every enum as `USER-DEFINED`, so the metadata queries read `pg_type.typtype` to tell enums apart, and report the longest label as the column's length. L2 and L3 compare the label text.
 
-Snowflake database, schema and table names may contain a backslash or an apostrophe, for example `NAFMAIN\TUSDB102`. Snowflake reads backslash escapes inside a string literal, so the Snowflake metadata queries (table metadata, column metadata, case-sensitive columns and object-type lookups) write each name through `escape_snowflake_sql_literal` / `quote_snowflake_sql_literal` in `shared/queries/sql_literals.py`, which double backslashes as well as single quotes. Use these helpers, not `escape_sql_literal`, for any name embedded in a Snowflake string literal.
+Snowflake database, schema and table names may contain a backslash or an apostrophe, for example `NAFMAIN\TUSDB102`. Snowflake reads backslash escapes inside a string literal, so the Snowflake metadata queries (table metadata, column metadata, case-sensitive columns, object-type lookups, and the column list and table list that column partitioning and the configuration generator read through `utils/catalog_literals.py`) write each name through `escape_snowflake_sql_literal` / `quote_snowflake_sql_literal` in `shared/queries/sql_literals.py`, which double backslashes as well as single quotes. Use these helpers, not `escape_sql_literal`, for any name embedded in a Snowflake string literal.
 
 ### Metrics Validation
 Compares statistical metrics for each column:
@@ -91,6 +91,14 @@ L1 and L2 read each column's catalog spelling rather than an upper-cased name, s
 
 ### Views
 
+SQL Server and Azure Synapse source scans delimit each configured object-name part
+through the shared identifier manager. Reserved words, dots inside quoted parts,
+escaped closing brackets, and omitted schema segments remain valid in generated
+SQL, including whitespace between a separator and a quoted part.
+The platform-aware table-context configuration boundary parses T-SQL references
+into actual name components once; repeated queries write those components without
+reinterpreting quote characters or reparsing the original configuration.
+
 For **Redshift** and **Teradata** sources, views are validated on the view itself (no `CREATE TABLE AS SELECT` materialization in local sync mode). Other platforms may still materialize views to temporary tables before L1/L2/L3. **Teradata** views are read via `DBC.ColumnsQV` (Teradata's Query View Column Information catalog), so view L1 covers the full field set — column existence, datatype, precision, scale, length, nullability, and ordinal position — matching the table path. When `DBC.ColumnsQV` is unavailable (QVCI disabled → Teradata error 9719, missing `SELECT` grant, any other failure of that catalog query, or a degenerate result with null `DATA_TYPE`), the extractor falls back to `HELP COLUMN` with `basic_validation=True` and emits one WARNING schema row per view naming the trigger and the remediation. If HELP fails or source schema metadata still has no usable `DATA_TYPE` (including a view validated as a table whose `ColumnsV` types are null), L1 fails that object and does not continue to L2/L3. **Redshift** uses full schema metadata like tables.
 
 ### Row Validation
@@ -99,6 +107,26 @@ Performs row-by-row comparison:
 - Field-level value comparison
 - Mismatch reporting
 
+Row-MD5 results preserve the exact key aliases projected from actual catalog names; joins and report predicates do not uppercase or case-fold those aliases. Distinct keys such as `MixedCase` and `MIXEDCASE` keep their own values. Join alignment accepts an exact `_IDX` projection alias when the key's direct alias is absent; reports read the resulting canonical aliases directly. Each report side takes its identifier and literal rules from the same query generator. Only the generated hash alias is recognized by its dialect spelling (`row_md5` on PostgreSQL/Redshift). An index named `ROW_MD5` conflicts with that reserved result column and is rejected rather than used as a hash. Empty results retain cursor headers and complete missing join/hash columns; populated results fail explicitly for a missing key alias (including a known exclusion) or a missing row-MD5 column rather than guessing another key by casing.
+
+For each mismatched row, the report holds a source and a target `SELECT` that find that row, and the run also saves a diff query per side. Each side's queries write names by that side's identifier rules: every part of the object name is read as that platform reads a configured name (an unquoted part folds to its storage case, a quoted part is kept) and written delimited, and every key column is written delimited too. So a SQL Server or Azure Synapse `Load.dbo.File` is queried as `[Load].[dbo].[File]`, the Snowflake target `LOAD.DBO.FILE` as `"LOAD"."DBO"."FILE"`, and a BigQuery key column as `` `Order Id` ``; a reserved word such as `File` stays valid. The report's table name and the saved query file names keep the configured name (`Load.dbo.File`). A string key is escaped by the query generator of the platform that runs that side (`escape_string_literal`). Most dialects use standard SQL and double an apostrophe. Snowflake also doubles backslashes, because it reads a backslash in a string literal as an escape, and BigQuery backslash-escapes both backslashes and apostrophes. A platform whose literals differ overrides `escape_string_literal` on its query generator.
+
+Redshift also doubles backslashes and apostrophes. PostgreSQL's standard-SQL escaping assumes `standard_conforming_strings=on` (the server default since PostgreSQL 9.1); the connector does not set this session option. The unit tests check generated literal text for all ten engine dialects, not live database parsing.
+
+Qualified source references retain parsed actual components until the identifier
+manager writes SQL. At the table-context boundary, an explicitly supplied actual
+schema fills an omitted schema position; it is not reparsed or replaced with
+`dbo`. Without a resolved schema, valid omitted-position syntax such as
+`tempdb..#snapshot` remains omitted. Separate internal temporary references do
+not inherit the source table's schema. Catalog predicates likewise escape actual
+names as string values rather than stripping literal brackets, quotes or padding.
+
+Row-query result labels retain their exact aliases. Catalog-metadata boundaries
+normalize only field labels (such as `DATA_TYPE` and `ROW_COUNT`) for metadata
+consumers; actual column-name values remain unchanged, including mixed case.
+
+A key present once on each side is a `MISMATCH` when its row MD5s differ. A NULL row MD5 on one side against a digest on the other is a difference too; NULL on both sides is a match. This is the `IS DISTINCT FROM` rule the Snowflake pushdown compare uses (`docs/common/l3-signature-pushdown-compare.md`).
+
 ---
 
 ## 📊 Reports
@@ -131,14 +159,74 @@ Raising `requires-python` here above the pinned version fails the audit until th
 
 ---
 
+## 🧩 Custom result processors
+
+`execute_query` and `execute_query_from_table_configuration` take an optional
+`result_processor(raw_result, apply_column_validated_uppercase)` that turns the
+connector's raw result into a DataFrame. A processor you pass is called for every
+result, an empty one included, with the `apply_column_validated_uppercase` flag,
+and the DataFrame it returns is the result, so the columns it gives an empty
+frame are kept. Without one, the platform's default processor is used and an
+empty result becomes an empty DataFrame with no columns.
+
+---
+
 ## 📚 Documentation
 
 For configuration options and examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
 
+Async validation reads extracted schema and metrics files laid out as described in [Async validation files](../docs/snowflake-data-validation/async-validation-files.md).
+
 ---
 
 ## Changelog
 
+### v1.48.0
+
+**New features**
+
+- Added a row checksum gate that validates a large matching table with one aggregate query per side before extracting row signatures.
+- Added support for Snowflake `IPADDRESS` as an opt-in target for PostgreSQL `INET`.
+- Added L1, L2, and L3 validation parity for Oracle Iceberg tables.
+- Added validation of `INTERVAL` and `NUMBER(*)` target encodings.
+- Added a validation config setting that names one `HASH_MD5` UDF for every Teradata source database.
+- Added the adoption row-hash check for partially migrated Snowflake targets.
+
+**Improvements**
+
+- Improved L2 metrics by dropping the `FLOAT` sum, adding exact sign counts, and aligning the text and binary metric sets.
+- Improved Oracle `SDO_GEOMETRY` validation by comparing the whole WKT by MD5 and keeping the SRID as EWKT on `GEOMETRY` targets.
+
+**Bug fixes**
+
+- Fixed Snowflake catalog and `SHOW` lookups not doubling backslashes in name literals.
+- Fixed Azure Synapse `VARCHAR` time values not being compared as the text CETAS stored.
+- Fixed `etl-validate` row validation by pairing each target column with its source type.
+- Fixed Oracle dates and timestamps before year 1 losing their sign.
+- Fixed Oracle integer spatial coordinates not being compared as Snowflake writes them.
+- Fixed Teradata L3 false mismatches on non-ASCII values in `LATIN` columns.
+
+### v1.47.0
+
+**New features**
+
+- Added L1, L2, and L3 validation of BigQuery Iceberg tables.
+- Added L2 and L3 comparison of Teradata `BLOB` and `CLOB` columns.
+
+**Improvements**
+
+- Improved Teradata L2 validation to compute the standard deviation of integer columns exactly.
+- Updated the Snowflake connector to `snowflake-connector-python` 5.x.
+
+**Bug fixes**
+
+- Fixed row validation losing join keys when their casing differed or the key column was excluded.
+- Fixed row validation of tables with duplicate keys by matching duplicate-key rows as multisets, and quoted SQL Server identifiers.
+- Fixed custom processing being dropped when a query returned no rows.
+- Fixed the saved row-diff SQL ending in an empty `WHERE` when every mismatched row was missing on one side.
+- Fixed the row drill-down treating a `NULL` `ROW_MD5` on one side as equal to a digest on the other.
+- Fixed Oracle `TIMESTAMP WITH LOCAL TIME ZONE` and `XMLTYPE` comparison by reading them as UTC text and compact `XMLSERIALIZE` text.
+
 ### v1.46.2
 
 **Improvements**

{snowflake_data_validation-1.46.2 → snowflake_data_validation-1.48.0}/pyproject.toml

@@ -36,7 +36,7 @@ dependencies = [
   "connectorx>=0.3.0",
   "cryptography>=50.0.0",
   "deepdiff>=8.0.0",
-  "snowflake-connector-python>=4.7.3,<5",
+  "snowflake-connector-python>=5,<6",
   "pandas>=2.2.3",
   "polars>=1.0.0",
   "pyarrow>=14.0.0",
@@ -106,7 +106,7 @@ db2 = [
 # musllinux build, and SDV is published to public PyPI. Inside DMO/DEA the
 # dependency is already declared, so SDV's logger tree is covered there.
 scai = [
-  "snowflake-code-unit-registry>=0.9.10",
+  "snowflake-code-unit-registry>=0.9.19",
 ]
 
 [project.urls]
@@ -192,6 +192,14 @@ exclude_lines = [
   "if __name__ == .__main__.:",
 ]
 
+# Ruff only reads the sources, so the linter needs neither the package nor its drivers.
+[tool.hatch.envs.linter]
+detached = true
+installer = "uv"
+dependencies = [
+  "ruff>=0.15",
+]
+
 [tool.hatch.envs.linter.scripts]
 check = [
   "echo 'Running linting checks...' && ruff check --config=../ruff.toml --statistics --verbose . || (echo '❌ LINTING FAILED: Please fix the above linting issues before proceeding. Use \"ruff check --config=../ruff.toml --fix .\" to auto-fix some issues, or fix them manually.' && exit 1)",