snowflake-data-validation 1.25.0__tar.gz → 1.26.0__tar.gz

{snowflake_data_validation-1.25.0 → snowflake_data_validation-1.26.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: snowflake-data-validation
-Version: 1.25.0
+Version: 1.26.0
 Summary: Snowflake Data Validation
 Project-URL: Bug Tracker, https://github.com/snowflake-eng/migrations-data-validation/issues
 Project-URL: Source code, https://github.com/snowflake-eng/migrations-data-validation/
@@ -89,9 +89,9 @@ Description-Content-Type: text/markdown
 [![License](https://img.shields.io/badge/License-Snowflake%20Conversion%20Software-blue)](https://www.snowflake.com/en/legal/technical-services-and-education/conversion-software-terms/)
 [![Python](https://img.shields.io/badge/python-3.10--3.13-blue)](https://www.python.org/downloads/)
 
-**Snowflake Data Validation** is a command-line tool and Python library for validating data migrations and ensuring data quality between source and target databases, with support for SQL Server, Redshift, Teradata, and Oracle as source systems.
+**Snowflake Data Validation** is a Python library for validating data migrations and ensuring data quality between source and target databases, with support for SQL Server, Redshift, Teradata, and Oracle as source systems.
 
-> 📖 **For detailed usage instructions, configuration examples, and CLI reference, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).**
+> 📖 **For detailed usage instructions and configuration examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).**
 
 ---
 
@@ -99,7 +99,6 @@ Description-Content-Type: text/markdown
 
 - **Multi-level validation**: Schema validation, statistical metrics, and row-level data integrity checks.
 - **Multiple source platforms**: SQL Server, Redshift, Teradata, Oracle.
-- **User-friendly CLI**: Comprehensive commands for automation and orchestration.
 - **Parallel processing**: Multi-threaded table validation for faster execution.
 - **Offline validation**: Extract source data as Parquet files for validation without source access.
 - **Flexible configuration**: YAML-based workflows with per-table customization.
@@ -139,27 +138,6 @@ pip install "snowflake-data-validation[all]"
 
 ---
 
-## 🔄 Execution Modes
-
-| Mode | Command | Description |
-|------|---------|-------------|
-| **Sync Validation** | `run-validation` | Real-time comparison between source and target databases |
-| **Source Extraction** | `source-validate` | Extract source data to Parquet files for offline validation |
-| **Async Validation** | `run-async-validation` | Validate using pre-extracted Parquet files |
-| **Script Generation** | `generate-validation-scripts` | Generate SQL scripts for manual execution |
-
-**Supported Dialects**: `sqlserver`, `snowflake`, `redshift`, `teradata`, `oracle`
-
-**CLI entry point**: from the repo (or an install), run the package module — for example:
-
-```bash
-python -m snowflake.snowflake_data_validation redshift run-validation -dvf path/to/config.yaml
-```
-
-Do not use `python -m snowflake.snowflake_data_validation.main_cli` for subcommands; that path exits after logging without dispatching to `redshift` / `sqlserver` / etc.
-
----
-
 ## 🔍 Validation Levels
 
 Validation runs **L1 schema**, **L2 metrics**, and optionally **L3 row** (and cell / hybrid modes where configured). Each level emits per-column (or per-criterion) rows with a `STATUS` of `SUCCESS` or `FAILURE`; the run is considered passing at that level only when **no failure rows** are present.
@@ -199,24 +177,32 @@ Performs row-by-row comparison:
 
 ---
 
-## CLI commands
-
-```bash
-snowflake-data-validation <source_dialect> <command> [options]
-# Short alias: sdv <source_dialect> <command> [options]
-```
+## 📚 Documentation
 
-Supported dialects include `sqlserver`, `teradata`, `redshift`, and `oracle`. Common commands: `run-validation`, `run-async-validation`, `source-validate`, and `generate-validation-scripts`. The dialect must match `source_platform` in your configuration file.
+For configuration options and examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
 
 ---
 
-## 📚 Documentation
+## Changelog
 
-For complete command reference, configuration options, and examples, see the [Data Validation CLI](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
+### v1.26.0
 
----
+**New features**
 
-## Changelog
+- Added charset-aware normalization to query generators.
+- Added Teradata non-Latin column UTF-8 hex hashing for server-side row validation.
+- Added key inference for Azure Synapse, BigQuery, and SQLite.
+- Added support for the `INTERVAL` data type.
+
+**Improvements**
+
+- Improved support for data validation of the `INTERVAL` data type.
+
+### v1.25.1
+
+**Improvements**
+
+- Removed the deprecated `sdv` CLI layer; Cloud Data Validation via the Data Exchange Agent is now the only supported execution path.
 
 ### v1.25.0
 

{snowflake_data_validation-1.25.0 → snowflake_data_validation-1.26.0}/README.md

@@ -3,9 +3,9 @@
 [![License](https://img.shields.io/badge/License-Snowflake%20Conversion%20Software-blue)](https://www.snowflake.com/en/legal/technical-services-and-education/conversion-software-terms/)
 [![Python](https://img.shields.io/badge/python-3.10--3.13-blue)](https://www.python.org/downloads/)
 
-**Snowflake Data Validation** is a command-line tool and Python library for validating data migrations and ensuring data quality between source and target databases, with support for SQL Server, Redshift, Teradata, and Oracle as source systems.
+**Snowflake Data Validation** is a Python library for validating data migrations and ensuring data quality between source and target databases, with support for SQL Server, Redshift, Teradata, and Oracle as source systems.
 
-> 📖 **For detailed usage instructions, configuration examples, and CLI reference, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).**
+> 📖 **For detailed usage instructions and configuration examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).**
 
 ---
 
@@ -13,7 +13,6 @@
 
 - **Multi-level validation**: Schema validation, statistical metrics, and row-level data integrity checks.
 - **Multiple source platforms**: SQL Server, Redshift, Teradata, Oracle.
-- **User-friendly CLI**: Comprehensive commands for automation and orchestration.
 - **Parallel processing**: Multi-threaded table validation for faster execution.
 - **Offline validation**: Extract source data as Parquet files for validation without source access.
 - **Flexible configuration**: YAML-based workflows with per-table customization.
@@ -53,27 +52,6 @@ pip install "snowflake-data-validation[all]"
 
 ---
 
-## 🔄 Execution Modes
-
-| Mode | Command | Description |
-|------|---------|-------------|
-| **Sync Validation** | `run-validation` | Real-time comparison between source and target databases |
-| **Source Extraction** | `source-validate` | Extract source data to Parquet files for offline validation |
-| **Async Validation** | `run-async-validation` | Validate using pre-extracted Parquet files |
-| **Script Generation** | `generate-validation-scripts` | Generate SQL scripts for manual execution |
-
-**Supported Dialects**: `sqlserver`, `snowflake`, `redshift`, `teradata`, `oracle`
-
-**CLI entry point**: from the repo (or an install), run the package module — for example:
-
-```bash
-python -m snowflake.snowflake_data_validation redshift run-validation -dvf path/to/config.yaml
-```
-
-Do not use `python -m snowflake.snowflake_data_validation.main_cli` for subcommands; that path exits after logging without dispatching to `redshift` / `sqlserver` / etc.
-
----
-
 ## 🔍 Validation Levels
 
 Validation runs **L1 schema**, **L2 metrics**, and optionally **L3 row** (and cell / hybrid modes where configured). Each level emits per-column (or per-criterion) rows with a `STATUS` of `SUCCESS` or `FAILURE`; the run is considered passing at that level only when **no failure rows** are present.
@@ -113,24 +91,32 @@ Performs row-by-row comparison:
 
 ---
 
-## CLI commands
-
-```bash
-snowflake-data-validation <source_dialect> <command> [options]
-# Short alias: sdv <source_dialect> <command> [options]
-```
+## 📚 Documentation
 
-Supported dialects include `sqlserver`, `teradata`, `redshift`, and `oracle`. Common commands: `run-validation`, `run-async-validation`, `source-validate`, and `generate-validation-scripts`. The dialect must match `source_platform` in your configuration file.
+For configuration options and examples, see the [official documentation](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
 
 ---
 
-## 📚 Documentation
+## Changelog
 
-For complete command reference, configuration options, and examples, see the [Data Validation CLI](https://docs.snowflake.com/en/migrations/snowconvert-docs/data-validation-cli/CLI_QUICK_REFERENCE).
+### v1.26.0
 
----
+**New features**
 
-## Changelog
+- Added charset-aware normalization to query generators.
+- Added Teradata non-Latin column UTF-8 hex hashing for server-side row validation.
+- Added key inference for Azure Synapse, BigQuery, and SQLite.
+- Added support for the `INTERVAL` data type.
+
+**Improvements**
+
+- Improved support for data validation of the `INTERVAL` data type.
+
+### v1.25.1
+
+**Improvements**
+
+- Removed the deprecated `sdv` CLI layer; Cloud Data Validation via the Data Exchange Agent is now the only supported execution path.
 
 ### v1.25.0
 

{snowflake_data_validation-1.25.0 → snowflake_data_validation-1.26.0}/pyproject.toml

@@ -60,10 +60,6 @@ name = "snowflake-data-validation"
 readme = "README.md"
 requires-python = '>=3.10'
 
-[project.scripts]
-sdv = "snowflake.snowflake_data_validation.main_cli:data_validation_app"
-snowflake-data-validation = "snowflake.snowflake_data_validation.main_cli:data_validation_app"
-
 [project.optional-dependencies]
 all = ["snowflake-data-validation[development,sqlserver,teradata,redshift,postgresql,azure-synapse,sqlite]"]
 development = [

{snowflake_data_validation-1.25.0 → snowflake_data_validation-1.26.0}/src/snowflake/snowflake_data_validation/__init__.py

@@ -25,7 +25,6 @@ Features
 --------
 - Multi-level data validation (Level 1: table metadata, Level 2: column metadata)
 - Support for multiple source systems (SQL Server, Snowflake)
-- CLI interface for validation operations
 - Configurable validation processes
 - Progress reporting and detailed validation reports
 
@@ -38,12 +37,6 @@ public
     Library API module providing decoupled validation steps (query generation,
     execution, validation) that can be called independently
 
-CLI Interface
-    Command-line interface available via `main_cli` module for:
-    - Setting up database connections
-    - Running validation operations
-    - Managing configuration files
-
 Supported Dialects
 ------------------
 - Snowflake to Snowflake validation
@@ -68,18 +61,6 @@ Programmatic API (Decoupled Steps):
 >>> result = public.execute_query(table_context, connector, schema_query)
 >>> validation_result = public.validate_schema(source_df, target_df)
 
-Command Line Interface:
-
-.. code-block:: bash
-
-    # Set up SQL Server connection
-    python -m snowflake.snowflake_data_validation sqlserver source-connection \
-        --host localhost --port 1433 --username user --password pass --database mydb
-
-    # Run validation
-    python -m snowflake.snowflake_data_validation sqlserver run-validation \
-        --data-validation-config-file config.json
-
 For more detailed examples and usage, please refer to the documentation.
 """
 

{snowflake_data_validation-1.25.0 → snowflake_data_validation-1.26.0}/src/snowflake/snowflake_data_validation/comparison_orchestrator.py

@@ -13,10 +13,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-import logging
-
-from collections.abc import Callable
-
 from snowflake.snowflake_data_validation.configuration.model.table_configuration import (
     TableConfiguration,
 )
@@ -26,9 +22,6 @@ from snowflake.snowflake_data_validation.configuration.model.validation_configur
 from snowflake.snowflake_data_validation.executer import (
     ExecutorFactory,
 )
-from snowflake.snowflake_data_validation.orchestration.console_progress_reporter import (
-    ConsoleProgressReporter,
-)
 from snowflake.snowflake_data_validation.orchestration.parallel_execution_engine import (
     ParallelExecutionEngine,
 )
@@ -38,6 +31,9 @@ from snowflake.snowflake_data_validation.orchestration.table_metadata_processor
 from snowflake.snowflake_data_validation.orchestration.validation_progress_reporter import (
     ValidationProgressReporter,
 )
+from snowflake.snowflake_data_validation.public.model import (
+    ValidationProgressCallback,
+)
 from snowflake.snowflake_data_validation.utils.arguments_manager_base import (
     ValidationEnvironmentObject,
 )
@@ -49,7 +45,6 @@ from snowflake.snowflake_data_validation.utils.constants import (
     ExecutionMode,
 )
 from snowflake.snowflake_data_validation.utils.context import Context
-from snowflake.snowflake_data_validation.utils.logging_utils import log
 from snowflake.snowflake_data_validation.utils.telemetry import (
     report_telemetry,
 )
@@ -58,21 +53,15 @@ from snowflake.snowflake_data_validation.validation.validation_database_manager
 )
 
 
-LOGGER = logging.getLogger(__name__)
-
-
 class ComparisonOrchestrator:
     """Orchestrator for validation operations that creates appropriate components based on command type."""
 
-    @log
     def __init__(
         self,
         connection_pool_manager: ConnectionPoolManager,
         context: Context,
         max_threads: int,
-        progress_callback: (
-            Callable[[str, list[str], Context | None], None] | None
-        ) = None,
+        progress_callback: ValidationProgressCallback | None = None,
     ):
         """Initialize the orchestrator.
 
@@ -86,73 +75,41 @@ class ComparisonOrchestrator:
                 Signature: (table_name: str, columns: list[str], context: Context | None) -> None
 
         """
-        LOGGER.debug("Initializing ComparisonOrchestrator")
-
         self.connection_pool_manager = connection_pool_manager
         self.context = context
         self.executor_factory = ExecutorFactory()
         self.max_threads = max_threads
 
-        # Initialize orchestration modules
         self.metadata_processor = TableMetadataProcessor()
         self.execution_engine = ParallelExecutionEngine(max_threads=self.max_threads)
         self.progress_reporter = ValidationProgressReporter(
             progress_callback=progress_callback,
         )
         self.db_manager = ValidationDatabaseManager()
-        self.console_progress_reporter = ConsoleProgressReporter(
-            output_handler=self.context.output_handler
-        )
-
-        LOGGER.debug(
-            "ComparisonOrchestrator initialized with max_threads=%d", self.max_threads
-        )
 
     @classmethod
-    @log(log_args=False)
     def from_validation_environment(
         cls,
         validation_env: ValidationEnvironmentObject,
-        progress_callback: (
-            Callable[[str, list[str], Context | None], None] | None
-        ) = None,
+        progress_callback: ValidationProgressCallback | None = None,
     ) -> "ComparisonOrchestrator":
         """Create a ComparisonOrchestrator from a ValidationEnvironmentObject.
 
         Args:
             validation_env: ValidationEnvironmentObject instance containing all required components
-            progress_callback: Optional callback for progress reporting. When
-                provided, it replaces the default stdout/pipe-based progress
-                reporting in ValidationProgressReporter.
-                Signature: (table_name: str, columns: list[str], context: Context | None) -> None
+            progress_callback: Optional callback for progress reporting.
 
         Returns:
             ComparisonOrchestrator: Configured orchestrator ready to run validation
 
         """
-        LOGGER.debug("Creating ComparisonOrchestrator from validation environment")
-
-        config = validation_env.context.configuration
-        num_tables: int = len(config.tables)
-        max_threads_config: str | int = config.max_threads
-
         # Hardcoded optimal pool size for now, can be adjusted later on based on changes in SnowConvert AI
         optimal_max_threads: int = TEMPORARY_DEFAULT_OPTIMAL_LIMIT
         # CpuOptimizer.get_optimal_thread_count(num_tables=num_tables, max_threads=max_threads_config)
         optimal_pool_size: int = TEMPORARY_DEFAULT_OPTIMAL_LIMIT
         # optimal_max_threads
 
-        LOGGER.debug(
-            "Using CPUOptimizer: num_tables=%d, config_max_threads=%s, optimal_max_threads=%d, optimal_pool_size=%d",
-            num_tables,
-            max_threads_config,
-            optimal_max_threads,
-            optimal_pool_size,
-        )
-
-        connection_pool_manager: ConnectionPoolManager = (
-            validation_env.create_connection_pool_manager(pool_size=optimal_pool_size)
-        )
+        connection_pool_manager: ConnectionPoolManager = validation_env.create_connection_pool_manager(pool_size=optimal_pool_size)
 
         return cls(
             connection_pool_manager=connection_pool_manager,
@@ -161,160 +118,54 @@ class ComparisonOrchestrator:
             progress_callback=progress_callback,
         )
 
-    @log
     @report_telemetry()
     def run_sync_comparison(self) -> None:
-        """Run the complete synchronous validation comparison process using parallel execution.
-
-        Uses the sync validation executor with metadata extractors for real-time validation.
-        Each table is processed in a separate thread.
-        """
-        LOGGER.info("Starting synchronous validation comparison")
-
+        """Run the complete synchronous validation comparison process using parallel execution."""
         with self.db_manager:
             self._execute_parallel_validation(ExecutionMode.SYNC_VALIDATION)
-            self.progress_reporter.flush_validation_reports(
-                self.context, self.db_manager
-            )
-            self.progress_reporter.log_row_validation_summary(self.context)
-
-            try:
-                self.progress_reporter.generate_execution_summary_report(
-                    context=self.context, db_manager=self.db_manager
-                )
-            except Exception as e:
-                LOGGER.warning("Failed to generate execution summary report: %s", e)
-
-            try:
-                results_summary = self.db_manager.get_results_summary_report_dataframe()
-
-                self.console_progress_reporter.generate_console_output_report(
-                    results_summary=results_summary
-                )
-                self.progress_reporter.generate_results_summary_report(
-                    context=self.context, results_summary=results_summary
-                )
-            except Exception as e:
-                LOGGER.warning("Error during report generation: %s", e)
+            self.progress_reporter.flush_validation_reports(self.context, self.db_manager)
+            self.progress_reporter.generate_execution_summary_report(context=self.context, db_manager=self.db_manager)
+            results_summary = self.db_manager.get_results_summary_report_dataframe()
+            self.progress_reporter.generate_results_summary_report(context=self.context, results_summary=results_summary)
 
-        LOGGER.info("Synchronous validation comparison completed")
-
-    @log
     @report_telemetry()
     def run_async_generation(self) -> None:
-        """Generate validation scripts for all tables in parallel.
-
-        Uses script printers to write SQL queries to files.
-        Each table is processed in a separate thread for maximum throughput.
-        """
-        LOGGER.info("Starting async script generation")
-
+        """Generate validation scripts for all tables in parallel."""
         with self.db_manager:
-            self._execute_parallel_validation(
-                ExecutionMode.ASYNC_GENERATION, use_script_printer=True
-            )
-            self.progress_reporter.flush_validation_reports(
-                self.context, self.db_manager
-            )
+            self._execute_parallel_validation(ExecutionMode.ASYNC_GENERATION, use_script_printer=True)
+            self.progress_reporter.flush_validation_reports(self.context, self.db_manager)
 
-        LOGGER.info("Async script generation completed")
-
-    @log
     @report_telemetry()
     def run_async_comparison(self) -> None:
-        """Run the asynchronous validation comparison process using parallel execution.
-
-        Uses the async validation executor with metadata extractors for deferred validation.
-        Each table is processed in a separate thread for maximum throughput.
-        """
-        LOGGER.info("Starting asynchronous validation comparison")
-
+        """Run the asynchronous validation comparison process using parallel execution."""
         with self.db_manager:
             self._execute_parallel_validation(ExecutionMode.ASYNC_VALIDATION)
-            self.progress_reporter.flush_validation_reports(
-                self.context, self.db_manager
-            )
-            self.progress_reporter.log_row_validation_summary(self.context)
-
-            try:
-                self.progress_reporter.generate_execution_summary_report(
-                    self.context, self.db_manager
-                )
-            except Exception as e:
-                LOGGER.warning("Failed to generate execution summary report: %s", e)
-
-        LOGGER.info("Asynchronous validation comparison completed")
+            self.progress_reporter.flush_validation_reports(self.context, self.db_manager)
+            self.progress_reporter.generate_execution_summary_report(self.context, self.db_manager)
 
-    @log
     @report_telemetry()
     def run_source_validation(self) -> None:
-        """Execute validation queries on source only and save as Parquet files.
-
-        Extracts schema and metrics data from source tables without performing
-        any validation or comparison. Results are saved as Parquet files for
-        later validation without needing source database access.
-        """
-        LOGGER.info("Starting source validation (data extraction)")
+        """Execute validation queries on source only and save as Parquet files."""
         self._execute_parallel_validation(ExecutionMode.SOURCE_VALIDATION)
-        LOGGER.info("Source validation completed - data saved as Parquet files")
 
-    @log(log_args=False)
     def _get_validation_configuration(
         self,
         table: TableConfiguration,
         default_configuration: ValidationConfiguration | None,
     ) -> ValidationConfiguration:
-        """Get the validation configuration for a table.
-
-        Args:
-            table: Table configuration object
-            default_configuration: Default validation configuration
-
-        Returns:
-            ValidationConfiguration: The configuration to use for this table
-
-        """
+        """Get the validation configuration for a table."""
         if table.validation_configuration:
-            LOGGER.debug(
-                "Using table-specific validation configuration for %s",
-                table.fully_qualified_name,
-            )
             return table.validation_configuration
         elif default_configuration:
-            LOGGER.debug(
-                "Using default validation configuration for %s",
-                table.fully_qualified_name,
-            )
             return default_configuration
         else:
-            LOGGER.debug(
-                "Using empty validation configuration for %s",
-                table.fully_qualified_name,
-            )
             return ValidationConfiguration()
 
-    @log
-    def _execute_parallel_validation(
-        self, execution_mode: ExecutionMode, use_script_printer: bool = False
-    ) -> None:
-        """Execute validation using parallel threads with connection pool.
-
-        Args:
-            execution_mode: The execution mode to use
-            use_script_printer: Whether to use script printers instead of metadata extractors
-
-        """
+    def _execute_parallel_validation(self, execution_mode: ExecutionMode, use_script_printer: bool = False) -> None:
+        """Execute validation using parallel threads with connection pool."""
         self.connection_pool_manager.initialize_pool()
 
-        LOGGER.debug(
-            "Executing parallel validation with execution_mode=%s, use_script_printer=%s",
-            execution_mode,
-            use_script_printer,
-        )
-
-        tables_to_validate = (
-            self.context.configuration.tables + self.context.configuration.views
-        )
+        tables_to_validate = self.context.configuration.tables + self.context.configuration.views
 
         self.execution_engine.execute_parallel_validation(
             tables=tables_to_validate,
@@ -329,9 +180,7 @@ class ComparisonOrchestrator:
                 column_selection_list=columns,
                 context=self.context,
             ),
-            validation_config_callback=lambda table, default_config: (
-                self._get_validation_configuration(
-                    table=table, default_configuration=default_config
-                )
+            validation_config_callback=lambda table, default_config: self._get_validation_configuration(
+                table=table, default_configuration=default_config
             ),
         )