google-pso-data-validator 5.1.0__tar.gz → 6.0.0__tar.gz

{google_pso_data_validator-5.1.0 → google_pso_data_validator-6.0.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: google-pso-data-validator
-Version: 5.1.0
+Version: 6.0.0
 Summary: A package to enable easy data validation
 Author: PSO DVT Engineering team
 Author-email: data-validator-eng@google.com
@@ -256,12 +256,14 @@ data-validation (--verbose or -v) (--log-level or -ll) validate row
                         Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
   [--trim-string-pks, -tsp]
                         Trims string based primary key values, intended for use when one engine uses padded string semantics (e.g. CHAR(n)) and the other does not (e.g. VARCHAR(n)).
+  [--case-insensitive-match, -cim]
+                        Performs a case insensitive match by adding an UPPER() before comparison.
 ```
 #### Generate Table Partitions for Large Table Row Validations
 
 When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
 
-The command generates and stores multiple YAML configs that represent chunks of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`). You can then run the configs in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
+The command generates and stores multiple YAML validations each representing a chunk of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in one YAML file. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
 
 The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions, ```partition-key``` is no longer supported.
 
@@ -289,13 +291,30 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         Directory Path to store YAML Config Files
                         GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                         Local: Provide a relative path of the target directory. Eg: `partitions_dir`
-  --partition-num [1-1000], -pn [1-1000]
-                        Number of partitions/config files to generate
+  --partition-num INT, -pn INT 
+                        Number of partitions into which the table should be split, e.g. 1000 or 10000
                         In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
+  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
+                        BigQuery destination for validation results. Defaults to stdout.
+                        See: *Validation Reports* section
+  [--service-account or -sa PATH_TO_SA_KEY]
+                        Service account to use for BigQuery result handler output.
+  [--parts-per-file INT], [-ppf INT] 
+                        Number of partitions in a yaml file, default value 1.
   [--filters SOURCE_FILTER:TARGET_FILTER]
                         Colon separated string values of source and target filters.
                         If target filter is not provided, the source filter will run on source and target tables.
                         See: *Filters* section
+  [--labels or -l KEY1=VALUE1,KEY2=VALUE2]
+                        Comma-separated key value pair labels for the run.
+  [--format or -fmt FORMAT]
+                        Format for stdout output. Supported formats are (text, csv, json, table). Defaults to table.
+  [--filter-status or -fs STATUSES_LIST]
+                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
+  [--trim-string-pks, -tsp]
+                        Trims string based primary key values, intended for use when one engine uses padded string semantics (e.g. CHAR(n)) and the other does not (e.g. VARCHAR(n)).
+  [--case-insensitive-match, -cim]
+                        Performs a case insensitive match by adding an UPPER() before comparison.
 ```
 #### Schema Validations
 
@@ -449,6 +468,10 @@ data-validation (--verbose or -v) (--log-level or -ll) validate custom-query row
                         Format for stdout output. Supported formats are (text, csv, json, table). Defaults to table.
   [--filter-status or -fs STATUSES_LIST]
                         Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
+  [--trim-string-pks, -tsp]
+                        Trims string based primary key values, intended for use when one engine uses padded string semantics (e.g. CHAR(n)) and the other does not (e.g. VARCHAR(n)).
+  [--case-insensitive-match, -cim]
+                        Performs a case insensitive match by adding an UPPER() before comparison.
 ```
 
 The [Examples](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/examples.md)

{google_pso_data_validator-5.1.0 → google_pso_data_validator-6.0.0}/README.md

@@ -210,12 +210,14 @@ data-validation (--verbose or -v) (--log-level or -ll) validate row
                         Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
   [--trim-string-pks, -tsp]
                         Trims string based primary key values, intended for use when one engine uses padded string semantics (e.g. CHAR(n)) and the other does not (e.g. VARCHAR(n)).
+  [--case-insensitive-match, -cim]
+                        Performs a case insensitive match by adding an UPPER() before comparison.
 ```
 #### Generate Table Partitions for Large Table Row Validations
 
 When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
 
-The command generates and stores multiple YAML configs that represent chunks of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`). You can then run the configs in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
+The command generates and stores multiple YAML validations each representing a chunk of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in one YAML file. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
 
 The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions, ```partition-key``` is no longer supported.
 
@@ -243,13 +245,30 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         Directory Path to store YAML Config Files
                         GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                         Local: Provide a relative path of the target directory. Eg: `partitions_dir`
-  --partition-num [1-1000], -pn [1-1000]
-                        Number of partitions/config files to generate
+  --partition-num INT, -pn INT 
+                        Number of partitions into which the table should be split, e.g. 1000 or 10000
                         In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
+  [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
+                        BigQuery destination for validation results. Defaults to stdout.
+                        See: *Validation Reports* section
+  [--service-account or -sa PATH_TO_SA_KEY]
+                        Service account to use for BigQuery result handler output.
+  [--parts-per-file INT], [-ppf INT] 
+                        Number of partitions in a yaml file, default value 1.
   [--filters SOURCE_FILTER:TARGET_FILTER]
                         Colon separated string values of source and target filters.
                         If target filter is not provided, the source filter will run on source and target tables.
                         See: *Filters* section
+  [--labels or -l KEY1=VALUE1,KEY2=VALUE2]
+                        Comma-separated key value pair labels for the run.
+  [--format or -fmt FORMAT]
+                        Format for stdout output. Supported formats are (text, csv, json, table). Defaults to table.
+  [--filter-status or -fs STATUSES_LIST]
+                        Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
+  [--trim-string-pks, -tsp]
+                        Trims string based primary key values, intended for use when one engine uses padded string semantics (e.g. CHAR(n)) and the other does not (e.g. VARCHAR(n)).
+  [--case-insensitive-match, -cim]
+                        Performs a case insensitive match by adding an UPPER() before comparison.
 ```
 #### Schema Validations
 
@@ -403,6 +422,10 @@ data-validation (--verbose or -v) (--log-level or -ll) validate custom-query row
                         Format for stdout output. Supported formats are (text, csv, json, table). Defaults to table.
   [--filter-status or -fs STATUSES_LIST]
                         Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
+  [--trim-string-pks, -tsp]
+                        Trims string based primary key values, intended for use when one engine uses padded string semantics (e.g. CHAR(n)) and the other does not (e.g. VARCHAR(n)).
+  [--case-insensitive-match, -cim]
+                        Performs a case insensitive match by adding an UPPER() before comparison.
 ```
 
 The [Examples](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/examples.md)

{google_pso_data_validator-5.1.0 → google_pso_data_validator-6.0.0}/data_validation/__main__.py

@@ -160,7 +160,7 @@ def get_aggregate_config(args, config_manager: ConfigManager):
     return aggregate_configs
 
 
-def get_calculated_config(args, config_manager):
+def get_calculated_config(args, config_manager) -> List[dict]:
     """Return list of formatted calculated objects.
 
     Args:
@@ -290,6 +290,17 @@ def build_config_from_args(args: Namespace, config_manager: ConfigManager):
             comparison_fields = cli_tools.get_arg_list(
                 args.comparison_fields, default_value=[]
             )
+
+            # As per #1190, add rstrip for Teradata string comparison fields
+            if (
+                config_manager.source_client.name == "teradata"
+                or config_manager.target_client.name == "teradata"
+            ):
+
+                comparison_fields = config_manager.add_rstrip_to_comp_fields(
+                    comparison_fields
+                )
+
             config_manager.append_comparison_fields(
                 config_manager.build_config_comparison_fields(comparison_fields)
             )
@@ -531,49 +542,49 @@ def run_validation(config_manager, dry_run=False, verbose=False):
         dry_run (bool): Print source and target SQL to stdout in lieu of validation.
         verbose (bool): Validation setting to log queries run.
     """
-    validator = DataValidation(
+    with DataValidation(
         config_manager.config,
         validation_builder=None,
         result_handler=None,
         verbose=verbose,
-    )
-
-    if dry_run:
-        sql_alchemy_clients = [
-            "mysql",
-            "oracle",
-            "postgres",
-            "db2",
-            "mssql",
-            "redshift",
-            "snowflake",
-        ]
-
-        source_query = validator.validation_builder.get_source_query().compile()
-        if config_manager.source_client.name in sql_alchemy_clients:
-            source_query = source_query.compile(
-                config_manager.source_client.con.engine,
-                compile_kwargs={"literal_binds": True},
-            )
+    ) as validator:
+
+        if dry_run:
+            sql_alchemy_clients = [
+                "mysql",
+                "oracle",
+                "postgres",
+                "db2",
+                "mssql",
+                "redshift",
+                "snowflake",
+            ]
+
+            source_query = validator.validation_builder.get_source_query().compile()
+            if config_manager.source_client.name in sql_alchemy_clients:
+                source_query = source_query.compile(
+                    config_manager.source_client.con.engine,
+                    compile_kwargs={"literal_binds": True},
+                )
 
-        target_query = validator.validation_builder.get_target_query().compile()
-        if config_manager.target_client.name in sql_alchemy_clients:
-            target_query = target_query.compile(
-                config_manager.target_client.con.engine,
-                compile_kwargs={"literal_binds": True},
-            )
+            target_query = validator.validation_builder.get_target_query().compile()
+            if config_manager.target_client.name in sql_alchemy_clients:
+                target_query = target_query.compile(
+                    config_manager.target_client.con.engine,
+                    compile_kwargs={"literal_binds": True},
+                )
 
-        print(
-            json.dumps(
-                {
-                    "source_query": str(source_query),
-                    "target_query": str(target_query),
-                },
-                indent=4,
+            print(
+                json.dumps(
+                    {
+                        "source_query": str(source_query),
+                        "target_query": str(target_query),
+                    },
+                    indent=4,
+                )
             )
-        )
-    else:
-        validator.execute()
+        else:
+            validator.execute()
 
 
 def run_validations(args, config_managers):
@@ -715,6 +726,7 @@ def main():
     elif args.command == "validate":
         validate(args)
     elif args.command == "generate-table-partitions":
+        cli_tools.check_no_yaml_files(args.partition_num, args.parts_per_file)
         partition_and_store_config_files(args)
     elif args.command == "deploy":
         from data_validation import app