google-pso-data-validator 6.2.0__tar.gz → 6.3.0__tar.gz

{google_pso_data_validator-6.2.0 → google_pso_data_validator-6.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: google-pso-data-validator
-Version: 6.2.0
+Version: 6.3.0
 Summary: A package to enable easy data validation
 Author: PSO DVT Engineering team
 Author-email: data-validator-eng@google.com
@@ -267,13 +267,13 @@ data-validation
   [--case-insensitive-match, -cim]
                         Performs a case insensitive match by adding an UPPER() before comparison.
 ```
-#### Generate Table Partitions for Large Table Row Validations
+#### Generate Partitions for Large Row Validations
 
-When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
+When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating partitions in order to perform row validations on large dataset (table or custom-query) to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
 
-The command generates and stores multiple YAML validations each representing a chunk of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in one YAML file. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
+The command generates and stores multiple YAML validations each representing a chunk of the large dataset using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in YAML files. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
 
-The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions, ```partition-key``` is no longer supported.
+The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. You can specify tables that are being validated or the source and target custom query. A parameter used in earlier versions, ```partition-key``` is no longer supported.
 
 ```
 data-validation
@@ -292,6 +292,17 @@ data-validation
                         Comma separated list of tables in the form schema.table=target_schema.target_table
                         Target schema name and table name are optional.
                         i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query SOURCE_QUERY, -sq SOURCE_QUERY
+                        Source sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query-file  SOURCE_QUERY_FILE, -sqf SOURCE_QUERY_FILE
+                        File containing the source sql command. Supports GCS and local paths.
+  --target-query TARGET_QUERY, -tq TARGET_QUERY
+                        Target sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --target-query-file TARGET_QUERY_FILE, -tqf TARGET_QUERY_FILE
+                        File containing the target sql command. Supports GCS and local paths.
   --primary-keys PRIMARY_KEYS, -pk PRIMARY_KEYS
                         Comma separated list of primary key columns 'col_a,col_b'.  See *Primary Keys* section
   --comparison-fields or -comp-fields FIELDS
@@ -303,6 +314,7 @@ data-validation
                         Directory Path to store YAML Config Files
                         GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                         Local: Provide a relative path of the target directory. Eg: `partitions_dir`
+                        If invoked with -tbls parameter, the validations are stored in a directory named <schema>.<table>, otherwise the directory is named `custom.<random_string>`
   --partition-num INT, -pn INT
                         Number of partitions into which the table should be split, e.g. 1000 or 10000
                         In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
@@ -581,10 +593,10 @@ View the complete YAML file for a Grouped Column validation on the
 
 ### Scaling DVT
 
-You can scale DVT for large table validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
+You can scale DVT for large validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
 
 
-We recommend first generating table partitions with the `generate-table-partitions` command for your large tables. Then, use Cloud Run or GKE to distribute validating each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
+We recommend first generating partitions with the `generate-table-partitions` command for your large datasets (tables or queries). Then, use Cloud Run or GKE to distribute the validation of each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
 
 When running DVT in a distributed fashion, both the `--kube-completions` and `--config-dir` flags are required. The `--kube-completions` flag specifies that the validation is being run in indexed completion mode in Kubernetes or as multiple independent tasks in Cloud Run. If the `-kc` option is used and you are not running in indexed mode, you will receive a warning and the container will process all the validations sequentially. If the `-kc` option is used and a config directory is not provided (i.e. a `--config-file` is provided instead), a warning is issued.
 
@@ -719,10 +731,15 @@ resultset that breaks down the count of rows per calendar date.
 Row level validations can involve either a hash/checksum, concat, or comparison fields.
 A hash validation (`--hash '*'`) will first sanitize the data with the following
 operations on all or selected columns: CAST to string, IFNULL replace with a default
-replacement string, RSTRIP, and UPPER. Then, it will CONCAT() the results
-and run a SHA256() hash and compare the source and target results. Since each row will
-be returned in the result set, it is recommended to utilize the `--use-random-row` feature
-to validate a subset of the table.
+replacement string and RSTRIP. Then, it will CONCAT() the results
+and run a SHA256() hash and compare the source and target results.
+
+When there are data type mismatches for columns, for example dates compared to timestamps and
+booleans compared with numeric columns, you may see other expressions in SQL statements which
+ensure that consistent values are used to build comparison values.
+
+Since each row will be returned in the result set if is recommended recommended to validate a
+subset of the table. The `--filters` and `--use-random-row` options can be used for this purpose.
 
 Please note that SHA256 is not a supported function on Teradata systems. If you wish to perform
 this comparison on Teradata you will need to [deploy a UDF to perform the conversion](https://github.com/akuroda/teradata-udf-sha2/blob/master/src/sha256.c).

{google_pso_data_validator-6.2.0 → google_pso_data_validator-6.3.0}/README.md

@@ -226,13 +226,13 @@ data-validation
   [--case-insensitive-match, -cim]
                         Performs a case insensitive match by adding an UPPER() before comparison.
 ```
-#### Generate Table Partitions for Large Table Row Validations
+#### Generate Partitions for Large Row Validations
 
-When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
+When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating partitions in order to perform row validations on large dataset (table or custom-query) to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
 
-The command generates and stores multiple YAML validations each representing a chunk of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in one YAML file. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
+The command generates and stores multiple YAML validations each representing a chunk of the large dataset using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in YAML files. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
 
-The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions, ```partition-key``` is no longer supported.
+The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. You can specify tables that are being validated or the source and target custom query. A parameter used in earlier versions, ```partition-key``` is no longer supported.
 
 ```
 data-validation
@@ -251,6 +251,17 @@ data-validation
                         Comma separated list of tables in the form schema.table=target_schema.target_table
                         Target schema name and table name are optional.
                         i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query SOURCE_QUERY, -sq SOURCE_QUERY
+                        Source sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query-file  SOURCE_QUERY_FILE, -sqf SOURCE_QUERY_FILE
+                        File containing the source sql command. Supports GCS and local paths.
+  --target-query TARGET_QUERY, -tq TARGET_QUERY
+                        Target sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --target-query-file TARGET_QUERY_FILE, -tqf TARGET_QUERY_FILE
+                        File containing the target sql command. Supports GCS and local paths.
   --primary-keys PRIMARY_KEYS, -pk PRIMARY_KEYS
                         Comma separated list of primary key columns 'col_a,col_b'.  See *Primary Keys* section
   --comparison-fields or -comp-fields FIELDS
@@ -262,6 +273,7 @@ data-validation
                         Directory Path to store YAML Config Files
                         GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                         Local: Provide a relative path of the target directory. Eg: `partitions_dir`
+                        If invoked with -tbls parameter, the validations are stored in a directory named <schema>.<table>, otherwise the directory is named `custom.<random_string>`
   --partition-num INT, -pn INT
                         Number of partitions into which the table should be split, e.g. 1000 or 10000
                         In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
@@ -540,10 +552,10 @@ View the complete YAML file for a Grouped Column validation on the
 
 ### Scaling DVT
 
-You can scale DVT for large table validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
+You can scale DVT for large validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
 
 
-We recommend first generating table partitions with the `generate-table-partitions` command for your large tables. Then, use Cloud Run or GKE to distribute validating each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
+We recommend first generating partitions with the `generate-table-partitions` command for your large datasets (tables or queries). Then, use Cloud Run or GKE to distribute the validation of each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
 
 When running DVT in a distributed fashion, both the `--kube-completions` and `--config-dir` flags are required. The `--kube-completions` flag specifies that the validation is being run in indexed completion mode in Kubernetes or as multiple independent tasks in Cloud Run. If the `-kc` option is used and you are not running in indexed mode, you will receive a warning and the container will process all the validations sequentially. If the `-kc` option is used and a config directory is not provided (i.e. a `--config-file` is provided instead), a warning is issued.
 
@@ -678,10 +690,15 @@ resultset that breaks down the count of rows per calendar date.
 Row level validations can involve either a hash/checksum, concat, or comparison fields.
 A hash validation (`--hash '*'`) will first sanitize the data with the following
 operations on all or selected columns: CAST to string, IFNULL replace with a default
-replacement string, RSTRIP, and UPPER. Then, it will CONCAT() the results
-and run a SHA256() hash and compare the source and target results. Since each row will
-be returned in the result set, it is recommended to utilize the `--use-random-row` feature
-to validate a subset of the table.
+replacement string and RSTRIP. Then, it will CONCAT() the results
+and run a SHA256() hash and compare the source and target results.
+
+When there are data type mismatches for columns, for example dates compared to timestamps and
+booleans compared with numeric columns, you may see other expressions in SQL statements which
+ensure that consistent values are used to build comparison values.
+
+Since each row will be returned in the result set if is recommended recommended to validate a
+subset of the table. The `--filters` and `--use-random-row` options can be used for this purpose.
 
 Please note that SHA256 is not a supported function on Teradata systems. If you wish to perform
 this comparison on Teradata you will need to [deploy a UDF to perform the conversion](https://github.com/akuroda/teradata-udf-sha2/blob/master/src/sha256.c).

{google_pso_data_validator-6.2.0 → google_pso_data_validator-6.3.0}/data_validation/__main__.py

@@ -160,7 +160,7 @@ def get_aggregate_config(args, config_manager: ConfigManager):
     return aggregate_configs
 
 
-def get_calculated_config(args, config_manager: ConfigManager) -> List[dict]:
+def _get_calculated_config(args, config_manager: ConfigManager) -> List[dict]:
     """Return list of formatted calculated objects.
 
     Args:
@@ -217,8 +217,41 @@ def get_calculated_config(args, config_manager: ConfigManager) -> List[dict]:
     return calculated_configs
 
 
+def _get_comparison_config(args, config_manager: ConfigManager) -> List[dict]:
+    col_list = (
+        None
+        if config_manager.comparison_fields == "*"
+        else cli_tools.get_arg_list(config_manager.comparison_fields)
+    )
+    comparison_fields = config_manager.build_comp_fields(
+        col_list,
+        args.exclude_columns,
+    )
+    # We can't have the PK columns in the comparison SQL twice therefore filter them out here if included.
+    comparison_fields = [
+        _
+        for _ in comparison_fields
+        if _ not in cli_tools.get_arg_list(args.primary_keys.casefold())
+    ]
+
+    # As per #1190, add rstrip for Teradata string comparison fields
+    if (
+        config_manager.source_client.name == "teradata"
+        or config_manager.target_client.name == "teradata"
+    ):
+        comparison_fields = config_manager.add_rstrip_to_comp_fields(comparison_fields)
+
+    return config_manager.build_config_comparison_fields(comparison_fields)
+
+
 def build_config_from_args(args: Namespace, config_manager: ConfigManager):
-    """Append build configs to ConfigManager object.
+    """This function is used to append build configs to the config manager for all validation commands and generate-table-partitions.
+    Instead of having two separate commands, e.g. validate row and validate custom-query row, generate-table-partitions
+    uses implicit choice of table or custom-query. A user can specify either tables or source/target query/file,
+    but not both. In the case of generate-table-partitions with custom query, the user will not provide
+    args.custom_query_type. However, the code will inject args.custom_query_type as 'row' before invoking
+    build_config_from_args.
+
 
     Args:
         args (Namespace): User specified Arguments
@@ -227,14 +260,14 @@ def build_config_from_args(args: Namespace, config_manager: ConfigManager):
 
     # Append SCHEMA_VALIDATION configs
     if config_manager.validation_type == consts.SCHEMA_VALIDATION:
-        if args.exclusion_columns is not None:
+        if args.exclusion_columns:
             exclusion_columns = cli_tools.get_arg_list(args.exclusion_columns)
             config_manager.append_exclusion_columns(
                 [col.casefold() for col in exclusion_columns]
             )
         config_manager.append_allow_list(args.allow_list, args.allow_list_file)
 
-    # Append CUSTOM_QUERY configs
+    # Append configs specific to CUSTOM_QUERY (i.e. query strings or strings from files)
     if config_manager.validation_type == consts.CUSTOM_QUERY:
         config_manager.append_custom_query_type(args.custom_query_type)
 
@@ -252,75 +285,37 @@ def build_config_from_args(args: Namespace, config_manager: ConfigManager):
             )
         )
 
-        # For custom-query column command
-        if args.custom_query_type == consts.COLUMN_VALIDATION.lower():
-            config_manager.append_aggregates(get_aggregate_config(args, config_manager))
-
-        # For custom-query row command
-        if args.custom_query_type == consts.ROW_VALIDATION.lower():
-            # Append Comparison fields
-            if args.comparison_fields is not None:
-                comparison_fields = cli_tools.get_arg_list(
-                    args.comparison_fields, default_value=[]
-                )
-
-                # As per #1190, add rstrip for Teradata string comparison fields
-                if (
-                    config_manager.source_client.name == "teradata"
-                    or config_manager.target_client.name == "teradata"
-                ):
-                    comparison_fields = config_manager.add_rstrip_to_comp_fields(
-                        comparison_fields
-                    )
-
-                config_manager.append_comparison_fields(
-                    config_manager.build_config_comparison_fields(comparison_fields)
-                )
-
-            # Append calculated fields: --hash/--concat
-            config_manager.append_calculated_fields(
-                get_calculated_config(args, config_manager)
-            )
-
-            # Append primary_keys
-            primary_keys = cli_tools.get_arg_list(args.primary_keys)
-            config_manager.append_primary_keys(
-                config_manager.build_column_configs(primary_keys)
-            )
-
-    # Append COLUMN_VALIDATION configs
-    if config_manager.validation_type == consts.COLUMN_VALIDATION:
+    # Append COLUMN_VALIDATION configs, including custom-query column validation
+    if (
+        config_manager.validation_type == consts.COLUMN_VALIDATION
+        or config_manager.validation_type == consts.CUSTOM_QUERY
+        and args.custom_query_type == consts.COLUMN_VALIDATION.lower()
+    ):
         config_manager.append_aggregates(get_aggregate_config(args, config_manager))
-        if args.grouped_columns is not None:
+        if (
+            config_manager.validation_type == consts.COLUMN_VALIDATION
+            and args.grouped_columns  # grouped_columns not supported in custom queries - at least now.
+        ):
             grouped_columns = cli_tools.get_arg_list(args.grouped_columns)
             config_manager.append_query_groups(
                 config_manager.build_column_configs(grouped_columns)
             )
 
-    # Append ROW_VALIDATION configs
-    if config_manager.validation_type == consts.ROW_VALIDATION:
+    # Append ROW_VALIDATION configs, including custom-query row validation
+    if (
+        config_manager.validation_type == consts.ROW_VALIDATION
+        or config_manager.validation_type == consts.CUSTOM_QUERY
+        and args.custom_query_type == consts.ROW_VALIDATION.lower()
+    ):
         # Append calculated fields: --hash/--concat
         config_manager.append_calculated_fields(
-            get_calculated_config(args, config_manager)
+            _get_calculated_config(args, config_manager)
         )
 
         # Append Comparison fields
-        if args.comparison_fields is not None:
-            comparison_fields = cli_tools.get_arg_list(
-                args.comparison_fields, default_value=[]
-            )
-
-            # As per #1190, add rstrip for Teradata string comparison fields
-            if (
-                config_manager.source_client.name == "teradata"
-                or config_manager.target_client.name == "teradata"
-            ):
-                comparison_fields = config_manager.add_rstrip_to_comp_fields(
-                    comparison_fields
-                )
-
+        if args.comparison_fields:
             config_manager.append_comparison_fields(
-                config_manager.build_config_comparison_fields(comparison_fields)
+                _get_comparison_config(args, config_manager)
             )
 
         # Append primary_keys
@@ -599,7 +594,11 @@ def partition_and_store_config_files(args: Namespace) -> None:
         None
     """
     # Default Validate Type
-    config_managers = build_config_managers_from_args(args, consts.ROW_VALIDATION)
+    if args.tables_list:
+        config_managers = build_config_managers_from_args(args, consts.ROW_VALIDATION)
+    else:
+        setattr(args, "custom_query_type", "row")
+        config_managers = build_config_managers_from_args(args, consts.CUSTOM_QUERY)
     partition_builder = PartitionBuilder(config_managers, args)
     partition_builder.partition_configs()
 

{google_pso_data_validator-6.2.0 → google_pso_data_validator-6.3.0}/data_validation/cli_tools.py

@@ -129,7 +129,7 @@ CONNECTION_SOURCE_FIELDS = {
     "FileSystem": [
         ["table_name", "Table name to use as reference for file data"],
         ["file_path", "The local, s3, or GCS file path to the data"],
-        ["file_type", "The file type of the file. 'csv' or 'json'"],
+        ["file_type", "The file type of the file. 'csv', 'orc', 'parquet' or 'json'"],
     ],
     "Impala": [
         ["host", "Desired Impala host"],
@@ -171,11 +171,45 @@ CONNECTION_SOURCE_FIELDS = {
 }
 
 
+def _check_custom_query_args(parser: argparse.ArgumentParser, parsed_args: Namespace):
+    # This is where we make additional checks if the arguments provided are what we expect
+    # For example, only one of -tbls and custom query options can be provided
+    if hasattr(parsed_args, "tables_list") and hasattr(
+        parsed_args, "source_query"
+    ):  # New Format
+        if (
+            parsed_args.tables_list
+        ):  # Tables_list is not None - so source and target queries all must be None
+            if (
+                parsed_args.source_query_file
+                or parsed_args.source_query
+                or parsed_args.target_query_file
+                or parsed_args.target_query
+            ):
+                parser.error(
+                    f"{parsed_args.command}: when --tables-list/-tbls is specified, --source-query-file/-sqf, --source-query/-sq, --target-query-file/-tqf and --target-query/-tq must not be specified"
+                )
+            else:
+                return
+        elif (parsed_args.source_query_file or parsed_args.source_query) and (
+            parsed_args.target_query_file or parsed_args.target_query
+        ):
+            return
+        else:
+            parser.error(
+                f"{parsed_args.command}: Must specify both source (--source-query-file/-sqf or --source-query/-sq) and target (--target-query-file/-tqf or --target-query/-tq) - when --tables-list/-tbls is not specified"
+            )
+    else:
+        return  # old format - only one of them is present
+
+
 def get_parsed_args() -> Namespace:
     """Return ArgParser with configured CLI arguments."""
     parser = configure_arg_parser()
     args = ["--help"] if len(sys.argv) == 1 else None
-    return parser.parse_args(args)
+    parsed_args = parser.parse_args(args)
+    _check_custom_query_args(parser, parsed_args)
+    return parsed_args
 
 
 def configure_arg_parser():
@@ -225,6 +259,7 @@ def _configure_partition_parser(subparsers):
         default=1,
         help="Number of partitions to be validated in a single yaml file.",
     )
+
     required_arguments.add_argument(
         "--config-dir",
         "-cdir",
@@ -242,6 +277,46 @@ def _configure_partition_parser(subparsers):
         help="Number of partitions into which the table should be split",
         type=_check_positive,
     )
+    # User can provide tables or custom queries, but not both
+    # However, Argparse does not support adding an argument_group to an argument_group or adding a
+    # mutually_exclusive_group or argument_group to a mutually_exclusive_group since version 3.11.
+    # We are only ensuring leaf level mutual exclusivity here and will need to check higher level
+    # mutual exclusivity in the code - i.e. a) when --tables-list is present, there can be no custom
+    # query parameters and b) when custom query parameters are specified, both source and target must be
+    # specified.
+    optional_arguments.add_argument(
+        "--tables-list",
+        "-tbls",
+        help=(
+            "Comma separated tables list in the form "
+            "'schema.table=target_schema.target_table'"
+        ),
+    )
+
+    source_mutually_exclusive = optional_arguments.add_mutually_exclusive_group()
+    source_mutually_exclusive.add_argument(
+        "--source-query-file",
+        "-sqf",
+        help="File containing the source sql query",
+    )
+    source_mutually_exclusive.add_argument(
+        "--source-query",
+        "-sq",
+        help="Source sql query",
+    )
+
+    # Group for mutually exclusive target query arguments. Either must be supplied
+    target_mutually_exclusive = optional_arguments.add_mutually_exclusive_group()
+    target_mutually_exclusive.add_argument(
+        "--target-query-file",
+        "-tqf",
+        help="File containing the target sql query",
+    )
+    target_mutually_exclusive.add_argument(
+        "--target-query",
+        "-tq",
+        help="Target sql query",
+    )
 
 
 def _configure_beta_parser(subparsers):
@@ -443,6 +518,7 @@ def _configure_row_parser(
         "--threshold",
         "-th",
         type=threshold_float,
+        default=0.0,
         help="Float max threshold for percent difference",
     )
     optional_arguments.add_argument(
@@ -454,6 +530,8 @@ def _configure_row_parser(
     optional_arguments.add_argument(
         "--filters",
         "-filters",
+        type=get_filters,
+        default=[],
         help="Filters in the format source_filter:target_filter",
     )
     optional_arguments.add_argument(
@@ -496,9 +574,10 @@ def _configure_row_parser(
             "-rbs",
             help="Row batch size used for random row filters (default 10,000).",
         )
-
-    # Group required arguments
-    if not is_custom_query:
+        # Generate table partitions follows a new argument spec where either the table names or queries can be provided, but not both.
+        # that is specified in configure_partition_parser. If we use the same spec for row and column validation, the custom query commands
+        # may get subsumed by validate and validate commands by specifying tables name or queries. Until this -tbls will be
+        # a required argument for validate row, validate column and validate schema.
         required_arguments.add_argument(
             "--tables-list",
             "-tbls",
@@ -506,6 +585,8 @@ def _configure_row_parser(
             required=True,
             help="Comma separated tables list in the form 'schema.table=target_schema.target_table'",
         )
+
+    # Group required arguments
     required_arguments.add_argument(
         "--primary-keys",
         "-pk",
@@ -603,24 +684,17 @@ def _configure_column_parser(column_parser):
         "--threshold",
         "-th",
         type=threshold_float,
+        default=0.0,
         help="Float max threshold for percent difference",
     )
     optional_arguments.add_argument(
         "--filters",
         "-filters",
+        type=get_filters,
+        default=[],
         help="Filters in the format source_filter:target_filter",
     )
-    optional_arguments.add_argument(
-        "--use-random-row",
-        "-rr",
-        action="store_true",
-        help="Finds a set of random rows of the first primary key supplied.",
-    )
-    optional_arguments.add_argument(
-        "--random-row-batch-size",
-        "-rbs",
-        help="Row batch size used for random row filters (default 10,000).",
-    )
+
     optional_arguments.add_argument(
         "--wildcard-include-string-len",
         "-wis",
@@ -818,25 +892,17 @@ def _configure_custom_query_column_parser(custom_query_column_parser):
     optional_arguments.add_argument(
         "--filters",
         "-filters",
+        type=get_filters,
+        default=[],
         help="Filters in the format source_filter:target_filter",
     )
     optional_arguments.add_argument(
         "--threshold",
         "-th",
         type=threshold_float,
+        default=0.0,
         help="Float max threshold for percent difference",
     )
-    optional_arguments.add_argument(
-        "--use-random-row",
-        "-rr",
-        action="store_true",
-        help="Finds a set of random rows of the first primary key supplied.",
-    )
-    optional_arguments.add_argument(
-        "--random-row-batch-size",
-        "-rbs",
-        help="Row batch size used for random row filters (default 10,000).",
-    )
 
     # Group required arguments
     required_arguments = custom_query_column_parser.add_argument_group(
@@ -910,6 +976,7 @@ def _add_common_arguments(
             "-cj",
             help="Store the validation config in the JSON File Path specified to be used for application use cases",
         )
+
     optional_arguments.add_argument(
         "--format",
         "-fmt",
@@ -1080,7 +1147,7 @@ def get_filters(filter_value: str) -> List[Dict]:
     filter_config = []
     if result := re.fullmatch(single_filter, filter_value):
         if result.group(0) == "":
-            raise ValueError("Empty string not allowed in filter")
+            raise argparse.ArgumentTypeError("Empty string not allowed in filter")
         filter_dict = {
             "type": "custom",
             "source": result.group(0),
@@ -1088,14 +1155,14 @@ def get_filters(filter_value: str) -> List[Dict]:
         }
     elif result := re.fullmatch(double_filter, filter_value):
         if result.group("source") == "" or result.group("target") == "":
-            raise ValueError("Empty string not allowed in filter")
+            raise argparse.ArgumentTypeError("Empty string not allowed in filter")
         filter_dict = {
             "type": "custom",
             "source": result.group("source"),
             "target": result.group("target"),
         }
     else:
-        raise ValueError("Unable to parse filter arguments.")
+        raise argparse.ArgumentTypeError("Unable to parse filter arguments.")
     filter_config.append(filter_dict)
     return filter_config
 
@@ -1315,8 +1382,8 @@ def get_pre_build_configs(args: Namespace, validate_cmd: str) -> List[Dict]:
         else:
             return get_arg_list(concat_arg)
 
-    # Since `generate-table-partitions` defaults to `validate_cmd=row`,
-    # `validate_cmd` is passed along while calling this method
+    # validate_cmd will be set to 'row`, or 'Custom-query' if invoked by generate-table-partitions depending
+    # on what is being partitioned. Otherwise validate_cmd will be set to None
     if validate_cmd is None:
         validate_cmd = args.validate_cmd.capitalize()
 
@@ -1339,14 +1406,9 @@ def get_pre_build_configs(args: Namespace, validate_cmd: str) -> List[Dict]:
     else:
         result_handler_config = None
 
-    # Get filter_config and threshold. Not supported in case of schema validation
-    filter_config = []
-    threshold = 0.0
-    if config_type != consts.SCHEMA_VALIDATION:
-        if args.filters:
-            filter_config = get_filters(args.filters)
-        if args.threshold:
-            threshold = args.threshold
+    # Set filter_config and threshold. Not supported in case of schema validation
+    filter_config = getattr(args, "filters", [])
+    threshold = getattr(args, "threshold", 0.0)
 
     # Get labels
     if args.labels is None:
@@ -1362,16 +1424,9 @@ def get_pre_build_configs(args: Namespace, validate_cmd: str) -> List[Dict]:
     # Get format: text, csv, json, table. Default is table
     format = args.format if args.format else "table"
 
-    # Get random row arguments. Not supported in case of schema validation,
-    # custom-query validation and generate-table-partitions
-    use_random_rows = None
-    random_row_batch_size = None
-    if (
-        args.command != "generate-table-partitions"
-        and config_type != consts.SCHEMA_VALIDATION
-    ):
-        use_random_rows = getattr(args, "use_random_row", False)
-        random_row_batch_size = getattr(args, "random_row_batch_size", None)
+    # Get random row arguments. Only in row validations these attributes can be present.
+    use_random_rows = getattr(args, "use_random_row", False)
+    random_row_batch_size = getattr(args, "random_row_batch_size", None)
 
     # Get table list. Not supported in case of custom query validation
     is_filesystem = source_client._source_type == "FileSystem"

{google_pso_data_validator-6.2.0 → google_pso_data_validator-6.3.0}/data_validation/clients.py

@@ -119,12 +119,16 @@ def get_pandas_client(table_name, file_path, file_type):
 
     table_name (str): Table name to use as reference for file data
     file_path (str): The local, s3, or GCS file path to the data
-    file_type (str): The file type of the file (csv or json)
+    file_type (str): The file type of the file (csv, json, orc or parquet)
     """
     if file_type == "csv":
         df = pandas.read_csv(file_path)
     elif file_type == "json":
         df = pandas.read_json(file_path)
+    elif file_type == "orc":
+        df = pandas.read_orc(file_path)
+    elif file_type == "parquet":
+        df = pandas.read_parquet(file_path)
     else:
         raise ValueError(f"Unknown Pandas File Type: {file_type}")