google-pso-data-validator 8.5.2__tar.gz → 8.7.0__tar.gz

{google_pso_data_validator-8.5.2/google_pso_data_validator.egg-info → google_pso_data_validator-8.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-pso-data-validator
-Version: 8.5.2
+Version: 8.7.0
 Summary: A package to enable easy data validation
 Author: PSO DVT Engineering team
 Author-email: data-validator-eng@google.com
@@ -39,7 +39,7 @@ Requires-Dist: 1.10.11; extra == "apache-airflow"
 Provides-Extra: pyspark
 Requires-Dist: 3.0.0; extra == "pyspark"
 Provides-Extra: develop
-Requires-Dist: black==22.3.0; extra == "develop"
+Requires-Dist: black==26.1.0; extra == "develop"
 Requires-Dist: flake8; extra == "develop"
 Requires-Dist: freezegun; extra == "develop"
 Requires-Dist: pyfakefs; extra == "develop"
@@ -83,7 +83,8 @@ DVT supports the following connection types:
 
 *   [AlloyDB](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#alloydb)
 *   [BigQuery](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-bigquery)
-*   [DB2](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#db2)
+*   [Db2 LUW](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#db2_luw)
+*   [Db2 z/OS](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#db2_zos)
 *   [FileSystem](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#filesystem)
 *   [Hive](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#hive)
 *   [Impala](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#impala)
@@ -130,7 +131,7 @@ The CLI is the main interface to use this tool and it has several different
 commands which can be used to create and run validations. DVT is designed to run in
 an environment connected to GCP services, specifically, BigQuery, GCS and Secret manager.
 If DVT is being run on-premises or in an environment with restricted access to GCP services, see
-[running DVT  at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
+[running DVT at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
 
 Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions, or Airflow
 ([Examples Here](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples)). See the [Validation Logic](https://github.com/GoogleCloudPlatform/professional-services-data-validator#validation-logic) section
@@ -719,17 +720,27 @@ target. The `find-tables` command:
 -   Pulls all tables in the source (applying a supplied `allowed-schemas` filter)
 -   Pulls all tables from the target
 -   Uses Jaro Similarity algorithm to match tables
+-   Optionally translates source schema names using mappings supplied in `allowed-schemas` before matching
 -   Finally, it prints a JSON list of tables which can be a reference for the
     validation run config.
 
 Note that our default value for the `score-cutoff` parameter is 1 and it seeks for identical matches. If no matches occur, reduce this value as deemed necessary. By using smaller numbers such as 0.7, 0.65 etc you can get more matches. For reference, we make use of [this jaro_similarity method](https://jamesturk.github.io/jellyfish/functions/#jaro-similarity) for the string comparison.
 
+
 ```shell
 data-validation find-tables --source-conn source --target-conn target \
     --allowed-schemas pso_data_validator \
     --score-cutoff 1
 ```
 
+If your source and target environments have different schema names (e.g., `prod_raw` vs `dwh_raw`), you can provide a mapping directly in the `--allowed-schemas` flag using the `source_schema=target_schema` syntax.
+
+```shell
+data-validation find-tables --source-conn source --target-conn target \
+    --allowed-schemas prod=dwh \
+    --score-cutoff 1
+```
+
 ### Using Beta CLI Features
 
 There may be occasions we want to release a new CLI feature under a Beta flag.

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/README.md

@@ -23,7 +23,8 @@ DVT supports the following connection types:
 
 *   [AlloyDB](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#alloydb)
 *   [BigQuery](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-bigquery)
-*   [DB2](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#db2)
+*   [Db2 LUW](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#db2_luw)
+*   [Db2 z/OS](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#db2_zos)
 *   [FileSystem](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#filesystem)
 *   [Hive](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#hive)
 *   [Impala](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#impala)
@@ -70,7 +71,7 @@ The CLI is the main interface to use this tool and it has several different
 commands which can be used to create and run validations. DVT is designed to run in
 an environment connected to GCP services, specifically, BigQuery, GCS and Secret manager.
 If DVT is being run on-premises or in an environment with restricted access to GCP services, see
-[running DVT  at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
+[running DVT at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
 
 Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions, or Airflow
 ([Examples Here](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples)). See the [Validation Logic](https://github.com/GoogleCloudPlatform/professional-services-data-validator#validation-logic) section
@@ -659,17 +660,27 @@ target. The `find-tables` command:
 -   Pulls all tables in the source (applying a supplied `allowed-schemas` filter)
 -   Pulls all tables from the target
 -   Uses Jaro Similarity algorithm to match tables
+-   Optionally translates source schema names using mappings supplied in `allowed-schemas` before matching
 -   Finally, it prints a JSON list of tables which can be a reference for the
     validation run config.
 
 Note that our default value for the `score-cutoff` parameter is 1 and it seeks for identical matches. If no matches occur, reduce this value as deemed necessary. By using smaller numbers such as 0.7, 0.65 etc you can get more matches. For reference, we make use of [this jaro_similarity method](https://jamesturk.github.io/jellyfish/functions/#jaro-similarity) for the string comparison.
 
+
 ```shell
 data-validation find-tables --source-conn source --target-conn target \
     --allowed-schemas pso_data_validator \
     --score-cutoff 1
 ```
 
+If your source and target environments have different schema names (e.g., `prod_raw` vs `dwh_raw`), you can provide a mapping directly in the `--allowed-schemas` flag using the `source_schema=target_schema` syntax.
+
+```shell
+data-validation find-tables --source-conn source --target-conn target \
+    --allowed-schemas prod=dwh \
+    --score-cutoff 1
+```
+
 ### Using Beta CLI Features
 
 There may be occasions we want to release a new CLI feature under a Beta flag.

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/__init__.py

@@ -14,6 +14,5 @@
 
 from importlib.metadata import version
 
-
 __version__ = version("google-pso-data-validator")
 __all__ = ["__version__"]

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/cli_tools.py

@@ -75,9 +75,13 @@ CONNECTION_SOURCE_FIELDS = {
         ["project_id", "GCP Project to use for BigQuery"],
         [
             "client_project_id",
-            "(Optional) BigQuery job/billing project (can differ from data project)",
+            "(Deprecated) BigQuery job/billing project (can differ from data project)",
         ],
-        ["google_service_account_key_path", "(Optional) GCP SA Key Path"],
+        [
+            "billing_project_id",
+            "(Optional) BigQuery billing project to override default billing project",
+        ],
+        ["google_service_account_key_path", "(Deprecated) GCP SA Key Path"],
         [
             "api_endpoint",
             '(Optional) GCP BigQuery API endpoint (e.g. "https://bigquery-mypsc.p.googleapis.com")',
@@ -149,7 +153,7 @@ CONNECTION_SOURCE_FIELDS = {
         ["project_id", "GCP Project to use for Spanner"],
         ["instance_id", "ID of Spanner instance to connect to"],
         ["database_id", "ID of Spanner database (schema) to connect to"],
-        ["google_service_account_key_path", "(Optional) GCP SA Key Path"],
+        ["google_service_account_key_path", "(Deprecated) GCP SA Key Path"],
         [
             "api_endpoint",
             '(Optional) GCP Spanner API endpoint (e.g. "https://spanner-mypsc.p.googleapis.com")',
@@ -199,13 +203,24 @@ CONNECTION_SOURCE_FIELDS = {
         ["http_path", "URL path of HTTP proxy"],
     ],
     consts.SOURCE_TYPE_DB2: [
-        ["host", "DB2 host"],
-        ["port", "DB2 port (50000 if not provided)"],
+        ["host", "Db2 host"],
+        ["port", "Db2 port (50000 if not provided)"],
         ["user", "Username to connect to"],
         ["password", "Password for authentication of user"],
-        ["database", "Database in DB2 to connect to"],
-        ["url", "URL link in DB2 to connect to"],
-        ["driver", "Driver link in DB2 to connect to (default ibm_db_sa)"],
+        ["database", "Database in Db2 to connect to"],
+        ["url", "URL link in Db2 to connect to"],
+        ["driver", "Driver link in Db2 to connect to (default ibm_db_sa)"],
+        ["connect_args", "(Optional) Additional connection argument mapping"],
+    ],
+    consts.SOURCE_TYPE_DB2_ZOS: [
+        ["host", "Db2 host"],
+        ["port", "Db2 port (50000 if not provided)"],
+        ["user", "Username to connect to"],
+        ["password", "Password for authentication of user"],
+        ["database", "Database in Db2 to connect to"],
+        ["url", "URL link in Db2 to connect to"],
+        ["driver", "Driver link in Db2 to connect to (default ibm_db_sa)"],
+        ["connect_args", "(Optional) Additional connection argument mapping"],
     ],
 }
 
@@ -404,7 +419,9 @@ def _configure_find_tables(subparsers):
         "--target-conn", "-tc", help="Target connection name."
     )
     find_tables_parser.add_argument(
-        "--allowed-schemas", "-as", help="List of source schemas to match."
+        "--allowed-schemas",
+        "-as",
+        help="List of source schemas to match. Can include mappings like 'source_schema=target_schema'.",
     )
     find_tables_parser.add_argument(
         "--include-views",

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/client_info.py

@@ -16,7 +16,6 @@ from google.api_core import client_info as http_client_info
 
 import data_validation
 
-
 APPLICATION_NAME = "google-pso-tool/data-validator"
 USER_AGENT = "{}/{}".format(APPLICATION_NAME, data_validation.__version__)
 

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/clients.py

@@ -53,6 +53,7 @@ IBIS_ALCHEMY_BACKENDS = [
     "oracle",
     "postgres",
     "db2",
+    "db2_zos",
     "mssql",
     "redshift",
     "snowflake",
@@ -91,8 +92,10 @@ except ImportError:
 # DB2 requires ibm_db_sa
 try:
     from third_party.ibis.ibis_db2.api import db2_connect
+    from third_party.ibis.ibis_db2_zos.api import db2_zos_connect
 except ImportError:
     db2_connect = _raise_missing_client_error("pip install ibm_db_sa")
+    db2_zos_connect = _raise_missing_client_error("pip install ibm_db_sa")
 
 # Sybase requires sqlalchemy_sybase package.
 try:
@@ -111,15 +114,13 @@ def get_google_bigquery_client(
     job_config = bigquery.QueryJobConfig(
         connection_properties=[bigquery.ConnectionProperty("time_zone", "UTC")]
     )
-    effective_project = quota_project_id or project_id
     options = None
     if api_endpoint or quota_project_id:
         options = client_options.ClientOptions(
             api_endpoint=api_endpoint,
-            quota_project_id=quota_project_id if quota_project_id else None,
         )
     return bigquery.Client(
-        project=effective_project,
+        project=quota_project_id,
         client_info=info,
         credentials=credentials,
         default_query_job_config=job_config,
@@ -136,7 +137,6 @@ def _get_google_bqstorage_client(
     if api_endpoint or quota_project_id:
         options = client_options.ClientOptions(
             api_endpoint=api_endpoint,
-            quota_project_id=quota_project_id if quota_project_id else None,
         )
     from google.cloud import bigquery_storage_v1 as bigquery_storage
 
@@ -152,20 +152,26 @@ def get_bigquery_client(
     credentials=None,
     api_endpoint: Optional[str] = None,
     storage_api_endpoint: Optional[str] = None,
-    client_project_id: Optional[str] = None,
+    client_project_id: Optional[str] = None,  # to be deprecated in the future
+    billing_project_id: Optional[str] = None,
 ):
+    if client_project_id:
+        logging.warning(
+            "client_project_id is deprecated and will be removed in the future, use --billing-project-id instead"
+        )
+        billing_project_id = client_project_id
     google_client = get_google_bigquery_client(
         project_id,
         credentials=credentials,
         api_endpoint=api_endpoint,
-        quota_project_id=client_project_id,
+        quota_project_id=billing_project_id,
     )
     bqstorage_client = None
     if storage_api_endpoint:
         bqstorage_client = _get_google_bqstorage_client(
             credentials=credentials,
             api_endpoint=storage_api_endpoint,
-            quota_project_id=client_project_id,
+            quota_project_id=billing_project_id,
         )
 
     return bigquery_connect(
@@ -227,6 +233,7 @@ def get_ibis_table(client, schema_name, table_name, database_name=None):
         "oracle",
         "postgres",
         "db2",
+        "db2_zos",
         "mssql",
         "redshift",
         "sybase",
@@ -342,10 +349,14 @@ def get_data_client(connection_config):
             consts.GOOGLE_SERVICE_ACCOUNT_KEY_PATH
         )
         if key_path:
-            decrypted_connection_config[
-                "credentials"
-            ] = google.oauth2.service_account.Credentials.from_service_account_file(
-                key_path
+            logging.warning(
+                """"Use of Service Account Keys is strongly discouraged. This option is deprecated and may be removed.
+                Run DVT with the service account identity by using application-default-credentials with --impersonate-service-account"""
+            )
+            decrypted_connection_config["credentials"] = (
+                google.oauth2.service_account.Credentials.from_service_account_file(
+                    key_path
+                )
             )
 
     if source_type not in CLIENT_LOOKUP:
@@ -430,4 +441,5 @@ CLIENT_LOOKUP = {
     consts.SOURCE_TYPE_SPANNER: spanner_connect,
     consts.SOURCE_TYPE_SYBASE: sybase_connect,
     consts.SOURCE_TYPE_DB2: db2_connect,
+    consts.SOURCE_TYPE_DB2_ZOS: db2_zos_connect,
 }

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/config_manager.py

@@ -667,12 +667,34 @@ class ConfigManager(object):
                 isinstance(source_type, dt.UUID) or isinstance(target_type, dt.UUID)
             )
 
+    def _is_db2_blob(self, source_column_name: str, target_column_name: str) -> bool:
+        """Returns True when either source or target column is Db2 (LUW and z/OS) BLOB data type."""
+        return self._is_raw_data_type(
+            "db2", source_column_name, target_column_name, ["BLOB"]
+        ) or self._is_raw_data_type(
+            "db2_zos", source_column_name, target_column_name, ["BLOB"]
+        )
+
     def _is_db2_xml(self, source_column_name: str, target_column_name: str) -> bool:
-        """Returns True when either source or target column is Oracle LOB data type."""
+        """Returns True when either source or target column is Db2 XML data type."""
         return self._is_raw_data_type(
             "db2", source_column_name, target_column_name, ["XML"]
         )
 
+    def _is_db2_zos_blob(
+        self, source_column_name: str, target_column_name: str
+    ) -> bool:
+        """Returns True when either source or target column is Db2 z/OS BLOB data type."""
+        return self._is_raw_data_type(
+            "db2_zos", source_column_name, target_column_name, ["BLOB"]
+        )
+
+    def _is_db2_zos_xml(self, source_column_name: str, target_column_name: str) -> bool:
+        """Returns True when either source or target column is Db2 z/OS XML data type."""
+        return self._is_raw_data_type(
+            "db2_zos", source_column_name, target_column_name, ["XML"]
+        )
+
     def _is_oracle_lob(self, source_column_name: str, target_column_name: str) -> bool:
         """Returns True when either source or target column is Oracle LOB data type.
 
@@ -1058,7 +1080,11 @@ class ConfigManager(object):
                     # Oracle BLOB is invalid for use with SQL COUNT function.
                     # The expression below returns True if client is Oracle which
                     # has the effect of triggering use of byte_length transformation.
-                    return self._is_oracle_lob(source_column, target_column)
+                    # Same for Db2 z/OS.
+                    return bool(
+                        self._is_oracle_lob(source_column, target_column)
+                        or self._is_db2_zos_blob(source_column, target_column)
+                    )
                 else:
                     # Convert to length for any min/max/sum on binary columns.
                     return True
@@ -1146,6 +1172,11 @@ class ConfigManager(object):
                     f"Skipping {agg_type} on {column} due to SQL Server image data type"
                 )
                 continue
+            elif self._is_db2_zos_xml(column, column):
+                logging.info(
+                    f"Skipping {agg_type} on {column} due to Db2 z/OS XML data type"
+                )
+                continue
             elif agg_type != "count" and self._is_db2_xml(column, column):
                 logging.info(
                     f"Skipping {agg_type} on {column} due to Db2 XML data type"
@@ -1351,6 +1382,14 @@ class ConfigManager(object):
                 )
                 result_source_columns.pop(source_column)
                 result_target_columns.pop(target_column)
+            if self._is_db2_blob(source_column, target_column):
+                # Db2 BLOB is incompatible with Db2s own HEX function.
+                # Users would need to use a custom query to a custom UDF to row validate these.
+                logging.info(
+                    f"Skipping column {source_column} due to Db2 BLOB data type"
+                )
+                result_source_columns.pop(source_column)
+                result_target_columns.pop(target_column)
 
         return result_source_columns, result_target_columns
 

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/consts.py

@@ -130,6 +130,7 @@ YAML_VALIDATIONS = "validations"
 # Connection key constants.
 SOURCE_TYPE_BIGQUERY = "BigQuery"
 SOURCE_TYPE_DB2 = "DB2"
+SOURCE_TYPE_DB2_ZOS = "DB2_ZOS"
 SOURCE_TYPE_FILESYSTEM = "FileSystem"
 SOURCE_TYPE_IMPALA = "Impala"
 SOURCE_TYPE_MSSQL = "MSSQL"
@@ -151,6 +152,7 @@ GOOGLE_SERVICE_ACCOUNT_KEY_PATH = "google_service_account_key_path"
 API_ENDPOINT = "api_endpoint"
 STORAGE_API_ENDPOINT = "storage_api_endpoint"
 CLIENT_PROJECT_ID = "client_project_id"
+BILLING_PROJECT_ID = "billing_project_id"
 
 # Result Handler Output Table Fields
 VALIDATION_TYPE = "validation_type"

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/find_tables.py

@@ -28,26 +28,37 @@ if TYPE_CHECKING:
     import ibis
 
 
-def _compare_match_tables(source_table_map, target_table_map, score_cutoff=0.8) -> list:
+def _compare_match_tables(
+    source_table_map: dict,
+    target_table_map: dict,
+    score_cutoff: float = 0.8,
+    schema_map: dict = None,
+) -> list:
     """Return dict config object from matching tables."""
     # TODO(dhercher): evaluate if improved comparison and score cutoffs should be used.
     table_configs = []
+    schema_map = schema_map or {}
 
     target_keys = target_table_map.keys()
     for source_key in source_table_map:
+        source_schema = source_table_map[source_key][consts.CONFIG_SCHEMA_NAME]
+        source_table = source_table_map[source_key][consts.CONFIG_TABLE_NAME]
+
+        # Apply schema mapping if it exists, when there is a mapping
+        # lookup_key is {mapped_schema}.{source_name} which should then
+        # find a match in target_keys.
+        lookup_schema = schema_map.get(source_schema, source_schema)
+        lookup_key = f"{lookup_schema}.{source_table}".casefold()
+
         target_key = jellyfish_distance.extract_closest_match(
-            source_key, target_keys, score_cutoff=score_cutoff
+            lookup_key, target_keys, score_cutoff=score_cutoff
         )
         if target_key is None:
             continue
 
         table_config = {
-            consts.CONFIG_SCHEMA_NAME: source_table_map[source_key][
-                consts.CONFIG_SCHEMA_NAME
-            ],
-            consts.CONFIG_TABLE_NAME: source_table_map[source_key][
-                consts.CONFIG_TABLE_NAME
-            ],
+            consts.CONFIG_SCHEMA_NAME: source_schema,
+            consts.CONFIG_TABLE_NAME: source_table,
             consts.CONFIG_TARGET_SCHEMA_NAME: target_table_map[target_key][
                 consts.CONFIG_SCHEMA_NAME
             ],
@@ -95,6 +106,7 @@ def get_mapped_table_configs(
     allowed_schemas: list = None,
     include_views: bool = False,
     score_cutoff: int = 1,
+    schema_map: dict = None,
 ) -> list:
     """Get table list from each client and match them together into a single list of dicts."""
     source_table_map = _get_table_map(
@@ -102,7 +114,10 @@ def get_mapped_table_configs(
     )
     target_table_map = _get_table_map(target_client, include_views=include_views)
     return _compare_match_tables(
-        source_table_map, target_table_map, score_cutoff=score_cutoff
+        source_table_map,
+        target_table_map,
+        score_cutoff=score_cutoff,
+        schema_map=schema_map,
     )
 
 
@@ -114,13 +129,22 @@ def find_tables_using_string_matching(args) -> str:
     source_client = clients.get_data_client(mgr.get_connection_config(args.source_conn))
     target_client = clients.get_data_client(mgr.get_connection_config(args.target_conn))
 
-    allowed_schemas = cli_tools.get_arg_list(args.allowed_schemas)
+    allowed_schemas_raw = cli_tools.get_arg_list(args.allowed_schemas) or []
+    allowed_schemas = []
+    schema_map = {}
+    for schema in allowed_schemas_raw:
+        if "=" in schema:
+            schema, target_schema = schema.split("=", 1)
+            schema_map[schema] = target_schema
+        allowed_schemas.append(schema)
+
     table_configs = get_mapped_table_configs(
         source_client,
         target_client,
         allowed_schemas=allowed_schemas,
         include_views=args.include_views,
         score_cutoff=score_cutoff,
+        schema_map=schema_map,
     )
     return json.dumps(table_configs)
 

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/gcs_helper.py

@@ -19,7 +19,6 @@ from typing import List
 from google.cloud import storage
 from data_validation import client_info
 
-
 WRITE_SUCCESS_STRING = "Success! Config output written to"
 DELETE_SUCCESS_STRING = "Successfully deleted"
 

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/query_builder/random_row_builder.py

@@ -19,9 +19,8 @@ import ibis
 from data_validation import clients
 from data_validation.query_builder.query_builder import QueryBuilder
 
-
 # Adding new data sources should be done by adding the Backend name here
-RANDOM_SORT_SUPPORTS = [
+SUPPORTS_SORT_BY_RANDOM = [
     "bigquery",
     "teradata",
     "impala",
@@ -36,6 +35,10 @@ RANDOM_SORT_SUPPORTS = [
     "sybase",
 ]
 
+SUPPORTS_SORT_BY_ALIASED_RANDOM = [
+    "db2_zos",
+]
+
 
 class RandomRowBuilder(object):
     def __init__(self, primary_keys: List[str], batch_size: int):
@@ -88,7 +91,16 @@ class RandomRowBuilder(object):
         self, data_client: ibis.backends.base.BaseBackend, table: ibis.Expr
     ) -> ibis.Expr:
         """Return a randomly sorted query if it is supported for the client."""
-        if data_client.name in RANDOM_SORT_SUPPORTS:
+        if data_client.name in SUPPORTS_SORT_BY_ALIASED_RANDOM:
+            random_sort_col = "random_sort"
+            return (
+                table[self.primary_keys]
+                .mutate(**{random_sort_col: ibis.random()})
+                .order_by(random_sort_col)
+                .limit(self.batch_size)[self.primary_keys]
+            )
+
+        if data_client.name in SUPPORTS_SORT_BY_RANDOM:
             # Teradata 'SAMPLE' and Spanner 'TABLESAMPLE' is random by nature
             # and does not require a sort by
             if data_client.name == "teradata" or data_client.name == "spanner":

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/result_handlers/base_backend.py

@@ -20,7 +20,6 @@ import ibis
 from data_validation import consts, util
 from data_validation.result_handlers import text as text_handler
 
-
 if TYPE_CHECKING:
     from pandas import DataFrame
 

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/result_handlers/bigquery.py

@@ -23,7 +23,6 @@ import google.oauth2.service_account
 from data_validation import clients, consts, exceptions, util
 from data_validation.result_handlers.base_backend import BaseBackendResultHandler
 
-
 BQRH_WRITE_MESSAGE = "Results written to BigQuery"
 BQRH_NO_WRITE_MESSAGE = "No results to write to BigQuery"
 

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/result_handlers/postgres.py

@@ -28,7 +28,6 @@ from data_validation.result_handlers.base_backend import (
     RH_NO_WRITE_MESSAGE,
 )
 
-
 if TYPE_CHECKING:
     from pandas import DataFrame
     from ibis.backends.base import BaseBackend

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/schema_validation.py

@@ -20,7 +20,6 @@ import re
 
 from data_validation import metadata, consts, clients, exceptions
 
-
 # Check for decimal data type with precision and/or scale. Permits hyphen in p/s for value ranges.
 DECIMAL_PRECISION_SCALE_PATTERN = re.compile(
     r"([!]?decimal)\(([0-9\-]+)(?:,[ ]*([0-9\-]+))?\)", re.I
@@ -187,7 +186,10 @@ def schema_validation_matching(
             )
         else:
             # Target data type mismatch
-            (higher_precision, lower_precision,) = parse_n_validate_datatypes(
+            (
+                higher_precision,
+                lower_precision,
+            ) = parse_n_validate_datatypes(
                 string_val(source_field_type), string_val(target_field_type)
             )
             if higher_precision:

{google_pso_data_validator-8.5.2 → google_pso_data_validator-8.7.0}/data_validation/util.py

@@ -16,6 +16,7 @@ import json
 import logging
 import re
 import time
+import uuid
 
 from data_validation import clients, exceptions
 
@@ -87,3 +88,19 @@ def ibis_table_to_sql(ibis_table: "IbisTable", alchemy_client: "BaseBackend") ->
     else:
         sql_string = str(ibis_table.compile())
     return sql_string
+
+
+def dvt_temp_object_name(prefix: str = "dvt_temp") -> str:
+    """Generate a random name for when DVT needs to create a temporary object.
+
+    Args:
+        prefix: The prefix to use for the temporary object name.
+
+    Returns:
+        A random name for when DVT needs to create a temporary object.
+    """
+    if not isinstance(prefix, str) or not re.match(r"^[a-zA-Z0-9_]+$", prefix):
+        raise exceptions.ValidationException(
+            f"Invalid prefix: '{prefix}'. Only alphanumeric and underscore characters are allowed."
+        )
+    return f"{prefix}_{uuid.uuid4().hex[:8].lower()}"