google-pso-data-validator 7.6.0__tar.gz → 7.7.0__tar.gz

{google_pso_data_validator-7.6.0/google_pso_data_validator.egg-info → google_pso_data_validator-7.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-pso-data-validator
-Version: 7.6.0
+Version: 7.7.0
 Summary: A package to enable easy data validation
 Author: PSO DVT Engineering team
 Author-email: data-validator-eng@google.com
@@ -125,8 +125,10 @@ please review the [Connections](https://github.com/GoogleCloudPlatform/professio
 ### Running Validations
 
 The CLI is the main interface to use this tool and it has several different
-commands which can be used to create and run validations. Below are the command
-syntax and options for running validations.
+commands which can be used to create and run validations. DVT is designed to run in
+an environment connected to GCP services, specifically, BigQuery, GCS and Secret manager.
+If DVT is being run on-premises or in an environment with restricted access to GCP services, see
+[running DVT  at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
 
 Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions, or Airflow
 ([Examples Here](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples)). See the [Validation Logic](https://github.com/GoogleCloudPlatform/professional-services-data-validator#validation-logic) section
@@ -584,6 +586,11 @@ For example, this flag can be used as follows:
     "target_query": "SELECT `hash__all`, `station_id`\nFROM ..."
 }
 ```
+#### Running DVT at on-prem
+On-premises environments can have limited access to GCP services. DVT supports using BigQuery for storing validation results, GCS for storage and
+the Secret Manager for secrets. You may also use BigQuery and Spanner as a source or target for validation. Service
+APIs (i.e. bigquery.googleapis.com) may not always be accessible due to firewall restrictions. Work with your network
+adminstrator to identify the way to access these services. They may set up a [Private Service Connect Endpoint](https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints). DVT supports accessing source and target tables in Spanner and BigQuery via endpoints set up in your network. Connection Parameters for [Spanner](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-spanner) and [BigQuery](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-bigquery) outline regarding how to specify endpoints.
 
 ### Running DVT with YAML Configuration Files
 

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/README.md

@@ -66,8 +66,10 @@ please review the [Connections](https://github.com/GoogleCloudPlatform/professio
 ### Running Validations
 
 The CLI is the main interface to use this tool and it has several different
-commands which can be used to create and run validations. Below are the command
-syntax and options for running validations.
+commands which can be used to create and run validations. DVT is designed to run in
+an environment connected to GCP services, specifically, BigQuery, GCS and Secret manager.
+If DVT is being run on-premises or in an environment with restricted access to GCP services, see
+[running DVT  at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
 
 Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions, or Airflow
 ([Examples Here](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples)). See the [Validation Logic](https://github.com/GoogleCloudPlatform/professional-services-data-validator#validation-logic) section
@@ -525,6 +527,11 @@ For example, this flag can be used as follows:
     "target_query": "SELECT `hash__all`, `station_id`\nFROM ..."
 }
 ```
+#### Running DVT at on-prem
+On-premises environments can have limited access to GCP services. DVT supports using BigQuery for storing validation results, GCS for storage and
+the Secret Manager for secrets. You may also use BigQuery and Spanner as a source or target for validation. Service
+APIs (i.e. bigquery.googleapis.com) may not always be accessible due to firewall restrictions. Work with your network
+adminstrator to identify the way to access these services. They may set up a [Private Service Connect Endpoint](https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints). DVT supports accessing source and target tables in Spanner and BigQuery via endpoints set up in your network. Connection Parameters for [Spanner](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-spanner) and [BigQuery](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-bigquery) outline regarding how to specify endpoints.
 
 ### Running DVT with YAML Configuration Files
 

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/cli_tools.py

@@ -76,7 +76,11 @@ CONNECTION_SOURCE_FIELDS = {
         ["google_service_account_key_path", "(Optional) GCP SA Key Path"],
         [
             "api_endpoint",
-            '(Optional) GCP BigQuery API endpoint (e.g. "https://mybq.p.googleapis.com")',
+            '(Optional) GCP BigQuery API endpoint (e.g. "https://bigquery-mypsc.p.googleapis.com")',
+        ],
+        [
+            "storage_api_endpoint",
+            '(Optional) GCP BigQuery Storage API endpoint (e.g. "https://bigquerystorage-mypsc.p.googleapis.com")',
         ],
     ],
     consts.SOURCE_TYPE_TERADATA: [
@@ -140,7 +144,7 @@ CONNECTION_SOURCE_FIELDS = {
         ["google_service_account_key_path", "(Optional) GCP SA Key Path"],
         [
             "api_endpoint",
-            '(Optional) GCP Spanner API endpoint (e.g. "https://mycs.p.googleapis.com")',
+            '(Optional) GCP Spanner API endpoint (e.g. "https://spanner-mypsc.p.googleapis.com")',
         ],
     ],
     consts.SOURCE_TYPE_FILESYSTEM: [
@@ -1284,6 +1288,9 @@ def _get_result_handler(rc_value: str, sa_file=None) -> dict:
                 consts.PROJECT_ID: conn_from_file["project_id"],
                 consts.TABLE_ID: config[1],
                 consts.API_ENDPOINT: conn_from_file.get("api_endpoint", None),
+                consts.STORAGE_API_ENDPOINT: conn_from_file.get(
+                    "storage_api_endpoint", None
+                ),
             }
         elif conn_from_file[consts.SOURCE_TYPE] == consts.SOURCE_TYPE_POSTGRES:
             result_handler = {

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/clients.py

@@ -27,6 +27,8 @@ import pandas
 
 from data_validation import client_info, consts, exceptions
 from data_validation.secret_manager import SecretManagerBuilder
+
+from third_party.ibis.ibis_bigquery.api import bigquery_connect
 from third_party.ibis.ibis_cloud_spanner.api import spanner_connect
 from third_party.ibis.ibis_impala.api import impala_connect
 from third_party.ibis.ibis_mssql.api import mssql_connect
@@ -111,24 +113,42 @@ def get_google_bigquery_client(
     )
 
 
+def _get_google_bqstorage_client(credentials=None, api_endpoint: str = None):
+    options = None
+    if api_endpoint:
+        options = client_options.ClientOptions(api_endpoint=api_endpoint)
+    from google.cloud import bigquery_storage_v1 as bigquery_storage
+
+    return bigquery_storage.BigQueryReadClient(
+        credentials=credentials,
+        client_options=options,
+    )
+
+
 def get_bigquery_client(
-    project_id: str, dataset_id: str = "", credentials=None, api_endpoint: str = None
+    project_id: str,
+    dataset_id: str = "",
+    credentials=None,
+    api_endpoint: str = None,
+    storage_api_endpoint: str = None,
 ):
     google_client = get_google_bigquery_client(
         project_id, credentials=credentials, api_endpoint=api_endpoint
     )
+    bqstorage_client = None
+    if storage_api_endpoint:
+        bqstorage_client = _get_google_bqstorage_client(
+            credentials=credentials, api_endpoint=storage_api_endpoint
+        )
 
-    ibis_client = ibis.bigquery.connect(
+    return bigquery_connect(
         project_id=project_id,
         dataset_id=dataset_id,
         credentials=credentials,
+        bigquery_client=google_client,
+        bqstorage_client=bqstorage_client,
     )
 
-    # Override the BigQuery client object to ensure the correct user agent is
-    # included and any api_endpoint is used.
-    ibis_client.client = google_client
-    return ibis_client
-
 
 def get_pandas_client(table_name, file_path, file_type):
     """Return pandas client and env with file loaded into DataFrame

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/combiner.py

@@ -406,18 +406,18 @@ def _pivot_result(
                             consts.AGGREGATION_TYPE
                         ),
                         ibis.literal(validation.get_table_name(result_type)).name(
-                            "table_name"
+                            consts.COMBINER_TABLE_NAME
                         ),
                         # Cast to string to ensure types match, even when column
                         # name is NULL (such as for count aggregations).
                         ibis.literal(validation.get_column_name(result_type))
                         .cast("string")
-                        .name("column_name"),
+                        .name(consts.COMBINER_COLUMN_NAME),
                         primary_keys,
                         ibis.literal(validation.num_random_rows).name(
                             consts.NUM_RANDOM_ROWS
                         ),
-                        result[field].cast("string").name("agg_value"),
+                        result[field].cast("string").name(consts.COMBINER_AGG_VALUE),
                     )
                     + join_on_fields
                 )
@@ -469,11 +469,11 @@ def _join_pivots(
         + [
             source[consts.VALIDATION_TYPE],
             source[consts.AGGREGATION_TYPE],
-            source[consts.CONFIG_TABLE_NAME],
-            source["column_name"],
+            source[consts.COMBINER_TABLE_NAME],
+            source[consts.COMBINER_COLUMN_NAME],
             source[consts.CONFIG_PRIMARY_KEYS],
             source[consts.NUM_RANDOM_ROWS],
-            source["agg_value"],
+            source[consts.COMBINER_AGG_VALUE],
             differences[consts.VALIDATION_DIFFERENCE],
             differences[consts.VALIDATION_PCT_DIFFERENCE],
             differences[consts.VALIDATION_PCT_THRESHOLD],
@@ -488,12 +488,12 @@ def _join_pivots(
         source_difference[consts.AGGREGATION_TYPE]
         .fillna(target[consts.AGGREGATION_TYPE])
         .name(consts.AGGREGATION_TYPE),
-        source_difference["table_name"].name(consts.SOURCE_TABLE_NAME),
-        source_difference["column_name"].name(consts.SOURCE_COLUMN_NAME),
-        source_difference["agg_value"].name(consts.SOURCE_AGG_VALUE),
-        target["table_name"].name(consts.TARGET_TABLE_NAME),
-        target["column_name"].name(consts.TARGET_COLUMN_NAME),
-        target["agg_value"].name(consts.TARGET_AGG_VALUE),
+        source_difference[consts.COMBINER_TABLE_NAME].name(consts.SOURCE_TABLE_NAME),
+        source_difference[consts.COMBINER_COLUMN_NAME].name(consts.SOURCE_COLUMN_NAME),
+        source_difference[consts.COMBINER_AGG_VALUE].name(consts.SOURCE_AGG_VALUE),
+        target[consts.COMBINER_TABLE_NAME].name(consts.TARGET_TABLE_NAME),
+        target[consts.COMBINER_COLUMN_NAME].name(consts.TARGET_COLUMN_NAME),
+        target[consts.COMBINER_AGG_VALUE].name(consts.TARGET_AGG_VALUE),
         group_by_columns,
         source_difference[consts.CONFIG_PRIMARY_KEYS],
         source_difference[consts.NUM_RANDOM_ROWS],

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/config_manager.py

@@ -611,7 +611,7 @@ class ConfigManager(object):
                     self.build_config_calculated_fields(
                         [casefold_source_columns[field.casefold()]],
                         [casefold_target_columns[field.casefold()]],
-                        "rstrip",
+                        consts.CALC_FIELD_RSTRIP,
                         alias,
                         0,
                     )
@@ -812,16 +812,16 @@ class ConfigManager(object):
                 consts.CONFIG_FIELD_ALIAS
             ]
             depth = 1
-            calc_func = "length"
+            calc_func = consts.CALC_FIELD_LENGTH
         elif column_type in ["string", "!string"]:
-            calc_func = "length"
+            calc_func = consts.CALC_FIELD_LENGTH
 
         elif self._is_uuid(column_type, target_column_type):
             calc_func = consts.CONFIG_CAST
             cast_type = consts.CONFIG_CAST_UUID_STRING
 
         elif column_type in ["binary", "!binary"]:
-            calc_func = "byte_length"
+            calc_func = consts.CALC_FIELD_BYTE_LENGTH
 
         elif column_type in ["timestamp", "!timestamp", "date", "!date"]:
             if (
@@ -841,7 +841,7 @@ class ConfigManager(object):
                 ]
                 depth = 1
 
-            calc_func = "epoch_seconds"
+            calc_func = consts.CALC_FIELD_EPOCH_SECONDS
             if agg_type == consts.CONFIG_TYPE_SUM:
                 # It is possible to exceed int64 when summing epoch_seconds therefore cast to string.
                 # See issue 1391 for details.
@@ -1138,7 +1138,7 @@ class ConfigManager(object):
         target_table: "ibis.expr.types.Table",
     ) -> dict:
         """Mutates col_config to contain any overrides. Also returns col_config for convenience."""
-        if col_config["calc_type"] != "cast":
+        if col_config["calc_type"] != consts.CALC_FIELD_CAST:
             return col_config
 
         source_table_schema = {k: v for k, v in source_table.schema().items()}
@@ -1186,13 +1186,19 @@ class ConfigManager(object):
 
     def _get_order_of_operations(self, calc_type: str) -> List[str]:
         """Return order of operations for row validation."""
-        order_of_operations = ["cast", "ifnull", "rstrip"]
+        order_of_operations = [
+            consts.CALC_FIELD_CAST,
+            consts.CALC_FIELD_IFNULL,
+            consts.CALC_FIELD_RSTRIP,
+        ]
         if self.case_insensitive_match():
-            order_of_operations.append("upper")
-        if calc_type == "hash":
-            order_of_operations.extend(["concat", "hash"])
-        elif calc_type == "concat":
-            order_of_operations.append("concat")
+            order_of_operations.append(consts.CALC_FIELD_UPPER)
+        if calc_type == consts.CALC_FIELD_HASH:
+            order_of_operations.extend(
+                [consts.CALC_FIELD_CONCAT, consts.CALC_FIELD_HASH]
+            )
+        elif calc_type == consts.CALC_FIELD_CONCAT:
+            order_of_operations.append(consts.CALC_FIELD_CONCAT)
 
         return order_of_operations
 
@@ -1241,7 +1247,7 @@ class ConfigManager(object):
                 previous_level = [x for x in casefold_source_columns.keys()]
             else:
                 previous_level = [k for k, v in column_aliases.items() if v == i - 1]
-            if calc in ["concat", "hash"]:
+            if calc in [consts.CALC_FIELD_CONCAT, consts.CALC_FIELD_HASH]:
                 col = {}
                 col["source_reference"] = previous_level
                 col["target_reference"] = previous_level

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/consts.py

@@ -143,6 +143,7 @@ PROJECT_ID = "project_id"
 TABLE_ID = "table_id"
 GOOGLE_SERVICE_ACCOUNT_KEY_PATH = "google_service_account_key_path"
 API_ENDPOINT = "api_endpoint"
+STORAGE_API_ENDPOINT = "storage_api_endpoint"
 
 # Result Handler Output Table Fields
 VALIDATION_TYPE = "validation_type"
@@ -181,6 +182,11 @@ FAILED_SOURCE_NOT_IN_TARGET = "failed_rows_present_in_source_not_in_target"
 FAILED_TARGET_NOT_IN_SOURCE = "failed_rows_present_in_target_not_in_source"
 FAILED_PRESENT_IN_BOTH_TABLES = "failed_rows_present_in_both_source_and_target"
 
+# Combiner only constants
+COMBINER_TABLE_NAME = "dvt_table_name"
+COMBINER_COLUMN_NAME = "dvt_column_name"
+COMBINER_AGG_VALUE = "dvt_agg_value"
+
 # SQL Template Formatting
 # TODO: should this be managed in query_builder if that is the only place its used?
 COUNT_STAR = "{count_star}"
@@ -239,3 +245,15 @@ MAX_CONCAT_COLUMNS_DEFAULTS = {
     # Minimizing risk of: [Error 3556] Too many columns defined for this table.
     "teradata": 500,
 }
+
+# CalculatedField expression constants.
+CALC_FIELD_CAST = "cast"
+CALC_FIELD_CONCAT = "concat"
+CALC_FIELD_BYTE_LENGTH = "byte_length"
+CALC_FIELD_EPOCH_SECONDS = "epoch_seconds"
+CALC_FIELD_HASH = "hash"
+CALC_FIELD_IFNULL = "ifnull"
+CALC_FIELD_LENGTH = "length"
+CALC_FIELD_PADDED_CHAR_LENGTH = "padded_char_length"
+CALC_FIELD_RSTRIP = "rstrip"
+CALC_FIELD_UPPER = "upper"

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/query_builder/query_builder.py

@@ -358,6 +358,14 @@ class CalculatedField(object):
             fields,
         )
 
+    @staticmethod
+    def padded_char_length(config, fields):
+        return CalculatedField(
+            ibis.expr.types.StringValue.padded_char_length,
+            config,
+            fields,
+        )
+
     @staticmethod
     def byte_length(config, fields):
         return CalculatedField(
@@ -403,7 +411,7 @@ class CalculatedField(object):
     @staticmethod
     def custom(config, fields):
         """Returns a CalculatedField instance built for any custom ibis expression
-        i.e. 'ibis.expr.api.StringValue.replace'. For a list of supported functions,
+        e.g. 'ibis.expr.api.StringValue.replace'. For a list of supported functions,
         see https://github.com/ibis-project/ibis/blob/1.4.0/ibis/expr/api.py
         Args:
             expr (Str): A custom ibis expression to be used as a calc field

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/result_handlers/bigquery.py

@@ -65,7 +65,7 @@ class BigQueryResultHandler(BaseBackendResultHandler):
                 Explicit credentials to use in case default credentials
                 aren't working properly.
             status_list (list): provided status to filter the results with
-            api_endpoint (str): BigQuery API endpoint (e.g. https://mybq.p.googleapis.com)
+            api_endpoint (str): BigQuery API endpoint (e.g. https://bigquery-mypsc.p.googleapis.com)
             text_format (str, optional):
                 This allows the user to influence the text results written via logger.debug.
                 See: https://github.com/GoogleCloudPlatform/professional-services-data-validator/issues/871

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/data_validation/validation_builder.py

@@ -11,8 +11,9 @@
 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 # See the License for the specific language governing permissions and
 # limitations under the License.
-import logging
 from copy import deepcopy
+import logging
+from typing import TYPE_CHECKING
 
 from data_validation import consts, metadata
 from data_validation.clients import get_max_in_list_size
@@ -25,6 +26,9 @@ from data_validation.query_builder.query_builder import (
     QueryBuilder,
 )
 
+if TYPE_CHECKING:
+    import ibis
+
 
 def list_to_sublists(id_list: list, max_size: int) -> list:
     """Return a list of items as a list of lists based on a max list length of max_size."""
@@ -87,6 +91,23 @@ class ValidationBuilder(object):
         else:
             return FilterField.isin(column_name, in_list)
 
+    def _is_padded_char(
+        self,
+        client: "ibis.backends.base.BaseBackend",
+        raw_column_metadata: dict,
+        column_name: str,
+    ) -> bool:
+        # Clients only need to implement _is_char_type_padded method if they support padded strings.
+        return (
+            (
+                client.is_char_type_padded(raw_column_metadata[column_name])
+                if raw_column_metadata.get(column_name)
+                else False
+            )
+            if hasattr(client, "is_char_type_padded") and raw_column_metadata
+            else False
+        )
+
     def clone(self):
         cloned_builder = ValidationBuilder(self.config_manager)
 
@@ -347,32 +368,63 @@ class ValidationBuilder(object):
             threshold=self.config_manager.threshold,
         )
 
-    def add_calc(self, calc_field):
+    def _get_calc_type(
+        self,
+        calc_field: dict,
+        column_name: str,
+        client: "ibis.backends.base.BaseBackend",
+        raw_data_types: dict,
+    ) -> str:
+        if calc_field[
+            consts.CONFIG_TYPE
+        ] == consts.CALC_FIELD_LENGTH and self._is_padded_char(
+            client,
+            raw_data_types,
+            column_name,
+        ):
+            calc_type = consts.CALC_FIELD_PADDED_CHAR_LENGTH
+        else:
+            calc_type = calc_field[consts.CONFIG_TYPE]
+        # Check if valid calc field and return correct object.
+        if not hasattr(CalculatedField, calc_type):
+            raise Exception("Unknown Calculation Type: {}".format(calc_type))
+        return calc_type
+
+    def add_calc(self, calc_field: dict):
         """Add CalculatedField to Queries
 
         Args:
             calc_field (Dict): An object with source, target, and cast info
         """
-        # prepare source and target payloads
+        # Prepare source and target payloads
         source_config = deepcopy(calc_field)
         source_fields = calc_field[consts.CONFIG_CALCULATED_SOURCE_COLUMNS]
         target_config = deepcopy(calc_field)
         target_fields = calc_field[consts.CONFIG_CALCULATED_TARGET_COLUMNS]
-        # grab calc field metadata
+        # Grab calc field metadata
         alias = calc_field[consts.CONFIG_FIELD_ALIAS]
-        calc_type = calc_field[consts.CONFIG_TYPE]
-        # check if valid calc field and return correct object
-        if not hasattr(CalculatedField, calc_type):
-            raise Exception("Unknown Calculation Type: {}".format(calc_type))
-        source_field = getattr(CalculatedField, calc_type)(
+        source_calc_type = self._get_calc_type(
+            calc_field,
+            source_fields[0],
+            self.source_client,
+            self.config_manager.get_source_raw_data_types(),
+        )
+        target_calc_type = self._get_calc_type(
+            calc_field,
+            target_fields[0],
+            self.target_client,
+            self.config_manager.get_target_raw_data_types(),
+        )
+
+        source_field = getattr(CalculatedField, source_calc_type)(
             config=source_config, fields=source_fields
         )
-        target_field = getattr(CalculatedField, calc_type)(
+        target_field = getattr(CalculatedField, target_calc_type)(
             config=target_config, fields=target_fields
         )
         self.source_builder.add_calculated_field(source_field)
         self.target_builder.add_calculated_field(target_field)
-        # register calc field under alias
+        # Register calc field under alias
         self.calculated_aliases[alias] = calc_field
 
     def get_source_query(self):

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0/google_pso_data_validator.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: google-pso-data-validator
-Version: 7.6.0
+Version: 7.7.0
 Summary: A package to enable easy data validation
 Author: PSO DVT Engineering team
 Author-email: data-validator-eng@google.com
@@ -125,8 +125,10 @@ please review the [Connections](https://github.com/GoogleCloudPlatform/professio
 ### Running Validations
 
 The CLI is the main interface to use this tool and it has several different
-commands which can be used to create and run validations. Below are the command
-syntax and options for running validations.
+commands which can be used to create and run validations. DVT is designed to run in
+an environment connected to GCP services, specifically, BigQuery, GCS and Secret manager.
+If DVT is being run on-premises or in an environment with restricted access to GCP services, see
+[running DVT  at on-prem](#running-dvt-at-on-prem). Below are the command syntax and options for running validations.
 
 Alternatives to running DVT in the CLI include deploying DVT to Cloud Run, Cloud Functions, or Airflow
 ([Examples Here](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples)). See the [Validation Logic](https://github.com/GoogleCloudPlatform/professional-services-data-validator#validation-logic) section
@@ -584,6 +586,11 @@ For example, this flag can be used as follows:
     "target_query": "SELECT `hash__all`, `station_id`\nFROM ..."
 }
 ```
+#### Running DVT at on-prem
+On-premises environments can have limited access to GCP services. DVT supports using BigQuery for storing validation results, GCS for storage and
+the Secret Manager for secrets. You may also use BigQuery and Spanner as a source or target for validation. Service
+APIs (i.e. bigquery.googleapis.com) may not always be accessible due to firewall restrictions. Work with your network
+adminstrator to identify the way to access these services. They may set up a [Private Service Connect Endpoint](https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints). DVT supports accessing source and target tables in Spanner and BigQuery via endpoints set up in your network. Connection Parameters for [Spanner](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-spanner) and [BigQuery](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/connections.md#google-bigquery) outline regarding how to specify endpoints.
 
 ### Running DVT with YAML Configuration Files
 

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/google_pso_data_validator.egg-info/SOURCES.txt

@@ -43,8 +43,8 @@ google_pso_data_validator.egg-info/top_level.txt
 third_party/ibis/ibis_addon/__init__.py
 third_party/ibis/ibis_addon/api.py
 third_party/ibis/ibis_addon/operations.py
-third_party/ibis/ibis_biquery/__init__.py
-third_party/ibis/ibis_biquery/api.py
+third_party/ibis/ibis_bigquery/__init__.py
+third_party/ibis/ibis_bigquery/api.py
 third_party/ibis/ibis_cloud_spanner/__init__.py
 third_party/ibis/ibis_cloud_spanner/api.py
 third_party/ibis/ibis_cloud_spanner/client.py

{google_pso_data_validator-7.6.0 → google_pso_data_validator-7.7.0}/setup.py

@@ -18,7 +18,7 @@ import setuptools
 
 name = "google-pso-data-validator"
 description = "A package to enable easy data validation"
-version = "7.6.0"
+version = "7.7.0"
 release_status = "Development Status :: 4 - Beta"
 
 with open("README.md", "r") as fh: