google-pso-data-validator 6.1.0__tar.gz → 6.3.0__tar.gz

{google_pso_data_validator-6.1.0/google_pso_data_validator.egg-info → google_pso_data_validator-6.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: google-pso-data-validator
-Version: 6.1.0
+Version: 6.3.0
 Summary: A package to enable easy data validation
 Author: PSO DVT Engineering team
 Author-email: data-validator-eng@google.com
@@ -14,31 +14,26 @@ Classifier: Operating System :: OS Independent
 Requires-Python: >=3.8
 Description-Content-Type: text/markdown
 License-File: LICENSE
-Requires-Dist: attrs==23.1.0
-Requires-Dist: grpcio==1.53.2
-Requires-Dist: lazy-object-proxy==1.9.0
-Requires-Dist: marshmallow==3.19.0
-Requires-Dist: fsspec>=2022.8.2
-Requires-Dist: google-api-python-client==2.91.0
+Requires-Dist: Flask>=2.2
+Requires-Dist: fsspec>=2024.9.0
+Requires-Dist: google-api-python-client>=2.144.0
+Requires-Dist: google-cloud-bigquery>=3.25.0
+Requires-Dist: google-cloud-bigquery-storage>=2.26.0
+Requires-Dist: google-cloud-secret-manager>=2.20.2
+Requires-Dist: google-cloud-spanner>=3.49.1
+Requires-Dist: google-cloud-storage>=2.18.2
 Requires-Dist: ibis-framework==5.1.0
-Requires-Dist: impyla==0.18.0
-Requires-Dist: SQLAlchemy==1.4.49
-Requires-Dist: PyMySQL==1.1.1
-Requires-Dist: psycopg2-binary==2.9.6
-Requires-Dist: PyYAML==6.0
-Requires-Dist: pandas==1.5.3
-Requires-Dist: proto-plus==1.22.3
+Requires-Dist: impyla>=0.19.0
+Requires-Dist: jellyfish>=1.1.0
+Requires-Dist: pandas==2.0.3
+Requires-Dist: parsy>=2.1
+Requires-Dist: psycopg2-binary>=2.9.9
 Requires-Dist: pyarrow==14.0.1
-Requires-Dist: pydata-google-auth==1.8.0
-Requires-Dist: google-cloud-bigquery==3.11.3
-Requires-Dist: google-cloud-bigquery-storage==2.20.0
-Requires-Dist: google-cloud-spanner==3.36.0
-Requires-Dist: google-cloud-storage==2.10.0
-Requires-Dist: jellyfish==1.0.0
-Requires-Dist: tabulate==0.9.0
-Requires-Dist: Flask==2.3.2
-Requires-Dist: parsy==2.1
-Requires-Dist: google-cloud-secret-manager>=2.16.0
+Requires-Dist: pydata-google-auth>=1.8.2
+Requires-Dist: PyMySQL>=1.1.1
+Requires-Dist: PyYAML>=6.0.2
+Requires-Dist: SQLAlchemy==1.4.49
+Requires-Dist: tabulate>=0.9.0
 Provides-Extra: apache-airflow
 Requires-Dist: 1.10.11; extra == "apache-airflow"
 Provides-Extra: pyspark
@@ -133,7 +128,12 @@ over all columns ('*') will only run over numeric columns, unless the
 `--wildcard-include-string-len` or `--wildcard-include-timestamp` flags are present.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate column
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate column
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -141,7 +141,7 @@ data-validation (--verbose or -v) (--log-level or -ll) validate column
                         Target connection details
                         See: *Connections* section for each data source
   --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
-                        Comma separated list of tables in the form schema.table=target_schema.target_table
+                        Comma separated list of tables in the form schema.table=target_schema.target_table. Or shorthand schema.* for all tables.
                         Target schema name and table name are optional.
                         i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
   [--grouped-columns or -gc GROUPED_COLUMNS]
@@ -182,6 +182,7 @@ data-validation (--verbose or -v) (--log-level or -ll) validate column
                         Format for stdout output. Supported formats are (text, csv, json, table). Defaults to table.
   [--filter-status or -fs STATUSES_LIST]
                         Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
+
 ```
 
 The default aggregation type is a 'COUNT *', which will run in addition to the validations you specify. To remove this default,
@@ -212,7 +213,12 @@ Under the hood, row validation uses
 apply functions such as IFNULL() or RTRIM(). These can be edited in the YAML or JSON config file to customize your row validation.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate row
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate row
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -261,17 +267,21 @@ data-validation (--verbose or -v) (--log-level or -ll) validate row
   [--case-insensitive-match, -cim]
                         Performs a case insensitive match by adding an UPPER() before comparison.
 ```
-#### Generate Table Partitions for Large Table Row Validations
+#### Generate Partitions for Large Row Validations
 
-When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
+When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating partitions in order to perform row validations on large dataset (table or custom-query) to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
 
-The command generates and stores multiple YAML validations each representing a chunk of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in one YAML file. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
+The command generates and stores multiple YAML validations each representing a chunk of the large dataset using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in YAML files. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
 
-The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions, ```partition-key``` is no longer supported.
+The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. You can specify tables that are being validated or the source and target custom query. A parameter used in earlier versions, ```partition-key``` is no longer supported.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
-
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  generate-table-partitions
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -282,6 +292,17 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         Comma separated list of tables in the form schema.table=target_schema.target_table
                         Target schema name and table name are optional.
                         i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query SOURCE_QUERY, -sq SOURCE_QUERY
+                        Source sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query-file  SOURCE_QUERY_FILE, -sqf SOURCE_QUERY_FILE
+                        File containing the source sql command. Supports GCS and local paths.
+  --target-query TARGET_QUERY, -tq TARGET_QUERY
+                        Target sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --target-query-file TARGET_QUERY_FILE, -tqf TARGET_QUERY_FILE
+                        File containing the target sql command. Supports GCS and local paths.
   --primary-keys PRIMARY_KEYS, -pk PRIMARY_KEYS
                         Comma separated list of primary key columns 'col_a,col_b'.  See *Primary Keys* section
   --comparison-fields or -comp-fields FIELDS
@@ -293,7 +314,8 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         Directory Path to store YAML Config Files
                         GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                         Local: Provide a relative path of the target directory. Eg: `partitions_dir`
-  --partition-num INT, -pn INT 
+                        If invoked with -tbls parameter, the validations are stored in a directory named <schema>.<table>, otherwise the directory is named `custom.<random_string>`
+  --partition-num INT, -pn INT
                         Number of partitions into which the table should be split, e.g. 1000 or 10000
                         In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
   [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
@@ -301,7 +323,7 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         See: *Validation Reports* section
   [--service-account or -sa PATH_TO_SA_KEY]
                         Service account to use for BigQuery result handler output.
-  [--parts-per-file INT], [-ppf INT] 
+  [--parts-per-file INT], [-ppf INT]
                         Number of partitions in a yaml file, default value 1.
   [--filters SOURCE_FILTER:TARGET_FILTER]
                         Colon separated string values of source and target filters.
@@ -326,7 +348,12 @@ types between source and target.
 Note: An exclamation point before a data type (`!string`) signifies the column is non-nullable or required.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate schema
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate schema
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -334,7 +361,7 @@ data-validation (--verbose or -v) (--log-level or -ll) validate schema
                         Target connection details
                         See: *Connections* section for each data source
   --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
-                        Comma separated list of tables in the form schema.table=target_schema.target_table
+                        Comma separated list of tables in the form schema.table=target_schema.target_table. Or shorthand schema.* for all tables.
                         Target schema name and table name are optional.
                         e.g.: 'bigquery-public-data.new_york_citibike.citibike_trips'
   [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
@@ -367,7 +394,12 @@ data-validation (--verbose or -v) (--log-level or -ll) validate schema
 Below is the command syntax for custom query column validations.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate custom-query column
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate custom-query column
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -429,7 +461,12 @@ in the SELECT statement of both source_query.sql and target_query.sql.  See *Pri
 Below is the command syntax for custom query row validations.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate custom-query row
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate custom-query row
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -487,7 +524,12 @@ The `validate` command takes a `--dry-run` command line flag that prints source
 and target SQL to stdout as JSON in lieu of performing a validation:
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate
   [--dry-run or -dr]    Prints source and target SQL to stdout in lieu of performing a validation.
 ```
 
@@ -518,7 +560,12 @@ GCS and local paths.
 You can use the `data-validation configs` command to run and view YAMLs.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) configs run
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  configs run
   [--config-file or -c CONFIG_FILE]
                         Path to YAML config file to run. Supports local and GCS paths.
   [--config-dir or -cdir CONFIG_DIR]
@@ -546,10 +593,10 @@ View the complete YAML file for a Grouped Column validation on the
 
 ### Scaling DVT
 
-You can scale DVT for large table validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
+You can scale DVT for large validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
 
 
-We recommend first generating table partitions with the `generate-table-partitions` command for your large tables. Then, use Cloud Run or GKE to distribute validating each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
+We recommend first generating partitions with the `generate-table-partitions` command for your large datasets (tables or queries). Then, use Cloud Run or GKE to distribute the validation of each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
 
 When running DVT in a distributed fashion, both the `--kube-completions` and `--config-dir` flags are required. The `--kube-completions` flag specifies that the validation is being run in indexed completion mode in Kubernetes or as multiple independent tasks in Cloud Run. If the `-kc` option is used and you are not running in indexed mode, you will receive a warning and the container will process all the validations sequentially. If the `-kc` option is used and a config directory is not provided (i.e. a `--config-file` is provided instead), a warning is issued.
 
@@ -684,10 +731,15 @@ resultset that breaks down the count of rows per calendar date.
 Row level validations can involve either a hash/checksum, concat, or comparison fields.
 A hash validation (`--hash '*'`) will first sanitize the data with the following
 operations on all or selected columns: CAST to string, IFNULL replace with a default
-replacement string, RSTRIP, and UPPER. Then, it will CONCAT() the results
-and run a SHA256() hash and compare the source and target results. Since each row will
-be returned in the result set, it is recommended to utilize the `--use-random-row` feature
-to validate a subset of the table.
+replacement string and RSTRIP. Then, it will CONCAT() the results
+and run a SHA256() hash and compare the source and target results.
+
+When there are data type mismatches for columns, for example dates compared to timestamps and
+booleans compared with numeric columns, you may see other expressions in SQL statements which
+ensure that consistent values are used to build comparison values.
+
+Since each row will be returned in the result set if is recommended recommended to validate a
+subset of the table. The `--filters` and `--use-random-row` options can be used for this purpose.
 
 Please note that SHA256 is not a supported function on Teradata systems. If you wish to perform
 this comparison on Teradata you will need to [deploy a UDF to perform the conversion](https://github.com/akuroda/teradata-udf-sha2/blob/master/src/sha256.c).

google_pso_data_validator-6.1.0/PKG-INFO → google_pso_data_validator-6.3.0/README.md

@@ -1,49 +1,3 @@
-Metadata-Version: 2.1
-Name: google-pso-data-validator
-Version: 6.1.0
-Summary: A package to enable easy data validation
-Author: PSO DVT Engineering team
-Author-email: data-validator-eng@google.com
-Classifier: Development Status :: 4 - Beta
-Classifier: License :: OSI Approved :: Apache Software License
-Classifier: Programming Language :: Python
-Classifier: Programming Language :: Python :: 3.8
-Classifier: Programming Language :: Python :: 3.9
-Classifier: Programming Language :: Python :: 3.10
-Classifier: Operating System :: OS Independent
-Requires-Python: >=3.8
-Description-Content-Type: text/markdown
-License-File: LICENSE
-Requires-Dist: attrs==23.1.0
-Requires-Dist: grpcio==1.53.2
-Requires-Dist: lazy-object-proxy==1.9.0
-Requires-Dist: marshmallow==3.19.0
-Requires-Dist: fsspec>=2022.8.2
-Requires-Dist: google-api-python-client==2.91.0
-Requires-Dist: ibis-framework==5.1.0
-Requires-Dist: impyla==0.18.0
-Requires-Dist: SQLAlchemy==1.4.49
-Requires-Dist: PyMySQL==1.1.1
-Requires-Dist: psycopg2-binary==2.9.6
-Requires-Dist: PyYAML==6.0
-Requires-Dist: pandas==1.5.3
-Requires-Dist: proto-plus==1.22.3
-Requires-Dist: pyarrow==14.0.1
-Requires-Dist: pydata-google-auth==1.8.0
-Requires-Dist: google-cloud-bigquery==3.11.3
-Requires-Dist: google-cloud-bigquery-storage==2.20.0
-Requires-Dist: google-cloud-spanner==3.36.0
-Requires-Dist: google-cloud-storage==2.10.0
-Requires-Dist: jellyfish==1.0.0
-Requires-Dist: tabulate==0.9.0
-Requires-Dist: Flask==2.3.2
-Requires-Dist: parsy==2.1
-Requires-Dist: google-cloud-secret-manager>=2.16.0
-Provides-Extra: apache-airflow
-Requires-Dist: 1.10.11; extra == "apache-airflow"
-Provides-Extra: pyspark
-Requires-Dist: 3.0.0; extra == "pyspark"
-
 # Data Validation Tool
 
 The Data Validation Tool is an open sourced Python CLI tool based on the
@@ -133,7 +87,12 @@ over all columns ('*') will only run over numeric columns, unless the
 `--wildcard-include-string-len` or `--wildcard-include-timestamp` flags are present.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate column
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate column
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -141,7 +100,7 @@ data-validation (--verbose or -v) (--log-level or -ll) validate column
                         Target connection details
                         See: *Connections* section for each data source
   --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
-                        Comma separated list of tables in the form schema.table=target_schema.target_table
+                        Comma separated list of tables in the form schema.table=target_schema.target_table. Or shorthand schema.* for all tables.
                         Target schema name and table name are optional.
                         i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
   [--grouped-columns or -gc GROUPED_COLUMNS]
@@ -182,6 +141,7 @@ data-validation (--verbose or -v) (--log-level or -ll) validate column
                         Format for stdout output. Supported formats are (text, csv, json, table). Defaults to table.
   [--filter-status or -fs STATUSES_LIST]
                         Comma separated list of statuses to filter the validation results. Supported statuses are (success, fail). If no list is provided, all statuses are returned.
+
 ```
 
 The default aggregation type is a 'COUNT *', which will run in addition to the validations you specify. To remove this default,
@@ -212,7 +172,12 @@ Under the hood, row validation uses
 apply functions such as IFNULL() or RTRIM(). These can be edited in the YAML or JSON config file to customize your row validation.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate row
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate row
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -261,17 +226,21 @@ data-validation (--verbose or -v) (--log-level or -ll) validate row
   [--case-insensitive-match, -cim]
                         Performs a case insensitive match by adding an UPPER() before comparison.
 ```
-#### Generate Table Partitions for Large Table Row Validations
+#### Generate Partitions for Large Row Validations
 
-When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating table partitions in order to perform row validations on large tables to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
+When performing row validations, Data Validation Tool brings each row into memory and can run into MemoryError. Below is the command syntax for generating partitions in order to perform row validations on large dataset (table or custom-query) to alleviate MemoryError. Each partition contains a range of primary key(s) and the ranges of keys across partitions are distinct. The partitions have nearly equal number of rows. See *Primary Keys* section
 
-The command generates and stores multiple YAML validations each representing a chunk of the large table using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in one YAML file. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
+The command generates and stores multiple YAML validations each representing a chunk of the large dataset using filters (`WHERE primary_key(s) >= X AND primary_key(s) < Y`) in YAML files. The parameter parts-per-file, specifies the number of validations in one YAML file. Each yaml file will have parts-per-file validations in it - except the last one which will contain the remaining partitions (i.e. parts-per-file may not divide partition-num evenly). You can then run the validations in the directory serially (or in parallel in multiple containers, VMs) with the `data-validation configs run --config-dir PATH` command as described [here](https://github.com/GoogleCloudPlatform/professional-services-data-validator#yaml-configuration-files).
 
-The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. A parameter used in earlier versions, ```partition-key``` is no longer supported.
+The command takes the same parameters as required for `Row Validation` *plus* a few parameters to support partitioning. Single and multiple primary keys are supported and keys can be of any indexable type, except for date and timestamp type. You can specify tables that are being validated or the source and target custom query. A parameter used in earlier versions, ```partition-key``` is no longer supported.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
-
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  generate-table-partitions
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -282,6 +251,17 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         Comma separated list of tables in the form schema.table=target_schema.target_table
                         Target schema name and table name are optional.
                         i.e 'bigquery-public-data.new_york_citibike.citibike_trips'
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query SOURCE_QUERY, -sq SOURCE_QUERY
+                        Source sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --source-query-file  SOURCE_QUERY_FILE, -sqf SOURCE_QUERY_FILE
+                        File containing the source sql command. Supports GCS and local paths.
+  --target-query TARGET_QUERY, -tq TARGET_QUERY
+                        Target sql query
+                        Either --tables-list or --source-query (or file) and --target-query (or file) must be provided
+  --target-query-file TARGET_QUERY_FILE, -tqf TARGET_QUERY_FILE
+                        File containing the target sql command. Supports GCS and local paths.
   --primary-keys PRIMARY_KEYS, -pk PRIMARY_KEYS
                         Comma separated list of primary key columns 'col_a,col_b'.  See *Primary Keys* section
   --comparison-fields or -comp-fields FIELDS
@@ -293,7 +273,8 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         Directory Path to store YAML Config Files
                         GCS: Provide a full gs:// path of the target directory. Eg: `gs://<BUCKET>/partitions_dir`
                         Local: Provide a relative path of the target directory. Eg: `partitions_dir`
-  --partition-num INT, -pn INT 
+                        If invoked with -tbls parameter, the validations are stored in a directory named <schema>.<table>, otherwise the directory is named `custom.<random_string>`
+  --partition-num INT, -pn INT
                         Number of partitions into which the table should be split, e.g. 1000 or 10000
                         In case this value exceeds the row count of the source/target table, it will be decreased to max(source_row_count, target_row_count)
   [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
@@ -301,7 +282,7 @@ data-validation (--verbose or -v) (--log-level or -ll) generate-table-partitions
                         See: *Validation Reports* section
   [--service-account or -sa PATH_TO_SA_KEY]
                         Service account to use for BigQuery result handler output.
-  [--parts-per-file INT], [-ppf INT] 
+  [--parts-per-file INT], [-ppf INT]
                         Number of partitions in a yaml file, default value 1.
   [--filters SOURCE_FILTER:TARGET_FILTER]
                         Colon separated string values of source and target filters.
@@ -326,7 +307,12 @@ types between source and target.
 Note: An exclamation point before a data type (`!string`) signifies the column is non-nullable or required.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate schema
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate schema
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -334,7 +320,7 @@ data-validation (--verbose or -v) (--log-level or -ll) validate schema
                         Target connection details
                         See: *Connections* section for each data source
   --tables-list or -tbls SOURCE_SCHEMA.SOURCE_TABLE=TARGET_SCHEMA.TARGET_TABLE
-                        Comma separated list of tables in the form schema.table=target_schema.target_table
+                        Comma separated list of tables in the form schema.table=target_schema.target_table. Or shorthand schema.* for all tables.
                         Target schema name and table name are optional.
                         e.g.: 'bigquery-public-data.new_york_citibike.citibike_trips'
   [--bq-result-handler or -bqrh PROJECT_ID.DATASET.TABLE]
@@ -367,7 +353,12 @@ data-validation (--verbose or -v) (--log-level or -ll) validate schema
 Below is the command syntax for custom query column validations.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate custom-query column
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate custom-query column
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -429,7 +420,12 @@ in the SELECT statement of both source_query.sql and target_query.sql.  See *Pri
 Below is the command syntax for custom query row validations.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate custom-query row
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate custom-query row
   --source-conn or -sc SOURCE_CONN
                         Source connection details
                         See: *Data Source Configurations* section for each data source
@@ -487,7 +483,12 @@ The `validate` command takes a `--dry-run` command line flag that prints source
 and target SQL to stdout as JSON in lieu of performing a validation:
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) validate
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  validate
   [--dry-run or -dr]    Prints source and target SQL to stdout in lieu of performing a validation.
 ```
 
@@ -518,7 +519,12 @@ GCS and local paths.
 You can use the `data-validation configs` command to run and view YAMLs.
 
 ```
-data-validation (--verbose or -v) (--log-level or -ll) configs run
+data-validation
+  [--verbose or -v ]
+                        Verbose logging
+  [--log-level or -ll]
+                        Log Level to be assigned. Supported levels are (DEBUG,INFO,WARNING,ERROR,CRITICAL). Defaults to INFO.
+  configs run
   [--config-file or -c CONFIG_FILE]
                         Path to YAML config file to run. Supports local and GCS paths.
   [--config-dir or -cdir CONFIG_DIR]
@@ -546,10 +552,10 @@ View the complete YAML file for a Grouped Column validation on the
 
 ### Scaling DVT
 
-You can scale DVT for large table validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
+You can scale DVT for large validations by running the tool in a distributed manner. To optimize the validation speed for large tables, you can use GKE Jobs ([Google Kubernetes Jobs](https://cloud.google.com/kubernetes-engine/docs/how-to/deploying-workloads-overview#batch_jobs)) or [Cloud Run Jobs](https://cloud.google.com/run/docs/create-jobs). If you are not familiar with Kubernetes or Cloud Run Jobs, see [Scaling DVT with Distributed Jobs](https://github.com/GoogleCloudPlatform/professional-services-data-validator/blob/develop/docs/internal/distributed_jobs.md) for a detailed overview.
 
 
-We recommend first generating table partitions with the `generate-table-partitions` command for your large tables. Then, use Cloud Run or GKE to distribute validating each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
+We recommend first generating partitions with the `generate-table-partitions` command for your large datasets (tables or queries). Then, use Cloud Run or GKE to distribute the validation of each chunk in parallel. See the [Cloud Run Jobs Quickstart sample](https://github.com/GoogleCloudPlatform/professional-services-data-validator/tree/develop/samples/cloud_run_jobs) to get started.
 
 When running DVT in a distributed fashion, both the `--kube-completions` and `--config-dir` flags are required. The `--kube-completions` flag specifies that the validation is being run in indexed completion mode in Kubernetes or as multiple independent tasks in Cloud Run. If the `-kc` option is used and you are not running in indexed mode, you will receive a warning and the container will process all the validations sequentially. If the `-kc` option is used and a config directory is not provided (i.e. a `--config-file` is provided instead), a warning is issued.
 
@@ -684,10 +690,15 @@ resultset that breaks down the count of rows per calendar date.
 Row level validations can involve either a hash/checksum, concat, or comparison fields.
 A hash validation (`--hash '*'`) will first sanitize the data with the following
 operations on all or selected columns: CAST to string, IFNULL replace with a default
-replacement string, RSTRIP, and UPPER. Then, it will CONCAT() the results
-and run a SHA256() hash and compare the source and target results. Since each row will
-be returned in the result set, it is recommended to utilize the `--use-random-row` feature
-to validate a subset of the table.
+replacement string and RSTRIP. Then, it will CONCAT() the results
+and run a SHA256() hash and compare the source and target results.
+
+When there are data type mismatches for columns, for example dates compared to timestamps and
+booleans compared with numeric columns, you may see other expressions in SQL statements which
+ensure that consistent values are used to build comparison values.
+
+Since each row will be returned in the result set if is recommended recommended to validate a
+subset of the table. The `--filters` and `--use-random-row` options can be used for this purpose.
 
 Please note that SHA256 is not a supported function on Teradata systems. If you wish to perform
 this comparison on Teradata you will need to [deploy a UDF to perform the conversion](https://github.com/akuroda/teradata-udf-sha2/blob/master/src/sha256.c).