table-validator 0.1.9__tar.gz → 0.1.11__tar.gz

{table_validator-0.1.9/table_validator.egg-info → table_validator-0.1.11}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: table-validator
-Version: 0.1.9
+Version: 0.1.11
 Summary: CLI tool for validating data migrations between Azure (Blob Storage / SQL Database) and Databricks Delta Lake catalogs
 License: MIT
 Requires-Python: >=3.9
@@ -15,6 +15,7 @@ Requires-Dist: databricks-sql-connector>=3.0
 Requires-Dist: pandas>=2.0
 Requires-Dist: numpy>=1.24
 Requires-Dist: azure-storage-blob>=12.14
+Requires-Dist: azure-identity>=1.15
 Requires-Dist: openpyxl>=3.1
 Requires-Dist: pyarrow>=14.0
 Requires-Dist: PyYAML>=6.0
@@ -23,6 +24,8 @@ Provides-Extra: dev
 Requires-Dist: pytest>=7.0; extra == "dev"
 Requires-Dist: ruff>=0.4; extra == "dev"
 Requires-Dist: black>=24.0; extra == "dev"
+Provides-Extra: spark
+Requires-Dist: pyspark>=3.4; extra == "spark"
 Dynamic: license-file
 
 # table-validator
@@ -38,6 +41,7 @@ Currently supported sources (target is always Databricks):
 - Databricks catalog → Databricks catalog
 - Azure Blob Storage (CSV / Excel / Parquet) → Databricks table
 - Azure SQL Database → Databricks catalog
+- Azure Synapse SQL pool (dedicated or serverless) → Databricks catalog
 
 More source platforms can be added behind the same connector/validator
 interfaces described below.
@@ -80,10 +84,11 @@ Interactive wizard that walks you through:
 
 1. Azure Storage account + container (optional, skip if you don't have a Blob source) and account key
 2. Azure SQL server + database (optional, skip if you don't have a SQL source) and username/password
-3. Databricks workspace URL, SQL Warehouse HTTP path, and personal access token
-4. Source table (catalog / schema / table)
-5. Target table (catalog / schema / table)
-6. Which validations to run (catalog / schema / column / row)
+3. Azure Synapse SQL pool endpoint + database (optional, skip if you don't have a Synapse source), then either a SQL login (username/password) or a Microsoft Entra ID service principal (tenant ID + client ID + client secret)
+4. Databricks workspace URL, SQL Warehouse HTTP path, and personal access token
+5. Source table (catalog / schema / table)
+6. Target table (catalog / schema / table)
+7. Which validations to run (catalog / schema / column / row)
 
 ```bash
 tablevalidator validate
@@ -104,6 +109,36 @@ tablevalidator open
 
 Opens the most recently generated report in your default spreadsheet app.
 
+## Notebook usage (zero config, inside Databricks)
+
+From inside a Databricks notebook cell, `validate_tables()` compares two
+tables using the notebook's own already-authenticated Spark session - no
+workspace URL, personal access token, or SQL Warehouse HTTP path to set
+up. This is a second, independent way to use the package alongside the
+CLI above (pick whichever fits: the CLI for a scheduled/scripted run
+against a SQL Warehouse, this for an ad-hoc in-notebook check) - both run
+the exact same full-depth comparison engine.
+
+```python
+%pip install table-validator
+
+from table_validator import validate_tables
+
+result = validate_tables(
+    "catalog1.schema1.table1",
+    "catalog1.schema1.table2",
+)
+print(result)
+```
+
+Optional keyword arguments: `primary_key` (a real key for row-level
+comparison instead of the ROW_NUMBER() fallback), `ignore_columns`,
+`only_columns`, `column_map` (for a renamed column). Outside a Databricks
+notebook (e.g. local development against a real Spark session), install
+the `spark` extra: `pip install "table-validator[spark]"` - inside an
+actual Databricks notebook, pyspark and an active session already exist,
+so the bare `%pip install table-validator` above is sufficient.
+
 ## Quickstart (Python API)
 
 Everything the CLI does is available as a library, built from the same
@@ -158,6 +193,9 @@ Other public entry points exported from `table_validator`:
 - `ValidatorConfig`, `default_config`, `save_config`, `require_config`,
   `ConfigNotFoundError` — the same config load/save layer the CLI wizard
   uses, if you want to construct or persist configuration programmatically.
+- `validate_tables` / `SparkConnector` — the notebook-native entry point
+  described above (lazily imported: importing `table_validator` itself
+  never requires `pyspark` to be installed).
 
 ## Where config and secrets are stored
 

{table_validator-0.1.9 → table_validator-0.1.11}/README.md

@@ -11,6 +11,7 @@ Currently supported sources (target is always Databricks):
 - Databricks catalog → Databricks catalog
 - Azure Blob Storage (CSV / Excel / Parquet) → Databricks table
 - Azure SQL Database → Databricks catalog
+- Azure Synapse SQL pool (dedicated or serverless) → Databricks catalog
 
 More source platforms can be added behind the same connector/validator
 interfaces described below.
@@ -53,10 +54,11 @@ Interactive wizard that walks you through:
 
 1. Azure Storage account + container (optional, skip if you don't have a Blob source) and account key
 2. Azure SQL server + database (optional, skip if you don't have a SQL source) and username/password
-3. Databricks workspace URL, SQL Warehouse HTTP path, and personal access token
-4. Source table (catalog / schema / table)
-5. Target table (catalog / schema / table)
-6. Which validations to run (catalog / schema / column / row)
+3. Azure Synapse SQL pool endpoint + database (optional, skip if you don't have a Synapse source), then either a SQL login (username/password) or a Microsoft Entra ID service principal (tenant ID + client ID + client secret)
+4. Databricks workspace URL, SQL Warehouse HTTP path, and personal access token
+5. Source table (catalog / schema / table)
+6. Target table (catalog / schema / table)
+7. Which validations to run (catalog / schema / column / row)
 
 ```bash
 tablevalidator validate
@@ -77,6 +79,36 @@ tablevalidator open
 
 Opens the most recently generated report in your default spreadsheet app.
 
+## Notebook usage (zero config, inside Databricks)
+
+From inside a Databricks notebook cell, `validate_tables()` compares two
+tables using the notebook's own already-authenticated Spark session - no
+workspace URL, personal access token, or SQL Warehouse HTTP path to set
+up. This is a second, independent way to use the package alongside the
+CLI above (pick whichever fits: the CLI for a scheduled/scripted run
+against a SQL Warehouse, this for an ad-hoc in-notebook check) - both run
+the exact same full-depth comparison engine.
+
+```python
+%pip install table-validator
+
+from table_validator import validate_tables
+
+result = validate_tables(
+    "catalog1.schema1.table1",
+    "catalog1.schema1.table2",
+)
+print(result)
+```
+
+Optional keyword arguments: `primary_key` (a real key for row-level
+comparison instead of the ROW_NUMBER() fallback), `ignore_columns`,
+`only_columns`, `column_map` (for a renamed column). Outside a Databricks
+notebook (e.g. local development against a real Spark session), install
+the `spark` extra: `pip install "table-validator[spark]"` - inside an
+actual Databricks notebook, pyspark and an active session already exist,
+so the bare `%pip install table-validator` above is sufficient.
+
 ## Quickstart (Python API)
 
 Everything the CLI does is available as a library, built from the same
@@ -131,6 +163,9 @@ Other public entry points exported from `table_validator`:
 - `ValidatorConfig`, `default_config`, `save_config`, `require_config`,
   `ConfigNotFoundError` — the same config load/save layer the CLI wizard
   uses, if you want to construct or persist configuration programmatically.
+- `validate_tables` / `SparkConnector` — the notebook-native entry point
+  described above (lazily imported: importing `table_validator` itself
+  never requires `pyspark` to be installed).
 
 ## Where config and secrets are stored
 

{table_validator-0.1.9 → table_validator-0.1.11}/pyproject.toml

@@ -19,6 +19,7 @@ dependencies = [
     "pandas>=2.0",
     "numpy>=1.24",
     "azure-storage-blob>=12.14",
+    "azure-identity>=1.15",
     "openpyxl>=3.1",
     "pyarrow>=14.0",
     "PyYAML>=6.0",
@@ -31,6 +32,13 @@ dev = [
     "ruff>=0.4",
     "black>=24.0",
 ]
+# Only needed to construct SparkConnector/call validate_tables() OUTSIDE a
+# real Databricks notebook (e.g. local dev/testing) - inside an actual
+# notebook, pyspark and an active SparkSession already exist ambiently, so
+# a bare `pip install table-validator` is sufficient there.
+spark = [
+    "pyspark>=3.4",
+]
 
 [project.scripts]
 tablevalidator = "table_validator.cli.main:app"

{table_validator-0.1.9 → table_validator-0.1.11}/table_validator/__init__.py

@@ -43,4 +43,31 @@ __all__ = [
     # Core request/response models
     "CatalogValidationRequest",
     "CatalogValidationResponse",
+    # Notebook-native API (lazily loaded - see __getattr__ below; pyspark
+    # is only imported when one of these two names is actually accessed).
+    "SparkConnector",
+    "validate_tables",
 ]
+
+
+def __getattr__(name: str):
+    """
+    PEP 562 module-level lazy attribute access, for the two names above
+    that transitively depend on pyspark. pyspark is NOT a hard dependency
+    of this package (CLI-only users never install it) - deferring the
+    import to actual first-access time, rather than importing eagerly
+    like everything else in this file, keeps `import table_validator`
+    working with no pyspark installed at all. Every other name in
+    __all__ is already bound by the eager imports above and resolved by
+    normal attribute lookup before Python ever calls this function -
+    __getattr__ only runs as a fallback for names lookup didn't resolve.
+    """
+    if name == "SparkConnector":
+        from table_validator.connectors.spark_connector import SparkConnector
+
+        return SparkConnector
+    if name == "validate_tables":
+        from table_validator.notebook import validate_tables
+
+        return validate_tables
+    raise AttributeError(f"module {__name__!r} has no attribute {name!r}")

{table_validator-0.1.9 → table_validator-0.1.11}/table_validator/auth/azure_auth.py

@@ -26,22 +26,36 @@ class AzureCredential:
 
     storage_account_key authenticates AzureConnector (Blob Storage);
     sql_username/sql_password authenticate AzureSqlConnector (Azure SQL
-    Database). Both are optional here since a given validation run may
-    only need one of the two.
+    Database); synapse_username/synapse_password authenticate the same
+    AzureSqlConnector class when pointed at a Synapse SQL pool instead
+    (Synapse SQL is protocol-identical T-SQL/ODBC, so no separate
+    connector class exists - only a separate credential pair, since a
+    Synapse SQL login is a distinct principal from an Azure SQL DB one).
+    synapse_client_secret is the alternative to that pair: the Entra ID
+    service-principal secret used when config.azure.synapse_auth_mode is
+    'entra_service_principal', paired with the non-secret tenant_id /
+    synapse_client_id that live in config.yaml.
+    All fields are optional here since a given validation run may only
+    need one pair.
     """
 
     storage_account_key: Optional[str] = None
     sql_username: Optional[str] = None
     sql_password: Optional[str] = None
+    synapse_username: Optional[str] = None
+    synapse_password: Optional[str] = None
+    synapse_client_secret: Optional[str] = None
 
 
 def get_azure_credential(config: ValidatorConfig, env_path: Path = ENV_PATH) -> AzureCredential:
     """
     Resolve Azure credentials for the given config.
 
-    Phase 1: reads AZURE_STORAGE_KEY / AZURE_SQL_USERNAME / AZURE_SQL_PASSWORD
-    from ~/.table_validator/.env. A later phase can swap this body for
-    Azure CLI / Service Principal auth without changing any caller.
+    Phase 1: reads AZURE_STORAGE_KEY / AZURE_SQL_USERNAME /
+    AZURE_SQL_PASSWORD / SYNAPSE_USERNAME / SYNAPSE_PASSWORD /
+    SYNAPSE_CLIENT_SECRET from ~/.table_validator/.env. A later phase can
+    swap this body for Azure CLI / managed-identity auth without changing
+    any caller.
     """
     values = dotenv_values(env_path) if env_path.exists() else {}
 
@@ -49,4 +63,7 @@ def get_azure_credential(config: ValidatorConfig, env_path: Path = ENV_PATH) ->
         storage_account_key=values.get("AZURE_STORAGE_KEY") or None,
         sql_username=values.get("AZURE_SQL_USERNAME") or None,
         sql_password=values.get("AZURE_SQL_PASSWORD") or None,
+        synapse_username=values.get("SYNAPSE_USERNAME") or None,
+        synapse_password=values.get("SYNAPSE_PASSWORD") or None,
+        synapse_client_secret=values.get("SYNAPSE_CLIENT_SECRET") or None,
     )

{table_validator-0.1.9 → table_validator-0.1.11}/table_validator/cli/main.py

@@ -19,7 +19,12 @@ from table_validator.cli.summary_table import (
 from table_validator.cli.partition_prompt import build_partition_prompt
 from table_validator.cli.wizard import run_configure_wizard
 from table_validator.config.manager import CONFIG_PATH, ConfigNotFoundError, require_config
-from table_validator.config.schema import SourceType, ValidationType, ValidatorConfig
+from table_validator.config.schema import (
+    SourceType,
+    SynapseAuthMode,
+    ValidationType,
+    ValidatorConfig,
+)
 from table_validator.connectors.azure_connector import AzureConnector, AzureSqlConnector
 from table_validator.connectors.databricks_connector import DatabricksConnector
 from table_validator.models import (
@@ -207,6 +212,18 @@ def validate(
             "prompt on its own, but --yes makes that explicit."
         ),
     ),
+    skip_category_summary: bool = typer.Option(
+        False,
+        "--skip-category-summary",
+        help=(
+            "Omit the 'Mismatch Categories' sheet (root-cause "
+            "classification + plain-English summary of Data Mismatches) "
+            "from the Excel report. Every other sheet, including Data "
+            "Mismatches itself, is unaffected - this recovers the exact "
+            "report shape from before the mismatch-categorization "
+            "feature existed, for anyone who doesn't want it."
+        ),
+    ),
 ) -> None:
     """Run validation and produce an Excel validation report."""
 
@@ -281,6 +298,8 @@ def validate(
             result = _run_blob_validation(config, azure_credential, databricks)
         elif config.source_type == SourceType.AZURE_SQL:
             result = _run_sql_validation(config, azure_credential, databricks)
+        elif config.source_type == SourceType.SYNAPSE:
+            result = _run_synapse_validation(config, azure_credential, databricks)
         else:
             result = _run_databricks_validation(config, databricks, mode=mode, yes=yes)
     except ValueError as exc:
@@ -310,6 +329,7 @@ def validate(
             result, str(output),
             source_type=config.source_type.value,
             enabled_validations=report_enabled_validations,
+            skip_category_summary=skip_category_summary,
         )
     except PermissionError:
         typer.secho(
@@ -444,6 +464,17 @@ def _run_databricks_validation(
         tables_restriction = [config.source_table.table]
         if config.source_table.table.lower() != config.target_table.table.lower():
             table_map = {config.source_table.table: config.target_table.table}
+    else:
+        # Schema-wide sweep (no single table named on both sides) - apply
+        # the wizard's optional only_tables allowlist / table_map renaming
+        # for this schema, if the user set either. Both are no-ops
+        # (tables_restriction stays None, table_map stays {}) unless the
+        # user actually configured them, so a plain schema sweep behaves
+        # exactly as before this feature existed.
+        if config.only_tables:
+            tables_restriction = list(config.only_tables)
+        if config.table_map:
+            table_map = dict(config.table_map)
 
     # A configured primary key only applies to the single named table
     # (not a catalog-wide sweep) - CatalogValidator looks it up by
@@ -603,6 +634,113 @@ def _run_sql_validation(
     return validator.validate(request)
 
 
+def _run_synapse_validation(
+    config: ValidatorConfig,
+    azure_credential,
+    databricks: DatabricksConnector,
+) -> CatalogValidationResponse:
+    """source_type == synapse: Azure Synapse SQL pool (dedicated or
+    serverless) against a Databricks catalog. Synapse SQL is
+    protocol-identical T-SQL over the same ODBC driver Azure SQL Database
+    uses, so this reuses AzureSqlConnector/AzureSqlValidator UNCHANGED,
+    just constructed with the Synapse server/database/credentials instead
+    of an Azure SQL DB's - no new connector or validator class, mirrors
+    _run_sql_validation's logic exactly (see that function for the
+    schema_map/table_map/primary_keys/FULL-mode rationale, identical
+    here).
+
+    Auth is the one place Synapse genuinely diverges from the Azure SQL
+    path: config.azure.synapse_auth_mode selects between a SQL login
+    (SYNAPSE_USERNAME/SYNAPSE_PASSWORD) and a Microsoft Entra ID service
+    principal (azure.tenant_id + azure.synapse_client_id +
+    SYNAPSE_CLIENT_SECRET). Entra is what a workspace with SQL
+    authentication disabled requires - the connector fetches an access
+    token and passes it via the ODBC access-token attribute rather than
+    sending UID/PWD at all.
+
+    Not yet verified against a live Synapse SQL pool for pool-specific
+    T-SQL quirks (dedicated pools have historically had some system-view/
+    HASHBYTES-adjacent gaps vs. Azure SQL DB) - flagged here rather than
+    assumed away; report back if a specific query fails against your pool
+    and it can be special-cased in AzureSqlConnector the same way
+    decimal_as_integer/float formatting already are for Databricks
+    compatibility.
+    """
+    if not config.azure.synapse_server or not config.azure.synapse_database:
+        raise ValueError(
+            "azure.synapse_server and azure.synapse_database are required "
+            "for a Synapse source. Run 'tablevalidator configure' first."
+        )
+
+    if config.azure.synapse_auth_mode == SynapseAuthMode.ENTRA_SERVICE_PRINCIPAL:
+        missing_entra = []
+        if not config.azure.tenant_id:
+            missing_entra.append("azure.tenant_id (in config.yaml)")
+        if not config.azure.synapse_client_id:
+            missing_entra.append("azure.synapse_client_id (in config.yaml)")
+        if not azure_credential.synapse_client_secret:
+            missing_entra.append("SYNAPSE_CLIENT_SECRET (in ~/.table_validator/.env)")
+        if missing_entra:
+            raise ValueError(
+                "Synapse is configured for Entra service-principal auth but "
+                "these are missing: " + ", ".join(missing_entra)
+                + ". Run 'tablevalidator configure' first."
+            )
+
+        synapse = AzureSqlConnector(
+            server=config.azure.synapse_server,
+            database=config.azure.synapse_database,
+            tenant_id=config.azure.tenant_id,
+            client_id=config.azure.synapse_client_id,
+            client_secret=azure_credential.synapse_client_secret,
+        )
+    else:
+        if not azure_credential.synapse_username or not azure_credential.synapse_password:
+            raise ValueError(
+                "No Synapse username/password found in ~/.table_validator/.env. "
+                "Run 'tablevalidator configure' first."
+            )
+
+        synapse = AzureSqlConnector(
+            server=config.azure.synapse_server,
+            database=config.azure.synapse_database,
+            username=azure_credential.synapse_username,
+            password=azure_credential.synapse_password,
+        )
+
+    schema_map: Dict[str, str] = {}
+    schemas_restriction = None
+    if config.synapse_source.schema_name and config.target_table.schema_name:
+        schema_map = {config.synapse_source.schema_name: config.target_table.schema_name}
+        schemas_restriction = [config.synapse_source.schema_name]
+
+    table_map: Dict[str, str] = {}
+    tables_restriction = None
+    if config.synapse_source.table and config.target_table.table:
+        table_map = {config.synapse_source.table: config.target_table.table}
+        tables_restriction = [config.synapse_source.table]
+
+    primary_keys: Dict[str, list] = {}
+    if config.primary_key and config.target_table.table:
+        primary_keys[config.target_table.table] = config.primary_key
+        if config.target_table.schema_name:
+            key = f"{config.target_table.schema_name}.{config.target_table.table}"
+            primary_keys[key] = config.primary_key
+
+    request = AzureSqlValidationRequest(
+        target_catalog=config.target_table.catalog or "",
+        schemas=schemas_restriction,
+        schema_map=schema_map,
+        tables=tables_restriction,
+        table_map=table_map,
+        primary_keys=primary_keys,
+        data_compare_mode=DataCompareMode.FULL,
+    )
+
+    validator = AzureSqlValidator(synapse, databricks)
+    return validator.validate(request)
+
+
 # ---------------------------------------------------------------------------
 # Helpers
 # ---------------------------------------------------------------------------
@@ -632,6 +770,11 @@ def _missing_config_fields(config: ValidatorConfig) -> list:
             missing.append("azure.sql_server")
         if not config.azure.sql_database:
             missing.append("azure.sql_database")
+    elif config.source_type == SourceType.SYNAPSE:
+        if not config.azure.synapse_server:
+            missing.append("azure.synapse_server")
+        if not config.azure.synapse_database:
+            missing.append("azure.synapse_database")
     else:
         if not config.source_table.catalog:
             missing.append("source_table.catalog")
@@ -696,6 +839,17 @@ def _describe_scope(config: ValidatorConfig) -> str:
             source_desc = f"sql:{database}.{config.sql_source.schema_name} (all tables)"
         else:
             source_desc = f"sql:{database}.{config.sql_source.schema_name}.{config.sql_source.table}"
+    elif config.source_type == SourceType.SYNAPSE:
+        database = config.azure.synapse_database or "?"
+        if not config.synapse_source.schema_name:
+            source_desc = f"synapse:{database} (all schemas)"
+        elif not config.synapse_source.table:
+            source_desc = f"synapse:{database}.{config.synapse_source.schema_name} (all tables)"
+        else:
+            source_desc = (
+                f"synapse:{database}.{config.synapse_source.schema_name}."
+                f"{config.synapse_source.table}"
+            )
     else:
         source_desc = _side(config.source_table)