tablevalidator-databricks 0.1.0__tar.gz → 0.1.1__tar.gz

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tablevalidator-databricks
-Version: 0.1.0
+Version: 0.1.1
 Summary: Generates a widget-driven Databricks notebook UI on top of the table-validator package - no code required to run a comparison.
 License: MIT
 Requires-Python: >=3.9
@@ -72,9 +72,19 @@ markers and opens it as a real notebook.
 
 **Cascading dropdowns**: Databricks widgets don't auto-refresh when an
 upstream selection changes. After changing Source/Target Catalog or
-Schema, re-run the notebook's "Create widgets" cell once to repopulate the
-dropdowns below it before running the rest of the notebook. This is a
-Databricks widget limitation, not a bug in the generated notebook.
+Schema, re-run the notebook's "Create / refresh widgets" cell once to
+repopulate the dropdowns below it before running the rest of the notebook.
+This is a Databricks widget limitation, not a bug in the generated
+notebook. If you skip this and run validation anyway with a stale Schema/
+Table selection, the notebook fails with a clear error telling you which
+cell to re-run, rather than silently comparing the wrong table.
+
+**Can't find a name in a dropdown?** Every Catalog/Schema/Table dropdown
+has a matching "(manual override)" text widget next to it. Type the exact
+name there and it's used instead of the dropdown - useful if something
+isn't showing up (a permissions lag, sort order, anything). It's used
+as-is, not checked against the live catalog listing. Leave every override
+blank to just use the dropdowns as normal.
 
 ## Why a separate package
 

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/README.md

@@ -58,9 +58,19 @@ markers and opens it as a real notebook.
 
 **Cascading dropdowns**: Databricks widgets don't auto-refresh when an
 upstream selection changes. After changing Source/Target Catalog or
-Schema, re-run the notebook's "Create widgets" cell once to repopulate the
-dropdowns below it before running the rest of the notebook. This is a
-Databricks widget limitation, not a bug in the generated notebook.
+Schema, re-run the notebook's "Create / refresh widgets" cell once to
+repopulate the dropdowns below it before running the rest of the notebook.
+This is a Databricks widget limitation, not a bug in the generated
+notebook. If you skip this and run validation anyway with a stale Schema/
+Table selection, the notebook fails with a clear error telling you which
+cell to re-run, rather than silently comparing the wrong table.
+
+**Can't find a name in a dropdown?** Every Catalog/Schema/Table dropdown
+has a matching "(manual override)" text widget next to it. Type the exact
+name there and it's used instead of the dropdown - useful if something
+isn't showing up (a permissions lag, sort order, anything). It's used
+as-is, not checked against the live catalog listing. Leave every override
+blank to just use the dropdowns as normal.
 
 ## Why a separate package
 

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/tablevalidator_databricks/cli/main.py

@@ -88,9 +88,11 @@ def info() -> None:
         "  3. Fill in the widgets and Run All\n"
         "     Pick Source/Target Catalog, Schema, Table, and which checks "
         "to run, then run the notebook top to bottom. Re-run the "
-        "'Create widgets' cell after changing a Catalog/Schema selection, "
-        "to repopulate the dropdowns below it (a Databricks widget "
-        "limitation, not a bug).\n"
+        "'Create / refresh widgets' cell after changing a Catalog/Schema "
+        "selection, to repopulate the dropdowns below it (a Databricks "
+        "widget limitation, not a bug). Can't find a name in a dropdown? "
+        "Each dropdown has a '(manual override)' text widget next to it - "
+        "type the exact name there instead.\n"
         "\n"
         "This package contains no validation logic of its own - every "
         "generated notebook calls directly into table-validator's own "

tablevalidator_databricks-0.1.1/tablevalidator_databricks/templates/widget_notebook.py.tmpl

@@ -0,0 +1,249 @@
+# Databricks notebook source
+# MAGIC %md
+# MAGIC # Table Validator
+# MAGIC Generated by `tablevalidator-databricks init --mode {{MODE}}`.
+# MAGIC
+# MAGIC Fill in the widgets above (Source/Target Catalog, Schema, Table, and
+# MAGIC which checks to run), then **Run All**. No code needs to be written or
+# MAGIC edited - every cell below just reads the widgets and calls the
+# MAGIC `table_validator` package's `validate_tables()` API.
+# MAGIC
+# MAGIC **Changing Catalog or Schema after the dropdowns are already built?**
+# MAGIC Databricks widgets have no on-change event, so picking a new Catalog
+# MAGIC does not repopulate the Schema dropdown by itself (same for Schema ->
+# MAGIC Table) - only re-running the **"Create / refresh widgets"** cell does.
+# MAGIC After changing Source/Target Catalog or Schema, re-run just that one
+# MAGIC cell (Shift+Enter) to refresh everything below it, then continue. This
+# MAGIC is a Databricks platform limitation, not a bug in this notebook.
+# MAGIC
+# MAGIC **Can't find a name in a dropdown?** Each Catalog/Schema/Table dropdown
+# MAGIC has a matching "(manual override)" text box right next to it. Type the
+# MAGIC exact name there and it's used INSTEAD of the dropdown - useful if
+# MAGIC something isn't showing up (a permissions lag, a name that doesn't sort
+# MAGIC into view, etc.). Leave every override blank to just use the dropdowns
+# MAGIC as normal.
+
+# COMMAND ----------
+
+# MAGIC %pip install table-validator
+
+# COMMAND ----------
+
+dbutils.library.restartPython()
+
+# COMMAND ----------
+
+from table_validator import validate_tables
+from table_validator.config.schema import ValidationType
+
+MODE = "{{MODE}}"
+SHOW_OPTIONAL_WIDGETS = {{SHOW_OPTIONAL_WIDGETS}}
+
+
+def _list_catalogs():
+    return sorted(r["catalog"] for r in spark.sql("SHOW CATALOGS").collect())
+
+
+def _list_schemas(catalog: str):
+    if not catalog:
+        return []
+    rows = spark.sql(f"SHOW SCHEMAS IN `{catalog}`").collect()
+    return sorted(r["databaseName"] for r in rows)
+
+
+def _list_tables(catalog: str, schema: str):
+    if not catalog or not schema:
+        return []
+    rows = spark.sql(f"SHOW TABLES IN `{catalog}`.`{schema}`").collect()
+    return sorted(r["tableName"] for r in rows)
+
+
+def _current_or_first(options, current):
+    """Keep a widget's existing selection across a re-run of this cell if
+    it's still valid for the (possibly new) upstream selection, otherwise
+    fall back to the first available option (or "" if there are none)."""
+    if current in options:
+        return current
+    return options[0] if options else ""
+
+# COMMAND ----------
+
+# MAGIC %md ### Create / refresh widgets
+# MAGIC **Run this cell (Shift+Enter) every time you change Source/Target
+# MAGIC Catalog or Schema**, to repopulate the Schema/Table dropdowns below
+# MAGIC them - Databricks widgets don't refresh automatically. Re-running it
+# MAGIC is always safe: your current Catalog/Schema/Table picks are kept if
+# MAGIC still valid, and only stale downstream selections reset.
+
+# COMMAND ----------
+
+def _get_widget(name: str, default: str = "") -> str:
+    try:
+        return dbutils.widgets.get(name)
+    except Exception:
+        return default
+
+
+def _effective_value(prefix: str) -> str:
+    """The manual-override text widget wins whenever it's non-blank,
+    otherwise fall back to the dropdown's current value - lets a name
+    that isn't showing up in a dropdown (permissions lag, sort order,
+    anything) be typed in directly instead. Used as-is, not validated
+    against the live catalog/schema/table list - same convention as
+    row_filter elsewhere in this notebook."""
+    override = _get_widget(f"{prefix}_override").strip()
+    return override or _get_widget(prefix)
+
+
+def _build_cascading_widgets(prefix: str, label_prefix: str) -> None:
+    """Build/refresh one side's (source or target) Catalog -> Schema ->
+    Table dropdowns from Unity Catalog, in one pass - each level is
+    re-queried from the CURRENT value of the level above, so this single
+    cell run always reflects whatever Catalog/Schema is selected right
+    now, not a stale snapshot from an earlier run. Also creates a
+    "(manual override)" text widget next to each dropdown, for a name
+    that isn't showing up in the dropdown for any reason - see
+    _effective_value() below for how the override takes priority."""
+    catalog_current = _current_or_first(catalogs, _effective_value(f"{prefix}_catalog"))
+    schemas = _list_schemas(catalog_current)
+    schema_current = _current_or_first(schemas, _effective_value(f"{prefix}_schema"))
+    tables = _list_tables(catalog_current, schema_current)
+    table_current = _current_or_first(tables, _effective_value(f"{prefix}_table"))
+
+    dbutils.widgets.dropdown(f"{prefix}_catalog", catalog_current, catalogs or [""], f"{label_prefix} Catalog")
+    dbutils.widgets.text(f"{prefix}_catalog_override", "", f"{label_prefix} Catalog (manual override)")
+    dbutils.widgets.dropdown(f"{prefix}_schema", schema_current, schemas or [""], f"{label_prefix} Schema")
+    dbutils.widgets.text(f"{prefix}_schema_override", "", f"{label_prefix} Schema (manual override)")
+    dbutils.widgets.dropdown(f"{prefix}_table", table_current, tables or [""], f"{label_prefix} Table")
+    dbutils.widgets.text(f"{prefix}_table_override", "", f"{label_prefix} Table (manual override)")
+
+
+catalogs = _list_catalogs()
+_build_cascading_widgets("source", "Source")
+_build_cascading_widgets("target", "Target")
+
+print(
+    "Widgets refreshed for the CURRENT Catalog/Schema selection.\n"
+    "Changed Catalog or Schema just now? Run this cell again (Shift+Enter) "
+    "before moving on - Databricks widgets don't repopulate on their own."
+)
+
+# COMMAND ----------
+
+# MAGIC %md ### Validation checks
+
+# COMMAND ----------
+
+_default_checks = ["Catalog & Schema", "Column", "Row"] if MODE != "schema" else ["Catalog & Schema"]
+dbutils.widgets.multiselect(
+    "checks", ",".join(_default_checks),
+    ["Catalog & Schema", "Column", "Row"], "Validation Checks",
+)
+
+if SHOW_OPTIONAL_WIDGETS:
+    dbutils.widgets.text("only_columns", "", "Only compare these columns (comma-separated, optional)")
+    dbutils.widgets.text("ignore_columns", "", "Skip these columns entirely (comma-separated, optional)")
+    dbutils.widgets.text("row_filter", "", "Row filter - SQL WHERE-fragment (optional)")
+    dbutils.widgets.text("primary_key", "", "Primary key column(s) (comma-separated, optional)")
+
+# COMMAND ----------
+
+# MAGIC %md ### Run validation
+
+# COMMAND ----------
+
+def _validate_current_selection(prefix: str, label: str) -> None:
+    """Guard against a stale Schema/Table dropdown: if the Schema/Table
+    widget's CURRENT value isn't actually in that catalog/schema right
+    now (e.g. the user picked a new Catalog but never re-ran the "Create
+    / refresh widgets" cell above), fail loudly here with a clear fix
+    instead of silently validating the wrong table. A field with a
+    manual override set is trusted as typed and skipped here - the whole
+    point of the override is to name something the live catalog listing
+    doesn't (currently) show, so it can never pass this check by
+    definition."""
+    catalog = _effective_value(f"{prefix}_catalog")
+    schema_overridden = bool(_get_widget(f"{prefix}_schema_override").strip())
+    table_overridden = bool(_get_widget(f"{prefix}_table_override").strip())
+    schema = _effective_value(f"{prefix}_schema")
+    table = _effective_value(f"{prefix}_table")
+
+    if not schema_overridden:
+        live_schemas = _list_schemas(catalog)
+        if schema not in live_schemas:
+            raise ValueError(
+                f"{label} Schema widget is set to {schema!r}, which is not "
+                f"a schema in catalog {catalog!r} right now. You likely "
+                f"changed the {label} Catalog without refreshing - go back "
+                "and re-run the 'Create / refresh widgets' cell, then "
+                "re-pick the Schema and Table before running this cell "
+                "again. (Or, if this schema genuinely isn't showing up in "
+                "the dropdown, type it into the Schema manual override "
+                "widget instead.)"
+            )
+    if not table_overridden:
+        live_tables = _list_tables(catalog, schema)
+        if table not in live_tables:
+            raise ValueError(
+                f"{label} Table widget is set to {table!r}, which is not a "
+                f"table in {catalog}.{schema} right now. You likely changed "
+                f"the {label} Schema without refreshing - go back and "
+                "re-run the 'Create / refresh widgets' cell, then re-pick "
+                "the Table before running this cell again. (Or, if this "
+                "table genuinely isn't showing up in the dropdown, type it "
+                "into the Table manual override widget instead.)"
+            )
+
+
+_validate_current_selection("source", "Source")
+_validate_current_selection("target", "Target")
+
+source = f"{_effective_value('source_catalog')}.{_effective_value('source_schema')}.{_effective_value('source_table')}"
+target = f"{_effective_value('target_catalog')}.{_effective_value('target_schema')}.{_effective_value('target_table')}"
+
+_CHECK_MAP = {
+    "Catalog & Schema": {ValidationType.CATALOG, ValidationType.SCHEMA},
+    "Column": {ValidationType.COLUMN},
+    "Row": {ValidationType.ROW},
+}
+selected_checks = [c for c in dbutils.widgets.get("checks").split(",") if c]
+enabled_validations = set()
+for check in selected_checks:
+    enabled_validations |= _CHECK_MAP.get(check, set())
+if not enabled_validations:
+    enabled_validations = None  # fall back to validate_tables()'s own "run everything" default
+
+kwargs = {"enabled_validations": enabled_validations}
+
+if SHOW_OPTIONAL_WIDGETS:
+    only_columns = [c.strip() for c in dbutils.widgets.get("only_columns").split(",") if c.strip()]
+    ignore_columns = [c.strip() for c in dbutils.widgets.get("ignore_columns").split(",") if c.strip()]
+    row_filter = dbutils.widgets.get("row_filter").strip()
+    primary_key = [c.strip() for c in dbutils.widgets.get("primary_key").split(",") if c.strip()]
+
+    if only_columns:
+        kwargs["only_columns"] = only_columns
+    if ignore_columns:
+        kwargs["ignore_columns"] = ignore_columns
+    if row_filter:
+        kwargs["row_filter"] = row_filter
+    if primary_key:
+        kwargs["primary_key"] = primary_key
+
+result = validate_tables(source, target, **kwargs)
+
+# COMMAND ----------
+
+# MAGIC %md ### Results
+
+# COMMAND ----------
+
+print(result)
+display(result.table_validation.to_dataframe())
+
+# Other sheets, mirroring the Excel report's own tabs - uncomment as needed:
+# display(result.column_validation.to_dataframe())
+# display(result.data_mismatches.to_dataframe())
+# display(result.row_hash_mismatches.to_dataframe())
+# display(result.mismatch_categories.to_dataframe())
+# display(result.suggestions.to_dataframe())

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/tablevalidator_databricks.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: tablevalidator-databricks
-Version: 0.1.0
+Version: 0.1.1
 Summary: Generates a widget-driven Databricks notebook UI on top of the table-validator package - no code required to run a comparison.
 License: MIT
 Requires-Python: >=3.9
@@ -72,9 +72,19 @@ markers and opens it as a real notebook.
 
 **Cascading dropdowns**: Databricks widgets don't auto-refresh when an
 upstream selection changes. After changing Source/Target Catalog or
-Schema, re-run the notebook's "Create widgets" cell once to repopulate the
-dropdowns below it before running the rest of the notebook. This is a
-Databricks widget limitation, not a bug in the generated notebook.
+Schema, re-run the notebook's "Create / refresh widgets" cell once to
+repopulate the dropdowns below it before running the rest of the notebook.
+This is a Databricks widget limitation, not a bug in the generated
+notebook. If you skip this and run validation anyway with a stale Schema/
+Table selection, the notebook fails with a clear error telling you which
+cell to re-run, rather than silently comparing the wrong table.
+
+**Can't find a name in a dropdown?** Every Catalog/Schema/Table dropdown
+has a matching "(manual override)" text widget next to it. Type the exact
+name there and it's used instead of the dropdown - useful if something
+isn't showing up (a permissions lag, sort order, anything). It's used
+as-is, not checked against the live catalog listing. Leave every override
+blank to just use the dropdowns as normal.
 
 ## Why a separate package
 

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/tablevalidator_databricks.egg-info/SOURCES.txt

@@ -15,4 +15,5 @@ tablevalidator_databricks/generator/__init__.py
 tablevalidator_databricks/generator/generator.py
 tablevalidator_databricks/templates/widget_notebook.py.tmpl
 tests/test_cli.py
+tests/test_generated_notebook_execution.py
 tests/test_generator.py

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/tablevalidator_databricks.egg-info/scm_file_list.json

@@ -9,6 +9,7 @@
     "tablevalidator_databricks/generator/generator.py",
     "tablevalidator_databricks/templates/widget_notebook.py.tmpl",
     "tests/test_cli.py",
+    "tests/test_generated_notebook_execution.py",
     "tests/test_generator.py"
   ]
 }

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/tablevalidator_databricks.egg-info/scm_version.json

@@ -1,7 +1,7 @@
 {
-  "tag": "0.1.0",
+  "tag": "0.1.1",
   "distance": 0,
-  "node": "g9124e37",
+  "node": "g1f85a99",
   "dirty": false,
   "branch": "HEAD",
   "node_date": "2026-09-08"

tablevalidator_databricks-0.1.1/tests/test_generated_notebook_execution.py

@@ -0,0 +1,231 @@
+"""Executes the generated notebook's Python cells against a fake
+Databricks runtime (dbutils/spark/display stubs), to catch real bugs in
+the widget wiring - not just that the file is syntactically valid.
+
+Strips the Databricks-magic-only lines (the notebook source header,
+`# COMMAND ----------` cell separators, `# MAGIC` lines) and execs the
+rest as one script, same approach used to hand-verify this template
+during development.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+from unittest.mock import MagicMock
+
+import pytest
+
+from tablevalidator_databricks.generator.generator import generate_notebook
+
+
+def _code_from_notebook(path: Path) -> str:
+    lines = path.read_text(encoding="utf-8").splitlines()
+    code_lines = [
+        line for line in lines
+        if line.strip() != "# Databricks notebook source"
+        and line.strip() != "# COMMAND ----------"
+        and not line.strip().startswith("# MAGIC")
+    ]
+    return "\n".join(code_lines)
+
+
+class _FakeWidgets:
+    def __init__(self, initial=None):
+        self._values = dict(initial or {})
+
+    def get(self, name):
+        return self._values.get(name, "")
+
+    def text(self, name, default, label):
+        self._values.setdefault(name, default)
+
+    def dropdown(self, name, default, choices, label):
+        # Real dbutils.widgets.dropdown does NOT overwrite an existing
+        # value just because the cell re-ran with a different `default` -
+        # it only sets it the first time. Match that.
+        self._values.setdefault(name, default)
+
+    def multiselect(self, name, default, choices, label):
+        self._values.setdefault(name, default)
+
+
+class _FakeRow(dict):
+    def __getitem__(self, key):
+        return dict.__getitem__(self, key)
+
+
+def _fake_spark(catalogs, schemas_by_catalog, tables_by_schema):
+    def sql(query):
+        result = MagicMock()
+        if "SHOW CATALOGS" in query:
+            result.collect.return_value = [_FakeRow(catalog=c) for c in catalogs]
+        elif "SHOW SCHEMAS IN" in query:
+            catalog = query.split("`")[1]
+            result.collect.return_value = [
+                _FakeRow(databaseName=s) for s in schemas_by_catalog.get(catalog, [])
+            ]
+        elif "SHOW TABLES IN" in query:
+            parts = query.split("`")
+            catalog, schema = parts[1], parts[3]
+            result.collect.return_value = [
+                _FakeRow(tableName=t) for t in tables_by_schema.get((catalog, schema), [])
+            ]
+        else:
+            result.collect.return_value = []
+        return result
+
+    spark = MagicMock()
+    spark.sql.side_effect = sql
+    return spark
+
+
+def _run_notebook(path: Path, widget_values, spark, validate_tables_mock):
+    code = _code_from_notebook(path)
+
+    fake_dbutils = MagicMock()
+    fake_dbutils.widgets = _FakeWidgets(initial=widget_values)
+    fake_dbutils.library.restartPython = MagicMock()
+
+    globs = {
+        "dbutils": fake_dbutils,
+        "spark": spark,
+        "display": MagicMock(),
+        "__name__": "__main__",
+    }
+
+    import table_validator
+    import table_validator.notebook as nb_mod
+
+    table_validator.validate_tables = validate_tables_mock
+    nb_mod.validate_tables = validate_tables_mock
+
+    exec(compile(code, str(path), "exec"), globs)
+    return globs
+
+
+@pytest.fixture
+def notebook_path(tmp_path: Path) -> Path:
+    path = tmp_path / "TableValidator.py"
+    generate_notebook(path, "full")
+    return path
+
+
+def test_happy_path_calls_validate_tables_with_resolved_names(notebook_path):
+    spark = _fake_spark(
+        catalogs=["main"],
+        schemas_by_catalog={"main": ["sales"]},
+        tables_by_schema={("main", "sales"): ["orders"]},
+    )
+    captured = {}
+
+    def fake_validate_tables(source, target, **kwargs):
+        captured["source"] = source
+        captured["target"] = target
+        captured["kwargs"] = kwargs
+        m = MagicMock()
+        m.__str__ = MagicMock(return_value="Overall status: PASS")
+        return m
+
+    _run_notebook(notebook_path, widget_values={}, spark=spark, validate_tables_mock=fake_validate_tables)
+
+    assert captured["source"] == "main.sales.orders"
+    assert captured["target"] == "main.sales.orders"
+
+
+def test_stale_schema_selection_raises_clear_error(notebook_path):
+    """Regression test for a real user-reported issue: after picking a
+    new Source Catalog, the Source Schema/Table widgets kept their value
+    from the PREVIOUS catalog because the 'Create / refresh widgets'
+    cell wasn't re-run. Must fail loudly instead of silently validating
+    against the wrong table."""
+    spark = _fake_spark(
+        catalogs=["for_validation1", "for_validation2"],
+        schemas_by_catalog={"for_validation1": ["alteryx_silver_source"]},
+        tables_by_schema={("for_validation1", "alteryx_silver_source"): ["from_bronze_us_counties"]},
+    )
+
+    with pytest.raises(ValueError, match="not a schema in catalog"):
+        _run_notebook(
+            notebook_path,
+            widget_values={
+                "source_catalog": "for_validation1",
+                "source_schema": "some_stale_schema_from_before",
+                "source_table": "some_stale_table",
+                "target_catalog": "for_validation1",
+                "target_schema": "alteryx_silver_source",
+                "target_table": "from_bronze_us_counties",
+            },
+            spark=spark,
+            validate_tables_mock=MagicMock(),
+        )
+
+
+def test_manual_override_bypasses_dropdown_and_stale_check(notebook_path):
+    """A table that isn't showing up in the dropdown must still be
+    usable by typing it into the "(manual override)" text widget - and
+    the override, not the dropdown, is what gets sent to
+    validate_tables()."""
+    spark = _fake_spark(
+        catalogs=["main"],
+        schemas_by_catalog={"main": ["sales"]},
+        # "hidden_table" deliberately absent - simulating a table not
+        # showing up in the dropdown.
+        tables_by_schema={("main", "sales"): ["orders"]},
+    )
+    captured = {}
+
+    def fake_validate_tables(source, target, **kwargs):
+        captured["source"] = source
+        captured["target"] = target
+        m = MagicMock()
+        m.__str__ = MagicMock(return_value="Overall status: PASS")
+        return m
+
+    _run_notebook(
+        notebook_path,
+        widget_values={
+            "source_catalog": "main", "source_schema": "sales", "source_table": "orders",
+            "target_catalog": "main", "target_schema": "sales",
+            "target_table_override": "hidden_table",
+        },
+        spark=spark,
+        validate_tables_mock=fake_validate_tables,
+    )
+
+    assert captured["target"] == "main.sales.hidden_table"
+
+
+def test_catalog_override_drives_schema_and_table_lookup(notebook_path):
+    """A manually-overridden Catalog must be used when querying for
+    Schemas/Tables too, not just at final source/target string
+    construction - otherwise typing a catalog override would leave the
+    Schema/Table dropdowns empty."""
+    spark = _fake_spark(
+        catalogs=["main"],  # "hidden_catalog" deliberately absent
+        schemas_by_catalog={"hidden_catalog": ["sales"], "main": ["sales"]},
+        tables_by_schema={
+            ("hidden_catalog", "sales"): ["orders"],
+            ("main", "sales"): ["orders"],
+        },
+    )
+    captured = {}
+
+    def fake_validate_tables(source, target, **kwargs):
+        captured["source"] = source
+        captured["target"] = target
+        m = MagicMock()
+        m.__str__ = MagicMock(return_value="Overall status: PASS")
+        return m
+
+    _run_notebook(
+        notebook_path,
+        widget_values={
+            "source_catalog_override": "hidden_catalog",
+            "source_schema": "sales", "source_table": "orders",
+            "target_catalog": "main", "target_schema": "sales", "target_table": "orders",
+        },
+        spark=spark,
+        validate_tables_mock=fake_validate_tables,
+    )
+
+    assert captured["source"] == "hidden_catalog.sales.orders"

{tablevalidator_databricks-0.1.0 → tablevalidator_databricks-0.1.1}/tests/test_generator.py

@@ -63,6 +63,22 @@ def test_generate_notebook_creates_parent_directories(tmp_path: Path) -> None:
     assert output.exists()
 
 
+def test_generate_notebook_includes_manual_override_widgets(tmp_path: Path) -> None:
+    """Every Catalog/Schema/Table dropdown gets a matching manual-override
+    text widget, for a name that isn't showing up in the dropdown. Widget
+    names are built at runtime via f-strings (f"{prefix}_catalog_override"),
+    not literal per-side text, so check for the shared template fragments
+    rather than the fully-resolved names."""
+    output = tmp_path / "TableValidator.py"
+    generate_notebook(output, "full")
+    content = output.read_text(encoding="utf-8")
+
+    assert '"{prefix}_catalog_override"' in content
+    assert '"{prefix}_schema_override"' in content
+    assert '"{prefix}_table_override"' in content
+    assert "(manual override)" in content
+
+
 def test_generate_notebook_no_leftover_placeholders(tmp_path: Path) -> None:
     """Regression guard: every {{...}} placeholder in the template must be
     substituted - a leftover placeholder would silently ship broken Python

tablevalidator_databricks-0.1.0/tablevalidator_databricks/templates/widget_notebook.py.tmpl

@@ -1,178 +0,0 @@
-# Databricks notebook source
-# MAGIC %md
-# MAGIC # Table Validator
-# MAGIC Generated by `tablevalidator-databricks init --mode {{MODE}}`.
-# MAGIC
-# MAGIC Fill in the widgets above (Source/Target Catalog, Schema, Table, and
-# MAGIC which checks to run), then **Run All**. No code needs to be written or
-# MAGIC edited - every cell below just reads the widgets and calls the
-# MAGIC `table_validator` package's `validate_tables()` API.
-# MAGIC
-# MAGIC **Cascading dropdowns**: Databricks widgets don't auto-refresh when an
-# MAGIC upstream selection changes. After picking a different Source/Target
-# MAGIC Catalog, re-run the "Create widgets" cell below to repopulate the Schema
-# MAGIC dropdown for that catalog; likewise after changing Schema, re-run it
-# MAGIC again to repopulate Table. This is a real Databricks widget limitation,
-# MAGIC not a bug in this notebook.
-
-# COMMAND ----------
-
-# MAGIC %pip install table-validator
-
-# COMMAND ----------
-
-dbutils.library.restartPython()
-
-# COMMAND ----------
-
-from table_validator import validate_tables
-from table_validator.config.schema import ValidationType
-
-MODE = "{{MODE}}"
-SHOW_OPTIONAL_WIDGETS = {{SHOW_OPTIONAL_WIDGETS}}
-
-
-def _list_catalogs():
-    return sorted(r["catalog"] for r in spark.sql("SHOW CATALOGS").collect())
-
-
-def _list_schemas(catalog: str):
-    if not catalog:
-        return []
-    rows = spark.sql(f"SHOW SCHEMAS IN `{catalog}`").collect()
-    return sorted(r["databaseName"] for r in rows)
-
-
-def _list_tables(catalog: str, schema: str):
-    if not catalog or not schema:
-        return []
-    rows = spark.sql(f"SHOW TABLES IN `{catalog}`.`{schema}`").collect()
-    return sorted(r["tableName"] for r in rows)
-
-
-def _current_or_first(options, current):
-    """Keep a widget's existing selection across a re-run of this cell if
-    it's still valid for the (possibly new) upstream selection, otherwise
-    fall back to the first available option (or "" if there are none)."""
-    if current in options:
-        return current
-    return options[0] if options else ""
-
-# COMMAND ----------
-
-# MAGIC %md ### Create widgets
-# MAGIC Re-run this cell after changing Source/Target Catalog or Schema, to
-# MAGIC repopulate the dropdowns below them.
-
-# COMMAND ----------
-
-def _get_widget(name: str, default: str = "") -> str:
-    try:
-        return dbutils.widgets.get(name)
-    except Exception:
-        return default
-
-
-catalogs = _list_catalogs()
-
-source_catalog_current = _current_or_first(catalogs, _get_widget("source_catalog"))
-target_catalog_current = _current_or_first(catalogs, _get_widget("target_catalog"))
-
-source_schemas = _list_schemas(source_catalog_current)
-target_schemas = _list_schemas(target_catalog_current)
-
-source_schema_current = _current_or_first(source_schemas, _get_widget("source_schema"))
-target_schema_current = _current_or_first(target_schemas, _get_widget("target_schema"))
-
-source_tables = _list_tables(source_catalog_current, source_schema_current)
-target_tables = _list_tables(target_catalog_current, target_schema_current)
-
-source_table_current = _current_or_first(source_tables, _get_widget("source_table"))
-target_table_current = _current_or_first(target_tables, _get_widget("target_table"))
-
-dbutils.widgets.dropdown("source_catalog", source_catalog_current, catalogs or [""], "Source Catalog")
-dbutils.widgets.dropdown("source_schema", source_schema_current, source_schemas or [""], "Source Schema")
-dbutils.widgets.dropdown("source_table", source_table_current, source_tables or [""], "Source Table")
-
-dbutils.widgets.dropdown("target_catalog", target_catalog_current, catalogs or [""], "Target Catalog")
-dbutils.widgets.dropdown("target_schema", target_schema_current, target_schemas or [""], "Target Schema")
-dbutils.widgets.dropdown("target_table", target_table_current, target_tables or [""], "Target Table")
-
-print(
-    "Widgets created. If you just changed Source/Target Catalog or Schema, "
-    "re-run this cell once more to repopulate the dropdowns below it."
-)
-
-# COMMAND ----------
-
-# MAGIC %md ### Validation checks
-
-# COMMAND ----------
-
-_default_checks = ["Catalog & Schema", "Column", "Row"] if MODE != "schema" else ["Catalog & Schema"]
-dbutils.widgets.multiselect(
-    "checks", ",".join(_default_checks),
-    ["Catalog & Schema", "Column", "Row"], "Validation Checks",
-)
-
-if SHOW_OPTIONAL_WIDGETS:
-    dbutils.widgets.text("only_columns", "", "Only compare these columns (comma-separated, optional)")
-    dbutils.widgets.text("ignore_columns", "", "Skip these columns entirely (comma-separated, optional)")
-    dbutils.widgets.text("row_filter", "", "Row filter - SQL WHERE-fragment (optional)")
-    dbutils.widgets.text("primary_key", "", "Primary key column(s) (comma-separated, optional)")
-
-# COMMAND ----------
-
-# MAGIC %md ### Run validation
-
-# COMMAND ----------
-
-source = f"{dbutils.widgets.get('source_catalog')}.{dbutils.widgets.get('source_schema')}.{dbutils.widgets.get('source_table')}"
-target = f"{dbutils.widgets.get('target_catalog')}.{dbutils.widgets.get('target_schema')}.{dbutils.widgets.get('target_table')}"
-
-_CHECK_MAP = {
-    "Catalog & Schema": {ValidationType.CATALOG, ValidationType.SCHEMA},
-    "Column": {ValidationType.COLUMN},
-    "Row": {ValidationType.ROW},
-}
-selected_checks = [c for c in dbutils.widgets.get("checks").split(",") if c]
-enabled_validations = set()
-for check in selected_checks:
-    enabled_validations |= _CHECK_MAP.get(check, set())
-if not enabled_validations:
-    enabled_validations = None  # fall back to validate_tables()'s own "run everything" default
-
-kwargs = {"enabled_validations": enabled_validations}
-
-if SHOW_OPTIONAL_WIDGETS:
-    only_columns = [c.strip() for c in dbutils.widgets.get("only_columns").split(",") if c.strip()]
-    ignore_columns = [c.strip() for c in dbutils.widgets.get("ignore_columns").split(",") if c.strip()]
-    row_filter = dbutils.widgets.get("row_filter").strip()
-    primary_key = [c.strip() for c in dbutils.widgets.get("primary_key").split(",") if c.strip()]
-
-    if only_columns:
-        kwargs["only_columns"] = only_columns
-    if ignore_columns:
-        kwargs["ignore_columns"] = ignore_columns
-    if row_filter:
-        kwargs["row_filter"] = row_filter
-    if primary_key:
-        kwargs["primary_key"] = primary_key
-
-result = validate_tables(source, target, **kwargs)
-
-# COMMAND ----------
-
-# MAGIC %md ### Results
-
-# COMMAND ----------
-
-print(result)
-display(result.table_validation.to_dataframe())
-
-# Other sheets, mirroring the Excel report's own tabs - uncomment as needed:
-# display(result.column_validation.to_dataframe())
-# display(result.data_mismatches.to_dataframe())
-# display(result.row_hash_mismatches.to_dataframe())
-# display(result.mismatch_categories.to_dataframe())
-# display(result.suggestions.to_dataframe())