tablevalidator-databricks 0.1.2__tar.gz

tablevalidator_databricks-0.1.2/PKG-INFO

@@ -0,0 +1,116 @@
+Metadata-Version: 2.4
+Name: tablevalidator-databricks
+Version: 0.1.2
+Summary: Generates a widget-driven Databricks notebook UI on top of the table-validator package - no code required to run a comparison.
+License: MIT
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: typer>=0.12
+Requires-Dist: table-validator>=0.1.19
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: ruff>=0.4; extra == "dev"
+Requires-Dist: black>=24.0; extra == "dev"
+
+# tablevalidator-databricks
+
+A widget-driven Databricks notebook UI for [`table-validator`](https://pypi.org/project/table-validator/)
+- generates a notebook with `dbutils.widgets` for picking source/target
+tables and validation checks, so you can run a comparison without writing
+any Python.
+
+This package contains **no validation logic of its own**. It only writes a
+notebook file whose cells call directly into `table-validator`'s own
+`validate_tables()` API - the exact same engine the CLI (`tablevalidator`)
+and the notebook-native Python API already use.
+
+## Install
+
+```bash
+pip install tablevalidator-databricks
+```
+
+Installing this package also installs `table-validator` (a dependency), so
+there's nothing else to install separately.
+
+## Usage
+
+```bash
+tablevalidator-databricks init
+```
+
+Writes `./TableValidator.py` - a ready-to-use Databricks notebook. Options:
+
+```bash
+tablevalidator-databricks init --output MyValidation.py --mode full
+```
+
+- `--mode basic` - only the check-group multiselect and table pickers.
+- `--mode full` (default) - also shows optional filter widgets (only/
+  ignore columns, row filter, primary key).
+- `--mode schema` - same widgets as `basic`; intended for a lightweight
+  schema-shape-only comparison (just deselect Column/Row in the checks
+  multiselect after opening the notebook).
+
+Then, in your Databricks workspace: **Workspace -> Import**, format
+"Source", and select the generated file. Databricks recognizes the
+`# Databricks notebook source` header and `# COMMAND ----------` cell
+markers and opens it as a real notebook.
+
+## What the generated notebook does
+
+1. Installs `table-validator` (`%pip install table-validator`).
+2. Queries Unity Catalog (`SHOW CATALOGS` / `SHOW SCHEMAS IN ...` /
+   `SHOW TABLES IN ...` via the notebook's own ambient Spark session) to
+   populate Source/Target Catalog, Schema, and Table dropdowns.
+3. Shows a "Validation Checks" multiselect (Catalog & Schema / Column /
+   Row - the three independently-selectable check groups the engine
+   actually supports) plus, in `full` mode, optional text widgets for
+   column filtering, a row filter, and a primary key.
+4. On run, reads every widget and calls
+   `validate_tables(source, target, enabled_validations=..., ...)`,
+   then prints a summary and displays the Table Validation sheet.
+
+**Cascading dropdowns**: Databricks widgets don't auto-refresh when an
+upstream selection changes. After changing Source/Target Catalog or
+Schema, re-run the notebook's "Create / refresh widgets" cell once to
+repopulate the dropdowns below it before running the rest of the notebook.
+This is a Databricks widget limitation, not a bug in the generated
+notebook. If you skip this and run validation anyway with a stale Schema/
+Table selection, the notebook fails with a clear error telling you which
+cell to re-run, rather than silently comparing the wrong table.
+
+**Can't find a name in a dropdown?** Every Catalog/Schema/Table dropdown
+has a matching "(manual override)" text widget next to it. Type the exact
+name there and it's used instead of the dropdown - useful if something
+isn't showing up (a permissions lag, sort order, anything). It's used
+as-is, not checked against the live catalog listing. Leave every override
+blank to just use the dropdowns as normal.
+
+**Validate every table in a schema, not just one**: pick
+"(all tables in schema)" as the Table dropdown (Source and/or Target
+independently) to compare every identically-named table in that schema in
+one run - the same schema-wide sweep `validate_tables()` itself supports.
+A primary key can't be set in this mode (leave the Primary Key widget
+blank). Comparing every schema in a catalog isn't supported yet - Schema
+must be a specific choice, even when Table is left as "(all ... in ...)".
+
+## Why a separate package
+
+`table-validator` stays focused on the actual comparison engine (CLI +
+notebook API + report generation). This package is purely a Databricks-
+specific UI layer on top of it - keeping the two independently versioned
+and installable means CLI-only or code-only users never need Databricks-
+specific tooling pulled in, and this package can add more Databricks UX
+(e.g. richer widgets) without touching the core engine at all.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT - see the `table-validator` repository's [LICENSE](../table_validator/LICENSE).

tablevalidator_databricks-0.1.2/README.md

@@ -0,0 +1,102 @@
+# tablevalidator-databricks
+
+A widget-driven Databricks notebook UI for [`table-validator`](https://pypi.org/project/table-validator/)
+- generates a notebook with `dbutils.widgets` for picking source/target
+tables and validation checks, so you can run a comparison without writing
+any Python.
+
+This package contains **no validation logic of its own**. It only writes a
+notebook file whose cells call directly into `table-validator`'s own
+`validate_tables()` API - the exact same engine the CLI (`tablevalidator`)
+and the notebook-native Python API already use.
+
+## Install
+
+```bash
+pip install tablevalidator-databricks
+```
+
+Installing this package also installs `table-validator` (a dependency), so
+there's nothing else to install separately.
+
+## Usage
+
+```bash
+tablevalidator-databricks init
+```
+
+Writes `./TableValidator.py` - a ready-to-use Databricks notebook. Options:
+
+```bash
+tablevalidator-databricks init --output MyValidation.py --mode full
+```
+
+- `--mode basic` - only the check-group multiselect and table pickers.
+- `--mode full` (default) - also shows optional filter widgets (only/
+  ignore columns, row filter, primary key).
+- `--mode schema` - same widgets as `basic`; intended for a lightweight
+  schema-shape-only comparison (just deselect Column/Row in the checks
+  multiselect after opening the notebook).
+
+Then, in your Databricks workspace: **Workspace -> Import**, format
+"Source", and select the generated file. Databricks recognizes the
+`# Databricks notebook source` header and `# COMMAND ----------` cell
+markers and opens it as a real notebook.
+
+## What the generated notebook does
+
+1. Installs `table-validator` (`%pip install table-validator`).
+2. Queries Unity Catalog (`SHOW CATALOGS` / `SHOW SCHEMAS IN ...` /
+   `SHOW TABLES IN ...` via the notebook's own ambient Spark session) to
+   populate Source/Target Catalog, Schema, and Table dropdowns.
+3. Shows a "Validation Checks" multiselect (Catalog & Schema / Column /
+   Row - the three independently-selectable check groups the engine
+   actually supports) plus, in `full` mode, optional text widgets for
+   column filtering, a row filter, and a primary key.
+4. On run, reads every widget and calls
+   `validate_tables(source, target, enabled_validations=..., ...)`,
+   then prints a summary and displays the Table Validation sheet.
+
+**Cascading dropdowns**: Databricks widgets don't auto-refresh when an
+upstream selection changes. After changing Source/Target Catalog or
+Schema, re-run the notebook's "Create / refresh widgets" cell once to
+repopulate the dropdowns below it before running the rest of the notebook.
+This is a Databricks widget limitation, not a bug in the generated
+notebook. If you skip this and run validation anyway with a stale Schema/
+Table selection, the notebook fails with a clear error telling you which
+cell to re-run, rather than silently comparing the wrong table.
+
+**Can't find a name in a dropdown?** Every Catalog/Schema/Table dropdown
+has a matching "(manual override)" text widget next to it. Type the exact
+name there and it's used instead of the dropdown - useful if something
+isn't showing up (a permissions lag, sort order, anything). It's used
+as-is, not checked against the live catalog listing. Leave every override
+blank to just use the dropdowns as normal.
+
+**Validate every table in a schema, not just one**: pick
+"(all tables in schema)" as the Table dropdown (Source and/or Target
+independently) to compare every identically-named table in that schema in
+one run - the same schema-wide sweep `validate_tables()` itself supports.
+A primary key can't be set in this mode (leave the Primary Key widget
+blank). Comparing every schema in a catalog isn't supported yet - Schema
+must be a specific choice, even when Table is left as "(all ... in ...)".
+
+## Why a separate package
+
+`table-validator` stays focused on the actual comparison engine (CLI +
+notebook API + report generation). This package is purely a Databricks-
+specific UI layer on top of it - keeping the two independently versioned
+and installable means CLI-only or code-only users never need Databricks-
+specific tooling pulled in, and this package can add more Databricks UX
+(e.g. richer widgets) without touching the core engine at all.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+## License
+
+MIT - see the `table-validator` repository's [LICENSE](../table_validator/LICENSE).

tablevalidator_databricks-0.1.2/pyproject.toml

@@ -0,0 +1,50 @@
+[build-system]
+requires = ["setuptools>=68", "setuptools_scm>=8"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "tablevalidator-databricks"
+dynamic = ["version"]
+description = "Generates a widget-driven Databricks notebook UI on top of the table-validator package - no code required to run a comparison."
+readme = "README.md"
+requires-python = ">=3.9"
+license = { text = "MIT" }
+dependencies = [
+    "typer>=0.12",
+    "table-validator>=0.1.19",
+]
+
+[project.optional-dependencies]
+dev = [
+    "pytest>=7.0",
+    "ruff>=0.4",
+    "black>=24.0",
+]
+
+[project.scripts]
+tablevalidator-databricks = "tablevalidator_databricks.cli.main:app"
+
+[tool.setuptools.packages.find]
+include = ["tablevalidator_databricks", "tablevalidator_databricks.*"]
+
+[tool.setuptools.package-data]
+tablevalidator_databricks = ["templates/*.tmpl"]
+
+[tool.setuptools_scm]
+root = ".."
+# Distinct tag prefix ("databricks-vX.Y.Z") from table-validator's own
+# "vX.Y.Z" tags (table_validator/pyproject.toml's tag_regex) - the two
+# packages are versioned and released completely independently, and this
+# package's own publish workflow only triggers on this tag pattern (see
+# .github/workflows/publish-databricks.yml). Do not reuse table-validator's
+# tag_regex here - a shared pattern would make both packages' publish
+# workflows fire off the same tag.
+#
+# git_describe_command scopes `git describe` itself to only this package's
+# tag pattern (via --match) - without this, `git describe` finds the
+# repo's overall nearest tag (e.g. table-validator's own "v0.1.19") and
+# setuptools_scm >=8 raises ValueError rather than silently falling back
+# to fallback_version when that tag doesn't match tag_regex.
+git_describe_command = "git describe --dirty --tags --long --match 'databricks-v*.*.*'"
+tag_regex = "^databricks-v(?P<version>\\d+\\.\\d+\\.\\d+)$"
+fallback_version = "0.0.0"

tablevalidator_databricks-0.1.2/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

tablevalidator_databricks-0.1.2/tablevalidator_databricks/__init__.py

@@ -0,0 +1,15 @@
+"""tablevalidator_databricks: generates a widget-driven Databricks notebook
+UI on top of the table-validator package's validate_tables() API.
+
+This package contains no validation logic of its own - it only writes a
+notebook file whose cells call into table_validator's existing engine
+(see tablevalidator_databricks/templates/widget_notebook.py.tmpl)."""
+
+from importlib.metadata import PackageNotFoundError, version
+
+try:
+    __version__ = version("tablevalidator-databricks")
+except PackageNotFoundError:
+    __version__ = "0.0.0"
+
+__all__ = ["__version__"]

tablevalidator_databricks-0.1.2/tablevalidator_databricks/cli/main.py

@@ -0,0 +1,101 @@
+"""CLI entry point: `tablevalidator-databricks` console script."""
+
+from pathlib import Path
+
+import typer
+
+from tablevalidator_databricks.generator.generator import VALID_MODES, generate_notebook
+
+app = typer.Typer(
+    name="tablevalidator-databricks",
+    help="Generate a widget-driven Databricks notebook UI for table-validator.",
+    no_args_is_help=True,
+)
+
+DEFAULT_OUTPUT_PATH = Path("TableValidator.py")
+
+
+@app.command()
+def init(
+    output: Path = typer.Option(
+        DEFAULT_OUTPUT_PATH,
+        "--output",
+        help="Path to write the generated notebook to (default: ./TableValidator.py).",
+    ),
+    mode: str = typer.Option(
+        "full",
+        "--mode",
+        help=(
+            "basic: only the check-group multiselect and table pickers. "
+            "full (default): also shows optional filter widgets (only/ "
+            "ignore columns, row filter, primary key). "
+            "schema: pre-selects only the Catalog & Schema check group."
+        ),
+    ),
+) -> None:
+    """Generate a Databricks notebook with dbutils widgets for source/
+    target catalog.schema.table selection and validation checks - fill in
+    the widgets and run the notebook, no code to write.
+
+    Import the generated file into a Databricks workspace via
+    Workspace -> Import, with format set to "Source" (or "File" -> upload
+    directly, depending on your workspace UI version) - Databricks
+    recognizes the "# Databricks notebook source" header and cell
+    (# COMMAND ----------) markers and opens it as a real notebook, not a
+    plain text file.
+    """
+    if mode not in VALID_MODES:
+        typer.secho(
+            f"Invalid --mode {mode!r} - must be one of {', '.join(VALID_MODES)}.",
+            fg=typer.colors.RED,
+        )
+        raise typer.Exit(code=1)
+
+    generate_notebook(output, mode)
+
+    typer.echo(f"Generated notebook: {output.resolve()}")
+    typer.echo(
+        "Import it into Databricks (Workspace -> Import, format 'Source'), "
+        "fill in the widgets, and Run All."
+    )
+
+
+@app.command()
+def info() -> None:
+    """Show what this tool does and how to use the generated notebook."""
+    typer.echo(
+        "\n"
+        "tablevalidator-databricks - Databricks notebook UI for table-validator\n"
+        "------------------------------------------------------------------------\n"
+        "Generates a Databricks notebook with dbutils widgets (dynamic "
+        "catalog/schema/table dropdowns sourced from Unity Catalog, plus a "
+        "validation-checks multiselect) as a wrapper around the "
+        "table-validator package's validate_tables() API - no Python code "
+        "to write, no separate credentials to configure (the notebook "
+        "reuses its own ambient Spark session, same as validate_tables() "
+        "itself).\n"
+        "\n"
+        "Workflow:\n"
+        "\n"
+        "  1. tablevalidator-databricks init\n"
+        "     Writes a ready-to-use notebook (./TableValidator.py by "
+        "default). Use --output to write elsewhere, --mode to pick basic/"
+        "full/schema (see 'tablevalidator-databricks init --help').\n"
+        "\n"
+        "  2. Import the generated file into Databricks\n"
+        "     Workspace -> Import, format 'Source'.\n"
+        "\n"
+        "  3. Fill in the widgets and Run All\n"
+        "     Pick Source/Target Catalog, Schema, Table, and which checks "
+        "to run, then run the notebook top to bottom. Re-run the "
+        "'Create / refresh widgets' cell after changing a Catalog/Schema "
+        "selection, to repopulate the dropdowns below it (a Databricks "
+        "widget limitation, not a bug). Can't find a name in a dropdown? "
+        "Each dropdown has a '(manual override)' text widget next to it - "
+        "type the exact name there instead.\n"
+        "\n"
+        "This package contains no validation logic of its own - every "
+        "generated notebook calls directly into table-validator's own "
+        "validate_tables() engine, so results are identical to calling it "
+        "yourself in a code cell.\n"
+    )

tablevalidator_databricks-0.1.2/tablevalidator_databricks/generator/generator.py

@@ -0,0 +1,39 @@
+"""
+Renders the bundled widget_notebook.py.tmpl into a real Databricks
+notebook file. No Jinja2 - the template has exactly two placeholders,
+substituted via plain str.replace(), so a templating dependency isn't
+worth adding for this.
+"""
+
+from __future__ import annotations
+
+import importlib.resources
+from pathlib import Path
+
+VALID_MODES = ("basic", "full", "schema")
+
+
+def generate_notebook(output: Path, mode: str) -> None:
+    """Write a widget-driven Databricks notebook to `output`.
+
+    mode controls which optional widgets/pre-selected checks the notebook
+    starts with - see tablevalidator_databricks/cli/main.py's `init`
+    command docstring for what each mode means. The underlying comparison
+    logic is identical regardless of mode; only the widgets shown differ,
+    so all three modes render from the same template file rather than
+    three separate ones.
+    """
+    if mode not in VALID_MODES:
+        raise ValueError(f"mode must be one of {VALID_MODES}, got {mode!r}")
+
+    template = (
+        importlib.resources.files("tablevalidator_databricks.templates")
+        .joinpath("widget_notebook.py.tmpl")
+        .read_text(encoding="utf-8")
+    )
+    rendered = template.replace("{{MODE}}", mode).replace(
+        "{{SHOW_OPTIONAL_WIDGETS}}", "True" if mode == "full" else "False",
+    )
+
+    output.parent.mkdir(parents=True, exist_ok=True)
+    output.write_text(rendered, encoding="utf-8")