databricks-schema 0.6.1__tar.gz → 0.8.0__tar.gz

{databricks_schema-0.6.1 → databricks_schema-0.8.0}/.gitignore

@@ -13,4 +13,4 @@ wheels/
 .test_output/
 
 # Claude Code
-.claude/
+.claude/*

{databricks_schema-0.6.1 → databricks_schema-0.8.0}/CLAUDE.md

@@ -21,9 +21,9 @@ databricks_schema/
   models.py      # Pydantic v2 models: Catalog, Schema, Table, Column, PrimaryKey, ForeignKey
   extractor.py   # CatalogExtractor — wraps databricks-sdk
   yaml_io.py     # schema/catalog to/from YAML and JSON; _strip_empty removes None + empty collections
-  diff.py        # diff_schemas / diff_catalog_with_dir; FieldChange, ColumnDiff, TableDiff, SchemaDiff, CatalogDiff
+  diff.py        # diff_schemas / diff_catalogs / diff_catalog_with_dir / diff_schema_dirs; FieldChange, ColumnDiff, TableDiff, SchemaDiff, CatalogDiff
   sql_gen.py     # schema_diff_to_sql — pure SQL generation from SchemaDiff; no SDK/IO
-  cli.py         # argparse CLI: extract, diff, generate-sql, list-catalogs, list-schemas
+  cli.py         # argparse CLI: extract, diff, generate-sql, validate, diff-files, list-catalogs, list-schemas
   __init__.py    # public re-exports
 tests/
   test_models.py
@@ -31,11 +31,14 @@ tests/
   test_yaml_io.py
   test_diff.py       # pure model comparison, no SDK calls
   test_sql_gen.py    # pure SQL generation tests, no SDK calls
+  test_validate.py
+skills/databricks-schema-cli/SKILL.md  # Agent Skill: teaches an AI agent to drive the CLI
 ```
 
 ## Conventions
 
 - Package manager: `uv`; do not use `pip` directly
+- Version bumps: use `uv version --bump <major|minor|patch>` (updates `pyproject.toml` and `uv.lock` together) — do not hand-edit the version string
 - Ruff: select E, W, F, I, UP; line-length 100; target py311
 - Use `X | None` not `Optional[X]`; use `datetime.UTC` not `timezone.utc`
 - `from __future__ import annotations` in every module
@@ -50,8 +53,11 @@ tests/
 - CLI: catalog is a required positional argument (not a flag)
 - `extract` `--format`/`-f` selects output format (`yaml` default, `json` opt-in); dest is `fmt`
 - `diff` auto-detects format from files present in the directory; exits 2 on mixed YAML+JSON
+- `diff`'s second positional arg is a directory if one exists at that path, else a second catalog name (`diff_catalogs`); first `catalog` arg is always the live side
 - `TableType` is re-exported from `databricks.sdk.service.catalog` — do not redefine it
-- `--include-metadata` flag (on `extract`, `diff`, `generate-sql`) enables `owner` + `storage_location`; both are excluded by default
+- `--include-metadata` flag (on `extract`, `diff`, `generate-sql`, `diff-files`) adds `owner` to output/comparison; `extract` additionally includes `storage_location` in its output (not compared by `diff`/`generate-sql`/`diff-files`); excluded by default
+- `--include-tags` flag (on `extract`, `diff`, `generate-sql`) enables Unity Catalog tag lookups; excluded by default
+- `--quiet`/`-q` flag (on `extract`, `diff`, `generate-sql`, `diff-files`) raises the log level to suppress progress messages; errors always print
 - `diff` command exits 0 (no changes) or 1 (differences found) — useful in CI
 - Diff result types are dataclasses (not Pydantic); comparison functions are pure (no SDK calls)
 - `generate-sql` auto-detects format from files; exits 2 on mixed YAML+JSON or empty directory
@@ -59,3 +65,4 @@ tests/
 - `sql_gen.py` is pure (no SDK, no I/O); diff direction: `FieldChange.old` = stored (target), `.new` = live
 - FK refs in SQL use same catalog as the source table (`ref_schema` + `ref_table` from model)
 - Unsupported field changes (`table_type`) emit `-- TODO: unsupported change: …` comments
+- When CLI commands, flags, or output shape change, update `skills/databricks-schema-cli/SKILL.md` to match

{databricks_schema-0.6.1 → databricks_schema-0.8.0}/PKG-INFO

@@ -1,15 +1,18 @@
 Metadata-Version: 2.4
 Name: databricks-schema
-Version: 0.6.1
+Version: 0.8.0
 Summary: Databricks Unity Catalog schema extractor
+Project-URL: Homepage, https://github.com/frizzleqq/databricks-schema
 Project-URL: Repository, https://github.com/frizzleqq/databricks-schema
 Project-URL: Issues, https://github.com/frizzleqq/databricks-schema/issues
+Project-URL: Changelog, https://github.com/frizzleqq/databricks-schema/releases
 Author: frizzleqq
 License-Expression: Apache-2.0
 License-File: LICENSE
 Keywords: databricks,schema,sql,unity-catalog
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -22,11 +25,19 @@ Description-Content-Type: text/markdown
 
 # databricks-schema
 
-A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas as YAML files. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
+A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
 
-## Overview
+## Getting Started
 
-Extract a catalog to YAML files, then diff those files against a catalog — the same one to detect drift or a different one to compare environments (e.g. prod vs test):
+Install from PyPI (requires Python 3.11+):
+
+```bash
+pip install databricks-schema
+```
+
+Configure your Databricks credentials (see [Authentication](#authentication) for all supported methods).
+
+Then extract, diff, and generate SQL for your Unity Catalog schemas:
 
 ```bash
 # 1. Find the catalog you want to snapshot
@@ -42,11 +53,76 @@ databricks-schema diff test_catalog ./schemas/
 databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
 ```
 
-The YAML files act as a version-controllable snapshot of your schema. The `diff` command exits with code `1` when differences are found, making it suitable for CI pipelines.
+## Databricks Authentication
+
+The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
+
+```bash
+export DATABRICKS_HOST=https://<databricks-url>
+export DATABRICKS_TOKEN=<your-personal-access-token>
+```
+
+Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
+
+You can also pass credentials directly as flags using `--host` / `--token` below.
+
+## Development
+
+Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
+
+```bash
+git clone https://github.com/frizzleqq/databricks-schema
+cd databricks-schema
+uv sync --all-groups  # includes pytest and ruff
+```
+
+```bash
+# Run tests
+uv run pytest
+
+# Lint
+uv run ruff check databricks_schema/ tests/
+
+# Format
+uv run ruff format databricks_schema/ tests/
+```
+
+## Agent Skill
+
+This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
+[`skills/databricks-schema-cli/SKILL.md`](skills/databricks-schema-cli/SKILL.md)
+that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
+explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into
+your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls
+or queries.
+
+The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
+with [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv tool install databricks-schema
+```
+
+To use the skill in your own project without cloning this repo, download it directly into your
+project's `.claude/skills/`:
+
+```bash
+mkdir -p .claude/skills/databricks-schema-cli
+curl -o .claude/skills/databricks-schema-cli/SKILL.md \
+  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
+```
+
+For agentic harnesses other than Claude Code that look under `.agents/skills/` instead:
+
+```bash
+mkdir -p .agents/skills/databricks-schema-cli
+curl -o .agents/skills/databricks-schema-cli/SKILL.md \
+  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
+```
 
 ## Output Format
 
-Each schema is written to `{output-dir}/{schema-name}.yaml`. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
+Each schema is written to `{output-dir}/{schema-name}.yaml` if `--output-dir` is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
 
 ```yaml
 name: main
@@ -82,41 +158,46 @@ tables:
           - id
 ```
 
-## Installation
-
-Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
+Without `--output-dir`, `extract` prints a single `Catalog` document to stdout instead — the same schema shape nested under a `schemas` list, with the catalog name as a top-level field:
 
-```bash
-git clone <repo>
-cd databricks-schema
-uv sync
+```yaml
+name: prod_catalog
+schemas:
+  - name: main
+    comment: Main production schema
+    tags:
+      env: prod
+    tables: [...]
 ```
 
-For development (includes pytest and ruff):
+## CLI Usage
 
-```bash
-uv sync --all-groups
+```
+databricks-schema [OPTIONS] COMMAND [ARGS]...
 ```
 
-## Authentication
+`extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them;
+results and errors are unaffected.
 
-The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
+### `list-catalogs`
+
+List all accessible catalogs:
 
 ```bash
-export DATABRICKS_HOST=https://<databricks-url>
-export DATABRICKS_TOKEN=<your-personal-access-token>
+databricks-schema list-catalogs
 ```
 
-Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
-
-You can also pass credentials directly as flags (see `--host` / `--token` below).
+### `list-schemas`
 
-## CLI Usage
+List schemas in a catalog:
 
-```
-databricks-schema [OPTIONS] COMMAND [ARGS]...
+```bash
+databricks-schema list-schemas <catalog>
 ```
 
+Both print one name per line by default. Pass `--format json` / `-f json` (or `--format yaml`) to
+get the names as a JSON array or YAML list instead.
+
 ### `extract`
 
 Extract all schemas from a catalog to YAML files:
@@ -125,7 +206,11 @@ Extract all schemas from a catalog to YAML files:
 databricks-schema extract <catalog> --output-dir ./schemas/
 ```
 
-Use `--format json` to write `.json` files instead of `.yaml`.
+Print to stdout instead of writing files (no `--output-dir`) — the catalog and its schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
+
+```bash
+databricks-schema extract <catalog>
+```
 
 Extract specific schemas only:
 
@@ -133,16 +218,17 @@ Extract specific schemas only:
 databricks-schema extract <catalog> --schema main --schema raw --output-dir ./schemas/
 ```
 
-Print a single schema to stdout (no `--output-dir`):
+Or extract a single schema or table by naming it directly, dotted onto the catalog (output keeps the same `Catalog`/`Schema` shape, just filtered down). This cannot be combined with `--schema`:
 
 ```bash
-databricks-schema extract <catalog> --schema main
+databricks-schema extract <catalog>.<schema>
+databricks-schema extract <catalog>.<schema>.<table>
 ```
 
-Skip tag lookups for faster extraction (tags will be absent from output):
+Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):
 
 ```bash
-databricks-schema extract <catalog> --output-dir ./schemas/ --no-tags
+databricks-schema extract <catalog> --output-dir ./schemas/ --include-tags
 ```
 
 Include additional metadata (`owner`, `storage_location`) in the output:
@@ -151,7 +237,7 @@ Include additional metadata (`owner`, `storage_location`) in the output:
 databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
 ```
 
-Control the number of parallel workers (default: 4):
+Control the number of parallel workers (default: 4), useful for large schemas with many tags:
 
 ```bash
 databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
@@ -165,38 +251,51 @@ Compare the live catalog against previously extracted schema files (format auto-
 databricks-schema diff <catalog> ./schemas/
 ```
 
+Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
+
+```
+~ Schema: main [MODIFIED]
+  ~ Table: users [MODIFIED]
+    ~ Column: score [MODIFIED]
+        data_type: 'int' -> 'double'
+    + Column: phone [ADDED]
+  + Table: events [ADDED]
+- Schema: legacy [REMOVED]
+```
+
+Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
+
+Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
+
+```bash
+databricks-schema diff dev_catalog prod_catalog
+```
+
 Compare specific schemas only:
 
 ```bash
 databricks-schema diff <catalog> ./schemas/ --schema main --schema raw
 ```
 
-Skip tag lookups during comparison:
+Include Unity Catalog tags in the comparison (excluded by default):
 
 ```bash
-databricks-schema diff <catalog> ./schemas/ --no-tags
+databricks-schema diff <catalog> ./schemas/ --include-tags
 ```
 
-Include additional metadata (`owner`, `storage_location`) in the comparison:
+Include `owner` in the comparison (excluded by default):
 
 ```bash
 databricks-schema diff <catalog> ./schemas/ --include-metadata
 ```
 
-Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
+Get the same comparison as structured JSON or YAML instead of the `+`/`-`/`~` tree (exit codes unchanged) — useful for scripts or agents:
 
-```
-~ Schema: main [MODIFIED]
-  ~ Table: users [MODIFIED]
-    ~ Column: score [MODIFIED]
-        data_type: 'int' -> 'double'
-    + Column: phone [ADDED]
-  + Table: events [ADDED]
-- Schema: legacy [REMOVED]
+```bash
+databricks-schema diff <catalog> ./schemas/ --format json
+databricks-schema diff <catalog> ./schemas/ --format yaml
 ```
 
-Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
-
 ### `generate-sql`
 
 Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:
@@ -223,13 +322,13 @@ Filter to specific schemas:
 databricks-schema generate-sql <catalog> ./schemas/ --schema main --schema raw
 ```
 
-Skip tag lookups for faster comparison:
+Include Unity Catalog tags in the comparison (excluded by default):
 
 ```bash
-databricks-schema generate-sql <catalog> ./schemas/ --no-tags
+databricks-schema generate-sql <catalog> ./schemas/ --include-tags
 ```
 
-Include additional metadata (`owner`, `storage_location`) in the comparison:
+Include `owner` in the comparison (excluded by default):
 
 ```bash
 databricks-schema generate-sql <catalog> ./schemas/ --include-metadata
@@ -243,28 +342,28 @@ Validate local schema files for structural integrity (no Databricks connection n
 databricks-schema validate ./schemas/
 ```
 
+Pass `--format json` or `--format yaml` for an `{"issues": [...]}` document instead of `OK`/`ERROR:` text.
+
 ### `diff-files`
 
-Compare two local directories of schema files (no Databricks connection needed):
+Compare two local directories of schema files (no Databricks connection needed). Same output,
+exit codes, `--format json`/`yaml`, and `--schema` / `--include-metadata` flags as `diff`:
 
 ```bash
 databricks-schema diff-files ./schemas-prod/ ./schemas-test/
 ```
 
-### `list-catalogs`
+### `json-schema`
 
-List all accessible catalogs:
+Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection
+needed. Useful before parsing `--format json`/`yaml` output (or `extract`'s output)
+programmatically. Add `--format yaml` to render the schema itself as YAML:
 
 ```bash
-databricks-schema list-catalogs
-```
-
-### `list-schemas`
-
-List schemas in a catalog:
-
-```bash
-databricks-schema list-schemas <catalog>
+databricks-schema json-schema catalog    # extract's stdout Catalog document
+databricks-schema json-schema schema     # one per-schema extract file
+databricks-schema json-schema diff       # diff / diff-files --format json/yaml
+databricks-schema json-schema validate   # validate --format json/yaml
 ```
 
 ## Python Library Usage
@@ -274,14 +373,14 @@ See also the [`examples/`](examples/) directory for standalone scripts.
 ```python
 from pathlib import Path
 from databricks_schema import CatalogExtractor, catalog_to_yaml, schema_from_yaml
-from databricks_schema import diff_catalog_with_dir, diff_schemas, schema_diff_to_sql
+from databricks_schema import diff_catalog_with_dir, diff_catalogs, diff_schemas, schema_diff_to_sql
 
 # Extract using configured auth (max_workers controls parallel table extraction)
 extractor = CatalogExtractor(max_workers=4)
 catalog = extractor.extract_catalog("my_catalog", schema_filter=["main", "raw"])
 
-# Skip tag lookups for faster extraction
-catalog = extractor.extract_catalog("my_catalog", include_tags=False)
+# Include Unity Catalog tags (extra API calls per entity; excluded by default)
+catalog = extractor.extract_catalog("my_catalog", include_tags=True)
 
 # Include additional metadata (owner, storage_location)
 catalog = extractor.extract_catalog("my_catalog", include_metadata=True)
@@ -299,6 +398,10 @@ if result.has_changes:
     for schema_diff in result.schemas:
         print(schema_diff.name, schema_diff.status)
 
+# Compare two live catalogs directly
+prod_catalog = extractor.extract_catalog("prod_catalog")
+result = diff_catalogs(live=catalog, stored=prod_catalog)
+
 # Compare two Schema objects directly
 stored = schema_from_yaml(open("schemas/main.yaml").read())
 diff = diff_schemas(live=catalog.schemas[0], stored=stored)
@@ -307,16 +410,3 @@ diff = diff_schemas(live=catalog.schemas[0], stored=stored)
 sql = schema_diff_to_sql("my_catalog", diff, stored_schema=stored, allow_drop=False)
 print(sql)
 ```
-
-## Development
-
-```bash
-# Run tests
-uv run pytest
-
-# Lint
-uv run ruff check databricks_schema/ tests/
-
-# Format
-uv run ruff format databricks_schema/ tests/
-```