databricks-schema 0.6.1__tar.gz → 0.7.0__tar.gz

databricks_schema-0.7.0/.claude/skills/databricks-schema-cli/SKILL.md

@@ -0,0 +1,203 @@
+---
+name: databricks-schema-cli
+description: Explore, snapshot, diff, and generate migration SQL for Databricks Unity Catalog schemas using the `databricks-schema` CLI. Use this whenever the user wants to list catalogs/schemas in a Databricks workspace, pull a Unity Catalog schema (tables, columns, keys, tags) into version-controllable YAML/JSON, compare a live catalog against stored schema files or two schema directories against each other, validate schema files, or generate Spark SQL DDL to reconcile a catalog with a desired state — even if they just say "what's in this Databricks catalog" or "diff prod vs what's checked in" without naming the tool.
+---
+
+# Databricks Unity Catalog exploration (`databricks-schema`)
+
+`databricks-schema` is a CLI that talks to a Databricks workspace via the Databricks SDK and
+represents Unity Catalog schemas as YAML/JSON files — one file per schema. Use it instead of
+writing ad-hoc SDK calls whenever the task is "look at / snapshot / diff / generate SQL for" a
+Unity Catalog schema.
+
+Call it as `databricks-schema <command> ...`. If the command isn't found, install it first (see
+`README.md`, e.g. `uv tool install databricks-schema`). Run `databricks-schema <command> --help`
+to confirm exact flags before relying on this document for anything unusual — this skill covers
+the common path, not every flag.
+
+## Authentication
+
+Don't pass `--host` / `--token` yourself. Just run the commands — the CLI resolves credentials
+on its own via the Databricks SDK, from `DATABRICKS_HOST`/`DATABRICKS_TOKEN` env vars, an active
+`databricks auth login` session, or a profile in `~/.databrickscfg`, whichever is already set up
+in the environment.
+
+If a command fails with an `Unauthenticated`/`PermissionDenied` error, that means none of those
+are configured (or the credential lacks access) — tell the user auth is missing/insufficient
+rather than trying to fix it yourself. Don't confuse this with a `NotFound` error, which means
+the catalog/schema name itself is wrong.
+
+## Reducing noise for agentic use
+
+`extract`, `diff`, `generate-sql`, and `diff-files` print progress lines (e.g. `Extracting
+catalog 'X'...`, `Comparing catalog 'X' against ...`) to stderr before/while they run. Pass
+`--quiet` / `-q` to suppress them — the actual result (extracted YAML/JSON, diff tree, generated
+SQL, exit code) is unaffected; only the informational narration is dropped. Errors are always
+printed regardless of `--quiet`.
+
+## Orienting yourself in a workspace
+
+```bash
+databricks-schema list-catalogs              # what catalogs can I see?
+databricks-schema list-schemas <catalog>      # what schemas are in this catalog?
+```
+
+Both just print names, one per line — good for a quick scan or for building a `--schema` filter
+list for the commands below.
+
+## Pulling a schema into a readable snapshot
+
+```bash
+# Printed straight to stdout as one Catalog document (no --output-dir) — good for
+# "show me schema X" or "show me this whole catalog"
+databricks-schema extract <catalog> --schema main
+databricks-schema extract <catalog>
+
+# Whole catalog (or a filtered set of schemas) written to one file per schema
+databricks-schema extract <catalog> --output-dir ./schemas/
+databricks-schema extract <catalog> --schema main --schema raw --output-dir ./schemas/
+
+# A single schema or table can also be named directly, dotted onto the catalog —
+# output keeps the same Catalog/Schema shape, just filtered down. Cannot combine
+# with --schema.
+databricks-schema extract <catalog>.<schema>
+databricks-schema extract <catalog>.<schema>.<table>
+```
+
+Notes:
+- `--schema` / `-s` is repeatable; omit it to extract every schema.
+- `<catalog>.<schema>` and `<catalog>.<schema>.<table>` are shortcuts for filtering to one schema
+  or one table; they exit with an error (code 2) if combined with `--schema`.
+- `--output-dir` writes one file per schema; omitting it prints one `Catalog` document to stdout
+  (`name` + `schemas: [...]`) covering all matching schemas.
+- `--format json` / `-f json` writes `.json` instead of `.yaml` (same structure either way).
+- `--include-metadata` adds `owner` and `storage_location` (excluded by default — smaller, more
+  diffable output).
+- `--include-tags` adds Unity Catalog tags (excluded by default — extra API calls per entity).
+- `--workers N` controls parallel table extraction (default 4); raise it for large catalogs.
+
+### Shape of the output
+
+Each schema file looks like this (fields with no value are omitted entirely):
+
+```yaml
+name: main
+comment: Main production schema
+tags:
+  env: prod
+tables:
+  - name: users
+    table_type: MANAGED
+    comment: User accounts
+    tags:
+      domain: identity
+    columns:
+      - name: id
+        data_type: bigint
+        nullable: false
+        comment: Primary key
+      - name: email
+        data_type: string
+    primary_key:
+      name: pk_users
+      columns: [id]
+    foreign_keys:
+      - name: fk_org
+        columns: [org_id]
+        ref_schema: orgs
+        ref_table: organizations
+        ref_columns: [id]
+```
+
+Read this directly to answer questions about a catalog's structure — column names/types,
+nullability, primary/foreign keys, per-object tags — without needing SDK calls of your own.
+Foreign keys reference `ref_schema` + `ref_table` only (same catalog as the source table).
+
+Stdout output (no `--output-dir`) wraps this in a `Catalog` document instead — same schema shape,
+nested under `schemas`, with the catalog name at the top level:
+
+```yaml
+name: prod_catalog
+schemas:
+  - name: main
+    comment: Main production schema
+    tags:
+      env: prod
+    tables: [...]
+```
+
+## Comparing catalog state
+
+```bash
+# Live catalog vs. a directory of stored schema files (format auto-detected: YAML or JSON)
+databricks-schema diff <catalog> ./schemas/
+
+# Live catalog vs. another live catalog (e.g. dev vs. prod) — no local files needed
+databricks-schema diff dev_catalog prod_catalog
+
+# Two local directories, no Databricks connection needed
+databricks-schema diff-files ./schemas-old/ ./schemas-new/
+```
+
+For `diff`, the second argument is a directory path if one exists on disk, otherwise it's read as
+a second catalog name and both catalogs are extracted live. Either way, the first `catalog`
+argument is the "live"/actual side and the second (`target`) is the baseline/reference side —
+that ordering decides which side of each diff shows as `+`/`-`.
+
+Both print a tree with `+` (added), `-` (removed), `~` (modified) markers, e.g.:
+
+```
+~ Schema: main [MODIFIED]
+  ~ Table: users [MODIFIED]
+    ~ Column: score [MODIFIED]
+        data_type: 'int' -> 'double'
+    + Column: phone [ADDED]
+  + Table: events [ADDED]
+- Schema: legacy [REMOVED]
+```
+
+**Exit codes matter — use them instead of parsing stdout when you just need a yes/no:**
+- `0` — no differences
+- `1` — differences found (this is normal, not a failure)
+- `2` — usage error (bad directory, mixed YAML+JSON in one directory, no schema files found)
+
+Same `--schema` and `--include-tags` flags apply as for `extract`. `--include-metadata` here only
+adds `owner` to the comparison — `extract`'s `storage_location` isn't diffed.
+
+## Validating schema files
+
+```bash
+databricks-schema validate ./schemas/
+```
+
+Checks structural integrity of local YAML/JSON files with no Databricks connection (e.g. after
+hand-editing one). Exits `0` and prints `OK — N schema(s) validated` on success, `1` with an
+`ERROR:` line per issue otherwise.
+
+## Generating migration SQL
+
+```bash
+databricks-schema generate-sql <catalog> ./schemas/                       # print to stdout
+databricks-schema generate-sql <catalog> ./schemas/ --output-dir ./migrations/  # one .sql per schema
+```
+
+Produces Databricks Spark SQL DDL to bring the *live* catalog in line with the *stored* files
+(create/alter tables and columns, add/drop keys, etc.). Destructive statements (`DROP SCHEMA`,
+`DROP TABLE`, `DROP COLUMN`) are emitted as commented-out SQL by default — pass `--allow-drop` to
+make them executable. Treat this as a review-then-run step, not something to pipe straight into
+execution, especially with `--allow-drop`. Unsupported changes (e.g. `table_type`) show up as
+`-- TODO: unsupported change: ...` comments rather than being silently dropped.
+
+Same `--schema`, `--include-tags`, `--include-metadata` filters apply as for `diff`.
+
+## Choosing the right command
+
+| Want to...                                              | Command        |
+|-----------------------------------------------------------|----------------|
+| See what catalogs/schemas exist                           | `list-catalogs`, `list-schemas` |
+| Read/summarize a live schema's structure                  | `extract` (no `--output-dir`, use `--schema`) |
+| Snapshot a catalog for version control                    | `extract --output-dir` |
+| Check if a catalog drifted from a checked-in snapshot      | `diff` |
+| Check if two snapshot directories differ (no live access) | `diff-files` |
+| Sanity-check hand-edited YAML/JSON before using it         | `validate` |
+| Produce SQL to reconcile live catalog with a snapshot      | `generate-sql` |

{databricks_schema-0.6.1 → databricks_schema-0.7.0}/.gitignore

@@ -13,4 +13,5 @@ wheels/
 .test_output/
 
 # Claude Code
-.claude/
+.claude/*
+!.claude/skills/

{databricks_schema-0.6.1 → databricks_schema-0.7.0}/CLAUDE.md

@@ -21,9 +21,9 @@ databricks_schema/
   models.py      # Pydantic v2 models: Catalog, Schema, Table, Column, PrimaryKey, ForeignKey
   extractor.py   # CatalogExtractor — wraps databricks-sdk
   yaml_io.py     # schema/catalog to/from YAML and JSON; _strip_empty removes None + empty collections
-  diff.py        # diff_schemas / diff_catalog_with_dir; FieldChange, ColumnDiff, TableDiff, SchemaDiff, CatalogDiff
+  diff.py        # diff_schemas / diff_catalogs / diff_catalog_with_dir / diff_schema_dirs; FieldChange, ColumnDiff, TableDiff, SchemaDiff, CatalogDiff
   sql_gen.py     # schema_diff_to_sql — pure SQL generation from SchemaDiff; no SDK/IO
-  cli.py         # argparse CLI: extract, diff, generate-sql, list-catalogs, list-schemas
+  cli.py         # argparse CLI: extract, diff, generate-sql, validate, diff-files, list-catalogs, list-schemas
   __init__.py    # public re-exports
 tests/
   test_models.py
@@ -31,6 +31,8 @@ tests/
   test_yaml_io.py
   test_diff.py       # pure model comparison, no SDK calls
   test_sql_gen.py    # pure SQL generation tests, no SDK calls
+  test_validate.py
+.claude/skills/databricks-schema-cli/SKILL.md  # Agent Skill: teaches an AI agent to drive the CLI
 ```
 
 ## Conventions
@@ -50,8 +52,11 @@ tests/
 - CLI: catalog is a required positional argument (not a flag)
 - `extract` `--format`/`-f` selects output format (`yaml` default, `json` opt-in); dest is `fmt`
 - `diff` auto-detects format from files present in the directory; exits 2 on mixed YAML+JSON
+- `diff`'s second positional arg is a directory if one exists at that path, else a second catalog name (`diff_catalogs`); first `catalog` arg is always the live side
 - `TableType` is re-exported from `databricks.sdk.service.catalog` — do not redefine it
-- `--include-metadata` flag (on `extract`, `diff`, `generate-sql`) enables `owner` + `storage_location`; both are excluded by default
+- `--include-metadata` flag (on `extract`, `diff`, `generate-sql`, `diff-files`) adds `owner` to output/comparison; `extract` additionally includes `storage_location` in its output (not compared by `diff`/`generate-sql`/`diff-files`); excluded by default
+- `--include-tags` flag (on `extract`, `diff`, `generate-sql`) enables Unity Catalog tag lookups; excluded by default
+- `--quiet`/`-q` flag (on `extract`, `diff`, `generate-sql`, `diff-files`) raises the log level to suppress progress messages; errors always print
 - `diff` command exits 0 (no changes) or 1 (differences found) — useful in CI
 - Diff result types are dataclasses (not Pydantic); comparison functions are pure (no SDK calls)
 - `generate-sql` auto-detects format from files; exits 2 on mixed YAML+JSON or empty directory
@@ -59,3 +64,4 @@ tests/
 - `sql_gen.py` is pure (no SDK, no I/O); diff direction: `FieldChange.old` = stored (target), `.new` = live
 - FK refs in SQL use same catalog as the source table (`ref_schema` + `ref_table` from model)
 - Unsupported field changes (`table_type`) emit `-- TODO: unsupported change: …` comments
+- When CLI commands, flags, or output shape change, update `.claude/skills/databricks-schema-cli/SKILL.md` to match

{databricks_schema-0.6.1 → databricks_schema-0.7.0}/PKG-INFO

@@ -1,15 +1,18 @@
 Metadata-Version: 2.4
 Name: databricks-schema
-Version: 0.6.1
+Version: 0.7.0
 Summary: Databricks Unity Catalog schema extractor
+Project-URL: Homepage, https://github.com/frizzleqq/databricks-schema
 Project-URL: Repository, https://github.com/frizzleqq/databricks-schema
 Project-URL: Issues, https://github.com/frizzleqq/databricks-schema/issues
+Project-URL: Changelog, https://github.com/frizzleqq/databricks-schema/releases
 Author: frizzleqq
 License-Expression: Apache-2.0
 License-File: LICENSE
 Keywords: databricks,schema,sql,unity-catalog
 Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: Apache Software License
 Classifier: Programming Language :: Python :: 3
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
@@ -24,9 +27,17 @@ Description-Content-Type: text/markdown
 
 A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas as YAML files. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
 
-## Overview
+## Getting Started
 
-Extract a catalog to YAML files, then diff those files against a catalog — the same one to detect drift or a different one to compare environments (e.g. prod vs test):
+Install from PyPI (requires Python 3.11+):
+
+```bash
+pip install databricks-schema
+```
+
+Configure your Databricks credentials (see [Authentication](#authentication) for all supported methods).
+
+Then extract, diff, and generate SQL for your Unity Catalog schemas:
 
 ```bash
 # 1. Find the catalog you want to snapshot
@@ -82,20 +93,16 @@ tables:
           - id
 ```
 
-## Installation
+Without `--output-dir`, `extract` prints a single `Catalog` document to stdout instead — the same schema shape nested under a `schemas` list, with the catalog name as a top-level field:
 
-Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
-
-```bash
-git clone <repo>
-cd databricks-schema
-uv sync
-```
-
-For development (includes pytest and ruff):
-
-```bash
-uv sync --all-groups
+```yaml
+name: prod_catalog
+schemas:
+  - name: main
+    comment: Main production schema
+    tags:
+      env: prod
+    tables: [...]
 ```
 
 ## Authentication
@@ -111,12 +118,61 @@ Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/prof
 
 You can also pass credentials directly as flags (see `--host` / `--token` below).
 
+## Agent Skill
+
+This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
+[`.claude/skills/databricks-schema-cli/SKILL.md`](.claude/skills/databricks-schema-cli/SKILL.md)
+that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
+explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Point an agent at
+this repo (or copy the skill into your own project) and it can use the CLI directly instead of
+writing ad-hoc Databricks SDK calls.
+
+The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
+with [uv](https://github.com/astral-sh/uv):
+
+```bash
+uv tool install databricks-schema
+```
+
+To use the skill in your own project without cloning this repo, download it directly:
+
+```bash
+mkdir -p .claude/skills/databricks-schema-cli
+curl -o .claude/skills/databricks-schema-cli/SKILL.md \
+  https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/.claude/skills/databricks-schema-cli/SKILL.md
+```
+
+## Development
+
+Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
+
+```bash
+git clone https://github.com/frizzleqq/databricks-schema
+cd databricks-schema
+uv sync --all-groups  # includes pytest and ruff
+```
+
+```bash
+# Run tests
+uv run pytest
+
+# Lint
+uv run ruff check databricks_schema/ tests/
+
+# Format
+uv run ruff format databricks_schema/ tests/
+```
+
 ## CLI Usage
 
 ```
 databricks-schema [OPTIONS] COMMAND [ARGS]...
 ```
 
+`extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting
+catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them; results and
+errors are unaffected.
+
 ### `extract`
 
 Extract all schemas from a catalog to YAML files:
@@ -133,16 +189,23 @@ Extract specific schemas only:
 databricks-schema extract <catalog> --schema main --schema raw --output-dir ./schemas/
 ```
 
-Print a single schema to stdout (no `--output-dir`):
+Or extract a single schema or table by naming it directly, dotted onto the catalog (output keeps the same `Catalog`/`Schema` shape, just filtered down). This cannot be combined with `--schema`:
+
+```bash
+databricks-schema extract <catalog>.<schema>
+databricks-schema extract <catalog>.<schema>.<table>
+```
+
+Print to stdout instead of writing files (no `--output-dir`) — the catalog and its matching schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
 
 ```bash
 databricks-schema extract <catalog> --schema main
 ```
 
-Skip tag lookups for faster extraction (tags will be absent from output):
+Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):
 
 ```bash
-databricks-schema extract <catalog> --output-dir ./schemas/ --no-tags
+databricks-schema extract <catalog> --output-dir ./schemas/ --include-tags
 ```
 
 Include additional metadata (`owner`, `storage_location`) in the output:
@@ -165,19 +228,25 @@ Compare the live catalog against previously extracted schema files (format auto-
 databricks-schema diff <catalog> ./schemas/
 ```
 
+Or compare two live catalogs directly, e.g. dev against prod — no local files needed. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
+
+```bash
+databricks-schema diff dev_catalog prod_catalog
+```
+
 Compare specific schemas only:
 
 ```bash
 databricks-schema diff <catalog> ./schemas/ --schema main --schema raw
 ```
 
-Skip tag lookups during comparison:
+Include Unity Catalog tags in the comparison (excluded by default):
 
 ```bash
-databricks-schema diff <catalog> ./schemas/ --no-tags
+databricks-schema diff <catalog> ./schemas/ --include-tags
 ```
 
-Include additional metadata (`owner`, `storage_location`) in the comparison:
+Include `owner` in the comparison (excluded by default):
 
 ```bash
 databricks-schema diff <catalog> ./schemas/ --include-metadata
@@ -223,13 +292,13 @@ Filter to specific schemas:
 databricks-schema generate-sql <catalog> ./schemas/ --schema main --schema raw
 ```
 
-Skip tag lookups for faster comparison:
+Include Unity Catalog tags in the comparison (excluded by default):
 
 ```bash
-databricks-schema generate-sql <catalog> ./schemas/ --no-tags
+databricks-schema generate-sql <catalog> ./schemas/ --include-tags
 ```
 
-Include additional metadata (`owner`, `storage_location`) in the comparison:
+Include `owner` in the comparison (excluded by default):
 
 ```bash
 databricks-schema generate-sql <catalog> ./schemas/ --include-metadata
@@ -245,7 +314,8 @@ databricks-schema validate ./schemas/
 
 ### `diff-files`
 
-Compare two local directories of schema files (no Databricks connection needed):
+Compare two local directories of schema files (no Databricks connection needed). Same output,
+exit codes, and `--schema` / `--include-metadata` flags as `diff`:
 
 ```bash
 databricks-schema diff-files ./schemas-prod/ ./schemas-test/
@@ -274,14 +344,14 @@ See also the [`examples/`](examples/) directory for standalone scripts.
 ```python
 from pathlib import Path
 from databricks_schema import CatalogExtractor, catalog_to_yaml, schema_from_yaml
-from databricks_schema import diff_catalog_with_dir, diff_schemas, schema_diff_to_sql
+from databricks_schema import diff_catalog_with_dir, diff_catalogs, diff_schemas, schema_diff_to_sql
 
 # Extract using configured auth (max_workers controls parallel table extraction)
 extractor = CatalogExtractor(max_workers=4)
 catalog = extractor.extract_catalog("my_catalog", schema_filter=["main", "raw"])
 
-# Skip tag lookups for faster extraction
-catalog = extractor.extract_catalog("my_catalog", include_tags=False)
+# Include Unity Catalog tags (extra API calls per entity; excluded by default)
+catalog = extractor.extract_catalog("my_catalog", include_tags=True)
 
 # Include additional metadata (owner, storage_location)
 catalog = extractor.extract_catalog("my_catalog", include_metadata=True)
@@ -299,6 +369,10 @@ if result.has_changes:
     for schema_diff in result.schemas:
         print(schema_diff.name, schema_diff.status)
 
+# Compare two live catalogs directly
+prod_catalog = extractor.extract_catalog("prod_catalog")
+result = diff_catalogs(live=catalog, stored=prod_catalog)
+
 # Compare two Schema objects directly
 stored = schema_from_yaml(open("schemas/main.yaml").read())
 diff = diff_schemas(live=catalog.schemas[0], stored=stored)
@@ -307,16 +381,3 @@ diff = diff_schemas(live=catalog.schemas[0], stored=stored)
 sql = schema_diff_to_sql("my_catalog", diff, stored_schema=stored, allow_drop=False)
 print(sql)
 ```
-
-## Development
-
-```bash
-# Run tests
-uv run pytest
-
-# Lint
-uv run ruff check databricks_schema/ tests/
-
-# Format
-uv run ruff format databricks_schema/ tests/
-```