databricks-schema 0.7.0__tar.gz → 0.8.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (32) hide show
  1. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.gitignore +0 -1
  2. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/CLAUDE.md +3 -2
  3. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/PKG-INFO +118 -89
  4. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/README.md +117 -88
  5. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/cli.py +259 -34
  6. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/diff.py +16 -1
  7. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/pyproject.toml +1 -1
  8. databricks_schema-0.8.0/skills/databricks-schema-cli/SKILL.md +200 -0
  9. databricks_schema-0.8.0/tests/test_cli.py +416 -0
  10. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_diff.py +56 -0
  11. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/uv.lock +1 -1
  12. databricks_schema-0.7.0/.claude/skills/databricks-schema-cli/SKILL.md +0 -203
  13. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.github/workflows/ci.yml +0 -0
  14. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.github/workflows/release.yml +0 -0
  15. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.python-version +0 -0
  16. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/LICENSE +0 -0
  17. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/__init__.py +0 -0
  18. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/extractor.py +0 -0
  19. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/models.py +0 -0
  20. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/sql_gen.py +0 -0
  21. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/validate.py +0 -0
  22. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/yaml_io.py +0 -0
  23. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/01_extract_schema.py +0 -0
  24. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/02_save_load_yaml.py +0 -0
  25. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/03_diff_schemas.py +0 -0
  26. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/04_generate_sql.py +0 -0
  27. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/__init__.py +0 -0
  28. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_extractor.py +0 -0
  29. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_models.py +0 -0
  30. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_sql_gen.py +0 -0
  31. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_validate.py +0 -0
  32. {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_yaml_io.py +0 -0
@@ -14,4 +14,3 @@ wheels/
14
14
 
15
15
  # Claude Code
16
16
  .claude/*
17
- !.claude/skills/
@@ -32,12 +32,13 @@ tests/
32
32
  test_diff.py # pure model comparison, no SDK calls
33
33
  test_sql_gen.py # pure SQL generation tests, no SDK calls
34
34
  test_validate.py
35
- .claude/skills/databricks-schema-cli/SKILL.md # Agent Skill: teaches an AI agent to drive the CLI
35
+ skills/databricks-schema-cli/SKILL.md # Agent Skill: teaches an AI agent to drive the CLI
36
36
  ```
37
37
 
38
38
  ## Conventions
39
39
 
40
40
  - Package manager: `uv`; do not use `pip` directly
41
+ - Version bumps: use `uv version --bump <major|minor|patch>` (updates `pyproject.toml` and `uv.lock` together) — do not hand-edit the version string
41
42
  - Ruff: select E, W, F, I, UP; line-length 100; target py311
42
43
  - Use `X | None` not `Optional[X]`; use `datetime.UTC` not `timezone.utc`
43
44
  - `from __future__ import annotations` in every module
@@ -64,4 +65,4 @@ tests/
64
65
  - `sql_gen.py` is pure (no SDK, no I/O); diff direction: `FieldChange.old` = stored (target), `.new` = live
65
66
  - FK refs in SQL use same catalog as the source table (`ref_schema` + `ref_table` from model)
66
67
  - Unsupported field changes (`table_type`) emit `-- TODO: unsupported change: …` comments
67
- - When CLI commands, flags, or output shape change, update `.claude/skills/databricks-schema-cli/SKILL.md` to match
68
+ - When CLI commands, flags, or output shape change, update `skills/databricks-schema-cli/SKILL.md` to match
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: databricks-schema
3
- Version: 0.7.0
3
+ Version: 0.8.0
4
4
  Summary: Databricks Unity Catalog schema extractor
5
5
  Project-URL: Homepage, https://github.com/frizzleqq/databricks-schema
6
6
  Project-URL: Repository, https://github.com/frizzleqq/databricks-schema
@@ -25,7 +25,7 @@ Description-Content-Type: text/markdown
25
25
 
26
26
  # databricks-schema
27
27
 
28
- A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas as YAML files. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
28
+ A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
29
29
 
30
30
  ## Getting Started
31
31
 
@@ -53,11 +53,76 @@ databricks-schema diff test_catalog ./schemas/
53
53
  databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
54
54
  ```
55
55
 
56
- The YAML files act as a version-controllable snapshot of your schema. The `diff` command exits with code `1` when differences are found, making it suitable for CI pipelines.
56
+ ## Databricks Authentication
57
+
58
+ The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
59
+
60
+ ```bash
61
+ export DATABRICKS_HOST=https://<databricks-url>
62
+ export DATABRICKS_TOKEN=<your-personal-access-token>
63
+ ```
64
+
65
+ Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
66
+
67
+ You can also pass credentials directly as flags using `--host` / `--token` below.
68
+
69
+ ## Development
70
+
71
+ Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
72
+
73
+ ```bash
74
+ git clone https://github.com/frizzleqq/databricks-schema
75
+ cd databricks-schema
76
+ uv sync --all-groups # includes pytest and ruff
77
+ ```
78
+
79
+ ```bash
80
+ # Run tests
81
+ uv run pytest
82
+
83
+ # Lint
84
+ uv run ruff check databricks_schema/ tests/
85
+
86
+ # Format
87
+ uv run ruff format databricks_schema/ tests/
88
+ ```
89
+
90
+ ## Agent Skill
91
+
92
+ This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
93
+ [`skills/databricks-schema-cli/SKILL.md`](skills/databricks-schema-cli/SKILL.md)
94
+ that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
95
+ explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into
96
+ your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls
97
+ or queries.
98
+
99
+ The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
100
+ with [uv](https://github.com/astral-sh/uv):
101
+
102
+ ```bash
103
+ uv tool install databricks-schema
104
+ ```
105
+
106
+ To use the skill in your own project without cloning this repo, download it directly into your
107
+ project's `.claude/skills/`:
108
+
109
+ ```bash
110
+ mkdir -p .claude/skills/databricks-schema-cli
111
+ curl -o .claude/skills/databricks-schema-cli/SKILL.md \
112
+ https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
113
+ ```
114
+
115
+ For agentic harnesses other than Claude Code that look under `.agents/skills/` instead:
116
+
117
+ ```bash
118
+ mkdir -p .agents/skills/databricks-schema-cli
119
+ curl -o .agents/skills/databricks-schema-cli/SKILL.md \
120
+ https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
121
+ ```
57
122
 
58
123
  ## Output Format
59
124
 
60
- Each schema is written to `{output-dir}/{schema-name}.yaml`. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
125
+ Each schema is written to `{output-dir}/{schema-name}.yaml` if `--output-dir` is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
61
126
 
62
127
  ```yaml
63
128
  name: main
@@ -105,73 +170,33 @@ schemas:
105
170
  tables: [...]
106
171
  ```
107
172
 
108
- ## Authentication
109
-
110
- The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
173
+ ## CLI Usage
111
174
 
112
- ```bash
113
- export DATABRICKS_HOST=https://<databricks-url>
114
- export DATABRICKS_TOKEN=<your-personal-access-token>
115
175
  ```
116
-
117
- Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
118
-
119
- You can also pass credentials directly as flags (see `--host` / `--token` below).
120
-
121
- ## Agent Skill
122
-
123
- This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
124
- [`.claude/skills/databricks-schema-cli/SKILL.md`](.claude/skills/databricks-schema-cli/SKILL.md)
125
- that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
126
- explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Point an agent at
127
- this repo (or copy the skill into your own project) and it can use the CLI directly instead of
128
- writing ad-hoc Databricks SDK calls.
129
-
130
- The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
131
- with [uv](https://github.com/astral-sh/uv):
132
-
133
- ```bash
134
- uv tool install databricks-schema
176
+ databricks-schema [OPTIONS] COMMAND [ARGS]...
135
177
  ```
136
178
 
137
- To use the skill in your own project without cloning this repo, download it directly:
179
+ `extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them;
180
+ results and errors are unaffected.
138
181
 
139
- ```bash
140
- mkdir -p .claude/skills/databricks-schema-cli
141
- curl -o .claude/skills/databricks-schema-cli/SKILL.md \
142
- https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/.claude/skills/databricks-schema-cli/SKILL.md
143
- ```
144
-
145
- ## Development
182
+ ### `list-catalogs`
146
183
 
147
- Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
184
+ List all accessible catalogs:
148
185
 
149
186
  ```bash
150
- git clone https://github.com/frizzleqq/databricks-schema
151
- cd databricks-schema
152
- uv sync --all-groups # includes pytest and ruff
187
+ databricks-schema list-catalogs
153
188
  ```
154
189
 
155
- ```bash
156
- # Run tests
157
- uv run pytest
158
-
159
- # Lint
160
- uv run ruff check databricks_schema/ tests/
161
-
162
- # Format
163
- uv run ruff format databricks_schema/ tests/
164
- ```
190
+ ### `list-schemas`
165
191
 
166
- ## CLI Usage
192
+ List schemas in a catalog:
167
193
 
168
- ```
169
- databricks-schema [OPTIONS] COMMAND [ARGS]...
194
+ ```bash
195
+ databricks-schema list-schemas <catalog>
170
196
  ```
171
197
 
172
- `extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting
173
- catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them; results and
174
- errors are unaffected.
198
+ Both print one name per line by default. Pass `--format json` / `-f json` (or `--format yaml`) to
199
+ get the names as a JSON array or YAML list instead.
175
200
 
176
201
  ### `extract`
177
202
 
@@ -181,7 +206,11 @@ Extract all schemas from a catalog to YAML files:
181
206
  databricks-schema extract <catalog> --output-dir ./schemas/
182
207
  ```
183
208
 
184
- Use `--format json` to write `.json` files instead of `.yaml`.
209
+ Print to stdout instead of writing files (no `--output-dir`) — the catalog and its schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
210
+
211
+ ```bash
212
+ databricks-schema extract <catalog>
213
+ ```
185
214
 
186
215
  Extract specific schemas only:
187
216
 
@@ -196,12 +225,6 @@ databricks-schema extract <catalog>.<schema>
196
225
  databricks-schema extract <catalog>.<schema>.<table>
197
226
  ```
198
227
 
199
- Print to stdout instead of writing files (no `--output-dir`) — the catalog and its matching schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
200
-
201
- ```bash
202
- databricks-schema extract <catalog> --schema main
203
- ```
204
-
205
228
  Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):
206
229
 
207
230
  ```bash
@@ -214,7 +237,7 @@ Include additional metadata (`owner`, `storage_location`) in the output:
214
237
  databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
215
238
  ```
216
239
 
217
- Control the number of parallel workers (default: 4):
240
+ Control the number of parallel workers (default: 4), useful for large schemas with many tags:
218
241
 
219
242
  ```bash
220
243
  databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
@@ -228,7 +251,21 @@ Compare the live catalog against previously extracted schema files (format auto-
228
251
  databricks-schema diff <catalog> ./schemas/
229
252
  ```
230
253
 
231
- Or compare two live catalogs directly, e.g. dev against prod — no local files needed. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
254
+ Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
255
+
256
+ ```
257
+ ~ Schema: main [MODIFIED]
258
+ ~ Table: users [MODIFIED]
259
+ ~ Column: score [MODIFIED]
260
+ data_type: 'int' -> 'double'
261
+ + Column: phone [ADDED]
262
+ + Table: events [ADDED]
263
+ - Schema: legacy [REMOVED]
264
+ ```
265
+
266
+ Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
267
+
268
+ Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
232
269
 
233
270
  ```bash
234
271
  databricks-schema diff dev_catalog prod_catalog
@@ -252,20 +289,13 @@ Include `owner` in the comparison (excluded by default):
252
289
  databricks-schema diff <catalog> ./schemas/ --include-metadata
253
290
  ```
254
291
 
255
- Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
292
+ Get the same comparison as structured JSON or YAML instead of the `+`/`-`/`~` tree (exit codes unchanged) — useful for scripts or agents:
256
293
 
257
- ```
258
- ~ Schema: main [MODIFIED]
259
- ~ Table: users [MODIFIED]
260
- ~ Column: score [MODIFIED]
261
- data_type: 'int' -> 'double'
262
- + Column: phone [ADDED]
263
- + Table: events [ADDED]
264
- - Schema: legacy [REMOVED]
294
+ ```bash
295
+ databricks-schema diff <catalog> ./schemas/ --format json
296
+ databricks-schema diff <catalog> ./schemas/ --format yaml
265
297
  ```
266
298
 
267
- Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
268
-
269
299
  ### `generate-sql`
270
300
 
271
301
  Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:
@@ -312,29 +342,28 @@ Validate local schema files for structural integrity (no Databricks connection n
312
342
  databricks-schema validate ./schemas/
313
343
  ```
314
344
 
345
+ Pass `--format json` or `--format yaml` for an `{"issues": [...]}` document instead of `OK`/`ERROR:` text.
346
+
315
347
  ### `diff-files`
316
348
 
317
349
  Compare two local directories of schema files (no Databricks connection needed). Same output,
318
- exit codes, and `--schema` / `--include-metadata` flags as `diff`:
350
+ exit codes, `--format json`/`yaml`, and `--schema` / `--include-metadata` flags as `diff`:
319
351
 
320
352
  ```bash
321
353
  databricks-schema diff-files ./schemas-prod/ ./schemas-test/
322
354
  ```
323
355
 
324
- ### `list-catalogs`
325
-
326
- List all accessible catalogs:
327
-
328
- ```bash
329
- databricks-schema list-catalogs
330
- ```
331
-
332
- ### `list-schemas`
356
+ ### `json-schema`
333
357
 
334
- List schemas in a catalog:
358
+ Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection
359
+ needed. Useful before parsing `--format json`/`yaml` output (or `extract`'s output)
360
+ programmatically. Add `--format yaml` to render the schema itself as YAML:
335
361
 
336
362
  ```bash
337
- databricks-schema list-schemas <catalog>
363
+ databricks-schema json-schema catalog # extract's stdout Catalog document
364
+ databricks-schema json-schema schema # one per-schema extract file
365
+ databricks-schema json-schema diff # diff / diff-files --format json/yaml
366
+ databricks-schema json-schema validate # validate --format json/yaml
338
367
  ```
339
368
 
340
369
  ## Python Library Usage
@@ -1,6 +1,6 @@
1
1
  # databricks-schema
2
2
 
3
- A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas as YAML files. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
3
+ A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
4
4
 
5
5
  ## Getting Started
6
6
 
@@ -28,11 +28,76 @@ databricks-schema diff test_catalog ./schemas/
28
28
  databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
29
29
  ```
30
30
 
31
- The YAML files act as a version-controllable snapshot of your schema. The `diff` command exits with code `1` when differences are found, making it suitable for CI pipelines.
31
+ ## Databricks Authentication
32
+
33
+ The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
34
+
35
+ ```bash
36
+ export DATABRICKS_HOST=https://<databricks-url>
37
+ export DATABRICKS_TOKEN=<your-personal-access-token>
38
+ ```
39
+
40
+ Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
41
+
42
+ You can also pass credentials directly as flags using `--host` / `--token` below.
43
+
44
+ ## Development
45
+
46
+ Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
47
+
48
+ ```bash
49
+ git clone https://github.com/frizzleqq/databricks-schema
50
+ cd databricks-schema
51
+ uv sync --all-groups # includes pytest and ruff
52
+ ```
53
+
54
+ ```bash
55
+ # Run tests
56
+ uv run pytest
57
+
58
+ # Lint
59
+ uv run ruff check databricks_schema/ tests/
60
+
61
+ # Format
62
+ uv run ruff format databricks_schema/ tests/
63
+ ```
64
+
65
+ ## Agent Skill
66
+
67
+ This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
68
+ [`skills/databricks-schema-cli/SKILL.md`](skills/databricks-schema-cli/SKILL.md)
69
+ that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
70
+ explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into
71
+ your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls
72
+ or queries.
73
+
74
+ The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
75
+ with [uv](https://github.com/astral-sh/uv):
76
+
77
+ ```bash
78
+ uv tool install databricks-schema
79
+ ```
80
+
81
+ To use the skill in your own project without cloning this repo, download it directly into your
82
+ project's `.claude/skills/`:
83
+
84
+ ```bash
85
+ mkdir -p .claude/skills/databricks-schema-cli
86
+ curl -o .claude/skills/databricks-schema-cli/SKILL.md \
87
+ https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
88
+ ```
89
+
90
+ For agentic harnesses other than Claude Code that look under `.agents/skills/` instead:
91
+
92
+ ```bash
93
+ mkdir -p .agents/skills/databricks-schema-cli
94
+ curl -o .agents/skills/databricks-schema-cli/SKILL.md \
95
+ https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
96
+ ```
32
97
 
33
98
  ## Output Format
34
99
 
35
- Each schema is written to `{output-dir}/{schema-name}.yaml`. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
100
+ Each schema is written to `{output-dir}/{schema-name}.yaml` if `--output-dir` is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
36
101
 
37
102
  ```yaml
38
103
  name: main
@@ -80,73 +145,33 @@ schemas:
80
145
  tables: [...]
81
146
  ```
82
147
 
83
- ## Authentication
84
-
85
- The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
148
+ ## CLI Usage
86
149
 
87
- ```bash
88
- export DATABRICKS_HOST=https://<databricks-url>
89
- export DATABRICKS_TOKEN=<your-personal-access-token>
90
150
  ```
91
-
92
- Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
93
-
94
- You can also pass credentials directly as flags (see `--host` / `--token` below).
95
-
96
- ## Agent Skill
97
-
98
- This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
99
- [`.claude/skills/databricks-schema-cli/SKILL.md`](.claude/skills/databricks-schema-cli/SKILL.md)
100
- that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
101
- explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Point an agent at
102
- this repo (or copy the skill into your own project) and it can use the CLI directly instead of
103
- writing ad-hoc Databricks SDK calls.
104
-
105
- The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
106
- with [uv](https://github.com/astral-sh/uv):
107
-
108
- ```bash
109
- uv tool install databricks-schema
151
+ databricks-schema [OPTIONS] COMMAND [ARGS]...
110
152
  ```
111
153
 
112
- To use the skill in your own project without cloning this repo, download it directly:
154
+ `extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them;
155
+ results and errors are unaffected.
113
156
 
114
- ```bash
115
- mkdir -p .claude/skills/databricks-schema-cli
116
- curl -o .claude/skills/databricks-schema-cli/SKILL.md \
117
- https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/.claude/skills/databricks-schema-cli/SKILL.md
118
- ```
119
-
120
- ## Development
157
+ ### `list-catalogs`
121
158
 
122
- Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
159
+ List all accessible catalogs:
123
160
 
124
161
  ```bash
125
- git clone https://github.com/frizzleqq/databricks-schema
126
- cd databricks-schema
127
- uv sync --all-groups # includes pytest and ruff
162
+ databricks-schema list-catalogs
128
163
  ```
129
164
 
130
- ```bash
131
- # Run tests
132
- uv run pytest
133
-
134
- # Lint
135
- uv run ruff check databricks_schema/ tests/
136
-
137
- # Format
138
- uv run ruff format databricks_schema/ tests/
139
- ```
165
+ ### `list-schemas`
140
166
 
141
- ## CLI Usage
167
+ List schemas in a catalog:
142
168
 
143
- ```
144
- databricks-schema [OPTIONS] COMMAND [ARGS]...
169
+ ```bash
170
+ databricks-schema list-schemas <catalog>
145
171
  ```
146
172
 
147
- `extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting
148
- catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them; results and
149
- errors are unaffected.
173
+ Both print one name per line by default. Pass `--format json` / `-f json` (or `--format yaml`) to
174
+ get the names as a JSON array or YAML list instead.
150
175
 
151
176
  ### `extract`
152
177
 
@@ -156,7 +181,11 @@ Extract all schemas from a catalog to YAML files:
156
181
  databricks-schema extract <catalog> --output-dir ./schemas/
157
182
  ```
158
183
 
159
- Use `--format json` to write `.json` files instead of `.yaml`.
184
+ Print to stdout instead of writing files (no `--output-dir`) — the catalog and its schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
185
+
186
+ ```bash
187
+ databricks-schema extract <catalog>
188
+ ```
160
189
 
161
190
  Extract specific schemas only:
162
191
 
@@ -171,12 +200,6 @@ databricks-schema extract <catalog>.<schema>
171
200
  databricks-schema extract <catalog>.<schema>.<table>
172
201
  ```
173
202
 
174
- Print to stdout instead of writing files (no `--output-dir`) — the catalog and its matching schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
175
-
176
- ```bash
177
- databricks-schema extract <catalog> --schema main
178
- ```
179
-
180
203
  Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):
181
204
 
182
205
  ```bash
@@ -189,7 +212,7 @@ Include additional metadata (`owner`, `storage_location`) in the output:
189
212
  databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
190
213
  ```
191
214
 
192
- Control the number of parallel workers (default: 4):
215
+ Control the number of parallel workers (default: 4), useful for large schemas with many tags:
193
216
 
194
217
  ```bash
195
218
  databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
@@ -203,7 +226,21 @@ Compare the live catalog against previously extracted schema files (format auto-
203
226
  databricks-schema diff <catalog> ./schemas/
204
227
  ```
205
228
 
206
- Or compare two live catalogs directly, e.g. dev against prod — no local files needed. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
229
+ Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
230
+
231
+ ```
232
+ ~ Schema: main [MODIFIED]
233
+ ~ Table: users [MODIFIED]
234
+ ~ Column: score [MODIFIED]
235
+ data_type: 'int' -> 'double'
236
+ + Column: phone [ADDED]
237
+ + Table: events [ADDED]
238
+ - Schema: legacy [REMOVED]
239
+ ```
240
+
241
+ Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
242
+
243
+ Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
207
244
 
208
245
  ```bash
209
246
  databricks-schema diff dev_catalog prod_catalog
@@ -227,20 +264,13 @@ Include `owner` in the comparison (excluded by default):
227
264
  databricks-schema diff <catalog> ./schemas/ --include-metadata
228
265
  ```
229
266
 
230
- Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
267
+ Get the same comparison as structured JSON or YAML instead of the `+`/`-`/`~` tree (exit codes unchanged) — useful for scripts or agents:
231
268
 
232
- ```
233
- ~ Schema: main [MODIFIED]
234
- ~ Table: users [MODIFIED]
235
- ~ Column: score [MODIFIED]
236
- data_type: 'int' -> 'double'
237
- + Column: phone [ADDED]
238
- + Table: events [ADDED]
239
- - Schema: legacy [REMOVED]
269
+ ```bash
270
+ databricks-schema diff <catalog> ./schemas/ --format json
271
+ databricks-schema diff <catalog> ./schemas/ --format yaml
240
272
  ```
241
273
 
242
- Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
243
-
244
274
  ### `generate-sql`
245
275
 
246
276
  Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:
@@ -287,29 +317,28 @@ Validate local schema files for structural integrity (no Databricks connection n
287
317
  databricks-schema validate ./schemas/
288
318
  ```
289
319
 
320
+ Pass `--format json` or `--format yaml` for an `{"issues": [...]}` document instead of `OK`/`ERROR:` text.
321
+
290
322
  ### `diff-files`
291
323
 
292
324
  Compare two local directories of schema files (no Databricks connection needed). Same output,
293
- exit codes, and `--schema` / `--include-metadata` flags as `diff`:
325
+ exit codes, `--format json`/`yaml`, and `--schema` / `--include-metadata` flags as `diff`:
294
326
 
295
327
  ```bash
296
328
  databricks-schema diff-files ./schemas-prod/ ./schemas-test/
297
329
  ```
298
330
 
299
- ### `list-catalogs`
300
-
301
- List all accessible catalogs:
302
-
303
- ```bash
304
- databricks-schema list-catalogs
305
- ```
306
-
307
- ### `list-schemas`
331
+ ### `json-schema`
308
332
 
309
- List schemas in a catalog:
333
+ Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection
334
+ needed. Useful before parsing `--format json`/`yaml` output (or `extract`'s output)
335
+ programmatically. Add `--format yaml` to render the schema itself as YAML:
310
336
 
311
337
  ```bash
312
- databricks-schema list-schemas <catalog>
338
+ databricks-schema json-schema catalog # extract's stdout Catalog document
339
+ databricks-schema json-schema schema # one per-schema extract file
340
+ databricks-schema json-schema diff # diff / diff-files --format json/yaml
341
+ databricks-schema json-schema validate # validate --format json/yaml
313
342
  ```
314
343
 
315
344
  ## Python Library Usage