databricks-schema 0.7.0__tar.gz → 0.8.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.gitignore +0 -1
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/CLAUDE.md +3 -2
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/PKG-INFO +118 -89
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/README.md +117 -88
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/cli.py +259 -34
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/diff.py +16 -1
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/pyproject.toml +1 -1
- databricks_schema-0.8.0/skills/databricks-schema-cli/SKILL.md +200 -0
- databricks_schema-0.8.0/tests/test_cli.py +416 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_diff.py +56 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/uv.lock +1 -1
- databricks_schema-0.7.0/.claude/skills/databricks-schema-cli/SKILL.md +0 -203
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.github/workflows/ci.yml +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.github/workflows/release.yml +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/.python-version +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/LICENSE +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/__init__.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/extractor.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/models.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/sql_gen.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/validate.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/databricks_schema/yaml_io.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/01_extract_schema.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/02_save_load_yaml.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/03_diff_schemas.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/examples/04_generate_sql.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/__init__.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_extractor.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_models.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_sql_gen.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_validate.py +0 -0
- {databricks_schema-0.7.0 → databricks_schema-0.8.0}/tests/test_yaml_io.py +0 -0
|
@@ -32,12 +32,13 @@ tests/
|
|
|
32
32
|
test_diff.py # pure model comparison, no SDK calls
|
|
33
33
|
test_sql_gen.py # pure SQL generation tests, no SDK calls
|
|
34
34
|
test_validate.py
|
|
35
|
-
|
|
35
|
+
skills/databricks-schema-cli/SKILL.md # Agent Skill: teaches an AI agent to drive the CLI
|
|
36
36
|
```
|
|
37
37
|
|
|
38
38
|
## Conventions
|
|
39
39
|
|
|
40
40
|
- Package manager: `uv`; do not use `pip` directly
|
|
41
|
+
- Version bumps: use `uv version --bump <major|minor|patch>` (updates `pyproject.toml` and `uv.lock` together) — do not hand-edit the version string
|
|
41
42
|
- Ruff: select E, W, F, I, UP; line-length 100; target py311
|
|
42
43
|
- Use `X | None` not `Optional[X]`; use `datetime.UTC` not `timezone.utc`
|
|
43
44
|
- `from __future__ import annotations` in every module
|
|
@@ -64,4 +65,4 @@ tests/
|
|
|
64
65
|
- `sql_gen.py` is pure (no SDK, no I/O); diff direction: `FieldChange.old` = stored (target), `.new` = live
|
|
65
66
|
- FK refs in SQL use same catalog as the source table (`ref_schema` + `ref_table` from model)
|
|
66
67
|
- Unsupported field changes (`table_type`) emit `-- TODO: unsupported change: …` comments
|
|
67
|
-
- When CLI commands, flags, or output shape change, update
|
|
68
|
+
- When CLI commands, flags, or output shape change, update `skills/databricks-schema-cli/SKILL.md` to match
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: databricks-schema
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.8.0
|
|
4
4
|
Summary: Databricks Unity Catalog schema extractor
|
|
5
5
|
Project-URL: Homepage, https://github.com/frizzleqq/databricks-schema
|
|
6
6
|
Project-URL: Repository, https://github.com/frizzleqq/databricks-schema
|
|
@@ -25,7 +25,7 @@ Description-Content-Type: text/markdown
|
|
|
25
25
|
|
|
26
26
|
# databricks-schema
|
|
27
27
|
|
|
28
|
-
A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas
|
|
28
|
+
A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
|
|
29
29
|
|
|
30
30
|
## Getting Started
|
|
31
31
|
|
|
@@ -53,11 +53,76 @@ databricks-schema diff test_catalog ./schemas/
|
|
|
53
53
|
databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
|
|
54
54
|
```
|
|
55
55
|
|
|
56
|
-
|
|
56
|
+
## Databricks Authentication
|
|
57
|
+
|
|
58
|
+
The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
|
|
59
|
+
|
|
60
|
+
```bash
|
|
61
|
+
export DATABRICKS_HOST=https://<databricks-url>
|
|
62
|
+
export DATABRICKS_TOKEN=<your-personal-access-token>
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
|
|
66
|
+
|
|
67
|
+
You can also pass credentials directly as flags using `--host` / `--token` below.
|
|
68
|
+
|
|
69
|
+
## Development
|
|
70
|
+
|
|
71
|
+
Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
|
|
72
|
+
|
|
73
|
+
```bash
|
|
74
|
+
git clone https://github.com/frizzleqq/databricks-schema
|
|
75
|
+
cd databricks-schema
|
|
76
|
+
uv sync --all-groups # includes pytest and ruff
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
```bash
|
|
80
|
+
# Run tests
|
|
81
|
+
uv run pytest
|
|
82
|
+
|
|
83
|
+
# Lint
|
|
84
|
+
uv run ruff check databricks_schema/ tests/
|
|
85
|
+
|
|
86
|
+
# Format
|
|
87
|
+
uv run ruff format databricks_schema/ tests/
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Agent Skill
|
|
91
|
+
|
|
92
|
+
This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
|
|
93
|
+
[`skills/databricks-schema-cli/SKILL.md`](skills/databricks-schema-cli/SKILL.md)
|
|
94
|
+
that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
|
|
95
|
+
explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into
|
|
96
|
+
your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls
|
|
97
|
+
or queries.
|
|
98
|
+
|
|
99
|
+
The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
|
|
100
|
+
with [uv](https://github.com/astral-sh/uv):
|
|
101
|
+
|
|
102
|
+
```bash
|
|
103
|
+
uv tool install databricks-schema
|
|
104
|
+
```
|
|
105
|
+
|
|
106
|
+
To use the skill in your own project without cloning this repo, download it directly into your
|
|
107
|
+
project's `.claude/skills/`:
|
|
108
|
+
|
|
109
|
+
```bash
|
|
110
|
+
mkdir -p .claude/skills/databricks-schema-cli
|
|
111
|
+
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
|
|
112
|
+
https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
|
|
113
|
+
```
|
|
114
|
+
|
|
115
|
+
For agentic harnesses other than Claude Code that look under `.agents/skills/` instead:
|
|
116
|
+
|
|
117
|
+
```bash
|
|
118
|
+
mkdir -p .agents/skills/databricks-schema-cli
|
|
119
|
+
curl -o .agents/skills/databricks-schema-cli/SKILL.md \
|
|
120
|
+
https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
|
|
121
|
+
```
|
|
57
122
|
|
|
58
123
|
## Output Format
|
|
59
124
|
|
|
60
|
-
Each schema is written to `{output-dir}/{schema-name}.yaml
|
|
125
|
+
Each schema is written to `{output-dir}/{schema-name}.yaml` if `--output-dir` is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
|
|
61
126
|
|
|
62
127
|
```yaml
|
|
63
128
|
name: main
|
|
@@ -105,73 +170,33 @@ schemas:
|
|
|
105
170
|
tables: [...]
|
|
106
171
|
```
|
|
107
172
|
|
|
108
|
-
##
|
|
109
|
-
|
|
110
|
-
The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
|
|
173
|
+
## CLI Usage
|
|
111
174
|
|
|
112
|
-
```bash
|
|
113
|
-
export DATABRICKS_HOST=https://<databricks-url>
|
|
114
|
-
export DATABRICKS_TOKEN=<your-personal-access-token>
|
|
115
175
|
```
|
|
116
|
-
|
|
117
|
-
Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
|
|
118
|
-
|
|
119
|
-
You can also pass credentials directly as flags (see `--host` / `--token` below).
|
|
120
|
-
|
|
121
|
-
## Agent Skill
|
|
122
|
-
|
|
123
|
-
This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
|
|
124
|
-
[`.claude/skills/databricks-schema-cli/SKILL.md`](.claude/skills/databricks-schema-cli/SKILL.md)
|
|
125
|
-
that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
|
|
126
|
-
explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Point an agent at
|
|
127
|
-
this repo (or copy the skill into your own project) and it can use the CLI directly instead of
|
|
128
|
-
writing ad-hoc Databricks SDK calls.
|
|
129
|
-
|
|
130
|
-
The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
|
|
131
|
-
with [uv](https://github.com/astral-sh/uv):
|
|
132
|
-
|
|
133
|
-
```bash
|
|
134
|
-
uv tool install databricks-schema
|
|
176
|
+
databricks-schema [OPTIONS] COMMAND [ARGS]...
|
|
135
177
|
```
|
|
136
178
|
|
|
137
|
-
|
|
179
|
+
`extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them;
|
|
180
|
+
results and errors are unaffected.
|
|
138
181
|
|
|
139
|
-
|
|
140
|
-
mkdir -p .claude/skills/databricks-schema-cli
|
|
141
|
-
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
|
|
142
|
-
https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/.claude/skills/databricks-schema-cli/SKILL.md
|
|
143
|
-
```
|
|
144
|
-
|
|
145
|
-
## Development
|
|
182
|
+
### `list-catalogs`
|
|
146
183
|
|
|
147
|
-
|
|
184
|
+
List all accessible catalogs:
|
|
148
185
|
|
|
149
186
|
```bash
|
|
150
|
-
|
|
151
|
-
cd databricks-schema
|
|
152
|
-
uv sync --all-groups # includes pytest and ruff
|
|
187
|
+
databricks-schema list-catalogs
|
|
153
188
|
```
|
|
154
189
|
|
|
155
|
-
|
|
156
|
-
# Run tests
|
|
157
|
-
uv run pytest
|
|
158
|
-
|
|
159
|
-
# Lint
|
|
160
|
-
uv run ruff check databricks_schema/ tests/
|
|
161
|
-
|
|
162
|
-
# Format
|
|
163
|
-
uv run ruff format databricks_schema/ tests/
|
|
164
|
-
```
|
|
190
|
+
### `list-schemas`
|
|
165
191
|
|
|
166
|
-
|
|
192
|
+
List schemas in a catalog:
|
|
167
193
|
|
|
168
|
-
```
|
|
169
|
-
databricks-schema
|
|
194
|
+
```bash
|
|
195
|
+
databricks-schema list-schemas <catalog>
|
|
170
196
|
```
|
|
171
197
|
|
|
172
|
-
|
|
173
|
-
|
|
174
|
-
errors are unaffected.
|
|
198
|
+
Both print one name per line by default. Pass `--format json` / `-f json` (or `--format yaml`) to
|
|
199
|
+
get the names as a JSON array or YAML list instead.
|
|
175
200
|
|
|
176
201
|
### `extract`
|
|
177
202
|
|
|
@@ -181,7 +206,11 @@ Extract all schemas from a catalog to YAML files:
|
|
|
181
206
|
databricks-schema extract <catalog> --output-dir ./schemas/
|
|
182
207
|
```
|
|
183
208
|
|
|
184
|
-
|
|
209
|
+
Print to stdout instead of writing files (no `--output-dir`) — the catalog and its schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
|
|
210
|
+
|
|
211
|
+
```bash
|
|
212
|
+
databricks-schema extract <catalog>
|
|
213
|
+
```
|
|
185
214
|
|
|
186
215
|
Extract specific schemas only:
|
|
187
216
|
|
|
@@ -196,12 +225,6 @@ databricks-schema extract <catalog>.<schema>
|
|
|
196
225
|
databricks-schema extract <catalog>.<schema>.<table>
|
|
197
226
|
```
|
|
198
227
|
|
|
199
|
-
Print to stdout instead of writing files (no `--output-dir`) — the catalog and its matching schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
|
|
200
|
-
|
|
201
|
-
```bash
|
|
202
|
-
databricks-schema extract <catalog> --schema main
|
|
203
|
-
```
|
|
204
|
-
|
|
205
228
|
Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):
|
|
206
229
|
|
|
207
230
|
```bash
|
|
@@ -214,7 +237,7 @@ Include additional metadata (`owner`, `storage_location`) in the output:
|
|
|
214
237
|
databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
|
|
215
238
|
```
|
|
216
239
|
|
|
217
|
-
Control the number of parallel workers (default: 4):
|
|
240
|
+
Control the number of parallel workers (default: 4), useful for large schemas with many tags:
|
|
218
241
|
|
|
219
242
|
```bash
|
|
220
243
|
databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
|
|
@@ -228,7 +251,21 @@ Compare the live catalog against previously extracted schema files (format auto-
|
|
|
228
251
|
databricks-schema diff <catalog> ./schemas/
|
|
229
252
|
```
|
|
230
253
|
|
|
231
|
-
|
|
254
|
+
Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
|
|
255
|
+
|
|
256
|
+
```
|
|
257
|
+
~ Schema: main [MODIFIED]
|
|
258
|
+
~ Table: users [MODIFIED]
|
|
259
|
+
~ Column: score [MODIFIED]
|
|
260
|
+
data_type: 'int' -> 'double'
|
|
261
|
+
+ Column: phone [ADDED]
|
|
262
|
+
+ Table: events [ADDED]
|
|
263
|
+
- Schema: legacy [REMOVED]
|
|
264
|
+
```
|
|
265
|
+
|
|
266
|
+
Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
|
|
267
|
+
|
|
268
|
+
Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
|
|
232
269
|
|
|
233
270
|
```bash
|
|
234
271
|
databricks-schema diff dev_catalog prod_catalog
|
|
@@ -252,20 +289,13 @@ Include `owner` in the comparison (excluded by default):
|
|
|
252
289
|
databricks-schema diff <catalog> ./schemas/ --include-metadata
|
|
253
290
|
```
|
|
254
291
|
|
|
255
|
-
|
|
292
|
+
Get the same comparison as structured JSON or YAML instead of the `+`/`-`/`~` tree (exit codes unchanged) — useful for scripts or agents:
|
|
256
293
|
|
|
257
|
-
```
|
|
258
|
-
|
|
259
|
-
|
|
260
|
-
~ Column: score [MODIFIED]
|
|
261
|
-
data_type: 'int' -> 'double'
|
|
262
|
-
+ Column: phone [ADDED]
|
|
263
|
-
+ Table: events [ADDED]
|
|
264
|
-
- Schema: legacy [REMOVED]
|
|
294
|
+
```bash
|
|
295
|
+
databricks-schema diff <catalog> ./schemas/ --format json
|
|
296
|
+
databricks-schema diff <catalog> ./schemas/ --format yaml
|
|
265
297
|
```
|
|
266
298
|
|
|
267
|
-
Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
|
|
268
|
-
|
|
269
299
|
### `generate-sql`
|
|
270
300
|
|
|
271
301
|
Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:
|
|
@@ -312,29 +342,28 @@ Validate local schema files for structural integrity (no Databricks connection n
|
|
|
312
342
|
databricks-schema validate ./schemas/
|
|
313
343
|
```
|
|
314
344
|
|
|
345
|
+
Pass `--format json` or `--format yaml` for an `{"issues": [...]}` document instead of `OK`/`ERROR:` text.
|
|
346
|
+
|
|
315
347
|
### `diff-files`
|
|
316
348
|
|
|
317
349
|
Compare two local directories of schema files (no Databricks connection needed). Same output,
|
|
318
|
-
exit codes, and `--schema` / `--include-metadata` flags as `diff`:
|
|
350
|
+
exit codes, `--format json`/`yaml`, and `--schema` / `--include-metadata` flags as `diff`:
|
|
319
351
|
|
|
320
352
|
```bash
|
|
321
353
|
databricks-schema diff-files ./schemas-prod/ ./schemas-test/
|
|
322
354
|
```
|
|
323
355
|
|
|
324
|
-
### `
|
|
325
|
-
|
|
326
|
-
List all accessible catalogs:
|
|
327
|
-
|
|
328
|
-
```bash
|
|
329
|
-
databricks-schema list-catalogs
|
|
330
|
-
```
|
|
331
|
-
|
|
332
|
-
### `list-schemas`
|
|
356
|
+
### `json-schema`
|
|
333
357
|
|
|
334
|
-
|
|
358
|
+
Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection
|
|
359
|
+
needed. Useful before parsing `--format json`/`yaml` output (or `extract`'s output)
|
|
360
|
+
programmatically. Add `--format yaml` to render the schema itself as YAML:
|
|
335
361
|
|
|
336
362
|
```bash
|
|
337
|
-
databricks-schema
|
|
363
|
+
databricks-schema json-schema catalog # extract's stdout Catalog document
|
|
364
|
+
databricks-schema json-schema schema # one per-schema extract file
|
|
365
|
+
databricks-schema json-schema diff # diff / diff-files --format json/yaml
|
|
366
|
+
databricks-schema json-schema validate # validate --format json/yaml
|
|
338
367
|
```
|
|
339
368
|
|
|
340
369
|
## Python Library Usage
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
# databricks-schema
|
|
2
2
|
|
|
3
|
-
A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas
|
|
3
|
+
A CLI tool and Python library that uses the Databricks SDK to extract and diff Unity Catalog schemas in YAML format. It can also generate Databricks Spark SQL to apply schema changes across catalogs.
|
|
4
4
|
|
|
5
5
|
## Getting Started
|
|
6
6
|
|
|
@@ -28,11 +28,76 @@ databricks-schema diff test_catalog ./schemas/
|
|
|
28
28
|
databricks-schema generate-sql test_catalog ./schemas/ --output-dir ./migrations/
|
|
29
29
|
```
|
|
30
30
|
|
|
31
|
-
|
|
31
|
+
## Databricks Authentication
|
|
32
|
+
|
|
33
|
+
The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
export DATABRICKS_HOST=https://<databricks-url>
|
|
37
|
+
export DATABRICKS_TOKEN=<your-personal-access-token>
|
|
38
|
+
```
|
|
39
|
+
|
|
40
|
+
Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
|
|
41
|
+
|
|
42
|
+
You can also pass credentials directly as flags using `--host` / `--token` below.
|
|
43
|
+
|
|
44
|
+
## Development
|
|
45
|
+
|
|
46
|
+
Requires Python 3.11+ and [uv](https://github.com/astral-sh/uv).
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
git clone https://github.com/frizzleqq/databricks-schema
|
|
50
|
+
cd databricks-schema
|
|
51
|
+
uv sync --all-groups # includes pytest and ruff
|
|
52
|
+
```
|
|
53
|
+
|
|
54
|
+
```bash
|
|
55
|
+
# Run tests
|
|
56
|
+
uv run pytest
|
|
57
|
+
|
|
58
|
+
# Lint
|
|
59
|
+
uv run ruff check databricks_schema/ tests/
|
|
60
|
+
|
|
61
|
+
# Format
|
|
62
|
+
uv run ruff format databricks_schema/ tests/
|
|
63
|
+
```
|
|
64
|
+
|
|
65
|
+
## Agent Skill
|
|
66
|
+
|
|
67
|
+
This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
|
|
68
|
+
[`skills/databricks-schema-cli/SKILL.md`](skills/databricks-schema-cli/SKILL.md)
|
|
69
|
+
that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
|
|
70
|
+
explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Copy the skill into
|
|
71
|
+
your own project and it can use the CLI directly instead of writing ad-hoc Databricks SDK calls
|
|
72
|
+
or queries.
|
|
73
|
+
|
|
74
|
+
The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
|
|
75
|
+
with [uv](https://github.com/astral-sh/uv):
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
uv tool install databricks-schema
|
|
79
|
+
```
|
|
80
|
+
|
|
81
|
+
To use the skill in your own project without cloning this repo, download it directly into your
|
|
82
|
+
project's `.claude/skills/`:
|
|
83
|
+
|
|
84
|
+
```bash
|
|
85
|
+
mkdir -p .claude/skills/databricks-schema-cli
|
|
86
|
+
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
|
|
87
|
+
https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
For agentic harnesses other than Claude Code that look under `.agents/skills/` instead:
|
|
91
|
+
|
|
92
|
+
```bash
|
|
93
|
+
mkdir -p .agents/skills/databricks-schema-cli
|
|
94
|
+
curl -o .agents/skills/databricks-schema-cli/SKILL.md \
|
|
95
|
+
https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/skills/databricks-schema-cli/SKILL.md
|
|
96
|
+
```
|
|
32
97
|
|
|
33
98
|
## Output Format
|
|
34
99
|
|
|
35
|
-
Each schema is written to `{output-dir}/{schema-name}.yaml
|
|
100
|
+
Each schema is written to `{output-dir}/{schema-name}.yaml` if `--output-dir` is specified. Fields with no value (null comments, empty tag dicts, empty FK lists) are omitted. Use `--format json` to write `.json` files with the same structure.
|
|
36
101
|
|
|
37
102
|
```yaml
|
|
38
103
|
name: main
|
|
@@ -80,73 +145,33 @@ schemas:
|
|
|
80
145
|
tables: [...]
|
|
81
146
|
```
|
|
82
147
|
|
|
83
|
-
##
|
|
84
|
-
|
|
85
|
-
The tool uses the [Databricks SDK](https://github.com/databricks/databricks-sdk-py) for auth. Configure it via environment variables:
|
|
148
|
+
## CLI Usage
|
|
86
149
|
|
|
87
|
-
```bash
|
|
88
|
-
export DATABRICKS_HOST=https://<databricks-url>
|
|
89
|
-
export DATABRICKS_TOKEN=<your-personal-access-token>
|
|
90
150
|
```
|
|
91
|
-
|
|
92
|
-
Or use a [Databricks CLI profile](https://docs.databricks.com/dev-tools/cli/profiles.html) (`~/.databrickscfg`) — the SDK will pick it up automatically.
|
|
93
|
-
|
|
94
|
-
You can also pass credentials directly as flags (see `--host` / `--token` below).
|
|
95
|
-
|
|
96
|
-
## Agent Skill
|
|
97
|
-
|
|
98
|
-
This repo ships an [Agent Skill](https://code.claude.com/docs/en/skills) at
|
|
99
|
-
[`.claude/skills/databricks-schema-cli/SKILL.md`](.claude/skills/databricks-schema-cli/SKILL.md)
|
|
100
|
-
that teaches an AI coding agent (e.g. Claude Code) how to use the `databricks-schema` CLI to
|
|
101
|
-
explore, snapshot, diff, and generate migration SQL for a live Unity Catalog. Point an agent at
|
|
102
|
-
this repo (or copy the skill into your own project) and it can use the CLI directly instead of
|
|
103
|
-
writing ad-hoc Databricks SDK calls.
|
|
104
|
-
|
|
105
|
-
The skill assumes the `databricks-schema` command is on `PATH`. Install it as a standalone tool
|
|
106
|
-
with [uv](https://github.com/astral-sh/uv):
|
|
107
|
-
|
|
108
|
-
```bash
|
|
109
|
-
uv tool install databricks-schema
|
|
151
|
+
databricks-schema [OPTIONS] COMMAND [ARGS]...
|
|
110
152
|
```
|
|
111
153
|
|
|
112
|
-
|
|
154
|
+
`extract`, `diff`, `generate-sql`, and `diff-files` print progress messages (e.g. `Extracting catalog 'X'...`) to stderr as they run. Pass `--quiet` / `-q` to suppress them;
|
|
155
|
+
results and errors are unaffected.
|
|
113
156
|
|
|
114
|
-
|
|
115
|
-
mkdir -p .claude/skills/databricks-schema-cli
|
|
116
|
-
curl -o .claude/skills/databricks-schema-cli/SKILL.md \
|
|
117
|
-
https://raw.githubusercontent.com/frizzleqq/databricks-schema/main/.claude/skills/databricks-schema-cli/SKILL.md
|
|
118
|
-
```
|
|
119
|
-
|
|
120
|
-
## Development
|
|
157
|
+
### `list-catalogs`
|
|
121
158
|
|
|
122
|
-
|
|
159
|
+
List all accessible catalogs:
|
|
123
160
|
|
|
124
161
|
```bash
|
|
125
|
-
|
|
126
|
-
cd databricks-schema
|
|
127
|
-
uv sync --all-groups # includes pytest and ruff
|
|
162
|
+
databricks-schema list-catalogs
|
|
128
163
|
```
|
|
129
164
|
|
|
130
|
-
|
|
131
|
-
# Run tests
|
|
132
|
-
uv run pytest
|
|
133
|
-
|
|
134
|
-
# Lint
|
|
135
|
-
uv run ruff check databricks_schema/ tests/
|
|
136
|
-
|
|
137
|
-
# Format
|
|
138
|
-
uv run ruff format databricks_schema/ tests/
|
|
139
|
-
```
|
|
165
|
+
### `list-schemas`
|
|
140
166
|
|
|
141
|
-
|
|
167
|
+
List schemas in a catalog:
|
|
142
168
|
|
|
143
|
-
```
|
|
144
|
-
databricks-schema
|
|
169
|
+
```bash
|
|
170
|
+
databricks-schema list-schemas <catalog>
|
|
145
171
|
```
|
|
146
172
|
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
errors are unaffected.
|
|
173
|
+
Both print one name per line by default. Pass `--format json` / `-f json` (or `--format yaml`) to
|
|
174
|
+
get the names as a JSON array or YAML list instead.
|
|
150
175
|
|
|
151
176
|
### `extract`
|
|
152
177
|
|
|
@@ -156,7 +181,11 @@ Extract all schemas from a catalog to YAML files:
|
|
|
156
181
|
databricks-schema extract <catalog> --output-dir ./schemas/
|
|
157
182
|
```
|
|
158
183
|
|
|
159
|
-
|
|
184
|
+
Print to stdout instead of writing files (no `--output-dir`) — the catalog and its schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
|
|
185
|
+
|
|
186
|
+
```bash
|
|
187
|
+
databricks-schema extract <catalog>
|
|
188
|
+
```
|
|
160
189
|
|
|
161
190
|
Extract specific schemas only:
|
|
162
191
|
|
|
@@ -171,12 +200,6 @@ databricks-schema extract <catalog>.<schema>
|
|
|
171
200
|
databricks-schema extract <catalog>.<schema>.<table>
|
|
172
201
|
```
|
|
173
202
|
|
|
174
|
-
Print to stdout instead of writing files (no `--output-dir`) — the catalog and its matching schemas are printed as a single `Catalog` document (`name` + `schemas: [...]`):
|
|
175
|
-
|
|
176
|
-
```bash
|
|
177
|
-
databricks-schema extract <catalog> --schema main
|
|
178
|
-
```
|
|
179
|
-
|
|
180
203
|
Include Unity Catalog tags in the output (extra API calls per entity; excluded by default):
|
|
181
204
|
|
|
182
205
|
```bash
|
|
@@ -189,7 +212,7 @@ Include additional metadata (`owner`, `storage_location`) in the output:
|
|
|
189
212
|
databricks-schema extract <catalog> --output-dir ./schemas/ --include-metadata
|
|
190
213
|
```
|
|
191
214
|
|
|
192
|
-
Control the number of parallel workers (default: 4):
|
|
215
|
+
Control the number of parallel workers (default: 4), useful for large schemas with many tags:
|
|
193
216
|
|
|
194
217
|
```bash
|
|
195
218
|
databricks-schema extract <catalog> --output-dir ./schemas/ --workers 8
|
|
@@ -203,7 +226,21 @@ Compare the live catalog against previously extracted schema files (format auto-
|
|
|
203
226
|
databricks-schema diff <catalog> ./schemas/
|
|
204
227
|
```
|
|
205
228
|
|
|
206
|
-
|
|
229
|
+
Exits with code `0` if no differences are found, `1` if there are — making it suitable for CI pipelines. Output example:
|
|
230
|
+
|
|
231
|
+
```
|
|
232
|
+
~ Schema: main [MODIFIED]
|
|
233
|
+
~ Table: users [MODIFIED]
|
|
234
|
+
~ Column: score [MODIFIED]
|
|
235
|
+
data_type: 'int' -> 'double'
|
|
236
|
+
+ Column: phone [ADDED]
|
|
237
|
+
+ Table: events [ADDED]
|
|
238
|
+
- Schema: legacy [REMOVED]
|
|
239
|
+
```
|
|
240
|
+
|
|
241
|
+
Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
|
|
242
|
+
|
|
243
|
+
Or compare two live catalogs directly, e.g. dev against prod. The second argument is treated as a directory if one exists at that path, otherwise as a catalog name:
|
|
207
244
|
|
|
208
245
|
```bash
|
|
209
246
|
databricks-schema diff dev_catalog prod_catalog
|
|
@@ -227,20 +264,13 @@ Include `owner` in the comparison (excluded by default):
|
|
|
227
264
|
databricks-schema diff <catalog> ./schemas/ --include-metadata
|
|
228
265
|
```
|
|
229
266
|
|
|
230
|
-
|
|
267
|
+
Get the same comparison as structured JSON or YAML instead of the `+`/`-`/`~` tree (exit codes unchanged) — useful for scripts or agents:
|
|
231
268
|
|
|
232
|
-
```
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
~ Column: score [MODIFIED]
|
|
236
|
-
data_type: 'int' -> 'double'
|
|
237
|
-
+ Column: phone [ADDED]
|
|
238
|
-
+ Table: events [ADDED]
|
|
239
|
-
- Schema: legacy [REMOVED]
|
|
269
|
+
```bash
|
|
270
|
+
databricks-schema diff <catalog> ./schemas/ --format json
|
|
271
|
+
databricks-schema diff <catalog> ./schemas/ --format yaml
|
|
240
272
|
```
|
|
241
273
|
|
|
242
|
-
Markers: `+` added in catalog, `-` removed from catalog, `~` modified.
|
|
243
|
-
|
|
244
274
|
### `generate-sql`
|
|
245
275
|
|
|
246
276
|
Generate Databricks Spark SQL statements to bring the live catalog in line with local schema files (format auto-detected, YAML or JSON, not mixed). Statements are printed to stdout by default:
|
|
@@ -287,29 +317,28 @@ Validate local schema files for structural integrity (no Databricks connection n
|
|
|
287
317
|
databricks-schema validate ./schemas/
|
|
288
318
|
```
|
|
289
319
|
|
|
320
|
+
Pass `--format json` or `--format yaml` for an `{"issues": [...]}` document instead of `OK`/`ERROR:` text.
|
|
321
|
+
|
|
290
322
|
### `diff-files`
|
|
291
323
|
|
|
292
324
|
Compare two local directories of schema files (no Databricks connection needed). Same output,
|
|
293
|
-
exit codes, and `--schema` / `--include-metadata` flags as `diff`:
|
|
325
|
+
exit codes, `--format json`/`yaml`, and `--schema` / `--include-metadata` flags as `diff`:
|
|
294
326
|
|
|
295
327
|
```bash
|
|
296
328
|
databricks-schema diff-files ./schemas-prod/ ./schemas-test/
|
|
297
329
|
```
|
|
298
330
|
|
|
299
|
-
### `
|
|
300
|
-
|
|
301
|
-
List all accessible catalogs:
|
|
302
|
-
|
|
303
|
-
```bash
|
|
304
|
-
databricks-schema list-catalogs
|
|
305
|
-
```
|
|
306
|
-
|
|
307
|
-
### `list-schemas`
|
|
331
|
+
### `json-schema`
|
|
308
332
|
|
|
309
|
-
|
|
333
|
+
Print the JSON Schema for one of the CLI's structured output shapes — no Databricks connection
|
|
334
|
+
needed. Useful before parsing `--format json`/`yaml` output (or `extract`'s output)
|
|
335
|
+
programmatically. Add `--format yaml` to render the schema itself as YAML:
|
|
310
336
|
|
|
311
337
|
```bash
|
|
312
|
-
databricks-schema
|
|
338
|
+
databricks-schema json-schema catalog # extract's stdout Catalog document
|
|
339
|
+
databricks-schema json-schema schema # one per-schema extract file
|
|
340
|
+
databricks-schema json-schema diff # diff / diff-files --format json/yaml
|
|
341
|
+
databricks-schema json-schema validate # validate --format json/yaml
|
|
313
342
|
```
|
|
314
343
|
|
|
315
344
|
## Python Library Usage
|