spark-sql-migrations 0.0.5__tar.gz → 0.0.7__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/PKG-INFO +56 -14
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/setup.py +62 -17
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/migration_templates/default_migration_template.sql +3 -0
- spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql +9 -0
- spark_sql_migrations-0.0.5/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table_all.sql → spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql +1 -2
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_sql/spark_sql.py +125 -98
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_utils.py +3 -3
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/PKG-INFO +56 -14
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/SOURCES.txt +3 -3
- spark_sql_migrations-0.0.5/spark_sql_migrations/migrations_initial/first_01_intial_create_cat_dbr_only.sql +0 -4
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/LICENSE +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/MANIFEST.in +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/backend_shim.py +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/setup.cfg +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/__init__.py +0 -0
- /spark_sql_migrations-0.0.5/spark_sql_migrations/migrations_initial/first_02_intial_create_schema_all.sql → /spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/py.typed +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_sql/__init__.py +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/dependency_links.txt +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/entry_points.txt +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/namespace_packages.txt +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/requires.txt +0 -0
- {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spark_sql_migrations
|
|
3
|
-
Version: 0.0.
|
|
3
|
+
Version: 0.0.7
|
|
4
4
|
Summary: Chained, idempotent SQL migrations for Spark and Databricks.
|
|
5
5
|
Home-page: https://github.com/mcjug2015/spark_sql_migrations
|
|
6
6
|
Author: Victor Semenov
|
|
@@ -79,28 +79,32 @@ Migrations come from two different owners, and they behave differently:
|
|
|
79
79
|
| | ships in the wheel | owned by you |
|
|
80
80
|
|---|---|---|
|
|
81
81
|
| **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
|
|
82
|
-
| **where** | `spark_sql_migrations/migrations_initial/` |
|
|
83
|
-
| **
|
|
82
|
+
| **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
|
|
83
|
+
| **ordered by** | filename | the `prev_revision_id` chain |
|
|
84
84
|
| **when applied** | until the version table exists | once, then recorded |
|
|
85
85
|
|
|
86
86
|
The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
|
|
87
87
|
skipped from then on. A run that fails partway through bootstrapping replays it, so it must
|
|
88
|
-
still be idempotent. Your own
|
|
88
|
+
still be idempotent. Your own migrations are applied once each and the head revision is
|
|
89
89
|
written to the version table.
|
|
90
90
|
|
|
91
|
-
|
|
91
|
+
Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
|
|
92
|
+
bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
|
|
92
93
|
|
|
93
|
-
|
|
94
|
+
## Laying out your chain
|
|
95
|
+
|
|
96
|
+
`--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
|
|
94
97
|
|
|
95
98
|
```
|
|
96
99
|
your_project/
|
|
97
|
-
└── migrations/
|
|
98
|
-
├──
|
|
99
|
-
└──
|
|
100
|
+
└── migrations/ <- this is --migrations-dir
|
|
101
|
+
├── 260831_01_create_test_table_e5ce0039b32b.sql
|
|
102
|
+
└── 260831_02_create_metrics_table_07990e2a101e.sql
|
|
100
103
|
```
|
|
101
104
|
|
|
102
|
-
|
|
103
|
-
|
|
105
|
+
One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
|
|
106
|
+
migration that otherwise runs everywhere — see below. SQL that only Databricks understands
|
|
107
|
+
must never sit ungated there, or your local runs will break.
|
|
104
108
|
|
|
105
109
|
## Running migrations
|
|
106
110
|
|
|
@@ -124,7 +128,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
|
|
|
124
128
|
|
|
125
129
|
|
|
126
130
|
def get_migrations_dir():
|
|
127
|
-
"""the
|
|
131
|
+
"""the directory holding this project's own migration chain."""
|
|
128
132
|
return os.path.dirname(__file__)
|
|
129
133
|
|
|
130
134
|
|
|
@@ -134,7 +138,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
|
|
|
134
138
|
`main` builds its own session via `get_spark()` and, being library code, leaves logging to
|
|
135
139
|
you. To supply your own — a session already
|
|
136
140
|
configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
|
|
137
|
-
|
|
141
|
+
migrations_dir)` directly.
|
|
138
142
|
|
|
139
143
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
140
144
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
@@ -144,7 +148,7 @@ before it is executed, so you can always read the exact SQL a run applied.
|
|
|
144
148
|
Generate one rather than hand-rolling the header:
|
|
145
149
|
|
|
146
150
|
```bash
|
|
147
|
-
spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations
|
|
151
|
+
spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations
|
|
148
152
|
```
|
|
149
153
|
|
|
150
154
|
That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
|
|
@@ -183,6 +187,44 @@ begin
|
|
|
183
187
|
end;
|
|
184
188
|
```
|
|
185
189
|
|
|
190
|
+
### Databricks-only statements
|
|
191
|
+
|
|
192
|
+
`is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
|
|
193
|
+
Databricks understands:
|
|
194
|
+
|
|
195
|
+
```sql
|
|
196
|
+
-- revision_id:e8614b76da6d;
|
|
197
|
+
-- prev_revision_id:57037b6c19b4;
|
|
198
|
+
{% if is_dbr %}
|
|
199
|
+
begin
|
|
200
|
+
grant READ FILES on external location `some-bucket` to `some_group`;
|
|
201
|
+
end;
|
|
202
|
+
{% endif %}
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
`create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
|
|
206
|
+
headers so they stay readable:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
spark-sql-migrations create_new_migration --message "read files on the cms zip external location" --output-path path/to/migrations --add-is-dbr
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
|
|
213
|
+
its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
|
|
214
|
+
script never gets the chance to run — and the whole script fails to compile, not just that
|
|
215
|
+
statement. Jinja removes the text before `spark.sql()` ever sees it.
|
|
216
|
+
|
|
217
|
+
A migration left holding nothing but comments is recorded in the version table and not
|
|
218
|
+
executed, so the chain advances in step on both engines. That recording is forward-only: a
|
|
219
|
+
catalog migrated locally past a gated revision will not pick that statement up if it is
|
|
220
|
+
later migrated from Databricks. For disposable test schemas that is exactly what you want;
|
|
221
|
+
for a long-lived catalog, don't migrate it from both.
|
|
222
|
+
|
|
223
|
+
`_spark_migrations_version` holds one row and one column, `version_num`. There is no
|
|
224
|
+
in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
|
|
225
|
+
a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
|
|
226
|
+
scratch, which also clears the revision history that layout recorded.
|
|
227
|
+
|
|
186
228
|
## Local vs Databricks
|
|
187
229
|
|
|
188
230
|
`get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
|
|
@@ -80,28 +80,32 @@ Migrations come from two different owners, and they behave differently:
|
|
|
80
80
|
| | ships in the wheel | owned by you |
|
|
81
81
|
|---|---|---|
|
|
82
82
|
| **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
|
|
83
|
-
| **where** | `spark_sql_migrations/migrations_initial/` |
|
|
84
|
-
| **
|
|
83
|
+
| **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
|
|
84
|
+
| **ordered by** | filename | the `prev_revision_id` chain |
|
|
85
85
|
| **when applied** | until the version table exists | once, then recorded |
|
|
86
86
|
|
|
87
87
|
The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
|
|
88
88
|
skipped from then on. A run that fails partway through bootstrapping replays it, so it must
|
|
89
|
-
still be idempotent. Your own
|
|
89
|
+
still be idempotent. Your own migrations are applied once each and the head revision is
|
|
90
90
|
written to the version table.
|
|
91
91
|
|
|
92
|
-
|
|
92
|
+
Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
|
|
93
|
+
bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
|
|
93
94
|
|
|
94
|
-
|
|
95
|
+
## Laying out your chain
|
|
96
|
+
|
|
97
|
+
`--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
|
|
95
98
|
|
|
96
99
|
```
|
|
97
100
|
your_project/
|
|
98
|
-
└── migrations/
|
|
99
|
-
├──
|
|
100
|
-
└──
|
|
101
|
+
└── migrations/ <- this is --migrations-dir
|
|
102
|
+
├── 260831_01_create_test_table_e5ce0039b32b.sql
|
|
103
|
+
└── 260831_02_create_metrics_table_07990e2a101e.sql
|
|
101
104
|
```
|
|
102
105
|
|
|
103
|
-
|
|
104
|
-
|
|
106
|
+
One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
|
|
107
|
+
migration that otherwise runs everywhere — see below. SQL that only Databricks understands
|
|
108
|
+
must never sit ungated there, or your local runs will break.
|
|
105
109
|
|
|
106
110
|
## Running migrations
|
|
107
111
|
|
|
@@ -128,7 +132,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
|
|
|
128
132
|
|
|
129
133
|
|
|
130
134
|
def get_migrations_dir():
|
|
131
|
-
\"\"\"the
|
|
135
|
+
\"\"\"the directory holding this project's own migration chain.\"\"\"
|
|
132
136
|
return os.path.dirname(__file__)
|
|
133
137
|
|
|
134
138
|
|
|
@@ -138,7 +142,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
|
|
|
138
142
|
`main` builds its own session via `get_spark()` and, being library code, leaves logging to
|
|
139
143
|
you. To supply your own — a session already
|
|
140
144
|
configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
|
|
141
|
-
|
|
145
|
+
migrations_dir)` directly.
|
|
142
146
|
|
|
143
147
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
144
148
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
@@ -150,7 +154,7 @@ Generate one rather than hand-rolling the header:
|
|
|
150
154
|
```bash
|
|
151
155
|
spark-sql-migrations create_new_migration \
|
|
152
156
|
--message "add batch id to metrics" \
|
|
153
|
-
--output-path path/to/migrations
|
|
157
|
+
--output-path path/to/migrations
|
|
154
158
|
```
|
|
155
159
|
|
|
156
160
|
That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
|
|
@@ -189,6 +193,47 @@ begin
|
|
|
189
193
|
end;
|
|
190
194
|
```
|
|
191
195
|
|
|
196
|
+
### Databricks-only statements
|
|
197
|
+
|
|
198
|
+
`is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
|
|
199
|
+
Databricks understands:
|
|
200
|
+
|
|
201
|
+
```sql
|
|
202
|
+
-- revision_id:e8614b76da6d;
|
|
203
|
+
-- prev_revision_id:57037b6c19b4;
|
|
204
|
+
{% if is_dbr %}
|
|
205
|
+
begin
|
|
206
|
+
grant READ FILES on external location `some-bucket` to `some_group`;
|
|
207
|
+
end;
|
|
208
|
+
{% endif %}
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
`create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
|
|
212
|
+
headers so they stay readable:
|
|
213
|
+
|
|
214
|
+
```bash
|
|
215
|
+
spark-sql-migrations create_new_migration \
|
|
216
|
+
--message "read files on the cms zip external location" \
|
|
217
|
+
--output-path path/to/migrations \
|
|
218
|
+
--add-is-dbr
|
|
219
|
+
```
|
|
220
|
+
|
|
221
|
+
This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
|
|
222
|
+
its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
|
|
223
|
+
script never gets the chance to run — and the whole script fails to compile, not just that
|
|
224
|
+
statement. Jinja removes the text before `spark.sql()` ever sees it.
|
|
225
|
+
|
|
226
|
+
A migration left holding nothing but comments is recorded in the version table and not
|
|
227
|
+
executed, so the chain advances in step on both engines. That recording is forward-only: a
|
|
228
|
+
catalog migrated locally past a gated revision will not pick that statement up if it is
|
|
229
|
+
later migrated from Databricks. For disposable test schemas that is exactly what you want;
|
|
230
|
+
for a long-lived catalog, don't migrate it from both.
|
|
231
|
+
|
|
232
|
+
`_spark_migrations_version` holds one row and one column, `version_num`. There is no
|
|
233
|
+
in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
|
|
234
|
+
a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
|
|
235
|
+
scratch, which also clears the revision history that layout recorded.
|
|
236
|
+
|
|
192
237
|
## Local vs Databricks
|
|
193
238
|
|
|
194
239
|
`get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
|
|
@@ -238,9 +283,9 @@ Apache License 2.0 — see [LICENSE](LICENSE).
|
|
|
238
283
|
'package_data': {
|
|
239
284
|
'spark_sql_migrations': (
|
|
240
285
|
'migration_templates/default_migration_template.sql',
|
|
241
|
-
'migrations_initial/
|
|
242
|
-
'migrations_initial/
|
|
243
|
-
'migrations_initial/
|
|
286
|
+
'migrations_initial/first_01_intial_create_cat.sql',
|
|
287
|
+
'migrations_initial/first_02_intial_create_schema.sql',
|
|
288
|
+
'migrations_initial/first_03_intial_create_state_table.sql',
|
|
244
289
|
'py.typed',
|
|
245
290
|
),
|
|
246
291
|
},
|
|
@@ -254,5 +299,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
|
|
|
254
299
|
},
|
|
255
300
|
'python_requires': '>=3.10',
|
|
256
301
|
'url': 'https://github.com/mcjug2015/spark_sql_migrations',
|
|
257
|
-
'version': '0.0.
|
|
302
|
+
'version': '0.0.7',
|
|
258
303
|
})
|
|
@@ -5,6 +5,9 @@ begin
|
|
|
5
5
|
-- would like to have applied to databricks db or local spark(for testing)
|
|
6
6
|
-- crutch scaffolding will use jinja to fill in the cat and schema that are sent to it
|
|
7
7
|
-- this is also taken care of in the gh build.
|
|
8
|
+
-- is_dbr is filled in too: statements only databricks understands go in a jinja
|
|
9
|
+
-- conditional on it, outside the begin/end, rather than in a second chain. pass
|
|
10
|
+
-- --add-is-dbr to create_new_migration to have that conditional written for you.
|
|
8
11
|
|
|
9
12
|
-- e.g:
|
|
10
13
|
-- create schema if not exists {{cat}}.{{schema}};
|
spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql
ADDED
|
@@ -0,0 +1,9 @@
|
|
|
1
|
+
-- revision_id:;
|
|
2
|
+
-- a catalog is a unity catalog object: local spark has spark_catalog and no way
|
|
3
|
+
-- to make another, so this one is gated to databricks and renders to nothing
|
|
4
|
+
-- else, leaving the bootstrap chain the same three files everywhere.
|
|
5
|
+
{% if is_dbr %}
|
|
6
|
+
begin
|
|
7
|
+
create catalog if not exists {{cat}};
|
|
8
|
+
end;
|
|
9
|
+
{% endif %}
|
|
@@ -3,15 +3,21 @@
|
|
|
3
3
|
Two kinds of migration exist, and they are owned by different parties:
|
|
4
4
|
|
|
5
5
|
* the *initial* chain and the new-migration template ship inside this package.
|
|
6
|
-
They bootstrap the catalog, the schema and the version table,
|
|
7
|
-
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
* the
|
|
11
|
-
|
|
12
|
-
``
|
|
13
|
-
|
|
14
|
-
|
|
6
|
+
They bootstrap the catalog, the schema and the version table, carry no
|
|
7
|
+
revision of their own, and are applied in filename order on every run until
|
|
8
|
+
the version table exists -- so a run that failed partway through replays
|
|
9
|
+
them, and they must be idempotent.
|
|
10
|
+
* the client chain belongs to the consuming project. The caller passes the
|
|
11
|
+
directory holding it as ``migrations_dir``; the chain is walked from its root
|
|
12
|
+
via ``prev_revision_id`` and only the tail not yet recorded in the version
|
|
13
|
+
table is applied.
|
|
14
|
+
|
|
15
|
+
Which engine a migration targets is decided inside the migration, not by where it
|
|
16
|
+
sits: every one of them is rendered through jinja before it is executed, with
|
|
17
|
+
``is_dbr`` in the render context, so a chain that runs everywhere can still gate
|
|
18
|
+
the statements one engine would reject. A migration left holding no statements is
|
|
19
|
+
recorded and not executed, which is what lets a project keep one chain and no
|
|
20
|
+
second head.
|
|
15
21
|
"""
|
|
16
22
|
|
|
17
23
|
import argparse
|
|
@@ -21,26 +27,28 @@ import os
|
|
|
21
27
|
import re
|
|
22
28
|
import uuid
|
|
23
29
|
from dataclasses import dataclass
|
|
30
|
+
from typing import Any, Dict, List, Optional
|
|
24
31
|
|
|
25
32
|
from jinja2 import Environment, FileSystemLoader, PackageLoader, select_autoescape
|
|
26
|
-
from pyspark.sql.
|
|
33
|
+
from pyspark.sql.session import SparkSession
|
|
34
|
+
from pyspark.sql.types import Row
|
|
27
35
|
|
|
28
36
|
from spark_sql_migrations.spark_utils import get_spark, is_dbr
|
|
29
37
|
|
|
30
38
|
logger = logging.getLogger(__name__)
|
|
31
39
|
|
|
32
40
|
|
|
33
|
-
ALL_SPARK = "all_spark"
|
|
34
|
-
DBR_ONLY = "dbr_only"
|
|
35
|
-
|
|
36
41
|
VERSION_TABLE = "_spark_migrations_version"
|
|
37
42
|
|
|
38
43
|
PACKAGE_NAME = "spark_sql_migrations"
|
|
39
44
|
INITIAL_MIGRATIONS_PACKAGE_PATH = "migrations_initial"
|
|
45
|
+
# where a run's rendered client migrations land, under its output folder
|
|
46
|
+
CLIENT_OUTPUT_DIRNAME = "migrations"
|
|
40
47
|
TEMPLATES_PACKAGE_PATH = "migration_templates"
|
|
41
48
|
DEFAULT_TEMPLATE_NAME = "default_migration_template.sql"
|
|
42
49
|
|
|
43
50
|
_HEADER_RE = re.compile(r"^--\s*(revision_id|prev_revision_id)\s*:\s*(\w*)\s*;$")
|
|
51
|
+
_COMMENT_OR_BLANK_RE = re.compile(r"^\s*(--.*)?$")
|
|
44
52
|
|
|
45
53
|
|
|
46
54
|
@dataclass(frozen=True)
|
|
@@ -50,18 +58,13 @@ class Migration:
|
|
|
50
58
|
template_name: str
|
|
51
59
|
|
|
52
60
|
|
|
53
|
-
def
|
|
54
|
-
"""the client-owned directory holding one chain."""
|
|
55
|
-
return os.path.join(migrations_root, f"{migration_type}_migrations")
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
def get_default_template_path():
|
|
61
|
+
def get_default_template_path() -> str:
|
|
59
62
|
"""the template this package ships; callers may pass their own instead."""
|
|
60
63
|
return os.path.join(os.path.dirname(__file__), "..", TEMPLATES_PACKAGE_PATH, DEFAULT_TEMPLATE_NAME)
|
|
61
64
|
|
|
62
65
|
|
|
63
|
-
def _parse_migration(migrations_dir, template_name):
|
|
64
|
-
headers = {}
|
|
66
|
+
def _parse_migration(migrations_dir: str, template_name: str) -> Migration:
|
|
67
|
+
headers: Dict[str, Any | str] = {}
|
|
65
68
|
migration_file_path = os.path.join(migrations_dir, template_name)
|
|
66
69
|
if os.path.getsize(migration_file_path) == 0:
|
|
67
70
|
raise ValueError(f"{template_name} is empty;")
|
|
@@ -84,8 +87,8 @@ def _parse_migration(migrations_dir, template_name):
|
|
|
84
87
|
)
|
|
85
88
|
|
|
86
89
|
|
|
87
|
-
def get_ordered_migration_objs(migration_objs):
|
|
88
|
-
by_revision = {}
|
|
90
|
+
def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migration]:
|
|
91
|
+
by_revision: Dict[str, Migration] = {}
|
|
89
92
|
for migration in migration_objs:
|
|
90
93
|
clash = by_revision.get(migration.revision_id)
|
|
91
94
|
if clash:
|
|
@@ -102,7 +105,7 @@ def get_ordered_migration_objs(migration_objs):
|
|
|
102
105
|
f"found {[m.template_name for m in roots]};"
|
|
103
106
|
)
|
|
104
107
|
|
|
105
|
-
next_by_revision = {}
|
|
108
|
+
next_by_revision: Dict[str, Migration] = {}
|
|
106
109
|
for migration in migration_objs:
|
|
107
110
|
if not migration.prev_revision_id:
|
|
108
111
|
continue
|
|
@@ -119,7 +122,7 @@ def get_ordered_migration_objs(migration_objs):
|
|
|
119
122
|
next_by_revision[migration.prev_revision_id] = migration
|
|
120
123
|
|
|
121
124
|
ordered = []
|
|
122
|
-
current = roots[0]
|
|
125
|
+
current: Migration | None = roots[0]
|
|
123
126
|
while current:
|
|
124
127
|
ordered.append(current)
|
|
125
128
|
current = next_by_revision.get(current.revision_id)
|
|
@@ -134,7 +137,7 @@ def get_ordered_migration_objs(migration_objs):
|
|
|
134
137
|
return ordered
|
|
135
138
|
|
|
136
139
|
|
|
137
|
-
def get_migrations_list(migrations_dir):
|
|
140
|
+
def get_migrations_list(migrations_dir: str) -> List[Migration]:
|
|
138
141
|
"""every migration in one directory, walked from its root to its head."""
|
|
139
142
|
if not os.path.isdir(migrations_dir):
|
|
140
143
|
raise ValueError(f"no migrations dir at {migrations_dir};")
|
|
@@ -150,36 +153,56 @@ def get_migrations_list(migrations_dir):
|
|
|
150
153
|
return get_ordered_migration_objs(migrations)
|
|
151
154
|
|
|
152
155
|
|
|
153
|
-
def get_table_version(spark
|
|
156
|
+
def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Row]:
|
|
157
|
+
"""the revision VERSION_TABLE records, or None if it holds no row yet."""
|
|
154
158
|
version_table = f"{cat}.{schema}.{VERSION_TABLE}"
|
|
155
159
|
if not spark.catalog.tableExists(version_table):
|
|
156
160
|
return None
|
|
157
|
-
return spark.read.table(version_table).
|
|
161
|
+
return spark.read.table(version_table).select("version_num").head()
|
|
158
162
|
|
|
159
163
|
|
|
160
|
-
def get_unapplied_migrations_list(spark, migrations_dir
|
|
161
|
-
"""the tail of one chain after the revision VERSION_TABLE records for it.
|
|
162
|
-
|
|
163
|
-
the whole chain when the table is missing or holds no row for this type.
|
|
164
|
-
"""
|
|
164
|
+
def get_unapplied_migrations_list(spark: SparkSession, migrations_dir: str, cat: str, schema: str) -> List[Migration]:
|
|
165
165
|
ordered = get_migrations_list(migrations_dir)
|
|
166
|
-
current = get_table_version(spark,
|
|
166
|
+
current = get_table_version(spark, cat, schema)
|
|
167
167
|
if not current:
|
|
168
168
|
return ordered
|
|
169
169
|
|
|
170
170
|
current_revision_id = current["version_num"]
|
|
171
171
|
|
|
172
|
+
if not ordered:
|
|
173
|
+
logger.warning(
|
|
174
|
+
f"{VERSION_TABLE} is at {current_revision_id}, but {migrations_dir} holds no migrations; "
|
|
175
|
+
f"treating the chain as retired;"
|
|
176
|
+
)
|
|
177
|
+
return ordered
|
|
178
|
+
|
|
172
179
|
for index, migration in enumerate(ordered):
|
|
173
180
|
if migration.revision_id == current_revision_id:
|
|
174
181
|
return ordered[index + 1 :]
|
|
175
182
|
|
|
176
|
-
raise ValueError(
|
|
177
|
-
f"{VERSION_TABLE} has {migration_type} at {current_revision_id}, "
|
|
178
|
-
f"which matches no migration in {migrations_dir};"
|
|
179
|
-
)
|
|
183
|
+
raise ValueError(f"{VERSION_TABLE} is at {current_revision_id}, which matches no migration in {migrations_dir};")
|
|
180
184
|
|
|
181
185
|
|
|
182
|
-
def
|
|
186
|
+
def gate_on_is_dbr(migration_content: str) -> str:
|
|
187
|
+
"""wrap a migration's body in the jinja conditional, below its headers.
|
|
188
|
+
|
|
189
|
+
the revision headers stay outside it: they are read off the file as comments,
|
|
190
|
+
and the first line that is neither comment nor blank ends that read -- so an
|
|
191
|
+
opening tag above them would hide them. a body-less template gets an empty
|
|
192
|
+
gate at the end, which renders to nothing either way.
|
|
193
|
+
"""
|
|
194
|
+
lines = migration_content.splitlines()
|
|
195
|
+
body_start = len(lines)
|
|
196
|
+
for index, line in enumerate(lines):
|
|
197
|
+
if line.strip() and not line.startswith("--"):
|
|
198
|
+
body_start = index
|
|
199
|
+
break
|
|
200
|
+
return "\n".join(lines[:body_start] + ["{% if is_dbr %}"] + lines[body_start:] + ["{% endif %}"]) + "\n"
|
|
201
|
+
|
|
202
|
+
|
|
203
|
+
def create_new_migration(
|
|
204
|
+
message: str, template_path: str, output_path: str, truncate_slug_length: int = 40, add_is_dbr: bool = False
|
|
205
|
+
) -> None:
|
|
183
206
|
rev_id = uuid.uuid4().hex[-12:]
|
|
184
207
|
|
|
185
208
|
date_prefix = datetime.datetime.today().strftime("%y%m%d")
|
|
@@ -201,149 +224,153 @@ def create_new_migration(message: str, template_path: str, output_path: str, tru
|
|
|
201
224
|
default_migration_content = open(template_path).read()
|
|
202
225
|
default_migration_content = default_migration_content.replace("{{revision_id}}", rev_id)
|
|
203
226
|
default_migration_content = default_migration_content.replace("{{prev_revision_id}}", last_migration_rev_id)
|
|
227
|
+
if add_is_dbr:
|
|
228
|
+
default_migration_content = gate_on_is_dbr(default_migration_content)
|
|
204
229
|
|
|
205
230
|
with open(migration_path, "w") as migration_file:
|
|
206
231
|
migration_file.write(default_migration_content)
|
|
207
232
|
|
|
208
233
|
|
|
209
|
-
def apply_template(output_dir, template, cat: str, schema: str):
|
|
210
|
-
result_sql = template.render(cat=cat, schema=schema)
|
|
234
|
+
def apply_template(output_dir: str, template, cat: str, schema: str) -> str:
|
|
235
|
+
result_sql = template.render(cat=cat, schema=schema, is_dbr=is_dbr())
|
|
211
236
|
with open(os.path.join(output_dir, template.name.replace(".sql", "_primed.sql")), "w") as file_handle:
|
|
212
237
|
file_handle.write(result_sql)
|
|
213
238
|
return result_sql
|
|
214
239
|
|
|
215
240
|
|
|
216
|
-
def get_ascending_letters_within_minute():
|
|
241
|
+
def get_ascending_letters_within_minute() -> str:
|
|
217
242
|
micros_since_minute = datetime.datetime.now() - datetime.datetime.now().replace(second=0, microsecond=0)
|
|
218
243
|
result = str(micros_since_minute.microseconds).translate(str.maketrans("0123456789", "ABCDEFGHIJ"))
|
|
219
244
|
return result
|
|
220
245
|
|
|
221
246
|
|
|
222
|
-
def get_output_folder(output_parent_path):
|
|
247
|
+
def get_output_folder(output_parent_path: str) -> str:
|
|
223
248
|
folder_name = f"{datetime.datetime.today().strftime('%Y%m%d_%H%M')}_{get_ascending_letters_within_minute()}"
|
|
224
249
|
return os.path.join(output_parent_path, folder_name)
|
|
225
250
|
|
|
226
251
|
|
|
227
|
-
def
|
|
228
|
-
|
|
229
|
-
|
|
230
|
-
|
|
231
|
-
|
|
232
|
-
return False
|
|
252
|
+
def has_no_statements(rendered_sql: str) -> bool:
|
|
253
|
+
for line in rendered_sql.splitlines():
|
|
254
|
+
if not _COMMENT_OR_BLANK_RE.match(line):
|
|
255
|
+
return False
|
|
256
|
+
return True
|
|
233
257
|
|
|
234
258
|
|
|
235
|
-
def
|
|
236
|
-
"""
|
|
259
|
+
def render_and_apply(spark: SparkSession, output_folder: str, template, cat: str, schema: str) -> None:
|
|
260
|
+
"""render one migration and run it, unless this engine was gated out of it."""
|
|
261
|
+
result_sql = apply_template(output_folder, template, cat=cat, schema=schema)
|
|
262
|
+
if has_no_statements(result_sql):
|
|
263
|
+
logger.info(f"{template.name} rendered no statements; nothing to apply here;")
|
|
264
|
+
return
|
|
265
|
+
spark.sql(result_sql)
|
|
266
|
+
|
|
267
|
+
|
|
268
|
+
def migrate_initial(spark: SparkSession, output_folder: str, cat: str, schema: str) -> None:
|
|
269
|
+
"""apply the package's own bootstrap chain, in filename order rather than by revision."""
|
|
237
270
|
env = Environment(
|
|
238
271
|
loader=PackageLoader(package_name=PACKAGE_NAME, package_path=INITIAL_MIGRATIONS_PACKAGE_PATH),
|
|
239
272
|
autoescape=select_autoescape(),
|
|
240
273
|
)
|
|
241
|
-
all_templates = env.list_templates(
|
|
274
|
+
all_templates = env.list_templates(extensions=["sql"])
|
|
242
275
|
|
|
243
276
|
logger.info(f"found {len(all_templates)} migrations; first five are {all_templates[:5]};")
|
|
244
277
|
for template_name in all_templates:
|
|
245
|
-
|
|
246
|
-
|
|
247
|
-
logger.info(f"invoked spark on {len(all_templates)} migrations;")
|
|
248
|
-
|
|
278
|
+
render_and_apply(spark, output_folder, env.get_template(template_name), cat=cat, schema=schema)
|
|
279
|
+
logger.info(f"rendered {len(all_templates)} initial migrations;")
|
|
249
280
|
|
|
250
|
-
def record_revision(spark, cat: str, schema: str, migration_type: str, revision_id: str):
|
|
251
|
-
"""point VERSION_TABLE's row for one migration type at revision_id.
|
|
252
281
|
|
|
253
|
-
|
|
254
|
-
after a crash between the two statements is harmless because migrations are
|
|
255
|
-
idempotent. the values are inlined rather than passed as `args` because
|
|
256
|
-
delta's DELETE does not bind named parameters.
|
|
257
|
-
"""
|
|
282
|
+
def record_revision(spark: SparkSession, cat: str, schema: str, revision_id: str) -> None:
|
|
258
283
|
version_table = f"{cat}.{schema}.{VERSION_TABLE}"
|
|
259
|
-
type_literal = migration_type.replace("'", "''")
|
|
260
284
|
revision_literal = revision_id.replace("'", "''")
|
|
261
|
-
spark.sql(f"delete from {version_table}
|
|
262
|
-
spark.sql(
|
|
263
|
-
f"insert into {version_table} (migration_type, version_num) " f"values ('{type_literal}', '{revision_literal}')"
|
|
264
|
-
)
|
|
285
|
+
spark.sql(f"delete from {version_table}")
|
|
286
|
+
spark.sql(f"insert into {version_table} (version_num) values ('{revision_literal}')")
|
|
265
287
|
|
|
266
288
|
|
|
267
|
-
def migrate_w_rev(spark, output_folder
|
|
289
|
+
def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str, cat: str, schema: str) -> None:
|
|
268
290
|
"""
|
|
269
|
-
apply the migrations of one client chain that VERSION_TABLE hasn't recorded yet.
|
|
270
291
|
TODO XXX permit applying up to a revision below the head revision
|
|
271
292
|
TODO XXX add backwards migrations, permit migrating backwards
|
|
272
293
|
"""
|
|
273
|
-
|
|
274
|
-
|
|
294
|
+
if not os.path.isdir(migrations_dir):
|
|
295
|
+
logger.info(f"no chain at {migrations_dir}; skipping it;")
|
|
296
|
+
return
|
|
297
|
+
unapplied = get_unapplied_migrations_list(spark, migrations_dir, cat=cat, schema=schema)
|
|
275
298
|
|
|
276
299
|
logger.info(
|
|
277
|
-
f"{
|
|
300
|
+
f"{len(unapplied)} unapplied migrations; "
|
|
278
301
|
f"first five are {[migration.template_name for migration in unapplied[:5]]};"
|
|
279
302
|
)
|
|
280
303
|
|
|
281
304
|
env = Environment(loader=FileSystemLoader(migrations_dir), autoescape=select_autoescape())
|
|
282
305
|
for migration in unapplied:
|
|
283
|
-
|
|
284
|
-
spark.
|
|
285
|
-
|
|
286
|
-
logger.info(f"applied {migration.template_name}; {migration_type} is now at {migration.revision_id};")
|
|
306
|
+
render_and_apply(spark, output_folder, env.get_template(migration.template_name), cat=cat, schema=schema)
|
|
307
|
+
record_revision(spark, cat, schema, migration.revision_id)
|
|
308
|
+
logger.info(f"{migration.template_name} done; now at {migration.revision_id};")
|
|
287
309
|
|
|
288
310
|
|
|
289
|
-
def run_migrations(spark, cat, schema, output_folder,
|
|
311
|
+
def run_migrations(spark: SparkSession, cat: str, schema: str, output_folder: str, migrations_dir: str) -> None:
|
|
290
312
|
initial_output_folder = os.path.join(output_folder, INITIAL_MIGRATIONS_PACKAGE_PATH)
|
|
291
|
-
|
|
292
|
-
all_spark_ouput_folder = os.path.join(output_folder, f"{ALL_SPARK}_migrations")
|
|
313
|
+
client_output_folder = os.path.join(output_folder, CLIENT_OUTPUT_DIRNAME)
|
|
293
314
|
os.makedirs(initial_output_folder, exist_ok=True)
|
|
294
|
-
os.makedirs(
|
|
295
|
-
os.makedirs(all_spark_ouput_folder, exist_ok=True)
|
|
315
|
+
os.makedirs(client_output_folder, exist_ok=True)
|
|
296
316
|
version_table = f"{cat}.{schema}.{VERSION_TABLE}"
|
|
297
317
|
if spark.catalog.tableExists(version_table):
|
|
298
318
|
logger.info(f"{version_table} already exists; skipping the initial migrations;")
|
|
299
319
|
else:
|
|
300
320
|
migrate_initial(spark, initial_output_folder, cat, schema)
|
|
301
|
-
|
|
302
|
-
migrate_w_rev(spark, dbr_ouput_folder, migrations_root, cat, schema, DBR_ONLY)
|
|
303
|
-
migrate_w_rev(spark, all_spark_ouput_folder, migrations_root, cat, schema, ALL_SPARK)
|
|
321
|
+
migrate_w_rev(spark, client_output_folder, migrations_dir, cat, schema)
|
|
304
322
|
|
|
305
323
|
|
|
306
|
-
def main(cat, schema, migrations_dir, output_parent_path=None):
|
|
324
|
+
def main(cat: str, schema: str, migrations_dir: str, output_parent_path: str | None = None) -> None:
|
|
307
325
|
output_folder = get_output_folder(output_parent_path or os.path.join(os.getcwd(), "migrations_out"))
|
|
308
326
|
os.makedirs(output_folder)
|
|
309
327
|
run_migrations(get_spark(), cat, schema, output_folder, migrations_dir)
|
|
310
328
|
|
|
311
329
|
|
|
312
|
-
def _cli_main(cat, schema, migrations_dir): # pragma: no cover
|
|
330
|
+
def _cli_main(cat: str, schema: str, migrations_dir: str) -> None: # pragma: no cover
|
|
313
331
|
main(cat, schema, migrations_dir)
|
|
314
332
|
|
|
315
333
|
|
|
316
|
-
def _cli_create_new_migration(
|
|
317
|
-
|
|
334
|
+
def _cli_create_new_migration(
|
|
335
|
+
message: str, output_path: str, template_path: str, add_is_dbr: bool
|
|
336
|
+
) -> None: # pragma: no cover
|
|
337
|
+
create_new_migration(message, template_path or get_default_template_path(), output_path, add_is_dbr=add_is_dbr)
|
|
318
338
|
|
|
319
339
|
|
|
320
|
-
def build_parser(): # pragma: no cover
|
|
340
|
+
def build_parser() -> argparse.ArgumentParser: # pragma: no cover
|
|
321
341
|
parser = argparse.ArgumentParser(description="spark_sql_migrations migration utilities")
|
|
322
342
|
subparsers = parser.add_subparsers(dest="command", required=True)
|
|
323
343
|
|
|
324
344
|
p_run = subparsers.add_parser("run", help="apply all pending migrations")
|
|
325
|
-
p_run.add_argument("--cat", default="spark_catalog")
|
|
326
|
-
p_run.add_argument("--schema", default="default")
|
|
327
|
-
p_run.add_argument(
|
|
328
|
-
"--migrations-dir", required=True, help="parent of all_spark_migrations/ and dbr_only_migrations/"
|
|
329
|
-
)
|
|
345
|
+
p_run.add_argument("--cat", type=str, default="spark_catalog")
|
|
346
|
+
p_run.add_argument("--schema", type=str, default="default")
|
|
347
|
+
p_run.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the migration chain")
|
|
330
348
|
p_run.set_defaults(func=_cli_main)
|
|
331
349
|
|
|
332
350
|
p_create = subparsers.add_parser("create_new_migration", help="create a new migration from the template")
|
|
333
|
-
p_create.add_argument("--message", required=True)
|
|
351
|
+
p_create.add_argument("--message", type=str, required=True)
|
|
334
352
|
p_create.add_argument(
|
|
335
353
|
"--output-path",
|
|
354
|
+
type=str,
|
|
336
355
|
required=True,
|
|
337
356
|
help="the chain directory to write the new migration into. if sql files are in there, the last sql's rev_id"
|
|
338
357
|
"will be set as the new migrations prev_rev_id.",
|
|
339
358
|
)
|
|
340
|
-
p_create.add_argument(
|
|
359
|
+
p_create.add_argument(
|
|
360
|
+
"--template-path", type=str, default=None, help="defaults to the template shipped in this package"
|
|
361
|
+
)
|
|
362
|
+
# the %% are argparse's: it interpolates help strings
|
|
363
|
+
p_create.add_argument(
|
|
364
|
+
"--add-is-dbr",
|
|
365
|
+
action="store_true",
|
|
366
|
+
help="wrap the new migration's body in {%% if is_dbr %%}, for statements only databricks understands",
|
|
367
|
+
)
|
|
341
368
|
p_create.set_defaults(func=_cli_create_new_migration)
|
|
342
369
|
|
|
343
370
|
return parser
|
|
344
371
|
|
|
345
372
|
|
|
346
|
-
def cli(argv=None): # pragma: no cover
|
|
373
|
+
def cli(argv: list[str] | None = None) -> None: # pragma: no cover
|
|
347
374
|
"""the `spark-sql-migrations` console command.
|
|
348
375
|
|
|
349
376
|
an application entry point, so unlike the rest of this package it configures
|
{spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_utils.py
RENAMED
|
@@ -11,7 +11,7 @@ METASTORE_DIR_ENV_VAR = "SPARK_METASTORE_DIR"
|
|
|
11
11
|
DEFAULT_WAREHOUSE_DIRNAME = "spark-warehouse"
|
|
12
12
|
|
|
13
13
|
|
|
14
|
-
def is_dbr():
|
|
14
|
+
def is_dbr() -> bool:
|
|
15
15
|
try:
|
|
16
16
|
from pyspark.dbutils import DBUtils # type: ignore # noqa: F401
|
|
17
17
|
|
|
@@ -21,7 +21,7 @@ def is_dbr():
|
|
|
21
21
|
return False
|
|
22
22
|
|
|
23
23
|
|
|
24
|
-
def get_warehouse_dir():
|
|
24
|
+
def get_warehouse_dir() -> str:
|
|
25
25
|
"""where local (non-DBR) spark keeps its warehouse.
|
|
26
26
|
|
|
27
27
|
resolved against the caller's cwd, never against __file__: installed from a
|
|
@@ -34,7 +34,7 @@ def get_warehouse_dir():
|
|
|
34
34
|
)
|
|
35
35
|
|
|
36
36
|
|
|
37
|
-
def get_spark(use_dbc=False):
|
|
37
|
+
def get_spark(use_dbc: bool = False) -> SparkSession:
|
|
38
38
|
if use_dbc or is_dbr():
|
|
39
39
|
from databricks.connect.session import DatabricksSession # type: ignore # pants: no-infer-dep
|
|
40
40
|
|
{spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/PKG-INFO
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spark_sql_migrations
|
|
3
|
-
Version: 0.0.
|
|
3
|
+
Version: 0.0.7
|
|
4
4
|
Summary: Chained, idempotent SQL migrations for Spark and Databricks.
|
|
5
5
|
Home-page: https://github.com/mcjug2015/spark_sql_migrations
|
|
6
6
|
Author: Victor Semenov
|
|
@@ -79,28 +79,32 @@ Migrations come from two different owners, and they behave differently:
|
|
|
79
79
|
| | ships in the wheel | owned by you |
|
|
80
80
|
|---|---|---|
|
|
81
81
|
| **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
|
|
82
|
-
| **where** | `spark_sql_migrations/migrations_initial/` |
|
|
83
|
-
| **
|
|
82
|
+
| **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
|
|
83
|
+
| **ordered by** | filename | the `prev_revision_id` chain |
|
|
84
84
|
| **when applied** | until the version table exists | once, then recorded |
|
|
85
85
|
|
|
86
86
|
The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
|
|
87
87
|
skipped from then on. A run that fails partway through bootstrapping replays it, so it must
|
|
88
|
-
still be idempotent. Your own
|
|
88
|
+
still be idempotent. Your own migrations are applied once each and the head revision is
|
|
89
89
|
written to the version table.
|
|
90
90
|
|
|
91
|
-
|
|
91
|
+
Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
|
|
92
|
+
bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
|
|
92
93
|
|
|
93
|
-
|
|
94
|
+
## Laying out your chain
|
|
95
|
+
|
|
96
|
+
`--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
|
|
94
97
|
|
|
95
98
|
```
|
|
96
99
|
your_project/
|
|
97
|
-
└── migrations/
|
|
98
|
-
├──
|
|
99
|
-
└──
|
|
100
|
+
└── migrations/ <- this is --migrations-dir
|
|
101
|
+
├── 260831_01_create_test_table_e5ce0039b32b.sql
|
|
102
|
+
└── 260831_02_create_metrics_table_07990e2a101e.sql
|
|
100
103
|
```
|
|
101
104
|
|
|
102
|
-
|
|
103
|
-
|
|
105
|
+
One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
|
|
106
|
+
migration that otherwise runs everywhere — see below. SQL that only Databricks understands
|
|
107
|
+
must never sit ungated there, or your local runs will break.
|
|
104
108
|
|
|
105
109
|
## Running migrations
|
|
106
110
|
|
|
@@ -124,7 +128,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
|
|
|
124
128
|
|
|
125
129
|
|
|
126
130
|
def get_migrations_dir():
|
|
127
|
-
"""the
|
|
131
|
+
"""the directory holding this project's own migration chain."""
|
|
128
132
|
return os.path.dirname(__file__)
|
|
129
133
|
|
|
130
134
|
|
|
@@ -134,7 +138,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
|
|
|
134
138
|
`main` builds its own session via `get_spark()` and, being library code, leaves logging to
|
|
135
139
|
you. To supply your own — a session already
|
|
136
140
|
configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
|
|
137
|
-
|
|
141
|
+
migrations_dir)` directly.
|
|
138
142
|
|
|
139
143
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
140
144
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
@@ -144,7 +148,7 @@ before it is executed, so you can always read the exact SQL a run applied.
|
|
|
144
148
|
Generate one rather than hand-rolling the header:
|
|
145
149
|
|
|
146
150
|
```bash
|
|
147
|
-
spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations
|
|
151
|
+
spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations
|
|
148
152
|
```
|
|
149
153
|
|
|
150
154
|
That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
|
|
@@ -183,6 +187,44 @@ begin
|
|
|
183
187
|
end;
|
|
184
188
|
```
|
|
185
189
|
|
|
190
|
+
### Databricks-only statements
|
|
191
|
+
|
|
192
|
+
`is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
|
|
193
|
+
Databricks understands:
|
|
194
|
+
|
|
195
|
+
```sql
|
|
196
|
+
-- revision_id:e8614b76da6d;
|
|
197
|
+
-- prev_revision_id:57037b6c19b4;
|
|
198
|
+
{% if is_dbr %}
|
|
199
|
+
begin
|
|
200
|
+
grant READ FILES on external location `some-bucket` to `some_group`;
|
|
201
|
+
end;
|
|
202
|
+
{% endif %}
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
`create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
|
|
206
|
+
headers so they stay readable:
|
|
207
|
+
|
|
208
|
+
```bash
|
|
209
|
+
spark-sql-migrations create_new_migration --message "read files on the cms zip external location" --output-path path/to/migrations --add-is-dbr
|
|
210
|
+
```
|
|
211
|
+
|
|
212
|
+
This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
|
|
213
|
+
its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
|
|
214
|
+
script never gets the chance to run — and the whole script fails to compile, not just that
|
|
215
|
+
statement. Jinja removes the text before `spark.sql()` ever sees it.
|
|
216
|
+
|
|
217
|
+
A migration left holding nothing but comments is recorded in the version table and not
|
|
218
|
+
executed, so the chain advances in step on both engines. That recording is forward-only: a
|
|
219
|
+
catalog migrated locally past a gated revision will not pick that statement up if it is
|
|
220
|
+
later migrated from Databricks. For disposable test schemas that is exactly what you want;
|
|
221
|
+
for a long-lived catalog, don't migrate it from both.
|
|
222
|
+
|
|
223
|
+
`_spark_migrations_version` holds one row and one column, `version_num`. There is no
|
|
224
|
+
in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
|
|
225
|
+
a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
|
|
226
|
+
scratch, which also clears the revision history that layout recorded.
|
|
227
|
+
|
|
186
228
|
## Local vs Databricks
|
|
187
229
|
|
|
188
230
|
`get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
|
{spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/SOURCES.txt
RENAMED
|
@@ -13,8 +13,8 @@ spark_sql_migrations.egg-info/namespace_packages.txt
|
|
|
13
13
|
spark_sql_migrations.egg-info/requires.txt
|
|
14
14
|
spark_sql_migrations.egg-info/top_level.txt
|
|
15
15
|
spark_sql_migrations/migration_templates/default_migration_template.sql
|
|
16
|
-
spark_sql_migrations/migrations_initial/
|
|
17
|
-
spark_sql_migrations/migrations_initial/
|
|
18
|
-
spark_sql_migrations/migrations_initial/
|
|
16
|
+
spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql
|
|
17
|
+
spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql
|
|
18
|
+
spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql
|
|
19
19
|
spark_sql_migrations/spark_sql/__init__.py
|
|
20
20
|
spark_sql_migrations/spark_sql/spark_sql.py
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_sql/__init__.py
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/requires.txt
RENAMED
|
File without changes
|
|
File without changes
|