spark-sql-migrations 0.0.7__tar.gz → 0.0.9__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/PKG-INFO +44 -1
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/setup.py +44 -1
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/spark_sql.py +252 -22
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/PKG-INFO +44 -1
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/LICENSE +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/MANIFEST.in +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/backend_shim.py +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/setup.cfg +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/__init__.py +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migration_templates/default_migration_template.sql +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/py.typed +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/__init__.py +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_utils.py +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/SOURCES.txt +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/dependency_links.txt +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/entry_points.txt +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/namespace_packages.txt +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/requires.txt +0 -0
- {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spark_sql_migrations
|
|
3
|
-
Version: 0.0.
|
|
3
|
+
Version: 0.0.9
|
|
4
4
|
Summary: Chained, idempotent SQL migrations for Spark and Databricks.
|
|
5
5
|
Home-page: https://github.com/mcjug2015/spark_sql_migrations
|
|
6
6
|
Author: Victor Semenov
|
|
@@ -143,6 +143,49 @@ migrations_dir)` directly.
|
|
|
143
143
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
144
144
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
145
145
|
|
|
146
|
+
## Reading the chain
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
spark-sql-migrations history --migrations-dir path/to/migrations
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
|
|
153
|
+
with `<base>` standing in for the revision a root migration follows:
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
|
|
157
|
+
e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
|
|
158
|
+
<base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
| Flag | Effect |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
|
|
164
|
+
| `--oldest-first` | walk from the root up instead of from the head down |
|
|
165
|
+
|
|
166
|
+
`-n 5` with the default order gives the five newest; with `--oldest-first`, the five
|
|
167
|
+
oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
|
|
168
|
+
session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
|
|
169
|
+
nothing about how far any catalog has been migrated.
|
|
170
|
+
|
|
171
|
+
Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
|
|
172
|
+
want it to describe. A fork is tagged where it happens and spelled out below the listing,
|
|
173
|
+
which is kept whole even when `-n` cut the listing short:
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
|
|
177
|
+
c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
|
|
178
|
+
b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
|
|
179
|
+
split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
|
|
180
|
+
c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Two migrations with an empty `prev_revision_id` (two chains in one directory) and
|
|
184
|
+
migrations no root reaches (a cycle) get the same treatment. From Python,
|
|
185
|
+
`history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
|
|
186
|
+
list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
|
|
187
|
+
branch points and orphans as objects.
|
|
188
|
+
|
|
146
189
|
## Writing a migration
|
|
147
190
|
|
|
148
191
|
Generate one rather than hand-rolling the header:
|
|
@@ -147,6 +147,49 @@ migrations_dir)` directly.
|
|
|
147
147
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
148
148
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
149
149
|
|
|
150
|
+
## Reading the chain
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
spark-sql-migrations history --migrations-dir path/to/migrations
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
|
|
157
|
+
with `<base>` standing in for the revision a root migration follows:
|
|
158
|
+
|
|
159
|
+
```
|
|
160
|
+
07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
|
|
161
|
+
e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
|
|
162
|
+
<base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
| Flag | Effect |
|
|
166
|
+
|---|---|
|
|
167
|
+
| `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
|
|
168
|
+
| `--oldest-first` | walk from the root up instead of from the head down |
|
|
169
|
+
|
|
170
|
+
`-n 5` with the default order gives the five newest; with `--oldest-first`, the five
|
|
171
|
+
oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
|
|
172
|
+
session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
|
|
173
|
+
nothing about how far any catalog has been migrated.
|
|
174
|
+
|
|
175
|
+
Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
|
|
176
|
+
want it to describe. A fork is tagged where it happens and spelled out below the listing,
|
|
177
|
+
which is kept whole even when `-n` cut the listing short:
|
|
178
|
+
|
|
179
|
+
```
|
|
180
|
+
c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
|
|
181
|
+
c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
|
|
182
|
+
b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
|
|
183
|
+
split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
|
|
184
|
+
c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Two migrations with an empty `prev_revision_id` (two chains in one directory) and
|
|
188
|
+
migrations no root reaches (a cycle) get the same treatment. From Python,
|
|
189
|
+
`history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
|
|
190
|
+
list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
|
|
191
|
+
branch points and orphans as objects.
|
|
192
|
+
|
|
150
193
|
## Writing a migration
|
|
151
194
|
|
|
152
195
|
Generate one rather than hand-rolling the header:
|
|
@@ -299,5 +342,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
|
|
|
299
342
|
},
|
|
300
343
|
'python_requires': '>=3.10',
|
|
301
344
|
'url': 'https://github.com/mcjug2015/spark_sql_migrations',
|
|
302
|
-
'version': '0.0.
|
|
345
|
+
'version': '0.0.9',
|
|
303
346
|
})
|
|
@@ -27,7 +27,7 @@ import os
|
|
|
27
27
|
import re
|
|
28
28
|
import uuid
|
|
29
29
|
from dataclasses import dataclass
|
|
30
|
-
from typing import Any, Dict, List, Optional
|
|
30
|
+
from typing import Any, Dict, List, Optional, Set
|
|
31
31
|
|
|
32
32
|
from jinja2 import Environment, FileSystemLoader, PackageLoader, select_autoescape
|
|
33
33
|
from pyspark.sql.session import SparkSession
|
|
@@ -39,6 +39,8 @@ logger = logging.getLogger(__name__)
|
|
|
39
39
|
|
|
40
40
|
|
|
41
41
|
VERSION_TABLE = "_spark_migrations_version"
|
|
42
|
+
# what `history` prints in place of the revision a root migration follows
|
|
43
|
+
BASE_LABEL = "<base>"
|
|
42
44
|
|
|
43
45
|
PACKAGE_NAME = "spark_sql_migrations"
|
|
44
46
|
INITIAL_MIGRATIONS_PACKAGE_PATH = "migrations_initial"
|
|
@@ -54,10 +56,29 @@ _COMMENT_OR_BLANK_RE = re.compile(r"^\s*(--.*)?$")
|
|
|
54
56
|
@dataclass(frozen=True)
|
|
55
57
|
class Migration:
|
|
56
58
|
revision_id: str
|
|
57
|
-
prev_revision_id: str
|
|
59
|
+
prev_revision_id: Optional[str]
|
|
58
60
|
template_name: str
|
|
59
61
|
|
|
60
62
|
|
|
63
|
+
@dataclass(frozen=True)
|
|
64
|
+
class ChainSummary:
|
|
65
|
+
"""a directory's chain as `history` found it, forks and all.
|
|
66
|
+
|
|
67
|
+
``ordered`` is oldest first and holds every migration in the directory, so a
|
|
68
|
+
listing built from it never silently drops a file. Everything else is what
|
|
69
|
+
makes the chain unapplicable: more than one ``heads`` entry is a split head,
|
|
70
|
+
a ``branch_points`` entry is where it split, more than one ``roots`` entry is
|
|
71
|
+
a second chain in the same directory, and ``orphans`` are files no walk from
|
|
72
|
+
any root reaches -- a cycle, in practice.
|
|
73
|
+
"""
|
|
74
|
+
|
|
75
|
+
ordered: List[Migration]
|
|
76
|
+
heads: List[Migration]
|
|
77
|
+
roots: List[Migration]
|
|
78
|
+
branch_points: Dict[str, List[Migration]]
|
|
79
|
+
orphans: List[Migration]
|
|
80
|
+
|
|
81
|
+
|
|
61
82
|
def get_default_template_path() -> str:
|
|
62
83
|
"""the template this package ships; callers may pass their own instead."""
|
|
63
84
|
return os.path.join(os.path.dirname(__file__), "..", TEMPLATES_PACKAGE_PATH, DEFAULT_TEMPLATE_NAME)
|
|
@@ -122,7 +143,7 @@ def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migratio
|
|
|
122
143
|
next_by_revision[migration.prev_revision_id] = migration
|
|
123
144
|
|
|
124
145
|
ordered = []
|
|
125
|
-
current: Migration
|
|
146
|
+
current: Optional[Migration] = roots[0]
|
|
126
147
|
while current:
|
|
127
148
|
ordered.append(current)
|
|
128
149
|
current = next_by_revision.get(current.revision_id)
|
|
@@ -137,20 +158,130 @@ def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migratio
|
|
|
137
158
|
return ordered
|
|
138
159
|
|
|
139
160
|
|
|
140
|
-
def
|
|
141
|
-
"""every
|
|
161
|
+
def read_migrations_dir(migrations_dir: str) -> List[Migration]:
|
|
162
|
+
"""every .sql file in one directory, parsed, in filename order and unwalked.
|
|
163
|
+
|
|
164
|
+
the headers are all this reads, so a chain that forks or points at nothing
|
|
165
|
+
comes back whole -- it is `get_ordered_migration_objs` that insists on a
|
|
166
|
+
single walkable chain, and `history` that reports one which is not.
|
|
167
|
+
"""
|
|
142
168
|
if not os.path.isdir(migrations_dir):
|
|
143
169
|
raise ValueError(f"no migrations dir at {migrations_dir};")
|
|
144
170
|
|
|
145
|
-
|
|
171
|
+
return [
|
|
146
172
|
_parse_migration(migrations_dir, template_name)
|
|
147
173
|
for template_name in sorted(os.listdir(migrations_dir))
|
|
148
174
|
if template_name.endswith(".sql")
|
|
149
175
|
]
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
def get_migrations_list(migrations_dir: str, max_revision_id: Optional[str] = None) -> List[Migration]:
|
|
179
|
+
"""every migration in one directory, walked from its root to its head.
|
|
180
|
+
|
|
181
|
+
``revision_id`` cuts the walk short: the chain is returned up to and including
|
|
182
|
+
the migration carrying that revision. A revision no migration in the directory
|
|
183
|
+
carries is not an error -- the whole chain comes back, same as for no revision
|
|
184
|
+
at all.
|
|
185
|
+
"""
|
|
186
|
+
migrations = read_migrations_dir(migrations_dir)
|
|
150
187
|
if not migrations:
|
|
151
188
|
return []
|
|
152
189
|
|
|
153
|
-
|
|
190
|
+
ordered = get_ordered_migration_objs(migrations)
|
|
191
|
+
if max_revision_id is None:
|
|
192
|
+
return ordered
|
|
193
|
+
|
|
194
|
+
for index, migration in enumerate(ordered):
|
|
195
|
+
if migration.revision_id == max_revision_id:
|
|
196
|
+
return ordered[: index + 1]
|
|
197
|
+
|
|
198
|
+
return ordered
|
|
199
|
+
|
|
200
|
+
|
|
201
|
+
def get_children_by_revision(migrations: List[Migration]) -> Dict[Optional[str], List[Migration]]:
|
|
202
|
+
"""each revision's followers, keyed by the revision they follow.
|
|
203
|
+
|
|
204
|
+
the root's followers sit under None. A revision with two followers is a fork,
|
|
205
|
+
which is why this returns a list and not one migration.
|
|
206
|
+
"""
|
|
207
|
+
children: Dict[Optional[str], List[Migration]] = {}
|
|
208
|
+
for migration in sorted(migrations, key=lambda migration: migration.template_name):
|
|
209
|
+
children.setdefault(migration.prev_revision_id, []).append(migration)
|
|
210
|
+
return children
|
|
211
|
+
|
|
212
|
+
|
|
213
|
+
def summarize_chain(migrations: List[Migration]) -> ChainSummary:
|
|
214
|
+
"""walk a chain for reading rather than for applying.
|
|
215
|
+
|
|
216
|
+
this raises nothing: a fork, a second root or a revision following one no file
|
|
217
|
+
carries is what the caller asked to be shown, not a reason to refuse. The walk
|
|
218
|
+
is depth first from each root, so one branch of a fork is listed to its head
|
|
219
|
+
before the next branch starts, and a revision already listed is not listed
|
|
220
|
+
again -- which is what keeps a cycle from looping here.
|
|
221
|
+
"""
|
|
222
|
+
children = get_children_by_revision(migrations)
|
|
223
|
+
known_revisions = {migration.revision_id for migration in migrations}
|
|
224
|
+
roots = [
|
|
225
|
+
migration
|
|
226
|
+
for migration in sorted(migrations, key=lambda migration: migration.template_name)
|
|
227
|
+
if not migration.prev_revision_id or migration.prev_revision_id not in known_revisions
|
|
228
|
+
]
|
|
229
|
+
|
|
230
|
+
ordered: List[Migration] = []
|
|
231
|
+
seen: Set[Migration] = set()
|
|
232
|
+
stack = list(reversed(roots))
|
|
233
|
+
while stack:
|
|
234
|
+
migration = stack.pop()
|
|
235
|
+
if migration in seen:
|
|
236
|
+
continue
|
|
237
|
+
seen.add(migration)
|
|
238
|
+
ordered.append(migration)
|
|
239
|
+
stack.extend(reversed(children.get(migration.revision_id, [])))
|
|
240
|
+
|
|
241
|
+
# orphan - somehow comes from no root
|
|
242
|
+
orphans = [migration for migration in migrations if migration not in seen]
|
|
243
|
+
ordered.extend(sorted(orphans, key=lambda migration: migration.template_name))
|
|
244
|
+
|
|
245
|
+
return ChainSummary(
|
|
246
|
+
ordered=ordered,
|
|
247
|
+
heads=[migration for migration in ordered if not children.get(migration.revision_id)],
|
|
248
|
+
roots=roots,
|
|
249
|
+
branch_points={
|
|
250
|
+
revision_id: followers for revision_id, followers in children.items() if revision_id and len(followers) > 1
|
|
251
|
+
},
|
|
252
|
+
orphans=orphans,
|
|
253
|
+
)
|
|
254
|
+
|
|
255
|
+
|
|
256
|
+
def format_history(summary: ChainSummary, limit: Optional[int] = None, oldest_first: bool = False) -> List[str]:
|
|
257
|
+
"""the listing, newest first like alembic's unless asked for the other end.
|
|
258
|
+
|
|
259
|
+
``limit`` cuts from the far end, so it keeps the newest with the default order
|
|
260
|
+
and the oldest with ``oldest_first`` -- either way, the end the listing starts
|
|
261
|
+
at is the end that survives. The notes are appended whole, even to a listing
|
|
262
|
+
cut short, because a split head is the thing worth seeing most.
|
|
263
|
+
"""
|
|
264
|
+
if limit is not None and limit < 1:
|
|
265
|
+
raise ValueError(f"a history limit lists at least one migration, got {limit};")
|
|
266
|
+
|
|
267
|
+
ordered = summary.ordered if oldest_first else list(reversed(summary.ordered))
|
|
268
|
+
if limit is not None:
|
|
269
|
+
ordered = ordered[:limit]
|
|
270
|
+
|
|
271
|
+
head_revisions = {migration.revision_id for migration in summary.heads}
|
|
272
|
+
branch_point_revisions = set(summary.branch_points)
|
|
273
|
+
lines = [format_history_line(migration, head_revisions, branch_point_revisions) for migration in ordered]
|
|
274
|
+
return lines + format_chain_notes(summary)
|
|
275
|
+
|
|
276
|
+
|
|
277
|
+
def history(migrations_dir: str, limit: Optional[int] = None, oldest_first: bool = False) -> List[str]:
|
|
278
|
+
"""the chain in one directory, as lines ready to be printed.
|
|
279
|
+
|
|
280
|
+
the files on disk are the whole of it: no catalog is read and no session is
|
|
281
|
+
built, so a chain can be read anywhere, whatever any catalog has been migrated
|
|
282
|
+
to. `get_table_version` is what says where a catalog actually stands.
|
|
283
|
+
"""
|
|
284
|
+
return format_history(summarize_chain(read_migrations_dir(migrations_dir)), limit=limit, oldest_first=oldest_first)
|
|
154
285
|
|
|
155
286
|
|
|
156
287
|
def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Row]:
|
|
@@ -161,8 +292,10 @@ def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Ro
|
|
|
161
292
|
return spark.read.table(version_table).select("version_num").head()
|
|
162
293
|
|
|
163
294
|
|
|
164
|
-
def get_unapplied_migrations_list(
|
|
165
|
-
|
|
295
|
+
def get_unapplied_migrations_list(
|
|
296
|
+
spark: SparkSession, migrations_dir: str, cat: str, schema: str, max_revision_id: Optional[str] = None
|
|
297
|
+
) -> List[Migration]:
|
|
298
|
+
ordered = get_migrations_list(migrations_dir, max_revision_id)
|
|
166
299
|
current = get_table_version(spark, cat, schema)
|
|
167
300
|
if not current:
|
|
168
301
|
return ordered
|
|
@@ -239,9 +372,15 @@ def apply_template(output_dir: str, template, cat: str, schema: str) -> str:
|
|
|
239
372
|
|
|
240
373
|
|
|
241
374
|
def get_ascending_letters_within_minute() -> str:
|
|
242
|
-
|
|
243
|
-
|
|
244
|
-
|
|
375
|
+
"""a letter stamp that sorts in the order it was taken, within one minute.
|
|
376
|
+
|
|
377
|
+
the full microseconds since the minute began, not timedelta.microseconds -- that
|
|
378
|
+
holds only the sub-second part and so starts over every second. zero-padded to a
|
|
379
|
+
fixed width, because these are compared as text once they are folder names.
|
|
380
|
+
"""
|
|
381
|
+
now = datetime.datetime.now()
|
|
382
|
+
micros_since_minute = (now - now.replace(second=0, microsecond=0)) // datetime.timedelta(microseconds=1)
|
|
383
|
+
return f"{micros_since_minute:08d}".translate(str.maketrans("0123456789", "ABCDEFGHIJ"))
|
|
245
384
|
|
|
246
385
|
|
|
247
386
|
def get_output_folder(output_parent_path: str) -> str:
|
|
@@ -286,7 +425,14 @@ def record_revision(spark: SparkSession, cat: str, schema: str, revision_id: str
|
|
|
286
425
|
spark.sql(f"insert into {version_table} (version_num) values ('{revision_literal}')")
|
|
287
426
|
|
|
288
427
|
|
|
289
|
-
def
|
|
428
|
+
def upgrade(
|
|
429
|
+
spark: SparkSession,
|
|
430
|
+
output_folder: str,
|
|
431
|
+
migrations_dir: str,
|
|
432
|
+
cat: str,
|
|
433
|
+
schema: str,
|
|
434
|
+
max_revision_id: Optional[str] = None,
|
|
435
|
+
) -> None:
|
|
290
436
|
"""
|
|
291
437
|
TODO XXX permit applying up to a revision below the head revision
|
|
292
438
|
TODO XXX add backwards migrations, permit migrating backwards
|
|
@@ -294,7 +440,9 @@ def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str,
|
|
|
294
440
|
if not os.path.isdir(migrations_dir):
|
|
295
441
|
logger.info(f"no chain at {migrations_dir}; skipping it;")
|
|
296
442
|
return
|
|
297
|
-
unapplied = get_unapplied_migrations_list(
|
|
443
|
+
unapplied = get_unapplied_migrations_list(
|
|
444
|
+
spark, migrations_dir, cat=cat, schema=schema, max_revision_id=max_revision_id
|
|
445
|
+
)
|
|
298
446
|
|
|
299
447
|
logger.info(
|
|
300
448
|
f"{len(unapplied)} unapplied migrations; "
|
|
@@ -308,7 +456,14 @@ def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str,
|
|
|
308
456
|
logger.info(f"{migration.template_name} done; now at {migration.revision_id};")
|
|
309
457
|
|
|
310
458
|
|
|
311
|
-
def
|
|
459
|
+
def run_upgrade(
|
|
460
|
+
spark: SparkSession,
|
|
461
|
+
cat: str,
|
|
462
|
+
schema: str,
|
|
463
|
+
output_folder: str,
|
|
464
|
+
migrations_dir: str,
|
|
465
|
+
max_revision_id: Optional[str] = None,
|
|
466
|
+
) -> None:
|
|
312
467
|
initial_output_folder = os.path.join(output_folder, INITIAL_MIGRATIONS_PACKAGE_PATH)
|
|
313
468
|
client_output_folder = os.path.join(output_folder, CLIENT_OUTPUT_DIRNAME)
|
|
314
469
|
os.makedirs(initial_output_folder, exist_ok=True)
|
|
@@ -318,17 +473,64 @@ def run_migrations(spark: SparkSession, cat: str, schema: str, output_folder: st
|
|
|
318
473
|
logger.info(f"{version_table} already exists; skipping the initial migrations;")
|
|
319
474
|
else:
|
|
320
475
|
migrate_initial(spark, initial_output_folder, cat, schema)
|
|
321
|
-
|
|
476
|
+
upgrade(spark, client_output_folder, migrations_dir, cat, schema, max_revision_id)
|
|
477
|
+
|
|
478
|
+
|
|
479
|
+
def format_history_line(
|
|
480
|
+
migration: Migration, head_revisions: Set[str], branch_point_revisions: Set[str]
|
|
481
|
+
) -> str: # pragma: no cover
|
|
482
|
+
"""one alembic-shaped line: `prev -> rev (tags), filename`."""
|
|
483
|
+
tags = []
|
|
484
|
+
if migration.revision_id in head_revisions:
|
|
485
|
+
tags.append("head")
|
|
486
|
+
if migration.revision_id in branch_point_revisions:
|
|
487
|
+
tags.append("branchpoint")
|
|
488
|
+
tag_text = f" ({', '.join(tags)})" if tags else ""
|
|
489
|
+
return f"{migration.prev_revision_id or BASE_LABEL} -> {migration.revision_id}{tag_text}, {migration.template_name}"
|
|
490
|
+
|
|
491
|
+
|
|
492
|
+
def format_chain_notes(summary: ChainSummary) -> List[str]: # pragma: no cover
|
|
493
|
+
"""what is wrong with this chain, below the listing, or nothing if it is sound."""
|
|
494
|
+
notes = []
|
|
495
|
+
if len(summary.heads) > 1:
|
|
496
|
+
notes.append(
|
|
497
|
+
f"split heads: {', '.join(migration.revision_id for migration in summary.heads)}; "
|
|
498
|
+
f"a chain with more than one head cannot be applied until it is rejoined;"
|
|
499
|
+
)
|
|
500
|
+
for revision_id, followers in sorted(summary.branch_points.items()):
|
|
501
|
+
notes.append(
|
|
502
|
+
f"{revision_id} is the prev_revision_id of "
|
|
503
|
+
f"{', '.join(migration.template_name for migration in followers)};"
|
|
504
|
+
)
|
|
505
|
+
if len(summary.roots) > 1:
|
|
506
|
+
notes.append(
|
|
507
|
+
f"more than one root: {', '.join(migration.template_name for migration in summary.roots)}; "
|
|
508
|
+
f"a chain has exactly one migration following no other;"
|
|
509
|
+
)
|
|
510
|
+
if summary.orphans:
|
|
511
|
+
notes.append(
|
|
512
|
+
f"no root reaches {', '.join(migration.template_name for migration in summary.orphans)}; "
|
|
513
|
+
f"they are listed together at the tail end of the chain order, having none of their own;"
|
|
514
|
+
)
|
|
515
|
+
return notes
|
|
322
516
|
|
|
323
517
|
|
|
324
|
-
def main(
|
|
518
|
+
def main(
|
|
519
|
+
cat: str,
|
|
520
|
+
schema: str,
|
|
521
|
+
migrations_dir: str,
|
|
522
|
+
output_parent_path: Optional[str] = None,
|
|
523
|
+
max_revision_id: Optional[str] = None,
|
|
524
|
+
) -> None:
|
|
325
525
|
output_folder = get_output_folder(output_parent_path or os.path.join(os.getcwd(), "migrations_out"))
|
|
326
526
|
os.makedirs(output_folder)
|
|
327
|
-
|
|
527
|
+
run_upgrade(get_spark(), cat, schema, output_folder, migrations_dir, max_revision_id)
|
|
328
528
|
|
|
329
529
|
|
|
330
|
-
def _cli_main(
|
|
331
|
-
|
|
530
|
+
def _cli_main(
|
|
531
|
+
cat: str, schema: str, migrations_dir: str, max_revision_id: Optional[str] = None
|
|
532
|
+
) -> None: # pragma: no cover
|
|
533
|
+
main(cat, schema, migrations_dir, max_revision_id=max_revision_id)
|
|
332
534
|
|
|
333
535
|
|
|
334
536
|
def _cli_create_new_migration(
|
|
@@ -337,14 +539,26 @@ def _cli_create_new_migration(
|
|
|
337
539
|
create_new_migration(message, template_path or get_default_template_path(), output_path, add_is_dbr=add_is_dbr)
|
|
338
540
|
|
|
339
541
|
|
|
542
|
+
def _cli_history(migrations_dir: str, limit: Optional[int], oldest_first: bool) -> None: # pragma: no cover
|
|
543
|
+
"""the listing goes to stdout, so it pipes; the log records go to stderr as always."""
|
|
544
|
+
for line in history(migrations_dir, limit=limit, oldest_first=oldest_first):
|
|
545
|
+
print(line)
|
|
546
|
+
|
|
547
|
+
|
|
340
548
|
def build_parser() -> argparse.ArgumentParser: # pragma: no cover
|
|
341
549
|
parser = argparse.ArgumentParser(description="spark_sql_migrations migration utilities")
|
|
342
550
|
subparsers = parser.add_subparsers(dest="command", required=True)
|
|
343
551
|
|
|
344
|
-
p_run = subparsers.add_parser("run", help="apply
|
|
552
|
+
p_run = subparsers.add_parser("run", help="upgrade to head/apply pending migrations")
|
|
345
553
|
p_run.add_argument("--cat", type=str, default="spark_catalog")
|
|
346
554
|
p_run.add_argument("--schema", type=str, default="default")
|
|
347
555
|
p_run.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the migration chain")
|
|
556
|
+
p_run.add_argument(
|
|
557
|
+
"--max-revision-id",
|
|
558
|
+
type=str,
|
|
559
|
+
default=None,
|
|
560
|
+
help="apply the migration carrying this revision_id and stop; defaults to applying the chain to its head",
|
|
561
|
+
)
|
|
348
562
|
p_run.set_defaults(func=_cli_main)
|
|
349
563
|
|
|
350
564
|
p_create = subparsers.add_parser("create_new_migration", help="create a new migration from the template")
|
|
@@ -367,10 +581,26 @@ def build_parser() -> argparse.ArgumentParser: # pragma: no cover
|
|
|
367
581
|
)
|
|
368
582
|
p_create.set_defaults(func=_cli_create_new_migration)
|
|
369
583
|
|
|
584
|
+
p_history = subparsers.add_parser("history", help="list the migration chain on disk, newest first")
|
|
585
|
+
p_history.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the chain")
|
|
586
|
+
p_history.add_argument(
|
|
587
|
+
"-n",
|
|
588
|
+
"--limit",
|
|
589
|
+
type=int,
|
|
590
|
+
default=None,
|
|
591
|
+
help="list at most this many migrations, counted from the end the listing starts at",
|
|
592
|
+
)
|
|
593
|
+
p_history.add_argument(
|
|
594
|
+
"--oldest-first",
|
|
595
|
+
action="store_true",
|
|
596
|
+
help="list from the root up instead of from the head down",
|
|
597
|
+
)
|
|
598
|
+
p_history.set_defaults(func=_cli_history)
|
|
599
|
+
|
|
370
600
|
return parser
|
|
371
601
|
|
|
372
602
|
|
|
373
|
-
def cli(argv: list[str]
|
|
603
|
+
def cli(argv: Optional[list[str]] = None) -> None: # pragma: no cover
|
|
374
604
|
"""the `spark-sql-migrations` console command.
|
|
375
605
|
|
|
376
606
|
an application entry point, so unlike the rest of this package it configures
|
{spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/PKG-INFO
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spark_sql_migrations
|
|
3
|
-
Version: 0.0.
|
|
3
|
+
Version: 0.0.9
|
|
4
4
|
Summary: Chained, idempotent SQL migrations for Spark and Databricks.
|
|
5
5
|
Home-page: https://github.com/mcjug2015/spark_sql_migrations
|
|
6
6
|
Author: Victor Semenov
|
|
@@ -143,6 +143,49 @@ migrations_dir)` directly.
|
|
|
143
143
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
144
144
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
145
145
|
|
|
146
|
+
## Reading the chain
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
spark-sql-migrations history --migrations-dir path/to/migrations
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
|
|
153
|
+
with `<base>` standing in for the revision a root migration follows:
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
|
|
157
|
+
e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
|
|
158
|
+
<base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
| Flag | Effect |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
|
|
164
|
+
| `--oldest-first` | walk from the root up instead of from the head down |
|
|
165
|
+
|
|
166
|
+
`-n 5` with the default order gives the five newest; with `--oldest-first`, the five
|
|
167
|
+
oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
|
|
168
|
+
session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
|
|
169
|
+
nothing about how far any catalog has been migrated.
|
|
170
|
+
|
|
171
|
+
Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
|
|
172
|
+
want it to describe. A fork is tagged where it happens and spelled out below the listing,
|
|
173
|
+
which is kept whole even when `-n` cut the listing short:
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
|
|
177
|
+
c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
|
|
178
|
+
b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
|
|
179
|
+
split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
|
|
180
|
+
c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Two migrations with an empty `prev_revision_id` (two chains in one directory) and
|
|
184
|
+
migrations no root reaches (a cycle) get the same treatment. From Python,
|
|
185
|
+
`history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
|
|
186
|
+
list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
|
|
187
|
+
branch points and orphans as objects.
|
|
188
|
+
|
|
146
189
|
## Writing a migration
|
|
147
190
|
|
|
148
191
|
Generate one rather than hand-rolling the header:
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/__init__.py
RENAMED
|
File without changes
|
{spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_utils.py
RENAMED
|
File without changes
|
{spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/SOURCES.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/requires.txt
RENAMED
|
File without changes
|
|
File without changes
|