spark-sql-migrations 0.0.8__tar.gz → 0.0.9__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/PKG-INFO +44 -1
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/setup.py +44 -1
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/spark_sql.py +186 -8
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/PKG-INFO +44 -1
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/LICENSE +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/MANIFEST.in +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/backend_shim.py +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/setup.cfg +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/__init__.py +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migration_templates/default_migration_template.sql +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/py.typed +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/__init__.py +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_utils.py +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/SOURCES.txt +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/dependency_links.txt +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/entry_points.txt +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/namespace_packages.txt +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/requires.txt +0 -0
- {spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/top_level.txt +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spark_sql_migrations
|
|
3
|
-
Version: 0.0.
|
|
3
|
+
Version: 0.0.9
|
|
4
4
|
Summary: Chained, idempotent SQL migrations for Spark and Databricks.
|
|
5
5
|
Home-page: https://github.com/mcjug2015/spark_sql_migrations
|
|
6
6
|
Author: Victor Semenov
|
|
@@ -143,6 +143,49 @@ migrations_dir)` directly.
|
|
|
143
143
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
144
144
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
145
145
|
|
|
146
|
+
## Reading the chain
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
spark-sql-migrations history --migrations-dir path/to/migrations
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
|
|
153
|
+
with `<base>` standing in for the revision a root migration follows:
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
|
|
157
|
+
e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
|
|
158
|
+
<base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
| Flag | Effect |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
|
|
164
|
+
| `--oldest-first` | walk from the root up instead of from the head down |
|
|
165
|
+
|
|
166
|
+
`-n 5` with the default order gives the five newest; with `--oldest-first`, the five
|
|
167
|
+
oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
|
|
168
|
+
session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
|
|
169
|
+
nothing about how far any catalog has been migrated.
|
|
170
|
+
|
|
171
|
+
Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
|
|
172
|
+
want it to describe. A fork is tagged where it happens and spelled out below the listing,
|
|
173
|
+
which is kept whole even when `-n` cut the listing short:
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
|
|
177
|
+
c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
|
|
178
|
+
b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
|
|
179
|
+
split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
|
|
180
|
+
c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Two migrations with an empty `prev_revision_id` (two chains in one directory) and
|
|
184
|
+
migrations no root reaches (a cycle) get the same treatment. From Python,
|
|
185
|
+
`history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
|
|
186
|
+
list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
|
|
187
|
+
branch points and orphans as objects.
|
|
188
|
+
|
|
146
189
|
## Writing a migration
|
|
147
190
|
|
|
148
191
|
Generate one rather than hand-rolling the header:
|
|
@@ -147,6 +147,49 @@ migrations_dir)` directly.
|
|
|
147
147
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
148
148
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
149
149
|
|
|
150
|
+
## Reading the chain
|
|
151
|
+
|
|
152
|
+
```bash
|
|
153
|
+
spark-sql-migrations history --migrations-dir path/to/migrations
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
|
|
157
|
+
with `<base>` standing in for the revision a root migration follows:
|
|
158
|
+
|
|
159
|
+
```
|
|
160
|
+
07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
|
|
161
|
+
e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
|
|
162
|
+
<base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
| Flag | Effect |
|
|
166
|
+
|---|---|
|
|
167
|
+
| `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
|
|
168
|
+
| `--oldest-first` | walk from the root up instead of from the head down |
|
|
169
|
+
|
|
170
|
+
`-n 5` with the default order gives the five newest; with `--oldest-first`, the five
|
|
171
|
+
oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
|
|
172
|
+
session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
|
|
173
|
+
nothing about how far any catalog has been migrated.
|
|
174
|
+
|
|
175
|
+
Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
|
|
176
|
+
want it to describe. A fork is tagged where it happens and spelled out below the listing,
|
|
177
|
+
which is kept whole even when `-n` cut the listing short:
|
|
178
|
+
|
|
179
|
+
```
|
|
180
|
+
c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
|
|
181
|
+
c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
|
|
182
|
+
b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
|
|
183
|
+
split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
|
|
184
|
+
c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
|
|
185
|
+
```
|
|
186
|
+
|
|
187
|
+
Two migrations with an empty `prev_revision_id` (two chains in one directory) and
|
|
188
|
+
migrations no root reaches (a cycle) get the same treatment. From Python,
|
|
189
|
+
`history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
|
|
190
|
+
list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
|
|
191
|
+
branch points and orphans as objects.
|
|
192
|
+
|
|
150
193
|
## Writing a migration
|
|
151
194
|
|
|
152
195
|
Generate one rather than hand-rolling the header:
|
|
@@ -299,5 +342,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
|
|
|
299
342
|
},
|
|
300
343
|
'python_requires': '>=3.10',
|
|
301
344
|
'url': 'https://github.com/mcjug2015/spark_sql_migrations',
|
|
302
|
-
'version': '0.0.
|
|
345
|
+
'version': '0.0.9',
|
|
303
346
|
})
|
|
@@ -27,7 +27,7 @@ import os
|
|
|
27
27
|
import re
|
|
28
28
|
import uuid
|
|
29
29
|
from dataclasses import dataclass
|
|
30
|
-
from typing import Any, Dict, List, Optional
|
|
30
|
+
from typing import Any, Dict, List, Optional, Set
|
|
31
31
|
|
|
32
32
|
from jinja2 import Environment, FileSystemLoader, PackageLoader, select_autoescape
|
|
33
33
|
from pyspark.sql.session import SparkSession
|
|
@@ -39,6 +39,8 @@ logger = logging.getLogger(__name__)
|
|
|
39
39
|
|
|
40
40
|
|
|
41
41
|
VERSION_TABLE = "_spark_migrations_version"
|
|
42
|
+
# what `history` prints in place of the revision a root migration follows
|
|
43
|
+
BASE_LABEL = "<base>"
|
|
42
44
|
|
|
43
45
|
PACKAGE_NAME = "spark_sql_migrations"
|
|
44
46
|
INITIAL_MIGRATIONS_PACKAGE_PATH = "migrations_initial"
|
|
@@ -58,6 +60,25 @@ class Migration:
|
|
|
58
60
|
template_name: str
|
|
59
61
|
|
|
60
62
|
|
|
63
|
+
@dataclass(frozen=True)
|
|
64
|
+
class ChainSummary:
|
|
65
|
+
"""a directory's chain as `history` found it, forks and all.
|
|
66
|
+
|
|
67
|
+
``ordered`` is oldest first and holds every migration in the directory, so a
|
|
68
|
+
listing built from it never silently drops a file. Everything else is what
|
|
69
|
+
makes the chain unapplicable: more than one ``heads`` entry is a split head,
|
|
70
|
+
a ``branch_points`` entry is where it split, more than one ``roots`` entry is
|
|
71
|
+
a second chain in the same directory, and ``orphans`` are files no walk from
|
|
72
|
+
any root reaches -- a cycle, in practice.
|
|
73
|
+
"""
|
|
74
|
+
|
|
75
|
+
ordered: List[Migration]
|
|
76
|
+
heads: List[Migration]
|
|
77
|
+
roots: List[Migration]
|
|
78
|
+
branch_points: Dict[str, List[Migration]]
|
|
79
|
+
orphans: List[Migration]
|
|
80
|
+
|
|
81
|
+
|
|
61
82
|
def get_default_template_path() -> str:
|
|
62
83
|
"""the template this package ships; callers may pass their own instead."""
|
|
63
84
|
return os.path.join(os.path.dirname(__file__), "..", TEMPLATES_PACKAGE_PATH, DEFAULT_TEMPLATE_NAME)
|
|
@@ -137,22 +158,32 @@ def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migratio
|
|
|
137
158
|
return ordered
|
|
138
159
|
|
|
139
160
|
|
|
140
|
-
def
|
|
141
|
-
"""every
|
|
161
|
+
def read_migrations_dir(migrations_dir: str) -> List[Migration]:
|
|
162
|
+
"""every .sql file in one directory, parsed, in filename order and unwalked.
|
|
142
163
|
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
at all.
|
|
164
|
+
the headers are all this reads, so a chain that forks or points at nothing
|
|
165
|
+
comes back whole -- it is `get_ordered_migration_objs` that insists on a
|
|
166
|
+
single walkable chain, and `history` that reports one which is not.
|
|
147
167
|
"""
|
|
148
168
|
if not os.path.isdir(migrations_dir):
|
|
149
169
|
raise ValueError(f"no migrations dir at {migrations_dir};")
|
|
150
170
|
|
|
151
|
-
|
|
171
|
+
return [
|
|
152
172
|
_parse_migration(migrations_dir, template_name)
|
|
153
173
|
for template_name in sorted(os.listdir(migrations_dir))
|
|
154
174
|
if template_name.endswith(".sql")
|
|
155
175
|
]
|
|
176
|
+
|
|
177
|
+
|
|
178
|
+
def get_migrations_list(migrations_dir: str, max_revision_id: Optional[str] = None) -> List[Migration]:
|
|
179
|
+
"""every migration in one directory, walked from its root to its head.
|
|
180
|
+
|
|
181
|
+
``revision_id`` cuts the walk short: the chain is returned up to and including
|
|
182
|
+
the migration carrying that revision. A revision no migration in the directory
|
|
183
|
+
carries is not an error -- the whole chain comes back, same as for no revision
|
|
184
|
+
at all.
|
|
185
|
+
"""
|
|
186
|
+
migrations = read_migrations_dir(migrations_dir)
|
|
156
187
|
if not migrations:
|
|
157
188
|
return []
|
|
158
189
|
|
|
@@ -167,6 +198,92 @@ def get_migrations_list(migrations_dir: str, max_revision_id: Optional[str] = No
|
|
|
167
198
|
return ordered
|
|
168
199
|
|
|
169
200
|
|
|
201
|
+
def get_children_by_revision(migrations: List[Migration]) -> Dict[Optional[str], List[Migration]]:
|
|
202
|
+
"""each revision's followers, keyed by the revision they follow.
|
|
203
|
+
|
|
204
|
+
the root's followers sit under None. A revision with two followers is a fork,
|
|
205
|
+
which is why this returns a list and not one migration.
|
|
206
|
+
"""
|
|
207
|
+
children: Dict[Optional[str], List[Migration]] = {}
|
|
208
|
+
for migration in sorted(migrations, key=lambda migration: migration.template_name):
|
|
209
|
+
children.setdefault(migration.prev_revision_id, []).append(migration)
|
|
210
|
+
return children
|
|
211
|
+
|
|
212
|
+
|
|
213
|
+
def summarize_chain(migrations: List[Migration]) -> ChainSummary:
|
|
214
|
+
"""walk a chain for reading rather than for applying.
|
|
215
|
+
|
|
216
|
+
this raises nothing: a fork, a second root or a revision following one no file
|
|
217
|
+
carries is what the caller asked to be shown, not a reason to refuse. The walk
|
|
218
|
+
is depth first from each root, so one branch of a fork is listed to its head
|
|
219
|
+
before the next branch starts, and a revision already listed is not listed
|
|
220
|
+
again -- which is what keeps a cycle from looping here.
|
|
221
|
+
"""
|
|
222
|
+
children = get_children_by_revision(migrations)
|
|
223
|
+
known_revisions = {migration.revision_id for migration in migrations}
|
|
224
|
+
roots = [
|
|
225
|
+
migration
|
|
226
|
+
for migration in sorted(migrations, key=lambda migration: migration.template_name)
|
|
227
|
+
if not migration.prev_revision_id or migration.prev_revision_id not in known_revisions
|
|
228
|
+
]
|
|
229
|
+
|
|
230
|
+
ordered: List[Migration] = []
|
|
231
|
+
seen: Set[Migration] = set()
|
|
232
|
+
stack = list(reversed(roots))
|
|
233
|
+
while stack:
|
|
234
|
+
migration = stack.pop()
|
|
235
|
+
if migration in seen:
|
|
236
|
+
continue
|
|
237
|
+
seen.add(migration)
|
|
238
|
+
ordered.append(migration)
|
|
239
|
+
stack.extend(reversed(children.get(migration.revision_id, [])))
|
|
240
|
+
|
|
241
|
+
# orphan - somehow comes from no root
|
|
242
|
+
orphans = [migration for migration in migrations if migration not in seen]
|
|
243
|
+
ordered.extend(sorted(orphans, key=lambda migration: migration.template_name))
|
|
244
|
+
|
|
245
|
+
return ChainSummary(
|
|
246
|
+
ordered=ordered,
|
|
247
|
+
heads=[migration for migration in ordered if not children.get(migration.revision_id)],
|
|
248
|
+
roots=roots,
|
|
249
|
+
branch_points={
|
|
250
|
+
revision_id: followers for revision_id, followers in children.items() if revision_id and len(followers) > 1
|
|
251
|
+
},
|
|
252
|
+
orphans=orphans,
|
|
253
|
+
)
|
|
254
|
+
|
|
255
|
+
|
|
256
|
+
def format_history(summary: ChainSummary, limit: Optional[int] = None, oldest_first: bool = False) -> List[str]:
|
|
257
|
+
"""the listing, newest first like alembic's unless asked for the other end.
|
|
258
|
+
|
|
259
|
+
``limit`` cuts from the far end, so it keeps the newest with the default order
|
|
260
|
+
and the oldest with ``oldest_first`` -- either way, the end the listing starts
|
|
261
|
+
at is the end that survives. The notes are appended whole, even to a listing
|
|
262
|
+
cut short, because a split head is the thing worth seeing most.
|
|
263
|
+
"""
|
|
264
|
+
if limit is not None and limit < 1:
|
|
265
|
+
raise ValueError(f"a history limit lists at least one migration, got {limit};")
|
|
266
|
+
|
|
267
|
+
ordered = summary.ordered if oldest_first else list(reversed(summary.ordered))
|
|
268
|
+
if limit is not None:
|
|
269
|
+
ordered = ordered[:limit]
|
|
270
|
+
|
|
271
|
+
head_revisions = {migration.revision_id for migration in summary.heads}
|
|
272
|
+
branch_point_revisions = set(summary.branch_points)
|
|
273
|
+
lines = [format_history_line(migration, head_revisions, branch_point_revisions) for migration in ordered]
|
|
274
|
+
return lines + format_chain_notes(summary)
|
|
275
|
+
|
|
276
|
+
|
|
277
|
+
def history(migrations_dir: str, limit: Optional[int] = None, oldest_first: bool = False) -> List[str]:
|
|
278
|
+
"""the chain in one directory, as lines ready to be printed.
|
|
279
|
+
|
|
280
|
+
the files on disk are the whole of it: no catalog is read and no session is
|
|
281
|
+
built, so a chain can be read anywhere, whatever any catalog has been migrated
|
|
282
|
+
to. `get_table_version` is what says where a catalog actually stands.
|
|
283
|
+
"""
|
|
284
|
+
return format_history(summarize_chain(read_migrations_dir(migrations_dir)), limit=limit, oldest_first=oldest_first)
|
|
285
|
+
|
|
286
|
+
|
|
170
287
|
def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Row]:
|
|
171
288
|
"""the revision VERSION_TABLE records, or None if it holds no row yet."""
|
|
172
289
|
version_table = f"{cat}.{schema}.{VERSION_TABLE}"
|
|
@@ -359,6 +476,45 @@ def run_upgrade(
|
|
|
359
476
|
upgrade(spark, client_output_folder, migrations_dir, cat, schema, max_revision_id)
|
|
360
477
|
|
|
361
478
|
|
|
479
|
+
def format_history_line(
|
|
480
|
+
migration: Migration, head_revisions: Set[str], branch_point_revisions: Set[str]
|
|
481
|
+
) -> str: # pragma: no cover
|
|
482
|
+
"""one alembic-shaped line: `prev -> rev (tags), filename`."""
|
|
483
|
+
tags = []
|
|
484
|
+
if migration.revision_id in head_revisions:
|
|
485
|
+
tags.append("head")
|
|
486
|
+
if migration.revision_id in branch_point_revisions:
|
|
487
|
+
tags.append("branchpoint")
|
|
488
|
+
tag_text = f" ({', '.join(tags)})" if tags else ""
|
|
489
|
+
return f"{migration.prev_revision_id or BASE_LABEL} -> {migration.revision_id}{tag_text}, {migration.template_name}"
|
|
490
|
+
|
|
491
|
+
|
|
492
|
+
def format_chain_notes(summary: ChainSummary) -> List[str]: # pragma: no cover
|
|
493
|
+
"""what is wrong with this chain, below the listing, or nothing if it is sound."""
|
|
494
|
+
notes = []
|
|
495
|
+
if len(summary.heads) > 1:
|
|
496
|
+
notes.append(
|
|
497
|
+
f"split heads: {', '.join(migration.revision_id for migration in summary.heads)}; "
|
|
498
|
+
f"a chain with more than one head cannot be applied until it is rejoined;"
|
|
499
|
+
)
|
|
500
|
+
for revision_id, followers in sorted(summary.branch_points.items()):
|
|
501
|
+
notes.append(
|
|
502
|
+
f"{revision_id} is the prev_revision_id of "
|
|
503
|
+
f"{', '.join(migration.template_name for migration in followers)};"
|
|
504
|
+
)
|
|
505
|
+
if len(summary.roots) > 1:
|
|
506
|
+
notes.append(
|
|
507
|
+
f"more than one root: {', '.join(migration.template_name for migration in summary.roots)}; "
|
|
508
|
+
f"a chain has exactly one migration following no other;"
|
|
509
|
+
)
|
|
510
|
+
if summary.orphans:
|
|
511
|
+
notes.append(
|
|
512
|
+
f"no root reaches {', '.join(migration.template_name for migration in summary.orphans)}; "
|
|
513
|
+
f"they are listed together at the tail end of the chain order, having none of their own;"
|
|
514
|
+
)
|
|
515
|
+
return notes
|
|
516
|
+
|
|
517
|
+
|
|
362
518
|
def main(
|
|
363
519
|
cat: str,
|
|
364
520
|
schema: str,
|
|
@@ -383,6 +539,12 @@ def _cli_create_new_migration(
|
|
|
383
539
|
create_new_migration(message, template_path or get_default_template_path(), output_path, add_is_dbr=add_is_dbr)
|
|
384
540
|
|
|
385
541
|
|
|
542
|
+
def _cli_history(migrations_dir: str, limit: Optional[int], oldest_first: bool) -> None: # pragma: no cover
|
|
543
|
+
"""the listing goes to stdout, so it pipes; the log records go to stderr as always."""
|
|
544
|
+
for line in history(migrations_dir, limit=limit, oldest_first=oldest_first):
|
|
545
|
+
print(line)
|
|
546
|
+
|
|
547
|
+
|
|
386
548
|
def build_parser() -> argparse.ArgumentParser: # pragma: no cover
|
|
387
549
|
parser = argparse.ArgumentParser(description="spark_sql_migrations migration utilities")
|
|
388
550
|
subparsers = parser.add_subparsers(dest="command", required=True)
|
|
@@ -419,6 +581,22 @@ def build_parser() -> argparse.ArgumentParser: # pragma: no cover
|
|
|
419
581
|
)
|
|
420
582
|
p_create.set_defaults(func=_cli_create_new_migration)
|
|
421
583
|
|
|
584
|
+
p_history = subparsers.add_parser("history", help="list the migration chain on disk, newest first")
|
|
585
|
+
p_history.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the chain")
|
|
586
|
+
p_history.add_argument(
|
|
587
|
+
"-n",
|
|
588
|
+
"--limit",
|
|
589
|
+
type=int,
|
|
590
|
+
default=None,
|
|
591
|
+
help="list at most this many migrations, counted from the end the listing starts at",
|
|
592
|
+
)
|
|
593
|
+
p_history.add_argument(
|
|
594
|
+
"--oldest-first",
|
|
595
|
+
action="store_true",
|
|
596
|
+
help="list from the root up instead of from the head down",
|
|
597
|
+
)
|
|
598
|
+
p_history.set_defaults(func=_cli_history)
|
|
599
|
+
|
|
422
600
|
return parser
|
|
423
601
|
|
|
424
602
|
|
{spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/PKG-INFO
RENAMED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: spark_sql_migrations
|
|
3
|
-
Version: 0.0.
|
|
3
|
+
Version: 0.0.9
|
|
4
4
|
Summary: Chained, idempotent SQL migrations for Spark and Databricks.
|
|
5
5
|
Home-page: https://github.com/mcjug2015/spark_sql_migrations
|
|
6
6
|
Author: Victor Semenov
|
|
@@ -143,6 +143,49 @@ migrations_dir)` directly.
|
|
|
143
143
|
Every rendered statement is written to a timestamped folder under `migrations_out/`
|
|
144
144
|
before it is executed, so you can always read the exact SQL a run applied.
|
|
145
145
|
|
|
146
|
+
## Reading the chain
|
|
147
|
+
|
|
148
|
+
```bash
|
|
149
|
+
spark-sql-migrations history --migrations-dir path/to/migrations
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
|
|
153
|
+
with `<base>` standing in for the revision a root migration follows:
|
|
154
|
+
|
|
155
|
+
```
|
|
156
|
+
07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
|
|
157
|
+
e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
|
|
158
|
+
<base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
|
|
159
|
+
```
|
|
160
|
+
|
|
161
|
+
| Flag | Effect |
|
|
162
|
+
|---|---|
|
|
163
|
+
| `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
|
|
164
|
+
| `--oldest-first` | walk from the root up instead of from the head down |
|
|
165
|
+
|
|
166
|
+
`-n 5` with the default order gives the five newest; with `--oldest-first`, the five
|
|
167
|
+
oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
|
|
168
|
+
session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
|
|
169
|
+
nothing about how far any catalog has been migrated.
|
|
170
|
+
|
|
171
|
+
Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
|
|
172
|
+
want it to describe. A fork is tagged where it happens and spelled out below the listing,
|
|
173
|
+
which is kept whole even when `-n` cut the listing short:
|
|
174
|
+
|
|
175
|
+
```
|
|
176
|
+
c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
|
|
177
|
+
c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
|
|
178
|
+
b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
|
|
179
|
+
split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
|
|
180
|
+
c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
|
|
181
|
+
```
|
|
182
|
+
|
|
183
|
+
Two migrations with an empty `prev_revision_id` (two chains in one directory) and
|
|
184
|
+
migrations no root reaches (a cycle) get the same treatment. From Python,
|
|
185
|
+
`history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
|
|
186
|
+
list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
|
|
187
|
+
branch points and orphans as objects.
|
|
188
|
+
|
|
146
189
|
## Writing a migration
|
|
147
190
|
|
|
148
191
|
Generate one rather than hand-rolling the header:
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/__init__.py
RENAMED
|
File without changes
|
{spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_utils.py
RENAMED
|
File without changes
|
{spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/SOURCES.txt
RENAMED
|
File without changes
|
|
File without changes
|
|
File without changes
|
|
File without changes
|
{spark_sql_migrations-0.0.8 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/requires.txt
RENAMED
|
File without changes
|
|
File without changes
|