spark-sql-migrations 0.0.7__tar.gz → 0.0.9__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (22) hide show
  1. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/PKG-INFO +44 -1
  2. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/setup.py +44 -1
  3. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/spark_sql.py +252 -22
  4. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/PKG-INFO +44 -1
  5. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/LICENSE +0 -0
  6. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/MANIFEST.in +0 -0
  7. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/backend_shim.py +0 -0
  8. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/setup.cfg +0 -0
  9. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/__init__.py +0 -0
  10. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migration_templates/default_migration_template.sql +0 -0
  11. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql +0 -0
  12. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql +0 -0
  13. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql +0 -0
  14. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/py.typed +0 -0
  15. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_sql/__init__.py +0 -0
  16. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations/spark_utils.py +0 -0
  17. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/SOURCES.txt +0 -0
  18. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/dependency_links.txt +0 -0
  19. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/entry_points.txt +0 -0
  20. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/namespace_packages.txt +0 -0
  21. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/requires.txt +0 -0
  22. {spark_sql_migrations-0.0.7 → spark_sql_migrations-0.0.9}/spark_sql_migrations.egg-info/top_level.txt +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: spark_sql_migrations
3
- Version: 0.0.7
3
+ Version: 0.0.9
4
4
  Summary: Chained, idempotent SQL migrations for Spark and Databricks.
5
5
  Home-page: https://github.com/mcjug2015/spark_sql_migrations
6
6
  Author: Victor Semenov
@@ -143,6 +143,49 @@ migrations_dir)` directly.
143
143
  Every rendered statement is written to a timestamped folder under `migrations_out/`
144
144
  before it is executed, so you can always read the exact SQL a run applied.
145
145
 
146
+ ## Reading the chain
147
+
148
+ ```bash
149
+ spark-sql-migrations history --migrations-dir path/to/migrations
150
+ ```
151
+
152
+ Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
153
+ with `<base>` standing in for the revision a root migration follows:
154
+
155
+ ```
156
+ 07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
157
+ e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
158
+ <base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
159
+ ```
160
+
161
+ | Flag | Effect |
162
+ |---|---|
163
+ | `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
164
+ | `--oldest-first` | walk from the root up instead of from the head down |
165
+
166
+ `-n 5` with the default order gives the five newest; with `--oldest-first`, the five
167
+ oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
168
+ session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
169
+ nothing about how far any catalog has been migrated.
170
+
171
+ Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
172
+ want it to describe. A fork is tagged where it happens and spelled out below the listing,
173
+ which is kept whole even when `-n` cut the listing short:
174
+
175
+ ```
176
+ c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
177
+ c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
178
+ b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
179
+ split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
180
+ c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
181
+ ```
182
+
183
+ Two migrations with an empty `prev_revision_id` (two chains in one directory) and
184
+ migrations no root reaches (a cycle) get the same treatment. From Python,
185
+ `history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
186
+ list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
187
+ branch points and orphans as objects.
188
+
146
189
  ## Writing a migration
147
190
 
148
191
  Generate one rather than hand-rolling the header:
@@ -147,6 +147,49 @@ migrations_dir)` directly.
147
147
  Every rendered statement is written to a timestamped folder under `migrations_out/`
148
148
  before it is executed, so you can always read the exact SQL a run applied.
149
149
 
150
+ ## Reading the chain
151
+
152
+ ```bash
153
+ spark-sql-migrations history --migrations-dir path/to/migrations
154
+ ```
155
+
156
+ Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
157
+ with `<base>` standing in for the revision a root migration follows:
158
+
159
+ ```
160
+ 07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
161
+ e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
162
+ <base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
163
+ ```
164
+
165
+ | Flag | Effect |
166
+ |---|---|
167
+ | `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
168
+ | `--oldest-first` | walk from the root up instead of from the head down |
169
+
170
+ `-n 5` with the default order gives the five newest; with `--oldest-first`, the five
171
+ oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
172
+ session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
173
+ nothing about how far any catalog has been migrated.
174
+
175
+ Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
176
+ want it to describe. A fork is tagged where it happens and spelled out below the listing,
177
+ which is kept whole even when `-n` cut the listing short:
178
+
179
+ ```
180
+ c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
181
+ c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
182
+ b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
183
+ split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
184
+ c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
185
+ ```
186
+
187
+ Two migrations with an empty `prev_revision_id` (two chains in one directory) and
188
+ migrations no root reaches (a cycle) get the same treatment. From Python,
189
+ `history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
190
+ list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
191
+ branch points and orphans as objects.
192
+
150
193
  ## Writing a migration
151
194
 
152
195
  Generate one rather than hand-rolling the header:
@@ -299,5 +342,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
299
342
  },
300
343
  'python_requires': '>=3.10',
301
344
  'url': 'https://github.com/mcjug2015/spark_sql_migrations',
302
- 'version': '0.0.7',
345
+ 'version': '0.0.9',
303
346
  })
@@ -27,7 +27,7 @@ import os
27
27
  import re
28
28
  import uuid
29
29
  from dataclasses import dataclass
30
- from typing import Any, Dict, List, Optional
30
+ from typing import Any, Dict, List, Optional, Set
31
31
 
32
32
  from jinja2 import Environment, FileSystemLoader, PackageLoader, select_autoescape
33
33
  from pyspark.sql.session import SparkSession
@@ -39,6 +39,8 @@ logger = logging.getLogger(__name__)
39
39
 
40
40
 
41
41
  VERSION_TABLE = "_spark_migrations_version"
42
+ # what `history` prints in place of the revision a root migration follows
43
+ BASE_LABEL = "<base>"
42
44
 
43
45
  PACKAGE_NAME = "spark_sql_migrations"
44
46
  INITIAL_MIGRATIONS_PACKAGE_PATH = "migrations_initial"
@@ -54,10 +56,29 @@ _COMMENT_OR_BLANK_RE = re.compile(r"^\s*(--.*)?$")
54
56
  @dataclass(frozen=True)
55
57
  class Migration:
56
58
  revision_id: str
57
- prev_revision_id: str | None
59
+ prev_revision_id: Optional[str]
58
60
  template_name: str
59
61
 
60
62
 
63
+ @dataclass(frozen=True)
64
+ class ChainSummary:
65
+ """a directory's chain as `history` found it, forks and all.
66
+
67
+ ``ordered`` is oldest first and holds every migration in the directory, so a
68
+ listing built from it never silently drops a file. Everything else is what
69
+ makes the chain unapplicable: more than one ``heads`` entry is a split head,
70
+ a ``branch_points`` entry is where it split, more than one ``roots`` entry is
71
+ a second chain in the same directory, and ``orphans`` are files no walk from
72
+ any root reaches -- a cycle, in practice.
73
+ """
74
+
75
+ ordered: List[Migration]
76
+ heads: List[Migration]
77
+ roots: List[Migration]
78
+ branch_points: Dict[str, List[Migration]]
79
+ orphans: List[Migration]
80
+
81
+
61
82
  def get_default_template_path() -> str:
62
83
  """the template this package ships; callers may pass their own instead."""
63
84
  return os.path.join(os.path.dirname(__file__), "..", TEMPLATES_PACKAGE_PATH, DEFAULT_TEMPLATE_NAME)
@@ -122,7 +143,7 @@ def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migratio
122
143
  next_by_revision[migration.prev_revision_id] = migration
123
144
 
124
145
  ordered = []
125
- current: Migration | None = roots[0]
146
+ current: Optional[Migration] = roots[0]
126
147
  while current:
127
148
  ordered.append(current)
128
149
  current = next_by_revision.get(current.revision_id)
@@ -137,20 +158,130 @@ def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migratio
137
158
  return ordered
138
159
 
139
160
 
140
- def get_migrations_list(migrations_dir: str) -> List[Migration]:
141
- """every migration in one directory, walked from its root to its head."""
161
+ def read_migrations_dir(migrations_dir: str) -> List[Migration]:
162
+ """every .sql file in one directory, parsed, in filename order and unwalked.
163
+
164
+ the headers are all this reads, so a chain that forks or points at nothing
165
+ comes back whole -- it is `get_ordered_migration_objs` that insists on a
166
+ single walkable chain, and `history` that reports one which is not.
167
+ """
142
168
  if not os.path.isdir(migrations_dir):
143
169
  raise ValueError(f"no migrations dir at {migrations_dir};")
144
170
 
145
- migrations = [
171
+ return [
146
172
  _parse_migration(migrations_dir, template_name)
147
173
  for template_name in sorted(os.listdir(migrations_dir))
148
174
  if template_name.endswith(".sql")
149
175
  ]
176
+
177
+
178
+ def get_migrations_list(migrations_dir: str, max_revision_id: Optional[str] = None) -> List[Migration]:
179
+ """every migration in one directory, walked from its root to its head.
180
+
181
+ ``revision_id`` cuts the walk short: the chain is returned up to and including
182
+ the migration carrying that revision. A revision no migration in the directory
183
+ carries is not an error -- the whole chain comes back, same as for no revision
184
+ at all.
185
+ """
186
+ migrations = read_migrations_dir(migrations_dir)
150
187
  if not migrations:
151
188
  return []
152
189
 
153
- return get_ordered_migration_objs(migrations)
190
+ ordered = get_ordered_migration_objs(migrations)
191
+ if max_revision_id is None:
192
+ return ordered
193
+
194
+ for index, migration in enumerate(ordered):
195
+ if migration.revision_id == max_revision_id:
196
+ return ordered[: index + 1]
197
+
198
+ return ordered
199
+
200
+
201
+ def get_children_by_revision(migrations: List[Migration]) -> Dict[Optional[str], List[Migration]]:
202
+ """each revision's followers, keyed by the revision they follow.
203
+
204
+ the root's followers sit under None. A revision with two followers is a fork,
205
+ which is why this returns a list and not one migration.
206
+ """
207
+ children: Dict[Optional[str], List[Migration]] = {}
208
+ for migration in sorted(migrations, key=lambda migration: migration.template_name):
209
+ children.setdefault(migration.prev_revision_id, []).append(migration)
210
+ return children
211
+
212
+
213
+ def summarize_chain(migrations: List[Migration]) -> ChainSummary:
214
+ """walk a chain for reading rather than for applying.
215
+
216
+ this raises nothing: a fork, a second root or a revision following one no file
217
+ carries is what the caller asked to be shown, not a reason to refuse. The walk
218
+ is depth first from each root, so one branch of a fork is listed to its head
219
+ before the next branch starts, and a revision already listed is not listed
220
+ again -- which is what keeps a cycle from looping here.
221
+ """
222
+ children = get_children_by_revision(migrations)
223
+ known_revisions = {migration.revision_id for migration in migrations}
224
+ roots = [
225
+ migration
226
+ for migration in sorted(migrations, key=lambda migration: migration.template_name)
227
+ if not migration.prev_revision_id or migration.prev_revision_id not in known_revisions
228
+ ]
229
+
230
+ ordered: List[Migration] = []
231
+ seen: Set[Migration] = set()
232
+ stack = list(reversed(roots))
233
+ while stack:
234
+ migration = stack.pop()
235
+ if migration in seen:
236
+ continue
237
+ seen.add(migration)
238
+ ordered.append(migration)
239
+ stack.extend(reversed(children.get(migration.revision_id, [])))
240
+
241
+ # orphan - somehow comes from no root
242
+ orphans = [migration for migration in migrations if migration not in seen]
243
+ ordered.extend(sorted(orphans, key=lambda migration: migration.template_name))
244
+
245
+ return ChainSummary(
246
+ ordered=ordered,
247
+ heads=[migration for migration in ordered if not children.get(migration.revision_id)],
248
+ roots=roots,
249
+ branch_points={
250
+ revision_id: followers for revision_id, followers in children.items() if revision_id and len(followers) > 1
251
+ },
252
+ orphans=orphans,
253
+ )
254
+
255
+
256
+ def format_history(summary: ChainSummary, limit: Optional[int] = None, oldest_first: bool = False) -> List[str]:
257
+ """the listing, newest first like alembic's unless asked for the other end.
258
+
259
+ ``limit`` cuts from the far end, so it keeps the newest with the default order
260
+ and the oldest with ``oldest_first`` -- either way, the end the listing starts
261
+ at is the end that survives. The notes are appended whole, even to a listing
262
+ cut short, because a split head is the thing worth seeing most.
263
+ """
264
+ if limit is not None and limit < 1:
265
+ raise ValueError(f"a history limit lists at least one migration, got {limit};")
266
+
267
+ ordered = summary.ordered if oldest_first else list(reversed(summary.ordered))
268
+ if limit is not None:
269
+ ordered = ordered[:limit]
270
+
271
+ head_revisions = {migration.revision_id for migration in summary.heads}
272
+ branch_point_revisions = set(summary.branch_points)
273
+ lines = [format_history_line(migration, head_revisions, branch_point_revisions) for migration in ordered]
274
+ return lines + format_chain_notes(summary)
275
+
276
+
277
+ def history(migrations_dir: str, limit: Optional[int] = None, oldest_first: bool = False) -> List[str]:
278
+ """the chain in one directory, as lines ready to be printed.
279
+
280
+ the files on disk are the whole of it: no catalog is read and no session is
281
+ built, so a chain can be read anywhere, whatever any catalog has been migrated
282
+ to. `get_table_version` is what says where a catalog actually stands.
283
+ """
284
+ return format_history(summarize_chain(read_migrations_dir(migrations_dir)), limit=limit, oldest_first=oldest_first)
154
285
 
155
286
 
156
287
  def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Row]:
@@ -161,8 +292,10 @@ def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Ro
161
292
  return spark.read.table(version_table).select("version_num").head()
162
293
 
163
294
 
164
- def get_unapplied_migrations_list(spark: SparkSession, migrations_dir: str, cat: str, schema: str) -> List[Migration]:
165
- ordered = get_migrations_list(migrations_dir)
295
+ def get_unapplied_migrations_list(
296
+ spark: SparkSession, migrations_dir: str, cat: str, schema: str, max_revision_id: Optional[str] = None
297
+ ) -> List[Migration]:
298
+ ordered = get_migrations_list(migrations_dir, max_revision_id)
166
299
  current = get_table_version(spark, cat, schema)
167
300
  if not current:
168
301
  return ordered
@@ -239,9 +372,15 @@ def apply_template(output_dir: str, template, cat: str, schema: str) -> str:
239
372
 
240
373
 
241
374
  def get_ascending_letters_within_minute() -> str:
242
- micros_since_minute = datetime.datetime.now() - datetime.datetime.now().replace(second=0, microsecond=0)
243
- result = str(micros_since_minute.microseconds).translate(str.maketrans("0123456789", "ABCDEFGHIJ"))
244
- return result
375
+ """a letter stamp that sorts in the order it was taken, within one minute.
376
+
377
+ the full microseconds since the minute began, not timedelta.microseconds -- that
378
+ holds only the sub-second part and so starts over every second. zero-padded to a
379
+ fixed width, because these are compared as text once they are folder names.
380
+ """
381
+ now = datetime.datetime.now()
382
+ micros_since_minute = (now - now.replace(second=0, microsecond=0)) // datetime.timedelta(microseconds=1)
383
+ return f"{micros_since_minute:08d}".translate(str.maketrans("0123456789", "ABCDEFGHIJ"))
245
384
 
246
385
 
247
386
  def get_output_folder(output_parent_path: str) -> str:
@@ -286,7 +425,14 @@ def record_revision(spark: SparkSession, cat: str, schema: str, revision_id: str
286
425
  spark.sql(f"insert into {version_table} (version_num) values ('{revision_literal}')")
287
426
 
288
427
 
289
- def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str, cat: str, schema: str) -> None:
428
+ def upgrade(
429
+ spark: SparkSession,
430
+ output_folder: str,
431
+ migrations_dir: str,
432
+ cat: str,
433
+ schema: str,
434
+ max_revision_id: Optional[str] = None,
435
+ ) -> None:
290
436
  """
291
437
  TODO XXX permit applying up to a revision below the head revision
292
438
  TODO XXX add backwards migrations, permit migrating backwards
@@ -294,7 +440,9 @@ def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str,
294
440
  if not os.path.isdir(migrations_dir):
295
441
  logger.info(f"no chain at {migrations_dir}; skipping it;")
296
442
  return
297
- unapplied = get_unapplied_migrations_list(spark, migrations_dir, cat=cat, schema=schema)
443
+ unapplied = get_unapplied_migrations_list(
444
+ spark, migrations_dir, cat=cat, schema=schema, max_revision_id=max_revision_id
445
+ )
298
446
 
299
447
  logger.info(
300
448
  f"{len(unapplied)} unapplied migrations; "
@@ -308,7 +456,14 @@ def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str,
308
456
  logger.info(f"{migration.template_name} done; now at {migration.revision_id};")
309
457
 
310
458
 
311
- def run_migrations(spark: SparkSession, cat: str, schema: str, output_folder: str, migrations_dir: str) -> None:
459
+ def run_upgrade(
460
+ spark: SparkSession,
461
+ cat: str,
462
+ schema: str,
463
+ output_folder: str,
464
+ migrations_dir: str,
465
+ max_revision_id: Optional[str] = None,
466
+ ) -> None:
312
467
  initial_output_folder = os.path.join(output_folder, INITIAL_MIGRATIONS_PACKAGE_PATH)
313
468
  client_output_folder = os.path.join(output_folder, CLIENT_OUTPUT_DIRNAME)
314
469
  os.makedirs(initial_output_folder, exist_ok=True)
@@ -318,17 +473,64 @@ def run_migrations(spark: SparkSession, cat: str, schema: str, output_folder: st
318
473
  logger.info(f"{version_table} already exists; skipping the initial migrations;")
319
474
  else:
320
475
  migrate_initial(spark, initial_output_folder, cat, schema)
321
- migrate_w_rev(spark, client_output_folder, migrations_dir, cat, schema)
476
+ upgrade(spark, client_output_folder, migrations_dir, cat, schema, max_revision_id)
477
+
478
+
479
+ def format_history_line(
480
+ migration: Migration, head_revisions: Set[str], branch_point_revisions: Set[str]
481
+ ) -> str: # pragma: no cover
482
+ """one alembic-shaped line: `prev -> rev (tags), filename`."""
483
+ tags = []
484
+ if migration.revision_id in head_revisions:
485
+ tags.append("head")
486
+ if migration.revision_id in branch_point_revisions:
487
+ tags.append("branchpoint")
488
+ tag_text = f" ({', '.join(tags)})" if tags else ""
489
+ return f"{migration.prev_revision_id or BASE_LABEL} -> {migration.revision_id}{tag_text}, {migration.template_name}"
490
+
491
+
492
+ def format_chain_notes(summary: ChainSummary) -> List[str]: # pragma: no cover
493
+ """what is wrong with this chain, below the listing, or nothing if it is sound."""
494
+ notes = []
495
+ if len(summary.heads) > 1:
496
+ notes.append(
497
+ f"split heads: {', '.join(migration.revision_id for migration in summary.heads)}; "
498
+ f"a chain with more than one head cannot be applied until it is rejoined;"
499
+ )
500
+ for revision_id, followers in sorted(summary.branch_points.items()):
501
+ notes.append(
502
+ f"{revision_id} is the prev_revision_id of "
503
+ f"{', '.join(migration.template_name for migration in followers)};"
504
+ )
505
+ if len(summary.roots) > 1:
506
+ notes.append(
507
+ f"more than one root: {', '.join(migration.template_name for migration in summary.roots)}; "
508
+ f"a chain has exactly one migration following no other;"
509
+ )
510
+ if summary.orphans:
511
+ notes.append(
512
+ f"no root reaches {', '.join(migration.template_name for migration in summary.orphans)}; "
513
+ f"they are listed together at the tail end of the chain order, having none of their own;"
514
+ )
515
+ return notes
322
516
 
323
517
 
324
- def main(cat: str, schema: str, migrations_dir: str, output_parent_path: str | None = None) -> None:
518
+ def main(
519
+ cat: str,
520
+ schema: str,
521
+ migrations_dir: str,
522
+ output_parent_path: Optional[str] = None,
523
+ max_revision_id: Optional[str] = None,
524
+ ) -> None:
325
525
  output_folder = get_output_folder(output_parent_path or os.path.join(os.getcwd(), "migrations_out"))
326
526
  os.makedirs(output_folder)
327
- run_migrations(get_spark(), cat, schema, output_folder, migrations_dir)
527
+ run_upgrade(get_spark(), cat, schema, output_folder, migrations_dir, max_revision_id)
328
528
 
329
529
 
330
- def _cli_main(cat: str, schema: str, migrations_dir: str) -> None: # pragma: no cover
331
- main(cat, schema, migrations_dir)
530
+ def _cli_main(
531
+ cat: str, schema: str, migrations_dir: str, max_revision_id: Optional[str] = None
532
+ ) -> None: # pragma: no cover
533
+ main(cat, schema, migrations_dir, max_revision_id=max_revision_id)
332
534
 
333
535
 
334
536
  def _cli_create_new_migration(
@@ -337,14 +539,26 @@ def _cli_create_new_migration(
337
539
  create_new_migration(message, template_path or get_default_template_path(), output_path, add_is_dbr=add_is_dbr)
338
540
 
339
541
 
542
+ def _cli_history(migrations_dir: str, limit: Optional[int], oldest_first: bool) -> None: # pragma: no cover
543
+ """the listing goes to stdout, so it pipes; the log records go to stderr as always."""
544
+ for line in history(migrations_dir, limit=limit, oldest_first=oldest_first):
545
+ print(line)
546
+
547
+
340
548
  def build_parser() -> argparse.ArgumentParser: # pragma: no cover
341
549
  parser = argparse.ArgumentParser(description="spark_sql_migrations migration utilities")
342
550
  subparsers = parser.add_subparsers(dest="command", required=True)
343
551
 
344
- p_run = subparsers.add_parser("run", help="apply all pending migrations")
552
+ p_run = subparsers.add_parser("run", help="upgrade to head/apply pending migrations")
345
553
  p_run.add_argument("--cat", type=str, default="spark_catalog")
346
554
  p_run.add_argument("--schema", type=str, default="default")
347
555
  p_run.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the migration chain")
556
+ p_run.add_argument(
557
+ "--max-revision-id",
558
+ type=str,
559
+ default=None,
560
+ help="apply the migration carrying this revision_id and stop; defaults to applying the chain to its head",
561
+ )
348
562
  p_run.set_defaults(func=_cli_main)
349
563
 
350
564
  p_create = subparsers.add_parser("create_new_migration", help="create a new migration from the template")
@@ -367,10 +581,26 @@ def build_parser() -> argparse.ArgumentParser: # pragma: no cover
367
581
  )
368
582
  p_create.set_defaults(func=_cli_create_new_migration)
369
583
 
584
+ p_history = subparsers.add_parser("history", help="list the migration chain on disk, newest first")
585
+ p_history.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the chain")
586
+ p_history.add_argument(
587
+ "-n",
588
+ "--limit",
589
+ type=int,
590
+ default=None,
591
+ help="list at most this many migrations, counted from the end the listing starts at",
592
+ )
593
+ p_history.add_argument(
594
+ "--oldest-first",
595
+ action="store_true",
596
+ help="list from the root up instead of from the head down",
597
+ )
598
+ p_history.set_defaults(func=_cli_history)
599
+
370
600
  return parser
371
601
 
372
602
 
373
- def cli(argv: list[str] | None = None) -> None: # pragma: no cover
603
+ def cli(argv: Optional[list[str]] = None) -> None: # pragma: no cover
374
604
  """the `spark-sql-migrations` console command.
375
605
 
376
606
  an application entry point, so unlike the rest of this package it configures
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: spark_sql_migrations
3
- Version: 0.0.7
3
+ Version: 0.0.9
4
4
  Summary: Chained, idempotent SQL migrations for Spark and Databricks.
5
5
  Home-page: https://github.com/mcjug2015/spark_sql_migrations
6
6
  Author: Victor Semenov
@@ -143,6 +143,49 @@ migrations_dir)` directly.
143
143
  Every rendered statement is written to a timestamped folder under `migrations_out/`
144
144
  before it is executed, so you can always read the exact SQL a run applied.
145
145
 
146
+ ## Reading the chain
147
+
148
+ ```bash
149
+ spark-sql-migrations history --migrations-dir path/to/migrations
150
+ ```
151
+
152
+ Prints one line per migration, newest first, in alembic's shape — `prev -> rev, file`,
153
+ with `<base>` standing in for the revision a root migration follows:
154
+
155
+ ```
156
+ 07990e2a101e -> 57037b6c19b4 (head), 260901_03_batch_id_for_metrics_e5ce0039b32b.sql
157
+ e5ce0039b32b -> 07990e2a101e, 260831_02_create_metrics_table_07990e2a101e.sql
158
+ <base> -> e5ce0039b32b, 260831_01_create_test_table_e5ce0039b32b.sql
159
+ ```
160
+
161
+ | Flag | Effect |
162
+ |---|---|
163
+ | `-n`, `--limit` | list at most this many, counted from the end the listing starts at |
164
+ | `--oldest-first` | walk from the root up instead of from the head down |
165
+
166
+ `-n 5` with the default order gives the five newest; with `--oldest-first`, the five
167
+ oldest. The files on disk are the whole of it: `history` reads no catalog and builds no
168
+ session, so it needs no `--cat`, no `--schema` and no Spark flavour installed, and it says
169
+ nothing about how far any catalog has been migrated.
170
+
171
+ Unlike `run`, `history` refuses nothing — a chain it could not apply is exactly what you
172
+ want it to describe. A fork is tagged where it happens and spelled out below the listing,
173
+ which is kept whole even when `-n` cut the listing short:
174
+
175
+ ```
176
+ c333 -> e555 (head), 260902_04_fork_the_chain_e555f1a2b3c4.sql
177
+ c333 -> d444 (head), 260902_04_add_a_column_d444a1b2c3d4.sql
178
+ b222 -> c333 (branchpoint), 260901_03_create_c_c333a1b2c3d4.sql
179
+ split heads: d444, e555; a chain with more than one head cannot be applied until it is rejoined;
180
+ c333 is the prev_revision_id of 260902_04_add_a_column_d444a1b2c3d4.sql, 260902_04_fork_the_chain_e555f1a2b3c4.sql;
181
+ ```
182
+
183
+ Two migrations with an empty `prev_revision_id` (two chains in one directory) and
184
+ migrations no root reaches (a cycle) get the same treatment. From Python,
185
+ `history(migrations_dir, limit=None, oldest_first=False)` returns those same lines as a
186
+ list, and `summarize_chain(read_migrations_dir(migrations_dir))` returns the heads, roots,
187
+ branch points and orphans as objects.
188
+
146
189
  ## Writing a migration
147
190
 
148
191
  Generate one rather than hand-rolling the header: