spark-sql-migrations 0.0.5__tar.gz → 0.0.7__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (23) hide show
  1. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/PKG-INFO +56 -14
  2. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/setup.py +62 -17
  3. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/migration_templates/default_migration_template.sql +3 -0
  4. spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql +9 -0
  5. spark_sql_migrations-0.0.5/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table_all.sql → spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql +1 -2
  6. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_sql/spark_sql.py +125 -98
  7. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_utils.py +3 -3
  8. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/PKG-INFO +56 -14
  9. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/SOURCES.txt +3 -3
  10. spark_sql_migrations-0.0.5/spark_sql_migrations/migrations_initial/first_01_intial_create_cat_dbr_only.sql +0 -4
  11. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/LICENSE +0 -0
  12. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/MANIFEST.in +0 -0
  13. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/backend_shim.py +0 -0
  14. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/setup.cfg +0 -0
  15. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/__init__.py +0 -0
  16. /spark_sql_migrations-0.0.5/spark_sql_migrations/migrations_initial/first_02_intial_create_schema_all.sql → /spark_sql_migrations-0.0.7/spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql +0 -0
  17. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/py.typed +0 -0
  18. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations/spark_sql/__init__.py +0 -0
  19. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/dependency_links.txt +0 -0
  20. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/entry_points.txt +0 -0
  21. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/namespace_packages.txt +0 -0
  22. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/requires.txt +0 -0
  23. {spark_sql_migrations-0.0.5 → spark_sql_migrations-0.0.7}/spark_sql_migrations.egg-info/top_level.txt +0 -0
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: spark_sql_migrations
3
- Version: 0.0.5
3
+ Version: 0.0.7
4
4
  Summary: Chained, idempotent SQL migrations for Spark and Databricks.
5
5
  Home-page: https://github.com/mcjug2015/spark_sql_migrations
6
6
  Author: Victor Semenov
@@ -79,28 +79,32 @@ Migrations come from two different owners, and they behave differently:
79
79
  | | ships in the wheel | owned by you |
80
80
  |---|---|---|
81
81
  | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
82
- | **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
83
- | **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
82
+ | **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
83
+ | **ordered by** | filename | the `prev_revision_id` chain |
84
84
  | **when applied** | until the version table exists | once, then recorded |
85
85
 
86
86
  The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
87
87
  skipped from then on. A run that fails partway through bootstrapping replays it, so it must
88
- still be idempotent. Your own chains are applied once each and their head revision is
88
+ still be idempotent. Your own migrations are applied once each and the head revision is
89
89
  written to the version table.
90
90
 
91
- ## Laying out your chains
91
+ Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
92
+ bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
92
93
 
93
- Point spark_sql_migrations at a directory holding one or both chains:
94
+ ## Laying out your chain
95
+
96
+ `--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
94
97
 
95
98
  ```
96
99
  your_project/
97
- └── migrations/ <- this is --migrations-dir
98
- ├── all_spark_migrations/ <- runs everywhere
99
- └── dbr_only_migrations/ <- runs only when is_dbr()
100
+ └── migrations/ <- this is --migrations-dir
101
+ ├── 260831_01_create_test_table_e5ce0039b32b.sql
102
+ └── 260831_02_create_metrics_table_07990e2a101e.sql
100
103
  ```
101
104
 
102
- Choose deliberately. SQL that only Databricks understands must not live in
103
- `all_spark_migrations/`, or your local runs will break.
105
+ One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
106
+ migration that otherwise runs everywhere — see below. SQL that only Databricks understands
107
+ must never sit ungated there, or your local runs will break.
104
108
 
105
109
  ## Running migrations
106
110
 
@@ -124,7 +128,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
124
128
 
125
129
 
126
130
  def get_migrations_dir():
127
- """the parent of this project's own migration chains."""
131
+ """the directory holding this project's own migration chain."""
128
132
  return os.path.dirname(__file__)
129
133
 
130
134
 
@@ -134,7 +138,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
134
138
  `main` builds its own session via `get_spark()` and, being library code, leaves logging to
135
139
  you. To supply your own — a session already
136
140
  configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
137
- migrations_root)` directly.
141
+ migrations_dir)` directly.
138
142
 
139
143
  Every rendered statement is written to a timestamped folder under `migrations_out/`
140
144
  before it is executed, so you can always read the exact SQL a run applied.
@@ -144,7 +148,7 @@ before it is executed, so you can always read the exact SQL a run applied.
144
148
  Generate one rather than hand-rolling the header:
145
149
 
146
150
  ```bash
147
- spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations/all_spark_migrations
151
+ spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations
148
152
  ```
149
153
 
150
154
  That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
@@ -183,6 +187,44 @@ begin
183
187
  end;
184
188
  ```
185
189
 
190
+ ### Databricks-only statements
191
+
192
+ `is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
193
+ Databricks understands:
194
+
195
+ ```sql
196
+ -- revision_id:e8614b76da6d;
197
+ -- prev_revision_id:57037b6c19b4;
198
+ {% if is_dbr %}
199
+ begin
200
+ grant READ FILES on external location `some-bucket` to `some_group`;
201
+ end;
202
+ {% endif %}
203
+ ```
204
+
205
+ `create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
206
+ headers so they stay readable:
207
+
208
+ ```bash
209
+ spark-sql-migrations create_new_migration --message "read files on the cms zip external location" --output-path path/to/migrations --add-is-dbr
210
+ ```
211
+
212
+ This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
213
+ its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
214
+ script never gets the chance to run — and the whole script fails to compile, not just that
215
+ statement. Jinja removes the text before `spark.sql()` ever sees it.
216
+
217
+ A migration left holding nothing but comments is recorded in the version table and not
218
+ executed, so the chain advances in step on both engines. That recording is forward-only: a
219
+ catalog migrated locally past a gated revision will not pick that statement up if it is
220
+ later migrated from Databricks. For disposable test schemas that is exactly what you want;
221
+ for a long-lived catalog, don't migrate it from both.
222
+
223
+ `_spark_migrations_version` holds one row and one column, `version_num`. There is no
224
+ in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
225
+ a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
226
+ scratch, which also clears the revision history that layout recorded.
227
+
186
228
  ## Local vs Databricks
187
229
 
188
230
  `get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
@@ -80,28 +80,32 @@ Migrations come from two different owners, and they behave differently:
80
80
  | | ships in the wheel | owned by you |
81
81
  |---|---|---|
82
82
  | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
83
- | **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
84
- | **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
83
+ | **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
84
+ | **ordered by** | filename | the `prev_revision_id` chain |
85
85
  | **when applied** | until the version table exists | once, then recorded |
86
86
 
87
87
  The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
88
88
  skipped from then on. A run that fails partway through bootstrapping replays it, so it must
89
- still be idempotent. Your own chains are applied once each and their head revision is
89
+ still be idempotent. Your own migrations are applied once each and the head revision is
90
90
  written to the version table.
91
91
 
92
- ## Laying out your chains
92
+ Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
93
+ bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
93
94
 
94
- Point spark_sql_migrations at a directory holding one or both chains:
95
+ ## Laying out your chain
96
+
97
+ `--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
95
98
 
96
99
  ```
97
100
  your_project/
98
- └── migrations/ <- this is --migrations-dir
99
- ├── all_spark_migrations/ <- runs everywhere
100
- └── dbr_only_migrations/ <- runs only when is_dbr()
101
+ └── migrations/ <- this is --migrations-dir
102
+ ├── 260831_01_create_test_table_e5ce0039b32b.sql
103
+ └── 260831_02_create_metrics_table_07990e2a101e.sql
101
104
  ```
102
105
 
103
- Choose deliberately. SQL that only Databricks understands must not live in
104
- `all_spark_migrations/`, or your local runs will break.
106
+ One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
107
+ migration that otherwise runs everywhere — see below. SQL that only Databricks understands
108
+ must never sit ungated there, or your local runs will break.
105
109
 
106
110
  ## Running migrations
107
111
 
@@ -128,7 +132,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
128
132
 
129
133
 
130
134
  def get_migrations_dir():
131
- \"\"\"the parent of this project's own migration chains.\"\"\"
135
+ \"\"\"the directory holding this project's own migration chain.\"\"\"
132
136
  return os.path.dirname(__file__)
133
137
 
134
138
 
@@ -138,7 +142,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
138
142
  `main` builds its own session via `get_spark()` and, being library code, leaves logging to
139
143
  you. To supply your own — a session already
140
144
  configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
141
- migrations_root)` directly.
145
+ migrations_dir)` directly.
142
146
 
143
147
  Every rendered statement is written to a timestamped folder under `migrations_out/`
144
148
  before it is executed, so you can always read the exact SQL a run applied.
@@ -150,7 +154,7 @@ Generate one rather than hand-rolling the header:
150
154
  ```bash
151
155
  spark-sql-migrations create_new_migration \
152
156
  --message "add batch id to metrics" \
153
- --output-path path/to/migrations/all_spark_migrations
157
+ --output-path path/to/migrations
154
158
  ```
155
159
 
156
160
  That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
@@ -189,6 +193,47 @@ begin
189
193
  end;
190
194
  ```
191
195
 
196
+ ### Databricks-only statements
197
+
198
+ `is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
199
+ Databricks understands:
200
+
201
+ ```sql
202
+ -- revision_id:e8614b76da6d;
203
+ -- prev_revision_id:57037b6c19b4;
204
+ {% if is_dbr %}
205
+ begin
206
+ grant READ FILES on external location `some-bucket` to `some_group`;
207
+ end;
208
+ {% endif %}
209
+ ```
210
+
211
+ `create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
212
+ headers so they stay readable:
213
+
214
+ ```bash
215
+ spark-sql-migrations create_new_migration \
216
+ --message "read files on the cms zip external location" \
217
+ --output-path path/to/migrations \
218
+ --add-is-dbr
219
+ ```
220
+
221
+ This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
222
+ its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
223
+ script never gets the chance to run — and the whole script fails to compile, not just that
224
+ statement. Jinja removes the text before `spark.sql()` ever sees it.
225
+
226
+ A migration left holding nothing but comments is recorded in the version table and not
227
+ executed, so the chain advances in step on both engines. That recording is forward-only: a
228
+ catalog migrated locally past a gated revision will not pick that statement up if it is
229
+ later migrated from Databricks. For disposable test schemas that is exactly what you want;
230
+ for a long-lived catalog, don't migrate it from both.
231
+
232
+ `_spark_migrations_version` holds one row and one column, `version_num`. There is no
233
+ in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
234
+ a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
235
+ scratch, which also clears the revision history that layout recorded.
236
+
192
237
  ## Local vs Databricks
193
238
 
194
239
  `get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
@@ -238,9 +283,9 @@ Apache License 2.0 — see [LICENSE](LICENSE).
238
283
  'package_data': {
239
284
  'spark_sql_migrations': (
240
285
  'migration_templates/default_migration_template.sql',
241
- 'migrations_initial/first_01_intial_create_cat_dbr_only.sql',
242
- 'migrations_initial/first_02_intial_create_schema_all.sql',
243
- 'migrations_initial/first_03_intial_create_state_table_all.sql',
286
+ 'migrations_initial/first_01_intial_create_cat.sql',
287
+ 'migrations_initial/first_02_intial_create_schema.sql',
288
+ 'migrations_initial/first_03_intial_create_state_table.sql',
244
289
  'py.typed',
245
290
  ),
246
291
  },
@@ -254,5 +299,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
254
299
  },
255
300
  'python_requires': '>=3.10',
256
301
  'url': 'https://github.com/mcjug2015/spark_sql_migrations',
257
- 'version': '0.0.5',
302
+ 'version': '0.0.7',
258
303
  })
@@ -5,6 +5,9 @@ begin
5
5
  -- would like to have applied to databricks db or local spark(for testing)
6
6
  -- crutch scaffolding will use jinja to fill in the cat and schema that are sent to it
7
7
  -- this is also taken care of in the gh build.
8
+ -- is_dbr is filled in too: statements only databricks understands go in a jinja
9
+ -- conditional on it, outside the begin/end, rather than in a second chain. pass
10
+ -- --add-is-dbr to create_new_migration to have that conditional written for you.
8
11
 
9
12
  -- e.g:
10
13
  -- create schema if not exists {{cat}}.{{schema}};
@@ -0,0 +1,9 @@
1
+ -- revision_id:;
2
+ -- a catalog is a unity catalog object: local spark has spark_catalog and no way
3
+ -- to make another, so this one is gated to databricks and renders to nothing
4
+ -- else, leaving the bootstrap chain the same three files everywhere.
5
+ {% if is_dbr %}
6
+ begin
7
+ create catalog if not exists {{cat}};
8
+ end;
9
+ {% endif %}
@@ -1,7 +1,6 @@
1
1
  -- revision_id:;
2
2
  begin
3
3
  create table if not exists {{cat}}.{{schema}}._spark_migrations_version (
4
- migration_type string not null,
5
4
  version_num string not null
6
5
  );
7
- end;
6
+ end;
@@ -3,15 +3,21 @@
3
3
  Two kinds of migration exist, and they are owned by different parties:
4
4
 
5
5
  * the *initial* chain and the new-migration template ship inside this package.
6
- They bootstrap the catalog, the schema and the version table, are selected by
7
- filename suffix (``_all.sql`` / ``_dbr_only.sql``) rather than by revision,
8
- and are applied on every run until the version table exists -- so a run that
9
- failed partway through replays them, and they must be idempotent.
10
- * the ``all_spark_migrations/`` and ``dbr_only_migrations/`` chains belong to
11
- the consuming project. The caller passes their parent directory as
12
- ``migrations_root``; each chain is walked from its root via
13
- ``prev_revision_id`` and only the tail not yet recorded in the version table
14
- is applied.
6
+ They bootstrap the catalog, the schema and the version table, carry no
7
+ revision of their own, and are applied in filename order on every run until
8
+ the version table exists -- so a run that failed partway through replays
9
+ them, and they must be idempotent.
10
+ * the client chain belongs to the consuming project. The caller passes the
11
+ directory holding it as ``migrations_dir``; the chain is walked from its root
12
+ via ``prev_revision_id`` and only the tail not yet recorded in the version
13
+ table is applied.
14
+
15
+ Which engine a migration targets is decided inside the migration, not by where it
16
+ sits: every one of them is rendered through jinja before it is executed, with
17
+ ``is_dbr`` in the render context, so a chain that runs everywhere can still gate
18
+ the statements one engine would reject. A migration left holding no statements is
19
+ recorded and not executed, which is what lets a project keep one chain and no
20
+ second head.
15
21
  """
16
22
 
17
23
  import argparse
@@ -21,26 +27,28 @@ import os
21
27
  import re
22
28
  import uuid
23
29
  from dataclasses import dataclass
30
+ from typing import Any, Dict, List, Optional
24
31
 
25
32
  from jinja2 import Environment, FileSystemLoader, PackageLoader, select_autoescape
26
- from pyspark.sql.functions import col
33
+ from pyspark.sql.session import SparkSession
34
+ from pyspark.sql.types import Row
27
35
 
28
36
  from spark_sql_migrations.spark_utils import get_spark, is_dbr
29
37
 
30
38
  logger = logging.getLogger(__name__)
31
39
 
32
40
 
33
- ALL_SPARK = "all_spark"
34
- DBR_ONLY = "dbr_only"
35
-
36
41
  VERSION_TABLE = "_spark_migrations_version"
37
42
 
38
43
  PACKAGE_NAME = "spark_sql_migrations"
39
44
  INITIAL_MIGRATIONS_PACKAGE_PATH = "migrations_initial"
45
+ # where a run's rendered client migrations land, under its output folder
46
+ CLIENT_OUTPUT_DIRNAME = "migrations"
40
47
  TEMPLATES_PACKAGE_PATH = "migration_templates"
41
48
  DEFAULT_TEMPLATE_NAME = "default_migration_template.sql"
42
49
 
43
50
  _HEADER_RE = re.compile(r"^--\s*(revision_id|prev_revision_id)\s*:\s*(\w*)\s*;$")
51
+ _COMMENT_OR_BLANK_RE = re.compile(r"^\s*(--.*)?$")
44
52
 
45
53
 
46
54
  @dataclass(frozen=True)
@@ -50,18 +58,13 @@ class Migration:
50
58
  template_name: str
51
59
 
52
60
 
53
- def get_migrations_dir(migrations_root, migration_type):
54
- """the client-owned directory holding one chain."""
55
- return os.path.join(migrations_root, f"{migration_type}_migrations")
56
-
57
-
58
- def get_default_template_path():
61
+ def get_default_template_path() -> str:
59
62
  """the template this package ships; callers may pass their own instead."""
60
63
  return os.path.join(os.path.dirname(__file__), "..", TEMPLATES_PACKAGE_PATH, DEFAULT_TEMPLATE_NAME)
61
64
 
62
65
 
63
- def _parse_migration(migrations_dir, template_name):
64
- headers = {}
66
+ def _parse_migration(migrations_dir: str, template_name: str) -> Migration:
67
+ headers: Dict[str, Any | str] = {}
65
68
  migration_file_path = os.path.join(migrations_dir, template_name)
66
69
  if os.path.getsize(migration_file_path) == 0:
67
70
  raise ValueError(f"{template_name} is empty;")
@@ -84,8 +87,8 @@ def _parse_migration(migrations_dir, template_name):
84
87
  )
85
88
 
86
89
 
87
- def get_ordered_migration_objs(migration_objs):
88
- by_revision = {}
90
+ def get_ordered_migration_objs(migration_objs: List[Migration]) -> List[Migration]:
91
+ by_revision: Dict[str, Migration] = {}
89
92
  for migration in migration_objs:
90
93
  clash = by_revision.get(migration.revision_id)
91
94
  if clash:
@@ -102,7 +105,7 @@ def get_ordered_migration_objs(migration_objs):
102
105
  f"found {[m.template_name for m in roots]};"
103
106
  )
104
107
 
105
- next_by_revision = {}
108
+ next_by_revision: Dict[str, Migration] = {}
106
109
  for migration in migration_objs:
107
110
  if not migration.prev_revision_id:
108
111
  continue
@@ -119,7 +122,7 @@ def get_ordered_migration_objs(migration_objs):
119
122
  next_by_revision[migration.prev_revision_id] = migration
120
123
 
121
124
  ordered = []
122
- current = roots[0]
125
+ current: Migration | None = roots[0]
123
126
  while current:
124
127
  ordered.append(current)
125
128
  current = next_by_revision.get(current.revision_id)
@@ -134,7 +137,7 @@ def get_ordered_migration_objs(migration_objs):
134
137
  return ordered
135
138
 
136
139
 
137
- def get_migrations_list(migrations_dir):
140
+ def get_migrations_list(migrations_dir: str) -> List[Migration]:
138
141
  """every migration in one directory, walked from its root to its head."""
139
142
  if not os.path.isdir(migrations_dir):
140
143
  raise ValueError(f"no migrations dir at {migrations_dir};")
@@ -150,36 +153,56 @@ def get_migrations_list(migrations_dir):
150
153
  return get_ordered_migration_objs(migrations)
151
154
 
152
155
 
153
- def get_table_version(spark, migration_type, cat: str, schema: str):
156
+ def get_table_version(spark: SparkSession, cat: str, schema: str) -> Optional[Row]:
157
+ """the revision VERSION_TABLE records, or None if it holds no row yet."""
154
158
  version_table = f"{cat}.{schema}.{VERSION_TABLE}"
155
159
  if not spark.catalog.tableExists(version_table):
156
160
  return None
157
- return spark.read.table(version_table).where(col("migration_type") == migration_type).select("version_num").head()
161
+ return spark.read.table(version_table).select("version_num").head()
158
162
 
159
163
 
160
- def get_unapplied_migrations_list(spark, migrations_dir, migration_type, cat: str, schema: str):
161
- """the tail of one chain after the revision VERSION_TABLE records for it.
162
-
163
- the whole chain when the table is missing or holds no row for this type.
164
- """
164
+ def get_unapplied_migrations_list(spark: SparkSession, migrations_dir: str, cat: str, schema: str) -> List[Migration]:
165
165
  ordered = get_migrations_list(migrations_dir)
166
- current = get_table_version(spark, migration_type, cat, schema)
166
+ current = get_table_version(spark, cat, schema)
167
167
  if not current:
168
168
  return ordered
169
169
 
170
170
  current_revision_id = current["version_num"]
171
171
 
172
+ if not ordered:
173
+ logger.warning(
174
+ f"{VERSION_TABLE} is at {current_revision_id}, but {migrations_dir} holds no migrations; "
175
+ f"treating the chain as retired;"
176
+ )
177
+ return ordered
178
+
172
179
  for index, migration in enumerate(ordered):
173
180
  if migration.revision_id == current_revision_id:
174
181
  return ordered[index + 1 :]
175
182
 
176
- raise ValueError(
177
- f"{VERSION_TABLE} has {migration_type} at {current_revision_id}, "
178
- f"which matches no migration in {migrations_dir};"
179
- )
183
+ raise ValueError(f"{VERSION_TABLE} is at {current_revision_id}, which matches no migration in {migrations_dir};")
180
184
 
181
185
 
182
- def create_new_migration(message: str, template_path: str, output_path: str, truncate_slug_length: int = 40):
186
+ def gate_on_is_dbr(migration_content: str) -> str:
187
+ """wrap a migration's body in the jinja conditional, below its headers.
188
+
189
+ the revision headers stay outside it: they are read off the file as comments,
190
+ and the first line that is neither comment nor blank ends that read -- so an
191
+ opening tag above them would hide them. a body-less template gets an empty
192
+ gate at the end, which renders to nothing either way.
193
+ """
194
+ lines = migration_content.splitlines()
195
+ body_start = len(lines)
196
+ for index, line in enumerate(lines):
197
+ if line.strip() and not line.startswith("--"):
198
+ body_start = index
199
+ break
200
+ return "\n".join(lines[:body_start] + ["{% if is_dbr %}"] + lines[body_start:] + ["{% endif %}"]) + "\n"
201
+
202
+
203
+ def create_new_migration(
204
+ message: str, template_path: str, output_path: str, truncate_slug_length: int = 40, add_is_dbr: bool = False
205
+ ) -> None:
183
206
  rev_id = uuid.uuid4().hex[-12:]
184
207
 
185
208
  date_prefix = datetime.datetime.today().strftime("%y%m%d")
@@ -201,149 +224,153 @@ def create_new_migration(message: str, template_path: str, output_path: str, tru
201
224
  default_migration_content = open(template_path).read()
202
225
  default_migration_content = default_migration_content.replace("{{revision_id}}", rev_id)
203
226
  default_migration_content = default_migration_content.replace("{{prev_revision_id}}", last_migration_rev_id)
227
+ if add_is_dbr:
228
+ default_migration_content = gate_on_is_dbr(default_migration_content)
204
229
 
205
230
  with open(migration_path, "w") as migration_file:
206
231
  migration_file.write(default_migration_content)
207
232
 
208
233
 
209
- def apply_template(output_dir, template, cat: str, schema: str):
210
- result_sql = template.render(cat=cat, schema=schema)
234
+ def apply_template(output_dir: str, template, cat: str, schema: str) -> str:
235
+ result_sql = template.render(cat=cat, schema=schema, is_dbr=is_dbr())
211
236
  with open(os.path.join(output_dir, template.name.replace(".sql", "_primed.sql")), "w") as file_handle:
212
237
  file_handle.write(result_sql)
213
238
  return result_sql
214
239
 
215
240
 
216
- def get_ascending_letters_within_minute():
241
+ def get_ascending_letters_within_minute() -> str:
217
242
  micros_since_minute = datetime.datetime.now() - datetime.datetime.now().replace(second=0, microsecond=0)
218
243
  result = str(micros_since_minute.microseconds).translate(str.maketrans("0123456789", "ABCDEFGHIJ"))
219
244
  return result
220
245
 
221
246
 
222
- def get_output_folder(output_parent_path):
247
+ def get_output_folder(output_parent_path: str) -> str:
223
248
  folder_name = f"{datetime.datetime.today().strftime('%Y%m%d_%H%M')}_{get_ascending_letters_within_minute()}"
224
249
  return os.path.join(output_parent_path, folder_name)
225
250
 
226
251
 
227
- def use_migration_file(fname):
228
- if fname.endswith("all.sql"):
229
- return True
230
- elif is_dbr() and fname.endswith("dbr_only.sql"):
231
- return True
232
- return False
252
+ def has_no_statements(rendered_sql: str) -> bool:
253
+ for line in rendered_sql.splitlines():
254
+ if not _COMMENT_OR_BLANK_RE.match(line):
255
+ return False
256
+ return True
233
257
 
234
258
 
235
- def migrate_initial(spark, output_folder, cat: str, schema: str):
236
- """apply the package's own bootstrap chain; selected by suffix, not revision."""
259
+ def render_and_apply(spark: SparkSession, output_folder: str, template, cat: str, schema: str) -> None:
260
+ """render one migration and run it, unless this engine was gated out of it."""
261
+ result_sql = apply_template(output_folder, template, cat=cat, schema=schema)
262
+ if has_no_statements(result_sql):
263
+ logger.info(f"{template.name} rendered no statements; nothing to apply here;")
264
+ return
265
+ spark.sql(result_sql)
266
+
267
+
268
+ def migrate_initial(spark: SparkSession, output_folder: str, cat: str, schema: str) -> None:
269
+ """apply the package's own bootstrap chain, in filename order rather than by revision."""
237
270
  env = Environment(
238
271
  loader=PackageLoader(package_name=PACKAGE_NAME, package_path=INITIAL_MIGRATIONS_PACKAGE_PATH),
239
272
  autoescape=select_autoescape(),
240
273
  )
241
- all_templates = env.list_templates(filter_func=use_migration_file)
274
+ all_templates = env.list_templates(extensions=["sql"])
242
275
 
243
276
  logger.info(f"found {len(all_templates)} migrations; first five are {all_templates[:5]};")
244
277
  for template_name in all_templates:
245
- result_sql = apply_template(output_folder, env.get_template(template_name), cat=cat, schema=schema)
246
- spark.sql(result_sql)
247
- logger.info(f"invoked spark on {len(all_templates)} migrations;")
248
-
278
+ render_and_apply(spark, output_folder, env.get_template(template_name), cat=cat, schema=schema)
279
+ logger.info(f"rendered {len(all_templates)} initial migrations;")
249
280
 
250
- def record_revision(spark, cat: str, schema: str, migration_type: str, revision_id: str):
251
- """point VERSION_TABLE's row for one migration type at revision_id.
252
281
 
253
- delete + insert rather than merge: there is one row per type, and a replay
254
- after a crash between the two statements is harmless because migrations are
255
- idempotent. the values are inlined rather than passed as `args` because
256
- delta's DELETE does not bind named parameters.
257
- """
282
+ def record_revision(spark: SparkSession, cat: str, schema: str, revision_id: str) -> None:
258
283
  version_table = f"{cat}.{schema}.{VERSION_TABLE}"
259
- type_literal = migration_type.replace("'", "''")
260
284
  revision_literal = revision_id.replace("'", "''")
261
- spark.sql(f"delete from {version_table} where migration_type = '{type_literal}'")
262
- spark.sql(
263
- f"insert into {version_table} (migration_type, version_num) " f"values ('{type_literal}', '{revision_literal}')"
264
- )
285
+ spark.sql(f"delete from {version_table}")
286
+ spark.sql(f"insert into {version_table} (version_num) values ('{revision_literal}')")
265
287
 
266
288
 
267
- def migrate_w_rev(spark, output_folder, migrations_root, cat: str, schema: str, migration_type: str):
289
+ def migrate_w_rev(spark: SparkSession, output_folder: str, migrations_dir: str, cat: str, schema: str) -> None:
268
290
  """
269
- apply the migrations of one client chain that VERSION_TABLE hasn't recorded yet.
270
291
  TODO XXX permit applying up to a revision below the head revision
271
292
  TODO XXX add backwards migrations, permit migrating backwards
272
293
  """
273
- migrations_dir = get_migrations_dir(migrations_root, migration_type)
274
- unapplied = get_unapplied_migrations_list(spark, migrations_dir, migration_type, cat=cat, schema=schema)
294
+ if not os.path.isdir(migrations_dir):
295
+ logger.info(f"no chain at {migrations_dir}; skipping it;")
296
+ return
297
+ unapplied = get_unapplied_migrations_list(spark, migrations_dir, cat=cat, schema=schema)
275
298
 
276
299
  logger.info(
277
- f"{migration_type} has {len(unapplied)} unapplied migrations; "
300
+ f"{len(unapplied)} unapplied migrations; "
278
301
  f"first five are {[migration.template_name for migration in unapplied[:5]]};"
279
302
  )
280
303
 
281
304
  env = Environment(loader=FileSystemLoader(migrations_dir), autoescape=select_autoescape())
282
305
  for migration in unapplied:
283
- result_sql = apply_template(output_folder, env.get_template(migration.template_name), cat=cat, schema=schema)
284
- spark.sql(result_sql)
285
- record_revision(spark, cat, schema, migration_type, migration.revision_id)
286
- logger.info(f"applied {migration.template_name}; {migration_type} is now at {migration.revision_id};")
306
+ render_and_apply(spark, output_folder, env.get_template(migration.template_name), cat=cat, schema=schema)
307
+ record_revision(spark, cat, schema, migration.revision_id)
308
+ logger.info(f"{migration.template_name} done; now at {migration.revision_id};")
287
309
 
288
310
 
289
- def run_migrations(spark, cat, schema, output_folder, migrations_root):
311
+ def run_migrations(spark: SparkSession, cat: str, schema: str, output_folder: str, migrations_dir: str) -> None:
290
312
  initial_output_folder = os.path.join(output_folder, INITIAL_MIGRATIONS_PACKAGE_PATH)
291
- dbr_ouput_folder = os.path.join(output_folder, f"{DBR_ONLY}_migrations")
292
- all_spark_ouput_folder = os.path.join(output_folder, f"{ALL_SPARK}_migrations")
313
+ client_output_folder = os.path.join(output_folder, CLIENT_OUTPUT_DIRNAME)
293
314
  os.makedirs(initial_output_folder, exist_ok=True)
294
- os.makedirs(dbr_ouput_folder, exist_ok=True)
295
- os.makedirs(all_spark_ouput_folder, exist_ok=True)
315
+ os.makedirs(client_output_folder, exist_ok=True)
296
316
  version_table = f"{cat}.{schema}.{VERSION_TABLE}"
297
317
  if spark.catalog.tableExists(version_table):
298
318
  logger.info(f"{version_table} already exists; skipping the initial migrations;")
299
319
  else:
300
320
  migrate_initial(spark, initial_output_folder, cat, schema)
301
- if is_dbr():
302
- migrate_w_rev(spark, dbr_ouput_folder, migrations_root, cat, schema, DBR_ONLY)
303
- migrate_w_rev(spark, all_spark_ouput_folder, migrations_root, cat, schema, ALL_SPARK)
321
+ migrate_w_rev(spark, client_output_folder, migrations_dir, cat, schema)
304
322
 
305
323
 
306
- def main(cat, schema, migrations_dir, output_parent_path=None):
324
+ def main(cat: str, schema: str, migrations_dir: str, output_parent_path: str | None = None) -> None:
307
325
  output_folder = get_output_folder(output_parent_path or os.path.join(os.getcwd(), "migrations_out"))
308
326
  os.makedirs(output_folder)
309
327
  run_migrations(get_spark(), cat, schema, output_folder, migrations_dir)
310
328
 
311
329
 
312
- def _cli_main(cat, schema, migrations_dir): # pragma: no cover
330
+ def _cli_main(cat: str, schema: str, migrations_dir: str) -> None: # pragma: no cover
313
331
  main(cat, schema, migrations_dir)
314
332
 
315
333
 
316
- def _cli_create_new_migration(message, output_path, template_path): # pragma: no cover
317
- create_new_migration(message, template_path or get_default_template_path(), output_path)
334
+ def _cli_create_new_migration(
335
+ message: str, output_path: str, template_path: str, add_is_dbr: bool
336
+ ) -> None: # pragma: no cover
337
+ create_new_migration(message, template_path or get_default_template_path(), output_path, add_is_dbr=add_is_dbr)
318
338
 
319
339
 
320
- def build_parser(): # pragma: no cover
340
+ def build_parser() -> argparse.ArgumentParser: # pragma: no cover
321
341
  parser = argparse.ArgumentParser(description="spark_sql_migrations migration utilities")
322
342
  subparsers = parser.add_subparsers(dest="command", required=True)
323
343
 
324
344
  p_run = subparsers.add_parser("run", help="apply all pending migrations")
325
- p_run.add_argument("--cat", default="spark_catalog")
326
- p_run.add_argument("--schema", default="default")
327
- p_run.add_argument(
328
- "--migrations-dir", required=True, help="parent of all_spark_migrations/ and dbr_only_migrations/"
329
- )
345
+ p_run.add_argument("--cat", type=str, default="spark_catalog")
346
+ p_run.add_argument("--schema", type=str, default="default")
347
+ p_run.add_argument("--migrations-dir", type=str, required=True, help="the directory holding the migration chain")
330
348
  p_run.set_defaults(func=_cli_main)
331
349
 
332
350
  p_create = subparsers.add_parser("create_new_migration", help="create a new migration from the template")
333
- p_create.add_argument("--message", required=True)
351
+ p_create.add_argument("--message", type=str, required=True)
334
352
  p_create.add_argument(
335
353
  "--output-path",
354
+ type=str,
336
355
  required=True,
337
356
  help="the chain directory to write the new migration into. if sql files are in there, the last sql's rev_id"
338
357
  "will be set as the new migrations prev_rev_id.",
339
358
  )
340
- p_create.add_argument("--template-path", default=None, help="defaults to the template shipped in this package")
359
+ p_create.add_argument(
360
+ "--template-path", type=str, default=None, help="defaults to the template shipped in this package"
361
+ )
362
+ # the %% are argparse's: it interpolates help strings
363
+ p_create.add_argument(
364
+ "--add-is-dbr",
365
+ action="store_true",
366
+ help="wrap the new migration's body in {%% if is_dbr %%}, for statements only databricks understands",
367
+ )
341
368
  p_create.set_defaults(func=_cli_create_new_migration)
342
369
 
343
370
  return parser
344
371
 
345
372
 
346
- def cli(argv=None): # pragma: no cover
373
+ def cli(argv: list[str] | None = None) -> None: # pragma: no cover
347
374
  """the `spark-sql-migrations` console command.
348
375
 
349
376
  an application entry point, so unlike the rest of this package it configures
@@ -11,7 +11,7 @@ METASTORE_DIR_ENV_VAR = "SPARK_METASTORE_DIR"
11
11
  DEFAULT_WAREHOUSE_DIRNAME = "spark-warehouse"
12
12
 
13
13
 
14
- def is_dbr():
14
+ def is_dbr() -> bool:
15
15
  try:
16
16
  from pyspark.dbutils import DBUtils # type: ignore # noqa: F401
17
17
 
@@ -21,7 +21,7 @@ def is_dbr():
21
21
  return False
22
22
 
23
23
 
24
- def get_warehouse_dir():
24
+ def get_warehouse_dir() -> str:
25
25
  """where local (non-DBR) spark keeps its warehouse.
26
26
 
27
27
  resolved against the caller's cwd, never against __file__: installed from a
@@ -34,7 +34,7 @@ def get_warehouse_dir():
34
34
  )
35
35
 
36
36
 
37
- def get_spark(use_dbc=False):
37
+ def get_spark(use_dbc: bool = False) -> SparkSession:
38
38
  if use_dbc or is_dbr():
39
39
  from databricks.connect.session import DatabricksSession # type: ignore # pants: no-infer-dep
40
40
 
@@ -1,6 +1,6 @@
1
1
  Metadata-Version: 2.4
2
2
  Name: spark_sql_migrations
3
- Version: 0.0.5
3
+ Version: 0.0.7
4
4
  Summary: Chained, idempotent SQL migrations for Spark and Databricks.
5
5
  Home-page: https://github.com/mcjug2015/spark_sql_migrations
6
6
  Author: Victor Semenov
@@ -79,28 +79,32 @@ Migrations come from two different owners, and they behave differently:
79
79
  | | ships in the wheel | owned by you |
80
80
  |---|---|---|
81
81
  | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
82
- | **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
83
- | **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
82
+ | **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
83
+ | **ordered by** | filename | the `prev_revision_id` chain |
84
84
  | **when applied** | until the version table exists | once, then recorded |
85
85
 
86
86
  The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
87
87
  skipped from then on. A run that fails partway through bootstrapping replays it, so it must
88
- still be idempotent. Your own chains are applied once each and their head revision is
88
+ still be idempotent. Your own migrations are applied once each and the head revision is
89
89
  written to the version table.
90
90
 
91
- ## Laying out your chains
91
+ Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
92
+ bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
92
93
 
93
- Point spark_sql_migrations at a directory holding one or both chains:
94
+ ## Laying out your chain
95
+
96
+ `--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
94
97
 
95
98
  ```
96
99
  your_project/
97
- └── migrations/ <- this is --migrations-dir
98
- ├── all_spark_migrations/ <- runs everywhere
99
- └── dbr_only_migrations/ <- runs only when is_dbr()
100
+ └── migrations/ <- this is --migrations-dir
101
+ ├── 260831_01_create_test_table_e5ce0039b32b.sql
102
+ └── 260831_02_create_metrics_table_07990e2a101e.sql
100
103
  ```
101
104
 
102
- Choose deliberately. SQL that only Databricks understands must not live in
103
- `all_spark_migrations/`, or your local runs will break.
105
+ One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
106
+ migration that otherwise runs everywhere — see below. SQL that only Databricks understands
107
+ must never sit ungated there, or your local runs will break.
104
108
 
105
109
  ## Running migrations
106
110
 
@@ -124,7 +128,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
124
128
 
125
129
 
126
130
  def get_migrations_dir():
127
- """the parent of this project's own migration chains."""
131
+ """the directory holding this project's own migration chain."""
128
132
  return os.path.dirname(__file__)
129
133
 
130
134
 
@@ -134,7 +138,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
134
138
  `main` builds its own session via `get_spark()` and, being library code, leaves logging to
135
139
  you. To supply your own — a session already
136
140
  configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
137
- migrations_root)` directly.
141
+ migrations_dir)` directly.
138
142
 
139
143
  Every rendered statement is written to a timestamped folder under `migrations_out/`
140
144
  before it is executed, so you can always read the exact SQL a run applied.
@@ -144,7 +148,7 @@ before it is executed, so you can always read the exact SQL a run applied.
144
148
  Generate one rather than hand-rolling the header:
145
149
 
146
150
  ```bash
147
- spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations/all_spark_migrations
151
+ spark-sql-migrations create_new_migration --message "add batch id to metrics" --output-path path/to/migrations
148
152
  ```
149
153
 
150
154
  That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
@@ -183,6 +187,44 @@ begin
183
187
  end;
184
188
  ```
185
189
 
190
+ ### Databricks-only statements
191
+
192
+ `is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
193
+ Databricks understands:
194
+
195
+ ```sql
196
+ -- revision_id:e8614b76da6d;
197
+ -- prev_revision_id:57037b6c19b4;
198
+ {% if is_dbr %}
199
+ begin
200
+ grant READ FILES on external location `some-bucket` to `some_group`;
201
+ end;
202
+ {% endif %}
203
+ ```
204
+
205
+ `create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
206
+ headers so they stay readable:
207
+
208
+ ```bash
209
+ spark-sql-migrations create_new_migration --message "read files on the cms zip external location" --output-path path/to/migrations --add-is-dbr
210
+ ```
211
+
212
+ This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
213
+ its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
214
+ script never gets the chance to run — and the whole script fails to compile, not just that
215
+ statement. Jinja removes the text before `spark.sql()` ever sees it.
216
+
217
+ A migration left holding nothing but comments is recorded in the version table and not
218
+ executed, so the chain advances in step on both engines. That recording is forward-only: a
219
+ catalog migrated locally past a gated revision will not pick that statement up if it is
220
+ later migrated from Databricks. For disposable test schemas that is exactly what you want;
221
+ for a long-lived catalog, don't migrate it from both.
222
+
223
+ `_spark_migrations_version` holds one row and one column, `version_num`. There is no
224
+ in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
225
+ a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
226
+ scratch, which also clears the revision history that layout recorded.
227
+
186
228
  ## Local vs Databricks
187
229
 
188
230
  `get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
@@ -13,8 +13,8 @@ spark_sql_migrations.egg-info/namespace_packages.txt
13
13
  spark_sql_migrations.egg-info/requires.txt
14
14
  spark_sql_migrations.egg-info/top_level.txt
15
15
  spark_sql_migrations/migration_templates/default_migration_template.sql
16
- spark_sql_migrations/migrations_initial/first_01_intial_create_cat_dbr_only.sql
17
- spark_sql_migrations/migrations_initial/first_02_intial_create_schema_all.sql
18
- spark_sql_migrations/migrations_initial/first_03_intial_create_state_table_all.sql
16
+ spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql
17
+ spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql
18
+ spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql
19
19
  spark_sql_migrations/spark_sql/__init__.py
20
20
  spark_sql_migrations/spark_sql/spark_sql.py
@@ -1,4 +0,0 @@
1
- -- revision_id:;
2
- begin
3
- create catalog if not exists {{cat}};
4
- end;