spark-sql-migrations 0.0.4__tar.gz → 0.0.6__tar.gz

{spark_sql_migrations-0.0.4 → spark_sql_migrations-0.0.6}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark_sql_migrations
-Version: 0.0.4
+Version: 0.0.6
 Summary: Chained, idempotent SQL migrations for Spark and Databricks.
 Home-page: https://github.com/mcjug2015/spark_sql_migrations
 Author: Victor Semenov
@@ -79,28 +79,32 @@ Migrations come from two different owners, and they behave differently:
 | | ships in the wheel | owned by you |
 |---|---|---|
 | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
-| **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
-| **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
+| **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
+| **ordered by** | filename | the `prev_revision_id` chain |
 | **when applied** | until the version table exists | once, then recorded |
 
 The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
 skipped from then on. A run that fails partway through bootstrapping replays it, so it must
-still be idempotent. Your own chains are applied once each and their head revision is
+still be idempotent. Your own migrations are applied once each and the head revision is
 written to the version table.
 
-## Laying out your chains
+Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
+bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
 
-Point spark_sql_migrations at a directory holding one or both chains:
+## Laying out your chain
+
+`--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
 
 ```
 your_project/
-└── migrations/                 <- this is --migrations-dir
-    ├── all_spark_migrations/   <- runs everywhere
-    └── dbr_only_migrations/    <- runs only when is_dbr()
+└── migrations/        <- this is --migrations-dir
+    ├── 260831_01_create_test_table_e5ce0039b32b.sql
+    └── 260831_02_create_metrics_table_07990e2a101e.sql
 ```
 
-Choose deliberately. SQL that only Databricks understands must not live in
-`all_spark_migrations/`, or your local runs will break.
+One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
+migration that otherwise runs everywhere — see below. SQL that only Databricks understands
+must never sit ungated there, or your local runs will break.
 
 ## Running migrations
 
@@ -124,7 +128,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
 
 
 def get_migrations_dir():
-    """the parent of this project's own migration chains."""
+    """the directory holding this project's own migration chain."""
     return os.path.dirname(__file__)
 
 
@@ -134,7 +138,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
 `main` builds its own session via `get_spark()` and, being library code, leaves logging to
 you. To supply your own — a session already
 configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
-migrations_root)` directly.
+migrations_dir)` directly.
 
 Every rendered statement is written to a timestamped folder under `migrations_out/`
 before it is executed, so you can always read the exact SQL a run applied.
@@ -144,7 +148,7 @@ before it is executed, so you can always read the exact SQL a run applied.
 Generate one rather than hand-rolling the header:
 
 ```bash
-spark-sql-migrations create_new_migration     --message "add batch id to metrics"     --output-path path/to/migrations/all_spark_migrations
+spark-sql-migrations create_new_migration     --message "add batch id to metrics"     --output-path path/to/migrations
 ```
 
 That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
@@ -183,6 +187,44 @@ begin
 end;
 ```
 
+### Databricks-only statements
+
+`is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
+Databricks understands:
+
+```sql
+-- revision_id:e8614b76da6d;
+-- prev_revision_id:57037b6c19b4;
+{% if is_dbr %}
+begin
+grant READ FILES on external location `some-bucket` to `some_group`;
+end;
+{% endif %}
+```
+
+`create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
+headers so they stay readable:
+
+```bash
+spark-sql-migrations create_new_migration     --message "read files on the cms zip external location"     --output-path path/to/migrations     --add-is-dbr
+```
+
+This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
+its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
+script never gets the chance to run — and the whole script fails to compile, not just that
+statement. Jinja removes the text before `spark.sql()` ever sees it.
+
+A migration left holding nothing but comments is recorded in the version table and not
+executed, so the chain advances in step on both engines. That recording is forward-only: a
+catalog migrated locally past a gated revision will not pick that statement up if it is
+later migrated from Databricks. For disposable test schemas that is exactly what you want;
+for a long-lived catalog, don't migrate it from both.
+
+`_spark_migrations_version` holds one row and one column, `version_num`. There is no
+in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
+a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
+scratch, which also clears the revision history that layout recorded.
+
 ## Local vs Databricks
 
 `get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured

{spark_sql_migrations-0.0.4 → spark_sql_migrations-0.0.6}/setup.py

@@ -80,28 +80,32 @@ Migrations come from two different owners, and they behave differently:
 | | ships in the wheel | owned by you |
 |---|---|---|
 | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
-| **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
-| **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
+| **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
+| **ordered by** | filename | the `prev_revision_id` chain |
 | **when applied** | until the version table exists | once, then recorded |
 
 The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
 skipped from then on. A run that fails partway through bootstrapping replays it, so it must
-still be idempotent. Your own chains are applied once each and their head revision is
+still be idempotent. Your own migrations are applied once each and the head revision is
 written to the version table.
 
-## Laying out your chains
+Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
+bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
 
-Point spark_sql_migrations at a directory holding one or both chains:
+## Laying out your chain
+
+`--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
 
 ```
 your_project/
-└── migrations/                 <- this is --migrations-dir
-    ├── all_spark_migrations/   <- runs everywhere
-    └── dbr_only_migrations/    <- runs only when is_dbr()
+└── migrations/        <- this is --migrations-dir
+    ├── 260831_01_create_test_table_e5ce0039b32b.sql
+    └── 260831_02_create_metrics_table_07990e2a101e.sql
 ```
 
-Choose deliberately. SQL that only Databricks understands must not live in
-`all_spark_migrations/`, or your local runs will break.
+One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
+migration that otherwise runs everywhere — see below. SQL that only Databricks understands
+must never sit ungated there, or your local runs will break.
 
 ## Running migrations
 
@@ -128,7 +132,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
 
 
 def get_migrations_dir():
-    \"\"\"the parent of this project's own migration chains.\"\"\"
+    \"\"\"the directory holding this project's own migration chain.\"\"\"
     return os.path.dirname(__file__)
 
 
@@ -138,7 +142,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
 `main` builds its own session via `get_spark()` and, being library code, leaves logging to
 you. To supply your own — a session already
 configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
-migrations_root)` directly.
+migrations_dir)` directly.
 
 Every rendered statement is written to a timestamped folder under `migrations_out/`
 before it is executed, so you can always read the exact SQL a run applied.
@@ -150,7 +154,7 @@ Generate one rather than hand-rolling the header:
 ```bash
 spark-sql-migrations create_new_migration \
     --message "add batch id to metrics" \
-    --output-path path/to/migrations/all_spark_migrations
+    --output-path path/to/migrations
 ```
 
 That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
@@ -189,6 +193,47 @@ begin
 end;
 ```
 
+### Databricks-only statements
+
+`is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
+Databricks understands:
+
+```sql
+-- revision_id:e8614b76da6d;
+-- prev_revision_id:57037b6c19b4;
+{% if is_dbr %}
+begin
+grant READ FILES on external location `some-bucket` to `some_group`;
+end;
+{% endif %}
+```
+
+`create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
+headers so they stay readable:
+
+```bash
+spark-sql-migrations create_new_migration \
+    --message "read files on the cms zip external location" \
+    --output-path path/to/migrations \
+    --add-is-dbr
+```
+
+This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
+its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
+script never gets the chance to run — and the whole script fails to compile, not just that
+statement. Jinja removes the text before `spark.sql()` ever sees it.
+
+A migration left holding nothing but comments is recorded in the version table and not
+executed, so the chain advances in step on both engines. That recording is forward-only: a
+catalog migrated locally past a gated revision will not pick that statement up if it is
+later migrated from Databricks. For disposable test schemas that is exactly what you want;
+for a long-lived catalog, don't migrate it from both.
+
+`_spark_migrations_version` holds one row and one column, `version_num`. There is no
+in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
+a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
+scratch, which also clears the revision history that layout recorded.
+
 ## Local vs Databricks
 
 `get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured
@@ -238,9 +283,9 @@ Apache License 2.0 — see [LICENSE](LICENSE).
     'package_data': {
         'spark_sql_migrations': (
             'migration_templates/default_migration_template.sql',
-            'migrations_initial/first_01_intial_create_cat_dbr_only.sql',
-            'migrations_initial/first_02_intial_create_schema_all.sql',
-            'migrations_initial/first_03_intial_create_state_table_all.sql',
+            'migrations_initial/first_01_intial_create_cat.sql',
+            'migrations_initial/first_02_intial_create_schema.sql',
+            'migrations_initial/first_03_intial_create_state_table.sql',
             'py.typed',
         ),
     },
@@ -254,5 +299,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
     },
     'python_requires': '>=3.10',
     'url': 'https://github.com/mcjug2015/spark_sql_migrations',
-    'version': '0.0.4',
+    'version': '0.0.6',
 })

{spark_sql_migrations-0.0.4 → spark_sql_migrations-0.0.6}/spark_sql_migrations/migration_templates/default_migration_template.sql

@@ -5,6 +5,9 @@ begin
 -- would like to have applied to databricks db or local spark(for testing)
 -- crutch scaffolding will use jinja to fill in the cat and schema that are sent to it
 -- this is also taken care of in the gh build.
+-- is_dbr is filled in too: statements only databricks understands go in a jinja
+-- conditional on it, outside the begin/end, rather than in a second chain. pass
+-- --add-is-dbr to create_new_migration to have that conditional written for you.
 
 -- e.g:
 -- create schema if not exists {{cat}}.{{schema}};

spark_sql_migrations-0.0.6/spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql

@@ -0,0 +1,9 @@
+-- revision_id:;
+-- a catalog is a unity catalog object: local spark has spark_catalog and no way
+-- to make another, so this one is gated to databricks and renders to nothing
+-- else, leaving the bootstrap chain the same three files everywhere.
+{% if is_dbr %}
+begin
+create catalog if not exists {{cat}};
+end;
+{% endif %}

spark_sql_migrations-0.0.4/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table_all.sql → spark_sql_migrations-0.0.6/spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql

@@ -1,7 +1,6 @@
 -- revision_id:;
 begin
 create table if not exists {{cat}}.{{schema}}._spark_migrations_version (
-    migration_type string not null,
     version_num string not null
 );
-end;
+end;

{spark_sql_migrations-0.0.4 → spark_sql_migrations-0.0.6}/spark_sql_migrations/spark_sql/spark_sql.py

@@ -3,15 +3,21 @@
 Two kinds of migration exist, and they are owned by different parties:
 
 * the *initial* chain and the new-migration template ship inside this package.
-  They bootstrap the catalog, the schema and the version table, are selected by
-  filename suffix (``_all.sql`` / ``_dbr_only.sql``) rather than by revision,
-  and are applied on every run until the version table exists -- so a run that
-  failed partway through replays them, and they must be idempotent.
-* the ``all_spark_migrations/`` and ``dbr_only_migrations/`` chains belong to
-  the consuming project. The caller passes their parent directory as
-  ``migrations_root``; each chain is walked from its root via
-  ``prev_revision_id`` and only the tail not yet recorded in the version table
-  is applied.
+  They bootstrap the catalog, the schema and the version table, carry no
+  revision of their own, and are applied in filename order on every run until
+  the version table exists -- so a run that failed partway through replays
+  them, and they must be idempotent.
+* the client chain belongs to the consuming project. The caller passes the
+  directory holding it as ``migrations_dir``; the chain is walked from its root
+  via ``prev_revision_id`` and only the tail not yet recorded in the version
+  table is applied.
+
+Which engine a migration targets is decided inside the migration, not by where it
+sits: every one of them is rendered through jinja before it is executed, with
+``is_dbr`` in the render context, so a chain that runs everywhere can still gate
+the statements one engine would reject. A migration left holding no statements is
+recorded and not executed, which is what lets a project keep one chain and no
+second head.
 """
 
 import argparse
@@ -23,24 +29,23 @@ import uuid
 from dataclasses import dataclass
 
 from jinja2 import Environment, FileSystemLoader, PackageLoader, select_autoescape
-from pyspark.sql.functions import col
 
 from spark_sql_migrations.spark_utils import get_spark, is_dbr
 
 logger = logging.getLogger(__name__)
 
 
-ALL_SPARK = "all_spark"
-DBR_ONLY = "dbr_only"
-
 VERSION_TABLE = "_spark_migrations_version"
 
 PACKAGE_NAME = "spark_sql_migrations"
 INITIAL_MIGRATIONS_PACKAGE_PATH = "migrations_initial"
+# where a run's rendered client migrations land, under its output folder
+CLIENT_OUTPUT_DIRNAME = "migrations"
 TEMPLATES_PACKAGE_PATH = "migration_templates"
 DEFAULT_TEMPLATE_NAME = "default_migration_template.sql"
 
 _HEADER_RE = re.compile(r"^--\s*(revision_id|prev_revision_id)\s*:\s*(\w*)\s*;$")
+_COMMENT_OR_BLANK_RE = re.compile(r"^\s*(--.*)?$")
 
 
 @dataclass(frozen=True)
@@ -50,11 +55,6 @@ class Migration:
     template_name: str
 
 
-def get_migrations_dir(migrations_root, migration_type):
-    """the client-owned directory holding one chain."""
-    return os.path.join(migrations_root, f"{migration_type}_migrations")
-
-
 def get_default_template_path():
     """the template this package ships; callers may pass their own instead."""
     return os.path.join(os.path.dirname(__file__), "..", TEMPLATES_PACKAGE_PATH, DEFAULT_TEMPLATE_NAME)
@@ -150,39 +150,56 @@ def get_migrations_list(migrations_dir):
     return get_ordered_migration_objs(migrations)
 
 
-def get_unapplied_migrations_list(spark, migrations_dir, migration_type, cat: str, schema: str):
-    """the tail of one chain after the revision VERSION_TABLE records for it.
-
-    the whole chain when the table is missing or holds no row for this type.
-    """
-    ordered = get_migrations_list(migrations_dir)
-
+def get_table_version(spark, cat: str, schema: str):
+    """the revision VERSION_TABLE records, or None if it holds no row yet."""
     version_table = f"{cat}.{schema}.{VERSION_TABLE}"
     if not spark.catalog.tableExists(version_table):
-        return ordered
+        return None
+    return spark.read.table(version_table).select("version_num").head()
 
-    current = (
-        spark.read.table(version_table).where(col("migration_type") == migration_type).select("version_num").head()
-    )
+
+def get_unapplied_migrations_list(spark, migrations_dir, cat: str, schema: str):
+    ordered = get_migrations_list(migrations_dir)
+    current = get_table_version(spark, cat, schema)
     if not current:
         return ordered
 
     current_revision_id = current["version_num"]
 
+    if not ordered:
+        logger.warning(
+            f"{VERSION_TABLE} is at {current_revision_id}, but {migrations_dir} holds no migrations; "
+            f"treating the chain as retired;"
+        )
+        return ordered
+
     for index, migration in enumerate(ordered):
         if migration.revision_id == current_revision_id:
             return ordered[index + 1 :]
 
-    raise ValueError(
-        f"{VERSION_TABLE} has {migration_type} at {current_revision_id}, "
-        f"which matches no migration in {migrations_dir};"
-    )
+    raise ValueError(f"{VERSION_TABLE} is at {current_revision_id}, which matches no migration in {migrations_dir};")
 
 
-def create_new_migration(message: str, template_path: str, output_path: str, truncate_slug_length: int = 40):
-    """
-    TODO XXX start looking up prev revision id from db or files and including it
+def gate_on_is_dbr(migration_content):
+    """wrap a migration's body in the jinja conditional, below its headers.
+
+    the revision headers stay outside it: they are read off the file as comments,
+    and the first line that is neither comment nor blank ends that read -- so an
+    opening tag above them would hide them. a body-less template gets an empty
+    gate at the end, which renders to nothing either way.
     """
+    lines = migration_content.splitlines()
+    body_start = len(lines)
+    for index, line in enumerate(lines):
+        if line.strip() and not line.startswith("--"):
+            body_start = index
+            break
+    return "\n".join(lines[:body_start] + ["{% if is_dbr %}"] + lines[body_start:] + ["{% endif %}"]) + "\n"
+
+
+def create_new_migration(
+    message: str, template_path: str, output_path: str, truncate_slug_length: int = 40, add_is_dbr: bool = False
+):
     rev_id = uuid.uuid4().hex[-12:]
 
     date_prefix = datetime.datetime.today().strftime("%y%m%d")
@@ -195,15 +212,24 @@ def create_new_migration(message: str, template_path: str, output_path: str, tru
 
     migration_path = os.path.join(output_path, migration_filename)
 
+    try:
+        ordered_migrations = get_migrations_list(output_path)
+        last_migration_rev_id = ordered_migrations[-1].revision_id if len(ordered_migrations) > 0 else ""
+    except ValueError:
+        last_migration_rev_id = ""
+
     default_migration_content = open(template_path).read()
     default_migration_content = default_migration_content.replace("{{revision_id}}", rev_id)
+    default_migration_content = default_migration_content.replace("{{prev_revision_id}}", last_migration_rev_id)
+    if add_is_dbr:
+        default_migration_content = gate_on_is_dbr(default_migration_content)
 
     with open(migration_path, "w") as migration_file:
         migration_file.write(default_migration_content)
 
 
 def apply_template(output_dir, template, cat: str, schema: str):
-    result_sql = template.render(cat=cat, schema=schema)
+    result_sql = template.render(cat=cat, schema=schema, is_dbr=is_dbr())
     with open(os.path.join(output_dir, template.name.replace(".sql", "_primed.sql")), "w") as file_handle:
         file_handle.write(result_sql)
     return result_sql
@@ -220,83 +246,76 @@ def get_output_folder(output_parent_path):
     return os.path.join(output_parent_path, folder_name)
 
 
-def use_migration_file(fname):
-    if fname.endswith("all.sql"):
-        return True
-    elif is_dbr() and fname.endswith("dbr_only.sql"):
-        return True
-    return False
+def has_no_statements(rendered_sql):
+    for line in rendered_sql.splitlines():
+        if not _COMMENT_OR_BLANK_RE.match(line):
+            return False
+    return True
+
+
+def render_and_apply(spark, output_folder, template, cat: str, schema: str):
+    """render one migration and run it, unless this engine was gated out of it."""
+    result_sql = apply_template(output_folder, template, cat=cat, schema=schema)
+    if has_no_statements(result_sql):
+        logger.info(f"{template.name} rendered no statements; nothing to apply here;")
+        return
+    spark.sql(result_sql)
 
 
 def migrate_initial(spark, output_folder, cat: str, schema: str):
-    """apply the package's own bootstrap chain; selected by suffix, not revision."""
+    """apply the package's own bootstrap chain, in filename order rather than by revision."""
     env = Environment(
         loader=PackageLoader(package_name=PACKAGE_NAME, package_path=INITIAL_MIGRATIONS_PACKAGE_PATH),
         autoescape=select_autoescape(),
     )
-    all_templates = env.list_templates(filter_func=use_migration_file)
+    all_templates = env.list_templates(extensions=["sql"])
 
     logger.info(f"found {len(all_templates)} migrations; first five are {all_templates[:5]};")
     for template_name in all_templates:
-        result_sql = apply_template(output_folder, env.get_template(template_name), cat=cat, schema=schema)
-        spark.sql(result_sql)
-    logger.info(f"invoked spark on {len(all_templates)} migrations;")
-
+        render_and_apply(spark, output_folder, env.get_template(template_name), cat=cat, schema=schema)
+    logger.info(f"rendered {len(all_templates)} initial migrations;")
 
-def record_revision(spark, cat: str, schema: str, migration_type: str, revision_id: str):
-    """point VERSION_TABLE's row for one migration type at revision_id.
 
-    delete + insert rather than merge: there is one row per type, and a replay
-    after a crash between the two statements is harmless because migrations are
-    idempotent. the values are inlined rather than passed as `args` because
-    delta's DELETE does not bind named parameters.
-    """
+def record_revision(spark, cat: str, schema: str, revision_id: str):
     version_table = f"{cat}.{schema}.{VERSION_TABLE}"
-    type_literal = migration_type.replace("'", "''")
     revision_literal = revision_id.replace("'", "''")
-    spark.sql(f"delete from {version_table} where migration_type = '{type_literal}'")
-    spark.sql(
-        f"insert into {version_table} (migration_type, version_num) " f"values ('{type_literal}', '{revision_literal}')"
-    )
+    spark.sql(f"delete from {version_table}")
+    spark.sql(f"insert into {version_table} (version_num) values ('{revision_literal}')")
 
 
-def migrate_w_rev(spark, output_folder, migrations_root, cat: str, schema: str, migration_type: str):
+def migrate_w_rev(spark, output_folder, migrations_dir, cat: str, schema: str):
     """
-    apply the migrations of one client chain that VERSION_TABLE hasn't recorded yet.
     TODO XXX permit applying up to a revision below the head revision
     TODO XXX add backwards migrations, permit migrating backwards
     """
-    migrations_dir = get_migrations_dir(migrations_root, migration_type)
-    unapplied = get_unapplied_migrations_list(spark, migrations_dir, migration_type, cat=cat, schema=schema)
+    if not os.path.isdir(migrations_dir):
+        logger.info(f"no chain at {migrations_dir}; skipping it;")
+        return
+    unapplied = get_unapplied_migrations_list(spark, migrations_dir, cat=cat, schema=schema)
 
     logger.info(
-        f"{migration_type} has {len(unapplied)} unapplied migrations; "
+        f"{len(unapplied)} unapplied migrations; "
         f"first five are {[migration.template_name for migration in unapplied[:5]]};"
     )
 
     env = Environment(loader=FileSystemLoader(migrations_dir), autoescape=select_autoescape())
     for migration in unapplied:
-        result_sql = apply_template(output_folder, env.get_template(migration.template_name), cat=cat, schema=schema)
-        spark.sql(result_sql)
-        record_revision(spark, cat, schema, migration_type, migration.revision_id)
-        logger.info(f"applied {migration.template_name}; {migration_type} is now at {migration.revision_id};")
+        render_and_apply(spark, output_folder, env.get_template(migration.template_name), cat=cat, schema=schema)
+        record_revision(spark, cat, schema, migration.revision_id)
+        logger.info(f"{migration.template_name} done; now at {migration.revision_id};")
 
 
-def run_migrations(spark, cat, schema, output_folder, migrations_root):
+def run_migrations(spark, cat, schema, output_folder, migrations_dir):
     initial_output_folder = os.path.join(output_folder, INITIAL_MIGRATIONS_PACKAGE_PATH)
-    dbr_ouput_folder = os.path.join(output_folder, f"{DBR_ONLY}_migrations")
-    all_spark_ouput_folder = os.path.join(output_folder, f"{ALL_SPARK}_migrations")
+    client_output_folder = os.path.join(output_folder, CLIENT_OUTPUT_DIRNAME)
     os.makedirs(initial_output_folder, exist_ok=True)
-    os.makedirs(dbr_ouput_folder, exist_ok=True)
-    os.makedirs(all_spark_ouput_folder, exist_ok=True)
+    os.makedirs(client_output_folder, exist_ok=True)
     version_table = f"{cat}.{schema}.{VERSION_TABLE}"
     if spark.catalog.tableExists(version_table):
         logger.info(f"{version_table} already exists; skipping the initial migrations;")
     else:
         migrate_initial(spark, initial_output_folder, cat, schema)
-    if is_dbr():
-        migrate_w_rev(spark, dbr_ouput_folder, migrations_root, cat, schema, DBR_ONLY)
-    migrate_w_rev(spark, all_spark_ouput_folder, migrations_root, cat, schema, ALL_SPARK)
+    migrate_w_rev(spark, client_output_folder, migrations_dir, cat, schema)
 
 
 def main(cat, schema, migrations_dir, output_parent_path=None):
@@ -309,8 +328,8 @@ def _cli_main(cat, schema, migrations_dir):  # pragma: no cover
     main(cat, schema, migrations_dir)
 
 
-def _cli_create_new_migration(message, output_path, template_path):  # pragma: no cover
-    create_new_migration(message, template_path or get_default_template_path(), output_path)
+def _cli_create_new_migration(message, output_path, template_path, add_is_dbr):  # pragma: no cover
+    create_new_migration(message, template_path or get_default_template_path(), output_path, add_is_dbr=add_is_dbr)
 
 
 def build_parser():  # pragma: no cover
@@ -320,15 +339,24 @@ def build_parser():  # pragma: no cover
     p_run = subparsers.add_parser("run", help="apply all pending migrations")
     p_run.add_argument("--cat", default="spark_catalog")
     p_run.add_argument("--schema", default="default")
-    p_run.add_argument(
-        "--migrations-dir", required=True, help="parent of all_spark_migrations/ and dbr_only_migrations/"
-    )
+    p_run.add_argument("--migrations-dir", required=True, help="the directory holding the migration chain")
     p_run.set_defaults(func=_cli_main)
 
     p_create = subparsers.add_parser("create_new_migration", help="create a new migration from the template")
     p_create.add_argument("--message", required=True)
-    p_create.add_argument("--output-path", required=True, help="the chain directory to write the new migration into")
+    p_create.add_argument(
+        "--output-path",
+        required=True,
+        help="the chain directory to write the new migration into. if sql files are in there, the last sql's rev_id"
+        "will be set as the new migrations prev_rev_id.",
+    )
     p_create.add_argument("--template-path", default=None, help="defaults to the template shipped in this package")
+    # the %% are argparse's: it interpolates help strings
+    p_create.add_argument(
+        "--add-is-dbr",
+        action="store_true",
+        help="wrap the new migration's body in {%% if is_dbr %%}, for statements only databricks understands",
+    )
     p_create.set_defaults(func=_cli_create_new_migration)
 
     return parser

{spark_sql_migrations-0.0.4 → spark_sql_migrations-0.0.6}/spark_sql_migrations.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark_sql_migrations
-Version: 0.0.4
+Version: 0.0.6
 Summary: Chained, idempotent SQL migrations for Spark and Databricks.
 Home-page: https://github.com/mcjug2015/spark_sql_migrations
 Author: Victor Semenov
@@ -79,28 +79,32 @@ Migrations come from two different owners, and they behave differently:
 | | ships in the wheel | owned by you |
 |---|---|---|
 | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
-| **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
-| **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
+| **where** | `spark_sql_migrations/migrations_initial/` | the directory you pass as `--migrations-dir` |
+| **ordered by** | filename | the `prev_revision_id` chain |
 | **when applied** | until the version table exists | once, then recorded |
 
 The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
 skipped from then on. A run that fails partway through bootstrapping replays it, so it must
-still be idempotent. Your own chains are applied once each and their head revision is
+still be idempotent. Your own migrations are applied once each and the head revision is
 written to the version table.
 
-## Laying out your chains
+Both kinds are rendered the same way, so both can gate a statement on `is_dbr` — the
+bootstrap chain uses it for `CREATE CATALOG`, which only Unity Catalog has.
 
-Point spark_sql_migrations at a directory holding one or both chains:
+## Laying out your chain
+
+`--migrations-dir` is the directory holding the chain — the `.sql` files sit in it directly:
 
 ```
 your_project/
-└── migrations/                 <- this is --migrations-dir
-    ├── all_spark_migrations/   <- runs everywhere
-    └── dbr_only_migrations/    <- runs only when is_dbr()
+└── migrations/        <- this is --migrations-dir
+    ├── 260831_01_create_test_table_e5ce0039b32b.sql
+    └── 260831_02_create_metrics_table_07990e2a101e.sql
 ```
 
-Choose deliberately. SQL that only Databricks understands must not live in
-`all_spark_migrations/`, or your local runs will break.
+One chain covers both engines: a Databricks-only *statement* gates on `is_dbr` inside a
+migration that otherwise runs everywhere — see below. SQL that only Databricks understands
+must never sit ungated there, or your local runs will break.
 
 ## Running migrations
 
@@ -124,7 +128,7 @@ from spark_sql_migrations.spark_sql.spark_sql import main
 
 
 def get_migrations_dir():
-    """the parent of this project's own migration chains."""
+    """the directory holding this project's own migration chain."""
     return os.path.dirname(__file__)
 
 
@@ -134,7 +138,7 @@ main(cat="spark_catalog", schema="default", migrations_dir=get_migrations_dir())
 `main` builds its own session via `get_spark()` and, being library code, leaves logging to
 you. To supply your own — a session already
 configured by your job, say — call `run_migrations(spark, cat, schema, output_folder,
-migrations_root)` directly.
+migrations_dir)` directly.
 
 Every rendered statement is written to a timestamped folder under `migrations_out/`
 before it is executed, so you can always read the exact SQL a run applied.
@@ -144,7 +148,7 @@ before it is executed, so you can always read the exact SQL a run applied.
 Generate one rather than hand-rolling the header:
 
 ```bash
-spark-sql-migrations create_new_migration     --message "add batch id to metrics"     --output-path path/to/migrations/all_spark_migrations
+spark-sql-migrations create_new_migration     --message "add batch id to metrics"     --output-path path/to/migrations
 ```
 
 That writes `<yymmdd>_<slug>_<revision_id>.sql` from the template shipped in the package
@@ -183,6 +187,44 @@ begin
 end;
 ```
 
+### Databricks-only statements
+
+`is_dbr` is in the render context, so a migration in the shared chain can carry SQL only
+Databricks understands:
+
+```sql
+-- revision_id:e8614b76da6d;
+-- prev_revision_id:57037b6c19b4;
+{% if is_dbr %}
+begin
+grant READ FILES on external location `some-bucket` to `some_group`;
+end;
+{% endif %}
+```
+
+`create_new_migration` writes that conditional for you with `--add-is-dbr`, below the
+headers so they stay readable:
+
+```bash
+spark-sql-migrations create_new_migration     --message "read files on the cms zip external location"     --output-path path/to/migrations     --add-is-dbr
+```
+
+This has to happen at render time, not in SQL. OSS Spark rejects `GRANT` and the rest of
+its unsupported-native-command list while *parsing*, so a `declare exit handler` in the same
+script never gets the chance to run — and the whole script fails to compile, not just that
+statement. Jinja removes the text before `spark.sql()` ever sees it.
+
+A migration left holding nothing but comments is recorded in the version table and not
+executed, so the chain advances in step on both engines. That recording is forward-only: a
+catalog migrated locally past a gated revision will not pick that statement up if it is
+later migrated from Databricks. For disposable test schemas that is exactly what you want;
+for a long-lived catalog, don't migrate it from both.
+
+`_spark_migrations_version` holds one row and one column, `version_num`. There is no
+in-place upgrade from the older two-chain shape, which carried a `migration_type` column and
+a row per chain: a catalog bootstrapped by an earlier release has to be rebuilt from
+scratch, which also clears the revision history that layout recorded.
+
 ## Local vs Databricks
 
 `get_spark()` returns a `DatabricksSession` when `is_dbr()` is true and a Delta-configured

{spark_sql_migrations-0.0.4 → spark_sql_migrations-0.0.6}/spark_sql_migrations.egg-info/SOURCES.txt

@@ -13,8 +13,8 @@ spark_sql_migrations.egg-info/namespace_packages.txt
 spark_sql_migrations.egg-info/requires.txt
 spark_sql_migrations.egg-info/top_level.txt
 spark_sql_migrations/migration_templates/default_migration_template.sql
-spark_sql_migrations/migrations_initial/first_01_intial_create_cat_dbr_only.sql
-spark_sql_migrations/migrations_initial/first_02_intial_create_schema_all.sql
-spark_sql_migrations/migrations_initial/first_03_intial_create_state_table_all.sql
+spark_sql_migrations/migrations_initial/first_01_intial_create_cat.sql
+spark_sql_migrations/migrations_initial/first_02_intial_create_schema.sql
+spark_sql_migrations/migrations_initial/first_03_intial_create_state_table.sql
 spark_sql_migrations/spark_sql/__init__.py
 spark_sql_migrations/spark_sql/spark_sql.py

spark_sql_migrations-0.0.4/spark_sql_migrations/migrations_initial/first_01_intial_create_cat_dbr_only.sql

@@ -1,4 +0,0 @@
--- revision_id:;
-begin
-create catalog if not exists {{cat}};
-end;