spark-sql-migrations 0.0.3__tar.gz → 0.0.4__tar.gz

{spark_sql_migrations-0.0.3 → spark_sql_migrations-0.0.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark_sql_migrations
-Version: 0.0.3
+Version: 0.0.4
 Summary: Chained, idempotent SQL migrations for Spark and Databricks.
 Home-page: https://github.com/mcjug2015/spark_sql_migrations
 Author: Victor Semenov
@@ -81,10 +81,12 @@ Migrations come from two different owners, and they behave differently:
 | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
 | **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
 | **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
-| **when applied** | every run | once, then recorded |
+| **when applied** | until the version table exists | once, then recorded |
 
-The bootstrap chain is re-applied on **every** run, so it must be idempotent. Your own
-chains are applied once each and their head revision is written to the version table.
+The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
+skipped from then on. A run that fails partway through bootstrapping replays it, so it must
+still be idempotent. Your own chains are applied once each and their head revision is
+written to the version table.
 
 ## Laying out your chains
 

{spark_sql_migrations-0.0.3 → spark_sql_migrations-0.0.4}/setup.py

@@ -82,10 +82,12 @@ Migrations come from two different owners, and they behave differently:
 | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
 | **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
 | **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
-| **when applied** | every run | once, then recorded |
+| **when applied** | until the version table exists | once, then recorded |
 
-The bootstrap chain is re-applied on **every** run, so it must be idempotent. Your own
-chains are applied once each and their head revision is written to the version table.
+The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
+skipped from then on. A run that fails partway through bootstrapping replays it, so it must
+still be idempotent. Your own chains are applied once each and their head revision is
+written to the version table.
 
 ## Laying out your chains
 
@@ -252,5 +254,5 @@ Apache License 2.0 — see [LICENSE](LICENSE).
     },
     'python_requires': '>=3.10',
     'url': 'https://github.com/mcjug2015/spark_sql_migrations',
-    'version': '0.0.3',
+    'version': '0.0.4',
 })

{spark_sql_migrations-0.0.3 → spark_sql_migrations-0.0.4}/spark_sql_migrations/spark_sql/spark_sql.py

@@ -5,7 +5,8 @@ Two kinds of migration exist, and they are owned by different parties:
 * the *initial* chain and the new-migration template ship inside this package.
   They bootstrap the catalog, the schema and the version table, are selected by
   filename suffix (``_all.sql`` / ``_dbr_only.sql``) rather than by revision,
-  and are re-applied on every run -- so they must be idempotent.
+  and are applied on every run until the version table exists -- so a run that
+  failed partway through replays them, and they must be idempotent.
 * the ``all_spark_migrations/`` and ``dbr_only_migrations/`` chains belong to
   the consuming project. The caller passes their parent directory as
   ``migrations_root``; each chain is walked from its root via
@@ -288,7 +289,11 @@ def run_migrations(spark, cat, schema, output_folder, migrations_root):
     os.makedirs(initial_output_folder, exist_ok=True)
     os.makedirs(dbr_ouput_folder, exist_ok=True)
     os.makedirs(all_spark_ouput_folder, exist_ok=True)
-    migrate_initial(spark, initial_output_folder, cat, schema)
+    version_table = f"{cat}.{schema}.{VERSION_TABLE}"
+    if spark.catalog.tableExists(version_table):
+        logger.info(f"{version_table} already exists; skipping the initial migrations;")
+    else:
+        migrate_initial(spark, initial_output_folder, cat, schema)
     if is_dbr():
         migrate_w_rev(spark, dbr_ouput_folder, migrations_root, cat, schema, DBR_ONLY)
     migrate_w_rev(spark, all_spark_ouput_folder, migrations_root, cat, schema, ALL_SPARK)

{spark_sql_migrations-0.0.3 → spark_sql_migrations-0.0.4}/spark_sql_migrations.egg-info/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: spark_sql_migrations
-Version: 0.0.3
+Version: 0.0.4
 Summary: Chained, idempotent SQL migrations for Spark and Databricks.
 Home-page: https://github.com/mcjug2015/spark_sql_migrations
 Author: Victor Semenov
@@ -81,10 +81,12 @@ Migrations come from two different owners, and they behave differently:
 | **what** | catalog, schema, `_spark_migrations_version` table | your tables and columns |
 | **where** | `spark_sql_migrations/migrations_initial/` | `all_spark_migrations/`, `dbr_only_migrations/` |
 | **selected by** | filename suffix — `_all.sql` everywhere, `_dbr_only.sql` on Databricks | the `prev_revision_id` chain |
-| **when applied** | every run | once, then recorded |
+| **when applied** | until the version table exists | once, then recorded |
 
-The bootstrap chain is re-applied on **every** run, so it must be idempotent. Your own
-chains are applied once each and their head revision is written to the version table.
+The bootstrap chain is applied on every run until `_spark_migrations_version` exists, and
+skipped from then on. A run that fails partway through bootstrapping replays it, so it must
+still be idempotent. Your own chains are applied once each and their head revision is
+written to the version table.
 
 ## Laying out your chains