ch-migrate-cli 0.6.0__tar.gz → 0.7.0__tar.gz

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ch-migrate-cli
-Version: 0.6.0
+Version: 0.7.0
 Summary: SQL-first ClickHouse schema migrations across environments: the ch-migrate command
 Project-URL: Homepage, https://github.com/DRYCodeWorks/ch-migrate
 Project-URL: Repository, https://github.com/DRYCodeWorks/ch-migrate
@@ -65,8 +65,15 @@ This README describes the source checkout, which may be ahead of PyPI. To try an
 
 ## Upgrade from 0.x
 
-0.6 requires Python 3.10+; 0.5.x is the last line supporting Python 3.9.
-A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades to 0.6; pin
+**From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
+`ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
+so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
+`SELECT ON system.*`, such as on ClickHouse Cloud, add the grants listed in the
+[CHANGELOG](CHANGELOG.md). A loose requirement such as `ch-migrate-cli` upgrades to
+0.7; pin `ch-migrate-cli==0.6.0` to stay on 0.6.
+
+**From 0.5:** 0.6 and later require Python 3.10+; 0.5.x is the last line supporting Python 3.9.
+A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades past 0.5; pin
 `ch-migrate-cli==0.5.1` until each project has run the command below.
 After upgrading the package, run this in each migration project:
 
@@ -269,13 +276,111 @@ information or the single-server/Cloud default for version-table DDL. Review
 that DDL before using it for a self-hosted Replicated database. Cloud qualification
 is separate from the local suite; the local replicated harness has one shard.
 
+### Migrations that wait
+
+`up` doesn't record a migration as applied until the work its statements started
+has finished. After each statement that can write, it polls `system.mutations` on
+the same checked ClickHouse session, so earlier `SET` statements stay in effect.
+`op.execute`, raw `op.get_bind()` execution and `run_sql` all pass through that
+check. Upgrades run in the CLI process, so killing it can't leave a child Alembic
+runner advancing revisions. `down` waits the same way.
+
+```bash
+ch-migrate up dev                 # No default waiting timeout
+ch-migrate up dev --timeout 120   # One waiting budget across the invocation
+```
+
+What `up` waits for:
+
+- **Every unfinished mutation in the project database,** including one another
+  writer started. Before the first migration runs, `up` waits for any already
+  queued, and a failed one blocks the run. The version table's own mutations are
+  left to [its bookkeeping](#the-version-table).
+- **Each replica that has to apply it.** Zero parts left isn't completion on its
+  own: the mutation must report `is_done`. A ReplicatedMergeTree mutation must be
+  finished on every replica, and a replica that hasn't received it yet is still
+  pending. Configure `cluster` for self-hosted replicated tables, or use a
+  Replicated database's automatic cluster; without either, `up` refuses a table
+  with more than one replica. SharedMergeTree on ClickHouse Cloud reports one
+  shared mutation.
+- **`ON CLUSTER` DDL on every host.** `up` sends these statements with
+  `distributed_ddl_output_mode = 'throw'`, so ClickHouse answers once every host
+  has run the statement, and a host that fails or doesn't finish in time fails the
+  migration with ClickHouse's message. The limit is 270 seconds, below the
+  client's read timeout, or what's left of `--timeout` if that's less. Mutations
+  the statement starts are then waited for as above.
+
+Statements ClickHouse already holds until they finish, such as a type-changing
+`MODIFY COLUMN`, a `MODIFY TTL` that materializes, or a lightweight `DELETE`,
+still hold their request as they did in 0.6.
+
+While it waits, `up` shows progress on stderr. In a terminal, each mutation gets a
+live bar weighted by bytes, with its parts and an estimate of the time left:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━  18.2 / 41.0 GiB · 23/41 parts · 1:12 · ~1:30 left
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 41.0 GiB · 2:03
+```
+
+Self-hosted replicated tables get one row per replica. When output isn't a
+terminal (CI logs, pipes, coding agents), `up` prints plain lines instead: one
+when a wait starts, one at each 10% of the parts or every 30 seconds, and one
+when it finishes:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  waiting on mutation_7.txt (demo_dev.events): 0/41 parts
+  waiting on mutation_7.txt (demo_dev.events): 5/41 parts · 0:14
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 2:03
+```
+
+Part sizes come from `system.parts`. If the migration role can't read it, the bar
+counts parts only and says so once; progress never changes whether `up` succeeds.
+
+**When `up` stops short.** After a timeout, Ctrl-C, dropped connection, killed
+process or failed statement, ClickHouse keeps working and the migration isn't
+recorded. `up` and `down` note each migration in `_ch_migrate_in_progress` when it
+starts and clear the note once its version is recorded, so the next run knows the
+migration was interrupted. That run waits for unfinished work, then:
+
+- **Runs the migration again on its own** when every statement in it passes the
+  idempotency gate without a waiver, so a repeat leaves the same result.
+- **Stops before sending anything** when the migration has a waived statement or
+  SQL the gate can't read, because that statement may already have applied. It
+  names each one. Check what the earlier run changed, fix the SQL if needed, then
+  name the revision: `ch-migrate up ENV --rerun-interrupted REV` (or `down`), or
+  `-x rerun_interrupted=REV` with `alembic`. Naming it accepts the repeat.
+
+Every `UPDATE`, `DELETE` and `INSERT ... SELECT` needs a waiver, so an interrupted
+backfill always takes that second path.
+
+**Failed mutation.** ClickHouse's `latest_fail_reason` stops the run, and `up`
+prints the scoped `KILL MUTATION` statement for an operator; ch-migrate never
+runs it. ClickHouse keeps retrying a failed mutation, and later mutations on that
+table queue behind it. If the cause is temporary, such as memory or disk, fix it
+and run `up` again. If the statement itself is wrong, kill the mutation, fix the
+SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's
+waived. Killing a mutation doesn't undo the parts it already rewrote.
+
+Run one migration runner per database, using a CI concurrency key or an external
+lock; the in-progress notes aren't a lock. Re-run `bootstrap` when upgrading a
+restricted migration user to obtain the `system.replicas` and `system.parts` grants.
+
 ### Re-runnable migrations
 
 `up` statically checks pending upgrade statements before Alembic executes any of
 them. Idempotency and standalone-SET errors refuse the whole run and print the source file, line,
 statement, and suggested fix. There is no command-line bypass; `--skip-mv-check`
 does not skip this gate. Other findings are warnings in `up`; standalone `lint`
-retains their configured severities.
+retains their configured severities. A direct `alembic upgrade` runs the same gate
+from `env.py` before its first migration.
+
+The gate reads SQL in `run_sql` files and in literal strings, f-strings and
+`text("...")` passed to `op.execute`, or to `execute` and `exec_driver_sql` on a
+connection such as `op.get_bind()`. SQL that only exists at run time, such as a
+variable, a built string or `run_sql(name)`, can't be checked, so it needs a
+reasoned waiver like any other statement the gate can't vouch for.
 
 Static lint rejects a standalone `SET` when `migrations/env.py` lacks the
 `CH_MIGRATE_ENV_VERSION = 2` marker: that statement would be ignored on the old
@@ -309,7 +414,7 @@ INSERT INTO {db}.target SELECT id FROM {db}.source;
 ```
 
 For inline Python SQL, use the same directive in a `#` comment directly above
-`op.execute`. Empty reasons remain errors. Waivers print as INFO with their
+the call that sends it. Empty reasons remain errors. Waivers print as INFO with their
 reasons in `lint` and `up`; they do not make the statement idempotent.
 `MODIFY`, `MATERIALIZE`, `CLEAR COLUMN`, `TRUNCATE`, `OPTIMIZE`, `SYSTEM`, grants,
 revokes, `REPLACE PARTITION`, attaching or detaching a whole partition, and
@@ -318,11 +423,12 @@ The reviewable classification corpus is `tests/corpus/idempotency.yaml`.
 
 **Recovery:** fix the cause of a partial failure, then run `ch-migrate up ENV`
 again. Earlier idempotent statements can run again; the version advances only
-when the revision completes. On ClickHouse Cloud, a column that the failed attempt
-dropped or renamed stays locked until that change's mutation finishes: a retry
-before then fails with "affected by mutation ... which is not finished yet", even
-with `IF EXISTS`. Wait for it in `system.mutations`, then run `up` again. Before retrying waived operations, review what the
-earlier attempt actually changed.
+when the revision completes. A dropped or renamed column stays locked until that
+change's mutation finishes, and `up` waits for it before running the next
+statement, so a retry after a later failure doesn't hit "affected by mutation ...
+which is not finished yet". If `up` was stopped while it waited, run it again: it
+waits for that mutation, then runs the revision again, unless the revision has a
+statement the gate can't vouch for (see below).
 
 **Existing history:** `upgrade-env` records the current local script head(s) in
 `config.yaml` as `lint.gate_baseline`, preserving YAML comments. It records
@@ -435,7 +541,7 @@ Example: `ch-migrate new dev add_status --table logs`
 
 ### `up`
 
-`ch-migrate up ENV [-r REV] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
+`ch-migrate up ENV [-r REV] [--timeout SECONDS] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--timeout` is a finite positive waiting limit; omitting it waits without a deadline. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
 
 Example: `ch-migrate up dev --revision abc123`
 
@@ -549,7 +655,13 @@ The legacy `CH_<ENV>_PASSWORD` remains supported. Never commit credentials or pa
 
 ### Hooks
 
-Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the migration batch; `post_migrate` runs after each revision. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do not put secrets in it.
+Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the
+migration batch; `post_migrate` runs after each revision body **before its version
+is completed**, so hook mutations cannot escape the waiting barrier. A resumed
+upgrade reuses its unfinished batch's receipts instead of repeating completed
+hook writes. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through
+the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do
+not put secrets in it.
 
 ```yaml
 hooks:

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.7.0}/README.md

@@ -23,8 +23,15 @@ This README describes the source checkout, which may be ahead of PyPI. To try an
 
 ## Upgrade from 0.x
 
-0.6 requires Python 3.10+; 0.5.x is the last line supporting Python 3.9.
-A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades to 0.6; pin
+**From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
+`ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
+so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
+`SELECT ON system.*`, such as on ClickHouse Cloud, add the grants listed in the
+[CHANGELOG](CHANGELOG.md). A loose requirement such as `ch-migrate-cli` upgrades to
+0.7; pin `ch-migrate-cli==0.6.0` to stay on 0.6.
+
+**From 0.5:** 0.6 and later require Python 3.10+; 0.5.x is the last line supporting Python 3.9.
+A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades past 0.5; pin
 `ch-migrate-cli==0.5.1` until each project has run the command below.
 After upgrading the package, run this in each migration project:
 
@@ -227,13 +234,111 @@ information or the single-server/Cloud default for version-table DDL. Review
 that DDL before using it for a self-hosted Replicated database. Cloud qualification
 is separate from the local suite; the local replicated harness has one shard.
 
+### Migrations that wait
+
+`up` doesn't record a migration as applied until the work its statements started
+has finished. After each statement that can write, it polls `system.mutations` on
+the same checked ClickHouse session, so earlier `SET` statements stay in effect.
+`op.execute`, raw `op.get_bind()` execution and `run_sql` all pass through that
+check. Upgrades run in the CLI process, so killing it can't leave a child Alembic
+runner advancing revisions. `down` waits the same way.
+
+```bash
+ch-migrate up dev                 # No default waiting timeout
+ch-migrate up dev --timeout 120   # One waiting budget across the invocation
+```
+
+What `up` waits for:
+
+- **Every unfinished mutation in the project database,** including one another
+  writer started. Before the first migration runs, `up` waits for any already
+  queued, and a failed one blocks the run. The version table's own mutations are
+  left to [its bookkeeping](#the-version-table).
+- **Each replica that has to apply it.** Zero parts left isn't completion on its
+  own: the mutation must report `is_done`. A ReplicatedMergeTree mutation must be
+  finished on every replica, and a replica that hasn't received it yet is still
+  pending. Configure `cluster` for self-hosted replicated tables, or use a
+  Replicated database's automatic cluster; without either, `up` refuses a table
+  with more than one replica. SharedMergeTree on ClickHouse Cloud reports one
+  shared mutation.
+- **`ON CLUSTER` DDL on every host.** `up` sends these statements with
+  `distributed_ddl_output_mode = 'throw'`, so ClickHouse answers once every host
+  has run the statement, and a host that fails or doesn't finish in time fails the
+  migration with ClickHouse's message. The limit is 270 seconds, below the
+  client's read timeout, or what's left of `--timeout` if that's less. Mutations
+  the statement starts are then waited for as above.
+
+Statements ClickHouse already holds until they finish, such as a type-changing
+`MODIFY COLUMN`, a `MODIFY TTL` that materializes, or a lightweight `DELETE`,
+still hold their request as they did in 0.6.
+
+While it waits, `up` shows progress on stderr. In a terminal, each mutation gets a
+live bar weighted by bytes, with its parts and an estimate of the time left:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━  18.2 / 41.0 GiB · 23/41 parts · 1:12 · ~1:30 left
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 41.0 GiB · 2:03
+```
+
+Self-hosted replicated tables get one row per replica. When output isn't a
+terminal (CI logs, pipes, coding agents), `up` prints plain lines instead: one
+when a wait starts, one at each 10% of the parts or every 30 seconds, and one
+when it finishes:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  waiting on mutation_7.txt (demo_dev.events): 0/41 parts
+  waiting on mutation_7.txt (demo_dev.events): 5/41 parts · 0:14
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 2:03
+```
+
+Part sizes come from `system.parts`. If the migration role can't read it, the bar
+counts parts only and says so once; progress never changes whether `up` succeeds.
+
+**When `up` stops short.** After a timeout, Ctrl-C, dropped connection, killed
+process or failed statement, ClickHouse keeps working and the migration isn't
+recorded. `up` and `down` note each migration in `_ch_migrate_in_progress` when it
+starts and clear the note once its version is recorded, so the next run knows the
+migration was interrupted. That run waits for unfinished work, then:
+
+- **Runs the migration again on its own** when every statement in it passes the
+  idempotency gate without a waiver, so a repeat leaves the same result.
+- **Stops before sending anything** when the migration has a waived statement or
+  SQL the gate can't read, because that statement may already have applied. It
+  names each one. Check what the earlier run changed, fix the SQL if needed, then
+  name the revision: `ch-migrate up ENV --rerun-interrupted REV` (or `down`), or
+  `-x rerun_interrupted=REV` with `alembic`. Naming it accepts the repeat.
+
+Every `UPDATE`, `DELETE` and `INSERT ... SELECT` needs a waiver, so an interrupted
+backfill always takes that second path.
+
+**Failed mutation.** ClickHouse's `latest_fail_reason` stops the run, and `up`
+prints the scoped `KILL MUTATION` statement for an operator; ch-migrate never
+runs it. ClickHouse keeps retrying a failed mutation, and later mutations on that
+table queue behind it. If the cause is temporary, such as memory or disk, fix it
+and run `up` again. If the statement itself is wrong, kill the mutation, fix the
+SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's
+waived. Killing a mutation doesn't undo the parts it already rewrote.
+
+Run one migration runner per database, using a CI concurrency key or an external
+lock; the in-progress notes aren't a lock. Re-run `bootstrap` when upgrading a
+restricted migration user to obtain the `system.replicas` and `system.parts` grants.
+
 ### Re-runnable migrations
 
 `up` statically checks pending upgrade statements before Alembic executes any of
 them. Idempotency and standalone-SET errors refuse the whole run and print the source file, line,
 statement, and suggested fix. There is no command-line bypass; `--skip-mv-check`
 does not skip this gate. Other findings are warnings in `up`; standalone `lint`
-retains their configured severities.
+retains their configured severities. A direct `alembic upgrade` runs the same gate
+from `env.py` before its first migration.
+
+The gate reads SQL in `run_sql` files and in literal strings, f-strings and
+`text("...")` passed to `op.execute`, or to `execute` and `exec_driver_sql` on a
+connection such as `op.get_bind()`. SQL that only exists at run time, such as a
+variable, a built string or `run_sql(name)`, can't be checked, so it needs a
+reasoned waiver like any other statement the gate can't vouch for.
 
 Static lint rejects a standalone `SET` when `migrations/env.py` lacks the
 `CH_MIGRATE_ENV_VERSION = 2` marker: that statement would be ignored on the old
@@ -267,7 +372,7 @@ INSERT INTO {db}.target SELECT id FROM {db}.source;
 ```
 
 For inline Python SQL, use the same directive in a `#` comment directly above
-`op.execute`. Empty reasons remain errors. Waivers print as INFO with their
+the call that sends it. Empty reasons remain errors. Waivers print as INFO with their
 reasons in `lint` and `up`; they do not make the statement idempotent.
 `MODIFY`, `MATERIALIZE`, `CLEAR COLUMN`, `TRUNCATE`, `OPTIMIZE`, `SYSTEM`, grants,
 revokes, `REPLACE PARTITION`, attaching or detaching a whole partition, and
@@ -276,11 +381,12 @@ The reviewable classification corpus is `tests/corpus/idempotency.yaml`.
 
 **Recovery:** fix the cause of a partial failure, then run `ch-migrate up ENV`
 again. Earlier idempotent statements can run again; the version advances only
-when the revision completes. On ClickHouse Cloud, a column that the failed attempt
-dropped or renamed stays locked until that change's mutation finishes: a retry
-before then fails with "affected by mutation ... which is not finished yet", even
-with `IF EXISTS`. Wait for it in `system.mutations`, then run `up` again. Before retrying waived operations, review what the
-earlier attempt actually changed.
+when the revision completes. A dropped or renamed column stays locked until that
+change's mutation finishes, and `up` waits for it before running the next
+statement, so a retry after a later failure doesn't hit "affected by mutation ...
+which is not finished yet". If `up` was stopped while it waited, run it again: it
+waits for that mutation, then runs the revision again, unless the revision has a
+statement the gate can't vouch for (see below).
 
 **Existing history:** `upgrade-env` records the current local script head(s) in
 `config.yaml` as `lint.gate_baseline`, preserving YAML comments. It records
@@ -393,7 +499,7 @@ Example: `ch-migrate new dev add_status --table logs`
 
 ### `up`
 
-`ch-migrate up ENV [-r REV] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
+`ch-migrate up ENV [-r REV] [--timeout SECONDS] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--timeout` is a finite positive waiting limit; omitting it waits without a deadline. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
 
 Example: `ch-migrate up dev --revision abc123`
 
@@ -507,7 +613,13 @@ The legacy `CH_<ENV>_PASSWORD` remains supported. Never commit credentials or pa
 
 ### Hooks
 
-Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the migration batch; `post_migrate` runs after each revision. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do not put secrets in it.
+Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the
+migration batch; `post_migrate` runs after each revision body **before its version
+is completed**, so hook mutations cannot escape the waiting barrier. A resumed
+upgrade reuses its unfinished batch's receipts instead of repeating completed
+hook writes. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through
+the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do
+not put secrets in it.
 
 ```yaml
 hooks:

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.7.0}/ch_migrate/__init__.py

@@ -5,7 +5,7 @@ Usage:
     from ch_migrate import read_sql, get_db, get_env_config, create_dictionary
 """
 
-__version__ = "0.6.0"
+__version__ = "0.7.0"
 
 
 from typing import Any

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.7.0}/ch_migrate/alembic_env.py

@@ -4,6 +4,7 @@ from __future__ import annotations
 
 import logging
 import os
+from functools import partial
 from logging.config import fileConfig
 from pathlib import Path
 from types import SimpleNamespace
@@ -18,6 +19,7 @@ from dotenv import load_dotenv
 from sqlalchemy import URL, Column, MetaData, String, Table, create_engine, pool, text
 from sqlalchemy.sql.dml import Insert
 
+from ch_migrate import ui
 from ch_migrate.config import get_env_config
 from ch_migrate.hooks import HookRegistry, run_hooks
 from ch_migrate.rebase import _literal_assignment
@@ -28,6 +30,8 @@ from ch_migrate.version_table import (
     inspect_version_table,
     sync_version_replica,
 )
+from ch_migrate.waiting import MigrationWaiter
+from ch_migrate.waiting.reruns import RerunGuard
 
 ENV_VERSION = 2
 DEFAULT_SESSION_TIMEOUT = 1800
@@ -96,18 +100,23 @@ def run() -> None:
     """Run the selected environment on one official-dialect session."""
     root = Path.cwd()
     load_dotenv(root / ".env.local")
-    env_config = get_env_config(os.environ.get("CH_ENVIRONMENT", "dev"), root / "config.yaml")
+    # `ch-migrate up` runs Alembic in its own process and names the environment directly.
+    environment = context.config.attributes.get(
+        "ch_migrate_environment", os.environ.get("CH_ENVIRONMENT", "dev")
+    )
+    env_config = get_env_config(environment, root / "config.yaml")
     os.environ["CH_DATABASE"] = env_config["database"]
     if env_config.get("cluster"):
         os.environ["CH_CLUSTER"] = env_config["cluster"]
     else:
         os.environ.pop("CH_CLUSTER", None)
     if context.config.config_file_name is not None:
-        fileConfig(context.config.config_file_name)
+        fileConfig(context.config.config_file_name, disable_existing_loggers=False)
+    _route_logs(context.config.attributes.get("ch_migrate_log_handler"))
     if context.is_offline_mode():
         _run_offline(env_config)
     else:
-        _run_online(env_config)
+        _run_online(env_config, environment)
 
 
 def has_current_env(path: Path) -> bool:
@@ -118,6 +127,18 @@ def has_current_env(path: Path) -> bool:
     )
 
 
+def _route_logs(handler: logging.Handler | None) -> None:
+    # `ch-migrate up` reports each migration in its own words, so Alembic's and
+    # ch-migrate's records go to its handler instead of alembic.ini's console handler.
+    if handler is None:
+        return
+    for name in ("alembic", "ch_migrate"):
+        named = logging.getLogger(name)
+        named.handlers = [handler]
+        named.propagate = False
+        named.setLevel(logging.INFO)
+
+
 def _run_offline(env_config: dict[str, Any]) -> None:
     context.configure(
         url=_url(env_config),
@@ -135,7 +156,7 @@ def _run_offline(env_config: dict[str, Any]) -> None:
         context.run_migrations()
 
 
-def _run_online(env_config: dict[str, Any]) -> None:
+def _run_online(env_config: dict[str, Any], environment: str) -> None:
     # The session lives on whichever replica its connection reached: ClickHouse Cloud routes
     # each TCP connection, not each session id. A pool of its own keeps the session on one
     # connection instead of sharing the driver's default pool with other in-process clients.
@@ -146,36 +167,56 @@ def _run_online(env_config: dict[str, Any]) -> None:
     )
     try:
         with engine.connect() as connection:
-            _run_connection(connection, env_config)
+            _run_connection(connection, env_config, environment)
     finally:
         engine.dispose()
         session_pool.clear()
 
 
-def _run_connection(connection, env_config: dict[str, Any]) -> None:
+def _run_connection(connection, env_config: dict[str, Any], environment: str) -> None:
     hooks = HookRegistry.from_config(env_config.get("hooks"))
-    database = env_config["database"]
+    state = _checked_session(connection, env_config)
+    context.configure(
+        connection=connection,
+        target_metadata=None,
+        version_table="alembic_version",
+        version_table_schema=state.database,
+        on_version_apply=_post_migrate(hooks, state.database),
+        ch_migrate_version_state=state,
+    )
+    waiter = MigrationWaiter(connection, state, context.config.attributes.get("ch_migrate_timeout"))
+    guard = _rerun_guard(waiter.client, state, environment)
+    try:
+        waiter.install(context.config)
+        with context.begin_transaction():
+            # Work an earlier run left unfinished finishes before anything is sent again.
+            waiter.wait(before_run=True)
+            run_hooks(
+                connection,
+                hooks.pre_migrate,
+                db=state.database,
+                phase="pre_migrate",
+                revision="all",
+            )
+            runtime = context.get_context()
+            runtime._migrations_fn = guard.steps(runtime._migrations_fn)
+            context.run_migrations()
+    finally:
+        waiter.close()
+
+
+def _checked_session(connection, env_config: dict[str, Any]) -> VersionTableState:
     # Establish the session before requiring its existence on every later request.
     connection.exec_driver_sql("SELECT 1")
     client = connection.connection.dbapi_connection.client
     client.set_client_setting("session_check", 1)
-    state = inspect_version_table(client, database, env_config.get("cluster"))
+    state = inspect_version_table(client, env_config["database"], env_config.get("cluster"))
     assert_version_mutations_healthy(client, state)
     state = sync_version_replica(client, state)
     if warning := state.warning():
         logger.warning(warning)
     connection.dialect.ddl_compiler = VersionTableDDLCompiler
-    context.configure(
-        connection=connection,
-        target_metadata=None,
-        version_table="alembic_version",
-        version_table_schema=database,
-        on_version_apply=_post_migrate(hooks, database),
-        ch_migrate_version_state=state,
-    )
-    with context.begin_transaction():
-        run_hooks(connection, hooks.pre_migrate, db=database, phase="pre_migrate", revision="all")
-        context.run_migrations()
+    return state
 
 
 def _url(env_config: dict[str, Any]) -> URL:
@@ -210,3 +251,27 @@ def _post_migrate(hooks: HookRegistry, database: str):
         )
 
     return after_revision
+
+
+def _rerun_guard(client, state: VersionTableState, environment: str) -> RerunGuard:
+    attributes = context.config.attributes
+    # `ch-migrate up` passes --rerun-interrupted directly; `down` and a direct alembic
+    # command pass it as -x rerun_interrupted=REV[,REV].
+    named = attributes.get("ch_migrate_rerun_interrupted") or context.get_x_argument(
+        as_dictionary=True
+    ).get("rerun_interrupted", "").split(",")
+    guard = RerunGuard(client, state, frozenset(name.strip() for name in named if name.strip()))
+    if not attributes.get("ch_migrate_gate_checked"):
+        # A direct `alembic upgrade` hasn't been through `ch-migrate up`'s gate.
+        guard.before_first_upgrade = partial(_enforce_gate, environment)
+    return guard
+
+
+def _enforce_gate(environment: str) -> None:
+    from ch_migrate.gate import enforce_up_gate  # lint imports this module
+
+    try:
+        enforce_up_gate(Path.cwd(), environment)
+    except ui.Failure as failure:
+        failure.show()
+        raise SystemExit(1) from None

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.7.0}/ch_migrate/bootstrap.py

@@ -147,6 +147,8 @@ def build_bootstrap_sql(
         f"GRANT SELECT ON system.databases TO {project}_migration_role;",
         f"GRANT SELECT ON system.tables TO {project}_migration_role;",
         f"GRANT SELECT ON system.mutations TO {project}_migration_role;",
+        f"GRANT CURRENT GRANTS(SELECT ON system.replicas) TO {project}_migration_role;",
+        f"GRANT CURRENT GRANTS(SELECT ON system.parts) TO {project}_migration_role;",
         f"GRANT CURRENT GRANTS(SYSTEM SYNC REPLICA ON {db}.*) TO {project}_migration_role;",
         f"GRANT CURRENT GRANTS(SYSTEM SYNC DATABASE REPLICA ON {db}.*) TO {project}_migration_role;",
         f"GRANT CURRENT GRANTS(CLUSTER ON *.*) TO {project}_migration_role;",
@@ -324,7 +326,9 @@ def run_bootstrap(
     secure = env_config.get("secure", True)
     port = env_config.get("port", 8443 if secure else 8123)
 
-    ui.step(f"Bootstrapping {env_name}: database {env_config['database']} on {host}, as {admin_user}")
+    ui.step(
+        f"Bootstrapping {env_name}: database {env_config['database']} on {host}, as {admin_user}"
+    )
     ui.detail(f"Migration user: {migration_user}")
     if dict_reader_name:
         ui.detail(f"Dict reader: {dict_reader_name}")