ch-migrate-cli 0.5.0__tar.gz → 0.6.0__tar.gz

{ch_migrate_cli-0.5.0 → ch_migrate_cli-0.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ch-migrate-cli
-Version: 0.5.0
+Version: 0.6.0
 Summary: SQL-first ClickHouse schema migrations across environments: the ch-migrate command
 Project-URL: Homepage, https://github.com/DRYCodeWorks/ch-migrate
 Project-URL: Repository, https://github.com/DRYCodeWorks/ch-migrate
@@ -14,22 +14,21 @@ Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
-Classifier: Programming Language :: Python :: 3.9
 Classifier: Programming Language :: Python :: 3.10
 Classifier: Programming Language :: Python :: 3.11
 Classifier: Programming Language :: Python :: 3.12
 Classifier: Programming Language :: Python :: 3.13
 Classifier: Programming Language :: Python :: 3.14
 Classifier: Topic :: Database
-Requires-Python: >=3.9
-Requires-Dist: alembic>=1.14.0
+Requires-Python: >=3.10
+Requires-Dist: alembic>=1.18.0
 Requires-Dist: boto3>=1.42.24
 Requires-Dist: click>=8.1.0
-Requires-Dist: clickhouse-connect>=0.7.0
-Requires-Dist: clickhouse-sqlalchemy>=0.3.0
+Requires-Dist: clickhouse-connect[alembic]>=1.9
 Requires-Dist: python-dotenv>=1.0.0
 Requires-Dist: pyyaml>=6.0
 Requires-Dist: rich>=13.0.0
+Requires-Dist: ruamel-yaml<0.20,>=0.19.1
 Provides-Extra: dev
 Requires-Dist: black>=23.0.0; extra == 'dev'
 Requires-Dist: isort>=5.12.0; extra == 'dev'
@@ -45,7 +44,7 @@ Description-Content-Type: text/markdown
 
 ## What it is
 
-`ch-migrate` manages SQL-first ClickHouse migrations across environments: author SQL files, bootstrap databases and roles, inspect migrations and dependencies, and compare schema snapshots. Alembic owns revision history; the database dialect owns DDL compilation. This operational layer complements ClickHouse's official Alembic integration rather than replacing it. This development line still uses `clickhouse-sqlalchemy` for Alembic connections; it does not imply an endorsement from ClickHouse.
+`ch-migrate` manages SQL-first ClickHouse migrations across environments: author SQL files, bootstrap databases and roles, inspect migrations and dependencies, and compare schema snapshots. Alembic owns revision history; ClickHouse's official `clickhouse-connect[alembic]` integration owns the dialect and DDL compilation. ch-migrate supplies the operational layer above them, without implying an endorsement from ClickHouse.
 
 Background: [ClickHouse migrations with Alembic](https://www.drycodeworks.com/articles/dev-guides/clickhouse-migrations-with-alembic).
 
@@ -58,12 +57,38 @@ pip install ch-migrate-cli
 ch-migrate --version
 ```
 
-The command is `ch-migrate`, the PyPI package is `ch-migrate-cli` (PyPI treats `ch-migrate` as the same name as the existing, unrelated `chmigrate`), and migrations import from `ch_migrate`. Versions up to 0.4.1 were published as `clickhouse-alembic` with the import package `clickhouse_alembic`; that import still works with a deprecation warning until 1.0, so existing migration files keep running. Replace `clickhouse_alembic` with `ch_migrate` in them when convenient, and run `ch-migrate upgrade-env` to refresh `migrations/env.py`.
+The command is `ch-migrate`, the PyPI package is `ch-migrate-cli` (PyPI treats `ch-migrate` as the same name as the existing, unrelated `chmigrate`), and migrations import from `ch_migrate`. Versions up to 0.4.1 were published as `clickhouse-alembic` with the import package `clickhouse_alembic`; that import still works with a deprecation warning until 1.0, so existing migration files keep running. Replace `clickhouse_alembic` with `ch_migrate` in them when convenient. For `migrations/env.py`, run `ch-migrate upgrade-env` only if you never edited it; if you did (for example to add connection settings), change its `clickhouse_alembic` imports to `ch_migrate` by hand instead, because `upgrade-env` replaces the whole file.
 
 To switch an existing install, remove the old package first, because both install the `ch-migrate` command and the `clickhouse_alembic` folder: `uv tool uninstall clickhouse-alembic && uv tool install ch-migrate-cli`, or `pip uninstall clickhouse-alembic && pip install ch-migrate-cli`. In a project that lists `clickhouse-alembic` as a dependency, replace it with `ch-migrate-cli`.
 
 This README describes the source checkout, which may be ahead of PyPI. To try an unreleased checkout locally, run `uv tool install .` in the repository. For development without installing a global tool, use `uv run --locked ch-migrate`.
 
+## Upgrade from 0.x
+
+0.6 requires Python 3.10+; 0.5.x is the last line supporting Python 3.9.
+A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades to 0.6; pin
+`ch-migrate-cli==0.5.1` until each project has run the command below.
+After upgrading the package, run this in each migration project:
+
+```bash
+ch-migrate upgrade-env
+```
+
+`init` and `upgrade-env` write a thin version-2 environment that delegates to the
+package. The old file is saved as `migrations/env.py.bak`; repeating the command
+on the current shim leaves that backup intact. Review any old customizations.
+Existing Python migrations and version tables continue to work without edits.
+`up`, `down`, and `history` refuse old environments before connecting and tell
+you to run `upgrade-env`; `status` still reports, so CI jobs that use it as a
+non-blocking reporter keep passing.
+
+One HTTP session spans a migration run, including later revisions in that run.
+A standalone `SET` therefore carries into following revisions. Prefer a
+statement's `SETTINGS` clause when the change should apply only to that query.
+`session_timeout` in an environment or `defaults` sets the idle timeout in
+seconds (default `1800`). If that session expires, the next request fails rather
+than silently recreating a session with default settings.
+
 ## Quick start
 
 Use a dedicated ClickHouse test server. The example uses local HTTP; replace the host and port with your server's address. For HTTPS, set `secure: true` and the HTTPS port (usually `8443`). Never run a trial migration against a shared or production database.
@@ -176,7 +201,141 @@ project/
 
 Keyword arguments to `run_sql` add or override substitutions. All other braces remain literal, including JSON and ClickHouse parameters such as `{id:UInt64}`. Doubled braces are not format escapes. Write statements that are safe to repeat where possible, such as `CREATE ... IF NOT EXISTS` and `DROP ... IF EXISTS`.
 
-To render without executing, set `CH_ENVIRONMENT` and run `alembic upgrade head --sql`. Existing projects need `ch-migrate upgrade-env` for the offline version-table and literal-rendering fixes. The package requires Alembic 1.14 or later for that extension point.
+To render without executing, set `CH_ENVIRONMENT` and run `alembic upgrade head --sql`. Existing projects need `ch-migrate upgrade-env` to use the packaged environment. The package requires Alembic 1.18 or later and the official `clickhouse-connect[alembic]` integration.
+
+### The version table
+
+New `alembic_version` tables use `ORDER BY version_num`. The runtime inspects
+`system.databases.engine` and the environment's `cluster` setting:
+
+| Deployment | New version table |
+|---|---|
+| Shared database on ClickHouse Cloud | `MergeTree`; Cloud supplies its shared implementation |
+| Self-hosted `Replicated` database | `ReplicatedMergeTree()` with server-default arguments |
+| Atomic database with `cluster` configured | `ReplicatedMergeTree` created `ON CLUSTER`, with a Keeper path containing the database name and shard macro |
+| Single server | `MergeTree` |
+
+The official Alembic implementation advances a version by **inserting the new
+row, then deleting the old row with `mutations_sync = 2`**. It does not issue an
+asynchronous version UPDATE. If the process stops while the delete is pending,
+both rows remain and Alembic refuses their overlapping history rather than
+repeating the migration. Let the version mutation finish, then retry. A
+server-reported failed version mutation stops the run with its reason; ch-migrate
+never kills that mutation.
+
+Before reading replicated heads, the tool catches up replicated database
+metadata and inserted version parts. The table barrier uses
+[`SYSTEM SYNC REPLICA ... LIGHTWEIGHT`](https://clickhouse.com/docs/reference/statements/system#sync-replica),
+which does not wait for held mutation tasks. Failure checks cover reachable
+hosts in the configured cluster, or the
+[Replicated database's automatically named cluster](https://clickhouse.com/docs/reference/engines/database-engines/replicated).
+An unavailable host is not evidence that its state is healthy.
+
+Self-hosted replicas need Keeper and suitable `{shard}`/`{replica}` macros.
+Configure authentication for a Replicated database's automatic cluster through
+its `collection_name` setting; the named collection uses `cluster_username`,
+`cluster_password`, and optionally `cluster_secret`/`cluster_secure_connection`.
+Keep those values in server-side secret configuration, not migration files.
+Provision the migration user on each node. Re-run bootstrap when upgrading to
+grant the system-table reads and available `CLUSTER`, remote-read, and replica
+synchronization privileges used by these checks.
+
+Existing 0.4.1 `ReplacingMergeTree ORDER BY updated` tables are **never converted
+automatically**. `upgrade-env` remains offline: it prints a conditional advisory
+instead of probing every configured environment. `status ENV` performs the live
+check and warns when existing state is not replicated but the deployment is.
+Do not route migrations across nodes until that state is reconciled.
+
+For a manual conversion:
+
+1. Stop every migration runner. Back up the old table's DDL and rows on each
+   node, and inspect unfinished version mutations.
+2. Reconcile one authoritative set of completed revision heads with the
+   revision graph. Do not infer it from the newest timestamp or blindly use
+   `FINAL`: the old timestamp key can collapse heads written in the same second.
+3. Keep the old tables as backups. On an Atomic cluster, rename each existing
+   old table on its own node, then create the replacement on the cluster.
+   In a Replicated database, issue schema changes once and let its DDL log
+   propagate them.
+4. Use `MergeTree ORDER BY version_num` on a single server, or the replicated
+   engine from the table above. For an Atomic cluster, use a fresh shared Keeper
+   path containing the database and the appropriate replica macros.
+5. Copy the reconciled, distinct heads **once** into the empty replacement,
+   synchronize replicas, and compare `status` through every routing endpoint.
+   Resume runners only after they agree. Keep backups until the result is verified.
+
+Offline SQL cannot inspect a live database engine: it uses configured `cluster`
+information or the single-server/Cloud default for version-table DDL. Review
+that DDL before using it for a self-hosted Replicated database. Cloud qualification
+is separate from the local suite; the local replicated harness has one shard.
+
+### Re-runnable migrations
+
+`up` statically checks pending upgrade statements before Alembic executes any of
+them. Idempotency and standalone-SET errors refuse the whole run and print the source file, line,
+statement, and suggested fix. There is no command-line bypass; `--skip-mv-check`
+does not skip this gate. Other findings are warnings in `up`; standalone `lint`
+retains their configured severities.
+
+Static lint rejects a standalone `SET` when `migrations/env.py` lacks the
+`CH_MIGRATE_ENV_VERSION = 2` marker: that statement would be ignored on the old
+connection. Run `ch-migrate upgrade-env`, or put the setting in the relevant
+statement's `SETTINGS` clause. Version-2 projects allow `SET`; `SETTINGS` clauses
+and the word `SET` inside string literals are not flagged. `standalone_set` is
+a gate rule whose severity cannot be lowered.
+
+Use `IF NOT EXISTS` for `CREATE` and `ALTER ... ADD`, and `IF EXISTS` for `DROP`,
+`ALTER ... DROP`, `RENAME COLUMN`, and table, dictionary, and database renames.
+This includes tables, views, materialized views, dictionaries, databases, users,
+roles, row policies, settings profiles, quotas, functions, and named collections.
+Column, index, projection, and constraint changes are checked separately.
+`CREATE OR REPLACE` is accepted. `ATTACH` and `DETACH` of an object need their
+matching `IF` form.
+
+Each finding says how to make its statement safe to run twice, using the
+statement's own names: the exact `IF` form, or what a retry would do and the
+usual fix. For example, a backfill into a table that only its migration writes
+can start with `TRUNCATE`, and `ATTACH PARTITION ... FROM` can usually be
+`REPLACE PARTITION ... FROM`.
+
+Inserts, exchanges, updates/deletes, `ATTACH PARTITION ... FROM`,
+`MOVE PARTITION ... TO TABLE`, and attaching or detaching a single `PART` need a
+reasoned waiver. Put it directly above the statement, without a separating blank
+line:
+
+```sql
+-- ch-migrate: allow-non-idempotent This backfill targets a reviewed deduplicating table
+INSERT INTO {db}.target SELECT id FROM {db}.source;
+```
+
+For inline Python SQL, use the same directive in a `#` comment directly above
+`op.execute`. Empty reasons remain errors. Waivers print as INFO with their
+reasons in `lint` and `up`; they do not make the statement idempotent.
+`MODIFY`, `MATERIALIZE`, `CLEAR COLUMN`, `TRUNCATE`, `OPTIMIZE`, `SYSTEM`, grants,
+revokes, `REPLACE PARTITION`, attaching or detaching a whole partition, and
+comments are not flagged by this rule.
+The reviewable classification corpus is `tests/corpus/idempotency.yaml`.
+
+**Recovery:** fix the cause of a partial failure, then run `ch-migrate up ENV`
+again. Earlier idempotent statements can run again; the version advances only
+when the revision completes. On ClickHouse Cloud, a column that the failed attempt
+dropped or renamed stays locked until that change's mutation finishes: a retry
+before then fails with "affected by mutation ... which is not finished yet", even
+with `IF EXISTS`. Wait for it in `system.mutations`, then run `up` again. Before retrying waived operations, review what the
+earlier attempt actually changed.
+
+**Existing history:** `upgrade-env` records the current local script head(s) in
+`config.yaml` as `lint.gate_baseline`, preserving YAML comments. It records
+script history, not the deployed database head. The baseline and its ancestors
+are exempt in every environment, including where they have not yet run.
+Multiple heads are stored as a list. `init` writes no baseline, so new projects
+gate every revision.
+
+Review and commit the baseline diff. Running `upgrade-env` again updates it to
+the current script heads; do not use that operation to hide new findings.
+Static `lint` checks revisions after the baseline, and live `lint ENV` intersects
+that scope with pending revisions. Lowering a gate rule through `lint.rules`
+is an error: use a reviewed baseline or an explicit statement waiver instead.
 
 ### Irreversible migrations
 
@@ -226,6 +385,9 @@ def downgrade():
 
 `new --exchange --table NAME` generates the existing shadow-table, copy, exchange, and drop scaffold. Coordinate or pause writers: this copy-and-swap pattern alone does not preserve inserts arriving during the copy. It is not an online-rebuild guarantee. Review the generated SQL and column mapping before applying it. The scaffold is marked irreversible because it drops the old table.
 
+The generated copy and exchange statements require explicit, reasoned waivers
+before `up` accepts them. The generator does not waive them automatically.
+
 For a controlled change, the underlying pattern is:
 
 ```sql
@@ -273,7 +435,7 @@ Example: `ch-migrate new dev add_status --table logs`
 
 ### `up`
 
-`ch-migrate up ENV [-r REV] [--skip-mv-check] [--verbose]` applies migrations to `head` by default, printing one line per migration. `-r/--revision` selects a target. `--skip-mv-check` bypasses materialized-view declaration validation; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
+`ch-migrate up ENV [-r REV] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
 
 Example: `ch-migrate up dev --revision abc123`
 
@@ -285,7 +447,7 @@ Example: `ch-migrate down dev --revision base`
 
 ### `status`
 
-`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Exits 1 if it cannot reach the database.
+`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Status is a report: it exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check.
 
 Example: `ch-migrate status dev`
 
@@ -297,7 +459,7 @@ Example: `ch-migrate history dev`
 
 ### `lint`
 
-`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks every revision statically without credentials or a connection. With an environment, it checks only pending revisions and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings alone do not.
+`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings and waiver INFO lines alone do not.
 
 Example: `ch-migrate lint`
 
@@ -335,7 +497,7 @@ Example: `ch-migrate rebase dev --onto abc123 --dry-run`
 
 ### `upgrade-env`
 
-`ch-migrate upgrade-env` replaces `migrations/env.py` with the installed version and backs up the old file as `env.py.bak`. No command-specific options. Review and reapply local customizations from the backup.
+`ch-migrate upgrade-env` replaces `migrations/env.py` with the installed version, backs up the old file as `env.py.bak`, and records current script heads as `lint.gate_baseline` while preserving YAML comments. No command-specific options. Review the baseline diff. It does not merge: any local customizations (connection settings, session pins, hooks) are dropped from the new file. Reapply them from the backup, or skip `upgrade-env` and edit a customized `env.py` by hand. An unchanged shim is not backed up again.
 
 Example: `ch-migrate upgrade-env`
 
@@ -401,7 +563,7 @@ Only configure the dictionary hook when that dictionary exists at every revision
 
 ### Lint configuration
 
-Set rule severities to `error`, `warn`, or `off`. `mv_validation_cutoff` can exclude older revisions from materialized-view declaration checks.
+Non-gate rule severities can be `error`, `warn`, or `off`. Gate rules must remain `error`. `mv_validation_cutoff` can exclude older revisions from materialized-view declaration checks.
 
 ```yaml
 lint:
@@ -409,7 +571,7 @@ lint:
   mv_validation_cutoff: "2026-01-01"
   rules:
     destructive_changes: warn
-    idempotency: warn
+    idempotency: error
     reserved_words: warn
 ```
 
@@ -417,7 +579,7 @@ Review findings rather than treating a successful command as a guarantee that a
 
 ### Bootstrap roles and Cloud notes
 
-Bootstrap creates `{project}_migration_role` for schema/data operations and introspection, including explicit `system.grants` access. Optional users add `{project}_readonly_role` (SELECT/SHOW) and `{project}_dict_role` (dictionary sources). Bootstrap uses explicit grants rather than `GRANT ALL` for Cloud compatibility.
+Bootstrap creates `{project}_migration_role` for schema/data operations and introspection, including explicit `system.grants`, `system.databases`, `system.tables`, and `system.mutations` access. It grants available cluster, remote-read, and synchronization rights through `CURRENT GRANTS`. Optional users add `{project}_readonly_role` (SELECT/SHOW) and `{project}_dict_role` (dictionary sources). Bootstrap uses explicit grants rather than `GRANT ALL` for Cloud compatibility.
 
 Use standard table engine names such as `MergeTree` and `ReplacingMergeTree`; ClickHouse Cloud supplies its shared variants. Cloud usually uses HTTPS port `8443`; local HTTP usually uses `8123`.
 
@@ -435,10 +597,62 @@ Run the opt-in real-server suite:
 uv run --locked pytest -q -m integration
 ```
 
-The fixture starts `clickhouse/clickhouse-server:26.3` in its own `chm-it-*` container on a random loopback port. Each test uses a separate database. Finalizers clean up on success, failure, and handled interrupts; a forced process kill cannot run finalizers. Docker-unavailable runs skip with a reason.
+The single-server fixture starts `clickhouse/clickhouse-server:26.3` in its own `chm-it-*` container on a random loopback port. Each test uses a separate database. Finalizers clean up on success, failure, and handled interrupts; a forced process kill cannot run finalizers. Docker-unavailable runs skip with a reason.
 
 `CH_MIGRATE_IT_IMAGE` overrides the image tag. `CH_MIGRATE_IT_URL` selects a dedicated test server instead of starting Docker. It is an HTTP(S) URL with credentials supplied only through the environment. Tests create and drop databases there: never select a shared or production server, and never commit the URL.
 
+For the single-server suite only:
+
+```bash
+uv run --locked pytest -q -m "integration and not cluster"
+```
+
+For the replicated harness:
+
+```bash
+uv run --locked --python 3.12 pytest -q -m "integration and cluster" -k harness
+```
+
+`clickhouse_cluster` owns a Docker network and two nodes, with Keeper embedded
+in node 1. `it_cluster` has one shard and two replicas. `cluster_project` creates
+an **Atomic database on both nodes with `ON CLUSTER`**; replicated tables use
+`ReplicatedMergeTree` with a shared Keeper path and per-node replica macros.
+The fixture exposes `clients[1]` and `clients[2]`, plus `stop_node(2)` and
+`start_node(2)`. Read `clients[2]` again after a restart; its connection is renewed.
+
+Tests prove row replication and both hosts' distributed-DDL completion. The
+node-down scenario uses a five-second distributed-DDL timeout, inspects the
+unfinished host, then proves catch-up after restart. Its HTTP response is
+buffered with `wait_end_of_query=1` so the timeout is not obscured by a truncated
+chunked response.
+
+Cluster fixtures use the same image-tag override, remove their own containers,
+anonymous volumes, and network, and never operate on external servers. Cluster
+tests skip when `CH_MIGRATE_IT_URL` is set. The integration CI job runs cluster
+tests on Python 3.12 only, and single-server tests on 3.10 and 3.14.
+
+### Against a ClickHouse Cloud test service
+
+Point `CH_MIGRATE_IT_URL` at a dedicated Cloud test service
+(`https://<user>:<url-encoded password>@<service>:8443`), loading the password
+from your secret store at run time. Use a user that can create and drop `it_*`
+databases, users and roles. The fixture waits up to 300 seconds for an idle
+service to wake.
+
+```bash
+uv run --locked pytest -q -m "integration and not cluster"   # the full single-server suite
+uv run --locked pytest -q -m "integration and cloud"         # Cloud-only checks
+```
+
+Tests marked `cloud` run only when the URL is a Cloud service (Shared
+databases); they check that a migration's `SET` reaches every later statement
+whichever replica serves it, and version-table round trips on SharedMergeTree.
+Cluster tests skip. Cloud grants no user `SYSTEM STOP MERGES`, so tests that
+hold a mutation pending skip with that reason; the self-hosted suite still
+covers them. Each test drops its own database. After an interrupted run,
+`scripts/cloud_sweep.py [--dry-run]` drops `it_*` databases no query has
+touched for 24 hours, with their test users and roles.
+
 ## License
 
 MIT License — see [LICENSE](LICENSE).