ch-migrate-cli 0.6.0__tar.gz → 0.8.0__tar.gz

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.8.0}/.gitignore

@@ -66,4 +66,6 @@ tmp/
 temp/
 
 examples/
+# Documentation examples are reviewed source, not local sample projects.
+!docs/examples/
 lambda-exporter/

{ch_migrate_cli-0.6.0 → ch_migrate_cli-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ch-migrate-cli
-Version: 0.6.0
+Version: 0.8.0
 Summary: SQL-first ClickHouse schema migrations across environments: the ch-migrate command
 Project-URL: Homepage, https://github.com/DRYCodeWorks/ch-migrate
 Project-URL: Repository, https://github.com/DRYCodeWorks/ch-migrate
@@ -32,6 +32,7 @@ Requires-Dist: ruamel-yaml<0.20,>=0.19.1
 Provides-Extra: dev
 Requires-Dist: black>=23.0.0; extra == 'dev'
 Requires-Dist: isort>=5.12.0; extra == 'dev'
+Requires-Dist: jsonschema>=4.18; extra == 'dev'
 Requires-Dist: mypy>=1.5.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
 Requires-Dist: pytest>=7.4.0; extra == 'dev'
@@ -65,8 +66,19 @@ This README describes the source checkout, which may be ahead of PyPI. To try an
 
 ## Upgrade from 0.x
 
-0.6 requires Python 3.10+; 0.5.x is the last line supporting Python 3.9.
-A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades to 0.6; pin
+**From 0.7:** 0.8 needs no `upgrade-env`, and nothing changes unless you pass
+`--json` (see [Drift detection in CI](#drift-detection-in-ci)). A loose requirement
+such as `ch-migrate-cli` upgrades to 0.8; pin `ch-migrate-cli==0.7.0` to stay on 0.7.
+
+**From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
+`ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
+so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
+`SELECT ON system.*`, such as on ClickHouse Cloud, add the grants listed in the
+[CHANGELOG](CHANGELOG.md). A loose requirement such as `ch-migrate-cli` upgrades to
+0.7; pin `ch-migrate-cli==0.6.0` to stay on 0.6.
+
+**From 0.5:** 0.6 and later require Python 3.10+; 0.5.x is the last line supporting Python 3.9.
+A loose requirement such as `ch-migrate-cli` or `~=0.5` upgrades past 0.5; pin
 `ch-migrate-cli==0.5.1` until each project has run the command below.
 After upgrading the package, run this in each migration project:
 
@@ -269,13 +281,111 @@ information or the single-server/Cloud default for version-table DDL. Review
 that DDL before using it for a self-hosted Replicated database. Cloud qualification
 is separate from the local suite; the local replicated harness has one shard.
 
+### Migrations that wait
+
+`up` doesn't record a migration as applied until the work its statements started
+has finished. After each statement that can write, it polls `system.mutations` on
+the same checked ClickHouse session, so earlier `SET` statements stay in effect.
+`op.execute`, raw `op.get_bind()` execution and `run_sql` all pass through that
+check. Upgrades run in the CLI process, so killing it can't leave a child Alembic
+runner advancing revisions. `down` waits the same way.
+
+```bash
+ch-migrate up dev                 # No default waiting timeout
+ch-migrate up dev --timeout 120   # One waiting budget across the invocation
+```
+
+What `up` waits for:
+
+- **Every unfinished mutation in the project database,** including one another
+  writer started. Before the first migration runs, `up` waits for any already
+  queued, and a failed one blocks the run. The version table's own mutations are
+  left to [its bookkeeping](#the-version-table).
+- **Each replica that has to apply it.** Zero parts left isn't completion on its
+  own: the mutation must report `is_done`. A ReplicatedMergeTree mutation must be
+  finished on every replica, and a replica that hasn't received it yet is still
+  pending. Configure `cluster` for self-hosted replicated tables, or use a
+  Replicated database's automatic cluster; without either, `up` refuses a table
+  with more than one replica. SharedMergeTree on ClickHouse Cloud reports one
+  shared mutation.
+- **`ON CLUSTER` DDL on every host.** `up` sends these statements with
+  `distributed_ddl_output_mode = 'throw'`, so ClickHouse answers once every host
+  has run the statement, and a host that fails or doesn't finish in time fails the
+  migration with ClickHouse's message. The limit is 270 seconds, below the
+  client's read timeout, or what's left of `--timeout` if that's less. Mutations
+  the statement starts are then waited for as above.
+
+Statements ClickHouse already holds until they finish, such as a type-changing
+`MODIFY COLUMN`, a `MODIFY TTL` that materializes, or a lightweight `DELETE`,
+still hold their request as they did in 0.6.
+
+While it waits, `up` shows progress on stderr. In a terminal, each mutation gets a
+live bar weighted by bytes, with its parts and an estimate of the time left:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  ━━━━━━━━━━━━━━━━━━━━╺━━━━━━━━━━━━━━━━━  18.2 / 41.0 GiB · 23/41 parts · 1:12 · ~1:30 left
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 41.0 GiB · 2:03
+```
+
+Self-hosted replicated tables get one row per replica. When output isn't a
+terminal (CI logs, pipes, coding agents), `up` prints plain lines instead: one
+when a wait starts, one at each 10% of the parts or every 30 seconds, and one
+when it finishes:
+
+```text
+→ Applying 3f9c2a1b  backfill_digest
+  waiting on mutation_7.txt (demo_dev.events): 0/41 parts
+  waiting on mutation_7.txt (demo_dev.events): 5/41 parts · 0:14
+✓ 3f9c2a1b  backfill_digest  mutation_7.txt · 41 parts · 2:03
+```
+
+Part sizes come from `system.parts`. If the migration role can't read it, the bar
+counts parts only and says so once; progress never changes whether `up` succeeds.
+
+**When `up` stops short.** After a timeout, Ctrl-C, dropped connection, killed
+process or failed statement, ClickHouse keeps working and the migration isn't
+recorded. `up` and `down` note each migration in `_ch_migrate_in_progress` when it
+starts and clear the note once its version is recorded, so the next run knows the
+migration was interrupted. That run waits for unfinished work, then:
+
+- **Runs the migration again on its own** when every statement in it passes the
+  idempotency gate without a waiver, so a repeat leaves the same result.
+- **Stops before sending anything** when the migration has a waived statement or
+  SQL the gate can't read, because that statement may already have applied. It
+  names each one. Check what the earlier run changed, fix the SQL if needed, then
+  name the revision: `ch-migrate up ENV --rerun-interrupted REV` (or `down`), or
+  `-x rerun_interrupted=REV` with `alembic`. Naming it accepts the repeat.
+
+Every `UPDATE`, `DELETE` and `INSERT ... SELECT` needs a waiver, so an interrupted
+backfill always takes that second path.
+
+**Failed mutation.** ClickHouse's `latest_fail_reason` stops the run, and `up`
+prints the scoped `KILL MUTATION` statement for an operator; ch-migrate never
+runs it. ClickHouse keeps retrying a failed mutation, and later mutations on that
+table queue behind it. If the cause is temporary, such as memory or disk, fix it
+and run `up` again. If the statement itself is wrong, kill the mutation, fix the
+SQL and run `up` again, naming the revision with `--rerun-interrupted` if it's
+waived. Killing a mutation doesn't undo the parts it already rewrote.
+
+Run one migration runner per database, using a CI concurrency key or an external
+lock; the in-progress notes aren't a lock. Re-run `bootstrap` when upgrading a
+restricted migration user to obtain the `system.replicas` and `system.parts` grants.
+
 ### Re-runnable migrations
 
 `up` statically checks pending upgrade statements before Alembic executes any of
 them. Idempotency and standalone-SET errors refuse the whole run and print the source file, line,
 statement, and suggested fix. There is no command-line bypass; `--skip-mv-check`
 does not skip this gate. Other findings are warnings in `up`; standalone `lint`
-retains their configured severities.
+retains their configured severities. A direct `alembic upgrade` runs the same gate
+from `env.py` before its first migration.
+
+The gate reads SQL in `run_sql` files and in literal strings, f-strings and
+`text("...")` passed to `op.execute`, or to `execute` and `exec_driver_sql` on a
+connection such as `op.get_bind()`. SQL that only exists at run time, such as a
+variable, a built string or `run_sql(name)`, can't be checked, so it needs a
+reasoned waiver like any other statement the gate can't vouch for.
 
 Static lint rejects a standalone `SET` when `migrations/env.py` lacks the
 `CH_MIGRATE_ENV_VERSION = 2` marker: that statement would be ignored on the old
@@ -309,7 +419,7 @@ INSERT INTO {db}.target SELECT id FROM {db}.source;
 ```
 
 For inline Python SQL, use the same directive in a `#` comment directly above
-`op.execute`. Empty reasons remain errors. Waivers print as INFO with their
+the call that sends it. Empty reasons remain errors. Waivers print as INFO with their
 reasons in `lint` and `up`; they do not make the statement idempotent.
 `MODIFY`, `MATERIALIZE`, `CLEAR COLUMN`, `TRUNCATE`, `OPTIMIZE`, `SYSTEM`, grants,
 revokes, `REPLACE PARTITION`, attaching or detaching a whole partition, and
@@ -318,11 +428,12 @@ The reviewable classification corpus is `tests/corpus/idempotency.yaml`.
 
 **Recovery:** fix the cause of a partial failure, then run `ch-migrate up ENV`
 again. Earlier idempotent statements can run again; the version advances only
-when the revision completes. On ClickHouse Cloud, a column that the failed attempt
-dropped or renamed stays locked until that change's mutation finishes: a retry
-before then fails with "affected by mutation ... which is not finished yet", even
-with `IF EXISTS`. Wait for it in `system.mutations`, then run `up` again. Before retrying waived operations, review what the
-earlier attempt actually changed.
+when the revision completes. A dropped or renamed column stays locked until that
+change's mutation finishes, and `up` waits for it before running the next
+statement, so a retry after a later failure doesn't hit "affected by mutation ...
+which is not finished yet". If `up` was stopped while it waited, run it again: it
+waits for that mutation, then runs the revision again, unless the revision has a
+statement the gate can't vouch for (see below).
 
 **Existing history:** `upgrade-env` records the current local script head(s) in
 `config.yaml` as `lint.gate_baseline`, preserving YAML comments. It records
@@ -409,12 +520,68 @@ def upgrade():
     create_dictionary("history/dictionaries/dict_users/001_create.sql")
 ```
 
+## Drift detection in CI
+
+After applying and verifying migrations in an environment, capture its expected
+schema:
+
+```bash
+ch-migrate snapshot dev
+mv migrations/sql/snapshots/20261002_120000 migrations/sql/snapshots/dev
+ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/dev --json
+```
+
+`snapshot` writes a new timestamped directory every time; replace the example
+timestamp with the one it reports. Give each environment's snapshot a fixed name,
+such as `snapshots/dev`, so `CH_MIGRATE_SNAPSHOT_DIR` never changes and the next
+snapshot shows up in review as a diff against this one. Review and commit that
+directory alongside the migration head. Do not regenerate the snapshot
+automatically in the drift job: doing so would accept an out-of-band change as the
+new expected schema. Re-snapshot after an intentional migration or an explicitly
+reviewed reconciliation, not merely to turn a failed check green. To re-snapshot,
+run `snapshot` again, delete the old `snapshots/dev`, and move the new directory
+into its place, so the commit also shows objects that were dropped.
+
+Copy [the GitHub Actions example](docs/examples/github-actions-drift.yml) into the
+consumer repository's `.github/workflows/`. It runs on a daily schedule and
+same-repository pull requests; fork PRs are excluded because they have no database
+secret. It does not use `pull_request_target`. All actions are pinned by commit.
+The checkout must contain `config.yaml` and the committed snapshot, and the runner
+must be able to reach the selected server.
+
+Configure these repository variables:
+
+| Variable | Value |
+|---|---|
+| `CH_MIGRATE_PACKAGE` | A pinned `ch-migrate-cli` requirement, such as `ch-migrate-cli==0.8.0`. 0.8.0 is the first release with `diff --json`; there is no fallback to an older one. |
+| `CH_MIGRATE_ENV` | An environment name in `config.yaml`, such as `dev`. |
+| `CH_MIGRATE_SNAPSHOT_DIR` | That environment's committed snapshot directory. |
+
+Set the repository secret `DRIFT_MIGRATION_PASSWORD` to the password for that
+environment's configured user. Prefer a dedicated schema-inspection account,
+not an admin account. The job writes a private `.env.local` without printing the
+value and removes the file even when comparison fails.
+
+`diff --json` exits **0** for matching schemas, **1** for drift, and **2** for a
+configuration/connection error. The workflow saves `drift.json`, uploads it even
+on comparison failure, then fails the job for either nonzero result. The report
+names changed tables and fields. ch-migrate's own test suite runs the example's
+shell steps against a real ClickHouse server, installing from the source checkout,
+for both a matching and a drifted schema.
+
+Without `--snapshot-dir`, `diff` sorts entries in `migrations/sql/snapshots/` by
+name and chooses the last one. It does not select a snapshot by environment or
+recorded migration revision. Use an explicit path in CI, especially when one
+repository holds snapshots for several environments.
+
 ## Command reference
 
 Every command accepts `--help`. Top-level `ch-migrate --version` reports the installed package version. `ENV` below names an entry in `config.yaml`.
 
 Output lines start with `→` for a step, `✓` for a result, `!` for a warning and `✗` for an error; warnings and errors go to stderr. Colour is dropped when output is not a terminal or `NO_COLOR` is set, and lines are never wrapped, so paths and SQL can be copied or grepped.
 
+`status`, `history`, `lint`, and `diff` accept `--json`. JSON mode writes one document to stdout; diagnostics go to stderr. Every document includes `"schema_version": 1` and `"command"`. The format can still change before 1.0; `schema_version` says which one a document follows. Errors include an `error` string rather than inventing a successful empty result. The schemas linked below use JSON Schema draft 2020-12.
+
 ### `init`
 
 `ch-migrate init [PATH] [-n NAME]` initializes the current directory by default. `-n/--name` sets the project name; otherwise it uses the directory name.
@@ -435,7 +602,7 @@ Example: `ch-migrate new dev add_status --table logs`
 
 ### `up`
 
-`ch-migrate up ENV [-r REV] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
+`ch-migrate up ENV [-r REV] [--timeout SECONDS] [--skip-mv-check] [--verbose]` applies migrations to `head` by default after the idempotency gate passes, printing one line per migration. `-r/--revision` selects a target. `--timeout` is a finite positive waiting limit; omitting it waits without a deadline. `--skip-mv-check` skips nonblocking materialized-view declaration checks, not the idempotency gate; use it only after reviewing those findings. If a migration fails, `up` names it, the SQL file, the statement and its line, and ClickHouse's error; `--verbose` adds the Python traceback.
 
 Example: `ch-migrate up dev --revision abc123`
 
@@ -447,22 +614,53 @@ Example: `ch-migrate down dev --revision base`
 
 ### `status`
 
-`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Status is a report: it exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check.
+`ch-migrate status ENV [--json]` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. Status is a report: human output exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check. JSON mode uses nonzero exits for unreachable or pending state.
 
 Example: `ch-migrate status dev`
 
+With `--json`, [the status schema](docs/schemas/status.schema.json) includes
+`database`, `current_heads`, `script_heads`, `pending`, `applied`, and `at_head`.
+`applied` includes ancestors resolved through the revision graph, not just stored
+heads. JSON exits 0 when the head sets match exactly, 1 when pending or diverged,
+and 2 when configuration or database state cannot be read. Human-mode exits are
+unchanged.
+
+```bash
+ch-migrate status dev --json | jq -e .current_heads
+# Check the actual head set, not the last element of applied:
+ch-migrate status dev --json | jq -e '.current_heads == ["PINNED_REVISION"]'
+```
+
+Replace `PINNED_REVISION` with the required revision. In a shell pipeline, enable
+`set -o pipefail` if the caller must also preserve ch-migrate's exit code.
+
 ### `history`
 
-`ch-migrate history ENV` displays the revision graph and applied state. No command-specific options.
+`ch-migrate history ENV [--json]` displays the revision graph and applied state.
 
 Example: `ch-migrate history dev`
 
+`ch-migrate history dev --json` follows [the history schema](docs/schemas/history.schema.json).
+Each entry in `revisions` includes its revision, all `down_revisions` (including
+merge parents), description, create date, path, applied status, and irreversible
+reason. A reversible revision has `irreversible: null`. If database state is
+unavailable, `applied` is null, the document includes `error`, and JSON exits 2.
+An unknown database head also makes applied status unknown instead of claiming
+that every local revision is unapplied.
+
 ### `lint`
 
-`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings and waiver INFO lines alone do not.
+`ch-migrate lint [ENV] [--json]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. Errors exit nonzero; warnings and waiver INFO lines alone do not.
 
 Example: `ch-migrate lint`
 
+`ch-migrate lint --json` follows [the lint schema](docs/schemas/lint.schema.json).
+`findings` include rule, severity, source file/line, message, `blocking`, and
+`waived`. `blocking` means the finding currently blocks `up`; a waived finding
+has its written reason and does not block. `counts` reports total, error, warning,
+info, blocking, and waived findings. JSON and human output use the same exit
+codes: 1 for lint/configuration errors, otherwise 0.
+
 Findings name the project-relative SQL file and statement line. Inline Python SQL
 points to its `op.execute` call. Extraction reads `run_sql`/`read_sql` file
 references and literal or f-string `op.execute` arguments without importing
@@ -479,10 +677,16 @@ Example: `ch-migrate deps dev --validate migrations/sql/history/tables/logs/chan
 
 ### `diff`
 
-`ch-migrate diff ENV [-s PATH]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift or an execution error.
+`ch-migrate diff ENV [-s PATH] [--json]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift. JSON errors exit 2; human-mode execution errors retain exit 1.
 
 Example: `ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/20261002_120000`
 
+`ch-migrate diff dev --json` follows [the diff schema](docs/schemas/diff.schema.json).
+`objects` identifies each object's type, name, status, and structural `details`
+(field, local value, remote value, and message). `in_sync` is true only when every
+object matches. Use the exit code as the drift gate; an error document does not
+contain a fabricated `in_sync: true`.
+
 ### `snapshot`
 
 `ch-migrate snapshot ENV [-e GLOB] [-f GLOB]` writes CREATE statements to a timestamped snapshot directory. `-e/--exclude` and `-f/--filter` accept repeated glob patterns for excluded and included objects.
@@ -549,7 +753,13 @@ The legacy `CH_<ENV>_PASSWORD` remains supported. Never commit credentials or pa
 
 ### Hooks
 
-Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the migration batch; `post_migrate` runs after each revision. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do not put secrets in it.
+Top-level hooks run SQL on the migration connection. `pre_migrate` runs before the
+migration batch; `post_migrate` runs after each revision body **before its version
+is completed**, so hook mutations cannot escape the waiting barrier. A resumed
+upgrade reuses its unfinished batch's receipts instead of repeating completed
+hook writes. `{db}` is substituted. Hooks execute as SQLAlchemy text, not through
+the SQL-file splitter; supply one statement per entry. Hook SQL is logged, so do
+not put secrets in it.
 
 ```yaml
 hooks: