ch-migrate-cli 0.7.0__tar.gz → 0.8.0__tar.gz

{ch_migrate_cli-0.7.0 → ch_migrate_cli-0.8.0}/.gitignore

@@ -66,4 +66,6 @@ tmp/
 temp/
 
 examples/
+# Documentation examples are reviewed source, not local sample projects.
+!docs/examples/
 lambda-exporter/

{ch_migrate_cli-0.7.0 → ch_migrate_cli-0.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: ch-migrate-cli
-Version: 0.7.0
+Version: 0.8.0
 Summary: SQL-first ClickHouse schema migrations across environments: the ch-migrate command
 Project-URL: Homepage, https://github.com/DRYCodeWorks/ch-migrate
 Project-URL: Repository, https://github.com/DRYCodeWorks/ch-migrate
@@ -32,6 +32,7 @@ Requires-Dist: ruamel-yaml<0.20,>=0.19.1
 Provides-Extra: dev
 Requires-Dist: black>=23.0.0; extra == 'dev'
 Requires-Dist: isort>=5.12.0; extra == 'dev'
+Requires-Dist: jsonschema>=4.18; extra == 'dev'
 Requires-Dist: mypy>=1.5.0; extra == 'dev'
 Requires-Dist: pytest-cov>=4.1.0; extra == 'dev'
 Requires-Dist: pytest>=7.4.0; extra == 'dev'
@@ -65,6 +66,10 @@ This README describes the source checkout, which may be ahead of PyPI. To try an
 
 ## Upgrade from 0.x
 
+**From 0.7:** 0.8 needs no `upgrade-env`, and nothing changes unless you pass
+`--json` (see [Drift detection in CI](#drift-detection-in-ci)). A loose requirement
+such as `ch-migrate-cli` upgrades to 0.8; pin `ch-migrate-cli==0.7.0` to stay on 0.7.
+
 **From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
 `ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
 so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
@@ -515,12 +520,68 @@ def upgrade():
     create_dictionary("history/dictionaries/dict_users/001_create.sql")
 ```
 
+## Drift detection in CI
+
+After applying and verifying migrations in an environment, capture its expected
+schema:
+
+```bash
+ch-migrate snapshot dev
+mv migrations/sql/snapshots/20261002_120000 migrations/sql/snapshots/dev
+ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/dev --json
+```
+
+`snapshot` writes a new timestamped directory every time; replace the example
+timestamp with the one it reports. Give each environment's snapshot a fixed name,
+such as `snapshots/dev`, so `CH_MIGRATE_SNAPSHOT_DIR` never changes and the next
+snapshot shows up in review as a diff against this one. Review and commit that
+directory alongside the migration head. Do not regenerate the snapshot
+automatically in the drift job: doing so would accept an out-of-band change as the
+new expected schema. Re-snapshot after an intentional migration or an explicitly
+reviewed reconciliation, not merely to turn a failed check green. To re-snapshot,
+run `snapshot` again, delete the old `snapshots/dev`, and move the new directory
+into its place, so the commit also shows objects that were dropped.
+
+Copy [the GitHub Actions example](docs/examples/github-actions-drift.yml) into the
+consumer repository's `.github/workflows/`. It runs on a daily schedule and
+same-repository pull requests; fork PRs are excluded because they have no database
+secret. It does not use `pull_request_target`. All actions are pinned by commit.
+The checkout must contain `config.yaml` and the committed snapshot, and the runner
+must be able to reach the selected server.
+
+Configure these repository variables:
+
+| Variable | Value |
+|---|---|
+| `CH_MIGRATE_PACKAGE` | A pinned `ch-migrate-cli` requirement, such as `ch-migrate-cli==0.8.0`. 0.8.0 is the first release with `diff --json`; there is no fallback to an older one. |
+| `CH_MIGRATE_ENV` | An environment name in `config.yaml`, such as `dev`. |
+| `CH_MIGRATE_SNAPSHOT_DIR` | That environment's committed snapshot directory. |
+
+Set the repository secret `DRIFT_MIGRATION_PASSWORD` to the password for that
+environment's configured user. Prefer a dedicated schema-inspection account,
+not an admin account. The job writes a private `.env.local` without printing the
+value and removes the file even when comparison fails.
+
+`diff --json` exits **0** for matching schemas, **1** for drift, and **2** for a
+configuration/connection error. The workflow saves `drift.json`, uploads it even
+on comparison failure, then fails the job for either nonzero result. The report
+names changed tables and fields. ch-migrate's own test suite runs the example's
+shell steps against a real ClickHouse server, installing from the source checkout,
+for both a matching and a drifted schema.
+
+Without `--snapshot-dir`, `diff` sorts entries in `migrations/sql/snapshots/` by
+name and chooses the last one. It does not select a snapshot by environment or
+recorded migration revision. Use an explicit path in CI, especially when one
+repository holds snapshots for several environments.
+
 ## Command reference
 
 Every command accepts `--help`. Top-level `ch-migrate --version` reports the installed package version. `ENV` below names an entry in `config.yaml`.
 
 Output lines start with `→` for a step, `✓` for a result, `!` for a warning and `✗` for an error; warnings and errors go to stderr. Colour is dropped when output is not a terminal or `NO_COLOR` is set, and lines are never wrapped, so paths and SQL can be copied or grepped.
 
+`status`, `history`, `lint`, and `diff` accept `--json`. JSON mode writes one document to stdout; diagnostics go to stderr. Every document includes `"schema_version": 1` and `"command"`. The format can still change before 1.0; `schema_version` says which one a document follows. Errors include an `error` string rather than inventing a successful empty result. The schemas linked below use JSON Schema draft 2020-12.
+
 ### `init`
 
 `ch-migrate init [PATH] [-n NAME]` initializes the current directory by default. `-n/--name` sets the project name; otherwise it uses the directory name.
@@ -553,22 +614,53 @@ Example: `ch-migrate down dev --revision base`
 
 ### `status`
 
-`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Status is a report: it exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check.
+`ch-migrate status ENV [--json]` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. Status is a report: human output exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check. JSON mode uses nonzero exits for unreachable or pending state.
 
 Example: `ch-migrate status dev`
 
+With `--json`, [the status schema](docs/schemas/status.schema.json) includes
+`database`, `current_heads`, `script_heads`, `pending`, `applied`, and `at_head`.
+`applied` includes ancestors resolved through the revision graph, not just stored
+heads. JSON exits 0 when the head sets match exactly, 1 when pending or diverged,
+and 2 when configuration or database state cannot be read. Human-mode exits are
+unchanged.
+
+```bash
+ch-migrate status dev --json | jq -e .current_heads
+# Check the actual head set, not the last element of applied:
+ch-migrate status dev --json | jq -e '.current_heads == ["PINNED_REVISION"]'
+```
+
+Replace `PINNED_REVISION` with the required revision. In a shell pipeline, enable
+`set -o pipefail` if the caller must also preserve ch-migrate's exit code.
+
 ### `history`
 
-`ch-migrate history ENV` displays the revision graph and applied state. No command-specific options.
+`ch-migrate history ENV [--json]` displays the revision graph and applied state.
 
 Example: `ch-migrate history dev`
 
+`ch-migrate history dev --json` follows [the history schema](docs/schemas/history.schema.json).
+Each entry in `revisions` includes its revision, all `down_revisions` (including
+merge parents), description, create date, path, applied status, and irreversible
+reason. A reversible revision has `irreversible: null`. If database state is
+unavailable, `applied` is null, the document includes `error`, and JSON exits 2.
+An unknown database head also makes applied status unknown instead of claiming
+that every local revision is unapplied.
+
 ### `lint`
 
-`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings and waiver INFO lines alone do not.
+`ch-migrate lint [ENV] [--json]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. Errors exit nonzero; warnings and waiver INFO lines alone do not.
 
 Example: `ch-migrate lint`
 
+`ch-migrate lint --json` follows [the lint schema](docs/schemas/lint.schema.json).
+`findings` include rule, severity, source file/line, message, `blocking`, and
+`waived`. `blocking` means the finding currently blocks `up`; a waived finding
+has its written reason and does not block. `counts` reports total, error, warning,
+info, blocking, and waived findings. JSON and human output use the same exit
+codes: 1 for lint/configuration errors, otherwise 0.
+
 Findings name the project-relative SQL file and statement line. Inline Python SQL
 points to its `op.execute` call. Extraction reads `run_sql`/`read_sql` file
 references and literal or f-string `op.execute` arguments without importing
@@ -585,10 +677,16 @@ Example: `ch-migrate deps dev --validate migrations/sql/history/tables/logs/chan
 
 ### `diff`
 
-`ch-migrate diff ENV [-s PATH]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift or an execution error.
+`ch-migrate diff ENV [-s PATH] [--json]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift. JSON errors exit 2; human-mode execution errors retain exit 1.
 
 Example: `ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/20261002_120000`
 
+`ch-migrate diff dev --json` follows [the diff schema](docs/schemas/diff.schema.json).
+`objects` identifies each object's type, name, status, and structural `details`
+(field, local value, remote value, and message). `in_sync` is true only when every
+object matches. Use the exit code as the drift gate; an error document does not
+contain a fabricated `in_sync: true`.
+
 ### `snapshot`
 
 `ch-migrate snapshot ENV [-e GLOB] [-f GLOB]` writes CREATE statements to a timestamped snapshot directory. `-e/--exclude` and `-f/--filter` accept repeated glob patterns for excluded and included objects.

{ch_migrate_cli-0.7.0 → ch_migrate_cli-0.8.0}/README.md

@@ -23,6 +23,10 @@ This README describes the source checkout, which may be ahead of PyPI. To try an
 
 ## Upgrade from 0.x
 
+**From 0.7:** 0.8 needs no `upgrade-env`, and nothing changes unless you pass
+`--json` (see [Drift detection in CI](#drift-detection-in-ci)). A loose requirement
+such as `ch-migrate-cli` upgrades to 0.8; pin `ch-migrate-cli==0.7.0` to stay on 0.7.
+
 **From 0.6:** 0.7 needs no `upgrade-env`. `up` now waits for the mutations and
 `ON CLUSTER` DDL its statements start (see [Migrations that wait](#migrations-that-wait)),
 so give CI jobs time to finish or pass `--timeout`. Where `bootstrap` couldn't grant
@@ -473,12 +477,68 @@ def upgrade():
     create_dictionary("history/dictionaries/dict_users/001_create.sql")
 ```
 
+## Drift detection in CI
+
+After applying and verifying migrations in an environment, capture its expected
+schema:
+
+```bash
+ch-migrate snapshot dev
+mv migrations/sql/snapshots/20261002_120000 migrations/sql/snapshots/dev
+ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/dev --json
+```
+
+`snapshot` writes a new timestamped directory every time; replace the example
+timestamp with the one it reports. Give each environment's snapshot a fixed name,
+such as `snapshots/dev`, so `CH_MIGRATE_SNAPSHOT_DIR` never changes and the next
+snapshot shows up in review as a diff against this one. Review and commit that
+directory alongside the migration head. Do not regenerate the snapshot
+automatically in the drift job: doing so would accept an out-of-band change as the
+new expected schema. Re-snapshot after an intentional migration or an explicitly
+reviewed reconciliation, not merely to turn a failed check green. To re-snapshot,
+run `snapshot` again, delete the old `snapshots/dev`, and move the new directory
+into its place, so the commit also shows objects that were dropped.
+
+Copy [the GitHub Actions example](docs/examples/github-actions-drift.yml) into the
+consumer repository's `.github/workflows/`. It runs on a daily schedule and
+same-repository pull requests; fork PRs are excluded because they have no database
+secret. It does not use `pull_request_target`. All actions are pinned by commit.
+The checkout must contain `config.yaml` and the committed snapshot, and the runner
+must be able to reach the selected server.
+
+Configure these repository variables:
+
+| Variable | Value |
+|---|---|
+| `CH_MIGRATE_PACKAGE` | A pinned `ch-migrate-cli` requirement, such as `ch-migrate-cli==0.8.0`. 0.8.0 is the first release with `diff --json`; there is no fallback to an older one. |
+| `CH_MIGRATE_ENV` | An environment name in `config.yaml`, such as `dev`. |
+| `CH_MIGRATE_SNAPSHOT_DIR` | That environment's committed snapshot directory. |
+
+Set the repository secret `DRIFT_MIGRATION_PASSWORD` to the password for that
+environment's configured user. Prefer a dedicated schema-inspection account,
+not an admin account. The job writes a private `.env.local` without printing the
+value and removes the file even when comparison fails.
+
+`diff --json` exits **0** for matching schemas, **1** for drift, and **2** for a
+configuration/connection error. The workflow saves `drift.json`, uploads it even
+on comparison failure, then fails the job for either nonzero result. The report
+names changed tables and fields. ch-migrate's own test suite runs the example's
+shell steps against a real ClickHouse server, installing from the source checkout,
+for both a matching and a drifted schema.
+
+Without `--snapshot-dir`, `diff` sorts entries in `migrations/sql/snapshots/` by
+name and chooses the last one. It does not select a snapshot by environment or
+recorded migration revision. Use an explicit path in CI, especially when one
+repository holds snapshots for several environments.
+
 ## Command reference
 
 Every command accepts `--help`. Top-level `ch-migrate --version` reports the installed package version. `ENV` below names an entry in `config.yaml`.
 
 Output lines start with `→` for a step, `✓` for a result, `!` for a warning and `✗` for an error; warnings and errors go to stderr. Colour is dropped when output is not a terminal or `NO_COLOR` is set, and lines are never wrapped, so paths and SQL can be copied or grepped.
 
+`status`, `history`, `lint`, and `diff` accept `--json`. JSON mode writes one document to stdout; diagnostics go to stderr. Every document includes `"schema_version": 1` and `"command"`. The format can still change before 1.0; `schema_version` says which one a document follows. Errors include an `error` string rather than inventing a successful empty result. The schemas linked below use JSON Schema draft 2020-12.
+
 ### `init`
 
 `ch-migrate init [PATH] [-n NAME]` initializes the current directory by default. `-n/--name` sets the project name; otherwise it uses the directory name.
@@ -511,22 +571,53 @@ Example: `ch-migrate down dev --revision base`
 
 ### `status`
 
-`ch-migrate status ENV` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. No command-specific options. Status is a report: it exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check.
+`ch-migrate status ENV [--json]` shows connection information, applied/pending counts, and head status, and names the `up` command when migrations are pending. Status is a report: human output exits 0 when the database is unreachable (with a warning) or migrations are pending, and exits 1 only when the configuration or `migrations/versions/` is missing, so CI can run it as a non-blocking check. JSON mode uses nonzero exits for unreachable or pending state.
 
 Example: `ch-migrate status dev`
 
+With `--json`, [the status schema](docs/schemas/status.schema.json) includes
+`database`, `current_heads`, `script_heads`, `pending`, `applied`, and `at_head`.
+`applied` includes ancestors resolved through the revision graph, not just stored
+heads. JSON exits 0 when the head sets match exactly, 1 when pending or diverged,
+and 2 when configuration or database state cannot be read. Human-mode exits are
+unchanged.
+
+```bash
+ch-migrate status dev --json | jq -e .current_heads
+# Check the actual head set, not the last element of applied:
+ch-migrate status dev --json | jq -e '.current_heads == ["PINNED_REVISION"]'
+```
+
+Replace `PINNED_REVISION` with the required revision. In a shell pipeline, enable
+`set -o pipefail` if the caller must also preserve ch-migrate's exit code.
+
 ### `history`
 
-`ch-migrate history ENV` displays the revision graph and applied state. No command-specific options.
+`ch-migrate history ENV [--json]` displays the revision graph and applied state.
 
 Example: `ch-migrate history dev`
 
+`ch-migrate history dev --json` follows [the history schema](docs/schemas/history.schema.json).
+Each entry in `revisions` includes its revision, all `down_revisions` (including
+merge parents), description, create date, path, applied status, and irreversible
+reason. A reversible revision has `irreversible: null`. If database state is
+unavailable, `applied` is null, the document includes `error`, and JSON exits 2.
+An unknown database head also makes applied status unknown instead of claiming
+that every local revision is unapplied.
+
 ### `lint`
 
-`ch-migrate lint [ENV]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. No command-specific options. Errors exit nonzero; warnings and waiver INFO lines alone do not.
+`ch-migrate lint [ENV] [--json]` analyzes upgrade statements, not downgrade SQL. Without `ENV`, it checks revisions after the gate baseline statically without credentials or a connection. With an environment, it checks only pending revisions in that scope and adds live size and dependency checks. If it cannot determine the pending set, it fails rather than silently checking a different scope. Errors exit nonzero; warnings and waiver INFO lines alone do not.
 
 Example: `ch-migrate lint`
 
+`ch-migrate lint --json` follows [the lint schema](docs/schemas/lint.schema.json).
+`findings` include rule, severity, source file/line, message, `blocking`, and
+`waived`. `blocking` means the finding currently blocks `up`; a waived finding
+has its written reason and does not block. `counts` reports total, error, warning,
+info, blocking, and waived findings. JSON and human output use the same exit
+codes: 1 for lint/configuration errors, otherwise 0.
+
 Findings name the project-relative SQL file and statement line. Inline Python SQL
 points to its `op.execute` call. Extraction reads `run_sql`/`read_sql` file
 references and literal or f-string `op.execute` arguments without importing
@@ -543,10 +634,16 @@ Example: `ch-migrate deps dev --validate migrations/sql/history/tables/logs/chan
 
 ### `diff`
 
-`ch-migrate diff ENV [-s PATH]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift or an execution error.
+`ch-migrate diff ENV [-s PATH] [--json]` compares the live schema with the latest snapshot. `-s/--snapshot-dir PATH` chooses another snapshot. Exit code 0 means no drift; 1 means drift. JSON errors exit 2; human-mode execution errors retain exit 1.
 
 Example: `ch-migrate diff dev --snapshot-dir migrations/sql/snapshots/20261002_120000`
 
+`ch-migrate diff dev --json` follows [the diff schema](docs/schemas/diff.schema.json).
+`objects` identifies each object's type, name, status, and structural `details`
+(field, local value, remote value, and message). `in_sync` is true only when every
+object matches. Use the exit code as the drift gate; an error document does not
+contain a fabricated `in_sync: true`.
+
 ### `snapshot`
 
 `ch-migrate snapshot ENV [-e GLOB] [-f GLOB]` writes CREATE statements to a timestamped snapshot directory. `-e/--exclude` and `-f/--filter` accept repeated glob patterns for excluded and included objects.

{ch_migrate_cli-0.7.0 → ch_migrate_cli-0.8.0}/ch_migrate/__init__.py

@@ -5,7 +5,7 @@ Usage:
     from ch_migrate import read_sql, get_db, get_env_config, create_dictionary
 """
 
-__version__ = "0.7.0"
+__version__ = "0.8.0"
 
 
 from typing import Any