trellis-pg 0.1.1 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 410251832893ecd2d20f86866c37a19d6759daea75a7f921725d0e943533ed39
-  data.tar.gz: 36360b89efadf99fbf39801c2b35d12525966cca924a2e00a86ef134e05016b4
+  metadata.gz: 989a2bb8e00efa189fc95684dca175ecbdf17e0e6227ea26462c837276418567
+  data.tar.gz: 70d484a63fece2c1e4c61821f2565b3431638e65b74b6ad724fd22396c5df6bb
 SHA512:
-  metadata.gz: 37f46e475451bb8e861c578418fe9b98eb031bbb03bb04cdd357af15e5c3655a60e0fcee81de6bc8f952b72215bc16fee8cecc72b301ee9b5cbf7b3722f9cec1
-  data.tar.gz: 5b0062cd93400242b4dc7a5c20408a28853adf2d49549085471e53df12e2122e4e09ba96668bf72271c39757637ddb1786d7c6aaeba332d79d27fc738a907d70
+  metadata.gz: a0dde1a7bc9e39fdf24a27fe6130591d0423ae1f89ff059b61d0d06b65f186686486b18552e448ba46c4bd8427a6043e9365022feef9e637cd54166904a4996c
+  data.tar.gz: 05b66f0c6d95a08173f2805d1b75cc2bbb00b53c484e62d3d3ec4f2a16a4b572c7f2320b5a2d04addd4903aa0d8708dfdbdf9da7325edae1959eb39f7553d06f

data/README.md

@@ -18,7 +18,7 @@ as `trellis/pg`. The module is still `Trellis`.
 
 ```ruby
 # Gemfile
-gem "trellis-pg", "~> 0.1.0"
+gem "trellis-pg", "~> 0.2.0"
 ```
 
 `Bundler.require`, which a Rails app's `config/application.rb` runs,
@@ -78,10 +78,38 @@ Nothing in the binding uses it.
   `table`, `columns`, `role` (`:group_by`, `:join_from` or `:join_joined`: the part of
   the definition that needs it), `opted_out` and a `create index` statement as `fix`);
   `self_check` refuses such a target unless it was defined `WITHOUT SOURCE
-  INDEX`. `status` lists the partitions of a partitioned source a definition
+  INDEX`. `status` lists the source indexes a `MIN`/`MAX` aggregate's probes
+  read and the source lacks (`extreme_index_gaps`, each a
+  `Trellis::ExtremeIndexGap` with `table`, `columns`, `argument`, `opted_out`
+  and `fix`); nothing pauses, and a deleted extreme is re-read by scanning
+  its group. `status` lists the partitions of a partitioned source a definition
   is being corrected for while it reads `:backfilling` for them
   (`correcting`, each a `Trellis::CorrectingNode` with `node`, `since` and
-  `keys_remaining`).
+  `keys_remaining`). `status` names what a Re-derive build waits on instead
+  of its own work (`build_wait_xid`, the transaction id of the fence it
+  waits out, `nil` when it waits on nothing). While a process of another
+  Trellis version is live, `status` reports that this process stands by
+  (`version_handover`, a `Trellis::VersionHandover` with `this_version`,
+  `processes`, each a `Trellis::OtherProcess` with `worker_id` and
+  `version` (`nil` for a release before 0.2.0),
+  `unregistered_staging_worker` and `migrations_pending`), and
+  state-changing calls raise a `version_handover` error until it is gone.
+  While this binary is below the catalog floor, `status` reports that this
+  process stands by (`below_catalog_floor`, a `Trellis::BelowCatalogFloor`
+  with `floor` and `this_version`), and state-changing calls raise a
+  `version_floor` error.
+  For a `JOIN` definition `status` reports its hidden tables (`join`, a
+  `Trellis::JoinStatus`): `hidden_tables`, each a `Trellis::HiddenTable` with
+  `table`, `name`, `estimated_rows` and `bytes`, the `P`-size warning
+  (`partials_warning`, a `Trellis::PartialsWarning`), a 1-1 `JOIN`'s fan-out
+  warnings (`fan_out_warnings`, one `Trellis::FanOutWarning` per joined
+  position past the threshold, naming its `position`) and the step its build
+  runs (`building`, a `Trellis::JoinBuildStep` with `step` and the `side`,
+  `part`, `leaf`, `remaining` or `position` its kind of step carries). `join` is
+  `nil` for any other definition. `status` reports the oldest Trellis release
+  that can run a definition (`floor`, `"major.minor.patch"`), which an
+  application reads before rolling back to an earlier Trellis release
+  ([upgrading](../../docs/upgrading.md#rolling-back)).
 - `Trellis.request_backfill(source_table)`,
   `Trellis.detect_capture_disruptions` (pauses now the readers of every table
   whose Trellis capture was changed outside Trellis, and returns the tables),
@@ -161,7 +189,7 @@ one stand by; that is harmless, and each polls the lock every 5 seconds (see
 At the defaults, a handle holds up to 20 Postgres connections (its call pool,
 opened as calls need them and closed after 60 seconds idle) and 3 threads. A
 worker handle (`staging: true, drain_threads: N`) also owns a background pool
-of up to `2N + 5` connections (`2N + 4` without staging, `1` with staging
+of up to `2N + 6` connections (`2N + 5` without staging, `2` with staging
 alone), closed after 60 seconds idle, and two connections that stay open: the
 staging session and one `LISTEN` shared by its drain threads. It holds 6
 threads: drain threads are tasks on its runtime, not OS threads. Its memory is
@@ -262,7 +290,9 @@ default one included.
   ```
 - **Errors** are raised as a subclass of `Trellis::Error` per error code
   (`ParseError`, `ValidationError`, `ConnectivityError`, `ConflictError`,
-  `NotFoundError`, `InternalError`, `TimeoutError`), each with `#code` (a
+  `NotFoundError`, `InternalError`, `TimeoutError`, `VersionHandoverError`,
+  `VersionFloorError`),
+  each with `#code` (a
   symbol) and `#message`. A code newer than the binding arrives as
   `UnknownError`, naming the code in its message. `ForkedHandleError` is a
   `ValidationError`. Raising is Ruby's idiom, so where the Elixir binding
@@ -493,9 +523,8 @@ true, drain_threads: 1`), because the resume rebuilds through them.
 A Rails app that uses Trellis dumps its schema as `structure.sql`
 (`config.active_record.schema_format = :sql`), and the dump carries Trellis's
 definitions, the way it carries `schema_migrations`. `schema.rb` can't carry
-another schema's tables or triggers, so with `schema_format = :ruby` the dump
-(`db:migrate`'s and `db:schema:dump`) raises `Trellis::ValidationError` and
-writes nothing.
+them, so with `schema_format = :ruby` the dump (`db:migrate`'s and
+`db:schema:dump`) raises `Trellis::ValidationError` and writes nothing.
 
 ```ruby
 # config/application.rb
@@ -503,49 +532,65 @@ config.active_record.schema_format = :sql
 config.trellis.connect = { url: ENV.fetch("DATABASE_URL") }
 ```
 
-Nothing else is set up. After `pg_dump`, `db:migrate`'s dump and
-`db:schema:dump` append `Trellis::Instance#dump_definitions_sql` for
-`config.trellis.connect` and each of `config.trellis.instances`: Trellis's
-migration history, the instance marker, the transform
-definitions and the seed rows capture needs. The Railtie also adds each
-instance's `schema` and `target_schema` to the `--schema` flags Rails passes
-`pg_dump` (nothing is added when Rails dumps every schema), so the Trellis
-tables, the target tables and the capture triggers are in the file.
-`db:schema:load`, `db:prepare`, `db:setup` and the test database
-`maintain_test_schema!` loads then build a database with every definition in
-place. Each loads as `waiting_to_backfill`, and the worker builds it from the
-source rows; a test database has none, so that is immediate. Runtime state
-(ledger, ring, build chunks, quarantine, claims, statuses) is never dumped.
-
-- The hook applies to one database config, `config.trellis.dump_database`
+Nothing else is set up. The file has the application's tables, the target
+tables (with their indexes and grants), and one row per definition: its
+source, target and `JOIN` table, its text as written, its grammar version
+and its floor. Trellis's own schema, the capture triggers on source tables
+and the 1-1 targets' ledgers are not in it, so regenerating the dump changes
+it only where a definition changed. `db:schema:load`, `db:prepare`,
+`db:setup` and the test database `maintain_test_schema!` loads then run
+`migrate` for `config.trellis.connect` and each of `config.trellis.instances`,
+which creates Trellis's tables and restores each definition into its dumped
+target. Each is restored unbuilt, and the worker builds it from the source
+rows; a test database has none, so that is immediate. A file loaded another
+way (`psql -f structure.sql`) needs `bin/rails trellis:migrate` after it.
+How the restore works, and what it refuses, is in
+[`docs/embedding.md`](../../docs/embedding.md#schema-dumps).
+
+- The hooks apply to one database config, `config.trellis.dump_database`
   (default `"primary"`): the one `config.trellis.connect` and
-  `config.trellis.instances` point at. Other databases dump as Rails dumps
-  them. The dump raises `Trellis::ValidationError`, writing nothing, when
-  `config.trellis.connect` or an instance points at a different database than
-  that config. They are compared by host, port and database name: the Rails
-  config's as libpq resolves them (`PGHOST`, `PGPORT` and `PGDATABASE`
-  included), Trellis's `url` from the url alone (port 5432, database the
-  user's name). A config with no host uses libpq's local socket, which
-  matches a local Trellis host (`localhost`, `127.0.0.1`, `::1` or a socket
-  directory). Other host spellings are not equated, so `localhost` is not
-  `127.0.0.1`. `db:prepare` in development also dumps the test database
-  after migrating it; that dump is checked against development's config of
-  the same name, which `config.trellis.connect` points at.
-- `pg_dump` and `dump_definitions_sql` read separate snapshots. The hook reads
-  the definitions before and after the `pg_dump` and redoes the dump when they
-  differ (three attempts, then `Trellis::ConflictError`), so a define or drop
-  that races a dump can't leave tables and definitions disagreeing.
+  `config.trellis.instances` point at. Other databases dump and load as Rails
+  does it. The dump and the load raise `Trellis::ValidationError`, the dump
+  writing nothing, when `config.trellis.connect` or an instance points at a
+  different database than that config. They are compared by host, port and
+  database name: the Rails config's as libpq resolves them (`PGHOST`,
+  `PGPORT` and `PGDATABASE` included), Trellis's `url` from the url alone
+  (port 5432, database the user's name). A config with no host uses libpq's
+  local socket, which matches a local Trellis host (`localhost`,
+  `127.0.0.1`, `::1` or a socket directory). Other host spellings are not
+  equated, so `localhost` is not `127.0.0.1`.
+- `db:prepare` in development also dumps and loads the test database, and
+  `db:test:prepare` loads it. That dump is checked against development's
+  config of the same name, which `config.trellis.connect` points at, and the
+  load's restore connects to the test database: the configured options with
+  the test config's database, and its host, port, user, password and other
+  libpq keywords (`sslmode`, `sslrootcert`, ...) where it names a non-blank
+  value (its `url` when it has one).
+- The hook adds each instance's exclusions (its Trellis schema, its ledgers)
+  to the flags Rails passes `pg_dump`, and its `target_schema` to the
+  `--schema` flags when Rails passes any. `pg_dump` can't leave out one
+  trigger, so the hook takes the capture triggers out of its output
+  (`Trellis::Instance#filter_schema_dump`).
+- `pg_dump` and `Trellis::Instance#schema_dump` read separate snapshots. The
+  hook reads the definitions before and after the `pg_dump` and redoes the
+  dump when they differ (three attempts, then `Trellis::ConflictError`), so a
+  define or drop that races a dump can't leave tables and definitions
+  disagreeing. A dump that fails once `pg_dump` has written `structure.sql`
+  removes it, since what Rails wrote has no definitions and would load
+  without a word; a file no `pg_dump` wrote to is left as it was.
 - A definition that was paused or quarantined where the dump was taken loads
-  as `waiting_to_backfill`: the pause is runtime state.
+  unpaused: the pause is runtime state.
+- A `structure.sql` written by an earlier release, which carries Trellis's
+  schema and its definitions' catalog rows, still loads as it did.
 - `pg_dump` 17 writes `CREATE SCHEMA public`, which a database made by
   `db:create` already has; the hook writes `CREATE SCHEMA IF NOT EXISTS
   public` instead.
 - The dump connects through `Trellis::Railtie.with_handle`, so it migrates
   Trellis's tables first if the database lacks them, and starts no worker.
 - It hooks `ActiveRecord::Tasks::DatabaseTasks.dump_schema` and
-  `PostgreSQLDatabaseTasks#structure_dump`, which Rails marks `:nodoc:` and
-  offers no extension point for. A Rails upgrade that renames them fails the
-  test suite's round trip (`test/schema_dump_test.rb`).
+  `.load_schema`, and `PostgreSQLDatabaseTasks#structure_dump`, which Rails
+  marks `:nodoc:` and offers no extension point for. A Rails upgrade that
+  renames them fails the test suite's round trip (`test/schema_dump_test.rb`).
 
 ## Layout
 

data/ext/trellis_ruby/Cargo.lock

@@ -1729,8 +1729,8 @@ dependencies = [
 
 [[package]]
 name = "trellis"
-version = "0.1.1"
-source = "git+https://github.com/salesforce-misc/trellis?rev=0e13fef8a731b23ae4650fd040eb1d0f93c01c44#0e13fef8a731b23ae4650fd040eb1d0f93c01c44"
+version = "0.2.0"
+source = "git+https://github.com/salesforce-misc/trellis?rev=f2afc74f5ee0a6ac309051da8678e0b2d862892e#f2afc74f5ee0a6ac309051da8678e0b2d862892e"
 dependencies = [
  "deadpool",
  "deadpool-postgres",
@@ -1739,7 +1739,6 @@ dependencies = [
  "metrics-exporter-prometheus",
  "metrics-util",
  "refinery",
- "regex",
  "rustls",
  "serde",
  "serde_json",
@@ -1752,8 +1751,8 @@ dependencies = [
 
 [[package]]
 name = "trellis-embed"
-version = "0.1.1"
-source = "git+https://github.com/salesforce-misc/trellis?rev=0e13fef8a731b23ae4650fd040eb1d0f93c01c44#0e13fef8a731b23ae4650fd040eb1d0f93c01c44"
+version = "0.2.0"
+source = "git+https://github.com/salesforce-misc/trellis?rev=f2afc74f5ee0a6ac309051da8678e0b2d862892e#f2afc74f5ee0a6ac309051da8678e0b2d862892e"
 dependencies = [
  "tracing",
  "trellis",
@@ -1761,7 +1760,7 @@ dependencies = [
 
 [[package]]
 name = "trellis_ruby"
-version = "0.1.1"
+version = "0.2.0"
 dependencies = [
  "magnus",
  "rb-sys",

data/ext/trellis_ruby/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "trellis_ruby"
-version = "0.1.1"
+version = "0.2.0"
 edition = "2024"
 publish = false
 
@@ -28,8 +28,8 @@ doctest = false
 ruby = ["dep:magnus", "dep:rb-sys"]
 
 [dependencies]
-trellis = { git = "https://github.com/salesforce-misc/trellis", rev = "0e13fef8a731b23ae4650fd040eb1d0f93c01c44" }
-trellis-embed = { git = "https://github.com/salesforce-misc/trellis", rev = "0e13fef8a731b23ae4650fd040eb1d0f93c01c44" }
+trellis = { git = "https://github.com/salesforce-misc/trellis", rev = "f2afc74f5ee0a6ac309051da8678e0b2d862892e" }
+trellis-embed = { git = "https://github.com/salesforce-misc/trellis", rev = "f2afc74f5ee0a6ac309051da8678e0b2d862892e" }
 magnus = { version = "0.9", optional = true, default-features = false, features = ["rb-sys"] }
 rb-sys = { version = "0.9", optional = true }
 

data/ext/trellis_ruby/src/lib.rs

@@ -61,14 +61,17 @@ use magnus::{
 };
 use trellis::{BlockingTrellis, Config, ErrorCode, PendingCall, TrellisOptions};
 use trellis_embed::{
-    DIVERGENCE_KINDS, ERROR_CODES, PlainApplied, PlainBackfillFailure, PlainCaptureFailure,
-    PlainCaptureWait, PlainConfig, PlainCorrectingNode, PlainDefinitionStatus,
-    PlainDefinitionSummary, PlainDivergence, PlainDrainFailure, PlainError, PlainHeldKeys,
-    PlainPoisonEntry, PlainQuarantineEntry, PlainSamplePage, PlainSelfCheckJob,
-    PlainSelfCheckReport, PlainSourceIndexGap, SELF_CHECK_MODES, SELF_CHECK_OUTCOMES,
-    SELF_CHECK_STATES, capture_failure_kind_names, decode_cursor, decode_watermark,
-    encode_watermark, index_role_names, quarantine_state_names, require_transform_statement,
-    self_check_mode, system_time_from_epoch_micros, transform_status_names,
+    DIVERGENCE_KINDS, ERROR_CODES, JOIN_BUILD_STEPS, JOIN_SIDE_PARTS, JOIN_SIDES, PlainApplied,
+    PlainBackfillFailure, PlainBelowCatalogFloor, PlainCaptureFailure, PlainCaptureWait,
+    PlainConfig, PlainCorrectingNode, PlainDefinitionStatus, PlainDefinitionSummary,
+    PlainDivergence, PlainDrainFailure, PlainError, PlainExtremeIndexGap, PlainFanOutWarning,
+    PlainHeldKeys, PlainHiddenTable, PlainJoinBuildStep, PlainJoinStatus, PlainOtherProcess,
+    PlainPartialsWarning, PlainPoisonEntry, PlainQuarantineEntry, PlainSamplePage,
+    PlainSelfCheckJob, PlainSelfCheckReport, PlainSourceIndexGap, PlainVersionHandover,
+    SELF_CHECK_MODES, SELF_CHECK_OUTCOMES, SELF_CHECK_STATES, capture_failure_kind_names,
+    decode_cursor, decode_watermark, encode_watermark, index_role_names, quarantine_state_names,
+    require_transform_statement, self_check_mode, system_time_from_epoch_micros,
+    transform_status_names,
 };
 
 /// What a blocking call produces: its value, or the engine's error as plain
@@ -234,6 +237,29 @@ impl Handle {
         rb_self.call(ruby, BlockingTrellis::start_dump_definitions_sql)
     }
 
+    /// What a schema dump of the database needs from this instance
+    /// ([`BlockingTrellis::schema_dump`]): `[pg_dump_args, appendix]`, the
+    /// arguments that leave Trellis's own tables out of `pg_dump` and the
+    /// SQL to append after its filtered output.
+    fn schema_dump(ruby: &Ruby, rb_self: &Self) -> Result<RArray, Error> {
+        let dump = rb_self.call(ruby, BlockingTrellis::start_schema_dump)?;
+        let args = ruby.ary_from_iter(dump.pg_dump_args);
+        Ok(ruby.ary_new_from_values(&[args.as_value(), ruby.str_new(&dump.appendix).as_value()]))
+    }
+
+    /// `pg_dump_output` without this instance's capture triggers
+    /// ([`BlockingTrellis::filter_schema_dump`]). Reads nothing from the
+    /// database.
+    fn filter_schema_dump(
+        ruby: &Ruby,
+        rb_self: &Self,
+        pg_dump_output: String,
+    ) -> Result<String, Error> {
+        rb_self
+            .with_engine(ruby, |trellis| trellis.filter_schema_dump(&pg_dump_output))?
+            .map_err(|err| raise(ruby, err.into()))
+    }
+
     /// Rebuilds every applying definition that reads `source_table`
     /// (`live -> backfilling`).
     fn request_backfill(ruby: &Ruby, rb_self: &Self, source_table: String) -> Result<(), Error> {
@@ -835,6 +861,9 @@ fn symbol_words() -> impl Iterator<Item = &'static str> {
         .chain(DIVERGENCE_KINDS)
         .chain(capture_failure_kind_names())
         .chain(index_role_names())
+        .chain(JOIN_BUILD_STEPS)
+        .chain(JOIN_SIDES)
+        .chain(JOIN_SIDE_PARTS)
 }
 
 /// A hash from symbol keys to `fields`' values: the shape every record
@@ -906,8 +935,21 @@ fn status_hash(ruby: &Ruby, status: PlainDefinitionStatus) -> Result<RHash, Erro
         .map(|failure| drain_failure_hash(ruby, failure))
         .transpose()?;
     let source_index_gaps = array(ruby, status.source_index_gaps, source_index_gap_hash)?;
+    let extreme_index_gaps = array(ruby, status.extreme_index_gaps, extreme_index_gap_hash)?;
     let correcting = array(ruby, status.correcting, correcting_hash)?;
     let paused_upstream_columns = ruby.ary_from_iter(status.paused_upstream_columns);
+    let version_handover = status
+        .version_handover
+        .map(|seen| version_handover_hash(ruby, seen))
+        .transpose()?;
+    let below_catalog_floor = status
+        .below_catalog_floor
+        .map(|seen| below_catalog_floor_hash(ruby, seen))
+        .transpose()?;
+    let join = status
+        .join
+        .map(|join| join_status_hash(ruby, join))
+        .transpose()?;
     record(
         ruby,
         [
@@ -918,11 +960,142 @@ fn status_hash(ruby: &Ruby, status: PlainDefinitionStatus) -> Result<RHash, Erro
             ("held_keys", ruby.into_value(held_keys)),
             ("drain_failure", ruby.into_value(drain_failure)),
             ("source_index_gaps", source_index_gaps.as_value()),
+            ("extreme_index_gaps", extreme_index_gaps.as_value()),
             ("correcting", correcting.as_value()),
             (
                 "paused_upstream_columns",
                 paused_upstream_columns.as_value(),
             ),
+            ("build_wait_xid", ruby.into_value(status.build_wait_xid)),
+            ("version_handover", ruby.into_value(version_handover)),
+            ("below_catalog_floor", ruby.into_value(below_catalog_floor)),
+            ("join", ruby.into_value(join)),
+            ("floor", ruby.into_value(status.floor)),
+        ],
+    )
+}
+
+/// A `JOIN` transform's hidden tables, warnings and build step.
+fn join_status_hash(ruby: &Ruby, join: PlainJoinStatus) -> Result<RHash, Error> {
+    let hidden_tables = array(ruby, join.hidden_tables, hidden_table_hash)?;
+    let partials_warning = join
+        .partials_warning
+        .map(|warning| partials_warning_hash(ruby, warning))
+        .transpose()?;
+    let fan_out_warnings = array(ruby, join.fan_out_warnings, fan_out_warning_hash)?;
+    let building = join
+        .building
+        .map(|step| join_build_step_hash(ruby, step))
+        .transpose()?;
+    record(
+        ruby,
+        [
+            ("hidden_tables", hidden_tables.as_value()),
+            ("partials_warning", ruby.into_value(partials_warning)),
+            ("fan_out_warnings", fan_out_warnings.as_value()),
+            ("building", ruby.into_value(building)),
+        ],
+    )
+}
+
+/// One hidden table of a `JOIN`.
+fn hidden_table_hash(ruby: &Ruby, table: PlainHiddenTable) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("table", ruby.into_value(table.table)),
+            ("name", ruby.into_value(table.name)),
+            ("estimated_rows", ruby.into_value(table.estimated_rows)),
+            ("bytes", ruby.into_value(table.bytes)),
+        ],
+    )
+}
+
+/// `P` has grown close to the fact table's size.
+fn partials_warning_hash(ruby: &Ruby, warning: PlainPartialsWarning) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("partials_rows", ruby.into_value(warning.partials_rows)),
+            ("fact_rows", ruby.into_value(warning.fact_rows)),
+        ],
+    )
+}
+
+/// A 1-1 `JOIN`'s very large group on one join key at one joined position.
+fn fan_out_warning_hash(ruby: &Ruby, warning: PlainFanOutWarning) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("position", ruby.into_value(warning.position)),
+            ("key", ruby.into_value(warning.key)),
+            ("rows", ruby.into_value(warning.rows)),
+            ("threshold", ruby.into_value(warning.threshold)),
+        ],
+    )
+}
+
+/// The step a `JOIN` build runs. `step`, `side` and `part` are symbols from
+/// [`JOIN_BUILD_STEPS`], [`JOIN_SIDES`] and [`JOIN_SIDE_PARTS`]; each of the
+/// others is set only for the step that carries it.
+fn join_build_step_hash(ruby: &Ruby, step: PlainJoinBuildStep) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("step", word_symbol(ruby, step.step).as_value()),
+            (
+                "side",
+                ruby.into_value(step.side.map(|word| word_symbol(ruby, word))),
+            ),
+            (
+                "part",
+                ruby.into_value(step.part.map(|word| word_symbol(ruby, word))),
+            ),
+            ("leaf", ruby.into_value(step.leaf)),
+            ("remaining", ruby.into_value(step.remaining)),
+            ("position", ruby.into_value(step.position)),
+        ],
+    )
+}
+
+/// Another Trellis version is live, so this binary stands by.
+fn version_handover_hash(ruby: &Ruby, seen: PlainVersionHandover) -> Result<RHash, Error> {
+    let processes = array(ruby, seen.processes, other_process_hash)?;
+    record(
+        ruby,
+        [
+            ("this_version", ruby.into_value(seen.this_version)),
+            ("processes", processes.as_value()),
+            (
+                "unregistered_staging_worker",
+                ruby.into_value(seen.unregistered_staging_worker),
+            ),
+            (
+                "migrations_pending",
+                ruby.into_value(seen.migrations_pending),
+            ),
+        ],
+    )
+}
+
+/// This binary is below the catalog floor, so it stands by.
+fn below_catalog_floor_hash(ruby: &Ruby, seen: PlainBelowCatalogFloor) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("floor", ruby.into_value(seen.floor)),
+            ("this_version", ruby.into_value(seen.this_version)),
+        ],
+    )
+}
+
+/// A live process of another version.
+fn other_process_hash(ruby: &Ruby, process: PlainOtherProcess) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("worker_id", ruby.into_value(process.worker_id)),
+            ("version", ruby.into_value(process.version)),
         ],
     )
 }
@@ -953,6 +1126,20 @@ fn source_index_gap_hash(ruby: &Ruby, gap: PlainSourceIndexGap) -> Result<RHash,
     )
 }
 
+/// A source index a `MIN`/`MAX` field's probe reads and the source lacks.
+fn extreme_index_gap_hash(ruby: &Ruby, gap: PlainExtremeIndexGap) -> Result<RHash, Error> {
+    record(
+        ruby,
+        [
+            ("table", ruby.into_value(gap.table)),
+            ("columns", ruby.into_value(gap.columns)),
+            ("argument", ruby.into_value(gap.argument)),
+            ("opted_out", ruby.into_value(gap.opted_out)),
+            ("fix", ruby.into_value(gap.fix)),
+        ],
+    )
+}
+
 /// A drain page that keeps failing with nothing charged or paused.
 fn drain_failure_hash(ruby: &Ruby, failure: PlainDrainFailure) -> Result<RHash, Error> {
     record(
@@ -1222,6 +1409,21 @@ fn index_roles(ruby: &Ruby) -> RArray {
     word_symbols(ruby, index_role_names())
 }
 
+/// Every step symbol a `JOIN` build's status can carry.
+fn join_build_steps(ruby: &Ruby) -> RArray {
+    word_symbols(ruby, JOIN_BUILD_STEPS)
+}
+
+/// Every side symbol a `JOIN` build step can carry.
+fn join_sides(ruby: &Ruby) -> RArray {
+    word_symbols(ruby, JOIN_SIDES)
+}
+
+/// Every part symbol a `JOIN` build's side step can carry.
+fn join_side_parts(ruby: &Ruby) -> RArray {
+    word_symbols(ruby, JOIN_SIDE_PARTS)
+}
+
 /// Every statement kind `trellis`'s grammar has, for the test asserting
 /// `apply`'s round trip covers each one.
 fn statement_kinds() -> Vec<&'static str> {
@@ -1265,6 +1467,9 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
     native.define_module_function("divergence_kinds", function!(divergence_kinds, 0))?;
     native.define_module_function("capture_failure_kinds", function!(capture_failure_kinds, 0))?;
     native.define_module_function("index_roles", function!(index_roles, 0))?;
+    native.define_module_function("join_build_steps", function!(join_build_steps, 0))?;
+    native.define_module_function("join_sides", function!(join_sides, 0))?;
+    native.define_module_function("join_side_parts", function!(join_side_parts, 0))?;
     native.define_module_function("statement_kinds", function!(statement_kinds, 0))?;
     native.define_module_function("render_prometheus", function!(render_prometheus, 0))?;
 
@@ -1279,6 +1484,8 @@ fn init(ruby: &Ruby) -> Result<(), Error> {
         "dump_definitions_sql",
         method!(Handle::dump_definitions_sql, 0),
     )?;
+    handle.define_method("schema_dump", method!(Handle::schema_dump, 0))?;
+    handle.define_method("filter_schema_dump", method!(Handle::filter_schema_dump, 1))?;
     handle.define_method("request_backfill", method!(Handle::request_backfill, 1))?;
     handle.define_method("release_key", method!(Handle::release_key, 3))?;
     handle.define_method("poisoned_since", method!(Handle::poisoned_since, 1))?;

data/lib/trellis/error.rb

@@ -14,7 +14,8 @@ module Trellis
     CODE = nil
 
     # One of :parse, :validation, :connectivity, :conflict, :not_found,
-    # :internal, :timeout, or :unknown for a code newer than this binding.
+    # :internal, :timeout, :version_handover, :version_floor, or :unknown for
+    # a code newer than this binding.
     def code
       self.class::CODE
     end
@@ -62,12 +63,33 @@ module Trellis
 
   # A bounded wait ran out of time, such as Trellis.await_converged's
   # timeout_ms, or Trellis.release_key's wait for the pages in flight on its
-  # table, or a call ran out of its 30-second deadline (the server stopped
+  # table, or a statement waited out the 30-second lock timeout behind another
+  # transaction's lock (a define, ALTER or DROP behind a drain page, say), or a
+  # call ran out of its 30-second deadline (the server stopped
   # what the call started and rolled it back). Expected, not a bug: retry it.
   class TimeoutError < Error
     CODE = :timeout
   end
 
+  # A Trellis process of another version is live on the instance, so this
+  # call changed nothing: a state-changing call, or Trellis.migrate with
+  # migrations pending, during a deploy that crosses a version boundary.
+  # Read-only calls keep working. Retry once the other version's processes
+  # have stopped; the message names the versions and their worker ids.
+  class VersionHandoverError < Error
+    CODE = :version_handover
+  end
+
+  # This Trellis version is below a compatibility floor the database
+  # records: the catalog floor, the oldest version that can read the
+  # instance's catalog, so Trellis.migrate or a state-changing call refused
+  # it; or a definition's floor, the oldest version that can run it, so DROP
+  # TRANSFORM refused it and changed nothing. Not retryable with this
+  # version: run the version the message names, or a later one.
+  class VersionFloorError < Error
+    CODE = :version_floor
+  end
+
   # An error code newer than this binding. The message names the code.
   class UnknownError < Error
     CODE = :unknown
@@ -97,7 +119,9 @@ module Trellis
       "conflict" => ConflictError,
       "not_found" => NotFoundError,
       "internal" => InternalError,
-      "timeout" => TimeoutError
+      "timeout" => TimeoutError,
+      "version_handover" => VersionHandoverError,
+      "version_floor" => VersionFloorError
     }.freeze
   end
 end

data/lib/trellis/instance.rb

@@ -65,7 +65,8 @@ module Trellis
   #   TRANSFORM) and reports what it did
   #   as an Applied. define is the one guarded convenience: an apply that
   #   refuses anything but a TRANSFORM statement before applying it.
-  # - Reads: status, definitions, dump_definitions_sql.
+  # - Reads: status, definitions, schema_dump, filter_schema_dump,
+  #   dump_definitions_sql.
   # - Quarantine: quarantined, quarantine_status, sample_quarantined,
   #   poisoned_since, release_key. Pausing and resuming are statements:
   #   instance.apply("RESUME TRANSFORM order_totals.total").
@@ -336,6 +337,35 @@ module Trellis
       handle.dump_definitions_sql
     end
 
+    # What a schema dump of this instance's database needs, as a frozen Hash:
+    # :pg_dump_args, the arguments (each one argument, not shell words) that
+    # leave Trellis's own schema and the 1-1 targets' ledgers out of
+    # `pg_dump`, and :appendix, the SQL to append after the dump. Run
+    # `pg_dump --schema-only` with pg_dump_args (and the target schema among
+    # the dump's `--schema`s, if it names any), pass the output through
+    # filter_schema_dump, and append the appendix. Both are the same for the
+    # same definitions, whatever their history. A database loaded from the file has the application's
+    # schema and the target tables; `migrate` restores every definition into
+    # its target, each unbuilt, and the worker builds them. Trellis's own
+    # schema, the capture triggers and the ledgers are not in the dump.
+    #
+    # Read from one snapshot, a different one from the pg_dump's: the Rails
+    # hook (Trellis::SchemaDump) reads it on both sides of the pg_dump and
+    # retries when the two differ.
+    def schema_dump
+      pg_dump_args, appendix = handle.schema_dump
+      { pg_dump_args: pg_dump_args.freeze, appendix: appendix.freeze }.freeze
+    end
+
+    # pg_dump_output, the output of a `pg_dump` run with schema_dump's
+    # pg_dump_args, without this instance's capture triggers on its source
+    # tables, which `pg_dump` can't leave out by itself. Reads nothing from
+    # the database. Raises Trellis::InternalError if a line still runs one of
+    # Trellis's functions afterwards: the dump wouldn't load.
+    def filter_schema_dump(pg_dump_output)
+      handle.filter_schema_dump(pg_dump_output)
+    end
+
     # Rebuilds every definition that reads source_table (a bare table name,
     # resolved like one in a statement), and returns once that rebuild is
     # queued.

data/lib/trellis/pg.rb

@@ -43,7 +43,8 @@ require_relative "metrics"
 #
 # - Lifecycle: connect, connected?, migrate, config, shutdown.
 # - Every statement form: apply, and define for a TRANSFORM statement alone.
-# - Reads: status, definitions, dump_definitions_sql.
+# - Reads: status, definitions, schema_dump, filter_schema_dump,
+#   dump_definitions_sql.
 # - Quarantine: quarantined, quarantine_status, sample_quarantined,
 #   poisoned_since, release_key.
 # - Operations: request_backfill, detect_capture_disruptions, has_live_drain_workers?,

data/lib/trellis/schema_dump.rb

@@ -2,46 +2,62 @@
 
 require "active_record"
 require "etc"
+require "fileutils"
 require "trellis/pg"
 
 module Trellis
   # Makes `structure.sql` carry Trellis's definitions, the way Rails makes it
   # carry `schema_migrations`: after `pg_dump` writes the schema, the dump
-  # appends the `INSERT`s that describe how the schema came to be. A database
-  # built from the file (`db:schema:load`, `db:prepare`, the test database
-  # `maintain_test_schema!` loads) then has every transform defined, with no
-  # Trellis step to run: Trellis's worker finds the definitions unbuilt and builds them (ADR-0010).
+  # appends the rows that describe how the schema came to be. The dump has
+  # the application's schema and the target tables, and one row per
+  # definition (Trellis::Instance#schema_dump). Trellis's own schema, the
+  # capture triggers on source tables and the 1-1 targets' ledgers are left
+  # out: a database built from the file gets them from the loading release,
+  # whose `migrate` restores each definition into its dumped target (ADR-0025).
   #
   # Needs `config.active_record.schema_format = :sql`: a `schema.rb` can't
-  # carry another schema's tables or triggers. With `:ruby`, the dump raises
+  # carry the definitions. With `:ruby`, the dump raises
   # Trellis::ValidationError and writes nothing: a `schema.rb` would load
   # into a database with target tables and no definitions.
   #
-  # It hooks the two places Rails offers no extension point for (both the
-  # `db:schema:dump` task and the dump after `db:migrate` call them):
+  # It hooks the places Rails offers no extension point for:
   #
-  # - ActiveRecord::Tasks::DatabaseTasks.dump_schema, whose `:sql` branch
-  #   is `pg_dump` followed by the `schema_migrations` rows. This hook adds
-  #   the Trellis rows after them: Trellis::Instance#dump_definitions_sql
-  #   for the default instance and each of config.trellis.instances. Those
-  #   rows are read before and after the `pg_dump`, which reads its own
-  #   snapshot, and the dump is redone (up to three times) when a define or
-  #   drop committed in between, so the DDL and the rows agree.
-  # - ActiveRecord::Tasks::PostgreSQLDatabaseTasks#structure_dump, which
-  #   passes `--schema` only for the search path's schemas. This hook adds
-  #   each instance's `schema` and `target_schema`, so the Trellis tables and
-  #   capture triggers are in the dump. (Nothing is added when Rails dumps
-  #   every schema anyway, `ActiveRecord.dump_schemas = :all` or no
-  #   `schema_search_path`.)
+  # - ActiveRecord::Tasks::DatabaseTasks.dump_schema (the `db:schema:dump`
+  #   task and the dump after `db:migrate` both call it), whose `:sql`
+  #   branch is `pg_dump` followed by the `schema_migrations` rows. This
+  #   hook takes the capture triggers out of the `pg_dump` output
+  #   (Trellis::Instance#filter_schema_dump) and adds the definitions'
+  #   rows after the `schema_migrations` ones, for the default instance and
+  #   each of config.trellis.instances. Those rows are read before and after
+  #   the `pg_dump`, which reads its own snapshot, and the dump is redone (up
+  #   to three times) when a define or drop committed in between, so the DDL
+  #   and the rows agree.
+  # - ActiveRecord::Tasks::PostgreSQLDatabaseTasks#structure_dump. This hook
+  #   adds each instance's `pg_dump_args` (which exclude the Trellis schema
+  #   and the ledgers) and, when Rails passes `--schema` for the search
+  #   path's schemas, each instance's `target_schema`.
+  # - ActiveRecord::Tasks::DatabaseTasks.load_schema, which
+  #   `db:schema:load`, `db:prepare` and the test database's
+  #   `maintain_test_schema!` load the file with. After the load, this hook
+  #   runs `migrate` for each configured instance, which restores the
+  #   definitions (a file in the older format, which carries Trellis's
+  #   schema, needs nothing more and loads as before). A load of another
+  #   environment's database (`db:prepare` in development prepares the test
+  #   database too, and `db:test:prepare` the test one) runs it against
+  #   that database: the instances' options with the database's host, port,
+  #   name, user and password from its config.
   #
-  # Both apply to one database: the config named by
+  # A load that doesn't go through Rails (`psql -f structure.sql`) is followed
+  # by `rails trellis:migrate`.
+  #
+  # All three apply to one database: the config named by
   # `config.trellis.dump_database` (default "primary"), the one
   # config.trellis.connect and config.trellis.instances point at. The other
-  # databases of a multi-database app dump as Rails dumps them.
+  # databases of a multi-database app dump and load as Rails does it.
   #
   # A dump lists definitions and nothing else of Trellis's state. Every
-  # definition loads as `waiting_to_backfill`, as a `define` leaves it,
-  # including one that was paused or quarantined where the dump was taken.
+  # definition loads unbuilt, as a `define` leaves it, including one that
+  # was paused or quarantined where the dump was taken.
   #
   # `pg_dump` 17 writes `CREATE SCHEMA public` when `public` is dumped,
   # which fails to load into a database that has one (every database
@@ -54,12 +70,9 @@ module Trellis
     # socket) matches, besides a socket directory.
     LOCAL_HOSTS = %w[localhost 127.0.0.1 ::1].freeze
 
-    HEADER = <<~SQL
-      --
-      -- Trellis definitions (Trellis::Instance#dump_definitions_sql)
-      --
-
-    SQL
+    # The thread-local the dump hook leaves the parts it read before the
+    # `pg_dump` in, for the structure-dump hook's arguments.
+    PARTS_KEY = :trellis_schema_dump_parts
 
     class << self
       # Installs the hooks. Idempotent.
@@ -68,6 +81,12 @@ module Trellis
         ::ActiveRecord::Tasks::PostgreSQLDatabaseTasks.prepend(StructureDumpHook)
       end
 
+      # The schema format of a dump or load of db_config: the format
+      # argument args carries, or Rails' default for db_config.
+      def format_of(db_config, args)
+        args.first || (db_config.respond_to?(:schema_format) ? db_config.schema_format : ::ActiveRecord.schema_format)
+      end
+
       # Whether Trellis's definitions belong in the dump of db_config: Trellis
       # is configured, and db_config is the database config.trellis.dump_database
       # names.
@@ -122,38 +141,136 @@ module Trellis
         (app&.config&.trellis&.dump_database || "primary").to_s
       end
 
-      # The `--schema` flags to add to the pg_dump of db_config, whose
-      # connection settings are configuration_hash.
-      def schema_flags(db_config, configuration_hash)
-        return [] unless applies?(db_config) && restricts_schemas?(configuration_hash)
-
-        schemas = []
-        default = Railtie.connect_options
-        schemas << default if default
-        schemas.concat(Railtie.instance_options.values)
-        schemas.flat_map { |options| [options.fetch(:schema, "trellis"), options.fetch(:target_schema, "public")] }
-               .uniq.map { |schema| "--schema=#{schema}" }
+      # The flags to add to the pg_dump of db_config, whose connection
+      # settings are configuration_hash: each instance's pg_dump_args, and its
+      # target schema when Rails names the schemas to dump. parts are the
+      # instances' schema_dump Hashes (Trellis::Instance#schema_dump), read now when the dump hook
+      # hasn't.
+      def schema_flags(db_config, configuration_hash, parts = nil)
+        return [] unless applies?(db_config)
+
+        parts ||= Thread.current[PARTS_KEY] || self.parts
+        flags = []
+        if restricts_schemas?(configuration_hash)
+          flags.concat(configured_options.map { |options| "--schema=#{exact_pattern(options.fetch(:target_schema, "public"))}" })
+        end
+        flags.concat(parts.flat_map { |part| part.fetch(:pg_dump_args) })
+        flags.uniq
+      end
+
+      # schema as a pg_dump pattern that matches exactly it: as is when it is
+      # lower case letters, digits and underscores, which pg_dump reads
+      # literally, and double-quoted otherwise, so its case and pattern
+      # characters (* ? .) are taken literally.
+      def exact_pattern(schema)
+        schema = schema.to_s
+        schema.match?(/\A[a-z_][a-z0-9_]*\z/) ? schema : "\"#{schema.gsub('"', '""')}\""
       end
 
-      # The definitions of every configured instance as SQL: each one's
-      # dump_definitions_sql.
-      def appendix
-        parts = []
-        parts << Railtie.with_handle(&:dump_definitions_sql) if Railtie.connect_options
-        Railtie.instance_options.each_key { |name| parts << Railtie.with_handle(name, &:dump_definitions_sql) }
-        parts.join("\n")
+      # Each configured instance's schema_dump Hashes (Trellis::Instance#schema_dump): the default
+      # instance's first, then config.trellis.instances' in order.
+      def parts
+        each_instance.map { |name| Railtie.with_handle(name, &:schema_dump) }
       end
 
-      # Appends appendix to the dump at filename, and makes its
-      # `CREATE SCHEMA public` loadable into a database that has one.
-      def finish(filename, appendix)
+      # The dump at filename without each instance's capture triggers, and
+      # its `CREATE SCHEMA public` loadable into a database that has one,
+      # followed by each of parts' appendix.
+      def finish(filename, parts)
         sql = File.read(filename, encoding: "utf-8")
         sql = sql.gsub(/^CREATE SCHEMA public;$/, "CREATE SCHEMA IF NOT EXISTS public;")
-        File.write(filename, "#{sql}#{HEADER}#{appendix}\n", encoding: "utf-8")
+        each_instance.each do |name|
+          sql = Railtie.with_handle(name) { |instance| instance.filter_schema_dump(sql) }
+        end
+        appendix = parts.map { |part| part.fetch(:appendix) }.join("\n")
+        File.write(filename, "#{sql}\n#{appendix}", encoding: "utf-8")
+      end
+
+      # Removes what a dump that failed left at filename, unless it is
+      # previous (the file's bytes before the dump, nil when there was none):
+      # a failure before `pg_dump` wrote to the file leaves it alone.
+      def discard_unfinished(filename, previous)
+        return unless File.exist?(filename)
+        return if previous && File.binread(filename) == previous
+
+        FileUtils.rm_f(filename)
+      end
+
+      # Restores the definitions a load of db_config's dump queued: `migrate`
+      # on each configured instance, in the database db_config names.
+      def restore(db_config)
+        verify_same_database!(db_config)
+        if db_config.env_name.to_s == ::ActiveRecord::Tasks::DatabaseTasks.env.to_s
+          Railtie.migrate_all
+          return
+        end
+
+        configured = []
+        configured << [nil, Railtie.connect_options] if Railtie.connect_options
+        Railtie.instance_options.each { |name, options| configured << [name, options] }
+        configured.each do |name, options|
+          what = name ? "config.trellis.instances[#{name.inspect}]" : "config.trellis.connect"
+          url = environment_url(db_config, options[:url], what)
+          instance = Trellis::Instance.connect(**options, url:, staging: false, drain_threads: 0)
+          begin
+            instance.migrate
+          ensure
+            instance.shutdown
+          end
+        end
       end
 
       private
 
+      # nil for the default instance (when config.trellis.connect is set),
+      # then the name of each of config.trellis.instances.
+      def each_instance
+        names = []
+        names << nil if Railtie.connect_options
+        names.concat(Railtie.instance_options.keys)
+      end
+
+      def configured_options
+        options = []
+        options << Railtie.connect_options if Railtie.connect_options
+        options.concat(Railtie.instance_options.values)
+      end
+
+      # A libpq connection string for the database db_config names, in
+      # another environment than configured_url's: configured_url's
+      # settings with db_config's database, and its host, port, user,
+      # password and other libpq keywords (sslmode, sslrootcert, ...) where
+      # db_config names them (a host left to libpq's default keeps
+      # configured_url's, a local one by verify_same_database!). A db_config
+      # with a url is that url.
+      def environment_url(db_config, configured_url, what)
+        hash = db_config.configuration_hash.transform_keys(&:to_sym)
+        return hash[:url].to_s if hash[:url]
+
+        keywords = conninfo(configured_url, what).reject { |_, value| value.to_s.empty? }
+        host, port, dbname = rails_database(hash)
+        keywords["dbname"] = dbname
+        keywords["port"] = port
+        if host
+          keywords["host"] = host
+          # A hostaddr would win over the host for the connection itself.
+          keywords.delete("hostaddr")
+        end
+        user = hash[:username] || hash[:user]
+        keywords["user"] = user.to_s if user
+        # Every other libpq keyword the config names (sslmode, sslrootcert,
+        # connect_timeout, ...) is the connection Rails makes: the adapter
+        # hands them all to libpq. A blank one names nothing, a blank password
+        # included: it doesn't blank the url's.
+        PG::Connection.conndefaults_hash.each_key do |key|
+          next if %i[host port dbname user].include?(key)
+
+          value = hash[key]
+          keywords[key.to_s] = value.to_s unless value.nil? || value.to_s.empty?
+        end
+        keywords.map { |key, value| "#{key}='#{value.to_s.gsub(/[\\']/) { |c| "\\#{c}" }}'" }.join(" ")
+      end
+
       # The keywords of a libpq connection string or URI. A failure names
       # what was being read and never the string or libpq's message, which
       # can quote part of a password (as Trellis's own parse does).
@@ -183,7 +300,8 @@ module Trellis
         hash = hash.transform_keys(&:to_sym)
         url = hash[:url] ? conninfo(hash[:url], "config.trellis.dump_database's url") : {}
         given = { host: hash[:host] || url["host"], port: hash[:port] || url["port"],
-                  dbname: hash[:database] || url["dbname"], user: hash[:username] || url["user"] }
+                  dbname: hash[:database] || hash[:dbname] || url["dbname"],
+                  user: hash[:username] || hash[:user] || url["user"] }
         defaults = PG::Connection.conndefaults_hash
         resolved = given.to_h do |key, value|
           value = defaults[key] if value.nil? || value.to_s.empty?
@@ -237,13 +355,11 @@ module Trellis
       def dump_schema(db_config, *args)
         return super unless SchemaDump.applies?(db_config) && db_config.schema_dump
 
-        format = args.first ||
-                 (db_config.respond_to?(:schema_format) ? db_config.schema_format : ::ActiveRecord.schema_format)
-
+        format = SchemaDump.format_of(db_config, args)
         unless format.to_sym == :sql
           raise Trellis::ValidationError,
                 "#{db_config.name}'s schema dump is #{format}, which can't carry Trellis's " \
-                "tables, triggers or definitions: set config.active_record.schema_format = :sql"
+                "definitions: set config.active_record.schema_format = :sql"
         end
 
         filename = schema_dump_path(db_config, format)
@@ -251,17 +367,45 @@ module Trellis
 
         SchemaDump.verify_same_database!(db_config)
 
-        SchemaDump::ATTEMPTS.times do
-          before = SchemaDump.appendix
-          super(db_config, format)
-          if SchemaDump.appendix == before
-            SchemaDump.finish(filename, before)
-            return nil
+        previous = File.exist?(filename) ? File.binread(filename) : nil
+        finished = false
+        begin
+          SchemaDump::ATTEMPTS.times do
+            before = SchemaDump.parts
+            begin
+              Thread.current[SchemaDump::PARTS_KEY] = before
+              super(db_config, format)
+            ensure
+              Thread.current[SchemaDump::PARTS_KEY] = nil
+            end
+            if SchemaDump.parts == before
+              SchemaDump.finish(filename, before)
+              finished = true
+              return nil
+            end
           end
+          raise Trellis::ConflictError,
+                "Trellis's definitions changed during each of #{SchemaDump::ATTEMPTS} dumps of " \
+                "#{db_config.name}: run the dump when no define or drop is running"
+        ensure
+          # Whatever Rails wrote is raw `pg_dump` output with the capture
+          # triggers in and no definitions: left in place, a later
+          # `db:schema:load` would load it without a word. A file no
+          # `pg_dump` wrote to (a failure before the first, or a `pg_dump`
+          # that failed before opening it) is the previous dump: left alone.
+          SchemaDump.discard_unfinished(filename, previous) unless finished
+        end
+      end
+
+      # After Rails loads the dump database's `structure.sql`, restores the
+      # definitions it queued (SchemaDump.restore).
+      def load_schema(db_config, *args)
+        result = super
+        if SchemaDump.applies?(db_config) && db_config.schema_dump &&
+           SchemaDump.format_of(db_config, args).to_sym == :sql
+          SchemaDump.restore(db_config)
         end
-        raise Trellis::ConflictError,
-              "Trellis's definitions changed during each of #{SchemaDump::ATTEMPTS} dumps of " \
-              "#{db_config.name}: run the dump when no define or drop is running"
+        result
       end
     end
 

data/lib/trellis/values.rb

@@ -78,6 +78,13 @@ module Trellis
   # Trellis.self_check refuses to walk the target unless it opted out, when
   # the walk scans the source once per page.
   #
+  # extreme_index_gaps is a warning too: the ExtremeIndexGaps for the source
+  # indexes a MIN/MAX aggregate's probes read and the source lacks ([] when
+  # it has no MIN or MAX, or they exist). Define and a resume refuse a
+  # MIN/MAX definition without them, so a gap is an index dropped since.
+  # Nothing is paused: when a group's extreme is deleted, the next one is
+  # re-read by scanning the group instead, exact but slower.
+  #
   # correcting lists the CorrectingNodes of its partitioned source it is being
   # corrected for, oldest first ([] when none): a partition was attached,
   # detached, dropped or truncated, and the definition re-derives the keys
@@ -92,15 +99,56 @@ module Trellis
   # over its frozen values, so it is stale against the source. Nothing is
   # paused or rebuilt: resuming the column writes through to the reader and
   # it reads :live again.
+  #
+  # build_wait_xid is set while its build waits on something other than its
+  # own work: the transaction id of a fence. The build's source is another
+  # definition's target, and no chunk runs until every transaction that was
+  # open when the fence was taken has ended (a writer of that target could
+  # otherwise commit after a chunk's snapshot without the build seeing its
+  # rows). Look in pg_stat_activity for a transaction with a backend_xid at
+  # or below it; the wait clears on its own when that transaction ends. nil
+  # when the build waits on nothing.
+  #
+  # version_handover is set while a process of another Trellis version is
+  # live on the instance (a VersionHandover): this process stands by (it
+  # stages, drains and builds nothing) and its state-changing calls raise
+  # VersionHandoverError until the other version is gone. A process killed
+  # without a clean shutdown keeps its row live, and so delays the takeover,
+  # for at most 30 seconds. nil when none is.
+  #
+  # below_catalog_floor is set while this binary's version is below the
+  # instance's catalog floor, the oldest Trellis version that can read its
+  # catalog (a BelowCatalogFloor): this process stands by (it stages, drains
+  # and builds nothing, and checks again every standby poll) and its
+  # state-changing calls raise VersionFloorError. Run a version at or above
+  # the floor. nil when this binary meets it.
+  #
+  # join is set for a JOIN definition, aggregate or 1-1, and nil for any
+  # other (a JoinStatus). It never holds anything up: it changes nothing
+  # about the definition's status. It reports the estimated size of the
+  # definition's hidden tables, the P-size warning, the fan-out warning of a
+  # 1-1 JOIN and the step its build runs.
+  #
+  # floor is the oldest Trellis release that can run the definition, as
+  # "major.minor.patch": the latest release any construct, key space, option
+  # or hidden-table layout it uses arrived in. Trellis records it when the
+  # definition is defined, altered or rebuilt by a resume. Before rolling the
+  # application back to an earlier Trellis release, drop each definition
+  # whose floor is above that release, under the newer one.
   Status = Data.define(:status, :backfill_failure, :capture_wait, :capture_failure,
                        :held_keys, :drain_failure, :source_index_gaps,
-                       :correcting, :paused_upstream_columns) do
+                       :extreme_index_gaps, :correcting, :paused_upstream_columns,
+                       :build_wait_xid, :version_handover, :below_catalog_floor,
+                       :join, :floor) do
     def self.from_native(hash)
       failure = hash[:backfill_failure]
       wait = hash[:capture_wait]
       capture_failure = hash[:capture_failure]
       held_keys = hash[:held_keys]
       drain_failure = hash[:drain_failure]
+      version_handover = hash[:version_handover]
+      below_catalog_floor = hash[:below_catalog_floor]
+      join = hash[:join]
       new(status: hash[:status],
           backfill_failure: failure && BackfillFailure.from_native(failure),
           capture_wait: wait && CaptureWait.from_native(wait),
@@ -108,11 +156,116 @@ module Trellis
           held_keys: held_keys && HeldKeys.from_native(held_keys),
           drain_failure: drain_failure && DrainFailure.from_native(drain_failure),
           source_index_gaps: hash.fetch(:source_index_gaps).map { |g| SourceIndexGap.new(**g) },
+          extreme_index_gaps: hash.fetch(:extreme_index_gaps).map { |g| ExtremeIndexGap.new(**g) },
           correcting: hash.fetch(:correcting).map { |c| CorrectingNode.from_native(c) },
-          paused_upstream_columns: hash.fetch(:paused_upstream_columns))
+          paused_upstream_columns: hash.fetch(:paused_upstream_columns),
+          build_wait_xid: hash.fetch(:build_wait_xid),
+          version_handover: version_handover && VersionHandover.from_native(version_handover),
+          below_catalog_floor: below_catalog_floor && BelowCatalogFloor.new(**below_catalog_floor),
+          join: join && JoinStatus.from_native(join),
+          floor: hash.fetch(:floor))
+    end
+  end
+
+  # A JOIN definition's hidden tables, as Trellis.status reports them: the
+  # estimated size of each (hidden_tables, HiddenTables: P, M, J and T, or R, M
+  # and J for a 1-1 JOIN, a table that doesn't exist left out), a warning when
+  # P has grown close to the fact table's size (partials_warning, a
+  # PartialsWarning), a 1-1 JOIN's warnings that one join key holds a very
+  # large group of rows (fan_out_warnings, FanOutWarnings, one per joined
+  # position past the threshold, in position order) and the step its build
+  # runs now, or runs next while no worker holds its job (building, a
+  # JoinBuildStep). partials_warning and building are nil, and
+  # fan_out_warnings is empty, when there is none.
+  # Sizes are the planner's estimates, as fresh as the table's last ANALYZE or
+  # autovacuum.
+  JoinStatus = Data.define(:hidden_tables, :partials_warning, :fan_out_warnings, :building) do
+    def self.from_native(hash)
+      partials_warning = hash.fetch(:partials_warning)
+      building = hash.fetch(:building)
+      new(hidden_tables: hash.fetch(:hidden_tables).map { |t| HiddenTable.new(**t) },
+          partials_warning: partials_warning && PartialsWarning.new(**partials_warning),
+          fan_out_warnings: hash.fetch(:fan_out_warnings).map { |w| FanOutWarning.new(**w) },
+          building: building && JoinBuildStep.new(**building))
+    end
+  end
+
+  # One hidden table of a JOIN, in a JoinStatus: which table it is (table,
+  # "P", "M", "J", "T" or a 1-1 JOIN's "R"), its name in the catalog schema
+  # (name), its estimated live rows (estimated_rows, 0 before the first
+  # analyze) and its total size on disk with its indexes, in bytes (bytes).
+  HiddenTable = Data.define(:table, :name, :estimated_rows, :bytes)
+
+  # P, the FROM table grouped by the definition's GROUP BY columns and join
+  # key, has grown close to the fact table's size, so the JOIN costs about as
+  # much as the table it summarises: P's estimated rows (partials_rows) reach
+  # half of the fact table's (fact_rows).
+  PartialsWarning = Data.define(:partials_rows, :fact_rows)
+
+  # A 1-1 JOIN has a join key with a very large group of a rows at one joined
+  # position: a change of that position's row on the key rewrites that many
+  # target rows, in chunks, before a watermark token past it converges. It
+  # names the joined position (position, 1 for the first table joined), the
+  # join key's text (key, its columns'
+  # values joined by ","), the group's live rows (rows, exact up to ten times
+  # the threshold) and the threshold it passed (threshold). The group is found
+  # by sampling, so a group just past the threshold can go unreported until it
+  # grows.
+  FanOutWarning = Data.define(:position, :key, :rows, :threshold)
+
+  # The step a JOIN build runs, in a JoinStatus. step is the kind:
+  #
+  # * :whole, the whole build in one transaction, because neither side is a
+  #   partitioned table with rows.
+  # * :side, a step of a build with a partitioned side. side is :from or
+  #   :joined, and part says what it builds of that side: :whole, one :leaf
+  #   (named by leaf, a schema.table; leaves build in oid order), the
+  #   :remainder (the cells that route to none of its leaves) or :unroutable
+  #   (the side's partition tree isn't recorded, or a leaf has no usable
+  #   constraint text, so the build fails on this side and backfill_failure
+  #   says why).
+  # * :final, the step that writes T and the target and completes the build.
+  # * A 1-1 JOIN's steps: :joined_rows, :joined_null_keys and :joined_gone for
+  #   the pass over the M of the joined position position (1 for the first
+  #   table joined), :planning_chunks while the plan job walks a's primary
+  #   key, :chunks while the chunks of that key run (remaining of them are
+  #   still to run), and :sweep_stale and :sweep_orphans for the two passes of
+  #   the sweep that ends a rebuild in place.
+  # * :unknown, a step added since this binding was built; side and part read
+  #   :unknown likewise for an unknown side or part.
+  #
+  # side, part, leaf, remaining and position are nil for the steps that don't
+  # carry them.
+  JoinBuildStep = Data.define(:step, :side, :part, :leaf, :remaining, :position)
+
+  # Another Trellis version is live on the instance, so this process changes
+  # nothing until it is gone, as Trellis.status reports it: this binary's
+  # version (this_version), the live processes holding it back (processes,
+  # OtherProcesses), whether the staging-worker lock is held by a session no
+  # registered process names, which is a staging worker of a release before
+  # 0.2.0 (unregistered_staging_worker), and whether the migrations this
+  # binary carries are pending (migrations_pending), in which case every
+  # live process holds it back, whatever its version.
+  VersionHandover = Data.define(:this_version, :processes, :unregistered_staging_worker,
+                                :migrations_pending) do
+    def self.from_native(hash)
+      new(this_version: hash.fetch(:this_version),
+          processes: hash.fetch(:processes).map { |p| OtherProcess.new(**p) },
+          unregistered_staging_worker: hash.fetch(:unregistered_staging_worker),
+          migrations_pending: hash.fetch(:migrations_pending))
     end
   end
 
+  # A live process of another Trellis version, in a VersionHandover: its
+  # worker id (worker_id, trellis-<host>-<pid>-<random>) and its binary's
+  # version (version, nil for a release before 0.2.0, which records none).
+  OtherProcess = Data.define(:worker_id, :version)
+
+  # This binary's version is below the instance's catalog floor, so this
+  # process changes nothing, as Trellis.status reports it: the floor the
+  # catalog records (floor) and this binary's version (this_version).
+  BelowCatalogFloor = Data.define(:floor, :this_version)
+
   # A node of a partitioned source a definition is being corrected for, as
   # Trellis.status reports it: the schema-qualified node (node), when the
   # staging worker queued its correction (since, a Time), and the keys still
@@ -135,6 +288,14 @@ module Trellis
   # (fix). Trellis doesn't create indexes on your tables.
   SourceIndexGap = Data.define(:table, :columns, :role, :opted_out, :fix)
 
+  # A source index a MIN/MAX field's probe reads and the source lacks, as
+  # Trellis.status reports it: the qualified table (table), the columns the
+  # index should lead with, the GROUP BY columns then the argument (columns),
+  # the source column the MIN/MAX reads (argument), whether the definition
+  # opted out with WITHOUT SOURCE INDEX (opted_out, which define refuses for
+  # MIN/MAX), and a create index statement to run (fix).
+  ExtremeIndexGap = Data.define(:table, :columns, :argument, :opted_out, :fix)
+
   # A drain page that keeps failing with nothing charged or paused, as
   # Trellis.status and Trellis.self_check report it: the segment whose page
   # fails (seg_seq), the qualified source tables it holds changes to
@@ -175,7 +336,8 @@ module Trellis
   end
 
   # Why capture of source_table is broken, or why the drain (or a build's
-  # merge) halted on the definition: kind is :capture or :halt, error is a
+  # merge) halted on the definition: kind is :capture, :halt, :disruption, or
+  # :unknown for a kind a newer Trellis release stored, error is a
   # sentence naming the cause, columns the columns it is about (empty when it
   # isn't about a column, and for a halt), and detected_at (a Time) when it
   # was first found.

data/lib/trellis/version.rb

@@ -3,5 +3,5 @@
 module Trellis
   # The trellis-pg gem's version: the Trellis release's, which a
   # `v<VERSION>` tag releases (docs/releasing.md).
-  VERSION = "0.1.1"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: trellis-pg
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.2.0
 platform: ruby
 authors:
 - Salesforce