exwiw 0.9.19 → 0.9.20

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f565edc88b4eba108ef4d9accf7bbaa7d48f0e1700007cc9d280b0033e357b6
-  data.tar.gz: 822e5cd0b9d33c81d0b396cc641fa960cf80bffbebd6bd078322460a140b37cf
+  metadata.gz: 328fbb843d5c15da7ea9f764adda96a5c58b4d24de615b9cac38ba305550ed05
+  data.tar.gz: 691448e490f27475d1ae17be240f89af5ea7c3ac54ddedfa544ca1b71517f5c1
 SHA512:
-  metadata.gz: 20da3c9ee09865762f40a04b6c4bf6cf24582545ba7c51945a69aed69798906a19a214c9bd2ad588e0b6740f29a3ebfa08a122845f9a28fe64b654ba723088a0
-  data.tar.gz: db91db3a7e1774d386f45d3582301098a23df86f48fe790fae3631895b87fe686dcce3d048a17827f69ce530f948ca423bb9028f8a4b2495d167e820edd6f3fb
+  metadata.gz: ee57c7925b2b912a3e9777947d251b8870f94572ec96b5d7fbd0a8d60dbb8aaefd33cde4e12e735b2ff2eb44b24e2f101b81e4bdfc78752f55788619a4ce5a6c
+  data.tar.gz: 9f3a2b95adb72322032b7ac4967d69e3c32a0df6fa75c64adc7f59be1f9de6b7958a8250c0abb2fd9857a5c063d8123f71c2d9678376757d1161ef34f484f368

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+## [0.9.20] - 2026-08-05
+
+### Changed
+
+- **PostgreSQL: the extensions a managed platform installs to run the source instance itself (Cloud SQL / AlloyDB) are now explicitly out of target and left out of `insert-000-schema.sql`.** A full-database `pg_dump` of a Cloud SQL instance emits `CREATE SCHEMA google_vacuum_mgmt` plus the `CREATE EXTENSION` / `COMMENT ON EXTENSION` for it; an AlloyDB one emits `google_columnar_engine` and `google_db_advisor`. exwiw already made these survivable — earlier releases wrapped both statements in `DO` blocks that warn-and-skip when the restore target cannot provide the extension — but they still occupied the dump, left an empty vendor schema behind on every restore, and logged a `WARNING` each time. They serve the *source* instance's operation (vacuum tuning, the in-memory columnar cache, index advice), carry no application data, are named by nothing in the application's own schema, and ship only with the managed platform, so no restore target outside it can ever create them. Both halves are now dropped: their schemas via `pg_dump --exclude-schema` (which also covers any object the platform adds inside one later), and the extension statements — not schema-qualified, so no `pg_dump` filter reaches them — via a new `DdlPostprocessor.strip_extensions` pass, with whatever was excluded named in the run's log rather than dropped silently. The set is an explicit list of names (`PostgresqlAdapter::PLATFORM_MANAGED_EXTENSIONS`), deliberately not a `google_*` prefix match: the prefixes are not reserved, so a prefix rule would also drop a schema an application legitimately owns — `google_calendar` for a Google Calendar integration — together with its tables. Adding a name costs a release, which is the right trade. Every other extension is still emitted wrapped in the warn-and-skip `DO` block, including two kinds that are also managed-platform-only: a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`, installable on any plain PostgreSQL), and an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which application SQL and dumped DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it. Data extraction never touched any of these objects — it is driven by the schema config, which has no entry for a table without an application model — so no `insert-*` / `delete-*` file changes.
+
 ## [0.9.19] - 2026-08-04
 
 ### Added

data/README.md

@@ -108,6 +108,13 @@ so you should import the dump in order.
 
 `insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. For `mysql`, set `EXWIW_MYSQLDUMP` to point at a specific `mysqldump` binary when the one on PATH is incompatible with the server (e.g. a MySQL 9.x `mysqldump` cannot load `mysql_native_password` against a server still using that auth plugin — `EXWIW_MYSQLDUMP=/path/to/mysql@8.0/bin/mysqldump`). The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`. For `mysql`, the source server's `DEFINER=user@host` stamp on views and triggers is stripped too, so restoring into a managed MySQL instance (which usually can't grant the privilege to recreate someone else's `DEFINER`) does not fail.
 
+For `postgresql`, the extensions a managed platform installs to run the source instance itself are treated as out of target and left out of the dump entirely — currently `google_vacuum_mgmt` (Cloud SQL / AlloyDB adaptive autovacuum), `google_columnar_engine` and `google_db_advisor` (AlloyDB). They serve the source instance's operation (vacuum tuning, the in-memory columnar cache, index advice), hold no application data, are referenced by nothing in the application's own schema, and ship only with the managed platform, so a restore target outside it can never create them. Their schemas are dropped via `pg_dump --exclude-schema` and their `CREATE EXTENSION` / `COMMENT ON EXTENSION` statements — which are not schema-qualified, so no `pg_dump` filter reaches them — are removed from the output; whatever was excluded is named in the run's log.
+
+The list is exact names, not a `google_*` prefix match: those prefixes are not reserved, so a prefix rule would also drop a schema an application legitimately owns (`google_calendar` for a Google Calendar integration) together with its tables. Every other extension is kept and wrapped in the usual warn-and-skip `DO` block, including two kinds that are also managed-platform-only:
+
+- a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`), since that one *is* installable on a plain PostgreSQL, and
+- an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which the application's own SQL and DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it, so it warns and skips instead.
+
 you need to delete the records before importing the dump,
 `delete-{idx}-{table_name}.sql` will help you to do that.
 This sql will delete "all" related records to the extract targets.

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -98,6 +98,41 @@ module Exwiw
         connection.exec("EXPLAIN #{sql}").values.map(&:first).join("\n")
       end
 
+      # Extensions a managed PostgreSQL platform installs to run the *source*
+      # instance itself, which exwiw treats as out of target and leaves out of the
+      # dump entirely. Each one here satisfies both conditions:
+      #
+      #   1. it ships only with the managed platform, so no restore target outside
+      #      that platform can create it, and
+      #   2. nothing in the application's own schema or queries references it — it
+      #      is operational machinery (autovacuum tuning, the in-memory columnar
+      #      cache, index advice), not a feature the app builds on.
+      #
+      # Condition 2 is what keeps this list short. AlloyDB's application-facing
+      # extensions — `google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl` —
+      # meet condition 1 but are called from application SQL and can be named by
+      # dumped DDL (a ScaNN index is `USING scann`), so dropping their CREATE would
+      # silently strand whatever refers to them. They stay in the dump, wrapped in
+      # the usual warn-and-skip DO block, like `pglogical` and like a third-party
+      # extension pulled in as a dependency of one listed here (`google_db_advisor`
+      # requires `hypopg`, which any plain PostgreSQL can install).
+      #
+      # Deliberately an explicit list of names, not a `google_`/`alloydb_` prefix
+      # match (which is how the old `pg_extension` query filtered, before switching
+      # to a full-database pg_dump dropped that query and the filter with it): the
+      # prefixes are not reserved, so a schema an application legitimately owns —
+      # `google_calendar` for a Google Calendar integration — would be matched and
+      # dropped together with its tables. The vendor gains extensions faster than
+      # this list does; that costs a release, which is the right trade against
+      # deleting an application's own objects. Its `rds_%` / `aiven_%` arms are not
+      # carried over for the same reason: naming their members takes a dump to
+      # confirm, and adding one here is a one-line change.
+      PLATFORM_MANAGED_EXTENSIONS = %w[
+        google_columnar_engine
+        google_db_advisor
+        google_vacuum_mgmt
+      ].freeze
+
       def dump_schema(ordered_tables, output_path)
         require 'open3'
 
@@ -117,10 +152,20 @@ module Exwiw
           '--schema-only',
           '--no-owner',
           '--no-acl',
+          # An extension that needs a schema of its own puts it under its own name
+          # (Cloud SQL's google_vacuum_mgmt does; the AlloyDB two install into
+          # public), so excluding the same names covers the schema half — including
+          # any object the platform adds inside one later. The patterns are exact
+          # names: pg_dump reads them psql \d-style, where `_` is literal and only
+          # `*` globs. A pattern matching nothing is not an error (that needs
+          # --strict-names), so these are harmless against a self-hosted server.
+          *PLATFORM_MANAGED_EXTENSIONS.map { |name| "--exclude-schema=#{name}" },
           @connection_config.database_name,
         ]
         env = { 'PGPASSWORD' => @connection_config.password.to_s }
 
+        log_excluded_platform_managed_schemas
+
         @logger.debug("  Running pg_dump for the whole database (#{@connection_config.database_name})...")
         stdout, stderr, status = Open3.capture3(env, *cmd)
         unless status.success?
@@ -141,7 +186,9 @@ module Exwiw
         # EXTENSION that pg_dump emits alongside is likewise wrapped to swallow
         # undefined_object, so a skipped extension's trailing comment does not
         # abort the restore either.
-        idempotent = stdout
+        # Platform-managed extensions are stripped first: the wrapping passes below
+        # rewrite the bare CREATE/COMMENT statements this removes.
+        idempotent = strip_platform_managed_extensions(stdout)
         idempotent = DdlPostprocessor.wrap_create_type_enum_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_create_extension_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_comment_on_extension_in_do_block(idempotent)
@@ -159,6 +206,37 @@ module Exwiw
         @logger.info("  Wrote full-database schema to #{output_path} (#{ordered_tables.size} table(s) in scope for data).")
       end
 
+      # Drop the platform-managed extensions from a raw pg_dump, naming the ones
+      # actually removed so the run records what it left out rather than silently
+      # dropping it. See PLATFORM_MANAGED_EXTENSIONS.
+      private def strip_platform_managed_extensions(sql)
+        removed = DdlPostprocessor.extension_names(sql) & PLATFORM_MANAGED_EXTENSIONS
+        unless removed.empty?
+          @logger.info("  Excluded platform-managed extension(s) from the schema dump: #{removed.join(', ')}.")
+        end
+
+        DdlPostprocessor.strip_extensions(sql, PLATFORM_MANAGED_EXTENSIONS)
+      end
+
+      # Name the --exclude-schema patterns the source instance actually has, so the
+      # run records the schema half of the exclusion the way
+      # #strip_platform_managed_extensions records the extension half. Only exact
+      # names are excluded, but nothing stops an application from owning one of
+      # them, and excluding a schema takes every object inside it along — that must
+      # not happen without the log saying so. See PLATFORM_MANAGED_EXTENSIONS.
+      private def log_excluded_platform_managed_schemas
+        # The names are bare identifiers from the constant, so they need no quoting
+        # inside the array literal ANY() takes.
+        result = connection.exec_params(
+          'SELECT nspname FROM pg_namespace WHERE nspname = ANY($1) ORDER BY nspname',
+          ["{#{PLATFORM_MANAGED_EXTENSIONS.join(',')}}"],
+        )
+        excluded = result.values.map(&:first)
+        return if excluded.empty?
+
+        @logger.info("  Excluded platform-managed schema(s) from the schema dump: #{excluded.join(', ')}.")
+      end
+
       # The INSERT header for this adapter. PostgreSQL uses bare identifiers,
       # quoted only when required (reserved word / unsafe characters).
       # #to_bulk_insert / #write_inserts (SqlBulkInsert) append the value tuples

data/lib/exwiw/ddl_postprocessor.rb

@@ -170,6 +170,50 @@ module Exwiw
       end
     end
 
+    # Every extension name a dump installs, in the order the CREATE EXTENSION
+    # statements appear. Used to report which of them #strip_extensions is about
+    # to drop; run it on the raw dump, before any wrapping pass.
+    def extension_names(sql)
+      sql.scan(CREATE_EXTENSION_RE).map { |(name)| name.delete('"') }
+    end
+
+    # Drop every trace of the extensions named in `names` — the ones a managed
+    # PostgreSQL platform installs to run the source instance itself (see
+    # PostgresqlAdapter::PLATFORM_MANAGED_EXTENSIONS). Removed rather than merely
+    # wrapped like any other extension, because they cannot be created anywhere
+    # outside that platform, so keeping them only leaves a restore-time WARNING
+    # and objects the target will never have.
+    #
+    # Three things are emitted per extension and all three must go, or a leftover
+    # references a name that is no longer installed: pg_dump's `-- Name: <ext>;
+    # Type: EXTENSION` header block, the `CREATE EXTENSION`, and the
+    # `COMMENT ON EXTENSION` (whose own header reads `-- Name: EXTENSION <ext>;
+    # Type: COMMENT`). Run this on the raw dump, before the wrapping passes: they
+    # rewrite the bare statements this matches into DO blocks.
+    def strip_extensions(sql, names)
+      return sql if names.empty?
+
+      # Whole names only: each match is anchored by the whitespace/quote before it
+      # and a \b after, so `google_vacuum_mgmt` never matches an extension merely
+      # containing it (`not_google_vacuum_mgmt`, `google_vacuum_mgmt_v2`).
+      names_re = Regexp.union(names.map { |n| Regexp.escape(n) })
+      name = /(?:"#{names_re}"|#{names_re}\b)/
+
+      # Each removal takes the blank lines that trailed the object too, so the
+      # surrounding statements keep pg_dump's spacing instead of gaining a gap.
+      trailing_blank_lines = /(?:[ \t]*\r?\n(?:[ \t]*\r?\n)*)?/
+
+      sql = sql.gsub(
+        /^--\r?\n-- Name: (?:EXTENSION\s+)?#{name};[ \t]*Type:[ \t]*(?:EXTENSION|COMMENT);[^\n]*\n--(?:\r?\n)+/i,
+        "",
+      )
+      sql = sql.gsub(/^[ \t]*CREATE\s+EXTENSION\b(?:\s+IF\s+NOT\s+EXISTS)?\s+#{name}[^;]*;#{trailing_blank_lines}/i, "")
+      sql.gsub(
+        /^[ \t]*COMMENT\s+ON\s+EXTENSION\s+#{name}\s+IS\s+(?:'(?:[^']|'')*'|NULL)\s*;#{trailing_blank_lines}/i,
+        "",
+      )
+    end
+
     # Generate idempotent CREATE TYPE ... AS ENUM statements.
     # +enum_types+ is an Array of Hashes with keys :schema, :name, :labels.
     def create_type_enum_statements(enum_types)

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.19"
+  VERSION = "0.9.20"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.19
+  version: 0.9.20
 platform: ruby
 authors:
 - Shia