exwiw 0.9.18 → 0.9.20

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 329bc5d4ceae914d8ba650d64b00a9d83916b1aa2f753cab6310ea61d3c6989a
-  data.tar.gz: 5f404952e90a272b7b4818257901f8e3dec7cbcdb0e4ae63310c728374790dad
+  metadata.gz: 328fbb843d5c15da7ea9f764adda96a5c58b4d24de615b9cac38ba305550ed05
+  data.tar.gz: 691448e490f27475d1ae17be240f89af5ea7c3ac54ddedfa544ca1b71517f5c1
 SHA512:
-  metadata.gz: 77b947ce4141e5ed17e6faf4a1f6e4f910e8f0a70d96274fff3e7d33be11a1adeed2fa2ddbe59787ce6f9b4572ab969432a50bcccd2efb758e1c1c423dedb8cd
-  data.tar.gz: de8c14e52a77e610d950381af8f0028b4e78c0f572e0e7f1bcbeda37d9c824c1d04ddf6683fe9e80150631dc56df9aac340598b30c53eba25d207c0bad8057f4
+  metadata.gz: ee57c7925b2b912a3e9777947d251b8870f94572ec96b5d7fbd0a8d60dbb8aaefd33cde4e12e735b2ff2eb44b24e2f101b81e4bdfc78752f55788619a4ce5a6c
+  data.tar.gz: 9f3a2b95adb72322032b7ac4967d69e3c32a0df6fa75c64adc7f59be1f9de6b7958a8250c0abb2fd9857a5c063d8123f71c2d9678376757d1161ef34f484f368

data/CHANGELOG.md

@@ -2,6 +2,18 @@
 
 ## [Unreleased]
 
+## [0.9.20] - 2026-08-05
+
+### Changed
+
+- **PostgreSQL: the extensions a managed platform installs to run the source instance itself (Cloud SQL / AlloyDB) are now explicitly out of target and left out of `insert-000-schema.sql`.** A full-database `pg_dump` of a Cloud SQL instance emits `CREATE SCHEMA google_vacuum_mgmt` plus the `CREATE EXTENSION` / `COMMENT ON EXTENSION` for it; an AlloyDB one emits `google_columnar_engine` and `google_db_advisor`. exwiw already made these survivable — earlier releases wrapped both statements in `DO` blocks that warn-and-skip when the restore target cannot provide the extension — but they still occupied the dump, left an empty vendor schema behind on every restore, and logged a `WARNING` each time. They serve the *source* instance's operation (vacuum tuning, the in-memory columnar cache, index advice), carry no application data, are named by nothing in the application's own schema, and ship only with the managed platform, so no restore target outside it can ever create them. Both halves are now dropped: their schemas via `pg_dump --exclude-schema` (which also covers any object the platform adds inside one later), and the extension statements — not schema-qualified, so no `pg_dump` filter reaches them — via a new `DdlPostprocessor.strip_extensions` pass, with whatever was excluded named in the run's log rather than dropped silently. The set is an explicit list of names (`PostgresqlAdapter::PLATFORM_MANAGED_EXTENSIONS`), deliberately not a `google_*` prefix match: the prefixes are not reserved, so a prefix rule would also drop a schema an application legitimately owns — `google_calendar` for a Google Calendar integration — together with its tables. Adding a name costs a release, which is the right trade. Every other extension is still emitted wrapped in the warn-and-skip `DO` block, including two kinds that are also managed-platform-only: a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`, installable on any plain PostgreSQL), and an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which application SQL and dumped DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it. Data extraction never touched any of these objects — it is driven by the schema config, which has no entry for a table without an application model — so no `insert-*` / `delete-*` file changes.
+
+## [0.9.19] - 2026-08-04
+
+### Added
+
+- **`batch_scope`: extract a table as one query per slice of the scope's id set, so a very large table stays index-driven instead of degrading into a full scan.** A scoped table reached through a `belongs_to` hop compiles to `SELECT t.* FROM t JOIN parent ON t.parent_id = parent.id AND parent.tenant_id IN (...)`, which is index-driven while the scope keeps few parent rows. Past some number of them the planner's estimate of "probe the foreign-key index once per parent row" exceeds its estimate of "scan the table once" and it switches to a sequential scan of the whole table — for a result set that is a small fraction of it. On a table of hundreds of millions of rows that scan exceeds the server's `statement_timeout` (or runs for hours), and a `filter` on the extracted table does not help: a predicate that reduces the *output* does not reduce the *work* once the plan is a scan. `batch_scope: { "table": "<scoped table>", "size": 1000 }` names the scoped table this one reaches, resolves that table's in-scope primary keys once (from its own extraction query, so it is narrowed by exactly the filter it would carry anyway), and extracts one `size`-sized slice of those ids at a time, each batch carrying `<batch table>.<pk> IN (<ids>)` in place of the scope filter. A literal id list of that size is exactly estimated and selective, so the foreign-key index is unambiguously the cheapest plan for every batch, and total work becomes proportional to the rows the table keeps rather than to the table's size. **The dumped rows are the same as the unbatched query's**, in batch-by-batch order: the slices partition the id set, so no row is dropped or emitted twice, and the ids are sorted in exwiw before slicing (not via `ORDER BY`, which would push a sort onto the source DB), so the batched output is reproducible run to run. The batch table may be any number of hops up the path (a table two hops below it names the same batch table, and the ids are applied where the path meets the scope, bounding the whole join chain), or the table itself when it carries the scope column. Because a batch key only splits an extraction correctly when *every* row the table keeps is selected through the batch table's scope filter, the supported shapes are deliberately narrow — scope-column mode, and either a directly scoped table naming itself or a single `belongs_to` join path terminating at the named table. Polymorphic arm `UNION`s, `reverse_scope`, referenced-by, the parent cascade, `scope_exempt` (on either side — an exempt batch table's id set would not be scoped, so its batches would reach outside the scope) and single `--target-table` mode are rejected with an explanation rather than silently mis-sliced (they keep rows by routes a batch of ids does not constrain, so every batch would re-emit them), and the rejection happens in the pre-flight validation, before any output is written. `exwiw explain` additionally prints the id-set query and its `EXPLAIN` for a batched table; it cannot show a batch's literal ids, since it executes no extraction SELECT. `delete-*.sql` is generated from the unbatched query as before, `bulk_insert_chunk_size` is independent (batches are query boundaries, chunks are statement boundaries), and output for every table without `batch_scope` is byte-identical.
+
 ## [0.9.18] - 2026-08-03
 
 ### Fixed

data/README.md

@@ -108,6 +108,13 @@ so you should import the dump in order.
 
 `insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. For `mysql`, set `EXWIW_MYSQLDUMP` to point at a specific `mysqldump` binary when the one on PATH is incompatible with the server (e.g. a MySQL 9.x `mysqldump` cannot load `mysql_native_password` against a server still using that auth plugin — `EXWIW_MYSQLDUMP=/path/to/mysql@8.0/bin/mysqldump`). The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`. For `mysql`, the source server's `DEFINER=user@host` stamp on views and triggers is stripped too, so restoring into a managed MySQL instance (which usually can't grant the privilege to recreate someone else's `DEFINER`) does not fail.
 
+For `postgresql`, the extensions a managed platform installs to run the source instance itself are treated as out of target and left out of the dump entirely — currently `google_vacuum_mgmt` (Cloud SQL / AlloyDB adaptive autovacuum), `google_columnar_engine` and `google_db_advisor` (AlloyDB). They serve the source instance's operation (vacuum tuning, the in-memory columnar cache, index advice), hold no application data, are referenced by nothing in the application's own schema, and ship only with the managed platform, so a restore target outside it can never create them. Their schemas are dropped via `pg_dump --exclude-schema` and their `CREATE EXTENSION` / `COMMENT ON EXTENSION` statements — which are not schema-qualified, so no `pg_dump` filter reaches them — are removed from the output; whatever was excluded is named in the run's log.
+
+The list is exact names, not a `google_*` prefix match: those prefixes are not reserved, so a prefix rule would also drop a schema an application legitimately owns (`google_calendar` for a Google Calendar integration) together with its tables. Every other extension is kept and wrapped in the usual warn-and-skip `DO` block, including two kinds that are also managed-platform-only:
+
+- a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`), since that one *is* installable on a plain PostgreSQL, and
+- an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which the application's own SQL and DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it, so it warns and skips instead.
+
 you need to delete the records before importing the dump,
 `delete-{idx}-{table_name}.sql` will help you to do that.
 This sql will delete "all" related records to the extract targets.
@@ -738,6 +745,60 @@ Unlike rails-managed entries, `columns` and `belongs_tos` are retained so the en
 
 If omitted, the adapter default applies: 10,000 rows per statement for the SQL adapters (1,000 documents per chunk for MongoDB). Tables at or below the chunk size still produce a single `INSERT` statement. To force a single statement regardless of table size, set a value larger than the table's row count.
 
+### Batched extraction (`batch_scope`)
+
+A scoped table is normally extracted with one query, whose scope filter sits on the table it joins up to:
+
+```sql
+SELECT activities.* FROM activities
+  JOIN customers ON activities.customer_id = customers.id
+                AND customers.tenant_id IN ('t1')
+```
+
+That is index-driven while the scope keeps few `customers`. Past some number of them the planner's estimate of "probe the foreign-key index once per customer" exceeds its estimate of "scan the table once", and it switches to a **sequential scan of the whole table** — for a result set that is a small fraction of it. On a table of hundreds of millions of rows the scan then exceeds the server's `statement_timeout`, or simply runs for hours. Note that no `filter` on the extracted table fixes this: a predicate that reduces the *output* does not reduce the *work* once the plan is a scan (it may not even change the plan).
+
+`batch_scope` removes the choice instead of arguing with the estimate. It names the scoped table this one reaches — the **batch table** — and exwiw resolves that table's in-scope primary keys once, then extracts one `size`-sized slice of those ids at a time:
+
+```json
+{
+  "name": "activities",
+  "primary_key": "id",
+  "batch_scope": { "table": "customers", "size": 1000 },
+  "belongs_tos": [{ "table_name": "customers", "foreign_key": "customer_id" }],
+  "columns": [{ "name": "id" }, { "name": "customer_id" }]
+}
+```
+
+Each batch runs with that slice's ids in place of the scope filter:
+
+```sql
+SELECT activities.* FROM activities
+  JOIN customers ON activities.customer_id = customers.id
+                AND customers.id IN (/* 1000 ids */)
+```
+
+An explicit id list of that size is exactly estimated and selective, so the foreign-key index is unambiguously the cheapest plan for every batch, and total work is proportional to the rows the table actually keeps rather than to the table's size.
+
+- **The dumped rows are the same as the unbatched query's** (in batch-by-batch order). The slices partition the id set — every id is in exactly one batch — so no row is dropped or emitted twice. The ids are sorted (in exwiw, not with `ORDER BY` — the id-set query stays cheap on the source DB) before slicing, so batch composition, and the dump, is reproducible run to run.
+- **`size` defaults to 1000** ids per batch.
+- The batch table's ids come from **its own extraction query**, so it is narrowed by exactly the filter it would carry in the unbatched query. They are held in memory for the extraction: one scope's worth of primary keys, orders of magnitude smaller than the table being batched.
+- The batch table may be **any number of hops up** the path — a table two hops below it (`activity_orders → activities → customers`) names `customers` too, and the batch ids are applied where the path meets the scope, bounding the whole join chain.
+- A table that **carries the scope column itself** batches by naming itself; each batch then filters `WHERE <pk> IN (<ids>)` directly. Note that the id-set query is then the same scope predicate over the same table, so this shape only avoids the scan when the scope column is indexed (ideally index-only) — the join shape above is the one that genuinely removes the planner's choice.
+- `delete-*.sql` is unaffected (it is generated from the unbatched query).
+- `bulk_insert_chunk_size` is independent: batches are query boundaries, chunks are `INSERT` statement boundaries.
+- With `--output-format=copy`, batching bounds each query's cost but not memory: COPY builds the whole table's body in memory, so all batches' rows are resident at once. Use the default INSERT format (which streams) when the kept rows themselves are huge.
+
+**Supported shapes.** A batch key only splits an extraction correctly when *every* row the table keeps is selected through the batch table's scope filter — otherwise a route the batch key does not constrain would keep the same rows in every batch, and the dump would repeat them (a primary-key conflict on import). So `batch_scope` requires [scope-column mode](#scope-column-mode) and one of:
+
+- the table is **directly scoped** (`scope_column`) and names itself, or
+- the table reaches the scope through a **single `belongs_to` join path** (path 2 in [the six scoping paths](#how-each-table-is-narrowed--the-six-scoping-paths)) whose scoped terminus is the named table.
+
+Every other shape — polymorphic arm `UNION`s, `reverse_scope`, referenced-by, the parent cascade, `scope_exempt` (on the batched table *or* the batch table, whose id set would then not be scoped), and single `--target-table` mode — is **rejected with an explanation** rather than silently mis-sliced, before any output is written. (In single-target mode the extraction is already anchored on a caller-supplied id list, so batching it means running exwiw once per slice of `--ids`.)
+
+`exwiw explain` prints the id-set query and its `EXPLAIN` after a batched table's own query, since that query is the part of a batched export the table's query does not show. It cannot show a batch's literal id list — `explain` resolves no ids, because it executes no extraction SELECT.
+
+Like `scope_column` / `scope_exempt` / `reverse_scope`, `batch_scope` is user-maintained: never emitted by `schema:generate`, and preserved across regeneration.
+
 ### Filter
 
 Some case, you don't need full records related to target. e.g. dump user access logs only for the last year.
@@ -745,6 +806,7 @@ Some case, you don't need full records related to target. e.g. dump user access
 
 - injected as it is in table condition(e.g. WHERE on mysql), so you are recommended to clearify table name of column to avoid ambiguity.
 - injected to every where / join clause, so it affects to all tables depends on filterted target-table. it results to data inconsistency.
+- a way to reduce the rows returned, which is **not** necessarily a way to reduce the work: on a large table the engine may keep (or switch to) a full scan and evaluate the filter per row. See [batched extraction](#batched-extraction-batch_scope) when the goal is to bound how much of the table is read.
 
 ### Masking
 

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -98,6 +98,41 @@ module Exwiw
         connection.exec("EXPLAIN #{sql}").values.map(&:first).join("\n")
       end
 
+      # Extensions a managed PostgreSQL platform installs to run the *source*
+      # instance itself, which exwiw treats as out of target and leaves out of the
+      # dump entirely. Each one here satisfies both conditions:
+      #
+      #   1. it ships only with the managed platform, so no restore target outside
+      #      that platform can create it, and
+      #   2. nothing in the application's own schema or queries references it — it
+      #      is operational machinery (autovacuum tuning, the in-memory columnar
+      #      cache, index advice), not a feature the app builds on.
+      #
+      # Condition 2 is what keeps this list short. AlloyDB's application-facing
+      # extensions — `google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl` —
+      # meet condition 1 but are called from application SQL and can be named by
+      # dumped DDL (a ScaNN index is `USING scann`), so dropping their CREATE would
+      # silently strand whatever refers to them. They stay in the dump, wrapped in
+      # the usual warn-and-skip DO block, like `pglogical` and like a third-party
+      # extension pulled in as a dependency of one listed here (`google_db_advisor`
+      # requires `hypopg`, which any plain PostgreSQL can install).
+      #
+      # Deliberately an explicit list of names, not a `google_`/`alloydb_` prefix
+      # match (which is how the old `pg_extension` query filtered, before switching
+      # to a full-database pg_dump dropped that query and the filter with it): the
+      # prefixes are not reserved, so a schema an application legitimately owns —
+      # `google_calendar` for a Google Calendar integration — would be matched and
+      # dropped together with its tables. The vendor gains extensions faster than
+      # this list does; that costs a release, which is the right trade against
+      # deleting an application's own objects. Its `rds_%` / `aiven_%` arms are not
+      # carried over for the same reason: naming their members takes a dump to
+      # confirm, and adding one here is a one-line change.
+      PLATFORM_MANAGED_EXTENSIONS = %w[
+        google_columnar_engine
+        google_db_advisor
+        google_vacuum_mgmt
+      ].freeze
+
       def dump_schema(ordered_tables, output_path)
         require 'open3'
 
@@ -117,10 +152,20 @@ module Exwiw
           '--schema-only',
           '--no-owner',
           '--no-acl',
+          # An extension that needs a schema of its own puts it under its own name
+          # (Cloud SQL's google_vacuum_mgmt does; the AlloyDB two install into
+          # public), so excluding the same names covers the schema half — including
+          # any object the platform adds inside one later. The patterns are exact
+          # names: pg_dump reads them psql \d-style, where `_` is literal and only
+          # `*` globs. A pattern matching nothing is not an error (that needs
+          # --strict-names), so these are harmless against a self-hosted server.
+          *PLATFORM_MANAGED_EXTENSIONS.map { |name| "--exclude-schema=#{name}" },
           @connection_config.database_name,
         ]
         env = { 'PGPASSWORD' => @connection_config.password.to_s }
 
+        log_excluded_platform_managed_schemas
+
         @logger.debug("  Running pg_dump for the whole database (#{@connection_config.database_name})...")
         stdout, stderr, status = Open3.capture3(env, *cmd)
         unless status.success?
@@ -141,7 +186,9 @@ module Exwiw
         # EXTENSION that pg_dump emits alongside is likewise wrapped to swallow
         # undefined_object, so a skipped extension's trailing comment does not
         # abort the restore either.
-        idempotent = stdout
+        # Platform-managed extensions are stripped first: the wrapping passes below
+        # rewrite the bare CREATE/COMMENT statements this removes.
+        idempotent = strip_platform_managed_extensions(stdout)
         idempotent = DdlPostprocessor.wrap_create_type_enum_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_create_extension_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_comment_on_extension_in_do_block(idempotent)
@@ -159,6 +206,37 @@ module Exwiw
         @logger.info("  Wrote full-database schema to #{output_path} (#{ordered_tables.size} table(s) in scope for data).")
       end
 
+      # Drop the platform-managed extensions from a raw pg_dump, naming the ones
+      # actually removed so the run records what it left out rather than silently
+      # dropping it. See PLATFORM_MANAGED_EXTENSIONS.
+      private def strip_platform_managed_extensions(sql)
+        removed = DdlPostprocessor.extension_names(sql) & PLATFORM_MANAGED_EXTENSIONS
+        unless removed.empty?
+          @logger.info("  Excluded platform-managed extension(s) from the schema dump: #{removed.join(', ')}.")
+        end
+
+        DdlPostprocessor.strip_extensions(sql, PLATFORM_MANAGED_EXTENSIONS)
+      end
+
+      # Name the --exclude-schema patterns the source instance actually has, so the
+      # run records the schema half of the exclusion the way
+      # #strip_platform_managed_extensions records the extension half. Only exact
+      # names are excluded, but nothing stops an application from owning one of
+      # them, and excluding a schema takes every object inside it along — that must
+      # not happen without the log saying so. See PLATFORM_MANAGED_EXTENSIONS.
+      private def log_excluded_platform_managed_schemas
+        # The names are bare identifiers from the constant, so they need no quoting
+        # inside the array literal ANY() takes.
+        result = connection.exec_params(
+          'SELECT nspname FROM pg_namespace WHERE nspname = ANY($1) ORDER BY nspname',
+          ["{#{PLATFORM_MANAGED_EXTENSIONS.join(',')}}"],
+        )
+        excluded = result.values.map(&:first)
+        return if excluded.empty?
+
+        @logger.info("  Excluded platform-managed schema(s) from the schema dump: #{excluded.join(', ')}.")
+      end
+
       # The INSERT header for this adapter. PostgreSQL uses bare identifiers,
       # quoted only when required (reserved word / unsafe characters).
       # #to_bulk_insert / #write_inserts (SqlBulkInsert) append the value tuples

data/lib/exwiw/batch_scope.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Opt-in config for batched extraction (see {BatchedExtraction} and the
+  # `batch_scope` section of README.md): `table` is the scoped table whose
+  # in-scope primary keys slice this table's extraction, `size` the ids per batch.
+  class BatchScope
+    include Serdes
+
+    DEFAULT_SIZE = 1_000
+
+    attribute :table, String
+    attribute :size, optional(Integer), skip_serializing_if_nil: true
+    attribute :comment, optional(String), skip_serializing_if_nil: true
+
+    def batch_size
+      size || DEFAULT_SIZE
+    end
+  end
+end

data/lib/exwiw/batched_extraction.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Extracts a table configured with `batch_scope` as one query per slice of the
+  # scope's id set, so each query stays index-driven instead of degrading into a
+  # full scan. Rows are the unbatched query's, in batch order; the slices
+  # partition the id set, so none is dropped or repeated. See README.md.
+  class BatchedExtraction
+    include Enumerable
+
+    attr_reader :terminus
+
+    def self.build(adapter:, table:, dump_target:, table_by_name:, logger:)
+      return nil unless table.respond_to?(:batch_scope) && table.batch_scope
+
+      new(
+        adapter: adapter,
+        table: table,
+        dump_target: dump_target,
+        table_by_name: table_by_name,
+        logger: logger,
+      )
+    end
+
+    def initialize(adapter:, table:, dump_target:, table_by_name:, logger:)
+      @adapter = adapter
+      @table = table
+      @dump_target = dump_target
+      @table_by_name = table_by_name
+      @logger = logger
+      @terminus = QueryAstBuilder
+        .new(table.name, table_by_name, dump_target, logger)
+        .batch_scope_terminus!
+    end
+
+    def batch_size
+      @table.batch_scope.batch_size
+    end
+
+    # The batch table's own extraction query projected to its primary key, so the
+    # ids are narrowed by exactly the filter the unbatched query would carry. The
+    # key is a plain column so masking configured on it cannot corrupt the ids.
+    def key_query_ast
+      @key_query_ast ||= begin
+        scoped = QueryAstBuilder.run(@terminus.name, @table_by_name, @dump_target, @logger)
+
+        QueryAst::Select.new.tap do |ast|
+          ast.from(scoped.from_table_name)
+          ast.select([TableColumn.from_symbol_keys(name: @terminus.primary_key)])
+          scoped.join_clauses.each { |join_clause| ast.join(join_clause) }
+          scoped.where_clauses.each { |where_clause| ast.where(where_clause) }
+        end
+      end
+    end
+
+    def batch_query_ast(ids)
+      QueryAstBuilder.run(@table.name, @table_by_name, @dump_target, @logger, batch_ids: ids)
+    end
+
+    # Resolve the id set and log the plan before extraction starts, so its cost
+    # (and an empty id set) is reported where it happens rather than mid-stream.
+    def prepare!
+      if key_ids.empty?
+        @logger.info("  No in-scope #{@terminus.name} ids to batch by; extracting nothing.")
+      else
+        @logger.info(
+          "  Extracting in #{batch_count} batch(es) of up to #{batch_size} " \
+          "#{@terminus.name}.#{@terminus.primary_key} value(s) (#{key_ids.size} in scope)."
+        )
+      end
+      self
+    end
+
+    # Drained in full (the connection must be free for the batch queries) and
+    # sorted here rather than via ORDER BY, which would push a sort onto the
+    # source DB. Any total order makes the batches reproducible run to run.
+    def key_ids
+      @key_ids ||= @adapter.execute(key_query_ast).map(&:first).sort
+    end
+
+    def batch_count
+      (key_ids.size + batch_size - 1) / batch_size
+    end
+
+    def each
+      return enum_for(:each) unless block_given?
+
+      extracted = 0
+      key_ids.each_slice(batch_size).with_index do |ids, idx|
+        rows = 0
+        @adapter.execute(batch_query_ast(ids)).each do |row|
+          rows += 1
+          yield row
+        end
+        extracted += rows
+        @logger.info("  Batch #{idx + 1}/#{batch_count}: #{rows} record(s), #{extracted} so far.")
+      end
+
+      self
+    end
+
+    # Only the COPY output format needs the count up front, and each batch answers
+    # it with its own count query, so this stays lazy.
+    def size
+      @size ||= key_ids.each_slice(batch_size).sum { |ids| @adapter.execute(batch_query_ast(ids)).size }
+    end
+    alias length size
+
+    def describe_plan
+      "-- batch_scope: extracted in batches of up to #{batch_size} #{@terminus.name}." \
+        "#{@terminus.primary_key} value(s). Each batch runs the query above with " \
+        "`#{@terminus.name}.#{@terminus.primary_key} IN (<batch ids>)` in place of the scope filter, " \
+        "over the ids of:"
+    end
+  end
+end

data/lib/exwiw/ddl_postprocessor.rb

@@ -170,6 +170,50 @@ module Exwiw
       end
     end
 
+    # Every extension name a dump installs, in the order the CREATE EXTENSION
+    # statements appear. Used to report which of them #strip_extensions is about
+    # to drop; run it on the raw dump, before any wrapping pass.
+    def extension_names(sql)
+      sql.scan(CREATE_EXTENSION_RE).map { |(name)| name.delete('"') }
+    end
+
+    # Drop every trace of the extensions named in `names` — the ones a managed
+    # PostgreSQL platform installs to run the source instance itself (see
+    # PostgresqlAdapter::PLATFORM_MANAGED_EXTENSIONS). Removed rather than merely
+    # wrapped like any other extension, because they cannot be created anywhere
+    # outside that platform, so keeping them only leaves a restore-time WARNING
+    # and objects the target will never have.
+    #
+    # Three things are emitted per extension and all three must go, or a leftover
+    # references a name that is no longer installed: pg_dump's `-- Name: <ext>;
+    # Type: EXTENSION` header block, the `CREATE EXTENSION`, and the
+    # `COMMENT ON EXTENSION` (whose own header reads `-- Name: EXTENSION <ext>;
+    # Type: COMMENT`). Run this on the raw dump, before the wrapping passes: they
+    # rewrite the bare statements this matches into DO blocks.
+    def strip_extensions(sql, names)
+      return sql if names.empty?
+
+      # Whole names only: each match is anchored by the whitespace/quote before it
+      # and a \b after, so `google_vacuum_mgmt` never matches an extension merely
+      # containing it (`not_google_vacuum_mgmt`, `google_vacuum_mgmt_v2`).
+      names_re = Regexp.union(names.map { |n| Regexp.escape(n) })
+      name = /(?:"#{names_re}"|#{names_re}\b)/
+
+      # Each removal takes the blank lines that trailed the object too, so the
+      # surrounding statements keep pg_dump's spacing instead of gaining a gap.
+      trailing_blank_lines = /(?:[ \t]*\r?\n(?:[ \t]*\r?\n)*)?/
+
+      sql = sql.gsub(
+        /^--\r?\n-- Name: (?:EXTENSION\s+)?#{name};[ \t]*Type:[ \t]*(?:EXTENSION|COMMENT);[^\n]*\n--(?:\r?\n)+/i,
+        "",
+      )
+      sql = sql.gsub(/^[ \t]*CREATE\s+EXTENSION\b(?:\s+IF\s+NOT\s+EXISTS)?\s+#{name}[^;]*;#{trailing_blank_lines}/i, "")
+      sql.gsub(
+        /^[ \t]*COMMENT\s+ON\s+EXTENSION\s+#{name}\s+IS\s+(?:'(?:[^']|'')*'|NULL)\s*;#{trailing_blank_lines}/i,
+        "",
+      )
+    end
+
     # Generate idempotent CREATE TYPE ... AS ENUM statements.
     # +enum_types+ is an Array of Hashes with keys :schema, :name, :labels.
     def create_type_enum_statements(enum_types)

data/lib/exwiw/explain_runner.rb

@@ -68,9 +68,34 @@ module Exwiw
         @io.puts "-- EXPLAIN:"
         @io.puts explain_text
         @io.puts
+
+        explain_batch_scope(adapter, table, table_by_name)
       end
     end
 
+    # A batched export also runs the query resolving the ids it is sliced by,
+    # which the table's own block above does not show. The per-batch query cannot
+    # be rendered faithfully here: explain resolves no ids, so #describe_plan
+    # explains the substitution instead.
+    private def explain_batch_scope(adapter, table, table_by_name)
+      batched = BatchedExtraction.build(
+        adapter: adapter,
+        table: table,
+        dump_target: @dump_target,
+        table_by_name: table_by_name,
+        logger: @logger,
+      )
+      return if batched.nil?
+
+      key_query_ast = batched.key_query_ast
+      @io.puts batched.describe_plan
+      @io.puts adapter.describe_query(key_query_ast)
+      @io.puts
+      @io.puts "-- EXPLAIN (batch_scope id set):"
+      @io.puts adapter.explain(key_query_ast, verbosity: @explain_verbosity)
+      @io.puts
+    end
+
     private def load_table_config(klass)
       Dir[File.join(@schema_dir, "*.json")].map do |file|
         json = JSON.parse(File.read(file))

data/lib/exwiw/query_ast_builder.rb

@@ -2,8 +2,8 @@
 
 module Exwiw
   class QueryAstBuilder
-    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
-      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path).run
+    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [], batch_ids: nil)
+      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path, batch_ids: batch_ids).run
     end
 
     # Scope-column mode classification for a single table. One of
@@ -26,35 +26,48 @@ module Exwiw
       !!(target && target.respond_to?(:scope_column) && target.scope_column)
     end
 
-    # Strict pre-flight for scope-column mode: abort if any extractable table
-    # cannot be scoped, so an unscoped (potentially sensitive) table is never
-    # silently dumped in full. No-op outside scope mode. `tables` is the set of
+    # Strict pre-flight: abort if any extractable table cannot be scoped (scope
+    # mode), or declares a `batch_scope` its scoping shape cannot be sliced by
+    # (both modes) — before any output is written. `tables` is the set of
     # dumpable configs (ignore:true tables are skipped — they are not extracted).
     def self.validate_scope!(tables, table_by_name, dump_target, logger)
-      return unless scope_mode?(table_by_name, dump_target)
+      # Unscopable is reported before a bad batch_scope shape — it is the more
+      # fundamental problem.
+      if scope_mode?(table_by_name, dump_target)
+        unscopable =
+          tables.reject(&:ignore).select do |table|
+            scope_category(table.name, table_by_name, dump_target, logger) == :unscopable
+          end
 
-      unscopable =
-        tables.reject(&:ignore).select do |table|
-          scope_category(table.name, table_by_name, dump_target, logger) == :unscopable
+        if unscopable.any?
+          names = unscopable.map(&:name).sort.join(", ")
+          raise ArgumentError,
+                "scope-column mode: #{unscopable.size} table(s) cannot be scoped: #{names}. " \
+                "For each, declare `scope_column: <column>` on the table to filter it directly, " \
+                "add a belongs_to path to a table that carries the scope column, mark it " \
+                "`scope_exempt: true` to export it in full, or set `ignore: true` to skip it."
         end
-      return if unscopable.empty?
+      end
+
+      tables.reject(&:ignore).each do |table|
+        next unless table.respond_to?(:batch_scope) && table.batch_scope
 
-      names = unscopable.map(&:name).sort.join(", ")
-      raise ArgumentError,
-            "scope-column mode: #{unscopable.size} table(s) cannot be scoped: #{names}. " \
-            "For each, declare `scope_column: <column>` on the table to filter it directly, " \
-            "add a belongs_to path to a table that carries the scope column, mark it " \
-            "`scope_exempt: true` to export it in full, or set `ignore: true` to skip it."
+        new(table.name, table_by_name, dump_target, logger).batch_scope_terminus!
+      end
     end
 
     attr_reader :table_name, :table_by_name, :dump_target
 
-    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
+    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [], batch_ids: nil)
       @table_name = table_name
       @table_by_name = table_by_name
       @dump_target = dump_target
       @logger = logger
       @allow_reverse = allow_reverse
+      # One batch's slice of the batch table's in-scope primary keys, set only by
+      # BatchedExtraction. Deliberately not threaded into the recursive builds
+      # below, which compile other tables' queries.
+      @batch_ids = batch_ids
       # @forward_path is the chain of tables currently being forward-resolved by
       # the "scope via an indirectly-scoped belongs_to parent" rescue
       # (build_belongs_to_scoped_clause). Each forward hop appends the table it is
@@ -612,6 +625,9 @@ module Exwiw
     end
 
     private def scope_where_clause(table)
+      batch_clause = batch_ids_clause(table)
+      return batch_clause if batch_clause
+
       Exwiw::QueryAst::WhereClause.new(
         column_name: resolved_scope_column(table),
         operator: :eq,
@@ -619,6 +635,95 @@ module Exwiw
       )
     end
 
+    # This batch's ids, in place of the batch table's scope filter. nil when the
+    # build is not batched or `table` is not the batch table.
+    private def batch_ids_clause(table)
+      return nil if @batch_ids.nil?
+
+      batch_scope = table_by_name.fetch(table_name).batch_scope
+      return nil if batch_scope.nil? || batch_scope.table != table.name
+
+      Exwiw::QueryAst::WhereClause.new(
+        column_name: table.primary_key,
+        operator: :eq,
+        value: @batch_ids
+      )
+    end
+
+    # The scoped table whose in-scope primary keys slice this table's extraction,
+    # or nil when it declares no `batch_scope`. Shapes are accepted only when
+    # every row the table keeps is selected through that table's scope filter —
+    # otherwise the unconstrained route would re-emit the same rows in every
+    # batch — so each rejection below explains itself to the config author.
+    def batch_scope_terminus!
+      table = table_by_name.fetch(table_name)
+      batch_scope = table.batch_scope
+      return nil if batch_scope.nil?
+
+      prefix = "Table '#{table.name}': batch_scope"
+
+      unless scope_mode?
+        raise ArgumentError,
+              "#{prefix} is supported in scope-column mode only. In single `--target-table` mode the " \
+              "extraction is already anchored on a caller-supplied id list, which can be batched by " \
+              "running exwiw once per slice of `--ids`."
+      end
+
+      if scope_exempt?(table)
+        raise ArgumentError,
+              "#{prefix} cannot apply: the table is exported in full (scope_exempt / rails-managed), " \
+              "so there is no scope filter to slice."
+      end
+
+      terminus = table_by_name[batch_scope.table]
+      if terminus.nil?
+        raise ArgumentError, "#{prefix} names table '#{batch_scope.table}', which is not in the schema."
+      end
+      if terminus.primary_key.nil?
+        raise ArgumentError, "#{prefix} table '#{terminus.name}' has no primary_key to slice the extraction by."
+      end
+      unless directly_scoped?(terminus)
+        raise ArgumentError,
+              "#{prefix} table '#{terminus.name}' does not carry the scope column " \
+              "(#{resolved_scope_column(terminus) || 'none declared'}), so its in-scope ids cannot be " \
+              "resolved. Name the scoped table this table joins up to."
+      end
+      # A scope_exempt terminus carries the column but its own extraction query is
+      # unfiltered, so the batches would substitute every tenant's ids for the
+      # scope filter the unbatched join still applies.
+      if scope_exempt?(terminus)
+        raise ArgumentError,
+              "#{prefix} table '#{terminus.name}' is exported in full (scope_exempt / rails-managed), " \
+              "so its id set is not scoped and every batch would reach outside the scope."
+      end
+
+      if directly_scoped?(table)
+        return terminus if terminus.name == table.name
+
+        raise ArgumentError,
+              "#{prefix} must name '#{table.name}' itself, which carries the scope column and is " \
+              "therefore filtered directly rather than through '#{terminus.name}'."
+      end
+
+      arms = scoped_arms(table)
+      unless arms.size == 1 && arms.first.path
+        raise ArgumentError,
+              "#{prefix} needs a single belongs_to join path from '#{table.name}' to the scope, but it is " \
+              "scoped another way (polymorphic arms / reverse_scope / referenced-by / the parent cascade), " \
+              "or not scoped at all. Those other id sets keep rows by routes a batch of '#{terminus.name}' " \
+              "ids does not constrain, so every batch would re-emit them."
+      end
+
+      path = arms.first.path
+      unless path.last == terminus.name
+        raise ArgumentError,
+              "#{prefix} table '#{terminus.name}' is not where '#{table.name}' reaches the scope " \
+              "(#{path.join(' -> ')}); name that path's scoped table, '#{path.last}'."
+      end
+
+      terminus
+    end
+
     # BFS over belongs_tos to the nearest *directly scoped* ancestor. Unlike the
     # target-mode walk, the returned path INCLUDES that ancestor: the scope column
     # lives on the ancestor itself (not on a foreign key of the child), so the

data/lib/exwiw/runner.rb

@@ -103,8 +103,22 @@ module Exwiw
           # both the INSERT and COPY branches below.
           row_transformer = RowTransformer.build(table)
 
+          # `batch_scope` splits the extraction into one query per slice of the
+          # scope's id set, streaming rows like any adapter result. `query_ast`
+          # stays the unbatched query — the DELETE file and the error message
+          # below describe that one.
+          phase = "resolving the batch_scope id set"
+          batched = BatchedExtraction.build(
+            adapter: adapter,
+            table: table,
+            dump_target: @dump_target,
+            table_by_name: table_by_name,
+            logger: @logger,
+          )
+          batched&.prepare!
+
           phase = "executing extraction query"
-          results = adapter.execute(query_ast)
+          results = batched || adapter.execute(query_ast)
           results = row_transformer.wrap(results) if row_transformer
           insert_idx = (idx + 1).to_s.rjust(3, '0')
 

data/lib/exwiw/table_config.rb

@@ -47,6 +47,10 @@ module Exwiw
     # schema generators.
     attribute :reverse_scope, Serdes::OptionalType.new(ReverseScope), skip_serializing_if_nil: true
 
+    # `batch_scope` splits this table's extraction into one query per slice of the
+    # scope's id set (see Exwiw::BatchScope). User-configured, never generated.
+    attribute :batch_scope, Serdes::OptionalType.new(BatchScope), skip_serializing_if_nil: true
+
     def self.from(hash)
       # Reject unknown keys before deserializing: Serdes silently drops them,
       # which would turn a typo'd or unsupported key into a silent no-op (see
@@ -76,6 +80,7 @@ module Exwiw
         hash.delete("belongs_tos")
         hash.delete("columns")
         hash.delete("reverse_scope")
+        hash.delete("batch_scope")
       end
       hash
     end
@@ -171,6 +176,7 @@ module Exwiw
         merged_table.scope_exempt = scope_exempt
         merged_table.scope_column = scope_column
         merged_table.reverse_scope = reverse_scope
+        merged_table.batch_scope = batch_scope
 
         # Structural facts of each belongs_to come from the freshly generated
         # config, but the user-owned `comment`/`ignore`/`ignore_type`/`references`
@@ -222,6 +228,10 @@ module Exwiw
           raise ArgumentError,
                 "Table '#{name}' has type=#{type}; reverse_scope must not be defined."
         end
+        if batch_scope
+          raise ArgumentError,
+                "Table '#{name}' has type=#{type}; batch_scope must not be defined."
+        end
       else
         # An ignore:true table is not extracted, so primary_key is not required
         # (e.g. a composite-primary-key table that exwiw does not support).
@@ -229,6 +239,12 @@ module Exwiw
           raise ArgumentError, "Table '#{name}' requires primary_key."
         end
 
+        if batch_scope && batch_scope.size && batch_scope.size < 1
+          raise ArgumentError,
+                "Table '#{name}': batch_scope size must be a positive number of ids per batch " \
+                "(got #{batch_scope.size})."
+        end
+
         columns.each { |column| validate_ruby_side_masking!(column) }
       end
     end

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.18"
+  VERSION = "0.9.20"
 end

data/lib/exwiw.rb

@@ -12,6 +12,7 @@ require_relative "exwiw/belongs_to"
 require_relative "exwiw/fake_data"
 require_relative "exwiw/table_column"
 require_relative "exwiw/reverse_scope"
+require_relative "exwiw/batch_scope"
 require_relative "exwiw/table_config"
 require_relative "exwiw/embedded_in"
 require_relative "exwiw/mongodb_field"
@@ -32,6 +33,7 @@ require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"
 require_relative "exwiw/row_transformer"
+require_relative "exwiw/batched_extraction"
 require_relative "exwiw/after_insert_hook"
 require_relative "exwiw/runner"
 require_relative "exwiw/explain_runner"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.18
+  version: 0.9.20
 platform: ruby
 authors:
 - Shia
@@ -64,6 +64,8 @@ files:
 - lib/exwiw/adapter/sql_bulk_insert.rb
 - lib/exwiw/adapter/sqlite_adapter.rb
 - lib/exwiw/after_insert_hook.rb
+- lib/exwiw/batch_scope.rb
+- lib/exwiw/batched_extraction.rb
 - lib/exwiw/belongs_to.rb
 - lib/exwiw/cli.rb
 - lib/exwiw/config_file.rb