exwiw 0.9.18 → 0.9.19

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 329bc5d4ceae914d8ba650d64b00a9d83916b1aa2f753cab6310ea61d3c6989a
-  data.tar.gz: 5f404952e90a272b7b4818257901f8e3dec7cbcdb0e4ae63310c728374790dad
+  metadata.gz: 9f565edc88b4eba108ef4d9accf7bbaa7d48f0e1700007cc9d280b0033e357b6
+  data.tar.gz: 822e5cd0b9d33c81d0b396cc641fa960cf80bffbebd6bd078322460a140b37cf
 SHA512:
-  metadata.gz: 77b947ce4141e5ed17e6faf4a1f6e4f910e8f0a70d96274fff3e7d33be11a1adeed2fa2ddbe59787ce6f9b4572ab969432a50bcccd2efb758e1c1c423dedb8cd
-  data.tar.gz: de8c14e52a77e610d950381af8f0028b4e78c0f572e0e7f1bcbeda37d9c824c1d04ddf6683fe9e80150631dc56df9aac340598b30c53eba25d207c0bad8057f4
+  metadata.gz: 20da3c9ee09865762f40a04b6c4bf6cf24582545ba7c51945a69aed69798906a19a214c9bd2ad588e0b6740f29a3ebfa08a122845f9a28fe64b654ba723088a0
+  data.tar.gz: db91db3a7e1774d386f45d3582301098a23df86f48fe790fae3631895b87fe686dcce3d048a17827f69ce530f948ca423bb9028f8a4b2495d167e820edd6f3fb

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+## [0.9.19] - 2026-08-04
+
+### Added
+
+- **`batch_scope`: extract a table as one query per slice of the scope's id set, so a very large table stays index-driven instead of degrading into a full scan.** A scoped table reached through a `belongs_to` hop compiles to `SELECT t.* FROM t JOIN parent ON t.parent_id = parent.id AND parent.tenant_id IN (...)`, which is index-driven while the scope keeps few parent rows. Past some number of them the planner's estimate of "probe the foreign-key index once per parent row" exceeds its estimate of "scan the table once" and it switches to a sequential scan of the whole table — for a result set that is a small fraction of it. On a table of hundreds of millions of rows that scan exceeds the server's `statement_timeout` (or runs for hours), and a `filter` on the extracted table does not help: a predicate that reduces the *output* does not reduce the *work* once the plan is a scan. `batch_scope: { "table": "<scoped table>", "size": 1000 }` names the scoped table this one reaches, resolves that table's in-scope primary keys once (from its own extraction query, so it is narrowed by exactly the filter it would carry anyway), and extracts one `size`-sized slice of those ids at a time, each batch carrying `<batch table>.<pk> IN (<ids>)` in place of the scope filter. A literal id list of that size is exactly estimated and selective, so the foreign-key index is unambiguously the cheapest plan for every batch, and total work becomes proportional to the rows the table keeps rather than to the table's size. **The dumped rows are the same as the unbatched query's**, in batch-by-batch order: the slices partition the id set, so no row is dropped or emitted twice, and the ids are sorted in exwiw before slicing (not via `ORDER BY`, which would push a sort onto the source DB), so the batched output is reproducible run to run. The batch table may be any number of hops up the path (a table two hops below it names the same batch table, and the ids are applied where the path meets the scope, bounding the whole join chain), or the table itself when it carries the scope column. Because a batch key only splits an extraction correctly when *every* row the table keeps is selected through the batch table's scope filter, the supported shapes are deliberately narrow — scope-column mode, and either a directly scoped table naming itself or a single `belongs_to` join path terminating at the named table. Polymorphic arm `UNION`s, `reverse_scope`, referenced-by, the parent cascade, `scope_exempt` (on either side — an exempt batch table's id set would not be scoped, so its batches would reach outside the scope) and single `--target-table` mode are rejected with an explanation rather than silently mis-sliced (they keep rows by routes a batch of ids does not constrain, so every batch would re-emit them), and the rejection happens in the pre-flight validation, before any output is written. `exwiw explain` additionally prints the id-set query and its `EXPLAIN` for a batched table; it cannot show a batch's literal ids, since it executes no extraction SELECT. `delete-*.sql` is generated from the unbatched query as before, `bulk_insert_chunk_size` is independent (batches are query boundaries, chunks are statement boundaries), and output for every table without `batch_scope` is byte-identical.
+
 ## [0.9.18] - 2026-08-03
 
 ### Fixed

data/README.md

@@ -738,6 +738,60 @@ Unlike rails-managed entries, `columns` and `belongs_tos` are retained so the en
 
 If omitted, the adapter default applies: 10,000 rows per statement for the SQL adapters (1,000 documents per chunk for MongoDB). Tables at or below the chunk size still produce a single `INSERT` statement. To force a single statement regardless of table size, set a value larger than the table's row count.
 
+### Batched extraction (`batch_scope`)
+
+A scoped table is normally extracted with one query, whose scope filter sits on the table it joins up to:
+
+```sql
+SELECT activities.* FROM activities
+  JOIN customers ON activities.customer_id = customers.id
+                AND customers.tenant_id IN ('t1')
+```
+
+That is index-driven while the scope keeps few `customers`. Past some number of them the planner's estimate of "probe the foreign-key index once per customer" exceeds its estimate of "scan the table once", and it switches to a **sequential scan of the whole table** — for a result set that is a small fraction of it. On a table of hundreds of millions of rows the scan then exceeds the server's `statement_timeout`, or simply runs for hours. Note that no `filter` on the extracted table fixes this: a predicate that reduces the *output* does not reduce the *work* once the plan is a scan (it may not even change the plan).
+
+`batch_scope` removes the choice instead of arguing with the estimate. It names the scoped table this one reaches — the **batch table** — and exwiw resolves that table's in-scope primary keys once, then extracts one `size`-sized slice of those ids at a time:
+
+```json
+{
+  "name": "activities",
+  "primary_key": "id",
+  "batch_scope": { "table": "customers", "size": 1000 },
+  "belongs_tos": [{ "table_name": "customers", "foreign_key": "customer_id" }],
+  "columns": [{ "name": "id" }, { "name": "customer_id" }]
+}
+```
+
+Each batch runs with that slice's ids in place of the scope filter:
+
+```sql
+SELECT activities.* FROM activities
+  JOIN customers ON activities.customer_id = customers.id
+                AND customers.id IN (/* 1000 ids */)
+```
+
+An explicit id list of that size is exactly estimated and selective, so the foreign-key index is unambiguously the cheapest plan for every batch, and total work is proportional to the rows the table actually keeps rather than to the table's size.
+
+- **The dumped rows are the same as the unbatched query's** (in batch-by-batch order). The slices partition the id set — every id is in exactly one batch — so no row is dropped or emitted twice. The ids are sorted (in exwiw, not with `ORDER BY` — the id-set query stays cheap on the source DB) before slicing, so batch composition, and the dump, is reproducible run to run.
+- **`size` defaults to 1000** ids per batch.
+- The batch table's ids come from **its own extraction query**, so it is narrowed by exactly the filter it would carry in the unbatched query. They are held in memory for the extraction: one scope's worth of primary keys, orders of magnitude smaller than the table being batched.
+- The batch table may be **any number of hops up** the path — a table two hops below it (`activity_orders → activities → customers`) names `customers` too, and the batch ids are applied where the path meets the scope, bounding the whole join chain.
+- A table that **carries the scope column itself** batches by naming itself; each batch then filters `WHERE <pk> IN (<ids>)` directly. Note that the id-set query is then the same scope predicate over the same table, so this shape only avoids the scan when the scope column is indexed (ideally index-only) — the join shape above is the one that genuinely removes the planner's choice.
+- `delete-*.sql` is unaffected (it is generated from the unbatched query).
+- `bulk_insert_chunk_size` is independent: batches are query boundaries, chunks are `INSERT` statement boundaries.
+- With `--output-format=copy`, batching bounds each query's cost but not memory: COPY builds the whole table's body in memory, so all batches' rows are resident at once. Use the default INSERT format (which streams) when the kept rows themselves are huge.
+
+**Supported shapes.** A batch key only splits an extraction correctly when *every* row the table keeps is selected through the batch table's scope filter — otherwise a route the batch key does not constrain would keep the same rows in every batch, and the dump would repeat them (a primary-key conflict on import). So `batch_scope` requires [scope-column mode](#scope-column-mode) and one of:
+
+- the table is **directly scoped** (`scope_column`) and names itself, or
+- the table reaches the scope through a **single `belongs_to` join path** (path 2 in [the six scoping paths](#how-each-table-is-narrowed--the-six-scoping-paths)) whose scoped terminus is the named table.
+
+Every other shape — polymorphic arm `UNION`s, `reverse_scope`, referenced-by, the parent cascade, `scope_exempt` (on the batched table *or* the batch table, whose id set would then not be scoped), and single `--target-table` mode — is **rejected with an explanation** rather than silently mis-sliced, before any output is written. (In single-target mode the extraction is already anchored on a caller-supplied id list, so batching it means running exwiw once per slice of `--ids`.)
+
+`exwiw explain` prints the id-set query and its `EXPLAIN` after a batched table's own query, since that query is the part of a batched export the table's query does not show. It cannot show a batch's literal id list — `explain` resolves no ids, because it executes no extraction SELECT.
+
+Like `scope_column` / `scope_exempt` / `reverse_scope`, `batch_scope` is user-maintained: never emitted by `schema:generate`, and preserved across regeneration.
+
 ### Filter
 
 Some case, you don't need full records related to target. e.g. dump user access logs only for the last year.
@@ -745,6 +799,7 @@ Some case, you don't need full records related to target. e.g. dump user access
 
 - injected as it is in table condition(e.g. WHERE on mysql), so you are recommended to clearify table name of column to avoid ambiguity.
 - injected to every where / join clause, so it affects to all tables depends on filterted target-table. it results to data inconsistency.
+- a way to reduce the rows returned, which is **not** necessarily a way to reduce the work: on a large table the engine may keep (or switch to) a full scan and evaluate the filter per row. See [batched extraction](#batched-extraction-batch_scope) when the goal is to bound how much of the table is read.
 
 ### Masking
 

data/lib/exwiw/batch_scope.rb

@@ -0,0 +1,20 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Opt-in config for batched extraction (see {BatchedExtraction} and the
+  # `batch_scope` section of README.md): `table` is the scoped table whose
+  # in-scope primary keys slice this table's extraction, `size` the ids per batch.
+  class BatchScope
+    include Serdes
+
+    DEFAULT_SIZE = 1_000
+
+    attribute :table, String
+    attribute :size, optional(Integer), skip_serializing_if_nil: true
+    attribute :comment, optional(String), skip_serializing_if_nil: true
+
+    def batch_size
+      size || DEFAULT_SIZE
+    end
+  end
+end

data/lib/exwiw/batched_extraction.rb

@@ -0,0 +1,116 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Extracts a table configured with `batch_scope` as one query per slice of the
+  # scope's id set, so each query stays index-driven instead of degrading into a
+  # full scan. Rows are the unbatched query's, in batch order; the slices
+  # partition the id set, so none is dropped or repeated. See README.md.
+  class BatchedExtraction
+    include Enumerable
+
+    attr_reader :terminus
+
+    def self.build(adapter:, table:, dump_target:, table_by_name:, logger:)
+      return nil unless table.respond_to?(:batch_scope) && table.batch_scope
+
+      new(
+        adapter: adapter,
+        table: table,
+        dump_target: dump_target,
+        table_by_name: table_by_name,
+        logger: logger,
+      )
+    end
+
+    def initialize(adapter:, table:, dump_target:, table_by_name:, logger:)
+      @adapter = adapter
+      @table = table
+      @dump_target = dump_target
+      @table_by_name = table_by_name
+      @logger = logger
+      @terminus = QueryAstBuilder
+        .new(table.name, table_by_name, dump_target, logger)
+        .batch_scope_terminus!
+    end
+
+    def batch_size
+      @table.batch_scope.batch_size
+    end
+
+    # The batch table's own extraction query projected to its primary key, so the
+    # ids are narrowed by exactly the filter the unbatched query would carry. The
+    # key is a plain column so masking configured on it cannot corrupt the ids.
+    def key_query_ast
+      @key_query_ast ||= begin
+        scoped = QueryAstBuilder.run(@terminus.name, @table_by_name, @dump_target, @logger)
+
+        QueryAst::Select.new.tap do |ast|
+          ast.from(scoped.from_table_name)
+          ast.select([TableColumn.from_symbol_keys(name: @terminus.primary_key)])
+          scoped.join_clauses.each { |join_clause| ast.join(join_clause) }
+          scoped.where_clauses.each { |where_clause| ast.where(where_clause) }
+        end
+      end
+    end
+
+    def batch_query_ast(ids)
+      QueryAstBuilder.run(@table.name, @table_by_name, @dump_target, @logger, batch_ids: ids)
+    end
+
+    # Resolve the id set and log the plan before extraction starts, so its cost
+    # (and an empty id set) is reported where it happens rather than mid-stream.
+    def prepare!
+      if key_ids.empty?
+        @logger.info("  No in-scope #{@terminus.name} ids to batch by; extracting nothing.")
+      else
+        @logger.info(
+          "  Extracting in #{batch_count} batch(es) of up to #{batch_size} " \
+          "#{@terminus.name}.#{@terminus.primary_key} value(s) (#{key_ids.size} in scope)."
+        )
+      end
+      self
+    end
+
+    # Drained in full (the connection must be free for the batch queries) and
+    # sorted here rather than via ORDER BY, which would push a sort onto the
+    # source DB. Any total order makes the batches reproducible run to run.
+    def key_ids
+      @key_ids ||= @adapter.execute(key_query_ast).map(&:first).sort
+    end
+
+    def batch_count
+      (key_ids.size + batch_size - 1) / batch_size
+    end
+
+    def each
+      return enum_for(:each) unless block_given?
+
+      extracted = 0
+      key_ids.each_slice(batch_size).with_index do |ids, idx|
+        rows = 0
+        @adapter.execute(batch_query_ast(ids)).each do |row|
+          rows += 1
+          yield row
+        end
+        extracted += rows
+        @logger.info("  Batch #{idx + 1}/#{batch_count}: #{rows} record(s), #{extracted} so far.")
+      end
+
+      self
+    end
+
+    # Only the COPY output format needs the count up front, and each batch answers
+    # it with its own count query, so this stays lazy.
+    def size
+      @size ||= key_ids.each_slice(batch_size).sum { |ids| @adapter.execute(batch_query_ast(ids)).size }
+    end
+    alias length size
+
+    def describe_plan
+      "-- batch_scope: extracted in batches of up to #{batch_size} #{@terminus.name}." \
+        "#{@terminus.primary_key} value(s). Each batch runs the query above with " \
+        "`#{@terminus.name}.#{@terminus.primary_key} IN (<batch ids>)` in place of the scope filter, " \
+        "over the ids of:"
+    end
+  end
+end

data/lib/exwiw/explain_runner.rb

@@ -68,9 +68,34 @@ module Exwiw
         @io.puts "-- EXPLAIN:"
         @io.puts explain_text
         @io.puts
+
+        explain_batch_scope(adapter, table, table_by_name)
       end
     end
 
+    # A batched export also runs the query resolving the ids it is sliced by,
+    # which the table's own block above does not show. The per-batch query cannot
+    # be rendered faithfully here: explain resolves no ids, so #describe_plan
+    # explains the substitution instead.
+    private def explain_batch_scope(adapter, table, table_by_name)
+      batched = BatchedExtraction.build(
+        adapter: adapter,
+        table: table,
+        dump_target: @dump_target,
+        table_by_name: table_by_name,
+        logger: @logger,
+      )
+      return if batched.nil?
+
+      key_query_ast = batched.key_query_ast
+      @io.puts batched.describe_plan
+      @io.puts adapter.describe_query(key_query_ast)
+      @io.puts
+      @io.puts "-- EXPLAIN (batch_scope id set):"
+      @io.puts adapter.explain(key_query_ast, verbosity: @explain_verbosity)
+      @io.puts
+    end
+
     private def load_table_config(klass)
       Dir[File.join(@schema_dir, "*.json")].map do |file|
         json = JSON.parse(File.read(file))

data/lib/exwiw/query_ast_builder.rb

@@ -2,8 +2,8 @@
 
 module Exwiw
   class QueryAstBuilder
-    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
-      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path).run
+    def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [], batch_ids: nil)
+      new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path, batch_ids: batch_ids).run
     end
 
     # Scope-column mode classification for a single table. One of
@@ -26,35 +26,48 @@ module Exwiw
       !!(target && target.respond_to?(:scope_column) && target.scope_column)
     end
 
-    # Strict pre-flight for scope-column mode: abort if any extractable table
-    # cannot be scoped, so an unscoped (potentially sensitive) table is never
-    # silently dumped in full. No-op outside scope mode. `tables` is the set of
+    # Strict pre-flight: abort if any extractable table cannot be scoped (scope
+    # mode), or declares a `batch_scope` its scoping shape cannot be sliced by
+    # (both modes) — before any output is written. `tables` is the set of
     # dumpable configs (ignore:true tables are skipped — they are not extracted).
     def self.validate_scope!(tables, table_by_name, dump_target, logger)
-      return unless scope_mode?(table_by_name, dump_target)
+      # Unscopable is reported before a bad batch_scope shape — it is the more
+      # fundamental problem.
+      if scope_mode?(table_by_name, dump_target)
+        unscopable =
+          tables.reject(&:ignore).select do |table|
+            scope_category(table.name, table_by_name, dump_target, logger) == :unscopable
+          end
 
-      unscopable =
-        tables.reject(&:ignore).select do |table|
-          scope_category(table.name, table_by_name, dump_target, logger) == :unscopable
+        if unscopable.any?
+          names = unscopable.map(&:name).sort.join(", ")
+          raise ArgumentError,
+                "scope-column mode: #{unscopable.size} table(s) cannot be scoped: #{names}. " \
+                "For each, declare `scope_column: <column>` on the table to filter it directly, " \
+                "add a belongs_to path to a table that carries the scope column, mark it " \
+                "`scope_exempt: true` to export it in full, or set `ignore: true` to skip it."
         end
-      return if unscopable.empty?
+      end
+
+      tables.reject(&:ignore).each do |table|
+        next unless table.respond_to?(:batch_scope) && table.batch_scope
 
-      names = unscopable.map(&:name).sort.join(", ")
-      raise ArgumentError,
-            "scope-column mode: #{unscopable.size} table(s) cannot be scoped: #{names}. " \
-            "For each, declare `scope_column: <column>` on the table to filter it directly, " \
-            "add a belongs_to path to a table that carries the scope column, mark it " \
-            "`scope_exempt: true` to export it in full, or set `ignore: true` to skip it."
+        new(table.name, table_by_name, dump_target, logger).batch_scope_terminus!
+      end
     end
 
     attr_reader :table_name, :table_by_name, :dump_target
 
-    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
+    def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [], batch_ids: nil)
       @table_name = table_name
       @table_by_name = table_by_name
       @dump_target = dump_target
       @logger = logger
       @allow_reverse = allow_reverse
+      # One batch's slice of the batch table's in-scope primary keys, set only by
+      # BatchedExtraction. Deliberately not threaded into the recursive builds
+      # below, which compile other tables' queries.
+      @batch_ids = batch_ids
       # @forward_path is the chain of tables currently being forward-resolved by
       # the "scope via an indirectly-scoped belongs_to parent" rescue
       # (build_belongs_to_scoped_clause). Each forward hop appends the table it is
@@ -612,6 +625,9 @@ module Exwiw
     end
 
     private def scope_where_clause(table)
+      batch_clause = batch_ids_clause(table)
+      return batch_clause if batch_clause
+
       Exwiw::QueryAst::WhereClause.new(
         column_name: resolved_scope_column(table),
         operator: :eq,
@@ -619,6 +635,95 @@ module Exwiw
       )
     end
 
+    # This batch's ids, in place of the batch table's scope filter. nil when the
+    # build is not batched or `table` is not the batch table.
+    private def batch_ids_clause(table)
+      return nil if @batch_ids.nil?
+
+      batch_scope = table_by_name.fetch(table_name).batch_scope
+      return nil if batch_scope.nil? || batch_scope.table != table.name
+
+      Exwiw::QueryAst::WhereClause.new(
+        column_name: table.primary_key,
+        operator: :eq,
+        value: @batch_ids
+      )
+    end
+
+    # The scoped table whose in-scope primary keys slice this table's extraction,
+    # or nil when it declares no `batch_scope`. Shapes are accepted only when
+    # every row the table keeps is selected through that table's scope filter —
+    # otherwise the unconstrained route would re-emit the same rows in every
+    # batch — so each rejection below explains itself to the config author.
+    def batch_scope_terminus!
+      table = table_by_name.fetch(table_name)
+      batch_scope = table.batch_scope
+      return nil if batch_scope.nil?
+
+      prefix = "Table '#{table.name}': batch_scope"
+
+      unless scope_mode?
+        raise ArgumentError,
+              "#{prefix} is supported in scope-column mode only. In single `--target-table` mode the " \
+              "extraction is already anchored on a caller-supplied id list, which can be batched by " \
+              "running exwiw once per slice of `--ids`."
+      end
+
+      if scope_exempt?(table)
+        raise ArgumentError,
+              "#{prefix} cannot apply: the table is exported in full (scope_exempt / rails-managed), " \
+              "so there is no scope filter to slice."
+      end
+
+      terminus = table_by_name[batch_scope.table]
+      if terminus.nil?
+        raise ArgumentError, "#{prefix} names table '#{batch_scope.table}', which is not in the schema."
+      end
+      if terminus.primary_key.nil?
+        raise ArgumentError, "#{prefix} table '#{terminus.name}' has no primary_key to slice the extraction by."
+      end
+      unless directly_scoped?(terminus)
+        raise ArgumentError,
+              "#{prefix} table '#{terminus.name}' does not carry the scope column " \
+              "(#{resolved_scope_column(terminus) || 'none declared'}), so its in-scope ids cannot be " \
+              "resolved. Name the scoped table this table joins up to."
+      end
+      # A scope_exempt terminus carries the column but its own extraction query is
+      # unfiltered, so the batches would substitute every tenant's ids for the
+      # scope filter the unbatched join still applies.
+      if scope_exempt?(terminus)
+        raise ArgumentError,
+              "#{prefix} table '#{terminus.name}' is exported in full (scope_exempt / rails-managed), " \
+              "so its id set is not scoped and every batch would reach outside the scope."
+      end
+
+      if directly_scoped?(table)
+        return terminus if terminus.name == table.name
+
+        raise ArgumentError,
+              "#{prefix} must name '#{table.name}' itself, which carries the scope column and is " \
+              "therefore filtered directly rather than through '#{terminus.name}'."
+      end
+
+      arms = scoped_arms(table)
+      unless arms.size == 1 && arms.first.path
+        raise ArgumentError,
+              "#{prefix} needs a single belongs_to join path from '#{table.name}' to the scope, but it is " \
+              "scoped another way (polymorphic arms / reverse_scope / referenced-by / the parent cascade), " \
+              "or not scoped at all. Those other id sets keep rows by routes a batch of '#{terminus.name}' " \
+              "ids does not constrain, so every batch would re-emit them."
+      end
+
+      path = arms.first.path
+      unless path.last == terminus.name
+        raise ArgumentError,
+              "#{prefix} table '#{terminus.name}' is not where '#{table.name}' reaches the scope " \
+              "(#{path.join(' -> ')}); name that path's scoped table, '#{path.last}'."
+      end
+
+      terminus
+    end
+
     # BFS over belongs_tos to the nearest *directly scoped* ancestor. Unlike the
     # target-mode walk, the returned path INCLUDES that ancestor: the scope column
     # lives on the ancestor itself (not on a foreign key of the child), so the

data/lib/exwiw/runner.rb

@@ -103,8 +103,22 @@ module Exwiw
           # both the INSERT and COPY branches below.
           row_transformer = RowTransformer.build(table)
 
+          # `batch_scope` splits the extraction into one query per slice of the
+          # scope's id set, streaming rows like any adapter result. `query_ast`
+          # stays the unbatched query — the DELETE file and the error message
+          # below describe that one.
+          phase = "resolving the batch_scope id set"
+          batched = BatchedExtraction.build(
+            adapter: adapter,
+            table: table,
+            dump_target: @dump_target,
+            table_by_name: table_by_name,
+            logger: @logger,
+          )
+          batched&.prepare!
+
           phase = "executing extraction query"
-          results = adapter.execute(query_ast)
+          results = batched || adapter.execute(query_ast)
           results = row_transformer.wrap(results) if row_transformer
           insert_idx = (idx + 1).to_s.rjust(3, '0')
 

data/lib/exwiw/table_config.rb

@@ -47,6 +47,10 @@ module Exwiw
     # schema generators.
     attribute :reverse_scope, Serdes::OptionalType.new(ReverseScope), skip_serializing_if_nil: true
 
+    # `batch_scope` splits this table's extraction into one query per slice of the
+    # scope's id set (see Exwiw::BatchScope). User-configured, never generated.
+    attribute :batch_scope, Serdes::OptionalType.new(BatchScope), skip_serializing_if_nil: true
+
     def self.from(hash)
       # Reject unknown keys before deserializing: Serdes silently drops them,
       # which would turn a typo'd or unsupported key into a silent no-op (see
@@ -76,6 +80,7 @@ module Exwiw
         hash.delete("belongs_tos")
         hash.delete("columns")
         hash.delete("reverse_scope")
+        hash.delete("batch_scope")
       end
       hash
     end
@@ -171,6 +176,7 @@ module Exwiw
         merged_table.scope_exempt = scope_exempt
         merged_table.scope_column = scope_column
         merged_table.reverse_scope = reverse_scope
+        merged_table.batch_scope = batch_scope
 
         # Structural facts of each belongs_to come from the freshly generated
         # config, but the user-owned `comment`/`ignore`/`ignore_type`/`references`
@@ -222,6 +228,10 @@ module Exwiw
           raise ArgumentError,
                 "Table '#{name}' has type=#{type}; reverse_scope must not be defined."
         end
+        if batch_scope
+          raise ArgumentError,
+                "Table '#{name}' has type=#{type}; batch_scope must not be defined."
+        end
       else
         # An ignore:true table is not extracted, so primary_key is not required
         # (e.g. a composite-primary-key table that exwiw does not support).
@@ -229,6 +239,12 @@ module Exwiw
           raise ArgumentError, "Table '#{name}' requires primary_key."
         end
 
+        if batch_scope && batch_scope.size && batch_scope.size < 1
+          raise ArgumentError,
+                "Table '#{name}': batch_scope size must be a positive number of ids per batch " \
+                "(got #{batch_scope.size})."
+        end
+
         columns.each { |column| validate_ruby_side_masking!(column) }
       end
     end

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.18"
+  VERSION = "0.9.19"
 end

data/lib/exwiw.rb

@@ -12,6 +12,7 @@ require_relative "exwiw/belongs_to"
 require_relative "exwiw/fake_data"
 require_relative "exwiw/table_column"
 require_relative "exwiw/reverse_scope"
+require_relative "exwiw/batch_scope"
 require_relative "exwiw/table_config"
 require_relative "exwiw/embedded_in"
 require_relative "exwiw/mongodb_field"
@@ -32,6 +33,7 @@ require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"
 require_relative "exwiw/row_transformer"
+require_relative "exwiw/batched_extraction"
 require_relative "exwiw/after_insert_hook"
 require_relative "exwiw/runner"
 require_relative "exwiw/explain_runner"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.18
+  version: 0.9.19
 platform: ruby
 authors:
 - Shia
@@ -64,6 +64,8 @@ files:
 - lib/exwiw/adapter/sql_bulk_insert.rb
 - lib/exwiw/adapter/sqlite_adapter.rb
 - lib/exwiw/after_insert_hook.rb
+- lib/exwiw/batch_scope.rb
+- lib/exwiw/batched_extraction.rb
 - lib/exwiw/belongs_to.rb
 - lib/exwiw/cli.rb
 - lib/exwiw/config_file.rb