exwiw 0.8.3 → 0.8.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 90d949cb54565ed644b599102cfabc16236fe2d83523594cd7b14f269118a9b2
-  data.tar.gz: 7236bb6ee6b9dda38aec93491f0266e334890e3c271b4f7d5d072d46a1e0f88b
+  metadata.gz: 3934deb015b3ede7c6a8caa25b857d1f7c8957b4efa3284cd26f21ff2620904d
+  data.tar.gz: e44fef95c3c273aa96b1dde37736212687da88d8d9b98abb5c44fb305d45b7a3
 SHA512:
-  metadata.gz: bcfac0aaf220b55dfa3172f94d2ad0a6e828f53253e367aa9203db7061695fa6a9188e22a92c18d51bd9a55b6a9bf840cf780a3647d4aca2a0cccff1a785a347
-  data.tar.gz: d891fc7101fea30d674b933213715a8a4534c459e191b9bf4fce31b3bda8a9be037bf09627419f43ef914d2a9288aa40e1cd4836a4d8e1e0901d73e320218cd3
+  metadata.gz: 1bf410fb503b270aca3f96191f34cb8ee67730ac49ea6a57d34550d2b603fed7404206d405c61f6dd3b67808f8865d73c80121d303c628dda513ebcf9c418b17
+  data.tar.gz: d1cbb8bd0ed7bd10f53d3d3169985844d4e5e509041c95a18d083dff458a06f26c09e325fea44bc5c1e14bf65b53fdc79e4dbe351f3e4e5f916e6c9135cfa71e

data/CHANGELOG.md

@@ -2,6 +2,16 @@
 
 ## [Unreleased]
 
+## [0.8.5] - 2026-06-25
+
+- **`--parallel-workers=N` parallelizes the MongoDB dump across forked processes (opt-in, byte-identical).** When set with `N≥2` on the mongodb adapter's `export`, exwiw runs an inter-collection fork schedule that decodes whole collections in parallel while preserving each collection's natural row order, so the output files are byte-identical to a serial run (same filenames, same content). Collections are classified into three dependency groups — reference data dumped in full (no `belongs_to`), the scoped DAG reachable to the dump target, and non-reachable reference data — which lets the heavy full-dump collections run concurrently with the scoped pass; only a handful of small `@state` hand-offs cross process boundaries. The win needs real cores: it clears ~2× from 4 workers and saturates there. Requires a dump target and a `fork`-capable runtime (CRuby on POSIX); it falls back to the serial path on JRuby/TruffleRuby/Windows or when no target is given. Also settable as `parallel_workers:` in the config file. The default remains the serial dump.
+
+## [0.8.4] - 2026-06-24
+
+### Fixed
+
+- **Scope id-sets are materialized and probed by JOIN instead of `<col> IN (subquery)`, removing a correlated full-scan on large tables.** The three id-set scope shapes — the multi-referencer `reverse_scope` `UNION`, the single-referencer reverse/`referenced_by` extraction, and the multi-hop forward (`via_scoped_parent`) cascade — were emitted as `<col> IN (<subquery>)`. On a large global-identity table (e.g. `users`) MySQL cannot turn a `UNION` subquery into a materialized semi-join and falls back to its IN-to-`EXISTS` rewrite: a correlated `DEPENDENT SUBQUERY`/`DEPENDENT UNION` re-evaluated **per outer row**, so the driving table is full-scanned and the union re-run for every row (the plan that ran for minutes and timed out on a production-scale identity table, even for an empty tenant). These clauses are now lifted into a `JOIN` against a materialized derived table — `JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids ON <table>.<col> = ids.exwiw_scope_id` — so the engine evaluates the id set **once** (the `DISTINCT` makes the derived table non-mergeable, hence materialized) and probes the outer table by primary key. The `DISTINCT` also dedups, so the result set is identical to the old `IN` form; the cascade nests the same way (each level materialized once); NULL exclusion, the forward-path cycle guard, single-parent/polymorphic skips, and PostgreSQL's `uuid`/`varchar` `::text` reconciliation are all preserved. All three SQL adapters (mysql / postgresql / sqlite). See the [README](README.md#why-a-join-not-in-subquery).
+
 ## [0.8.3] - 2026-06-24
 
 ### Fixed

data/README.md

@@ -179,15 +179,18 @@ Each table is resolved as follows:
   nearest such table and applies the scope filter there (the same join machinery
   the single-target mode uses).
 - **`belongs_to` a parent that is itself scoped but carries no scope column of its
-  own** → exwiw constrains this table to the parent's in-scope ids via a subquery
-  (`fk IN (SELECT parent.pk FROM <parent's scoped query>)`). This covers a *hub*
-  table that has no scope column and is scoped only because an extractable child
-  references it (see referenced-by below): the hub's other `belongs_to` children
-  ride along to just the in-scope rows instead of being dumped in full. The parent
-  itself may be scoped the same way, so this **cascades across multiple hops**
-  (each a single unambiguous scopable parent) and the subquery nests
-  correspondingly; the recursion terminates on a genuine `belongs_to` cycle (a
-  table already on the path is left `:unscopable` rather than looped on).
+  own** → exwiw constrains this table to the parent's in-scope ids by joining it to
+  the parent's scoped query, materialized as a derived table
+  (`JOIN (SELECT DISTINCT parent.pk … FROM <parent's scoped query>) … ON fk = …`).
+  This covers a *hub* table that has no scope column and is scoped only because an
+  extractable child references it (see referenced-by below): the hub's other
+  `belongs_to` children ride along to just the in-scope rows instead of being dumped
+  in full. The parent itself may be scoped the same way, so this **cascades across
+  multiple hops** (each a single unambiguous scopable parent) and the derived-table
+  JOINs nest correspondingly; the recursion terminates on a genuine `belongs_to`
+  cycle (a table already on the path is left `:unscopable` rather than looped on).
+  (See [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery) for the
+  materialization rationale.)
 - **Cannot be scoped at all** (no scope column and no path to one) → exwiw
   **aborts** and lists the offending tables, so an unscoped table is never silently
   dumped in full. For each, either declare a `scope_column`, add a `belongs_to`
@@ -568,15 +571,19 @@ The same type filter is applied on the join path — and in the matching `delete
 ActiveStorage is handled automatically — no ActiveStorage-specific configuration is required. The `has_one_attached` / `has_many_attached` macros don't add a column to the owning model; they generate ordinary associations that exwiw already understands:
 
 - **`active_storage_attachments`** is the polymorphic join row (`belongs_to :record, polymorphic: true` + `belongs_to :blob`). `exwiw:schema:generate` expands the polymorphic `record` into one `belongs_to` per model that declared `has_*_attached` (found via the generated `has_* ..., as: :record` reflections), exactly like any other [polymorphic `belongs_to`](#polymorphic-belongs_to). So only the attachments whose owner is among the dumped rows are extracted.
-- **`active_storage_blobs`** has no `belongs_to` of its own (attachments point *at* it), so it has no path to the dump target. exwiw narrows it via **reverse / "referenced_by" extraction**: a parent table referenced by exactly one constrained, non-polymorphic child is constrained to just the referenced ids instead of dumping every row:
+- **`active_storage_blobs`** has no `belongs_to` of its own (attachments point *at* it), so it has no path to the dump target. exwiw narrows it via **reverse / "referenced_by" extraction**: a parent table referenced by exactly one constrained, non-polymorphic child is constrained to just the referenced ids instead of dumping every row. The id set is materialized once and joined back (see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery)):
 
   ```sql
   SELECT active_storage_blobs.* FROM active_storage_blobs
-  WHERE active_storage_blobs.id IN (
-    SELECT active_storage_attachments.blob_id FROM active_storage_attachments
-    WHERE active_storage_attachments.record_id IN (/* owner subquery */)
-      AND active_storage_attachments.record_type = '...'
-  )
+  JOIN (
+    SELECT DISTINCT exwiw_scope_src_0.blob_id AS exwiw_scope_id
+    FROM (
+      SELECT active_storage_attachments.blob_id FROM active_storage_attachments
+      WHERE active_storage_attachments.record_id IN (/* owner subquery */)
+        AND active_storage_attachments.record_type = '...'
+    ) AS exwiw_scope_src_0
+  ) AS exwiw_scope_ids_0
+    ON active_storage_blobs.id = exwiw_scope_ids_0.exwiw_scope_id
   ```
 
   `active_storage_variant_records` also references blobs, but since it has no path of its own to the dump target it doesn't constrain anything and is ignored as a referencer — blobs stays narrowed to the attachment-referenced ids. (A parent referenced by *multiple* constrained children currently falls back to dumping all of its rows.)
@@ -603,18 +610,22 @@ The automatic reverse extraction above narrows a table referenced by **exactly o
 }
 ```
 
-produces (each arm reuses that referencer's own scope, so a per-tenant run keeps only that tenant's ids):
+produces (each arm reuses that referencer's own scope, so a per-tenant run keeps only that tenant's ids; the `UNION` id set is materialized once and joined back — see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery)):
 
 ```sql
 SELECT users.* FROM users
-WHERE users.id IN (
-  SELECT customers.user_id FROM customers WHERE <customers' scope> AND customers.user_id IS NOT NULL
-  UNION
-  SELECT staff.user_id FROM staff WHERE <staff' scope> AND staff.user_id IS NOT NULL
-  UNION
-  SELECT business_entity_customers.kantan_yoyaku_user_id FROM business_entity_customers
-    WHERE <…' scope> AND business_entity_customers.kantan_yoyaku_user_id IS NOT NULL
-)
+JOIN (
+  SELECT DISTINCT exwiw_scope_src_0.user_id AS exwiw_scope_id
+  FROM (
+    SELECT customers.user_id FROM customers WHERE <customers' scope> AND customers.user_id IS NOT NULL
+    UNION
+    SELECT staff.user_id FROM staff WHERE <staff' scope> AND staff.user_id IS NOT NULL
+    UNION
+    SELECT business_entity_customers.kantan_yoyaku_user_id FROM business_entity_customers
+      WHERE <…' scope> AND business_entity_customers.kantan_yoyaku_user_id IS NOT NULL
+  ) AS exwiw_scope_src_0
+) AS exwiw_scope_ids_0
+  ON users.id = exwiw_scope_ids_0.exwiw_scope_id
 ```
 
 Notes:
@@ -625,6 +636,19 @@ Notes:
 - **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`. The cascade is **multi-hop**, so a table several `belongs_to` hops below the reverse-scoped table (e.g. `end_user_profiles → end_users → users`) also tightens automatically, with no config of its own.
 - Works in both single-target and scope-column mode. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
 
+### Why a JOIN, not `IN (subquery)`
+
+Every scope id-set above — the multi-referencer `reverse_scope` `UNION`, the single-referencer reverse extraction, and the multi-hop forward cascade — is emitted as a `JOIN` to a `SELECT DISTINCT` derived table rather than `<col> IN (<subquery>)`:
+
+```sql
+… JOIN (SELECT DISTINCT src.<id> AS exwiw_scope_id FROM (<id-set subquery>) AS src) AS ids
+    ON <table>.<col> = ids.exwiw_scope_id
+```
+
+Both forms select the **same rows** — the `DISTINCT` dedups, so the join never fans out — but the query plans differ sharply on a large table. As `<col> IN (… UNION …)`, MySQL cannot turn a `UNION` subquery into a materialized semi-join and falls back to its IN-to-`EXISTS` rewrite: a **correlated `DEPENDENT SUBQUERY`** re-evaluated for every outer row, i.e. a full scan of the (potentially huge) outer table multiplied by the cost of the union. The derived-table form forces the engine to evaluate the id set **once** (the `DISTINCT` makes the derived table non-mergeable, hence materialized) and then probe the outer table by its primary key. On a global-identity table such as `users` this is the difference between a full table scan and an index lookup; the cascade nests the same way, so each level is materialized once instead of being re-evaluated by the level above.
+
+All three SQL adapters (mysql / postgresql / sqlite) emit this shape. PostgreSQL additionally reconciles a `uuid`/`varchar` type mismatch by casting the join key and the projected id to `text`, exactly as the old `IN` form did.
+
 ### Rails-managed tables (special `type` values)
 
 Some tables are owned by Rails itself rather than the application — they have no ActiveRecord model and Rails reserves the right to evolve their column shape between versions (e.g. `schema_migrations`, `ar_internal_metadata`). exwiw treats them as a distinct category via the `type` field on a table config:
@@ -745,6 +769,7 @@ The MongoDB adapter is experimental. To use it:
 - `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
 - `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only** (the SQL adapters have no equivalent).
 - Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
+- `--parallel-workers=N` (opt-in, `export` only) forks `N` worker processes that decode whole collections in parallel — the dominant cost on a large dump is the driver's BSON→Ruby decode, and each worker decodes its own collections in their natural order, so the output stays **byte-identical** to a serial run (same filenames and content). It needs a dump target (the schedule is built around the scoped DAG) and a `fork`-capable runtime (CRuby on POSIX), falling back to the serial path otherwise; it also accepts `parallel_workers:` in the config file. The speedup needs real cores to spend — it reaches ~2× from 4 workers and saturates there. The default is serial. See [`docs/mongodb-dump-parallelism-2x-notes.md`](docs/mongodb-dump-parallelism-2x-notes.md) for the schedule and measurements.
 - Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
   ```bash
   mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl

data/docs/mongodb-dump-parallelism-2x-notes.md

@@ -134,13 +134,24 @@ bounded by chunk size, and `N×` that stays well under the 7 GiB container limit
 
 ## Status
 
-This is a **measured, byte-identical proof** (bench prototype, not yet integrated
-into the Runner/CLI). Integrating it means re-introducing process orchestration
-and `@state` sidecar IPC — the machinery `optimization-notes.md` deliberately
-removed as over-engineered for a flag. It is recorded here because, unlike that
-removed work, this schedule is (a) byte-identical by construction, (b) measured
-past the 2× target on a real extraction, and (c) the lever the task explicitly
-invited (scale the task to go faster). A production version would gate it behind a
-worker-count option, fall back to serial where `fork` is unavailable
-(Windows/JRuby), and reuse the existing genuine-anchor classification to derive
-the three groups.
+**Integrated and shipped behind an opt-in flag.** The schedule lives in
+`Exwiw::MongodbParallelPlan` (the static, DB-free classification) and
+`Exwiw::MongodbParallelDumper` (the fork orchestrator: per-group pools, LPT
+bin-packing, the `@state` Marshal-sidecar IPC, and the Phase-2 cascade). The
+`Runner` delegates the whole schema+inserts pass to the dumper when the mongodb
+adapter is used with `--parallel-workers=N` (N≥2), a genuine-anchor dump target is
+present, and the runtime can `fork`; otherwise it runs the serial loop unchanged.
+The CLI exposes `--parallel-workers` / config-file `parallel_workers` (mongodb +
+`export` only). The after-insert hook runs identically on both paths.
+
+End-to-end verification through the real `exwiw export` CLI on the same staging
+restore: **189/189 output files byte-identical** to the serial CLI run (same
+filenames, 0 content mismatches), at **2.19× wall-clock** (serial 7.13 s → N=4
+3.25 s; N=2 3.99 s = 1.81×; N=6 saturates at 3.25 s). Per the curve above the win
+materializes from ~4 real cores.
+
+This was the machinery `optimization-notes.md` deliberately removed as
+over-engineered for a flag — re-introduced here because, unlike that removed work,
+this schedule is byte-identical by construction, measured past the 2× target on a
+real extraction, and the lever the task explicitly invited (scale the task to go
+faster). It is **strictly opt-in**: the default remains the serial path.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -70,6 +70,25 @@ module Exwiw
         @state = {}
       end
 
+      # Propagation @state accessor, used ONLY by MongodbParallelDumper to seed a
+      # forked worker with the slice of parent ids its collections reference and to
+      # harvest the ids downstream collections will `$in`-match against (handed
+      # between processes as Marshal sidecars). The serial Runner never touches
+      # these — it relies on the in-process capture during #execute.
+      attr_accessor :state
+
+      # Cheap, metadata-only document-count estimate for `collection_name`, used by
+      # the parallel dumper to weight collections for LPT bin-packing. This only
+      # influences which worker processes a collection (never the output bytes), so
+      # an imprecise estimate is harmless. Reads collection metadata rather than
+      # running a COLLSCAN; returns 0 on any error (e.g. a collection absent from
+      # this database just sorts to the lowest weight).
+      def estimated_count(collection_name)
+        db[collection_name].estimated_document_count
+      rescue StandardError
+        0
+      end
+
       def dumpable?(config)
         !config.embedded?
       end

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -229,11 +229,18 @@ module Exwiw
       def compile_ast(query_ast, count_only: false)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if count_only
                  "COUNT(*)"
                elsif query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
                end
@@ -256,14 +263,43 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table:
+      #
+      #   JOIN (SELECT DISTINCT src.<proj> AS exwiw_scope_id
+      #         FROM (<subquery>) AS src) AS ids
+      #     ON <table>.<col> = ids.exwiw_scope_id
+      #
+      # The DISTINCT makes the derived table non-mergeable, so MySQL materializes
+      # the id-set once and probes this table by it (PK/index lookup) — instead
+      # of full-scanning this table and re-evaluating a correlated
+      # `IN (… UNION …)` per row (the DEPENDENT SUBQUERY / IN-to-EXISTS fallback,
+      # which a UNION subquery cannot be turned into a materialized semi-join).
+      # DISTINCT also dedups, so the join never fans out: the row set is identical
+      # to `<col> IN (subquery)`.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -301,9 +301,16 @@ module Exwiw
       def compile_ast(query_ast, select_cast_to: nil)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  cols = query_ast.columns.map { |col| compile_column_name(query_ast, col) }
                  cols = cols.map { |c| "#{c}::#{select_cast_to}" } if select_cast_to
@@ -337,14 +344,45 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table
+      # (see MysqlAdapter#compile_scope_join for the full rationale). The DISTINCT
+      # forces the engine to materialize the id-set once and probe this table by
+      # it, instead of full-scanning and re-evaluating a correlated subquery per
+      # row; it also dedups, so the join is row-for-row identical to
+      # `<col> IN (subquery)`.
+      #
+      # Type reconciliation mirrors the old IN form: when the outer column and
+      # the projected id clash (e.g. uuid vs varchar), #compile_subquery already
+      # casts every arm to text, so the derived `exwiw_scope_id` is text and the
+      # outer key is cast to match.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+
+        inner_sql = compile_subquery(subquery, outer_table: from_table_name, outer_column: where_clause.column_name)
+        cast_to = subquery_cast_to(subquery, from_table_name, where_clause.column_name)
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+        outer_key = "#{outer_key}::#{cast_to}" if cast_to
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{inner_sql}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -198,11 +198,18 @@ module Exwiw
       def compile_ast(query_ast, count_only: false)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 
+        # Lift scope id-set clauses (reverse_scope UNION / forward cascade /
+        # single referenced_by) out of `WHERE <col> IN (subquery)` and into a
+        # JOIN against a materialized derived table. See #compile_scope_join.
+        scope_clauses, plain_where_clauses = partition_scope_clauses(query_ast.where_clauses)
+
         sql = "SELECT "
         sql += if count_only
                  "COUNT(*)"
                elsif query_ast.select_all
-                 "*"
+                 # A lifted scope JOIN brings a derived table into FROM, so a bare
+                 # `*` would also project its column. Qualify to this table's own.
+                 scope_clauses.any? ? "#{query_ast.from_table_name}.*" : "*"
                else
                  query_ast.columns.map { |col| compile_column_name(query_ast, col) }.join(', ')
                end
@@ -225,14 +232,36 @@ module Exwiw
           end
         end
 
-        if query_ast.where_clauses.any?
+        scope_clauses.each_with_index do |where_clause, idx|
+          sql += " #{compile_scope_join(query_ast.from_table_name, where_clause, idx)}"
+        end
+
+        if plain_where_clauses.any?
           sql += " WHERE "
-          sql += query_ast.where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
+          sql += plain_where_clauses.map { |where| compile_where_condition(where, query_ast.from_table_name) }.join(' AND ')
         end
 
         sql
       end
 
+      # Render a scope id-set clause as a JOIN to a materialized derived table
+      # (see MysqlAdapter#compile_scope_join for the full rationale). The DISTINCT
+      # forces the engine to materialize the id-set once and probe this table by
+      # it, instead of full-scanning and re-evaluating a correlated subquery per
+      # row; it also dedups, so the join is row-for-row identical to
+      # `<col> IN (subquery)`.
+      private def compile_scope_join(from_table_name, where_clause, idx)
+        subquery = where_clause.value
+        projection = subquery_projection_name(subquery)
+        src_alias = "exwiw_scope_src_#{idx}"
+        ids_alias = "exwiw_scope_ids_#{idx}"
+        outer_key = "#{from_table_name}.#{where_clause.column_name}"
+
+        "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
+          "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
+          "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter.rb

@@ -242,6 +242,43 @@ module Exwiw
       private def null_preserving(ast, column, masked_expr)
         "CASE WHEN #{ast.from_table_name}.#{column.name} IS NOT NULL THEN #{masked_expr} ELSE NULL END"
       end
+
+      # Split an outer query's WHERE clauses into the scope id-set clauses to
+      # lift into a materialized derived-table JOIN (see each adapter's
+      # #compile_scope_join) and the remaining plain clauses (kept in WHERE).
+      # Returns [scope_clauses, plain_clauses]; #partition keeps each clause in
+      # its original order *within* its own group. The two groups are emitted in
+      # different SQL positions (a JOIN vs the WHERE), so their interleaving is
+      # irrelevant — only the order within each group matters, and that is kept.
+      private def partition_scope_clauses(where_clauses)
+        where_clauses.partition { |where_clause| scope_subquery_clause?(where_clause) }
+      end
+
+      # Whether a WHERE clause is a scope id-set probe that should be lifted into
+      # a JOIN against a materialized derived table. Only the SelectSubquery /
+      # UnionSubquery shapes (reverse_scope UNION, forward cascade, single
+      # referenced_by) qualify: they project over potentially huge tables and, as
+      # `<col> IN (subquery)`, can degrade into a correlated DEPENDENT SUBQUERY
+      # re-evaluated per outer row. The flat ids_field `Subquery` is deliberately
+      # left as a plain IN — it is a small, bounded, uncorrelated probe.
+      private def scope_subquery_clause?(where_clause)
+        where_clause.is_a?(Exwiw::QueryAst::WhereClause) &&
+          where_clause.operator == :in_subquery &&
+          (where_clause.value.is_a?(Exwiw::QueryAst::SelectSubquery) ||
+            where_clause.value.is_a?(Exwiw::QueryAst::UnionSubquery))
+      end
+
+      # The bare name of the single column a scope subquery projects, used to
+      # reference it inside the materialized derived table. For a UNION the
+      # output column name comes from the first arm.
+      private def subquery_projection_name(subquery)
+        case subquery
+        when Exwiw::QueryAst::SelectSubquery
+          subquery.query.columns.first.name
+        when Exwiw::QueryAst::UnionSubquery
+          subquery.queries.first.columns.first.name
+        end
+      end
     end
 
     # @params [Exwiw::QueryAst] query_ast

data/lib/exwiw/cli.rb

@@ -35,6 +35,7 @@ module Exwiw
       ids
       ids_field
       scope_column
+      parallel_workers
     ].freeze
 
     # Database connection settings are environment-specific (and sometimes
@@ -44,7 +45,7 @@ module Exwiw
 
     # Keys that only make sense for `export`. They are skipped when merging config
     # for `explain` so a shared config file does not trip validate_explain_only!.
-    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook].freeze
+    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook parallel_workers].freeze
 
     def self.start(argv)
       new(argv).run
@@ -80,6 +81,7 @@ module Exwiw
       @output_format = nil
       @insert_only = nil
       @after_insert_hook_path = nil
+      @parallel_workers = nil
       # nil (not :info) so we can tell "user passed --log-level" from the default,
       # letting a config-file value fill in; the :info default is applied later.
       @log_level = nil
@@ -125,6 +127,7 @@ module Exwiw
           output_format: @output_format,
           insert_only: @insert_only,
           after_insert_hook_path: @after_insert_hook_path,
+          parallel_workers: @parallel_workers,
           cli_options: build_cli_options_hash,
           logger: logger,
         ).run
@@ -165,6 +168,7 @@ module Exwiw
       resolve_scope_column!
       resolve_ids_field!
       resolve_uri_option!
+      resolve_parallel_workers!
 
       if @subcommand == "explain"
         validate_explain_only!
@@ -316,6 +320,7 @@ module Exwiw
       end
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
+      @parallel_workers ||= parse_parallel_workers(config["parallel_workers"]) if config.key?("parallel_workers")
     end
 
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
@@ -409,6 +414,38 @@ module Exwiw
       end
     end
 
+    # `--parallel-workers` opts into the MongoDB fork-parallel dump schedule
+    # (docs/mongodb-dump-parallelism-2x-notes.md). It is mongodb-only (the SQL
+    # adapters shell out to their own dumpers) and must be a positive integer;
+    # N<2 is accepted but runs serially. Runs after the adapter name is normalized
+    # so the family check is reliable. `explain` rejection is handled separately
+    # by validate_explain_only!.
+    private def resolve_parallel_workers!
+      return if @parallel_workers.nil?
+
+      if @database_adapter != "mongodb"
+        $stderr.puts "--parallel-workers is only supported by the mongodb adapter"
+        exit 1
+      end
+
+      if @parallel_workers < 1
+        $stderr.puts "--parallel-workers must be a positive integer (got #{@parallel_workers})"
+        exit 1
+      end
+    end
+
+    # Coerce a config-file `parallel_workers` (YAML scalar) to Integer, matching
+    # the CLI flag's Integer coercion. A non-integer value is a config typo, so
+    # fail fast rather than silently dropping it.
+    private def parse_parallel_workers(value)
+      return nil if value.nil?
+
+      Integer(value)
+    rescue ArgumentError, TypeError
+      $stderr.puts "config 'parallel_workers' must be an integer (got #{value.inspect})"
+      exit 1
+    end
+
     private def validate_explain_only!
       if @database_adapter == "mongodb"
         $stderr.puts "mongodb adapter is not yet supported by 'explain' subcommand"
@@ -420,6 +457,7 @@ module Exwiw
       rejected << "--output-format" unless @output_format.nil?
       rejected << "--insert-only" unless @insert_only.nil?
       rejected << "--after-insert-hook" unless @after_insert_hook_path.nil?
+      rejected << "--parallel-workers" unless @parallel_workers.nil?
 
       unless rejected.empty?
         $stderr.puts "The following options are not applicable in 'explain' subcommand: #{rejected.join(', ')}"
@@ -526,6 +564,7 @@ module Exwiw
         opts.on("--after-insert-hook=PATH", "Path to a .rb or .sh post-processing hook executed after all insert/delete files are written (export subcommand only)") do |v|
           @after_insert_hook_path = File.expand_path(v)
         end
+        opts.on("--parallel-workers=N", Integer, "Fork N workers for the MongoDB dump's parallel schedule (mongodb + export only; N>=2 enables it, default is serial). Output is byte-identical to serial; falls back to serial where fork is unavailable.") { |v| @parallel_workers = v }
         opts.on("--log-level=LEVEL", "Log level (debug, info). default is info") { |v| @log_level = v.to_sym }
 
         opts.on("--help", "Print this help") do

data/lib/exwiw/mongodb_parallel_dumper.rb

@@ -0,0 +1,290 @@
+# frozen_string_literal: true
+
+require "set"
+require "fileutils"
+require "tmpdir"
+
+module Exwiw
+  # Runs the inter-collection fork schedule from
+  # docs/mongodb-dump-parallelism-2x-notes.md, producing output **byte-identical**
+  # to the serial Runner while parallelizing the dominant cost (the Mongo driver's
+  # BSON->Ruby decode) across processes — each worker decodes its own collections
+  # in their natural order, so order is preserved and the result still matches a
+  # serial dump.
+  #
+  # It consumes the static, config-derived classification from MongodbParallelPlan
+  # (the three groups + cascade adjacency + ref_bt components) and adds the live
+  # orchestration the plan deliberately leaves out: a fork pool per group, LPT
+  # bin-packing on a per-collection cost weight, @state Marshal sidecar IPC for the
+  # handful of referenced leaves, and the Phase-2 cascade reprocess.
+  #
+  # The schedule (one parent process + a pool of `workers` forks):
+  #
+  #   Phase 1 (concurrent): fork the leaf pool; the parent meanwhile dumps the
+  #     schema and processes the WHOLE genuine DAG optimistically (no leaf @state
+  #     yet), recording each genuine collection's row count.
+  #   Barrier: wait for the leaf pool; load the Marshal sidecars the consumed
+  #     leaves wrote into the parent's @state.
+  #   Phase 2 (cascade): reprocess only the genuine collections whose output can
+  #     change now that leaf @state is present (the direct-leaf referencers),
+  #     cascading to genuine children of any whose row count actually changed.
+  #   Phase 3: fork the ref_bt collections as dependency-closed components, each
+  #     worker owning whole components (processed in topological order) seeded with
+  #     the leaf @state its members reference.
+  #
+  # Output bytes are independent of the schedule: every collection writes its own
+  # insert-NNN-<name>.<ext> file (the index taken over the plan's full ordering,
+  # exactly as the serial Runner numbers them) and the per-collection write is the
+  # same build_query -> execute -> write_inserts pass the Runner performs. The
+  # bin-packing only decides which worker runs which collection, never the bytes.
+  #
+  # fork is required; callers must check {.available?} and fall back to the serial
+  # Runner on JRuby/TruffleRuby/Windows.
+  class MongodbParallelDumper
+    # True when the runtime can `fork` (CRuby on a POSIX OS). On JRuby/TruffleRuby
+    # and Windows it cannot — the caller must run the serial Runner instead.
+    def self.available?
+      Process.respond_to?(:fork)
+    end
+
+    # Longest-Processing-Time bin-packing: assign `items` to `bins` bins, heaviest
+    # first onto the currently least-loaded bin. Returns an Array of `bins` arrays
+    # (some may be empty when items < bins). `weight` is called exactly once per
+    # item (it may be DB-backed, so it must not be invoked repeatedly). Pure — no
+    # DB, no IO — so it is unit-tested directly.
+    def self.bin_pack(items, bins, &weight)
+      raise ArgumentError, "bins must be >= 1 (got #{bins})" if bins < 1
+
+      weighted = items.map { |item| [item, weight.call(item)] }.sort_by { |(_, w)| -w }
+      groups = Array.new(bins) { [] }
+      loads = Array.new(bins, 0)
+      weighted.each do |(item, w)|
+        i = (0...bins).min_by { |j| loads[j] }
+        groups[i] << item
+        loads[i] += w
+      end
+      groups
+    end
+
+    # @param connection_config [ConnectionConfig] used to build a FRESH adapter in
+    #   the parent and in every fork (a Mongo client cannot be shared across fork)
+    # @param plan [MongodbParallelPlan] the static classification for this dump
+    # @param dump_target [DumpTarget]
+    # @param table_by_name [Hash{String=>config}] ALL configs (embedded included),
+    #   exactly as Runner builds it
+    # @param output_dir [String]
+    # @param workers [Integer] fork pool size (>= 1)
+    # @param logger [Logger]
+    # @param weight_for [#call, nil] optional name -> numeric cost weight for LPT;
+    #   defaults to the adapter's metadata-only estimated document count
+    def initialize(connection_config:, plan:, dump_target:, table_by_name:, output_dir:, workers:, logger:, weight_for: nil)
+      raise ArgumentError, "workers must be >= 1 (got #{workers})" if workers < 1
+
+      @connection_config = connection_config
+      @plan = plan
+      @dump_target = dump_target
+      @table_by_name = table_by_name
+      @output_dir = output_dir
+      @workers = workers
+      @logger = logger
+      @weight_for = weight_for
+    end
+
+    # Execute the full schedule. Assumes the caller has already cleaned the output
+    # directory (the Runner does this before handing off), mirroring the serial
+    # path which dumps the schema into a freshly-cleaned dir. Returns a small stats
+    # Hash. Raises if any worker pool reports a non-zero exit.
+    def run
+      raise "fork is unavailable on this runtime; run the serial Runner instead" unless self.class.available?
+
+      FileUtils.mkdir_p(@output_dir)
+      parent = build_adapter
+
+      Dir.mktmpdir("exwiw-mongo-parallel-") do |sidecar_dir|
+        phase1_leaf_and_genuine(parent, sidecar_dir)
+        phase2_cascade(parent, sidecar_dir)
+        phase3_ref_components(parent, sidecar_dir)
+      end
+
+      {
+        workers: @workers,
+        genuine: @plan.genuine.size,
+        leaves: @plan.leaves.size,
+        ref_bt: @plan.ref_bt.size,
+        components: @plan.reference_components.map(&:size).sort.reverse,
+      }
+    end
+
+    private
+
+    # Phase 1: fork the leaf pool to run concurrently while the parent dumps the
+    # schema (parent-only, needs no @state) and processes the whole genuine DAG
+    # optimistically. The genuine row counts captured here seed the Phase-2 cascade.
+    def phase1_leaf_and_genuine(parent, sidecar_dir)
+      leaf_master = fork do
+        ok = run_leaf_pool(sidecar_dir)
+        exit!(ok ? 0 : 1)
+      end
+
+      schema_path = File.join(@output_dir, "insert-000-schema.#{parent.schema_output_extension}")
+      ordered_tables = @plan.ordered_all.map { |name| @table_by_name.fetch(name) }
+      @logger.info("Writing schema to #{schema_path}...")
+      parent.dump_schema(ordered_tables, schema_path)
+
+      @logger.info("Processing #{@plan.genuine.size} genuine collection(s) (parent, optimistic pass)...")
+      @row_counts = {}
+      @plan.genuine.each { |name| @row_counts[name] = process_collection(parent, name) }
+
+      Process.wait(leaf_master)
+      raise "exwiw parallel leaf pool failed (exit #{$?.exitstatus})" unless $?.exitstatus&.zero?
+    end
+
+    # Barrier + Phase 2: load the consumed-leaf @state the leaf workers handed back,
+    # then reprocess only the genuine collections whose output can change now that
+    # leaf @state is present, cascading to genuine children of any that changed.
+    def phase2_cascade(parent, sidecar_dir)
+      load_sidecars(parent, @plan.consumed_leaves, sidecar_dir)
+
+      queue = @plan.direct_leaf_genuine.dup
+      seen = Set.new
+      until queue.empty?
+        name = queue.shift
+        next if seen.include?(name)
+
+        seen << name
+        new_count = process_collection(parent, name)
+        next if new_count == @row_counts[name]
+
+        @row_counts[name] = new_count
+        @plan.genuine_children[name].each { |child| queue << child }
+      end
+      @logger.info("Cascade reprocessed #{seen.size} genuine collection(s) with leaf @state.") unless seen.empty?
+    end
+
+    # Phase 3: fork the ref_bt collections as dependency-closed weakly-connected
+    # components in a single pool (no level barriers, no cross-worker IPC). Each
+    # worker owns whole components and processes their members in topological order,
+    # seeded only with the leaf @state those members reference.
+    def phase3_ref_components(parent, sidecar_dir)
+      components = @plan.reference_components
+      return if components.empty?
+
+      leaf_state = parent.state
+      groups = self.class.bin_pack(components, @workers) { |component| component.sum { |name| weight_of(parent, name) } }
+
+      pids = groups.reject(&:empty?).map do |group|
+        members = group.flatten
+        seed = leaf_state.slice(*parents_of(members))
+        fork { run_component_worker(group, seed) }
+      end
+      ok = pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
+      raise "exwiw parallel ref_bt pool failed" unless ok
+
+      @logger.info("Processed #{@plan.ref_bt.size} ref_bt collection(s) in #{groups.reject(&:empty?).size} worker(s).")
+    end
+
+    # Fork `@workers` leaf workers (LPT-packed on cost weight so the single heaviest
+    # leaf sits alone) and wait for them. Each worker writes a Marshal sidecar for
+    # the consumed leaves it produced. Runs inside the leaf_master fork, so its own
+    # weight adapter and the worker connections never touch the parent's.
+    def run_leaf_pool(sidecar_dir)
+      return true if @plan.leaves.empty?
+
+      weight_adapter = build_adapter
+      groups = self.class.bin_pack(@plan.leaves, @workers) { |name| weight_of(weight_adapter, name) }
+
+      pids = groups.reject(&:empty?).map do |group|
+        fork { run_leaf_worker(group, sidecar_dir) }
+      end
+      pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
+    rescue StandardError => e
+      @logger.error("exwiw parallel leaf master error: #{e.class}: #{e.message}")
+      false
+    end
+
+    def run_leaf_worker(group, sidecar_dir)
+      adapter = build_adapter
+      group.each { |name| process_collection(adapter, name) }
+      group.each do |name|
+        next unless @plan.consumed_leaves.include?(name)
+        next unless adapter.state.key?(name)
+
+        File.binwrite(File.join(sidecar_dir, "#{name}.marshal"), Marshal.dump(adapter.state[name]))
+      end
+      exit!(0)
+    rescue StandardError => e
+      @logger.error("exwiw parallel leaf worker error (#{group.first}..): #{e.class}: #{e.message}")
+      exit!(1)
+    end
+
+    def run_component_worker(group, seed)
+      adapter = build_adapter
+      adapter.state = seed unless seed.empty?
+      # Each component is already topologically ordered (parent before child) and
+      # dependency-closed over intra-ref_bt edges, so a plain serial walk suffices.
+      group.each { |component| component.each { |name| process_collection(adapter, name) } }
+      exit!(0)
+    rescue StandardError => e
+      @logger.error("exwiw parallel ref_bt worker error (#{group.first&.first}..): #{e.class}: #{e.message}")
+      exit!(1)
+    end
+
+    # Extract one collection to its insert-NNN-<name>.<ext> file. This mirrors the
+    # serial Runner's non-COPY insert path exactly — same filename (index taken over
+    # the plan's full ordering), same pre/post hooks (nil for MongoDB), same
+    # streaming write_inserts + trailing "\n", and the same empty-result handling
+    # (delete the just-opened file) — so the bytes are identical regardless of which
+    # process writes them. Returns the row count.
+    def process_collection(adapter, name)
+      table = @table_by_name.fetch(name)
+      query = adapter.build_query(table, @dump_target, @table_by_name)
+      results = adapter.execute(query)
+
+      insert_idx = (@plan.index_of.fetch(name) + 1).to_s.rjust(3, "0")
+      path = File.join(@output_dir, "insert-#{insert_idx}-#{name}.#{adapter.output_extension}")
+      chunk_size = table.bulk_insert_chunk_size || adapter.default_bulk_insert_chunk_size
+
+      record_num = 0
+      File.open(path, "w") do |file|
+        pre = adapter.pre_insert_sql(table)
+        file.puts(pre) if pre
+        _statement_count, record_num = adapter.write_inserts(file, results, table, chunk_size)
+        file.print("\n")
+        post = adapter.post_insert_sql(table)
+        file.puts(post) if post
+      end
+      File.delete(path) if record_num.zero?
+      record_num
+    end
+
+    # Merge the Marshal sidecars the leaf workers wrote (one per consumed leaf that
+    # actually produced rows) into `adapter`'s @state, so the cascade reprocess and
+    # the ref_bt workers can constrain on those leaf ids.
+    def load_sidecars(adapter, names, sidecar_dir)
+      state = adapter.state
+      names.each do |name|
+        path = File.join(sidecar_dir, "#{name}.marshal")
+        state[name] = Marshal.load(File.binread(path)) if File.exist?(path)
+      end
+    end
+
+    # The distinct belongs_to parent names of `names`, used to slice the leaf @state
+    # a worker is seeded with down to only the keys its collections reference.
+    def parents_of(names)
+      names.flat_map { |name| @table_by_name.fetch(name).belongs_tos.map(&:table_name) }.uniq
+    end
+
+    def weight_of(adapter, name)
+      return @weight_for.call(name) if @weight_for
+
+      adapter.estimated_count(name)
+    end
+
+    # A fresh adapter (and thus a fresh, lazily-opened Mongo connection). Built per
+    # process — the parent and every fork get their own; a Mongo client must never
+    # be shared across a fork boundary.
+    def build_adapter
+      Adapter.build(@connection_config, @logger)
+    end
+  end
+end

data/lib/exwiw/mongodb_parallel_plan.rb

@@ -0,0 +1,271 @@
+# frozen_string_literal: true
+
+require "set"
+
+module Exwiw
+  # Classifies a MongoDB dump's collections into the three dependency groups the
+  # inter-collection fork schedule needs, plus the derived adjacency that
+  # schedule consumes. See docs/mongodb-dump-parallelism-2x-notes.md for the why;
+  # this class is the static, config-derived half of that plan.
+  #
+  # It is a pure function of the loaded configs and the dump target — no DB
+  # access — so it can be computed once up front and unit-tested without a live
+  # MongoDB. The fork orchestration (worker pools, LPT bin-packing on output-size
+  # weights, @state Marshal sidecars, the Phase-2 cascade loop) lives elsewhere
+  # and consumes the structures produced here.
+  #
+  # Input contract: `configs` are MongodbCollectionConfig already passed through
+  # `#reject_ignored_members!` (exactly as Runner#load_table_config produces
+  # them), so every surviving belongs_to has a non-nil `table_name`. ignore:true
+  # *collections* are still present in `configs` — they contribute to the schema
+  # and to the file-index ordering, but their data extraction is skipped — and
+  # are therefore excluded from the three processing groups.
+  #
+  # The three groups partition the extractable collections exactly:
+  #
+  # - **genuine**  — reachable to the dump target by following belongs_to edges
+  #                  (the scoped DAG). Includes the target itself.
+  # - **leaf**     — no belongs_to at all: reference/master data dumped in full,
+  #                  with no input dependencies (embarrassingly parallel).
+  # - **ref_bt**   — has belongs_to but is NOT reachable to the target: reference
+  #                  data scoped by the adapter's strict-AND fallback. Its
+  #                  internal edges form shallow components.
+  #
+  # `reachable` mirrors MongodbAdapter#genuine_scope_set exactly (fixpoint over
+  # all non-embedded configs, including ignore:true ones), so the genuine set
+  # here matches the adapter's runtime scoping classification.
+  class MongodbParallelPlan
+    EMPTY_NAMES = [].freeze
+    private_constant :EMPTY_NAMES
+
+    # @param configs [Array<MongodbCollectionConfig>] reject_ignored_members!'d
+    # @param target_table_name [String] the dump target collection
+    # @param logger [Logger, nil] forwarded to DetermineTableProcessingOrder
+    def initialize(configs:, target_table_name:, logger: nil)
+      @by = configs.each_with_object({}) { |c, h| h[c.name] = c }
+      @target_table_name = target_table_name
+
+      dumpable = configs.reject(&:embedded?)
+      # The file index (insert-NNN-) is taken over the FULL processing order,
+      # including ignore:true collections, so the orchestrated run's filenames
+      # are byte-identical to the serial Runner's (which numbers files the same
+      # way). Data extraction, however, skips ignore:true — see #extractable.
+      @ordered_all = DetermineTableProcessingOrder.run(dumpable, logger: logger).freeze
+      @index_of = @ordered_all.each_with_index.to_h.freeze
+      @extractable = @ordered_all.reject { |n| @by[n].ignore }.freeze
+
+      @reachable = compute_reachable
+      classify
+      derive_consumed_leaves
+      derive_cascade_adjacency
+      @reference_components = compute_reference_components.freeze
+    end
+
+    # Full processing order, INCLUDING ignore:true collections — the sequence the
+    # file index (insert-NNN-) is numbered over.
+    attr_reader :ordered_all
+
+    # name => 0-based position in #ordered_all (the file index is position + 1).
+    attr_reader :index_of
+
+    # #ordered_all minus ignore:true collections — the collections whose data is
+    # actually extracted. Union of the three groups below.
+    attr_reader :extractable
+
+    # The three groups (each a subset of #extractable, in #ordered_all order):
+
+    # genuine — reachable to the dump target (includes the target).
+    attr_reader :genuine
+
+    # leaf — no belongs_to; reference/master data with no input dependencies.
+    attr_reader :leaves
+
+    # ref_bt — has belongs_to but not reachable to the target.
+    attr_reader :ref_bt
+
+    # ref_bt collections as dependency-closed weakly-connected components over
+    # intra-ref_bt belongs_to edges, each returned in a valid topological order
+    # (a parent before its child). A whole component can be processed serially by
+    # one worker with no cross-worker @state IPC and no level barriers, seeded
+    # only with the leaf @state its members reference.
+    attr_reader :reference_components
+
+    # Leaf collections referenced (via belongs_to) by some non-leaf extractable
+    # collection (genuine OR ref_bt). These are the only leaves whose captured
+    # @state a downstream collection can need, so they are the ones a leaf worker
+    # must hand back (e.g. as a Marshal sidecar). Set<String>.
+    attr_reader :consumed_leaves
+
+    # genuine collections that directly reference a leaf — the only genuine
+    # collections whose output can change once leaf @state is present (and only
+    # at runtime, when their genuine anchor turns out empty and they fall back to
+    # the leaf clause). These seed the Phase-2 cascade reprocess.
+    attr_reader :direct_leaf_genuine
+
+    # name => genuine children (genuine collections that belongs_to it), keyed
+    # only by reachable parents. Drives the Phase-2 cascade: when a reprocessed
+    # collection's row count changes, its genuine children are re-enqueued.
+    attr_reader :genuine_children
+
+    # The set of collection names genuinely scoped by the target (the target plus
+    # everything that can reach it through belongs_to). Exposed for inspection.
+    attr_reader :reachable
+
+    def summary
+      {
+        extractable: @extractable.size,
+        genuine: @genuine.size,
+        leaves: @leaves.size,
+        ref_bt: @ref_bt.size,
+        consumed_leaves: @consumed_leaves.size,
+        direct_leaf_genuine: @direct_leaf_genuine.size,
+        reference_components: @reference_components.map(&:size).sort.reverse,
+      }
+    end
+
+    private
+
+    # Fixpoint over non-embedded configs: the target, plus every collection that
+    # can reach it by following belongs_to (child -> parent) transitively.
+    # Mirrors MongodbAdapter#genuine_scope_set (same traversal, same inclusion of
+    # ignore:true collections) so the genuine set matches the adapter's runtime
+    # scoping decision.
+    def compute_reachable
+      reachable = Set.new([@target_table_name])
+      loop do
+        added = false
+        @by.each_value do |cfg|
+          next if cfg.embedded? || reachable.include?(cfg.name)
+          next unless cfg.belongs_tos.any? { |rel| reachable.include?(rel.table_name) }
+
+          reachable << cfg.name
+          added = true
+        end
+        break unless added
+      end
+      reachable
+    end
+
+    def classify
+      # The three groups partition #extractable: reachable -> genuine; otherwise
+      # leaf (no belongs_to) -> leaves; otherwise -> ref_bt. The target is
+      # reachable (it seeds the set), so it lands in genuine and is never
+      # mis-grouped as a leaf even when it has no belongs_to of its own — which
+      # would otherwise double-process it (leaf pool AND parent).
+      @genuine = []
+      @leaves = []
+      @ref_bt = []
+      @extractable.each do |name|
+        if @reachable.include?(name)
+          @genuine << name
+        elsif leaf?(name)
+          @leaves << name
+        else
+          @ref_bt << name
+        end
+      end
+      @genuine.freeze
+      @leaves.freeze
+      @ref_bt.freeze
+      # Membership against the leaf *group* (which excludes the target), not the
+      # raw structural #leaf? predicate. The target has no belongs_to and is thus
+      # structurally leaf-like, but it is genuine — processed by the parent, not a
+      # leaf worker — so a belongs_to to the target must not count as referencing
+      # a leaf (it would wrongly demand a sidecar / seed the cascade).
+      @leaf_set = @leaves.to_set
+    end
+
+    def derive_consumed_leaves
+      consumed = Set.new
+      (@genuine + @ref_bt).each do |name|
+        @by[name].belongs_tos.each do |rel|
+          consumed << rel.table_name if @leaf_set.include?(rel.table_name)
+        end
+      end
+      @consumed_leaves = consumed.freeze
+    end
+
+    def derive_cascade_adjacency
+      @direct_leaf_genuine = @genuine.select do |name|
+        @by[name].belongs_tos.any? { |rel| @leaf_set.include?(rel.table_name) }
+      end.freeze
+
+      children = Hash.new { |h, k| h[k] = [] }
+      @genuine.each do |name|
+        @by[name].belongs_tos.each do |rel|
+          children[rel.table_name] << name if @reachable.include?(rel.table_name)
+        end
+      end
+      # Freeze with a non-mutating default so a lookup of a parent with no genuine
+      # children returns [] without trying to write into the frozen hash.
+      children.default_proc = nil
+      children.default = EMPTY_NAMES
+      @genuine_children = children.freeze
+    end
+
+    # ref_bt as dependency-closed weakly-connected components over intra-ref_bt
+    # belongs_to edges, each topo-ordered. Ported from the bench prototype: build
+    # the directed (child indegree) and undirected (component) views of the
+    # intra-ref_bt edges, find weakly-connected components, then Kahn-order each.
+    def compute_reference_components
+      ref_set = @ref_bt.to_set
+      children = Hash.new { |h, k| h[k] = [] }
+      adjacency = Hash.new { |h, k| h[k] = [] }
+      @ref_bt.each do |name|
+        @by[name].belongs_tos.each do |rel|
+          next unless ref_set.include?(rel.table_name)
+
+          children[rel.table_name] << name
+          adjacency[rel.table_name] << name
+          adjacency[name] << rel.table_name
+        end
+      end
+
+      seen = Set.new
+      components = []
+      @ref_bt.each do |start|
+        next if seen.include?(start)
+
+        stack = [start]
+        members = []
+        until stack.empty?
+          node = stack.pop
+          next if seen.include?(node)
+
+          seen << node
+          members << node
+          adjacency[node].each { |neighbor| stack << neighbor unless seen.include?(neighbor) }
+        end
+        components << members
+      end
+
+      components.map { |members| topo_order(members, children) }
+    end
+
+    # Kahn topological order of `members` over intra-component belongs_to edges
+    # (parent before child). `children` is the directed intra-ref_bt adjacency.
+    def topo_order(members, children)
+      member_set = members.to_set
+      indegree = members.to_h do |name|
+        [name, @by[name].belongs_tos.count { |rel| member_set.include?(rel.table_name) }]
+      end
+      queue = members.select { |name| indegree[name].zero? }
+      ordered = []
+      until queue.empty?
+        node = queue.shift
+        ordered << node
+        children[node].each do |child|
+          next unless member_set.include?(child)
+
+          indegree[child] -= 1
+          queue << child if indegree[child].zero?
+        end
+      end
+      ordered
+    end
+
+    def leaf?(name)
+      (cfg = @by[name]) && !cfg.embedded? && cfg.belongs_tos.empty?
+    end
+  end
+end

data/lib/exwiw/runner.rb

@@ -13,6 +13,7 @@ module Exwiw
       output_format: 'insert',
       insert_only: false,
       after_insert_hook_path: nil,
+      parallel_workers: nil,
       cli_options: {}
     )
       @connection_config = connection_config
@@ -22,6 +23,7 @@ module Exwiw
       @output_format = output_format
       @insert_only = insert_only
       @after_insert_hook_path = after_insert_hook_path
+      @parallel_workers = parallel_workers
       @cli_options = cli_options
       @logger = logger
     end
@@ -49,6 +51,19 @@ module Exwiw
 
       clean_output_dir!
 
+      # Opt-in MongoDB inter-collection fork parallelism (see
+      # docs/mongodb-dump-parallelism-2x-notes.md). It is byte-identical to the
+      # serial loop below — same filenames (the file index is taken over the same
+      # full processing order) and same per-collection bytes — so it is a drop-in
+      # replacement for the whole schema+inserts pass, after which the common
+      # after-insert hook still runs. Everything before this point (validation,
+      # scope check, ordering, output-dir clean) applies to both paths.
+      if use_mongodb_parallel?(adapter)
+        dump_mongodb_parallel(configs, table_by_name)
+        run_after_insert_hook(adapter, ordered_table_names.size)
+        return
+      end
+
       ordered_tables = ordered_table_names.map { |n| table_by_name.fetch(n) }
       schema_path = File.join(@output_dir, "insert-000-schema.#{adapter.schema_output_extension}")
       @logger.info("Writing schema to #{schema_path}...")
@@ -161,17 +176,71 @@ module Exwiw
         end
       end
 
-      if @after_insert_hook_path
-        @logger.info("Running after-insert hook: #{@after_insert_hook_path}")
-        AfterInsertHook.run(
-          path: @after_insert_hook_path,
-          cli_options: @cli_options,
-          output_dir: @output_dir,
-          next_idx: total_size + 1,
-          output_extension: adapter.output_extension,
-          logger: @logger,
-        )
+      run_after_insert_hook(adapter, total_size)
+    end
+
+    # Run the post-processing hook (no-op when none configured). `total_size` is
+    # the count of processed tables/collections; the hook's first output file is
+    # numbered just past them. Shared by the serial and parallel dump paths.
+    private def run_after_insert_hook(adapter, total_size)
+      return unless @after_insert_hook_path
+
+      @logger.info("Running after-insert hook: #{@after_insert_hook_path}")
+      AfterInsertHook.run(
+        path: @after_insert_hook_path,
+        cli_options: @cli_options,
+        output_dir: @output_dir,
+        next_idx: total_size + 1,
+        output_extension: adapter.output_extension,
+        logger: @logger,
+      )
+    end
+
+    # True when the opt-in MongoDB fork-parallel dump should run instead of the
+    # serial loop: the mongodb adapter, a worker count > 1, a genuine-anchor dump
+    # target (the schedule is built around the scoped DAG), and a runtime that can
+    # fork. Anything else falls back to the serial path (warning when the user
+    # explicitly asked for parallelism but it cannot apply).
+    private def use_mongodb_parallel?(adapter)
+      return false unless adapter.is_a?(Adapter::MongodbAdapter)
+      return false unless @parallel_workers && @parallel_workers > 1
+
+      if @dump_target.table_name.nil?
+        @logger.warn("--parallel-workers ignored: MongoDB parallelism needs a --target-collection; running serially.")
+        return false
       end
+
+      unless MongodbParallelDumper.available?
+        @logger.warn("--parallel-workers ignored: fork is unavailable on this runtime; running serially.")
+        return false
+      end
+
+      true
+    end
+
+    # Build the static plan and hand the whole schema+inserts pass to the fork
+    # orchestrator. `configs` are the reject_ignored_members!'d configs (the plan
+    # rejects embedded and orders them itself, identically to the serial path).
+    private def dump_mongodb_parallel(configs, table_by_name)
+      plan = MongodbParallelPlan.new(
+        configs: configs,
+        target_table_name: @dump_target.table_name,
+        logger: @logger,
+      )
+      @logger.info(
+        "MongoDB parallel dump with #{@parallel_workers} worker(s): " \
+        "genuine=#{plan.genuine.size}, leaves=#{plan.leaves.size}, ref_bt=#{plan.ref_bt.size}."
+      )
+      stats = MongodbParallelDumper.new(
+        connection_config: @connection_config,
+        plan: plan,
+        dump_target: @dump_target,
+        table_by_name: table_by_name,
+        output_dir: @output_dir,
+        workers: @parallel_workers,
+        logger: @logger,
+      ).run
+      @logger.info("MongoDB parallel dump complete: #{stats.inspect}")
     end
 
     # Empty the output dir before writing so each export starts from a clean

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.3"
+  VERSION = "0.8.5"
 end

data/lib/exwiw.rb

@@ -23,6 +23,8 @@ require_relative "exwiw/adapter/mysql_adapter"
 require_relative "exwiw/adapter/postgresql_adapter"
 require_relative "exwiw/adapter/mongodb_adapter"
 require_relative "exwiw/determine_table_processing_order"
+require_relative "exwiw/mongodb_parallel_plan"
+require_relative "exwiw/mongodb_parallel_dumper"
 require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.3
+  version: 0.8.5
 platform: ruby
 authors:
 - Shia
@@ -72,6 +72,8 @@ files:
 - lib/exwiw/mongo_query.rb
 - lib/exwiw/mongodb_collection_config.rb
 - lib/exwiw/mongodb_field.rb
+- lib/exwiw/mongodb_parallel_dumper.rb
+- lib/exwiw/mongodb_parallel_plan.rb
 - lib/exwiw/mongoid_schema_generator.rb
 - lib/exwiw/query_ast.rb
 - lib/exwiw/query_ast_builder.rb