exwiw 0.8.4 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ee150e9fd830a7829be8c0eebbeca19f0e2d1b1dbcce50fe78e63d62e878997c
-  data.tar.gz: 31377340d8737ece209e1c0755eeac49ac40930562503e2c4d04f6362938e9d9
+  metadata.gz: 171e3c591c208afcb89c2a9330c2c80d67f8f86ab370858b4690d03c799d28bc
+  data.tar.gz: f913a31f6f8e2a29a9e4b2a296926c1266c17cc8655fd4cf33110899f2b468fd
 SHA512:
-  metadata.gz: 26151e0bf763b6f48e993d2025b4bac804a373fe2e3cfd71bb70d91b86b453826838ec683ebfd3f4f4336bb83b8a12d567d7dbda7f8a4ef48fc7f50bbe98145a
-  data.tar.gz: b29a2da42750dde93cd53eb56a11a592478134b690ccba31b59cd312f3685dad2713e63897d09cb77246a8be37dfe55890f1a8505c1aa8b5fd020b7bc7760a03
+  metadata.gz: a8d6e99e0881f4cdaa8d6d737cfbd0107382a6585f119df7d3068007ebb768a976b47fd59f78fb2d931ed95147c707523b3c8e5d1e9af0507584de93796e1ec1
+  data.tar.gz: f1cbada5c1adb8892cc297a5e864051da3bdc0b7447cbe1c386b32dde4020cbf1ac6be94935d7bb547c2ec7f4482ebf79b3e1df796bb526db7fd7d5fb42ce662

data/CHANGELOG.md

@@ -2,6 +2,17 @@
 
 ## [Unreleased]
 
+## [0.9.0] - 2026-06-30
+
+### Added
+
+- **`exwiw explain` now supports the mongodb adapter.** It runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) for the `find` each collection would be dumped with, and prints the explain document as JSON alongside the find description. Verbosity is configurable via the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins): `queryPlanner` (default), `executionStats`, or `allPlansExecution`. The default `queryPlanner` only **plans** the query — it does not execute it or scan any documents — so it is safe to point at a production source; `executionStats`/`allPlansExecution` run the real extraction query to gather runtime stats and should be used deliberately. Because a scoped collection's `belongs_to` ids are captured at runtime (which `explain` never does), `explain` fills each scoped collection's real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id instead of the match-nothing `{_id: {$in: []}}` fallback — so the plan reports the real `IXSCAN`/`COLLSCAN` (queryPlanner selects an index by field, not value; only the bound values are fake). `ExplainRunner` now prints the adapter-agnostic query description (the compiled SELECT for SQL, the find description for mongodb), so the SQL adapters' output is unchanged.
+- **The MongoDB fork-parallel dump (`--parallel-workers`) logs a per-collection extracted-record summary on completion.** The serial path already logs each collection's row count inline, but the parallel path previously emitted only an aggregate (`genuine`/`leaves`/`ref_bt` counts). The leaf and `ref_bt` collections are extracted in forked workers whose counts never reached the parent, so the per-collection numbers were unavailable. Each worker now hands its counts back through the same Marshal-sidecar IPC the consumed-leaf `@state` already uses; the parent merges them with the genuine counts it holds and logs one `name: count` line per collection in processing order (matching the `insert-NNN-` file numbering), plus a `total_records` field on the returned stats. This is purely additive metadata — it does not touch the output files, so the byte-identity guarantee is unchanged.
+
+## [0.8.5] - 2026-06-25
+
+- **`--parallel-workers=N` parallelizes the MongoDB dump across forked processes (opt-in, byte-identical).** When set with `N≥2` on the mongodb adapter's `export`, exwiw runs an inter-collection fork schedule that decodes whole collections in parallel while preserving each collection's natural row order, so the output files are byte-identical to a serial run (same filenames, same content). Collections are classified into three dependency groups — reference data dumped in full (no `belongs_to`), the scoped DAG reachable to the dump target, and non-reachable reference data — which lets the heavy full-dump collections run concurrently with the scoped pass; only a handful of small `@state` hand-offs cross process boundaries. The win needs real cores: it clears ~2× from 4 workers and saturates there. Requires a dump target and a `fork`-capable runtime (CRuby on POSIX); it falls back to the serial path on JRuby/TruffleRuby/Windows or when no target is given. Also settable as `parallel_workers:` in the config file. The default remains the serial dump.
+
 ## [0.8.4] - 2026-06-24
 
 ### Fixed

data/README.md

@@ -59,7 +59,7 @@ driver implements that auth handshake itself and sidesteps the issue.
 exwiw has two subcommands:
 
 - `export` (default) — generate INSERT/COPY SQL files. If the subcommand is omitted, `export` is assumed.
-- `explain` — print the compiled SQL and its `EXPLAIN` output for each query that `export` would run, without executing the SELECTs.
+- `explain` — print each query `export` would run together with its `EXPLAIN` output. SQL adapters compile the SELECT without executing it; mongodb runs the server's explain (defaulting to the execution-free `queryPlanner`).
 
 ### `exwiw export`
 
@@ -115,7 +115,7 @@ idx meaning is the same as insert sql.
 
 ### `exwiw explain`
 
-Print the compiled SQL and its `EXPLAIN` output (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) for each query that `export` would run, to stdout. No SELECT is executed. Supported for `mysql`, `postgresql`, and `sqlite`. The `mongodb` adapter is not yet supported.
+Print the query each `export` would run together with its `EXPLAIN` output, to stdout. For the SQL adapters (`mysql`, `postgresql`, `sqlite`) this is the compiled SELECT plus its `EXPLAIN` (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) — no SELECT is executed. For `mongodb` it is the `find` description plus the server's explain document as JSON.
 
 ```bash
 # preview the queries exwiw would run, without executing the SELECTs
@@ -129,6 +129,29 @@ exwiw explain \
 
 The `--output-dir`, `--output-format`, `--insert-only`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
 
+#### MongoDB explain verbosity
+
+The mongodb explain runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) at a configurable verbosity. The default, **`queryPlanner`, only plans the query and does not execute it**, so it is safe to point at a production source. Set it with the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins):
+
+| verbosity | behaviour |
+|---|---|
+| `queryPlanner` (default) | Plans the query only. **The query is not executed** — no documents are scanned. |
+| `executionStats` | **Runs the query** and reports runtime statistics (docs examined, time, etc.). |
+| `allPlansExecution` | Runs the winning plan **and the rejected candidate plans** to gather their stats. |
+
+```bash
+# inspect index usage of the real extraction query (executes it)
+EXWIW_MONGODB_EXPLAIN_VERBOSITY=executionStats exwiw explain \
+  --adapter=mongodb \
+  --uri="mongodb+srv://reader@cluster/app_production" \
+  --schema-dir=exwiw/schema \
+  --target-collection=shops --ids=...
+```
+
+`executionStats` and `allPlansExecution` execute the extraction query against the source, so use them deliberately on large/production collections.
+
+> **Scoped collections use a placeholder id.** A non-target collection is scoped by its parents' ids, which a real dump captures while running each parent query — but `explain` runs nothing. So for scoped collections, `explain` fills the real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id rather than real values. The plan still reflects the real dump: `queryPlanner` chooses an index by the queried *field*, not its value, so whether a scoped extraction does an `IXSCAN` or a `COLLSCAN` is reported correctly. Only the bound *values* are fake. (The target collection uses the real `--ids`; reference collections with no `belongs_to` use the real `{}` full scan.)
+
 ### Scope-column mode
 
 The default `--target-table` extraction assumes the schema converges on a single
@@ -286,6 +309,7 @@ Notes:
 - **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
 - Unknown keys are rejected so a typo surfaces immediately.
 - Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
+- `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [`exwiw explain`](#mongodb-explain-verbosity).
 
 ### Generator
 
@@ -769,6 +793,7 @@ The MongoDB adapter is experimental. To use it:
 - `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
 - `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only** (the SQL adapters have no equivalent).
 - Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
+- `--parallel-workers=N` (opt-in, `export` only) forks `N` worker processes that decode whole collections in parallel — the dominant cost on a large dump is the driver's BSON→Ruby decode, and each worker decodes its own collections in their natural order, so the output stays **byte-identical** to a serial run (same filenames and content). It needs a dump target (the schedule is built around the scoped DAG) and a `fork`-capable runtime (CRuby on POSIX), falling back to the serial path otherwise; it also accepts `parallel_workers:` in the config file. The speedup needs real cores to spend — it reaches ~2× from 4 workers and saturates there. The default is serial. See [`docs/mongodb-dump-parallelism-2x-notes.md`](docs/mongodb-dump-parallelism-2x-notes.md) for the schedule and measurements.
 - Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
   ```bash
   mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl

data/docs/mongodb-dump-parallelism-2x-notes.md

@@ -134,13 +134,24 @@ bounded by chunk size, and `N×` that stays well under the 7 GiB container limit
 
 ## Status
 
-This is a **measured, byte-identical proof** (bench prototype, not yet integrated
-into the Runner/CLI). Integrating it means re-introducing process orchestration
-and `@state` sidecar IPC — the machinery `optimization-notes.md` deliberately
-removed as over-engineered for a flag. It is recorded here because, unlike that
-removed work, this schedule is (a) byte-identical by construction, (b) measured
-past the 2× target on a real extraction, and (c) the lever the task explicitly
-invited (scale the task to go faster). A production version would gate it behind a
-worker-count option, fall back to serial where `fork` is unavailable
-(Windows/JRuby), and reuse the existing genuine-anchor classification to derive
-the three groups.
+**Integrated and shipped behind an opt-in flag.** The schedule lives in
+`Exwiw::MongodbParallelPlan` (the static, DB-free classification) and
+`Exwiw::MongodbParallelDumper` (the fork orchestrator: per-group pools, LPT
+bin-packing, the `@state` Marshal-sidecar IPC, and the Phase-2 cascade). The
+`Runner` delegates the whole schema+inserts pass to the dumper when the mongodb
+adapter is used with `--parallel-workers=N` (N≥2), a genuine-anchor dump target is
+present, and the runtime can `fork`; otherwise it runs the serial loop unchanged.
+The CLI exposes `--parallel-workers` / config-file `parallel_workers` (mongodb +
+`export` only). The after-insert hook runs identically on both paths.
+
+End-to-end verification through the real `exwiw export` CLI on the same staging
+restore: **189/189 output files byte-identical** to the serial CLI run (same
+filenames, 0 content mismatches), at **2.19× wall-clock** (serial 7.13 s → N=4
+3.25 s; N=2 3.99 s = 1.81×; N=6 saturates at 3.25 s). Per the curve above the win
+materializes from ~4 real cores.
+
+This was the machinery `optimization-notes.md` deliberately removed as
+over-engineered for a flag — re-introduced here because, unlike that removed work,
+this schedule is byte-identical by construction, measured past the 2× target on a
+real extraction, and the lever the task explicitly invited (scale the task to go
+faster). It is **strictly opt-in**: the default remains the serial path.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -68,6 +68,47 @@ module Exwiw
       def initialize(connection_config, logger)
         super
         @state = {}
+        @explain_placeholder = false
+      end
+
+      # A recognizably-fake ObjectId substituted for captured parent ids when
+      # building scope filters in `explain` placeholder mode (see
+      # #explain_scope_with_placeholders! and #parent_state_for).
+      EXPLAIN_PLACEHOLDER_OID_HEX = "ffffffffffffffffffffffff"
+
+      # Switch #build_query into placeholder-scope mode for `explain`. The
+      # mongodb scope of a non-target collection is its parents' captured ids,
+      # which the serial dump harvests at runtime in #execute. `explain` never
+      # executes a query, so that @state stays empty and every scoped child would
+      # otherwise fall back to the match-nothing `{_id: {$in: []}}` filter —
+      # hiding which field (e.g. a foreign key) the real dump would actually
+      # filter on, and thus whether it is indexed. In this mode #parent_state_for
+      # returns a placeholder id for each dumped parent instead of reading
+      # @state, so build_query emits the real foreign-key filter shape with a
+      # dummy value. queryPlanner picks an index by the queried FIELD, not the
+      # value, so index selection (IXSCAN vs COLLSCAN) is reported correctly even
+      # though the bound value is fake.
+      def explain_scope_with_placeholders!
+        @explain_placeholder = true
+      end
+
+      # Propagation @state accessor, used ONLY by MongodbParallelDumper to seed a
+      # forked worker with the slice of parent ids its collections reference and to
+      # harvest the ids downstream collections will `$in`-match against (handed
+      # between processes as Marshal sidecars). The serial Runner never touches
+      # these — it relies on the in-process capture during #execute.
+      attr_accessor :state
+
+      # Cheap, metadata-only document-count estimate for `collection_name`, used by
+      # the parallel dumper to weight collections for LPT bin-packing. This only
+      # influences which worker processes a collection (never the output bytes), so
+      # an imprecise estimate is harmless. Reads collection metadata rather than
+      # running a COLLSCAN; returns 0 on any error (e.g. a collection absent from
+      # this database just sorts to the lowest weight).
+      def estimated_count(collection_name)
+        db[collection_name].estimated_document_count
+      rescue StandardError
+        0
       end
 
       def dumpable?(config)
@@ -174,8 +215,30 @@ module Exwiw
         raise NotImplementedError, "MongodbAdapter does not support bulk delete"
       end
 
-      def explain(_query)
-        raise NotImplementedError, "MongodbAdapter does not support explain yet"
+      # Default explain verbosity. `queryPlanner` asks the server to PLAN the
+      # query without executing it, so it is safe to run against a production
+      # source — no documents are scanned or returned. `executionStats` and
+      # `allPlansExecution` actually run the query to collect runtime stats (the
+      # latter also runs the rejected candidate plans), so they carry the cost of
+      # the real extraction query.
+      DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
+
+      # Server-side explain for the find this dump would issue, returned as
+      # pretty-printed relaxed extended JSON (same value semantics as the dumped
+      # documents). `verbosity` is one of queryPlanner / executionStats /
+      # allPlansExecution; see DEFAULT_EXPLAIN_VERBOSITY for the safety
+      # implications. A String verbosity is passed through to the driver as-is.
+      def explain(query, verbosity: nil)
+        verbosity ||= DEFAULT_EXPLAIN_VERBOSITY
+        @logger.debug("  Running explain (verbosity=#{verbosity}) on '#{query.collection}': filter=#{query.filter.inspect}")
+
+        result = db[query.collection]
+          .find(query.filter)
+          .projection(query.projection)
+          .comment(query_comment_text("collection=#{query.collection}"))
+          .explain(verbosity: verbosity)
+
+        JSON.pretty_generate(result.as_extended_json(mode: :relaxed))
       end
 
       def describe_query(query)
@@ -459,7 +522,21 @@ module Exwiw
       # constrained by: the values of the parent field the FK references
       # (`relation.references`, default the parent primary_key). nil when the
       # parent has not been executed yet.
+      #
+      # In `explain` placeholder mode (#explain_scope_with_placeholders!) there is
+      # no captured state, so return a one-element placeholder id for any dumped
+      # (non-embedded) parent — enough to make the child's foreign-key clause
+      # appear with its real field, so the plan reflects the real dump. An
+      # embedded parent is not dumped on its own and yields nil, exactly as a real
+      # run's empty @state would.
       private def parent_state_for(relation, config_by_name)
+        if @explain_placeholder
+          parent = config_by_name[relation.table_name]
+          return nil if parent.nil? || parent.embedded?
+
+          return [explain_placeholder_id]
+        end
+
         parent_fields = @state[relation.table_name]
         return nil if parent_fields.nil?
 
@@ -468,6 +545,17 @@ module Exwiw
         parent_fields[reference_field]
       end
 
+      # The placeholder id used in explain placeholder mode. Built lazily because
+      # build_query can run before any db access loads bson; falls back to the
+      # hex String if bson is genuinely unavailable (the filter shape is what
+      # matters, not the value's type).
+      private def explain_placeholder_id
+        require 'bson' unless defined?(::BSON::ObjectId)
+        @explain_placeholder_id ||= ::BSON::ObjectId.from_string(EXPLAIN_PLACEHOLDER_OID_HEX)
+      rescue LoadError
+        EXPLAIN_PLACEHOLDER_OID_HEX
+      end
+
       # A masking plan compiled once per collection config and reused for every
       # document of that collection. `masked_fields` is `[field_name,
       # template_segments]` for each field carrying a `replace_with`;

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -68,7 +68,7 @@ module Exwiw
         StreamingResult.new(client: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN: \n#{sql}")

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -90,7 +90,7 @@ module Exwiw
         StreamingResult.new(connection: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN: \n#{sql}")

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -71,7 +71,7 @@ module Exwiw
         StreamingResult.new(connection: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN QUERY PLAN: \n#{sql}")

data/lib/exwiw/adapter.rb

@@ -182,11 +182,23 @@ module Exwiw
 
       # Run the database-specific EXPLAIN for the given query and return the
       # output as a single string for `explain` subcommand to print.
-      # SQL adapters override; MongodbAdapter currently raises.
-      def explain(_query_ast)
+      # `verbosity` selects the explain detail level; it is mongodb-only (the
+      # MongoDB explain command's queryPlanner / executionStats /
+      # allPlansExecution) and the SQL adapters accept it only to keep the
+      # contract uniform — they ignore it. SQL adapters override; MongodbAdapter
+      # implements verbosity.
+      def explain(_query_ast, verbosity: nil)
         raise NotImplementedError, "#{self.class.name} does not implement #explain"
       end
 
+      # Hook letting `explain` tell an adapter whose scope is resolved at runtime
+      # (mongodb captures parent ids during execution) to build placeholder scope
+      # filters instead, since explain executes nothing. No-op by default: the
+      # SQL adapters embed scoping in the query itself, so their `explain` already
+      # reflects the real query without any captured state.
+      def explain_scope_with_placeholders!
+      end
+
       # Identifier text prepended to every query exwiw sends to the (often
       # production) source DB, so the statement is recognizable in the
       # processlist / slow-query log / db.currentOp() and can be killed if it

data/lib/exwiw/cli.rb

@@ -35,8 +35,16 @@ module Exwiw
       ids
       ids_field
       scope_column
+      parallel_workers
+      explain_verbosity
     ].freeze
 
+    # MongoDB explain verbosity levels (passed through to the server's explain
+    # command). `queryPlanner` only plans the query and is the safe default —
+    # the query is not executed; the other two run it to gather runtime stats.
+    EXPLAIN_VERBOSITIES = %w[queryPlanner executionStats allPlansExecution].freeze
+    DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
+
     # Database connection settings are environment-specific (and sometimes
     # secret-adjacent), so they must be passed via CLI/env, never the committed
     # config file. `adapter` is the one connection-ish key allowed in config.
@@ -44,7 +52,7 @@ module Exwiw
 
     # Keys that only make sense for `export`. They are skipped when merging config
     # for `explain` so a shared config file does not trip validate_explain_only!.
-    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook].freeze
+    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook parallel_workers].freeze
 
     def self.start(argv)
       new(argv).run
@@ -80,6 +88,8 @@ module Exwiw
       @output_format = nil
       @insert_only = nil
       @after_insert_hook_path = nil
+      @parallel_workers = nil
+      @explain_verbosity = nil
       # nil (not :info) so we can tell "user passed --log-level" from the default,
       # letting a config-file value fill in; the :info default is applied later.
       @log_level = nil
@@ -125,6 +135,7 @@ module Exwiw
           output_format: @output_format,
           insert_only: @insert_only,
           after_insert_hook_path: @after_insert_hook_path,
+          parallel_workers: @parallel_workers,
           cli_options: build_cli_options_hash,
           logger: logger,
         ).run
@@ -135,6 +146,7 @@ module Exwiw
           dump_target: dump_target,
           logger: logger,
           io: $stdout,
+          explain_verbosity: @explain_verbosity,
         ).run
       end
     end
@@ -165,9 +177,11 @@ module Exwiw
       resolve_scope_column!
       resolve_ids_field!
       resolve_uri_option!
+      resolve_parallel_workers!
 
       if @subcommand == "explain"
         validate_explain_only!
+        resolve_explain_verbosity!
       end
 
       if @database_adapter != "sqlite"
@@ -316,6 +330,8 @@ module Exwiw
       end
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
+      @parallel_workers ||= parse_parallel_workers(config["parallel_workers"]) if config.key?("parallel_workers")
+      @explain_verbosity ||= config["explain_verbosity"]
     end
 
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
@@ -409,17 +425,45 @@ module Exwiw
       end
     end
 
-    private def validate_explain_only!
-      if @database_adapter == "mongodb"
-        $stderr.puts "mongodb adapter is not yet supported by 'explain' subcommand"
+    # `--parallel-workers` opts into the MongoDB fork-parallel dump schedule
+    # (docs/mongodb-dump-parallelism-2x-notes.md). It is mongodb-only (the SQL
+    # adapters shell out to their own dumpers) and must be a positive integer;
+    # N<2 is accepted but runs serially. Runs after the adapter name is normalized
+    # so the family check is reliable. `explain` rejection is handled separately
+    # by validate_explain_only!.
+    private def resolve_parallel_workers!
+      return if @parallel_workers.nil?
+
+      if @database_adapter != "mongodb"
+        $stderr.puts "--parallel-workers is only supported by the mongodb adapter"
+        exit 1
+      end
+
+      if @parallel_workers < 1
+        $stderr.puts "--parallel-workers must be a positive integer (got #{@parallel_workers})"
         exit 1
       end
+    end
+
+    # Coerce a config-file `parallel_workers` (YAML scalar) to Integer, matching
+    # the CLI flag's Integer coercion. A non-integer value is a config typo, so
+    # fail fast rather than silently dropping it.
+    private def parse_parallel_workers(value)
+      return nil if value.nil?
 
+      Integer(value)
+    rescue ArgumentError, TypeError
+      $stderr.puts "config 'parallel_workers' must be an integer (got #{value.inspect})"
+      exit 1
+    end
+
+    private def validate_explain_only!
       rejected = []
       rejected << "--output-dir" unless @output_dir.nil?
       rejected << "--output-format" unless @output_format.nil?
       rejected << "--insert-only" unless @insert_only.nil?
       rejected << "--after-insert-hook" unless @after_insert_hook_path.nil?
+      rejected << "--parallel-workers" unless @parallel_workers.nil?
 
       unless rejected.empty?
         $stderr.puts "The following options are not applicable in 'explain' subcommand: #{rejected.join(', ')}"
@@ -427,6 +471,24 @@ module Exwiw
       end
     end
 
+    # Resolve the MongoDB explain verbosity for an `explain` run. It is
+    # mongodb-only, so this no-ops (and skips validation) for the SQL adapters,
+    # which ignore verbosity. The env var wins over the config-file value (it is
+    # the more run-specific override, like a CLI flag would be); when neither is
+    # set the safe, execution-free `queryPlanner` default is used.
+    private def resolve_explain_verbosity!
+      return unless @database_adapter == "mongodb"
+
+      env = ENV["EXWIW_MONGODB_EXPLAIN_VERBOSITY"]
+      @explain_verbosity = env unless env.nil? || env.empty?
+      @explain_verbosity ||= DEFAULT_EXPLAIN_VERBOSITY
+
+      unless EXPLAIN_VERBOSITIES.include?(@explain_verbosity)
+        $stderr.puts "Invalid explain verbosity '#{@explain_verbosity}'. Available options are: #{EXPLAIN_VERBOSITIES.join(', ')}"
+        exit 1
+      end
+    end
+
     # The export clears @output_dir before writing (see Runner#clean_output_dir!).
     # That is destructive, so when running interactively (stdin is a tty) ask for
     # confirmation first. In non-interactive contexts (CI, pipes) we proceed
@@ -495,7 +557,9 @@ module Exwiw
           Subcommands:
             export    Generate INSERT/COPY SQL files (default when omitted).
             explain   Print EXPLAIN output for each extraction query to stdout.
-                      (not yet supported for the mongodb adapter)
+                      For mongodb, set verbosity via EXWIW_MONGODB_EXPLAIN_VERBOSITY
+                      or `explain_verbosity:` in config (queryPlanner (default,
+                      no query is executed) | executionStats | allPlansExecution).
         BANNER
         opts.version = Exwiw::VERSION
 
@@ -526,6 +590,7 @@ module Exwiw
         opts.on("--after-insert-hook=PATH", "Path to a .rb or .sh post-processing hook executed after all insert/delete files are written (export subcommand only)") do |v|
           @after_insert_hook_path = File.expand_path(v)
         end
+        opts.on("--parallel-workers=N", Integer, "Fork N workers for the MongoDB dump's parallel schedule (mongodb + export only; N>=2 enables it, default is serial). Output is byte-identical to serial; falls back to serial where fork is unavailable.") { |v| @parallel_workers = v }
         opts.on("--log-level=LEVEL", "Log level (debug, info). default is info") { |v| @log_level = v.to_sym }
 
         opts.on("--help", "Print this help") do

data/lib/exwiw/explain_runner.rb

@@ -7,17 +7,24 @@ module Exwiw
       schema_dir:,
       dump_target:,
       logger:,
-      io: $stdout
+      io: $stdout,
+      explain_verbosity: nil
     )
       @connection_config = connection_config
       @schema_dir = schema_dir
       @dump_target = dump_target
       @logger = logger
       @io = io
+      # mongodb-only; nil lets the adapter pick its safe default (queryPlanner).
+      @explain_verbosity = explain_verbosity
     end
 
     def run
       adapter = Adapter.build(@connection_config, @logger)
+      # explain executes nothing, so an adapter whose non-target scope is captured
+      # at runtime (mongodb) has no parent ids to filter children by; ask it to
+      # build placeholder scope filters so each query reflects the real dump.
+      adapter.explain_scope_with_placeholders!
       configs = load_table_config(adapter.class.table_config_class)
       validate_ignored(configs)
 
@@ -44,11 +51,13 @@ module Exwiw
         @logger.debug("Explaining '#{table_name}'... (#{idx + 1}/#{total_size})")
 
         query_ast = adapter.build_query(table, @dump_target, table_by_name)
-        sql = adapter.commented_sql(query_ast)
-        explain_text = adapter.explain(query_ast)
+        # describe_query is the adapter-agnostic "what query is this": the
+        # commented SELECT for SQL adapters, the find description for mongodb.
+        query_text = adapter.describe_query(query_ast)
+        explain_text = adapter.explain(query_ast, verbosity: @explain_verbosity)
 
         @io.puts "-- [#{idx + 1}/#{total_size}] #{table_name}"
-        @io.puts sql
+        @io.puts query_text
         @io.puts
         @io.puts "-- EXPLAIN:"
         @io.puts explain_text

data/lib/exwiw/mongodb_parallel_dumper.rb

@@ -0,0 +1,333 @@
+# frozen_string_literal: true
+
+require "set"
+require "fileutils"
+require "tmpdir"
+
+module Exwiw
+  # Runs the inter-collection fork schedule from
+  # docs/mongodb-dump-parallelism-2x-notes.md, producing output **byte-identical**
+  # to the serial Runner while parallelizing the dominant cost (the Mongo driver's
+  # BSON->Ruby decode) across processes — each worker decodes its own collections
+  # in their natural order, so order is preserved and the result still matches a
+  # serial dump.
+  #
+  # It consumes the static, config-derived classification from MongodbParallelPlan
+  # (the three groups + cascade adjacency + ref_bt components) and adds the live
+  # orchestration the plan deliberately leaves out: a fork pool per group, LPT
+  # bin-packing on a per-collection cost weight, @state Marshal sidecar IPC for the
+  # handful of referenced leaves, and the Phase-2 cascade reprocess.
+  #
+  # The schedule (one parent process + a pool of `workers` forks):
+  #
+  #   Phase 1 (concurrent): fork the leaf pool; the parent meanwhile dumps the
+  #     schema and processes the WHOLE genuine DAG optimistically (no leaf @state
+  #     yet), recording each genuine collection's row count.
+  #   Barrier: wait for the leaf pool; load the Marshal sidecars the consumed
+  #     leaves wrote into the parent's @state.
+  #   Phase 2 (cascade): reprocess only the genuine collections whose output can
+  #     change now that leaf @state is present (the direct-leaf referencers),
+  #     cascading to genuine children of any whose row count actually changed.
+  #   Phase 3: fork the ref_bt collections as dependency-closed components, each
+  #     worker owning whole components (processed in topological order) seeded with
+  #     the leaf @state its members reference.
+  #
+  # Output bytes are independent of the schedule: every collection writes its own
+  # insert-NNN-<name>.<ext> file (the index taken over the plan's full ordering,
+  # exactly as the serial Runner numbers them) and the per-collection write is the
+  # same build_query -> execute -> write_inserts pass the Runner performs. The
+  # bin-packing only decides which worker runs which collection, never the bytes.
+  #
+  # fork is required; callers must check {.available?} and fall back to the serial
+  # Runner on JRuby/TruffleRuby/Windows.
+  class MongodbParallelDumper
+    # Filename prefix for the per-worker record-count sidecars. Distinct from the
+    # consumed-leaf @state sidecars (`<name>.marshal`) so the two never collide in
+    # the shared sidecar dir and the counts can be globbed back independently.
+    COUNTS_SIDECAR_PREFIX = "__exwiw_counts__"
+    private_constant :COUNTS_SIDECAR_PREFIX
+
+    # True when the runtime can `fork` (CRuby on a POSIX OS). On JRuby/TruffleRuby
+    # and Windows it cannot — the caller must run the serial Runner instead.
+    def self.available?
+      Process.respond_to?(:fork)
+    end
+
+    # Longest-Processing-Time bin-packing: assign `items` to `bins` bins, heaviest
+    # first onto the currently least-loaded bin. Returns an Array of `bins` arrays
+    # (some may be empty when items < bins). `weight` is called exactly once per
+    # item (it may be DB-backed, so it must not be invoked repeatedly). Pure — no
+    # DB, no IO — so it is unit-tested directly.
+    def self.bin_pack(items, bins, &weight)
+      raise ArgumentError, "bins must be >= 1 (got #{bins})" if bins < 1
+
+      weighted = items.map { |item| [item, weight.call(item)] }.sort_by { |(_, w)| -w }
+      groups = Array.new(bins) { [] }
+      loads = Array.new(bins, 0)
+      weighted.each do |(item, w)|
+        i = (0...bins).min_by { |j| loads[j] }
+        groups[i] << item
+        loads[i] += w
+      end
+      groups
+    end
+
+    # @param connection_config [ConnectionConfig] used to build a FRESH adapter in
+    #   the parent and in every fork (a Mongo client cannot be shared across fork)
+    # @param plan [MongodbParallelPlan] the static classification for this dump
+    # @param dump_target [DumpTarget]
+    # @param table_by_name [Hash{String=>config}] ALL configs (embedded included),
+    #   exactly as Runner builds it
+    # @param output_dir [String]
+    # @param workers [Integer] fork pool size (>= 1)
+    # @param logger [Logger]
+    # @param weight_for [#call, nil] optional name -> numeric cost weight for LPT;
+    #   defaults to the adapter's metadata-only estimated document count
+    def initialize(connection_config:, plan:, dump_target:, table_by_name:, output_dir:, workers:, logger:, weight_for: nil)
+      raise ArgumentError, "workers must be >= 1 (got #{workers})" if workers < 1
+
+      @connection_config = connection_config
+      @plan = plan
+      @dump_target = dump_target
+      @table_by_name = table_by_name
+      @output_dir = output_dir
+      @workers = workers
+      @logger = logger
+      @weight_for = weight_for
+    end
+
+    # Execute the full schedule. Assumes the caller has already cleaned the output
+    # directory (the Runner does this before handing off), mirroring the serial
+    # path which dumps the schema into a freshly-cleaned dir. Returns a small stats
+    # Hash. Raises if any worker pool reports a non-zero exit.
+    def run
+      raise "fork is unavailable on this runtime; run the serial Runner instead" unless self.class.available?
+
+      FileUtils.mkdir_p(@output_dir)
+      parent = build_adapter
+      @counts = {}
+
+      Dir.mktmpdir("exwiw-mongo-parallel-") do |sidecar_dir|
+        phase1_leaf_and_genuine(parent, sidecar_dir)
+        phase2_cascade(parent, sidecar_dir)
+        phase3_ref_components(parent, sidecar_dir)
+        collect_counts(sidecar_dir)
+      end
+
+      log_count_summary
+
+      {
+        workers: @workers,
+        genuine: @plan.genuine.size,
+        leaves: @plan.leaves.size,
+        ref_bt: @plan.ref_bt.size,
+        components: @plan.reference_components.map(&:size).sort.reverse,
+        total_records: @counts.values.sum,
+      }
+    end
+
+    private
+
+    # Phase 1: fork the leaf pool to run concurrently while the parent dumps the
+    # schema (parent-only, needs no @state) and processes the whole genuine DAG
+    # optimistically. The genuine row counts captured here seed the Phase-2 cascade.
+    def phase1_leaf_and_genuine(parent, sidecar_dir)
+      leaf_master = fork do
+        ok = run_leaf_pool(sidecar_dir)
+        exit!(ok ? 0 : 1)
+      end
+
+      schema_path = File.join(@output_dir, "insert-000-schema.#{parent.schema_output_extension}")
+      ordered_tables = @plan.ordered_all.map { |name| @table_by_name.fetch(name) }
+      @logger.info("Writing schema to #{schema_path}...")
+      parent.dump_schema(ordered_tables, schema_path)
+
+      @logger.info("Processing #{@plan.genuine.size} genuine collection(s) (parent, optimistic pass)...")
+      @row_counts = {}
+      @plan.genuine.each { |name| @row_counts[name] = process_collection(parent, name) }
+
+      Process.wait(leaf_master)
+      raise "exwiw parallel leaf pool failed (exit #{$?.exitstatus})" unless $?.exitstatus&.zero?
+    end
+
+    # Barrier + Phase 2: load the consumed-leaf @state the leaf workers handed back,
+    # then reprocess only the genuine collections whose output can change now that
+    # leaf @state is present, cascading to genuine children of any that changed.
+    def phase2_cascade(parent, sidecar_dir)
+      load_sidecars(parent, @plan.consumed_leaves, sidecar_dir)
+
+      queue = @plan.direct_leaf_genuine.dup
+      seen = Set.new
+      until queue.empty?
+        name = queue.shift
+        next if seen.include?(name)
+
+        seen << name
+        new_count = process_collection(parent, name)
+        next if new_count == @row_counts[name]
+
+        @row_counts[name] = new_count
+        @plan.genuine_children[name].each { |child| queue << child }
+      end
+      @logger.info("Cascade reprocessed #{seen.size} genuine collection(s) with leaf @state.") unless seen.empty?
+    end
+
+    # Phase 3: fork the ref_bt collections as dependency-closed weakly-connected
+    # components in a single pool (no level barriers, no cross-worker IPC). Each
+    # worker owns whole components and processes their members in topological order,
+    # seeded only with the leaf @state those members reference.
+    def phase3_ref_components(parent, sidecar_dir)
+      components = @plan.reference_components
+      return if components.empty?
+
+      leaf_state = parent.state
+      groups = self.class.bin_pack(components, @workers) { |component| component.sum { |name| weight_of(parent, name) } }
+
+      pids = groups.reject(&:empty?).map do |group|
+        members = group.flatten
+        seed = leaf_state.slice(*parents_of(members))
+        fork { run_component_worker(group, seed, sidecar_dir) }
+      end
+      ok = pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
+      raise "exwiw parallel ref_bt pool failed" unless ok
+
+      @logger.info("Processed #{@plan.ref_bt.size} ref_bt collection(s) in #{groups.reject(&:empty?).size} worker(s).")
+    end
+
+    # Fork `@workers` leaf workers (LPT-packed on cost weight so the single heaviest
+    # leaf sits alone) and wait for them. Each worker writes a Marshal sidecar for
+    # the consumed leaves it produced. Runs inside the leaf_master fork, so its own
+    # weight adapter and the worker connections never touch the parent's.
+    def run_leaf_pool(sidecar_dir)
+      return true if @plan.leaves.empty?
+
+      weight_adapter = build_adapter
+      groups = self.class.bin_pack(@plan.leaves, @workers) { |name| weight_of(weight_adapter, name) }
+
+      pids = groups.reject(&:empty?).map do |group|
+        fork { run_leaf_worker(group, sidecar_dir) }
+      end
+      pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
+    rescue StandardError => e
+      @logger.error("exwiw parallel leaf master error: #{e.class}: #{e.message}")
+      false
+    end
+
+    def run_leaf_worker(group, sidecar_dir)
+      adapter = build_adapter
+      counts = {}
+      group.each { |name| counts[name] = process_collection(adapter, name) }
+      group.each do |name|
+        next unless @plan.consumed_leaves.include?(name)
+        next unless adapter.state.key?(name)
+
+        File.binwrite(File.join(sidecar_dir, "#{name}.marshal"), Marshal.dump(adapter.state[name]))
+      end
+      write_counts_sidecar(sidecar_dir, group.first, counts)
+      exit!(0)
+    rescue StandardError => e
+      @logger.error("exwiw parallel leaf worker error (#{group.first}..): #{e.class}: #{e.message}")
+      exit!(1)
+    end
+
+    def run_component_worker(group, seed, sidecar_dir)
+      adapter = build_adapter
+      adapter.state = seed unless seed.empty?
+      # Each component is already topologically ordered (parent before child) and
+      # dependency-closed over intra-ref_bt edges, so a plain serial walk suffices.
+      counts = {}
+      group.each { |component| component.each { |name| counts[name] = process_collection(adapter, name) } }
+      write_counts_sidecar(sidecar_dir, group.first.first, counts)
+      exit!(0)
+    rescue StandardError => e
+      @logger.error("exwiw parallel ref_bt worker error (#{group.first&.first}..): #{e.class}: #{e.message}")
+      exit!(1)
+    end
+
+    # Extract one collection to its insert-NNN-<name>.<ext> file. This mirrors the
+    # serial Runner's non-COPY insert path exactly — same filename (index taken over
+    # the plan's full ordering), same pre/post hooks (nil for MongoDB), same
+    # streaming write_inserts + trailing "\n", and the same empty-result handling
+    # (delete the just-opened file) — so the bytes are identical regardless of which
+    # process writes them. Returns the row count.
+    def process_collection(adapter, name)
+      table = @table_by_name.fetch(name)
+      query = adapter.build_query(table, @dump_target, @table_by_name)
+      results = adapter.execute(query)
+
+      insert_idx = (@plan.index_of.fetch(name) + 1).to_s.rjust(3, "0")
+      path = File.join(@output_dir, "insert-#{insert_idx}-#{name}.#{adapter.output_extension}")
+      chunk_size = table.bulk_insert_chunk_size || adapter.default_bulk_insert_chunk_size
+
+      record_num = 0
+      File.open(path, "w") do |file|
+        pre = adapter.pre_insert_sql(table)
+        file.puts(pre) if pre
+        _statement_count, record_num = adapter.write_inserts(file, results, table, chunk_size)
+        file.print("\n")
+        post = adapter.post_insert_sql(table)
+        file.puts(post) if post
+      end
+      File.delete(path) if record_num.zero?
+      record_num
+    end
+
+    # Merge the Marshal sidecars the leaf workers wrote (one per consumed leaf that
+    # actually produced rows) into `adapter`'s @state, so the cascade reprocess and
+    # the ref_bt workers can constrain on those leaf ids.
+    def load_sidecars(adapter, names, sidecar_dir)
+      state = adapter.state
+      names.each do |name|
+        path = File.join(sidecar_dir, "#{name}.marshal")
+        state[name] = Marshal.load(File.binread(path)) if File.exist?(path)
+      end
+    end
+
+    # Gather every collection's extracted record count back into the parent for the
+    # completion summary. genuine counts are already in @row_counts (parent-local,
+    # post-cascade); the leaf and ref_bt collections were extracted in forked
+    # workers, so each worker wrote a small Marshal counts sidecar (the same IPC
+    # pattern the consumed-leaf @state uses) that we merge here before the tmpdir is
+    # removed.
+    def collect_counts(sidecar_dir)
+      @counts.merge!(@row_counts)
+      Dir.glob(File.join(sidecar_dir, "#{COUNTS_SIDECAR_PREFIX}*.marshal")).each do |path|
+        @counts.merge!(Marshal.load(File.binread(path)))
+      end
+    end
+
+    def write_counts_sidecar(sidecar_dir, tag, counts)
+      File.binwrite(File.join(sidecar_dir, "#{COUNTS_SIDECAR_PREFIX}#{tag}.marshal"), Marshal.dump(counts))
+    end
+
+    # Log one line per extractable collection in the dump's processing order (the
+    # same order the insert-NNN- files are numbered), so the counts cross-reference
+    # directly with the output files. A collection that matched no rows shows 0
+    # (its output file was deleted).
+    def log_count_summary
+      total = @counts.values.sum
+      nonzero = @counts.count { |_, count| count.positive? }
+      @logger.info("Per-collection extracted record counts (#{total} record(s) across #{nonzero} collection(s)):")
+      @plan.extractable.each { |name| @logger.info("  #{name}: #{@counts.fetch(name, 0)}") }
+    end
+
+    # The distinct belongs_to parent names of `names`, used to slice the leaf @state
+    # a worker is seeded with down to only the keys its collections reference.
+    def parents_of(names)
+      names.flat_map { |name| @table_by_name.fetch(name).belongs_tos.map(&:table_name) }.uniq
+    end
+
+    def weight_of(adapter, name)
+      return @weight_for.call(name) if @weight_for
+
+      adapter.estimated_count(name)
+    end
+
+    # A fresh adapter (and thus a fresh, lazily-opened Mongo connection). Built per
+    # process — the parent and every fork get their own; a Mongo client must never
+    # be shared across a fork boundary.
+    def build_adapter
+      Adapter.build(@connection_config, @logger)
+    end
+  end
+end

data/lib/exwiw/mongodb_parallel_plan.rb

@@ -0,0 +1,271 @@
+# frozen_string_literal: true
+
+require "set"
+
+module Exwiw
+  # Classifies a MongoDB dump's collections into the three dependency groups the
+  # inter-collection fork schedule needs, plus the derived adjacency that
+  # schedule consumes. See docs/mongodb-dump-parallelism-2x-notes.md for the why;
+  # this class is the static, config-derived half of that plan.
+  #
+  # It is a pure function of the loaded configs and the dump target — no DB
+  # access — so it can be computed once up front and unit-tested without a live
+  # MongoDB. The fork orchestration (worker pools, LPT bin-packing on output-size
+  # weights, @state Marshal sidecars, the Phase-2 cascade loop) lives elsewhere
+  # and consumes the structures produced here.
+  #
+  # Input contract: `configs` are MongodbCollectionConfig already passed through
+  # `#reject_ignored_members!` (exactly as Runner#load_table_config produces
+  # them), so every surviving belongs_to has a non-nil `table_name`. ignore:true
+  # *collections* are still present in `configs` — they contribute to the schema
+  # and to the file-index ordering, but their data extraction is skipped — and
+  # are therefore excluded from the three processing groups.
+  #
+  # The three groups partition the extractable collections exactly:
+  #
+  # - **genuine**  — reachable to the dump target by following belongs_to edges
+  #                  (the scoped DAG). Includes the target itself.
+  # - **leaf**     — no belongs_to at all: reference/master data dumped in full,
+  #                  with no input dependencies (embarrassingly parallel).
+  # - **ref_bt**   — has belongs_to but is NOT reachable to the target: reference
+  #                  data scoped by the adapter's strict-AND fallback. Its
+  #                  internal edges form shallow components.
+  #
+  # `reachable` mirrors MongodbAdapter#genuine_scope_set exactly (fixpoint over
+  # all non-embedded configs, including ignore:true ones), so the genuine set
+  # here matches the adapter's runtime scoping classification.
+  class MongodbParallelPlan
+    EMPTY_NAMES = [].freeze
+    private_constant :EMPTY_NAMES
+
+    # @param configs [Array<MongodbCollectionConfig>] reject_ignored_members!'d
+    # @param target_table_name [String] the dump target collection
+    # @param logger [Logger, nil] forwarded to DetermineTableProcessingOrder
+    def initialize(configs:, target_table_name:, logger: nil)
+      @by = configs.each_with_object({}) { |c, h| h[c.name] = c }
+      @target_table_name = target_table_name
+
+      dumpable = configs.reject(&:embedded?)
+      # The file index (insert-NNN-) is taken over the FULL processing order,
+      # including ignore:true collections, so the orchestrated run's filenames
+      # are byte-identical to the serial Runner's (which numbers files the same
+      # way). Data extraction, however, skips ignore:true — see #extractable.
+      @ordered_all = DetermineTableProcessingOrder.run(dumpable, logger: logger).freeze
+      @index_of = @ordered_all.each_with_index.to_h.freeze
+      @extractable = @ordered_all.reject { |n| @by[n].ignore }.freeze
+
+      @reachable = compute_reachable
+      classify
+      derive_consumed_leaves
+      derive_cascade_adjacency
+      @reference_components = compute_reference_components.freeze
+    end
+
+    # Full processing order, INCLUDING ignore:true collections — the sequence the
+    # file index (insert-NNN-) is numbered over.
+    attr_reader :ordered_all
+
+    # name => 0-based position in #ordered_all (the file index is position + 1).
+    attr_reader :index_of
+
+    # #ordered_all minus ignore:true collections — the collections whose data is
+    # actually extracted. Union of the three groups below.
+    attr_reader :extractable
+
+    # The three groups (each a subset of #extractable, in #ordered_all order):
+
+    # genuine — reachable to the dump target (includes the target).
+    attr_reader :genuine
+
+    # leaf — no belongs_to; reference/master data with no input dependencies.
+    attr_reader :leaves
+
+    # ref_bt — has belongs_to but not reachable to the target.
+    attr_reader :ref_bt
+
+    # ref_bt collections as dependency-closed weakly-connected components over
+    # intra-ref_bt belongs_to edges, each returned in a valid topological order
+    # (a parent before its child). A whole component can be processed serially by
+    # one worker with no cross-worker @state IPC and no level barriers, seeded
+    # only with the leaf @state its members reference.
+    attr_reader :reference_components
+
+    # Leaf collections referenced (via belongs_to) by some non-leaf extractable
+    # collection (genuine OR ref_bt). These are the only leaves whose captured
+    # @state a downstream collection can need, so they are the ones a leaf worker
+    # must hand back (e.g. as a Marshal sidecar). Set<String>.
+    attr_reader :consumed_leaves
+
+    # genuine collections that directly reference a leaf — the only genuine
+    # collections whose output can change once leaf @state is present (and only
+    # at runtime, when their genuine anchor turns out empty and they fall back to
+    # the leaf clause). These seed the Phase-2 cascade reprocess.
+    attr_reader :direct_leaf_genuine
+
+    # name => genuine children (genuine collections that belongs_to it), keyed
+    # only by reachable parents. Drives the Phase-2 cascade: when a reprocessed
+    # collection's row count changes, its genuine children are re-enqueued.
+    attr_reader :genuine_children
+
+    # The set of collection names genuinely scoped by the target (the target plus
+    # everything that can reach it through belongs_to). Exposed for inspection.
+    attr_reader :reachable
+
+    def summary
+      {
+        extractable: @extractable.size,
+        genuine: @genuine.size,
+        leaves: @leaves.size,
+        ref_bt: @ref_bt.size,
+        consumed_leaves: @consumed_leaves.size,
+        direct_leaf_genuine: @direct_leaf_genuine.size,
+        reference_components: @reference_components.map(&:size).sort.reverse,
+      }
+    end
+
+    private
+
+    # Fixpoint over non-embedded configs: the target, plus every collection that
+    # can reach it by following belongs_to (child -> parent) transitively.
+    # Mirrors MongodbAdapter#genuine_scope_set (same traversal, same inclusion of
+    # ignore:true collections) so the genuine set matches the adapter's runtime
+    # scoping decision.
+    def compute_reachable
+      reachable = Set.new([@target_table_name])
+      loop do
+        added = false
+        @by.each_value do |cfg|
+          next if cfg.embedded? || reachable.include?(cfg.name)
+          next unless cfg.belongs_tos.any? { |rel| reachable.include?(rel.table_name) }
+
+          reachable << cfg.name
+          added = true
+        end
+        break unless added
+      end
+      reachable
+    end
+
+    def classify
+      # The three groups partition #extractable: reachable -> genuine; otherwise
+      # leaf (no belongs_to) -> leaves; otherwise -> ref_bt. The target is
+      # reachable (it seeds the set), so it lands in genuine and is never
+      # mis-grouped as a leaf even when it has no belongs_to of its own — which
+      # would otherwise double-process it (leaf pool AND parent).
+      @genuine = []
+      @leaves = []
+      @ref_bt = []
+      @extractable.each do |name|
+        if @reachable.include?(name)
+          @genuine << name
+        elsif leaf?(name)
+          @leaves << name
+        else
+          @ref_bt << name
+        end
+      end
+      @genuine.freeze
+      @leaves.freeze
+      @ref_bt.freeze
+      # Membership against the leaf *group* (which excludes the target), not the
+      # raw structural #leaf? predicate. The target has no belongs_to and is thus
+      # structurally leaf-like, but it is genuine — processed by the parent, not a
+      # leaf worker — so a belongs_to to the target must not count as referencing
+      # a leaf (it would wrongly demand a sidecar / seed the cascade).
+      @leaf_set = @leaves.to_set
+    end
+
+    def derive_consumed_leaves
+      consumed = Set.new
+      (@genuine + @ref_bt).each do |name|
+        @by[name].belongs_tos.each do |rel|
+          consumed << rel.table_name if @leaf_set.include?(rel.table_name)
+        end
+      end
+      @consumed_leaves = consumed.freeze
+    end
+
+    def derive_cascade_adjacency
+      @direct_leaf_genuine = @genuine.select do |name|
+        @by[name].belongs_tos.any? { |rel| @leaf_set.include?(rel.table_name) }
+      end.freeze
+
+      children = Hash.new { |h, k| h[k] = [] }
+      @genuine.each do |name|
+        @by[name].belongs_tos.each do |rel|
+          children[rel.table_name] << name if @reachable.include?(rel.table_name)
+        end
+      end
+      # Freeze with a non-mutating default so a lookup of a parent with no genuine
+      # children returns [] without trying to write into the frozen hash.
+      children.default_proc = nil
+      children.default = EMPTY_NAMES
+      @genuine_children = children.freeze
+    end
+
+    # ref_bt as dependency-closed weakly-connected components over intra-ref_bt
+    # belongs_to edges, each topo-ordered. Ported from the bench prototype: build
+    # the directed (child indegree) and undirected (component) views of the
+    # intra-ref_bt edges, find weakly-connected components, then Kahn-order each.
+    def compute_reference_components
+      ref_set = @ref_bt.to_set
+      children = Hash.new { |h, k| h[k] = [] }
+      adjacency = Hash.new { |h, k| h[k] = [] }
+      @ref_bt.each do |name|
+        @by[name].belongs_tos.each do |rel|
+          next unless ref_set.include?(rel.table_name)
+
+          children[rel.table_name] << name
+          adjacency[rel.table_name] << name
+          adjacency[name] << rel.table_name
+        end
+      end
+
+      seen = Set.new
+      components = []
+      @ref_bt.each do |start|
+        next if seen.include?(start)
+
+        stack = [start]
+        members = []
+        until stack.empty?
+          node = stack.pop
+          next if seen.include?(node)
+
+          seen << node
+          members << node
+          adjacency[node].each { |neighbor| stack << neighbor unless seen.include?(neighbor) }
+        end
+        components << members
+      end
+
+      components.map { |members| topo_order(members, children) }
+    end
+
+    # Kahn topological order of `members` over intra-component belongs_to edges
+    # (parent before child). `children` is the directed intra-ref_bt adjacency.
+    def topo_order(members, children)
+      member_set = members.to_set
+      indegree = members.to_h do |name|
+        [name, @by[name].belongs_tos.count { |rel| member_set.include?(rel.table_name) }]
+      end
+      queue = members.select { |name| indegree[name].zero? }
+      ordered = []
+      until queue.empty?
+        node = queue.shift
+        ordered << node
+        children[node].each do |child|
+          next unless member_set.include?(child)
+
+          indegree[child] -= 1
+          queue << child if indegree[child].zero?
+        end
+      end
+      ordered
+    end
+
+    def leaf?(name)
+      (cfg = @by[name]) && !cfg.embedded? && cfg.belongs_tos.empty?
+    end
+  end
+end

data/lib/exwiw/runner.rb

@@ -13,6 +13,7 @@ module Exwiw
       output_format: 'insert',
       insert_only: false,
       after_insert_hook_path: nil,
+      parallel_workers: nil,
       cli_options: {}
     )
       @connection_config = connection_config
@@ -22,6 +23,7 @@ module Exwiw
       @output_format = output_format
       @insert_only = insert_only
       @after_insert_hook_path = after_insert_hook_path
+      @parallel_workers = parallel_workers
       @cli_options = cli_options
       @logger = logger
     end
@@ -49,6 +51,19 @@ module Exwiw
 
       clean_output_dir!
 
+      # Opt-in MongoDB inter-collection fork parallelism (see
+      # docs/mongodb-dump-parallelism-2x-notes.md). It is byte-identical to the
+      # serial loop below — same filenames (the file index is taken over the same
+      # full processing order) and same per-collection bytes — so it is a drop-in
+      # replacement for the whole schema+inserts pass, after which the common
+      # after-insert hook still runs. Everything before this point (validation,
+      # scope check, ordering, output-dir clean) applies to both paths.
+      if use_mongodb_parallel?(adapter)
+        dump_mongodb_parallel(configs, table_by_name)
+        run_after_insert_hook(adapter, ordered_table_names.size)
+        return
+      end
+
       ordered_tables = ordered_table_names.map { |n| table_by_name.fetch(n) }
       schema_path = File.join(@output_dir, "insert-000-schema.#{adapter.schema_output_extension}")
       @logger.info("Writing schema to #{schema_path}...")
@@ -161,17 +176,71 @@ module Exwiw
         end
       end
 
-      if @after_insert_hook_path
-        @logger.info("Running after-insert hook: #{@after_insert_hook_path}")
-        AfterInsertHook.run(
-          path: @after_insert_hook_path,
-          cli_options: @cli_options,
-          output_dir: @output_dir,
-          next_idx: total_size + 1,
-          output_extension: adapter.output_extension,
-          logger: @logger,
-        )
+      run_after_insert_hook(adapter, total_size)
+    end
+
+    # Run the post-processing hook (no-op when none configured). `total_size` is
+    # the count of processed tables/collections; the hook's first output file is
+    # numbered just past them. Shared by the serial and parallel dump paths.
+    private def run_after_insert_hook(adapter, total_size)
+      return unless @after_insert_hook_path
+
+      @logger.info("Running after-insert hook: #{@after_insert_hook_path}")
+      AfterInsertHook.run(
+        path: @after_insert_hook_path,
+        cli_options: @cli_options,
+        output_dir: @output_dir,
+        next_idx: total_size + 1,
+        output_extension: adapter.output_extension,
+        logger: @logger,
+      )
+    end
+
+    # True when the opt-in MongoDB fork-parallel dump should run instead of the
+    # serial loop: the mongodb adapter, a worker count > 1, a genuine-anchor dump
+    # target (the schedule is built around the scoped DAG), and a runtime that can
+    # fork. Anything else falls back to the serial path (warning when the user
+    # explicitly asked for parallelism but it cannot apply).
+    private def use_mongodb_parallel?(adapter)
+      return false unless adapter.is_a?(Adapter::MongodbAdapter)
+      return false unless @parallel_workers && @parallel_workers > 1
+
+      if @dump_target.table_name.nil?
+        @logger.warn("--parallel-workers ignored: MongoDB parallelism needs a --target-collection; running serially.")
+        return false
       end
+
+      unless MongodbParallelDumper.available?
+        @logger.warn("--parallel-workers ignored: fork is unavailable on this runtime; running serially.")
+        return false
+      end
+
+      true
+    end
+
+    # Build the static plan and hand the whole schema+inserts pass to the fork
+    # orchestrator. `configs` are the reject_ignored_members!'d configs (the plan
+    # rejects embedded and orders them itself, identically to the serial path).
+    private def dump_mongodb_parallel(configs, table_by_name)
+      plan = MongodbParallelPlan.new(
+        configs: configs,
+        target_table_name: @dump_target.table_name,
+        logger: @logger,
+      )
+      @logger.info(
+        "MongoDB parallel dump with #{@parallel_workers} worker(s): " \
+        "genuine=#{plan.genuine.size}, leaves=#{plan.leaves.size}, ref_bt=#{plan.ref_bt.size}."
+      )
+      stats = MongodbParallelDumper.new(
+        connection_config: @connection_config,
+        plan: plan,
+        dump_target: @dump_target,
+        table_by_name: table_by_name,
+        output_dir: @output_dir,
+        workers: @parallel_workers,
+        logger: @logger,
+      ).run
+      @logger.info("MongoDB parallel dump complete: #{stats.inspect}")
     end
 
     # Empty the output dir before writing so each export starts from a clean

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.4"
+  VERSION = "0.9.0"
 end

data/lib/exwiw.rb

@@ -23,6 +23,8 @@ require_relative "exwiw/adapter/mysql_adapter"
 require_relative "exwiw/adapter/postgresql_adapter"
 require_relative "exwiw/adapter/mongodb_adapter"
 require_relative "exwiw/determine_table_processing_order"
+require_relative "exwiw/mongodb_parallel_plan"
+require_relative "exwiw/mongodb_parallel_dumper"
 require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.4
+  version: 0.9.0
 platform: ruby
 authors:
 - Shia
@@ -72,6 +72,8 @@ files:
 - lib/exwiw/mongo_query.rb
 - lib/exwiw/mongodb_collection_config.rb
 - lib/exwiw/mongodb_field.rb
+- lib/exwiw/mongodb_parallel_dumper.rb
+- lib/exwiw/mongodb_parallel_plan.rb
 - lib/exwiw/mongoid_schema_generator.rb
 - lib/exwiw/query_ast.rb
 - lib/exwiw/query_ast_builder.rb