exwiw 0.8.4 → 0.8.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ee150e9fd830a7829be8c0eebbeca19f0e2d1b1dbcce50fe78e63d62e878997c
-  data.tar.gz: 31377340d8737ece209e1c0755eeac49ac40930562503e2c4d04f6362938e9d9
+  metadata.gz: 3934deb015b3ede7c6a8caa25b857d1f7c8957b4efa3284cd26f21ff2620904d
+  data.tar.gz: e44fef95c3c273aa96b1dde37736212687da88d8d9b98abb5c44fb305d45b7a3
 SHA512:
-  metadata.gz: 26151e0bf763b6f48e993d2025b4bac804a373fe2e3cfd71bb70d91b86b453826838ec683ebfd3f4f4336bb83b8a12d567d7dbda7f8a4ef48fc7f50bbe98145a
-  data.tar.gz: b29a2da42750dde93cd53eb56a11a592478134b690ccba31b59cd312f3685dad2713e63897d09cb77246a8be37dfe55890f1a8505c1aa8b5fd020b7bc7760a03
+  metadata.gz: 1bf410fb503b270aca3f96191f34cb8ee67730ac49ea6a57d34550d2b603fed7404206d405c61f6dd3b67808f8865d73c80121d303c628dda513ebcf9c418b17
+  data.tar.gz: d1cbb8bd0ed7bd10f53d3d3169985844d4e5e509041c95a18d083dff458a06f26c09e325fea44bc5c1e14bf65b53fdc79e4dbe351f3e4e5f916e6c9135cfa71e

data/CHANGELOG.md

@@ -2,6 +2,10 @@
 
 ## [Unreleased]
 
+## [0.8.5] - 2026-06-25
+
+- **`--parallel-workers=N` parallelizes the MongoDB dump across forked processes (opt-in, byte-identical).** When set with `N≥2` on the mongodb adapter's `export`, exwiw runs an inter-collection fork schedule that decodes whole collections in parallel while preserving each collection's natural row order, so the output files are byte-identical to a serial run (same filenames, same content). Collections are classified into three dependency groups — reference data dumped in full (no `belongs_to`), the scoped DAG reachable to the dump target, and non-reachable reference data — which lets the heavy full-dump collections run concurrently with the scoped pass; only a handful of small `@state` hand-offs cross process boundaries. The win needs real cores: it clears ~2× from 4 workers and saturates there. Requires a dump target and a `fork`-capable runtime (CRuby on POSIX); it falls back to the serial path on JRuby/TruffleRuby/Windows or when no target is given. Also settable as `parallel_workers:` in the config file. The default remains the serial dump.
+
 ## [0.8.4] - 2026-06-24
 
 ### Fixed

data/README.md

@@ -769,6 +769,7 @@ The MongoDB adapter is experimental. To use it:
 - `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
 - `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only** (the SQL adapters have no equivalent).
 - Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
+- `--parallel-workers=N` (opt-in, `export` only) forks `N` worker processes that decode whole collections in parallel — the dominant cost on a large dump is the driver's BSON→Ruby decode, and each worker decodes its own collections in their natural order, so the output stays **byte-identical** to a serial run (same filenames and content). It needs a dump target (the schedule is built around the scoped DAG) and a `fork`-capable runtime (CRuby on POSIX), falling back to the serial path otherwise; it also accepts `parallel_workers:` in the config file. The speedup needs real cores to spend — it reaches ~2× from 4 workers and saturates there. The default is serial. See [`docs/mongodb-dump-parallelism-2x-notes.md`](docs/mongodb-dump-parallelism-2x-notes.md) for the schedule and measurements.
 - Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
   ```bash
   mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl

data/docs/mongodb-dump-parallelism-2x-notes.md

@@ -134,13 +134,24 @@ bounded by chunk size, and `N×` that stays well under the 7 GiB container limit
 
 ## Status
 
-This is a **measured, byte-identical proof** (bench prototype, not yet integrated
-into the Runner/CLI). Integrating it means re-introducing process orchestration
-and `@state` sidecar IPC — the machinery `optimization-notes.md` deliberately
-removed as over-engineered for a flag. It is recorded here because, unlike that
-removed work, this schedule is (a) byte-identical by construction, (b) measured
-past the 2× target on a real extraction, and (c) the lever the task explicitly
-invited (scale the task to go faster). A production version would gate it behind a
-worker-count option, fall back to serial where `fork` is unavailable
-(Windows/JRuby), and reuse the existing genuine-anchor classification to derive
-the three groups.
+**Integrated and shipped behind an opt-in flag.** The schedule lives in
+`Exwiw::MongodbParallelPlan` (the static, DB-free classification) and
+`Exwiw::MongodbParallelDumper` (the fork orchestrator: per-group pools, LPT
+bin-packing, the `@state` Marshal-sidecar IPC, and the Phase-2 cascade). The
+`Runner` delegates the whole schema+inserts pass to the dumper when the mongodb
+adapter is used with `--parallel-workers=N` (N≥2), a genuine-anchor dump target is
+present, and the runtime can `fork`; otherwise it runs the serial loop unchanged.
+The CLI exposes `--parallel-workers` / config-file `parallel_workers` (mongodb +
+`export` only). The after-insert hook runs identically on both paths.
+
+End-to-end verification through the real `exwiw export` CLI on the same staging
+restore: **189/189 output files byte-identical** to the serial CLI run (same
+filenames, 0 content mismatches), at **2.19× wall-clock** (serial 7.13 s → N=4
+3.25 s; N=2 3.99 s = 1.81×; N=6 saturates at 3.25 s). Per the curve above the win
+materializes from ~4 real cores.
+
+This was the machinery `optimization-notes.md` deliberately removed as
+over-engineered for a flag — re-introduced here because, unlike that removed work,
+this schedule is byte-identical by construction, measured past the 2× target on a
+real extraction, and the lever the task explicitly invited (scale the task to go
+faster). It is **strictly opt-in**: the default remains the serial path.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -70,6 +70,25 @@ module Exwiw
         @state = {}
       end
 
+      # Propagation @state accessor, used ONLY by MongodbParallelDumper to seed a
+      # forked worker with the slice of parent ids its collections reference and to
+      # harvest the ids downstream collections will `$in`-match against (handed
+      # between processes as Marshal sidecars). The serial Runner never touches
+      # these — it relies on the in-process capture during #execute.
+      attr_accessor :state
+
+      # Cheap, metadata-only document-count estimate for `collection_name`, used by
+      # the parallel dumper to weight collections for LPT bin-packing. This only
+      # influences which worker processes a collection (never the output bytes), so
+      # an imprecise estimate is harmless. Reads collection metadata rather than
+      # running a COLLSCAN; returns 0 on any error (e.g. a collection absent from
+      # this database just sorts to the lowest weight).
+      def estimated_count(collection_name)
+        db[collection_name].estimated_document_count
+      rescue StandardError
+        0
+      end
+
       def dumpable?(config)
         !config.embedded?
       end

data/lib/exwiw/cli.rb

@@ -35,6 +35,7 @@ module Exwiw
       ids
       ids_field
       scope_column
+      parallel_workers
     ].freeze
 
     # Database connection settings are environment-specific (and sometimes
@@ -44,7 +45,7 @@ module Exwiw
 
     # Keys that only make sense for `export`. They are skipped when merging config
     # for `explain` so a shared config file does not trip validate_explain_only!.
-    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook].freeze
+    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook parallel_workers].freeze
 
     def self.start(argv)
       new(argv).run
@@ -80,6 +81,7 @@ module Exwiw
       @output_format = nil
       @insert_only = nil
       @after_insert_hook_path = nil
+      @parallel_workers = nil
       # nil (not :info) so we can tell "user passed --log-level" from the default,
       # letting a config-file value fill in; the :info default is applied later.
       @log_level = nil
@@ -125,6 +127,7 @@ module Exwiw
           output_format: @output_format,
           insert_only: @insert_only,
           after_insert_hook_path: @after_insert_hook_path,
+          parallel_workers: @parallel_workers,
           cli_options: build_cli_options_hash,
           logger: logger,
         ).run
@@ -165,6 +168,7 @@ module Exwiw
       resolve_scope_column!
       resolve_ids_field!
       resolve_uri_option!
+      resolve_parallel_workers!
 
       if @subcommand == "explain"
         validate_explain_only!
@@ -316,6 +320,7 @@ module Exwiw
       end
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
+      @parallel_workers ||= parse_parallel_workers(config["parallel_workers"]) if config.key?("parallel_workers")
     end
 
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
@@ -409,6 +414,38 @@ module Exwiw
       end
     end
 
+    # `--parallel-workers` opts into the MongoDB fork-parallel dump schedule
+    # (docs/mongodb-dump-parallelism-2x-notes.md). It is mongodb-only (the SQL
+    # adapters shell out to their own dumpers) and must be a positive integer;
+    # N<2 is accepted but runs serially. Runs after the adapter name is normalized
+    # so the family check is reliable. `explain` rejection is handled separately
+    # by validate_explain_only!.
+    private def resolve_parallel_workers!
+      return if @parallel_workers.nil?
+
+      if @database_adapter != "mongodb"
+        $stderr.puts "--parallel-workers is only supported by the mongodb adapter"
+        exit 1
+      end
+
+      if @parallel_workers < 1
+        $stderr.puts "--parallel-workers must be a positive integer (got #{@parallel_workers})"
+        exit 1
+      end
+    end
+
+    # Coerce a config-file `parallel_workers` (YAML scalar) to Integer, matching
+    # the CLI flag's Integer coercion. A non-integer value is a config typo, so
+    # fail fast rather than silently dropping it.
+    private def parse_parallel_workers(value)
+      return nil if value.nil?
+
+      Integer(value)
+    rescue ArgumentError, TypeError
+      $stderr.puts "config 'parallel_workers' must be an integer (got #{value.inspect})"
+      exit 1
+    end
+
     private def validate_explain_only!
       if @database_adapter == "mongodb"
         $stderr.puts "mongodb adapter is not yet supported by 'explain' subcommand"
@@ -420,6 +457,7 @@ module Exwiw
       rejected << "--output-format" unless @output_format.nil?
       rejected << "--insert-only" unless @insert_only.nil?
       rejected << "--after-insert-hook" unless @after_insert_hook_path.nil?
+      rejected << "--parallel-workers" unless @parallel_workers.nil?
 
       unless rejected.empty?
         $stderr.puts "The following options are not applicable in 'explain' subcommand: #{rejected.join(', ')}"
@@ -526,6 +564,7 @@ module Exwiw
         opts.on("--after-insert-hook=PATH", "Path to a .rb or .sh post-processing hook executed after all insert/delete files are written (export subcommand only)") do |v|
           @after_insert_hook_path = File.expand_path(v)
         end
+        opts.on("--parallel-workers=N", Integer, "Fork N workers for the MongoDB dump's parallel schedule (mongodb + export only; N>=2 enables it, default is serial). Output is byte-identical to serial; falls back to serial where fork is unavailable.") { |v| @parallel_workers = v }
         opts.on("--log-level=LEVEL", "Log level (debug, info). default is info") { |v| @log_level = v.to_sym }
 
         opts.on("--help", "Print this help") do

data/lib/exwiw/mongodb_parallel_dumper.rb

@@ -0,0 +1,290 @@
+# frozen_string_literal: true
+
+require "set"
+require "fileutils"
+require "tmpdir"
+
+module Exwiw
+  # Runs the inter-collection fork schedule from
+  # docs/mongodb-dump-parallelism-2x-notes.md, producing output **byte-identical**
+  # to the serial Runner while parallelizing the dominant cost (the Mongo driver's
+  # BSON->Ruby decode) across processes — each worker decodes its own collections
+  # in their natural order, so order is preserved and the result still matches a
+  # serial dump.
+  #
+  # It consumes the static, config-derived classification from MongodbParallelPlan
+  # (the three groups + cascade adjacency + ref_bt components) and adds the live
+  # orchestration the plan deliberately leaves out: a fork pool per group, LPT
+  # bin-packing on a per-collection cost weight, @state Marshal sidecar IPC for the
+  # handful of referenced leaves, and the Phase-2 cascade reprocess.
+  #
+  # The schedule (one parent process + a pool of `workers` forks):
+  #
+  #   Phase 1 (concurrent): fork the leaf pool; the parent meanwhile dumps the
+  #     schema and processes the WHOLE genuine DAG optimistically (no leaf @state
+  #     yet), recording each genuine collection's row count.
+  #   Barrier: wait for the leaf pool; load the Marshal sidecars the consumed
+  #     leaves wrote into the parent's @state.
+  #   Phase 2 (cascade): reprocess only the genuine collections whose output can
+  #     change now that leaf @state is present (the direct-leaf referencers),
+  #     cascading to genuine children of any whose row count actually changed.
+  #   Phase 3: fork the ref_bt collections as dependency-closed components, each
+  #     worker owning whole components (processed in topological order) seeded with
+  #     the leaf @state its members reference.
+  #
+  # Output bytes are independent of the schedule: every collection writes its own
+  # insert-NNN-<name>.<ext> file (the index taken over the plan's full ordering,
+  # exactly as the serial Runner numbers them) and the per-collection write is the
+  # same build_query -> execute -> write_inserts pass the Runner performs. The
+  # bin-packing only decides which worker runs which collection, never the bytes.
+  #
+  # fork is required; callers must check {.available?} and fall back to the serial
+  # Runner on JRuby/TruffleRuby/Windows.
+  class MongodbParallelDumper
+    # True when the runtime can `fork` (CRuby on a POSIX OS). On JRuby/TruffleRuby
+    # and Windows it cannot — the caller must run the serial Runner instead.
+    def self.available?
+      Process.respond_to?(:fork)
+    end
+
+    # Longest-Processing-Time bin-packing: assign `items` to `bins` bins, heaviest
+    # first onto the currently least-loaded bin. Returns an Array of `bins` arrays
+    # (some may be empty when items < bins). `weight` is called exactly once per
+    # item (it may be DB-backed, so it must not be invoked repeatedly). Pure — no
+    # DB, no IO — so it is unit-tested directly.
+    def self.bin_pack(items, bins, &weight)
+      raise ArgumentError, "bins must be >= 1 (got #{bins})" if bins < 1
+
+      weighted = items.map { |item| [item, weight.call(item)] }.sort_by { |(_, w)| -w }
+      groups = Array.new(bins) { [] }
+      loads = Array.new(bins, 0)
+      weighted.each do |(item, w)|
+        i = (0...bins).min_by { |j| loads[j] }
+        groups[i] << item
+        loads[i] += w
+      end
+      groups
+    end
+
+    # @param connection_config [ConnectionConfig] used to build a FRESH adapter in
+    #   the parent and in every fork (a Mongo client cannot be shared across fork)
+    # @param plan [MongodbParallelPlan] the static classification for this dump
+    # @param dump_target [DumpTarget]
+    # @param table_by_name [Hash{String=>config}] ALL configs (embedded included),
+    #   exactly as Runner builds it
+    # @param output_dir [String]
+    # @param workers [Integer] fork pool size (>= 1)
+    # @param logger [Logger]
+    # @param weight_for [#call, nil] optional name -> numeric cost weight for LPT;
+    #   defaults to the adapter's metadata-only estimated document count
+    def initialize(connection_config:, plan:, dump_target:, table_by_name:, output_dir:, workers:, logger:, weight_for: nil)
+      raise ArgumentError, "workers must be >= 1 (got #{workers})" if workers < 1
+
+      @connection_config = connection_config
+      @plan = plan
+      @dump_target = dump_target
+      @table_by_name = table_by_name
+      @output_dir = output_dir
+      @workers = workers
+      @logger = logger
+      @weight_for = weight_for
+    end
+
+    # Execute the full schedule. Assumes the caller has already cleaned the output
+    # directory (the Runner does this before handing off), mirroring the serial
+    # path which dumps the schema into a freshly-cleaned dir. Returns a small stats
+    # Hash. Raises if any worker pool reports a non-zero exit.
+    def run
+      raise "fork is unavailable on this runtime; run the serial Runner instead" unless self.class.available?
+
+      FileUtils.mkdir_p(@output_dir)
+      parent = build_adapter
+
+      Dir.mktmpdir("exwiw-mongo-parallel-") do |sidecar_dir|
+        phase1_leaf_and_genuine(parent, sidecar_dir)
+        phase2_cascade(parent, sidecar_dir)
+        phase3_ref_components(parent, sidecar_dir)
+      end
+
+      {
+        workers: @workers,
+        genuine: @plan.genuine.size,
+        leaves: @plan.leaves.size,
+        ref_bt: @plan.ref_bt.size,
+        components: @plan.reference_components.map(&:size).sort.reverse,
+      }
+    end
+
+    private
+
+    # Phase 1: fork the leaf pool to run concurrently while the parent dumps the
+    # schema (parent-only, needs no @state) and processes the whole genuine DAG
+    # optimistically. The genuine row counts captured here seed the Phase-2 cascade.
+    def phase1_leaf_and_genuine(parent, sidecar_dir)
+      leaf_master = fork do
+        ok = run_leaf_pool(sidecar_dir)
+        exit!(ok ? 0 : 1)
+      end
+
+      schema_path = File.join(@output_dir, "insert-000-schema.#{parent.schema_output_extension}")
+      ordered_tables = @plan.ordered_all.map { |name| @table_by_name.fetch(name) }
+      @logger.info("Writing schema to #{schema_path}...")
+      parent.dump_schema(ordered_tables, schema_path)
+
+      @logger.info("Processing #{@plan.genuine.size} genuine collection(s) (parent, optimistic pass)...")
+      @row_counts = {}
+      @plan.genuine.each { |name| @row_counts[name] = process_collection(parent, name) }
+
+      Process.wait(leaf_master)
+      raise "exwiw parallel leaf pool failed (exit #{$?.exitstatus})" unless $?.exitstatus&.zero?
+    end
+
+    # Barrier + Phase 2: load the consumed-leaf @state the leaf workers handed back,
+    # then reprocess only the genuine collections whose output can change now that
+    # leaf @state is present, cascading to genuine children of any that changed.
+    def phase2_cascade(parent, sidecar_dir)
+      load_sidecars(parent, @plan.consumed_leaves, sidecar_dir)
+
+      queue = @plan.direct_leaf_genuine.dup
+      seen = Set.new
+      until queue.empty?
+        name = queue.shift
+        next if seen.include?(name)
+
+        seen << name
+        new_count = process_collection(parent, name)
+        next if new_count == @row_counts[name]
+
+        @row_counts[name] = new_count
+        @plan.genuine_children[name].each { |child| queue << child }
+      end
+      @logger.info("Cascade reprocessed #{seen.size} genuine collection(s) with leaf @state.") unless seen.empty?
+    end
+
+    # Phase 3: fork the ref_bt collections as dependency-closed weakly-connected
+    # components in a single pool (no level barriers, no cross-worker IPC). Each
+    # worker owns whole components and processes their members in topological order,
+    # seeded only with the leaf @state those members reference.
+    def phase3_ref_components(parent, sidecar_dir)
+      components = @plan.reference_components
+      return if components.empty?
+
+      leaf_state = parent.state
+      groups = self.class.bin_pack(components, @workers) { |component| component.sum { |name| weight_of(parent, name) } }
+
+      pids = groups.reject(&:empty?).map do |group|
+        members = group.flatten
+        seed = leaf_state.slice(*parents_of(members))
+        fork { run_component_worker(group, seed) }
+      end
+      ok = pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
+      raise "exwiw parallel ref_bt pool failed" unless ok
+
+      @logger.info("Processed #{@plan.ref_bt.size} ref_bt collection(s) in #{groups.reject(&:empty?).size} worker(s).")
+    end
+
+    # Fork `@workers` leaf workers (LPT-packed on cost weight so the single heaviest
+    # leaf sits alone) and wait for them. Each worker writes a Marshal sidecar for
+    # the consumed leaves it produced. Runs inside the leaf_master fork, so its own
+    # weight adapter and the worker connections never touch the parent's.
+    def run_leaf_pool(sidecar_dir)
+      return true if @plan.leaves.empty?
+
+      weight_adapter = build_adapter
+      groups = self.class.bin_pack(@plan.leaves, @workers) { |name| weight_of(weight_adapter, name) }
+
+      pids = groups.reject(&:empty?).map do |group|
+        fork { run_leaf_worker(group, sidecar_dir) }
+      end
+      pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
+    rescue StandardError => e
+      @logger.error("exwiw parallel leaf master error: #{e.class}: #{e.message}")
+      false
+    end
+
+    def run_leaf_worker(group, sidecar_dir)
+      adapter = build_adapter
+      group.each { |name| process_collection(adapter, name) }
+      group.each do |name|
+        next unless @plan.consumed_leaves.include?(name)
+        next unless adapter.state.key?(name)
+
+        File.binwrite(File.join(sidecar_dir, "#{name}.marshal"), Marshal.dump(adapter.state[name]))
+      end
+      exit!(0)
+    rescue StandardError => e
+      @logger.error("exwiw parallel leaf worker error (#{group.first}..): #{e.class}: #{e.message}")
+      exit!(1)
+    end
+
+    def run_component_worker(group, seed)
+      adapter = build_adapter
+      adapter.state = seed unless seed.empty?
+      # Each component is already topologically ordered (parent before child) and
+      # dependency-closed over intra-ref_bt edges, so a plain serial walk suffices.
+      group.each { |component| component.each { |name| process_collection(adapter, name) } }
+      exit!(0)
+    rescue StandardError => e
+      @logger.error("exwiw parallel ref_bt worker error (#{group.first&.first}..): #{e.class}: #{e.message}")
+      exit!(1)
+    end
+
+    # Extract one collection to its insert-NNN-<name>.<ext> file. This mirrors the
+    # serial Runner's non-COPY insert path exactly — same filename (index taken over
+    # the plan's full ordering), same pre/post hooks (nil for MongoDB), same
+    # streaming write_inserts + trailing "\n", and the same empty-result handling
+    # (delete the just-opened file) — so the bytes are identical regardless of which
+    # process writes them. Returns the row count.
+    def process_collection(adapter, name)
+      table = @table_by_name.fetch(name)
+      query = adapter.build_query(table, @dump_target, @table_by_name)
+      results = adapter.execute(query)
+
+      insert_idx = (@plan.index_of.fetch(name) + 1).to_s.rjust(3, "0")
+      path = File.join(@output_dir, "insert-#{insert_idx}-#{name}.#{adapter.output_extension}")
+      chunk_size = table.bulk_insert_chunk_size || adapter.default_bulk_insert_chunk_size
+
+      record_num = 0
+      File.open(path, "w") do |file|
+        pre = adapter.pre_insert_sql(table)
+        file.puts(pre) if pre
+        _statement_count, record_num = adapter.write_inserts(file, results, table, chunk_size)
+        file.print("\n")
+        post = adapter.post_insert_sql(table)
+        file.puts(post) if post
+      end
+      File.delete(path) if record_num.zero?
+      record_num
+    end
+
+    # Merge the Marshal sidecars the leaf workers wrote (one per consumed leaf that
+    # actually produced rows) into `adapter`'s @state, so the cascade reprocess and
+    # the ref_bt workers can constrain on those leaf ids.
+    def load_sidecars(adapter, names, sidecar_dir)
+      state = adapter.state
+      names.each do |name|
+        path = File.join(sidecar_dir, "#{name}.marshal")
+        state[name] = Marshal.load(File.binread(path)) if File.exist?(path)
+      end
+    end
+
+    # The distinct belongs_to parent names of `names`, used to slice the leaf @state
+    # a worker is seeded with down to only the keys its collections reference.
+    def parents_of(names)
+      names.flat_map { |name| @table_by_name.fetch(name).belongs_tos.map(&:table_name) }.uniq
+    end
+
+    def weight_of(adapter, name)
+      return @weight_for.call(name) if @weight_for
+
+      adapter.estimated_count(name)
+    end
+
+    # A fresh adapter (and thus a fresh, lazily-opened Mongo connection). Built per
+    # process — the parent and every fork get their own; a Mongo client must never
+    # be shared across a fork boundary.
+    def build_adapter
+      Adapter.build(@connection_config, @logger)
+    end
+  end
+end

data/lib/exwiw/mongodb_parallel_plan.rb

@@ -0,0 +1,271 @@
+# frozen_string_literal: true
+
+require "set"
+
+module Exwiw
+  # Classifies a MongoDB dump's collections into the three dependency groups the
+  # inter-collection fork schedule needs, plus the derived adjacency that
+  # schedule consumes. See docs/mongodb-dump-parallelism-2x-notes.md for the why;
+  # this class is the static, config-derived half of that plan.
+  #
+  # It is a pure function of the loaded configs and the dump target — no DB
+  # access — so it can be computed once up front and unit-tested without a live
+  # MongoDB. The fork orchestration (worker pools, LPT bin-packing on output-size
+  # weights, @state Marshal sidecars, the Phase-2 cascade loop) lives elsewhere
+  # and consumes the structures produced here.
+  #
+  # Input contract: `configs` are MongodbCollectionConfig already passed through
+  # `#reject_ignored_members!` (exactly as Runner#load_table_config produces
+  # them), so every surviving belongs_to has a non-nil `table_name`. ignore:true
+  # *collections* are still present in `configs` — they contribute to the schema
+  # and to the file-index ordering, but their data extraction is skipped — and
+  # are therefore excluded from the three processing groups.
+  #
+  # The three groups partition the extractable collections exactly:
+  #
+  # - **genuine**  — reachable to the dump target by following belongs_to edges
+  #                  (the scoped DAG). Includes the target itself.
+  # - **leaf**     — no belongs_to at all: reference/master data dumped in full,
+  #                  with no input dependencies (embarrassingly parallel).
+  # - **ref_bt**   — has belongs_to but is NOT reachable to the target: reference
+  #                  data scoped by the adapter's strict-AND fallback. Its
+  #                  internal edges form shallow components.
+  #
+  # `reachable` mirrors MongodbAdapter#genuine_scope_set exactly (fixpoint over
+  # all non-embedded configs, including ignore:true ones), so the genuine set
+  # here matches the adapter's runtime scoping classification.
+  class MongodbParallelPlan
+    EMPTY_NAMES = [].freeze
+    private_constant :EMPTY_NAMES
+
+    # @param configs [Array<MongodbCollectionConfig>] reject_ignored_members!'d
+    # @param target_table_name [String] the dump target collection
+    # @param logger [Logger, nil] forwarded to DetermineTableProcessingOrder
+    def initialize(configs:, target_table_name:, logger: nil)
+      @by = configs.each_with_object({}) { |c, h| h[c.name] = c }
+      @target_table_name = target_table_name
+
+      dumpable = configs.reject(&:embedded?)
+      # The file index (insert-NNN-) is taken over the FULL processing order,
+      # including ignore:true collections, so the orchestrated run's filenames
+      # are byte-identical to the serial Runner's (which numbers files the same
+      # way). Data extraction, however, skips ignore:true — see #extractable.
+      @ordered_all = DetermineTableProcessingOrder.run(dumpable, logger: logger).freeze
+      @index_of = @ordered_all.each_with_index.to_h.freeze
+      @extractable = @ordered_all.reject { |n| @by[n].ignore }.freeze
+
+      @reachable = compute_reachable
+      classify
+      derive_consumed_leaves
+      derive_cascade_adjacency
+      @reference_components = compute_reference_components.freeze
+    end
+
+    # Full processing order, INCLUDING ignore:true collections — the sequence the
+    # file index (insert-NNN-) is numbered over.
+    attr_reader :ordered_all
+
+    # name => 0-based position in #ordered_all (the file index is position + 1).
+    attr_reader :index_of
+
+    # #ordered_all minus ignore:true collections — the collections whose data is
+    # actually extracted. Union of the three groups below.
+    attr_reader :extractable
+
+    # The three groups (each a subset of #extractable, in #ordered_all order):
+
+    # genuine — reachable to the dump target (includes the target).
+    attr_reader :genuine
+
+    # leaf — no belongs_to; reference/master data with no input dependencies.
+    attr_reader :leaves
+
+    # ref_bt — has belongs_to but not reachable to the target.
+    attr_reader :ref_bt
+
+    # ref_bt collections as dependency-closed weakly-connected components over
+    # intra-ref_bt belongs_to edges, each returned in a valid topological order
+    # (a parent before its child). A whole component can be processed serially by
+    # one worker with no cross-worker @state IPC and no level barriers, seeded
+    # only with the leaf @state its members reference.
+    attr_reader :reference_components
+
+    # Leaf collections referenced (via belongs_to) by some non-leaf extractable
+    # collection (genuine OR ref_bt). These are the only leaves whose captured
+    # @state a downstream collection can need, so they are the ones a leaf worker
+    # must hand back (e.g. as a Marshal sidecar). Set<String>.
+    attr_reader :consumed_leaves
+
+    # genuine collections that directly reference a leaf — the only genuine
+    # collections whose output can change once leaf @state is present (and only
+    # at runtime, when their genuine anchor turns out empty and they fall back to
+    # the leaf clause). These seed the Phase-2 cascade reprocess.
+    attr_reader :direct_leaf_genuine
+
+    # name => genuine children (genuine collections that belongs_to it), keyed
+    # only by reachable parents. Drives the Phase-2 cascade: when a reprocessed
+    # collection's row count changes, its genuine children are re-enqueued.
+    attr_reader :genuine_children
+
+    # The set of collection names genuinely scoped by the target (the target plus
+    # everything that can reach it through belongs_to). Exposed for inspection.
+    attr_reader :reachable
+
+    def summary
+      {
+        extractable: @extractable.size,
+        genuine: @genuine.size,
+        leaves: @leaves.size,
+        ref_bt: @ref_bt.size,
+        consumed_leaves: @consumed_leaves.size,
+        direct_leaf_genuine: @direct_leaf_genuine.size,
+        reference_components: @reference_components.map(&:size).sort.reverse,
+      }
+    end
+
+    private
+
+    # Fixpoint over non-embedded configs: the target, plus every collection that
+    # can reach it by following belongs_to (child -> parent) transitively.
+    # Mirrors MongodbAdapter#genuine_scope_set (same traversal, same inclusion of
+    # ignore:true collections) so the genuine set matches the adapter's runtime
+    # scoping decision.
+    def compute_reachable
+      reachable = Set.new([@target_table_name])
+      loop do
+        added = false
+        @by.each_value do |cfg|
+          next if cfg.embedded? || reachable.include?(cfg.name)
+          next unless cfg.belongs_tos.any? { |rel| reachable.include?(rel.table_name) }
+
+          reachable << cfg.name
+          added = true
+        end
+        break unless added
+      end
+      reachable
+    end
+
+    def classify
+      # The three groups partition #extractable: reachable -> genuine; otherwise
+      # leaf (no belongs_to) -> leaves; otherwise -> ref_bt. The target is
+      # reachable (it seeds the set), so it lands in genuine and is never
+      # mis-grouped as a leaf even when it has no belongs_to of its own — which
+      # would otherwise double-process it (leaf pool AND parent).
+      @genuine = []
+      @leaves = []
+      @ref_bt = []
+      @extractable.each do |name|
+        if @reachable.include?(name)
+          @genuine << name
+        elsif leaf?(name)
+          @leaves << name
+        else
+          @ref_bt << name
+        end
+      end
+      @genuine.freeze
+      @leaves.freeze
+      @ref_bt.freeze
+      # Membership against the leaf *group* (which excludes the target), not the
+      # raw structural #leaf? predicate. The target has no belongs_to and is thus
+      # structurally leaf-like, but it is genuine — processed by the parent, not a
+      # leaf worker — so a belongs_to to the target must not count as referencing
+      # a leaf (it would wrongly demand a sidecar / seed the cascade).
+      @leaf_set = @leaves.to_set
+    end
+
+    def derive_consumed_leaves
+      consumed = Set.new
+      (@genuine + @ref_bt).each do |name|
+        @by[name].belongs_tos.each do |rel|
+          consumed << rel.table_name if @leaf_set.include?(rel.table_name)
+        end
+      end
+      @consumed_leaves = consumed.freeze
+    end
+
+    def derive_cascade_adjacency
+      @direct_leaf_genuine = @genuine.select do |name|
+        @by[name].belongs_tos.any? { |rel| @leaf_set.include?(rel.table_name) }
+      end.freeze
+
+      children = Hash.new { |h, k| h[k] = [] }
+      @genuine.each do |name|
+        @by[name].belongs_tos.each do |rel|
+          children[rel.table_name] << name if @reachable.include?(rel.table_name)
+        end
+      end
+      # Freeze with a non-mutating default so a lookup of a parent with no genuine
+      # children returns [] without trying to write into the frozen hash.
+      children.default_proc = nil
+      children.default = EMPTY_NAMES
+      @genuine_children = children.freeze
+    end
+
+    # ref_bt as dependency-closed weakly-connected components over intra-ref_bt
+    # belongs_to edges, each topo-ordered. Ported from the bench prototype: build
+    # the directed (child indegree) and undirected (component) views of the
+    # intra-ref_bt edges, find weakly-connected components, then Kahn-order each.
+    def compute_reference_components
+      ref_set = @ref_bt.to_set
+      children = Hash.new { |h, k| h[k] = [] }
+      adjacency = Hash.new { |h, k| h[k] = [] }
+      @ref_bt.each do |name|
+        @by[name].belongs_tos.each do |rel|
+          next unless ref_set.include?(rel.table_name)
+
+          children[rel.table_name] << name
+          adjacency[rel.table_name] << name
+          adjacency[name] << rel.table_name
+        end
+      end
+
+      seen = Set.new
+      components = []
+      @ref_bt.each do |start|
+        next if seen.include?(start)
+
+        stack = [start]
+        members = []
+        until stack.empty?
+          node = stack.pop
+          next if seen.include?(node)
+
+          seen << node
+          members << node
+          adjacency[node].each { |neighbor| stack << neighbor unless seen.include?(neighbor) }
+        end
+        components << members
+      end
+
+      components.map { |members| topo_order(members, children) }
+    end
+
+    # Kahn topological order of `members` over intra-component belongs_to edges
+    # (parent before child). `children` is the directed intra-ref_bt adjacency.
+    def topo_order(members, children)
+      member_set = members.to_set
+      indegree = members.to_h do |name|
+        [name, @by[name].belongs_tos.count { |rel| member_set.include?(rel.table_name) }]
+      end
+      queue = members.select { |name| indegree[name].zero? }
+      ordered = []
+      until queue.empty?
+        node = queue.shift
+        ordered << node
+        children[node].each do |child|
+          next unless member_set.include?(child)
+
+          indegree[child] -= 1
+          queue << child if indegree[child].zero?
+        end
+      end
+      ordered
+    end
+
+    def leaf?(name)
+      (cfg = @by[name]) && !cfg.embedded? && cfg.belongs_tos.empty?
+    end
+  end
+end

data/lib/exwiw/runner.rb

@@ -13,6 +13,7 @@ module Exwiw
       output_format: 'insert',
       insert_only: false,
       after_insert_hook_path: nil,
+      parallel_workers: nil,
       cli_options: {}
     )
       @connection_config = connection_config
@@ -22,6 +23,7 @@ module Exwiw
       @output_format = output_format
       @insert_only = insert_only
       @after_insert_hook_path = after_insert_hook_path
+      @parallel_workers = parallel_workers
       @cli_options = cli_options
       @logger = logger
     end
@@ -49,6 +51,19 @@ module Exwiw
 
       clean_output_dir!
 
+      # Opt-in MongoDB inter-collection fork parallelism (see
+      # docs/mongodb-dump-parallelism-2x-notes.md). It is byte-identical to the
+      # serial loop below — same filenames (the file index is taken over the same
+      # full processing order) and same per-collection bytes — so it is a drop-in
+      # replacement for the whole schema+inserts pass, after which the common
+      # after-insert hook still runs. Everything before this point (validation,
+      # scope check, ordering, output-dir clean) applies to both paths.
+      if use_mongodb_parallel?(adapter)
+        dump_mongodb_parallel(configs, table_by_name)
+        run_after_insert_hook(adapter, ordered_table_names.size)
+        return
+      end
+
       ordered_tables = ordered_table_names.map { |n| table_by_name.fetch(n) }
       schema_path = File.join(@output_dir, "insert-000-schema.#{adapter.schema_output_extension}")
       @logger.info("Writing schema to #{schema_path}...")
@@ -161,17 +176,71 @@ module Exwiw
         end
       end
 
-      if @after_insert_hook_path
-        @logger.info("Running after-insert hook: #{@after_insert_hook_path}")
-        AfterInsertHook.run(
-          path: @after_insert_hook_path,
-          cli_options: @cli_options,
-          output_dir: @output_dir,
-          next_idx: total_size + 1,
-          output_extension: adapter.output_extension,
-          logger: @logger,
-        )
+      run_after_insert_hook(adapter, total_size)
+    end
+
+    # Run the post-processing hook (no-op when none configured). `total_size` is
+    # the count of processed tables/collections; the hook's first output file is
+    # numbered just past them. Shared by the serial and parallel dump paths.
+    private def run_after_insert_hook(adapter, total_size)
+      return unless @after_insert_hook_path
+
+      @logger.info("Running after-insert hook: #{@after_insert_hook_path}")
+      AfterInsertHook.run(
+        path: @after_insert_hook_path,
+        cli_options: @cli_options,
+        output_dir: @output_dir,
+        next_idx: total_size + 1,
+        output_extension: adapter.output_extension,
+        logger: @logger,
+      )
+    end
+
+    # True when the opt-in MongoDB fork-parallel dump should run instead of the
+    # serial loop: the mongodb adapter, a worker count > 1, a genuine-anchor dump
+    # target (the schedule is built around the scoped DAG), and a runtime that can
+    # fork. Anything else falls back to the serial path (warning when the user
+    # explicitly asked for parallelism but it cannot apply).
+    private def use_mongodb_parallel?(adapter)
+      return false unless adapter.is_a?(Adapter::MongodbAdapter)
+      return false unless @parallel_workers && @parallel_workers > 1
+
+      if @dump_target.table_name.nil?
+        @logger.warn("--parallel-workers ignored: MongoDB parallelism needs a --target-collection; running serially.")
+        return false
       end
+
+      unless MongodbParallelDumper.available?
+        @logger.warn("--parallel-workers ignored: fork is unavailable on this runtime; running serially.")
+        return false
+      end
+
+      true
+    end
+
+    # Build the static plan and hand the whole schema+inserts pass to the fork
+    # orchestrator. `configs` are the reject_ignored_members!'d configs (the plan
+    # rejects embedded and orders them itself, identically to the serial path).
+    private def dump_mongodb_parallel(configs, table_by_name)
+      plan = MongodbParallelPlan.new(
+        configs: configs,
+        target_table_name: @dump_target.table_name,
+        logger: @logger,
+      )
+      @logger.info(
+        "MongoDB parallel dump with #{@parallel_workers} worker(s): " \
+        "genuine=#{plan.genuine.size}, leaves=#{plan.leaves.size}, ref_bt=#{plan.ref_bt.size}."
+      )
+      stats = MongodbParallelDumper.new(
+        connection_config: @connection_config,
+        plan: plan,
+        dump_target: @dump_target,
+        table_by_name: table_by_name,
+        output_dir: @output_dir,
+        workers: @parallel_workers,
+        logger: @logger,
+      ).run
+      @logger.info("MongoDB parallel dump complete: #{stats.inspect}")
     end
 
     # Empty the output dir before writing so each export starts from a clean

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.4"
+  VERSION = "0.8.5"
 end

data/lib/exwiw.rb

@@ -23,6 +23,8 @@ require_relative "exwiw/adapter/mysql_adapter"
 require_relative "exwiw/adapter/postgresql_adapter"
 require_relative "exwiw/adapter/mongodb_adapter"
 require_relative "exwiw/determine_table_processing_order"
+require_relative "exwiw/mongodb_parallel_plan"
+require_relative "exwiw/mongodb_parallel_dumper"
 require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.4
+  version: 0.8.5
 platform: ruby
 authors:
 - Shia
@@ -72,6 +72,8 @@ files:
 - lib/exwiw/mongo_query.rb
 - lib/exwiw/mongodb_collection_config.rb
 - lib/exwiw/mongodb_field.rb
+- lib/exwiw/mongodb_parallel_dumper.rb
+- lib/exwiw/mongodb_parallel_plan.rb
 - lib/exwiw/mongoid_schema_generator.rb
 - lib/exwiw/query_ast.rb
 - lib/exwiw/query_ast_builder.rb