exwiw 0.8.5 → 0.9.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3934deb015b3ede7c6a8caa25b857d1f7c8957b4efa3284cd26f21ff2620904d
-  data.tar.gz: e44fef95c3c273aa96b1dde37736212687da88d8d9b98abb5c44fb305d45b7a3
+  metadata.gz: 171e3c591c208afcb89c2a9330c2c80d67f8f86ab370858b4690d03c799d28bc
+  data.tar.gz: f913a31f6f8e2a29a9e4b2a296926c1266c17cc8655fd4cf33110899f2b468fd
 SHA512:
-  metadata.gz: 1bf410fb503b270aca3f96191f34cb8ee67730ac49ea6a57d34550d2b603fed7404206d405c61f6dd3b67808f8865d73c80121d303c628dda513ebcf9c418b17
-  data.tar.gz: d1cbb8bd0ed7bd10f53d3d3169985844d4e5e509041c95a18d083dff458a06f26c09e325fea44bc5c1e14bf65b53fdc79e4dbe351f3e4e5f916e6c9135cfa71e
+  metadata.gz: a8d6e99e0881f4cdaa8d6d737cfbd0107382a6585f119df7d3068007ebb768a976b47fd59f78fb2d931ed95147c707523b3c8e5d1e9af0507584de93796e1ec1
+  data.tar.gz: f1cbada5c1adb8892cc297a5e864051da3bdc0b7447cbe1c386b32dde4020cbf1ac6be94935d7bb547c2ec7f4482ebf79b3e1df796bb526db7fd7d5fb42ce662

data/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## [Unreleased]
 
+## [0.9.0] - 2026-06-30
+
+### Added
+
+- **`exwiw explain` now supports the mongodb adapter.** It runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) for the `find` each collection would be dumped with, and prints the explain document as JSON alongside the find description. Verbosity is configurable via the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins): `queryPlanner` (default), `executionStats`, or `allPlansExecution`. The default `queryPlanner` only **plans** the query — it does not execute it or scan any documents — so it is safe to point at a production source; `executionStats`/`allPlansExecution` run the real extraction query to gather runtime stats and should be used deliberately. Because a scoped collection's `belongs_to` ids are captured at runtime (which `explain` never does), `explain` fills each scoped collection's real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id instead of the match-nothing `{_id: {$in: []}}` fallback — so the plan reports the real `IXSCAN`/`COLLSCAN` (queryPlanner selects an index by field, not value; only the bound values are fake). `ExplainRunner` now prints the adapter-agnostic query description (the compiled SELECT for SQL, the find description for mongodb), so the SQL adapters' output is unchanged.
+- **The MongoDB fork-parallel dump (`--parallel-workers`) logs a per-collection extracted-record summary on completion.** The serial path already logs each collection's row count inline, but the parallel path previously emitted only an aggregate (`genuine`/`leaves`/`ref_bt` counts). The leaf and `ref_bt` collections are extracted in forked workers whose counts never reached the parent, so the per-collection numbers were unavailable. Each worker now hands its counts back through the same Marshal-sidecar IPC the consumed-leaf `@state` already uses; the parent merges them with the genuine counts it holds and logs one `name: count` line per collection in processing order (matching the `insert-NNN-` file numbering), plus a `total_records` field on the returned stats. This is purely additive metadata — it does not touch the output files, so the byte-identity guarantee is unchanged.
+
 ## [0.8.5] - 2026-06-25
 
 - **`--parallel-workers=N` parallelizes the MongoDB dump across forked processes (opt-in, byte-identical).** When set with `N≥2` on the mongodb adapter's `export`, exwiw runs an inter-collection fork schedule that decodes whole collections in parallel while preserving each collection's natural row order, so the output files are byte-identical to a serial run (same filenames, same content). Collections are classified into three dependency groups — reference data dumped in full (no `belongs_to`), the scoped DAG reachable to the dump target, and non-reachable reference data — which lets the heavy full-dump collections run concurrently with the scoped pass; only a handful of small `@state` hand-offs cross process boundaries. The win needs real cores: it clears ~2× from 4 workers and saturates there. Requires a dump target and a `fork`-capable runtime (CRuby on POSIX); it falls back to the serial path on JRuby/TruffleRuby/Windows or when no target is given. Also settable as `parallel_workers:` in the config file. The default remains the serial dump.

data/README.md

@@ -59,7 +59,7 @@ driver implements that auth handshake itself and sidesteps the issue.
 exwiw has two subcommands:
 
 - `export` (default) — generate INSERT/COPY SQL files. If the subcommand is omitted, `export` is assumed.
-- `explain` — print the compiled SQL and its `EXPLAIN` output for each query that `export` would run, without executing the SELECTs.
+- `explain` — print each query `export` would run together with its `EXPLAIN` output. SQL adapters compile the SELECT without executing it; mongodb runs the server's explain (defaulting to the execution-free `queryPlanner`).
 
 ### `exwiw export`
 
@@ -115,7 +115,7 @@ idx meaning is the same as insert sql.
 
 ### `exwiw explain`
 
-Print the compiled SQL and its `EXPLAIN` output (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) for each query that `export` would run, to stdout. No SELECT is executed. Supported for `mysql`, `postgresql`, and `sqlite`. The `mongodb` adapter is not yet supported.
+Print the query each `export` would run together with its `EXPLAIN` output, to stdout. For the SQL adapters (`mysql`, `postgresql`, `sqlite`) this is the compiled SELECT plus its `EXPLAIN` (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) — no SELECT is executed. For `mongodb` it is the `find` description plus the server's explain document as JSON.
 
 ```bash
 # preview the queries exwiw would run, without executing the SELECTs
@@ -129,6 +129,29 @@ exwiw explain \
 
 The `--output-dir`, `--output-format`, `--insert-only`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
 
+#### MongoDB explain verbosity
+
+The mongodb explain runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) at a configurable verbosity. The default, **`queryPlanner`, only plans the query and does not execute it**, so it is safe to point at a production source. Set it with the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins):
+
+| verbosity | behaviour |
+|---|---|
+| `queryPlanner` (default) | Plans the query only. **The query is not executed** — no documents are scanned. |
+| `executionStats` | **Runs the query** and reports runtime statistics (docs examined, time, etc.). |
+| `allPlansExecution` | Runs the winning plan **and the rejected candidate plans** to gather their stats. |
+
+```bash
+# inspect index usage of the real extraction query (executes it)
+EXWIW_MONGODB_EXPLAIN_VERBOSITY=executionStats exwiw explain \
+  --adapter=mongodb \
+  --uri="mongodb+srv://reader@cluster/app_production" \
+  --schema-dir=exwiw/schema \
+  --target-collection=shops --ids=...
+```
+
+`executionStats` and `allPlansExecution` execute the extraction query against the source, so use them deliberately on large/production collections.
+
+> **Scoped collections use a placeholder id.** A non-target collection is scoped by its parents' ids, which a real dump captures while running each parent query — but `explain` runs nothing. So for scoped collections, `explain` fills the real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id rather than real values. The plan still reflects the real dump: `queryPlanner` chooses an index by the queried *field*, not its value, so whether a scoped extraction does an `IXSCAN` or a `COLLSCAN` is reported correctly. Only the bound *values* are fake. (The target collection uses the real `--ids`; reference collections with no `belongs_to` use the real `{}` full scan.)
+
 ### Scope-column mode
 
 The default `--target-table` extraction assumes the schema converges on a single
@@ -286,6 +309,7 @@ Notes:
 - **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
 - Unknown keys are rejected so a typo surfaces immediately.
 - Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
+- `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [`exwiw explain`](#mongodb-explain-verbosity).
 
 ### Generator
 

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -68,6 +68,28 @@ module Exwiw
       def initialize(connection_config, logger)
         super
         @state = {}
+        @explain_placeholder = false
+      end
+
+      # A recognizably-fake ObjectId substituted for captured parent ids when
+      # building scope filters in `explain` placeholder mode (see
+      # #explain_scope_with_placeholders! and #parent_state_for).
+      EXPLAIN_PLACEHOLDER_OID_HEX = "ffffffffffffffffffffffff"
+
+      # Switch #build_query into placeholder-scope mode for `explain`. The
+      # mongodb scope of a non-target collection is its parents' captured ids,
+      # which the serial dump harvests at runtime in #execute. `explain` never
+      # executes a query, so that @state stays empty and every scoped child would
+      # otherwise fall back to the match-nothing `{_id: {$in: []}}` filter —
+      # hiding which field (e.g. a foreign key) the real dump would actually
+      # filter on, and thus whether it is indexed. In this mode #parent_state_for
+      # returns a placeholder id for each dumped parent instead of reading
+      # @state, so build_query emits the real foreign-key filter shape with a
+      # dummy value. queryPlanner picks an index by the queried FIELD, not the
+      # value, so index selection (IXSCAN vs COLLSCAN) is reported correctly even
+      # though the bound value is fake.
+      def explain_scope_with_placeholders!
+        @explain_placeholder = true
       end
 
       # Propagation @state accessor, used ONLY by MongodbParallelDumper to seed a
@@ -193,8 +215,30 @@ module Exwiw
         raise NotImplementedError, "MongodbAdapter does not support bulk delete"
       end
 
-      def explain(_query)
-        raise NotImplementedError, "MongodbAdapter does not support explain yet"
+      # Default explain verbosity. `queryPlanner` asks the server to PLAN the
+      # query without executing it, so it is safe to run against a production
+      # source — no documents are scanned or returned. `executionStats` and
+      # `allPlansExecution` actually run the query to collect runtime stats (the
+      # latter also runs the rejected candidate plans), so they carry the cost of
+      # the real extraction query.
+      DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
+
+      # Server-side explain for the find this dump would issue, returned as
+      # pretty-printed relaxed extended JSON (same value semantics as the dumped
+      # documents). `verbosity` is one of queryPlanner / executionStats /
+      # allPlansExecution; see DEFAULT_EXPLAIN_VERBOSITY for the safety
+      # implications. A String verbosity is passed through to the driver as-is.
+      def explain(query, verbosity: nil)
+        verbosity ||= DEFAULT_EXPLAIN_VERBOSITY
+        @logger.debug("  Running explain (verbosity=#{verbosity}) on '#{query.collection}': filter=#{query.filter.inspect}")
+
+        result = db[query.collection]
+          .find(query.filter)
+          .projection(query.projection)
+          .comment(query_comment_text("collection=#{query.collection}"))
+          .explain(verbosity: verbosity)
+
+        JSON.pretty_generate(result.as_extended_json(mode: :relaxed))
       end
 
       def describe_query(query)
@@ -478,7 +522,21 @@ module Exwiw
       # constrained by: the values of the parent field the FK references
       # (`relation.references`, default the parent primary_key). nil when the
       # parent has not been executed yet.
+      #
+      # In `explain` placeholder mode (#explain_scope_with_placeholders!) there is
+      # no captured state, so return a one-element placeholder id for any dumped
+      # (non-embedded) parent — enough to make the child's foreign-key clause
+      # appear with its real field, so the plan reflects the real dump. An
+      # embedded parent is not dumped on its own and yields nil, exactly as a real
+      # run's empty @state would.
       private def parent_state_for(relation, config_by_name)
+        if @explain_placeholder
+          parent = config_by_name[relation.table_name]
+          return nil if parent.nil? || parent.embedded?
+
+          return [explain_placeholder_id]
+        end
+
         parent_fields = @state[relation.table_name]
         return nil if parent_fields.nil?
 
@@ -487,6 +545,17 @@ module Exwiw
         parent_fields[reference_field]
       end
 
+      # The placeholder id used in explain placeholder mode. Built lazily because
+      # build_query can run before any db access loads bson; falls back to the
+      # hex String if bson is genuinely unavailable (the filter shape is what
+      # matters, not the value's type).
+      private def explain_placeholder_id
+        require 'bson' unless defined?(::BSON::ObjectId)
+        @explain_placeholder_id ||= ::BSON::ObjectId.from_string(EXPLAIN_PLACEHOLDER_OID_HEX)
+      rescue LoadError
+        EXPLAIN_PLACEHOLDER_OID_HEX
+      end
+
       # A masking plan compiled once per collection config and reused for every
       # document of that collection. `masked_fields` is `[field_name,
       # template_segments]` for each field carrying a `replace_with`;

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -68,7 +68,7 @@ module Exwiw
         StreamingResult.new(client: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN: \n#{sql}")

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -90,7 +90,7 @@ module Exwiw
         StreamingResult.new(connection: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN: \n#{sql}")

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -71,7 +71,7 @@ module Exwiw
         StreamingResult.new(connection: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN QUERY PLAN: \n#{sql}")

data/lib/exwiw/adapter.rb

@@ -182,11 +182,23 @@ module Exwiw
 
       # Run the database-specific EXPLAIN for the given query and return the
       # output as a single string for `explain` subcommand to print.
-      # SQL adapters override; MongodbAdapter currently raises.
-      def explain(_query_ast)
+      # `verbosity` selects the explain detail level; it is mongodb-only (the
+      # MongoDB explain command's queryPlanner / executionStats /
+      # allPlansExecution) and the SQL adapters accept it only to keep the
+      # contract uniform — they ignore it. SQL adapters override; MongodbAdapter
+      # implements verbosity.
+      def explain(_query_ast, verbosity: nil)
         raise NotImplementedError, "#{self.class.name} does not implement #explain"
       end
 
+      # Hook letting `explain` tell an adapter whose scope is resolved at runtime
+      # (mongodb captures parent ids during execution) to build placeholder scope
+      # filters instead, since explain executes nothing. No-op by default: the
+      # SQL adapters embed scoping in the query itself, so their `explain` already
+      # reflects the real query without any captured state.
+      def explain_scope_with_placeholders!
+      end
+
       # Identifier text prepended to every query exwiw sends to the (often
       # production) source DB, so the statement is recognizable in the
       # processlist / slow-query log / db.currentOp() and can be killed if it

data/lib/exwiw/cli.rb

@@ -36,8 +36,15 @@ module Exwiw
       ids_field
       scope_column
       parallel_workers
+      explain_verbosity
     ].freeze
 
+    # MongoDB explain verbosity levels (passed through to the server's explain
+    # command). `queryPlanner` only plans the query and is the safe default —
+    # the query is not executed; the other two run it to gather runtime stats.
+    EXPLAIN_VERBOSITIES = %w[queryPlanner executionStats allPlansExecution].freeze
+    DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
+
     # Database connection settings are environment-specific (and sometimes
     # secret-adjacent), so they must be passed via CLI/env, never the committed
     # config file. `adapter` is the one connection-ish key allowed in config.
@@ -82,6 +89,7 @@ module Exwiw
       @insert_only = nil
       @after_insert_hook_path = nil
       @parallel_workers = nil
+      @explain_verbosity = nil
       # nil (not :info) so we can tell "user passed --log-level" from the default,
       # letting a config-file value fill in; the :info default is applied later.
       @log_level = nil
@@ -138,6 +146,7 @@ module Exwiw
           dump_target: dump_target,
           logger: logger,
           io: $stdout,
+          explain_verbosity: @explain_verbosity,
         ).run
       end
     end
@@ -172,6 +181,7 @@ module Exwiw
 
       if @subcommand == "explain"
         validate_explain_only!
+        resolve_explain_verbosity!
       end
 
       if @database_adapter != "sqlite"
@@ -321,6 +331,7 @@ module Exwiw
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
       @parallel_workers ||= parse_parallel_workers(config["parallel_workers"]) if config.key?("parallel_workers")
+      @explain_verbosity ||= config["explain_verbosity"]
     end
 
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
@@ -447,11 +458,6 @@ module Exwiw
     end
 
     private def validate_explain_only!
-      if @database_adapter == "mongodb"
-        $stderr.puts "mongodb adapter is not yet supported by 'explain' subcommand"
-        exit 1
-      end
-
       rejected = []
       rejected << "--output-dir" unless @output_dir.nil?
       rejected << "--output-format" unless @output_format.nil?
@@ -465,6 +471,24 @@ module Exwiw
       end
     end
 
+    # Resolve the MongoDB explain verbosity for an `explain` run. It is
+    # mongodb-only, so this no-ops (and skips validation) for the SQL adapters,
+    # which ignore verbosity. The env var wins over the config-file value (it is
+    # the more run-specific override, like a CLI flag would be); when neither is
+    # set the safe, execution-free `queryPlanner` default is used.
+    private def resolve_explain_verbosity!
+      return unless @database_adapter == "mongodb"
+
+      env = ENV["EXWIW_MONGODB_EXPLAIN_VERBOSITY"]
+      @explain_verbosity = env unless env.nil? || env.empty?
+      @explain_verbosity ||= DEFAULT_EXPLAIN_VERBOSITY
+
+      unless EXPLAIN_VERBOSITIES.include?(@explain_verbosity)
+        $stderr.puts "Invalid explain verbosity '#{@explain_verbosity}'. Available options are: #{EXPLAIN_VERBOSITIES.join(', ')}"
+        exit 1
+      end
+    end
+
     # The export clears @output_dir before writing (see Runner#clean_output_dir!).
     # That is destructive, so when running interactively (stdin is a tty) ask for
     # confirmation first. In non-interactive contexts (CI, pipes) we proceed
@@ -533,7 +557,9 @@ module Exwiw
           Subcommands:
             export    Generate INSERT/COPY SQL files (default when omitted).
             explain   Print EXPLAIN output for each extraction query to stdout.
-                      (not yet supported for the mongodb adapter)
+                      For mongodb, set verbosity via EXWIW_MONGODB_EXPLAIN_VERBOSITY
+                      or `explain_verbosity:` in config (queryPlanner (default,
+                      no query is executed) | executionStats | allPlansExecution).
         BANNER
         opts.version = Exwiw::VERSION
 

data/lib/exwiw/explain_runner.rb

@@ -7,17 +7,24 @@ module Exwiw
       schema_dir:,
       dump_target:,
       logger:,
-      io: $stdout
+      io: $stdout,
+      explain_verbosity: nil
     )
       @connection_config = connection_config
       @schema_dir = schema_dir
       @dump_target = dump_target
       @logger = logger
       @io = io
+      # mongodb-only; nil lets the adapter pick its safe default (queryPlanner).
+      @explain_verbosity = explain_verbosity
     end
 
     def run
       adapter = Adapter.build(@connection_config, @logger)
+      # explain executes nothing, so an adapter whose non-target scope is captured
+      # at runtime (mongodb) has no parent ids to filter children by; ask it to
+      # build placeholder scope filters so each query reflects the real dump.
+      adapter.explain_scope_with_placeholders!
       configs = load_table_config(adapter.class.table_config_class)
       validate_ignored(configs)
 
@@ -44,11 +51,13 @@ module Exwiw
         @logger.debug("Explaining '#{table_name}'... (#{idx + 1}/#{total_size})")
 
         query_ast = adapter.build_query(table, @dump_target, table_by_name)
-        sql = adapter.commented_sql(query_ast)
-        explain_text = adapter.explain(query_ast)
+        # describe_query is the adapter-agnostic "what query is this": the
+        # commented SELECT for SQL adapters, the find description for mongodb.
+        query_text = adapter.describe_query(query_ast)
+        explain_text = adapter.explain(query_ast, verbosity: @explain_verbosity)
 
         @io.puts "-- [#{idx + 1}/#{total_size}] #{table_name}"
-        @io.puts sql
+        @io.puts query_text
         @io.puts
         @io.puts "-- EXPLAIN:"
         @io.puts explain_text

data/lib/exwiw/mongodb_parallel_dumper.rb

@@ -41,6 +41,12 @@ module Exwiw
   # fork is required; callers must check {.available?} and fall back to the serial
   # Runner on JRuby/TruffleRuby/Windows.
   class MongodbParallelDumper
+    # Filename prefix for the per-worker record-count sidecars. Distinct from the
+    # consumed-leaf @state sidecars (`<name>.marshal`) so the two never collide in
+    # the shared sidecar dir and the counts can be globbed back independently.
+    COUNTS_SIDECAR_PREFIX = "__exwiw_counts__"
+    private_constant :COUNTS_SIDECAR_PREFIX
+
     # True when the runtime can `fork` (CRuby on a POSIX OS). On JRuby/TruffleRuby
     # and Windows it cannot — the caller must run the serial Runner instead.
     def self.available?
@@ -99,19 +105,24 @@ module Exwiw
 
       FileUtils.mkdir_p(@output_dir)
       parent = build_adapter
+      @counts = {}
 
       Dir.mktmpdir("exwiw-mongo-parallel-") do |sidecar_dir|
         phase1_leaf_and_genuine(parent, sidecar_dir)
         phase2_cascade(parent, sidecar_dir)
         phase3_ref_components(parent, sidecar_dir)
+        collect_counts(sidecar_dir)
       end
 
+      log_count_summary
+
       {
         workers: @workers,
         genuine: @plan.genuine.size,
         leaves: @plan.leaves.size,
         ref_bt: @plan.ref_bt.size,
         components: @plan.reference_components.map(&:size).sort.reverse,
+        total_records: @counts.values.sum,
       }
     end
 
@@ -175,7 +186,7 @@ module Exwiw
       pids = groups.reject(&:empty?).map do |group|
         members = group.flatten
         seed = leaf_state.slice(*parents_of(members))
-        fork { run_component_worker(group, seed) }
+        fork { run_component_worker(group, seed, sidecar_dir) }
       end
       ok = pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
       raise "exwiw parallel ref_bt pool failed" unless ok
@@ -204,25 +215,29 @@ module Exwiw
 
     def run_leaf_worker(group, sidecar_dir)
       adapter = build_adapter
-      group.each { |name| process_collection(adapter, name) }
+      counts = {}
+      group.each { |name| counts[name] = process_collection(adapter, name) }
       group.each do |name|
         next unless @plan.consumed_leaves.include?(name)
         next unless adapter.state.key?(name)
 
         File.binwrite(File.join(sidecar_dir, "#{name}.marshal"), Marshal.dump(adapter.state[name]))
       end
+      write_counts_sidecar(sidecar_dir, group.first, counts)
       exit!(0)
     rescue StandardError => e
       @logger.error("exwiw parallel leaf worker error (#{group.first}..): #{e.class}: #{e.message}")
       exit!(1)
     end
 
-    def run_component_worker(group, seed)
+    def run_component_worker(group, seed, sidecar_dir)
       adapter = build_adapter
       adapter.state = seed unless seed.empty?
       # Each component is already topologically ordered (parent before child) and
       # dependency-closed over intra-ref_bt edges, so a plain serial walk suffices.
-      group.each { |component| component.each { |name| process_collection(adapter, name) } }
+      counts = {}
+      group.each { |component| component.each { |name| counts[name] = process_collection(adapter, name) } }
+      write_counts_sidecar(sidecar_dir, group.first.first, counts)
       exit!(0)
     rescue StandardError => e
       @logger.error("exwiw parallel ref_bt worker error (#{group.first&.first}..): #{e.class}: #{e.message}")
@@ -268,6 +283,34 @@ module Exwiw
       end
     end
 
+    # Gather every collection's extracted record count back into the parent for the
+    # completion summary. genuine counts are already in @row_counts (parent-local,
+    # post-cascade); the leaf and ref_bt collections were extracted in forked
+    # workers, so each worker wrote a small Marshal counts sidecar (the same IPC
+    # pattern the consumed-leaf @state uses) that we merge here before the tmpdir is
+    # removed.
+    def collect_counts(sidecar_dir)
+      @counts.merge!(@row_counts)
+      Dir.glob(File.join(sidecar_dir, "#{COUNTS_SIDECAR_PREFIX}*.marshal")).each do |path|
+        @counts.merge!(Marshal.load(File.binread(path)))
+      end
+    end
+
+    def write_counts_sidecar(sidecar_dir, tag, counts)
+      File.binwrite(File.join(sidecar_dir, "#{COUNTS_SIDECAR_PREFIX}#{tag}.marshal"), Marshal.dump(counts))
+    end
+
+    # Log one line per extractable collection in the dump's processing order (the
+    # same order the insert-NNN- files are numbered), so the counts cross-reference
+    # directly with the output files. A collection that matched no rows shows 0
+    # (its output file was deleted).
+    def log_count_summary
+      total = @counts.values.sum
+      nonzero = @counts.count { |_, count| count.positive? }
+      @logger.info("Per-collection extracted record counts (#{total} record(s) across #{nonzero} collection(s)):")
+      @plan.extractable.each { |name| @logger.info("  #{name}: #{@counts.fetch(name, 0)}") }
+    end
+
     # The distinct belongs_to parent names of `names`, used to slice the leaf @state
     # a worker is seeded with down to only the keys its collections reference.
     def parents_of(names)

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.5"
+  VERSION = "0.9.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.5
+  version: 0.9.0
 platform: ruby
 authors:
 - Shia