exwiw 0.8.5 → 0.9.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3934deb015b3ede7c6a8caa25b857d1f7c8957b4efa3284cd26f21ff2620904d
-  data.tar.gz: e44fef95c3c273aa96b1dde37736212687da88d8d9b98abb5c44fb305d45b7a3
+  metadata.gz: 577420a290ea24adc64ef6ba21914ac71b1c98f86163f9d1d8a88643cc89e15b
+  data.tar.gz: eaddb0073169543aa1bea944cb1b075fbd8258a034f4cffb9cf0011338ba260a
 SHA512:
-  metadata.gz: 1bf410fb503b270aca3f96191f34cb8ee67730ac49ea6a57d34550d2b603fed7404206d405c61f6dd3b67808f8865d73c80121d303c628dda513ebcf9c418b17
-  data.tar.gz: d1cbb8bd0ed7bd10f53d3d3169985844d4e5e509041c95a18d083dff458a06f26c09e325fea44bc5c1e14bf65b53fdc79e4dbe351f3e4e5f916e6c9135cfa71e
+  metadata.gz: c21735b0dcda7b30b0519944c37f24610564e91801a29e2c0f7cc3632cc0f24c532d10c078127bccffd0a2f258a5695cdabe25094edf228b5e3203a629ddf175
+  data.tar.gz: 7e9ce9682b281c097485da409befc466dd06fc655c1612278b9e98f6bccdea575aa443ff4292777be1e4e68f88d826554702eecd6b94bddc31b1e4edc01c923b

data/CHANGELOG.md

@@ -2,6 +2,19 @@
 
 ## [Unreleased]
 
+## [0.9.1] - 2026-06-30
+
+### Fixed
+
+- **The `reverse_scope` satellite cascade now fires in single-target (`--target-table`) mode, not only scope-column mode.** A table that `belongs_to` a `reverse_scope`'d hub (a "satellite") is scoped by constraining it to the hub's in-scope ids — the multi-hop forward (`via_scoped_parent`) cascade. That cascade ran only in scope-column mode: in single-target / PK-anchor mode the hub itself was scoped via `reverse_scope`, but its `belongs_to` children fell through to a full dump — a silent cross-tenant export in a multi-tenant schema, despite the README promising the cascade works in "both single-target and scope-column mode." The single-target build now runs the same `build_belongs_to_scoped_clause` cascade for a non-target table that has no `belongs_to` path to the dump target, so satellites tighten to the kept ids (multi-hop, keeping the single-unambiguous-parent rule, polymorphic skip, and forward-path cycle guard); it can only narrow such a table's output, never widen it. Scope-column mode is unchanged — the cascade logic is reused as-is. Because single-target mode has no `validate_scope!` pre-flight, a satellite the cascade cannot resolve to a single scopable parent (e.g. it `belongs_to` two scopable hubs) is still dumped in full but now logs a warning. The README also documents that `referenced_by` scoping takes precedence over the hub cascade (which can under-scope) and how to force the hub cascade. SQL adapters only.
+
+## [0.9.0] - 2026-06-30
+
+### Added
+
+- **`exwiw explain` now supports the mongodb adapter.** It runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) for the `find` each collection would be dumped with, and prints the explain document as JSON alongside the find description. Verbosity is configurable via the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins): `queryPlanner` (default), `executionStats`, or `allPlansExecution`. The default `queryPlanner` only **plans** the query — it does not execute it or scan any documents — so it is safe to point at a production source; `executionStats`/`allPlansExecution` run the real extraction query to gather runtime stats and should be used deliberately. Because a scoped collection's `belongs_to` ids are captured at runtime (which `explain` never does), `explain` fills each scoped collection's real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id instead of the match-nothing `{_id: {$in: []}}` fallback — so the plan reports the real `IXSCAN`/`COLLSCAN` (queryPlanner selects an index by field, not value; only the bound values are fake). `ExplainRunner` now prints the adapter-agnostic query description (the compiled SELECT for SQL, the find description for mongodb), so the SQL adapters' output is unchanged.
+- **The MongoDB fork-parallel dump (`--parallel-workers`) logs a per-collection extracted-record summary on completion.** The serial path already logs each collection's row count inline, but the parallel path previously emitted only an aggregate (`genuine`/`leaves`/`ref_bt` counts). The leaf and `ref_bt` collections are extracted in forked workers whose counts never reached the parent, so the per-collection numbers were unavailable. Each worker now hands its counts back through the same Marshal-sidecar IPC the consumed-leaf `@state` already uses; the parent merges them with the genuine counts it holds and logs one `name: count` line per collection in processing order (matching the `insert-NNN-` file numbering), plus a `total_records` field on the returned stats. This is purely additive metadata — it does not touch the output files, so the byte-identity guarantee is unchanged.
+
 ## [0.8.5] - 2026-06-25
 
 - **`--parallel-workers=N` parallelizes the MongoDB dump across forked processes (opt-in, byte-identical).** When set with `N≥2` on the mongodb adapter's `export`, exwiw runs an inter-collection fork schedule that decodes whole collections in parallel while preserving each collection's natural row order, so the output files are byte-identical to a serial run (same filenames, same content). Collections are classified into three dependency groups — reference data dumped in full (no `belongs_to`), the scoped DAG reachable to the dump target, and non-reachable reference data — which lets the heavy full-dump collections run concurrently with the scoped pass; only a handful of small `@state` hand-offs cross process boundaries. The win needs real cores: it clears ~2× from 4 workers and saturates there. Requires a dump target and a `fork`-capable runtime (CRuby on POSIX); it falls back to the serial path on JRuby/TruffleRuby/Windows or when no target is given. Also settable as `parallel_workers:` in the config file. The default remains the serial dump.

data/README.md

@@ -59,7 +59,7 @@ driver implements that auth handshake itself and sidesteps the issue.
 exwiw has two subcommands:
 
 - `export` (default) — generate INSERT/COPY SQL files. If the subcommand is omitted, `export` is assumed.
-- `explain` — print the compiled SQL and its `EXPLAIN` output for each query that `export` would run, without executing the SELECTs.
+- `explain` — print each query `export` would run together with its `EXPLAIN` output. SQL adapters compile the SELECT without executing it; mongodb runs the server's explain (defaulting to the execution-free `queryPlanner`).
 
 ### `exwiw export`
 
@@ -115,7 +115,7 @@ idx meaning is the same as insert sql.
 
 ### `exwiw explain`
 
-Print the compiled SQL and its `EXPLAIN` output (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) for each query that `export` would run, to stdout. No SELECT is executed. Supported for `mysql`, `postgresql`, and `sqlite`. The `mongodb` adapter is not yet supported.
+Print the query each `export` would run together with its `EXPLAIN` output, to stdout. For the SQL adapters (`mysql`, `postgresql`, `sqlite`) this is the compiled SELECT plus its `EXPLAIN` (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) — no SELECT is executed. For `mongodb` it is the `find` description plus the server's explain document as JSON.
 
 ```bash
 # preview the queries exwiw would run, without executing the SELECTs
@@ -129,6 +129,29 @@ exwiw explain \
 
 The `--output-dir`, `--output-format`, `--insert-only`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
 
+#### MongoDB explain verbosity
+
+The mongodb explain runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) at a configurable verbosity. The default, **`queryPlanner`, only plans the query and does not execute it**, so it is safe to point at a production source. Set it with the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins):
+
+| verbosity | behaviour |
+|---|---|
+| `queryPlanner` (default) | Plans the query only. **The query is not executed** — no documents are scanned. |
+| `executionStats` | **Runs the query** and reports runtime statistics (docs examined, time, etc.). |
+| `allPlansExecution` | Runs the winning plan **and the rejected candidate plans** to gather their stats. |
+
+```bash
+# inspect index usage of the real extraction query (executes it)
+EXWIW_MONGODB_EXPLAIN_VERBOSITY=executionStats exwiw explain \
+  --adapter=mongodb \
+  --uri="mongodb+srv://reader@cluster/app_production" \
+  --schema-dir=exwiw/schema \
+  --target-collection=shops --ids=...
+```
+
+`executionStats` and `allPlansExecution` execute the extraction query against the source, so use them deliberately on large/production collections.
+
+> **Scoped collections use a placeholder id.** A non-target collection is scoped by its parents' ids, which a real dump captures while running each parent query — but `explain` runs nothing. So for scoped collections, `explain` fills the real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id rather than real values. The plan still reflects the real dump: `queryPlanner` chooses an index by the queried *field*, not its value, so whether a scoped extraction does an `IXSCAN` or a `COLLSCAN` is reported correctly. Only the bound *values* are fake. (The target collection uses the real `--ids`; reference collections with no `belongs_to` use the real `{}` full scan.)
+
 ### Scope-column mode
 
 The default `--target-table` extraction assumes the schema converges on a single
@@ -197,6 +220,13 @@ Each table is resolved as follows:
   path, set `ignore: true` to skip it, or mark it `scope_exempt: true` (below) to
   export it in full.
 
+> **Note — referenced-by is preferred over the hub cascade.** A table that is
+> *both* `belongs_to` a scoped hub *and* referenced-by a constrained child is
+> scoped to the (narrower) referenced-by id-set, not the hub cascade, so the hub's
+> other children the child does not reference are dropped (under-scoping). To force
+> the broader hub cascade, set `ignore: true` on the child's `belongs_to` edge that
+> points at this table.
+
 Scope-column mode is SQL-only (mysql / postgresql / sqlite). It works with `exwiw
 explain` too, which is the recommended way to preview the queries before exporting.
 
@@ -286,6 +316,7 @@ Notes:
 - **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
 - Unknown keys are rejected so a typo surfaces immediately.
 - Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
+- `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [`exwiw explain`](#mongodb-explain-verbosity).
 
 ### Generator
 
@@ -634,7 +665,7 @@ Notes:
 - **Only scoped referencers belong in `via`.** Each arm's query must come out constrained; an unconstrained referencer (e.g. a `scope_exempt` table, or one with no path to a scope) would project *every* id and union the whole table back — so such an arm is **skipped with a warning** rather than silently widening the dump. An unknown table is likewise skipped with a warning. If no arm survives, the table stays unscopable and (in [scope-column mode](#scope-column-mode)) the run aborts via `validate_scope!`.
 - **NULLs are excluded** per arm (`IS NOT NULL`).
 - **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`. The cascade is **multi-hop**, so a table several `belongs_to` hops below the reverse-scoped table (e.g. `end_user_profiles → end_users → users`) also tightens automatically, with no config of its own.
-- Works in both single-target and scope-column mode. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
+- Works in both single-target and scope-column mode. In single-target mode there is no scope-column pre-flight (`validate_scope!`), so a satellite the cascade cannot resolve to a single scopable parent (e.g. it `belongs_to` two scopable hubs) is dumped in full with a warning rather than aborting. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
 
 ### Why a JOIN, not `IN (subquery)`
 

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -68,6 +68,28 @@ module Exwiw
       def initialize(connection_config, logger)
         super
         @state = {}
+        @explain_placeholder = false
+      end
+
+      # A recognizably-fake ObjectId substituted for captured parent ids when
+      # building scope filters in `explain` placeholder mode (see
+      # #explain_scope_with_placeholders! and #parent_state_for).
+      EXPLAIN_PLACEHOLDER_OID_HEX = "ffffffffffffffffffffffff"
+
+      # Switch #build_query into placeholder-scope mode for `explain`. The
+      # mongodb scope of a non-target collection is its parents' captured ids,
+      # which the serial dump harvests at runtime in #execute. `explain` never
+      # executes a query, so that @state stays empty and every scoped child would
+      # otherwise fall back to the match-nothing `{_id: {$in: []}}` filter —
+      # hiding which field (e.g. a foreign key) the real dump would actually
+      # filter on, and thus whether it is indexed. In this mode #parent_state_for
+      # returns a placeholder id for each dumped parent instead of reading
+      # @state, so build_query emits the real foreign-key filter shape with a
+      # dummy value. queryPlanner picks an index by the queried FIELD, not the
+      # value, so index selection (IXSCAN vs COLLSCAN) is reported correctly even
+      # though the bound value is fake.
+      def explain_scope_with_placeholders!
+        @explain_placeholder = true
       end
 
       # Propagation @state accessor, used ONLY by MongodbParallelDumper to seed a
@@ -193,8 +215,30 @@ module Exwiw
         raise NotImplementedError, "MongodbAdapter does not support bulk delete"
       end
 
-      def explain(_query)
-        raise NotImplementedError, "MongodbAdapter does not support explain yet"
+      # Default explain verbosity. `queryPlanner` asks the server to PLAN the
+      # query without executing it, so it is safe to run against a production
+      # source — no documents are scanned or returned. `executionStats` and
+      # `allPlansExecution` actually run the query to collect runtime stats (the
+      # latter also runs the rejected candidate plans), so they carry the cost of
+      # the real extraction query.
+      DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
+
+      # Server-side explain for the find this dump would issue, returned as
+      # pretty-printed relaxed extended JSON (same value semantics as the dumped
+      # documents). `verbosity` is one of queryPlanner / executionStats /
+      # allPlansExecution; see DEFAULT_EXPLAIN_VERBOSITY for the safety
+      # implications. A String verbosity is passed through to the driver as-is.
+      def explain(query, verbosity: nil)
+        verbosity ||= DEFAULT_EXPLAIN_VERBOSITY
+        @logger.debug("  Running explain (verbosity=#{verbosity}) on '#{query.collection}': filter=#{query.filter.inspect}")
+
+        result = db[query.collection]
+          .find(query.filter)
+          .projection(query.projection)
+          .comment(query_comment_text("collection=#{query.collection}"))
+          .explain(verbosity: verbosity)
+
+        JSON.pretty_generate(result.as_extended_json(mode: :relaxed))
       end
 
       def describe_query(query)
@@ -478,7 +522,21 @@ module Exwiw
       # constrained by: the values of the parent field the FK references
       # (`relation.references`, default the parent primary_key). nil when the
       # parent has not been executed yet.
+      #
+      # In `explain` placeholder mode (#explain_scope_with_placeholders!) there is
+      # no captured state, so return a one-element placeholder id for any dumped
+      # (non-embedded) parent — enough to make the child's foreign-key clause
+      # appear with its real field, so the plan reflects the real dump. An
+      # embedded parent is not dumped on its own and yields nil, exactly as a real
+      # run's empty @state would.
       private def parent_state_for(relation, config_by_name)
+        if @explain_placeholder
+          parent = config_by_name[relation.table_name]
+          return nil if parent.nil? || parent.embedded?
+
+          return [explain_placeholder_id]
+        end
+
         parent_fields = @state[relation.table_name]
         return nil if parent_fields.nil?
 
@@ -487,6 +545,17 @@ module Exwiw
         parent_fields[reference_field]
       end
 
+      # The placeholder id used in explain placeholder mode. Built lazily because
+      # build_query can run before any db access loads bson; falls back to the
+      # hex String if bson is genuinely unavailable (the filter shape is what
+      # matters, not the value's type).
+      private def explain_placeholder_id
+        require 'bson' unless defined?(::BSON::ObjectId)
+        @explain_placeholder_id ||= ::BSON::ObjectId.from_string(EXPLAIN_PLACEHOLDER_OID_HEX)
+      rescue LoadError
+        EXPLAIN_PLACEHOLDER_OID_HEX
+      end
+
       # A masking plan compiled once per collection config and reused for every
       # document of that collection. `masked_fields` is `[field_name,
       # template_segments]` for each field carrying a `replace_with`;

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -68,7 +68,7 @@ module Exwiw
         StreamingResult.new(client: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN: \n#{sql}")

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -90,7 +90,7 @@ module Exwiw
         StreamingResult.new(connection: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN: \n#{sql}")

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -71,7 +71,7 @@ module Exwiw
         StreamingResult.new(connection: connection, data_sql: data_sql, count_sql: count_sql)
       end
 
-      def explain(query_ast)
+      def explain(query_ast, verbosity: nil)
         sql = commented_sql(query_ast)
 
         @logger.debug("  Executing EXPLAIN QUERY PLAN: \n#{sql}")

data/lib/exwiw/adapter.rb

@@ -182,11 +182,23 @@ module Exwiw
 
       # Run the database-specific EXPLAIN for the given query and return the
       # output as a single string for `explain` subcommand to print.
-      # SQL adapters override; MongodbAdapter currently raises.
-      def explain(_query_ast)
+      # `verbosity` selects the explain detail level; it is mongodb-only (the
+      # MongoDB explain command's queryPlanner / executionStats /
+      # allPlansExecution) and the SQL adapters accept it only to keep the
+      # contract uniform — they ignore it. SQL adapters override; MongodbAdapter
+      # implements verbosity.
+      def explain(_query_ast, verbosity: nil)
         raise NotImplementedError, "#{self.class.name} does not implement #explain"
       end
 
+      # Hook letting `explain` tell an adapter whose scope is resolved at runtime
+      # (mongodb captures parent ids during execution) to build placeholder scope
+      # filters instead, since explain executes nothing. No-op by default: the
+      # SQL adapters embed scoping in the query itself, so their `explain` already
+      # reflects the real query without any captured state.
+      def explain_scope_with_placeholders!
+      end
+
       # Identifier text prepended to every query exwiw sends to the (often
       # production) source DB, so the statement is recognizable in the
       # processlist / slow-query log / db.currentOp() and can be killed if it

data/lib/exwiw/cli.rb

@@ -36,8 +36,15 @@ module Exwiw
       ids_field
       scope_column
       parallel_workers
+      explain_verbosity
     ].freeze
 
+    # MongoDB explain verbosity levels (passed through to the server's explain
+    # command). `queryPlanner` only plans the query and is the safe default —
+    # the query is not executed; the other two run it to gather runtime stats.
+    EXPLAIN_VERBOSITIES = %w[queryPlanner executionStats allPlansExecution].freeze
+    DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
+
     # Database connection settings are environment-specific (and sometimes
     # secret-adjacent), so they must be passed via CLI/env, never the committed
     # config file. `adapter` is the one connection-ish key allowed in config.
@@ -82,6 +89,7 @@ module Exwiw
       @insert_only = nil
       @after_insert_hook_path = nil
       @parallel_workers = nil
+      @explain_verbosity = nil
       # nil (not :info) so we can tell "user passed --log-level" from the default,
       # letting a config-file value fill in; the :info default is applied later.
       @log_level = nil
@@ -138,6 +146,7 @@ module Exwiw
           dump_target: dump_target,
           logger: logger,
           io: $stdout,
+          explain_verbosity: @explain_verbosity,
         ).run
       end
     end
@@ -172,6 +181,7 @@ module Exwiw
 
       if @subcommand == "explain"
         validate_explain_only!
+        resolve_explain_verbosity!
       end
 
       if @database_adapter != "sqlite"
@@ -321,6 +331,7 @@ module Exwiw
       @ids_field ||= config["ids_field"]
       @scope_column ||= config["scope_column"]
       @parallel_workers ||= parse_parallel_workers(config["parallel_workers"]) if config.key?("parallel_workers")
+      @explain_verbosity ||= config["explain_verbosity"]
     end
 
     # Strip a trailing slash (like the CLI's dir options) and expand relative to
@@ -447,11 +458,6 @@ module Exwiw
     end
 
     private def validate_explain_only!
-      if @database_adapter == "mongodb"
-        $stderr.puts "mongodb adapter is not yet supported by 'explain' subcommand"
-        exit 1
-      end
-
       rejected = []
       rejected << "--output-dir" unless @output_dir.nil?
       rejected << "--output-format" unless @output_format.nil?
@@ -465,6 +471,24 @@ module Exwiw
       end
     end
 
+    # Resolve the MongoDB explain verbosity for an `explain` run. It is
+    # mongodb-only, so this no-ops (and skips validation) for the SQL adapters,
+    # which ignore verbosity. The env var wins over the config-file value (it is
+    # the more run-specific override, like a CLI flag would be); when neither is
+    # set the safe, execution-free `queryPlanner` default is used.
+    private def resolve_explain_verbosity!
+      return unless @database_adapter == "mongodb"
+
+      env = ENV["EXWIW_MONGODB_EXPLAIN_VERBOSITY"]
+      @explain_verbosity = env unless env.nil? || env.empty?
+      @explain_verbosity ||= DEFAULT_EXPLAIN_VERBOSITY
+
+      unless EXPLAIN_VERBOSITIES.include?(@explain_verbosity)
+        $stderr.puts "Invalid explain verbosity '#{@explain_verbosity}'. Available options are: #{EXPLAIN_VERBOSITIES.join(', ')}"
+        exit 1
+      end
+    end
+
     # The export clears @output_dir before writing (see Runner#clean_output_dir!).
     # That is destructive, so when running interactively (stdin is a tty) ask for
     # confirmation first. In non-interactive contexts (CI, pipes) we proceed
@@ -533,7 +557,9 @@ module Exwiw
           Subcommands:
             export    Generate INSERT/COPY SQL files (default when omitted).
             explain   Print EXPLAIN output for each extraction query to stdout.
-                      (not yet supported for the mongodb adapter)
+                      For mongodb, set verbosity via EXWIW_MONGODB_EXPLAIN_VERBOSITY
+                      or `explain_verbosity:` in config (queryPlanner (default,
+                      no query is executed) | executionStats | allPlansExecution).
         BANNER
         opts.version = Exwiw::VERSION
 

data/lib/exwiw/explain_runner.rb

@@ -7,17 +7,24 @@ module Exwiw
       schema_dir:,
       dump_target:,
       logger:,
-      io: $stdout
+      io: $stdout,
+      explain_verbosity: nil
     )
       @connection_config = connection_config
       @schema_dir = schema_dir
       @dump_target = dump_target
       @logger = logger
       @io = io
+      # mongodb-only; nil lets the adapter pick its safe default (queryPlanner).
+      @explain_verbosity = explain_verbosity
     end
 
     def run
       adapter = Adapter.build(@connection_config, @logger)
+      # explain executes nothing, so an adapter whose non-target scope is captured
+      # at runtime (mongodb) has no parent ids to filter children by; ask it to
+      # build placeholder scope filters so each query reflects the real dump.
+      adapter.explain_scope_with_placeholders!
       configs = load_table_config(adapter.class.table_config_class)
       validate_ignored(configs)
 
@@ -44,11 +51,13 @@ module Exwiw
         @logger.debug("Explaining '#{table_name}'... (#{idx + 1}/#{total_size})")
 
         query_ast = adapter.build_query(table, @dump_target, table_by_name)
-        sql = adapter.commented_sql(query_ast)
-        explain_text = adapter.explain(query_ast)
+        # describe_query is the adapter-agnostic "what query is this": the
+        # commented SELECT for SQL adapters, the find description for mongodb.
+        query_text = adapter.describe_query(query_ast)
+        explain_text = adapter.explain(query_ast, verbosity: @explain_verbosity)
 
         @io.puts "-- [#{idx + 1}/#{total_size}] #{table_name}"
-        @io.puts sql
+        @io.puts query_text
         @io.puts
         @io.puts "-- EXPLAIN:"
         @io.puts explain_text

data/lib/exwiw/mongodb_parallel_dumper.rb

@@ -41,6 +41,12 @@ module Exwiw
   # fork is required; callers must check {.available?} and fall back to the serial
   # Runner on JRuby/TruffleRuby/Windows.
   class MongodbParallelDumper
+    # Filename prefix for the per-worker record-count sidecars. Distinct from the
+    # consumed-leaf @state sidecars (`<name>.marshal`) so the two never collide in
+    # the shared sidecar dir and the counts can be globbed back independently.
+    COUNTS_SIDECAR_PREFIX = "__exwiw_counts__"
+    private_constant :COUNTS_SIDECAR_PREFIX
+
     # True when the runtime can `fork` (CRuby on a POSIX OS). On JRuby/TruffleRuby
     # and Windows it cannot — the caller must run the serial Runner instead.
     def self.available?
@@ -99,19 +105,24 @@ module Exwiw
 
       FileUtils.mkdir_p(@output_dir)
       parent = build_adapter
+      @counts = {}
 
       Dir.mktmpdir("exwiw-mongo-parallel-") do |sidecar_dir|
         phase1_leaf_and_genuine(parent, sidecar_dir)
         phase2_cascade(parent, sidecar_dir)
         phase3_ref_components(parent, sidecar_dir)
+        collect_counts(sidecar_dir)
       end
 
+      log_count_summary
+
       {
         workers: @workers,
         genuine: @plan.genuine.size,
         leaves: @plan.leaves.size,
         ref_bt: @plan.ref_bt.size,
         components: @plan.reference_components.map(&:size).sort.reverse,
+        total_records: @counts.values.sum,
       }
     end
 
@@ -175,7 +186,7 @@ module Exwiw
       pids = groups.reject(&:empty?).map do |group|
         members = group.flatten
         seed = leaf_state.slice(*parents_of(members))
-        fork { run_component_worker(group, seed) }
+        fork { run_component_worker(group, seed, sidecar_dir) }
       end
       ok = pids.map { |pid| Process.wait(pid); $?.exitstatus&.zero? }.all?
       raise "exwiw parallel ref_bt pool failed" unless ok
@@ -204,25 +215,29 @@ module Exwiw
 
     def run_leaf_worker(group, sidecar_dir)
       adapter = build_adapter
-      group.each { |name| process_collection(adapter, name) }
+      counts = {}
+      group.each { |name| counts[name] = process_collection(adapter, name) }
       group.each do |name|
         next unless @plan.consumed_leaves.include?(name)
         next unless adapter.state.key?(name)
 
         File.binwrite(File.join(sidecar_dir, "#{name}.marshal"), Marshal.dump(adapter.state[name]))
       end
+      write_counts_sidecar(sidecar_dir, group.first, counts)
       exit!(0)
     rescue StandardError => e
       @logger.error("exwiw parallel leaf worker error (#{group.first}..): #{e.class}: #{e.message}")
       exit!(1)
     end
 
-    def run_component_worker(group, seed)
+    def run_component_worker(group, seed, sidecar_dir)
       adapter = build_adapter
       adapter.state = seed unless seed.empty?
       # Each component is already topologically ordered (parent before child) and
       # dependency-closed over intra-ref_bt edges, so a plain serial walk suffices.
-      group.each { |component| component.each { |name| process_collection(adapter, name) } }
+      counts = {}
+      group.each { |component| component.each { |name| counts[name] = process_collection(adapter, name) } }
+      write_counts_sidecar(sidecar_dir, group.first.first, counts)
       exit!(0)
     rescue StandardError => e
       @logger.error("exwiw parallel ref_bt worker error (#{group.first&.first}..): #{e.class}: #{e.message}")
@@ -268,6 +283,34 @@ module Exwiw
       end
     end
 
+    # Gather every collection's extracted record count back into the parent for the
+    # completion summary. genuine counts are already in @row_counts (parent-local,
+    # post-cascade); the leaf and ref_bt collections were extracted in forked
+    # workers, so each worker wrote a small Marshal counts sidecar (the same IPC
+    # pattern the consumed-leaf @state uses) that we merge here before the tmpdir is
+    # removed.
+    def collect_counts(sidecar_dir)
+      @counts.merge!(@row_counts)
+      Dir.glob(File.join(sidecar_dir, "#{COUNTS_SIDECAR_PREFIX}*.marshal")).each do |path|
+        @counts.merge!(Marshal.load(File.binread(path)))
+      end
+    end
+
+    def write_counts_sidecar(sidecar_dir, tag, counts)
+      File.binwrite(File.join(sidecar_dir, "#{COUNTS_SIDECAR_PREFIX}#{tag}.marshal"), Marshal.dump(counts))
+    end
+
+    # Log one line per extractable collection in the dump's processing order (the
+    # same order the insert-NNN- files are numbered), so the counts cross-reference
+    # directly with the output files. A collection that matched no rows shows 0
+    # (its output file was deleted).
+    def log_count_summary
+      total = @counts.values.sum
+      nonzero = @counts.count { |_, count| count.positive? }
+      @logger.info("Per-collection extracted record counts (#{total} record(s) across #{nonzero} collection(s)):")
+      @plan.extractable.each { |name| @logger.info("  #{name}: #{@counts.fetch(name, 0)}") }
+    end
+
     # The distinct belongs_to parent names of `names`, used to slice the leaf @state
     # a worker is seeded with down to only the keys its collections reference.
     def parents_of(names)

data/lib/exwiw/query_ast_builder.rb

@@ -86,6 +86,28 @@ module Exwiw
         where_clauses.push(reverse_clause) if reverse_clause
       end
 
+      # Forward cascade. A satellite of a reverse_scope'd (or referenced-by-scoped)
+      # hub has no belongs_to path to the dump target, so the clauses above stay
+      # empty and it would dump every row. When its belongs_to parent is itself
+      # scoped, constrain this table to the parent's in-scope ids — the same
+      # multi-hop cascade scope-column mode performs in build_scoped.
+      if table.name != dump_target.table_name &&
+         where_clauses.empty? && join_clauses.empty? &&
+         forward_scope_allowed?(table)
+        parent_clause = build_belongs_to_scoped_clause(table)
+        if parent_clause
+          where_clauses.push(parent_clause)
+        elsif @allow_reverse && @forward_path.empty? && !scope_exempt?(table) &&
+              scopable_parent_candidates(table).size > 1
+          @logger.warn(
+            "  #{table.name} belongs_to multiple scopable parents; the cascade cannot " \
+            "pick one unambiguously, so it is dumped in full. If this is intended, set " \
+            "`scope_exempt: true`. Otherwise, scope it through a single parent (e.g. ignore one belongs_to edge), " \
+            "or switch to scope-column mode to scope it directly."
+          )
+        end
+      end
+
       QueryAst::Select.new.tap do |ast|
         ast.from(table.name)
         if table.rails_managed?
@@ -290,48 +312,18 @@ module Exwiw
       )
     end
 
-    # Scope-column mode. Builds a `fk IN (SELECT parent.pk FROM <parent
-    # extraction query>)` clause for a table whose belongs_to parent is itself
-    # scopable but carries no scope column of its own — so find_path_to_scoped
-    # cannot terminate on it (via_path fails) and nothing references this table
-    # (referenced_by fails). The classic shape is a hub scoped only via
-    # referenced_by (e.g. CDP `customer_accounts`, scoped by the `customers` that
-    # reference it) with sibling detail tables (`customer_account_details`, ...)
-    # hanging off it. Constraining those siblings to the hub's in-scope ids keeps
-    # them out of a full dump. Returns nil when there is no single, unambiguous
-    # scopable parent, leaving the caller on the unscopable path.
+    # Builds a `fk IN (SELECT parent.pk FROM <parent extraction query>)` clause
+    # for a table whose belongs_to parent is itself scopable but carries no scope
+    # column of its own — so find_path_to_scoped cannot terminate on it (via_path
+    # fails) and nothing references this table (referenced_by fails). The classic
+    # shape is a hub scoped only via referenced_by (e.g. CDP `customer_accounts`,
+    # scoped by the `customers` that reference it) with sibling detail tables
+    # (`customer_account_details`, ...) hanging off it. Constraining those
+    # siblings to the hub's in-scope ids keeps them out of a full dump. Returns
+    # nil when there is no single, unambiguous scopable parent, leaving the caller
+    # on the unscopable path. Used by both scope-column and single-target mode.
     private def build_belongs_to_scoped_clause(table)
-      # This table plus every ancestor currently being forward-resolved. A
-      # candidate parent already on this path would close a belongs_to cycle, so
-      # it is skipped; threading the grown path into the parent build lets the
-      # cascade recurse N hops (users -> end_users -> end_user_profiles -> ...)
-      # and terminate only when a table reappears.
-      forward_path = @forward_path + [table.name]
-
-      candidates = table.belongs_tos.filter_map do |relation|
-        # A polymorphic belongs_to points at several parent tables through one
-        # column, so it cannot project to a single parent id set; skip it.
-        next if relation.polymorphic?
-
-        parent = table_by_name[relation.table_name]
-        next if parent.nil?
-
-        # Cycle guard: descending into a parent already on the forward path would
-        # loop (a -> b -> a). Stop, leaving this table on the :unscopable path.
-        next if forward_path.include?(parent.name)
-
-        # Build the parent's own scoped query. allow_reverse stays true so the
-        # parent may be scoped via referenced_by, and forward scoping stays
-        # enabled so a parent that is itself scoped via *its* parent resolves
-        # too — this is what makes the cascade multi-hop.
-        parent_query = self.class.run(parent.name, table_by_name, dump_target, @logger, allow_reverse: true, forward_path: forward_path)
-
-        # Only a constrained parent narrows anything; an unconstrained parent
-        # would select every pk (i.e. dump all) and not help.
-        next unless parent_query.where_clauses.any? || parent_query.join_clauses.any?
-
-        [relation, parent, parent_query]
-      end
+      candidates = scopable_parent_candidates(table)
 
       # Only the unambiguous single-parent case. Multiple scopable parents would
       # need their subqueries combined (not supported); fall back to unscopable.
@@ -360,6 +352,45 @@ module Exwiw
       )
     end
 
+    # The scopable belongs_to parents of `table`: each non-polymorphic parent
+    # whose own extraction query comes out constrained, paired with the relation
+    # and that query. Shared by build_belongs_to_scoped_clause (which requires
+    # exactly one) and the single-target full-dump warning (which flags two or
+    # more, since the cascade then cannot disambiguate).
+    private def scopable_parent_candidates(table)
+      # Memoized: #run can resolve this twice for the same table (once via
+      # build_belongs_to_scoped_clause, once for the ambiguous-parent warning),
+      # and each pass recursively builds every parent's query.
+      (@scopable_parent_candidates ||= {})[table.name] ||= begin
+        # This table plus every ancestor currently being forward-resolved; a
+        # candidate parent already on this path would close a belongs_to cycle, so
+        # it is skipped. Threading the grown path into the parent build lets the
+        # cascade recurse N hops and terminate only when a table reappears.
+        forward_path = @forward_path + [table.name]
+
+        table.belongs_tos.filter_map do |relation|
+          # A polymorphic belongs_to points at several parent tables through one
+          # column, so it cannot project to a single parent id set.
+          next if relation.polymorphic?
+
+          parent = table_by_name[relation.table_name]
+          next if parent.nil?
+          next if forward_path.include?(parent.name)
+
+          # allow_reverse and forward scoping stay enabled so the parent may itself
+          # be scoped via referenced_by or via *its* parent — this is what makes the
+          # cascade multi-hop.
+          parent_query = self.class.run(parent.name, table_by_name, dump_target, @logger, allow_reverse: true, forward_path: forward_path)
+
+          # Only a constrained parent narrows anything; an unconstrained parent
+          # would select every pk (i.e. dump all) and not help.
+          next unless parent_query.where_clauses.any? || parent_query.join_clauses.any?
+
+          [relation, parent, parent_query]
+        end
+      end
+    end
+
     private def build_where_clauses(table, dump_target)
       clauses = []
 

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.8.5"
+  VERSION = "0.9.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.8.5
+  version: 0.9.1
 platform: ruby
 authors:
 - Shia