exwiw 0.6.0 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 39362410df244fffa463a86c845062f0e9bacac723e15a6697a50c631db0d5cd
-  data.tar.gz: 9670677e7c822886ed6268e476008c81a1caba4e2eacb286bf0e747fef5d3f3c
+  metadata.gz: b66bdb8ccf3d2043c0a903c70071e0f070b41b9fd89dbc09e3c5385c9b4916c1
+  data.tar.gz: cf1224f47635113127f2e1fcf38f161c539f7b68f0bb8c85ba0c428fa6a202d1
 SHA512:
-  metadata.gz: 4af4db84210fbd9da8b6b32e136c1f12af370bc719c5aeee2a1f7183046f1ffae5e8dcdc0bb6d856e30fe800a5febe12aef4d4ff2d2e8c27b162fcc4a056c7f4
-  data.tar.gz: 5064dfee83c653ae0c73ae07edf33299752c748c6f9efc5e413424e7b3a92b769f6a99901688a2d1d3415ad3a0939ee807095b1cf24b2aab46fcf89402113a90
+  metadata.gz: 41a4ef3c8a6da41ccbb92d7e42a519c5260a1bbcf53ce9cdcfe3a18dbca55d509c7ca1492e726bdb82bfc8c101a0a6bcee1dec1c051ceae44910009801d4b64d
+  data.tar.gz: ce5cdc2e96fcacbfc2a7c3ca23c72a22a1c3e765da2e69ad21fd9ebd87bbfdc3619e54a0c6eb5ae94321e4dc1b8651ca73a965cf33509ecafba9349b8a894883

data/CHANGELOG.md

@@ -2,6 +2,8 @@
 
 ## [Unreleased]
 
+## [0.6.1] - 2026-06-20
+
 ## [0.6.0] - 2026-06-20
 
 ### Added

data/docs/mongodb-scoping-fullscan-notes.md

@@ -0,0 +1,145 @@
+# MongoDB scoping full-scan diagnosis (nullable-FK belongs_to)
+
+Why a related collection's dump can be "especially slow" — dumping far more
+records than the scope implies and scanning the whole collection — when running
+an exwiw config against a MongoDB backup. The headline finding: the slowness is a
+**symptom of a scoping bug**, not of serialization/decode cost.
+
+## Reproduction setup
+
+- Backup: serve a raw WiredTiger dbpath with a standalone `mongod` (the same
+  `mongo:7` image the repo's `compose.yml` uses) on a spare port:
+
+  ```
+  docker run -d --name exwiw-restore-mongo --user 0:0 --entrypoint mongod \
+    -p 27018:27017 -v "<backup-dbpath>:/data/db" \
+    mongo:7 --dbpath /data/db --bind_ip_all
+  ```
+
+  Notes: run as root (`--user 0:0`) so mongod can read a backup's `0600` files;
+  bypass the image entrypoint (`--entrypoint mongod`) so it does not `gosu`-drop
+  back to the `mongodb` user. A backup carrying a `WiredTiger.backup` marker runs
+  recovery on the first start and **writes into the backup dir** (expected for a
+  restore). Starting a standalone from a replica-set backup yields a harmless
+  `system.replset` warning. Local DB connections may be blocked by a dev sandbox —
+  run measurement commands with the sandbox disabled.
+
+- Run: `bundle exec exwiw export --config <app>/exwiw/exwiw.yml
+  --adapter=mongodb --host=localhost --port=27018 --database=<database>
+  --ids=<target-id> --output-dir=… --log-level=debug`.
+  (The optional-argument CLI flags — `--ids`, `--output-dir` — must use `=`; the
+  space form passes `nil` and crashes in the option callback.)
+
+## Baseline measurement (worked example, warm cache)
+
+Full run ≈ **69s** over a few hundred non-empty collections. Per-collection wall
+time (gap between consecutive `Processing table` log markers) was dominated by a
+single collection:
+
+| collection | time | records |
+|---|---|---|
+| **items** | **41.0s (59% of the whole run)** | 18,739 |
+| (other collections) | ~5s and below each | — |
+
+A "full run ≈ 10 min" figure is the **cold-cache** version (first read pages the
+`items` data off the backup over the bind mount). The *relative* shape (items
+dominates) is the same warm or cold.
+
+## Root cause: nullable belongs_to FK used as a hard `$in` AND constraint
+
+The scope is tiny: one parent entity (`<target-id>`) → **1 store** (linked by
+`business_entity_id` = the entity's **`uuid`**, correctly configured with
+`references: uuid`) → **127 items** (`{store_id: <that store>}`, indexed, ~2ms).
+
+But the run dumped **18,739** items, not 127, and scanned the whole collection.
+Why:
+
+1. `MongodbAdapter#related_collection_filter` ANDs **every** belongs_to whose
+   parent produced ids. The `stores` filter became:
+
+   ```
+   { user_id:             {$in: [580 ids]},
+     deleted_user_id:     {$in: [96 ids]},     # nullable FK
+     business_entity_id:  {$in: ["<target-id>"]} }
+   ```
+
+   The one matching store has **`deleted_user_id` absent (null)**, so it can never
+   satisfy `deleted_user_id ∈ {96 ids}`. The AND yields **0 stores** → `stores`
+   logs "No records matched. skip this table."
+
+2. With `stores` empty, `@state["stores"]` carries no ids, so when `items` is
+   built its `store_id` belongs_to contributes nothing. The remaining belongs_tos
+   are to **reference/master data that is dumped in full** —
+   `large_categories` and `medium_categories`. So the items filter degenerated to:
+
+   ```
+   { large_category_id:   {$in: [98 ids]},
+     medium_category_id:  {$in: [846 ids]} }
+   ```
+
+3. `items` has **no index on `large_category_id` / `medium_category_id`** (it does
+   have `store_id_1`). So this filter forces a **full COLLSCAN of all 2.43M items**
+   — and the Runner scans it **twice**: once for `StreamingResult#size`
+   (`count_documents`) and again for the fetch.
+
+Isolated phase breakdown of the degenerate items query (warm): `count_documents`
+3.58s + fetch/decode 1.47s + serialize 0.26s. Serialization (the native
+`Exwiw::ExtJson` C ext) is **not** the bottleneck — the COLLSCAN is, and it is far
+worse cold.
+
+The same nullable-FK problem applies one level down: the store's 127 items
+themselves have `*_category_id` = null, so even `{store_id, large_category,
+medium_category}` ANDed returns **0**. The only filter that yields the correct 127
+is `{store_id: {$in: [store]}}` **alone**.
+
+## Implemented fix: genuine-anchor scoping (MongodbAdapter#related_collection_filter)
+
+Scope flows from the dump target along belongs_to edges. The fix classifies each
+belongs_to parent of a non-target collection by whether it is **genuinely scoped**
+— reachable back to the dump target through belongs_to chains
+(`#genuine_scope_set`, a fixpoint over the configs) — and applies the constraint
+accordingly:
+
+- **Anchor (strict).** Among the genuine parents, the most selective one (fewest
+  captured ids) is applied strictly (`{fk: {$in: [...]}}`). It carries the real
+  narrowing and, being strict, bounds the result to a small set — which keeps both
+  this query and the `$in` sets it feeds downstream from ballooning.
+- **Other genuine parents (null-aware).** `{fk: {$in: [nil, ...]}}` (Mongo's
+  `$in: [nil]` matches both explicit nulls and missing fields), so a row whose
+  nullable refinement FK is null is not excluded by it.
+- **Reference parents (dropped).** A parent NOT reachable to the dump target is
+  reference/master data dumped in full; its id set is "all/most of a table" and is
+  not a real scope, so when a genuine anchor exists it is dropped entirely.
+- **No genuine parent:** fall back to the historical strict-AND of whatever
+  constraints exist (preserves prior behaviour for unreachable collections).
+
+For this extraction: `stores` → `{business_entity_id ∈ {<target-id>}}` (anchor;
+`user_id`/`deleted_user_id` → reference leaks, dropped) → **1 store**; `items` →
+`{store_id ∈ {store}}` strict anchor with the nullable refinement FKs null-aware
+and the `*_category` references dropped → **127** via the `store_id_1` index.
+
+### Measured result (warm, same cache)
+
+Full run **58.8s → 11.0s ≈ 5.4×**; `items` 41s double COLLSCAN → ~11ms indexed
+(≈3700×). Correctness also fixed: `stores` 0→1, `items` 18,739 (leaked COLLSCAN)
+→127. Byte-identical existing snapshots (the seed graph is fully genuine and has no
+null FKs, so anchor-strict + null-aware ≡ the prior strict-AND).
+
+### Approaches considered and rejected
+
+- **Unconditional null-aware on every belongs_to** (the original iter-1 direction):
+  catastrophic. A collection that belongs_to only reference data dumped in full
+  becomes ~the whole table once null-aware; the resulting child `$in` then exceeds
+  Mongo's **48 MB max message size** and the run crashes. Null-awareness must NOT
+  be applied to a collection's only/anchor scope.
+- **Null-aware on all genuine parents (no anchor distinction):** makes the genuine
+  *anchor* itself null-aware too — `stores` then matched every store with a null
+  `business_entity_id` (a not-fully-backfilled column) → hundreds of thousands of
+  stores → a ~39 MB child filter on a downstream collection (**MaxBSONSize**).
+  Hence the anchor stays strict.
+- **Scope by the single most-selective genuine parent alone (drop other genuine):**
+  fast and correct here, but drops legitimate AND-narrowing for multi-parent
+  collections (e.g. `order_items` ∈ orders AND products) and moves seed snapshots.
+- Pure-performance tweaks that keep the (incorrect) 18,739-row output —
+  `--cursor-parallel` (changes row order, treats the symptom) or skipping the
+  redundant `count_documents` scan (~½ only) — were rejected as the primary fix.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -123,7 +123,7 @@ module Exwiw
               { config.primary_key => { "$in" => coerce_ids(dump_target.ids) } }
             end
           else
-            related_collection_filter(config, config_by_name)
+            related_collection_filter(config, config_by_name, dump_target)
           end
 
         Exwiw::MongoQuery::Find.new(
@@ -352,23 +352,74 @@ module Exwiw
       # the values were captured from that field in #execute, so their BSON type
       # already matches the stored FK — no coercion.
       #
-      # A belongs_to whose parent produced no ids contributes no constraint:
-      # either the parent matched nothing, or it is not dumped here (e.g. an
-      # embedded collection, or one excluded from the run). If that leaves the
-      # filter empty even though the collection HAS belongs_to, the collection
-      # cannot be scoped from the dump target — and falling back to an empty `{}`
-      # filter would scan and dump the ENTIRE collection across every scope. That
-      # is never what a scoped extraction wants, so constrain it to match nothing
-      # and warn instead. (A collection with no belongs_to at all is genuine
-      # reference/master data and is still dumped in full via `{}`.)
-      private def related_collection_filter(config, config_by_name)
-        filter = config.belongs_tos.each_with_object({}) do |relation, acc|
+      # Scope flows from the dump target along belongs_to edges. A belongs_to is
+      # classified by whether its parent is *genuinely scoped* — reachable back to
+      # the dump target through belongs_to chains (see #genuine_scope_set) — which
+      # determines how its constraint is applied:
+      #
+      # - Among the genuine parents, the most selective one (fewest captured ids)
+      #   is the ANCHOR and is applied strictly. It carries the real narrowing and,
+      #   being strict, bounds the result to a small set — which keeps both this
+      #   query and the `$in` sets it feeds downstream from ballooning.
+      #
+      # - The OTHER genuine parents are applied null-aware: a row whose (nullable)
+      #   FK is null/absent has no reference through that relation and must not be
+      #   excluded by it. `nil` is added to the `$in` set (Mongo's `$in: [nil]`
+      #   matches both explicit nulls and missing fields). Without this, a nullable
+      #   genuine FK that is null on otherwise in-scope rows ANDs the result to
+      #   empty — dropping legitimate rows, and (when it zeroes a parent) making
+      #   children lose that parent's selective+indexed scope and degenerate to a
+      #   full COLLSCAN. See docs/mongodb-scoping-fullscan-notes.md. Null-aware is
+      #   applied to non-anchor parents only: making the sole/anchor scope itself
+      #   null-aware would match every row whose FK is null (e.g. a not-yet-
+      #   backfilled column), ballooning the result instead of scoping it.
+      #
+      # - Reference parents (NOT reachable to the dump target — master/reference
+      #   data dumped in full, or only reachable via such data) produce a non-
+      #   scoping id set: "all/most of a reference table", which neither narrows
+      #   meaningfully nor, made null-aware, stays bounded. So when the collection
+      #   has a genuine parent to anchor on, reference-parent constraints are
+      #   dropped entirely.
+      #
+      # When NO genuine parent produced ids, the collection is not reachable from
+      # the dump target; fall back to the historical strict-AND of whatever
+      # constraints exist (bounded, preserves prior behavior).
+      #
+      # A belongs_to whose parent produced no ids contributes no constraint: either
+      # the parent matched nothing, or it is not dumped here (e.g. an embedded
+      # collection, or one excluded from the run). If that leaves the filter empty
+      # even though the collection HAS belongs_to, the collection cannot be scoped
+      # from the dump target — and an empty `{}` filter would scan and dump the
+      # ENTIRE collection across every scope. That is never what a scoped
+      # extraction wants, so constrain it to match nothing and warn instead. (A
+      # collection with no belongs_to at all is genuine reference/master data and
+      # is still dumped in full via `{}`.)
+      private def related_collection_filter(config, config_by_name, dump_target)
+        genuine = genuine_scope_set(config_by_name, dump_target.table_name)
+
+        genuine_clauses = []
+        reference_clauses = []
+        config.belongs_tos.each do |relation|
           values = parent_state_for(relation, config_by_name)
           next if values.nil? || values.empty?
 
-          acc[relation.foreign_key] = { "$in" => values }
+          target = genuine.include?(relation.table_name) ? genuine_clauses : reference_clauses
+          target << [relation.foreign_key, values]
         end
 
+        filter =
+          if genuine_clauses.any?
+            anchor_index = (0...genuine_clauses.size).min_by { |i| genuine_clauses[i][1].size }
+            genuine_clauses.each_with_index.each_with_object({}) do |((foreign_key, values), index), acc|
+              acc[foreign_key] =
+                index == anchor_index ? { "$in" => values } : { "$in" => [nil] + values }
+            end
+          else
+            reference_clauses.each_with_object({}) do |(foreign_key, values), acc|
+              acc[foreign_key] = { "$in" => values }
+            end
+          end
+
         return filter unless filter.empty? && config.belongs_tos.any?
 
         @logger.warn(
@@ -379,6 +430,31 @@ module Exwiw
         { config.primary_key => { "$in" => [] } }
       end
 
+      # The set of collection names *genuinely scoped* by the dump target: the
+      # target itself, plus every collection that can reach it by following
+      # belongs_to edges (child -> parent) transitively. Computed by fixpoint over
+      # the configs. Everything outside this set is reference/master data (or only
+      # reachable through it) whose belongs_to id sets do not represent a real
+      # scope. Memoized per target name; the configs do not mutate mid-run.
+      private def genuine_scope_set(config_by_name, target_name)
+        (@genuine_scope_set_cache ||= {})[target_name] ||=
+          begin
+            reachable = Set.new([target_name])
+            loop do
+              added = false
+              config_by_name.each_value do |cfg|
+                next if cfg.embedded? || reachable.include?(cfg.name)
+                next unless cfg.belongs_tos.any? { |relation| reachable.include?(relation.table_name) }
+
+                reachable << cfg.name
+                added = true
+              end
+              break unless added
+            end
+            reachable
+          end
+      end
+
       # The captured parent-collection values a child belongs_to should be
       # constrained by: the values of the parent field the FK references
       # (`relation.references`, default the parent primary_key). nil when the

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.6.0"
+  VERSION = "0.6.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.6.1
 platform: ruby
 authors:
 - Shia
@@ -36,6 +36,7 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- docs/mongodb-scoping-fullscan-notes.md
 - docs/optimization-notes.md
 - docs/optimize-mongodb-export-with-native-ext.md
 - docs/plans/2026-05-15-insert-000-schema-file.md