exwiw 0.9.7 → 0.9.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 70b1d51562a341ea502ebb5d22ed3aa625d902264d9112bf338aa6142268dd0a
-  data.tar.gz: e1110a13b9d95cddec6d32f0f829360ded75742f4d17c6969a4ebf51d272b454
+  metadata.gz: 8b734fe407ce3c430c3197b04db275df36640849f6795d7636614d8fab0c6574
+  data.tar.gz: 4be4a94e18f929ef13c97072248e4c7b7e867ad80160841a81c0146a929b9e21
 SHA512:
-  metadata.gz: acd60932985992f2863640102a34d79e87434b566ee96484abbf3a36f3b2d08e7700a13a63657f9f45cd97c8f0f76d83f221b13e23ae3b0b7b257500f9f99890
-  data.tar.gz: 0de13c9385276a68b8ceb582777da4c9f6b329d1f796db5dfc1411ea936843e0aff0f6a5d178c1a883230b7f0bc7378daedd5382ac9dcfc935879ad74df924f9
+  metadata.gz: cdb946b9f0b3ae2bfca4aa74aca93454cbc0ae6f332f0900216c3a35be9d8bdcef2b2082744bcc6262b6bc9cf72afae9a477c85f7e00d60da8d6eaa4ed55b825
+  data.tar.gz: 12f63468a2d834e558b98e73937e739aa2ab4581cf83699e0a675e280cfd6e6f158d136156400e3ec7c7672beaa63189a6e90c727cc1c87281412e061c72b417

data/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 ## [Unreleased]
 
+## [0.9.9] - 2026-07-09
+
+### Changed
+
+- **`replace_with_fake_data` name types now produce a coherent identity, and gained kana types.** The person-family types (`human_name`, `first_name`, `last_name`) previously drew from three independent faker pools, so a single seed's first name, last name, and full name were unrelated people. They now share one per-locale pool of whole people keyed by the seed, so for a given seed `human_name` always equals `last_name` + `first_name` (ordered per locale: `姓 名` for `ja`, `First Last` otherwise). Three kana types are added — `human_name_kana`, `first_name_kana`, `last_name_kana` — for the reading of the same person. Because faker's `ja` locale has kanji names with no readings, exwiw bundles its own paired (kanji, katakana) dataset (`Exwiw::JapaneseNames`) and builds the `ja` name pool from it, so kana matches the kanji and a `ja`-name-only config needs no faker at all. Kana types require `locale: ja` (they raise a clear error otherwise). The person pool is sized independently of the other types at 20,000 (`PERSON_POOL_SIZE`), and for `ja` is filled with 20,000 *distinct* people enumerated from a 142×142 name dataset (rather than sampled with replacement, which left ~37% of slots duplicated), so the name space is large and evenly used. Determinism for the `ja` name family is now pinned to exwiw's bundled dataset version rather than the faker version; other types and locales are unchanged.
+
+## [0.9.8] - 2026-07-09
+
+### Added
+
+- **The MongoDB adapter now supports `reverse_scope` (multi-referencer reverse scoping).** The `reverse_scope.via` key that `TableConfig` already offers for the SQL adapters is now accepted by `MongodbCollectionConfig` with the same shape and semantics: a global-identity collection with no `belongs_to` path to the dump target, referenced by several scoped collections, is constrained to the union of the ids its enumerated referencer arms actually point at instead of being dumped in full. Where SQL emits a `UNION` subquery, the mongodb adapter captures each arm's foreign-key values **at runtime** while the referencer collection streams (the existing parent-id propagation mechanism, extended to via-arm columns; `null`s are dropped per arm, array-valued columns contribute one id per element, and captured values keep their native BSON types). Because of that runtime capture, `DetermineTableProcessingOrder` gains a mongodb-only mode that orders a reverse-scoped collection **after** all of its `via` referencers (inverting an arm's own `belongs_to` edge back to the collection) and aborts with a clear error when the arms create a genuine ordering cycle; the SQL processing order is unchanged. Arm hygiene mirrors SQL (an unknown / embedded / undumped / unscoped referencer is skipped with a warning rather than silently widening the dump), a collection with its own `belongs_to` path to the target keeps that scope (`reverse_scope` is ignored, as in SQL), satellites of the reverse-scoped collection tighten automatically, `exwiw explain` shows the real filter shape with a placeholder id, and `--parallel-workers` falls back to serial with a warning when a reverse-scoped collection is present (the parallel schedule does not express the new ordering constraint yet). Masking/fields behavior is unchanged, and the key stays user-owned: never emitted by the schema generator and preserved across regeneration.
+
+### Changed
+
+- **Unknown keys in schema config JSON are now rejected on load instead of being silently dropped.** Serdes deserialization is lenient, so a key that matched no declared attribute was discarded without a word — turning a typo (`reverse_scop`, `bulk_insert_chunk_sise`) or a key another adapter supports but this one does not (`raw_sql`/`map` on a MongoDB field; `reverse_scope` on a MongoDB collection before this release) into a silent no-op: the dump ran and the requested masking/scoping simply never happened. `TableConfig.from` / `MongodbCollectionConfig.from` now validate the raw hash against the declared attributes — including the nested `belongs_tos` / `columns` / `fields` / `reverse_scope` / `embedded_in` / `replace_with_fake_data` entries — and raise `Exwiw::UnknownConfigKeyError` (an `ArgumentError` subclass) naming the key(s), the table/collection, the nested position, and the allowed keys; `export`/`explain` prepend the offending file path. This is a deliberate hard error with no opt-out: every declared key still passes — including the documentation-only `comment` on table/collection configs and their `belongs_tos`/`columns`/`fields` entries, which remains the supported place for free-form notes — so a config that only uses supported keys is unaffected, while anything now rejected was already being ignored. If a config carries stray keys, remove them or fold them into `comment`.
+
 ## [0.9.7] - 2026-07-08
 
 ### Added

data/README.md

@@ -451,12 +451,12 @@ This is an example of the one table schema:
     "primary_key": "id",
     "filter": "users.id > 0",
     "bulk_insert_chunk_size": 1000,
-    "belongs_to": [{
-        "name": "companies",
+    "belongs_tos": [{
+        "table_name": "companies",
         "foreign_key": "company_id"
     }],
     "columns": [{
-        "name": "id",
+        "name": "id"
     }, {
         "name": "email",
         "replace_with": "user{id}@example.com"
@@ -468,6 +468,12 @@ This is an example of the one table schema:
 
 `--schema-dir` will use all json files in the specified directory.
 
+#### Unknown keys are rejected
+
+Loading a table/collection config with a key that no declared attribute accepts is an **error** (`Exwiw::UnknownConfigKeyError`, an `ArgumentError` subclass) naming the key, the table/collection, the offending file, and the allowed keys. This also applies to the nested `belongs_tos` / `columns` / `fields` / `reverse_scope` / `embedded_in` / `replace_with_fake_data` entries. Previously such keys were silently dropped, which turned a typo (`reverse_scop`) — or a key another adapter supports but this one does not (e.g. `raw_sql` on a MongoDB field) — into a silent no-op: the config loaded, the dump ran, and the requested masking/scoping simply never happened.
+
+For free-form annotations, use the `comment` key — it is a declared, documentation-only attribute on table/collection configs and on their `belongs_tos` / `columns` / `fields` entries, so it always passes (see [Ignore / annotate a column or `belongs_to`](#ignore--annotate-a-column-or-belongs_to)).
+
 ### Output format
 
 By default, exwiw generates `INSERT` statements. For PostgreSQL, you can pass `--output-format=copy` to generate `COPY FROM stdin` format instead, which is significantly faster for bulk loading.
@@ -680,6 +686,7 @@ Notes:
 - **NULLs are excluded** per arm (`IS NOT NULL`).
 - **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`. The cascade is **multi-hop**, so a table several `belongs_to` hops below the reverse-scoped table (e.g. `end_user_profiles → end_users → users`) also tightens automatically, with no config of its own.
 - Works in both single-target and scope-column mode. In single-target mode there is no scope-column pre-flight (`validate_scope!`), so a satellite the cascade cannot resolve to a single scopable parent (e.g. it `belongs_to` two scopable hubs) is dumped in full with a warning rather than aborting. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
+- **The MongoDB adapter supports `reverse_scope` too** — same config shape and semantics, but the id set is captured at runtime instead of being emitted as a `UNION` subquery. See [`reverse_scope` on collections](#reverse_scope-on-collections) under MongoDB notes.
 
 ### Why a JOIN, not `IN (subquery)`
 
@@ -806,9 +813,10 @@ which is equivalent to `"replace_with": "user{id}@example.com"`.
   automatic NULL preservation** — the proc receives `nil` and decides.
 - `map` is exclusive with the other masking keys on the same column
   (`raw_sql` / `replace_with` / `replace_with_fake_data`).
-- SQL adapters only. The MongoDB adapter silently drops the key (like
-  `raw_sql`). Because the transform runs in the exwiw process, it is invisible
-  to `explain`.
+- SQL adapters only. On the MongoDB adapter the key is rejected on load, like
+  `raw_sql` (see [Unknown keys are rejected](#unknown-keys-are-rejected)).
+  Because the transform runs in the exwiw process, it is invisible to
+  `explain`.
 
 **Security note**: `map` executes arbitrary Ruby from the schema config. Treat
 config files with the same trust as your Gemfile — only load trusted configs.
@@ -842,36 +850,61 @@ fake value, across tables, runs, and adapters:
   because their text forms differ per adapter. A `NULL` seed value hashes `""`
   (still deterministic).
 - Like `replace_with`, it **preserves NULL** in the target column.
-- `locale` (optional) sets the faker locale used to generate the candidate
-  values, e.g. `{ "seed": "id", "type": "human_name", "locale": "ja" }` produces
-  Japanese names.
+- `locale` (optional) sets the locale used to build the candidate values, e.g.
+  `{ "seed": "id", "type": "human_name", "locale": "ja" }` produces Japanese
+  names.
 - Supported `type`s:
 
-  | type | example output |
-  |------|----------------|
-  | `human_name` | `Adrianna Kilback` |
-  | `first_name` | `Adrianna` |
-  | `last_name` | `Kilback` |
-  | `phone_number` | `(555) 123-4567` |
-  | `address` | `282 Kevin Brook, Imogeneborough, CA 58517` |
-  | `company_name` | `Hirthe-Ritchie` |
-  | `email` | `cliff.fay.9d6b804eff5a3f57@example.com` |
-  | `username` | `cliff.fay_9d6b804eff5a3f57` |
-
-- Values are drawn from a pool of 10,000 pre-generated candidates per
-  (type, locale), so distinct seeds can share a fake value. The
+  | type | example output (en) | example output (`locale: ja`) |
+  |------|----------------|----------------|
+  | `human_name` | `Adrianna Kilback` | `山田 太郎` |
+  | `first_name` | `Adrianna` | `太郎` |
+  | `last_name` | `Kilback` | `山田` |
+  | `human_name_kana` | — (ja only) | `ヤマダ タロウ` |
+  | `first_name_kana` | — (ja only) | `タロウ` |
+  | `last_name_kana` | — (ja only) | `ヤマダ` |
+  | `phone_number` | `(555) 123-4567` | |
+  | `address` | `282 Kevin Brook, Imogeneborough, CA 58517` | |
+  | `company_name` | `Hirthe-Ritchie` | |
+  | `email` | `cliff.fay.9d6b804eff5a3f57@example.com` | |
+  | `username` | `cliff.fay_9d6b804eff5a3f57` | |
+
+- **Coherent identity across the name family.** The person-family types
+  (`human_name`, `first_name`, `last_name` and their `*_kana` counterparts) all
+  draw from a single shared pool of people per locale, keyed by the same seed —
+  so for one seed value the last name, first name, full name, and every kana
+  reading belong to the **same person**: `human_name` always equals
+  `last_name` + `first_name`, and `human_name_kana` matches `human_name`. Full
+  names are ordered per locale (`姓 名` for `ja`, `First Last` otherwise).
+- **Kana (`*_kana`) types require `locale: ja`.** faker's `ja` locale ships
+  kanji names with no reading, so exwiw bundles its own paired (kanji, katakana)
+  dataset for `ja`; this is what lets a fake person carry a kana reading that
+  actually matches its kanji. Requesting a `*_kana` type with any other locale
+  raises a clear error at build time.
+- Values are drawn from a pre-generated pool per (type, locale), so distinct
+  seeds can share a fake value. The name family uses one shared **person** pool
+  of 20,000 identities — for `ja` these are 20,000 *distinct* people enumerated
+  from the bundled (kanji, kana) dataset (142 surnames × 142 given names); the
+  other types use an independent 10,000-candidate pool. The
   uniqueness-sensitive types (`email`, `username`) additionally embed a 64-bit
   hex token derived from the seed hash, so they stay collision-free under a
   unique index even at millions of rows (collision probability at 5M distinct
   seeds ≈ 7e-7) and always use the `example.com` domain.
-- **Determinism caveat**: values are stable for a given faker gem version +
-  locale. Upgrading faker (or changing `locale`) regenerates the pool and maps
+- **Determinism caveat**: values are stable for a given locale plus the version
+  of the value source — the faker gem for the non-`ja` name family and the
+  independent types, and exwiw's bundled dataset for the `ja` name family.
+  Upgrading that source (or changing `locale`) regenerates the pool and maps
   seeds to different values. The seed→value mapping itself never changes within
   one version.
 - The faker gem is **not** a runtime dependency of exwiw — add `gem "faker"` to
-  your Gemfile to use this mode (exwiw raises a clear error otherwise).
+  your Gemfile to use this mode (exwiw raises a clear error otherwise). A config
+  that uses **only** `ja` person types needs no faker (that pool is built
+  entirely from the bundled dataset); faker is required for every other type
+  and locale.
 - Exclusive with the other masking keys on the same column. SQL adapters only
-  (the MongoDB adapter silently drops the key), and invisible to `explain`.
+  (the MongoDB adapter rejects the key on load, see
+  [Unknown keys are rejected](#unknown-keys-are-rejected)), and invisible to
+  `explain`.
 
 **Performance**: this is a per-row Ruby transform, measured at ~1.5–1.6µs/row
 per fake column (so ≈ +8s per 5M rows per column; ~+40% against a local sqlite
@@ -906,9 +939,39 @@ The MongoDB adapter is experimental. To use it:
   mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
   ```
 - Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
-- `raw_sql`, `map`, and `replace_with_fake_data` are not supported (the `MongodbField` schema does not declare them; such keys in scenario JSON are silently dropped on load). Use `replace_with` for masking.
+- `raw_sql`, `map`, and `replace_with_fake_data` are not supported (the `MongodbField` schema does not declare them; such keys in a config are rejected on load — see [Unknown keys are rejected](#unknown-keys-are-rejected)). Use `replace_with` for masking.
 - The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
 
+#### `reverse_scope` on collections
+
+[Multi-referencer reverse scoping](#reverse-scope-for-multi-referencer-tables-reverse_scope) works on `MongodbCollectionConfig` with the same config shape and the same semantics as the SQL adapters — a global-identity collection (say `accounts`) with no `belongs_to` path to the dump target, but referenced by several scoped collections, is constrained to the union of the ids those referencers actually point at instead of being dumped in full:
+
+```json
+{
+  "name": "accounts",
+  "primary_key": "_id",
+  "reverse_scope": {
+    "via": [
+      { "table": "articles", "column": "author_account_id" },
+      { "table": "invitations", "column": "invitee_account_id" }
+    ]
+  },
+  "belongs_tos": [],
+  "fields": [{ "name": "_id" }, { "name": "name" }]
+}
+```
+
+Where the SQL adapters emit a `UNION` subquery, MongoDB has no cross-collection joins, so the adapter captures each arm's column values **at runtime** while the referencer collection streams (the same mechanism that already propagates parent ids to children), then filters the reverse-scoped collection with `{"_id": {"$in": [<union of captured ids>]}}`. Consequences of that runtime capture:
+
+- **Processing order**: a reverse-scoped collection is dumped **after** all of its `via` referencers (an arm's own `belongs_to` back to the reverse-scoped collection is inverted rather than kept — the declaration states ids flow referencer → collection). If the arms form a genuine ordering cycle with the `belongs_to` graph, the export aborts with an error naming the cycle members. SQL processing order is unchanged (its INSERT output must stay loadable in foreign-key order).
+- **Arm hygiene mirrors SQL**: an arm whose referencer is unknown, embedded, not dumped, or itself unscoped (no path to the dump target and no `reverse_scope` of its own) is **skipped with a warning** — an unscoped referencer's ids span every scope and would silently widen the dump. Per-arm `null`/absent foreign keys are dropped (the SQL `IS NOT NULL`), an array-valued foreign-key column contributes one id per element, and captured values keep their native BSON types (an `ObjectId` foreign key matches an `ObjectId` `_id` with no coercion).
+- **Precedence mirrors SQL**: a collection with its own `belongs_to` path to the dump target is scoped by that path; a `reverse_scope` declared on it is ignored.
+- **Satellites need no config**, as in SQL: a collection that `belongs_to` the reverse-scoped collection tightens to the kept ids automatically through the ordinary captured-parent-id mechanism.
+- **`--parallel-workers` falls back to serial** (with a warning) when any collection declares `reverse_scope` — the parallel schedule does not express the referencers-first ordering constraint yet.
+- **`exwiw explain`** shows the real `{"_id": {"$in": [...]}}` filter shape with a placeholder id, like the other runtime-captured scopes.
+
+Masking (`replace_with`) and `fields` behavior on a reverse-scoped collection are unchanged. Like the SQL key, `reverse_scope` is user-owned: `exwiw:mongoid:schema:generate` never emits it and regeneration preserves a hand-added value.
+
 #### Embedded documents
 
 MongoDB models often store one-to-many relationships as embedded subdocument arrays (e.g. `users` documents with a `posts: [...]` field). To mask fields inside embedded subdocuments, declare a separate config with `embedded_in`:

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -413,18 +413,34 @@ module Exwiw
       # The distinct set of this collection's fields that downstream children
       # constrain on (each child belongs_to's `references`, defaulting to this
       # collection's primary_key), with primary_key always included so the
-      # historical primary-key-keyed propagation keeps working.
+      # historical primary-key-keyed propagation keeps working. A collection
+      # named as a `reverse_scope.via` referencer additionally captures the
+      # arm's foreign-key column, so the reverse-scoped collection (processed
+      # later) can constrain itself to the ids this collection points at.
       private def propagation_keys_for(config, config_by_name)
         referenced = config_by_name.each_value.flat_map do |child|
           next [] if child.embedded?
 
-          child.belongs_tos
+          keys = child.belongs_tos
             .select { |relation| relation.table_name == config.name }
             .map { |relation| relation.references || config.primary_key }
+
+          reverse_scope_arms_of(child).each do |via|
+            keys << via.column if via.table == config.name
+          end
+
+          keys
         end
         ([config.primary_key] + referenced).uniq
       end
 
+      # The `reverse_scope.via` arms of `config`, or [] when it declares none.
+      private def reverse_scope_arms_of(config)
+        return [] unless config.respond_to?(:reverse_scope)
+
+        config.reverse_scope&.via || []
+      end
+
       # Build the scoping filter for a non-target collection from its belongs_to
       # parents' captured ids. Each belongs_to is constrained by the parent field
       # the FK references (`relation.references`, default the parent primary_key);
@@ -476,6 +492,24 @@ module Exwiw
       private def related_collection_filter(config, config_by_name, dump_target)
         genuine = genuine_scope_set(config_by_name, dump_target.table_name)
 
+        # Opt-in multi-referencer reverse scope (MongodbCollectionConfig
+        # #reverse_scope), mirroring the SQL adapters' semantics: it applies only
+        # to a collection with no belongs_to path to the dump target (exactly
+        # when SQL's reverse extraction is attempted — a genuinely scoped
+        # collection keeps its belongs_to scope and reverse_scope is ignored),
+        # and when it applies it IS the scope filter, replacing the
+        # reference-parent strict-AND fallback. When no arm survives, fall
+        # through to the historical behavior, as SQL falls back to dump-all.
+        if !genuine.include?(config.name) && reverse_scope_arms_of(config).any?
+          filter = reverse_scope_filter(config, config_by_name, genuine)
+          return filter unless filter.nil?
+        elsif genuine.include?(config.name) && reverse_scope_arms_of(config).any?
+          @logger.debug(
+            "  Collection '#{config.name}' declares reverse_scope but is genuinely scoped " \
+            "via belongs_to; reverse_scope is ignored (same precedence as the SQL adapters)."
+          )
+        end
+
         genuine_clauses = []
         reference_clauses = []
         config.belongs_tos.each do |relation|
@@ -509,6 +543,88 @@ module Exwiw
         { config.primary_key => { "$in" => [] } }
       end
 
+      # Build the `pk $in <union of referenced ids>` filter for a reverse-scoped
+      # collection (MongodbCollectionConfig#reverse_scope). Each `via` arm names
+      # a referencer collection and the foreign-key column on it that points at
+      # this collection's primary key; the referencer was dumped earlier under
+      # its own scope (DetermineTableProcessingOrder orders every arm before the
+      # reverse-scoped collection) and #execute captured the arm column's values
+      # into @state, so the union here holds only in-scope ids. Values are used
+      # exactly as captured (native BSON — ObjectId FKs stay ObjectIds, string
+      # FKs stay strings), array-valued columns are flattened (one referenced id
+      # per element), and nil/absent FKs are dropped, mirroring the SQL arms'
+      # `IS NOT NULL`.
+      #
+      # An arm is skipped with a warning when its referencer is unknown,
+      # embedded, produced no captured state (not dumped — e.g. ignore:true), or
+      # is itself unscoped (neither genuinely scoped nor reverse-scoped): an
+      # unscoped referencer's captured ids span every scope and would silently
+      # widen the dump, exactly the case the SQL adapters skip. Returns nil when
+      # no arm survives, letting the caller fall through to the historical
+      # behavior (SQL parity: dump-all fallback).
+      #
+      # In `explain` placeholder mode there is no captured state; a placeholder
+      # id keeps the real filter shape (`pk $in [...]`) so index selection is
+      # reported correctly.
+      private def reverse_scope_filter(config, config_by_name, genuine)
+        ids = []
+        any_arm = false
+
+        reverse_scope_arms_of(config).each do |via|
+          referencer = config_by_name[via.table]
+          if referencer.nil? || referencer.embedded?
+            @logger.warn(
+              "  #{config.name}.reverse_scope references #{referencer.nil? ? 'unknown' : 'embedded'} " \
+              "collection '#{via.table}'; skipping arm."
+            )
+            next
+          end
+
+          unless genuine.include?(via.table) || reverse_scope_arms_of(referencer).any?
+            @logger.warn(
+              "  #{config.name}.reverse_scope arm '#{via.table}.#{via.column}' is not scoped; " \
+              "skipping it (an unscoped arm would union ids from every scope back). " \
+              "Make '#{via.table}' scopable or remove it from reverse_scope.via."
+            )
+            next
+          end
+
+          if @explain_placeholder
+            any_arm = true
+            ids << explain_placeholder_id
+            next
+          end
+
+          captured = @state[via.table]
+          if captured.nil?
+            @logger.warn(
+              "  #{config.name}.reverse_scope arm '#{via.table}.#{via.column}' has no captured " \
+              "state ('#{via.table}' was not dumped before '#{config.name}'); skipping arm."
+            )
+            next
+          end
+
+          any_arm = true
+          values = captured[via.column]
+          if values.nil?
+            @logger.warn(
+              "  #{config.name}.reverse_scope arm column '#{via.table}.#{via.column}' was not " \
+              "captured while dumping '#{via.table}'; treating the arm as empty."
+            )
+            next
+          end
+
+          ids.concat(values)
+        end
+
+        return nil unless any_arm
+
+        ids = ids.flat_map { |value| value.is_a?(Array) ? value : [value] }
+        ids.compact!
+        ids.uniq!
+        { config.primary_key => { "$in" => ids } }
+      end
+
       # The set of collection names *genuinely scoped* by the dump target: the
       # target itself, plus every collection that can reach it by following
       # belongs_to edges (child -> parent) transitively. Computed by fixpoint over

data/lib/exwiw/determine_table_processing_order.rb

@@ -8,8 +8,18 @@ module Exwiw
 
     # @param tables [Array<Exwiw::TableConfig>] tables
     # @param logger [Logger, nil] receives a warning when a cycle has to be broken
+    # @param runtime_reverse_scope [Boolean] when true (mongodb), a table
+    #   declaring `reverse_scope` is ordered AFTER its `via` referencer tables:
+    #   the adapter builds the reverse filter from ids captured at runtime while
+    #   the referencers were dumped, so they must be processed first. Each arm's
+    #   own belongs_to edge back to the reverse-scoped table (the usual
+    #   `referencer.fk -> hub` relation) is inverted rather than kept, since the
+    #   declaration states ids flow referencer -> hub. False (the default)
+    #   preserves the historical belongs_to-only ordering — the SQL adapters
+    #   scope via subqueries and need the hub emitted before its referencers so
+    #   the INSERT output stays loadable in foreign-key order.
     # @return [Array<String>] sorted table names
-    def run(tables, logger: nil)
+    def run(tables, logger: nil, runtime_reverse_scope: false)
       return tables.map(&:name) if tables.size < 2
 
       ordered_table_names = []
@@ -19,31 +29,45 @@ module Exwiw
         acc[table.name] = table
       end
 
-      # Only belongs_to relations whose target is also in this run constrain the
-      # order. A belongs_to pointing at a table that is not being processed here
-      # — e.g. an embedded MongoDB collection (masked through its parent, never
-      # dumped on its own) or any table excluded from the run — is not something
-      # we can or need to order against, so it must never block resolution.
-      # Without this, such a dependency would stay unresolved forever and
-      # masquerade as a circular dependency, freezing every table that
-      # (transitively) references it.
+      reverse_scope_deps = runtime_reverse_scope ? compute_reverse_scope_dependencies(tables) : {}
+      dependencies_by_name = tables.each_with_object({}) do |table, acc|
+        acc[table.name] = compute_dependencies(table, reverse_scope_deps)
+      end
+
+      # Only relations whose target is also in this run constrain the order. A
+      # dependency pointing at a table that is not being processed here — e.g.
+      # an embedded MongoDB collection (masked through its parent, never dumped
+      # on its own) or any table excluded from the run — is not something we can
+      # or need to order against, so it must never block resolution. Without
+      # this, such a dependency would stay unresolved forever and masquerade as
+      # a circular dependency, freezing every table that (transitively)
+      # references it.
       present_names = table_by_name.keys.to_set
 
       loop do
         break if table_by_name.empty?
 
         resolvable = table_by_name.values.select do |table|
-          unresolved_dependencies(table, present_names, ordered).empty?
+          unresolved_dependencies(table.name, dependencies_by_name, present_names, ordered).empty?
         end
 
         if resolvable.empty?
           # No table has all its (in-run) dependencies satisfied, yet tables
-          # remain: the belongs_to graph has a genuine cycle and no strict
-          # topological order exists. Rather than aborting the whole export, break
-          # the cycle by emitting one cycle member; see pick_cycle_victim for how
-          # the member is chosen. Warn so the dropped constraint is visible.
-          victim = pick_cycle_victim(table_by_name.values, present_names, ordered)
-          warn_cycle_break(logger, victim, unresolved_dependencies(victim, present_names, ordered))
+          # remain: the dependency graph has a genuine cycle and no strict
+          # topological order exists.
+          #
+          # When a reverse_scope ordering edge participates in the cycle, there
+          # is no safe way out: emitting the reverse-scoped table before an arm
+          # would build its filter from missing state (silently dropping rows),
+          # so fail loudly instead of guessing.
+          detect_reverse_scope_cycle!(table_by_name, dependencies_by_name, reverse_scope_deps, present_names, ordered)
+
+          # Otherwise the cycle is a plain belongs_to cycle. Rather than
+          # aborting the whole export, break it by emitting one cycle member;
+          # see pick_cycle_victim for how the member is chosen. Warn so the
+          # dropped constraint is visible.
+          victim = pick_cycle_victim(table_by_name.values, dependencies_by_name, present_names, ordered)
+          warn_cycle_break(logger, victim, unresolved_dependencies(victim.name, dependencies_by_name, present_names, ordered))
           resolvable = [victim]
         end
 
@@ -67,13 +91,66 @@ module Exwiw
       table.belongs_tos.map(&:table_name)
     end
 
-    # The dependencies still blocking `table`: belongs_to targets that are part
-    # of this run, not yet ordered, and not the table itself (a self-referential
-    # belongs_to never blocks).
-    private_class_method def unresolved_dependencies(table, present_names, ordered)
-      compute_table_dependencies(table).uniq.select do |dep|
-        present_names.include?(dep) && !ordered.include?(dep) && dep != table.name
+    # reverse_scope ordering edges: reverse-scoped table name => its `via`
+    # referencer table names (the tables that must be processed before it).
+    private_class_method def compute_reverse_scope_dependencies(tables)
+      tables.each_with_object({}) do |table, acc|
+        next unless table.respond_to?(:reverse_scope)
+
+        arm_tables = (table.reverse_scope&.via || []).map(&:table).uniq
+        acc[table.name] = arm_tables if arm_tables.any?
+      end
+    end
+
+    # The ordering dependencies of `table`: its belongs_to targets — minus any
+    # belongs_to pointing at a reverse-scoped table that names `table` as a
+    # `via` arm (that edge is inverted: the arm feeds the reverse-scoped table
+    # its ids, so the arm goes first) — plus, when `table` itself is
+    # reverse-scoped, its `via` referencer tables.
+    private_class_method def compute_dependencies(table, reverse_scope_deps)
+      deps = compute_table_dependencies(table).reject do |dep|
+        reverse_scope_deps[dep]&.include?(table.name)
+      end
+      deps += reverse_scope_deps[table.name] || []
+      deps.uniq
+    end
+
+    # The dependencies still blocking `table_name`: dependency targets that are
+    # part of this run, not yet ordered, and not the table itself (a
+    # self-referential dependency never blocks).
+    private_class_method def unresolved_dependencies(table_name, dependencies_by_name, present_names, ordered)
+      dependencies_by_name.fetch(table_name).select do |dep|
+        present_names.include?(dep) && !ordered.include?(dep) && dep != table_name
+      end
+    end
+
+    # Raise when the stall is caused by a reverse_scope ordering edge: a
+    # reverse-scoped table and one of its `via` arms sit in the same non-trivial
+    # strongly-connected component, so no processing order can put every arm
+    # before the table. Typical shape: two reverse-scoped tables naming each
+    # other as arms, or an arm that (transitively) belongs_to the table it
+    # feeds. A plain belongs_to cycle (no reverse_scope edge involved) returns
+    # without raising, leaving the historical cycle-break to handle it.
+    private_class_method def detect_reverse_scope_cycle!(table_by_name, dependencies_by_name, reverse_scope_deps, present_names, ordered)
+      return if reverse_scope_deps.empty?
+
+      adjacency = table_by_name.each_key.each_with_object({}) do |name, acc|
+        acc[name] = unresolved_dependencies(name, dependencies_by_name, present_names, ordered)
       end
+      cyclic_names = strongly_connected_members(adjacency)
+
+      offenders = cyclic_names.select do |name|
+        (reverse_scope_deps[name] || []).any? { |arm| cyclic_names.include?(arm) }
+      end
+      return if offenders.empty?
+
+      details = offenders.sort.map { |name| "'#{name}' (via: #{(reverse_scope_deps[name] & cyclic_names.to_a).sort.join(', ')})" }
+      raise ArgumentError,
+            "reverse_scope creates an ordering cycle: #{details.join('; ')}. " \
+            "A reverse-scoped collection must be processed after all of its reverse_scope.via " \
+            "referencers, but these dependencies form a cycle with the belongs_to/reverse_scope " \
+            "graph (cycle members: #{cyclic_names.to_a.sort.join(', ')}). Remove one of the " \
+            "reverse_scope arms, or break the belongs_to edge that closes the cycle with `ignore: true`."
     end
 
     # Choose the next table to emit when the order is stuck in a cycle. Only
@@ -85,26 +162,27 @@ module Exwiw
     # collapsing to "match every row" (a cross-scope over-extraction risk for the
     # mongodb adapter); break remaining ties by fewest unresolved dependencies,
     # then by name, for determinism.
-    private_class_method def pick_cycle_victim(remaining, present_names, ordered)
+    private_class_method def pick_cycle_victim(remaining, dependencies_by_name, present_names, ordered)
       adjacency = remaining.each_with_object({}) do |table, acc|
-        acc[table.name] = unresolved_dependencies(table, present_names, ordered)
+        acc[table.name] = unresolved_dependencies(table.name, dependencies_by_name, present_names, ordered)
       end
       cyclic_names = strongly_connected_members(adjacency)
 
       candidates = remaining.select { |table| cyclic_names.include?(table.name) }
       candidates = remaining if candidates.empty? # defensive; a stall implies a cycle
 
-      anchored = candidates.select { |table| ordered_parent?(table, present_names, ordered) }
+      anchored = candidates.select { |table| ordered_parent?(table.name, dependencies_by_name, present_names, ordered) }
       pool = anchored.empty? ? candidates : anchored
 
-      pool.min_by { |table| [unresolved_dependencies(table, present_names, ordered).size, table.name] }
+      pool.min_by { |table| [unresolved_dependencies(table.name, dependencies_by_name, present_names, ordered).size, table.name] }
     end
 
-    # True when `table` has a belongs_to whose target was already ordered, so its
-    # extraction filter will be constrained rather than an unscoped full scan.
-    private_class_method def ordered_parent?(table, present_names, ordered)
-      compute_table_dependencies(table).any? do |dep|
-        dep != table.name && present_names.include?(dep) && ordered.include?(dep)
+    # True when `table_name` has a dependency whose target was already ordered,
+    # so its extraction filter will be constrained rather than an unscoped full
+    # scan.
+    private_class_method def ordered_parent?(table_name, dependencies_by_name, present_names, ordered)
+      dependencies_by_name.fetch(table_name).any? do |dep|
+        dep != table_name && present_names.include?(dep) && ordered.include?(dep)
       end
     end
 

data/lib/exwiw/explain_runner.rb

@@ -38,7 +38,13 @@ module Exwiw
       QueryAstBuilder.validate_scope!(dumpable_configs, table_by_name, @dump_target, @logger)
 
       @logger.debug("Determining table processing order...")
-      ordered_table_names = DetermineTableProcessingOrder.run(dumpable_configs, logger: @logger)
+      # Match the export's processing order (see Runner#run): mongodb orders a
+      # reverse-scoped collection after its `via` referencers.
+      ordered_table_names = DetermineTableProcessingOrder.run(
+        dumpable_configs,
+        logger: @logger,
+        runtime_reverse_scope: adapter.is_a?(Adapter::MongodbAdapter),
+      )
 
       total_size = ordered_table_names.size
       ordered_table_names.each_with_index do |table_name, idx|
@@ -68,7 +74,12 @@ module Exwiw
     private def load_table_config(klass)
       Dir[File.join(@schema_dir, "*.json")].map do |file|
         json = JSON.parse(File.read(file))
-        klass.from(json).reject_ignored_members!
+        begin
+          klass.from(json).reject_ignored_members!
+        rescue UnknownConfigKeyError => e
+          # `.from` knows the table, not the file; point at the offending file.
+          raise UnknownConfigKeyError, "#{file}: #{e.message}", e.backtrace
+        end
       end
     end