exwiw 0.9.23 → 0.9.24

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9cbe5a94be08902db6990812f48eb07b3aa5cae7a507acb0f8c3cf1f90edc175
-  data.tar.gz: 19b92d8feee9e874135fc0c9f833d290aacdb1be35c6bcd3da72c7c212352327
+  metadata.gz: 73740836024ec336feb7138c5a94e396df206de932de00737404d556fc403e0a
+  data.tar.gz: de53ac3f5e4c93f6761524041261c2becb65baa9bd396e528ca0ffa9b82b518e
 SHA512:
-  metadata.gz: 75e631242a39ba72bd16ed7d28435f935c9d591d89888632f5fd835b2b727d6f497a976b91bbcc69bdbc8c85fbfb68d2265488d3b07b19baa37001a7d832d173
-  data.tar.gz: abcfacf75eff9c74324fae77915f10b86ab7258f5251bf5fa7adbe73a3af318fd06ffc4e791e7abc00439b1007901d43e80865ff0394ada7eb8deba040707d1d
+  metadata.gz: ac457715cf1eaf521840903e2b2fbb6c5f503230d0abfb7cc24510067c2aa68c8b841d370d8eeed3e0c0ee009a105bea87cfe455a21c50b5bd462feb48c14f22
+  data.tar.gz: 945c4c46077e9ee6678f6fbc6e72b13186e393c4f06035dd1e9d63f10efa3c1fe6adb67a05f470e1330a48d5d8f34d008c67cb8795ba7dbda8294178f13082a0

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+## [0.9.24] - 2026-08-17
+
+### Fixed
+
+- **MongoDB: `ignore: true` is now honored on embedded configs instead of silently exporting the subdocuments raw.** An ignored embedded config keeps its path out of the parent's projection, so the whole subdocument is absent from the dump ([docs/mongodb.md](docs/mongodb.md#excluding-an-embedded-path-ignore-true)); an ignored *field* of an embedded config is deleted from each subdocument before masking runs, as on a top-level collection. That deletion also fixes a leak there: an ignored field a child `references` is fetched as a propagation key and used to be exported. A primary key marked `ignore: true` — previously a silent no-op — now raises `ArgumentError` on load: dropping `_id` would make a restore assign fresh ids and break every reference pointing at the documents.
+
 ## [0.9.23] - 2026-08-13
 
 ### Fixed

data/README.md

@@ -683,6 +683,7 @@ Constraints:
 - If another non-ignored table has a `belongs_to` entry pointing at an ignored table, exwiw raises `ArgumentError` on load. Remove the `belongs_to` entry on the referencing table, or unset `ignore` on the referenced table.
 - Specifying an ignored table as `--target-table` raises `ArgumentError`.
 - `ignore: true` is preserved by `exwiw:schema:generate` regenerations (the receiver value wins over the auto-generated config).
+- On a MongoDB [embedded config](docs/mongodb.md#excluding-an-embedded-path-ignore-true), which is masked through its parent rather than dumped on its own, `ignore: true` excludes the embedded path from the parent's projection — the subdocument is left out of the dump entirely.
 
 ### Ignore / annotate a column or `belongs_to`
 
@@ -706,6 +707,8 @@ Individual `columns` (SQL) / `fields` (MongoDB) and `belongs_tos` entries accept
 }
 ```
 
+On a MongoDB [embedded config](docs/mongodb.md#embedded-documents) an ignored field cannot be left out of the query — the subdocuments arrive as part of whatever the parent collection fetched — so it is removed from each subdocument during masking instead. The result is the same: the field is absent from the dump. A MongoDB config marking its primary key `ignore: true` is rejected on load — the dump has to keep the identifier, or a restore would assign fresh ids and break every reference pointing at the documents.
+
 The ignored entries are removed only at runtime, right after the config is loaded from file; the JSON on disk keeps them. Both `comment` and `ignore` are **preserved across `exwiw:schema:generate` / `exwiw:mongoid:schema:generate` regenerations** (the hand-edited value wins over the auto-generated config), just like `replace_with`. This applies to the MongoDB `MongodbCollectionConfig` (`fields` / `belongs_tos`) as well.
 
 ### `needs_mask_decision`

data/docs/mongodb.md

@@ -38,6 +38,7 @@ exwiw can export a MongoDB database with `--adapter=mongodb`. This document coll
 
 - `replace_with_fake_data` is supported on a field ([full reference](../README.md#replace_with_fake_data)) — its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the same fake value as the SQL adapters for the same seed. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), works inside embedded subdocuments, and is exclusive with `replace_with` on the same field. Add `gem "faker"` to use it (except a config using only `ja` person types). `raw_sql` and `map` are **not** supported (the `MongodbField` schema does not declare them; such keys are rejected on load — see [Unknown keys are rejected](../README.md#unknown-keys-are-rejected)); use `replace_with` for template masking.
 - The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
+- A field marked [`ignore: true`](../README.md#ignore--annotate-a-column-or-belongs_to) is absent from the dump on an embedded config too, though it gets there differently: a top-level collection's ignored field is left out of the projection (unless a child references it — then it is fetched as a propagation key and dropped during masking too), while an embedded one arrives inside the parent's document and is deleted from the subdocument during masking. It is deleted before the collection's own `replace_with` / `replace_with_fake_data` run, so — as on a top-level collection — nothing masking the same subdocument can read it. The primary key cannot be marked `ignore: true` (that raises `ArgumentError` on load): dropping it would leave the dumped documents with no identifier, so a restore would assign fresh ids and break every reference pointing at them.
 
 ## `reverse_scope` on collections
 
@@ -107,6 +108,30 @@ At runtime:
 - Cross-collection references from inside an embedded subdocument (`belongs_tos` on an embedded config) are not supported and raise `ArgumentError` on load.
 - Specifying an embedded config as `--target-table` raises `NotImplementedError`; pass the top-level collection name instead.
 
+### Excluding an embedded path (`ignore: true`)
+
+A parent collection fetches an embedded path only because some config claims it, so marking the embedded config [`ignore: true`](../README.md#ignore-a-table) leaves the path out of the parent's projection: the **whole subdocument is absent from the dump**, and any embedded chain hanging off it goes with it.
+
+```jsonc
+// posts is embedded in users.posts, but its content is not wanted in the dump
+{
+  "name": "posts",
+  "primary_key": "_id",
+  "ignore": true,
+  "comment": "subdocument is environment-specific; drop it rather than mask it",
+  "embedded_in": { "collection_name": "users", "path": "posts" },
+  "belongs_tos": [],
+  "fields": [{ "name": "_id" }, { "name": "title" }]
+}
+```
+
+This is the only way to keep an embedded path out of the dump — MongoDB does not allow mixing exclusions into the inclusion projection exwiw builds — and it is how to drop a subdocument that must not survive into the restored database at all, rather than be masked into a fake value. Two consequences worth knowing:
+
+- Dropping is not masking: a field that must keep a plausible value belongs in `replace_with` / `replace_with_fake_data`, not here. An absent path also means an application reading it sees "not set", which is a different state from "set to a masked value".
+- The path is only excluded when *no* config asks for it. If the parent collection also declares the path in its own `fields`, that declaration still pulls the raw subdocument into the dump — now with no masking applied to it, since the ignored config's rules no longer run. Remove the parent's field entry (or mark it `ignore: true`) as well.
+
+Like every other `ignore`, it is preserved across `exwiw:mongoid:schema:generate` regenerations, so the exclusion survives a config refresh.
+
 ## `exwiw explain` verbosity
 
 The mongodb explain runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) at a configurable verbosity. The default, **`queryPlanner`, only plans the query and does not execute it**, so it is safe to point at a production source. Set it with the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins):

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -383,9 +383,18 @@ module Exwiw
               "collection-level `filter` is not supported by MongodbAdapter (collection: #{config.name})"
       end
 
+      # Index the embedded configs by the collection they are embedded in, so
+      # the parent's projection can pull their paths in and its mask plan can
+      # descend into them.
+      #
+      # An `ignore: true` child is left out, which keeps its path out of the
+      # parent's inclusion projection and so out of the dump — the only way to
+      # exclude an embedded path (see docs/mongodb.md). Its own embedded
+      # children need no handling: they are never reached.
       private def index_embedded_children(config_by_name)
         config_by_name.each_value.with_object({}) do |child, acc|
           next unless child.embedded?
+          next if child.ignore
 
           (acc[child.embedded_in.collection_name] ||= []) << child
         end
@@ -706,9 +715,10 @@ module Exwiw
       # document of that collection. `masked_fields` is `[field_name,
       # template_segments]` for each field carrying a `replace_with`;
       # `faked_fields` is `[field_name, deriver, seed_field]` for each field
-      # carrying a `replace_with_fake_data`; `embedded` is one EmbeddedMask per
-      # embedded child.
-      MaskPlan = Struct.new(:masked_fields, :faked_fields, :embedded)
+      # carrying a `replace_with_fake_data`; `dropped_fields` is the name of each
+      # field marked `ignore:true`; `embedded` is one EmbeddedMask per embedded
+      # child.
+      MaskPlan = Struct.new(:masked_fields, :faked_fields, :dropped_fields, :embedded)
 
       # A pre-resolved embedded-child mask: the parent path split once into
       # `prefix` (the containers to descend into) and `last` (the field holding
@@ -741,11 +751,15 @@ module Exwiw
           acc << [field.name, mask]
         end
         faked_fields = build_faked_fields(config)
+        # Mostly for an embedded config, but a top-level one fetches an ignored
+        # field too when it is a propagation key (a child's `references`), and
+        # this is what keeps it out of the dump.
+        dropped_fields = config.ignored_field_names
         embedded = embedded_children_of(config).map do |child|
           *prefix, last = child.embedded_in.path.split(".")
           EmbeddedMask.new(prefix, last, build_mask_plan(child))
         end
-        MaskPlan.new(masked_fields, faked_fields, embedded)
+        MaskPlan.new(masked_fields, faked_fields, dropped_fields, embedded)
       end
 
       # Compile each `replace_with_fake_data` field into `[field_name, deriver,
@@ -775,13 +789,19 @@ module Exwiw
         end
       end
 
-      # Apply a precompiled MaskPlan to a document in place: render each
-      # `replace_with` field, then each `replace_with_fake_data` field, then
-      # descend into each embedded child (recursing into its own plan). Fake
-      # fields are applied after replace_with so a fake seed reads the already-
-      # masked value — matching the SQL adapters, where replace_with runs in the
-      # database before the Ruby-side fake transform sees the row.
+      # Apply a precompiled MaskPlan to a document in place: drop each
+      # `ignore:true` field, then render each `replace_with` field, then each
+      # `replace_with_fake_data` field, then descend into each embedded child
+      # (recursing into its own plan). Fake fields are applied after replace_with
+      # so a fake seed reads the already-masked value — matching the SQL
+      # adapters, where replace_with runs in the database before the Ruby-side
+      # fake transform sees the row.
+      #
+      # Dropping comes first so an ignored field is invisible to the masking that
+      # follows, as on a top-level collection — normally not fetched, and dropped
+      # here when the projection pulled it in as a propagation key.
       private def apply_mask_plan!(doc, plan)
+        plan.dropped_fields.each { |name| doc.delete(name) }
         plan.masked_fields.each do |name, mask|
           # Preserve a NULL / absent source value instead of clobbering it into a
           # masked literal. `doc[name].nil?` is true for both an explicit nil and

data/lib/exwiw/mongodb_collection_config.rb

@@ -67,6 +67,7 @@ module Exwiw
       instance.__send__(:validate_embedded!)
       instance.__send__(:validate_belongs_tos!)
       instance.__send__(:validate_fake_data!)
+      instance.__send__(:validate_ignored_fields!)
       instance
     end
 
@@ -78,11 +79,20 @@ module Exwiw
       !embedded_in.nil?
     end
 
+    # The names of the fields flagged `ignore:true`, kept answerable after
+    # #reject_ignored_members! has dropped the entries: an embedded config's
+    # subdocuments arrive inside the parent's document, so masking is the only
+    # place they can be removed (see MongodbAdapter#build_mask_plan).
+    def ignored_field_names
+      @ignored_field_names ||= fields.select(&:ignore).map(&:name)
+    end
+
     # Drop the belongs_tos/fields flagged `ignore:true` so they are excluded from
     # extraction. The config files on disk keep these entries; this is applied to
     # the runtime config right after it is loaded (see Runner#load_table_config).
     def reject_ignored_members!
       self.belongs_tos = belongs_tos.reject(&:ignore)
+      ignored_field_names # memoize before the entries are dropped
       self.fields = fields.reject(&:ignore)
       self
     end
@@ -197,6 +207,21 @@ module Exwiw
       end
     end
 
+    # A primary key marked `ignore: true` is rejected on load. Honoring it
+    # would delete the identifier from every dumped document — restore assigns
+    # fresh ids, breaking every reference pointing at them, and a `{_id}` mask
+    # template renders empty on every row — and silently not honoring it (the
+    # historical behavior, since the projection always fetches the key) would
+    # leave the config lying about what the dump contains.
+    private def validate_ignored_fields!
+      return unless fields.any? { |f| f.ignore && f.name == primary_key }
+
+      raise ArgumentError,
+            "MongodbCollectionConfig '#{name}': the primary key '#{primary_key}' must not be " \
+            "marked ignore: true (the dump has to keep the identifier; remove the ignore, " \
+            "or use ignore: true on the collection to exclude it entirely)."
+    end
+
     private def validate_embedded!
       return unless embedded?
 

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.23"
+  VERSION = "0.9.24"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.23
+  version: 0.9.24
 platform: ruby
 authors:
 - Shia