exwiw 0.9.11 → 0.9.12

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6f85d2a708bc4da44d7a32b0ef91dcf42967b65f5ddcfca516e0ce674de568b4
-  data.tar.gz: 0eb9ea0cd15abd41d64380170323070c0961a37a1bd796bfd9c1c63484c8b6d3
+  metadata.gz: 4be92980b62c26dbb27cef847c3534134cf9e5d27c4676dc2ff28468168e52ed
+  data.tar.gz: 357c2efdbe2e818fa4e5cce3739b5d77bb05b0dda50817dc9a8e622746b3d6ce
 SHA512:
-  metadata.gz: 645793841437d64ad976c84e89dc7f0bde405dbc549a953b870085a6fac26886060535a69bc3f41f5480f4dc31d5c15ce97b598d0146bb96981004cdcee3679e
-  data.tar.gz: 15efb4638a90daf07b9cc83cf2f6613fd0ecee93daaff6fbad40608002d0bf26c4bf61a64a83389d7e672101a6d15c3b4250e34b7daa202d176c62d931b1a7a5
+  metadata.gz: 632102dbdeb9b713abf435fff5224565675f7d34843e6a1e447d96b71920018ff2a79f9f27971afc153a27813e6cd68cefc7df790dbc00fe8a371d8d4cb32c5a
+  data.tar.gz: c7dbd4ec63a7b711a276fb4e0c07e5e481621664c5bd07cdc2452b5a60c38c6843e11891cb46d966a969c57e6fe99714d20d5f786a6d9af087d4966180401f58

data/CHANGELOG.md

@@ -2,10 +2,13 @@
 
 ## [Unreleased]
 
+## [0.9.12] - 2026-07-09
+
 ## [0.9.11] - 2026-07-09
 
 ### Added
 
+- **The MongoDB adapter now supports `replace_with_fake_data`.** The masking mode previously limited to the SQL adapters is now accepted on a `MongodbField`: its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the **same** fake value as the SQL adapters for the same seed value — the value-derivation logic (`build_value_deriver` et al.) was extracted from `RowTransformer` into shared class methods so both families produce byte-identical output for a given seed, type, and locale. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), recurses into embedded subdocuments, and preserves `null`/absent values. `MongodbCollectionConfig` validates the key on load (exclusive with `replace_with` on the same field, unknown types rejected, seed field must resolve) and preserves it across schema regeneration. Add `gem "faker"` to use it (except a config using only `ja` person types, which build from exwiw's bundled dataset). `raw_sql` and `map` remain unsupported on MongoDB. See [MongoDB notes](README.md#mongodb-notes).
 - **Reserved-word and special-character identifiers are now quoted (SQL adapters).** Table and column names that are reserved words (`order`, `from`, `group`, …) or contain characters invalid as a bare identifier are conditionally quoted — backticks on mysql, double quotes on postgresql/sqlite — across every emission path: SELECT projection and masking expressions, FROM/JOIN, WHERE, subqueries, materialized scope JOINs, INSERT / COPY headers and DELETE. Previously only the mysql INSERT header was quoted ([#83](https://github.com/heyinc/exwiw/pull/83)), so a reserved table name broke the extraction SELECT everywhere, a reserved column broke postgresql/sqlite INSERTs, and sqlite rejected even table-qualified reserved columns. Ordinary names stay bare, so output for existing configs is byte-identical (mysql INSERT headers keep their always-backtick form from 0.4.5). Dotted table names are treated as schema qualification and quoted per part (`billing.order` → `` billing.`order` ``), preserving Rails multi-schema `table_name`s. The reserved-word lists cover MySQL 8.4 and 9.x (including 9's `LIBRARY`) plus MariaDB-specific words, PostgreSQL's fully reserved key words, and — for sqlite — exactly the keywords that fail to parse bare in exwiw's emission positions, so fallback-accepted names like `key` are not churned. The lists are enforced by specs that probe the live mysql/postgresql servers' own keyword catalogs and re-derive the sqlite set empirically.
 
 ### Fixed
@@ -33,7 +36,6 @@
 ### Changed
 
 - **Unknown keys in schema config JSON are now rejected on load instead of being silently dropped.** Serdes deserialization is lenient, so a key that matched no declared attribute was discarded without a word — turning a typo (`reverse_scop`, `bulk_insert_chunk_sise`) or a key another adapter supports but this one does not (`raw_sql`/`map` on a MongoDB field; `reverse_scope` on a MongoDB collection before this release) into a silent no-op: the dump ran and the requested masking/scoping simply never happened. `TableConfig.from` / `MongodbCollectionConfig.from` now validate the raw hash against the declared attributes — including the nested `belongs_tos` / `columns` / `fields` / `reverse_scope` / `embedded_in` / `replace_with_fake_data` entries — and raise `Exwiw::UnknownConfigKeyError` (an `ArgumentError` subclass) naming the key(s), the table/collection, the nested position, and the allowed keys; `export`/`explain` prepend the offending file path. This is a deliberate hard error with no opt-out: every declared key still passes — including the documentation-only `comment` on table/collection configs and their `belongs_tos`/`columns`/`fields` entries, which remains the supported place for free-form notes — so a config that only uses supported keys is unaffected, while anything now rejected was already being ignored. If a config carries stray keys, remove them or fold them into `comment`.
->>>>>>> 71a0958eea6e031d2beeb7349bea35cb460414ca
 
 ## [0.9.7] - 2026-07-08
 

data/README.md

@@ -901,10 +901,10 @@ fake value, across tables, runs, and adapters:
   that uses **only** `ja` person types needs no faker (that pool is built
   entirely from the bundled dataset); faker is required for every other type
   and locale.
-- Exclusive with the other masking keys on the same column. SQL adapters only
-  (the MongoDB adapter rejects the key on load, see
-  [Unknown keys are rejected](#unknown-keys-are-rejected)), and invisible to
-  `explain`.
+- Exclusive with the other masking keys on the same column, and invisible to
+  `explain`. Also supported by the MongoDB adapter on a `MongodbField` (seed
+  names a field of the collection, or `_id`), where it is applied document-side
+  after `replace_with` — see [MongoDB notes](#mongodb-notes).
 
 **Performance**: this is a per-row Ruby transform, measured at ~1.5–1.6µs/row
 per fake column (so ≈ +8s per 5M rows per column; ~+40% against a local sqlite
@@ -939,7 +939,7 @@ The MongoDB adapter is experimental. To use it:
   mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
   ```
 - Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
-- `raw_sql`, `map`, and `replace_with_fake_data` are not supported (the `MongodbField` schema does not declare them; such keys in a config are rejected on load — see [Unknown keys are rejected](#unknown-keys-are-rejected)). Use `replace_with` for masking.
+- `replace_with_fake_data` is supported on a field ([full reference](#replace_with_fake_data)) — its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the same fake value as the SQL adapters for the same seed. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), works inside embedded subdocuments, and is exclusive with `replace_with` on the same field. Add `gem "faker"` to use it (except a config using only `ja` person types). `raw_sql` and `map` are **not** supported (the `MongodbField` schema does not declare them; such keys are rejected on load — see [Unknown keys are rejected](#unknown-keys-are-rejected)); use `replace_with` for template masking.
 - The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
 
 #### `reverse_scope` on collections

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -691,8 +691,10 @@ module Exwiw
       # A masking plan compiled once per collection config and reused for every
       # document of that collection. `masked_fields` is `[field_name,
       # template_segments]` for each field carrying a `replace_with`;
-      # `embedded` is one EmbeddedMask per embedded child.
-      MaskPlan = Struct.new(:masked_fields, :embedded)
+      # `faked_fields` is `[field_name, deriver, seed_field]` for each field
+      # carrying a `replace_with_fake_data`; `embedded` is one EmbeddedMask per
+      # embedded child.
+      MaskPlan = Struct.new(:masked_fields, :faked_fields, :embedded)
 
       # A pre-resolved embedded-child mask: the parent path split once into
       # `prefix` (the containers to descend into) and `last` (the field holding
@@ -721,17 +723,47 @@ module Exwiw
 
           acc << [field.name, compile_template(field.replace_with)]
         end
+        faked_fields = build_faked_fields(config)
         embedded = embedded_children_of(config).map do |child|
           *prefix, last = child.embedded_in.path.split(".")
           EmbeddedMask.new(prefix, last, build_mask_plan(child))
         end
-        MaskPlan.new(masked_fields, embedded)
+        MaskPlan.new(masked_fields, faked_fields, embedded)
+      end
+
+      # Compile each `replace_with_fake_data` field into `[field_name, deriver,
+      # seed_field]`. The deriver (RowTransformer.build_value_deriver) is the
+      # same one the SQL adapters use, so a given seed value produces a
+      # byte-identical fake value across adapters. The seed is re-resolved here
+      # against the effective (post-ignore) fields, so a seed pointing at an
+      # `ignore:true` field — accepted by the load-time validation, which sees
+      # the full field list — is caught at dump time rather than silently
+      # hashing an absent value.
+      private def build_faked_fields(config)
+        config.fields.each_with_object([]) do |field, acc|
+          fake_data = field.replace_with_fake_data
+          next unless fake_data
+
+          seed_field = fake_data.seed.delete_prefix("#{config.name}.")
+          if seed_field != config.primary_key && config.fields.none? { |f| f.name == seed_field }
+            raise ArgumentError,
+                  "replace_with_fake_data for collection '#{config.name}' field '#{field.name}': " \
+                  "seed '#{fake_data.seed}' does not resolve to an extracted field (is it ignore:true?)"
+          end
+
+          deriver = RowTransformer.build_value_deriver(
+            fake_data, "collection '#{config.name}' field '#{field.name}'"
+          )
+          acc << [field.name, deriver, seed_field]
+        end
       end
 
-      # Apply a precompiled MaskPlan to a document in place: render each masked
-      # field, then descend into each embedded child (recursing into its own
-      # plan). Equivalent to the old apply_replace_with! + apply_embedded_masking!
-      # pair, with all per-config lookups hoisted into the plan.
+      # Apply a precompiled MaskPlan to a document in place: render each
+      # `replace_with` field, then each `replace_with_fake_data` field, then
+      # descend into each embedded child (recursing into its own plan). Fake
+      # fields are applied after replace_with so a fake seed reads the already-
+      # masked value — matching the SQL adapters, where replace_with runs in the
+      # database before the Ruby-side fake transform sees the row.
       private def apply_mask_plan!(doc, plan)
         plan.masked_fields.each do |name, segments|
           # Preserve a NULL / absent source value instead of clobbering it into a
@@ -741,6 +773,14 @@ module Exwiw
 
           doc[name] = render_template(segments, doc)
         end
+        plan.faked_fields.each do |name, deriver, seed_field|
+          # NULL-preserving like replace_with (an absent key stays absent). The
+          # seed is read from the current doc; a nil/absent seed hashes ""
+          # deterministically.
+          next if doc[name].nil?
+
+          doc[name] = deriver.call(doc[seed_field])
+        end
         plan.embedded.each do |child|
           container = child.prefix.reduce(doc) { |acc, seg| acc.is_a?(Hash) ? acc[seg] : nil }
           next unless container.is_a?(Hash)

data/lib/exwiw/mongodb_collection_config.rb

@@ -66,6 +66,7 @@ module Exwiw
       instance = super
       instance.__send__(:validate_embedded!)
       instance.__send__(:validate_belongs_tos!)
+      instance.__send__(:validate_fake_data!)
       instance
     end
 
@@ -92,8 +93,8 @@ module Exwiw
     # - structural facts come from the freshly generated config: primary_key,
     #   belongs_tos, embedded_in.
     # - user customizations are kept from the receiver: filter, ignore,
-    #   bulk_insert_chunk_size, query_timeout_ms, and each field's `replace_with`
-    #   masking rule.
+    #   bulk_insert_chunk_size, query_timeout_ms, and each field's
+    #   `replace_with` / `replace_with_fake_data` masking rule.
     # - generated fields drive the field list (so added/removed fields track the
     #   model), but a matching receiver field wins to retain its masking.
     def merge(passed)
@@ -139,6 +140,7 @@ module Exwiw
           receiver = receiver_field_by_name[pf.name]
           if receiver
             pf.replace_with = receiver.replace_with if receiver.replace_with
+            pf.replace_with_fake_data = receiver.replace_with_fake_data if receiver.replace_with_fake_data
             pf.comment = receiver.comment if receiver.comment
             pf.ignore = receiver.ignore unless receiver.ignore.nil?
           end
@@ -147,6 +149,42 @@ module Exwiw
       end
     end
 
+    # Ruby-side masking validation, mirroring TableConfig#validate_ruby_side_masking!
+    # for the SQL adapters: `replace_with_fake_data` is exclusive with
+    # `replace_with` on the same field, the type must be supported, and the seed
+    # must name a field of this collection (bare or `name.`-qualified) or its
+    # primary key. Deliberately static (no faker require, no pool build) so
+    # schema regeneration never triggers value generation; the seed is resolved
+    # again against the effective (post-ignore) fields at dump time in
+    # MongodbAdapter#build_mask_plan.
+    private def validate_fake_data!
+      fields.each do |field|
+        fake_data = field.replace_with_fake_data
+        next unless fake_data
+
+        if field.replace_with
+          raise ArgumentError,
+                "MongodbCollectionConfig '#{name}' field '#{field.name}': replace_with and " \
+                "replace_with_fake_data cannot be combined; use only one."
+        end
+
+        supported_types = RowTransformer::PERSON_TYPES.keys + RowTransformer::FAKE_TYPES.keys
+        unless supported_types.include?(fake_data.type)
+          raise ArgumentError,
+                "MongodbCollectionConfig '#{name}' field '#{field.name}': unknown " \
+                "replace_with_fake_data type '#{fake_data.type}' (supported: #{supported_types.join(', ')})."
+        end
+
+        seed_field = fake_data.seed.delete_prefix("#{name}.")
+        if seed_field.include?(".") || (seed_field != primary_key && fields.none? { |f| f.name == seed_field })
+          raise ArgumentError,
+                "MongodbCollectionConfig '#{name}' field '#{field.name}': replace_with_fake_data " \
+                "seed '#{fake_data.seed}' does not name a field of this collection " \
+                "(use 'field' or '#{name}.field')."
+        end
+      end
+    end
+
     private def validate_embedded!
       return unless embedded?
 

data/lib/exwiw/mongodb_field.rb

@@ -6,6 +6,13 @@ module Exwiw
 
     attribute :name, String
     attribute :replace_with, optional(String), skip_serializing_if_nil: true
+    # Ruby-process-side masking: replace the value with a deterministic fake
+    # value derived from a seed field (see FakeData / RowTransformer). Unlike the
+    # SQL adapters — where replace_with runs in the database and fake data needs
+    # a separate streaming transform — the MongoDB adapter already masks
+    # document-side, so this is applied inside the collection's mask plan
+    # (MongodbAdapter#apply_mask_plan!), after `replace_with`.
+    attribute :replace_with_fake_data, Serdes::OptionalType.new(FakeData), skip_serializing_if_nil: true
     # The Mongoid model's Ruby accessor when the stored document key (`name`)
     # was renamed via `field :ctry, as: :country`. Purely informational — exwiw
     # masks/projects by `name` (the storage key) — but surfacing the accessor

data/lib/exwiw/row_transformer.rb

@@ -216,6 +216,65 @@ module Exwiw
     # sampling with replacement (which wastes ~37% of slots to duplicates), this
     # gives PERSON_POOL_SIZE distinct identities — so JapaneseNames must supply
     # at least that many combinations.
+    # Compile a FakeData config into a callable `seed_value -> fake value`,
+    # requiring faker if the (type, locale) needs it. Shared by the SQL
+    # RowTransformer (per-row array) and the MongoDB mask plan (per-document
+    # hash): both derive byte-identical fake values from the same seed, keeping
+    # `replace_with_fake_data` consistent across adapters. The caller owns NULL
+    # preservation of the *target* (a nil seed still hashes "", deterministically,
+    # exactly like the SQL path). `error_context` names the offending column/field
+    # in raised messages.
+    def self.build_value_deriver(fake_data, error_context)
+      type = fake_data.type
+      unless PERSON_TYPES.key?(type) || FAKE_TYPES.key?(type)
+        raise ArgumentError,
+              "replace_with_fake_data for #{error_context}: unknown type '#{type}'. " \
+              "Supported: #{(PERSON_TYPES.keys + FAKE_TYPES.keys).join(', ')}"
+      end
+      require_faker! if type_needs_faker?(type, fake_data.locale)
+
+      if PERSON_TYPES.key?(type)
+        build_person_deriver(fake_data, error_context)
+      else
+        build_independent_deriver(fake_data)
+      end
+    end
+
+    # Person-family deriver: pick a coherent Person from the per-locale pool.
+    # The digest[0,8] index is shared with every other person column/field, so
+    # one seed's name values all belong to the same person.
+    def self.build_person_deriver(fake_data, error_context)
+      type = fake_data.type
+      locale = fake_data.locale
+      pool = person_pool(locale)
+      extractor = PERSON_TYPES.fetch(type)
+
+      if KANA_TYPES.include?(type) && pool.first.last_name_kana.nil?
+        raise ArgumentError,
+              "replace_with_fake_data for #{error_context}: type '#{type}' needs kana readings, " \
+              "which are only available with locale: ja (got locale: #{locale.inspect})"
+      end
+
+      lambda do |seed_value|
+        digest = Digest::SHA256.digest(seed_value.to_s)
+        person = pool[digest[0, 8].unpack1("Q>") % PERSON_POOL_SIZE]
+        extractor.call(person, locale)
+      end
+    end
+
+    # Independent-type deriver: pick a value from its own (type, locale) pool,
+    # composing a 64-bit token for uniqueness-sensitive types (email/username).
+    def self.build_independent_deriver(fake_data)
+      pool = fake_pool(fake_data.type, fake_data.locale)
+      compose = FAKE_TYPES.fetch(fake_data.type)[:compose]
+
+      lambda do |seed_value|
+        digest = Digest::SHA256.digest(seed_value.to_s)
+        base = pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
+        compose ? compose.call(base, digest[8, 8].unpack1("H*")) : base
+      end
+    end
+
     def self.build_japanese_person_pool(random)
       surnames = JapaneseNames::SURNAMES
       given_names = JapaneseNames::GIVEN_NAMES
@@ -338,47 +397,29 @@ module Exwiw
       seed_index
     end
 
-    # Person-family type: pick a coherent Person from the per-locale pool. The
+    # Person-family type: wrap the shared per-locale person deriver with this
+    # column's NULL preservation (a NULL target stays NULL). The deriver's
     # digest[0,8] index is shared with every other person column, so one seed's
     # name columns all belong to the same person.
     private def compile_person_fake(fake_data, column, column_index, seed_index)
-      type = fake_data.type
-      locale = fake_data.locale
-      pool = self.class.person_pool(locale)
-      extractor = PERSON_TYPES.fetch(type)
-
-      if KANA_TYPES.include?(type) && pool.first.last_name_kana.nil?
-        raise ArgumentError,
-              "replace_with_fake_data for column '#{@table_name}.#{column.name}': " \
-              "type '#{type}' needs kana readings, which are only available with locale: ja " \
-              "(got locale: #{locale.inspect})"
-      end
-
-      # NULL-preserving like replace_with: a NULL target stays NULL.
+      deriver = self.class.build_person_deriver(fake_data, "column '#{@table_name}.#{column.name}'")
       lambda do |row|
         next nil if row[column_index].nil?
 
-        digest = Digest::SHA256.digest(row[seed_index].to_s)
-        person = pool[digest[0, 8].unpack1("Q>") % PERSON_POOL_SIZE]
-        extractor.call(person, locale)
+        deriver.call(row[seed_index])
       end
     end
 
-    # Independent type: pick a value from its own (type, locale) pool.
+    # Independent type: wrap the shared per-(type, locale) deriver with this
+    # column's NULL preservation. A nil seed value hashes "" (deterministic).
+    # Seed values are normalized with to_s so sqlite's native Integer 123 and
+    # pg/mysql's string "123" pick the same fake value.
     private def compile_independent_fake(fake_data, column_index, seed_index)
-      pool = self.class.fake_pool(fake_data.type, fake_data.locale)
-      compose = FAKE_TYPES.fetch(fake_data.type)[:compose]
-
-      # NULL-preserving like replace_with: a NULL target stays NULL. A nil seed
-      # value hashes "" (deterministic). Seed values are normalized with to_s so
-      # sqlite's native Integer 123 and pg/mysql's string "123" pick the same
-      # fake value.
+      deriver = self.class.build_independent_deriver(fake_data)
       lambda do |row|
         next nil if row[column_index].nil?
 
-        digest = Digest::SHA256.digest(row[seed_index].to_s)
-        base = pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
-        compose ? compose.call(base, digest[8, 8].unpack1("H*")) : base
+        deriver.call(row[seed_index])
       end
     end
   end

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.11"
+  VERSION = "0.9.12"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.11
+  version: 0.9.12
 platform: ruby
 authors:
 - Shia