exwiw 0.9.11 → 0.9.13

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6f85d2a708bc4da44d7a32b0ef91dcf42967b65f5ddcfca516e0ce674de568b4
-  data.tar.gz: 0eb9ea0cd15abd41d64380170323070c0961a37a1bd796bfd9c1c63484c8b6d3
+  metadata.gz: bc7d217b16a94d775d54d12c668886e433a44212a92631ea47258c194d71b542
+  data.tar.gz: 8ca5d14a2289315c1595ac249dc6bfc9ecdea1806403321850adadfcb8ae18d4
 SHA512:
-  metadata.gz: 645793841437d64ad976c84e89dc7f0bde405dbc549a953b870085a6fac26886060535a69bc3f41f5480f4dc31d5c15ce97b598d0146bb96981004cdcee3679e
-  data.tar.gz: 15efb4638a90daf07b9cc83cf2f6613fd0ecee93daaff6fbad40608002d0bf26c4bf61a64a83389d7e672101a6d15c3b4250e34b7daa202d176c62d931b1a7a5
+  metadata.gz: 134aa36ec540862c384ea51cbab3f1cd04a8ed14710bac097cb6aa9aeaf97dae03b9b74d752da3c3bd594390b70a5238961ad30a8713ca0b5c6403114c4bc666
+  data.tar.gz: 3b7a46ac569963c46faead09ad5f7357e3be1a815e0fa8bcc641a3287e8e1aa07a18af62cb552e8a09b4615e64376892df16e8aae0ad4cba048ee05b6555e6b5

data/CHANGELOG.md

@@ -2,10 +2,19 @@
 
 ## [Unreleased]
 
+## [0.9.13] - 2026-07-28
+
+### Changed
+
+- **The mysql adapter materializes each scope id-set into a session `TEMPORARY TABLE` and reuses it across tables.** Previously the same scope subquery (e.g. the dump target's users id-set, including a multi-arm `reverse_scope` UNION) was embedded — and re-evaluated by the server — in every descendant table's extraction query, which dominated export time on large tenants. During `export`, each distinct id-set is now computed once (`CREATE TEMPORARY TABLE ... AS SELECT DISTINCT ...`, indexed), and descendant queries JOIN the temp table; nested scopes are materialized bottom-up so outer sets build from already-materialized inner ones. The dumped rows are identical. `explain` still compiles the inline form and executes nothing. Read-only replicas are supported: `NO_ENGINE_SUBSTITUTION` is stripped from the session `sql_mode` before the first CREATE, so e.g. Aurora MySQL 3 reader instances (which cannot create InnoDB temp tables) substitute a permitted engine instead of failing with ERROR 3161. If temp table creation still fails (e.g. the DB user lacks `CREATE TEMPORARY TABLES`), materialization is disabled for the run with a warning and extraction falls back to the previous inline subqueries. postgresql/sqlite adapters are unchanged.
+
+## [0.9.12] - 2026-07-09
+
 ## [0.9.11] - 2026-07-09
 
 ### Added
 
+- **The MongoDB adapter now supports `replace_with_fake_data`.** The masking mode previously limited to the SQL adapters is now accepted on a `MongodbField`: its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the **same** fake value as the SQL adapters for the same seed value — the value-derivation logic (`build_value_deriver` et al.) was extracted from `RowTransformer` into shared class methods so both families produce byte-identical output for a given seed, type, and locale. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), recurses into embedded subdocuments, and preserves `null`/absent values. `MongodbCollectionConfig` validates the key on load (exclusive with `replace_with` on the same field, unknown types rejected, seed field must resolve) and preserves it across schema regeneration. Add `gem "faker"` to use it (except a config using only `ja` person types, which build from exwiw's bundled dataset). `raw_sql` and `map` remain unsupported on MongoDB. See [MongoDB notes](README.md#mongodb-notes).
 - **Reserved-word and special-character identifiers are now quoted (SQL adapters).** Table and column names that are reserved words (`order`, `from`, `group`, …) or contain characters invalid as a bare identifier are conditionally quoted — backticks on mysql, double quotes on postgresql/sqlite — across every emission path: SELECT projection and masking expressions, FROM/JOIN, WHERE, subqueries, materialized scope JOINs, INSERT / COPY headers and DELETE. Previously only the mysql INSERT header was quoted ([#83](https://github.com/heyinc/exwiw/pull/83)), so a reserved table name broke the extraction SELECT everywhere, a reserved column broke postgresql/sqlite INSERTs, and sqlite rejected even table-qualified reserved columns. Ordinary names stay bare, so output for existing configs is byte-identical (mysql INSERT headers keep their always-backtick form from 0.4.5). Dotted table names are treated as schema qualification and quoted per part (`billing.order` → `` billing.`order` ``), preserving Rails multi-schema `table_name`s. The reserved-word lists cover MySQL 8.4 and 9.x (including 9's `LIBRARY`) plus MariaDB-specific words, PostgreSQL's fully reserved key words, and — for sqlite — exactly the keywords that fail to parse bare in exwiw's emission positions, so fallback-accepted names like `key` are not churned. The lists are enforced by specs that probe the live mysql/postgresql servers' own keyword catalogs and re-derive the sqlite set empirically.
 
 ### Fixed
@@ -33,7 +42,6 @@
 ### Changed
 
 - **Unknown keys in schema config JSON are now rejected on load instead of being silently dropped.** Serdes deserialization is lenient, so a key that matched no declared attribute was discarded without a word — turning a typo (`reverse_scop`, `bulk_insert_chunk_sise`) or a key another adapter supports but this one does not (`raw_sql`/`map` on a MongoDB field; `reverse_scope` on a MongoDB collection before this release) into a silent no-op: the dump ran and the requested masking/scoping simply never happened. `TableConfig.from` / `MongodbCollectionConfig.from` now validate the raw hash against the declared attributes — including the nested `belongs_tos` / `columns` / `fields` / `reverse_scope` / `embedded_in` / `replace_with_fake_data` entries — and raise `Exwiw::UnknownConfigKeyError` (an `ArgumentError` subclass) naming the key(s), the table/collection, the nested position, and the allowed keys; `export`/`explain` prepend the offending file path. This is a deliberate hard error with no opt-out: every declared key still passes — including the documentation-only `comment` on table/collection configs and their `belongs_tos`/`columns`/`fields` entries, which remains the supported place for free-form notes — so a config that only uses supported keys is unaffected, while anything now rejected was already being ignored. If a config carries stray keys, remove them or fold them into `comment`.
->>>>>>> 71a0958eea6e031d2beeb7349bea35cb460414ca
 
 ## [0.9.7] - 2026-07-08
 

data/README.md

@@ -901,10 +901,10 @@ fake value, across tables, runs, and adapters:
   that uses **only** `ja` person types needs no faker (that pool is built
   entirely from the bundled dataset); faker is required for every other type
   and locale.
-- Exclusive with the other masking keys on the same column. SQL adapters only
-  (the MongoDB adapter rejects the key on load, see
-  [Unknown keys are rejected](#unknown-keys-are-rejected)), and invisible to
-  `explain`.
+- Exclusive with the other masking keys on the same column, and invisible to
+  `explain`. Also supported by the MongoDB adapter on a `MongodbField` (seed
+  names a field of the collection, or `_id`), where it is applied document-side
+  after `replace_with` — see [MongoDB notes](#mongodb-notes).
 
 **Performance**: this is a per-row Ruby transform, measured at ~1.5–1.6µs/row
 per fake column (so ≈ +8s per 5M rows per column; ~+40% against a local sqlite
@@ -939,7 +939,7 @@ The MongoDB adapter is experimental. To use it:
   mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
   ```
 - Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
-- `raw_sql`, `map`, and `replace_with_fake_data` are not supported (the `MongodbField` schema does not declare them; such keys in a config are rejected on load — see [Unknown keys are rejected](#unknown-keys-are-rejected)). Use `replace_with` for masking.
+- `replace_with_fake_data` is supported on a field ([full reference](#replace_with_fake_data)) — its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the same fake value as the SQL adapters for the same seed. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), works inside embedded subdocuments, and is exclusive with `replace_with` on the same field. Add `gem "faker"` to use it (except a config using only `ja` person types). `raw_sql` and `map` are **not** supported (the `MongodbField` schema does not declare them; such keys are rejected on load — see [Unknown keys are rejected](#unknown-keys-are-rejected)); use `replace_with` for template masking.
 - The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
 
 #### `reverse_scope` on collections

data/lib/exwiw/adapter/identifier_quoting.rb

@@ -119,7 +119,7 @@ module Exwiw
         day_minute day_second dec decimal declare default delayed delete
         dense_rank desc describe deterministic distinct distinctrow div double
         drop dual each else elseif empty enclosed escaped except exists exit
-        explain false fetch first_value float float4 float8 for force foreign
+        explain external false fetch first_value float float4 float8 for force foreign
         from fulltext function generated get grant group grouping groups
         having high_priority hour_microsecond hour_minute hour_second if
         ignore in index infile inner inout insensitive insert int int1 int2
@@ -132,13 +132,13 @@ module Exwiw
         minute_second mod modifies natural not no_write_to_binlog nth_value
         ntile null numeric of on optimize optimizer_costs option optionally
         or order out outer outfile over partition percent_rank precision
-        primary procedure purge range rank read reads read_write real
+        primary procedure purge qualify range rank read reads read_write real
         recursive references regexp release rename repeat replace require
         resignal restrict return revoke right rlike row rows row_number
         schema schemas second_microsecond select sensitive separator set show
         signal smallint spatial specific sql sqlexception sqlstate sqlwarning
         sql_big_result sql_calc_found_rows sql_small_result ssl starting
-        stored straight_join system table terminated then tinyblob tinyint
+        stored straight_join system table tablesample terminated then tinyblob tinyint
         tinytext to trailing trigger true undo union unique unlock unsigned
         update usage use using utc_date utc_time utc_timestamp values
         varbinary varchar varcharacter varying virtual when where while

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -691,8 +691,10 @@ module Exwiw
       # A masking plan compiled once per collection config and reused for every
       # document of that collection. `masked_fields` is `[field_name,
       # template_segments]` for each field carrying a `replace_with`;
-      # `embedded` is one EmbeddedMask per embedded child.
-      MaskPlan = Struct.new(:masked_fields, :embedded)
+      # `faked_fields` is `[field_name, deriver, seed_field]` for each field
+      # carrying a `replace_with_fake_data`; `embedded` is one EmbeddedMask per
+      # embedded child.
+      MaskPlan = Struct.new(:masked_fields, :faked_fields, :embedded)
 
       # A pre-resolved embedded-child mask: the parent path split once into
       # `prefix` (the containers to descend into) and `last` (the field holding
@@ -721,17 +723,47 @@ module Exwiw
 
           acc << [field.name, compile_template(field.replace_with)]
         end
+        faked_fields = build_faked_fields(config)
         embedded = embedded_children_of(config).map do |child|
           *prefix, last = child.embedded_in.path.split(".")
           EmbeddedMask.new(prefix, last, build_mask_plan(child))
         end
-        MaskPlan.new(masked_fields, embedded)
+        MaskPlan.new(masked_fields, faked_fields, embedded)
+      end
+
+      # Compile each `replace_with_fake_data` field into `[field_name, deriver,
+      # seed_field]`. The deriver (RowTransformer.build_value_deriver) is the
+      # same one the SQL adapters use, so a given seed value produces a
+      # byte-identical fake value across adapters. The seed is re-resolved here
+      # against the effective (post-ignore) fields, so a seed pointing at an
+      # `ignore:true` field — accepted by the load-time validation, which sees
+      # the full field list — is caught at dump time rather than silently
+      # hashing an absent value.
+      private def build_faked_fields(config)
+        config.fields.each_with_object([]) do |field, acc|
+          fake_data = field.replace_with_fake_data
+          next unless fake_data
+
+          seed_field = fake_data.seed.delete_prefix("#{config.name}.")
+          if seed_field != config.primary_key && config.fields.none? { |f| f.name == seed_field }
+            raise ArgumentError,
+                  "replace_with_fake_data for collection '#{config.name}' field '#{field.name}': " \
+                  "seed '#{fake_data.seed}' does not resolve to an extracted field (is it ignore:true?)"
+          end
+
+          deriver = RowTransformer.build_value_deriver(
+            fake_data, "collection '#{config.name}' field '#{field.name}'"
+          )
+          acc << [field.name, deriver, seed_field]
+        end
       end
 
-      # Apply a precompiled MaskPlan to a document in place: render each masked
-      # field, then descend into each embedded child (recursing into its own
-      # plan). Equivalent to the old apply_replace_with! + apply_embedded_masking!
-      # pair, with all per-config lookups hoisted into the plan.
+      # Apply a precompiled MaskPlan to a document in place: render each
+      # `replace_with` field, then each `replace_with_fake_data` field, then
+      # descend into each embedded child (recursing into its own plan). Fake
+      # fields are applied after replace_with so a fake seed reads the already-
+      # masked value — matching the SQL adapters, where replace_with runs in the
+      # database before the Ruby-side fake transform sees the row.
       private def apply_mask_plan!(doc, plan)
         plan.masked_fields.each do |name, segments|
           # Preserve a NULL / absent source value instead of clobbering it into a
@@ -741,6 +773,14 @@ module Exwiw
 
           doc[name] = render_template(segments, doc)
         end
+        plan.faked_fields.each do |name, deriver, seed_field|
+          # NULL-preserving like replace_with (an absent key stays absent). The
+          # seed is read from the current doc; a nil/absent seed hashes ""
+          # deterministically.
+          next if doc[name].nil?
+
+          doc[name] = deriver.call(doc[seed_field])
+        end
         plan.embedded.each do |child|
           container = child.prefix.reduce(doc) { |acc, seg| acc.is_a?(Hash) ? acc[seg] : nil }
           next unless container.is_a?(Hash)

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -59,11 +59,15 @@ module Exwiw
       end
 
       def execute(query_ast)
-        data_sql = commented_sql(query_ast)
+        data_sql = nil
+        count_sql = nil
         # Count via the same FROM/JOIN/WHERE (projection replaced by COUNT(*)) so
         # the Runner can skip empty tables and log the row count without draining
         # the stream. See StreamingResult for why this is not a subquery wrap.
-        count_sql = "#{sql_query_comment(query_ast)} #{compile_ast(query_ast, count_only: true)}"
+        with_scope_materialization do
+          data_sql = commented_sql(query_ast)
+          count_sql = "#{sql_query_comment(query_ast)} #{compile_ast(query_ast, count_only: true)}"
+        end
 
         @logger.debug("  Executing SQL (streaming): \n#{data_sql}")
         StreamingResult.new(client: connection, data_sql: data_sql, count_sql: count_sql)
@@ -292,16 +296,86 @@ module Exwiw
       # to `<col> IN (subquery)`.
       private def compile_scope_join(from_table_name, where_clause, idx)
         subquery = where_clause.value
-        projection = quote_identifier(subquery_projection_name(subquery))
-        src_alias = "exwiw_scope_src_#{idx}"
         ids_alias = "exwiw_scope_ids_#{idx}"
         outer_key = qualified_name(from_table_name, where_clause.column_name)
 
+        if (scope_table = materialized_scope_table(subquery))
+          return "JOIN #{quote_table_name(scope_table)} AS #{ids_alias} " \
+            "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
+        end
+
+        projection = quote_identifier(subquery_projection_name(subquery))
+        src_alias = "exwiw_scope_src_#{idx}"
+
         "JOIN (SELECT DISTINCT #{src_alias}.#{projection} AS exwiw_scope_id " \
           "FROM (#{compile_subquery(subquery)}) AS #{src_alias}) AS #{ids_alias} " \
           "ON #{outer_key} = #{ids_alias}.exwiw_scope_id"
       end
 
+      # The same scope subquery (e.g. the target-tenant users id-set) is embedded
+      # in every descendant table's extraction query, so the source DB would
+      # re-evaluate it once per table — the dominant cost on large tenants.
+      # During #execute, materialize each distinct id-set once into a session
+      # TEMPORARY TABLE and JOIN that instead. Keyed by the compiled SELECT, so
+      # nested scopes reuse already-materialized parents. #explain and
+      # #describe_query compile without the flag and stay side-effect free.
+      private def with_scope_materialization
+        @materialize_scopes = true
+        yield
+      ensure
+        @materialize_scopes = false
+      end
+
+      private def materialized_scope_table(subquery)
+        return nil unless @materialize_scopes
+        return nil if @scope_materialization_disabled
+
+        select_sql = "SELECT DISTINCT exwiw_scope_src.#{quote_identifier(subquery_projection_name(subquery))} " \
+          "AS exwiw_scope_id FROM (#{compile_subquery(subquery)}) AS exwiw_scope_src"
+
+        @scope_id_tables ||= {}
+        begin
+          prepare_scope_session
+          @scope_id_tables[select_sql] ||= create_scope_id_table(select_sql)
+        rescue StandardError => e
+          # e.g. the DB user lacks CREATE TEMPORARY TABLES; keep extracting with
+          # the inline (per-query) scope subqueries instead of failing the run.
+          @scope_materialization_disabled = true
+          @logger.warn("Disabling scope id-set materialization (#{e.class}: #{e.message}); " \
+            "falling back to inline scope subqueries.")
+          nil
+        end
+      end
+
+      # Read-only replicas (e.g. Aurora MySQL 3 reader instances) cannot create
+      # InnoDB temporary tables: with NO_ENGINE_SUBSTITUTION in sql_mode the
+      # CREATE fails outright (ERROR 3161) instead of substituting a permitted
+      # engine such as MyISAM. Strip that flag once per session — it only
+      # governs DDL engine fallback, not query semantics.
+      private def prepare_scope_session
+        return if @scope_session_prepared
+
+        mode = connection.query("SELECT @@SESSION.sql_mode").rows.dig(0, 0).to_s
+        cleaned = mode.split(',').reject { |part| part == 'NO_ENGINE_SUBSTITUTION' }.join(',')
+        connection.query("SET SESSION sql_mode = '#{cleaned}'") unless cleaned == mode
+        @scope_session_prepared = true
+      end
+
+      # If a step after CREATE fails, the temp table stays in the session until
+      # disconnect, but it is never referenced: the name is only cached (and thus
+      # only joined) after all three statements succeed, and the caller disables
+      # materialization for the rest of the run.
+      private def create_scope_id_table(select_sql)
+        name = "exwiw_scope_id_set_#{@scope_id_tables.size}"
+        connection.query("CREATE TEMPORARY TABLE #{quote_table_name(name)} AS #{select_sql}")
+        # ROW_COUNT() reads the CTAS insert count in O(1); a COUNT(*) would
+        # re-scan the whole id-set just for this log line.
+        count = connection.query("SELECT ROW_COUNT()").rows.dig(0, 0)
+        connection.query("ALTER TABLE #{quote_table_name(name)} ADD INDEX `index_exwiw_scope_id` (exwiw_scope_id)")
+        @logger.info("  Materialized scope id set #{name} (#{count} ids).")
+        name
+      end
+
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/mongodb_collection_config.rb

@@ -66,6 +66,7 @@ module Exwiw
       instance = super
       instance.__send__(:validate_embedded!)
       instance.__send__(:validate_belongs_tos!)
+      instance.__send__(:validate_fake_data!)
       instance
     end
 
@@ -92,8 +93,8 @@ module Exwiw
     # - structural facts come from the freshly generated config: primary_key,
     #   belongs_tos, embedded_in.
     # - user customizations are kept from the receiver: filter, ignore,
-    #   bulk_insert_chunk_size, query_timeout_ms, and each field's `replace_with`
-    #   masking rule.
+    #   bulk_insert_chunk_size, query_timeout_ms, and each field's
+    #   `replace_with` / `replace_with_fake_data` masking rule.
     # - generated fields drive the field list (so added/removed fields track the
     #   model), but a matching receiver field wins to retain its masking.
     def merge(passed)
@@ -139,6 +140,7 @@ module Exwiw
           receiver = receiver_field_by_name[pf.name]
           if receiver
             pf.replace_with = receiver.replace_with if receiver.replace_with
+            pf.replace_with_fake_data = receiver.replace_with_fake_data if receiver.replace_with_fake_data
             pf.comment = receiver.comment if receiver.comment
             pf.ignore = receiver.ignore unless receiver.ignore.nil?
           end
@@ -147,6 +149,42 @@ module Exwiw
       end
     end
 
+    # Ruby-side masking validation, mirroring TableConfig#validate_ruby_side_masking!
+    # for the SQL adapters: `replace_with_fake_data` is exclusive with
+    # `replace_with` on the same field, the type must be supported, and the seed
+    # must name a field of this collection (bare or `name.`-qualified) or its
+    # primary key. Deliberately static (no faker require, no pool build) so
+    # schema regeneration never triggers value generation; the seed is resolved
+    # again against the effective (post-ignore) fields at dump time in
+    # MongodbAdapter#build_mask_plan.
+    private def validate_fake_data!
+      fields.each do |field|
+        fake_data = field.replace_with_fake_data
+        next unless fake_data
+
+        if field.replace_with
+          raise ArgumentError,
+                "MongodbCollectionConfig '#{name}' field '#{field.name}': replace_with and " \
+                "replace_with_fake_data cannot be combined; use only one."
+        end
+
+        supported_types = RowTransformer::PERSON_TYPES.keys + RowTransformer::FAKE_TYPES.keys
+        unless supported_types.include?(fake_data.type)
+          raise ArgumentError,
+                "MongodbCollectionConfig '#{name}' field '#{field.name}': unknown " \
+                "replace_with_fake_data type '#{fake_data.type}' (supported: #{supported_types.join(', ')})."
+        end
+
+        seed_field = fake_data.seed.delete_prefix("#{name}.")
+        if seed_field.include?(".") || (seed_field != primary_key && fields.none? { |f| f.name == seed_field })
+          raise ArgumentError,
+                "MongodbCollectionConfig '#{name}' field '#{field.name}': replace_with_fake_data " \
+                "seed '#{fake_data.seed}' does not name a field of this collection " \
+                "(use 'field' or '#{name}.field')."
+        end
+      end
+    end
+
     private def validate_embedded!
       return unless embedded?
 

data/lib/exwiw/mongodb_field.rb

@@ -6,6 +6,13 @@ module Exwiw
 
     attribute :name, String
     attribute :replace_with, optional(String), skip_serializing_if_nil: true
+    # Ruby-process-side masking: replace the value with a deterministic fake
+    # value derived from a seed field (see FakeData / RowTransformer). Unlike the
+    # SQL adapters — where replace_with runs in the database and fake data needs
+    # a separate streaming transform — the MongoDB adapter already masks
+    # document-side, so this is applied inside the collection's mask plan
+    # (MongodbAdapter#apply_mask_plan!), after `replace_with`.
+    attribute :replace_with_fake_data, Serdes::OptionalType.new(FakeData), skip_serializing_if_nil: true
     # The Mongoid model's Ruby accessor when the stored document key (`name`)
     # was renamed via `field :ctry, as: :country`. Purely informational — exwiw
     # masks/projects by `name` (the storage key) — but surfacing the accessor

data/lib/exwiw/row_transformer.rb

@@ -216,6 +216,65 @@ module Exwiw
     # sampling with replacement (which wastes ~37% of slots to duplicates), this
     # gives PERSON_POOL_SIZE distinct identities — so JapaneseNames must supply
     # at least that many combinations.
+    # Compile a FakeData config into a callable `seed_value -> fake value`,
+    # requiring faker if the (type, locale) needs it. Shared by the SQL
+    # RowTransformer (per-row array) and the MongoDB mask plan (per-document
+    # hash): both derive byte-identical fake values from the same seed, keeping
+    # `replace_with_fake_data` consistent across adapters. The caller owns NULL
+    # preservation of the *target* (a nil seed still hashes "", deterministically,
+    # exactly like the SQL path). `error_context` names the offending column/field
+    # in raised messages.
+    def self.build_value_deriver(fake_data, error_context)
+      type = fake_data.type
+      unless PERSON_TYPES.key?(type) || FAKE_TYPES.key?(type)
+        raise ArgumentError,
+              "replace_with_fake_data for #{error_context}: unknown type '#{type}'. " \
+              "Supported: #{(PERSON_TYPES.keys + FAKE_TYPES.keys).join(', ')}"
+      end
+      require_faker! if type_needs_faker?(type, fake_data.locale)
+
+      if PERSON_TYPES.key?(type)
+        build_person_deriver(fake_data, error_context)
+      else
+        build_independent_deriver(fake_data)
+      end
+    end
+
+    # Person-family deriver: pick a coherent Person from the per-locale pool.
+    # The digest[0,8] index is shared with every other person column/field, so
+    # one seed's name values all belong to the same person.
+    def self.build_person_deriver(fake_data, error_context)
+      type = fake_data.type
+      locale = fake_data.locale
+      pool = person_pool(locale)
+      extractor = PERSON_TYPES.fetch(type)
+
+      if KANA_TYPES.include?(type) && pool.first.last_name_kana.nil?
+        raise ArgumentError,
+              "replace_with_fake_data for #{error_context}: type '#{type}' needs kana readings, " \
+              "which are only available with locale: ja (got locale: #{locale.inspect})"
+      end
+
+      lambda do |seed_value|
+        digest = Digest::SHA256.digest(seed_value.to_s)
+        person = pool[digest[0, 8].unpack1("Q>") % PERSON_POOL_SIZE]
+        extractor.call(person, locale)
+      end
+    end
+
+    # Independent-type deriver: pick a value from its own (type, locale) pool,
+    # composing a 64-bit token for uniqueness-sensitive types (email/username).
+    def self.build_independent_deriver(fake_data)
+      pool = fake_pool(fake_data.type, fake_data.locale)
+      compose = FAKE_TYPES.fetch(fake_data.type)[:compose]
+
+      lambda do |seed_value|
+        digest = Digest::SHA256.digest(seed_value.to_s)
+        base = pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
+        compose ? compose.call(base, digest[8, 8].unpack1("H*")) : base
+      end
+    end
+
     def self.build_japanese_person_pool(random)
       surnames = JapaneseNames::SURNAMES
       given_names = JapaneseNames::GIVEN_NAMES
@@ -338,47 +397,29 @@ module Exwiw
       seed_index
     end
 
-    # Person-family type: pick a coherent Person from the per-locale pool. The
+    # Person-family type: wrap the shared per-locale person deriver with this
+    # column's NULL preservation (a NULL target stays NULL). The deriver's
     # digest[0,8] index is shared with every other person column, so one seed's
     # name columns all belong to the same person.
     private def compile_person_fake(fake_data, column, column_index, seed_index)
-      type = fake_data.type
-      locale = fake_data.locale
-      pool = self.class.person_pool(locale)
-      extractor = PERSON_TYPES.fetch(type)
-
-      if KANA_TYPES.include?(type) && pool.first.last_name_kana.nil?
-        raise ArgumentError,
-              "replace_with_fake_data for column '#{@table_name}.#{column.name}': " \
-              "type '#{type}' needs kana readings, which are only available with locale: ja " \
-              "(got locale: #{locale.inspect})"
-      end
-
-      # NULL-preserving like replace_with: a NULL target stays NULL.
+      deriver = self.class.build_person_deriver(fake_data, "column '#{@table_name}.#{column.name}'")
       lambda do |row|
         next nil if row[column_index].nil?
 
-        digest = Digest::SHA256.digest(row[seed_index].to_s)
-        person = pool[digest[0, 8].unpack1("Q>") % PERSON_POOL_SIZE]
-        extractor.call(person, locale)
+        deriver.call(row[seed_index])
       end
     end
 
-    # Independent type: pick a value from its own (type, locale) pool.
+    # Independent type: wrap the shared per-(type, locale) deriver with this
+    # column's NULL preservation. A nil seed value hashes "" (deterministic).
+    # Seed values are normalized with to_s so sqlite's native Integer 123 and
+    # pg/mysql's string "123" pick the same fake value.
     private def compile_independent_fake(fake_data, column_index, seed_index)
-      pool = self.class.fake_pool(fake_data.type, fake_data.locale)
-      compose = FAKE_TYPES.fetch(fake_data.type)[:compose]
-
-      # NULL-preserving like replace_with: a NULL target stays NULL. A nil seed
-      # value hashes "" (deterministic). Seed values are normalized with to_s so
-      # sqlite's native Integer 123 and pg/mysql's string "123" pick the same
-      # fake value.
+      deriver = self.class.build_independent_deriver(fake_data)
       lambda do |row|
         next nil if row[column_index].nil?
 
-        digest = Digest::SHA256.digest(row[seed_index].to_s)
-        base = pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
-        compose ? compose.call(base, digest[8, 8].unpack1("H*")) : base
+        deriver.call(row[seed_index])
       end
     end
   end

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.11"
+  VERSION = "0.9.13"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.11
+  version: 0.9.13
 platform: ruby
 authors:
 - Shia