exwiw 0.9.6 → 0.9.7

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4a9938f5cd311ecefb5568ed27f35ed8594a9ba15e0041e241c423b25af12cba
-  data.tar.gz: f838bd892aecb7a717eac84f1bf52045cec0b6c1881c3a26c29bd4983c845326
+  metadata.gz: 70b1d51562a341ea502ebb5d22ed3aa625d902264d9112bf338aa6142268dd0a
+  data.tar.gz: e1110a13b9d95cddec6d32f0f829360ded75742f4d17c6969a4ebf51d272b454
 SHA512:
-  metadata.gz: 17568187de281651d94abf50190bd0926a3a85eaaf468f9dcdc43b1254583c8cc416f5282e55de16f8af7a44dc1635f95d10f6a06b33de668ab98ace8e7eda79
-  data.tar.gz: 8530a5f2eaf0c15dda1c604e6bc65e923b8f91322990352e3de6ccb3c8dc832f2184e76496941cc4d64c49089b1639bc7b3743f20e79c3ad6083acff1b9c6f26
+  metadata.gz: acd60932985992f2863640102a34d79e87434b566ee96484abbf3a36f3b2d08e7700a13a63657f9f45cd97c8f0f76d83f221b13e23ae3b0b7b257500f9f99890
+  data.tar.gz: 0de13c9385276a68b8ceb582777da4c9f6b329d1f796db5dfc1411ea936843e0aff0f6a5d178c1a883230b7f0bc7378daedd5382ac9dcfc935879ad74df924f9

data/CHANGELOG.md

@@ -2,6 +2,13 @@
 
 ## [Unreleased]
 
+## [0.9.7] - 2026-07-08
+
+### Added
+
+- **`map` masking mode (SQL adapters).** The long-promised `map` column key is now implemented: its value is evaluated once as Ruby and must yield a `Proc`, which is then called for every fetched row with a row accessor (`r['column_name']` reads any column's fetched value) and whose return value replaces the column value in the dump. Unlike `replace_with`/`raw_sql` (which compile into the `SELECT` and run in the database), `map` runs in the exwiw process — it can express transforms SQL cannot, at a measured cost of well under a microsecond per row. There is no automatic NULL preservation (the proc receives `nil` and decides), the key is exclusive with the other masking keys on the same column, and it executes arbitrary Ruby from the schema config — only load trusted configs. mysql/postgresql/sqlite only (the MongoDB adapter silently drops the key, like `raw_sql`); both the INSERT and the PostgreSQL COPY output paths are covered, and the transform streams with the existing bounded-memory dump (no change to the memory profile).
+- **`replace_with_fake_data` masking mode (SQL adapters).** Replaces a column with realistic fake data from the [faker](https://github.com/faker-ruby/faker) gem, picked **deterministically** by a SHA-256 hash of a seed column's value — `{ "seed": "users.id", "type": "human_name" }` maps the same id to the same fake name across tables, runs, and adapters (seed values are `to_s`-normalized, so sqlite's integer ids agree with pg/mysql's string form). NULL targets stay NULL (like `replace_with`); an optional `"locale"` picks the faker locale (e.g. `"ja"`). Supported types: `human_name`, `first_name`, `last_name`, `phone_number`, `address`, `company_name`, plus uniqueness-sensitive `email`/`username`, which embed a 64-bit hex token from the seed hash so unique indexes survive multi-million-row dumps. Values are drawn from a pool of 10,000 candidates pre-generated per (type, locale) under a fixed seed — this keeps the per-row cost at ~1.5µs per fake column, ≈ +8s per 5M rows (a naive per-row faker call is ~30µs), and makes values stable for a given faker version + locale (upgrading faker remaps them). faker is deliberately **not** a runtime dependency: add `gem "faker"` to the consuming Gemfile. Benchmarks and design notes in `docs/row-transform-masking-notes.md` / `script/bench_row_transform.rb`.
+
 ## [0.9.6] - 2026-07-08
 
 ### Added

data/README.md

@@ -788,19 +788,99 @@ If it used with `replace_with`, `replace_with` will be ignored.
 
 #### `map`
 
-XXX: TODO
+The value is evaluated as Ruby code once (per table, at dump time), must yield a
+`Proc`, and the proc is called for every fetched row. Its return value replaces
+the column value in the dump:
 
-Given value will be evaluated as Ruby code, and treated as the proc.
-
-```
-"map": "proc { |r| 'user' + v['id'].to_s + '@example.com' }"
+```jsonc
+{ "name": "email", "map": "proc { |r| 'user' + r['id'].to_s + '@example.com' }" }
 ```
 
 which is equivalent to `"replace_with": "user{id}@example.com"`.
 
-Notice this is the most powerful option, but you should be careful to use this option.
-Because this transformation occured on exwiw process, so much slower than other options.
-Most of case, this option is not recommended.
+- `r['column_name']` reads any column of the current row — the value as fetched
+  from the database (i.e. after SQL-side masking such as another column's
+  `replace_with`, before Ruby-side transforms). `r` is only valid inside the
+  call; do not retain it.
+- Return a `String`, `Numeric`, or `nil`. Unlike `replace_with` there is **no
+  automatic NULL preservation** — the proc receives `nil` and decides.
+- `map` is exclusive with the other masking keys on the same column
+  (`raw_sql` / `replace_with` / `replace_with_fake_data`).
+- SQL adapters only. The MongoDB adapter silently drops the key (like
+  `raw_sql`). Because the transform runs in the exwiw process, it is invisible
+  to `explain`.
+
+**Security note**: `map` executes arbitrary Ruby from the schema config. Treat
+config files with the same trust as your Gemfile — only load trusted configs.
+
+This is the most powerful option, but it runs per row in the exwiw process
+rather than in the database. The measured dispatch cost is small, though
+(~0.6–0.8µs/row plus whatever the proc body does — see
+[`docs/row-transform-masking-notes.md`](docs/row-transform-masking-notes.md)).
+Prefer `replace_with`/`raw_sql` when they can express the transform; reach for
+`map` when they cannot.
+
+#### `replace_with_fake_data`
+
+Replaces the value with realistic-looking fake data generated by the
+[faker](https://github.com/faker-ruby/faker) gem, picked **deterministically**
+from the value of a seed column — the same seed value always maps to the same
+fake value, across tables, runs, and adapters:
+
+```jsonc
+{
+  "name": "name",
+  "replace_with_fake_data": { "seed": "users.id", "type": "human_name" }
+}
+```
+
+- `seed` names a column of the same table, bare (`"id"`) or table-qualified
+  (`"users.id"`). The seed value is hashed (SHA-256, after `to_s`
+  normalization, so sqlite's integer `123` and postgres/mysql's string `"123"`
+  agree) and the hash picks the fake value. Use a stable identifier (integer or
+  string primary key) as the seed; float/decimal/binary columns are discouraged
+  because their text forms differ per adapter. A `NULL` seed value hashes `""`
+  (still deterministic).
+- Like `replace_with`, it **preserves NULL** in the target column.
+- `locale` (optional) sets the faker locale used to generate the candidate
+  values, e.g. `{ "seed": "id", "type": "human_name", "locale": "ja" }` produces
+  Japanese names.
+- Supported `type`s:
+
+  | type | example output |
+  |------|----------------|
+  | `human_name` | `Adrianna Kilback` |
+  | `first_name` | `Adrianna` |
+  | `last_name` | `Kilback` |
+  | `phone_number` | `(555) 123-4567` |
+  | `address` | `282 Kevin Brook, Imogeneborough, CA 58517` |
+  | `company_name` | `Hirthe-Ritchie` |
+  | `email` | `cliff.fay.9d6b804eff5a3f57@example.com` |
+  | `username` | `cliff.fay_9d6b804eff5a3f57` |
+
+- Values are drawn from a pool of 10,000 pre-generated candidates per
+  (type, locale), so distinct seeds can share a fake value. The
+  uniqueness-sensitive types (`email`, `username`) additionally embed a 64-bit
+  hex token derived from the seed hash, so they stay collision-free under a
+  unique index even at millions of rows (collision probability at 5M distinct
+  seeds ≈ 7e-7) and always use the `example.com` domain.
+- **Determinism caveat**: values are stable for a given faker gem version +
+  locale. Upgrading faker (or changing `locale`) regenerates the pool and maps
+  seeds to different values. The seed→value mapping itself never changes within
+  one version.
+- The faker gem is **not** a runtime dependency of exwiw — add `gem "faker"` to
+  your Gemfile to use this mode (exwiw raises a clear error otherwise).
+- Exclusive with the other masking keys on the same column. SQL adapters only
+  (the MongoDB adapter silently drops the key), and invisible to `explain`.
+
+**Performance**: this is a per-row Ruby transform, measured at ~1.5–1.6µs/row
+per fake column (so ≈ +8s per 5M rows per column; ~+40% against a local sqlite
+fetch — the worst case — and proportionally less against a network database,
+where the fetch dominates). Values are drawn from a pool pre-generated once,
+not by calling faker per row (which would be ~20× slower). Memory is
+unaffected: the transform streams with the dump. See
+[`docs/row-transform-masking-notes.md`](docs/row-transform-masking-notes.md)
+for the benchmark, and `script/bench_row_transform.rb` to measure on your data.
 
 ### MongoDB notes
 
@@ -826,7 +906,7 @@ The MongoDB adapter is experimental. To use it:
   mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
   ```
 - Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
-- `raw_sql` is not supported (the `MongodbField` schema does not declare it; any `raw_sql` keys in scenario JSON are silently dropped on load). Use `replace_with` for masking.
+- `raw_sql`, `map`, and `replace_with_fake_data` are not supported (the `MongodbField` schema does not declare them; such keys in scenario JSON are silently dropped on load). Use `replace_with` for masking.
 - The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
 
 #### Embedded documents

data/docs/row-transform-masking-notes.md

@@ -0,0 +1,156 @@
+# Ruby-side masking (`map` / `replace_with_fake_data`): design & benchmark notes
+
+Companion to [`sql-dump-optimization-notes.md`](./sql-dump-optimization-notes.md).
+That document records how the SQL dump path became a bounded-memory streaming
+pipeline; this one records the cost of the first **per-row Ruby work** added on
+top of it — the `RowTransformer` that implements the `map` and
+`replace_with_fake_data` masking modes — and the design decisions the numbers
+drove.
+
+The reproducible harness is `script/bench_row_transform.rb`. The correctness
+anchors are `spec/row_transformer_spec.rb` (determinism contract) and
+`spec/insert_output_snapshot_spec.rb` (byte-exact output for `map`).
+
+## Why these modes cost anything at all
+
+`replace_with` / `raw_sql` compile into the `SELECT` and run **in the
+database**; the Ruby pipeline (streaming fetch → `write_inserts`) previously
+did zero per-row transform work. `map` (arbitrary Ruby proc) and
+`replace_with_fake_data` (faker values) cannot be pushed into SQL, so they run
+in the exwiw process, per row, between the cursor and the INSERT/COPY writer:
+
+```
+adapter.execute --> StreamingResult --(RowTransformer::TransformedResult#each)--> write_inserts / to_copy_from_stdin
+```
+
+The wrapper delegates `#size` (so the COPY path's upfront count and
+`each_slice`'s allocation-hint COUNT are unchanged) and transforms rows one at
+a time off `#each`, so the bounded-memory profile of the streaming dump is
+preserved. When no column opts in, `RowTransformer.build` returns nil and the
+pipeline is byte-for-byte the pre-existing one.
+
+## Design decisions the measurements drove
+
+### Fake values come from a pre-generated pool, not per-row faker calls
+
+A naive deterministic design — per row, seed faker's RNG with the hash and
+call the generator —
+
+```ruby
+Faker::Config.random = Random.new(seed_hash)
+Faker::Name.name
+```
+
+measures **~30 µs/row** (I18n lookups + regex templating per call). At 5M rows
+that is ~150 s for one column: unacceptable.
+
+Instead, `RowTransformer` pre-generates a pool of `POOL_SIZE = 10_000`
+candidate values per `(type, locale)` under a fixed RNG seed (~0.2 s, once),
+and per row only hashes the seed value and indexes the pool:
+
+```ruby
+digest = Digest::SHA256.digest(seed_value.to_s)
+pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
+```
+
+That is ~1.4 µs/row — **~21× cheaper** — and just as deterministic (values are
+stable for a given faker version + locale; the pool regenerates identically).
+
+### One SHA-256 digest supplies both the pool index and the uniqueness token
+
+`Zlib.crc32` is ~16× faster than SHA-256 (0.055 vs 0.87 µs/op), but 32 bits is
+unusable for the uniqueness token the `email`/`username` types embed: a
+birthday collision is even money at ~77k distinct seeds, far below the
+multi-million-row target. One SHA-256 per row provides 64 bits for the pool
+index (bytes 0–7) and an independent 64-bit hex token (bytes 8–15) whose
+collision probability at 5M distinct seeds is ≈ 7e-7. The hash is not the
+bottleneck (see numbers below), so there is no fast-path variant.
+
+### Zero-allocation row accessor for `map`
+
+The proc receives a single reused accessor object (`r['column']` resolves
+through a frozen name→index Hash built once per table), not a per-row Hash of
+the whole row. Replacements are computed from the original row before any is
+written back, so transformed columns can read each other's *pre*-transform
+values regardless of column order. Rows are mutated in place when the driver
+allows it; sqlite3's `Statement#each` yields **frozen** rows, which are duped
+first (the one extra allocation on that adapter).
+
+## Measurements
+
+Environment: Apple Silicon (arm64, 64 GB), Ruby 4.0.5, faker 3.8.0, sqlite
+adapter for serialization. 8-column rows, ~211 B/row of output. Run:
+
+```bash
+BENCH_ROWS=2000000 bundle exec ruby script/bench_row_transform.rb
+BENCH_ROWS=5000000 BENCH_PART_A=0 bundle exec ruby script/bench_row_transform.rb  # E2E only
+```
+
+### Part A — serialization path only (2M rows, 422 MB output)
+
+The rows are pre-materialized, so the delta is exactly the transform cost on
+the `write_inserts` path — the *most pessimistic* relative view, since a real
+dump also pays the DB fetch.
+
+| variant | wall | vs baseline | per row |
+|---|---|---|---|
+| baseline (no transform) | 6.99 s (286k rows/s) | — | — |
+| `map` ×1 column | 8.27 s | +18% | +0.64 µs |
+| fake ×1 column (email) | 10.17 s | +45% | +1.59 µs |
+| fake ×3 columns | 13.85 s | +98% | +3.43 µs (~1.1 µs/col) |
+
+### Per-operation microbench
+
+| operation | µs/op |
+|---|---|
+| SHA-256 digest → u64 index + hex token | 0.87 |
+| `Zlib.crc32` (comparison only) | 0.055 |
+| fake pipeline, transform 1 row (email) | 1.42 |
+| `map` dispatch, transform 1 row | 0.83 |
+| naive per-row `Faker::Name.name` (rejected design) | ~30 |
+
+### Part B — end-to-end against live sqlite (5M rows, ~1.06 GB output)
+
+`execute` + wrap + `write_inserts`, fresh process (`BENCH_PART_A=0`; see
+pitfalls below):
+
+| variant | wall | vs baseline | per row |
+|---|---|---|---|
+| baseline | 19.97 s (250k rows/s) | — | — |
+| fake ×1 column (email) | 27.98 s | +40% | +1.60 µs |
+
+RSS stayed flat at ~70–100 MB for both variants across the whole 5M-row /
+1 GB dump — the transform does not disturb the streaming memory profile.
+
+The 2M-row E2E run reproduces the same per-row cost (+1.55 µs/row).
+
+### Reading the numbers
+
+- **The stable metric is µs/row per masked column**: ~0.6–0.8 µs for `map`
+  (plus whatever the user's proc does), ~1.5–1.6 µs for fake data
+  (SHA-256 + pool lookup + compose + the sqlite frozen-row dup).
+- At **5M rows, one fake column costs ~8 s** of wall clock.
+- The relative percentages above are worst-case: a local sqlite fetch is the
+  fastest possible source (baseline ~4 µs/row), so +1.6 µs reads as +40%.
+  Against a network mysql/postgresql source the same absolute overhead is a
+  much smaller fraction; on the fully-diluted end (2M E2E measured in a
+  process with a large heap) it read as +21%. Unused tables/columns cost
+  exactly zero (`build` returns nil; nothing is wrapped).
+
+## Benchmarking pitfalls found on the way
+
+Recorded because both would silently corrupt a rerun:
+
+1. **A 10 ms `ps` RSS sampler is catastrophic on a large heap.** The
+   `script/bench_sql_dump.rb`-style background sampler forks the whole process
+   per sample; with the ~1 GB rows array a 2M-row `write_inserts` went from
+   10.8 s to **502 s**. This script reports before/after RSS (2 forks per
+   phase) instead; peak-transient capture matters less here because the
+   transform adds no table-sized structure.
+2. **The first measured variant absorbs one-time heap expansion.** With only a
+   partial warm pass, the baseline (measured first) ran ~15% slower than the
+   later variants — enough to make `map` appear *faster* than no transform.
+   The script now does a full-size unmeasured warm pass first. Relatedly,
+   running Part A's materialized array before Part B in one process inflates
+   Part B's GC costs (the 5M E2E overhead read +6.0 µs/row instead of
+   +1.6 µs/row); hence `BENCH_PART_A=0` for clean E2E numbers.

data/lib/exwiw/fake_data.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Configuration for a column's `replace_with_fake_data` masking mode
+  # (see {RowTransformer}): the value is replaced with a fake value picked
+  # deterministically from a faker-generated pool, keyed by a hash of the
+  # `seed` column's value — so the same seed value always maps to the same
+  # fake value, across runs and adapters.
+  #
+  # `seed` names a column of the same table, either bare (`"id"`) or
+  # table-qualified (`"users.id"`). `type` selects the kind of fake value
+  # (see RowTransformer::FAKE_TYPES). `locale` optionally sets the faker
+  # locale used to build the pool (e.g. `"ja"`); nil uses faker's default.
+  class FakeData
+    include Serdes
+
+    attribute :seed, String
+    attribute :type, String
+    attribute :locale, optional(String), skip_serializing_if_nil: true
+  end
+end

data/lib/exwiw/row_transformer.rb

@@ -0,0 +1,243 @@
+# frozen_string_literal: true
+
+require "digest"
+
+module Exwiw
+  # Applies the Ruby-process-side masking modes — `map` and
+  # `replace_with_fake_data` — to the rows streamed out of a SQL adapter's
+  # #execute. Unlike replace_with/raw_sql, which compile into the SELECT and
+  # run in the database, these run in the exwiw process, so they can execute
+  # arbitrary Ruby (`map`) or derive deterministic fake values (`fake data`).
+  #
+  # Built once per table (.build compiles the map procs, resolves seed column
+  # indexes, and pre-generates the fake-value pools), then #wrap decorates the
+  # adapter's StreamingResult: rows are transformed one at a time as the
+  # stream is drained, so the bounded-memory profile of the streaming dump is
+  # preserved. #size delegates to the underlying result, keeping the COPY
+  # path's upfront count and each_slice's allocation-hint COUNT identical to
+  # an unwrapped run.
+  #
+  # SQL adapters only: .build returns nil for configs without a `columns`
+  # list (MongodbCollectionConfig has `fields`; rails-managed tables have no
+  # columns), and for tables where no column carries a Ruby-side mode.
+  class RowTransformer
+    # Fake values are picked from a pool of this many pre-generated values.
+    # Distinct seed values beyond the pool size reuse pool entries (fine for
+    # fake data — determinism, not uniqueness, is the contract; uniqueness-
+    # sensitive types compose a 64-bit token on top, see FAKE_TYPES).
+    POOL_SIZE = 10_000
+
+    # Fixed Random seed for pool generation, so a pool is deterministic for a
+    # given faker gem version + locale.
+    POOL_RANDOM_SEED = 715_517
+
+    # Supported `replace_with_fake_data` types. `pool` builds one candidate
+    # value (called POOL_SIZE times under a seeded Faker random). Types with
+    # `compose` are uniqueness-sensitive: the pooled base alone would collide
+    # under a unique index at scale, so the final value composes the base with
+    # a 64-bit hex token derived from the same seed digest (collision
+    # probability at 5M distinct seeds ≈ 7e-7).
+    FAKE_TYPES = {
+      "human_name"   => { pool: -> { Faker::Name.name } },
+      "first_name"   => { pool: -> { Faker::Name.first_name } },
+      "last_name"    => { pool: -> { Faker::Name.last_name } },
+      "phone_number" => { pool: -> { Faker::PhoneNumber.phone_number } },
+      "address"      => { pool: -> { Faker::Address.full_address } },
+      "company_name" => { pool: -> { Faker::Company.name } },
+      "email"        => { pool: -> { Faker::Internet.username(specifier: 5..12) },
+                          compose: ->(base, token) { "#{base}.#{token}@example.com" } },
+      "username"     => { pool: -> { Faker::Internet.username(specifier: 5..12) },
+                          compose: ->(base, token) { "#{base}_#{token}" } },
+    }.freeze
+
+    # The `r` a map proc receives: read-only access to the current row's
+    # values by column name (`r['id']`). One instance is reused across all
+    # rows (zero per-row allocation) — do not retain it outside the proc.
+    class Row
+      def initialize(table_name, name_to_index)
+        @table_name = table_name
+        @name_to_index = name_to_index
+        @values = nil
+      end
+
+      attr_writer :values
+
+      def [](column_name)
+        index = @name_to_index[column_name]
+        unless index
+          raise ArgumentError,
+                "unknown column '#{column_name}' in map proc for table '#{@table_name}' " \
+                "(available: #{@name_to_index.keys.join(', ')})"
+        end
+        @values[index]
+      end
+    end
+
+    # Lazy Enumerable decorator returned by #wrap. Mirrors the adapters'
+    # StreamingResult contract (#size COUNT delegation, sized enum_for, #each
+    # returning self) so it is a drop-in for both write_inserts and
+    # to_copy_from_stdin.
+    class TransformedResult
+      include Enumerable
+
+      def initialize(inner, transformer)
+        @inner = inner
+        @transformer = transformer
+      end
+
+      def size
+        @inner.size
+      end
+      alias length size
+
+      def each
+        return enum_for(:each) { size } unless block_given?
+
+        @inner.each { |row| yield @transformer.transform(row) }
+        self
+      end
+    end
+
+    # -> RowTransformer | nil. nil when the table carries no Ruby-side mode,
+    # so the Runner can skip wrapping entirely (the unused path stays
+    # byte-identical and cost-free).
+    def self.build(table)
+      return nil unless table.respond_to?(:columns)
+
+      columns = table.columns
+      return nil if columns.nil? || columns.none? { |c| c.map || c.replace_with_fake_data }
+
+      new(table)
+    end
+
+    def self.require_faker!
+      require "faker"
+    rescue LoadError
+      raise LoadError,
+            "replace_with_fake_data requires the faker gem. " \
+            "Add `gem \"faker\"` to your Gemfile (it is not a runtime dependency of exwiw)."
+    end
+
+    # Pools are memoized per (type, locale): every column sharing a type+locale
+    # sees the same pool, which is what makes equal seed values map to equal
+    # fake values across tables and runs.
+    def self.fake_pool(type, locale)
+      @fake_pools ||= {}
+      @fake_pools[[type, locale]] ||= build_fake_pool(type, locale)
+    end
+
+    def self.build_fake_pool(type, locale)
+      spec = FAKE_TYPES.fetch(type)
+      previous_locale = Faker::Config.locale
+      previous_random = Faker::Config.random
+      Faker::Config.locale = locale if locale
+      Faker::Config.random = Random.new(POOL_RANDOM_SEED)
+      Array.new(POOL_SIZE) { spec[:pool].call.freeze }.freeze
+    ensure
+      Faker::Config.locale = previous_locale
+      Faker::Config.random = previous_random
+    end
+
+    def initialize(table)
+      @table_name = table.name
+      @name_to_index = {}
+      table.columns.each_with_index { |column, index| @name_to_index[column.name] = index }
+      @name_to_index.freeze
+      @row_accessor = Row.new(@table_name, @name_to_index)
+
+      self.class.require_faker! if table.columns.any?(&:replace_with_fake_data)
+
+      @transforms = table.columns.each_with_index.filter_map do |column, index|
+        if column.map
+          [index, compile_map(column)]
+        elsif column.replace_with_fake_data
+          [index, compile_fake(column, index)]
+        end
+      end
+      @replacement_buffer = Array.new(@transforms.size)
+    end
+
+    def wrap(results)
+      TransformedResult.new(results, self)
+    end
+
+    # Replacements are all computed from the original row before any is
+    # written back, so a map proc / fake seed always reads the pre-transform
+    # (post-SQL-masking) value regardless of column order. Rows are mutated in
+    # place when possible (each cursor yields a fresh Array per row), but
+    # sqlite3's Statement#each yields frozen rows — those are duped first.
+    def transform(row)
+      @transforms.each_with_index do |(_, callable), k|
+        @replacement_buffer[k] = callable.call(row)
+      end
+      row = row.dup if row.frozen?
+      @transforms.each_with_index do |(index, _), k|
+        row[index] = @replacement_buffer[k]
+      end
+      row
+    end
+
+    private def compile_map(column)
+      # eval is the documented contract of `map`: the schema config supplies a
+      # Ruby proc source and exwiw runs it. Configs are trusted local files
+      # (same trust level as the Gemfile); the README warns to only load
+      # trusted configs. Evaluated once per table, at dump time only — never
+      # during schema generation/regeneration.
+      evaluated =
+        begin
+          eval(column.map, TOPLEVEL_BINDING.dup, "(exwiw map for #{@table_name}.#{column.name})") # rubocop:disable Security/Eval
+        rescue StandardError, ScriptError => e
+          raise ArgumentError,
+                "map for column '#{@table_name}.#{column.name}' failed to eval: #{e.class}: #{e.message}"
+        end
+      unless evaluated.is_a?(Proc)
+        raise ArgumentError,
+              "map for column '#{@table_name}.#{column.name}' must evaluate to a Proc, got #{evaluated.class}"
+      end
+
+      row_accessor = @row_accessor
+      table_name = @table_name
+      column_name = column.name
+      lambda do |row|
+        row_accessor.values = row
+        begin
+          evaluated.call(row_accessor)
+        rescue StandardError => e
+          raise "map proc for column '#{table_name}.#{column_name}' raised: #{e.class}: #{e.message}"
+        end
+      end
+    end
+
+    private def compile_fake(column, column_index)
+      fake_data = column.replace_with_fake_data
+      spec = FAKE_TYPES.fetch(fake_data.type)
+
+      # Re-resolve the seed here, against the effective (post-ignore) columns:
+      # load-time validation sees the full column list, so a seed column that
+      # was ignore:true is only caught at dump time.
+      seed_column = fake_data.seed.delete_prefix("#{@table_name}.")
+      seed_index = @name_to_index[seed_column]
+      unless seed_index
+        raise ArgumentError,
+              "replace_with_fake_data for column '#{@table_name}.#{column.name}': " \
+              "seed '#{fake_data.seed}' does not resolve to an extracted column " \
+              "(is it ignore:true?)"
+      end
+
+      pool = self.class.fake_pool(fake_data.type, fake_data.locale)
+      compose = spec[:compose]
+
+      # NULL-preserving like replace_with: a NULL target stays NULL. A nil
+      # seed value hashes "" (deterministic). Seed values are normalized with
+      # to_s so sqlite's native Integer 123 and pg/mysql's string "123" pick
+      # the same fake value.
+      lambda do |row|
+        next nil if row[column_index].nil?
+
+        digest = Digest::SHA256.digest(row[seed_index].to_s)
+        base = pool[digest[0, 8].unpack1("Q>") % POOL_SIZE]
+        compose ? compose.call(base, digest[8, 8].unpack1("H*")) : base
+      end
+    end
+  end
+end

data/lib/exwiw/runner.rb

@@ -86,9 +86,17 @@ module Exwiw
         # turning the fetched rows into SQL/JSONL, the rescue below can report
         # both the failing step and the exact extraction query that produced the
         # data being processed.
-        phase = "executing extraction query"
+        phase = "compiling row transforms (map/replace_with_fake_data)"
         begin
+          # Ruby-side masking (map / replace_with_fake_data): wrap the streamed
+          # results so rows are transformed as they are drained. nil (and thus
+          # a byte-identical, cost-free run) when no column opts in; covers
+          # both the INSERT and COPY branches below.
+          row_transformer = RowTransformer.build(table)
+
+          phase = "executing extraction query"
           results = adapter.execute(query_ast)
+          results = row_transformer.wrap(results) if row_transformer
           insert_idx = (idx + 1).to_s.rjust(3, '0')
 
           if @output_format == 'copy'

data/lib/exwiw/table_column.rb

@@ -7,6 +7,11 @@ module Exwiw
     attribute :name, String
     attribute :replace_with, optional(String), skip_serializing_if_nil: true
     attribute :raw_sql, optional(String), skip_serializing_if_nil: true
+    # Ruby-process-side masking modes, applied to the fetched rows by
+    # RowTransformer (SQL adapters only) — unlike replace_with/raw_sql, which
+    # compile into the SELECT and run in the database.
+    attribute :map, optional(String), skip_serializing_if_nil: true
+    attribute :replace_with_fake_data, Serdes::OptionalType.new(FakeData), skip_serializing_if_nil: true
     # User-owned fields preserved across schema regeneration (see
     # TableConfig#merge). `ignore:true` drops the column from extraction (SELECT /
     # INSERT) once the config is loaded (see TableConfig#reject_ignored_members!).

data/lib/exwiw/table_config.rb

@@ -219,6 +219,42 @@ module Exwiw
         if primary_key.nil? && !ignore
           raise ArgumentError, "Table '#{name}' requires primary_key."
         end
+
+        columns.each { |column| validate_ruby_side_masking!(column) }
+      end
+    end
+
+    # The Ruby-side masking modes (`map` / `replace_with_fake_data`) are strictly
+    # exclusive — with each other and with the SQL-side modes. Only the new keys
+    # are restricted: the legacy raw_sql > replace_with precedence stays lenient.
+    # Deliberately static (no faker require, no proc eval) so schema
+    # regeneration never executes config Ruby; the seed column and the map proc
+    # are resolved at dump time by RowTransformer.build.
+    private def validate_ruby_side_masking!(column)
+      ruby_side = [column.map && "map", column.replace_with_fake_data && "replace_with_fake_data"].compact
+      return if ruby_side.empty?
+
+      sql_side = [column.raw_sql && "raw_sql", column.replace_with && "replace_with"].compact
+      if ruby_side.size > 1 || sql_side.any?
+        raise ArgumentError,
+              "Table '#{name}' column '#{column.name}': #{(ruby_side + sql_side).join('/')} cannot be combined; " \
+              "use at most one of map/replace_with_fake_data, without raw_sql/replace_with."
+      end
+
+      fake_data = column.replace_with_fake_data
+      return unless fake_data
+
+      unless RowTransformer::FAKE_TYPES.key?(fake_data.type)
+        raise ArgumentError,
+              "Table '#{name}' column '#{column.name}': unknown replace_with_fake_data type '#{fake_data.type}' " \
+              "(supported: #{RowTransformer::FAKE_TYPES.keys.join(', ')})."
+      end
+
+      seed_column = fake_data.seed.delete_prefix("#{name}.")
+      if seed_column.include?(".") || columns.none? { |c| c.name == seed_column }
+        raise ArgumentError,
+              "Table '#{name}' column '#{column.name}': replace_with_fake_data seed '#{fake_data.seed}' " \
+              "does not name a column of this table (use 'column' or '#{name}.column')."
       end
     end
 

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.6"
+  VERSION = "0.9.7"
 end

data/lib/exwiw.rb

@@ -8,6 +8,7 @@ require "serdes"
 require_relative "exwiw/ext_json"
 require_relative "exwiw/config_file"
 require_relative "exwiw/belongs_to"
+require_relative "exwiw/fake_data"
 require_relative "exwiw/table_column"
 require_relative "exwiw/reverse_scope"
 require_relative "exwiw/table_config"
@@ -28,6 +29,7 @@ require_relative "exwiw/mongodb_parallel_dumper"
 require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"
+require_relative "exwiw/row_transformer"
 require_relative "exwiw/after_insert_hook"
 require_relative "exwiw/runner"
 require_relative "exwiw/explain_runner"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.6
+  version: 0.9.7
 platform: ruby
 authors:
 - Shia
@@ -47,6 +47,7 @@ files:
 - docs/plans/2026-05-29-rails-managed-tables.md
 - docs/plans/2026-05-31-ids-column-for-sql-adapters.md
 - docs/plans/2026-06-19-mongodb-export-remove-parallelism-native-ext.md
+- docs/row-transform-masking-notes.md
 - docs/scope-column-redesign.md
 - docs/sql-dump-optimization-notes.md
 - exe/exwiw
@@ -69,6 +70,7 @@ files:
 - lib/exwiw/embedded_in.rb
 - lib/exwiw/explain_runner.rb
 - lib/exwiw/ext_json.rb
+- lib/exwiw/fake_data.rb
 - lib/exwiw/mongo_query.rb
 - lib/exwiw/mongodb_collection_config.rb
 - lib/exwiw/mongodb_field.rb
@@ -79,6 +81,7 @@ files:
 - lib/exwiw/query_ast_builder.rb
 - lib/exwiw/railtie.rb
 - lib/exwiw/reverse_scope.rb
+- lib/exwiw/row_transformer.rb
 - lib/exwiw/runner.rb
 - lib/exwiw/schema_generator.rb
 - lib/exwiw/table_column.rb