exwiw 0.9.20 → 0.9.21

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 328fbb843d5c15da7ea9f764adda96a5c58b4d24de615b9cac38ba305550ed05
-  data.tar.gz: 691448e490f27475d1ae17be240f89af5ea7c3ac54ddedfa544ca1b71517f5c1
+  metadata.gz: 61f6374ae972477087c63744e9a3702acc795872f7d08599cef69bc7898563db
+  data.tar.gz: 5a140799d94c5804fec5988c0643acbcc79c33ed442e7be36fe2b65bcce14a3b
 SHA512:
-  metadata.gz: ee57c7925b2b912a3e9777947d251b8870f94572ec96b5d7fbd0a8d60dbb8aaefd33cde4e12e735b2ff2eb44b24e2f101b81e4bdfc78752f55788619a4ce5a6c
-  data.tar.gz: 9f3a2b95adb72322032b7ac4967d69e3c32a0df6fa75c64adc7f59be1f9de6b7958a8250c0abb2fd9857a5c063d8123f71c2d9678376757d1161ef34f484f368
+  metadata.gz: 78154be467c8920c5011f527957635145089c07bba5268ce4ebe7ac3272622243acb36b05bd1b538740f76a9167dafe48498a022b458bdd11610544d2c10d446
+  data.tar.gz: 7aea16379a332986f78b18f07a1c6c258745c407a804510de1451c17324c1533f7f17fda6ed725bc4e982e5b5824395fd96f01ffa30fc46c90b0a2e10d25c033

data/CHANGELOG.md

@@ -2,6 +2,24 @@
 
 ## [Unreleased]
 
+## [0.9.21] - 2026-08-06
+
+### Changed
+
+- **`schema:generate` now runs in safe mode by default, so a column a migration just added cannot start being exported before anyone has judged whether it holds personal data.** Every column the config does not have yet is emitted **masked** and flagged `needs_mask_decision: true`; existing entries keep their resolved state through the usual merge, so in practice this marks exactly the new columns. `EXWIW_NEW_COLUMNS=plain` opts out, which is what a first-time bootstrap wants — every column of every table is new there. `SchemaGenerator.new` / `.from_rails_application` default to safe too, so a caller driving the generator from its own task or script gets the same behavior. A column that has a default of its own is masked with that default — a value the column provably holds, and the one the application treats as neutral, so masking a `default: true` flag does not quietly turn the feature off for every row (a default the database computes, like `now()`, is not a constant and does not count, and neither does a JSON object — `{...}` in a mask is a column placeholder, so those fall back to `{}`). Otherwise the mask follows the column type: `masked-{primary key}` for text (with `@example.com` when the column name mentions mail), `0` for numbers, `false` for booleans, a fixed date/timestamp, `{}` for JSON; text always takes the template, since its mask has to vary per row. Three kinds of column are flagged but never masked: the primary key and the foreign keys/types the `belongs_tos` join on (masking them would break the joins and leave the dump referencing rows that were never exported); types no constant safely fits (`uuid`, `binary`, enums, array columns — which report their member type, so a scalar default would not fit — and text columns too short to hold the masked value), since a default the column cannot hold would fail the restore the dump feeds; and columns covered by a unique index unless the mask varies per row, since a constant would collapse every row onto one value and break the restore with a duplicate key. If the index list cannot be read at all, every column is treated as unique-indexed rather than risking the latter. ActiveRecord only for now — `schema:generate_mongoid` does not flag new fields yet.
+
+### Added
+
+- **`rake exwiw:schema:check` reports how the committed config differs from what the application would generate now, plus the columns still waiting for a masking decision.** Added/removed/changed tables and columns and the flagged columns come out as sorted JSON with a non-zero exit, each entry prefixed with its database in a multi-database app; `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` additionally writes the report to a file, so a caller need not assume stdout carries nothing else. It regenerates into a throwaway copy rather than over the config directory, so it runs on a working tree it must not modify — a CI check that keeps a schema change from being merged until the config is reconciled and every new column's masking is decided. ActiveRecord only, like safe mode.
+
+- **A column/field can carry `needs_mask_decision: true`, marking a masking decision nobody has made yet.** Extraction ignores the key entirely — what the column exports is whatever `replace_with` / `ignore` say — so it is purely a decision-tracking bit that tooling can attach and report, letting a check refuse to merge a schema change while any column still carries one. The on-disk state wins over regeneration: removing the key is how the decision is recorded, so a regenerated config must not restore it.
+
+- **`replace_with` accepts a non-String value, so a column that is not text can be masked without changing its type.** A String mask is rendered as a template and emitted as text, which is the wrong shape for an integer/boolean column (and would change a MongoDB field's BSON type). A JSON scalar — `"replace_with": 0`, `false`, `1.5` — is used verbatim instead: the SQL adapters emit it as a typed literal rather than concatenating it into text, and the MongoDB adapter assigns the value as-is. `{}` placeholders are only interpreted in the String form, and NULL preservation applies to both. Because `false` is a valid mask value and falsy in Ruby, the checks that decide whether a column is masked now test for nil, so a `replace_with: false` column is no longer emitted unmasked.
+
+### Fixed
+
+- **A `{}` in a `replace_with` template is emitted literally instead of compiling to a reference to a column with no name.** The SQL adapters read `{...}` as a placeholder even when it was empty, so `"replace_with": "{}"` compiled to `CONCAT(t."")` (`t.``` on mysql, `(t."")` on sqlite) and the extraction failed on that table. The MongoDB adapter never had the bug — its placeholder pattern requires at least one character — so the same config meant two different things per adapter. A brace pair with nothing between it names no column and is now a literal everywhere, which is what makes `{}` usable as an empty-JSON mask (the default safe-mode mask for a `json`/`jsonb` column). Templates that name a real column are unaffected, byte for byte.
+
 ## [0.9.20] - 2026-08-05
 
 ### Changed

data/README.md

@@ -351,6 +351,52 @@ EXWIW_SCHEMA_DIR_PATH=custom_directory bundle exec rake exwiw:schema:generate
 
 As with the CLI, a relative `schema_dir` in the config file is resolved relative to the config file's own directory.
 
+#### Safe mode (masking new columns by default)
+
+A migration that adds a column would otherwise leave `schema:generate` emitting it unmasked, so
+it starts being exported the moment the config is regenerated — before anyone has judged whether
+it holds personal data. So `schema:generate` runs in **safe mode by default**: every column the
+config does not have yet is emitted **masked** and flagged
+[`needs_mask_decision: true`](#needs_mask_decision).
+
+Columns already in the config keep whatever they say — the merge that preserves `replace_with` /
+`comment` / `ignore` preserves a resolved decision too — so in practice this marks exactly the
+columns a migration just added.
+
+```bash
+bundle exec rake exwiw:schema:generate           # safe mode
+EXWIW_NEW_COLUMNS=plain bundle exec rake exwiw:schema:generate   # opt out
+```
+
+Opting out is for the **first-time bootstrap** of a config, where every column of every table is
+new and safe mode would flag the whole thing at once. Use it nowhere else: a column committed
+under `plain` carries no flag, so nothing afterwards can tell it apart from one whose masking was
+decided.
+
+A column that has a **default of its own** is masked with that default: it is a value the column
+provably holds, and it is what the application treats as neutral, so masking a `default: true`
+flag does not quietly turn the feature off for every row in the dump. A default the database
+computes (`now()`) is not a constant and does not count, and neither does a JSON object — `{...}`
+in a mask is a column placeholder, so those fall back to `{}`. Otherwise the mask depends on the column
+type: `masked-{primary key}` for text (with `@example.com` appended when the column name mentions
+mail, so it stays a valid address), `0` for numbers, `false` for booleans, a fixed date/timestamp,
+and `{}` for JSON. Text always takes the template rather than its default, since the mask has to
+vary per row. Three kinds of
+column are flagged but deliberately **not** masked:
+
+- **The primary key, and the foreign keys/types the `belongs_tos` join on.** Masking them
+  would break the joins and leave the dump referencing rows that were never exported.
+- **Types no constant safely fits** — `uuid`, `binary`, enums, array columns (which report their
+  member type, so a scalar default would not fit), and text columns too short to hold the masked
+  value. An invalid default would fail the restore the dump feeds, which is worse than exporting
+  the column while the flag keeps the change from being merged.
+- **Columns covered by a unique index**, unless the mask varies per row (the text masks do, via
+  the primary key). A constant would collapse every row onto one value and break the restore with
+  a duplicate key.
+
+Safe mode is ActiveRecord-only for now: `schema:generate_mongoid` does not flag new fields yet,
+though the `needs_mask_decision` key itself is understood on a MongoDB field.
+
 #### Tidying stale config (`schema:tidy`)
 
 `schema:generate` adds and updates config files for the tables it finds, but it never deletes the config file of a table that has been dropped from the application. To reconcile the existing config against the current schema, run:
@@ -368,6 +414,43 @@ Because it reads the database directly, a table that still exists in the databas
 
 It respects `EXWIW_SCHEMA_DIR_PATH` and the per-database subdirectory layout in the same way as `schema:generate`. Unlike `generate`, `tidy` never adds or regenerates entries — every surviving table/column (including hand-edited `comment` / `ignore` / `replace_with`) is left untouched, so it is safe to run on a customized config. The task prints which tables and columns it removed (or that the config was already tidy). Stale `belongs_tos` are not pruned by `tidy`; rerun `schema:generate` to refresh those.
 
+#### Checking the config against the schema
+
+`schema:check` reports how the committed config differs from what the application would
+generate now — without writing anything, so it can run on a working tree it must not modify:
+
+```bash
+bundle exec rake exwiw:schema:check
+```
+
+It regenerates into a throwaway copy of the config directory (safe mode + `tidy`) and prints
+the comparison as JSON, then exits non-zero when anything needs attention:
+
+```json
+{
+  "added_tables": [],
+  "added_columns": ["users.contact_email"],
+  "removed_tables": [],
+  "removed_columns": [],
+  "changed_tables": ["users"],
+  "needs_mask_decision": ["orders.memo"]
+}
+```
+
+`added_*` / `removed_*` / `changed_tables` mean the config no longer matches the schema — run
+`schema:generate` and `schema:tidy` to reconcile it. `needs_mask_decision` lists the columns
+whose masking nobody has decided on yet (see [the flag](#needs_mask_decision)). The exit code
+makes it usable as a CI check that keeps a schema change from being merged until both are
+resolved; the JSON is stable and sorted, so it can be posted as-is. In a multi-database app each
+entry is prefixed with its database (`primary/users.email`), so the same table name in two
+databases stays distinct.
+
+Set `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` to have the same JSON written to a file, which spares a
+caller from assuming stdout carries nothing else (application boot is free to print).
+
+Like safe mode, this is ActiveRecord-only — it regenerates through `SchemaGenerator`, so a
+Mongoid config directory is not supported yet.
+
 #### Multiple databases
 
 If the application uses Rails' multiple-database support (`connects_to`), `schema:generate` buckets models by the database they connect to and writes each database's config files into its own subdirectory of the output directory, named after the database config name (`primary`, `analytics`, ...):
@@ -535,6 +618,27 @@ Individual `columns` (SQL) / `fields` (MongoDB) and `belongs_tos` entries accept
 
 The ignored entries are removed only at runtime, right after the config is loaded from file; the JSON on disk keeps them. Both `comment` and `ignore` are **preserved across `exwiw:schema:generate` / `exwiw:mongoid:schema:generate` regenerations** (the hand-edited value wins over the auto-generated config), just like `replace_with`. This applies to the MongoDB `MongodbCollectionConfig` (`fields` / `belongs_tos`) as well.
 
+### `needs_mask_decision`
+
+A column/field may also carry `needs_mask_decision: true`, marking a column whose masking
+nobody has decided on yet:
+
+```json
+{ "name": "contact_email", "replace_with": "masked-{id}@example.com", "needs_mask_decision": true }
+```
+
+Extraction ignores the key entirely — what the column exports is whatever `replace_with` /
+`ignore` say. It exists so the decision can be tracked and required: `schema:generate`'s
+[safe mode](#safe-mode-masking-new-columns-by-default) attaches it to every newly discovered
+column together with a default mask, and [`schema:check`](#checking-the-config-against-the-schema)
+reports the columns that still carry it, so CI can keep a pull request red until each one is
+resolved. Resolving it means removing the key — after keeping the mask (ideally recording why
+in `comment`), replacing it with a real masking rule, dropping `replace_with` to export the
+raw value, or setting `ignore: true`.
+
+Like `comment` / `ignore`, the on-disk state wins over regeneration: once removed,
+`schema:generate` does not bring it back.
+
 ### Polymorphic `belongs_to`
 
 A Rails polymorphic association (`belongs_to :reviewable, polymorphic: true`) does not point at a single table — the target row is selected at runtime by a type column. exwiw models this as **one `belongs_to` entry per concrete target table**, each carrying two extra fields:
@@ -826,6 +930,23 @@ absent field) is left as-is instead of being replaced by the masked literal, so
 string is a real value and is still masked. Because of this you do not need to hand-write a
 `raw_sql` `CASE WHEN ... IS NOT NULL ...` to keep NULLs.
 
+A **non-String** value (number or boolean) is used verbatim instead of being rendered as a
+template, so a column that is not text keeps its type:
+
+```jsonc
+{ "name": "score",  "replace_with": 0 }      // integer column -> SELECT emits the literal 0
+{ "name": "active", "replace_with": false }  // boolean column
+{ "name": "email",  "replace_with": "masked-{id}@example.com" }  // template, as above
+```
+
+The SQL adapters emit it as a typed literal (not concatenated into text) and the MongoDB
+adapter assigns it as-is, so the field keeps its BSON type. NULL preservation applies to both
+forms.
+
+In the String form, a `{...}` placeholder must name a column: an empty brace pair (`{}`) names
+nothing, so it is emitted literally — which is what makes `"replace_with": "{}"` a usable
+empty-JSON mask, on every adapter.
+
 #### `raw_sql`
 
 It will used instead of the original value.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -733,9 +733,12 @@ module Exwiw
 
       private def build_mask_plan(config)
         masked_fields = config.fields.each_with_object([]) do |field, acc|
-          next unless field.replace_with
+          next if field.replace_with.nil?
 
-          acc << [field.name, compile_template(field.replace_with)]
+          # A non-String replace_with (see Exwiw::MaskValue) is stored verbatim
+          # so the field keeps its BSON type; only a template compiles to segments.
+          mask = Exwiw::MaskValue.scalar?(field.replace_with) ? field.replace_with : compile_template(field.replace_with)
+          acc << [field.name, mask]
         end
         faked_fields = build_faked_fields(config)
         embedded = embedded_children_of(config).map do |child|
@@ -779,13 +782,13 @@ module Exwiw
       # masked value — matching the SQL adapters, where replace_with runs in the
       # database before the Ruby-side fake transform sees the row.
       private def apply_mask_plan!(doc, plan)
-        plan.masked_fields.each do |name, segments|
+        plan.masked_fields.each do |name, mask|
           # Preserve a NULL / absent source value instead of clobbering it into a
           # masked literal. `doc[name].nil?` is true for both an explicit nil and
           # an absent key, so an absent key is left absent (not created).
           next if doc[name].nil?
 
-          doc[name] = render_template(segments, doc)
+          doc[name] = mask.is_a?(Array) ? render_template(mask, doc) : mask
         end
         plan.faked_fields.each do |name, deriver, seed_field|
           # NULL-preserving like replace_with (an absent key stays absent). The

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -486,15 +486,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(", ")
           null_preserving(ast, column, "CONCAT(#{replaced})")
         else

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -588,15 +588,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(", ")
           null_preserving(ast, column, "CONCAT(#{replaced})")
         else

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -329,15 +329,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(" || ")
           null_preserving(ast, column, "(#{replaced})")
         else

data/lib/exwiw/adapter.rb

@@ -263,6 +263,33 @@ module Exwiw
         "CASE WHEN #{qualified_name(ast.from_table_name, column.name)} IS NOT NULL THEN #{masked_expr} ELSE NULL END"
       end
 
+      # Split a `replace_with` template into the expressions it concatenates: a
+      # `{column}` placeholder becomes that column's qualified name, everything
+      # else a quoted literal. An empty brace pair names no column, so it stays
+      # literal — which is what makes `"replace_with": "{}"` an empty-JSON mask.
+      # A quote in a literal is doubled; a backslash is not, so a hand-written
+      # mask containing one means what MySQL makes of it.
+      private def mask_template_parts(ast, column)
+        column.value.scan(/#{Exwiw::MaskValue::PLACEHOLDER}|[^{}]+|[{}]/).map do |part|
+          if part.size > 1 && part.start_with?("{")
+            qualified_name(ast.from_table_name, part[1..-2])
+          else
+            "'#{part.gsub("'", "''")}'"
+          end
+        end
+      end
+
+      # A non-String `replace_with` (see Exwiw::MaskValue) is emitted as a typed
+      # literal rather than concatenated into text, so an integer column masked
+      # with `0` keeps its numeric type.
+      private def scalar_literal(value)
+        case value
+        when true then "TRUE"
+        when false then "FALSE"
+        else value.to_s
+        end
+      end
+
       # Split an outer query's WHERE clauses into the scope id-set clauses to
       # lift into a materialized derived-table JOIN (see each adapter's
       # #compile_scope_join) and the remaining plain clauses (kept in WHERE).

data/lib/exwiw/default_mask.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+require "date"
+
+module Exwiw
+  # The `replace_with` value safe mode attaches to a newly discovered column.
+  #
+  # Only types a constant can safely stand in for are masked: a default the
+  # column cannot hold would fail the restore the dump feeds, which is worse
+  # than exporting the column while its `needs_mask_decision` flag keeps the
+  # change from being merged.
+  module DefaultMask
+    FIXED_DATE = "2000-01-01"
+    FIXED_TIME = "2000-01-01 00:00:00"
+    EMPTY_JSON = "{}"
+    JSON_TYPES = %i[json jsonb].freeze
+
+    # Skip the mask when the column is too short to hold `masked-<primary key>`
+    # for a realistic key; the rendered length is only known per row at dump time.
+    MIN_TEXT_LIMIT = 20
+    MIN_EMAIL_LIMIT = 40
+
+    module_function
+
+    # The default mask for a column, or nil when no safe constant fits it.
+    # `primary_key` is what keeps a text mask unique per row, so without one text
+    # is left unmasked too. Under a unique index (`unique`) only a mask that
+    # varies per row is allowed, or every row would collide on restore.
+    def for(name:, type:, limit:, primary_key:, array: false, unique: false, column_default: nil)
+      return nil if array
+
+      mask =
+        case type
+        when :string, :text then text_mask(name, limit, primary_key)
+        else constant_mask(type, column_default)
+        end
+
+      return nil if mask.nil?
+      return nil if unique && !varies_per_row?(mask, primary_key)
+
+      mask
+    end
+
+    # The column's own default wins over the per-type constant: it is a value the
+    # column provably holds and the one the application treats as neutral, so
+    # masking a `default: true` flag does not turn the feature off for every row.
+    # A default the database computes (`now()`) arrives as nil and falls through.
+    def constant_mask(type, column_default)
+      from_default = default_value(type, column_default)
+      return from_default unless from_default.nil?
+
+      case type
+      when :integer, :decimal, :float then 0
+      when :boolean then false
+      when :date then FIXED_DATE
+      when :datetime, :timestamp, :time then FIXED_TIME
+      when :json, :jsonb then EMPTY_JSON
+      end
+    end
+
+    # The default as a mask value, or nil when it cannot be one. A JSON column's
+    # default is serialized as JSON whatever Ruby class it arrives as, so a string
+    # default keeps its quoting. Anything whose JSON contains an object is
+    # rejected rather than mis-parsed, so an array of objects falls back to the
+    # empty-JSON constant too.
+    def default_value(type, value)
+      return nil if value.nil?
+
+      mask = JSON_TYPES.include?(type) ? value.to_json : scalar_default(value)
+      return nil if mask.is_a?(String) && mask.match?(MaskValue::PLACEHOLDER)
+
+      mask
+    end
+
+    def scalar_default(value)
+      case value
+      when nil then nil
+      when true, false, Integer, Float, String then value
+      when Numeric then value.to_f
+      when Time, DateTime then value.strftime("%Y-%m-%d %H:%M:%S")
+      when Date then value.strftime("%Y-%m-%d")
+      when Hash, Array then value.to_json
+      end
+    end
+
+    def text_mask(name, limit, primary_key)
+      return nil if primary_key.nil?
+
+      if name.to_s.include?("mail")
+        return nil if limit && limit < MIN_EMAIL_LIMIT
+
+        "masked-{#{primary_key}}@example.com"
+      else
+        return nil if limit && limit < MIN_TEXT_LIMIT
+
+        "masked-{#{primary_key}}"
+      end
+    end
+
+    def varies_per_row?(mask, primary_key)
+      mask.is_a?(String) && mask.include?("{#{primary_key}}")
+    end
+  end
+end

data/lib/exwiw/mask_value.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Serdes type for a `replace_with` value: a template String with `{column}`
+  # placeholders, rendered as text, or a non-String JSON scalar used verbatim so
+  # a column that is not text keeps its type (in MongoDB, its BSON type).
+  class MaskValue < Serdes::TypeBase
+    PERMITTED = [String, Integer, Float].freeze
+
+    # What counts as a `{column}` placeholder: an empty brace pair names no
+    # column, which is what lets `{}` be an empty-JSON mask. One definition,
+    # shared by the adapters that parse a template and the generator that must
+    # avoid emitting one — keep it capture-free, since the splitter's `scan`
+    # reads whole matches.
+    PLACEHOLDER = /\{[^{}]+\}/
+
+    def permit?(value)
+      value == true || value == false || PERMITTED.any? { |type| value.is_a?(type) }
+    end
+
+    # Whether `value` is used verbatim rather than rendered as a template.
+    def self.scalar?(value)
+      !value.is_a?(String) && !value.nil?
+    end
+
+    def to_s
+      "mask_value"
+    end
+  end
+end

data/lib/exwiw/mongodb_collection_config.rb

@@ -139,10 +139,13 @@ module Exwiw
         merged.fields = passed.fields.map do |pf|
           receiver = receiver_field_by_name[pf.name]
           if receiver
-            pf.replace_with = receiver.replace_with if receiver.replace_with
+            pf.replace_with = receiver.replace_with unless receiver.replace_with.nil?
             pf.replace_with_fake_data = receiver.replace_with_fake_data if receiver.replace_with_fake_data
             pf.comment = receiver.comment if receiver.comment
             pf.ignore = receiver.ignore unless receiver.ignore.nil?
+            # Receiver wins even when unset: clearing the flag is how a human
+            # records the decision, and regeneration must not bring it back.
+            pf.needs_mask_decision = receiver.needs_mask_decision
           end
           pf
         end
@@ -162,7 +165,7 @@ module Exwiw
         fake_data = field.replace_with_fake_data
         next unless fake_data
 
-        if field.replace_with
+        unless field.replace_with.nil?
           raise ArgumentError,
                 "MongodbCollectionConfig '#{name}' field '#{field.name}': replace_with and " \
                 "replace_with_fake_data cannot be combined; use only one."

data/lib/exwiw/mongodb_field.rb

@@ -5,7 +5,7 @@ module Exwiw
     include Serdes
 
     attribute :name, String
-    attribute :replace_with, optional(String), skip_serializing_if_nil: true
+    attribute :replace_with, Serdes::OptionalType.new(MaskValue.new), skip_serializing_if_nil: true
     # Ruby-process-side masking: replace the value with a deterministic fake
     # value derived from a seed field (see FakeData / RowTransformer). Unlike the
     # SQL adapters — where replace_with runs in the database and fake data needs
@@ -23,6 +23,10 @@ module Exwiw
     # once the config is loaded (see MongodbCollectionConfig#reject_ignored_members!).
     attribute :comment, optional(String), skip_serializing_if_nil: true
     attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    # See TableColumn#needs_mask_decision.
+    attribute :needs_mask_decision,
+              Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)),
+              skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))

data/lib/exwiw/query_ast.rb

@@ -177,7 +177,7 @@ module Exwiw
         columns.map do |c|
           if c.raw_sql
             QueryAst::ColumnValue::RawSql.new(name: c.name, value: c.raw_sql)
-          elsif c.replace_with
+          elsif !c.replace_with.nil?
             QueryAst::ColumnValue::ReplaceWith.new(name: c.name, value: c.replace_with)
           else
             QueryAst::ColumnValue::Plain.new(name: c.name, value: c.name)

data/lib/exwiw/schema_check.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "json"
+require "pathname"
+require "tmpdir"
+
+module Exwiw
+  # Reports how the committed schema config differs from what the application
+  # would generate now, plus the columns still waiting for a masking decision.
+  # Regenerating into a copy rather than over the config directory lets it run
+  # on a working tree it must not modify (CI). ActiveRecord only.
+  class SchemaCheck
+    CATEGORIES = %w[
+      added_tables added_columns removed_tables removed_columns changed_tables needs_mask_decision
+    ].freeze
+
+    def self.from_rails_application(schema_dir:)
+      Rails.application.eager_load!
+      new(models: ActiveRecord::Base.descendants, schema_dir: schema_dir)
+    end
+
+    def initialize(models:, schema_dir:)
+      @models = models
+      @schema_dir = schema_dir
+    end
+
+    # The report as a plain Hash. Every list is sorted, so a given state always
+    # produces the same output (it ends up in a CI comment).
+    def run
+      committed = read_configs(@schema_dir)
+      regenerated = Dir.mktmpdir do |tmp_dir|
+        regenerate_into(tmp_dir)
+        read_configs(tmp_dir)
+      end
+
+      report = diff(committed, regenerated)
+      report["needs_mask_decision"] = flagged_columns(committed)
+      report
+    end
+
+    # Whether the report requires someone to act: the config is out of date, or
+    # a column's masking has not been decided yet.
+    def self.clean?(report)
+      CATEGORIES.all? { |category| report.fetch(category, []).empty? }
+    end
+
+    private def regenerate_into(tmp_dir)
+      FileUtils.cp_r(File.join(@schema_dir, "."), tmp_dir) if Dir.exist?(@schema_dir)
+      # Explicit: safe mode is not optional here, whatever the library default is.
+      SchemaGenerator.new(models: @models, output_dir: tmp_dir, safe_new_columns: true).generate!
+      SchemaGenerator.new(models: @models, output_dir: tmp_dir).tidy!
+    end
+
+    # Every config file under `dir`, keyed by its path relative to `dir` so the
+    # per-database subdirectories stay distinct. Hand-editing these files is the
+    # workflow this drives, so a syntax error has to name the file it is in.
+    private def read_configs(dir)
+      return {} unless Dir.exist?(dir)
+
+      Dir[File.join(dir, "**", "*.json")].each_with_object({}) do |path, acc|
+        key = Pathname.new(path).relative_path_from(Pathname.new(dir)).to_s
+        begin
+          acc[key] = JSON.parse(File.read(path))
+        rescue JSON::ParserError => e
+          raise JSON::ParserError, "invalid JSON in schema config '#{path}': #{e.message}"
+        end
+      end
+    end
+
+    private def diff(committed, regenerated)
+      report = CATEGORIES.to_h { |category| [category, []] }
+
+      (committed.keys | regenerated.keys).sort.each do |key|
+        before = committed[key]
+        after = regenerated[key]
+
+        if before.nil?
+          report["added_tables"] << table_label(key, after)
+          next
+        end
+        if after.nil?
+          report["removed_tables"] << table_label(key, before)
+          next
+        end
+        next if before == after
+
+        label = table_label(key, before)
+        report["changed_tables"] << label
+        added, removed = column_diff(before, after)
+        report["added_columns"] += added.sort.map { |column| "#{label}.#{column}" }
+        report["removed_columns"] += removed.sort.map { |column| "#{label}.#{column}" }
+      end
+
+      report
+    end
+
+    private def column_diff(before, after)
+      before_names = column_names(before)
+      after_names = column_names(after)
+      [after_names - before_names, before_names - after_names]
+    end
+
+    # `fields` is the MongoDB config's spelling of `columns`.
+    private def column_names(config)
+      (config["columns"] || config["fields"] || []).map { |column| column["name"] }
+    end
+
+    # How a table is named in the report. The database is part of the label, or
+    # the same table name in two of them collides (each has `schema_migrations`).
+    private def table_label(key, config)
+      name = config&.fetch("name", nil) || File.basename(key, ".json")
+      db = File.dirname(key)
+      db == "." ? name : "#{db}/#{name}"
+    end
+
+    private def flagged_columns(committed)
+      committed.flat_map do |key, config|
+        (config["columns"] || config["fields"] || [])
+          .select { |column| column["needs_mask_decision"] }
+          .map { |column| "#{table_label(key, config)}.#{column['name']}" }
+      end.sort
+    end
+  end
+end

data/lib/exwiw/schema_generator.rb

@@ -43,14 +43,20 @@ module Exwiw
     # belongs_to to it.
     ACTIVE_STORAGE_VARIANT_RECORDS_TABLE = "active_storage_variant_records"
 
-    def self.from_rails_application(output_dir:)
+    def self.from_rails_application(output_dir:, safe_new_columns: true)
       Rails.application.eager_load!
-      new(models: ActiveRecord::Base.descendants, output_dir: output_dir)
+      new(models: ActiveRecord::Base.descendants, output_dir: output_dir, safe_new_columns: safe_new_columns)
     end
 
-    def initialize(models:, output_dir:)
+    # `safe_new_columns` (the default) emits every column masked — as far as its
+    # type allows, see DefaultMask — and flagged `needs_mask_decision: true`.
+    # #merge lets an existing entry win, so in practice only columns a migration
+    # has just added keep that treatment. Pass false to bootstrap a config, where
+    # every column is new and flagging all of them at once is noise.
+    def initialize(models:, output_dir:, safe_new_columns: true)
       @models = models
       @output_dir = output_dir
+      @safe_new_columns = safe_new_columns
     end
 
     def generate!
@@ -239,11 +245,12 @@ module Exwiw
             columns: representative.column_names.map { |name| { name: name } },
           )
         else
+          belongs_tos = aggregate_belongs_tos(model_group)
           TableConfig.from_symbol_keys(
             name: table_name,
             primary_key: primary_key,
-            belongs_tos: aggregate_belongs_tos(model_group),
-            columns: representative.column_names.map { |name| { name: name } },
+            belongs_tos: belongs_tos,
+            columns: build_columns(representative, primary_key, belongs_tos, conn),
           )
         end
       end
@@ -251,6 +258,58 @@ module Exwiw
       tables_from_models + build_rails_managed_tables(conn)
     end
 
+    # The `columns` entries for a table: just the name, or — in safe mode — also
+    # a default mask and the `needs_mask_decision` flag. The primary key and the
+    # foreign keys/types the belongs_tos join on are flagged but never masked,
+    # since masking them would break the joins.
+    private def build_columns(representative, primary_key, belongs_tos, conn)
+      names = representative.column_names
+      return names.map { |name| { name: name } } unless @safe_new_columns
+
+      structural = belongs_tos.flat_map { |bt| [bt[:foreign_key], bt[:foreign_type]] }.compact.to_set
+      structural << primary_key if primary_key
+      columns_by_name = representative.columns.each_with_object({}) { |column, acc| acc[column.name] = column }
+      unique = unique_column_names(conn, representative.table_name)
+      defaults = representative.column_defaults
+
+      names.map do |name|
+        entry = { name: name, needs_mask_decision: true }
+        next entry if structural.include?(name)
+
+        column = columns_by_name[name]
+        mask = column && DefaultMask.for(
+          name: name,
+          type: column.type,
+          limit: column.limit,
+          primary_key: primary_key,
+          # `integer[]` reports `:integer`, so a scalar default would not fit.
+          array: column.respond_to?(:array?) && column.array?,
+          unique: unique.nil? || unique.include?(name),
+          column_default: defaults[name],
+        )
+        mask.nil? ? entry : entry.merge(replace_with: mask)
+      end
+    end
+
+    # Columns covered by a unique index, or nil when they could not be read —
+    # callers treat that as "assume every column is unique", since masking a
+    # unique column with a constant breaks the restore the dump feeds.
+    #
+    # `Array()` because an expression index reports a String (`lower((email)::text)`)
+    # rather than a list; wrapping keeps it from being walked character by
+    # character, and it simply never matches a column name.
+    private def unique_column_names(conn, table_name)
+      conn.indexes(table_name).select(&:unique).flat_map { |index| Array(index.columns) }.to_set
+    rescue StandardError => e
+      # Once per run: a systematic failure would otherwise repeat per table.
+      unless @warned_missing_indexes
+        @warned_missing_indexes = true
+        warn "exwiw: could not read the indexes of '#{table_name}' (#{e.class}); " \
+             "treating every column as unique-indexed so no constant mask is emitted."
+      end
+      nil
+    end
+
     private def concrete_models
       @models.reject(&:abstract_class?).select(&:table_exists?)
     end

data/lib/exwiw/table_column.rb

@@ -5,7 +5,7 @@ module Exwiw
     include Serdes
 
     attribute :name, String
-    attribute :replace_with, optional(String), skip_serializing_if_nil: true
+    attribute :replace_with, Serdes::OptionalType.new(MaskValue.new), skip_serializing_if_nil: true
     attribute :raw_sql, optional(String), skip_serializing_if_nil: true
     # Ruby-process-side masking modes, applied to the fetched rows by
     # RowTransformer (SQL adapters only) — unlike replace_with/raw_sql, which
@@ -17,6 +17,12 @@ module Exwiw
     # INSERT) once the config is loaded (see TableConfig#reject_ignored_members!).
     attribute :comment, optional(String), skip_serializing_if_nil: true
     attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    # Marks a column whose masking nobody has decided on yet. Extraction never
+    # reads it; `schema:generate` emits it and `schema:check` reports it, so CI
+    # can require the decision.
+    attribute :needs_mask_decision,
+              Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)),
+              skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))

data/lib/exwiw/table_config.rb

@@ -259,7 +259,7 @@ module Exwiw
       ruby_side = [column.map && "map", column.replace_with_fake_data && "replace_with_fake_data"].compact
       return if ruby_side.empty?
 
-      sql_side = [column.raw_sql && "raw_sql", column.replace_with && "replace_with"].compact
+      sql_side = [column.raw_sql && "raw_sql", !column.replace_with.nil? && "replace_with"].select { |v| v }
       if ruby_side.size > 1 || sql_side.any?
         raise ArgumentError,
               "Table '#{name}' column '#{column.name}': #{(ruby_side + sql_side).join('/')} cannot be combined; " \

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.20"
+  VERSION = "0.9.21"
 end

data/lib/exwiw.rb

@@ -10,6 +10,8 @@ require_relative "exwiw/config_file"
 require_relative "exwiw/strict_keys"
 require_relative "exwiw/belongs_to"
 require_relative "exwiw/fake_data"
+require_relative "exwiw/mask_value"
+require_relative "exwiw/default_mask"
 require_relative "exwiw/table_column"
 require_relative "exwiw/reverse_scope"
 require_relative "exwiw/batch_scope"
@@ -38,6 +40,7 @@ require_relative "exwiw/after_insert_hook"
 require_relative "exwiw/runner"
 require_relative "exwiw/explain_runner"
 require_relative "exwiw/schema_generator"
+require_relative "exwiw/schema_check"
 require_relative "exwiw/mongoid_schema_generator"
 
 begin

data/lib/tasks/exwiw.rake

@@ -13,12 +13,18 @@ namespace :exwiw do
       ENV["EXWIW_SCHEMA_DIR_PATH"] || Exwiw::ConfigFile.schema_dir || "exwiw/schema"
     end
 
+    # Safe mode is on unless EXWIW_NEW_COLUMNS=plain, which a first-time
+    # bootstrap wants: there every column is new, so safe mode would flag the
+    # whole config at once. See SchemaGenerator#initialize.
+    safe_new_columns = lambda { ENV["EXWIW_NEW_COLUMNS"] != "plain" }
+
     desc "Generate schema from application"
     task generate: :environment do
       require "exwiw"
 
       groups = Exwiw::SchemaGenerator.from_rails_application(
         output_dir: resolve_schema_dir.call,
+        safe_new_columns: safe_new_columns.call,
       ).generate!
 
       # Surface cross-database belongs_tos the generator auto-ignored: these
@@ -38,6 +44,24 @@ namespace :exwiw do
       end
     end
 
+    desc "Report how the committed schema config differs from the application, without changing it"
+    task check: :environment do
+      require "exwiw"
+
+      report = Exwiw::SchemaCheck.from_rails_application(schema_dir: resolve_schema_dir.call).run
+      json = JSON.pretty_generate(report)
+      puts json
+      # A file too, so a caller need not assume stdout carries only the JSON.
+      File.write(ENV["EXWIW_SCHEMA_CHECK_OUTPUT"], json + "\n") if ENV["EXWIW_SCHEMA_CHECK_OUTPUT"]
+
+      unless Exwiw::SchemaCheck.clean?(report)
+        $stderr.puts "exwiw: the schema config is out of date or has undecided masking; " \
+                     "run `rake exwiw:schema:generate exwiw:schema:tidy` " \
+                     "and resolve every `needs_mask_decision` column."
+        exit 1
+      end
+    end
+
     desc "Remove tables/columns from the schema config that no longer exist in the application"
     task tidy: :environment do
       require "exwiw"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.20
+  version: 0.9.21
 platform: ruby
 authors:
 - Shia
@@ -70,12 +70,14 @@ files:
 - lib/exwiw/cli.rb
 - lib/exwiw/config_file.rb
 - lib/exwiw/ddl_postprocessor.rb
+- lib/exwiw/default_mask.rb
 - lib/exwiw/determine_table_processing_order.rb
 - lib/exwiw/embedded_in.rb
 - lib/exwiw/explain_runner.rb
 - lib/exwiw/ext_json.rb
 - lib/exwiw/fake_data.rb
 - lib/exwiw/japanese_names.rb
+- lib/exwiw/mask_value.rb
 - lib/exwiw/mongo_query.rb
 - lib/exwiw/mongodb_collection_config.rb
 - lib/exwiw/mongodb_field.rb
@@ -88,6 +90,7 @@ files:
 - lib/exwiw/reverse_scope.rb
 - lib/exwiw/row_transformer.rb
 - lib/exwiw/runner.rb
+- lib/exwiw/schema_check.rb
 - lib/exwiw/schema_generator.rb
 - lib/exwiw/strict_keys.rb
 - lib/exwiw/table_column.rb