exwiw 0.9.19 → 0.9.21

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9f565edc88b4eba108ef4d9accf7bbaa7d48f0e1700007cc9d280b0033e357b6
-  data.tar.gz: 822e5cd0b9d33c81d0b396cc641fa960cf80bffbebd6bd078322460a140b37cf
+  metadata.gz: 61f6374ae972477087c63744e9a3702acc795872f7d08599cef69bc7898563db
+  data.tar.gz: 5a140799d94c5804fec5988c0643acbcc79c33ed442e7be36fe2b65bcce14a3b
 SHA512:
-  metadata.gz: 20da3c9ee09865762f40a04b6c4bf6cf24582545ba7c51945a69aed69798906a19a214c9bd2ad588e0b6740f29a3ebfa08a122845f9a28fe64b654ba723088a0
-  data.tar.gz: db91db3a7e1774d386f45d3582301098a23df86f48fe790fae3631895b87fe686dcce3d048a17827f69ce530f948ca423bb9028f8a4b2495d167e820edd6f3fb
+  metadata.gz: 78154be467c8920c5011f527957635145089c07bba5268ce4ebe7ac3272622243acb36b05bd1b538740f76a9167dafe48498a022b458bdd11610544d2c10d446
+  data.tar.gz: 7aea16379a332986f78b18f07a1c6c258745c407a804510de1451c17324c1533f7f17fda6ed725bc4e982e5b5824395fd96f01ffa30fc46c90b0a2e10d25c033

data/CHANGELOG.md

@@ -2,6 +2,30 @@
 
 ## [Unreleased]
 
+## [0.9.21] - 2026-08-06
+
+### Changed
+
+- **`schema:generate` now runs in safe mode by default, so a column a migration just added cannot start being exported before anyone has judged whether it holds personal data.** Every column the config does not have yet is emitted **masked** and flagged `needs_mask_decision: true`; existing entries keep their resolved state through the usual merge, so in practice this marks exactly the new columns. `EXWIW_NEW_COLUMNS=plain` opts out, which is what a first-time bootstrap wants — every column of every table is new there. `SchemaGenerator.new` / `.from_rails_application` default to safe too, so a caller driving the generator from its own task or script gets the same behavior. A column that has a default of its own is masked with that default — a value the column provably holds, and the one the application treats as neutral, so masking a `default: true` flag does not quietly turn the feature off for every row (a default the database computes, like `now()`, is not a constant and does not count, and neither does a JSON object — `{...}` in a mask is a column placeholder, so those fall back to `{}`). Otherwise the mask follows the column type: `masked-{primary key}` for text (with `@example.com` when the column name mentions mail), `0` for numbers, `false` for booleans, a fixed date/timestamp, `{}` for JSON; text always takes the template, since its mask has to vary per row. Three kinds of column are flagged but never masked: the primary key and the foreign keys/types the `belongs_tos` join on (masking them would break the joins and leave the dump referencing rows that were never exported); types no constant safely fits (`uuid`, `binary`, enums, array columns — which report their member type, so a scalar default would not fit — and text columns too short to hold the masked value), since a default the column cannot hold would fail the restore the dump feeds; and columns covered by a unique index unless the mask varies per row, since a constant would collapse every row onto one value and break the restore with a duplicate key. If the index list cannot be read at all, every column is treated as unique-indexed rather than risking the latter. ActiveRecord only for now — `schema:generate_mongoid` does not flag new fields yet.
+
+### Added
+
+- **`rake exwiw:schema:check` reports how the committed config differs from what the application would generate now, plus the columns still waiting for a masking decision.** Added/removed/changed tables and columns and the flagged columns come out as sorted JSON with a non-zero exit, each entry prefixed with its database in a multi-database app; `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` additionally writes the report to a file, so a caller need not assume stdout carries nothing else. It regenerates into a throwaway copy rather than over the config directory, so it runs on a working tree it must not modify — a CI check that keeps a schema change from being merged until the config is reconciled and every new column's masking is decided. ActiveRecord only, like safe mode.
+
+- **A column/field can carry `needs_mask_decision: true`, marking a masking decision nobody has made yet.** Extraction ignores the key entirely — what the column exports is whatever `replace_with` / `ignore` say — so it is purely a decision-tracking bit that tooling can attach and report, letting a check refuse to merge a schema change while any column still carries one. The on-disk state wins over regeneration: removing the key is how the decision is recorded, so a regenerated config must not restore it.
+
+- **`replace_with` accepts a non-String value, so a column that is not text can be masked without changing its type.** A String mask is rendered as a template and emitted as text, which is the wrong shape for an integer/boolean column (and would change a MongoDB field's BSON type). A JSON scalar — `"replace_with": 0`, `false`, `1.5` — is used verbatim instead: the SQL adapters emit it as a typed literal rather than concatenating it into text, and the MongoDB adapter assigns the value as-is. `{}` placeholders are only interpreted in the String form, and NULL preservation applies to both. Because `false` is a valid mask value and falsy in Ruby, the checks that decide whether a column is masked now test for nil, so a `replace_with: false` column is no longer emitted unmasked.
+
+### Fixed
+
+- **A `{}` in a `replace_with` template is emitted literally instead of compiling to a reference to a column with no name.** The SQL adapters read `{...}` as a placeholder even when it was empty, so `"replace_with": "{}"` compiled to `CONCAT(t."")` (`t.``` on mysql, `(t."")` on sqlite) and the extraction failed on that table. The MongoDB adapter never had the bug — its placeholder pattern requires at least one character — so the same config meant two different things per adapter. A brace pair with nothing between it names no column and is now a literal everywhere, which is what makes `{}` usable as an empty-JSON mask (the default safe-mode mask for a `json`/`jsonb` column). Templates that name a real column are unaffected, byte for byte.
+
+## [0.9.20] - 2026-08-05
+
+### Changed
+
+- **PostgreSQL: the extensions a managed platform installs to run the source instance itself (Cloud SQL / AlloyDB) are now explicitly out of target and left out of `insert-000-schema.sql`.** A full-database `pg_dump` of a Cloud SQL instance emits `CREATE SCHEMA google_vacuum_mgmt` plus the `CREATE EXTENSION` / `COMMENT ON EXTENSION` for it; an AlloyDB one emits `google_columnar_engine` and `google_db_advisor`. exwiw already made these survivable — earlier releases wrapped both statements in `DO` blocks that warn-and-skip when the restore target cannot provide the extension — but they still occupied the dump, left an empty vendor schema behind on every restore, and logged a `WARNING` each time. They serve the *source* instance's operation (vacuum tuning, the in-memory columnar cache, index advice), carry no application data, are named by nothing in the application's own schema, and ship only with the managed platform, so no restore target outside it can ever create them. Both halves are now dropped: their schemas via `pg_dump --exclude-schema` (which also covers any object the platform adds inside one later), and the extension statements — not schema-qualified, so no `pg_dump` filter reaches them — via a new `DdlPostprocessor.strip_extensions` pass, with whatever was excluded named in the run's log rather than dropped silently. The set is an explicit list of names (`PostgresqlAdapter::PLATFORM_MANAGED_EXTENSIONS`), deliberately not a `google_*` prefix match: the prefixes are not reserved, so a prefix rule would also drop a schema an application legitimately owns — `google_calendar` for a Google Calendar integration — together with its tables. Adding a name costs a release, which is the right trade. Every other extension is still emitted wrapped in the warn-and-skip `DO` block, including two kinds that are also managed-platform-only: a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`, installable on any plain PostgreSQL), and an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which application SQL and dumped DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it. Data extraction never touched any of these objects — it is driven by the schema config, which has no entry for a table without an application model — so no `insert-*` / `delete-*` file changes.
+
 ## [0.9.19] - 2026-08-04
 
 ### Added

data/README.md

@@ -108,6 +108,13 @@ so you should import the dump in order.
 
 `insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. For `mysql`, set `EXWIW_MYSQLDUMP` to point at a specific `mysqldump` binary when the one on PATH is incompatible with the server (e.g. a MySQL 9.x `mysqldump` cannot load `mysql_native_password` against a server still using that auth plugin — `EXWIW_MYSQLDUMP=/path/to/mysql@8.0/bin/mysqldump`). The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`. For `mysql`, the source server's `DEFINER=user@host` stamp on views and triggers is stripped too, so restoring into a managed MySQL instance (which usually can't grant the privilege to recreate someone else's `DEFINER`) does not fail.
 
+For `postgresql`, the extensions a managed platform installs to run the source instance itself are treated as out of target and left out of the dump entirely — currently `google_vacuum_mgmt` (Cloud SQL / AlloyDB adaptive autovacuum), `google_columnar_engine` and `google_db_advisor` (AlloyDB). They serve the source instance's operation (vacuum tuning, the in-memory columnar cache, index advice), hold no application data, are referenced by nothing in the application's own schema, and ship only with the managed platform, so a restore target outside it can never create them. Their schemas are dropped via `pg_dump --exclude-schema` and their `CREATE EXTENSION` / `COMMENT ON EXTENSION` statements — which are not schema-qualified, so no `pg_dump` filter reaches them — are removed from the output; whatever was excluded is named in the run's log.
+
+The list is exact names, not a `google_*` prefix match: those prefixes are not reserved, so a prefix rule would also drop a schema an application legitimately owns (`google_calendar` for a Google Calendar integration) together with its tables. Every other extension is kept and wrapped in the usual warn-and-skip `DO` block, including two kinds that are also managed-platform-only:
+
+- a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`), since that one *is* installable on a plain PostgreSQL, and
+- an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which the application's own SQL and DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it, so it warns and skips instead.
+
 you need to delete the records before importing the dump,
 `delete-{idx}-{table_name}.sql` will help you to do that.
 This sql will delete "all" related records to the extract targets.
@@ -344,6 +351,52 @@ EXWIW_SCHEMA_DIR_PATH=custom_directory bundle exec rake exwiw:schema:generate
 
 As with the CLI, a relative `schema_dir` in the config file is resolved relative to the config file's own directory.
 
+#### Safe mode (masking new columns by default)
+
+A migration that adds a column would otherwise leave `schema:generate` emitting it unmasked, so
+it starts being exported the moment the config is regenerated — before anyone has judged whether
+it holds personal data. So `schema:generate` runs in **safe mode by default**: every column the
+config does not have yet is emitted **masked** and flagged
+[`needs_mask_decision: true`](#needs_mask_decision).
+
+Columns already in the config keep whatever they say — the merge that preserves `replace_with` /
+`comment` / `ignore` preserves a resolved decision too — so in practice this marks exactly the
+columns a migration just added.
+
+```bash
+bundle exec rake exwiw:schema:generate           # safe mode
+EXWIW_NEW_COLUMNS=plain bundle exec rake exwiw:schema:generate   # opt out
+```
+
+Opting out is for the **first-time bootstrap** of a config, where every column of every table is
+new and safe mode would flag the whole thing at once. Use it nowhere else: a column committed
+under `plain` carries no flag, so nothing afterwards can tell it apart from one whose masking was
+decided.
+
+A column that has a **default of its own** is masked with that default: it is a value the column
+provably holds, and it is what the application treats as neutral, so masking a `default: true`
+flag does not quietly turn the feature off for every row in the dump. A default the database
+computes (`now()`) is not a constant and does not count, and neither does a JSON object — `{...}`
+in a mask is a column placeholder, so those fall back to `{}`. Otherwise the mask depends on the column
+type: `masked-{primary key}` for text (with `@example.com` appended when the column name mentions
+mail, so it stays a valid address), `0` for numbers, `false` for booleans, a fixed date/timestamp,
+and `{}` for JSON. Text always takes the template rather than its default, since the mask has to
+vary per row. Three kinds of
+column are flagged but deliberately **not** masked:
+
+- **The primary key, and the foreign keys/types the `belongs_tos` join on.** Masking them
+  would break the joins and leave the dump referencing rows that were never exported.
+- **Types no constant safely fits** — `uuid`, `binary`, enums, array columns (which report their
+  member type, so a scalar default would not fit), and text columns too short to hold the masked
+  value. An invalid default would fail the restore the dump feeds, which is worse than exporting
+  the column while the flag keeps the change from being merged.
+- **Columns covered by a unique index**, unless the mask varies per row (the text masks do, via
+  the primary key). A constant would collapse every row onto one value and break the restore with
+  a duplicate key.
+
+Safe mode is ActiveRecord-only for now: `schema:generate_mongoid` does not flag new fields yet,
+though the `needs_mask_decision` key itself is understood on a MongoDB field.
+
 #### Tidying stale config (`schema:tidy`)
 
 `schema:generate` adds and updates config files for the tables it finds, but it never deletes the config file of a table that has been dropped from the application. To reconcile the existing config against the current schema, run:
@@ -361,6 +414,43 @@ Because it reads the database directly, a table that still exists in the databas
 
 It respects `EXWIW_SCHEMA_DIR_PATH` and the per-database subdirectory layout in the same way as `schema:generate`. Unlike `generate`, `tidy` never adds or regenerates entries — every surviving table/column (including hand-edited `comment` / `ignore` / `replace_with`) is left untouched, so it is safe to run on a customized config. The task prints which tables and columns it removed (or that the config was already tidy). Stale `belongs_tos` are not pruned by `tidy`; rerun `schema:generate` to refresh those.
 
+#### Checking the config against the schema
+
+`schema:check` reports how the committed config differs from what the application would
+generate now — without writing anything, so it can run on a working tree it must not modify:
+
+```bash
+bundle exec rake exwiw:schema:check
+```
+
+It regenerates into a throwaway copy of the config directory (safe mode + `tidy`) and prints
+the comparison as JSON, then exits non-zero when anything needs attention:
+
+```json
+{
+  "added_tables": [],
+  "added_columns": ["users.contact_email"],
+  "removed_tables": [],
+  "removed_columns": [],
+  "changed_tables": ["users"],
+  "needs_mask_decision": ["orders.memo"]
+}
+```
+
+`added_*` / `removed_*` / `changed_tables` mean the config no longer matches the schema — run
+`schema:generate` and `schema:tidy` to reconcile it. `needs_mask_decision` lists the columns
+whose masking nobody has decided on yet (see [the flag](#needs_mask_decision)). The exit code
+makes it usable as a CI check that keeps a schema change from being merged until both are
+resolved; the JSON is stable and sorted, so it can be posted as-is. In a multi-database app each
+entry is prefixed with its database (`primary/users.email`), so the same table name in two
+databases stays distinct.
+
+Set `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` to have the same JSON written to a file, which spares a
+caller from assuming stdout carries nothing else (application boot is free to print).
+
+Like safe mode, this is ActiveRecord-only — it regenerates through `SchemaGenerator`, so a
+Mongoid config directory is not supported yet.
+
 #### Multiple databases
 
 If the application uses Rails' multiple-database support (`connects_to`), `schema:generate` buckets models by the database they connect to and writes each database's config files into its own subdirectory of the output directory, named after the database config name (`primary`, `analytics`, ...):
@@ -528,6 +618,27 @@ Individual `columns` (SQL) / `fields` (MongoDB) and `belongs_tos` entries accept
 
 The ignored entries are removed only at runtime, right after the config is loaded from file; the JSON on disk keeps them. Both `comment` and `ignore` are **preserved across `exwiw:schema:generate` / `exwiw:mongoid:schema:generate` regenerations** (the hand-edited value wins over the auto-generated config), just like `replace_with`. This applies to the MongoDB `MongodbCollectionConfig` (`fields` / `belongs_tos`) as well.
 
+### `needs_mask_decision`
+
+A column/field may also carry `needs_mask_decision: true`, marking a column whose masking
+nobody has decided on yet:
+
+```json
+{ "name": "contact_email", "replace_with": "masked-{id}@example.com", "needs_mask_decision": true }
+```
+
+Extraction ignores the key entirely — what the column exports is whatever `replace_with` /
+`ignore` say. It exists so the decision can be tracked and required: `schema:generate`'s
+[safe mode](#safe-mode-masking-new-columns-by-default) attaches it to every newly discovered
+column together with a default mask, and [`schema:check`](#checking-the-config-against-the-schema)
+reports the columns that still carry it, so CI can keep a pull request red until each one is
+resolved. Resolving it means removing the key — after keeping the mask (ideally recording why
+in `comment`), replacing it with a real masking rule, dropping `replace_with` to export the
+raw value, or setting `ignore: true`.
+
+Like `comment` / `ignore`, the on-disk state wins over regeneration: once removed,
+`schema:generate` does not bring it back.
+
 ### Polymorphic `belongs_to`
 
 A Rails polymorphic association (`belongs_to :reviewable, polymorphic: true`) does not point at a single table — the target row is selected at runtime by a type column. exwiw models this as **one `belongs_to` entry per concrete target table**, each carrying two extra fields:
@@ -819,6 +930,23 @@ absent field) is left as-is instead of being replaced by the masked literal, so
 string is a real value and is still masked. Because of this you do not need to hand-write a
 `raw_sql` `CASE WHEN ... IS NOT NULL ...` to keep NULLs.
 
+A **non-String** value (number or boolean) is used verbatim instead of being rendered as a
+template, so a column that is not text keeps its type:
+
+```jsonc
+{ "name": "score",  "replace_with": 0 }      // integer column -> SELECT emits the literal 0
+{ "name": "active", "replace_with": false }  // boolean column
+{ "name": "email",  "replace_with": "masked-{id}@example.com" }  // template, as above
+```
+
+The SQL adapters emit it as a typed literal (not concatenated into text) and the MongoDB
+adapter assigns it as-is, so the field keeps its BSON type. NULL preservation applies to both
+forms.
+
+In the String form, a `{...}` placeholder must name a column: an empty brace pair (`{}`) names
+nothing, so it is emitted literally — which is what makes `"replace_with": "{}"` a usable
+empty-JSON mask, on every adapter.
+
 #### `raw_sql`
 
 It will used instead of the original value.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -733,9 +733,12 @@ module Exwiw
 
       private def build_mask_plan(config)
         masked_fields = config.fields.each_with_object([]) do |field, acc|
-          next unless field.replace_with
+          next if field.replace_with.nil?
 
-          acc << [field.name, compile_template(field.replace_with)]
+          # A non-String replace_with (see Exwiw::MaskValue) is stored verbatim
+          # so the field keeps its BSON type; only a template compiles to segments.
+          mask = Exwiw::MaskValue.scalar?(field.replace_with) ? field.replace_with : compile_template(field.replace_with)
+          acc << [field.name, mask]
         end
         faked_fields = build_faked_fields(config)
         embedded = embedded_children_of(config).map do |child|
@@ -779,13 +782,13 @@ module Exwiw
       # masked value — matching the SQL adapters, where replace_with runs in the
       # database before the Ruby-side fake transform sees the row.
       private def apply_mask_plan!(doc, plan)
-        plan.masked_fields.each do |name, segments|
+        plan.masked_fields.each do |name, mask|
           # Preserve a NULL / absent source value instead of clobbering it into a
           # masked literal. `doc[name].nil?` is true for both an explicit nil and
           # an absent key, so an absent key is left absent (not created).
           next if doc[name].nil?
 
-          doc[name] = render_template(segments, doc)
+          doc[name] = mask.is_a?(Array) ? render_template(mask, doc) : mask
         end
         plan.faked_fields.each do |name, deriver, seed_field|
           # NULL-preserving like replace_with (an absent key stays absent). The

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -486,15 +486,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(", ")
           null_preserving(ast, column, "CONCAT(#{replaced})")
         else

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -98,6 +98,41 @@ module Exwiw
         connection.exec("EXPLAIN #{sql}").values.map(&:first).join("\n")
       end
 
+      # Extensions a managed PostgreSQL platform installs to run the *source*
+      # instance itself, which exwiw treats as out of target and leaves out of the
+      # dump entirely. Each one here satisfies both conditions:
+      #
+      #   1. it ships only with the managed platform, so no restore target outside
+      #      that platform can create it, and
+      #   2. nothing in the application's own schema or queries references it — it
+      #      is operational machinery (autovacuum tuning, the in-memory columnar
+      #      cache, index advice), not a feature the app builds on.
+      #
+      # Condition 2 is what keeps this list short. AlloyDB's application-facing
+      # extensions — `google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl` —
+      # meet condition 1 but are called from application SQL and can be named by
+      # dumped DDL (a ScaNN index is `USING scann`), so dropping their CREATE would
+      # silently strand whatever refers to them. They stay in the dump, wrapped in
+      # the usual warn-and-skip DO block, like `pglogical` and like a third-party
+      # extension pulled in as a dependency of one listed here (`google_db_advisor`
+      # requires `hypopg`, which any plain PostgreSQL can install).
+      #
+      # Deliberately an explicit list of names, not a `google_`/`alloydb_` prefix
+      # match (which is how the old `pg_extension` query filtered, before switching
+      # to a full-database pg_dump dropped that query and the filter with it): the
+      # prefixes are not reserved, so a schema an application legitimately owns —
+      # `google_calendar` for a Google Calendar integration — would be matched and
+      # dropped together with its tables. The vendor gains extensions faster than
+      # this list does; that costs a release, which is the right trade against
+      # deleting an application's own objects. Its `rds_%` / `aiven_%` arms are not
+      # carried over for the same reason: naming their members takes a dump to
+      # confirm, and adding one here is a one-line change.
+      PLATFORM_MANAGED_EXTENSIONS = %w[
+        google_columnar_engine
+        google_db_advisor
+        google_vacuum_mgmt
+      ].freeze
+
       def dump_schema(ordered_tables, output_path)
         require 'open3'
 
@@ -117,10 +152,20 @@ module Exwiw
           '--schema-only',
           '--no-owner',
           '--no-acl',
+          # An extension that needs a schema of its own puts it under its own name
+          # (Cloud SQL's google_vacuum_mgmt does; the AlloyDB two install into
+          # public), so excluding the same names covers the schema half — including
+          # any object the platform adds inside one later. The patterns are exact
+          # names: pg_dump reads them psql \d-style, where `_` is literal and only
+          # `*` globs. A pattern matching nothing is not an error (that needs
+          # --strict-names), so these are harmless against a self-hosted server.
+          *PLATFORM_MANAGED_EXTENSIONS.map { |name| "--exclude-schema=#{name}" },
           @connection_config.database_name,
         ]
         env = { 'PGPASSWORD' => @connection_config.password.to_s }
 
+        log_excluded_platform_managed_schemas
+
         @logger.debug("  Running pg_dump for the whole database (#{@connection_config.database_name})...")
         stdout, stderr, status = Open3.capture3(env, *cmd)
         unless status.success?
@@ -141,7 +186,9 @@ module Exwiw
         # EXTENSION that pg_dump emits alongside is likewise wrapped to swallow
         # undefined_object, so a skipped extension's trailing comment does not
         # abort the restore either.
-        idempotent = stdout
+        # Platform-managed extensions are stripped first: the wrapping passes below
+        # rewrite the bare CREATE/COMMENT statements this removes.
+        idempotent = strip_platform_managed_extensions(stdout)
         idempotent = DdlPostprocessor.wrap_create_type_enum_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_create_extension_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_comment_on_extension_in_do_block(idempotent)
@@ -159,6 +206,37 @@ module Exwiw
         @logger.info("  Wrote full-database schema to #{output_path} (#{ordered_tables.size} table(s) in scope for data).")
       end
 
+      # Drop the platform-managed extensions from a raw pg_dump, naming the ones
+      # actually removed so the run records what it left out rather than silently
+      # dropping it. See PLATFORM_MANAGED_EXTENSIONS.
+      private def strip_platform_managed_extensions(sql)
+        removed = DdlPostprocessor.extension_names(sql) & PLATFORM_MANAGED_EXTENSIONS
+        unless removed.empty?
+          @logger.info("  Excluded platform-managed extension(s) from the schema dump: #{removed.join(', ')}.")
+        end
+
+        DdlPostprocessor.strip_extensions(sql, PLATFORM_MANAGED_EXTENSIONS)
+      end
+
+      # Name the --exclude-schema patterns the source instance actually has, so the
+      # run records the schema half of the exclusion the way
+      # #strip_platform_managed_extensions records the extension half. Only exact
+      # names are excluded, but nothing stops an application from owning one of
+      # them, and excluding a schema takes every object inside it along — that must
+      # not happen without the log saying so. See PLATFORM_MANAGED_EXTENSIONS.
+      private def log_excluded_platform_managed_schemas
+        # The names are bare identifiers from the constant, so they need no quoting
+        # inside the array literal ANY() takes.
+        result = connection.exec_params(
+          'SELECT nspname FROM pg_namespace WHERE nspname = ANY($1) ORDER BY nspname',
+          ["{#{PLATFORM_MANAGED_EXTENSIONS.join(',')}}"],
+        )
+        excluded = result.values.map(&:first)
+        return if excluded.empty?
+
+        @logger.info("  Excluded platform-managed schema(s) from the schema dump: #{excluded.join(', ')}.")
+      end
+
       # The INSERT header for this adapter. PostgreSQL uses bare identifiers,
       # quoted only when required (reserved word / unsafe characters).
       # #to_bulk_insert / #write_inserts (SqlBulkInsert) append the value tuples
@@ -510,15 +588,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(", ")
           null_preserving(ast, column, "CONCAT(#{replaced})")
         else

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -329,15 +329,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(" || ")
           null_preserving(ast, column, "(#{replaced})")
         else

data/lib/exwiw/adapter.rb

@@ -263,6 +263,33 @@ module Exwiw
         "CASE WHEN #{qualified_name(ast.from_table_name, column.name)} IS NOT NULL THEN #{masked_expr} ELSE NULL END"
       end
 
+      # Split a `replace_with` template into the expressions it concatenates: a
+      # `{column}` placeholder becomes that column's qualified name, everything
+      # else a quoted literal. An empty brace pair names no column, so it stays
+      # literal — which is what makes `"replace_with": "{}"` an empty-JSON mask.
+      # A quote in a literal is doubled; a backslash is not, so a hand-written
+      # mask containing one means what MySQL makes of it.
+      private def mask_template_parts(ast, column)
+        column.value.scan(/#{Exwiw::MaskValue::PLACEHOLDER}|[^{}]+|[{}]/).map do |part|
+          if part.size > 1 && part.start_with?("{")
+            qualified_name(ast.from_table_name, part[1..-2])
+          else
+            "'#{part.gsub("'", "''")}'"
+          end
+        end
+      end
+
+      # A non-String `replace_with` (see Exwiw::MaskValue) is emitted as a typed
+      # literal rather than concatenated into text, so an integer column masked
+      # with `0` keeps its numeric type.
+      private def scalar_literal(value)
+        case value
+        when true then "TRUE"
+        when false then "FALSE"
+        else value.to_s
+        end
+      end
+
       # Split an outer query's WHERE clauses into the scope id-set clauses to
       # lift into a materialized derived-table JOIN (see each adapter's
       # #compile_scope_join) and the remaining plain clauses (kept in WHERE).

data/lib/exwiw/ddl_postprocessor.rb

@@ -170,6 +170,50 @@ module Exwiw
       end
     end
 
+    # Every extension name a dump installs, in the order the CREATE EXTENSION
+    # statements appear. Used to report which of them #strip_extensions is about
+    # to drop; run it on the raw dump, before any wrapping pass.
+    def extension_names(sql)
+      sql.scan(CREATE_EXTENSION_RE).map { |(name)| name.delete('"') }
+    end
+
+    # Drop every trace of the extensions named in `names` — the ones a managed
+    # PostgreSQL platform installs to run the source instance itself (see
+    # PostgresqlAdapter::PLATFORM_MANAGED_EXTENSIONS). Removed rather than merely
+    # wrapped like any other extension, because they cannot be created anywhere
+    # outside that platform, so keeping them only leaves a restore-time WARNING
+    # and objects the target will never have.
+    #
+    # Three things are emitted per extension and all three must go, or a leftover
+    # references a name that is no longer installed: pg_dump's `-- Name: <ext>;
+    # Type: EXTENSION` header block, the `CREATE EXTENSION`, and the
+    # `COMMENT ON EXTENSION` (whose own header reads `-- Name: EXTENSION <ext>;
+    # Type: COMMENT`). Run this on the raw dump, before the wrapping passes: they
+    # rewrite the bare statements this matches into DO blocks.
+    def strip_extensions(sql, names)
+      return sql if names.empty?
+
+      # Whole names only: each match is anchored by the whitespace/quote before it
+      # and a \b after, so `google_vacuum_mgmt` never matches an extension merely
+      # containing it (`not_google_vacuum_mgmt`, `google_vacuum_mgmt_v2`).
+      names_re = Regexp.union(names.map { |n| Regexp.escape(n) })
+      name = /(?:"#{names_re}"|#{names_re}\b)/
+
+      # Each removal takes the blank lines that trailed the object too, so the
+      # surrounding statements keep pg_dump's spacing instead of gaining a gap.
+      trailing_blank_lines = /(?:[ \t]*\r?\n(?:[ \t]*\r?\n)*)?/
+
+      sql = sql.gsub(
+        /^--\r?\n-- Name: (?:EXTENSION\s+)?#{name};[ \t]*Type:[ \t]*(?:EXTENSION|COMMENT);[^\n]*\n--(?:\r?\n)+/i,
+        "",
+      )
+      sql = sql.gsub(/^[ \t]*CREATE\s+EXTENSION\b(?:\s+IF\s+NOT\s+EXISTS)?\s+#{name}[^;]*;#{trailing_blank_lines}/i, "")
+      sql.gsub(
+        /^[ \t]*COMMENT\s+ON\s+EXTENSION\s+#{name}\s+IS\s+(?:'(?:[^']|'')*'|NULL)\s*;#{trailing_blank_lines}/i,
+        "",
+      )
+    end
+
     # Generate idempotent CREATE TYPE ... AS ENUM statements.
     # +enum_types+ is an Array of Hashes with keys :schema, :name, :labels.
     def create_type_enum_statements(enum_types)

data/lib/exwiw/default_mask.rb

@@ -0,0 +1,104 @@
+# frozen_string_literal: true
+
+require "date"
+
+module Exwiw
+  # The `replace_with` value safe mode attaches to a newly discovered column.
+  #
+  # Only types a constant can safely stand in for are masked: a default the
+  # column cannot hold would fail the restore the dump feeds, which is worse
+  # than exporting the column while its `needs_mask_decision` flag keeps the
+  # change from being merged.
+  module DefaultMask
+    FIXED_DATE = "2000-01-01"
+    FIXED_TIME = "2000-01-01 00:00:00"
+    EMPTY_JSON = "{}"
+    JSON_TYPES = %i[json jsonb].freeze
+
+    # Skip the mask when the column is too short to hold `masked-<primary key>`
+    # for a realistic key; the rendered length is only known per row at dump time.
+    MIN_TEXT_LIMIT = 20
+    MIN_EMAIL_LIMIT = 40
+
+    module_function
+
+    # The default mask for a column, or nil when no safe constant fits it.
+    # `primary_key` is what keeps a text mask unique per row, so without one text
+    # is left unmasked too. Under a unique index (`unique`) only a mask that
+    # varies per row is allowed, or every row would collide on restore.
+    def for(name:, type:, limit:, primary_key:, array: false, unique: false, column_default: nil)
+      return nil if array
+
+      mask =
+        case type
+        when :string, :text then text_mask(name, limit, primary_key)
+        else constant_mask(type, column_default)
+        end
+
+      return nil if mask.nil?
+      return nil if unique && !varies_per_row?(mask, primary_key)
+
+      mask
+    end
+
+    # The column's own default wins over the per-type constant: it is a value the
+    # column provably holds and the one the application treats as neutral, so
+    # masking a `default: true` flag does not turn the feature off for every row.
+    # A default the database computes (`now()`) arrives as nil and falls through.
+    def constant_mask(type, column_default)
+      from_default = default_value(type, column_default)
+      return from_default unless from_default.nil?
+
+      case type
+      when :integer, :decimal, :float then 0
+      when :boolean then false
+      when :date then FIXED_DATE
+      when :datetime, :timestamp, :time then FIXED_TIME
+      when :json, :jsonb then EMPTY_JSON
+      end
+    end
+
+    # The default as a mask value, or nil when it cannot be one. A JSON column's
+    # default is serialized as JSON whatever Ruby class it arrives as, so a string
+    # default keeps its quoting. Anything whose JSON contains an object is
+    # rejected rather than mis-parsed, so an array of objects falls back to the
+    # empty-JSON constant too.
+    def default_value(type, value)
+      return nil if value.nil?
+
+      mask = JSON_TYPES.include?(type) ? value.to_json : scalar_default(value)
+      return nil if mask.is_a?(String) && mask.match?(MaskValue::PLACEHOLDER)
+
+      mask
+    end
+
+    def scalar_default(value)
+      case value
+      when nil then nil
+      when true, false, Integer, Float, String then value
+      when Numeric then value.to_f
+      when Time, DateTime then value.strftime("%Y-%m-%d %H:%M:%S")
+      when Date then value.strftime("%Y-%m-%d")
+      when Hash, Array then value.to_json
+      end
+    end
+
+    def text_mask(name, limit, primary_key)
+      return nil if primary_key.nil?
+
+      if name.to_s.include?("mail")
+        return nil if limit && limit < MIN_EMAIL_LIMIT
+
+        "masked-{#{primary_key}}@example.com"
+      else
+        return nil if limit && limit < MIN_TEXT_LIMIT
+
+        "masked-{#{primary_key}}"
+      end
+    end
+
+    def varies_per_row?(mask, primary_key)
+      mask.is_a?(String) && mask.include?("{#{primary_key}}")
+    end
+  end
+end

data/lib/exwiw/mask_value.rb

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+module Exwiw
+  # Serdes type for a `replace_with` value: a template String with `{column}`
+  # placeholders, rendered as text, or a non-String JSON scalar used verbatim so
+  # a column that is not text keeps its type (in MongoDB, its BSON type).
+  class MaskValue < Serdes::TypeBase
+    PERMITTED = [String, Integer, Float].freeze
+
+    # What counts as a `{column}` placeholder: an empty brace pair names no
+    # column, which is what lets `{}` be an empty-JSON mask. One definition,
+    # shared by the adapters that parse a template and the generator that must
+    # avoid emitting one — keep it capture-free, since the splitter's `scan`
+    # reads whole matches.
+    PLACEHOLDER = /\{[^{}]+\}/
+
+    def permit?(value)
+      value == true || value == false || PERMITTED.any? { |type| value.is_a?(type) }
+    end
+
+    # Whether `value` is used verbatim rather than rendered as a template.
+    def self.scalar?(value)
+      !value.is_a?(String) && !value.nil?
+    end
+
+    def to_s
+      "mask_value"
+    end
+  end
+end

data/lib/exwiw/mongodb_collection_config.rb

@@ -139,10 +139,13 @@ module Exwiw
         merged.fields = passed.fields.map do |pf|
           receiver = receiver_field_by_name[pf.name]
           if receiver
-            pf.replace_with = receiver.replace_with if receiver.replace_with
+            pf.replace_with = receiver.replace_with unless receiver.replace_with.nil?
             pf.replace_with_fake_data = receiver.replace_with_fake_data if receiver.replace_with_fake_data
             pf.comment = receiver.comment if receiver.comment
             pf.ignore = receiver.ignore unless receiver.ignore.nil?
+            # Receiver wins even when unset: clearing the flag is how a human
+            # records the decision, and regeneration must not bring it back.
+            pf.needs_mask_decision = receiver.needs_mask_decision
           end
           pf
         end
@@ -162,7 +165,7 @@ module Exwiw
         fake_data = field.replace_with_fake_data
         next unless fake_data
 
-        if field.replace_with
+        unless field.replace_with.nil?
           raise ArgumentError,
                 "MongodbCollectionConfig '#{name}' field '#{field.name}': replace_with and " \
                 "replace_with_fake_data cannot be combined; use only one."

data/lib/exwiw/mongodb_field.rb

@@ -5,7 +5,7 @@ module Exwiw
     include Serdes
 
     attribute :name, String
-    attribute :replace_with, optional(String), skip_serializing_if_nil: true
+    attribute :replace_with, Serdes::OptionalType.new(MaskValue.new), skip_serializing_if_nil: true
     # Ruby-process-side masking: replace the value with a deterministic fake
     # value derived from a seed field (see FakeData / RowTransformer). Unlike the
     # SQL adapters — where replace_with runs in the database and fake data needs
@@ -23,6 +23,10 @@ module Exwiw
     # once the config is loaded (see MongodbCollectionConfig#reject_ignored_members!).
     attribute :comment, optional(String), skip_serializing_if_nil: true
     attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    # See TableColumn#needs_mask_decision.
+    attribute :needs_mask_decision,
+              Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)),
+              skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))

data/lib/exwiw/query_ast.rb

@@ -177,7 +177,7 @@ module Exwiw
         columns.map do |c|
           if c.raw_sql
             QueryAst::ColumnValue::RawSql.new(name: c.name, value: c.raw_sql)
-          elsif c.replace_with
+          elsif !c.replace_with.nil?
             QueryAst::ColumnValue::ReplaceWith.new(name: c.name, value: c.replace_with)
           else
             QueryAst::ColumnValue::Plain.new(name: c.name, value: c.name)

data/lib/exwiw/schema_check.rb

@@ -0,0 +1,125 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "json"
+require "pathname"
+require "tmpdir"
+
+module Exwiw
+  # Reports how the committed schema config differs from what the application
+  # would generate now, plus the columns still waiting for a masking decision.
+  # Regenerating into a copy rather than over the config directory lets it run
+  # on a working tree it must not modify (CI). ActiveRecord only.
+  class SchemaCheck
+    CATEGORIES = %w[
+      added_tables added_columns removed_tables removed_columns changed_tables needs_mask_decision
+    ].freeze
+
+    def self.from_rails_application(schema_dir:)
+      Rails.application.eager_load!
+      new(models: ActiveRecord::Base.descendants, schema_dir: schema_dir)
+    end
+
+    def initialize(models:, schema_dir:)
+      @models = models
+      @schema_dir = schema_dir
+    end
+
+    # The report as a plain Hash. Every list is sorted, so a given state always
+    # produces the same output (it ends up in a CI comment).
+    def run
+      committed = read_configs(@schema_dir)
+      regenerated = Dir.mktmpdir do |tmp_dir|
+        regenerate_into(tmp_dir)
+        read_configs(tmp_dir)
+      end
+
+      report = diff(committed, regenerated)
+      report["needs_mask_decision"] = flagged_columns(committed)
+      report
+    end
+
+    # Whether the report requires someone to act: the config is out of date, or
+    # a column's masking has not been decided yet.
+    def self.clean?(report)
+      CATEGORIES.all? { |category| report.fetch(category, []).empty? }
+    end
+
+    private def regenerate_into(tmp_dir)
+      FileUtils.cp_r(File.join(@schema_dir, "."), tmp_dir) if Dir.exist?(@schema_dir)
+      # Explicit: safe mode is not optional here, whatever the library default is.
+      SchemaGenerator.new(models: @models, output_dir: tmp_dir, safe_new_columns: true).generate!
+      SchemaGenerator.new(models: @models, output_dir: tmp_dir).tidy!
+    end
+
+    # Every config file under `dir`, keyed by its path relative to `dir` so the
+    # per-database subdirectories stay distinct. Hand-editing these files is the
+    # workflow this drives, so a syntax error has to name the file it is in.
+    private def read_configs(dir)
+      return {} unless Dir.exist?(dir)
+
+      Dir[File.join(dir, "**", "*.json")].each_with_object({}) do |path, acc|
+        key = Pathname.new(path).relative_path_from(Pathname.new(dir)).to_s
+        begin
+          acc[key] = JSON.parse(File.read(path))
+        rescue JSON::ParserError => e
+          raise JSON::ParserError, "invalid JSON in schema config '#{path}': #{e.message}"
+        end
+      end
+    end
+
+    private def diff(committed, regenerated)
+      report = CATEGORIES.to_h { |category| [category, []] }
+
+      (committed.keys | regenerated.keys).sort.each do |key|
+        before = committed[key]
+        after = regenerated[key]
+
+        if before.nil?
+          report["added_tables"] << table_label(key, after)
+          next
+        end
+        if after.nil?
+          report["removed_tables"] << table_label(key, before)
+          next
+        end
+        next if before == after
+
+        label = table_label(key, before)
+        report["changed_tables"] << label
+        added, removed = column_diff(before, after)
+        report["added_columns"] += added.sort.map { |column| "#{label}.#{column}" }
+        report["removed_columns"] += removed.sort.map { |column| "#{label}.#{column}" }
+      end
+
+      report
+    end
+
+    private def column_diff(before, after)
+      before_names = column_names(before)
+      after_names = column_names(after)
+      [after_names - before_names, before_names - after_names]
+    end
+
+    # `fields` is the MongoDB config's spelling of `columns`.
+    private def column_names(config)
+      (config["columns"] || config["fields"] || []).map { |column| column["name"] }
+    end
+
+    # How a table is named in the report. The database is part of the label, or
+    # the same table name in two of them collides (each has `schema_migrations`).
+    private def table_label(key, config)
+      name = config&.fetch("name", nil) || File.basename(key, ".json")
+      db = File.dirname(key)
+      db == "." ? name : "#{db}/#{name}"
+    end
+
+    private def flagged_columns(committed)
+      committed.flat_map do |key, config|
+        (config["columns"] || config["fields"] || [])
+          .select { |column| column["needs_mask_decision"] }
+          .map { |column| "#{table_label(key, config)}.#{column['name']}" }
+      end.sort
+    end
+  end
+end

data/lib/exwiw/schema_generator.rb

@@ -43,14 +43,20 @@ module Exwiw
     # belongs_to to it.
     ACTIVE_STORAGE_VARIANT_RECORDS_TABLE = "active_storage_variant_records"
 
-    def self.from_rails_application(output_dir:)
+    def self.from_rails_application(output_dir:, safe_new_columns: true)
       Rails.application.eager_load!
-      new(models: ActiveRecord::Base.descendants, output_dir: output_dir)
+      new(models: ActiveRecord::Base.descendants, output_dir: output_dir, safe_new_columns: safe_new_columns)
     end
 
-    def initialize(models:, output_dir:)
+    # `safe_new_columns` (the default) emits every column masked — as far as its
+    # type allows, see DefaultMask — and flagged `needs_mask_decision: true`.
+    # #merge lets an existing entry win, so in practice only columns a migration
+    # has just added keep that treatment. Pass false to bootstrap a config, where
+    # every column is new and flagging all of them at once is noise.
+    def initialize(models:, output_dir:, safe_new_columns: true)
       @models = models
       @output_dir = output_dir
+      @safe_new_columns = safe_new_columns
     end
 
     def generate!
@@ -239,11 +245,12 @@ module Exwiw
             columns: representative.column_names.map { |name| { name: name } },
           )
         else
+          belongs_tos = aggregate_belongs_tos(model_group)
           TableConfig.from_symbol_keys(
             name: table_name,
             primary_key: primary_key,
-            belongs_tos: aggregate_belongs_tos(model_group),
-            columns: representative.column_names.map { |name| { name: name } },
+            belongs_tos: belongs_tos,
+            columns: build_columns(representative, primary_key, belongs_tos, conn),
           )
         end
       end
@@ -251,6 +258,58 @@ module Exwiw
       tables_from_models + build_rails_managed_tables(conn)
     end
 
+    # The `columns` entries for a table: just the name, or — in safe mode — also
+    # a default mask and the `needs_mask_decision` flag. The primary key and the
+    # foreign keys/types the belongs_tos join on are flagged but never masked,
+    # since masking them would break the joins.
+    private def build_columns(representative, primary_key, belongs_tos, conn)
+      names = representative.column_names
+      return names.map { |name| { name: name } } unless @safe_new_columns
+
+      structural = belongs_tos.flat_map { |bt| [bt[:foreign_key], bt[:foreign_type]] }.compact.to_set
+      structural << primary_key if primary_key
+      columns_by_name = representative.columns.each_with_object({}) { |column, acc| acc[column.name] = column }
+      unique = unique_column_names(conn, representative.table_name)
+      defaults = representative.column_defaults
+
+      names.map do |name|
+        entry = { name: name, needs_mask_decision: true }
+        next entry if structural.include?(name)
+
+        column = columns_by_name[name]
+        mask = column && DefaultMask.for(
+          name: name,
+          type: column.type,
+          limit: column.limit,
+          primary_key: primary_key,
+          # `integer[]` reports `:integer`, so a scalar default would not fit.
+          array: column.respond_to?(:array?) && column.array?,
+          unique: unique.nil? || unique.include?(name),
+          column_default: defaults[name],
+        )
+        mask.nil? ? entry : entry.merge(replace_with: mask)
+      end
+    end
+
+    # Columns covered by a unique index, or nil when they could not be read —
+    # callers treat that as "assume every column is unique", since masking a
+    # unique column with a constant breaks the restore the dump feeds.
+    #
+    # `Array()` because an expression index reports a String (`lower((email)::text)`)
+    # rather than a list; wrapping keeps it from being walked character by
+    # character, and it simply never matches a column name.
+    private def unique_column_names(conn, table_name)
+      conn.indexes(table_name).select(&:unique).flat_map { |index| Array(index.columns) }.to_set
+    rescue StandardError => e
+      # Once per run: a systematic failure would otherwise repeat per table.
+      unless @warned_missing_indexes
+        @warned_missing_indexes = true
+        warn "exwiw: could not read the indexes of '#{table_name}' (#{e.class}); " \
+             "treating every column as unique-indexed so no constant mask is emitted."
+      end
+      nil
+    end
+
     private def concrete_models
       @models.reject(&:abstract_class?).select(&:table_exists?)
     end

data/lib/exwiw/table_column.rb

@@ -5,7 +5,7 @@ module Exwiw
     include Serdes
 
     attribute :name, String
-    attribute :replace_with, optional(String), skip_serializing_if_nil: true
+    attribute :replace_with, Serdes::OptionalType.new(MaskValue.new), skip_serializing_if_nil: true
     attribute :raw_sql, optional(String), skip_serializing_if_nil: true
     # Ruby-process-side masking modes, applied to the fetched rows by
     # RowTransformer (SQL adapters only) — unlike replace_with/raw_sql, which
@@ -17,6 +17,12 @@ module Exwiw
     # INSERT) once the config is loaded (see TableConfig#reject_ignored_members!).
     attribute :comment, optional(String), skip_serializing_if_nil: true
     attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    # Marks a column whose masking nobody has decided on yet. Extraction never
+    # reads it; `schema:generate` emits it and `schema:check` reports it, so CI
+    # can require the decision.
+    attribute :needs_mask_decision,
+              Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)),
+              skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))

data/lib/exwiw/table_config.rb

@@ -259,7 +259,7 @@ module Exwiw
       ruby_side = [column.map && "map", column.replace_with_fake_data && "replace_with_fake_data"].compact
       return if ruby_side.empty?
 
-      sql_side = [column.raw_sql && "raw_sql", column.replace_with && "replace_with"].compact
+      sql_side = [column.raw_sql && "raw_sql", !column.replace_with.nil? && "replace_with"].select { |v| v }
       if ruby_side.size > 1 || sql_side.any?
         raise ArgumentError,
               "Table '#{name}' column '#{column.name}': #{(ruby_side + sql_side).join('/')} cannot be combined; " \

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.19"
+  VERSION = "0.9.21"
 end

data/lib/exwiw.rb

@@ -10,6 +10,8 @@ require_relative "exwiw/config_file"
 require_relative "exwiw/strict_keys"
 require_relative "exwiw/belongs_to"
 require_relative "exwiw/fake_data"
+require_relative "exwiw/mask_value"
+require_relative "exwiw/default_mask"
 require_relative "exwiw/table_column"
 require_relative "exwiw/reverse_scope"
 require_relative "exwiw/batch_scope"
@@ -38,6 +40,7 @@ require_relative "exwiw/after_insert_hook"
 require_relative "exwiw/runner"
 require_relative "exwiw/explain_runner"
 require_relative "exwiw/schema_generator"
+require_relative "exwiw/schema_check"
 require_relative "exwiw/mongoid_schema_generator"
 
 begin

data/lib/tasks/exwiw.rake

@@ -13,12 +13,18 @@ namespace :exwiw do
       ENV["EXWIW_SCHEMA_DIR_PATH"] || Exwiw::ConfigFile.schema_dir || "exwiw/schema"
     end
 
+    # Safe mode is on unless EXWIW_NEW_COLUMNS=plain, which a first-time
+    # bootstrap wants: there every column is new, so safe mode would flag the
+    # whole config at once. See SchemaGenerator#initialize.
+    safe_new_columns = lambda { ENV["EXWIW_NEW_COLUMNS"] != "plain" }
+
     desc "Generate schema from application"
     task generate: :environment do
       require "exwiw"
 
       groups = Exwiw::SchemaGenerator.from_rails_application(
         output_dir: resolve_schema_dir.call,
+        safe_new_columns: safe_new_columns.call,
       ).generate!
 
       # Surface cross-database belongs_tos the generator auto-ignored: these
@@ -38,6 +44,24 @@ namespace :exwiw do
       end
     end
 
+    desc "Report how the committed schema config differs from the application, without changing it"
+    task check: :environment do
+      require "exwiw"
+
+      report = Exwiw::SchemaCheck.from_rails_application(schema_dir: resolve_schema_dir.call).run
+      json = JSON.pretty_generate(report)
+      puts json
+      # A file too, so a caller need not assume stdout carries only the JSON.
+      File.write(ENV["EXWIW_SCHEMA_CHECK_OUTPUT"], json + "\n") if ENV["EXWIW_SCHEMA_CHECK_OUTPUT"]
+
+      unless Exwiw::SchemaCheck.clean?(report)
+        $stderr.puts "exwiw: the schema config is out of date or has undecided masking; " \
+                     "run `rake exwiw:schema:generate exwiw:schema:tidy` " \
+                     "and resolve every `needs_mask_decision` column."
+        exit 1
+      end
+    end
+
     desc "Remove tables/columns from the schema config that no longer exist in the application"
     task tidy: :environment do
       require "exwiw"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.19
+  version: 0.9.21
 platform: ruby
 authors:
 - Shia
@@ -70,12 +70,14 @@ files:
 - lib/exwiw/cli.rb
 - lib/exwiw/config_file.rb
 - lib/exwiw/ddl_postprocessor.rb
+- lib/exwiw/default_mask.rb
 - lib/exwiw/determine_table_processing_order.rb
 - lib/exwiw/embedded_in.rb
 - lib/exwiw/explain_runner.rb
 - lib/exwiw/ext_json.rb
 - lib/exwiw/fake_data.rb
 - lib/exwiw/japanese_names.rb
+- lib/exwiw/mask_value.rb
 - lib/exwiw/mongo_query.rb
 - lib/exwiw/mongodb_collection_config.rb
 - lib/exwiw/mongodb_field.rb
@@ -88,6 +90,7 @@ files:
 - lib/exwiw/reverse_scope.rb
 - lib/exwiw/row_transformer.rb
 - lib/exwiw/runner.rb
+- lib/exwiw/schema_check.rb
 - lib/exwiw/schema_generator.rb
 - lib/exwiw/strict_keys.rb
 - lib/exwiw/table_column.rb