exwiw 0.9.20 → 0.9.22

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 328fbb843d5c15da7ea9f764adda96a5c58b4d24de615b9cac38ba305550ed05
-  data.tar.gz: 691448e490f27475d1ae17be240f89af5ea7c3ac54ddedfa544ca1b71517f5c1
+  metadata.gz: 3abd99db26266c75e79f46cd0fc14a2a0344308031422385ca22dd2a77ab0130
+  data.tar.gz: 9c78446793e1c91947bcf7aa6b933d623c87d004d84fbcb3df8b86886e136482
 SHA512:
-  metadata.gz: ee57c7925b2b912a3e9777947d251b8870f94572ec96b5d7fbd0a8d60dbb8aaefd33cde4e12e735b2ff2eb44b24e2f101b81e4bdfc78752f55788619a4ce5a6c
-  data.tar.gz: 9f3a2b95adb72322032b7ac4967d69e3c32a0df6fa75c64adc7f59be1f9de6b7958a8250c0abb2fd9857a5c063d8123f71c2d9678376757d1161ef34f484f368
+  metadata.gz: 0406c29183e0a896ae9de40e24622239caa4cf06b898784d1aa24633d479b0e4ee5717f8d755e5c323af1dd290e18e92f201f4d2ac6f192feb71e9c1a947eee2
+  data.tar.gz: 63e754207a3c08d97d9b9ceed684b5e112c770c02e9638dd80e9ba958fd1d9ecc96784e6d0431cb27c818e254177d62b506ea587fc1ba55d7068125c889a7fd2

data/CHANGELOG.md

@@ -2,6 +2,38 @@
 
 ## [Unreleased]
 
+## [0.9.22] - 2026-08-13
+
+### Added
+
+- **The safe-mode / check workflow now covers Mongoid applications: `schema:generate_mongoid` flags new fields, and `rake exwiw:schema:check_mongoid` / `exwiw:schema:tidy_mongoid` complete the loop.** `generate_mongoid` runs in safe mode by default (opt out with `EXWIW_NEW_COLUMNS=plain`, same as `generate`): every field the config does not have yet is emitted flagged `needs_mask_decision: true`, with a default `replace_with` where the field's Mongoid type allows one — the same per-type defaults the ActiveRecord generator emits, with `masked-{_id}` for text (the MongoDB adapter renders the placeholder per document). Structural fields are flagged but never masked: `_id`, the STI discriminator (`Model.discriminator_key`), and every `belongs_to` foreign key declared by the collection's models — including the ones the generator deliberately does not emit as `belongs_tos` (polymorphic relations, and referenced relations on embedded documents), whose foreign keys and polymorphic type fields are still live cross-collection references. Uniqueness comes from the model's `index_specifications` (any field of a unique index only gets a mask that varies per document); an index never declared on the model is invisible here, since class-level introspection is what keeps the whole workflow connection-free. `tidy_mongoid` deletes the config file of a collection no model stores into any more — the removal `generate_mongoid` can never do — using exactly the collection grouping `generate!` writes files from, so the two cannot disagree. A config declaring `embedded_in` is never deleted, whatever its name: when two embedded classes share a collection name only one of them can be generated, so the other exists only as a hand-written file named for where it lives — a name that matches no model by construction, and whose deletion would silently export those subdocuments unmasked. A file that does not parse is left in place with a warning for the same reason: its name cannot be trusted to say what it describes. `check_mongoid` mirrors `check` (same JSON report, `EXWIW_SCHEMA_CHECK_OUTPUT`, non-zero exit on drift or an unresolved flag), regenerating into a throwaway copy via `SchemaCheck.from_mongoid_application`; `SchemaCheck` itself now takes an injectable `regenerator:`, which is what makes a non-ActiveRecord schema source pluggable.
+
+- **`exwiw schema generate|check|tidy --from-db` regenerates and checks a schema config straight from a live database connection, so an application exwiw cannot load — any language, no Rails — gets the same safe-mode / check workflow.** The introspectors (MySQL and PostgreSQL) read tables, single-column primary keys, ordinal-ordered columns with types mapped to the same symbols `DefaultMask` masks by, unique indexes, and single-column foreign-key constraints; views and composite foreign keys are out (the latter skipped with a warning), a composite primary key is emitted as the same `ignore: true` signpost the ActiveRecord generator writes, and a table with **no** primary key — which ActiveRecord can never report but a database is free to have — is emitted `ignore: true` with a comment naming the fix. Safe mode matches `schema:generate` (`EXWIW_NEW_COLUMNS=plain` opts out) and `schema check` matches `rake exwiw:schema:check` (same JSON report, `EXWIW_SCHEMA_CHECK_OUTPUT`, exit 1 on drift or an unresolved `needs_mask_decision` — and exit 2 when the check could not run at all, so an unreachable database never reads as "config out of date"). One deliberate departure from the model-driven generator: **introspection only ever adds a `belongs_to`, never rewrites the list.** A foreign-key constraint is strictly weaker evidence than an application model — plenty of schemas express a relation only in application code, and hand-written belongs_tos are load-bearing extraction paths — so the existing config's relations are kept verbatim (their foreign-key columns are protected from default masks too) and constraint-discovered edges are appended; removal is `schema tidy`'s explicit job, driven by the target table actually being gone, with `ignore: true` tombstones kept. Connection settings come from the same flags/env the exporter uses; `DATABASE_PASSWORD` may be empty here (CI databases commonly run with trust auth), while `export`'s requirement is unchanged. A `primary_key` the user writes by hand on one of the `ignore: true` signpost tables survives regeneration and promotes the table to an ordinary config: the signposts exist to be acted on, and the database still reports no usable key, so the config is the only record of that decision.
+
+### Fixed
+
+- **Regenerating a Mongoid config no longer re-masks a field somebody deliberately unmasked.** `MongodbCollectionConfig#merge` kept the generated side's `replace_with` when the on-disk field had none — harmless while the generator emitted no masks, but wrong under safe mode, which proposes default masks: a field whose flag was resolved as "export raw" would silently get its mask back on the next regeneration. For a field the config already has, every decision-owned attribute (`replace_with`, `replace_with_fake_data`, `comment`, `ignore`, `needs_mask_decision`) now comes from the on-disk side even when unset; only structural facts (`name`, `mongoid_field_name`) track the model.
+
+- `schema:generate` no longer drops a hand-set `bulk_insert_chunk_size` when regenerating an existing config. The key is user-owned — the generator never emits it — but the merge took it from the freshly generated side (always empty) instead of carrying the existing value over like `filter`, `scope_column` and the other user-owned keys, so every regeneration silently deleted it.
+
+## [0.9.21] - 2026-08-06
+
+### Changed
+
+- **`schema:generate` now runs in safe mode by default, so a column a migration just added cannot start being exported before anyone has judged whether it holds personal data.** Every column the config does not have yet is emitted **masked** and flagged `needs_mask_decision: true`; existing entries keep their resolved state through the usual merge, so in practice this marks exactly the new columns. `EXWIW_NEW_COLUMNS=plain` opts out, which is what a first-time bootstrap wants — every column of every table is new there. `SchemaGenerator.new` / `.from_rails_application` default to safe too, so a caller driving the generator from its own task or script gets the same behavior. A column that has a default of its own is masked with that default — a value the column provably holds, and the one the application treats as neutral, so masking a `default: true` flag does not quietly turn the feature off for every row (a default the database computes, like `now()`, is not a constant and does not count, and neither does a JSON object — `{...}` in a mask is a column placeholder, so those fall back to `{}`). Otherwise the mask follows the column type: `masked-{primary key}` for text (with `@example.com` when the column name mentions mail), `0` for numbers, `false` for booleans, a fixed date/timestamp, `{}` for JSON; text always takes the template, since its mask has to vary per row. Three kinds of column are flagged but never masked: the primary key and the foreign keys/types the `belongs_tos` join on (masking them would break the joins and leave the dump referencing rows that were never exported); types no constant safely fits (`uuid`, `binary`, enums, array columns — which report their member type, so a scalar default would not fit — and text columns too short to hold the masked value), since a default the column cannot hold would fail the restore the dump feeds; and columns covered by a unique index unless the mask varies per row, since a constant would collapse every row onto one value and break the restore with a duplicate key. If the index list cannot be read at all, every column is treated as unique-indexed rather than risking the latter. ActiveRecord only for now — `schema:generate_mongoid` does not flag new fields yet.
+
+### Added
+
+- **`rake exwiw:schema:check` reports how the committed config differs from what the application would generate now, plus the columns still waiting for a masking decision.** Added/removed/changed tables and columns and the flagged columns come out as sorted JSON with a non-zero exit, each entry prefixed with its database in a multi-database app; `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` additionally writes the report to a file, so a caller need not assume stdout carries nothing else. It regenerates into a throwaway copy rather than over the config directory, so it runs on a working tree it must not modify — a CI check that keeps a schema change from being merged until the config is reconciled and every new column's masking is decided. ActiveRecord only, like safe mode.
+
+- **A column/field can carry `needs_mask_decision: true`, marking a masking decision nobody has made yet.** Extraction ignores the key entirely — what the column exports is whatever `replace_with` / `ignore` say — so it is purely a decision-tracking bit that tooling can attach and report, letting a check refuse to merge a schema change while any column still carries one. The on-disk state wins over regeneration: removing the key is how the decision is recorded, so a regenerated config must not restore it.
+
+- **`replace_with` accepts a non-String value, so a column that is not text can be masked without changing its type.** A String mask is rendered as a template and emitted as text, which is the wrong shape for an integer/boolean column (and would change a MongoDB field's BSON type). A JSON scalar — `"replace_with": 0`, `false`, `1.5` — is used verbatim instead: the SQL adapters emit it as a typed literal rather than concatenating it into text, and the MongoDB adapter assigns the value as-is. `{}` placeholders are only interpreted in the String form, and NULL preservation applies to both. Because `false` is a valid mask value and falsy in Ruby, the checks that decide whether a column is masked now test for nil, so a `replace_with: false` column is no longer emitted unmasked.
+
+### Fixed
+
+- **A `{}` in a `replace_with` template is emitted literally instead of compiling to a reference to a column with no name.** The SQL adapters read `{...}` as a placeholder even when it was empty, so `"replace_with": "{}"` compiled to `CONCAT(t."")` (`t.``` on mysql, `(t."")` on sqlite) and the extraction failed on that table. The MongoDB adapter never had the bug — its placeholder pattern requires at least one character — so the same config meant two different things per adapter. A brace pair with nothing between it names no column and is now a literal everywhere, which is what makes `{}` usable as an empty-JSON mask (the default safe-mode mask for a `json`/`jsonb` column). Templates that name a real column are unaffected, byte for byte.
+
 ## [0.9.20] - 2026-08-05
 
 ### Changed

data/README.md

@@ -19,7 +19,7 @@ You want to export only the data you want to export.
 
 - Export the full list of INSERT sql for the specified conditions.
 - Provide serveral masking options for sensitive columns.
-- Provide config generator for ActiveRecord.
+- Provide config generator for ActiveRecord, for Mongoid, and from a live database connection (any application, any language).
 
 ## Installation
 
@@ -56,10 +56,11 @@ driver implements that auth handshake itself and sidesteps the issue.
 
 ## Usage
 
-exwiw has two subcommands:
+exwiw has three subcommands:
 
 - `export` (default) — generate INSERT/COPY SQL files. If the subcommand is omitted, `export` is assumed.
 - `explain` — print each query `export` would run together with its `EXPLAIN` output. SQL adapters compile the SELECT without executing it; mongodb runs the server's explain (defaulting to the execution-free `queryPlanner`).
+- `schema generate|check|tidy --from-db` — maintain the schema config by reading a live database, for applications that cannot be loaded to generate it from their models. See [Non-Rails applications](#non-rails-applications-exwiw-schema---from-db).
 
 ### `exwiw export`
 
@@ -326,7 +327,7 @@ Notes:
 - **Database connection settings stay on the CLI/environment.** `host`, `port`, `user`, `database`, `uri`, and `password` are **rejected** in the config file (exwiw exits with an error). `adapter` is the one connection-related key that *is* allowed in the file.
 - **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
 - Unknown keys are rejected so a typo surfaces immediately.
-- Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
+- Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain` or `schema`, so a single config file can be shared by every subcommand.
 - `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [MongoDB support](docs/mongodb.md#exwiw-explain-verbosity).
 - `mongodb_query_timeout_ms` sets the global, server-enforced query timeout (mongodb only); the `--mongodb-query-timeout-ms` CLI flag overrides it. Ignored by the SQL adapters. See [MongoDB support](docs/mongodb.md).
 
@@ -351,6 +352,65 @@ EXWIW_SCHEMA_DIR_PATH=custom_directory bundle exec rake exwiw:schema:generate
 
 As with the CLI, a relative `schema_dir` in the config file is resolved relative to the config file's own directory.
 
+An application with more than one schema source — ActiveRecord models and Mongoid documents, say —
+should give each source its own directory (`EXWIW_SCHEMA_DIR_PATH`), because every task judges the
+whole directory against the one source it reads. Left sharing a directory, `tidy` / `check` see the
+other source's configs as belonging to tables and collections that no longer exist, and report or
+remove them.
+
+#### Safe mode (masking new columns by default)
+
+A migration that adds a column would otherwise leave `schema:generate` emitting it unmasked, so
+it starts being exported the moment the config is regenerated — before anyone has judged whether
+it holds personal data. So `schema:generate` runs in **safe mode by default**: every column the
+config does not have yet is emitted **masked** and flagged
+[`needs_mask_decision: true`](#needs_mask_decision).
+
+Columns already in the config keep whatever they say — the merge that preserves `replace_with` /
+`comment` / `ignore` preserves a resolved decision too — so in practice this marks exactly the
+columns a migration just added.
+
+```bash
+bundle exec rake exwiw:schema:generate           # safe mode
+EXWIW_NEW_COLUMNS=plain bundle exec rake exwiw:schema:generate   # opt out
+```
+
+Opting out is for the **first-time bootstrap** of a config, where every column of every table is
+new and safe mode would flag the whole thing at once. Use it nowhere else: a column committed
+under `plain` carries no flag, so nothing afterwards can tell it apart from one whose masking was
+decided.
+
+A column that has a **default of its own** is masked with that default: it is a value the column
+provably holds, and it is what the application treats as neutral, so masking a `default: true`
+flag does not quietly turn the feature off for every row in the dump. A default the database
+computes (`now()`) is not a constant and does not count, and neither does a JSON object — `{...}`
+in a mask is a column placeholder, so those fall back to `{}`. Otherwise the mask depends on the column
+type: `masked-{primary key}` for text (with `@example.com` appended when the column name mentions
+mail, so it stays a valid address), `0` for numbers, `false` for booleans, a fixed date/timestamp,
+and `{}` for JSON. Text always takes the template rather than its default, since the mask has to
+vary per row. Three kinds of
+column are flagged but deliberately **not** masked:
+
+- **The primary key, and the foreign keys/types the `belongs_tos` join on.** Masking them
+  would break the joins and leave the dump referencing rows that were never exported.
+- **Types no constant safely fits** — `uuid`, `binary`, enums, array columns (which report their
+  member type, so a scalar default would not fit), and text columns too short to hold the masked
+  value. An invalid default would fail the restore the dump feeds, which is worse than exporting
+  the column while the flag keeps the change from being merged.
+- **Columns covered by a unique index**, unless the mask varies per row (the text masks do, via
+  the primary key). A constant would collapse every row onto one value and break the restore with
+  a duplicate key.
+
+`schema:generate_mongoid` runs in safe mode too, on the same `EXWIW_NEW_COLUMNS=plain` opt-out. The
+masks come from the Mongoid field type (`String`, `Integer`, `Float`, `BigDecimal`,
+`Mongoid::Boolean`, `Date`, `Time` / `DateTime` / `ActiveSupport::TimeWithZone`); a field of any
+other type — `Hash`, `Array`, a typeless field, a BSON type — is flagged but not masked, as is a
+field covered by a unique index unless its mask varies per document. The structural fields are the
+`_id` primary key, the STI discriminator (`_type`) and every `belongs_to` foreign key of the
+collection: flagged, never masked. The foreign keys are read from the models, so the ones the
+config itself drops (a polymorphic `belongs_to`, a `belongs_to` on an embedded document) are
+covered too.
+
 #### Tidying stale config (`schema:tidy`)
 
 `schema:generate` adds and updates config files for the tables it finds, but it never deletes the config file of a table that has been dropped from the application. To reconcile the existing config against the current schema, run:
@@ -368,6 +428,47 @@ Because it reads the database directly, a table that still exists in the databas
 
 It respects `EXWIW_SCHEMA_DIR_PATH` and the per-database subdirectory layout in the same way as `schema:generate`. Unlike `generate`, `tidy` never adds or regenerates entries — every surviving table/column (including hand-edited `comment` / `ignore` / `replace_with`) is left untouched, so it is safe to run on a customized config. The task prints which tables and columns it removed (or that the config was already tidy). Stale `belongs_tos` are not pruned by `tidy`; rerun `schema:generate` to refresh those.
 
+#### Checking the config against the schema
+
+`schema:check` reports how the committed config differs from what the application would
+generate now — without writing anything, so it can run on a working tree it must not modify:
+
+```bash
+bundle exec rake exwiw:schema:check
+```
+
+It regenerates into a throwaway copy of the config directory (safe mode + `tidy`) and prints
+the comparison as JSON, then exits non-zero when anything needs attention:
+
+```json
+{
+  "added_tables": [],
+  "added_columns": ["users.contact_email"],
+  "removed_tables": [],
+  "removed_columns": [],
+  "changed_tables": ["users"],
+  "needs_mask_decision": ["orders.memo"]
+}
+```
+
+`added_*` / `removed_*` / `changed_tables` mean the config no longer matches the schema — run
+`schema:generate` and `schema:tidy` to reconcile it. `needs_mask_decision` lists the columns
+whose masking nobody has decided on yet (see [the flag](#needs_mask_decision)). The exit code
+makes it usable as a CI check that keeps a schema change from being merged until both are
+resolved; the JSON is stable and sorted, so it can be posted as-is. In a multi-database app each
+entry is prefixed with its database (`primary/users.email`), so the same table name in two
+databases stays distinct.
+
+Set `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` to have the same JSON written to a file, which spares a
+caller from assuming stdout carries nothing else (application boot is free to print).
+
+A Mongoid config directory has its own task, `schema:check_mongoid`, with the same output,
+the same `EXWIW_SCHEMA_CHECK_OUTPUT` file and the same exit code — it just regenerates through
+`MongoidSchemaGenerator` (safe mode + `tidy_mongoid`) instead. Collections and fields are
+reported under the same keys as tables and columns. An application that cannot be loaded to
+generate from its models at all can run the same check against its database instead: see
+[Non-Rails applications](#non-rails-applications-exwiw-schema---from-db).
+
 #### Multiple databases
 
 If the application uses Rails' multiple-database support (`connects_to`), `schema:generate` buckets models by the database they connect to and writes each database's config files into its own subdirectory of the output directory, named after the database config name (`primary`, `analytics`, ...):
@@ -396,10 +497,82 @@ For MongoDB applications backed by [Mongoid](https://www.mongodb.com/docs/mongoi
 
 ```bash
 bundle exec rake exwiw:schema:generate_mongoid
+bundle exec rake exwiw:schema:tidy_mongoid   # delete the config of a collection no model stores into
+bundle exec rake exwiw:schema:check_mongoid  # report the difference without changing anything
 ```
 
 What it derives from each model (fields, `belongs_tos`, `embedded_in`, STI handling), how to annotate constructs exwiw cannot represent with `ignore` / `ignore_type`, and the `EXWIW_SKIP_UNSUPPORTED=1` bootstrap flag are all documented in [MongoDB support](docs/mongodb.md#generating-config-from-mongoid-models).
 
+`tidy_mongoid` reconciles against the *models* rather than a live connection (MongoDB has no schema to read, and a collection exists only once something is written to it): a config file whose collection no model stores into any more is deleted, and nothing else is touched — fields already track the models through `generate_mongoid`. `check_mongoid` runs both into a throwaway copy, so it never writes to the working tree.
+
+#### Non-Rails applications (`exwiw schema ... --from-db`)
+
+The rake tasks above read the application's models, which requires loading the application — so
+they are only available where that is possible. For an application written in any other language
+(or a Ruby one exwiw cannot boot), the same three operations are available on the CLI, reading
+the **database** instead of the models:
+
+```bash
+# generate / refresh the config from the live schema
+exwiw schema generate --from-db -a postgresql -h db.example.com -p 5432 -u app --database=app --schema-dir=exwiw/schema
+
+# report how the committed config differs from the database (exits 1 when it needs work)
+exwiw schema check --from-db -a postgresql -h db.example.com -p 5432 -u app --database=app --schema-dir=exwiw/schema
+
+# remove tables/columns/relations the database no longer has
+exwiw schema tidy --from-db -a postgresql -h db.example.com -p 5432 -u app --database=app --schema-dir=exwiw/schema
+```
+
+- `--from-db` is **required**: the schema source is always stated explicitly rather than inferred.
+- mysql and postgresql only. sqlite is not supported, and a MongoDB schema lives in the
+  application rather than the database (use `schema:generate_mongoid`).
+- The usual connection flags and `DATABASE_PASSWORD` apply, and `adapter` / `schema_dir` may come
+  from [the config file](#config-file-exwiwyml) instead (`--schema-dir` wins). Unlike `export`,
+  an empty or absent `DATABASE_PASSWORD` is accepted, since these commands are commonly pointed
+  at a CI database that runs with trust authentication.
+- `generate` creates the schema directory if it does not exist; `check` and `tidy` require it.
+- Everything else behaves as the rake tasks do: [safe mode](#safe-mode-masking-new-columns-by-default)
+  is on unless `EXWIW_NEW_COLUMNS=plain`, `check` prints the same JSON report, honours
+  `EXWIW_SCHEMA_CHECK_OUTPUT`, and exits 1 when the config needs attention. A check that could
+  not *run* (an unreachable database, a malformed config) exits with a different status, so CI
+  can tell the two apart.
+- One run covers one database — the connection addresses one — so the files are written flat into
+  the schema directory. There is no per-database subdirectory layout here; a second database is a
+  second run against a second connection.
+
+The database is read through its catalog only: tables, columns and their types/defaults, primary
+keys, unique indexes and foreign keys. Views are skipped, since they hold no rows of their own,
+and a table with no primary key is emitted with `ignore: true` and a comment saying what to add
+to export it (exwiw identifies and joins rows by primary key). Following that advice sticks: a
+`primary_key` written by hand — on a table the database reports none for, or one column of a
+composite key that identifies a row on its own — is kept by later runs, which then treat the table
+as an ordinary one rather than re-imposing the signpost `type` / `comment`.
+
+**`belongs_tos` are only ever added, never rewritten.** A foreign-key constraint is weaker
+evidence than an application model: plenty of schemas express a relation only in application code,
+and a `belongs_to` is the path extraction follows to reach a table, so silently dropping one
+narrows the dump. Regeneration therefore keeps every relation the config already declares —
+in its existing order, with its `comment` / `ignore` / `ignore_type` — and appends only the
+foreign-key-backed relations that are not there yet. A hand-written relation's foreign-key column
+is treated as structural too, so safe mode never masks it. Removing a relation is `tidy`'s job:
+it drops a `belongs_to` whose target table no longer exists in the database (an `ignore: true`
+entry is kept, since it records a decision), alongside the tables and columns that are gone.
+
+So a relation the database does not know about is declared once, by hand, and survives from then
+on:
+
+```json
+{
+    "name": "orders",
+    "primary_key": "id",
+    "belongs_tos": [{
+        "table_name": "buyers",
+        "foreign_key": "buyer_id",
+        "comment": "enforced in the application; no foreign key in the database"
+    }]
+}
+```
+
 ### Configuration
 
 This is an example of the one table schema:
@@ -535,6 +708,28 @@ Individual `columns` (SQL) / `fields` (MongoDB) and `belongs_tos` entries accept
 
 The ignored entries are removed only at runtime, right after the config is loaded from file; the JSON on disk keeps them. Both `comment` and `ignore` are **preserved across `exwiw:schema:generate` / `exwiw:mongoid:schema:generate` regenerations** (the hand-edited value wins over the auto-generated config), just like `replace_with`. This applies to the MongoDB `MongodbCollectionConfig` (`fields` / `belongs_tos`) as well.
 
+### `needs_mask_decision`
+
+A column/field may also carry `needs_mask_decision: true`, marking a column whose masking
+nobody has decided on yet:
+
+```json
+{ "name": "contact_email", "replace_with": "masked-{id}@example.com", "needs_mask_decision": true }
+```
+
+Extraction ignores the key entirely — what the column exports is whatever `replace_with` /
+`ignore` say. It exists so the decision can be tracked and required:
+[safe mode](#safe-mode-masking-new-columns-by-default) attaches it to every newly discovered
+column/field together with a default mask, and
+[`schema:check`](#checking-the-config-against-the-schema) — `schema:check_mongoid` for a Mongoid
+config — reports the ones that still carry it, so CI can keep a pull request red until each is
+resolved. Resolving it means removing the key — after keeping the mask (ideally recording why
+in `comment`), replacing it with a real masking rule, dropping `replace_with` to export the
+raw value, or setting `ignore: true`.
+
+Like `comment` / `ignore`, the on-disk state wins over regeneration: once removed,
+`schema:generate` does not bring it back.
+
 ### Polymorphic `belongs_to`
 
 A Rails polymorphic association (`belongs_to :reviewable, polymorphic: true`) does not point at a single table — the target row is selected at runtime by a type column. exwiw models this as **one `belongs_to` entry per concrete target table**, each carrying two extra fields:
@@ -826,6 +1021,23 @@ absent field) is left as-is instead of being replaced by the masked literal, so
 string is a real value and is still masked. Because of this you do not need to hand-write a
 `raw_sql` `CASE WHEN ... IS NOT NULL ...` to keep NULLs.
 
+A **non-String** value (number or boolean) is used verbatim instead of being rendered as a
+template, so a column that is not text keeps its type:
+
+```jsonc
+{ "name": "score",  "replace_with": 0 }      // integer column -> SELECT emits the literal 0
+{ "name": "active", "replace_with": false }  // boolean column
+{ "name": "email",  "replace_with": "masked-{id}@example.com" }  // template, as above
+```
+
+The SQL adapters emit it as a typed literal (not concatenated into text) and the MongoDB
+adapter assigns it as-is, so the field keeps its BSON type. NULL preservation applies to both
+forms.
+
+In the String form, a `{...}` placeholder must name a column: an empty brace pair (`{}`) names
+nothing, so it is emitted literally — which is what makes `"replace_with": "{}"` a usable
+empty-JSON mask, on every adapter.
+
 #### `raw_sql`
 
 It will used instead of the original value.

data/docs/mongodb.md

@@ -149,6 +149,19 @@ Models in an inheritance hierarchy whose subclasses share the base's collection
 
 Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, `bulk_insert_chunk_size`, and `query_timeout_ms` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [Output](#output)). Polymorphic `belongs_to` is not yet expanded by this task.
 
+### Safe mode, `tidy_mongoid` and `check_mongoid`
+
+Like the ActiveRecord task, `generate_mongoid` runs in [safe mode](../README.md#safe-mode-masking-new-columns-by-default) by default (`EXWIW_NEW_COLUMNS=plain` opts out for a first bootstrap): a field the config does not have yet is emitted masked — as far as its Mongoid type allows — and flagged `needs_mask_decision: true`, so a field added to a model does not start being exported before somebody has judged whether it holds personal data. The `_id` primary key, the `_type` STI discriminator and every `belongs_to` foreign key are flagged but never masked (masking them would rewrite the very references the dump is assembled from), and neither is a field whose type no constant fits (`Hash`, `Array`, a typeless field, a BSON type) or one covered by a unique index unless its mask varies per document. A field's own `default:` is preferred over the per-type constant. A decision recorded on disk — a mask kept, replaced, dropped, or the flag removed — wins over every later regeneration.
+
+```bash
+bundle exec rake exwiw:schema:tidy_mongoid   # delete the config of a collection no model stores into
+bundle exec rake exwiw:schema:check_mongoid  # report the difference, writing nothing
+```
+
+`tidy_mongoid` is the counterpart `generate_mongoid` cannot be: it deletes the config file of a collection no model stores into any more. It reconciles against the **models**, not a live connection — MongoDB has no schema to read, and a collection exists only once a document is written to it — using exactly the grouping `generate_mongoid` writes files from, embedded collections included. Fields are left alone, since `generate_mongoid` already tracks them through the model.
+
+`check_mongoid` mirrors [`schema:check`](../README.md#checking-the-config-against-the-schema): it regenerates (safe mode + tidy) into a throwaway copy of the config directory, prints the same sorted JSON report — collections and fields under the same keys as tables and columns — honors `EXWIW_SCHEMA_CHECK_OUTPUT`, and exits non-zero while anything is out of date or undecided, so it can gate a pull request. It regenerates in strict mode: a newly introduced construct exwiw cannot represent fails the task rather than reporting clean.
+
 By default the task **aborts** when a model uses a construct exwiw cannot represent: a `belongs_to` whose target class can no longer be resolved (a stale relation left behind after its model was removed), or a polymorphic / self-referential-cyclic / ambiguous / unresolvable-parent `embedded_in` (see the cases above).
 
 ### Honoring an explicit `ignore` (the recommended way to keep these out)

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -733,9 +733,12 @@ module Exwiw
 
       private def build_mask_plan(config)
         masked_fields = config.fields.each_with_object([]) do |field, acc|
-          next unless field.replace_with
+          next if field.replace_with.nil?
 
-          acc << [field.name, compile_template(field.replace_with)]
+          # A non-String replace_with (see Exwiw::MaskValue) is stored verbatim
+          # so the field keeps its BSON type; only a template compiles to segments.
+          mask = Exwiw::MaskValue.scalar?(field.replace_with) ? field.replace_with : compile_template(field.replace_with)
+          acc << [field.name, mask]
         end
         faked_fields = build_faked_fields(config)
         embedded = embedded_children_of(config).map do |child|
@@ -779,13 +782,13 @@ module Exwiw
       # masked value — matching the SQL adapters, where replace_with runs in the
       # database before the Ruby-side fake transform sees the row.
       private def apply_mask_plan!(doc, plan)
-        plan.masked_fields.each do |name, segments|
+        plan.masked_fields.each do |name, mask|
           # Preserve a NULL / absent source value instead of clobbering it into a
           # masked literal. `doc[name].nil?` is true for both an explicit nil and
           # an absent key, so an absent key is left absent (not created).
           next if doc[name].nil?
 
-          doc[name] = render_template(segments, doc)
+          doc[name] = mask.is_a?(Array) ? render_template(mask, doc) : mask
         end
         plan.faked_fields.each do |name, deriver, seed_field|
           # NULL-preserving like replace_with (an absent key stays absent). The

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -486,15 +486,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(", ")
           null_preserving(ast, column, "CONCAT(#{replaced})")
         else

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -588,15 +588,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(", ")
           null_preserving(ast, column, "CONCAT(#{replaced})")
         else

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -329,15 +329,12 @@ module Exwiw
         when Exwiw::QueryAst::ColumnValue::RawSql
           column.value
         when Exwiw::QueryAst::ColumnValue::ReplaceWith
-          parts = column.value.scan(/[^{}]+|\{[^{}]*\}/).map do |part|
-            if part.start_with?('{')
-              name = part[1..-2]
-              qualified_name(ast.from_table_name, name)
-            else
-              "'#{part}'"
-            end
+          if Exwiw::MaskValue.scalar?(column.value)
+            return null_preserving(ast, column, scalar_literal(column.value))
           end
 
+          parts = mask_template_parts(ast, column)
+
           replaced = parts.join(" || ")
           null_preserving(ast, column, "(#{replaced})")
         else

data/lib/exwiw/adapter.rb

@@ -263,6 +263,33 @@ module Exwiw
         "CASE WHEN #{qualified_name(ast.from_table_name, column.name)} IS NOT NULL THEN #{masked_expr} ELSE NULL END"
       end
 
+      # Split a `replace_with` template into the expressions it concatenates: a
+      # `{column}` placeholder becomes that column's qualified name, everything
+      # else a quoted literal. An empty brace pair names no column, so it stays
+      # literal — which is what makes `"replace_with": "{}"` an empty-JSON mask.
+      # A quote in a literal is doubled; a backslash is not, so a hand-written
+      # mask containing one means what MySQL makes of it.
+      private def mask_template_parts(ast, column)
+        column.value.scan(/#{Exwiw::MaskValue::PLACEHOLDER}|[^{}]+|[{}]/).map do |part|
+          if part.size > 1 && part.start_with?("{")
+            qualified_name(ast.from_table_name, part[1..-2])
+          else
+            "'#{part.gsub("'", "''")}'"
+          end
+        end
+      end
+
+      # A non-String `replace_with` (see Exwiw::MaskValue) is emitted as a typed
+      # literal rather than concatenated into text, so an integer column masked
+      # with `0` keeps its numeric type.
+      private def scalar_literal(value)
+        case value
+        when true then "TRUE"
+        when false then "FALSE"
+        else value.to_s
+        end
+      end
+
       # Split an outer query's WHERE clauses into the scope id-set clauses to
       # lift into a materialized derived-table JOIN (see each adapter's
       # #compile_scope_join) and the remaining plain clauses (kept in WHERE).