exwiw 0.9.21 → 0.9.23

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 61f6374ae972477087c63744e9a3702acc795872f7d08599cef69bc7898563db
-  data.tar.gz: 5a140799d94c5804fec5988c0643acbcc79c33ed442e7be36fe2b65bcce14a3b
+  metadata.gz: 9cbe5a94be08902db6990812f48eb07b3aa5cae7a507acb0f8c3cf1f90edc175
+  data.tar.gz: 19b92d8feee9e874135fc0c9f833d290aacdb1be35c6bcd3da72c7c212352327
 SHA512:
-  metadata.gz: 78154be467c8920c5011f527957635145089c07bba5268ce4ebe7ac3272622243acb36b05bd1b538740f76a9167dafe48498a022b458bdd11610544d2c10d446
-  data.tar.gz: 7aea16379a332986f78b18f07a1c6c258745c407a804510de1451c17324c1533f7f17fda6ed725bc4e982e5b5824395fd96f01ffa30fc46c90b0a2e10d25c033
+  metadata.gz: 75e631242a39ba72bd16ed7d28435f935c9d591d89888632f5fd835b2b727d6f497a976b91bbcc69bdbc8c85fbfb68d2265488d3b07b19baa37001a7d832d173
+  data.tar.gz: abcfacf75eff9c74324fae77915f10b86ab7258f5251bf5fa7adbe73a3af318fd06ffc4e791e7abc00439b1007901d43e80865ff0394ada7eb8deba040707d1d

data/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ## [Unreleased]
 
+## [0.9.23] - 2026-08-13
+
+### Fixed
+
+- **A collection stored into by both a top-level and an embedded model is now generated as the top-level collection it is, instead of silently ceasing to be dumped.** `MongoidSchemaGenerator` treated a collection group as embedded as soon as *any* model in it was — and an embedded class's collection name derives from its class name alone, so it collides with any top-level model that stores into a collection of that name, with nothing declaring the two to be related and neither side wrong. The emitted config then carried an `embedded_in` and an empty `belongs_tos`, with three consequences: `MongodbAdapter#dumpable?` (`!embedded?`) silently never dumped the collection's root documents; the top-level models' `belongs_tos` — the extraction paths that reach everything scoped below them — disappeared from the config; and a hand-maintained top-level config could not survive regeneration either, because `MongodbCollectionConfig#merge` always takes the freshly generated `embedded_in`, so `schema:check_mongoid` reported that file as changed on every run with no way to resolve it. Such a collection is genuinely top-level — it has root documents that must be dumped — so it is now built from its **root** models alone (their fields, their `belongs_tos`, and safe mode's structural/unique sets), with no `embedded_in`, and the collision is named on stderr instead: the embedded documents of the same name are **not** covered by that config and are masked, as before, through a config written by hand with an `embedded_in` and a name of its own (generation and `tidy_mongoid` both leave `embedded_in` configs alone, so such a file is stable). The collision is deliberately *not* recorded as a generated `comment` on the config, since a generated comment wins the merge and would overwrite — on every run — the note a user wrote about this very situation. Only a *root* model counts as evidence of a root collection, which is the second shape this distinguishes: an **embedded family under a plain base class** (`Address` holding the shared fields and declaring no `embedded_in`, `BillingAddress < Address` declaring one and inheriting the collection name) is not mixed at all. `embedded?` answers "does this class declare an `embedded_in`", not "does this collection have root documents", and a shared base stores nothing at the root — its fields describe subdocuments. Such a family keeps generating exactly the embedded config it always did (the same field union, in the same order, with the embedding derived from a class that actually declares one), because flipping it to a top-level config would delete the config masking those subdocuments: the same worst-case failure in the opposite direction. A non-embedded model whose embedded descendants are in the group is therefore always read as their base class, conservatively so even if the application also stored root documents under it — that keeps the behavior (and the limitation) the generator always had rather than trading a silent masking regression for a silent dumping one. Collections stored into only by top-level models, or only by embedded ones, are unaffected, including the unrepresentable embedding shapes that abort or, under `EXWIW_SKIP_UNSUPPORTED=1`, emit an `ignore: true` config. Related: a model declared `store_in collection: nil` — how an application says a document class is never persisted and only wants Mongoid's casting — is now skipped entirely instead of contributing to a config for the nameless group every such class collapses into, which describes nothing and, as a top-level config, would instruct the dump to read a collection with no name. They are dropped where the generator selects the models it describes, so they leave `tidy_mongoid`'s live-collection set as well.
+
+## [0.9.22] - 2026-08-13
+
+### Added
+
+- **The safe-mode / check workflow now covers Mongoid applications: `schema:generate_mongoid` flags new fields, and `rake exwiw:schema:check_mongoid` / `exwiw:schema:tidy_mongoid` complete the loop.** `generate_mongoid` runs in safe mode by default (opt out with `EXWIW_NEW_COLUMNS=plain`, same as `generate`): every field the config does not have yet is emitted flagged `needs_mask_decision: true`, with a default `replace_with` where the field's Mongoid type allows one — the same per-type defaults the ActiveRecord generator emits, with `masked-{_id}` for text (the MongoDB adapter renders the placeholder per document). Structural fields are flagged but never masked: `_id`, the STI discriminator (`Model.discriminator_key`), and every `belongs_to` foreign key declared by the collection's models — including the ones the generator deliberately does not emit as `belongs_tos` (polymorphic relations, and referenced relations on embedded documents), whose foreign keys and polymorphic type fields are still live cross-collection references. Uniqueness comes from the model's `index_specifications` (any field of a unique index only gets a mask that varies per document); an index never declared on the model is invisible here, since class-level introspection is what keeps the whole workflow connection-free. `tidy_mongoid` deletes the config file of a collection no model stores into any more — the removal `generate_mongoid` can never do — using exactly the collection grouping `generate!` writes files from, so the two cannot disagree. A config declaring `embedded_in` is never deleted, whatever its name: when two embedded classes share a collection name only one of them can be generated, so the other exists only as a hand-written file named for where it lives — a name that matches no model by construction, and whose deletion would silently export those subdocuments unmasked. A file that does not parse is left in place with a warning for the same reason: its name cannot be trusted to say what it describes. `check_mongoid` mirrors `check` (same JSON report, `EXWIW_SCHEMA_CHECK_OUTPUT`, non-zero exit on drift or an unresolved flag), regenerating into a throwaway copy via `SchemaCheck.from_mongoid_application`; `SchemaCheck` itself now takes an injectable `regenerator:`, which is what makes a non-ActiveRecord schema source pluggable.
+
+- **`exwiw schema generate|check|tidy --from-db` regenerates and checks a schema config straight from a live database connection, so an application exwiw cannot load — any language, no Rails — gets the same safe-mode / check workflow.** The introspectors (MySQL and PostgreSQL) read tables, single-column primary keys, ordinal-ordered columns with types mapped to the same symbols `DefaultMask` masks by, unique indexes, and single-column foreign-key constraints; views and composite foreign keys are out (the latter skipped with a warning), a composite primary key is emitted as the same `ignore: true` signpost the ActiveRecord generator writes, and a table with **no** primary key — which ActiveRecord can never report but a database is free to have — is emitted `ignore: true` with a comment naming the fix. Safe mode matches `schema:generate` (`EXWIW_NEW_COLUMNS=plain` opts out) and `schema check` matches `rake exwiw:schema:check` (same JSON report, `EXWIW_SCHEMA_CHECK_OUTPUT`, exit 1 on drift or an unresolved `needs_mask_decision` — and exit 2 when the check could not run at all, so an unreachable database never reads as "config out of date"). One deliberate departure from the model-driven generator: **introspection only ever adds a `belongs_to`, never rewrites the list.** A foreign-key constraint is strictly weaker evidence than an application model — plenty of schemas express a relation only in application code, and hand-written belongs_tos are load-bearing extraction paths — so the existing config's relations are kept verbatim (their foreign-key columns are protected from default masks too) and constraint-discovered edges are appended; removal is `schema tidy`'s explicit job, driven by the target table actually being gone, with `ignore: true` tombstones kept. Connection settings come from the same flags/env the exporter uses; `DATABASE_PASSWORD` may be empty here (CI databases commonly run with trust auth), while `export`'s requirement is unchanged. A `primary_key` the user writes by hand on one of the `ignore: true` signpost tables survives regeneration and promotes the table to an ordinary config: the signposts exist to be acted on, and the database still reports no usable key, so the config is the only record of that decision.
+
+### Fixed
+
+- **Regenerating a Mongoid config no longer re-masks a field somebody deliberately unmasked.** `MongodbCollectionConfig#merge` kept the generated side's `replace_with` when the on-disk field had none — harmless while the generator emitted no masks, but wrong under safe mode, which proposes default masks: a field whose flag was resolved as "export raw" would silently get its mask back on the next regeneration. For a field the config already has, every decision-owned attribute (`replace_with`, `replace_with_fake_data`, `comment`, `ignore`, `needs_mask_decision`) now comes from the on-disk side even when unset; only structural facts (`name`, `mongoid_field_name`) track the model.
+
+- `schema:generate` no longer drops a hand-set `bulk_insert_chunk_size` when regenerating an existing config. The key is user-owned — the generator never emits it — but the merge took it from the freshly generated side (always empty) instead of carrying the existing value over like `filter`, `scope_column` and the other user-owned keys, so every regeneration silently deleted it.
+
 ## [0.9.21] - 2026-08-06
 
 ### Changed

data/README.md

@@ -19,7 +19,7 @@ You want to export only the data you want to export.
 
 - Export the full list of INSERT sql for the specified conditions.
 - Provide serveral masking options for sensitive columns.
-- Provide config generator for ActiveRecord.
+- Provide config generator for ActiveRecord, for Mongoid, and from a live database connection (any application, any language).
 
 ## Installation
 
@@ -56,10 +56,11 @@ driver implements that auth handshake itself and sidesteps the issue.
 
 ## Usage
 
-exwiw has two subcommands:
+exwiw has three subcommands:
 
 - `export` (default) — generate INSERT/COPY SQL files. If the subcommand is omitted, `export` is assumed.
 - `explain` — print each query `export` would run together with its `EXPLAIN` output. SQL adapters compile the SELECT without executing it; mongodb runs the server's explain (defaulting to the execution-free `queryPlanner`).
+- `schema generate|check|tidy --from-db` — maintain the schema config by reading a live database, for applications that cannot be loaded to generate it from their models. See [Non-Rails applications](#non-rails-applications-exwiw-schema---from-db).
 
 ### `exwiw export`
 
@@ -326,7 +327,7 @@ Notes:
 - **Database connection settings stay on the CLI/environment.** `host`, `port`, `user`, `database`, `uri`, and `password` are **rejected** in the config file (exwiw exits with an error). `adapter` is the one connection-related key that *is* allowed in the file.
 - **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
 - Unknown keys are rejected so a typo surfaces immediately.
-- Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
+- Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain` or `schema`, so a single config file can be shared by every subcommand.
 - `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [MongoDB support](docs/mongodb.md#exwiw-explain-verbosity).
 - `mongodb_query_timeout_ms` sets the global, server-enforced query timeout (mongodb only); the `--mongodb-query-timeout-ms` CLI flag overrides it. Ignored by the SQL adapters. See [MongoDB support](docs/mongodb.md).
 
@@ -351,6 +352,12 @@ EXWIW_SCHEMA_DIR_PATH=custom_directory bundle exec rake exwiw:schema:generate
 
 As with the CLI, a relative `schema_dir` in the config file is resolved relative to the config file's own directory.
 
+An application with more than one schema source — ActiveRecord models and Mongoid documents, say —
+should give each source its own directory (`EXWIW_SCHEMA_DIR_PATH`), because every task judges the
+whole directory against the one source it reads. Left sharing a directory, `tidy` / `check` see the
+other source's configs as belonging to tables and collections that no longer exist, and report or
+remove them.
+
 #### Safe mode (masking new columns by default)
 
 A migration that adds a column would otherwise leave `schema:generate` emitting it unmasked, so
@@ -394,8 +401,15 @@ column are flagged but deliberately **not** masked:
   the primary key). A constant would collapse every row onto one value and break the restore with
   a duplicate key.
 
-Safe mode is ActiveRecord-only for now: `schema:generate_mongoid` does not flag new fields yet,
-though the `needs_mask_decision` key itself is understood on a MongoDB field.
+`schema:generate_mongoid` runs in safe mode too, on the same `EXWIW_NEW_COLUMNS=plain` opt-out. The
+masks come from the Mongoid field type (`String`, `Integer`, `Float`, `BigDecimal`,
+`Mongoid::Boolean`, `Date`, `Time` / `DateTime` / `ActiveSupport::TimeWithZone`); a field of any
+other type — `Hash`, `Array`, a typeless field, a BSON type — is flagged but not masked, as is a
+field covered by a unique index unless its mask varies per document. The structural fields are the
+`_id` primary key, the STI discriminator (`_type`) and every `belongs_to` foreign key of the
+collection: flagged, never masked. The foreign keys are read from the models, so the ones the
+config itself drops (a polymorphic `belongs_to`, a `belongs_to` on an embedded document) are
+covered too.
 
 #### Tidying stale config (`schema:tidy`)
 
@@ -448,8 +462,12 @@ databases stays distinct.
 Set `EXWIW_SCHEMA_CHECK_OUTPUT=<path>` to have the same JSON written to a file, which spares a
 caller from assuming stdout carries nothing else (application boot is free to print).
 
-Like safe mode, this is ActiveRecord-only — it regenerates through `SchemaGenerator`, so a
-Mongoid config directory is not supported yet.
+A Mongoid config directory has its own task, `schema:check_mongoid`, with the same output,
+the same `EXWIW_SCHEMA_CHECK_OUTPUT` file and the same exit code — it just regenerates through
+`MongoidSchemaGenerator` (safe mode + `tidy_mongoid`) instead. Collections and fields are
+reported under the same keys as tables and columns. An application that cannot be loaded to
+generate from its models at all can run the same check against its database instead: see
+[Non-Rails applications](#non-rails-applications-exwiw-schema---from-db).
 
 #### Multiple databases
 
@@ -479,10 +497,82 @@ For MongoDB applications backed by [Mongoid](https://www.mongodb.com/docs/mongoi
 
 ```bash
 bundle exec rake exwiw:schema:generate_mongoid
+bundle exec rake exwiw:schema:tidy_mongoid   # delete the config of a collection no model stores into
+bundle exec rake exwiw:schema:check_mongoid  # report the difference without changing anything
 ```
 
 What it derives from each model (fields, `belongs_tos`, `embedded_in`, STI handling), how to annotate constructs exwiw cannot represent with `ignore` / `ignore_type`, and the `EXWIW_SKIP_UNSUPPORTED=1` bootstrap flag are all documented in [MongoDB support](docs/mongodb.md#generating-config-from-mongoid-models).
 
+`tidy_mongoid` reconciles against the *models* rather than a live connection (MongoDB has no schema to read, and a collection exists only once something is written to it): a config file whose collection no model stores into any more is deleted, and nothing else is touched — fields already track the models through `generate_mongoid`. `check_mongoid` runs both into a throwaway copy, so it never writes to the working tree.
+
+#### Non-Rails applications (`exwiw schema ... --from-db`)
+
+The rake tasks above read the application's models, which requires loading the application — so
+they are only available where that is possible. For an application written in any other language
+(or a Ruby one exwiw cannot boot), the same three operations are available on the CLI, reading
+the **database** instead of the models:
+
+```bash
+# generate / refresh the config from the live schema
+exwiw schema generate --from-db -a postgresql -h db.example.com -p 5432 -u app --database=app --schema-dir=exwiw/schema
+
+# report how the committed config differs from the database (exits 1 when it needs work)
+exwiw schema check --from-db -a postgresql -h db.example.com -p 5432 -u app --database=app --schema-dir=exwiw/schema
+
+# remove tables/columns/relations the database no longer has
+exwiw schema tidy --from-db -a postgresql -h db.example.com -p 5432 -u app --database=app --schema-dir=exwiw/schema
+```
+
+- `--from-db` is **required**: the schema source is always stated explicitly rather than inferred.
+- mysql and postgresql only. sqlite is not supported, and a MongoDB schema lives in the
+  application rather than the database (use `schema:generate_mongoid`).
+- The usual connection flags and `DATABASE_PASSWORD` apply, and `adapter` / `schema_dir` may come
+  from [the config file](#config-file-exwiwyml) instead (`--schema-dir` wins). Unlike `export`,
+  an empty or absent `DATABASE_PASSWORD` is accepted, since these commands are commonly pointed
+  at a CI database that runs with trust authentication.
+- `generate` creates the schema directory if it does not exist; `check` and `tidy` require it.
+- Everything else behaves as the rake tasks do: [safe mode](#safe-mode-masking-new-columns-by-default)
+  is on unless `EXWIW_NEW_COLUMNS=plain`, `check` prints the same JSON report, honours
+  `EXWIW_SCHEMA_CHECK_OUTPUT`, and exits 1 when the config needs attention. A check that could
+  not *run* (an unreachable database, a malformed config) exits with a different status, so CI
+  can tell the two apart.
+- One run covers one database — the connection addresses one — so the files are written flat into
+  the schema directory. There is no per-database subdirectory layout here; a second database is a
+  second run against a second connection.
+
+The database is read through its catalog only: tables, columns and their types/defaults, primary
+keys, unique indexes and foreign keys. Views are skipped, since they hold no rows of their own,
+and a table with no primary key is emitted with `ignore: true` and a comment saying what to add
+to export it (exwiw identifies and joins rows by primary key). Following that advice sticks: a
+`primary_key` written by hand — on a table the database reports none for, or one column of a
+composite key that identifies a row on its own — is kept by later runs, which then treat the table
+as an ordinary one rather than re-imposing the signpost `type` / `comment`.
+
+**`belongs_tos` are only ever added, never rewritten.** A foreign-key constraint is weaker
+evidence than an application model: plenty of schemas express a relation only in application code,
+and a `belongs_to` is the path extraction follows to reach a table, so silently dropping one
+narrows the dump. Regeneration therefore keeps every relation the config already declares —
+in its existing order, with its `comment` / `ignore` / `ignore_type` — and appends only the
+foreign-key-backed relations that are not there yet. A hand-written relation's foreign-key column
+is treated as structural too, so safe mode never masks it. Removing a relation is `tidy`'s job:
+it drops a `belongs_to` whose target table no longer exists in the database (an `ignore: true`
+entry is kept, since it records a decision), alongside the tables and columns that are gone.
+
+So a relation the database does not know about is declared once, by hand, and survives from then
+on:
+
+```json
+{
+    "name": "orders",
+    "primary_key": "id",
+    "belongs_tos": [{
+        "table_name": "buyers",
+        "foreign_key": "buyer_id",
+        "comment": "enforced in the application; no foreign key in the database"
+    }]
+}
+```
+
 ### Configuration
 
 This is an example of the one table schema:
@@ -628,10 +718,11 @@ nobody has decided on yet:
 ```
 
 Extraction ignores the key entirely — what the column exports is whatever `replace_with` /
-`ignore` say. It exists so the decision can be tracked and required: `schema:generate`'s
+`ignore` say. It exists so the decision can be tracked and required:
 [safe mode](#safe-mode-masking-new-columns-by-default) attaches it to every newly discovered
-column together with a default mask, and [`schema:check`](#checking-the-config-against-the-schema)
-reports the columns that still carry it, so CI can keep a pull request red until each one is
+column/field together with a default mask, and
+[`schema:check`](#checking-the-config-against-the-schema) — `schema:check_mongoid` for a Mongoid
+config — reports the ones that still carry it, so CI can keep a pull request red until each is
 resolved. Resolving it means removing the key — after keeping the mask (ideally recording why
 in `comment`), replacing it with a real masking rule, dropping `replace_with` to export the
 raw value, or setting `ignore: true`.

data/docs/mongodb.md

@@ -138,7 +138,7 @@ For MongoDB applications backed by [Mongoid](https://www.mongodb.com/docs/mongoi
 bundle exec rake exwiw:schema:generate_mongoid
 ```
 
-It is a distinct task and class (`Exwiw::MongoidSchemaGenerator`) from the ActiveRecord generator because the two ORMs expose entirely different metadata. From each model it derives:
+It is a distinct task and class (`Exwiw::MongoidSchemaGenerator`) from the ActiveRecord generator because the two ORMs expose entirely different metadata. Every application document model is described, except one declared `store_in collection: nil` — the way an application says a document class is never persisted and only wants Mongoid's casting: it has no collection name to describe, so it is skipped (and does not count as a live collection when tidying). From each remaining model it derives:
 
 - the collection name and the `_id` primary key,
 - `fields` from the declared Mongoid fields (referenced `belongs_to` foreign keys such as `shop_id`, and the `created_at` / `updated_at` columns added by `Mongoid::Timestamps`, are ordinary fields — their BSON `ObjectId` / `Date` values serialize as MongoDB Extended JSON at dump time). For an aliased field (`field :ctry, as: :country`), the generator emits the **stored** document key (`ctry`), never the Ruby accessor (`country`), so masking and projection target the key that actually appears in the document, and additionally records the accessor as `mongoid_field_name` on that field so the short key stays understandable (association aliases such as `shop => shop_id` and the built-in `id => _id` are not field renames and are not annotated),
@@ -147,8 +147,28 @@ It is a distinct task and class (`Exwiw::MongoidSchemaGenerator`) from the Activ
 
 Models in an inheritance hierarchy whose subclasses share the base's collection (Mongoid STI, distinguished by the auto-added `_type` discriminator) collapse into a single config: the generator discovers the subclasses via `descendants` (Mongoid registers only the base class in `Mongoid.models`) and unions every class's `fields` and `belongs_tos` into the collection config, so subclass-only fields and associations are not lost.
 
+A collection name can also be shared by embedded and non-embedded models, in two shapes the generator tells apart. `embedded?` only answers "does this class declare an `embedded_in`", which is not the same question as "does this collection have root documents":
+
+- **An embedded family under a plain base class** — `Address` holding the shared fields and declaring no `embedded_in`, with `BillingAddress < Address` declaring one and inheriting the collection name. Nothing is stored at the root under the base; it is part of the embedded family, and its fields describe subdocuments. The collection stays **embedded**, exactly as before: one `embedded_in` config unioning the base's fields with its subclasses', derived from a class that actually declares the embedding. (A non-embedded model whose embedded descendants are in the same group is always read as their base — conservatively so, even if the application also stored root documents under it, since the alternative would silently delete the config masking those subdocuments.)
+- **A genuine collision** — an unrelated top-level model stores into a collection whose name an embedded class derives from its own class name. Such a collection is genuinely **top-level**: it has root documents that must be dumped, so its config is built from the root models only (their fields and `belongs_tos`) with no `embedded_in`. An embedded base in the same group does not contribute its fields. Representing the group as embedded would make `dumpable?` skip the collection and silently drop those documents from the export.
+
+The collision is reported on stderr, because the embedded documents of the same name are **not** covered by that config: to mask them, add a config by hand with an `embedded_in` and a `name` / file name of your own choosing (generation and `tidy_mongoid` both leave `embedded_in` configs alone, so a hand-written one is stable). This is deliberately not recorded as a `comment` on the generated config — a generated comment takes precedence when merging, so it would overwrite a note you wrote about this very situation on every run.
+
 Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, `bulk_insert_chunk_size`, and `query_timeout_ms` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [Output](#output)). Polymorphic `belongs_to` is not yet expanded by this task.
 
+### Safe mode, `tidy_mongoid` and `check_mongoid`
+
+Like the ActiveRecord task, `generate_mongoid` runs in [safe mode](../README.md#safe-mode-masking-new-columns-by-default) by default (`EXWIW_NEW_COLUMNS=plain` opts out for a first bootstrap): a field the config does not have yet is emitted masked — as far as its Mongoid type allows — and flagged `needs_mask_decision: true`, so a field added to a model does not start being exported before somebody has judged whether it holds personal data. The `_id` primary key, the `_type` STI discriminator and every `belongs_to` foreign key are flagged but never masked (masking them would rewrite the very references the dump is assembled from), and neither is a field whose type no constant fits (`Hash`, `Array`, a typeless field, a BSON type) or one covered by a unique index unless its mask varies per document. A field's own `default:` is preferred over the per-type constant. A decision recorded on disk — a mask kept, replaced, dropped, or the flag removed — wins over every later regeneration.
+
+```bash
+bundle exec rake exwiw:schema:tidy_mongoid   # delete the config of a collection no model stores into
+bundle exec rake exwiw:schema:check_mongoid  # report the difference, writing nothing
+```
+
+`tidy_mongoid` is the counterpart `generate_mongoid` cannot be: it deletes the config file of a collection no model stores into any more. It reconciles against the **models**, not a live connection — MongoDB has no schema to read, and a collection exists only once a document is written to it — using exactly the grouping `generate_mongoid` writes files from, embedded collections included. Fields are left alone, since `generate_mongoid` already tracks them through the model.
+
+`check_mongoid` mirrors [`schema:check`](../README.md#checking-the-config-against-the-schema): it regenerates (safe mode + tidy) into a throwaway copy of the config directory, prints the same sorted JSON report — collections and fields under the same keys as tables and columns — honors `EXWIW_SCHEMA_CHECK_OUTPUT`, and exits non-zero while anything is out of date or undecided, so it can gate a pull request. It regenerates in strict mode: a newly introduced construct exwiw cannot represent fails the task rather than reporting clean.
+
 By default the task **aborts** when a model uses a construct exwiw cannot represent: a `belongs_to` whose target class can no longer be resolved (a stale relation left behind after its model was removed), or a polymorphic / self-referential-cyclic / ambiguous / unresolvable-parent `embedded_in` (see the cases above).
 
 ### Honoring an explicit `ignore` (the recommended way to keep these out)

data/lib/exwiw/cli.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require 'fileutils'
 require 'logger'
 require 'optparse'
 require 'pathname'
@@ -11,7 +12,18 @@ require 'exwiw'
 
 module Exwiw
   class CLI
-    KNOWN_SUBCOMMANDS = %w[export explain].freeze
+    KNOWN_SUBCOMMANDS = %w[export explain schema].freeze
+
+    # The verbs `exwiw schema` takes, mirroring the `exwiw:schema:*` rake tasks
+    # a Rails application would run instead.
+    SCHEMA_VERBS = %w[generate check tidy].freeze
+
+    # `schema check` exit codes. 1 means "the config needs work" — that is the
+    # signal CI acts on — so a failure to *perform* the check (an unreachable
+    # database, a malformed config) must not use it, or an infrastructure
+    # problem reads as a schema problem and gets "fixed" by regenerating.
+    SCHEMA_CHECK_DIRTY_EXIT = 1
+    SCHEMA_CHECK_ERROR_EXIT = 2
 
     # Config file loaded automatically when --config is omitted, if one exists in
     # the current directory. Kept at the project root (rather than under exwiw/)
@@ -52,7 +64,8 @@ module Exwiw
     REJECTED_CONNECTION_KEYS = %w[host port user database uri password].freeze
 
     # Keys that only make sense for `export`. They are skipped when merging config
-    # for `explain` so a shared config file does not trip validate_explain_only!.
+    # for `explain` so a shared config file does not trip validate_explain_only!,
+    # and for `schema`, which performs no export at all.
     EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook parallel_workers].freeze
 
     def self.start(argv)
@@ -69,6 +82,16 @@ module Exwiw
           "export"
         end
 
+      # `schema` is the one subcommand taking a verb of its own
+      # (`exwiw schema generate`), so the next positional argument belongs to
+      # it. An unknown or missing verb is not consumed here — validation
+      # reports it with the list of valid ones rather than OptionParser
+      # failing on a stray argument.
+      @schema_verb =
+        if @subcommand == "schema" && !@argv.empty? && !@argv.first.start_with?("-")
+          @argv.shift
+        end
+
       @help = @argv.empty?
 
       @database_host = nil
@@ -78,6 +101,7 @@ module Exwiw
       @connection_uri = nil
       @output_dir = nil
       @schema_dir = nil
+      @from_db = false
       @config_file_path = nil
       @database_adapter = nil
       @database_name = nil
@@ -151,9 +175,94 @@ module Exwiw
           io: $stdout,
           explain_verbosity: @explain_verbosity,
         ).run
+      when "schema"
+        run_schema(connection_config)
       end
     end
 
+    # `exwiw schema generate|check|tidy --from-db`: the same three operations
+    # the `exwiw:schema:*` rake tasks perform for a Rails application, driven
+    # from a database connection instead of from the application's models.
+    private def run_schema(connection_config)
+      introspector = DbIntrospector.build(connection_config)
+
+      case @schema_verb
+      when "generate" then run_schema_generate(introspector)
+      when "tidy" then run_schema_tidy(introspector)
+      when "check" then run_schema_check(introspector)
+      end
+    end
+
+    private def run_schema_generate(introspector)
+      DbSchemaGenerator.new(
+        introspector: introspector,
+        output_dir: @schema_dir,
+        safe_new_columns: schema_safe_new_columns?,
+      ).generate!
+
+      puts "exwiw: wrote the schema config for #{@database_name} to #{@schema_dir}."
+    end
+
+    private def run_schema_tidy(introspector)
+      result = DbSchemaGenerator.new(introspector: introspector, output_dir: @schema_dir).tidy!
+
+      if result.empty?
+        puts "exwiw: schema config is already tidy; nothing to remove."
+        return
+      end
+
+      result.removed_tables.each do |name|
+        puts "exwiw: removed config for table '#{name}' (no longer exists in the database)."
+      end
+      result.removed_columns.each do |table_name, columns|
+        puts "exwiw: removed column(s) #{columns.join(', ')} from '#{table_name}' (no longer in the table)."
+      end
+      result.removed_belongs_tos.each do |table_name, targets|
+        puts "exwiw: removed belongs_to(s) to #{targets.join(', ')} from '#{table_name}' " \
+             "(the target table no longer exists)."
+      end
+    end
+
+    private def run_schema_check(introspector)
+      # Regenerate exactly as `generate` + `tidy` would, into the copy
+      # SchemaCheck hands over, so the report is the diff against a config that
+      # has been through both steps rather than only the first.
+      regenerator = lambda do |tmp_dir|
+        # Explicit: safe mode is not optional here, whatever the library default is.
+        DbSchemaGenerator.new(introspector: introspector, output_dir: tmp_dir, safe_new_columns: true).generate!
+        DbSchemaGenerator.new(introspector: introspector, output_dir: tmp_dir).tidy!
+      end
+
+      report =
+        begin
+          SchemaCheck.new(schema_dir: @schema_dir, regenerator: regenerator).run
+        rescue StandardError => e
+          # See SCHEMA_CHECK_ERROR_EXIT: not being able to run the check is a
+          # different answer from "the config is out of date".
+          $stderr.puts "exwiw: could not run the schema check (#{e.class}: #{e.message})"
+          exit SCHEMA_CHECK_ERROR_EXIT
+        end
+
+      json = JSON.pretty_generate(report)
+      puts json
+      # A file too, so a caller need not assume stdout carries only the JSON.
+      File.write(ENV["EXWIW_SCHEMA_CHECK_OUTPUT"], json + "\n") if ENV["EXWIW_SCHEMA_CHECK_OUTPUT"]
+
+      return if SchemaCheck.clean?(report)
+
+      $stderr.puts "exwiw: the schema config is out of date or has undecided masking; " \
+                   "run `exwiw schema generate --from-db` (then `exwiw schema tidy --from-db`) " \
+                   "and resolve every `needs_mask_decision` column."
+      exit SCHEMA_CHECK_DIRTY_EXIT
+    end
+
+    # Safe mode is on unless EXWIW_NEW_COLUMNS=plain, matching the rake task: a
+    # first-time bootstrap wants it off, since there every column is new and
+    # flagging the whole config at once is noise.
+    private def schema_safe_new_columns?
+      ENV["EXWIW_NEW_COLUMNS"] != "plain"
+    end
+
     private def validate_options!
       # Fill in any options not given on the CLI from the config file. Done first
       # so a config-provided `adapter` is in place before normalization below.
@@ -188,6 +297,16 @@ module Exwiw
         resolve_explain_verbosity!
       end
 
+      # `schema` shares only the connection options with the other subcommands:
+      # it neither extracts rows nor reads a dump target, so every check below
+      # is about an export it will not perform. Its own requirements — a schema
+      # source, an introspectable adapter, a schema dir the verb can use — are
+      # validated instead.
+      if @subcommand == "schema"
+        validate_schema_options!
+        return
+      end
+
       if @database_adapter != "sqlite"
         # When a connection URI is supplied (mongodb only), host/port/database
         # are read from the URI, so none of them are required on the CLI.
@@ -315,7 +434,9 @@ module Exwiw
 
       # For `explain`, drop export-only keys so a config shared with `export`
       # does not make validate_explain_only! reject the run.
-      config = config.reject { |k, _| EXPORT_ONLY_CONFIG_KEYS.include?(k) } if @subcommand == "explain"
+      if @subcommand == "explain" || @subcommand == "schema"
+        config = config.reject { |k, _| EXPORT_ONLY_CONFIG_KEYS.include?(k) }
+      end
 
       @database_adapter ||= config["adapter"]
       @schema_dir ||= expand_dir(config["schema_dir"], base)
@@ -494,6 +615,66 @@ module Exwiw
       exit 1
     end
 
+    # Everything `exwiw schema <verb>` needs, and nothing the export path
+    # requires. Runs instead of the export/explain validations, not after them.
+    private def validate_schema_options!
+      unless SCHEMA_VERBS.include?(@schema_verb)
+        $stderr.puts "Usage: exwiw schema #{SCHEMA_VERBS.join('|')} --from-db [options] " \
+                     "(got #{@schema_verb.inspect})"
+        exit 1
+      end
+
+      # The schema source is spelled out rather than assumed. Reading a live
+      # database is one way to describe an application's schema and not the
+      # only conceivable one, so the flag keeps the command unambiguous today
+      # and leaves the reader expecting alternatives tomorrow.
+      unless @from_db
+        $stderr.puts "`exwiw schema #{@schema_verb}` requires --from-db to say where the schema is read from " \
+                     "(a live database connection). A Rails application can use the exwiw:schema:#{@schema_verb} " \
+                     "rake task instead, which reads its models."
+        exit 1
+      end
+
+      unless DbIntrospector::SUPPORTED_ADAPTERS.include?(@database_adapter)
+        $stderr.puts "--from-db supports the #{DbIntrospector::SUPPORTED_ADAPTERS.join(' and ')} adapters only " \
+                     "(got '#{@database_adapter}'). sqlite and mongodb schemas are not read this way."
+        exit 1
+      end
+
+      {
+        "Target database host" => @database_host,
+        "Target database port" => @database_port,
+        "Target database name" => @database_name,
+        "Database user" => @database_user,
+      }.each do |name, value|
+        if value.nil?
+          $stderr.puts "#{name} is required"
+          exit 1
+        end
+      end
+
+      # Deliberately no DATABASE_PASSWORD requirement, unlike export: schema
+      # generation is commonly run in CI against a throwaway database started
+      # with trust/empty authentication, and refusing an empty password would
+      # make the check unrunnable exactly where it is most wanted. The password
+      # is still used when set.
+
+      if @schema_dir.nil?
+        $stderr.puts "Schema dir is required (pass --schema-dir or set schema_dir in the config file)"
+        exit 1
+      end
+
+      # `generate` is also the bootstrap command, so it creates the directory;
+      # `check` and `tidy` only ever read an existing config, and a missing
+      # directory there means the wrong path far more often than an empty one.
+      if @schema_verb == "generate"
+        FileUtils.mkdir_p(@schema_dir)
+      elsif !Dir.exist?(@schema_dir)
+        $stderr.puts "Schema dir does not exist: #{@schema_dir}"
+        exit 1
+      end
+    end
+
     private def validate_explain_only!
       rejected = []
       rejected << "--output-dir" unless @output_dir.nil?
@@ -603,6 +784,11 @@ module Exwiw
                       For mongodb, set verbosity via EXWIW_MONGODB_EXPLAIN_VERBOSITY
                       or `explain_verbosity:` in config (queryPlanner (default,
                       no query is executed) | executionStats | allPlansExecution).
+            schema    Maintain the schema config (generate | check | tidy) by
+                      reading a live database, for applications that cannot be
+                      loaded to generate it from their models. Requires --from-db;
+                      mysql and postgresql only. `check` prints a JSON report and
+                      exits 1 when the config needs work.
         BANNER
         opts.version = Exwiw::VERSION
 
@@ -617,6 +803,7 @@ module Exwiw
           v = v.end_with?("/") ? v[0..-2] : v
           @schema_dir = File.expand_path(v)
         end
+        opts.on("--from-db", "Read the schema from the database the connection options point at (schema subcommand only; mysql/postgresql). Required by `exwiw schema`.") { @from_db = true }
         opts.on("-c", "--config=CONFIG_FILE_PATH", "Path to the exwiw config YAML. Defaults to ./#{DEFAULT_CONFIG_PATHS.first} (or .#{File.extname(DEFAULT_CONFIG_PATHS.last)}) when present. CLI options take precedence; paths inside the file are resolved relative to the file.") do |v|
           @config_file_path = File.expand_path(v)
         end