exwiw 0.9.17 → 0.9.19
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +13 -1
- data/README.md +84 -168
- data/docs/mongodb.md +192 -0
- data/lib/exwiw/adapter/mysql_client.rb +3 -0
- data/lib/exwiw/batch_scope.rb +20 -0
- data/lib/exwiw/batched_extraction.rb +116 -0
- data/lib/exwiw/explain_runner.rb +25 -0
- data/lib/exwiw/query_ast_builder.rb +122 -17
- data/lib/exwiw/runner.rb +15 -1
- data/lib/exwiw/table_config.rb +16 -0
- data/lib/exwiw/version.rb +1 -1
- data/lib/exwiw.rb +2 -0
- metadata +4 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 9f565edc88b4eba108ef4d9accf7bbaa7d48f0e1700007cc9d280b0033e357b6
|
|
4
|
+
data.tar.gz: 822e5cd0b9d33c81d0b396cc641fa960cf80bffbebd6bd078322460a140b37cf
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 20da3c9ee09865762f40a04b6c4bf6cf24582545ba7c51945a69aed69798906a19a214c9bd2ad588e0b6740f29a3ebfa08a122845f9a28fe64b654ba723088a0
|
|
7
|
+
data.tar.gz: db91db3a7e1774d386f45d3582301098a23df86f48fe790fae3631895b87fe686dcce3d048a17827f69ce530f948ca423bb9028f8a4b2495d167e820edd6f3fb
|
data/CHANGELOG.md
CHANGED
|
@@ -2,6 +2,18 @@
|
|
|
2
2
|
|
|
3
3
|
## [Unreleased]
|
|
4
4
|
|
|
5
|
+
## [0.9.19] - 2026-08-04
|
|
6
|
+
|
|
7
|
+
### Added
|
|
8
|
+
|
|
9
|
+
- **`batch_scope`: extract a table as one query per slice of the scope's id set, so a very large table stays index-driven instead of degrading into a full scan.** A scoped table reached through a `belongs_to` hop compiles to `SELECT t.* FROM t JOIN parent ON t.parent_id = parent.id AND parent.tenant_id IN (...)`, which is index-driven while the scope keeps few parent rows. Past some number of them the planner's estimate of "probe the foreign-key index once per parent row" exceeds its estimate of "scan the table once" and it switches to a sequential scan of the whole table — for a result set that is a small fraction of it. On a table of hundreds of millions of rows that scan exceeds the server's `statement_timeout` (or runs for hours), and a `filter` on the extracted table does not help: a predicate that reduces the *output* does not reduce the *work* once the plan is a scan. `batch_scope: { "table": "<scoped table>", "size": 1000 }` names the scoped table this one reaches, resolves that table's in-scope primary keys once (from its own extraction query, so it is narrowed by exactly the filter it would carry anyway), and extracts one `size`-sized slice of those ids at a time, each batch carrying `<batch table>.<pk> IN (<ids>)` in place of the scope filter. A literal id list of that size is exactly estimated and selective, so the foreign-key index is unambiguously the cheapest plan for every batch, and total work becomes proportional to the rows the table keeps rather than to the table's size. **The dumped rows are the same as the unbatched query's**, in batch-by-batch order: the slices partition the id set, so no row is dropped or emitted twice, and the ids are sorted in exwiw before slicing (not via `ORDER BY`, which would push a sort onto the source DB), so the batched output is reproducible run to run. The batch table may be any number of hops up the path (a table two hops below it names the same batch table, and the ids are applied where the path meets the scope, bounding the whole join chain), or the table itself when it carries the scope column. Because a batch key only splits an extraction correctly when *every* row the table keeps is selected through the batch table's scope filter, the supported shapes are deliberately narrow — scope-column mode, and either a directly scoped table naming itself or a single `belongs_to` join path terminating at the named table. Polymorphic arm `UNION`s, `reverse_scope`, referenced-by, the parent cascade, `scope_exempt` (on either side — an exempt batch table's id set would not be scoped, so its batches would reach outside the scope) and single `--target-table` mode are rejected with an explanation rather than silently mis-sliced (they keep rows by routes a batch of ids does not constrain, so every batch would re-emit them), and the rejection happens in the pre-flight validation, before any output is written. `exwiw explain` additionally prints the id-set query and its `EXPLAIN` for a batched table; it cannot show a batch's literal ids, since it executes no extraction SELECT. `delete-*.sql` is generated from the unbatched query as before, `bulk_insert_chunk_size` is independent (batches are query boundaries, chunks are statement boundaries), and output for every table without `batch_scope` is byte-identical.
|
|
10
|
+
|
|
11
|
+
## [0.9.18] - 2026-08-03
|
|
12
|
+
|
|
13
|
+
### Fixed
|
|
14
|
+
|
|
15
|
+
- **mysql: scope id-set materialization now works under the mysql2 driver instead of silently disabling itself.** `MysqlClient#query` read `fields` off the driver's return value, but mysql2 returns `nil` for a statement with no result set — exactly what materialization issues (`SET SESSION sql_mode`, `CREATE TEMPORARY TABLE`, `ALTER TABLE ... ADD INDEX`). The resulting `NoMethodError` was caught by the adapter's own safety net, which logged `Disabling scope id-set materialization` and fell back to inline scope subqueries for the whole run, so every descendant table re-evaluated the same id-set subquery (for a multi-arm `reverse_scope` UNION over a large identity table, the dominant cost of the export) — the 0.9.13 optimization never took effect on mysql2 at all. The trilogy driver always returns a result object, which is why it was unaffected. `#query` now returns an empty `Result` for a statement with no result set, and a spec exercises `SET` / `CREATE TEMPORARY TABLE` / `ALTER TABLE` against a live server on both drivers. Because a materialization failure is invisible in the output — the adapter warns and falls back to inline subqueries, producing identical rows — a new mysql e2e scenario (`e2e/test_with_mysql_scope.sh`, scope-column mode over the two-tenant fixture) additionally asserts that the run materializes an id-set and never logs the fallback. Dumped rows are unchanged; only the number of times the scope subquery is executed is.
|
|
16
|
+
|
|
5
17
|
## [0.9.17] - 2026-08-03
|
|
6
18
|
|
|
7
19
|
### Fixed
|
|
@@ -39,7 +51,7 @@
|
|
|
39
51
|
|
|
40
52
|
### Added
|
|
41
53
|
|
|
42
|
-
- **The MongoDB adapter now supports `replace_with_fake_data`.** The masking mode previously limited to the SQL adapters is now accepted on a `MongodbField`: its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the **same** fake value as the SQL adapters for the same seed value — the value-derivation logic (`build_value_deriver` et al.) was extracted from `RowTransformer` into shared class methods so both families produce byte-identical output for a given seed, type, and locale. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), recurses into embedded subdocuments, and preserves `null`/absent values. `MongodbCollectionConfig` validates the key on load (exclusive with `replace_with` on the same field, unknown types rejected, seed field must resolve) and preserves it across schema regeneration. Add `gem "faker"` to use it (except a config using only `ja` person types, which build from exwiw's bundled dataset). `raw_sql` and `map` remain unsupported on MongoDB. See [MongoDB
|
|
54
|
+
- **The MongoDB adapter now supports `replace_with_fake_data`.** The masking mode previously limited to the SQL adapters is now accepted on a `MongodbField`: its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the **same** fake value as the SQL adapters for the same seed value — the value-derivation logic (`build_value_deriver` et al.) was extracted from `RowTransformer` into shared class methods so both families produce byte-identical output for a given seed, type, and locale. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), recurses into embedded subdocuments, and preserves `null`/absent values. `MongodbCollectionConfig` validates the key on load (exclusive with `replace_with` on the same field, unknown types rejected, seed field must resolve) and preserves it across schema regeneration. Add `gem "faker"` to use it (except a config using only `ja` person types, which build from exwiw's bundled dataset). `raw_sql` and `map` remain unsupported on MongoDB. See [MongoDB support](docs/mongodb.md#masking).
|
|
43
55
|
- **Reserved-word and special-character identifiers are now quoted (SQL adapters).** Table and column names that are reserved words (`order`, `from`, `group`, …) or contain characters invalid as a bare identifier are conditionally quoted — backticks on mysql, double quotes on postgresql/sqlite — across every emission path: SELECT projection and masking expressions, FROM/JOIN, WHERE, subqueries, materialized scope JOINs, INSERT / COPY headers and DELETE. Previously only the mysql INSERT header was quoted ([#83](https://github.com/heyinc/exwiw/pull/83)), so a reserved table name broke the extraction SELECT everywhere, a reserved column broke postgresql/sqlite INSERTs, and sqlite rejected even table-qualified reserved columns. Ordinary names stay bare, so output for existing configs is byte-identical (mysql INSERT headers keep their always-backtick form from 0.4.5). Dotted table names are treated as schema qualification and quoted per part (`billing.order` → `` billing.`order` ``), preserving Rails multi-schema `table_name`s. The reserved-word lists cover MySQL 8.4 and 9.x (including 9's `LIBRARY`) plus MariaDB-specific words, PostgreSQL's fully reserved key words, and — for sqlite — exactly the keywords that fail to parse bare in exwiw's emission positions, so fallback-accepted names like `key` are not churned. The lists are enforced by specs that probe the live mysql/postgresql servers' own keyword catalogs and re-derive the sqlite set empirically.
|
|
44
56
|
|
|
45
57
|
### Fixed
|
data/README.md
CHANGED
|
@@ -40,7 +40,7 @@ gem install exwiw
|
|
|
40
40
|
- mysql
|
|
41
41
|
- postgresql
|
|
42
42
|
- sqlite
|
|
43
|
-
- mongodb (
|
|
43
|
+
- mongodb (see [MongoDB support](docs/mongodb.md))
|
|
44
44
|
|
|
45
45
|
For MySQL, exwiw connects through whichever of the `mysql2` or `trilogy` gem is
|
|
46
46
|
available (preferring `mysql2`), so an app on either driver works without any
|
|
@@ -129,28 +129,31 @@ exwiw explain \
|
|
|
129
129
|
|
|
130
130
|
The `--output-dir`, `--output-format`, `--insert-only`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
|
|
131
131
|
|
|
132
|
-
|
|
132
|
+
MongoDB-specific explain behavior — the configurable verbosity (`queryPlanner` / `executionStats` / `allPlansExecution`) and how scoped collections are shown — is described in [MongoDB support](docs/mongodb.md#exwiw-explain-verbosity).
|
|
133
133
|
|
|
134
|
-
|
|
134
|
+
### How each table is narrowed — the six scoping paths
|
|
135
135
|
|
|
136
|
-
|
|
137
|
-
|---|---|
|
|
138
|
-
| `queryPlanner` (default) | Plans the query only. **The query is not executed** — no documents are scanned. |
|
|
139
|
-
| `executionStats` | **Runs the query** and reports runtime statistics (docs examined, time, etc.). |
|
|
140
|
-
| `allPlansExecution` | Runs the winning plan **and the rejected candidate plans** to gather their stats. |
|
|
136
|
+
Only the dump target itself is filtered by `--ids` directly. Every *other* table must be **scoped** — narrowed to just the rows related to the target — some other way, and a table that cannot be scoped at all is dumped in full (or, in scope-column mode, aborts the run). exwiw resolves each table through the **first** of these six paths that applies:
|
|
141
137
|
|
|
142
|
-
|
|
143
|
-
|
|
144
|
-
|
|
145
|
-
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
138
|
+
| # | Path | When it applies | Resulting query shape |
|
|
139
|
+
|---|------|-----------------|-----------------------|
|
|
140
|
+
| 1 | **Direct filter** | The table is the `--target-table` itself; or, in [scope-column mode](#scope-column-mode), it declares a `scope_column` | `WHERE pk IN (ids)` / `WHERE scope_column IN (ids)` |
|
|
141
|
+
| 2 | **`belongs_to` join walk** | The table reaches the target (or a scope-column table) by following its `belongs_to` edges | `WHERE fk IN (ids)` for a single hop; a chain of `JOIN`s for longer paths |
|
|
142
|
+
| 3 | **Referenced-by (automatic reverse)** | No `belongs_to` path of its own, but **exactly one** already-constrained table points at it by foreign key | Constrained to the ids that referencer's own query selects |
|
|
143
|
+
| 4 | **`reverse_scope` (declared reverse)** | Referenced by **many** scoped tables — typically a global-identity table like `users` — and the referencers are enumerated in its config | Constrained to the `UNION` of the enumerated referencers' ids |
|
|
144
|
+
| 5 | **Scoped-parent cascade** | No path or referencer, but a `belongs_to` parent is itself scoped (by any path above) | Constrained to the parent's in-scope primary keys; cascades over multiple hops |
|
|
145
|
+
| 6 | **Full dump** | Nothing relates the table to the target | All rows. In scope-column mode this **aborts** unless the table opts in with `scope_exempt: true` |
|
|
150
146
|
|
|
151
|
-
|
|
147
|
+
How the paths behave and interact:
|
|
152
148
|
|
|
153
|
-
|
|
149
|
+
1. **Direct filter.** In the default single-target mode the target is anchored on its primary key (or a custom field via the mongodb-only `--ids-field`). In [scope-column mode](#scope-column-mode) there is no single anchor: every table that declares a `scope_column` is filtered on that column directly.
|
|
150
|
+
2. **`belongs_to` join walk** — the "normal join" path. exwiw BFS-walks `belongs_to` edges to the nearest terminus (the target table, or a directly scoped table in scope-column mode) and compiles the shortest path into `INNER JOIN`s. A [polymorphic `belongs_to`](#polymorphic-belongs_to) hop additionally pins the type column; in scope-column mode a polymorphic hop is resolved for **every** concrete arm and the arms are `UNION`ed (see [Every arm is extracted](#every-arm-is-extracted-scope-column-mode)).
|
|
151
|
+
3. **Referenced-by** handles a table with no outgoing path that is pointed *at* by a constrained child — `active_storage_blobs`, referenced by `active_storage_attachments.blob_id`, is the canonical case (see [ActiveStorage](#activestorage-has_one_attached--has_many_attached)). It is automatic but deliberately narrow: it requires a single, non-polymorphic referencer. With two or more referencers it steps aside (path 6) unless you declare `reverse_scope`.
|
|
152
|
+
4. **[`reverse_scope`](#reverse-scope-for-multi-referencer-tables-reverse_scope)** is the declared, multi-referencer form of path 3: the config enumerates which referencers' (already scoped) queries feed the id set. Unscoped arms are skipped with a warning rather than widening the dump.
|
|
153
|
+
5. **Scoped-parent cascade** rescues satellites: a table whose only link is a `belongs_to` toward a hub that is itself scoped (e.g. via referenced-by or `reverse_scope`) is constrained to that parent's in-scope ids. The cascade recurses hop by hop (each level requires a single unambiguous scopable parent) and stops on `belongs_to` cycles.
|
|
154
|
+
6. **Full dump** is the fallback for a genuinely unrelated table — intended for reference/master data. Single-target mode dumps it in full (with a warning when an ambiguous cascade was the reason); scope-column mode refuses to run instead, unless the table is explicitly marked [`scope_exempt: true`](#scope_exempt-intentional-full-dump) (Rails-managed tables are exempt automatically).
|
|
155
|
+
|
|
156
|
+
Paths 3–5 all materialize their id set once and probe it via a `JOIN` on a `SELECT DISTINCT` derived table rather than `IN (subquery)` — see [Why a JOIN, not `IN (subquery)`](#why-a-join-not-in-subquery). Scope-column mode classifies every table up front with these same paths (`:direct` / `:via_path` / `:referenced_by` / `:via_scoped_parent` / `:exempt` / `:unscopable` in `QueryAstBuilder#scope_category`) and aborts before extracting anything if any table lands on `:unscopable`. The MongoDB adapter follows the same model, except id sets are captured at runtime while parent collections stream instead of being expressed as SQL subqueries — see [MongoDB support](docs/mongodb.md).
|
|
154
157
|
|
|
155
158
|
### Scope-column mode
|
|
156
159
|
|
|
@@ -317,8 +320,8 @@ Notes:
|
|
|
317
320
|
- **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
|
|
318
321
|
- Unknown keys are rejected so a typo surfaces immediately.
|
|
319
322
|
- Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain`, so a single config file can be shared by both subcommands.
|
|
320
|
-
- `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [
|
|
321
|
-
- `mongodb_query_timeout_ms` sets the global, server-enforced query timeout (mongodb only); the `--mongodb-query-timeout-ms` CLI flag overrides it. Ignored by the SQL adapters. See [MongoDB
|
|
323
|
+
- `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [MongoDB support](docs/mongodb.md#exwiw-explain-verbosity).
|
|
324
|
+
- `mongodb_query_timeout_ms` sets the global, server-enforced query timeout (mongodb only); the `--mongodb-query-timeout-ms` CLI flag overrides it. Ignored by the SQL adapters. See [MongoDB support](docs/mongodb.md).
|
|
322
325
|
|
|
323
326
|
### Generator
|
|
324
327
|
|
|
@@ -382,64 +385,13 @@ A `belongs_to` whose target model lives in a *different* database (e.g. a `prima
|
|
|
382
385
|
|
|
383
386
|
#### Mongoid applications
|
|
384
387
|
|
|
385
|
-
For MongoDB applications backed by [Mongoid](https://www.mongodb.com/docs/mongoid/), a separate rake task introspects Mongoid document models and emits `MongodbCollectionConfig` files
|
|
388
|
+
For MongoDB applications backed by [Mongoid](https://www.mongodb.com/docs/mongoid/), a separate rake task introspects Mongoid document models and emits `MongodbCollectionConfig` files:
|
|
386
389
|
|
|
387
390
|
```bash
|
|
388
391
|
bundle exec rake exwiw:schema:generate_mongoid
|
|
389
392
|
```
|
|
390
393
|
|
|
391
|
-
|
|
392
|
-
|
|
393
|
-
- the collection name and the `_id` primary key,
|
|
394
|
-
- `fields` from the declared Mongoid fields (referenced `belongs_to` foreign keys such as `shop_id`, and the `created_at` / `updated_at` columns added by `Mongoid::Timestamps`, are ordinary fields — their BSON `ObjectId` / `Date` values serialize as MongoDB Extended JSON at dump time). For an aliased field (`field :ctry, as: :country`), the generator emits the **stored** document key (`ctry`), never the Ruby accessor (`country`), so masking and projection target the key that actually appears in the document, and additionally records the accessor as `mongoid_field_name` on that field so the short key stays understandable (association aliases such as `shop => shop_id` and the built-in `id => _id` are not field renames and are not annotated),
|
|
395
|
-
- `belongs_tos` from referenced `belongs_to` associations (`{ table_name, foreign_key }`). A referenced `belongs_to` declared on an *embedded* document is dropped (cross-collection refs from inside embedded subdocuments are unsupported — see [MongoDB notes](#mongodb-notes)), but its foreign-key column is still kept as an ordinary field. A `has_and_belongs_to_many` association is also dropped (its foreign keys are stored as an array field, e.g. `tag_ids`, which exwiw cannot follow as a single-valued foreign key), while that `*_ids` array column is kept as an ordinary field,
|
|
396
|
-
- `embedded_in` from `embedded_in` / `embeds_many` / `embeds_one` associations. Each embedded config names its *immediate* parent collection and the document key it lives under (`store_as`, defaulting to the relation name); nested embedding is represented as a chain (`comments` → `embedded_in` `posts`, `posts` → `embedded_in` `users`) rather than a flattened dot-path, matching how the adapter recurses through array and Hash subdocuments. The document key is resolved by locating the parent's `embeds_one` / `embeds_many` that stores this collection. (Mongoid's computed inverse is frequently `nil` when no explicit `inverse_of:` is set, so exwiw matches by the collection the parent's embedding relations store rather than trusting that inverse — this also resolves an STI subclass embedded through a relation declared against its base class.) When the same collection is embedded under several keys in the parent, the path is ambiguous and treated as unrepresentable (see below). A *polymorphic* `embedded_in` (`embedded_in :addressable, polymorphic: true`) has no single embedding parent collection and so cannot be expressed as an `embedded_in` config. A *self-referential / cyclic* embedding (Mongoid's `recursively_embeds_many` / `recursively_embeds_one`) makes a collection both a top-level document and embedded inside documents of its own type; exwiw represents a collection as either top-level or embedded, not both, so it cannot emit an `embedded_in` config that would silently make the collection undumpable. These unrepresentable shapes are handled best-effort by default and abort only in strict mode (see below).
|
|
397
|
-
|
|
398
|
-
Models in an inheritance hierarchy whose subclasses share the base's collection (Mongoid STI, distinguished by the auto-added `_type` discriminator) collapse into a single config: the generator discovers the subclasses via `descendants` (Mongoid registers only the base class in `Mongoid.models`) and unions every class's `fields` and `belongs_tos` into the collection config, so subclass-only fields and associations are not lost.
|
|
399
|
-
|
|
400
|
-
Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, `bulk_insert_chunk_size`, and `query_timeout_ms` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [MongoDB notes](#mongodb-notes)). Polymorphic `belongs_to` is not yet expanded by this task.
|
|
401
|
-
|
|
402
|
-
By default the task **aborts** when a model uses a construct exwiw cannot represent: a `belongs_to` whose target class can no longer be resolved (a stale relation left behind after its model was removed), or a polymorphic / self-referential-cyclic / ambiguous / unresolvable-parent `embedded_in` (see the cases above).
|
|
403
|
-
|
|
404
|
-
#### Honoring an explicit `ignore` (the recommended way to keep these out)
|
|
405
|
-
|
|
406
|
-
When you have reviewed such a construct and decided exwiw should leave it alone, mark it `ignore: true` in its config on disk. The generator **honors an explicit `ignore` and skips re-introspecting it**, so it never aborts the run on something you have already triaged — and your annotation survives regeneration. Two granularities:
|
|
407
|
-
|
|
408
|
-
- A whole **collection** exwiw cannot represent (e.g. a polymorphic / ambiguous `embedded_in`) — mark the collection config `"ignore": true`. To actually dump/mask it later, define its `embedded_in` config by hand (see [Embedded documents](#embedded-documents)).
|
|
409
|
-
- A single **`belongs_to`** that no longer resolves while the rest of its collection is fine (e.g. a stale relation pointing at a removed model) — mark that entry `"ignore": true`, with no `table_name`. The relation is dropped from extraction (`#reject_ignored_members!`) while its foreign-key column stays an ordinary field, and the collection keeps dumping.
|
|
410
|
-
|
|
411
|
-
Record *why* with the optional **`ignore_type`** (a free-form tag exwiw never interprets — e.g. `"need_code_fix"` for an application-side bug, `"unsupported"` for a shape exwiw cannot express) and a **`comment`**. Both are user-owned and preserved across regeneration; the generator never emits `ignore_type` itself.
|
|
412
|
-
|
|
413
|
-
```json
|
|
414
|
-
// orders.json — a stale belongs_to flagged for a code fix; the collection still dumps
|
|
415
|
-
{
|
|
416
|
-
"name": "orders",
|
|
417
|
-
"primary_key": "_id",
|
|
418
|
-
"belongs_to": [
|
|
419
|
-
{ "table_name": "shops", "foreign_key": "shop_id" },
|
|
420
|
-
{
|
|
421
|
-
"foreign_key": "coupon_id",
|
|
422
|
-
"ignore": true,
|
|
423
|
-
"ignore_type": "need_code_fix",
|
|
424
|
-
"comment": "FIXME: belongs_to :coupon -> Coupon does not exist (dead relation)."
|
|
425
|
-
}
|
|
426
|
-
],
|
|
427
|
-
"fields": [ /* ... coupon_id is kept as an ordinary field ... */ ]
|
|
428
|
-
}
|
|
429
|
-
```
|
|
430
|
-
|
|
431
|
-
#### First bootstrap pass: `EXWIW_SKIP_UNSUPPORTED=1`
|
|
432
|
-
|
|
433
|
-
For the very first pass against a large app — before any `ignore` annotations exist — set `EXWIW_SKIP_UNSUPPORTED=1` to keep going past *un-annotated* unrepresentable constructs instead of aborting one at a time:
|
|
434
|
-
|
|
435
|
-
```bash
|
|
436
|
-
EXWIW_SKIP_UNSUPPORTED=1 bundle exec rake exwiw:schema:generate_mongoid
|
|
437
|
-
```
|
|
438
|
-
|
|
439
|
-
- An unresolvable `belongs_to` is dropped from the collection's `belongs_tos` (its foreign-key column is still kept as an ordinary field, like the polymorphic / HABTM cases) and a warning naming the relation is printed to stderr.
|
|
440
|
-
- An unrepresentable `embedded_in` collection is emitted as a **top-level** config marked `"ignore": true` with a `comment` recording why, and a warning is printed.
|
|
441
|
-
|
|
442
|
-
Review the stderr warnings, annotate the affected configs (`ignore` / `ignore_type` / `comment`), and subsequent runs complete without the flag because the generator honors those explicit ignores.
|
|
394
|
+
What it derives from each model (fields, `belongs_tos`, `embedded_in`, STI handling), how to annotate constructs exwiw cannot represent with `ignore` / `ignore_type`, and the `EXWIW_SKIP_UNSUPPORTED=1` bootstrap flag are all documented in [MongoDB support](docs/mongodb.md#generating-config-from-mongoid-models).
|
|
443
395
|
|
|
444
396
|
### Configuration
|
|
445
397
|
|
|
@@ -717,7 +669,7 @@ Notes:
|
|
|
717
669
|
- **NULLs are excluded** per arm (`IS NOT NULL`).
|
|
718
670
|
- **Satellites need no config.** A table that `belongs_to` the reverse-scoped table (e.g. `end_users.id → users.id`, or `identities.user_id → users.id`) tightens to the kept ids automatically through the normal cascade — only the reverse-scoped table itself declares `reverse_scope`. The cascade is **multi-hop**, so a table several `belongs_to` hops below the reverse-scoped table (e.g. `end_user_profiles → end_users → users`) also tightens automatically, with no config of its own.
|
|
719
671
|
- Works in both single-target and scope-column mode. In single-target mode there is no scope-column pre-flight (`validate_scope!`), so a satellite the cascade cannot resolve to a single scopable parent (e.g. it `belongs_to` two scopable hubs) is dumped in full with a warning rather than aborting. Polymorphic foreign keys are not eligible as anchors (the named `column` is always a concrete column).
|
|
720
|
-
- **The MongoDB adapter supports `reverse_scope` too** — same config shape and semantics, but the id set is captured at runtime instead of being emitted as a `UNION` subquery. See [`reverse_scope` on collections](#reverse_scope-on-collections) under MongoDB
|
|
672
|
+
- **The MongoDB adapter supports `reverse_scope` too** — same config shape and semantics, but the id set is captured at runtime instead of being emitted as a `UNION` subquery. See [`reverse_scope` on collections](docs/mongodb.md#reverse_scope-on-collections) under MongoDB support.
|
|
721
673
|
|
|
722
674
|
### Why a JOIN, not `IN (subquery)`
|
|
723
675
|
|
|
@@ -786,6 +738,60 @@ Unlike rails-managed entries, `columns` and `belongs_tos` are retained so the en
|
|
|
786
738
|
|
|
787
739
|
If omitted, the adapter default applies: 10,000 rows per statement for the SQL adapters (1,000 documents per chunk for MongoDB). Tables at or below the chunk size still produce a single `INSERT` statement. To force a single statement regardless of table size, set a value larger than the table's row count.
|
|
788
740
|
|
|
741
|
+
### Batched extraction (`batch_scope`)
|
|
742
|
+
|
|
743
|
+
A scoped table is normally extracted with one query, whose scope filter sits on the table it joins up to:
|
|
744
|
+
|
|
745
|
+
```sql
|
|
746
|
+
SELECT activities.* FROM activities
|
|
747
|
+
JOIN customers ON activities.customer_id = customers.id
|
|
748
|
+
AND customers.tenant_id IN ('t1')
|
|
749
|
+
```
|
|
750
|
+
|
|
751
|
+
That is index-driven while the scope keeps few `customers`. Past some number of them the planner's estimate of "probe the foreign-key index once per customer" exceeds its estimate of "scan the table once", and it switches to a **sequential scan of the whole table** — for a result set that is a small fraction of it. On a table of hundreds of millions of rows the scan then exceeds the server's `statement_timeout`, or simply runs for hours. Note that no `filter` on the extracted table fixes this: a predicate that reduces the *output* does not reduce the *work* once the plan is a scan (it may not even change the plan).
|
|
752
|
+
|
|
753
|
+
`batch_scope` removes the choice instead of arguing with the estimate. It names the scoped table this one reaches — the **batch table** — and exwiw resolves that table's in-scope primary keys once, then extracts one `size`-sized slice of those ids at a time:
|
|
754
|
+
|
|
755
|
+
```json
|
|
756
|
+
{
|
|
757
|
+
"name": "activities",
|
|
758
|
+
"primary_key": "id",
|
|
759
|
+
"batch_scope": { "table": "customers", "size": 1000 },
|
|
760
|
+
"belongs_tos": [{ "table_name": "customers", "foreign_key": "customer_id" }],
|
|
761
|
+
"columns": [{ "name": "id" }, { "name": "customer_id" }]
|
|
762
|
+
}
|
|
763
|
+
```
|
|
764
|
+
|
|
765
|
+
Each batch runs with that slice's ids in place of the scope filter:
|
|
766
|
+
|
|
767
|
+
```sql
|
|
768
|
+
SELECT activities.* FROM activities
|
|
769
|
+
JOIN customers ON activities.customer_id = customers.id
|
|
770
|
+
AND customers.id IN (/* 1000 ids */)
|
|
771
|
+
```
|
|
772
|
+
|
|
773
|
+
An explicit id list of that size is exactly estimated and selective, so the foreign-key index is unambiguously the cheapest plan for every batch, and total work is proportional to the rows the table actually keeps rather than to the table's size.
|
|
774
|
+
|
|
775
|
+
- **The dumped rows are the same as the unbatched query's** (in batch-by-batch order). The slices partition the id set — every id is in exactly one batch — so no row is dropped or emitted twice. The ids are sorted (in exwiw, not with `ORDER BY` — the id-set query stays cheap on the source DB) before slicing, so batch composition, and the dump, is reproducible run to run.
|
|
776
|
+
- **`size` defaults to 1000** ids per batch.
|
|
777
|
+
- The batch table's ids come from **its own extraction query**, so it is narrowed by exactly the filter it would carry in the unbatched query. They are held in memory for the extraction: one scope's worth of primary keys, orders of magnitude smaller than the table being batched.
|
|
778
|
+
- The batch table may be **any number of hops up** the path — a table two hops below it (`activity_orders → activities → customers`) names `customers` too, and the batch ids are applied where the path meets the scope, bounding the whole join chain.
|
|
779
|
+
- A table that **carries the scope column itself** batches by naming itself; each batch then filters `WHERE <pk> IN (<ids>)` directly. Note that the id-set query is then the same scope predicate over the same table, so this shape only avoids the scan when the scope column is indexed (ideally index-only) — the join shape above is the one that genuinely removes the planner's choice.
|
|
780
|
+
- `delete-*.sql` is unaffected (it is generated from the unbatched query).
|
|
781
|
+
- `bulk_insert_chunk_size` is independent: batches are query boundaries, chunks are `INSERT` statement boundaries.
|
|
782
|
+
- With `--output-format=copy`, batching bounds each query's cost but not memory: COPY builds the whole table's body in memory, so all batches' rows are resident at once. Use the default INSERT format (which streams) when the kept rows themselves are huge.
|
|
783
|
+
|
|
784
|
+
**Supported shapes.** A batch key only splits an extraction correctly when *every* row the table keeps is selected through the batch table's scope filter — otherwise a route the batch key does not constrain would keep the same rows in every batch, and the dump would repeat them (a primary-key conflict on import). So `batch_scope` requires [scope-column mode](#scope-column-mode) and one of:
|
|
785
|
+
|
|
786
|
+
- the table is **directly scoped** (`scope_column`) and names itself, or
|
|
787
|
+
- the table reaches the scope through a **single `belongs_to` join path** (path 2 in [the six scoping paths](#how-each-table-is-narrowed--the-six-scoping-paths)) whose scoped terminus is the named table.
|
|
788
|
+
|
|
789
|
+
Every other shape — polymorphic arm `UNION`s, `reverse_scope`, referenced-by, the parent cascade, `scope_exempt` (on the batched table *or* the batch table, whose id set would then not be scoped), and single `--target-table` mode — is **rejected with an explanation** rather than silently mis-sliced, before any output is written. (In single-target mode the extraction is already anchored on a caller-supplied id list, so batching it means running exwiw once per slice of `--ids`.)
|
|
790
|
+
|
|
791
|
+
`exwiw explain` prints the id-set query and its `EXPLAIN` after a batched table's own query, since that query is the part of a batched export the table's query does not show. It cannot show a batch's literal id list — `explain` resolves no ids, because it executes no extraction SELECT.
|
|
792
|
+
|
|
793
|
+
Like `scope_column` / `scope_exempt` / `reverse_scope`, `batch_scope` is user-maintained: never emitted by `schema:generate`, and preserved across regeneration.
|
|
794
|
+
|
|
789
795
|
### Filter
|
|
790
796
|
|
|
791
797
|
Some case, you don't need full records related to target. e.g. dump user access logs only for the last year.
|
|
@@ -793,6 +799,7 @@ Some case, you don't need full records related to target. e.g. dump user access
|
|
|
793
799
|
|
|
794
800
|
- injected as it is in table condition(e.g. WHERE on mysql), so you are recommended to clearify table name of column to avoid ambiguity.
|
|
795
801
|
- injected to every where / join clause, so it affects to all tables depends on filterted target-table. it results to data inconsistency.
|
|
802
|
+
- a way to reduce the rows returned, which is **not** necessarily a way to reduce the work: on a large table the engine may keep (or switch to) a full scan and evaluate the filter per row. See [batched extraction](#batched-extraction-batch_scope) when the goal is to bound how much of the table is read.
|
|
796
803
|
|
|
797
804
|
### Masking
|
|
798
805
|
|
|
@@ -935,7 +942,7 @@ fake value, across tables, runs, and adapters:
|
|
|
935
942
|
- Exclusive with the other masking keys on the same column, and invisible to
|
|
936
943
|
`explain`. Also supported by the MongoDB adapter on a `MongodbField` (seed
|
|
937
944
|
names a field of the collection, or `_id`), where it is applied document-side
|
|
938
|
-
after `replace_with` — see [MongoDB
|
|
945
|
+
after `replace_with` — see [MongoDB support](docs/mongodb.md#masking).
|
|
939
946
|
|
|
940
947
|
**Performance**: this is a per-row Ruby transform, measured at ~1.5–1.6µs/row
|
|
941
948
|
per fake column (so ≈ +8s per 5M rows per column; ~+40% against a local sqlite
|
|
@@ -946,100 +953,9 @@ unaffected: the transform streams with the dump. See
|
|
|
946
953
|
[`docs/row-transform-masking-notes.md`](docs/row-transform-masking-notes.md)
|
|
947
954
|
for the benchmark, and `script/bench_row_transform.rb` to measure on your data.
|
|
948
955
|
|
|
949
|
-
### MongoDB
|
|
950
|
-
|
|
951
|
-
The MongoDB adapter is experimental. To use it:
|
|
952
|
-
|
|
953
|
-
- Add `gem "mongo"` to your Gemfile in addition to `exwiw` (it is not declared as a runtime dependency of the gem).
|
|
954
|
-
- Set `--adapter=mongodb`. `--user` / `DATABASE_PASSWORD` are optional and only needed when your MongoDB requires authentication.
|
|
955
|
-
- `--uri=URI` connects with a full MongoDB connection string (`mongodb://...` or `mongodb+srv://...`) instead of `--host`/`--port`. Use it for managed/replica-set deployments (e.g. Atlas) where TLS, `replicaSet`, `authSource`, and credentials must be expressed — put them in the URI's query string (e.g. `mongodb+srv://user:pass@cluster.example.com/?authSource=admin&tls=true`). The URI takes precedence: when given, `--host`/`--port`/`--user`/`DATABASE_PASSWORD` are ignored, and `--host`/`--port`/`--database` are no longer required on the CLI. `--database`, if still passed, overrides the database in the URI path; otherwise the database from the URI is used. This flag is **mongodb-only** (the SQL adapters shell out to their own client binaries and have no equivalent). The URI may carry credentials, so it is never written to logs.
|
|
956
|
-
- The MongoDB adapter consumes a separate config type, `MongodbCollectionConfig`, with MongoDB-native naming. Use `fields` (instead of the SQL adapters' `columns`), and set `"primary_key": "_id"`. Foreign keys (`shop_id`, `user_id`, ...) stay as ordinary fields.
|
|
957
|
-
- `--ids` values are coerced to the type actually stored in `_id` before filtering: integer-looking ids become `Integer`, 24-char hex ids become `BSON::ObjectId` (Mongoid's default `_id` type — a plain String would never match an ObjectId), and any other string is left as-is.
|
|
958
|
-
- `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
|
|
959
|
-
- `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only** (the SQL adapters have no equivalent).
|
|
960
|
-
- Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
|
|
961
|
-
- `--mongodb-query-timeout-ms=N` sets a global, **server-enforced** timeout (in milliseconds) on every query exwiw issues — the find cursor's whole lifetime (the initial batch and every `getMore` the streaming dump walks), the count, and an executing `explain`. Past the deadline the server aborts the operation and exwiw fails the run, so an accidentally heavy or unscoped query cannot keep pinning the (often production) source. It is **mongodb-only** (the SQL adapters shell out to their own clients) and may also be set as `mongodb_query_timeout_ms:` in the config file. The default is no timeout. A single collection can opt to a different limit with a `query_timeout_ms` key in its schema config (a sibling of `bulk_insert_chunk_size`), which overrides the global for that collection's find/count — use it to give a known-large collection more headroom, or to cap one that tends to run away. Hand-edited `query_timeout_ms` values are preserved across schema regeneration.
|
|
962
|
-
- `--parallel-workers=N` (opt-in, `export` only) forks `N` worker processes that decode whole collections in parallel — the dominant cost on a large dump is the driver's BSON→Ruby decode, and each worker decodes its own collections in their natural order, so the output stays **byte-identical** to a serial run (same filenames and content). It needs a dump target (the schedule is built around the scoped DAG) and a `fork`-capable runtime (CRuby on POSIX), falling back to the serial path otherwise; it also accepts `parallel_workers:` in the config file. The speedup needs real cores to spend — it reaches ~2× from 4 workers and saturates there. The default is serial. See [`docs/mongodb-dump-parallelism-2x-notes.md`](docs/mongodb-dump-parallelism-2x-notes.md) for the schedule and measurements.
|
|
963
|
-
- Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
|
|
964
|
-
```bash
|
|
965
|
-
mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl
|
|
966
|
-
```
|
|
967
|
-
- A Ruby [after-insert hook](#after-insert-hook) can seed extra documents into named collections with `insert_jsonl(collection, template)`; each targeted collection gets its own `insert-NNN-<collection>.jsonl` file numbered after the dump's own files, so the filename-based `mongoimport` convention above applies to hook output unchanged.
|
|
968
|
-
- The leading `dump/insert-000-schema.js` contains `db.createCollection(...)` and `db.<col>.createIndex(...)` calls for every top-level collection (indexes are introspected from the source via `listIndexes`; the auto-created `_id_` index is skipped). Apply it with mongosh **before** running `mongoimport`:
|
|
969
|
-
```bash
|
|
970
|
-
mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
|
|
971
|
-
```
|
|
972
|
-
- Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
|
|
973
|
-
- `replace_with_fake_data` is supported on a field ([full reference](#replace_with_fake_data)) — its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the same fake value as the SQL adapters for the same seed. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), works inside embedded subdocuments, and is exclusive with `replace_with` on the same field. Add `gem "faker"` to use it (except a config using only `ja` person types). `raw_sql` and `map` are **not** supported (the `MongodbField` schema does not declare them; such keys are rejected on load — see [Unknown keys are rejected](#unknown-keys-are-rejected)); use `replace_with` for template masking.
|
|
974
|
-
- The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
|
|
975
|
-
|
|
976
|
-
#### `reverse_scope` on collections
|
|
977
|
-
|
|
978
|
-
[Multi-referencer reverse scoping](#reverse-scope-for-multi-referencer-tables-reverse_scope) works on `MongodbCollectionConfig` with the same config shape and the same semantics as the SQL adapters — a global-identity collection (say `accounts`) with no `belongs_to` path to the dump target, but referenced by several scoped collections, is constrained to the union of the ids those referencers actually point at instead of being dumped in full:
|
|
979
|
-
|
|
980
|
-
```json
|
|
981
|
-
{
|
|
982
|
-
"name": "accounts",
|
|
983
|
-
"primary_key": "_id",
|
|
984
|
-
"reverse_scope": {
|
|
985
|
-
"via": [
|
|
986
|
-
{ "table": "articles", "column": "author_account_id" },
|
|
987
|
-
{ "table": "invitations", "column": "invitee_account_id" }
|
|
988
|
-
]
|
|
989
|
-
},
|
|
990
|
-
"belongs_tos": [],
|
|
991
|
-
"fields": [{ "name": "_id" }, { "name": "name" }]
|
|
992
|
-
}
|
|
993
|
-
```
|
|
994
|
-
|
|
995
|
-
Where the SQL adapters emit a `UNION` subquery, MongoDB has no cross-collection joins, so the adapter captures each arm's column values **at runtime** while the referencer collection streams (the same mechanism that already propagates parent ids to children), then filters the reverse-scoped collection with `{"_id": {"$in": [<union of captured ids>]}}`. Consequences of that runtime capture:
|
|
996
|
-
|
|
997
|
-
- **Processing order**: a reverse-scoped collection is dumped **after** all of its `via` referencers (an arm's own `belongs_to` back to the reverse-scoped collection is inverted rather than kept — the declaration states ids flow referencer → collection). If the arms form a genuine ordering cycle with the `belongs_to` graph, the export aborts with an error naming the cycle members. SQL processing order is unchanged (its INSERT output must stay loadable in foreign-key order).
|
|
998
|
-
- **Arm hygiene mirrors SQL**: an arm whose referencer is unknown, embedded, not dumped, or itself unscoped (no path to the dump target and no `reverse_scope` of its own) is **skipped with a warning** — an unscoped referencer's ids span every scope and would silently widen the dump. Per-arm `null`/absent foreign keys are dropped (the SQL `IS NOT NULL`), an array-valued foreign-key column contributes one id per element, and captured values keep their native BSON types (an `ObjectId` foreign key matches an `ObjectId` `_id` with no coercion).
|
|
999
|
-
- **Precedence mirrors SQL**: a collection with its own `belongs_to` path to the dump target is scoped by that path; a `reverse_scope` declared on it is ignored.
|
|
1000
|
-
- **Satellites need no config**, as in SQL: a collection that `belongs_to` the reverse-scoped collection tightens to the kept ids automatically through the ordinary captured-parent-id mechanism.
|
|
1001
|
-
- **`--parallel-workers` falls back to serial** (with a warning) when any collection declares `reverse_scope` — the parallel schedule does not express the referencers-first ordering constraint yet.
|
|
1002
|
-
- **`exwiw explain`** shows the real `{"_id": {"$in": [...]}}` filter shape with a placeholder id, like the other runtime-captured scopes.
|
|
1003
|
-
|
|
1004
|
-
Masking (`replace_with`) and `fields` behavior on a reverse-scoped collection are unchanged. Like the SQL key, `reverse_scope` is user-owned: `exwiw:mongoid:schema:generate` never emits it and regeneration preserves a hand-added value.
|
|
1005
|
-
|
|
1006
|
-
#### Embedded documents
|
|
1007
|
-
|
|
1008
|
-
MongoDB models often store one-to-many relationships as embedded subdocument arrays (e.g. `users` documents with a `posts: [...]` field). To mask fields inside embedded subdocuments, declare a separate config with `embedded_in`:
|
|
1009
|
-
|
|
1010
|
-
```jsonc
|
|
1011
|
-
// e2e/users.json — top-level collection
|
|
1012
|
-
{
|
|
1013
|
-
"name": "users",
|
|
1014
|
-
"primary_key": "_id",
|
|
1015
|
-
"belongs_tos": [{ "table_name": "shops", "foreign_key": "shop_id" }],
|
|
1016
|
-
"fields": [
|
|
1017
|
-
{ "name": "_id" },
|
|
1018
|
-
{ "name": "name", "replace_with": "masked{_id}" },
|
|
1019
|
-
{ "name": "shop_id" }
|
|
1020
|
-
]
|
|
1021
|
-
}
|
|
1022
|
-
|
|
1023
|
-
// e2e/posts.json — embedded under users.posts
|
|
1024
|
-
{
|
|
1025
|
-
"name": "posts",
|
|
1026
|
-
"primary_key": "_id",
|
|
1027
|
-
"embedded_in": { "collection_name": "users", "path": "posts" },
|
|
1028
|
-
"belongs_tos": [],
|
|
1029
|
-
"fields": [
|
|
1030
|
-
{ "name": "_id" },
|
|
1031
|
-
{ "name": "title", "replace_with": "masked-{_id}" }
|
|
1032
|
-
]
|
|
1033
|
-
}
|
|
1034
|
-
```
|
|
1035
|
-
|
|
1036
|
-
At runtime:
|
|
956
|
+
### MongoDB
|
|
1037
957
|
|
|
1038
|
-
|
|
1039
|
-
- `path` accepts dot-separated paths for nested fields (e.g. `"profile.contacts"`).
|
|
1040
|
-
- Both arrays of subdocuments and a single Hash subdocument at `path` are supported. Multiple levels of nesting work via embedded chains.
|
|
1041
|
-
- Cross-collection references from inside an embedded subdocument (`belongs_tos` on an embedded config) are not supported and raise `ArgumentError` on load.
|
|
1042
|
-
- Specifying an embedded config as `--target-table` raises `NotImplementedError`; pass the top-level collection name instead.
|
|
958
|
+
exwiw can export MongoDB databases too (`--adapter=mongodb`): JSONL output importable with `mongoimport`, schema/index DDL for `mongosh`, masking inside embedded documents, `reverse_scope` on collections, Mongoid-based config generation, a server-enforced query timeout, and parallel dump workers. Everything MongoDB-specific is documented in [docs/mongodb.md](docs/mongodb.md).
|
|
1043
959
|
|
|
1044
960
|
## How it works
|
|
1045
961
|
|
data/docs/mongodb.md
ADDED
|
@@ -0,0 +1,192 @@
|
|
|
1
|
+
# MongoDB support
|
|
2
|
+
|
|
3
|
+
exwiw can export a MongoDB database with `--adapter=mongodb`. This document collects everything MongoDB-specific: setup, CLI flags, output format, masking, scoping on collections, embedded documents, `explain`, and generating config from Mongoid models. Everything else (masking reference, `reverse_scope` semantics, config file, hooks, ...) is shared with the SQL adapters and documented in the [README](../README.md).
|
|
4
|
+
|
|
5
|
+
## Setup
|
|
6
|
+
|
|
7
|
+
- Add `gem "mongo"` to your Gemfile in addition to `exwiw` (it is not declared as a runtime dependency of the gem).
|
|
8
|
+
- Set `--adapter=mongodb`. `--user` / `DATABASE_PASSWORD` are optional and only needed when your MongoDB requires authentication.
|
|
9
|
+
|
|
10
|
+
## Connecting and selecting the target
|
|
11
|
+
|
|
12
|
+
- `--uri=URI` connects with a full MongoDB connection string (`mongodb://...` or `mongodb+srv://...`) instead of `--host`/`--port`. Use it for managed/replica-set deployments (e.g. Atlas) where TLS, `replicaSet`, `authSource`, and credentials must be expressed — put them in the URI's query string (e.g. `mongodb+srv://user:pass@cluster.example.com/?authSource=admin&tls=true`). The URI takes precedence: when given, `--host`/`--port`/`--user`/`DATABASE_PASSWORD` are ignored, and `--host`/`--port`/`--database` are no longer required on the CLI. `--database`, if still passed, overrides the database in the URI path; otherwise the database from the URI is used. This flag is **mongodb-only** (the SQL adapters shell out to their own client binaries and have no equivalent). The URI may carry credentials, so it is never written to logs.
|
|
13
|
+
- The MongoDB adapter consumes a separate config type, `MongodbCollectionConfig`, with MongoDB-native naming. Use `fields` (instead of the SQL adapters' `columns`), and set `"primary_key": "_id"`. Foreign keys (`shop_id`, `user_id`, ...) stay as ordinary fields.
|
|
14
|
+
- `--ids` values are coerced to the type actually stored in `_id` before filtering: integer-looking ids become `Integer`, 24-char hex ids become `BSON::ObjectId` (Mongoid's default `_id` type — a plain String would never match an ObjectId), and any other string is left as-is.
|
|
15
|
+
- `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
|
|
16
|
+
- `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only** (the SQL adapters have no equivalent).
|
|
17
|
+
|
|
18
|
+
## Performance and safety
|
|
19
|
+
|
|
20
|
+
- Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`optimization-notes.md`](optimization-notes.md) for the performance investigation and [`optimize-mongodb-export-with-native-ext.md`](optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
|
|
21
|
+
- `--mongodb-query-timeout-ms=N` sets a global, **server-enforced** timeout (in milliseconds) on every query exwiw issues — the find cursor's whole lifetime (the initial batch and every `getMore` the streaming dump walks), the count, and an executing `explain`. Past the deadline the server aborts the operation and exwiw fails the run, so an accidentally heavy or unscoped query cannot keep pinning the (often production) source. It is **mongodb-only** (the SQL adapters shell out to their own clients) and may also be set as `mongodb_query_timeout_ms:` in the config file. The default is no timeout. A single collection can opt to a different limit with a `query_timeout_ms` key in its schema config (a sibling of `bulk_insert_chunk_size`), which overrides the global for that collection's find/count — use it to give a known-large collection more headroom, or to cap one that tends to run away. Hand-edited `query_timeout_ms` values are preserved across schema regeneration.
|
|
22
|
+
- `--parallel-workers=N` (opt-in, `export` only) forks `N` worker processes that decode whole collections in parallel — the dominant cost on a large dump is the driver's BSON→Ruby decode, and each worker decodes its own collections in their natural order, so the output stays **byte-identical** to a serial run (same filenames and content). It needs a dump target (the schedule is built around the scoped DAG) and a `fork`-capable runtime (CRuby on POSIX), falling back to the serial path otherwise; it also accepts `parallel_workers:` in the config file. The speedup needs real cores to spend — it reaches ~2× from 4 workers and saturates there. The default is serial. See [`mongodb-dump-parallelism-2x-notes.md`](mongodb-dump-parallelism-2x-notes.md) for the schedule and measurements.
|
|
23
|
+
|
|
24
|
+
## Output
|
|
25
|
+
|
|
26
|
+
- Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
|
|
27
|
+
```bash
|
|
28
|
+
mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl
|
|
29
|
+
```
|
|
30
|
+
- A Ruby [after-insert hook](../README.md#after-insert-hook) can seed extra documents into named collections with `insert_jsonl(collection, template)`; each targeted collection gets its own `insert-NNN-<collection>.jsonl` file numbered after the dump's own files, so the filename-based `mongoimport` convention above applies to hook output unchanged.
|
|
31
|
+
- The leading `dump/insert-000-schema.js` contains `db.createCollection(...)` and `db.<col>.createIndex(...)` calls for every top-level collection (indexes are introspected from the source via `listIndexes`; the auto-created `_id_` index is skipped). Apply it with mongosh **before** running `mongoimport`:
|
|
32
|
+
```bash
|
|
33
|
+
mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
|
|
34
|
+
```
|
|
35
|
+
- Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
|
|
36
|
+
|
|
37
|
+
## Masking
|
|
38
|
+
|
|
39
|
+
- `replace_with_fake_data` is supported on a field ([full reference](../README.md#replace_with_fake_data)) — its `seed` names a field of the same collection (bare or `collection.`-qualified) or the primary key (`_id`), and it derives the same fake value as the SQL adapters for the same seed. It is applied document-side after `replace_with` (so a fake seed reads the already-masked value, matching the SQL adapters where `replace_with` runs in the database first), works inside embedded subdocuments, and is exclusive with `replace_with` on the same field. Add `gem "faker"` to use it (except a config using only `ja` person types). `raw_sql` and `map` are **not** supported (the `MongodbField` schema does not declare them; such keys are rejected on load — see [Unknown keys are rejected](../README.md#unknown-keys-are-rejected)); use `replace_with` for template masking.
|
|
40
|
+
- The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
|
|
41
|
+
|
|
42
|
+
## `reverse_scope` on collections
|
|
43
|
+
|
|
44
|
+
[Multi-referencer reverse scoping](../README.md#reverse-scope-for-multi-referencer-tables-reverse_scope) works on `MongodbCollectionConfig` with the same config shape and the same semantics as the SQL adapters — a global-identity collection (say `accounts`) with no `belongs_to` path to the dump target, but referenced by several scoped collections, is constrained to the union of the ids those referencers actually point at instead of being dumped in full:
|
|
45
|
+
|
|
46
|
+
```json
|
|
47
|
+
{
|
|
48
|
+
"name": "accounts",
|
|
49
|
+
"primary_key": "_id",
|
|
50
|
+
"reverse_scope": {
|
|
51
|
+
"via": [
|
|
52
|
+
{ "table": "articles", "column": "author_account_id" },
|
|
53
|
+
{ "table": "invitations", "column": "invitee_account_id" }
|
|
54
|
+
]
|
|
55
|
+
},
|
|
56
|
+
"belongs_tos": [],
|
|
57
|
+
"fields": [{ "name": "_id" }, { "name": "name" }]
|
|
58
|
+
}
|
|
59
|
+
```
|
|
60
|
+
|
|
61
|
+
Where the SQL adapters emit a `UNION` subquery, MongoDB has no cross-collection joins, so the adapter captures each arm's column values **at runtime** while the referencer collection streams (the same mechanism that already propagates parent ids to children), then filters the reverse-scoped collection with `{"_id": {"$in": [<union of captured ids>]}}`. Consequences of that runtime capture:
|
|
62
|
+
|
|
63
|
+
- **Processing order**: a reverse-scoped collection is dumped **after** all of its `via` referencers (an arm's own `belongs_to` back to the reverse-scoped collection is inverted rather than kept — the declaration states ids flow referencer → collection). If the arms form a genuine ordering cycle with the `belongs_to` graph, the export aborts with an error naming the cycle members. SQL processing order is unchanged (its INSERT output must stay loadable in foreign-key order).
|
|
64
|
+
- **Arm hygiene mirrors SQL**: an arm whose referencer is unknown, embedded, not dumped, or itself unscoped (no path to the dump target and no `reverse_scope` of its own) is **skipped with a warning** — an unscoped referencer's ids span every scope and would silently widen the dump. Per-arm `null`/absent foreign keys are dropped (the SQL `IS NOT NULL`), an array-valued foreign-key column contributes one id per element, and captured values keep their native BSON types (an `ObjectId` foreign key matches an `ObjectId` `_id` with no coercion).
|
|
65
|
+
- **Precedence mirrors SQL**: a collection with its own `belongs_to` path to the dump target is scoped by that path; a `reverse_scope` declared on it is ignored.
|
|
66
|
+
- **Satellites need no config**, as in SQL: a collection that `belongs_to` the reverse-scoped collection tightens to the kept ids automatically through the ordinary captured-parent-id mechanism.
|
|
67
|
+
- **`--parallel-workers` falls back to serial** (with a warning) when any collection declares `reverse_scope` — the parallel schedule does not express the referencers-first ordering constraint yet.
|
|
68
|
+
- **`exwiw explain`** shows the real `{"_id": {"$in": [...]}}` filter shape with a placeholder id, like the other runtime-captured scopes.
|
|
69
|
+
|
|
70
|
+
Masking (`replace_with`) and `fields` behavior on a reverse-scoped collection are unchanged. Like the SQL key, `reverse_scope` is user-owned: `exwiw:mongoid:schema:generate` never emits it and regeneration preserves a hand-added value.
|
|
71
|
+
|
|
72
|
+
## Embedded documents
|
|
73
|
+
|
|
74
|
+
MongoDB models often store one-to-many relationships as embedded subdocument arrays (e.g. `users` documents with a `posts: [...]` field). To mask fields inside embedded subdocuments, declare a separate config with `embedded_in`:
|
|
75
|
+
|
|
76
|
+
```jsonc
|
|
77
|
+
// e2e/users.json — top-level collection
|
|
78
|
+
{
|
|
79
|
+
"name": "users",
|
|
80
|
+
"primary_key": "_id",
|
|
81
|
+
"belongs_tos": [{ "table_name": "shops", "foreign_key": "shop_id" }],
|
|
82
|
+
"fields": [
|
|
83
|
+
{ "name": "_id" },
|
|
84
|
+
{ "name": "name", "replace_with": "masked{_id}" },
|
|
85
|
+
{ "name": "shop_id" }
|
|
86
|
+
]
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
// e2e/posts.json — embedded under users.posts
|
|
90
|
+
{
|
|
91
|
+
"name": "posts",
|
|
92
|
+
"primary_key": "_id",
|
|
93
|
+
"embedded_in": { "collection_name": "users", "path": "posts" },
|
|
94
|
+
"belongs_tos": [],
|
|
95
|
+
"fields": [
|
|
96
|
+
{ "name": "_id" },
|
|
97
|
+
{ "name": "title", "replace_with": "masked-{_id}" }
|
|
98
|
+
]
|
|
99
|
+
}
|
|
100
|
+
```
|
|
101
|
+
|
|
102
|
+
At runtime:
|
|
103
|
+
|
|
104
|
+
- `posts` is **not** dumped as its own jsonl file. Its `replace_with` rules are applied to the subdocuments inside the parent `users` document at the path `posts`.
|
|
105
|
+
- `path` accepts dot-separated paths for nested fields (e.g. `"profile.contacts"`).
|
|
106
|
+
- Both arrays of subdocuments and a single Hash subdocument at `path` are supported. Multiple levels of nesting work via embedded chains.
|
|
107
|
+
- Cross-collection references from inside an embedded subdocument (`belongs_tos` on an embedded config) are not supported and raise `ArgumentError` on load.
|
|
108
|
+
- Specifying an embedded config as `--target-table` raises `NotImplementedError`; pass the top-level collection name instead.
|
|
109
|
+
|
|
110
|
+
## `exwiw explain` verbosity
|
|
111
|
+
|
|
112
|
+
The mongodb explain runs the server's [explain command](https://www.mongodb.com/docs/manual/reference/command/explain/) at a configurable verbosity. The default, **`queryPlanner`, only plans the query and does not execute it**, so it is safe to point at a production source. Set it with the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` environment variable or the `explain_verbosity:` config key (the env var wins):
|
|
113
|
+
|
|
114
|
+
| verbosity | behaviour |
|
|
115
|
+
|---|---|
|
|
116
|
+
| `queryPlanner` (default) | Plans the query only. **The query is not executed** — no documents are scanned. |
|
|
117
|
+
| `executionStats` | **Runs the query** and reports runtime statistics (docs examined, time, etc.). |
|
|
118
|
+
| `allPlansExecution` | Runs the winning plan **and the rejected candidate plans** to gather their stats. |
|
|
119
|
+
|
|
120
|
+
```bash
|
|
121
|
+
# inspect index usage of the real extraction query (executes it)
|
|
122
|
+
EXWIW_MONGODB_EXPLAIN_VERBOSITY=executionStats exwiw explain \
|
|
123
|
+
--adapter=mongodb \
|
|
124
|
+
--uri="mongodb+srv://reader@cluster/app_production" \
|
|
125
|
+
--schema-dir=exwiw/schema \
|
|
126
|
+
--target-collection=shops --ids=...
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
`executionStats` and `allPlansExecution` execute the extraction query against the source, so use them deliberately on large/production collections.
|
|
130
|
+
|
|
131
|
+
> **Scoped collections use a placeholder id.** A non-target collection is scoped by its parents' ids, which a real dump captures while running each parent query — but `explain` runs nothing. So for scoped collections, `explain` fills the real foreign-key filter (e.g. `users` → `{ shop_id: { $in: [...] } }`) with a placeholder id rather than real values. The plan still reflects the real dump: `queryPlanner` chooses an index by the queried *field*, not its value, so whether a scoped extraction does an `IXSCAN` or a `COLLSCAN` is reported correctly. Only the bound *values* are fake. (The target collection uses the real `--ids`; reference collections with no `belongs_to` use the real `{}` full scan.)
|
|
132
|
+
|
|
133
|
+
## Generating config from Mongoid models
|
|
134
|
+
|
|
135
|
+
For MongoDB applications backed by [Mongoid](https://www.mongodb.com/docs/mongoid/), a separate rake task introspects Mongoid document models and emits `MongodbCollectionConfig` files (the `fields` / `_id` / `embedded_in` shape described in this document):
|
|
136
|
+
|
|
137
|
+
```bash
|
|
138
|
+
bundle exec rake exwiw:schema:generate_mongoid
|
|
139
|
+
```
|
|
140
|
+
|
|
141
|
+
It is a distinct task and class (`Exwiw::MongoidSchemaGenerator`) from the ActiveRecord generator because the two ORMs expose entirely different metadata. From each model it derives:
|
|
142
|
+
|
|
143
|
+
- the collection name and the `_id` primary key,
|
|
144
|
+
- `fields` from the declared Mongoid fields (referenced `belongs_to` foreign keys such as `shop_id`, and the `created_at` / `updated_at` columns added by `Mongoid::Timestamps`, are ordinary fields — their BSON `ObjectId` / `Date` values serialize as MongoDB Extended JSON at dump time). For an aliased field (`field :ctry, as: :country`), the generator emits the **stored** document key (`ctry`), never the Ruby accessor (`country`), so masking and projection target the key that actually appears in the document, and additionally records the accessor as `mongoid_field_name` on that field so the short key stays understandable (association aliases such as `shop => shop_id` and the built-in `id => _id` are not field renames and are not annotated),
|
|
145
|
+
- `belongs_tos` from referenced `belongs_to` associations (`{ table_name, foreign_key }`). A referenced `belongs_to` declared on an *embedded* document is dropped (cross-collection refs from inside embedded subdocuments are unsupported — see [Embedded documents](#embedded-documents)), but its foreign-key column is still kept as an ordinary field. A `has_and_belongs_to_many` association is also dropped (its foreign keys are stored as an array field, e.g. `tag_ids`, which exwiw cannot follow as a single-valued foreign key), while that `*_ids` array column is kept as an ordinary field,
|
|
146
|
+
- `embedded_in` from `embedded_in` / `embeds_many` / `embeds_one` associations. Each embedded config names its *immediate* parent collection and the document key it lives under (`store_as`, defaulting to the relation name); nested embedding is represented as a chain (`comments` → `embedded_in` `posts`, `posts` → `embedded_in` `users`) rather than a flattened dot-path, matching how the adapter recurses through array and Hash subdocuments. The document key is resolved by locating the parent's `embeds_one` / `embeds_many` that stores this collection. (Mongoid's computed inverse is frequently `nil` when no explicit `inverse_of:` is set, so exwiw matches by the collection the parent's embedding relations store rather than trusting that inverse — this also resolves an STI subclass embedded through a relation declared against its base class.) When the same collection is embedded under several keys in the parent, the path is ambiguous and treated as unrepresentable (see below). A *polymorphic* `embedded_in` (`embedded_in :addressable, polymorphic: true`) has no single embedding parent collection and so cannot be expressed as an `embedded_in` config. A *self-referential / cyclic* embedding (Mongoid's `recursively_embeds_many` / `recursively_embeds_one`) makes a collection both a top-level document and embedded inside documents of its own type; exwiw represents a collection as either top-level or embedded, not both, so it cannot emit an `embedded_in` config that would silently make the collection undumpable. These unrepresentable shapes are handled best-effort by default and abort only in strict mode (see below).
|
|
147
|
+
|
|
148
|
+
Models in an inheritance hierarchy whose subclasses share the base's collection (Mongoid STI, distinguished by the auto-added `_type` discriminator) collapse into a single config: the generator discovers the subclasses via `descendants` (Mongoid registers only the base class in `Mongoid.models`) and unions every class's `fields` and `belongs_tos` into the collection config, so subclass-only fields and associations are not lost.
|
|
149
|
+
|
|
150
|
+
Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, `bulk_insert_chunk_size`, and `query_timeout_ms` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [Output](#output)). Polymorphic `belongs_to` is not yet expanded by this task.
|
|
151
|
+
|
|
152
|
+
By default the task **aborts** when a model uses a construct exwiw cannot represent: a `belongs_to` whose target class can no longer be resolved (a stale relation left behind after its model was removed), or a polymorphic / self-referential-cyclic / ambiguous / unresolvable-parent `embedded_in` (see the cases above).
|
|
153
|
+
|
|
154
|
+
### Honoring an explicit `ignore` (the recommended way to keep these out)
|
|
155
|
+
|
|
156
|
+
When you have reviewed such a construct and decided exwiw should leave it alone, mark it `ignore: true` in its config on disk. The generator **honors an explicit `ignore` and skips re-introspecting it**, so it never aborts the run on something you have already triaged — and your annotation survives regeneration. Two granularities:
|
|
157
|
+
|
|
158
|
+
- A whole **collection** exwiw cannot represent (e.g. a polymorphic / ambiguous `embedded_in`) — mark the collection config `"ignore": true`. To actually dump/mask it later, define its `embedded_in` config by hand (see [Embedded documents](#embedded-documents)).
|
|
159
|
+
- A single **`belongs_to`** that no longer resolves while the rest of its collection is fine (e.g. a stale relation pointing at a removed model) — mark that entry `"ignore": true`, with no `table_name`. The relation is dropped from extraction (`#reject_ignored_members!`) while its foreign-key column stays an ordinary field, and the collection keeps dumping.
|
|
160
|
+
|
|
161
|
+
Record *why* with the optional **`ignore_type`** (a free-form tag exwiw never interprets — e.g. `"need_code_fix"` for an application-side bug, `"unsupported"` for a shape exwiw cannot express) and a **`comment`**. Both are user-owned and preserved across regeneration; the generator never emits `ignore_type` itself.
|
|
162
|
+
|
|
163
|
+
```json
|
|
164
|
+
// orders.json — a stale belongs_to flagged for a code fix; the collection still dumps
|
|
165
|
+
{
|
|
166
|
+
"name": "orders",
|
|
167
|
+
"primary_key": "_id",
|
|
168
|
+
"belongs_to": [
|
|
169
|
+
{ "table_name": "shops", "foreign_key": "shop_id" },
|
|
170
|
+
{
|
|
171
|
+
"foreign_key": "coupon_id",
|
|
172
|
+
"ignore": true,
|
|
173
|
+
"ignore_type": "need_code_fix",
|
|
174
|
+
"comment": "FIXME: belongs_to :coupon -> Coupon does not exist (dead relation)."
|
|
175
|
+
}
|
|
176
|
+
],
|
|
177
|
+
"fields": [ /* ... coupon_id is kept as an ordinary field ... */ ]
|
|
178
|
+
}
|
|
179
|
+
```
|
|
180
|
+
|
|
181
|
+
### First bootstrap pass: `EXWIW_SKIP_UNSUPPORTED=1`
|
|
182
|
+
|
|
183
|
+
For the very first pass against a large app — before any `ignore` annotations exist — set `EXWIW_SKIP_UNSUPPORTED=1` to keep going past *un-annotated* unrepresentable constructs instead of aborting one at a time:
|
|
184
|
+
|
|
185
|
+
```bash
|
|
186
|
+
EXWIW_SKIP_UNSUPPORTED=1 bundle exec rake exwiw:schema:generate_mongoid
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
- An unresolvable `belongs_to` is dropped from the collection's `belongs_tos` (its foreign-key column is still kept as an ordinary field, like the polymorphic / HABTM cases) and a warning naming the relation is printed to stderr.
|
|
190
|
+
- An unrepresentable `embedded_in` collection is emitted as a **top-level** config marked `"ignore": true` with a `comment` recording why, and a warning is printed.
|
|
191
|
+
|
|
192
|
+
Review the stderr warnings, annotate the affected configs (`ignore` / `ignore_type` / `comment`), and subsequent runs complete without the flag because the generator honors those explicit ignores.
|
|
@@ -107,6 +107,9 @@ module Exwiw
|
|
|
107
107
|
case @driver
|
|
108
108
|
when :mysql2
|
|
109
109
|
res = raw.query(sql, cast: false, as: :array)
|
|
110
|
+
# mysql2 returns nil for a statement with no result set (SET, DDL, ...).
|
|
111
|
+
return Result.new([], []) if res.nil?
|
|
112
|
+
|
|
110
113
|
rows = res.to_a.map { |row| row.map { |value| self.class.stringify_value(value) } }
|
|
111
114
|
Result.new(res.fields, rows)
|
|
112
115
|
when :trilogy
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Exwiw
|
|
4
|
+
# Opt-in config for batched extraction (see {BatchedExtraction} and the
|
|
5
|
+
# `batch_scope` section of README.md): `table` is the scoped table whose
|
|
6
|
+
# in-scope primary keys slice this table's extraction, `size` the ids per batch.
|
|
7
|
+
class BatchScope
|
|
8
|
+
include Serdes
|
|
9
|
+
|
|
10
|
+
DEFAULT_SIZE = 1_000
|
|
11
|
+
|
|
12
|
+
attribute :table, String
|
|
13
|
+
attribute :size, optional(Integer), skip_serializing_if_nil: true
|
|
14
|
+
attribute :comment, optional(String), skip_serializing_if_nil: true
|
|
15
|
+
|
|
16
|
+
def batch_size
|
|
17
|
+
size || DEFAULT_SIZE
|
|
18
|
+
end
|
|
19
|
+
end
|
|
20
|
+
end
|
|
@@ -0,0 +1,116 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module Exwiw
|
|
4
|
+
# Extracts a table configured with `batch_scope` as one query per slice of the
|
|
5
|
+
# scope's id set, so each query stays index-driven instead of degrading into a
|
|
6
|
+
# full scan. Rows are the unbatched query's, in batch order; the slices
|
|
7
|
+
# partition the id set, so none is dropped or repeated. See README.md.
|
|
8
|
+
class BatchedExtraction
|
|
9
|
+
include Enumerable
|
|
10
|
+
|
|
11
|
+
attr_reader :terminus
|
|
12
|
+
|
|
13
|
+
def self.build(adapter:, table:, dump_target:, table_by_name:, logger:)
|
|
14
|
+
return nil unless table.respond_to?(:batch_scope) && table.batch_scope
|
|
15
|
+
|
|
16
|
+
new(
|
|
17
|
+
adapter: adapter,
|
|
18
|
+
table: table,
|
|
19
|
+
dump_target: dump_target,
|
|
20
|
+
table_by_name: table_by_name,
|
|
21
|
+
logger: logger,
|
|
22
|
+
)
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def initialize(adapter:, table:, dump_target:, table_by_name:, logger:)
|
|
26
|
+
@adapter = adapter
|
|
27
|
+
@table = table
|
|
28
|
+
@dump_target = dump_target
|
|
29
|
+
@table_by_name = table_by_name
|
|
30
|
+
@logger = logger
|
|
31
|
+
@terminus = QueryAstBuilder
|
|
32
|
+
.new(table.name, table_by_name, dump_target, logger)
|
|
33
|
+
.batch_scope_terminus!
|
|
34
|
+
end
|
|
35
|
+
|
|
36
|
+
def batch_size
|
|
37
|
+
@table.batch_scope.batch_size
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
# The batch table's own extraction query projected to its primary key, so the
|
|
41
|
+
# ids are narrowed by exactly the filter the unbatched query would carry. The
|
|
42
|
+
# key is a plain column so masking configured on it cannot corrupt the ids.
|
|
43
|
+
def key_query_ast
|
|
44
|
+
@key_query_ast ||= begin
|
|
45
|
+
scoped = QueryAstBuilder.run(@terminus.name, @table_by_name, @dump_target, @logger)
|
|
46
|
+
|
|
47
|
+
QueryAst::Select.new.tap do |ast|
|
|
48
|
+
ast.from(scoped.from_table_name)
|
|
49
|
+
ast.select([TableColumn.from_symbol_keys(name: @terminus.primary_key)])
|
|
50
|
+
scoped.join_clauses.each { |join_clause| ast.join(join_clause) }
|
|
51
|
+
scoped.where_clauses.each { |where_clause| ast.where(where_clause) }
|
|
52
|
+
end
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def batch_query_ast(ids)
|
|
57
|
+
QueryAstBuilder.run(@table.name, @table_by_name, @dump_target, @logger, batch_ids: ids)
|
|
58
|
+
end
|
|
59
|
+
|
|
60
|
+
# Resolve the id set and log the plan before extraction starts, so its cost
|
|
61
|
+
# (and an empty id set) is reported where it happens rather than mid-stream.
|
|
62
|
+
def prepare!
|
|
63
|
+
if key_ids.empty?
|
|
64
|
+
@logger.info(" No in-scope #{@terminus.name} ids to batch by; extracting nothing.")
|
|
65
|
+
else
|
|
66
|
+
@logger.info(
|
|
67
|
+
" Extracting in #{batch_count} batch(es) of up to #{batch_size} " \
|
|
68
|
+
"#{@terminus.name}.#{@terminus.primary_key} value(s) (#{key_ids.size} in scope)."
|
|
69
|
+
)
|
|
70
|
+
end
|
|
71
|
+
self
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
# Drained in full (the connection must be free for the batch queries) and
|
|
75
|
+
# sorted here rather than via ORDER BY, which would push a sort onto the
|
|
76
|
+
# source DB. Any total order makes the batches reproducible run to run.
|
|
77
|
+
def key_ids
|
|
78
|
+
@key_ids ||= @adapter.execute(key_query_ast).map(&:first).sort
|
|
79
|
+
end
|
|
80
|
+
|
|
81
|
+
def batch_count
|
|
82
|
+
(key_ids.size + batch_size - 1) / batch_size
|
|
83
|
+
end
|
|
84
|
+
|
|
85
|
+
def each
|
|
86
|
+
return enum_for(:each) unless block_given?
|
|
87
|
+
|
|
88
|
+
extracted = 0
|
|
89
|
+
key_ids.each_slice(batch_size).with_index do |ids, idx|
|
|
90
|
+
rows = 0
|
|
91
|
+
@adapter.execute(batch_query_ast(ids)).each do |row|
|
|
92
|
+
rows += 1
|
|
93
|
+
yield row
|
|
94
|
+
end
|
|
95
|
+
extracted += rows
|
|
96
|
+
@logger.info(" Batch #{idx + 1}/#{batch_count}: #{rows} record(s), #{extracted} so far.")
|
|
97
|
+
end
|
|
98
|
+
|
|
99
|
+
self
|
|
100
|
+
end
|
|
101
|
+
|
|
102
|
+
# Only the COPY output format needs the count up front, and each batch answers
|
|
103
|
+
# it with its own count query, so this stays lazy.
|
|
104
|
+
def size
|
|
105
|
+
@size ||= key_ids.each_slice(batch_size).sum { |ids| @adapter.execute(batch_query_ast(ids)).size }
|
|
106
|
+
end
|
|
107
|
+
alias length size
|
|
108
|
+
|
|
109
|
+
def describe_plan
|
|
110
|
+
"-- batch_scope: extracted in batches of up to #{batch_size} #{@terminus.name}." \
|
|
111
|
+
"#{@terminus.primary_key} value(s). Each batch runs the query above with " \
|
|
112
|
+
"`#{@terminus.name}.#{@terminus.primary_key} IN (<batch ids>)` in place of the scope filter, " \
|
|
113
|
+
"over the ids of:"
|
|
114
|
+
end
|
|
115
|
+
end
|
|
116
|
+
end
|
data/lib/exwiw/explain_runner.rb
CHANGED
|
@@ -68,9 +68,34 @@ module Exwiw
|
|
|
68
68
|
@io.puts "-- EXPLAIN:"
|
|
69
69
|
@io.puts explain_text
|
|
70
70
|
@io.puts
|
|
71
|
+
|
|
72
|
+
explain_batch_scope(adapter, table, table_by_name)
|
|
71
73
|
end
|
|
72
74
|
end
|
|
73
75
|
|
|
76
|
+
# A batched export also runs the query resolving the ids it is sliced by,
|
|
77
|
+
# which the table's own block above does not show. The per-batch query cannot
|
|
78
|
+
# be rendered faithfully here: explain resolves no ids, so #describe_plan
|
|
79
|
+
# explains the substitution instead.
|
|
80
|
+
private def explain_batch_scope(adapter, table, table_by_name)
|
|
81
|
+
batched = BatchedExtraction.build(
|
|
82
|
+
adapter: adapter,
|
|
83
|
+
table: table,
|
|
84
|
+
dump_target: @dump_target,
|
|
85
|
+
table_by_name: table_by_name,
|
|
86
|
+
logger: @logger,
|
|
87
|
+
)
|
|
88
|
+
return if batched.nil?
|
|
89
|
+
|
|
90
|
+
key_query_ast = batched.key_query_ast
|
|
91
|
+
@io.puts batched.describe_plan
|
|
92
|
+
@io.puts adapter.describe_query(key_query_ast)
|
|
93
|
+
@io.puts
|
|
94
|
+
@io.puts "-- EXPLAIN (batch_scope id set):"
|
|
95
|
+
@io.puts adapter.explain(key_query_ast, verbosity: @explain_verbosity)
|
|
96
|
+
@io.puts
|
|
97
|
+
end
|
|
98
|
+
|
|
74
99
|
private def load_table_config(klass)
|
|
75
100
|
Dir[File.join(@schema_dir, "*.json")].map do |file|
|
|
76
101
|
json = JSON.parse(File.read(file))
|
|
@@ -2,8 +2,8 @@
|
|
|
2
2
|
|
|
3
3
|
module Exwiw
|
|
4
4
|
class QueryAstBuilder
|
|
5
|
-
def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
|
|
6
|
-
new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path).run
|
|
5
|
+
def self.run(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [], batch_ids: nil)
|
|
6
|
+
new(table_name, table_by_name, dump_target, logger, allow_reverse: allow_reverse, forward_path: forward_path, batch_ids: batch_ids).run
|
|
7
7
|
end
|
|
8
8
|
|
|
9
9
|
# Scope-column mode classification for a single table. One of
|
|
@@ -26,35 +26,48 @@ module Exwiw
|
|
|
26
26
|
!!(target && target.respond_to?(:scope_column) && target.scope_column)
|
|
27
27
|
end
|
|
28
28
|
|
|
29
|
-
# Strict pre-flight
|
|
30
|
-
#
|
|
31
|
-
#
|
|
29
|
+
# Strict pre-flight: abort if any extractable table cannot be scoped (scope
|
|
30
|
+
# mode), or declares a `batch_scope` its scoping shape cannot be sliced by
|
|
31
|
+
# (both modes) — before any output is written. `tables` is the set of
|
|
32
32
|
# dumpable configs (ignore:true tables are skipped — they are not extracted).
|
|
33
33
|
def self.validate_scope!(tables, table_by_name, dump_target, logger)
|
|
34
|
-
|
|
34
|
+
# Unscopable is reported before a bad batch_scope shape — it is the more
|
|
35
|
+
# fundamental problem.
|
|
36
|
+
if scope_mode?(table_by_name, dump_target)
|
|
37
|
+
unscopable =
|
|
38
|
+
tables.reject(&:ignore).select do |table|
|
|
39
|
+
scope_category(table.name, table_by_name, dump_target, logger) == :unscopable
|
|
40
|
+
end
|
|
35
41
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
42
|
+
if unscopable.any?
|
|
43
|
+
names = unscopable.map(&:name).sort.join(", ")
|
|
44
|
+
raise ArgumentError,
|
|
45
|
+
"scope-column mode: #{unscopable.size} table(s) cannot be scoped: #{names}. " \
|
|
46
|
+
"For each, declare `scope_column: <column>` on the table to filter it directly, " \
|
|
47
|
+
"add a belongs_to path to a table that carries the scope column, mark it " \
|
|
48
|
+
"`scope_exempt: true` to export it in full, or set `ignore: true` to skip it."
|
|
39
49
|
end
|
|
40
|
-
|
|
50
|
+
end
|
|
51
|
+
|
|
52
|
+
tables.reject(&:ignore).each do |table|
|
|
53
|
+
next unless table.respond_to?(:batch_scope) && table.batch_scope
|
|
41
54
|
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
"scope-column mode: #{unscopable.size} table(s) cannot be scoped: #{names}. " \
|
|
45
|
-
"For each, declare `scope_column: <column>` on the table to filter it directly, " \
|
|
46
|
-
"add a belongs_to path to a table that carries the scope column, mark it " \
|
|
47
|
-
"`scope_exempt: true` to export it in full, or set `ignore: true` to skip it."
|
|
55
|
+
new(table.name, table_by_name, dump_target, logger).batch_scope_terminus!
|
|
56
|
+
end
|
|
48
57
|
end
|
|
49
58
|
|
|
50
59
|
attr_reader :table_name, :table_by_name, :dump_target
|
|
51
60
|
|
|
52
|
-
def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [])
|
|
61
|
+
def initialize(table_name, table_by_name, dump_target, logger, allow_reverse: true, forward_path: [], batch_ids: nil)
|
|
53
62
|
@table_name = table_name
|
|
54
63
|
@table_by_name = table_by_name
|
|
55
64
|
@dump_target = dump_target
|
|
56
65
|
@logger = logger
|
|
57
66
|
@allow_reverse = allow_reverse
|
|
67
|
+
# One batch's slice of the batch table's in-scope primary keys, set only by
|
|
68
|
+
# BatchedExtraction. Deliberately not threaded into the recursive builds
|
|
69
|
+
# below, which compile other tables' queries.
|
|
70
|
+
@batch_ids = batch_ids
|
|
58
71
|
# @forward_path is the chain of tables currently being forward-resolved by
|
|
59
72
|
# the "scope via an indirectly-scoped belongs_to parent" rescue
|
|
60
73
|
# (build_belongs_to_scoped_clause). Each forward hop appends the table it is
|
|
@@ -612,6 +625,9 @@ module Exwiw
|
|
|
612
625
|
end
|
|
613
626
|
|
|
614
627
|
private def scope_where_clause(table)
|
|
628
|
+
batch_clause = batch_ids_clause(table)
|
|
629
|
+
return batch_clause if batch_clause
|
|
630
|
+
|
|
615
631
|
Exwiw::QueryAst::WhereClause.new(
|
|
616
632
|
column_name: resolved_scope_column(table),
|
|
617
633
|
operator: :eq,
|
|
@@ -619,6 +635,95 @@ module Exwiw
|
|
|
619
635
|
)
|
|
620
636
|
end
|
|
621
637
|
|
|
638
|
+
# This batch's ids, in place of the batch table's scope filter. nil when the
|
|
639
|
+
# build is not batched or `table` is not the batch table.
|
|
640
|
+
private def batch_ids_clause(table)
|
|
641
|
+
return nil if @batch_ids.nil?
|
|
642
|
+
|
|
643
|
+
batch_scope = table_by_name.fetch(table_name).batch_scope
|
|
644
|
+
return nil if batch_scope.nil? || batch_scope.table != table.name
|
|
645
|
+
|
|
646
|
+
Exwiw::QueryAst::WhereClause.new(
|
|
647
|
+
column_name: table.primary_key,
|
|
648
|
+
operator: :eq,
|
|
649
|
+
value: @batch_ids
|
|
650
|
+
)
|
|
651
|
+
end
|
|
652
|
+
|
|
653
|
+
# The scoped table whose in-scope primary keys slice this table's extraction,
|
|
654
|
+
# or nil when it declares no `batch_scope`. Shapes are accepted only when
|
|
655
|
+
# every row the table keeps is selected through that table's scope filter —
|
|
656
|
+
# otherwise the unconstrained route would re-emit the same rows in every
|
|
657
|
+
# batch — so each rejection below explains itself to the config author.
|
|
658
|
+
def batch_scope_terminus!
|
|
659
|
+
table = table_by_name.fetch(table_name)
|
|
660
|
+
batch_scope = table.batch_scope
|
|
661
|
+
return nil if batch_scope.nil?
|
|
662
|
+
|
|
663
|
+
prefix = "Table '#{table.name}': batch_scope"
|
|
664
|
+
|
|
665
|
+
unless scope_mode?
|
|
666
|
+
raise ArgumentError,
|
|
667
|
+
"#{prefix} is supported in scope-column mode only. In single `--target-table` mode the " \
|
|
668
|
+
"extraction is already anchored on a caller-supplied id list, which can be batched by " \
|
|
669
|
+
"running exwiw once per slice of `--ids`."
|
|
670
|
+
end
|
|
671
|
+
|
|
672
|
+
if scope_exempt?(table)
|
|
673
|
+
raise ArgumentError,
|
|
674
|
+
"#{prefix} cannot apply: the table is exported in full (scope_exempt / rails-managed), " \
|
|
675
|
+
"so there is no scope filter to slice."
|
|
676
|
+
end
|
|
677
|
+
|
|
678
|
+
terminus = table_by_name[batch_scope.table]
|
|
679
|
+
if terminus.nil?
|
|
680
|
+
raise ArgumentError, "#{prefix} names table '#{batch_scope.table}', which is not in the schema."
|
|
681
|
+
end
|
|
682
|
+
if terminus.primary_key.nil?
|
|
683
|
+
raise ArgumentError, "#{prefix} table '#{terminus.name}' has no primary_key to slice the extraction by."
|
|
684
|
+
end
|
|
685
|
+
unless directly_scoped?(terminus)
|
|
686
|
+
raise ArgumentError,
|
|
687
|
+
"#{prefix} table '#{terminus.name}' does not carry the scope column " \
|
|
688
|
+
"(#{resolved_scope_column(terminus) || 'none declared'}), so its in-scope ids cannot be " \
|
|
689
|
+
"resolved. Name the scoped table this table joins up to."
|
|
690
|
+
end
|
|
691
|
+
# A scope_exempt terminus carries the column but its own extraction query is
|
|
692
|
+
# unfiltered, so the batches would substitute every tenant's ids for the
|
|
693
|
+
# scope filter the unbatched join still applies.
|
|
694
|
+
if scope_exempt?(terminus)
|
|
695
|
+
raise ArgumentError,
|
|
696
|
+
"#{prefix} table '#{terminus.name}' is exported in full (scope_exempt / rails-managed), " \
|
|
697
|
+
"so its id set is not scoped and every batch would reach outside the scope."
|
|
698
|
+
end
|
|
699
|
+
|
|
700
|
+
if directly_scoped?(table)
|
|
701
|
+
return terminus if terminus.name == table.name
|
|
702
|
+
|
|
703
|
+
raise ArgumentError,
|
|
704
|
+
"#{prefix} must name '#{table.name}' itself, which carries the scope column and is " \
|
|
705
|
+
"therefore filtered directly rather than through '#{terminus.name}'."
|
|
706
|
+
end
|
|
707
|
+
|
|
708
|
+
arms = scoped_arms(table)
|
|
709
|
+
unless arms.size == 1 && arms.first.path
|
|
710
|
+
raise ArgumentError,
|
|
711
|
+
"#{prefix} needs a single belongs_to join path from '#{table.name}' to the scope, but it is " \
|
|
712
|
+
"scoped another way (polymorphic arms / reverse_scope / referenced-by / the parent cascade), " \
|
|
713
|
+
"or not scoped at all. Those other id sets keep rows by routes a batch of '#{terminus.name}' " \
|
|
714
|
+
"ids does not constrain, so every batch would re-emit them."
|
|
715
|
+
end
|
|
716
|
+
|
|
717
|
+
path = arms.first.path
|
|
718
|
+
unless path.last == terminus.name
|
|
719
|
+
raise ArgumentError,
|
|
720
|
+
"#{prefix} table '#{terminus.name}' is not where '#{table.name}' reaches the scope " \
|
|
721
|
+
"(#{path.join(' -> ')}); name that path's scoped table, '#{path.last}'."
|
|
722
|
+
end
|
|
723
|
+
|
|
724
|
+
terminus
|
|
725
|
+
end
|
|
726
|
+
|
|
622
727
|
# BFS over belongs_tos to the nearest *directly scoped* ancestor. Unlike the
|
|
623
728
|
# target-mode walk, the returned path INCLUDES that ancestor: the scope column
|
|
624
729
|
# lives on the ancestor itself (not on a foreign key of the child), so the
|
data/lib/exwiw/runner.rb
CHANGED
|
@@ -103,8 +103,22 @@ module Exwiw
|
|
|
103
103
|
# both the INSERT and COPY branches below.
|
|
104
104
|
row_transformer = RowTransformer.build(table)
|
|
105
105
|
|
|
106
|
+
# `batch_scope` splits the extraction into one query per slice of the
|
|
107
|
+
# scope's id set, streaming rows like any adapter result. `query_ast`
|
|
108
|
+
# stays the unbatched query — the DELETE file and the error message
|
|
109
|
+
# below describe that one.
|
|
110
|
+
phase = "resolving the batch_scope id set"
|
|
111
|
+
batched = BatchedExtraction.build(
|
|
112
|
+
adapter: adapter,
|
|
113
|
+
table: table,
|
|
114
|
+
dump_target: @dump_target,
|
|
115
|
+
table_by_name: table_by_name,
|
|
116
|
+
logger: @logger,
|
|
117
|
+
)
|
|
118
|
+
batched&.prepare!
|
|
119
|
+
|
|
106
120
|
phase = "executing extraction query"
|
|
107
|
-
results = adapter.execute(query_ast)
|
|
121
|
+
results = batched || adapter.execute(query_ast)
|
|
108
122
|
results = row_transformer.wrap(results) if row_transformer
|
|
109
123
|
insert_idx = (idx + 1).to_s.rjust(3, '0')
|
|
110
124
|
|
data/lib/exwiw/table_config.rb
CHANGED
|
@@ -47,6 +47,10 @@ module Exwiw
|
|
|
47
47
|
# schema generators.
|
|
48
48
|
attribute :reverse_scope, Serdes::OptionalType.new(ReverseScope), skip_serializing_if_nil: true
|
|
49
49
|
|
|
50
|
+
# `batch_scope` splits this table's extraction into one query per slice of the
|
|
51
|
+
# scope's id set (see Exwiw::BatchScope). User-configured, never generated.
|
|
52
|
+
attribute :batch_scope, Serdes::OptionalType.new(BatchScope), skip_serializing_if_nil: true
|
|
53
|
+
|
|
50
54
|
def self.from(hash)
|
|
51
55
|
# Reject unknown keys before deserializing: Serdes silently drops them,
|
|
52
56
|
# which would turn a typo'd or unsupported key into a silent no-op (see
|
|
@@ -76,6 +80,7 @@ module Exwiw
|
|
|
76
80
|
hash.delete("belongs_tos")
|
|
77
81
|
hash.delete("columns")
|
|
78
82
|
hash.delete("reverse_scope")
|
|
83
|
+
hash.delete("batch_scope")
|
|
79
84
|
end
|
|
80
85
|
hash
|
|
81
86
|
end
|
|
@@ -171,6 +176,7 @@ module Exwiw
|
|
|
171
176
|
merged_table.scope_exempt = scope_exempt
|
|
172
177
|
merged_table.scope_column = scope_column
|
|
173
178
|
merged_table.reverse_scope = reverse_scope
|
|
179
|
+
merged_table.batch_scope = batch_scope
|
|
174
180
|
|
|
175
181
|
# Structural facts of each belongs_to come from the freshly generated
|
|
176
182
|
# config, but the user-owned `comment`/`ignore`/`ignore_type`/`references`
|
|
@@ -222,6 +228,10 @@ module Exwiw
|
|
|
222
228
|
raise ArgumentError,
|
|
223
229
|
"Table '#{name}' has type=#{type}; reverse_scope must not be defined."
|
|
224
230
|
end
|
|
231
|
+
if batch_scope
|
|
232
|
+
raise ArgumentError,
|
|
233
|
+
"Table '#{name}' has type=#{type}; batch_scope must not be defined."
|
|
234
|
+
end
|
|
225
235
|
else
|
|
226
236
|
# An ignore:true table is not extracted, so primary_key is not required
|
|
227
237
|
# (e.g. a composite-primary-key table that exwiw does not support).
|
|
@@ -229,6 +239,12 @@ module Exwiw
|
|
|
229
239
|
raise ArgumentError, "Table '#{name}' requires primary_key."
|
|
230
240
|
end
|
|
231
241
|
|
|
242
|
+
if batch_scope && batch_scope.size && batch_scope.size < 1
|
|
243
|
+
raise ArgumentError,
|
|
244
|
+
"Table '#{name}': batch_scope size must be a positive number of ids per batch " \
|
|
245
|
+
"(got #{batch_scope.size})."
|
|
246
|
+
end
|
|
247
|
+
|
|
232
248
|
columns.each { |column| validate_ruby_side_masking!(column) }
|
|
233
249
|
end
|
|
234
250
|
end
|
data/lib/exwiw/version.rb
CHANGED
data/lib/exwiw.rb
CHANGED
|
@@ -12,6 +12,7 @@ require_relative "exwiw/belongs_to"
|
|
|
12
12
|
require_relative "exwiw/fake_data"
|
|
13
13
|
require_relative "exwiw/table_column"
|
|
14
14
|
require_relative "exwiw/reverse_scope"
|
|
15
|
+
require_relative "exwiw/batch_scope"
|
|
15
16
|
require_relative "exwiw/table_config"
|
|
16
17
|
require_relative "exwiw/embedded_in"
|
|
17
18
|
require_relative "exwiw/mongodb_field"
|
|
@@ -32,6 +33,7 @@ require_relative "exwiw/mongo_query"
|
|
|
32
33
|
require_relative "exwiw/query_ast"
|
|
33
34
|
require_relative "exwiw/query_ast_builder"
|
|
34
35
|
require_relative "exwiw/row_transformer"
|
|
36
|
+
require_relative "exwiw/batched_extraction"
|
|
35
37
|
require_relative "exwiw/after_insert_hook"
|
|
36
38
|
require_relative "exwiw/runner"
|
|
37
39
|
require_relative "exwiw/explain_runner"
|
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: exwiw
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.9.
|
|
4
|
+
version: 0.9.19
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Shia
|
|
@@ -38,6 +38,7 @@ files:
|
|
|
38
38
|
- README.md
|
|
39
39
|
- docs/mongodb-dump-parallelism-2x-notes.md
|
|
40
40
|
- docs/mongodb-scoping-fullscan-notes.md
|
|
41
|
+
- docs/mongodb.md
|
|
41
42
|
- docs/optimization-notes.md
|
|
42
43
|
- docs/optimize-mongodb-export-with-native-ext.md
|
|
43
44
|
- docs/plans/2026-05-15-insert-000-schema-file.md
|
|
@@ -63,6 +64,8 @@ files:
|
|
|
63
64
|
- lib/exwiw/adapter/sql_bulk_insert.rb
|
|
64
65
|
- lib/exwiw/adapter/sqlite_adapter.rb
|
|
65
66
|
- lib/exwiw/after_insert_hook.rb
|
|
67
|
+
- lib/exwiw/batch_scope.rb
|
|
68
|
+
- lib/exwiw/batched_extraction.rb
|
|
66
69
|
- lib/exwiw/belongs_to.rb
|
|
67
70
|
- lib/exwiw/cli.rb
|
|
68
71
|
- lib/exwiw/config_file.rb
|