exwiw 0.5.2 → 0.5.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 567683d65df5d9f147ab9415a67baf48a80e21ad32e1ef7635c624dfc3d28c47
-  data.tar.gz: 1513b577f6f2368df60edc45a54c96495ece4f1ee9b453e92adb8991f182fcdf
+  metadata.gz: 4df9132fdf081f28d9d0c53ade6bba8267f0f5ec5c40e19ab42fa45d9cfa5a10
+  data.tar.gz: 4f20a649c126a4617cd5e90fd09eb2eb7f92b068408d5ef9e927eac98a5fadda
 SHA512:
-  metadata.gz: a9680642eb34f99ed3f0c2924154a5171edf541286dfed14befe15b8c029271419a13d3295694847b1143898b0200bf6eb1d6448e6665dbad7bef026e3c3fbbb
-  data.tar.gz: 30e6ef9f988965b85f899fdb0646b6e4e2befd68f95a247a9edfeddb8d3a6088f611f07d5376c7d3b9f4f58f147072e03294a140312dec1622994fb6da175720
+  metadata.gz: 309984a32b8cf593b5499427aa089c1c235332a5ca0cda584a2520d3dd0b073f33839adbe20487bce6bc31c20ba66434ce5e0cb1eab5e53bbdfa72bcf115707f
+  data.tar.gz: de045c9fef6510561b9fd4e713e6a9ba4be7c145f570aaf3da108cd7a4bf3512d0f8757c33e40470bf78b3e7dbebe5244013978c21a974610bca86ceeab54fd9

data/CHANGELOG.md

@@ -2,6 +2,12 @@
 
 ## [Unreleased]
 
+## [0.5.3] - 2026-06-19
+
+### Changed
+
+- **MongoDB: dumps now stream, bounding peak memory regardless of collection size** (default; no flag, byte-identical output). The adapter previously loaded each collection's entire result set into memory (`.to_a`) and built the whole collection's JSONL output as one string, so peak memory scaled with collection size. It now wraps the Mongo cursor in a lazy streaming result and writes output in chunks, so at most one chunk of documents (plus the small FK-propagation key arrays) is resident at a time. On a 20k-document × 30-embed collection this cut peak RSS by hundreds of MB and was also faster (less GC pressure). The per-document Extended-JSON masking was also precompiled per collection config, trimming per-document encoding cost. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the full investigation, and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the proposed native-encoder follow-up.
+
 ## [0.5.2] - 2026-06-18
 
 ### Fixed

data/README.md

@@ -647,6 +647,7 @@ The MongoDB adapter is experimental. To use it:
 - `--ids` values are coerced to the type actually stored in `_id` before filtering: integer-looking ids become `Integer`, 24-char hex ids become `BSON::ObjectId` (Mongoid's default `_id` type — a plain String would never match an ObjectId), and any other string is left as-is.
 - `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
 - `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only**; the SQL adapters use `--ids-column` instead (see below).
+- Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. The dominant remaining cost is encoding each document to MongoDB Extended JSON (pure Ruby); see [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the proposed native-encoder follow-up. Benchmark your own data with `script/bench_mongodb_dump.rb`.
 - Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
   ```bash
   mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl
@@ -664,7 +665,7 @@ The MongoDB adapter is experimental. To use it:
 MongoDB models often store one-to-many relationships as embedded subdocument arrays (e.g. `users` documents with a `posts: [...]` field). To mask fields inside embedded subdocuments, declare a separate config with `embedded_in`:
 
 ```jsonc
-// scenario/users.json — top-level collection
+// e2e/users.json — top-level collection
 {
   "name": "users",
   "primary_key": "_id",
@@ -676,7 +677,7 @@ MongoDB models often store one-to-many relationships as embedded subdocument arr
   ]
 }
 
-// scenario/posts.json — embedded under users.posts
+// e2e/posts.json — embedded under users.posts
 {
   "name": "posts",
   "primary_key": "_id",

data/docs/optimization-notes.md

@@ -0,0 +1,126 @@
+# MongoDB dump performance: investigation notes
+
+This records what was learned while making the MongoDB adapter's dump faster and
+lighter, **what shipped**, and **what was explored and deliberately removed**.
+It exists so the removed work isn't re-discovered from scratch and so the
+trade-offs behind the current design are legible.
+
+The reproducible harness is `script/bench_mongodb_dump.rb` (seeds a synthetic
+large/embed-heavy dataset and measures the dump phases). The correctness anchor
+throughout is `spec/insert_output_snapshot_spec.rb` — a **byte-exact** snapshot
+of the dump output; every change below was required to keep it green.
+
+## The two hotspots
+
+On an embed-heavy benchmark (20k users × 30 embedded posts → ~154 MB JSONL):
+
+1. **Memory.** Two compounding costs. `MongodbAdapter#execute` did `.to_a`,
+   loading the entire result set onto the heap (~600–900 MB / ~9.5M Ruby
+   objects for 20k docs). Separately, with no chunking the Runner built the whole
+   collection's JSONL output as **one giant string** before writing, held
+   simultaneously with the result set.
+2. **CPU.** `doc.as_extended_json(mode: :relaxed)` is ~82% of per-document
+   serialization (~104µs of ~124µs for a 30-post doc). It recursively rebuilds
+   the document into a new intermediate Hash tree, so cost scales with embedding
+   depth/count; `JSON.generate` over that tree is comparatively cheap (~10µs).
+
+## What shipped (default, no flags, byte-identical)
+
+- **Chunked output streaming.** The Runner writes each bulk-insert chunk straight
+  to the file instead of joining the whole table's output into one string.
+  `MongodbAdapter` sets a positive `default_bulk_insert_chunk_size` (1000) so
+  MongoDB output is chunked by default while SQL adapters keep one statement per
+  table. Cut peak RSS ~112 MB and was ~30% faster, byte-identical.
+- **Streaming result set.** `#execute` returns a lazy `StreamingResult` wrapping
+  the Mongo cursor instead of `.to_a`. The Runner pulls documents through
+  `each_slice`, so only one chunk is resident at a time. `#size` is answered with
+  a cheap `count_documents` (index-only) rather than draining the cursor, and the
+  FK-propagation `@state` is captured *as the cursor streams* and published once
+  the pass completes (the Runner always fully consumes a non-empty result, so
+  propagation is unaffected). Cut peak RSS growth ~360 MB and wall time ~40%.
+- **Precompiled masking (`MaskPlan`).** Masking runs over every document **and**
+  every embedded subdocument, so per-config decisions (which fields carry a
+  `replace_with`, how each template splits, where embedded children live) were
+  recomputed many times per document. Compiling a `MaskPlan` once per collection
+  config dropped per-document masking ~17–22% and ~35 allocations/doc, scaling
+  down with embedding count. Byte-identical.
+
+Net default result: memory is bounded by chunk size rather than collection size,
+with a meaningful wall-time improvement and no API/flag surface.
+
+## What was explored and removed
+
+After the memory work, the remaining cost was almost entirely the pure-Ruby
+`as_extended_json`. Threads give **zero** speedup (it holds the GVL), and a
+hand-rolled pure-Ruby fused encoder is **slower** than `as_extended_json +
+JSON.generate` (per-leaf `.to_json` C-call overhead; `JSON.generate` does the
+whole tree in one C pass). `bson` 5.2.0 has no native Extended-JSON serializer to
+borrow (`to_extended_json` is literally `as_extended_json(**opts).to_json`, all
+pure Ruby). That left two levers, both of which were built, measured, and then
+**removed for being disproportionately complex**:
+
+### Fork-parallel serialization (`--parallel-workers=N`)
+
+Forked `N` worker processes to serialize contiguous document slices in parallel,
+parent concatenating parts in order (byte-identical). The *serialization step*
+parallelized ~2–2.5× at 4–8 workers, **but the end-to-end dump speedup was only
+~1.1–1.4×** on embed-heavy data. Reason (Amdahl): ~40% of dump wall time is the
+**serial Mongo cursor BSON→Ruby decode** in the parent, which serialization
+parallelism cannot touch — capping the win — and fork/concat overhead eroded most
+of the rest.
+
+### Cursor-parallel fetch (`--cursor-parallel`)
+
+Went further: split each collection into `N` disjoint `_id` ranges, each fetched
+by a forked worker with its own connection+cursor, so the **decode** was
+parallelized too. Measured byte-identical at ~2.5–5.5× depending on dataset size
+— a real, larger win. But it required a lot of machinery:
+
+- `_id`-range partitioning (an index-only scan + range split) — `MongoIdPartitioner`.
+- A fork orchestrator writing ordered part files + Marshal'd state sidecars — `ForkedPartWriter`.
+- Distributed FK-propagation: each worker captures its range's `@state` slice and
+  the parent merges them in range order — `PropagationCapture`.
+- A per-worker fresh-connection builder (the Mongo driver is not fork-safe).
+- New adapter seams (`write_bulk_insert`, `write_inserts`) and CLI→Runner→Adapter
+  threading for two flags + their validations.
+- A user-visible caveat: per-range cursors must `sort(_id)`, so the output is
+  **ordered by `_id` rather than natural order** — a different byte stream
+  (semantically equivalent re-import), so it could not be the snapshot-tested
+  default.
+
+### Why both were removed
+
+The cursor-parallel win was real but bought with multi-process orchestration,
+IPC, distributed state, a fresh-connection-per-worker requirement, fork
+fallbacks for Windows/JRuby, and a non-default output ordering — a large,
+permanently-maintained surface for a single adapter's export path. The
+maintainer's call was that this is **over-engineered** for the benefit, and that
+the CPU hotspot is better addressed by the lever every earlier iteration kept
+pointing at: a native (C) Extended-JSON encoder. The memory wins above are
+unrelated to the parallelism and were kept.
+
+## Where the speedup goes next
+
+A C extension can collapse `as_extended_json + JSON.generate` into one native
+tree-walk (no intermediate tree, no second pass), as a flag-free, fork-free,
+single-process win — bounded by the same serial-decode ceiling (~2.5×) the
+`--parallel-workers` path hit, since it also doesn't touch the driver's decode.
+The full design (byte-identity strategy, fast-path vs Ruby-delegate types,
+optional-load + pure-Ruby fallback, packaging) is in
+[`optimize-mongodb-export-with-native-ext.md`](./optimize-mongodb-export-with-native-ext.md).
+
+## Methodology notes (for re-running)
+
+- The CPU hotspot reproduces **with no database**: the Mongo driver hands back
+  plain Ruby `Hash` + `BSON::ObjectId`/`Time`, so synthesizing that shape in
+  memory yields the exact `as_extended_json + JSON.generate` cost and runs under
+  the normal sandbox. DB-touching measurement needs live mongo on `localhost:27017`
+  (the dev sandbox blocks it — disable the sandbox for those runs).
+- In-process sequential bench passes accumulate RSS (Ruby reclaims to the OS
+  lazily), which inflates a later serial baseline and overstates a parallel
+  speedup; isolate sections in fresh processes for defensible numbers.
+- Chunk size never changes output **bytes** — the Runner inserts the same `"\n"`
+  between chunks that `to_bulk_insert` inserts between documents — so it is purely
+  a memory/throughput knob, safe against the snapshot guard.
+- Ruby 4.0 removed the `benchmark` stdlib from default gems; use
+  `Process.clock_gettime(Process::CLOCK_MONOTONIC)`.

data/docs/optimize-mongodb-export-with-native-ext.md

@@ -0,0 +1,229 @@
+# Design: optional native (C) extension for the MongoDB Extended-JSON encoder
+
+Status: **proposed / not implemented.** This document captures the design for a
+future change. It is the planned successor to the fork/cursor parallelism that
+was removed (see [`optimization-notes.md`](./optimization-notes.md)).
+
+## Motivation
+
+When the MongoDB adapter dumps embed-heavy documents, the dominant CPU cost is
+turning each decoded Mongo document (a Ruby `Hash` containing `BSON::ObjectId` /
+`Time` / nested `Hash`+`Array`) into one JSONL line of MongoDB **Relaxed
+Extended JSON**. Today that is:
+
+```ruby
+# lib/exwiw/adapter/mongodb_adapter.rb
+JSON.generate(doc.as_extended_json(mode: :relaxed))
+```
+
+`as_extended_json` (in the pure-Ruby `bson` gem) **recursively rebuilds the
+whole document into a new intermediate Hash tree** (`ObjectId -> {"$oid"=>…}`,
+`Time -> {"$date"=>…}`, every subdoc/array re-allocated), and then
+`JSON.generate` walks that tree a *second* time. For a 30-embedded-post doc this
+was measured at ~130µs/doc and is ~82% of per-document serialization cost.
+
+Earlier experiments established the levers:
+
+- Threads give **zero** speedup — `as_extended_json` is pure Ruby and holds the GVL.
+- A pure-Ruby fused single-pass encoder is **slower** (per-leaf `.to_json` C-call
+  overhead beats it; `JSON.generate` does the whole tree in one C pass).
+- Multi-process parallelism worked but was judged over-engineered and removed.
+
+The remaining lever is a **C extension**: one native walk that emits the
+Extended-JSON text directly — no intermediate transformed-Hash tree, no second
+JSON pass.
+
+## Goals / non-goals
+
+- **Goal:** a native encoder that is **byte-for-byte identical** to the current
+  pure-Ruby path, behind an **optional** load with a pure-Ruby fallback so
+  exwiw stays installable as a pure-Ruby gem (JRuby/TruffleRuby, or any host
+  where compilation fails, keep working).
+- **Non-goal:** speeding up the Mongo cursor's BSON→Ruby *decode*. That lives in
+  the `mongo`/`bson` driver and is ~40% of total dump wall time. A serialization
+  C extension is therefore bounded to roughly the same end-to-end ceiling
+  (~2.5×) the removed `--parallel-workers` path had — it does **not** reach the
+  removed cursor-parallel path's 3.4–5.5×. This is an accepted trade for a far
+  simpler, flag-free, fork-free implementation.
+
+## Exact serialization semantics to reproduce (verified against bson 5.2.0)
+
+The byte-exact anchor is `spec/insert_output_snapshot_spec.rb` (committed
+`spec/insert_output_snapshots/mongodb/*.jsonl` fixtures). The C encoder must
+reproduce the following exactly. All rows below were verified empirically and,
+for `Time`, against `bson-5.2.0/lib/bson/time.rb:72-89`.
+
+| Ruby value | Relaxed Extended JSON output | Notes |
+|---|---|---|
+| `BSON::ObjectId` | `{"$oid":"<24 lowercase hex>"}` | hex via `ObjectId#to_s` |
+| `Time` (year 1970..9999, sub-second) | `{"$date":"2021-01-02T03:04:05.678Z"}` | floor to ms, `strftime('%Y-%m-%dT%H:%M:%S.%LZ')` |
+| `Time` (year 1970..9999, whole second) | `{"$date":"2021-01-02T03:04:05Z"}` | **no** fraction when `usec == 0` |
+| `Time` (year <1970 or >9999) | `{"$date":{"$numberLong":"<ms>"}}` | `ms = sec*1000 + usec.divmod(1000).first` |
+| `Integer` (fits int64) | bare `42` / `9000000000` | |
+| `Integer` (outside int64) | **raises `RangeError`** | `"Integer … too big to be represented as a MongoDB integer"` |
+| `Float` | `JSON.generate(float)` form | `1e20 → 1e+20` (**not** `Float#to_s`'s `1.0e+20`); `100.0 → 100.0`; `-0.0 → -0.0` |
+| `String` | JSON string | escape only `\b \t \n \f \r \" \\`; other `<0x20` as lowercase `\u00xx`; `/`, DEL, U+2028/U+2029, non-ASCII left raw |
+| `true` / `false` / `nil` | `true` / `false` / `null` | |
+| `Hash` | `{…}` | **insertion order** preserved; keys are Strings (JSON-escaped) |
+| `Array` | `[…]` | |
+
+`bson/time.rb` boundary (the highest-risk piece), verified verbatim:
+
+```ruby
+def as_extended_json(**options)
+  if options[:mode] == :relaxed && (1970..9999).include?(utc_time.year)
+    if utc_time.usec != 0
+      utc_time = utc_time.floor(3)            # floor to millisecond
+      {'$date' => utc_time.strftime('%Y-%m-%dT%H:%M:%S.%LZ')}
+    else
+      {'$date' => utc_time.strftime('%Y-%m-%dT%H:%M:%SZ')}
+    end
+  else
+    msec = utc_time.usec.divmod(1000).first
+    {'$date' => {'$numberLong' => (sec * 1000 + msec).to_s}}
+  end
+end
+```
+
+## Fast-path vs delegate (the byte-identity strategy)
+
+The encoder splits values into a **native fast path** and a **Ruby delegate**:
+
+- **Native (in C):** `Hash`, `Array`, `String`, `Integer` within int64,
+  `true`/`false`/`nil`, and `BSON::ObjectId`. These are the structural bulk plus
+  the single most common leaf (`_id`), and their formatting is simple and stable.
+- **Delegate to Ruby** — call back into
+  `JSON.generate(value.as_extended_json(mode: :relaxed))` for the individual
+  value and splice the returned fragment into the buffer:
+  - `Float` — `Float#to_s` diverges from `JSON.generate` for scientific notation
+    (`1e20`), so never reformat floats in C.
+  - `Time` — variable fractional digits + the `[1970,9999]` `$numberLong`
+    boundary + ms flooring; too fragile to risk in C v1.
+  - out-of-int64 `Integer` — must surface the identical `RangeError`.
+  - any unrecognized class — `Decimal128`, `BSON::Binary`, `Symbol`, `Regexp`,
+    `Date`, `BSON::Timestamp`, etc.
+
+**Why delegating is provably byte-identical:** `Hash#as_extended_json` and
+`Array#as_extended_json` are *non-transforming structural recursion* — they map
+over children and call `as_extended_json` on each. So the bytes produced for any
+sub-value `v` by `JSON.generate(v.as_extended_json(mode: :relaxed))` are exactly
+the bytes that the whole-document `JSON.generate(doc.as_extended_json(...))`
+would have produced for that position. The native walk can therefore hand any
+value it does not want to format to Ruby and splice the result, with no
+divergence.
+
+`Time` and `Float` are candidates for later promotion into the native path if a
+benchmark shows the per-leaf delegate call is a meaningful fraction of the win
+(timestamp-heavy docs call out to Ruby once per `Time` field).
+
+## C source & buffer design
+
+- `Exwiw::ExtJson.encode_native(doc) -> String` — returns one JSONL line, **no**
+  trailing `\n` (the caller/Runner owns separators).
+- Recursive emitter writing into a single growing buffer (`rb_str_buf_new` +
+  `rb_str_cat`/`rb_str_buf_cat`, or a `malloc` buffer finalized to an
+  `rb_utf8_str_new`). Result string is UTF-8.
+- Type dispatch via `TYPE()` for the immediates/`T_HASH`/`T_ARRAY`/`T_STRING`/
+  `T_FLOAT`/`T_FIXNUM`/`T_BIGNUM`, and a cached `BSON::ObjectId` class reference
+  (`rb_const_get`) compared with `rb_obj_is_kind_of` for ObjectId.
+- `Hash`: `rb_hash_foreach` preserves insertion order; emit `key:value` pairs;
+  keys are Strings run through the same string escaper.
+- Delegate path: a cached `ID` for a Ruby helper (e.g.
+  `Exwiw::ExtJson.encode_fragment(v)`), `rb_funcall`'d, returning the JSON
+  fragment String to splice. The `RangeError` for oversized integers propagates
+  naturally through the delegate.
+- String escaper implemented in C to match the table above (no per-leaf Ruby
+  call for the common String case).
+
+## Packaging, optional load, fallback
+
+- **gemspec:** `spec.extensions = ["ext/exwiw/ext_json/extconf.rb"]`. With
+  `extensions` set, `gem install exwiw` compiles automatically; hosts that can't
+  compile fall back at runtime (below).
+- **`ext/exwiw/ext_json/extconf.rb`:** `require "mkmf"` (stdlib) +
+  `create_makefile("exwiw/ext_json_native")`. The compiled lib is named
+  `ext_json_native` (distinct from the `ext_json.rb` shim) to avoid a
+  `require` self-collision.
+- **`ext/exwiw/ext_json/ext_json.c`:** the emitter; defines
+  `Exwiw::ExtJson.encode_native`.
+- **`lib/exwiw/ext_json.rb`** (the shim, always loaded):
+
+  ```ruby
+  require "json"
+
+  module Exwiw
+    module ExtJson
+      module_function
+
+      # Pure-Ruby fragment encoder used by both the fallback and the native
+      # delegate path. Byte-identical to today's behavior.
+      def encode_fragment(value)
+        JSON.generate(value.respond_to?(:as_extended_json) ? value.as_extended_json(mode: :relaxed) : value)
+      end
+
+      begin
+        require "exwiw/ext_json_native"   # defines Exwiw::ExtJson.encode_native
+        def encode(doc) = encode_native(doc)
+      rescue LoadError
+        def encode(doc) = encode_fragment(doc)   # exact current behavior
+      end
+    end
+  end
+  ```
+
+- **`Rakefile`:** `require "rake/extensiontask"`;
+  `Rake::ExtensionTask.new("ext_json_native") { |e| e.ext_dir = "ext/exwiw/ext_json" }`;
+  make the `spec` task depend on `compile`.
+- **Dev dependency:** add `rake-compiler` (Gemfile / gemspec dev deps). `mkmf`
+  is stdlib, no runtime dep added.
+- **`.gitignore`:** ignore built artifacts (`lib/exwiw/*.bundle`,
+  `lib/exwiw/*.so`, `ext/**/*.o`, `ext/**/Makefile`). Commit only the `ext/`
+  sources; the gemspec ships files via `git ls-files`.
+- **`lib/exwiw.rb`:** add `require_relative "exwiw/ext_json"`.
+
+## Integration point
+
+In `lib/exwiw/adapter/mongodb_adapter.rb`, the per-document serialize step
+becomes (masking still runs in Ruby first; only the encode changes):
+
+```ruby
+def to_bulk_insert(rows, config)
+  plan = mask_plan(config)
+  rows.map do |doc|
+    apply_mask_plan!(doc, plan)
+    Exwiw::ExtJson.encode(doc)   # was: JSON.generate(extended_json(doc))
+  end.join("\n")
+end
+```
+
+The private `#extended_json` helper is removed — its logic (including the
+`respond_to?(:as_extended_json)` guard) moves into `ExtJson.encode_fragment`.
+
+## Test & benchmark strategy
+
+- **`spec/ext_json_spec.rb`** (DB-free; the primary byte-identity guard, runs in
+  normal CI): assert `encode_native(doc) == encode_fragment(doc)` over a fuzz of
+  representative shapes — ObjectId; nested hashes/arrays; `Time` across the year
+  boundary, whole-second (no fraction), and sub-second; strings with control
+  chars / quotes / backslashes / non-ASCII / U+2028; ints, bignums (assert the
+  same `RangeError`), floats (`1e20`, `-0.0`, `100.0`); `nil`; empty
+  hash/array/string. Skip the native half with a clear message when the lib
+  isn't compiled, so the suite still passes on a fallback-only host.
+- **`spec/insert_output_snapshot_spec.rb`** (live mongo on 27017): the byte-exact
+  fixtures must stay green with the native encoder built.
+- **Microbench** (extend `script/bench_mongodb_dump.rb`): native-encode vs
+  Ruby-fallback throughput on DB-free synthesized embed-heavy docs, plus the live
+  path on 20k×30, to quantify the real speedup.
+
+## Risk register
+
+1. **Time formatting** — variable fraction + ms flooring + `$numberLong`
+   boundary. Mitigated by delegating `Time` to Ruby in v1.
+2. **Float formatting** — `Float#to_s` ≠ `JSON.generate`. Mitigated by delegating.
+3. **String escaping** — must match JSON exactly. Implemented in C, fuzz-tested
+   vs the Ruby fallback.
+4. **Hash key order** — preserved via `rb_hash_foreach`.
+5. **Oversized integers** — delegate so the same `RangeError` surfaces.
+6. **Encoding** — emit UTF-8; pass non-ASCII bytes through unescaped (matches JSON).
+7. **Build/portability** — optional load + pure-Ruby fallback keeps non-CRuby and
+   no-compiler installs working.

data/docs/plans/2026-05-15-insert-000-schema-file.md

@@ -117,7 +117,7 @@ Error: `pg_dump` not found in PATH. exwiw needs pg_dump to generate insert-000-s
 | `lib/exwiw/ddl_postprocessor.rb` (新規) | `IF NOT EXISTS` 書き換え / DO ブロックラップ |
 | `lib/exwiw.rb` | 新規ファイルの require |
 | `README.md` | `dump/` の出力に `insert-000-schema.{sql,js}` を追記、import 手順を更新 |
-| `spec/adapter/sqlite3_adapter_spec.rb` | `dump_schema` 統合テスト (`scenario/initdb/init.sqlite3` に対して実行し、出力が `CREATE TABLE IF NOT EXISTS` を含むことを assert) |
+| `spec/adapter/sqlite3_adapter_spec.rb` | `dump_schema` 統合テスト (`e2e/initdb/init.sqlite3` に対して実行し、出力が `CREATE TABLE IF NOT EXISTS` を含むことを assert) |
 | `spec/adapter/mongodb_adapter_spec.rb` | `dump_schema` テスト (db スタブで `listIndexes` を返し、出力 JS を assert) |
 | `spec/runner_spec.rb` | `insert-000-schema.sql` が `output_dir` に書かれることを assert (Sqlite3 経由で実際に流れることを確認) |
 
@@ -134,9 +134,9 @@ Error: `pg_dump` not found in PATH. exwiw needs pg_dump to generate insert-000-s
 2. `bundle exec rspec spec/adapter/mongodb_adapter_spec.rb` — mongo クライアントをスタブして JS 出力に `db.createCollection("users")` と該当 collection の `createIndex(...)` が含まれることを確認。
 
 ### E2E (scenario スクリプト経由)
-3. `scenario/test_with_sqlite3.sh` を実行し、`dump/insert-000-schema.sql` が生成されることと、空 DB に対して `sqlite3 empty.db < dump/insert-000-schema.sql && for f in dump/insert-*.sql; do sqlite3 empty.db < $f; done` が成功することを確認する。
-4. `scenario/test_with_mysql2.sh`, `scenario/test_with_postgresql.sh` も同様に、`mysql empty_db < dump/insert-000-schema.sql` / `psql empty_db -f dump/insert-000-schema.sql` が成功 → 続けて insert ファイル群が流せることを確認。**`mysqldump` / `pg_dump` を docker compose のコンテナ内 (`compose.yml` で起動する DB コンテナ) で実行する必要がある場合は、scenario スクリプトを更新する。**
-5. `scenario/test_with_mongodb.sh` を実行し、`dump/insert-000-schema.js` が出力されることと、空 DB に対して `mongosh "mongodb://localhost/empty_db" < dump/insert-000-schema.js` が成功すること、続いて `mongoimport` で各 jsonl が流せることを確認。
+3. `e2e/test_with_sqlite3.sh` を実行し、`dump/insert-000-schema.sql` が生成されることと、空 DB に対して `sqlite3 empty.db < dump/insert-000-schema.sql && for f in dump/insert-*.sql; do sqlite3 empty.db < $f; done` が成功することを確認する。
+4. `e2e/test_with_mysql2.sh`, `e2e/test_with_postgresql.sh` も同様に、`mysql empty_db < dump/insert-000-schema.sql` / `psql empty_db -f dump/insert-000-schema.sql` が成功 → 続けて insert ファイル群が流せることを確認。**`mysqldump` / `pg_dump` を docker compose のコンテナ内 (`compose.yml` で起動する DB コンテナ) で実行する必要がある場合は、scenario スクリプトを更新する。**
+5. `e2e/test_with_mongodb.sh` を実行し、`dump/insert-000-schema.js` が出力されることと、空 DB に対して `mongosh "mongodb://localhost/empty_db" < dump/insert-000-schema.js` が成功すること、続いて `mongoimport` で各 jsonl が流せることを確認。
 6. **idempotency 確認**: 同じ schema ファイルを 2 回流してもエラーにならないこと (`IF NOT EXISTS` / `DO $$ EXCEPTION WHEN duplicate_object` / `try/catch on createCollection` が効いている)。
 
 ### 手動確認のチェックポイント

data/docs/plans/2026-05-16-mongodb-from-clean-scenario.md

@@ -6,9 +6,9 @@
 `createCollection` / `createIndex` を書き出す実装を既に持っているが、scenario 側で
 これを apply するパスが無く、CI でも検証できていなかった。具体的なギャップ:
 
-1. `scenario/setup_with_mongodb.rb` は seed を `insert_many` で流すだけで、index を一切作っていない
+1. `e2e/setup_with_mongodb.rb` は seed を `insert_many` で流すだけで、index を一切作っていない
 2. その結果 `tmp/mongodb/insert-000-schema.js` は `createCollection` 行のみで `createIndex` が 0 行
-3. `scenario/import_with_mongodb.rb` は `insert-*.jsonl` だけを glob して処理しており、`insert-000-schema.js` を一切実行しない
+3. `e2e/import_with_mongodb.rb` は `insert-*.jsonl` だけを glob して処理しており、`insert-000-schema.js` を一切実行しない
 
 sqlite3 / mysql2 / postgresql で導入済みの「from clean DB から立ち上げる」流れと
 MongoDB の `insert-000-schema.js` が連動していない状態だった (issue #16)。
@@ -25,10 +25,10 @@ MongoDB の `insert-000-schema.js` が連動していない状態だった (issu
 ### scenario 層
 | パス | 変更 |
 |---|---|
-| `scenario/setup_with_mongodb.rb` | seed 流し込みの後に 3 種類の代表的 index を作る (unique `shops.name` / plain `users.email` / 複合 `orders.shop_id+user_id`) |
-| `scenario/import_with_mongodb.rb` | `--no-drop` と `--input-dir DIR` フラグを追加。from-clean は drop すると schema.js が作った index ごと消えてしまうため |
-| `scenario/verify_with_mongodb.rb` | `--with-indexes` で target collection の index を assert (default scenario では import 時に drop されるのでスキップ) |
-| `scenario/test_with_mongodb_from_clean.sh` (新規) | `mongosh dropDatabase` → exwiw 実行 → `mongosh insert-000-schema.js` → `import --no-drop --input-dir tmp/mongodb-clean` → `verify --with-indexes` |
+| `e2e/setup_with_mongodb.rb` | seed 流し込みの後に 3 種類の代表的 index を作る (unique `shops.name` / plain `users.email` / 複合 `orders.shop_id+user_id`) |
+| `e2e/import_with_mongodb.rb` | `--no-drop` と `--input-dir DIR` フラグを追加。from-clean は drop すると schema.js が作った index ごと消えてしまうため |
+| `e2e/verify_with_mongodb.rb` | `--with-indexes` で target collection の index を assert (default scenario では import 時に drop されるのでスキップ) |
+| `e2e/test_with_mongodb_from_clean.sh` (新規) | `mongosh dropDatabase` → exwiw 実行 → `mongosh insert-000-schema.js` → `import --no-drop --input-dir tmp/mongodb-clean` → `verify --with-indexes` |
 | `.github/workflows/scenario.yml` | with_mongodb job に `mongodb-mongosh` install ステップと `test_with_mongodb_from_clean.sh` 実行ステップを追加。apt repo の codename は `jammy` 固定 (ubuntu-latest が noble に上がる前提) |
 
 ### snapshot test 層
@@ -56,8 +56,8 @@ MongoDB の `insert-000-schema.js` が連動していない状態だった (issu
 
 ## Verification
 
-- `bash scenario/test_with_mongodb.sh` 既存 scenario 維持を確認 ✓
-- `bash scenario/test_with_mongodb_from_clean.sh` 新規 scenario 通過を確認
+- `bash e2e/test_with_mongodb.sh` 既存 scenario 維持を確認 ✓
+- `bash e2e/test_with_mongodb_from_clean.sh` 新規 scenario 通過を確認
   (indexes round-trip OK) ✓
 - `bundle exec rspec` 全 153 examples / 0 failures ✓
 - `tmp/mongodb-clean/insert-000-schema.js` を目視で確認:

data/docs/plans/2026-05-22-postgres-copy-mode-scenario-test.md

@@ -9,22 +9,22 @@
 
 しかし **生成された COPY-mode SQL を実際に `psql -f` で取り込めるかを検証する end-to-end テストが存在しない**。ユーザーは COPY モードで invalid な SQL が出ているのではと疑っており、それを実DBに対して検証したい。
 
-既存の INSERT モードは `scenario/test_with_postgresql.sh` が `psql -f` での再取込まで含めて検証している。これに対応する COPY モード版が無い状態。
+既存の INSERT モードは `e2e/test_with_postgresql.sh` が `psql -f` での再取込まで含めて検証している。これに対応する COPY モード版が無い状態。
 
 ゴール: COPY モード出力を実際に psql に食わせる E2E シナリオ + スナップショット回帰テストを追加し、潜在的な invalid SQL を表面化する。
 
 ## 変更ファイル
 
-1. **新規** `scenario/test_with_postgresql_copy.sh` — E2E シェル
+1. **新規** `e2e/test_with_postgresql_copy.sh` — E2E シェル
 2. **修正** `spec/insert_output_snapshot_spec.rb` — COPY 用の SCENARIOS エントリと `snapshot_subdir` 対応
 3. **修正** `.github/workflows/scenario.yml` — `with_postgres` ジョブに新ステップ
 4. **新規** `spec/insert_output_snapshots/postgresql-copy/insert-*.sql` — `UPDATE_SNAPSHOTS=1` で自動生成
 
 ## 詳細
 
-### 1. `scenario/test_with_postgresql_copy.sh`
+### 1. `e2e/test_with_postgresql_copy.sh`
 
-`scenario/test_with_postgresql.sh` を雛形にして以下のみ差し替え:
+`e2e/test_with_postgresql.sh` を雛形にして以下のみ差し替え:
 
 - `FROM_DATABASE_NAME="exwiw_scenario_prod_db_copy"`
 - `TO_DATABASE_NAME="exwiw_scenario_dev_db_copy"`（並列実行されても既存シナリオと衝突しない名前）
@@ -47,7 +47,7 @@
 ```ruby
 {
   adapter: "postgresql",
-  config_dir: "scenario/postgresql-schema",
+  config_dir: "e2e/postgresql-schema",
   output_format: "copy",
   snapshot_subdir: "postgresql-copy",
   connection: { adapter: "postgresql", database_name: "exwiw_test",
@@ -64,7 +64,7 @@
 
 ```yaml
       - name: Run exwiw (copy mode)
-        run: scenario/test_with_postgresql_copy.sh
+        run: e2e/test_with_postgresql_copy.sh
 ```
 
 `postgres:17-alpine` サービスと `postgresql-client-17` インストールは既存ステップで完了済みなので追加不要。
@@ -80,7 +80,7 @@ UPDATE_SNAPSHOTS=1 bundle exec rspec spec/insert_output_snapshot_spec.rb
 ## 検証手順
 
 1. ローカルで `docker compose up -d postgres` を起動
-2. `bash scenario/test_with_postgresql_copy.sh` を実行 — exit 0 ならば COPY モード SQL は psql 経由で valid。non-zero なら invalid SQL が表面化（その時点で原因を特定して別途修正）
+2. `bash e2e/test_with_postgresql_copy.sh` を実行 — exit 0 ならば COPY モード SQL は psql 経由で valid。non-zero なら invalid SQL が表面化（その時点で原因を特定して別途修正）
 3. `UPDATE_SNAPSHOTS=1 bundle exec rspec spec/insert_output_snapshot_spec.rb` でスナップショットを生成
 4. `bundle exec rspec spec/insert_output_snapshot_spec.rb` を `UPDATE_SNAPSHOTS` 無しで再実行し、全シナリオ（sqlite3 / mysql2 / postgresql / postgresql-copy / mongodb）が通ることを確認
 5. CI 上で `with_postgres` ジョブの新ステップ `Run exwiw (copy mode)` が通る（または invalid SQL を検出する）ことを確認

data/docs/plans/2026-05-31-ids-column-for-sql-adapters.md

@@ -81,7 +81,7 @@ indirect / polymorphic を一律に正しく扱える。
   `ids_field` 指定時に対象テーブルの WHERE が主キーではなく当該カラムになることを確認する
   ケースを追加。
 - `explain` サブコマンド（SQL のみ対応）で end-to-end 確認:
-  既存 scenario（例 `scenario/sqlite3-schema`）に対し
+  既存 scenario（例 `e2e/sqlite3-schema`）に対し
   `--target-table=... --ids=... --ids-column=<col>` を渡し、出力 SQL の WHERE が
   `<table>.<col> IN (...)` になることを目視確認。
 - `bundle exec rspec`（全体）でリグレッションが無いこと。

data/docs/plans/2026-06-19-mongodb-export-remove-parallelism-native-ext.md

@@ -0,0 +1,70 @@
+# Plan: Back out MongoDB fork/cursor parallelism → optional Extended-JSON C extension
+
+## Context (why)
+
+This branch (`gnhf/rt-rails-mongodb-dum-ed518c`) shipped a large MongoDB-dump perf effort across 18 iterations. The memory wins (iter 2–5: streaming result set + chunked output + precompiled `MaskPlan`) are clean, byte-identical, no-flag defaults and stay. But the CPU/throughput half (iter 6–18) grew into heavy multi-process machinery: two CLI flags (`--parallel-workers`, `--cursor-parallel`), four components (`ParallelSerializer`, `MongoIdPartitioner`, `PropagationCapture`, `ForkedPartWriter`), adapter `write_bulk_insert`/`write_inserts` seams, CLI→Runner→Adapter threading, fork orchestration, Marshal IPC, `_id`-range partitioning, distributed `@state` merge, and a sorted-output caveat — to buy ~1.1–1.4× (serialize-fork) / ~2.5–5.5× (cursor-parallel).
+
+The maintainer's decision: **this is over-engineered.** Preserve the findings as `docs/optimization-notes.md`, **remove the parallelism**, and address the CPU hotspot with a much simpler lever the earlier iterations kept pointing at — a **C extension** for the dominant `as_extended_json(mode: :relaxed) + JSON.generate` cost.
+
+**Honest scope of the win:** a C extension speeds only the per-document Extended-JSON *serialization* (~82% of per-doc serialize cost). It cannot touch the Mongo cursor's BSON→Ruby *decode* (~40% of total wall time, inside the driver) — that was cursor-parallel's job and is being removed. So end-to-end is bounded ~2.5× (the same Amdahl ceiling `--parallel-workers` hit), not cursor-parallel's 3.4–5.5×. The trade is deliberate: far simpler code + a no-flag, no-fork, single-process CPU win, for a smaller peak speedup. Memory behavior is unchanged (streaming stays the default).
+
+## Part 1 — Remove the fork/cursor parallelism (keep streaming + MaskPlan)
+
+Verified: `git diff b389204..HEAD` on the core lib files is **entirely** parallelism (`b389204` = iter-5 "MaskPlan" commit). So the iter-5 versions are exactly the "keep streaming, drop parallel" baseline.
+
+**Restore to `b389204` (clean streaming baseline):**
+- `lib/exwiw/runner.rb` — back to the inline `each_slice` chunk loop (`file.print(adapter.to_bulk_insert(chunk))`), no `write_inserts`/`parallel_workers`/`cursor_parallel`.
+- `lib/exwiw/adapter.rb` — `Base#initialize(connection_config, logger)` (drop the two kwargs + ivars), `self.build(connection_config, logger)`; drop the `write_bulk_insert`/`write_inserts` seams (they exist only for parallelism). Keep `default_bulk_insert_chunk_size` (streaming).
+- `lib/exwiw/adapter/mongodb_adapter.rb` — iter-5 form: `StreamingResult` without `query`/`keys` readers; `db` with inline client construction (drop `build_client`); no `write_bulk_insert`/`write_inserts`/`cursor_parallel`/`parallel_workers`/partitioner/capture. Keep `StreamingResult`, `MaskPlan`, chunking. (Then patch `serialize_document` in Part 3.)
+- `lib/exwiw/cli.rb` — drop `--parallel-workers`/`--cursor-parallel` flags, ivars, `validate_parallel_workers!`/`validate_cursor_parallel!`, and the Runner kwargs.
+- `lib/exwiw.rb` — drop the four parallel `require_relative` lines (Part 3 adds the `ext_json` require).
+
+**Delete (components + specs + probes):**
+- `lib/exwiw/{parallel_serializer,forked_part_writer,mongo_id_partitioner,propagation_capture}.rb`
+- `spec/{parallel_serializer,forked_part_writer,mongo_id_partitioner,propagation_capture}_spec.rb`
+- `script/bench_mongodb_parallel_probe.rb`, `script/bench_mongodb_cursor_parallel_probe.rb`
+
+**Restore to `b389204`:** `spec/adapter_spec.rb`, `spec/cli_spec.rb`, `spec/adapter/mongodb_adapter_spec.rb`, `script/bench_mongodb_dump.rb` (all post-iter-5 changes there are parallel-only).
+
+**Edit, don't restore:** `README.md`, `CHANGELOG.md` — remove the `--parallel-workers` / `--cursor-parallel` / `EXWIW_MONGODB_*` sections; the `[Unreleased]` entry becomes "streaming/chunked MongoDB dump by default + optional Extended-JSON C extension", pointing to `docs/optimization-notes.md`.
+
+## Part 2 — `docs/optimization-notes.md`
+
+Distill the 18-iteration log (`.gnhf/runs/.../notes.md`) into a durable doc: the two hotspots (result-set memory; `as_extended_json` CPU), what shipped by default (streaming + chunking + `MaskPlan`), and the **explored-and-removed** parallelism — fork serialize (~1.1–1.4×), cursor-parallel (~3.4–5.5× but heavy: IPC, `_id` partitioning, distributed `@state`, sorted output) — with the Amdahl reasoning (serial cursor decode floor) and *why* it was removed in favor of the C extension. This is the knowledge-preservation the maintainer asked for.
+
+## Part 3 — Optional Extended-JSON C extension — DOCUMENT ONLY (not implemented now)
+
+**Revised scope (per maintainer):** Part 3 is NOT implemented in this pass. Instead, write the design below into `docs/optimize-mongodb-export-with-native-ext.md` as a future-work / design doc. Only Part 1 and Part 2 are executed now.
+
+Design to capture: replaces `JSON.generate(doc.as_extended_json(mode: :relaxed))` with one native tree-walk that emits the Relaxed Extended JSON line directly — no intermediate transformed-Hash tree, no second JSON pass.
+
+**New files**
+- `ext/exwiw/ext_json/extconf.rb` — `mkmf` (stdlib), `create_makefile("exwiw/ext_json_native")`.
+- `ext/exwiw/ext_json/ext_json.c` — defines `Exwiw::ExtJson.encode_native(doc) -> String` (one JSONL line, no trailing `\n`). Recursive emitter into a growth buffer.
+- `lib/exwiw/ext_json.rb` — shim: `require "exwiw/ext_json_native"` (distinct name avoids self-collision); on `LoadError`, define a pure-Ruby `encode`. Exposes one stable `Exwiw::ExtJson.encode(doc)`. Fallback is **byte-for-byte today's code**:
+  ```ruby
+  JSON.generate(doc.respond_to?(:as_extended_json) ? doc.as_extended_json(mode: :relaxed) : doc)
+  ```
+
+**Native fast-path vs delegate (byte-identity strategy, from the Plan agent's findings)**
+- Native in C: `Hash` (insertion order via `rb_hash_foreach`; String keys), `Array`, `String` (escape only `\b\t\n\f\r\"\\`; lowercase `\u00xx` for other <0x20; leave `/`, DEL, U+2028/9, non-ASCII raw), `Integer` within int64, `true`/`false`/`nil`, and `BSON::ObjectId` → `{"$oid":"<24 hex>"}` (hex via `to_s`).
+- **Delegate to Ruby** (call back to a fallback helper returning the JSON fragment for that value): `Float` (`Float#to_s` ≠ `JSON.generate` for sci-notation, e.g. `1e20`), `Time` (variable fractional digits + the years-[1970,9999] `$numberLong` boundary — highest risk), out-of-int64 `Integer` (must preserve the existing `RangeError`), and any unrecognized class (Decimal128, Binary, Symbol, Regexp, Date, …). This is provably byte-identical because `Hash/Array#as_extended_json` are non-transforming structural recursion, so `JSON.generate(v.as_extended_json(mode: :relaxed))` on any sub-value matches the whole-tree output. Time/Float are candidates for later native promotion if the benchmark justifies the added risk.
+
+**Packaging / wiring**
+- `exwiw.gemspec` — `spec.extensions = ["ext/exwiw/ext_json/extconf.rb"]` (auto-compiles on `gem install`; fallback covers platforms that can't).
+- `Rakefile` — `Rake::ExtensionTask.new("ext_json_native")`; make `spec` depend on `compile`.
+- Add `rake-compiler` as a dev dep (Gemfile/gemspec). `mkmf` is stdlib.
+- `.gitignore` — ignore built artifacts (`lib/exwiw/*.bundle`, `lib/exwiw/*.so`, `ext/**/*.o`, `ext/**/Makefile`); commit only `ext/` sources (gemspec ships via `git ls-files`).
+- `lib/exwiw.rb` — `require_relative "exwiw/ext_json"`.
+- `lib/exwiw/adapter/mongodb_adapter.rb` — `serialize_document`/`to_bulk_insert` inner call becomes `ExtJson.encode(doc)` (after `apply_mask_plan!`); remove the now-unused private `#extended_json`.
+
+## Verification (Part 1 + Part 2 scope)
+
+- **`bundle exec rspec`** — full suite green after the revert. The parallel specs are deleted; the restored specs match iter-5. (`spec/insert_output_snapshot_spec.rb` and other mongodb-touching specs need live mongo on 27017 → sandbox disabled; the rest run normally.)
+- **`spec/insert_output_snapshot_spec.rb`** (mongodb fixtures, live mongo) — the byte-exact guard; output bytes must be unchanged by the revert (streaming default is byte-identical to iter-5).
+- `git grep -nE 'parallel_workers|cursor_parallel|ParallelSerializer|MongoIdPartitioner|PropagationCapture|ForkedPartWriter'` returns nothing in `lib/`, `spec/`, `exe/`, `README.md` after the revert (confirms full removal).
+- `docs/optimize-mongodb-export-with-native-ext.md` and `docs/optimization-notes.md` exist and read coherently.
+
+## Notes
+- No `git rebase`/`push -f` (history may stay messy; backing out via forward edits + deletions, not history rewrite).
+- After implementation, offer to commit the plan via the remember-plan skill.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -14,6 +14,57 @@ module Exwiw
         Exwiw::MongodbCollectionConfig
       end
 
+      # A lazy, streaming stand-in for the materialized result array #execute
+      # used to return. Wrapping the Mongo cursor (instead of `.to_a`) keeps the
+      # dump's dominant memory cost — the full result set — off the heap: the
+      # Runner pulls documents through `each_slice`, so at most one chunk of
+      # documents (plus the small propagation-key arrays) is resident at a time,
+      # even for large or embed-heavy collections.
+      #
+      # It satisfies the two things the Runner asks of an execute result:
+      #   - #size: the record count, used to skip empty collections and to log.
+      #     Answered with a `count_documents` query (which only walks index
+      #     entries, far cheaper than fetching every document) rather than by
+      #     draining the cursor.
+      #   - #each (via Enumerable / each_slice): a single streaming pass over the
+      #     cursor. While streaming it captures — per propagation key, BEFORE
+      #     handing the document to the caller's masking — the values downstream
+      #     children will `$in`-match against, publishing them into @state once
+      #     the pass completes.
+      #
+      # Contract note: unlike the old `.to_a` execute, which populated @state
+      # eagerly, this defers state capture until the result is consumed. The
+      # Runner always fully consumes a non-empty result before any child
+      # collection is processed, so propagation is unaffected; a caller that only
+      # needs @state must iterate the result (e.g. `.to_a`).
+      class StreamingResult
+        include Enumerable
+
+        def initialize(view:, collection:, keys:, state:)
+          @view = view
+          @collection = collection
+          @keys = keys
+          @state = state
+        end
+
+        def size
+          @size ||= @view.count_documents
+        end
+        alias length size
+
+        def each
+          return enum_for(:each) { size } unless block_given?
+
+          captured = @keys.each_with_object({}) { |key, acc| acc[key] = [] }
+          @view.each do |doc|
+            @keys.each { |key| captured[key] << doc[key] }
+            yield doc
+          end
+          @state[@collection] = captured
+          self
+        end
+      end
+
       def initialize(connection_config, logger)
         super
         @state = {}
@@ -86,22 +137,24 @@ module Exwiw
       def execute(query)
         @logger.debug("  Executing Mongo find on '#{query.collection}': filter=#{query.filter.inspect} projection=#{query.projection.inspect}")
 
-        docs = db[query.collection]
+        view = db[query.collection]
           .find(query.filter)
           .projection(query.projection)
           .comment(query_comment_text("collection=#{query.collection}"))
-          .to_a
 
-        # Stash, per referenced field, the values children will `$in`-match
-        # against. @propagation_keys is set by the build_query call for this same
+        # Per referenced field, the values children will `$in`-match against.
+        # @propagation_keys is set by the build_query call for this same
         # collection; fall back to the primary key if execute is driven without a
         # preceding build_query (e.g. in isolation from a test).
         keys = @propagation_keys || [query.primary_key]
-        @state[query.collection] = keys.each_with_object({}) do |key, acc|
-          acc[key] = docs.map { |doc| doc[key] }
-        end
 
-        docs
+        # Return a streaming view of the result set rather than `.to_a`-ing the
+        # whole collection into memory. The Runner pulls documents through
+        # `each_slice`, so only one chunk's worth is resident at a time even for
+        # large / embed-heavy collections — the dump's dominant memory cost. The
+        # propagation-key values are captured as the cursor streams and published
+        # into @state once the pass completes (see StreamingResult).
+        StreamingResult.new(view: view, collection: query.collection, keys: keys, state: @state)
       end
 
       # NOTE: relies on @embedded_children_by_parent set by a prior build_query
@@ -110,9 +163,9 @@ module Exwiw
       # to_bulk_insert (SQL adapters don't need it). Safe in Runner, fragile in
       # tests — call build_query first.
       def to_bulk_insert(rows, config)
+        plan = mask_plan(config)
         rows.map do |doc|
-          apply_replace_with!(doc, config)
-          apply_embedded_masking!(doc, config)
+          apply_mask_plan!(doc, plan)
           JSON.generate(extended_json(doc))
         end.join("\n")
       end
@@ -135,6 +188,20 @@ module Exwiw
         'jsonl'
       end
 
+      # Bound how many documents are serialized at once when a collection config
+      # carries no explicit bulk_insert_chunk_size. A MongoDB dump is one JSONL
+      # line per document and, without chunking, the Runner would materialize the
+      # entire collection's output as a single giant string while the full
+      # in-memory result set is still alive — doubling peak memory on large or
+      # embed-heavy collections. Chunking lets the Runner stream each slice to the
+      # file and release its serialized string (and the transient extended-JSON
+      # trees) before building the next.
+      DEFAULT_BULK_INSERT_CHUNK_SIZE = 1_000
+
+      def default_bulk_insert_chunk_size
+        DEFAULT_BULK_INSERT_CHUNK_SIZE
+      end
+
       def schema_output_extension
         'js'
       end
@@ -325,41 +392,101 @@ module Exwiw
         parent_fields[reference_field]
       end
 
-      private def apply_replace_with!(doc, config)
-        config.fields.each do |field|
+      # A masking plan compiled once per collection config and reused for every
+      # document of that collection. `masked_fields` is `[field_name,
+      # template_segments]` for each field carrying a `replace_with`;
+      # `embedded` is one EmbeddedMask per embedded child.
+      MaskPlan = Struct.new(:masked_fields, :embedded)
+
+      # A pre-resolved embedded-child mask: the parent path split once into
+      # `prefix` (the containers to descend into) and `last` (the field holding
+      # the subdocument(s)), plus the child's own MaskPlan.
+      EmbeddedMask = Struct.new(:prefix, :last, :plan)
+
+      # Build (or fetch) the cached MaskPlan for `config`. Masking runs over every
+      # document AND every embedded subdocument, so for an embed-heavy collection
+      # the same per-config decisions — which fields carry a `replace_with`, how
+      # each template splits into segments, where the embedded children live —
+      # were previously recomputed tens of times per document. Compiling them once
+      # per config lets #apply_mask_plan! do nothing but the work that actually
+      # varies per document (rendering templates, descending into subdocuments),
+      # so the saved per-subdocument overhead scales down with embedding count.
+      #
+      # Cached by config name: names are unique within a run and the configs do
+      # not mutate mid-dump. Relies on @embedded_children_by_parent, set by the
+      # build_query call that always precedes to_bulk_insert (see #to_bulk_insert).
+      private def mask_plan(config)
+        (@mask_plans ||= {})[config.name] ||= build_mask_plan(config)
+      end
+
+      private def build_mask_plan(config)
+        masked_fields = config.fields.each_with_object([]) do |field, acc|
           next unless field.replace_with
 
-          doc[field.name] = field.replace_with.gsub(/\{([^{}]+)\}/) do
-            ref = Regexp.last_match(1)
-            (doc.key?(ref) ? doc[ref] : nil).to_s
-          end
+          acc << [field.name, compile_template(field.replace_with)]
+        end
+        embedded = embedded_children_of(config).map do |child|
+          *prefix, last = child.embedded_in.path.split(".")
+          EmbeddedMask.new(prefix, last, build_mask_plan(child))
         end
+        MaskPlan.new(masked_fields, embedded)
       end
 
-      private def apply_embedded_masking!(doc, parent_config)
-        embedded_children_of(parent_config).each do |child|
-          walk(doc, child.embedded_in.path) do |subdoc|
-            apply_replace_with!(subdoc, child)
-            apply_embedded_masking!(subdoc, child)
+      # Apply a precompiled MaskPlan to a document in place: render each masked
+      # field, then descend into each embedded child (recursing into its own
+      # plan). Equivalent to the old apply_replace_with! + apply_embedded_masking!
+      # pair, with all per-config lookups hoisted into the plan.
+      private def apply_mask_plan!(doc, plan)
+        plan.masked_fields.each do |name, segments|
+          doc[name] = render_template(segments, doc)
+        end
+        plan.embedded.each do |child|
+          container = child.prefix.reduce(doc) { |acc, seg| acc.is_a?(Hash) ? acc[seg] : nil }
+          next unless container.is_a?(Hash)
+
+          case (value = container[child.last])
+          when Array then value.each { |sub| apply_mask_plan!(sub, child.plan) if sub.is_a?(Hash) }
+          when Hash  then apply_mask_plan!(value, child.plan)
           end
         end
       end
 
-      private def embedded_children_of(parent_config)
-        @embedded_children_by_parent.fetch(parent_config.name, [])
+      PLACEHOLDER_PATTERN = /\{([^{}]+)\}/
+
+      # Split a `replace_with` template into a flat list of segments (called once
+      # per masked field at plan-build time, see #build_mask_plan). A segment is
+      # either a literal String or a 1-element Array `[ref]` marking a `{ref}`
+      # placeholder. #render_template then concatenates them, skipping the regex
+      # scan / block / `Regexp.last_match` a per-document `gsub` would repeat (~2.5x
+      # faster per field). The segment walk reproduces the old gsub byte-for-byte
+      # (missing keys render as "", literals pass through unchanged).
+      private def compile_template(template)
+        segments = []
+        pos = 0
+        while (md = PLACEHOLDER_PATTERN.match(template, pos))
+          segments << template[pos...md.begin(0)] if md.begin(0) > pos
+          segments << [md[1]]
+          pos = md.end(0)
+        end
+        segments << template[pos..] if pos < template.length
+        segments
       end
 
-      private def walk(doc, dotted_path)
-        segments = dotted_path.split(".")
-        *prefix, last = segments
-        container = prefix.reduce(doc) { |acc, seg| acc.is_a?(Hash) ? acc[seg] : nil }
-        return unless container.is_a?(Hash)
-
-        value = container[last]
-        case value
-        when Array then value.each { |sub| yield sub if sub.is_a?(Hash) }
-        when Hash  then yield value
+      private def render_template(segments, doc)
+        out = +''
+        segments.each do |seg|
+          if seg.is_a?(Array)
+            ref = seg[0]
+            out << (doc.key?(ref) ? doc[ref] : nil).to_s
+          else
+            out << seg
+          end
         end
+        out
+      end
+
+      private def embedded_children_of(parent_config)
+        @embedded_children_by_parent.fetch(parent_config.name, [])
       end
 
       private def extended_json(doc)

data/lib/exwiw/adapter.rb

@@ -113,6 +113,16 @@ module Exwiw
         raise NotImplementedError, "COPY format is not supported by #{self.class.name}"
       end
 
+      # Default bulk-insert chunk size when a table config does not set one.
+      # The Runner streams each chunk straight to the output file, so a non-nil
+      # value here bounds how much serialized output (and how many transient
+      # intermediate objects) live in memory at once. SQL adapters keep nil
+      # (one statement per table, as before); adapters whose output is large
+      # and built per-row (e.g. MongoDB JSONL) override with a positive value.
+      def default_bulk_insert_chunk_size
+        nil
+      end
+
       # Run the database-specific EXPLAIN for the given query and return the
       # output as a single string for `explain` subcommand to print.
       # SQL adapters override; MongodbAdapter currently raises.

data/lib/exwiw/runner.rb

@@ -97,18 +97,36 @@ module Exwiw
           else
             phase = "generating INSERT statement"
             @logger.debug("  Generate INSERT statement...")
-            chunk_size = table.bulk_insert_chunk_size
-            chunks = chunk_size ? results.each_slice(chunk_size).to_a : [results]
-            insert_sql = chunks.map { |chunk_rows| adapter.to_bulk_insert(chunk_rows, table) }.join("\n")
-
-            @logger.info("  Generated INSERT statement for #{record_num} records (#{chunks.size} statement(s)).")
+            # Stream each chunk straight to the file instead of building the whole
+            # table's INSERT/JSONL output as one string first. This keeps only a
+            # single chunk's serialized text (and its transient intermediate
+            # objects) in memory at a time — important for large MongoDB
+            # collections, whose one-giant-chunk JSONL would otherwise be held in
+            # full alongside the already-large in-memory result set.
+            #
+            # The chunk size falls back to the adapter's default when the table
+            # config does not set one (SQL adapters: nil -> one statement, as
+            # before; MongoDB: a positive default so the output is chunked). The
+            # bytes written are identical to joining the chunks with "\n" and
+            # appending a trailing newline, matching the previous `file.puts`.
+            chunk_size = table.bulk_insert_chunk_size || adapter.default_bulk_insert_chunk_size
+            chunks = chunk_size ? results.each_slice(chunk_size) : [results]
+
+            statement_count = 0
             File.open(File.join(@output_dir, "insert-#{insert_idx}-#{table_name}.#{adapter.output_extension}"), 'w') do |file|
               pre = adapter.pre_insert_sql(table)
               file.puts(pre) if pre
-              file.puts(insert_sql)
+              chunks.each do |chunk_rows|
+                file.print("\n") if statement_count.positive?
+                file.print(adapter.to_bulk_insert(chunk_rows, table))
+                statement_count += 1
+              end
+              file.print("\n")
               post = adapter.post_insert_sql(table)
               file.puts(post) if post
             end
+
+            @logger.info("  Generated INSERT statement for #{record_num} records (#{statement_count} statement(s)).")
           end
 
           if adapter.supports_bulk_delete? && !@insert_only && !(table.respond_to?(:rails_managed?) && table.rails_managed?)

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.5.2"
+  VERSION = "0.5.3"
 end

data/mise.toml

@@ -1,6 +1,6 @@
 [env]
-# Prepend scenario/bin so `pg_dump` resolves to the wrapper that delegates to
+# Prepend e2e/bin so `pg_dump` resolves to the wrapper that delegates to
 # the postgres container (compose.yml). exwiw's PostgreSQL adapter shells out
 # to pg_dump, which requires a server/client major-version match — the dev DB
 # is postgres:17 while host clients are often older (e.g. Homebrew pg14).
-_.path = ["./scenario/bin"]
+_.path = ["./e2e/bin"]

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.5.2
+  version: 0.5.3
 platform: ruby
 authors:
 - Shia
@@ -35,12 +35,15 @@ files:
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
+- docs/optimization-notes.md
+- docs/optimize-mongodb-export-with-native-ext.md
 - docs/plans/2026-05-15-insert-000-schema-file.md
 - docs/plans/2026-05-16-mongodb-from-clean-scenario.md
 - docs/plans/2026-05-22-after-insert-hook.md
 - docs/plans/2026-05-22-postgres-copy-mode-scenario-test.md
 - docs/plans/2026-05-29-rails-managed-tables.md
 - docs/plans/2026-05-31-ids-column-for-sql-adapters.md
+- docs/plans/2026-06-19-mongodb-export-remove-parallelism-native-ext.md
 - exe/exwiw
 - lib/exwiw.rb
 - lib/exwiw/adapter.rb