exwiw 0.5.3 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4df9132fdf081f28d9d0c53ade6bba8267f0f5ec5c40e19ab42fa45d9cfa5a10
-  data.tar.gz: 4f20a649c126a4617cd5e90fd09eb2eb7f92b068408d5ef9e927eac98a5fadda
+  metadata.gz: b66bdb8ccf3d2043c0a903c70071e0f070b41b9fd89dbc09e3c5385c9b4916c1
+  data.tar.gz: cf1224f47635113127f2e1fcf38f161c539f7b68f0bb8c85ba0c428fa6a202d1
 SHA512:
-  metadata.gz: 309984a32b8cf593b5499427aa089c1c235332a5ca0cda584a2520d3dd0b073f33839adbe20487bce6bc31c20ba66434ce5e0cb1eab5e53bbdfa72bcf115707f
-  data.tar.gz: de045c9fef6510561b9fd4e713e6a9ba4be7c145f570aaf3da108cd7a4bf3512d0f8757c33e40470bf78b3e7dbebe5244013978c21a974610bca86ceeab54fd9
+  metadata.gz: 41a4ef3c8a6da41ccbb92d7e42a519c5260a1bbcf53ce9cdcfe3a18dbca55d509c7ca1492e726bdb82bfc8c101a0a6bcee1dec1c051ceae44910009801d4b64d
+  data.tar.gz: ce5cdc2e96fcacbfc2a7c3ca23c72a22a1c3e765da2e69ad21fd9ebd87bbfdc3619e54a0c6eb5ae94321e4dc1b8651ca73a965cf33509ecafba9349b8a894883

data/CHANGELOG.md

@@ -2,6 +2,15 @@
 
 ## [Unreleased]
 
+## [0.6.1] - 2026-06-20
+
+## [0.6.0] - 2026-06-20
+
+### Added
+
+- Optimize memory usage https://github.com/heyinc/exwiw/pull/118 
+- **MongoDB: optional native (C) encoder for the Extended-JSON dump path** (no flag, byte-identical output, pure-Ruby fallback). Encoding each document to MongoDB Relaxed Extended JSON — previously `JSON.generate(doc.as_extended_json(mode: :relaxed))`, which rebuilds the whole document into an intermediate transformed Hash tree and then walks it again — was the dominant per-document CPU cost (~82% of serialization on embed-heavy data). A new C extension (`ext/exwiw/ext_json/`) emits the JSONL line in a single native tree-walk. It formats the structural bulk plus the leaves that dominate a dumped document — `Hash`, `Array`, `String`, fixnum `Integer`, `true`/`false`/`nil`, `BSON::ObjectId` (`_id`), and in-range `Time` (the Mongoid `created_at`/`updated_at` timestamps) — and delegates everything else (`Float`, out-of-int64 `Integer`, out-of-range `Time`, `Symbol`, `Decimal128`, …) back to the exact pure-Ruby path, so the output is provably byte-for-byte identical. On a 30-embedded-post timestamp-heavy document this serializes ~2.8× faster. With `gem install exwiw` the extension compiles automatically; hosts that cannot compile (JRuby/TruffleRuby, no toolchain) fall back to the pure-Ruby encoder, so exwiw stays installable as a pure-Ruby gem. See [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md).
+
 ## [0.5.3] - 2026-06-19
 
 ### Changed

data/README.md

@@ -647,7 +647,7 @@ The MongoDB adapter is experimental. To use it:
 - `--ids` values are coerced to the type actually stored in `_id` before filtering: integer-looking ids become `Integer`, 24-char hex ids become `BSON::ObjectId` (Mongoid's default `_id` type — a plain String would never match an ObjectId), and any other string is left as-is.
 - `--target-collection=COLLECTION` is a mongodb-only alias of `--target-table` (use whichever reads better for MongoDB). Specifying both, or using `--target-collection` with a non-mongodb adapter, is an error.
 - `--ids-field=FIELD` matches `--ids` against `FIELD` on the target collection instead of its primary key (e.g. `--target-collection=users --ids=a@example.com --ids-field=email`). Downstream foreign-key propagation still keys off the primary key, so only the target collection's filter changes. Unlike the primary-key path, the supplied ids are **not** type-coerced (the stored type of a custom field is unknown), so pass values matching the field's actual type. This flag is **mongodb-only**; the SQL adapters use `--ids-column` instead (see below).
-- Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. The dominant remaining cost is encoding each document to MongoDB Extended JSON (pure Ruby); see [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the proposed native-encoder follow-up. Benchmark your own data with `script/bench_mongodb_dump.rb`.
+- Large or embedded-document-heavy dumps are streamed automatically: the adapter reads the collection through a lazy cursor (not `.to_a`) and writes JSONL in chunks, so peak memory is bounded by the chunk size rather than the collection size — no flag to set. Encoding each document to MongoDB Extended JSON is accelerated by an **optional native (C) extension** that compiles automatically on `gem install`; where it cannot compile, exwiw falls back to a byte-identical pure-Ruby encoder. See [`docs/optimization-notes.md`](docs/optimization-notes.md) for the performance investigation and [`docs/optimize-mongodb-export-with-native-ext.md`](docs/optimize-mongodb-export-with-native-ext.md) for the native encoder's design. Benchmark your own data with `script/bench_mongodb_dump.rb`.
 - Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
   ```bash
   mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl

data/docs/mongodb-scoping-fullscan-notes.md

@@ -0,0 +1,145 @@
+# MongoDB scoping full-scan diagnosis (nullable-FK belongs_to)
+
+Why a related collection's dump can be "especially slow" — dumping far more
+records than the scope implies and scanning the whole collection — when running
+an exwiw config against a MongoDB backup. The headline finding: the slowness is a
+**symptom of a scoping bug**, not of serialization/decode cost.
+
+## Reproduction setup
+
+- Backup: serve a raw WiredTiger dbpath with a standalone `mongod` (the same
+  `mongo:7` image the repo's `compose.yml` uses) on a spare port:
+
+  ```
+  docker run -d --name exwiw-restore-mongo --user 0:0 --entrypoint mongod \
+    -p 27018:27017 -v "<backup-dbpath>:/data/db" \
+    mongo:7 --dbpath /data/db --bind_ip_all
+  ```
+
+  Notes: run as root (`--user 0:0`) so mongod can read a backup's `0600` files;
+  bypass the image entrypoint (`--entrypoint mongod`) so it does not `gosu`-drop
+  back to the `mongodb` user. A backup carrying a `WiredTiger.backup` marker runs
+  recovery on the first start and **writes into the backup dir** (expected for a
+  restore). Starting a standalone from a replica-set backup yields a harmless
+  `system.replset` warning. Local DB connections may be blocked by a dev sandbox —
+  run measurement commands with the sandbox disabled.
+
+- Run: `bundle exec exwiw export --config <app>/exwiw/exwiw.yml
+  --adapter=mongodb --host=localhost --port=27018 --database=<database>
+  --ids=<target-id> --output-dir=… --log-level=debug`.
+  (The optional-argument CLI flags — `--ids`, `--output-dir` — must use `=`; the
+  space form passes `nil` and crashes in the option callback.)
+
+## Baseline measurement (worked example, warm cache)
+
+Full run ≈ **69s** over a few hundred non-empty collections. Per-collection wall
+time (gap between consecutive `Processing table` log markers) was dominated by a
+single collection:
+
+| collection | time | records |
+|---|---|---|
+| **items** | **41.0s (59% of the whole run)** | 18,739 |
+| (other collections) | ~5s and below each | — |
+
+A "full run ≈ 10 min" figure is the **cold-cache** version (first read pages the
+`items` data off the backup over the bind mount). The *relative* shape (items
+dominates) is the same warm or cold.
+
+## Root cause: nullable belongs_to FK used as a hard `$in` AND constraint
+
+The scope is tiny: one parent entity (`<target-id>`) → **1 store** (linked by
+`business_entity_id` = the entity's **`uuid`**, correctly configured with
+`references: uuid`) → **127 items** (`{store_id: <that store>}`, indexed, ~2ms).
+
+But the run dumped **18,739** items, not 127, and scanned the whole collection.
+Why:
+
+1. `MongodbAdapter#related_collection_filter` ANDs **every** belongs_to whose
+   parent produced ids. The `stores` filter became:
+
+   ```
+   { user_id:             {$in: [580 ids]},
+     deleted_user_id:     {$in: [96 ids]},     # nullable FK
+     business_entity_id:  {$in: ["<target-id>"]} }
+   ```
+
+   The one matching store has **`deleted_user_id` absent (null)**, so it can never
+   satisfy `deleted_user_id ∈ {96 ids}`. The AND yields **0 stores** → `stores`
+   logs "No records matched. skip this table."
+
+2. With `stores` empty, `@state["stores"]` carries no ids, so when `items` is
+   built its `store_id` belongs_to contributes nothing. The remaining belongs_tos
+   are to **reference/master data that is dumped in full** —
+   `large_categories` and `medium_categories`. So the items filter degenerated to:
+
+   ```
+   { large_category_id:   {$in: [98 ids]},
+     medium_category_id:  {$in: [846 ids]} }
+   ```
+
+3. `items` has **no index on `large_category_id` / `medium_category_id`** (it does
+   have `store_id_1`). So this filter forces a **full COLLSCAN of all 2.43M items**
+   — and the Runner scans it **twice**: once for `StreamingResult#size`
+   (`count_documents`) and again for the fetch.
+
+Isolated phase breakdown of the degenerate items query (warm): `count_documents`
+3.58s + fetch/decode 1.47s + serialize 0.26s. Serialization (the native
+`Exwiw::ExtJson` C ext) is **not** the bottleneck — the COLLSCAN is, and it is far
+worse cold.
+
+The same nullable-FK problem applies one level down: the store's 127 items
+themselves have `*_category_id` = null, so even `{store_id, large_category,
+medium_category}` ANDed returns **0**. The only filter that yields the correct 127
+is `{store_id: {$in: [store]}}` **alone**.
+
+## Implemented fix: genuine-anchor scoping (MongodbAdapter#related_collection_filter)
+
+Scope flows from the dump target along belongs_to edges. The fix classifies each
+belongs_to parent of a non-target collection by whether it is **genuinely scoped**
+— reachable back to the dump target through belongs_to chains
+(`#genuine_scope_set`, a fixpoint over the configs) — and applies the constraint
+accordingly:
+
+- **Anchor (strict).** Among the genuine parents, the most selective one (fewest
+  captured ids) is applied strictly (`{fk: {$in: [...]}}`). It carries the real
+  narrowing and, being strict, bounds the result to a small set — which keeps both
+  this query and the `$in` sets it feeds downstream from ballooning.
+- **Other genuine parents (null-aware).** `{fk: {$in: [nil, ...]}}` (Mongo's
+  `$in: [nil]` matches both explicit nulls and missing fields), so a row whose
+  nullable refinement FK is null is not excluded by it.
+- **Reference parents (dropped).** A parent NOT reachable to the dump target is
+  reference/master data dumped in full; its id set is "all/most of a table" and is
+  not a real scope, so when a genuine anchor exists it is dropped entirely.
+- **No genuine parent:** fall back to the historical strict-AND of whatever
+  constraints exist (preserves prior behaviour for unreachable collections).
+
+For this extraction: `stores` → `{business_entity_id ∈ {<target-id>}}` (anchor;
+`user_id`/`deleted_user_id` → reference leaks, dropped) → **1 store**; `items` →
+`{store_id ∈ {store}}` strict anchor with the nullable refinement FKs null-aware
+and the `*_category` references dropped → **127** via the `store_id_1` index.
+
+### Measured result (warm, same cache)
+
+Full run **58.8s → 11.0s ≈ 5.4×**; `items` 41s double COLLSCAN → ~11ms indexed
+(≈3700×). Correctness also fixed: `stores` 0→1, `items` 18,739 (leaked COLLSCAN)
+→127. Byte-identical existing snapshots (the seed graph is fully genuine and has no
+null FKs, so anchor-strict + null-aware ≡ the prior strict-AND).
+
+### Approaches considered and rejected
+
+- **Unconditional null-aware on every belongs_to** (the original iter-1 direction):
+  catastrophic. A collection that belongs_to only reference data dumped in full
+  becomes ~the whole table once null-aware; the resulting child `$in` then exceeds
+  Mongo's **48 MB max message size** and the run crashes. Null-awareness must NOT
+  be applied to a collection's only/anchor scope.
+- **Null-aware on all genuine parents (no anchor distinction):** makes the genuine
+  *anchor* itself null-aware too — `stores` then matched every store with a null
+  `business_entity_id` (a not-fully-backfilled column) → hundreds of thousands of
+  stores → a ~39 MB child filter on a downstream collection (**MaxBSONSize**).
+  Hence the anchor stays strict.
+- **Scope by the single most-selective genuine parent alone (drop other genuine):**
+  fast and correct here, but drops legitimate AND-narrowing for multi-parent
+  collections (e.g. `order_items` ∈ orders AND products) and moves seed snapshots.
+- Pure-performance tweaks that keep the (incorrect) 18,739-row output —
+  `--cursor-parallel` (changes row order, treats the symptom) or skipping the
+  redundant `count_documents` scan (~½ only) — were rejected as the primary fix.

data/docs/optimize-mongodb-export-with-native-ext.md

@@ -1,8 +1,11 @@
 # Design: optional native (C) extension for the MongoDB Extended-JSON encoder
 
-Status: **proposed / not implemented.** This document captures the design for a
-future change. It is the planned successor to the fork/cursor parallelism that
-was removed (see [`optimization-notes.md`](./optimization-notes.md)).
+Status: **implemented.** This document captured the design; it now describes the
+shipped encoder. Source: `ext/exwiw/ext_json/ext_json.c` (native emitter) and
+`lib/exwiw/ext_json.rb` (the optional-load shim + pure-Ruby fallback); the
+byte-identity guard is `spec/ext_json_spec.rb`. It is the successor to the
+fork/cursor parallelism that was removed (see
+[`optimization-notes.md`](./optimization-notes.md)).
 
 ## Motivation
 
@@ -90,15 +93,23 @@ end
 The encoder splits values into a **native fast path** and a **Ruby delegate**:
 
 - **Native (in C):** `Hash`, `Array`, `String`, `Integer` within int64,
-  `true`/`false`/`nil`, and `BSON::ObjectId`. These are the structural bulk plus
-  the single most common leaf (`_id`), and their formatting is simple and stable.
+  `true`/`false`/`nil`, `BSON::ObjectId`, and **in-range `Time`** (years
+  1970..9999). These are the structural bulk plus the two most common leaves in
+  a dumped document — `_id` and the Mongoid `created_at`/`updated_at` timestamps.
+  The in-range Time path resolves the absolute instant with `rb_time_timespec`
+  (epoch seconds + nanoseconds, no `rb_funcall`), formats with `gmtime_r` +
+  `snprintf`, and reproduces bson's rule exactly: a `.mmm` fraction iff
+  `nsec >= 1000` (i.e. `usec != 0`), with the millisecond floored to
+  `nsec / 1e6`. The in-range window is the half-open epoch-second range
+  `[0, 253402300800)`.
 - **Delegate to Ruby** — call back into
   `JSON.generate(value.as_extended_json(mode: :relaxed))` for the individual
   value and splice the returned fragment into the buffer:
   - `Float` — `Float#to_s` diverges from `JSON.generate` for scientific notation
     (`1e20`), so never reformat floats in C.
-  - `Time` — variable fractional digits + the `[1970,9999]` `$numberLong`
-    boundary + ms flooring; too fragile to risk in C v1.
+  - **out-of-range `Time`** (year < 1970 or > 9999) — its `$numberLong` form
+    involves negative-epoch arithmetic, is vanishingly rare in dumped data, and
+    is left to Ruby. The in-range ISO branch is handled natively (above).
   - out-of-int64 `Integer` — must surface the identical `RangeError`.
   - any unrecognized class — `Decimal128`, `BSON::Binary`, `Symbol`, `Regexp`,
     `Date`, `BSON::Timestamp`, etc.
@@ -112,9 +123,14 @@ would have produced for that position. The native walk can therefore hand any
 value it does not want to format to Ruby and splice the result, with no
 divergence.
 
-`Time` and `Float` are candidates for later promotion into the native path if a
-benchmark shows the per-leaf delegate call is a meaningful fraction of the win
-(timestamp-heavy docs call out to Ruby once per `Time` field).
+`Time` was promoted into the native path because the benchmark showed it was
+decisive: with `Time` delegated, a 30-embedded-post timestamp-heavy document
+(32 `Time` fields) sped up only ~1.03× — the per-`Time` `rb_funcall` +
+`as_extended_json` Hash allocation + second `JSON.generate` pass erased the win.
+Formatting in-range `Time` natively brings the same document to ~2.8× (the
+serialization-step ceiling). `Float` remains delegated: matching
+`JSON.generate`'s shortest-round-trip float formatting in C (not `Float#to_s`)
+is not worth the risk for the few floats a typical document carries.
 
 ## C source & buffer design
 
@@ -218,7 +234,11 @@ The private `#extended_json` helper is removed — its logic (including the
 ## Risk register
 
 1. **Time formatting** — variable fraction + ms flooring + `$numberLong`
-   boundary. Mitigated by delegating `Time` to Ruby in v1.
+   boundary. In-range years (1970..9999) are formatted natively via
+   `rb_time_timespec` + `gmtime_r`; the rare out-of-range `$numberLong` form is
+   delegated. Mitigated by a dense byte-identity fuzz over the whole in-range
+   epoch span with mixed nanosecond precision, plus the boundary/sub-ms edges in
+   `spec/ext_json_spec.rb`.
 2. **Float formatting** — `Float#to_s` ≠ `JSON.generate`. Mitigated by delegating.
 3. **String escaping** — must match JSON exactly. Implemented in C, fuzz-tested
    vs the Ruby fallback.

data/docs/sql-dump-optimization-notes.md

@@ -0,0 +1,278 @@
+# SQL dump performance: investigation notes
+
+Companion to [`optimization-notes.md`](./optimization-notes.md) (which covers the
+MongoDB adapter). This records the speed/memory bottlenecks of the **SQL**
+adapters' dump path (mysql / postgresql / sqlite), measured against a baseline,
+so a future iteration can address them. **Nothing is fixed yet** — this is the
+measurement + bottleneck-analysis step.
+
+The reproducible harness is `script/bench_sql_dump.rb`. It seeds a synthetic
+table and measures the two Runner phases per table; it also measures the
+serialization step with no DB at all. The correctness anchor for any future fix
+is `spec/insert_output_snapshot_spec.rb` — the **byte-exact** snapshot of dump
+output.
+
+> **Status:** **both hotspots are fixed for all three SQL adapters.** Hotspot #2
+> (the whole-table INSERT string) — see
+> [Resolution #2](#resolution-hotspot-2-streamed-single-insert). Hotspot #1
+> (full result-set materialization in `execute`) — postgresql, mysql, and sqlite
+> all stream the fetch now; see
+> [Resolution #1](#resolution-hotspot-1-streaming-fetch-postgresql--mysql).
+
+## The two hotspots (same shape as MongoDB had pre-optimization)
+
+The Runner drives, per table:
+
+1. **execute** — the adapter materializes the **entire** result set into a Ruby
+   array-of-arrays before anything is written:
+   - postgresql: `connection.exec(sql).values`
+   - mysql: `res.to_a.map { |row| row.map { stringify } }` (also re-allocates
+     every value as a normalized String)
+   - sqlite: `connection.execute(sql)`
+
+   Memory here is proportional to the **table size**, independent of any chunking
+   downstream.
+
+2. **to_bulk_insert** — SQL adapters set **no** `default_bulk_insert_chunk_size`
+   (it is `nil`), so the Runner treats the whole table as one chunk and
+   `to_bulk_insert` builds the **entire** `INSERT INTO ... VALUES (...),(...);`
+   as one giant String — first an `Array` of N per-row tuple strings, then the
+   joined result — held simultaneously with the result set from step 1.
+
+## Baseline (200,000 rows, 8 columns, ~41.2 MB output)
+
+Measured via `bench_sql_dump.rb` (sandbox disabled — it needs `ps` for RSS and
+localhost for the live DB). RSS is sticky across in-process phases, so read the
+peaks as upper bounds and the *deltas* as the signal.
+
+| adapter | execute peak (Δ) | + whole-string write peak | result-set objs |
+|---------|------------------|---------------------------|-----------------|
+| postgresql | 471.7 MB (+65)  | 494.0 MB | 1.8M |
+| mysql      | 413.4 MB (+52)  | 554.9 MB | 2.4M |
+| sqlite     | 434.5 MB        | 526.6 MB | 1.4M |
+
+For a **41 MB** dump the process peaks near **0.5 GB** — ~12× the output size —
+because the full result set *and* the whole-table INSERT string are resident at
+once. Both costs scale linearly with the table, so a large table OOMs the same
+way the embed-heavy MongoDB collection did.
+
+Per-value serialization is cheap and not the bottleneck: `escape_value` is
+~0.4–1.3 µs/op and a one-row `to_bulk_insert` ~5.5 µs/op. The cost is **memory**
+(holding everything at once), not CPU.
+
+## The byte-identity catch (why this differs from MongoDB)
+
+MongoDB's fix was a `default_bulk_insert_chunk_size`, which is byte-identical
+because JSONL chunks join with the same `"\n"` `to_bulk_insert` already inserts
+between docs. **That does not transfer to SQL.** Each `to_bulk_insert` call wraps
+its rows in its own `INSERT INTO ... VALUES ...;` statement, so a chunk size > 0
+turns one INSERT into *many* INSERT statements — semantically equivalent on
+re-import, but a **different byte stream** that breaks the snapshot guard.
+
+The byte-identical lever for SQL is instead to **stream the tuples into a single
+INSERT statement**: emit the adapter's exact `INSERT ... VALUES\n` header once,
+then write each row's `(...)` tuple (reusing the adapter's own `escape_value`)
+separated by `",\n"`, then `;`. The bench implements this as `write_streamed`
+and asserts byte-for-byte identity with the whole-string path — confirmed
+**true** for all three adapters (the header must reuse the adapter's quoting,
+e.g. MySQL's backticks, or it diverges).
+
+`write_streamed` cuts the to_bulk_insert peak by ~110–120 MB, **but** naive
+per-row `IO#print` makes it ~2–2.4× slower than building one string and writing
+it once. So the production fix wants **chunk-buffered streaming**: build an
+N-row substring in memory, write it, repeat — managing the `",\n"` separator
+across chunk boundaries — to bound memory *without* the per-row IO penalty,
+while still emitting a single INSERT statement (byte-identical).
+
+## Resolution: hotspot #2 (streamed single INSERT)
+
+Implemented. The Runner no longer builds the per-table INSERT as one giant
+String; it delegates writing to a new adapter seam `Adapter#write_inserts(io,
+results, table, chunk_size)`:
+
+- `Adapter::Base#write_inserts` keeps the old behavior (write `to_bulk_insert`
+  per chunk, joined by `"\n"`), so MongoDB and any future adapter are unchanged.
+- The SQL adapters mix in `Adapter::SqlBulkInsert`, which **streams** the single
+  `INSERT INTO ... VALUES <tuples>;` statement to the file `STREAM_FLUSH_ROWS`
+  (2,000) tuples at a time. Each flush is one fast `map`+`join` (the same path
+  `to_bulk_insert` uses), and the `",\n"` printed between slices reproduces the
+  exact separator between tuples — so the bytes are **identical** to the
+  whole-table build. The three duplicate `to_bulk_insert` methods collapsed into
+  the shared module; each adapter now only supplies `insert_header` (its
+  identifier quoting) and `escape_value`.
+
+Verified byte-for-byte by `spec/insert_output_snapshot_spec.rb` (live DB, all
+three adapters) and a flush-boundary sanity check; full suite green.
+
+Measured (`bench_sql_dump.rb`, 200k rows / ~41.2 MB output):
+
+| adapter | whole-string peak | streamed peak | Δ peak |
+|---------|-------------------|---------------|--------|
+| postgresql | 367 MB | 226 MB | −141 MB |
+| mysql      | 325 MB | 214 MB | −111 MB |
+| sqlite     | 353 MB | 207 MB | −146 MB |
+
+So hotspot #2's contribution (~110–145 MB on a 41 MB dump — the whole INSERT
+string *plus* the transient 200k-tuple `Array` and its join) is gone; the write
+buffer is now bounded to ~2,000 tuples regardless of table size.
+
+**Speed:** streaming is *not* slower than the whole-string build. Measured in
+isolation (post-`GC.start`, no sampler thread) the streamed write at
+`flush_rows=2000` was ~0.67 s vs ~0.83 s for the one giant `map`+`join` — small
+chunks stay in cache and avoid repeatedly growing/copying a 41 MB String. The
+~1.3× "slowdown" the in-process bench shows is an artifact of its background RSS
+sampler thread (`ps` every 10 ms) plus run-ordering (streamed runs first, cold),
+not the algorithm. The earlier worry about a per-row `IO#print` penalty only
+applied to the naive row-at-a-time prototype, which `flush_rows` slicing avoids.
+
+## Resolution: hotspot #1 (streaming fetch, postgresql + mysql)
+
+Implemented for **postgresql**. `PostgresqlAdapter#execute` no longer returns
+`connection.exec(sql).values` (the whole result set as a Ruby array-of-arrays);
+it returns a lazy `PostgresqlAdapter::StreamingResult` that pulls rows off the
+wire one at a time via libpq **single-row mode** (`send_query` +
+`set_single_row_mode` + a `get_result` loop, each yielding one row's
+text-format `Array<String|nil>`). The Runner drives it exactly like the old
+array — `#size` then a single `each_slice` pass — so nothing else changed and
+the output is byte-identical (verified by `insert_output_snapshot_spec`, both
+the `insert` and `copy` pg scenarios).
+
+It mirrors `MongodbAdapter::StreamingResult`, with two SQL-specific points:
+
+- **`#size`** can't be answered cheaply from the cursor, so it runs a separate
+  `SELECT COUNT(*) FROM (<query>) AS exwiw_count_src` (comment-prefixed, like
+  the data query). Postgres prunes the wrapped subquery's unused projection, so
+  the COUNT transfers no row data — but it does re-run the query plan. This is
+  the deliberate cost of keeping the Runner contract (`#size` before iteration,
+  used to skip empty tables and log the count) unchanged, so MongoDB and the
+  other SQL adapters are untouched. (MongoDB's `count_documents` is an
+  index-only walk and cheaper; the SQL COUNT is the analogue.)
+- the streaming pass ties up the connection until fully drained. The Runner
+  always drains it (`write_inserts`) before issuing `post_insert_sql` / DELETE
+  on the same connection, so the ordering holds. `StreamingResult#each` also
+  drains any queued results if iteration is abandoned mid-stream (a SQL error
+  surfaced by `#check`, or the consumer raising), so the connection stays
+  usable.
+
+Measured in **isolated fresh processes** (one per path, so the peak is not
+polluted by other phases — RSS is sticky), 200k rows / ~41.2 MB output:
+
+| pg fetch path | peak RSS | Δ over baseline |
+|---------------|----------|-----------------|
+| materialize (`exec(sql).values`) + streamed write (OLD) | ~360 MB | ~320 MB |
+| **single-row stream + streamed write (NEW)** | **~48 MB** | **~12 MB** |
+
+So the full result set (~320 MB of Ruby strings/arrays for 200k×8) is no longer
+resident: peak drops ~**310 MB (~87%)** and is now *below* the 41 MB output
+size, because the row is pulled one at a time and the write buffer is bounded to
+~2,000 tuples (hotspot #2's fix). Speed is unchanged (~1.8 s both paths on the
+in-process bench; the COUNT is cheap on an indexed seed). The reproducible A/B
+is in `bench_sql_dump.rb` Part B (`execute(stream)` vs `execute(materialize)`,
+with a byte-identity assertion).
+
+Implemented for **mysql** too. `MysqlAdapter#execute` now returns a
+`MysqlAdapter::StreamingResult` (an Enumerable mirroring the pg one) instead of
+`connection.query(sql).rows`. The new `MysqlClient#stream_rows` pulls rows off
+the wire one at a time via mysql2's server-side stream (`stream: true` +
+`cache_rows: false`), yielding the same `Array<String|nil>` rows `#query`
+buffered — so the generated INSERT is byte-identical (verified by
+`insert_output_snapshot_spec`).
+
+Two MySQL specifics differ from the pg path:
+
+- **`#size`** is a separate `SELECT COUNT(*)` of the same query, but **not** a
+  subquery wrap. MySQL rejects a derived table with duplicate column names,
+  which a rails-managed `SELECT *` joined to another table produces
+  (`Duplicate column name 'id'`); Postgres tolerates it, MySQL does not. So
+  mysql replaces the projection with `COUNT(*)` instead
+  (`compile_ast(count_only: true)`) — exact because exwiw's extraction queries
+  have no DISTINCT/GROUP BY/LIMIT, so the count is independent of the projected
+  columns (confirmed against live data: `COUNT(*)` over the bare FROM/JOIN/WHERE
+  equals the streamed row count for both plain and `SELECT *`+join queries).
+- **abandoned streams.** mysql2 requires a streamed result to be fully consumed
+  before the next query on the connection, or it raises "Commands out of sync".
+  `stream_rows` drains the remainder (re-entering `res.each`, which continues
+  from where it stopped) if the consumer block raises, so the connection stays
+  usable for the next table. `trilogy` has no streaming cursor
+  (no `QUERY_FLAGS_STREAMING`), so it buffers and yields — parity, no memory
+  win; trilogy is a test-only driver, production uses mysql2.
+
+Measured in **isolated fresh processes** (one per path), 200k rows / ~40.7 MB
+output:
+
+| mysql fetch path | peak RSS | Δ over baseline |
+|------------------|----------|-----------------|
+| materialize (`query(sql).rows`) + streamed write (OLD) | ~340 MB | ~300 MB |
+| **single-row stream + streamed write (NEW)** | **~50 MB** | **~10 MB** |
+
+So peak drops ~**290 MB (~85%)**, now just above the 40.7 MB output — the same
+shape as the pg result. Speed is unchanged-to-faster (the materialize path also
+builds the whole array first). `bench_sql_dump.rb` Part B now shows the delta
+for mysql too (it was equivalent before, when mysql still materialized).
+
+Implemented for **sqlite** too, closing hotspot #1 for all three SQL adapters.
+`SqliteAdapter#execute` no longer returns `connection.execute(sql)` (which
+buffers the whole result into a Ruby array); it returns a
+`SqliteAdapter::StreamingResult` (Enumerable, mirroring the pg/mysql ones) whose
+`#each` walks the result one row at a time through SQLite's **statement cursor**
+— `connection.prepare(data_sql)` then `Statement#each` (which maps to
+`sqlite3_step`), closing the statement in an `ensure` so an abandoned mid-stream
+iteration still releases the cursor. The rows are the same `Array` of
+native-typed values `Database#execute` produced, so the generated INSERT is
+byte-identical (verified by `insert_output_snapshot_spec` and a direct cursor
+vs. `#execute` comparison).
+
+SQLite specifics vs. the pg/mysql paths:
+
+- **`#size`** runs a separate `SELECT COUNT(*)` of the same query with the
+  projection replaced by `COUNT(*)` (`compile_ast(count_only: true)`, the same
+  trick mysql uses) — exact because exwiw's extraction queries have no
+  DISTINCT/GROUP BY/LIMIT. SQLite would also tolerate a duplicate-column
+  subquery wrap (unlike mysql), but the `count_only` form is shared and avoids
+  the extra subquery.
+- **no connection contention.** SQLite is an embedded, single-connection engine
+  that allows multiple active prepared statements at once, so the `#size` COUNT
+  and the data cursor don't fight over the connection the way the pg/mysql
+  single-row streams tie up the wire. No drain dance is needed; just close the
+  statement.
+
+Measured in **isolated fresh processes** (one per path), 200k rows / ~40.5 MB
+output:
+
+| sqlite fetch path | peak RSS | Δ over baseline |
+|-------------------|----------|-----------------|
+| materialize (`Database#execute`) + streamed write (OLD) | ~298 MB | ~257 MB |
+| **statement-cursor stream + streamed write (NEW)** | **~59 MB** | **~18 MB** |
+
+So peak drops ~**240 MB (~80%)**, the same shape as pg/mysql, and it is
+**faster** (~0.84 s vs ~1.68 s) — the materialize path pays to build the whole
+Ruby array up front, the cursor does not. `bench_sql_dump.rb` Part B now shows a
+real delta for sqlite too (it was equivalent before, when sqlite still
+materialized).
+
+## Status: both hotspots closed for all three SQL adapters
+
+1. **Bounded-memory write** (hotspot #2) — done for mysql / postgresql / sqlite;
+   see [Resolution #2](#resolution-hotspot-2-streamed-single-insert).
+2. **Streaming result fetch** (hotspot #1) — done for postgresql (libpq
+   single-row mode), mysql (mysql2 `stream: true`), and sqlite (statement
+   cursor); see Resolution #1 above.
+
+There is no remaining materialization hotspot in the SQL dump path: peak RSS is
+now bounded (well below the output size) and independent of table size for every
+SQL adapter, the same property the MongoDB streaming work achieved. The
+`trilogy` driver still buffers (it has no streaming cursor flag), but it is a
+test-only driver — production mysql uses mysql2.
+
+## Methodology notes
+
+- The serialization hotspot reproduces **with no database** (Part A): synthesize
+  the array-of-String-arrays the drivers return and measure `to_bulk_insert`.
+  The live-DB part (Part B) measures `execute` and needs a reachable DB; the dev
+  sandbox blocks localhost (and `ps`), so disable the sandbox for bench runs.
+- Run order matters: the bench measures the STREAMED path **before** the WHOLE
+  path so the transient giant String doesn't pollute the streamed peak (RSS is
+  reclaimed lazily). For defensible absolute numbers, isolate phases in fresh
+  processes.
+- Ruby 4.0 removed the `benchmark` stdlib; the harness uses
+  `Process.clock_gettime(Process::CLOCK_MONOTONIC)`.