exwiw 0.9.4 → 0.9.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c472ab8d5098ed7cd796e8549329dadba6b4cce31b2d44b956f463699643a353
-  data.tar.gz: c5b53ba657142131d1dacdb44b3bca0117b55052384d9da2257108be6073b511
+  metadata.gz: 4a9938f5cd311ecefb5568ed27f35ed8594a9ba15e0041e241c423b25af12cba
+  data.tar.gz: f838bd892aecb7a717eac84f1bf52045cec0b6c1881c3a26c29bd4983c845326
 SHA512:
-  metadata.gz: 79879008903ad2eff63b7c14851611fd4d6b8161726240faef77cf4533a1c172a9cbed140abe71dde1e1026de521d4d81b5d2eff074d977a8883476979f10ad0
-  data.tar.gz: bbb86bbd868783c891a1a45a337977160e569c920ded2e3a8782c098e702dabfe0f6bbbad599fd0db0b0949731a471193529c65f382de6d22c31524c776468ef
+  metadata.gz: 17568187de281651d94abf50190bd0926a3a85eaaf468f9dcdc43b1254583c8cc416f5282e55de16f8af7a44dc1635f95d10f6a06b33de668ab98ace8e7eda79
+  data.tar.gz: 8530a5f2eaf0c15dda1c604e6bc65e923b8f91322990352e3de6ccb3c8dc832f2184e76496941cc4d64c49089b1639bc7b3743f20e79c3ad6083acff1b9c6f26

data/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 ## [Unreleased]
 
+## [0.9.6] - 2026-07-08
+
+### Added
+
+- **Ruby after-insert hooks can seed named MongoDB collections with `insert_jsonl(collection, template)`.** The single-argument `insert_sql(template)` / `insert_jsonl(template)` form writes all hook output to one collection-less `insert-{N+1}-after_insert.{ext}` file. That is fine for SQL — the statements name their table in-band — but MongoDB import derives the target collection from the filename (`insert-NNN-<collection>.jsonl` → `mongoimport --collection <collection>`), so a hook had no way to seed documents into specific collections. The new two-argument form (**mongodb-only**; a SQL adapter raises `ArgumentError`) appends the ERB-rendered extended-JSON lines to the named collection's own buffer; after the hook finishes each targeted collection is written to its own `insert-NNN-<collection>.jsonl`, numbered sequentially after the dump's own files (and after the collection-less `after_insert` file when both forms are used), so the existing filename-based import convention applies to hook output unchanged. The single-argument form is untouched.
+
+### Fixed
+
+- **The MySQL adapter now escapes newline and control characters (`\n`, `\r`, `\0`, `\Z`) in string values, matching mysqldump.** They were previously written raw, so a value containing a newline was emitted with a literal line break inside its `VALUES` tuple. That is valid SQL on its own, but it breaks any consumer that splits a dump into statements on a semicolon-newline boundary when a string value contains `;` immediately followed by a newline — the statement is cut mid-value, leaving an unterminated string literal. Escaping keeps every tuple on a single line. Single-quote escaping (doubling) is unchanged, and the SQLite/PostgreSQL adapters are unaffected. MySQL insert output changes only for string values that contain these characters.
+
+## [0.9.5] - 2026-07-07
+
+### Fixed
+
+- **PostgreSQL: a `COMMENT ON EXTENSION` for a skipped extension no longer aborts the restore.** pg_dump emits `COMMENT ON EXTENSION <name> IS '...'` right after each `CREATE EXTENSION`. When the `CREATE` was skipped by the graceful `DO … EXCEPTION WHEN feature_not_supported` block (a target that cannot provide the extension — e.g. AlloyDB's `google_vacuum_mgmt` restored into vanilla PostgreSQL), the bare `COMMENT` then failed with `undefined_object` (`extension "…" does not exist`) and aborted the whole restore. A new `DdlPostprocessor.wrap_comment_on_extension_in_do_block` pass wraps each `COMMENT ON EXTENSION` in a `DO $exwiw$ … EXCEPTION WHEN undefined_object THEN NULL … $exwiw$` block, so it applies when the extension exists and is a no-op when it was skipped. This makes any managed-platform extension (AlloyDB `google_*`, `alloydb_scann`, etc.) skip cleanly, not just the one reported.
+
 ## [0.9.4] - 2026-07-07
 
 ### Changed

data/README.md

@@ -488,10 +488,11 @@ By default, exwiw generates `delete-*.sql` files alongside the `insert-*.sql` fi
 
 `--after-insert-hook=PATH` runs a post-processing hook **after** all per-table insert/delete files have been written. The hook can be either a Ruby file (`.rb`) or any executable script (e.g. `.sh`).
 
-**Ruby hook (`.rb`)**: provides a tiny DSL with two builtins:
+**Ruby hook (`.rb`)**: provides a tiny DSL with these builtins:
 
 - `cli_options` — Hash of all parsed CLI options (e.g. `cli_options.fetch(:ids)` returns the `--ids` array).
 - `insert_sql(template)` — appends an ERB-rendered string to a buffer. After the hook finishes, the buffer is concatenated and written to `insert-{N+1}-after_insert.{ext}` where `{N+1}` is one past the last per-table insert file. For the MongoDB adapter the equivalent alias `insert_jsonl(template)` is available; output goes to `insert-{N+1}-after_insert.jsonl`. Multiple `insert_sql` calls in a single hook are joined with `"\n"` into the same file. If no `insert_sql` call is made, no file is created.
+- `insert_jsonl(collection, template)` — **MongoDB adapter only**. SQL statements name their table in-band, but JSONL documents do not — the import convention derives the target collection from the filename — so the two-argument form writes the ERB-rendered extended-JSON lines to the named collection's own `insert-NNN-<collection>.jsonl` file, importable with the same `mongoimport --collection <collection>` convention as the per-collection dump files. Multiple calls targeting the same collection are appended (joined with `"\n"`) into that collection's file; distinct collections get one file each, numbered sequentially after the last per-collection dump file (the collection-less `after_insert` buffer, when also used, keeps `{N+1}` and the collection files follow it). Calling this form with a SQL adapter raises an error.
 
 Example `hooks/seed_default_users.rb`:
 
@@ -504,6 +505,17 @@ insert_sql <<~SQL
 SQL
 ```
 
+MongoDB example seeding two collections (`insert-{N+1}-users.jsonl` and `insert-{N+2}-posts.jsonl`):
+
+```ruby
+insert_jsonl 'users', <<~JSONL
+  <%- cli_options.fetch(:ids).each do |shop_id| -%>
+  {"shop_id":{"$oid":"<%= shop_id %>"},"email":"default@example.com"}
+  <%- end -%>
+JSONL
+insert_jsonl 'posts', '{"title":"welcome"}'
+```
+
 **Shell hook**: anything other than `.rb` is exec'd as a child process. It is a pure side-effect hook — exwiw does not capture its stdout. The hook receives these env vars and inherits `DATABASE_PASSWORD` from the parent:
 
 - `EXWIW_OUTPUT_DIR`, `EXWIW_SCHEMA_DIR`
@@ -808,6 +820,7 @@ The MongoDB adapter is experimental. To use it:
   ```bash
   mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl
   ```
+- A Ruby [after-insert hook](#after-insert-hook) can seed extra documents into named collections with `insert_jsonl(collection, template)`; each targeted collection gets its own `insert-NNN-<collection>.jsonl` file numbered after the dump's own files, so the filename-based `mongoimport` convention above applies to hook output unchanged.
 - The leading `dump/insert-000-schema.js` contains `db.createCollection(...)` and `db.<col>.createIndex(...)` calls for every top-level collection (indexes are introspected from the source via `listIndexes`; the auto-created `_id_` index is skipped). Apply it with mongosh **before** running `mongoimport`:
   ```bash
   mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -340,12 +340,24 @@ module Exwiw
           "WHERE #{subquery.table_name}.#{subquery.where_column} IN (#{inner_values.join(', ')})"
       end
 
+      # Backslash and control-character escapes, matching mysqldump. Escaping
+      # newlines keeps every VALUES tuple on a single line, so a value that
+      # itself contains a newline cannot break consumers that split the dump
+      # into statements on a semicolon-newline boundary.
+      SPECIAL_CHARACTER_ESCAPES = {
+        "\\" => "\\\\",
+        "\n" => "\\n",
+        "\r" => "\\r",
+        "\u0000" => "\\0",
+        "\u001A" => "\\Z",
+      }.freeze
+
       private def escape_value(value)
         case value
         when nil
           "NULL"
         when String
-          qv = value.gsub('\\') { '\\\\' }
+          qv = value.gsub(/[\\\r\n\u0000\u001A]/) { |char| SPECIAL_CHARACTER_ESCAPES.fetch(char) }
           qv = escape_single_quote(qv)
           "'#{qv}'"
         else

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -136,10 +136,14 @@ module Exwiw
         # robustness: enums swallow duplicate_object on re-restore; extensions
         # warn-and-skip feature_not_supported / invalid_schema_name so a target
         # that cannot provide the extension (e.g. pglogical's schema absent, or a
-        # managed-platform extension) does not abort the restore.
+        # managed-platform extension) does not abort the restore. The COMMENT ON
+        # EXTENSION that pg_dump emits alongside is likewise wrapped to swallow
+        # undefined_object, so a skipped extension's trailing comment does not
+        # abort the restore either.
         idempotent = stdout
         idempotent = DdlPostprocessor.wrap_create_type_enum_in_do_block(idempotent)
         idempotent = DdlPostprocessor.wrap_create_extension_in_do_block(idempotent)
+        idempotent = DdlPostprocessor.wrap_comment_on_extension_in_do_block(idempotent)
         idempotent = DdlPostprocessor.add_if_not_exists_to_create_schema(idempotent)
         idempotent = DdlPostprocessor.add_if_not_exists_to_create_sequence(idempotent)
         idempotent = DdlPostprocessor.add_if_not_exists_to_create_table(idempotent)

data/lib/exwiw/after_insert_hook.rb

@@ -6,26 +6,51 @@ module Exwiw
   class AfterInsertHook
     def self.run(path:, cli_options:, output_dir:, next_idx:, output_extension:, logger:)
       ext = File.extname(path)
-      idx_str = next_idx.to_s.rjust(3, '0')
-      output_path = File.join(output_dir, "insert-#{idx_str}-after_insert.#{output_extension}")
 
       if ext == '.rb'
-        run_ruby(path: path, cli_options: cli_options, output_path: output_path, logger: logger)
+        run_ruby(
+          path: path,
+          cli_options: cli_options,
+          output_dir: output_dir,
+          next_idx: next_idx,
+          output_extension: output_extension,
+          logger: logger,
+        )
       else
         run_shell(path: path, cli_options: cli_options, output_dir: output_dir, logger: logger)
       end
     end
 
-    def self.run_ruby(path:, cli_options:, output_path:, logger:)
-      ctx = Context.new(cli_options)
+    def self.run_ruby(path:, cli_options:, output_dir:, next_idx:, output_extension:, logger:)
+      ctx = Context.new(cli_options, output_extension: output_extension)
       ctx.instance_eval(File.read(path), path)
+
+      # One output file per buffer, numbered sequentially from next_idx: first
+      # the shared collection-less buffer (kept at next_idx so existing hooks
+      # produce the same filename as before), then one file per targeted
+      # collection in first-targeted order. Collection files reuse the
+      # insert-NNN-<collection>.{ext} naming of the per-table dump files, so
+      # filename-driven import tooling handles them unchanged.
+      outputs = []
       content = ctx.collected.join("\n")
-      if content.empty?
+      outputs << ['after_insert', content] unless content.empty?
+      ctx.collected_by_collection.each do |collection, chunks|
+        body = chunks.join("\n")
+        next if body.empty?
+        outputs << [collection, body]
+      end
+
+      if outputs.empty?
         logger.info("After-insert hook produced no output; skipping file write.")
         return
       end
-      File.write(output_path, content)
-      logger.info("Wrote after-insert hook output to #{output_path}")
+
+      outputs.each_with_index do |(name, body), offset|
+        idx_str = (next_idx + offset).to_s.rjust(3, '0')
+        output_path = File.join(output_dir, "insert-#{idx_str}-#{name}.#{output_extension}")
+        File.write(output_path, body)
+        logger.info("Wrote after-insert hook output to #{output_path}")
+      end
     end
 
     def self.run_shell(path:, cli_options:, output_dir:, logger:)
@@ -48,17 +73,52 @@ module Exwiw
     end
 
     class Context
-      attr_reader :cli_options, :collected
+      # Collection names become part of the output filename, so restrict them
+      # to plain path-safe names (no separators, no leading dot).
+      COLLECTION_NAME_PATTERN = /\A[A-Za-z0-9_][A-Za-z0-9_.-]*\z/
+
+      attr_reader :cli_options, :collected, :collected_by_collection
 
-      def initialize(cli_options)
+      def initialize(cli_options, output_extension: nil)
         @cli_options = cli_options
+        @output_extension = output_extension
         @collected = []
+        @collected_by_collection = {}
       end
 
       def insert_sql(template)
-        @collected << ERB.new(template, trim_mode: '-').result(binding)
+        @collected << render(template)
+      end
+
+      # With one argument, identical to insert_sql: the rendered output goes to
+      # the shared collection-less buffer (insert-NNN-after_insert.{ext}).
+      #
+      # With two arguments (MongoDB adapter only), the rendered output is
+      # appended to the named collection's own buffer and written to
+      # insert-NNN-<collection>.jsonl. SQL statements name their table in-band,
+      # but JSONL documents do not — the import convention derives the target
+      # collection from the filename — so this is the only way a hook can seed
+      # documents into a specific collection.
+      def insert_jsonl(collection_or_template, template = nil)
+        return insert_sql(collection_or_template) if template.nil?
+
+        unless @output_extension == 'jsonl'
+          raise ArgumentError,
+                "insert_jsonl(collection, template) is only supported for the MongoDB adapter; " \
+                "SQL statements already name their table, use insert_sql(template) instead."
+        end
+
+        collection = collection_or_template.to_s
+        unless collection.match?(COLLECTION_NAME_PATTERN)
+          raise ArgumentError, "invalid collection name for insert_jsonl: #{collection_or_template.inspect}"
+        end
+
+        (@collected_by_collection[collection] ||= []) << render(template)
+      end
+
+      private def render(template)
+        ERB.new(template, trim_mode: '-').result(binding)
       end
-      alias_method :insert_jsonl, :insert_sql
     end
   end
 end

data/lib/exwiw/ddl_postprocessor.rb

@@ -106,6 +106,26 @@ module Exwiw
       end
     end
 
+    # pg_dump emits `COMMENT ON EXTENSION <name> IS '...';` right after the
+    # matching CREATE EXTENSION. When the CREATE was skipped (its DO block caught
+    # feature_not_supported because the target cannot provide the extension —
+    # e.g. AlloyDB's google_vacuum_mgmt restored into vanilla PostgreSQL), this
+    # bare COMMENT then aborts the whole restore with `undefined_object` (42704,
+    # "extension ... does not exist"). Wrap each COMMENT in a DO block that
+    # swallows undefined_object, so it applies when the extension exists and is a
+    # no-op when it was skipped. The IS clause is matched as a whole single-quoted
+    # string (doubled quotes escaped) or NULL, so an embedded `;` does not end the
+    # match early.
+    COMMENT_ON_EXTENSION_RE =
+      /^[ \t]*COMMENT\s+ON\s+EXTENSION\s+(?:"[^"]+"|[^\s]+)\s+IS\s+(?:'(?:[^']|'')*'|NULL)\s*;/i.freeze
+
+    def wrap_comment_on_extension_in_do_block(sql)
+      sql.gsub(COMMENT_ON_EXTENSION_RE) do
+        stmt = Regexp.last_match(0).strip
+        "DO $exwiw$ BEGIN #{stmt} EXCEPTION WHEN undefined_object THEN NULL; END $exwiw$;"
+      end
+    end
+
     # Generate idempotent CREATE TYPE ... AS ENUM statements.
     # +enum_types+ is an Array of Hashes with keys :schema, :name, :labels.
     def create_type_enum_statements(enum_types)

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.4"
+  VERSION = "0.9.6"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.4
+  version: 0.9.6
 platform: ruby
 authors:
 - Shia