exwiw 0.9.24 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 73740836024ec336feb7138c5a94e396df206de932de00737404d556fc403e0a
-  data.tar.gz: de53ac3f5e4c93f6761524041261c2becb65baa9bd396e528ca0ffa9b82b518e
+  metadata.gz: ecbb0b978dfd64c1e48c396e600662a9f7d4bb9c2c8f0e0050440987b53c815d
+  data.tar.gz: eff38c81ce04abb165f0edcad7316e1820d44abd54c6596ce450357c11a9c201
 SHA512:
-  metadata.gz: ac457715cf1eaf521840903e2b2fbb6c5f503230d0abfb7cc24510067c2aa68c8b841d370d8eeed3e0c0ee009a105bea87cfe455a21c50b5bd462feb48c14f22
-  data.tar.gz: 945c4c46077e9ee6678f6fbc6e72b13186e393c4f06035dd1e9d63f10efa3c1fe6adb67a05f470e1330a48d5d8f34d008c67cb8795ba7dbda8294178f13082a0
+  metadata.gz: 2b119cb478ae9e6bb019bb2d0d94272ed84f7bb20ac3013236260b8c98433c9c97f37b3d4dafc132a70736d0d6eb4a20e4fc0e44e473dfc5d1a91f7dbda560a0
+  data.tar.gz: 17268aa8ec3ad2959521cc4b16d453e86480935ae98ba43e4f38fb4545dada3af23980fc79b8f3751268fa79e078e87c7e7388e8bab7ce9fefbef9e6344c9453

data/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 ## [Unreleased]
 
+## [1.0.0] - 2026-09-01
+
+### Removed
+
+- **BREAKING — delete SQL support is removed ([#166](https://github.com/heyinc/exwiw/issues/166)): `export` no longer generates `delete-NNN-<table>.sql` files.** They existed to clear an already-populated import target before re-inserting; the supported workflow is now to import into an empty database (`insert-000-schema.sql` provisions one) or to clear the target yourself. Every export now behaves as `--insert-only` always did, so the `--insert-only` flag and the `insert_only:` config key have nothing left to toggle — both are still **accepted but ignored**, with a warning naming them obsolete, so an existing invocation or committed config file keeps working across the upgrade; remove them at your leisure (a future release will reject them). For the adapter API, `to_bulk_delete`, `supports_bulk_delete?`, and `pre_delete_sql` are removed.
+
+### Changed
+
+- **BREAKING — PostgreSQL: `insert-000-schema.sql` now contains the source's triggers, wrapped in `DO $exwiw$ ... EXCEPTION WHEN duplicate_object`.** They used to be deleted from the dump, so a restored database silently ran none of them: an audit trigger recorded nothing, a timestamp trigger never fired, and the dump gave whoever restored it no way to add them back. The deletion existed because a `--table` dump emitted `CREATE TRIGGER` without the `CREATE FUNCTION` it references, failing the restore with `PG::UndefinedFunction`; the dump has since become a whole-database one that carries the functions too, so it was a leftover workaround rather than intended behavior — `mysqldump` and sqlite's `sqlite_master` have always emitted triggers. It is nonetheless a breaking change: dumps grow the trigger statements. They do not fire during the load, because each `insert-*.sql` now opens with the `session_replication_role = 'replica'` block described below. The DO block keeps the schema re-appliable, which a bare `CREATE TRIGGER` is not (`CREATE OR REPLACE TRIGGER` is PostgreSQL 14+ only and `pg_dump` never emits it).
+
+- **PostgreSQL: each `insert-NNN-<table>.sql` now opens with a `session_replication_role = 'replica'` block, so triggers and foreign keys do not fire while the target is loaded.** With the source's triggers now in the schema, an unguarded load would run every one of them once per inserted row — over rows that already carry the values those triggers produced on the source — and would enforce FKs against a target that is only complete once every `insert-*.sql` has been applied. This is the counterpart of the `FOREIGN_KEY_CHECKS=0` that `MysqlAdapter#pre_insert_sql` has always emitted. Setting the parameter requires superuser (`rds_superuser` on RDS); when the restoring role lacks it the block catches `insufficient_privilege` and downgrades to a `WARNING`, so the load still runs — with triggers firing, as it did before this release. The setting is connection-scoped (`set_config(..., false)`) and is not reset at the end of the file — every file re-arms it itself, so concatenating them works regardless, but sourcing them into a session that goes on to do other work leaves that session in replica mode.
+
+### Fixed
+
+- **PostgreSQL: a `CREATE TRIGGER` inside a function body is no longer rewritten.** The removed `DdlPostprocessor.strip_triggers` matched `^CREATE TRIGGER` anywhere, so it also deleted such a line from a dollar-quoted body, leaving a function that still compiled but no longer installed its trigger. Its replacement, `wrap_create_trigger_in_do_block`, anchors on the `-- Name: <table> <trigger>; Type: TRIGGER` header `pg_dump` writes before each trigger, which never appears inside a body. `CREATE CONSTRAINT TRIGGER` is handled, and a `;` inside a quoted trigger argument no longer truncates the statement.
+
 ## [0.9.24] - 2026-08-17
 
 ### Fixed

data/README.md

@@ -102,12 +102,15 @@ The output dir is emptied before each export so it never mixes files from a prev
 
 - `dump/insert-000-schema.sql` — idempotent `CREATE TABLE IF NOT EXISTS ...` for every table in scope. Apply this first to provision an empty database.
 - `dump/insert-{idx}-{table_name}.sql`
-- `dump/delete-{idx}-{table_name}.sql`
 
 idx means the order of the dump. bigger idx might depend on smaller idx,
 so you should import the dump in order.
 
-`insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. For `mysql`, set `EXWIW_MYSQLDUMP` to point at a specific `mysqldump` binary when the one on PATH is incompatible with the server (e.g. a MySQL 9.x `mysqldump` cannot load `mysql_native_password` against a server still using that auth plugin — `EXWIW_MYSQLDUMP=/path/to/mysql@8.0/bin/mysqldump`). The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`. For `mysql`, the source server's `DEFINER=user@host` stamp on views and triggers is stripped too, so restoring into a managed MySQL instance (which usually can't grant the privilege to recreate someone else's `DEFINER`) does not fail.
+exwiw generates INSERT statements only — it does not generate DELETE statements. Import into an empty database (`insert-000-schema.sql` provisions one), or clear the target's rows yourself before importing.
+
+`insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. For `mysql`, set `EXWIW_MYSQLDUMP` to point at a specific `mysqldump` binary when the one on PATH is incompatible with the server (e.g. a MySQL 9.x `mysqldump` cannot load `mysql_native_password` against a server still using that auth plugin — `EXWIW_MYSQLDUMP=/path/to/mysql@8.0/bin/mysqldump`). The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` and `CREATE TRIGGER` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`. For `mysql`, the source server's `DEFINER=user@host` stamp on views and triggers is stripped too, so restoring into a managed MySQL instance (which usually can't grant the privilege to recreate someone else's `DEFINER`) does not fail.
+
+The schema file carries the source's triggers. They are suppressed while the target is loaded: each `insert-NNN-<table>.sql` opens with a block that sets `session_replication_role = 'replica'` for the connection, which turns off both user triggers and foreign-key enforcement for the statements that follow (the PostgreSQL counterpart of the `FOREIGN_KEY_CHECKS=0` `mysql` dumps already carry). Setting it requires superuser (`rds_superuser` on RDS); if the restoring role lacks the privilege the block reports a `WARNING` and the load proceeds with triggers firing. The setting applies to the connection and is not reset at the end of each file — every file re-arms it itself, so `psql -f` per file and `cat insert-*.sql | psql` both work, but a session that sources these files and then goes on to do other work stays in replica mode; reset it yourself (`SET session_replication_role = 'origin'`) in that case. `sqlite` has no equivalent and loads with its triggers active.
 
 For `postgresql`, the extensions a managed platform installs to run the source instance itself are treated as out of target and left out of the dump entirely — currently `google_vacuum_mgmt` (Cloud SQL / AlloyDB adaptive autovacuum), `google_columnar_engine` and `google_db_advisor` (AlloyDB). They serve the source instance's operation (vacuum tuning, the in-memory columnar cache, index advice), hold no application data, are referenced by nothing in the application's own schema, and ship only with the managed platform, so a restore target outside it can never create them. Their schemas are dropped via `pg_dump --exclude-schema` and their `CREATE EXTENSION` / `COMMENT ON EXTENSION` statements — which are not schema-qualified, so no `pg_dump` filter reaches them — are removed from the output; whatever was excluded is named in the run's log.
 
@@ -116,11 +119,6 @@ The list is exact names, not a `google_*` prefix match: those prefixes are not r
 - a third-party extension pulled in as a dependency of an excluded one (`google_db_advisor` requires `hypopg`), since that one *is* installable on a plain PostgreSQL, and
 - an application-facing platform extension (`google_ml_integration`, `alloydb_scann`, `alloydb_ai_nl`), which the application's own SQL and DDL can name (a ScaNN index is `USING scann`) — removing its `CREATE` would strand whatever refers to it, so it warns and skips instead.
 
-you need to delete the records before importing the dump,
-`delete-{idx}-{table_name}.sql` will help you to do that.
-This sql will delete "all" related records to the extract targets.
-idx meaning is the same as insert sql.
-
 ### `exwiw explain`
 
 Print the query each `export` would run together with its `EXPLAIN` output, to stdout. For the SQL adapters (`mysql`, `postgresql`, `sqlite`) this is the compiled SELECT plus its `EXPLAIN` (estimate-only; `EXPLAIN QUERY PLAN` on SQLite) — no SELECT is executed. For `mongodb` it is the `find` description plus the server's explain document as JSON.
@@ -135,7 +133,7 @@ exwiw explain \
   --target-table=shops --ids=1
 ```
 
-The `--output-dir`, `--output-format`, `--insert-only`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
+The `--output-dir`, `--output-format`, and `--after-insert-hook` options are dump-specific and rejected when used with `explain`.
 
 MongoDB-specific explain behavior — the configurable verbosity (`queryPlanner` / `executionStats` / `allPlansExecution`) and how scoped collections are shown — is described in [MongoDB support](docs/mongodb.md#exwiw-explain-verbosity).
 
@@ -307,7 +305,6 @@ adapter: postgresql
 schema_dir: exwiw/schema
 output_dir: dump
 output_format: insert        # insert | copy
-insert_only: false
 after_insert_hook: hooks/seed.rb
 log_level: info              # debug | info
 # target_table / ids / ids_field / scope_column may also be set here
@@ -326,8 +323,8 @@ Notes:
 
 - **Database connection settings stay on the CLI/environment.** `host`, `port`, `user`, `database`, `uri`, and `password` are **rejected** in the config file (exwiw exits with an error). `adapter` is the one connection-related key that *is* allowed in the file.
 - **Relative paths in the config (`schema_dir`, `output_dir`, `after_insert_hook`) are resolved relative to the config file's own directory**, not the current working directory. So with the config at the project root, `schema_dir: exwiw/schema` reads naturally, and an absolute `--config=/path/to/exwiw.yml` works no matter where you run from. (CLI path flags remain relative to the current directory — each source resolves relative to where it is written.) Absolute paths are used as-is.
-- Unknown keys are rejected so a typo surfaces immediately.
-- Export-only keys (`output_dir`, `output_format`, `insert_only`, `after_insert_hook`) are ignored when running `explain` or `schema`, so a single config file can be shared by every subcommand.
+- Unknown keys are rejected so a typo surfaces immediately. (`insert_only`, whose behavior was removed, is the one grandfathered key: accepted and ignored with a warning.)
+- Export-only keys (`output_dir`, `output_format`, `after_insert_hook`) are ignored when running `explain` or `schema`, so a single config file can be shared by every subcommand.
 - `explain_verbosity` sets the mongodb `explain` verbosity (`queryPlanner` | `executionStats` | `allPlansExecution`, default `queryPlanner`); the `EXWIW_MONGODB_EXPLAIN_VERBOSITY` env var overrides it. Ignored by the SQL adapters and by `export`. See [MongoDB support](docs/mongodb.md#exwiw-explain-verbosity).
 - `mongodb_query_timeout_ms` sets the global, server-enforced query timeout (mongodb only); the `--mongodb-query-timeout-ms` CLI flag overrides it. Ignored by the SQL adapters. See [MongoDB support](docs/mongodb.md).
 
@@ -618,13 +615,9 @@ psql -d app_dev -f dump/insert-001-shops.sql
 
 `--output-format=copy` is only supported with the `postgresql` adapter.
 
-### Skip DELETE SQL output
-
-By default, exwiw generates `delete-*.sql` files alongside the `insert-*.sql` files so that an existing dataset can be cleared before re-inserting. Pass `--insert-only` when you only need the insert files.
-
 ### After-insert hook
 
-`--after-insert-hook=PATH` runs a post-processing hook **after** all per-table insert/delete files have been written. The hook can be either a Ruby file (`.rb`) or any executable script (e.g. `.sh`).
+`--after-insert-hook=PATH` runs a post-processing hook **after** all per-table insert files have been written. The hook can be either a Ruby file (`.rb`) or any executable script (e.g. `.sh`).
 
 **Ruby hook (`.rb`)**: provides a tiny DSL with these builtins:
 
@@ -666,7 +659,7 @@ Note: Ruby hooks are evaluated via `instance_eval` inside the exwiw process —
 
 ### Ignore a table
 
-Set `"ignore": true` on a table's config JSON to exclude it from data extraction. The table's DDL is still emitted into `insert-000-schema.{sql,js}` so the schema stays consistent, but no `insert-*` / `delete-*` files are generated for it and the table is never queried.
+Set `"ignore": true` on a table's config JSON to exclude it from data extraction. The table's DDL is still emitted into `insert-000-schema.{sql,js}` so the schema stays consistent, but no `insert-*` files are generated for it and the table is never queried.
 
 ```json
 {
@@ -772,7 +765,7 @@ WHERE reviews.reviewable_id IN (/* products subquery */)
   AND reviews.reviewable_type = 'Product'
 ```
 
-The same type filter is applied on the join path — and in the matching `delete-*.sql` bulk-delete subquery — when the polymorphic table is an intermediate hop rather than the directly-dumped table.
+The same type filter is applied on the join path when the polymorphic table is an intermediate hop rather than the directly-dumped table.
 
 #### Every arm is extracted (scope-column mode)
 
@@ -912,7 +905,6 @@ Behavior at dump time:
 
 - Extraction uses `SELECT *` so the dump is robust against Rails-side column additions.
 - `INSERT` statements omit the column list (`INSERT INTO schema_migrations VALUES (...)`). For PostgreSQL `--output-format=copy`, the `COPY` header similarly omits the column list (`COPY schema_migrations FROM stdin;`).
-- No `delete-*.sql` file is generated for rails-managed tables, to avoid wiping migration history on the import target.
 
 Constraints:
 
@@ -982,7 +974,6 @@ An explicit id list of that size is exactly estimated and selective, so the fore
 - The batch table's ids come from **its own extraction query**, so it is narrowed by exactly the filter it would carry in the unbatched query. They are held in memory for the extraction: one scope's worth of primary keys, orders of magnitude smaller than the table being batched.
 - The batch table may be **any number of hops up** the path — a table two hops below it (`activity_orders → activities → customers`) names `customers` too, and the batch ids are applied where the path meets the scope, bounding the whole join chain.
 - A table that **carries the scope column itself** batches by naming itself; each batch then filters `WHERE <pk> IN (<ids>)` directly. Note that the id-set query is then the same scope predicate over the same table, so this shape only avoids the scan when the scope column is indexed (ideally index-only) — the join shape above is the one that genuinely removes the planner's choice.
-- `delete-*.sql` is unaffected (it is generated from the unbatched query).
 - `bulk_insert_chunk_size` is independent: batches are query boundaries, chunks are `INSERT` statement boundaries.
 - With `--output-format=copy`, batching bounds each query's cost but not memory: COPY builds the whole table's body in memory, so all batches' rows are resident at once. Use the default INSERT format (which streams) when the kept rows themselves are huge.
 
@@ -1186,9 +1177,6 @@ exwiw can export MongoDB databases too (`--adapter=mongodb`): JSONL output impor
 - Generate the full list of INSERT sql based on the specified conditions.
   - If the processing table has no relation with target tables, then dump all records.
   - If the processing table has relation with target tables, then dump the records which are related to the target tables.
-- Generate the full list of DELETE sql based on the specified conditions.
-  - If the processing table has no relation with target tables, then delete all records.
-  - If the processing table has relation with target tables, then delete the records which are related to the target tables.
 
 ## Development
 

data/docs/mongodb.md

@@ -32,7 +32,7 @@ exwiw can export a MongoDB database with `--adapter=mongodb`. This document coll
   ```bash
   mongosh "mongodb://localhost/app_dev" dump/insert-000-schema.js
   ```
-- Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
+- exwiw emits no delete files; drop the database / collection yourself before importing if needed.
 
 ## Masking
 

data/lib/exwiw/adapter/identifier_quoting.rb

@@ -170,8 +170,8 @@ module Exwiw
 
       # The subset of SQLite keywords (https://sqlite.org/lang_keywords.html)
       # that actually fail to parse as bare identifiers in the positions exwiw
-      # emits (qualified column, INSERT column list, FROM/DELETE/JOIN table
-      # name, CASE masking, derived-table scope JOIN). SQLite's parser accepts
+      # emits (qualified column, INSERT column list, FROM/JOIN table name,
+      # CASE masking, derived-table scope JOIN). SQLite's parser accepts
       # the other ~half of its keywords as identifiers via fallback (e.g.
       # `key`, `temp`, `row`), and those are deliberately NOT quoted so output
       # for such names stays byte-identical with previous releases — a name in

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -217,10 +217,6 @@ module Exwiw
         end.join("\n")
       end
 
-      def to_bulk_delete(_query, _config)
-        raise NotImplementedError, "MongodbAdapter does not support bulk delete"
-      end
-
       # Default explain verbosity. `queryPlanner` asks the server to PLAN the
       # query without executing it, so it is safe to run against a production
       # source — no documents are scanned or returned. `executionStats` and
@@ -325,10 +321,6 @@ module Exwiw
         @logger.info("  Wrote schema for #{collections.size} collection(s) to #{output_path}.")
       end
 
-      def supports_bulk_delete?
-        false
-      end
-
       # `--ids` from the CLI arrives as Strings. Mongo compares types strictly,
       # so the textual ids must be coerced to the type actually stored in `_id`:
       #

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -26,7 +26,7 @@ module Exwiw
       #     so the row count is independent of the projected columns.
       #   - the stream ties up the connection until fully drained. The Runner
       #     always drains it (write_inserts) before any further query
-      #     (post_insert_sql / DELETE), and MysqlClient#stream_rows drains the
+      #     (post_insert_sql), and MysqlClient#stream_rows drains the
       #     remainder if iteration is abandoned, so the connection stays usable.
       class StreamingResult
         include Enumerable
@@ -171,64 +171,6 @@ module Exwiw
         end
       end
 
-      def to_bulk_delete(select_query_ast, table)
-        raise NotImplementedError unless select_query_ast.is_a?(Exwiw::QueryAst::Select)
-
-        sql = "DELETE FROM #{quote_table_name(select_query_ast.from_table_name)}"
-
-        if select_query_ast.join_clauses.empty?
-          # Ignore filter option, because bulk delete is for cleaning before import,
-          # so it should delete all records to avoid foreign key violation & data consistancy.
-          compiled_where_conditions = select_query_ast.
-            where_clauses.
-            select { |where| where.is_a?(Exwiw::QueryAst::WhereClause) }.
-            map do |where|
-            compile_delete_where_condition(where, select_query_ast.from_table_name)
-          end
-
-          if compiled_where_conditions.size > 0
-            sql += "\nWHERE "
-            sql += compiled_where_conditions.join(' AND ')
-          end
-          sql += ";"
-
-          return sql
-        end
-
-        subquery_ast = Exwiw::QueryAst::Select.new
-        first_join = select_query_ast.join_clauses.first.clone
-
-        subquery_ast.from(first_join.join_table_name)
-        primay_key_col = table.columns.find { |col| col.name == table.primary_key }
-        subquery_ast.select([primay_key_col])
-        select_query_ast.join_clauses[1..].each do |join|
-          subquery_ast.join(join)
-        end
-        first_join.where_clauses.each do |where|
-          # Ignore filter option, because bulk delete is for cleaning before import,
-          # so it should delete all records to avoid foreign key violation & data consistancy.
-          subquery_ast.where(where) if where.is_a?(Exwiw::QueryAst::WhereClause)
-        end
-
-        foreign_key = first_join.foreign_key
-        subquery_sql = compile_ast(subquery_ast)
-        sql += "\nWHERE #{qualified_name(select_query_ast.from_table_name, foreign_key)} IN (#{subquery_sql})"
-
-        # first_join.base_where_clauses holds conditions on the outer
-        # delete-target table (from_table_name), such as a polymorphic type
-        # column. They are not part of the subquery, so add them to the outer
-        # WHERE. This prevents deleting rows that belong to a different
-        # polymorphic type.
-        first_join.base_where_clauses.each do |where|
-          next unless where.is_a?(Exwiw::QueryAst::WhereClause)
-
-          sql += " AND #{compile_where_condition(where, select_query_ast.from_table_name)}"
-        end
-        sql += ";"
-
-        sql
-      end
-
       # @param count_only [Boolean] emit `SELECT COUNT(*)` instead of the
       #   projected columns (used by StreamingResult#size). Safe because exwiw's
       #   extraction queries have no DISTINCT/GROUP BY/LIMIT, so the count does
@@ -377,39 +319,6 @@ module Exwiw
         name
       end
 
-      # A WHERE condition for the DELETE statement.
-      #
-      # MySQL refuses a subquery that reads the table being deleted from
-      # ("You can't specify target table 'x' for update in FROM clause"), and a
-      # polymorphic multi-arm scope produces exactly that: each arm selects the
-      # join table's own primary key, so the delete's `pk IN (…)` reads the
-      # delete target. Wrapping the subquery in a derived table lifts the
-      # restriction — MySQL materializes the derived table before the DELETE
-      # runs, so the rows deleted are the ones the SELECT matched.
-      #
-      # Only that self-referencing shape is wrapped; every other subquery
-      # (a scope id-set projected from *another* table, the ids_field probe)
-      # compiles exactly as before.
-      private def compile_delete_where_condition(where_clause, table_name)
-        if where_clause.operator == :in_subquery && delete_target_self_reference?(where_clause.value, table_name)
-          key = qualified_name(table_name, where_clause.column_name)
-          return "#{key} IN (SELECT * FROM (#{compile_subquery(where_clause.value)}) AS exwiw_delete_src)"
-        end
-
-        compile_where_condition(where_clause, table_name)
-      end
-
-      private def delete_target_self_reference?(subquery, table_name)
-        case subquery
-        when Exwiw::QueryAst::SelectSubquery
-          Exwiw::QueryAst.reads_table?(subquery.query, table_name)
-        when Exwiw::QueryAst::UnionSubquery
-          subquery.queries.any? { |query| Exwiw::QueryAst.reads_table?(query, table_name) }
-        else
-          false
-        end
-      end
-
       private def compile_where_condition(where_clause, table_name)
         # Use as it is if it's a raw query
         return where_clause if where_clause.is_a?(String)

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -24,8 +24,8 @@ module Exwiw
       #     unchanged, so MongoDB and the other SQL adapters are untouched.
       #   - the streaming pass ties up the connection until fully drained. The
       #     Runner always drains it (write_inserts) before issuing any further
-      #     query (post_insert_sql / DELETE) on the same connection, so the
-      #     ordering invariant holds.
+      #     query (post_insert_sql) on the same connection, so the ordering
+      #     invariant holds.
       class StreamingResult
         include Enumerable
 
@@ -186,6 +186,13 @@ module Exwiw
         # EXTENSION that pg_dump emits alongside is likewise wrapped to swallow
         # undefined_object, so a skipped extension's trailing comment does not
         # abort the restore either.
+        # Triggers are wrapped the same way. They used to be stripped instead,
+        # because a `--table` dump emitted CREATE TRIGGER without the
+        # CREATE FUNCTION it referenced; a whole-database dump carries both, so a
+        # target can have the source's triggers instead of silently running none
+        # (mysqldump and sqlite_master always emitted theirs). They do fire while
+        # the `insert-*.sql` files are applied, so a load that must not fire them
+        # disables them for the session, as a mysql restore already had to.
         # Platform-managed extensions are stripped first: the wrapping passes below
         # rewrite the bare CREATE/COMMENT statements this removes.
         idempotent = strip_platform_managed_extensions(stdout)
@@ -197,7 +204,7 @@ module Exwiw
         idempotent = DdlPostprocessor.add_if_not_exists_to_create_table(idempotent)
         idempotent = DdlPostprocessor.add_if_not_exists_to_create_index(idempotent)
         idempotent = DdlPostprocessor.wrap_add_constraint_in_do_block(idempotent)
-        idempotent = DdlPostprocessor.strip_triggers(idempotent)
+        idempotent = DdlPostprocessor.wrap_create_trigger_in_do_block(idempotent)
 
         File.open(output_path, 'w') do |file|
           file.puts("-- Auto-generated by exwiw via pg_dump. Idempotent DDL for postgresql.")
@@ -266,6 +273,44 @@ module Exwiw
         lines.join("\n")
       end
 
+      def pre_insert_sql(_table)
+        suppress_triggers_sql
+      end
+
+      # Suppress trigger and foreign-key enforcement for the statements that
+      # follow. insert-000-schema.sql carries the source's triggers now, so an
+      # unguarded pass would fire every one of them per row it touches — on the
+      # insert side over rows that already carry the values those triggers
+      # produced on the source, and in an order that is only guaranteed to
+      # satisfy FKs once every insert-*.sql has been applied. `replica` mode
+      # turns off both user and system (RI) triggers, matching what MysqlAdapter
+      # does with FOREIGN_KEY_CHECKS.
+      #
+      # Setting the parameter needs superuser (or a grant on it), which a
+      # restore role may not have; the pass itself is still valid without it,
+      # so a failure is downgraded to a WARNING rather than aborting the file.
+      #
+      # Unlike MysqlAdapter — and unlike `pg_dump --disable-triggers`, which
+      # pairs each table's DISABLE TRIGGER ALL with an ENABLE — no counterpart
+      # reset is emitted, so the file is NOT self-contained in session state:
+      # sourcing these files into a session that goes on to do other work
+      # leaves that session in replica mode. `is_local = false` still bounds it
+      # to the connection, and every file re-arms the setting itself, so
+      # concatenating the files works either way — the reset would only buy
+      # tidiness, at the cost of emitting it unconditionally (post_insert_sql
+      # returns nil for a table with no serial PK) inside a second exception
+      # handler (a bare reset re-raises insufficient_privilege on the
+      # unprivileged path and would abort the file).
+      private def suppress_triggers_sql
+        <<~SQL.chomp
+          DO $exwiw$ BEGIN
+            PERFORM set_config('session_replication_role', 'replica', false);
+          EXCEPTION WHEN insufficient_privilege THEN
+            RAISE WARNING 'exwiw: could not disable triggers for the load (%): %', SQLSTATE, SQLERRM;
+          END $exwiw$;
+        SQL
+      end
+
       # Transcribe the FROM-side sequence cursor backing `table.primary_key`
       # onto the import target. Without this, importing into a clean DB leaves
       # the sequence at 1 while the inserted rows occupy higher IDs, so the
@@ -302,73 +347,6 @@ module Exwiw
         "SELECT pg_catalog.setval('#{escape_single_quote(seq_name)}', #{last_value}, #{is_called_sql});"
       end
 
-      def to_bulk_delete(select_query_ast, table)
-        raise NotImplementedError unless select_query_ast.is_a?(Exwiw::QueryAst::Select)
-
-        sql = "DELETE FROM #{quote_table_name(select_query_ast.from_table_name)}"
-
-        if select_query_ast.join_clauses.empty?
-          # Ignore filter option, because bulk delete is for cleaning before import,
-          # so it should delete all records to avoid foreign key violation & data consistancy.
-          compiled_where_conditions = select_query_ast.
-            where_clauses.
-            select { |where| where.is_a?(Exwiw::QueryAst::WhereClause) }.
-            map do |where|
-            compile_where_condition(where, select_query_ast.from_table_name)
-          end
-
-          if compiled_where_conditions.size > 0
-            sql += "\nWHERE "
-            sql += compiled_where_conditions.join(' AND ')
-          end
-          sql += ";"
-
-          return sql
-        end
-
-        subquery_ast = Exwiw::QueryAst::Select.new
-        first_join = select_query_ast.join_clauses.first.clone
-
-        subquery_ast.from(first_join.join_table_name)
-        primay_key_col = table.columns.find { |col| col.name == table.primary_key }
-        subquery_ast.select([primay_key_col])
-        select_query_ast.join_clauses[1..].each do |join|
-          subquery_ast.join(join)
-        end
-        first_join.where_clauses.each do |where|
-          # Ignore filter option, because bulk delete is for cleaning before import,
-          # so it should delete all records to avoid foreign key violation & data consistancy.
-          subquery_ast.where(where) if where.is_a?(Exwiw::QueryAst::WhereClause)
-        end
-
-        foreign_key = first_join.foreign_key
-        outer_table = select_query_ast.from_table_name
-        inner_table = first_join.join_table_name
-        inner_column = first_join.primary_key
-        cast_to = types_need_cast?(
-          column_pg_type(outer_table, foreign_key),
-          column_pg_type(inner_table, inner_column)
-        ) ? 'text' : nil
-        subquery_sql = compile_ast(subquery_ast, select_cast_to: cast_to)
-        outer_expr = qualified_name(outer_table, foreign_key)
-        outer_expr = "#{outer_expr}::text" if cast_to
-        sql += "\nWHERE #{outer_expr} IN (#{subquery_sql})"
-
-        # first_join.base_where_clauses holds conditions on the outer
-        # delete-target table (from_table_name), such as a polymorphic type
-        # column. They are not part of the subquery, so add them to the outer
-        # WHERE. This prevents deleting rows that belong to a different
-        # polymorphic type.
-        first_join.base_where_clauses.each do |where|
-          next unless where.is_a?(Exwiw::QueryAst::WhereClause)
-
-          sql += " AND #{compile_where_condition(where, select_query_ast.from_table_name)}"
-        end
-        sql += ";"
-
-        sql
-      end
-
       def compile_ast(query_ast, select_cast_to: nil)
         raise NotImplementedError unless query_ast.is_a?(Exwiw::QueryAst::Select)
 

data/lib/exwiw/adapter/sqlite_adapter.rb

@@ -137,64 +137,6 @@ module Exwiw
         end
       end
 
-      def to_bulk_delete(select_query_ast, table)
-        raise NotImplementedError unless select_query_ast.is_a?(Exwiw::QueryAst::Select)
-
-        sql = "DELETE FROM #{quote_table_name(select_query_ast.from_table_name)}"
-
-        if select_query_ast.join_clauses.empty?
-          # Ignore filter option, because bulk delete is for cleaning before import,
-          # so it should delete all records to avoid foreign key violation & data consistancy.
-          compiled_where_conditions = select_query_ast.
-            where_clauses.
-            select { |where| where.is_a?(Exwiw::QueryAst::WhereClause) }.
-            map do |where|
-            compile_where_condition(where, select_query_ast.from_table_name)
-          end
-
-          if compiled_where_conditions.size > 0
-            sql += "\nWHERE "
-            sql += compiled_where_conditions.join(' AND ')
-          end
-          sql += ";"
-
-          return sql
-        end
-
-        subquery_ast = Exwiw::QueryAst::Select.new
-        first_join = select_query_ast.join_clauses.first.clone
-
-        subquery_ast.from(first_join.join_table_name)
-        primay_key_col = table.columns.find { |col| col.name == table.primary_key }
-        subquery_ast.select([primay_key_col])
-        select_query_ast.join_clauses[1..].each do |join|
-          subquery_ast.join(join)
-        end
-        first_join.where_clauses.each do |where|
-          # Ignore filter option, because bulk delete is for cleaning before import,
-          # so it should delete all records to avoid foreign key violation & data consistancy.
-          subquery_ast.where(where) if where.is_a?(Exwiw::QueryAst::WhereClause)
-        end
-
-        foreign_key = first_join.foreign_key
-        subquery_sql = compile_ast(subquery_ast)
-        sql += "\nWHERE #{qualified_name(select_query_ast.from_table_name, foreign_key)} IN (#{subquery_sql})"
-
-        # first_join.base_where_clauses holds conditions on the outer
-        # delete-target table (from_table_name), such as a polymorphic type
-        # column. They are not part of the subquery, so add them to the outer
-        # WHERE. This prevents deleting rows that belong to a different
-        # polymorphic type.
-        first_join.base_where_clauses.each do |where|
-          next unless where.is_a?(Exwiw::QueryAst::WhereClause)
-
-          sql += " AND #{compile_where_condition(where, select_query_ast.from_table_name)}"
-        end
-        sql += ";"
-
-        sql
-      end
-
       # @param count_only [Boolean] emit `SELECT COUNT(*)` instead of the
       #   projected columns (used by StreamingResult#size). Safe because exwiw's
       #   extraction queries have no DISTINCT/GROUP BY/LIMIT, so the count does

data/lib/exwiw/adapter.rb

@@ -82,11 +82,6 @@ module Exwiw
       def dump_schema(ordered_tables, output_path)
       end
 
-      # Whether this adapter emits delete-NNN-*.sql files.
-      def supports_bulk_delete?
-        true
-      end
-
       # Whether the given config produces its own dump output and needs an
       # independent processing pass. SQL adapters always do; non-SQL adapters
       # may exclude e.g. embedded subdocument configs.
@@ -339,12 +334,6 @@ module Exwiw
       raise NotImplementedError
     end
 
-    # @params [Exwiw::QueryAst] select_query_ast
-    # @params [Exwiw::TableConfig] table
-    def to_bulk_delete(select_query_ast, table)
-      raise NotImplementedError
-    end
-
     def self.build(connection_config, logger)
       case normalize_name(connection_config.adapter)
       when 'sqlite'

data/lib/exwiw/cli.rb

@@ -39,7 +39,6 @@ module Exwiw
       schema_dir
       output_dir
       output_format
-      insert_only
       after_insert_hook
       log_level
       target_table
@@ -58,6 +57,11 @@ module Exwiw
     EXPLAIN_VERBOSITIES = %w[queryPlanner executionStats allPlansExecution].freeze
     DEFAULT_EXPLAIN_VERBOSITY = "queryPlanner"
 
+    # Keys that configured behavior that no longer exists (insert_only toggled
+    # the removed delete-*.sql generation). Accepted so a committed config file
+    # keeps working across the upgrade, warned about so it gets cleaned up.
+    OBSOLETE_CONFIG_KEYS = %w[insert_only].freeze
+
     # Database connection settings are environment-specific (and sometimes
     # secret-adjacent), so they must be passed via CLI/env, never the committed
     # config file. `adapter` is the one connection-ish key allowed in config.
@@ -66,7 +70,7 @@ module Exwiw
     # Keys that only make sense for `export`. They are skipped when merging config
     # for `explain` so a shared config file does not trip validate_explain_only!,
     # and for `schema`, which performs no export at all.
-    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format insert_only after_insert_hook parallel_workers].freeze
+    EXPORT_ONLY_CONFIG_KEYS = %w[output_dir output_format after_insert_hook parallel_workers].freeze
 
     def self.start(argv)
       new(argv).run
@@ -111,7 +115,6 @@ module Exwiw
       @ids_field = nil
       @scope_column = nil
       @output_format = nil
-      @insert_only = nil
       @after_insert_hook_path = nil
       @parallel_workers = nil
       @mongodb_query_timeout_ms = nil
@@ -160,7 +163,6 @@ module Exwiw
           schema_dir: @schema_dir,
           dump_target: dump_target,
           output_format: @output_format,
-          insert_only: @insert_only,
           after_insert_hook_path: @after_insert_hook_path,
           parallel_workers: @parallel_workers,
           cli_options: build_cli_options_hash,
@@ -333,7 +335,6 @@ module Exwiw
       if @subcommand == "export"
         @output_dir ||= "dump"
         @output_format ||= "insert"
-        @insert_only = @insert_only ? true : false
 
         valid_output_formats = ["insert", "copy"]
         unless valid_output_formats.include?(@output_format)
@@ -426,6 +427,10 @@ module Exwiw
           $stderr.puts "'#{key}' is a database connection setting and must be passed via the CLI/environment, not the config file (#{path})"
           exit 1
         end
+        if OBSOLETE_CONFIG_KEYS.include?(key)
+          $stderr.puts "warning: config key '#{key}' in #{path} is obsolete and ignored (exwiw no longer generates delete-*.sql files); remove it from the config file"
+          next
+        end
         unless ALLOWED_CONFIG_KEYS.include?(key)
           $stderr.puts "Unknown config key '#{key}' in #{path}. Allowed keys: #{ALLOWED_CONFIG_KEYS.join(', ')}"
           exit 1
@@ -443,7 +448,6 @@ module Exwiw
       @output_dir ||= expand_dir(config["output_dir"], base)
       @after_insert_hook_path ||= (File.expand_path(config["after_insert_hook"], base) if config["after_insert_hook"])
       @output_format ||= config["output_format"]
-      @insert_only = config["insert_only"] if @insert_only.nil? && config.key?("insert_only")
       @log_level ||= config["log_level"]&.to_sym
       @target_table_name ||= config["target_table"]
       @target_collection_name ||= config["target_collection"]
@@ -679,7 +683,6 @@ module Exwiw
       rejected = []
       rejected << "--output-dir" unless @output_dir.nil?
       rejected << "--output-format" unless @output_format.nil?
-      rejected << "--insert-only" unless @insert_only.nil?
       rejected << "--after-insert-hook" unless @after_insert_hook_path.nil?
       rejected << "--parallel-workers" unless @parallel_workers.nil?
 
@@ -744,7 +747,6 @@ module Exwiw
         ids_field: @ids_field,
         scope_column: @scope_column,
         output_format: @output_format,
-        insert_only: @insert_only,
         log_level: @log_level,
         after_insert_hook: @after_insert_hook_path,
       }.freeze
@@ -816,8 +818,10 @@ module Exwiw
         opts.on("--ids-field=[FIELD]", "Field on the target collection that --ids is matched against. Defaults to the primary key. (mongodb adapter only)") { |v| @ids_field = v }
         opts.on("--scope-column=[COLUMN]", "DEPRECATED. Filter every table by this shared global column (--ids are its values) instead of a single --target-table. SQL adapters only; mutually exclusive with --target-table. Prefer declaring a per-table `scope_column:` in the schema config and running with --target-table.") { |v| @scope_column = v }
         opts.on("--output-format=[FORMAT]", "Output format: insert (default) or copy (PostgreSQL only, export subcommand only)") { |v| @output_format = v }
-        opts.on("--insert-only", "Do not generate DELETE SQL files (export subcommand only)") { @insert_only = true }
-        opts.on("--after-insert-hook=PATH", "Path to a .rb or .sh post-processing hook executed after all insert/delete files are written (export subcommand only)") do |v|
+        opts.on("--insert-only", "DEPRECATED: ignored. exwiw no longer generates delete-*.sql files, so every export is insert-only.") do
+          $stderr.puts "warning: --insert-only is obsolete and ignored (exwiw no longer generates delete-*.sql files); remove the flag"
+        end
+        opts.on("--after-insert-hook=PATH", "Path to a .rb or .sh post-processing hook executed after all insert files are written (export subcommand only)") do |v|
           @after_insert_hook_path = File.expand_path(v)
         end
         opts.on("--parallel-workers=N", Integer, "Fork N workers for the MongoDB dump's parallel schedule (mongodb + export only; N>=2 enables it, default is serial). Output is byte-identical to serial; falls back to serial where fork is unavailable.") { |v| @parallel_workers = v }

data/lib/exwiw/ddl_postprocessor.rb

@@ -58,10 +58,50 @@ module Exwiw
       end
     end
 
-    # pg_dump --table includes triggers but not the referenced function
-    # definitions, causing UndefinedFunction errors on the target DB.
-    def strip_triggers(sql)
-      sql.gsub(/^[ \t]*CREATE\s+(?:OR\s+REPLACE\s+)?(?:CONSTRAINT\s+)?TRIGGER\b[^;]*;\r?\n?/i, "")
+    # One trigger as pg_dump writes it: its header comment block ($1) followed by
+    # the `CREATE TRIGGER` statement ($2).
+    #
+    #   --
+    #   -- Name: users set_timestamp; Type: TRIGGER; Schema: public; Owner: -
+    #   --
+    #
+    #   CREATE TRIGGER set_timestamp BEFORE UPDATE ON public.users ...;
+    #
+    # The header is what keeps the pass off a `CREATE TRIGGER` line inside a
+    # dollar-quoted function body — a function that installs a trigger itself,
+    # whose meaning a rewrite would change. pg_dump never writes a header block
+    # inside a body, so requiring one limits the match to top-level statements.
+    # (The stripping pass this replaced matched the bare form and deleted such a
+    # line, leaving a function that compiled but installed nothing.)
+    #
+    # The statement ends at the first semicolon outside a single-quoted string, so
+    # a `;` in a WHEN clause or an argument (`EXECUTE FUNCTION f('a;b')`) does not
+    # truncate it. A trigger definition has no body, so no dollar quoting either.
+    CREATE_TRIGGER_RE = /
+      (^--\r?\n
+       --[ \t]+Name:[^\n]*;[ \t]*Type:[ \t]*TRIGGER;[^\n]*\r?\n
+       --\r?\n
+       (?:[ \t]*\r?\n)*)
+      ([ \t]*CREATE\s+(?:OR\s+REPLACE\s+)?(?:CONSTRAINT\s+)?TRIGGER\b
+       (?:[^;']|'(?:[^']|'')*')*;)
+    /xi.freeze
+
+    # A bare `CREATE TRIGGER ...;` is not idempotent (a second restore raises
+    # `duplicate_object`, 42710), so each statement goes into a DO block that
+    # swallows that error, in the form #wrap_add_constraint_in_do_block uses; the
+    # header is left as written. `CREATE OR REPLACE TRIGGER` would be shorter but
+    # is PostgreSQL 14+ only and pg_dump never emits it.
+    def wrap_create_trigger_in_do_block(sql)
+      sql.gsub(CREATE_TRIGGER_RE) do
+        header = Regexp.last_match(1)
+        stmt = Regexp.last_match(2).strip
+        <<~SQL.chomp
+          #{header}DO $exwiw$ BEGIN
+            #{stmt}
+          EXCEPTION WHEN duplicate_object THEN NULL;
+          END $exwiw$;
+        SQL
+      end
     end
 
     # A user@host pair as mysqldump writes it. Each side is independently a

data/lib/exwiw/runner.rb

@@ -11,7 +11,6 @@ module Exwiw
       dump_target:,
       logger:,
       output_format: 'insert',
-      insert_only: false,
       after_insert_hook_path: nil,
       parallel_workers: nil,
       cli_options: {}
@@ -21,7 +20,6 @@ module Exwiw
       @schema_dir = schema_dir
       @dump_target = dump_target
       @output_format = output_format
-      @insert_only = insert_only
       @after_insert_hook_path = after_insert_hook_path
       @parallel_workers = parallel_workers
       @cli_options = cli_options
@@ -105,8 +103,8 @@ module Exwiw
 
           # `batch_scope` splits the extraction into one query per slice of the
           # scope's id set, streaming rows like any adapter result. `query_ast`
-          # stays the unbatched query — the DELETE file and the error message
-          # below describe that one.
+          # stays the unbatched query — the error message below describes that
+          # one.
           phase = "resolving the batch_scope id set"
           batched = BatchedExtraction.build(
             adapter: adapter,
@@ -184,21 +182,6 @@ module Exwiw
 
             @logger.info("  Generated INSERT statement for #{record_num} records (#{statement_count} statement(s)).")
           end
-
-          if adapter.supports_bulk_delete? && !@insert_only && !(table.respond_to?(:rails_managed?) && table.rails_managed?)
-            phase = "generating DELETE statement"
-            @logger.debug("  Generate DELETE statement...")
-            delete_sql = adapter.to_bulk_delete(query_ast, table)
-            if @logger.debug?
-              @logger.debug("  Generated DELETE statement:\n#{delete_sql}")
-            else
-              @logger.info("  Generated DELETE statement.")
-            end
-            delete_idx = (total_size - idx).to_s.rjust(3, '0')
-            File.open(File.join(@output_dir, "delete-#{delete_idx}-#{table_name}.#{adapter.output_extension}"), 'w') do |file|
-              file.puts(delete_sql)
-            end
-          end
         rescue => e
           @logger.error("Error while #{phase} for table '#{table_name}' (#{idx + 1}/#{total_size}): #{e.class}: #{e.message}")
           @logger.error("  Extraction query that produced the data being processed:")

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.9.24"
+  VERSION = "1.0.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.9.24
+  version: 1.0.0
 platform: ruby
 authors:
 - Shia