exwiw 0.3.2 → 0.3.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 83bb3c7245d9174e5d16e85e6de2a530b58ca445b570d692c5198682fc43b9ef
-  data.tar.gz: cee17f919dab949c82942eae146a1d6182909ab6ef595e8cfc7d4c3f744d1ede
+  metadata.gz: 3b68b4eed496cf67cbac408c7f1c233a881b9778b46f0f5a67d0ff50b9321e03
+  data.tar.gz: 521a8cf7d5af0dee4538e407d06d9605d37cb059f5023ee922e11807ff6d1b17
 SHA512:
-  metadata.gz: d655b0e5d55759932d7ab6646cb9ed9d0b699e6c96e6061863091eded7914c736b74c558e8a96d132ea94ef668376ae372a9eacf23f978d52540bf31c38c7c18
-  data.tar.gz: 5e068572f70d2b1869e5dffceba9f681d0b153c0947da3fd7b35d746e7f33bbc6c2d5fa25d23c9dfa0f190e14d3304ec6f2940aa19168749aa05b303c572ea10
+  metadata.gz: bdd88c3f13879d23a4ecc7c239f22db2c5c5ddbd3bb06b838ae570d48309ba4f2460a6260d9f848da71717dcf43cae0de3b593e3e6e99f43406d700b39e79a37
+  data.tar.gz: ac2e73630034663ef1ef299fd09925843c086776d10e5998d30758e079a4066f388bdb76672867acacbb57024a5a990336c46e5d8f21d42548862125d1004748

data/CHANGELOG.md

@@ -2,6 +2,26 @@
 
 ## [Unreleased]
 
+## [0.3.4] - 2026-05-31
+
+### Changed
+
+- **Breaking:** the table/collection-level config attribute `skip` is renamed to `ignore`. There is no alias — config files using `"skip": true` must be updated to `"ignore": true` (`exwiw:schema:generate` / `exwiw:mongoid:schema:generate` now emit `ignore`, e.g. for composite-primary-key tables). The library accessors are renamed accordingly (`TableConfig#skip` → `#ignore`, `MongodbCollectionConfig#skip` → `#ignore`).
+
+### Added
+
+- `columns` / `fields` and `belongs_tos` entries now accept optional `comment` (a free-form note) and `ignore: true`. An ignored column/field is excluded from the `SELECT` and generated `INSERT`; an ignored `belongs_to` is removed from dependency ordering and query building. These user-owned values are preserved across `exwiw:schema:generate` / `exwiw:mongoid:schema:generate` regenerations (the hand-edited value wins over the auto-generated config), like `replace_with`. The ignored entries are dropped at runtime right after the config is loaded from file, so the JSON on disk keeps them. Applies to both the SQL `TableConfig` and the MongoDB `MongodbCollectionConfig`.
+
+### Fixed
+
+fixed HABTM relationship bug https://github.com/heyinc/exwiw/pull/56
+
+## [0.3.3] - 2026-05-31
+
+### Changed
+
+- `export` now empties `--output-dir` before writing, so a run never mixes files from a previous export. When running interactively (stdin is a tty) and the dir already has contents, exwiw asks for confirmation before removing them; in non-interactive contexts (CI, pipes) it proceeds without prompting.
+
 ## [0.3.2] - 2026-05-31
 
 ### Changed

data/README.md

@@ -42,22 +42,18 @@ gem install exwiw
 - sqlite
 - mongodb (experimental, see [MongoDB notes](#mongodb-notes))
 
-Adapter names are driver-agnostic — they name the database, not the Ruby driver
-or Rails ActiveRecord adapter. For backward compatibility and to absorb the Rails
-adapter spelling, the following aliases are also accepted (case-insensitive):
-
-| `--adapter` value | Aliases accepted |
-| --- | --- |
-| `mysql` | `mysql2` |
-| `sqlite` | `sqlite3` |
-
-So `--adapter=mysql2` and `--adapter=mysql` both select the same MySQL adapter.
-
 For MySQL, exwiw connects through whichever of the `mysql2` or `trilogy` gem is
 available (preferring `mysql2`), so an app on either driver works without any
 extra setup. There is no separate `trilogy` adapter name — pass `--adapter=mysql`
 either way.
 
+Set `EXWIW_MYSQL_DRIVER=trilogy` (or `mysql2`) to force a specific driver. This
+is useful when the `mysql2` gem is linked against a `libmysqlclient` that can no
+longer load the server's auth plugin — e.g. a MySQL 9.x client drops the
+`mysql_native_password` plugin and raises `Authentication plugin
+'mysql_native_password' cannot be loaded` on connect. The pure-Ruby `trilogy`
+driver implements that auth handshake itself and sidesteps the issue.
+
 ## Usage
 
 exwiw has two subcommands:
@@ -101,6 +97,8 @@ exwiw \
 
 This command will generate sql files in the `dump` directory.
 
+The output dir is emptied before each export so it never mixes files from a previous run (defaulting to `dump/` when `--output-dir` is omitted). When run interactively (stdin is a tty) and the dir already contains files, exwiw asks for confirmation before removing them; in non-interactive contexts (CI, pipes) it proceeds without prompting.
+
 - `dump/insert-000-schema.sql` — idempotent `CREATE TABLE IF NOT EXISTS ...` for every table in scope. Apply this first to provision an empty database.
 - `dump/insert-{idx}-{table_name}.sql`
 - `dump/delete-{idx}-{table_name}.sql`
@@ -108,7 +106,7 @@ This command will generate sql files in the `dump` directory.
 idx means the order of the dump. bigger idx might depend on smaller idx,
 so you should import the dump in order.
 
-`insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`.
+`insert-000-schema.sql` is generated by shelling out to the database client tools (`mysqldump` for `mysql`, `pg_dump` for `postgresql`, and the sqlite3 driver for `sqlite`), so the corresponding client must be available on PATH when running exwiw. For `mysql`, set `EXWIW_MYSQLDUMP` to point at a specific `mysqldump` binary when the one on PATH is incompatible with the server (e.g. a MySQL 9.x `mysqldump` cannot load `mysql_native_password` against a server still using that auth plugin — `EXWIW_MYSQLDUMP=/path/to/mysql@8.0/bin/mysqldump`). The output is post-processed to make it idempotent: `CREATE TABLE IF NOT EXISTS`, `CREATE INDEX IF NOT EXISTS` (where the engine supports it), and PostgreSQL's `ALTER TABLE ... ADD CONSTRAINT` statements are wrapped in `DO $$ ... EXCEPTION WHEN duplicate_object`.
 
 you need to delete the records before importing the dump,
 `delete-{idx}-{table_name}.sql` will help you to do that.
@@ -183,7 +181,7 @@ It is a distinct task and class (`Exwiw::MongoidSchemaGenerator`) from the Activ
 
 Models in an inheritance hierarchy whose subclasses share the base's collection (Mongoid STI, distinguished by the auto-added `_type` discriminator) collapse into a single config: the generator discovers the subclasses via `descendants` (Mongoid registers only the base class in `Mongoid.models`) and unions every class's `fields` and `belongs_tos` into the collection config, so subclass-only fields and associations are not lost.
 
-Regeneration preserves hand-edited `replace_with`, `filter`, `skip`, and `bulk_insert_chunk_size` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [MongoDB notes](#mongodb-notes)). Polymorphic `belongs_to` is not yet expanded by this task.
+Regeneration preserves hand-edited `replace_with`, `filter`, `ignore`, and `bulk_insert_chunk_size` values, like the ActiveRecord generator. Indexes are not written to the config — they are introspected from the live database at dump time (see [MongoDB notes](#mongodb-notes)). Polymorphic `belongs_to` is not yet expanded by this task.
 
 ### Configuration
 
@@ -258,15 +256,15 @@ A non-zero exit code from the shell hook aborts exwiw.
 
 Note: Ruby hooks are evaluated via `instance_eval` inside the exwiw process — only pass paths you trust.
 
-### Skip a table
+### Ignore a table
 
-Set `"skip": true` on a table's config JSON to exclude it from data extraction. The table's DDL is still emitted into `insert-000-schema.{sql,js}` so the schema stays consistent, but no `insert-*` / `delete-*` files are generated for it and the table is never queried.
+Set `"ignore": true` on a table's config JSON to exclude it from data extraction. The table's DDL is still emitted into `insert-000-schema.{sql,js}` so the schema stays consistent, but no `insert-*` / `delete-*` files are generated for it and the table is never queried.
 
 ```json
 {
   "name": "audit_logs",
   "primary_key": "id",
-  "skip": true,
+  "ignore": true,
   "belongs_tos": [],
   "columns": [{ "name": "id" }]
 }
@@ -274,9 +272,33 @@ Set `"skip": true` on a table's config JSON to exclude it from data extraction.
 
 Constraints:
 
-- If another non-skipped table has a `belongs_to` entry pointing at a skipped table, exwiw raises `ArgumentError` on load. Remove the `belongs_to` entry on the referencing table, or unset `skip` on the referenced table.
-- Specifying a skipped table as `--target-table` raises `ArgumentError`.
-- `skip: true` is preserved by `exwiw:schema:generate` regenerations (the receiver value wins over the auto-generated config).
+- If another non-ignored table has a `belongs_to` entry pointing at an ignored table, exwiw raises `ArgumentError` on load. Remove the `belongs_to` entry on the referencing table, or unset `ignore` on the referenced table.
+- Specifying an ignored table as `--target-table` raises `ArgumentError`.
+- `ignore: true` is preserved by `exwiw:schema:generate` regenerations (the receiver value wins over the auto-generated config).
+
+### Ignore / annotate a column or `belongs_to`
+
+Individual `columns` (SQL) / `fields` (MongoDB) and `belongs_tos` entries accept two optional, **user-owned** keys:
+
+- `comment` — a free-form note. Purely informational; exwiw never reads it.
+- `ignore: true` — drops that entry from extraction. An ignored column/field is excluded from the `SELECT` and the generated `INSERT` (the column still exists in the target schema, since the DDL comes from the source database — exwiw just does not copy its data). An ignored `belongs_to` is removed from dependency ordering and query building, so the relation is not traversed.
+
+```json
+{
+  "name": "users",
+  "primary_key": "id",
+  "belongs_tos": [
+    { "table_name": "companies", "foreign_key": "company_id" },
+    { "table_name": "audit_logs", "foreign_key": "log_id", "ignore": true, "comment": "huge table, not needed for this export" }
+  ],
+  "columns": [
+    { "name": "id" },
+    { "name": "secret_token", "ignore": true, "comment": "do not copy credentials" }
+  ]
+}
+```
+
+The ignored entries are removed only at runtime, right after the config is loaded from file; the JSON on disk keeps them. Both `comment` and `ignore` are **preserved across `exwiw:schema:generate` / `exwiw:mongoid:schema:generate` regenerations** (the hand-edited value wins over the auto-generated config), just like `replace_with`. This applies to the MongoDB `MongodbCollectionConfig` (`fields` / `belongs_tos`) as well.
 
 ### Polymorphic `belongs_to`
 
@@ -352,20 +374,20 @@ Constraints:
 
 ### Composite primary keys (unsupported)
 
-exwiw does not yet support tables with a composite primary key. When `exwiw:schema:generate` encounters a model whose `primary_key` is an array, it still emits a config entry so the table is not silently dropped, but marks it `skip: true`, tags it `type: "unsupported_composite_primary_key"`, and records the key columns in a `comment`:
+exwiw does not yet support tables with a composite primary key. When `exwiw:schema:generate` encounters a model whose `primary_key` is an array, it still emits a config entry so the table is not silently dropped, but marks it `ignore: true`, tags it `type: "unsupported_composite_primary_key"`, and records the key columns in a `comment`:
 
 ```json
 {
   "name": "composite_pk_records",
   "type": "unsupported_composite_primary_key",
-  "skip": true,
+  "ignore": true,
   "comment": "exwiw does not support composite primary keys (organization_id, location_id); data extraction is skipped.",
   "belongs_tos": [],
   "columns": [{ "name": "organization_id" }, { "name": "location_id" }, { "name": "name" }]
 }
 ```
 
-Unlike rails-managed entries, `columns` and `belongs_tos` are retained so the entry is ready to wire up once composite-key support lands. The `type` is purely a marker — `skip: true` is what actually excludes the table from extraction, so removing `skip` (and supplying a workable `primary_key`) lets you opt the table back in manually.
+Unlike rails-managed entries, `columns` and `belongs_tos` are retained so the entry is ready to wire up once composite-key support lands. The `type` is purely a marker — `ignore: true` is what actually excludes the table from extraction, so removing `ignore` (and supplying a workable `primary_key`) lets you opt the table back in manually.
 
 ### Bulk insert chunk size
 

data/lib/exwiw/adapter/mysql_adapter.rb

@@ -34,8 +34,16 @@ module Exwiw
           return
         end
 
+        # The mysqldump binary is invoked directly (not via the mysql2/trilogy
+        # driver), so point EXWIW_MYSQLDUMP at a specific binary when the one on
+        # PATH is incompatible with the server — e.g. a MySQL 9.x client whose
+        # mysqldump cannot load `mysql_native_password` ("plugin ... cannot be
+        # loaded", exit 2) against a server still using that auth plugin.
+        mysqldump_bin = ENV['EXWIW_MYSQLDUMP']
+        mysqldump_bin = 'mysqldump' if mysqldump_bin.nil? || mysqldump_bin.empty?
+
         cmd = [
-          'mysqldump',
+          mysqldump_bin,
           "--host=#{@connection_config.host}",
           "--port=#{@connection_config.port}",
           "--user=#{@connection_config.user}",
@@ -58,11 +66,18 @@ module Exwiw
         ]
         env = { 'MYSQL_PWD' => @connection_config.password.to_s }
 
-        @logger.debug("  Running mysqldump for #{table_names.size} table(s)...")
-        stdout, stderr, status = Open3.capture3(env, *cmd)
+        @logger.debug("  Running #{mysqldump_bin} for #{table_names.size} table(s)...")
+        stdout, stderr, status =
+          begin
+            Open3.capture3(env, *cmd)
+          rescue Errno::ENOENT
+            raise "Failed to run `#{mysqldump_bin}`. Ensure the mysql client is installed and on PATH, " \
+                  "or set EXWIW_MYSQLDUMP to a mysqldump binary."
+          end
         unless status.success?
           if stderr.include?('command not found') || stderr.empty?
-            raise "Failed to run `mysqldump`. Ensure the mysql client is installed and on PATH. stderr: #{stderr}"
+            raise "Failed to run `#{mysqldump_bin}`. Ensure the mysql client is installed and on PATH, " \
+                  "or set EXWIW_MYSQLDUMP to a mysqldump binary. stderr: #{stderr}"
           end
           raise "mysqldump failed (exit #{status.exitstatus}): #{stderr}"
         end

data/lib/exwiw/adapter/mysql_client.rb

@@ -18,9 +18,27 @@ module Exwiw
       # Immutable value object: a query's column names and its rows.
       Result = Data.define(:fields, :rows)
 
+      DRIVERS = [:mysql2, :trilogy].freeze
+
       # Pick the available driver, preferring mysql2 (exwiw's historical default).
       # require returns false when already loaded, so this is safe to call repeatedly.
+      #
+      # Set EXWIW_MYSQL_DRIVER=trilogy to force the pure-Ruby trilogy driver. This
+      # is useful when the mysql2 gem is linked against a libmysqlclient that can
+      # no longer load the server's auth plugin (e.g. MySQL 9.x client dropped the
+      # `mysql_native_password` plugin .so, raising "Authentication plugin
+      # 'mysql_native_password' cannot be loaded" on connect).
       def self.detect_driver
+        forced = ENV['EXWIW_MYSQL_DRIVER']
+        if forced && !forced.empty?
+          sym = forced.to_sym
+          unless DRIVERS.include?(sym)
+            raise ArgumentError,
+                  "EXWIW_MYSQL_DRIVER must be one of #{DRIVERS.join(', ')}, got #{forced.inspect}."
+          end
+          return sym
+        end
+
         require 'mysql2'
         :mysql2
       rescue LoadError

data/lib/exwiw/belongs_to.rb

@@ -12,6 +12,12 @@ module Exwiw
     # non-polymorphic belongs_to.
     attribute :foreign_type, optional(String), skip_serializing_if_nil: true
     attribute :type_value, optional(String), skip_serializing_if_nil: true
+    # User-owned fields. The schema generators never emit them, but a user can
+    # add them by hand and they survive regeneration (see TableConfig#merge /
+    # MongodbCollectionConfig#merge). `ignore:true` drops the relation from
+    # extraction once the config is loaded (see #reject_ignored_members!).
+    attribute :comment, optional(String), skip_serializing_if_nil: true
+    attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))
@@ -21,6 +27,13 @@ module Exwiw
       !foreign_type.nil?
     end
 
+    # Structural identity used to match a freshly generated belongs_to against a
+    # user-maintained one during merge. `comment`/`ignore` are user-owned and so
+    # are intentionally excluded.
+    def identity
+      [table_name, foreign_key, foreign_type, type_value]
+    end
+
     def to_hash
       super.compact
     end

data/lib/exwiw/cli.rb

@@ -76,6 +76,7 @@ module Exwiw
 
       case @subcommand
       when "export"
+        confirm_output_dir_clear!
         Runner.new(
           connection_config: connection_config,
           output_dir: @output_dir,
@@ -270,6 +271,28 @@ module Exwiw
       end
     end
 
+    # The export clears @output_dir before writing (see Runner#clean_output_dir!).
+    # That is destructive, so when running interactively (stdin is a tty) ask for
+    # confirmation first. In non-interactive contexts (CI, pipes) we proceed
+    # without prompting. Only prompt when there is actually something to delete.
+    private def confirm_output_dir_clear!
+      return unless $stdin.tty?
+      return unless Dir.exist?(@output_dir)
+
+      entries = Dir.each_child(@output_dir).to_a
+      return if entries.empty?
+
+      $stderr.puts "All contents of the output dir will be removed before export:"
+      $stderr.puts "  #{@output_dir} (#{entries.size} entr#{entries.size == 1 ? 'y' : 'ies'})"
+      $stderr.print "Continue? [y/N]: "
+
+      answer = $stdin.gets&.strip&.downcase
+      unless answer == "y" || answer == "yes"
+        $stderr.puts "Aborted."
+        exit 1
+      end
+    end
+
     private def build_cli_options_hash
       {
         database_host: @database_host,
@@ -322,7 +345,7 @@ module Exwiw
         opts.on("-h", "--host=HOST", "Target database host") { |v| @database_host = v }
         opts.on("-p", "--port=PORT", "Target database port") { |v| @database_port = v }
         opts.on("-u", "--user=USERNAME", "Target database user") { |v| @database_user = v }
-        opts.on("-o", "--output-dir=[DUMP_DIR_PATH]", "Output file path. default is dump/ (export subcommand only)") do |v|
+        opts.on("-o", "--output-dir=[DUMP_DIR_PATH]", "Output file path. default is dump/. Its contents are emptied before each export (export subcommand only)") do |v|
           v = v.end_with?("/") ? v[0..-2] : v
           @output_dir = File.expand_path(v)
         end

data/lib/exwiw/determine_table_processing_order.rb

@@ -24,6 +24,10 @@ module Exwiw
           not_resolved_names.empty?
         end
 
+        if tables_with_no_dependencies.empty?
+          raise ArgumentError, build_cycle_error_message(table_by_name, ordered_table_names)
+        end
+
         tables_with_no_dependencies.each do |table|
           ordered_table_names << table.name
           table_by_name.delete(table.name)
@@ -38,5 +42,25 @@ module Exwiw
         acc << relation.table_name
       end
     end
+
+    # When no table can be resolved but some remain, the belongs_to graph
+    # contains a cycle (e.g. A belongs_to B and B belongs_to A). A topological
+    # order cannot exist, so report the offending tables instead of looping
+    # forever.
+    private_class_method def cycle_diagnostics(table_by_name, ordered_table_names)
+      table_by_name.values.map do |table|
+        unresolved = (compute_table_dependencies(table) - ordered_table_names - [table.name]).uniq
+        "  #{table.name} -> #{unresolved.join(', ')}"
+      end
+    end
+
+    private_class_method def build_cycle_error_message(table_by_name, ordered_table_names)
+      "Circular belongs_to dependency detected among tables: " \
+        "#{table_by_name.keys.sort.join(', ')}. " \
+        "A processing order cannot be determined. " \
+        "Remove one of the belongs_to entries forming the cycle.\n" \
+        "Unresolved dependencies:\n" \
+        "#{cycle_diagnostics(table_by_name, ordered_table_names).join("\n")}"
+    end
   end
 end

data/lib/exwiw/explain_runner.rb

@@ -19,7 +19,7 @@ module Exwiw
     def run
       adapter = Adapter.build(@connection_config, @logger)
       configs = load_table_config(adapter.class.table_config_class)
-      validate_skipped(configs)
+      validate_ignored(configs)
 
       table_by_name = configs.each_with_object({}) { |config, hash| hash[config.name] = config }
 
@@ -32,8 +32,8 @@ module Exwiw
       total_size = ordered_table_names.size
       ordered_table_names.each_with_index do |table_name, idx|
         table = table_by_name.fetch(table_name)
-        if table.skip
-          @logger.debug("Skipping explain for '#{table_name}' (skip:true)")
+        if table.ignore
+          @logger.debug("Skipping explain for '#{table_name}' (ignore:true)")
           next
         end
 
@@ -55,30 +55,30 @@ module Exwiw
     private def load_table_config(klass)
       Dir[File.join(@config_dir, "*.json")].map do |file|
         json = JSON.parse(File.read(file))
-        klass.from(json)
+        klass.from(json).reject_ignored_members!
       end
     end
 
-    private def validate_skipped(configs)
-      skipped_names = configs.select { |c| c.skip }.map(&:name).to_set
-      return if skipped_names.empty?
+    private def validate_ignored(configs)
+      ignored_names = configs.select { |c| c.ignore }.map(&:name).to_set
+      return if ignored_names.empty?
 
       configs.each do |config|
-        next if config.skip
+        next if config.ignore
         next unless config.respond_to?(:belongs_tos)
 
-        dangling = config.belongs_tos.select { |rel| skipped_names.include?(rel.table_name) }
+        dangling = config.belongs_tos.select { |rel| ignored_names.include?(rel.table_name) }
         next if dangling.empty?
 
         raise ArgumentError,
-              "Table '#{config.name}' has belongs_to references to skipped table(s): " \
+              "Table '#{config.name}' has belongs_to references to ignored table(s): " \
               "#{dangling.map(&:table_name).join(', ')}. " \
-              "Remove the belongs_to entries or unset `skip` on the referenced table."
+              "Remove the belongs_to entries or unset `ignore` on the referenced table."
       end
 
-      if @dump_target.table_name && skipped_names.include?(@dump_target.table_name)
+      if @dump_target.table_name && ignored_names.include?(@dump_target.table_name)
         raise ArgumentError,
-              "--target-table '#{@dump_target.table_name}' is marked skip:true and cannot be used as a dump target."
+              "--target-table '#{@dump_target.table_name}' is marked ignore:true and cannot be used as a dump target."
       end
     end
   end

data/lib/exwiw/mongodb_collection_config.rb

@@ -13,7 +13,7 @@ module Exwiw
     attribute :belongs_tos, array(BelongsTo)
     attribute :fields, array(MongodbField)
     attribute :bulk_insert_chunk_size, optional(Integer), skip_serializing_if_nil: true
-    attribute :skip, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
 
     # Marks this config as physically embedded inside another collection's
     # documents. When set, this config is not processed as a standalone dump
@@ -35,12 +35,21 @@ module Exwiw
       !embedded_in.nil?
     end
 
+    # Drop the belongs_tos/fields flagged `ignore:true` so they are excluded from
+    # extraction. The config files on disk keep these entries; this is applied to
+    # the runtime config right after it is loaded (see Runner#load_table_config).
+    def reject_ignored_members!
+      self.belongs_tos = belongs_tos.reject(&:ignore)
+      self.fields = fields.reject(&:ignore)
+      self
+    end
+
     # Merge an auto-generated config (`passed`) into this user-maintained one so
     # that `MongoidSchemaGenerator` regenerations preserve hand-edited values.
     #
     # - structural facts come from the freshly generated config: primary_key,
     #   belongs_tos, embedded_in.
-    # - user customizations are kept from the receiver: filter, skip,
+    # - user customizations are kept from the receiver: filter, ignore,
     #   bulk_insert_chunk_size, and each field's `replace_with` masking rule.
     # - generated fields drive the field list (so added/removed fields track the
     #   model), but a matching receiver field wins to retain its masking.
@@ -51,18 +60,34 @@ module Exwiw
         merged.name = name
         merged.primary_key = passed.primary_key
         merged.filter = filter
-        merged.belongs_tos = passed.belongs_tos
         merged.bulk_insert_chunk_size = bulk_insert_chunk_size
-        merged.skip = skip
+        merged.ignore = ignore
         merged.embedded_in = passed.embedded_in
 
+        # Structural facts of each belongs_to come from the freshly generated
+        # config, but the user-owned `comment`/`ignore` carry over when the same
+        # relation still exists.
+        receiver_belongs_to_by_identity = belongs_tos.each_with_object({}) { |bt, h| h[bt.identity] = bt }
+        merged.belongs_tos = passed.belongs_tos.map do |pbt|
+          receiver_bt = receiver_belongs_to_by_identity[pbt.identity]
+          if receiver_bt
+            pbt.comment = receiver_bt.comment if receiver_bt.comment
+            pbt.ignore = receiver_bt.ignore unless receiver_bt.ignore.nil?
+          end
+          pbt
+        end
+
         # Take each field from the freshly generated config (so structural facts
         # like `mongoid_field_name` track the model) but carry over the user's
-        # hand-edited `replace_with` masking when the field still exists.
+        # hand-edited `replace_with`/`comment`/`ignore` when the field still exists.
         receiver_field_by_name = fields.each_with_object({}) { |f, h| h[f.name] = f }
         merged.fields = passed.fields.map do |pf|
           receiver = receiver_field_by_name[pf.name]
-          pf.replace_with = receiver.replace_with if receiver&.replace_with
+          if receiver
+            pf.replace_with = receiver.replace_with if receiver.replace_with
+            pf.comment = receiver.comment if receiver.comment
+            pf.ignore = receiver.ignore unless receiver.ignore.nil?
+          end
           pf
         end
       end

data/lib/exwiw/mongodb_field.rb

@@ -11,6 +11,11 @@ module Exwiw
     # masks/projects by `name` (the storage key) — but surfacing the accessor
     # keeps an otherwise cryptic short key understandable in the config.
     attribute :mongoid_field_name, optional(String), skip_serializing_if_nil: true
+    # User-owned fields preserved across schema regeneration (see
+    # MongodbCollectionConfig#merge). `ignore:true` drops the field from extraction
+    # once the config is loaded (see MongodbCollectionConfig#reject_ignored_members!).
+    attribute :comment, optional(String), skip_serializing_if_nil: true
+    attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))

data/lib/exwiw/runner.rb

@@ -30,7 +30,7 @@ module Exwiw
       adapter = Adapter.build(@connection_config, @logger)
       configs = load_table_config(adapter.class.table_config_class)
 
-      validate_skipped(configs)
+      validate_ignored(configs)
       validate_rails_managed_target!(configs)
 
       table_by_name = configs.each_with_object({}) { |config, hash| hash[config.name] = config }
@@ -41,9 +41,7 @@ module Exwiw
       @logger.info("Determining table processing order...")
       ordered_table_names = DetermineTableProcessingOrder.run(configs.select { |c| adapter.dumpable?(c) })
 
-      if !Dir.exist?(@output_dir)
-        FileUtils.mkdir_p(@output_dir)
-      end
+      clean_output_dir!
 
       ordered_tables = ordered_table_names.map { |n| table_by_name.fetch(n) }
       schema_path = File.join(@output_dir, "insert-000-schema.#{adapter.schema_output_extension}")
@@ -54,8 +52,8 @@ module Exwiw
       ordered_table_names.each_with_index do |table_name, idx|
         table = table_by_name.fetch(table_name)
 
-        if table.skip
-          @logger.info("Skipping data extraction for '#{table_name}' (skip:true)")
+        if table.ignore
+          @logger.info("Skipping data extraction for '#{table_name}' (ignore:true)")
           next
         end
 
@@ -123,36 +121,57 @@ module Exwiw
       end
     end
 
+    # Empty the output dir before writing so each export starts from a clean
+    # slate and never mixes files from a previous run. Remove the contents
+    # (including dotfiles) rather than the dir itself, preserving the dir's own
+    # permissions/inode. The CLI is responsible for confirming this with the
+    # user when running interactively.
+    private def clean_output_dir!
+      if Dir.exist?(@output_dir)
+        entries = Dir.each_child(@output_dir).to_a
+        unless entries.empty?
+          @logger.info("Cleaning output dir #{@output_dir} (#{entries.size} entr#{entries.size == 1 ? 'y' : 'ies'})...")
+          entries.each { |entry| FileUtils.rm_rf(File.join(@output_dir, entry)) }
+        end
+      else
+        FileUtils.mkdir_p(@output_dir)
+      end
+    end
+
     private def load_table_config(klass)
       Dir[File.join(@config_dir, "*.json")].map do |file|
         json = JSON.parse(File.read(file))
-        klass.from(json)
+        # Drop belongs_tos/columns(fields) flagged ignore:true so they are not
+        # considered during extraction. Done here (after loading from file)
+        # rather than in `.from` so the schema generators keep the full config
+        # and can preserve the ignored entries on regeneration.
+        klass.from(json).reject_ignored_members!
       end
     end
 
-    private def validate_skipped(configs)
-      skipped_names = configs.select { |c| c.skip }.map(&:name).to_set
-      return if skipped_names.empty?
+    private def validate_ignored(configs)
+      ignored_names = configs.select { |c| c.ignore }.map(&:name).to_set
+      return if ignored_names.empty?
 
       configs.each do |config|
-        next if config.skip
+        next if config.ignore
         next unless config.respond_to?(:belongs_tos)
 
-        dangling = config.belongs_tos.select { |rel| skipped_names.include?(rel.table_name) }
+        dangling = config.belongs_tos.select { |rel| ignored_names.include?(rel.table_name) }
         next if dangling.empty?
 
         raise ArgumentError,
-              "Table '#{config.name}' has belongs_to references to skipped table(s): " \
+              "Table '#{config.name}' has belongs_to references to ignored table(s): " \
               "#{dangling.map(&:table_name).join(', ')}. " \
-              "Remove the belongs_to entries or unset `skip` on the referenced table."
+              "Remove the belongs_to entries or unset `ignore` on the referenced table."
       end
 
-      if @dump_target.table_name && skipped_names.include?(@dump_target.table_name)
+      if @dump_target.table_name && ignored_names.include?(@dump_target.table_name)
         raise ArgumentError,
-              "--target-table '#{@dump_target.table_name}' is marked skip:true and cannot be used as a dump target."
+              "--target-table '#{@dump_target.table_name}' is marked ignore:true and cannot be used as a dump target."
       end
 
-      skipped_names.each { |n| @logger.info("Table '#{n}' is marked skip:true (schema will be included, data extraction skipped)") }
+      ignored_names.each { |n| @logger.info("Table '#{n}' is marked ignore:true (schema will be included, data extraction skipped)") }
     end
 
     private def validate_rails_managed_target!(configs)

data/lib/exwiw/schema_generator.rb

@@ -80,15 +80,15 @@ module Exwiw
 
         # Tables with a composite primary key (`representative.primary_key` is an
         # Array) are not supported yet. Emit them with `primary_key` omitted,
-        # `skip: true`, and a `type` that marks them as unsupported — the `type`
+        # `ignore: true`, and a `type` that marks them as unsupported — the `type`
         # acts as a signpost for adding support later. The config file itself is
-        # still generated so a user can manually remove `skip` and wire it up when
-        # needed.
+        # still generated so a user can manually remove `ignore` and wire it up
+        # when needed.
         if primary_key.is_a?(Array)
           TableConfig.from_symbol_keys(
             name: table_name,
             type: TableConfig::UNSUPPORTED_COMPOSITE_PRIMARY_KEY,
-            skip: true,
+            ignore: true,
             comment: "exwiw does not support composite primary keys " \
                      "(#{primary_key.join(', ')}); data extraction is skipped.",
             belongs_tos: aggregate_belongs_tos(model_group),
@@ -147,7 +147,7 @@ module Exwiw
     end
 
     private def aggregate_belongs_tos(models)
-      belongs_to_assocs = models.flat_map { |m| m.reflect_on_all_associations(:belongs_to) }
+      belongs_to_assocs = models.flat_map { |m| belongs_to_associations_for(m) }
 
       non_polymorphic = belongs_to_assocs
         .reject(&:polymorphic?)
@@ -174,6 +174,27 @@ module Exwiw
       (non_polymorphic + polymorphic).uniq
     end
 
+    # `belongs_to` reflections for a model, with the synthetic HABTM left-side
+    # association removed.
+    #
+    # Rails backs every `has_and_belongs_to_many` with an anonymous join model
+    # (`HABTM_*`, a concrete `ActiveRecord::Base` descendant whose table is the
+    # join table). That join model declares two belongs_tos: the "right side"
+    # (named after the association, e.g. `belongs_to :tags` -> `tag_id`), which
+    # is correct, and a synthetic `belongs_to :left_side`. The left-side
+    # association is built with `anonymous_class:` and no `foreign_key:`, so AR
+    # derives its foreign key from the reflection name -> `left_side_id`, a
+    # column that does not exist in the join table. Dropping it leaves the join
+    # table with only its genuine foreign keys; the right-side reflections of
+    # the two HABTM_* models together still supply both (`post_id` + `tag_id`).
+    private def belongs_to_associations_for(model)
+      assocs = model.reflect_on_all_associations(:belongs_to)
+      return assocs unless model.respond_to?(:left_reflection)
+
+      left = model.left_reflection
+      assocs.reject { |assoc| assoc.equal?(left) }
+    end
+
     # Enumerate the concrete models that can be targets of the polymorphic
     # association `association_name`, by looking them up from every model's
     # `has_many` / `has_one` `as:` option. The order of `concrete_models` depends

data/lib/exwiw/table_column.rb

@@ -7,6 +7,11 @@ module Exwiw
     attribute :name, String
     attribute :replace_with, optional(String), skip_serializing_if_nil: true
     attribute :raw_sql, optional(String), skip_serializing_if_nil: true
+    # User-owned fields preserved across schema regeneration (see
+    # TableConfig#merge). `ignore:true` drops the column from extraction (SELECT /
+    # INSERT) once the config is loaded (see TableConfig#reject_ignored_members!).
+    attribute :comment, optional(String), skip_serializing_if_nil: true
+    attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(hash.transform_keys(&:to_s))

data/lib/exwiw/table_config.rb

@@ -12,7 +12,7 @@ module Exwiw
     ].freeze
 
     # type marking a table with a composite primary key, which exwiw does not
-    # support yet. schema:generate attaches it together with skip:true. Unlike
+    # support yet. schema:generate attaches it together with ignore:true. Unlike
     # rails-managed tables, columns/belongs_tos are retained so it can serve as a
     # signpost for adding support later.
     UNSUPPORTED_COMPOSITE_PRIMARY_KEY = "unsupported_composite_primary_key"
@@ -25,7 +25,7 @@ module Exwiw
     attribute :belongs_tos, array(BelongsTo), default: []
     attribute :columns, array(TableColumn), default: []
     attribute :bulk_insert_chunk_size, optional(Integer), skip_serializing_if_nil: true
-    attribute :skip, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
+    attribute :ignore, Serdes::OptionalType.new(Serdes::ConcreteType.new(Boolean)), skip_serializing_if_nil: true
 
     def self.from(hash)
       config = super
@@ -54,6 +54,16 @@ module Exwiw
       columns.map(&:name)
     end
 
+    # Drop the belongs_tos/columns flagged `ignore:true` so they are excluded
+    # from extraction (dependency ordering, SELECT projection, INSERT). The
+    # config files on disk keep these entries; this is applied to the runtime
+    # config right after it is loaded from a file (see Runner#load_table_config).
+    def reject_ignored_members!
+      self.belongs_tos = belongs_tos.reject(&:ignore)
+      self.columns = columns.reject(&:ignore)
+      self
+    end
+
     def belongs_to(table_name)
       belongs_tos.find { |relation| relation.table_name == table_name }
     end
@@ -111,9 +121,22 @@ module Exwiw
         merged_table.type = passed_table.type
         merged_table.comment = comment
         merged_table.filter = filter
-        merged_table.belongs_tos = passed_table.belongs_tos
         merged_table.bulk_insert_chunk_size = passed_table.bulk_insert_chunk_size
-        merged_table.skip = skip
+        merged_table.ignore = ignore
+
+        # Structural facts of each belongs_to come from the freshly generated
+        # config, but the user-owned `comment`/`ignore` carry over when the same
+        # relation still exists.
+        receiver_belongs_to_by_identity = belongs_tos.each_with_object({}) { |bt, hash| hash[bt.identity] = bt }
+        merged_table.belongs_tos =
+          passed_table.belongs_tos.map do |passed_belongs_to|
+            receiver_belongs_to = receiver_belongs_to_by_identity[passed_belongs_to.identity]
+            if receiver_belongs_to
+              passed_belongs_to.comment = receiver_belongs_to.comment if receiver_belongs_to.comment
+              passed_belongs_to.ignore = receiver_belongs_to.ignore unless receiver_belongs_to.ignore.nil?
+            end
+            passed_belongs_to
+          end
 
         receiver_column_by_name = columns.each_with_object({}) { |column, hash| hash[column.name] = column }
 
@@ -144,9 +167,9 @@ module Exwiw
                 "Table '#{name}' has type=#{type}; columns must not be defined."
         end
       else
-        # A skip:true table is not extracted, so primary_key is not required
+        # An ignore:true table is not extracted, so primary_key is not required
         # (e.g. a composite-primary-key table that exwiw does not support).
-        if primary_key.nil? && !skip
+        if primary_key.nil? && !ignore
           raise ArgumentError, "Table '#{name}' requires primary_key."
         end
       end

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.3.2"
+  VERSION = "0.3.4"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.3.2
+  version: 0.3.4
 platform: ruby
 authors:
 - Shia