exwiw 0.1.4 → 0.1.6

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b5120d6bb2e0b1e32096c8461b5feba65248b3c48e491e94fda0b439a73cfe05
-  data.tar.gz: 11ebd135b111b71666c505dd6fb964007cc6fd88143c96d0fb0a5d3df2c7f99e
+  metadata.gz: c3fe2fb0c2cff1c962899799f4cfe22981a7b730759d456dc01ab8d19254dbec
+  data.tar.gz: 7bf651780e74074b50e9b0345a7cdcb9ec0281cd3d60ff04764cf03c1804b32e
 SHA512:
-  metadata.gz: 10a26785c8ee4d505494fc8576dc9267b4fb20f79a944962dbd6f2d659286b0fe90f06523e18ea597561f6ee18d74255acc765665b992504544b62a05983f4c5
-  data.tar.gz: 4bf778905971b929da83c0af4e82245a33efc2a5a303ab4a403cb32e68ab3c07c1811cd69562ef70789f515986d92309c3f6b17011245fdbebf498ce311901d0
+  metadata.gz: a8ee7520812cb8a655cd4f9b8dffa3afdd622311c6b39c5b7942aae5743c3642cb44a7b6b2fdd56ffdf967a8225c220866691a8a001bf75606c70871d4d731ca
+  data.tar.gz: 10182f0c69f0d773e1a515f932148c7b8795a51ba64e46141d0630459c815594e470ac62e4d082a695426ec254430dcac6c3cdbaff74b47e34c49f0dfa9a901a

data/.rubocop.yml

@@ -0,0 +1,10 @@
+plugins:
+  - rubocop-greppable_rails
+
+AllCops:
+  TargetRubyVersion: 3.3
+  DisabledByDefault: true
+  SuggestExtensions: false
+
+GreppableRails/UseInlineAccessModifier:
+  Enabled: true

data/CHANGELOG.md

@@ -2,6 +2,28 @@
 
 ## [Unreleased]
 
+### Added
+
+- Add `bulk_insert_chunk_size` table config to split the generated `INSERT` statement into chunks of the specified size. ([#8](https://github.com/riseshia/exwiw/pull/8))
+- Add experimental MongoDB adapter (`--adapter=mongodb`) that exports collections as JSONL (`insert-{idx}-{collection}.jsonl`). Embedded documents and collection-level `filter` are not supported. ([#10](https://github.com/riseshia/exwiw/pull/10))
+- Introduce `MongodbCollectionConfig` for the MongoDB adapter, with MongoDB-native naming (`fields` instead of `columns`). ([#10](https://github.com/riseshia/exwiw/pull/10))
+
+### Changed
+
+- **Breaking (MongoDB only):** scenario JSON for the MongoDB adapter must use `fields` instead of `columns`. SQL adapters (`mysql2`, `postgresql`, `sqlite3`) are unaffected. ([#10](https://github.com/riseshia/exwiw/pull/10))
+- Bump minimum required Ruby version to 3.3.0 and drop Ruby 3.2 from the CI matrix (3.2 reached EOL on 2026-03-31).
+- Refactor `Adapter` contract to support non-SQL backends: introduce `Adapter#build_query`, `Adapter#output_extension`, and `Adapter#supports_bulk_delete?` hooks. SQL adapters retain existing behavior. ([#9](https://github.com/riseshia/exwiw/pull/9))
+- Extract `exwiw:schema:generate` logic into `Exwiw::SchemaGenerator` so it can be exercised under RSpec without the Rake harness. ([#11](https://github.com/riseshia/exwiw/pull/11))
+
+### Fixed
+
+- Fix MySQL host access for local rspec runs and switch local dev scripts to inject the password via `MYSQL_PWD` env on `docker compose exec` instead of the `-p` CLI flag. ([#5](https://github.com/riseshia/exwiw/pull/5))
+- Expand `~` in path arguments and validate the existence of `--config-dir`. ([#6](https://github.com/riseshia/exwiw/pull/6))
+- Fix incorrect left-side table in `JOIN ... ON` clause for join chains with 3+ hops, which caused `no such column` / `column does not exist` errors at execute time. ([#7](https://github.com/riseshia/exwiw/pull/7))
+- Fix hard-coded `id` primary key in `QueryAstBuilder` so non-`id` primary keys are honored when expanding `dump_target.ids` into `WHERE`. ([#9](https://github.com/riseshia/exwiw/pull/9))
+- `exwiw:schema:generate` now aggregates `belongs_to` reflections across STI subclasses sharing one table; previously the first-seen class won and subclass associations could be silently dropped. ([#11](https://github.com/riseshia/exwiw/pull/11))
+- `exwiw:schema:generate` now fails fast with `Exwiw::SchemaGenerator::MultipleDatabasesNotSupportedError` when models span multiple `connects_to` databases instead of silently producing a partial schema bound to a single connection. ([#11](https://github.com/riseshia/exwiw/pull/11))
+
 ## [0.1.4] - 2026-04-04
 
 ### Fixed

data/README.md

@@ -40,6 +40,7 @@ gem install exwiw
 - mysql2
 - postgresql
 - sqlite3
+- mongodb (experimental, see [MongoDB notes](#mongodb-notes))
 
 ## Usage
 
@@ -98,6 +99,7 @@ This is an example of the one table schema:
     "name": "users",
     "primary_key": "id",
     "filter": "users.id > 0",
+    "bulk_insert_chunk_size": 1000,
     "belongs_to": [{
         "name": "companies",
         "foreign_key": "company_id"
@@ -115,6 +117,12 @@ This is an example of the one table schema:
 
 `--config-dir` will use all json files in the specified directory.
 
+### Bulk insert chunk size
+
+`bulk_insert_chunk_size` splits the generated `INSERT` statement into multiple statements, each containing at most the specified number of rows. This is useful when the number of records per table is large enough to hit limits like MySQL's `max_allowed_packet`.
+
+If omitted, all records for a table are emitted as a single `INSERT` statement.
+
 ### Filter
 
 Some case, you don't need full records related to target. e.g. dump user access logs only for the last year.
@@ -163,6 +171,21 @@ Notice this is the most powerful option, but you should be careful to use this o
 Because this transformation occured on exwiw process, so much slower than other options.
 Most of case, this option is not recommended.
 
+### MongoDB notes
+
+The MongoDB adapter is experimental. To use it:
+
+- Add `gem "mongo"` to your Gemfile in addition to `exwiw` (it is not declared as a runtime dependency of the gem).
+- Set `--adapter=mongodb`. `--user` / `DATABASE_PASSWORD` are optional and only needed when your MongoDB requires authentication.
+- The MongoDB adapter consumes a separate config type, `MongodbCollectionConfig`, with MongoDB-native naming. Use `fields` (instead of the SQL adapters' `columns`), and set `"primary_key": "_id"`. Foreign keys (`shop_id`, `user_id`, ...) stay as ordinary fields.
+- Output is JSON Lines (`insert-{idx}-{collection}.jsonl`) using MongoDB Extended JSON (relaxed mode). Import with `mongoimport`:
+  ```bash
+  mongoimport --db app_dev --collection users --file dump/insert-002-users.jsonl
+  ```
+- Unlike SQL adapters, the MongoDB adapter does not emit `delete-*.jsonl` files (drop the database / collection yourself before importing if needed).
+- `raw_sql` is not supported (the `MongodbField` schema does not declare it; any `raw_sql` keys in scenario JSON are silently dropped on load). Use `replace_with` for masking.
+- The MongoDB adapter does not support the collection-level `filter` field (it raises `NotImplementedError` if set, since the SQL-string filter cannot be applied to MongoDB).
+
 ## How it works
 
 - Load the table information from the specified config file.

data/lib/exwiw/adapter/mongodb_adapter.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+require 'json'
+
+# NOTE: This adapter consumes MongodbCollectionConfig (`fields` instead of
+# the SQL adapters' `columns`). It assumes a "flat" document schema where
+# references between collections are expressed as scalar foreign keys
+# (e.g. `shop_id` on `users`); the forward fan-out strategy here cannot
+# follow references that live inside embedded structures.
+module Exwiw
+  module Adapter
+    class MongodbAdapter < Base
+      def self.table_config_class
+        Exwiw::MongodbCollectionConfig
+      end
+
+      def initialize(connection_config, logger)
+        super
+        @state = {}
+      end
+
+      def build_query(config, dump_target, _config_by_name)
+        reject_filter!(config)
+
+        filter =
+          if config.name == dump_target.table_name
+            { config.primary_key => { "$in" => coerce_ids(dump_target.ids) } }
+          else
+            constrained = config.belongs_tos.select do |relation|
+              @state.key?(relation.table_name) && !@state[relation.table_name].empty?
+            end
+
+            if constrained.empty?
+              {}
+            else
+              constrained.each_with_object({}) do |relation, acc|
+                acc[relation.foreign_key] = { "$in" => @state[relation.table_name] }
+              end
+            end
+          end
+
+        Exwiw::MongoQuery::Find.new(
+          collection: config.name,
+          primary_key: config.primary_key,
+          filter: filter,
+          projection: build_projection(config),
+        )
+      end
+
+      def execute(query)
+        @logger.debug("  Executing Mongo find on '#{query.collection}': filter=#{query.filter.inspect} projection=#{query.projection.inspect}")
+
+        docs = db[query.collection].find(query.filter).projection(query.projection).to_a
+
+        @state[query.collection] = docs.map { |doc| doc[query.primary_key] }
+
+        docs
+      end
+
+      def to_bulk_insert(rows, config)
+        rows.map do |doc|
+          apply_replace_with!(doc, config)
+          JSON.generate(extended_json(doc))
+        end.join("\n")
+      end
+
+      def to_bulk_delete(_query, _config)
+        raise NotImplementedError, "MongodbAdapter does not support bulk delete"
+      end
+
+      def output_extension
+        'jsonl'
+      end
+
+      def supports_bulk_delete?
+        false
+      end
+
+      # `--ids` from the CLI arrives as Strings. Mongo compares types strictly,
+      # so integer-looking ids are coerced to Integer. Other strings (e.g. ObjectId
+      # hex) are left as-is.
+      private def coerce_ids(ids)
+        Array(ids).map do |id|
+          if id.is_a?(String) && id.match?(/\A-?\d+\z/)
+            id.to_i
+          else
+            id
+          end
+        end
+      end
+
+      private def reject_filter!(config)
+        return if config.filter.nil? || config.filter.to_s.empty?
+
+        raise NotImplementedError,
+              "collection-level `filter` is not supported by MongodbAdapter (collection: #{config.name})"
+      end
+
+      private def build_projection(config)
+        projection = {}
+        # Always include primary key so masking templates referencing it work,
+        # even if it is not declared in fields.
+        projection[config.primary_key] = 1
+        config.fields.each do |field|
+          projection[field.name] = 1
+        end
+        projection
+      end
+
+      private def apply_replace_with!(doc, config)
+        config.fields.each do |field|
+          next unless field.replace_with
+
+          doc[field.name] = field.replace_with.gsub(/\{([^{}]+)\}/) do
+            ref = Regexp.last_match(1)
+            (doc.key?(ref) ? doc[ref] : nil).to_s
+          end
+        end
+      end
+
+      private def extended_json(doc)
+        if doc.respond_to?(:as_extended_json)
+          doc.as_extended_json(mode: :relaxed)
+        else
+          doc
+        end
+      end
+
+      private def db
+        @db ||=
+          begin
+            require 'mongo'
+            address = "#{@connection_config.host}:#{@connection_config.port}"
+            options = { database: @connection_config.database_name }
+            if @connection_config.user && !@connection_config.user.to_s.empty?
+              options[:user] = @connection_config.user
+              options[:password] = @connection_config.password
+            end
+            Mongo::Logger.logger.level = ::Logger::WARN
+            Mongo::Client.new([address], **options)
+          end
+      end
+    end
+  end
+end

data/lib/exwiw/adapter/mysql2_adapter.rb

@@ -3,6 +3,10 @@
 module Exwiw
   module Adapter
     class Mysql2Adapter < Base
+      def build_query(table, dump_target, table_by_name)
+        Exwiw::QueryAstBuilder.run(table.name, table_by_name, dump_target, @logger)
+      end
+
       def execute(query_ast)
         sql = compile_ast(query_ast)
 
@@ -79,7 +83,7 @@ module Exwiw
         sql += " FROM #{query_ast.from_table_name}"
 
         query_ast.join_clauses.each do |join|
-          sql += " JOIN #{join.join_table_name} ON #{query_ast.from_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
+          sql += " JOIN #{join.join_table_name} ON #{join.base_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
 
           join.where_clauses.each do |where|
             compiled_where_condition = compile_where_condition(where, join.join_table_name)

data/lib/exwiw/adapter/postgresql_adapter.rb

@@ -3,6 +3,10 @@
 module Exwiw
   module Adapter
     class PostgresqlAdapter < Base
+      def build_query(table, dump_target, table_by_name)
+        Exwiw::QueryAstBuilder.run(table.name, table_by_name, dump_target, @logger)
+      end
+
       def execute(query_ast)
         sql = compile_ast(query_ast)
 
@@ -79,7 +83,7 @@ module Exwiw
         sql += " FROM #{query_ast.from_table_name}"
 
         query_ast.join_clauses.each do |join|
-          sql += " JOIN #{join.join_table_name} ON #{query_ast.from_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
+          sql += " JOIN #{join.join_table_name} ON #{join.base_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
 
           join.where_clauses.each do |where|
             compiled_where_condition = compile_where_condition(where, join.join_table_name)

data/lib/exwiw/adapter/sqlite3_adapter.rb

@@ -3,6 +3,10 @@
 module Exwiw
   module Adapter
     class Sqlite3Adapter < Base
+      def build_query(table, dump_target, table_by_name)
+        Exwiw::QueryAstBuilder.run(table.name, table_by_name, dump_target, @logger)
+      end
+
       def execute(query_ast)
         sql = compile_ast(query_ast)
 
@@ -79,7 +83,7 @@ module Exwiw
         sql += " FROM #{query_ast.from_table_name}"
 
         query_ast.join_clauses.each do |join|
-          sql += " JOIN #{join.join_table_name} ON #{query_ast.from_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
+          sql += " JOIN #{join.join_table_name} ON #{join.base_table_name}.#{join.foreign_key} = #{join.join_table_name}.#{join.primary_key}"
 
           join.where_clauses.each do |where|
             compiled_where_condition = compile_where_condition(where, join.join_table_name)
@@ -157,7 +161,7 @@ module Exwiw
         @connection ||=
           begin
             require 'sqlite3'
-            SQLite3::Database.new(@connection_config.database_name)
+            SQLite3::Database.new(File.expand_path(@connection_config.database_name))
           end
       end
     end

data/lib/exwiw/adapter.rb

@@ -9,6 +9,31 @@ module Exwiw
         @connection_config = connection_config
         @logger = logger
       end
+
+      # The config class that this adapter consumes. Runner uses this to
+      # decide which Serdes type to load scenario JSON into. SQL adapters
+      # share the SQL-shaped TableConfig; non-SQL adapters override.
+      def self.table_config_class
+        TableConfig
+      end
+
+      # @params [Exwiw::TableConfig] table
+      # @params [Exwiw::DumpTarget] dump_target
+      # @params [Hash{String => Exwiw::TableConfig}] table_by_name
+      # @return [Object] adapter-specific query object (e.g. Exwiw::QueryAst::Select for SQL)
+      def build_query(table, dump_target, table_by_name)
+        raise NotImplementedError
+      end
+
+      # File extension used for dump output (e.g. 'sql' for SQL, 'jsonl' for MongoDB).
+      def output_extension
+        'sql'
+      end
+
+      # Whether this adapter emits delete-NNN-*.sql files.
+      def supports_bulk_delete?
+        true
+      end
     end
 
     # @params [Exwiw::QueryAst] query_ast
@@ -36,6 +61,8 @@ module Exwiw
         Adapter::Mysql2Adapter.new(connection_config, logger)
       when 'postgresql'
         Adapter::PostgresqlAdapter.new(connection_config, logger)
+      when 'mongodb'
+        Adapter::MongodbAdapter.new(connection_config, logger)
       else
         raise 'Unsupported adapter'
       end

data/lib/exwiw/cli.rb

@@ -67,34 +67,46 @@ module Exwiw
 
     private def validate_options!
       if @database_adapter != "sqlite3"
-        {
+        required_options = {
           "Target database host" => @database_host,
           "Target database port" => @database_port,
-          "Database user" => @database_user,
           "Target database name" => @database_name,
-        }.each do |k, v|
+        }
+        required_options["Database user"] = @database_user unless @database_adapter == "mongodb"
+        required_options.each do |k, v|
           if v.nil?
             $stderr.puts "#{k} is required"
             exit 1
           end
         end
 
-        if @config_dir.nil?
-          $stderr.puts "Config dir is required"
-        end
-
-        if @database_password.nil? || @database_password.empty?
+        if @database_adapter != "mongodb" && (@database_password.nil? || @database_password.empty?)
           $stderr.puts "environment variable 'DATABASE_PASSWORD' is required"
           exit 1
         end
       end
 
-      valid_adapters = ["mysql2", "postgresql", "sqlite3"]
+      valid_adapters = ["mysql2", "postgresql", "sqlite3", "mongodb"]
       unless valid_adapters.include?(@database_adapter)
         $stderr.puts "Invalid adapter. Available options are: #{valid_adapters.join(', ')}"
         exit 1
       end
 
+      if @config_dir.nil?
+        $stderr.puts "Config dir is required"
+        exit 1
+      end
+
+      unless Dir.exist?(@config_dir)
+        $stderr.puts "Config dir does not exist: #{@config_dir}"
+        exit 1
+      end
+
+      if Dir.glob(File.join(@config_dir, "*.json")).empty?
+        $stderr.puts "Config dir contains no .json files: #{@config_dir}"
+        exit 1
+      end
+
       if @target_table_name.nil? || @target_table_name.empty?
         $stderr.puts "Target table is required"
         exit 1
@@ -130,10 +142,12 @@ module Exwiw
         opts.on("-p", "--port=PORT", "Target database port") { |v| @database_port = v }
         opts.on("-u", "--user=USERNAME", "Target database user") { |v| @database_user = v }
         opts.on("-o", "--output-dir=[DUMP_DIR_PATH]", "Output file path. default is dump/") do |v|
-          @output_dir = v.end_with?("/") ? v[0..-2] : v
+          v = v.end_with?("/") ? v[0..-2] : v
+          @output_dir = File.expand_path(v)
         end
         opts.on("-c", "--config-dir=CONFIG_DIR_PATH", "Config dir path.") do |v|
-          @config_dir = v.end_with?("/") ? v[0..-2] : v
+          v = v.end_with?("/") ? v[0..-2] : v
+          @config_dir = File.expand_path(v)
         end
         opts.on("-a", "--adapter=ADAPTER", "Database adapter") { |v| @database_adapter = v }
         opts.on("--database=DATABASE", "Target database name") { |v| @database_name = v }

data/lib/exwiw/mongo_query.rb

@@ -0,0 +1,16 @@
+# frozen_string_literal: true
+
+module Exwiw
+  module MongoQuery
+    Find = Struct.new(:collection, :primary_key, :filter, :projection, keyword_init: true) do
+      def to_h
+        {
+          collection: collection,
+          primary_key: primary_key,
+          filter: filter,
+          projection: projection,
+        }
+      end
+    end
+  end
+end

data/lib/exwiw/mongodb_collection_config.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Exwiw
+  class MongodbCollectionConfig
+    include Serdes
+
+    # MongoDB-native names. Intentionally re-declared instead of inheriting
+    # from TableConfig — Serdes does not propagate attribute declarations
+    # across class boundaries.
+    attribute :name, String
+    attribute :primary_key, String
+    attribute :filter, optional(String), skip_serializing_if_nil: true
+    attribute :belongs_tos, array(BelongsTo)
+    attribute :fields, array(MongodbField)
+    attribute :bulk_insert_chunk_size, optional(Integer), skip_serializing_if_nil: true
+
+    def self.from_symbol_keys(hash)
+      from(JSON.parse(hash.to_json))
+    end
+  end
+end

data/lib/exwiw/mongodb_field.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Exwiw
+  class MongodbField
+    include Serdes
+
+    attribute :name, String
+    attribute :replace_with, optional(String), skip_serializing_if_nil: true
+
+    def self.from_symbol_keys(hash)
+      from(hash.transform_keys(&:to_s))
+    end
+
+    def to_hash
+      super.compact
+    end
+  end
+end

data/lib/exwiw/query_ast.rb

@@ -32,7 +32,7 @@ module Exwiw
         {
           column_name: column_name,
           operator: operator,
-          value: value
+          value: value,
         }
       end
     end

data/lib/exwiw/query_ast_builder.rb

@@ -77,7 +77,7 @@ module Exwiw
 
       if table.name == dump_target.table_name
         clauses.push Exwiw::QueryAst::WhereClause.new(
-          column_name: 'id',
+          column_name: table.primary_key,
           operator: :eq,
           value: dump_target.ids
         )

data/lib/exwiw/runner.rb

@@ -20,7 +20,7 @@ module Exwiw
 
     def run
       adapter = Adapter.build(@connection_config, @logger)
-      tables = load_table_config
+      tables = load_table_config(adapter.class.table_config_class)
 
       @logger.info("Determining table processing order...")
       ordered_table_names = DetermineTableProcessingOrder.run(tables)
@@ -36,7 +36,7 @@ module Exwiw
         @logger.info("Processing table '#{table_name}'... (#{idx + 1}/#{total_size})")
         table = table_by_name.fetch(table_name)
 
-        query_ast = QueryAstBuilder.run(table.name, table_by_name, @dump_target, @logger)
+        query_ast = adapter.build_query(table, @dump_target, table_by_name)
         results = adapter.execute(query_ast)
         record_num = results.size
 
@@ -44,34 +44,38 @@ module Exwiw
           @logger.info("  No records matched. skip this table.")
           next
         end
-        @logger.debug("  Generate INSERT SQL...")
+        @logger.debug("  Generate INSERT statement...")
 
-        insert_sql = adapter.to_bulk_insert(results, table)
+        chunk_size = table.bulk_insert_chunk_size
+        chunks = chunk_size ? results.each_slice(chunk_size).to_a : [results]
+        insert_sql = chunks.map { |chunk_rows| adapter.to_bulk_insert(chunk_rows, table) }.join("\n")
 
-        @logger.info("  Generated INSERT SQL for #{record_num} records.")
+        @logger.info("  Generated INSERT statement for #{record_num} records (#{chunks.size} statement(s)).")
         insert_idx = (idx + 1).to_s.rjust(3, '0')
-        File.open(File.join(@output_dir, "insert-#{insert_idx}-#{table_name}.sql"), 'w') do |file|
+        File.open(File.join(@output_dir, "insert-#{insert_idx}-#{table_name}.#{adapter.output_extension}"), 'w') do |file|
           file.puts(insert_sql)
         end
 
-        @logger.debug("  Generate DELETE SQL...")
-        delete_sql = adapter.to_bulk_delete(query_ast, table)
-        if @logger.debug?
-          @logger.debug("  Generated DELETE SQL:\n#{delete_sql}")
-        else
-          @logger.info("  Generated DELETE SQL.")
-        end
-        delete_idx = (total_size - idx).to_s.rjust(3, '0')
-        File.open(File.join(@output_dir, "delete-#{delete_idx}-#{table_name}.sql"), 'w') do |file|
-          file.puts(delete_sql)
+        if adapter.supports_bulk_delete?
+          @logger.debug("  Generate DELETE statement...")
+          delete_sql = adapter.to_bulk_delete(query_ast, table)
+          if @logger.debug?
+            @logger.debug("  Generated DELETE statement:\n#{delete_sql}")
+          else
+            @logger.info("  Generated DELETE statement.")
+          end
+          delete_idx = (total_size - idx).to_s.rjust(3, '0')
+          File.open(File.join(@output_dir, "delete-#{delete_idx}-#{table_name}.#{adapter.output_extension}"), 'w') do |file|
+            file.puts(delete_sql)
+          end
         end
       end
     end
 
-    private def load_table_config
+    private def load_table_config(klass)
       Dir[File.join(@config_dir, "*.json")].map do |file|
         json = JSON.parse(File.read(file))
-        TableConfig.from(json)
+        klass.from(json)
       end
     end
 

data/lib/exwiw/schema_generator.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+require "fileutils"
+require "json"
+
+module Exwiw
+  class SchemaGenerator
+    class MultipleDatabasesNotSupportedError < StandardError; end
+
+    def self.from_rails_application(output_dir:)
+      Rails.application.eager_load!
+      new(models: ActiveRecord::Base.descendants, output_dir: output_dir)
+    end
+
+    def initialize(models:, output_dir:)
+      @models = models
+      @output_dir = output_dir
+    end
+
+    def generate!
+      tables = build_tables
+      write_files(tables)
+      tables
+    end
+
+    def build_tables
+      models = concrete_models
+      validate_single_database!(models)
+
+      models.group_by(&:table_name).map do |table_name, model_group|
+        representative = model_group.first
+        TableConfig.from_symbol_keys(
+          name: table_name,
+          primary_key: representative.primary_key,
+          belongs_tos: aggregate_belongs_tos(model_group),
+          columns: representative.column_names.map { |name| { name: name } },
+        )
+      end
+    end
+
+    def write_files(tables)
+      FileUtils.mkdir_p(@output_dir)
+
+      tables.each do |table|
+        path = File.join(@output_dir, "#{table.name}.json")
+        config_to_write =
+          if File.exist?(path)
+            TableConfig.from(JSON.parse(File.read(path))).merge(table)
+          else
+            table
+          end
+        File.write(path, JSON.pretty_generate(config_to_write.to_hash) + "\n")
+      end
+    end
+
+    private def concrete_models
+      @models.reject(&:abstract_class?).select(&:table_exists?)
+    end
+
+    private def aggregate_belongs_tos(models)
+      pairs = models
+        .flat_map { |m| m.reflect_on_all_associations(:belongs_to) }
+        .reject(&:polymorphic?) # XXX: Support polymorphic
+        .map { |assoc| [assoc.table_name, assoc.foreign_key] }
+        .uniq
+
+      pairs.map do |table_name, foreign_key|
+        { table_name: table_name, foreign_key: foreign_key }
+      end
+    end
+
+    # `connection_specification_name` is a quasi-private API but has been stable
+    # across Rails 6.1 - 8.x. With Rails multi-DB (`connects_to`), every
+    # descendant of the same abstract base shares one spec name regardless of
+    # role/shard, so distinct values across concrete models indicate genuinely
+    # separate databases.
+    private def validate_single_database!(models)
+      return if models.empty?
+
+      specs = models.map(&:connection_specification_name).uniq
+      return if specs.size <= 1
+
+      raise MultipleDatabasesNotSupportedError, <<~MSG
+        exwiw does not yet support Rails multiple-database setup.
+        Detected connection specifications: #{specs.inspect}
+        Track progress at https://github.com/riseshia/exwiw/issues
+      MSG
+    end
+  end
+end

data/lib/exwiw/table_config.rb

@@ -9,6 +9,7 @@ module Exwiw
     attribute :filter, optional(String), skip_serializing_if_nil: true
     attribute :belongs_tos, array(BelongsTo)
     attribute :columns, array(TableColumn)
+    attribute :bulk_insert_chunk_size, optional(Integer), skip_serializing_if_nil: true
 
     def self.from_symbol_keys(hash)
       from(JSON.parse(hash.to_json))
@@ -74,6 +75,7 @@ module Exwiw
         merged_table.primary_key = passed_table.primary_key
         merged_table.filter = filter
         merged_table.belongs_tos = passed_table.belongs_tos
+        merged_table.bulk_insert_chunk_size = passed_table.bulk_insert_chunk_size
 
         receiver_column_by_name = columns.each_with_object({}) { |column, hash| hash[column.name] = column }
 

data/lib/exwiw/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Exwiw
-  VERSION = "0.1.4"
+  VERSION = "0.1.6"
 end

data/lib/exwiw.rb

@@ -5,17 +5,22 @@ require_relative "exwiw/version"
 require "json"
 require "serdes"
 
+require_relative "exwiw/belongs_to"
+require_relative "exwiw/table_column"
+require_relative "exwiw/table_config"
+require_relative "exwiw/mongodb_field"
+require_relative "exwiw/mongodb_collection_config"
 require_relative "exwiw/adapter"
 require_relative "exwiw/adapter/sqlite3_adapter"
 require_relative "exwiw/adapter/mysql2_adapter"
 require_relative "exwiw/adapter/postgresql_adapter"
+require_relative "exwiw/adapter/mongodb_adapter"
 require_relative "exwiw/determine_table_processing_order"
+require_relative "exwiw/mongo_query"
 require_relative "exwiw/query_ast"
 require_relative "exwiw/query_ast_builder"
 require_relative "exwiw/runner"
-require_relative "exwiw/belongs_to"
-require_relative "exwiw/table_column"
-require_relative "exwiw/table_config"
+require_relative "exwiw/schema_generator"
 
 begin
   require 'rails'

data/lib/tasks/exwiw.rake

@@ -4,59 +4,11 @@ namespace :exwiw do
   namespace :schema do
     desc "Generate schema from application"
     task generate: :environment do
-      require "json"
       require "exwiw"
-      require "fileutils"
 
-      Rails.application.eager_load!
-
-      table_by_name = {}
-
-      ActiveRecord::Base.descendants.each do |model|
-        next if model.abstract_class?
-        next unless model.table_exists?
-        next if table_by_name[model.table_name]
-
-        belongs_tos = model.reflect_on_all_associations(:belongs_to).map do |assoc|
-          if assoc.polymorphic?
-            # XXX: Support polymorphic
-            next
-          else
-            Exwiw::BelongsTo.from_symbol_keys({
-              table_name: assoc.table_name,
-              foreign_key: assoc.foreign_key,
-            })
-          end
-        end
-
-        columns = model.column_names.map do |name|
-          Exwiw::TableColumn.from_symbol_keys({ name: name })
-        end
-
-        table = Exwiw::TableConfig.from_symbol_keys({
-          name: model.table_name,
-          primary_key: model.primary_key,
-          belongs_tos: belongs_tos.compact,
-          columns: columns,
-        })
-        table_by_name[table.name] = table
-      end
-
-      tables = table_by_name.values
-
-      output_dir = ENV['OUTPUT_DIR_PATH'] || "exwiw"
-      FileUtils.mkdir_p(output_dir)
-
-      tables.each do |table|
-        path = File.join(output_dir, "#{table.name}.json")
-        if File.exist?(path)
-          current_config = Exwiw::TableConfig.from(JSON.parse(File.read(path)))
-          merged_config = current_config.merge(table)
-          File.write(path, JSON.pretty_generate(merged_config.to_hash) + "\n")
-        else
-          File.write(path, JSON.pretty_generate(table.to_hash) + "\n")
-        end
-      end
+      Exwiw::SchemaGenerator.from_rails_application(
+        output_dir: ENV["OUTPUT_DIR_PATH"] || "exwiw",
+      ).generate!
     end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: exwiw
 version: !ruby/object:Gem::Version
-  version: 0.1.4
+  version: 0.1.6
 platform: ruby
 authors:
 - Shia
@@ -32,22 +32,28 @@ executables:
 extensions: []
 extra_rdoc_files: []
 files:
+- ".rubocop.yml"
 - CHANGELOG.md
 - LICENSE.txt
 - README.md
 - exe/exwiw
 - lib/exwiw.rb
 - lib/exwiw/adapter.rb
+- lib/exwiw/adapter/mongodb_adapter.rb
 - lib/exwiw/adapter/mysql2_adapter.rb
 - lib/exwiw/adapter/postgresql_adapter.rb
 - lib/exwiw/adapter/sqlite3_adapter.rb
 - lib/exwiw/belongs_to.rb
 - lib/exwiw/cli.rb
 - lib/exwiw/determine_table_processing_order.rb
+- lib/exwiw/mongo_query.rb
+- lib/exwiw/mongodb_collection_config.rb
+- lib/exwiw/mongodb_field.rb
 - lib/exwiw/query_ast.rb
 - lib/exwiw/query_ast_builder.rb
 - lib/exwiw/railtie.rb
 - lib/exwiw/runner.rb
+- lib/exwiw/schema_generator.rb
 - lib/exwiw/table_column.rb
 - lib/exwiw/table_config.rb
 - lib/exwiw/version.rb
@@ -66,14 +72,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 3.2.0
+      version: 3.3.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 4.0.3
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Ruby gem that allows you to export records from a database to a dump file.
 test_files: []