active_record_importer 0.3.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 0b0c2019e863558605d2703fb80890c8ba209192
-  data.tar.gz: 480be20d7fc0440427e4107f7b50e7e19572ad81
+SHA256:
+  metadata.gz: a9432ca865104e5e0298a6cc01cbdedd8d5e5512ab8418f54301731aad44994e
+  data.tar.gz: 84c9f8bd766fb6d81194b7c0ab44b65447f0fd18562bfc9b91630e4af6f861c4
 SHA512:
-  metadata.gz: f9191b603ceb3e77bff828ac70ad0a9301b854a1956cc1ef218e00177c2f8d5b20b0722e98bcc99cb9f99663126e29e53d3b57eebe60d14c88eb248008092441
-  data.tar.gz: 4be44dcf15fcd9a9981ff0942c63a1a1e6d21691cf27832cb3d3755a78636f4fe79193fbe26eb5bb4ccc437cbd0e6c0ab5583d73e37f25e67e78b1a8d7d53265
+  metadata.gz: '09eb496352a2b2b29f3a79f033689167b6bf9f743e0b14da5364211e60e4176db44d821ea94d23250028eac4ada508171fbcfcfc45cc1a3f93b39c000e4c3608'
+  data.tar.gz: c0450b85a0cb842663939cf69f65dd1ac857f351bd761b4f9d1ef84eac39647edb177a644ccb03564a53622525bdb2d872b370b28e681836ecfbeb11dce58359

data/CHANGELOG.md

@@ -0,0 +1,40 @@
+# Changelog
+
+## 1.0.0
+
+### Added
+- Rows are persisted in bulk per CSV chunk through
+  [activerecord-import](https://github.com/zdennis/activerecord-import):
+  one bulk `INSERT` for new records and one bulk upsert (primary key conflict)
+  for existing records, inside a single transaction per chunk.
+- Existing records matched by `find_options` are loaded with a single query per chunk.
+- Validations are kept: every row is validated (`valid?`) before persistence and invalid
+  rows are written to the failed file with their errors, exactly as before.
+- When the database rejects a bulk statement (e.g. a unique index without a model
+  validation), the group is retried row by row in savepoints so only the offending rows fail.
+- `acts_as_importable bulk_insert: false` saves rows one by one (still one transaction
+  per chunk) so the model's own ActiveRecord save callbacks keep running.
+- Spec suite running against PostgreSQL (default) and MySQL (`DB=mysql`).
+
+### Changed
+- **Bulk mode skips the model's own ActiveRecord save callbacks** (`before_save`,
+  `after_create`, ...). Use `bulk_insert: false` if your model depends on them during import.
+  Importer hooks (`before_save`, `after_save`, `state_machine_attr`) still run; when
+  `after_save` or `state_machine_attr` are configured the chunk is saved row by row.
+- Default `chunk_size` is now 500 (was `nil`, which processed one row at a time).
+- Requires Ruby >= 2.7 and ActiveRecord >= 6.0 (was ActiveRecord >= 4.0).
+- `activerecord-import` is a new runtime dependency.
+- RSpec 3 for development.
+
+### Fixed
+- Importer options were stored in a class variable shared by every importable model;
+  each model now keeps its own options.
+- `default_attributes` given as `Proc` or `Symbol` were never evaluated, and
+  `importing: true` was not set on imported instances.
+- `before_save`/`after_save` given as a single `Symbol` or `Proc` raised `NameError`.
+- The failed file header row was not written as CSV.
+- `File.exists?` (removed in Ruby 3.2) replaced with `File.exist?`.
+
+## 0.4.0 and earlier
+
+See the git history.

data/Gemfile

@@ -2,3 +2,10 @@ source 'https://rubygems.org'
 
 # Specify your gem's dependencies in active_record_importer.gemspec
 gemspec
+
+# Database adapters used by the spec suite (see README "Development").
+# Select the database with DB=postgresql (default) or DB=mysql.
+group :test do
+  gem 'pg'
+  gem 'trilogy'
+end

data/README.md

@@ -1,11 +1,21 @@
 # ActiveRecordImporter
 
-Supports only Rails 4 and 5
-
 This gem helps you insert/update records easily. For now, it only accepts CSV file.
 This also helps you monitor how many rows are imported, and how many rows failed.
 This gem also allows you to easily import to any model with few configurations.
 
+Rows are validated one by one but persisted **in bulk** (one `INSERT` per chunk,
+one upsert per chunk) through [activerecord-import](https://github.com/zdennis/activerecord-import).
+See [How rows are persisted](#how-rows-are-persisted).
+
+The spec suite runs against PostgreSQL and MySQL (ActiveRecord 7.2); see [Development](#development).
+
+## Requirements
+
+* Ruby >= 2.7
+* ActiveRecord >= 6.0
+* PostgreSQL (>= 9.5) or MySQL for bulk upserts; other databases fall back to row-by-row saving
+
 I'll release an update to enable this on background job.
 
 ## Installation
@@ -25,9 +35,17 @@ Or install it yourself as:
     $ gem install active_record_importer
 
 ## Usage
-### For version 0.3.0
+### For version 1.0.0
+
+Rows are now validated one by one but persisted in bulk per chunk (see
+[How rows are persisted](#how-rows-are-persisted)). If your model relies on its own
+ActiveRecord save callbacks during import, add `bulk_insert: false` to `acts_as_importable`.
+See the [CHANGELOG](CHANGELOG.md) for the full list of changes.
 
-For the newest version (0.3.0), you don't have to create Import table/model and controller.
+### For version 0.4.0
+
+For the newest version (0.4.0), you don't have to create Import table/model and controller.
+I already fixed the errors when there's no Import table/model on version 0.3.0.
 You just need to add the `acts_as_importable` in your model you want to be importable, and you may now run:
 
 ```ruby
@@ -231,6 +249,38 @@ end
 
 `I'll add more options SOON!`
 
+### How rows are persisted
+
+Every CSV chunk (`chunk_size` / `batch_size` rows, 500 by default) goes through these steps:
+
+1. The attributes of every row are built (`default_attributes`, `store_accessors`, timestamps...).
+2. For `upsert` and `error_duplicate`, the existing records matching the `find_options`
+   of the whole chunk are loaded with **a single query**.
+3. An instance is built for every row, the `before_save` callback runs, and the instance is
+   **validated** (`valid?`). Invalid rows are kept aside with their errors
+   (e.g. `Validation failed: Email can't be blank`).
+4. Valid new records are inserted with **one bulk `INSERT`**, and valid existing records are
+   updated with **one bulk upsert** on the primary key, inside one transaction per chunk.
+   If the database rejects a bulk statement (e.g. a unique index without a model validation),
+   that group is retried row by row in savepoints, so only the offending rows fail.
+5. Failed rows are written to the failed file with an `import_errors` column, and
+   `imported_rows` / `failed_rows` are updated on the `Import` record.
+
+Bulk persistence skips the model's own ActiveRecord save callbacks (like `activerecord-import` does).
+Rows are saved one by one (still within one transaction per chunk) when:
+
+* the importer has `after_save` or `state_machine_attr` options, since they need a persisted instance, or
+* bulk insert is disabled for the model:
+
+```ruby
+class User < ActiveRecord::Base
+  acts_as_importable bulk_insert: false
+end
+```
+
+Duplicates inside the same chunk follow the `insert_method`: with `upsert` they are merged into
+a single record, with `error_duplicate` the second occurrence fails with `Duplicate record found!`.
+
 
 ### Create Imports Controller:
 ```ruby
@@ -268,7 +318,24 @@ Headers of your csv file should be formatted/transformed to column names of your
 
 ## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+After checking out the repo, run `bin/setup` to install dependencies.
+
+The specs run against a real database. By default they use PostgreSQL
+(database `active_record_importer_test`, created automatically if missing):
+
+    $ bundle exec rspec
+    $ DB=postgresql PG_HOST=localhost PG_USER=postgres PG_PASSWORD=secret bundle exec rspec
+
+To run them against MySQL (uses the `trilogy` adapter, or `MYSQL_ADAPTER=mysql2`):
+
+    $ docker run -d --name ari_mysql -e MYSQL_ROOT_PASSWORD=root \
+        -e MYSQL_DATABASE=active_record_importer_test -p 3307:3306 mysql:8.0
+    $ DB=mysql MYSQL_PORT=3307 MYSQL_PASSWORD=root bundle exec rspec
+
+Supported variables: `DB` (`postgresql`|`mysql`), `DB_NAME`, `PG_HOST`, `PG_PORT`, `PG_USER`,
+`PG_PASSWORD`, `MYSQL_ADAPTER`, `MYSQL_HOST`, `MYSQL_PORT`, `MYSQL_USER`, `MYSQL_PASSWORD`.
+
+You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
 To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 

data/active_record_importer.gemspec

@@ -10,7 +10,7 @@ Gem::Specification.new do |spec|
   spec.email         = ['kapitan_03@yahoo.com']
 
   spec.summary       = 'Active Record Importer'
-  spec.description   = 'Smart gem for importing rails models'
+  spec.description   = 'Smart gem for importing CSV files to ActiveRecord models'
   spec.homepage      = 'https://github.com/michaelnera/active_record_importer'
   spec.license       = 'MIT'
 
@@ -19,10 +19,14 @@ Gem::Specification.new do |spec|
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.require_paths = ['lib']
 
-  spec.add_development_dependency 'rspec', '~> 2.99'
+  spec.required_ruby_version = '>= 2.7'
 
-  spec.add_runtime_dependency 'activerecord', '>= 4.0'
-  spec.add_runtime_dependency 'activesupport', '>= 4.0'
+  spec.add_development_dependency 'rake', '>= 12.0'
+  spec.add_development_dependency 'rspec', '~> 3.0'
+
+  spec.add_runtime_dependency 'activerecord', '>= 6.0'
+  spec.add_runtime_dependency 'activerecord-import', '>= 1.0'
+  spec.add_runtime_dependency 'activesupport', '>= 6.0'
   spec.add_runtime_dependency 'enumerize', '>= 2.0', '~> 2.0.1'
   spec.add_runtime_dependency 'virtus', '>= 1.0', '~> 1.0'
   spec.add_runtime_dependency 'smarter_csv', '>= 1.0', '~> 1.0'

data/lib/active_record_importer/attribute/attributes_builder.rb

@@ -30,6 +30,7 @@ module ActiveRecordImporter
         default_attributes.each do |key, value|
           def_attrs[key] = fetch_value(value)
         end
+        def_attrs
       end
 
       def force_encode_attributes

data/lib/active_record_importer/batch_importer.rb

@@ -1,65 +1,92 @@
 module ActiveRecordImporter
+  ##
+  # Imports one chunk of csv rows:
+  #
+  #   1. builds the attributes of every row
+  #   2. preloads, in one query, the records matched by the find options
+  #   3. builds and validates an instance for every row
+  #   4. persists the valid instances in bulk (see BatchPersister)
+  #   5. writes the failed rows (with their errors) and updates the counters
+  ##
   class BatchImporter
     include Virtus.model
 
-    attribute :import, Import
     attribute :importable
+    attribute :import
     attribute :data, Array, default: []
-    attribute :failed_file, FailedFileBuilder, default: :initialize_failed_file
 
     def process!
       @imported_count, @failed_count = 0, 0
 
-      data.each do |row_attrs|
-        next if row_attrs.blank?
-        process_row(row_attrs.symbolize_keys!)
-      end
-
+      processors = build_processors
+      persist(processors)
+      record_results(processors)
       set_import_count
-      finalize_batch_import
     end
 
     private
 
-    def initialize_failed_file
-      return unless import
-      FailedFileBuilder.new(import)
+    def build_processors
+      processors = rows.map { |row_attrs| build_processor(row_attrs) }
+      processors.each(&:prepare)
+      lookup.preload(processors.map(&:find_attributes_for_lookup))
+      processors.each(&:build)
+      processors
     end
 
-    def process_row(row_attrs)
-      processor =
-        DataProcessor.new(
-          import: import,
-          importable: importable,
-          row_attrs: row_attrs
-        )
-      return @imported_count += 1 if processor.process
+    def rows
+      data.reject(&:blank?).map { |row_attrs| row_attrs.to_hash.symbolize_keys }
+    end
 
-      collect_failed_rows(row_attrs, processor.row_errors)
-      @failed_count += 1
+    def build_processor(row_attrs)
+      DataProcessor.new(
+        import: import,
+        importable: importable,
+        row_attrs: row_attrs,
+        lookup: lookup
+      )
     end
 
-    def set_import_count
-      return unless import
+    def lookup
+      @lookup ||= RecordPreloader.new(importable)
+    end
 
-      Import.update_counters(import.id, imported_rows: @imported_count)
-      Import.update_counters(import.id, failed_rows: @failed_count)
+    def persist(processors)
+      BatchPersister.new(
+        importable: importable,
+        processors: processors.select(&:ready?)
+      ).persist!
     end
 
-    def collect_failed_rows(row_attrs, errors)
-      return puts errors.inspect unless failed_file
-      @failed_file.failed_rows << row_attrs.merge(import_errors: errors)
+    def record_results(processors)
+      processors.each do |processor|
+        if processor.failed?
+          write_failed_row(processor.row_attrs, processor.row_errors)
+          @failed_count += 1
+        else
+          @imported_count += 1
+        end
+      end
     end
 
-    def finalize_batch_import
-      return unless failed_file
-      @failed_file.build
+    def set_import_count
+      return unless import
+
+      import.class.update_counters(
+        import.id,
+        imported_rows: @imported_count,
+        failed_rows: @failed_count
+      )
     end
 
-    delegate :importer_options, to: :importable
+    def failed_file
+      return unless import
+      @failed_file ||= FailedFileBuilder.new(import)
+    end
 
-    def csv_options
-      importer_options.csv_opts.to_hash
+    def write_failed_row(row_attrs, errors)
+      return puts errors.inspect unless failed_file
+      failed_file.write(row_attrs.merge(import_errors: errors))
     end
   end
 end

data/lib/active_record_importer/batch_persister.rb

@@ -0,0 +1,103 @@
+module ActiveRecordImporter
+  ##
+  # Persists the already validated instances of a chunk.
+  #
+  # When possible, new records are inserted with a single bulk INSERT and
+  # existing records are updated with a single bulk upsert on the primary
+  # key (through activerecord-import). If a bulk statement fails because of
+  # a database level error (e.g. a unique index), the affected group is
+  # retried row by row inside savepoints so only the offending rows fail.
+  #
+  # When importer callbacks need a persisted instance (after_save or
+  # state_machine_attr), or bulk insert is disabled for the model, every
+  # row is saved individually inside one transaction per chunk.
+  ##
+  class BatchPersister
+    include Virtus.model
+
+    attribute :importable, Class
+    attribute :processors, Array, default: []
+
+    def persist!
+      return if processors.empty?
+
+      importable.transaction do
+        bulk? ? persist_in_bulk : persist_each(processors)
+      end
+    end
+
+    private
+
+    delegate :importer_options, to: :importable
+    delegate :after_save, :state_machine_attr, :bulk_insert, to: :importer_options
+
+    def bulk?
+      bulk_insert != false && !callbacks_required? && bulk_import_supported?
+    end
+
+    def callbacks_required?
+      after_save.present? || state_machine_attr.present?
+    end
+
+    def bulk_import_supported?
+      importable.respond_to?(:bulk_import)
+    end
+
+    def bulk_update_supported?
+      connection = importable.connection
+      connection.respond_to?(:supports_on_duplicate_key_update?) &&
+        connection.supports_on_duplicate_key_update?
+    end
+
+    def persist_in_bulk
+      new_processors, existing_processors =
+        processors.partition { |processor| processor.instance.new_record? }
+
+      bulk_insert_records(new_processors)
+      bulk_update_records(existing_processors)
+    end
+
+    def bulk_insert_records(group)
+      return if group.empty?
+      bulk_import(group, validate: false)
+    end
+
+    def bulk_update_records(group)
+      return if group.empty?
+      return persist_each(group) unless bulk_update_supported?
+
+      columns = changed_columns(group)
+      return if columns.empty?
+
+      bulk_import(group, validate: false, on_duplicate_key_update: columns)
+    end
+
+    def bulk_import(group, options)
+      instances = group.map(&:instance).uniq
+
+      importable.transaction(requires_new: true) do
+        importable.bulk_import(instances, options)
+      end
+    rescue ActiveRecord::ActiveRecordError
+      persist_each(group)
+    end
+
+    def changed_columns(group)
+      columns = group.flat_map { |processor| processor.instance.changed }.uniq
+      (columns - Array(importable.primary_key).map(&:to_s)).map(&:to_sym)
+    end
+
+    def persist_each(group)
+      group.each do |processor|
+        begin
+          importable.transaction(requires_new: true) do
+            processor.instance.save!
+            processor.after_persist
+          end
+        rescue => exception
+          processor.mark_failed(exception)
+        end
+      end
+    end
+  end
+end

data/lib/active_record_importer/data_processor.rb

@@ -1,44 +1,86 @@
 module ActiveRecordImporter
+  ##
+  # Prepares a single csv row: builds its attributes and find attributes,
+  # builds and validates the instance, and keeps the row errors.
+  # Persistence happens later, for the whole chunk, in BatchPersister.
+  ##
   class DataProcessor
     include Virtus.model
 
-    attribute :import, Import
     attribute :importable, Class
+    attribute :import
     attribute :insert_method, String, default: :set_insert_method
     attribute :row_attrs, Hash
     attribute :instance_attrs, Hash
     attribute :find_options, String, default: :set_find_options
     attribute :find_attributes, Hash
-    attribute :row_errors, Array
+    attribute :row_errors, String
     attribute :instance
+    attribute :lookup, Object, default: :default_lookup
 
     def process
+      prepare
+      build
+    end
+
+    ##
+    # Step 1: build the attributes used for the instance and the lookup
+    ##
+    def prepare
       fetch_instance_attributes
       fetch_find_attributes
-      create_instance
+      !failed?
     end
 
-    private
+    ##
+    # Step 2: build and validate the instance
+    ##
+    def build
+      return false if failed?
+
+      @instance =
+        InstanceBuilder.new(
+          importable: importable,
+          insert_method: insert_method,
+          find_attributes: find_attributes,
+          instance_attrs: attributes_without_state_machine_attrs,
+          lookup: lookup
+        ).build
+      true
+    rescue => exception
+      append_errors(exception)
+      false
+    end
 
-    def create_instance
-      ActiveRecord::Base.transaction do
-        begin
-          @instance =
-            InstanceBuilder.new(
-              importable: importable,
-              insert_method: insert_method,
-              find_attributes: find_attributes,
-              instance_attrs: attributes_without_state_machine_attrs
-            ).build
-
-          methods_after_upsert
-          true
-        rescue => exception
-          append_errors(exception, true)
-        end
-      end
+    ##
+    # Find attributes that need to be preloaded for this row
+    ##
+    def find_attributes_for_lookup
+      return if failed? || insert_method.inquiry.insert?
+      find_attributes
+    end
+
+    ##
+    # Called once the instance is persisted (per-row mode only)
+    ##
+    def after_persist
+      methods_after_upsert
+    end
+
+    def mark_failed(error)
+      append_errors(error)
+    end
+
+    def failed?
+      row_errors.present?
+    end
+
+    def ready?
+      !failed? && instance.present?
     end
 
+    private
+
     def fetch_instance_attributes
       @instance_attrs = Attribute::AttributesBuilder.new(
         importable, row_attrs
@@ -48,6 +90,8 @@ module ActiveRecordImporter
     end
 
     def fetch_find_attributes
+      return if failed?
+
       @find_attributes = Attribute::FindOptionsBuilder.new(
         importable: importable,
         find_options: find_options,
@@ -81,7 +125,7 @@ module ActiveRecordImporter
     end
 
     def attributes_without_state_machine_attrs
-      instance_attrs.except(*state_machine_attr)
+      instance_attrs.except(*Array(state_machine_attr))
     end
 
     def skip_callbacks?
@@ -94,12 +138,10 @@ module ActiveRecordImporter
       ImportCallbacker.new(instance, after_save).call
     end
 
-    def append_errors(error, rollback = false)
+    def append_errors(error)
       return if error.blank?
 
-      message = error.is_a?(Exception) ? error.message : error
-      @row_errors = message
-      fail ActiveRecord::Rollback if rollback
+      @row_errors = error.is_a?(Exception) ? error.message : error.to_s
     end
 
     def set_insert_method
@@ -111,5 +153,9 @@ module ActiveRecordImporter
     def set_find_options
       import.try(:find_options) || importer_options.find_options.join(',')
     end
+
+    def default_lookup
+      RecordPreloader.new(importable)
+    end
   end
 end

data/lib/active_record_importer/dispatcher.rb

@@ -2,13 +2,14 @@ module ActiveRecordImporter
   class Dispatcher
     include Virtus.model
 
-    attribute :import, Import
     attribute :importable, Class
     attribute :execute, Boolean, default: true
+    attribute :import
     attribute :import_file
 
     def call
       divide_and_conquer
+      create_import_failed_file
     end
 
     private
@@ -19,13 +20,14 @@ module ActiveRecordImporter
           queue_or_execute(collection)
         end
       end
+
       true
     end
 
     def csv_options
       opts = klass_csv_opts
-      return opts if import.nil? || import.batch_size.blank?
-      opts.merge(chunk_size: import.batch_size)
+      return opts if import.nil? || import.batch_size.to_i <= 0
+      opts.merge(chunk_size: import.batch_size.to_i)
     end
 
     def klass_csv_opts
@@ -45,8 +47,36 @@ module ActiveRecordImporter
       ).process!
     end
 
-    def queue(collection)
+    def queue(_collection)
       # To follow
     end
+
+    def create_import_failed_file
+      return if import.nil? || !File.exist?(temp_failed_file_path)
+      File.open(temp_failed_file_path) do |file|
+        import.failed_file = file
+
+        # I forced to save it as 'text/csv' because
+        # the file is being saved as 'text/x-pascal'
+        # and I still have no idea why?!?
+
+        import.failed_file_content_type = 'text/csv'
+        import.save!
+      end
+
+      destroy_temp_failed_file
+    end
+
+    def destroy_temp_failed_file
+      FileUtils.rm(temp_failed_file_path)
+    end
+
+    def temp_failed_file_path
+      "/tmp/#{target_file_name}"
+    end
+
+    def target_file_name
+      "failed_file_#{import.id}.csv"
+    end
   end
 end