activerecord-import 0.27.0 → 0.28.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 00e3b31f9b8fbe29c0ed8b6a58b6a1afeb0a3f67
-  data.tar.gz: b854ffc00c14c4cfc64a67819a367b35941cdb80
+  metadata.gz: dbd7badbb2e98c6725ddaf5edbb713c4f79bfc01
+  data.tar.gz: 0912df3959070820db94063d0172a43f4b027a9f
 SHA512:
-  metadata.gz: 164ff30dd1c1f19becb91dd2e7c0d3c9d022489b5065a337743b5eb84a497e58dd6ddaee43a401127cf06c60f3320d4501c962dd91885127b86acea4542315d9
-  data.tar.gz: 0dad63cbd8a90ee1f31483796a470088cbd4f35eca9579f707e394500ab6e2ab9524150e49d3a2bee60b79b367c8974855dd5af413c33b326ed4796d0852e118
+  metadata.gz: 22fb70781c58df1795c41e5b8e954b8a89e6c85be0dbf33f2c2fb9265847778f99f414cbd7fe9ea945d6aa43f89d7557ea41541f938fa357a7bbcb2c0e6114df
+  data.tar.gz: b480446464f626518ac128b2e79d4e4dbb2b50e8e52d6781812a0ee1594e5328430214ad03be9590c09dcfd07661ae54130b53ccde1b9b7d2131d2f40c80a633

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## Changes in 0.28.1
+
+### Fixes
+
+* Fix issue where ActiveRecord presence validations were being mutated.
+  Limited custom presence validation to bulk imports.
+
+## Changes in 0.28.0
+
+### New Features
+
+* Allow updated timestamps to be manually set.Thanks to @Rob117, @jkowens via \#570.
+
+### Fixes
+
+* Fix validating presence of belongs_to associations. Existence
+  of the parent record is not validated, but the foreign key field
+  cannot be empty. Thanks to @Rob117, @jkowens via \#575.
+
 ## Changes in 0.27.0
 
 ### New Features

data/README.markdown

@@ -21,12 +21,401 @@ and then the reviews:
 That would be about 4M SQL insert statements vs 3, which results in vastly improved performance. In our case, it converted
 an 18 hour batch process to <2 hrs.
 
+The gem provides the following high-level features:
+
+* activerecord-import can work with raw columns and arrays of values (fastest)
+* activerecord-import works with model objects (faster)
+* activerecord-import can perform validations (fast)
+* activerecord-import can perform on duplicate key updates (requires MySQL or Postgres 9.5+)
+
 ## Table of Contents
 
+* [Examples](#examples)
+  * [Introduction](#introduction)
+  * [Columns and Arrays](#columns-and-arrays)
+  * [Hashes](#hashes)
+  * [ActiveRecord Models](#activerecord-models)
+  * [Batching](#batching)
+  * [Recursive](#recursive)
+* [Options](#options)
+  * [Duplicate Key Ignore](#duplicate-key-ignore)
+  * [Duplicate Key Update](#duplicate-key-update)
+* [Return Info](#return-info)
+* [Counter Cache](#counter-cache)
+* [ActiveRecord Timestamps](#activerecord-timestamps)
 * [Callbacks](#callbacks)
+* [Supported Adapters](#supported-adapters)
 * [Additional Adapters](#additional-adapters)
+* [Requiring](#requiring)
+  * [Autoloading via Bundler](#autoloading-via-bundler)
+  * [Manually Loading](#manually-loading)
 * [Load Path Setup](#load-path-setup)
+* [Conflicts With Other Gems](#conflicts-with-other-gems)
 * [More Information](#more-information)
+* [Contributing](#contributing)
+  * [Running Tests](#running-tests)
+
+### Examples
+
+#### Introduction
+
+Without `activerecord-import`, you'd write something like this:
+
+```ruby
+10.times do |i|
+  Book.create! :name => "book #{i}"
+end
+```
+
+This would end up making 10 SQL calls. YUCK!  With `activerecord-import`, you can instead do this:
+
+```ruby
+books = []
+10.times do |i|
+  books << Book.new(:name => "book #{i}")
+end
+Book.import books    # or use import!
+```
+
+and only have 1 SQL call. Much better!
+
+#### Columns and Arrays
+
+The `import` method can take an array of column names (string or symbols) and an array of arrays. Each child array represents an individual record and its list of values in the same order as the columns. This is the fastest import mechanism and also the most primitive.
+
+```ruby
+columns = [ :title, :author ]
+values = [ ['Book1', 'FooManChu'], ['Book2', 'Bob Jones'] ]
+
+# Importing without model validations
+Book.import columns, values, :validate => false
+
+# Import with model validations
+Book.import columns, values, :validate => true
+
+# when not specified :validate defaults to true
+Book.import columns, values
+```
+
+#### Hashes
+
+The `import` method can take an array of hashes. The keys map to the column names in the database.
+
+```ruby
+values = [{ title: 'Book1', author: 'FooManChu' }, { title: 'Book2', author: 'Bob Jones'}]
+
+# Importing without model validations
+Book.import values, validate: false
+
+# Import with model validations
+Book.import values, validate: true
+
+# when not specified :validate defaults to true
+Book.import values
+```
+h2. Import Using Hashes and Explicit Column Names
+
+The `import` method can take an array of column names and an array of hash objects. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field:
+
+```ruby
+books = [
+  { title: "Book 1", author: "FooManChu" },
+  { title: "Book 2", author: "Bob Jones" }
+]
+columns = [ :title ]
+
+# without validations
+Book.import columns, books, validate: false
+
+# with validations
+Book.import columns, books, validate: true
+
+# when not specified :validate defaults to true
+Book.import columns, books
+
+# result in table books
+# title  | author
+#--------|--------
+# Book 1 | NULL
+# Book 2 | NULL
+
+```
+
+Using hashes will only work if the columns are consistent in every hash of the array. If this does not hold, an exception will be raised. There are two workarounds: use the array to instantiate an array of ActiveRecord objects and then pass that into `import` or divide the array into multiple ones with consistent columns and import each one separately.
+
+See https://github.com/zdennis/activerecord-import/issues/507 for discussion.
+
+```ruby
+arr = [
+  { bar: 'abc' },
+  { baz: 'xyz' },
+  { bar: '123', baz: '456' }
+]
+
+# An exception will be raised
+Foo.import arr
+
+# better
+arr.map! { |args| Foo.new(args) }
+Foo.import arr
+
+# better
+arr.group_by(&:keys).each_value do |v|
+ Foo.import v
+end
+```
+
+#### ActiveRecord Models
+
+The `import` method can take an array of models. The attributes will be pulled off from each model by looking at the columns available on the model.
+
+```ruby
+books = [
+  Book.new(:title => "Book 1", :author => "FooManChu"),
+  Book.new(:title => "Book 2", :author => "Bob Jones")
+]
+
+# without validations
+Book.import books, :validate => false
+
+# with validations
+Book.import books, :validate => true
+
+# when not specified :validate defaults to true
+Book.import books
+```
+
+The `import` method can take an array of column names and an array of models. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field:
+
+```ruby
+books = [
+  Book.new(:title => "Book 1", :author => "FooManChu"),
+  Book.new(:title => "Book 2", :author => "Bob Jones")
+]
+columns = [ :title ]
+
+# without validations
+Book.import columns, books, :validate => false
+
+# with validations
+Book.import columns, books, :validate => true
+
+# when not specified :validate defaults to true
+Book.import columns, books
+
+# result in table books
+# title  | author
+#--------|--------
+# Book 1 | NULL
+# Book 2 | NULL
+
+```
+
+#### Batching
+
+The `import` method can take a `batch_size` option to control the number of rows to insert per INSERT statement. The default is the total number of records being inserted so there is a single INSERT statement.
+
+```ruby
+books = [
+  Book.new(:title => "Book 1", :author => "FooManChu"),
+  Book.new(:title => "Book 2", :author => "Bob Jones"),
+  Book.new(:title => "Book 1", :author => "John Doe"),
+  Book.new(:title => "Book 2", :author => "Richard Wright")
+]
+columns = [ :title ]
+
+# 2 INSERT statements for 4 records
+Book.import columns, books, :batch_size => 2
+```
+
+#### Recursive
+
+NOTE: This only works with PostgreSQL.
+
+Assume that Books <code>has_many</code> Reviews.
+
+```ruby
+books = []
+10.times do |i|
+  book = Book.new(:name => "book #{i}")
+  book.reviews.build(:title => "Excellent")
+  books << book
+end
+Book.import books, recursive: true
+```
+
+### Options
+
+Key                     | Options               | Default            | Description
+----------------------- | --------------------- | ------------------ | -----------
+:validate               | `true`/`false`        | `true`             | Whether or not to run `ActiveRecord` validations (uniqueness skipped).
+:validate_uniqueness    | `true`/`false`        | `false`            | Whether or not to run uniqueness validations, has potential pitfalls, use with caution (requires `>= v0.27.0`).
+:on_duplicate_key_ignore| `true`/`false`        | `false`            | Allows skipping records with duplicate keys. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-ignore) for more details.
+:ignore                 | `true`/`false`        | `false`            | Alias for :on_duplicate_key_ignore.
+:on_duplicate_key_update| :all, `Array`, `Hash` | N/A                | Allows upsert logic to be used. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-update) for more details.
+:synchronize            | `Array`               | N/A                | An array of ActiveRecord instances. This synchronizes existing instances in memory with updates from the import.
+:timestamps             | `true`/`false`        | `true`             | Enables/disables timestamps on imported records.
+:recursive              | `true`/`false`        | `false`            | Imports has_many/has_one associations (PostgreSQL only).
+:batch_size             | `Integer`             | total # of records | Max number of records to insert per import
+:raise_error            | `true`/`false`        | `false`            | Throws an exception if there are invalid records. `import!` is a shortcut for this.
+
+
+#### Duplicate Key Ignore
+
+[MySQL](http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html), [SQLite](https://www.sqlite.org/lang_insert.html), and [PostgreSQL](https://www.postgresql.org/docs/current/static/sql-insert.html#SQL-ON-CONFLICT) (9.5+) support `on_duplicate_key_ignore` which allows you to skip records if a primary or unique key constraint is violated.
+
+For Postgres 9.5+ it adds `ON CONFLICT DO NOTHING`, for MySQL it uses `INSERT IGNORE`, and for SQLite it uses `INSERT OR IGNORE`. Cannot be enabled on a recursive import. For database adapters that normally support setting primary keys on imported objects, this option prevents that from occurring.
+
+```ruby
+book = Book.create! title: "Book1", author: "FooManChu"
+book.title = "Updated Book Title"
+book.author = "Bob Barker"
+
+Book.import [book], on_duplicate_key_ignore: true
+
+book.reload.title  # => "Book1"     (stayed the same)
+book.reload.author # => "FooManChu" (stayed the same)
+```
+
+The option `:on_duplicate_key_ignore` is bypassed when `:recursive` is enabled for [PostgreSQL imports](https://github.com/zdennis/activerecord-import/wiki#recursive-example-postgresql-only).
+
+#### Duplicate Key Update
+
+MySQL, PostgreSQL (9.5+), and SQLite (3.24.0+) support `on duplicate key update` (also known as "upsert") which allows you to specify fields whose values should be updated if a primary or unique key constraint is violated.
+
+One big difference between MySQL and PostgreSQL support is that MySQL will handle any conflict that happens, but PostgreSQL requires that you specify which columns the conflict would occur over. SQLite models its upsert support after PostgreSQL.
+
+This will use MySQL's `ON DUPLICATE KEY UPDATE` or Postgres/SQLite `ON CONFLICT DO UPDATE` to do upsert.
+
+Basic Update
+
+```ruby
+book = Book.create! title: "Book1", author: "FooManChu"
+book.title = "Updated Book Title"
+book.author = "Bob Barker"
+
+# MySQL version
+Book.import [book], on_duplicate_key_update: [:title]
+
+# PostgreSQL version
+Book.import [book], on_duplicate_key_update: {conflict_target: [:id], columns: [:title]}
+
+# PostgreSQL shorthand version (conflict target must be primary key)
+Book.import [book], on_duplicate_key_update: [:title]
+
+book.reload.title  # => "Updated Book Title" (changed)
+book.reload.author # => "FooManChu"          (stayed the same)
+```
+
+Using the value from another column
+
+```ruby
+book = Book.create! title: "Book1", author: "FooManChu"
+book.title = "Updated Book Title"
+
+# MySQL version
+Book.import [book], on_duplicate_key_update: {author: :title}
+
+# PostgreSQL version (no shorthand version)
+Book.import [book], on_duplicate_key_update: {
+  conflict_target: [:id], columns: {author: :title}
+}
+
+book.reload.title  # => "Book1"              (stayed the same)
+book.reload.author # => "Updated Book Title" (changed)
+```
+
+Using Custom SQL
+
+```ruby
+book = Book.create! title: "Book1", author: "FooManChu"
+book.author = "Bob Barker"
+
+# MySQL version
+Book.import [book], on_duplicate_key_update: "author = values(author)"
+
+# PostgreSQL version
+Book.import [book], on_duplicate_key_update: {
+  conflict_target: [:id], columns: "author = excluded.author"
+}
+
+# PostgreSQL shorthand version (conflict target must be primary key)
+Book.import [book], on_duplicate_key_update: "author = excluded.author"
+
+book.reload.title  # => "Book1"      (stayed the same)
+book.reload.author # => "Bob Barker" (changed)
+```
+
+PostgreSQL Using constraints
+
+```ruby
+book = Book.create! title: "Book1", author: "FooManChu", edition: 3, published_at: nil
+book.published_at = Time.now
+
+# in migration
+execute <<-SQL
+      ALTER TABLE books
+        ADD CONSTRAINT for_upsert UNIQUE (title, author, edition);
+    SQL
+
+# PostgreSQL version
+Book.import [book], on_duplicate_key_update: {constraint_name: :for_upsert, columns: [:published_at]}
+
+
+book.reload.title  # => "Book1"          (stayed the same)
+book.reload.author # => "FooManChu"      (stayed the same)
+book.reload.edition # => 3               (stayed the same)
+book.reload.published_at # => 2017-10-09 (changed)
+```
+
+```ruby
+Book.import books, validate_uniqueness: true
+```
+
+### Return Info
+
+The `import` method returns a `Result` object that responds to `failed_instances` and `num_inserts`. Additionally, for users of Postgres, there will be two arrays `ids` and `results` that can be accessed`.
+
+```ruby
+articles = [
+  Article.new(author_id: 1, title: 'First Article', content: 'This is the first article'),
+  Article.new(author_id: 2, title: 'Second Article', content: ''),
+  Article.new(author_id: 3, content: '')
+]
+
+demo = Article.import(articles), returning: :title # => #<struct ActiveRecord::Import::Result
+
+demo.failed_instances
+=> [#<Article id: 3, author_id: 3, title: nil, content: "", created_at: nil, updated_at: nil>]
+
+demo.num_inserts
+=> 1,
+
+demo.ids
+=> ["1", "2"] # for Postgres
+=> [] # for other DBs
+
+demo.results
+=> ["First Article", "Second Article"] # for Postgres
+=> [] for other DBs
+```
+
+### Counter Cache
+
+When running `import`, `activerecord-import` does not automatically update counter cache columns. To update these columns, you will need to do one of the following:
+
+* Provide values to the column as an argument on your object that is passed in.
+* Manually update the column after the record has been imported.
+
+### ActiveRecord Timestamps
+
+If you're familiar with ActiveRecord you're probably familiar with its timestamp columns: created_at, created_on, updated_at, updated_on, etc. When importing data the timestamp fields will continue to work as expected and each timestamp column will be set.
+
+Should you wish to specify those columns, you may use the option `timestamps: false`.
+
+However, it is also possible to set just `:created_at` in specific records. In this case despite using `timestamps: true`,  `:created_at` will be updated only in records where that field is `nil`. Same rule applies for record associations when enabling the option `recursive: true`.
+
+If you are using custom time zones, these will be respected when performing imports as well as long as `ActiveRecord::Base.default_timezone` is set, which for practically all Rails apps it is
 
 ### Callbacks
 
@@ -34,7 +423,7 @@ ActiveRecord callbacks related to [creating](http://guides.rubyonrails.org/activ
 
 If you do have a collection of in-memory ActiveRecord objects you can do something like this:
 
-```
+```ruby
 books.each do |book|
   book.run_callbacks(:save) { false }
   book.run_callbacks(:create) { false }
@@ -46,7 +435,7 @@ This will run before_create and before_save callbacks on each item. The `false`
 
 If that is an issue, another possible approach is to loop through your models first to do validations and then only run callbacks on and import the valid models.
 
-```
+```ruby
 valid_books = []
 invalid_books = []
 
@@ -66,6 +455,24 @@ end
 Book.import valid_books, validate: false
 ```
 
+### Supported Adapters
+
+The following database adapters are currently supported:
+
+* MySQL - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.1.0 and higher)
+* MySQL2 - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.2.0 and higher)
+* PostgreSQL - supports core import functionality (included in activerecord-import 0.1.0 and higher)
+* SQLite3 - supports core import functionality (included in activerecord-import 0.1.0 and higher)
+* Oracle - supports core import functionality through DML trigger (available as an external gem: [activerecord-import-oracle_enhanced](https://github.com/keeguon/activerecord-import-oracle_enhanced)
+* SQL Server - supports core import functionality (available as an external gem: [activerecord-import-sqlserver](https://github.com/keeguon/activerecord-import-sqlserver)
+
+If your adapter isn't listed here, please consider creating an external gem as described in the README to provide support. If you do, feel free to update this wiki to include a link to the new adapter's repository!
+
+To test which features are supported by your adapter, use the following methods on a model class:
+* `supports_import?(*args)`
+* `supports_on_duplicate_key_update?`
+* `supports_setting_primary_key_of_imported_objects?`
+
 ### Additional Adapters
 
 Additional adapters can be provided by gems external to activerecord-import by providing an adapter that matches the naming convention setup by activerecord-import (and subsequently activerecord) for dynamically loading adapters.  This involves also providing a folder on the load path that follows the activerecord-import naming convention to allow activerecord-import to dynamically load the file.
@@ -73,17 +480,54 @@ Additional adapters can be provided by gems external to activerecord-import by p
 When `ActiveRecord::Import.require_adapter("fake_name")` is called the require will be:
 
 ```ruby
-  require 'activerecord-import/active_record/adapters/fake_name_adapter'
+require 'activerecord-import/active_record/adapters/fake_name_adapter'
 ```
 
 This allows an external gem to dynamically add an adapter without the need to add any file/code to the core activerecord-import gem.
 
+### Requiring
+
+Note: These instructions will only work if you are using version 0.2.0 or higher.
+
+#### Autoloading via Bundler
+
+If you are using Rails or otherwise autoload your dependencies via Bundler, all you need to do add the gem to your `Gemfile` like so:
+
+```ruby
+gem 'activerecord-import'
+```
+
+#### Manually Loading
+
+You may want to manually load activerecord-import for one reason or another. First, add the `require: false` argument like so:
+
+```ruby
+gem 'activerecord-import', require: false
+```
+
+This will allow you to load up activerecord-import in the file or files where you are using it and only load the parts you need.
+If you are doing this within Rails and ActiveRecord has established a database connection (such as within a controller), you will need to do extra initialization work:
+
+```ruby
+require 'activerecord-import/base'
+# load the appropriate database adapter (postgresql, mysql2, sqlite3, etc)
+require 'activerecord-import/active_record/adapters/postgresql_adapter'
+```
+
+If your gem dependencies aren’t autoloaded, and your script will be establishing a database connection, then simply require activerecord-import after ActiveRecord has been loaded, i.e.:
+
+```ruby
+require 'active_record'
+require 'activerecord-import'
+```
+
 ### Load Path Setup
 To understand how rubygems loads code you can reference the following:
 
   http://guides.rubygems.org/patterns/#loading_code
 
 And an example of how active_record dynamically load adapters:
+
   https://github.com/rails/rails/blob/master/activerecord/lib/active_record/connection_adapters/connection_specification.rb
 
 In summary, when a gem is loaded rubygems adds the `lib` folder of the gem to the global load path `$LOAD_PATH` so that all `require` lookups will not propagate through all of the folders on the load path. When a `require` is issued each folder on the `$LOAD_PATH` is checked for the file and/or folder referenced. This allows a gem (like activerecord-import) to define push the activerecord-import folder (or namespace) on the `$LOAD_PATH` and any adapters provided by activerecord-import will be found by rubygems when the require is issued.
@@ -105,12 +549,62 @@ activerecord-import-fake_name/
 
 When rubygems pushes the `lib` folder onto the load path a `require` will now find `activerecord-import/active_record/adapters/fake_name_adapter` as it runs through the lookup process for a ruby file under that path in `$LOAD_PATH`
 
+
+### Conflicts With Other Gems
+
+`activerecord-import` adds the `.import` method onto `ActiveRecord::Base`. There are other gems, such as `elasticsearch-rails`, that do the same thing. In conflicts such as this, there is an aliased method named `.bulk_import` that can be used interchangeably.
+
+If you are using the `apartment` gem, there is a weird triple interaction between that gem, `activerecord-import`, and `activerecord` involving caching of the `sequence_name` of a model. This can be worked around by explcitly setting this value within the model. For example:
+
+```ruby
+class Post < ActiveRecord::Base
+  self.sequence_name = "posts_seq"
+end
+```
+
+Another way to work around the issue is to call `.reset_sequence_name` on the model. For example:
+
+```ruby
+schemas.all.each do |schema|
+  Apartment::Tenant.switch! schema.name
+  ActiveRecord::Base.transaction do
+    Post.reset_sequence_name
+
+    Post.import posts
+  end
+end
+```
+
+See https://github.com/zdennis/activerecord-import/issues/233 for further discussion.
+
 ### More Information
 
 For more information on activerecord-import please see its wiki: https://github.com/zdennis/activerecord-import/wiki
 
 To document new information, please add to the README instead of the wiki. See https://github.com/zdennis/activerecord-import/issues/397 for discussion.
 
+### Contributing
+
+#### Running Tests
+
+The first thing you need to do is set up your database(s):
+
+* copy `test/database.yml.sample` to `test/database.yml`
+* modify `test/database.yml` for your database settings
+* create databases as needed
+
+After that, you can run the tests. They run against multiple tests and ActiveRecord versions.
+
+This is one example of how to run the tests:
+
+```ruby
+rm Gemfile.lock
+AR_VERSION=4.2 bundle install
+AR_VERSION=4.2 bundle exec rake test:postgresql test:sqlite3 test:mysql2
+```
+
+Once you have pushed up your changes, you can find your CI results [here](https://travis-ci.org/zdennis/activerecord-import/).
+
 # License
 
 This is licensed under the ruby license.
@@ -131,3 +625,4 @@ Zach Dennis (zach.dennis@gmail.com)
 * Thibaud Guillaume-Gentil
 * Mark Van Holstyn
 * Victor Costan
+* Dillon Welch

data/lib/activerecord-import/import.rb

@@ -24,33 +24,57 @@ module ActiveRecord::Import #:nodoc:
   end
 
   class Validator
-    def initialize(options = {})
+    def initialize(klass, options = {})
       @options = options
+      init_validations(klass)
+    end
+
+    def init_validations(klass)
+      @validate_callbacks = klass._validate_callbacks.dup
+
+      klass._validate_callbacks.each_with_index do |callback, i|
+        filter = callback.raw_filter
+
+        if filter.class.name =~ /Validations::PresenceValidator/
+          callback = callback.dup
+          filter = filter.dup
+          associations = klass.reflect_on_all_associations(:belongs_to)
+          attrs = filter.instance_variable_get(:@attributes).dup
+          associations.each do |assoc|
+            if (index = attrs.index(assoc.name))
+              key = assoc.foreign_key.to_sym
+              attrs[index] = key unless attrs.include?(key)
+            end
+          end
+          filter.instance_variable_set(:@attributes, attrs)
+          if @validate_callbacks.respond_to?(:chain, true)
+            @validate_callbacks.send(:chain).tap do |chain|
+              callback.instance_variable_set(:@filter, filter)
+              chain[i] = callback
+            end
+          else
+            callback.raw_filter = filter
+            callback.filter = callback.send(:_compile_filter, filter)
+            @validate_callbacks[i] = callback
+          end
+        elsif !@options[:validate_uniqueness] && filter.is_a?(ActiveRecord::Validations::UniquenessValidator)
+          @validate_callbacks.delete(callback)
+        end
+      end
     end
 
     def valid_model?(model)
       validation_context = @options[:validate_with_context]
       validation_context ||= (model.new_record? ? :create : :update)
-
       current_context = model.send(:validation_context)
+
       begin
         model.send(:validation_context=, validation_context)
         model.errors.clear
 
-        validate_callbacks = model._validate_callbacks.dup
-        associations = model.class.reflect_on_all_associations(:belongs_to).map(&:name)
-
-        model._validate_callbacks.each do |callback|
-          filter = callback.raw_filter
-          if (!@options[:validate_uniqueness] && filter.is_a?(ActiveRecord::Validations::UniquenessValidator)) ||
-             (defined?(ActiveRecord::Validations::PresenceValidator) && filter.is_a?(ActiveRecord::Validations::PresenceValidator) && associations.include?(filter.attributes.first))
-            validate_callbacks.delete(callback)
-          end
-        end
-
         model.run_callbacks(:validation) do
           if defined?(ActiveSupport::Callbacks::Filters::Environment) # ActiveRecord >= 4.1
-            runner = validate_callbacks.compile
+            runner = @validate_callbacks.compile
             env = ActiveSupport::Callbacks::Filters::Environment.new(model, false, nil)
             if runner.respond_to?(:call) # ActiveRecord < 5.1
               runner.call(env)
@@ -69,10 +93,10 @@ module ActiveRecord::Import #:nodoc:
               runner.invoke_before(env)
               runner.invoke_after(env)
             end
-          elsif validate_callbacks.method(:compile).arity == 0 # ActiveRecord = 4.0
-            model.instance_eval validate_callbacks.compile
+          elsif @validate_callbacks.method(:compile).arity == 0 # ActiveRecord = 4.0
+            model.instance_eval @validate_callbacks.compile
           else # ActiveRecord 3.x
-            model.instance_eval validate_callbacks.compile(nil, model)
+            model.instance_eval @validate_callbacks.compile(nil, model)
           end
         end
 
@@ -520,7 +544,7 @@ class ActiveRecord::Base
       options[:locking_column] = locking_column if attribute_names.include?(locking_column)
 
       is_validating = options[:validate_with_context].present? ? true : options[:validate]
-      validator = ActiveRecord::Import::Validator.new(options)
+      validator = ActiveRecord::Import::Validator.new(self, options)
 
       # assume array of model objects
       if args.last.is_a?( Array ) && args.last.first.is_a?(ActiveRecord::Base)
@@ -553,6 +577,14 @@ class ActiveRecord::Base
           serialized_attributes
         end
 
+        update_attrs = if record_timestamps && options[:timestamps]
+          if respond_to?(:timestamp_attributes_for_update, true)
+            send(:timestamp_attributes_for_update).map(&:to_sym)
+          else
+            new.send(:timestamp_attributes_for_update_in_model)
+          end
+        end
+
         array_of_attributes = []
 
         models.each do |model|
@@ -566,9 +598,13 @@ class ActiveRecord::Base
           end
 
           array_of_attributes << column_names.map do |name|
-            if stored_attrs.key?(name.to_sym) ||
-               serialized_attrs.key?(name) ||
-               default_values.key?(name.to_s)
+            if model.persisted? &&
+               update_attrs && update_attrs.include?(name.to_sym) &&
+               !model.send("#{name}_changed?")
+              nil
+            elsif stored_attrs.key?(name.to_sym) ||
+                  serialized_attrs.key?(name.to_s) ||
+                  default_values[name.to_s]
               model.read_attribute(name.to_s)
             else
               model.read_attribute_before_type_cast(name.to_s)
@@ -655,7 +691,7 @@ class ActiveRecord::Base
       timestamps = {}
 
       # record timestamps unless disabled in ActiveRecord::Base
-      if record_timestamps && options.delete( :timestamps )
+      if record_timestamps && options[:timestamps]
         timestamps = add_special_rails_stamps column_names, array_of_attributes, options
       end
 
@@ -961,7 +997,7 @@ class ActiveRecord::Base
           index = column_names.index(column) || column_names.index(column.to_sym)
           if index
             # replace every instance of the array of attributes with our value
-            array_of_attributes.each { |arr| arr[index] = timestamp if arr[index].nil? || action == :update }
+            array_of_attributes.each { |arr| arr[index] = timestamp if arr[index].nil? }
           else
             column_names << column
             array_of_attributes.each { |arr| arr << timestamp }

data/lib/activerecord-import/version.rb

@@ -1,5 +1,5 @@
 module ActiveRecord
   module Import
-    VERSION = "0.27.0".freeze
+    VERSION = "0.28.1".freeze
   end
 end

data/test/import_test.rb

@@ -303,6 +303,27 @@ describe "#import" do
         end
       end
     end
+
+    context "when validatoring presence of belongs_to association" do
+      it "should not import records without foreign key" do
+        assert_no_difference "UserToken.count" do
+          UserToken.import [:token], [['12345abcdef67890']]
+        end
+      end
+
+      it "should import records with foreign key" do
+        assert_difference "UserToken.count", +1 do
+          UserToken.import [:user_name, :token], [%w("Bob", "12345abcdef67890")]
+        end
+      end
+
+      it "should not mutate the defined validations" do
+        UserToken.import [:user_name, :token], [%w("Bob", "12345abcdef67890")]
+        ut = UserToken.new
+        ut.valid?
+        assert_includes ut.errors.messages, :user
+      end
+    end
   end
 
   context "without :validation option" do
@@ -499,11 +520,11 @@ describe "#import" do
 
     context "when the timestamps columns are present" do
       setup do
-        @existing_book = Book.create(title: "Fell", author_name: "Curry", publisher: "Bayer", created_at: 2.years.ago.utc, created_on: 2.years.ago.utc)
+        @existing_book = Book.create(title: "Fell", author_name: "Curry", publisher: "Bayer", created_at: 2.years.ago.utc, created_on: 2.years.ago.utc, updated_at: 2.years.ago.utc, updated_on: 2.years.ago.utc)
         ActiveRecord::Base.default_timezone = :utc
         Timecop.freeze(time) do
           assert_difference "Book.count", +2 do
-            Book.import %w(title author_name publisher created_at created_on), [["LDAP", "Big Bird", "Del Rey", nil, nil], [@existing_book.title, @existing_book.author_name, @existing_book.publisher, @existing_book.created_at, @existing_book.created_on]]
+            Book.import %w(title author_name publisher created_at created_on updated_at updated_on), [["LDAP", "Big Bird", "Del Rey", nil, nil, nil, nil], [@existing_book.title, @existing_book.author_name, @existing_book.publisher, @existing_book.created_at, @existing_book.created_on, @existing_book.updated_at, @existing_book.updated_on]]
           end
         end
         @new_book, @existing_book = Book.last 2
@@ -532,6 +553,23 @@ describe "#import" do
       it "should set the updated_on column for new records" do
         assert_in_delta time.to_i, @new_book.updated_on.to_i, 1.second
       end
+
+      it "should not set the updated_at column for existing records" do
+        assert_equal 2.years.ago.utc.strftime("%Y:%d"), @existing_book.updated_at.strftime("%Y:%d")
+      end
+
+      it "should not set the updated_on column for existing records" do
+        assert_equal 2.years.ago.utc.strftime("%Y:%d"), @existing_book.updated_on.strftime("%Y:%d")
+      end
+
+      it "should not set the updated_at column on models if changed" do
+        timestamp = Time.now.utc
+        books = [
+          Book.new(author_name: "Foo", title: "Baz", created_at: timestamp, updated_at: timestamp)
+        ]
+        Book.import books
+        assert_equal timestamp.strftime("%Y:%d"), Book.last.updated_at.strftime("%Y:%d")
+      end
     end
 
     context "when a custom time zone is set" do

data/test/models/user_token.rb

@@ -1,3 +1,4 @@
 class UserToken < ActiveRecord::Base
   belongs_to :user, primary_key: :name, foreign_key: :user_name
+  validates :user, presence: true
 end

data/test/support/shared_examples/on_duplicate_key_update.rb

@@ -246,6 +246,26 @@ def should_support_basic_on_duplicate_key_update
         end
       end
 
+      context "with timestamps enabled" do
+        let(:time) { Chronic.parse("5 minutes from now") }
+
+        it 'should not overwrite changed updated_at with current timestamp' do
+          topic = Topic.create(author_name: "Jane Doe", title: "Book")
+          timestamp = Time.now.utc
+          topic.updated_at = timestamp
+          Topic.import [topic], on_duplicate_key_update: :all, validate: false
+          assert_equal timestamp.to_s, Topic.last.updated_at.to_s
+        end
+
+        it 'should update updated_at with current timestamp' do
+          topic = Topic.create(author_name: "Jane Doe", title: "Book")
+          Timecop.freeze(time) do
+            Topic.import [topic], on_duplicate_key_update: [:updated_at], validate: false
+            assert_in_delta time.to_i, topic.reload.updated_at.to_i, 1.second
+          end
+        end
+      end
+
       context "with validation checks turned off" do
         asssertion_group(:should_support_on_duplicate_key_update) do
           should_not_update_fields_not_mentioned

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: activerecord-import
 version: !ruby/object:Gem::Version
-  version: 0.27.0
+  version: 0.28.1
 platform: ruby
 authors:
 - Zach Dennis
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-10-09 00:00:00.000000000 Z
+date: 2019-01-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord