activerecord-import 0.19.0 → 1.0.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (43) hide show
  1. checksums.yaml +5 -5
  2. data/.travis.yml +22 -12
  3. data/CHANGELOG.md +166 -0
  4. data/Gemfile +13 -10
  5. data/README.markdown +548 -5
  6. data/Rakefile +2 -1
  7. data/benchmarks/lib/cli_parser.rb +2 -1
  8. data/gemfiles/5.1.gemfile +1 -0
  9. data/gemfiles/5.2.gemfile +2 -0
  10. data/lib/activerecord-import/adapters/abstract_adapter.rb +2 -2
  11. data/lib/activerecord-import/adapters/mysql_adapter.rb +16 -10
  12. data/lib/activerecord-import/adapters/postgresql_adapter.rb +59 -15
  13. data/lib/activerecord-import/adapters/sqlite3_adapter.rb +126 -3
  14. data/lib/activerecord-import/base.rb +4 -6
  15. data/lib/activerecord-import/import.rb +384 -126
  16. data/lib/activerecord-import/synchronize.rb +1 -1
  17. data/lib/activerecord-import/value_sets_parser.rb +14 -0
  18. data/lib/activerecord-import/version.rb +1 -1
  19. data/lib/activerecord-import.rb +2 -15
  20. data/test/adapters/makara_postgis.rb +1 -0
  21. data/test/import_test.rb +148 -14
  22. data/test/makara_postgis/import_test.rb +8 -0
  23. data/test/models/account.rb +3 -0
  24. data/test/models/bike_maker.rb +7 -0
  25. data/test/models/topic.rb +10 -0
  26. data/test/models/user.rb +3 -0
  27. data/test/models/user_token.rb +4 -0
  28. data/test/schema/generic_schema.rb +20 -0
  29. data/test/schema/mysql2_schema.rb +19 -0
  30. data/test/schema/postgresql_schema.rb +1 -0
  31. data/test/schema/sqlite3_schema.rb +13 -0
  32. data/test/support/factories.rb +9 -8
  33. data/test/support/generate.rb +6 -6
  34. data/test/support/mysql/import_examples.rb +14 -2
  35. data/test/support/postgresql/import_examples.rb +142 -0
  36. data/test/support/shared_examples/on_duplicate_key_update.rb +252 -1
  37. data/test/support/shared_examples/recursive_import.rb +41 -11
  38. data/test/support/sqlite3/import_examples.rb +187 -10
  39. data/test/synchronize_test.rb +8 -0
  40. data/test/test_helper.rb +9 -1
  41. data/test/value_sets_bytes_parser_test.rb +13 -2
  42. metadata +20 -5
  43. data/test/schema/mysql_schema.rb +0 -16
data/README.markdown CHANGED
@@ -21,27 +21,513 @@ and then the reviews:
21
21
  That would be about 4M SQL insert statements vs 3, which results in vastly improved performance. In our case, it converted
22
22
  an 18 hour batch process to <2 hrs.
23
23
 
24
- ### More Information : Usage and Examples in Wiki
24
+ The gem provides the following high-level features:
25
25
 
26
- For more information on activerecord-import please see its wiki: https://github.com/zdennis/activerecord-import/wiki
26
+ * activerecord-import can work with raw columns and arrays of values (fastest)
27
+ * activerecord-import works with model objects (faster)
28
+ * activerecord-import can perform validations (fast)
29
+ * activerecord-import can perform on duplicate key updates (requires MySQL or Postgres 9.5+)
30
+
31
+ ## Table of Contents
32
+
33
+ * [Examples](#examples)
34
+ * [Introduction](#introduction)
35
+ * [Columns and Arrays](#columns-and-arrays)
36
+ * [Hashes](#hashes)
37
+ * [ActiveRecord Models](#activerecord-models)
38
+ * [Batching](#batching)
39
+ * [Recursive](#recursive)
40
+ * [Options](#options)
41
+ * [Duplicate Key Ignore](#duplicate-key-ignore)
42
+ * [Duplicate Key Update](#duplicate-key-update)
43
+ * [Return Info](#return-info)
44
+ * [Counter Cache](#counter-cache)
45
+ * [ActiveRecord Timestamps](#activerecord-timestamps)
46
+ * [Callbacks](#callbacks)
47
+ * [Supported Adapters](#supported-adapters)
48
+ * [Additional Adapters](#additional-adapters)
49
+ * [Requiring](#requiring)
50
+ * [Autoloading via Bundler](#autoloading-via-bundler)
51
+ * [Manually Loading](#manually-loading)
52
+ * [Load Path Setup](#load-path-setup)
53
+ * [Conflicts With Other Gems](#conflicts-with-other-gems)
54
+ * [More Information](#more-information)
55
+ * [Contributing](#contributing)
56
+ * [Running Tests](#running-tests)
57
+
58
+ ### Examples
59
+
60
+ #### Introduction
61
+
62
+ Without `activerecord-import`, you'd write something like this:
63
+
64
+ ```ruby
65
+ 10.times do |i|
66
+ Book.create! :name => "book #{i}"
67
+ end
68
+ ```
69
+
70
+ This would end up making 10 SQL calls. YUCK! With `activerecord-import`, you can instead do this:
71
+
72
+ ```ruby
73
+ books = []
74
+ 10.times do |i|
75
+ books << Book.new(:name => "book #{i}")
76
+ end
77
+ Book.import books # or use import!
78
+ ```
79
+
80
+ and only have 1 SQL call. Much better!
81
+
82
+ #### Columns and Arrays
83
+
84
+ The `import` method can take an array of column names (string or symbols) and an array of arrays. Each child array represents an individual record and its list of values in the same order as the columns. This is the fastest import mechanism and also the most primitive.
85
+
86
+ ```ruby
87
+ columns = [ :title, :author ]
88
+ values = [ ['Book1', 'FooManChu'], ['Book2', 'Bob Jones'] ]
89
+
90
+ # Importing without model validations
91
+ Book.import columns, values, :validate => false
92
+
93
+ # Import with model validations
94
+ Book.import columns, values, :validate => true
95
+
96
+ # when not specified :validate defaults to true
97
+ Book.import columns, values
98
+ ```
99
+
100
+ #### Hashes
101
+
102
+ The `import` method can take an array of hashes. The keys map to the column names in the database.
103
+
104
+ ```ruby
105
+ values = [{ title: 'Book1', author: 'FooManChu' }, { title: 'Book2', author: 'Bob Jones'}]
106
+
107
+ # Importing without model validations
108
+ Book.import values, validate: false
109
+
110
+ # Import with model validations
111
+ Book.import values, validate: true
112
+
113
+ # when not specified :validate defaults to true
114
+ Book.import values
115
+ ```
116
+ #### Import Using Hashes and Explicit Column Names
117
+
118
+ The `import` method can take an array of column names and an array of hash objects. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field:
119
+
120
+ ```ruby
121
+ books = [
122
+ { title: "Book 1", author: "FooManChu" },
123
+ { title: "Book 2", author: "Bob Jones" }
124
+ ]
125
+ columns = [ :title ]
126
+
127
+ # without validations
128
+ Book.import columns, books, validate: false
129
+
130
+ # with validations
131
+ Book.import columns, books, validate: true
132
+
133
+ # when not specified :validate defaults to true
134
+ Book.import columns, books
135
+
136
+ # result in table books
137
+ # title | author
138
+ #--------|--------
139
+ # Book 1 | NULL
140
+ # Book 2 | NULL
141
+
142
+ ```
143
+
144
+ Using hashes will only work if the columns are consistent in every hash of the array. If this does not hold, an exception will be raised. There are two workarounds: use the array to instantiate an array of ActiveRecord objects and then pass that into `import` or divide the array into multiple ones with consistent columns and import each one separately.
145
+
146
+ See https://github.com/zdennis/activerecord-import/issues/507 for discussion.
147
+
148
+ ```ruby
149
+ arr = [
150
+ { bar: 'abc' },
151
+ { baz: 'xyz' },
152
+ { bar: '123', baz: '456' }
153
+ ]
154
+
155
+ # An exception will be raised
156
+ Foo.import arr
157
+
158
+ # better
159
+ arr.map! { |args| Foo.new(args) }
160
+ Foo.import arr
161
+
162
+ # better
163
+ arr.group_by(&:keys).each_value do |v|
164
+ Foo.import v
165
+ end
166
+ ```
167
+
168
+ #### ActiveRecord Models
169
+
170
+ The `import` method can take an array of models. The attributes will be pulled off from each model by looking at the columns available on the model.
171
+
172
+ ```ruby
173
+ books = [
174
+ Book.new(:title => "Book 1", :author => "FooManChu"),
175
+ Book.new(:title => "Book 2", :author => "Bob Jones")
176
+ ]
177
+
178
+ # without validations
179
+ Book.import books, :validate => false
180
+
181
+ # with validations
182
+ Book.import books, :validate => true
183
+
184
+ # when not specified :validate defaults to true
185
+ Book.import books
186
+ ```
187
+
188
+ The `import` method can take an array of column names and an array of models. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field:
189
+
190
+ ```ruby
191
+ books = [
192
+ Book.new(:title => "Book 1", :author => "FooManChu"),
193
+ Book.new(:title => "Book 2", :author => "Bob Jones")
194
+ ]
195
+ columns = [ :title ]
196
+
197
+ # without validations
198
+ Book.import columns, books, :validate => false
199
+
200
+ # with validations
201
+ Book.import columns, books, :validate => true
202
+
203
+ # when not specified :validate defaults to true
204
+ Book.import columns, books
205
+
206
+ # result in table books
207
+ # title | author
208
+ #--------|--------
209
+ # Book 1 | NULL
210
+ # Book 2 | NULL
211
+
212
+ ```
213
+
214
+ #### Batching
215
+
216
+ The `import` method can take a `batch_size` option to control the number of rows to insert per INSERT statement. The default is the total number of records being inserted so there is a single INSERT statement.
217
+
218
+ ```ruby
219
+ books = [
220
+ Book.new(:title => "Book 1", :author => "FooManChu"),
221
+ Book.new(:title => "Book 2", :author => "Bob Jones"),
222
+ Book.new(:title => "Book 1", :author => "John Doe"),
223
+ Book.new(:title => "Book 2", :author => "Richard Wright")
224
+ ]
225
+ columns = [ :title ]
226
+
227
+ # 2 INSERT statements for 4 records
228
+ Book.import columns, books, :batch_size => 2
229
+ ```
230
+
231
+ #### Recursive
232
+
233
+ NOTE: This only works with PostgreSQL.
234
+
235
+ Assume that Books <code>has_many</code> Reviews.
236
+
237
+ ```ruby
238
+ books = []
239
+ 10.times do |i|
240
+ book = Book.new(:name => "book #{i}")
241
+ book.reviews.build(:title => "Excellent")
242
+ books << book
243
+ end
244
+ Book.import books, recursive: true
245
+ ```
246
+
247
+ ### Options
248
+
249
+ Key | Options | Default | Description
250
+ ----------------------- | --------------------- | ------------------ | -----------
251
+ :validate | `true`/`false` | `true` | Whether or not to run `ActiveRecord` validations (uniqueness skipped).
252
+ :validate_uniqueness | `true`/`false` | `false` | Whether or not to run uniqueness validations, has potential pitfalls, use with caution (requires `>= v0.27.0`).
253
+ :on_duplicate_key_ignore| `true`/`false` | `false` | Allows skipping records with duplicate keys. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-ignore) for more details.
254
+ :ignore | `true`/`false` | `false` | Alias for :on_duplicate_key_ignore.
255
+ :on_duplicate_key_update| :all, `Array`, `Hash` | N/A | Allows upsert logic to be used. See [here](https://github.com/zdennis/activerecord-import/#duplicate-key-update) for more details.
256
+ :synchronize | `Array` | N/A | An array of ActiveRecord instances. This synchronizes existing instances in memory with updates from the import.
257
+ :timestamps | `true`/`false` | `true` | Enables/disables timestamps on imported records.
258
+ :recursive | `true`/`false` | `false` | Imports has_many/has_one associations (PostgreSQL only).
259
+ :batch_size | `Integer` | total # of records | Max number of records to insert per import
260
+ :raise_error | `true`/`false` | `false` | Throws an exception if there are invalid records. `import!` is a shortcut for this.
261
+
262
+
263
+ #### Duplicate Key Ignore
264
+
265
+ [MySQL](http://dev.mysql.com/doc/refman/5.0/en/insert-on-duplicate.html), [SQLite](https://www.sqlite.org/lang_insert.html), and [PostgreSQL](https://www.postgresql.org/docs/current/static/sql-insert.html#SQL-ON-CONFLICT) (9.5+) support `on_duplicate_key_ignore` which allows you to skip records if a primary or unique key constraint is violated.
266
+
267
+ For Postgres 9.5+ it adds `ON CONFLICT DO NOTHING`, for MySQL it uses `INSERT IGNORE`, and for SQLite it uses `INSERT OR IGNORE`. Cannot be enabled on a recursive import. For database adapters that normally support setting primary keys on imported objects, this option prevents that from occurring.
268
+
269
+ ```ruby
270
+ book = Book.create! title: "Book1", author: "FooManChu"
271
+ book.title = "Updated Book Title"
272
+ book.author = "Bob Barker"
273
+
274
+ Book.import [book], on_duplicate_key_ignore: true
275
+
276
+ book.reload.title # => "Book1" (stayed the same)
277
+ book.reload.author # => "FooManChu" (stayed the same)
278
+ ```
279
+
280
+ The option `:on_duplicate_key_ignore` is bypassed when `:recursive` is enabled for [PostgreSQL imports](https://github.com/zdennis/activerecord-import/wiki#recursive-example-postgresql-only).
281
+
282
+ #### Duplicate Key Update
283
+
284
+ MySQL, PostgreSQL (9.5+), and SQLite (3.24.0+) support `on duplicate key update` (also known as "upsert") which allows you to specify fields whose values should be updated if a primary or unique key constraint is violated.
285
+
286
+ One big difference between MySQL and PostgreSQL support is that MySQL will handle any conflict that happens, but PostgreSQL requires that you specify which columns the conflict would occur over. SQLite models its upsert support after PostgreSQL.
287
+
288
+ This will use MySQL's `ON DUPLICATE KEY UPDATE` or Postgres/SQLite `ON CONFLICT DO UPDATE` to do upsert.
289
+
290
+ Basic Update
291
+
292
+ ```ruby
293
+ book = Book.create! title: "Book1", author: "FooManChu"
294
+ book.title = "Updated Book Title"
295
+ book.author = "Bob Barker"
296
+
297
+ # MySQL version
298
+ Book.import [book], on_duplicate_key_update: [:title]
299
+
300
+ # PostgreSQL version
301
+ Book.import [book], on_duplicate_key_update: {conflict_target: [:id], columns: [:title]}
302
+
303
+ # PostgreSQL shorthand version (conflict target must be primary key)
304
+ Book.import [book], on_duplicate_key_update: [:title]
305
+
306
+ book.reload.title # => "Updated Book Title" (changed)
307
+ book.reload.author # => "FooManChu" (stayed the same)
308
+ ```
309
+
310
+ Using the value from another column
311
+
312
+ ```ruby
313
+ book = Book.create! title: "Book1", author: "FooManChu"
314
+ book.title = "Updated Book Title"
315
+
316
+ # MySQL version
317
+ Book.import [book], on_duplicate_key_update: {author: :title}
318
+
319
+ # PostgreSQL version (no shorthand version)
320
+ Book.import [book], on_duplicate_key_update: {
321
+ conflict_target: [:id], columns: {author: :title}
322
+ }
323
+
324
+ book.reload.title # => "Book1" (stayed the same)
325
+ book.reload.author # => "Updated Book Title" (changed)
326
+ ```
327
+
328
+ Using Custom SQL
329
+
330
+ ```ruby
331
+ book = Book.create! title: "Book1", author: "FooManChu"
332
+ book.author = "Bob Barker"
333
+
334
+ # MySQL version
335
+ Book.import [book], on_duplicate_key_update: "author = values(author)"
336
+
337
+ # PostgreSQL version
338
+ Book.import [book], on_duplicate_key_update: {
339
+ conflict_target: [:id], columns: "author = excluded.author"
340
+ }
341
+
342
+ # PostgreSQL shorthand version (conflict target must be primary key)
343
+ Book.import [book], on_duplicate_key_update: "author = excluded.author"
344
+
345
+ book.reload.title # => "Book1" (stayed the same)
346
+ book.reload.author # => "Bob Barker" (changed)
347
+ ```
348
+
349
+ PostgreSQL Using constraints
350
+
351
+ ```ruby
352
+ book = Book.create! title: "Book1", author: "FooManChu", edition: 3, published_at: nil
353
+ book.published_at = Time.now
354
+
355
+ # in migration
356
+ execute <<-SQL
357
+ ALTER TABLE books
358
+ ADD CONSTRAINT for_upsert UNIQUE (title, author, edition);
359
+ SQL
360
+
361
+ # PostgreSQL version
362
+ Book.import [book], on_duplicate_key_update: {constraint_name: :for_upsert, columns: [:published_at]}
363
+
364
+
365
+ book.reload.title # => "Book1" (stayed the same)
366
+ book.reload.author # => "FooManChu" (stayed the same)
367
+ book.reload.edition # => 3 (stayed the same)
368
+ book.reload.published_at # => 2017-10-09 (changed)
369
+ ```
370
+
371
+ ```ruby
372
+ Book.import books, validate_uniqueness: true
373
+ ```
374
+
375
+ ### Return Info
376
+
377
+ The `import` method returns a `Result` object that responds to `failed_instances` and `num_inserts`. Additionally, for users of Postgres, there will be two arrays `ids` and `results` that can be accessed`.
378
+
379
+ ```ruby
380
+ articles = [
381
+ Article.new(author_id: 1, title: 'First Article', content: 'This is the first article'),
382
+ Article.new(author_id: 2, title: 'Second Article', content: ''),
383
+ Article.new(author_id: 3, content: '')
384
+ ]
385
+
386
+ demo = Article.import(articles), returning: :title # => #<struct ActiveRecord::Import::Result
387
+
388
+ demo.failed_instances
389
+ => [#<Article id: 3, author_id: 3, title: nil, content: "", created_at: nil, updated_at: nil>]
390
+
391
+ demo.num_inserts
392
+ => 1,
393
+
394
+ demo.ids
395
+ => ["1", "2"] # for Postgres
396
+ => [] # for other DBs
397
+
398
+ demo.results
399
+ => ["First Article", "Second Article"] # for Postgres
400
+ => [] for other DBs
401
+ ```
402
+
403
+ ### Counter Cache
404
+
405
+ When running `import`, `activerecord-import` does not automatically update counter cache columns. To update these columns, you will need to do one of the following:
406
+
407
+ * Provide values to the column as an argument on your object that is passed in.
408
+ * Manually update the column after the record has been imported.
409
+
410
+ ### ActiveRecord Timestamps
411
+
412
+ If you're familiar with ActiveRecord you're probably familiar with its timestamp columns: created_at, created_on, updated_at, updated_on, etc. When importing data the timestamp fields will continue to work as expected and each timestamp column will be set.
413
+
414
+ Should you wish to specify those columns, you may use the option `timestamps: false`.
415
+
416
+ However, it is also possible to set just `:created_at` in specific records. In this case despite using `timestamps: true`, `:created_at` will be updated only in records where that field is `nil`. Same rule applies for record associations when enabling the option `recursive: true`.
417
+
418
+ If you are using custom time zones, these will be respected when performing imports as well as long as `ActiveRecord::Base.default_timezone` is set, which for practically all Rails apps it is
419
+
420
+ ### Callbacks
421
+
422
+ ActiveRecord callbacks related to [creating](http://guides.rubyonrails.org/active_record_callbacks.html#creating-an-object), [updating](http://guides.rubyonrails.org/active_record_callbacks.html#updating-an-object), or [destroying](http://guides.rubyonrails.org/active_record_callbacks.html#destroying-an-object) records (other than `before_validation` and `after_validation`) will NOT be called when calling the import method. This is because it is mass importing rows of data and doesn't necessarily have access to in-memory ActiveRecord objects.
423
+
424
+ If you do have a collection of in-memory ActiveRecord objects you can do something like this:
425
+
426
+ ```ruby
427
+ books.each do |book|
428
+ book.run_callbacks(:save) { false }
429
+ book.run_callbacks(:create) { false }
430
+ end
431
+ Book.import(books)
432
+ ```
433
+
434
+ This will run before_create and before_save callbacks on each item. The `false` argument is needed to prevent after_save being run, which wouldn't make sense prior to bulk import. Something to note in this example is that the before_create and before_save callbacks will run before the validation callbacks.
435
+
436
+ If that is an issue, another possible approach is to loop through your models first to do validations and then only run callbacks on and import the valid models.
437
+
438
+ ```ruby
439
+ valid_books = []
440
+ invalid_books = []
441
+
442
+ books.each do |book|
443
+ if book.valid?
444
+ valid_books << book
445
+ else
446
+ invalid_books << book
447
+ end
448
+ end
449
+
450
+ valid_books.each do |book|
451
+ book.run_callbacks(:save) { false }
452
+ book.run_callbacks(:create) { false }
453
+ end
454
+
455
+ Book.import valid_books, validate: false
456
+ ```
457
+
458
+ ### Supported Adapters
459
+
460
+ The following database adapters are currently supported:
461
+
462
+ * MySQL - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.1.0 and higher)
463
+ * MySQL2 - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.2.0 and higher)
464
+ * PostgreSQL - supports core import functionality (included in activerecord-import 0.1.0 and higher)
465
+ * SQLite3 - supports core import functionality (included in activerecord-import 0.1.0 and higher)
466
+ * Oracle - supports core import functionality through DML trigger (available as an external gem: [activerecord-import-oracle_enhanced](https://github.com/keeguon/activerecord-import-oracle_enhanced)
467
+ * SQL Server - supports core import functionality (available as an external gem: [activerecord-import-sqlserver](https://github.com/keeguon/activerecord-import-sqlserver)
468
+
469
+ If your adapter isn't listed here, please consider creating an external gem as described in the README to provide support. If you do, feel free to update this wiki to include a link to the new adapter's repository!
470
+
471
+ To test which features are supported by your adapter, use the following methods on a model class:
472
+ * `supports_import?(*args)`
473
+ * `supports_on_duplicate_key_update?`
474
+ * `supports_setting_primary_key_of_imported_objects?`
475
+
476
+ ### Additional Adapters
27
477
 
28
- ## Additional Adapters
29
478
  Additional adapters can be provided by gems external to activerecord-import by providing an adapter that matches the naming convention setup by activerecord-import (and subsequently activerecord) for dynamically loading adapters. This involves also providing a folder on the load path that follows the activerecord-import naming convention to allow activerecord-import to dynamically load the file.
30
479
 
31
480
  When `ActiveRecord::Import.require_adapter("fake_name")` is called the require will be:
32
481
 
33
482
  ```ruby
34
- require 'activerecord-import/active_record/adapters/fake_name_adapter'
483
+ require 'activerecord-import/active_record/adapters/fake_name_adapter'
35
484
  ```
36
485
 
37
486
  This allows an external gem to dynamically add an adapter without the need to add any file/code to the core activerecord-import gem.
38
487
 
488
+ ### Requiring
489
+
490
+ Note: These instructions will only work if you are using version 0.2.0 or higher.
491
+
492
+ #### Autoloading via Bundler
493
+
494
+ If you are using Rails or otherwise autoload your dependencies via Bundler, all you need to do add the gem to your `Gemfile` like so:
495
+
496
+ ```ruby
497
+ gem 'activerecord-import'
498
+ ```
499
+
500
+ #### Manually Loading
501
+
502
+ You may want to manually load activerecord-import for one reason or another. First, add the `require: false` argument like so:
503
+
504
+ ```ruby
505
+ gem 'activerecord-import', require: false
506
+ ```
507
+
508
+ This will allow you to load up activerecord-import in the file or files where you are using it and only load the parts you need.
509
+ If you are doing this within Rails and ActiveRecord has established a database connection (such as within a controller), you will need to do extra initialization work:
510
+
511
+ ```ruby
512
+ require 'activerecord-import/base'
513
+ # load the appropriate database adapter (postgresql, mysql2, sqlite3, etc)
514
+ require 'activerecord-import/active_record/adapters/postgresql_adapter'
515
+ ```
516
+
517
+ If your gem dependencies aren’t autoloaded, and your script will be establishing a database connection, then simply require activerecord-import after ActiveRecord has been loaded, i.e.:
518
+
519
+ ```ruby
520
+ require 'active_record'
521
+ require 'activerecord-import'
522
+ ```
523
+
39
524
  ### Load Path Setup
40
525
  To understand how rubygems loads code you can reference the following:
41
526
 
42
527
  http://guides.rubygems.org/patterns/#loading_code
43
528
 
44
529
  And an example of how active_record dynamically load adapters:
530
+
45
531
  https://github.com/rails/rails/blob/master/activerecord/lib/active_record/connection_adapters/connection_specification.rb
46
532
 
47
533
  In summary, when a gem is loaded rubygems adds the `lib` folder of the gem to the global load path `$LOAD_PATH` so that all `require` lookups will not propagate through all of the folders on the load path. When a `require` is issued each folder on the `$LOAD_PATH` is checked for the file and/or folder referenced. This allows a gem (like activerecord-import) to define push the activerecord-import folder (or namespace) on the `$LOAD_PATH` and any adapters provided by activerecord-import will be found by rubygems when the require is issued.
@@ -52,17 +538,73 @@ If `fake_name` adapter is needed by a gem (potentially called `activerecord-impo
52
538
  activerecord-import-fake_name/
53
539
  |-- activerecord-import-fake_name.gemspec
54
540
  |-- lib
541
+ | |-- activerecord-import-fake_name.rb
55
542
  | |-- activerecord-import-fake_name
56
543
  | | |-- version.rb
57
544
  | |-- activerecord-import
58
545
  | | |-- active_record
59
546
  | | | |-- adapters
60
547
  | | | |-- fake_name_adapter.rb
61
- |--activerecord-import-fake_name.rb
62
548
  ```
63
549
 
64
550
  When rubygems pushes the `lib` folder onto the load path a `require` will now find `activerecord-import/active_record/adapters/fake_name_adapter` as it runs through the lookup process for a ruby file under that path in `$LOAD_PATH`
65
551
 
552
+
553
+ ### Conflicts With Other Gems
554
+
555
+ `activerecord-import` adds the `.import` method onto `ActiveRecord::Base`. There are other gems, such as `elasticsearch-rails`, that do the same thing. In conflicts such as this, there is an aliased method named `.bulk_import` that can be used interchangeably.
556
+
557
+ If you are using the `apartment` gem, there is a weird triple interaction between that gem, `activerecord-import`, and `activerecord` involving caching of the `sequence_name` of a model. This can be worked around by explcitly setting this value within the model. For example:
558
+
559
+ ```ruby
560
+ class Post < ActiveRecord::Base
561
+ self.sequence_name = "posts_seq"
562
+ end
563
+ ```
564
+
565
+ Another way to work around the issue is to call `.reset_sequence_name` on the model. For example:
566
+
567
+ ```ruby
568
+ schemas.all.each do |schema|
569
+ Apartment::Tenant.switch! schema.name
570
+ ActiveRecord::Base.transaction do
571
+ Post.reset_sequence_name
572
+
573
+ Post.import posts
574
+ end
575
+ end
576
+ ```
577
+
578
+ See https://github.com/zdennis/activerecord-import/issues/233 for further discussion.
579
+
580
+ ### More Information
581
+
582
+ For more information on activerecord-import please see its wiki: https://github.com/zdennis/activerecord-import/wiki
583
+
584
+ To document new information, please add to the README instead of the wiki. See https://github.com/zdennis/activerecord-import/issues/397 for discussion.
585
+
586
+ ### Contributing
587
+
588
+ #### Running Tests
589
+
590
+ The first thing you need to do is set up your database(s):
591
+
592
+ * copy `test/database.yml.sample` to `test/database.yml`
593
+ * modify `test/database.yml` for your database settings
594
+ * create databases as needed
595
+
596
+ After that, you can run the tests. They run against multiple tests and ActiveRecord versions.
597
+
598
+ This is one example of how to run the tests:
599
+
600
+ ```ruby
601
+ rm Gemfile.lock
602
+ AR_VERSION=4.2 bundle install
603
+ AR_VERSION=4.2 bundle exec rake test:postgresql test:sqlite3 test:mysql2
604
+ ```
605
+
606
+ Once you have pushed up your changes, you can find your CI results [here](https://travis-ci.org/zdennis/activerecord-import/).
607
+
66
608
  # License
67
609
 
68
610
  This is licensed under the ruby license.
@@ -83,3 +625,4 @@ Zach Dennis (zach.dennis@gmail.com)
83
625
  * Thibaud Guillaume-Gentil
84
626
  * Mark Van Holstyn
85
627
  * Victor Costan
628
+ * Dillon Welch
data/Rakefile CHANGED
@@ -23,6 +23,7 @@ ADAPTERS = %w(
23
23
  postgresql
24
24
  postgresql_makara
25
25
  postgis
26
+ makara_postgis
26
27
  sqlite3
27
28
  spatialite
28
29
  seamless_database_pool
@@ -31,7 +32,7 @@ ADAPTERS.each do |adapter|
31
32
  namespace :test do
32
33
  desc "Runs #{adapter} database tests."
33
34
  Rake::TestTask.new(adapter) do |t|
34
- # FactoryGirl has an issue with warnings, so turn off, so noisy
35
+ # FactoryBot has an issue with warnings, so turn off, so noisy
35
36
  # t.warning = true
36
37
  t.test_files = FileList["test/adapters/#{adapter}.rb", "test/*_test.rb", "test/active_record/*_test.rb", "test/#{adapter}/**/*_test.rb"]
37
38
  end
@@ -42,7 +42,8 @@ module BenchmarkOptionParser
42
42
  table_types: {},
43
43
  delete_on_finish: true,
44
44
  number_of_objects: [],
45
- outputs: [] )
45
+ outputs: []
46
+ )
46
47
 
47
48
  opt_parser = OptionParser.new do |opts|
48
49
  opts.banner = BANNER
data/gemfiles/5.1.gemfile CHANGED
@@ -1 +1,2 @@
1
1
  gem 'activerecord', '~> 5.1.0'
2
+ gem 'composite_primary_keys', '~> 10.0'
@@ -0,0 +1,2 @@
1
+ gem 'activerecord', '~> 5.2.0'
2
+ gem 'composite_primary_keys', '~> 11.0'
@@ -16,7 +16,7 @@ module ActiveRecord::Import::AbstractAdapter
16
16
  sql2insert = base_sql + values.join( ',' ) + post_sql
17
17
  insert( sql2insert, *args )
18
18
 
19
- [number_of_inserts, []]
19
+ ActiveRecord::Import::Result.new([], number_of_inserts, [], [])
20
20
  end
21
21
 
22
22
  def pre_sql_statements(options)
@@ -45,7 +45,7 @@ module ActiveRecord::Import::AbstractAdapter
45
45
  post_sql_statements = []
46
46
 
47
47
  if supports_on_duplicate_key_update? && options[:on_duplicate_key_update]
48
- post_sql_statements << sql_for_on_duplicate_key_update( table_name, options[:on_duplicate_key_update], options[:primary_key] )
48
+ post_sql_statements << sql_for_on_duplicate_key_update( table_name, options[:on_duplicate_key_update], options[:primary_key], options[:locking_column] )
49
49
  elsif options[:on_duplicate_key_update]
50
50
  logger.warn "Ignoring on_duplicate_key_update because it is not supported by the database."
51
51
  end