activerecord-import 2.2.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f7cd19fa557186daf34db96db71f26a7db70b15e66a0a42eaab33be304945ab0
-  data.tar.gz: ff8afe763d437210ea1c78e21bacd0b4ae84f113fbadbda087077bf402f08e64
+  metadata.gz: 44eca8fc38325819cfc2eeb8c9a6151182f5de96135af6ad3ae1cc145c5f7740
+  data.tar.gz: 9f6ef423efde4813da41b8eb5ba218d4166bb3a86beba83d72b4bbf9e70af27f
 SHA512:
-  metadata.gz: 402d2087c8c35a2adfa3787ee4e183883df74cc1f4ad2276de1227ba9778b630bfbee546faa3b53b7dcbe1142232bd7d03de046ccab5c6caa5c475a3bc4f15c1
-  data.tar.gz: ef5ce7d4dba744b3c58a0affbe326b50d764c7e4c4726b468ad4d18eacee13fbaa8d7b5d78459bc048e8eeb0628cf34b52a59f759f0a1f4d9d491b7439d5845a
+  metadata.gz: f534b0b2a47c29484b8957109794ab8924c758993e93ecd871104f97ce8402856999c40d1e845ff7a2857fb92e12291a0d93259a9bdd17d4b60f81cb20ac5e42
+  data.tar.gz: f9e8de71d80edd76c978a1d85807b6a94247da8366dfda54a4c1e2c01ed25c31767d836c2ec487d39377379492dd896f320a986c1a158cfda7f810e7b3ba3a80

data/CHANGELOG.md

@@ -1,3 +1,127 @@
+## Changes in 3.0.0
+
+### Breaking Changes
+
+* Drop support for Ruby < 3.1 and ActiveRecord < 6.1. via #918 and #943.
+* MariaDB 10.5+ now imports with `INSERT ... RETURNING`, as PostgreSQL does. Imports return
+  `ids` and `results`, imported models get their ids and are marked persisted, and
+  `recursive: true` imports associations (it was ignored on MariaDB before). MariaDB doesn't
+  report affected rows for `INSERT ... RETURNING`, so `affected_rows` is now `nil` on MariaDB
+  10.5+ unless `no_returning: true` is passed. As on PostgreSQL, `on_duplicate_key_ignore` is
+  ignored for recursive imports. MySQL and MariaDB before 10.5 are unchanged. via #926.
+* When only some rows have a value for a column the database fills in (a uuid or other primary
+  key with a default function, or a column with a default function), the other rows now get the
+  database's default instead of `NULL` on PostgreSQL and MySQL/MariaDB, and assigned primary keys
+  are no longer dropped when the first model doesn't have one. SQLite can't use `DEFAULT` in
+  `VALUES`, so such a mixed import now raises `ArgumentError` there. Fixes #702, #777, #805 and
+  #827 via #929.
+* `on_duplicate_key_update: :all` and `columns: :all` no longer update the timestamps recorded on
+  create (`created_at`, `created_on`) when import records timestamps. via #927.
+* Recursive imports now also import existing records that were assigned to a new parent, so they're
+  linked to it as `save` would do. Without `on_duplicate_key_update` that raises a duplicate key
+  error, where the records used to be silently left unlinked. Fixes #804 via #939.
+* Importing with `synchronize` now keeps the synchronized models' `previous_changes`, rather than
+  clearing them. Fixes #738 via #933.
+
+### New Features
+
+* Add `require "activerecord-import/without_aliases"` to load the library without the `import` and
+  `import!` aliases, for gems such as `elasticsearch-model` that define their own `import`. Use
+  `bulk_import`/`bulk_import!` instead. Fixes #592 via #915.
+* `import` accepts array-like arguments such as an `ActiveRecord::Relation` or a `Set`, and invalid
+  arguments now raise an error saying what was received. Fixes #785 via #935.
+* Annotations from the current scope (`Model.annotate(...)`) are added to the INSERT statements as
+  SQL comments. Fixes #872 via #937.
+* When activerecord-import is required after the database connection is established, import now
+  loads the adapter's import support itself instead of inserting one row at a time. Adapters
+  without bulk import support log a warning when import falls back to row-by-row inserts.
+  via #925 and #941.
+* Validate imported models with `valid?`, so `around` validate callbacks are supported. via #919.
+
+### Fixes
+
+* Report `affected_rows` with the mysql2 adapter. It was always `nil` with mysql2, and only
+  reported with trilogy. via #944.
+* Clear the query cache after MySQL imports on ActiveRecord < 7.1. via #926.
+* Don't import the associations of models that failed validation. Fixes #857. Thanks to
+  @jacob-carlborg-apoex via #858.
+* Fix `NoMethodError` in `returning_selections` when there's no model. Thanks to @mirelon via #892.
+* Don't mark models that failed validation as persisted when `track_validation_failures` is set.
+  Fixes #888. Thanks to @OskarEichler via #903.
+* Leave models unchanged when `all_or_none` aborts an import. Thanks to @OskarEichler via #904.
+* Include the owner's foreign key and polymorphic type when importing models with explicit columns
+  through an association. Thanks to @OskarEichler via #905.
+* Support string hash keys in association imports. Thanks to @OskarEichler via #909.
+* Use the canonical polymorphic name (e.g. for STI owners) in association imports. Thanks to
+  @OskarEichler via #910.
+* Raise `ArgumentError` for rows with fewer values than columns, before running any SQL. Thanks
+  to @OskarEichler via #912.
+* Quote PostgreSQL `RETURNING` column names and generated conflict targets. Thanks to
+  @OskarEichler via #907 and #908.
+* Skip uniqueness validations on associated records unless `validate_uniqueness: true`. Fixes #841
+  via #916.
+* Use each model's own class for `options[:model]` in recursive imports. via #920.
+* Leave columns the database populates (e.g. a uuid with a default function) out of the INSERT
+  when no imported model assigns them, as `save` does. via #921.
+* Don't follow an attribute alias when its name is also a column. via #922.
+* Don't modify the options hash passed to `import!`. via #924.
+* Don't send a `RETURNING` clause on PostgreSQL with `no_returning: true`. Fixes #826 via #930.
+* Support `returning: :id` with composite primary keys that include an `id` column. Fixes #842
+  via #931.
+* Use the association's `primary_key` for foreign keys in recursive imports. Fixes #878 via #932.
+* Serialize values of STI subclass models with the subclass's attribute types, e.g. for enums
+  defined only on the subclass. Fixes #789 via #934.
+* Fix `NoMethodError` when filling in composite `belongs_to` foreign keys. via #938.
+
+### Performance
+
+* Match synchronized records in linear rather than quadratic time. Thanks to @OskarEichler via #902.
+* Collect returned ids and results across batches without recopying them. Thanks to @OskarEichler
+  via #906.
+
+### Development
+
+* Test against MariaDB 10.11 and 11.8 in CI. via #926.
+* Run the composite primary key tests on ActiveRecord 7.1, and speed up the JRuby CI job.
+  via #923, #928 and #945.
+* Run the database tests locally against Docker without a hand-written `test/database.yml`.
+  via #945.
+* Document the memory use of large imports and what gets logged. via #940.
+* Split `import_helper` into smaller steps. via #947.
+
+## Changes in 2.3.0
+
+### New Features
+
+* Add Support for trilogy_proxy from [active_record_proxy_adapters](https://rubygems.org/gems/active_record_proxy_adapters) by @mateuscruz via #887.
+* Add Support for sqlite3_proxy from [active_record_proxy_adapters](https://rubygems.org/gems/active_record_proxy_adapters) by @mateuscruz via #887.
+* Add affected rows support for the MySQL adapter. `ActiveRecord::Import::Result` now carries an
+  `affected_rows` member, which reports the number of rows the server actually reported as
+  affected when the adapter exposes it (`nil` otherwise). This is most useful with
+  `on_duplicate_key_update`, where the count distinguishes inserted from updated rows.
+  Thanks to @ehasrouni via #882 and #885.
+
+### Fixes
+
+* Fix `NoMethodError: undefined method 'validation_context='` when importing with validations
+  against Rails main (8.2.0.alpha). Rails removed ActiveModel's private `validation_context=`
+  writer in rails/rails@0f9d1270834a6407a59637650bf910d8ae826169 in favor of a validation
+  context object reachable through `context_for_validation`. `ActiveRecord::Import::Validator`
+  now sets and restores the validation context through whichever API the loaded ActiveModel
+  provides. Thanks to @yu-yaba via #900.
+* Fix adapter-specific import support (e.g. `on_duplicate_key_update`) never loading for
+  a connection established via `connects_to`/`ActiveRecord::ConnectionAdapters::
+  ConnectionHandler#establish_connection` instead of `ActiveRecord::Base.establish_connection`
+  directly - most commonly hit with a Rails multi-database secondary connection. The lazy
+  adapter loader now hooks `ConnectionHandler#establish_connection` instead of
+  `ActiveRecord::Base.establish_connection`, so it fires for every connection regardless of
+  how it was established. Thanks to @mateuscruz via #886.
+* Fix `active_record_proxy_adapters` support, which was not loading the import adapter for
+  proxied connections. Thanks to @mateuscruz via #880.
+* Use `with_connection` instead of `ActiveRecord::Base.connection`, which is deprecated as of
+  ActiveRecord 7.2. Thanks to @nerdrew via #875 and #876.
+* Reduce the size of the published gem by excluding test files. Thanks to @yuri-zubov via #893.
+
 ## Changes in 2.2.0
 
 ### New Features

data/README.markdown

@@ -37,6 +37,7 @@ The gem provides the following high-level features:
   * [ActiveRecord Models](#activerecord-models)
   * [Batching](#batching)
   * [Recursive](#recursive)
+  * [Annotating the SQL](#annotating-the-sql)
 * [Options](#options)
   * [Duplicate Key Ignore](#duplicate-key-ignore)
   * [Duplicate Key Update](#duplicate-key-update)
@@ -44,6 +45,7 @@ The gem provides the following high-level features:
 * [Counter Cache](#counter-cache)
 * [ActiveRecord Timestamps](#activerecord-timestamps)
 * [Callbacks](#callbacks)
+* [Logging](#logging)
 * [Supported Adapters](#supported-adapters)
 * [Additional Adapters](#additional-adapters)
 * [Requiring](#requiring)
@@ -121,20 +123,20 @@ Book.import values
 The `import` method can take an array of column names and an array of hash objects. The column names are used to determine what fields of data should be imported. The following example will only import books with the `title` field:
 
 ```ruby
-books = [
+values = [
   { title: "Book 1", author: "George Orwell" },
   { title: "Book 2", author: "Bob Jones" }
 ]
 columns = [ :title ]
 
 # without validations
-Book.import columns, books, validate: false
+Book.import columns, values, validate: false
 
 # with validations
-Book.import columns, books, validate: true
+Book.import columns, values, validate: true
 
 # when not specified :validate defaults to true
-Book.import columns, books
+Book.import columns, values
 
 # result in table books
 # title  | author
@@ -231,6 +233,16 @@ columns = [ :title ]
 Book.import columns, books, batch_size: 2
 ```
 
+`batch_size` only controls how many rows go into each INSERT statement. The whole import is still prepared up front: every record or row passed to `import`, and the SQL for all of them, is held in memory until the import finishes. For very large imports, load and import the data in chunks, so only one chunk is in memory at a time:
+
+```ruby
+Source.find_in_batches(batch_size: 1_000) do |sources|
+  Book.import sources.map { |source| Book.new(title: source.title, author: source.author) }
+end
+```
+
+Each `import` call validates, inserts and returns its own result, so options like `all_or_none` apply to each chunk rather than to all of the data.
+
 If your import is particularly large or slow (possibly due to [callbacks](#callbacks)) whilst batch importing, you might want a way to report back on progress. This is supported by passing a callable as the `batch_progress` option. e.g:
 
 ```ruby
@@ -246,7 +258,7 @@ Book.import columns, books, batch_size: 2, batch_progress: my_proc
 #### Recursive
 
 > **Note**
-> This only works with PostgreSQL and ActiveRecord objects. This won't work with hashes or arrays as recursive inputs.
+> This only works with PostgreSQL and MariaDB 10.5+, and with ActiveRecord objects. This won't work with hashes or arrays as recursive inputs.
 
 Assume that Books <code>has_many</code> Reviews.
 
@@ -260,12 +272,25 @@ end
 Book.import books, recursive: true
 ```
 
+#### Annotating the SQL
+
+To add SQL comments to an import, import through a relation annotated with ActiveRecord's [`annotate`](https://api.rubyonrails.org/classes/ActiveRecord/QueryMethods.html#method-i-annotate), or within its `scoping` block. The comments are added to every INSERT statement, including when the import is batched or split into several statements.
+
+```ruby
+Book.annotate("nightly import").import books
+# INSERT INTO "books" (...) VALUES (...) /* nightly import */
+
+Book.annotate("nightly import").scoping do
+  Book.import books
+end
+```
+
 ### Options
 
 Key                       | Options               | Default            | Description
 ------------------------- | --------------------- | ------------------ | -----------
 :validate                 | `true`/`false`        | `true`             | Whether or not to run `ActiveRecord` validations (uniqueness skipped). This option will always be true when using `import!`.
-:validate_uniqueness      | `true`/`false`        | `false`            | Whether or not to run ActiveRecord uniqueness validations. Beware this will incur an sql query per-record (N+1 queries). (requires `>= v0.27.0`).
+:validate_uniqueness      | `true`/`false`        | `false`            | Whether or not to run ActiveRecord uniqueness validations, including on associated records that are validated with the model. Beware this will incur an sql query per-record (N+1 queries). (requires `>= v0.27.0`).
 :validate_with_context    | `Symbol`              |`:create`/`:update` | Allows passing an ActiveModel validation context for each model. Default is `:create` for new records and `:update` for existing ones.
 :track_validation_failures| `true`/`false`        | `false`            | When this is set to true, `failed_instances` will be an array of arrays, with each inner array having the form `[:index_in_dataset, :object_with_errors]`
 :on_duplicate_key_ignore  | `true`/`false`        | `false`            | Allows skipping records with duplicate keys. See [here](#duplicate-key-ignore) for more details.
@@ -273,9 +298,9 @@ Key                       | Options               | Default            | Descrip
 :on_duplicate_key_update  | :all, `Array`, `Hash` | N/A                | Allows upsert logic to be used. See [here](#duplicate-key-update) for more details.
 :synchronize              | `Array`               | N/A                | An array of ActiveRecord instances. This synchronizes existing instances in memory with updates from the import.
 :timestamps               | `true`/`false`        | `true`             | Enables/disables timestamps on imported records.
-:recursive                | `true`/`false`        | `false`            | Imports has_many/has_one associations (PostgreSQL only).
+:recursive                | `true`/`false`        | `false`            | Imports has_many/has_one associations (PostgreSQL and MariaDB 10.5+ only).
 :recursive_on_duplicate_key_update | `Hash`       | N/A                | Allows upsert logic to be used for recursive associations. The hash key is the association name and the value has the same options as `:on_duplicate_key_update`. See [here](#duplicate-key-update) for more details.
-:batch_size               | `Integer`             | total # of records | Max number of records to insert per import
+:batch_size               | `Integer`             | total # of records | Max number of records to insert per INSERT statement. See [Batching](#batching).
 :raise_error              | `true`/`false`        | `false`            | Raises an exception at the first invalid record. This means there will not be a result object returned. The `import!` method is a shortcut for this.
 :all_or_none              | `true`/`false`        | `false`            | Will not import any records if there is a record with validation errors.
 
@@ -296,7 +321,7 @@ book.reload.title  # => "Book1"     (stayed the same)
 book.reload.author # => "George Orwell" (stayed the same)
 ```
 
-The option `:on_duplicate_key_ignore` is bypassed when `:recursive` is enabled for [PostgreSQL imports](https://github.com/zdennis/activerecord-import/wiki#recursive-example-postgresql-only).
+The option `:on_duplicate_key_ignore` is bypassed when `:recursive` is enabled for [PostgreSQL and MariaDB 10.5+ imports](https://rubydoc.info/gems/activerecord-import/ActiveRecord/Base#label-Options) (`:recursive` is not supported with other databases).
 
 #### Duplicate Key Update
 
@@ -416,7 +441,7 @@ Book.import books, validate_uniqueness: true
 
 ### Return Info
 
-The `import` method returns a `Result` object that responds to `failed_instances` and `num_inserts`. Additionally, for users of Postgres, there will be two arrays `ids` and `results` that can be accessed.
+The `import` method returns a `Result` object that responds to `failed_instances` and `num_inserts`. Additionally, for users of Postgres and MariaDB 10.5+, there will be two arrays `ids` and `results` that can be accessed. These are populated with `INSERT ... RETURNING`, which also sets the ids of imported models. On MariaDB this means `affected_rows` isn't available, since MariaDB doesn't report it for `INSERT ... RETURNING`; pass `no_returning: true` to get `affected_rows` instead of `ids`.
 
 ```ruby
 articles = [
@@ -434,11 +459,11 @@ demo.num_inserts
 => 1,
 
 demo.ids
-=> ["1", "2"] # for Postgres
+=> ["1", "2"] # for Postgres and MariaDB 10.5+
 => [] # for other DBs
 
 demo.results
-=> ["First Article", "Second Article"] # for Postgres
+=> ["First Article", "Second Article"] # for Postgres and MariaDB 10.5+
 => [] # for other DBs
 ```
 
@@ -499,18 +524,29 @@ end
 Book.import valid_books, validate: false
 ```
 
+### Logging
+
+Imports are logged like ActiveRecord's own queries, named `<Model> Create Many`. The imported values are part of the SQL rather than bind parameters, so a logged INSERT includes every value it imports. That can make for very long log lines, and ActiveRecord's `filter_attributes` doesn't apply to them, so the values of sensitive columns appear in the log too. ActiveRecord's own `insert_all` and `upsert_all` log the same way.
+
+If that's a concern, silence the logger around imports. For example, with Rails' logger:
+
+```ruby
+ActiveRecord::Base.logger.silence { Book.import books }
+```
+
 ### Supported Adapters
 
 The following database adapters are currently supported:
 
 * MySQL - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.1.0 and higher)
 * MySQL2 - supports core import functionality plus on duplicate key update support (included in activerecord-import 0.2.0 and higher)
+* MariaDB - supported through the MySQL2 and Trilogy adapters, with the same functionality as MySQL. MariaDB 10.5+ also supports `ids`, `returning` and recursive imports.
 * PostgreSQL - supports core import functionality (included in activerecord-import 0.1.0 and higher)
 * SQLite3 - supports core import functionality (included in activerecord-import 0.1.0 and higher)
-* Oracle - supports core import functionality through DML trigger (available as an external gem: [activerecord-import-oracle_enhanced](https://github.com/keeguon/activerecord-import-oracle_enhanced)
-* SQL Server - supports core import functionality (available as an external gem: [activerecord-import-sqlserver](https://github.com/keeguon/activerecord-import-sqlserver)
+* Oracle - supports core import functionality through DML trigger (available as an external gem: [activerecord-import-oracle_enhanced](https://github.com/keeguon/activerecord-import-oracle_enhanced))
+* SQL Server - supports core import functionality (available as an external gem: [activerecord-import-sqlserver](https://github.com/keeguon/activerecord-import-sqlserver))
 
-If your adapter isn't listed here, please consider creating an external gem as described in the README to provide support. If you do, feel free to update this wiki to include a link to the new adapter's repository!
+If your adapter isn't listed here, please consider creating an external gem as described in the README to provide support. If you do, feel free to update this README to include a link to the new adapter's repository!
 
 To test which features are supported by your adapter, use the following methods on a model class:
 * `supports_import?(*args)`
@@ -550,12 +586,15 @@ You may want to manually load activerecord-import for one reason or another. Fir
 gem 'activerecord-import', require: false
 ```
 
-This will allow you to load up activerecord-import in the file or files where you are using it and only load the parts you need.
-If you are doing this within Rails and ActiveRecord has established a database connection (such as within a controller), you will need to do extra initialization work:
+This will allow you to load up activerecord-import in the file or files where you are using it and only load the parts you need:
 
 ```ruby
 require 'activerecord-import/base'
-# load the appropriate database adapter (postgresql, mysql2, sqlite3, etc)
+```
+
+This works even if ActiveRecord has already established a database connection (such as within a controller): the import support for the connection's adapter (postgresql, mysql2, sqlite3, etc) is loaded the first time you import. If your adapter goes by a name activerecord-import doesn't recognize, load its support yourself:
+
+```ruby
 require 'activerecord-import/active_record/adapters/postgresql_adapter'
 ```
 
@@ -599,6 +638,16 @@ When rubygems pushes the `lib` folder onto the load path a `require` will now fi
 
 Activerecord-Import adds the `.import` method onto `ActiveRecord::Base`. There are other gems, such as `elasticsearch-rails`, that do the same thing. In conflicts such as this, there is an aliased method named `.bulk_import` that can be used interchangeably.
 
+Some gems, such as `elasticsearch-model`, won't define their own `.import` on a model if one already exists. To load activerecord-import without the `import` and `import!` aliases, so only `bulk_import` and `bulk_import!` are defined, require it like so:
+
+```ruby
+gem 'activerecord-import', require: 'activerecord-import/without_aliases'
+```
+
+This takes precedence even if `activerecord-import` or `activerecord-import/base` is also required, for example by another gem, regardless of the order they're required in. To confirm the aliases aren't defined, check that `ActiveRecord::Base.respond_to?(:import)` returns `false` in a console.
+
+If you're loading activerecord-import manually (see [Manually Loading](#manually-loading)), require `activerecord-import/without_aliases` in place of `activerecord-import/base`.
+
 If you are using the `apartment` gem, there is a weird triple interaction between that gem, `activerecord-import`, and `activerecord` involving caching of the `sequence_name` of a model. This can be worked around by explicitly setting this value within the model. For example:
 
 ```ruby
@@ -624,7 +673,7 @@ See https://github.com/zdennis/activerecord-import/issues/233 for further discus
 
 ### More Information
 
-For more information on Activerecord-Import please see its wiki: https://github.com/zdennis/activerecord-import/wiki
+For more information on Activerecord-Import please see its docs: https://rubydoc.info/gems/activerecord-import/ActiveRecord/Base
 
 To document new information, please add to the README instead of the wiki. See https://github.com/zdennis/activerecord-import/issues/397 for discussion.
 
@@ -632,22 +681,36 @@ To document new information, please add to the README instead of the wiki. See h
 
 #### Running Tests
 
-The first thing you need to do is set up your database(s):
+The tests run against each database adapter (`bundle exec rake -T` lists them) and against multiple ActiveRecord versions, chosen with `AR_VERSION`. The SQLite tests need no setup. For MySQL, MariaDB and PostgreSQL, the quickest way to get databases is Docker (see [Docker Setup](#docker-setup)):
+
+```bash
+docker compose up -d
+rm -f Gemfile.lock
+AR_VERSION=8.1 bundle install
+AR_VERSION=8.1 bundle exec rake test:sqlite3 test:postgresql test:mysql2 test:trilogy
+```
 
-* copy `test/database.yml.sample` to `test/database.yml`
-* modify `test/database.yml` for your database settings
-* create databases as needed
+Without a `test/database.yml`, the tests use `test/database.default.yml`, which connects to the Docker databases by default and can be pointed elsewhere with environment variables:
 
-After that, you can run the tests. They run against multiple tests and ActiveRecord versions.
+| Variable | Default | |
+| --- | --- | --- |
+| `DB_HOST` | `127.0.0.1` | Host for all databases |
+| `DB_DATABASE` | `activerecord_import_test` | Database name |
+| `MYSQL_HOST`, `PG_HOST` | `DB_HOST` | Host for MySQL/MariaDB or PostgreSQL |
+| `MYSQL_PORT` | `13306` | MySQL/MariaDB port; Docker's MariaDB is on `13307` |
+| `PG_PORT` | `15432` | PostgreSQL port |
+| `MYSQL_USERNAME`, `MYSQL_PASSWORD` | `root`, empty | |
+| `PG_USERNAME`, `PG_PASSWORD` | `postgres`, empty | |
 
-This is one example of how to run the tests:
+For example, to run the MySQL tests against Docker's MariaDB, or against a MySQL server you run yourself:
 
 ```bash
-rm Gemfile.lock
-AR_VERSION=7.0 bundle install
-AR_VERSION=7.0 bundle exec rake test:postgresql test:sqlite3 test:mysql2
+MYSQL_PORT=13307 AR_VERSION=8.1 bundle exec rake test:mysql2
+MYSQL_PORT=3306 MYSQL_PASSWORD=secret AR_VERSION=8.1 bundle exec rake test:mysql2
 ```
 
+The PostgreSQL tests need the `hstore`, `pgcrypto` and `uuid-ossp` extensions, which they create themselves when connecting as a superuser, and `test:postgis` also needs PostGIS. If you'd rather keep your settings in a file, copy `test/database.yml.sample` to `test/database.yml` and edit it; when that file exists it is used instead (it can also use ERB).
+
 Once you have pushed up your changes, you can find your CI results [here](https://github.com/zdennis/activerecord-import/actions).
 
 #### Docker Setup
@@ -657,11 +720,15 @@ Before you begin, make sure you have [Docker](https://www.docker.com/products/do
 ```bash
 brew install docker && brew install docker-compose
 ```
-##### Steps
 
-1. In your terminal run `docker-compose up --build`
-1. In another tab/window run `docker-compose exec app bash`
-1. In that same terminal run the mysql2 test by running `bundle exec rake test:mysql2`
+`docker compose up -d` starts MySQL 8.0, MariaDB 11.8 and PostgreSQL with PostGIS (the image CI uses), listening on localhost ports 13306, 13307 and 15432 so they don't collide with databases you may already run. To use other ports, set `MYSQL_PORT`, `MARIADB_PORT` or `PG_PORT` when starting them, and the same variables (with `MYSQL_PORT` set to the MariaDB port to test MariaDB) when running the tests. `docker compose down -v` stops them and deletes their data.
+
+To run the tests inside Docker instead of with a local Ruby:
+
+1. In your terminal run `docker compose --profile app up -d --build`
+1. Then run `docker compose exec app bash`
+1. In that shell run `bundle install` (the image's gems are for `AR_VERSION=8.1`; to test another version, `rm Gemfile.lock` and run `AR_VERSION=7.2 bundle install`, then set `AR_VERSION` when running the tests too)
+1. Run the mysql2 tests with `bundle exec rake test:mysql2` (set `MYSQL_HOST=mariadb` to use MariaDB)
 
 ## Issue Triage [![Open Source Helpers](https://www.codetriage.com/zdennis/activerecord-import/badges/users.svg)](https://www.codetriage.com/zdennis/activerecord-import)
 

data/lib/activerecord-import/active_record/adapters/mysql2_proxy_adapter.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "active_record/connection_adapters/mysql2_proxy_adapter"
+require "activerecord-import/adapters/mysql2_proxy_adapter"
+
+ActiveSupport.on_load(:active_record_mysql2proxyadapter) do |klass|
+  klass.include(ActiveRecord::Import::Mysql2ProxyAdapter)
+end

data/lib/activerecord-import/active_record/adapters/postgresql_proxy_adapter.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "active_record/connection_adapters/postgresql_proxy_adapter"
+require "activerecord-import/adapters/postgresql_proxy_adapter"
+
+ActiveSupport.on_load(:active_record_postgresqlproxyadapter) do |klass|
+  klass.include(ActiveRecord::Import::PostgreSQLProxyAdapter)
+end

data/lib/activerecord-import/active_record/adapters/sqlite3_proxy_adapter.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "active_record/connection_adapters/sqlite3_proxy_adapter"
+require "activerecord-import/adapters/sqlite3_proxy_adapter"
+
+ActiveSupport.on_load(:active_record_sqlite3proxyadapter) do |klass|
+  klass.include(ActiveRecord::Import::SQLite3ProxyAdapter)
+end

data/lib/activerecord-import/active_record/adapters/trilogy_proxy_adapter.rb

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "active_record/connection_adapters/trilogy_proxy_adapter"
+require "activerecord-import/adapters/trilogy_proxy_adapter"
+
+ActiveSupport.on_load(:active_record_trilogyproxyadapter) do |klass|
+  klass.include(ActiveRecord::Import::TrilogyProxyAdapter)
+end

data/lib/activerecord-import/adapters/abstract_adapter.rb

@@ -62,6 +62,59 @@ module ActiveRecord::Import::AbstractAdapter
       post_sql_statements
     end
 
+    # Returns the selections for an INSERT ... RETURNING clause: the primary key
+    # and any +:returning+ columns. Known columns are quoted, anything else (such
+    # as raw SQL) is returned unchanged.
+    def returning_selections(options)
+      selections = []
+      column_names = Array(options[:model]&.column_names)
+
+      selections += Array(options[:primary_key]) if options[:primary_key].present?
+      selections += Array(options[:returning]) if options[:returning].present?
+
+      selections.map do |selection|
+        column_names.include?(selection.to_s) ? quote_column_name(selection) : selection
+      end
+    end
+
+    # Splits the values returned by an INSERT ... RETURNING into the ids and the
+    # +:returning+ results. +returned_values+ is a Hash with the returned
+    # +:values+, and the +:columns+ when more than one column was returned.
+    def ids_and_results_from_returned_values( returned_values, options ) # :nodoc:
+      if options[:returning].blank?
+        [Array(returned_values[:values]), []]
+      elsif options[:primary_key].blank?
+        options[:returning_columns] ||= returned_values[:columns]
+        [[], Array(returned_values[:values])]
+      else
+        # split primary key and returning columns
+        ids, results, options[:returning_columns] = split_ids_and_results(returned_values, options)
+        [ids, results]
+      end
+    end
+
+    def split_ids_and_results( selections, options )
+      ids = []
+      returning_values = []
+
+      columns = Array(selections[:columns])
+      values = Array(selections[:values])
+      id_indexes = Array(options[:primary_key]).map { |key| columns.index(key) }
+      returning_columns = columns.reject.with_index { |_, index| id_indexes.include?(index) }
+      returning_indexes = returning_columns.map { |column| columns.index(column) }
+
+      values.each do |value|
+        value_array = Array(value)
+        ids << id_indexes.map { |index| value_array[index] }
+        returning_values << returning_indexes.map { |index| value_array[index] }
+      end
+
+      ids.map!(&:first) if id_indexes.size == 1
+      returning_values.map!(&:first) if returning_columns.size == 1
+
+      [ids, returning_values, returning_columns]
+    end
+
     def increment_locking_column!(table_name, results, locking_column)
       if locking_column.present?
         results << "\"#{locking_column}\"=#{table_name}.\"#{locking_column}\"+1"
@@ -71,5 +124,10 @@ module ActiveRecord::Import::AbstractAdapter
     def supports_on_duplicate_key_update?
       false
     end
+
+    # Whether a value in an INSERT's VALUES can be DEFAULT, to use the column's default
+    def supports_default_keyword_in_values?
+      true
+    end
   end
 end

data/lib/activerecord-import/adapters/active_record_proxy_adapter.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module ActiveRecord::Import::ActiveRecordProxyAdapter
+  def insert_many(*args, **kwargs, &block)
+    sticking_to_primary { super(*args, **kwargs, &block) }
+  end
+
+  def insert(*args, **kwargs, &block)
+    sticking_to_primary { super(*args, **kwargs, &block) }
+  end
+
+  def sticking_to_primary(&block)
+    ActiveRecord::Base.connected_to(role: writing_role, &block)
+  end
+
+  def writing_role
+    ActiveRecord.try(:writing_role) || ActiveRecord::Base.try(:writing_role) || :writing
+  end
+end

data/lib/activerecord-import/adapters/mysql2_proxy_adapter.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+require "activerecord-import/adapters/mysql2_adapter"
+require "activerecord-import/adapters/active_record_proxy_adapter"
+
+module ActiveRecord::Import::Mysql2ProxyAdapter
+  include ActiveRecord::Import::Mysql2Adapter
+  include ActiveRecord::Import::ActiveRecordProxyAdapter
+end