online_migrations 0.9.2 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 579a0db844c82c2c9153a1a3f44bfcffe929011112b132afe73b1dfdf68f4583
-  data.tar.gz: b88f26034f0c1d044607ef191ca3feae71e3c6093cb05bd9a277790447da01af
+  metadata.gz: 9ceda570f99a1712496ac5c9549859c7dc2cad7e3cc4ab59c97a22ba17f3bfd0
+  data.tar.gz: dfc5a83147a92bf5e04a532210e72b6f32f561f032b2a1b746ff142bf4e16635
 SHA512:
-  metadata.gz: 229a2a31c44a358425a684ea8d1771d98ecee6412c9b4480e30e2cd68e53c0c9e77039959fa8f379ad0a83fb9f592d133d29fe43e2c79ae4dbf674cd9c210598
-  data.tar.gz: 9255d5ad7b7d350021cc553b5d91bec852cfbecc0b9c7d6296606beffae266de72096cdf5a4d12dd88d921fb0af60a9e870d08a676a90fad552c5bd2d36096f6
+  metadata.gz: 0ff6cce820134ef6b893f9405b71a0d8f8cd42dd79616123aaa492983671ebcff98dbff21477afa78b29342d40098822da4b8de0306489faf956b21ee44e2113
+  data.tar.gz: 42dcf132290c9affe812ef7489397d4c05d48f33069f3ee09710fced6aaf8ac928fee63c1498333657a824f98885e3af5e706e4dcc4a089b9cd62284ad0223d1

data/CHANGELOG.md

@@ -1,5 +1,46 @@
 ## master (unreleased)
 
+## 0.11.0 (2024-01-09)
+
+- Support sharding for background migrations
+
+    Now, if a `relation` inside background migration definition is defined on a sharded model,
+    then that background migration would automatically run on all the shards.
+
+    To get all the new sharding related schema changes, you need to run:
+
+    ```sh
+    $ bin/rails generate online_migrations:upgrade
+    $ bin/rails db:migrate
+    ```
+
+- Change background migration `progress` to return values in range from 0.0 to 100.0
+
+    Previously, these were values in range from 0.0 to 1.0 and caused confusion
+
+- Copy exclusion constraints when changing column type
+- Update `revert_finalize_columns_type_change` to not remove indexes, foreign keys etc
+- Fix verbose query logging when `ActiveRecord::Base.logger` is `nil`
+- Add a shortcut for running background migrations
+
+    ```ruby
+    # Before:
+    OnlineMigrations::BackgroundMigrations::Scheduler.run
+    # After
+    OnlineMigrations.run_background_migrations
+    ```
+
+- Add support for `:type_cast_function` to `initialize_column_type_change` helper
+- Drop support for Ruby < 2.7 and Rails < 6.1
+
+## 0.10.0 (2023-12-12)
+
+- Add `auto_analyze` configuration option
+- Add `alphabetize_schema` configuration option
+- Fix `backfill_column_for_type_change_in_background` for cast expressions
+- Fix copying indexes with long names when changing column type
+- Enhance error messages with the link to the detailed description
+
 ## 0.9.2 (2023-11-02)
 
 - Fix checking which expression indexes to copy when changing column type

data/README.md

@@ -16,10 +16,12 @@ See [comparison to `strong_migrations`](#comparison-to-strong_migrations)
 
 ## Requirements
 
-- Ruby 2.1+
-- Rails 4.2+
+- Ruby 2.7+
+- Rails 6.1+
 - PostgreSQL 9.6+
 
+For older Ruby and Rails versions you can use '< 0.11' version of this gem.
+
 **Note**: Since some migration helpers use database `VIEW`s to implement their logic, it is recommended to use `structure.sql` schema format, or otherwise add some gem (like [scenic](https://github.com/scenic-views/scenic)) to be able to dump them into the `schema.rb`.
 
 ## Installation
@@ -35,10 +37,20 @@ And then run:
 ```sh
 $ bundle install
 $ bin/rails generate online_migrations:install
+$ bin/rails db:migrate
 ```
 
 **Note**: If you do not have plans on using [background migrations](docs/background_migrations.md) feature, then you can delete the generated migration and regenerate it later, if needed.
 
+### Upgrading
+
+If you're already using [background migrations](docs/background_migrations.md), your background migrations tables may require additional columns. After every upgrade run:
+
+```sh
+$ bin/rails generate online_migrations:upgrade
+$ bin/rails db:migrate
+```
+
 ## Motivation
 
 Writing a safe migration can be daunting. Numerous articles have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of PostgreSQL, remembering all the subtleties of explicit locking can be problematic.
@@ -90,7 +102,7 @@ class AddAdminToUsers < ActiveRecord::Migration[7.1]
     execute "SET statement_timeout TO '5s'"
     change_column_null :users, :admin, false
   end
-  
+
   def down
     remove_column :users, :admin
   end
@@ -177,30 +189,22 @@ end
 
 1. Ignore the column:
 
-  ```ruby
-  # For Active Record 5+
-  class User < ApplicationRecord
-    self.ignored_columns = ["name"]
-  end
-
-  # For Active Record < 5
-  class User < ActiveRecord::Base
-    def self.columns
-      super.reject { |c| c.name == "name" }
-    end
-  end
-  ```
+   ```ruby
+   class User < ApplicationRecord
+     self.ignored_columns = ["name"]
+   end
+   ```
 
 2. Deploy
 3. Wrap column removing in a `safety_assured` block:
 
-  ```ruby
-  class RemoveNameFromUsers < ActiveRecord::Migration[7.1]
-    def change
-      safety_assured { remove_column :users, :name }
-    end
-  end
-  ```
+   ```ruby
+   class RemoveNameFromUsers < ActiveRecord::Migration[7.1]
+     def change
+       safety_assured { remove_column :users, :name }
+     end
+   end
+   ```
 
 4. Remove column ignoring from `User` model
 5. Deploy
@@ -316,67 +320,82 @@ Type | Safe Changes
 
 :white_check_mark: **Good**
 
+#### "Classic" approach (abstract)
+
+1. Create a new column
+2. Write to both columns
+3. Backfill data from the old column to the new column
+4. Move reads from the old column to the new column
+5. Stop writing to the old column
+6. Drop the old column
+
+#### :bullettrain_side: Concrete steps for Active Record
+
 **Note**: The following steps can also be used to change the primary key's type (e.g., from `integer` to `bigint`).
 
 A safer approach can be accomplished in several steps:
 
 1. Create a new column and keep column's data in sync:
 
-  ```ruby
-  class InitializeChangeFilesSizeType < ActiveRecord::Migration[7.1]
-    def change
-      initialize_column_type_change :files, :size, :bigint
-    end
-  end
-  ```
+   ```ruby
+   class InitializeChangeFilesSizeType < ActiveRecord::Migration[7.1]
+     def change
+       initialize_column_type_change :files, :size, :bigint
+     end
+   end
+   ```
 
-**Note**: `initialize_column_type_change` accepts additional options (like `:limit`, `:default` etc)
-which will be passed to `add_column` when creating a new column, so you can override previous values.
+   **Note**: `initialize_column_type_change` accepts additional options (like `:limit`, `:default` etc)
+   which will be passed to `add_column` when creating a new column, so you can override previous values.
 
 2. Backfill data from the old column to the new column:
 
-  ```ruby
-  class BackfillChangeFilesSizeType < ActiveRecord::Migration[7.1]
-    disable_ddl_transaction!
+   ```ruby
+   class BackfillChangeFilesSizeType < ActiveRecord::Migration[7.1]
+     disable_ddl_transaction!
 
-    def up
-      backfill_column_for_type_change :files, :size
-    end
+     def up
+       backfill_column_for_type_change :files, :size
+     end
 
-    def down
-      # no op
-    end
-  end
-  ```
+     def down
+       # no op
+     end
+   end
+   ```
 
-3. Copy indexes, foreign keys, check constraints, NOT NULL constraint, swap new column in place:
+3. Make sure your application works with values in both formats (when read from the database, converting
+during writes works automatically). For most column type changes, this does not need any updates in the app.
+4. Deploy
+5. Copy indexes, foreign keys, check constraints, NOT NULL constraint, swap new column in place:
 
-  ```ruby
-  class FinalizeChangeFilesSizeType < ActiveRecord::Migration[7.1]
-    disable_ddl_transaction!
+   ```ruby
+   class FinalizeChangeFilesSizeType < ActiveRecord::Migration[7.1]
+     disable_ddl_transaction!
 
-    def change
-      finalize_column_type_change :files, :size
-    end
-  end
-  ```
+     def change
+       finalize_column_type_change :files, :size
+     end
+   end
+   ```
 
-4. Deploy
-5. Finally, if everything is working as expected, remove copy trigger and old column:
+6. Deploy
+7. Finally, if everything works as expected, remove copy trigger and old column:
 
-  ```ruby
-  class CleanupChangeFilesSizeType < ActiveRecord::Migration[7.1]
-    def up
-      cleanup_column_type_change :files, :size
-    end
+   ```ruby
+   class CleanupChangeFilesSizeType < ActiveRecord::Migration[7.1]
+     def up
+       cleanup_column_type_change :files, :size
+     end
 
-    def down
-      initialize_column_type_change :files, :size, :integer
-    end
-  end
-  ```
+     def down
+       initialize_column_type_change :files, :size, :integer
+     end
+   end
+   ```
 
-6. Deploy
+8. Remove changes from step 3, if any
+9. Deploy
 
 ### Renaming a column
 
@@ -430,67 +449,61 @@ To work around this limitation, we need to tell Active Record to acquire this in
 
 1. Instruct Rails that you are going to rename a column:
 
-```ruby
-OnlineMigrations.config.column_renames = {
-  "users" => {
-    "name" => "first_name"
-  }
-}
-```
-NOTE: You also need to temporarily enable partial writes (is disabled by default in Active Record >= 7)
-until the process of column rename is fully done.
-```ruby
-# config/application.rb
-# For Active Record >= 7
-config.active_record.partial_inserts = true
+   ```ruby
+   OnlineMigrations.config.column_renames = {
+     "users" => {
+       "name" => "first_name"
+     }
+   }
+   ```
 
-# Or for Active Record < 7
-config.active_record.partial_writes = true
-```
+   **Note**: You also need to temporarily enable partial writes (is disabled by default in Active Record >= 7)
+   until the process of column rename is fully done.
+
+   ```ruby
+   # config/application.rb
+   # For Active Record >= 7
+   config.active_record.partial_inserts = true
+
+   # Or for Active Record < 7
+   config.active_record.partial_writes = true
+   ```
 
 2. Deploy
 3. Tell the database that you are going to rename a column. This will not actually rename any columns,
 nor any data/indexes/foreign keys copying will be made, so will be instantaneous.
 It will use a combination of a VIEW and column aliasing to work with both column names simultaneously
 
-```ruby
-class InitializeRenameUsersNameToFirstName < ActiveRecord::Migration[7.1]
-  def change
-    initialize_column_rename :users, :name, :first_name
-  end
-end
-```
+   ```ruby
+   class InitializeRenameUsersNameToFirstName < ActiveRecord::Migration[7.1]
+     def change
+       initialize_column_rename :users, :name, :first_name
+     end
+   end
+   ```
 
 4. Replace usages of the old column with a new column in the codebase
 5. If you enabled Active Record `enumerate_columns_in_select_statements` setting in your application
-  (is disabled by default in Active Record >= 7), then you need to ignore old column:
+   (is disabled by default in Active Record >= 7), then you need to ignore old column:
 
-  ```ruby
-  # For Active Record 5+
-  class User < ApplicationRecord
-    self.ignored_columns = ["name"]
-  end
-
-  # For Active Record < 5
-  class User < ActiveRecord::Base
-    def self.columns
-      super.reject { |c| c.name == "name" }
-    end
-  end
-  ```
+   ```ruby
+   class User < ApplicationRecord
+     self.ignored_columns = ["name"]
+   end
+   ```
 
 6. Deploy
 7. Remove the column rename config from step 1
 8. Remove the column ignore from step 5, if added
 9. Remove the VIEW created in step 3 and finally rename the column:
 
-```ruby
-class FinalizeRenameUsersNameToFirstName < ActiveRecord::Migration[7.1]
-  def change
-    finalize_column_rename :users, :name, :first_name
-  end
-end
-```
+   ```ruby
+   class FinalizeRenameUsersNameToFirstName < ActiveRecord::Migration[7.1]
+     def change
+       finalize_column_rename :users, :name, :first_name
+     end
+   end
+   ```
 
 10. Deploy
 
@@ -546,35 +559,35 @@ To work around this limitation, we need to tell Active Record to acquire this in
 
 1. Instruct Rails that you are going to rename a table:
 
-```ruby
-OnlineMigrations.config.table_renames = {
-  "clients" => "users"
-}
-```
+   ```ruby
+   OnlineMigrations.config.table_renames = {
+     "clients" => "users"
+   }
+   ```
 
 2. Deploy
 3. Create a VIEW:
 
-```ruby
-class InitializeRenameClientsToUsers < ActiveRecord::Migration[7.1]
-  def change
-    initialize_table_rename :clients, :users
-  end
-end
-```
+   ```ruby
+   class InitializeRenameClientsToUsers < ActiveRecord::Migration[7.1]
+     def change
+       initialize_table_rename :clients, :users
+     end
+   end
+   ```
 
 4. Replace usages of the old table with a new table in the codebase
 5. Remove the table rename config from step 1
 6. Deploy
 7. Remove the VIEW created in step 3:
 
-```ruby
-class FinalizeRenameClientsToUsers < ActiveRecord::Migration[7.1]
-  def change
-    finalize_table_rename :clients, :users
-  end
-end
-```
+   ```ruby
+   class FinalizeRenameClientsToUsers < ActiveRecord::Migration[7.1]
+     def change
+       finalize_table_rename :clients, :users
+     end
+   end
+   ```
 
 8. Deploy
 
@@ -1024,7 +1037,7 @@ end
 
 :x: **Bad**
 
-Adding multiple foreign keys in a single migration blocks reads and writes on all involved tables until migration is completed.
+Adding multiple foreign keys in a single migration blocks writes on all involved tables until migration is completed.
 Avoid adding foreign key more than once per migration file, unless the source and target tables are identical.
 
 ```ruby
@@ -1157,19 +1170,11 @@ A safer approach is to:
 
 1. ignore the column:
 
-  ```ruby
-  # For Active Record 5+
-  class User < ApplicationRecord
-    self.ignored_columns = ["type"]
-  end
-
-  # For Active Record < 5
-  class User < ActiveRecord::Base
-    def self.columns
-      super.reject { |c| c.name == "type" }
-    end
-  end
-  ```
+   ```ruby
+   class User < ApplicationRecord
+     self.ignored_columns = ["type"]
+   end
+   ```
 
 2. deploy
 3. remove the column ignoring from step 1 and apply initial code changes
@@ -1281,18 +1286,18 @@ The main differences are:
 
 1. `strong_migrations` provides you **text guidance** on how to run migrations safer and you should implement them yourself. This new gem has actual [**code helpers**](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/schema_statements.rb) (and suggests them when fails on unsafe migrations) you can use to do what you want. See [example](#example) for an example.
 
-It has migrations helpers for:
+   It has migrations helpers for:
 
-* renaming tables/columns
-* changing columns types (including changing primary/foreign keys from `integer` to `bigint`)
-* adding columns with default values
-* backfilling data
-* adding different types of constraints
-* and others
+     * renaming tables/columns
+     * changing columns types (including changing primary/foreign keys from `integer` to `bigint`)
+     * adding columns with default values
+     * backfilling data
+     * adding different types of constraints
+     * and others
 
 2. This gem has a [powerful internal framework](https://github.com/fatkodima/online_migrations/blob/master/docs/background_migrations.md) for running data migrations on very large tables using background migrations.
 
-For example, you can use background migrations to migrate data that’s stored in a single JSON column to a separate table instead; backfill values from one column to another (as one of the steps when changing column type); or backfill some column’s value from an API.
+   For example, you can use background migrations to migrate data that’s stored in a single JSON column to a separate table instead; backfill values from one column to another (as one of the steps when changing column type); or backfill some column’s value from an API.
 
 3. Yet, it has more checks for unsafe changes (see [checks](#checks)).
 

data/docs/background_migrations.md

@@ -18,7 +18,7 @@ Start a background migrations scheduler. For example, to run it on cron using [w
 
 ```ruby
 every 1.minute do
-  runner "OnlineMigrations::BackgroundMigrations::Scheduler.run"
+  runner "OnlineMigrations.run_background_migrations"
 end
 ```
 
@@ -116,13 +116,15 @@ enqueue_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
 
 ## Predefined background migrations
 
-* `BackfillColumn` - backfills column(s) with scalar values (enqueue using `backfill_column_in_background`)
+* `BackfillColumn` - backfills column(s) with scalar values (enqueue using `backfill_column_in_background`; or `backfill_column_for_type_change_in_background` if backfilling column for which type change is in progress)
 * `CopyColumn` - copies data from one column(s) to other(s) (enqueue using `copy_column_in_background`)
 * `DeleteAssociatedRecords` - deletes records associated with a parent object (enqueue using `delete_associated_records_in_background`)
 * `DeleteOrphanedRecords` - deletes records with one or more missing relations (enqueue using `delete_orphaned_records_in_background`)
 * `PerformActionOnRelation` - performs specific action on a relation or individual records (enqueue using `perform_action_on_relation_in_background`)
 * `ResetCounters` - resets one or more counter caches to their correct value (enqueue using `reset_counters_in_background`)
 
+**Note**: These migration helpers should be run inside the migration against the database where background migrations tables are defined.
+
 ## Testing
 
 At a minimum, it's recommended that the `#process_batch` method in your background migration is tested. You may also want to test the `#relation` and `#count` methods if they are sufficiently complex.
@@ -137,17 +139,18 @@ require "test_helper"
 module OnlineMigrations
   module BackgroundMigrations
     class BackfillProjectIssuesCountTest < ActiveSupport::TestCase
-      test "#process_batch performs a background migration iteration" do
-        rails = Project.create!(name: "rails")
+      test "#process_batch performs an iteration" do
+        rails = Project.create!(name: "Ruby on Rails")
         postgres = Project.create!(name: "PostgreSQL")
 
         2.times { rails.issues.create! }
-        _postgres_issue = postgres.issues.create!
+        postgres.issues.create!
 
-        BackfillProjectIssuesCount.new.process_batch(Project.all)
+        migration = BackfillProjectIssuesCount.new
+        migration.process_batch(migration.relation)
 
-        assert_equal 2, rails.issues_count
-        assert_equal 1, postgres.issues_count
+        assert_equal 2, rails.reload.issues_count
+        assert_equal 1, postgres.reload.issues_count
       end
     end
   end
@@ -220,7 +223,7 @@ To get the progress (assuming `#count` method on background migration class was
 
 ```ruby
 migration = OnlineMigrations::BackgroundMigrations::Migration.find(id)
-migration.progress # value from 0 to 1.0
+migration.progress # value from 0 to 100.0
 ```
 
 **Note**: It will be easier to work with background migrations through some kind of Web UI, but until it is implemented, we can work with them only manually.
@@ -229,7 +232,19 @@ migration.progress # value from 0 to 1.0
 
 There are a few configurable options for the Background Migrations. Custom configurations should be placed in a `online_migrations.rb` initializer.
 
-**Note**: Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_migrations/config.rb) for the list of all available configuration options.
+Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_migrations/config.rb) for the list of all available configuration options.
+
+**Note**: You can dynamically change certain migration parameters while the migration is run.
+For example,
+```ruby
+migration = OnlineMigrations::BackgroundMigrations::Migration.find(id)
+migration.update!(
+  batch_size: 50_000,      # The # of records migration will update per run
+  sub_batch_size: 10_000,  # The # of records migration will update via single `UPDATE`
+  batch_pause: 1.second,   # Minimum time (in seconds) between successive migration runs
+  sub_batch_pause_ms: 20   # Minimum time (in ms) between successive migration `UPDATE`s
+)
+```
 
 ### Throttling
 
@@ -293,3 +308,21 @@ OnlineMigrations.config.background_migrations.backtrace_cleaner = cleaner
 ```
 
 If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.
+
+### Multiple databases and sharding
+
+If you have multiple databases or sharding, you may need to configure where background migrations related tables live
+by configuring the parent model:
+
+```ruby
+# config/initializers/online_migrations.rb
+
+# Referring to one of the databases
+OnlineMigrations::BackgroundMigrations::ApplicationRecord.connects_to database: { writing: :animals }
+
+# Referring to one of the shards (via `:database` option)
+OnlineMigrations::BackgroundMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
+```
+
+By default, ActiveRecord uses the database config named `:primary` (if exists) under the environment section from the `database.yml`.
+Otherwise, the first config under the environment section is used.