online_migrations 0.10.0 → 0.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 966629fafbc4d8c9de88240bc705799f66ab205754cd1acc99a1ae129d737de5
-  data.tar.gz: 61379d6eeec17e532314551d5a643ed465ba23bac69739be063f2fd704657451
+  metadata.gz: 9ceda570f99a1712496ac5c9549859c7dc2cad7e3cc4ab59c97a22ba17f3bfd0
+  data.tar.gz: dfc5a83147a92bf5e04a532210e72b6f32f561f032b2a1b746ff142bf4e16635
 SHA512:
-  metadata.gz: ae49cb8499bc7b086d9b1883106926d150a1227f495e1aff7e1f259197d68065a20b819797d5e7c9199d413be746e9deb94f7c9bc6005a53b9b94bbefd2d88cb
-  data.tar.gz: 4c36ad512328174d04600b7d10001721ed985ab73184cbfd7c2f2cde70788a32ef4a4e8de754134206510b29e17920da06820903f86a809766aa3c5aa4005408
+  metadata.gz: 0ff6cce820134ef6b893f9405b71a0d8f8cd42dd79616123aaa492983671ebcff98dbff21477afa78b29342d40098822da4b8de0306489faf956b21ee44e2113
+  data.tar.gz: 42dcf132290c9affe812ef7489397d4c05d48f33069f3ee09710fced6aaf8ac928fee63c1498333657a824f98885e3af5e706e4dcc4a089b9cd62284ad0223d1

data/CHANGELOG.md

@@ -1,5 +1,38 @@
 ## master (unreleased)
 
+## 0.11.0 (2024-01-09)
+
+- Support sharding for background migrations
+
+    Now, if a `relation` inside background migration definition is defined on a sharded model,
+    then that background migration would automatically run on all the shards.
+
+    To get all the new sharding related schema changes, you need to run:
+
+    ```sh
+    $ bin/rails generate online_migrations:upgrade
+    $ bin/rails db:migrate
+    ```
+
+- Change background migration `progress` to return values in range from 0.0 to 100.0
+
+    Previously, these were values in range from 0.0 to 1.0 and caused confusion
+
+- Copy exclusion constraints when changing column type
+- Update `revert_finalize_columns_type_change` to not remove indexes, foreign keys etc
+- Fix verbose query logging when `ActiveRecord::Base.logger` is `nil`
+- Add a shortcut for running background migrations
+
+    ```ruby
+    # Before:
+    OnlineMigrations::BackgroundMigrations::Scheduler.run
+    # After
+    OnlineMigrations.run_background_migrations
+    ```
+
+- Add support for `:type_cast_function` to `initialize_column_type_change` helper
+- Drop support for Ruby < 2.7 and Rails < 6.1
+
 ## 0.10.0 (2023-12-12)
 
 - Add `auto_analyze` configuration option

data/README.md

@@ -16,10 +16,12 @@ See [comparison to `strong_migrations`](#comparison-to-strong_migrations)
 
 ## Requirements
 
-- Ruby 2.1+
-- Rails 4.2+
+- Ruby 2.7+
+- Rails 6.1+
 - PostgreSQL 9.6+
 
+For older Ruby and Rails versions you can use '< 0.11' version of this gem.
+
 **Note**: Since some migration helpers use database `VIEW`s to implement their logic, it is recommended to use `structure.sql` schema format, or otherwise add some gem (like [scenic](https://github.com/scenic-views/scenic)) to be able to dump them into the `schema.rb`.
 
 ## Installation
@@ -35,10 +37,20 @@ And then run:
 ```sh
 $ bundle install
 $ bin/rails generate online_migrations:install
+$ bin/rails db:migrate
 ```
 
 **Note**: If you do not have plans on using [background migrations](docs/background_migrations.md) feature, then you can delete the generated migration and regenerate it later, if needed.
 
+### Upgrading
+
+If you're already using [background migrations](docs/background_migrations.md), your background migrations tables may require additional columns. After every upgrade run:
+
+```sh
+$ bin/rails generate online_migrations:upgrade
+$ bin/rails db:migrate
+```
+
 ## Motivation
 
 Writing a safe migration can be daunting. Numerous articles have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of PostgreSQL, remembering all the subtleties of explicit locking can be problematic.
@@ -178,17 +190,9 @@ end
 1. Ignore the column:
 
    ```ruby
-   # For Active Record 5+
    class User < ApplicationRecord
      self.ignored_columns = ["name"]
    end
-
-   # For Active Record < 5
-   class User < ActiveRecord::Base
-     def self.columns
-       super.reject { |c| c.name == "name" }
-     end
-   end
    ```
 
 2. Deploy
@@ -316,6 +320,17 @@ Type | Safe Changes
 
 :white_check_mark: **Good**
 
+#### "Classic" approach (abstract)
+
+1. Create a new column
+2. Write to both columns
+3. Backfill data from the old column to the new column
+4. Move reads from the old column to the new column
+5. Stop writing to the old column
+6. Drop the old column
+
+#### :bullettrain_side: Concrete steps for Active Record
+
 **Note**: The following steps can also be used to change the primary key's type (e.g., from `integer` to `bigint`).
 
 A safer approach can be accomplished in several steps:
@@ -349,7 +364,10 @@ A safer approach can be accomplished in several steps:
    end
    ```
 
-3. Copy indexes, foreign keys, check constraints, NOT NULL constraint, swap new column in place:
+3. Make sure your application works with values in both formats (when read from the database, converting
+during writes works automatically). For most column type changes, this does not need any updates in the app.
+4. Deploy
+5. Copy indexes, foreign keys, check constraints, NOT NULL constraint, swap new column in place:
 
    ```ruby
    class FinalizeChangeFilesSizeType < ActiveRecord::Migration[7.1]
@@ -361,8 +379,8 @@ A safer approach can be accomplished in several steps:
    end
    ```
 
-4. Deploy
-5. Finally, if everything is working as expected, remove copy trigger and old column:
+6. Deploy
+7. Finally, if everything works as expected, remove copy trigger and old column:
 
    ```ruby
    class CleanupChangeFilesSizeType < ActiveRecord::Migration[7.1]
@@ -376,7 +394,8 @@ A safer approach can be accomplished in several steps:
    end
    ```
 
-6. Deploy
+8. Remove changes from step 3, if any
+9. Deploy
 
 ### Renaming a column
 
@@ -468,17 +487,9 @@ It will use a combination of a VIEW and column aliasing to work with both column
    (is disabled by default in Active Record >= 7), then you need to ignore old column:
 
    ```ruby
-   # For Active Record 5+
    class User < ApplicationRecord
      self.ignored_columns = ["name"]
    end
-
-   # For Active Record < 5
-   class User < ActiveRecord::Base
-     def self.columns
-       super.reject { |c| c.name == "name" }
-     end
-   end
    ```
 
 6. Deploy
@@ -1026,7 +1037,7 @@ end
 
 :x: **Bad**
 
-Adding multiple foreign keys in a single migration blocks reads and writes on all involved tables until migration is completed.
+Adding multiple foreign keys in a single migration blocks writes on all involved tables until migration is completed.
 Avoid adding foreign key more than once per migration file, unless the source and target tables are identical.
 
 ```ruby
@@ -1160,17 +1171,9 @@ A safer approach is to:
 1. ignore the column:
 
    ```ruby
-   # For Active Record 5+
    class User < ApplicationRecord
      self.ignored_columns = ["type"]
    end
-
-   # For Active Record < 5
-   class User < ActiveRecord::Base
-     def self.columns
-       super.reject { |c| c.name == "type" }
-     end
-   end
    ```
 
 2. deploy

data/docs/background_migrations.md

@@ -18,7 +18,7 @@ Start a background migrations scheduler. For example, to run it on cron using [w
 
 ```ruby
 every 1.minute do
-  runner "OnlineMigrations::BackgroundMigrations::Scheduler.run"
+  runner "OnlineMigrations.run_background_migrations"
 end
 ```
 
@@ -116,13 +116,15 @@ enqueue_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
 
 ## Predefined background migrations
 
-* `BackfillColumn` - backfills column(s) with scalar values (enqueue using `backfill_column_in_background`)
+* `BackfillColumn` - backfills column(s) with scalar values (enqueue using `backfill_column_in_background`; or `backfill_column_for_type_change_in_background` if backfilling column for which type change is in progress)
 * `CopyColumn` - copies data from one column(s) to other(s) (enqueue using `copy_column_in_background`)
 * `DeleteAssociatedRecords` - deletes records associated with a parent object (enqueue using `delete_associated_records_in_background`)
 * `DeleteOrphanedRecords` - deletes records with one or more missing relations (enqueue using `delete_orphaned_records_in_background`)
 * `PerformActionOnRelation` - performs specific action on a relation or individual records (enqueue using `perform_action_on_relation_in_background`)
 * `ResetCounters` - resets one or more counter caches to their correct value (enqueue using `reset_counters_in_background`)
 
+**Note**: These migration helpers should be run inside the migration against the database where background migrations tables are defined.
+
 ## Testing
 
 At a minimum, it's recommended that the `#process_batch` method in your background migration is tested. You may also want to test the `#relation` and `#count` methods if they are sufficiently complex.
@@ -221,7 +223,7 @@ To get the progress (assuming `#count` method on background migration class was
 
 ```ruby
 migration = OnlineMigrations::BackgroundMigrations::Migration.find(id)
-migration.progress # value from 0 to 1.0
+migration.progress # value from 0 to 100.0
 ```
 
 **Note**: It will be easier to work with background migrations through some kind of Web UI, but until it is implemented, we can work with them only manually.
@@ -230,7 +232,19 @@ migration.progress # value from 0 to 1.0
 
 There are a few configurable options for the Background Migrations. Custom configurations should be placed in a `online_migrations.rb` initializer.
 
-**Note**: Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_migrations/config.rb) for the list of all available configuration options.
+Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_migrations/config.rb) for the list of all available configuration options.
+
+**Note**: You can dynamically change certain migration parameters while the migration is run.
+For example,
+```ruby
+migration = OnlineMigrations::BackgroundMigrations::Migration.find(id)
+migration.update!(
+  batch_size: 50_000,      # The # of records migration will update per run
+  sub_batch_size: 10_000,  # The # of records migration will update via single `UPDATE`
+  batch_pause: 1.second,   # Minimum time (in seconds) between successive migration runs
+  sub_batch_pause_ms: 20   # Minimum time (in ms) between successive migration `UPDATE`s
+)
+```
 
 ### Throttling
 
@@ -294,3 +308,21 @@ OnlineMigrations.config.background_migrations.backtrace_cleaner = cleaner
 ```
 
 If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.
+
+### Multiple databases and sharding
+
+If you have multiple databases or sharding, you may need to configure where background migrations related tables live
+by configuring the parent model:
+
+```ruby
+# config/initializers/online_migrations.rb
+
+# Referring to one of the databases
+OnlineMigrations::BackgroundMigrations::ApplicationRecord.connects_to database: { writing: :animals }
+
+# Referring to one of the shards (via `:database` option)
+OnlineMigrations::BackgroundMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
+```
+
+By default, ActiveRecord uses the database config named `:primary` (if exists) under the environment section from the `database.yml`.
+Otherwise, the first config under the environment section is used.

data/docs/configuring.md

@@ -86,14 +86,15 @@ config.lock_retrier = OnlineMigrations::ExponentialLockRetrier.new(
   attempts: 30,                # attempt 30 retries
   base_delay: 0.01.seconds,    # starting with delay of 10ms between each unsuccessful try, increasing exponentially
   max_delay: 1.minute,         # maximum delay is 1 minute
-  lock_timeout: 0.05.seconds   # and 50ms set as lock timeout for each try
+  lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try
 )
 ```
 
 When statement within transaction fails - the whole transaction is retried.
 
 To permanently disable lock retries, you can set `lock_retrier` to `nil`.
-To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable.
+
+To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable. This is useful when you are deploying a hotfix and do not want to wait too long while the lock retrier safely tries to acquire the lock, but try to acquire the lock immediately with the default configured lock timeout value.
 
 **Note**: Statements are retried by default, unless lock retries are disabled. It is possible to implement more sophisticated lock retriers. See [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/lock_retrier.rb) for the examples.
 

data/lib/generators/online_migrations/install_generator.rb

@@ -15,20 +15,16 @@ module OnlineMigrations
     end
 
     def create_migration_file
-      migration_template("migration.rb", File.join(migrations_dir, "install_online_migrations.rb"))
+      migration_template("migration.rb", File.join(db_migrate_path, "install_online_migrations.rb"))
     end
 
     private
       def migration_parent
-        Utils.migration_parent_string
+        "ActiveRecord::Migration[#{Utils.ar_version}]"
       end
 
       def start_after
-        self.class.current_migration_number(migrations_dir)
-      end
-
-      def migrations_dir
-        Utils.ar_version >= 5.1 ? db_migrate_path : "db/migrate"
+        self.class.current_migration_number(db_migrate_path)
       end
   end
 end

data/lib/generators/online_migrations/templates/add_sharding_to_online_migrations.rb.tt

@@ -0,0 +1,18 @@
+class AddShardingToOnlineMigrations < <%= migration_parent %>
+  def change
+    safety_assured do
+      remove_index :background_migrations, [:migration_name, :arguments], unique: true
+
+      change_table :background_migrations do |t|
+        t.bigint :parent_id
+        t.string :shard
+        t.boolean :composite, default: false, null: false
+
+        t.foreign_key :background_migrations, column: :parent_id, on_delete: :cascade
+
+        t.index [:migration_name, :arguments, :shard],
+          unique: true, name: :index_background_migrations_on_unique_configuration
+      end
+    end
+  end
+end

data/lib/generators/online_migrations/templates/initializer.rb.tt

@@ -31,12 +31,14 @@ OnlineMigrations.configure do |config|
   # config.alphabetize_schema = true
 
   # Disable specific checks.
-  # For the list of available checks look at `lib/error_messages` folder.
+  # For the list of available checks look at the `error_messages.rb` file inside
+  # the `online_migrations` gem.
   # config.disable_check(:remove_index)
 
   # Enable specific checks. All checks are enabled by default,
   # but this may change in the future.
-  # For the list of available checks look at `lib/error_messages` folder.
+  # For the list of available checks look at the `error_messages.rb` file inside
+  # the `online_migrations` gem.
   # config.enable_check(:remove_index)
 
   # Configure whether to log every SQL query happening in a migration.
@@ -55,7 +57,7 @@ OnlineMigrations.configure do |config|
     attempts: 30,                # attempt 30 retries
     base_delay: 0.01.seconds,    # starting with delay of 10ms between each unsuccessful try, increasing exponentially
     max_delay: 1.minute,         # up to the maximum delay of 1 minute
-    lock_timeout: 0.05.seconds   # and 50ms set as lock timeout for each try
+    lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try
   )
 
   # Configure tables that are in the process of being renamed.

data/lib/generators/online_migrations/templates/migration.rb.tt

@@ -1,6 +1,7 @@
 class InstallOnlineMigrations < <%= migration_parent %>
   def change
     create_table :background_migrations do |t|
+      t.bigint :parent_id
       t.string :migration_name, null: false
       t.jsonb :arguments, default: [], null: false
       t.string :batch_column_name, null: false
@@ -13,9 +14,13 @@ class InstallOnlineMigrations < <%= migration_parent %>
       t.integer :sub_batch_pause_ms, null: false
       t.integer :batch_max_attempts, null: false
       t.string :status, default: "enqueued", null: false
-      t.timestamps null: false
+      t.string :shard
+      t.boolean :composite, default: false, null: false
+      t.timestamps
 
-      t.index [:migration_name, :arguments],
+      t.foreign_key :background_migrations, column: :parent_id, on_delete: :cascade
+
+      t.index [:migration_name, :arguments, :shard],
         unique: true, name: :index_background_migrations_on_unique_configuration
     end
 
@@ -34,7 +39,7 @@ class InstallOnlineMigrations < <%= migration_parent %>
       t.string :error_class
       t.string :error_message
       t.string :backtrace, array: true
-      t.timestamps null: false
+      t.timestamps
 
       t.foreign_key :background_migrations, column: :migration_id, on_delete: :cascade
 

data/lib/generators/online_migrations/upgrade_generator.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record/migration"
+
+module OnlineMigrations
+  # @private
+  class UpgradeGenerator < Rails::Generators::Base
+    include ActiveRecord::Generators::Migration
+
+    source_root File.expand_path("templates", __dir__)
+
+    def copy_templates
+      migrations_to_be_applied.each do |migration|
+        migration_template("#{migration}.rb", File.join(db_migrate_path, "#{migration}.rb"))
+      end
+    end
+
+    private
+      def migrations_to_be_applied
+        connection = BackgroundMigrations::Migration.connection
+        columns = connection.columns(BackgroundMigrations::Migration.table_name).map(&:name)
+
+        migrations = []
+        migrations << "add_sharding_to_online_migrations" if !columns.include?("shard")
+        migrations
+      end
+
+      def migration_parent
+        "ActiveRecord::Migration[#{Utils.ar_version}]"
+      end
+  end
+end

data/lib/online_migrations/background_migrations/application_record.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module OnlineMigrations
+  module BackgroundMigrations
+    # Base class for all records used by this gem.
+    #
+    # Can be extended to setup different database where all tables related to
+    # online_migrations will live.
+    class ApplicationRecord < ActiveRecord::Base
+      self.abstract_class = true
+    end
+  end
+end

data/lib/online_migrations/background_migrations/backfill_column.rb

@@ -22,7 +22,7 @@ module OnlineMigrations
           quoted_column = connection.quote_column_name(column)
           model.unscoped.where("#{quoted_column} != ? OR #{quoted_column} IS NULL", value)
         else
-          Utils.ar_where_not_multiple_conditions(model.unscoped, updates)
+          model.unscoped.where.not(updates)
         end
       end
 

data/lib/online_migrations/background_migrations/copy_column.rb

@@ -25,14 +25,10 @@ module OnlineMigrations
       end
 
       def relation
-        relation = model
+        model
           .unscoped
-          .where(copy_to.map { |to_column| [to_column, nil] }.to_h)
-
-        Utils.ar_where_not_multiple_conditions(
-          relation,
-          copy_from.map { |from_column| [from_column, nil] }.to_h
-        )
+          .where(copy_to.index_with(nil))
+          .where.not(copy_from.index_with(nil))
       end
 
       def process_batch(relation)
@@ -43,13 +39,8 @@ module OnlineMigrations
           old_value = arel_table[from_column]
           if (type_cast_function = type_cast_functions[from_column])
             old_value =
-              if type_cast_function =~ /\A\w+\z/
-                if Utils.ar_version <= 5.2
-                  # Active Record <= 5.2 does not support quoting of Arel::Nodes::NamedFunction
-                  Arel.sql("#{type_cast_function}(#{connection.quote_column_name(from_column)})")
-                else
-                  Arel::Nodes::NamedFunction.new(type_cast_function, [old_value])
-                end
+              if type_cast_function.match?(/\A\w+\z/)
+                Arel::Nodes::NamedFunction.new(type_cast_function, [old_value])
               else
                 # We got a cast expression.
                 Arel.sql(type_cast_function)
@@ -58,12 +49,7 @@ module OnlineMigrations
           old_value
         end
 
-        if Utils.ar_version <= 4.2
-          stmt = Arel::UpdateManager.new(arel.engine)
-        else
-          stmt = Arel::UpdateManager.new
-        end
-
+        stmt = Arel::UpdateManager.new
         stmt.table(arel_table)
         stmt.wheres = arel.constraints
 

data/lib/online_migrations/background_migrations/delete_orphaned_records.rb

@@ -12,29 +12,11 @@ module OnlineMigrations
       end
 
       def relation
-        # For Active Record 6.1+ we can use `where.missing`
-        # https://github.com/rails/rails/pull/34727
-        associations.inject(model.unscoped) do |relation, association|
-          reflection = model.reflect_on_association(association)
-          if reflection.nil?
-            raise ArgumentError, "'#{model.name}' has no association called '#{association}'"
-          end
-
-          # left_joins was added in Active Record 5.0 - https://github.com/rails/rails/pull/12071
-          relation
-            .left_joins(association)
-            .where(reflection.table_name => { reflection.association_primary_key => nil })
-        end
+        model.unscoped.where.missing(*associations)
       end
 
       def process_batch(relation)
-        if Utils.ar_version > 5.0
-          relation.delete_all
-        else
-          # Older Active Record generates incorrect query when running delete_all
-          primary_key = model.primary_key
-          model.unscoped.where(primary_key => relation.select(primary_key)).delete_all
-        end
+        relation.delete_all
       end
 
       def count