online_migrations 0.9.2 → 0.11.0

data/docs/configuring.md

@@ -3,6 +3,8 @@
 There are a few configurable options for the gem. Custom configurations should be placed in a `online_migrations.rb` initializer.
 
 ```ruby
+# config/initializers/online_migrations.rb
+
 OnlineMigrations.configure do |config|
   # ...
 end
@@ -15,8 +17,6 @@ end
 Add your own custom checks with:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.add_check do |method, args|
   if method == :add_column && args[0].to_s == "users"
     stop!("No more columns on the users table")
@@ -33,8 +33,6 @@ Use the `stop!` method to stop migrations.
 Disable specific checks with:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.disable_check(:remove_index)
 ```
 
@@ -45,8 +43,6 @@ Check the [source code](https://github.com/fatkodima/online_migrations/blob/mast
 By default, checks are disabled when migrating down. Enable them with:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.check_down = true
 ```
 
@@ -55,8 +51,6 @@ config.check_down = true
 You can customize specific error messages:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.error_messages[:add_column_default] = "Your custom instructions"
 ```
 
@@ -88,20 +82,19 @@ ALTER ROLE myuser SET statement_timeout = '15s';
 You can configure this gem to automatically retry statements that exceed the lock timeout:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.lock_retrier = OnlineMigrations::ExponentialLockRetrier.new(
   attempts: 30,                # attempt 30 retries
   base_delay: 0.01.seconds,    # starting with delay of 10ms between each unsuccessful try, increasing exponentially
   max_delay: 1.minute,         # maximum delay is 1 minute
-  lock_timeout: 0.05.seconds   # and 50ms set as lock timeout for each try
+  lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try
 )
 ```
 
 When statement within transaction fails - the whole transaction is retried.
 
 To permanently disable lock retries, you can set `lock_retrier` to `nil`.
-To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable.
+
+To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable. This is useful when you are deploying a hotfix and do not want to wait too long while the lock retrier safely tries to acquire the lock, but try to acquire the lock immediately with the default configured lock timeout value.
 
 **Note**: Statements are retried by default, unless lock retries are disabled. It is possible to implement more sophisticated lock retriers. See [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/lock_retrier.rb) for the examples.
 
@@ -110,8 +103,6 @@ To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_
 To mark migrations as safe that were created before installing this gem, configure the migration version starting after which checks are performed:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.start_after = 20220101000000
 
 # or if you use multiple databases (Active Record 6+)
@@ -125,8 +116,6 @@ Use the version from your latest migration.
 If your development database version is different from production, you can specify the production version so the right checks run in development.
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.target_version = 10 # or "12.9" etc
 
 # or if you use multiple databases (Active Record 6+)
@@ -200,9 +189,25 @@ So you can actually check which steps are performed.
 To enable verbose sql logs:
 
 ```ruby
-# config/initializers/online_migrations.rb
-
 config.verbose_sql_logs = true
 ```
 
 This feature is enabled by default in a production Rails environment. You can override this setting via `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
+
+## Analyze Tables
+
+Analyze tables automatically (to update planner statistics) after an index is added.
+Add to an initializer file:
+
+```ruby
+config.auto_analyze = true
+```
+
+## Schema Sanity
+
+Columns can flip order in `db/schema.rb` when you have multiple developers. One way to prevent this is to [alphabetize them](https://www.pgrs.net/2008/03/12/alphabetize-schema-rb-columns/).
+To alphabetize columns:
+
+```ruby
+config.alphabetize_schema = true
+```

data/lib/generators/online_migrations/install_generator.rb

@@ -15,20 +15,16 @@ module OnlineMigrations
     end
 
     def create_migration_file
-      migration_template("migration.rb", File.join(migrations_dir, "install_online_migrations.rb"))
+      migration_template("migration.rb", File.join(db_migrate_path, "install_online_migrations.rb"))
     end
 
     private
       def migration_parent
-        Utils.migration_parent_string
+        "ActiveRecord::Migration[#{Utils.ar_version}]"
       end
 
       def start_after
-        self.class.current_migration_number(migrations_dir)
-      end
-
-      def migrations_dir
-        Utils.ar_version >= 5.1 ? db_migrate_path : "db/migrate"
+        self.class.current_migration_number(db_migrate_path)
       end
   end
 end

data/lib/generators/online_migrations/templates/add_sharding_to_online_migrations.rb.tt

@@ -0,0 +1,18 @@
+class AddShardingToOnlineMigrations < <%= migration_parent %>
+  def change
+    safety_assured do
+      remove_index :background_migrations, [:migration_name, :arguments], unique: true
+
+      change_table :background_migrations do |t|
+        t.bigint :parent_id
+        t.string :shard
+        t.boolean :composite, default: false, null: false
+
+        t.foreign_key :background_migrations, column: :parent_id, on_delete: :cascade
+
+        t.index [:migration_name, :arguments, :shard],
+          unique: true, name: :index_background_migrations_on_unique_configuration
+      end
+    end
+  end
+end

data/lib/generators/online_migrations/templates/initializer.rb.tt

@@ -23,13 +23,22 @@ OnlineMigrations.configure do |config|
   # It is considered safe to perform most of the dangerous operations on them.
   # config.small_tables = []
 
+  # Analyze tables after indexes are added.
+  # Outdated statistics can sometimes hurt performance.
+  # config.auto_analyze = true
+
+  # Alphabetize table columns when dumping the schema.
+  # config.alphabetize_schema = true
+
   # Disable specific checks.
-  # For the list of available checks look at `lib/error_messages` folder.
+  # For the list of available checks look at the `error_messages.rb` file inside
+  # the `online_migrations` gem.
   # config.disable_check(:remove_index)
 
   # Enable specific checks. All checks are enabled by default,
   # but this may change in the future.
-  # For the list of available checks look at `lib/error_messages` folder.
+  # For the list of available checks look at the `error_messages.rb` file inside
+  # the `online_migrations` gem.
   # config.enable_check(:remove_index)
 
   # Configure whether to log every SQL query happening in a migration.
@@ -48,7 +57,7 @@ OnlineMigrations.configure do |config|
     attempts: 30,                # attempt 30 retries
     base_delay: 0.01.seconds,    # starting with delay of 10ms between each unsuccessful try, increasing exponentially
     max_delay: 1.minute,         # up to the maximum delay of 1 minute
-    lock_timeout: 0.05.seconds   # and 50ms set as lock timeout for each try
+    lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try
   )
 
   # Configure tables that are in the process of being renamed.

data/lib/generators/online_migrations/templates/migration.rb.tt

@@ -1,6 +1,7 @@
 class InstallOnlineMigrations < <%= migration_parent %>
   def change
     create_table :background_migrations do |t|
+      t.bigint :parent_id
       t.string :migration_name, null: false
       t.jsonb :arguments, default: [], null: false
       t.string :batch_column_name, null: false
@@ -13,9 +14,13 @@ class InstallOnlineMigrations < <%= migration_parent %>
       t.integer :sub_batch_pause_ms, null: false
       t.integer :batch_max_attempts, null: false
       t.string :status, default: "enqueued", null: false
-      t.timestamps null: false
+      t.string :shard
+      t.boolean :composite, default: false, null: false
+      t.timestamps
 
-      t.index [:migration_name, :arguments],
+      t.foreign_key :background_migrations, column: :parent_id, on_delete: :cascade
+
+      t.index [:migration_name, :arguments, :shard],
         unique: true, name: :index_background_migrations_on_unique_configuration
     end
 
@@ -34,7 +39,7 @@ class InstallOnlineMigrations < <%= migration_parent %>
       t.string :error_class
       t.string :error_message
       t.string :backtrace, array: true
-      t.timestamps null: false
+      t.timestamps
 
       t.foreign_key :background_migrations, column: :migration_id, on_delete: :cascade
 

data/lib/generators/online_migrations/upgrade_generator.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record/migration"
+
+module OnlineMigrations
+  # @private
+  class UpgradeGenerator < Rails::Generators::Base
+    include ActiveRecord::Generators::Migration
+
+    source_root File.expand_path("templates", __dir__)
+
+    def copy_templates
+      migrations_to_be_applied.each do |migration|
+        migration_template("#{migration}.rb", File.join(db_migrate_path, "#{migration}.rb"))
+      end
+    end
+
+    private
+      def migrations_to_be_applied
+        connection = BackgroundMigrations::Migration.connection
+        columns = connection.columns(BackgroundMigrations::Migration.table_name).map(&:name)
+
+        migrations = []
+        migrations << "add_sharding_to_online_migrations" if !columns.include?("shard")
+        migrations
+      end
+
+      def migration_parent
+        "ActiveRecord::Migration[#{Utils.ar_version}]"
+      end
+  end
+end

data/lib/online_migrations/background_migrations/application_record.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+module OnlineMigrations
+  module BackgroundMigrations
+    # Base class for all records used by this gem.
+    #
+    # Can be extended to setup different database where all tables related to
+    # online_migrations will live.
+    class ApplicationRecord < ActiveRecord::Base
+      self.abstract_class = true
+    end
+  end
+end

data/lib/online_migrations/background_migrations/backfill_column.rb

@@ -22,7 +22,7 @@ module OnlineMigrations
           quoted_column = connection.quote_column_name(column)
           model.unscoped.where("#{quoted_column} != ? OR #{quoted_column} IS NULL", value)
         else
-          Utils.ar_where_not_multiple_conditions(model.unscoped, updates)
+          model.unscoped.where.not(updates)
         end
       end
 

data/lib/online_migrations/background_migrations/copy_column.rb

@@ -25,14 +25,10 @@ module OnlineMigrations
       end
 
       def relation
-        relation = model
+        model
           .unscoped
-          .where(copy_to.map { |to_column| [to_column, nil] }.to_h)
-
-        Utils.ar_where_not_multiple_conditions(
-          relation,
-          copy_from.map { |from_column| [from_column, nil] }.to_h
-        )
+          .where(copy_to.index_with(nil))
+          .where.not(copy_from.index_with(nil))
       end
 
       def process_batch(relation)
@@ -42,22 +38,18 @@ module OnlineMigrations
         old_values = copy_from.map do |from_column|
           old_value = arel_table[from_column]
           if (type_cast_function = type_cast_functions[from_column])
-            if Utils.ar_version <= 5.2
-              # Active Record <= 5.2 does not support quoting of Arel::Nodes::NamedFunction
-              old_value = Arel.sql("#{type_cast_function}(#{connection.quote_column_name(from_column)})")
-            else
-              old_value = Arel::Nodes::NamedFunction.new(type_cast_function, [old_value])
-            end
+            old_value =
+              if type_cast_function.match?(/\A\w+\z/)
+                Arel::Nodes::NamedFunction.new(type_cast_function, [old_value])
+              else
+                # We got a cast expression.
+                Arel.sql(type_cast_function)
+              end
           end
           old_value
         end
 
-        if Utils.ar_version <= 4.2
-          stmt = Arel::UpdateManager.new(arel.engine)
-        else
-          stmt = Arel::UpdateManager.new
-        end
-
+        stmt = Arel::UpdateManager.new
         stmt.table(arel_table)
         stmt.wheres = arel.constraints
 

data/lib/online_migrations/background_migrations/delete_orphaned_records.rb

@@ -12,29 +12,11 @@ module OnlineMigrations
       end
 
       def relation
-        # For Active Record 6.1+ we can use `where.missing`
-        # https://github.com/rails/rails/pull/34727
-        associations.inject(model.unscoped) do |relation, association|
-          reflection = model.reflect_on_association(association)
-          if reflection.nil?
-            raise ArgumentError, "'#{model.name}' has no association called '#{association}'"
-          end
-
-          # left_joins was added in Active Record 5.0 - https://github.com/rails/rails/pull/12071
-          relation
-            .left_joins(association)
-            .where(reflection.table_name => { reflection.association_primary_key => nil })
-        end
+        model.unscoped.where.missing(*associations)
       end
 
       def process_batch(relation)
-        if Utils.ar_version > 5.0
-          relation.delete_all
-        else
-          # Older Active Record generates incorrect query when running delete_all
-          primary_key = model.primary_key
-          model.unscoped.where(primary_key => relation.select(primary_key)).delete_all
-        end
+        relation.delete_all
       end
 
       def count

data/lib/online_migrations/background_migrations/migration.rb

@@ -2,7 +2,7 @@
 
 module OnlineMigrations
   module BackgroundMigrations
-    class Migration < ActiveRecord::Base
+    class Migration < ApplicationRecord
       STATUSES = [
         :enqueued,    # The migration has been enqueued by the user.
         :running,     # The migration is being performed by a migration executor.
@@ -15,25 +15,29 @@ module OnlineMigrations
       self.table_name = :background_migrations
 
       scope :queue_order, -> { order(created_at: :asc) }
+      scope :runnable, -> { where(composite: false) }
       scope :active, -> { where(status: [statuses[:enqueued], statuses[:running]]) }
+      scope :except_succeeded, -> { where.not(status: :succeeded) }
       scope :for_migration_name, ->(migration_name) { where(migration_name: normalize_migration_name(migration_name)) }
       scope :for_configuration, ->(migration_name, arguments) do
         for_migration_name(migration_name).where("arguments = ?", arguments.to_json)
       end
 
-      enum status: STATUSES.map { |status| [status, status.to_s] }.to_h
+      enum status: STATUSES.index_with(&:to_s)
 
+      belongs_to :parent, class_name: name, optional: true
+      has_many :children, class_name: name, foreign_key: :parent_id
       has_many :migration_jobs
 
       validates :migration_name, :batch_column_name, presence: true
 
-      validates :min_value, :max_value, :batch_size, :sub_batch_size,
-                  presence: true, numericality: { greater_than: 0 }
+      validates :batch_size, :sub_batch_size, presence: true, numericality: { greater_than: 0 }
+      validates :min_value, :max_value, presence: true, numericality: { greater_than: 0, unless: :composite? }
 
       validates :batch_pause, :sub_batch_pause_ms, presence: true,
                   numericality: { greater_than_or_equal_to: 0 }
-      validates :rows_count, numericality: { greater_than_or_equal_to: 0 }, allow_nil: true
-      validates :arguments, uniqueness: { scope: :migration_name }
+      validates :rows_count, numericality: { greater_than_or_equal_to: 0 }, allow_nil: true, unless: :composite?
+      validates :arguments, uniqueness: { scope: [:migration_name, :shard] }
 
       validate :validate_batch_column_values
       validate :validate_batch_sizes
@@ -43,6 +47,8 @@ module OnlineMigrations
       validates_with MigrationStatusValidator, on: :update
 
       before_validation :set_defaults
+      before_create :create_child_migrations, if: :composite?
+      before_update :copy_attributes_to_children, if: :composite?
 
       # @private
       def self.normalize_migration_name(migration_name)
@@ -58,28 +64,53 @@ module OnlineMigrations
         succeeded? || failed?
       end
 
+      # Overwrite enum's generated method to correctly work for composite migrations.
+      def paused!
+        return super if !composite?
+
+        transaction do
+          super
+          children.each { |child| child.paused! if child.enqueued? || child.running? }
+        end
+      end
+
+      # Overwrite enum's generated method to correctly work for composite migrations.
+      def running!
+        return super if !composite?
+
+        transaction do
+          super
+          children.each { |child| child.running! if child.paused? }
+        end
+      end
+
       def last_job
-        migration_jobs.order(max_value: :desc).first
+        migration_jobs.order(:max_value).last
       end
 
       def last_completed_job
-        migration_jobs.completed.order(finished_at: :desc).first
+        migration_jobs.completed.order(:finished_at).last
       end
 
       # Returns the progress of the background migration.
       #
       # @return [Float, nil]
-      #   - when background migration is configured to not to track progress, returns `nil`
-      #   - otherwise returns value in range of 0.0 and 1.0
+      #   - when background migration is configured to not track progress, returns `nil`
+      #   - otherwise returns value in range from 0.0 to 100.0
       #
       def progress
         if succeeded?
-          1.0
+          100.0
+        elsif composite?
+          progresses = children.map(&:progress).compact
+          if progresses.any?
+            (progresses.sum / progresses.size).round(2)
+          end
         elsif rows_count
           jobs_rows_count = migration_jobs.succeeded.sum(:batch_size)
           # The last migration job may need to process the amount of rows
           # less than the batch size, so we can get a value > 1.0.
-          [jobs_rows_count.to_f / rows_count, 1.0].min
+          ([jobs_rows_count.to_f / rows_count, 1.0].min * 100).round(2)
         end
       end
 
@@ -95,13 +126,19 @@ module OnlineMigrations
         migration_object.relation
       end
 
+      def migration_model
+        migration_relation.model
+      end
+
       # Returns whether the interval between previous step run has passed.
       # @return [Boolean]
       #
       def interval_elapsed?
-        if migration_jobs.running.exists?
+        last_active_job = migration_jobs.active.order(:updated_at).last
+
+        if last_active_job && !last_active_job.stuck?
           false
-        elsif (job = last_completed_job)
+        elsif batch_pause > 0 && (job = last_completed_job)
           job.finished_at + batch_pause <= Time.current
         else
           true
@@ -123,6 +160,14 @@ module OnlineMigrations
         end
       end
 
+      # @private
+      def on_shard(&block)
+        abstract_class = find_abstract_class(migration_model)
+
+        shard = (self.shard || abstract_class.default_shard).to_sym
+        abstract_class.connected_to(shard: shard, role: :writing, &block)
+      end
+
       # @private
       def reset_failed_jobs_attempts
         iterator = BatchIterator.new(migration_jobs.failed.attempts_exceeded)
@@ -138,16 +183,10 @@ module OnlineMigrations
 
         # rubocop:disable Lint/UnreachableLoop
         iterator.each_batch(of: batch_size, column: batch_column_name, start: next_min_value) do |relation|
-          if Utils.ar_version <= 4.2
-            # Active Record <= 4.2 does not support pluck with Arel nodes
-            quoted_column = self.class.connection.quote_column_name(batch_column_name)
-            batch_range = relation.pluck("MIN(#{quoted_column}), MAX(#{quoted_column})").first
-          else
-            min = relation.arel_table[batch_column_name].minimum
-            max = relation.arel_table[batch_column_name].maximum
+          min = relation.arel_table[batch_column_name].minimum
+          max = relation.arel_table[batch_column_name].maximum
+          batch_range = relation.pick(min, max)
 
-            batch_range = relation.pluck(min, max).first
-          end
           break
         end
         # rubocop:enable Lint/UnreachableLoop
@@ -162,6 +201,10 @@ module OnlineMigrations
         [min_value, max_value]
       end
 
+      protected
+        attr_accessor :child
+        alias child? child
+
       private
         def validate_batch_column_values
           if max_value.to_i < min_value.to_i
@@ -176,7 +219,13 @@ module OnlineMigrations
         end
 
         def validate_jobs_status
-          if succeeded? && migration_jobs.except_succeeded.exists?
+          if composite?
+            if succeeded? && children.except_succeeded.exists?
+              errors.add(:base, "all child migrations must be succeeded")
+            elsif failed? && !children.failed.exists?
+              errors.add(:base, "at least one child migration must be failed")
+            end
+          elsif succeeded? && migration_jobs.except_succeeded.exists?
             errors.add(:base, "all migration jobs must be succeeded")
           elsif failed? && !migration_jobs.failed.exists?
             errors.add(:base, "at least one migration job must be failed")
@@ -185,12 +234,30 @@ module OnlineMigrations
 
         def set_defaults
           if migration_relation.is_a?(ActiveRecord::Relation)
-            self.batch_column_name  ||= migration_relation.primary_key
-            self.min_value          ||= migration_relation.minimum(batch_column_name)
-            self.max_value          ||= migration_relation.maximum(batch_column_name)
-
-            count = migration_object.count
-            self.rows_count = count if count != :no_count
+            if !child?
+              shards = Utils.shard_names(migration_model)
+              self.composite = shards.size > 1
+            end
+
+            self.batch_column_name ||= migration_relation.primary_key
+
+            if composite?
+              self.min_value = self.max_value = self.rows_count = -1 # not relevant
+            else
+              on_shard do
+                self.min_value ||= migration_relation.minimum(batch_column_name)
+                self.max_value ||= migration_relation.maximum(batch_column_name)
+
+                # This can be the case when run in development on empty tables
+                if min_value.nil?
+                  # integer IDs minimum value is 1
+                  self.min_value = self.max_value = 1
+                end
+
+                count = migration_object.count
+                self.rows_count = count if count != :no_count
+              end
+            end
           end
 
           config = ::OnlineMigrations.config.background_migrations
@@ -199,12 +266,27 @@ module OnlineMigrations
           self.batch_pause          ||= config.batch_pause
           self.sub_batch_pause_ms   ||= config.sub_batch_pause_ms
           self.batch_max_attempts   ||= config.batch_max_attempts
+        end
+
+        def create_child_migrations
+          shards = Utils.shard_names(migration_model)
 
-          # This can be the case when run in development on empty tables
-          if min_value.nil?
-            # integer IDs minimum value is 1
-            self.min_value = self.max_value = 1
+          children = shards.map do |shard|
+            child = Migration.new(migration_name: migration_name, arguments: arguments, shard: shard)
+            child.child = true
+            child
           end
+
+          self.children = children
+        end
+
+        def copy_attributes_to_children
+          attributes = [:batch_size, :sub_batch_size, :batch_pause, :sub_batch_pause_ms, :batch_max_attempts]
+          updates = {}
+          attributes.each do |attribute|
+            updates[attribute] = read_attribute(attribute) if attribute_changed?(attribute)
+          end
+          children.update_all(updates) if updates.any?
         end
 
         def next_min_value
@@ -214,6 +296,13 @@ module OnlineMigrations
             min_value
           end
         end
+
+        def find_abstract_class(model)
+          model.ancestors.find do |parent|
+            parent == ActiveRecord::Base ||
+              (parent.is_a?(Class) && parent.abstract_class?)
+          end
+        end
     end
   end
 end

data/lib/online_migrations/background_migrations/migration_helpers.rb

@@ -223,10 +223,6 @@ module OnlineMigrations
       #     For smaller tables it is probably better and easier to directly find and delete orpahed records.
       #
       def delete_orphaned_records_in_background(model_name, *associations, **options)
-        if Utils.ar_version <= 4.2
-          raise "#{__method__} does not support Active Record <= 4.2 yet"
-        end
-
         model_name = model_name.name if model_name.is_a?(Class)
 
         enqueue_background_migration(