online_migrations 0.11.1 → 0.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: c08a823ebaea1855fcfaae87c3252cbc4a727f04ae564cd14cfcb8e2f43ebb2b
-  data.tar.gz: f02621f945b90de34217f588d26a243b506bc47c9433feab784b8db43ecd83de
+  metadata.gz: 34bd3f3fc2c18bc49962183603c7f2c85e167d978eafd3fb840b639fa98de60d
+  data.tar.gz: 0ae0b82440ea7dcec1183c987395742fab0c93618b384f770d7acd3fecd41cdf
 SHA512:
-  metadata.gz: 7bb01d066a03e7bb85c6d6e37f6c3e67351dc1a890579d9eb4c8ef7a200785918c863a7b2f1104a1d531772da0405daa1f9724cad6b224a92da08fba3fac2479
-  data.tar.gz: 21acc8fbfb9a2aab13d5768f1df613d74e9145d55148bdb8b04b53e5687bca9fd0e6f88178282dec9f0642fdad2fd188e979d152b697b771554adf866aa417e3
+  metadata.gz: 321b877ffe09ccf2edb94b43d1060f5ebc8d83c1c058a41dc1d248f4d52c161b5a4576186363349c6c027d20890375728590271d24bdb75647d0556a57202aa9
+  data.tar.gz: d6a25cf51e31b772e4556b18de85ed88c5019b0469a8d6299e955063042c2e939f4f3f228e05ce4fdc8a4060b2ee2f4160a1e5d477a8769439b103f03b103fb8

data/CHANGELOG.md

@@ -1,5 +1,29 @@
 ## master (unreleased)
 
+## 0.13.0 (2024-01-22)
+
+- Add ability to configure the path where generated background migrations will be placed
+
+    ```ruby
+    # It is placed in lib/ by default.
+    config.background_migrations.migrations_path = "app/lib"
+    ```
+
+- Reduce number of queries needed to calculate batch ranges for background migrations
+- Fix `finalize_column_type_change` to not recreate already existing indexes on the temporary column
+- Remove potentially heavy queries used to get the ranges of a background migration
+
+## 0.12.0 (2024-01-18)
+
+- Require passing model name for background migration helpers when using multiple databases
+- Add `statement_timeout` configuration option
+
+- Make `lock_timeout` argument optional for `config.lock_retrier`
+
+    This way, a default lock timeout value will be used (configured in `database.yml` or for the database user).
+
+- Fix a bug that can lead to unfinished children of a sharded background migration
+
 ## 0.11.1 (2024-01-11)
 
 - Fix calculation of batch ranges for sharded background migrations

data/docs/background_migrations.md

@@ -255,7 +255,7 @@ Specify the throttle condition as a block:
 ```ruby
 # config/initializers/online_migrations.rb
 
-OnlineMigrations.config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
+config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
 ```
 
 Note that it's up to you to define a throttling condition that makes sense for your app. For example, you can check various PostgreSQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
@@ -269,7 +269,7 @@ If you want to integrate with an exception monitoring service (e.g. Bugsnag), yo
 ```ruby
 # config/initializers/online_migrations.rb
 
-OnlineMigrations.config.background_migrations.error_handler = ->(error, errored_job) do
+config.background_migrations.error_handler = ->(error, errored_job) do
   Bugsnag.notify(error) do |notification|
     notification.add_metadata(:background_migration, { name: errored_job.migration_name })
   end
@@ -281,22 +281,34 @@ The error handler should be a lambda that accepts 2 arguments:
 * `error`: The exception that was raised.
 * `errored_job`: An `OnlineMigrations::BackgroundMigrations::MigrationJob` object that represents a failed batch.
 
+### Customizing the background migrations path
+
+`OnlineMigrations.config.background_migrations.migrations_path` can be configured to define where generated background migrations will be placed.
+
+```ruby
+# config/initializers/online_migrations.rb
+
+config.background_migrations.migrations_path = "app/lib"
+```
+
+If no value is specified, it will default to `"lib"`.
+
 ### Customizing the background migrations module
 
-`OnlineMigrations.config.background_migrations.migrations_module` can be configured to define the module in which
+`config.background_migrations.migrations_module` can be configured to define the module in which
 background migrations will be placed.
 
 ```ruby
 # config/initializers/online_migrations.rb
 
-OnlineMigrations.config.background_migrations.migrations_module = "BackgroundMigrationsModule"
+config.background_migrations.migrations_module = "BackgroundMigrationsModule"
 ```
 
-If no value is specified, it will default to `OnlineMigrations::BackgroundMigrations`.
+If no value is specified, it will default to `"OnlineMigrations::BackgroundMigrations"`.
 
 ### Customizing the backtrace cleaner
 
-`OnlineMigrations.config.background_migrations.backtrace_cleaner` can be configured to specify a backtrace cleaner to use when a Background Migration errors and the backtrace is cleaned and persisted. An `ActiveSupport::BacktraceCleaner` should be used.
+`config.background_migrations.backtrace_cleaner` can be configured to specify a backtrace cleaner to use when a Background Migration errors and the backtrace is cleaned and persisted. An `ActiveSupport::BacktraceCleaner` should be used.
 
 ```ruby
 # config/initializers/online_migrations.rb
@@ -304,7 +316,7 @@ If no value is specified, it will default to `OnlineMigrations::BackgroundMigrat
 cleaner = ActiveSupport::BacktraceCleaner.new
 cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
 
-OnlineMigrations.config.background_migrations.backtrace_cleaner = cleaner
+config.background_migrations.backtrace_cleaner = cleaner
 ```
 
 If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.

data/docs/configuring.md

@@ -59,22 +59,33 @@ Check the [source code](https://github.com/fatkodima/online_migrations/blob/mast
 ## Migration Timeouts
 
 It’s extremely important to set a short lock timeout for migrations. This way, if a migration can't acquire a lock in a timely manner, other statements won't be stuck behind it.
+We also recommend setting a long statement timeout so migrations can run for a while.
 
-Add timeouts to `config/database.yml`:
+You can configure a statement timeout for migrations via:
 
-```yml
-production:
-  connect_timeout: 5
-  variables:
-    lock_timeout: 10s
-    statement_timeout: 15s
+```ruby
+config.statement_timeout = 1.hour
 ```
 
+and a lock timeout for migrations can be configured via the `lock_retrier`.
+
 Or set the timeouts directly on the database user that runs migrations:
 
 ```sql
 ALTER ROLE myuser SET lock_timeout = '10s';
-ALTER ROLE myuser SET statement_timeout = '15s';
+ALTER ROLE myuser SET statement_timeout = '1h';
+```
+
+## App Timeouts
+
+We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.
+
+```yml
+production:
+  connect_timeout: 5
+  variables:
+    lock_timeout: 10s
+    statement_timeout: 15s
 ```
 
 ## Lock Timeout Retries
@@ -86,7 +97,7 @@ config.lock_retrier = OnlineMigrations::ExponentialLockRetrier.new(
   attempts: 30,                # attempt 30 retries
   base_delay: 0.01.seconds,    # starting with delay of 10ms between each unsuccessful try, increasing exponentially
   max_delay: 1.minute,         # maximum delay is 1 minute
-  lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try
+  lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try. Remove this line to use a default lock timeout.
 )
 ```
 
@@ -192,7 +203,7 @@ To enable verbose sql logs:
 config.verbose_sql_logs = true
 ```
 
-This feature is enabled by default in a production Rails environment. You can override this setting via `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
+This feature is enabled by default in a staging and production Rails environments. You can override this setting via `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
 
 ## Analyze Tables
 

data/lib/generators/online_migrations/background_migration_generator.rb

@@ -9,8 +9,11 @@ module OnlineMigrations
     desc "This generator creates a background migration file."
 
     def create_background_migration_file
+      migrations_module_file_path = migrations_module.underscore
+
       template_file = File.join(
-        "lib/#{migrations_module_file_path}",
+        config.migrations_path,
+        migrations_module_file_path,
         class_path,
         "#{file_name}.rb"
       )
@@ -18,12 +21,12 @@ module OnlineMigrations
     end
 
     private
-      def migrations_module_file_path
-        migrations_module.underscore
+      def migrations_module
+        config.migrations_module
       end
 
-      def migrations_module
-        OnlineMigrations.config.background_migrations.migrations_module
+      def config
+        OnlineMigrations.config.background_migrations
       end
   end
 end

data/lib/generators/online_migrations/templates/initializer.rb.tt

@@ -4,6 +4,9 @@ OnlineMigrations.configure do |config|
   # Configure the migration version starting after which checks are performed.
   # config.start_after = <%= start_after %>
 
+  # Configure statement timeout used for migrations.
+  config.statement_timeout = 1.hour
+
   # Set the version of the production database so the right checks are run in development.
   # config.target_version = 10
 
@@ -48,7 +51,7 @@ OnlineMigrations.configure do |config|
   # a better grasp of what is going on for high-level statements like add_column_with_default.
   #
   # Note: It can be overridden by `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
-  config.verbose_sql_logs = defined?(Rails) && Rails.env.production?
+  config.verbose_sql_logs = defined?(Rails.env) && (Rails.env.production? || Rails.env.staging?)
 
   # Lock retries.
   # Configure your custom lock retrier (see LockRetrier).
@@ -57,7 +60,7 @@ OnlineMigrations.configure do |config|
     attempts: 30,                # attempt 30 retries
     base_delay: 0.01.seconds,    # starting with delay of 10ms between each unsuccessful try, increasing exponentially
     max_delay: 1.minute,         # up to the maximum delay of 1 minute
-    lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try
+    lock_timeout: 0.2.seconds    # and 200ms set as lock timeout for each try. Remove this line to use a default lock timeout.
   )
 
   # Configure tables that are in the process of being renamed.
@@ -75,6 +78,13 @@ OnlineMigrations.configure do |config|
   # end
 
   # ==> Background migrations configuration
+
+  # The path where generated background migrations will be placed.
+  # config.background_migrations.migrations_path = "lib"
+
+  # The module in which background migrations will be placed.
+  # config.background_migrations.migrations_module = "OnlineMigrations::BackgroundMigrations"
+
   # The number of rows to process in a single background migration run.
   # config.background_migrations.batch_size = 20_000
 

data/lib/online_migrations/background_migrations/config.rb

@@ -4,7 +4,11 @@ module OnlineMigrations
   module BackgroundMigrations
     # Class representing configuration options for background migrations.
     class Config
-      # The module to namespace background migrations in
+      # The path where generated background migrations will be placed
+      # @return [String] defaults to "lib"
+      attr_accessor :migrations_path
+
+      # The module in which background migrations will be placed
       # @return [String] defaults to "OnlineMigrations::BackgroundMigrations"
       attr_accessor :migrations_module
 
@@ -75,6 +79,7 @@ module OnlineMigrations
       attr_accessor :error_handler
 
       def initialize
+        @migrations_path = "lib"
         @migrations_module = "OnlineMigrations::BackgroundMigrations"
         @batch_size = 20_000
         @sub_batch_size = 1000

data/lib/online_migrations/background_migrations/migration.rb

@@ -2,6 +2,11 @@
 
 module OnlineMigrations
   module BackgroundMigrations
+    # Class representing background data migration.
+    #
+    # @note The records of this class should not be created manually, but via
+    #   `enqueue_background_migration` helper inside migrations.
+    #
     class Migration < ApplicationRecord
       STATUSES = [
         :enqueued,    # The migration has been enqueued by the user.
@@ -23,11 +28,13 @@ module OnlineMigrations
         for_migration_name(migration_name).where("arguments = ?", arguments.to_json)
       end
 
+      alias_attribute :name, :migration_name
+
       enum status: STATUSES.index_with(&:to_s)
 
       belongs_to :parent, class_name: name, optional: true
-      has_many :children, class_name: name, foreign_key: :parent_id
-      has_many :migration_jobs
+      has_many :children, class_name: name, foreign_key: :parent_id, dependent: :delete_all
+      has_many :migration_jobs, dependent: :delete_all
 
       validates :migration_name, :batch_column_name, presence: true
 
@@ -47,7 +54,6 @@ module OnlineMigrations
       validates_with MigrationStatusValidator, on: :update
 
       before_validation :set_defaults
-      before_create :create_child_migrations, if: :composite?
       before_update :copy_attributes_to_children, if: :composite?
 
       # @private
@@ -57,8 +63,10 @@ module OnlineMigrations
       end
 
       def migration_name=(class_name)
+        class_name = class_name.name if class_name.is_a?(Class)
         write_attribute(:migration_name, self.class.normalize_migration_name(class_name))
       end
+      alias name= migration_name=
 
       def completed?
         succeeded? || failed?
@@ -189,10 +197,11 @@ module OnlineMigrations
 
         on_shard do
           # rubocop:disable Lint/UnreachableLoop
-          iterator.each_batch(of: batch_size, column: batch_column_name, start: next_min_value) do |relation|
-            min = relation.arel_table[batch_column_name].minimum
-            max = relation.arel_table[batch_column_name].maximum
-            batch_range = relation.pick(min, max)
+          iterator.each_batch(of: batch_size, column: batch_column_name, start: next_min_value) do |relation, min_value, max_value|
+            if max_value.nil?
+              max_value = relation.pick(relation.arel_table[batch_column_name].maximum)
+            end
+            batch_range = [min_value, max_value]
 
             break
           end
@@ -209,10 +218,6 @@ module OnlineMigrations
         [min_value, max_value]
       end
 
-      protected
-        attr_accessor :child
-        alias child? child
-
       private
         def validate_batch_column_values
           if max_value.to_i < min_value.to_i
@@ -242,25 +247,16 @@ module OnlineMigrations
 
         def set_defaults
           if migration_relation.is_a?(ActiveRecord::Relation)
-            if !child?
-              shards = Utils.shard_names(migration_model)
-              self.composite = shards.size > 1
-            end
-
             self.batch_column_name ||= migration_relation.primary_key
 
             if composite?
               self.min_value = self.max_value = self.rows_count = -1 # not relevant
             else
               on_shard do
-                self.min_value ||= migration_relation.minimum(batch_column_name)
-                self.max_value ||= migration_relation.maximum(batch_column_name)
-
-                # This can be the case when run in development on empty tables
-                if min_value.nil?
-                  # integer IDs minimum value is 1
-                  self.min_value = self.max_value = 1
-                end
+                # Getting exact min/max values can be a very heavy operation
+                # and is not needed practically.
+                self.min_value ||= 1
+                self.max_value ||= migration_model.unscoped.maximum(batch_column_name) || self.min_value
 
                 count = migration_object.count
                 self.rows_count = count if count != :no_count
@@ -276,18 +272,6 @@ module OnlineMigrations
           self.batch_max_attempts   ||= config.batch_max_attempts
         end
 
-        def create_child_migrations
-          shards = Utils.shard_names(migration_model)
-
-          children = shards.map do |shard|
-            child = Migration.new(migration_name: migration_name, arguments: arguments, shard: shard)
-            child.child = true
-            child
-          end
-
-          self.children = children
-        end
-
         def copy_attributes_to_children
           attributes = [:batch_size, :sub_batch_size, :batch_pause, :sub_batch_pause_ms, :batch_max_attempts]
           updates = {}

data/lib/online_migrations/background_migrations/migration_helpers.rb

@@ -42,6 +42,10 @@ module OnlineMigrations
       # @see #backfill_column_in_background
       #
       def backfill_columns_in_background(table_name, updates, model_name: nil, **options)
+        if model_name.nil? && Utils.multiple_databases?
+          raise ArgumentError, "You must pass a :model_name when using multiple databases."
+        end
+
         model_name = model_name.name if model_name.is_a?(Class)
 
         enqueue_background_migration(
@@ -99,6 +103,10 @@ module OnlineMigrations
       #
       def backfill_columns_for_type_change_in_background(table_name, *column_names, model_name: nil,
                                                          type_cast_functions: {}, **options)
+        if model_name.nil? && Utils.multiple_databases?
+          raise ArgumentError, "You must pass a :model_name when using multiple databases."
+        end
+
         tmp_columns = column_names.map { |column_name| "#{column_name}_for_type_change" }
         model_name = model_name.name if model_name.is_a?(Class)
 
@@ -153,6 +161,10 @@ module OnlineMigrations
       # @see #copy_column_in_background
       #
       def copy_columns_in_background(table_name, copy_from, copy_to, model_name: nil, type_cast_functions: {}, **options)
+        if model_name.nil? && Utils.multiple_databases?
+          raise ArgumentError, "You must pass a :model_name when using multiple databases."
+        end
+
         model_name = model_name.name if model_name.is_a?(Class)
 
         enqueue_background_migration(
@@ -358,23 +370,41 @@ module OnlineMigrations
       #     in development and test environments
       #
       def enqueue_background_migration(migration_name, *arguments, **options)
+        migration = create_background_migration(migration_name, *arguments, **options)
+
+        # For convenience in dev/test environments
+        if Utils.developer_env?
+          runner = MigrationRunner.new(migration)
+          runner.run_all_migration_jobs
+        end
+
+        migration
+      end
+
+      # @private
+      def create_background_migration(migration_name, *arguments, **options)
         options.assert_valid_keys(:batch_column_name, :min_value, :max_value, :batch_size, :sub_batch_size,
             :batch_pause, :sub_batch_pause_ms, :batch_max_attempts)
 
-        migration_name = migration_name.name if migration_name.is_a?(Class)
-
-        migration = Migration.create!(
+        migration = Migration.new(
           migration_name: migration_name,
           arguments: arguments,
           **options
         )
 
-        # For convenience in dev/test environments
-        if Utils.developer_env?
-          runner = MigrationRunner.new(migration)
-          runner.run_all_migration_jobs
+        shards = Utils.shard_names(migration.migration_model)
+        if shards.size > 1
+          migration.children = shards.map do |shard|
+            child = migration.dup
+            child.shard = shard
+            child
+          end
+
+          migration.composite = true
         end
 
+        # This will save all the records using a transaction.
+        migration.save!
         migration
       end
     end

data/lib/online_migrations/background_migrations/migration_runner.rb

@@ -94,8 +94,10 @@ module OnlineMigrations
 
       private
         def mark_as_running
-          migration.running!
-          migration.parent.running! if migration.parent && migration.parent.enqueued?
+          Migration.transaction do
+            migration.running!
+            migration.parent.running! if migration.parent && migration.parent.enqueued?
+          end
         end
 
         def should_throttle?

data/lib/online_migrations/batch_iterator.rb

@@ -19,37 +19,42 @@ module OnlineMigrations
       end
 
       relation = apply_limits(self.relation, column, start, finish, order)
+      base_relation = relation.reselect(column).reorder(column => order)
 
-      base_relation = relation.except(:select)
-        .select(column)
-        .reorder(column => order)
-
-      start_row = base_relation.uncached { base_relation.first }
-
-      return if !start_row
+      start_id = start || begin
+        start_row = base_relation.uncached { base_relation.first }
+        start_row[column] if start_row
+      end
 
-      start_id = start_row[column]
       arel_table = relation.arel_table
 
-      0.step do |index|
+      while start_id
         if order == :asc
           start_cond = arel_table[column].gteq(start_id)
         else
           start_cond = arel_table[column].lteq(start_id)
         end
 
-        stop_row = base_relation.uncached do
+        last_row, stop_row = base_relation.uncached do
           base_relation
             .where(start_cond)
-            .offset(of)
-            .first
+            .offset(of - 1)
+            .first(2)
+        end
+
+        if last_row.nil?
+          # We are at the end of the table.
+          last_row, stop_row = base_relation.uncached do
+            base_relation
+              .where(start_cond)
+              .last(2)
+          end
         end
 
         batch_relation = relation.where(start_cond)
 
         if stop_row
           stop_id = stop_row[column]
-          start_id = stop_id
 
           if order == :asc
             stop_cond = arel_table[column].lt(stop_id)
@@ -64,10 +69,14 @@ module OnlineMigrations
         # efficient UPDATE queries, hence we get rid of it.
         batch_relation = batch_relation.except(:order)
 
+        last_id = (last_row && last_row[column]) || start_id
+
         # Retaining the results in the query cache would undermine the point of batching.
-        batch_relation.uncached { yield batch_relation, index }
+        batch_relation.uncached { yield batch_relation, start_id, last_id }
+
+        break if stop_row.nil?
 
-        break if !stop_row
+        start_id = stop_id
       end
     end
 

data/lib/online_migrations/change_column_type_helpers.rb

@@ -416,15 +416,8 @@ module OnlineMigrations
               end
             end
 
-          if index.name.include?(from_column)
-            name = index.name.gsub(from_column, to_column)
-          end
-
-          name = index_name(table_name, new_columns) if !name || name.length > max_identifier_length
-
           options = {
             unique: index.unique,
-            name: name,
             length: index.lengths,
             order: index.orders,
           }

data/lib/online_migrations/command_checker.rb

@@ -31,6 +31,7 @@ module OnlineMigrations
 
     def check(command, *args, &block)
       check_database_version
+      set_statement_timeout
       check_lock_timeout
 
       if !safe?
@@ -98,6 +99,16 @@ module OnlineMigrations
         @database_version_checked = true
       end
 
+      def set_statement_timeout
+        if !@statement_timeout_set
+          if (statement_timeout = OnlineMigrations.config.statement_timeout)
+            # TODO: inline this method call after deprecated `disable_statement_timeout` method removal.
+            connection.__set_statement_timeout(statement_timeout)
+          end
+          @statement_timeout_set = true
+        end
+      end
+
       def check_lock_timeout
         limit = OnlineMigrations.config.lock_timeout_limit
 

data/lib/online_migrations/config.rb

@@ -35,6 +35,12 @@ module OnlineMigrations
       end
     end
 
+    # Statement timeout used for migrations (in seconds)
+    #
+    # @return [Numeric]
+    #
+    attr_accessor :statement_timeout
+
     # Set the database version against which the checks will be performed
     #
     # If your development database version is different from production, you can specify
@@ -158,7 +164,7 @@ module OnlineMigrations
     # migration failure in production. This is also useful in development to get
     # a better grasp of what is going on for high-level statements like add_column_with_default.
     #
-    # This feature is enabled by default in a production Rails environment.
+    # This feature is enabled by default in a staging and production Rails environments.
     # @return [Boolean]
     #
     # @note: It can be overridden by `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.

data/lib/online_migrations/lock_retrier.rb

@@ -60,9 +60,7 @@ module OnlineMigrations
     #
     # @param _attempt [Integer] attempt number
     #
-    def lock_timeout(_attempt)
-      raise NotImplementedError
-    end
+    def lock_timeout(_attempt); end
 
     # Returns sleep time after unsuccessful lock attempt (in seconds)
     #
@@ -143,9 +141,9 @@ module OnlineMigrations
     #
     # @param attempts [Integer] Maximum number of attempts
     # @param delay [Numeric] Sleep time after unsuccessful lock attempt (in seconds)
-    # @param lock_timeout [Numeric] Database lock timeout value (in seconds)
+    # @param lock_timeout [Numeric, nil] Database lock timeout value (in seconds)
     #
-    def initialize(attempts:, delay:, lock_timeout:)
+    def initialize(attempts:, delay:, lock_timeout: nil)
       super()
       @attempts = attempts
       @delay = delay
@@ -196,7 +194,7 @@ module OnlineMigrations
     # @param max_delay [Numeric] Maximum sleep time after unsuccessful lock attempt (in seconds)
     # @param lock_timeout [Numeric] Database lock timeout value (in seconds)
     #
-    def initialize(attempts:, base_delay:, max_delay:, lock_timeout:)
+    def initialize(attempts:, base_delay:, max_delay:, lock_timeout: nil)
       super()
       @attempts = attempts
       @base_delay = base_delay

data/lib/online_migrations/schema_statements.rb

@@ -680,34 +680,40 @@ module OnlineMigrations
     #
     # @see https://edgeapi.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-add_index
     #
-    def add_index(table_name, column_name, options = {})
-      algorithm = options[:algorithm]
+    def add_index(table_name, column_name, **options)
+      __ensure_not_in_transaction! if options[:algorithm] == :concurrently
 
-      __ensure_not_in_transaction! if algorithm == :concurrently
-
-      column_names = index_column_names(column_name || options[:column])
-
-      index_name = options[:name]
-      index_name ||= index_name(table_name, column_names)
-
-      if index_exists?(table_name, column_name, **options)
+      # Rewrite this with `IndexDefinition#defined_for?` when Active Record >= 7.1 is supported.
+      # See https://github.com/rails/rails/pull/45160.
+      index = indexes(table_name).find { |i| __index_defined_for?(i, column_name, **options) }
+      if index
         schema = __schema_for_table(table_name)
 
-        if __index_valid?(index_name, schema: schema)
-          Utils.say("Index was not created because it already exists (this may be due to an aborted migration " \
-                    "or similar): table_name: #{table_name}, column_name: #{column_name}")
+        if __index_valid?(index.name, schema: schema)
+          Utils.say("Index was not created because it already exists.")
           return
         else
           Utils.say("Recreating invalid index: table_name: #{table_name}, column_name: #{column_name}")
-          remove_index(table_name, column_name, name: index_name, algorithm: algorithm)
+          remove_index(table_name, column_name, **options)
         end
       end
 
-      disable_statement_timeout do
+      if OnlineMigrations.config.statement_timeout
         # "CREATE INDEX CONCURRENTLY" requires a "SHARE UPDATE EXCLUSIVE" lock.
         # It only conflicts with constraint validations, creating/removing indexes,
         # and some other "ALTER TABLE"s.
-        super(table_name, column_name, **options.merge(name: index_name))
+        super
+      else
+        OnlineMigrations.deprecator.warn(<<~MSG)
+          Running `add_index` without a statement timeout is deprecated.
+          Configure an explicit statement timeout in the initializer file via `config.statement_timeout`
+          or the default database statement timeout will be used.
+          Example, `config.statement_timeout = 1.hour`.
+        MSG
+
+        disable_statement_timeout do
+          super
+        end
       end
     end
 
@@ -716,23 +722,32 @@ module OnlineMigrations
     # @see https://edgeapi.rubyonrails.org/classes/ActiveRecord/ConnectionAdapters/SchemaStatements.html#method-i-remove_index
     #
     def remove_index(table_name, column_name = nil, **options)
-      algorithm = options[:algorithm]
-
-      __ensure_not_in_transaction! if algorithm == :concurrently
+      if column_name.blank? && options[:column].blank? && options[:name].blank?
+        raise ArgumentError, "No name or columns specified"
+      end
 
-      column_names = index_column_names(column_name || options[:column])
+      __ensure_not_in_transaction! if options[:algorithm] == :concurrently
 
-      if index_exists?(table_name, column_names, **options)
-        disable_statement_timeout do
+      if index_exists?(table_name, column_name, **options)
+        if OnlineMigrations.config.statement_timeout
           # "DROP INDEX CONCURRENTLY" requires a "SHARE UPDATE EXCLUSIVE" lock.
           # It only conflicts with constraint validations, other creating/removing indexes,
           # and some "ALTER TABLE"s.
+          super(table_name, column_name, **options)
+        else
+          OnlineMigrations.deprecator.warn(<<~MSG)
+            Running `remove_index` without a statement timeout is deprecated.
+            Configure an explicit statement timeout in the initializer file via `config.statement_timeout`
+            or the default database statement timeout will be used.
+            Example, `config.statement_timeout = 1.hour`.
+          MSG
 
-          super(table_name, **options.merge(column: column_names))
+          disable_statement_timeout do
+            super(table_name, column_name, **options)
+          end
         end
       else
-        Utils.say("Index was not removed because it does not exist (this may be due to an aborted migration " \
-                  "or similar): table_name: #{table_name}, column_name: #{column_names}")
+        Utils.say("Index was not removed because it does not exist.")
       end
     end
 
@@ -778,11 +793,22 @@ module OnlineMigrations
       # Skip costly operation if already validated.
       return if foreign_key.validated?
 
-      disable_statement_timeout do
+      if OnlineMigrations.config.statement_timeout
         # "VALIDATE CONSTRAINT" requires a "SHARE UPDATE EXCLUSIVE" lock.
         # It only conflicts with other validations, creating/removing indexes,
         # and some other "ALTER TABLE"s.
         super
+      else
+        OnlineMigrations.deprecator.warn(<<~MSG)
+          Running `validate_foreign_key` without a statement timeout is deprecated.
+          Configure an explicit statement timeout in the initializer file via `config.statement_timeout`
+          or the default database statement timeout will be used.
+          Example, `config.statement_timeout = 1.hour`.
+        MSG
+
+        disable_statement_timeout do
+          super
+        end
       end
     end
 
@@ -811,11 +837,22 @@ module OnlineMigrations
       # Skip costly operation if already validated.
       return if check_constraint.validated?
 
-      disable_statement_timeout do
+      if OnlineMigrations.config.statement_timeout
         # "VALIDATE CONSTRAINT" requires a "SHARE UPDATE EXCLUSIVE" lock.
         # It only conflicts with other validations, creating/removing indexes,
         # and some other "ALTER TABLE"s.
         super
+      else
+        OnlineMigrations.deprecator.warn(<<~MSG)
+          Running `validate_check_constraint` without a statement timeout is deprecated.
+          Configure an explicit statement timeout in the initializer file via `config.statement_timeout`
+          or the default database statement timeout will be used.
+          Example, `config.statement_timeout = 1.hour`.
+        MSG
+
+        disable_statement_timeout do
+          super
+        end
       end
     end
 
@@ -855,28 +892,26 @@ module OnlineMigrations
       end
     end
 
-    # Disables statement timeout while executing &block
-    #
-    # Long-running migrations may take more than the timeout allowed by the database.
-    # Disable the session's statement timeout to ensure migrations don't get killed prematurely.
-    #
-    # Statement timeouts are already disabled in `add_index`, `remove_index`,
-    # `validate_foreign_key`, and `validate_check_constraint` helpers.
-    #
-    # @return [void]
-    #
-    # @example
-    #   disable_statement_timeout do
-    #     add_index(:users, :email, unique: true, algorithm: :concurrently)
-    #   end
-    #
+    # @private
     def disable_statement_timeout
-      prev_value = select_value("SHOW statement_timeout")
-      execute("SET statement_timeout TO 0")
+      OnlineMigrations.deprecator.warn(<<~MSG)
+        `disable_statement_timeout` is deprecated and will be removed. Configure an explicit
+        statement timeout in the initializer file via `config.statement_timeout` or the default
+        database statement timeout will be used. Example, `config.statement_timeout = 1.hour`.
+      MSG
 
+      prev_value = select_value("SHOW statement_timeout")
+      __set_statement_timeout(0)
       yield
     ensure
-      execute("SET statement_timeout TO #{quote(prev_value)}")
+      __set_statement_timeout(prev_value)
+    end
+
+    # @private
+    def __set_statement_timeout(timeout)
+      # use ceil to prevent no timeout for values under 1 ms
+      timeout = (timeout.to_f * 1000).ceil if !timeout.is_a?(String)
+      execute("SET statement_timeout TO #{quote(timeout)}")
     end
 
     # @private
@@ -905,6 +940,17 @@ module OnlineMigrations
         end
       end
 
+      # Will not be needed for Active Record >= 7.1
+      def __index_defined_for?(index, columns = nil, name: nil, unique: nil, valid: nil, include: nil, nulls_not_distinct: nil, **options)
+        columns = options[:column] if columns.blank?
+        (columns.nil? || Array(index.columns) == Array(columns).map(&:to_s)) &&
+          (name.nil? || index.name == name.to_s) &&
+          (unique.nil? || index.unique == unique) &&
+          (valid.nil? || index.valid == valid) &&
+          (include.nil? || Array(index.include) == Array(include).map(&:to_s)) &&
+          (nulls_not_distinct.nil? || index.nulls_not_distinct == nulls_not_distinct)
+      end
+
       def __not_null_constraint_exists?(table_name, column_name, name: nil)
         name ||= __not_null_constraint_name(table_name, column_name)
         __check_constraint_exists?(table_name, name: name)

data/lib/online_migrations/utils.rb

@@ -10,8 +10,17 @@ module OnlineMigrations
         ActiveRecord.version.to_s.to_f
       end
 
+      def env
+        if defined?(Rails.env)
+          Rails.env
+        else
+          # default to production for safety
+          ENV["RACK_ENV"] || "production"
+        end
+      end
+
       def developer_env?
-        defined?(Rails.env) && (Rails.env.development? || Rails.env.test?)
+        env == "development" || env == "test"
       end
 
       def say(message)
@@ -137,6 +146,11 @@ module OnlineMigrations
           return pool_manager.shard_names.uniq if pool_manager
         end
       end
+
+      def multiple_databases?
+        db_config = ActiveRecord::Base.configurations.configs_for(env_name: env)
+        db_config.reject(&:replica?).size > 1
+      end
     end
   end
 end

data/lib/online_migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module OnlineMigrations
-  VERSION = "0.11.1"
+  VERSION = "0.13.0"
 end

data/lib/online_migrations.rb

@@ -1,7 +1,19 @@
 # frozen_string_literal: true
 
 require "active_record"
+
 require "online_migrations/version"
+require "online_migrations/utils"
+require "online_migrations/change_column_type_helpers"
+require "online_migrations/background_migrations/migration_helpers"
+require "online_migrations/schema_statements"
+require "online_migrations/migration"
+require "online_migrations/migrator"
+require "online_migrations/schema_dumper"
+require "online_migrations/database_tasks"
+require "online_migrations/command_recorder"
+require "online_migrations/error_messages"
+require "online_migrations/config"
 
 module OnlineMigrations
   class Error < StandardError; end
@@ -9,15 +21,8 @@ module OnlineMigrations
 
   extend ActiveSupport::Autoload
 
-  autoload :Utils
-  autoload :ErrorMessages
-  autoload :Config
   autoload :BatchIterator
   autoload :VerboseSqlLogs
-  autoload :Migration
-  autoload :Migrator
-  autoload :SchemaDumper
-  autoload :DatabaseTasks
   autoload :ForeignKeysCollector
   autoload :IndexDefinition
   autoload :IndexesCollector
@@ -36,10 +41,7 @@ module OnlineMigrations
     autoload :NullLockRetrier
   end
 
-  autoload :CommandRecorder
   autoload :CopyTrigger
-  autoload :ChangeColumnTypeHelpers
-  autoload :SchemaStatements
 
   module BackgroundMigrations
     extend ActiveSupport::Autoload
@@ -59,7 +61,6 @@ module OnlineMigrations
     autoload :Migration
     autoload :MigrationJobRunner
     autoload :MigrationRunner
-    autoload :MigrationHelpers
     autoload :Scheduler
   end
 
@@ -80,6 +81,15 @@ module OnlineMigrations
       BackgroundMigrations::Scheduler.run
     end
 
+    def deprecator
+      @deprecator ||=
+        if Utils.ar_version >= 7.1
+          ActiveSupport::Deprecation.new(nil, "online_migrations")
+        else
+          ActiveSupport::Deprecation
+        end
+    end
+
     # @private
     def load
       require "active_record/connection_adapters/postgresql_adapter"

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: online_migrations
 version: !ruby/object:Gem::Version
-  version: 0.11.1
+  version: 0.13.0
 platform: ruby
 authors:
 - fatkodima
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-01-11 00:00:00.000000000 Z
+date: 2024-01-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord