online_migrations 0.14.1 → 0.15.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 330d5ed7f11aae3b85e7fb5aab5a1ea66d809da704cf193420b6d4206f72d82b
-  data.tar.gz: 36d47a6f2a1cae3de33331a4ec9e6b21fc7b3b8b09a7d5e9c34d1be6c92e5d35
+  metadata.gz: 2b1b826df875fae97622c702205853da01d67153172ef73fbb0edf28a2f7ff60
+  data.tar.gz: 18ae161255543e7ab7c266efc94f1ca166a344aa18b92dd4ab2f37209fb42547
 SHA512:
-  metadata.gz: e07937136867b4b6bc2d0dfb2e5ccb8471faf6ce0339a247bc0c63d591cc163e084da731e86fc01ba839c7744a51e9c63e99ce632452e471f9eed98a95b9dd02
-  data.tar.gz: f1a33636d5fbd01f25a0cee32af0d74e377c804eaef0feee2c778c5318a93bbe8942dc03e8ea2bad0a610d0046a8d4e42b505d0991c7d215cc9842a24d5e5065
+  metadata.gz: 705e4ce816bd8b4fcbcb78871589274bdf37acca01fbdf031883e87403a403782d28990ebed10acdf57adcbf3fb4e30e1dbf187123029bf6b1b7d31f26ccd3ca
+  data.tar.gz: 60ea354440b96645c5c03345d60793a66ba61d8aa1b946737098bc1449ba67e0b24caeb4581c525b3b014c5dd44b0ac28e0c539c8e3454ad67a3a1610cffdd79

data/CHANGELOG.md

@@ -1,5 +1,12 @@
 ## master (unreleased)
 
+## 0.15.0 (2024-03-19)
+
+- Reraise errors when running background migrations inline
+- Add `remove_background_migration` migration helper
+- Allow adding bigint foreign keys referencing integer primary keys
+- Fix `add_reference_concurrently` to check for mismatched key types
+
 ## 0.14.1 (2024-02-21)
 
 - Fix `MigrationRunner` to consider `run_background_migrations_inline` proc

data/README.md

@@ -191,7 +191,7 @@ end
 
    ```ruby
    class User < ApplicationRecord
-     self.ignored_columns = ["name"]
+     self.ignored_columns += ["name"]
    end
    ```
 
@@ -488,7 +488,7 @@ It will use a combination of a VIEW and column aliasing to work with both column
 
    ```ruby
    class User < ApplicationRecord
-     self.ignored_columns = ["name"]
+     self.ignored_columns += ["name"]
    end
    ```
 
@@ -1172,7 +1172,7 @@ A safer approach is to:
 
    ```ruby
    class User < ApplicationRecord
-     self.ignored_columns = ["type"]
+     self.ignored_columns += ["type"]
    end
    ```
 

data/docs/background_migrations.md

@@ -30,7 +30,8 @@ A generator is provided to create background migrations. Generate a new backgrou
 $ bin/rails generate online_migrations:background_migration backfill_project_issues_count
 ```
 
-This creates the background migration file `lib/online_migrations/background_migrations/backfill_project_issues_count.rb`.
+This creates the background migration file `lib/online_migrations/background_migrations/backfill_project_issues_count.rb`
+and the regular migration file `db/migrate/xxxxxxxxxxxxxx_enqueue_backfill_project_issues_count.rb` where we enqueue it.
 
 The generated class is a subclass of `OnlineMigrations::BackgroundMigration` that implements:
 
@@ -76,12 +77,16 @@ end
 You can enqueue your background migration to be run by the scheduler via:
 
 ```ruby
-# db/migrate/xxxxxxxxxxxxxx_backfill_project_issues_count.rb
-# ...
-def up
-  enqueue_background_migration("BackfillProjectIssuesCount")
+# db/migrate/xxxxxxxxxxxxxx_enqueue_backfill_project_issues_count.rb
+class EnqueueBackfillProjectIssuesCount < ActiveRecord::Migration[7.1]
+  def up
+    enqueue_background_migration("BackfillProjectIssuesCount")
+  end
+
+  def down
+    remove_background_migration("BackfillProjectIssuesCount")
+  end
 end
-# ...
 ```
 
 `enqueue_background_migration` accepts additional configuration options which controls how the background migration is run. Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_migrations/migration_helpers.rb) for the list of all available configuration options.
@@ -106,7 +111,17 @@ end
 And pass them when enqueuing:
 
 ```ruby
-enqueue_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
+def up
+  enqueue_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
+end
+```
+
+Make sure to also pass the arguments inside the `down` method of the migration:
+
+```ruby
+def down
+  remove_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
+end
 ```
 
 ## Considerations when writing Background Migrations

data/docs/configuring.md

@@ -101,13 +101,15 @@ config.lock_retrier = OnlineMigrations::ExponentialLockRetrier.new(
 )
 ```
 
-When statement within transaction fails - the whole transaction is retried.
+When a statement within transaction fails - the whole transaction is retried. If any statement fails when running outside a transaction (e.g. using `disable_ddl_transaction!`) then only that statement is retried.
 
-To permanently disable lock retries, you can set `lock_retrier` to `nil`.
+**Note**: Statements are retried by default, unless lock retries are disabled. It is possible to implement more sophisticated lock retriers. See [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/lock_retrier.rb) for the examples.
 
 To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable. This is useful when you are deploying a hotfix and do not want to wait too long while the lock retrier safely tries to acquire the lock, but try to acquire the lock immediately with the default configured lock timeout value.
 
-**Note**: Statements are retried by default, unless lock retries are disabled. It is possible to implement more sophisticated lock retriers. See [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/lock_retrier.rb) for the examples.
+To permanently disable lock retries, you can set `lock_retrier` to `nil`.
+
+Finally, if your lock retrier implementation does not have an explicit `lock_timeout` value configured, then the timeout behavior will fallback to the database configuration (`config/database.yml`) or the PostgreSQL server config value ([off by default](https://www.postgresql.org/docs/current/runtime-config-client.html#GUC-LOCK-TIMEOUT)). Take care configuring this value, as this fallback may result in your migrations running without a lock timeout!
 
 ## Existing Migrations
 

data/lib/generators/online_migrations/background_migration_generator.rb

@@ -1,12 +1,15 @@
 # frozen_string_literal: true
 
 require "rails/generators"
+require "rails/generators/active_record/migration"
 
 module OnlineMigrations
   # @private
   class BackgroundMigrationGenerator < Rails::Generators::NamedBase
+    include ActiveRecord::Generators::Migration
+
     source_root File.expand_path("templates", __dir__)
-    desc "This generator creates a background migration file."
+    desc "This generator creates a background migration related files."
 
     def create_background_migration_file
       migrations_module_file_path = migrations_module.underscore
@@ -20,6 +23,10 @@ module OnlineMigrations
       template("background_migration.rb", template_file)
     end
 
+    def create_migration_file
+      migration_template("migration.rb", File.join(db_migrate_path, "enqueue_#{file_name}.rb"))
+    end
+
     private
       def migrations_module
         config.migrations_module
@@ -28,5 +35,9 @@ module OnlineMigrations
       def config
         OnlineMigrations.config.background_migrations
       end
+
+      def migration_parent
+        "ActiveRecord::Migration[#{Utils.ar_version}]"
+      end
   end
 end

data/lib/generators/online_migrations/install_generator.rb

@@ -15,7 +15,7 @@ module OnlineMigrations
     end
 
     def create_migration_file
-      migration_template("migration.rb", File.join(db_migrate_path, "install_online_migrations.rb"))
+      migration_template("install_migration.rb", File.join(db_migrate_path, "install_online_migrations.rb"))
     end
 
     private

data/lib/generators/online_migrations/templates/install_migration.rb.tt

@@ -0,0 +1,51 @@
+class InstallOnlineMigrations < <%= migration_parent %>
+  def change
+    create_table :background_migrations do |t|
+      t.bigint :parent_id
+      t.string :migration_name, null: false
+      t.jsonb :arguments, default: [], null: false
+      t.string :batch_column_name, null: false
+      t.bigint :min_value, null: false
+      t.bigint :max_value, null: false
+      t.bigint :rows_count
+      t.integer :batch_size, null: false
+      t.integer :sub_batch_size, null: false
+      t.integer :batch_pause, null: false
+      t.integer :sub_batch_pause_ms, null: false
+      t.integer :batch_max_attempts, null: false
+      t.string :status, default: "enqueued", null: false
+      t.string :shard
+      t.boolean :composite, default: false, null: false
+      t.timestamps
+
+      t.foreign_key :background_migrations, column: :parent_id, on_delete: :cascade
+
+      t.index [:migration_name, :arguments, :shard],
+        unique: true, name: :index_background_migrations_on_unique_configuration
+    end
+
+    create_table :background_migration_jobs do |t|
+      t.bigint :migration_id, null: false
+      t.bigint :min_value, null: false
+      t.bigint :max_value, null: false
+      t.integer :batch_size, null: false
+      t.integer :sub_batch_size, null: false
+      t.integer :pause_ms, null: false
+      t.datetime :started_at
+      t.datetime :finished_at
+      t.string :status, default: "enqueued", null: false
+      t.integer :max_attempts, null: false
+      t.integer :attempts, default: 0, null: false
+      t.string :error_class
+      t.string :error_message
+      t.string :backtrace, array: true
+      t.timestamps
+
+      t.foreign_key :background_migrations, column: :migration_id, on_delete: :cascade
+
+      t.index [:migration_id, :max_value], name: :index_background_migration_jobs_on_max_value
+      t.index [:migration_id, :status, :updated_at], name: :index_background_migration_jobs_on_updated_at
+      t.index [:migration_id, :finished_at], name: :index_background_migration_jobs_on_finished_at
+    end
+  end
+end

data/lib/generators/online_migrations/templates/migration.rb.tt

@@ -1,51 +1,10 @@
-class InstallOnlineMigrations < <%= migration_parent %>
-  def change
-    create_table :background_migrations do |t|
-      t.bigint :parent_id
-      t.string :migration_name, null: false
-      t.jsonb :arguments, default: [], null: false
-      t.string :batch_column_name, null: false
-      t.bigint :min_value, null: false
-      t.bigint :max_value, null: false
-      t.bigint :rows_count
-      t.integer :batch_size, null: false
-      t.integer :sub_batch_size, null: false
-      t.integer :batch_pause, null: false
-      t.integer :sub_batch_pause_ms, null: false
-      t.integer :batch_max_attempts, null: false
-      t.string :status, default: "enqueued", null: false
-      t.string :shard
-      t.boolean :composite, default: false, null: false
-      t.timestamps
-
-      t.foreign_key :background_migrations, column: :parent_id, on_delete: :cascade
-
-      t.index [:migration_name, :arguments, :shard],
-        unique: true, name: :index_background_migrations_on_unique_configuration
-    end
-
-    create_table :background_migration_jobs do |t|
-      t.bigint :migration_id, null: false
-      t.bigint :min_value, null: false
-      t.bigint :max_value, null: false
-      t.integer :batch_size, null: false
-      t.integer :sub_batch_size, null: false
-      t.integer :pause_ms, null: false
-      t.datetime :started_at
-      t.datetime :finished_at
-      t.string :status, default: "enqueued", null: false
-      t.integer :max_attempts, null: false
-      t.integer :attempts, default: 0, null: false
-      t.string :error_class
-      t.string :error_message
-      t.string :backtrace, array: true
-      t.timestamps
-
-      t.foreign_key :background_migrations, column: :migration_id, on_delete: :cascade
+class Enqueue<%= class_name %> < <%= migration_parent %>
+  def up
+    enqueue_background_migration("<%= class_name %>", ...args)
+  end
 
-      t.index [:migration_id, :max_value], name: :index_background_migration_jobs_on_max_value
-      t.index [:migration_id, :status, :updated_at], name: :index_background_migration_jobs_on_updated_at
-      t.index [:migration_id, :finished_at], name: :index_background_migration_jobs_on_finished_at
-    end
+  def down
+    # Make sure to pass the same arguments as in the "up" method, if any.
+    remove_background_migration("<%= class_name %>", ...args)
   end
 end

data/lib/online_migrations/background_migrations/background_migration_class_validator.rb

@@ -16,13 +16,6 @@ module OnlineMigrations
           return
         end
 
-        if relation.joins_values.present? && !record.batch_column_name.to_s.include?(".")
-          record.errors.add(
-            :batch_column_name,
-            "must be a fully-qualified column if you join a table"
-          )
-        end
-
         if relation.arel.orders.present? || relation.arel.taken.present?
           record.errors.add(
             :migration_name,

data/lib/online_migrations/background_migrations/delete_associated_records.rb

@@ -21,7 +21,7 @@ module OnlineMigrations
       end
 
       def process_batch(relation)
-        relation.delete_all(:delete_all)
+        relation.delete_all
       end
     end
   end

data/lib/online_migrations/background_migrations/migration_helpers.rb

@@ -372,8 +372,7 @@ module OnlineMigrations
       def enqueue_background_migration(migration_name, *arguments, **options)
         migration = create_background_migration(migration_name, *arguments, **options)
 
-        run_inline = OnlineMigrations.config.run_background_migrations_inline
-        if run_inline && run_inline.call
+        if Utils.run_background_migrations_inline?
           runner = MigrationRunner.new(migration)
           runner.run_all_migration_jobs
         end
@@ -381,11 +380,26 @@ module OnlineMigrations
         migration
       end
 
+      # Removes the background migration for the given class name and arguments, if exists.
+      #
+      # @param migration_name [String, Class] Background migration job class name
+      # @param arguments [Array] Extra arguments the migration was originally created with
+      #
+      # @example
+      #   remove_background_migration("BackfillProjectIssuesCount")
+      #
+      def remove_background_migration(migration_name, *arguments)
+        migration_name = migration_name.name if migration_name.is_a?(Class)
+        Migration.for_configuration(migration_name, arguments).delete_all
+      end
+
       # @private
       def create_background_migration(migration_name, *arguments, **options)
         options.assert_valid_keys(:batch_column_name, :min_value, :max_value, :batch_size, :sub_batch_size,
             :batch_pause, :sub_batch_pause_ms, :batch_max_attempts)
 
+        migration_name = migration_name.name if migration_name.is_a?(Class)
+
         migration = Migration.new(
           migration_name: migration_name,
           arguments: arguments,

data/lib/online_migrations/background_migrations/migration_job.rb

@@ -44,7 +44,7 @@ module OnlineMigrations
         enum status: STATUSES.index_with(&:to_s)
       end
 
-      delegate :migration_class, :migration_object, :migration_relation, :batch_column_name,
+      delegate :migration_name, :migration_class, :migration_object, :migration_relation, :batch_column_name,
         :arguments, :batch_pause, to: :migration
 
       belongs_to :migration
@@ -73,7 +73,10 @@ module OnlineMigrations
           status: self.class.statuses[:enqueued],
           attempts: 0,
           started_at: nil,
-          finished_at: nil
+          finished_at: nil,
+          error_class: nil,
+          error_message: nil,
+          backtrace: nil
         )
       end
 

data/lib/online_migrations/background_migrations/migration_job_runner.rb

@@ -46,6 +46,7 @@ module OnlineMigrations
         )
 
         ::OnlineMigrations.config.background_migrations.error_handler.call(e, migration_job)
+        raise if Utils.run_background_migrations_inline?
       end
 
       private

data/lib/online_migrations/command_checker.rb

@@ -527,6 +527,30 @@ module OnlineMigrations
       end
       alias add_belongs_to add_reference
 
+      def add_reference_concurrently(table_name, ref_name, **options)
+        # Always added by default in 5.0+
+        index = options.fetch(:index, true)
+
+        if index.is_a?(Hash) && index[:using].to_s == "hash" && postgresql_version < Gem::Version.new("10")
+          raise_error :add_hash_index
+        end
+
+        foreign_key = options.fetch(:foreign_key, false)
+
+        if foreign_key
+          foreign_table_name = Utils.foreign_table_name(ref_name, options)
+          @foreign_key_tables << foreign_table_name.to_s
+        end
+
+        if !options[:polymorphic]
+          type = (options[:type] || :bigint).to_sym
+          column_name = "#{ref_name}_id"
+
+          foreign_key_options = foreign_key.is_a?(Hash) ? foreign_key : {}
+          check_mismatched_foreign_key_type(table_name, column_name, type, **foreign_key_options)
+        end
+      end
+
       def add_index(table_name, column_name, **options)
         if options[:using].to_s == "hash" && postgresql_version < Gem::Version.new("10")
           raise_error :add_hash_index
@@ -830,8 +854,14 @@ module OnlineMigrations
           if connection.table_exists?(foreign_table_name)
             primary_key = options[:primary_key] || connection.primary_key(foreign_table_name)
             primary_key_column = column_for(foreign_table_name, primary_key)
+            return if primary_key_column.nil?
+
+            primary_key_type = primary_key_column.sql_type.to_sym
+            # Having bigint foreign keys is safe and people should
+            # detect integer primary keys via some other tools.
+            return if type == :bigint && primary_key_type == :integer
 
-            if primary_key_column && type != primary_key_column.sql_type.to_sym
+            if type != primary_key_type
               raise_error :mismatched_foreign_key_type,
                 table_name: table_name, column_name: column_name
             end

data/lib/online_migrations/command_recorder.rb

@@ -26,6 +26,7 @@ module OnlineMigrations
       :add_reference_concurrently,
       :change_column_type_in_background,
       :enqueue_background_migration,
+      :remove_background_migration,
 
       # column type change helpers
       :initialize_column_type_change,
@@ -77,6 +78,10 @@ module OnlineMigrations
 
       include StraightReversions
 
+      def invert_add_reference_concurrently(args)
+        [:remove_reference, args]
+      end
+
       def invert_swap_column_names(args)
         table_name, column1, column2 = args
         [:swap_column_names, [table_name, column2, column1]]

data/lib/online_migrations/error_messages.rb

@@ -110,7 +110,7 @@ A safer approach is to:
 1. ignore the column:
 
   class <%= model %> < ApplicationRecord
-    self.ignored_columns = [\"<%= column_name %>\"]
+    self.ignored_columns += [\"<%= column_name %>\"]
   end
 
 2. deploy
@@ -151,7 +151,7 @@ It will use a combination of a VIEW and column aliasing to work with both column
 <% if enumerate_columns_in_select_statements %>
 5. Ignore old column
 
-  self.ignored_columns = [:<%= column_name %>]
+  self.ignored_columns += [:<%= column_name %>]
 
 6. Deploy
 7. Remove the column rename config from step 1
@@ -298,7 +298,7 @@ A safer approach is to:
 1. Ignore the column:
 
   class <%= model %> < ApplicationRecord
-    self.ignored_columns = <%= columns %>
+    self.ignored_columns += <%= columns %>
   end
 
 2. Deploy

data/lib/online_migrations/lock_retrier.rb

@@ -229,11 +229,9 @@ module OnlineMigrations
     end
 
     def lock_timeout(*)
-      0
     end
 
     def delay(*)
-      0
     end
 
     def with_lock_retries

data/lib/online_migrations/utils.rb

@@ -151,6 +151,11 @@ module OnlineMigrations
         db_config = ActiveRecord::Base.configurations.configs_for(env_name: env)
         db_config.reject(&:replica?).size > 1
       end
+
+      def run_background_migrations_inline?
+        run_inline = OnlineMigrations.config.run_background_migrations_inline
+        run_inline && run_inline.call
+      end
     end
   end
 end

data/lib/online_migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module OnlineMigrations
-  VERSION = "0.14.1"
+  VERSION = "0.15.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: online_migrations
 version: !ruby/object:Gem::Version
-  version: 0.14.1
+  version: 0.15.0
 platform: ruby
 authors:
 - fatkodima
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-02-21 00:00:00.000000000 Z
+date: 2024-03-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -41,6 +41,7 @@ files:
 - lib/generators/online_migrations/templates/add_sharding_to_online_migrations.rb.tt
 - lib/generators/online_migrations/templates/background_migration.rb.tt
 - lib/generators/online_migrations/templates/initializer.rb.tt
+- lib/generators/online_migrations/templates/install_migration.rb.tt
 - lib/generators/online_migrations/templates/migration.rb.tt
 - lib/generators/online_migrations/upgrade_generator.rb
 - lib/online_migrations.rb