online_migrations 0.5.3 → 0.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0ac8a2eb2657fcc3536a9b56d16298f90f8e1a5e626ba52a6e9703b5bcaa7df0
-  data.tar.gz: c5357f94854fb43d1b8404ed9dcfafc2178430aeb9c6d69555f50629a94da3bd
+  metadata.gz: 625229a6ef3cc2b9a6a6201ae2033b4560ca395482d797115563abd1dace6716
+  data.tar.gz: bc24c264f8a6be48e9232959a6e654e5a814c9872317d88e788eca8260dd6a7e
 SHA512:
-  metadata.gz: 4ccbc274c76ab01f8ece559b3fee27c1c5c0c495225b393c3a4afb7941f0b61d3aab9ef2885195be78ed00f29bd5e6b18e3fe0cb044b745cd791aacf34283492
-  data.tar.gz: 04fb250e30e7620ac1bd06fddf24a5fdfe616f285422e47e19423e23be0c8117e57d45e45029a2e1212cc5e6ad9baf3605fac8b038152fdc73b2d9a818ba211c
+  metadata.gz: 8054e3f5fec6fbc066cc3d6184a3002a2e77b21dee5ee7adca1025fa907c3e2263fae206b25150b662e37cd1dfac46cd66a54c82d725848020c30ed0cecef5c9
+  data.tar.gz: 2ad342552fbf328c341875192aba31c441b2bd5a72d3fac9edfef9caa9fa2eab003c9ae7d441d4778b3425f82aed140290de82df3291c030e0b1e7867871ae47

data/BACKGROUND_MIGRATIONS.md

@@ -120,7 +120,7 @@ enqueue_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
 * `CopyColumn` - copies data from one column(s) to other(s) (enqueue using `copy_column_in_background`)
 * `DeleteAssociatedRecords` - deletes records associated with a parent object (enqueue using `delete_associated_records_in_background`)
 * `DeleteOrphanedRecords` - deletes records with one or more missing relations (enqueue using `delete_orphaned_records_in_background`)
-* `PerformActionOnRelation` - performs specific action on a relation or indvidual records (enqueue using `perform_action_on_relation_in_background`)
+* `PerformActionOnRelation` - performs specific action on a relation or individual records (enqueue using `perform_action_on_relation_in_background`)
 * `ResetCounters` - resets one or more counter caches to their correct value (enqueue using `reset_counters_in_background`)
 
 ## Testing
@@ -240,7 +240,7 @@ Specify the throttle condition as a block:
 ```ruby
 # config/initializers/online_migrations.rb
 
-OnlineMigrations.config.backround_migrations.throttler = -> { DatabaseStatus.unhealthy? }
+OnlineMigrations.config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
 ```
 
 Note that it's up to you to define a throttling condition that makes sense for your app. For example, you can check various PostgreSQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
@@ -254,7 +254,7 @@ If you want to integrate with an exception monitoring service (e.g. Bugsnag), yo
 ```ruby
 # config/initializers/online_migrations.rb
 
-OnlineMigrations.config.backround_migrations.error_handler = ->(error, errored_job) do
+OnlineMigrations.config.background_migrations.error_handler = ->(error, errored_job) do
   Bugsnag.notify(error) do |notification|
     notification.add_metadata(:background_migration, { name: errored_job.migration_name })
   end

data/CHANGELOG.md

@@ -1,5 +1,16 @@
 ## master (unreleased)
 
+## 0.6.0 (2023-02-04)
+
+- Ignore internal Active Record migrations compatibility related options when suggesting a safe column type change
+- Added check for `add_exclusion_constraint`
+- Fix preserving old column options (`:comment` and `:collation`) when changing column type
+- Set `NOT NULL` during new column creation when changing column type for PostgreSQL >= 11
+
+## 0.5.4 (2023-01-03)
+
+- Support ruby 3.2.0
+
 ## 0.5.3 (2022-11-10)
 
 - Fix removing index by name

data/README.md

@@ -20,6 +20,8 @@ See [comparison to `strong_migrations`](#comparison-to-strong_migrations)
 - Rails 4.2+
 - PostgreSQL 9.6+
 
+**Note**: Since some migration helpers use database `VIEW`s to implement their logic, it is recommended to use `structure.sql` schema format, or otherwise add some gem (like [scenic](https://github.com/scenic-views/scenic)) to be able to dump them into the `schema.rb`.
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -69,7 +71,7 @@ class AddAdminToUsers < ActiveRecord::Migration[7.0]
   # Do not wrap the migration in a transaction so that locks are held for a shorter time.
   disable_ddl_transaction!
 
-  def change
+  def up
     # Lower PostgreSQL's lock timeout to avoid statement queueing.
     execute "SET lock_timeout TO '5s'" # The lock_timeout duration is customizable.
 
@@ -88,6 +90,10 @@ class AddAdminToUsers < ActiveRecord::Migration[7.0]
     execute "SET statement_timeout TO '5s'"
     change_column_null :users, :admin, false
   end
+  
+  def down
+    remove_column :users, :admin
+  end
 end
 ```
 
@@ -136,6 +142,7 @@ Potentially dangerous operations:
 - [replacing an index](#replacing-an-index)
 - [adding a reference](#adding-a-reference)
 - [adding a foreign key](#adding-a-foreign-key)
+- [adding an exclusion constraint](#adding-an-exclusion-constraint)
 - [adding a json column](#adding-a-json-column)
 - [using primary key with short integer type](#using-primary-key-with-short-integer-type)
 - [hash indexes](#hash-indexes)
@@ -317,6 +324,9 @@ A safer approach can be accomplished in several steps:
   end
   ```
 
+**Note**: `initialize_column_type_change` accepts additional options (like `:limit`, `:default` etc)
+which will be passed to `add_column` when creating a new column, so you can override previous values.
+
 2. Backfill data from the old column to the new column:
 
   ```ruby
@@ -391,7 +401,7 @@ For the previous example, to rename `name` column to `first_name` of the `users`
 ```sql
 BEGIN;
 ALTER TABLE users RENAME TO users_column_rename;
-CREATE VIEW users AS SELECT *, first_name AS name FROM users;
+CREATE VIEW users AS SELECT *, first_name AS name FROM users_column_rename;
 COMMIT;
 ```
 
@@ -410,9 +420,21 @@ OnlineMigrations.config.column_renames = {
   }
 }
 ```
+NOTE: You also need to temporarily enable partial writes (is disabled by default in Active Record >= 7)
+until the process of column rename is fully done.
+```ruby
+# config/application.rb
+# For Active Record >= 7
+config.active_record.partial_inserts = true
+
+# Or for Active Record < 7
+config.active_record.partial_writes = true
+```
 
 2. Deploy
-3. Create a VIEW with aliased column:
+3. Tell the database that you are going to rename a column. This will not actually rename any columns,
+nor any data/indexes/foreign keys copying will be made, so will be instantaneous.
+It will use a combination of a VIEW and column aliasing to work with both column names simultaneously
 
 ```ruby
 class InitializeRenameUsersNameToFirstName < ActiveRecord::Migration[7.0]
@@ -425,7 +447,7 @@ end
 4. Replace usages of the old column with a new column in the codebase
 5. Deploy
 6. Remove the column rename config from step 1
-7. Remove the VIEW created in step 3:
+7. Remove the VIEW created in step 3 and finally rename the column:
 
 ```ruby
 class FinalizeRenameUsersNameToFirstName < ActiveRecord::Migration[7.0]
@@ -794,6 +816,24 @@ end
 
 **Note**: If you forget `disable_ddl_transaction!`, the migration will fail.
 
+### Adding an exclusion constraint
+
+:x: **Bad**
+
+Adding an exclusion constraint blocks reads and writes while every row is checked.
+
+```ruby
+class AddExclusionContraint < ActiveRecord::Migration[7.1]
+  def change
+    add_exclusion_constraint :users, "number WITH =", using: :gist
+  end
+end
+```
+
+:white_check_mark: **Good**
+
+[Let us know](https://github.com/fatkodima/online_migrations/issues/new) if you have a safe way to do this (exclusion constraints cannot be marked `NOT VALID`).
+
 ### Adding a json column
 
 :x: **Bad**

data/lib/generators/online_migrations/templates/initializer.rb.tt

@@ -38,7 +38,7 @@ OnlineMigrations.configure do |config|
   # migration failure in production. This is also useful in development to get
   # a better grasp of what is going on for high-level statements like add_column_with_default.
   #
-  # Note: It can be overriden by `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
+  # Note: It can be overridden by `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
   config.verbose_sql_logs = defined?(Rails) && Rails.env.production?
 
   # Lock retries.
@@ -67,25 +67,25 @@ OnlineMigrations.configure do |config|
 
   # ==> Background migrations configuration
   # The number of rows to process in a single background migration run.
-  # config.backround_migrations.batch_size = 20_000
+  # config.background_migrations.batch_size = 20_000
 
   # The smaller batches size that the batches will be divided into.
-  # config.backround_migrations.sub_batch_size = 1000
+  # config.background_migrations.sub_batch_size = 1000
 
   # The pause interval between each background migration job's execution (in seconds).
-  # config.backround_migrations.batch_pause = 0.seconds
+  # config.background_migrations.batch_pause = 0.seconds
 
   # The number of milliseconds to sleep between each sub_batch execution.
-  # config.backround_migrations.sub_batch_pause_ms = 100
+  # config.background_migrations.sub_batch_pause_ms = 100
 
   # Maximum number of batch run attempts.
   # When attempts are exhausted, the individual batch is marked as failed.
-  # config.backround_migrations.batch_max_attempts = 5
+  # config.background_migrations.batch_max_attempts = 5
 
   # Configure custom throttler for background migrations.
   # It will be called before each batch run.
   # If throttled, the current run will be retried next time.
-  # config.backround_migrations.throttler = -> { DatabaseStatus.unhealthy? }
+  # config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
 
   # The number of seconds that must pass before the running job is considered stuck.
   # config.background_migrations.stuck_jobs_timeout = 1.hour
@@ -95,7 +95,7 @@ OnlineMigrations.configure do |config|
   config.background_migrations.backtrace_cleaner = Rails.backtrace_cleaner
 
   # The callback to perform when an error occurs in the migration job.
-  # config.backround_migrations.error_handler = ->(error, errored_job) do
+  # config.background_migrations.error_handler = ->(error, errored_job) do
   #   Bugsnag.notify(error) do |notification|
   #     notification.add_metadata(:background_migration, { name: errored_job.migration_name })
   #   end

data/lib/online_migrations/background_migrations/config.rb

@@ -43,7 +43,7 @@ module OnlineMigrations
       # @return [Proc]
       #
       # @example
-      #   OnlineMigrations.config.backround_migrations.throttler = -> { DatabaseStatus.unhealthy? }
+      #   OnlineMigrations.config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
       #
       attr_reader :throttler
 
@@ -64,7 +64,7 @@ module OnlineMigrations
       # The callback to perform when an error occurs in the migration job.
       #
       # @example
-      #   OnlineMigrations.config.backround_migrations.error_handler = ->(error, errored_job) do
+      #   OnlineMigrations.config.background_migrations.error_handler = ->(error, errored_job) do
       #     Bugsnag.notify(error) do |notification|
       #       notification.add_metadata(:background_migration, { name: errored_job.migration_name })
       #     end

data/lib/online_migrations/change_column_type_helpers.rb

@@ -107,19 +107,26 @@ module OnlineMigrations
       transaction do
         columns_and_types.each do |(column_name, new_type)|
           old_col = __column_for(table_name, column_name)
+          old_col_options = __options_from_column(old_col, [:collation, :comment])
           column_options = options[column_name] || {}
           tmp_column_name = conversions[column_name]
 
-          if raw_connection.server_version >= 11_00_00 &&
-             primary_key(table_name) == column_name.to_s && old_col.type == :integer
-            # If the column to be converted is a Primary Key, set it to
-            # `NOT NULL DEFAULT 0` and we'll copy the correct values when backfilling.
-            # That way, we skip the expensive validation step required to add
-            #  a `NOT NULL` constraint at the end of the process.
-            add_column(table_name, tmp_column_name, new_type,
-              **column_options.merge(default: old_col.default || 0, null: false))
+          if raw_connection.server_version >= 11_00_00
+            if primary_key(table_name) == column_name.to_s && old_col.type == :integer
+              # If the column to be converted is a Primary Key, set it to
+              # `NOT NULL DEFAULT 0` and we'll copy the correct values when backfilling.
+              # That way, we skip the expensive validation step required to add
+              #  a `NOT NULL` constraint at the end of the process.
+              add_column(table_name, tmp_column_name, new_type,
+                **old_col_options.merge(column_options).merge(default: old_col.default || 0, null: false))
+            else
+              unless old_col.default.nil?
+                old_col_options = old_col_options.merge(default: old_col.default, null: old_col.null)
+              end
+              add_column(table_name, tmp_column_name, new_type, **old_col_options.merge(column_options))
+            end
           else
-            add_column(table_name, tmp_column_name, new_type, **column_options)
+            add_column(table_name, tmp_column_name, new_type, **old_col_options.merge(column_options))
             change_column_default(table_name, tmp_column_name, old_col.default) unless old_col.default.nil?
           end
         end
@@ -377,6 +384,17 @@ module OnlineMigrations
         "#{column_name}_for_type_change"
       end
 
+      def __options_from_column(column, options)
+        result = {}
+        options.each do |option|
+          if column.respond_to?(option)
+            value = column.public_send(option)
+            result[option] = value unless value.nil?
+          end
+        end
+        result
+      end
+
       def __copy_triggers_name(table_name, from_column, to_column)
         CopyTrigger.on_table(table_name, connection: self).name(from_column, to_column)
       end

data/lib/online_migrations/command_checker.rb

@@ -42,6 +42,7 @@ module OnlineMigrations
 
       true
     end
+    ruby2_keywords(:check) if respond_to?(:ruby2_keywords, true)
 
     private
       def check_database_version
@@ -228,6 +229,10 @@ module OnlineMigrations
 
         type = type.to_sym
 
+        # Ignore internal Active Record migrations compatibility related
+        # options, like `_uses_legacy_table_name` etc. They all are starting with "_".
+        options = options.reject { |key, _| key.to_s.start_with?("_") }
+
         existing_column = column_for(table_name, column_name)
         if existing_column
           existing_type = existing_column.type.to_sym
@@ -271,7 +276,7 @@ module OnlineMigrations
                 !options[:limit] || (existing_column.limit && options[:limit] >= existing_column.limit)
               end
             when :numeric, :decimal
-              # numeric and decimal are equivalent and can be used interchangably
+              # numeric and decimal are equivalent and can be used interchangeably
               [:numeric, :decimal].include?(existing_type) &&
               (
                 (
@@ -517,6 +522,12 @@ module OnlineMigrations
         end
       end
 
+      def add_exclusion_constraint(table_name, _expression, **_options)
+        unless new_or_small_table?(table_name)
+          raise_error :add_exclusion_constraint
+        end
+      end
+
       def add_check_constraint(table_name, expression, **options)
         if !new_or_small_table?(table_name) && options[:validate] != false
           name = options[:name] || check_constraint_name(table_name, expression)

data/lib/online_migrations/config.rb

@@ -153,7 +153,7 @@ module OnlineMigrations
     # This feature is enabled by default in a production Rails environment.
     # @return [Boolean]
     #
-    # @note: It can be overriden by `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
+    # @note: It can be overridden by `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
     #
     attr_accessor :verbose_sql_logs
 

data/lib/online_migrations/error_messages.rb

@@ -131,7 +131,8 @@ migration_helpers provides a safer approach to do this:
     }
   }
 <% unless partial_writes %>
-  NOTE: You also need to temporarily enable partial writes until the process of column rename is fully done.
+  NOTE: You also need to temporarily enable partial writes (is disabled by default in Active Record >= 7)
+  until the process of column rename is fully done.
   # config/application.rb
   config.active_record.<%= partial_writes_setting %> = true
 <% end %>
@@ -150,7 +151,7 @@ It will use a combination of a VIEW and column aliasing to work with both column
 4. Replace usages of the old column with a new column in the codebase
 5. Deploy
 6. Remove the column rename config from step 1
-7. Remove the VIEW created in step 3:
+7. Remove the VIEW created in step 3 and finally rename the column:
 
   class Finalize<%= migration_name %> < <%= migration_parent %>
     def change
@@ -175,6 +176,9 @@ A safer approach can be accomplished in several steps:
     end
   end
 
+**Note**: `initialize_column_type_change` accepts additional options (like `:limit`, `:default` etc)
+which will be passed to `add_column` when creating a new column, so you can override previous values.
+
 2. Backfill data from the old column to the new column:
 
   class Backfill<%= migration_name %> < <%= migration_parent %>
@@ -380,6 +384,9 @@ end",
 "Validating a foreign key while holding heavy locks on tables is dangerous.
 Use disable_ddl_transaction! or a separate migration.",
 
+      add_exclusion_constraint:
+"Adding an exclusion constraint blocks reads and writes while every row is checked.",
+
       add_check_constraint:
 "Adding a check constraint blocks reads and writes while every row is checked.
 A safer approach is to add the check constraint without validating existing rows,

data/lib/online_migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module OnlineMigrations
-  VERSION = "0.5.3"
+  VERSION = "0.6.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: online_migrations
 version: !ruby/object:Gem::Version
-  version: 0.5.3
+  version: 0.6.0
 platform: ruby
 authors:
 - fatkodima
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-11-10 00:00:00.000000000 Z
+date: 2023-02-04 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -101,7 +101,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.6
+rubygems_version: 3.4.3
 signing_key:
 specification_version: 4
 summary: Catch unsafe PostgreSQL migrations in development and run them easier in