safe-pg-migrations 3.1.4 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 06ca39e801ba6836f1059e9a3f66c9d54d1f1920a39f421904ae592944ae4281
-  data.tar.gz: b15dec6a1f06f7a6801ec32a497672059d7132792495a47745fb4f2916bf9a78
+  metadata.gz: 3edbea6934dae51bfef3563e0d24e04d3b589bf417ffd9635687265f1b973613
+  data.tar.gz: b4714af68927b76b3580b9d74b83d96f73bdaa8127395ec030379d29886d70b0
 SHA512:
-  metadata.gz: 330c11df6898b0a4a13114e75f1bcdec8ae9935c8ebbba427bf634a6d4ba896c6e56994edeaa761ede31c024b60bdca82282181057585469d358a2ae7dc6aec2
-  data.tar.gz: 47623a5dd9404a438af11dd7334b7989da829b775128f4076d3ea7e6a236e2ce2a068d38603aae9e4501b89e8988a4e6d14da54501afaa1c2f927aca6f087992
+  metadata.gz: 774fac16319205c8bafb5e1f346aa23bfc6de00df261bb307ec0a095be56da56242c2103fb3e9dfc7e9f4d7ce0023817eb5f8680476d3e0dc61fd328ebb23a72
+  data.tar.gz: 7f6b0661848d67fb0193155b21211486413fa39d1769caa7524f8429bc8303f1b71faba7fd502f0a137352c09f61a4487644cb62bf710fcd903b4d281cde326d

data/README.md

@@ -116,11 +116,25 @@ PG will still needs to update every row of the table, and will most likely state
 <details>
 <summary>Safe add_column - adding a volatile default value</summary>
 
-**Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a volatile default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
+> **⚠️ ERROR**
+>
+> Using `default_value_backfill: :update_in_batches` with **volatile defaults** (like `NOW()`, `clock_timestamp()`, `random()`, `gen_random_uuid()`) is **not allowed** and will raise an error.
+>
+> Volatile defaults are non-deterministic functions that are evaluated per row, causing migrations to hang for a very long time on large tables. You must backfill them manually with proper monitoring and control.
+>
+> **Non-volatile defaults** (like fixed strings, numbers, booleans) are still safe to use with automatic backfill.
+
+**Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a **non-volatile** default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
 
 ```ruby
+# ✅ SAFE - non-volatile default with automatic backfill
 safety_assured do
-  add_column :users, :created_at, default: 'clock_timestamp()', default_value_backfill: :update_in_batches
+  add_column :users, :status, :string, default: 'active', default_value_backfill: :update_in_batches
+end
+
+# ❌ NOT ALLOWED - volatile default with automatic backfill (will raise an error)
+safety_assured do
+  add_column :users, :created_at, :datetime, default: -> { 'clock_timestamp()' }, default_value_backfill: :update_in_batches
 end
 ```
 
@@ -132,18 +146,44 @@ More specifically, it will:
 4. change the column to `null: false`, if defined in the parameters, following the algorithm we have defined below.
 
 ---
-**NOTE**
-
-Data backfill take time. If your table is big, your migrations will (safely) hangs for a while. You might want to backfill data manually instead, to do so you will need two migrations
-
-1. First migration :
-
-    a. adds the column without default and without null constraint;
-
-    b. add the default value.
-
-2. manual data backfill (rake task, manual operation, ...)
-3. Second migration which change the column to null false (with **Safe PG Migrations**, `change_column_null` is safe and can be used; see section below)
+**NOTE: Manual Backfill for Volatile Defaults**
+
+For **volatile defaults** (non-deterministic functions), you MUST backfill manually. Split the operation into multiple steps in this EXACT order:
+
+1. **ALTER COLUMN SET DEFAULT** (for new and updated rows)
+   ```ruby
+   change_column_default :users, :created_at, -> { 'NOW()' }
+   ```
+
+2. **ADD CONSTRAINT CHECK NOT NULL NOT VALID** (for new and updated rows, only if you need NOT NULL)
+   ```ruby
+   add_check_constraint :users, "created_at IS NOT NULL",
+                        name: "check_users_created_at_not_null",
+                        validate: false
+   ```
+
+3. **BACKFILL** the column using a job (chunk over the primary key)
+   ```ruby
+   # Your own script to backfill in batches
+   User.in_batches.update_all("created_at = NOW()")
+   ```
+
+4. **VALIDATE CONSTRAINT** (check whole table, only if you added the constraint)
+   ```ruby
+   validate_check_constraint :users, name: "check_users_created_at_not_null"
+   ```
+
+5. **ALTER COLUMN SET NOT NULL** (only if you need NOT NULL)
+   ```ruby
+   change_column_null :users, :created_at, false
+   ```
+
+6. **DROP CONSTRAINT** (only if you added the constraint)
+   ```ruby
+   remove_check_constraint :users, name: "check_users_created_at_not_null"
+   ```
+
+For **non-volatile defaults**, data backfill with `:update_in_batches` is safe but takes time. If your table is very large, you might want to backfill manually for better control.
 ---
 
 `default_value_backfill:` also accept the value `:auto` which is set by default. In this case, **Safe PG Migrations** will not backfill data and will let PostgreSQL handle it itself.
@@ -161,7 +201,7 @@ It is also possible to set a threshold for the table size, above which the migra
 
 Creating an index requires a `SHARE` lock on the target table which blocks all write on the table while the index is created (which can take some time on a large table). This is usually not practical in a live environment. Thus, **Safe PG Migrations** ensures indexes are created concurrently.
 
-As `CREATE INDEX CONCURRENTLY` and `DROP INDEX CONCURRENTLY` are non-blocking operations (ie: read/write operations on the table are still possible), **Safe PG Migrations** sets a lock timeout to 30 seconds for those 2 specific statements.
+As `CREATE INDEX CONCURRENTLY` and `DROP INDEX CONCURRENTLY` are non-blocking operations (ie: read/write operations on the table are still possible), **Safe PG Migrations** sets a statement and lock timeout to 0 seconds for those 2 specific statements.
 
 If you still get lock timeout while adding / removing indexes, it might be for one of those reasons:
 

data/lib/safe-pg-migrations/base.rb

@@ -7,6 +7,7 @@ require 'safe-pg-migrations/helpers/index_helper'
 require 'safe-pg-migrations/helpers/batch_over'
 require 'safe-pg-migrations/helpers/session_setting_management'
 require 'safe-pg-migrations/helpers/statements_helper'
+require 'safe-pg-migrations/helpers/volatile_default'
 require 'safe-pg-migrations/plugins/verbose_sql_logger'
 require 'safe-pg-migrations/plugins/blocking_activity_logger'
 require 'safe-pg-migrations/plugins/statement_insurer/add_column'

data/lib/safe-pg-migrations/helpers/batch_over.rb

@@ -2,6 +2,31 @@
 
 module SafePgMigrations
   module Helpers
+    # This helper class allows to iterate over records in batches, in a similar
+    # way to ActiveRecord's `in_batches` method with the :use_ranges option,
+    # which was introduced in ActiveRecord 7.1, see:
+    #
+    #   - https://api.rubyonrails.org/classes/ActiveRecord/Batches.html#method-i-in_batches
+    #   - https://github.com/rails/rails/blob/v7.1.0/activerecord/CHANGELOG.md
+    #   - https://github.com/rails/rails/pull/45414
+    #   - https://github.com/rails/rails/commit/620f24782977b8e53e06cf0e2c905a591936e990
+    #
+    # In ActiveRecord 8.1, `in_baches(use_ranges: true)` was optimized further
+    # to use less cpu, memory, and bandwidth, see:
+    #
+    #   - https://github.com/rails/rails/releases/tag/v8.1.0
+    #   - https://github.com/rails/rails/pull/51243
+    #   - https://github.com/rails/rails/commit/c097bf6c24443323da8fe64030dd963951121dea
+    #
+    # If using ActiveRecord 8.1 or later, it's recommended to use the built-in
+    # method, e.g.
+    #
+    #   User.in_batches(of: 100, use_ranges: true).each { |batch| ... }
+    #
+    # Otherwise, this helper can be used as a fallback:
+    #
+    #   SafePgMigrations::Helpers::BatchOver.new(User, of: 100).each_batch { |batch| ... }
+    #
     class BatchOver
       def initialize(model, of: SafePgMigrations.config.backfill_batch_size)
         @model = model

data/lib/safe-pg-migrations/helpers/volatile_default.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  module Helpers
+    module VolatileDefault
+      VOLATILE_DEFAULT_PATTERNS = [
+        /\bclock_timestamp\s*\(/i,
+        /\bnow\s*\(/i,
+        /\bcurrent_timestamp\b/i,
+        /\bcurrent_time\b/i,
+        /\bcurrent_date\b/i,
+        /\brandom\s*\(/i,
+        /\buuid_generate/i,
+        /\bgen_random_uuid\s*\(/i,
+        /\btimeofday\s*\(/i,
+        /\btransaction_timestamp\s*\(/i,
+        /\bstatement_timestamp\s*\(/i,
+        /\bnextval\s*\(/i,
+        /\bgen_random_bytes\s*\(/i,
+      ].freeze
+
+      module_function
+
+      def volatile_default?(default)
+        return false if default.nil?
+        return true if default.is_a?(Proc)
+        return false unless default.is_a?(String)
+
+        VOLATILE_DEFAULT_PATTERNS.any? { |pattern| default.match?(pattern) }
+      end
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb

@@ -8,6 +8,11 @@ module SafePgMigrations
 
         options.delete(:default_value_backfill)
 
+        default = options[:default]
+
+        # Raise if using automatic backfill with a volatile default
+        raise_on_volatile_default(table_name, column_name, default) if volatile_default?(default)
+
         raise <<~ERROR unless backfill_column_default_safe?(table_name)
           Table #{table_name} has more than #{SafePgMigrations.config.default_value_backfill_threshold} rows.
           Backfilling the default value for column #{column_name} on table #{table_name} would take too long.
@@ -53,11 +58,55 @@ module SafePgMigrations
 
         Helpers::Logger.say_method_call(:backfill_column_default, table_name, column_name)
 
-        Helpers::BatchOver.new(model).each_batch do |batch|
+        batch_handler = lambda do |batch|
           batch.update_all("#{quoted_column_name} = DEFAULT")
 
           sleep SafePgMigrations.config.backfill_pause
         end
+
+        backfill_batch_size = SafePgMigrations.config.backfill_batch_size
+
+        if ActiveRecord.version >= Gem::Version.new('8.1')
+          model.in_batches(of: backfill_batch_size, use_ranges: true).each(&batch_handler)
+        else
+          Helpers::BatchOver.new(model, of: backfill_batch_size).each_batch(&batch_handler)
+        end
+      end
+
+      def volatile_default?(default)
+        Helpers::VolatileDefault.volatile_default?(default)
+      end
+
+      def raise_on_volatile_default(table_name, column_name, default)
+        default_display = default.is_a?(Proc) ? '<Proc>' : default
+
+        raise <<~ERROR
+          Using default_value_backfill: :update_in_batches with volatile default '#{default_display}'
+          on #{table_name}.#{column_name} is not allowed.
+
+          Volatile defaults are non-deterministic functions like gen_random_uuid(), now(), or clock_timestamp().
+          They are evaluated per row and can cause migrations to hang for a very long time on large tables.
+          You should backfill them "manually" with proper monitoring and control.
+
+          Split the operation into multiple steps in this EXACT order:
+
+          1. ALTER COLUMN SET DEFAULT (for new and updated rows)
+             change_column_default :#{table_name}, :#{column_name}, '#{default_display}'
+          2. ADD CONSTRAINT CHECK NOT NULL NOT VALID (for new and updated rows)
+             # Only if you need NOT NULL:
+             add_check_constraint :#{table_name}, "#{column_name} IS NOT NULL", name: "check_#{table_name}_#{column_name}_not_null", validate: false
+          3. BACKFILL the column (using a job or something else, chucking by PK)
+             # Your own script to backfill in batches
+          4. VALIDATE CONSTRAINT (check whole table)
+             # Only if you added the constraint in step 3:
+             validate_check_constraint :#{table_name}, name: "check_#{table_name}_#{column_name}_not_null"
+          5. ALTER COLUMN SET NOT NULL
+             # Only if you need NOT NULL:
+             change_column_null :#{table_name}, :#{column_name}, false
+          6. DROP CONSTRAINT
+             # Only if you added the constraint in step 3:
+             remove_check_constraint :#{table_name}, name: "check_#{table_name}_#{column_name}_not_null"
+        ERROR
       end
     end
   end

data/lib/safe-pg-migrations/plugins/statement_insurer.rb

@@ -47,18 +47,14 @@ module SafePgMigrations
       super do |td|
         yield td if block_given?
         td.indexes.map! do |key, index_options|
-          index_options[:algorithm] ||= :default
+          index_options[:algorithm] = nil unless index_options.key?(:algorithm)
           [key, index_options]
         end
       end
     end
 
     def add_index(table_name, column_name, **options)
-      if options[:algorithm] == :default
-        options.delete :algorithm
-      else
-        options[:algorithm] = :concurrently
-      end
+      options[:algorithm] = :concurrently unless options.key?(:algorithm)
 
       Helpers::Logger.say_method_call(:add_index, table_name, column_name, **options)
       without_timeout { super(table_name, column_name, **options) }

data/lib/safe-pg-migrations/plugins/strong_migrations_integration.rb

@@ -12,27 +12,12 @@ module SafePgMigrations
           next unless method == :add_column
 
           options = args.last.is_a?(Hash) ? args.last : {}
-
+          default = options[:default]
           default_value_backfill = options.fetch(:default_value_backfill, :auto)
 
-          if default_value_backfill == :update_in_batches
-            check_message = <<~CHECK
-              default_value_backfill: :update_in_batches will take time if the table is too big.
-
-              Your configuration sets a pause of #{SafePgMigrations.config.backfill_pause} seconds between batches of
-              #{SafePgMigrations.config.backfill_batch_size} rows. Each batch execution will take time as well. Please
-              check that the estimated duration of the migration is acceptable
-              before adding `safety_assured`.
-            CHECK
-
-            check_message += <<~CHECK if SafePgMigrations.config.default_value_backfill_threshold
+          next unless default_value_backfill == :update_in_batches
 
-              Also, please note that SafePgMigrations is configured to raise if the table has more than
-              #{SafePgMigrations.config.default_value_backfill_threshold} rows.
-            CHECK
-
-            stop! check_message
-          end
+          stop! StrongMigrationsIntegration.send(:backfill_check_message, default)
         end
       end
 
@@ -41,6 +26,39 @@ module SafePgMigrations
       def strong_migration_available?
         Object.const_defined? :StrongMigrations
       end
+
+      def backfill_check_message(default)
+        if Helpers::VolatileDefault.volatile_default?(default)
+          default_display = default.is_a?(Proc) ? '<Proc>' : default
+
+          <<~CHECK
+            Using default_value_backfill: :update_in_batches with volatile default '#{default_display}' is not allowed.
+
+            Volatile defaults (like NOW(), clock_timestamp(), random()) are evaluated per row and can cause
+            migrations to hang for a very long time on large tables.
+
+            Please backfill volatile defaults manually instead. See the safe-pg-migrations README for the
+            recommended approach.
+          CHECK
+        else
+          check_message = <<~CHECK
+            default_value_backfill: :update_in_batches will take time if the table is too big.
+
+            Your configuration sets a pause of #{SafePgMigrations.config.backfill_pause} seconds between batches of
+            #{SafePgMigrations.config.backfill_batch_size} rows. Each batch execution will take time as well. Please
+            check that the estimated duration of the migration is acceptable
+            before adding `safety_assured`.
+          CHECK
+
+          check_message += <<~CHECK if SafePgMigrations.config.default_value_backfill_threshold
+
+            Also, please note that SafePgMigrations is configured to raise if the table has more than
+            #{SafePgMigrations.config.default_value_backfill_threshold} rows.
+          CHECK
+
+          check_message
+        end
+      end
     end
 
     SAFE_METHODS = %i[
@@ -64,9 +82,20 @@ module SafePgMigrations
     def add_column(table_name, *args, **options)
       return super unless respond_to?(:safety_assured)
 
-      return safety_assured { super } if options.fetch(:default_value_backfill, :auto) == :auto
+      default_value_backfill = options.fetch(:default_value_backfill, :auto)
 
+      # Auto backfill is safe - use safety_assured
+      return safety_assured { super } if default_value_backfill == :auto
+
+      # :update_in_batches always requires explicit safety_assured (volatile defaults will be
+      # blocked by the check above before reaching this point)
       super
     end
+
+    private
+
+    def volatile_default?(default)
+      Helpers::VolatileDefault.volatile_default?(default)
+    end
   end
 end

data/lib/safe-pg-migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  VERSION = '3.1.4'
+  VERSION = '4.0.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: safe-pg-migrations
 version: !ruby/object:Gem::Version
-  version: 3.1.4
+  version: 4.0.0
 platform: ruby
 authors:
 - Matthieu Prat
@@ -11,7 +11,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2025-03-26 00:00:00.000000000 Z
+date: 2026-03-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -61,6 +61,7 @@ files:
 - lib/safe-pg-migrations/helpers/satisfied_helper.rb
 - lib/safe-pg-migrations/helpers/session_setting_management.rb
 - lib/safe-pg-migrations/helpers/statements_helper.rb
+- lib/safe-pg-migrations/helpers/volatile_default.rb
 - lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
 - lib/safe-pg-migrations/plugins/idempotent_statements.rb
 - lib/safe-pg-migrations/plugins/statement_insurer.rb