online_migrations 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4706d65d4ed40219b3680266294f9382d1db51ac168768b77f55e1a73a2d3de3
-  data.tar.gz: b828d4c3d8cc9ef1268eeaa00f86bc3fda41546963334cb356916b1ad481b3bc
+  metadata.gz: 87bf9f8f917c2d14e8a2e5584a2710dec5ccd66051730b8bb1b5a0b765b70ccb
+  data.tar.gz: 0cbf86b34c7d62f1466c877df231c2d39f3f4d5c2bd4f4dbef080f56e3254a74
 SHA512:
-  metadata.gz: f70ab747e025f34a546da3d97aad6d7a175006fca9c48a3ae402b1683715b1eeb5128213bec52612c9178e457a638f9eb7d370445cf43b7c164f6d9bb548026f
-  data.tar.gz: e001963c1fc82ec925a4b1c917689171e35dc3c4f0fba61f8a29ec0e2f5b2fedefa6733828d81176b489a345113e054f5f174c007152fc407d55e3184853cc77
+  metadata.gz: 3184f370a4cb3a3fa150ff24644355bc6c928eaa88331a0e18ee6f2c2448917898f5165675381162701f49881018516533b97e2364204a5a6ebd8bb19b3063cc
+  data.tar.gz: 954f9d33eba0e94020e0c9de77fd47097cd192915ad60a6c08aff8ab6d7a54b06a6938a50191c6b75596ae06b427076809472098b90a633549f2208ca358c656

data/BACKGROUND_MIGRATIONS.md

@@ -114,6 +114,12 @@ enqueue_background_migration("MyMigrationWithArgs", arg1, arg2, ...)
 * **Isolation**: Background migrations should be isolated and not use application code (for example, models defined in `app/models`). Since these migrations can take a long time to run it's possible for new versions to be deployed while they are still running.
 * **Idempotence**: It should be safe to run `process_batch` multiple times for the same elements. It's important if the Background Migration errors and you run it again, because the same element that errored may be processed again. Make sure that in case that your migration job is going to be retried data integrity is guaranteed.
 
+## Predefined background migrations
+
+* `BackfillColumn` - backfills column(s) with scalar values
+* `CopyColumn` - copies data from one column(s) to other(s)
+* `ResetCounters` - resets one or more counter caches to their correct value
+
 ## Testing
 
 At a minimum, it's recommended that the `#process_batch` method in your background migration is tested. You may also want to test the `#relation` and `#count` methods if they are sufficiently complex.

data/CHANGELOG.md

@@ -1,5 +1,49 @@
 ## master (unreleased)
 
+- Lazy load this gem
+
+- Add ability to reset counter caches using background migrations
+
+    ```ruby
+      class User < ApplicationRecord
+        has_many :projects
+      end
+
+      class Project < ApplicationRecord
+        belongs_to :user, counter_cache: true
+      end
+
+      class ResetUsersProjectsCount < ActiveRecord::Migration[7.0]
+        def up
+          reset_counters_in_background("User", :projects)
+        end
+      end
+    ```
+
+- Accept `0` as `batch_pause` value for background migrations
+- Ignore default scopes in `CopyColumn` and `BackfillColumn` background migrations
+- Raise an error for unsupported database versions
+- Fix backfilling code in suggestion for changing column's NOT NULL
+
+New safe operations
+
+- Changing between `text` and `citext` when not indexed
+- Changing a `string` column to a `citext` column when not indexed
+- Changing a `citext` column to a `string` column with no length limit
+- Increasing the `:precision` of an `interval` column
+- Changing a `cidr` column to an `inet` column
+- Changing an `xml` column to a `text` column
+- Changing an `xml` column to a `string` column with no `:limit`
+- Changing a `bit` column to a `bit_varying` column
+- Increasing or removing the `:limit` of a `bit_varying` column
+
+New unsafe operations
+
+- Decreasing `:precision` of a `datetime` column
+- Decreasing `:limit` of a `timestamptz` column
+- Decreasing `:limit` of a `bit_varying` column
+- Adding a `:limit` to a `bit_varying` column
+
 ## 0.3.0 (2022-02-10)
 
 - Support ActiveRecord 7.0+ versioned schemas

data/README.md

@@ -10,6 +10,10 @@ Catch unsafe PostgreSQL migrations in development and run them easier in product
 
 [![Build Status](https://github.com/fatkodima/online_migrations/actions/workflows/test.yml/badge.svg?branch=master)](https://github.com/fatkodima/online_migrations/actions/workflows/test.yml)
 
+## Cool, but there is a `strong_migrations` already
+
+See [comparison to `strong_migrations`](#comparison-to-strong_migrations)
+
 ## Requirements
 
 - Ruby 2.1+
@@ -282,12 +286,20 @@ end
 
 A few changes don't require a table rewrite (and are safe) in PostgreSQL:
 
-- Increasing the length limit of a `varchar` column (or removing the limit)
-- Changing a `varchar` column to a `text` column
-- Changing a `text` column to a `varchar` column with no length limit
-- Increasing the precision of a `decimal` or `numeric` column
-- Making a `decimal` or `numeric` column unconstrained
-- Changing between `timestamp` and `timestamptz` columns when session time zone is UTC in PostgreSQL 12+
+Type | Safe Changes
+--- | ---
+`bit` | Changing to `bit_varying`
+`bit_varying` | Increasing or removing `:limit`
+`cidr` | Changing to `inet`
+`citext` | Changing to `text` if not indexed, changing to `string` with no `:limit` if not indexed
+`datetime` | Increasing or removing `:precision`, changing to `timestamptz` when session time zone is UTC in PostgreSQL 12+
+`decimal` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
+`interval` | Increasing or removing `:precision`
+`numeric` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
+`string` | Increasing or removing `:limit`, changing to `text`, changing to `citext` if not indexed
+`text` | Changing to `string` with no `:limit`, changing to `citext` if not indexed
+`timestamptz` | Increasing or removing `:limit`, changing to `datetime` when session time zone is UTC in PostgreSQL 12+
+`xml` | Changing to `text`, changing to `string` with no `:limit`
 
 :white_check_mark: **Good**
 
@@ -1292,6 +1304,33 @@ Background migrations:
 - add UI
 - support batching over non-integer and multiple columns
 
+## Comparison to `strong_migrations`
+
+This gem was heavily inspired by the `strong_migrations` and GitLab's approaches to database migrations. This gem is a superset of `strong_migrations`, feature-wise, and has the same APIs.
+
+The main differences are:
+
+1. `strong_migrations` provides you **text guidance** on how to run migrations safer and you should implement them yourself. This new gem has actual [**code helpers**](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/schema_statements.rb) (and suggests them when fails on unsafe migrations) you can use to do what you want. See [example](#example) for an example.
+
+It has migrations helpers for:
+
+* renaming tables/columns
+* changing columns types (including changing primary/foreign keys from `integer` to `bigint`)
+* adding columns with default values
+* backfilling data
+* adding different types of constraints
+* and others
+
+2. This gem has a [powerful internal framework](https://github.com/fatkodima/online_migrations/blob/master/BACKGROUND_MIGRATIONS.md) for running data migrations on very large tables using background migrations.
+
+For example, you can use background migrations to migrate data that’s stored in a single JSON column to a separate table instead; backfill values from one column to another (as one of the steps when changing column type); or backfill some column’s value from an API.
+
+3. Yet, it has more checks for unsafe changes (see [checks](#checks)).
+
+4. Currently, this gem supports only PostgreSQL, while `strong_migrations` also checks `MySQL` and `MariaDB` migrations.
+
+5. This gem is more flexible in terms of configuration - see [config file](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/config.rb) for additional configuration options.
+
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/online_migrations/background_migrations/backfill_column.rb

@@ -20,9 +20,9 @@ module OnlineMigrations
           # Otherwise, the SQL is `WHERE column != value`. This condition ignores column
           # with NULLs in it, so we need to also manually check for NULLs.
           quoted_column = connection.quote_column_name(column)
-          model.where("#{quoted_column} != ? OR #{quoted_column} IS NULL", value)
+          model.unscoped.where("#{quoted_column} != ? OR #{quoted_column} IS NULL", value)
         else
-          Utils.ar_where_not_multiple_conditions(model, updates)
+          Utils.ar_where_not_multiple_conditions(model.unscoped, updates)
         end
       end
 
@@ -42,7 +42,7 @@ module OnlineMigrations
           @model ||= if model_name.present?
                        Object.const_get(model_name, false)
                      else
-                       Utils.define_model(ActiveRecord::Base.connection, table_name)
+                       Utils.define_model(table_name)
                      end
         end
 

data/lib/online_migrations/background_migrations/copy_column.rb

@@ -26,6 +26,7 @@ module OnlineMigrations
 
       def relation
         relation = model
+          .unscoped
           .where(copy_to.map { |to_column| [to_column, nil] }.to_h)
 
         Utils.ar_where_not_multiple_conditions(
@@ -78,7 +79,7 @@ module OnlineMigrations
           @model ||= if model_name.present?
                        Object.const_get(model_name, false)
                      else
-                       Utils.define_model(ActiveRecord::Base.connection, table_name)
+                       Utils.define_model(table_name)
                      end
         end
 

data/lib/online_migrations/background_migrations/migration.rb

@@ -27,10 +27,11 @@ module OnlineMigrations
 
       validates :migration_name, :batch_column_name, presence: true
 
-      validates :batch_pause, :min_value, :max_value, :batch_size, :sub_batch_size,
+      validates :min_value, :max_value, :batch_size, :sub_batch_size,
                   presence: true, numericality: { greater_than: 0 }
 
-      validates :sub_batch_pause_ms, presence: true, numericality: { greater_than_or_equal_to: 0 }
+      validates :batch_pause, :sub_batch_pause_ms, presence: true,
+                  numericality: { greater_than_or_equal_to: 0 }
       validates :rows_count, numericality: { greater_than_or_equal_to: 0 }, allow_nil: true
       validates :arguments, uniqueness: { scope: :migration_name }
 

data/lib/online_migrations/background_migrations/migration_helpers.rb

@@ -166,6 +166,45 @@ module OnlineMigrations
         )
       end
 
+      # Resets one or more counter caches to their correct value using background migrations.
+      # This is useful when adding new counter caches, or if the counter has been corrupted or modified directly by SQL.
+      #
+      # @param model_name [String]
+      # @param counters [Array]
+      # @param touch [Boolean, Symbol, Array] touch timestamp columns when updating.
+      #   - when `true` - will touch `updated_at` and/or `updated_on`
+      #   - when `Symbol` or `Array` - will touch specific column(s)
+      # @param options [Hash] used to control the behavior of background migration.
+      #     See `#enqueue_background_migration`
+      #
+      # @return [OnlineMigrations::BackgroundMigrations::Migration]
+      #
+      # @example
+      #     reset_counters_in_background("User", :projects, :friends, touch: true)
+      #
+      # @example Touch specific column
+      #     reset_counters_in_background("User", :projects, touch: :touched_at)
+      #
+      # @example Touch with specific time value
+      #     reset_counters_in_background("User", :projects, touch: [time: 2.days.ago])
+      #
+      # @see https://api.rubyonrails.org/classes/ActiveRecord/CounterCache/ClassMethods.html#method-i-reset_counters
+      #
+      # @note This method is better suited for extra large tables (100s of millions of records).
+      #     For smaller tables it is probably better and easier to use `reset_counters` from the ActiveRecord.
+      #
+      def reset_counters_in_background(model_name, *counters, touch: nil, **options)
+        model_name = model_name.name if model_name.is_a?(Class)
+
+        enqueue_background_migration(
+          "ResetCounters",
+          model_name,
+          counters,
+          { touch: touch },
+          **options
+        )
+      end
+
       # Creates a background migration for the given job class name.
       #
       # A background migration runs one job at a time, computing the bounds of the next batch

data/lib/online_migrations/background_migrations/reset_counters.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+module OnlineMigrations
+  module BackgroundMigrations
+    # @private
+    class ResetCounters < BackgroundMigration
+      attr_reader :model, :counters, :touch
+
+      def initialize(model_name, counters, options = {})
+        @model = Object.const_get(model_name, false)
+        @counters = counters
+        @touch = options[:touch]
+      end
+
+      def relation
+        model.unscoped
+      end
+
+      def process_batch(relation)
+        updates = counters.map do |counter_association|
+          has_many_association = has_many_association(counter_association)
+
+          foreign_key  = has_many_association.foreign_key.to_s
+          child_class  = has_many_association.klass
+          reflection   = child_class._reflections.values.find { |e| e.belongs_to? && e.foreign_key.to_s == foreign_key && e.options[:counter_cache].present? }
+          counter_name = reflection.counter_cache_column
+
+          quoted_association_table = connection.quote_table_name(has_many_association.table_name)
+          count_subquery = <<-SQL.strip_heredoc
+            SELECT COUNT(*)
+            FROM #{quoted_association_table}
+            WHERE #{quoted_association_table}.#{connection.quote_column_name(foreign_key)} =
+              #{model.quoted_table_name}.#{model.quoted_primary_key}
+          SQL
+
+          "#{connection.quote_column_name(counter_name)} = (#{count_subquery})"
+        end
+
+        if touch
+          names = touch if touch != true
+          names = Array.wrap(names)
+          options = names.extract_options!
+          touch_updates = touch_attributes_with_time(*names, **options)
+          # In ActiveRecord 4.2 sanitize_sql_for_assignment is protected
+          updates << model.send(:sanitize_sql_for_assignment, touch_updates)
+        end
+
+        relation.update_all(updates.join(", "))
+      end
+
+      def count
+        # Exact counts are expensive on large tables, since PostgreSQL
+        # needs to do a full scan. An estimated count should give a pretty decent
+        # approximation of rows count in this case.
+        Utils.estimated_count(connection, model.table_name)
+      end
+
+      private
+        def has_many_association(counter_association) # rubocop:disable Naming/PredicateName
+          has_many_association = model.reflect_on_association(counter_association)
+
+          unless has_many_association
+            has_many = model.reflect_on_all_associations(:has_many)
+
+            has_many_association = has_many.find do |association|
+              counter_cache_column = association.counter_cache_column
+
+              # ActiveRecord <= 4.2 is able to return only explicitly provided `counter_cache` column.
+              if !counter_cache_column && Utils.ar_version <= 4.2
+                counter_cache_column = "#{association.name}_count"
+              end
+              counter_cache_column && counter_cache_column.to_sym == counter_association.to_sym
+            end
+
+            counter_association = has_many_association.plural_name if has_many_association
+          end
+          raise ArgumentError, "'#{model.name}' has no association called '#{counter_association}'" unless has_many_association
+
+          if has_many_association.is_a?(ActiveRecord::Reflection::ThroughReflection)
+            has_many_association = has_many_association.through_reflection
+          end
+
+          has_many_association
+        end
+
+        def touch_attributes_with_time(*names, time: nil)
+          attribute_names = timestamp_attributes_for_update & model.column_names
+          attribute_names |= names.map(&:to_s)
+          attribute_names.map { |attribute_name| [attribute_name, time || Time.current] }.to_h
+        end
+
+        def timestamp_attributes_for_update
+          ["updated_at", "updated_on"].map { |name| model.attribute_aliases[name] || name }
+        end
+
+        def connection
+          model.connection
+        end
+    end
+  end
+end

data/lib/online_migrations/change_column_type_helpers.rb

@@ -110,7 +110,7 @@ module OnlineMigrations
           column_options = options[column_name] || {}
           tmp_column_name = conversions[column_name]
 
-          if @connection.server_version >= 11_00_00 &&
+          if __raw_connection.server_version >= 11_00_00 &&
              primary_key(table_name) == column_name.to_s && old_col.type == :integer
             # If the column to be converted is a Primary Key, set it to
             # `NOT NULL DEFAULT 0` and we'll copy the correct values when backfilling.

data/lib/online_migrations/command_checker.rb

@@ -27,6 +27,7 @@ module OnlineMigrations
     end
 
     def check(command, *args, &block)
+      check_database_version
       check_lock_timeout
 
       unless safe?
@@ -43,6 +44,22 @@ module OnlineMigrations
     end
 
     private
+      def check_database_version
+        return if defined?(@database_version_checked)
+
+        adapter = connection.adapter_name
+        case adapter
+        when /postg/i
+          if postgresql_version < Gem::Version.new("9.6")
+            raise "#{adapter} < 9.6 is not supported"
+          end
+        else
+          raise "#{adapter} is not supported"
+        end
+
+        @database_version_checked = true
+      end
+
       def check_lock_timeout
         limit = OnlineMigrations.config.lock_timeout_limit
 
@@ -206,10 +223,21 @@ module OnlineMigrations
 
         type = type.to_sym
 
-        existing_column = connection.columns(table_name).find { |c| c.name == column_name.to_s }
+        existing_column = column_for(table_name, column_name)
         if existing_column
           existing_type = existing_column.type.to_sym
 
+          # To get a list of binary-coercible types:
+          #
+          # SELECT stype.typname AS source, ttype.typname AS target
+          # FROM pg_cast
+          #   INNER JOIN pg_type stype ON pg_cast.castsource = stype.oid
+          #   INNER JOIN pg_type ttype ON pg_cast.casttarget = ttype.oid
+          # WHERE castmethod = 'b'
+          # ORDER BY 1, 2
+
+          # https://www.postgresql.org/docs/release/9.2.0/#AEN124164
+
           safe =
             case type
             when :string
@@ -218,12 +246,25 @@ module OnlineMigrations
               case existing_type
               when :string
                 !options[:limit] || (existing_column.limit && options[:limit] >= existing_column.limit)
-              when :text
+              when :text, :xml
                 !options[:limit]
+              when :citext
+                !options[:limit] && !indexed?(table_name, column_name)
               end
             when :text
-              # safe to change varchar to text (and text to text)
-              [:string, :text].include?(existing_type)
+              [:string, :text, :xml].include?(existing_type) ||
+              (existing_type == :citext && !indexed?(table_name, column_name))
+            when :citext
+              [:string, :text].include?(existing_type) && !indexed?(table_name, column_name)
+            when :bit_varying
+              case existing_type
+              when :bit
+                !options[:limit]
+              when :bit_varying
+                # safe to increase limit or remove it
+                # not safe to decrease limit or add a limit
+                !options[:limit] || (existing_column.limit && options[:limit] >= existing_column.limit)
+              end
             when :numeric, :decimal
               # numeric and decimal are equivalent and can be used interchangably
               [:numeric, :decimal].include?(existing_type) &&
@@ -239,9 +280,25 @@ module OnlineMigrations
                 )
               )
             when :datetime, :timestamp, :timestamptz
-              [:timestamp, :timestamptz].include?(existing_type) &&
-              postgresql_version >= Gem::Version.new("12") &&
-              connection.select_value("SHOW timezone") == "UTC"
+              # precision for datetime
+              # limit for timestamp, timestamptz
+              precision = (type == :datetime ? options[:precision] : options[:limit]) || 6
+              existing_precision = existing_column.precision || existing_column.limit || 6
+
+              [:datetime, :timestamp, :timestamptz].include?(existing_type) &&
+              precision >= existing_precision &&
+              (type == existing_type ||
+                (postgresql_version >= Gem::Version.new("12") &&
+                connection.select_value("SHOW timezone") == "UTC"))
+            when :interval
+              precision = options[:precision] || options[:limit] || 6
+              existing_precision = existing_column.precision || existing_column.limit || 6
+
+              # PostgreSQL interval data type was added in https://github.com/rails/rails/pull/16919
+              (existing_type == :interval || (Utils.ar_version < 6.1 && existing_column.sql_type.start_with?("interval"))) &&
+              precision >= existing_precision
+            when :inet
+              existing_type == :cidr
             else
               type == existing_type &&
               options[:limit] == existing_column.limit &&
@@ -281,15 +338,13 @@ module OnlineMigrations
             constraint_name = "#{table_name}_#{column_name}_null"
             vars = {
               add_constraint_code: command_str(:add_not_null_constraint, table_name, column_name, name: constraint_name, validate: false),
-              backfill_code: nil,
               validate_constraint_code: command_str(:validate_not_null_constraint, table_name, column_name, name: constraint_name),
               remove_constraint_code: nil,
+              table_name: table_name,
+              column_name: column_name,
+              default: default,
             }
 
-            if !default.nil?
-              vars[:backfill_code] = command_str(:update_column_in_batches, table_name, column_name, default)
-            end
-
             if postgresql_version >= Gem::Version.new("12")
               vars[:remove_constraint_code] = command_str(:remove_check_constraint, table_name, name: constraint_name)
               vars[:change_column_null_code] = command_str(:change_column_null, table_name, column_name, false)
@@ -537,7 +592,7 @@ module OnlineMigrations
             target_version.to_s
           else
             # For rails 6.0+ we can use connection.database_version
-            pg_connection = connection.instance_variable_get(:@connection)
+            pg_connection = connection.send(:__raw_connection)
             database_version = pg_connection.server_version
             patch = database_version % 100
             database_version /= 100
@@ -669,6 +724,10 @@ module OnlineMigrations
           [:integer, :bigint, :serial, :bigserial, :uuid].include?(type)
       end
 
+      def indexed?(table_name, column_name)
+        connection.indexes(table_name).any? { |index| index.columns.include?(column_name.to_s) }
+      end
+
       def column_for(table_name, column_name)
         connection.columns(table_name).find { |column| column.name == column_name.to_s }
       end

data/lib/online_migrations/error_messages.rb

@@ -214,11 +214,13 @@ class <%= migration_name %> < <%= migration_parent %>
 
   def change
     <%= add_constraint_code %>
-<% if backfill_code %>
+<% unless default.nil? %>
 
     # Passing a default value to change_column_null runs a single UPDATE query,
     # which can cause downtime. Instead, backfill the existing rows in batches.
-    <%= backfill_code %>
+    update_column_in_batches(:<%= table_name %>, :<%= column_name %>, <%= default.inspect %>) do |relation|
+      relation.where(<%= column_name %>: nil)
+    end
 
 <% end %>
     <%= validate_constraint_code %>

data/lib/online_migrations/schema_statements.rb

@@ -81,7 +81,7 @@ module OnlineMigrations
         end
       end
 
-      model = Utils.define_model(self, table_name)
+      model = Utils.define_model(table_name, self)
 
       conditions = columns_and_values.map do |(column_name, value)|
         value = Arel.sql(value.call.to_s) if value.is_a?(Proc)
@@ -396,7 +396,7 @@ module OnlineMigrations
         raise ArgumentError, "Expressions as default are not supported"
       end
 
-      if @connection.server_version >= 11_00_00 && !Utils.volatile_default?(self, type, default)
+      if __raw_connection.server_version >= 11_00_00 && !Utils.volatile_default?(self, type, default)
         add_column(table_name, column_name, type, **options)
       else
         __ensure_not_in_transaction!
@@ -422,7 +422,7 @@ module OnlineMigrations
           add_not_null_constraint(table_name, column_name, validate: false)
           validate_not_null_constraint(table_name, column_name)
 
-          if @connection.server_version >= 12_00_00
+          if __raw_connection.server_version >= 12_00_00
             # In PostgreSQL 12+ it is safe to "promote" a CHECK constraint to `NOT NULL` for the column
             change_column_null(table_name, column_name, false)
             remove_not_null_constraint(table_name, column_name)
@@ -1046,5 +1046,14 @@ module OnlineMigrations
         _, schema = table_name.to_s.split(".").reverse
         schema ? quote(schema) : "current_schema()"
       end
+
+      def __raw_connection
+        # ActiveRecord > 7.0.2.2 (https://github.com/rails/rails/pull/44530)
+        if defined?(@raw_connection)
+          @raw_connection
+        else
+          @connection
+        end
+      end
   end
 end

data/lib/online_migrations/utils.rb

@@ -53,7 +53,7 @@ module OnlineMigrations
         end
       end
 
-      def define_model(connection, table_name)
+      def define_model(table_name, connection = ActiveRecord::Base.connection)
         Class.new(ActiveRecord::Base) do
           self.table_name = table_name
           self.inheritance_column = :_type_disabled
@@ -101,7 +101,14 @@ module OnlineMigrations
           WHERE relname = #{quoted_table}
             AND relnamespace = current_schema()::regnamespace
         SQL
-        count.to_i if count
+
+        if count
+          count = count.to_i
+          # If the table has never yet been vacuumed or analyzed, reltuples contains -1
+          # indicating that the row count is unknown.
+          count = 0 if count == -1
+          count
+        end
       end
 
       def ar_where_not_multiple_conditions(relation, conditions)

data/lib/online_migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module OnlineMigrations
-  VERSION = "0.3.0"
+  VERSION = "0.4.0"
 end

data/lib/online_migrations.rb

@@ -1,46 +1,61 @@
 # frozen_string_literal: true
 
 require "active_record"
-
-require "online_migrations/utils"
-require "online_migrations/error_messages"
-require "online_migrations/config"
-require "online_migrations/batch_iterator"
-require "online_migrations/verbose_sql_logs"
-require "online_migrations/migration"
-require "online_migrations/migrator"
-require "online_migrations/database_tasks"
-require "online_migrations/foreign_key_definition"
-require "online_migrations/foreign_keys_collector"
-require "online_migrations/index_definition"
-require "online_migrations/indexes_collector"
-require "online_migrations/command_checker"
-require "online_migrations/schema_cache"
-require "online_migrations/background_migration"
-require "online_migrations/background_migrations/config"
-require "online_migrations/background_migrations/migration_status_validator"
-require "online_migrations/background_migrations/migration_job_status_validator"
-require "online_migrations/background_migrations/background_migration_class_validator"
-require "online_migrations/background_migrations/backfill_column"
-require "online_migrations/background_migrations/copy_column"
-require "online_migrations/background_migrations/migration_job"
-require "online_migrations/background_migrations/migration"
-require "online_migrations/background_migrations/migration_job_runner"
-require "online_migrations/background_migrations/migration_runner"
-require "online_migrations/background_migrations/migration_helpers"
-require "online_migrations/background_migrations/advisory_lock"
-require "online_migrations/background_migrations/scheduler"
-require "online_migrations/lock_retrier"
-require "online_migrations/command_recorder"
-require "online_migrations/copy_trigger"
-require "online_migrations/change_column_type_helpers"
-require "online_migrations/schema_statements"
 require "online_migrations/version"
 
 module OnlineMigrations
   class Error < StandardError; end
   class UnsafeMigration < Error; end
 
+  extend ActiveSupport::Autoload
+
+  autoload :Utils
+  autoload :ErrorMessages
+  autoload :Config
+  autoload :BatchIterator
+  autoload :VerboseSqlLogs
+  autoload :Migration
+  autoload :Migrator
+  autoload :DatabaseTasks
+  autoload :ForeignKeyDefinition
+  autoload :ForeignKeysCollector
+  autoload :IndexDefinition
+  autoload :IndexesCollector
+  autoload :CommandChecker
+  autoload :SchemaCache
+  autoload :BackgroundMigration
+
+  autoload_at "online_migrations/lock_retrier" do
+    autoload :LockRetrier
+    autoload :ConstantLockRetrier
+    autoload :ExponentialLockRetrier
+    autoload :NullLockRetrier
+  end
+
+  autoload :CommandRecorder
+  autoload :CopyTrigger
+  autoload :ChangeColumnTypeHelpers
+  autoload :SchemaStatements
+
+  module BackgroundMigrations
+    extend ActiveSupport::Autoload
+
+    autoload :Config
+    autoload :MigrationStatusValidator
+    autoload :MigrationJobStatusValidator
+    autoload :BackgroundMigrationClassValidator
+    autoload :BackfillColumn
+    autoload :CopyColumn
+    autoload :ResetCounters
+    autoload :MigrationJob
+    autoload :Migration
+    autoload :MigrationJobRunner
+    autoload :MigrationRunner
+    autoload :MigrationHelpers
+    autoload :AdvisoryLock
+    autoload :Scheduler
+  end
+
   class << self
     # @private
     attr_accessor :current_migration

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: online_migrations
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - fatkodima
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-02-10 00:00:00.000000000 Z
+date: 2022-03-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -31,25 +31,10 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/workflows/test.yml"
-- ".gitignore"
-- ".rubocop.yml"
-- ".yardopts"
 - BACKGROUND_MIGRATIONS.md
 - CHANGELOG.md
-- Gemfile
-- Gemfile.lock
 - LICENSE.txt
 - README.md
-- Rakefile
-- gemfiles/activerecord_42.gemfile
-- gemfiles/activerecord_50.gemfile
-- gemfiles/activerecord_51.gemfile
-- gemfiles/activerecord_52.gemfile
-- gemfiles/activerecord_60.gemfile
-- gemfiles/activerecord_61.gemfile
-- gemfiles/activerecord_70.gemfile
-- gemfiles/activerecord_head.gemfile
 - lib/generators/online_migrations/background_migration_generator.rb
 - lib/generators/online_migrations/install_generator.rb
 - lib/generators/online_migrations/templates/background_migration.rb.tt
@@ -69,6 +54,7 @@ files:
 - lib/online_migrations/background_migrations/migration_job_status_validator.rb
 - lib/online_migrations/background_migrations/migration_runner.rb
 - lib/online_migrations/background_migrations/migration_status_validator.rb
+- lib/online_migrations/background_migrations/reset_counters.rb
 - lib/online_migrations/background_migrations/scheduler.rb
 - lib/online_migrations/batch_iterator.rb
 - lib/online_migrations/change_column_type_helpers.rb
@@ -90,7 +76,6 @@ files:
 - lib/online_migrations/utils.rb
 - lib/online_migrations/verbose_sql_logs.rb
 - lib/online_migrations/version.rb
-- online_migrations.gemspec
 homepage: https://github.com/fatkodima/online_migrations
 licenses:
 - MIT

data/.github/workflows/test.yml

@@ -1,112 +0,0 @@
-name: Test
-on: [push, pull_request]
-
-jobs:
-  # Run the linter first for rapid feedback if some trivial stylistic issues
-  # slipped through the cracks.
-  lint:
-    runs-on: ubuntu-latest
-    steps:
-      - uses: actions/checkout@v2
-      - uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: 3.1.0
-          bundler-cache: true
-      - run: bundle exec rubocop
-
-  test:
-    needs: lint
-    runs-on: ubuntu-latest
-    services:
-      postgres:
-        image: postgres
-        env:
-          POSTGRES_DB: online_migrations_test
-          POSTGRES_PASSWORD: postgres
-        options: >-
-          --health-cmd pg_isready
-          --health-interval 10s
-          --health-timeout 5s
-          --health-retries 5
-        ports:
-          - 5432:5432
-    strategy:
-      matrix:
-        include:
-          - ruby-version: 2.4.10
-            gemfile: gemfiles/activerecord_42.gemfile
-          - ruby-version: 2.5.9
-            gemfile: gemfiles/activerecord_42.gemfile
-          - ruby-version: 2.6.9
-            gemfile: gemfiles/activerecord_42.gemfile
-
-          - ruby-version: 2.4.10
-            gemfile: gemfiles/activerecord_50.gemfile
-          - ruby-version: 2.5.9
-            gemfile: gemfiles/activerecord_50.gemfile
-          - ruby-version: 2.6.9
-            gemfile: gemfiles/activerecord_50.gemfile
-          - ruby-version: 2.7.5
-            gemfile: gemfiles/activerecord_50.gemfile
-
-          - ruby-version: 2.4.10
-            gemfile: gemfiles/activerecord_51.gemfile
-          - ruby-version: 2.5.9
-            gemfile: gemfiles/activerecord_51.gemfile
-          - ruby-version: 2.6.9
-            gemfile: gemfiles/activerecord_51.gemfile
-          - ruby-version: 2.7.5
-            gemfile: gemfiles/activerecord_51.gemfile
-
-          - ruby-version: 2.4.10
-            gemfile: gemfiles/activerecord_52.gemfile
-          - ruby-version: 2.5.9
-            gemfile: gemfiles/activerecord_52.gemfile
-          - ruby-version: 2.6.9
-            gemfile: gemfiles/activerecord_52.gemfile
-          - ruby-version: 2.7.5
-            gemfile: gemfiles/activerecord_52.gemfile
-
-          - ruby-version: 2.5.9
-            gemfile: gemfiles/activerecord_60.gemfile
-          - ruby-version: 2.6.9
-            gemfile: gemfiles/activerecord_60.gemfile
-          - ruby-version: 2.7.5
-            gemfile: gemfiles/activerecord_60.gemfile
-          - ruby-version: 3.0.3
-            gemfile: gemfiles/activerecord_60.gemfile
-
-          - ruby-version: 2.5.9
-            gemfile: gemfiles/activerecord_61.gemfile
-          - ruby-version: 2.6.9
-            gemfile: gemfiles/activerecord_61.gemfile
-          - ruby-version: 2.7.4
-            gemfile: gemfiles/activerecord_61.gemfile
-          - ruby-version: 3.0.3
-            gemfile: gemfiles/activerecord_61.gemfile
-          - ruby-version: 3.1.0
-            gemfile: gemfiles/activerecord_61.gemfile
-
-          - ruby-version: 2.7.4
-            gemfile: gemfiles/activerecord_70.gemfile
-          - ruby-version: 3.0.3
-            gemfile: gemfiles/activerecord_70.gemfile
-          - ruby-version: 3.1.0
-            gemfile: gemfiles/activerecord_70.gemfile
-
-          - ruby-version: 2.7.4
-            gemfile: gemfiles/activerecord_head.gemfile
-          - ruby-version: 3.0.3
-            gemfile: gemfiles/activerecord_head.gemfile
-          - ruby-version: 3.1.0
-            gemfile: gemfiles/activerecord_head.gemfile
-    env:
-      BUNDLE_GEMFILE: ${{ github.workspace }}/${{ matrix.gemfile }}
-    steps:
-      - uses: actions/checkout@v2
-      - uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: ${{ matrix.ruby-version }}
-          bundler-cache: true
-      - name: Run the test suite
-        run: bundle exec rake test

data/.gitignore

@@ -1,11 +0,0 @@
-/.bundle/
-/.yardoc
-/_yardoc/
-/coverage/
-/doc/
-/pkg/
-/spec/reports/
-/tmp/
-
-gemfiles/**.lock
-/debug.log

data/.rubocop.yml

@@ -1,120 +0,0 @@
-AllCops:
-  NewCops: enable
-  SuggestExtensions: false
-
-Style/StringLiterals:
-  EnforcedStyle: double_quotes
-
-Style/RescueModifier:
-  Exclude:
-    - test/**/*
-
-Style/IfUnlessModifier:
-  Enabled: false
-
-Style/SymbolArray:
-  EnforcedStyle: brackets
-
-Style/WordArray:
-  EnforcedStyle: brackets
-
-Style/GuardClause:
-  Enabled: false
-
-Style/Documentation:
-  Enabled: false
-
-Style/MutableConstant:
-  Enabled: false
-
-Style/MissingRespondToMissing:
-  Enabled: false
-
-Style/TrailingCommaInHashLiteral:
-  EnforcedStyleForMultiline: comma
-
-Style/TrailingCommaInArrayLiteral:
-  EnforcedStyleForMultiline: comma
-
-Style/NumericPredicate:
-  Enabled: false
-
-Style/NegatedIf:
-  Enabled: false
-
-Style/ConditionalAssignment:
-  Enabled: false
-
-# only for ruby 2.3+
-Style/SafeNavigation:
-  Enabled: false
-
-Style/NumericLiterals:
-  Enabled: false
-
-Style/Next:
-  Enabled: false
-
-Style/GlobalVars:
-  Exclude:
-    - test/**/*
-
-Style/Lambda:
-  EnforcedStyle: literal
-
-Style/WhileUntilModifier:
-  Enabled: false
-
-Style/HashAsLastArrayItem:
-  Enabled: false
-
-# only for ruby 2.6+
-Style/MapToHash:
-  Enabled: false
-
-# we can not use new syntax for older rubies
-Lint/ErbNewArguments:
-  Enabled: false
-
-Lint/MissingSuper:
-  Enabled: false
-
-Layout/EmptyLinesAroundAccessModifier:
-  EnforcedStyle: only_before
-
-Layout/IndentationConsistency:
-  EnforcedStyle: indented_internal_methods
-
-Layout/ArgumentAlignment:
-  Enabled: false
-
-Layout/LineLength:
-  Enabled: false
-
-Layout/MultilineMethodCallIndentation:
-  Enabled: false
-
-Layout/HashAlignment:
-  Enabled: false
-
-Naming/VariableNumber:
-  Exclude:
-    - test/**/*
-
-Naming/MethodParameterName:
-  AllowedNames:
-    - of
-    - fk
-
-Naming/AccessorMethodName:
-  Exclude:
-    - test/**/*
-
-Gemspec/RequiredRubyVersion:
-  Enabled: false
-
-Gemspec/RequireMFA:
-  Enabled: false
-
-Metrics:
-  Enabled: false

data/.yardopts

@@ -1 +0,0 @@
---no-private --markup markdown lib/**/**.rb - README.md

data/Gemfile

@@ -1,27 +0,0 @@
-# frozen_string_literal: true
-
-source "https://rubygems.org"
-
-# Specify your gem's dependencies in online_migrations.gemspec
-gemspec
-
-gem "minitest", "~> 5.0"
-gem "rake", "~> 12.0"
-gem "yard"
-
-if defined?(@ar_gem_requirement)
-  gem "activerecord", @ar_gem_requirement
-  gem "railties", @ar_gem_requirement
-else
-  gem "activerecord" # latest
-  gem "railties" # to test generator
-
-  # Run Rubocop only on latest rubies, because it is incompatible with older versions.
-  gem "rubocop", "~> 1.24"
-end
-
-if defined?(@pg_gem_requirement)
-  gem "pg", @pg_gem_requirement
-else
-  gem "pg", "~> 1.2"
-end

data/Gemfile.lock

@@ -1,108 +0,0 @@
-PATH
-  remote: .
-  specs:
-    online_migrations (0.3.0)
-      activerecord (>= 4.2)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    actionpack (7.0.2)
-      actionview (= 7.0.2)
-      activesupport (= 7.0.2)
-      rack (~> 2.0, >= 2.2.0)
-      rack-test (>= 0.6.3)
-      rails-dom-testing (~> 2.0)
-      rails-html-sanitizer (~> 1.0, >= 1.2.0)
-    actionview (7.0.2)
-      activesupport (= 7.0.2)
-      builder (~> 3.1)
-      erubi (~> 1.4)
-      rails-dom-testing (~> 2.0)
-      rails-html-sanitizer (~> 1.1, >= 1.2.0)
-    activemodel (7.0.2)
-      activesupport (= 7.0.2)
-    activerecord (7.0.2)
-      activemodel (= 7.0.2)
-      activesupport (= 7.0.2)
-    activesupport (7.0.2)
-      concurrent-ruby (~> 1.0, >= 1.0.2)
-      i18n (>= 1.6, < 2)
-      minitest (>= 5.1)
-      tzinfo (~> 2.0)
-    ast (2.4.2)
-    builder (3.2.4)
-    concurrent-ruby (1.1.9)
-    crass (1.0.6)
-    erubi (1.10.0)
-    i18n (1.9.1)
-      concurrent-ruby (~> 1.0)
-    loofah (2.13.0)
-      crass (~> 1.0.2)
-      nokogiri (>= 1.5.9)
-    method_source (1.0.0)
-    mini_portile2 (2.7.1)
-    minitest (5.15.0)
-    nokogiri (1.13.1)
-      mini_portile2 (~> 2.7.0)
-      racc (~> 1.4)
-    parallel (1.21.0)
-    parser (3.1.0.0)
-      ast (~> 2.4.1)
-    pg (1.3.1)
-    racc (1.6.0)
-    rack (2.2.3)
-    rack-test (1.1.0)
-      rack (>= 1.0, < 3)
-    rails-dom-testing (2.0.3)
-      activesupport (>= 4.2.0)
-      nokogiri (>= 1.6)
-    rails-html-sanitizer (1.4.2)
-      loofah (~> 2.3)
-    railties (7.0.2)
-      actionpack (= 7.0.2)
-      activesupport (= 7.0.2)
-      method_source
-      rake (>= 12.2)
-      thor (~> 1.0)
-      zeitwerk (~> 2.5)
-    rainbow (3.1.1)
-    rake (12.3.3)
-    regexp_parser (2.2.0)
-    rexml (3.2.5)
-    rubocop (1.25.1)
-      parallel (~> 1.10)
-      parser (>= 3.1.0.0)
-      rainbow (>= 2.2.2, < 4.0)
-      regexp_parser (>= 1.8, < 3.0)
-      rexml
-      rubocop-ast (>= 1.15.1, < 2.0)
-      ruby-progressbar (~> 1.7)
-      unicode-display_width (>= 1.4.0, < 3.0)
-    rubocop-ast (1.15.1)
-      parser (>= 3.0.1.1)
-    ruby-progressbar (1.11.0)
-    thor (1.2.1)
-    tzinfo (2.0.4)
-      concurrent-ruby (~> 1.0)
-    unicode-display_width (2.1.0)
-    webrick (1.7.0)
-    yard (0.9.27)
-      webrick (~> 1.7.0)
-    zeitwerk (2.5.4)
-
-PLATFORMS
-  ruby
-
-DEPENDENCIES
-  activerecord
-  minitest (~> 5.0)
-  online_migrations!
-  pg (~> 1.2)
-  railties
-  rake (~> 12.0)
-  rubocop (~> 1.24)
-  yard
-
-BUNDLED WITH
-   2.1.4

data/Rakefile

@@ -1,23 +0,0 @@
-# frozen_string_literal: true
-
-require "bundler/gem_tasks"
-require "rake/testtask"
-
-Rake::TestTask.new(:test) do |t|
-  t.libs << "test"
-  t.libs << "lib"
-  t.test_files = FileList["test/**/*_test.rb"]
-end
-
-require "rdoc/task"
-
-RDoc::Task.new(:rdoc) do |rdoc|
-  rdoc.rdoc_dir = "rdoc"
-  rdoc.title    = "OnlineMigrations"
-  rdoc.options << "--line-numbers"
-  rdoc.rdoc_files.include("README.md")
-  rdoc.rdoc_files.include("BACKGROUND_MIGRATIONS.md")
-  rdoc.rdoc_files.include("lib/**/*.rb")
-end
-
-task default: :test

data/gemfiles/activerecord_42.gemfile

@@ -1,6 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 4.2.0"
-@pg_gem_requirement = "< 1"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_50.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 5.0.0"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_51.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 5.1.0"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_52.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 5.2.0"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_60.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 6.0.0"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_61.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 6.1.0"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_70.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = "~> 7.0.0"
-
-eval_gemfile "../Gemfile"

data/gemfiles/activerecord_head.gemfile

@@ -1,5 +0,0 @@
-# frozen_string_literal: true
-
-@ar_gem_requirement = { github: "rails/rails", branch: "main" }
-
-eval_gemfile "../Gemfile"

data/online_migrations.gemspec

@@ -1,28 +0,0 @@
-# frozen_string_literal: true
-
-require_relative "lib/online_migrations/version"
-
-Gem::Specification.new do |spec|
-  spec.name          = "online_migrations"
-  spec.version       = OnlineMigrations::VERSION
-  spec.authors       = ["fatkodima"]
-  spec.email         = ["fatkodima123@gmail.com"]
-
-  spec.summary       = "Catch unsafe PostgreSQL migrations in development and run them easier in production"
-  spec.homepage      = "https://github.com/fatkodima/online_migrations"
-  spec.license       = "MIT"
-  spec.required_ruby_version = Gem::Requirement.new(">= 2.1.0")
-
-  spec.metadata["homepage_uri"] = spec.homepage
-  spec.metadata["source_code_uri"] = spec.homepage
-  spec.metadata["changelog_uri"] = "#{spec.homepage}/blob/master/CHANGELOG.md"
-
-  # Specify which files should be added to the gem when it is released.
-  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
-  spec.files = Dir.chdir(File.expand_path(__dir__)) do
-    `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
-  end
-  spec.require_paths = ["lib"]
-
-  spec.add_dependency "activerecord", ">= 4.2"
-end