safe-pg-migrations 1.3.0 → 1.4.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fdee6e752da3fbdec93e5fd5c72fb370497ddd66f1cc27f33609c2a31e364c53
-  data.tar.gz: 143a5dcbf614abed765fac78b16969d900ded5ec574f11d939d802b94d7e2586
+  metadata.gz: c5137a033537f088b0a0c3f41dc8b632d5c37ef81cf5ac25bc62d6b01e62c56c
+  data.tar.gz: 5f0029750d93bcb04ffcb65e8a8bf0b6a8420ee1f6d7c12a85efac967e3b5c93
 SHA512:
-  metadata.gz: 41c264aa05c7b8ad5fe90f7ecdc88fa700504562e69695d5fd9f4dc9ac63315fb666818554e1387251b367aed241fccff31154589f70309905e1ed1cfae90c68
-  data.tar.gz: 9386e6a8b0366387543998ec5cec42e719d2a4138d5eb52f13f6580d2a3d31e48bd438760b358beb1ab5eb32bd8598927d69bafd03756108a514e88eeb431022
+  metadata.gz: afbb64bcc3de2680200ec2cc735b6fa7fc7bb02a71b056be7dcafd3a02a9a11895398632e128ad727e6b089e149ccf75617c5f82674bd1472b16af79c15c0f77
+  data.tar.gz: '0946ab447b6c503ff9965f71edba84e973ddc6e5bc658fe9785672d23c3dc6b0ace071e9081727defd8585dba3d6faedb22145bd9fe8c84e0cd735e4ce44457f'

data/README.md

@@ -2,6 +2,8 @@
 
 ActiveRecord migrations for Postgres made safe.
 
+![safe-pg-migrations](./logo.png)
+
 ## Requirements
 
 - Ruby 2.5+
@@ -91,7 +93,7 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
 - In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
 - In order to add an index concurrently, we have to be outside a transaction.
 
-Note that if a migration fails, it won't be rollbacked. This can result in migrations being partially applied. In that case, they need to be manually reverted.
+Note that if a migration fails, it won't be rolled back. This can result in migrations being partially applied. In that case, they need to be manually reverted.
 
 </details>
 
@@ -108,7 +110,7 @@ PG will still needs to update every row of the table, and will most likely state
 
 <blockquote>
 
-**Note: Pre-postgre 11**
+**Note: Pre-postgres 11**
 Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
 
 **Safe PG Migrations** makes it safe by:
@@ -120,7 +122,7 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
 
 Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
 
-</blockquote>    
+</blockquote>
 
 </details>
 
@@ -137,7 +139,7 @@ If you still get lock timeout while adding / removing indexes, it might be for o
 
 </details>
 
-<details><summary id="safe_add_foreign_key">safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
+<details><summary id="safe_add_foreign_key">Safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
 
 Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
 
@@ -150,7 +152,7 @@ Adding the constraint itself is rather fast, the major part of the time is spent
 
 <details><summary>Retry after lock timeout</summary>
 
-When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
+When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
 </details>
 
 <details><summary>Blocking activity logging</summary>
@@ -219,11 +221,11 @@ SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retry
 SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
 ```
 
-## Runnings tests
+## Running tests
 
 ```bash
 bundle
-psql -h localhost -U postgres -c 'CREATE DATABASE safe_pg_migrations_test'
+psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
 rake test
 ```
 
@@ -257,3 +259,4 @@ Interesting reads:
 - [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
 - [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
 - [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
+- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)

data/lib/safe-pg-migrations/base.rb

@@ -6,17 +6,19 @@ require 'safe-pg-migrations/plugins/verbose_sql_logger'
 require 'safe-pg-migrations/plugins/blocking_activity_logger'
 require 'safe-pg-migrations/plugins/statement_insurer'
 require 'safe-pg-migrations/plugins/statement_retrier'
-require 'safe-pg-migrations/plugins/idem_potent_statements'
+require 'safe-pg-migrations/plugins/idempotent_statements'
 require 'safe-pg-migrations/plugins/useless_statements_logger'
+require 'safe-pg-migrations/plugins/legacy_active_record_support'
 
 module SafePgMigrations
   # Order matters: the bottom-most plugin will have precedence
   PLUGINS = [
     BlockingActivityLogger,
-    IdemPotentStatements,
+    IdempotentStatements,
     StatementRetrier,
     StatementInsurer,
     UselessStatementsLogger,
+    LegacyActiveRecordSupport,
   ].freeze
 
   class << self
@@ -85,7 +87,16 @@ module SafePgMigrations
       true
     end
 
-    SAFE_METHODS = %i[execute add_column add_index add_reference add_belongs_to change_column_null].freeze
+    SAFE_METHODS = %i[
+      execute
+      add_column
+      add_index
+      add_reference
+      add_belongs_to
+      change_column_null
+      add_foreign_key
+    ].freeze
+
     SAFE_METHODS.each do |method|
       define_method method do |*args|
         return super(*args) unless respond_to?(:safety_assured)

data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  module BlockingActivityLogger
+  module BlockingActivityLogger # rubocop:disable Metrics/ModuleLength
     FILTERED_COLUMNS = %w[
       blocked_activity.xact_start
       blocked_locks.locktype
@@ -113,6 +113,7 @@ module SafePgMigrations
     end
 
     def format_start_time(start_time, reference_time = Time.now)
+      start_time = Time.parse(start_time) unless start_time.is_a? Time
       duration = (reference_time - start_time).round
       "transaction started #{duration} #{'second'.pluralize(duration)} ago"
     end

data/lib/safe-pg-migrations/plugins/{idem_potent_statements.rb → idempotent_statements.rb}

@@ -1,15 +1,23 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  module IdemPotentStatements
+  module IdempotentStatements
     ruby2_keywords def add_index(table_name, column_name, *args)
       options = args.last.is_a?(Hash) ? args.last : {}
-      index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
-      return super unless index_name_exists?(table_name, index_name)
 
-      return if index_valid?(index_name)
+      index_definition = index_definition(table_name, column_name, **options)
 
-      remove_index(table_name, name: index_name)
+      return super unless index_name_exists?(index_definition.table, index_definition.name)
+
+      if index_valid?(index_definition.name)
+        SafePgMigrations.say(
+          "/!\\ Index '#{index_definition.name}' already exists in '#{table_name}'. Skipping statement.",
+          true
+        )
+        return
+      end
+
+      remove_index(table_name, name: index_definition.name)
       super
     end
 
@@ -62,6 +70,13 @@ module SafePgMigrations
       end
     end
 
+    protected
+
+    def index_definition(table_name, column_name, **options)
+      index_definition, = add_index_options(table_name, column_name, **options)
+      index_definition
+    end
+
     private
 
     def index_valid?(index_name)

data/lib/safe-pg-migrations/plugins/legacy_active_record_support.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  module LegacyActiveRecordSupport
+    ruby2_keywords def validate_foreign_key(*args)
+      return super(*args) if satisfied? '>=6.0.0'
+
+      from_table, to_table, options = args
+      super(from_table, to_table || options)
+    end
+
+    ruby2_keywords def foreign_key_exists?(*args)
+      return super(*args) if satisfied? '>=6.0.0'
+
+      from_table, to_table, options = args
+      super(from_table, to_table || options)
+    end
+
+    protected
+
+    IndexDefinition = Struct.new(:table, :name)
+
+    def index_definition(table_name, column_name, **options)
+      return super(table_name, column_name, **options) if satisfied? '>=6.1.0'
+
+      index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
+      validate_index_length!(table_name, index_name, options.fetch(:internal, false))
+
+      IndexDefinition.new(table_name, index_name)
+    end
+
+    private
+
+    def satisfied?(version)
+      Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/statement_insurer.rb

@@ -1,7 +1,7 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  module StatementInsurer # rubocop:disable Metrics/ModuleLength
+  module StatementInsurer
     PG_11_VERSION_NUM = 110_000
 
     %i[change_column_null change_column].each do |method|
@@ -85,20 +85,11 @@ module SafePgMigrations
     end
 
     def backfill_column_default(table_name, column_name)
-      quoted_table_name = quote_table_name(table_name)
+      model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
       quoted_column_name = quote_column_name(column_name)
-      primary_key_offset = 0
-      loop do
-        ids = query_values <<~SQL.squish
-          SELECT id FROM #{quoted_table_name} WHERE id > #{primary_key_offset}
-          ORDER BY id LIMIT #{SafePgMigrations.config.batch_size}
-        SQL
-        break if ids.empty?
-
-        primary_key_offset = ids.last
-        execute <<~SQL.squish
-          UPDATE #{quoted_table_name} SET #{quoted_column_name} = DEFAULT WHERE id IN (#{ids.join(',')})
-        SQL
+
+      model.in_batches(of: SafePgMigrations.config.batch_size).each do |relation|
+        relation.update_all("#{quoted_column_name} = DEFAULT")
       end
     end
 

data/lib/safe-pg-migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  VERSION = '1.3.0'
+  VERSION = '1.4.2'
 end

metadata

@@ -1,15 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: safe-pg-migrations
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.4.2
 platform: ruby
 authors:
 - Matthieu Prat
 - Romain Choquet
+- Thomas Hareau
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2021-06-28 00:00:00.000000000 Z
+date: 2022-03-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -53,120 +54,8 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.0.4
-- !ruby/object:Gem::Dependency
-  name: bundler
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: minitest
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: mocha
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: pg
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: pry
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: pry-coolline
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rake
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-- !ruby/object:Gem::Dependency
-  name: rubocop
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :development
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 description: Make your PG migrations safe.
-email: matthieuprat@gmail.com
+email:
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -177,7 +66,8 @@ files:
 - lib/safe-pg-migrations/base.rb
 - lib/safe-pg-migrations/configuration.rb
 - lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
-- lib/safe-pg-migrations/plugins/idem_potent_statements.rb
+- lib/safe-pg-migrations/plugins/idempotent_statements.rb
+- lib/safe-pg-migrations/plugins/legacy_active_record_support.rb
 - lib/safe-pg-migrations/plugins/statement_insurer.rb
 - lib/safe-pg-migrations/plugins/statement_retrier.rb
 - lib/safe-pg-migrations/plugins/useless_statements_logger.rb
@@ -187,7 +77,12 @@ files:
 homepage: https://github.com/doctolib/safe-pg-migrations
 licenses:
 - MIT
-metadata: {}
+metadata:
+  bug_tracker_uri: https://github.com/doctolib/safe-pg-migrations/issues
+  homepage_uri: https://github.com/doctolib/safe-pg-migrations#safe-pg-migrations
+  mailing_list_uri: https://doctolib.engineering/engineering-news-ruby-rails-react
+  source_code_uri: https://github.com/doctolib/safe-pg-migrations
+  contributors_uri: https://github.com/doctolib/safe-pg-migrations/graphs/contributors
 post_install_message:
 rdoc_options: []
 require_paths: