safe-pg-migrations 0.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: abc17a43f436f429bc2f8983a45b1a61af67a4f8
-  data.tar.gz: f1c0fdc3261c34fe99218e935e502c639d36f4f1
+SHA256:
+  metadata.gz: d806334c473708774d2180dbe91d2483b49f0d27b0a223a2973bc95d986d4030
+  data.tar.gz: 9b5c2d329d5cc1bd12c80944420e20eb5454e5b0a0f7fda2d6ec3fc3c3a5e042
 SHA512:
-  metadata.gz: 3bc4cb7434a1b9059efbc0c69a7a3f73a329b478023db08b0f53d5345ad4346798be592b581e0db0413a574c6c72b2369dcad98402ae4f89a88411525967211d
-  data.tar.gz: 670b3ef74768301708b85e904defe0566dedfb04762ea85fb2cccf0dccb666a614719430e36e6204e399a7861c59494843750fb6321add67144a49c3fc941846
+  metadata.gz: 6786e68cce9dbd91a4633c80526a1fd63f6fbfa335d6e2b53d29b71ba0b566f9bb32169745bb13f89a059b82b0c0b21585a9bffdb9ebc751aa0e4a467f0f7934
+  data.tar.gz: 299a69f1d9413279fd61908a7d8f6eb65bd4c9912abcc1101ca8acbfee39c40d33086efa28a9f2133c0679c1ad9cdbbfe3ce84338f2175ad882b7d1ace707654

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2018 Doctolib
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,256 @@
+# safe-pg-migrations
+
+ActiveRecord migrations for Postgres made safe.
+
+## Requirements
+
+- Ruby 2.5+
+- Rails 5.2+
+- PostgreSQL 9.3+
+
+## Usage
+
+Just drop this line in your Gemfile:
+
+```rb
+gem 'safe-pg-migrations'
+```
+
+## Example
+
+Consider the following migration:
+
+```rb
+class AddAdminToUsers < ActiveRecord::Migration[5.2]
+  def change
+    add_column :users, :admin, :boolean, default: false, null: false
+  end
+end
+```
+
+If the `users` table is large, running this migration on a live Postgres 9 database will likely cause downtime. **Safe PG Migrations** hooks into Active Record so that the following gets executed instead:
+
+```rb
+class AddAdminToUsers < ActiveRecord::Migration[5.2]
+  # Do not wrap the migration in a transaction so that locks are held for a shorter time.
+  disable_ddl_transaction!
+
+  def change
+    # Lower Postgres' lock timeout to avoid statement queueing. Acts like a seatbelt.
+    execute "SET lock_timeout TO '5s'" # The lock_timeout duration is customizable.
+
+    # Add the column without the default value and the not-null constraint.
+    add_column :users, :admin, :boolean
+
+    # Set the column's default value.
+    change_column_default :users, :admin, false
+
+    # Backfill the column in batches.
+    User.in_batches.update_all(admin: false)
+
+    # Add the not-null constraint. Beforehand, set a short statement timeout so that
+    # Postgres does not spend too much time performing the full table scan to verify
+    # the column contains no nulls.
+    execute "SET statement_timeout TO '5s'"
+    change_column_null :users, :admin, false
+  end
+end
+```
+
+Under the hood, **Safe PG Migrations** patches `ActiveRecord::Migration` and extends `ActiveRecord::Base.connection` to make potentially dangerous methods—like `add_column`—safe.
+
+## Motivation
+
+Writing a safe migration can be daunting. Numerous articles have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of Postgres, remembering all the subtleties of explicit locking is not a piece of cake.
+
+Active Record means developers don't have to be proficient in SQL to interact with a database. In the same way, **Safe PG Migrations** was created so that developers don't have to understand the ins and outs of Postgres to write a safe migration.
+
+## Feature set
+
+<details><summary>Lock timeout</summary>
+
+Most DDL operations (e.g. adding a column, removing a column or adding a default value to a column) take an `ACCESS EXCLUSIVE` lock on the table they are altering. While these operations wait to acquire their lock, other statements are blocked. Before running a migration, **Safe PG Migrations** sets a short lock timeout (default to 5 seconds) so that statements are not blocked for too long.
+
+See [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html) and [Migrations and Long Transactions](https://www.fin.com/post/2018/1/migrations-and-long-transactions) for detailed explanations of the matter.
+</details>
+
+<details><summary>Statement timeout</summary>
+
+Adding a foreign key or a not-null constraint can take a lot of time on a large table. The problem is that those operations take `ACCESS EXCLUSIVE` locks. We clearly don't want them to hold these locks for too long. Thus, **Safe PG Migrations** runs them with a short statement timeout (default to 5 seconds).
+
+See [Zero-downtime Postgres migrations - the hard parts](https://gocardless.com/blog/zero-downtime-postgres-migrations-the-hard-parts/) for a detailed explanation on the subject.
+</details>
+
+<details><summary>Prevent wrapping migrations in transaction</summary>
+
+When **Safe PG Migrations** is used, migrations are not wrapped in a transaction. This is for several reasons:
+
+- We want to release locks as soon as possible.
+- In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
+- In order to add an index concurrently, we have to be outside a transaction.
+
+Note that if a migration fails, it won't be rollbacked. This can result in migrations being partially applied. In that case, they need to be manually reverted.
+
+</details>
+
+<details>
+<summary>Safe <code>add_column</code></summary>
+
+**Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
+
+Beware though, when adding a volatile default value: 
+```ruby
+add_column :users, :created_at, default: 'clock_timestamp()'
+```
+PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
+
+<blockquote>
+
+**Note: Pre-postgre 11**
+Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
+
+**Safe PG Migrations** makes it safe by:
+
+1.  Adding the column without the default value and the not null constraint,
+2.  Then set the default value on the column,
+3.  Then backfilling the column,
+4.  And then adding the not null constraint with a short statement timeout.
+
+Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
+
+</blockquote>    
+
+</details>
+
+<details><summary>Safe <code>add_index</code> and <code>remove_index</code></summary>
+
+Creating an index requires a `SHARE` lock on the target table which blocks all write on the table while the index is created (which can take some time on a large table). This is usually not practical in a live environment. Thus, **Safe PG Migrations** ensures indexes are created concurrently.
+
+As `CREATE INDEX CONCURRENTLY` and `DROP INDEX CONCURRENTLY` are non-blocking operations (ie: read/write operations on the table are still possible), **Safe PG Migrations** sets a lock timeout to 30 seconds for those 2 specific statements.
+
+If you still get lock timeout while adding / removing indexes, it might be for one of those reasons:
+
+- Long-running queries are active on the table. To create / remove an index, PG needs to wait for the queries that are actually running to finish before starting the index creation / removal. The blocking activity logger might help you to pinpoint the culprit queries.
+- A vacuum / autovacuum is running on the table, holding a ShareUpdateExclusiveLock, you are most likely out of luck for the current migration, but you may try to [optimize your autovacuums settings](https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in-postgresql-and-autovacuum-internals/).
+
+</details>
+
+<details><summary>safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
+
+Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
+
+Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus safe-pg-migrations ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps: 
+
+1. adding the constraint *without validation*, will not validate existing rows;
+2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
+
+</details>
+
+<details><summary>Retry after lock timeout</summary>
+
+When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
+</details>
+
+<details><summary>Blocking activity logging</summary>
+
+If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
+</details>
+
+<details><summary>Verbose SQL logging</summary>
+
+For any operation, **Safe PG Migrations** can output the performed SQL queries. This feature is enabled by default in a production Rails environment. If you want to explicit enable it, for example in your development environment you can use:
+```bash
+export SAFE_PG_MIGRATIONS_VERBOSE=1
+```
+
+Instead of the traditional output:
+```ruby
+add_index :users, :age
+
+== 20191215132355 SampleIndex: migrating ======================================
+-- add_index(:users, :age)
+   -> add_index("users", :age, {:algorithm=>:concurrently})
+   -> 0.0175s
+== 20191215132355 SampleIndex: migrated (0.0200s) =============================
+```
+**Safe PG Migrations** will output the following logs:
+```ruby
+add_index :users, :age
+
+== 20191215132355 SampleIndex: migrating ======================================
+   (0.3ms)  SHOW lock_timeout
+   (0.3ms)  SET lock_timeout TO '5s'
+-- add_index(:users, :age)
+   -> add_index("users", :age, {:algorithm=>:concurrently})
+   (0.3ms)  SHOW statement_timeout
+   (0.3ms)  SET statement_timeout TO 0
+   (0.3ms)  SHOW lock_timeout
+   (0.3ms)  SET lock_timeout TO '30s'
+   (3.5ms)  CREATE INDEX CONCURRENTLY "index_users_on_age" ON "users"  ("age")
+   (0.3ms)  SET lock_timeout TO '5s'
+   (0.2ms)  SET statement_timeout TO '1min'
+   -> 0.0093s
+   (0.2ms)  SET lock_timeout TO '0'
+== 20191215132355 SampleIndex: migrated (0.0114s) =============================
+```
+So you can actually check that the `CREATE INDEX` statement will be performed concurrently, without any statement timeout and with a lock timeout of 30 seconds.
+
+*Nb: The `SHOW` statements are used by **Safe PG Migrations** to query settings for their original values in order to restore them after the work is done*
+
+</details>
+
+## Configuration
+
+**Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
+
+```ruby
+SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
+
+SafePgMigrations.config.index_lock_timeout = 30.seconds # Lock timeout used for CREATE / DROP INDEX
+
+SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be smaller than safe_timeout and index_lock_timeout
+
+SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
+
+SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
+
+SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
+```
+
+## Runnings tests
+
+```bash
+bundle
+psql -h localhost -U postgres -c 'CREATE DATABASE safe_pg_migrations_test'
+rake test
+```
+
+## Authors
+
+- [Matthieu Prat](https://github.com/matthieuprat)
+- [Romain Choquet](https://github.com/rchoquet)
+- [Paul-Etienne Coisne](https://github.com/coisnepe)
+
+## License
+
+[MIT](https://github.com/doctolib/safe-pg-migrations/blob/master/LICENSE) © [Doctolib](https://github.com/doctolib/)
+
+## Additional resources
+
+Alternatives:
+
+- https://github.com/gocardless/activerecord-safer_migrations
+- https://github.com/ankane/strong_migrations
+- https://github.com/LendingHome/zero_downtime_migrations
+
+Interesting reads:
+
+- [When Postgres blocks: 7 tips for dealing with locks](https://www.citusdata.com/blog/2018/02/22/seven-tips-for-dealing-with-postgres-locks/)
+- [Migrations and Long Transactions in Postgres
+](https://www.fin.com/post/2018/1/migrations-and-long-transactions)
+- [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html)
+- [Adding a NOT NULL CONSTRAINT on PG Faster with Minimal Locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c)
+- [Adding columns with default values to really large tables in Postgres + Rails](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/)
+- [Rails migrations with no downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
+- [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
+- [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)

data/lib/safe-pg-migrations.rb

@@ -1,5 +1,3 @@
-class SafePgMigrations
-  def self.hello
-    puts 'Hello world!'
-  end
-end
+# frozen_string_literal: true
+
+require 'safe-pg-migrations/railtie' if defined?(Rails)

data/lib/safe-pg-migrations/base.rb

@@ -0,0 +1,93 @@
+# frozen_string_literal: true
+
+require 'safe-pg-migrations/configuration'
+require 'safe-pg-migrations/plugins/verbose_sql_logger'
+require 'safe-pg-migrations/plugins/blocking_activity_logger'
+require 'safe-pg-migrations/plugins/statement_insurer'
+require 'safe-pg-migrations/plugins/statement_retrier'
+require 'safe-pg-migrations/plugins/idem_potent_statements'
+
+module SafePgMigrations
+  # Order matters: the bottom-most plugin will have precedence
+  PLUGINS = [
+    BlockingActivityLogger,
+    IdemPotentStatements,
+    StatementRetrier,
+    StatementInsurer,
+  ].freeze
+
+  class << self
+    attr_reader :current_migration, :pg_version_num
+
+    def setup_and_teardown(migration, connection)
+      @pg_version_num = get_pg_version_num(connection)
+      @alternate_connection = nil
+      @current_migration = migration
+      stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
+      PLUGINS.each { |plugin| connection.extend(plugin) }
+
+      connection.with_setting(:lock_timeout, SafePgMigrations.config.pg_safe_timeout) { yield }
+    ensure
+      close_alternate_connection
+      @current_migration = nil
+      stdout_sql_logger&.teardown
+    end
+
+    def get_pg_version_num(connection)
+      connection.query_value('SHOW server_version_num').to_i
+    end
+
+    def alternate_connection
+      @alternate_connection ||= ActiveRecord::Base.connection_pool.send(:new_connection)
+    end
+
+    def close_alternate_connection
+      return unless @alternate_connection
+
+      @alternate_connection.disconnect!
+      @alternate_connection = nil
+    end
+
+    def say(*args)
+      return unless current_migration
+
+      current_migration.say(*args)
+    end
+
+    def say_method_call(method, *args)
+      say "#{method}(#{args.map(&:inspect) * ', '})", true
+    end
+
+    def verbose?
+      return ENV['SAFE_PG_MIGRATIONS_VERBOSE'] == '1' if ENV['SAFE_PG_MIGRATIONS_VERBOSE']
+      return Rails.env.production? if defined?(Rails)
+
+      false
+    end
+
+    def config
+      @config ||= Configuration.new
+    end
+  end
+
+  module Migration
+    def exec_migration(connection, direction)
+      SafePgMigrations.setup_and_teardown(self, connection) do
+        super(connection, direction)
+      end
+    end
+
+    def disable_ddl_transaction
+      true
+    end
+
+    SAFE_METHODS = %i[execute add_column add_index add_reference add_belongs_to change_column_null].freeze
+    SAFE_METHODS.each do |method|
+      define_method method do |*args|
+        return super(*args) unless respond_to?(:safety_assured)
+
+        safety_assured { super(*args) }
+      end
+    end
+  end
+end

data/lib/safe-pg-migrations/configuration.rb

@@ -0,0 +1,36 @@
+# frozen_string_literal: true
+
+require 'active_support/core_ext/numeric/time'
+
+module SafePgMigrations
+  class Configuration
+    attr_accessor :safe_timeout
+    attr_accessor :index_lock_timeout
+    attr_accessor :blocking_activity_logger_margin
+    attr_accessor :batch_size
+    attr_accessor :retry_delay
+    attr_accessor :max_tries
+
+    def initialize
+      self.safe_timeout = 5.seconds
+      self.index_lock_timeout = 30.seconds
+      self.blocking_activity_logger_margin = 1.second
+      self.batch_size = 1000
+      self.retry_delay = 1.minute
+      self.max_tries = 5
+    end
+
+    def pg_safe_timeout
+      pg_duration(safe_timeout)
+    end
+
+    def pg_index_lock_timeout
+      pg_duration(index_lock_timeout)
+    end
+
+    def pg_duration(duration)
+      value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
+      "#{value}#{unit}"
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb

@@ -0,0 +1,90 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  module BlockingActivityLogger
+    SELECT_BLOCKING_QUERIES_SQL = <<~SQL.squish
+      SELECT blocking_activity.query
+      FROM pg_catalog.pg_locks           blocked_locks
+      JOIN pg_catalog.pg_stat_activity   blocked_activity
+        ON blocked_activity.pid = blocked_locks.pid
+      JOIN pg_catalog.pg_locks           blocking_locks
+        ON blocking_locks.locktype = blocked_locks.locktype
+        AND blocking_locks.DATABASE      IS NOT DISTINCT FROM blocked_locks.DATABASE
+        AND blocking_locks.relation      IS NOT DISTINCT FROM blocked_locks.relation
+        AND blocking_locks.page          IS NOT DISTINCT FROM blocked_locks.page
+        AND blocking_locks.tuple         IS NOT DISTINCT FROM blocked_locks.tuple
+        AND blocking_locks.virtualxid    IS NOT DISTINCT FROM blocked_locks.virtualxid
+        AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
+        AND blocking_locks.classid       IS NOT DISTINCT FROM blocked_locks.classid
+        AND blocking_locks.objid         IS NOT DISTINCT FROM blocked_locks.objid
+        AND blocking_locks.objsubid      IS NOT DISTINCT FROM blocked_locks.objsubid
+        AND blocking_locks.pid != blocked_locks.pid
+      JOIN pg_catalog.pg_stat_activity   blocking_activity
+        ON blocking_activity.pid = blocking_locks.pid
+      WHERE blocked_locks.pid = %d
+    SQL
+
+    %i[
+      add_column remove_column add_foreign_key remove_foreign_key change_column_default
+      change_column_null create_table add_index remove_index
+    ].each do |method|
+      define_method method do |*args, &block|
+        log_blocking_queries(method) { super(*args, &block) }
+      end
+    end
+
+    private
+
+    def delay_before_logging(method)
+      timeout_delay =
+        if %i[add_index remove_index].include?(method)
+          SafePgMigrations.config.index_lock_timeout
+        else
+          SafePgMigrations.config.safe_timeout
+        end
+
+      timeout_delay - SafePgMigrations.config.blocking_activity_logger_margin
+    end
+
+    def log_blocking_queries(method)
+      blocking_queries_retriever_thread =
+        Thread.new do
+          sleep delay_before_logging(method)
+          SafePgMigrations.alternate_connection.query_values(SELECT_BLOCKING_QUERIES_SQL % raw_connection.backend_pid)
+        end
+
+      yield
+
+      blocking_queries_retriever_thread.kill
+    rescue ActiveRecord::LockWaitTimeout
+      SafePgMigrations.say 'Lock timeout.', true
+      queries =
+        begin
+          blocking_queries_retriever_thread.value
+        rescue StandardError => e
+          SafePgMigrations.say("Error while retrieving blocking queries: #{e}", true)
+          nil
+        end
+
+      raise if queries.nil?
+
+      if queries.empty?
+        SafePgMigrations.say 'Could not find any blocking query.', true
+      else
+        SafePgMigrations.say(
+          "Statement was being blocked by the following #{'query'.pluralize(queries.size)}:", true
+        )
+        SafePgMigrations.say '', true
+        queries.each { |query| SafePgMigrations.say "  #{query}", true }
+        SafePgMigrations.say(
+          'Beware, some of those queries might run in a transaction. In this case the locking query might be '\
+          'located elsewhere in the transaction',
+          true
+        )
+        SafePgMigrations.say '', true
+      end
+
+      raise
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/idem_potent_statements.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  module IdemPotentStatements
+    def add_index(table_name, column_name, **options)
+      index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
+      return super unless index_name_exists?(table_name, index_name)
+
+      return if index_valid?(index_name)
+
+      remove_index(table_name, name: index_name)
+      super
+    end
+
+    def add_column(table_name, column_name, type, options = {})
+      return super unless column_exists?(table_name, column_name)
+
+      SafePgMigrations.say("/!\\ Column '#{column_name}' already exists in '#{table_name}'. Skipping statement.", true)
+    end
+
+    def remove_column(table_name, column_name, type = nil, options = {})
+      return super if column_exists?(table_name, column_name)
+
+      SafePgMigrations.say("/!\\ Column '#{column_name}' not found on table '#{table_name}'. Skipping statement.", true)
+    end
+
+    def remove_index(table_name, options = {})
+      index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, options)
+
+      return super if index_name_exists?(table_name, index_name)
+
+      SafePgMigrations.say("/!\\ Index '#{index_name}' not found on table '#{table_name}'. Skipping statement.", true)
+    end
+
+    def add_foreign_key(from_table, to_table, **options)
+      options_or_to_table = options.slice(:name, :column).presence || to_table
+      return super unless foreign_key_exists?(from_table, options_or_to_table)
+
+      SafePgMigrations.say(
+        "/!\\ Foreign key '#{from_table}' -> '#{to_table}' already exists. Skipping statement.",
+        true
+      )
+    end
+
+    def create_table(table_name, comment: nil, **options)
+      return super unless table_exists?(table_name)
+
+      SafePgMigrations.say(
+        "/!\\ Table '#{table_name}' already exists. Skipping statement.",
+        true
+      )
+    end
+
+    private
+
+    def index_valid?(index_name)
+      query_value <<~SQL.squish
+        SELECT indisvalid
+        FROM pg_index i
+        JOIN pg_class c
+          ON i.indexrelid = c.oid
+        WHERE c.relname = '#{index_name}';
+      SQL
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/statement_insurer.rb

@@ -0,0 +1,110 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  module StatementInsurer
+    PG_11_VERSION_NUM = 110_000
+
+    %i[change_column_null change_column create_table].each do |method|
+      define_method method do |*args, &block|
+        with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
+      end
+    end
+
+    def add_column(table_name, column_name, type, **options) # rubocop:disable Metrics/CyclomaticComplexity
+      need_default_value_backfill = SafePgMigrations.pg_version_num < PG_11_VERSION_NUM
+
+      default = options.delete(:default) if need_default_value_backfill
+      null = options.delete(:null)
+
+      if !default.nil? || null == false
+        SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, options)
+      end
+
+      super
+
+      if need_default_value_backfill && !default.nil?
+        SafePgMigrations.say_method_call(:change_column_default, table_name, column_name, default)
+        change_column_default(table_name, column_name, default)
+
+        SafePgMigrations.say_method_call(:backfill_column_default, table_name, column_name)
+        backfill_column_default(table_name, column_name)
+      end
+
+      if null == false # rubocop:disable Style/GuardClause
+        SafePgMigrations.say_method_call(:change_column_null, table_name, column_name, null)
+        change_column_null(table_name, column_name, null)
+      end
+    end
+
+    def add_foreign_key(from_table, to_table, **options)
+      validate_present = options.key? :validate
+      options[:validate] = false unless validate_present
+
+      with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super }
+
+      options_or_to_table = options.slice(:name, :column).presence || to_table
+      without_statement_timeout { validate_foreign_key from_table, options_or_to_table } unless validate_present
+    end
+
+    def add_index(table_name, column_name, **options)
+      options[:algorithm] = :concurrently
+      SafePgMigrations.say_method_call(:add_index, table_name, column_name, options)
+
+      with_index_timeouts { super }
+    end
+
+    def remove_index(table_name, options = {})
+      options = { column: options } unless options.is_a?(Hash)
+      options[:algorithm] = :concurrently
+      SafePgMigrations.say_method_call(:remove_index, table_name, options)
+
+      with_index_timeouts { super }
+    end
+
+    def backfill_column_default(table_name, column_name)
+      quoted_table_name = quote_table_name(table_name)
+      quoted_column_name = quote_column_name(column_name)
+      primary_key_offset = 0
+      loop do
+        ids = query_values <<~SQL.squish
+          SELECT id FROM #{quoted_table_name} WHERE id > #{primary_key_offset}
+          ORDER BY id LIMIT #{SafePgMigrations.config.batch_size}
+        SQL
+        break if ids.empty?
+
+        primary_key_offset = ids.last
+        execute <<~SQL.squish
+          UPDATE #{quoted_table_name} SET #{quoted_column_name} = DEFAULT WHERE id IN (#{ids.join(',')})
+        SQL
+      end
+    end
+
+    def with_setting(key, value)
+      old_value = query_value("SHOW #{key}")
+      execute("SET #{key} TO #{quote(value)}")
+      begin
+        yield
+      ensure
+        begin
+          execute("SET #{key} TO #{quote(old_value)}")
+        rescue ActiveRecord::StatementInvalid => e
+          # Swallow `PG::InFailedSqlTransaction` exceptions so as to keep the
+          # original exception (if any).
+          raise unless e.cause.is_a?(PG::InFailedSqlTransaction)
+        end
+      end
+    end
+
+    def without_statement_timeout
+      with_setting(:statement_timeout, 0) { yield }
+    end
+
+    def with_index_timeouts
+      without_statement_timeout do
+        with_setting(:lock_timeout, SafePgMigrations.config.pg_index_lock_timeout) do
+          yield
+        end
+      end
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/statement_retrier.rb

@@ -0,0 +1,35 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  module StatementRetrier
+    RETRIABLE_SCHEMA_STATEMENTS = %i[
+      add_column add_foreign_key remove_foreign_key change_column_default
+      change_column_null add_index remove_index remove_column
+    ].freeze
+
+    RETRIABLE_SCHEMA_STATEMENTS.each do |method|
+      define_method method do |*args, &block|
+        retry_if_lock_timeout { super(*args, &block) }
+      end
+    end
+
+    private
+
+    def retry_if_lock_timeout
+      remaining_tries = SafePgMigrations.config.max_tries
+      begin
+        remaining_tries -= 1
+        yield
+      rescue ActiveRecord::LockWaitTimeout
+        raise if transaction_open? # Retrying is useless if we're inside a transaction.
+        raise unless remaining_tries > 0
+
+        retry_delay = SafePgMigrations.config.retry_delay
+        SafePgMigrations.say "Retrying in #{retry_delay} seconds...", true
+        sleep retry_delay
+        SafePgMigrations.say 'Retrying now.', true
+        retry
+      end
+    end
+  end
+end

data/lib/safe-pg-migrations/plugins/verbose_sql_logger.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  class VerboseSqlLogger
+    def setup
+      @activerecord_logger_was = ActiveRecord::Base.logger
+      @verbose_query_logs_was = ActiveRecord::Base.verbose_query_logs
+      @colorize_logging_was = ActiveRecord::LogSubscriber.colorize_logging
+
+      disable_marginalia if defined?(Marginalia)
+
+      stdout_logger = Logger.new($stdout, formatter: ->(_severity, _time, _progname, query) { "#{query}\n" })
+      ActiveRecord::Base.logger = stdout_logger
+      ActiveRecord::LogSubscriber.colorize_logging = colorize_logging?
+      # Do not output caller method, we know it is coming from the migration
+      ActiveRecord::Base.verbose_query_logs = false
+      self
+    end
+
+    def teardown
+      ActiveRecord::Base.verbose_query_logs = @verbose_query_logs_was
+      ActiveRecord::LogSubscriber.colorize_logging = @colorize_logging_was
+      ActiveRecord::Base.logger = @activerecord_logger_was
+      enable_marginalia if defined?(Marginalia)
+    end
+
+    private
+
+    def colorize_logging?
+      defined?(Rails) && Rails.env.development?
+    end
+
+    # Marginalia annotations will most likely pollute the output
+    def disable_marginalia
+      @marginalia_components_were = Marginalia::Comment.components
+      Marginalia::Comment.components = []
+    end
+
+    def enable_marginalia
+      Marginalia::Comment.components = @marginalia_components_were
+    end
+  end
+end

data/lib/safe-pg-migrations/railtie.rb

@@ -0,0 +1,13 @@
+# frozen_string_literal: true
+
+require 'safe-pg-migrations/base'
+
+module SafePgMigrations
+  class Railtie < Rails::Railtie
+    initializer 'safe_pg_migrations.insert_into_active_record' do
+      ActiveSupport.on_load :active_record do
+        ActiveRecord::Migration.prepend(SafePgMigrations::Migration)
+      end
+    end
+  end
+end

data/lib/safe-pg-migrations/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  VERSION = '1.1.0'
+end

metadata

@@ -1,28 +1,179 @@
 --- !ruby/object:Gem::Specification
 name: safe-pg-migrations
 version: !ruby/object:Gem::Version
-  version: 0.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Matthieu Prat
 - Romain Choquet
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2018-10-08 00:00:00.000000000 Z
-dependencies: []
+date: 2020-11-03 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: activerecord
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.2'
+- !ruby/object:Gem::Dependency
+  name: activesupport
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '5.2'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pg
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: pry-coolline
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Make your PG migrations safe.
 email: matthieuprat@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
+- LICENSE
+- README.md
 - lib/safe-pg-migrations.rb
-homepage: http://rubygems.org/doctolib/safe-pg-migrations
+- lib/safe-pg-migrations/base.rb
+- lib/safe-pg-migrations/configuration.rb
+- lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
+- lib/safe-pg-migrations/plugins/idem_potent_statements.rb
+- lib/safe-pg-migrations/plugins/statement_insurer.rb
+- lib/safe-pg-migrations/plugins/statement_retrier.rb
+- lib/safe-pg-migrations/plugins/verbose_sql_logger.rb
+- lib/safe-pg-migrations/railtie.rb
+- lib/safe-pg-migrations/version.rb
+homepage: https://github.com/doctolib/safe-pg-migrations
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -30,16 +181,16 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.13
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.7.3
+signing_key:
 specification_version: 4
 summary: Make your PG migrations safe.
 test_files: []