safe-pg-migrations 0.0.3 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 064f4f95d22160ab489fbe40a4af338110144c74
-  data.tar.gz: 6c7e864e8b88251d9c3defb93c986af186e19a97
+  metadata.gz: d3e47cc5a8440b59b3e25dc7a5e99a9212c4709c
+  data.tar.gz: 7347fdd053c1edd34b3ec58d9021b126ec794eb3
 SHA512:
-  metadata.gz: 68157b6bbaf2f8b5a363f0796e7e98a5cfa19ccd6e6bdc2981e4b523dbc24b63fda023807c0d13964da544f1323d186ce180265e0f3f9e66dd6faa193d601baa
-  data.tar.gz: 7b0cc78b1552683538548f6ab57b05154f80009f022073c4c993d4226f1cda6f91576f0a090493b7f5f4dc61073ad6aaf1d63100e78ec984ee25c89d2b7770f3
+  metadata.gz: 74058e22a0e55ae44e29b8e74096bf0b4bf6b7c83a7fafce8c6aa3595c50bafec2d93c6ce502d304f89b44e236a3f3770fac91ff7e9bb449a3289cca0a06ab89
+  data.tar.gz: 7a38389555cb2cd8c0271882389aed8e27b1d22ef7cb16976c729041ef315803ad7de0a71400f478d50c7c741e1b10a5a37109940de5a5e7a8809800eecdff25

data/README.md

@@ -28,7 +28,7 @@ class AddAdminToUsers < ActiveRecord::Migration[5.2]
 end
 ```
 
-If the `users` table is large, running this migration on a live Postgres database will likely cause downtime. **Safe PG Migrations** hooks into Active Record so that the following gets executed instead:
+If the `users` table is large, running this migration on a live Postgres 9 database will likely cause downtime. **Safe PG Migrations** hooks into Active Record so that the following gets executed instead:
 
 ```rb
 class AddAdminToUsers < ActiveRecord::Migration[5.2]
@@ -69,19 +69,19 @@ Active Record means developers don't have to be proficient in SQL to interact wi
 
 ### Lock timeout
 
-Most DDL operations (e.g. adding a column, removing a column or adding a default value to a column) take an `ACCESS EXCLUSIVE` lock on the table they are altering. While these operations wait to acquire their lock, other statements are blocked. Before running a migration, **Safe PG Migrations** sets a short lock timeout so that statements are not blocked for too long.
+Most DDL operations (e.g. adding a column, removing a column or adding a default value to a column) take an `ACCESS EXCLUSIVE` lock on the table they are altering. While these operations wait to acquire their lock, other statements are blocked. Before running a migration, **Safe PG Migrations** sets a short lock timeout (default to 5 seconds) so that statements are not blocked for too long.
 
 See [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html) and [Migrations and Long Transactions](https://www.fin.com/post/2018/1/migrations-and-long-transactions) for detailed explanations of the matter.
 
 ### Statement timeout
 
-Adding a foreign key or a not-null constraint can take a lot of time on a large table. The problem is that those operations take `ACCESS EXCLUSIVE` locks. We clearly don't want them to hold these locks for too long. Thus, **Safe PG Migrations** runs them with a short statement timeout.
+Adding a foreign key or a not-null constraint can take a lot of time on a large table. The problem is that those operations take `ACCESS EXCLUSIVE` locks. We clearly don't want them to hold these locks for too long. Thus, **Safe PG Migrations** runs them with a short statement timeout (default to 5 seconds).
 
 See [Zero-downtime Postgres migrations - the hard parts](https://gocardless.com/blog/zero-downtime-postgres-migrations-the-hard-parts/) for a detailed explanation on the subject.
 
 ### Prevent wrapping migrations in transaction
 
-When **Safe PG Migrations** is enabled (which is the case by default if `Rails.env.production?` is true), migrations are not wrapped in a transaction. This is for several reasons:
+When **Safe PG Migrations** is used, migrations are not wrapped in a transaction. This is for several reasons:
 
 - We want to release locks as soon as possible.
 - In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
@@ -91,6 +91,8 @@ Note that if a migration fails, it won't be rollbacked. This can result in migra
 
 ### Safe `add_column`
 
+#### Pre Postgres 11 behavior
+
 Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
 
 **Safe PG Migrations** makes it safe by:
@@ -102,18 +104,95 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
 
 Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
 
+#### Postgres 11 behavior
+
+**Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
+
+Beware though, when adding a volatile default value: 
+```ruby
+add_column :users, :created_at, default: 'clock_timestamp()'
+```
+PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
+
 ### Concurrent indexes
 
 Creating an index requires a `SHARE` lock on the target table which blocks all write on the table while the index is created (which can take some time on a large table). This is usually not practical in a live environment. Thus, **Safe PG Migrations** ensures indexes are created concurrently.
 
+As `CREATE INDEX CONCURRENTLY` and `DROP INDEX CONCURRENTLY` are non-blocking operations (ie: read/write operations on the table are still possible), **Safe PG Migrations** sets a lock timeout to 30 seconds for those 2 specific statements.
+
+If you still get lock timeout while adding / removing indexes, it might be for one of those reasons:
+
+- Long-running queries are active on the table. To create / remove an index, PG needs to wait for the queries that are actually running to finish before starting the index creation / removal. The blocking activity logger might help you to pinpoint the culprit queries.
+- A vacuum / autovacuum is running on the table, holding a ShareUpdateExclusiveLock, you are most likely out of luck for the current migration, but you may try to [optimize your autovacuums settings](https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in-postgresql-and-autovacuum-internals/).
+
+
 ### Retry after lock timeout
 
-When a statement fails with a lock timeout, **Safe PG Migrations** retries them (5 times max).
+When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
 
 ### Blocking activity logging
 
 If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
 
+### Verbose SQL logging
+
+For any operation, **Safe PG Migrations** can output the performed SQL queries. This feature is enabled by default in a production Rails environment. If you want to explicit enable it, for example in your development environment you can use:
+```bash
+export SAFE_PG_MIGRATIONS_VERBOSE=1
+```
+
+Instead of the traditional output:
+```ruby
+add_index :users, :age
+
+== 20191215132355 SampleIndex: migrating ======================================
+-- add_index(:users, :age)
+   -> add_index("users", :age, {:algorithm=>:concurrently})
+   -> 0.0175s
+== 20191215132355 SampleIndex: migrated (0.0200s) =============================
+```
+**Sage PG Migrations** will output the following logs:
+```ruby
+add_index :users, :age
+
+== 20191215132355 SampleIndex: migrating ======================================
+   (0.3ms)  SHOW lock_timeout
+   (0.3ms)  SET lock_timeout TO '5s'
+-- add_index(:users, :age)
+   -> add_index("users", :age, {:algorithm=>:concurrently})
+   (0.3ms)  SHOW statement_timeout
+   (0.3ms)  SET statement_timeout TO 0
+   (0.3ms)  SHOW lock_timeout
+   (0.3ms)  SET lock_timeout TO '30s'
+   (3.5ms)  CREATE INDEX CONCURRENTLY "index_users_on_age" ON "users"  ("age")
+   (0.3ms)  SET lock_timeout TO '5s'
+   (0.2ms)  SET statement_timeout TO '1min'
+   -> 0.0093s
+   (0.2ms)  SET lock_timeout TO '0'
+== 20191215132355 SampleIndex: migrated (0.0114s) =============================
+```
+So you can actually check that the `CREATE INDEX` statement will be performed concurrently, without any statement timeout and with a lock timeout of 30 seconds.
+
+*Nb: The `SHOW` statements are used by **Safe PG Migrations** to query settings for their original values in order to restore them after the work is done*
+
+## Configuration
+
+**Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
+
+```ruby
+SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
+
+SafePgMigrations.config.index_lock_timeout = 30.seconds # Lock timeout used for CREATE / DROP INDEX
+
+SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be smaller than safe_timeout and index_lock_timeout
+
+SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
+
+SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
+
+SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
+```
+
 ## Runnings tests
 
 ```bash
@@ -142,11 +221,12 @@ Alternatives:
 
 Interesting reads:
 
-- https://www.citusdata.com/blog/2018/02/22/seven-tips-for-dealing-with-postgres-locks/
-- https://www.fin.com/post/2018/1/migrations-and-long-transactions
-- http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html
-- https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c
-- https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/
-- https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/
-- https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/
-- https://blog.codeship.com/rails-migrations-zero-downtime/
+- [When Postgres blocks: 7 tips for dealing with locks](https://www.citusdata.com/blog/2018/02/22/seven-tips-for-dealing-with-postgres-locks/)
+- [Migrations and Long Transactions in Postgres
+](https://www.fin.com/post/2018/1/migrations-and-long-transactions)
+- [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html)
+- [Adding a NOT NULL CONSTRAINT on PG Faster with Minimal Locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c)
+- [Adding columns with default values to really large tables in Postgres + Rails](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/)
+- [Rails migrations with no downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
+- [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
+- [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)

data/lib/safe-pg-migrations/base.rb

@@ -1,6 +1,7 @@
 # frozen_string_literal: true
 
 require 'safe-pg-migrations/configuration'
+require 'safe-pg-migrations/plugins/verbose_sql_logger'
 require 'safe-pg-migrations/plugins/blocking_activity_logger'
 require 'safe-pg-migrations/plugins/statement_insurer'
 require 'safe-pg-migrations/plugins/statement_retrier'
@@ -16,17 +17,24 @@ module SafePgMigrations
   ].freeze
 
   class << self
-    attr_reader :current_migration
-    attr_accessor :enabled
+    attr_reader :current_migration, :pg_version_num
 
     def setup_and_teardown(migration, connection)
+      @pg_version_num = get_pg_version_num(connection)
       @alternate_connection = nil
       @current_migration = migration
+      stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
       PLUGINS.each { |plugin| connection.extend(plugin) }
-      connection.with_setting(:lock_timeout, SafePgMigrations.config.safe_timeout) { yield }
+
+      connection.with_setting(:lock_timeout, SafePgMigrations.config.pg_safe_timeout) { yield }
     ensure
       close_alternate_connection
       @current_migration = nil
+      stdout_sql_logger&.teardown
+    end
+
+    def get_pg_version_num(connection)
+      connection.query_value('SHOW server_version_num').to_i
     end
 
     def alternate_connection
@@ -50,9 +58,8 @@ module SafePgMigrations
       say "#{method}(#{args.map(&:inspect) * ', '})", true
     end
 
-    def enabled?
-      return ENV['SAFE_PG_MIGRATIONS'] == '1' if ENV['SAFE_PG_MIGRATIONS']
-      return enabled unless enabled.nil?
+    def verbose?
+      return ENV['SAFE_PG_MIGRATIONS_VERBOSE'] == '1' if ENV['SAFE_PG_MIGRATIONS_VERBOSE']
       return Rails.env.production? if defined?(Rails)
 
       false
@@ -71,7 +78,7 @@ module SafePgMigrations
     end
 
     def disable_ddl_transaction
-      SafePgMigrations.enabled? || super
+      true
     end
 
     SAFE_METHODS = %i[execute add_column add_index add_reference add_belongs_to change_column_null].freeze

data/lib/safe-pg-migrations/configuration.rb

@@ -5,17 +5,32 @@ require 'active_support/core_ext/numeric/time'
 module SafePgMigrations
   class Configuration
     attr_accessor :safe_timeout
-    attr_accessor :blocking_activity_logger_delay # Must be close to but smaller than safe_timeout.
+    attr_accessor :index_lock_timeout
+    attr_accessor :blocking_activity_logger_margin
     attr_accessor :batch_size
     attr_accessor :retry_delay
     attr_accessor :max_tries
 
     def initialize
-      self.safe_timeout = '5s'
-      self.blocking_activity_logger_delay = 4.seconds
+      self.safe_timeout = 5.seconds
+      self.index_lock_timeout = 30.seconds
+      self.blocking_activity_logger_margin = 1.second
       self.batch_size = 1000
-      self.retry_delay = 2.minutes
+      self.retry_delay = 1.minute
       self.max_tries = 5
     end
+
+    def pg_safe_timeout
+      pg_duration(safe_timeout)
+    end
+
+    def pg_index_lock_timeout
+      pg_duration(index_lock_timeout)
+    end
+
+    def pg_duration(duration)
+      value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
+      "#{value}#{unit}"
+    end
   end
 end

data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb

@@ -29,16 +29,27 @@ module SafePgMigrations
       change_column_null create_table add_index remove_index
     ].each do |method|
       define_method method do |*args, &block|
-        log_blocking_queries { super(*args, &block) }
+        log_blocking_queries(method) { super(*args, &block) }
       end
     end
 
     private
 
-    def log_blocking_queries
+    def delay_before_logging(method)
+      timeout_delay =
+        if %i[add_index remove_index].include?(method)
+          SafePgMigrations.config.index_lock_timeout
+        else
+          SafePgMigrations.config.safe_timeout
+        end
+
+      timeout_delay - SafePgMigrations.config.blocking_activity_logger_margin
+    end
+
+    def log_blocking_queries(method)
       blocking_queries_retriever_thread =
         Thread.new do
-          sleep SafePgMigrations.config.blocking_activity_logger_delay
+          sleep delay_before_logging(method)
           SafePgMigrations.alternate_connection.query_values(SELECT_BLOCKING_QUERIES_SQL % raw_connection.backend_pid)
         end
 
@@ -65,6 +76,11 @@ module SafePgMigrations
         )
         SafePgMigrations.say '', true
         queries.each { |query| SafePgMigrations.say "  #{query}", true }
+        SafePgMigrations.say(
+          'Beware, some of those queries might run in a transaction. In this case the locking query might be '\
+          'located elsewhere in the transaction',
+          true
+        )
         SafePgMigrations.say '', true
       end
 

data/lib/safe-pg-migrations/plugins/statement_insurer.rb

@@ -2,14 +2,18 @@
 
 module SafePgMigrations
   module StatementInsurer
+    PG_11_VERSION_NUM = 110_000
+
     %i[change_column_null add_foreign_key create_table].each do |method|
       define_method method do |*args, &block|
-        with_setting(:statement_timeout, SafePgMigrations.config.safe_timeout) { super(*args, &block) }
+        with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
       end
     end
 
     def add_column(table_name, column_name, type, **options)
-      default = options.delete(:default)
+      need_default_value_backfill = SafePgMigrations.pg_version_num < PG_11_VERSION_NUM
+
+      default = options.delete(:default) if need_default_value_backfill
       null = options.delete(:null)
 
       if !default.nil? || null == false
@@ -18,7 +22,7 @@ module SafePgMigrations
 
       super
 
-      unless default.nil?
+      if need_default_value_backfill && !default.nil?
         SafePgMigrations.say_method_call(:change_column_default, table_name, column_name, default)
         change_column_default(table_name, column_name, default)
 
@@ -33,20 +37,18 @@ module SafePgMigrations
     end
 
     def add_index(table_name, column_name, **options)
-      if SafePgMigrations.enabled?
-        options[:algorithm] = :concurrently
-        SafePgMigrations.say_method_call(:add_index, table_name, column_name, **options)
-      end
-      without_statement_timeout { super }
+      options[:algorithm] = :concurrently
+      SafePgMigrations.say_method_call(:add_index, table_name, column_name, **options)
+
+      with_index_timeouts { super }
     end
 
     def remove_index(table_name, options = {})
       options = { column: options } unless options.is_a?(Hash)
-      if SafePgMigrations.enabled?
-        options[:algorithm] = :concurrently
-        SafePgMigrations.say_method_call(:remove_index, table_name, **options)
-      end
-      without_statement_timeout { super }
+      options[:algorithm] = :concurrently
+      SafePgMigrations.say_method_call(:remove_index, table_name, **options)
+
+      with_index_timeouts { super }
     end
 
     def backfill_column_default(table_name, column_name)
@@ -86,5 +88,13 @@ module SafePgMigrations
     def without_statement_timeout
       with_setting(:statement_timeout, 0) { yield }
     end
+
+    def with_index_timeouts
+      without_statement_timeout do
+        with_setting(:lock_timeout, SafePgMigrations.config.pg_index_lock_timeout) do
+          yield
+        end
+      end
+    end
   end
 end

data/lib/safe-pg-migrations/plugins/verbose_sql_logger.rb

@@ -0,0 +1,43 @@
+# frozen_string_literal: true
+
+module SafePgMigrations
+  class VerboseSqlLogger
+    def setup
+      @activerecord_logger_was = ActiveRecord::Base.logger
+      @verbose_query_logs_was = ActiveRecord::Base.verbose_query_logs
+      @colorize_logging_was = ActiveRecord::LogSubscriber.colorize_logging
+
+      disable_marginalia if defined?(Marginalia)
+
+      stdout_logger = Logger.new($stdout, formatter: ->(_severity, _time, _progname, query) { "#{query}\n" })
+      ActiveRecord::Base.logger = stdout_logger
+      ActiveRecord::LogSubscriber.colorize_logging = colorize_logging?
+      # Do not output caller method, we know it is coming from the migration
+      ActiveRecord::Base.verbose_query_logs = false
+      self
+    end
+
+    def teardown
+      ActiveRecord::Base.verbose_query_logs = @verbose_query_logs_was
+      ActiveRecord::LogSubscriber.colorize_logging = @colorize_logging_was
+      ActiveRecord::Base.logger = @activerecord_logger_was
+      enable_marginalia if defined?(Marginalia)
+    end
+
+    private
+
+    def colorize_logging?
+      defined?(Rails) && Rails.env.development?
+    end
+
+    # Marginalia annotations will most likely pollute the output
+    def disable_marginalia
+      @marginalia_components_were = Marginalia::Comment.components
+      Marginalia::Comment.components = []
+    end
+
+    def enable_marginalia
+      Marginalia::Comment.components = @marginalia_components_were
+    end
+  end
+end

data/lib/safe-pg-migrations/railtie.rb

@@ -4,7 +4,7 @@ require 'safe-pg-migrations/base'
 
 module SafePgMigrations
   class Railtie < Rails::Railtie
-    initializer 'sage_pg_migrations.insert_into_active_record' do
+    initializer 'safe_pg_migrations.insert_into_active_record' do
       ActiveSupport.on_load :active_record do
         ActiveRecord::Migration.prepend(SafePgMigrations::Migration)
       end

data/lib/safe-pg-migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  VERSION = '0.0.3'
+  VERSION = '1.0.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: safe-pg-migrations
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 1.0.0
 platform: ruby
 authors:
 - Matthieu Prat
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-11-22 00:00:00.000000000 Z
+date: 2019-12-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -166,6 +166,7 @@ files:
 - lib/safe-pg-migrations/plugins/idem_potent_statements.rb
 - lib/safe-pg-migrations/plugins/statement_insurer.rb
 - lib/safe-pg-migrations/plugins/statement_retrier.rb
+- lib/safe-pg-migrations/plugins/verbose_sql_logger.rb
 - lib/safe-pg-migrations/railtie.rb
 - lib/safe-pg-migrations/version.rb
 homepage: https://github.com/doctolib/safe-pg-migrations