safe-pg-migrations 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: d3e47cc5a8440b59b3e25dc7a5e99a9212c4709c
-  data.tar.gz: 7347fdd053c1edd34b3ec58d9021b126ec794eb3
+SHA256:
+  metadata.gz: d806334c473708774d2180dbe91d2483b49f0d27b0a223a2973bc95d986d4030
+  data.tar.gz: 9b5c2d329d5cc1bd12c80944420e20eb5454e5b0a0f7fda2d6ec3fc3c3a5e042
 SHA512:
-  metadata.gz: 74058e22a0e55ae44e29b8e74096bf0b4bf6b7c83a7fafce8c6aa3595c50bafec2d93c6ce502d304f89b44e236a3f3770fac91ff7e9bb449a3289cca0a06ab89
-  data.tar.gz: 7a38389555cb2cd8c0271882389aed8e27b1d22ef7cb16976c729041ef315803ad7de0a71400f478d50c7c741e1b10a5a37109940de5a5e7a8809800eecdff25
+  metadata.gz: 6786e68cce9dbd91a4633c80526a1fd63f6fbfa335d6e2b53d29b71ba0b566f9bb32169745bb13f89a059b82b0c0b21585a9bffdb9ebc751aa0e4a467f0f7934
+  data.tar.gz: 299a69f1d9413279fd61908a7d8f6eb65bd4c9912abcc1101ca8acbfee39c40d33086efa28a9f2133c0679c1ad9cdbbfe3ce84338f2175ad882b7d1ace707654

data/README.md

@@ -1,10 +1,10 @@
-# safe-pg-migrations [![Build Status](https://travis-ci.org/doctolib/safe-pg-migrations.svg?branch=master)](https://travis-ci.org/doctolib/safe-pg-migrations)
+# safe-pg-migrations
 
 ActiveRecord migrations for Postgres made safe.
 
 ## Requirements
 
-- Ruby 2.3+
+- Ruby 2.5+
 - Rails 5.2+
 - PostgreSQL 9.3+
 
@@ -67,19 +67,21 @@ Active Record means developers don't have to be proficient in SQL to interact wi
 
 ## Feature set
 
-### Lock timeout
+<details><summary>Lock timeout</summary>
 
 Most DDL operations (e.g. adding a column, removing a column or adding a default value to a column) take an `ACCESS EXCLUSIVE` lock on the table they are altering. While these operations wait to acquire their lock, other statements are blocked. Before running a migration, **Safe PG Migrations** sets a short lock timeout (default to 5 seconds) so that statements are not blocked for too long.
 
 See [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html) and [Migrations and Long Transactions](https://www.fin.com/post/2018/1/migrations-and-long-transactions) for detailed explanations of the matter.
+</details>
 
-### Statement timeout
+<details><summary>Statement timeout</summary>
 
 Adding a foreign key or a not-null constraint can take a lot of time on a large table. The problem is that those operations take `ACCESS EXCLUSIVE` locks. We clearly don't want them to hold these locks for too long. Thus, **Safe PG Migrations** runs them with a short statement timeout (default to 5 seconds).
 
 See [Zero-downtime Postgres migrations - the hard parts](https://gocardless.com/blog/zero-downtime-postgres-migrations-the-hard-parts/) for a detailed explanation on the subject.
+</details>
 
-### Prevent wrapping migrations in transaction
+<details><summary>Prevent wrapping migrations in transaction</summary>
 
 When **Safe PG Migrations** is used, migrations are not wrapped in a transaction. This is for several reasons:
 
@@ -89,10 +91,22 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
 
 Note that if a migration fails, it won't be rollbacked. This can result in migrations being partially applied. In that case, they need to be manually reverted.
 
-### Safe `add_column`
+</details>
 
-#### Pre Postgres 11 behavior
+<details>
+<summary>Safe <code>add_column</code></summary>
 
+**Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
+
+Beware though, when adding a volatile default value: 
+```ruby
+add_column :users, :created_at, default: 'clock_timestamp()'
+```
+PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
+
+<blockquote>
+
+**Note: Pre-postgre 11**
 Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
 
 **Safe PG Migrations** makes it safe by:
@@ -104,17 +118,11 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
 
 Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
 
-#### Postgres 11 behavior
-
-**Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
+</blockquote>    
 
-Beware though, when adding a volatile default value: 
-```ruby
-add_column :users, :created_at, default: 'clock_timestamp()'
-```
-PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
+</details>
 
-### Concurrent indexes
+<details><summary>Safe <code>add_index</code> and <code>remove_index</code></summary>
 
 Creating an index requires a `SHARE` lock on the target table which blocks all write on the table while the index is created (which can take some time on a large table). This is usually not practical in a live environment. Thus, **Safe PG Migrations** ensures indexes are created concurrently.
 
@@ -125,16 +133,30 @@ If you still get lock timeout while adding / removing indexes, it might be for o
 - Long-running queries are active on the table. To create / remove an index, PG needs to wait for the queries that are actually running to finish before starting the index creation / removal. The blocking activity logger might help you to pinpoint the culprit queries.
 - A vacuum / autovacuum is running on the table, holding a ShareUpdateExclusiveLock, you are most likely out of luck for the current migration, but you may try to [optimize your autovacuums settings](https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in-postgresql-and-autovacuum-internals/).
 
+</details>
+
+<details><summary>safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
 
-### Retry after lock timeout
+Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
+
+Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus safe-pg-migrations ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps: 
+
+1. adding the constraint *without validation*, will not validate existing rows;
+2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
+
+</details>
+
+<details><summary>Retry after lock timeout</summary>
 
 When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
+</details>
 
-### Blocking activity logging
+<details><summary>Blocking activity logging</summary>
 
 If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
+</details>
 
-### Verbose SQL logging
+<details><summary>Verbose SQL logging</summary>
 
 For any operation, **Safe PG Migrations** can output the performed SQL queries. This feature is enabled by default in a production Rails environment. If you want to explicit enable it, for example in your development environment you can use:
 ```bash
@@ -151,7 +173,7 @@ add_index :users, :age
    -> 0.0175s
 == 20191215132355 SampleIndex: migrated (0.0200s) =============================
 ```
-**Sage PG Migrations** will output the following logs:
+**Safe PG Migrations** will output the following logs:
 ```ruby
 add_index :users, :age
 
@@ -175,6 +197,8 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
 
 *Nb: The `SHOW` statements are used by **Safe PG Migrations** to query settings for their original values in order to restore them after the work is done*
 
+</details>
+
 ## Configuration
 
 **Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):

data/lib/safe-pg-migrations/plugins/idem_potent_statements.rb

@@ -12,6 +12,12 @@ module SafePgMigrations
       super
     end
 
+    def add_column(table_name, column_name, type, options = {})
+      return super unless column_exists?(table_name, column_name)
+
+      SafePgMigrations.say("/!\\ Column '#{column_name}' already exists in '#{table_name}'. Skipping statement.", true)
+    end
+
     def remove_column(table_name, column_name, type = nil, options = {})
       return super if column_exists?(table_name, column_name)
 
@@ -26,6 +32,25 @@ module SafePgMigrations
       SafePgMigrations.say("/!\\ Index '#{index_name}' not found on table '#{table_name}'. Skipping statement.", true)
     end
 
+    def add_foreign_key(from_table, to_table, **options)
+      options_or_to_table = options.slice(:name, :column).presence || to_table
+      return super unless foreign_key_exists?(from_table, options_or_to_table)
+
+      SafePgMigrations.say(
+        "/!\\ Foreign key '#{from_table}' -> '#{to_table}' already exists. Skipping statement.",
+        true
+      )
+    end
+
+    def create_table(table_name, comment: nil, **options)
+      return super unless table_exists?(table_name)
+
+      SafePgMigrations.say(
+        "/!\\ Table '#{table_name}' already exists. Skipping statement.",
+        true
+      )
+    end
+
     private
 
     def index_valid?(index_name)

data/lib/safe-pg-migrations/plugins/statement_insurer.rb

@@ -4,20 +4,20 @@ module SafePgMigrations
   module StatementInsurer
     PG_11_VERSION_NUM = 110_000
 
-    %i[change_column_null add_foreign_key create_table].each do |method|
+    %i[change_column_null change_column create_table].each do |method|
       define_method method do |*args, &block|
         with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
       end
     end
 
-    def add_column(table_name, column_name, type, **options)
+    def add_column(table_name, column_name, type, **options) # rubocop:disable Metrics/CyclomaticComplexity
       need_default_value_backfill = SafePgMigrations.pg_version_num < PG_11_VERSION_NUM
 
       default = options.delete(:default) if need_default_value_backfill
       null = options.delete(:null)
 
       if !default.nil? || null == false
-        SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, **options)
+        SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, options)
       end
 
       super
@@ -36,9 +36,19 @@ module SafePgMigrations
       end
     end
 
+    def add_foreign_key(from_table, to_table, **options)
+      validate_present = options.key? :validate
+      options[:validate] = false unless validate_present
+
+      with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super }
+
+      options_or_to_table = options.slice(:name, :column).presence || to_table
+      without_statement_timeout { validate_foreign_key from_table, options_or_to_table } unless validate_present
+    end
+
     def add_index(table_name, column_name, **options)
       options[:algorithm] = :concurrently
-      SafePgMigrations.say_method_call(:add_index, table_name, column_name, **options)
+      SafePgMigrations.say_method_call(:add_index, table_name, column_name, options)
 
       with_index_timeouts { super }
     end
@@ -46,7 +56,7 @@ module SafePgMigrations
     def remove_index(table_name, options = {})
       options = { column: options } unless options.is_a?(Hash)
       options[:algorithm] = :concurrently
-      SafePgMigrations.say_method_call(:remove_index, table_name, **options)
+      SafePgMigrations.say_method_call(:remove_index, table_name, options)
 
       with_index_timeouts { super }
     end

data/lib/safe-pg-migrations/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SafePgMigrations
-  VERSION = '1.0.0'
+  VERSION = '1.1.0'
 end

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: safe-pg-migrations
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - Matthieu Prat
 - Romain Choquet
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-12-17 00:00:00.000000000 Z
+date: 2020-11-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -173,7 +173,7 @@ homepage: https://github.com/doctolib/safe-pg-migrations
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -181,16 +181,16 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.3'
+      version: '2.5'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.6.14.1
-signing_key: 
+rubyforge_project:
+rubygems_version: 2.7.3
+signing_key:
 specification_version: 4
 summary: Make your PG migrations safe.
 test_files: []