safe-pg-migrations 1.0.0 → 1.2.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: d3e47cc5a8440b59b3e25dc7a5e99a9212c4709c
4
- data.tar.gz: 7347fdd053c1edd34b3ec58d9021b126ec794eb3
2
+ SHA256:
3
+ metadata.gz: 8b01cc2376f8bcfa1908bdc642a51bd444420540eef7aa01dc403f13ea0b19c8
4
+ data.tar.gz: 0c09d3bc52ce84542fd50b119cef667714a6058d5fcf59f1be83af72318b3862
5
5
  SHA512:
6
- metadata.gz: 74058e22a0e55ae44e29b8e74096bf0b4bf6b7c83a7fafce8c6aa3595c50bafec2d93c6ce502d304f89b44e236a3f3770fac91ff7e9bb449a3289cca0a06ab89
7
- data.tar.gz: 7a38389555cb2cd8c0271882389aed8e27b1d22ef7cb16976c729041ef315803ad7de0a71400f478d50c7c741e1b10a5a37109940de5a5e7a8809800eecdff25
6
+ metadata.gz: f7a822bbb6cc2e924c8da76ca1501cec16588b4972e850fd07e7fbc120e159dfef715a8293c3399454aa9072e2cc6c2da8927e6dc84f15c72670bb16182fe53c
7
+ data.tar.gz: 4821e1fb76ce6e868a2032eb0e53aceea8f745f31fce521333949c20143150bd03950187d7286d63d1d9c7269b6015f8b9be8e38833d3e6df659f2f8e5940902
data/README.md CHANGED
@@ -1,10 +1,10 @@
1
- # safe-pg-migrations [![Build Status](https://travis-ci.org/doctolib/safe-pg-migrations.svg?branch=master)](https://travis-ci.org/doctolib/safe-pg-migrations)
1
+ # safe-pg-migrations
2
2
 
3
3
  ActiveRecord migrations for Postgres made safe.
4
4
 
5
5
  ## Requirements
6
6
 
7
- - Ruby 2.3+
7
+ - Ruby 2.5+
8
8
  - Rails 5.2+
9
9
  - PostgreSQL 9.3+
10
10
 
@@ -16,6 +16,8 @@ Just drop this line in your Gemfile:
16
16
  gem 'safe-pg-migrations'
17
17
  ```
18
18
 
19
+ **Note: Do not run migrations via PgBouncer connection if it is configured to use transactional or statement pooling modes. You must run migrations via a direct Postgres connection, or configure PgBouncer to use session pooling mode.**
20
+
19
21
  ## Example
20
22
 
21
23
  Consider the following migration:
@@ -61,25 +63,27 @@ Under the hood, **Safe PG Migrations** patches `ActiveRecord::Migration` and ext
61
63
 
62
64
  ## Motivation
63
65
 
64
- Writing a safe migration can be daunting. Numerous articles have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of Postgres, remembering all the subtleties of explicit locking is not a piece of cake.
66
+ Writing a safe migration can be daunting. Numerous articles, [including ours](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9), have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of Postgres, remembering all the subtleties of explicit locking is not a piece of cake.
65
67
 
66
68
  Active Record means developers don't have to be proficient in SQL to interact with a database. In the same way, **Safe PG Migrations** was created so that developers don't have to understand the ins and outs of Postgres to write a safe migration.
67
69
 
68
70
  ## Feature set
69
71
 
70
- ### Lock timeout
72
+ <details><summary>Lock timeout</summary>
71
73
 
72
74
  Most DDL operations (e.g. adding a column, removing a column or adding a default value to a column) take an `ACCESS EXCLUSIVE` lock on the table they are altering. While these operations wait to acquire their lock, other statements are blocked. Before running a migration, **Safe PG Migrations** sets a short lock timeout (default to 5 seconds) so that statements are not blocked for too long.
73
75
 
74
76
  See [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html) and [Migrations and Long Transactions](https://www.fin.com/post/2018/1/migrations-and-long-transactions) for detailed explanations of the matter.
77
+ </details>
75
78
 
76
- ### Statement timeout
79
+ <details><summary>Statement timeout</summary>
77
80
 
78
81
  Adding a foreign key or a not-null constraint can take a lot of time on a large table. The problem is that those operations take `ACCESS EXCLUSIVE` locks. We clearly don't want them to hold these locks for too long. Thus, **Safe PG Migrations** runs them with a short statement timeout (default to 5 seconds).
79
82
 
80
83
  See [Zero-downtime Postgres migrations - the hard parts](https://gocardless.com/blog/zero-downtime-postgres-migrations-the-hard-parts/) for a detailed explanation on the subject.
84
+ </details>
81
85
 
82
- ### Prevent wrapping migrations in transaction
86
+ <details><summary>Prevent wrapping migrations in transaction</summary>
83
87
 
84
88
  When **Safe PG Migrations** is used, migrations are not wrapped in a transaction. This is for several reasons:
85
89
 
@@ -89,10 +93,22 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
89
93
 
90
94
  Note that if a migration fails, it won't be rollbacked. This can result in migrations being partially applied. In that case, they need to be manually reverted.
91
95
 
92
- ### Safe `add_column`
96
+ </details>
97
+
98
+ <details>
99
+ <summary>Safe <code>add_column</code></summary>
100
+
101
+ **Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
102
+
103
+ Beware though, when adding a volatile default value:
104
+ ```ruby
105
+ add_column :users, :created_at, default: 'clock_timestamp()'
106
+ ```
107
+ PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
93
108
 
94
- #### Pre Postgres 11 behavior
109
+ <blockquote>
95
110
 
111
+ **Note: Pre-postgre 11**
96
112
  Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
97
113
 
98
114
  **Safe PG Migrations** makes it safe by:
@@ -104,17 +120,11 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
104
120
 
105
121
  Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
106
122
 
107
- #### Postgres 11 behavior
123
+ </blockquote>
108
124
 
109
- **Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
125
+ </details>
110
126
 
111
- Beware though, when adding a volatile default value:
112
- ```ruby
113
- add_column :users, :created_at, default: 'clock_timestamp()'
114
- ```
115
- PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
116
-
117
- ### Concurrent indexes
127
+ <details><summary id="safe_add_remove_index">Safe <code>add_index</code> and <code>remove_index</code></summary>
118
128
 
119
129
  Creating an index requires a `SHARE` lock on the target table which blocks all write on the table while the index is created (which can take some time on a large table). This is usually not practical in a live environment. Thus, **Safe PG Migrations** ensures indexes are created concurrently.
120
130
 
@@ -125,16 +135,30 @@ If you still get lock timeout while adding / removing indexes, it might be for o
125
135
  - Long-running queries are active on the table. To create / remove an index, PG needs to wait for the queries that are actually running to finish before starting the index creation / removal. The blocking activity logger might help you to pinpoint the culprit queries.
126
136
  - A vacuum / autovacuum is running on the table, holding a ShareUpdateExclusiveLock, you are most likely out of luck for the current migration, but you may try to [optimize your autovacuums settings](https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in-postgresql-and-autovacuum-internals/).
127
137
 
138
+ </details>
139
+
140
+ <details><summary id="safe_add_foreign_key">safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
128
141
 
129
- ### Retry after lock timeout
142
+ Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
143
+
144
+ Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus safe-pg-migrations ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
145
+
146
+ 1. adding the constraint *without validation*, will not validate existing rows;
147
+ 2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
148
+
149
+ </details>
150
+
151
+ <details><summary>Retry after lock timeout</summary>
130
152
 
131
153
  When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
154
+ </details>
132
155
 
133
- ### Blocking activity logging
156
+ <details><summary>Blocking activity logging</summary>
134
157
 
135
158
  If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
159
+ </details>
136
160
 
137
- ### Verbose SQL logging
161
+ <details><summary>Verbose SQL logging</summary>
138
162
 
139
163
  For any operation, **Safe PG Migrations** can output the performed SQL queries. This feature is enabled by default in a production Rails environment. If you want to explicit enable it, for example in your development environment you can use:
140
164
  ```bash
@@ -151,7 +175,7 @@ add_index :users, :age
151
175
  -> 0.0175s
152
176
  == 20191215132355 SampleIndex: migrated (0.0200s) =============================
153
177
  ```
154
- **Sage PG Migrations** will output the following logs:
178
+ **Safe PG Migrations** will output the following logs:
155
179
  ```ruby
156
180
  add_index :users, :age
157
181
 
@@ -175,6 +199,8 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
175
199
 
176
200
  *Nb: The `SHOW` statements are used by **Safe PG Migrations** to query settings for their original values in order to restore them after the work is done*
177
201
 
202
+ </details>
203
+
178
204
  ## Configuration
179
205
 
180
206
  **Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
@@ -182,9 +208,9 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
182
208
  ```ruby
183
209
  SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
184
210
 
185
- SafePgMigrations.config.index_lock_timeout = 30.seconds # Lock timeout used for CREATE / DROP INDEX
211
+ SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
186
212
 
187
- SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be smaller than safe_timeout and index_lock_timeout
213
+ SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
188
214
 
189
215
  SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
190
216
 
@@ -230,3 +256,4 @@ Interesting reads:
230
256
  - [Rails migrations with no downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
231
257
  - [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
232
258
  - [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
259
+ - [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
@@ -6,6 +6,7 @@ require 'safe-pg-migrations/plugins/blocking_activity_logger'
6
6
  require 'safe-pg-migrations/plugins/statement_insurer'
7
7
  require 'safe-pg-migrations/plugins/statement_retrier'
8
8
  require 'safe-pg-migrations/plugins/idem_potent_statements'
9
+ require 'safe-pg-migrations/plugins/useless_statements_logger'
9
10
 
10
11
  module SafePgMigrations
11
12
  # Order matters: the bottom-most plugin will have precedence
@@ -14,6 +15,7 @@ module SafePgMigrations
14
15
  IdemPotentStatements,
15
16
  StatementRetrier,
16
17
  StatementInsurer,
18
+ UselessStatementsLogger,
17
19
  ].freeze
18
20
 
19
21
  class << self
@@ -78,6 +80,7 @@ module SafePgMigrations
78
80
  end
79
81
 
80
82
  def disable_ddl_transaction
83
+ UselessStatementsLogger.warn_useless '`disable_ddl_transaction`' if super
81
84
  true
82
85
  end
83
86
 
@@ -5,16 +5,16 @@ require 'active_support/core_ext/numeric/time'
5
5
  module SafePgMigrations
6
6
  class Configuration
7
7
  attr_accessor :safe_timeout
8
- attr_accessor :index_lock_timeout
9
8
  attr_accessor :blocking_activity_logger_margin
9
+ attr_accessor :blocking_activity_logger_verbose
10
10
  attr_accessor :batch_size
11
11
  attr_accessor :retry_delay
12
12
  attr_accessor :max_tries
13
13
 
14
14
  def initialize
15
15
  self.safe_timeout = 5.seconds
16
- self.index_lock_timeout = 30.seconds
17
16
  self.blocking_activity_logger_margin = 1.second
17
+ self.blocking_activity_logger_verbose = true
18
18
  self.batch_size = 1000
19
19
  self.retry_delay = 1.minute
20
20
  self.max_tries = 5
@@ -24,10 +24,6 @@ module SafePgMigrations
24
24
  pg_duration(safe_timeout)
25
25
  end
26
26
 
27
- def pg_index_lock_timeout
28
- pg_duration(index_lock_timeout)
29
- end
30
-
31
27
  def pg_duration(duration)
32
28
  value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
33
29
  "#{value}#{unit}"
@@ -2,55 +2,63 @@
2
2
 
3
3
  module SafePgMigrations
4
4
  module BlockingActivityLogger
5
- SELECT_BLOCKING_QUERIES_SQL = <<~SQL.squish
6
- SELECT blocking_activity.query
7
- FROM pg_catalog.pg_locks blocked_locks
8
- JOIN pg_catalog.pg_stat_activity blocked_activity
9
- ON blocked_activity.pid = blocked_locks.pid
10
- JOIN pg_catalog.pg_locks blocking_locks
11
- ON blocking_locks.locktype = blocked_locks.locktype
12
- AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
13
- AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
14
- AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
15
- AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
16
- AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
17
- AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
18
- AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
19
- AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
20
- AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
21
- AND blocking_locks.pid != blocked_locks.pid
22
- JOIN pg_catalog.pg_stat_activity blocking_activity
23
- ON blocking_activity.pid = blocking_locks.pid
24
- WHERE blocked_locks.pid = %d
25
- SQL
5
+ FILTERED_COLUMNS = %w[
6
+ blocked_activity.xact_start
7
+ blocked_locks.locktype
8
+ blocked_locks.mode
9
+ blocking_activity.pid
10
+ blocked_locks.transactionid
11
+ ].freeze
12
+
13
+ VERBOSE_COLUMNS = %w[
14
+ blocking_activity.query
15
+ blocked_activity.xact_start
16
+ ].freeze
26
17
 
27
18
  %i[
28
- add_column remove_column add_foreign_key remove_foreign_key change_column_default
29
- change_column_null create_table add_index remove_index
19
+ add_column remove_column add_foreign_key remove_foreign_key change_column_default change_column_null create_table
30
20
  ].each do |method|
31
21
  define_method method do |*args, &block|
32
- log_blocking_queries(method) { super(*args, &block) }
22
+ log_blocking_queries { super(*args, &block) }
33
23
  end
34
24
  end
35
25
 
36
26
  private
37
27
 
38
- def delay_before_logging(method)
39
- timeout_delay =
40
- if %i[add_index remove_index].include?(method)
41
- SafePgMigrations.config.index_lock_timeout
42
- else
43
- SafePgMigrations.config.safe_timeout
44
- end
28
+ def select_blocking_queries_sql
29
+ columns = SafePgMigrations.config.blocking_activity_logger_verbose ? VERBOSE_COLUMNS : FILTERED_COLUMNS
45
30
 
46
- timeout_delay - SafePgMigrations.config.blocking_activity_logger_margin
31
+ <<~SQL.squish
32
+ SELECT #{columns.join(', ')}
33
+ FROM pg_catalog.pg_locks blocked_locks
34
+ JOIN pg_catalog.pg_stat_activity blocked_activity
35
+ ON blocked_activity.pid = blocked_locks.pid
36
+ JOIN pg_catalog.pg_locks blocking_locks
37
+ ON blocking_locks.locktype = blocked_locks.locktype
38
+ AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
39
+ AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
40
+ AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
41
+ AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
42
+ AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
43
+ AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
44
+ AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
45
+ AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
46
+ AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
47
+ AND blocking_locks.pid != blocked_locks.pid
48
+ JOIN pg_catalog.pg_stat_activity blocking_activity
49
+ ON blocking_activity.pid = blocking_locks.pid
50
+ WHERE blocked_locks.pid = %d
51
+ SQL
47
52
  end
48
53
 
49
- def log_blocking_queries(method)
54
+ def log_blocking_queries
55
+ delay_before_logging =
56
+ SafePgMigrations.config.safe_timeout - SafePgMigrations.config.blocking_activity_logger_margin
57
+
50
58
  blocking_queries_retriever_thread =
51
59
  Thread.new do
52
- sleep delay_before_logging(method)
53
- SafePgMigrations.alternate_connection.query_values(SELECT_BLOCKING_QUERIES_SQL % raw_connection.backend_pid)
60
+ sleep delay_before_logging
61
+ SafePgMigrations.alternate_connection.query(select_blocking_queries_sql % raw_connection.backend_pid)
54
62
  end
55
63
 
56
64
  yield
@@ -75,7 +83,7 @@ module SafePgMigrations
75
83
  "Statement was being blocked by the following #{'query'.pluralize(queries.size)}:", true
76
84
  )
77
85
  SafePgMigrations.say '', true
78
- queries.each { |query| SafePgMigrations.say " #{query}", true }
86
+ output_blocking_queries(queries)
79
87
  SafePgMigrations.say(
80
88
  'Beware, some of those queries might run in a transaction. In this case the locking query might be '\
81
89
  'located elsewhere in the transaction',
@@ -86,5 +94,26 @@ module SafePgMigrations
86
94
 
87
95
  raise
88
96
  end
97
+
98
+ def output_blocking_queries(queries)
99
+ if SafePgMigrations.config.blocking_activity_logger_verbose
100
+ queries.each { |query, start_time| SafePgMigrations.say "#{format_start_time start_time}: #{query}", true }
101
+ else
102
+ queries.each do |start_time, locktype, mode, pid, transactionid|
103
+ SafePgMigrations.say(
104
+ "#{format_start_time(start_time)}: lock type: #{locktype || 'null'}, " \
105
+ "lock mode: #{mode || 'null'}, " \
106
+ "lock pid: #{pid || 'null'}, " \
107
+ "lock transactionid: #{transactionid || 'null'}",
108
+ true
109
+ )
110
+ end
111
+ end
112
+ end
113
+
114
+ def format_start_time(start_time, reference_time = Time.now)
115
+ duration = (reference_time - start_time).round
116
+ "transaction started #{duration} #{'second'.pluralize(duration)} ago"
117
+ end
89
118
  end
90
119
  end
@@ -12,6 +12,12 @@ module SafePgMigrations
12
12
  super
13
13
  end
14
14
 
15
+ def add_column(table_name, column_name, type, options = {})
16
+ return super unless column_exists?(table_name, column_name)
17
+
18
+ SafePgMigrations.say("/!\\ Column '#{column_name}' already exists in '#{table_name}'. Skipping statement.", true)
19
+ end
20
+
15
21
  def remove_column(table_name, column_name, type = nil, options = {})
16
22
  return super if column_exists?(table_name, column_name)
17
23
 
@@ -26,6 +32,32 @@ module SafePgMigrations
26
32
  SafePgMigrations.say("/!\\ Index '#{index_name}' not found on table '#{table_name}'. Skipping statement.", true)
27
33
  end
28
34
 
35
+ def add_foreign_key(from_table, to_table, **options)
36
+ options_or_to_table = options.slice(:name, :column).presence || to_table
37
+ return super unless foreign_key_exists?(from_table, options_or_to_table)
38
+
39
+ SafePgMigrations.say(
40
+ "/!\\ Foreign key '#{from_table}' -> '#{to_table}' already exists. Skipping statement.",
41
+ true
42
+ )
43
+ end
44
+
45
+ def create_table(table_name, comment: nil, **options)
46
+ return super if options[:force] || !table_exists?(table_name)
47
+
48
+ SafePgMigrations.say "/!\\ Table '#{table_name}' already exists.", true
49
+
50
+ td = create_table_definition(table_name, **options)
51
+
52
+ yield td if block_given?
53
+
54
+ SafePgMigrations.say(td.indexes.empty? ? '-- Skipping statement' : '-- Creating indexes', true)
55
+
56
+ td.indexes.each do |column_name, index_options|
57
+ add_index(table_name, column_name, index_options)
58
+ end
59
+ end
60
+
29
61
  private
30
62
 
31
63
  def index_valid?(index_name)
@@ -4,25 +4,25 @@ module SafePgMigrations
4
4
  module StatementInsurer
5
5
  PG_11_VERSION_NUM = 110_000
6
6
 
7
- %i[change_column_null add_foreign_key create_table].each do |method|
7
+ %i[change_column_null change_column].each do |method|
8
8
  define_method method do |*args, &block|
9
9
  with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
10
10
  end
11
11
  end
12
12
 
13
13
  def add_column(table_name, column_name, type, **options)
14
- need_default_value_backfill = SafePgMigrations.pg_version_num < PG_11_VERSION_NUM
14
+ return super if SafePgMigrations.pg_version_num >= PG_11_VERSION_NUM
15
15
 
16
- default = options.delete(:default) if need_default_value_backfill
16
+ default = options.delete(:default)
17
17
  null = options.delete(:null)
18
18
 
19
19
  if !default.nil? || null == false
20
- SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, **options)
20
+ SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, options)
21
21
  end
22
22
 
23
23
  super
24
24
 
25
- if need_default_value_backfill && !default.nil?
25
+ unless default.nil?
26
26
  SafePgMigrations.say_method_call(:change_column_default, table_name, column_name, default)
27
27
  change_column_default(table_name, column_name, default)
28
28
 
@@ -36,19 +36,46 @@ module SafePgMigrations
36
36
  end
37
37
  end
38
38
 
39
+ def add_foreign_key(from_table, to_table, **options)
40
+ validate_present = options.key? :validate
41
+ options[:validate] = false unless validate_present
42
+
43
+ with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super }
44
+
45
+ options_or_to_table = options.slice(:name, :column).presence || to_table
46
+ without_statement_timeout { validate_foreign_key from_table, options_or_to_table } unless validate_present
47
+ end
48
+
49
+ def create_table(*)
50
+ with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) do
51
+ super do |td|
52
+ yield td if block_given?
53
+ td.indexes.map! do |key, index_options|
54
+ index_options[:algorithm] ||= :default
55
+ [key, index_options]
56
+ end
57
+ end
58
+ end
59
+ end
60
+
39
61
  def add_index(table_name, column_name, **options)
40
- options[:algorithm] = :concurrently
41
- SafePgMigrations.say_method_call(:add_index, table_name, column_name, **options)
62
+ if options[:algorithm] == :default
63
+ options.delete :algorithm
64
+ else
65
+ options[:algorithm] = :concurrently
66
+ end
42
67
 
43
- with_index_timeouts { super }
68
+ SafePgMigrations.say_method_call(:add_index, table_name, column_name, options)
69
+
70
+ without_timeout { super }
44
71
  end
45
72
 
46
73
  def remove_index(table_name, options = {})
47
74
  options = { column: options } unless options.is_a?(Hash)
48
75
  options[:algorithm] = :concurrently
49
- SafePgMigrations.say_method_call(:remove_index, table_name, **options)
76
+ SafePgMigrations.say_method_call(:remove_index, table_name, options)
50
77
 
51
- with_index_timeouts { super }
78
+ without_timeout { super }
52
79
  end
53
80
 
54
81
  def backfill_column_default(table_name, column_name)
@@ -89,12 +116,12 @@ module SafePgMigrations
89
116
  with_setting(:statement_timeout, 0) { yield }
90
117
  end
91
118
 
92
- def with_index_timeouts
93
- without_statement_timeout do
94
- with_setting(:lock_timeout, SafePgMigrations.config.pg_index_lock_timeout) do
95
- yield
96
- end
97
- end
119
+ def without_lock_timeout
120
+ with_setting(:lock_timeout, 0) { yield }
121
+ end
122
+
123
+ def without_timeout
124
+ without_statement_timeout { without_lock_timeout { yield } }
98
125
  end
99
126
  end
100
127
  end
@@ -3,8 +3,7 @@
3
3
  module SafePgMigrations
4
4
  module StatementRetrier
5
5
  RETRIABLE_SCHEMA_STATEMENTS = %i[
6
- add_column add_foreign_key remove_foreign_key change_column_default
7
- change_column_null add_index remove_index remove_column
6
+ add_column add_foreign_key remove_foreign_key change_column_default change_column_null remove_column
8
7
  ].freeze
9
8
 
10
9
  RETRIABLE_SCHEMA_STATEMENTS.each do |method|
@@ -0,0 +1,33 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SafePgMigrations
4
+ module UselessStatementsLogger
5
+ def self.warn_useless(action, link = nil, *args)
6
+ SafePgMigrations.say "/!\\ No need to explicitly use #{action}, safe-pg-migrations does it for you", *args
7
+ SafePgMigrations.say "\t see #{link} for more details", *args if link
8
+ end
9
+
10
+ def add_index(*, **options)
11
+ warn_for_index(**options)
12
+ super
13
+ end
14
+
15
+ def remove_index(table_name, options = {})
16
+ warn_for_index(options) if options.is_a? Hash
17
+ super
18
+ end
19
+
20
+ def add_foreign_key(*, **options)
21
+ if options[:validate] == false
22
+ UselessStatementsLogger.warn_useless '`validate: :false`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_foreign_key'
23
+ end
24
+ super
25
+ end
26
+
27
+ def warn_for_index(**options)
28
+ return unless options[:algorithm] == :concurrently
29
+
30
+ UselessStatementsLogger.warn_useless '`algorithm: :concurrently`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_remove_index'
31
+ end
32
+ end
33
+ end
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SafePgMigrations
4
- VERSION = '1.0.0'
4
+ VERSION = '1.2.3'
5
5
  end
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: safe-pg-migrations
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0
4
+ version: 1.2.3
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matthieu Prat
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2019-12-17 00:00:00.000000000 Z
12
+ date: 2021-04-30 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: activerecord
@@ -166,6 +166,7 @@ files:
166
166
  - lib/safe-pg-migrations/plugins/idem_potent_statements.rb
167
167
  - lib/safe-pg-migrations/plugins/statement_insurer.rb
168
168
  - lib/safe-pg-migrations/plugins/statement_retrier.rb
169
+ - lib/safe-pg-migrations/plugins/useless_statements_logger.rb
169
170
  - lib/safe-pg-migrations/plugins/verbose_sql_logger.rb
170
171
  - lib/safe-pg-migrations/railtie.rb
171
172
  - lib/safe-pg-migrations/version.rb
@@ -181,7 +182,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
181
182
  requirements:
182
183
  - - ">="
183
184
  - !ruby/object:Gem::Version
184
- version: '2.3'
185
+ version: '2.5'
185
186
  required_rubygems_version: !ruby/object:Gem::Requirement
186
187
  requirements:
187
188
  - - ">="
@@ -189,7 +190,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
189
190
  version: '0'
190
191
  requirements: []
191
192
  rubyforge_project:
192
- rubygems_version: 2.6.14.1
193
+ rubygems_version: 2.7.3
193
194
  signing_key:
194
195
  specification_version: 4
195
196
  summary: Make your PG migrations safe.