safe-pg-migrations 1.0.0 → 1.2.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/README.md +50 -23
- data/lib/safe-pg-migrations/base.rb +3 -0
- data/lib/safe-pg-migrations/configuration.rb +2 -6
- data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb +65 -36
- data/lib/safe-pg-migrations/plugins/idem_potent_statements.rb +32 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +43 -16
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +1 -2
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +33 -0
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +5 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 8b01cc2376f8bcfa1908bdc642a51bd444420540eef7aa01dc403f13ea0b19c8
|
4
|
+
data.tar.gz: 0c09d3bc52ce84542fd50b119cef667714a6058d5fcf59f1be83af72318b3862
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f7a822bbb6cc2e924c8da76ca1501cec16588b4972e850fd07e7fbc120e159dfef715a8293c3399454aa9072e2cc6c2da8927e6dc84f15c72670bb16182fe53c
|
7
|
+
data.tar.gz: 4821e1fb76ce6e868a2032eb0e53aceea8f745f31fce521333949c20143150bd03950187d7286d63d1d9c7269b6015f8b9be8e38833d3e6df659f2f8e5940902
|
data/README.md
CHANGED
@@ -1,10 +1,10 @@
|
|
1
|
-
# safe-pg-migrations
|
1
|
+
# safe-pg-migrations
|
2
2
|
|
3
3
|
ActiveRecord migrations for Postgres made safe.
|
4
4
|
|
5
5
|
## Requirements
|
6
6
|
|
7
|
-
- Ruby 2.
|
7
|
+
- Ruby 2.5+
|
8
8
|
- Rails 5.2+
|
9
9
|
- PostgreSQL 9.3+
|
10
10
|
|
@@ -16,6 +16,8 @@ Just drop this line in your Gemfile:
|
|
16
16
|
gem 'safe-pg-migrations'
|
17
17
|
```
|
18
18
|
|
19
|
+
**Note: Do not run migrations via PgBouncer connection if it is configured to use transactional or statement pooling modes. You must run migrations via a direct Postgres connection, or configure PgBouncer to use session pooling mode.**
|
20
|
+
|
19
21
|
## Example
|
20
22
|
|
21
23
|
Consider the following migration:
|
@@ -61,25 +63,27 @@ Under the hood, **Safe PG Migrations** patches `ActiveRecord::Migration` and ext
|
|
61
63
|
|
62
64
|
## Motivation
|
63
65
|
|
64
|
-
Writing a safe migration can be daunting. Numerous articles have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of Postgres, remembering all the subtleties of explicit locking is not a piece of cake.
|
66
|
+
Writing a safe migration can be daunting. Numerous articles, [including ours](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9), have been written on the topic and a few gems are trying to address the problem. Even for someone who has a pretty good command of Postgres, remembering all the subtleties of explicit locking is not a piece of cake.
|
65
67
|
|
66
68
|
Active Record means developers don't have to be proficient in SQL to interact with a database. In the same way, **Safe PG Migrations** was created so that developers don't have to understand the ins and outs of Postgres to write a safe migration.
|
67
69
|
|
68
70
|
## Feature set
|
69
71
|
|
70
|
-
|
72
|
+
<details><summary>Lock timeout</summary>
|
71
73
|
|
72
74
|
Most DDL operations (e.g. adding a column, removing a column or adding a default value to a column) take an `ACCESS EXCLUSIVE` lock on the table they are altering. While these operations wait to acquire their lock, other statements are blocked. Before running a migration, **Safe PG Migrations** sets a short lock timeout (default to 5 seconds) so that statements are not blocked for too long.
|
73
75
|
|
74
76
|
See [PostgreSQL Alter Table and Long Transactions](http://www.joshuakehn.com/2017/9/9/postgresql-alter-table-and-long-transactions.html) and [Migrations and Long Transactions](https://www.fin.com/post/2018/1/migrations-and-long-transactions) for detailed explanations of the matter.
|
77
|
+
</details>
|
75
78
|
|
76
|
-
|
79
|
+
<details><summary>Statement timeout</summary>
|
77
80
|
|
78
81
|
Adding a foreign key or a not-null constraint can take a lot of time on a large table. The problem is that those operations take `ACCESS EXCLUSIVE` locks. We clearly don't want them to hold these locks for too long. Thus, **Safe PG Migrations** runs them with a short statement timeout (default to 5 seconds).
|
79
82
|
|
80
83
|
See [Zero-downtime Postgres migrations - the hard parts](https://gocardless.com/blog/zero-downtime-postgres-migrations-the-hard-parts/) for a detailed explanation on the subject.
|
84
|
+
</details>
|
81
85
|
|
82
|
-
|
86
|
+
<details><summary>Prevent wrapping migrations in transaction</summary>
|
83
87
|
|
84
88
|
When **Safe PG Migrations** is used, migrations are not wrapped in a transaction. This is for several reasons:
|
85
89
|
|
@@ -89,10 +93,22 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
|
|
89
93
|
|
90
94
|
Note that if a migration fails, it won't be rollbacked. This can result in migrations being partially applied. In that case, they need to be manually reverted.
|
91
95
|
|
92
|
-
|
96
|
+
</details>
|
97
|
+
|
98
|
+
<details>
|
99
|
+
<summary>Safe <code>add_column</code></summary>
|
100
|
+
|
101
|
+
**Safe PG Migrations** gracefully handle the upgrade to PG11 by **not** backfilling default value for existing rows, as the [database engine is now natively handling it](https://www.postgresql.org/docs/11/ddl-alter.html#DDL-ALTER-ADDING-A-COLUMN).
|
102
|
+
|
103
|
+
Beware though, when adding a volatile default value:
|
104
|
+
```ruby
|
105
|
+
add_column :users, :created_at, default: 'clock_timestamp()'
|
106
|
+
```
|
107
|
+
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
|
93
108
|
|
94
|
-
|
109
|
+
<blockquote>
|
95
110
|
|
111
|
+
**Note: Pre-postgre 11**
|
96
112
|
Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
97
113
|
|
98
114
|
**Safe PG Migrations** makes it safe by:
|
@@ -104,17 +120,11 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
|
|
104
120
|
|
105
121
|
Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
|
106
122
|
|
107
|
-
|
123
|
+
</blockquote>
|
108
124
|
|
109
|
-
|
125
|
+
</details>
|
110
126
|
|
111
|
-
|
112
|
-
```ruby
|
113
|
-
add_column :users, :created_at, default: 'clock_timestamp()'
|
114
|
-
```
|
115
|
-
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
|
116
|
-
|
117
|
-
### Concurrent indexes
|
127
|
+
<details><summary id="safe_add_remove_index">Safe <code>add_index</code> and <code>remove_index</code></summary>
|
118
128
|
|
119
129
|
Creating an index requires a `SHARE` lock on the target table which blocks all write on the table while the index is created (which can take some time on a large table). This is usually not practical in a live environment. Thus, **Safe PG Migrations** ensures indexes are created concurrently.
|
120
130
|
|
@@ -125,16 +135,30 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
125
135
|
- Long-running queries are active on the table. To create / remove an index, PG needs to wait for the queries that are actually running to finish before starting the index creation / removal. The blocking activity logger might help you to pinpoint the culprit queries.
|
126
136
|
- A vacuum / autovacuum is running on the table, holding a ShareUpdateExclusiveLock, you are most likely out of luck for the current migration, but you may try to [optimize your autovacuums settings](https://www.percona.com/blog/2018/08/10/tuning-autovacuum-in-postgresql-and-autovacuum-internals/).
|
127
137
|
|
138
|
+
</details>
|
139
|
+
|
140
|
+
<details><summary id="safe_add_foreign_key">safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
|
128
141
|
|
129
|
-
|
142
|
+
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
143
|
+
|
144
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus safe-pg-migrations ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
|
145
|
+
|
146
|
+
1. adding the constraint *without validation*, will not validate existing rows;
|
147
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
148
|
+
|
149
|
+
</details>
|
150
|
+
|
151
|
+
<details><summary>Retry after lock timeout</summary>
|
130
152
|
|
131
153
|
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
154
|
+
</details>
|
132
155
|
|
133
|
-
|
156
|
+
<details><summary>Blocking activity logging</summary>
|
134
157
|
|
135
158
|
If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
|
159
|
+
</details>
|
136
160
|
|
137
|
-
|
161
|
+
<details><summary>Verbose SQL logging</summary>
|
138
162
|
|
139
163
|
For any operation, **Safe PG Migrations** can output the performed SQL queries. This feature is enabled by default in a production Rails environment. If you want to explicit enable it, for example in your development environment you can use:
|
140
164
|
```bash
|
@@ -151,7 +175,7 @@ add_index :users, :age
|
|
151
175
|
-> 0.0175s
|
152
176
|
== 20191215132355 SampleIndex: migrated (0.0200s) =============================
|
153
177
|
```
|
154
|
-
**
|
178
|
+
**Safe PG Migrations** will output the following logs:
|
155
179
|
```ruby
|
156
180
|
add_index :users, :age
|
157
181
|
|
@@ -175,6 +199,8 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
|
|
175
199
|
|
176
200
|
*Nb: The `SHOW` statements are used by **Safe PG Migrations** to query settings for their original values in order to restore them after the work is done*
|
177
201
|
|
202
|
+
</details>
|
203
|
+
|
178
204
|
## Configuration
|
179
205
|
|
180
206
|
**Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
|
@@ -182,9 +208,9 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
|
|
182
208
|
```ruby
|
183
209
|
SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
|
184
210
|
|
185
|
-
SafePgMigrations.config.
|
211
|
+
SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
|
186
212
|
|
187
|
-
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be
|
213
|
+
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
188
214
|
|
189
215
|
SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
|
190
216
|
|
@@ -230,3 +256,4 @@ Interesting reads:
|
|
230
256
|
- [Rails migrations with no downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
|
231
257
|
- [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
|
232
258
|
- [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
|
259
|
+
- [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
|
@@ -6,6 +6,7 @@ require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
|
6
6
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
7
7
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
8
8
|
require 'safe-pg-migrations/plugins/idem_potent_statements'
|
9
|
+
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
9
10
|
|
10
11
|
module SafePgMigrations
|
11
12
|
# Order matters: the bottom-most plugin will have precedence
|
@@ -14,6 +15,7 @@ module SafePgMigrations
|
|
14
15
|
IdemPotentStatements,
|
15
16
|
StatementRetrier,
|
16
17
|
StatementInsurer,
|
18
|
+
UselessStatementsLogger,
|
17
19
|
].freeze
|
18
20
|
|
19
21
|
class << self
|
@@ -78,6 +80,7 @@ module SafePgMigrations
|
|
78
80
|
end
|
79
81
|
|
80
82
|
def disable_ddl_transaction
|
83
|
+
UselessStatementsLogger.warn_useless '`disable_ddl_transaction`' if super
|
81
84
|
true
|
82
85
|
end
|
83
86
|
|
@@ -5,16 +5,16 @@ require 'active_support/core_ext/numeric/time'
|
|
5
5
|
module SafePgMigrations
|
6
6
|
class Configuration
|
7
7
|
attr_accessor :safe_timeout
|
8
|
-
attr_accessor :index_lock_timeout
|
9
8
|
attr_accessor :blocking_activity_logger_margin
|
9
|
+
attr_accessor :blocking_activity_logger_verbose
|
10
10
|
attr_accessor :batch_size
|
11
11
|
attr_accessor :retry_delay
|
12
12
|
attr_accessor :max_tries
|
13
13
|
|
14
14
|
def initialize
|
15
15
|
self.safe_timeout = 5.seconds
|
16
|
-
self.index_lock_timeout = 30.seconds
|
17
16
|
self.blocking_activity_logger_margin = 1.second
|
17
|
+
self.blocking_activity_logger_verbose = true
|
18
18
|
self.batch_size = 1000
|
19
19
|
self.retry_delay = 1.minute
|
20
20
|
self.max_tries = 5
|
@@ -24,10 +24,6 @@ module SafePgMigrations
|
|
24
24
|
pg_duration(safe_timeout)
|
25
25
|
end
|
26
26
|
|
27
|
-
def pg_index_lock_timeout
|
28
|
-
pg_duration(index_lock_timeout)
|
29
|
-
end
|
30
|
-
|
31
27
|
def pg_duration(duration)
|
32
28
|
value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
|
33
29
|
"#{value}#{unit}"
|
@@ -2,55 +2,63 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module BlockingActivityLogger
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
|
18
|
-
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
|
19
|
-
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
|
20
|
-
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
|
21
|
-
AND blocking_locks.pid != blocked_locks.pid
|
22
|
-
JOIN pg_catalog.pg_stat_activity blocking_activity
|
23
|
-
ON blocking_activity.pid = blocking_locks.pid
|
24
|
-
WHERE blocked_locks.pid = %d
|
25
|
-
SQL
|
5
|
+
FILTERED_COLUMNS = %w[
|
6
|
+
blocked_activity.xact_start
|
7
|
+
blocked_locks.locktype
|
8
|
+
blocked_locks.mode
|
9
|
+
blocking_activity.pid
|
10
|
+
blocked_locks.transactionid
|
11
|
+
].freeze
|
12
|
+
|
13
|
+
VERBOSE_COLUMNS = %w[
|
14
|
+
blocking_activity.query
|
15
|
+
blocked_activity.xact_start
|
16
|
+
].freeze
|
26
17
|
|
27
18
|
%i[
|
28
|
-
add_column remove_column add_foreign_key remove_foreign_key change_column_default
|
29
|
-
change_column_null create_table add_index remove_index
|
19
|
+
add_column remove_column add_foreign_key remove_foreign_key change_column_default change_column_null create_table
|
30
20
|
].each do |method|
|
31
21
|
define_method method do |*args, &block|
|
32
|
-
log_blocking_queries
|
22
|
+
log_blocking_queries { super(*args, &block) }
|
33
23
|
end
|
34
24
|
end
|
35
25
|
|
36
26
|
private
|
37
27
|
|
38
|
-
def
|
39
|
-
|
40
|
-
if %i[add_index remove_index].include?(method)
|
41
|
-
SafePgMigrations.config.index_lock_timeout
|
42
|
-
else
|
43
|
-
SafePgMigrations.config.safe_timeout
|
44
|
-
end
|
28
|
+
def select_blocking_queries_sql
|
29
|
+
columns = SafePgMigrations.config.blocking_activity_logger_verbose ? VERBOSE_COLUMNS : FILTERED_COLUMNS
|
45
30
|
|
46
|
-
|
31
|
+
<<~SQL.squish
|
32
|
+
SELECT #{columns.join(', ')}
|
33
|
+
FROM pg_catalog.pg_locks blocked_locks
|
34
|
+
JOIN pg_catalog.pg_stat_activity blocked_activity
|
35
|
+
ON blocked_activity.pid = blocked_locks.pid
|
36
|
+
JOIN pg_catalog.pg_locks blocking_locks
|
37
|
+
ON blocking_locks.locktype = blocked_locks.locktype
|
38
|
+
AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
|
39
|
+
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
|
40
|
+
AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
|
41
|
+
AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
|
42
|
+
AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
|
43
|
+
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
|
44
|
+
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
|
45
|
+
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
|
46
|
+
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
|
47
|
+
AND blocking_locks.pid != blocked_locks.pid
|
48
|
+
JOIN pg_catalog.pg_stat_activity blocking_activity
|
49
|
+
ON blocking_activity.pid = blocking_locks.pid
|
50
|
+
WHERE blocked_locks.pid = %d
|
51
|
+
SQL
|
47
52
|
end
|
48
53
|
|
49
|
-
def log_blocking_queries
|
54
|
+
def log_blocking_queries
|
55
|
+
delay_before_logging =
|
56
|
+
SafePgMigrations.config.safe_timeout - SafePgMigrations.config.blocking_activity_logger_margin
|
57
|
+
|
50
58
|
blocking_queries_retriever_thread =
|
51
59
|
Thread.new do
|
52
|
-
sleep delay_before_logging
|
53
|
-
SafePgMigrations.alternate_connection.
|
60
|
+
sleep delay_before_logging
|
61
|
+
SafePgMigrations.alternate_connection.query(select_blocking_queries_sql % raw_connection.backend_pid)
|
54
62
|
end
|
55
63
|
|
56
64
|
yield
|
@@ -75,7 +83,7 @@ module SafePgMigrations
|
|
75
83
|
"Statement was being blocked by the following #{'query'.pluralize(queries.size)}:", true
|
76
84
|
)
|
77
85
|
SafePgMigrations.say '', true
|
78
|
-
queries
|
86
|
+
output_blocking_queries(queries)
|
79
87
|
SafePgMigrations.say(
|
80
88
|
'Beware, some of those queries might run in a transaction. In this case the locking query might be '\
|
81
89
|
'located elsewhere in the transaction',
|
@@ -86,5 +94,26 @@ module SafePgMigrations
|
|
86
94
|
|
87
95
|
raise
|
88
96
|
end
|
97
|
+
|
98
|
+
def output_blocking_queries(queries)
|
99
|
+
if SafePgMigrations.config.blocking_activity_logger_verbose
|
100
|
+
queries.each { |query, start_time| SafePgMigrations.say "#{format_start_time start_time}: #{query}", true }
|
101
|
+
else
|
102
|
+
queries.each do |start_time, locktype, mode, pid, transactionid|
|
103
|
+
SafePgMigrations.say(
|
104
|
+
"#{format_start_time(start_time)}: lock type: #{locktype || 'null'}, " \
|
105
|
+
"lock mode: #{mode || 'null'}, " \
|
106
|
+
"lock pid: #{pid || 'null'}, " \
|
107
|
+
"lock transactionid: #{transactionid || 'null'}",
|
108
|
+
true
|
109
|
+
)
|
110
|
+
end
|
111
|
+
end
|
112
|
+
end
|
113
|
+
|
114
|
+
def format_start_time(start_time, reference_time = Time.now)
|
115
|
+
duration = (reference_time - start_time).round
|
116
|
+
"transaction started #{duration} #{'second'.pluralize(duration)} ago"
|
117
|
+
end
|
89
118
|
end
|
90
119
|
end
|
@@ -12,6 +12,12 @@ module SafePgMigrations
|
|
12
12
|
super
|
13
13
|
end
|
14
14
|
|
15
|
+
def add_column(table_name, column_name, type, options = {})
|
16
|
+
return super unless column_exists?(table_name, column_name)
|
17
|
+
|
18
|
+
SafePgMigrations.say("/!\\ Column '#{column_name}' already exists in '#{table_name}'. Skipping statement.", true)
|
19
|
+
end
|
20
|
+
|
15
21
|
def remove_column(table_name, column_name, type = nil, options = {})
|
16
22
|
return super if column_exists?(table_name, column_name)
|
17
23
|
|
@@ -26,6 +32,32 @@ module SafePgMigrations
|
|
26
32
|
SafePgMigrations.say("/!\\ Index '#{index_name}' not found on table '#{table_name}'. Skipping statement.", true)
|
27
33
|
end
|
28
34
|
|
35
|
+
def add_foreign_key(from_table, to_table, **options)
|
36
|
+
options_or_to_table = options.slice(:name, :column).presence || to_table
|
37
|
+
return super unless foreign_key_exists?(from_table, options_or_to_table)
|
38
|
+
|
39
|
+
SafePgMigrations.say(
|
40
|
+
"/!\\ Foreign key '#{from_table}' -> '#{to_table}' already exists. Skipping statement.",
|
41
|
+
true
|
42
|
+
)
|
43
|
+
end
|
44
|
+
|
45
|
+
def create_table(table_name, comment: nil, **options)
|
46
|
+
return super if options[:force] || !table_exists?(table_name)
|
47
|
+
|
48
|
+
SafePgMigrations.say "/!\\ Table '#{table_name}' already exists.", true
|
49
|
+
|
50
|
+
td = create_table_definition(table_name, **options)
|
51
|
+
|
52
|
+
yield td if block_given?
|
53
|
+
|
54
|
+
SafePgMigrations.say(td.indexes.empty? ? '-- Skipping statement' : '-- Creating indexes', true)
|
55
|
+
|
56
|
+
td.indexes.each do |column_name, index_options|
|
57
|
+
add_index(table_name, column_name, index_options)
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
29
61
|
private
|
30
62
|
|
31
63
|
def index_valid?(index_name)
|
@@ -4,25 +4,25 @@ module SafePgMigrations
|
|
4
4
|
module StatementInsurer
|
5
5
|
PG_11_VERSION_NUM = 110_000
|
6
6
|
|
7
|
-
%i[change_column_null
|
7
|
+
%i[change_column_null change_column].each do |method|
|
8
8
|
define_method method do |*args, &block|
|
9
9
|
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
|
10
10
|
end
|
11
11
|
end
|
12
12
|
|
13
13
|
def add_column(table_name, column_name, type, **options)
|
14
|
-
|
14
|
+
return super if SafePgMigrations.pg_version_num >= PG_11_VERSION_NUM
|
15
15
|
|
16
|
-
default = options.delete(:default)
|
16
|
+
default = options.delete(:default)
|
17
17
|
null = options.delete(:null)
|
18
18
|
|
19
19
|
if !default.nil? || null == false
|
20
|
-
SafePgMigrations.say_method_call(:add_column, table_name, column_name, type,
|
20
|
+
SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, options)
|
21
21
|
end
|
22
22
|
|
23
23
|
super
|
24
24
|
|
25
|
-
|
25
|
+
unless default.nil?
|
26
26
|
SafePgMigrations.say_method_call(:change_column_default, table_name, column_name, default)
|
27
27
|
change_column_default(table_name, column_name, default)
|
28
28
|
|
@@ -36,19 +36,46 @@ module SafePgMigrations
|
|
36
36
|
end
|
37
37
|
end
|
38
38
|
|
39
|
+
def add_foreign_key(from_table, to_table, **options)
|
40
|
+
validate_present = options.key? :validate
|
41
|
+
options[:validate] = false unless validate_present
|
42
|
+
|
43
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super }
|
44
|
+
|
45
|
+
options_or_to_table = options.slice(:name, :column).presence || to_table
|
46
|
+
without_statement_timeout { validate_foreign_key from_table, options_or_to_table } unless validate_present
|
47
|
+
end
|
48
|
+
|
49
|
+
def create_table(*)
|
50
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) do
|
51
|
+
super do |td|
|
52
|
+
yield td if block_given?
|
53
|
+
td.indexes.map! do |key, index_options|
|
54
|
+
index_options[:algorithm] ||= :default
|
55
|
+
[key, index_options]
|
56
|
+
end
|
57
|
+
end
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
39
61
|
def add_index(table_name, column_name, **options)
|
40
|
-
options[:algorithm]
|
41
|
-
|
62
|
+
if options[:algorithm] == :default
|
63
|
+
options.delete :algorithm
|
64
|
+
else
|
65
|
+
options[:algorithm] = :concurrently
|
66
|
+
end
|
42
67
|
|
43
|
-
|
68
|
+
SafePgMigrations.say_method_call(:add_index, table_name, column_name, options)
|
69
|
+
|
70
|
+
without_timeout { super }
|
44
71
|
end
|
45
72
|
|
46
73
|
def remove_index(table_name, options = {})
|
47
74
|
options = { column: options } unless options.is_a?(Hash)
|
48
75
|
options[:algorithm] = :concurrently
|
49
|
-
SafePgMigrations.say_method_call(:remove_index, table_name,
|
76
|
+
SafePgMigrations.say_method_call(:remove_index, table_name, options)
|
50
77
|
|
51
|
-
|
78
|
+
without_timeout { super }
|
52
79
|
end
|
53
80
|
|
54
81
|
def backfill_column_default(table_name, column_name)
|
@@ -89,12 +116,12 @@ module SafePgMigrations
|
|
89
116
|
with_setting(:statement_timeout, 0) { yield }
|
90
117
|
end
|
91
118
|
|
92
|
-
def
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
119
|
+
def without_lock_timeout
|
120
|
+
with_setting(:lock_timeout, 0) { yield }
|
121
|
+
end
|
122
|
+
|
123
|
+
def without_timeout
|
124
|
+
without_statement_timeout { without_lock_timeout { yield } }
|
98
125
|
end
|
99
126
|
end
|
100
127
|
end
|
@@ -3,8 +3,7 @@
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module StatementRetrier
|
5
5
|
RETRIABLE_SCHEMA_STATEMENTS = %i[
|
6
|
-
add_column add_foreign_key remove_foreign_key change_column_default
|
7
|
-
change_column_null add_index remove_index remove_column
|
6
|
+
add_column add_foreign_key remove_foreign_key change_column_default change_column_null remove_column
|
8
7
|
].freeze
|
9
8
|
|
10
9
|
RETRIABLE_SCHEMA_STATEMENTS.each do |method|
|
@@ -0,0 +1,33 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module UselessStatementsLogger
|
5
|
+
def self.warn_useless(action, link = nil, *args)
|
6
|
+
SafePgMigrations.say "/!\\ No need to explicitly use #{action}, safe-pg-migrations does it for you", *args
|
7
|
+
SafePgMigrations.say "\t see #{link} for more details", *args if link
|
8
|
+
end
|
9
|
+
|
10
|
+
def add_index(*, **options)
|
11
|
+
warn_for_index(**options)
|
12
|
+
super
|
13
|
+
end
|
14
|
+
|
15
|
+
def remove_index(table_name, options = {})
|
16
|
+
warn_for_index(options) if options.is_a? Hash
|
17
|
+
super
|
18
|
+
end
|
19
|
+
|
20
|
+
def add_foreign_key(*, **options)
|
21
|
+
if options[:validate] == false
|
22
|
+
UselessStatementsLogger.warn_useless '`validate: :false`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_foreign_key'
|
23
|
+
end
|
24
|
+
super
|
25
|
+
end
|
26
|
+
|
27
|
+
def warn_for_index(**options)
|
28
|
+
return unless options[:algorithm] == :concurrently
|
29
|
+
|
30
|
+
UselessStatementsLogger.warn_useless '`algorithm: :concurrently`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_remove_index'
|
31
|
+
end
|
32
|
+
end
|
33
|
+
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: safe-pg-migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.2.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthieu Prat
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2021-04-30 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: activerecord
|
@@ -166,6 +166,7 @@ files:
|
|
166
166
|
- lib/safe-pg-migrations/plugins/idem_potent_statements.rb
|
167
167
|
- lib/safe-pg-migrations/plugins/statement_insurer.rb
|
168
168
|
- lib/safe-pg-migrations/plugins/statement_retrier.rb
|
169
|
+
- lib/safe-pg-migrations/plugins/useless_statements_logger.rb
|
169
170
|
- lib/safe-pg-migrations/plugins/verbose_sql_logger.rb
|
170
171
|
- lib/safe-pg-migrations/railtie.rb
|
171
172
|
- lib/safe-pg-migrations/version.rb
|
@@ -181,7 +182,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
181
182
|
requirements:
|
182
183
|
- - ">="
|
183
184
|
- !ruby/object:Gem::Version
|
184
|
-
version: '2.
|
185
|
+
version: '2.5'
|
185
186
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
186
187
|
requirements:
|
187
188
|
- - ">="
|
@@ -189,7 +190,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
189
190
|
version: '0'
|
190
191
|
requirements: []
|
191
192
|
rubyforge_project:
|
192
|
-
rubygems_version: 2.
|
193
|
+
rubygems_version: 2.7.3
|
193
194
|
signing_key:
|
194
195
|
specification_version: 4
|
195
196
|
summary: Make your PG migrations safe.
|