safe-pg-migrations 1.2.2 → 1.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +12 -7
- data/lib/safe-pg-migrations/base.rb +18 -5
- data/lib/safe-pg-migrations/configuration.rb +2 -0
- data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb +59 -24
- data/lib/safe-pg-migrations/plugins/{idem_potent_statements.rb → idempotent_statements.rb} +27 -15
- data/lib/safe-pg-migrations/plugins/legacy_active_record_support.rb +23 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +23 -26
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +1 -0
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +12 -7
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +29 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2fb891279f68c60f15f81b6e7ccbc425750443c3ee294080c2286bb059ed2e89
|
4
|
+
data.tar.gz: f4a8d59c4ee5abf004e4141db1d09d3cde33ff42f180c03526767535f20a5acd
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5b8c48e973fc0296a54c2a8cceaa3b3c8c8d3380cf15a87ea5154e2ebfc28a8aa852172fe020946d8c758a4d1518bf22fe06eda08d31d318d9176df88820ac8e
|
7
|
+
data.tar.gz: e88101b365a400e21ca76cf64b1818fba36e8875958989e1864391faf476b793596bbcfbd6bed4c53db46d93a16981334f45fe83124dc04e2eb7132029a712e3
|
data/README.md
CHANGED
@@ -2,6 +2,8 @@
|
|
2
2
|
|
3
3
|
ActiveRecord migrations for Postgres made safe.
|
4
4
|
|
5
|
+
![safe-pg-migrations](./logo.png)
|
6
|
+
|
5
7
|
## Requirements
|
6
8
|
|
7
9
|
- Ruby 2.5+
|
@@ -91,7 +93,7 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
|
|
91
93
|
- In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
|
92
94
|
- In order to add an index concurrently, we have to be outside a transaction.
|
93
95
|
|
94
|
-
Note that if a migration fails, it won't be
|
96
|
+
Note that if a migration fails, it won't be rolled back. This can result in migrations being partially applied. In that case, they need to be manually reverted.
|
95
97
|
|
96
98
|
</details>
|
97
99
|
|
@@ -108,7 +110,7 @@ PG will still needs to update every row of the table, and will most likely state
|
|
108
110
|
|
109
111
|
<blockquote>
|
110
112
|
|
111
|
-
**Note: Pre-
|
113
|
+
**Note: Pre-postgres 11**
|
112
114
|
Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
113
115
|
|
114
116
|
**Safe PG Migrations** makes it safe by:
|
@@ -120,7 +122,7 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
|
|
120
122
|
|
121
123
|
Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
|
122
124
|
|
123
|
-
</blockquote>
|
125
|
+
</blockquote>
|
124
126
|
|
125
127
|
</details>
|
126
128
|
|
@@ -137,7 +139,7 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
137
139
|
|
138
140
|
</details>
|
139
141
|
|
140
|
-
<details><summary id="safe_add_foreign_key">
|
142
|
+
<details><summary id="safe_add_foreign_key">Safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
|
141
143
|
|
142
144
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
143
145
|
|
@@ -150,7 +152,7 @@ Adding the constraint itself is rather fast, the major part of the time is spent
|
|
150
152
|
|
151
153
|
<details><summary>Retry after lock timeout</summary>
|
152
154
|
|
153
|
-
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of
|
155
|
+
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
154
156
|
</details>
|
155
157
|
|
156
158
|
<details><summary>Blocking activity logging</summary>
|
@@ -208,6 +210,8 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
|
|
208
210
|
```ruby
|
209
211
|
SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
|
210
212
|
|
213
|
+
SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
|
214
|
+
|
211
215
|
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
212
216
|
|
213
217
|
SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
|
@@ -217,11 +221,11 @@ SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retry
|
|
217
221
|
SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
|
218
222
|
```
|
219
223
|
|
220
|
-
##
|
224
|
+
## Running tests
|
221
225
|
|
222
226
|
```bash
|
223
227
|
bundle
|
224
|
-
psql -h localhost -
|
228
|
+
psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
|
225
229
|
rake test
|
226
230
|
```
|
227
231
|
|
@@ -255,3 +259,4 @@ Interesting reads:
|
|
255
259
|
- [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
|
256
260
|
- [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
|
257
261
|
- [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
|
262
|
+
- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
|
@@ -1,21 +1,24 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require 'ruby2_keywords'
|
3
4
|
require 'safe-pg-migrations/configuration'
|
4
5
|
require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
5
6
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
6
7
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
7
8
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
8
|
-
require 'safe-pg-migrations/plugins/
|
9
|
+
require 'safe-pg-migrations/plugins/idempotent_statements'
|
9
10
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
11
|
+
require 'safe-pg-migrations/plugins/legacy_active_record_support'
|
10
12
|
|
11
13
|
module SafePgMigrations
|
12
14
|
# Order matters: the bottom-most plugin will have precedence
|
13
15
|
PLUGINS = [
|
14
16
|
BlockingActivityLogger,
|
15
|
-
|
17
|
+
IdempotentStatements,
|
16
18
|
StatementRetrier,
|
17
19
|
StatementInsurer,
|
18
20
|
UselessStatementsLogger,
|
21
|
+
LegacyActiveRecordSupport,
|
19
22
|
].freeze
|
20
23
|
|
21
24
|
class << self
|
@@ -50,13 +53,13 @@ module SafePgMigrations
|
|
50
53
|
@alternate_connection = nil
|
51
54
|
end
|
52
55
|
|
53
|
-
def say(*args)
|
56
|
+
ruby2_keywords def say(*args)
|
54
57
|
return unless current_migration
|
55
58
|
|
56
59
|
current_migration.say(*args)
|
57
60
|
end
|
58
61
|
|
59
|
-
def say_method_call(method, *args)
|
62
|
+
ruby2_keywords def say_method_call(method, *args)
|
60
63
|
say "#{method}(#{args.map(&:inspect) * ', '})", true
|
61
64
|
end
|
62
65
|
|
@@ -84,13 +87,23 @@ module SafePgMigrations
|
|
84
87
|
true
|
85
88
|
end
|
86
89
|
|
87
|
-
SAFE_METHODS = %i[
|
90
|
+
SAFE_METHODS = %i[
|
91
|
+
execute
|
92
|
+
add_column
|
93
|
+
add_index
|
94
|
+
add_reference
|
95
|
+
add_belongs_to
|
96
|
+
change_column_null
|
97
|
+
add_foreign_key
|
98
|
+
].freeze
|
99
|
+
|
88
100
|
SAFE_METHODS.each do |method|
|
89
101
|
define_method method do |*args|
|
90
102
|
return super(*args) unless respond_to?(:safety_assured)
|
91
103
|
|
92
104
|
safety_assured { super(*args) }
|
93
105
|
end
|
106
|
+
ruby2_keywords method
|
94
107
|
end
|
95
108
|
end
|
96
109
|
end
|
@@ -6,6 +6,7 @@ module SafePgMigrations
|
|
6
6
|
class Configuration
|
7
7
|
attr_accessor :safe_timeout
|
8
8
|
attr_accessor :blocking_activity_logger_margin
|
9
|
+
attr_accessor :blocking_activity_logger_verbose
|
9
10
|
attr_accessor :batch_size
|
10
11
|
attr_accessor :retry_delay
|
11
12
|
attr_accessor :max_tries
|
@@ -13,6 +14,7 @@ module SafePgMigrations
|
|
13
14
|
def initialize
|
14
15
|
self.safe_timeout = 5.seconds
|
15
16
|
self.blocking_activity_logger_margin = 1.second
|
17
|
+
self.blocking_activity_logger_verbose = true
|
16
18
|
self.batch_size = 1000
|
17
19
|
self.retry_delay = 1.minute
|
18
20
|
self.max_tries = 5
|
@@ -1,28 +1,19 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module BlockingActivityLogger
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
|
18
|
-
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
|
19
|
-
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
|
20
|
-
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
|
21
|
-
AND blocking_locks.pid != blocked_locks.pid
|
22
|
-
JOIN pg_catalog.pg_stat_activity blocking_activity
|
23
|
-
ON blocking_activity.pid = blocking_locks.pid
|
24
|
-
WHERE blocked_locks.pid = %d
|
25
|
-
SQL
|
4
|
+
module BlockingActivityLogger # rubocop:disable Metrics/ModuleLength
|
5
|
+
FILTERED_COLUMNS = %w[
|
6
|
+
blocked_activity.xact_start
|
7
|
+
blocked_locks.locktype
|
8
|
+
blocked_locks.mode
|
9
|
+
blocking_activity.pid
|
10
|
+
blocked_locks.transactionid
|
11
|
+
].freeze
|
12
|
+
|
13
|
+
VERBOSE_COLUMNS = %w[
|
14
|
+
blocking_activity.query
|
15
|
+
blocked_activity.xact_start
|
16
|
+
].freeze
|
26
17
|
|
27
18
|
%i[
|
28
19
|
add_column remove_column add_foreign_key remove_foreign_key change_column_default change_column_null create_table
|
@@ -30,10 +21,37 @@ module SafePgMigrations
|
|
30
21
|
define_method method do |*args, &block|
|
31
22
|
log_blocking_queries { super(*args, &block) }
|
32
23
|
end
|
24
|
+
ruby2_keywords method
|
33
25
|
end
|
34
26
|
|
35
27
|
private
|
36
28
|
|
29
|
+
def select_blocking_queries_sql
|
30
|
+
columns = SafePgMigrations.config.blocking_activity_logger_verbose ? VERBOSE_COLUMNS : FILTERED_COLUMNS
|
31
|
+
|
32
|
+
<<~SQL.squish
|
33
|
+
SELECT #{columns.join(', ')}
|
34
|
+
FROM pg_catalog.pg_locks blocked_locks
|
35
|
+
JOIN pg_catalog.pg_stat_activity blocked_activity
|
36
|
+
ON blocked_activity.pid = blocked_locks.pid
|
37
|
+
JOIN pg_catalog.pg_locks blocking_locks
|
38
|
+
ON blocking_locks.locktype = blocked_locks.locktype
|
39
|
+
AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
|
40
|
+
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
|
41
|
+
AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
|
42
|
+
AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
|
43
|
+
AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
|
44
|
+
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
|
45
|
+
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
|
46
|
+
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
|
47
|
+
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
|
48
|
+
AND blocking_locks.pid != blocked_locks.pid
|
49
|
+
JOIN pg_catalog.pg_stat_activity blocking_activity
|
50
|
+
ON blocking_activity.pid = blocking_locks.pid
|
51
|
+
WHERE blocked_locks.pid = %d
|
52
|
+
SQL
|
53
|
+
end
|
54
|
+
|
37
55
|
def log_blocking_queries
|
38
56
|
delay_before_logging =
|
39
57
|
SafePgMigrations.config.safe_timeout - SafePgMigrations.config.blocking_activity_logger_margin
|
@@ -41,7 +59,7 @@ module SafePgMigrations
|
|
41
59
|
blocking_queries_retriever_thread =
|
42
60
|
Thread.new do
|
43
61
|
sleep delay_before_logging
|
44
|
-
SafePgMigrations.alternate_connection.query(
|
62
|
+
SafePgMigrations.alternate_connection.query(select_blocking_queries_sql % raw_connection.backend_pid)
|
45
63
|
end
|
46
64
|
|
47
65
|
yield
|
@@ -66,7 +84,7 @@ module SafePgMigrations
|
|
66
84
|
"Statement was being blocked by the following #{'query'.pluralize(queries.size)}:", true
|
67
85
|
)
|
68
86
|
SafePgMigrations.say '', true
|
69
|
-
queries
|
87
|
+
output_blocking_queries(queries)
|
70
88
|
SafePgMigrations.say(
|
71
89
|
'Beware, some of those queries might run in a transaction. In this case the locking query might be '\
|
72
90
|
'located elsewhere in the transaction',
|
@@ -78,7 +96,24 @@ module SafePgMigrations
|
|
78
96
|
raise
|
79
97
|
end
|
80
98
|
|
99
|
+
def output_blocking_queries(queries)
|
100
|
+
if SafePgMigrations.config.blocking_activity_logger_verbose
|
101
|
+
queries.each { |query, start_time| SafePgMigrations.say "#{format_start_time start_time}: #{query}", true }
|
102
|
+
else
|
103
|
+
queries.each do |start_time, locktype, mode, pid, transactionid|
|
104
|
+
SafePgMigrations.say(
|
105
|
+
"#{format_start_time(start_time)}: lock type: #{locktype || 'null'}, " \
|
106
|
+
"lock mode: #{mode || 'null'}, " \
|
107
|
+
"lock pid: #{pid || 'null'}, " \
|
108
|
+
"lock transactionid: #{transactionid || 'null'}",
|
109
|
+
true
|
110
|
+
)
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
81
115
|
def format_start_time(start_time, reference_time = Time.now)
|
116
|
+
start_time = Time.parse(start_time) unless start_time.is_a? Time
|
82
117
|
duration = (reference_time - start_time).round
|
83
118
|
"transaction started #{duration} #{'second'.pluralize(duration)} ago"
|
84
119
|
end
|
@@ -1,30 +1,40 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module
|
5
|
-
def add_index(table_name, column_name,
|
6
|
-
|
7
|
-
return super unless index_name_exists?(table_name, index_name)
|
4
|
+
module IdempotentStatements
|
5
|
+
ruby2_keywords def add_index(table_name, column_name, *args)
|
6
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
8
7
|
|
9
|
-
|
8
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
10
9
|
|
11
|
-
|
10
|
+
return super unless index_name_exists?(index_definition.table, index_definition.name)
|
11
|
+
|
12
|
+
if index_valid?(index_definition.name)
|
13
|
+
SafePgMigrations.say(
|
14
|
+
"/!\\ Index '#{index_definition.name}' already exists in '#{table_name}'. Skipping statement.",
|
15
|
+
true
|
16
|
+
)
|
17
|
+
return
|
18
|
+
end
|
19
|
+
|
20
|
+
remove_index(table_name, name: index_definition.name)
|
12
21
|
super
|
13
22
|
end
|
14
23
|
|
15
|
-
def add_column(table_name, column_name, type,
|
24
|
+
ruby2_keywords def add_column(table_name, column_name, type, *)
|
16
25
|
return super unless column_exists?(table_name, column_name)
|
17
26
|
|
18
27
|
SafePgMigrations.say("/!\\ Column '#{column_name}' already exists in '#{table_name}'. Skipping statement.", true)
|
19
28
|
end
|
20
29
|
|
21
|
-
def remove_column(table_name, column_name, type = nil,
|
30
|
+
ruby2_keywords def remove_column(table_name, column_name, type = nil, *)
|
22
31
|
return super if column_exists?(table_name, column_name)
|
23
32
|
|
24
33
|
SafePgMigrations.say("/!\\ Column '#{column_name}' not found on table '#{table_name}'. Skipping statement.", true)
|
25
34
|
end
|
26
35
|
|
27
|
-
def remove_index(table_name,
|
36
|
+
ruby2_keywords def remove_index(table_name, *args)
|
37
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
28
38
|
index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, options)
|
29
39
|
|
30
40
|
return super if index_name_exists?(table_name, index_name)
|
@@ -32,9 +42,10 @@ module SafePgMigrations
|
|
32
42
|
SafePgMigrations.say("/!\\ Index '#{index_name}' not found on table '#{table_name}'. Skipping statement.", true)
|
33
43
|
end
|
34
44
|
|
35
|
-
def add_foreign_key(from_table, to_table,
|
36
|
-
|
37
|
-
|
45
|
+
ruby2_keywords def add_foreign_key(from_table, to_table, *args)
|
46
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
47
|
+
suboptions = options.slice(:name, :column)
|
48
|
+
return super unless foreign_key_exists?(from_table, suboptions.present? ? nil : to_table, **suboptions)
|
38
49
|
|
39
50
|
SafePgMigrations.say(
|
40
51
|
"/!\\ Foreign key '#{from_table}' -> '#{to_table}' already exists. Skipping statement.",
|
@@ -42,19 +53,20 @@ module SafePgMigrations
|
|
42
53
|
)
|
43
54
|
end
|
44
55
|
|
45
|
-
def create_table(table_name,
|
56
|
+
ruby2_keywords def create_table(table_name, *args)
|
57
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
46
58
|
return super if options[:force] || !table_exists?(table_name)
|
47
59
|
|
48
60
|
SafePgMigrations.say "/!\\ Table '#{table_name}' already exists.", true
|
49
61
|
|
50
|
-
td = create_table_definition(table_name,
|
62
|
+
td = create_table_definition(table_name, *args)
|
51
63
|
|
52
64
|
yield td if block_given?
|
53
65
|
|
54
66
|
SafePgMigrations.say(td.indexes.empty? ? '-- Skipping statement' : '-- Creating indexes', true)
|
55
67
|
|
56
68
|
td.indexes.each do |column_name, index_options|
|
57
|
-
add_index(table_name, column_name, index_options)
|
69
|
+
add_index(table_name, column_name, **index_options)
|
58
70
|
end
|
59
71
|
end
|
60
72
|
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module LegacyActiveRecordSupport
|
5
|
+
ruby2_keywords def validate_foreign_key(from_table, to_table = nil, **options)
|
6
|
+
return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
|
7
|
+
|
8
|
+
super(from_table, to_table, **options)
|
9
|
+
end
|
10
|
+
|
11
|
+
ruby2_keywords def foreign_key_exists?(from_table, to_table = nil, **options)
|
12
|
+
return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
|
13
|
+
|
14
|
+
super(from_table, to_table, **options)
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def satisfied?(version)
|
20
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
@@ -8,9 +8,11 @@ module SafePgMigrations
|
|
8
8
|
define_method method do |*args, &block|
|
9
9
|
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
|
10
10
|
end
|
11
|
+
ruby2_keywords method
|
11
12
|
end
|
12
13
|
|
13
|
-
def add_column(table_name, column_name, type,
|
14
|
+
ruby2_keywords def add_column(table_name, column_name, type, *args) # rubocop:disable Metrics/CyclomaticComplexity
|
15
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
14
16
|
return super if SafePgMigrations.pg_version_num >= PG_11_VERSION_NUM
|
15
17
|
|
16
18
|
default = options.delete(:default)
|
@@ -36,17 +38,21 @@ module SafePgMigrations
|
|
36
38
|
end
|
37
39
|
end
|
38
40
|
|
39
|
-
def add_foreign_key(from_table, to_table,
|
41
|
+
ruby2_keywords def add_foreign_key(from_table, to_table, *args)
|
42
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
40
43
|
validate_present = options.key? :validate
|
41
44
|
options[:validate] = false unless validate_present
|
45
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) do
|
46
|
+
super(from_table, to_table, **options)
|
47
|
+
end
|
42
48
|
|
43
|
-
|
49
|
+
return if validate_present
|
44
50
|
|
45
|
-
|
46
|
-
without_statement_timeout { validate_foreign_key from_table,
|
51
|
+
suboptions = options.slice(:name, :column)
|
52
|
+
without_statement_timeout { validate_foreign_key from_table, suboptions.present? ? nil : to_table, **suboptions }
|
47
53
|
end
|
48
54
|
|
49
|
-
def create_table(*)
|
55
|
+
ruby2_keywords def create_table(*)
|
50
56
|
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) do
|
51
57
|
super do |td|
|
52
58
|
yield td if block_given?
|
@@ -65,34 +71,25 @@ module SafePgMigrations
|
|
65
71
|
options[:algorithm] = :concurrently
|
66
72
|
end
|
67
73
|
|
68
|
-
SafePgMigrations.say_method_call(:add_index, table_name, column_name, options)
|
74
|
+
SafePgMigrations.say_method_call(:add_index, table_name, column_name, **options)
|
69
75
|
|
70
|
-
without_timeout { super }
|
76
|
+
without_timeout { super(table_name, column_name, **options) }
|
71
77
|
end
|
72
78
|
|
73
|
-
def remove_index(table_name,
|
74
|
-
options = { column:
|
75
|
-
options[:algorithm] = :concurrently
|
76
|
-
SafePgMigrations.say_method_call(:remove_index, table_name, options)
|
79
|
+
ruby2_keywords def remove_index(table_name, *args)
|
80
|
+
options = args.last.is_a?(Hash) ? args.last : { column: args.last }
|
81
|
+
options[:algorithm] = :concurrently unless options.key?(:algorithm)
|
82
|
+
SafePgMigrations.say_method_call(:remove_index, table_name, **options)
|
77
83
|
|
78
|
-
without_timeout { super }
|
84
|
+
without_timeout { super(table_name, **options) }
|
79
85
|
end
|
80
86
|
|
81
87
|
def backfill_column_default(table_name, column_name)
|
82
|
-
|
88
|
+
model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
|
83
89
|
quoted_column_name = quote_column_name(column_name)
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
SELECT id FROM #{quoted_table_name} WHERE id > #{primary_key_offset}
|
88
|
-
ORDER BY id LIMIT #{SafePgMigrations.config.batch_size}
|
89
|
-
SQL
|
90
|
-
break if ids.empty?
|
91
|
-
|
92
|
-
primary_key_offset = ids.last
|
93
|
-
execute <<~SQL.squish
|
94
|
-
UPDATE #{quoted_table_name} SET #{quoted_column_name} = DEFAULT WHERE id IN (#{ids.join(',')})
|
95
|
-
SQL
|
90
|
+
|
91
|
+
model.in_batches(of: SafePgMigrations.config.batch_size).each do |relation|
|
92
|
+
relation.update_all("#{quoted_column_name} = DEFAULT")
|
96
93
|
end
|
97
94
|
end
|
98
95
|
|
@@ -2,22 +2,27 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module UselessStatementsLogger
|
5
|
-
|
6
|
-
|
7
|
-
|
5
|
+
class << self
|
6
|
+
ruby2_keywords def warn_useless(action, link = nil, *args)
|
7
|
+
SafePgMigrations.say "/!\\ No need to explicitly use #{action}, safe-pg-migrations does it for you", *args
|
8
|
+
SafePgMigrations.say "\t see #{link} for more details", *args if link
|
9
|
+
end
|
8
10
|
end
|
9
11
|
|
10
|
-
def add_index(
|
12
|
+
ruby2_keywords def add_index(*args)
|
13
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
11
14
|
warn_for_index(**options)
|
12
15
|
super
|
13
16
|
end
|
14
17
|
|
15
|
-
def remove_index(table_name,
|
16
|
-
|
18
|
+
ruby2_keywords def remove_index(table_name, *args)
|
19
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
20
|
+
warn_for_index(**options) unless options.empty?
|
17
21
|
super
|
18
22
|
end
|
19
23
|
|
20
|
-
def add_foreign_key(
|
24
|
+
ruby2_keywords def add_foreign_key(*args)
|
25
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
21
26
|
if options[:validate] == false
|
22
27
|
UselessStatementsLogger.warn_useless '`validate: :false`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_foreign_key'
|
23
28
|
end
|
metadata
CHANGED
@@ -1,15 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: safe-pg-migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthieu Prat
|
8
8
|
- Romain Choquet
|
9
|
+
- Thomas Hareau
|
9
10
|
autorequire:
|
10
11
|
bindir: bin
|
11
12
|
cert_chain: []
|
12
|
-
date:
|
13
|
+
date: 2022-02-24 00:00:00.000000000 Z
|
13
14
|
dependencies:
|
14
15
|
- !ruby/object:Gem::Dependency
|
15
16
|
name: activerecord
|
@@ -39,6 +40,20 @@ dependencies:
|
|
39
40
|
- - ">="
|
40
41
|
- !ruby/object:Gem::Version
|
41
42
|
version: '5.2'
|
43
|
+
- !ruby/object:Gem::Dependency
|
44
|
+
name: ruby2_keywords
|
45
|
+
requirement: !ruby/object:Gem::Requirement
|
46
|
+
requirements:
|
47
|
+
- - ">="
|
48
|
+
- !ruby/object:Gem::Version
|
49
|
+
version: 0.0.4
|
50
|
+
type: :runtime
|
51
|
+
prerelease: false
|
52
|
+
version_requirements: !ruby/object:Gem::Requirement
|
53
|
+
requirements:
|
54
|
+
- - ">="
|
55
|
+
- !ruby/object:Gem::Version
|
56
|
+
version: 0.0.4
|
42
57
|
- !ruby/object:Gem::Dependency
|
43
58
|
name: bundler
|
44
59
|
requirement: !ruby/object:Gem::Requirement
|
@@ -152,7 +167,7 @@ dependencies:
|
|
152
167
|
- !ruby/object:Gem::Version
|
153
168
|
version: '0'
|
154
169
|
description: Make your PG migrations safe.
|
155
|
-
email:
|
170
|
+
email:
|
156
171
|
executables: []
|
157
172
|
extensions: []
|
158
173
|
extra_rdoc_files: []
|
@@ -163,7 +178,8 @@ files:
|
|
163
178
|
- lib/safe-pg-migrations/base.rb
|
164
179
|
- lib/safe-pg-migrations/configuration.rb
|
165
180
|
- lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
|
166
|
-
- lib/safe-pg-migrations/plugins/
|
181
|
+
- lib/safe-pg-migrations/plugins/idempotent_statements.rb
|
182
|
+
- lib/safe-pg-migrations/plugins/legacy_active_record_support.rb
|
167
183
|
- lib/safe-pg-migrations/plugins/statement_insurer.rb
|
168
184
|
- lib/safe-pg-migrations/plugins/statement_retrier.rb
|
169
185
|
- lib/safe-pg-migrations/plugins/useless_statements_logger.rb
|
@@ -173,7 +189,12 @@ files:
|
|
173
189
|
homepage: https://github.com/doctolib/safe-pg-migrations
|
174
190
|
licenses:
|
175
191
|
- MIT
|
176
|
-
metadata:
|
192
|
+
metadata:
|
193
|
+
bug_tracker_uri: https://github.com/doctolib/safe-pg-migrations/issues
|
194
|
+
homepage_uri: https://github.com/doctolib/safe-pg-migrations#safe-pg-migrations
|
195
|
+
mailing_list_uri: https://doctolib.engineering/engineering-news-ruby-rails-react
|
196
|
+
source_code_uri: https://github.com/doctolib/safe-pg-migrations
|
197
|
+
contributors_uri: https://github.com/doctolib/safe-pg-migrations/graphs/contributors
|
177
198
|
post_install_message:
|
178
199
|
rdoc_options: []
|
179
200
|
require_paths:
|
@@ -183,6 +204,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
183
204
|
- - ">="
|
184
205
|
- !ruby/object:Gem::Version
|
185
206
|
version: '2.5'
|
207
|
+
- - "<"
|
208
|
+
- !ruby/object:Gem::Version
|
209
|
+
version: '4'
|
186
210
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
187
211
|
requirements:
|
188
212
|
- - ">="
|