safe-pg-migrations 1.2.2 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +12 -7
- data/lib/safe-pg-migrations/base.rb +18 -5
- data/lib/safe-pg-migrations/configuration.rb +2 -0
- data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb +59 -24
- data/lib/safe-pg-migrations/plugins/{idem_potent_statements.rb → idempotent_statements.rb} +27 -15
- data/lib/safe-pg-migrations/plugins/legacy_active_record_support.rb +23 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +23 -26
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +1 -0
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +12 -7
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +29 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2fb891279f68c60f15f81b6e7ccbc425750443c3ee294080c2286bb059ed2e89
|
4
|
+
data.tar.gz: f4a8d59c4ee5abf004e4141db1d09d3cde33ff42f180c03526767535f20a5acd
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5b8c48e973fc0296a54c2a8cceaa3b3c8c8d3380cf15a87ea5154e2ebfc28a8aa852172fe020946d8c758a4d1518bf22fe06eda08d31d318d9176df88820ac8e
|
7
|
+
data.tar.gz: e88101b365a400e21ca76cf64b1818fba36e8875958989e1864391faf476b793596bbcfbd6bed4c53db46d93a16981334f45fe83124dc04e2eb7132029a712e3
|
data/README.md
CHANGED
@@ -2,6 +2,8 @@
|
|
2
2
|
|
3
3
|
ActiveRecord migrations for Postgres made safe.
|
4
4
|
|
5
|
+

|
6
|
+
|
5
7
|
## Requirements
|
6
8
|
|
7
9
|
- Ruby 2.5+
|
@@ -91,7 +93,7 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
|
|
91
93
|
- In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
|
92
94
|
- In order to add an index concurrently, we have to be outside a transaction.
|
93
95
|
|
94
|
-
Note that if a migration fails, it won't be
|
96
|
+
Note that if a migration fails, it won't be rolled back. This can result in migrations being partially applied. In that case, they need to be manually reverted.
|
95
97
|
|
96
98
|
</details>
|
97
99
|
|
@@ -108,7 +110,7 @@ PG will still needs to update every row of the table, and will most likely state
|
|
108
110
|
|
109
111
|
<blockquote>
|
110
112
|
|
111
|
-
**Note: Pre-
|
113
|
+
**Note: Pre-postgres 11**
|
112
114
|
Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
113
115
|
|
114
116
|
**Safe PG Migrations** makes it safe by:
|
@@ -120,7 +122,7 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
|
|
120
122
|
|
121
123
|
Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
|
122
124
|
|
123
|
-
</blockquote>
|
125
|
+
</blockquote>
|
124
126
|
|
125
127
|
</details>
|
126
128
|
|
@@ -137,7 +139,7 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
137
139
|
|
138
140
|
</details>
|
139
141
|
|
140
|
-
<details><summary id="safe_add_foreign_key">
|
142
|
+
<details><summary id="safe_add_foreign_key">Safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
|
141
143
|
|
142
144
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
143
145
|
|
@@ -150,7 +152,7 @@ Adding the constraint itself is rather fast, the major part of the time is spent
|
|
150
152
|
|
151
153
|
<details><summary>Retry after lock timeout</summary>
|
152
154
|
|
153
|
-
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of
|
155
|
+
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
154
156
|
</details>
|
155
157
|
|
156
158
|
<details><summary>Blocking activity logging</summary>
|
@@ -208,6 +210,8 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
|
|
208
210
|
```ruby
|
209
211
|
SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
|
210
212
|
|
213
|
+
SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
|
214
|
+
|
211
215
|
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
212
216
|
|
213
217
|
SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
|
@@ -217,11 +221,11 @@ SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retry
|
|
217
221
|
SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
|
218
222
|
```
|
219
223
|
|
220
|
-
##
|
224
|
+
## Running tests
|
221
225
|
|
222
226
|
```bash
|
223
227
|
bundle
|
224
|
-
psql -h localhost -
|
228
|
+
psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
|
225
229
|
rake test
|
226
230
|
```
|
227
231
|
|
@@ -255,3 +259,4 @@ Interesting reads:
|
|
255
259
|
- [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
|
256
260
|
- [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
|
257
261
|
- [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
|
262
|
+
- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
|
@@ -1,21 +1,24 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
|
+
require 'ruby2_keywords'
|
3
4
|
require 'safe-pg-migrations/configuration'
|
4
5
|
require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
5
6
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
6
7
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
7
8
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
8
|
-
require 'safe-pg-migrations/plugins/
|
9
|
+
require 'safe-pg-migrations/plugins/idempotent_statements'
|
9
10
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
11
|
+
require 'safe-pg-migrations/plugins/legacy_active_record_support'
|
10
12
|
|
11
13
|
module SafePgMigrations
|
12
14
|
# Order matters: the bottom-most plugin will have precedence
|
13
15
|
PLUGINS = [
|
14
16
|
BlockingActivityLogger,
|
15
|
-
|
17
|
+
IdempotentStatements,
|
16
18
|
StatementRetrier,
|
17
19
|
StatementInsurer,
|
18
20
|
UselessStatementsLogger,
|
21
|
+
LegacyActiveRecordSupport,
|
19
22
|
].freeze
|
20
23
|
|
21
24
|
class << self
|
@@ -50,13 +53,13 @@ module SafePgMigrations
|
|
50
53
|
@alternate_connection = nil
|
51
54
|
end
|
52
55
|
|
53
|
-
def say(*args)
|
56
|
+
ruby2_keywords def say(*args)
|
54
57
|
return unless current_migration
|
55
58
|
|
56
59
|
current_migration.say(*args)
|
57
60
|
end
|
58
61
|
|
59
|
-
def say_method_call(method, *args)
|
62
|
+
ruby2_keywords def say_method_call(method, *args)
|
60
63
|
say "#{method}(#{args.map(&:inspect) * ', '})", true
|
61
64
|
end
|
62
65
|
|
@@ -84,13 +87,23 @@ module SafePgMigrations
|
|
84
87
|
true
|
85
88
|
end
|
86
89
|
|
87
|
-
SAFE_METHODS = %i[
|
90
|
+
SAFE_METHODS = %i[
|
91
|
+
execute
|
92
|
+
add_column
|
93
|
+
add_index
|
94
|
+
add_reference
|
95
|
+
add_belongs_to
|
96
|
+
change_column_null
|
97
|
+
add_foreign_key
|
98
|
+
].freeze
|
99
|
+
|
88
100
|
SAFE_METHODS.each do |method|
|
89
101
|
define_method method do |*args|
|
90
102
|
return super(*args) unless respond_to?(:safety_assured)
|
91
103
|
|
92
104
|
safety_assured { super(*args) }
|
93
105
|
end
|
106
|
+
ruby2_keywords method
|
94
107
|
end
|
95
108
|
end
|
96
109
|
end
|
@@ -6,6 +6,7 @@ module SafePgMigrations
|
|
6
6
|
class Configuration
|
7
7
|
attr_accessor :safe_timeout
|
8
8
|
attr_accessor :blocking_activity_logger_margin
|
9
|
+
attr_accessor :blocking_activity_logger_verbose
|
9
10
|
attr_accessor :batch_size
|
10
11
|
attr_accessor :retry_delay
|
11
12
|
attr_accessor :max_tries
|
@@ -13,6 +14,7 @@ module SafePgMigrations
|
|
13
14
|
def initialize
|
14
15
|
self.safe_timeout = 5.seconds
|
15
16
|
self.blocking_activity_logger_margin = 1.second
|
17
|
+
self.blocking_activity_logger_verbose = true
|
16
18
|
self.batch_size = 1000
|
17
19
|
self.retry_delay = 1.minute
|
18
20
|
self.max_tries = 5
|
@@ -1,28 +1,19 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module BlockingActivityLogger
|
5
|
-
|
6
|
-
|
7
|
-
|
8
|
-
|
9
|
-
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
15
|
-
|
16
|
-
|
17
|
-
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
|
18
|
-
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
|
19
|
-
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
|
20
|
-
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
|
21
|
-
AND blocking_locks.pid != blocked_locks.pid
|
22
|
-
JOIN pg_catalog.pg_stat_activity blocking_activity
|
23
|
-
ON blocking_activity.pid = blocking_locks.pid
|
24
|
-
WHERE blocked_locks.pid = %d
|
25
|
-
SQL
|
4
|
+
module BlockingActivityLogger # rubocop:disable Metrics/ModuleLength
|
5
|
+
FILTERED_COLUMNS = %w[
|
6
|
+
blocked_activity.xact_start
|
7
|
+
blocked_locks.locktype
|
8
|
+
blocked_locks.mode
|
9
|
+
blocking_activity.pid
|
10
|
+
blocked_locks.transactionid
|
11
|
+
].freeze
|
12
|
+
|
13
|
+
VERBOSE_COLUMNS = %w[
|
14
|
+
blocking_activity.query
|
15
|
+
blocked_activity.xact_start
|
16
|
+
].freeze
|
26
17
|
|
27
18
|
%i[
|
28
19
|
add_column remove_column add_foreign_key remove_foreign_key change_column_default change_column_null create_table
|
@@ -30,10 +21,37 @@ module SafePgMigrations
|
|
30
21
|
define_method method do |*args, &block|
|
31
22
|
log_blocking_queries { super(*args, &block) }
|
32
23
|
end
|
24
|
+
ruby2_keywords method
|
33
25
|
end
|
34
26
|
|
35
27
|
private
|
36
28
|
|
29
|
+
def select_blocking_queries_sql
|
30
|
+
columns = SafePgMigrations.config.blocking_activity_logger_verbose ? VERBOSE_COLUMNS : FILTERED_COLUMNS
|
31
|
+
|
32
|
+
<<~SQL.squish
|
33
|
+
SELECT #{columns.join(', ')}
|
34
|
+
FROM pg_catalog.pg_locks blocked_locks
|
35
|
+
JOIN pg_catalog.pg_stat_activity blocked_activity
|
36
|
+
ON blocked_activity.pid = blocked_locks.pid
|
37
|
+
JOIN pg_catalog.pg_locks blocking_locks
|
38
|
+
ON blocking_locks.locktype = blocked_locks.locktype
|
39
|
+
AND blocking_locks.DATABASE IS NOT DISTINCT FROM blocked_locks.DATABASE
|
40
|
+
AND blocking_locks.relation IS NOT DISTINCT FROM blocked_locks.relation
|
41
|
+
AND blocking_locks.page IS NOT DISTINCT FROM blocked_locks.page
|
42
|
+
AND blocking_locks.tuple IS NOT DISTINCT FROM blocked_locks.tuple
|
43
|
+
AND blocking_locks.virtualxid IS NOT DISTINCT FROM blocked_locks.virtualxid
|
44
|
+
AND blocking_locks.transactionid IS NOT DISTINCT FROM blocked_locks.transactionid
|
45
|
+
AND blocking_locks.classid IS NOT DISTINCT FROM blocked_locks.classid
|
46
|
+
AND blocking_locks.objid IS NOT DISTINCT FROM blocked_locks.objid
|
47
|
+
AND blocking_locks.objsubid IS NOT DISTINCT FROM blocked_locks.objsubid
|
48
|
+
AND blocking_locks.pid != blocked_locks.pid
|
49
|
+
JOIN pg_catalog.pg_stat_activity blocking_activity
|
50
|
+
ON blocking_activity.pid = blocking_locks.pid
|
51
|
+
WHERE blocked_locks.pid = %d
|
52
|
+
SQL
|
53
|
+
end
|
54
|
+
|
37
55
|
def log_blocking_queries
|
38
56
|
delay_before_logging =
|
39
57
|
SafePgMigrations.config.safe_timeout - SafePgMigrations.config.blocking_activity_logger_margin
|
@@ -41,7 +59,7 @@ module SafePgMigrations
|
|
41
59
|
blocking_queries_retriever_thread =
|
42
60
|
Thread.new do
|
43
61
|
sleep delay_before_logging
|
44
|
-
SafePgMigrations.alternate_connection.query(
|
62
|
+
SafePgMigrations.alternate_connection.query(select_blocking_queries_sql % raw_connection.backend_pid)
|
45
63
|
end
|
46
64
|
|
47
65
|
yield
|
@@ -66,7 +84,7 @@ module SafePgMigrations
|
|
66
84
|
"Statement was being blocked by the following #{'query'.pluralize(queries.size)}:", true
|
67
85
|
)
|
68
86
|
SafePgMigrations.say '', true
|
69
|
-
queries
|
87
|
+
output_blocking_queries(queries)
|
70
88
|
SafePgMigrations.say(
|
71
89
|
'Beware, some of those queries might run in a transaction. In this case the locking query might be '\
|
72
90
|
'located elsewhere in the transaction',
|
@@ -78,7 +96,24 @@ module SafePgMigrations
|
|
78
96
|
raise
|
79
97
|
end
|
80
98
|
|
99
|
+
def output_blocking_queries(queries)
|
100
|
+
if SafePgMigrations.config.blocking_activity_logger_verbose
|
101
|
+
queries.each { |query, start_time| SafePgMigrations.say "#{format_start_time start_time}: #{query}", true }
|
102
|
+
else
|
103
|
+
queries.each do |start_time, locktype, mode, pid, transactionid|
|
104
|
+
SafePgMigrations.say(
|
105
|
+
"#{format_start_time(start_time)}: lock type: #{locktype || 'null'}, " \
|
106
|
+
"lock mode: #{mode || 'null'}, " \
|
107
|
+
"lock pid: #{pid || 'null'}, " \
|
108
|
+
"lock transactionid: #{transactionid || 'null'}",
|
109
|
+
true
|
110
|
+
)
|
111
|
+
end
|
112
|
+
end
|
113
|
+
end
|
114
|
+
|
81
115
|
def format_start_time(start_time, reference_time = Time.now)
|
116
|
+
start_time = Time.parse(start_time) unless start_time.is_a? Time
|
82
117
|
duration = (reference_time - start_time).round
|
83
118
|
"transaction started #{duration} #{'second'.pluralize(duration)} ago"
|
84
119
|
end
|
@@ -1,30 +1,40 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module
|
5
|
-
def add_index(table_name, column_name,
|
6
|
-
|
7
|
-
return super unless index_name_exists?(table_name, index_name)
|
4
|
+
module IdempotentStatements
|
5
|
+
ruby2_keywords def add_index(table_name, column_name, *args)
|
6
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
8
7
|
|
9
|
-
|
8
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
10
9
|
|
11
|
-
|
10
|
+
return super unless index_name_exists?(index_definition.table, index_definition.name)
|
11
|
+
|
12
|
+
if index_valid?(index_definition.name)
|
13
|
+
SafePgMigrations.say(
|
14
|
+
"/!\\ Index '#{index_definition.name}' already exists in '#{table_name}'. Skipping statement.",
|
15
|
+
true
|
16
|
+
)
|
17
|
+
return
|
18
|
+
end
|
19
|
+
|
20
|
+
remove_index(table_name, name: index_definition.name)
|
12
21
|
super
|
13
22
|
end
|
14
23
|
|
15
|
-
def add_column(table_name, column_name, type,
|
24
|
+
ruby2_keywords def add_column(table_name, column_name, type, *)
|
16
25
|
return super unless column_exists?(table_name, column_name)
|
17
26
|
|
18
27
|
SafePgMigrations.say("/!\\ Column '#{column_name}' already exists in '#{table_name}'. Skipping statement.", true)
|
19
28
|
end
|
20
29
|
|
21
|
-
def remove_column(table_name, column_name, type = nil,
|
30
|
+
ruby2_keywords def remove_column(table_name, column_name, type = nil, *)
|
22
31
|
return super if column_exists?(table_name, column_name)
|
23
32
|
|
24
33
|
SafePgMigrations.say("/!\\ Column '#{column_name}' not found on table '#{table_name}'. Skipping statement.", true)
|
25
34
|
end
|
26
35
|
|
27
|
-
def remove_index(table_name,
|
36
|
+
ruby2_keywords def remove_index(table_name, *args)
|
37
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
28
38
|
index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, options)
|
29
39
|
|
30
40
|
return super if index_name_exists?(table_name, index_name)
|
@@ -32,9 +42,10 @@ module SafePgMigrations
|
|
32
42
|
SafePgMigrations.say("/!\\ Index '#{index_name}' not found on table '#{table_name}'. Skipping statement.", true)
|
33
43
|
end
|
34
44
|
|
35
|
-
def add_foreign_key(from_table, to_table,
|
36
|
-
|
37
|
-
|
45
|
+
ruby2_keywords def add_foreign_key(from_table, to_table, *args)
|
46
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
47
|
+
suboptions = options.slice(:name, :column)
|
48
|
+
return super unless foreign_key_exists?(from_table, suboptions.present? ? nil : to_table, **suboptions)
|
38
49
|
|
39
50
|
SafePgMigrations.say(
|
40
51
|
"/!\\ Foreign key '#{from_table}' -> '#{to_table}' already exists. Skipping statement.",
|
@@ -42,19 +53,20 @@ module SafePgMigrations
|
|
42
53
|
)
|
43
54
|
end
|
44
55
|
|
45
|
-
def create_table(table_name,
|
56
|
+
ruby2_keywords def create_table(table_name, *args)
|
57
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
46
58
|
return super if options[:force] || !table_exists?(table_name)
|
47
59
|
|
48
60
|
SafePgMigrations.say "/!\\ Table '#{table_name}' already exists.", true
|
49
61
|
|
50
|
-
td = create_table_definition(table_name,
|
62
|
+
td = create_table_definition(table_name, *args)
|
51
63
|
|
52
64
|
yield td if block_given?
|
53
65
|
|
54
66
|
SafePgMigrations.say(td.indexes.empty? ? '-- Skipping statement' : '-- Creating indexes', true)
|
55
67
|
|
56
68
|
td.indexes.each do |column_name, index_options|
|
57
|
-
add_index(table_name, column_name, index_options)
|
69
|
+
add_index(table_name, column_name, **index_options)
|
58
70
|
end
|
59
71
|
end
|
60
72
|
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module LegacyActiveRecordSupport
|
5
|
+
ruby2_keywords def validate_foreign_key(from_table, to_table = nil, **options)
|
6
|
+
return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
|
7
|
+
|
8
|
+
super(from_table, to_table, **options)
|
9
|
+
end
|
10
|
+
|
11
|
+
ruby2_keywords def foreign_key_exists?(from_table, to_table = nil, **options)
|
12
|
+
return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
|
13
|
+
|
14
|
+
super(from_table, to_table, **options)
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def satisfied?(version)
|
20
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
@@ -8,9 +8,11 @@ module SafePgMigrations
|
|
8
8
|
define_method method do |*args, &block|
|
9
9
|
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) { super(*args, &block) }
|
10
10
|
end
|
11
|
+
ruby2_keywords method
|
11
12
|
end
|
12
13
|
|
13
|
-
def add_column(table_name, column_name, type,
|
14
|
+
ruby2_keywords def add_column(table_name, column_name, type, *args) # rubocop:disable Metrics/CyclomaticComplexity
|
15
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
14
16
|
return super if SafePgMigrations.pg_version_num >= PG_11_VERSION_NUM
|
15
17
|
|
16
18
|
default = options.delete(:default)
|
@@ -36,17 +38,21 @@ module SafePgMigrations
|
|
36
38
|
end
|
37
39
|
end
|
38
40
|
|
39
|
-
def add_foreign_key(from_table, to_table,
|
41
|
+
ruby2_keywords def add_foreign_key(from_table, to_table, *args)
|
42
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
40
43
|
validate_present = options.key? :validate
|
41
44
|
options[:validate] = false unless validate_present
|
45
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) do
|
46
|
+
super(from_table, to_table, **options)
|
47
|
+
end
|
42
48
|
|
43
|
-
|
49
|
+
return if validate_present
|
44
50
|
|
45
|
-
|
46
|
-
without_statement_timeout { validate_foreign_key from_table,
|
51
|
+
suboptions = options.slice(:name, :column)
|
52
|
+
without_statement_timeout { validate_foreign_key from_table, suboptions.present? ? nil : to_table, **suboptions }
|
47
53
|
end
|
48
54
|
|
49
|
-
def create_table(*)
|
55
|
+
ruby2_keywords def create_table(*)
|
50
56
|
with_setting(:statement_timeout, SafePgMigrations.config.pg_safe_timeout) do
|
51
57
|
super do |td|
|
52
58
|
yield td if block_given?
|
@@ -65,34 +71,25 @@ module SafePgMigrations
|
|
65
71
|
options[:algorithm] = :concurrently
|
66
72
|
end
|
67
73
|
|
68
|
-
SafePgMigrations.say_method_call(:add_index, table_name, column_name, options)
|
74
|
+
SafePgMigrations.say_method_call(:add_index, table_name, column_name, **options)
|
69
75
|
|
70
|
-
without_timeout { super }
|
76
|
+
without_timeout { super(table_name, column_name, **options) }
|
71
77
|
end
|
72
78
|
|
73
|
-
def remove_index(table_name,
|
74
|
-
options = { column:
|
75
|
-
options[:algorithm] = :concurrently
|
76
|
-
SafePgMigrations.say_method_call(:remove_index, table_name, options)
|
79
|
+
ruby2_keywords def remove_index(table_name, *args)
|
80
|
+
options = args.last.is_a?(Hash) ? args.last : { column: args.last }
|
81
|
+
options[:algorithm] = :concurrently unless options.key?(:algorithm)
|
82
|
+
SafePgMigrations.say_method_call(:remove_index, table_name, **options)
|
77
83
|
|
78
|
-
without_timeout { super }
|
84
|
+
without_timeout { super(table_name, **options) }
|
79
85
|
end
|
80
86
|
|
81
87
|
def backfill_column_default(table_name, column_name)
|
82
|
-
|
88
|
+
model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
|
83
89
|
quoted_column_name = quote_column_name(column_name)
|
84
|
-
|
85
|
-
|
86
|
-
|
87
|
-
SELECT id FROM #{quoted_table_name} WHERE id > #{primary_key_offset}
|
88
|
-
ORDER BY id LIMIT #{SafePgMigrations.config.batch_size}
|
89
|
-
SQL
|
90
|
-
break if ids.empty?
|
91
|
-
|
92
|
-
primary_key_offset = ids.last
|
93
|
-
execute <<~SQL.squish
|
94
|
-
UPDATE #{quoted_table_name} SET #{quoted_column_name} = DEFAULT WHERE id IN (#{ids.join(',')})
|
95
|
-
SQL
|
90
|
+
|
91
|
+
model.in_batches(of: SafePgMigrations.config.batch_size).each do |relation|
|
92
|
+
relation.update_all("#{quoted_column_name} = DEFAULT")
|
96
93
|
end
|
97
94
|
end
|
98
95
|
|
@@ -2,22 +2,27 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module UselessStatementsLogger
|
5
|
-
|
6
|
-
|
7
|
-
|
5
|
+
class << self
|
6
|
+
ruby2_keywords def warn_useless(action, link = nil, *args)
|
7
|
+
SafePgMigrations.say "/!\\ No need to explicitly use #{action}, safe-pg-migrations does it for you", *args
|
8
|
+
SafePgMigrations.say "\t see #{link} for more details", *args if link
|
9
|
+
end
|
8
10
|
end
|
9
11
|
|
10
|
-
def add_index(
|
12
|
+
ruby2_keywords def add_index(*args)
|
13
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
11
14
|
warn_for_index(**options)
|
12
15
|
super
|
13
16
|
end
|
14
17
|
|
15
|
-
def remove_index(table_name,
|
16
|
-
|
18
|
+
ruby2_keywords def remove_index(table_name, *args)
|
19
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
20
|
+
warn_for_index(**options) unless options.empty?
|
17
21
|
super
|
18
22
|
end
|
19
23
|
|
20
|
-
def add_foreign_key(
|
24
|
+
ruby2_keywords def add_foreign_key(*args)
|
25
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
21
26
|
if options[:validate] == false
|
22
27
|
UselessStatementsLogger.warn_useless '`validate: :false`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_foreign_key'
|
23
28
|
end
|
metadata
CHANGED
@@ -1,15 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: safe-pg-migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthieu Prat
|
8
8
|
- Romain Choquet
|
9
|
+
- Thomas Hareau
|
9
10
|
autorequire:
|
10
11
|
bindir: bin
|
11
12
|
cert_chain: []
|
12
|
-
date:
|
13
|
+
date: 2022-02-24 00:00:00.000000000 Z
|
13
14
|
dependencies:
|
14
15
|
- !ruby/object:Gem::Dependency
|
15
16
|
name: activerecord
|
@@ -39,6 +40,20 @@ dependencies:
|
|
39
40
|
- - ">="
|
40
41
|
- !ruby/object:Gem::Version
|
41
42
|
version: '5.2'
|
43
|
+
- !ruby/object:Gem::Dependency
|
44
|
+
name: ruby2_keywords
|
45
|
+
requirement: !ruby/object:Gem::Requirement
|
46
|
+
requirements:
|
47
|
+
- - ">="
|
48
|
+
- !ruby/object:Gem::Version
|
49
|
+
version: 0.0.4
|
50
|
+
type: :runtime
|
51
|
+
prerelease: false
|
52
|
+
version_requirements: !ruby/object:Gem::Requirement
|
53
|
+
requirements:
|
54
|
+
- - ">="
|
55
|
+
- !ruby/object:Gem::Version
|
56
|
+
version: 0.0.4
|
42
57
|
- !ruby/object:Gem::Dependency
|
43
58
|
name: bundler
|
44
59
|
requirement: !ruby/object:Gem::Requirement
|
@@ -152,7 +167,7 @@ dependencies:
|
|
152
167
|
- !ruby/object:Gem::Version
|
153
168
|
version: '0'
|
154
169
|
description: Make your PG migrations safe.
|
155
|
-
email:
|
170
|
+
email:
|
156
171
|
executables: []
|
157
172
|
extensions: []
|
158
173
|
extra_rdoc_files: []
|
@@ -163,7 +178,8 @@ files:
|
|
163
178
|
- lib/safe-pg-migrations/base.rb
|
164
179
|
- lib/safe-pg-migrations/configuration.rb
|
165
180
|
- lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
|
166
|
-
- lib/safe-pg-migrations/plugins/
|
181
|
+
- lib/safe-pg-migrations/plugins/idempotent_statements.rb
|
182
|
+
- lib/safe-pg-migrations/plugins/legacy_active_record_support.rb
|
167
183
|
- lib/safe-pg-migrations/plugins/statement_insurer.rb
|
168
184
|
- lib/safe-pg-migrations/plugins/statement_retrier.rb
|
169
185
|
- lib/safe-pg-migrations/plugins/useless_statements_logger.rb
|
@@ -173,7 +189,12 @@ files:
|
|
173
189
|
homepage: https://github.com/doctolib/safe-pg-migrations
|
174
190
|
licenses:
|
175
191
|
- MIT
|
176
|
-
metadata:
|
192
|
+
metadata:
|
193
|
+
bug_tracker_uri: https://github.com/doctolib/safe-pg-migrations/issues
|
194
|
+
homepage_uri: https://github.com/doctolib/safe-pg-migrations#safe-pg-migrations
|
195
|
+
mailing_list_uri: https://doctolib.engineering/engineering-news-ruby-rails-react
|
196
|
+
source_code_uri: https://github.com/doctolib/safe-pg-migrations
|
197
|
+
contributors_uri: https://github.com/doctolib/safe-pg-migrations/graphs/contributors
|
177
198
|
post_install_message:
|
178
199
|
rdoc_options: []
|
179
200
|
require_paths:
|
@@ -183,6 +204,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
183
204
|
- - ">="
|
184
205
|
- !ruby/object:Gem::Version
|
185
206
|
version: '2.5'
|
207
|
+
- - "<"
|
208
|
+
- !ruby/object:Gem::Version
|
209
|
+
version: '4'
|
186
210
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
187
211
|
requirements:
|
188
212
|
- - ">="
|