safe-pg-migrations 2.0.0 → 2.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +98 -3
- data/lib/safe-pg-migrations/base.rb +12 -3
- data/lib/safe-pg-migrations/configuration.rb +15 -5
- data/lib/safe-pg-migrations/helpers/batch_over.rb +56 -0
- data/lib/safe-pg-migrations/helpers/index_helper.rb +24 -0
- data/lib/safe-pg-migrations/helpers/satisfied_helper.rb +31 -0
- data/lib/safe-pg-migrations/plugins/idempotent_statements.rb +43 -13
- data/lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb +52 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +44 -5
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +1 -1
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +8 -0
- data/lib/safe-pg-migrations/polyfills/index_definition_polyfill.rb +2 -2
- data/lib/safe-pg-migrations/polyfills/verbose_query_logs_polyfill.rb +2 -4
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +6 -3
- data/lib/safe-pg-migrations/polyfills/satisfied_helper.rb +0 -13
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2fafd98448f0ff697da386a337dc22753be2144feedba4793f87a351581a5607
|
4
|
+
data.tar.gz: 9653ae6395d4c9f27b69dba9dee9142a4535384c42e8d05fac9ca2e2ef1bdce6
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 64297c9ba4f040e7afa21fefa2acb2a26e27fb0bfee3d9f9cbf902ee4bb0ac087f4e22591a7b5470cf73dc7bc73345e59904649c02d7540bd4693372e7069549
|
7
|
+
data.tar.gz: 7a2640fd2494480d560cfa6f19059550ecc316eaf729df6d8f97a434c189f4565c49d47acbb02f437d9b1285a466a5615a79a84d6ba1dfb3d94bb54f87b9e788
|
data/README.md
CHANGED
@@ -109,7 +109,43 @@ Beware though, when adding a volatile default value:
|
|
109
109
|
```ruby
|
110
110
|
add_column :users, :created_at, default: 'clock_timestamp()'
|
111
111
|
```
|
112
|
-
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case,
|
112
|
+
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, **Safe PG Migrations** can automatically backfill data when the option `default_value_backfill:` is set to `:update_in_batches`.
|
113
|
+
|
114
|
+
</details>
|
115
|
+
|
116
|
+
<details>
|
117
|
+
<summary>Safe add_column - adding a volatile default value</summary>
|
118
|
+
|
119
|
+
**Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a volatile default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
|
120
|
+
|
121
|
+
```ruby
|
122
|
+
add_column :users, :created_at, default: 'clock_timestamp()', default_value_backfill: :update_in_batches
|
123
|
+
```
|
124
|
+
|
125
|
+
More specifically, it will:
|
126
|
+
|
127
|
+
1. create the column without default value and without null constraint. This ensure the `ACCESS EXCLUSIVE` lock is acquired for the least amount of time;
|
128
|
+
2. add the default value, without data backfill. An `ACCESS EXCLUSIVE` lock is acquired and released immediately;
|
129
|
+
3. backfill data, in batch of `SafePgMigrations.config.backfill_batch_size` and with a pause of `SafePgMigrations.config.backfill_pause` between each batch;
|
130
|
+
4. change the column to `null: false`, if defined in the parameters, following the algorithm we have defined below.
|
131
|
+
|
132
|
+
---
|
133
|
+
**NOTE**
|
134
|
+
|
135
|
+
Data backfill take time. If your table is big, your migrations will (safely) hangs for a while. You might want to backfill data manually instead, to do so you will need two migrations
|
136
|
+
|
137
|
+
1. First migration :
|
138
|
+
|
139
|
+
a. adds the column without default and without null constraint;
|
140
|
+
|
141
|
+
b. add the default value.
|
142
|
+
|
143
|
+
2. manual data backfill (rake task, manual operation, ...)
|
144
|
+
3. Second migration which change the column to null false (with **Safe PG Migrations**, `change_column_null` is safe and can be used; see section below)
|
145
|
+
|
146
|
+
---
|
147
|
+
|
148
|
+
`default_value_backfill:` also accept the value `:auto` which is set by default. In this case, **Safe PG Migrations** will not backfill data and will let PostgreSQL handle it itself.
|
113
149
|
|
114
150
|
</details>
|
115
151
|
|
@@ -130,13 +166,70 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
130
166
|
|
131
167
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
132
168
|
|
133
|
-
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus
|
169
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus **Safe PG Migrations** ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
|
170
|
+
|
171
|
+
1. adding the constraint *without validation*, will not validate existing rows;
|
172
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
173
|
+
|
174
|
+
</details>
|
175
|
+
|
176
|
+
|
177
|
+
<details><summary id="safe_add_check_constraint">Safe <code>add_check_constraint</code> (ActiveRecord > 6.1)</summary>
|
178
|
+
|
179
|
+
Adding a check constraint requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
180
|
+
|
181
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
182
|
+
Thus **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least amount of time by
|
183
|
+
splitting the constraint addition in two steps:
|
134
184
|
|
135
185
|
1. adding the constraint *without validation*, will not validate existing rows;
|
136
186
|
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
137
187
|
|
138
188
|
</details>
|
139
189
|
|
190
|
+
<details><summary id="safe_change_column_null">Safe <code>change_column_null</code> (ActiveRecord and PG version dependant)</summary>
|
191
|
+
|
192
|
+
Changing the nullability of a column requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
193
|
+
|
194
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
195
|
+
|
196
|
+
**Safe PG Migrations** acts differently depending on the version you are on.
|
197
|
+
|
198
|
+
### Recent versions of PG and Active Record (> 12 and > 6.1)
|
199
|
+
|
200
|
+
Starting on PostgreSQL versions 12, adding the column NOT NULL constraint is safe if a check constraint validates the
|
201
|
+
nullability of the same column. **Safe PG Migrations** also relies on add_check_constraint, which was introduced in
|
202
|
+
ActiveRecord 6.1.
|
203
|
+
|
204
|
+
If these requirements are met, **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least
|
205
|
+
amount of time by splitting the constraint addition in several steps:
|
206
|
+
|
207
|
+
1. adding a `IS NOT NULL` constraint *without validation*, will not validate existing rows but block read or write;
|
208
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table;
|
209
|
+
3. changing the not null status of the column, thanks to the NOT NULL constraint without having to scan the table sequentially;
|
210
|
+
4. dropping the `IS NOT NULL` constraint.
|
211
|
+
|
212
|
+
### Older versions of PG or ActiveRecord
|
213
|
+
|
214
|
+
If the version of PostgreSQL is below 12, or if the version of ActiveRecord is below 6.1, **Safe PG Migrations** will only
|
215
|
+
wrap ActiveRecord method into a statement timeout and lock timeout.
|
216
|
+
|
217
|
+
### Call with a default parameter
|
218
|
+
|
219
|
+
Calling change_column_null with a default parameter [is dangerous](https://github.com/rails/rails/blob/716baea69f989b64f5bfeaff880c2512377bebab/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb#L446)
|
220
|
+
and is likely not to finish in the statement timeout defined by **Safe PG Migrations**. For this reason, when the default
|
221
|
+
parameter is given, **Safe PG Migrations** will simply forward it to activerecord methods without trying to improve it
|
222
|
+
|
223
|
+
### Dropping a NULL constraint
|
224
|
+
|
225
|
+
Dropping a null constraint still requires an `ACCESS EXCLUSIVE` lock, but does not require extra operation to reduce the
|
226
|
+
amount of time during which the lock is held. **Safe PG Migrations** only wrap methods of activerecord in lock and statement
|
227
|
+
timeouts
|
228
|
+
|
229
|
+
</details>
|
230
|
+
|
231
|
+
|
232
|
+
|
140
233
|
<details><summary>Retry after lock timeout</summary>
|
141
234
|
|
142
235
|
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
@@ -201,7 +294,9 @@ SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the ra
|
|
201
294
|
|
202
295
|
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
203
296
|
|
204
|
-
SafePgMigrations.config.
|
297
|
+
SafePgMigrations.config.backfill_batch_size = 100_000 # Size of the batches used for backfilling when adding a column with a default value
|
298
|
+
|
299
|
+
SafePgMigrations.config.backfill_pause = 0.5.second # Delay between each batch during a backfill. This ensure replication can happen safely.
|
205
300
|
|
206
301
|
SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
|
207
302
|
|
@@ -1,13 +1,16 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require 'safe-pg-migrations/configuration'
|
4
|
+
require 'safe-pg-migrations/helpers/satisfied_helper'
|
5
|
+
require 'safe-pg-migrations/helpers/index_helper'
|
6
|
+
require 'safe-pg-migrations/helpers/batch_over'
|
4
7
|
require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
5
8
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
9
|
+
require 'safe-pg-migrations/plugins/statement_insurer/add_column'
|
6
10
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
7
11
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
8
12
|
require 'safe-pg-migrations/plugins/idempotent_statements'
|
9
13
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
10
|
-
require 'safe-pg-migrations/polyfills/satisfied_helper'
|
11
14
|
require 'safe-pg-migrations/polyfills/index_definition_polyfill'
|
12
15
|
require 'safe-pg-migrations/polyfills/verbose_query_logs_polyfill'
|
13
16
|
|
@@ -23,15 +26,16 @@ module SafePgMigrations
|
|
23
26
|
].freeze
|
24
27
|
|
25
28
|
class << self
|
26
|
-
attr_reader :current_migration
|
29
|
+
attr_reader :current_migration, :pg_version_num
|
27
30
|
|
28
31
|
def setup_and_teardown(migration, connection, &block)
|
32
|
+
@pg_version_num = get_pg_version_num(connection)
|
29
33
|
@alternate_connection = nil
|
30
34
|
@current_migration = migration
|
31
35
|
stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
|
32
36
|
PLUGINS.each { |plugin| connection.extend(plugin) }
|
33
37
|
|
34
|
-
connection.with_setting
|
38
|
+
connection.with_setting :lock_timeout, SafePgMigrations.config.pg_lock_timeout, &block
|
35
39
|
ensure
|
36
40
|
close_alternate_connection
|
37
41
|
@current_migration = nil
|
@@ -72,6 +76,10 @@ module SafePgMigrations
|
|
72
76
|
def config
|
73
77
|
@config ||= Configuration.new
|
74
78
|
end
|
79
|
+
|
80
|
+
def get_pg_version_num(connection)
|
81
|
+
connection.query_value('SHOW server_version_num').to_i
|
82
|
+
end
|
75
83
|
end
|
76
84
|
|
77
85
|
module Migration
|
@@ -102,6 +110,7 @@ module SafePgMigrations
|
|
102
110
|
add_belongs_to
|
103
111
|
change_column_null
|
104
112
|
add_foreign_key
|
113
|
+
add_check_constraint
|
105
114
|
].freeze
|
106
115
|
|
107
116
|
SAFE_METHODS.each do |method|
|
@@ -4,22 +4,32 @@ require 'active_support/core_ext/numeric/time'
|
|
4
4
|
|
5
5
|
module SafePgMigrations
|
6
6
|
class Configuration
|
7
|
-
attr_accessor :safe_timeout, :blocking_activity_logger_margin, :blocking_activity_logger_verbose,
|
8
|
-
:retry_delay, :max_tries
|
7
|
+
attr_accessor :safe_timeout, :blocking_activity_logger_margin, :blocking_activity_logger_verbose,
|
8
|
+
:backfill_batch_size, :backfill_pause, :retry_delay, :max_tries
|
9
9
|
|
10
10
|
def initialize
|
11
11
|
self.safe_timeout = 5.seconds
|
12
12
|
self.blocking_activity_logger_margin = 1.second
|
13
13
|
self.blocking_activity_logger_verbose = true
|
14
|
-
self.
|
14
|
+
self.backfill_batch_size = 100_000
|
15
|
+
self.backfill_pause = 0.5.second
|
15
16
|
self.retry_delay = 1.minute
|
16
17
|
self.max_tries = 5
|
17
18
|
end
|
18
19
|
|
19
|
-
def
|
20
|
-
pg_duration
|
20
|
+
def pg_statement_timeout
|
21
|
+
pg_duration safe_timeout
|
21
22
|
end
|
22
23
|
|
24
|
+
def pg_lock_timeout
|
25
|
+
# if statement timeout and lock timeout have the same value, statement timeout will raise in priority. We actually
|
26
|
+
# need the opposite for BlockingActivityLogger to detect lock timeouts correctly.
|
27
|
+
# By reducing the lock timeout by a very small margin, we ensure that the lock timeout is raised in priority
|
28
|
+
pg_duration safe_timeout * 0.99
|
29
|
+
end
|
30
|
+
|
31
|
+
private
|
32
|
+
|
23
33
|
def pg_duration(duration)
|
24
34
|
value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
|
25
35
|
"#{value}#{unit}"
|
@@ -0,0 +1,56 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
class BatchOver
|
6
|
+
def initialize(model, of: SafePgMigrations.config.backfill_batch_size)
|
7
|
+
@model = model
|
8
|
+
@of = of
|
9
|
+
|
10
|
+
@current_range = nil
|
11
|
+
end
|
12
|
+
|
13
|
+
def each_batch
|
14
|
+
yield scope.where(primary_key => @current_range) while next_batch
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def next_batch
|
20
|
+
return if endless?
|
21
|
+
|
22
|
+
first = next_scope.take
|
23
|
+
|
24
|
+
return unless first
|
25
|
+
|
26
|
+
last = next_scope.offset(@of).take
|
27
|
+
|
28
|
+
first_key = first[primary_key]
|
29
|
+
last_key = last.nil? ? nil : last[primary_key]
|
30
|
+
|
31
|
+
@current_range = first_key...last_key
|
32
|
+
end
|
33
|
+
|
34
|
+
def next_scope
|
35
|
+
return scope if @current_range.nil?
|
36
|
+
return scope.none if endless?
|
37
|
+
|
38
|
+
scope.where(primary_key => @current_range.end..)
|
39
|
+
end
|
40
|
+
|
41
|
+
def scope
|
42
|
+
@model.order(primary_key => :asc)
|
43
|
+
end
|
44
|
+
|
45
|
+
def endless?
|
46
|
+
return false if @current_range.nil?
|
47
|
+
|
48
|
+
@current_range.end.nil?
|
49
|
+
end
|
50
|
+
|
51
|
+
def primary_key
|
52
|
+
@model.primary_key
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module IndexHelper
|
6
|
+
def index_definition(table_name, column_name, **options)
|
7
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
8
|
+
index_definition
|
9
|
+
end
|
10
|
+
|
11
|
+
private
|
12
|
+
|
13
|
+
def index_valid?(index_name)
|
14
|
+
query_value <<~SQL.squish
|
15
|
+
SELECT indisvalid
|
16
|
+
FROM pg_index i
|
17
|
+
JOIN pg_class c
|
18
|
+
ON i.indexrelid = c.oid
|
19
|
+
WHERE c.relname = '#{index_name}';
|
20
|
+
SQL
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module SatisfiedHelper
|
6
|
+
class << self
|
7
|
+
def satisfies_change_column_null_requirements?
|
8
|
+
satisfies_add_check_constraints? && SafePgMigrations.pg_version_num >= 120_000
|
9
|
+
end
|
10
|
+
|
11
|
+
def satisfies_add_check_constraints!
|
12
|
+
return if satisfies_add_check_constraints?
|
13
|
+
|
14
|
+
raise NotImplementedError, 'add_check_constraint is not supported in your ActiveRecord version'
|
15
|
+
end
|
16
|
+
|
17
|
+
def satisfies_add_check_constraints?
|
18
|
+
satisfied? '>=6.1.0'
|
19
|
+
end
|
20
|
+
|
21
|
+
def satisfies_add_column_update_rows_backfill?
|
22
|
+
satisfies_change_column_null_requirements?
|
23
|
+
end
|
24
|
+
|
25
|
+
def satisfied?(version)
|
26
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -2,6 +2,8 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module IdempotentStatements
|
5
|
+
include SafePgMigrations::Helpers::IndexHelper
|
6
|
+
|
5
7
|
ruby2_keywords def add_index(table_name, column_name, *args)
|
6
8
|
options = args.last.is_a?(Hash) ? args.last : {}
|
7
9
|
|
@@ -80,23 +82,51 @@ module SafePgMigrations
|
|
80
82
|
end
|
81
83
|
end
|
82
84
|
|
83
|
-
|
85
|
+
def add_check_constraint(table_name, expression, **options)
|
86
|
+
constraint_definition = check_constraint_for table_name,
|
87
|
+
**check_constraint_options(table_name, expression, options)
|
88
|
+
|
89
|
+
return super if constraint_definition.nil?
|
90
|
+
|
91
|
+
SafePgMigrations.say "/!\\ Constraint '#{constraint_definition.name}' already exists. Skipping statement.", true
|
92
|
+
end
|
93
|
+
|
94
|
+
def change_column_null(table_name, column_name, null, *)
|
95
|
+
column = column_for(table_name, column_name)
|
84
96
|
|
85
|
-
|
86
|
-
|
87
|
-
|
97
|
+
return super if column.null != null
|
98
|
+
|
99
|
+
SafePgMigrations.say(
|
100
|
+
"/!\\ Column '#{table_name}.#{column.name}' is already set to 'null: #{null}'. Skipping statement.",
|
101
|
+
true
|
102
|
+
)
|
88
103
|
end
|
89
104
|
|
90
|
-
|
105
|
+
def validate_check_constraint(table_name, **options)
|
106
|
+
constraint_definition = check_constraint_for!(table_name, **options)
|
107
|
+
|
108
|
+
return super unless constraint_definition.validated?
|
91
109
|
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
110
|
+
SafePgMigrations.say "/!\\ Constraint '#{constraint_definition.name}' already validated. Skipping statement.",
|
111
|
+
true
|
112
|
+
end
|
113
|
+
|
114
|
+
def change_column_default(table_name, column_name, default_or_changes)
|
115
|
+
column = column_for(table_name, column_name)
|
116
|
+
|
117
|
+
previous_alter_statement = change_column_default_for_alter(table_name, column_name, column.default)
|
118
|
+
new_alter_statement = change_column_default_for_alter(table_name, column_name, default_or_changes)
|
119
|
+
|
120
|
+
# NOTE: PG change_column_default is already idempotent.
|
121
|
+
# We try to detect it because it still takes an ACCESS EXCLUSIVE lock
|
122
|
+
|
123
|
+
return super if new_alter_statement != previous_alter_statement
|
124
|
+
|
125
|
+
SafePgMigrations.say(
|
126
|
+
"/!\\ Column '#{table_name}.#{column.name}' is already set to 'default: #{column.default}'. " \
|
127
|
+
'Skipping statement.',
|
128
|
+
true
|
129
|
+
)
|
100
130
|
end
|
101
131
|
end
|
102
132
|
end
|
@@ -0,0 +1,52 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module StatementInsurer
|
5
|
+
module AddColumn
|
6
|
+
ruby2_keywords def add_column(table_name, column_name, type, *args)
|
7
|
+
options = args.last.is_a?(Hash) && args.last
|
8
|
+
options ||= {}
|
9
|
+
|
10
|
+
if should_keep_default_implementation?(**options)
|
11
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) { return super }
|
12
|
+
end
|
13
|
+
|
14
|
+
default = options.delete(:default)
|
15
|
+
null = options.delete(:null)
|
16
|
+
|
17
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
18
|
+
SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, options)
|
19
|
+
super table_name, column_name, type, **options
|
20
|
+
end
|
21
|
+
|
22
|
+
SafePgMigrations.say_method_call(:change_column_default, table_name, column_name, default)
|
23
|
+
change_column_default(table_name, column_name, default)
|
24
|
+
|
25
|
+
SafePgMigrations.say_method_call(:backfill_column_default, table_name, column_name)
|
26
|
+
without_statement_timeout do
|
27
|
+
backfill_column_default(table_name, column_name)
|
28
|
+
end
|
29
|
+
|
30
|
+
change_column_null(table_name, column_name, null) if null == false
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
def should_keep_default_implementation?(default: nil, default_value_backfill: :auto, **)
|
36
|
+
default_value_backfill != :update_in_batches || !default ||
|
37
|
+
!SafePgMigrations::Helpers::SatisfiedHelper.satisfies_add_column_update_rows_backfill?
|
38
|
+
end
|
39
|
+
|
40
|
+
def backfill_column_default(table_name, column_name)
|
41
|
+
model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
|
42
|
+
quoted_column_name = quote_column_name(column_name)
|
43
|
+
|
44
|
+
SafePgMigrations::Helpers::BatchOver.new(model).each_batch do |batch|
|
45
|
+
batch
|
46
|
+
.update_all("#{quoted_column_name} = DEFAULT")
|
47
|
+
sleep SafePgMigrations.config.backfill_pause
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
@@ -2,18 +2,41 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module StatementInsurer
|
5
|
-
|
5
|
+
include AddColumn
|
6
|
+
|
7
|
+
%i[change_column].each do |method|
|
6
8
|
define_method method do |*args, &block|
|
7
|
-
with_setting(:statement_timeout, SafePgMigrations.config.
|
9
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) { super(*args, &block) }
|
8
10
|
end
|
9
11
|
ruby2_keywords method
|
10
12
|
end
|
11
13
|
|
14
|
+
def validate_check_constraint(table_name, **options)
|
15
|
+
without_statement_timeout do
|
16
|
+
super
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def add_check_constraint(table_name, expression, **options)
|
21
|
+
Helpers::SatisfiedHelper.satisfies_add_check_constraints!
|
22
|
+
return unless supports_check_constraints?
|
23
|
+
|
24
|
+
options = check_constraint_options(table_name, expression, options)
|
25
|
+
|
26
|
+
SafePgMigrations.say_method_call :add_check_constraint, table_name, expression, **options, validate: false
|
27
|
+
super table_name, expression, **options, validate: false
|
28
|
+
|
29
|
+
return unless options.fetch(:validate, true)
|
30
|
+
|
31
|
+
SafePgMigrations.say_method_call :validate_check_constraint, table_name, name: options[:name]
|
32
|
+
validate_check_constraint table_name, name: options[:name]
|
33
|
+
end
|
34
|
+
|
12
35
|
ruby2_keywords def add_foreign_key(from_table, to_table, *args)
|
13
36
|
options = args.last.is_a?(Hash) ? args.last : {}
|
14
|
-
validate_present = options.key?
|
37
|
+
validate_present = options.key?(:validate)
|
15
38
|
options[:validate] = false unless validate_present
|
16
|
-
with_setting(:statement_timeout, SafePgMigrations.config.
|
39
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
17
40
|
super(from_table, to_table, **options)
|
18
41
|
end
|
19
42
|
|
@@ -24,7 +47,7 @@ module SafePgMigrations
|
|
24
47
|
end
|
25
48
|
|
26
49
|
ruby2_keywords def create_table(*)
|
27
|
-
with_setting(:statement_timeout, SafePgMigrations.config.
|
50
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
28
51
|
super do |td|
|
29
52
|
yield td if block_given?
|
30
53
|
td.indexes.map! do |key, index_options|
|
@@ -57,6 +80,22 @@ module SafePgMigrations
|
|
57
80
|
without_timeout { super(table_name, **options) }
|
58
81
|
end
|
59
82
|
|
83
|
+
def change_column_null(table_name, column_name, null, default = nil)
|
84
|
+
if default || null || !Helpers::SatisfiedHelper.satisfies_change_column_null_requirements?
|
85
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) { return super }
|
86
|
+
end
|
87
|
+
|
88
|
+
add_check_constraint table_name, "#{column_name} IS NOT NULL"
|
89
|
+
|
90
|
+
SafePgMigrations.say_method_call :change_column_null, table_name, column_name, false
|
91
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
92
|
+
super table_name, column_name, false
|
93
|
+
end
|
94
|
+
|
95
|
+
SafePgMigrations.say_method_call :remove_check_constraint, table_name, "#{column_name} IS NOT NULL"
|
96
|
+
remove_check_constraint table_name, "#{column_name} IS NOT NULL"
|
97
|
+
end
|
98
|
+
|
60
99
|
def with_setting(key, value)
|
61
100
|
old_value = query_value("SHOW #{key}")
|
62
101
|
execute("SET #{key} TO #{quote(value)}")
|
@@ -3,7 +3,7 @@
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module StatementRetrier
|
5
5
|
RETRIABLE_SCHEMA_STATEMENTS = %i[
|
6
|
-
add_column add_foreign_key remove_foreign_key change_column_default change_column_null remove_column
|
6
|
+
add_column add_foreign_key remove_foreign_key change_column_default change_column_null remove_column drop_table
|
7
7
|
].freeze
|
8
8
|
|
9
9
|
RETRIABLE_SCHEMA_STATEMENTS.each do |method|
|
@@ -29,6 +29,14 @@ module SafePgMigrations
|
|
29
29
|
super
|
30
30
|
end
|
31
31
|
|
32
|
+
ruby2_keywords def add_check_constraint(*args)
|
33
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
34
|
+
if options[:validate] == false
|
35
|
+
UselessStatementsLogger.warn_useless '`validate: :false`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_check_constraint'
|
36
|
+
end
|
37
|
+
super
|
38
|
+
end
|
39
|
+
|
32
40
|
def warn_for_index(**options)
|
33
41
|
return unless options[:algorithm] == :concurrently
|
34
42
|
|
@@ -3,14 +3,14 @@
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module Polyfills
|
5
5
|
module IndexDefinitionPolyfill
|
6
|
-
include SatisfiedHelper
|
6
|
+
include Helpers::SatisfiedHelper
|
7
7
|
|
8
8
|
protected
|
9
9
|
|
10
10
|
IndexDefinition = Struct.new(:table, :name)
|
11
11
|
|
12
12
|
def index_definition(table_name, column_name, **options)
|
13
|
-
return super(table_name, column_name, **options) if satisfied? '>=6.1.0'
|
13
|
+
return super(table_name, column_name, **options) if Helpers::SatisfiedHelper.satisfied? '>=6.1.0'
|
14
14
|
|
15
15
|
index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
|
16
16
|
validate_index_length!(table_name, index_name, options.fetch(:internal, false))
|
@@ -4,16 +4,14 @@ module SafePgMigrations
|
|
4
4
|
module Polyfills
|
5
5
|
module VerboseQueryLogsPolyfill
|
6
6
|
class << self
|
7
|
-
include SatisfiedHelper
|
8
|
-
|
9
7
|
def verbose_query_logs
|
10
|
-
return ActiveRecord.verbose_query_logs if satisfied? '>=7.0.0'
|
8
|
+
return ActiveRecord.verbose_query_logs if Helpers::SatisfiedHelper.satisfied? '>=7.0.0'
|
11
9
|
|
12
10
|
ActiveRecord::Base.verbose_query_logs
|
13
11
|
end
|
14
12
|
|
15
13
|
def verbose_query_logs=(value)
|
16
|
-
if satisfied? '>=7.0.0'
|
14
|
+
if Helpers::SatisfiedHelper.satisfied? '>=7.0.0'
|
17
15
|
ActiveRecord.verbose_query_logs = value
|
18
16
|
return
|
19
17
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: safe-pg-migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthieu Prat
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2023-
|
13
|
+
date: 2023-06-29 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: activerecord
|
@@ -51,16 +51,19 @@ files:
|
|
51
51
|
- lib/safe-pg-migrations.rb
|
52
52
|
- lib/safe-pg-migrations/base.rb
|
53
53
|
- lib/safe-pg-migrations/configuration.rb
|
54
|
+
- lib/safe-pg-migrations/helpers/batch_over.rb
|
54
55
|
- lib/safe-pg-migrations/helpers/blocking_activity_formatter.rb
|
55
56
|
- lib/safe-pg-migrations/helpers/blocking_activity_selector.rb
|
57
|
+
- lib/safe-pg-migrations/helpers/index_helper.rb
|
58
|
+
- lib/safe-pg-migrations/helpers/satisfied_helper.rb
|
56
59
|
- lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
|
57
60
|
- lib/safe-pg-migrations/plugins/idempotent_statements.rb
|
58
61
|
- lib/safe-pg-migrations/plugins/statement_insurer.rb
|
62
|
+
- lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb
|
59
63
|
- lib/safe-pg-migrations/plugins/statement_retrier.rb
|
60
64
|
- lib/safe-pg-migrations/plugins/useless_statements_logger.rb
|
61
65
|
- lib/safe-pg-migrations/plugins/verbose_sql_logger.rb
|
62
66
|
- lib/safe-pg-migrations/polyfills/index_definition_polyfill.rb
|
63
|
-
- lib/safe-pg-migrations/polyfills/satisfied_helper.rb
|
64
67
|
- lib/safe-pg-migrations/polyfills/verbose_query_logs_polyfill.rb
|
65
68
|
- lib/safe-pg-migrations/railtie.rb
|
66
69
|
- lib/safe-pg-migrations/version.rb
|
@@ -1,13 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module SafePgMigrations
|
4
|
-
module Polyfills
|
5
|
-
module SatisfiedHelper
|
6
|
-
private
|
7
|
-
|
8
|
-
def satisfied?(version)
|
9
|
-
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
10
|
-
end
|
11
|
-
end
|
12
|
-
end
|
13
|
-
end
|