safe-pg-migrations 2.0.0 → 2.2.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +143 -12
- data/lib/safe-pg-migrations/base.rb +38 -36
- data/lib/safe-pg-migrations/configuration.rb +49 -5
- data/lib/safe-pg-migrations/helpers/batch_over.rb +56 -0
- data/lib/safe-pg-migrations/helpers/blocking_activity_formatter.rb +23 -22
- data/lib/safe-pg-migrations/helpers/index_helper.rb +24 -0
- data/lib/safe-pg-migrations/helpers/logger.rb +47 -0
- data/lib/safe-pg-migrations/helpers/satisfied_helper.rb +31 -0
- data/lib/safe-pg-migrations/helpers/session_setting_management.rb +35 -0
- data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb +25 -9
- data/lib/safe-pg-migrations/plugins/idempotent_statements.rb +73 -32
- data/lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb +65 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer/change_column_null.rb +46 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +49 -43
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +3 -3
- data/lib/safe-pg-migrations/plugins/strong_migrations_integration.rb +75 -0
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +12 -2
- data/lib/safe-pg-migrations/polyfills/index_definition_polyfill.rb +2 -2
- data/lib/safe-pg-migrations/polyfills/verbose_query_logs_polyfill.rb +2 -4
- data/lib/safe-pg-migrations/railtie.rb +2 -0
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +10 -3
- data/lib/safe-pg-migrations/polyfills/satisfied_helper.rb +0 -13
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 624821b7134dee05186350edada6f27faaf8c820976837d32403dc5fffa4880a
|
4
|
+
data.tar.gz: bf41324b8d563de7bc893a302d45d8f175e890bb888a2f52e0b14be7676f942c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 76601dcf0b97013edc36de6ac351beb8e4f6c532a0e223cba95d1a29e4eb7f18b8a4e7b3e3f28e4e7bde02144a4de8c512c19e2725396578dcb322e040963168
|
7
|
+
data.tar.gz: 767876e75af3292f9c4d3fc82810ab37bb3d38dad7b8b87ee6ba7068ec5e4aa9967ac0e53170be77bda07dd3424f2d3b3b3a03d9b8dc6ebcda94cee21e61fcfc
|
data/README.md
CHANGED
@@ -109,7 +109,51 @@ Beware though, when adding a volatile default value:
|
|
109
109
|
```ruby
|
110
110
|
add_column :users, :created_at, default: 'clock_timestamp()'
|
111
111
|
```
|
112
|
-
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case,
|
112
|
+
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, **Safe PG Migrations** can automatically backfill data when the option `default_value_backfill:` is set to `:update_in_batches`.
|
113
|
+
|
114
|
+
</details>
|
115
|
+
|
116
|
+
<details>
|
117
|
+
<summary>Safe add_column - adding a volatile default value</summary>
|
118
|
+
|
119
|
+
**Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a volatile default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
|
120
|
+
|
121
|
+
```ruby
|
122
|
+
safety_assured do
|
123
|
+
add_column :users, :created_at, default: 'clock_timestamp()', default_value_backfill: :update_in_batches
|
124
|
+
end
|
125
|
+
```
|
126
|
+
|
127
|
+
More specifically, it will:
|
128
|
+
|
129
|
+
1. create the column without default value and without null constraint. This ensure the `ACCESS EXCLUSIVE` lock is acquired for the least amount of time;
|
130
|
+
2. add the default value, without data backfill. An `ACCESS EXCLUSIVE` lock is acquired and released immediately;
|
131
|
+
3. backfill data, in batch of `SafePgMigrations.config.backfill_batch_size` and with a pause of `SafePgMigrations.config.backfill_pause` between each batch;
|
132
|
+
4. change the column to `null: false`, if defined in the parameters, following the algorithm we have defined below.
|
133
|
+
|
134
|
+
---
|
135
|
+
**NOTE**
|
136
|
+
|
137
|
+
Data backfill take time. If your table is big, your migrations will (safely) hangs for a while. You might want to backfill data manually instead, to do so you will need two migrations
|
138
|
+
|
139
|
+
1. First migration :
|
140
|
+
|
141
|
+
a. adds the column without default and without null constraint;
|
142
|
+
|
143
|
+
b. add the default value.
|
144
|
+
|
145
|
+
2. manual data backfill (rake task, manual operation, ...)
|
146
|
+
3. Second migration which change the column to null false (with **Safe PG Migrations**, `change_column_null` is safe and can be used; see section below)
|
147
|
+
---
|
148
|
+
|
149
|
+
`default_value_backfill:` also accept the value `:auto` which is set by default. In this case, **Safe PG Migrations** will not backfill data and will let PostgreSQL handle it itself.
|
150
|
+
|
151
|
+
### Preventing :update_in_batches when the table is too big
|
152
|
+
|
153
|
+
`add_column` with `default_value_backfill: :update_in_batches` can be dangerous on big tables. To avoid unwanted long migrations, **Safe PG Migrations** does not automatically mark this usage as safe when used with `strong-migrations`, usage of `safety_assured` is required.
|
154
|
+
|
155
|
+
It is also possible to set a threshold for the table size, above which the migration will fail. This can be done by setting the `default_value_backfill_threshold:` option in the configuration.
|
156
|
+
|
113
157
|
|
114
158
|
</details>
|
115
159
|
|
@@ -130,13 +174,70 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
130
174
|
|
131
175
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
132
176
|
|
133
|
-
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus
|
177
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus **Safe PG Migrations** ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
|
134
178
|
|
135
179
|
1. adding the constraint *without validation*, will not validate existing rows;
|
136
180
|
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
137
181
|
|
138
182
|
</details>
|
139
183
|
|
184
|
+
|
185
|
+
<details><summary id="safe_add_check_constraint">Safe <code>add_check_constraint</code> (ActiveRecord > 6.1)</summary>
|
186
|
+
|
187
|
+
Adding a check constraint requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
188
|
+
|
189
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
190
|
+
Thus **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least amount of time by
|
191
|
+
splitting the constraint addition in two steps:
|
192
|
+
|
193
|
+
1. adding the constraint *without validation*, will not validate existing rows;
|
194
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
195
|
+
|
196
|
+
</details>
|
197
|
+
|
198
|
+
<details><summary id="safe_change_column_null">Safe <code>change_column_null</code> (ActiveRecord and PG version dependant)</summary>
|
199
|
+
|
200
|
+
Changing the nullability of a column requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
201
|
+
|
202
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
203
|
+
|
204
|
+
**Safe PG Migrations** acts differently depending on the version you are on.
|
205
|
+
|
206
|
+
### Recent versions of PG and Active Record (> 12 and > 6.1)
|
207
|
+
|
208
|
+
Starting on PostgreSQL versions 12, adding the column NOT NULL constraint is safe if a check constraint validates the
|
209
|
+
nullability of the same column. **Safe PG Migrations** also relies on add_check_constraint, which was introduced in
|
210
|
+
ActiveRecord 6.1.
|
211
|
+
|
212
|
+
If these requirements are met, **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least
|
213
|
+
amount of time by splitting the constraint addition in several steps:
|
214
|
+
|
215
|
+
1. adding a `IS NOT NULL` constraint *without validation*, will not validate existing rows but block read or write;
|
216
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table;
|
217
|
+
3. changing the not null status of the column, thanks to the NOT NULL constraint without having to scan the table sequentially;
|
218
|
+
4. dropping the `IS NOT NULL` constraint.
|
219
|
+
|
220
|
+
### Older versions of PG or ActiveRecord
|
221
|
+
|
222
|
+
If the version of PostgreSQL is below 12, or if the version of ActiveRecord is below 6.1, **Safe PG Migrations** will only
|
223
|
+
wrap ActiveRecord method into a statement timeout and lock timeout.
|
224
|
+
|
225
|
+
### Call with a default parameter
|
226
|
+
|
227
|
+
Calling change_column_null with a default parameter [is dangerous](https://github.com/rails/rails/blob/716baea69f989b64f5bfeaff880c2512377bebab/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb#L446)
|
228
|
+
and is likely not to finish in the statement timeout defined by **Safe PG Migrations**. For this reason, when the default
|
229
|
+
parameter is given, **Safe PG Migrations** will simply forward it to activerecord methods without trying to improve it
|
230
|
+
|
231
|
+
### Dropping a NULL constraint
|
232
|
+
|
233
|
+
Dropping a null constraint still requires an `ACCESS EXCLUSIVE` lock, but does not require extra operation to reduce the
|
234
|
+
amount of time during which the lock is held. **Safe PG Migrations** only wrap methods of activerecord in lock and statement
|
235
|
+
timeouts
|
236
|
+
|
237
|
+
</details>
|
238
|
+
|
239
|
+
|
240
|
+
|
140
241
|
<details><summary>Retry after lock timeout</summary>
|
141
242
|
|
142
243
|
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
@@ -145,6 +246,35 @@ When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5
|
|
145
246
|
<details><summary>Blocking activity logging</summary>
|
146
247
|
|
147
248
|
If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
|
249
|
+
|
250
|
+
---
|
251
|
+
**NOTE**
|
252
|
+
|
253
|
+
Data logged by the Blocking activity logger can be sensitive (it will contain raw SQL queries, which can be hashes of password, user information, ...)
|
254
|
+
|
255
|
+
If you cannot afford to log this type of data, you can either
|
256
|
+
* Set `SafePgMigrations.config.blocking_activity_logger_verbose = false`. In this case, the logger will only log the pid of the blocking statement, which should be enough to investigate;
|
257
|
+
* Provide a different logger through `SafePgMigrations.config.sensitive_logger = YourLogger.new`. Instead of using the default IO stream, SafePgMigrations will send sensitive data to the given logger which can be managed as you wish.
|
258
|
+
|
259
|
+
---
|
260
|
+
|
261
|
+
</details>
|
262
|
+
|
263
|
+
<details><summary>Dropping a table</summary>
|
264
|
+
|
265
|
+
Dropping a table can be difficult to achieve in a small amount of time if it holds several foreign keys to busy tables.
|
266
|
+
To remove the table, PostgreSQL will have to acquire an access exclusive lock on all the tables referenced by the foreign keys.
|
267
|
+
|
268
|
+
To solve this issue, **Safe Pg Migrations** will drop the foreign keys before dropping the table.
|
269
|
+
|
270
|
+
---
|
271
|
+
**NOTE**
|
272
|
+
|
273
|
+
Dropping a table is a dangerous operation by nature. **Safe Pg Migrations** will not prevent the deletion of a table which
|
274
|
+
would still be in use.
|
275
|
+
|
276
|
+
---
|
277
|
+
|
148
278
|
</details>
|
149
279
|
|
150
280
|
<details><summary>Verbose SQL logging</summary>
|
@@ -195,31 +325,32 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
|
|
195
325
|
**Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
|
196
326
|
|
197
327
|
```ruby
|
198
|
-
SafePgMigrations.config.safe_timeout = 5.seconds #
|
328
|
+
SafePgMigrations.config.safe_timeout = 5.seconds # Statement timeout used for all DDL operations except from CREATE / DROP INDEX
|
329
|
+
|
330
|
+
SafePgMigrations.config.lock_timeout = nil # Lock timeout used for all DDL operations except from CREATE / DROP INDEX. If not set, safe_timeout will be used with a deduction of 1% to ensure that the lock timeout is raised in priority
|
199
331
|
|
200
332
|
SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
|
201
333
|
|
334
|
+
SafePgMigrations.config.sensitive_logger = nil # When given, sensitive data will be sent to this logger instead of the standard output. Must implement method `info`.
|
335
|
+
|
202
336
|
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
203
337
|
|
204
|
-
SafePgMigrations.config.
|
338
|
+
SafePgMigrations.config.backfill_batch_size = 100_000 # Size of the batches used for backfilling when adding a column with a default value
|
205
339
|
|
206
|
-
SafePgMigrations.config.
|
340
|
+
SafePgMigrations.config.backfill_pause = 0.5.second # Delay between each batch during a backfill. This ensure replication can happen safely.
|
207
341
|
|
208
|
-
SafePgMigrations.config.
|
209
|
-
```
|
342
|
+
SafePgMigrations.config.default_value_backfill_threshold = nil # When set, batch backfill will only be available if the table is under the given threshold. If the number of rows is higher (according to stats), the migration will fail.
|
210
343
|
|
211
|
-
|
344
|
+
SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
|
212
345
|
|
213
|
-
|
214
|
-
bundle
|
215
|
-
psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
|
216
|
-
rake test
|
346
|
+
SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
|
217
347
|
```
|
218
348
|
|
219
349
|
## Authors
|
220
350
|
|
221
351
|
- [Matthieu Prat](https://github.com/matthieuprat)
|
222
352
|
- [Romain Choquet](https://github.com/rchoquet)
|
353
|
+
- [Thomas Hareau](https://github.com/ThHareau)
|
223
354
|
- [Paul-Etienne Coisne](https://github.com/coisnepe)
|
224
355
|
|
225
356
|
## License
|
@@ -1,13 +1,20 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require 'safe-pg-migrations/configuration'
|
4
|
+
require 'safe-pg-migrations/helpers/logger'
|
5
|
+
require 'safe-pg-migrations/helpers/satisfied_helper'
|
6
|
+
require 'safe-pg-migrations/helpers/index_helper'
|
7
|
+
require 'safe-pg-migrations/helpers/batch_over'
|
8
|
+
require 'safe-pg-migrations/helpers/session_setting_management'
|
4
9
|
require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
5
10
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
11
|
+
require 'safe-pg-migrations/plugins/statement_insurer/add_column'
|
12
|
+
require 'safe-pg-migrations/plugins/statement_insurer/change_column_null'
|
6
13
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
7
14
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
8
15
|
require 'safe-pg-migrations/plugins/idempotent_statements'
|
9
16
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
10
|
-
require 'safe-pg-migrations/
|
17
|
+
require 'safe-pg-migrations/plugins/strong_migrations_integration'
|
11
18
|
require 'safe-pg-migrations/polyfills/index_definition_polyfill'
|
12
19
|
require 'safe-pg-migrations/polyfills/verbose_query_logs_polyfill'
|
13
20
|
|
@@ -23,19 +30,34 @@ module SafePgMigrations
|
|
23
30
|
].freeze
|
24
31
|
|
25
32
|
class << self
|
26
|
-
attr_reader :current_migration
|
33
|
+
attr_reader :current_migration, :pg_version_num
|
27
34
|
|
28
35
|
def setup_and_teardown(migration, connection, &block)
|
36
|
+
@pg_version_num = get_pg_version_num(connection)
|
29
37
|
@alternate_connection = nil
|
30
|
-
@current_migration = migration
|
31
|
-
stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
|
32
|
-
PLUGINS.each { |plugin| connection.extend(plugin) }
|
33
38
|
|
34
|
-
|
39
|
+
with_current_migration(migration) do
|
40
|
+
stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
|
41
|
+
|
42
|
+
VerboseSqlLogger.new.setup if verbose?
|
43
|
+
PLUGINS.each { |plugin| connection.extend(plugin) }
|
44
|
+
|
45
|
+
connection.with_setting :lock_timeout, SafePgMigrations.config.pg_lock_timeout do
|
46
|
+
connection.with_setting :statement_timeout, SafePgMigrations.config.pg_statement_timeout, &block
|
47
|
+
end
|
48
|
+
ensure
|
49
|
+
stdout_sql_logger&.teardown
|
50
|
+
end
|
35
51
|
ensure
|
36
52
|
close_alternate_connection
|
53
|
+
end
|
54
|
+
|
55
|
+
def with_current_migration(migration, &block)
|
56
|
+
@current_migration = migration
|
57
|
+
|
58
|
+
yield block
|
59
|
+
ensure
|
37
60
|
@current_migration = nil
|
38
|
-
stdout_sql_logger&.teardown
|
39
61
|
end
|
40
62
|
|
41
63
|
def alternate_connection
|
@@ -49,16 +71,6 @@ module SafePgMigrations
|
|
49
71
|
@alternate_connection = nil
|
50
72
|
end
|
51
73
|
|
52
|
-
ruby2_keywords def say(*args)
|
53
|
-
return unless current_migration
|
54
|
-
|
55
|
-
current_migration.say(*args)
|
56
|
-
end
|
57
|
-
|
58
|
-
ruby2_keywords def say_method_call(method, *args)
|
59
|
-
say "#{method}(#{args.map(&:inspect) * ', '})", true
|
60
|
-
end
|
61
|
-
|
62
74
|
def verbose?
|
63
75
|
unless current_migration.class._safe_pg_migrations_verbose.nil?
|
64
76
|
return current_migration.class._safe_pg_migrations_verbose
|
@@ -72,9 +84,15 @@ module SafePgMigrations
|
|
72
84
|
def config
|
73
85
|
@config ||= Configuration.new
|
74
86
|
end
|
87
|
+
|
88
|
+
def get_pg_version_num(connection)
|
89
|
+
connection.query_value('SHOW server_version_num').to_i
|
90
|
+
end
|
75
91
|
end
|
76
92
|
|
77
93
|
module Migration
|
94
|
+
include StrongMigrationsIntegration
|
95
|
+
|
78
96
|
module ClassMethods
|
79
97
|
attr_accessor :_safe_pg_migrations_verbose
|
80
98
|
|
@@ -90,27 +108,11 @@ module SafePgMigrations
|
|
90
108
|
end
|
91
109
|
|
92
110
|
def disable_ddl_transaction
|
93
|
-
|
94
|
-
|
95
|
-
end
|
111
|
+
SafePgMigrations.with_current_migration(self) do
|
112
|
+
UselessStatementsLogger.warn_useless '`disable_ddl_transaction`' if super
|
96
113
|
|
97
|
-
|
98
|
-
execute
|
99
|
-
add_column
|
100
|
-
add_index
|
101
|
-
add_reference
|
102
|
-
add_belongs_to
|
103
|
-
change_column_null
|
104
|
-
add_foreign_key
|
105
|
-
].freeze
|
106
|
-
|
107
|
-
SAFE_METHODS.each do |method|
|
108
|
-
define_method method do |*args|
|
109
|
-
return super(*args) unless respond_to?(:safety_assured)
|
110
|
-
|
111
|
-
safety_assured { super(*args) }
|
114
|
+
true
|
112
115
|
end
|
113
|
-
ruby2_keywords method
|
114
116
|
end
|
115
117
|
end
|
116
118
|
end
|
@@ -4,22 +4,66 @@ require 'active_support/core_ext/numeric/time'
|
|
4
4
|
|
5
5
|
module SafePgMigrations
|
6
6
|
class Configuration
|
7
|
-
attr_accessor
|
8
|
-
|
7
|
+
attr_accessor(*%i[
|
8
|
+
blocking_activity_logger_margin
|
9
|
+
blocking_activity_logger_verbose
|
10
|
+
default_value_backfill_threshold
|
11
|
+
backfill_batch_size
|
12
|
+
backfill_pause
|
13
|
+
retry_delay
|
14
|
+
max_tries
|
15
|
+
sensitive_logger
|
16
|
+
])
|
17
|
+
attr_reader :lock_timeout, :safe_timeout
|
9
18
|
|
10
19
|
def initialize
|
20
|
+
self.default_value_backfill_threshold = nil
|
11
21
|
self.safe_timeout = 5.seconds
|
22
|
+
self.lock_timeout = nil
|
12
23
|
self.blocking_activity_logger_margin = 1.second
|
13
24
|
self.blocking_activity_logger_verbose = true
|
14
|
-
self.
|
25
|
+
self.backfill_batch_size = 100_000
|
26
|
+
self.backfill_pause = 0.5.second
|
15
27
|
self.retry_delay = 1.minute
|
16
28
|
self.max_tries = 5
|
29
|
+
self.sensitive_logger = nil
|
17
30
|
end
|
18
31
|
|
19
|
-
def
|
20
|
-
|
32
|
+
def lock_timeout=(value)
|
33
|
+
raise 'Setting lock timeout to 0 disables the lock timeout and is dangerous' if value == 0.seconds
|
34
|
+
|
35
|
+
unless value.nil? || value < safe_timeout
|
36
|
+
raise ArgumentError, "Lock timeout (#{value}) cannot be greater than safe timeout (#{safe_timeout})"
|
37
|
+
end
|
38
|
+
|
39
|
+
@lock_timeout = value
|
40
|
+
end
|
41
|
+
|
42
|
+
def safe_timeout=(value)
|
43
|
+
raise 'Setting safe timeout to 0 disables the safe timeout and is dangerous' unless value
|
44
|
+
|
45
|
+
unless lock_timeout.nil? || value > lock_timeout
|
46
|
+
raise ArgumentError, "Safe timeout (#{value}) cannot be less than lock timeout (#{lock_timeout})"
|
47
|
+
end
|
48
|
+
|
49
|
+
@safe_timeout = value
|
21
50
|
end
|
22
51
|
|
52
|
+
def pg_statement_timeout
|
53
|
+
pg_duration safe_timeout
|
54
|
+
end
|
55
|
+
|
56
|
+
def pg_lock_timeout
|
57
|
+
return pg_duration lock_timeout if lock_timeout
|
58
|
+
|
59
|
+
# if statement timeout and lock timeout have the same value, statement timeout will raise in priority. We actually
|
60
|
+
# need the opposite for BlockingActivityLogger to detect lock timeouts correctly.
|
61
|
+
# By reducing the lock timeout by a very small margin, we ensure that the lock timeout is raised in priority
|
62
|
+
pg_duration safe_timeout * 0.99
|
63
|
+
end
|
64
|
+
|
65
|
+
private
|
66
|
+
|
23
67
|
def pg_duration(duration)
|
24
68
|
value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
|
25
69
|
"#{value}#{unit}"
|
@@ -0,0 +1,56 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
class BatchOver
|
6
|
+
def initialize(model, of: SafePgMigrations.config.backfill_batch_size)
|
7
|
+
@model = model
|
8
|
+
@of = of
|
9
|
+
|
10
|
+
@current_range = nil
|
11
|
+
end
|
12
|
+
|
13
|
+
def each_batch
|
14
|
+
yield scope.where(primary_key => @current_range) while next_batch
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def next_batch
|
20
|
+
return if endless?
|
21
|
+
|
22
|
+
first = next_scope.take
|
23
|
+
|
24
|
+
return unless first
|
25
|
+
|
26
|
+
last = next_scope.offset(@of).take
|
27
|
+
|
28
|
+
first_key = first[primary_key]
|
29
|
+
last_key = last.nil? ? nil : last[primary_key]
|
30
|
+
|
31
|
+
@current_range = first_key...last_key
|
32
|
+
end
|
33
|
+
|
34
|
+
def next_scope
|
35
|
+
return scope if @current_range.nil?
|
36
|
+
return scope.none if endless?
|
37
|
+
|
38
|
+
scope.where(primary_key => @current_range.end..)
|
39
|
+
end
|
40
|
+
|
41
|
+
def scope
|
42
|
+
@model.order(primary_key => :asc)
|
43
|
+
end
|
44
|
+
|
45
|
+
def endless?
|
46
|
+
return false if @current_range.nil?
|
47
|
+
|
48
|
+
@current_range.end.nil?
|
49
|
+
end
|
50
|
+
|
51
|
+
def primary_key
|
52
|
+
@model.primary_key
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
@@ -5,20 +5,20 @@ module SafePgMigrations
|
|
5
5
|
module BlockingActivityFormatter
|
6
6
|
def log_queries(queries)
|
7
7
|
if queries.empty?
|
8
|
-
|
8
|
+
Logger.say 'Could not find any blocking query.', sub_item: true
|
9
9
|
else
|
10
|
-
|
11
|
-
|
12
|
-
|
13
|
-
|
14
|
-
|
10
|
+
Logger.say <<~MESSAGE.squish, sub_item: true
|
11
|
+
Statement was being blocked by the following #{'query'.pluralize(queries.size)}:
|
12
|
+
MESSAGE
|
13
|
+
|
14
|
+
Logger.say '', sub_item: true
|
15
15
|
output_blocking_queries(queries)
|
16
|
-
|
17
|
-
|
18
|
-
|
19
|
-
|
20
|
-
|
21
|
-
|
16
|
+
Logger.say <<~MESSAGE.squish, sub_item: true
|
17
|
+
Beware, some of those queries might run in a transaction. In this case the locking query might be located
|
18
|
+
elsewhere in the transaction
|
19
|
+
MESSAGE
|
20
|
+
|
21
|
+
Logger.say '', sub_item: true
|
22
22
|
end
|
23
23
|
end
|
24
24
|
|
@@ -27,8 +27,10 @@ module SafePgMigrations
|
|
27
27
|
def output_blocking_queries(queries)
|
28
28
|
if SafePgMigrations.config.blocking_activity_logger_verbose
|
29
29
|
queries.each do |pid, query, start_time|
|
30
|
-
|
31
|
-
|
30
|
+
Logger.say(
|
31
|
+
"Query with pid #{pid || 'null'} started #{format_start_time start_time}: #{query}",
|
32
|
+
sub_item: true, sensitive: true
|
33
|
+
)
|
32
34
|
end
|
33
35
|
else
|
34
36
|
output_confidentially_blocking_queries(queries)
|
@@ -37,14 +39,13 @@ module SafePgMigrations
|
|
37
39
|
|
38
40
|
def output_confidentially_blocking_queries(queries)
|
39
41
|
queries.each do |start_time, locktype, mode, pid, transactionid|
|
40
|
-
|
41
|
-
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
)
|
42
|
+
Logger.say <<~MESSAGE.squish, sub_item: true, sensitive: true
|
43
|
+
Query with pid #{pid || 'null'}
|
44
|
+
started #{format_start_time(start_time)}:
|
45
|
+
lock type: #{locktype || 'null'},
|
46
|
+
lock mode: #{mode || 'null'},
|
47
|
+
lock transactionid: #{transactionid || 'null'}",
|
48
|
+
MESSAGE
|
48
49
|
end
|
49
50
|
end
|
50
51
|
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module IndexHelper
|
6
|
+
def index_definition(table_name, column_name, **options)
|
7
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
8
|
+
index_definition
|
9
|
+
end
|
10
|
+
|
11
|
+
private
|
12
|
+
|
13
|
+
def index_valid?(index_name)
|
14
|
+
query_value <<~SQL.squish
|
15
|
+
SELECT indisvalid
|
16
|
+
FROM pg_index i
|
17
|
+
JOIN pg_class c
|
18
|
+
ON i.indexrelid = c.oid
|
19
|
+
WHERE c.relname = '#{index_name}';
|
20
|
+
SQL
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,47 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module Logger
|
6
|
+
class << self
|
7
|
+
def say(message, sub_item: false, sensitive: false, warn_sensitive_logs: true)
|
8
|
+
return unless SafePgMigrations.current_migration
|
9
|
+
|
10
|
+
if sensitive
|
11
|
+
log_sensitive message, sub_item: sub_item
|
12
|
+
if warn_sensitive_logs && sensitive_logger?
|
13
|
+
log 'Sensitive data sent to sensitive logger', sub_item: sub_item
|
14
|
+
end
|
15
|
+
else
|
16
|
+
log message, sub_item: sub_item
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def say_method_call(method, *args, sensitive: false, warn_sensitive_logs: true, **options)
|
21
|
+
args += [options] unless options.empty?
|
22
|
+
|
23
|
+
say "#{method}(#{args.map(&:inspect) * ', '})",
|
24
|
+
sub_item: true, sensitive: sensitive, warn_sensitive_logs: warn_sensitive_logs
|
25
|
+
end
|
26
|
+
|
27
|
+
private
|
28
|
+
|
29
|
+
def log(message, sub_item:)
|
30
|
+
SafePgMigrations.current_migration.say message, sub_item
|
31
|
+
end
|
32
|
+
|
33
|
+
def log_sensitive(message, sub_item:)
|
34
|
+
if sensitive_logger?
|
35
|
+
SafePgMigrations.config.sensitive_logger.info message
|
36
|
+
else
|
37
|
+
log message, sub_item: sub_item
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
def sensitive_logger?
|
42
|
+
SafePgMigrations.config.sensitive_logger
|
43
|
+
end
|
44
|
+
end
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module SatisfiedHelper
|
6
|
+
class << self
|
7
|
+
def satisfies_change_column_null_requirements?
|
8
|
+
satisfies_add_check_constraints? && SafePgMigrations.pg_version_num >= 120_000
|
9
|
+
end
|
10
|
+
|
11
|
+
def satisfies_add_check_constraints!
|
12
|
+
return if satisfies_add_check_constraints?
|
13
|
+
|
14
|
+
raise NotImplementedError, 'add_check_constraint is not supported in your ActiveRecord version'
|
15
|
+
end
|
16
|
+
|
17
|
+
def satisfies_add_check_constraints?
|
18
|
+
satisfied? '>=6.1.0'
|
19
|
+
end
|
20
|
+
|
21
|
+
def satisfies_add_column_update_rows_backfill?
|
22
|
+
satisfies_change_column_null_requirements?
|
23
|
+
end
|
24
|
+
|
25
|
+
def satisfied?(version)
|
26
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|