safe-pg-migrations 2.0.0 → 2.2.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +143 -12
- data/lib/safe-pg-migrations/base.rb +38 -36
- data/lib/safe-pg-migrations/configuration.rb +49 -5
- data/lib/safe-pg-migrations/helpers/batch_over.rb +56 -0
- data/lib/safe-pg-migrations/helpers/blocking_activity_formatter.rb +23 -22
- data/lib/safe-pg-migrations/helpers/index_helper.rb +24 -0
- data/lib/safe-pg-migrations/helpers/logger.rb +47 -0
- data/lib/safe-pg-migrations/helpers/satisfied_helper.rb +31 -0
- data/lib/safe-pg-migrations/helpers/session_setting_management.rb +35 -0
- data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb +25 -9
- data/lib/safe-pg-migrations/plugins/idempotent_statements.rb +73 -32
- data/lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb +65 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer/change_column_null.rb +46 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +49 -43
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +3 -3
- data/lib/safe-pg-migrations/plugins/strong_migrations_integration.rb +75 -0
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +12 -2
- data/lib/safe-pg-migrations/polyfills/index_definition_polyfill.rb +2 -2
- data/lib/safe-pg-migrations/polyfills/verbose_query_logs_polyfill.rb +2 -4
- data/lib/safe-pg-migrations/railtie.rb +2 -0
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +10 -3
- data/lib/safe-pg-migrations/polyfills/satisfied_helper.rb +0 -13
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 624821b7134dee05186350edada6f27faaf8c820976837d32403dc5fffa4880a
|
|
4
|
+
data.tar.gz: bf41324b8d563de7bc893a302d45d8f175e890bb888a2f52e0b14be7676f942c
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 76601dcf0b97013edc36de6ac351beb8e4f6c532a0e223cba95d1a29e4eb7f18b8a4e7b3e3f28e4e7bde02144a4de8c512c19e2725396578dcb322e040963168
|
|
7
|
+
data.tar.gz: 767876e75af3292f9c4d3fc82810ab37bb3d38dad7b8b87ee6ba7068ec5e4aa9967ac0e53170be77bda07dd3424f2d3b3b3a03d9b8dc6ebcda94cee21e61fcfc
|
data/README.md
CHANGED
|
@@ -109,7 +109,51 @@ Beware though, when adding a volatile default value:
|
|
|
109
109
|
```ruby
|
|
110
110
|
add_column :users, :created_at, default: 'clock_timestamp()'
|
|
111
111
|
```
|
|
112
|
-
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case,
|
|
112
|
+
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, **Safe PG Migrations** can automatically backfill data when the option `default_value_backfill:` is set to `:update_in_batches`.
|
|
113
|
+
|
|
114
|
+
</details>
|
|
115
|
+
|
|
116
|
+
<details>
|
|
117
|
+
<summary>Safe add_column - adding a volatile default value</summary>
|
|
118
|
+
|
|
119
|
+
**Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a volatile default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
|
|
120
|
+
|
|
121
|
+
```ruby
|
|
122
|
+
safety_assured do
|
|
123
|
+
add_column :users, :created_at, default: 'clock_timestamp()', default_value_backfill: :update_in_batches
|
|
124
|
+
end
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
More specifically, it will:
|
|
128
|
+
|
|
129
|
+
1. create the column without default value and without null constraint. This ensure the `ACCESS EXCLUSIVE` lock is acquired for the least amount of time;
|
|
130
|
+
2. add the default value, without data backfill. An `ACCESS EXCLUSIVE` lock is acquired and released immediately;
|
|
131
|
+
3. backfill data, in batch of `SafePgMigrations.config.backfill_batch_size` and with a pause of `SafePgMigrations.config.backfill_pause` between each batch;
|
|
132
|
+
4. change the column to `null: false`, if defined in the parameters, following the algorithm we have defined below.
|
|
133
|
+
|
|
134
|
+
---
|
|
135
|
+
**NOTE**
|
|
136
|
+
|
|
137
|
+
Data backfill take time. If your table is big, your migrations will (safely) hangs for a while. You might want to backfill data manually instead, to do so you will need two migrations
|
|
138
|
+
|
|
139
|
+
1. First migration :
|
|
140
|
+
|
|
141
|
+
a. adds the column without default and without null constraint;
|
|
142
|
+
|
|
143
|
+
b. add the default value.
|
|
144
|
+
|
|
145
|
+
2. manual data backfill (rake task, manual operation, ...)
|
|
146
|
+
3. Second migration which change the column to null false (with **Safe PG Migrations**, `change_column_null` is safe and can be used; see section below)
|
|
147
|
+
---
|
|
148
|
+
|
|
149
|
+
`default_value_backfill:` also accept the value `:auto` which is set by default. In this case, **Safe PG Migrations** will not backfill data and will let PostgreSQL handle it itself.
|
|
150
|
+
|
|
151
|
+
### Preventing :update_in_batches when the table is too big
|
|
152
|
+
|
|
153
|
+
`add_column` with `default_value_backfill: :update_in_batches` can be dangerous on big tables. To avoid unwanted long migrations, **Safe PG Migrations** does not automatically mark this usage as safe when used with `strong-migrations`, usage of `safety_assured` is required.
|
|
154
|
+
|
|
155
|
+
It is also possible to set a threshold for the table size, above which the migration will fail. This can be done by setting the `default_value_backfill_threshold:` option in the configuration.
|
|
156
|
+
|
|
113
157
|
|
|
114
158
|
</details>
|
|
115
159
|
|
|
@@ -130,13 +174,70 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
|
130
174
|
|
|
131
175
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
|
132
176
|
|
|
133
|
-
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus
|
|
177
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus **Safe PG Migrations** ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
|
|
134
178
|
|
|
135
179
|
1. adding the constraint *without validation*, will not validate existing rows;
|
|
136
180
|
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
|
137
181
|
|
|
138
182
|
</details>
|
|
139
183
|
|
|
184
|
+
|
|
185
|
+
<details><summary id="safe_add_check_constraint">Safe <code>add_check_constraint</code> (ActiveRecord > 6.1)</summary>
|
|
186
|
+
|
|
187
|
+
Adding a check constraint requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
|
188
|
+
|
|
189
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
|
190
|
+
Thus **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least amount of time by
|
|
191
|
+
splitting the constraint addition in two steps:
|
|
192
|
+
|
|
193
|
+
1. adding the constraint *without validation*, will not validate existing rows;
|
|
194
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
|
195
|
+
|
|
196
|
+
</details>
|
|
197
|
+
|
|
198
|
+
<details><summary id="safe_change_column_null">Safe <code>change_column_null</code> (ActiveRecord and PG version dependant)</summary>
|
|
199
|
+
|
|
200
|
+
Changing the nullability of a column requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
|
201
|
+
|
|
202
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
|
203
|
+
|
|
204
|
+
**Safe PG Migrations** acts differently depending on the version you are on.
|
|
205
|
+
|
|
206
|
+
### Recent versions of PG and Active Record (> 12 and > 6.1)
|
|
207
|
+
|
|
208
|
+
Starting on PostgreSQL versions 12, adding the column NOT NULL constraint is safe if a check constraint validates the
|
|
209
|
+
nullability of the same column. **Safe PG Migrations** also relies on add_check_constraint, which was introduced in
|
|
210
|
+
ActiveRecord 6.1.
|
|
211
|
+
|
|
212
|
+
If these requirements are met, **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least
|
|
213
|
+
amount of time by splitting the constraint addition in several steps:
|
|
214
|
+
|
|
215
|
+
1. adding a `IS NOT NULL` constraint *without validation*, will not validate existing rows but block read or write;
|
|
216
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table;
|
|
217
|
+
3. changing the not null status of the column, thanks to the NOT NULL constraint without having to scan the table sequentially;
|
|
218
|
+
4. dropping the `IS NOT NULL` constraint.
|
|
219
|
+
|
|
220
|
+
### Older versions of PG or ActiveRecord
|
|
221
|
+
|
|
222
|
+
If the version of PostgreSQL is below 12, or if the version of ActiveRecord is below 6.1, **Safe PG Migrations** will only
|
|
223
|
+
wrap ActiveRecord method into a statement timeout and lock timeout.
|
|
224
|
+
|
|
225
|
+
### Call with a default parameter
|
|
226
|
+
|
|
227
|
+
Calling change_column_null with a default parameter [is dangerous](https://github.com/rails/rails/blob/716baea69f989b64f5bfeaff880c2512377bebab/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb#L446)
|
|
228
|
+
and is likely not to finish in the statement timeout defined by **Safe PG Migrations**. For this reason, when the default
|
|
229
|
+
parameter is given, **Safe PG Migrations** will simply forward it to activerecord methods without trying to improve it
|
|
230
|
+
|
|
231
|
+
### Dropping a NULL constraint
|
|
232
|
+
|
|
233
|
+
Dropping a null constraint still requires an `ACCESS EXCLUSIVE` lock, but does not require extra operation to reduce the
|
|
234
|
+
amount of time during which the lock is held. **Safe PG Migrations** only wrap methods of activerecord in lock and statement
|
|
235
|
+
timeouts
|
|
236
|
+
|
|
237
|
+
</details>
|
|
238
|
+
|
|
239
|
+
|
|
240
|
+
|
|
140
241
|
<details><summary>Retry after lock timeout</summary>
|
|
141
242
|
|
|
142
243
|
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
|
@@ -145,6 +246,35 @@ When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5
|
|
|
145
246
|
<details><summary>Blocking activity logging</summary>
|
|
146
247
|
|
|
147
248
|
If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
|
|
249
|
+
|
|
250
|
+
---
|
|
251
|
+
**NOTE**
|
|
252
|
+
|
|
253
|
+
Data logged by the Blocking activity logger can be sensitive (it will contain raw SQL queries, which can be hashes of password, user information, ...)
|
|
254
|
+
|
|
255
|
+
If you cannot afford to log this type of data, you can either
|
|
256
|
+
* Set `SafePgMigrations.config.blocking_activity_logger_verbose = false`. In this case, the logger will only log the pid of the blocking statement, which should be enough to investigate;
|
|
257
|
+
* Provide a different logger through `SafePgMigrations.config.sensitive_logger = YourLogger.new`. Instead of using the default IO stream, SafePgMigrations will send sensitive data to the given logger which can be managed as you wish.
|
|
258
|
+
|
|
259
|
+
---
|
|
260
|
+
|
|
261
|
+
</details>
|
|
262
|
+
|
|
263
|
+
<details><summary>Dropping a table</summary>
|
|
264
|
+
|
|
265
|
+
Dropping a table can be difficult to achieve in a small amount of time if it holds several foreign keys to busy tables.
|
|
266
|
+
To remove the table, PostgreSQL will have to acquire an access exclusive lock on all the tables referenced by the foreign keys.
|
|
267
|
+
|
|
268
|
+
To solve this issue, **Safe Pg Migrations** will drop the foreign keys before dropping the table.
|
|
269
|
+
|
|
270
|
+
---
|
|
271
|
+
**NOTE**
|
|
272
|
+
|
|
273
|
+
Dropping a table is a dangerous operation by nature. **Safe Pg Migrations** will not prevent the deletion of a table which
|
|
274
|
+
would still be in use.
|
|
275
|
+
|
|
276
|
+
---
|
|
277
|
+
|
|
148
278
|
</details>
|
|
149
279
|
|
|
150
280
|
<details><summary>Verbose SQL logging</summary>
|
|
@@ -195,31 +325,32 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
|
|
|
195
325
|
**Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
|
|
196
326
|
|
|
197
327
|
```ruby
|
|
198
|
-
SafePgMigrations.config.safe_timeout = 5.seconds #
|
|
328
|
+
SafePgMigrations.config.safe_timeout = 5.seconds # Statement timeout used for all DDL operations except from CREATE / DROP INDEX
|
|
329
|
+
|
|
330
|
+
SafePgMigrations.config.lock_timeout = nil # Lock timeout used for all DDL operations except from CREATE / DROP INDEX. If not set, safe_timeout will be used with a deduction of 1% to ensure that the lock timeout is raised in priority
|
|
199
331
|
|
|
200
332
|
SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
|
|
201
333
|
|
|
334
|
+
SafePgMigrations.config.sensitive_logger = nil # When given, sensitive data will be sent to this logger instead of the standard output. Must implement method `info`.
|
|
335
|
+
|
|
202
336
|
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
|
203
337
|
|
|
204
|
-
SafePgMigrations.config.
|
|
338
|
+
SafePgMigrations.config.backfill_batch_size = 100_000 # Size of the batches used for backfilling when adding a column with a default value
|
|
205
339
|
|
|
206
|
-
SafePgMigrations.config.
|
|
340
|
+
SafePgMigrations.config.backfill_pause = 0.5.second # Delay between each batch during a backfill. This ensure replication can happen safely.
|
|
207
341
|
|
|
208
|
-
SafePgMigrations.config.
|
|
209
|
-
```
|
|
342
|
+
SafePgMigrations.config.default_value_backfill_threshold = nil # When set, batch backfill will only be available if the table is under the given threshold. If the number of rows is higher (according to stats), the migration will fail.
|
|
210
343
|
|
|
211
|
-
|
|
344
|
+
SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
|
|
212
345
|
|
|
213
|
-
|
|
214
|
-
bundle
|
|
215
|
-
psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
|
|
216
|
-
rake test
|
|
346
|
+
SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
|
|
217
347
|
```
|
|
218
348
|
|
|
219
349
|
## Authors
|
|
220
350
|
|
|
221
351
|
- [Matthieu Prat](https://github.com/matthieuprat)
|
|
222
352
|
- [Romain Choquet](https://github.com/rchoquet)
|
|
353
|
+
- [Thomas Hareau](https://github.com/ThHareau)
|
|
223
354
|
- [Paul-Etienne Coisne](https://github.com/coisnepe)
|
|
224
355
|
|
|
225
356
|
## License
|
|
@@ -1,13 +1,20 @@
|
|
|
1
1
|
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
require 'safe-pg-migrations/configuration'
|
|
4
|
+
require 'safe-pg-migrations/helpers/logger'
|
|
5
|
+
require 'safe-pg-migrations/helpers/satisfied_helper'
|
|
6
|
+
require 'safe-pg-migrations/helpers/index_helper'
|
|
7
|
+
require 'safe-pg-migrations/helpers/batch_over'
|
|
8
|
+
require 'safe-pg-migrations/helpers/session_setting_management'
|
|
4
9
|
require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
|
5
10
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
|
11
|
+
require 'safe-pg-migrations/plugins/statement_insurer/add_column'
|
|
12
|
+
require 'safe-pg-migrations/plugins/statement_insurer/change_column_null'
|
|
6
13
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
|
7
14
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
|
8
15
|
require 'safe-pg-migrations/plugins/idempotent_statements'
|
|
9
16
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
|
10
|
-
require 'safe-pg-migrations/
|
|
17
|
+
require 'safe-pg-migrations/plugins/strong_migrations_integration'
|
|
11
18
|
require 'safe-pg-migrations/polyfills/index_definition_polyfill'
|
|
12
19
|
require 'safe-pg-migrations/polyfills/verbose_query_logs_polyfill'
|
|
13
20
|
|
|
@@ -23,19 +30,34 @@ module SafePgMigrations
|
|
|
23
30
|
].freeze
|
|
24
31
|
|
|
25
32
|
class << self
|
|
26
|
-
attr_reader :current_migration
|
|
33
|
+
attr_reader :current_migration, :pg_version_num
|
|
27
34
|
|
|
28
35
|
def setup_and_teardown(migration, connection, &block)
|
|
36
|
+
@pg_version_num = get_pg_version_num(connection)
|
|
29
37
|
@alternate_connection = nil
|
|
30
|
-
@current_migration = migration
|
|
31
|
-
stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
|
|
32
|
-
PLUGINS.each { |plugin| connection.extend(plugin) }
|
|
33
38
|
|
|
34
|
-
|
|
39
|
+
with_current_migration(migration) do
|
|
40
|
+
stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
|
|
41
|
+
|
|
42
|
+
VerboseSqlLogger.new.setup if verbose?
|
|
43
|
+
PLUGINS.each { |plugin| connection.extend(plugin) }
|
|
44
|
+
|
|
45
|
+
connection.with_setting :lock_timeout, SafePgMigrations.config.pg_lock_timeout do
|
|
46
|
+
connection.with_setting :statement_timeout, SafePgMigrations.config.pg_statement_timeout, &block
|
|
47
|
+
end
|
|
48
|
+
ensure
|
|
49
|
+
stdout_sql_logger&.teardown
|
|
50
|
+
end
|
|
35
51
|
ensure
|
|
36
52
|
close_alternate_connection
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
def with_current_migration(migration, &block)
|
|
56
|
+
@current_migration = migration
|
|
57
|
+
|
|
58
|
+
yield block
|
|
59
|
+
ensure
|
|
37
60
|
@current_migration = nil
|
|
38
|
-
stdout_sql_logger&.teardown
|
|
39
61
|
end
|
|
40
62
|
|
|
41
63
|
def alternate_connection
|
|
@@ -49,16 +71,6 @@ module SafePgMigrations
|
|
|
49
71
|
@alternate_connection = nil
|
|
50
72
|
end
|
|
51
73
|
|
|
52
|
-
ruby2_keywords def say(*args)
|
|
53
|
-
return unless current_migration
|
|
54
|
-
|
|
55
|
-
current_migration.say(*args)
|
|
56
|
-
end
|
|
57
|
-
|
|
58
|
-
ruby2_keywords def say_method_call(method, *args)
|
|
59
|
-
say "#{method}(#{args.map(&:inspect) * ', '})", true
|
|
60
|
-
end
|
|
61
|
-
|
|
62
74
|
def verbose?
|
|
63
75
|
unless current_migration.class._safe_pg_migrations_verbose.nil?
|
|
64
76
|
return current_migration.class._safe_pg_migrations_verbose
|
|
@@ -72,9 +84,15 @@ module SafePgMigrations
|
|
|
72
84
|
def config
|
|
73
85
|
@config ||= Configuration.new
|
|
74
86
|
end
|
|
87
|
+
|
|
88
|
+
def get_pg_version_num(connection)
|
|
89
|
+
connection.query_value('SHOW server_version_num').to_i
|
|
90
|
+
end
|
|
75
91
|
end
|
|
76
92
|
|
|
77
93
|
module Migration
|
|
94
|
+
include StrongMigrationsIntegration
|
|
95
|
+
|
|
78
96
|
module ClassMethods
|
|
79
97
|
attr_accessor :_safe_pg_migrations_verbose
|
|
80
98
|
|
|
@@ -90,27 +108,11 @@ module SafePgMigrations
|
|
|
90
108
|
end
|
|
91
109
|
|
|
92
110
|
def disable_ddl_transaction
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
end
|
|
111
|
+
SafePgMigrations.with_current_migration(self) do
|
|
112
|
+
UselessStatementsLogger.warn_useless '`disable_ddl_transaction`' if super
|
|
96
113
|
|
|
97
|
-
|
|
98
|
-
execute
|
|
99
|
-
add_column
|
|
100
|
-
add_index
|
|
101
|
-
add_reference
|
|
102
|
-
add_belongs_to
|
|
103
|
-
change_column_null
|
|
104
|
-
add_foreign_key
|
|
105
|
-
].freeze
|
|
106
|
-
|
|
107
|
-
SAFE_METHODS.each do |method|
|
|
108
|
-
define_method method do |*args|
|
|
109
|
-
return super(*args) unless respond_to?(:safety_assured)
|
|
110
|
-
|
|
111
|
-
safety_assured { super(*args) }
|
|
114
|
+
true
|
|
112
115
|
end
|
|
113
|
-
ruby2_keywords method
|
|
114
116
|
end
|
|
115
117
|
end
|
|
116
118
|
end
|
|
@@ -4,22 +4,66 @@ require 'active_support/core_ext/numeric/time'
|
|
|
4
4
|
|
|
5
5
|
module SafePgMigrations
|
|
6
6
|
class Configuration
|
|
7
|
-
attr_accessor
|
|
8
|
-
|
|
7
|
+
attr_accessor(*%i[
|
|
8
|
+
blocking_activity_logger_margin
|
|
9
|
+
blocking_activity_logger_verbose
|
|
10
|
+
default_value_backfill_threshold
|
|
11
|
+
backfill_batch_size
|
|
12
|
+
backfill_pause
|
|
13
|
+
retry_delay
|
|
14
|
+
max_tries
|
|
15
|
+
sensitive_logger
|
|
16
|
+
])
|
|
17
|
+
attr_reader :lock_timeout, :safe_timeout
|
|
9
18
|
|
|
10
19
|
def initialize
|
|
20
|
+
self.default_value_backfill_threshold = nil
|
|
11
21
|
self.safe_timeout = 5.seconds
|
|
22
|
+
self.lock_timeout = nil
|
|
12
23
|
self.blocking_activity_logger_margin = 1.second
|
|
13
24
|
self.blocking_activity_logger_verbose = true
|
|
14
|
-
self.
|
|
25
|
+
self.backfill_batch_size = 100_000
|
|
26
|
+
self.backfill_pause = 0.5.second
|
|
15
27
|
self.retry_delay = 1.minute
|
|
16
28
|
self.max_tries = 5
|
|
29
|
+
self.sensitive_logger = nil
|
|
17
30
|
end
|
|
18
31
|
|
|
19
|
-
def
|
|
20
|
-
|
|
32
|
+
def lock_timeout=(value)
|
|
33
|
+
raise 'Setting lock timeout to 0 disables the lock timeout and is dangerous' if value == 0.seconds
|
|
34
|
+
|
|
35
|
+
unless value.nil? || value < safe_timeout
|
|
36
|
+
raise ArgumentError, "Lock timeout (#{value}) cannot be greater than safe timeout (#{safe_timeout})"
|
|
37
|
+
end
|
|
38
|
+
|
|
39
|
+
@lock_timeout = value
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
def safe_timeout=(value)
|
|
43
|
+
raise 'Setting safe timeout to 0 disables the safe timeout and is dangerous' unless value
|
|
44
|
+
|
|
45
|
+
unless lock_timeout.nil? || value > lock_timeout
|
|
46
|
+
raise ArgumentError, "Safe timeout (#{value}) cannot be less than lock timeout (#{lock_timeout})"
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
@safe_timeout = value
|
|
21
50
|
end
|
|
22
51
|
|
|
52
|
+
def pg_statement_timeout
|
|
53
|
+
pg_duration safe_timeout
|
|
54
|
+
end
|
|
55
|
+
|
|
56
|
+
def pg_lock_timeout
|
|
57
|
+
return pg_duration lock_timeout if lock_timeout
|
|
58
|
+
|
|
59
|
+
# if statement timeout and lock timeout have the same value, statement timeout will raise in priority. We actually
|
|
60
|
+
# need the opposite for BlockingActivityLogger to detect lock timeouts correctly.
|
|
61
|
+
# By reducing the lock timeout by a very small margin, we ensure that the lock timeout is raised in priority
|
|
62
|
+
pg_duration safe_timeout * 0.99
|
|
63
|
+
end
|
|
64
|
+
|
|
65
|
+
private
|
|
66
|
+
|
|
23
67
|
def pg_duration(duration)
|
|
24
68
|
value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
|
|
25
69
|
"#{value}#{unit}"
|
|
@@ -0,0 +1,56 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SafePgMigrations
|
|
4
|
+
module Helpers
|
|
5
|
+
class BatchOver
|
|
6
|
+
def initialize(model, of: SafePgMigrations.config.backfill_batch_size)
|
|
7
|
+
@model = model
|
|
8
|
+
@of = of
|
|
9
|
+
|
|
10
|
+
@current_range = nil
|
|
11
|
+
end
|
|
12
|
+
|
|
13
|
+
def each_batch
|
|
14
|
+
yield scope.where(primary_key => @current_range) while next_batch
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
private
|
|
18
|
+
|
|
19
|
+
def next_batch
|
|
20
|
+
return if endless?
|
|
21
|
+
|
|
22
|
+
first = next_scope.take
|
|
23
|
+
|
|
24
|
+
return unless first
|
|
25
|
+
|
|
26
|
+
last = next_scope.offset(@of).take
|
|
27
|
+
|
|
28
|
+
first_key = first[primary_key]
|
|
29
|
+
last_key = last.nil? ? nil : last[primary_key]
|
|
30
|
+
|
|
31
|
+
@current_range = first_key...last_key
|
|
32
|
+
end
|
|
33
|
+
|
|
34
|
+
def next_scope
|
|
35
|
+
return scope if @current_range.nil?
|
|
36
|
+
return scope.none if endless?
|
|
37
|
+
|
|
38
|
+
scope.where(primary_key => @current_range.end..)
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def scope
|
|
42
|
+
@model.order(primary_key => :asc)
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def endless?
|
|
46
|
+
return false if @current_range.nil?
|
|
47
|
+
|
|
48
|
+
@current_range.end.nil?
|
|
49
|
+
end
|
|
50
|
+
|
|
51
|
+
def primary_key
|
|
52
|
+
@model.primary_key
|
|
53
|
+
end
|
|
54
|
+
end
|
|
55
|
+
end
|
|
56
|
+
end
|
|
@@ -5,20 +5,20 @@ module SafePgMigrations
|
|
|
5
5
|
module BlockingActivityFormatter
|
|
6
6
|
def log_queries(queries)
|
|
7
7
|
if queries.empty?
|
|
8
|
-
|
|
8
|
+
Logger.say 'Could not find any blocking query.', sub_item: true
|
|
9
9
|
else
|
|
10
|
-
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
14
|
-
|
|
10
|
+
Logger.say <<~MESSAGE.squish, sub_item: true
|
|
11
|
+
Statement was being blocked by the following #{'query'.pluralize(queries.size)}:
|
|
12
|
+
MESSAGE
|
|
13
|
+
|
|
14
|
+
Logger.say '', sub_item: true
|
|
15
15
|
output_blocking_queries(queries)
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
21
|
-
|
|
16
|
+
Logger.say <<~MESSAGE.squish, sub_item: true
|
|
17
|
+
Beware, some of those queries might run in a transaction. In this case the locking query might be located
|
|
18
|
+
elsewhere in the transaction
|
|
19
|
+
MESSAGE
|
|
20
|
+
|
|
21
|
+
Logger.say '', sub_item: true
|
|
22
22
|
end
|
|
23
23
|
end
|
|
24
24
|
|
|
@@ -27,8 +27,10 @@ module SafePgMigrations
|
|
|
27
27
|
def output_blocking_queries(queries)
|
|
28
28
|
if SafePgMigrations.config.blocking_activity_logger_verbose
|
|
29
29
|
queries.each do |pid, query, start_time|
|
|
30
|
-
|
|
31
|
-
|
|
30
|
+
Logger.say(
|
|
31
|
+
"Query with pid #{pid || 'null'} started #{format_start_time start_time}: #{query}",
|
|
32
|
+
sub_item: true, sensitive: true
|
|
33
|
+
)
|
|
32
34
|
end
|
|
33
35
|
else
|
|
34
36
|
output_confidentially_blocking_queries(queries)
|
|
@@ -37,14 +39,13 @@ module SafePgMigrations
|
|
|
37
39
|
|
|
38
40
|
def output_confidentially_blocking_queries(queries)
|
|
39
41
|
queries.each do |start_time, locktype, mode, pid, transactionid|
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
)
|
|
42
|
+
Logger.say <<~MESSAGE.squish, sub_item: true, sensitive: true
|
|
43
|
+
Query with pid #{pid || 'null'}
|
|
44
|
+
started #{format_start_time(start_time)}:
|
|
45
|
+
lock type: #{locktype || 'null'},
|
|
46
|
+
lock mode: #{mode || 'null'},
|
|
47
|
+
lock transactionid: #{transactionid || 'null'}",
|
|
48
|
+
MESSAGE
|
|
48
49
|
end
|
|
49
50
|
end
|
|
50
51
|
|
|
@@ -0,0 +1,24 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SafePgMigrations
|
|
4
|
+
module Helpers
|
|
5
|
+
module IndexHelper
|
|
6
|
+
def index_definition(table_name, column_name, **options)
|
|
7
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
|
8
|
+
index_definition
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
private
|
|
12
|
+
|
|
13
|
+
def index_valid?(index_name)
|
|
14
|
+
query_value <<~SQL.squish
|
|
15
|
+
SELECT indisvalid
|
|
16
|
+
FROM pg_index i
|
|
17
|
+
JOIN pg_class c
|
|
18
|
+
ON i.indexrelid = c.oid
|
|
19
|
+
WHERE c.relname = '#{index_name}';
|
|
20
|
+
SQL
|
|
21
|
+
end
|
|
22
|
+
end
|
|
23
|
+
end
|
|
24
|
+
end
|
|
@@ -0,0 +1,47 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SafePgMigrations
|
|
4
|
+
module Helpers
|
|
5
|
+
module Logger
|
|
6
|
+
class << self
|
|
7
|
+
def say(message, sub_item: false, sensitive: false, warn_sensitive_logs: true)
|
|
8
|
+
return unless SafePgMigrations.current_migration
|
|
9
|
+
|
|
10
|
+
if sensitive
|
|
11
|
+
log_sensitive message, sub_item: sub_item
|
|
12
|
+
if warn_sensitive_logs && sensitive_logger?
|
|
13
|
+
log 'Sensitive data sent to sensitive logger', sub_item: sub_item
|
|
14
|
+
end
|
|
15
|
+
else
|
|
16
|
+
log message, sub_item: sub_item
|
|
17
|
+
end
|
|
18
|
+
end
|
|
19
|
+
|
|
20
|
+
def say_method_call(method, *args, sensitive: false, warn_sensitive_logs: true, **options)
|
|
21
|
+
args += [options] unless options.empty?
|
|
22
|
+
|
|
23
|
+
say "#{method}(#{args.map(&:inspect) * ', '})",
|
|
24
|
+
sub_item: true, sensitive: sensitive, warn_sensitive_logs: warn_sensitive_logs
|
|
25
|
+
end
|
|
26
|
+
|
|
27
|
+
private
|
|
28
|
+
|
|
29
|
+
def log(message, sub_item:)
|
|
30
|
+
SafePgMigrations.current_migration.say message, sub_item
|
|
31
|
+
end
|
|
32
|
+
|
|
33
|
+
def log_sensitive(message, sub_item:)
|
|
34
|
+
if sensitive_logger?
|
|
35
|
+
SafePgMigrations.config.sensitive_logger.info message
|
|
36
|
+
else
|
|
37
|
+
log message, sub_item: sub_item
|
|
38
|
+
end
|
|
39
|
+
end
|
|
40
|
+
|
|
41
|
+
def sensitive_logger?
|
|
42
|
+
SafePgMigrations.config.sensitive_logger
|
|
43
|
+
end
|
|
44
|
+
end
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
@@ -0,0 +1,31 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
module SafePgMigrations
|
|
4
|
+
module Helpers
|
|
5
|
+
module SatisfiedHelper
|
|
6
|
+
class << self
|
|
7
|
+
def satisfies_change_column_null_requirements?
|
|
8
|
+
satisfies_add_check_constraints? && SafePgMigrations.pg_version_num >= 120_000
|
|
9
|
+
end
|
|
10
|
+
|
|
11
|
+
def satisfies_add_check_constraints!
|
|
12
|
+
return if satisfies_add_check_constraints?
|
|
13
|
+
|
|
14
|
+
raise NotImplementedError, 'add_check_constraint is not supported in your ActiveRecord version'
|
|
15
|
+
end
|
|
16
|
+
|
|
17
|
+
def satisfies_add_check_constraints?
|
|
18
|
+
satisfied? '>=6.1.0'
|
|
19
|
+
end
|
|
20
|
+
|
|
21
|
+
def satisfies_add_column_update_rows_backfill?
|
|
22
|
+
satisfies_change_column_null_requirements?
|
|
23
|
+
end
|
|
24
|
+
|
|
25
|
+
def satisfied?(version)
|
|
26
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
|
27
|
+
end
|
|
28
|
+
end
|
|
29
|
+
end
|
|
30
|
+
end
|
|
31
|
+
end
|