safe-pg-migrations 2.0.0 → 2.2.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 878eca03499c2fa44ef8e046f5828b2b0978e692ade4440f9489a29889a99b49
4
- data.tar.gz: 88a9aa0adf58a3a716565d8a3482839ffd546570ab7238f67daecb0c2396a152
3
+ metadata.gz: 624821b7134dee05186350edada6f27faaf8c820976837d32403dc5fffa4880a
4
+ data.tar.gz: bf41324b8d563de7bc893a302d45d8f175e890bb888a2f52e0b14be7676f942c
5
5
  SHA512:
6
- metadata.gz: 15cd514d22cf2b5f182faf16a50a4775fa528754357eeeaa0c81f2f93bf25d12375ae2c7742e8e762addd01ae3b5aeeedb09c758e9e6da063f4b0beca8544c45
7
- data.tar.gz: d930ac0e741025ea05743a2ad8f6da979932f67cd9e93e13c64b22581bb073b164212144d152f9a0729a93e49098cb9279caba57a91b746502ccb259cb18bbcb
6
+ metadata.gz: 76601dcf0b97013edc36de6ac351beb8e4f6c532a0e223cba95d1a29e4eb7f18b8a4e7b3e3f28e4e7bde02144a4de8c512c19e2725396578dcb322e040963168
7
+ data.tar.gz: 767876e75af3292f9c4d3fc82810ab37bb3d38dad7b8b87ee6ba7068ec5e4aa9967ac0e53170be77bda07dd3424f2d3b3b3a03d9b8dc6ebcda94cee21e61fcfc
data/README.md CHANGED
@@ -109,7 +109,51 @@ Beware though, when adding a volatile default value:
109
109
  ```ruby
110
110
  add_column :users, :created_at, default: 'clock_timestamp()'
111
111
  ```
112
- PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, your best bet is to add the column without default, set the default, and backfill existing rows.
112
+ PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, **Safe PG Migrations** can automatically backfill data when the option `default_value_backfill:` is set to `:update_in_batches`.
113
+
114
+ </details>
115
+
116
+ <details>
117
+ <summary>Safe add_column - adding a volatile default value</summary>
118
+
119
+ **Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a volatile default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
120
+
121
+ ```ruby
122
+ safety_assured do
123
+ add_column :users, :created_at, default: 'clock_timestamp()', default_value_backfill: :update_in_batches
124
+ end
125
+ ```
126
+
127
+ More specifically, it will:
128
+
129
+ 1. create the column without default value and without null constraint. This ensure the `ACCESS EXCLUSIVE` lock is acquired for the least amount of time;
130
+ 2. add the default value, without data backfill. An `ACCESS EXCLUSIVE` lock is acquired and released immediately;
131
+ 3. backfill data, in batch of `SafePgMigrations.config.backfill_batch_size` and with a pause of `SafePgMigrations.config.backfill_pause` between each batch;
132
+ 4. change the column to `null: false`, if defined in the parameters, following the algorithm we have defined below.
133
+
134
+ ---
135
+ **NOTE**
136
+
137
+ Data backfill take time. If your table is big, your migrations will (safely) hangs for a while. You might want to backfill data manually instead, to do so you will need two migrations
138
+
139
+ 1. First migration :
140
+
141
+ a. adds the column without default and without null constraint;
142
+
143
+ b. add the default value.
144
+
145
+ 2. manual data backfill (rake task, manual operation, ...)
146
+ 3. Second migration which change the column to null false (with **Safe PG Migrations**, `change_column_null` is safe and can be used; see section below)
147
+ ---
148
+
149
+ `default_value_backfill:` also accept the value `:auto` which is set by default. In this case, **Safe PG Migrations** will not backfill data and will let PostgreSQL handle it itself.
150
+
151
+ ### Preventing :update_in_batches when the table is too big
152
+
153
+ `add_column` with `default_value_backfill: :update_in_batches` can be dangerous on big tables. To avoid unwanted long migrations, **Safe PG Migrations** does not automatically mark this usage as safe when used with `strong-migrations`, usage of `safety_assured` is required.
154
+
155
+ It is also possible to set a threshold for the table size, above which the migration will fail. This can be done by setting the `default_value_backfill_threshold:` option in the configuration.
156
+
113
157
 
114
158
  </details>
115
159
 
@@ -130,13 +174,70 @@ If you still get lock timeout while adding / removing indexes, it might be for o
130
174
 
131
175
  Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
132
176
 
133
- Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus safe-pg-migrations ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
177
+ Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus **Safe PG Migrations** ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
134
178
 
135
179
  1. adding the constraint *without validation*, will not validate existing rows;
136
180
  2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
137
181
 
138
182
  </details>
139
183
 
184
+
185
+ <details><summary id="safe_add_check_constraint">Safe <code>add_check_constraint</code> (ActiveRecord > 6.1)</summary>
186
+
187
+ Adding a check constraint requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
188
+
189
+ Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
190
+ Thus **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least amount of time by
191
+ splitting the constraint addition in two steps:
192
+
193
+ 1. adding the constraint *without validation*, will not validate existing rows;
194
+ 2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
195
+
196
+ </details>
197
+
198
+ <details><summary id="safe_change_column_null">Safe <code>change_column_null</code> (ActiveRecord and PG version dependant)</summary>
199
+
200
+ Changing the nullability of a column requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
201
+
202
+ Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
203
+
204
+ **Safe PG Migrations** acts differently depending on the version you are on.
205
+
206
+ ### Recent versions of PG and Active Record (> 12 and > 6.1)
207
+
208
+ Starting on PostgreSQL versions 12, adding the column NOT NULL constraint is safe if a check constraint validates the
209
+ nullability of the same column. **Safe PG Migrations** also relies on add_check_constraint, which was introduced in
210
+ ActiveRecord 6.1.
211
+
212
+ If these requirements are met, **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least
213
+ amount of time by splitting the constraint addition in several steps:
214
+
215
+ 1. adding a `IS NOT NULL` constraint *without validation*, will not validate existing rows but block read or write;
216
+ 2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table;
217
+ 3. changing the not null status of the column, thanks to the NOT NULL constraint without having to scan the table sequentially;
218
+ 4. dropping the `IS NOT NULL` constraint.
219
+
220
+ ### Older versions of PG or ActiveRecord
221
+
222
+ If the version of PostgreSQL is below 12, or if the version of ActiveRecord is below 6.1, **Safe PG Migrations** will only
223
+ wrap ActiveRecord method into a statement timeout and lock timeout.
224
+
225
+ ### Call with a default parameter
226
+
227
+ Calling change_column_null with a default parameter [is dangerous](https://github.com/rails/rails/blob/716baea69f989b64f5bfeaff880c2512377bebab/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb#L446)
228
+ and is likely not to finish in the statement timeout defined by **Safe PG Migrations**. For this reason, when the default
229
+ parameter is given, **Safe PG Migrations** will simply forward it to activerecord methods without trying to improve it
230
+
231
+ ### Dropping a NULL constraint
232
+
233
+ Dropping a null constraint still requires an `ACCESS EXCLUSIVE` lock, but does not require extra operation to reduce the
234
+ amount of time during which the lock is held. **Safe PG Migrations** only wrap methods of activerecord in lock and statement
235
+ timeouts
236
+
237
+ </details>
238
+
239
+
240
+
140
241
  <details><summary>Retry after lock timeout</summary>
141
242
 
142
243
  When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
@@ -145,6 +246,35 @@ When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5
145
246
  <details><summary>Blocking activity logging</summary>
146
247
 
147
248
  If a statement fails with a lock timeout, **Safe PG Migrations** will try to tell you what was the blocking statement.
249
+
250
+ ---
251
+ **NOTE**
252
+
253
+ Data logged by the Blocking activity logger can be sensitive (it will contain raw SQL queries, which can be hashes of password, user information, ...)
254
+
255
+ If you cannot afford to log this type of data, you can either
256
+ * Set `SafePgMigrations.config.blocking_activity_logger_verbose = false`. In this case, the logger will only log the pid of the blocking statement, which should be enough to investigate;
257
+ * Provide a different logger through `SafePgMigrations.config.sensitive_logger = YourLogger.new`. Instead of using the default IO stream, SafePgMigrations will send sensitive data to the given logger which can be managed as you wish.
258
+
259
+ ---
260
+
261
+ </details>
262
+
263
+ <details><summary>Dropping a table</summary>
264
+
265
+ Dropping a table can be difficult to achieve in a small amount of time if it holds several foreign keys to busy tables.
266
+ To remove the table, PostgreSQL will have to acquire an access exclusive lock on all the tables referenced by the foreign keys.
267
+
268
+ To solve this issue, **Safe Pg Migrations** will drop the foreign keys before dropping the table.
269
+
270
+ ---
271
+ **NOTE**
272
+
273
+ Dropping a table is a dangerous operation by nature. **Safe Pg Migrations** will not prevent the deletion of a table which
274
+ would still be in use.
275
+
276
+ ---
277
+
148
278
  </details>
149
279
 
150
280
  <details><summary>Verbose SQL logging</summary>
@@ -195,31 +325,32 @@ So you can actually check that the `CREATE INDEX` statement will be performed co
195
325
  **Safe PG Migrations** can be customized, here is an example of a Rails initializer (the values are the default ones):
196
326
 
197
327
  ```ruby
198
- SafePgMigrations.config.safe_timeout = 5.seconds # Lock and statement timeout used for all DDL operations except from CREATE / DROP INDEX
328
+ SafePgMigrations.config.safe_timeout = 5.seconds # Statement timeout used for all DDL operations except from CREATE / DROP INDEX
329
+
330
+ SafePgMigrations.config.lock_timeout = nil # Lock timeout used for all DDL operations except from CREATE / DROP INDEX. If not set, safe_timeout will be used with a deduction of 1% to ensure that the lock timeout is raised in priority
199
331
 
200
332
  SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the raw blocking queries on timeout. When false, outputs information about the lock instead
201
333
 
334
+ SafePgMigrations.config.sensitive_logger = nil # When given, sensitive data will be sent to this logger instead of the standard output. Must implement method `info`.
335
+
202
336
  SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
203
337
 
204
- SafePgMigrations.config.batch_size = 1000 # Size of the batches used for backfilling when adding a column with a default value pre-PG11
338
+ SafePgMigrations.config.backfill_batch_size = 100_000 # Size of the batches used for backfilling when adding a column with a default value
205
339
 
206
- SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
340
+ SafePgMigrations.config.backfill_pause = 0.5.second # Delay between each batch during a backfill. This ensure replication can happen safely.
207
341
 
208
- SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
209
- ```
342
+ SafePgMigrations.config.default_value_backfill_threshold = nil # When set, batch backfill will only be available if the table is under the given threshold. If the number of rows is higher (according to stats), the migration will fail.
210
343
 
211
- ## Running tests
344
+ SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
212
345
 
213
- ```bash
214
- bundle
215
- psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
216
- rake test
346
+ SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
217
347
  ```
218
348
 
219
349
  ## Authors
220
350
 
221
351
  - [Matthieu Prat](https://github.com/matthieuprat)
222
352
  - [Romain Choquet](https://github.com/rchoquet)
353
+ - [Thomas Hareau](https://github.com/ThHareau)
223
354
  - [Paul-Etienne Coisne](https://github.com/coisnepe)
224
355
 
225
356
  ## License
@@ -1,13 +1,20 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  require 'safe-pg-migrations/configuration'
4
+ require 'safe-pg-migrations/helpers/logger'
5
+ require 'safe-pg-migrations/helpers/satisfied_helper'
6
+ require 'safe-pg-migrations/helpers/index_helper'
7
+ require 'safe-pg-migrations/helpers/batch_over'
8
+ require 'safe-pg-migrations/helpers/session_setting_management'
4
9
  require 'safe-pg-migrations/plugins/verbose_sql_logger'
5
10
  require 'safe-pg-migrations/plugins/blocking_activity_logger'
11
+ require 'safe-pg-migrations/plugins/statement_insurer/add_column'
12
+ require 'safe-pg-migrations/plugins/statement_insurer/change_column_null'
6
13
  require 'safe-pg-migrations/plugins/statement_insurer'
7
14
  require 'safe-pg-migrations/plugins/statement_retrier'
8
15
  require 'safe-pg-migrations/plugins/idempotent_statements'
9
16
  require 'safe-pg-migrations/plugins/useless_statements_logger'
10
- require 'safe-pg-migrations/polyfills/satisfied_helper'
17
+ require 'safe-pg-migrations/plugins/strong_migrations_integration'
11
18
  require 'safe-pg-migrations/polyfills/index_definition_polyfill'
12
19
  require 'safe-pg-migrations/polyfills/verbose_query_logs_polyfill'
13
20
 
@@ -23,19 +30,34 @@ module SafePgMigrations
23
30
  ].freeze
24
31
 
25
32
  class << self
26
- attr_reader :current_migration
33
+ attr_reader :current_migration, :pg_version_num
27
34
 
28
35
  def setup_and_teardown(migration, connection, &block)
36
+ @pg_version_num = get_pg_version_num(connection)
29
37
  @alternate_connection = nil
30
- @current_migration = migration
31
- stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
32
- PLUGINS.each { |plugin| connection.extend(plugin) }
33
38
 
34
- connection.with_setting(:lock_timeout, SafePgMigrations.config.pg_safe_timeout, &block)
39
+ with_current_migration(migration) do
40
+ stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
41
+
42
+ VerboseSqlLogger.new.setup if verbose?
43
+ PLUGINS.each { |plugin| connection.extend(plugin) }
44
+
45
+ connection.with_setting :lock_timeout, SafePgMigrations.config.pg_lock_timeout do
46
+ connection.with_setting :statement_timeout, SafePgMigrations.config.pg_statement_timeout, &block
47
+ end
48
+ ensure
49
+ stdout_sql_logger&.teardown
50
+ end
35
51
  ensure
36
52
  close_alternate_connection
53
+ end
54
+
55
+ def with_current_migration(migration, &block)
56
+ @current_migration = migration
57
+
58
+ yield block
59
+ ensure
37
60
  @current_migration = nil
38
- stdout_sql_logger&.teardown
39
61
  end
40
62
 
41
63
  def alternate_connection
@@ -49,16 +71,6 @@ module SafePgMigrations
49
71
  @alternate_connection = nil
50
72
  end
51
73
 
52
- ruby2_keywords def say(*args)
53
- return unless current_migration
54
-
55
- current_migration.say(*args)
56
- end
57
-
58
- ruby2_keywords def say_method_call(method, *args)
59
- say "#{method}(#{args.map(&:inspect) * ', '})", true
60
- end
61
-
62
74
  def verbose?
63
75
  unless current_migration.class._safe_pg_migrations_verbose.nil?
64
76
  return current_migration.class._safe_pg_migrations_verbose
@@ -72,9 +84,15 @@ module SafePgMigrations
72
84
  def config
73
85
  @config ||= Configuration.new
74
86
  end
87
+
88
+ def get_pg_version_num(connection)
89
+ connection.query_value('SHOW server_version_num').to_i
90
+ end
75
91
  end
76
92
 
77
93
  module Migration
94
+ include StrongMigrationsIntegration
95
+
78
96
  module ClassMethods
79
97
  attr_accessor :_safe_pg_migrations_verbose
80
98
 
@@ -90,27 +108,11 @@ module SafePgMigrations
90
108
  end
91
109
 
92
110
  def disable_ddl_transaction
93
- UselessStatementsLogger.warn_useless '`disable_ddl_transaction`' if super
94
- true
95
- end
111
+ SafePgMigrations.with_current_migration(self) do
112
+ UselessStatementsLogger.warn_useless '`disable_ddl_transaction`' if super
96
113
 
97
- SAFE_METHODS = %i[
98
- execute
99
- add_column
100
- add_index
101
- add_reference
102
- add_belongs_to
103
- change_column_null
104
- add_foreign_key
105
- ].freeze
106
-
107
- SAFE_METHODS.each do |method|
108
- define_method method do |*args|
109
- return super(*args) unless respond_to?(:safety_assured)
110
-
111
- safety_assured { super(*args) }
114
+ true
112
115
  end
113
- ruby2_keywords method
114
116
  end
115
117
  end
116
118
  end
@@ -4,22 +4,66 @@ require 'active_support/core_ext/numeric/time'
4
4
 
5
5
  module SafePgMigrations
6
6
  class Configuration
7
- attr_accessor :safe_timeout, :blocking_activity_logger_margin, :blocking_activity_logger_verbose, :batch_size,
8
- :retry_delay, :max_tries
7
+ attr_accessor(*%i[
8
+ blocking_activity_logger_margin
9
+ blocking_activity_logger_verbose
10
+ default_value_backfill_threshold
11
+ backfill_batch_size
12
+ backfill_pause
13
+ retry_delay
14
+ max_tries
15
+ sensitive_logger
16
+ ])
17
+ attr_reader :lock_timeout, :safe_timeout
9
18
 
10
19
  def initialize
20
+ self.default_value_backfill_threshold = nil
11
21
  self.safe_timeout = 5.seconds
22
+ self.lock_timeout = nil
12
23
  self.blocking_activity_logger_margin = 1.second
13
24
  self.blocking_activity_logger_verbose = true
14
- self.batch_size = 1000
25
+ self.backfill_batch_size = 100_000
26
+ self.backfill_pause = 0.5.second
15
27
  self.retry_delay = 1.minute
16
28
  self.max_tries = 5
29
+ self.sensitive_logger = nil
17
30
  end
18
31
 
19
- def pg_safe_timeout
20
- pg_duration(safe_timeout)
32
+ def lock_timeout=(value)
33
+ raise 'Setting lock timeout to 0 disables the lock timeout and is dangerous' if value == 0.seconds
34
+
35
+ unless value.nil? || value < safe_timeout
36
+ raise ArgumentError, "Lock timeout (#{value}) cannot be greater than safe timeout (#{safe_timeout})"
37
+ end
38
+
39
+ @lock_timeout = value
40
+ end
41
+
42
+ def safe_timeout=(value)
43
+ raise 'Setting safe timeout to 0 disables the safe timeout and is dangerous' unless value
44
+
45
+ unless lock_timeout.nil? || value > lock_timeout
46
+ raise ArgumentError, "Safe timeout (#{value}) cannot be less than lock timeout (#{lock_timeout})"
47
+ end
48
+
49
+ @safe_timeout = value
21
50
  end
22
51
 
52
+ def pg_statement_timeout
53
+ pg_duration safe_timeout
54
+ end
55
+
56
+ def pg_lock_timeout
57
+ return pg_duration lock_timeout if lock_timeout
58
+
59
+ # if statement timeout and lock timeout have the same value, statement timeout will raise in priority. We actually
60
+ # need the opposite for BlockingActivityLogger to detect lock timeouts correctly.
61
+ # By reducing the lock timeout by a very small margin, we ensure that the lock timeout is raised in priority
62
+ pg_duration safe_timeout * 0.99
63
+ end
64
+
65
+ private
66
+
23
67
  def pg_duration(duration)
24
68
  value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
25
69
  "#{value}#{unit}"
@@ -0,0 +1,56 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SafePgMigrations
4
+ module Helpers
5
+ class BatchOver
6
+ def initialize(model, of: SafePgMigrations.config.backfill_batch_size)
7
+ @model = model
8
+ @of = of
9
+
10
+ @current_range = nil
11
+ end
12
+
13
+ def each_batch
14
+ yield scope.where(primary_key => @current_range) while next_batch
15
+ end
16
+
17
+ private
18
+
19
+ def next_batch
20
+ return if endless?
21
+
22
+ first = next_scope.take
23
+
24
+ return unless first
25
+
26
+ last = next_scope.offset(@of).take
27
+
28
+ first_key = first[primary_key]
29
+ last_key = last.nil? ? nil : last[primary_key]
30
+
31
+ @current_range = first_key...last_key
32
+ end
33
+
34
+ def next_scope
35
+ return scope if @current_range.nil?
36
+ return scope.none if endless?
37
+
38
+ scope.where(primary_key => @current_range.end..)
39
+ end
40
+
41
+ def scope
42
+ @model.order(primary_key => :asc)
43
+ end
44
+
45
+ def endless?
46
+ return false if @current_range.nil?
47
+
48
+ @current_range.end.nil?
49
+ end
50
+
51
+ def primary_key
52
+ @model.primary_key
53
+ end
54
+ end
55
+ end
56
+ end
@@ -5,20 +5,20 @@ module SafePgMigrations
5
5
  module BlockingActivityFormatter
6
6
  def log_queries(queries)
7
7
  if queries.empty?
8
- SafePgMigrations.say 'Could not find any blocking query.', true
8
+ Logger.say 'Could not find any blocking query.', sub_item: true
9
9
  else
10
- SafePgMigrations.say(
11
- "Statement was being blocked by the following #{'query'.pluralize(queries.size)}:",
12
- true
13
- )
14
- SafePgMigrations.say '', true
10
+ Logger.say <<~MESSAGE.squish, sub_item: true
11
+ Statement was being blocked by the following #{'query'.pluralize(queries.size)}:
12
+ MESSAGE
13
+
14
+ Logger.say '', sub_item: true
15
15
  output_blocking_queries(queries)
16
- SafePgMigrations.say(
17
- 'Beware, some of those queries might run in a transaction. In this case the locking query might be ' \
18
- 'located elsewhere in the transaction',
19
- true
20
- )
21
- SafePgMigrations.say '', true
16
+ Logger.say <<~MESSAGE.squish, sub_item: true
17
+ Beware, some of those queries might run in a transaction. In this case the locking query might be located
18
+ elsewhere in the transaction
19
+ MESSAGE
20
+
21
+ Logger.say '', sub_item: true
22
22
  end
23
23
  end
24
24
 
@@ -27,8 +27,10 @@ module SafePgMigrations
27
27
  def output_blocking_queries(queries)
28
28
  if SafePgMigrations.config.blocking_activity_logger_verbose
29
29
  queries.each do |pid, query, start_time|
30
- SafePgMigrations.say "Query with pid #{pid || 'null'} started #{format_start_time start_time}: #{query}",
31
- true
30
+ Logger.say(
31
+ "Query with pid #{pid || 'null'} started #{format_start_time start_time}: #{query}",
32
+ sub_item: true, sensitive: true
33
+ )
32
34
  end
33
35
  else
34
36
  output_confidentially_blocking_queries(queries)
@@ -37,14 +39,13 @@ module SafePgMigrations
37
39
 
38
40
  def output_confidentially_blocking_queries(queries)
39
41
  queries.each do |start_time, locktype, mode, pid, transactionid|
40
- SafePgMigrations.say(
41
- "Query with pid #{pid || 'null'} " \
42
- "started #{format_start_time(start_time)}: " \
43
- "lock type: #{locktype || 'null'}, " \
44
- "lock mode: #{mode || 'null'}, " \
45
- "lock transactionid: #{transactionid || 'null'}",
46
- true
47
- )
42
+ Logger.say <<~MESSAGE.squish, sub_item: true, sensitive: true
43
+ Query with pid #{pid || 'null'}
44
+ started #{format_start_time(start_time)}:
45
+ lock type: #{locktype || 'null'},
46
+ lock mode: #{mode || 'null'},
47
+ lock transactionid: #{transactionid || 'null'}",
48
+ MESSAGE
48
49
  end
49
50
  end
50
51
 
@@ -0,0 +1,24 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SafePgMigrations
4
+ module Helpers
5
+ module IndexHelper
6
+ def index_definition(table_name, column_name, **options)
7
+ index_definition, = add_index_options(table_name, column_name, **options)
8
+ index_definition
9
+ end
10
+
11
+ private
12
+
13
+ def index_valid?(index_name)
14
+ query_value <<~SQL.squish
15
+ SELECT indisvalid
16
+ FROM pg_index i
17
+ JOIN pg_class c
18
+ ON i.indexrelid = c.oid
19
+ WHERE c.relname = '#{index_name}';
20
+ SQL
21
+ end
22
+ end
23
+ end
24
+ end
@@ -0,0 +1,47 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SafePgMigrations
4
+ module Helpers
5
+ module Logger
6
+ class << self
7
+ def say(message, sub_item: false, sensitive: false, warn_sensitive_logs: true)
8
+ return unless SafePgMigrations.current_migration
9
+
10
+ if sensitive
11
+ log_sensitive message, sub_item: sub_item
12
+ if warn_sensitive_logs && sensitive_logger?
13
+ log 'Sensitive data sent to sensitive logger', sub_item: sub_item
14
+ end
15
+ else
16
+ log message, sub_item: sub_item
17
+ end
18
+ end
19
+
20
+ def say_method_call(method, *args, sensitive: false, warn_sensitive_logs: true, **options)
21
+ args += [options] unless options.empty?
22
+
23
+ say "#{method}(#{args.map(&:inspect) * ', '})",
24
+ sub_item: true, sensitive: sensitive, warn_sensitive_logs: warn_sensitive_logs
25
+ end
26
+
27
+ private
28
+
29
+ def log(message, sub_item:)
30
+ SafePgMigrations.current_migration.say message, sub_item
31
+ end
32
+
33
+ def log_sensitive(message, sub_item:)
34
+ if sensitive_logger?
35
+ SafePgMigrations.config.sensitive_logger.info message
36
+ else
37
+ log message, sub_item: sub_item
38
+ end
39
+ end
40
+
41
+ def sensitive_logger?
42
+ SafePgMigrations.config.sensitive_logger
43
+ end
44
+ end
45
+ end
46
+ end
47
+ end
@@ -0,0 +1,31 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SafePgMigrations
4
+ module Helpers
5
+ module SatisfiedHelper
6
+ class << self
7
+ def satisfies_change_column_null_requirements?
8
+ satisfies_add_check_constraints? && SafePgMigrations.pg_version_num >= 120_000
9
+ end
10
+
11
+ def satisfies_add_check_constraints!
12
+ return if satisfies_add_check_constraints?
13
+
14
+ raise NotImplementedError, 'add_check_constraint is not supported in your ActiveRecord version'
15
+ end
16
+
17
+ def satisfies_add_check_constraints?
18
+ satisfied? '>=6.1.0'
19
+ end
20
+
21
+ def satisfies_add_column_update_rows_backfill?
22
+ satisfies_change_column_null_requirements?
23
+ end
24
+
25
+ def satisfied?(version)
26
+ Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
27
+ end
28
+ end
29
+ end
30
+ end
31
+ end