safe-pg-migrations 2.0.0 → 2.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +98 -3
- data/lib/safe-pg-migrations/base.rb +12 -3
- data/lib/safe-pg-migrations/configuration.rb +15 -5
- data/lib/safe-pg-migrations/helpers/batch_over.rb +56 -0
- data/lib/safe-pg-migrations/helpers/index_helper.rb +24 -0
- data/lib/safe-pg-migrations/helpers/satisfied_helper.rb +31 -0
- data/lib/safe-pg-migrations/plugins/idempotent_statements.rb +43 -13
- data/lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb +52 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +44 -5
- data/lib/safe-pg-migrations/plugins/statement_retrier.rb +1 -1
- data/lib/safe-pg-migrations/plugins/useless_statements_logger.rb +8 -0
- data/lib/safe-pg-migrations/polyfills/index_definition_polyfill.rb +2 -2
- data/lib/safe-pg-migrations/polyfills/verbose_query_logs_polyfill.rb +2 -4
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +6 -3
- data/lib/safe-pg-migrations/polyfills/satisfied_helper.rb +0 -13
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2fafd98448f0ff697da386a337dc22753be2144feedba4793f87a351581a5607
|
4
|
+
data.tar.gz: 9653ae6395d4c9f27b69dba9dee9142a4535384c42e8d05fac9ca2e2ef1bdce6
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 64297c9ba4f040e7afa21fefa2acb2a26e27fb0bfee3d9f9cbf902ee4bb0ac087f4e22591a7b5470cf73dc7bc73345e59904649c02d7540bd4693372e7069549
|
7
|
+
data.tar.gz: 7a2640fd2494480d560cfa6f19059550ecc316eaf729df6d8f97a434c189f4565c49d47acbb02f437d9b1285a466a5615a79a84d6ba1dfb3d94bb54f87b9e788
|
data/README.md
CHANGED
@@ -109,7 +109,43 @@ Beware though, when adding a volatile default value:
|
|
109
109
|
```ruby
|
110
110
|
add_column :users, :created_at, default: 'clock_timestamp()'
|
111
111
|
```
|
112
|
-
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case,
|
112
|
+
PG will still needs to update every row of the table, and will most likely statement timeout for big table. In this case, **Safe PG Migrations** can automatically backfill data when the option `default_value_backfill:` is set to `:update_in_batches`.
|
113
|
+
|
114
|
+
</details>
|
115
|
+
|
116
|
+
<details>
|
117
|
+
<summary>Safe add_column - adding a volatile default value</summary>
|
118
|
+
|
119
|
+
**Safe PG Migrations** provides the extra option parameter `default_value_backfill:`. When your migration is adding a volatile default value, the option `:update_in_batches` can be set. It will automatically backfill the value in a safe manner.
|
120
|
+
|
121
|
+
```ruby
|
122
|
+
add_column :users, :created_at, default: 'clock_timestamp()', default_value_backfill: :update_in_batches
|
123
|
+
```
|
124
|
+
|
125
|
+
More specifically, it will:
|
126
|
+
|
127
|
+
1. create the column without default value and without null constraint. This ensure the `ACCESS EXCLUSIVE` lock is acquired for the least amount of time;
|
128
|
+
2. add the default value, without data backfill. An `ACCESS EXCLUSIVE` lock is acquired and released immediately;
|
129
|
+
3. backfill data, in batch of `SafePgMigrations.config.backfill_batch_size` and with a pause of `SafePgMigrations.config.backfill_pause` between each batch;
|
130
|
+
4. change the column to `null: false`, if defined in the parameters, following the algorithm we have defined below.
|
131
|
+
|
132
|
+
---
|
133
|
+
**NOTE**
|
134
|
+
|
135
|
+
Data backfill take time. If your table is big, your migrations will (safely) hangs for a while. You might want to backfill data manually instead, to do so you will need two migrations
|
136
|
+
|
137
|
+
1. First migration :
|
138
|
+
|
139
|
+
a. adds the column without default and without null constraint;
|
140
|
+
|
141
|
+
b. add the default value.
|
142
|
+
|
143
|
+
2. manual data backfill (rake task, manual operation, ...)
|
144
|
+
3. Second migration which change the column to null false (with **Safe PG Migrations**, `change_column_null` is safe and can be used; see section below)
|
145
|
+
|
146
|
+
---
|
147
|
+
|
148
|
+
`default_value_backfill:` also accept the value `:auto` which is set by default. In this case, **Safe PG Migrations** will not backfill data and will let PostgreSQL handle it itself.
|
113
149
|
|
114
150
|
</details>
|
115
151
|
|
@@ -130,13 +166,70 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
130
166
|
|
131
167
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
132
168
|
|
133
|
-
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus
|
169
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint. Thus **Safe PG Migrations** ensures that adding a foreign key holds blocking locks for the least amount of time by splitting the foreign key creation in two steps:
|
170
|
+
|
171
|
+
1. adding the constraint *without validation*, will not validate existing rows;
|
172
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
173
|
+
|
174
|
+
</details>
|
175
|
+
|
176
|
+
|
177
|
+
<details><summary id="safe_add_check_constraint">Safe <code>add_check_constraint</code> (ActiveRecord > 6.1)</summary>
|
178
|
+
|
179
|
+
Adding a check constraint requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
180
|
+
|
181
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
182
|
+
Thus **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least amount of time by
|
183
|
+
splitting the constraint addition in two steps:
|
134
184
|
|
135
185
|
1. adding the constraint *without validation*, will not validate existing rows;
|
136
186
|
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table
|
137
187
|
|
138
188
|
</details>
|
139
189
|
|
190
|
+
<details><summary id="safe_change_column_null">Safe <code>change_column_null</code> (ActiveRecord and PG version dependant)</summary>
|
191
|
+
|
192
|
+
Changing the nullability of a column requires an `ACCESS EXCLUSIVE` lock, which **prevent writing and reading in the tables** [as soon as the lock is requested](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9).
|
193
|
+
|
194
|
+
Adding the constraint itself is rather fast, the major part of the time is spent on validating this constraint.
|
195
|
+
|
196
|
+
**Safe PG Migrations** acts differently depending on the version you are on.
|
197
|
+
|
198
|
+
### Recent versions of PG and Active Record (> 12 and > 6.1)
|
199
|
+
|
200
|
+
Starting on PostgreSQL versions 12, adding the column NOT NULL constraint is safe if a check constraint validates the
|
201
|
+
nullability of the same column. **Safe PG Migrations** also relies on add_check_constraint, which was introduced in
|
202
|
+
ActiveRecord 6.1.
|
203
|
+
|
204
|
+
If these requirements are met, **Safe PG Migrations** ensures that adding a constraints holds blocking locks for the least
|
205
|
+
amount of time by splitting the constraint addition in several steps:
|
206
|
+
|
207
|
+
1. adding a `IS NOT NULL` constraint *without validation*, will not validate existing rows but block read or write;
|
208
|
+
2. validating the constraint, will validate existing rows in the table, without blocking read or write on the table;
|
209
|
+
3. changing the not null status of the column, thanks to the NOT NULL constraint without having to scan the table sequentially;
|
210
|
+
4. dropping the `IS NOT NULL` constraint.
|
211
|
+
|
212
|
+
### Older versions of PG or ActiveRecord
|
213
|
+
|
214
|
+
If the version of PostgreSQL is below 12, or if the version of ActiveRecord is below 6.1, **Safe PG Migrations** will only
|
215
|
+
wrap ActiveRecord method into a statement timeout and lock timeout.
|
216
|
+
|
217
|
+
### Call with a default parameter
|
218
|
+
|
219
|
+
Calling change_column_null with a default parameter [is dangerous](https://github.com/rails/rails/blob/716baea69f989b64f5bfeaff880c2512377bebab/activerecord/lib/active_record/connection_adapters/postgresql/schema_statements.rb#L446)
|
220
|
+
and is likely not to finish in the statement timeout defined by **Safe PG Migrations**. For this reason, when the default
|
221
|
+
parameter is given, **Safe PG Migrations** will simply forward it to activerecord methods without trying to improve it
|
222
|
+
|
223
|
+
### Dropping a NULL constraint
|
224
|
+
|
225
|
+
Dropping a null constraint still requires an `ACCESS EXCLUSIVE` lock, but does not require extra operation to reduce the
|
226
|
+
amount of time during which the lock is held. **Safe PG Migrations** only wrap methods of activerecord in lock and statement
|
227
|
+
timeouts
|
228
|
+
|
229
|
+
</details>
|
230
|
+
|
231
|
+
|
232
|
+
|
140
233
|
<details><summary>Retry after lock timeout</summary>
|
141
234
|
|
142
235
|
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
@@ -201,7 +294,9 @@ SafePgMigrations.config.blocking_activity_logger_verbose = true # Outputs the ra
|
|
201
294
|
|
202
295
|
SafePgMigrations.config.blocking_activity_logger_margin = 1.second # Delay to output blocking queries before timeout. Must be shorter than safe_timeout
|
203
296
|
|
204
|
-
SafePgMigrations.config.
|
297
|
+
SafePgMigrations.config.backfill_batch_size = 100_000 # Size of the batches used for backfilling when adding a column with a default value
|
298
|
+
|
299
|
+
SafePgMigrations.config.backfill_pause = 0.5.second # Delay between each batch during a backfill. This ensure replication can happen safely.
|
205
300
|
|
206
301
|
SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retryable statements
|
207
302
|
|
@@ -1,13 +1,16 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
require 'safe-pg-migrations/configuration'
|
4
|
+
require 'safe-pg-migrations/helpers/satisfied_helper'
|
5
|
+
require 'safe-pg-migrations/helpers/index_helper'
|
6
|
+
require 'safe-pg-migrations/helpers/batch_over'
|
4
7
|
require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
5
8
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
9
|
+
require 'safe-pg-migrations/plugins/statement_insurer/add_column'
|
6
10
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
7
11
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
8
12
|
require 'safe-pg-migrations/plugins/idempotent_statements'
|
9
13
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
10
|
-
require 'safe-pg-migrations/polyfills/satisfied_helper'
|
11
14
|
require 'safe-pg-migrations/polyfills/index_definition_polyfill'
|
12
15
|
require 'safe-pg-migrations/polyfills/verbose_query_logs_polyfill'
|
13
16
|
|
@@ -23,15 +26,16 @@ module SafePgMigrations
|
|
23
26
|
].freeze
|
24
27
|
|
25
28
|
class << self
|
26
|
-
attr_reader :current_migration
|
29
|
+
attr_reader :current_migration, :pg_version_num
|
27
30
|
|
28
31
|
def setup_and_teardown(migration, connection, &block)
|
32
|
+
@pg_version_num = get_pg_version_num(connection)
|
29
33
|
@alternate_connection = nil
|
30
34
|
@current_migration = migration
|
31
35
|
stdout_sql_logger = VerboseSqlLogger.new.setup if verbose?
|
32
36
|
PLUGINS.each { |plugin| connection.extend(plugin) }
|
33
37
|
|
34
|
-
connection.with_setting
|
38
|
+
connection.with_setting :lock_timeout, SafePgMigrations.config.pg_lock_timeout, &block
|
35
39
|
ensure
|
36
40
|
close_alternate_connection
|
37
41
|
@current_migration = nil
|
@@ -72,6 +76,10 @@ module SafePgMigrations
|
|
72
76
|
def config
|
73
77
|
@config ||= Configuration.new
|
74
78
|
end
|
79
|
+
|
80
|
+
def get_pg_version_num(connection)
|
81
|
+
connection.query_value('SHOW server_version_num').to_i
|
82
|
+
end
|
75
83
|
end
|
76
84
|
|
77
85
|
module Migration
|
@@ -102,6 +110,7 @@ module SafePgMigrations
|
|
102
110
|
add_belongs_to
|
103
111
|
change_column_null
|
104
112
|
add_foreign_key
|
113
|
+
add_check_constraint
|
105
114
|
].freeze
|
106
115
|
|
107
116
|
SAFE_METHODS.each do |method|
|
@@ -4,22 +4,32 @@ require 'active_support/core_ext/numeric/time'
|
|
4
4
|
|
5
5
|
module SafePgMigrations
|
6
6
|
class Configuration
|
7
|
-
attr_accessor :safe_timeout, :blocking_activity_logger_margin, :blocking_activity_logger_verbose,
|
8
|
-
:retry_delay, :max_tries
|
7
|
+
attr_accessor :safe_timeout, :blocking_activity_logger_margin, :blocking_activity_logger_verbose,
|
8
|
+
:backfill_batch_size, :backfill_pause, :retry_delay, :max_tries
|
9
9
|
|
10
10
|
def initialize
|
11
11
|
self.safe_timeout = 5.seconds
|
12
12
|
self.blocking_activity_logger_margin = 1.second
|
13
13
|
self.blocking_activity_logger_verbose = true
|
14
|
-
self.
|
14
|
+
self.backfill_batch_size = 100_000
|
15
|
+
self.backfill_pause = 0.5.second
|
15
16
|
self.retry_delay = 1.minute
|
16
17
|
self.max_tries = 5
|
17
18
|
end
|
18
19
|
|
19
|
-
def
|
20
|
-
pg_duration
|
20
|
+
def pg_statement_timeout
|
21
|
+
pg_duration safe_timeout
|
21
22
|
end
|
22
23
|
|
24
|
+
def pg_lock_timeout
|
25
|
+
# if statement timeout and lock timeout have the same value, statement timeout will raise in priority. We actually
|
26
|
+
# need the opposite for BlockingActivityLogger to detect lock timeouts correctly.
|
27
|
+
# By reducing the lock timeout by a very small margin, we ensure that the lock timeout is raised in priority
|
28
|
+
pg_duration safe_timeout * 0.99
|
29
|
+
end
|
30
|
+
|
31
|
+
private
|
32
|
+
|
23
33
|
def pg_duration(duration)
|
24
34
|
value, unit = duration.integer? ? [duration, 's'] : [(duration * 1000).to_i, 'ms']
|
25
35
|
"#{value}#{unit}"
|
@@ -0,0 +1,56 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
class BatchOver
|
6
|
+
def initialize(model, of: SafePgMigrations.config.backfill_batch_size)
|
7
|
+
@model = model
|
8
|
+
@of = of
|
9
|
+
|
10
|
+
@current_range = nil
|
11
|
+
end
|
12
|
+
|
13
|
+
def each_batch
|
14
|
+
yield scope.where(primary_key => @current_range) while next_batch
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def next_batch
|
20
|
+
return if endless?
|
21
|
+
|
22
|
+
first = next_scope.take
|
23
|
+
|
24
|
+
return unless first
|
25
|
+
|
26
|
+
last = next_scope.offset(@of).take
|
27
|
+
|
28
|
+
first_key = first[primary_key]
|
29
|
+
last_key = last.nil? ? nil : last[primary_key]
|
30
|
+
|
31
|
+
@current_range = first_key...last_key
|
32
|
+
end
|
33
|
+
|
34
|
+
def next_scope
|
35
|
+
return scope if @current_range.nil?
|
36
|
+
return scope.none if endless?
|
37
|
+
|
38
|
+
scope.where(primary_key => @current_range.end..)
|
39
|
+
end
|
40
|
+
|
41
|
+
def scope
|
42
|
+
@model.order(primary_key => :asc)
|
43
|
+
end
|
44
|
+
|
45
|
+
def endless?
|
46
|
+
return false if @current_range.nil?
|
47
|
+
|
48
|
+
@current_range.end.nil?
|
49
|
+
end
|
50
|
+
|
51
|
+
def primary_key
|
52
|
+
@model.primary_key
|
53
|
+
end
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
@@ -0,0 +1,24 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module IndexHelper
|
6
|
+
def index_definition(table_name, column_name, **options)
|
7
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
8
|
+
index_definition
|
9
|
+
end
|
10
|
+
|
11
|
+
private
|
12
|
+
|
13
|
+
def index_valid?(index_name)
|
14
|
+
query_value <<~SQL.squish
|
15
|
+
SELECT indisvalid
|
16
|
+
FROM pg_index i
|
17
|
+
JOIN pg_class c
|
18
|
+
ON i.indexrelid = c.oid
|
19
|
+
WHERE c.relname = '#{index_name}';
|
20
|
+
SQL
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
24
|
+
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module Helpers
|
5
|
+
module SatisfiedHelper
|
6
|
+
class << self
|
7
|
+
def satisfies_change_column_null_requirements?
|
8
|
+
satisfies_add_check_constraints? && SafePgMigrations.pg_version_num >= 120_000
|
9
|
+
end
|
10
|
+
|
11
|
+
def satisfies_add_check_constraints!
|
12
|
+
return if satisfies_add_check_constraints?
|
13
|
+
|
14
|
+
raise NotImplementedError, 'add_check_constraint is not supported in your ActiveRecord version'
|
15
|
+
end
|
16
|
+
|
17
|
+
def satisfies_add_check_constraints?
|
18
|
+
satisfied? '>=6.1.0'
|
19
|
+
end
|
20
|
+
|
21
|
+
def satisfies_add_column_update_rows_backfill?
|
22
|
+
satisfies_change_column_null_requirements?
|
23
|
+
end
|
24
|
+
|
25
|
+
def satisfied?(version)
|
26
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -2,6 +2,8 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module IdempotentStatements
|
5
|
+
include SafePgMigrations::Helpers::IndexHelper
|
6
|
+
|
5
7
|
ruby2_keywords def add_index(table_name, column_name, *args)
|
6
8
|
options = args.last.is_a?(Hash) ? args.last : {}
|
7
9
|
|
@@ -80,23 +82,51 @@ module SafePgMigrations
|
|
80
82
|
end
|
81
83
|
end
|
82
84
|
|
83
|
-
|
85
|
+
def add_check_constraint(table_name, expression, **options)
|
86
|
+
constraint_definition = check_constraint_for table_name,
|
87
|
+
**check_constraint_options(table_name, expression, options)
|
88
|
+
|
89
|
+
return super if constraint_definition.nil?
|
90
|
+
|
91
|
+
SafePgMigrations.say "/!\\ Constraint '#{constraint_definition.name}' already exists. Skipping statement.", true
|
92
|
+
end
|
93
|
+
|
94
|
+
def change_column_null(table_name, column_name, null, *)
|
95
|
+
column = column_for(table_name, column_name)
|
84
96
|
|
85
|
-
|
86
|
-
|
87
|
-
|
97
|
+
return super if column.null != null
|
98
|
+
|
99
|
+
SafePgMigrations.say(
|
100
|
+
"/!\\ Column '#{table_name}.#{column.name}' is already set to 'null: #{null}'. Skipping statement.",
|
101
|
+
true
|
102
|
+
)
|
88
103
|
end
|
89
104
|
|
90
|
-
|
105
|
+
def validate_check_constraint(table_name, **options)
|
106
|
+
constraint_definition = check_constraint_for!(table_name, **options)
|
107
|
+
|
108
|
+
return super unless constraint_definition.validated?
|
91
109
|
|
92
|
-
|
93
|
-
|
94
|
-
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
110
|
+
SafePgMigrations.say "/!\\ Constraint '#{constraint_definition.name}' already validated. Skipping statement.",
|
111
|
+
true
|
112
|
+
end
|
113
|
+
|
114
|
+
def change_column_default(table_name, column_name, default_or_changes)
|
115
|
+
column = column_for(table_name, column_name)
|
116
|
+
|
117
|
+
previous_alter_statement = change_column_default_for_alter(table_name, column_name, column.default)
|
118
|
+
new_alter_statement = change_column_default_for_alter(table_name, column_name, default_or_changes)
|
119
|
+
|
120
|
+
# NOTE: PG change_column_default is already idempotent.
|
121
|
+
# We try to detect it because it still takes an ACCESS EXCLUSIVE lock
|
122
|
+
|
123
|
+
return super if new_alter_statement != previous_alter_statement
|
124
|
+
|
125
|
+
SafePgMigrations.say(
|
126
|
+
"/!\\ Column '#{table_name}.#{column.name}' is already set to 'default: #{column.default}'. " \
|
127
|
+
'Skipping statement.',
|
128
|
+
true
|
129
|
+
)
|
100
130
|
end
|
101
131
|
end
|
102
132
|
end
|
@@ -0,0 +1,52 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module StatementInsurer
|
5
|
+
module AddColumn
|
6
|
+
ruby2_keywords def add_column(table_name, column_name, type, *args)
|
7
|
+
options = args.last.is_a?(Hash) && args.last
|
8
|
+
options ||= {}
|
9
|
+
|
10
|
+
if should_keep_default_implementation?(**options)
|
11
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) { return super }
|
12
|
+
end
|
13
|
+
|
14
|
+
default = options.delete(:default)
|
15
|
+
null = options.delete(:null)
|
16
|
+
|
17
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
18
|
+
SafePgMigrations.say_method_call(:add_column, table_name, column_name, type, options)
|
19
|
+
super table_name, column_name, type, **options
|
20
|
+
end
|
21
|
+
|
22
|
+
SafePgMigrations.say_method_call(:change_column_default, table_name, column_name, default)
|
23
|
+
change_column_default(table_name, column_name, default)
|
24
|
+
|
25
|
+
SafePgMigrations.say_method_call(:backfill_column_default, table_name, column_name)
|
26
|
+
without_statement_timeout do
|
27
|
+
backfill_column_default(table_name, column_name)
|
28
|
+
end
|
29
|
+
|
30
|
+
change_column_null(table_name, column_name, null) if null == false
|
31
|
+
end
|
32
|
+
|
33
|
+
private
|
34
|
+
|
35
|
+
def should_keep_default_implementation?(default: nil, default_value_backfill: :auto, **)
|
36
|
+
default_value_backfill != :update_in_batches || !default ||
|
37
|
+
!SafePgMigrations::Helpers::SatisfiedHelper.satisfies_add_column_update_rows_backfill?
|
38
|
+
end
|
39
|
+
|
40
|
+
def backfill_column_default(table_name, column_name)
|
41
|
+
model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
|
42
|
+
quoted_column_name = quote_column_name(column_name)
|
43
|
+
|
44
|
+
SafePgMigrations::Helpers::BatchOver.new(model).each_batch do |batch|
|
45
|
+
batch
|
46
|
+
.update_all("#{quoted_column_name} = DEFAULT")
|
47
|
+
sleep SafePgMigrations.config.backfill_pause
|
48
|
+
end
|
49
|
+
end
|
50
|
+
end
|
51
|
+
end
|
52
|
+
end
|
@@ -2,18 +2,41 @@
|
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module StatementInsurer
|
5
|
-
|
5
|
+
include AddColumn
|
6
|
+
|
7
|
+
%i[change_column].each do |method|
|
6
8
|
define_method method do |*args, &block|
|
7
|
-
with_setting(:statement_timeout, SafePgMigrations.config.
|
9
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) { super(*args, &block) }
|
8
10
|
end
|
9
11
|
ruby2_keywords method
|
10
12
|
end
|
11
13
|
|
14
|
+
def validate_check_constraint(table_name, **options)
|
15
|
+
without_statement_timeout do
|
16
|
+
super
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
def add_check_constraint(table_name, expression, **options)
|
21
|
+
Helpers::SatisfiedHelper.satisfies_add_check_constraints!
|
22
|
+
return unless supports_check_constraints?
|
23
|
+
|
24
|
+
options = check_constraint_options(table_name, expression, options)
|
25
|
+
|
26
|
+
SafePgMigrations.say_method_call :add_check_constraint, table_name, expression, **options, validate: false
|
27
|
+
super table_name, expression, **options, validate: false
|
28
|
+
|
29
|
+
return unless options.fetch(:validate, true)
|
30
|
+
|
31
|
+
SafePgMigrations.say_method_call :validate_check_constraint, table_name, name: options[:name]
|
32
|
+
validate_check_constraint table_name, name: options[:name]
|
33
|
+
end
|
34
|
+
|
12
35
|
ruby2_keywords def add_foreign_key(from_table, to_table, *args)
|
13
36
|
options = args.last.is_a?(Hash) ? args.last : {}
|
14
|
-
validate_present = options.key?
|
37
|
+
validate_present = options.key?(:validate)
|
15
38
|
options[:validate] = false unless validate_present
|
16
|
-
with_setting(:statement_timeout, SafePgMigrations.config.
|
39
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
17
40
|
super(from_table, to_table, **options)
|
18
41
|
end
|
19
42
|
|
@@ -24,7 +47,7 @@ module SafePgMigrations
|
|
24
47
|
end
|
25
48
|
|
26
49
|
ruby2_keywords def create_table(*)
|
27
|
-
with_setting(:statement_timeout, SafePgMigrations.config.
|
50
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
28
51
|
super do |td|
|
29
52
|
yield td if block_given?
|
30
53
|
td.indexes.map! do |key, index_options|
|
@@ -57,6 +80,22 @@ module SafePgMigrations
|
|
57
80
|
without_timeout { super(table_name, **options) }
|
58
81
|
end
|
59
82
|
|
83
|
+
def change_column_null(table_name, column_name, null, default = nil)
|
84
|
+
if default || null || !Helpers::SatisfiedHelper.satisfies_change_column_null_requirements?
|
85
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) { return super }
|
86
|
+
end
|
87
|
+
|
88
|
+
add_check_constraint table_name, "#{column_name} IS NOT NULL"
|
89
|
+
|
90
|
+
SafePgMigrations.say_method_call :change_column_null, table_name, column_name, false
|
91
|
+
with_setting(:statement_timeout, SafePgMigrations.config.pg_statement_timeout) do
|
92
|
+
super table_name, column_name, false
|
93
|
+
end
|
94
|
+
|
95
|
+
SafePgMigrations.say_method_call :remove_check_constraint, table_name, "#{column_name} IS NOT NULL"
|
96
|
+
remove_check_constraint table_name, "#{column_name} IS NOT NULL"
|
97
|
+
end
|
98
|
+
|
60
99
|
def with_setting(key, value)
|
61
100
|
old_value = query_value("SHOW #{key}")
|
62
101
|
execute("SET #{key} TO #{quote(value)}")
|
@@ -3,7 +3,7 @@
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module StatementRetrier
|
5
5
|
RETRIABLE_SCHEMA_STATEMENTS = %i[
|
6
|
-
add_column add_foreign_key remove_foreign_key change_column_default change_column_null remove_column
|
6
|
+
add_column add_foreign_key remove_foreign_key change_column_default change_column_null remove_column drop_table
|
7
7
|
].freeze
|
8
8
|
|
9
9
|
RETRIABLE_SCHEMA_STATEMENTS.each do |method|
|
@@ -29,6 +29,14 @@ module SafePgMigrations
|
|
29
29
|
super
|
30
30
|
end
|
31
31
|
|
32
|
+
ruby2_keywords def add_check_constraint(*args)
|
33
|
+
options = args.last.is_a?(Hash) ? args.last : {}
|
34
|
+
if options[:validate] == false
|
35
|
+
UselessStatementsLogger.warn_useless '`validate: :false`', 'https://github.com/doctolib/safe-pg-migrations#safe_add_check_constraint'
|
36
|
+
end
|
37
|
+
super
|
38
|
+
end
|
39
|
+
|
32
40
|
def warn_for_index(**options)
|
33
41
|
return unless options[:algorithm] == :concurrently
|
34
42
|
|
@@ -3,14 +3,14 @@
|
|
3
3
|
module SafePgMigrations
|
4
4
|
module Polyfills
|
5
5
|
module IndexDefinitionPolyfill
|
6
|
-
include SatisfiedHelper
|
6
|
+
include Helpers::SatisfiedHelper
|
7
7
|
|
8
8
|
protected
|
9
9
|
|
10
10
|
IndexDefinition = Struct.new(:table, :name)
|
11
11
|
|
12
12
|
def index_definition(table_name, column_name, **options)
|
13
|
-
return super(table_name, column_name, **options) if satisfied? '>=6.1.0'
|
13
|
+
return super(table_name, column_name, **options) if Helpers::SatisfiedHelper.satisfied? '>=6.1.0'
|
14
14
|
|
15
15
|
index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
|
16
16
|
validate_index_length!(table_name, index_name, options.fetch(:internal, false))
|
@@ -4,16 +4,14 @@ module SafePgMigrations
|
|
4
4
|
module Polyfills
|
5
5
|
module VerboseQueryLogsPolyfill
|
6
6
|
class << self
|
7
|
-
include SatisfiedHelper
|
8
|
-
|
9
7
|
def verbose_query_logs
|
10
|
-
return ActiveRecord.verbose_query_logs if satisfied? '>=7.0.0'
|
8
|
+
return ActiveRecord.verbose_query_logs if Helpers::SatisfiedHelper.satisfied? '>=7.0.0'
|
11
9
|
|
12
10
|
ActiveRecord::Base.verbose_query_logs
|
13
11
|
end
|
14
12
|
|
15
13
|
def verbose_query_logs=(value)
|
16
|
-
if satisfied? '>=7.0.0'
|
14
|
+
if Helpers::SatisfiedHelper.satisfied? '>=7.0.0'
|
17
15
|
ActiveRecord.verbose_query_logs = value
|
18
16
|
return
|
19
17
|
end
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: safe-pg-migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthieu Prat
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2023-
|
13
|
+
date: 2023-06-29 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: activerecord
|
@@ -51,16 +51,19 @@ files:
|
|
51
51
|
- lib/safe-pg-migrations.rb
|
52
52
|
- lib/safe-pg-migrations/base.rb
|
53
53
|
- lib/safe-pg-migrations/configuration.rb
|
54
|
+
- lib/safe-pg-migrations/helpers/batch_over.rb
|
54
55
|
- lib/safe-pg-migrations/helpers/blocking_activity_formatter.rb
|
55
56
|
- lib/safe-pg-migrations/helpers/blocking_activity_selector.rb
|
57
|
+
- lib/safe-pg-migrations/helpers/index_helper.rb
|
58
|
+
- lib/safe-pg-migrations/helpers/satisfied_helper.rb
|
56
59
|
- lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
|
57
60
|
- lib/safe-pg-migrations/plugins/idempotent_statements.rb
|
58
61
|
- lib/safe-pg-migrations/plugins/statement_insurer.rb
|
62
|
+
- lib/safe-pg-migrations/plugins/statement_insurer/add_column.rb
|
59
63
|
- lib/safe-pg-migrations/plugins/statement_retrier.rb
|
60
64
|
- lib/safe-pg-migrations/plugins/useless_statements_logger.rb
|
61
65
|
- lib/safe-pg-migrations/plugins/verbose_sql_logger.rb
|
62
66
|
- lib/safe-pg-migrations/polyfills/index_definition_polyfill.rb
|
63
|
-
- lib/safe-pg-migrations/polyfills/satisfied_helper.rb
|
64
67
|
- lib/safe-pg-migrations/polyfills/verbose_query_logs_polyfill.rb
|
65
68
|
- lib/safe-pg-migrations/railtie.rb
|
66
69
|
- lib/safe-pg-migrations/version.rb
|
@@ -1,13 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
module SafePgMigrations
|
4
|
-
module Polyfills
|
5
|
-
module SatisfiedHelper
|
6
|
-
private
|
7
|
-
|
8
|
-
def satisfied?(version)
|
9
|
-
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
10
|
-
end
|
11
|
-
end
|
12
|
-
end
|
13
|
-
end
|