safe-pg-migrations 1.3.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +10 -7
- data/lib/safe-pg-migrations/base.rb +14 -3
- data/lib/safe-pg-migrations/plugins/blocking_activity_logger.rb +2 -1
- data/lib/safe-pg-migrations/plugins/{idem_potent_statements.rb → idempotent_statements.rb} +13 -5
- data/lib/safe-pg-migrations/plugins/legacy_active_record_support.rb +23 -0
- data/lib/safe-pg-migrations/plugins/statement_insurer.rb +5 -14
- data/lib/safe-pg-migrations/version.rb +1 -1
- metadata +12 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2fb891279f68c60f15f81b6e7ccbc425750443c3ee294080c2286bb059ed2e89
|
4
|
+
data.tar.gz: f4a8d59c4ee5abf004e4141db1d09d3cde33ff42f180c03526767535f20a5acd
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5b8c48e973fc0296a54c2a8cceaa3b3c8c8d3380cf15a87ea5154e2ebfc28a8aa852172fe020946d8c758a4d1518bf22fe06eda08d31d318d9176df88820ac8e
|
7
|
+
data.tar.gz: e88101b365a400e21ca76cf64b1818fba36e8875958989e1864391faf476b793596bbcfbd6bed4c53db46d93a16981334f45fe83124dc04e2eb7132029a712e3
|
data/README.md
CHANGED
@@ -2,6 +2,8 @@
|
|
2
2
|
|
3
3
|
ActiveRecord migrations for Postgres made safe.
|
4
4
|
|
5
|
+

|
6
|
+
|
5
7
|
## Requirements
|
6
8
|
|
7
9
|
- Ruby 2.5+
|
@@ -91,7 +93,7 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
|
|
91
93
|
- In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
|
92
94
|
- In order to add an index concurrently, we have to be outside a transaction.
|
93
95
|
|
94
|
-
Note that if a migration fails, it won't be
|
96
|
+
Note that if a migration fails, it won't be rolled back. This can result in migrations being partially applied. In that case, they need to be manually reverted.
|
95
97
|
|
96
98
|
</details>
|
97
99
|
|
@@ -108,7 +110,7 @@ PG will still needs to update every row of the table, and will most likely state
|
|
108
110
|
|
109
111
|
<blockquote>
|
110
112
|
|
111
|
-
**Note: Pre-
|
113
|
+
**Note: Pre-postgres 11**
|
112
114
|
Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
113
115
|
|
114
116
|
**Safe PG Migrations** makes it safe by:
|
@@ -120,7 +122,7 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
|
|
120
122
|
|
121
123
|
Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
|
122
124
|
|
123
|
-
</blockquote>
|
125
|
+
</blockquote>
|
124
126
|
|
125
127
|
</details>
|
126
128
|
|
@@ -137,7 +139,7 @@ If you still get lock timeout while adding / removing indexes, it might be for o
|
|
137
139
|
|
138
140
|
</details>
|
139
141
|
|
140
|
-
<details><summary id="safe_add_foreign_key">
|
142
|
+
<details><summary id="safe_add_foreign_key">Safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
|
141
143
|
|
142
144
|
Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
|
143
145
|
|
@@ -150,7 +152,7 @@ Adding the constraint itself is rather fast, the major part of the time is spent
|
|
150
152
|
|
151
153
|
<details><summary>Retry after lock timeout</summary>
|
152
154
|
|
153
|
-
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of
|
155
|
+
When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
|
154
156
|
</details>
|
155
157
|
|
156
158
|
<details><summary>Blocking activity logging</summary>
|
@@ -219,11 +221,11 @@ SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retry
|
|
219
221
|
SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
|
220
222
|
```
|
221
223
|
|
222
|
-
##
|
224
|
+
## Running tests
|
223
225
|
|
224
226
|
```bash
|
225
227
|
bundle
|
226
|
-
psql -h localhost -
|
228
|
+
psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
|
227
229
|
rake test
|
228
230
|
```
|
229
231
|
|
@@ -257,3 +259,4 @@ Interesting reads:
|
|
257
259
|
- [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
|
258
260
|
- [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
|
259
261
|
- [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
|
262
|
+
- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
|
@@ -6,17 +6,19 @@ require 'safe-pg-migrations/plugins/verbose_sql_logger'
|
|
6
6
|
require 'safe-pg-migrations/plugins/blocking_activity_logger'
|
7
7
|
require 'safe-pg-migrations/plugins/statement_insurer'
|
8
8
|
require 'safe-pg-migrations/plugins/statement_retrier'
|
9
|
-
require 'safe-pg-migrations/plugins/
|
9
|
+
require 'safe-pg-migrations/plugins/idempotent_statements'
|
10
10
|
require 'safe-pg-migrations/plugins/useless_statements_logger'
|
11
|
+
require 'safe-pg-migrations/plugins/legacy_active_record_support'
|
11
12
|
|
12
13
|
module SafePgMigrations
|
13
14
|
# Order matters: the bottom-most plugin will have precedence
|
14
15
|
PLUGINS = [
|
15
16
|
BlockingActivityLogger,
|
16
|
-
|
17
|
+
IdempotentStatements,
|
17
18
|
StatementRetrier,
|
18
19
|
StatementInsurer,
|
19
20
|
UselessStatementsLogger,
|
21
|
+
LegacyActiveRecordSupport,
|
20
22
|
].freeze
|
21
23
|
|
22
24
|
class << self
|
@@ -85,7 +87,16 @@ module SafePgMigrations
|
|
85
87
|
true
|
86
88
|
end
|
87
89
|
|
88
|
-
SAFE_METHODS = %i[
|
90
|
+
SAFE_METHODS = %i[
|
91
|
+
execute
|
92
|
+
add_column
|
93
|
+
add_index
|
94
|
+
add_reference
|
95
|
+
add_belongs_to
|
96
|
+
change_column_null
|
97
|
+
add_foreign_key
|
98
|
+
].freeze
|
99
|
+
|
89
100
|
SAFE_METHODS.each do |method|
|
90
101
|
define_method method do |*args|
|
91
102
|
return super(*args) unless respond_to?(:safety_assured)
|
@@ -1,7 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module BlockingActivityLogger
|
4
|
+
module BlockingActivityLogger # rubocop:disable Metrics/ModuleLength
|
5
5
|
FILTERED_COLUMNS = %w[
|
6
6
|
blocked_activity.xact_start
|
7
7
|
blocked_locks.locktype
|
@@ -113,6 +113,7 @@ module SafePgMigrations
|
|
113
113
|
end
|
114
114
|
|
115
115
|
def format_start_time(start_time, reference_time = Time.now)
|
116
|
+
start_time = Time.parse(start_time) unless start_time.is_a? Time
|
116
117
|
duration = (reference_time - start_time).round
|
117
118
|
"transaction started #{duration} #{'second'.pluralize(duration)} ago"
|
118
119
|
end
|
@@ -1,15 +1,23 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module
|
4
|
+
module IdempotentStatements
|
5
5
|
ruby2_keywords def add_index(table_name, column_name, *args)
|
6
6
|
options = args.last.is_a?(Hash) ? args.last : {}
|
7
|
-
index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
|
8
|
-
return super unless index_name_exists?(table_name, index_name)
|
9
7
|
|
10
|
-
|
8
|
+
index_definition, = add_index_options(table_name, column_name, **options)
|
11
9
|
|
12
|
-
|
10
|
+
return super unless index_name_exists?(index_definition.table, index_definition.name)
|
11
|
+
|
12
|
+
if index_valid?(index_definition.name)
|
13
|
+
SafePgMigrations.say(
|
14
|
+
"/!\\ Index '#{index_definition.name}' already exists in '#{table_name}'. Skipping statement.",
|
15
|
+
true
|
16
|
+
)
|
17
|
+
return
|
18
|
+
end
|
19
|
+
|
20
|
+
remove_index(table_name, name: index_definition.name)
|
13
21
|
super
|
14
22
|
end
|
15
23
|
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module SafePgMigrations
|
4
|
+
module LegacyActiveRecordSupport
|
5
|
+
ruby2_keywords def validate_foreign_key(from_table, to_table = nil, **options)
|
6
|
+
return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
|
7
|
+
|
8
|
+
super(from_table, to_table, **options)
|
9
|
+
end
|
10
|
+
|
11
|
+
ruby2_keywords def foreign_key_exists?(from_table, to_table = nil, **options)
|
12
|
+
return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
|
13
|
+
|
14
|
+
super(from_table, to_table, **options)
|
15
|
+
end
|
16
|
+
|
17
|
+
private
|
18
|
+
|
19
|
+
def satisfied?(version)
|
20
|
+
Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
|
21
|
+
end
|
22
|
+
end
|
23
|
+
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
2
|
|
3
3
|
module SafePgMigrations
|
4
|
-
module StatementInsurer
|
4
|
+
module StatementInsurer
|
5
5
|
PG_11_VERSION_NUM = 110_000
|
6
6
|
|
7
7
|
%i[change_column_null change_column].each do |method|
|
@@ -85,20 +85,11 @@ module SafePgMigrations
|
|
85
85
|
end
|
86
86
|
|
87
87
|
def backfill_column_default(table_name, column_name)
|
88
|
-
|
88
|
+
model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
|
89
89
|
quoted_column_name = quote_column_name(column_name)
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
SELECT id FROM #{quoted_table_name} WHERE id > #{primary_key_offset}
|
94
|
-
ORDER BY id LIMIT #{SafePgMigrations.config.batch_size}
|
95
|
-
SQL
|
96
|
-
break if ids.empty?
|
97
|
-
|
98
|
-
primary_key_offset = ids.last
|
99
|
-
execute <<~SQL.squish
|
100
|
-
UPDATE #{quoted_table_name} SET #{quoted_column_name} = DEFAULT WHERE id IN (#{ids.join(',')})
|
101
|
-
SQL
|
90
|
+
|
91
|
+
model.in_batches(of: SafePgMigrations.config.batch_size).each do |relation|
|
92
|
+
relation.update_all("#{quoted_column_name} = DEFAULT")
|
102
93
|
end
|
103
94
|
end
|
104
95
|
|
metadata
CHANGED
@@ -1,15 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: safe-pg-migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Matthieu Prat
|
8
8
|
- Romain Choquet
|
9
|
+
- Thomas Hareau
|
9
10
|
autorequire:
|
10
11
|
bindir: bin
|
11
12
|
cert_chain: []
|
12
|
-
date:
|
13
|
+
date: 2022-02-24 00:00:00.000000000 Z
|
13
14
|
dependencies:
|
14
15
|
- !ruby/object:Gem::Dependency
|
15
16
|
name: activerecord
|
@@ -166,7 +167,7 @@ dependencies:
|
|
166
167
|
- !ruby/object:Gem::Version
|
167
168
|
version: '0'
|
168
169
|
description: Make your PG migrations safe.
|
169
|
-
email:
|
170
|
+
email:
|
170
171
|
executables: []
|
171
172
|
extensions: []
|
172
173
|
extra_rdoc_files: []
|
@@ -177,7 +178,8 @@ files:
|
|
177
178
|
- lib/safe-pg-migrations/base.rb
|
178
179
|
- lib/safe-pg-migrations/configuration.rb
|
179
180
|
- lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
|
180
|
-
- lib/safe-pg-migrations/plugins/
|
181
|
+
- lib/safe-pg-migrations/plugins/idempotent_statements.rb
|
182
|
+
- lib/safe-pg-migrations/plugins/legacy_active_record_support.rb
|
181
183
|
- lib/safe-pg-migrations/plugins/statement_insurer.rb
|
182
184
|
- lib/safe-pg-migrations/plugins/statement_retrier.rb
|
183
185
|
- lib/safe-pg-migrations/plugins/useless_statements_logger.rb
|
@@ -187,7 +189,12 @@ files:
|
|
187
189
|
homepage: https://github.com/doctolib/safe-pg-migrations
|
188
190
|
licenses:
|
189
191
|
- MIT
|
190
|
-
metadata:
|
192
|
+
metadata:
|
193
|
+
bug_tracker_uri: https://github.com/doctolib/safe-pg-migrations/issues
|
194
|
+
homepage_uri: https://github.com/doctolib/safe-pg-migrations#safe-pg-migrations
|
195
|
+
mailing_list_uri: https://doctolib.engineering/engineering-news-ruby-rails-react
|
196
|
+
source_code_uri: https://github.com/doctolib/safe-pg-migrations
|
197
|
+
contributors_uri: https://github.com/doctolib/safe-pg-migrations/graphs/contributors
|
191
198
|
post_install_message:
|
192
199
|
rdoc_options: []
|
193
200
|
require_paths:
|