safe-pg-migrations 1.3.0 → 1.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: fdee6e752da3fbdec93e5fd5c72fb370497ddd66f1cc27f33609c2a31e364c53
4
- data.tar.gz: 143a5dcbf614abed765fac78b16969d900ded5ec574f11d939d802b94d7e2586
3
+ metadata.gz: 2fb891279f68c60f15f81b6e7ccbc425750443c3ee294080c2286bb059ed2e89
4
+ data.tar.gz: f4a8d59c4ee5abf004e4141db1d09d3cde33ff42f180c03526767535f20a5acd
5
5
  SHA512:
6
- metadata.gz: 41c264aa05c7b8ad5fe90f7ecdc88fa700504562e69695d5fd9f4dc9ac63315fb666818554e1387251b367aed241fccff31154589f70309905e1ed1cfae90c68
7
- data.tar.gz: 9386e6a8b0366387543998ec5cec42e719d2a4138d5eb52f13f6580d2a3d31e48bd438760b358beb1ab5eb32bd8598927d69bafd03756108a514e88eeb431022
6
+ metadata.gz: 5b8c48e973fc0296a54c2a8cceaa3b3c8c8d3380cf15a87ea5154e2ebfc28a8aa852172fe020946d8c758a4d1518bf22fe06eda08d31d318d9176df88820ac8e
7
+ data.tar.gz: e88101b365a400e21ca76cf64b1818fba36e8875958989e1864391faf476b793596bbcfbd6bed4c53db46d93a16981334f45fe83124dc04e2eb7132029a712e3
data/README.md CHANGED
@@ -2,6 +2,8 @@
2
2
 
3
3
  ActiveRecord migrations for Postgres made safe.
4
4
 
5
+ ![safe-pg-migrations](./logo.png)
6
+
5
7
  ## Requirements
6
8
 
7
9
  - Ruby 2.5+
@@ -91,7 +93,7 @@ When **Safe PG Migrations** is used, migrations are not wrapped in a transaction
91
93
  - In order to be able to retry statements that have failed because of a lock timeout, we have to be outside a transaction.
92
94
  - In order to add an index concurrently, we have to be outside a transaction.
93
95
 
94
- Note that if a migration fails, it won't be rollbacked. This can result in migrations being partially applied. In that case, they need to be manually reverted.
96
+ Note that if a migration fails, it won't be rolled back. This can result in migrations being partially applied. In that case, they need to be manually reverted.
95
97
 
96
98
  </details>
97
99
 
@@ -108,7 +110,7 @@ PG will still needs to update every row of the table, and will most likely state
108
110
 
109
111
  <blockquote>
110
112
 
111
- **Note: Pre-postgre 11**
113
+ **Note: Pre-postgres 11**
112
114
  Adding a column with a default value and a not-null constraint is [dangerous](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
113
115
 
114
116
  **Safe PG Migrations** makes it safe by:
@@ -120,7 +122,7 @@ Adding a column with a default value and a not-null constraint is [dangerous](ht
120
122
 
121
123
  Note: the addition of the not null constraint may timeout. In that case, you may want to add the not-null constraint as initially not valid and validate it in a separate statement. See [Adding a not-null constraint on Postgres with minimal locking](https://medium.com/doctolib-engineering/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
122
124
 
123
- </blockquote>
125
+ </blockquote>
124
126
 
125
127
  </details>
126
128
 
@@ -137,7 +139,7 @@ If you still get lock timeout while adding / removing indexes, it might be for o
137
139
 
138
140
  </details>
139
141
 
140
- <details><summary id="safe_add_foreign_key">safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
142
+ <details><summary id="safe_add_foreign_key">Safe <code>add_foreign_key</code> (and <code>add_reference</code>)</summary>
141
143
 
142
144
  Adding a foreign key requires a `SHARE ROW EXCLUSIVE` lock, which **prevent writing in the tables** while the migration is running.
143
145
 
@@ -150,7 +152,7 @@ Adding the constraint itself is rather fast, the major part of the time is spent
150
152
 
151
153
  <details><summary>Retry after lock timeout</summary>
152
154
 
153
- When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retryable statments](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
155
+ When a statement fails with a lock timeout, **Safe PG Migrations** retries it (5 times max) [list of retriable statements](https://github.com/doctolib/safe-pg-migrations/blob/66933256252b6bbf12e404b829a361dbba30e684/lib/safe-pg-migrations/plugins/statement_retrier.rb#L5)
154
156
  </details>
155
157
 
156
158
  <details><summary>Blocking activity logging</summary>
@@ -219,11 +221,11 @@ SafePgMigrations.config.retry_delay = 1.minute # Delay between retries for retry
219
221
  SafePgMigrations.config.max_tries = 5 # Number of retries before abortion of the migration
220
222
  ```
221
223
 
222
- ## Runnings tests
224
+ ## Running tests
223
225
 
224
226
  ```bash
225
227
  bundle
226
- psql -h localhost -U postgres -c 'CREATE DATABASE safe_pg_migrations_test'
228
+ psql -h localhost -c 'CREATE DATABASE safe_pg_migrations_test'
227
229
  rake test
228
230
  ```
229
231
 
@@ -257,3 +259,4 @@ Interesting reads:
257
259
  - [Safe Operations For High Volume PostgreSQL](https://www.braintreepayments.com/blog/safe-operations-for-high-volume-postgresql/)
258
260
  - [Rails Migrations with Zero Downtime](https://blog.codeship.com/rails-migrations-zero-downtime/)
259
261
  - [Stop worrying about PostgreSQL locks in your Rails migrations](https://medium.com/doctolib/stop-worrying-about-postgresql-locks-in-your-rails-migrations-3426027e9cc9)
262
+ - [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/paypal-tech/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
@@ -6,17 +6,19 @@ require 'safe-pg-migrations/plugins/verbose_sql_logger'
6
6
  require 'safe-pg-migrations/plugins/blocking_activity_logger'
7
7
  require 'safe-pg-migrations/plugins/statement_insurer'
8
8
  require 'safe-pg-migrations/plugins/statement_retrier'
9
- require 'safe-pg-migrations/plugins/idem_potent_statements'
9
+ require 'safe-pg-migrations/plugins/idempotent_statements'
10
10
  require 'safe-pg-migrations/plugins/useless_statements_logger'
11
+ require 'safe-pg-migrations/plugins/legacy_active_record_support'
11
12
 
12
13
  module SafePgMigrations
13
14
  # Order matters: the bottom-most plugin will have precedence
14
15
  PLUGINS = [
15
16
  BlockingActivityLogger,
16
- IdemPotentStatements,
17
+ IdempotentStatements,
17
18
  StatementRetrier,
18
19
  StatementInsurer,
19
20
  UselessStatementsLogger,
21
+ LegacyActiveRecordSupport,
20
22
  ].freeze
21
23
 
22
24
  class << self
@@ -85,7 +87,16 @@ module SafePgMigrations
85
87
  true
86
88
  end
87
89
 
88
- SAFE_METHODS = %i[execute add_column add_index add_reference add_belongs_to change_column_null].freeze
90
+ SAFE_METHODS = %i[
91
+ execute
92
+ add_column
93
+ add_index
94
+ add_reference
95
+ add_belongs_to
96
+ change_column_null
97
+ add_foreign_key
98
+ ].freeze
99
+
89
100
  SAFE_METHODS.each do |method|
90
101
  define_method method do |*args|
91
102
  return super(*args) unless respond_to?(:safety_assured)
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SafePgMigrations
4
- module BlockingActivityLogger
4
+ module BlockingActivityLogger # rubocop:disable Metrics/ModuleLength
5
5
  FILTERED_COLUMNS = %w[
6
6
  blocked_activity.xact_start
7
7
  blocked_locks.locktype
@@ -113,6 +113,7 @@ module SafePgMigrations
113
113
  end
114
114
 
115
115
  def format_start_time(start_time, reference_time = Time.now)
116
+ start_time = Time.parse(start_time) unless start_time.is_a? Time
116
117
  duration = (reference_time - start_time).round
117
118
  "transaction started #{duration} #{'second'.pluralize(duration)} ago"
118
119
  end
@@ -1,15 +1,23 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SafePgMigrations
4
- module IdemPotentStatements
4
+ module IdempotentStatements
5
5
  ruby2_keywords def add_index(table_name, column_name, *args)
6
6
  options = args.last.is_a?(Hash) ? args.last : {}
7
- index_name = options.key?(:name) ? options[:name].to_s : index_name(table_name, index_column_names(column_name))
8
- return super unless index_name_exists?(table_name, index_name)
9
7
 
10
- return if index_valid?(index_name)
8
+ index_definition, = add_index_options(table_name, column_name, **options)
11
9
 
12
- remove_index(table_name, name: index_name)
10
+ return super unless index_name_exists?(index_definition.table, index_definition.name)
11
+
12
+ if index_valid?(index_definition.name)
13
+ SafePgMigrations.say(
14
+ "/!\\ Index '#{index_definition.name}' already exists in '#{table_name}'. Skipping statement.",
15
+ true
16
+ )
17
+ return
18
+ end
19
+
20
+ remove_index(table_name, name: index_definition.name)
13
21
  super
14
22
  end
15
23
 
@@ -0,0 +1,23 @@
1
+ # frozen_string_literal: true
2
+
3
+ module SafePgMigrations
4
+ module LegacyActiveRecordSupport
5
+ ruby2_keywords def validate_foreign_key(from_table, to_table = nil, **options)
6
+ return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
7
+
8
+ super(from_table, to_table, **options)
9
+ end
10
+
11
+ ruby2_keywords def foreign_key_exists?(from_table, to_table = nil, **options)
12
+ return super(from_table, to_table || options) unless satisfied? '>=6.0.0'
13
+
14
+ super(from_table, to_table, **options)
15
+ end
16
+
17
+ private
18
+
19
+ def satisfied?(version)
20
+ Gem::Requirement.new(version).satisfied_by? Gem::Version.new(::ActiveRecord::VERSION::STRING)
21
+ end
22
+ end
23
+ end
@@ -1,7 +1,7 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SafePgMigrations
4
- module StatementInsurer # rubocop:disable Metrics/ModuleLength
4
+ module StatementInsurer
5
5
  PG_11_VERSION_NUM = 110_000
6
6
 
7
7
  %i[change_column_null change_column].each do |method|
@@ -85,20 +85,11 @@ module SafePgMigrations
85
85
  end
86
86
 
87
87
  def backfill_column_default(table_name, column_name)
88
- quoted_table_name = quote_table_name(table_name)
88
+ model = Class.new(ActiveRecord::Base) { self.table_name = table_name }
89
89
  quoted_column_name = quote_column_name(column_name)
90
- primary_key_offset = 0
91
- loop do
92
- ids = query_values <<~SQL.squish
93
- SELECT id FROM #{quoted_table_name} WHERE id > #{primary_key_offset}
94
- ORDER BY id LIMIT #{SafePgMigrations.config.batch_size}
95
- SQL
96
- break if ids.empty?
97
-
98
- primary_key_offset = ids.last
99
- execute <<~SQL.squish
100
- UPDATE #{quoted_table_name} SET #{quoted_column_name} = DEFAULT WHERE id IN (#{ids.join(',')})
101
- SQL
90
+
91
+ model.in_batches(of: SafePgMigrations.config.batch_size).each do |relation|
92
+ relation.update_all("#{quoted_column_name} = DEFAULT")
102
93
  end
103
94
  end
104
95
 
@@ -1,5 +1,5 @@
1
1
  # frozen_string_literal: true
2
2
 
3
3
  module SafePgMigrations
4
- VERSION = '1.3.0'
4
+ VERSION = '1.4.0'
5
5
  end
metadata CHANGED
@@ -1,15 +1,16 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: safe-pg-migrations
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.3.0
4
+ version: 1.4.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Matthieu Prat
8
8
  - Romain Choquet
9
+ - Thomas Hareau
9
10
  autorequire:
10
11
  bindir: bin
11
12
  cert_chain: []
12
- date: 2021-06-28 00:00:00.000000000 Z
13
+ date: 2022-02-24 00:00:00.000000000 Z
13
14
  dependencies:
14
15
  - !ruby/object:Gem::Dependency
15
16
  name: activerecord
@@ -166,7 +167,7 @@ dependencies:
166
167
  - !ruby/object:Gem::Version
167
168
  version: '0'
168
169
  description: Make your PG migrations safe.
169
- email: matthieuprat@gmail.com
170
+ email:
170
171
  executables: []
171
172
  extensions: []
172
173
  extra_rdoc_files: []
@@ -177,7 +178,8 @@ files:
177
178
  - lib/safe-pg-migrations/base.rb
178
179
  - lib/safe-pg-migrations/configuration.rb
179
180
  - lib/safe-pg-migrations/plugins/blocking_activity_logger.rb
180
- - lib/safe-pg-migrations/plugins/idem_potent_statements.rb
181
+ - lib/safe-pg-migrations/plugins/idempotent_statements.rb
182
+ - lib/safe-pg-migrations/plugins/legacy_active_record_support.rb
181
183
  - lib/safe-pg-migrations/plugins/statement_insurer.rb
182
184
  - lib/safe-pg-migrations/plugins/statement_retrier.rb
183
185
  - lib/safe-pg-migrations/plugins/useless_statements_logger.rb
@@ -187,7 +189,12 @@ files:
187
189
  homepage: https://github.com/doctolib/safe-pg-migrations
188
190
  licenses:
189
191
  - MIT
190
- metadata: {}
192
+ metadata:
193
+ bug_tracker_uri: https://github.com/doctolib/safe-pg-migrations/issues
194
+ homepage_uri: https://github.com/doctolib/safe-pg-migrations#safe-pg-migrations
195
+ mailing_list_uri: https://doctolib.engineering/engineering-news-ruby-rails-react
196
+ source_code_uri: https://github.com/doctolib/safe-pg-migrations
197
+ contributors_uri: https://github.com/doctolib/safe-pg-migrations/graphs/contributors
191
198
  post_install_message:
192
199
  rdoc_options: []
193
200
  require_paths: