strong_migrations 0.6.6 → 0.7.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +27 -0
- data/LICENSE.txt +1 -1
- data/README.md +180 -133
- data/lib/generators/strong_migrations/install_generator.rb +11 -0
- data/lib/generators/strong_migrations/templates/initializer.rb.tt +7 -2
- data/lib/strong_migrations.rb +31 -51
- data/lib/strong_migrations/checker.rb +105 -35
- data/lib/strong_migrations/railtie.rb +0 -4
- data/lib/strong_migrations/version.rb +1 -1
- data/lib/tasks/strong_migrations.rake +0 -6
- metadata +7 -8
- data/lib/strong_migrations/migration_helpers.rb +0 -117
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f3410e3379861436cbec2b46796069db388fffea222fa50ff46b054a153fbdaf
|
4
|
+
data.tar.gz: f328bee4412388114e888557d9b691ed3dad10bd0a5bf1ffb93c09f0b48d6696
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7e588f76685aa486f38949629b2f0aebe5f1acf92a5b53a29b4e4e1f5f334439eec3ef915240736caf55cff1f21197369305ec073c6adbbedaf6477aabb282cb
|
7
|
+
data.tar.gz: 3e350c8a68ec076295d36aac330ff9eae5ad75329b9775fb8839319f7fbc99c62b8ea58ca5fc57ec36c750a510f9ae743a76fd6fedc7fd755d1e07b7ab7230b5
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,30 @@
|
|
1
|
+
## 0.7.2 (2020-10-25)
|
2
|
+
|
3
|
+
- Added support for float timeouts
|
4
|
+
|
5
|
+
## 0.7.1 (2020-07-27)
|
6
|
+
|
7
|
+
- Added `target_version` option to replace database-specific options
|
8
|
+
|
9
|
+
## 0.7.0 (2020-07-22)
|
10
|
+
|
11
|
+
- Added `check_down` option
|
12
|
+
- Added check for `change_column` with `null: false`
|
13
|
+
- Added check for `validate_foreign_key`
|
14
|
+
- Improved error messages
|
15
|
+
- Made auto analyze less verbose in Postgres
|
16
|
+
- Decreasing the length limit of a `varchar` column or adding a limit is not safe in Postgres
|
17
|
+
- Removed safety checks for `db` rake tasks (Rails 5+ handles this)
|
18
|
+
|
19
|
+
## 0.6.8 (2020-05-13)
|
20
|
+
|
21
|
+
- `change_column_null` on a column with a `NOT NULL` constraint is safe in Postgres 12+
|
22
|
+
|
23
|
+
## 0.6.7 (2020-05-13)
|
24
|
+
|
25
|
+
- Improved comments in initializer
|
26
|
+
- Fixed string timeouts for Postgres
|
27
|
+
|
1
28
|
## 0.6.6 (2020-05-08)
|
2
29
|
|
3
30
|
- Added warnings for missing and long lock timeouts
|
data/LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -21,9 +21,40 @@ gem 'strong_migrations'
|
|
21
21
|
And run:
|
22
22
|
|
23
23
|
```sh
|
24
|
+
bundle install
|
24
25
|
rails generate strong_migrations:install
|
25
26
|
```
|
26
27
|
|
28
|
+
Strong Migrations sets a long statement timeout for migrations so you can set a [short statement timeout](#app-timeouts) for your application.
|
29
|
+
|
30
|
+
## How It Works
|
31
|
+
|
32
|
+
When you run a migration that’s potentially dangerous, you’ll see an error message like:
|
33
|
+
|
34
|
+
```txt
|
35
|
+
=== Dangerous operation detected #strong_migrations ===
|
36
|
+
|
37
|
+
Active Record caches attributes, which causes problems
|
38
|
+
when removing columns. Be sure to ignore the column:
|
39
|
+
|
40
|
+
class User < ApplicationRecord
|
41
|
+
self.ignored_columns = ["name"]
|
42
|
+
end
|
43
|
+
|
44
|
+
Deploy the code, then wrap this step in a safety_assured { ... } block.
|
45
|
+
|
46
|
+
class RemoveColumn < ActiveRecord::Migration[6.0]
|
47
|
+
def change
|
48
|
+
safety_assured { remove_column :users, :name }
|
49
|
+
end
|
50
|
+
end
|
51
|
+
```
|
52
|
+
|
53
|
+
An operation is classified as dangerous if it either:
|
54
|
+
|
55
|
+
- Blocks reads or writes for more than a few seconds (after a lock is acquired)
|
56
|
+
- Has a good chance of causing application errors
|
57
|
+
|
27
58
|
## Checks
|
28
59
|
|
29
60
|
Potentially dangerous operations:
|
@@ -31,21 +62,19 @@ Potentially dangerous operations:
|
|
31
62
|
- [removing a column](#removing-a-column)
|
32
63
|
- [adding a column with a default value](#adding-a-column-with-a-default-value)
|
33
64
|
- [backfilling data](#backfilling-data)
|
34
|
-
- [changing the type of a column](#
|
35
|
-
- [renaming a column](#renaming-
|
65
|
+
- [changing the type of a column](#changing-the-type-of-a-column)
|
66
|
+
- [renaming a column](#renaming-a-column)
|
36
67
|
- [renaming a table](#renaming-a-table)
|
37
68
|
- [creating a table with the force option](#creating-a-table-with-the-force-option)
|
38
|
-
- [
|
69
|
+
- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
|
39
70
|
- [executing SQL directly](#executing-SQL-directly)
|
40
71
|
|
41
72
|
Postgres-specific checks:
|
42
73
|
|
43
|
-
- [adding an index non-concurrently](#adding-an-index)
|
44
|
-
- [removing an index non-concurrently](#removing-an-index)
|
74
|
+
- [adding an index non-concurrently](#adding-an-index-non-concurrently)
|
45
75
|
- [adding a reference](#adding-a-reference)
|
46
76
|
- [adding a foreign key](#adding-a-foreign-key)
|
47
77
|
- [adding a json column](#adding-a-json-column)
|
48
|
-
- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
|
49
78
|
|
50
79
|
Best practices:
|
51
80
|
|
@@ -57,7 +86,7 @@ You can also add [custom checks](#custom-checks) or [disable specific checks](#d
|
|
57
86
|
|
58
87
|
#### Bad
|
59
88
|
|
60
|
-
|
89
|
+
Active Record caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
|
61
90
|
|
62
91
|
```ruby
|
63
92
|
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[6.0]
|
@@ -69,7 +98,7 @@ end
|
|
69
98
|
|
70
99
|
#### Good
|
71
100
|
|
72
|
-
1. Tell
|
101
|
+
1. Tell Active Record to ignore the column from its cache
|
73
102
|
|
74
103
|
```ruby
|
75
104
|
class User < ApplicationRecord
|
@@ -92,11 +121,9 @@ end
|
|
92
121
|
|
93
122
|
### Adding a column with a default value
|
94
123
|
|
95
|
-
Note: This operation is safe in Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+.
|
96
|
-
|
97
124
|
#### Bad
|
98
125
|
|
99
|
-
|
126
|
+
In earlier versions of Postgres, MySQL, and MariaDB, adding a column with a default value to an existing table causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
100
127
|
|
101
128
|
```ruby
|
102
129
|
class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
|
@@ -106,6 +133,8 @@ class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
|
|
106
133
|
end
|
107
134
|
```
|
108
135
|
|
136
|
+
In Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+, this no longer requires a table rewrite and is safe.
|
137
|
+
|
109
138
|
#### Good
|
110
139
|
|
111
140
|
Instead, add the column without a default value, then change the default.
|
@@ -129,7 +158,7 @@ See the next section for how to backfill.
|
|
129
158
|
|
130
159
|
#### Bad
|
131
160
|
|
132
|
-
|
161
|
+
Active Record creates a transaction around each migration, and backfilling in the same transaction that alters a table keeps the table locked for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
133
162
|
|
134
163
|
```ruby
|
135
164
|
class AddSomeColumnToUsers < ActiveRecord::Migration[6.0]
|
@@ -159,19 +188,11 @@ class BackfillSomeColumn < ActiveRecord::Migration[6.0]
|
|
159
188
|
end
|
160
189
|
```
|
161
190
|
|
162
|
-
###
|
191
|
+
### Changing the type of a column
|
163
192
|
|
164
193
|
#### Bad
|
165
194
|
|
166
|
-
|
167
|
-
class RenameSomeColumn < ActiveRecord::Migration[6.0]
|
168
|
-
def change
|
169
|
-
rename_column :users, :some_column, :new_name
|
170
|
-
end
|
171
|
-
end
|
172
|
-
```
|
173
|
-
|
174
|
-
or
|
195
|
+
Changing the type of a column causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
175
196
|
|
176
197
|
```ruby
|
177
198
|
class ChangeSomeColumnType < ActiveRecord::Migration[6.0]
|
@@ -181,17 +202,44 @@ class ChangeSomeColumnType < ActiveRecord::Migration[6.0]
|
|
181
202
|
end
|
182
203
|
```
|
183
204
|
|
184
|
-
A few changes are safe in Postgres:
|
205
|
+
A few changes don’t require a table rewrite (and are safe) in Postgres:
|
185
206
|
|
186
|
-
-
|
207
|
+
- Increasing the length limit of a `varchar` column (or removing the limit)
|
208
|
+
- Changing a `varchar` column to a `text` column
|
209
|
+
- Changing a `text` column to a `varchar` column with no length limit
|
187
210
|
- Increasing the precision of a `decimal` or `numeric` column
|
188
211
|
- Making a `decimal` or `numeric` column unconstrained
|
189
212
|
- Changing between `timestamp` and `timestamptz` columns when session time zone is UTC in Postgres 12+
|
190
213
|
|
191
214
|
And a few in MySQL and MariaDB:
|
192
215
|
|
193
|
-
- Increasing the length of a `varchar` column from under 255 up to 255
|
194
|
-
- Increasing the length of a `varchar` column over 255
|
216
|
+
- Increasing the length limit of a `varchar` column from under 255 up to 255
|
217
|
+
- Increasing the length limit of a `varchar` column from over 255 to the max limit
|
218
|
+
|
219
|
+
#### Good
|
220
|
+
|
221
|
+
A safer approach is to:
|
222
|
+
|
223
|
+
1. Create a new column
|
224
|
+
2. Write to both columns
|
225
|
+
3. Backfill data from the old column to the new column
|
226
|
+
4. Move reads from the old column to the new column
|
227
|
+
5. Stop writing to the old column
|
228
|
+
6. Drop the old column
|
229
|
+
|
230
|
+
### Renaming a column
|
231
|
+
|
232
|
+
#### Bad
|
233
|
+
|
234
|
+
Renaming a column that’s in use will cause errors in your application.
|
235
|
+
|
236
|
+
```ruby
|
237
|
+
class RenameSomeColumn < ActiveRecord::Migration[6.0]
|
238
|
+
def change
|
239
|
+
rename_column :users, :some_column, :new_name
|
240
|
+
end
|
241
|
+
end
|
242
|
+
```
|
195
243
|
|
196
244
|
#### Good
|
197
245
|
|
@@ -208,6 +256,8 @@ A safer approach is to:
|
|
208
256
|
|
209
257
|
#### Bad
|
210
258
|
|
259
|
+
Renaming a table that’s in use will cause errors in your application.
|
260
|
+
|
211
261
|
```ruby
|
212
262
|
class RenameUsersToCustomers < ActiveRecord::Migration[6.0]
|
213
263
|
def change
|
@@ -257,33 +307,57 @@ class CreateUsers < ActiveRecord::Migration[6.0]
|
|
257
307
|
end
|
258
308
|
```
|
259
309
|
|
260
|
-
|
310
|
+
If you intend to drop an existing table, run `drop_table` first.
|
311
|
+
|
312
|
+
### Setting NOT NULL on an existing column
|
261
313
|
|
262
314
|
#### Bad
|
263
315
|
|
264
|
-
|
316
|
+
Setting `NOT NULL` on an existing column blocks reads and writes while every row is checked.
|
265
317
|
|
266
318
|
```ruby
|
267
|
-
class
|
319
|
+
class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
268
320
|
def change
|
269
|
-
change_column_null :users, :some_column, false
|
321
|
+
change_column_null :users, :some_column, false
|
270
322
|
end
|
271
323
|
end
|
272
324
|
```
|
273
325
|
|
274
|
-
#### Good
|
326
|
+
#### Good - Postgres
|
327
|
+
|
328
|
+
Instead, add a check constraint:
|
329
|
+
|
330
|
+
```ruby
|
331
|
+
class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
332
|
+
def change
|
333
|
+
safety_assured do
|
334
|
+
execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
|
335
|
+
end
|
336
|
+
end
|
337
|
+
end
|
338
|
+
```
|
275
339
|
|
276
|
-
|
340
|
+
Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column, but it won’t show up in `schema.rb`. In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
|
277
341
|
|
278
342
|
```ruby
|
279
|
-
class
|
343
|
+
class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
280
344
|
def change
|
345
|
+
safety_assured do
|
346
|
+
execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
|
347
|
+
end
|
348
|
+
|
349
|
+
# in Postgres 12+, you can then safely set NOT NULL on the column
|
281
350
|
change_column_null :users, :some_column, false
|
351
|
+
safety_assured do
|
352
|
+
execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
|
353
|
+
end
|
282
354
|
end
|
283
355
|
end
|
284
356
|
```
|
285
357
|
|
286
|
-
|
358
|
+
#### Good - MySQL and MariaDB
|
359
|
+
|
360
|
+
[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this.
|
287
361
|
|
288
362
|
### Executing SQL directly
|
289
363
|
|
@@ -297,11 +371,11 @@ class ExecuteSQL < ActiveRecord::Migration[6.0]
|
|
297
371
|
end
|
298
372
|
```
|
299
373
|
|
300
|
-
### Adding an index
|
374
|
+
### Adding an index non-concurrently
|
301
375
|
|
302
376
|
#### Bad
|
303
377
|
|
304
|
-
In Postgres, adding an index non-concurrently
|
378
|
+
In Postgres, adding an index non-concurrently blocks writes.
|
305
379
|
|
306
380
|
```ruby
|
307
381
|
class AddSomeIndexToUsers < ActiveRecord::Migration[6.0]
|
@@ -333,41 +407,11 @@ With [gindex](https://github.com/ankane/gindex), you can generate an index migra
|
|
333
407
|
rails g index table column
|
334
408
|
```
|
335
409
|
|
336
|
-
### Removing an index
|
337
|
-
|
338
|
-
Note: This check is [opt-in](#opt-in-checks).
|
339
|
-
|
340
|
-
#### Bad
|
341
|
-
|
342
|
-
In Postgres, removing an index non-concurrently locks the table for a brief period.
|
343
|
-
|
344
|
-
```ruby
|
345
|
-
class RemoveSomeIndexFromUsers < ActiveRecord::Migration[6.0]
|
346
|
-
def change
|
347
|
-
remove_index :users, :some_column
|
348
|
-
end
|
349
|
-
end
|
350
|
-
```
|
351
|
-
|
352
|
-
#### Good
|
353
|
-
|
354
|
-
Remove indexes concurrently.
|
355
|
-
|
356
|
-
```ruby
|
357
|
-
class RemoveSomeIndexFromUsers < ActiveRecord::Migration[6.0]
|
358
|
-
disable_ddl_transaction!
|
359
|
-
|
360
|
-
def change
|
361
|
-
remove_index :users, column: :some_column, algorithm: :concurrently
|
362
|
-
end
|
363
|
-
end
|
364
|
-
```
|
365
|
-
|
366
410
|
### Adding a reference
|
367
411
|
|
368
412
|
#### Bad
|
369
413
|
|
370
|
-
Rails adds an index non-concurrently to references by default, which
|
414
|
+
Rails adds an index non-concurrently to references by default, which blocks writes in Postgres.
|
371
415
|
|
372
416
|
```ruby
|
373
417
|
class AddReferenceToUsers < ActiveRecord::Migration[6.0]
|
@@ -395,7 +439,7 @@ end
|
|
395
439
|
|
396
440
|
#### Bad
|
397
441
|
|
398
|
-
In Postgres,
|
442
|
+
In Postgres, adding a foreign key blocks writes on both tables.
|
399
443
|
|
400
444
|
```ruby
|
401
445
|
class AddForeignKeyOnUsers < ActiveRecord::Migration[6.0]
|
@@ -417,7 +461,7 @@ end
|
|
417
461
|
|
418
462
|
#### Good
|
419
463
|
|
420
|
-
|
464
|
+
Add the foreign key without validating existing rows, then validate them in a separate migration.
|
421
465
|
|
422
466
|
For Rails 5.2+, use:
|
423
467
|
|
@@ -429,7 +473,7 @@ class AddForeignKeyOnUsers < ActiveRecord::Migration[6.0]
|
|
429
473
|
end
|
430
474
|
```
|
431
475
|
|
432
|
-
Then
|
476
|
+
Then:
|
433
477
|
|
434
478
|
```ruby
|
435
479
|
class ValidateForeignKeyOnUsers < ActiveRecord::Migration[6.0]
|
@@ -451,7 +495,7 @@ class AddForeignKeyOnUsers < ActiveRecord::Migration[5.1]
|
|
451
495
|
end
|
452
496
|
```
|
453
497
|
|
454
|
-
Then
|
498
|
+
Then:
|
455
499
|
|
456
500
|
```ruby
|
457
501
|
class ValidateForeignKeyOnUsers < ActiveRecord::Migration[5.1]
|
@@ -467,7 +511,7 @@ end
|
|
467
511
|
|
468
512
|
#### Bad
|
469
513
|
|
470
|
-
In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries.
|
514
|
+
In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries in your application.
|
471
515
|
|
472
516
|
```ruby
|
473
517
|
class AddPropertiesToUsers < ActiveRecord::Migration[6.0]
|
@@ -489,48 +533,6 @@ class AddPropertiesToUsers < ActiveRecord::Migration[6.0]
|
|
489
533
|
end
|
490
534
|
```
|
491
535
|
|
492
|
-
### Setting NOT NULL on an existing column
|
493
|
-
|
494
|
-
#### Bad
|
495
|
-
|
496
|
-
In Postgres, setting `NOT NULL` on an existing column requires an `AccessExclusiveLock`, which is expensive on large tables.
|
497
|
-
|
498
|
-
```ruby
|
499
|
-
class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
500
|
-
def change
|
501
|
-
change_column_null :users, :some_column, false
|
502
|
-
end
|
503
|
-
end
|
504
|
-
```
|
505
|
-
|
506
|
-
#### Good
|
507
|
-
|
508
|
-
Instead, add a constraint:
|
509
|
-
|
510
|
-
```ruby
|
511
|
-
class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
512
|
-
def change
|
513
|
-
safety_assured do
|
514
|
-
execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
|
515
|
-
end
|
516
|
-
end
|
517
|
-
end
|
518
|
-
```
|
519
|
-
|
520
|
-
Then validate it in a separate migration.
|
521
|
-
|
522
|
-
```ruby
|
523
|
-
class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
524
|
-
def change
|
525
|
-
safety_assured do
|
526
|
-
execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
|
527
|
-
end
|
528
|
-
end
|
529
|
-
end
|
530
|
-
```
|
531
|
-
|
532
|
-
Note: This is not 100% the same as `NOT NULL` column constraint. Here’s a [good explanation](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
|
533
|
-
|
534
536
|
### Keeping non-unique indexes to three columns or less
|
535
537
|
|
536
538
|
#### Bad
|
@@ -591,16 +593,12 @@ Note: Since `remove_column` always requires a `safety_assured` block, it’s not
|
|
591
593
|
|
592
594
|
## Opt-in Checks
|
593
595
|
|
594
|
-
|
596
|
+
### Removing an index non-concurrently
|
595
597
|
|
596
|
-
|
597
|
-
StrongMigrations.enable_check(:remove_index)
|
598
|
-
```
|
599
|
-
|
600
|
-
To start a check only after a specific migration, use:
|
598
|
+
Postgres supports removing indexes concurrently, but removing them non-concurrently shouldn’t be an issue for most applications. You can enable this check with:
|
601
599
|
|
602
600
|
```ruby
|
603
|
-
StrongMigrations.enable_check(:remove_index
|
601
|
+
StrongMigrations.enable_check(:remove_index)
|
604
602
|
```
|
605
603
|
|
606
604
|
## Disable Checks
|
@@ -613,6 +611,14 @@ StrongMigrations.disable_check(:add_index)
|
|
613
611
|
|
614
612
|
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
|
615
613
|
|
614
|
+
## Down Migrations / Rollbacks
|
615
|
+
|
616
|
+
By default, checks are disabled when migrating down. Enable them with:
|
617
|
+
|
618
|
+
```ruby
|
619
|
+
StrongMigrations.check_down = true
|
620
|
+
```
|
621
|
+
|
616
622
|
## Custom Messages
|
617
623
|
|
618
624
|
To customize specific messages, create an initializer with:
|
@@ -623,7 +629,7 @@ StrongMigrations.error_messages[:add_column_default] = "Your custom instructions
|
|
623
629
|
|
624
630
|
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
|
625
631
|
|
626
|
-
## Timeouts
|
632
|
+
## Migration Timeouts
|
627
633
|
|
628
634
|
It’s extremely important to set a short lock timeout for migrations. This way, if a migration can’t acquire a lock in a timely manner, other statements won’t be stuck behind it. We also recommend setting a long statement timeout so migrations can run for a while.
|
629
635
|
|
@@ -643,6 +649,49 @@ ALTER ROLE myuser SET statement_timeout = '1h';
|
|
643
649
|
|
644
650
|
Note: If you use PgBouncer in transaction mode, you must set timeouts on the database user.
|
645
651
|
|
652
|
+
## App Timeouts
|
653
|
+
|
654
|
+
We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.
|
655
|
+
|
656
|
+
For Postgres:
|
657
|
+
|
658
|
+
```yml
|
659
|
+
production:
|
660
|
+
connect_timeout: 5
|
661
|
+
variables:
|
662
|
+
statement_timeout: 15s
|
663
|
+
lock_timeout: 10s
|
664
|
+
```
|
665
|
+
|
666
|
+
Note: If you use PgBouncer in transaction mode, you must set the statement and lock timeouts on the database user as shown above.
|
667
|
+
|
668
|
+
For MySQL:
|
669
|
+
|
670
|
+
```yml
|
671
|
+
production:
|
672
|
+
connect_timeout: 5
|
673
|
+
read_timeout: 5
|
674
|
+
write_timeout: 5
|
675
|
+
variables:
|
676
|
+
max_execution_time: 15000 # ms
|
677
|
+
lock_wait_timeout: 10 # sec
|
678
|
+
|
679
|
+
```
|
680
|
+
|
681
|
+
For MariaDB:
|
682
|
+
|
683
|
+
```yml
|
684
|
+
production:
|
685
|
+
connect_timeout: 5
|
686
|
+
read_timeout: 5
|
687
|
+
write_timeout: 5
|
688
|
+
variables:
|
689
|
+
max_statement_time: 15 # sec
|
690
|
+
lock_wait_timeout: 10 # sec
|
691
|
+
```
|
692
|
+
|
693
|
+
For HTTP connections, Redis, and other services, check out [this guide](https://github.com/ankane/the-ultimate-guide-to-ruby-timeouts).
|
694
|
+
|
646
695
|
## Existing Migrations
|
647
696
|
|
648
697
|
To mark migrations as safe that were created before installing this gem, create an initializer with:
|
@@ -658,11 +707,11 @@ Use the version from your latest migration.
|
|
658
707
|
If your development database version is different from production, you can specify the production version so the right checks run in development.
|
659
708
|
|
660
709
|
```ruby
|
661
|
-
StrongMigrations.
|
662
|
-
StrongMigrations.target_mysql_version = "8.0.12"
|
663
|
-
StrongMigrations.target_mariadb_version = "10.3.2"
|
710
|
+
StrongMigrations.target_version = 10 # or "8.0.12", "10.3.2", etc
|
664
711
|
```
|
665
712
|
|
713
|
+
The major version works well for Postgres, while the full version is recommended for MySQL and MariaDB.
|
714
|
+
|
666
715
|
For safety, this option only affects development and test environments. In other environments, the actual server version is always used.
|
667
716
|
|
668
717
|
## Analyze Tables
|
@@ -690,22 +739,20 @@ Columns can flip order in `db/schema.rb` when you have multiple developers. One
|
|
690
739
|
task "db:schema:dump": "strong_migrations:alphabetize_columns"
|
691
740
|
```
|
692
741
|
|
693
|
-
## Dangerous Tasks
|
694
|
-
|
695
|
-
For safety, dangerous database tasks are disabled in production - `db:drop`, `db:reset`, `db:schema:load`, and `db:structure:load`. To get around this, use:
|
696
|
-
|
697
|
-
```sh
|
698
|
-
SAFETY_ASSURED=1 rails db:drop
|
699
|
-
```
|
700
|
-
|
701
742
|
## Permissions
|
702
743
|
|
703
744
|
We recommend using a [separate database user](https://ankane.org/postgres-users) for migrations when possible so you don’t need to grant your app user permission to alter tables.
|
704
745
|
|
746
|
+
## Smaller Projects
|
747
|
+
|
748
|
+
You probably don’t need this gem for smaller projects, as operations that are unsafe at scale can be perfectly safe on smaller, low-traffic tables.
|
749
|
+
|
705
750
|
## Additional Reading
|
706
751
|
|
707
752
|
- [Rails Migrations with No Downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
|
708
753
|
- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/braintree-product-technology/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
|
754
|
+
- [An Overview of DDL Algorithms in MySQL](https://mydbops.wordpress.com/2020/03/04/an-overview-of-ddl-algorithms-in-mysql-covers-mysql-8/)
|
755
|
+
- [MariaDB InnoDB Online DDL Overview](https://mariadb.com/kb/en/innodb-online-ddl-overview/)
|
709
756
|
|
710
757
|
## Credits
|
711
758
|
|
@@ -12,6 +12,17 @@ module StrongMigrations
|
|
12
12
|
def start_after
|
13
13
|
Time.now.utc.strftime("%Y%m%d%H%M%S")
|
14
14
|
end
|
15
|
+
|
16
|
+
def target_version
|
17
|
+
case ActiveRecord::Base.connection_config[:adapter].to_s
|
18
|
+
when /mysql/
|
19
|
+
# could try to connect to database and check for MariaDB
|
20
|
+
# but this should be fine
|
21
|
+
'"8.0.12"'
|
22
|
+
else
|
23
|
+
"10"
|
24
|
+
end
|
25
|
+
end
|
15
26
|
end
|
16
27
|
end
|
17
28
|
end
|
@@ -1,14 +1,19 @@
|
|
1
1
|
# Mark existing migrations as safe
|
2
2
|
StrongMigrations.start_after = <%= start_after %>
|
3
3
|
|
4
|
-
# Set timeouts
|
4
|
+
# Set timeouts for migrations
|
5
5
|
# If you use PgBouncer in transaction mode, delete these lines and set timeouts on the database user
|
6
6
|
StrongMigrations.lock_timeout = 10.seconds
|
7
7
|
StrongMigrations.statement_timeout = 1.hour
|
8
8
|
|
9
|
-
# Analyze tables
|
9
|
+
# Analyze tables after indexes are added
|
10
|
+
# Outdated statistics can sometimes hurt performance
|
10
11
|
StrongMigrations.auto_analyze = true
|
11
12
|
|
13
|
+
# Set the version of the production database
|
14
|
+
# so the right checks are run in development
|
15
|
+
# StrongMigrations.target_version = <%= target_version %>
|
16
|
+
|
12
17
|
# Add custom checks
|
13
18
|
# StrongMigrations.add_check do |method, args|
|
14
19
|
# if method == :add_index && args[0].to_s == "users"
|
data/lib/strong_migrations.rb
CHANGED
@@ -5,7 +5,6 @@ require "active_support"
|
|
5
5
|
require "strong_migrations/checker"
|
6
6
|
require "strong_migrations/database_tasks"
|
7
7
|
require "strong_migrations/migration"
|
8
|
-
require "strong_migrations/migration_helpers"
|
9
8
|
require "strong_migrations/version"
|
10
9
|
|
11
10
|
# integrations
|
@@ -18,7 +17,7 @@ module StrongMigrations
|
|
18
17
|
class << self
|
19
18
|
attr_accessor :auto_analyze, :start_after, :checks, :error_messages,
|
20
19
|
:target_postgresql_version, :target_mysql_version, :target_mariadb_version,
|
21
|
-
:enabled_checks, :lock_timeout, :statement_timeout, :
|
20
|
+
:enabled_checks, :lock_timeout, :statement_timeout, :check_down, :target_version
|
22
21
|
attr_writer :lock_timeout_limit
|
23
22
|
end
|
24
23
|
self.auto_analyze = false
|
@@ -26,7 +25,7 @@ module StrongMigrations
|
|
26
25
|
self.checks = []
|
27
26
|
self.error_messages = {
|
28
27
|
add_column_default:
|
29
|
-
"Adding a column with a non-null default
|
28
|
+
"Adding a column with a non-null default blocks %{rewrite_blocks} while the entire table is rewritten.
|
30
29
|
Instead, add the column without a default value, then change the default.
|
31
30
|
|
32
31
|
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
@@ -51,12 +50,18 @@ class Backfill%{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
51
50
|
end",
|
52
51
|
|
53
52
|
add_column_json:
|
54
|
-
"There's no equality operator for the json column type, which can
|
55
|
-
|
53
|
+
"There's no equality operator for the json column type, which can cause errors for
|
54
|
+
existing SELECT DISTINCT queries in your application. Use jsonb instead.
|
55
|
+
|
56
|
+
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
57
|
+
def change
|
58
|
+
%{command}
|
59
|
+
end
|
60
|
+
end",
|
56
61
|
|
57
62
|
change_column:
|
58
|
-
"Changing the type of an existing column
|
59
|
-
|
63
|
+
"Changing the type of an existing column blocks %{rewrite_blocks}
|
64
|
+
while the entire table is rewritten. A safer approach is to:
|
60
65
|
|
61
66
|
1. Create a new column
|
62
67
|
2. Write to both columns
|
@@ -65,7 +70,10 @@ table and indexes to be rewritten. A safer approach is to:
|
|
65
70
|
5. Stop writing to the old column
|
66
71
|
6. Drop the old column",
|
67
72
|
|
68
|
-
|
73
|
+
change_column_with_not_null:
|
74
|
+
"Changing the type is safe, but setting NOT NULL is not.",
|
75
|
+
|
76
|
+
remove_column: "Active Record caches attributes, which causes problems
|
69
77
|
when removing columns. Be sure to ignore the column%{column_suffix}:
|
70
78
|
|
71
79
|
class %{model} < %{base_model}
|
@@ -81,7 +89,8 @@ class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
81
89
|
end",
|
82
90
|
|
83
91
|
rename_column:
|
84
|
-
"Renaming a column
|
92
|
+
"Renaming a column that's in use will cause errors
|
93
|
+
in your application. A safer approach is to:
|
85
94
|
|
86
95
|
1. Create a new column
|
87
96
|
2. Write to both columns
|
@@ -91,7 +100,8 @@ end",
|
|
91
100
|
6. Drop the old column",
|
92
101
|
|
93
102
|
rename_table:
|
94
|
-
"Renaming a table
|
103
|
+
"Renaming a table that's in use will cause errors
|
104
|
+
in your application. A safer approach is to:
|
95
105
|
|
96
106
|
1. Create a new table. Don't forget to recreate indexes from the old table
|
97
107
|
2. Write to both tables
|
@@ -112,7 +122,7 @@ class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
112
122
|
end",
|
113
123
|
|
114
124
|
add_index:
|
115
|
-
"Adding an index non-concurrently
|
125
|
+
"Adding an index non-concurrently blocks writes. Instead, use:
|
116
126
|
|
117
127
|
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
118
128
|
disable_ddl_transaction!
|
@@ -123,7 +133,7 @@ class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
123
133
|
end",
|
124
134
|
|
125
135
|
remove_index:
|
126
|
-
"Removing an index non-concurrently
|
136
|
+
"Removing an index non-concurrently blocks writes. Instead, use:
|
127
137
|
|
128
138
|
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
129
139
|
disable_ddl_transaction!
|
@@ -166,9 +176,8 @@ class Backfill%{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
166
176
|
end",
|
167
177
|
|
168
178
|
change_column_null_postgresql:
|
169
|
-
"Setting NOT NULL on
|
170
|
-
|
171
|
-
validate it in a separate migration with a more agreeable RowShareLock.
|
179
|
+
"Setting NOT NULL on an existing column blocks reads and writes while every row is checked.
|
180
|
+
Instead, add a check constraint and validate it in a separate migration.
|
172
181
|
|
173
182
|
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
174
183
|
def change
|
@@ -182,26 +191,13 @@ class Validate%{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
182
191
|
end
|
183
192
|
end",
|
184
193
|
|
185
|
-
change_column_null_postgresql_helper:
|
186
|
-
"Setting NOT NULL on a column requires an AccessExclusiveLock,
|
187
|
-
which is expensive on large tables. Instead, we can use a constraint and
|
188
|
-
validate it in a separate step with a more agreeable RowShareLock.
|
189
|
-
|
190
|
-
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
191
|
-
disable_ddl_transaction!
|
192
|
-
|
193
|
-
def change
|
194
|
-
%{command}
|
195
|
-
end
|
196
|
-
end",
|
197
|
-
|
198
194
|
change_column_null_mysql:
|
199
195
|
"Setting NOT NULL on an existing column is not safe with your database engine.",
|
200
196
|
|
201
197
|
add_foreign_key:
|
202
|
-
"
|
203
|
-
|
204
|
-
|
198
|
+
"Adding a foreign key blocks writes on both tables. Instead,
|
199
|
+
add the foreign key without validating existing rows,
|
200
|
+
then validate them in a separate migration.
|
205
201
|
|
206
202
|
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
207
203
|
def change
|
@@ -215,21 +211,12 @@ class Validate%{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
|
215
211
|
end
|
216
212
|
end",
|
217
213
|
|
218
|
-
|
219
|
-
"
|
220
|
-
|
221
|
-
with a more agreeable RowShareLock.
|
222
|
-
|
223
|
-
class %{migration_name} < ActiveRecord::Migration%{migration_suffix}
|
224
|
-
disable_ddl_transaction!
|
225
|
-
|
226
|
-
def change
|
227
|
-
%{command}
|
228
|
-
end
|
229
|
-
end",
|
214
|
+
validate_foreign_key:
|
215
|
+
"Validating a foreign key while writes are blocked is dangerous.
|
216
|
+
Use disable_ddl_transaction! or a separate migration."
|
230
217
|
}
|
231
218
|
self.enabled_checks = (error_messages.keys - [:remove_index]).map { |k| [k, {}] }.to_h
|
232
|
-
self.
|
219
|
+
self.check_down = false
|
233
220
|
|
234
221
|
# private
|
235
222
|
def self.developer_env?
|
@@ -263,13 +250,6 @@ end",
|
|
263
250
|
false
|
264
251
|
end
|
265
252
|
end
|
266
|
-
|
267
|
-
# def self.enable_helpers
|
268
|
-
# unless helpers
|
269
|
-
# ActiveRecord::Migration.include(StrongMigrations::MigrationHelpers)
|
270
|
-
# self.helpers = true
|
271
|
-
# end
|
272
|
-
# end
|
273
253
|
end
|
274
254
|
|
275
255
|
ActiveSupport.on_load(:active_record) do
|
@@ -96,11 +96,13 @@ Then add the NOT NULL constraint in separate migrations."
|
|
96
96
|
change_command: command_str("change_column_default", [table, column, default]),
|
97
97
|
remove_command: command_str("remove_column", [table, column]),
|
98
98
|
code: backfill_code(table, column, default),
|
99
|
-
append: append
|
99
|
+
append: append,
|
100
|
+
rewrite_blocks: rewrite_blocks
|
100
101
|
end
|
101
102
|
|
102
103
|
if type.to_s == "json" && postgresql?
|
103
|
-
raise_error :add_column_json
|
104
|
+
raise_error :add_column_json,
|
105
|
+
command: command_str("add_column", [table, column, :jsonb, options])
|
104
106
|
end
|
105
107
|
when :change_column
|
106
108
|
table, column, type, options = args
|
@@ -109,15 +111,24 @@ Then add the NOT NULL constraint in separate migrations."
|
|
109
111
|
safe = false
|
110
112
|
existing_column = connection.columns(table).find { |c| c.name.to_s == column.to_s }
|
111
113
|
if existing_column
|
112
|
-
|
114
|
+
existing_type = existing_column.sql_type.split("(").first
|
113
115
|
if postgresql?
|
114
116
|
case type.to_s
|
115
|
-
when "string"
|
116
|
-
# safe to
|
117
|
-
safe
|
117
|
+
when "string"
|
118
|
+
# safe to increase limit or remove it
|
119
|
+
# not safe to decrease limit or add a limit
|
120
|
+
case existing_type
|
121
|
+
when "character varying"
|
122
|
+
safe = !options[:limit] || (existing_column.limit && options[:limit] >= existing_column.limit)
|
123
|
+
when "text"
|
124
|
+
safe = !options[:limit]
|
125
|
+
end
|
126
|
+
when "text"
|
127
|
+
# safe to change varchar to text (and text to text)
|
128
|
+
safe = ["character varying", "text"].include?(existing_type)
|
118
129
|
when "numeric", "decimal"
|
119
130
|
# numeric and decimal are equivalent and can be used interchangably
|
120
|
-
safe = ["numeric", "decimal"].include?(
|
131
|
+
safe = ["numeric", "decimal"].include?(existing_type) &&
|
121
132
|
(
|
122
133
|
(
|
123
134
|
# unconstrained
|
@@ -130,7 +141,7 @@ Then add the NOT NULL constraint in separate migrations."
|
|
130
141
|
)
|
131
142
|
)
|
132
143
|
when "datetime", "timestamp", "timestamptz"
|
133
|
-
safe = ["timestamp without time zone", "timestamp with time zone"].include?(
|
144
|
+
safe = ["timestamp without time zone", "timestamp with time zone"].include?(existing_type) &&
|
134
145
|
postgresql_version >= Gem::Version.new("12") &&
|
135
146
|
connection.select_all("SHOW timezone").first["TimeZone"] == "UTC"
|
136
147
|
end
|
@@ -142,13 +153,19 @@ Then add the NOT NULL constraint in separate migrations."
|
|
142
153
|
# increased limit, but doesn't change number of length bytes
|
143
154
|
# 1-255 = 1 byte, 256-65532 = 2 bytes, 65533+ = too big for varchar
|
144
155
|
limit = options[:limit] || 255
|
145
|
-
safe = ["varchar"].include?(
|
156
|
+
safe = ["varchar"].include?(existing_type) &&
|
146
157
|
limit >= existing_column.limit &&
|
147
158
|
(limit <= 255 || existing_column.limit > 255)
|
148
159
|
end
|
149
160
|
end
|
150
161
|
end
|
151
|
-
|
162
|
+
|
163
|
+
# unsafe to set NOT NULL for safe types
|
164
|
+
if safe && existing_column.null && options[:null] == false
|
165
|
+
raise_error :change_column_with_not_null
|
166
|
+
end
|
167
|
+
|
168
|
+
raise_error :change_column, rewrite_blocks: rewrite_blocks unless safe
|
152
169
|
when :create_table
|
153
170
|
table, options = args
|
154
171
|
options ||= {}
|
@@ -176,7 +193,7 @@ Then add the NOT NULL constraint in separate migrations."
|
|
176
193
|
end
|
177
194
|
|
178
195
|
if options.delete(:foreign_key)
|
179
|
-
headline = "Adding a
|
196
|
+
headline = "Adding a foreign key blocks writes on both tables."
|
180
197
|
append = "
|
181
198
|
|
182
199
|
Then add the foreign key in separate migrations."
|
@@ -196,16 +213,25 @@ Then add the foreign key in separate migrations."
|
|
196
213
|
table, column, null, default = args
|
197
214
|
if !null
|
198
215
|
if postgresql?
|
199
|
-
|
200
|
-
|
201
|
-
|
202
|
-
|
216
|
+
safe = false
|
217
|
+
if postgresql_version >= Gem::Version.new("12")
|
218
|
+
# TODO likely need to quote the column in some situations
|
219
|
+
safe = constraints(table).any? { |c| c["def"] == "CHECK ((#{column} IS NOT NULL))" }
|
220
|
+
end
|
221
|
+
|
222
|
+
unless safe
|
203
223
|
# match https://github.com/nullobject/rein
|
204
224
|
constraint_name = "#{table}_#{column}_null"
|
205
225
|
|
226
|
+
validate_constraint_code = String.new(constraint_str("ALTER TABLE %s VALIDATE CONSTRAINT %s", [table, constraint_name]))
|
227
|
+
if postgresql_version >= Gem::Version.new("12")
|
228
|
+
validate_constraint_code << "\n #{command_str(:change_column_null, [table, column, null])}"
|
229
|
+
validate_constraint_code << "\n #{constraint_str("ALTER TABLE %s DROP CONSTRAINT %s", [table, constraint_name])}"
|
230
|
+
end
|
231
|
+
|
206
232
|
raise_error :change_column_null_postgresql,
|
207
233
|
add_constraint_code: constraint_str("ALTER TABLE %s ADD CONSTRAINT %s CHECK (%s IS NOT NULL) NOT VALID", [table, constraint_name, column]),
|
208
|
-
validate_constraint_code:
|
234
|
+
validate_constraint_code: validate_constraint_code
|
209
235
|
end
|
210
236
|
elsif mysql? || mariadb?
|
211
237
|
raise_error :change_column_null_mysql
|
@@ -222,10 +248,7 @@ Then add the foreign key in separate migrations."
|
|
222
248
|
validate = options.fetch(:validate, true) || ActiveRecord::VERSION::STRING < "5.2"
|
223
249
|
|
224
250
|
if postgresql? && validate
|
225
|
-
if
|
226
|
-
raise_error :add_foreign_key_helper,
|
227
|
-
command: command_str(:add_foreign_key_safely, [from_table, to_table, options])
|
228
|
-
elsif ActiveRecord::VERSION::STRING < "5.2"
|
251
|
+
if ActiveRecord::VERSION::STRING < "5.2"
|
229
252
|
# fk name logic from rails
|
230
253
|
primary_key = options[:primary_key] || "id"
|
231
254
|
column = options[:column] || "#{to_table.to_s.singularize}_id"
|
@@ -241,6 +264,10 @@ Then add the foreign key in separate migrations."
|
|
241
264
|
validate_foreign_key_code: command_str("validate_foreign_key", [from_table, to_table])
|
242
265
|
end
|
243
266
|
end
|
267
|
+
when :validate_foreign_key
|
268
|
+
if postgresql? && writes_blocked?
|
269
|
+
raise_error :validate_foreign_key
|
270
|
+
end
|
244
271
|
end
|
245
272
|
|
246
273
|
StrongMigrations.checks.each do |check|
|
@@ -250,9 +277,10 @@ Then add the foreign key in separate migrations."
|
|
250
277
|
|
251
278
|
result = yield
|
252
279
|
|
280
|
+
# outdated statistics + a new index can hurt performance of existing queries
|
253
281
|
if StrongMigrations.auto_analyze && direction == :up && method == :add_index
|
254
282
|
if postgresql?
|
255
|
-
connection.execute "ANALYZE
|
283
|
+
connection.execute "ANALYZE #{connection.quote_table_name(args[0].to_s)}"
|
256
284
|
elsif mariadb? || mysql?
|
257
285
|
connection.execute "ANALYZE TABLE #{connection.quote_table_name(args[0].to_s)}"
|
258
286
|
end
|
@@ -261,15 +289,17 @@ Then add the foreign key in separate migrations."
|
|
261
289
|
result
|
262
290
|
end
|
263
291
|
|
264
|
-
|
292
|
+
private
|
293
|
+
|
265
294
|
def set_timeouts
|
266
295
|
if !@timeouts_set
|
267
296
|
if StrongMigrations.statement_timeout
|
268
297
|
statement =
|
269
298
|
if postgresql?
|
270
|
-
"SET statement_timeout TO #{connection.quote(StrongMigrations.statement_timeout
|
299
|
+
"SET statement_timeout TO #{connection.quote(postgresql_timeout(StrongMigrations.statement_timeout))}"
|
271
300
|
elsif mysql?
|
272
|
-
|
301
|
+
# use ceil to prevent no timeout for values under 1 ms
|
302
|
+
"SET max_execution_time = #{connection.quote((StrongMigrations.statement_timeout.to_f * 1000).ceil)}"
|
273
303
|
elsif mariadb?
|
274
304
|
"SET max_statement_time = #{connection.quote(StrongMigrations.statement_timeout)}"
|
275
305
|
else
|
@@ -282,7 +312,7 @@ Then add the foreign key in separate migrations."
|
|
282
312
|
if StrongMigrations.lock_timeout
|
283
313
|
statement =
|
284
314
|
if postgresql?
|
285
|
-
"SET lock_timeout TO #{connection.quote(StrongMigrations.lock_timeout
|
315
|
+
"SET lock_timeout TO #{connection.quote(postgresql_timeout(StrongMigrations.lock_timeout))}"
|
286
316
|
elsif mysql? || mariadb?
|
287
317
|
"SET lock_wait_timeout = #{connection.quote(StrongMigrations.lock_timeout)}"
|
288
318
|
else
|
@@ -296,8 +326,6 @@ Then add the foreign key in separate migrations."
|
|
296
326
|
end
|
297
327
|
end
|
298
328
|
|
299
|
-
private
|
300
|
-
|
301
329
|
def connection
|
302
330
|
@migration.connection
|
303
331
|
end
|
@@ -307,7 +335,8 @@ Then add the foreign key in separate migrations."
|
|
307
335
|
end
|
308
336
|
|
309
337
|
def safe?
|
310
|
-
@safe || ENV["SAFETY_ASSURED"] || @migration.is_a?(ActiveRecord::Schema) ||
|
338
|
+
@safe || ENV["SAFETY_ASSURED"] || @migration.is_a?(ActiveRecord::Schema) ||
|
339
|
+
(direction == :down && !StrongMigrations.check_down) || version_safe?
|
311
340
|
end
|
312
341
|
|
313
342
|
def version_safe?
|
@@ -352,6 +381,7 @@ Then add the foreign key in separate migrations."
|
|
352
381
|
end
|
353
382
|
|
354
383
|
def target_version(target_version)
|
384
|
+
target_version ||= StrongMigrations.target_version
|
355
385
|
version =
|
356
386
|
if target_version && StrongMigrations.developer_env?
|
357
387
|
target_version.to_s
|
@@ -369,22 +399,25 @@ Then add the foreign key in separate migrations."
|
|
369
399
|
lock_timeout = connection.select_all("SHOW lock_timeout").first["lock_timeout"]
|
370
400
|
lock_timeout_sec = timeout_to_sec(lock_timeout)
|
371
401
|
if lock_timeout_sec == 0
|
372
|
-
warn "[strong_migrations]
|
402
|
+
warn "[strong_migrations] DANGER: No lock timeout set"
|
373
403
|
elsif lock_timeout_sec > limit
|
374
|
-
warn "[strong_migrations]
|
404
|
+
warn "[strong_migrations] DANGER: Lock timeout is longer than #{limit} seconds: #{lock_timeout}"
|
375
405
|
end
|
376
406
|
elsif mysql? || mariadb?
|
377
407
|
lock_timeout = connection.select_all("SHOW VARIABLES LIKE 'lock_wait_timeout'").first["Value"]
|
408
|
+
# lock timeout is an integer
|
378
409
|
if lock_timeout.to_i > limit
|
379
|
-
warn "[strong_migrations]
|
410
|
+
warn "[strong_migrations] DANGER: Lock timeout is longer than #{limit} seconds: #{lock_timeout}"
|
380
411
|
end
|
381
412
|
end
|
382
413
|
@lock_timeout_checked = true
|
383
414
|
end
|
384
415
|
end
|
385
416
|
|
417
|
+
# units: https://www.postgresql.org/docs/current/config-setting.html
|
386
418
|
def timeout_to_sec(timeout)
|
387
|
-
|
419
|
+
units = {
|
420
|
+
"us" => 0.001,
|
388
421
|
"ms" => 1,
|
389
422
|
"s" => 1000,
|
390
423
|
"min" => 1000 * 60,
|
@@ -392,7 +425,7 @@ Then add the foreign key in separate migrations."
|
|
392
425
|
"d" => 1000 * 60 * 60 * 24
|
393
426
|
}
|
394
427
|
timeout_ms = timeout.to_i
|
395
|
-
|
428
|
+
units.each do |k, v|
|
396
429
|
if timeout.end_with?(k)
|
397
430
|
timeout_ms *= v
|
398
431
|
break
|
@@ -401,8 +434,28 @@ Then add the foreign key in separate migrations."
|
|
401
434
|
timeout_ms / 1000.0
|
402
435
|
end
|
403
436
|
|
404
|
-
def
|
405
|
-
|
437
|
+
def postgresql_timeout(timeout)
|
438
|
+
if timeout.is_a?(String)
|
439
|
+
timeout
|
440
|
+
else
|
441
|
+
# use ceil to prevent no timeout for values under 1 ms
|
442
|
+
(timeout.to_f * 1000).ceil
|
443
|
+
end
|
444
|
+
end
|
445
|
+
|
446
|
+
def constraints(table_name)
|
447
|
+
query = <<~SQL
|
448
|
+
SELECT
|
449
|
+
conname AS name,
|
450
|
+
pg_get_constraintdef(oid) AS def
|
451
|
+
FROM
|
452
|
+
pg_constraint
|
453
|
+
WHERE
|
454
|
+
contype = 'c' AND
|
455
|
+
convalidated AND
|
456
|
+
conrelid = #{connection.quote(connection.quote_table_name(table_name))}::regclass
|
457
|
+
SQL
|
458
|
+
connection.select_all(query.squish).to_a
|
406
459
|
end
|
407
460
|
|
408
461
|
def raise_error(message_key, header: nil, append: nil, **vars)
|
@@ -449,6 +502,23 @@ Then add the foreign key in separate migrations."
|
|
449
502
|
"#{command} #{str_args.join(", ")}"
|
450
503
|
end
|
451
504
|
|
505
|
+
def writes_blocked?
|
506
|
+
query = <<~SQL
|
507
|
+
SELECT
|
508
|
+
relation::regclass::text
|
509
|
+
FROM
|
510
|
+
pg_locks
|
511
|
+
WHERE
|
512
|
+
mode IN ('ShareRowExclusiveLock', 'AccessExclusiveLock') AND
|
513
|
+
pid = pg_backend_pid()
|
514
|
+
SQL
|
515
|
+
connection.select_all(query.squish).any?
|
516
|
+
end
|
517
|
+
|
518
|
+
def rewrite_blocks
|
519
|
+
mysql? || mariadb? ? "writes" : "reads and writes"
|
520
|
+
end
|
521
|
+
|
452
522
|
def backfill_code(table, column, default)
|
453
523
|
model = table.to_s.classify
|
454
524
|
"#{model}.unscoped.in_batches do |relation| \n relation.update_all #{column}: #{default.inspect}\n sleep(0.01)\n end"
|
@@ -5,10 +5,6 @@ module StrongMigrations
|
|
5
5
|
class Railtie < Rails::Railtie
|
6
6
|
rake_tasks do
|
7
7
|
load "tasks/strong_migrations.rake"
|
8
|
-
|
9
|
-
["db:drop", "db:reset", "db:schema:load", "db:structure:load"].each do |t|
|
10
|
-
Rake::Task[t].enhance ["strong_migrations:safety_assured"]
|
11
|
-
end
|
12
8
|
end
|
13
9
|
end
|
14
10
|
end
|
@@ -1,10 +1,4 @@
|
|
1
|
-
# https://nithinbekal.com/posts/safe-rake-tasks
|
2
|
-
|
3
1
|
namespace :strong_migrations do
|
4
|
-
task safety_assured: :environment do
|
5
|
-
raise "Set SAFETY_ASSURED=1 to run this task in production" if Rails.env.production? && !ENV["SAFETY_ASSURED"]
|
6
|
-
end
|
7
|
-
|
8
2
|
# https://www.pgrs.net/2008/03/13/alphabetize-schema-rb-columns/
|
9
3
|
task :alphabetize_columns do
|
10
4
|
$stderr.puts "Dumping schema"
|
metadata
CHANGED
@@ -1,16 +1,16 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: strong_migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 0.7.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Andrew Kane
|
8
8
|
- Bob Remeika
|
9
9
|
- David Waller
|
10
|
-
autorequire:
|
10
|
+
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2020-
|
13
|
+
date: 2020-10-25 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: activerecord
|
@@ -96,7 +96,7 @@ dependencies:
|
|
96
96
|
- - ">="
|
97
97
|
- !ruby/object:Gem::Version
|
98
98
|
version: '0'
|
99
|
-
description:
|
99
|
+
description:
|
100
100
|
email:
|
101
101
|
- andrew@chartkick.com
|
102
102
|
- bob.remeika@gmail.com
|
@@ -115,7 +115,6 @@ files:
|
|
115
115
|
- lib/strong_migrations/checker.rb
|
116
116
|
- lib/strong_migrations/database_tasks.rb
|
117
117
|
- lib/strong_migrations/migration.rb
|
118
|
-
- lib/strong_migrations/migration_helpers.rb
|
119
118
|
- lib/strong_migrations/railtie.rb
|
120
119
|
- lib/strong_migrations/version.rb
|
121
120
|
- lib/tasks/strong_migrations.rake
|
@@ -123,7 +122,7 @@ homepage: https://github.com/ankane/strong_migrations
|
|
123
122
|
licenses:
|
124
123
|
- MIT
|
125
124
|
metadata: {}
|
126
|
-
post_install_message:
|
125
|
+
post_install_message:
|
127
126
|
rdoc_options: []
|
128
127
|
require_paths:
|
129
128
|
- lib
|
@@ -138,8 +137,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
138
137
|
- !ruby/object:Gem::Version
|
139
138
|
version: '0'
|
140
139
|
requirements: []
|
141
|
-
rubygems_version: 3.1.
|
142
|
-
signing_key:
|
140
|
+
rubygems_version: 3.1.4
|
141
|
+
signing_key:
|
143
142
|
specification_version: 4
|
144
143
|
summary: Catch unsafe migrations in development
|
145
144
|
test_files: []
|
@@ -1,117 +0,0 @@
|
|
1
|
-
module StrongMigrations
|
2
|
-
module MigrationHelpers
|
3
|
-
def add_foreign_key_safely(from_table, to_table, **options)
|
4
|
-
ensure_postgresql(__method__)
|
5
|
-
ensure_not_in_transaction(__method__)
|
6
|
-
|
7
|
-
reversible do |dir|
|
8
|
-
dir.up do
|
9
|
-
if ActiveRecord::VERSION::STRING >= "5.2"
|
10
|
-
add_foreign_key(from_table, to_table, options.merge(validate: false))
|
11
|
-
validate_foreign_key(from_table, to_table)
|
12
|
-
else
|
13
|
-
options = connection.foreign_key_options(from_table, to_table, options)
|
14
|
-
fk_name, column, primary_key = options.values_at(:name, :column, :primary_key)
|
15
|
-
primary_key ||= "id"
|
16
|
-
|
17
|
-
statement = ["ALTER TABLE %s ADD CONSTRAINT %s FOREIGN KEY (%s) REFERENCES %s (%s)"]
|
18
|
-
statement << on_delete_update_statement(:delete, options[:on_delete]) if options[:on_delete]
|
19
|
-
statement << on_delete_update_statement(:update, options[:on_update]) if options[:on_update]
|
20
|
-
statement << "NOT VALID"
|
21
|
-
|
22
|
-
safety_assured do
|
23
|
-
execute quote_identifiers(statement.join(" "), [from_table, fk_name, column, to_table, primary_key])
|
24
|
-
execute quote_identifiers("ALTER TABLE %s VALIDATE CONSTRAINT %s", [from_table, fk_name])
|
25
|
-
end
|
26
|
-
end
|
27
|
-
end
|
28
|
-
|
29
|
-
dir.down do
|
30
|
-
remove_foreign_key(from_table, to_table)
|
31
|
-
end
|
32
|
-
end
|
33
|
-
end
|
34
|
-
|
35
|
-
def add_null_constraint_safely(table_name, column_name, name: nil)
|
36
|
-
ensure_postgresql(__method__)
|
37
|
-
ensure_not_in_transaction(__method__)
|
38
|
-
|
39
|
-
reversible do |dir|
|
40
|
-
dir.up do
|
41
|
-
name ||= null_constraint_name(table_name, column_name)
|
42
|
-
|
43
|
-
safety_assured do
|
44
|
-
execute quote_identifiers("ALTER TABLE %s ADD CONSTRAINT %s CHECK (%s IS NOT NULL) NOT VALID", [table_name, name, column_name])
|
45
|
-
execute quote_identifiers("ALTER TABLE %s VALIDATE CONSTRAINT %s", [table_name, name])
|
46
|
-
end
|
47
|
-
end
|
48
|
-
|
49
|
-
dir.down do
|
50
|
-
remove_null_constraint_safely(table_name, column_name)
|
51
|
-
end
|
52
|
-
end
|
53
|
-
end
|
54
|
-
|
55
|
-
# removing constraints is safe, but this method is safe to reverse as well
|
56
|
-
def remove_null_constraint_safely(table_name, column_name, name: nil)
|
57
|
-
# could also ensure in transaction so it can be reversed
|
58
|
-
# but that's more of a concern for a reversible migrations check
|
59
|
-
ensure_postgresql(__method__)
|
60
|
-
|
61
|
-
reversible do |dir|
|
62
|
-
dir.up do
|
63
|
-
name ||= null_constraint_name(table_name, column_name)
|
64
|
-
|
65
|
-
safety_assured do
|
66
|
-
execute quote_identifiers("ALTER TABLE %s DROP CONSTRAINT %s", [table_name, name])
|
67
|
-
end
|
68
|
-
end
|
69
|
-
|
70
|
-
dir.down do
|
71
|
-
add_null_constraint_safely(table_name, column_name)
|
72
|
-
end
|
73
|
-
end
|
74
|
-
end
|
75
|
-
|
76
|
-
private
|
77
|
-
|
78
|
-
def ensure_postgresql(method_name)
|
79
|
-
raise StrongMigrations::Error, "`#{method_name}` is intended for Postgres only" unless postgresql?
|
80
|
-
end
|
81
|
-
|
82
|
-
def postgresql?
|
83
|
-
%w(PostgreSQL PostGIS).include?(connection.adapter_name)
|
84
|
-
end
|
85
|
-
|
86
|
-
def ensure_not_in_transaction(method_name)
|
87
|
-
if connection.transaction_open?
|
88
|
-
raise StrongMigrations::Error, "Cannot run `#{method_name}` inside a transaction. Use `disable_ddl_transaction` to disable the transaction."
|
89
|
-
end
|
90
|
-
end
|
91
|
-
|
92
|
-
# match https://github.com/nullobject/rein
|
93
|
-
def null_constraint_name(table_name, column_name)
|
94
|
-
"#{table_name}_#{column_name}_null"
|
95
|
-
end
|
96
|
-
|
97
|
-
def on_delete_update_statement(delete_or_update, action)
|
98
|
-
on = delete_or_update.to_s.upcase
|
99
|
-
|
100
|
-
case action
|
101
|
-
when :nullify
|
102
|
-
"ON #{on} SET NULL"
|
103
|
-
when :cascade
|
104
|
-
"ON #{on} CASCADE"
|
105
|
-
when :restrict
|
106
|
-
"ON #{on} RESTRICT"
|
107
|
-
else
|
108
|
-
# same error message as Active Record
|
109
|
-
raise "'#{action}' is not supported for :on_update or :on_delete.\nSupported values are: :nullify, :cascade, :restrict"
|
110
|
-
end
|
111
|
-
end
|
112
|
-
|
113
|
-
def quote_identifiers(statement, identifiers)
|
114
|
-
statement % identifiers.map { |v| connection.quote_table_name(v) }
|
115
|
-
end
|
116
|
-
end
|
117
|
-
end
|