strong_migrations 0.6.8 → 1.2.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +93 -0
- data/LICENSE.txt +1 -1
- data/README.md +285 -117
- data/lib/generators/strong_migrations/install_generator.rb +29 -0
- data/lib/generators/strong_migrations/templates/initializer.rb.tt +10 -3
- data/lib/strong_migrations/adapters/abstract_adapter.rb +61 -0
- data/lib/strong_migrations/adapters/mariadb_adapter.rb +29 -0
- data/lib/strong_migrations/adapters/mysql_adapter.rb +87 -0
- data/lib/strong_migrations/adapters/postgresql_adapter.rb +227 -0
- data/lib/strong_migrations/checker.rb +116 -401
- data/lib/strong_migrations/checks.rb +421 -0
- data/lib/strong_migrations/database_tasks.rb +2 -1
- data/lib/strong_migrations/error_messages.rb +227 -0
- data/lib/strong_migrations/migration.rb +10 -2
- data/lib/strong_migrations/migrator.rb +19 -0
- data/lib/strong_migrations/railtie.rb +0 -4
- data/lib/strong_migrations/safe_methods.rb +118 -0
- data/lib/strong_migrations/version.rb +1 -1
- data/lib/strong_migrations.rb +20 -180
- data/lib/tasks/strong_migrations.rake +0 -6
- metadata +19 -81
data/README.md
CHANGED
@@ -4,18 +4,18 @@ Catch unsafe migrations in development
|
|
4
4
|
|
5
5
|
✓ Detects potentially dangerous operations<br /> ✓ Prevents them from running by default<br /> ✓ Provides instructions on safer ways to do what you want
|
6
6
|
|
7
|
-
Supports
|
7
|
+
Supports PostgreSQL, MySQL, and MariaDB
|
8
8
|
|
9
9
|
:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
|
10
10
|
|
11
|
-
[![Build Status](https://
|
11
|
+
[![Build Status](https://github.com/ankane/strong_migrations/workflows/build/badge.svg?branch=master)](https://github.com/ankane/strong_migrations/actions)
|
12
12
|
|
13
13
|
## Installation
|
14
14
|
|
15
15
|
Add this line to your application’s Gemfile:
|
16
16
|
|
17
17
|
```ruby
|
18
|
-
gem
|
18
|
+
gem "strong_migrations"
|
19
19
|
```
|
20
20
|
|
21
21
|
And run:
|
@@ -25,6 +25,36 @@ bundle install
|
|
25
25
|
rails generate strong_migrations:install
|
26
26
|
```
|
27
27
|
|
28
|
+
Strong Migrations sets a long statement timeout for migrations so you can set a [short statement timeout](#app-timeouts) for your application.
|
29
|
+
|
30
|
+
## How It Works
|
31
|
+
|
32
|
+
When you run a migration that’s potentially dangerous, you’ll see an error message like:
|
33
|
+
|
34
|
+
```txt
|
35
|
+
=== Dangerous operation detected #strong_migrations ===
|
36
|
+
|
37
|
+
Active Record caches attributes, which causes problems
|
38
|
+
when removing columns. Be sure to ignore the column:
|
39
|
+
|
40
|
+
class User < ApplicationRecord
|
41
|
+
self.ignored_columns = ["name"]
|
42
|
+
end
|
43
|
+
|
44
|
+
Deploy the code, then wrap this step in a safety_assured { ... } block.
|
45
|
+
|
46
|
+
class RemoveColumn < ActiveRecord::Migration[7.0]
|
47
|
+
def change
|
48
|
+
safety_assured { remove_column :users, :name }
|
49
|
+
end
|
50
|
+
end
|
51
|
+
```
|
52
|
+
|
53
|
+
An operation is classified as dangerous if it either:
|
54
|
+
|
55
|
+
- Blocks reads or writes for more than a few seconds (after a lock is acquired)
|
56
|
+
- Has a good chance of causing application errors
|
57
|
+
|
28
58
|
## Checks
|
29
59
|
|
30
60
|
Potentially dangerous operations:
|
@@ -32,11 +62,11 @@ Potentially dangerous operations:
|
|
32
62
|
- [removing a column](#removing-a-column)
|
33
63
|
- [adding a column with a default value](#adding-a-column-with-a-default-value)
|
34
64
|
- [backfilling data](#backfilling-data)
|
35
|
-
- [changing the type of a column](#
|
36
|
-
- [renaming a column](#renaming-
|
65
|
+
- [changing the type of a column](#changing-the-type-of-a-column)
|
66
|
+
- [renaming a column](#renaming-a-column)
|
37
67
|
- [renaming a table](#renaming-a-table)
|
38
68
|
- [creating a table with the force option](#creating-a-table-with-the-force-option)
|
39
|
-
- [
|
69
|
+
- [adding a check constraint](#adding-a-check-constraint)
|
40
70
|
- [executing SQL directly](#executing-SQL-directly)
|
41
71
|
|
42
72
|
Postgres-specific checks:
|
@@ -57,10 +87,10 @@ You can also add [custom checks](#custom-checks) or [disable specific checks](#d
|
|
57
87
|
|
58
88
|
#### Bad
|
59
89
|
|
60
|
-
|
90
|
+
Active Record caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
|
61
91
|
|
62
92
|
```ruby
|
63
|
-
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[
|
93
|
+
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[7.0]
|
64
94
|
def change
|
65
95
|
remove_column :users, :some_column
|
66
96
|
end
|
@@ -69,7 +99,7 @@ end
|
|
69
99
|
|
70
100
|
#### Good
|
71
101
|
|
72
|
-
1. Tell
|
102
|
+
1. Tell Active Record to ignore the column from its cache
|
73
103
|
|
74
104
|
```ruby
|
75
105
|
class User < ApplicationRecord
|
@@ -81,7 +111,7 @@ end
|
|
81
111
|
3. Write a migration to remove the column (wrap in `safety_assured` block)
|
82
112
|
|
83
113
|
```ruby
|
84
|
-
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[
|
114
|
+
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[7.0]
|
85
115
|
def change
|
86
116
|
safety_assured { remove_column :users, :some_column }
|
87
117
|
end
|
@@ -89,29 +119,30 @@ end
|
|
89
119
|
```
|
90
120
|
|
91
121
|
4. Deploy and run migration
|
122
|
+
5. Remove the line added in step 1
|
92
123
|
|
93
124
|
### Adding a column with a default value
|
94
125
|
|
95
|
-
Note: This operation is safe in Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+.
|
96
|
-
|
97
126
|
#### Bad
|
98
127
|
|
99
|
-
|
128
|
+
In earlier versions of Postgres, MySQL, and MariaDB, adding a column with a default value to an existing table causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
100
129
|
|
101
130
|
```ruby
|
102
|
-
class AddSomeColumnToUsers < ActiveRecord::Migration[
|
131
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
103
132
|
def change
|
104
133
|
add_column :users, :some_column, :text, default: "default_value"
|
105
134
|
end
|
106
135
|
end
|
107
136
|
```
|
108
137
|
|
138
|
+
In Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+, this no longer requires a table rewrite and is safe.
|
139
|
+
|
109
140
|
#### Good
|
110
141
|
|
111
142
|
Instead, add the column without a default value, then change the default.
|
112
143
|
|
113
144
|
```ruby
|
114
|
-
class AddSomeColumnToUsers < ActiveRecord::Migration[
|
145
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
115
146
|
def up
|
116
147
|
add_column :users, :some_column, :text
|
117
148
|
change_column_default :users, :some_column, "default_value"
|
@@ -129,10 +160,10 @@ See the next section for how to backfill.
|
|
129
160
|
|
130
161
|
#### Bad
|
131
162
|
|
132
|
-
|
163
|
+
Active Record creates a transaction around each migration, and backfilling in the same transaction that alters a table keeps the table locked for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
133
164
|
|
134
165
|
```ruby
|
135
|
-
class AddSomeColumnToUsers < ActiveRecord::Migration[
|
166
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
136
167
|
def change
|
137
168
|
add_column :users, :some_column, :text
|
138
169
|
User.update_all some_column: "default_value"
|
@@ -147,7 +178,7 @@ Also, running a single query to update data can cause issues for large tables.
|
|
147
178
|
There are three keys to backfilling safely: batching, throttling, and running it outside a transaction. Use the Rails console or a separate migration with `disable_ddl_transaction!`.
|
148
179
|
|
149
180
|
```ruby
|
150
|
-
class BackfillSomeColumn < ActiveRecord::Migration[
|
181
|
+
class BackfillSomeColumn < ActiveRecord::Migration[7.0]
|
151
182
|
disable_ddl_transaction!
|
152
183
|
|
153
184
|
def up
|
@@ -159,40 +190,66 @@ class BackfillSomeColumn < ActiveRecord::Migration[6.0]
|
|
159
190
|
end
|
160
191
|
```
|
161
192
|
|
162
|
-
###
|
193
|
+
### Changing the type of a column
|
163
194
|
|
164
195
|
#### Bad
|
165
196
|
|
197
|
+
Changing the type of a column causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
198
|
+
|
166
199
|
```ruby
|
167
|
-
class
|
200
|
+
class ChangeSomeColumnType < ActiveRecord::Migration[7.0]
|
168
201
|
def change
|
169
|
-
|
202
|
+
change_column :users, :some_column, :new_type
|
170
203
|
end
|
171
204
|
end
|
172
205
|
```
|
173
206
|
|
174
|
-
|
207
|
+
Some changes don’t require a table rewrite and are safe in Postgres:
|
208
|
+
|
209
|
+
Type | Safe Changes
|
210
|
+
--- | ---
|
211
|
+
`cidr` | Changing to `inet`
|
212
|
+
`citext` | Changing to `text` if not indexed, changing to `string` with no `:limit` if not indexed
|
213
|
+
`datetime` | Increasing or removing `:precision`, changing to `timestamptz` when session time zone is UTC in Postgres 12+
|
214
|
+
`decimal` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
|
215
|
+
`interval` | Increasing or removing `:precision`
|
216
|
+
`numeric` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
|
217
|
+
`string` | Increasing or removing `:limit`, changing to `text`, changing `citext` if not indexed
|
218
|
+
`text` | Changing to `string` with no `:limit`, changing to `citext` if not indexed
|
219
|
+
`time` | Increasing or removing `:precision`
|
220
|
+
`timestamptz` | Increasing or removing `:limit`, changing to `datetime` when session time zone is UTC in Postgres 12+
|
221
|
+
|
222
|
+
And some in MySQL and MariaDB:
|
223
|
+
|
224
|
+
Type | Safe Changes
|
225
|
+
--- | ---
|
226
|
+
`string` | Increasing `:limit` from under 255 up to 255, increasing `:limit` from over 255 to the max
|
227
|
+
|
228
|
+
#### Good
|
229
|
+
|
230
|
+
A safer approach is to:
|
231
|
+
|
232
|
+
1. Create a new column
|
233
|
+
2. Write to both columns
|
234
|
+
3. Backfill data from the old column to the new column
|
235
|
+
4. Move reads from the old column to the new column
|
236
|
+
5. Stop writing to the old column
|
237
|
+
6. Drop the old column
|
238
|
+
|
239
|
+
### Renaming a column
|
240
|
+
|
241
|
+
#### Bad
|
242
|
+
|
243
|
+
Renaming a column that’s in use will cause errors in your application.
|
175
244
|
|
176
245
|
```ruby
|
177
|
-
class
|
246
|
+
class RenameSomeColumn < ActiveRecord::Migration[7.0]
|
178
247
|
def change
|
179
|
-
|
248
|
+
rename_column :users, :some_column, :new_name
|
180
249
|
end
|
181
250
|
end
|
182
251
|
```
|
183
252
|
|
184
|
-
A few changes are safe in Postgres:
|
185
|
-
|
186
|
-
- Changing between `varchar` and `text` columns
|
187
|
-
- Increasing the precision of a `decimal` or `numeric` column
|
188
|
-
- Making a `decimal` or `numeric` column unconstrained
|
189
|
-
- Changing between `timestamp` and `timestamptz` columns when session time zone is UTC in Postgres 12+
|
190
|
-
|
191
|
-
And a few in MySQL and MariaDB:
|
192
|
-
|
193
|
-
- Increasing the length of a `varchar` column from under 255 up to 255
|
194
|
-
- Increasing the length of a `varchar` column over 255
|
195
|
-
|
196
253
|
#### Good
|
197
254
|
|
198
255
|
A safer approach is to:
|
@@ -208,8 +265,10 @@ A safer approach is to:
|
|
208
265
|
|
209
266
|
#### Bad
|
210
267
|
|
268
|
+
Renaming a table that’s in use will cause errors in your application.
|
269
|
+
|
211
270
|
```ruby
|
212
|
-
class RenameUsersToCustomers < ActiveRecord::Migration[
|
271
|
+
class RenameUsersToCustomers < ActiveRecord::Migration[7.0]
|
213
272
|
def change
|
214
273
|
rename_table :users, :customers
|
215
274
|
end
|
@@ -234,7 +293,7 @@ A safer approach is to:
|
|
234
293
|
The `force` option can drop an existing table.
|
235
294
|
|
236
295
|
```ruby
|
237
|
-
class CreateUsers < ActiveRecord::Migration[
|
296
|
+
class CreateUsers < ActiveRecord::Migration[7.0]
|
238
297
|
def change
|
239
298
|
create_table :users, force: true do |t|
|
240
299
|
# ...
|
@@ -248,7 +307,7 @@ end
|
|
248
307
|
Create tables without the `force` option.
|
249
308
|
|
250
309
|
```ruby
|
251
|
-
class CreateUsers < ActiveRecord::Migration[
|
310
|
+
class CreateUsers < ActiveRecord::Migration[7.0]
|
252
311
|
def change
|
253
312
|
create_table :users do |t|
|
254
313
|
# ...
|
@@ -259,40 +318,54 @@ end
|
|
259
318
|
|
260
319
|
If you intend to drop an existing table, run `drop_table` first.
|
261
320
|
|
262
|
-
###
|
321
|
+
### Adding a check constraint
|
322
|
+
|
323
|
+
:turtle: Safe by default available
|
263
324
|
|
264
325
|
#### Bad
|
265
326
|
|
266
|
-
|
327
|
+
Adding a check constraint blocks reads and writes in Postgres and blocks writes in MySQL and MariaDB while every row is checked.
|
267
328
|
|
268
329
|
```ruby
|
269
|
-
class
|
330
|
+
class AddCheckConstraint < ActiveRecord::Migration[7.0]
|
270
331
|
def change
|
271
|
-
|
332
|
+
add_check_constraint :users, "price > 0", name: "price_check"
|
272
333
|
end
|
273
334
|
end
|
274
335
|
```
|
275
336
|
|
276
|
-
#### Good
|
337
|
+
#### Good - Postgres
|
277
338
|
|
278
|
-
|
339
|
+
Add the check constraint without validating existing rows:
|
279
340
|
|
280
341
|
```ruby
|
281
|
-
class
|
342
|
+
class AddCheckConstraint < ActiveRecord::Migration[7.0]
|
282
343
|
def change
|
283
|
-
|
344
|
+
add_check_constraint :users, "price > 0", name: "price_check", validate: false
|
345
|
+
end
|
346
|
+
end
|
347
|
+
```
|
348
|
+
|
349
|
+
Then validate them in a separate migration.
|
350
|
+
|
351
|
+
```ruby
|
352
|
+
class ValidateCheckConstraint < ActiveRecord::Migration[7.0]
|
353
|
+
def change
|
354
|
+
validate_check_constraint :users, name: "price_check"
|
284
355
|
end
|
285
356
|
end
|
286
357
|
```
|
287
358
|
|
288
|
-
|
359
|
+
#### Good - MySQL and MariaDB
|
360
|
+
|
361
|
+
[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this (check constraints can be added with `NOT ENFORCED`, but enforcing blocks writes).
|
289
362
|
|
290
363
|
### Executing SQL directly
|
291
364
|
|
292
365
|
Strong Migrations can’t ensure safety for raw SQL statements. Make really sure that what you’re doing is safe, then use:
|
293
366
|
|
294
367
|
```ruby
|
295
|
-
class ExecuteSQL < ActiveRecord::Migration[
|
368
|
+
class ExecuteSQL < ActiveRecord::Migration[7.0]
|
296
369
|
def change
|
297
370
|
safety_assured { execute "..." }
|
298
371
|
end
|
@@ -301,12 +374,14 @@ end
|
|
301
374
|
|
302
375
|
### Adding an index non-concurrently
|
303
376
|
|
377
|
+
:turtle: Safe by default available
|
378
|
+
|
304
379
|
#### Bad
|
305
380
|
|
306
|
-
In Postgres, adding an index non-concurrently
|
381
|
+
In Postgres, adding an index non-concurrently blocks writes.
|
307
382
|
|
308
383
|
```ruby
|
309
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
384
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
310
385
|
def change
|
311
386
|
add_index :users, :some_column
|
312
387
|
end
|
@@ -318,7 +393,7 @@ end
|
|
318
393
|
Add indexes concurrently.
|
319
394
|
|
320
395
|
```ruby
|
321
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
396
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
322
397
|
disable_ddl_transaction!
|
323
398
|
|
324
399
|
def change
|
@@ -337,12 +412,14 @@ rails g index table column
|
|
337
412
|
|
338
413
|
### Adding a reference
|
339
414
|
|
415
|
+
:turtle: Safe by default available
|
416
|
+
|
340
417
|
#### Bad
|
341
418
|
|
342
|
-
Rails adds an index non-concurrently to references by default, which
|
419
|
+
Rails adds an index non-concurrently to references by default, which blocks writes in Postgres.
|
343
420
|
|
344
421
|
```ruby
|
345
|
-
class AddReferenceToUsers < ActiveRecord::Migration[
|
422
|
+
class AddReferenceToUsers < ActiveRecord::Migration[7.0]
|
346
423
|
def change
|
347
424
|
add_reference :users, :city
|
348
425
|
end
|
@@ -354,7 +431,7 @@ end
|
|
354
431
|
Make sure the index is added concurrently.
|
355
432
|
|
356
433
|
```ruby
|
357
|
-
class AddReferenceToUsers < ActiveRecord::Migration[
|
434
|
+
class AddReferenceToUsers < ActiveRecord::Migration[7.0]
|
358
435
|
disable_ddl_transaction!
|
359
436
|
|
360
437
|
def change
|
@@ -365,12 +442,14 @@ end
|
|
365
442
|
|
366
443
|
### Adding a foreign key
|
367
444
|
|
445
|
+
:turtle: Safe by default available
|
446
|
+
|
368
447
|
#### Bad
|
369
448
|
|
370
|
-
In Postgres,
|
449
|
+
In Postgres, adding a foreign key blocks writes on both tables.
|
371
450
|
|
372
451
|
```ruby
|
373
|
-
class AddForeignKeyOnUsers < ActiveRecord::Migration[
|
452
|
+
class AddForeignKeyOnUsers < ActiveRecord::Migration[7.0]
|
374
453
|
def change
|
375
454
|
add_foreign_key :users, :orders
|
376
455
|
end
|
@@ -380,7 +459,7 @@ end
|
|
380
459
|
or
|
381
460
|
|
382
461
|
```ruby
|
383
|
-
class AddReferenceToUsers < ActiveRecord::Migration[
|
462
|
+
class AddReferenceToUsers < ActiveRecord::Migration[7.0]
|
384
463
|
def change
|
385
464
|
add_reference :users, :order, foreign_key: true
|
386
465
|
end
|
@@ -389,107 +468,111 @@ end
|
|
389
468
|
|
390
469
|
#### Good
|
391
470
|
|
392
|
-
|
393
|
-
|
394
|
-
For Rails 5.2+, use:
|
471
|
+
Add the foreign key without validating existing rows:
|
395
472
|
|
396
473
|
```ruby
|
397
|
-
class AddForeignKeyOnUsers < ActiveRecord::Migration[
|
474
|
+
class AddForeignKeyOnUsers < ActiveRecord::Migration[7.0]
|
398
475
|
def change
|
399
476
|
add_foreign_key :users, :orders, validate: false
|
400
477
|
end
|
401
478
|
end
|
402
479
|
```
|
403
480
|
|
404
|
-
Then validate
|
481
|
+
Then validate them in a separate migration.
|
405
482
|
|
406
483
|
```ruby
|
407
|
-
class ValidateForeignKeyOnUsers < ActiveRecord::Migration[
|
484
|
+
class ValidateForeignKeyOnUsers < ActiveRecord::Migration[7.0]
|
408
485
|
def change
|
409
486
|
validate_foreign_key :users, :orders
|
410
487
|
end
|
411
488
|
end
|
412
489
|
```
|
413
490
|
|
414
|
-
|
491
|
+
### Adding a json column
|
492
|
+
|
493
|
+
#### Bad
|
494
|
+
|
495
|
+
In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries in your application.
|
415
496
|
|
416
497
|
```ruby
|
417
|
-
class
|
498
|
+
class AddPropertiesToUsers < ActiveRecord::Migration[7.0]
|
418
499
|
def change
|
419
|
-
|
420
|
-
execute 'ALTER TABLE "users" ADD CONSTRAINT "fk_rails_c1e9b98e31" FOREIGN KEY ("order_id") REFERENCES "orders" ("id") NOT VALID'
|
421
|
-
end
|
500
|
+
add_column :users, :properties, :json
|
422
501
|
end
|
423
502
|
end
|
424
503
|
```
|
425
504
|
|
426
|
-
|
505
|
+
#### Good
|
506
|
+
|
507
|
+
Use `jsonb` instead.
|
427
508
|
|
428
509
|
```ruby
|
429
|
-
class
|
510
|
+
class AddPropertiesToUsers < ActiveRecord::Migration[7.0]
|
430
511
|
def change
|
431
|
-
|
432
|
-
execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "fk_rails_c1e9b98e31"'
|
433
|
-
end
|
512
|
+
add_column :users, :properties, :jsonb
|
434
513
|
end
|
435
514
|
end
|
436
515
|
```
|
437
516
|
|
438
|
-
###
|
517
|
+
### Setting NOT NULL on an existing column
|
518
|
+
|
519
|
+
:turtle: Safe by default available
|
439
520
|
|
440
521
|
#### Bad
|
441
522
|
|
442
|
-
In Postgres,
|
523
|
+
In Postgres, setting `NOT NULL` on an existing column blocks reads and writes while every row is checked.
|
443
524
|
|
444
525
|
```ruby
|
445
|
-
class
|
526
|
+
class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
|
446
527
|
def change
|
447
|
-
|
528
|
+
change_column_null :users, :some_column, false
|
448
529
|
end
|
449
530
|
end
|
450
531
|
```
|
451
532
|
|
452
533
|
#### Good
|
453
534
|
|
454
|
-
|
535
|
+
Instead, add a check constraint.
|
536
|
+
|
537
|
+
For Rails 6.1, use:
|
455
538
|
|
456
539
|
```ruby
|
457
|
-
class
|
540
|
+
class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
|
458
541
|
def change
|
459
|
-
|
542
|
+
add_check_constraint :users, "some_column IS NOT NULL", name: "users_some_column_null", validate: false
|
460
543
|
end
|
461
544
|
end
|
462
545
|
```
|
463
546
|
|
464
|
-
|
465
|
-
|
466
|
-
#### Bad
|
467
|
-
|
468
|
-
In Postgres, setting `NOT NULL` on an existing column requires an `AccessExclusiveLock`, which is expensive on large tables.
|
547
|
+
For Rails < 6.1, use:
|
469
548
|
|
470
549
|
```ruby
|
471
550
|
class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
472
551
|
def change
|
473
|
-
|
552
|
+
safety_assured do
|
553
|
+
execute 'ALTER TABLE "users" ADD CONSTRAINT "users_some_column_null" CHECK ("some_column" IS NOT NULL) NOT VALID'
|
554
|
+
end
|
474
555
|
end
|
475
556
|
end
|
476
557
|
```
|
477
558
|
|
478
|
-
|
559
|
+
Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column (but it won’t show up in `schema.rb` in Rails < 6.1). In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
|
479
560
|
|
480
|
-
|
561
|
+
For Rails 6.1, use:
|
481
562
|
|
482
563
|
```ruby
|
483
|
-
class
|
564
|
+
class ValidateSomeColumnNotNull < ActiveRecord::Migration[7.0]
|
484
565
|
def change
|
485
|
-
|
486
|
-
|
487
|
-
|
566
|
+
validate_check_constraint :users, name: "users_some_column_null"
|
567
|
+
|
568
|
+
# in Postgres 12+, you can then safely set NOT NULL on the column
|
569
|
+
change_column_null :users, :some_column, false
|
570
|
+
remove_check_constraint :users, name: "users_some_column_null"
|
488
571
|
end
|
489
572
|
end
|
490
573
|
```
|
491
574
|
|
492
|
-
|
575
|
+
For Rails < 6.1, use:
|
493
576
|
|
494
577
|
```ruby
|
495
578
|
class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
@@ -498,7 +581,7 @@ class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
|
498
581
|
execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
|
499
582
|
end
|
500
583
|
|
501
|
-
# in Postgres 12+, you can safely
|
584
|
+
# in Postgres 12+, you can then safely set NOT NULL on the column
|
502
585
|
change_column_null :users, :some_column, false
|
503
586
|
safety_assured do
|
504
587
|
execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
|
@@ -507,8 +590,6 @@ class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
|
507
590
|
end
|
508
591
|
```
|
509
592
|
|
510
|
-
Note: This is not 100% the same as `NOT NULL` column constraint before Postgres 12. Here’s a [good explanation](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c).
|
511
|
-
|
512
593
|
### Keeping non-unique indexes to three columns or less
|
513
594
|
|
514
595
|
#### Bad
|
@@ -516,7 +597,7 @@ Note: This is not 100% the same as `NOT NULL` column constraint before Postgres
|
|
516
597
|
Adding a non-unique index with more than three columns rarely improves performance.
|
517
598
|
|
518
599
|
```ruby
|
519
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
600
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
520
601
|
def change
|
521
602
|
add_index :users, [:a, :b, :c, :d]
|
522
603
|
end
|
@@ -528,7 +609,7 @@ end
|
|
528
609
|
Instead, start an index with columns that narrow down the results the most.
|
529
610
|
|
530
611
|
```ruby
|
531
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
612
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
532
613
|
def change
|
533
614
|
add_index :users, [:b, :d]
|
534
615
|
end
|
@@ -542,7 +623,7 @@ For Postgres, be sure to add them concurrently.
|
|
542
623
|
To mark a step in the migration as safe, despite using a method that might otherwise be dangerous, wrap it in a `safety_assured` block.
|
543
624
|
|
544
625
|
```ruby
|
545
|
-
class MySafeMigration < ActiveRecord::Migration[
|
626
|
+
class MySafeMigration < ActiveRecord::Migration[7.0]
|
546
627
|
def change
|
547
628
|
safety_assured { remove_column :users, :some_column }
|
548
629
|
end
|
@@ -551,6 +632,21 @@ end
|
|
551
632
|
|
552
633
|
Certain methods like `execute` and `change_table` cannot be inspected and are prevented from running by default. Make sure what you’re doing is really safe and use this pattern.
|
553
634
|
|
635
|
+
## Safe by Default
|
636
|
+
|
637
|
+
Make operations safe by default.
|
638
|
+
|
639
|
+
- adding and removing an index
|
640
|
+
- adding a foreign key
|
641
|
+
- adding a check constraint
|
642
|
+
- setting NOT NULL on an existing column
|
643
|
+
|
644
|
+
Add to `config/initializers/strong_migrations.rb`:
|
645
|
+
|
646
|
+
```ruby
|
647
|
+
StrongMigrations.safe_by_default = true
|
648
|
+
```
|
649
|
+
|
554
650
|
## Custom Checks
|
555
651
|
|
556
652
|
Add your own custom checks with:
|
@@ -587,6 +683,14 @@ StrongMigrations.disable_check(:add_index)
|
|
587
683
|
|
588
684
|
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
|
589
685
|
|
686
|
+
## Down Migrations / Rollbacks
|
687
|
+
|
688
|
+
By default, checks are disabled when migrating down. Enable them with:
|
689
|
+
|
690
|
+
```ruby
|
691
|
+
StrongMigrations.check_down = true
|
692
|
+
```
|
693
|
+
|
590
694
|
## Custom Messages
|
591
695
|
|
592
696
|
To customize specific messages, create an initializer with:
|
@@ -595,9 +699,9 @@ To customize specific messages, create an initializer with:
|
|
595
699
|
StrongMigrations.error_messages[:add_column_default] = "Your custom instructions"
|
596
700
|
```
|
597
701
|
|
598
|
-
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
|
702
|
+
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations/error_messages.rb) for the list of keys.
|
599
703
|
|
600
|
-
## Timeouts
|
704
|
+
## Migration Timeouts
|
601
705
|
|
602
706
|
It’s extremely important to set a short lock timeout for migrations. This way, if a migration can’t acquire a lock in a timely manner, other statements won’t be stuck behind it. We also recommend setting a long statement timeout so migrations can run for a while.
|
603
707
|
|
@@ -617,6 +721,68 @@ ALTER ROLE myuser SET statement_timeout = '1h';
|
|
617
721
|
|
618
722
|
Note: If you use PgBouncer in transaction mode, you must set timeouts on the database user.
|
619
723
|
|
724
|
+
## Lock Timeout Retries [experimental]
|
725
|
+
|
726
|
+
There’s the option to automatically retry statements when the lock timeout is reached. Here’s how it works:
|
727
|
+
|
728
|
+
- If a lock timeout happens outside a transaction, the statement is retried
|
729
|
+
- If it happens inside the DDL transaction, the entire migration is retried (only applicable to Postgres)
|
730
|
+
|
731
|
+
Add to `config/initializers/strong_migrations.rb`:
|
732
|
+
|
733
|
+
```ruby
|
734
|
+
StrongMigrations.lock_timeout_retries = 3
|
735
|
+
```
|
736
|
+
|
737
|
+
Set the delay between retries with:
|
738
|
+
|
739
|
+
```ruby
|
740
|
+
StrongMigrations.lock_timeout_retry_delay = 10.seconds
|
741
|
+
```
|
742
|
+
|
743
|
+
## App Timeouts
|
744
|
+
|
745
|
+
We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.
|
746
|
+
|
747
|
+
For Postgres:
|
748
|
+
|
749
|
+
```yml
|
750
|
+
production:
|
751
|
+
connect_timeout: 5
|
752
|
+
variables:
|
753
|
+
statement_timeout: 15s
|
754
|
+
lock_timeout: 10s
|
755
|
+
```
|
756
|
+
|
757
|
+
Note: If you use PgBouncer in transaction mode, you must set the statement and lock timeouts on the database user as shown above.
|
758
|
+
|
759
|
+
For MySQL:
|
760
|
+
|
761
|
+
```yml
|
762
|
+
production:
|
763
|
+
connect_timeout: 5
|
764
|
+
read_timeout: 5
|
765
|
+
write_timeout: 5
|
766
|
+
variables:
|
767
|
+
max_execution_time: 15000 # ms
|
768
|
+
lock_wait_timeout: 10 # sec
|
769
|
+
|
770
|
+
```
|
771
|
+
|
772
|
+
For MariaDB:
|
773
|
+
|
774
|
+
```yml
|
775
|
+
production:
|
776
|
+
connect_timeout: 5
|
777
|
+
read_timeout: 5
|
778
|
+
write_timeout: 5
|
779
|
+
variables:
|
780
|
+
max_statement_time: 15 # sec
|
781
|
+
lock_wait_timeout: 10 # sec
|
782
|
+
```
|
783
|
+
|
784
|
+
For HTTP connections, Redis, and other services, check out [this guide](https://github.com/ankane/the-ultimate-guide-to-ruby-timeouts).
|
785
|
+
|
620
786
|
## Existing Migrations
|
621
787
|
|
622
788
|
To mark migrations as safe that were created before installing this gem, create an initializer with:
|
@@ -632,11 +798,11 @@ Use the version from your latest migration.
|
|
632
798
|
If your development database version is different from production, you can specify the production version so the right checks run in development.
|
633
799
|
|
634
800
|
```ruby
|
635
|
-
StrongMigrations.
|
636
|
-
StrongMigrations.target_mysql_version = "8.0.12"
|
637
|
-
StrongMigrations.target_mariadb_version = "10.3.2"
|
801
|
+
StrongMigrations.target_version = 10 # or "8.0.12", "10.3.2", etc
|
638
802
|
```
|
639
803
|
|
804
|
+
The major version works well for Postgres, while the full version is recommended for MySQL and MariaDB.
|
805
|
+
|
640
806
|
For safety, this option only affects development and test environments. In other environments, the actual server version is always used.
|
641
807
|
|
642
808
|
## Analyze Tables
|
@@ -649,11 +815,15 @@ StrongMigrations.auto_analyze = true
|
|
649
815
|
|
650
816
|
## Faster Migrations
|
651
817
|
|
652
|
-
Only dump the schema when adding a new migration. If you use Git,
|
818
|
+
Only dump the schema when adding a new migration. If you use Git, add to the end of your `Rakefile`:
|
653
819
|
|
654
|
-
```
|
655
|
-
|
656
|
-
|
820
|
+
```rb
|
821
|
+
task :faster_migrations do
|
822
|
+
ActiveRecord::Base.dump_schema_after_migration = Rails.env.development? &&
|
823
|
+
`git status db/migrate/ --porcelain`.present?
|
824
|
+
end
|
825
|
+
|
826
|
+
task "db:migrate": "faster_migrations"
|
657
827
|
```
|
658
828
|
|
659
829
|
## Schema Sanity
|
@@ -664,22 +834,20 @@ Columns can flip order in `db/schema.rb` when you have multiple developers. One
|
|
664
834
|
task "db:schema:dump": "strong_migrations:alphabetize_columns"
|
665
835
|
```
|
666
836
|
|
667
|
-
## Dangerous Tasks
|
668
|
-
|
669
|
-
For safety, dangerous database tasks are disabled in production - `db:drop`, `db:reset`, `db:schema:load`, and `db:structure:load`. To get around this, use:
|
670
|
-
|
671
|
-
```sh
|
672
|
-
SAFETY_ASSURED=1 rails db:drop
|
673
|
-
```
|
674
|
-
|
675
837
|
## Permissions
|
676
838
|
|
677
839
|
We recommend using a [separate database user](https://ankane.org/postgres-users) for migrations when possible so you don’t need to grant your app user permission to alter tables.
|
678
840
|
|
841
|
+
## Smaller Projects
|
842
|
+
|
843
|
+
You probably don’t need this gem for smaller projects, as operations that are unsafe at scale can be perfectly safe on smaller, low-traffic tables.
|
844
|
+
|
679
845
|
## Additional Reading
|
680
846
|
|
681
847
|
- [Rails Migrations with No Downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
|
682
848
|
- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/braintree-product-technology/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
|
849
|
+
- [An Overview of DDL Algorithms in MySQL](https://mydbops.wordpress.com/2020/03/04/an-overview-of-ddl-algorithms-in-mysql-covers-mysql-8/)
|
850
|
+
- [MariaDB InnoDB Online DDL Overview](https://mariadb.com/kb/en/innodb-online-ddl-overview/)
|
683
851
|
|
684
852
|
## Credits
|
685
853
|
|