strong_migrations 0.6.8 → 1.6.1
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +141 -0
- data/LICENSE.txt +1 -1
- data/README.md +349 -108
- data/lib/generators/strong_migrations/install_generator.rb +29 -0
- data/lib/generators/strong_migrations/templates/initializer.rb.tt +10 -3
- data/lib/strong_migrations/adapters/abstract_adapter.rb +77 -0
- data/lib/strong_migrations/adapters/mariadb_adapter.rb +32 -0
- data/lib/strong_migrations/adapters/mysql_adapter.rb +107 -0
- data/lib/strong_migrations/adapters/postgresql_adapter.rb +230 -0
- data/lib/strong_migrations/checker.rb +125 -403
- data/lib/strong_migrations/checks.rb +483 -0
- data/lib/strong_migrations/database_tasks.rb +2 -1
- data/lib/strong_migrations/error_messages.rb +250 -0
- data/lib/strong_migrations/migration.rb +10 -2
- data/lib/strong_migrations/migrator.rb +19 -0
- data/lib/strong_migrations/railtie.rb +0 -4
- data/lib/strong_migrations/safe_methods.rb +128 -0
- data/lib/strong_migrations/schema_dumper.rb +21 -0
- data/lib/strong_migrations/version.rb +1 -1
- data/lib/strong_migrations.rb +40 -185
- data/lib/tasks/strong_migrations.rake +2 -13
- metadata +20 -82
- data/lib/strong_migrations/alphabetize_columns.rb +0 -11
data/README.md
CHANGED
@@ -4,18 +4,18 @@ Catch unsafe migrations in development
|
|
4
4
|
|
5
5
|
✓ Detects potentially dangerous operations<br /> ✓ Prevents them from running by default<br /> ✓ Provides instructions on safer ways to do what you want
|
6
6
|
|
7
|
-
Supports
|
7
|
+
Supports PostgreSQL, MySQL, and MariaDB
|
8
8
|
|
9
9
|
:tangerine: Battle-tested at [Instacart](https://www.instacart.com/opensource)
|
10
10
|
|
11
|
-
[![Build Status](https://
|
11
|
+
[![Build Status](https://github.com/ankane/strong_migrations/workflows/build/badge.svg?branch=master)](https://github.com/ankane/strong_migrations/actions)
|
12
12
|
|
13
13
|
## Installation
|
14
14
|
|
15
15
|
Add this line to your application’s Gemfile:
|
16
16
|
|
17
17
|
```ruby
|
18
|
-
gem
|
18
|
+
gem "strong_migrations"
|
19
19
|
```
|
20
20
|
|
21
21
|
And run:
|
@@ -25,6 +25,36 @@ bundle install
|
|
25
25
|
rails generate strong_migrations:install
|
26
26
|
```
|
27
27
|
|
28
|
+
Strong Migrations sets a long statement timeout for migrations so you can set a [short statement timeout](#app-timeouts) for your application.
|
29
|
+
|
30
|
+
## How It Works
|
31
|
+
|
32
|
+
When you run a migration that’s potentially dangerous, you’ll see an error message like:
|
33
|
+
|
34
|
+
```txt
|
35
|
+
=== Dangerous operation detected #strong_migrations ===
|
36
|
+
|
37
|
+
Active Record caches attributes, which causes problems
|
38
|
+
when removing columns. Be sure to ignore the column:
|
39
|
+
|
40
|
+
class User < ApplicationRecord
|
41
|
+
self.ignored_columns = ["name"]
|
42
|
+
end
|
43
|
+
|
44
|
+
Deploy the code, then wrap this step in a safety_assured { ... } block.
|
45
|
+
|
46
|
+
class RemoveColumn < ActiveRecord::Migration[7.0]
|
47
|
+
def change
|
48
|
+
safety_assured { remove_column :users, :name }
|
49
|
+
end
|
50
|
+
end
|
51
|
+
```
|
52
|
+
|
53
|
+
An operation is classified as dangerous if it either:
|
54
|
+
|
55
|
+
- Blocks reads or writes for more than a few seconds (after a lock is acquired)
|
56
|
+
- Has a good chance of causing application errors
|
57
|
+
|
28
58
|
## Checks
|
29
59
|
|
30
60
|
Potentially dangerous operations:
|
@@ -32,11 +62,12 @@ Potentially dangerous operations:
|
|
32
62
|
- [removing a column](#removing-a-column)
|
33
63
|
- [adding a column with a default value](#adding-a-column-with-a-default-value)
|
34
64
|
- [backfilling data](#backfilling-data)
|
35
|
-
- [
|
36
|
-
- [
|
65
|
+
- [adding a stored generated column](#adding-a-stored-generated-column)
|
66
|
+
- [changing the type of a column](#changing-the-type-of-a-column)
|
67
|
+
- [renaming a column](#renaming-a-column)
|
37
68
|
- [renaming a table](#renaming-a-table)
|
38
69
|
- [creating a table with the force option](#creating-a-table-with-the-force-option)
|
39
|
-
- [
|
70
|
+
- [adding a check constraint](#adding-a-check-constraint)
|
40
71
|
- [executing SQL directly](#executing-SQL-directly)
|
41
72
|
|
42
73
|
Postgres-specific checks:
|
@@ -44,9 +75,14 @@ Postgres-specific checks:
|
|
44
75
|
- [adding an index non-concurrently](#adding-an-index-non-concurrently)
|
45
76
|
- [adding a reference](#adding-a-reference)
|
46
77
|
- [adding a foreign key](#adding-a-foreign-key)
|
78
|
+
- [adding an exclusion constraint](#adding-an-exclusion-constraint)
|
47
79
|
- [adding a json column](#adding-a-json-column)
|
48
80
|
- [setting NOT NULL on an existing column](#setting-not-null-on-an-existing-column)
|
49
81
|
|
82
|
+
Config-specific checks:
|
83
|
+
|
84
|
+
- [changing the default value of a column](#changing-the-default-value-of-a-column)
|
85
|
+
|
50
86
|
Best practices:
|
51
87
|
|
52
88
|
- [keeping non-unique indexes to three columns or less](#keeping-non-unique-indexes-to-three-columns-or-less)
|
@@ -57,10 +93,10 @@ You can also add [custom checks](#custom-checks) or [disable specific checks](#d
|
|
57
93
|
|
58
94
|
#### Bad
|
59
95
|
|
60
|
-
|
96
|
+
Active Record caches database columns at runtime, so if you drop a column, it can cause exceptions until your app reboots.
|
61
97
|
|
62
98
|
```ruby
|
63
|
-
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[
|
99
|
+
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[7.0]
|
64
100
|
def change
|
65
101
|
remove_column :users, :some_column
|
66
102
|
end
|
@@ -69,7 +105,7 @@ end
|
|
69
105
|
|
70
106
|
#### Good
|
71
107
|
|
72
|
-
1. Tell
|
108
|
+
1. Tell Active Record to ignore the column from its cache
|
73
109
|
|
74
110
|
```ruby
|
75
111
|
class User < ApplicationRecord
|
@@ -77,41 +113,42 @@ end
|
|
77
113
|
end
|
78
114
|
```
|
79
115
|
|
80
|
-
2. Deploy code
|
116
|
+
2. Deploy the code
|
81
117
|
3. Write a migration to remove the column (wrap in `safety_assured` block)
|
82
118
|
|
83
119
|
```ruby
|
84
|
-
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[
|
120
|
+
class RemoveSomeColumnFromUsers < ActiveRecord::Migration[7.0]
|
85
121
|
def change
|
86
122
|
safety_assured { remove_column :users, :some_column }
|
87
123
|
end
|
88
124
|
end
|
89
125
|
```
|
90
126
|
|
91
|
-
4. Deploy and run migration
|
127
|
+
4. Deploy and run the migration
|
128
|
+
5. Remove the line added in step 1
|
92
129
|
|
93
130
|
### Adding a column with a default value
|
94
131
|
|
95
|
-
Note: This operation is safe in Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+.
|
96
|
-
|
97
132
|
#### Bad
|
98
133
|
|
99
|
-
|
134
|
+
In earlier versions of Postgres, MySQL, and MariaDB, adding a column with a default value to an existing table causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
100
135
|
|
101
136
|
```ruby
|
102
|
-
class AddSomeColumnToUsers < ActiveRecord::Migration[
|
137
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
103
138
|
def change
|
104
139
|
add_column :users, :some_column, :text, default: "default_value"
|
105
140
|
end
|
106
141
|
end
|
107
142
|
```
|
108
143
|
|
144
|
+
In Postgres 11+, MySQL 8.0.12+, and MariaDB 10.3.2+, this no longer requires a table rewrite and is safe (except for volatile functions like `gen_random_uuid()`).
|
145
|
+
|
109
146
|
#### Good
|
110
147
|
|
111
148
|
Instead, add the column without a default value, then change the default.
|
112
149
|
|
113
150
|
```ruby
|
114
|
-
class AddSomeColumnToUsers < ActiveRecord::Migration[
|
151
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
115
152
|
def up
|
116
153
|
add_column :users, :some_column, :text
|
117
154
|
change_column_default :users, :some_column, "default_value"
|
@@ -129,10 +166,10 @@ See the next section for how to backfill.
|
|
129
166
|
|
130
167
|
#### Bad
|
131
168
|
|
132
|
-
|
169
|
+
Active Record creates a transaction around each migration, and backfilling in the same transaction that alters a table keeps the table locked for the [duration of the backfill](https://wework.github.io/data/2015/11/05/add-columns-with-default-values-to-large-tables-in-rails-postgres/).
|
133
170
|
|
134
171
|
```ruby
|
135
|
-
class AddSomeColumnToUsers < ActiveRecord::Migration[
|
172
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
136
173
|
def change
|
137
174
|
add_column :users, :some_column, :text
|
138
175
|
User.update_all some_column: "default_value"
|
@@ -147,7 +184,7 @@ Also, running a single query to update data can cause issues for large tables.
|
|
147
184
|
There are three keys to backfilling safely: batching, throttling, and running it outside a transaction. Use the Rails console or a separate migration with `disable_ddl_transaction!`.
|
148
185
|
|
149
186
|
```ruby
|
150
|
-
class BackfillSomeColumn < ActiveRecord::Migration[
|
187
|
+
class BackfillSomeColumn < ActiveRecord::Migration[7.0]
|
151
188
|
disable_ddl_transaction!
|
152
189
|
|
153
190
|
def up
|
@@ -159,39 +196,83 @@ class BackfillSomeColumn < ActiveRecord::Migration[6.0]
|
|
159
196
|
end
|
160
197
|
```
|
161
198
|
|
162
|
-
###
|
199
|
+
### Adding a stored generated column
|
163
200
|
|
164
201
|
#### Bad
|
165
202
|
|
203
|
+
Adding a stored generated column causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
204
|
+
|
166
205
|
```ruby
|
167
|
-
class
|
206
|
+
class AddSomeColumnToUsers < ActiveRecord::Migration[7.0]
|
168
207
|
def change
|
169
|
-
|
208
|
+
add_column :users, :some_column, :virtual, type: :string, as: "...", stored: true
|
170
209
|
end
|
171
210
|
end
|
172
211
|
```
|
173
212
|
|
174
|
-
|
213
|
+
#### Good
|
214
|
+
|
215
|
+
Add a non-generated column and use callbacks or triggers instead (or a virtual generated column with MySQL and MariaDB).
|
216
|
+
|
217
|
+
### Changing the type of a column
|
218
|
+
|
219
|
+
#### Bad
|
220
|
+
|
221
|
+
Changing the type of a column causes the entire table to be rewritten. During this time, reads and writes are blocked in Postgres, and writes are blocked in MySQL and MariaDB.
|
175
222
|
|
176
223
|
```ruby
|
177
|
-
class ChangeSomeColumnType < ActiveRecord::Migration[
|
224
|
+
class ChangeSomeColumnType < ActiveRecord::Migration[7.0]
|
178
225
|
def change
|
179
226
|
change_column :users, :some_column, :new_type
|
180
227
|
end
|
181
228
|
end
|
182
229
|
```
|
183
230
|
|
184
|
-
|
231
|
+
Some changes don’t require a table rewrite and are safe in Postgres:
|
185
232
|
|
186
|
-
|
187
|
-
|
188
|
-
|
189
|
-
|
233
|
+
Type | Safe Changes
|
234
|
+
--- | ---
|
235
|
+
`cidr` | Changing to `inet`
|
236
|
+
`citext` | Changing to `text` if not indexed, changing to `string` with no `:limit` if not indexed
|
237
|
+
`datetime` | Increasing or removing `:precision`, changing to `timestamptz` when session time zone is UTC in Postgres 12+
|
238
|
+
`decimal` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
|
239
|
+
`interval` | Increasing or removing `:precision`
|
240
|
+
`numeric` | Increasing `:precision` at same `:scale`, removing `:precision` and `:scale`
|
241
|
+
`string` | Increasing or removing `:limit`, changing to `text`, changing `citext` if not indexed
|
242
|
+
`text` | Changing to `string` with no `:limit`, changing to `citext` if not indexed
|
243
|
+
`time` | Increasing or removing `:precision`
|
244
|
+
`timestamptz` | Increasing or removing `:limit`, changing to `datetime` when session time zone is UTC in Postgres 12+
|
190
245
|
|
191
|
-
And
|
246
|
+
And some in MySQL and MariaDB:
|
192
247
|
|
193
|
-
|
194
|
-
|
248
|
+
Type | Safe Changes
|
249
|
+
--- | ---
|
250
|
+
`string` | Increasing `:limit` from under 63 up to 63, increasing `:limit` from over 63 to the max (the threshold can be different if using an encoding other than `utf8mb4` - for instance, it’s 85 for `utf8mb3` and 255 for `latin1`)
|
251
|
+
|
252
|
+
#### Good
|
253
|
+
|
254
|
+
A safer approach is to:
|
255
|
+
|
256
|
+
1. Create a new column
|
257
|
+
2. Write to both columns
|
258
|
+
3. Backfill data from the old column to the new column
|
259
|
+
4. Move reads from the old column to the new column
|
260
|
+
5. Stop writing to the old column
|
261
|
+
6. Drop the old column
|
262
|
+
|
263
|
+
### Renaming a column
|
264
|
+
|
265
|
+
#### Bad
|
266
|
+
|
267
|
+
Renaming a column that’s in use will cause errors in your application.
|
268
|
+
|
269
|
+
```ruby
|
270
|
+
class RenameSomeColumn < ActiveRecord::Migration[7.0]
|
271
|
+
def change
|
272
|
+
rename_column :users, :some_column, :new_name
|
273
|
+
end
|
274
|
+
end
|
275
|
+
```
|
195
276
|
|
196
277
|
#### Good
|
197
278
|
|
@@ -208,8 +289,10 @@ A safer approach is to:
|
|
208
289
|
|
209
290
|
#### Bad
|
210
291
|
|
292
|
+
Renaming a table that’s in use will cause errors in your application.
|
293
|
+
|
211
294
|
```ruby
|
212
|
-
class RenameUsersToCustomers < ActiveRecord::Migration[
|
295
|
+
class RenameUsersToCustomers < ActiveRecord::Migration[7.0]
|
213
296
|
def change
|
214
297
|
rename_table :users, :customers
|
215
298
|
end
|
@@ -234,7 +317,7 @@ A safer approach is to:
|
|
234
317
|
The `force` option can drop an existing table.
|
235
318
|
|
236
319
|
```ruby
|
237
|
-
class CreateUsers < ActiveRecord::Migration[
|
320
|
+
class CreateUsers < ActiveRecord::Migration[7.0]
|
238
321
|
def change
|
239
322
|
create_table :users, force: true do |t|
|
240
323
|
# ...
|
@@ -248,7 +331,7 @@ end
|
|
248
331
|
Create tables without the `force` option.
|
249
332
|
|
250
333
|
```ruby
|
251
|
-
class CreateUsers < ActiveRecord::Migration[
|
334
|
+
class CreateUsers < ActiveRecord::Migration[7.0]
|
252
335
|
def change
|
253
336
|
create_table :users do |t|
|
254
337
|
# ...
|
@@ -259,40 +342,54 @@ end
|
|
259
342
|
|
260
343
|
If you intend to drop an existing table, run `drop_table` first.
|
261
344
|
|
262
|
-
###
|
345
|
+
### Adding a check constraint
|
346
|
+
|
347
|
+
:turtle: Safe by default available
|
263
348
|
|
264
349
|
#### Bad
|
265
350
|
|
266
|
-
|
351
|
+
Adding a check constraint blocks reads and writes in Postgres and blocks writes in MySQL and MariaDB while every row is checked.
|
267
352
|
|
268
353
|
```ruby
|
269
|
-
class
|
354
|
+
class AddCheckConstraint < ActiveRecord::Migration[7.0]
|
270
355
|
def change
|
271
|
-
|
356
|
+
add_check_constraint :users, "price > 0", name: "price_check"
|
272
357
|
end
|
273
358
|
end
|
274
359
|
```
|
275
360
|
|
276
|
-
#### Good
|
361
|
+
#### Good - Postgres
|
277
362
|
|
278
|
-
|
363
|
+
Add the check constraint without validating existing rows:
|
279
364
|
|
280
365
|
```ruby
|
281
|
-
class
|
366
|
+
class AddCheckConstraint < ActiveRecord::Migration[7.0]
|
282
367
|
def change
|
283
|
-
|
368
|
+
add_check_constraint :users, "price > 0", name: "price_check", validate: false
|
284
369
|
end
|
285
370
|
end
|
286
371
|
```
|
287
372
|
|
288
|
-
|
373
|
+
Then validate them in a separate migration.
|
374
|
+
|
375
|
+
```ruby
|
376
|
+
class ValidateCheckConstraint < ActiveRecord::Migration[7.0]
|
377
|
+
def change
|
378
|
+
validate_check_constraint :users, name: "price_check"
|
379
|
+
end
|
380
|
+
end
|
381
|
+
```
|
382
|
+
|
383
|
+
#### Good - MySQL and MariaDB
|
384
|
+
|
385
|
+
[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this (check constraints can be added with `NOT ENFORCED`, but enforcing blocks writes).
|
289
386
|
|
290
387
|
### Executing SQL directly
|
291
388
|
|
292
389
|
Strong Migrations can’t ensure safety for raw SQL statements. Make really sure that what you’re doing is safe, then use:
|
293
390
|
|
294
391
|
```ruby
|
295
|
-
class ExecuteSQL < ActiveRecord::Migration[
|
392
|
+
class ExecuteSQL < ActiveRecord::Migration[7.0]
|
296
393
|
def change
|
297
394
|
safety_assured { execute "..." }
|
298
395
|
end
|
@@ -301,12 +398,14 @@ end
|
|
301
398
|
|
302
399
|
### Adding an index non-concurrently
|
303
400
|
|
401
|
+
:turtle: Safe by default available
|
402
|
+
|
304
403
|
#### Bad
|
305
404
|
|
306
|
-
In Postgres, adding an index non-concurrently
|
405
|
+
In Postgres, adding an index non-concurrently blocks writes.
|
307
406
|
|
308
407
|
```ruby
|
309
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
408
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
310
409
|
def change
|
311
410
|
add_index :users, :some_column
|
312
411
|
end
|
@@ -318,7 +417,7 @@ end
|
|
318
417
|
Add indexes concurrently.
|
319
418
|
|
320
419
|
```ruby
|
321
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
420
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
322
421
|
disable_ddl_transaction!
|
323
422
|
|
324
423
|
def change
|
@@ -337,12 +436,14 @@ rails g index table column
|
|
337
436
|
|
338
437
|
### Adding a reference
|
339
438
|
|
439
|
+
:turtle: Safe by default available
|
440
|
+
|
340
441
|
#### Bad
|
341
442
|
|
342
|
-
Rails adds an index non-concurrently to references by default, which
|
443
|
+
Rails adds an index non-concurrently to references by default, which blocks writes in Postgres.
|
343
444
|
|
344
445
|
```ruby
|
345
|
-
class AddReferenceToUsers < ActiveRecord::Migration[
|
446
|
+
class AddReferenceToUsers < ActiveRecord::Migration[7.0]
|
346
447
|
def change
|
347
448
|
add_reference :users, :city
|
348
449
|
end
|
@@ -354,7 +455,7 @@ end
|
|
354
455
|
Make sure the index is added concurrently.
|
355
456
|
|
356
457
|
```ruby
|
357
|
-
class AddReferenceToUsers < ActiveRecord::Migration[
|
458
|
+
class AddReferenceToUsers < ActiveRecord::Migration[7.0]
|
358
459
|
disable_ddl_transaction!
|
359
460
|
|
360
461
|
def change
|
@@ -365,12 +466,14 @@ end
|
|
365
466
|
|
366
467
|
### Adding a foreign key
|
367
468
|
|
469
|
+
:turtle: Safe by default available
|
470
|
+
|
368
471
|
#### Bad
|
369
472
|
|
370
|
-
In Postgres,
|
473
|
+
In Postgres, adding a foreign key blocks writes on both tables.
|
371
474
|
|
372
475
|
```ruby
|
373
|
-
class AddForeignKeyOnUsers < ActiveRecord::Migration[
|
476
|
+
class AddForeignKeyOnUsers < ActiveRecord::Migration[7.0]
|
374
477
|
def change
|
375
478
|
add_foreign_key :users, :orders
|
376
479
|
end
|
@@ -380,7 +483,7 @@ end
|
|
380
483
|
or
|
381
484
|
|
382
485
|
```ruby
|
383
|
-
class AddReferenceToUsers < ActiveRecord::Migration[
|
486
|
+
class AddReferenceToUsers < ActiveRecord::Migration[7.0]
|
384
487
|
def change
|
385
488
|
add_reference :users, :order, foreign_key: true
|
386
489
|
end
|
@@ -389,60 +492,52 @@ end
|
|
389
492
|
|
390
493
|
#### Good
|
391
494
|
|
392
|
-
|
393
|
-
|
394
|
-
For Rails 5.2+, use:
|
495
|
+
Add the foreign key without validating existing rows:
|
395
496
|
|
396
497
|
```ruby
|
397
|
-
class AddForeignKeyOnUsers < ActiveRecord::Migration[
|
498
|
+
class AddForeignKeyOnUsers < ActiveRecord::Migration[7.0]
|
398
499
|
def change
|
399
500
|
add_foreign_key :users, :orders, validate: false
|
400
501
|
end
|
401
502
|
end
|
402
503
|
```
|
403
504
|
|
404
|
-
Then validate
|
505
|
+
Then validate them in a separate migration.
|
405
506
|
|
406
507
|
```ruby
|
407
|
-
class ValidateForeignKeyOnUsers < ActiveRecord::Migration[
|
508
|
+
class ValidateForeignKeyOnUsers < ActiveRecord::Migration[7.0]
|
408
509
|
def change
|
409
510
|
validate_foreign_key :users, :orders
|
410
511
|
end
|
411
512
|
end
|
412
513
|
```
|
413
514
|
|
414
|
-
|
515
|
+
### Adding an exclusion constraint
|
415
516
|
|
416
|
-
|
417
|
-
class AddForeignKeyOnUsers < ActiveRecord::Migration[5.1]
|
418
|
-
def change
|
419
|
-
safety_assured do
|
420
|
-
execute 'ALTER TABLE "users" ADD CONSTRAINT "fk_rails_c1e9b98e31" FOREIGN KEY ("order_id") REFERENCES "orders" ("id") NOT VALID'
|
421
|
-
end
|
422
|
-
end
|
423
|
-
end
|
424
|
-
```
|
517
|
+
#### Bad
|
425
518
|
|
426
|
-
|
519
|
+
In Postgres, adding an exclusion constraint blocks reads and writes while every row is checked.
|
427
520
|
|
428
521
|
```ruby
|
429
|
-
class
|
522
|
+
class AddExclusionContraint < ActiveRecord::Migration[7.1]
|
430
523
|
def change
|
431
|
-
|
432
|
-
execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "fk_rails_c1e9b98e31"'
|
433
|
-
end
|
524
|
+
add_exclusion_constraint :users, "number WITH =", using: :gist
|
434
525
|
end
|
435
526
|
end
|
436
527
|
```
|
437
528
|
|
529
|
+
#### Good
|
530
|
+
|
531
|
+
[Let us know](https://github.com/ankane/strong_migrations/issues/new) if you have a safe way to do this (exclusion constraints cannot be marked `NOT VALID`).
|
532
|
+
|
438
533
|
### Adding a json column
|
439
534
|
|
440
535
|
#### Bad
|
441
536
|
|
442
|
-
In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries.
|
537
|
+
In Postgres, there’s no equality operator for the `json` column type, which can cause errors for existing `SELECT DISTINCT` queries in your application.
|
443
538
|
|
444
539
|
```ruby
|
445
|
-
class AddPropertiesToUsers < ActiveRecord::Migration[
|
540
|
+
class AddPropertiesToUsers < ActiveRecord::Migration[7.0]
|
446
541
|
def change
|
447
542
|
add_column :users, :properties, :json
|
448
543
|
end
|
@@ -454,7 +549,7 @@ end
|
|
454
549
|
Use `jsonb` instead.
|
455
550
|
|
456
551
|
```ruby
|
457
|
-
class AddPropertiesToUsers < ActiveRecord::Migration[
|
552
|
+
class AddPropertiesToUsers < ActiveRecord::Migration[7.0]
|
458
553
|
def change
|
459
554
|
add_column :users, :properties, :jsonb
|
460
555
|
end
|
@@ -463,12 +558,14 @@ end
|
|
463
558
|
|
464
559
|
### Setting NOT NULL on an existing column
|
465
560
|
|
561
|
+
:turtle: Safe by default available
|
562
|
+
|
466
563
|
#### Bad
|
467
564
|
|
468
|
-
In Postgres, setting `NOT NULL` on an existing column
|
565
|
+
In Postgres, setting `NOT NULL` on an existing column blocks reads and writes while every row is checked.
|
469
566
|
|
470
567
|
```ruby
|
471
|
-
class SetSomeColumnNotNull < ActiveRecord::Migration[
|
568
|
+
class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
|
472
569
|
def change
|
473
570
|
change_column_null :users, :some_column, false
|
474
571
|
end
|
@@ -477,7 +574,19 @@ end
|
|
477
574
|
|
478
575
|
#### Good
|
479
576
|
|
480
|
-
Instead, add a constraint
|
577
|
+
Instead, add a check constraint.
|
578
|
+
|
579
|
+
For Rails 6.1, use:
|
580
|
+
|
581
|
+
```ruby
|
582
|
+
class SetSomeColumnNotNull < ActiveRecord::Migration[7.0]
|
583
|
+
def change
|
584
|
+
add_check_constraint :users, "some_column IS NOT NULL", name: "users_some_column_null", validate: false
|
585
|
+
end
|
586
|
+
end
|
587
|
+
```
|
588
|
+
|
589
|
+
For Rails < 6.1, use:
|
481
590
|
|
482
591
|
```ruby
|
483
592
|
class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
@@ -489,7 +598,23 @@ class SetSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
|
489
598
|
end
|
490
599
|
```
|
491
600
|
|
492
|
-
Then validate it in a separate migration.
|
601
|
+
Then validate it in a separate migration. A `NOT NULL` check constraint is [functionally equivalent](https://medium.com/doctolib/adding-a-not-null-constraint-on-pg-faster-with-minimal-locking-38b2c00c4d1c) to setting `NOT NULL` on the column (but it won’t show up in `schema.rb` in Rails < 6.1). In Postgres 12+, once the check constraint is validated, you can safely set `NOT NULL` on the column and drop the check constraint.
|
602
|
+
|
603
|
+
For Rails 6.1, use:
|
604
|
+
|
605
|
+
```ruby
|
606
|
+
class ValidateSomeColumnNotNull < ActiveRecord::Migration[7.0]
|
607
|
+
def change
|
608
|
+
validate_check_constraint :users, name: "users_some_column_null"
|
609
|
+
|
610
|
+
# in Postgres 12+, you can then safely set NOT NULL on the column
|
611
|
+
change_column_null :users, :some_column, false
|
612
|
+
remove_check_constraint :users, name: "users_some_column_null"
|
613
|
+
end
|
614
|
+
end
|
615
|
+
```
|
616
|
+
|
617
|
+
For Rails < 6.1, use:
|
493
618
|
|
494
619
|
```ruby
|
495
620
|
class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
@@ -498,7 +623,7 @@ class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
|
498
623
|
execute 'ALTER TABLE "users" VALIDATE CONSTRAINT "users_some_column_null"'
|
499
624
|
end
|
500
625
|
|
501
|
-
# in Postgres 12+, you can safely
|
626
|
+
# in Postgres 12+, you can then safely set NOT NULL on the column
|
502
627
|
change_column_null :users, :some_column, false
|
503
628
|
safety_assured do
|
504
629
|
execute 'ALTER TABLE "users" DROP CONSTRAINT "users_some_column_null"'
|
@@ -507,7 +632,35 @@ class ValidateSomeColumnNotNull < ActiveRecord::Migration[6.0]
|
|
507
632
|
end
|
508
633
|
```
|
509
634
|
|
510
|
-
|
635
|
+
### Changing the default value of a column
|
636
|
+
|
637
|
+
#### Bad
|
638
|
+
|
639
|
+
Rails < 7 enables partial writes by default, which can cause incorrect values to be inserted when changing the default value of a column.
|
640
|
+
|
641
|
+
```ruby
|
642
|
+
class ChangeSomeColumnDefault < ActiveRecord::Migration[6.1]
|
643
|
+
def change
|
644
|
+
change_column_default :users, :some_column, from: "old", to: "new"
|
645
|
+
end
|
646
|
+
end
|
647
|
+
|
648
|
+
User.create!(some_column: "old") # can insert "new"
|
649
|
+
```
|
650
|
+
|
651
|
+
#### Good
|
652
|
+
|
653
|
+
Disable partial writes in `config/application.rb`. For Rails < 7, use:
|
654
|
+
|
655
|
+
```ruby
|
656
|
+
config.active_record.partial_writes = false
|
657
|
+
```
|
658
|
+
|
659
|
+
For Rails 7, use:
|
660
|
+
|
661
|
+
```ruby
|
662
|
+
config.active_record.partial_inserts = false
|
663
|
+
```
|
511
664
|
|
512
665
|
### Keeping non-unique indexes to three columns or less
|
513
666
|
|
@@ -516,7 +669,7 @@ Note: This is not 100% the same as `NOT NULL` column constraint before Postgres
|
|
516
669
|
Adding a non-unique index with more than three columns rarely improves performance.
|
517
670
|
|
518
671
|
```ruby
|
519
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
672
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
520
673
|
def change
|
521
674
|
add_index :users, [:a, :b, :c, :d]
|
522
675
|
end
|
@@ -528,7 +681,7 @@ end
|
|
528
681
|
Instead, start an index with columns that narrow down the results the most.
|
529
682
|
|
530
683
|
```ruby
|
531
|
-
class AddSomeIndexToUsers < ActiveRecord::Migration[
|
684
|
+
class AddSomeIndexToUsers < ActiveRecord::Migration[7.0]
|
532
685
|
def change
|
533
686
|
add_index :users, [:b, :d]
|
534
687
|
end
|
@@ -542,7 +695,7 @@ For Postgres, be sure to add them concurrently.
|
|
542
695
|
To mark a step in the migration as safe, despite using a method that might otherwise be dangerous, wrap it in a `safety_assured` block.
|
543
696
|
|
544
697
|
```ruby
|
545
|
-
class MySafeMigration < ActiveRecord::Migration[
|
698
|
+
class MySafeMigration < ActiveRecord::Migration[7.0]
|
546
699
|
def change
|
547
700
|
safety_assured { remove_column :users, :some_column }
|
548
701
|
end
|
@@ -551,6 +704,21 @@ end
|
|
551
704
|
|
552
705
|
Certain methods like `execute` and `change_table` cannot be inspected and are prevented from running by default. Make sure what you’re doing is really safe and use this pattern.
|
553
706
|
|
707
|
+
## Safe by Default
|
708
|
+
|
709
|
+
Make operations safe by default.
|
710
|
+
|
711
|
+
- adding and removing an index
|
712
|
+
- adding a foreign key
|
713
|
+
- adding a check constraint
|
714
|
+
- setting NOT NULL on an existing column
|
715
|
+
|
716
|
+
Add to `config/initializers/strong_migrations.rb`:
|
717
|
+
|
718
|
+
```ruby
|
719
|
+
StrongMigrations.safe_by_default = true
|
720
|
+
```
|
721
|
+
|
554
722
|
## Custom Checks
|
555
723
|
|
556
724
|
Add your own custom checks with:
|
@@ -585,7 +753,15 @@ Disable specific checks with:
|
|
585
753
|
StrongMigrations.disable_check(:add_index)
|
586
754
|
```
|
587
755
|
|
588
|
-
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
|
756
|
+
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations/error_messages.rb) for the list of keys.
|
757
|
+
|
758
|
+
## Down Migrations / Rollbacks
|
759
|
+
|
760
|
+
By default, checks are disabled when migrating down. Enable them with:
|
761
|
+
|
762
|
+
```ruby
|
763
|
+
StrongMigrations.check_down = true
|
764
|
+
```
|
589
765
|
|
590
766
|
## Custom Messages
|
591
767
|
|
@@ -595,9 +771,9 @@ To customize specific messages, create an initializer with:
|
|
595
771
|
StrongMigrations.error_messages[:add_column_default] = "Your custom instructions"
|
596
772
|
```
|
597
773
|
|
598
|
-
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations.rb) for the list of keys.
|
774
|
+
Check the [source code](https://github.com/ankane/strong_migrations/blob/master/lib/strong_migrations/error_messages.rb) for the list of keys.
|
599
775
|
|
600
|
-
## Timeouts
|
776
|
+
## Migration Timeouts
|
601
777
|
|
602
778
|
It’s extremely important to set a short lock timeout for migrations. This way, if a migration can’t acquire a lock in a timely manner, other statements won’t be stuck behind it. We also recommend setting a long statement timeout so migrations can run for a while.
|
603
779
|
|
@@ -617,6 +793,68 @@ ALTER ROLE myuser SET statement_timeout = '1h';
|
|
617
793
|
|
618
794
|
Note: If you use PgBouncer in transaction mode, you must set timeouts on the database user.
|
619
795
|
|
796
|
+
## Lock Timeout Retries [experimental]
|
797
|
+
|
798
|
+
There’s the option to automatically retry statements when the lock timeout is reached. Here’s how it works:
|
799
|
+
|
800
|
+
- If a lock timeout happens outside a transaction, the statement is retried
|
801
|
+
- If it happens inside the DDL transaction, the entire migration is retried (only applicable to Postgres)
|
802
|
+
|
803
|
+
Add to `config/initializers/strong_migrations.rb`:
|
804
|
+
|
805
|
+
```ruby
|
806
|
+
StrongMigrations.lock_timeout_retries = 3
|
807
|
+
```
|
808
|
+
|
809
|
+
Set the delay between retries with:
|
810
|
+
|
811
|
+
```ruby
|
812
|
+
StrongMigrations.lock_timeout_retry_delay = 10.seconds
|
813
|
+
```
|
814
|
+
|
815
|
+
## App Timeouts
|
816
|
+
|
817
|
+
We recommend adding timeouts to `config/database.yml` to prevent connections from hanging and individual queries from taking up too many resources in controllers, jobs, the Rails console, and other places.
|
818
|
+
|
819
|
+
For Postgres:
|
820
|
+
|
821
|
+
```yml
|
822
|
+
production:
|
823
|
+
connect_timeout: 5
|
824
|
+
variables:
|
825
|
+
statement_timeout: 15s
|
826
|
+
lock_timeout: 10s
|
827
|
+
```
|
828
|
+
|
829
|
+
Note: If you use PgBouncer in transaction mode, you must set the statement and lock timeouts on the database user as shown above.
|
830
|
+
|
831
|
+
For MySQL:
|
832
|
+
|
833
|
+
```yml
|
834
|
+
production:
|
835
|
+
connect_timeout: 5
|
836
|
+
read_timeout: 5
|
837
|
+
write_timeout: 5
|
838
|
+
variables:
|
839
|
+
max_execution_time: 15000 # ms
|
840
|
+
lock_wait_timeout: 10 # sec
|
841
|
+
|
842
|
+
```
|
843
|
+
|
844
|
+
For MariaDB:
|
845
|
+
|
846
|
+
```yml
|
847
|
+
production:
|
848
|
+
connect_timeout: 5
|
849
|
+
read_timeout: 5
|
850
|
+
write_timeout: 5
|
851
|
+
variables:
|
852
|
+
max_statement_time: 15 # sec
|
853
|
+
lock_wait_timeout: 10 # sec
|
854
|
+
```
|
855
|
+
|
856
|
+
For HTTP connections, Redis, and other services, check out [this guide](https://github.com/ankane/the-ultimate-guide-to-ruby-timeouts).
|
857
|
+
|
620
858
|
## Existing Migrations
|
621
859
|
|
622
860
|
To mark migrations as safe that were created before installing this gem, create an initializer with:
|
@@ -632,13 +870,19 @@ Use the version from your latest migration.
|
|
632
870
|
If your development database version is different from production, you can specify the production version so the right checks run in development.
|
633
871
|
|
634
872
|
```ruby
|
635
|
-
StrongMigrations.
|
636
|
-
StrongMigrations.target_mysql_version = "8.0.12"
|
637
|
-
StrongMigrations.target_mariadb_version = "10.3.2"
|
873
|
+
StrongMigrations.target_version = 10 # or "8.0.12", "10.3.2", etc
|
638
874
|
```
|
639
875
|
|
876
|
+
The major version works well for Postgres, while the full version is recommended for MySQL and MariaDB.
|
877
|
+
|
640
878
|
For safety, this option only affects development and test environments. In other environments, the actual server version is always used.
|
641
879
|
|
880
|
+
If your app has multiple databases with different versions, with Rails 6.1+, you can use:
|
881
|
+
|
882
|
+
```ruby
|
883
|
+
StrongMigrations.target_version = {primary: 13, catalog: 15}
|
884
|
+
```
|
885
|
+
|
642
886
|
## Analyze Tables
|
643
887
|
|
644
888
|
Analyze tables automatically (to update planner statistics) after an index is added. Create an initializer with:
|
@@ -649,37 +893,34 @@ StrongMigrations.auto_analyze = true
|
|
649
893
|
|
650
894
|
## Faster Migrations
|
651
895
|
|
652
|
-
Only dump the schema when adding a new migration. If you use Git,
|
896
|
+
Only dump the schema when adding a new migration. If you use Git, add to `config/environments/development.rb`:
|
653
897
|
|
654
|
-
```
|
655
|
-
|
656
|
-
`git status db/migrate/ --porcelain`.present?
|
898
|
+
```rb
|
899
|
+
config.active_record.dump_schema_after_migration = `git status db/migrate/ --porcelain`.present?
|
657
900
|
```
|
658
901
|
|
659
902
|
## Schema Sanity
|
660
903
|
|
661
|
-
Columns can flip order in `db/schema.rb` when you have multiple developers. One way to prevent this is to [alphabetize them](https://www.pgrs.net/2008/03/12/alphabetize-schema-rb-columns/). Add to
|
904
|
+
Columns can flip order in `db/schema.rb` when you have multiple developers. One way to prevent this is to [alphabetize them](https://www.pgrs.net/2008/03/12/alphabetize-schema-rb-columns/). Add to `config/initializers/strong_migrations.rb`:
|
662
905
|
|
663
906
|
```ruby
|
664
|
-
|
665
|
-
```
|
666
|
-
|
667
|
-
## Dangerous Tasks
|
668
|
-
|
669
|
-
For safety, dangerous database tasks are disabled in production - `db:drop`, `db:reset`, `db:schema:load`, and `db:structure:load`. To get around this, use:
|
670
|
-
|
671
|
-
```sh
|
672
|
-
SAFETY_ASSURED=1 rails db:drop
|
907
|
+
StrongMigrations.alphabetize_schema = true
|
673
908
|
```
|
674
909
|
|
675
910
|
## Permissions
|
676
911
|
|
677
912
|
We recommend using a [separate database user](https://ankane.org/postgres-users) for migrations when possible so you don’t need to grant your app user permission to alter tables.
|
678
913
|
|
914
|
+
## Smaller Projects
|
915
|
+
|
916
|
+
You probably don’t need this gem for smaller projects, as operations that are unsafe at scale can be perfectly safe on smaller, low-traffic tables.
|
917
|
+
|
679
918
|
## Additional Reading
|
680
919
|
|
681
920
|
- [Rails Migrations with No Downtime](https://pedro.herokuapp.com/past/2011/7/13/rails_migrations_with_no_downtime/)
|
682
921
|
- [PostgreSQL at Scale: Database Schema Changes Without Downtime](https://medium.com/braintree-product-technology/postgresql-at-scale-database-schema-changes-without-downtime-20d3749ed680)
|
922
|
+
- [An Overview of DDL Algorithms in MySQL](https://mydbops.wordpress.com/2020/03/04/an-overview-of-ddl-algorithms-in-mysql-covers-mysql-8/)
|
923
|
+
- [MariaDB InnoDB Online DDL Overview](https://mariadb.com/kb/en/innodb-online-ddl-overview/)
|
683
924
|
|
684
925
|
## Credits
|
685
926
|
|