online_migrations 0.7.1 → 0.7.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +7 -0
- data/README.md +48 -213
- data/docs/configuring.md +208 -0
- data/lib/online_migrations/command_checker.rb +2 -1
- data/lib/online_migrations/error_messages.rb +27 -2
- data/lib/online_migrations/schema_statements.rb +2 -2
- data/lib/online_migrations/utils.rb +8 -0
- data/lib/online_migrations/version.rb +1 -1
- metadata +5 -4
- /data/{BACKGROUND_MIGRATIONS.md → docs/background_migrations.md} +0 -0
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 882229e8b11af47ea2e30e16aa8ff55dd482ce1131175ac0287ba3ac2c24142f
|
4
|
+
data.tar.gz: d1e2c65e980c4dc32aa15498dedd3ed7b186e56f4765ff12e25f4cb977797f81
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 1b008939f6db560c92d0c1742e91a62782a0629dc5700e51e3f16269d76d1e394b1fbed45df1f2066777edf787e77c248271d5b056f747300ec48e0b7f5064c7
|
7
|
+
data.tar.gz: 3d46aa208a7675a4a9a3762dedb7cc25a5cf62724bc7af3774eb5c25e30b07573fe87f3e8803828e98095af7eed71d1e4724a3ec25e20b845d1d8e1c7ca62126
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,12 @@
|
|
1
1
|
## master (unreleased)
|
2
2
|
|
3
|
+
## 0.7.2 (2023-03-08)
|
4
|
+
|
5
|
+
- Suggest additional steps for safely renaming a column if Active Record `enumerate_columns_in_select_statements`
|
6
|
+
setting is enabled (implemented in Active Record 7+, disabled by default)
|
7
|
+
- Fix `add_reference_concurrently` to correctly check for existence of foreign keys
|
8
|
+
- Fix column quoting in `add_not_null_constraint`
|
9
|
+
|
3
10
|
## 0.7.1 (2023-02-22)
|
4
11
|
|
5
12
|
- Fix Schema Cache to correctly retrieve metadata from renamed tables
|
data/README.md
CHANGED
@@ -37,7 +37,7 @@ $ bundle install
|
|
37
37
|
$ bin/rails generate online_migrations:install
|
38
38
|
```
|
39
39
|
|
40
|
-
**Note**: If you do not have plans on using [background migrations](
|
40
|
+
**Note**: If you do not have plans on using [background migrations](docs/background_migrations.md) feature, then you can delete the generated migration and regenerate it later, if needed.
|
41
41
|
|
42
42
|
## Motivation
|
43
43
|
|
@@ -388,6 +388,17 @@ end
|
|
388
388
|
|
389
389
|
:white_check_mark: **Good**
|
390
390
|
|
391
|
+
#### "Classic" approach (abstract)
|
392
|
+
|
393
|
+
1. Create a new column
|
394
|
+
2. Write to both columns
|
395
|
+
3. Backfill data from the old column to the new column
|
396
|
+
4. Move reads from the old column to the new column
|
397
|
+
5. Stop writing to the old column
|
398
|
+
6. Drop the old column
|
399
|
+
|
400
|
+
#### :bullettrain_side: Enhanced approach (with concrete steps for Active Record)
|
401
|
+
|
391
402
|
The "classic" approach suggests creating a new column and copy data/indexes/etc to it from the old column. This can be costly for very large tables. There is a trick that helps to avoid such heavy operations.
|
392
403
|
|
393
404
|
The technique is built on top of database views, using the following steps:
|
@@ -445,9 +456,27 @@ end
|
|
445
456
|
```
|
446
457
|
|
447
458
|
4. Replace usages of the old column with a new column in the codebase
|
448
|
-
5.
|
449
|
-
|
450
|
-
|
459
|
+
5. If you enabled Active Record `enumerate_columns_in_select_statements` setting in your application
|
460
|
+
(is disabled by default in Active Record >= 7), then you need to ignore old column:
|
461
|
+
|
462
|
+
```ruby
|
463
|
+
# For ActiveRecord 5+
|
464
|
+
class User < ApplicationRecord
|
465
|
+
self.ignored_columns = ["name"]
|
466
|
+
end
|
467
|
+
|
468
|
+
# For ActiveRecord < 5
|
469
|
+
class User < ActiveRecord::Base
|
470
|
+
def self.columns
|
471
|
+
super.reject { |c| c.name == "name" }
|
472
|
+
end
|
473
|
+
end
|
474
|
+
```
|
475
|
+
|
476
|
+
6. Deploy
|
477
|
+
7. Remove the column rename config from step 1
|
478
|
+
8. Remove the column ignore from step 5, if added
|
479
|
+
9. Remove the VIEW created in step 3 and finally rename the column:
|
451
480
|
|
452
481
|
```ruby
|
453
482
|
class FinalizeRenameUsersNameToFirstName < ActiveRecord::Migration[7.0]
|
@@ -457,7 +486,7 @@ class FinalizeRenameUsersNameToFirstName < ActiveRecord::Migration[7.0]
|
|
457
486
|
end
|
458
487
|
```
|
459
488
|
|
460
|
-
|
489
|
+
10. Deploy
|
461
490
|
|
462
491
|
### Renaming a table
|
463
492
|
|
@@ -475,6 +504,17 @@ end
|
|
475
504
|
|
476
505
|
:white_check_mark: **Good**
|
477
506
|
|
507
|
+
#### "Classic" approach (abstract)
|
508
|
+
|
509
|
+
1. Create a new table
|
510
|
+
2. Write to both tables
|
511
|
+
3. Backfill data from the old table to new table
|
512
|
+
4. Move reads from the old table to the new table
|
513
|
+
5. Stop writing to the old table
|
514
|
+
6. Drop the old table
|
515
|
+
|
516
|
+
#### :bullettrain_side: Enhanced approach (with concrete steps for Active Record)
|
517
|
+
|
478
518
|
The "classic" approach suggests creating a new table and copy data/indexes/etc to it from the old table. This can be costly for very large tables. There is a trick that helps to avoid such heavy operations.
|
479
519
|
|
480
520
|
The technique is built on top of database views, using the following steps:
|
@@ -1087,216 +1127,11 @@ Certain methods like `execute` and `change_table` cannot be inspected and are pr
|
|
1087
1127
|
|
1088
1128
|
## Configuring the gem
|
1089
1129
|
|
1090
|
-
|
1091
|
-
|
1092
|
-
```ruby
|
1093
|
-
OnlineMigrations.configure do |config|
|
1094
|
-
# ...
|
1095
|
-
end
|
1096
|
-
```
|
1097
|
-
|
1098
|
-
**Note**: Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/config.rb) for the list of all available configuration options.
|
1099
|
-
|
1100
|
-
### Custom checks
|
1101
|
-
|
1102
|
-
Add your own custom checks with:
|
1103
|
-
|
1104
|
-
```ruby
|
1105
|
-
# config/initializers/online_migrations.rb
|
1106
|
-
|
1107
|
-
config.add_check do |method, args|
|
1108
|
-
if method == :add_column && args[0].to_s == "users"
|
1109
|
-
stop!("No more columns on the users table")
|
1110
|
-
end
|
1111
|
-
end
|
1112
|
-
```
|
1113
|
-
|
1114
|
-
Use the `stop!` method to stop migrations.
|
1115
|
-
|
1116
|
-
**Note**: Since `remove_column`, `execute` and `change_table` always require a `safety_assured` block, it's not possible to add a custom check for these operations.
|
1117
|
-
|
1118
|
-
### Disable Checks
|
1119
|
-
|
1120
|
-
Disable specific checks with:
|
1121
|
-
|
1122
|
-
```ruby
|
1123
|
-
# config/initializers/online_migrations.rb
|
1124
|
-
|
1125
|
-
config.disable_check(:remove_index)
|
1126
|
-
```
|
1127
|
-
|
1128
|
-
Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/error_messages.rb) for the list of keys.
|
1129
|
-
|
1130
|
-
### Down Migrations / Rollbacks
|
1131
|
-
|
1132
|
-
By default, checks are disabled when migrating down. Enable them with:
|
1133
|
-
|
1134
|
-
```ruby
|
1135
|
-
# config/initializers/online_migrations.rb
|
1136
|
-
|
1137
|
-
config.check_down = true
|
1138
|
-
```
|
1139
|
-
|
1140
|
-
### Custom Messages
|
1141
|
-
|
1142
|
-
You can customize specific error messages:
|
1143
|
-
|
1144
|
-
```ruby
|
1145
|
-
# config/initializers/online_migrations.rb
|
1146
|
-
|
1147
|
-
config.error_messages[:add_column_default] = "Your custom instructions"
|
1148
|
-
```
|
1149
|
-
|
1150
|
-
Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/error_messages.rb) for the list of keys.
|
1151
|
-
|
1152
|
-
### Migration Timeouts
|
1153
|
-
|
1154
|
-
It’s extremely important to set a short lock timeout for migrations. This way, if a migration can't acquire a lock in a timely manner, other statements won't be stuck behind it.
|
1155
|
-
|
1156
|
-
Add timeouts to `config/database.yml`:
|
1157
|
-
|
1158
|
-
```yml
|
1159
|
-
production:
|
1160
|
-
connect_timeout: 5
|
1161
|
-
variables:
|
1162
|
-
lock_timeout: 10s
|
1163
|
-
statement_timeout: 15s
|
1164
|
-
```
|
1165
|
-
|
1166
|
-
Or set the timeouts directly on the database user that runs migrations:
|
1167
|
-
|
1168
|
-
```sql
|
1169
|
-
ALTER ROLE myuser SET lock_timeout = '10s';
|
1170
|
-
ALTER ROLE myuser SET statement_timeout = '15s';
|
1171
|
-
```
|
1172
|
-
|
1173
|
-
### Lock Timeout Retries
|
1174
|
-
|
1175
|
-
You can configure this gem to automatically retry statements that exceed the lock timeout:
|
1176
|
-
|
1177
|
-
```ruby
|
1178
|
-
# config/initializers/online_migrations.rb
|
1179
|
-
|
1180
|
-
config.lock_retrier = OnlineMigrations::ExponentialLockRetrier.new(
|
1181
|
-
attempts: 30, # attempt 30 retries
|
1182
|
-
base_delay: 0.01.seconds, # starting with delay of 10ms between each unsuccessful try, increasing exponentially
|
1183
|
-
max_delay: 1.minute, # maximum delay is 1 minute
|
1184
|
-
lock_timeout: 0.05.seconds # and 50ms set as lock timeout for each try
|
1185
|
-
)
|
1186
|
-
```
|
1187
|
-
|
1188
|
-
When statement within transaction fails - the whole transaction is retried.
|
1189
|
-
|
1190
|
-
To permanently disable lock retries, you can set `lock_retrier` to `nil`.
|
1191
|
-
To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable.
|
1192
|
-
|
1193
|
-
**Note**: Statements are retried by default, unless lock retries are disabled. It is possible to implement more sophisticated lock retriers. See [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/lock_retrier.rb) for the examples.
|
1194
|
-
|
1195
|
-
### Existing Migrations
|
1196
|
-
|
1197
|
-
To mark migrations as safe that were created before installing this gem, configure the migration version starting after which checks are performed:
|
1198
|
-
|
1199
|
-
```ruby
|
1200
|
-
# config/initializers/online_migrations.rb
|
1201
|
-
|
1202
|
-
config.start_after = 20220101000000
|
1203
|
-
|
1204
|
-
# or if you use multiple databases (ActiveRecord 6+)
|
1205
|
-
config.start_after = { primary: 20211112000000, animals: 20220101000000 }
|
1206
|
-
```
|
1207
|
-
|
1208
|
-
Use the version from your latest migration.
|
1209
|
-
|
1210
|
-
### Target Version
|
1211
|
-
|
1212
|
-
If your development database version is different from production, you can specify the production version so the right checks run in development.
|
1213
|
-
|
1214
|
-
```ruby
|
1215
|
-
# config/initializers/online_migrations.rb
|
1216
|
-
|
1217
|
-
config.target_version = 10 # or "12.9" etc
|
1218
|
-
|
1219
|
-
# or if you use multiple databases (ActiveRecord 6+)
|
1220
|
-
config.target_version = { primary: 10, animals: 14.1 }
|
1221
|
-
```
|
1222
|
-
|
1223
|
-
For safety, this option only affects development and test environments. In other environments, the actual server version is always used.
|
1224
|
-
|
1225
|
-
### Small Tables
|
1226
|
-
|
1227
|
-
Most projects have tables that are known to be small in size. These are usually "settings", "prices", "plans" etc. It is considered safe to perform most of the dangerous operations on them, like adding indexes, columns etc.
|
1228
|
-
|
1229
|
-
To mark tables as small:
|
1230
|
-
|
1231
|
-
```ruby
|
1232
|
-
config.small_tables = [:settings, :prices]
|
1233
|
-
```
|
1234
|
-
|
1235
|
-
### Verbose SQL logs
|
1236
|
-
|
1237
|
-
For any operation, **Online Migrations** can output the performed SQL queries.
|
1238
|
-
|
1239
|
-
This is useful to demystify `online_migrations` inner workings, and to better investigate migration failure in production. This is also useful in development to get a better grasp of what is going on for high-level statements like `add_column_with_default`.
|
1240
|
-
|
1241
|
-
Consider migration, running on PostgreSQL < 11:
|
1242
|
-
|
1243
|
-
```ruby
|
1244
|
-
class AddAdminToUsers < ActiveRecord::Migration[7.0]
|
1245
|
-
disable_ddl_transaction!
|
1246
|
-
|
1247
|
-
def change
|
1248
|
-
add_column_with_default :users, :admin, :boolean, default: false
|
1249
|
-
end
|
1250
|
-
end
|
1251
|
-
```
|
1252
|
-
|
1253
|
-
Instead of the traditional output:
|
1254
|
-
|
1255
|
-
```
|
1256
|
-
== 20220106214827 AddAdminToUsers: migrating ==================================
|
1257
|
-
-- add_column_with_default(:users, :admin, :boolean, {:default=>false})
|
1258
|
-
-> 0.1423s
|
1259
|
-
== 20220106214827 AddAdminToUsers: migrated (0.1462s) =========================
|
1260
|
-
```
|
1261
|
-
|
1262
|
-
**Online Migrations** will output the following logs:
|
1263
|
-
|
1264
|
-
```
|
1265
|
-
== 20220106214827 AddAdminToUsers: migrating ==================================
|
1266
|
-
(0.3ms) SHOW lock_timeout
|
1267
|
-
(0.2ms) SET lock_timeout TO '50ms'
|
1268
|
-
-- add_column_with_default(:users, :admin, :boolean, {:default=>false})
|
1269
|
-
TRANSACTION (0.1ms) BEGIN
|
1270
|
-
(37.7ms) ALTER TABLE "users" ADD "admin" boolean DEFAULT NULL
|
1271
|
-
(0.5ms) ALTER TABLE "users" ALTER COLUMN "admin" SET DEFAULT FALSE
|
1272
|
-
TRANSACTION (0.3ms) COMMIT
|
1273
|
-
Load (0.3ms) SELECT "users"."id" FROM "users" WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) ORDER BY "users"."id" ASC LIMIT $1 [["LIMIT", 1]]
|
1274
|
-
Load (0.5ms) SELECT "users"."id" FROM "users" WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1 ORDER BY "users"."id" ASC LIMIT $1 OFFSET $2 [["LIMIT", 1], ["OFFSET", 1000]]
|
1275
|
-
#<Class:0x00007f8ae3703f08> Update All (9.6ms) UPDATE "users" SET "admin" = $1 WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1 AND "users"."id" < 1001 [["admin", false]]
|
1276
|
-
Load (0.8ms) SELECT "users"."id" FROM "users" WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1001 ORDER BY "users"."id" ASC LIMIT $1 OFFSET $2 [["LIMIT", 1], ["OFFSET", 1000]]
|
1277
|
-
#<Class:0x00007f8ae3703f08> Update All (1.5ms) UPDATE "users" SET "admin" = $1 WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1001 [["admin", false]]
|
1278
|
-
-> 0.1814s
|
1279
|
-
(0.4ms) SET lock_timeout TO '5s'
|
1280
|
-
== 20220106214827 AddAdminToUsers: migrated (0.1840s) =========================
|
1281
|
-
```
|
1282
|
-
|
1283
|
-
So you can actually check which steps are performed.
|
1284
|
-
|
1285
|
-
**Note**: The `SHOW` statements are used by **Online Migrations** to query settings for their original values in order to restore them after the work is done.
|
1286
|
-
|
1287
|
-
To enable verbose sql logs:
|
1288
|
-
|
1289
|
-
```ruby
|
1290
|
-
# config/initializers/online_migrations.rb
|
1291
|
-
|
1292
|
-
config.verbose_sql_logs = true
|
1293
|
-
```
|
1294
|
-
|
1295
|
-
This feature is enabled by default in a production Rails environment. You can override this setting via `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
|
1130
|
+
Read [configuring.md](docs/configuring.md).
|
1296
1131
|
|
1297
1132
|
## Background Migrations
|
1298
1133
|
|
1299
|
-
Read [
|
1134
|
+
Read [background_migrations.md](docs/background_migrations.md) on how to perform data migrations on large tables.
|
1300
1135
|
|
1301
1136
|
## Credits
|
1302
1137
|
|
@@ -1361,7 +1196,7 @@ It has migrations helpers for:
|
|
1361
1196
|
* adding different types of constraints
|
1362
1197
|
* and others
|
1363
1198
|
|
1364
|
-
2. This gem has a [powerful internal framework](https://github.com/fatkodima/online_migrations/blob/master/
|
1199
|
+
2. This gem has a [powerful internal framework](https://github.com/fatkodima/online_migrations/blob/master/docs/background_migrations.md) for running data migrations on very large tables using background migrations.
|
1365
1200
|
|
1366
1201
|
For example, you can use background migrations to migrate data that’s stored in a single JSON column to a separate table instead; backfill values from one column to another (as one of the steps when changing column type); or backfill some column’s value from an API.
|
1367
1202
|
|
data/docs/configuring.md
ADDED
@@ -0,0 +1,208 @@
|
|
1
|
+
# Configuring
|
2
|
+
|
3
|
+
There are a few configurable options for the gem. Custom configurations should be placed in a `online_migrations.rb` initializer.
|
4
|
+
|
5
|
+
```ruby
|
6
|
+
OnlineMigrations.configure do |config|
|
7
|
+
# ...
|
8
|
+
end
|
9
|
+
```
|
10
|
+
|
11
|
+
**Note**: Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/config.rb) for the list of all available configuration options.
|
12
|
+
|
13
|
+
## Custom checks
|
14
|
+
|
15
|
+
Add your own custom checks with:
|
16
|
+
|
17
|
+
```ruby
|
18
|
+
# config/initializers/online_migrations.rb
|
19
|
+
|
20
|
+
config.add_check do |method, args|
|
21
|
+
if method == :add_column && args[0].to_s == "users"
|
22
|
+
stop!("No more columns on the users table")
|
23
|
+
end
|
24
|
+
end
|
25
|
+
```
|
26
|
+
|
27
|
+
Use the `stop!` method to stop migrations.
|
28
|
+
|
29
|
+
**Note**: Since `remove_column`, `execute` and `change_table` always require a `safety_assured` block, it's not possible to add a custom check for these operations.
|
30
|
+
|
31
|
+
## Disable Checks
|
32
|
+
|
33
|
+
Disable specific checks with:
|
34
|
+
|
35
|
+
```ruby
|
36
|
+
# config/initializers/online_migrations.rb
|
37
|
+
|
38
|
+
config.disable_check(:remove_index)
|
39
|
+
```
|
40
|
+
|
41
|
+
Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/error_messages.rb) for the list of keys.
|
42
|
+
|
43
|
+
## Down Migrations / Rollbacks
|
44
|
+
|
45
|
+
By default, checks are disabled when migrating down. Enable them with:
|
46
|
+
|
47
|
+
```ruby
|
48
|
+
# config/initializers/online_migrations.rb
|
49
|
+
|
50
|
+
config.check_down = true
|
51
|
+
```
|
52
|
+
|
53
|
+
## Custom Messages
|
54
|
+
|
55
|
+
You can customize specific error messages:
|
56
|
+
|
57
|
+
```ruby
|
58
|
+
# config/initializers/online_migrations.rb
|
59
|
+
|
60
|
+
config.error_messages[:add_column_default] = "Your custom instructions"
|
61
|
+
```
|
62
|
+
|
63
|
+
Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/error_messages.rb) for the list of keys.
|
64
|
+
|
65
|
+
## Migration Timeouts
|
66
|
+
|
67
|
+
It’s extremely important to set a short lock timeout for migrations. This way, if a migration can't acquire a lock in a timely manner, other statements won't be stuck behind it.
|
68
|
+
|
69
|
+
Add timeouts to `config/database.yml`:
|
70
|
+
|
71
|
+
```yml
|
72
|
+
production:
|
73
|
+
connect_timeout: 5
|
74
|
+
variables:
|
75
|
+
lock_timeout: 10s
|
76
|
+
statement_timeout: 15s
|
77
|
+
```
|
78
|
+
|
79
|
+
Or set the timeouts directly on the database user that runs migrations:
|
80
|
+
|
81
|
+
```sql
|
82
|
+
ALTER ROLE myuser SET lock_timeout = '10s';
|
83
|
+
ALTER ROLE myuser SET statement_timeout = '15s';
|
84
|
+
```
|
85
|
+
|
86
|
+
## Lock Timeout Retries
|
87
|
+
|
88
|
+
You can configure this gem to automatically retry statements that exceed the lock timeout:
|
89
|
+
|
90
|
+
```ruby
|
91
|
+
# config/initializers/online_migrations.rb
|
92
|
+
|
93
|
+
config.lock_retrier = OnlineMigrations::ExponentialLockRetrier.new(
|
94
|
+
attempts: 30, # attempt 30 retries
|
95
|
+
base_delay: 0.01.seconds, # starting with delay of 10ms between each unsuccessful try, increasing exponentially
|
96
|
+
max_delay: 1.minute, # maximum delay is 1 minute
|
97
|
+
lock_timeout: 0.05.seconds # and 50ms set as lock timeout for each try
|
98
|
+
)
|
99
|
+
```
|
100
|
+
|
101
|
+
When statement within transaction fails - the whole transaction is retried.
|
102
|
+
|
103
|
+
To permanently disable lock retries, you can set `lock_retrier` to `nil`.
|
104
|
+
To temporarily disable lock retries while running migrations, set `DISABLE_LOCK_RETRIES` env variable.
|
105
|
+
|
106
|
+
**Note**: Statements are retried by default, unless lock retries are disabled. It is possible to implement more sophisticated lock retriers. See [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/lock_retrier.rb) for the examples.
|
107
|
+
|
108
|
+
## Existing Migrations
|
109
|
+
|
110
|
+
To mark migrations as safe that were created before installing this gem, configure the migration version starting after which checks are performed:
|
111
|
+
|
112
|
+
```ruby
|
113
|
+
# config/initializers/online_migrations.rb
|
114
|
+
|
115
|
+
config.start_after = 20220101000000
|
116
|
+
|
117
|
+
# or if you use multiple databases (ActiveRecord 6+)
|
118
|
+
config.start_after = { primary: 20211112000000, animals: 20220101000000 }
|
119
|
+
```
|
120
|
+
|
121
|
+
Use the version from your latest migration.
|
122
|
+
|
123
|
+
## Target Version
|
124
|
+
|
125
|
+
If your development database version is different from production, you can specify the production version so the right checks run in development.
|
126
|
+
|
127
|
+
```ruby
|
128
|
+
# config/initializers/online_migrations.rb
|
129
|
+
|
130
|
+
config.target_version = 10 # or "12.9" etc
|
131
|
+
|
132
|
+
# or if you use multiple databases (ActiveRecord 6+)
|
133
|
+
config.target_version = { primary: 10, animals: 14.1 }
|
134
|
+
```
|
135
|
+
|
136
|
+
For safety, this option only affects development and test environments. In other environments, the actual server version is always used.
|
137
|
+
|
138
|
+
## Small Tables
|
139
|
+
|
140
|
+
Most projects have tables that are known to be small in size. These are usually "settings", "prices", "plans" etc. It is considered safe to perform most of the dangerous operations on them, like adding indexes, columns etc.
|
141
|
+
|
142
|
+
To mark tables as small:
|
143
|
+
|
144
|
+
```ruby
|
145
|
+
config.small_tables = [:settings, :prices]
|
146
|
+
```
|
147
|
+
|
148
|
+
## Verbose SQL logs
|
149
|
+
|
150
|
+
For any operation, **Online Migrations** can output the performed SQL queries.
|
151
|
+
|
152
|
+
This is useful to demystify `online_migrations` inner workings, and to better investigate migration failure in production. This is also useful in development to get a better grasp of what is going on for high-level statements like `add_column_with_default`.
|
153
|
+
|
154
|
+
Consider migration, running on PostgreSQL < 11:
|
155
|
+
|
156
|
+
```ruby
|
157
|
+
class AddAdminToUsers < ActiveRecord::Migration[7.0]
|
158
|
+
disable_ddl_transaction!
|
159
|
+
|
160
|
+
def change
|
161
|
+
add_column_with_default :users, :admin, :boolean, default: false
|
162
|
+
end
|
163
|
+
end
|
164
|
+
```
|
165
|
+
|
166
|
+
Instead of the traditional output:
|
167
|
+
|
168
|
+
```
|
169
|
+
== 20220106214827 AddAdminToUsers: migrating ==================================
|
170
|
+
-- add_column_with_default(:users, :admin, :boolean, {:default=>false})
|
171
|
+
-> 0.1423s
|
172
|
+
== 20220106214827 AddAdminToUsers: migrated (0.1462s) =========================
|
173
|
+
```
|
174
|
+
|
175
|
+
**Online Migrations** will output the following logs:
|
176
|
+
|
177
|
+
```
|
178
|
+
== 20220106214827 AddAdminToUsers: migrating ==================================
|
179
|
+
(0.3ms) SHOW lock_timeout
|
180
|
+
(0.2ms) SET lock_timeout TO '50ms'
|
181
|
+
-- add_column_with_default(:users, :admin, :boolean, {:default=>false})
|
182
|
+
TRANSACTION (0.1ms) BEGIN
|
183
|
+
(37.7ms) ALTER TABLE "users" ADD "admin" boolean DEFAULT NULL
|
184
|
+
(0.5ms) ALTER TABLE "users" ALTER COLUMN "admin" SET DEFAULT FALSE
|
185
|
+
TRANSACTION (0.3ms) COMMIT
|
186
|
+
Load (0.3ms) SELECT "users"."id" FROM "users" WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) ORDER BY "users"."id" ASC LIMIT $1 [["LIMIT", 1]]
|
187
|
+
Load (0.5ms) SELECT "users"."id" FROM "users" WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1 ORDER BY "users"."id" ASC LIMIT $1 OFFSET $2 [["LIMIT", 1], ["OFFSET", 1000]]
|
188
|
+
#<Class:0x00007f8ae3703f08> Update All (9.6ms) UPDATE "users" SET "admin" = $1 WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1 AND "users"."id" < 1001 [["admin", false]]
|
189
|
+
Load (0.8ms) SELECT "users"."id" FROM "users" WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1001 ORDER BY "users"."id" ASC LIMIT $1 OFFSET $2 [["LIMIT", 1], ["OFFSET", 1000]]
|
190
|
+
#<Class:0x00007f8ae3703f08> Update All (1.5ms) UPDATE "users" SET "admin" = $1 WHERE ("users"."admin" != FALSE OR "users"."admin" IS NULL) AND "users"."id" >= 1001 [["admin", false]]
|
191
|
+
-> 0.1814s
|
192
|
+
(0.4ms) SET lock_timeout TO '5s'
|
193
|
+
== 20220106214827 AddAdminToUsers: migrated (0.1840s) =========================
|
194
|
+
```
|
195
|
+
|
196
|
+
So you can actually check which steps are performed.
|
197
|
+
|
198
|
+
**Note**: The `SHOW` statements are used by **Online Migrations** to query settings for their original values in order to restore them after the work is done.
|
199
|
+
|
200
|
+
To enable verbose sql logs:
|
201
|
+
|
202
|
+
```ruby
|
203
|
+
# config/initializers/online_migrations.rb
|
204
|
+
|
205
|
+
config.verbose_sql_logs = true
|
206
|
+
```
|
207
|
+
|
208
|
+
This feature is enabled by default in a production Rails environment. You can override this setting via `ONLINE_MIGRATIONS_VERBOSE_SQL_LOGS` environment variable.
|
@@ -220,7 +220,8 @@ module OnlineMigrations
|
|
220
220
|
new_column: new_column,
|
221
221
|
model: table_name.to_s.classify,
|
222
222
|
partial_writes: Utils.ar_partial_writes?,
|
223
|
-
partial_writes_setting: Utils.ar_partial_writes_setting
|
223
|
+
partial_writes_setting: Utils.ar_partial_writes_setting,
|
224
|
+
enumerate_columns_in_select_statements: Utils.ar_enumerate_columns_in_select_statements
|
224
225
|
end
|
225
226
|
end
|
226
227
|
|
@@ -149,17 +149,42 @@ It will use a combination of a VIEW and column aliasing to work with both column
|
|
149
149
|
end
|
150
150
|
|
151
151
|
4. Replace usages of the old column with a new column in the codebase
|
152
|
+
<% if enumerate_columns_in_select_statements %>
|
153
|
+
5. Ignore old column
|
154
|
+
|
155
|
+
<% if ar_version >= 5 %>
|
156
|
+
self.ignored_columns = [:<%= column_name %>]
|
157
|
+
<% else %>
|
158
|
+
def self.columns
|
159
|
+
super.reject { |c| c.name == \"<%= column_name %>\" }
|
160
|
+
end
|
161
|
+
<% end %>
|
162
|
+
|
163
|
+
6. Deploy
|
164
|
+
7. Remove the column rename config from step 1
|
165
|
+
8. Remove the column ignore from step 5
|
166
|
+
9. Remove the VIEW created in step 3 and finally rename the column:
|
167
|
+
|
168
|
+
class Finalize<%= migration_name %> < <%= migration_parent %>
|
169
|
+
def change
|
170
|
+
finalize_column_rename :<%= table_name %>, :<%= column_name %>, :<%= new_column %>
|
171
|
+
end
|
172
|
+
end
|
173
|
+
|
174
|
+
10. Deploy
|
175
|
+
<% else %>
|
152
176
|
5. Deploy
|
153
177
|
6. Remove the column rename config from step 1
|
154
178
|
7. Remove the VIEW created in step 3 and finally rename the column:
|
155
179
|
|
156
180
|
class Finalize<%= migration_name %> < <%= migration_parent %>
|
157
181
|
def change
|
158
|
-
finalize_column_rename
|
182
|
+
finalize_column_rename :<%= table_name %>, :<%= column_name %>, :<%= new_column %>
|
159
183
|
end
|
160
184
|
end
|
161
185
|
|
162
|
-
8. Deploy
|
186
|
+
8. Deploy
|
187
|
+
<% end %>",
|
163
188
|
|
164
189
|
change_column_with_not_null:
|
165
190
|
"Changing the type is safe, but setting NOT NULL is not.",
|
@@ -507,7 +507,7 @@ module OnlineMigrations
|
|
507
507
|
__not_null_constraint_exists?(table_name, column_name, name: name)
|
508
508
|
Utils.say("NOT NULL constraint was not created: column #{table_name}.#{column_name} is already defined as `NOT NULL`")
|
509
509
|
else
|
510
|
-
expression = "#{column_name} IS NOT NULL"
|
510
|
+
expression = "#{quote_column_name(column_name)} IS NOT NULL"
|
511
511
|
name ||= __not_null_constraint_name(table_name, column_name)
|
512
512
|
add_check_constraint(table_name, expression, name: name, validate: false)
|
513
513
|
|
@@ -687,7 +687,7 @@ module OnlineMigrations
|
|
687
687
|
foreign_key = {} if foreign_key == true
|
688
688
|
|
689
689
|
foreign_table_name = Utils.foreign_table_name(ref_name, foreign_key)
|
690
|
-
add_foreign_key(table_name, foreign_table_name, **foreign_key.merge(validate: false))
|
690
|
+
add_foreign_key(table_name, foreign_table_name, **foreign_key.merge(column: column_name, validate: false))
|
691
691
|
|
692
692
|
if foreign_key[:validate] != false
|
693
693
|
validate_foreign_key(table_name, foreign_table_name, **foreign_key)
|
@@ -90,6 +90,14 @@ module OnlineMigrations
|
|
90
90
|
end
|
91
91
|
end
|
92
92
|
|
93
|
+
def ar_enumerate_columns_in_select_statements
|
94
|
+
if ar_version >= 7
|
95
|
+
ActiveRecord::Base.enumerate_columns_in_select_statements
|
96
|
+
else
|
97
|
+
false
|
98
|
+
end
|
99
|
+
end
|
100
|
+
|
93
101
|
# Returns estimated rows count for a table.
|
94
102
|
# https://www.citusdata.com/blog/2016/10/12/count-performance/
|
95
103
|
def estimated_count(connection, table_name)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: online_migrations
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.7.
|
4
|
+
version: 0.7.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- fatkodima
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2023-
|
11
|
+
date: 2023-03-08 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: activerecord
|
@@ -31,10 +31,11 @@ executables: []
|
|
31
31
|
extensions: []
|
32
32
|
extra_rdoc_files: []
|
33
33
|
files:
|
34
|
-
- BACKGROUND_MIGRATIONS.md
|
35
34
|
- CHANGELOG.md
|
36
35
|
- LICENSE.txt
|
37
36
|
- README.md
|
37
|
+
- docs/background_migrations.md
|
38
|
+
- docs/configuring.md
|
38
39
|
- lib/generators/online_migrations/background_migration_generator.rb
|
39
40
|
- lib/generators/online_migrations/install_generator.rb
|
40
41
|
- lib/generators/online_migrations/templates/background_migration.rb.tt
|
@@ -101,7 +102,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
101
102
|
- !ruby/object:Gem::Version
|
102
103
|
version: '0'
|
103
104
|
requirements: []
|
104
|
-
rubygems_version: 3.4.
|
105
|
+
rubygems_version: 3.4.7
|
105
106
|
signing_key:
|
106
107
|
specification_version: 4
|
107
108
|
summary: Catch unsafe PostgreSQL migrations in development and run them easier in
|
File without changes
|