online_migrations 0.15.0 → 0.16.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +6 -0
- data/README.md +10 -6
- data/docs/{background_migrations.md → background_data_migrations.md} +4 -33
- data/docs/background_schema_migrations.md +163 -0
- data/docs/configuring.md +30 -1
- data/lib/generators/online_migrations/templates/create_background_schema_migrations.rb.tt +29 -0
- data/lib/generators/online_migrations/templates/initializer.rb.tt +24 -11
- data/lib/generators/online_migrations/templates/install_migration.rb.tt +24 -0
- data/lib/generators/online_migrations/upgrade_generator.rb +5 -0
- data/lib/online_migrations/application_record.rb +11 -0
- data/lib/online_migrations/background_migrations/config.rb +27 -24
- data/lib/online_migrations/background_migrations/migration.rb +1 -8
- data/lib/online_migrations/background_migrations/migration_job_runner.rb +1 -1
- data/lib/online_migrations/background_migrations/migration_runner.rb +1 -1
- data/lib/online_migrations/background_schema_migrations/config.rb +40 -0
- data/lib/online_migrations/background_schema_migrations/migration.rb +205 -0
- data/lib/online_migrations/background_schema_migrations/migration_helpers.rb +76 -0
- data/lib/online_migrations/background_schema_migrations/migration_runner.rb +110 -0
- data/lib/online_migrations/background_schema_migrations/migration_status_validator.rb +33 -0
- data/lib/online_migrations/background_schema_migrations/scheduler.rb +30 -0
- data/lib/online_migrations/config.rb +32 -0
- data/lib/online_migrations/schema_statements.rb +1 -0
- data/lib/online_migrations/utils.rb +7 -0
- data/lib/online_migrations/version.rb +1 -1
- data/lib/online_migrations.rb +19 -2
- metadata +12 -4
- data/lib/online_migrations/background_migrations/application_record.rb +0 -13
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 48ddbce66257c9010d8ab7361d8860cd898854207b5a1f4c6aa2b6b797aec2e1
|
4
|
+
data.tar.gz: c21fbb5f535f788e13068e339eebe77f05a378c68c00244c4522a10d50a28a0d
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: a5bbff2a8cca45e5fce79d653db3246bc41a4ab9d314f44131bbf74f2659d610fdaf3e87ac5e22baf7928d0cdb481e19eacaa1ed8945e4ee32f1afac3223273f
|
7
|
+
data.tar.gz: 6e3861f420327e0bccec7ab8e7c0a508e7cf224312c366297834091467d7cdb5e4fac8a305dbb59f7c0650d199d9a17062ab0408fd180b6fbaa05946bfd27545
|
data/CHANGELOG.md
CHANGED
@@ -1,5 +1,11 @@
|
|
1
1
|
## master (unreleased)
|
2
2
|
|
3
|
+
## 0.16.0 (2024-03-28)
|
4
|
+
|
5
|
+
- Add support for asynchronous creation/removal of indexes
|
6
|
+
|
7
|
+
See `docs/background_schema_migrations.md` for the feature description.
|
8
|
+
|
3
9
|
## 0.15.0 (2024-03-19)
|
4
10
|
|
5
11
|
- Reraise errors when running background migrations inline
|
data/README.md
CHANGED
@@ -40,11 +40,11 @@ $ bin/rails generate online_migrations:install
|
|
40
40
|
$ bin/rails db:migrate
|
41
41
|
```
|
42
42
|
|
43
|
-
**Note**: If you do not have plans on using [background migrations](docs/
|
43
|
+
**Note**: If you do not have plans on using [background data migrations](docs/background_data_migrations.md) or [background schema migrations](docs/background_schema_migrations.md) features, then you can delete the generated migration and regenerate it later, if needed.
|
44
44
|
|
45
45
|
### Upgrading
|
46
46
|
|
47
|
-
If you're already using [background migrations](docs/
|
47
|
+
If you're already using [background data migrations](docs/background_data_migrations.md) or [background schema migrations](docs/background_schema_migrations.md), your background migrations tables may require additional columns. After every upgrade run:
|
48
48
|
|
49
49
|
```sh
|
50
50
|
$ bin/rails generate online_migrations:upgrade
|
@@ -285,7 +285,7 @@ end
|
|
285
285
|
```
|
286
286
|
|
287
287
|
**Note**: If you forget `disable_ddl_transaction!`, the migration will fail.
|
288
|
-
**Note**: You may consider [background migrations](#background-migrations) to run data changes on large tables.
|
288
|
+
**Note**: You may consider [background data migrations](#background-data-migrations) or [background schema migrations](#background-schema-migrations) to run data changes on large tables.
|
289
289
|
|
290
290
|
### Changing the type of a column
|
291
291
|
|
@@ -1228,9 +1228,13 @@ Certain methods like `execute` and `change_table` cannot be inspected and are pr
|
|
1228
1228
|
|
1229
1229
|
Read [configuring.md](docs/configuring.md).
|
1230
1230
|
|
1231
|
-
## Background Migrations
|
1231
|
+
## Background Data Migrations
|
1232
1232
|
|
1233
|
-
Read [
|
1233
|
+
Read [background_data_migrations.md](docs/background_data_migrations.md) on how to perform data migrations on large tables.
|
1234
|
+
|
1235
|
+
## Background Schema Migrations
|
1236
|
+
|
1237
|
+
Read [background_schema_migrations.md](docs/background_schema_migrations.md) on how to perform background schema migrations on large tables.
|
1234
1238
|
|
1235
1239
|
## Credits
|
1236
1240
|
|
@@ -1295,7 +1299,7 @@ The main differences are:
|
|
1295
1299
|
* adding different types of constraints
|
1296
1300
|
* and others
|
1297
1301
|
|
1298
|
-
2. This gem has a
|
1302
|
+
2. This gem has a powerful internal framework for running [data migrations](docs/background_data_migrations.md) and [schema migrations](docs/background_schema_migrations.md) on very large tables in background.
|
1299
1303
|
|
1300
1304
|
For example, you can use background migrations to migrate data that’s stored in a single JSON column to a separate table instead; backfill values from one column to another (as one of the steps when changing column type); or backfill some column’s value from an API.
|
1301
1305
|
|
@@ -1,4 +1,4 @@
|
|
1
|
-
# Background Migrations
|
1
|
+
# Background Data Migrations
|
2
2
|
|
3
3
|
When a project grows, your database starts to be heavy and changing the data through the deployment process can be very painful.
|
4
4
|
|
@@ -18,7 +18,7 @@ Start a background migrations scheduler. For example, to run it on cron using [w
|
|
18
18
|
|
19
19
|
```ruby
|
20
20
|
every 1.minute do
|
21
|
-
runner "OnlineMigrations.
|
21
|
+
runner "OnlineMigrations.run_background_data_migrations"
|
22
22
|
end
|
23
23
|
```
|
24
24
|
|
@@ -261,20 +261,6 @@ migration.update!(
|
|
261
261
|
)
|
262
262
|
```
|
263
263
|
|
264
|
-
### Throttling
|
265
|
-
|
266
|
-
Background Migrations often modify a lot of data and can be taxing on your database. There is a throttling mechanism that can be used to throttle a background migration when a given condition is met. If a migration is throttled, it will be interrupted and retried on the next Scheduler cycle run.
|
267
|
-
|
268
|
-
Specify the throttle condition as a block:
|
269
|
-
|
270
|
-
```ruby
|
271
|
-
# config/initializers/online_migrations.rb
|
272
|
-
|
273
|
-
config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
|
274
|
-
```
|
275
|
-
|
276
|
-
Note that it's up to you to define a throttling condition that makes sense for your app. For example, you can check various PostgreSQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
|
277
|
-
|
278
264
|
### Customizing the error handler
|
279
265
|
|
280
266
|
Exceptions raised while a Background Migration is performing are rescued and information about the error is persisted in the database.
|
@@ -321,21 +307,6 @@ config.background_migrations.migrations_module = "BackgroundMigrationsModule"
|
|
321
307
|
|
322
308
|
If no value is specified, it will default to `"OnlineMigrations::BackgroundMigrations"`.
|
323
309
|
|
324
|
-
### Customizing the backtrace cleaner
|
325
|
-
|
326
|
-
`config.background_migrations.backtrace_cleaner` can be configured to specify a backtrace cleaner to use when a Background Migration errors and the backtrace is cleaned and persisted. An `ActiveSupport::BacktraceCleaner` should be used.
|
327
|
-
|
328
|
-
```ruby
|
329
|
-
# config/initializers/online_migrations.rb
|
330
|
-
|
331
|
-
cleaner = ActiveSupport::BacktraceCleaner.new
|
332
|
-
cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
|
333
|
-
|
334
|
-
config.background_migrations.backtrace_cleaner = cleaner
|
335
|
-
```
|
336
|
-
|
337
|
-
If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.
|
338
|
-
|
339
310
|
### Multiple databases and sharding
|
340
311
|
|
341
312
|
If you have multiple databases or sharding, you may need to configure where background migrations related tables live
|
@@ -345,10 +316,10 @@ by configuring the parent model:
|
|
345
316
|
# config/initializers/online_migrations.rb
|
346
317
|
|
347
318
|
# Referring to one of the databases
|
348
|
-
OnlineMigrations::
|
319
|
+
OnlineMigrations::ApplicationRecord.connects_to database: { writing: :animals }
|
349
320
|
|
350
321
|
# Referring to one of the shards (via `:database` option)
|
351
|
-
OnlineMigrations::
|
322
|
+
OnlineMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
|
352
323
|
```
|
353
324
|
|
354
325
|
By default, ActiveRecord uses the database config named `:primary` (if exists) under the environment section from the `database.yml`.
|
@@ -0,0 +1,163 @@
|
|
1
|
+
# Background Schema Migrations
|
2
|
+
|
3
|
+
When a project grows, your database starts to be heavy and performing schema changes through the deployment process can be very painful.
|
4
|
+
|
5
|
+
E.g., for very large tables, index creation can be a challenge to manage. While adding indexes `CONCURRENTLY` creates indexes in a way that does not block ordinary traffic, it can still be problematic when index creation runs for many hours. Necessary database operations like autovacuum cannot run, and the deployment process is usually blocked waiting for index creation to finish.
|
6
|
+
|
7
|
+
**Note**: You probably don't need to use this feature for smaller projects, since performing schema changes directly on smaller databases will be perfectly fine and will not block the deployment too much.
|
8
|
+
|
9
|
+
## Installation
|
10
|
+
|
11
|
+
Make sure you have migration files generated when installed this gem:
|
12
|
+
|
13
|
+
```sh
|
14
|
+
$ bin/rails generate online_migrations:install
|
15
|
+
```
|
16
|
+
|
17
|
+
Start a background migrations scheduler. For example, to run it on cron using [whenever gem](https://github.com/javan/whenever) add the following lines to its `schedule.rb` file:
|
18
|
+
|
19
|
+
```ruby
|
20
|
+
every 1.minute do
|
21
|
+
runner "OnlineMigrations.run_background_schema_migrations"
|
22
|
+
end
|
23
|
+
```
|
24
|
+
|
25
|
+
or run it manually when the deployment is finished, from the rails console:
|
26
|
+
|
27
|
+
```rb
|
28
|
+
[production] (main)> OnlineMigrations.run_background_schema_migrations
|
29
|
+
```
|
30
|
+
|
31
|
+
**Note**: Scheduler will perform only one migration at a time, to not load the database too much. If you enqueued multiple migrations or a migration for multiple shards, you need to call this method a few times.
|
32
|
+
|
33
|
+
**Note**: Make sure that the process that runs the scheduler does not die until the migration is finished.
|
34
|
+
|
35
|
+
## Enqueueing a Background Schema Migration
|
36
|
+
|
37
|
+
Currently, only helpers for adding/removing indexes are provided.
|
38
|
+
|
39
|
+
Background schema migrations should be performed in 2 steps:
|
40
|
+
|
41
|
+
1. Create a PR that schedules the index to be created/removed
|
42
|
+
2. Verify that the PR was deployed and that the index was actually created/removed on production.
|
43
|
+
Create a follow-up PR with a regular migration that creates/removes an index synchronously (will be a no op when run on production) and commit the schema changes for `schema.rb`/`structure.sql`
|
44
|
+
|
45
|
+
To schedule an index creation:
|
46
|
+
|
47
|
+
```ruby
|
48
|
+
# db/migrate/xxxxxxxxxxxxxx_add_index_to_users_email_in_background.rb
|
49
|
+
def up
|
50
|
+
add_index_in_background(:users, :email, unique: true)
|
51
|
+
end
|
52
|
+
```
|
53
|
+
|
54
|
+
To schedule an index removal:
|
55
|
+
|
56
|
+
```ruby
|
57
|
+
# db/migrate/xxxxxxxxxxxxxx_remove_index_from_users_email_in_background.rb
|
58
|
+
def up
|
59
|
+
remove_index_in_background(:users, name: "index_users_on_email")
|
60
|
+
end
|
61
|
+
```
|
62
|
+
|
63
|
+
`add_index_in_background`/`remove_index_in_background` accept additional configuration options which controls how the background schema migration is run. Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_schema_migrations/migration_helpers.rb) for the list of all available configuration options.
|
64
|
+
|
65
|
+
## Instrumentation
|
66
|
+
|
67
|
+
Background schema migrations use the [ActiveSupport::Notifications](http://api.rubyonrails.org/classes/ActiveSupport/Notifications.html) API.
|
68
|
+
|
69
|
+
You can subscribe to `background_schema_migrations` events and log it, graph it, etc.
|
70
|
+
|
71
|
+
To get notified about specific type of events, subscribe to the event name followed by the `background_schema_migrations` namespace. E.g. for retries use:
|
72
|
+
|
73
|
+
```ruby
|
74
|
+
# config/initializers/online_migrations.rb
|
75
|
+
ActiveSupport::Notifications.subscribe("retried.background_schema_migrations") do |name, start, finish, id, payload|
|
76
|
+
# background schema migration object is available in payload[:background_schema_migration]
|
77
|
+
|
78
|
+
# Your code here
|
79
|
+
end
|
80
|
+
```
|
81
|
+
|
82
|
+
If you want to subscribe to every `background_schema_migrations` event, use:
|
83
|
+
|
84
|
+
```ruby
|
85
|
+
# config/initializers/online_migrations.rb
|
86
|
+
ActiveSupport::Notifications.subscribe(/background_schema_migrations/) do |name, start, finish, id, payload|
|
87
|
+
# background schema migration object is available in payload[:background_schema_migration]
|
88
|
+
|
89
|
+
# Your code here
|
90
|
+
end
|
91
|
+
```
|
92
|
+
|
93
|
+
Available events:
|
94
|
+
|
95
|
+
* `started.background_schema_migrations`
|
96
|
+
* `run.background_schema_migrations`
|
97
|
+
* `completed.background_schema_migrations`
|
98
|
+
* `retried.background_schema_migrations`
|
99
|
+
* `throttled.background_schema_migrations`
|
100
|
+
|
101
|
+
## Monitoring Background Schema Migrations
|
102
|
+
|
103
|
+
Background Schema Migrations can be in various states during its execution:
|
104
|
+
|
105
|
+
* **enqueued**: A migration has been enqueued by the user.
|
106
|
+
* **running**: A migration is being performed by a migration executor.
|
107
|
+
* **failed**: A migration raises an exception when running.
|
108
|
+
* **succeeded**: A migration finished without error.
|
109
|
+
|
110
|
+
## Configuring
|
111
|
+
|
112
|
+
There are a few configurable options for the Background Schema Migrations. Custom configurations should be placed in a `online_migrations.rb` initializer.
|
113
|
+
|
114
|
+
Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_schema_migrations/config.rb) for the list of all available configuration options.
|
115
|
+
|
116
|
+
**Note**: You can dynamically change certain migration parameters while the migration is run.
|
117
|
+
For example,
|
118
|
+
```ruby
|
119
|
+
migration = OnlineMigrations::BackgroundSchemaMigrations::Migration.find(id)
|
120
|
+
migration.update!(
|
121
|
+
statement_timeout: 2.hours, # The statement timeout value used when running the migration
|
122
|
+
max_attempts: 10 # The # of attempts the failing migration will be retried
|
123
|
+
)
|
124
|
+
```
|
125
|
+
|
126
|
+
### Customizing the error handler
|
127
|
+
|
128
|
+
Exceptions raised while a Background Schema Migration is performing are rescued and information about the error is persisted in the database.
|
129
|
+
|
130
|
+
If you want to integrate with an exception monitoring service (e.g. Bugsnag), you can define an error handler:
|
131
|
+
|
132
|
+
```ruby
|
133
|
+
# config/initializers/online_migrations.rb
|
134
|
+
|
135
|
+
OnlineMigrations.config.background_schema_migrations.error_handler = ->(error, errored_migration) do
|
136
|
+
Bugsnag.notify(error) do |notification|
|
137
|
+
notification.add_metadata(:background_schema_migration, { name: errored_migration.name })
|
138
|
+
end
|
139
|
+
end
|
140
|
+
```
|
141
|
+
|
142
|
+
The error handler should be a lambda that accepts 2 arguments:
|
143
|
+
|
144
|
+
* `error`: The exception that was raised.
|
145
|
+
* `errored_migration`: An `OnlineMigrations::BackgroundSchemaMigrations::Migration` object that represents a failed migration.
|
146
|
+
|
147
|
+
### Multiple databases and sharding
|
148
|
+
|
149
|
+
If you have multiple databases or sharding, you may need to configure where background migrations related tables live
|
150
|
+
by configuring the parent model:
|
151
|
+
|
152
|
+
```ruby
|
153
|
+
# config/initializers/online_migrations.rb
|
154
|
+
|
155
|
+
# Referring to one of the databases
|
156
|
+
OnlineMigrations::ApplicationRecord.connects_to database: { writing: :animals }
|
157
|
+
|
158
|
+
# Referring to one of the shards (via `:database` option)
|
159
|
+
OnlineMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
|
160
|
+
```
|
161
|
+
|
162
|
+
By default, ActiveRecord uses the database config named `:primary` (if exists) under the environment section from the `database.yml`.
|
163
|
+
Otherwise, the first config under the environment section is used.
|
data/docs/configuring.md
CHANGED
@@ -216,7 +216,7 @@ Add to an initializer file:
|
|
216
216
|
config.auto_analyze = true
|
217
217
|
```
|
218
218
|
|
219
|
-
|
219
|
+
## Running background migrations inline
|
220
220
|
|
221
221
|
`config.run_background_migrations_inline` can be configured with a proc to decide whether background migrations should be run inline. For convenience defaults to true for development and test environments.
|
222
222
|
|
@@ -227,6 +227,35 @@ config.run_background_migrations_inline = -> { Rails.env.local? }
|
|
227
227
|
|
228
228
|
Set to `nil` to avoid running background migrations inline.
|
229
229
|
|
230
|
+
## Throttling
|
231
|
+
|
232
|
+
Background data and schema migrations can be taxing on your database. There is a throttling mechanism that can be used to throttle a background migration when a given condition is met. If a migration is throttled, it will be interrupted and retried on the next Scheduler cycle run.
|
233
|
+
|
234
|
+
Specify the throttle condition as a block:
|
235
|
+
|
236
|
+
```ruby
|
237
|
+
# config/initializers/online_migrations.rb
|
238
|
+
|
239
|
+
OnlineMigrations.config.throttler = -> { DatabaseStatus.unhealthy? }
|
240
|
+
```
|
241
|
+
|
242
|
+
**Note**: It's up to you to define a throttling condition that makes sense for your app. For example, you can check various PostgreSQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
|
243
|
+
|
244
|
+
## Customizing the backtrace cleaner
|
245
|
+
|
246
|
+
`OnlineMigrations.config.backtrace_cleaner` can be configured to specify a backtrace cleaner to use when a background data or schema migration errors and the backtrace is cleaned and persisted. An `ActiveSupport::BacktraceCleaner` should be used.
|
247
|
+
|
248
|
+
```ruby
|
249
|
+
# config/initializers/online_migrations.rb
|
250
|
+
|
251
|
+
cleaner = ActiveSupport::BacktraceCleaner.new
|
252
|
+
cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
|
253
|
+
|
254
|
+
OnlineMigrations.config.backtrace_cleaner = cleaner
|
255
|
+
```
|
256
|
+
|
257
|
+
If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.
|
258
|
+
|
230
259
|
## Schema Sanity
|
231
260
|
|
232
261
|
Columns can flip order in `db/schema.rb` when you have multiple developers. One way to prevent this is to [alphabetize them](https://www.pgrs.net/2008/03/12/alphabetize-schema-rb-columns/).
|
@@ -0,0 +1,29 @@
|
|
1
|
+
class CreateBackgroundSchemaMigrations < <%= migration_parent %>
|
2
|
+
def change
|
3
|
+
# You can remove this migration for now and regenerate it later if you do not have plans
|
4
|
+
# to use background schema migrations, like adding indexes in the background.
|
5
|
+
create_table :background_schema_migrations do |t|
|
6
|
+
t.bigint :parent_id
|
7
|
+
t.string :migration_name, null: false
|
8
|
+
t.string :table_name, null: false
|
9
|
+
t.string :definition, null: false
|
10
|
+
t.string :status, default: "enqueued", null: false
|
11
|
+
t.string :shard
|
12
|
+
t.boolean :composite, default: false, null: false
|
13
|
+
t.integer :statement_timeout
|
14
|
+
t.datetime :started_at
|
15
|
+
t.datetime :finished_at
|
16
|
+
t.integer :max_attempts, null: false
|
17
|
+
t.integer :attempts, default: 0, null: false
|
18
|
+
t.string :error_class
|
19
|
+
t.string :error_message
|
20
|
+
t.string :backtrace, array: true
|
21
|
+
t.string :connection_class_name
|
22
|
+
t.timestamps
|
23
|
+
|
24
|
+
t.foreign_key :background_schema_migrations, column: :parent_id, on_delete: :cascade
|
25
|
+
|
26
|
+
t.index [:migration_name, :shard], unique: true, name: :index_background_schema_migrations_on_unique_configuration
|
27
|
+
end
|
28
|
+
end
|
29
|
+
end
|
@@ -77,12 +77,20 @@ OnlineMigrations.configure do |config|
|
|
77
77
|
# end
|
78
78
|
# end
|
79
79
|
|
80
|
-
# Decide whether background migrations should be run inline.
|
80
|
+
# Decide whether background data and schema migrations should be run inline.
|
81
81
|
# Convenient for development and test environments.
|
82
82
|
config.run_background_migrations_inline = -> { Rails.env.local? }
|
83
83
|
|
84
|
-
#
|
84
|
+
# Configure custom throttler for background data and schema migrations.
|
85
|
+
# It will be called before each run.
|
86
|
+
# If throttled, the current run will be retried next time.
|
87
|
+
# config.throttler = -> { DatabaseStatus.unhealthy? }
|
88
|
+
|
89
|
+
# The Active Support backtrace cleaner that will be used to clean the
|
90
|
+
# backtrace of a background data or schema migration that errors.
|
91
|
+
config.backtrace_cleaner = Rails.backtrace_cleaner
|
85
92
|
|
93
|
+
# ==> Background data migrations configuration
|
86
94
|
# The path where generated background migrations will be placed.
|
87
95
|
# config.background_migrations.migrations_path = "lib"
|
88
96
|
|
@@ -105,22 +113,27 @@ OnlineMigrations.configure do |config|
|
|
105
113
|
# When attempts are exhausted, the individual batch is marked as failed.
|
106
114
|
# config.background_migrations.batch_max_attempts = 5
|
107
115
|
|
108
|
-
# Configure custom throttler for background migrations.
|
109
|
-
# It will be called before each batch run.
|
110
|
-
# If throttled, the current run will be retried next time.
|
111
|
-
# config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
|
112
|
-
|
113
116
|
# The number of seconds that must pass before the running job is considered stuck.
|
114
117
|
# config.background_migrations.stuck_jobs_timeout = 1.hour
|
115
118
|
|
116
|
-
# The Active Support backtrace cleaner that will be used to clean the
|
117
|
-
# backtrace of a migration job that errors.
|
118
|
-
config.background_migrations.backtrace_cleaner = Rails.backtrace_cleaner
|
119
|
-
|
120
119
|
# The callback to perform when an error occurs in the migration job.
|
121
120
|
# config.background_migrations.error_handler = ->(error, errored_job) do
|
122
121
|
# Bugsnag.notify(error) do |notification|
|
123
122
|
# notification.add_metadata(:background_migration, { name: errored_job.migration_name })
|
124
123
|
# end
|
125
124
|
# end
|
125
|
+
|
126
|
+
# ==> Background schema migrations configuration
|
127
|
+
# When attempts are exhausted, the failing migration stops to be retried.
|
128
|
+
# config.background_schema_migrations.max_attempts = 5
|
129
|
+
|
130
|
+
# Statement timeout value used when running background schema migration.
|
131
|
+
# config.background_schema_migrations.statement_timeout = 1.hour
|
132
|
+
|
133
|
+
# The callback to perform when an error occurs during the background schema migration.
|
134
|
+
# config.background_schema_migrations.error_handler = ->(error, errored_migration) do
|
135
|
+
# Bugsnag.notify(error) do |notification|
|
136
|
+
# notification.add_metadata(:background_schema_migration, { name: errored_migration.name })
|
137
|
+
# end
|
138
|
+
# end
|
126
139
|
end
|
@@ -47,5 +47,29 @@ class InstallOnlineMigrations < <%= migration_parent %>
|
|
47
47
|
t.index [:migration_id, :status, :updated_at], name: :index_background_migration_jobs_on_updated_at
|
48
48
|
t.index [:migration_id, :finished_at], name: :index_background_migration_jobs_on_finished_at
|
49
49
|
end
|
50
|
+
|
51
|
+
create_table :background_schema_migrations do |t|
|
52
|
+
t.bigint :parent_id
|
53
|
+
t.string :migration_name, null: false
|
54
|
+
t.string :table_name, null: false
|
55
|
+
t.string :definition, null: false
|
56
|
+
t.string :status, default: "enqueued", null: false
|
57
|
+
t.string :shard
|
58
|
+
t.boolean :composite, default: false, null: false
|
59
|
+
t.integer :statement_timeout
|
60
|
+
t.datetime :started_at
|
61
|
+
t.datetime :finished_at
|
62
|
+
t.integer :max_attempts, null: false
|
63
|
+
t.integer :attempts, default: 0, null: false
|
64
|
+
t.string :error_class
|
65
|
+
t.string :error_message
|
66
|
+
t.string :backtrace, array: true
|
67
|
+
t.string :connection_class_name
|
68
|
+
t.timestamps
|
69
|
+
|
70
|
+
t.foreign_key :background_schema_migrations, column: :parent_id, on_delete: :cascade
|
71
|
+
|
72
|
+
t.index [:migration_name, :shard], unique: true, name: :index_background_schema_migrations_on_unique_configuration
|
73
|
+
end
|
50
74
|
end
|
51
75
|
end
|
@@ -23,6 +23,11 @@ module OnlineMigrations
|
|
23
23
|
|
24
24
|
migrations = []
|
25
25
|
migrations << "add_sharding_to_online_migrations" if !columns.include?("shard")
|
26
|
+
|
27
|
+
if !connection.table_exists?(BackgroundSchemaMigrations::Migration.table_name)
|
28
|
+
migrations << "create_background_schema_migrations"
|
29
|
+
end
|
30
|
+
|
26
31
|
migrations
|
27
32
|
end
|
28
33
|
|
@@ -0,0 +1,11 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module OnlineMigrations
|
4
|
+
# Base class for all records used by this gem.
|
5
|
+
#
|
6
|
+
# Can be extended to setup different database where all tables related to
|
7
|
+
# online_migrations will live.
|
8
|
+
class ApplicationRecord < ActiveRecord::Base
|
9
|
+
self.abstract_class = true
|
10
|
+
end
|
11
|
+
end
|
@@ -39,17 +39,13 @@ module OnlineMigrations
|
|
39
39
|
#
|
40
40
|
attr_accessor :batch_max_attempts
|
41
41
|
|
42
|
-
|
43
|
-
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
# @example
|
50
|
-
# OnlineMigrations.config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
|
51
|
-
#
|
52
|
-
attr_reader :throttler
|
42
|
+
def throttler
|
43
|
+
OnlineMigrations.deprecator.warn(<<~MSG)
|
44
|
+
`config.background_migrations.throttler` is deprecated and will be removed.
|
45
|
+
Use `config.throttler` instead.
|
46
|
+
MSG
|
47
|
+
OnlineMigrations.config.throttler
|
48
|
+
end
|
53
49
|
|
54
50
|
# The number of seconds that must pass before the running job is considered stuck
|
55
51
|
#
|
@@ -57,13 +53,21 @@ module OnlineMigrations
|
|
57
53
|
#
|
58
54
|
attr_accessor :stuck_jobs_timeout
|
59
55
|
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
65
|
-
|
66
|
-
|
56
|
+
def backtrace_cleaner
|
57
|
+
OnlineMigrations.deprecator.warn(<<~MSG)
|
58
|
+
`config.background_migrations.backtrace_cleaner` is deprecated and will be removed.
|
59
|
+
Use `config.backtrace_cleaner` instead.
|
60
|
+
MSG
|
61
|
+
OnlineMigrations.config.backtrace_cleaner
|
62
|
+
end
|
63
|
+
|
64
|
+
def backtrace_cleaner=(value)
|
65
|
+
OnlineMigrations.deprecator.warn(<<~MSG)
|
66
|
+
`config.background_migrations.backtrace_cleaner=` is deprecated and will be removed.
|
67
|
+
Use `config.backtrace_cleaner=` instead.
|
68
|
+
MSG
|
69
|
+
OnlineMigrations.config.backtrace_cleaner = value
|
70
|
+
end
|
67
71
|
|
68
72
|
# The callback to perform when an error occurs in the migration job.
|
69
73
|
#
|
@@ -86,17 +90,16 @@ module OnlineMigrations
|
|
86
90
|
@batch_pause = 0.seconds
|
87
91
|
@sub_batch_pause_ms = 100
|
88
92
|
@batch_max_attempts = 5
|
89
|
-
@throttler = -> { false }
|
90
93
|
@stuck_jobs_timeout = 1.hour
|
91
94
|
@error_handler = ->(error, errored_job) {}
|
92
95
|
end
|
93
96
|
|
94
97
|
def throttler=(value)
|
95
|
-
|
96
|
-
|
97
|
-
|
98
|
-
|
99
|
-
|
98
|
+
OnlineMigrations.deprecator.warn(<<~MSG)
|
99
|
+
`config.background_migrations.throttler=` is deprecated and will be removed.
|
100
|
+
Use `config.throttler=` instead.
|
101
|
+
MSG
|
102
|
+
OnlineMigrations.config.throttler = value
|
100
103
|
end
|
101
104
|
end
|
102
105
|
end
|
@@ -181,7 +181,7 @@ module OnlineMigrations
|
|
181
181
|
|
182
182
|
# @private
|
183
183
|
def on_shard(&block)
|
184
|
-
abstract_class =
|
184
|
+
abstract_class = Utils.find_connection_class(migration_model)
|
185
185
|
|
186
186
|
shard = (self.shard || abstract_class.default_shard).to_sym
|
187
187
|
abstract_class.connected_to(shard: shard, role: :writing, &block)
|
@@ -290,13 +290,6 @@ module OnlineMigrations
|
|
290
290
|
min_value
|
291
291
|
end
|
292
292
|
end
|
293
|
-
|
294
|
-
def find_abstract_class(model)
|
295
|
-
model.ancestors.find do |parent|
|
296
|
-
parent == ActiveRecord::Base ||
|
297
|
-
(parent.is_a?(Class) && parent.abstract_class?)
|
298
|
-
end
|
299
|
-
end
|
300
293
|
end
|
301
294
|
end
|
302
295
|
end
|
@@ -35,7 +35,7 @@ module OnlineMigrations
|
|
35
35
|
|
36
36
|
migration_job.update!(status: :succeeded, finished_at: Time.current)
|
37
37
|
rescue Exception => e # rubocop:disable Lint/RescueException
|
38
|
-
backtrace_cleaner = ::OnlineMigrations.config.
|
38
|
+
backtrace_cleaner = ::OnlineMigrations.config.backtrace_cleaner
|
39
39
|
|
40
40
|
migration_job.update!(
|
41
41
|
status: :failed,
|
@@ -0,0 +1,40 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module OnlineMigrations
|
4
|
+
module BackgroundSchemaMigrations
|
5
|
+
# Class representing configuration options for background schema migrations.
|
6
|
+
class Config
|
7
|
+
# Maximum number of run attempts
|
8
|
+
#
|
9
|
+
# When attempts are exhausted, the migration is marked as failed.
|
10
|
+
# @return [Integer] defaults to 5
|
11
|
+
#
|
12
|
+
attr_accessor :max_attempts
|
13
|
+
|
14
|
+
# Statement timeout value used when running background schema migration.
|
15
|
+
#
|
16
|
+
# @return [Integer] defaults to 1 hour
|
17
|
+
#
|
18
|
+
attr_accessor :statement_timeout
|
19
|
+
|
20
|
+
# The callback to perform when an error occurs in the migration.
|
21
|
+
#
|
22
|
+
# @example
|
23
|
+
# OnlineMigrations.config.background_schema_migrations.error_handler = ->(error, errored_migration) do
|
24
|
+
# Bugsnag.notify(error) do |notification|
|
25
|
+
# notification.add_metadata(:background_schema_migration, { name: errored_migration.name })
|
26
|
+
# end
|
27
|
+
# end
|
28
|
+
#
|
29
|
+
# @return [Proc] the callback to perform when an error occurs in the migration
|
30
|
+
#
|
31
|
+
attr_accessor :error_handler
|
32
|
+
|
33
|
+
def initialize
|
34
|
+
@max_attempts = 5
|
35
|
+
@statement_timeout = 1.hour
|
36
|
+
@error_handler = ->(error, errored_migration) {}
|
37
|
+
end
|
38
|
+
end
|
39
|
+
end
|
40
|
+
end
|