online_migrations 0.15.0 → 0.16.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (28) hide show
  1. checksums.yaml +4 -4
  2. data/CHANGELOG.md +6 -0
  3. data/README.md +10 -6
  4. data/docs/{background_migrations.md → background_data_migrations.md} +4 -33
  5. data/docs/background_schema_migrations.md +163 -0
  6. data/docs/configuring.md +30 -1
  7. data/lib/generators/online_migrations/templates/create_background_schema_migrations.rb.tt +29 -0
  8. data/lib/generators/online_migrations/templates/initializer.rb.tt +24 -11
  9. data/lib/generators/online_migrations/templates/install_migration.rb.tt +24 -0
  10. data/lib/generators/online_migrations/upgrade_generator.rb +5 -0
  11. data/lib/online_migrations/application_record.rb +11 -0
  12. data/lib/online_migrations/background_migrations/config.rb +27 -24
  13. data/lib/online_migrations/background_migrations/migration.rb +1 -8
  14. data/lib/online_migrations/background_migrations/migration_job_runner.rb +1 -1
  15. data/lib/online_migrations/background_migrations/migration_runner.rb +1 -1
  16. data/lib/online_migrations/background_schema_migrations/config.rb +40 -0
  17. data/lib/online_migrations/background_schema_migrations/migration.rb +205 -0
  18. data/lib/online_migrations/background_schema_migrations/migration_helpers.rb +76 -0
  19. data/lib/online_migrations/background_schema_migrations/migration_runner.rb +110 -0
  20. data/lib/online_migrations/background_schema_migrations/migration_status_validator.rb +33 -0
  21. data/lib/online_migrations/background_schema_migrations/scheduler.rb +30 -0
  22. data/lib/online_migrations/config.rb +32 -0
  23. data/lib/online_migrations/schema_statements.rb +1 -0
  24. data/lib/online_migrations/utils.rb +7 -0
  25. data/lib/online_migrations/version.rb +1 -1
  26. data/lib/online_migrations.rb +19 -2
  27. metadata +12 -4
  28. data/lib/online_migrations/background_migrations/application_record.rb +0 -13
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2b1b826df875fae97622c702205853da01d67153172ef73fbb0edf28a2f7ff60
4
- data.tar.gz: 18ae161255543e7ab7c266efc94f1ca166a344aa18b92dd4ab2f37209fb42547
3
+ metadata.gz: 48ddbce66257c9010d8ab7361d8860cd898854207b5a1f4c6aa2b6b797aec2e1
4
+ data.tar.gz: c21fbb5f535f788e13068e339eebe77f05a378c68c00244c4522a10d50a28a0d
5
5
  SHA512:
6
- metadata.gz: 705e4ce816bd8b4fcbcb78871589274bdf37acca01fbdf031883e87403a403782d28990ebed10acdf57adcbf3fb4e30e1dbf187123029bf6b1b7d31f26ccd3ca
7
- data.tar.gz: 60ea354440b96645c5c03345d60793a66ba61d8aa1b946737098bc1449ba67e0b24caeb4581c525b3b014c5dd44b0ac28e0c539c8e3454ad67a3a1610cffdd79
6
+ metadata.gz: a5bbff2a8cca45e5fce79d653db3246bc41a4ab9d314f44131bbf74f2659d610fdaf3e87ac5e22baf7928d0cdb481e19eacaa1ed8945e4ee32f1afac3223273f
7
+ data.tar.gz: 6e3861f420327e0bccec7ab8e7c0a508e7cf224312c366297834091467d7cdb5e4fac8a305dbb59f7c0650d199d9a17062ab0408fd180b6fbaa05946bfd27545
data/CHANGELOG.md CHANGED
@@ -1,5 +1,11 @@
1
1
  ## master (unreleased)
2
2
 
3
+ ## 0.16.0 (2024-03-28)
4
+
5
+ - Add support for asynchronous creation/removal of indexes
6
+
7
+ See `docs/background_schema_migrations.md` for the feature description.
8
+
3
9
  ## 0.15.0 (2024-03-19)
4
10
 
5
11
  - Reraise errors when running background migrations inline
data/README.md CHANGED
@@ -40,11 +40,11 @@ $ bin/rails generate online_migrations:install
40
40
  $ bin/rails db:migrate
41
41
  ```
42
42
 
43
- **Note**: If you do not have plans on using [background migrations](docs/background_migrations.md) feature, then you can delete the generated migration and regenerate it later, if needed.
43
+ **Note**: If you do not have plans on using [background data migrations](docs/background_data_migrations.md) or [background schema migrations](docs/background_schema_migrations.md) features, then you can delete the generated migration and regenerate it later, if needed.
44
44
 
45
45
  ### Upgrading
46
46
 
47
- If you're already using [background migrations](docs/background_migrations.md), your background migrations tables may require additional columns. After every upgrade run:
47
+ If you're already using [background data migrations](docs/background_data_migrations.md) or [background schema migrations](docs/background_schema_migrations.md), your background migrations tables may require additional columns. After every upgrade run:
48
48
 
49
49
  ```sh
50
50
  $ bin/rails generate online_migrations:upgrade
@@ -285,7 +285,7 @@ end
285
285
  ```
286
286
 
287
287
  **Note**: If you forget `disable_ddl_transaction!`, the migration will fail.
288
- **Note**: You may consider [background migrations](#background-migrations) to run data changes on large tables.
288
+ **Note**: You may consider [background data migrations](#background-data-migrations) or [background schema migrations](#background-schema-migrations) to run data changes on large tables.
289
289
 
290
290
  ### Changing the type of a column
291
291
 
@@ -1228,9 +1228,13 @@ Certain methods like `execute` and `change_table` cannot be inspected and are pr
1228
1228
 
1229
1229
  Read [configuring.md](docs/configuring.md).
1230
1230
 
1231
- ## Background Migrations
1231
+ ## Background Data Migrations
1232
1232
 
1233
- Read [background_migrations.md](docs/background_migrations.md) on how to perform data migrations on large tables.
1233
+ Read [background_data_migrations.md](docs/background_data_migrations.md) on how to perform data migrations on large tables.
1234
+
1235
+ ## Background Schema Migrations
1236
+
1237
+ Read [background_schema_migrations.md](docs/background_schema_migrations.md) on how to perform background schema migrations on large tables.
1234
1238
 
1235
1239
  ## Credits
1236
1240
 
@@ -1295,7 +1299,7 @@ The main differences are:
1295
1299
  * adding different types of constraints
1296
1300
  * and others
1297
1301
 
1298
- 2. This gem has a [powerful internal framework](https://github.com/fatkodima/online_migrations/blob/master/docs/background_migrations.md) for running data migrations on very large tables using background migrations.
1302
+ 2. This gem has a powerful internal framework for running [data migrations](docs/background_data_migrations.md) and [schema migrations](docs/background_schema_migrations.md) on very large tables in background.
1299
1303
 
1300
1304
  For example, you can use background migrations to migrate data that’s stored in a single JSON column to a separate table instead; backfill values from one column to another (as one of the steps when changing column type); or backfill some column’s value from an API.
1301
1305
 
@@ -1,4 +1,4 @@
1
- # Background Migrations
1
+ # Background Data Migrations
2
2
 
3
3
  When a project grows, your database starts to be heavy and changing the data through the deployment process can be very painful.
4
4
 
@@ -18,7 +18,7 @@ Start a background migrations scheduler. For example, to run it on cron using [w
18
18
 
19
19
  ```ruby
20
20
  every 1.minute do
21
- runner "OnlineMigrations.run_background_migrations"
21
+ runner "OnlineMigrations.run_background_data_migrations"
22
22
  end
23
23
  ```
24
24
 
@@ -261,20 +261,6 @@ migration.update!(
261
261
  )
262
262
  ```
263
263
 
264
- ### Throttling
265
-
266
- Background Migrations often modify a lot of data and can be taxing on your database. There is a throttling mechanism that can be used to throttle a background migration when a given condition is met. If a migration is throttled, it will be interrupted and retried on the next Scheduler cycle run.
267
-
268
- Specify the throttle condition as a block:
269
-
270
- ```ruby
271
- # config/initializers/online_migrations.rb
272
-
273
- config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
274
- ```
275
-
276
- Note that it's up to you to define a throttling condition that makes sense for your app. For example, you can check various PostgreSQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
277
-
278
264
  ### Customizing the error handler
279
265
 
280
266
  Exceptions raised while a Background Migration is performing are rescued and information about the error is persisted in the database.
@@ -321,21 +307,6 @@ config.background_migrations.migrations_module = "BackgroundMigrationsModule"
321
307
 
322
308
  If no value is specified, it will default to `"OnlineMigrations::BackgroundMigrations"`.
323
309
 
324
- ### Customizing the backtrace cleaner
325
-
326
- `config.background_migrations.backtrace_cleaner` can be configured to specify a backtrace cleaner to use when a Background Migration errors and the backtrace is cleaned and persisted. An `ActiveSupport::BacktraceCleaner` should be used.
327
-
328
- ```ruby
329
- # config/initializers/online_migrations.rb
330
-
331
- cleaner = ActiveSupport::BacktraceCleaner.new
332
- cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
333
-
334
- config.background_migrations.backtrace_cleaner = cleaner
335
- ```
336
-
337
- If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.
338
-
339
310
  ### Multiple databases and sharding
340
311
 
341
312
  If you have multiple databases or sharding, you may need to configure where background migrations related tables live
@@ -345,10 +316,10 @@ by configuring the parent model:
345
316
  # config/initializers/online_migrations.rb
346
317
 
347
318
  # Referring to one of the databases
348
- OnlineMigrations::BackgroundMigrations::ApplicationRecord.connects_to database: { writing: :animals }
319
+ OnlineMigrations::ApplicationRecord.connects_to database: { writing: :animals }
349
320
 
350
321
  # Referring to one of the shards (via `:database` option)
351
- OnlineMigrations::BackgroundMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
322
+ OnlineMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
352
323
  ```
353
324
 
354
325
  By default, ActiveRecord uses the database config named `:primary` (if exists) under the environment section from the `database.yml`.
@@ -0,0 +1,163 @@
1
+ # Background Schema Migrations
2
+
3
+ When a project grows, your database starts to be heavy and performing schema changes through the deployment process can be very painful.
4
+
5
+ E.g., for very large tables, index creation can be a challenge to manage. While adding indexes `CONCURRENTLY` creates indexes in a way that does not block ordinary traffic, it can still be problematic when index creation runs for many hours. Necessary database operations like autovacuum cannot run, and the deployment process is usually blocked waiting for index creation to finish.
6
+
7
+ **Note**: You probably don't need to use this feature for smaller projects, since performing schema changes directly on smaller databases will be perfectly fine and will not block the deployment too much.
8
+
9
+ ## Installation
10
+
11
+ Make sure you have migration files generated when installed this gem:
12
+
13
+ ```sh
14
+ $ bin/rails generate online_migrations:install
15
+ ```
16
+
17
+ Start a background migrations scheduler. For example, to run it on cron using [whenever gem](https://github.com/javan/whenever) add the following lines to its `schedule.rb` file:
18
+
19
+ ```ruby
20
+ every 1.minute do
21
+ runner "OnlineMigrations.run_background_schema_migrations"
22
+ end
23
+ ```
24
+
25
+ or run it manually when the deployment is finished, from the rails console:
26
+
27
+ ```rb
28
+ [production] (main)> OnlineMigrations.run_background_schema_migrations
29
+ ```
30
+
31
+ **Note**: Scheduler will perform only one migration at a time, to not load the database too much. If you enqueued multiple migrations or a migration for multiple shards, you need to call this method a few times.
32
+
33
+ **Note**: Make sure that the process that runs the scheduler does not die until the migration is finished.
34
+
35
+ ## Enqueueing a Background Schema Migration
36
+
37
+ Currently, only helpers for adding/removing indexes are provided.
38
+
39
+ Background schema migrations should be performed in 2 steps:
40
+
41
+ 1. Create a PR that schedules the index to be created/removed
42
+ 2. Verify that the PR was deployed and that the index was actually created/removed on production.
43
+ Create a follow-up PR with a regular migration that creates/removes an index synchronously (will be a no op when run on production) and commit the schema changes for `schema.rb`/`structure.sql`
44
+
45
+ To schedule an index creation:
46
+
47
+ ```ruby
48
+ # db/migrate/xxxxxxxxxxxxxx_add_index_to_users_email_in_background.rb
49
+ def up
50
+ add_index_in_background(:users, :email, unique: true)
51
+ end
52
+ ```
53
+
54
+ To schedule an index removal:
55
+
56
+ ```ruby
57
+ # db/migrate/xxxxxxxxxxxxxx_remove_index_from_users_email_in_background.rb
58
+ def up
59
+ remove_index_in_background(:users, name: "index_users_on_email")
60
+ end
61
+ ```
62
+
63
+ `add_index_in_background`/`remove_index_in_background` accept additional configuration options which controls how the background schema migration is run. Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_schema_migrations/migration_helpers.rb) for the list of all available configuration options.
64
+
65
+ ## Instrumentation
66
+
67
+ Background schema migrations use the [ActiveSupport::Notifications](http://api.rubyonrails.org/classes/ActiveSupport/Notifications.html) API.
68
+
69
+ You can subscribe to `background_schema_migrations` events and log it, graph it, etc.
70
+
71
+ To get notified about specific type of events, subscribe to the event name followed by the `background_schema_migrations` namespace. E.g. for retries use:
72
+
73
+ ```ruby
74
+ # config/initializers/online_migrations.rb
75
+ ActiveSupport::Notifications.subscribe("retried.background_schema_migrations") do |name, start, finish, id, payload|
76
+ # background schema migration object is available in payload[:background_schema_migration]
77
+
78
+ # Your code here
79
+ end
80
+ ```
81
+
82
+ If you want to subscribe to every `background_schema_migrations` event, use:
83
+
84
+ ```ruby
85
+ # config/initializers/online_migrations.rb
86
+ ActiveSupport::Notifications.subscribe(/background_schema_migrations/) do |name, start, finish, id, payload|
87
+ # background schema migration object is available in payload[:background_schema_migration]
88
+
89
+ # Your code here
90
+ end
91
+ ```
92
+
93
+ Available events:
94
+
95
+ * `started.background_schema_migrations`
96
+ * `run.background_schema_migrations`
97
+ * `completed.background_schema_migrations`
98
+ * `retried.background_schema_migrations`
99
+ * `throttled.background_schema_migrations`
100
+
101
+ ## Monitoring Background Schema Migrations
102
+
103
+ Background Schema Migrations can be in various states during its execution:
104
+
105
+ * **enqueued**: A migration has been enqueued by the user.
106
+ * **running**: A migration is being performed by a migration executor.
107
+ * **failed**: A migration raises an exception when running.
108
+ * **succeeded**: A migration finished without error.
109
+
110
+ ## Configuring
111
+
112
+ There are a few configurable options for the Background Schema Migrations. Custom configurations should be placed in a `online_migrations.rb` initializer.
113
+
114
+ Check the [source code](https://github.com/fatkodima/online_migrations/blob/master/lib/online_migrations/background_schema_migrations/config.rb) for the list of all available configuration options.
115
+
116
+ **Note**: You can dynamically change certain migration parameters while the migration is run.
117
+ For example,
118
+ ```ruby
119
+ migration = OnlineMigrations::BackgroundSchemaMigrations::Migration.find(id)
120
+ migration.update!(
121
+ statement_timeout: 2.hours, # The statement timeout value used when running the migration
122
+ max_attempts: 10 # The # of attempts the failing migration will be retried
123
+ )
124
+ ```
125
+
126
+ ### Customizing the error handler
127
+
128
+ Exceptions raised while a Background Schema Migration is performing are rescued and information about the error is persisted in the database.
129
+
130
+ If you want to integrate with an exception monitoring service (e.g. Bugsnag), you can define an error handler:
131
+
132
+ ```ruby
133
+ # config/initializers/online_migrations.rb
134
+
135
+ OnlineMigrations.config.background_schema_migrations.error_handler = ->(error, errored_migration) do
136
+ Bugsnag.notify(error) do |notification|
137
+ notification.add_metadata(:background_schema_migration, { name: errored_migration.name })
138
+ end
139
+ end
140
+ ```
141
+
142
+ The error handler should be a lambda that accepts 2 arguments:
143
+
144
+ * `error`: The exception that was raised.
145
+ * `errored_migration`: An `OnlineMigrations::BackgroundSchemaMigrations::Migration` object that represents a failed migration.
146
+
147
+ ### Multiple databases and sharding
148
+
149
+ If you have multiple databases or sharding, you may need to configure where background migrations related tables live
150
+ by configuring the parent model:
151
+
152
+ ```ruby
153
+ # config/initializers/online_migrations.rb
154
+
155
+ # Referring to one of the databases
156
+ OnlineMigrations::ApplicationRecord.connects_to database: { writing: :animals }
157
+
158
+ # Referring to one of the shards (via `:database` option)
159
+ OnlineMigrations::ApplicationRecord.connects_to database: { writing: :shard_one }
160
+ ```
161
+
162
+ By default, ActiveRecord uses the database config named `:primary` (if exists) under the environment section from the `database.yml`.
163
+ Otherwise, the first config under the environment section is used.
data/docs/configuring.md CHANGED
@@ -216,7 +216,7 @@ Add to an initializer file:
216
216
  config.auto_analyze = true
217
217
  ```
218
218
 
219
- ### Running background migrations inline
219
+ ## Running background migrations inline
220
220
 
221
221
  `config.run_background_migrations_inline` can be configured with a proc to decide whether background migrations should be run inline. For convenience defaults to true for development and test environments.
222
222
 
@@ -227,6 +227,35 @@ config.run_background_migrations_inline = -> { Rails.env.local? }
227
227
 
228
228
  Set to `nil` to avoid running background migrations inline.
229
229
 
230
+ ## Throttling
231
+
232
+ Background data and schema migrations can be taxing on your database. There is a throttling mechanism that can be used to throttle a background migration when a given condition is met. If a migration is throttled, it will be interrupted and retried on the next Scheduler cycle run.
233
+
234
+ Specify the throttle condition as a block:
235
+
236
+ ```ruby
237
+ # config/initializers/online_migrations.rb
238
+
239
+ OnlineMigrations.config.throttler = -> { DatabaseStatus.unhealthy? }
240
+ ```
241
+
242
+ **Note**: It's up to you to define a throttling condition that makes sense for your app. For example, you can check various PostgreSQL metrics such as replication lag, DB threads, whether DB writes are available, etc.
243
+
244
+ ## Customizing the backtrace cleaner
245
+
246
+ `OnlineMigrations.config.backtrace_cleaner` can be configured to specify a backtrace cleaner to use when a background data or schema migration errors and the backtrace is cleaned and persisted. An `ActiveSupport::BacktraceCleaner` should be used.
247
+
248
+ ```ruby
249
+ # config/initializers/online_migrations.rb
250
+
251
+ cleaner = ActiveSupport::BacktraceCleaner.new
252
+ cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
253
+
254
+ OnlineMigrations.config.backtrace_cleaner = cleaner
255
+ ```
256
+
257
+ If none is specified, the default `Rails.backtrace_cleaner` will be used to clean backtraces.
258
+
230
259
  ## Schema Sanity
231
260
 
232
261
  Columns can flip order in `db/schema.rb` when you have multiple developers. One way to prevent this is to [alphabetize them](https://www.pgrs.net/2008/03/12/alphabetize-schema-rb-columns/).
@@ -0,0 +1,29 @@
1
+ class CreateBackgroundSchemaMigrations < <%= migration_parent %>
2
+ def change
3
+ # You can remove this migration for now and regenerate it later if you do not have plans
4
+ # to use background schema migrations, like adding indexes in the background.
5
+ create_table :background_schema_migrations do |t|
6
+ t.bigint :parent_id
7
+ t.string :migration_name, null: false
8
+ t.string :table_name, null: false
9
+ t.string :definition, null: false
10
+ t.string :status, default: "enqueued", null: false
11
+ t.string :shard
12
+ t.boolean :composite, default: false, null: false
13
+ t.integer :statement_timeout
14
+ t.datetime :started_at
15
+ t.datetime :finished_at
16
+ t.integer :max_attempts, null: false
17
+ t.integer :attempts, default: 0, null: false
18
+ t.string :error_class
19
+ t.string :error_message
20
+ t.string :backtrace, array: true
21
+ t.string :connection_class_name
22
+ t.timestamps
23
+
24
+ t.foreign_key :background_schema_migrations, column: :parent_id, on_delete: :cascade
25
+
26
+ t.index [:migration_name, :shard], unique: true, name: :index_background_schema_migrations_on_unique_configuration
27
+ end
28
+ end
29
+ end
@@ -77,12 +77,20 @@ OnlineMigrations.configure do |config|
77
77
  # end
78
78
  # end
79
79
 
80
- # Decide whether background migrations should be run inline.
80
+ # Decide whether background data and schema migrations should be run inline.
81
81
  # Convenient for development and test environments.
82
82
  config.run_background_migrations_inline = -> { Rails.env.local? }
83
83
 
84
- # ==> Background migrations configuration
84
+ # Configure custom throttler for background data and schema migrations.
85
+ # It will be called before each run.
86
+ # If throttled, the current run will be retried next time.
87
+ # config.throttler = -> { DatabaseStatus.unhealthy? }
88
+
89
+ # The Active Support backtrace cleaner that will be used to clean the
90
+ # backtrace of a background data or schema migration that errors.
91
+ config.backtrace_cleaner = Rails.backtrace_cleaner
85
92
 
93
+ # ==> Background data migrations configuration
86
94
  # The path where generated background migrations will be placed.
87
95
  # config.background_migrations.migrations_path = "lib"
88
96
 
@@ -105,22 +113,27 @@ OnlineMigrations.configure do |config|
105
113
  # When attempts are exhausted, the individual batch is marked as failed.
106
114
  # config.background_migrations.batch_max_attempts = 5
107
115
 
108
- # Configure custom throttler for background migrations.
109
- # It will be called before each batch run.
110
- # If throttled, the current run will be retried next time.
111
- # config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
112
-
113
116
  # The number of seconds that must pass before the running job is considered stuck.
114
117
  # config.background_migrations.stuck_jobs_timeout = 1.hour
115
118
 
116
- # The Active Support backtrace cleaner that will be used to clean the
117
- # backtrace of a migration job that errors.
118
- config.background_migrations.backtrace_cleaner = Rails.backtrace_cleaner
119
-
120
119
  # The callback to perform when an error occurs in the migration job.
121
120
  # config.background_migrations.error_handler = ->(error, errored_job) do
122
121
  # Bugsnag.notify(error) do |notification|
123
122
  # notification.add_metadata(:background_migration, { name: errored_job.migration_name })
124
123
  # end
125
124
  # end
125
+
126
+ # ==> Background schema migrations configuration
127
+ # When attempts are exhausted, the failing migration stops to be retried.
128
+ # config.background_schema_migrations.max_attempts = 5
129
+
130
+ # Statement timeout value used when running background schema migration.
131
+ # config.background_schema_migrations.statement_timeout = 1.hour
132
+
133
+ # The callback to perform when an error occurs during the background schema migration.
134
+ # config.background_schema_migrations.error_handler = ->(error, errored_migration) do
135
+ # Bugsnag.notify(error) do |notification|
136
+ # notification.add_metadata(:background_schema_migration, { name: errored_migration.name })
137
+ # end
138
+ # end
126
139
  end
@@ -47,5 +47,29 @@ class InstallOnlineMigrations < <%= migration_parent %>
47
47
  t.index [:migration_id, :status, :updated_at], name: :index_background_migration_jobs_on_updated_at
48
48
  t.index [:migration_id, :finished_at], name: :index_background_migration_jobs_on_finished_at
49
49
  end
50
+
51
+ create_table :background_schema_migrations do |t|
52
+ t.bigint :parent_id
53
+ t.string :migration_name, null: false
54
+ t.string :table_name, null: false
55
+ t.string :definition, null: false
56
+ t.string :status, default: "enqueued", null: false
57
+ t.string :shard
58
+ t.boolean :composite, default: false, null: false
59
+ t.integer :statement_timeout
60
+ t.datetime :started_at
61
+ t.datetime :finished_at
62
+ t.integer :max_attempts, null: false
63
+ t.integer :attempts, default: 0, null: false
64
+ t.string :error_class
65
+ t.string :error_message
66
+ t.string :backtrace, array: true
67
+ t.string :connection_class_name
68
+ t.timestamps
69
+
70
+ t.foreign_key :background_schema_migrations, column: :parent_id, on_delete: :cascade
71
+
72
+ t.index [:migration_name, :shard], unique: true, name: :index_background_schema_migrations_on_unique_configuration
73
+ end
50
74
  end
51
75
  end
@@ -23,6 +23,11 @@ module OnlineMigrations
23
23
 
24
24
  migrations = []
25
25
  migrations << "add_sharding_to_online_migrations" if !columns.include?("shard")
26
+
27
+ if !connection.table_exists?(BackgroundSchemaMigrations::Migration.table_name)
28
+ migrations << "create_background_schema_migrations"
29
+ end
30
+
26
31
  migrations
27
32
  end
28
33
 
@@ -0,0 +1,11 @@
1
+ # frozen_string_literal: true
2
+
3
+ module OnlineMigrations
4
+ # Base class for all records used by this gem.
5
+ #
6
+ # Can be extended to setup different database where all tables related to
7
+ # online_migrations will live.
8
+ class ApplicationRecord < ActiveRecord::Base
9
+ self.abstract_class = true
10
+ end
11
+ end
@@ -39,17 +39,13 @@ module OnlineMigrations
39
39
  #
40
40
  attr_accessor :batch_max_attempts
41
41
 
42
- # Allows to throttle background migrations based on external signal (e.g. database health)
43
- #
44
- # It will be called before each batch run.
45
- # If throttled, the current run will be retried next time.
46
- #
47
- # @return [Proc]
48
- #
49
- # @example
50
- # OnlineMigrations.config.background_migrations.throttler = -> { DatabaseStatus.unhealthy? }
51
- #
52
- attr_reader :throttler
42
+ def throttler
43
+ OnlineMigrations.deprecator.warn(<<~MSG)
44
+ `config.background_migrations.throttler` is deprecated and will be removed.
45
+ Use `config.throttler` instead.
46
+ MSG
47
+ OnlineMigrations.config.throttler
48
+ end
53
49
 
54
50
  # The number of seconds that must pass before the running job is considered stuck
55
51
  #
@@ -57,13 +53,21 @@ module OnlineMigrations
57
53
  #
58
54
  attr_accessor :stuck_jobs_timeout
59
55
 
60
- # The Active Support backtrace cleaner that will be used to clean the
61
- # backtrace of a migration job that errors.
62
- #
63
- # @return [ActiveSupport::BacktraceCleaner, nil] the backtrace cleaner to
64
- # use when cleaning a job's backtrace. Defaults to `Rails.backtrace_cleaner`
65
- #
66
- attr_accessor :backtrace_cleaner
56
+ def backtrace_cleaner
57
+ OnlineMigrations.deprecator.warn(<<~MSG)
58
+ `config.background_migrations.backtrace_cleaner` is deprecated and will be removed.
59
+ Use `config.backtrace_cleaner` instead.
60
+ MSG
61
+ OnlineMigrations.config.backtrace_cleaner
62
+ end
63
+
64
+ def backtrace_cleaner=(value)
65
+ OnlineMigrations.deprecator.warn(<<~MSG)
66
+ `config.background_migrations.backtrace_cleaner=` is deprecated and will be removed.
67
+ Use `config.backtrace_cleaner=` instead.
68
+ MSG
69
+ OnlineMigrations.config.backtrace_cleaner = value
70
+ end
67
71
 
68
72
  # The callback to perform when an error occurs in the migration job.
69
73
  #
@@ -86,17 +90,16 @@ module OnlineMigrations
86
90
  @batch_pause = 0.seconds
87
91
  @sub_batch_pause_ms = 100
88
92
  @batch_max_attempts = 5
89
- @throttler = -> { false }
90
93
  @stuck_jobs_timeout = 1.hour
91
94
  @error_handler = ->(error, errored_job) {}
92
95
  end
93
96
 
94
97
  def throttler=(value)
95
- if !value.respond_to?(:call)
96
- raise ArgumentError, "background_migrations throttler must be a callable."
97
- end
98
-
99
- @throttler = value
98
+ OnlineMigrations.deprecator.warn(<<~MSG)
99
+ `config.background_migrations.throttler=` is deprecated and will be removed.
100
+ Use `config.throttler=` instead.
101
+ MSG
102
+ OnlineMigrations.config.throttler = value
100
103
  end
101
104
  end
102
105
  end
@@ -181,7 +181,7 @@ module OnlineMigrations
181
181
 
182
182
  # @private
183
183
  def on_shard(&block)
184
- abstract_class = find_abstract_class(migration_model)
184
+ abstract_class = Utils.find_connection_class(migration_model)
185
185
 
186
186
  shard = (self.shard || abstract_class.default_shard).to_sym
187
187
  abstract_class.connected_to(shard: shard, role: :writing, &block)
@@ -290,13 +290,6 @@ module OnlineMigrations
290
290
  min_value
291
291
  end
292
292
  end
293
-
294
- def find_abstract_class(model)
295
- model.ancestors.find do |parent|
296
- parent == ActiveRecord::Base ||
297
- (parent.is_a?(Class) && parent.abstract_class?)
298
- end
299
- end
300
293
  end
301
294
  end
302
295
  end
@@ -35,7 +35,7 @@ module OnlineMigrations
35
35
 
36
36
  migration_job.update!(status: :succeeded, finished_at: Time.current)
37
37
  rescue Exception => e # rubocop:disable Lint/RescueException
38
- backtrace_cleaner = ::OnlineMigrations.config.background_migrations.backtrace_cleaner
38
+ backtrace_cleaner = ::OnlineMigrations.config.backtrace_cleaner
39
39
 
40
40
  migration_job.update!(
41
41
  status: :failed,
@@ -105,7 +105,7 @@ module OnlineMigrations
105
105
  end
106
106
 
107
107
  def should_throttle?
108
- ::OnlineMigrations.config.background_migrations.throttler.call
108
+ ::OnlineMigrations.config.throttler.call
109
109
  end
110
110
 
111
111
  def find_or_create_next_migration_job
@@ -0,0 +1,40 @@
1
+ # frozen_string_literal: true
2
+
3
+ module OnlineMigrations
4
+ module BackgroundSchemaMigrations
5
+ # Class representing configuration options for background schema migrations.
6
+ class Config
7
+ # Maximum number of run attempts
8
+ #
9
+ # When attempts are exhausted, the migration is marked as failed.
10
+ # @return [Integer] defaults to 5
11
+ #
12
+ attr_accessor :max_attempts
13
+
14
+ # Statement timeout value used when running background schema migration.
15
+ #
16
+ # @return [Integer] defaults to 1 hour
17
+ #
18
+ attr_accessor :statement_timeout
19
+
20
+ # The callback to perform when an error occurs in the migration.
21
+ #
22
+ # @example
23
+ # OnlineMigrations.config.background_schema_migrations.error_handler = ->(error, errored_migration) do
24
+ # Bugsnag.notify(error) do |notification|
25
+ # notification.add_metadata(:background_schema_migration, { name: errored_migration.name })
26
+ # end
27
+ # end
28
+ #
29
+ # @return [Proc] the callback to perform when an error occurs in the migration
30
+ #
31
+ attr_accessor :error_handler
32
+
33
+ def initialize
34
+ @max_attempts = 5
35
+ @statement_timeout = 1.hour
36
+ @error_handler = ->(error, errored_migration) {}
37
+ end
38
+ end
39
+ end
40
+ end