online_migrations 0.31.2 → 0.33.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +23 -0
- data/docs/background_data_migrations.md +12 -2
- data/docs/background_schema_migrations.md +9 -1
- data/lib/generators/online_migrations/templates/background_data_migrations_remove_iteration_pause_default.rb.tt +7 -0
- data/lib/generators/online_migrations/templates/background_migrations_change_status_default.rb.tt +8 -0
- data/lib/generators/online_migrations/templates/background_schema_migrations_change_unique_index.rb.tt +1 -1
- data/lib/generators/online_migrations/templates/install_migration.rb.tt +4 -4
- data/lib/generators/online_migrations/upgrade_generator.rb +11 -1
- data/lib/online_migrations/background_data_migrations/migration.rb +32 -17
- data/lib/online_migrations/background_data_migrations/migration_helpers.rb +5 -2
- data/lib/online_migrations/background_data_migrations/migration_job.rb +14 -11
- data/lib/online_migrations/background_data_migrations/migration_status_validator.rb +14 -7
- data/lib/online_migrations/background_data_migrations/scheduler.rb +6 -5
- data/lib/online_migrations/background_schema_migrations/migration.rb +33 -8
- data/lib/online_migrations/background_schema_migrations/migration_helpers.rb +9 -7
- data/lib/online_migrations/background_schema_migrations/migration_runner.rb +1 -1
- data/lib/online_migrations/background_schema_migrations/migration_status_validator.rb +7 -4
- data/lib/online_migrations/background_schema_migrations/scheduler.rb +1 -1
- data/lib/online_migrations/change_column_type_helpers.rb +4 -1
- data/lib/online_migrations/command_checker.rb +1 -1
- data/lib/online_migrations/error_messages.rb +2 -2
- data/lib/online_migrations/schema_statements.rb +27 -0
- data/lib/online_migrations/version.rb +1 -1
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 7f092b18d7bc864ba2232a41727e136bb86469279bdc3ad4c04a49a49628fb95
|
|
4
|
+
data.tar.gz: b7349d9b1c818ed4fefd078f7262bb380bb22a34109be13d2ead3191f338dbc9
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4a52e29b327dbecae7f1c31fa4020fa5b428c5f830ea34215695bfd827c787c90146dda6dd12045a5c7f4cc2136741bb1a08b76b2a3ad7273f69c028d192a764
|
|
7
|
+
data.tar.gz: 6175ee34939fbdc6c21605320c36cfdeca971458381e1d6e465e7b464b302914f87f69c7bbcfc50c3b150fc9b9d6d8d34e0d0a1fd2a0573940aa1efa36e983a5
|
data/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,28 @@
|
|
|
1
1
|
## master (unreleased)
|
|
2
2
|
|
|
3
|
+
## 0.33.0 (2026-02-04)
|
|
4
|
+
|
|
5
|
+
- Change background migrations default status to "pending"
|
|
6
|
+
|
|
7
|
+
Note: Run `bin/rails generate online_migrations:upgrade` if using background migrations.
|
|
8
|
+
|
|
9
|
+
- Fix copying partial indexes when changing column type
|
|
10
|
+
|
|
11
|
+
## 0.32.0 (2026-01-07)
|
|
12
|
+
|
|
13
|
+
- Add ability to create delayed background migrations
|
|
14
|
+
|
|
15
|
+
```ruby
|
|
16
|
+
add_index_in_background(:users, :name, delay: true)
|
|
17
|
+
enqueue_background_data_migration("MyMigration", delay: true)
|
|
18
|
+
```
|
|
19
|
+
|
|
20
|
+
They will start only after approval from the user.
|
|
21
|
+
|
|
22
|
+
- Fix enqueueing background data migrations with the same name on different tables
|
|
23
|
+
|
|
24
|
+
Note: Run `bin/rails generate online_migrations:upgrade` if using background schema migrations.
|
|
25
|
+
|
|
3
26
|
## 0.31.2 (2025-11-13)
|
|
4
27
|
|
|
5
28
|
- Fix running background data migrations inline
|
|
@@ -300,7 +300,8 @@ end
|
|
|
300
300
|
|
|
301
301
|
Data Migrations can be in various states during its execution:
|
|
302
302
|
|
|
303
|
-
* **
|
|
303
|
+
* **pending**: A migration has been created by the user.
|
|
304
|
+
* **enqueued**: A migration has been enqueued by the scheduler.
|
|
304
305
|
* **running**: A migration is being performed by a migration executor.
|
|
305
306
|
* **pausing**: A migration has been told to pause but is finishing work.
|
|
306
307
|
* **paused**: A migration was paused in the middle of the run by the user.
|
|
@@ -312,7 +313,8 @@ Data Migrations can be in various states during its execution:
|
|
|
312
313
|
migration.pause
|
|
313
314
|
```
|
|
314
315
|
|
|
315
|
-
* **
|
|
316
|
+
* **errored**: A migration raised an error during last run.
|
|
317
|
+
* **failed**: A migration raises an error when running and retry attempts exceeded.
|
|
316
318
|
* **succeeded**: A migration finished without error.
|
|
317
319
|
* **cancelling**: A migration has been told to cancel but is finishing work.
|
|
318
320
|
* **cancelled**: A migration was cancelled by the user.
|
|
@@ -324,6 +326,14 @@ Data Migrations can be in various states during its execution:
|
|
|
324
326
|
migration.cancel
|
|
325
327
|
```
|
|
326
328
|
|
|
329
|
+
* **delayed**: A migration was created, but waiting approval from the user to start running.
|
|
330
|
+
|
|
331
|
+
To create a delayed migration, you can pass a `delayed: true` option:
|
|
332
|
+
|
|
333
|
+
```ruby
|
|
334
|
+
enqueue_background_data_migration("MyMigration", delay: true)
|
|
335
|
+
```
|
|
336
|
+
|
|
327
337
|
To get the progress (assuming `#count` method on data migration class was defined):
|
|
328
338
|
|
|
329
339
|
```ruby
|
|
@@ -132,13 +132,21 @@ Available events:
|
|
|
132
132
|
|
|
133
133
|
Background Schema Migrations can be in various states during its execution:
|
|
134
134
|
|
|
135
|
-
* **
|
|
135
|
+
* **pending**: A migration has been created by the user.
|
|
136
136
|
* **running**: A migration is being performed by a migration executor.
|
|
137
137
|
* **errored**: A migration raised an error during last run.
|
|
138
138
|
* **failed**: A migration raises an error when running and retry attempts exceeded.
|
|
139
139
|
* **succeeded**: A migration finished without error.
|
|
140
140
|
* **cancelled**: A migration was cancelled by the user.
|
|
141
141
|
|
|
142
|
+
* **delayed**: A migration was created, but waiting approval from the user to start running.
|
|
143
|
+
|
|
144
|
+
To create a delayed migration, you can pass a `delayed: true` option:
|
|
145
|
+
|
|
146
|
+
```ruby
|
|
147
|
+
add_index_in_background(:users, :name, delay: true)
|
|
148
|
+
```
|
|
149
|
+
|
|
142
150
|
## Configuring
|
|
143
151
|
|
|
144
152
|
There are a few configurable options for the Background Schema Migrations. Custom configurations should be placed in a `online_migrations.rb` initializer.
|
data/lib/generators/online_migrations/templates/background_migrations_change_status_default.rb.tt
ADDED
|
@@ -0,0 +1,8 @@
|
|
|
1
|
+
class BackgroundDataMigrationsChangeStatusDefault < <%= migration_parent %>
|
|
2
|
+
def change
|
|
3
|
+
safety_assured do
|
|
4
|
+
change_column_default :background_data_migrations, :status, from: "enqueued", to: "pending"
|
|
5
|
+
change_column_default :background_schema_migrations, :status, from: "enqueued", to: "pending"
|
|
6
|
+
end
|
|
7
|
+
end
|
|
8
|
+
end
|
|
@@ -2,7 +2,7 @@ class BackgroundSchemaMigrationsChangeUniqueIndex < <%= migration_parent %>
|
|
|
2
2
|
def change
|
|
3
3
|
safety_assured("Table is small") do
|
|
4
4
|
remove_index :background_schema_migrations, name: :index_background_schema_migrations_on_unique_configuration
|
|
5
|
-
add_index :background_schema_migrations, [:migration_name, :shard, :connection_class_name], unique: true,
|
|
5
|
+
add_index :background_schema_migrations, [:migration_name, :table_name, :shard, :connection_class_name], unique: true,
|
|
6
6
|
name: :index_background_schema_migrations_on_unique_configuration
|
|
7
7
|
end
|
|
8
8
|
end
|
|
@@ -3,7 +3,7 @@ class InstallOnlineMigrations < <%= migration_parent %>
|
|
|
3
3
|
create_table :background_data_migrations do |t|
|
|
4
4
|
t.string :migration_name, null: false
|
|
5
5
|
t.jsonb :arguments, default: [], null: false
|
|
6
|
-
t.string :status, default: "
|
|
6
|
+
t.string :status, default: "pending", null: false
|
|
7
7
|
t.string :shard
|
|
8
8
|
t.string :cursor
|
|
9
9
|
t.string :jid
|
|
@@ -13,7 +13,7 @@ class InstallOnlineMigrations < <%= migration_parent %>
|
|
|
13
13
|
t.bigint :tick_count, default: 0, null: false
|
|
14
14
|
t.float :time_running, default: 0.0, null: false
|
|
15
15
|
t.integer :max_attempts, null: false
|
|
16
|
-
t.float :iteration_pause,
|
|
16
|
+
t.float :iteration_pause, null: false
|
|
17
17
|
t.string :error_class
|
|
18
18
|
t.string :error_message
|
|
19
19
|
t.string :backtrace, array: true
|
|
@@ -28,7 +28,7 @@ class InstallOnlineMigrations < <%= migration_parent %>
|
|
|
28
28
|
t.string :migration_name, null: false
|
|
29
29
|
t.string :table_name, null: false
|
|
30
30
|
t.string :definition, null: false
|
|
31
|
-
t.string :status, default: "
|
|
31
|
+
t.string :status, default: "pending", null: false
|
|
32
32
|
t.string :shard
|
|
33
33
|
t.integer :statement_timeout
|
|
34
34
|
t.datetime :started_at
|
|
@@ -41,7 +41,7 @@ class InstallOnlineMigrations < <%= migration_parent %>
|
|
|
41
41
|
t.string :connection_class_name
|
|
42
42
|
t.timestamps
|
|
43
43
|
|
|
44
|
-
t.index [:migration_name, :shard, :connection_class_name], unique: true,
|
|
44
|
+
t.index [:migration_name, :table_name, :shard, :connection_class_name], unique: true,
|
|
45
45
|
name: :index_background_schema_migrations_on_unique_configuration
|
|
46
46
|
end
|
|
47
47
|
end
|
|
@@ -30,7 +30,7 @@ module OnlineMigrations
|
|
|
30
30
|
end
|
|
31
31
|
|
|
32
32
|
indexes = connection.indexes(:background_schema_migrations)
|
|
33
|
-
unique_index = indexes.find { |i| i.unique && i.columns.sort == ["connection_class_name", "migration_name", "shard"] }
|
|
33
|
+
unique_index = indexes.find { |i| i.unique && i.columns.sort == ["connection_class_name", "migration_name", "shard", "table_name"] }
|
|
34
34
|
if !unique_index
|
|
35
35
|
migrations << "background_schema_migrations_change_unique_index"
|
|
36
36
|
end
|
|
@@ -47,6 +47,16 @@ module OnlineMigrations
|
|
|
47
47
|
migrations << "background_data_migrations_add_iteration_pause"
|
|
48
48
|
end
|
|
49
49
|
|
|
50
|
+
iteration_pause_column = connection.columns(:background_data_migrations).find { |c| c.name == "iteration_pause" }
|
|
51
|
+
if iteration_pause_column && iteration_pause_column.default
|
|
52
|
+
migrations << "background_data_migrations_remove_iteration_pause_default"
|
|
53
|
+
end
|
|
54
|
+
|
|
55
|
+
status_column = connection.columns(:background_data_migrations).find { |c| c.name == "status" }
|
|
56
|
+
if status_column.default == "enqueued"
|
|
57
|
+
migrations << "background_migrations_change_status_default"
|
|
58
|
+
end
|
|
59
|
+
|
|
50
60
|
migrations
|
|
51
61
|
end
|
|
52
62
|
|
|
@@ -11,21 +11,26 @@ module OnlineMigrations
|
|
|
11
11
|
include ShardAware
|
|
12
12
|
|
|
13
13
|
STATUSES = [
|
|
14
|
-
"
|
|
14
|
+
"pending", # The migration has been created by the user.
|
|
15
|
+
"enqueued", # The migration has been enqueued by the scheduler.
|
|
15
16
|
"running", # The migration is being performed by a migration executor.
|
|
16
17
|
"pausing", # The migration has been told to pause but is finishing work.
|
|
17
18
|
"paused", # The migration was paused in the middle of the run by the user.
|
|
18
|
-
"
|
|
19
|
+
"errored", # The migration raised an error during last run.
|
|
20
|
+
"failed", # The migration raises an error when running and retry attempts exceeded.
|
|
19
21
|
"succeeded", # The migration finished without error.
|
|
20
22
|
"cancelling", # The migration has been told to cancel but is finishing work.
|
|
21
23
|
"cancelled", # The migration was cancelled by the user.
|
|
24
|
+
"delayed", # The migration was created, but waiting approval from the user to start running.
|
|
22
25
|
]
|
|
23
26
|
|
|
24
27
|
COMPLETED_STATUSES = ["succeeded", "failed", "cancelled"]
|
|
25
28
|
|
|
26
29
|
ACTIVE_STATUSES = [
|
|
30
|
+
"pending",
|
|
27
31
|
"enqueued",
|
|
28
32
|
"running",
|
|
33
|
+
"failed",
|
|
29
34
|
"pausing",
|
|
30
35
|
"paused",
|
|
31
36
|
"cancelling",
|
|
@@ -86,7 +91,7 @@ module OnlineMigrations
|
|
|
86
91
|
end
|
|
87
92
|
|
|
88
93
|
# Returns whether the migration is active, which is defined as
|
|
89
|
-
# having a status of enqueued, running, pausing, paused, or cancelling.
|
|
94
|
+
# having a status of pending, enqueued, running, pausing, paused, or cancelling.
|
|
90
95
|
#
|
|
91
96
|
# @return [Boolean] whether the migration is active.
|
|
92
97
|
#
|
|
@@ -106,22 +111,18 @@ module OnlineMigrations
|
|
|
106
111
|
end
|
|
107
112
|
|
|
108
113
|
# Returns whether a migration is stuck, which is defined as having a status of
|
|
109
|
-
# cancelling or pausing, and not having been updated in the last 5 minutes.
|
|
114
|
+
# running, cancelling or pausing, and not having been updated in the last 5 minutes.
|
|
110
115
|
#
|
|
111
116
|
# @return [Boolean] whether the migration is stuck.
|
|
112
117
|
#
|
|
113
118
|
def stuck?
|
|
114
119
|
stuck_timeout = OnlineMigrations.config.background_data_migrations.stuck_timeout
|
|
115
|
-
(cancelling? || pausing?) && updated_at <= stuck_timeout.ago
|
|
120
|
+
(running? || cancelling? || pausing?) && updated_at <= stuck_timeout.ago
|
|
116
121
|
end
|
|
117
122
|
|
|
118
123
|
# @private
|
|
119
124
|
def start
|
|
120
|
-
if
|
|
121
|
-
update!(started_at: Time.current)
|
|
122
|
-
data_migration.after_start
|
|
123
|
-
true
|
|
124
|
-
elsif enqueued?
|
|
125
|
+
if enqueued?
|
|
125
126
|
update!(status: :running, started_at: Time.current)
|
|
126
127
|
data_migration.after_start
|
|
127
128
|
true
|
|
@@ -130,6 +131,19 @@ module OnlineMigrations
|
|
|
130
131
|
end
|
|
131
132
|
end
|
|
132
133
|
|
|
134
|
+
# Enqueue this data migration. No-op if migration is not delayed.
|
|
135
|
+
#
|
|
136
|
+
# @return [Boolean] whether this data migration was enqueued.
|
|
137
|
+
#
|
|
138
|
+
def enqueue
|
|
139
|
+
if delayed?
|
|
140
|
+
pending!
|
|
141
|
+
true
|
|
142
|
+
else
|
|
143
|
+
false
|
|
144
|
+
end
|
|
145
|
+
end
|
|
146
|
+
|
|
133
147
|
# Cancel this data migration. No-op if migration is completed.
|
|
134
148
|
#
|
|
135
149
|
# @return [Boolean] whether this data migration was cancelled.
|
|
@@ -137,9 +151,9 @@ module OnlineMigrations
|
|
|
137
151
|
def cancel
|
|
138
152
|
return false if completed?
|
|
139
153
|
|
|
140
|
-
if paused? || stuck?
|
|
154
|
+
if paused? || delayed? || stuck?
|
|
141
155
|
update!(status: :cancelled, finished_at: Time.current)
|
|
142
|
-
elsif enqueued?
|
|
156
|
+
elsif pending? || enqueued? || errored?
|
|
143
157
|
cancelled!
|
|
144
158
|
else
|
|
145
159
|
cancelling!
|
|
@@ -155,7 +169,7 @@ module OnlineMigrations
|
|
|
155
169
|
def pause
|
|
156
170
|
return false if completed?
|
|
157
171
|
|
|
158
|
-
if enqueued? || stuck?
|
|
172
|
+
if pending? || enqueued? || delayed? || stuck? || errored?
|
|
159
173
|
paused!
|
|
160
174
|
else
|
|
161
175
|
pausing!
|
|
@@ -170,7 +184,7 @@ module OnlineMigrations
|
|
|
170
184
|
#
|
|
171
185
|
def resume
|
|
172
186
|
if paused?
|
|
173
|
-
|
|
187
|
+
pending!
|
|
174
188
|
true
|
|
175
189
|
else
|
|
176
190
|
false
|
|
@@ -221,13 +235,14 @@ module OnlineMigrations
|
|
|
221
235
|
end
|
|
222
236
|
|
|
223
237
|
# @private
|
|
224
|
-
def persist_error(error)
|
|
238
|
+
def persist_error(error, attempt)
|
|
225
239
|
backtrace = error.backtrace
|
|
226
240
|
backtrace_cleaner = OnlineMigrations.config.backtrace_cleaner
|
|
227
241
|
backtrace = backtrace_cleaner.clean(backtrace) if backtrace_cleaner
|
|
242
|
+
status = attempt >= max_attempts ? :failed : :errored
|
|
228
243
|
|
|
229
244
|
update!(
|
|
230
|
-
status:
|
|
245
|
+
status: status,
|
|
231
246
|
finished_at: Time.current,
|
|
232
247
|
error_class: error.class.name,
|
|
233
248
|
error_message: error.message,
|
|
@@ -276,7 +291,7 @@ module OnlineMigrations
|
|
|
276
291
|
def retry
|
|
277
292
|
if failed?
|
|
278
293
|
update!(
|
|
279
|
-
status: :
|
|
294
|
+
status: :pending,
|
|
280
295
|
started_at: nil,
|
|
281
296
|
finished_at: nil,
|
|
282
297
|
error_class: nil,
|
|
@@ -338,6 +338,7 @@ module OnlineMigrations
|
|
|
338
338
|
#
|
|
339
339
|
# @param migration_name [String, Class] Background migration class name
|
|
340
340
|
# @param arguments [Array] Extra arguments to pass to the migration instance when the migration runs
|
|
341
|
+
# @param delay [Boolean] Whether this migration should be delayed and approved by the user to start running.
|
|
341
342
|
# @option options [Integer] :max_attempts (5) Maximum number of batch run attempts
|
|
342
343
|
# @option options [String, nil] :connection_class_name Class name to use to get connections
|
|
343
344
|
#
|
|
@@ -368,7 +369,7 @@ module OnlineMigrations
|
|
|
368
369
|
# @note For convenience, the enqueued background data migration is run inline
|
|
369
370
|
# in development and test environments
|
|
370
371
|
#
|
|
371
|
-
def enqueue_background_data_migration(migration_name, *arguments, **options)
|
|
372
|
+
def enqueue_background_data_migration(migration_name, *arguments, delay: false, **options)
|
|
372
373
|
options.assert_valid_keys(:max_attempts, :iteration_pause, :connection_class_name)
|
|
373
374
|
|
|
374
375
|
migration_name = migration_name.name if migration_name.is_a?(Class)
|
|
@@ -381,13 +382,15 @@ module OnlineMigrations
|
|
|
381
382
|
connection_class = options[:connection_class_name].constantize
|
|
382
383
|
shards = Utils.shard_names(connection_class)
|
|
383
384
|
shards = [nil] if shards.size == 1
|
|
385
|
+
status = delay ? :delayed : :pending
|
|
384
386
|
|
|
385
387
|
shards.each do |shard|
|
|
386
388
|
# Can't use `find_or_create_by` or hash syntax here, because it does not correctly work with json `arguments`.
|
|
387
389
|
migration = Migration.where(migration_name: migration_name, shard: shard).where("arguments = ?", arguments.to_json).first
|
|
388
|
-
migration ||= Migration.create!(**options, migration_name: migration_name, arguments: arguments, shard: shard)
|
|
390
|
+
migration ||= Migration.create!(**options, status: status, migration_name: migration_name, arguments: arguments, shard: shard)
|
|
389
391
|
|
|
390
392
|
if Utils.run_background_migrations_inline? && !migration.succeeded?
|
|
393
|
+
migration.update_column(:status, :enqueued) if !migration.enqueued?
|
|
391
394
|
job = OnlineMigrations.config.background_data_migrations.job
|
|
392
395
|
job.constantize.perform_inline(migration.id)
|
|
393
396
|
end
|
|
@@ -8,23 +8,17 @@ module OnlineMigrations
|
|
|
8
8
|
|
|
9
9
|
sidekiq_options backtrace: true
|
|
10
10
|
|
|
11
|
-
sidekiq_retry_in do |count,
|
|
11
|
+
sidekiq_retry_in do |count, exception, jobhash|
|
|
12
12
|
migration_id = jobhash["args"].fetch(0)
|
|
13
13
|
migration = Migration.find(migration_id)
|
|
14
|
+
migration.persist_error(exception, count + 1)
|
|
15
|
+
OnlineMigrations.config.background_data_migrations.error_handler.call(exception, migration)
|
|
14
16
|
|
|
15
17
|
if count + 1 >= migration.max_attempts
|
|
16
18
|
:kill
|
|
17
19
|
end
|
|
18
20
|
end
|
|
19
21
|
|
|
20
|
-
sidekiq_retries_exhausted do |jobhash, exception|
|
|
21
|
-
migration_id = jobhash["args"].fetch(0)
|
|
22
|
-
migration = Migration.find(migration_id)
|
|
23
|
-
migration.persist_error(exception)
|
|
24
|
-
|
|
25
|
-
OnlineMigrations.config.background_data_migrations.error_handler.call(exception, migration)
|
|
26
|
-
end
|
|
27
|
-
|
|
28
22
|
TICKER_INTERVAL = 5 # seconds
|
|
29
23
|
|
|
30
24
|
def initialize
|
|
@@ -57,6 +51,15 @@ module OnlineMigrations
|
|
|
57
51
|
end
|
|
58
52
|
|
|
59
53
|
def on_resume
|
|
54
|
+
if @migration.errored? # the job was retried
|
|
55
|
+
@migration.update!(
|
|
56
|
+
status: :running,
|
|
57
|
+
error_class: nil,
|
|
58
|
+
error_message: nil,
|
|
59
|
+
backtrace: nil
|
|
60
|
+
)
|
|
61
|
+
end
|
|
62
|
+
|
|
60
63
|
@data_migration.after_resume
|
|
61
64
|
end
|
|
62
65
|
|
|
@@ -126,7 +129,7 @@ module OnlineMigrations
|
|
|
126
129
|
end
|
|
127
130
|
|
|
128
131
|
def each_iteration(item, _migration_id)
|
|
129
|
-
if @migration.cancelling? || @migration.pausing? || @migration.paused?
|
|
132
|
+
if @migration.cancelling? || @migration.cancelled? || @migration.pausing? || @migration.paused?
|
|
130
133
|
# Finish this exact sidekiq job. When the migration is paused
|
|
131
134
|
# and will be resumed, a new job will be enqueued.
|
|
132
135
|
finished = true
|
|
@@ -140,7 +143,7 @@ module OnlineMigrations
|
|
|
140
143
|
@migration.data_migration.process(item)
|
|
141
144
|
|
|
142
145
|
# Migration is refreshed regularly by ticker.
|
|
143
|
-
pause = @migration.iteration_pause
|
|
146
|
+
pause = @migration.iteration_pause.to_f
|
|
144
147
|
sleep(pause) if pause > 0
|
|
145
148
|
end
|
|
146
149
|
@ticker.tick
|
|
@@ -6,6 +6,7 @@ module OnlineMigrations
|
|
|
6
6
|
class MigrationStatusValidator < ActiveModel::Validator
|
|
7
7
|
# Valid status transitions a Migration can make.
|
|
8
8
|
VALID_STATUS_TRANSITIONS = {
|
|
9
|
+
"pending" => ["enqueued", "paused", "cancelled"],
|
|
9
10
|
# enqueued -> running occurs when the migration starts performing.
|
|
10
11
|
# enqueued -> paused occurs when the migration is paused before starting.
|
|
11
12
|
# enqueued -> cancelled occurs when the migration is cancelled before starting.
|
|
@@ -15,11 +16,13 @@ module OnlineMigrations
|
|
|
15
16
|
# running -> succeeded occurs when the migration completes successfully.
|
|
16
17
|
# running -> pausing occurs when a user pauses the migration as it's performing.
|
|
17
18
|
# running -> cancelling occurs when a user cancels the migration as it's performing.
|
|
18
|
-
# running ->
|
|
19
|
+
# running -> errored occurs when the migration raised an error during the last run.
|
|
20
|
+
# running -> failed occurs when the migration raises an error when running and retry attempts exceeded.
|
|
19
21
|
"running" => [
|
|
20
22
|
"succeeded",
|
|
21
23
|
"pausing",
|
|
22
24
|
"cancelling",
|
|
25
|
+
"errored",
|
|
23
26
|
"failed",
|
|
24
27
|
],
|
|
25
28
|
# pausing -> paused occurs when the migration actually halts performing and
|
|
@@ -32,19 +35,23 @@ module OnlineMigrations
|
|
|
32
35
|
# nothing in its collection to process.
|
|
33
36
|
# pausing -> failed occurs when the job raises an exception after the
|
|
34
37
|
# user has paused it.
|
|
35
|
-
"pausing" => ["paused", "cancelling", "succeeded", "failed"],
|
|
36
|
-
# paused ->
|
|
38
|
+
"pausing" => ["paused", "cancelling", "succeeded", "errored", "failed"],
|
|
39
|
+
# paused -> pending occurs when the migration is resumed after being paused.
|
|
37
40
|
# paused -> cancelled when the user cancels the migration after it is paused.
|
|
38
|
-
"paused" => ["
|
|
39
|
-
|
|
40
|
-
|
|
41
|
+
"paused" => ["pending", "cancelled"],
|
|
42
|
+
"errored" => ["running", "failed", "cancelled", "paused"],
|
|
43
|
+
# failed -> pending occurs when the migration is retried after encounting an error.
|
|
44
|
+
"failed" => ["pending"],
|
|
41
45
|
# cancelling -> cancelled occurs when the migration actually halts performing
|
|
42
46
|
# and occupies a status of cancelled.
|
|
43
47
|
# cancelling -> succeeded occurs when the migration completes immediately after
|
|
44
48
|
# being cancelled. See description for pausing -> succeeded.
|
|
45
49
|
# cancelling -> failed occurs when the job raises an exception after the
|
|
46
50
|
# user has cancelled it.
|
|
47
|
-
"cancelling" => ["cancelled", "succeeded", "failed"],
|
|
51
|
+
"cancelling" => ["cancelled", "succeeded", "errored", "failed"],
|
|
52
|
+
# delayed -> pending occurs when the delayed migration was approved by the user to start running.
|
|
53
|
+
# delayed -> cancelled occurs when the delayed migration was cancelled.
|
|
54
|
+
"delayed" => ["pending", "cancelled"],
|
|
48
55
|
}
|
|
49
56
|
|
|
50
57
|
def validate(record)
|
|
@@ -34,13 +34,13 @@ module OnlineMigrations
|
|
|
34
34
|
relation = relation.where(shard: shard) if shard
|
|
35
35
|
|
|
36
36
|
with_lock do
|
|
37
|
-
|
|
38
|
-
|
|
37
|
+
stuck_migrations, active_migrations = relation.running.partition(&:stuck?)
|
|
38
|
+
runnable_migrations = relation.pending + stuck_migrations
|
|
39
39
|
|
|
40
40
|
# Ensure no more than 'concurrency' migrations are running at the same time.
|
|
41
|
-
remaining_to_enqueue = concurrency -
|
|
41
|
+
remaining_to_enqueue = concurrency - active_migrations.count
|
|
42
42
|
if remaining_to_enqueue > 0
|
|
43
|
-
migrations_to_enqueue =
|
|
43
|
+
migrations_to_enqueue = runnable_migrations.take(remaining_to_enqueue)
|
|
44
44
|
migrations_to_enqueue.each do |migration|
|
|
45
45
|
enqueue_migration(migration)
|
|
46
46
|
end
|
|
@@ -67,10 +67,11 @@ module OnlineMigrations
|
|
|
67
67
|
def enqueue_migration(migration)
|
|
68
68
|
job = OnlineMigrations.config.background_data_migrations.job
|
|
69
69
|
job_class = job.constantize
|
|
70
|
+
migration.update!(status: :enqueued)
|
|
70
71
|
|
|
71
72
|
jid = job_class.perform_async(migration.id)
|
|
72
73
|
if jid
|
|
73
|
-
migration.update!(
|
|
74
|
+
migration.update!(jid: jid)
|
|
74
75
|
end
|
|
75
76
|
end
|
|
76
77
|
end
|
|
@@ -11,12 +11,13 @@ module OnlineMigrations
|
|
|
11
11
|
include ShardAware
|
|
12
12
|
|
|
13
13
|
STATUSES = [
|
|
14
|
-
"
|
|
14
|
+
"pending", # The migration has been created by the user.
|
|
15
15
|
"running", # The migration is being performed by a migration executor.
|
|
16
16
|
"errored", # The migration raised an error during last run.
|
|
17
17
|
"failed", # The migration raises an error when running and retry attempts exceeded.
|
|
18
18
|
"succeeded", # The migration finished without error.
|
|
19
19
|
"cancelled", # The migration was cancelled by the user.
|
|
20
|
+
"delayed", # The migration was created, but waiting approval from the user to start running.
|
|
20
21
|
]
|
|
21
22
|
|
|
22
23
|
MAX_IDENTIFIER_LENGTH = 63
|
|
@@ -24,7 +25,7 @@ module OnlineMigrations
|
|
|
24
25
|
self.table_name = :background_schema_migrations
|
|
25
26
|
|
|
26
27
|
scope :queue_order, -> { order(created_at: :asc) }
|
|
27
|
-
scope :active, -> { where(status: [:
|
|
28
|
+
scope :active, -> { where(status: [:pending, :running, :errored]) }
|
|
28
29
|
|
|
29
30
|
alias_attribute :name, :migration_name
|
|
30
31
|
|
|
@@ -33,7 +34,7 @@ module OnlineMigrations
|
|
|
33
34
|
validates :table_name, presence: true, length: { maximum: MAX_IDENTIFIER_LENGTH }
|
|
34
35
|
validates :definition, presence: true
|
|
35
36
|
validates :migration_name, presence: true, uniqueness: {
|
|
36
|
-
scope: [:connection_class_name, :shard],
|
|
37
|
+
scope: [:table_name, :connection_class_name, :shard],
|
|
37
38
|
message: ->(object, data) do
|
|
38
39
|
message = "(#{data[:value]}) has already been taken."
|
|
39
40
|
if object.index_addition?
|
|
@@ -58,16 +59,14 @@ module OnlineMigrations
|
|
|
58
59
|
end
|
|
59
60
|
|
|
60
61
|
# Returns whether the migration is active, which is defined as
|
|
61
|
-
# having a status of
|
|
62
|
+
# having a status of pending, or running.
|
|
62
63
|
#
|
|
63
64
|
# @return [Boolean] whether the migration is active.
|
|
64
65
|
#
|
|
65
66
|
def active?
|
|
66
|
-
|
|
67
|
+
pending? || running?
|
|
67
68
|
end
|
|
68
69
|
|
|
69
|
-
alias cancel cancelled!
|
|
70
|
-
|
|
71
70
|
# Returns whether this migration is pausable.
|
|
72
71
|
#
|
|
73
72
|
def pausable?
|
|
@@ -99,7 +98,7 @@ module OnlineMigrations
|
|
|
99
98
|
def retry
|
|
100
99
|
if failed?
|
|
101
100
|
update!(
|
|
102
|
-
status: :
|
|
101
|
+
status: :pending,
|
|
103
102
|
attempts: 0,
|
|
104
103
|
started_at: nil,
|
|
105
104
|
finished_at: nil,
|
|
@@ -114,6 +113,32 @@ module OnlineMigrations
|
|
|
114
113
|
end
|
|
115
114
|
end
|
|
116
115
|
|
|
116
|
+
# Enqueue this data migration. No-op if migration is not delayed.
|
|
117
|
+
#
|
|
118
|
+
# @return [Boolean] whether this data migration was enqueued.
|
|
119
|
+
#
|
|
120
|
+
def enqueue
|
|
121
|
+
if delayed?
|
|
122
|
+
pending!
|
|
123
|
+
true
|
|
124
|
+
else
|
|
125
|
+
false
|
|
126
|
+
end
|
|
127
|
+
end
|
|
128
|
+
|
|
129
|
+
# Cancel this schema migration. No-op if migration is completed.
|
|
130
|
+
#
|
|
131
|
+
# @return [Boolean] whether this schema migration was cancelled.
|
|
132
|
+
#
|
|
133
|
+
def cancel
|
|
134
|
+
if completed?
|
|
135
|
+
false
|
|
136
|
+
elsif pending? || errored? || delayed? || stuck?
|
|
137
|
+
cancelled!
|
|
138
|
+
true
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
|
|
117
142
|
def index_addition?
|
|
118
143
|
definition.match?(/create (unique )?index/i)
|
|
119
144
|
end
|
|
@@ -4,7 +4,7 @@ module OnlineMigrations
|
|
|
4
4
|
module BackgroundSchemaMigrations
|
|
5
5
|
module MigrationHelpers
|
|
6
6
|
def add_index_in_background(table_name, column_name, **options)
|
|
7
|
-
migration_options = options.extract!(:max_attempts, :statement_timeout, :connection_class_name)
|
|
7
|
+
migration_options = options.extract!(:max_attempts, :statement_timeout, :connection_class_name, :delay)
|
|
8
8
|
|
|
9
9
|
options[:algorithm] = :concurrently
|
|
10
10
|
index, algorithm, if_not_exists = add_index_options(table_name, column_name, **options)
|
|
@@ -34,7 +34,7 @@ module OnlineMigrations
|
|
|
34
34
|
def remove_index_in_background(table_name, column_name = nil, name:, **options)
|
|
35
35
|
raise ArgumentError, "Index name must be specified" if name.blank?
|
|
36
36
|
|
|
37
|
-
migration_options = options.extract!(:max_attempts, :statement_timeout, :connection_class_name)
|
|
37
|
+
migration_options = options.extract!(:max_attempts, :statement_timeout, :connection_class_name, :delay)
|
|
38
38
|
|
|
39
39
|
if !index_exists?(table_name, column_name, **options, name: name)
|
|
40
40
|
Utils.raise_or_say("Index deletion was not enqueued because the index does not exist.")
|
|
@@ -46,7 +46,7 @@ module OnlineMigrations
|
|
|
46
46
|
end
|
|
47
47
|
|
|
48
48
|
def validate_foreign_key_in_background(from_table, to_table = nil, **options)
|
|
49
|
-
migration_options = options.extract!(:max_attempts, :statement_timeout, :connection_class_name)
|
|
49
|
+
migration_options = options.extract!(:max_attempts, :statement_timeout, :connection_class_name, :delay)
|
|
50
50
|
|
|
51
51
|
if !foreign_key_exists?(from_table, to_table, **options)
|
|
52
52
|
Utils.raise_or_say("Foreign key validation was not enqueued because the foreign key does not exist.")
|
|
@@ -87,7 +87,7 @@ module OnlineMigrations
|
|
|
87
87
|
end
|
|
88
88
|
end
|
|
89
89
|
|
|
90
|
-
def enqueue_background_schema_migration(migration_name, table_name, connection_class_name: nil, **options)
|
|
90
|
+
def enqueue_background_schema_migration(migration_name, table_name, connection_class_name: nil, delay: false, **options)
|
|
91
91
|
options.assert_valid_keys(:definition, :max_attempts, :statement_timeout)
|
|
92
92
|
|
|
93
93
|
if Utils.multiple_databases? && !connection_class_name
|
|
@@ -107,13 +107,15 @@ module OnlineMigrations
|
|
|
107
107
|
shards = Utils.shard_names(connection_class)
|
|
108
108
|
shards = [nil] if shards.size == 1
|
|
109
109
|
|
|
110
|
+
status = delay ? :delayed : :pending
|
|
111
|
+
|
|
110
112
|
shards.each do |shard|
|
|
111
|
-
migration = Migration.create_with(**options,
|
|
112
|
-
.find_or_create_by!(migration_name: migration_name, shard: shard, connection_class_name: connection_class_name)
|
|
113
|
+
migration = Migration.create_with(**options, status: status)
|
|
114
|
+
.find_or_create_by!(migration_name: migration_name, table_name: table_name, shard: shard, connection_class_name: connection_class_name)
|
|
113
115
|
|
|
114
116
|
if Utils.run_background_migrations_inline?
|
|
115
117
|
# Run migration again in development.
|
|
116
|
-
migration.update_column(:status, :
|
|
118
|
+
migration.update_column(:status, :pending) if !migration.pending?
|
|
117
119
|
|
|
118
120
|
runner = MigrationRunner.new(migration)
|
|
119
121
|
runner.run
|
|
@@ -13,7 +13,7 @@ module OnlineMigrations
|
|
|
13
13
|
def run
|
|
14
14
|
return if migration.cancelled? || migration.succeeded?
|
|
15
15
|
|
|
16
|
-
migration.running! if migration.
|
|
16
|
+
migration.running! if migration.pending? || migration.errored?
|
|
17
17
|
migration_payload = { migration: migration }
|
|
18
18
|
|
|
19
19
|
if migration.attempts == 0
|
|
@@ -5,8 +5,8 @@ module OnlineMigrations
|
|
|
5
5
|
# @private
|
|
6
6
|
class MigrationStatusValidator < ActiveModel::Validator
|
|
7
7
|
VALID_STATUS_TRANSITIONS = {
|
|
8
|
-
#
|
|
9
|
-
"
|
|
8
|
+
# pending -> running occurs when the migration starts performing.
|
|
9
|
+
"pending" => ["running", "cancelled"],
|
|
10
10
|
# running -> succeeded occurs when the migration completes successfully.
|
|
11
11
|
# running -> errored occurs when the migration raised an error during the last run.
|
|
12
12
|
# running -> failed occurs when the migration raises an error when running and retry attempts exceeded.
|
|
@@ -14,9 +14,12 @@ module OnlineMigrations
|
|
|
14
14
|
# errored -> running occurs when previously errored migration starts running
|
|
15
15
|
# errored -> failed occurs when the migration raises an error when running and retry attempts exceeded.
|
|
16
16
|
"errored" => ["running", "failed", "cancelled"],
|
|
17
|
-
# failed ->
|
|
17
|
+
# failed -> pending occurs when the failed migration is enqueued to be retried.
|
|
18
18
|
# failed -> running occurs when the failed migration is retried.
|
|
19
|
-
"failed" => ["
|
|
19
|
+
"failed" => ["pending", "running", "cancelled"],
|
|
20
|
+
# delayed -> pending occurs when the delayed migration was approved by the user to start running.
|
|
21
|
+
# delayed -> cancelled occurs when the delayed migration was cancelled.
|
|
22
|
+
"delayed" => ["pending", "cancelled"],
|
|
20
23
|
}
|
|
21
24
|
|
|
22
25
|
def validate(record)
|
|
@@ -36,7 +36,7 @@ module OnlineMigrations
|
|
|
36
36
|
private
|
|
37
37
|
def find_migration(**options)
|
|
38
38
|
stuck_migrations, active_migrations = Migration.running.partition(&:stuck?)
|
|
39
|
-
runnable_migrations = (Migration.
|
|
39
|
+
runnable_migrations = (Migration.pending + Migration.errored + stuck_migrations).sort_by(&:created_at)
|
|
40
40
|
|
|
41
41
|
if options.key?(:shard)
|
|
42
42
|
runnable_migrations = runnable_migrations.select { |migration| migration.shard.to_s == options[:shard].to_s }
|
|
@@ -436,7 +436,10 @@ module OnlineMigrations
|
|
|
436
436
|
}
|
|
437
437
|
|
|
438
438
|
options[:using] = index.using if index.using
|
|
439
|
-
|
|
439
|
+
|
|
440
|
+
if index.where
|
|
441
|
+
options[:where] = index.where.gsub(/\b#{from_column}\b/, to_column)
|
|
442
|
+
end
|
|
440
443
|
|
|
441
444
|
if index.opclasses.present?
|
|
442
445
|
opclasses = index.opclasses.dup
|
|
@@ -426,7 +426,7 @@ module OnlineMigrations
|
|
|
426
426
|
validate_constraint_code: command_str(:validate_not_null_constraint, table_name, column_name, name: constraint_name),
|
|
427
427
|
table_name: table_name,
|
|
428
428
|
column_name: column_name,
|
|
429
|
-
|
|
429
|
+
default_value: default,
|
|
430
430
|
remove_constraint_code: command_str(:remove_check_constraint, table_name, name: constraint_name),
|
|
431
431
|
change_column_null_code: command_str(:change_column_null, table_name, column_name, false),
|
|
432
432
|
}
|
|
@@ -277,11 +277,11 @@ class <%= migration_name %> < <%= migration_parent %>
|
|
|
277
277
|
|
|
278
278
|
def change
|
|
279
279
|
<%= add_constraint_code %>
|
|
280
|
-
<% unless
|
|
280
|
+
<% unless default_value.nil? %>
|
|
281
281
|
|
|
282
282
|
# Passing a default value to change_column_null runs a single UPDATE query,
|
|
283
283
|
# which can cause downtime. Instead, backfill the existing rows in batches.
|
|
284
|
-
update_column_in_batches(:<%= table_name %>, :<%= column_name %>, <%=
|
|
284
|
+
update_column_in_batches(:<%= table_name %>, :<%= column_name %>, <%= default_value.inspect %>) do |relation|
|
|
285
285
|
relation.where(<%= column_name %>: nil)
|
|
286
286
|
end
|
|
287
287
|
|
|
@@ -852,6 +852,33 @@ module OnlineMigrations
|
|
|
852
852
|
end
|
|
853
853
|
|
|
854
854
|
# @private
|
|
855
|
+
# From rails 8.2 this will be used by fixtures code.
|
|
856
|
+
# https://github.com/rails/rails/commit/3415223ed2765c61ae348622dc8d2681efd910d7
|
|
857
|
+
def reset_column_sequences!(tables)
|
|
858
|
+
views = self.views
|
|
859
|
+
|
|
860
|
+
table_renames = OnlineMigrations.config.table_renames
|
|
861
|
+
renamed_tables = table_renames.slice(*views)
|
|
862
|
+
|
|
863
|
+
column_renames = OnlineMigrations.config.column_renames
|
|
864
|
+
renamed_columns = column_renames.slice(*views)
|
|
865
|
+
|
|
866
|
+
tables = tables.map do |table|
|
|
867
|
+
if renamed_tables.key?(table)
|
|
868
|
+
renamed_tables[table]
|
|
869
|
+
elsif renamed_columns.key?(table)
|
|
870
|
+
__tmp_table_name_for_column_rename(table)
|
|
871
|
+
else
|
|
872
|
+
table
|
|
873
|
+
end
|
|
874
|
+
end
|
|
875
|
+
|
|
876
|
+
super
|
|
877
|
+
end
|
|
878
|
+
|
|
879
|
+
# @private
|
|
880
|
+
# Was used by fixtures code in rails < 8.2.
|
|
881
|
+
# Delete when rails < 8.2 is no longer supported.
|
|
855
882
|
def pk_and_sequence_for(table)
|
|
856
883
|
views = self.views
|
|
857
884
|
|
metadata
CHANGED
|
@@ -1,13 +1,13 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: online_migrations
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 0.
|
|
4
|
+
version: 0.33.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- fatkodima
|
|
8
8
|
bindir: bin
|
|
9
9
|
cert_chain: []
|
|
10
|
-
date:
|
|
10
|
+
date: 2026-02-04 00:00:00.000000000 Z
|
|
11
11
|
dependencies:
|
|
12
12
|
- !ruby/object:Gem::Dependency
|
|
13
13
|
name: activerecord
|
|
@@ -41,6 +41,8 @@ files:
|
|
|
41
41
|
- lib/generators/online_migrations/templates/add_sharding_to_online_migrations.rb.tt
|
|
42
42
|
- lib/generators/online_migrations/templates/add_timestamps_to_background_migrations.rb.tt
|
|
43
43
|
- lib/generators/online_migrations/templates/background_data_migrations_add_iteration_pause.rb.tt
|
|
44
|
+
- lib/generators/online_migrations/templates/background_data_migrations_remove_iteration_pause_default.rb.tt
|
|
45
|
+
- lib/generators/online_migrations/templates/background_migrations_change_status_default.rb.tt
|
|
44
46
|
- lib/generators/online_migrations/templates/background_schema_migrations_change_unique_index.rb.tt
|
|
45
47
|
- lib/generators/online_migrations/templates/change_background_data_migrations.rb.tt
|
|
46
48
|
- lib/generators/online_migrations/templates/create_background_schema_migrations.rb.tt
|