maintenance_tasks 1.5.0 → 1.6.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +159 -44
- data/app/helpers/maintenance_tasks/application_helper.rb +1 -0
- data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb +20 -4
- data/app/models/maintenance_tasks/application_record.rb +1 -0
- data/app/models/maintenance_tasks/{csv_collection.rb → csv_collection_builder.rb} +11 -12
- data/app/models/maintenance_tasks/null_collection_builder.rb +31 -0
- data/app/models/maintenance_tasks/progress.rb +8 -3
- data/app/models/maintenance_tasks/run.rb +26 -5
- data/app/models/maintenance_tasks/runner.rb +17 -5
- data/app/models/maintenance_tasks/runs_page.rb +1 -0
- data/app/{tasks → models}/maintenance_tasks/task.rb +95 -13
- data/app/models/maintenance_tasks/task_data.rb +2 -1
- data/app/validators/maintenance_tasks/run_status_validator.rb +1 -0
- data/config/routes.rb +1 -0
- data/db/migrate/20201211151756_create_maintenance_tasks_runs.rb +1 -0
- data/db/migrate/20210225152418_remove_index_on_task_name.rb +1 -0
- data/db/migrate/20210517131953_add_arguments_to_maintenance_tasks_runs.rb +1 -0
- data/lib/generators/maintenance_tasks/install_generator.rb +1 -0
- data/lib/generators/maintenance_tasks/templates/task.rb.tt +3 -1
- data/lib/maintenance_tasks/cli.rb +6 -5
- data/lib/maintenance_tasks/engine.rb +1 -0
- data/lib/maintenance_tasks.rb +2 -1
- metadata +6 -5
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 8affacb1fe5aa5298899bb27875132a4b299b1bd89e239b8fc234df1d00e171d
|
4
|
+
data.tar.gz: cd9351e3abed12d1c1bede76cea7ac1d664159983b3ece57fe2707a0de971db5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c0459cf005532667f4d5bbc8a1b05adc5e4aa57bccb4b4a3e9b9338589ff73043272201e5cd0a86ed0abfa38430b18ae5021ec1991a81f064444665be8de4e3f
|
7
|
+
data.tar.gz: d1f9b671f215a2d4a908a9a2353f02d6800511ea8d3337b4074712ec3aed7bdcd3a6b243960915a11e4ee31ca1b756d1992ce3af5a5cd27457a9016b42136842
|
data/README.md
CHANGED
@@ -57,6 +57,7 @@ Example:
|
|
57
57
|
|
58
58
|
```ruby
|
59
59
|
# app/tasks/maintenance/update_posts_task.rb
|
60
|
+
|
60
61
|
module Maintenance
|
61
62
|
class UpdatePostsTask < MaintenanceTasks::Task
|
62
63
|
def collection
|
@@ -68,7 +69,7 @@ module Maintenance
|
|
68
69
|
end
|
69
70
|
|
70
71
|
def process(post)
|
71
|
-
post.update!(content:
|
72
|
+
post.update!(content: "New content!")
|
72
73
|
end
|
73
74
|
end
|
74
75
|
end
|
@@ -78,8 +79,8 @@ end
|
|
78
79
|
|
79
80
|
You can also write a Task that iterates on a CSV file. Note that writing CSV
|
80
81
|
Tasks **requires Active Storage to be configured**. Ensure that the dependency
|
81
|
-
is specified in your application's Gemfile, and that you've followed the
|
82
|
-
|
82
|
+
is specified in your application's Gemfile, and that you've followed the [setup
|
83
|
+
instuctions][setup].
|
83
84
|
|
84
85
|
[setup]: https://edgeguides.rubyonrails.org/active_storage_overview.html#setup
|
85
86
|
|
@@ -95,6 +96,7 @@ The generated task is a subclass of `MaintenanceTasks::Task` that implements:
|
|
95
96
|
|
96
97
|
```ruby
|
97
98
|
# app/tasks/maintenance/import_posts_task.rb
|
99
|
+
|
98
100
|
module Maintenance
|
99
101
|
class ImportPostsTask < MaintenanceTasks::Task
|
100
102
|
csv_collection
|
@@ -112,16 +114,20 @@ title,content
|
|
112
114
|
My Title,Hello World!
|
113
115
|
```
|
114
116
|
|
117
|
+
The files uploaded to your Active Storage service provider will be renamed first
|
118
|
+
to include an ISO8601 timestamp and the Task name in snake case format.
|
119
|
+
|
115
120
|
### Processing Batch Collections
|
116
121
|
|
117
122
|
The Maintenance Tasks gem supports processing Active Records in batches. This
|
118
123
|
can reduce the number of calls your Task makes to the database. Use
|
119
|
-
`ActiveRecord::Batches#in_batches` on the relation returned by your collection
|
120
|
-
|
121
|
-
specified.
|
124
|
+
`ActiveRecord::Batches#in_batches` on the relation returned by your collection
|
125
|
+
to specify that your Task should process batches instead of records. Active
|
126
|
+
Record defaults to 1000 records by batch, but a custom size can be specified.
|
122
127
|
|
123
128
|
```ruby
|
124
129
|
# app/tasks/maintenance/update_posts_in_batches_task.rb
|
130
|
+
|
125
131
|
module Maintenance
|
126
132
|
class UpdatePostsInBatchesTask < MaintenanceTasks::Task
|
127
133
|
def collection
|
@@ -147,10 +153,11 @@ your collection, and your Task's progress will be displayed in terms of batches
|
|
147
153
|
**Important!** Batches should only be used if `#process` is performing a batch
|
148
154
|
operation such as `#update_all` or `#delete_all`. If you need to iterate over
|
149
155
|
individual records, you should define a collection that [returns an
|
150
|
-
`ActiveRecord::Relation`](#creating-a-task). This uses batching
|
151
|
-
|
152
|
-
|
153
|
-
records when calling `each` (or any `Enumerable` method)
|
156
|
+
`ActiveRecord::Relation`](#creating-a-task). This uses batching internally, but
|
157
|
+
loads the records with one SQL query. Conversely, batch collections load the
|
158
|
+
primary keys of the records of the batch first, and then perform an additional
|
159
|
+
query to load the records when calling `each` (or any `Enumerable` method)
|
160
|
+
inside `#process`.
|
154
161
|
|
155
162
|
### Throttling
|
156
163
|
|
@@ -162,6 +169,7 @@ Specify the throttle condition as a block:
|
|
162
169
|
|
163
170
|
```ruby
|
164
171
|
# app/tasks/maintenance/update_posts_throttled_task.rb
|
172
|
+
|
165
173
|
module Maintenance
|
166
174
|
class UpdatePostsThrottledTask < MaintenanceTasks::Task
|
167
175
|
throttle_on(backoff: 1.minute) do
|
@@ -195,12 +203,12 @@ conditions.
|
|
195
203
|
### Custom Task Parameters
|
196
204
|
|
197
205
|
Tasks may need additional information, supplied via parameters, to run.
|
198
|
-
Parameters can be defined as Active Model Attributes in a Task, and then
|
199
|
-
|
200
|
-
`#process`.
|
206
|
+
Parameters can be defined as Active Model Attributes in a Task, and then become
|
207
|
+
accessible to any of Task's methods: `#collection`, `#count`, or `#process`.
|
201
208
|
|
202
209
|
```ruby
|
203
210
|
# app/tasks/maintenance/update_posts_via_params_task.rb
|
211
|
+
|
204
212
|
module Maintenance
|
205
213
|
class UpdatePostsViaParamsTask < MaintenanceTasks::Task
|
206
214
|
attribute :updated_content, :string
|
@@ -227,6 +235,73 @@ to run. Since arguments are specified in the user interface via text area
|
|
227
235
|
inputs, it's important to check that they conform to the format your Task
|
228
236
|
expects, and to sanitize any inputs if necessary.
|
229
237
|
|
238
|
+
### Using Task Callbacks
|
239
|
+
|
240
|
+
The Task provides callbacks that hook into its life cycle.
|
241
|
+
|
242
|
+
Available callbacks are:
|
243
|
+
|
244
|
+
`after_start`
|
245
|
+
`after_pause`
|
246
|
+
`after_interrupt`
|
247
|
+
`after_cancel`
|
248
|
+
`after_complete`
|
249
|
+
`after_error`
|
250
|
+
|
251
|
+
```ruby
|
252
|
+
module Maintenance
|
253
|
+
class UpdatePostsTask < MaintenanceTasks::Task
|
254
|
+
after_start :notify
|
255
|
+
|
256
|
+
def notify
|
257
|
+
NotifyJob.perform_later(self.class.name)
|
258
|
+
end
|
259
|
+
|
260
|
+
# ...
|
261
|
+
end
|
262
|
+
end
|
263
|
+
```
|
264
|
+
|
265
|
+
Note: The `after_error` callback is guaranteed to complete,
|
266
|
+
so any exceptions raised in your callback code are ignored.
|
267
|
+
If your `after_error` callback code can raise an exception,
|
268
|
+
you'll need to rescue it and handle it appropriately
|
269
|
+
within the callback.
|
270
|
+
|
271
|
+
```ruby
|
272
|
+
module Maintenance
|
273
|
+
class UpdatePostsTask < MaintenanceTasks::Task
|
274
|
+
after_error :dangerous_notify
|
275
|
+
|
276
|
+
def dangerous_notify
|
277
|
+
# This error is rescued in favour of the original error causing the error flow.
|
278
|
+
raise NotDeliveredError
|
279
|
+
end
|
280
|
+
|
281
|
+
# ...
|
282
|
+
end
|
283
|
+
end
|
284
|
+
```
|
285
|
+
|
286
|
+
If any of the other callbacks cause an exception,
|
287
|
+
it will be handled by the error handler,
|
288
|
+
and will cause the task to stop running.
|
289
|
+
|
290
|
+
Callback behaviour can be shared across all tasks using an initializer.
|
291
|
+
|
292
|
+
```ruby
|
293
|
+
# config/initializer/maintenance_tasks.rb
|
294
|
+
Rails.autoloaders.main.on_load("MaintenanceTasks::Task") do
|
295
|
+
MaintenanceTasks::Task.class_eval do
|
296
|
+
after_start(:notify)
|
297
|
+
|
298
|
+
private
|
299
|
+
|
300
|
+
def notify; end
|
301
|
+
end
|
302
|
+
end
|
303
|
+
```
|
304
|
+
|
230
305
|
### Considerations when writing Tasks
|
231
306
|
|
232
307
|
MaintenanceTasks relies on the queue adapter configured for your application to
|
@@ -259,7 +334,7 @@ Example:
|
|
259
334
|
```ruby
|
260
335
|
# test/tasks/maintenance/update_posts_task_test.rb
|
261
336
|
|
262
|
-
require
|
337
|
+
require "test_helper"
|
263
338
|
|
264
339
|
module Maintenance
|
265
340
|
class UpdatePostsTaskTest < ActiveSupport::TestCase
|
@@ -268,7 +343,7 @@ module Maintenance
|
|
268
343
|
|
269
344
|
Maintenance::UpdatePostsTask.process(post)
|
270
345
|
|
271
|
-
assert_equal
|
346
|
+
assert_equal "New content!", post.content
|
272
347
|
end
|
273
348
|
end
|
274
349
|
end
|
@@ -281,20 +356,50 @@ takes a `CSV::Row` as an argument. You can pass a row, or a hash with string
|
|
281
356
|
keys to `#process` from your test.
|
282
357
|
|
283
358
|
```ruby
|
284
|
-
#
|
359
|
+
# test/tasks/maintenance/import_posts_task_test.rb
|
360
|
+
|
361
|
+
require "test_helper"
|
362
|
+
|
285
363
|
module Maintenance
|
286
364
|
class ImportPostsTaskTest < ActiveSupport::TestCase
|
287
365
|
test "#process performs a task iteration" do
|
288
366
|
assert_difference -> { Post.count } do
|
289
367
|
Maintenance::UpdatePostsTask.process({
|
290
|
-
|
291
|
-
|
368
|
+
"title" => "My Title",
|
369
|
+
"content" => "Hello World!",
|
292
370
|
})
|
293
371
|
end
|
294
372
|
|
295
373
|
post = Post.last
|
296
|
-
assert_equal
|
297
|
-
assert_equal
|
374
|
+
assert_equal "My Title", post.title
|
375
|
+
assert_equal "Hello World!", post.content
|
376
|
+
end
|
377
|
+
end
|
378
|
+
end
|
379
|
+
```
|
380
|
+
|
381
|
+
### Writing tests for a Task with parameters
|
382
|
+
|
383
|
+
Tests for tasks with parameters need to instatiate the task class in order to
|
384
|
+
assign attributes. Once the task instance is setup, you may test `#process`
|
385
|
+
normally.
|
386
|
+
|
387
|
+
```ruby
|
388
|
+
# test/tasks/maintenance/update_posts_via_params_task_test.rb
|
389
|
+
|
390
|
+
require "test_helper"
|
391
|
+
|
392
|
+
module Maintenance
|
393
|
+
class UpdatePostsViaParamsTaskTest < ActiveSupport::TestCase
|
394
|
+
setup do
|
395
|
+
@task = UpdatePostsViaParamsTask.new
|
396
|
+
@task.updated_content = "Testing"
|
397
|
+
end
|
398
|
+
|
399
|
+
test "#process performs a task iteration" do
|
400
|
+
assert_difference -> { Post.first.content } do
|
401
|
+
task.process(Post.first)
|
402
|
+
end
|
298
403
|
end
|
299
404
|
end
|
300
405
|
end
|
@@ -313,20 +418,21 @@ $ bundle exec maintenance_tasks perform Maintenance::UpdatePostsTask
|
|
313
418
|
To run a Task that processes CSVs from the command line, use the --csv option:
|
314
419
|
|
315
420
|
```bash
|
316
|
-
$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv
|
421
|
+
$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv "path/to/my_csv.csv"
|
317
422
|
```
|
318
423
|
|
319
424
|
To run a Task that takes arguments from the command line, use the --arguments
|
320
|
-
option, passing arguments as a set of
|
425
|
+
option, passing arguments as a set of \<key>:\<value> pairs:
|
321
426
|
|
322
427
|
```bash
|
323
|
-
$ bundle exec maintenance_tasks perform Maintenance::ParamsTask
|
428
|
+
$ bundle exec maintenance_tasks perform Maintenance::ParamsTask \
|
429
|
+
--arguments post_ids:1,2,3 content:"Hello, World!"
|
324
430
|
```
|
325
431
|
|
326
432
|
You can also run a Task in Ruby by sending `run` with a Task name to Runner:
|
327
433
|
|
328
434
|
```ruby
|
329
|
-
MaintenanceTasks::Runner.run(name:
|
435
|
+
MaintenanceTasks::Runner.run(name: "Maintenance::UpdatePostsTask")
|
330
436
|
```
|
331
437
|
|
332
438
|
To run a Task that processes CSVs using the Runner, provide a Hash containing an
|
@@ -334,8 +440,8 @@ open IO object and a filename to `run`:
|
|
334
440
|
|
335
441
|
```ruby
|
336
442
|
MaintenanceTasks::Runner.run(
|
337
|
-
name:
|
338
|
-
csv_file: { io: File.open(
|
443
|
+
name: "Maintenance::ImportPostsTask",
|
444
|
+
csv_file: { io: File.open("path/to/my_csv.csv"), filename: "my_csv.csv" }
|
339
445
|
)
|
340
446
|
```
|
341
447
|
|
@@ -358,8 +464,8 @@ a Task can be in:
|
|
358
464
|
* **enqueued**: A Task that is waiting to be performed after a user has
|
359
465
|
instructed it to run.
|
360
466
|
* **running**: A Task that is currently being performed by a job worker.
|
361
|
-
* **pausing**: A Task that was paused by a user, but needs to finish work
|
362
|
-
|
467
|
+
* **pausing**: A Task that was paused by a user, but needs to finish work before
|
468
|
+
stopping.
|
363
469
|
* **paused**: A Task that was paused by a user and is not performing. It can be
|
364
470
|
resumed.
|
365
471
|
* **interrupted**: A Task that has been momentarily interrupted by the job
|
@@ -433,6 +539,7 @@ you can define an error handler:
|
|
433
539
|
|
434
540
|
```ruby
|
435
541
|
# config/initializers/maintenance_tasks.rb
|
542
|
+
|
436
543
|
MaintenanceTasks.error_handler = ->(error, task_context, _errored_element) do
|
437
544
|
Bugsnag.notify(error) do |notification|
|
438
545
|
notification.add_tab(:task, task_context)
|
@@ -448,18 +555,18 @@ The error handler should be a lambda that accepts three arguments:
|
|
448
555
|
* `task_name`: The name of the Task that errored
|
449
556
|
* `started_at`: The time the Task started
|
450
557
|
* `ended_at`: The time the Task errored
|
558
|
+
|
451
559
|
Note that `task_context` may be empty if the Task produced an error before any
|
452
560
|
context could be gathered (for example, if deserializing the job to process
|
453
561
|
your Task failed).
|
454
|
-
* `errored_element`: The element, if any, that was being processed when the
|
455
|
-
|
456
|
-
|
457
|
-
|
458
|
-
|
459
|
-
|
460
|
-
|
461
|
-
|
462
|
-
report.
|
562
|
+
* `errored_element`: The element, if any, that was being processed when the Task
|
563
|
+
raised an exception. If you would like to pass this object to your exception
|
564
|
+
monitoring service, make sure you **sanitize the object** to avoid leaking
|
565
|
+
sensitive data and **convert it to a format** that is compatible with your bug
|
566
|
+
tracker. For example, Bugsnag only sends the id and class name of Active
|
567
|
+
Record objects in order to protect sensitive data. CSV rows, on the other
|
568
|
+
hand, are converted to strings and passed raw to Bugsnag, so make sure to
|
569
|
+
filter any personal data from these objects before adding them to a report.
|
463
570
|
|
464
571
|
#### Customizing the maintenance tasks module
|
465
572
|
|
@@ -468,7 +575,8 @@ tasks will be placed.
|
|
468
575
|
|
469
576
|
```ruby
|
470
577
|
# config/initializers/maintenance_tasks.rb
|
471
|
-
|
578
|
+
|
579
|
+
MaintenanceTasks.tasks_module = "TaskModule"
|
472
580
|
```
|
473
581
|
|
474
582
|
If no value is specified, it will default to `Maintenance`.
|
@@ -481,9 +589,11 @@ maintenance tasks in your application.
|
|
481
589
|
|
482
590
|
```ruby
|
483
591
|
# config/initializers/maintenance_tasks.rb
|
592
|
+
|
484
593
|
MaintenanceTasks.job = 'CustomTaskJob'
|
485
594
|
|
486
595
|
# app/jobs/custom_task_job.rb
|
596
|
+
|
487
597
|
class CustomTaskJob < MaintenanceTasks::TaskJob
|
488
598
|
queue_as :low_priority
|
489
599
|
end
|
@@ -491,8 +601,8 @@ end
|
|
491
601
|
|
492
602
|
The Job class **must inherit** from `MaintenanceTasks::TaskJob`.
|
493
603
|
|
494
|
-
Note that `retry_on` is not supported for custom Job
|
495
|
-
|
604
|
+
Note that `retry_on` is not supported for custom Job classes, so failed jobs
|
605
|
+
cannot be retried.
|
496
606
|
|
497
607
|
#### Customizing the rate at which task progress gets updated
|
498
608
|
|
@@ -502,6 +612,7 @@ task progress gets persisted to the database. It can be a `Numeric` value or an
|
|
502
612
|
|
503
613
|
```ruby
|
504
614
|
# config/initializers/maintenance_tasks.rb
|
615
|
+
|
505
616
|
MaintenanceTasks.ticker_delay = 2.seconds
|
506
617
|
```
|
507
618
|
|
@@ -516,6 +627,7 @@ key, as specified in your application's `config/storage.yml`:
|
|
516
627
|
|
517
628
|
```yaml
|
518
629
|
# config/storage.yml
|
630
|
+
|
519
631
|
user_data:
|
520
632
|
service: GCS
|
521
633
|
credentials: <%= Rails.root.join("path/to/user/data/keyfile.json") %>
|
@@ -531,6 +643,7 @@ internal:
|
|
531
643
|
|
532
644
|
```ruby
|
533
645
|
# config/initializers/maintenance_tasks.rb
|
646
|
+
|
534
647
|
MaintenanceTasks.active_storage_service = :internal
|
535
648
|
```
|
536
649
|
|
@@ -545,6 +658,7 @@ An `ActiveSupport::BacktraceCleaner` should be used.
|
|
545
658
|
|
546
659
|
```ruby
|
547
660
|
# config/initializers/maintenance_tasks.rb
|
661
|
+
|
548
662
|
cleaner = ActiveSupport::BacktraceCleaner.new
|
549
663
|
cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
|
550
664
|
|
@@ -568,10 +682,11 @@ This ensures that new migrations are installed and run as well.
|
|
568
682
|
**What if I've deleted my previous Maintenance Task migrations?**
|
569
683
|
|
570
684
|
The install command will attempt to reinstall these old migrations and migrating
|
571
|
-
the database will cause problems. Use `bin/rails
|
572
|
-
to copy the gem's migrations to your
|
573
|
-
notes to see if any new migrations were
|
574
|
-
Ensure that these are kept, but remove any
|
685
|
+
the database will cause problems. Use `bin/rails
|
686
|
+
maintenance_tasks:install:migrations` to copy the gem's migrations to your
|
687
|
+
`db/migrate` folder. Check the release notes to see if any new migrations were
|
688
|
+
added since your last gem upgrade. Ensure that these are kept, but remove any
|
689
|
+
migrations that already ran.
|
575
690
|
|
576
691
|
Run the migrations using `bin/rails db:migrate`.
|
577
692
|
|
@@ -1,4 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
module MaintenanceTasks
|
3
4
|
# Concern that holds the behaviour of the job that runs the tasks. It is
|
4
5
|
# included in {TaskJob} and if MaintenanceTasks.job is overridden, it must be
|
@@ -90,11 +91,11 @@ module MaintenanceTasks
|
|
90
91
|
def before_perform
|
91
92
|
@run = arguments.first
|
92
93
|
@task = @run.task
|
93
|
-
if @task.
|
94
|
+
if @task.has_csv_content?
|
94
95
|
@task.csv_content = @run.csv_file.download
|
95
96
|
end
|
96
97
|
|
97
|
-
@run.running
|
98
|
+
@run.running
|
98
99
|
|
99
100
|
@ticker = Ticker.new(MaintenanceTasks.ticker_delay) do |ticks, duration|
|
100
101
|
@run.persist_progress(ticks, duration)
|
@@ -105,22 +106,29 @@ module MaintenanceTasks
|
|
105
106
|
count = @task.count
|
106
107
|
count = @enumerator&.size if count == :no_count
|
107
108
|
@run.update!(started_at: Time.now, tick_total: count)
|
109
|
+
@task.run_callbacks(:start)
|
108
110
|
end
|
109
111
|
|
110
112
|
def on_complete
|
111
113
|
@run.status = :succeeded
|
112
114
|
@run.ended_at = Time.now
|
115
|
+
@task.run_callbacks(:complete)
|
113
116
|
end
|
114
117
|
|
115
118
|
def on_shutdown
|
116
119
|
if @run.cancelling?
|
117
120
|
@run.status = :cancelled
|
121
|
+
@task.run_callbacks(:cancel)
|
118
122
|
@run.ended_at = Time.now
|
123
|
+
elsif @run.pausing?
|
124
|
+
@run.status = :paused
|
125
|
+
@task.run_callbacks(:pause)
|
119
126
|
else
|
120
|
-
@run.status =
|
121
|
-
@
|
127
|
+
@run.status = :interrupted
|
128
|
+
@task.run_callbacks(:interrupt)
|
122
129
|
end
|
123
130
|
|
131
|
+
@run.cursor = cursor_position
|
124
132
|
@ticker.persist
|
125
133
|
end
|
126
134
|
|
@@ -163,7 +171,15 @@ module MaintenanceTasks
|
|
163
171
|
task_context = {}
|
164
172
|
end
|
165
173
|
errored_element = @errored_element if defined?(@errored_element)
|
174
|
+
run_error_callback
|
175
|
+
ensure
|
166
176
|
MaintenanceTasks.error_handler.call(error, task_context, errored_element)
|
167
177
|
end
|
178
|
+
|
179
|
+
def run_error_callback
|
180
|
+
@task.run_callbacks(:error) if defined?(@task)
|
181
|
+
rescue
|
182
|
+
nil
|
183
|
+
end
|
168
184
|
end
|
169
185
|
end
|
@@ -3,22 +3,16 @@
|
|
3
3
|
require "csv"
|
4
4
|
|
5
5
|
module MaintenanceTasks
|
6
|
-
#
|
7
|
-
# processing CSV files.
|
6
|
+
# Strategy for building a Task that processes CSV files.
|
8
7
|
#
|
9
8
|
# @api private
|
10
|
-
|
11
|
-
# The contents of a CSV file to be processed by a Task.
|
12
|
-
#
|
13
|
-
# @return [String] the content of the CSV file to process.
|
14
|
-
attr_accessor :csv_content
|
15
|
-
|
9
|
+
class CsvCollectionBuilder
|
16
10
|
# Defines the collection to be iterated over, based on the provided CSV.
|
17
11
|
#
|
18
12
|
# @return [CSV] the CSV object constructed from the specified CSV content,
|
19
13
|
# with headers.
|
20
|
-
def collection
|
21
|
-
CSV.new(csv_content, headers: true)
|
14
|
+
def collection(task)
|
15
|
+
CSV.new(task.csv_content, headers: true)
|
22
16
|
end
|
23
17
|
|
24
18
|
# The number of rows to be processed. Excludes the header row from the count
|
@@ -26,8 +20,13 @@ module MaintenanceTasks
|
|
26
20
|
# an approximation based on the number of new lines.
|
27
21
|
#
|
28
22
|
# @return [Integer] the approximate number of rows to process.
|
29
|
-
def count
|
30
|
-
csv_content.count("\n") - 1
|
23
|
+
def count(task)
|
24
|
+
task.csv_content.count("\n") - 1
|
25
|
+
end
|
26
|
+
|
27
|
+
# Return that the Task processes CSV content.
|
28
|
+
def has_csv_content?
|
29
|
+
true
|
31
30
|
end
|
32
31
|
end
|
33
32
|
end
|
@@ -0,0 +1,31 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
module MaintenanceTasks
|
4
|
+
# Base strategy for building a collection-based Task to be performed.
|
5
|
+
class NullCollectionBuilder
|
6
|
+
# Placeholder method to raise in case a subclass fails to implement the
|
7
|
+
# expected instance method.
|
8
|
+
#
|
9
|
+
# @raise [NotImplementedError] with a message advising subclasses to
|
10
|
+
# implement an override for this method.
|
11
|
+
def collection(task)
|
12
|
+
raise NoMethodError, "#{task.class.name} must implement `collection`."
|
13
|
+
end
|
14
|
+
|
15
|
+
# Total count of iterations to be performed.
|
16
|
+
#
|
17
|
+
# Tasks override this method to define the total amount of iterations
|
18
|
+
# expected at the start of the run. Return +nil+ if the amount is
|
19
|
+
# undefined, or counting would be prohibitive for your database.
|
20
|
+
#
|
21
|
+
# @return [Integer, nil]
|
22
|
+
def count(task)
|
23
|
+
:no_count
|
24
|
+
end
|
25
|
+
|
26
|
+
# Return that the Task does not process CSV content.
|
27
|
+
def has_csv_content?
|
28
|
+
false
|
29
|
+
end
|
30
|
+
end
|
31
|
+
end
|
@@ -50,14 +50,19 @@ module MaintenanceTasks
|
|
50
50
|
def text
|
51
51
|
count = @run.tick_count
|
52
52
|
total = @run.tick_total
|
53
|
+
|
53
54
|
if !total?
|
54
|
-
"Processed #{
|
55
|
+
"Processed #{number_to_delimited(count)} "\
|
56
|
+
"#{"item".pluralize(count)}."
|
55
57
|
elsif over_total?
|
56
|
-
"Processed #{
|
58
|
+
"Processed #{number_to_delimited(count)} "\
|
59
|
+
"#{"item".pluralize(count)} "\
|
60
|
+
"(expected #{number_to_delimited(total)})."
|
57
61
|
else
|
58
62
|
percentage = 100.0 * count / total
|
59
63
|
|
60
|
-
"Processed #{count} out of
|
64
|
+
"Processed #{number_to_delimited(count)} out of "\
|
65
|
+
"#{number_to_delimited(total)} #{"item".pluralize(total)} "\
|
61
66
|
"(#{number_to_percentage(percentage, precision: 0)})."
|
62
67
|
end
|
63
68
|
end
|
@@ -1,4 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
module MaintenanceTasks
|
3
4
|
# Model that persists information related to a task being run from the UI.
|
4
5
|
#
|
@@ -25,6 +26,10 @@ module MaintenanceTasks
|
|
25
26
|
:cancelling,
|
26
27
|
:interrupted,
|
27
28
|
]
|
29
|
+
STOPPING_STATUSES = [
|
30
|
+
:pausing,
|
31
|
+
:cancelling,
|
32
|
+
]
|
28
33
|
COMPLETED_STATUSES = [:succeeded, :errored, :cancelled]
|
29
34
|
COMPLETED_RUNS_LIMIT = 10
|
30
35
|
STUCK_TASK_TIMEOUT = 5.minutes
|
@@ -87,6 +92,7 @@ module MaintenanceTasks
|
|
87
92
|
#
|
88
93
|
# @param error [StandardError] the Error being persisted.
|
89
94
|
def persist_error(error)
|
95
|
+
self.started_at ||= Time.now
|
90
96
|
update!(
|
91
97
|
status: :errored,
|
92
98
|
error_class: error.class.to_s,
|
@@ -103,8 +109,8 @@ module MaintenanceTasks
|
|
103
109
|
#
|
104
110
|
# @return [MaintenanceTasks::Run] the Run record with its updated status.
|
105
111
|
def reload_status
|
106
|
-
updated_status =
|
107
|
-
|
112
|
+
updated_status = self.class.uncached do
|
113
|
+
self.class.where(id: id).pluck(:status).first
|
108
114
|
end
|
109
115
|
self.status = updated_status
|
110
116
|
clear_attribute_changes([:status])
|
@@ -116,7 +122,7 @@ module MaintenanceTasks
|
|
116
122
|
#
|
117
123
|
# @return [Boolean] whether the Run is stopping.
|
118
124
|
def stopping?
|
119
|
-
|
125
|
+
STOPPING_STATUSES.include?(status.to_sym)
|
120
126
|
end
|
121
127
|
|
122
128
|
# Returns whether the Run is stopped, which is defined as having a status of
|
@@ -167,6 +173,21 @@ module MaintenanceTasks
|
|
167
173
|
seconds_to_finished.seconds
|
168
174
|
end
|
169
175
|
|
176
|
+
# Mark a Run as running.
|
177
|
+
#
|
178
|
+
# If the run is stopping already, it will not transition to running.
|
179
|
+
def running
|
180
|
+
return if stopping?
|
181
|
+
updated = self.class.where(id: id).where.not(status: STOPPING_STATUSES)
|
182
|
+
.update_all(status: :running, updated_at: Time.now) > 0
|
183
|
+
if updated
|
184
|
+
self.status = :running
|
185
|
+
clear_attribute_changes([:status])
|
186
|
+
else
|
187
|
+
reload_status
|
188
|
+
end
|
189
|
+
end
|
190
|
+
|
170
191
|
# Cancels a Run.
|
171
192
|
#
|
172
193
|
# If the Run is paused, it will transition directly to cancelled, since the
|
@@ -200,9 +221,9 @@ module MaintenanceTasks
|
|
200
221
|
# should not have an attachment to be valid. The appropriate error is added
|
201
222
|
# if the Run does not meet the above criteria.
|
202
223
|
def csv_attachment_presence
|
203
|
-
if Task.named(task_name)
|
224
|
+
if Task.named(task_name).has_csv_content? && !csv_file.attached?
|
204
225
|
errors.add(:csv_file, "must be attached to CSV Task.")
|
205
|
-
elsif !
|
226
|
+
elsif !Task.named(task_name).has_csv_content? && csv_file.present?
|
206
227
|
errors.add(:csv_file, "should not be attached to non-CSV Task.")
|
207
228
|
end
|
208
229
|
rescue Task::NotFoundError
|
@@ -47,11 +47,14 @@ module MaintenanceTasks
|
|
47
47
|
# creating the Run.
|
48
48
|
# @raise [ActiveRecord::ValueTooLong] if the creation of the Run fails due
|
49
49
|
# to a value being too long for the column type.
|
50
|
-
def run(name:, csv_file: nil, arguments: {})
|
51
|
-
run =
|
52
|
-
|
53
|
-
|
54
|
-
|
50
|
+
def run(name:, csv_file: nil, arguments: {}, run_model: Run)
|
51
|
+
run = run_model.active.find_by(task_name: name) ||
|
52
|
+
run_model.new(task_name: name, arguments: arguments)
|
53
|
+
if csv_file
|
54
|
+
run.csv_file.attach(csv_file)
|
55
|
+
run.csv_file.filename = filename(name)
|
56
|
+
end
|
57
|
+
job = instantiate_job(run)
|
55
58
|
run.job_id = job.job_id
|
56
59
|
yield run if block_given?
|
57
60
|
run.enqueued!
|
@@ -70,5 +73,14 @@ module MaintenanceTasks
|
|
70
73
|
run.persist_error(error)
|
71
74
|
raise EnqueuingError, run
|
72
75
|
end
|
76
|
+
|
77
|
+
def filename(task_name)
|
78
|
+
formatted_task_name = task_name.underscore.gsub("/", "_")
|
79
|
+
"#{Time.now.utc.strftime("%Y%m%dT%H%M%SZ")}_#{formatted_task_name}.csv"
|
80
|
+
end
|
81
|
+
|
82
|
+
def instantiate_job(run)
|
83
|
+
MaintenanceTasks.job.constantize.new(run)
|
84
|
+
end
|
73
85
|
end
|
74
86
|
end
|
@@ -1,8 +1,10 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
module MaintenanceTasks
|
3
4
|
# Base class that is inherited by the host application's task classes.
|
4
5
|
class Task
|
5
6
|
extend ActiveSupport::DescendantsTracker
|
7
|
+
include ActiveSupport::Callbacks
|
6
8
|
include ActiveModel::Attributes
|
7
9
|
include ActiveModel::AttributeAssignment
|
8
10
|
include ActiveModel::Validations
|
@@ -16,6 +18,11 @@ module MaintenanceTasks
|
|
16
18
|
# @api private
|
17
19
|
class_attribute :throttle_conditions, default: []
|
18
20
|
|
21
|
+
class_attribute :collection_builder_strategy,
|
22
|
+
default: NullCollectionBuilder.new
|
23
|
+
|
24
|
+
define_callbacks :start, :complete, :error, :cancel, :pause, :interrupt
|
25
|
+
|
19
26
|
class << self
|
20
27
|
# Finds a Task with the given name.
|
21
28
|
#
|
@@ -47,11 +54,19 @@ module MaintenanceTasks
|
|
47
54
|
# An input to upload a CSV will be added in the form to start a Run. The
|
48
55
|
# collection and count method are implemented.
|
49
56
|
def csv_collection
|
50
|
-
|
57
|
+
unless defined?(ActiveStorage)
|
51
58
|
raise NotImplementedError, "Active Storage needs to be installed\n"\
|
52
59
|
"To resolve this issue run: bin/rails active_storage:install"
|
53
60
|
end
|
54
|
-
|
61
|
+
|
62
|
+
self.collection_builder_strategy = CsvCollectionBuilder.new
|
63
|
+
end
|
64
|
+
|
65
|
+
# Returns whether the Task handles CSV.
|
66
|
+
#
|
67
|
+
# @return [Boolean] whether the Task handles CSV.
|
68
|
+
def has_csv_content?
|
69
|
+
collection_builder_strategy.has_csv_content?
|
55
70
|
end
|
56
71
|
|
57
72
|
# Processes one item.
|
@@ -94,6 +109,54 @@ module MaintenanceTasks
|
|
94
109
|
]
|
95
110
|
end
|
96
111
|
|
112
|
+
# Initialize a callback to run after the task starts.
|
113
|
+
#
|
114
|
+
# @param filter_list apply filters to the callback
|
115
|
+
# (see https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-set_callback)
|
116
|
+
def after_start(*filter_list, &block)
|
117
|
+
set_callback(:start, :after, *filter_list, &block)
|
118
|
+
end
|
119
|
+
|
120
|
+
# Initialize a callback to run after the task completes.
|
121
|
+
#
|
122
|
+
# @param filter_list apply filters to the callback
|
123
|
+
# (see https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-set_callback)
|
124
|
+
def after_complete(*filter_list, &block)
|
125
|
+
set_callback(:complete, :after, *filter_list, &block)
|
126
|
+
end
|
127
|
+
|
128
|
+
# Initialize a callback to run after the task pauses.
|
129
|
+
#
|
130
|
+
# @param filter_list apply filters to the callback
|
131
|
+
# (see https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-set_callback)
|
132
|
+
def after_pause(*filter_list, &block)
|
133
|
+
set_callback(:pause, :after, *filter_list, &block)
|
134
|
+
end
|
135
|
+
|
136
|
+
# Initialize a callback to run after the task is interrupted.
|
137
|
+
#
|
138
|
+
# @param filter_list apply filters to the callback
|
139
|
+
# (see https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-set_callback)
|
140
|
+
def after_interrupt(*filter_list, &block)
|
141
|
+
set_callback(:interrupt, :after, *filter_list, &block)
|
142
|
+
end
|
143
|
+
|
144
|
+
# Initialize a callback to run after the task is cancelled.
|
145
|
+
#
|
146
|
+
# @param filter_list apply filters to the callback
|
147
|
+
# (see https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-set_callback)
|
148
|
+
def after_cancel(*filter_list, &block)
|
149
|
+
set_callback(:cancel, :after, *filter_list, &block)
|
150
|
+
end
|
151
|
+
|
152
|
+
# Initialize a callback to run after the task produces an error.
|
153
|
+
#
|
154
|
+
# @param filter_list apply filters to the callback
|
155
|
+
# (see https://api.rubyonrails.org/classes/ActiveSupport/Callbacks/ClassMethods.html#method-i-set_callback)
|
156
|
+
def after_error(*filter_list, &block)
|
157
|
+
set_callback(:error, :after, *filter_list, &block)
|
158
|
+
end
|
159
|
+
|
97
160
|
private
|
98
161
|
|
99
162
|
def load_constants
|
@@ -103,13 +166,36 @@ module MaintenanceTasks
|
|
103
166
|
end
|
104
167
|
end
|
105
168
|
|
106
|
-
#
|
107
|
-
# expected instance method.
|
169
|
+
# The contents of a CSV file to be processed by a Task.
|
108
170
|
#
|
109
|
-
# @
|
110
|
-
|
171
|
+
# @return [String] the content of the CSV file to process.
|
172
|
+
def csv_content
|
173
|
+
raise NoMethodError unless has_csv_content?
|
174
|
+
|
175
|
+
@csv_content
|
176
|
+
end
|
177
|
+
|
178
|
+
# Set the contents of a CSV file to be processed by a Task.
|
179
|
+
#
|
180
|
+
# @param csv_content [String] the content of the CSV file to process.
|
181
|
+
def csv_content=(csv_content)
|
182
|
+
raise NoMethodError unless has_csv_content?
|
183
|
+
|
184
|
+
@csv_content = csv_content
|
185
|
+
end
|
186
|
+
|
187
|
+
# Returns whether the Task handles CSV.
|
188
|
+
#
|
189
|
+
# @return [Boolean] whether the Task handles CSV.
|
190
|
+
def has_csv_content?
|
191
|
+
self.class.has_csv_content?
|
192
|
+
end
|
193
|
+
|
194
|
+
# The collection to be processed, delegated to the strategy.
|
195
|
+
#
|
196
|
+
# @return the collection.
|
111
197
|
def collection
|
112
|
-
|
198
|
+
self.class.collection_builder_strategy.collection(self)
|
113
199
|
end
|
114
200
|
|
115
201
|
# Placeholder method to raise in case a subclass fails to implement the
|
@@ -123,15 +209,11 @@ module MaintenanceTasks
|
|
123
209
|
raise NoMethodError, "#{self.class.name} must implement `process`."
|
124
210
|
end
|
125
211
|
|
126
|
-
# Total count of iterations to be performed.
|
127
|
-
#
|
128
|
-
# Tasks override this method to define the total amount of iterations
|
129
|
-
# expected at the start of the run. Return +nil+ if the amount is
|
130
|
-
# undefined, or counting would be prohibitive for your database.
|
212
|
+
# Total count of iterations to be performed, delegated to the strategy.
|
131
213
|
#
|
132
214
|
# @return [Integer, nil]
|
133
215
|
def count
|
134
|
-
|
216
|
+
self.class.collection_builder_strategy.count(self)
|
135
217
|
end
|
136
218
|
end
|
137
219
|
end
|
@@ -1,4 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
module MaintenanceTasks
|
3
4
|
# Class that represents the data related to a Task. Such information can be
|
4
5
|
# sourced from a Task or from existing Run records for a Task that was since
|
@@ -129,7 +130,7 @@ module MaintenanceTasks
|
|
129
130
|
|
130
131
|
# @return [Boolean] whether the Task inherits from CsvTask.
|
131
132
|
def csv_task?
|
132
|
-
!deleted? && Task.named(name)
|
133
|
+
!deleted? && Task.named(name).has_csv_content?
|
133
134
|
end
|
134
135
|
|
135
136
|
# @return [Array<String>] the names of parameters the Task accepts.
|
data/config/routes.rb
CHANGED
@@ -9,7 +9,9 @@ module <%= tasks_module %>
|
|
9
9
|
end
|
10
10
|
|
11
11
|
def process(element)
|
12
|
-
# The work to be done in a single iteration of the task
|
12
|
+
# The work to be done in a single iteration of the task.
|
13
|
+
# This should be idempotent, as the same element may be processed more
|
14
|
+
# than once if the task is interrupted and resumed.
|
13
15
|
end
|
14
16
|
|
15
17
|
def count
|
@@ -25,12 +25,13 @@ module MaintenanceTasks
|
|
25
25
|
LONGDESC
|
26
26
|
|
27
27
|
# Specify the CSV file to process for CSV Tasks
|
28
|
-
|
29
|
-
|
30
|
-
|
28
|
+
desc = "Supply a CSV file to be processed by a CSV Task, "\
|
29
|
+
"--csv path/to/csv/file.csv"
|
30
|
+
option :csv, desc: desc
|
31
31
|
# Specify arguments to supply to a Task supporting parameters
|
32
|
-
|
33
|
-
"
|
32
|
+
desc = "Supply arguments for a Task that accepts parameters as a set of "\
|
33
|
+
"<key>:<value> pairs."
|
34
|
+
option :arguments, type: :hash, desc: desc
|
34
35
|
|
35
36
|
# Command to run a Task.
|
36
37
|
#
|
data/lib/maintenance_tasks.rb
CHANGED
@@ -1,4 +1,5 @@
|
|
1
1
|
# frozen_string_literal: true
|
2
|
+
|
2
3
|
require "action_controller"
|
3
4
|
require "action_view"
|
4
5
|
require "active_job"
|
@@ -73,7 +74,7 @@ module MaintenanceTasks
|
|
73
74
|
unless error_handler.arity == 3
|
74
75
|
ActiveSupport::Deprecation.warn(
|
75
76
|
"MaintenanceTasks.error_handler should be a lambda that takes three "\
|
76
|
-
|
77
|
+
"arguments: error, task_context, and errored_element."
|
77
78
|
)
|
78
79
|
@error_handler = ->(error, _task_context, _errored_element) do
|
79
80
|
error_handler.call(error)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: maintenance_tasks
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.6.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Shopify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-11-05 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: actionpack
|
@@ -97,14 +97,15 @@ files:
|
|
97
97
|
- app/jobs/concerns/maintenance_tasks/task_job_concern.rb
|
98
98
|
- app/jobs/maintenance_tasks/task_job.rb
|
99
99
|
- app/models/maintenance_tasks/application_record.rb
|
100
|
-
- app/models/maintenance_tasks/
|
100
|
+
- app/models/maintenance_tasks/csv_collection_builder.rb
|
101
|
+
- app/models/maintenance_tasks/null_collection_builder.rb
|
101
102
|
- app/models/maintenance_tasks/progress.rb
|
102
103
|
- app/models/maintenance_tasks/run.rb
|
103
104
|
- app/models/maintenance_tasks/runner.rb
|
104
105
|
- app/models/maintenance_tasks/runs_page.rb
|
106
|
+
- app/models/maintenance_tasks/task.rb
|
105
107
|
- app/models/maintenance_tasks/task_data.rb
|
106
108
|
- app/models/maintenance_tasks/ticker.rb
|
107
|
-
- app/tasks/maintenance_tasks/task.rb
|
108
109
|
- app/validators/maintenance_tasks/run_status_validator.rb
|
109
110
|
- app/views/layouts/maintenance_tasks/_navbar.html.erb
|
110
111
|
- app/views/layouts/maintenance_tasks/application.html.erb
|
@@ -143,7 +144,7 @@ homepage: https://github.com/Shopify/maintenance_tasks
|
|
143
144
|
licenses:
|
144
145
|
- MIT
|
145
146
|
metadata:
|
146
|
-
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v1.
|
147
|
+
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v1.6.0
|
147
148
|
allowed_push_host: https://rubygems.org
|
148
149
|
post_install_message:
|
149
150
|
rdoc_options: []
|