maintenance_tasks 1.10.3 → 2.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +94 -50
- data/app/controllers/maintenance_tasks/application_controller.rb +1 -2
- data/app/controllers/maintenance_tasks/runs_controller.rb +29 -1
- data/app/controllers/maintenance_tasks/tasks_controller.rb +8 -21
- data/app/helpers/maintenance_tasks/tasks_helper.rb +5 -4
- data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb +5 -5
- data/app/models/maintenance_tasks/batch_csv_collection_builder.rb +1 -1
- data/app/models/maintenance_tasks/run.rb +27 -8
- data/app/models/maintenance_tasks/runner.rb +18 -10
- data/app/models/maintenance_tasks/runs_page.rb +10 -4
- data/app/models/maintenance_tasks/task.rb +7 -13
- data/app/models/maintenance_tasks/task_data_index.rb +87 -0
- data/app/models/maintenance_tasks/task_data_show.rb +96 -0
- data/app/validators/maintenance_tasks/run_status_validator.rb +1 -1
- data/app/views/layouts/maintenance_tasks/application.html.erb +2 -2
- data/app/views/maintenance_tasks/runs/_run.html.erb +17 -0
- data/app/views/maintenance_tasks/tasks/_task.html.erb +1 -1
- data/app/views/maintenance_tasks/tasks/show.html.erb +32 -62
- data/config/routes.rb +2 -5
- data/db/migrate/20211210152329_add_lock_version_to_maintenance_tasks_runs.rb +7 -2
- data/db/migrate/20220713131925_add_index_on_task_name_and_status_to_runs.rb +19 -0
- data/lib/generators/maintenance_tasks/task_generator.rb +9 -5
- data/lib/generators/maintenance_tasks/templates/no_collection_task_test.rb.tt +1 -0
- data/lib/generators/maintenance_tasks/templates/task_spec.rb.tt +1 -0
- data/lib/generators/maintenance_tasks/templates/task_test.rb.tt +1 -0
- data/lib/maintenance_tasks/cli.rb +19 -3
- data/lib/maintenance_tasks/engine.rb +3 -13
- data/lib/maintenance_tasks.rb +2 -21
- metadata +6 -4
- data/app/models/maintenance_tasks/task_data.rb +0 -166
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2dcb5b54d6f3fd47db28a3f57a487c201e2109b3eb8f78efbe3c972a374999e8
|
4
|
+
data.tar.gz: eeb95960a0cac8de08aa3af7d12d47f6cfbee4d875ab74f8989f4d17ef545f6c
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e99ebde772755c6f3d88d48e65df883805b2ceb4798334f00175bc8e4c83d093bef79931bf0c0c86b303c3d94d2e0b44b3b506b1813ea6e01597ccf47f2c8182
|
7
|
+
data.tar.gz: 58d11ad3fa36721e9f29a49cecf586d2f362aaf155667ce6b7702b7acd7bb43f1defa2a2a4ed1fa259d563876b9cec5d5f7b66f4d5dcf43747009e64552fa804
|
data/README.md
CHANGED
@@ -33,8 +33,28 @@ take a look at the [Active Job documentation][active-job-docs].
|
|
33
33
|
[async-adapter]: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html
|
34
34
|
[active-job-docs]: https://guides.rubyonrails.org/active_job_basics.html#setting-the-backend
|
35
35
|
|
36
|
+
### Autoloading
|
37
|
+
|
38
|
+
The Maintenance Tasks framework does not support autoloading in `:classic` mode.
|
39
|
+
Please ensure your application is using
|
40
|
+
[Zeitwerk](https://github.com/fxn/zeitwerk) to load your code. For more
|
41
|
+
information, please consult the [Rails guides on autoloading and reloading
|
42
|
+
constants](https://guides.rubyonrails.org/autoloading_and_reloading_constants.html).
|
43
|
+
|
36
44
|
## Usage
|
37
45
|
|
46
|
+
The typical Maintenance Tasks workflow is as follows:
|
47
|
+
|
48
|
+
1. [Generate a class describing the Task](#creating-a-task) and the work to be done.
|
49
|
+
2. Run the Task
|
50
|
+
- either by [using the included web UI](#running-a-task-from-the-web-ui),
|
51
|
+
- or by [using the command line](#running-a-task-from-the-command-line),
|
52
|
+
- or by [using Ruby](#running-a-task-from-ruby).
|
53
|
+
3. [Monitor the Task](#monitoring-your-tasks-status)
|
54
|
+
- either by using the included web UI,
|
55
|
+
- or by manually checking your task’s run’s status in your database.
|
56
|
+
4. Optionally, delete the Task code if you no longer need it.
|
57
|
+
|
38
58
|
### Creating a Task
|
39
59
|
|
40
60
|
A generator is provided to create tasks. Generate a new task by running:
|
@@ -50,8 +70,13 @@ The generated task is a subclass of `MaintenanceTasks::Task` that implements:
|
|
50
70
|
* `collection`: return an Active Record Relation or an Array to be iterated
|
51
71
|
over.
|
52
72
|
* `process`: do the work of your maintenance task on a single record
|
53
|
-
|
54
|
-
|
73
|
+
|
74
|
+
Optionally, tasks can also implement a custom `#count` method, defining the
|
75
|
+
number of elements that will be iterated over. Your task’s `tick_total` will be
|
76
|
+
calculated automatically based on the collection size, but this value may be
|
77
|
+
overridden if desired using the `#count` method (this might be done, for
|
78
|
+
example, to avoid the query that would be produced to determine the size of your
|
79
|
+
collection).
|
55
80
|
|
56
81
|
Example:
|
57
82
|
|
@@ -64,10 +89,6 @@ module Maintenance
|
|
64
89
|
Post.all
|
65
90
|
end
|
66
91
|
|
67
|
-
def count
|
68
|
-
collection.count
|
69
|
-
end
|
70
|
-
|
71
92
|
def process(post)
|
72
93
|
post.update!(content: "New content!")
|
73
94
|
end
|
@@ -79,10 +100,12 @@ end
|
|
79
100
|
|
80
101
|
You can also write a Task that iterates on a CSV file. Note that writing CSV
|
81
102
|
Tasks **requires Active Storage to be configured**. Ensure that the dependency
|
82
|
-
is specified in your application
|
83
|
-
|
103
|
+
is specified in your application’s Gemfile, and that you’ve followed the [setup
|
104
|
+
instructions][storage-setup]. See also [Customizing which Active Storage service
|
105
|
+
to use][storage-customizing].
|
84
106
|
|
85
|
-
[setup]: https://edgeguides.rubyonrails.org/active_storage_overview.html#setup
|
107
|
+
[storage-setup]: https://edgeguides.rubyonrails.org/active_storage_overview.html#setup
|
108
|
+
[storage-customizing]: #customizing-which-active-storage-service-to-use
|
86
109
|
|
87
110
|
Generate a CSV Task by running:
|
88
111
|
|
@@ -115,12 +138,12 @@ My Title,Hello World!
|
|
115
138
|
```
|
116
139
|
|
117
140
|
The files uploaded to your Active Storage service provider will be renamed
|
118
|
-
to include an
|
141
|
+
to include an ISO 8601 timestamp and the Task name in snake case format.
|
119
142
|
The CSV is expected to have a trailing newline at the end of the file.
|
120
143
|
|
121
144
|
#### Batch CSV Tasks
|
122
145
|
|
123
|
-
Tasks can process CSVs in batches. Add the `in_batches` option to your task
|
146
|
+
Tasks can process CSVs in batches. Add the `in_batches` option to your task’s
|
124
147
|
`csv_collection` macro:
|
125
148
|
|
126
149
|
```ruby
|
@@ -137,12 +160,12 @@ module Maintenance
|
|
137
160
|
end
|
138
161
|
```
|
139
162
|
|
140
|
-
As with a regular CSV task, ensure you
|
163
|
+
As with a regular CSV task, ensure you’ve implemented the following method:
|
141
164
|
|
142
165
|
* `process`: do the work of your Task on a batch (array of `CSV::Row` objects).
|
143
166
|
|
144
167
|
Note that `#count` is calculated automatically based on the number of batches in
|
145
|
-
your collection, and your Task
|
168
|
+
your collection, and your Task’s progress will be displayed in terms of batches
|
146
169
|
(not the total number of rows in your CSV).
|
147
170
|
|
148
171
|
### Processing Batch Collections
|
@@ -169,13 +192,13 @@ module Maintenance
|
|
169
192
|
end
|
170
193
|
```
|
171
194
|
|
172
|
-
Ensure that you
|
195
|
+
Ensure that you’ve implemented the following methods:
|
173
196
|
|
174
197
|
* `collection`: return an `ActiveRecord::Batches::BatchEnumerator`.
|
175
198
|
* `process`: do the work of your Task on a batch (`ActiveRecord::Relation`).
|
176
199
|
|
177
200
|
Note that `#count` is calculated automatically based on the number of batches in
|
178
|
-
your collection, and your Task
|
201
|
+
your collection, and your Task’s progress will be displayed in terms of batches
|
179
202
|
(not the number of records in the relation).
|
180
203
|
|
181
204
|
**Important!** Batches should only be used if `#process` is performing a batch
|
@@ -187,7 +210,7 @@ primary keys of the records of the batch first, and then perform an additional
|
|
187
210
|
query to load the records when calling `each` (or any `Enumerable` method)
|
188
211
|
inside `#process`.
|
189
212
|
|
190
|
-
### Tasks that don
|
213
|
+
### Tasks that don’t need a Collection
|
191
214
|
|
192
215
|
Sometimes, you might want to run a Task that performs a single operation, such
|
193
216
|
as enqueuing another background job or hitting an external API. The gem supports
|
@@ -238,10 +261,6 @@ module Maintenance
|
|
238
261
|
Post.all
|
239
262
|
end
|
240
263
|
|
241
|
-
def count
|
242
|
-
collection.count
|
243
|
-
end
|
244
|
-
|
245
264
|
def process(post)
|
246
265
|
post.update!(content: "New content added on #{Time.now.utc}")
|
247
266
|
end
|
@@ -249,7 +268,7 @@ module Maintenance
|
|
249
268
|
end
|
250
269
|
```
|
251
270
|
|
252
|
-
Note that it
|
271
|
+
Note that it’s up to you to define a throttling condition that makes sense for
|
253
272
|
your app. Shopify implements `DatabaseStatus.healthy?` to check various MySQL
|
254
273
|
metrics such as replication lag, DB threads, whether DB writes are available,
|
255
274
|
etc.
|
@@ -258,7 +277,7 @@ Tasks can define multiple throttle conditions. Throttle conditions are inherited
|
|
258
277
|
by descendants, and new conditions will be appended without impacting existing
|
259
278
|
conditions.
|
260
279
|
|
261
|
-
The backoff can also be specified as a
|
280
|
+
The backoff can also be specified as a Proc:
|
262
281
|
|
263
282
|
```ruby
|
264
283
|
# app/tasks/maintenance/update_posts_throttled_task.rb
|
@@ -272,11 +291,12 @@ module Maintenance
|
|
272
291
|
end
|
273
292
|
end
|
274
293
|
```
|
294
|
+
|
275
295
|
### Custom Task Parameters
|
276
296
|
|
277
297
|
Tasks may need additional information, supplied via parameters, to run.
|
278
298
|
Parameters can be defined as Active Model Attributes in a Task, and then become
|
279
|
-
accessible to any of Task
|
299
|
+
accessible to any of Task’s methods: `#collection`, `#count`, or `#process`.
|
280
300
|
|
281
301
|
```ruby
|
282
302
|
# app/tasks/maintenance/update_posts_via_params_task.rb
|
@@ -290,10 +310,6 @@ module Maintenance
|
|
290
310
|
Post.all
|
291
311
|
end
|
292
312
|
|
293
|
-
def count
|
294
|
-
collection.count
|
295
|
-
end
|
296
|
-
|
297
313
|
def process(post)
|
298
314
|
post.update!(content: updated_content)
|
299
315
|
end
|
@@ -304,7 +320,7 @@ end
|
|
304
320
|
Tasks can leverage Active Model Validations when defining parameters. Arguments
|
305
321
|
supplied to a Task accepting parameters will be validated before the Task starts
|
306
322
|
to run. Since arguments are specified in the user interface via text area
|
307
|
-
inputs, it
|
323
|
+
inputs, it’s important to check that they conform to the format your Task
|
308
324
|
expects, and to sanitize any inputs if necessary.
|
309
325
|
|
310
326
|
### Using Task Callbacks
|
@@ -337,7 +353,7 @@ end
|
|
337
353
|
Note: The `after_error` callback is guaranteed to complete,
|
338
354
|
so any exceptions raised in your callback code are ignored.
|
339
355
|
If your `after_error` callback code can raise an exception,
|
340
|
-
you
|
356
|
+
you’ll need to rescue it and handle it appropriately
|
341
357
|
within the callback.
|
342
358
|
|
343
359
|
```ruby
|
@@ -386,7 +402,7 @@ depend on the queue adapter but in general, you should follow these rules:
|
|
386
402
|
safely interrupted and resumed.
|
387
403
|
* Idempotency of `Task#process`: it should be safe to run `process` multiple
|
388
404
|
times for the same element of the collection. Read more in [this Sidekiq best
|
389
|
-
practice][sidekiq-idempotent]. It
|
405
|
+
practice][sidekiq-idempotent]. It’s important if the Task errors and you run
|
390
406
|
it again, because the same element that errored the Task may well be processed
|
391
407
|
again. It especially matters in the situation described above, when the
|
392
408
|
iteration duration exceeds the timeout: if the job is re-enqueued, multiple
|
@@ -394,10 +410,24 @@ depend on the queue adapter but in general, you should follow these rules:
|
|
394
410
|
|
395
411
|
[sidekiq-idempotent]: https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional
|
396
412
|
|
413
|
+
#### Task object life cycle and memoization
|
414
|
+
|
415
|
+
When the Task runs or resumes, the Runner enqueues a job, which processes the
|
416
|
+
Task. That job will instantiate a Task object which will live for the duration
|
417
|
+
of the job. The first time the job runs, it will call `count`. Every time a job
|
418
|
+
runs, it will call `collection` on the Task object, and then `process`
|
419
|
+
for each item in the collection, until the job stops. The job stops when either the
|
420
|
+
collection is finished processing or after the maximum job runtime has expired.
|
421
|
+
|
422
|
+
This means memoization can be misleading within `process`, since the memoized
|
423
|
+
values will be available for subsequent calls to `process` within the same job.
|
424
|
+
Still, memoization can be used for throttling or reporting, and you can use [Task
|
425
|
+
callbacks](#using-task-callbacks) to persist or log a report for example.
|
426
|
+
|
397
427
|
### Writing tests for a Task
|
398
428
|
|
399
429
|
The task generator will also create a test file for your task in the folder
|
400
|
-
`test/tasks/maintenance/`. At a minimum, it
|
430
|
+
`test/tasks/maintenance/`. At a minimum, it’s recommended that the `#process`
|
401
431
|
method in your task be tested. You may also want to test the `#collection` and
|
402
432
|
`#count` methods for your task if they are sufficiently complex.
|
403
433
|
|
@@ -452,7 +482,7 @@ end
|
|
452
482
|
|
453
483
|
### Writing tests for a Task with parameters
|
454
484
|
|
455
|
-
Tests for tasks with parameters need to
|
485
|
+
Tests for tasks with parameters need to instantiate the task class in order to
|
456
486
|
assign attributes. Once the task instance is setup, you may test `#process`
|
457
487
|
normally.
|
458
488
|
|
@@ -479,21 +509,32 @@ end
|
|
479
509
|
|
480
510
|
### Running a Task
|
481
511
|
|
512
|
+
#### Running a Task from the Web UI
|
513
|
+
|
482
514
|
You can run your new Task by accessing the Web UI and clicking on "Run".
|
483
515
|
|
516
|
+
#### Running a Task from the command line
|
517
|
+
|
484
518
|
Alternatively, you can run your Task in the command line:
|
485
519
|
|
486
520
|
```sh-session
|
487
521
|
bundle exec maintenance_tasks perform Maintenance::UpdatePostsTask
|
488
522
|
```
|
489
523
|
|
490
|
-
To run a Task that processes CSVs from the command line, use the
|
524
|
+
To run a Task that processes CSVs from the command line, use the `--csv` option:
|
491
525
|
|
492
526
|
```sh-session
|
493
527
|
bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv "path/to/my_csv.csv"
|
494
528
|
```
|
495
529
|
|
496
|
-
|
530
|
+
The `--csv` option also works with CSV content coming from the standard input:
|
531
|
+
|
532
|
+
```sh-session
|
533
|
+
curl "some/remote/csv" |
|
534
|
+
bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv
|
535
|
+
```
|
536
|
+
|
537
|
+
To run a Task that takes arguments from the command line, use the `--arguments`
|
497
538
|
option, passing arguments as a set of \<key>:\<value> pairs:
|
498
539
|
|
499
540
|
```sh-session
|
@@ -501,6 +542,8 @@ bundle exec maintenance_tasks perform Maintenance::ParamsTask \
|
|
501
542
|
--arguments post_ids:1,2,3 content:"Hello, World!"
|
502
543
|
```
|
503
544
|
|
545
|
+
#### Running a Task from Ruby
|
546
|
+
|
504
547
|
You can also run a Task in Ruby by sending `run` with a Task name to Runner:
|
505
548
|
|
506
549
|
```ruby
|
@@ -527,7 +570,7 @@ MaintenanceTasks::Runner.run(
|
|
527
570
|
)
|
528
571
|
```
|
529
572
|
|
530
|
-
### Monitoring your Task
|
573
|
+
### Monitoring your Task’s status
|
531
574
|
|
532
575
|
The web UI will provide updates on the status of your Task. Here are the states
|
533
576
|
a Task can be in:
|
@@ -566,7 +609,7 @@ By default, a running Task will be interrupted after running for more 5 minutes.
|
|
566
609
|
This is [configured in the `job-iteration` gem][max-job-runtime] and can be
|
567
610
|
tweaked in an initializer if necessary.
|
568
611
|
|
569
|
-
[max-job-runtime]: https://github.com/Shopify/job-iteration/blob
|
612
|
+
[max-job-runtime]: https://github.com/Shopify/job-iteration/blob/-/guides/best-practices.md#max-job-runtime
|
570
613
|
|
571
614
|
Running tasks will also be interrupted and re-enqueued when needed. For example
|
572
615
|
[when Sidekiq workers shuts down for a deploy][sidekiq-deploy]:
|
@@ -581,13 +624,13 @@ Running tasks will also be interrupted and re-enqueued when needed. For example
|
|
581
624
|
|
582
625
|
When Sidekiq is stopping, it will give workers 25 seconds to finish before
|
583
626
|
forcefully terminating them (this is the default but can be configured with the
|
584
|
-
`--timeout` option).
|
585
|
-
to re-enqueue the job so your Task will be resumed. However, the position in
|
586
|
-
collection won
|
627
|
+
`--timeout` option). Before the worker threads are terminated, Sidekiq will try
|
628
|
+
to re-enqueue the job so your Task will be resumed. However, the position in
|
629
|
+
the collection won’t be persisted so at least one iteration may run again.
|
587
630
|
|
588
631
|
#### Help! My Task is stuck
|
589
632
|
|
590
|
-
Finally, if the queue adapter configured for your application doesn
|
633
|
+
Finally, if the queue adapter configured for your application doesn’t have this
|
591
634
|
property, or if Sidekiq crashes, is forcefully terminated, or is unable to
|
592
635
|
re-enqueue the jobs that were in progress, the Task may be in a seemingly stuck
|
593
636
|
situation where it appears to be running but is not. In that situation, pausing
|
@@ -662,7 +705,7 @@ maintenance tasks in your application.
|
|
662
705
|
```ruby
|
663
706
|
# config/initializers/maintenance_tasks.rb
|
664
707
|
|
665
|
-
MaintenanceTasks.job =
|
708
|
+
MaintenanceTasks.job = "CustomTaskJob"
|
666
709
|
|
667
710
|
# app/jobs/custom_task_job.rb
|
668
711
|
|
@@ -693,9 +736,9 @@ If no value is specified, it will default to 1 second.
|
|
693
736
|
#### Customizing which Active Storage service to use
|
694
737
|
|
695
738
|
The Active Storage framework in Rails 6.1 and up supports multiple storage
|
696
|
-
services
|
697
|
-
`MaintenanceTasks.active_storage_service` can be configured with the service
|
698
|
-
key, as specified in your application
|
739
|
+
services. To specify which service to use,
|
740
|
+
`MaintenanceTasks.active_storage_service` can be configured with the service’s
|
741
|
+
key, as specified in your application’s `config/storage.yml`:
|
699
742
|
|
700
743
|
```yaml
|
701
744
|
# config/storage.yml
|
@@ -720,7 +763,8 @@ MaintenanceTasks.active_storage_service = :internal
|
|
720
763
|
```
|
721
764
|
|
722
765
|
There is no need to configure this option if your application uses only one
|
723
|
-
storage service
|
766
|
+
storage service. `Rails.configuration.active_storage.service` is used by
|
767
|
+
default.
|
724
768
|
|
725
769
|
#### Customizing the backtrace cleaner
|
726
770
|
|
@@ -751,13 +795,13 @@ bin/rails generate maintenance_tasks:install
|
|
751
795
|
|
752
796
|
This ensures that new migrations are installed and run as well.
|
753
797
|
|
754
|
-
**What if I
|
798
|
+
**What if I’ve deleted my previous Maintenance Task migrations?**
|
755
799
|
|
756
800
|
The install command will attempt to reinstall these old migrations and migrating
|
757
801
|
the database will cause problems. Use `bin/rails
|
758
|
-
maintenance_tasks:install:migrations` to copy the gem
|
802
|
+
maintenance_tasks:install:migrations` to copy the gem’s migrations to your
|
759
803
|
`db/migrate` folder. Check the release notes to see if any new migrations were
|
760
|
-
added since your last gem upgrade.
|
804
|
+
added since your last gem upgrade. Ensure that these are kept, but remove any
|
761
805
|
migrations that already ran.
|
762
806
|
|
763
807
|
Run the migrations using `bin/rails db:migrate`.
|
@@ -783,8 +827,8 @@ Once a release is ready, follow these steps:
|
|
783
827
|
* Deploy via [Shipit][shipit] and see the new version on
|
784
828
|
<https://rubygems.org/gems/maintenance_tasks>.
|
785
829
|
* Ensure the release has documented all changes and publish it.
|
786
|
-
* Create a new [draft release on GitHub][release] with the title
|
787
|
-
Release
|
830
|
+
* Create a new [draft release on GitHub][release] with the title “Upcoming
|
831
|
+
Release”. The tag version can be left blank. This will be the starting point
|
788
832
|
for documenting changes related to the next release.
|
789
833
|
|
790
834
|
[release]: https://help.github.com/articles/creating-releases/
|
@@ -21,8 +21,7 @@ module MaintenanceTasks
|
|
21
21
|
end
|
22
22
|
|
23
23
|
before_action do
|
24
|
-
request.content_security_policy_nonce_generator ||=
|
25
|
-
->(_request) { SecureRandom.base64(16) }
|
24
|
+
request.content_security_policy_nonce_generator ||= ->(_request) { SecureRandom.base64(16) }
|
26
25
|
request.content_security_policy_nonce_directives = ["style-src"]
|
27
26
|
end
|
28
27
|
|
@@ -6,7 +6,25 @@ module MaintenanceTasks
|
|
6
6
|
#
|
7
7
|
# @api private
|
8
8
|
class RunsController < ApplicationController
|
9
|
-
before_action :set_run
|
9
|
+
before_action :set_run, except: :create
|
10
|
+
|
11
|
+
# Creates a Run for a given Task and redirects to the Task page.
|
12
|
+
def create(&block)
|
13
|
+
task = Runner.run(
|
14
|
+
name: params.fetch(:task_id),
|
15
|
+
csv_file: params[:csv_file],
|
16
|
+
arguments: params.fetch(:task_arguments, {}).permit!.to_h,
|
17
|
+
&block
|
18
|
+
)
|
19
|
+
redirect_to(task_path(task))
|
20
|
+
rescue ActiveRecord::RecordInvalid => error
|
21
|
+
redirect_to(task_path(error.record.task_name), alert: error.message)
|
22
|
+
rescue ActiveRecord::ValueTooLong => error
|
23
|
+
task_name = params.fetch(:id)
|
24
|
+
redirect_to(task_path(task_name), alert: error.message)
|
25
|
+
rescue Runner::EnqueuingError => error
|
26
|
+
redirect_to(task_path(error.run.task_name), alert: error.message)
|
27
|
+
end
|
10
28
|
|
11
29
|
# Updates a Run status to paused.
|
12
30
|
def pause
|
@@ -24,6 +42,16 @@ module MaintenanceTasks
|
|
24
42
|
redirect_to(task_path(@run.task_name), alert: error.message)
|
25
43
|
end
|
26
44
|
|
45
|
+
# Resumes a previously paused Run.
|
46
|
+
def resume
|
47
|
+
Runner.resume(@run)
|
48
|
+
redirect_to(task_path(@run.task_name))
|
49
|
+
rescue ActiveRecord::RecordInvalid => error
|
50
|
+
redirect_to(task_path(@run.task_name), alert: error.message)
|
51
|
+
rescue Runner::EnqueuingError => error
|
52
|
+
redirect_to(task_path(@run.task_name), alert: error.message)
|
53
|
+
end
|
54
|
+
|
27
55
|
private
|
28
56
|
|
29
57
|
def set_run
|
@@ -12,33 +12,20 @@ module MaintenanceTasks
|
|
12
12
|
# Renders the maintenance_tasks/tasks page, displaying
|
13
13
|
# available tasks to users, grouped by category.
|
14
14
|
def index
|
15
|
-
@available_tasks =
|
15
|
+
@available_tasks = TaskDataIndex.available_tasks.group_by(&:category)
|
16
16
|
end
|
17
17
|
|
18
18
|
# Renders the page responsible for providing Task actions to users.
|
19
19
|
# Shows running and completed instances of the Task.
|
20
20
|
def show
|
21
|
-
@task = TaskData.find(params.fetch(:id))
|
22
|
-
set_refresh if @task.last_run&.active?
|
23
|
-
@runs_page = RunsPage.new(@task.previous_runs, params[:cursor])
|
24
|
-
end
|
25
|
-
|
26
|
-
# Runs a given Task and redirects to the Task page.
|
27
|
-
def run(&block)
|
28
|
-
task = Runner.run(
|
29
|
-
name: params.fetch(:id),
|
30
|
-
csv_file: params[:csv_file],
|
31
|
-
arguments: params.fetch(:task_arguments, {}).permit!.to_h,
|
32
|
-
&block
|
33
|
-
)
|
34
|
-
redirect_to(task_path(task))
|
35
|
-
rescue ActiveRecord::RecordInvalid => error
|
36
|
-
redirect_to(task_path(error.record.task_name), alert: error.message)
|
37
|
-
rescue ActiveRecord::ValueTooLong => error
|
38
21
|
task_name = params.fetch(:id)
|
39
|
-
|
40
|
-
|
41
|
-
|
22
|
+
@task = TaskDataShow.new(task_name)
|
23
|
+
@task.active_runs.load
|
24
|
+
set_refresh if @task.active_runs.any?
|
25
|
+
@runs_page = RunsPage.new(@task.completed_runs, params[:cursor])
|
26
|
+
if @task.active_runs.none? && @runs_page.records.none?
|
27
|
+
Task.named(task_name)
|
28
|
+
end
|
42
29
|
end
|
43
30
|
|
44
31
|
private
|
@@ -47,7 +47,7 @@ module MaintenanceTasks
|
|
47
47
|
progress_bar = tag.progress(
|
48
48
|
value: progress.value,
|
49
49
|
max: progress.max,
|
50
|
-
class: ["progress"] + STATUS_COLOURS.fetch(run.status)
|
50
|
+
class: ["progress"] + STATUS_COLOURS.fetch(run.status),
|
51
51
|
)
|
52
52
|
progress_text = tag.p(tag.i(progress.text))
|
53
53
|
tag.div(progress_bar + progress_text, class: "block")
|
@@ -97,16 +97,17 @@ module MaintenanceTasks
|
|
97
97
|
def csv_file_download_path(run)
|
98
98
|
Rails.application.routes.url_helpers.rails_blob_path(
|
99
99
|
run.csv_file,
|
100
|
-
only_path: true
|
100
|
+
only_path: true,
|
101
101
|
)
|
102
102
|
end
|
103
103
|
|
104
104
|
# Return the appropriate field tag for the parameter
|
105
105
|
def parameter_field(form_builder, parameter_name)
|
106
106
|
case form_builder.object.class.attribute_types[parameter_name]
|
107
|
-
when ActiveModel::Type::Integer
|
108
|
-
ActiveModel::Type::Float
|
107
|
+
when ActiveModel::Type::Integer
|
109
108
|
form_builder.number_field(parameter_name)
|
109
|
+
when ActiveModel::Type::Decimal, ActiveModel::Type::Float
|
110
|
+
form_builder.number_field(parameter_name, { step: "any" })
|
110
111
|
when ActiveModel::Type::DateTime
|
111
112
|
form_builder.datetime_field(parameter_name)
|
112
113
|
when ActiveModel::Type::Date
|
@@ -35,7 +35,7 @@ module MaintenanceTasks
|
|
35
35
|
collection = @task.collection
|
36
36
|
@enumerator = nil
|
37
37
|
|
38
|
-
collection_enum = case collection
|
38
|
+
@collection_enum = case collection
|
39
39
|
when :no_collection
|
40
40
|
enumerator_builder.build_once_enumerator(cursor: nil)
|
41
41
|
when ActiveRecord::Relation
|
@@ -50,7 +50,7 @@ module MaintenanceTasks
|
|
50
50
|
|
51
51
|
# For now, only support automatic count based on the enumerator for
|
52
52
|
# batches
|
53
|
-
|
53
|
+
enumerator_builder.active_record_on_batch_relations(
|
54
54
|
collection.relation,
|
55
55
|
cursor: cursor,
|
56
56
|
batch_size: collection.batch_size,
|
@@ -71,7 +71,7 @@ module MaintenanceTasks
|
|
71
71
|
Array, or CSV.
|
72
72
|
MSG
|
73
73
|
end
|
74
|
-
throttle_enumerator(collection_enum)
|
74
|
+
throttle_enumerator(@collection_enum)
|
75
75
|
end
|
76
76
|
|
77
77
|
def throttle_enumerator(collection_enum)
|
@@ -79,7 +79,7 @@ module MaintenanceTasks
|
|
79
79
|
enumerator_builder.build_throttle_enumerator(
|
80
80
|
enum,
|
81
81
|
throttle_on: condition[:throttle_on],
|
82
|
-
backoff: condition[:backoff].call
|
82
|
+
backoff: condition[:backoff].call,
|
83
83
|
)
|
84
84
|
end
|
85
85
|
end
|
@@ -123,7 +123,7 @@ module MaintenanceTasks
|
|
123
123
|
|
124
124
|
def on_start
|
125
125
|
count = @task.count
|
126
|
-
count = @
|
126
|
+
count = @collection_enum.size if count == :no_count
|
127
127
|
@run.start(count)
|
128
128
|
end
|
129
129
|
|
@@ -37,18 +37,25 @@ module MaintenanceTasks
|
|
37
37
|
|
38
38
|
enum status: STATUSES.to_h { |status| [status, status.to_s] }
|
39
39
|
|
40
|
-
validates :task_name, on: :create, inclusion: {
|
41
|
-
Task.available_tasks.map(&:to_s)
|
42
|
-
}
|
40
|
+
validates :task_name, on: :create, inclusion: {
|
41
|
+
in: ->(_) { Task.available_tasks.map(&:to_s) },
|
42
|
+
}
|
43
43
|
validate :csv_attachment_presence, on: :create
|
44
|
+
validate :csv_content_type, on: :create
|
44
45
|
validate :validate_task_arguments, on: :create
|
45
46
|
|
46
47
|
attr_readonly :task_name
|
47
48
|
|
48
|
-
|
49
|
-
|
49
|
+
if Rails.gem_version >= Gem::Version.new("7.1.alpha")
|
50
|
+
serialize :backtrace, coder: YAML
|
51
|
+
serialize :arguments, coder: JSON
|
52
|
+
else
|
53
|
+
serialize :backtrace
|
54
|
+
serialize :arguments, JSON
|
55
|
+
end
|
50
56
|
|
51
57
|
scope :active, -> { where(status: ACTIVE_STATUSES) }
|
58
|
+
scope :completed, -> { where(status: COMPLETED_STATUSES) }
|
52
59
|
|
53
60
|
# Ensure ActiveStorage is in use before preloading the attachments
|
54
61
|
scope :with_attached_csv, -> do
|
@@ -117,7 +124,7 @@ module MaintenanceTasks
|
|
117
124
|
id,
|
118
125
|
tick_count: number_of_ticks,
|
119
126
|
time_running: duration,
|
120
|
-
touch: true
|
127
|
+
touch: true,
|
121
128
|
)
|
122
129
|
if locking_enabled?
|
123
130
|
locking_column = self.class.locking_column
|
@@ -346,6 +353,18 @@ module MaintenanceTasks
|
|
346
353
|
nil
|
347
354
|
end
|
348
355
|
|
356
|
+
# Performs validation on the content type of the :csv_file attachment.
|
357
|
+
# A Run for a Task that uses CsvCollection must have a present :csv_file
|
358
|
+
# and a content type of "text/csv" to be valid. The appropriate error is
|
359
|
+
# added if the Run does not meet the above criteria.
|
360
|
+
def csv_content_type
|
361
|
+
if csv_file.present? && csv_file.content_type != "text/csv"
|
362
|
+
errors.add(:csv_file, "must be a CSV")
|
363
|
+
end
|
364
|
+
rescue Task::NotFoundError
|
365
|
+
nil
|
366
|
+
end
|
367
|
+
|
349
368
|
# Support iterating over ActiveModel::Errors in Rails 6.0 and Rails 6.1+.
|
350
369
|
# To be removed when Rails 6.0 is no longer supported.
|
351
370
|
if Rails::VERSION::STRING.match?(/^6.0/)
|
@@ -358,7 +377,7 @@ module MaintenanceTasks
|
|
358
377
|
.map { |attribute, message| "#{attribute.inspect} #{message}" }
|
359
378
|
errors.add(
|
360
379
|
:arguments,
|
361
|
-
"are invalid: #{error_messages.join("; ")}"
|
380
|
+
"are invalid: #{error_messages.join("; ")}",
|
362
381
|
)
|
363
382
|
end
|
364
383
|
rescue Task::NotFoundError
|
@@ -374,7 +393,7 @@ module MaintenanceTasks
|
|
374
393
|
.map { |error| "#{error.attribute.inspect} #{error.message}" }
|
375
394
|
errors.add(
|
376
395
|
:arguments,
|
377
|
-
"are invalid: #{error_messages.join("; ")}"
|
396
|
+
"are invalid: #{error_messages.join("; ")}",
|
378
397
|
)
|
379
398
|
end
|
380
399
|
rescue Task::NotFoundError
|