maintenance_tasks 2.5.1 → 2.7.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +200 -37
- data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb +39 -0
- data/app/models/maintenance_tasks/batch_csv_collection_builder.rb +4 -3
- data/app/models/maintenance_tasks/csv_collection_builder.rb +11 -8
- data/app/models/maintenance_tasks/progress.rb +5 -5
- data/app/models/maintenance_tasks/run.rb +31 -1
- data/app/models/maintenance_tasks/runner.rb +1 -1
- data/app/models/maintenance_tasks/task.rb +23 -46
- data/lib/generators/maintenance_tasks/task_generator.rb +2 -2
- data/lib/maintenance_tasks/cli.rb +2 -2
- data/lib/tasks/maintenance_tasks_tasks.rake +1 -0
- metadata +4 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 2869ba1c05cb80edb54803f50037bb45b388a5489bec724a79617d2890e78fdc
|
|
4
|
+
data.tar.gz: 5a936d433f102642e972f4c04e7fd96817193f0075b252e942c1ff86bb6a0e06
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 5e054a909fd05bacc7482caa7cba9cc69ed15850bdec1e0bbf09466426ad6bcfadd12c086a4efe185eb5be1edc40b3fbacaca34b555fac39fa0dcfac2df50405
|
|
7
|
+
data.tar.gz: fa1c8baed4e9cc6b34acd03e9c16ef2266da288293da212fb96c03d6a6be2d87fb5599e167e4d629e80f16a80127b611ffe6b8895867958a92675dca06457e48
|
data/README.md
CHANGED
|
@@ -45,10 +45,12 @@ If your task updates your database schema instead of data, use a migration
|
|
|
45
45
|
instead of a maintenance task.
|
|
46
46
|
|
|
47
47
|
If your task happens regularly, consider Active Jobs with a scheduler or cron,
|
|
48
|
-
[job-iteration jobs]
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
48
|
+
[job-iteration jobs][job-iteration] and/or [custom rails_admin
|
|
49
|
+
UIs][rails-admin-engines] instead of the Maintenance Tasks gem. Maintenance
|
|
50
|
+
tasks should be ephemeral, to suit their intentionally limited UI. They should
|
|
51
|
+
not repeat.
|
|
52
|
+
|
|
53
|
+
[job-iteration]: https://github.com/shopify/job-iteration
|
|
52
54
|
|
|
53
55
|
To create seed data for a new application, use the provided Rails `db/seeds.rb`
|
|
54
56
|
file instead.
|
|
@@ -91,10 +93,12 @@ take a look at the [Active Job documentation][active-job-docs].
|
|
|
91
93
|
### Autoloading
|
|
92
94
|
|
|
93
95
|
The Maintenance Tasks framework does not support autoloading in `:classic` mode.
|
|
94
|
-
Please ensure your application is using
|
|
95
|
-
[Zeitwerk](https://github.com/fxn/zeitwerk) to load your code. For more
|
|
96
|
+
Please ensure your application is using [Zeitwerk][] to load your code. For more
|
|
96
97
|
information, please consult the [Rails guides on autoloading and reloading
|
|
97
|
-
constants]
|
|
98
|
+
constants][autoloading].
|
|
99
|
+
|
|
100
|
+
[Zeitwerk]: https://github.com/fxn/zeitwerk
|
|
101
|
+
[autoloading]: https://guides.rubyonrails.org/autoloading_and_reloading_constants.html
|
|
98
102
|
|
|
99
103
|
## Usage
|
|
100
104
|
|
|
@@ -187,15 +191,57 @@ module Maintenance
|
|
|
187
191
|
end
|
|
188
192
|
```
|
|
189
193
|
|
|
194
|
+
`posts.csv`:
|
|
190
195
|
```csv
|
|
191
|
-
# posts.csv
|
|
192
196
|
title,content
|
|
193
197
|
My Title,Hello World!
|
|
194
198
|
```
|
|
195
199
|
|
|
196
200
|
The files uploaded to your Active Storage service provider will be renamed to
|
|
197
|
-
include an ISO 8601 timestamp and the Task name in snake case format.
|
|
198
|
-
|
|
201
|
+
include an ISO 8601 timestamp and the Task name in snake case format.
|
|
202
|
+
|
|
203
|
+
The implicit `#count` method loads and parses the entire file to determine the
|
|
204
|
+
accurate number of rows. With files with millions of rows, it takes several
|
|
205
|
+
seconds to process. Consider skipping the count (defining a `count` that returns
|
|
206
|
+
`nil`) or use an approximation, eg: count the number of new lines:
|
|
207
|
+
|
|
208
|
+
```ruby
|
|
209
|
+
def count(task)
|
|
210
|
+
task.csv_content.count("\n") - 1
|
|
211
|
+
end
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
#### CSV options
|
|
215
|
+
|
|
216
|
+
Tasks can pass [options for Ruby's CSV parser][csv-parse-options] by adding
|
|
217
|
+
keyword arguments to `csv_collection`:
|
|
218
|
+
|
|
219
|
+
[csv-parse-options]: https://ruby-doc.org/3.3.0/stdlibs/csv/CSV.html#class-CSV-label-Options+for+Parsing
|
|
220
|
+
|
|
221
|
+
```ruby
|
|
222
|
+
# app/tasks/maintenance/import_posts_task.rb
|
|
223
|
+
|
|
224
|
+
module Maintenance
|
|
225
|
+
class ImportPosts
|
|
226
|
+
csv_collection(skip_lines: /^#/, converters: ->(field) { field.strip })
|
|
227
|
+
|
|
228
|
+
def process(row)
|
|
229
|
+
Post.create!(title: row["title"], content: row["content"])
|
|
230
|
+
end
|
|
231
|
+
end
|
|
232
|
+
end
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
These options instruct Ruby's CSV parser to skip lines that start with a `#`,
|
|
236
|
+
and removes the leading and trailing spaces from any field, so that the
|
|
237
|
+
following file will be processed identically as the previous example:
|
|
238
|
+
|
|
239
|
+
`posts.csv`:
|
|
240
|
+
```csv
|
|
241
|
+
# A comment
|
|
242
|
+
title,content
|
|
243
|
+
My Title ,Hello World!
|
|
244
|
+
```
|
|
199
245
|
|
|
200
246
|
#### Batch CSV Tasks
|
|
201
247
|
|
|
@@ -305,11 +351,11 @@ If you have a special use case requiring iteration over an unsupported
|
|
|
305
351
|
collection type, such as external resources fetched from some API, you can
|
|
306
352
|
implement the `enumerator_builder(cursor:)` method in your task.
|
|
307
353
|
|
|
308
|
-
This method should return an `Enumerator`, yielding pairs of
|
|
309
|
-
|
|
310
|
-
|
|
311
|
-
|
|
312
|
-
|
|
354
|
+
This method should return an `Enumerator`, yielding pairs of `[item, cursor]`.
|
|
355
|
+
Maintenance Tasks takes care of persisting the current cursor position and will
|
|
356
|
+
provide it as the `cursor` argument if your task is interrupted or resumed. The
|
|
357
|
+
`cursor` is stored as a `String`, so your custom enumerator should handle
|
|
358
|
+
serializing/deserializing the value if required.
|
|
313
359
|
|
|
314
360
|
```ruby
|
|
315
361
|
# app/tasks/maintenance/custom_enumerator_task.rb
|
|
@@ -377,7 +423,7 @@ module Maintenance
|
|
|
377
423
|
throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
|
|
378
424
|
DatabaseStatus.unhealthy?
|
|
379
425
|
end
|
|
380
|
-
...
|
|
426
|
+
# ...
|
|
381
427
|
end
|
|
382
428
|
end
|
|
383
429
|
```
|
|
@@ -413,6 +459,100 @@ to run. Since arguments are specified in the user interface via text area
|
|
|
413
459
|
inputs, it’s important to check that they conform to the format your Task
|
|
414
460
|
expects, and to sanitize any inputs if necessary.
|
|
415
461
|
|
|
462
|
+
### Custom cursor columns to improve performance
|
|
463
|
+
|
|
464
|
+
The [job-iteration gem][job-iteration], on which this gem depends, adds an
|
|
465
|
+
`order by` clause to the relation returned by the `collection` method, in order
|
|
466
|
+
to iterate through records. It defaults to order on the `id` column.
|
|
467
|
+
|
|
468
|
+
The [job-iteration gem][job-iteration] supports configuring which columns are
|
|
469
|
+
used to order the cursor, as documented in
|
|
470
|
+
[`build_active_record_enumerator_on_records`][ji-ar-enumerator-doc].
|
|
471
|
+
|
|
472
|
+
[ji-ar-enumerator-doc]: https://www.rubydoc.info/gems/job-iteration/JobIteration/EnumeratorBuilder#build_active_record_enumerator_on_records-instance_method
|
|
473
|
+
|
|
474
|
+
The `maintenance-tasks` gem exposes the ability that `job-iteration` provides to
|
|
475
|
+
control the cursor columns, through the `cursor_columns` method in the
|
|
476
|
+
`MaintenanceTasks::Task` class. If the `cursor_columns` method returns `nil`,
|
|
477
|
+
the query is ordered by the primary key. If cursor columns values change during
|
|
478
|
+
an iteration, records may be skipped or yielded multiple times.
|
|
479
|
+
|
|
480
|
+
```ruby
|
|
481
|
+
module Maintenance
|
|
482
|
+
class UpdatePostsTask < MaintenanceTasks::Task
|
|
483
|
+
def cursor_columns
|
|
484
|
+
[:created_at, :id]
|
|
485
|
+
end
|
|
486
|
+
|
|
487
|
+
def collection
|
|
488
|
+
Post.where(created_at: 2.days.ago...1.hour.ago)
|
|
489
|
+
end
|
|
490
|
+
|
|
491
|
+
def process(post)
|
|
492
|
+
post.update!(content: "updated content")
|
|
493
|
+
end
|
|
494
|
+
end
|
|
495
|
+
end
|
|
496
|
+
```
|
|
497
|
+
|
|
498
|
+
### Subscribing to instrumentation events
|
|
499
|
+
|
|
500
|
+
If you are interested in actioning a specific task event, please refer to the [Using Task Callbacks](#using-task-callbacks) section below. However, if you want to subscribe to all events, irrespective of the task, you can use the following Active Support notifications:
|
|
501
|
+
|
|
502
|
+
```ruby
|
|
503
|
+
enqueued.maintenance_tasks # This event is published when a task has been enqueued by the user.
|
|
504
|
+
succeeded.maintenance_tasks # This event is published when a task has finished without any errors.
|
|
505
|
+
cancelled.maintenance_tasks # This event is published when the user explicitly halts the execution of a task.
|
|
506
|
+
paused.maintenance_tasks # This event is published when a task is paused by the user in the middle of its run.
|
|
507
|
+
errored.maintenance_tasks # This event is published when the task's code produces an unhandled exception.
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
These notifications offer a way to monitor the lifecycle of maintenance tasks in your application.
|
|
511
|
+
|
|
512
|
+
Usage example:
|
|
513
|
+
|
|
514
|
+
```ruby
|
|
515
|
+
ActiveSupport::Notifications.subscribe("succeeded.maintenance_tasks") do |*, payload|
|
|
516
|
+
task_name = payload[:task_name]
|
|
517
|
+
arguments = payload[:arguments]
|
|
518
|
+
metadata = payload[:metadata]
|
|
519
|
+
job_id = payload[:job_id]
|
|
520
|
+
run_id = payload[:run_id]
|
|
521
|
+
time_running = payload[:time_running]
|
|
522
|
+
started_at = payload[:started_at]
|
|
523
|
+
ended_at = payload[:ended_at]
|
|
524
|
+
rescue => e
|
|
525
|
+
Rails.logger.error(e)
|
|
526
|
+
end
|
|
527
|
+
|
|
528
|
+
ActiveSupport::Notifications.subscribe("errored.maintenance_tasks") do |*, payload|
|
|
529
|
+
task_name = payload[:task_name]
|
|
530
|
+
error = payload[:error]
|
|
531
|
+
error_message = error[:message]
|
|
532
|
+
error_class = error[:class]
|
|
533
|
+
error_backtrace = error[:backtrace]
|
|
534
|
+
rescue => e
|
|
535
|
+
Rails.logger.error(e)
|
|
536
|
+
end
|
|
537
|
+
|
|
538
|
+
# or
|
|
539
|
+
|
|
540
|
+
class MaintenanceTasksInstrumenter < ActiveSupport::Subscriber
|
|
541
|
+
attach_to :maintenance_tasks
|
|
542
|
+
|
|
543
|
+
def enqueued(event)
|
|
544
|
+
task_name = event.payload[:task_name]
|
|
545
|
+
arguments = event.payload[:arguments]
|
|
546
|
+
metadata = event.payload[:metadata]
|
|
547
|
+
|
|
548
|
+
SlackNotifier.broadcast(SLACK_CHANNEL,
|
|
549
|
+
"Job #{task_name} was started by #{metadata[:user_email]}} with arguments #{arguments.to_s.truncate(255)}")
|
|
550
|
+
rescue => e
|
|
551
|
+
Rails.logger.error(e)
|
|
552
|
+
end
|
|
553
|
+
end
|
|
554
|
+
```
|
|
555
|
+
|
|
416
556
|
### Using Task Callbacks
|
|
417
557
|
|
|
418
558
|
The Task provides callbacks that hook into its life cycle.
|
|
@@ -463,21 +603,6 @@ end
|
|
|
463
603
|
If any of the other callbacks cause an exception, it will be handled by the
|
|
464
604
|
error handler, and will cause the task to stop running.
|
|
465
605
|
|
|
466
|
-
Callback behaviour can be shared across all tasks using an initializer.
|
|
467
|
-
|
|
468
|
-
```ruby
|
|
469
|
-
# config/initializer/maintenance_tasks.rb
|
|
470
|
-
Rails.autoloaders.main.on_load("MaintenanceTasks::Task") do
|
|
471
|
-
MaintenanceTasks::Task.class_eval do
|
|
472
|
-
after_start(:notify)
|
|
473
|
-
|
|
474
|
-
private
|
|
475
|
-
|
|
476
|
-
def notify; end
|
|
477
|
-
end
|
|
478
|
-
end
|
|
479
|
-
```
|
|
480
|
-
|
|
481
606
|
### Considerations when writing Tasks
|
|
482
607
|
|
|
483
608
|
Maintenance Tasks relies on the queue adapter configured for your application to
|
|
@@ -595,6 +720,42 @@ module Maintenance
|
|
|
595
720
|
end
|
|
596
721
|
```
|
|
597
722
|
|
|
723
|
+
### Writing tests for a Task that uses a custom enumerator
|
|
724
|
+
|
|
725
|
+
Tests for tasks that use custom enumerators need to instantiate the task class
|
|
726
|
+
in order to call `#build_enumerator`. Once the task instance is set up, validate
|
|
727
|
+
that `#build_enumerator` returns an enumerator yielding pairs of [item, cursor]
|
|
728
|
+
as expected.
|
|
729
|
+
|
|
730
|
+
```ruby
|
|
731
|
+
# test/tasks/maintenance/custom_enumerating_task.rb
|
|
732
|
+
|
|
733
|
+
require "test_helper"
|
|
734
|
+
|
|
735
|
+
module Maintenance
|
|
736
|
+
class CustomEnumeratingTaskTest < ActiveSupport::TestCase
|
|
737
|
+
setup do
|
|
738
|
+
@task = CustomEnumeratingTask.new
|
|
739
|
+
end
|
|
740
|
+
|
|
741
|
+
test "#build_enumerator returns enumerator yielding pairs of [item, cursor]" do
|
|
742
|
+
enum = @task.build_enumerator(cursor: 0)
|
|
743
|
+
expected_items = [:b, :c]
|
|
744
|
+
|
|
745
|
+
assert_equal 2, enum.size
|
|
746
|
+
|
|
747
|
+
enum.each_with_index do |item, cursor|
|
|
748
|
+
assert_equal expected_items[cursor], item
|
|
749
|
+
end
|
|
750
|
+
end
|
|
751
|
+
|
|
752
|
+
test "#process performs a task iteration" do
|
|
753
|
+
# ...
|
|
754
|
+
end
|
|
755
|
+
end
|
|
756
|
+
end
|
|
757
|
+
```
|
|
758
|
+
|
|
598
759
|
### Running a Task
|
|
599
760
|
|
|
600
761
|
#### Running a Task from the Web UI
|
|
@@ -917,7 +1078,7 @@ MaintenanceTasks.parent_controller = "Services::CustomController"
|
|
|
917
1078
|
class Services::CustomController < ActionController::Base
|
|
918
1079
|
include CustomSecurityThings
|
|
919
1080
|
include CustomLoggingThings
|
|
920
|
-
...
|
|
1081
|
+
# ...
|
|
921
1082
|
end
|
|
922
1083
|
```
|
|
923
1084
|
|
|
@@ -928,12 +1089,14 @@ If no value is specified, it will default to `"ActionController::Base"`.
|
|
|
928
1089
|
|
|
929
1090
|
#### Configure time after which the task will be considered stuck
|
|
930
1091
|
|
|
931
|
-
To specify a time duration after which a task is considered stuck if it has not
|
|
932
|
-
you can configure `MaintenanceTasks.stuck_task_duration`. This
|
|
933
|
-
job infrastructure events that may prevent the
|
|
1092
|
+
To specify a time duration after which a task is considered stuck if it has not
|
|
1093
|
+
been updated, you can configure `MaintenanceTasks.stuck_task_duration`. This
|
|
1094
|
+
duration should account for job infrastructure events that may prevent the
|
|
1095
|
+
maintenance tasks job from being executed and cancelling the task.
|
|
934
1096
|
|
|
935
|
-
The value for `MaintenanceTasks.stuck_task_duration` must be an
|
|
936
|
-
If no value is specified, it will default to 5
|
|
1097
|
+
The value for `MaintenanceTasks.stuck_task_duration` must be an
|
|
1098
|
+
`ActiveSupport::Duration`. If no value is specified, it will default to 5
|
|
1099
|
+
minutes.
|
|
937
1100
|
|
|
938
1101
|
### Metadata
|
|
939
1102
|
|
|
@@ -33,6 +33,45 @@ module MaintenanceTasks
|
|
|
33
33
|
def build_enumerator(_run, cursor:)
|
|
34
34
|
cursor ||= @run.cursor
|
|
35
35
|
@collection_enum = @task.enumerator_builder(cursor: cursor)
|
|
36
|
+
|
|
37
|
+
@collection_enum ||= case (collection = @task.collection)
|
|
38
|
+
when :no_collection
|
|
39
|
+
enumerator_builder.build_once_enumerator(cursor: nil)
|
|
40
|
+
when ActiveRecord::Relation
|
|
41
|
+
enumerator_builder.active_record_on_records(collection, cursor: cursor, columns: @task.cursor_columns)
|
|
42
|
+
when ActiveRecord::Batches::BatchEnumerator
|
|
43
|
+
if collection.start || collection.finish
|
|
44
|
+
raise ArgumentError, <<~MSG.squish
|
|
45
|
+
#{@task.class.name}#collection cannot support
|
|
46
|
+
a batch enumerator with the "start" or "finish" options.
|
|
47
|
+
MSG
|
|
48
|
+
end
|
|
49
|
+
|
|
50
|
+
# For now, only support automatic count based on the enumerator for
|
|
51
|
+
# batches
|
|
52
|
+
enumerator_builder.active_record_on_batch_relations(
|
|
53
|
+
collection.relation,
|
|
54
|
+
cursor: cursor,
|
|
55
|
+
batch_size: collection.batch_size,
|
|
56
|
+
columns: @task.cursor_columns,
|
|
57
|
+
)
|
|
58
|
+
when Array
|
|
59
|
+
enumerator_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
|
|
60
|
+
when BatchCsvCollectionBuilder::BatchCsv
|
|
61
|
+
JobIteration::CsvEnumerator.new(collection.csv).batches(
|
|
62
|
+
batch_size: collection.batch_size,
|
|
63
|
+
cursor: cursor&.to_i,
|
|
64
|
+
)
|
|
65
|
+
when CSV
|
|
66
|
+
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
|
|
67
|
+
else
|
|
68
|
+
raise ArgumentError, <<~MSG.squish
|
|
69
|
+
#{@task.class.name}#collection must be either an
|
|
70
|
+
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
|
|
71
|
+
Array, or CSV.
|
|
72
|
+
MSG
|
|
73
|
+
end
|
|
74
|
+
|
|
36
75
|
throttle_enumerator(@collection_enum)
|
|
37
76
|
end
|
|
38
77
|
|
|
@@ -12,16 +12,17 @@ module MaintenanceTasks
|
|
|
12
12
|
# Initialize a BatchCsvCollectionBuilder with a batch size.
|
|
13
13
|
#
|
|
14
14
|
# @param batch_size [Integer] the number of CSV rows in a batch.
|
|
15
|
-
|
|
15
|
+
# @param csv_options [Hash] options to pass to the CSV parser.
|
|
16
|
+
def initialize(batch_size, **csv_options)
|
|
16
17
|
@batch_size = batch_size
|
|
17
|
-
super()
|
|
18
|
+
super(**csv_options)
|
|
18
19
|
end
|
|
19
20
|
|
|
20
21
|
# Defines the collection to be iterated over, based on the provided CSV.
|
|
21
22
|
# Includes the CSV and the batch size.
|
|
22
23
|
def collection(task)
|
|
23
24
|
BatchCsv.new(
|
|
24
|
-
csv: CSV.new(task.csv_content,
|
|
25
|
+
csv: CSV.new(task.csv_content, **@csv_options),
|
|
25
26
|
batch_size: @batch_size,
|
|
26
27
|
)
|
|
27
28
|
end
|
|
@@ -5,24 +5,27 @@ require "csv"
|
|
|
5
5
|
module MaintenanceTasks
|
|
6
6
|
# Strategy for building a Task that processes CSV files.
|
|
7
7
|
#
|
|
8
|
+
# @param csv_options [Hash] options to pass to the CSV parser.
|
|
8
9
|
# @api private
|
|
9
10
|
class CsvCollectionBuilder
|
|
11
|
+
def initialize(**csv_options)
|
|
12
|
+
@csv_options = csv_options
|
|
13
|
+
end
|
|
14
|
+
|
|
10
15
|
# Defines the collection to be iterated over, based on the provided CSV.
|
|
11
16
|
#
|
|
12
|
-
# @return [CSV] the CSV object constructed from the specified CSV content
|
|
13
|
-
# with headers.
|
|
17
|
+
# @return [CSV] the CSV object constructed from the specified CSV content.
|
|
14
18
|
def collection(task)
|
|
15
|
-
CSV.new(task.csv_content,
|
|
19
|
+
CSV.new(task.csv_content, **@csv_options)
|
|
16
20
|
end
|
|
17
21
|
|
|
18
|
-
# The number of rows to be processed.
|
|
19
|
-
#
|
|
20
|
-
# Note that
|
|
21
|
-
# newlines.
|
|
22
|
+
# The number of rows to be processed.
|
|
23
|
+
# It uses the CSV library for an accurate row count.
|
|
24
|
+
# Note that the entire file is loaded. It will take several seconds with files with millions of rows.
|
|
22
25
|
#
|
|
23
26
|
# @return [Integer] the approximate number of rows to process.
|
|
24
27
|
def count(task)
|
|
25
|
-
task.csv_content.count
|
|
28
|
+
CSV.new(task.csv_content, **@csv_options).count
|
|
26
29
|
end
|
|
27
30
|
|
|
28
31
|
# Return that the Task processes CSV content.
|
|
@@ -52,17 +52,17 @@ module MaintenanceTasks
|
|
|
52
52
|
total = @run.tick_total
|
|
53
53
|
|
|
54
54
|
if !total?
|
|
55
|
-
"Processed #{number_to_delimited(count)} "\
|
|
55
|
+
"Processed #{number_to_delimited(count)} " \
|
|
56
56
|
"#{"item".pluralize(count)}."
|
|
57
57
|
elsif over_total?
|
|
58
|
-
"Processed #{number_to_delimited(count)} "\
|
|
59
|
-
"#{"item".pluralize(count)} "\
|
|
58
|
+
"Processed #{number_to_delimited(count)} " \
|
|
59
|
+
"#{"item".pluralize(count)} " \
|
|
60
60
|
"(expected #{number_to_delimited(total)})."
|
|
61
61
|
else
|
|
62
62
|
percentage = 100.0 * count / total
|
|
63
63
|
|
|
64
|
-
"Processed #{number_to_delimited(count)} out of "\
|
|
65
|
-
"#{number_to_delimited(total)} #{"item".pluralize(total)} "\
|
|
64
|
+
"Processed #{number_to_delimited(count)} out of " \
|
|
65
|
+
"#{number_to_delimited(total)} #{"item".pluralize(total)} " \
|
|
66
66
|
"(#{number_to_percentage(percentage, precision: 0)})."
|
|
67
67
|
end
|
|
68
68
|
end
|
|
@@ -33,7 +33,13 @@ module MaintenanceTasks
|
|
|
33
33
|
]
|
|
34
34
|
COMPLETED_STATUSES = [:succeeded, :errored, :cancelled]
|
|
35
35
|
|
|
36
|
-
|
|
36
|
+
if Rails.gem_version >= Gem::Version.new("7.0.alpha")
|
|
37
|
+
enum :status, STATUSES.to_h { |status| [status, status.to_s] }
|
|
38
|
+
else
|
|
39
|
+
enum status: STATUSES.to_h { |status| [status, status.to_s] }
|
|
40
|
+
end
|
|
41
|
+
|
|
42
|
+
after_save :instrument_status_change
|
|
37
43
|
|
|
38
44
|
validate :task_name_belongs_to_a_valid_task, on: :create
|
|
39
45
|
validate :csv_attachment_presence, on: :create
|
|
@@ -448,6 +454,30 @@ module MaintenanceTasks
|
|
|
448
454
|
|
|
449
455
|
private
|
|
450
456
|
|
|
457
|
+
def instrument_status_change
|
|
458
|
+
return unless status_previously_changed? || id_previously_changed?
|
|
459
|
+
return if running? || pausing? || cancelling? || interrupted?
|
|
460
|
+
|
|
461
|
+
attr = {
|
|
462
|
+
run_id: id,
|
|
463
|
+
job_id: job_id,
|
|
464
|
+
task_name: task_name,
|
|
465
|
+
arguments: arguments,
|
|
466
|
+
metadata: metadata,
|
|
467
|
+
time_running: time_running,
|
|
468
|
+
started_at: started_at,
|
|
469
|
+
ended_at: ended_at,
|
|
470
|
+
}
|
|
471
|
+
|
|
472
|
+
attr[:error] = {
|
|
473
|
+
message: error_message,
|
|
474
|
+
class: error_class,
|
|
475
|
+
backtrace: backtrace,
|
|
476
|
+
} if errored?
|
|
477
|
+
|
|
478
|
+
ActiveSupport::Notifications.instrument("#{status}.maintenance_tasks", attr)
|
|
479
|
+
end
|
|
480
|
+
|
|
451
481
|
def run_task_callbacks(callback)
|
|
452
482
|
task.run_callbacks(callback)
|
|
453
483
|
rescue Task::NotFoundError
|
|
@@ -74,7 +74,7 @@ module MaintenanceTasks
|
|
|
74
74
|
|
|
75
75
|
def enqueue(run, job)
|
|
76
76
|
unless job.enqueue
|
|
77
|
-
raise "The job to perform #{run.task_name} could not be enqueued. "\
|
|
77
|
+
raise "The job to perform #{run.task_name} could not be enqueued. " \
|
|
78
78
|
"Enqueuing has been prevented by a callback."
|
|
79
79
|
end
|
|
80
80
|
rescue => error
|
|
@@ -65,20 +65,24 @@ module MaintenanceTasks
|
|
|
65
65
|
# Make this Task a task that handles CSV.
|
|
66
66
|
#
|
|
67
67
|
# @param in_batches [Integer] optionally, supply a batch size if the CSV
|
|
68
|
-
#
|
|
68
|
+
# should be processed in batches.
|
|
69
|
+
# @param csv_options [Hash] optionally, supply options for the CSV parser.
|
|
70
|
+
# If not given, defaults to: <code>{ headers: true }</code>
|
|
71
|
+
# @see https://ruby-doc.org/3.3.0/stdlibs/csv/CSV.html#class-CSV-label-Options+for+Parsing
|
|
69
72
|
#
|
|
70
73
|
# An input to upload a CSV will be added in the form to start a Run. The
|
|
71
74
|
# collection and count method are implemented.
|
|
72
|
-
def csv_collection(in_batches: nil)
|
|
75
|
+
def csv_collection(in_batches: nil, **csv_options)
|
|
73
76
|
unless defined?(ActiveStorage)
|
|
74
|
-
raise NotImplementedError, "Active Storage needs to be installed\n"\
|
|
77
|
+
raise NotImplementedError, "Active Storage needs to be installed\n" \
|
|
75
78
|
"To resolve this issue run: bin/rails active_storage:install"
|
|
76
79
|
end
|
|
77
80
|
|
|
81
|
+
csv_options[:headers] = true unless csv_options.key?(:headers)
|
|
78
82
|
self.collection_builder_strategy = if in_batches
|
|
79
|
-
BatchCsvCollectionBuilder.new(in_batches)
|
|
83
|
+
BatchCsvCollectionBuilder.new(in_batches, **csv_options)
|
|
80
84
|
else
|
|
81
|
-
CsvCollectionBuilder.new
|
|
85
|
+
CsvCollectionBuilder.new(**csv_options)
|
|
82
86
|
end
|
|
83
87
|
end
|
|
84
88
|
|
|
@@ -230,6 +234,18 @@ module MaintenanceTasks
|
|
|
230
234
|
self.class.collection_builder_strategy.collection(self)
|
|
231
235
|
end
|
|
232
236
|
|
|
237
|
+
# The columns used to build the `ORDER BY` clause of the query for iteration.
|
|
238
|
+
#
|
|
239
|
+
# If cursor_columns returns nil, the query is ordered by the primary key.
|
|
240
|
+
# If cursor columns values change during an iteration, records may be skipped or yielded multiple times.
|
|
241
|
+
# More details in the documentation of JobIteration::EnumeratorBuilder.build_active_record_enumerator_on_records:
|
|
242
|
+
# https://www.rubydoc.info/gems/job-iteration/JobIteration/EnumeratorBuilder#build_active_record_enumerator_on_records-instance_method
|
|
243
|
+
#
|
|
244
|
+
# @return the cursor_columns.
|
|
245
|
+
def cursor_columns
|
|
246
|
+
nil
|
|
247
|
+
end
|
|
248
|
+
|
|
233
249
|
# Placeholder method to raise in case a subclass fails to implement the
|
|
234
250
|
# expected instance method.
|
|
235
251
|
#
|
|
@@ -248,7 +264,7 @@ module MaintenanceTasks
|
|
|
248
264
|
self.class.collection_builder_strategy.count(self)
|
|
249
265
|
end
|
|
250
266
|
|
|
251
|
-
# Default
|
|
267
|
+
# Default enumerator builder. You may override this method to return any
|
|
252
268
|
# Enumerator yielding pairs of `[item, item_cursor]`.
|
|
253
269
|
#
|
|
254
270
|
# @param cursor [String, nil] cursor position to resume from, or nil on
|
|
@@ -256,46 +272,7 @@ module MaintenanceTasks
|
|
|
256
272
|
#
|
|
257
273
|
# @return [Enumerator]
|
|
258
274
|
def enumerator_builder(cursor:)
|
|
259
|
-
|
|
260
|
-
|
|
261
|
-
job_iteration_builder = JobIteration::EnumeratorBuilder.new(nil)
|
|
262
|
-
|
|
263
|
-
case collection
|
|
264
|
-
when :no_collection
|
|
265
|
-
job_iteration_builder.build_once_enumerator(cursor: nil)
|
|
266
|
-
when ActiveRecord::Relation
|
|
267
|
-
job_iteration_builder.active_record_on_records(collection, cursor: cursor)
|
|
268
|
-
when ActiveRecord::Batches::BatchEnumerator
|
|
269
|
-
if collection.start || collection.finish
|
|
270
|
-
raise ArgumentError, <<~MSG.squish
|
|
271
|
-
#{self.class.name}#collection cannot support
|
|
272
|
-
a batch enumerator with the "start" or "finish" options.
|
|
273
|
-
MSG
|
|
274
|
-
end
|
|
275
|
-
|
|
276
|
-
# For now, only support automatic count based on the enumerator for
|
|
277
|
-
# batches
|
|
278
|
-
job_iteration_builder.active_record_on_batch_relations(
|
|
279
|
-
collection.relation,
|
|
280
|
-
cursor: cursor,
|
|
281
|
-
batch_size: collection.batch_size,
|
|
282
|
-
)
|
|
283
|
-
when Array
|
|
284
|
-
job_iteration_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
|
|
285
|
-
when BatchCsvCollectionBuilder::BatchCsv
|
|
286
|
-
JobIteration::CsvEnumerator.new(collection.csv).batches(
|
|
287
|
-
batch_size: collection.batch_size,
|
|
288
|
-
cursor: cursor&.to_i,
|
|
289
|
-
)
|
|
290
|
-
when CSV
|
|
291
|
-
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
|
|
292
|
-
else
|
|
293
|
-
raise ArgumentError, <<~MSG.squish
|
|
294
|
-
#{self.class.name}#collection must be either an
|
|
295
|
-
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
|
|
296
|
-
Array, or CSV.
|
|
297
|
-
MSG
|
|
298
|
-
end
|
|
275
|
+
nil
|
|
299
276
|
end
|
|
300
277
|
end
|
|
301
278
|
end
|
|
@@ -6,7 +6,7 @@ module MaintenanceTasks
|
|
|
6
6
|
# @api private
|
|
7
7
|
class TaskGenerator < Rails::Generators::NamedBase
|
|
8
8
|
source_root File.expand_path("templates", __dir__)
|
|
9
|
-
desc "This generator creates a task file at app/tasks and a corresponding "\
|
|
9
|
+
desc "This generator creates a task file at app/tasks and a corresponding " \
|
|
10
10
|
"test."
|
|
11
11
|
|
|
12
12
|
class_option :csv,
|
|
@@ -24,7 +24,7 @@ module MaintenanceTasks
|
|
|
24
24
|
# Creates the Task file.
|
|
25
25
|
def create_task_file
|
|
26
26
|
if options[:csv] && options[:no_collection]
|
|
27
|
-
raise "Multiple Task type options provided. Please use either "\
|
|
27
|
+
raise "Multiple Task type options provided. Please use either " \
|
|
28
28
|
"--csv or --no-collection."
|
|
29
29
|
end
|
|
30
30
|
template_file = File.join(
|
|
@@ -23,11 +23,11 @@ module MaintenanceTasks
|
|
|
23
23
|
DESC
|
|
24
24
|
|
|
25
25
|
# Specify the CSV file to process for CSV Tasks
|
|
26
|
-
desc = "Supply a CSV file to be processed by a CSV Task, "\
|
|
26
|
+
desc = "Supply a CSV file to be processed by a CSV Task, " \
|
|
27
27
|
"--csv path/to/csv/file.csv"
|
|
28
28
|
option :csv, lazy_default: :stdin, desc: desc
|
|
29
29
|
# Specify arguments to supply to a Task supporting parameters
|
|
30
|
-
desc = "Supply arguments for a Task that accepts parameters as a set of "\
|
|
30
|
+
desc = "Supply arguments for a Task that accepts parameters as a set of " \
|
|
31
31
|
"<key>:<value> pairs."
|
|
32
32
|
option :arguments, type: :hash, desc: desc
|
|
33
33
|
|
metadata
CHANGED
|
@@ -1,14 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: maintenance_tasks
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 2.
|
|
4
|
+
version: 2.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Shopify Engineering
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: exe
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date: 2024-
|
|
11
|
+
date: 2024-04-16 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: actionpack
|
|
@@ -171,7 +171,7 @@ homepage: https://github.com/Shopify/maintenance_tasks
|
|
|
171
171
|
licenses:
|
|
172
172
|
- MIT
|
|
173
173
|
metadata:
|
|
174
|
-
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.
|
|
174
|
+
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.7.0
|
|
175
175
|
allowed_push_host: https://rubygems.org
|
|
176
176
|
post_install_message:
|
|
177
177
|
rdoc_options: []
|
|
@@ -188,7 +188,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
188
188
|
- !ruby/object:Gem::Version
|
|
189
189
|
version: '0'
|
|
190
190
|
requirements: []
|
|
191
|
-
rubygems_version: 3.5.
|
|
191
|
+
rubygems_version: 3.5.9
|
|
192
192
|
signing_key:
|
|
193
193
|
specification_version: 4
|
|
194
194
|
summary: A Rails engine for queuing and managing maintenance tasks
|