maintenance_tasks 2.5.1 → 2.7.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +200 -37
- data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb +39 -0
- data/app/models/maintenance_tasks/batch_csv_collection_builder.rb +4 -3
- data/app/models/maintenance_tasks/csv_collection_builder.rb +11 -8
- data/app/models/maintenance_tasks/progress.rb +5 -5
- data/app/models/maintenance_tasks/run.rb +31 -1
- data/app/models/maintenance_tasks/runner.rb +1 -1
- data/app/models/maintenance_tasks/task.rb +23 -46
- data/lib/generators/maintenance_tasks/task_generator.rb +2 -2
- data/lib/maintenance_tasks/cli.rb +2 -2
- data/lib/tasks/maintenance_tasks_tasks.rake +1 -0
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2869ba1c05cb80edb54803f50037bb45b388a5489bec724a79617d2890e78fdc
|
4
|
+
data.tar.gz: 5a936d433f102642e972f4c04e7fd96817193f0075b252e942c1ff86bb6a0e06
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 5e054a909fd05bacc7482caa7cba9cc69ed15850bdec1e0bbf09466426ad6bcfadd12c086a4efe185eb5be1edc40b3fbacaca34b555fac39fa0dcfac2df50405
|
7
|
+
data.tar.gz: fa1c8baed4e9cc6b34acd03e9c16ef2266da288293da212fb96c03d6a6be2d87fb5599e167e4d629e80f16a80127b611ffe6b8895867958a92675dca06457e48
|
data/README.md
CHANGED
@@ -45,10 +45,12 @@ If your task updates your database schema instead of data, use a migration
|
|
45
45
|
instead of a maintenance task.
|
46
46
|
|
47
47
|
If your task happens regularly, consider Active Jobs with a scheduler or cron,
|
48
|
-
[job-iteration jobs]
|
49
|
-
|
50
|
-
|
51
|
-
|
48
|
+
[job-iteration jobs][job-iteration] and/or [custom rails_admin
|
49
|
+
UIs][rails-admin-engines] instead of the Maintenance Tasks gem. Maintenance
|
50
|
+
tasks should be ephemeral, to suit their intentionally limited UI. They should
|
51
|
+
not repeat.
|
52
|
+
|
53
|
+
[job-iteration]: https://github.com/shopify/job-iteration
|
52
54
|
|
53
55
|
To create seed data for a new application, use the provided Rails `db/seeds.rb`
|
54
56
|
file instead.
|
@@ -91,10 +93,12 @@ take a look at the [Active Job documentation][active-job-docs].
|
|
91
93
|
### Autoloading
|
92
94
|
|
93
95
|
The Maintenance Tasks framework does not support autoloading in `:classic` mode.
|
94
|
-
Please ensure your application is using
|
95
|
-
[Zeitwerk](https://github.com/fxn/zeitwerk) to load your code. For more
|
96
|
+
Please ensure your application is using [Zeitwerk][] to load your code. For more
|
96
97
|
information, please consult the [Rails guides on autoloading and reloading
|
97
|
-
constants]
|
98
|
+
constants][autoloading].
|
99
|
+
|
100
|
+
[Zeitwerk]: https://github.com/fxn/zeitwerk
|
101
|
+
[autoloading]: https://guides.rubyonrails.org/autoloading_and_reloading_constants.html
|
98
102
|
|
99
103
|
## Usage
|
100
104
|
|
@@ -187,15 +191,57 @@ module Maintenance
|
|
187
191
|
end
|
188
192
|
```
|
189
193
|
|
194
|
+
`posts.csv`:
|
190
195
|
```csv
|
191
|
-
# posts.csv
|
192
196
|
title,content
|
193
197
|
My Title,Hello World!
|
194
198
|
```
|
195
199
|
|
196
200
|
The files uploaded to your Active Storage service provider will be renamed to
|
197
|
-
include an ISO 8601 timestamp and the Task name in snake case format.
|
198
|
-
|
201
|
+
include an ISO 8601 timestamp and the Task name in snake case format.
|
202
|
+
|
203
|
+
The implicit `#count` method loads and parses the entire file to determine the
|
204
|
+
accurate number of rows. With files with millions of rows, it takes several
|
205
|
+
seconds to process. Consider skipping the count (defining a `count` that returns
|
206
|
+
`nil`) or use an approximation, eg: count the number of new lines:
|
207
|
+
|
208
|
+
```ruby
|
209
|
+
def count(task)
|
210
|
+
task.csv_content.count("\n") - 1
|
211
|
+
end
|
212
|
+
```
|
213
|
+
|
214
|
+
#### CSV options
|
215
|
+
|
216
|
+
Tasks can pass [options for Ruby's CSV parser][csv-parse-options] by adding
|
217
|
+
keyword arguments to `csv_collection`:
|
218
|
+
|
219
|
+
[csv-parse-options]: https://ruby-doc.org/3.3.0/stdlibs/csv/CSV.html#class-CSV-label-Options+for+Parsing
|
220
|
+
|
221
|
+
```ruby
|
222
|
+
# app/tasks/maintenance/import_posts_task.rb
|
223
|
+
|
224
|
+
module Maintenance
|
225
|
+
class ImportPosts
|
226
|
+
csv_collection(skip_lines: /^#/, converters: ->(field) { field.strip })
|
227
|
+
|
228
|
+
def process(row)
|
229
|
+
Post.create!(title: row["title"], content: row["content"])
|
230
|
+
end
|
231
|
+
end
|
232
|
+
end
|
233
|
+
```
|
234
|
+
|
235
|
+
These options instruct Ruby's CSV parser to skip lines that start with a `#`,
|
236
|
+
and removes the leading and trailing spaces from any field, so that the
|
237
|
+
following file will be processed identically as the previous example:
|
238
|
+
|
239
|
+
`posts.csv`:
|
240
|
+
```csv
|
241
|
+
# A comment
|
242
|
+
title,content
|
243
|
+
My Title ,Hello World!
|
244
|
+
```
|
199
245
|
|
200
246
|
#### Batch CSV Tasks
|
201
247
|
|
@@ -305,11 +351,11 @@ If you have a special use case requiring iteration over an unsupported
|
|
305
351
|
collection type, such as external resources fetched from some API, you can
|
306
352
|
implement the `enumerator_builder(cursor:)` method in your task.
|
307
353
|
|
308
|
-
This method should return an `Enumerator`, yielding pairs of
|
309
|
-
|
310
|
-
|
311
|
-
|
312
|
-
|
354
|
+
This method should return an `Enumerator`, yielding pairs of `[item, cursor]`.
|
355
|
+
Maintenance Tasks takes care of persisting the current cursor position and will
|
356
|
+
provide it as the `cursor` argument if your task is interrupted or resumed. The
|
357
|
+
`cursor` is stored as a `String`, so your custom enumerator should handle
|
358
|
+
serializing/deserializing the value if required.
|
313
359
|
|
314
360
|
```ruby
|
315
361
|
# app/tasks/maintenance/custom_enumerator_task.rb
|
@@ -377,7 +423,7 @@ module Maintenance
|
|
377
423
|
throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
|
378
424
|
DatabaseStatus.unhealthy?
|
379
425
|
end
|
380
|
-
...
|
426
|
+
# ...
|
381
427
|
end
|
382
428
|
end
|
383
429
|
```
|
@@ -413,6 +459,100 @@ to run. Since arguments are specified in the user interface via text area
|
|
413
459
|
inputs, it’s important to check that they conform to the format your Task
|
414
460
|
expects, and to sanitize any inputs if necessary.
|
415
461
|
|
462
|
+
### Custom cursor columns to improve performance
|
463
|
+
|
464
|
+
The [job-iteration gem][job-iteration], on which this gem depends, adds an
|
465
|
+
`order by` clause to the relation returned by the `collection` method, in order
|
466
|
+
to iterate through records. It defaults to order on the `id` column.
|
467
|
+
|
468
|
+
The [job-iteration gem][job-iteration] supports configuring which columns are
|
469
|
+
used to order the cursor, as documented in
|
470
|
+
[`build_active_record_enumerator_on_records`][ji-ar-enumerator-doc].
|
471
|
+
|
472
|
+
[ji-ar-enumerator-doc]: https://www.rubydoc.info/gems/job-iteration/JobIteration/EnumeratorBuilder#build_active_record_enumerator_on_records-instance_method
|
473
|
+
|
474
|
+
The `maintenance-tasks` gem exposes the ability that `job-iteration` provides to
|
475
|
+
control the cursor columns, through the `cursor_columns` method in the
|
476
|
+
`MaintenanceTasks::Task` class. If the `cursor_columns` method returns `nil`,
|
477
|
+
the query is ordered by the primary key. If cursor columns values change during
|
478
|
+
an iteration, records may be skipped or yielded multiple times.
|
479
|
+
|
480
|
+
```ruby
|
481
|
+
module Maintenance
|
482
|
+
class UpdatePostsTask < MaintenanceTasks::Task
|
483
|
+
def cursor_columns
|
484
|
+
[:created_at, :id]
|
485
|
+
end
|
486
|
+
|
487
|
+
def collection
|
488
|
+
Post.where(created_at: 2.days.ago...1.hour.ago)
|
489
|
+
end
|
490
|
+
|
491
|
+
def process(post)
|
492
|
+
post.update!(content: "updated content")
|
493
|
+
end
|
494
|
+
end
|
495
|
+
end
|
496
|
+
```
|
497
|
+
|
498
|
+
### Subscribing to instrumentation events
|
499
|
+
|
500
|
+
If you are interested in actioning a specific task event, please refer to the [Using Task Callbacks](#using-task-callbacks) section below. However, if you want to subscribe to all events, irrespective of the task, you can use the following Active Support notifications:
|
501
|
+
|
502
|
+
```ruby
|
503
|
+
enqueued.maintenance_tasks # This event is published when a task has been enqueued by the user.
|
504
|
+
succeeded.maintenance_tasks # This event is published when a task has finished without any errors.
|
505
|
+
cancelled.maintenance_tasks # This event is published when the user explicitly halts the execution of a task.
|
506
|
+
paused.maintenance_tasks # This event is published when a task is paused by the user in the middle of its run.
|
507
|
+
errored.maintenance_tasks # This event is published when the task's code produces an unhandled exception.
|
508
|
+
```
|
509
|
+
|
510
|
+
These notifications offer a way to monitor the lifecycle of maintenance tasks in your application.
|
511
|
+
|
512
|
+
Usage example:
|
513
|
+
|
514
|
+
```ruby
|
515
|
+
ActiveSupport::Notifications.subscribe("succeeded.maintenance_tasks") do |*, payload|
|
516
|
+
task_name = payload[:task_name]
|
517
|
+
arguments = payload[:arguments]
|
518
|
+
metadata = payload[:metadata]
|
519
|
+
job_id = payload[:job_id]
|
520
|
+
run_id = payload[:run_id]
|
521
|
+
time_running = payload[:time_running]
|
522
|
+
started_at = payload[:started_at]
|
523
|
+
ended_at = payload[:ended_at]
|
524
|
+
rescue => e
|
525
|
+
Rails.logger.error(e)
|
526
|
+
end
|
527
|
+
|
528
|
+
ActiveSupport::Notifications.subscribe("errored.maintenance_tasks") do |*, payload|
|
529
|
+
task_name = payload[:task_name]
|
530
|
+
error = payload[:error]
|
531
|
+
error_message = error[:message]
|
532
|
+
error_class = error[:class]
|
533
|
+
error_backtrace = error[:backtrace]
|
534
|
+
rescue => e
|
535
|
+
Rails.logger.error(e)
|
536
|
+
end
|
537
|
+
|
538
|
+
# or
|
539
|
+
|
540
|
+
class MaintenanceTasksInstrumenter < ActiveSupport::Subscriber
|
541
|
+
attach_to :maintenance_tasks
|
542
|
+
|
543
|
+
def enqueued(event)
|
544
|
+
task_name = event.payload[:task_name]
|
545
|
+
arguments = event.payload[:arguments]
|
546
|
+
metadata = event.payload[:metadata]
|
547
|
+
|
548
|
+
SlackNotifier.broadcast(SLACK_CHANNEL,
|
549
|
+
"Job #{task_name} was started by #{metadata[:user_email]}} with arguments #{arguments.to_s.truncate(255)}")
|
550
|
+
rescue => e
|
551
|
+
Rails.logger.error(e)
|
552
|
+
end
|
553
|
+
end
|
554
|
+
```
|
555
|
+
|
416
556
|
### Using Task Callbacks
|
417
557
|
|
418
558
|
The Task provides callbacks that hook into its life cycle.
|
@@ -463,21 +603,6 @@ end
|
|
463
603
|
If any of the other callbacks cause an exception, it will be handled by the
|
464
604
|
error handler, and will cause the task to stop running.
|
465
605
|
|
466
|
-
Callback behaviour can be shared across all tasks using an initializer.
|
467
|
-
|
468
|
-
```ruby
|
469
|
-
# config/initializer/maintenance_tasks.rb
|
470
|
-
Rails.autoloaders.main.on_load("MaintenanceTasks::Task") do
|
471
|
-
MaintenanceTasks::Task.class_eval do
|
472
|
-
after_start(:notify)
|
473
|
-
|
474
|
-
private
|
475
|
-
|
476
|
-
def notify; end
|
477
|
-
end
|
478
|
-
end
|
479
|
-
```
|
480
|
-
|
481
606
|
### Considerations when writing Tasks
|
482
607
|
|
483
608
|
Maintenance Tasks relies on the queue adapter configured for your application to
|
@@ -595,6 +720,42 @@ module Maintenance
|
|
595
720
|
end
|
596
721
|
```
|
597
722
|
|
723
|
+
### Writing tests for a Task that uses a custom enumerator
|
724
|
+
|
725
|
+
Tests for tasks that use custom enumerators need to instantiate the task class
|
726
|
+
in order to call `#build_enumerator`. Once the task instance is set up, validate
|
727
|
+
that `#build_enumerator` returns an enumerator yielding pairs of [item, cursor]
|
728
|
+
as expected.
|
729
|
+
|
730
|
+
```ruby
|
731
|
+
# test/tasks/maintenance/custom_enumerating_task.rb
|
732
|
+
|
733
|
+
require "test_helper"
|
734
|
+
|
735
|
+
module Maintenance
|
736
|
+
class CustomEnumeratingTaskTest < ActiveSupport::TestCase
|
737
|
+
setup do
|
738
|
+
@task = CustomEnumeratingTask.new
|
739
|
+
end
|
740
|
+
|
741
|
+
test "#build_enumerator returns enumerator yielding pairs of [item, cursor]" do
|
742
|
+
enum = @task.build_enumerator(cursor: 0)
|
743
|
+
expected_items = [:b, :c]
|
744
|
+
|
745
|
+
assert_equal 2, enum.size
|
746
|
+
|
747
|
+
enum.each_with_index do |item, cursor|
|
748
|
+
assert_equal expected_items[cursor], item
|
749
|
+
end
|
750
|
+
end
|
751
|
+
|
752
|
+
test "#process performs a task iteration" do
|
753
|
+
# ...
|
754
|
+
end
|
755
|
+
end
|
756
|
+
end
|
757
|
+
```
|
758
|
+
|
598
759
|
### Running a Task
|
599
760
|
|
600
761
|
#### Running a Task from the Web UI
|
@@ -917,7 +1078,7 @@ MaintenanceTasks.parent_controller = "Services::CustomController"
|
|
917
1078
|
class Services::CustomController < ActionController::Base
|
918
1079
|
include CustomSecurityThings
|
919
1080
|
include CustomLoggingThings
|
920
|
-
...
|
1081
|
+
# ...
|
921
1082
|
end
|
922
1083
|
```
|
923
1084
|
|
@@ -928,12 +1089,14 @@ If no value is specified, it will default to `"ActionController::Base"`.
|
|
928
1089
|
|
929
1090
|
#### Configure time after which the task will be considered stuck
|
930
1091
|
|
931
|
-
To specify a time duration after which a task is considered stuck if it has not
|
932
|
-
you can configure `MaintenanceTasks.stuck_task_duration`. This
|
933
|
-
job infrastructure events that may prevent the
|
1092
|
+
To specify a time duration after which a task is considered stuck if it has not
|
1093
|
+
been updated, you can configure `MaintenanceTasks.stuck_task_duration`. This
|
1094
|
+
duration should account for job infrastructure events that may prevent the
|
1095
|
+
maintenance tasks job from being executed and cancelling the task.
|
934
1096
|
|
935
|
-
The value for `MaintenanceTasks.stuck_task_duration` must be an
|
936
|
-
If no value is specified, it will default to 5
|
1097
|
+
The value for `MaintenanceTasks.stuck_task_duration` must be an
|
1098
|
+
`ActiveSupport::Duration`. If no value is specified, it will default to 5
|
1099
|
+
minutes.
|
937
1100
|
|
938
1101
|
### Metadata
|
939
1102
|
|
@@ -33,6 +33,45 @@ module MaintenanceTasks
|
|
33
33
|
def build_enumerator(_run, cursor:)
|
34
34
|
cursor ||= @run.cursor
|
35
35
|
@collection_enum = @task.enumerator_builder(cursor: cursor)
|
36
|
+
|
37
|
+
@collection_enum ||= case (collection = @task.collection)
|
38
|
+
when :no_collection
|
39
|
+
enumerator_builder.build_once_enumerator(cursor: nil)
|
40
|
+
when ActiveRecord::Relation
|
41
|
+
enumerator_builder.active_record_on_records(collection, cursor: cursor, columns: @task.cursor_columns)
|
42
|
+
when ActiveRecord::Batches::BatchEnumerator
|
43
|
+
if collection.start || collection.finish
|
44
|
+
raise ArgumentError, <<~MSG.squish
|
45
|
+
#{@task.class.name}#collection cannot support
|
46
|
+
a batch enumerator with the "start" or "finish" options.
|
47
|
+
MSG
|
48
|
+
end
|
49
|
+
|
50
|
+
# For now, only support automatic count based on the enumerator for
|
51
|
+
# batches
|
52
|
+
enumerator_builder.active_record_on_batch_relations(
|
53
|
+
collection.relation,
|
54
|
+
cursor: cursor,
|
55
|
+
batch_size: collection.batch_size,
|
56
|
+
columns: @task.cursor_columns,
|
57
|
+
)
|
58
|
+
when Array
|
59
|
+
enumerator_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
|
60
|
+
when BatchCsvCollectionBuilder::BatchCsv
|
61
|
+
JobIteration::CsvEnumerator.new(collection.csv).batches(
|
62
|
+
batch_size: collection.batch_size,
|
63
|
+
cursor: cursor&.to_i,
|
64
|
+
)
|
65
|
+
when CSV
|
66
|
+
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
|
67
|
+
else
|
68
|
+
raise ArgumentError, <<~MSG.squish
|
69
|
+
#{@task.class.name}#collection must be either an
|
70
|
+
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
|
71
|
+
Array, or CSV.
|
72
|
+
MSG
|
73
|
+
end
|
74
|
+
|
36
75
|
throttle_enumerator(@collection_enum)
|
37
76
|
end
|
38
77
|
|
@@ -12,16 +12,17 @@ module MaintenanceTasks
|
|
12
12
|
# Initialize a BatchCsvCollectionBuilder with a batch size.
|
13
13
|
#
|
14
14
|
# @param batch_size [Integer] the number of CSV rows in a batch.
|
15
|
-
|
15
|
+
# @param csv_options [Hash] options to pass to the CSV parser.
|
16
|
+
def initialize(batch_size, **csv_options)
|
16
17
|
@batch_size = batch_size
|
17
|
-
super()
|
18
|
+
super(**csv_options)
|
18
19
|
end
|
19
20
|
|
20
21
|
# Defines the collection to be iterated over, based on the provided CSV.
|
21
22
|
# Includes the CSV and the batch size.
|
22
23
|
def collection(task)
|
23
24
|
BatchCsv.new(
|
24
|
-
csv: CSV.new(task.csv_content,
|
25
|
+
csv: CSV.new(task.csv_content, **@csv_options),
|
25
26
|
batch_size: @batch_size,
|
26
27
|
)
|
27
28
|
end
|
@@ -5,24 +5,27 @@ require "csv"
|
|
5
5
|
module MaintenanceTasks
|
6
6
|
# Strategy for building a Task that processes CSV files.
|
7
7
|
#
|
8
|
+
# @param csv_options [Hash] options to pass to the CSV parser.
|
8
9
|
# @api private
|
9
10
|
class CsvCollectionBuilder
|
11
|
+
def initialize(**csv_options)
|
12
|
+
@csv_options = csv_options
|
13
|
+
end
|
14
|
+
|
10
15
|
# Defines the collection to be iterated over, based on the provided CSV.
|
11
16
|
#
|
12
|
-
# @return [CSV] the CSV object constructed from the specified CSV content
|
13
|
-
# with headers.
|
17
|
+
# @return [CSV] the CSV object constructed from the specified CSV content.
|
14
18
|
def collection(task)
|
15
|
-
CSV.new(task.csv_content,
|
19
|
+
CSV.new(task.csv_content, **@csv_options)
|
16
20
|
end
|
17
21
|
|
18
|
-
# The number of rows to be processed.
|
19
|
-
#
|
20
|
-
# Note that
|
21
|
-
# newlines.
|
22
|
+
# The number of rows to be processed.
|
23
|
+
# It uses the CSV library for an accurate row count.
|
24
|
+
# Note that the entire file is loaded. It will take several seconds with files with millions of rows.
|
22
25
|
#
|
23
26
|
# @return [Integer] the approximate number of rows to process.
|
24
27
|
def count(task)
|
25
|
-
task.csv_content.count
|
28
|
+
CSV.new(task.csv_content, **@csv_options).count
|
26
29
|
end
|
27
30
|
|
28
31
|
# Return that the Task processes CSV content.
|
@@ -52,17 +52,17 @@ module MaintenanceTasks
|
|
52
52
|
total = @run.tick_total
|
53
53
|
|
54
54
|
if !total?
|
55
|
-
"Processed #{number_to_delimited(count)} "\
|
55
|
+
"Processed #{number_to_delimited(count)} " \
|
56
56
|
"#{"item".pluralize(count)}."
|
57
57
|
elsif over_total?
|
58
|
-
"Processed #{number_to_delimited(count)} "\
|
59
|
-
"#{"item".pluralize(count)} "\
|
58
|
+
"Processed #{number_to_delimited(count)} " \
|
59
|
+
"#{"item".pluralize(count)} " \
|
60
60
|
"(expected #{number_to_delimited(total)})."
|
61
61
|
else
|
62
62
|
percentage = 100.0 * count / total
|
63
63
|
|
64
|
-
"Processed #{number_to_delimited(count)} out of "\
|
65
|
-
"#{number_to_delimited(total)} #{"item".pluralize(total)} "\
|
64
|
+
"Processed #{number_to_delimited(count)} out of " \
|
65
|
+
"#{number_to_delimited(total)} #{"item".pluralize(total)} " \
|
66
66
|
"(#{number_to_percentage(percentage, precision: 0)})."
|
67
67
|
end
|
68
68
|
end
|
@@ -33,7 +33,13 @@ module MaintenanceTasks
|
|
33
33
|
]
|
34
34
|
COMPLETED_STATUSES = [:succeeded, :errored, :cancelled]
|
35
35
|
|
36
|
-
|
36
|
+
if Rails.gem_version >= Gem::Version.new("7.0.alpha")
|
37
|
+
enum :status, STATUSES.to_h { |status| [status, status.to_s] }
|
38
|
+
else
|
39
|
+
enum status: STATUSES.to_h { |status| [status, status.to_s] }
|
40
|
+
end
|
41
|
+
|
42
|
+
after_save :instrument_status_change
|
37
43
|
|
38
44
|
validate :task_name_belongs_to_a_valid_task, on: :create
|
39
45
|
validate :csv_attachment_presence, on: :create
|
@@ -448,6 +454,30 @@ module MaintenanceTasks
|
|
448
454
|
|
449
455
|
private
|
450
456
|
|
457
|
+
def instrument_status_change
|
458
|
+
return unless status_previously_changed? || id_previously_changed?
|
459
|
+
return if running? || pausing? || cancelling? || interrupted?
|
460
|
+
|
461
|
+
attr = {
|
462
|
+
run_id: id,
|
463
|
+
job_id: job_id,
|
464
|
+
task_name: task_name,
|
465
|
+
arguments: arguments,
|
466
|
+
metadata: metadata,
|
467
|
+
time_running: time_running,
|
468
|
+
started_at: started_at,
|
469
|
+
ended_at: ended_at,
|
470
|
+
}
|
471
|
+
|
472
|
+
attr[:error] = {
|
473
|
+
message: error_message,
|
474
|
+
class: error_class,
|
475
|
+
backtrace: backtrace,
|
476
|
+
} if errored?
|
477
|
+
|
478
|
+
ActiveSupport::Notifications.instrument("#{status}.maintenance_tasks", attr)
|
479
|
+
end
|
480
|
+
|
451
481
|
def run_task_callbacks(callback)
|
452
482
|
task.run_callbacks(callback)
|
453
483
|
rescue Task::NotFoundError
|
@@ -74,7 +74,7 @@ module MaintenanceTasks
|
|
74
74
|
|
75
75
|
def enqueue(run, job)
|
76
76
|
unless job.enqueue
|
77
|
-
raise "The job to perform #{run.task_name} could not be enqueued. "\
|
77
|
+
raise "The job to perform #{run.task_name} could not be enqueued. " \
|
78
78
|
"Enqueuing has been prevented by a callback."
|
79
79
|
end
|
80
80
|
rescue => error
|
@@ -65,20 +65,24 @@ module MaintenanceTasks
|
|
65
65
|
# Make this Task a task that handles CSV.
|
66
66
|
#
|
67
67
|
# @param in_batches [Integer] optionally, supply a batch size if the CSV
|
68
|
-
#
|
68
|
+
# should be processed in batches.
|
69
|
+
# @param csv_options [Hash] optionally, supply options for the CSV parser.
|
70
|
+
# If not given, defaults to: <code>{ headers: true }</code>
|
71
|
+
# @see https://ruby-doc.org/3.3.0/stdlibs/csv/CSV.html#class-CSV-label-Options+for+Parsing
|
69
72
|
#
|
70
73
|
# An input to upload a CSV will be added in the form to start a Run. The
|
71
74
|
# collection and count method are implemented.
|
72
|
-
def csv_collection(in_batches: nil)
|
75
|
+
def csv_collection(in_batches: nil, **csv_options)
|
73
76
|
unless defined?(ActiveStorage)
|
74
|
-
raise NotImplementedError, "Active Storage needs to be installed\n"\
|
77
|
+
raise NotImplementedError, "Active Storage needs to be installed\n" \
|
75
78
|
"To resolve this issue run: bin/rails active_storage:install"
|
76
79
|
end
|
77
80
|
|
81
|
+
csv_options[:headers] = true unless csv_options.key?(:headers)
|
78
82
|
self.collection_builder_strategy = if in_batches
|
79
|
-
BatchCsvCollectionBuilder.new(in_batches)
|
83
|
+
BatchCsvCollectionBuilder.new(in_batches, **csv_options)
|
80
84
|
else
|
81
|
-
CsvCollectionBuilder.new
|
85
|
+
CsvCollectionBuilder.new(**csv_options)
|
82
86
|
end
|
83
87
|
end
|
84
88
|
|
@@ -230,6 +234,18 @@ module MaintenanceTasks
|
|
230
234
|
self.class.collection_builder_strategy.collection(self)
|
231
235
|
end
|
232
236
|
|
237
|
+
# The columns used to build the `ORDER BY` clause of the query for iteration.
|
238
|
+
#
|
239
|
+
# If cursor_columns returns nil, the query is ordered by the primary key.
|
240
|
+
# If cursor columns values change during an iteration, records may be skipped or yielded multiple times.
|
241
|
+
# More details in the documentation of JobIteration::EnumeratorBuilder.build_active_record_enumerator_on_records:
|
242
|
+
# https://www.rubydoc.info/gems/job-iteration/JobIteration/EnumeratorBuilder#build_active_record_enumerator_on_records-instance_method
|
243
|
+
#
|
244
|
+
# @return the cursor_columns.
|
245
|
+
def cursor_columns
|
246
|
+
nil
|
247
|
+
end
|
248
|
+
|
233
249
|
# Placeholder method to raise in case a subclass fails to implement the
|
234
250
|
# expected instance method.
|
235
251
|
#
|
@@ -248,7 +264,7 @@ module MaintenanceTasks
|
|
248
264
|
self.class.collection_builder_strategy.count(self)
|
249
265
|
end
|
250
266
|
|
251
|
-
# Default
|
267
|
+
# Default enumerator builder. You may override this method to return any
|
252
268
|
# Enumerator yielding pairs of `[item, item_cursor]`.
|
253
269
|
#
|
254
270
|
# @param cursor [String, nil] cursor position to resume from, or nil on
|
@@ -256,46 +272,7 @@ module MaintenanceTasks
|
|
256
272
|
#
|
257
273
|
# @return [Enumerator]
|
258
274
|
def enumerator_builder(cursor:)
|
259
|
-
|
260
|
-
|
261
|
-
job_iteration_builder = JobIteration::EnumeratorBuilder.new(nil)
|
262
|
-
|
263
|
-
case collection
|
264
|
-
when :no_collection
|
265
|
-
job_iteration_builder.build_once_enumerator(cursor: nil)
|
266
|
-
when ActiveRecord::Relation
|
267
|
-
job_iteration_builder.active_record_on_records(collection, cursor: cursor)
|
268
|
-
when ActiveRecord::Batches::BatchEnumerator
|
269
|
-
if collection.start || collection.finish
|
270
|
-
raise ArgumentError, <<~MSG.squish
|
271
|
-
#{self.class.name}#collection cannot support
|
272
|
-
a batch enumerator with the "start" or "finish" options.
|
273
|
-
MSG
|
274
|
-
end
|
275
|
-
|
276
|
-
# For now, only support automatic count based on the enumerator for
|
277
|
-
# batches
|
278
|
-
job_iteration_builder.active_record_on_batch_relations(
|
279
|
-
collection.relation,
|
280
|
-
cursor: cursor,
|
281
|
-
batch_size: collection.batch_size,
|
282
|
-
)
|
283
|
-
when Array
|
284
|
-
job_iteration_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
|
285
|
-
when BatchCsvCollectionBuilder::BatchCsv
|
286
|
-
JobIteration::CsvEnumerator.new(collection.csv).batches(
|
287
|
-
batch_size: collection.batch_size,
|
288
|
-
cursor: cursor&.to_i,
|
289
|
-
)
|
290
|
-
when CSV
|
291
|
-
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
|
292
|
-
else
|
293
|
-
raise ArgumentError, <<~MSG.squish
|
294
|
-
#{self.class.name}#collection must be either an
|
295
|
-
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
|
296
|
-
Array, or CSV.
|
297
|
-
MSG
|
298
|
-
end
|
275
|
+
nil
|
299
276
|
end
|
300
277
|
end
|
301
278
|
end
|
@@ -6,7 +6,7 @@ module MaintenanceTasks
|
|
6
6
|
# @api private
|
7
7
|
class TaskGenerator < Rails::Generators::NamedBase
|
8
8
|
source_root File.expand_path("templates", __dir__)
|
9
|
-
desc "This generator creates a task file at app/tasks and a corresponding "\
|
9
|
+
desc "This generator creates a task file at app/tasks and a corresponding " \
|
10
10
|
"test."
|
11
11
|
|
12
12
|
class_option :csv,
|
@@ -24,7 +24,7 @@ module MaintenanceTasks
|
|
24
24
|
# Creates the Task file.
|
25
25
|
def create_task_file
|
26
26
|
if options[:csv] && options[:no_collection]
|
27
|
-
raise "Multiple Task type options provided. Please use either "\
|
27
|
+
raise "Multiple Task type options provided. Please use either " \
|
28
28
|
"--csv or --no-collection."
|
29
29
|
end
|
30
30
|
template_file = File.join(
|
@@ -23,11 +23,11 @@ module MaintenanceTasks
|
|
23
23
|
DESC
|
24
24
|
|
25
25
|
# Specify the CSV file to process for CSV Tasks
|
26
|
-
desc = "Supply a CSV file to be processed by a CSV Task, "\
|
26
|
+
desc = "Supply a CSV file to be processed by a CSV Task, " \
|
27
27
|
"--csv path/to/csv/file.csv"
|
28
28
|
option :csv, lazy_default: :stdin, desc: desc
|
29
29
|
# Specify arguments to supply to a Task supporting parameters
|
30
|
-
desc = "Supply arguments for a Task that accepts parameters as a set of "\
|
30
|
+
desc = "Supply arguments for a Task that accepts parameters as a set of " \
|
31
31
|
"<key>:<value> pairs."
|
32
32
|
option :arguments, type: :hash, desc: desc
|
33
33
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: maintenance_tasks
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 2.
|
4
|
+
version: 2.7.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Shopify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2024-
|
11
|
+
date: 2024-04-16 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: actionpack
|
@@ -171,7 +171,7 @@ homepage: https://github.com/Shopify/maintenance_tasks
|
|
171
171
|
licenses:
|
172
172
|
- MIT
|
173
173
|
metadata:
|
174
|
-
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.
|
174
|
+
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.7.0
|
175
175
|
allowed_push_host: https://rubygems.org
|
176
176
|
post_install_message:
|
177
177
|
rdoc_options: []
|
@@ -188,7 +188,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
188
188
|
- !ruby/object:Gem::Version
|
189
189
|
version: '0'
|
190
190
|
requirements: []
|
191
|
-
rubygems_version: 3.5.
|
191
|
+
rubygems_version: 3.5.9
|
192
192
|
signing_key:
|
193
193
|
specification_version: 4
|
194
194
|
summary: A Rails engine for queuing and managing maintenance tasks
|