maintenance_tasks 1.3.0 → 1.4.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +102 -17
- data/app/controllers/maintenance_tasks/tasks_controller.rb +5 -1
- data/app/helpers/maintenance_tasks/tasks_helper.rb +0 -14
- data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb +24 -4
- data/app/models/maintenance_tasks/run.rb +75 -7
- data/app/models/maintenance_tasks/runner.rb +8 -2
- data/app/models/maintenance_tasks/task_data.rb +9 -0
- data/app/tasks/maintenance_tasks/task.rb +4 -0
- data/app/views/maintenance_tasks/runs/info/_paused.html.erb +2 -2
- data/app/views/maintenance_tasks/runs/info/_running.html.erb +2 -2
- data/app/views/maintenance_tasks/tasks/index.html.erb +2 -2
- data/app/views/maintenance_tasks/tasks/show.html.erb +10 -0
- data/db/migrate/20210517131953_add_arguments_to_maintenance_tasks_runs.rb +6 -0
- data/lib/maintenance_tasks.rb +2 -0
- data/lib/maintenance_tasks/cli.rb +8 -2
- data/lib/patches/active_record_batch_enumerator.rb +23 -0
- metadata +6 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6c3c4c94cf355432e0d558e68bfc3f275972164c8ed932d2505b7c6450f0d124
|
4
|
+
data.tar.gz: 1b80ad4b58b9546d5ab2af77d2d0e64f8bd8b72dc6641ff445d33a67dae03860
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7402e24bfbc370c6af54b441d2b1050f252ee44f51ab11612875fed7fa6822645420422dc33405b1665596e66f85f8fe1f8b1e443d023d451a626913afe6dc98
|
7
|
+
data.tar.gz: 8f4666aba4abc6f01d77b238de6b6d511d24a5d4932f5ec2499f0e91faefe2e8a3b18435109864c821495b2efde464dd3212a76eae5276b4e622eb1cb6f7da3c
|
data/README.md
CHANGED
@@ -10,7 +10,7 @@ To install the gem and run the install generator, execute:
|
|
10
10
|
|
11
11
|
```bash
|
12
12
|
$ bundle add maintenance_tasks
|
13
|
-
$ rails generate maintenance_tasks:install
|
13
|
+
$ bin/rails generate maintenance_tasks:install
|
14
14
|
```
|
15
15
|
|
16
16
|
The generator creates and runs a migration to add the necessary table to your
|
@@ -23,12 +23,12 @@ handler](#customizing-the-error-handler) for more information.
|
|
23
23
|
|
24
24
|
### Active Job Dependency
|
25
25
|
|
26
|
-
The Maintenance Tasks framework relies on
|
27
|
-
Tasks. The default queuing backend for
|
26
|
+
The Maintenance Tasks framework relies on Active Job behind the scenes to run
|
27
|
+
Tasks. The default queuing backend for Active Job is
|
28
28
|
[asynchronous][async-adapter]. It is **strongly recommended** to change this to
|
29
29
|
a persistent backend so that Task progress is not lost during code or
|
30
30
|
infrastructure changes. For more information on configuring a queuing backend,
|
31
|
-
take a look at the [
|
31
|
+
take a look at the [Active Job documentation][active-job-docs].
|
32
32
|
|
33
33
|
[async-adapter]: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html
|
34
34
|
[active-job-docs]: https://guides.rubyonrails.org/active_job_basics.html#setting-the-backend
|
@@ -40,7 +40,7 @@ take a look at the [ActiveJob documentation][active-job-docs].
|
|
40
40
|
A generator is provided to create tasks. Generate a new task by running:
|
41
41
|
|
42
42
|
```bash
|
43
|
-
$ rails generate maintenance_tasks:task update_posts
|
43
|
+
$ bin/rails generate maintenance_tasks:task update_posts
|
44
44
|
```
|
45
45
|
|
46
46
|
This creates the task file `app/tasks/maintenance/update_posts_task.rb`.
|
@@ -77,7 +77,7 @@ end
|
|
77
77
|
### Creating a CSV Task
|
78
78
|
|
79
79
|
You can also write a Task that iterates on a CSV file. Note that writing CSV
|
80
|
-
Tasks **requires
|
80
|
+
Tasks **requires Active Storage to be configured**. Ensure that the dependency
|
81
81
|
is specified in your application's Gemfile, and that you've followed the
|
82
82
|
[setup instuctions][setup].
|
83
83
|
|
@@ -86,7 +86,7 @@ is specified in your application's Gemfile, and that you've followed the
|
|
86
86
|
Generate a CSV Task by running:
|
87
87
|
|
88
88
|
```bash
|
89
|
-
$ rails generate maintenance_tasks:task import_posts --csv
|
89
|
+
$ bin/rails generate maintenance_tasks:task import_posts --csv
|
90
90
|
```
|
91
91
|
|
92
92
|
The generated task is a subclass of `MaintenanceTasks::Task` that implements:
|
@@ -112,6 +112,46 @@ title,content
|
|
112
112
|
My Title,Hello World!
|
113
113
|
```
|
114
114
|
|
115
|
+
### Processing Batch Collections
|
116
|
+
|
117
|
+
The Maintenance Tasks gem supports processing Active Records in batches. This
|
118
|
+
can reduce the number of calls your Task makes to the database. Use
|
119
|
+
`ActiveRecord::Batches#in_batches` on the relation returned by your collection to specify that your Task should process
|
120
|
+
batches instead of records. Active Record defaults to 1000 records by batch, but a custom size can be
|
121
|
+
specified.
|
122
|
+
|
123
|
+
```ruby
|
124
|
+
# app/tasks/maintenance/update_posts_in_batches_task.rb
|
125
|
+
module Maintenance
|
126
|
+
class UpdatePostsInBatchesTask < MaintenanceTasks::Task
|
127
|
+
def collection
|
128
|
+
Post.in_batches
|
129
|
+
end
|
130
|
+
|
131
|
+
def process(batch_of_posts)
|
132
|
+
batch_of_posts.update_all(content: "New content added on #{Time.now.utc}")
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
```
|
137
|
+
|
138
|
+
Ensure that you've implemented the following methods:
|
139
|
+
|
140
|
+
* `collection`: return an `ActiveRecord::Batches::BatchEnumerator`.
|
141
|
+
* `process`: do the work of your Task on a batch (`ActiveRecord::Relation`).
|
142
|
+
|
143
|
+
Note that `#count` is calculated automatically based on the number of batches in
|
144
|
+
your collection, and your Task's progress will be displayed in terms of batches
|
145
|
+
(not the number of records in the relation).
|
146
|
+
|
147
|
+
**Important!** Batches should only be used if `#process` is performing a batch
|
148
|
+
operation such as `#update_all` or `#delete_all`. If you need to iterate over
|
149
|
+
individual records, you should define a collection that [returns an
|
150
|
+
`ActiveRecord::Relation`](#creating-a-task). This uses batching
|
151
|
+
internally, but loads the records with one SQL query. Conversely, batch
|
152
|
+
collections load the primary keys of the records of the batch first, and then perform an additional query to load the
|
153
|
+
records when calling `each` (or any `Enumerable` method) inside `#process`.
|
154
|
+
|
115
155
|
### Throttling
|
116
156
|
|
117
157
|
Maintenance Tasks often modify a lot of data and can be taxing on your database.
|
@@ -145,13 +185,48 @@ end
|
|
145
185
|
|
146
186
|
Note that it's up to you to define a throttling condition that makes sense for
|
147
187
|
your app. Shopify implements `DatabaseStatus.healthy?` to check various MySQL
|
148
|
-
metrics such as replication lag, DB threads, whether DB writes are available,
|
188
|
+
metrics such as replication lag, DB threads, whether DB writes are available,
|
149
189
|
etc.
|
150
190
|
|
151
191
|
Tasks can define multiple throttle conditions. Throttle conditions are inherited
|
152
192
|
by descendants, and new conditions will be appended without impacting existing
|
153
193
|
conditions.
|
154
194
|
|
195
|
+
### Custom Task Parameters
|
196
|
+
|
197
|
+
Tasks may need additional information, supplied via parameters, to run.
|
198
|
+
Parameters can be defined as Active Model Attributes in a Task, and then
|
199
|
+
become accessible to any of Task's methods: `#collection`, `#count`, or
|
200
|
+
`#process`.
|
201
|
+
|
202
|
+
```ruby
|
203
|
+
# app/tasks/maintenance/update_posts_via_params_task.rb
|
204
|
+
module Maintenance
|
205
|
+
class UpdatePostsViaParamsTask < MaintenanceTasks::Task
|
206
|
+
attribute :updated_content, :string
|
207
|
+
validates :updated_content, presence: true
|
208
|
+
|
209
|
+
def collection
|
210
|
+
Post.all
|
211
|
+
end
|
212
|
+
|
213
|
+
def count
|
214
|
+
collection.count
|
215
|
+
end
|
216
|
+
|
217
|
+
def process(post)
|
218
|
+
post.update!(content: updated_content)
|
219
|
+
end
|
220
|
+
end
|
221
|
+
end
|
222
|
+
```
|
223
|
+
|
224
|
+
Tasks can leverage Active Model Validations when defining parameters. Arguments
|
225
|
+
supplied to a Task accepting parameters will be validated before the Task starts
|
226
|
+
to run. Since arguments are specified in the user interface via text area
|
227
|
+
inputs, it's important to check that they conform to the format your Task
|
228
|
+
expects, and to sanitize any inputs if necessary.
|
229
|
+
|
155
230
|
### Considerations when writing Tasks
|
156
231
|
|
157
232
|
MaintenanceTasks relies on the queue adapter configured for your application to
|
@@ -241,6 +316,13 @@ To run a Task that processes CSVs from the command line, use the --csv option:
|
|
241
316
|
$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv 'path/to/my_csv.csv'
|
242
317
|
```
|
243
318
|
|
319
|
+
To run a Task that takes arguments from the command line, use the --arguments
|
320
|
+
option, passing arguments as a set of <key>:<value> pairs:
|
321
|
+
|
322
|
+
```bash
|
323
|
+
$ bundle exec maintenance_tasks perform Maintenance::ParamsTask --arguments post_ids:1,2,3 content:"Hello, World!"
|
324
|
+
```
|
325
|
+
|
244
326
|
You can also run a Task in Ruby by sending `run` with a Task name to Runner:
|
245
327
|
|
246
328
|
```ruby
|
@@ -257,6 +339,16 @@ MaintenanceTasks::Runner.run(
|
|
257
339
|
)
|
258
340
|
```
|
259
341
|
|
342
|
+
To run a Task that takes arguments using the Runner, provide a Hash containing
|
343
|
+
the set of arguments (`{ parameter_name: argument_value }`) to `run`:
|
344
|
+
|
345
|
+
```ruby
|
346
|
+
MaintenanceTasks::Runner.run(
|
347
|
+
name: "Maintenance::ParamsTask",
|
348
|
+
arguments: { post_ids: "1,2,3" }
|
349
|
+
)
|
350
|
+
```
|
351
|
+
|
260
352
|
### Monitoring your Task's status
|
261
353
|
|
262
354
|
The web UI will provide updates on the status of your Task. Here are the states
|
@@ -364,7 +456,7 @@ The error handler should be a lambda that accepts three arguments:
|
|
364
456
|
exception monitoring service, make sure you **sanitize the object** to avoid
|
365
457
|
leaking sensitive data and **convert it to a format** that is compatible with
|
366
458
|
your bug tracker. For example, Bugsnag only sends the id and class name of
|
367
|
-
|
459
|
+
Active Record objects in order to protect sensitive data. CSV rows, on the
|
368
460
|
other hand, are converted to strings and passed raw to Bugsnag, so make sure
|
369
461
|
to filter any personal data from these objects before adding them to a
|
370
462
|
report.
|
@@ -451,7 +543,7 @@ Use bundler to check for and upgrade to newer versions. After installing a new
|
|
451
543
|
version, re-run the install command:
|
452
544
|
|
453
545
|
```bash
|
454
|
-
$ rails generate maintenance_tasks:install
|
546
|
+
$ bin/rails generate maintenance_tasks:install
|
455
547
|
```
|
456
548
|
|
457
549
|
This ensures that new migrations are installed and run as well.
|
@@ -464,12 +556,6 @@ pull requests. You can find the contribution guidelines on
|
|
464
556
|
|
465
557
|
[contributing]: https://github.com/Shopify/maintenance_tasks/blob/main/.github/CONTRIBUTING.md
|
466
558
|
|
467
|
-
### Dependabot updates
|
468
|
-
|
469
|
-
Whenever Dependabot creates a PR for a gem bump, check out the branch locally
|
470
|
-
and run `bin/update-gemfile <gem>` to ensure all the gemfiles have the gem
|
471
|
-
updated consistently.
|
472
|
-
|
473
559
|
## Releasing new versions
|
474
560
|
|
475
561
|
Updates should be added to the latest draft release on GitHub as Pull Requests
|
@@ -478,7 +564,6 @@ are merged.
|
|
478
564
|
Once a release is ready, follow these steps:
|
479
565
|
|
480
566
|
* Update `spec.version` in `maintenance_tasks.gemspec`.
|
481
|
-
* Run `bin/gemfile-update install` to bump the version in all the lockfiles.
|
482
567
|
* Open a PR and merge on approval.
|
483
568
|
* Deploy via [Shipit][shipit] and see the new version on
|
484
569
|
<https://rubygems.org/gems/maintenance_tasks>.
|
@@ -27,11 +27,15 @@ module MaintenanceTasks
|
|
27
27
|
def run
|
28
28
|
task = Runner.run(
|
29
29
|
name: params.fetch(:id),
|
30
|
-
csv_file: params[:csv_file]
|
30
|
+
csv_file: params[:csv_file],
|
31
|
+
arguments: params.fetch(:task_arguments, {}).permit!.to_h,
|
31
32
|
)
|
32
33
|
redirect_to(task_path(task))
|
33
34
|
rescue ActiveRecord::RecordInvalid => error
|
34
35
|
redirect_to(task_path(error.record.task_name), alert: error.message)
|
36
|
+
rescue ActiveRecord::ValueTooLong => error
|
37
|
+
task_name = params.fetch(:id)
|
38
|
+
redirect_to(task_path(task_name), alert: error.message)
|
35
39
|
rescue Runner::EnqueuingError => error
|
36
40
|
redirect_to(task_path(error.run.task_name), alert: error.message)
|
37
41
|
end
|
@@ -63,20 +63,6 @@ module MaintenanceTasks
|
|
63
63
|
tag.span(status.capitalize, class: ["tag"] + STATUS_COLOURS.fetch(status))
|
64
64
|
end
|
65
65
|
|
66
|
-
# Returns the distance between now and the Run's expected completion time,
|
67
|
-
# if the Run has an estimated_completion_time.
|
68
|
-
#
|
69
|
-
# @param run [MaintenanceTasks::Run] the Run for which the estimated time to
|
70
|
-
# completion is being calculated.
|
71
|
-
# return [String, nil] the distance in words, or nil if the Run has no
|
72
|
-
# estimated completion time.
|
73
|
-
def estimated_time_to_completion(run)
|
74
|
-
estimated_completion_time = run.estimated_completion_time
|
75
|
-
if estimated_completion_time.present?
|
76
|
-
time_ago_in_words(estimated_completion_time)
|
77
|
-
end
|
78
|
-
end
|
79
|
-
|
80
66
|
# Reports the approximate elapsed time a Run has been processed so far based
|
81
67
|
# on the Run's time running attribute.
|
82
68
|
#
|
@@ -32,17 +32,35 @@ module MaintenanceTasks
|
|
32
32
|
def build_enumerator(_run, cursor:)
|
33
33
|
cursor ||= @run.cursor
|
34
34
|
collection = @task.collection
|
35
|
+
@enumerator = nil
|
35
36
|
|
36
37
|
collection_enum = case collection
|
37
38
|
when ActiveRecord::Relation
|
38
39
|
enumerator_builder.active_record_on_records(collection, cursor: cursor)
|
40
|
+
when ActiveRecord::Batches::BatchEnumerator
|
41
|
+
if collection.start || collection.finish
|
42
|
+
raise ArgumentError, <<~MSG.squish
|
43
|
+
#{@task.class.name}#collection cannot support
|
44
|
+
a batch enumerator with the "start" or "finish" options.
|
45
|
+
MSG
|
46
|
+
end
|
47
|
+
# For now, only support automatic count based on the enumerator for
|
48
|
+
# batches
|
49
|
+
@enumerator = enumerator_builder.active_record_on_batch_relations(
|
50
|
+
collection.relation,
|
51
|
+
cursor: cursor,
|
52
|
+
batch_size: collection.batch_size,
|
53
|
+
)
|
39
54
|
when Array
|
40
55
|
enumerator_builder.build_array_enumerator(collection, cursor: cursor)
|
41
56
|
when CSV
|
42
57
|
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor)
|
43
58
|
else
|
44
|
-
raise ArgumentError,
|
45
|
-
|
59
|
+
raise ArgumentError, <<~MSG.squish
|
60
|
+
#{@task.class.name}#collection must be either an
|
61
|
+
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
|
62
|
+
Array, or CSV.
|
63
|
+
MSG
|
46
64
|
end
|
47
65
|
|
48
66
|
@task.throttle_conditions.reduce(collection_enum) do |enum, condition|
|
@@ -71,7 +89,7 @@ module MaintenanceTasks
|
|
71
89
|
|
72
90
|
def before_perform
|
73
91
|
@run = arguments.first
|
74
|
-
@task =
|
92
|
+
@task = @run.task
|
75
93
|
if @task.respond_to?(:csv_content=)
|
76
94
|
@task.csv_content = @run.csv_file.download
|
77
95
|
end
|
@@ -85,7 +103,9 @@ module MaintenanceTasks
|
|
85
103
|
end
|
86
104
|
|
87
105
|
def on_start
|
88
|
-
|
106
|
+
count = @task.count
|
107
|
+
count = @enumerator&.size if count == :no_count
|
108
|
+
@run.update!(started_at: Time.now, tick_total: count)
|
89
109
|
end
|
90
110
|
|
91
111
|
def on_complete
|
@@ -34,10 +34,12 @@ module MaintenanceTasks
|
|
34
34
|
Task.available_tasks.map(&:to_s)
|
35
35
|
} }
|
36
36
|
validate :csv_attachment_presence, on: :create
|
37
|
+
validate :validate_task_arguments, on: :create
|
37
38
|
|
38
39
|
attr_readonly :task_name
|
39
40
|
|
40
41
|
serialize :backtrace
|
42
|
+
serialize :arguments, JSON
|
41
43
|
|
42
44
|
scope :active, -> { where(status: ACTIVE_STATUSES) }
|
43
45
|
|
@@ -149,19 +151,19 @@ module MaintenanceTasks
|
|
149
151
|
ACTIVE_STATUSES.include?(status.to_sym)
|
150
152
|
end
|
151
153
|
|
152
|
-
# Returns the
|
153
|
-
# ticks left and the average time needed to process a tick.
|
154
|
-
#
|
155
|
-
# zero.
|
154
|
+
# Returns the duration left for the Run to finish based on the number of
|
155
|
+
# ticks left and the average time needed to process a tick. Returns nil if
|
156
|
+
# the Run is completed, or if tick_count or tick_total is zero.
|
156
157
|
#
|
157
|
-
# @return [
|
158
|
-
|
158
|
+
# @return [ActiveSupport::Duration] the estimated duration left for the Run
|
159
|
+
# to finish.
|
160
|
+
def time_to_completion
|
159
161
|
return if completed? || tick_count == 0 || tick_total.to_i == 0
|
160
162
|
|
161
163
|
processed_per_second = (tick_count.to_f / time_running)
|
162
164
|
ticks_left = (tick_total - tick_count)
|
163
165
|
seconds_to_finished = ticks_left / processed_per_second
|
164
|
-
|
166
|
+
seconds_to_finished.seconds
|
165
167
|
end
|
166
168
|
|
167
169
|
# Cancels a Run.
|
@@ -206,6 +208,42 @@ module MaintenanceTasks
|
|
206
208
|
nil
|
207
209
|
end
|
208
210
|
|
211
|
+
# Support iterating over ActiveModel::Errors in Rails 6.0 and Rails 6.1+.
|
212
|
+
# To be removed when Rails 6.0 is no longer supported.
|
213
|
+
if Rails::VERSION::STRING.match?(/^6.0/)
|
214
|
+
# Performs validation on the arguments to use for the Task. If the Task is
|
215
|
+
# invalid, the errors are added to the Run.
|
216
|
+
def validate_task_arguments
|
217
|
+
arguments_match_task_attributes if arguments.present?
|
218
|
+
if task.invalid?
|
219
|
+
error_messages = task.errors
|
220
|
+
.map { |attribute, message| "#{attribute.inspect} #{message}" }
|
221
|
+
errors.add(
|
222
|
+
:arguments,
|
223
|
+
"are invalid: #{error_messages.join("; ")}"
|
224
|
+
)
|
225
|
+
end
|
226
|
+
rescue Task::NotFoundError
|
227
|
+
nil
|
228
|
+
end
|
229
|
+
else
|
230
|
+
# Performs validation on the arguments to use for the Task. If the Task is
|
231
|
+
# invalid, the errors are added to the Run.
|
232
|
+
def validate_task_arguments
|
233
|
+
arguments_match_task_attributes if arguments.present?
|
234
|
+
if task.invalid?
|
235
|
+
error_messages = task.errors
|
236
|
+
.map { |error| "#{error.attribute.inspect} #{error.message}" }
|
237
|
+
errors.add(
|
238
|
+
:arguments,
|
239
|
+
"are invalid: #{error_messages.join("; ")}"
|
240
|
+
)
|
241
|
+
end
|
242
|
+
rescue Task::NotFoundError
|
243
|
+
nil
|
244
|
+
end
|
245
|
+
end
|
246
|
+
|
209
247
|
# Fetches the attached ActiveStorage CSV file for the run. Checks first
|
210
248
|
# whether the ActiveStorage::Attachment table exists so that we are
|
211
249
|
# compatible with apps that are not using ActiveStorage.
|
@@ -216,5 +254,35 @@ module MaintenanceTasks
|
|
216
254
|
return unless ActiveStorage::Attachment.table_exists?
|
217
255
|
super
|
218
256
|
end
|
257
|
+
|
258
|
+
# Returns a Task instance for this Run. Assigns any attributes to the Task
|
259
|
+
# based on the Run's parameters. Note that the Task instance is not supplied
|
260
|
+
# with :csv_content yet if it's a CSV Task. This is done in the job, since
|
261
|
+
# downloading the CSV file can take some time.
|
262
|
+
#
|
263
|
+
# @return [Task] a Task instance.
|
264
|
+
def task
|
265
|
+
@task ||= begin
|
266
|
+
task = Task.named(task_name).new
|
267
|
+
if task.attribute_names.any? && arguments.present?
|
268
|
+
task.assign_attributes(arguments)
|
269
|
+
end
|
270
|
+
task
|
271
|
+
rescue ActiveModel::UnknownAttributeError
|
272
|
+
task
|
273
|
+
end
|
274
|
+
end
|
275
|
+
|
276
|
+
private
|
277
|
+
|
278
|
+
def arguments_match_task_attributes
|
279
|
+
invalid_argument_keys = arguments.keys - task.attribute_names
|
280
|
+
if invalid_argument_keys.any?
|
281
|
+
error_message = <<~MSG.squish
|
282
|
+
Unknown parameters: #{invalid_argument_keys.map(&:to_sym).join(", ")}
|
283
|
+
MSG
|
284
|
+
errors.add(:base, error_message)
|
285
|
+
end
|
286
|
+
end
|
219
287
|
end
|
220
288
|
end
|
@@ -37,18 +37,24 @@ module MaintenanceTasks
|
|
37
37
|
# for the Task to iterate over when running, in the form of an attachable
|
38
38
|
# (see https://edgeapi.rubyonrails.org/classes/ActiveStorage/Attached/One.html#method-i-attach).
|
39
39
|
# Value is nil if the Task does not use CSV iteration.
|
40
|
+
# @param arguments [Hash] the arguments to persist to the Run and to make
|
41
|
+
# accessible to the Task.
|
40
42
|
#
|
41
43
|
# @return [Task] the Task that was run.
|
42
44
|
#
|
43
45
|
# @raise [EnqueuingError] if an error occurs while enqueuing the Run.
|
44
46
|
# @raise [ActiveRecord::RecordInvalid] if validation errors occur while
|
45
47
|
# creating the Run.
|
46
|
-
|
47
|
-
|
48
|
+
# @raise [ActiveRecord::ValueTooLong] if the creation of the Run fails due
|
49
|
+
# to a value being too long for the column type.
|
50
|
+
def run(name:, csv_file: nil, arguments: {})
|
51
|
+
run = Run.active.find_by(task_name: name) ||
|
52
|
+
Run.new(task_name: name, arguments: arguments)
|
48
53
|
run.csv_file.attach(csv_file) if csv_file
|
49
54
|
|
50
55
|
run.enqueued!
|
51
56
|
enqueue(run)
|
57
|
+
yield run if block_given?
|
52
58
|
Task.named(name)
|
53
59
|
end
|
54
60
|
|
@@ -132,6 +132,15 @@ module MaintenanceTasks
|
|
132
132
|
!deleted? && Task.named(name) < CsvCollection
|
133
133
|
end
|
134
134
|
|
135
|
+
# @return [Array<String>] the names of parameters the Task accepts.
|
136
|
+
def parameter_names
|
137
|
+
if deleted?
|
138
|
+
[]
|
139
|
+
else
|
140
|
+
Task.named(name).attribute_names
|
141
|
+
end
|
142
|
+
end
|
143
|
+
|
135
144
|
private
|
136
145
|
|
137
146
|
def runs
|
@@ -3,6 +3,9 @@ module MaintenanceTasks
|
|
3
3
|
# Base class that is inherited by the host application's task classes.
|
4
4
|
class Task
|
5
5
|
extend ActiveSupport::DescendantsTracker
|
6
|
+
include ActiveModel::Attributes
|
7
|
+
include ActiveModel::AttributeAssignment
|
8
|
+
include ActiveModel::Validations
|
6
9
|
|
7
10
|
class NotFoundError < NameError; end
|
8
11
|
|
@@ -128,6 +131,7 @@ module MaintenanceTasks
|
|
128
131
|
#
|
129
132
|
# @return [Integer, nil]
|
130
133
|
def count
|
134
|
+
:no_count
|
131
135
|
end
|
132
136
|
end
|
133
137
|
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
<p>
|
2
2
|
Ran for <%= time_running_in_words run %> until paused,
|
3
|
-
<% if run.
|
4
|
-
<%=
|
3
|
+
<% if (time_to_completion = run.time_to_completion) %>
|
4
|
+
<%= distance_of_time_in_words(time_to_completion) %> remaining.
|
5
5
|
<% else %>
|
6
6
|
processed <%= pluralize run.tick_count, 'item' %> so far.
|
7
7
|
<% end %>
|
@@ -2,8 +2,8 @@
|
|
2
2
|
<div class="content is-large">
|
3
3
|
<h3 class="title is-3"> The MaintenanceTasks gem has been successfully installed! </h3>
|
4
4
|
<p>
|
5
|
-
Any new Tasks will show up here. To start writing your first Task,
|
6
|
-
run <code>rails generate maintenance_tasks:task my_task</code>.
|
5
|
+
Any new Tasks will show up here. To start writing your first Task,
|
6
|
+
run <code>bin/rails generate maintenance_tasks:task my_task</code>.
|
7
7
|
</p>
|
8
8
|
</div>
|
9
9
|
<% else %>
|
@@ -16,6 +16,16 @@
|
|
16
16
|
<%= form.file_field :csv_file %>
|
17
17
|
</div>
|
18
18
|
<% end %>
|
19
|
+
<% if @task.parameter_names.any? %>
|
20
|
+
<div class="block">
|
21
|
+
<%= form.fields_for :task_arguments do |ff| %>
|
22
|
+
<% @task.parameter_names.each do |parameter| %>
|
23
|
+
<%= ff.label parameter, "#{parameter}: ", class: "label" %>
|
24
|
+
<%= ff.text_area parameter, class: "textarea" %>
|
25
|
+
<% end %>
|
26
|
+
<% end %>
|
27
|
+
</div>
|
28
|
+
<% end %>
|
19
29
|
<div class="block">
|
20
30
|
<%= form.submit 'Run', class: "button is-success", disabled: @task.deleted? %>
|
21
31
|
</div>
|
data/lib/maintenance_tasks.rb
CHANGED
@@ -7,6 +7,8 @@ require "active_record"
|
|
7
7
|
require "job-iteration"
|
8
8
|
require "maintenance_tasks/engine"
|
9
9
|
|
10
|
+
require "patches/active_record_batch_enumerator"
|
11
|
+
|
10
12
|
# The engine's namespace module. It provides isolation between the host
|
11
13
|
# application's code and the engine-specific code. Top-level engine constants
|
12
14
|
# and variables are defined under this module.
|
@@ -28,15 +28,21 @@ module MaintenanceTasks
|
|
28
28
|
option :csv, desc: "Supply a CSV file to be processed by a CSV Task, "\
|
29
29
|
'--csv "path/to/csv/file.csv"'
|
30
30
|
|
31
|
+
# Specify arguments to supply to a Task supporting parameters
|
32
|
+
option :arguments, type: :hash, desc: "Supply arguments for a Task that "\
|
33
|
+
"accepts parameters as a set of <key>:<value> pairs."
|
34
|
+
|
31
35
|
# Command to run a Task.
|
32
36
|
#
|
33
37
|
# It instantiates a Runner and sends a run message with the given Task name.
|
34
38
|
# If a CSV file is supplied using the --csv option, an attachable with the
|
35
|
-
# File IO object is sent along with the Task name to run.
|
39
|
+
# File IO object is sent along with the Task name to run. If arguments are
|
40
|
+
# supplied using the --arguments option, these are also passed to run.
|
36
41
|
#
|
37
42
|
# @param name [String] the name of the Task to be run.
|
38
43
|
def perform(name)
|
39
|
-
|
44
|
+
arguments = options[:arguments] || {}
|
45
|
+
task = Runner.run(name: name, csv_file: csv_file, arguments: arguments)
|
40
46
|
say_status(:success, "#{task.name} was enqueued.", :green)
|
41
47
|
rescue => error
|
42
48
|
say_status(:error, error.message, :red)
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
# TODO: Remove this patch once all supported Rails versions include the changes
|
4
|
+
# upstream - https://github.com/rails/rails/pull/42312/commits/a031a43d969c87542c4ee8d0d338d55fcbb53376
|
5
|
+
module ActiveRecordBatchEnumerator
|
6
|
+
# The primary key value from which the BatchEnumerator starts,
|
7
|
+
# inclusive of the value.
|
8
|
+
attr_reader :start
|
9
|
+
|
10
|
+
# The primary key value at which the BatchEnumerator ends,
|
11
|
+
# inclusive of the value.
|
12
|
+
attr_reader :finish
|
13
|
+
|
14
|
+
# The relation from which the BatchEnumerator yields batches.
|
15
|
+
attr_reader :relation
|
16
|
+
|
17
|
+
# The size of the batches yielded by the BatchEnumerator.
|
18
|
+
def batch_size
|
19
|
+
@of
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
ActiveRecord::Batches::BatchEnumerator.include(ActiveRecordBatchEnumerator)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: maintenance_tasks
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Shopify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-06-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: actionpack
|
@@ -125,6 +125,7 @@ files:
|
|
125
125
|
- config/routes.rb
|
126
126
|
- db/migrate/20201211151756_create_maintenance_tasks_runs.rb
|
127
127
|
- db/migrate/20210225152418_remove_index_on_task_name.rb
|
128
|
+
- db/migrate/20210517131953_add_arguments_to_maintenance_tasks_runs.rb
|
128
129
|
- exe/maintenance_tasks
|
129
130
|
- lib/generators/maintenance_tasks/install_generator.rb
|
130
131
|
- lib/generators/maintenance_tasks/task_generator.rb
|
@@ -135,17 +136,15 @@ files:
|
|
135
136
|
- lib/maintenance_tasks.rb
|
136
137
|
- lib/maintenance_tasks/cli.rb
|
137
138
|
- lib/maintenance_tasks/engine.rb
|
139
|
+
- lib/patches/active_record_batch_enumerator.rb
|
138
140
|
- lib/tasks/maintenance_tasks_tasks.rake
|
139
141
|
homepage: https://github.com/Shopify/maintenance_tasks
|
140
142
|
licenses:
|
141
143
|
- MIT
|
142
144
|
metadata:
|
143
|
-
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v1.
|
145
|
+
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v1.4.0
|
144
146
|
allowed_push_host: https://rubygems.org
|
145
|
-
post_install_message:
|
146
|
-
Thank you for installing Maintenance Tasks 1.3.0. To complete, please run:
|
147
|
-
|
148
|
-
rails generate maintenance_tasks:install
|
147
|
+
post_install_message:
|
149
148
|
rdoc_options: []
|
150
149
|
require_paths:
|
151
150
|
- lib
|