maintenance_tasks 1.3.0 → 1.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +102 -17
- data/app/controllers/maintenance_tasks/tasks_controller.rb +5 -1
- data/app/helpers/maintenance_tasks/tasks_helper.rb +0 -14
- data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb +24 -4
- data/app/models/maintenance_tasks/run.rb +75 -7
- data/app/models/maintenance_tasks/runner.rb +8 -2
- data/app/models/maintenance_tasks/task_data.rb +9 -0
- data/app/tasks/maintenance_tasks/task.rb +4 -0
- data/app/views/maintenance_tasks/runs/info/_paused.html.erb +2 -2
- data/app/views/maintenance_tasks/runs/info/_running.html.erb +2 -2
- data/app/views/maintenance_tasks/tasks/index.html.erb +2 -2
- data/app/views/maintenance_tasks/tasks/show.html.erb +10 -0
- data/db/migrate/20210517131953_add_arguments_to_maintenance_tasks_runs.rb +6 -0
- data/lib/maintenance_tasks.rb +2 -0
- data/lib/maintenance_tasks/cli.rb +8 -2
- data/lib/patches/active_record_batch_enumerator.rb +23 -0
- metadata +6 -7
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 6c3c4c94cf355432e0d558e68bfc3f275972164c8ed932d2505b7c6450f0d124
|
4
|
+
data.tar.gz: 1b80ad4b58b9546d5ab2af77d2d0e64f8bd8b72dc6641ff445d33a67dae03860
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 7402e24bfbc370c6af54b441d2b1050f252ee44f51ab11612875fed7fa6822645420422dc33405b1665596e66f85f8fe1f8b1e443d023d451a626913afe6dc98
|
7
|
+
data.tar.gz: 8f4666aba4abc6f01d77b238de6b6d511d24a5d4932f5ec2499f0e91faefe2e8a3b18435109864c821495b2efde464dd3212a76eae5276b4e622eb1cb6f7da3c
|
data/README.md
CHANGED
@@ -10,7 +10,7 @@ To install the gem and run the install generator, execute:
|
|
10
10
|
|
11
11
|
```bash
|
12
12
|
$ bundle add maintenance_tasks
|
13
|
-
$ rails generate maintenance_tasks:install
|
13
|
+
$ bin/rails generate maintenance_tasks:install
|
14
14
|
```
|
15
15
|
|
16
16
|
The generator creates and runs a migration to add the necessary table to your
|
@@ -23,12 +23,12 @@ handler](#customizing-the-error-handler) for more information.
|
|
23
23
|
|
24
24
|
### Active Job Dependency
|
25
25
|
|
26
|
-
The Maintenance Tasks framework relies on
|
27
|
-
Tasks. The default queuing backend for
|
26
|
+
The Maintenance Tasks framework relies on Active Job behind the scenes to run
|
27
|
+
Tasks. The default queuing backend for Active Job is
|
28
28
|
[asynchronous][async-adapter]. It is **strongly recommended** to change this to
|
29
29
|
a persistent backend so that Task progress is not lost during code or
|
30
30
|
infrastructure changes. For more information on configuring a queuing backend,
|
31
|
-
take a look at the [
|
31
|
+
take a look at the [Active Job documentation][active-job-docs].
|
32
32
|
|
33
33
|
[async-adapter]: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html
|
34
34
|
[active-job-docs]: https://guides.rubyonrails.org/active_job_basics.html#setting-the-backend
|
@@ -40,7 +40,7 @@ take a look at the [ActiveJob documentation][active-job-docs].
|
|
40
40
|
A generator is provided to create tasks. Generate a new task by running:
|
41
41
|
|
42
42
|
```bash
|
43
|
-
$ rails generate maintenance_tasks:task update_posts
|
43
|
+
$ bin/rails generate maintenance_tasks:task update_posts
|
44
44
|
```
|
45
45
|
|
46
46
|
This creates the task file `app/tasks/maintenance/update_posts_task.rb`.
|
@@ -77,7 +77,7 @@ end
|
|
77
77
|
### Creating a CSV Task
|
78
78
|
|
79
79
|
You can also write a Task that iterates on a CSV file. Note that writing CSV
|
80
|
-
Tasks **requires
|
80
|
+
Tasks **requires Active Storage to be configured**. Ensure that the dependency
|
81
81
|
is specified in your application's Gemfile, and that you've followed the
|
82
82
|
[setup instuctions][setup].
|
83
83
|
|
@@ -86,7 +86,7 @@ is specified in your application's Gemfile, and that you've followed the
|
|
86
86
|
Generate a CSV Task by running:
|
87
87
|
|
88
88
|
```bash
|
89
|
-
$ rails generate maintenance_tasks:task import_posts --csv
|
89
|
+
$ bin/rails generate maintenance_tasks:task import_posts --csv
|
90
90
|
```
|
91
91
|
|
92
92
|
The generated task is a subclass of `MaintenanceTasks::Task` that implements:
|
@@ -112,6 +112,46 @@ title,content
|
|
112
112
|
My Title,Hello World!
|
113
113
|
```
|
114
114
|
|
115
|
+
### Processing Batch Collections
|
116
|
+
|
117
|
+
The Maintenance Tasks gem supports processing Active Records in batches. This
|
118
|
+
can reduce the number of calls your Task makes to the database. Use
|
119
|
+
`ActiveRecord::Batches#in_batches` on the relation returned by your collection to specify that your Task should process
|
120
|
+
batches instead of records. Active Record defaults to 1000 records by batch, but a custom size can be
|
121
|
+
specified.
|
122
|
+
|
123
|
+
```ruby
|
124
|
+
# app/tasks/maintenance/update_posts_in_batches_task.rb
|
125
|
+
module Maintenance
|
126
|
+
class UpdatePostsInBatchesTask < MaintenanceTasks::Task
|
127
|
+
def collection
|
128
|
+
Post.in_batches
|
129
|
+
end
|
130
|
+
|
131
|
+
def process(batch_of_posts)
|
132
|
+
batch_of_posts.update_all(content: "New content added on #{Time.now.utc}")
|
133
|
+
end
|
134
|
+
end
|
135
|
+
end
|
136
|
+
```
|
137
|
+
|
138
|
+
Ensure that you've implemented the following methods:
|
139
|
+
|
140
|
+
* `collection`: return an `ActiveRecord::Batches::BatchEnumerator`.
|
141
|
+
* `process`: do the work of your Task on a batch (`ActiveRecord::Relation`).
|
142
|
+
|
143
|
+
Note that `#count` is calculated automatically based on the number of batches in
|
144
|
+
your collection, and your Task's progress will be displayed in terms of batches
|
145
|
+
(not the number of records in the relation).
|
146
|
+
|
147
|
+
**Important!** Batches should only be used if `#process` is performing a batch
|
148
|
+
operation such as `#update_all` or `#delete_all`. If you need to iterate over
|
149
|
+
individual records, you should define a collection that [returns an
|
150
|
+
`ActiveRecord::Relation`](#creating-a-task). This uses batching
|
151
|
+
internally, but loads the records with one SQL query. Conversely, batch
|
152
|
+
collections load the primary keys of the records of the batch first, and then perform an additional query to load the
|
153
|
+
records when calling `each` (or any `Enumerable` method) inside `#process`.
|
154
|
+
|
115
155
|
### Throttling
|
116
156
|
|
117
157
|
Maintenance Tasks often modify a lot of data and can be taxing on your database.
|
@@ -145,13 +185,48 @@ end
|
|
145
185
|
|
146
186
|
Note that it's up to you to define a throttling condition that makes sense for
|
147
187
|
your app. Shopify implements `DatabaseStatus.healthy?` to check various MySQL
|
148
|
-
metrics such as replication lag, DB threads, whether DB writes are available,
|
188
|
+
metrics such as replication lag, DB threads, whether DB writes are available,
|
149
189
|
etc.
|
150
190
|
|
151
191
|
Tasks can define multiple throttle conditions. Throttle conditions are inherited
|
152
192
|
by descendants, and new conditions will be appended without impacting existing
|
153
193
|
conditions.
|
154
194
|
|
195
|
+
### Custom Task Parameters
|
196
|
+
|
197
|
+
Tasks may need additional information, supplied via parameters, to run.
|
198
|
+
Parameters can be defined as Active Model Attributes in a Task, and then
|
199
|
+
become accessible to any of Task's methods: `#collection`, `#count`, or
|
200
|
+
`#process`.
|
201
|
+
|
202
|
+
```ruby
|
203
|
+
# app/tasks/maintenance/update_posts_via_params_task.rb
|
204
|
+
module Maintenance
|
205
|
+
class UpdatePostsViaParamsTask < MaintenanceTasks::Task
|
206
|
+
attribute :updated_content, :string
|
207
|
+
validates :updated_content, presence: true
|
208
|
+
|
209
|
+
def collection
|
210
|
+
Post.all
|
211
|
+
end
|
212
|
+
|
213
|
+
def count
|
214
|
+
collection.count
|
215
|
+
end
|
216
|
+
|
217
|
+
def process(post)
|
218
|
+
post.update!(content: updated_content)
|
219
|
+
end
|
220
|
+
end
|
221
|
+
end
|
222
|
+
```
|
223
|
+
|
224
|
+
Tasks can leverage Active Model Validations when defining parameters. Arguments
|
225
|
+
supplied to a Task accepting parameters will be validated before the Task starts
|
226
|
+
to run. Since arguments are specified in the user interface via text area
|
227
|
+
inputs, it's important to check that they conform to the format your Task
|
228
|
+
expects, and to sanitize any inputs if necessary.
|
229
|
+
|
155
230
|
### Considerations when writing Tasks
|
156
231
|
|
157
232
|
MaintenanceTasks relies on the queue adapter configured for your application to
|
@@ -241,6 +316,13 @@ To run a Task that processes CSVs from the command line, use the --csv option:
|
|
241
316
|
$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv 'path/to/my_csv.csv'
|
242
317
|
```
|
243
318
|
|
319
|
+
To run a Task that takes arguments from the command line, use the --arguments
|
320
|
+
option, passing arguments as a set of <key>:<value> pairs:
|
321
|
+
|
322
|
+
```bash
|
323
|
+
$ bundle exec maintenance_tasks perform Maintenance::ParamsTask --arguments post_ids:1,2,3 content:"Hello, World!"
|
324
|
+
```
|
325
|
+
|
244
326
|
You can also run a Task in Ruby by sending `run` with a Task name to Runner:
|
245
327
|
|
246
328
|
```ruby
|
@@ -257,6 +339,16 @@ MaintenanceTasks::Runner.run(
|
|
257
339
|
)
|
258
340
|
```
|
259
341
|
|
342
|
+
To run a Task that takes arguments using the Runner, provide a Hash containing
|
343
|
+
the set of arguments (`{ parameter_name: argument_value }`) to `run`:
|
344
|
+
|
345
|
+
```ruby
|
346
|
+
MaintenanceTasks::Runner.run(
|
347
|
+
name: "Maintenance::ParamsTask",
|
348
|
+
arguments: { post_ids: "1,2,3" }
|
349
|
+
)
|
350
|
+
```
|
351
|
+
|
260
352
|
### Monitoring your Task's status
|
261
353
|
|
262
354
|
The web UI will provide updates on the status of your Task. Here are the states
|
@@ -364,7 +456,7 @@ The error handler should be a lambda that accepts three arguments:
|
|
364
456
|
exception monitoring service, make sure you **sanitize the object** to avoid
|
365
457
|
leaking sensitive data and **convert it to a format** that is compatible with
|
366
458
|
your bug tracker. For example, Bugsnag only sends the id and class name of
|
367
|
-
|
459
|
+
Active Record objects in order to protect sensitive data. CSV rows, on the
|
368
460
|
other hand, are converted to strings and passed raw to Bugsnag, so make sure
|
369
461
|
to filter any personal data from these objects before adding them to a
|
370
462
|
report.
|
@@ -451,7 +543,7 @@ Use bundler to check for and upgrade to newer versions. After installing a new
|
|
451
543
|
version, re-run the install command:
|
452
544
|
|
453
545
|
```bash
|
454
|
-
$ rails generate maintenance_tasks:install
|
546
|
+
$ bin/rails generate maintenance_tasks:install
|
455
547
|
```
|
456
548
|
|
457
549
|
This ensures that new migrations are installed and run as well.
|
@@ -464,12 +556,6 @@ pull requests. You can find the contribution guidelines on
|
|
464
556
|
|
465
557
|
[contributing]: https://github.com/Shopify/maintenance_tasks/blob/main/.github/CONTRIBUTING.md
|
466
558
|
|
467
|
-
### Dependabot updates
|
468
|
-
|
469
|
-
Whenever Dependabot creates a PR for a gem bump, check out the branch locally
|
470
|
-
and run `bin/update-gemfile <gem>` to ensure all the gemfiles have the gem
|
471
|
-
updated consistently.
|
472
|
-
|
473
559
|
## Releasing new versions
|
474
560
|
|
475
561
|
Updates should be added to the latest draft release on GitHub as Pull Requests
|
@@ -478,7 +564,6 @@ are merged.
|
|
478
564
|
Once a release is ready, follow these steps:
|
479
565
|
|
480
566
|
* Update `spec.version` in `maintenance_tasks.gemspec`.
|
481
|
-
* Run `bin/gemfile-update install` to bump the version in all the lockfiles.
|
482
567
|
* Open a PR and merge on approval.
|
483
568
|
* Deploy via [Shipit][shipit] and see the new version on
|
484
569
|
<https://rubygems.org/gems/maintenance_tasks>.
|
@@ -27,11 +27,15 @@ module MaintenanceTasks
|
|
27
27
|
def run
|
28
28
|
task = Runner.run(
|
29
29
|
name: params.fetch(:id),
|
30
|
-
csv_file: params[:csv_file]
|
30
|
+
csv_file: params[:csv_file],
|
31
|
+
arguments: params.fetch(:task_arguments, {}).permit!.to_h,
|
31
32
|
)
|
32
33
|
redirect_to(task_path(task))
|
33
34
|
rescue ActiveRecord::RecordInvalid => error
|
34
35
|
redirect_to(task_path(error.record.task_name), alert: error.message)
|
36
|
+
rescue ActiveRecord::ValueTooLong => error
|
37
|
+
task_name = params.fetch(:id)
|
38
|
+
redirect_to(task_path(task_name), alert: error.message)
|
35
39
|
rescue Runner::EnqueuingError => error
|
36
40
|
redirect_to(task_path(error.run.task_name), alert: error.message)
|
37
41
|
end
|
@@ -63,20 +63,6 @@ module MaintenanceTasks
|
|
63
63
|
tag.span(status.capitalize, class: ["tag"] + STATUS_COLOURS.fetch(status))
|
64
64
|
end
|
65
65
|
|
66
|
-
# Returns the distance between now and the Run's expected completion time,
|
67
|
-
# if the Run has an estimated_completion_time.
|
68
|
-
#
|
69
|
-
# @param run [MaintenanceTasks::Run] the Run for which the estimated time to
|
70
|
-
# completion is being calculated.
|
71
|
-
# return [String, nil] the distance in words, or nil if the Run has no
|
72
|
-
# estimated completion time.
|
73
|
-
def estimated_time_to_completion(run)
|
74
|
-
estimated_completion_time = run.estimated_completion_time
|
75
|
-
if estimated_completion_time.present?
|
76
|
-
time_ago_in_words(estimated_completion_time)
|
77
|
-
end
|
78
|
-
end
|
79
|
-
|
80
66
|
# Reports the approximate elapsed time a Run has been processed so far based
|
81
67
|
# on the Run's time running attribute.
|
82
68
|
#
|
@@ -32,17 +32,35 @@ module MaintenanceTasks
|
|
32
32
|
def build_enumerator(_run, cursor:)
|
33
33
|
cursor ||= @run.cursor
|
34
34
|
collection = @task.collection
|
35
|
+
@enumerator = nil
|
35
36
|
|
36
37
|
collection_enum = case collection
|
37
38
|
when ActiveRecord::Relation
|
38
39
|
enumerator_builder.active_record_on_records(collection, cursor: cursor)
|
40
|
+
when ActiveRecord::Batches::BatchEnumerator
|
41
|
+
if collection.start || collection.finish
|
42
|
+
raise ArgumentError, <<~MSG.squish
|
43
|
+
#{@task.class.name}#collection cannot support
|
44
|
+
a batch enumerator with the "start" or "finish" options.
|
45
|
+
MSG
|
46
|
+
end
|
47
|
+
# For now, only support automatic count based on the enumerator for
|
48
|
+
# batches
|
49
|
+
@enumerator = enumerator_builder.active_record_on_batch_relations(
|
50
|
+
collection.relation,
|
51
|
+
cursor: cursor,
|
52
|
+
batch_size: collection.batch_size,
|
53
|
+
)
|
39
54
|
when Array
|
40
55
|
enumerator_builder.build_array_enumerator(collection, cursor: cursor)
|
41
56
|
when CSV
|
42
57
|
JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor)
|
43
58
|
else
|
44
|
-
raise ArgumentError,
|
45
|
-
|
59
|
+
raise ArgumentError, <<~MSG.squish
|
60
|
+
#{@task.class.name}#collection must be either an
|
61
|
+
Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
|
62
|
+
Array, or CSV.
|
63
|
+
MSG
|
46
64
|
end
|
47
65
|
|
48
66
|
@task.throttle_conditions.reduce(collection_enum) do |enum, condition|
|
@@ -71,7 +89,7 @@ module MaintenanceTasks
|
|
71
89
|
|
72
90
|
def before_perform
|
73
91
|
@run = arguments.first
|
74
|
-
@task =
|
92
|
+
@task = @run.task
|
75
93
|
if @task.respond_to?(:csv_content=)
|
76
94
|
@task.csv_content = @run.csv_file.download
|
77
95
|
end
|
@@ -85,7 +103,9 @@ module MaintenanceTasks
|
|
85
103
|
end
|
86
104
|
|
87
105
|
def on_start
|
88
|
-
|
106
|
+
count = @task.count
|
107
|
+
count = @enumerator&.size if count == :no_count
|
108
|
+
@run.update!(started_at: Time.now, tick_total: count)
|
89
109
|
end
|
90
110
|
|
91
111
|
def on_complete
|
@@ -34,10 +34,12 @@ module MaintenanceTasks
|
|
34
34
|
Task.available_tasks.map(&:to_s)
|
35
35
|
} }
|
36
36
|
validate :csv_attachment_presence, on: :create
|
37
|
+
validate :validate_task_arguments, on: :create
|
37
38
|
|
38
39
|
attr_readonly :task_name
|
39
40
|
|
40
41
|
serialize :backtrace
|
42
|
+
serialize :arguments, JSON
|
41
43
|
|
42
44
|
scope :active, -> { where(status: ACTIVE_STATUSES) }
|
43
45
|
|
@@ -149,19 +151,19 @@ module MaintenanceTasks
|
|
149
151
|
ACTIVE_STATUSES.include?(status.to_sym)
|
150
152
|
end
|
151
153
|
|
152
|
-
# Returns the
|
153
|
-
# ticks left and the average time needed to process a tick.
|
154
|
-
#
|
155
|
-
# zero.
|
154
|
+
# Returns the duration left for the Run to finish based on the number of
|
155
|
+
# ticks left and the average time needed to process a tick. Returns nil if
|
156
|
+
# the Run is completed, or if tick_count or tick_total is zero.
|
156
157
|
#
|
157
|
-
# @return [
|
158
|
-
|
158
|
+
# @return [ActiveSupport::Duration] the estimated duration left for the Run
|
159
|
+
# to finish.
|
160
|
+
def time_to_completion
|
159
161
|
return if completed? || tick_count == 0 || tick_total.to_i == 0
|
160
162
|
|
161
163
|
processed_per_second = (tick_count.to_f / time_running)
|
162
164
|
ticks_left = (tick_total - tick_count)
|
163
165
|
seconds_to_finished = ticks_left / processed_per_second
|
164
|
-
|
166
|
+
seconds_to_finished.seconds
|
165
167
|
end
|
166
168
|
|
167
169
|
# Cancels a Run.
|
@@ -206,6 +208,42 @@ module MaintenanceTasks
|
|
206
208
|
nil
|
207
209
|
end
|
208
210
|
|
211
|
+
# Support iterating over ActiveModel::Errors in Rails 6.0 and Rails 6.1+.
|
212
|
+
# To be removed when Rails 6.0 is no longer supported.
|
213
|
+
if Rails::VERSION::STRING.match?(/^6.0/)
|
214
|
+
# Performs validation on the arguments to use for the Task. If the Task is
|
215
|
+
# invalid, the errors are added to the Run.
|
216
|
+
def validate_task_arguments
|
217
|
+
arguments_match_task_attributes if arguments.present?
|
218
|
+
if task.invalid?
|
219
|
+
error_messages = task.errors
|
220
|
+
.map { |attribute, message| "#{attribute.inspect} #{message}" }
|
221
|
+
errors.add(
|
222
|
+
:arguments,
|
223
|
+
"are invalid: #{error_messages.join("; ")}"
|
224
|
+
)
|
225
|
+
end
|
226
|
+
rescue Task::NotFoundError
|
227
|
+
nil
|
228
|
+
end
|
229
|
+
else
|
230
|
+
# Performs validation on the arguments to use for the Task. If the Task is
|
231
|
+
# invalid, the errors are added to the Run.
|
232
|
+
def validate_task_arguments
|
233
|
+
arguments_match_task_attributes if arguments.present?
|
234
|
+
if task.invalid?
|
235
|
+
error_messages = task.errors
|
236
|
+
.map { |error| "#{error.attribute.inspect} #{error.message}" }
|
237
|
+
errors.add(
|
238
|
+
:arguments,
|
239
|
+
"are invalid: #{error_messages.join("; ")}"
|
240
|
+
)
|
241
|
+
end
|
242
|
+
rescue Task::NotFoundError
|
243
|
+
nil
|
244
|
+
end
|
245
|
+
end
|
246
|
+
|
209
247
|
# Fetches the attached ActiveStorage CSV file for the run. Checks first
|
210
248
|
# whether the ActiveStorage::Attachment table exists so that we are
|
211
249
|
# compatible with apps that are not using ActiveStorage.
|
@@ -216,5 +254,35 @@ module MaintenanceTasks
|
|
216
254
|
return unless ActiveStorage::Attachment.table_exists?
|
217
255
|
super
|
218
256
|
end
|
257
|
+
|
258
|
+
# Returns a Task instance for this Run. Assigns any attributes to the Task
|
259
|
+
# based on the Run's parameters. Note that the Task instance is not supplied
|
260
|
+
# with :csv_content yet if it's a CSV Task. This is done in the job, since
|
261
|
+
# downloading the CSV file can take some time.
|
262
|
+
#
|
263
|
+
# @return [Task] a Task instance.
|
264
|
+
def task
|
265
|
+
@task ||= begin
|
266
|
+
task = Task.named(task_name).new
|
267
|
+
if task.attribute_names.any? && arguments.present?
|
268
|
+
task.assign_attributes(arguments)
|
269
|
+
end
|
270
|
+
task
|
271
|
+
rescue ActiveModel::UnknownAttributeError
|
272
|
+
task
|
273
|
+
end
|
274
|
+
end
|
275
|
+
|
276
|
+
private
|
277
|
+
|
278
|
+
def arguments_match_task_attributes
|
279
|
+
invalid_argument_keys = arguments.keys - task.attribute_names
|
280
|
+
if invalid_argument_keys.any?
|
281
|
+
error_message = <<~MSG.squish
|
282
|
+
Unknown parameters: #{invalid_argument_keys.map(&:to_sym).join(", ")}
|
283
|
+
MSG
|
284
|
+
errors.add(:base, error_message)
|
285
|
+
end
|
286
|
+
end
|
219
287
|
end
|
220
288
|
end
|
@@ -37,18 +37,24 @@ module MaintenanceTasks
|
|
37
37
|
# for the Task to iterate over when running, in the form of an attachable
|
38
38
|
# (see https://edgeapi.rubyonrails.org/classes/ActiveStorage/Attached/One.html#method-i-attach).
|
39
39
|
# Value is nil if the Task does not use CSV iteration.
|
40
|
+
# @param arguments [Hash] the arguments to persist to the Run and to make
|
41
|
+
# accessible to the Task.
|
40
42
|
#
|
41
43
|
# @return [Task] the Task that was run.
|
42
44
|
#
|
43
45
|
# @raise [EnqueuingError] if an error occurs while enqueuing the Run.
|
44
46
|
# @raise [ActiveRecord::RecordInvalid] if validation errors occur while
|
45
47
|
# creating the Run.
|
46
|
-
|
47
|
-
|
48
|
+
# @raise [ActiveRecord::ValueTooLong] if the creation of the Run fails due
|
49
|
+
# to a value being too long for the column type.
|
50
|
+
def run(name:, csv_file: nil, arguments: {})
|
51
|
+
run = Run.active.find_by(task_name: name) ||
|
52
|
+
Run.new(task_name: name, arguments: arguments)
|
48
53
|
run.csv_file.attach(csv_file) if csv_file
|
49
54
|
|
50
55
|
run.enqueued!
|
51
56
|
enqueue(run)
|
57
|
+
yield run if block_given?
|
52
58
|
Task.named(name)
|
53
59
|
end
|
54
60
|
|
@@ -132,6 +132,15 @@ module MaintenanceTasks
|
|
132
132
|
!deleted? && Task.named(name) < CsvCollection
|
133
133
|
end
|
134
134
|
|
135
|
+
# @return [Array<String>] the names of parameters the Task accepts.
|
136
|
+
def parameter_names
|
137
|
+
if deleted?
|
138
|
+
[]
|
139
|
+
else
|
140
|
+
Task.named(name).attribute_names
|
141
|
+
end
|
142
|
+
end
|
143
|
+
|
135
144
|
private
|
136
145
|
|
137
146
|
def runs
|
@@ -3,6 +3,9 @@ module MaintenanceTasks
|
|
3
3
|
# Base class that is inherited by the host application's task classes.
|
4
4
|
class Task
|
5
5
|
extend ActiveSupport::DescendantsTracker
|
6
|
+
include ActiveModel::Attributes
|
7
|
+
include ActiveModel::AttributeAssignment
|
8
|
+
include ActiveModel::Validations
|
6
9
|
|
7
10
|
class NotFoundError < NameError; end
|
8
11
|
|
@@ -128,6 +131,7 @@ module MaintenanceTasks
|
|
128
131
|
#
|
129
132
|
# @return [Integer, nil]
|
130
133
|
def count
|
134
|
+
:no_count
|
131
135
|
end
|
132
136
|
end
|
133
137
|
end
|
@@ -1,7 +1,7 @@
|
|
1
1
|
<p>
|
2
2
|
Ran for <%= time_running_in_words run %> until paused,
|
3
|
-
<% if run.
|
4
|
-
<%=
|
3
|
+
<% if (time_to_completion = run.time_to_completion) %>
|
4
|
+
<%= distance_of_time_in_words(time_to_completion) %> remaining.
|
5
5
|
<% else %>
|
6
6
|
processed <%= pluralize run.tick_count, 'item' %> so far.
|
7
7
|
<% end %>
|
@@ -2,8 +2,8 @@
|
|
2
2
|
<div class="content is-large">
|
3
3
|
<h3 class="title is-3"> The MaintenanceTasks gem has been successfully installed! </h3>
|
4
4
|
<p>
|
5
|
-
Any new Tasks will show up here. To start writing your first Task,
|
6
|
-
run <code>rails generate maintenance_tasks:task my_task</code>.
|
5
|
+
Any new Tasks will show up here. To start writing your first Task,
|
6
|
+
run <code>bin/rails generate maintenance_tasks:task my_task</code>.
|
7
7
|
</p>
|
8
8
|
</div>
|
9
9
|
<% else %>
|
@@ -16,6 +16,16 @@
|
|
16
16
|
<%= form.file_field :csv_file %>
|
17
17
|
</div>
|
18
18
|
<% end %>
|
19
|
+
<% if @task.parameter_names.any? %>
|
20
|
+
<div class="block">
|
21
|
+
<%= form.fields_for :task_arguments do |ff| %>
|
22
|
+
<% @task.parameter_names.each do |parameter| %>
|
23
|
+
<%= ff.label parameter, "#{parameter}: ", class: "label" %>
|
24
|
+
<%= ff.text_area parameter, class: "textarea" %>
|
25
|
+
<% end %>
|
26
|
+
<% end %>
|
27
|
+
</div>
|
28
|
+
<% end %>
|
19
29
|
<div class="block">
|
20
30
|
<%= form.submit 'Run', class: "button is-success", disabled: @task.deleted? %>
|
21
31
|
</div>
|
data/lib/maintenance_tasks.rb
CHANGED
@@ -7,6 +7,8 @@ require "active_record"
|
|
7
7
|
require "job-iteration"
|
8
8
|
require "maintenance_tasks/engine"
|
9
9
|
|
10
|
+
require "patches/active_record_batch_enumerator"
|
11
|
+
|
10
12
|
# The engine's namespace module. It provides isolation between the host
|
11
13
|
# application's code and the engine-specific code. Top-level engine constants
|
12
14
|
# and variables are defined under this module.
|
@@ -28,15 +28,21 @@ module MaintenanceTasks
|
|
28
28
|
option :csv, desc: "Supply a CSV file to be processed by a CSV Task, "\
|
29
29
|
'--csv "path/to/csv/file.csv"'
|
30
30
|
|
31
|
+
# Specify arguments to supply to a Task supporting parameters
|
32
|
+
option :arguments, type: :hash, desc: "Supply arguments for a Task that "\
|
33
|
+
"accepts parameters as a set of <key>:<value> pairs."
|
34
|
+
|
31
35
|
# Command to run a Task.
|
32
36
|
#
|
33
37
|
# It instantiates a Runner and sends a run message with the given Task name.
|
34
38
|
# If a CSV file is supplied using the --csv option, an attachable with the
|
35
|
-
# File IO object is sent along with the Task name to run.
|
39
|
+
# File IO object is sent along with the Task name to run. If arguments are
|
40
|
+
# supplied using the --arguments option, these are also passed to run.
|
36
41
|
#
|
37
42
|
# @param name [String] the name of the Task to be run.
|
38
43
|
def perform(name)
|
39
|
-
|
44
|
+
arguments = options[:arguments] || {}
|
45
|
+
task = Runner.run(name: name, csv_file: csv_file, arguments: arguments)
|
40
46
|
say_status(:success, "#{task.name} was enqueued.", :green)
|
41
47
|
rescue => error
|
42
48
|
say_status(:error, error.message, :red)
|
@@ -0,0 +1,23 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
# TODO: Remove this patch once all supported Rails versions include the changes
|
4
|
+
# upstream - https://github.com/rails/rails/pull/42312/commits/a031a43d969c87542c4ee8d0d338d55fcbb53376
|
5
|
+
module ActiveRecordBatchEnumerator
|
6
|
+
# The primary key value from which the BatchEnumerator starts,
|
7
|
+
# inclusive of the value.
|
8
|
+
attr_reader :start
|
9
|
+
|
10
|
+
# The primary key value at which the BatchEnumerator ends,
|
11
|
+
# inclusive of the value.
|
12
|
+
attr_reader :finish
|
13
|
+
|
14
|
+
# The relation from which the BatchEnumerator yields batches.
|
15
|
+
attr_reader :relation
|
16
|
+
|
17
|
+
# The size of the batches yielded by the BatchEnumerator.
|
18
|
+
def batch_size
|
19
|
+
@of
|
20
|
+
end
|
21
|
+
end
|
22
|
+
|
23
|
+
ActiveRecord::Batches::BatchEnumerator.include(ActiveRecordBatchEnumerator)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: maintenance_tasks
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.
|
4
|
+
version: 1.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Shopify Engineering
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-
|
11
|
+
date: 2021-06-07 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: actionpack
|
@@ -125,6 +125,7 @@ files:
|
|
125
125
|
- config/routes.rb
|
126
126
|
- db/migrate/20201211151756_create_maintenance_tasks_runs.rb
|
127
127
|
- db/migrate/20210225152418_remove_index_on_task_name.rb
|
128
|
+
- db/migrate/20210517131953_add_arguments_to_maintenance_tasks_runs.rb
|
128
129
|
- exe/maintenance_tasks
|
129
130
|
- lib/generators/maintenance_tasks/install_generator.rb
|
130
131
|
- lib/generators/maintenance_tasks/task_generator.rb
|
@@ -135,17 +136,15 @@ files:
|
|
135
136
|
- lib/maintenance_tasks.rb
|
136
137
|
- lib/maintenance_tasks/cli.rb
|
137
138
|
- lib/maintenance_tasks/engine.rb
|
139
|
+
- lib/patches/active_record_batch_enumerator.rb
|
138
140
|
- lib/tasks/maintenance_tasks_tasks.rake
|
139
141
|
homepage: https://github.com/Shopify/maintenance_tasks
|
140
142
|
licenses:
|
141
143
|
- MIT
|
142
144
|
metadata:
|
143
|
-
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v1.
|
145
|
+
source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v1.4.0
|
144
146
|
allowed_push_host: https://rubygems.org
|
145
|
-
post_install_message:
|
146
|
-
Thank you for installing Maintenance Tasks 1.3.0. To complete, please run:
|
147
|
-
|
148
|
-
rails generate maintenance_tasks:install
|
147
|
+
post_install_message:
|
149
148
|
rdoc_options: []
|
150
149
|
require_paths:
|
151
150
|
- lib
|