maintenance_tasks 2.3.1 → 2.3.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dadbcebb3f103324f6dd5d4319777ffbb5caa5c935912f092299275d27c40f58
-  data.tar.gz: a58d54eed126d14c9e3e036cc76238f3c738de40c66038882f4824230137c791
+  metadata.gz: 1f292dbc04fbbb71588c0be3af0bc95bfb5de5ca27f9aa7562b681cad39c6241
+  data.tar.gz: d631953126ffbd114721fea778fc8d43f5c86c20461c04943c3127a1b6f9e0d3
 SHA512:
-  metadata.gz: d0ffc76f0c5a2b7a3520ac7e8ae5f41e346f03eb84500238c1e73fec7dfea11586aa6d6b2c44841b72b550c023f3e7f4926dab686b823e475a22ec31e1b513ae
-  data.tar.gz: ac810b5fd6f1a9965bdf9556035196d4603bf84dd03297fb1f02ba5aa31c38935db9bca1d8195dc0950875083d0e372a0c540226bf80b9beb4986624072f71d6
+  metadata.gz: 146510c2a63f74b6084b43f9fa939c017e242b79ae0c453008b17b8e5c5e2332bd0d972ab4d8dce4d9d365836bcada8df8a7f1a7d717dc1d733232edd0830a4e
+  data.tar.gz: d1df2c04b6274843b32e4feb44a4bfb1eec3e64717ae67d78bbb440c08c34f24f63882d880dfc602529b2971f9bbfc9ce74345b848ecd3f780d1ab9270a610ec

data/README.md

@@ -1,9 +1,63 @@
-# MaintenanceTasks
+# Maintenance Tasks
 
 A Rails engine for queuing and managing maintenance tasks.
 
+By ”maintenance task”, this project means a data migration, i.e. code that
+changes data in the database, often to support schema migrations. For example,
+in order to introduce a new `NOT NULL` column, it has to be added as nullable
+first, backfilled with values, before finally being changed to `NOT NULL`.
+This engine helps with the second part of this process, backfilling.
+
+Maintenance tasks are collection-based tasks, usually using Active Record,
+that update the data in your database. They can be paused or interrupted.
+Maintenance tasks can operate [in batches](#processing-batch-collections) and
+use [throttling](#throttling) to control the load on your database.
+
+Maintenance tasks aren't meant to happen on a regular basis. They're used as
+needed, or as one-offs. Normally maintenance tasks are ephemeral, so they are
+used briefly and then deleted.
+
+The Rails engine has a web-based UI for listing maintenance tasks, seeing
+their status, and starting, pausing and restarting them.
+
 [![Link to demo video](static/demo.png)](https://www.youtube.com/watch?v=BTuvTQxlFzs)
 
+## Should I Use Maintenance Tasks?
+
+Maintenance tasks have a limited, specific job UI. While the engine can be
+used to provide a user interface for other data changes, such as data changes
+for support requests, we recommend you use regular application code for those
+use cases instead. These inevitably require more flexibility than this engine
+will be able to provide.
+
+If your task shouldn't run as an Active Job, it probably isn't a good match
+for this gem. If your task doesn't need to run in the background,
+consider a runner script instead. If your task doesn't need to be
+interruptible, consider a normal Active Job.
+
+Maintenance tasks can be interrupted between iterations. If your task [isn't
+collection-based](#tasks-that-dont-need-a-collection) (no CSV file or database
+table) or has very large batches, it will get limited benefit from throttling
+(pausing between iterations) or interrupting. This might be fine, or the added
+complexity of maintenance Tasks over normal Active Jobs may not be worthwhile.
+
+If your task updates your database schema instead of data, use a migration
+instead of a maintenance task.
+
+If your task happens regularly, consider Active Jobs with a scheduler or cron,
+[job-iteration jobs](https://github.com/shopify/job-iteration) and/or [custom
+rails_admin UIs][rails-admin-engines] instead of the Maintenance Tasks gem.
+Maintenance tasks should be ephemeral, to suit their intentionally limited UI.
+
+To create seed data for a new application, use the provided Rails
+`db/seeds.rb` file instead.
+
+If your application can't handle a half-completed migration, maintenance tasks
+are probably the wrong tool. Remember that maintenance tasks are intentionally
+pausable and can be cancelled halfway.
+
+[rails-admin-engines]: https://www.ruby-toolbox.com/categories/rails_admin_interfaces
+
 ## Installation
 
 To install the gem and run the install generator, execute:
@@ -168,6 +222,9 @@ Note that `#count` is calculated automatically based on the number of batches in
 your collection, and your Task’s progress will be displayed in terms of batches
 (not the total number of rows in your CSV).
 
+Non-batched CSV tasks will have an effective batch size of 1, which can reduce
+the efficiency of your database operations.
+
 ### Processing Batch Collections
 
 The Maintenance Tasks gem supports processing Active Records in batches. This
@@ -213,7 +270,7 @@ inside `#process`.
 ### Tasks that don’t need a Collection
 
 Sometimes, you might want to run a Task that performs a single operation, such
-as enqueuing another background job or hitting an external API. The gem supports
+as enqueuing another background job or querying an external API. The gem supports
 collection-less tasks.
 
 Generate a collection-less Task by running:
@@ -242,10 +299,12 @@ end
 
 ### Throttling
 
-Maintenance Tasks often modify a lot of data and can be taxing on your database.
+Maintenance tasks often modify a lot of data and can be taxing on your database.
 The gem provides a throttling mechanism that can be used to throttle a Task when
-a given condition is met. If a Task is throttled, it will be interrupted and
-retried after a backoff period has passed. The default backoff is 30 seconds.
+a given condition is met. If a Task is throttled (the throttle block returns
+true), it will be interrupted and retried after a backoff period has passed. The
+default backoff is 30 seconds.
+
 Specify the throttle condition as a block:
 
 ```ruby
@@ -277,7 +336,7 @@ Tasks can define multiple throttle conditions. Throttle conditions are inherited
 by descendants, and new conditions will be appended without impacting existing
 conditions.
 
-The backoff can also be specified as a Proc:
+The backoff can also be specified as a Proc that receives no arguments:
 
 ```ruby
 # app/tasks/maintenance/update_posts_throttled_task.rb
@@ -362,7 +421,7 @@ module Maintenance
     after_error :dangerous_notify
 
     def dangerous_notify
-      # This error is rescued in favour of the original error causing the error flow.
+      # This error is rescued and ignored in favour of the original error causing the error flow.
       raise NotDeliveredError
     end
 
@@ -392,21 +451,21 @@ end
 
 ### Considerations when writing Tasks
 
-MaintenanceTasks relies on the queue adapter configured for your application to
+Maintenance Tasks relies on the queue adapter configured for your application to
 run the job which is processing your Task. The guidelines for writing Task may
 depend on the queue adapter but in general, you should follow these rules:
 
 * Duration of `Task#process`: processing a single element of the collection
   should take less than 25 seconds, or the duration set as a timeout for Sidekiq
-  or the queue adapter configured in your application. It allows the Task to be
-  safely interrupted and resumed.
+  or the queue adapter configured in your application. Short batches allow the
+  Task to be safely interrupted and resumed.
 * Idempotency of `Task#process`: it should be safe to run `process` multiple
   times for the same element of the collection. Read more in [this Sidekiq best
   practice][sidekiq-idempotent]. It’s important if the Task errors and you run
-  it again, because the same element that errored the Task may well be processed
-  again. It especially matters in the situation described above, when the
-  iteration duration exceeds the timeout: if the job is re-enqueued, multiple
-  elements may be processed again.
+  it again, because the same element that caused the Task to give an error may
+  well be processed again. It especially matters in the situation described
+  above, when the iteration duration exceeds the timeout: if the job is
+  re-enqueued, multiple elements may be processed again.
 
 [sidekiq-idempotent]: https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional
 
@@ -636,8 +695,8 @@ re-enqueue the jobs that were in progress, the Task may be in a seemingly stuck
 situation where it appears to be running but is not. In that situation, pausing
 or cancelling it will not result in the Task being paused or cancelled, as the
 Task will get stuck in a state of `pausing` or `cancelling`. As a work-around,
-if a Task is `cancelling` for more than 5 minutes, you will be able to cancel it
-for good, which will just mark it as cancelled, allowing you to run it again.
+if a Task is `cancelling` for more than 5 minutes, you can cancel it again.
+It will then be marked as fully cancelled, allowing you to run it again.
 
 ### Configuring the gem
 
@@ -840,8 +899,8 @@ or email of the user who performed the maintenance task.
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
-MaintenanceTasks.metadata = ->(context) do
- { user_email: context.current_user.email }
+MaintenanceTasks.metadata = ->() do
+  { user_email: current_user.email }
 end
 ```
 

data/app/controllers/maintenance_tasks/runs_controller.rb

@@ -13,7 +13,7 @@ module MaintenanceTasks
       task = Runner.run(
         name: params.fetch(:task_id),
         csv_file: params[:csv_file],
-        arguments: params.fetch(:task_arguments, {}).permit!.to_h,
+        arguments: params.fetch(:task, {}).permit!.to_h,
         metadata: instance_exec(&MaintenanceTasks.metadata || -> {}),
         &block
       )

data/app/models/maintenance_tasks/task.rb

@@ -41,15 +41,27 @@ module MaintenanceTasks
         task
       end
 
-      # Returns a list of concrete classes that inherit from the Task
-      # superclass.
+      # Loads and returns a list of concrete classes that inherit
+      # from the Task superclass.
       #
       # @return [Array<Class>] the list of classes.
-      def available_tasks
+      def load_all
         load_constants
         descendants
       end
 
+      # Loads and returns a list of concrete classes that inherit
+      # from the Task superclass.
+      #
+      # @return [Array<Class>] the list of classes.
+      def available_tasks
+        warn(<<~MSG.squish, category: :deprecated)
+          MaintenanceTasks::Task.available_tasks is deprecated and will be
+          removed from maintenance-tasks 3.0.0. Use .load_all instead.
+        MSG
+        load_all
+      end
+
       # Make this Task a task that handles CSV.
       #
       # @param in_batches [Integer] optionally, supply a batch size if the CSV

data/app/models/maintenance_tasks/task_data_index.rb

@@ -25,7 +25,7 @@ module MaintenanceTasks
       def available_tasks
         tasks = []
 
-        task_names = Task.available_tasks.map(&:name)
+        task_names = Task.load_all.map(&:name)
 
         active_runs = Run.with_attached_csv.active.where(task_name: task_names)
         active_runs.each do |run|

data/app/views/maintenance_tasks/tasks/show.html.erb

@@ -15,7 +15,7 @@
     <% parameter_names = @task.parameter_names %>
     <% if parameter_names.any? %>
       <div class="block">
-        <%= form.fields_for :task_arguments, @task.new do |ff| %>
+        <%= fields_for :task, @task.new do |ff| %>
           <% parameter_names.each do |parameter_name| %>
             <div class="field">
               <%= ff.label parameter_name, parameter_name, class: "label is-family-monospace" %>

data/lib/maintenance_tasks/cli.rb

@@ -16,15 +16,6 @@ module MaintenanceTasks
 
     desc "perform [TASK NAME]", "Runs the given Maintenance Task"
 
-    long_desc <<-LONGDESC
-      `maintenance_tasks perform` will run the Maintenance Task specified by the
-      [TASK NAME] argument.
-
-      Available Tasks:
-
-      #{MaintenanceTasks::Task.available_tasks.join("\n\n")}
-    LONGDESC
-
     # Specify the CSV file to process for CSV Tasks
     desc = "Supply a CSV file to be processed by a CSV Task, "\
       "--csv path/to/csv/file.csv"
@@ -50,6 +41,21 @@ module MaintenanceTasks
       say_status(:error, error.message, :red)
     end
 
+    # `long_desc` only allows us to use a static string as "long description".
+    # By redefining the `#long_description` method on the "perform" Command
+    # object instead, we make it dynamic, thus delaying the task loading
+    # process until it's actually required.
+    commands["perform"].define_singleton_method(:long_description) do
+      <<~LONGDESC
+        `maintenance_tasks perform` will run the Maintenance Task specified by
+        the [TASK NAME] argument.
+
+        Available Tasks:
+
+        #{Task.load_all.map(&:name).sort.join("\n\n")}
+      LONGDESC
+    end
+
     private
 
     def csv_file

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: maintenance_tasks
 version: !ruby/object:Gem::Version
-  version: 2.3.1
+  version: 2.3.3
 platform: ruby
 authors:
 - Shopify Engineering
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2023-08-24 00:00:00.000000000 Z
+date: 2023-10-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -56,14 +56,14 @@ dependencies:
   name: job-iteration
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.3.6
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - ">="
       - !ruby/object:Gem::Version
         version: 1.3.6
 - !ruby/object:Gem::Dependency
@@ -156,7 +156,7 @@ homepage: https://github.com/Shopify/maintenance_tasks
 licenses:
 - MIT
 metadata:
-  source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.3.1
+  source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.3.3
   allowed_push_host: https://rubygems.org
 post_install_message: 
 rdoc_options: []
@@ -173,7 +173,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.4.18
+rubygems_version: 3.4.20
 signing_key: 
 specification_version: 4
 summary: A Rails engine for queuing and managing maintenance tasks