maintenance_tasks 1.3.0 → 1.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 649f66cc11a303666134c75aad8575b81eb376e358f69312180e21a14528ce50
-  data.tar.gz: b564b46e647c467c5d93a6198ab112729623059cfe194d1115d095c8ab8d9953
+  metadata.gz: 6019b1623a8cc72e877154f52e8437339598f0cab121189d78692e0989e86e61
+  data.tar.gz: 21563dc6dffadd2e58dd551fe2b7d71297b2da36500722f5d9fed91892b1fd6e
 SHA512:
-  metadata.gz: f313350c5a80fa8840b23976eb7cde8776f825bc25a9b6a8fb7ce5d6f1abe8148d547ae7b2c1368a72caf980a2a12377e33b7c325e1447de3c0a0b0686d87a8d
-  data.tar.gz: '084950e7be06503e12d437294c784d02e2f0449c76db6297c5f95fcfd40845e4ce7e3354e16fbe12dd99b82554946f16fbc3415c047c17109aab1303c6b81b9b'
+  metadata.gz: 93228d08b49fab144297cb44a0aa4b9be53144ff13b88c2eba384ef29fdb47b9a8caefaf94aaf1fbfbf4584f4d544396720c17d86f3bad31508fd66dfd7507b8
+  data.tar.gz: cab4cebb685fddf978eeebd62a5c16867215f849d424b9ba948beba0e93f42a5fe11ee1eccc15174c1494fa16a2fa100d3a7745c5eacef2527d425895805cf29

data/README.md

@@ -10,7 +10,7 @@ To install the gem and run the install generator, execute:
 
 ```bash
 $ bundle add maintenance_tasks
-$ rails generate maintenance_tasks:install
+$ bin/rails generate maintenance_tasks:install
 ```
 
 The generator creates and runs a migration to add the necessary table to your
@@ -23,12 +23,12 @@ handler](#customizing-the-error-handler) for more information.
 
 ### Active Job Dependency
 
-The Maintenance Tasks framework relies on ActiveJob behind the scenes to run
-Tasks. The default queuing backend for ActiveJob is
+The Maintenance Tasks framework relies on Active Job behind the scenes to run
+Tasks. The default queuing backend for Active Job is
 [asynchronous][async-adapter]. It is **strongly recommended** to change this to
 a persistent backend so that Task progress is not lost during code or
 infrastructure changes. For more information on configuring a queuing backend,
-take a look at the [ActiveJob documentation][active-job-docs].
+take a look at the [Active Job documentation][active-job-docs].
 
 [async-adapter]: https://api.rubyonrails.org/classes/ActiveJob/QueueAdapters/AsyncAdapter.html
 [active-job-docs]: https://guides.rubyonrails.org/active_job_basics.html#setting-the-backend
@@ -40,7 +40,7 @@ take a look at the [ActiveJob documentation][active-job-docs].
 A generator is provided to create tasks. Generate a new task by running:
 
 ```bash
-$ rails generate maintenance_tasks:task update_posts
+$ bin/rails generate maintenance_tasks:task update_posts
 ```
 
 This creates the task file `app/tasks/maintenance/update_posts_task.rb`.
@@ -57,6 +57,7 @@ Example:
 
 ```ruby
 # app/tasks/maintenance/update_posts_task.rb
+
 module Maintenance
   class UpdatePostsTask < MaintenanceTasks::Task
     def collection
@@ -68,7 +69,7 @@ module Maintenance
     end
 
     def process(post)
-      post.update!(content: 'New content!')
+      post.update!(content: "New content!")
     end
   end
 end
@@ -77,16 +78,16 @@ end
 ### Creating a CSV Task
 
 You can also write a Task that iterates on a CSV file. Note that writing CSV
-Tasks **requires ActiveStorage to be configured**. Ensure that the dependency
-is specified in your application's Gemfile, and that you've followed the
-[setup instuctions][setup].
+Tasks **requires Active Storage to be configured**. Ensure that the dependency
+is specified in your application's Gemfile, and that you've followed the [setup
+instuctions][setup].
 
 [setup]: https://edgeguides.rubyonrails.org/active_storage_overview.html#setup
 
 Generate a CSV Task by running:
 
 ```bash
-$ rails generate maintenance_tasks:task import_posts --csv
+$ bin/rails generate maintenance_tasks:task import_posts --csv
 ```
 
 The generated task is a subclass of `MaintenanceTasks::Task` that implements:
@@ -95,6 +96,7 @@ The generated task is a subclass of `MaintenanceTasks::Task` that implements:
 
 ```ruby
 # app/tasks/maintenance/import_posts_task.rb
+
 module Maintenance
   class ImportPostsTask < MaintenanceTasks::Task
     csv_collection
@@ -112,6 +114,52 @@ title,content
 My Title,Hello World!
 ```
 
+The files uploaded to your Active Storage service provider will be renamed
+to include an ISO8601 timestamp and the Task name in snake case format.
+The CSV is expected to have a trailing newline at the end of the file.
+
+### Processing Batch Collections
+
+The Maintenance Tasks gem supports processing Active Records in batches. This
+can reduce the number of calls your Task makes to the database. Use
+`ActiveRecord::Batches#in_batches` on the relation returned by your collection
+to specify that your Task should process batches instead of records. Active
+Record defaults to 1000 records by batch, but a custom size can be specified.
+
+```ruby
+# app/tasks/maintenance/update_posts_in_batches_task.rb
+
+module Maintenance
+  class UpdatePostsInBatchesTask < MaintenanceTasks::Task
+    def collection
+      Post.in_batches
+    end
+
+    def process(batch_of_posts)
+      batch_of_posts.update_all(content: "New content added on #{Time.now.utc}")
+    end
+  end
+end
+```
+
+Ensure that you've implemented the following methods:
+
+* `collection`: return an `ActiveRecord::Batches::BatchEnumerator`.
+* `process`: do the work of your Task on a batch (`ActiveRecord::Relation`).
+
+Note that `#count` is calculated automatically based on the number of batches in
+your collection, and your Task's progress will be displayed in terms of batches
+(not the number of records in the relation).
+
+**Important!** Batches should only be used if `#process` is performing a batch
+operation such as `#update_all` or `#delete_all`. If you need to iterate over
+individual records, you should define a collection that [returns an
+`ActiveRecord::Relation`](#creating-a-task). This uses batching internally, but
+loads the records with one SQL query. Conversely, batch collections load the
+primary keys of the records of the batch first, and then perform an additional
+query to load the records when calling `each` (or any `Enumerable` method)
+inside `#process`.
+
 ### Throttling
 
 Maintenance Tasks often modify a lot of data and can be taxing on your database.
@@ -122,6 +170,7 @@ Specify the throttle condition as a block:
 
 ```ruby
 # app/tasks/maintenance/update_posts_throttled_task.rb
+
 module Maintenance
   class UpdatePostsThrottledTask < MaintenanceTasks::Task
     throttle_on(backoff: 1.minute) do
@@ -145,13 +194,129 @@ end
 
 Note that it's up to you to define a throttling condition that makes sense for
 your app. Shopify implements `DatabaseStatus.healthy?` to check various MySQL
-metrics such as replication lag, DB threads, whether DB writes are available, 
+metrics such as replication lag, DB threads, whether DB writes are available,
 etc.
 
 Tasks can define multiple throttle conditions. Throttle conditions are inherited
 by descendants, and new conditions will be appended without impacting existing
 conditions.
 
+The backoff can also be specified as a proc:
+
+```ruby
+# app/tasks/maintenance/update_posts_throttled_task.rb
+
+module Maintenance
+  class UpdatePostsThrottledTask < MaintenanceTasks::Task
+    throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
+      DatabaseStatus.unhealthy?
+    end
+    ...
+  end
+end
+```
+### Custom Task Parameters
+
+Tasks may need additional information, supplied via parameters, to run.
+Parameters can be defined as Active Model Attributes in a Task, and then become
+accessible to any of Task's methods: `#collection`, `#count`, or `#process`.
+
+```ruby
+# app/tasks/maintenance/update_posts_via_params_task.rb
+
+module Maintenance
+  class UpdatePostsViaParamsTask < MaintenanceTasks::Task
+    attribute :updated_content, :string
+    validates :updated_content, presence: true
+
+    def collection
+      Post.all
+    end
+
+    def count
+      collection.count
+    end
+
+    def process(post)
+      post.update!(content: updated_content)
+    end
+  end
+end
+```
+
+Tasks can leverage Active Model Validations when defining parameters. Arguments
+supplied to a Task accepting parameters will be validated before the Task starts
+to run. Since arguments are specified in the user interface via text area
+inputs, it's important to check that they conform to the format your Task
+expects, and to sanitize any inputs if necessary.
+
+### Using Task Callbacks
+
+The Task provides callbacks that hook into its life cycle.
+
+Available callbacks are:
+
+* `after_start`
+* `after_pause`
+* `after_interrupt`
+* `after_cancel`
+* `after_complete`
+* `after_error`
+
+```ruby
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    after_start :notify
+
+    def notify
+      NotifyJob.perform_later(self.class.name)
+    end
+
+    # ...
+  end
+end
+```
+
+Note: The `after_error` callback is guaranteed to complete,
+so any exceptions raised in your callback code are ignored.
+If your `after_error` callback code can raise an exception,
+you'll need to rescue it and handle it appropriately
+within the callback.
+
+```ruby
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    after_error :dangerous_notify
+
+    def dangerous_notify
+      # This error is rescued in favour of the original error causing the error flow.
+      raise NotDeliveredError
+    end
+
+    # ...
+  end
+end
+```
+
+If any of the other callbacks cause an exception,
+it will be handled by the error handler,
+and will cause the task to stop running.
+
+Callback behaviour can be shared across all tasks using an initializer.
+
+```ruby
+# config/initializer/maintenance_tasks.rb
+Rails.autoloaders.main.on_load("MaintenanceTasks::Task") do
+  MaintenanceTasks::Task.class_eval do
+    after_start(:notify)
+
+    private
+
+    def notify; end
+  end
+end
+```
+
 ### Considerations when writing Tasks
 
 MaintenanceTasks relies on the queue adapter configured for your application to
@@ -184,7 +349,7 @@ Example:
 ```ruby
 # test/tasks/maintenance/update_posts_task_test.rb
 
-require 'test_helper'
+require "test_helper"
 
 module Maintenance
   class UpdatePostsTaskTest < ActiveSupport::TestCase
@@ -193,7 +358,7 @@ module Maintenance
 
       Maintenance::UpdatePostsTask.process(post)
 
-      assert_equal 'New content!', post.content
+      assert_equal "New content!", post.content
     end
   end
 end
@@ -206,20 +371,50 @@ takes a `CSV::Row` as an argument. You can pass a row, or a hash with string
 keys to `#process` from your test.
 
 ```ruby
-# app/tasks/maintenance/import_posts_task_test.rb
+# test/tasks/maintenance/import_posts_task_test.rb
+
+require "test_helper"
+
 module Maintenance
   class ImportPostsTaskTest < ActiveSupport::TestCase
     test "#process performs a task iteration" do
       assert_difference -> { Post.count } do
         Maintenance::UpdatePostsTask.process({
-          'title' => 'My Title',
-          'content' => 'Hello World!',
+          "title" => "My Title",
+          "content" => "Hello World!",
         })
       end
 
       post = Post.last
-      assert_equal 'My Title', post.title
-      assert_equal 'Hello World!', post.content
+      assert_equal "My Title", post.title
+      assert_equal "Hello World!", post.content
+    end
+  end
+end
+```
+
+### Writing tests for a Task with parameters
+
+Tests for tasks with parameters need to instatiate the task class in order to
+assign attributes. Once the task instance is setup, you may test `#process`
+normally.
+
+```ruby
+# test/tasks/maintenance/update_posts_via_params_task_test.rb
+
+require "test_helper"
+
+module Maintenance
+  class UpdatePostsViaParamsTaskTest < ActiveSupport::TestCase
+    setup do
+      @task = UpdatePostsViaParamsTask.new
+      @task.updated_content = "Testing"
+    end
+
+    test "#process performs a task iteration" do
+      assert_difference -> { Post.first.content } do
+        task.process(Post.first)
+      end
     end
   end
 end
@@ -238,13 +433,21 @@ $ bundle exec maintenance_tasks perform Maintenance::UpdatePostsTask
 To run a Task that processes CSVs from the command line, use the --csv option:
 
 ```bash
-$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv 'path/to/my_csv.csv'
+$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv "path/to/my_csv.csv"
+```
+
+To run a Task that takes arguments from the command line, use the --arguments
+option, passing arguments as a set of \<key>:\<value> pairs:
+
+```bash
+$ bundle exec maintenance_tasks perform Maintenance::ParamsTask \
+  --arguments post_ids:1,2,3 content:"Hello, World!"
 ```
 
 You can also run a Task in Ruby by sending `run` with a Task name to Runner:
 
 ```ruby
-MaintenanceTasks::Runner.run(name: 'Maintenance::UpdatePostsTask')
+MaintenanceTasks::Runner.run(name: "Maintenance::UpdatePostsTask")
 ```
 
 To run a Task that processes CSVs using the Runner, provide a Hash containing an
@@ -252,8 +455,18 @@ open IO object and a filename to `run`:
 
 ```ruby
 MaintenanceTasks::Runner.run(
-  name: 'Maintenance::ImportPostsTask'
-  csv_file: { io: File.open('path/to/my_csv.csv'), filename: 'my_csv.csv' }
+  name: "Maintenance::ImportPostsTask",
+  csv_file: { io: File.open("path/to/my_csv.csv"), filename: "my_csv.csv" }
+)
+```
+
+To run a Task that takes arguments using the Runner, provide a Hash containing
+the set of arguments (`{ parameter_name: argument_value }`) to `run`:
+
+```ruby
+MaintenanceTasks::Runner.run(
+  name: "Maintenance::ParamsTask",
+  arguments: { post_ids: "1,2,3" }
 )
 ```
 
@@ -266,8 +479,8 @@ a Task can be in:
 * **enqueued**: A Task that is waiting to be performed after a user has
   instructed it to run.
 * **running**: A Task that is currently being performed by a job worker.
-* **pausing**: A Task that was paused by a user, but needs to finish work
-  before stopping.
+* **pausing**: A Task that was paused by a user, but needs to finish work before
+  stopping.
 * **paused**: A Task that was paused by a user and is not performing. It can be
   resumed.
 * **interrupted**: A Task that has been momentarily interrupted by the job
@@ -341,9 +554,10 @@ you can define an error handler:
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.error_handler = ->(error, task_context, _errored_element) do
   Bugsnag.notify(error) do |notification|
-    notification.add_tab(:task, task_context)
+    notification.add_metadata(:task, task_context)
   end
 end
 ```
@@ -356,18 +570,18 @@ The error handler should be a lambda that accepts three arguments:
   * `task_name`: The name of the Task that errored
   * `started_at`: The time the Task started
   * `ended_at`: The time the Task errored
+
   Note that `task_context` may be empty if the Task produced an error before any
   context could be gathered (for example, if deserializing the job to process
   your Task failed).
-* `errored_element`: The element, if any, that was being processed when the
-  Task raised an exception. If you would like to pass this object to your
-  exception monitoring service, make sure you **sanitize the object** to avoid
-  leaking sensitive data and **convert it to a format** that is compatible with
-  your bug tracker. For example, Bugsnag only sends the id and class name of
-  ActiveRecord objects in order to protect sensitive data. CSV rows, on the
-  other hand, are converted to strings and passed raw to Bugsnag, so make sure
-  to filter any personal data from these objects before adding them to a
-  report.
+* `errored_element`: The element, if any, that was being processed when the Task
+  raised an exception. If you would like to pass this object to your exception
+  monitoring service, make sure you **sanitize the object** to avoid leaking
+  sensitive data and **convert it to a format** that is compatible with your bug
+  tracker. For example, Bugsnag only sends the id and class name of Active
+  Record objects in order to protect sensitive data. CSV rows, on the other
+  hand, are converted to strings and passed raw to Bugsnag, so make sure to
+  filter any personal data from these objects before adding them to a report.
 
 #### Customizing the maintenance tasks module
 
@@ -376,7 +590,8 @@ tasks will be placed.
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
-MaintenanceTasks.tasks_module = 'TaskModule'
+
+MaintenanceTasks.tasks_module = "TaskModule"
 ```
 
 If no value is specified, it will default to `Maintenance`.
@@ -389,9 +604,11 @@ maintenance tasks in your application.
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.job = 'CustomTaskJob'
 
 # app/jobs/custom_task_job.rb
+
 class CustomTaskJob < MaintenanceTasks::TaskJob
   queue_as :low_priority
 end
@@ -399,8 +616,8 @@ end
 
 The Job class **must inherit** from `MaintenanceTasks::TaskJob`.
 
-Note that `retry_on` is not supported for custom Job
-classes, so failed jobs cannot be retried.
+Note that `retry_on` is not supported for custom Job classes, so failed jobs
+cannot be retried.
 
 #### Customizing the rate at which task progress gets updated
 
@@ -410,6 +627,7 @@ task progress gets persisted to the database. It can be a `Numeric` value or an
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.ticker_delay = 2.seconds
 ```
 
@@ -424,6 +642,7 @@ key, as specified in your application's `config/storage.yml`:
 
 ```yaml
 # config/storage.yml
+
 user_data:
   service: GCS
   credentials: <%= Rails.root.join("path/to/user/data/keyfile.json") %>
@@ -439,23 +658,53 @@ internal:
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.active_storage_service = :internal
 ```
 
 There is no need to configure this option if your application uses only one
 storage service per environment.
 
+#### Customizing the backtrace cleaner
+
+`MaintenanceTasks.backtrace_cleaner` can be configured to specify a backtrace
+cleaner to use when a Task errors and the backtrace is cleaned and persisted.
+An `ActiveSupport::BacktraceCleaner` should be used.
+
+```ruby
+# config/initializers/maintenance_tasks.rb
+
+cleaner = ActiveSupport::BacktraceCleaner.new
+cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
+
+MaintenanceTasks.backtrace_cleaner = cleaner
+```
+
+If none is specified, the default `Rails.backtrace_cleaner` will be used to
+clean backtraces.
+
 ## Upgrading
 
 Use bundler to check for and upgrade to newer versions. After installing a new
 version, re-run the install command:
 
 ```bash
-$ rails generate maintenance_tasks:install
+$ bin/rails generate maintenance_tasks:install
 ```
 
 This ensures that new migrations are installed and run as well.
 
+**What if I've deleted my previous Maintenance Task migrations?**
+
+The install command will attempt to reinstall these old migrations and migrating
+the database will cause problems. Use `bin/rails
+maintenance_tasks:install:migrations` to copy the gem's migrations to your
+`db/migrate` folder. Check the release notes to see if any new migrations were
+added since your last gem upgrade.  Ensure that these are kept, but remove any
+migrations that already ran.
+
+Run the migrations using `bin/rails db:migrate`.
+
 ## Contributing
 
 Would you like to report an issue or contribute with code? We accept issues and
@@ -464,12 +713,6 @@ pull requests. You can find the contribution guidelines on
 
 [contributing]: https://github.com/Shopify/maintenance_tasks/blob/main/.github/CONTRIBUTING.md
 
-### Dependabot updates
-
-Whenever Dependabot creates a PR for a gem bump, check out the branch locally
-and run `bin/update-gemfile <gem>` to ensure all the gemfiles have the gem
-updated consistently.
-
 ## Releasing new versions
 
 Updates should be added to the latest draft release on GitHub as Pull Requests
@@ -478,7 +721,7 @@ are merged.
 Once a release is ready, follow these steps:
 
 * Update `spec.version` in `maintenance_tasks.gemspec`.
-* Run `bin/gemfile-update install` to bump the version in all the lockfiles.
+* Run `bundle install` to bump the `Gemfile.lock` version of the gem.
 * Open a PR and merge on approval.
 * Deploy via [Shipit][shipit] and see the new version on
   <https://rubygems.org/gems/maintenance_tasks>.

data/app/controllers/maintenance_tasks/tasks_controller.rb

@@ -24,14 +24,19 @@ module MaintenanceTasks
     end
 
     # Runs a given Task and redirects to the Task page.
-    def run
+    def run(&block)
       task = Runner.run(
         name: params.fetch(:id),
-        csv_file: params[:csv_file]
+        csv_file: params[:csv_file],
+        arguments: params.fetch(:task_arguments, {}).permit!.to_h,
+        &block
       )
       redirect_to(task_path(task))
     rescue ActiveRecord::RecordInvalid => error
       redirect_to(task_path(error.record.task_name), alert: error.message)
+    rescue ActiveRecord::ValueTooLong => error
+      task_name = params.fetch(:id)
+      redirect_to(task_path(task_name), alert: error.message)
     rescue Runner::EnqueuingError => error
       redirect_to(task_path(error.run.task_name), alert: error.message)
     end

data/app/helpers/maintenance_tasks/application_helper.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
+
 module MaintenanceTasks
   # Module for common view helpers.
   #

data/app/helpers/maintenance_tasks/tasks_helper.rb

@@ -63,20 +63,6 @@ module MaintenanceTasks
       tag.span(status.capitalize, class: ["tag"] + STATUS_COLOURS.fetch(status))
     end
 
-    # Returns the distance between now and the Run's expected completion time,
-    # if the Run has an estimated_completion_time.
-    #
-    # @param run [MaintenanceTasks::Run] the Run for which the estimated time to
-    #   completion is being calculated.
-    # return [String, nil] the distance in words, or nil if the Run has no
-    #   estimated completion time.
-    def estimated_time_to_completion(run)
-      estimated_completion_time = run.estimated_completion_time
-      if estimated_completion_time.present?
-        time_ago_in_words(estimated_completion_time)
-      end
-    end
-
     # Reports the approximate elapsed time a Run has been processed so far based
     # on the Run's time running attribute.
     #
@@ -114,5 +100,24 @@ module MaintenanceTasks
         only_path: true
       )
     end
+
+    # Return the appropriate field tag for the parameter
+    def parameter_field(form_builder, parameter_name)
+      case form_builder.object.class.attribute_types[parameter_name]
+      when ActiveModel::Type::Integer, ActiveModel::Type::Decimal,
+           ActiveModel::Type::Float
+        form_builder.number_field(parameter_name)
+      when ActiveModel::Type::DateTime
+        form_builder.datetime_field(parameter_name)
+      when ActiveModel::Type::Date
+        form_builder.date_field(parameter_name)
+      when ActiveModel::Type::Time
+        form_builder.time_field(parameter_name)
+      when ActiveModel::Type::Boolean
+        form_builder.check_box(parameter_name)
+      else
+        form_builder.text_area(parameter_name, class: "textarea")
+      end
+    end
   end
 end