maintenance_tasks 1.4.0 → 1.8.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6c3c4c94cf355432e0d558e68bfc3f275972164c8ed932d2505b7c6450f0d124
-  data.tar.gz: 1b80ad4b58b9546d5ab2af77d2d0e64f8bd8b72dc6641ff445d33a67dae03860
+  metadata.gz: e66a4f3f2755a31f9dac0125d41d63d5b1b8ca28a4f11642c8648d94dbc2a044
+  data.tar.gz: 1cf5d80050866d6e78d4cf392c9ff888953a3d15da8c19f83f511aaaf94640d2
 SHA512:
-  metadata.gz: 7402e24bfbc370c6af54b441d2b1050f252ee44f51ab11612875fed7fa6822645420422dc33405b1665596e66f85f8fe1f8b1e443d023d451a626913afe6dc98
-  data.tar.gz: 8f4666aba4abc6f01d77b238de6b6d511d24a5d4932f5ec2499f0e91faefe2e8a3b18435109864c821495b2efde464dd3212a76eae5276b4e622eb1cb6f7da3c
+  metadata.gz: ffc019d73a707936a483349862481e5a227e270be3578f859b7d385fa7d997d327082a6f090881dcb83306feaf496d36f03d75421533ac507d7472adc5cad991
+  data.tar.gz: ae2ca92ea5b989e11d6ffaa50bd6ce1e5fcd41f735f81e6261ae496b81de2b90e7da2f58fdc04803dc77d44f6c0a8814a86794413f2e7ce374aa557c7e0073fb

data/README.md

@@ -57,6 +57,7 @@ Example:
 
 ```ruby
 # app/tasks/maintenance/update_posts_task.rb
+
 module Maintenance
   class UpdatePostsTask < MaintenanceTasks::Task
     def collection
@@ -68,7 +69,7 @@ module Maintenance
     end
 
     def process(post)
-      post.update!(content: 'New content!')
+      post.update!(content: "New content!")
     end
   end
 end
@@ -78,8 +79,8 @@ end
 
 You can also write a Task that iterates on a CSV file. Note that writing CSV
 Tasks **requires Active Storage to be configured**. Ensure that the dependency
-is specified in your application's Gemfile, and that you've followed the
-[setup instuctions][setup].
+is specified in your application's Gemfile, and that you've followed the [setup
+instuctions][setup].
 
 [setup]: https://edgeguides.rubyonrails.org/active_storage_overview.html#setup
 
@@ -95,6 +96,7 @@ The generated task is a subclass of `MaintenanceTasks::Task` that implements:
 
 ```ruby
 # app/tasks/maintenance/import_posts_task.rb
+
 module Maintenance
   class ImportPostsTask < MaintenanceTasks::Task
     csv_collection
@@ -112,16 +114,21 @@ title,content
 My Title,Hello World!
 ```
 
+The files uploaded to your Active Storage service provider will be renamed
+to include an ISO8601 timestamp and the Task name in snake case format.
+The CSV is expected to have a trailing newline at the end of the file.
+
 ### Processing Batch Collections
 
 The Maintenance Tasks gem supports processing Active Records in batches. This
 can reduce the number of calls your Task makes to the database. Use
-`ActiveRecord::Batches#in_batches` on the relation returned by your collection to specify that your Task should process
-batches instead of records. Active Record defaults to 1000 records by batch, but a custom size can be
-specified.
+`ActiveRecord::Batches#in_batches` on the relation returned by your collection
+to specify that your Task should process batches instead of records. Active
+Record defaults to 1000 records by batch, but a custom size can be specified.
 
 ```ruby
 # app/tasks/maintenance/update_posts_in_batches_task.rb
+
 module Maintenance
   class UpdatePostsInBatchesTask < MaintenanceTasks::Task
     def collection
@@ -147,10 +154,41 @@ your collection, and your Task's progress will be displayed in terms of batches
 **Important!** Batches should only be used if `#process` is performing a batch
 operation such as `#update_all` or `#delete_all`. If you need to iterate over
 individual records, you should define a collection that [returns an
-`ActiveRecord::Relation`](#creating-a-task). This uses batching
-internally, but loads the records with one SQL query. Conversely, batch
-collections load the primary keys of the records of the batch first, and then perform an additional query to load the
-records when calling `each` (or any `Enumerable` method) inside `#process`.
+`ActiveRecord::Relation`](#creating-a-task). This uses batching internally, but
+loads the records with one SQL query. Conversely, batch collections load the
+primary keys of the records of the batch first, and then perform an additional
+query to load the records when calling `each` (or any `Enumerable` method)
+inside `#process`.
+
+### Tasks that don't need a Collection
+
+Sometimes, you might want to run a Task that performs a single operation, such
+as enqueuing another background job or hitting an external API. The gem supports
+collection-less tasks.
+
+Generate a collection-less Task by running:
+
+```bash
+$ bin/rails generate maintenance_tasks:task no_collection_task --no-collection
+```
+
+The generated task is a subclass of `MaintenanceTasks::Task` that implements:
+
+* `process`: do the work of your maintenance task
+
+```ruby
+# app/tasks/maintenance/no_collection_task.rb
+
+module Maintenance
+  class NoCollectionTask < MaintenanceTasks::Task
+    no_collection
+
+    def process
+      SomeAsyncJob.perform_later
+    end
+  end
+end
+```
 
 ### Throttling
 
@@ -162,6 +200,7 @@ Specify the throttle condition as a block:
 
 ```ruby
 # app/tasks/maintenance/update_posts_throttled_task.rb
+
 module Maintenance
   class UpdatePostsThrottledTask < MaintenanceTasks::Task
     throttle_on(backoff: 1.minute) do
@@ -192,15 +231,29 @@ Tasks can define multiple throttle conditions. Throttle conditions are inherited
 by descendants, and new conditions will be appended without impacting existing
 conditions.
 
+The backoff can also be specified as a proc:
+
+```ruby
+# app/tasks/maintenance/update_posts_throttled_task.rb
+
+module Maintenance
+  class UpdatePostsThrottledTask < MaintenanceTasks::Task
+    throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
+      DatabaseStatus.unhealthy?
+    end
+    ...
+  end
+end
+```
 ### Custom Task Parameters
 
 Tasks may need additional information, supplied via parameters, to run.
-Parameters can be defined as Active Model Attributes in a Task, and then
-become accessible to any of Task's methods: `#collection`, `#count`, or
-`#process`.
+Parameters can be defined as Active Model Attributes in a Task, and then become
+accessible to any of Task's methods: `#collection`, `#count`, or `#process`.
 
 ```ruby
 # app/tasks/maintenance/update_posts_via_params_task.rb
+
 module Maintenance
   class UpdatePostsViaParamsTask < MaintenanceTasks::Task
     attribute :updated_content, :string
@@ -227,6 +280,73 @@ to run. Since arguments are specified in the user interface via text area
 inputs, it's important to check that they conform to the format your Task
 expects, and to sanitize any inputs if necessary.
 
+### Using Task Callbacks
+
+The Task provides callbacks that hook into its life cycle.
+
+Available callbacks are:
+
+* `after_start`
+* `after_pause`
+* `after_interrupt`
+* `after_cancel`
+* `after_complete`
+* `after_error`
+
+```ruby
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    after_start :notify
+
+    def notify
+      NotifyJob.perform_later(self.class.name)
+    end
+
+    # ...
+  end
+end
+```
+
+Note: The `after_error` callback is guaranteed to complete,
+so any exceptions raised in your callback code are ignored.
+If your `after_error` callback code can raise an exception,
+you'll need to rescue it and handle it appropriately
+within the callback.
+
+```ruby
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    after_error :dangerous_notify
+
+    def dangerous_notify
+      # This error is rescued in favour of the original error causing the error flow.
+      raise NotDeliveredError
+    end
+
+    # ...
+  end
+end
+```
+
+If any of the other callbacks cause an exception,
+it will be handled by the error handler,
+and will cause the task to stop running.
+
+Callback behaviour can be shared across all tasks using an initializer.
+
+```ruby
+# config/initializer/maintenance_tasks.rb
+Rails.autoloaders.main.on_load("MaintenanceTasks::Task") do
+  MaintenanceTasks::Task.class_eval do
+    after_start(:notify)
+
+    private
+
+    def notify; end
+  end
+end
+```
+
 ### Considerations when writing Tasks
 
 MaintenanceTasks relies on the queue adapter configured for your application to
@@ -259,7 +379,7 @@ Example:
 ```ruby
 # test/tasks/maintenance/update_posts_task_test.rb
 
-require 'test_helper'
+require "test_helper"
 
 module Maintenance
   class UpdatePostsTaskTest < ActiveSupport::TestCase
@@ -268,7 +388,7 @@ module Maintenance
 
       Maintenance::UpdatePostsTask.process(post)
 
-      assert_equal 'New content!', post.content
+      assert_equal "New content!", post.content
     end
   end
 end
@@ -281,20 +401,50 @@ takes a `CSV::Row` as an argument. You can pass a row, or a hash with string
 keys to `#process` from your test.
 
 ```ruby
-# app/tasks/maintenance/import_posts_task_test.rb
+# test/tasks/maintenance/import_posts_task_test.rb
+
+require "test_helper"
+
 module Maintenance
   class ImportPostsTaskTest < ActiveSupport::TestCase
     test "#process performs a task iteration" do
       assert_difference -> { Post.count } do
         Maintenance::UpdatePostsTask.process({
-          'title' => 'My Title',
-          'content' => 'Hello World!',
+          "title" => "My Title",
+          "content" => "Hello World!",
         })
       end
 
       post = Post.last
-      assert_equal 'My Title', post.title
-      assert_equal 'Hello World!', post.content
+      assert_equal "My Title", post.title
+      assert_equal "Hello World!", post.content
+    end
+  end
+end
+```
+
+### Writing tests for a Task with parameters
+
+Tests for tasks with parameters need to instatiate the task class in order to
+assign attributes. Once the task instance is setup, you may test `#process`
+normally.
+
+```ruby
+# test/tasks/maintenance/update_posts_via_params_task_test.rb
+
+require "test_helper"
+
+module Maintenance
+  class UpdatePostsViaParamsTaskTest < ActiveSupport::TestCase
+    setup do
+      @task = UpdatePostsViaParamsTask.new
+      @task.updated_content = "Testing"
+    end
+
+    test "#process performs a task iteration" do
+      assert_difference -> { Post.first.content } do
+        task.process(Post.first)
+      end
     end
   end
 end
@@ -313,20 +463,21 @@ $ bundle exec maintenance_tasks perform Maintenance::UpdatePostsTask
 To run a Task that processes CSVs from the command line, use the --csv option:
 
 ```bash
-$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv 'path/to/my_csv.csv'
+$ bundle exec maintenance_tasks perform Maintenance::ImportPostsTask --csv "path/to/my_csv.csv"
 ```
 
 To run a Task that takes arguments from the command line, use the --arguments
-option, passing arguments as a set of <key>:<value> pairs:
+option, passing arguments as a set of \<key>:\<value> pairs:
 
 ```bash
-$ bundle exec maintenance_tasks perform Maintenance::ParamsTask --arguments post_ids:1,2,3 content:"Hello, World!"
+$ bundle exec maintenance_tasks perform Maintenance::ParamsTask \
+  --arguments post_ids:1,2,3 content:"Hello, World!"
 ```
 
 You can also run a Task in Ruby by sending `run` with a Task name to Runner:
 
 ```ruby
-MaintenanceTasks::Runner.run(name: 'Maintenance::UpdatePostsTask')
+MaintenanceTasks::Runner.run(name: "Maintenance::UpdatePostsTask")
 ```
 
 To run a Task that processes CSVs using the Runner, provide a Hash containing an
@@ -334,8 +485,8 @@ open IO object and a filename to `run`:
 
 ```ruby
 MaintenanceTasks::Runner.run(
-  name: 'Maintenance::ImportPostsTask'
-  csv_file: { io: File.open('path/to/my_csv.csv'), filename: 'my_csv.csv' }
+  name: "Maintenance::ImportPostsTask",
+  csv_file: { io: File.open("path/to/my_csv.csv"), filename: "my_csv.csv" }
 )
 ```
 
@@ -358,8 +509,8 @@ a Task can be in:
 * **enqueued**: A Task that is waiting to be performed after a user has
   instructed it to run.
 * **running**: A Task that is currently being performed by a job worker.
-* **pausing**: A Task that was paused by a user, but needs to finish work
-  before stopping.
+* **pausing**: A Task that was paused by a user, but needs to finish work before
+  stopping.
 * **paused**: A Task that was paused by a user and is not performing. It can be
   resumed.
 * **interrupted**: A Task that has been momentarily interrupted by the job
@@ -433,9 +584,10 @@ you can define an error handler:
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.error_handler = ->(error, task_context, _errored_element) do
   Bugsnag.notify(error) do |notification|
-    notification.add_tab(:task, task_context)
+    notification.add_metadata(:task, task_context)
   end
 end
 ```
@@ -448,18 +600,18 @@ The error handler should be a lambda that accepts three arguments:
   * `task_name`: The name of the Task that errored
   * `started_at`: The time the Task started
   * `ended_at`: The time the Task errored
+
   Note that `task_context` may be empty if the Task produced an error before any
   context could be gathered (for example, if deserializing the job to process
   your Task failed).
-* `errored_element`: The element, if any, that was being processed when the
-  Task raised an exception. If you would like to pass this object to your
-  exception monitoring service, make sure you **sanitize the object** to avoid
-  leaking sensitive data and **convert it to a format** that is compatible with
-  your bug tracker. For example, Bugsnag only sends the id and class name of
-  Active Record objects in order to protect sensitive data. CSV rows, on the
-  other hand, are converted to strings and passed raw to Bugsnag, so make sure
-  to filter any personal data from these objects before adding them to a
-  report.
+* `errored_element`: The element, if any, that was being processed when the Task
+  raised an exception. If you would like to pass this object to your exception
+  monitoring service, make sure you **sanitize the object** to avoid leaking
+  sensitive data and **convert it to a format** that is compatible with your bug
+  tracker. For example, Bugsnag only sends the id and class name of Active
+  Record objects in order to protect sensitive data. CSV rows, on the other
+  hand, are converted to strings and passed raw to Bugsnag, so make sure to
+  filter any personal data from these objects before adding them to a report.
 
 #### Customizing the maintenance tasks module
 
@@ -468,7 +620,8 @@ tasks will be placed.
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
-MaintenanceTasks.tasks_module = 'TaskModule'
+
+MaintenanceTasks.tasks_module = "TaskModule"
 ```
 
 If no value is specified, it will default to `Maintenance`.
@@ -481,9 +634,11 @@ maintenance tasks in your application.
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.job = 'CustomTaskJob'
 
 # app/jobs/custom_task_job.rb
+
 class CustomTaskJob < MaintenanceTasks::TaskJob
   queue_as :low_priority
 end
@@ -491,8 +646,8 @@ end
 
 The Job class **must inherit** from `MaintenanceTasks::TaskJob`.
 
-Note that `retry_on` is not supported for custom Job
-classes, so failed jobs cannot be retried.
+Note that `retry_on` is not supported for custom Job classes, so failed jobs
+cannot be retried.
 
 #### Customizing the rate at which task progress gets updated
 
@@ -502,6 +657,7 @@ task progress gets persisted to the database. It can be a `Numeric` value or an
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.ticker_delay = 2.seconds
 ```
 
@@ -516,6 +672,7 @@ key, as specified in your application's `config/storage.yml`:
 
 ```yaml
 # config/storage.yml
+
 user_data:
   service: GCS
   credentials: <%= Rails.root.join("path/to/user/data/keyfile.json") %>
@@ -531,12 +688,31 @@ internal:
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+
 MaintenanceTasks.active_storage_service = :internal
 ```
 
 There is no need to configure this option if your application uses only one
 storage service per environment.
 
+#### Customizing the backtrace cleaner
+
+`MaintenanceTasks.backtrace_cleaner` can be configured to specify a backtrace
+cleaner to use when a Task errors and the backtrace is cleaned and persisted.
+An `ActiveSupport::BacktraceCleaner` should be used.
+
+```ruby
+# config/initializers/maintenance_tasks.rb
+
+cleaner = ActiveSupport::BacktraceCleaner.new
+cleaner.add_silencer { |line| line =~ /ignore_this_dir/ }
+
+MaintenanceTasks.backtrace_cleaner = cleaner
+```
+
+If none is specified, the default `Rails.backtrace_cleaner` will be used to
+clean backtraces.
+
 ## Upgrading
 
 Use bundler to check for and upgrade to newer versions. After installing a new
@@ -548,6 +724,17 @@ $ bin/rails generate maintenance_tasks:install
 
 This ensures that new migrations are installed and run as well.
 
+**What if I've deleted my previous Maintenance Task migrations?**
+
+The install command will attempt to reinstall these old migrations and migrating
+the database will cause problems. Use `bin/rails
+maintenance_tasks:install:migrations` to copy the gem's migrations to your
+`db/migrate` folder. Check the release notes to see if any new migrations were
+added since your last gem upgrade.  Ensure that these are kept, but remove any
+migrations that already ran.
+
+Run the migrations using `bin/rails db:migrate`.
+
 ## Contributing
 
 Would you like to report an issue or contribute with code? We accept issues and
@@ -564,6 +751,7 @@ are merged.
 Once a release is ready, follow these steps:
 
 * Update `spec.version` in `maintenance_tasks.gemspec`.
+* Run `bundle install` to bump the `Gemfile.lock` version of the gem.
 * Open a PR and merge on approval.
 * Deploy via [Shipit][shipit] and see the new version on
   <https://rubygems.org/gems/maintenance_tasks>.

data/app/controllers/maintenance_tasks/tasks_controller.rb

@@ -24,11 +24,12 @@ module MaintenanceTasks
     end
 
     # Runs a given Task and redirects to the Task page.
-    def run
+    def run(&block)
       task = Runner.run(
         name: params.fetch(:id),
         csv_file: params[:csv_file],
         arguments: params.fetch(:task_arguments, {}).permit!.to_h,
+        &block
       )
       redirect_to(task_path(task))
     rescue ActiveRecord::RecordInvalid => error

data/app/helpers/maintenance_tasks/application_helper.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
+
 module MaintenanceTasks
   # Module for common view helpers.
   #

data/app/helpers/maintenance_tasks/tasks_helper.rb

@@ -100,5 +100,24 @@ module MaintenanceTasks
         only_path: true
       )
     end
+
+    # Return the appropriate field tag for the parameter
+    def parameter_field(form_builder, parameter_name)
+      case form_builder.object.class.attribute_types[parameter_name]
+      when ActiveModel::Type::Integer, ActiveModel::Type::Decimal,
+           ActiveModel::Type::Float
+        form_builder.number_field(parameter_name)
+      when ActiveModel::Type::DateTime
+        form_builder.datetime_field(parameter_name)
+      when ActiveModel::Type::Date
+        form_builder.date_field(parameter_name)
+      when ActiveModel::Type::Time
+        form_builder.time_field(parameter_name)
+      when ActiveModel::Type::Boolean
+        form_builder.check_box(parameter_name)
+      else
+        form_builder.text_area(parameter_name, class: "textarea")
+      end
+    end
   end
 end

data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
+
 module MaintenanceTasks
   # Concern that holds the behaviour of the job that runs the tasks. It is
   # included in {TaskJob} and if MaintenanceTasks.job is overridden, it must be
@@ -11,8 +12,8 @@ module MaintenanceTasks
       before_perform(:before_perform)
 
       on_start(:on_start)
-      on_complete(:on_complete)
       on_shutdown(:on_shutdown)
+      on_complete(:on_complete)
 
       after_perform(:after_perform)
 
@@ -35,6 +36,8 @@ module MaintenanceTasks
       @enumerator = nil
 
       collection_enum = case collection
+      when :no_collection
+        enumerator_builder.build_once_enumerator(cursor: nil)
       when ActiveRecord::Relation
         enumerator_builder.active_record_on_records(collection, cursor: cursor)
       when ActiveRecord::Batches::BatchEnumerator
@@ -62,9 +65,16 @@ module MaintenanceTasks
           Array, or CSV.
         MSG
       end
+      throttle_enumerator(collection_enum)
+    end
 
+    def throttle_enumerator(collection_enum)
       @task.throttle_conditions.reduce(collection_enum) do |enum, condition|
-        enumerator_builder.build_throttle_enumerator(enum, **condition)
+        enumerator_builder.build_throttle_enumerator(
+          enum,
+          throttle_on: condition[:throttle_on],
+          backoff: condition[:backoff].call
+        )
       end
     end
 
@@ -81,7 +91,11 @@ module MaintenanceTasks
     end
 
     def task_iteration(input)
-      @task.process(input)
+      if @task.no_collection?
+        @task.process
+      else
+        @task.process(input)
+      end
     rescue => error
       @errored_element = input
       raise error
@@ -90,12 +104,11 @@ module MaintenanceTasks
     def before_perform
       @run = arguments.first
       @task = @run.task
-      if @task.respond_to?(:csv_content=)
+      if @task.has_csv_content?
         @task.csv_content = @run.csv_file.download
       end
-      @run.job_id = job_id
 
-      @run.running! unless @run.stopping?
+      @run.running
 
       @ticker = Ticker.new(MaintenanceTasks.ticker_delay) do |ticks, duration|
         @run.persist_progress(ticks, duration)
@@ -105,26 +118,19 @@ module MaintenanceTasks
     def on_start
       count = @task.count
       count = @enumerator&.size if count == :no_count
-      @run.update!(started_at: Time.now, tick_total: count)
-    end
-
-    def on_complete
-      @run.status = :succeeded
-      @run.ended_at = Time.now
+      @run.start(count)
     end
 
     def on_shutdown
-      if @run.cancelling?
-        @run.status = :cancelled
-        @run.ended_at = Time.now
-      else
-        @run.status = @run.pausing? ? :paused : :interrupted
-        @run.cursor = cursor_position
-      end
-
+      @run.job_shutdown
+      @run.cursor = cursor_position
       @ticker.persist
     end
 
+    def on_complete
+      @run.complete
+    end
+
     # We are reopening a private part of Job Iteration's API here, so we should
     # ensure the method is still defined upstream. This way, in the case where
     # the method changes upstream, we catch it at load time instead of at
@@ -143,7 +149,7 @@ module MaintenanceTasks
     end
 
     def after_perform
-      @run.save!
+      @run.persist_transition
       if defined?(@reenqueue_iteration_job) && @reenqueue_iteration_job
         reenqueue_iteration_job(should_ignore: false)
       end
@@ -164,6 +170,7 @@ module MaintenanceTasks
         task_context = {}
       end
       errored_element = @errored_element if defined?(@errored_element)
+    ensure
       MaintenanceTasks.error_handler.call(error, task_context, errored_element)
     end
   end

data/app/models/maintenance_tasks/application_record.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
+
 module MaintenanceTasks
   # Base class for all records used by this engine.
   #