maintenance_tasks 2.10.1 → 2.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4b660fdfcef27420bf9b08ddc50c9cfac5ff0977c4435b54ecd2b468a3e4fde8
-  data.tar.gz: 30bac0da63839406dc623d07c85904b7a727d3af5c177736cbd72ea828ca41c9
+  metadata.gz: 8402f1f4b8919e892a6334373ad397d49b5c6c78893f2f107b990b90653435d1
+  data.tar.gz: 1459d5b92233a1196aeb83c80bec9da525f04e975a50f6a415cdcd8dc647bd59
 SHA512:
-  metadata.gz: 071dafa23a89bd949e40f10ef199ae892d5ae81ca2ad1e15e6c7de0260dafbd55f735d67c310f5de7ef499991982eb3bad69fce4024f8f8761e4357ac3fdc42b
-  data.tar.gz: 77265626bdeafaa6486a3843846d519168be0712399ebe3cd7a4ecbb42f5888f208f566346d6d628643ee0dcf06ff1c4b914f2ab3ef9ec73f60b9444ef3f05c0
+  metadata.gz: 3dff16b85a650f4d73cf79df20267d1278b1181f2ba6717742581909ebfd39145dfdd9b9cf1453edaa785474fb6f7dda87af2bad534e6398bf010945b02f2da3
+  data.tar.gz: c7b75c1edc684fad66c5080ee3642fe469038e1f4ab2329ead0fd3c65e5f0ca2a940250dc988658dce672e2de0d056741b9a317310afa23165ba27794722ce1d

data/README.md

@@ -74,9 +74,11 @@ The generator creates and runs a migration to add the necessary table to your
 database. It also mounts Maintenance Tasks in your `config/routes.rb`. By
 default the web UI can be accessed in the new `/maintenance_tasks` path.
 
-In case you use an exception reporting service (e.g. Bugsnag) you might want to
-define an error handler. See [Customizing the error
-handler](#customizing-the-error-handler) for more information.
+This gem uses the [Rails Error Reporter][rails-error-reporting] to report errors.
+If you are using a bug tracking service you may want to subscribe to the
+reporter. See [Reporting Errors](#reporting-errors) for more information.
+
+[rails-error-reporting]: https://guides.rubyonrails.org/error_reporting.html
 
 ### Active Job Dependency
 
@@ -114,12 +116,12 @@ The typical Maintenance Tasks workflow is as follows:
 1. [Generate a class describing the Task](#creating-a-task) and the work to be
    done.
 2. Run the Task
-    - either by [using the included web UI](#running-a-task-from-the-web-ui),
-    - or by [using the command line](#running-a-task-from-the-command-line),
-    - or by [using Ruby](#running-a-task-from-ruby).
+   - either by [using the included web UI](#running-a-task-from-the-web-ui),
+   - or by [using the command line](#running-a-task-from-the-command-line),
+   - or by [using Ruby](#running-a-task-from-ruby).
 3. [Monitor the Task](#monitoring-your-tasks-status)
-    - either by using the included web UI,
-    - or by manually checking your task’s run’s status in your database.
+   - either by using the included web UI,
+   - or by manually checking your task’s run’s status in your database.
 4. Optionally, delete the Task code if you no longer need it.
 
 ### Creating a Task
@@ -168,7 +170,8 @@ end
 When processing records from an Active Record Relation, records are fetched in
 batches internally, and then each record is passed to the `#process` method.
 Maintenance Tasks will query the database to fetch records in batches of 100 by
-default, but the batch size can be modified using the `collection_batch_size` macro:
+default, but the batch size can be modified using the `collection_batch_size`
+macro:
 
 ```ruby
 # app/tasks/maintenance/update_posts_task.rb
@@ -502,13 +505,44 @@ set of values will be used to populate a dropdown in the user interface. The
 following types are supported:
 
 * Arrays
-* Procs and lambdas that optionally accept the Task instance, and return an Array.
-* Callable objects that receive one argument, the Task instance, and return an Array.
+* Procs and lambdas that optionally accept the Task instance, and return an
+  Array.
+* Callable objects that receive one argument, the Task instance, and return an
+  Array.
 * Methods that return an Array, called on the Task instance.
 
 For enumerables that don't match the supported types, a text field will be
 rendered instead.
 
+### Masking Task Parameters
+
+Task attributes can be masked in the UI by adding `mask_attribute` class method
+in the task class. This will replace the value in the arguments list with
+`[FILTERED]` in the UI.
+
+```ruby
+# app/tasks/maintenance/sensitive_params_task.rb
+
+module Maintenance
+  class SensitiveParamsTask < MaintenanceTasks::Task
+    attribute :sensitive_content, :string
+
+    mask_attribute :sensitive_content
+  end
+end
+```
+
+If you have any filtered parameters in the global [Rails parameter
+filter][rails-parameter-filter], they will be automatically taken into account
+when masking the parameters, which means that you can mask parameters across all
+tasks by adding them to the global rails parameters filter.
+
+[rails-parameter-filter]:https://guides.rubyonrails.org/configuring.html#config-filter-parameters
+
+```ruby
+Rails.application.config.filter_parameters += %i[token]
+```
+
 ### Custom cursor columns to improve performance
 
 The [job-iteration gem][job-iteration], on which this gem depends, adds an
@@ -899,10 +933,10 @@ a Task can be in:
 
 The Maintenance Tasks engine uses Rails sessions for flash messages and storing
 the CSRF token. For the engine to work in an API-only Rails application, you
-need to add a [session middleware][] and the `ActionDispatch::Flash`
-middleware. The engine also defines a strict [Content Security Policy][], make
-sure to include `ActionDispatch::ContentSecurityPolicy::Middleware` in your
-app's middleware stack to ensure the CSP is delivered to the user's browser.
+need to add a [session middleware][] and the `ActionDispatch::Flash` middleware.
+The engine also defines a strict [Content Security Policy][], make sure to
+include `ActionDispatch::ContentSecurityPolicy::Middleware` in your app's
+middleware stack to ensure the CSP is delivered to the user's browser.
 
 [session middleware]: https://guides.rubyonrails.org/api_app.html#using-session-middlewares
 [Content Security Policy]: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
@@ -926,8 +960,8 @@ module YourApplication
 end
 ```
 
-You can read more in the [Using Rails for API-only Applications][rails api] Rails
-guide.
+You can read more in the [Using Rails for API-only Applications][rails api]
+Rails guide.
 
 [rails api]: https://guides.rubyonrails.org/api_app.html
 
@@ -944,9 +978,9 @@ infrastructure or code changes.
 This means a Task can safely be interrupted, re-enqueued and resumed without any
 intervention at the end of an iteration, after the `process` method returns.
 
-By default, a running Task will be interrupted after running for more 5 minutes.
-This is [configured in the `job-iteration` gem][max-job-runtime] and can be
-tweaked in an initializer if necessary.
+By default, a running Task will be interrupted after running for more than 5
+minutes. This is [configured in the `job-iteration` gem][max-job-runtime] and
+can be tweaked in an initializer if necessary.
 
 [max-job-runtime]: https://github.com/Shopify/job-iteration/blob/-/guides/best-practices.md#max-job-runtime
 
@@ -988,44 +1022,89 @@ If you are stuck in `pausing` and wish to preserve your tasks's position
 There are a few configurable options for the gem. Custom configurations should
 be placed in a `maintenance_tasks.rb` initializer.
 
-#### Customizing the error handler
+#### Reporting errors
 
 Exceptions raised while a Task is performing are rescued and information about
 the error is persisted and visible in the UI.
 
-If you want to integrate with an exception monitoring service (e.g. Bugsnag),
-you can define an error handler:
+Errors are also sent to `Rails.error.report`, which can be configured by
+your application. See the [Error Reporting in Rails
+Applications][rails-error-reporting] guide for more details.
+
+Reports to the error reporter will contain the following data:
+
+* `error`: The exception that was raised.
+* `context`: A hash with additional information about the Task and the error:
+   * `task_name`: The name of the Task that errored
+   * `started_at`: The time the Task started
+   * `ended_at`: The time the Task errored
+   * `run_id`: The id of the errored Task run
+   * `tick_count`: The tick count at the time of the error
+   * `errored_element`: The element, if any, that was being processed when the
+* `source`: This will be `maintenance-tasks`
+* `handled`: the value of `MaintenanceTasks.report_errors_as_handled` (default `true`, see below)
+
+Note that `context` may be empty if the Task produced an error before any
+context could be gathered (for example, if deserializing the job to process your
+Task failed).
+
+Here's an example custom subscriber to the Rails error reporter for integrating
+with an exception monitoring service (Bugsnag):
 
 ```ruby
 # config/initializers/maintenance_tasks.rb
+MaintenanceTasks.report_errors_as_handled = false
+
+class MaintenanceTasksErrorSubscriber
+  def report(error, handled:, severity:, context:, source: nil)
+    return unless source == "maintenance-tasks"
 
-MaintenanceTasks.error_handler = ->(error, task_context, _errored_element) do
-  Bugsnag.notify(error) do |notification|
-    notification.add_metadata(:task, task_context)
+    unless handled
+      Bugsnag.notify(error) do |notification|
+        notification.add_metadata(:task, context)
+      end
+    else
+      Rails.logger.info(error)
+    end
   end
 end
+
+Rails.error.subscribe(MaintenanceTasksErrorSubscriber.new)
 ```
 
-The error handler should be a lambda that accepts three arguments:
+`MaintenanceTasks.report_errors_as_handled` determines the value for `handled` in this example.
+By default (for backwards compatibility) this is `true`.
+Setting this to `false` provides more accurate error reporting as it allows to distinguish between
+expected (e.g., via `report_on`) and unexpected errors in error subscribers.
+`false` will be the default in v3.0.
 
-* `error`: The exception that was raised.
-* `task_context`: A hash with additional information about the Task and the
-  error:
-  * `task_name`: The name of the Task that errored
-  * `started_at`: The time the Task started
-  * `ended_at`: The time the Task errored
-
-  Note that `task_context` may be empty if the Task produced an error before any
-  context could be gathered (for example, if deserializing the job to process
-  your Task failed).
-* `errored_element`: The element, if any, that was being processed when the Task
-  raised an exception. If you would like to pass this object to your exception
-  monitoring service, make sure you **sanitize the object** to avoid leaking
-  sensitive data and **convert it to a format** that is compatible with your bug
-  tracker. For example, Bugsnag only sends the id and class name of Active
-  Record objects in order to protect sensitive data. CSV rows, on the other
-  hand, are converted to strings and passed raw to Bugsnag, so make sure to
-  filter any personal data from these objects before adding them to a report.
+#### Reporting errors during iteration
+
+By default, errors raised during task iteration will be raised to the
+application and iteration will stop. However, you may want to handle some errors
+and continue iteration. `MaintenanceTasks::Task.report_on` can be used to rescue
+certain exceptions and report them to the Rails error reporter. Any keyword
+arguments are passed to
+[ActiveSupport::ErrorReporter#report][as-error-reporter-report]:
+
+[as-error-reporter-report]: https://api.rubyonrails.org/classes/ActiveSupport/ErrorReporter.html#method-i-report
+
+```ruby
+class MyTask < MaintenanceTasks::Task
+  report_on(MyException, OtherException, severity: :info, context: {task_name: "my_task"})
+end
+```
+
+`MaintenanceTasks::Task` also includes `ActiveSupport::Rescuable` which you can
+use to implement custom error handling.
+
+```ruby
+class MyTask < MaintenanceTasks::Task
+  rescue_from(MyException) do |exception|
+    handle(exception)
+  end
+end
+```
 
 #### Customizing the maintenance tasks module
 
@@ -1188,6 +1267,42 @@ The value for `MaintenanceTasks.stuck_task_duration` must be an
 `ActiveSupport::Duration`. If no value is specified, it will default to 5
 minutes.
 
+#### Configure status reload frequency
+
+`MaintenanceTasks.status_reload_frequency` can be configured to specify how often
+the run status should be reloaded during iteration. By default, the status is
+reloaded every second, but this can be increased to improve performance. Note that increasing the reload interval impacts how quickly
+your task will stop if it is paused or interrupted.
+
+```ruby
+# config/initializers/maintenance_tasks.rb
+MaintenanceTasks.status_reload_frequency = 10.seconds  # Reload status every 10 seconds
+```
+
+Individual tasks can also override this setting using the `reload_status_every` method:
+
+```ruby
+# app/tasks/maintenance/update_posts_task.rb
+
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    # Reload status every 5 seconds instead of the global default
+    reload_status_every(5.seconds)
+
+    def collection
+      Post.all
+    end
+
+    def process(post)
+      post.update!(content: "New content!")
+    end
+  end
+end
+```
+
+This optimization can significantly reduce database queries, especially for short iterations.
+This is especially useful if the task doesn't need to check for cancellation/pausing very often.
+
 #### Metadata
 
 `MaintenanceTasks.metadata` can be configured to specify a proc from which to

data/app/controllers/maintenance_tasks/application_controller.rb

@@ -8,16 +8,26 @@ module MaintenanceTasks
     BULMA_CDN = "https://cdn.jsdelivr.net"
 
     content_security_policy do |policy|
-      policy.style_src(
+      policy.style_src_elem(
         BULMA_CDN,
-        # ruby syntax highlighting
-        "'sha256-y9V0na/WU44EUNI/HDP7kZ7mfEci4PAOIjYOOan6JMA='",
+        # <style> tag in app/views/layouts/maintenance_tasks/application.html.erb
+        "'sha256-WHHDQLdkleXnAN5zs0GDXC5ls41CHUaVsJtVpaNx+EM='",
       )
-      policy.script_src(
-        # page refresh script
+      capybara_lockstep_scripts = [
+        "'sha256-1AoN3ZtJC5OvqkMgrYvhZjp4kI8QjJjO7TAyKYiDw+U='",
+        "'sha256-QVSzZi6ZsX/cu4h+hIs1iVivG1BxUmJggiEsGDIXBG0='", # with debug on
+      ] if defined?(Capybara::Lockstep)
+      policy.script_src_elem(
+        # <script> tag in app/views/layouts/maintenance_tasks/application.html.erb
         "'sha256-NiHKryHWudRC2IteTqmY9v1VkaDUA/5jhgXkMTkgo2w='",
+        # <script> tag for capybara-lockstep
+        *capybara_lockstep_scripts,
       )
+
+      policy.require_trusted_types_for # disable because we use new DOMParser().parseFromString
       policy.frame_ancestors(:self)
+      policy.connect_src(:self)
+      policy.form_action(:self)
     end
 
     protect_from_forgery with: :exception

data/app/helpers/maintenance_tasks/application_helper.rb

@@ -14,7 +14,7 @@ module MaintenanceTasks
     # @param datetime [ActiveSupport::TimeWithZone] the time to be presented.
     # @return [String] the HTML to render with the relative datetime in words.
     def time_ago(datetime)
-      time_tag(datetime, title: datetime.utc.iso8601, class: "is-clickable") do
+      time_tag(datetime, title: datetime.utc, class: "is-clickable") do
         time_ago_in_words(datetime) + " ago"
       end
     end

data/app/helpers/maintenance_tasks/tasks_helper.rb

@@ -47,7 +47,7 @@ module MaintenanceTasks
       progress_bar = tag.progress(
         value: progress.value,
         max: progress.max,
-        class: ["progress"] + STATUS_COLOURS.fetch(run.status),
+        class: ["progress", "mt-4"] + STATUS_COLOURS.fetch(run.status),
       )
       progress_text = tag.p(tag.i(progress.text))
       tag.div(progress_bar + progress_text, class: "block")
@@ -60,7 +60,10 @@ module MaintenanceTasks
     # @return [String] the span element containing the status, with the
     #   appropriate tag class attached.
     def status_tag(status)
-      tag.span(status.capitalize, class: ["tag"] + STATUS_COLOURS.fetch(status))
+      tag.span(
+        status.capitalize,
+        class: ["tag", "has-text-weight-medium", "pr-2", "mr-4"] + STATUS_COLOURS.fetch(status),
+      )
     end
 
     # Reports the approximate elapsed time a Run has been processed so far based
@@ -101,7 +104,7 @@ module MaintenanceTasks
       )
     end
 
-    # Resolves values covered by the inclusion validator for a task attribute.
+    # Resolves values covered by the inclusion validator for a Task attribute.
     # Supported option types:
     # - Arrays
     # - Procs and lambdas that optionally accept the Task instance, and return an Array.
@@ -112,7 +115,7 @@ module MaintenanceTasks
     #
     # Returned values are used to populate a dropdown list of options.
     #
-    # @param task_class [Class<Task>] The task class for which the value needs to be resolved.
+    # @param task [Task] The Task for which the value needs to be resolved.
     # @param parameter_name [String] The parameter name.
     #
     # @return [Array] value of the resolved inclusion option.
@@ -149,24 +152,27 @@ module MaintenanceTasks
     # If the parameter has a `validates_inclusion_of` validator, return a dropdown list of options instead.
     def parameter_field(form_builder, parameter_name)
       inclusion_values = resolve_inclusion_value(form_builder.object, parameter_name)
-      return form_builder.select(parameter_name, inclusion_values, prompt: "Select a value") if inclusion_values
+      if inclusion_values
+        return tag.div(form_builder.select(parameter_name, inclusion_values, prompt: "Select a value"), class: "select")
+      end
 
       case form_builder.object.class.attribute_types[parameter_name]
       when ActiveModel::Type::Integer
-        form_builder.number_field(parameter_name)
+        form_builder.number_field(parameter_name, class: "input")
       when ActiveModel::Type::Decimal, ActiveModel::Type::Float
-        form_builder.number_field(parameter_name, { step: "any" })
+        form_builder.number_field(parameter_name, { step: "any", class: "input" })
       when ActiveModel::Type::DateTime
-        form_builder.datetime_field(parameter_name) + datetime_field_help_text
+        form_builder.datetime_field(parameter_name, class: "input") + datetime_field_help_text
       when ActiveModel::Type::Date
-        form_builder.date_field(parameter_name)
+        form_builder.date_field(parameter_name, class: "input")
       when ActiveModel::Type::Time
-        form_builder.time_field(parameter_name)
+        form_builder.time_field(parameter_name, class: "input")
       when ActiveModel::Type::Boolean
-        form_builder.check_box(parameter_name)
+        form_builder.check_box(parameter_name, class: "checkbox")
       else
         form_builder.text_area(parameter_name, class: "textarea")
       end
+        .then { |input| tag.div(input, class: "control") }
     end
 
     # Return helper text for the datetime-local form field.
@@ -182,5 +188,16 @@ module MaintenanceTasks
         class: "content is-small",
       )
     end
+
+    # Checks if an attribute is required for a given Task.
+    #
+    # @param task [MaintenanceTasks::TaskDataShow] The TaskDataShow instance.
+    # @param parameter_name [Symbol] The name of the attribute to check.
+    # @return [Boolean] Whether the attribute is required.
+    def attribute_required?(task, parameter_name)
+      task.class.validators_on(parameter_name).any? do |validator|
+        validator.kind == :presence
+      end
+    end
   end
 end

data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb

@@ -101,7 +101,8 @@ module MaintenanceTasks
       throw(:abort, :skip_complete_callbacks) if @run.stopping?
       task_iteration(input)
       @ticker.tick
-      @run.reload_status
+
+      reload_run_status
     end
 
     def task_iteration(input)
@@ -112,7 +113,7 @@ module MaintenanceTasks
       end
     rescue => error
       @errored_element = input
-      raise error
+      raise error unless @task.rescue_with_handler(error)
     end
 
     def before_perform
@@ -127,6 +128,8 @@ module MaintenanceTasks
       @ticker = Ticker.new(MaintenanceTasks.ticker_delay) do |ticks, duration|
         @run.persist_progress(ticks, duration)
       end
+
+      @last_status_reload = nil
     end
 
     def on_start
@@ -181,11 +184,37 @@ module MaintenanceTasks
           task_name: @run.task_name,
           started_at: @run.started_at,
           ended_at: @run.ended_at,
+          run_id: @run.id,
+          tick_count: @run.tick_count,
         }
       end
-      errored_element = @errored_element if defined?(@errored_element)
+      task_context[:errored_element] = @errored_element if defined?(@errored_element)
     ensure
-      MaintenanceTasks.error_handler.call(error, task_context, errored_element)
+      if MaintenanceTasks.instance_variable_get(:@error_handler)
+        errored_element = task_context.delete(:errored_element)
+        MaintenanceTasks.error_handler.call(error, task_context.except(:run_id, :tick_count), errored_element)
+      else
+        Rails.error.report(
+          error,
+          handled: MaintenanceTasks.report_errors_as_handled,
+          context: task_context,
+          source: "maintenance-tasks",
+        )
+      end
+    end
+
+    def reload_run_status
+      return unless should_reload_status?
+
+      @run.reload_status
+      @last_status_reload = Time.now
+    end
+
+    def should_reload_status?
+      return true if @last_status_reload.nil?
+
+      time_since_last_reload = Time.now - @last_status_reload
+      time_since_last_reload >= @task.status_reload_frequency
     end
   end
 end