maintenance_tasks 2.11.0 → 2.13.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4ac2fd148131a277d37ddc609173b8549d01d3e6a9f78786cf729bd5da4f65b0
-  data.tar.gz: a2db7dfe8a9b0a958d943f88305fbe6d8988fd171901c49cd688070d8125124e
+  metadata.gz: 8402f1f4b8919e892a6334373ad397d49b5c6c78893f2f107b990b90653435d1
+  data.tar.gz: 1459d5b92233a1196aeb83c80bec9da525f04e975a50f6a415cdcd8dc647bd59
 SHA512:
-  metadata.gz: c8e4c707d55ca479cca9f1592cc65359cc460d70cfd1e4ea4b2caf7c0a92617b4f243feb83c9f3657cb81a76691ed660c91bdafba73634b819a0fa44a719a2c8
-  data.tar.gz: 0efab90ffbdea1131f72de57443c8ed01fc30550d88c3ebbfe77b8e48a37c27dec8ec2ed7591ebdf9ea463e82ff4df7fcb649b111076a7302e10fce5720eb437
+  metadata.gz: 3dff16b85a650f4d73cf79df20267d1278b1181f2ba6717742581909ebfd39145dfdd9b9cf1453edaa785474fb6f7dda87af2bad534e6398bf010945b02f2da3
+  data.tar.gz: c7b75c1edc684fad66c5080ee3642fe469038e1f4ab2329ead0fd3c65e5f0ca2a940250dc988658dce672e2de0d056741b9a317310afa23165ba27794722ce1d

data/README.md

@@ -74,9 +74,11 @@ The generator creates and runs a migration to add the necessary table to your
 database. It also mounts Maintenance Tasks in your `config/routes.rb`. By
 default the web UI can be accessed in the new `/maintenance_tasks` path.
 
-This gem uses the [Rails Error Reporter](https://guides.rubyonrails.org/error_reporting.html) to report errors. If you are using a bug
-tracking service you may want to subscribe to the reporter. See [Reporting Errors](#reporting-errors)
-for more information.
+This gem uses the [Rails Error Reporter][rails-error-reporting] to report errors.
+If you are using a bug tracking service you may want to subscribe to the
+reporter. See [Reporting Errors](#reporting-errors) for more information.
+
+[rails-error-reporting]: https://guides.rubyonrails.org/error_reporting.html
 
 ### Active Job Dependency
 
@@ -114,12 +116,12 @@ The typical Maintenance Tasks workflow is as follows:
 1. [Generate a class describing the Task](#creating-a-task) and the work to be
    done.
 2. Run the Task
-    - either by [using the included web UI](#running-a-task-from-the-web-ui),
-    - or by [using the command line](#running-a-task-from-the-command-line),
-    - or by [using Ruby](#running-a-task-from-ruby).
+   - either by [using the included web UI](#running-a-task-from-the-web-ui),
+   - or by [using the command line](#running-a-task-from-the-command-line),
+   - or by [using Ruby](#running-a-task-from-ruby).
 3. [Monitor the Task](#monitoring-your-tasks-status)
-    - either by using the included web UI,
-    - or by manually checking your task’s run’s status in your database.
+   - either by using the included web UI,
+   - or by manually checking your task’s run’s status in your database.
 4. Optionally, delete the Task code if you no longer need it.
 
 ### Creating a Task
@@ -168,7 +170,8 @@ end
 When processing records from an Active Record Relation, records are fetched in
 batches internally, and then each record is passed to the `#process` method.
 Maintenance Tasks will query the database to fetch records in batches of 100 by
-default, but the batch size can be modified using the `collection_batch_size` macro:
+default, but the batch size can be modified using the `collection_batch_size`
+macro:
 
 ```ruby
 # app/tasks/maintenance/update_posts_task.rb
@@ -502,13 +505,44 @@ set of values will be used to populate a dropdown in the user interface. The
 following types are supported:
 
 * Arrays
-* Procs and lambdas that optionally accept the Task instance, and return an Array.
-* Callable objects that receive one argument, the Task instance, and return an Array.
+* Procs and lambdas that optionally accept the Task instance, and return an
+  Array.
+* Callable objects that receive one argument, the Task instance, and return an
+  Array.
 * Methods that return an Array, called on the Task instance.
 
 For enumerables that don't match the supported types, a text field will be
 rendered instead.
 
+### Masking Task Parameters
+
+Task attributes can be masked in the UI by adding `mask_attribute` class method
+in the task class. This will replace the value in the arguments list with
+`[FILTERED]` in the UI.
+
+```ruby
+# app/tasks/maintenance/sensitive_params_task.rb
+
+module Maintenance
+  class SensitiveParamsTask < MaintenanceTasks::Task
+    attribute :sensitive_content, :string
+
+    mask_attribute :sensitive_content
+  end
+end
+```
+
+If you have any filtered parameters in the global [Rails parameter
+filter][rails-parameter-filter], they will be automatically taken into account
+when masking the parameters, which means that you can mask parameters across all
+tasks by adding them to the global rails parameters filter.
+
+[rails-parameter-filter]:https://guides.rubyonrails.org/configuring.html#config-filter-parameters
+
+```ruby
+Rails.application.config.filter_parameters += %i[token]
+```
+
 ### Custom cursor columns to improve performance
 
 The [job-iteration gem][job-iteration], on which this gem depends, adds an
@@ -899,10 +933,10 @@ a Task can be in:
 
 The Maintenance Tasks engine uses Rails sessions for flash messages and storing
 the CSRF token. For the engine to work in an API-only Rails application, you
-need to add a [session middleware][] and the `ActionDispatch::Flash`
-middleware. The engine also defines a strict [Content Security Policy][], make
-sure to include `ActionDispatch::ContentSecurityPolicy::Middleware` in your
-app's middleware stack to ensure the CSP is delivered to the user's browser.
+need to add a [session middleware][] and the `ActionDispatch::Flash` middleware.
+The engine also defines a strict [Content Security Policy][], make sure to
+include `ActionDispatch::ContentSecurityPolicy::Middleware` in your app's
+middleware stack to ensure the CSP is delivered to the user's browser.
 
 [session middleware]: https://guides.rubyonrails.org/api_app.html#using-session-middlewares
 [Content Security Policy]: https://developer.mozilla.org/en-US/docs/Web/HTTP/CSP
@@ -926,8 +960,8 @@ module YourApplication
 end
 ```
 
-You can read more in the [Using Rails for API-only Applications][rails api] Rails
-guide.
+You can read more in the [Using Rails for API-only Applications][rails api]
+Rails guide.
 
 [rails api]: https://guides.rubyonrails.org/api_app.html
 
@@ -944,9 +978,9 @@ infrastructure or code changes.
 This means a Task can safely be interrupted, re-enqueued and resumed without any
 intervention at the end of an iteration, after the `process` method returns.
 
-By default, a running Task will be interrupted after running for more 5 minutes.
-This is [configured in the `job-iteration` gem][max-job-runtime] and can be
-tweaked in an initializer if necessary.
+By default, a running Task will be interrupted after running for more than 5
+minutes. This is [configured in the `job-iteration` gem][max-job-runtime] and
+can be tweaked in an initializer if necessary.
 
 [max-job-runtime]: https://github.com/Shopify/job-iteration/blob/-/guides/best-practices.md#max-job-runtime
 
@@ -993,45 +1027,76 @@ be placed in a `maintenance_tasks.rb` initializer.
 Exceptions raised while a Task is performing are rescued and information about
 the error is persisted and visible in the UI.
 
-Errors are also sent to the `Rails.error.reporter`, which can be configured by your
-application. See the [Error Reporting in Rails Applications](https://guides.rubyonrails.org/error_reporting.html) guide for more details.
+Errors are also sent to `Rails.error.report`, which can be configured by
+your application. See the [Error Reporting in Rails
+Applications][rails-error-reporting] guide for more details.
 
 Reports to the error reporter will contain the following data:
 
 * `error`: The exception that was raised.
-* `context`: A hash with additional information about the Task and the
-  error:
-  * `task_name`: The name of the Task that errored
-  * `started_at`: The time the Task started
-  * `ended_at`: The time the Task errored
-  * `run_id`: The id of the errored Task run
-  * `tick_count`: The tick count at the time of the error
-  * `errored_element`: The element, if any, that was being processed when the Task
-    raised an exception. If you would like to pass this object to your exception
-    monitoring service, make sure you **sanitize the object** to avoid leaking
-    sensitive data and **convert it to a format** that is compatible with your bug
-    tracker.
-* `source`: This will be `maintenance_tasks`
+* `context`: A hash with additional information about the Task and the error:
+   * `task_name`: The name of the Task that errored
+   * `started_at`: The time the Task started
+   * `ended_at`: The time the Task errored
+   * `run_id`: The id of the errored Task run
+   * `tick_count`: The tick count at the time of the error
+   * `errored_element`: The element, if any, that was being processed when the
+* `source`: This will be `maintenance-tasks`
+* `handled`: the value of `MaintenanceTasks.report_errors_as_handled` (default `true`, see below)
 
 Note that `context` may be empty if the Task produced an error before any
-context could be gathered (for example, if deserializing the job to process
-your Task failed).
+context could be gathered (for example, if deserializing the job to process your
+Task failed).
+
+Here's an example custom subscriber to the Rails error reporter for integrating
+with an exception monitoring service (Bugsnag):
+
+```ruby
+# config/initializers/maintenance_tasks.rb
+MaintenanceTasks.report_errors_as_handled = false
+
+class MaintenanceTasksErrorSubscriber
+  def report(error, handled:, severity:, context:, source: nil)
+    return unless source == "maintenance-tasks"
+
+    unless handled
+      Bugsnag.notify(error) do |notification|
+        notification.add_metadata(:task, context)
+      end
+    else
+      Rails.logger.info(error)
+    end
+  end
+end
+
+Rails.error.subscribe(MaintenanceTasksErrorSubscriber.new)
+```
+
+`MaintenanceTasks.report_errors_as_handled` determines the value for `handled` in this example.
+By default (for backwards compatibility) this is `true`.
+Setting this to `false` provides more accurate error reporting as it allows to distinguish between
+expected (e.g., via `report_on`) and unexpected errors in error subscribers.
+`false` will be the default in v3.0.
 
 #### Reporting errors during iteration
 
-By default, errors raised during task iteration will be raised to the application
-and iteration will stop. However, you may want to handle some errors and continue
-iteration. `MaintenanceTasks::Task.report_on` can be used to rescue certain
-exceptions and report them to the Rails error reporter.
+By default, errors raised during task iteration will be raised to the
+application and iteration will stop. However, you may want to handle some errors
+and continue iteration. `MaintenanceTasks::Task.report_on` can be used to rescue
+certain exceptions and report them to the Rails error reporter. Any keyword
+arguments are passed to
+[ActiveSupport::ErrorReporter#report][as-error-reporter-report]:
+
+[as-error-reporter-report]: https://api.rubyonrails.org/classes/ActiveSupport/ErrorReporter.html#method-i-report
 
 ```ruby
 class MyTask < MaintenanceTasks::Task
-  report_on(MyException)
+  report_on(MyException, OtherException, severity: :info, context: {task_name: "my_task"})
 end
 ```
 
-`MaintenanceTasks::Task` also includes `ActiveSupport::Rescuable` which you can use
-to implement custom error handling.
+`MaintenanceTasks::Task` also includes `ActiveSupport::Rescuable` which you can
+use to implement custom error handling.
 
 ```ruby
 class MyTask < MaintenanceTasks::Task
@@ -1202,6 +1267,42 @@ The value for `MaintenanceTasks.stuck_task_duration` must be an
 `ActiveSupport::Duration`. If no value is specified, it will default to 5
 minutes.
 
+#### Configure status reload frequency
+
+`MaintenanceTasks.status_reload_frequency` can be configured to specify how often
+the run status should be reloaded during iteration. By default, the status is
+reloaded every second, but this can be increased to improve performance. Note that increasing the reload interval impacts how quickly
+your task will stop if it is paused or interrupted.
+
+```ruby
+# config/initializers/maintenance_tasks.rb
+MaintenanceTasks.status_reload_frequency = 10.seconds  # Reload status every 10 seconds
+```
+
+Individual tasks can also override this setting using the `reload_status_every` method:
+
+```ruby
+# app/tasks/maintenance/update_posts_task.rb
+
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    # Reload status every 5 seconds instead of the global default
+    reload_status_every(5.seconds)
+
+    def collection
+      Post.all
+    end
+
+    def process(post)
+      post.update!(content: "New content!")
+    end
+  end
+end
+```
+
+This optimization can significantly reduce database queries, especially for short iterations.
+This is especially useful if the task doesn't need to check for cancellation/pausing very often.
+
 #### Metadata
 
 `MaintenanceTasks.metadata` can be configured to specify a proc from which to

data/app/controllers/maintenance_tasks/application_controller.rb

@@ -8,16 +8,26 @@ module MaintenanceTasks
     BULMA_CDN = "https://cdn.jsdelivr.net"
 
     content_security_policy do |policy|
-      policy.style_src(
+      policy.style_src_elem(
         BULMA_CDN,
-        # ruby syntax highlighting
-        "'sha256-y9V0na/WU44EUNI/HDP7kZ7mfEci4PAOIjYOOan6JMA='",
+        # <style> tag in app/views/layouts/maintenance_tasks/application.html.erb
+        "'sha256-WHHDQLdkleXnAN5zs0GDXC5ls41CHUaVsJtVpaNx+EM='",
       )
-      policy.script_src(
-        # page refresh script
+      capybara_lockstep_scripts = [
+        "'sha256-1AoN3ZtJC5OvqkMgrYvhZjp4kI8QjJjO7TAyKYiDw+U='",
+        "'sha256-QVSzZi6ZsX/cu4h+hIs1iVivG1BxUmJggiEsGDIXBG0='", # with debug on
+      ] if defined?(Capybara::Lockstep)
+      policy.script_src_elem(
+        # <script> tag in app/views/layouts/maintenance_tasks/application.html.erb
         "'sha256-NiHKryHWudRC2IteTqmY9v1VkaDUA/5jhgXkMTkgo2w='",
+        # <script> tag for capybara-lockstep
+        *capybara_lockstep_scripts,
       )
+
+      policy.require_trusted_types_for # disable because we use new DOMParser().parseFromString
       policy.frame_ancestors(:self)
+      policy.connect_src(:self)
+      policy.form_action(:self)
     end
 
     protect_from_forgery with: :exception

data/app/helpers/maintenance_tasks/tasks_helper.rb

@@ -47,7 +47,7 @@ module MaintenanceTasks
       progress_bar = tag.progress(
         value: progress.value,
         max: progress.max,
-        class: ["progress"] + STATUS_COLOURS.fetch(run.status),
+        class: ["progress", "mt-4"] + STATUS_COLOURS.fetch(run.status),
       )
       progress_text = tag.p(tag.i(progress.text))
       tag.div(progress_bar + progress_text, class: "block")
@@ -60,7 +60,10 @@ module MaintenanceTasks
     # @return [String] the span element containing the status, with the
     #   appropriate tag class attached.
     def status_tag(status)
-      tag.span(status.capitalize, class: ["tag"] + STATUS_COLOURS.fetch(status))
+      tag.span(
+        status.capitalize,
+        class: ["tag", "has-text-weight-medium", "pr-2", "mr-4"] + STATUS_COLOURS.fetch(status),
+      )
     end
 
     # Reports the approximate elapsed time a Run has been processed so far based
@@ -101,7 +104,7 @@ module MaintenanceTasks
       )
     end
 
-    # Resolves values covered by the inclusion validator for a task attribute.
+    # Resolves values covered by the inclusion validator for a Task attribute.
     # Supported option types:
     # - Arrays
     # - Procs and lambdas that optionally accept the Task instance, and return an Array.
@@ -112,7 +115,7 @@ module MaintenanceTasks
     #
     # Returned values are used to populate a dropdown list of options.
     #
-    # @param task_class [Class<Task>] The task class for which the value needs to be resolved.
+    # @param task [Task] The Task for which the value needs to be resolved.
     # @param parameter_name [String] The parameter name.
     #
     # @return [Array] value of the resolved inclusion option.
@@ -149,24 +152,27 @@ module MaintenanceTasks
     # If the parameter has a `validates_inclusion_of` validator, return a dropdown list of options instead.
     def parameter_field(form_builder, parameter_name)
       inclusion_values = resolve_inclusion_value(form_builder.object, parameter_name)
-      return form_builder.select(parameter_name, inclusion_values, prompt: "Select a value") if inclusion_values
+      if inclusion_values
+        return tag.div(form_builder.select(parameter_name, inclusion_values, prompt: "Select a value"), class: "select")
+      end
 
       case form_builder.object.class.attribute_types[parameter_name]
       when ActiveModel::Type::Integer
-        form_builder.number_field(parameter_name)
+        form_builder.number_field(parameter_name, class: "input")
       when ActiveModel::Type::Decimal, ActiveModel::Type::Float
-        form_builder.number_field(parameter_name, { step: "any" })
+        form_builder.number_field(parameter_name, { step: "any", class: "input" })
       when ActiveModel::Type::DateTime
-        form_builder.datetime_field(parameter_name) + datetime_field_help_text
+        form_builder.datetime_field(parameter_name, class: "input") + datetime_field_help_text
       when ActiveModel::Type::Date
-        form_builder.date_field(parameter_name)
+        form_builder.date_field(parameter_name, class: "input")
       when ActiveModel::Type::Time
-        form_builder.time_field(parameter_name)
+        form_builder.time_field(parameter_name, class: "input")
       when ActiveModel::Type::Boolean
-        form_builder.check_box(parameter_name)
+        form_builder.check_box(parameter_name, class: "checkbox")
       else
         form_builder.text_area(parameter_name, class: "textarea")
       end
+        .then { |input| tag.div(input, class: "control") }
     end
 
     # Return helper text for the datetime-local form field.
@@ -182,5 +188,16 @@ module MaintenanceTasks
         class: "content is-small",
       )
     end
+
+    # Checks if an attribute is required for a given Task.
+    #
+    # @param task [MaintenanceTasks::TaskDataShow] The TaskDataShow instance.
+    # @param parameter_name [Symbol] The name of the attribute to check.
+    # @return [Boolean] Whether the attribute is required.
+    def attribute_required?(task, parameter_name)
+      task.class.validators_on(parameter_name).any? do |validator|
+        validator.kind == :presence
+      end
+    end
   end
 end

data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb

@@ -101,7 +101,8 @@ module MaintenanceTasks
       throw(:abort, :skip_complete_callbacks) if @run.stopping?
       task_iteration(input)
       @ticker.tick
-      @run.reload_status
+
+      reload_run_status
     end
 
     def task_iteration(input)
@@ -127,6 +128,8 @@ module MaintenanceTasks
       @ticker = Ticker.new(MaintenanceTasks.ticker_delay) do |ticks, duration|
         @run.persist_progress(ticks, duration)
       end
+
+      @last_status_reload = nil
     end
 
     def on_start
@@ -190,11 +193,28 @@ module MaintenanceTasks
       if MaintenanceTasks.instance_variable_get(:@error_handler)
         errored_element = task_context.delete(:errored_element)
         MaintenanceTasks.error_handler.call(error, task_context.except(:run_id, :tick_count), errored_element)
-      elsif Rails.gem_version >= Gem::Version.new("7.1")
-        Rails.error.report(error, context: task_context, source: "maintenance-tasks")
       else
-        Rails.error.report(error, handled: true, context: task_context)
+        Rails.error.report(
+          error,
+          handled: MaintenanceTasks.report_errors_as_handled,
+          context: task_context,
+          source: "maintenance-tasks",
+        )
       end
     end
+
+    def reload_run_status
+      return unless should_reload_status?
+
+      @run.reload_status
+      @last_status_reload = Time.now
+    end
+
+    def should_reload_status?
+      return true if @last_status_reload.nil?
+
+      time_since_last_reload = Time.now - @last_status_reload
+      time_since_last_reload >= @task.status_reload_frequency
+    end
   end
 end