maintenance_tasks 2.5.1 → 2.6.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8952908cba45fd0a79cfcf0d334e89eab4e3a554ba4c0b1acffb318c7e92730f
-  data.tar.gz: bc1312f6269c51fe80480f7af90281c89c36bc9c78f052f65f4b5853effa5a41
+  metadata.gz: 2dca6b60a4a6d6366eaf05b43d1e7d1244c59baef10fe9d25561cab727a70638
+  data.tar.gz: c9827e001e131747ea2cb20301ef9749f394bd603a1d657afac0024e1feefc79
 SHA512:
-  metadata.gz: c32f23abae224617aef9148ac7e890a95fb799997e0900b6a5ad72fbb988cadb25ef61ef6a35e803f2483cce93f6ad20d379c202f2a513f72c339d553e830a12
-  data.tar.gz: cead5f36f1038614e23da3c6fa3661be58d00ac326a859681a3740d818977f0c3c311860f567d96701dcd04bc1019e001ad0903c20833444b2a0669dd9aac6e9
+  metadata.gz: 269fe1ab563047cf368cb552b91305c0a99b25affa6bea95553f55541c6e5a947726e441b3487181146d90d75bf9e9e1e853fc3302fda483ce0d00032ad7bceb
+  data.tar.gz: 7b1ae78d3049cb3c28c17bcf068acc735441e4b4a3de6babeb8107294408a1ee139c03b6bad12e1004f0bef59c834314429079df0a11c6384148f007bc3a44a9

data/README.md

@@ -10,8 +10,8 @@ engine helps with the second part of this process, backfilling.
 
 Maintenance tasks are collection-based tasks, usually using Active Record, that
 update the data in your database. They can be paused or interrupted. Maintenance
-tasks can operate [in batches](#processing-batch-collections) and use
-[throttling](#throttling) to control the load on your database.
+tasks can operate [in batches][#processing-batch-collections] and use
+[throttling][#throttling] to control the load on your database.
 
 Maintenance tasks aren't meant to happen on a regular basis. They're used as
 needed, or as one-offs. Normally maintenance tasks are ephemeral, so they are
@@ -45,10 +45,12 @@ If your task updates your database schema instead of data, use a migration
 instead of a maintenance task.
 
 If your task happens regularly, consider Active Jobs with a scheduler or cron,
-[job-iteration jobs](https://github.com/shopify/job-iteration) and/or [custom
-rails_admin UIs][rails-admin-engines] instead of the Maintenance Tasks gem.
-Maintenance tasks should be ephemeral, to suit their intentionally limited UI.
-They should not repeat.
+[job-iteration jobs][job-iteration] and/or [custom rails_admin
+UIs][rails-admin-engines] instead of the Maintenance Tasks gem. Maintenance
+tasks should be ephemeral, to suit their intentionally limited UI. They should
+not repeat.
+
+[job-iteration]: https://github.com/shopify/job-iteration
 
 To create seed data for a new application, use the provided Rails `db/seeds.rb`
 file instead.
@@ -91,10 +93,12 @@ take a look at the [Active Job documentation][active-job-docs].
 ### Autoloading
 
 The Maintenance Tasks framework does not support autoloading in `:classic` mode.
-Please ensure your application is using
-[Zeitwerk](https://github.com/fxn/zeitwerk) to load your code.  For more
+Please ensure your application is using [Zeitwerk][] to load your code. For more
 information, please consult the [Rails guides on autoloading and reloading
-constants](https://guides.rubyonrails.org/autoloading_and_reloading_constants.html).
+constants][autoloading].
+
+[Zeitwerk]: https://github.com/fxn/zeitwerk
+[autoloading]: https://guides.rubyonrails.org/autoloading_and_reloading_constants.html
 
 ## Usage
 
@@ -305,11 +309,11 @@ If you have a special use case requiring iteration over an unsupported
 collection type, such as external resources fetched from some API, you can
 implement the `enumerator_builder(cursor:)` method in your task.
 
-This method should return an `Enumerator`, yielding pairs of
-`[item, cursor]`. Maintenance Tasks takes care of persisting the current
-cursor position and will provide it as the `cursor` argument if your task is
-interrupted or resumed. The `cursor` is stored as a `String`, so your custom
-enumerator should handle serializing/deserializing the value if required.
+This method should return an `Enumerator`, yielding pairs of `[item, cursor]`.
+Maintenance Tasks takes care of persisting the current cursor position and will
+provide it as the `cursor` argument if your task is interrupted or resumed. The
+`cursor` is stored as a `String`, so your custom enumerator should handle
+serializing/deserializing the value if required.
 
 ```ruby
 # app/tasks/maintenance/custom_enumerator_task.rb
@@ -377,7 +381,7 @@ module Maintenance
     throttle_on(backoff: -> { RandomBackoffGenerator.generate_duration } ) do
       DatabaseStatus.unhealthy?
     end
-    ...
+    # ...
   end
 end
 ```
@@ -413,6 +417,42 @@ to run. Since arguments are specified in the user interface via text area
 inputs, it’s important to check that they conform to the format your Task
 expects, and to sanitize any inputs if necessary.
 
+### Custom cursor columns to improve performance
+
+The [job-iteration gem][job-iteration], on which this gem depends, adds an
+`order by` clause to the relation returned by the `collection` method, in order
+to iterate through records. It defaults to order on the `id` column.
+
+The [job-iteration gem][job-iteration] supports configuring which columns are
+used to order the cursor, as documented in
+[`build_active_record_enumerator_on_records`][ji-ar-enumerator-doc].
+
+[ji-ar-enumerator-doc]: https://www.rubydoc.info/gems/job-iteration/JobIteration/EnumeratorBuilder#build_active_record_enumerator_on_records-instance_method
+
+The `maintenance-tasks` gem exposes the ability that `job-iteration` provides to
+control the cursor columns, through the `cursor_columns` method in the
+`MaintenanceTasks::Task` class. If the `cursor_columns` method returns `nil`,
+the query is ordered by the primary key. If cursor columns values change during
+an iteration, records may be skipped or yielded multiple times.
+
+```ruby
+module Maintenance
+  class UpdatePostsTask < MaintenanceTasks::Task
+    def cursor_columns
+      [:created_at, :id]
+    end
+
+    def collection
+      Post.where(created_at: 2.days.ago...1.hour.ago)
+    end
+
+    def process(post)
+      post.update!(content: "updated content")
+    end
+  end
+end
+```
+
 ### Using Task Callbacks
 
 The Task provides callbacks that hook into its life cycle.
@@ -595,6 +635,42 @@ module Maintenance
 end
 ```
 
+### Writing tests for a Task that uses a custom enumerator
+
+Tests for tasks that use custom enumerators need to instantiate the task class
+in order to call `#build_enumerator`. Once the task instance is set up, validate
+that `#build_enumerator` returns an enumerator yielding pairs of [item, cursor]
+as expected.
+
+```ruby
+# test/tasks/maintenance/custom_enumerating_task.rb
+
+require "test_helper"
+
+module Maintenance
+  class CustomEnumeratingTaskTest < ActiveSupport::TestCase
+    setup do
+      @task = CustomEnumeratingTask.new
+    end
+
+    test "#build_enumerator returns enumerator yielding pairs of [item, cursor]" do
+      enum = @task.build_enumerator(cursor: 0)
+      expected_items = [:b, :c]
+
+      assert_equal 2, enum.size
+
+      enum.each_with_index do |item, cursor|
+        assert_equal expected_items[cursor], item
+      end
+    end
+
+    test "#process performs a task iteration" do
+      # ...
+    end
+  end
+end
+```
+
 ### Running a Task
 
 #### Running a Task from the Web UI
@@ -917,7 +993,7 @@ MaintenanceTasks.parent_controller = "Services::CustomController"
 class Services::CustomController < ActionController::Base
   include CustomSecurityThings
   include CustomLoggingThings
-  ...
+  # ...
 end
 ```
 
@@ -928,12 +1004,14 @@ If no value is specified, it will default to `"ActionController::Base"`.
 
 #### Configure time after which the task will be considered stuck
 
-To specify a time duration after which a task is considered stuck if it has not been updated,
-you can configure `MaintenanceTasks.stuck_task_duration`. This duration should account for
-job infrastructure events that may prevent the maintenance tasks job from being executed and cancelling the task.
+To specify a time duration after which a task is considered stuck if it has not
+been updated, you can configure `MaintenanceTasks.stuck_task_duration`. This
+duration should account for job infrastructure events that may prevent the
+maintenance tasks job from being executed and cancelling the task.
 
-The value for `MaintenanceTasks.stuck_task_duration` must be an `ActiveSupport::Duration`.
-If no value is specified, it will default to 5 minutes.
+The value for `MaintenanceTasks.stuck_task_duration` must be an
+`ActiveSupport::Duration`. If no value is specified, it will default to 5
+minutes.
 
 ### Metadata
 

data/app/jobs/concerns/maintenance_tasks/task_job_concern.rb

@@ -33,6 +33,45 @@ module MaintenanceTasks
     def build_enumerator(_run, cursor:)
       cursor ||= @run.cursor
       @collection_enum = @task.enumerator_builder(cursor: cursor)
+
+      @collection_enum ||= case (collection = @task.collection)
+      when :no_collection
+        enumerator_builder.build_once_enumerator(cursor: nil)
+      when ActiveRecord::Relation
+        enumerator_builder.active_record_on_records(collection, cursor: cursor, columns: @task.cursor_columns)
+      when ActiveRecord::Batches::BatchEnumerator
+        if collection.start || collection.finish
+          raise ArgumentError, <<~MSG.squish
+            #{@task.class.name}#collection cannot support
+            a batch enumerator with the "start" or "finish" options.
+          MSG
+        end
+
+        # For now, only support automatic count based on the enumerator for
+        # batches
+        enumerator_builder.active_record_on_batch_relations(
+          collection.relation,
+          cursor: cursor,
+          batch_size: collection.batch_size,
+          columns: @task.cursor_columns,
+        )
+      when Array
+        enumerator_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
+      when BatchCsvCollectionBuilder::BatchCsv
+        JobIteration::CsvEnumerator.new(collection.csv).batches(
+          batch_size: collection.batch_size,
+          cursor: cursor&.to_i,
+        )
+      when CSV
+        JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
+      else
+        raise ArgumentError, <<~MSG.squish
+          #{@task.class.name}#collection must be either an
+          Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
+          Array, or CSV.
+        MSG
+      end
+
       throttle_enumerator(@collection_enum)
     end
 

data/app/models/maintenance_tasks/run.rb

@@ -33,7 +33,11 @@ module MaintenanceTasks
     ]
     COMPLETED_STATUSES = [:succeeded, :errored, :cancelled]
 
-    enum status: STATUSES.to_h { |status| [status, status.to_s] }
+    if Rails.gem_version >= Gem::Version.new("7.0.alpha")
+      enum :status, STATUSES.to_h { |status| [status, status.to_s] }
+    else
+      enum status: STATUSES.to_h { |status| [status, status.to_s] }
+    end
 
     validate :task_name_belongs_to_a_valid_task, on: :create
     validate :csv_attachment_presence, on: :create

data/app/models/maintenance_tasks/task.rb

@@ -230,6 +230,18 @@ module MaintenanceTasks
       self.class.collection_builder_strategy.collection(self)
     end
 
+    # The columns used to build the `ORDER BY` clause of the query for iteration.
+    #
+    # If cursor_columns returns nil, the query is ordered by the primary key.
+    # If cursor columns values change during an iteration, records may be skipped or yielded multiple times.
+    # More details in the documentation of JobIteration::EnumeratorBuilder.build_active_record_enumerator_on_records:
+    # https://www.rubydoc.info/gems/job-iteration/JobIteration/EnumeratorBuilder#build_active_record_enumerator_on_records-instance_method
+    #
+    # @return the cursor_columns.
+    def cursor_columns
+      nil
+    end
+
     # Placeholder method to raise in case a subclass fails to implement the
     # expected instance method.
     #
@@ -248,7 +260,7 @@ module MaintenanceTasks
       self.class.collection_builder_strategy.count(self)
     end
 
-    # Default enumeration builder. You may override this method to return any
+    # Default enumerator builder. You may override this method to return any
     # Enumerator yielding pairs of `[item, item_cursor]`.
     #
     # @param cursor [String, nil] cursor position to resume from, or nil on
@@ -256,46 +268,7 @@ module MaintenanceTasks
     #
     # @return [Enumerator]
     def enumerator_builder(cursor:)
-      collection = self.collection
-
-      job_iteration_builder = JobIteration::EnumeratorBuilder.new(nil)
-
-      case collection
-      when :no_collection
-        job_iteration_builder.build_once_enumerator(cursor: nil)
-      when ActiveRecord::Relation
-        job_iteration_builder.active_record_on_records(collection, cursor: cursor)
-      when ActiveRecord::Batches::BatchEnumerator
-        if collection.start || collection.finish
-          raise ArgumentError, <<~MSG.squish
-            #{self.class.name}#collection cannot support
-            a batch enumerator with the "start" or "finish" options.
-          MSG
-        end
-
-        # For now, only support automatic count based on the enumerator for
-        # batches
-        job_iteration_builder.active_record_on_batch_relations(
-          collection.relation,
-          cursor: cursor,
-          batch_size: collection.batch_size,
-        )
-      when Array
-        job_iteration_builder.build_array_enumerator(collection, cursor: cursor&.to_i)
-      when BatchCsvCollectionBuilder::BatchCsv
-        JobIteration::CsvEnumerator.new(collection.csv).batches(
-          batch_size: collection.batch_size,
-          cursor: cursor&.to_i,
-        )
-      when CSV
-        JobIteration::CsvEnumerator.new(collection).rows(cursor: cursor&.to_i)
-      else
-        raise ArgumentError, <<~MSG.squish
-          #{self.class.name}#collection must be either an
-          Active Record Relation, ActiveRecord::Batches::BatchEnumerator,
-          Array, or CSV.
-        MSG
-      end
+      nil
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: maintenance_tasks
 version: !ruby/object:Gem::Version
-  version: 2.5.1
+  version: 2.6.0
 platform: ruby
 authors:
 - Shopify Engineering
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2024-02-09 00:00:00.000000000 Z
+date: 2024-02-12 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: actionpack
@@ -171,7 +171,7 @@ homepage: https://github.com/Shopify/maintenance_tasks
 licenses:
 - MIT
 metadata:
-  source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.5.1
+  source_code_uri: https://github.com/Shopify/maintenance_tasks/tree/v2.6.0
   allowed_push_host: https://rubygems.org
 post_install_message: 
 rdoc_options: []