sidekiq-iteration 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 82316cffa840b2c9619792b6f0c5bb7ec696f964ad81edf6b0ed5861339ca064
-  data.tar.gz: 8337c0e87e6be8858d5b9d868c8a04b05f91de315be2e9aca9c40e8447c78644
+  metadata.gz: 342540da75582c7f102f6ead29643a5196038978c5626b0c6a00a43db04f4f1f
+  data.tar.gz: cfb7cf80031976e5c68b2503d6ab0a13ffe92efc016419ad20944d3c5ce3e56d
 SHA512:
-  metadata.gz: 63712780bca873613cbe3ef89ff0037c3eaa5633a28382eb02427311360714a89f092e5efef29873efa75067d22f32745eea3f04d7606442e29da33e2e2e6a08
-  data.tar.gz: 4ded7fc6ab772c019154e6559027c87d389a356a16915d2c869e318fb26f5339dd98355a0f1eb422358c1cd9f7fe31bc2a70d5627a577f8f45394db15d0593b7
+  metadata.gz: 9c22d0b3d74888b394fcb26ca759a0a9a74e762f06c1d655b1dcb94b791c5d4be3880c6a127d8b5c0d149317fa4f25e1a2550026a14bde62b3743de7b058557c
+  data.tar.gz: 13ca6cd11f437d9c1b25e6dbd380f018674b1b80b7b4fece7c3a91c042b59b9dc8f8e1bc6a42a14325ec89ab4af4b022d4c3bd00e15aca6cb6189082e03d099a

data/CHANGELOG.md

@@ -1,5 +1,14 @@
 ## master (unreleased)
 
+## 0.2.0 (2022-11-11)
+
+- Fix storing run metadata when the job fails for sidekiq < 6.5.2
+
+- Make enumerators resume from the last cursor position
+
+  This fixes `NestedEnumerator` to work correctly. Previously, each intermediate enumerator
+  was resumed from the next cursor position, possibly skipping remaining inner items.
+
 ## 0.1.0 (2022-11-02)
 
 - First release

data/README.md

@@ -2,7 +2,7 @@
 
 [![Build Status](https://github.com/fatkodima/sidekiq-iteration/actions/workflows/ci.yml/badge.svg?branch=master)](https://github.com/fatkodima/sidekiq-iteration/actions/workflows/ci.yml)
 
-Meet Iteration, an extension for [Sidekiq](https://github.com/mperham/sidekiq) that makes your jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
+Meet Iteration, an extension for [Sidekiq](https://github.com/mperham/sidekiq) that makes your long-running jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
 
 ## Background
 
@@ -136,10 +136,10 @@ class BatchesJob
 end
 ```
 
-### Iterating over batches of Active Record Relations
+### Iterating over Active Record Relations
 
 ```ruby
-class BatchesAsRelationJob
+class RelationsJob
   include Sidekiq::Job
   include SidekiqIteration::Iteration
 
@@ -151,14 +151,14 @@ class BatchesAsRelationJob
     )
   end
 
-  def each_iteration(batch_of_comments, product_id)
-    # batch_of_comments will be a Comment::ActiveRecord_Relation
-    batch_of_comments.update_all(deleted: true)
+  def each_iteration(comments_relation, product_id)
+    # comments_relation will be a Comment::ActiveRecord_Relation
+    comments_relation.update_all(deleted: true)
   end
 end
 ```
 
-### Iterating over arrays
+### Iterating over arbitrary arrays
 
 ```ruby
 class ArrayJob
@@ -228,10 +228,15 @@ For more detailed documentation, see [rubydoc](https://rubydoc.info/gems/sidekiq
 
 ## API
 
-Iteration job must respond to `build_enumerator` and `each_iteration` methods. `build_enumerator` must return [`Enumerator`](https://ruby-doc.org/core-3.1.2/Enumerator.htmll) object that respects the `cursor` value.
+Iteration job must respond to `build_enumerator` and `each_iteration` methods. `build_enumerator` must return [`Enumerator`](https://ruby-doc.org/core-3.1.2/Enumerator.html) object that respects the `cursor` value.
 
 ## FAQ
 
+**Advantages of this pattern over splitting a large job into many small jobs?**
+* Having one job is easier for redis in terms of memory, time and # of requests needed for enqueuing.
+* It simplifies sidekiq monitoring, because you have a predictable number of jobs in the queues, instead of having thousands of them at one time and millions at another. Also easier to navigate its web UI.
+* You can stop/pause/delete just one job, if something goes wrong. With many jobs it is harder and can take a long time, if it is critical to stop it right now.
+
 **Why can't I just iterate in `#perform` method and do whatever I want?** You can, but then your job has to comply with a long list of requirements, such as the ones above. This creates leaky abstractions more easily, when instead we can expose a more powerful abstraction for developers without exposing the underlying infrastructure.
 
 **What happens when my job is interrupted?** A checkpoint will be persisted to Redis after the current `each_iteration`, and the job will be re-enqueued. Once it's popped off the queue, the worker will work off from the next iteration.

data/guides/best-practices.md

@@ -1,5 +1,10 @@
 # Best practices
 
+## Considerations when writing jobs
+
+* Duration of `#each_iteration`: processing a single element from the enumerator builded in `#build_enumerator` should take less than 25 seconds, or the duration set as a timeout for Sidekiq. It allows the job to be safely interrupted and resumed.
+* Idempotency of `#each_iteration`: it should be safe to run `#each_iteration` multiple times for the same element from the enumerator. Read more in [this Sidekiq best practice](https://github.com/mperham/sidekiq/wiki/Best-Practices#2-make-your-job-idempotent-and-transactional). It's important if the job errors and you run it again, because the same element that errored the job may be processed again. It especially matters in the situation described above, when the iteration duration exceeds the timeout: if the job is re-enqueued, multiple elements may be processed again.
+
 ## Batch iteration
 
 Regardless of the active record enumerator used in the task, `sidekiq-iteration` gem loads records in batches of 100 (by default).

data/lib/sidekiq_iteration/active_record_enumerator.rb

@@ -1,28 +1,45 @@
 # frozen_string_literal: true
 
-require_relative "active_record_cursor"
-
 module SidekiqIteration
-  # Builds Enumerator based on ActiveRecord Relation. Supports enumerating on rows and batches.
   # @private
   class ActiveRecordEnumerator
-    SQL_DATETIME_WITH_NSEC = "%Y-%m-%d %H:%M:%S.%N"
+    SQL_DATETIME_WITH_NSEC = "%Y-%m-%d %H:%M:%S.%6N"
 
     def initialize(relation, columns: nil, batch_size: 100, cursor: nil)
       unless relation.is_a?(ActiveRecord::Relation)
         raise ArgumentError, "relation must be an ActiveRecord::Relation"
       end
 
-      @relation = relation
+      @primary_key = "#{relation.table_name}.#{relation.primary_key}"
+      @columns = Array(columns&.map(&:to_s) || @primary_key)
+      @primary_key_index = @columns.index(@primary_key) || @columns.index(relation.primary_key)
+      @pluck_columns = if @primary_key_index
+                         @columns
+                       else
+                         @columns + [@primary_key]
+                       end
       @batch_size = batch_size
-      @columns = Array(columns || "#{relation.table_name}.#{relation.primary_key}")
-      @cursor = cursor
+      @cursor = Array.wrap(cursor)
+      raise ArgumentError, "Must specify at least one column" if @columns.empty?
+      if relation.joins_values.present? && !@columns.all?(/\./)
+        raise ArgumentError, "You need to specify fully-qualified columns if you join a table"
+      end
+
+      if relation.arel.orders.present? || relation.arel.taken.present?
+        raise ArgumentError,
+          "The relation cannot use ORDER BY or LIMIT due to the way how iteration with a cursor is designed. " \
+          "You can use other ways to limit the number of rows, e.g. a WHERE condition on the primary key column."
+      end
+
+      @base_relation = relation.reorder(@columns.join(", "))
+      @iteration_count = 0
     end
 
     def records
-      Enumerator.new(-> { size }) do |yielder|
+      Enumerator.new(-> { records_size }) do |yielder|
         batches.each do |batch, _|
           batch.each do |record|
+            @iteration_count += 1
             yielder.yield(record, cursor_value(record))
           end
         end
@@ -30,40 +47,145 @@ module SidekiqIteration
     end
 
     def batches
-      cursor = ActiveRecordCursor.new(@relation, @columns, @cursor)
-      Enumerator.new(-> { size }) do |yielder|
-        while (records = cursor.next_batch(@batch_size))
-          yielder.yield(records, cursor_value(records.last)) if records.any?
+      Enumerator.new(-> { records_size }) do |yielder|
+        while (batch = next_batch(load: true))
+          @iteration_count += 1
+          yielder.yield(batch, cursor_value(batch.last))
         end
       end
     end
 
-    def size
-      @relation.count(:all)
+    def relations
+      Enumerator.new(-> { relations_size }) do |yielder|
+        while (batch = next_batch(load: false))
+          @iteration_count += 1
+          yielder.yield(batch, unwrap_array(@cursor))
+        end
+      end
     end
 
     private
+      def records_size
+        @base_relation.count(:all)
+      end
+
+      def relations_size
+        (records_size + @batch_size - 1) / @batch_size # ceiling division
+      end
+
+      def next_batch(load:)
+        batch_relation = @base_relation.limit(@batch_size)
+        if conditions.any?
+          batch_relation = batch_relation.where(*conditions)
+        end
+
+        records = nil
+        cursor_values, ids = batch_relation.uncached do
+          if load
+            records = batch_relation.records
+            pluck_columns(records)
+          else
+            pluck_columns(batch_relation)
+          end
+        end
+
+        cursor = cursor_values.last
+        return unless cursor.present?
+
+        # The primary key was plucked, but original cursor did not include it, so we should remove it
+        cursor.pop unless @primary_key_index
+        @cursor = Array.wrap(cursor)
+
+        # Yields relations by selecting the primary keys of records in the batch.
+        # Post.where(published: nil) results in an enumerator of relations like:
+        # Post.where(published: nil, ids: batch_of_ids)
+        relation = @base_relation.where(@primary_key => ids)
+        relation.send(:load_records, records) if load
+        relation
+      end
+
+      def pluck_columns(batch)
+        columns =
+          if batch.is_a?(Array)
+            @pluck_columns.map { |column| column.to_s.split(".").last }
+          else
+            @pluck_columns
+          end
+
+        if columns.size == 1 # only the primary key
+          column_values = batch.pluck(columns.first)
+          return [column_values, column_values]
+        end
+
+        column_values = batch.pluck(*columns)
+        primary_key_index = @primary_key_index || -1
+        primary_key_values = column_values.map { |values| values[primary_key_index] }
+
+        serialize_column_values!(column_values)
+        [column_values, primary_key_values]
+      end
+
       def cursor_value(record)
         positions = @columns.map do |column|
           attribute_name = column.to_s.split(".").last
-          column_value(record, attribute_name)
+          column_value(record[attribute_name])
+        end
+
+        unwrap_array(positions)
+      end
+
+      def conditions
+        return [] if @cursor.empty?
+
+        binds = []
+        sql = build_starts_after_conditions(0, binds)
+
+        # Start from the record pointed by cursor.
+        # We use the property that `>=` is equivalent to `> or =`.
+        if @iteration_count == 0
+          binds.unshift(*@cursor)
+          columns_equality = @columns.map { |column| "#{column} = ?" }.join(" AND ")
+          sql = "(#{columns_equality}) OR (#{sql})"
         end
 
-        if positions.size == 1
-          positions.first
+        [sql, *binds]
+      end
+
+      # (x, y) > (a, b) iff (x > a or (x = a and y > b))
+      def build_starts_after_conditions(index, binds)
+        column = @columns[index]
+
+        if index < @cursor.size - 1
+          binds << @cursor[index] << @cursor[index]
+          "#{column} > ? OR (#{column} = ? AND (#{build_starts_after_conditions(index + 1, binds)}))"
         else
-          positions
+          binds << @cursor[index]
+          if @columns.size == @cursor.size
+            "#{column} > ?"
+          else
+            "#{column} >= ?"
+          end
         end
       end
 
-      def column_value(record, attribute)
-        value = record.read_attribute(attribute.to_sym)
-        case record.class.columns_hash.fetch(attribute).type
-        when :datetime
+      def serialize_column_values!(column_values)
+        column_values.map! { |values| values.map! { |value| column_value(value) } }
+      end
+
+      def column_value(value)
+        if value.is_a?(Time)
           value.strftime(SQL_DATETIME_WITH_NSEC)
         else
           value
         end
       end
+
+      def unwrap_array(array)
+        if array.size == 1
+          array.first
+        else
+          array
+        end
+      end
   end
 end

data/lib/sidekiq_iteration/csv_enumerator.rb

@@ -49,7 +49,7 @@ module SidekiqIteration
     def rows(cursor:)
       @csv.lazy
         .each_with_index
-        .drop(count_of_processed_rows(cursor))
+        .drop(cursor || 0)
         .to_enum { count_of_rows_in_file }
     end
 
@@ -60,7 +60,7 @@ module SidekiqIteration
       @csv.lazy
         .each_slice(batch_size)
         .with_index
-        .drop(count_of_processed_rows(cursor))
+        .drop(cursor || 0)
         .to_enum { (count_of_rows_in_file.to_f / batch_size).ceil }
     end
 
@@ -73,13 +73,5 @@ module SidekiqIteration
         count -= 1 if @csv.headers
         count
       end
-
-      def count_of_processed_rows(cursor)
-        if cursor
-          cursor + 1
-        else
-          0
-        end
-      end
   end
 end

data/lib/sidekiq_iteration/enumerators.rb

@@ -1,7 +1,6 @@
 # frozen_string_literal: true
 
 require_relative "active_record_enumerator"
-require_relative "active_record_batch_enumerator"
 require_relative "csv_enumerator"
 require_relative "nested_enumerator"
 
@@ -22,8 +21,7 @@ module SidekiqIteration
         raise ArgumentError, "array cannot contain ActiveRecord objects"
       end
 
-      drop = cursor ? cursor + 1 : 0
-      array.each_with_index.drop(drop).to_enum { array.size }
+      array.each_with_index.drop(cursor || 0).to_enum { array.size }
     end
 
     # Builds Enumerator from Active Record Relation. Each Enumerator tick moves the cursor one row forward.
@@ -115,7 +113,7 @@ module SidekiqIteration
     #   end
     #
     def active_record_relations_enumerator(scope, cursor:, **options)
-      ActiveRecordBatchEnumerator.new(scope, cursor: cursor, **options).each
+      ActiveRecordEnumerator.new(scope, cursor: cursor, **options).relations
     end
 
     # Builds Enumerator from a CSV file.

data/lib/sidekiq_iteration/job_retry_patch.rb

@@ -7,6 +7,17 @@ module SidekiqIteration
   module JobRetryPatch
     private
       def process_retry(jobinst, msg, queue, exception)
+        add_sidekiq_iteration_metadata(jobinst, msg)
+        super
+      end
+
+      # The method was renamed in https://github.com/mperham/sidekiq/commit/0676a5202e89aa9da4ad7991f4111b97a9d8a0a4.
+      def attempt_retry(jobinst, msg, queue, exception)
+        add_sidekiq_iteration_metadata(jobinst, msg)
+        super
+      end
+
+      def add_sidekiq_iteration_metadata(jobinst, msg)
         if jobinst.is_a?(Iteration)
           unless msg["args"].last.is_a?(Hash)
             msg["args"].push({})
@@ -19,12 +30,14 @@ module SidekiqIteration
             "total_time" => jobinst.total_time,
           }
         end
-
-        super
       end
   end
 end
 
-if Sidekiq::JobRetry.instance_method(:process_retry)
+if Sidekiq::JobRetry.private_method_defined?(:process_retry) ||
+   Sidekiq::JobRetry.private_method_defined?(:attempt_retry)
   Sidekiq::JobRetry.prepend(SidekiqIteration::JobRetryPatch)
+else
+  raise "Sidekiq #{Sidekiq::VERSION} removed the #process_retry method. " \
+        "Please open an issue at the `sidekiq-iteration` gem."
 end

data/lib/sidekiq_iteration/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module SidekiqIteration
-  VERSION = "0.1.0"
+  VERSION = "0.2.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq-iteration
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.2.0
 platform: ruby
 authors:
 - fatkodima
@@ -9,7 +9,7 @@ authors:
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-11-02 00:00:00.000000000 Z
+date: 2022-11-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sidekiq
@@ -41,8 +41,6 @@ files:
 - guides/throttling.md
 - lib/sidekiq-iteration.rb
 - lib/sidekiq_iteration.rb
-- lib/sidekiq_iteration/active_record_batch_enumerator.rb
-- lib/sidekiq_iteration/active_record_cursor.rb
 - lib/sidekiq_iteration/active_record_enumerator.rb
 - lib/sidekiq_iteration/csv_enumerator.rb
 - lib/sidekiq_iteration/enumerators.rb

data/lib/sidekiq_iteration/active_record_batch_enumerator.rb

@@ -1,127 +0,0 @@
-# frozen_string_literal: true
-
-module SidekiqIteration
-  # Batch Enumerator based on ActiveRecord Relation.
-  # @private
-  class ActiveRecordBatchEnumerator
-    include Enumerable
-
-    SQL_DATETIME_WITH_NSEC = "%Y-%m-%d %H:%M:%S.%N"
-
-    def initialize(relation, columns: nil, batch_size: 100, cursor: nil)
-      @primary_key = "#{relation.table_name}.#{relation.primary_key}"
-      @columns = Array(columns&.map(&:to_s) || @primary_key)
-      @primary_key_index = @columns.index(@primary_key) || @columns.index(relation.primary_key)
-      @pluck_columns = if @primary_key_index
-                         @columns
-                       else
-                         @columns + [@primary_key]
-                       end
-      @batch_size = batch_size
-      @cursor = Array.wrap(cursor)
-      @initial_cursor = @cursor
-      raise ArgumentError, "Must specify at least one column" if @columns.empty?
-      if relation.joins_values.present? && !@columns.all?(/\./)
-        raise ArgumentError, "You need to specify fully-qualified columns if you join a table"
-      end
-
-      if relation.arel.orders.present? || relation.arel.taken.present?
-        raise ArgumentError,
-          "The relation cannot use ORDER BY or LIMIT due to the way how iteration with a cursor is designed. " \
-          "You can use other ways to limit the number of rows, e.g. a WHERE condition on the primary key column."
-      end
-
-      @base_relation = relation.reorder(@columns.join(", "))
-    end
-
-    def each
-      return to_enum { size } unless block_given?
-
-      while (relation = next_batch)
-        yield relation, cursor_value
-      end
-    end
-
-    def size
-      (@base_relation.count(:all) + @batch_size - 1) / @batch_size # ceiling division
-    end
-
-    private
-      def next_batch
-        relation = @base_relation.limit(@batch_size)
-        if conditions.any?
-          relation = relation.where(*conditions)
-        end
-
-        cursor_values, ids = relation.uncached do
-          pluck_columns(relation)
-        end
-
-        cursor = cursor_values.last
-        unless cursor.present?
-          @cursor = @initial_cursor
-          return
-        end
-        # The primary key was plucked, but original cursor did not include it, so we should remove it
-        cursor.pop unless @primary_key_index
-        @cursor = Array.wrap(cursor)
-
-        # Yields relations by selecting the primary keys of records in the batch.
-        # Post.where(published: nil) results in an enumerator of relations like:
-        # Post.where(published: nil, ids: batch_of_ids)
-        @base_relation.where(@primary_key => ids)
-      end
-
-      def pluck_columns(relation)
-        if @pluck_columns.size == 1 # only the primary key
-          column_values = relation.pluck(*@pluck_columns)
-          return [column_values, column_values]
-        end
-
-        column_values = relation.pluck(*@pluck_columns)
-        primary_key_index = @primary_key_index || -1
-        primary_key_values = column_values.map { |values| values[primary_key_index] }
-
-        serialize_column_values!(column_values)
-        [column_values, primary_key_values]
-      end
-
-      def cursor_value
-        if @cursor.size == 1
-          @cursor.first
-        else
-          @cursor
-        end
-      end
-
-      def conditions
-        column_index = @cursor.size - 1
-        column = @columns[column_index]
-        where_clause = if @columns.size == @cursor.size
-                         "#{column} > ?"
-                       else
-                         "#{column} >= ?"
-                       end
-        while column_index > 0
-          column_index -= 1
-          column = @columns[column_index]
-          where_clause = "#{column} > ? OR (#{column} = ? AND (#{where_clause}))"
-        end
-        ret = @cursor.reduce([where_clause]) { |params, value| params << value << value }
-        ret.pop
-        ret
-      end
-
-      def serialize_column_values!(column_values)
-        column_values.map! { |values| values.map! { |value| column_value(value) } }
-      end
-
-      def column_value(value)
-        if value.is_a?(Time)
-          value.strftime(SQL_DATETIME_WITH_NSEC)
-        else
-          value
-        end
-      end
-  end
-end

data/lib/sidekiq_iteration/active_record_cursor.rb

@@ -1,89 +0,0 @@
-# frozen_string_literal: true
-
-module SidekiqIteration
-  # @private
-  class ActiveRecordCursor
-    include Comparable
-
-    attr_reader :position, :reached_end
-
-    def initialize(relation, columns = nil, position = nil)
-      columns ||= "#{relation.table_name}.#{relation.primary_key}"
-      @columns = Array.wrap(columns)
-      raise ArgumentError, "Must specify at least one column" if @columns.empty?
-
-      self.position = Array.wrap(position)
-      if relation.joins_values.present? && !@columns.all?(/\./)
-        raise ArgumentError, "You need to specify fully-qualified columns if you join a table"
-      end
-
-      if relation.arel.orders.present? || relation.arel.taken.present?
-        raise ArgumentError,
-          "The relation cannot use ORDER BY or LIMIT due to the way how iteration with a cursor is designed. " \
-          "You can use other ways to limit the number of rows, e.g. a WHERE condition on the primary key column."
-      end
-
-      @base_relation = relation.reorder(@columns.join(", "))
-      @reached_end = false
-    end
-
-    def <=>(other)
-      if reached_end == other.reached_end
-        position <=> other.position
-      else
-        reached_end ? 1 : -1
-      end
-    end
-
-    def position=(position)
-      raise ArgumentError, "Cursor position cannot contain nil values" if position.any?(&:nil?)
-
-      @position = position
-    end
-
-    def next_batch(batch_size)
-      return if @reached_end
-
-      relation = @base_relation.limit(batch_size)
-
-      if (conditions = self.conditions).any?
-        relation = relation.where(*conditions)
-      end
-
-      records = relation.uncached do
-        relation.to_a
-      end
-
-      update_from_record(records.last) if records.any?
-      @reached_end = records.size < batch_size
-
-      records if records.any?
-    end
-
-    private
-      def conditions
-        i = @position.size - 1
-        column = @columns[i]
-        conditions = if @columns.size == @position.size
-                       "#{column} > ?"
-                     else
-                       "#{column} >= ?"
-                     end
-        while i > 0
-          i -= 1
-          column = @columns[i]
-          conditions = "#{column} > ? OR (#{column} = ? AND (#{conditions}))"
-        end
-        ret = @position.reduce([conditions]) { |params, value| params << value << value }
-        ret.pop
-        ret
-      end
-
-      def update_from_record(record)
-        self.position = @columns.map do |column|
-          method = column.to_s.split(".").last
-          record.send(method)
-        end
-      end
-  end
-end