job-iteration 1.1.11 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 18324439fe7b98c7f1bca543c078ddbfdff18af5a7e97a87b21b48890e919769
-  data.tar.gz: 3d912ea06a5a66ee841fbd605e8dfc414bd1760c61d9edf74445e79ae16a6466
+  metadata.gz: eff37be02274e3c0db10de1cf3e49a10eea5b5b838b19800d143853aee94c08f
+  data.tar.gz: '085dc8b7eec8897393461e43fb4ece8acaddc6705165a2b6de0856cef54b1215'
 SHA512:
-  metadata.gz: 622368d1208ea23014188c028f832f51f899b6bdfbca8938c6396dcca48f913679ee7f41a035c5cbe0376346ea40a583c4615bd665251cf7dc36a30aeb37904a
-  data.tar.gz: 51931d565cffb4600141e1e96dc9a1a6b8a522589c8b0b38dc4cd1846e8d59a9f150d0d0bfcde4aaee587478af57418093a9e313b2f17c9f2c54ffb6cf18382a
+  metadata.gz: 22352326237ca16a23fc4177611187d4c6d426c07397caecd09165303726ce9ba00acaa882e35fb580b19bbe53f056ddde642298268a329dd02c88f61f82b5b7
+  data.tar.gz: 583406e5180f2721895673aa0410a390dce378c9c30e625151b5455fa8c1331867e94cc1cf2518b8b29cb0fbe908af517ee88a88429fd5f71f34a80617f5d8e6

data/.github/workflows/ci.yml

@@ -13,11 +13,9 @@ jobs:
         - 6379:6379
     strategy:
       matrix:
-        ruby: [2.5, 2.6, 2.7, 3.0]
+        ruby: [2.6, 2.7, 3.0]
         gemfile: [rails_5_2, rails_6_0, rails_edge]
         exclude:
-          - ruby: 2.5
-            gemfile: rails_edge
           - ruby: 2.6
             gemfile: rails_edge
           - ruby: 3.0

data/.rubocop.yml

@@ -2,7 +2,7 @@ inherit_gem:
   rubocop-shopify: rubocop.yml
 
 AllCops:
-  TargetRubyVersion: 2.4.4
+  TargetRubyVersion: 2.6.5
   Exclude:
     - 'vendor/bundle/**/*'
 Lint/SuppressedException:

data/CHANGELOG.md

@@ -1,15 +1,35 @@
 ### Master (unreleased)
 
+## v1.2.0 (Sept 21, 2021)
+- [107](https://github.com/Shopify/job-iteration/pull/107) - Remove broken links from README
+- [108](https://github.com/Shopify/job-iteration/pull/108) - Drop support for ruby 2.5
+- [110](https://github.com/Shopify/job-iteration/pull/110) - Update rubocop TargetRubyVersion
+
+## v1.1.14 (May 28, 2021)
+
+#### Bug fix
+- [84](https://github.com/Shopify/job-iteration/pull/84) - Call adjust_total_time before running on_complete callbacks
+- [94](https://github.com/Shopify/job-iteration/pull/94) - Remove unnecessary break
+- [95](https://github.com/Shopify/job-iteration/pull/95) - ActiveRecordBatchEnumerator#each should rewind at the end
+- [97](https://github.com/Shopify/job-iteration/pull/97) - Batch enumerator size returns the number of batches, not records
+
+## v1.1.13 (May 20, 2021)
+
 #### New feature
+- [91](https://github.com/Shopify/job-iteration/pull/91) - Add enumerator yielding batches as Active Record Relations
+
+## v1.1.12 (April 19, 2021)
 
 #### Bug fix
 
+- [77](https://github.com/Shopify/job-iteration/pull/77) - Defer enforce cursor be serializable until 2.0.0
 
 ## v1.1.11 (April 19, 2021)
 
 #### Bug fix
 
 - [73](https://github.com/Shopify/job-iteration/pull/73) - Enforce cursor be serializable
+  _This is reverted in 1.1.12 as it breaks behaviour in some apps._
 
 ## v1.1.10 (March 30, 2021)
 

data/README.md

@@ -1,6 +1,6 @@
 # Job Iteration API
 
-[![Build Status](https://travis-ci.com/Shopify/job-iteration.svg?branch=master)](https://travis-ci.com/Shopify/job-iteration)
+[![CI](https://github.com/Shopify/job-iteration/actions/workflows/ci.yml/badge.svg)](https://github.com/Shopify/job-iteration/actions/workflows/ci.yml)
 
 Meet Iteration, an extension for [ActiveJob](https://github.com/rails/rails/tree/master/activejob) that makes your jobs interruptible and resumable, saving all progress that the job has made (aka checkpoint for jobs).
 
@@ -77,7 +77,28 @@ class BatchesJob < ApplicationJob
 
   def each_iteration(batch_of_comments, product_id)
     # batch_of_comments will contain batches of 100 records
-    Comment.where(id: batch_of_comments.map(&:id)).update_all(deleted: true)
+    batch_of_comments.each do |comment|
+      DeleteCommentJob.perform_later(comment)
+    end
+  end
+end
+```
+
+```ruby
+class BatchesAsRelationJob < ApplicationJob
+  include JobIteration::Iteration
+
+  def build_enumerator(product_id, cursor:)
+    enumerator_builder.active_record_on_batch_relations(
+      Product.find(product_id).comments,
+      cursor: cursor,
+      batch_size: 100,
+    )
+  end
+
+  def each_iteration(batch_of_comments, product_id)
+    # batch_of_comments will be a Comment::ActiveRecord_Relation
+    batch_of_comments.update_all(deleted: true)
   end
 end
 ```
@@ -150,7 +171,7 @@ There a few configuration assumptions that are required for Iteration to work wi
 
 **Why is it important that `each_iteration` takes less than 30 seconds?** When the job worker is scheduled for restart or shutdown, it gets a notice to finish remaining unit of work. To guarantee that no progress is lost we need to make sure that `each_iteration` completes within a reasonable amount of time.
 
-**What do I do if each iteration takes a long time, because it's doing nested operations?** If your `each_iteration` is complex, we recommend enqueuing another job, which will run your nested business logic. We may expose primitives in the future to do this more effectively, but this is not terribly common today. We recommend to read https://goo.gl/UobaaU to learn more about nested operations.
+**What do I do if each iteration takes a long time, because it's doing nested operations?** If your `each_iteration` is complex, we recommend enqueuing another job, which will run your nested business logic. We may expose primitives in the future to do this more effectively, but this is not terribly common today.
 
 **Why do I use have to use this ugly helper in `build_enumerator`? Why can't you automatically infer it?** This is how the first version of the API worked. We checked the type of object returned by `build_enumerable`, and whether it was ActiveRecord Relation or an Array, we used the matching adapter. This caused opaque type branching in Iteration internals and it didn’t allow developers to craft their own Enumerators and control the cursor value. We made a decision to _always_ return Enumerator instance from `build_enumerator`. Now we provide explicit helpers to convert ActiveRecord Relation or an Array to Enumerator, and for more complex iteration flows developers can build their own `Enumerator` objects.
 

data/bin/test

@@ -0,0 +1,32 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+def main
+  begin
+    command = create_command
+  rescue ArgumentError => e
+    abort(e.message)
+  end
+  puts "Running #{command.join(" ")}"
+  system(*command)
+end
+
+def create_command
+  case ARGV.length
+  when 0
+    ["bundle", "exec", "rake", "test"]
+  when 1
+    filename = ARGV[0]
+    ["bundle", "exec", "rake", "test", "TEST=#{filename}"]
+  when 2
+    filename = ARGV[0]
+    test_name = ARGV[1]
+    test_name_with_underscores = test_name.tr(" ", "_")
+    test_name_pattern = "/#{Regexp.escape(test_name_with_underscores)}/"
+    ["bundle", "exec", "rake", "test", "TEST=#{filename}", "TESTOPTS=\"--name=#{test_name_pattern} -v\""]
+  else
+    raise ArgumentError, "Too many arguments. Did you forget to put the test name in quotes?"
+  end
+end
+
+main

data/dev.yml

@@ -13,7 +13,43 @@ up:
   - custom:
       name: Create Job Iteration database
       meet: mysql -uroot -h job-iteration.railgun -e "CREATE DATABASE job_iteration_test"
-      met?: mysql -uroot -h job-iteration.railgun job_iteration_test -e "SELECT 1"
+      met?: mysql -uroot -h job-iteration.railgun job_iteration_test -e "SELECT 1" &> /dev/null
 
 commands:
-  test: bundle exec rake
+  test:
+    run:  bin/test "$@"
+    syntax:
+      optional: filename testnamepattern
+    aliases: [t]
+    desc: run tests
+    long_desc: |
+      {{bold:Default}}
+      =======
+      Run the entire test suite.
+
+        Examples:
+        {{command:dev test}}
+        {{command:dev t}}
+
+      {{bold:Run all tests in a file}}
+      ========================
+      Include the file path.
+
+        Example:
+        {{command:dev test test/unit/iteration_test.rb}}
+
+      {{bold:Run a single test in a given file}}
+      ========================
+      Include the file path and the name of the test you'd like to run.
+
+        Example:
+        {{command:dev test test/unit/iteration_test.rb test_that_it_has_a_version_number}}
+
+      {{bold:Run all tests in a given file whose name contains a string}}
+      ========================
+      Include the file path and the string that the test names should contain.
+
+        Example:
+        {{command:dev test test/unit/iteration_test.rb version_number}}
+  style:
+    run: bundle exec rubocop -a

data/job-iteration.gemspec

@@ -5,9 +5,10 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require "job-iteration/version"
 
 Gem::Specification.new do |spec|
+  spec.required_ruby_version = ">= 2.6"
   spec.name          = "job-iteration"
   spec.version       = JobIteration::VERSION
-  spec.authors       = %w(Shopify)
+  spec.authors       = ["Shopify"]
   spec.email         = ["ops-accounts+shipit@shopify.com"]
 
   spec.summary       = "Makes your background jobs interruptible and resumable."
@@ -20,7 +21,7 @@ Gem::Specification.new do |spec|
   end
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
-  spec.require_paths = %w(lib)
+  spec.require_paths = ["lib"]
 
   spec.metadata["changelog_uri"] = "https://github.com/Shopify/job-iteration/blob/master/CHANGELOG.md"
   spec.metadata["allowed_push_host"] = "https://rubygems.org"

data/lib/job-iteration/active_record_batch_enumerator.rb

@@ -0,0 +1,117 @@
+# frozen_string_literal: true
+
+module JobIteration
+  # Builds Batch Enumerator based on ActiveRecord Relation.
+  # @see EnumeratorBuilder
+  class ActiveRecordBatchEnumerator
+    include Enumerable
+
+    SQL_DATETIME_WITH_NSEC = "%Y-%m-%d %H:%M:%S.%N"
+
+    def initialize(relation, columns: nil, batch_size: 100, cursor: nil)
+      @batch_size = batch_size
+      @primary_key = "#{relation.table_name}.#{relation.primary_key}"
+      @columns = Array(columns&.map(&:to_s) || @primary_key)
+      @primary_key_index = @columns.index(@primary_key) || @columns.index(relation.primary_key)
+      @pluck_columns = if @primary_key_index
+        @columns
+      else
+        @columns.dup << @primary_key
+      end
+      @cursor = Array.wrap(cursor)
+      @initial_cursor = @cursor
+      raise ArgumentError, "Must specify at least one column" if @columns.empty?
+      if relation.joins_values.present? && !@columns.all? { |column| column.to_s.include?(".") }
+        raise ArgumentError, "You need to specify fully-qualified columns if you join a table"
+      end
+
+      if relation.arel.orders.present? || relation.arel.taken.present?
+        raise ConditionNotSupportedError
+      end
+
+      @base_relation = relation.reorder(@columns.join(","))
+    end
+
+    def each
+      return to_enum { size } unless block_given?
+      while (relation = next_batch)
+        yield relation, cursor_value
+      end
+    end
+
+    def size
+      (@base_relation.count + @batch_size - 1) / @batch_size # ceiling division
+    end
+
+    private
+
+    def next_batch
+      relation = @base_relation.limit(@batch_size)
+      if conditions.any?
+        relation = relation.where(*conditions)
+      end
+
+      cursor_values, ids = relation.uncached do
+        pluck_columns(relation)
+      end
+
+      cursor = cursor_values.last
+      unless cursor.present?
+        @cursor = @initial_cursor
+        return
+      end
+      # The primary key was plucked, but original cursor did not include it, so we should remove it
+      cursor.pop unless @primary_key_index
+      @cursor = Array.wrap(cursor)
+
+      # Yields relations by selecting the primary keys of records in the batch.
+      # Post.where(published: nil) results in an enumerator of relations like: Post.where(ids: batch_of_ids)
+      @base_relation.where(@primary_key => ids)
+    end
+
+    def pluck_columns(relation)
+      if @pluck_columns.size == 1 # only the primary key
+        column_values = relation.pluck(*@pluck_columns)
+        return [column_values, column_values]
+      end
+
+      column_values = relation.pluck(*@pluck_columns)
+      primary_key_index = @primary_key_index || -1
+      primary_key_values = column_values.map { |values| values[primary_key_index] }
+
+      serialize_column_values!(column_values)
+      [column_values, primary_key_values]
+    end
+
+    def cursor_value
+      return @cursor.first if @cursor.size == 1
+      @cursor
+    end
+
+    def conditions
+      column_index = @cursor.size - 1
+      column = @columns[column_index]
+      where_clause = if @columns.size == @cursor.size
+        "#{column} > ?"
+      else
+        "#{column} >= ?"
+      end
+      while column_index > 0
+        column_index -= 1
+        column = @columns[column_index]
+        where_clause = "#{column} > ? OR (#{column} = ? AND (#{where_clause}))"
+      end
+      ret = @cursor.reduce([where_clause]) { |params, value| params << value << value }
+      ret.pop
+      ret
+    end
+
+    def serialize_column_values!(column_values)
+      column_values.map! { |values| values.map! { |value| column_value(value) } }
+    end
+
+    def column_value(value)
+      value.is_a?(Time) ? value.strftime(SQL_DATETIME_WITH_NSEC) : value
+    end
+  end
+end

data/lib/job-iteration/enumerator_builder.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
+require_relative "./active_record_batch_enumerator"
 require_relative "./active_record_enumerator"
 require_relative "./csv_enumerator"
 require_relative "./throttle_enumerator"
@@ -86,6 +87,11 @@ module JobIteration
     #        WHERE (created_at > '$LAST_CREATED_AT_CURSOR'
     #          OR (created_at = '$LAST_CREATED_AT_CURSOR' AND (id > '$LAST_ID_CURSOR')))
     #        ORDER BY created_at, id LIMIT 100
+    #
+    # As a result of this query pattern, if the values in these columns change for the records in scope during
+    # iteration, they may be skipped or yielded multiple times depending on the nature of the update and the
+    # cursor's value. If the value gets updated to a greater value than the cursor's value, it will get yielded
+    # again. Similarly, if the value gets updated to a lesser value than the curor's value, it will get skipped.
     def build_active_record_enumerator_on_records(scope, cursor:, **args)
       enum = build_active_record_enumerator(
         scope,
@@ -95,7 +101,7 @@ module JobIteration
       wrap(self, enum)
     end
 
-    # Builds Enumerator from Active Record Relation and enumerates on batches.
+    # Builds Enumerator from Active Record Relation and enumerates on batches of records.
     # Each Enumerator tick moves the cursor +batch_size+ rows forward.
     #
     # +batch_size:+ sets how many records will be fetched in one batch. Defaults to 100.
@@ -110,6 +116,16 @@ module JobIteration
       wrap(self, enum)
     end
 
+    # Builds Enumerator from Active Record Relation and enumerates on batches, yielding Active Record Relations.
+    # See documentation for #build_active_record_enumerator_on_batches.
+    def build_active_record_enumerator_on_batch_relations(scope, cursor:, **args)
+      JobIteration::ActiveRecordBatchEnumerator.new(
+        scope,
+        cursor: cursor,
+        **args
+      ).each
+    end
+
     def build_throttle_enumerator(enum, throttle_on:, backoff:)
       JobIteration::ThrottleEnumerator.new(
         enum,
@@ -124,6 +140,7 @@ module JobIteration
     alias_method :array, :build_array_enumerator
     alias_method :active_record_on_records, :build_active_record_enumerator_on_records
     alias_method :active_record_on_batches, :build_active_record_enumerator_on_batches
+    alias_method :active_record_on_batch_relations, :build_active_record_enumerator_on_batch_relations
     alias_method :throttle, :build_throttle_enumerator
 
     private

data/lib/job-iteration/iteration.rb

@@ -142,7 +142,8 @@ module JobIteration
       arguments = arguments.dup.freeze
       found_record = false
       enumerator.each do |object_from_enumerator, index|
-        assert_valid_cursor!(index)
+        # Deferred until 2.0.0
+        # assert_valid_cursor!(index)
 
         record_unit_of_work do
           found_record = true
@@ -161,6 +162,8 @@ module JobIteration
         "times_interrupted=#{times_interrupted} cursor_position=#{cursor_position}"
       ) unless found_record
 
+      adjust_total_time
+
       true
     end
 
@@ -248,8 +251,6 @@ module JobIteration
     end
 
     def output_interrupt_summary
-      adjust_total_time
-
       message = "[JobIteration::Iteration] Completed iterating. times_interrupted=%d total_time=%.3f"
       logger.info(Kernel.format(message, times_interrupted, total_time))
     end

data/lib/job-iteration/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module JobIteration
-  VERSION = "1.1.11"
+  VERSION = "1.2.0"
 end

data/lib/job-iteration.rb

@@ -38,16 +38,14 @@ module JobIteration
   def load_integrations
     loaded = nil
     INTEGRATIONS.each do |integration|
-      begin
-        load_integration(integration)
-        if loaded
-          raise IntegrationLoadError,
-            "#{loaded} integration has already been loaded, but #{integration} is also available. " \
-            "Iteration will only work with one integration."
-        end
-        loaded = integration
-      rescue LoadError
+      load_integration(integration)
+      if loaded
+        raise IntegrationLoadError,
+          "#{loaded} integration has already been loaded, but #{integration} is also available. " \
+          "Iteration will only work with one integration."
       end
+      loaded = integration
+    rescue LoadError
     end
   end
 

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: job-iteration
 version: !ruby/object:Gem::Version
-  version: 1.1.11
+  version: 1.2.0
 platform: ruby
 authors:
 - Shopify
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-04-19 00:00:00.000000000 Z
+date: 2021-09-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activerecord
@@ -56,6 +56,7 @@ files:
 - README.md
 - Rakefile
 - bin/setup
+- bin/test
 - dev.yml
 - gemfiles/rails_5_2.gemfile
 - gemfiles/rails_6_0.gemfile
@@ -66,6 +67,7 @@ files:
 - guides/throttling.md
 - job-iteration.gemspec
 - lib/job-iteration.rb
+- lib/job-iteration/active_record_batch_enumerator.rb
 - lib/job-iteration/active_record_cursor.rb
 - lib/job-iteration/active_record_enumerator.rb
 - lib/job-iteration/csv_enumerator.rb
@@ -91,14 +93,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '0'
+      version: '2.6'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3
+rubygems_version: 3.2.20
 signing_key: 
 specification_version: 4
 summary: Makes your background jobs interruptible and resumable.