job-iteration 1.9.0 → 1.11.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b32eb2fd6eb2dc54f59eff462a3cde3882192134c2964ffc05b737957e3d4abb
-  data.tar.gz: 255864c2fac9d3cc0f75da1929d18c1aeb8e63ff83ea3dc6c19038c0e4894e54
+  metadata.gz: 61c40af6f75909b71f400462b9d64f994530f0fb68367ddbe6479f1b5fe8d831
+  data.tar.gz: a293c68062cd9714d522a2c3c5e81292bcd5a9f8172cbf3f537f0676e0de5731
 SHA512:
-  metadata.gz: 7c241d626e4c92c4424dd08745feab35f1606fafbf6f0d610f34a704d16702a52623f646e5715925b331509257c737053b25bf90091f61a948b6fba2315cc659
-  data.tar.gz: 19b821e84f93954cea21b0709455ac5ad12b2633f70c2182ae93e75d2e883a0985dda1cf7c56ac89295caab1720ef3508857201d0c33f18b35ffad76996f093b
+  metadata.gz: a773c72a419db3fe0677ba323ec0a2bdca6c02e34f322e7519ff080752c66e8ce7c973ca3b9031e84e2b70e27eb61dcf5875dfd38ef0b4421983991202256c71
+  data.tar.gz: 2abedd098be50d634f68eade3e681042aef00a87c8a3b0fb7cc88a91143cafade08bc18435f7479399c2ccbb660563ff781c1b4e9248b8bff7cf22190504383e

data/CHANGELOG.md

@@ -1,5 +1,9 @@
 ### Main (unreleased)
 
+### Breaking Changes
+
+nil
+
 ### Changes
 
 nil
@@ -12,7 +16,30 @@ nil
 
 nil
 
-## v1.9.0 (Feb 3, 2024)
+## v1.11.0 (Jul 14, 2025)
+
+### Security fixes
+
+- [595](https://github.com/Shopify/job-iteration/pull/595) [CVE-2025-53623] Fixes a security issue in the `CSVEnumerator` where the filename was directly interpolated into a bash command.
+
+### Bug fixes
+
+- [590](https://github.com/Shopify/job-iteration/pull/590) Fix a compatibilty issue between the Sorbet DSL compiler and the latest Tapioca.
+- [593](https://github.com/Shopify/job-iteration/pull/593) Properly support required and optional positional arguments in the Sorbet DSL compiler.
+- [594](https://github.com/Shopify/job-iteration/pull/594) Clean up the size calculation in the `CSVEnumerator`.
+
+## v1.10.0 (Mar 20, 2025)
+
+### Breaking Changes
+
+- [544](https://github.com/Shopify/job-iteration/pull/544) - Drop support for Ruby 2.6 and 2.7, and Rails 5.2 and 6.0. The minimum supported Ruby version is now 3.0, and the minimum supported Rails version is 6.1.
+
+### Features
+
+- [547](https://github.com/Shopify/job-iteration/pull/547) Add interruption adapter for [DelayedJob](https://github.com/collectiveidea/delayed_job).
+- [556](https://github.com/Shopify/job-iteration/pull/556) Add support for generic job classes in the Tapioca Sorbet compiler.
+
+## v1.9.0 (Feb 3, 2025)
 
 ### Features
 

data/README.md

@@ -151,7 +151,25 @@ class NestedIterationJob < ApplicationJob
 end
 ```
 
-Iteration hooks into Sidekiq and Resque out of the box to support graceful interruption. No extra configuration is required.
+Iteration hooks into most popular queue adapters out of the box to support graceful interruption. No extra configuration is required.
+
+## Supported dependencies
+
+Job-iteration currently supports the following queue adapters (in order of implementation):
+
+- [Resque](https://github.com/resque/resque)
+- [Sidekiq](https://github.com/sidekiq/sidekiq/)
+- [GoodJob](https://github.com/bensheldon/good_job)
+- [Solid Queue](https://github.com/rails/solid_queue)
+- [Amazon SQS](https://github.com/aws/aws-activejob-sqs-ruby)
+- [Delayed::Job](https://github.com/collectiveidea/delayed_job)
+
+It supports the following platforms:
+
+- Ruby 3.0 and later
+- Rails 6.1 and later
+
+Support for older platforms that have reached end of life may occasionally be dropped if maintaining backwards compatibility is cumbersome.
 
 ## Guides
 

data/job-iteration.gemspec

@@ -5,7 +5,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require "job-iteration/version"
 
 Gem::Specification.new do |spec|
-  spec.required_ruby_version = ">= 2.6"
+  spec.required_ruby_version = ">= 3.0"
   spec.name          = "job-iteration"
   spec.version       = JobIteration::VERSION
   spec.authors       = ["Shopify"]
@@ -16,8 +16,8 @@ Gem::Specification.new do |spec|
   spec.homepage      = "https://github.com/shopify/job-iteration"
   spec.license       = "MIT"
 
-  spec.files         = %x(git ls-files -z).split("\x0").reject do |f|
-    f.match(%r{^(test|spec|features)/})
+  spec.files = ["CHANGELOG.md", "LICENSE.txt", "README.md", "job-iteration.gemspec"] + Dir.glob("{lib,exe}/**/*", File::FNM_DOTMATCH).reject do |f|
+    File.directory?(f) || f.match(%r{^(test|spec|features)/})
   end
   spec.bindir        = "exe"
   spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
@@ -26,5 +26,5 @@ Gem::Specification.new do |spec|
   spec.metadata["changelog_uri"] = "https://github.com/Shopify/job-iteration/blob/main/CHANGELOG.md"
   spec.metadata["allowed_push_host"] = "https://rubygems.org"
 
-  spec.add_dependency("activejob", ">= 5.2")
+  spec.add_dependency("activejob", ">= 6.1")
 end

data/lib/job-iteration/csv_enumerator.rb

@@ -43,23 +43,19 @@ module JobIteration
         .each_slice(batch_size)
         .with_index
         .drop(count_of_processed_rows(cursor))
-        .to_enum { (count_of_rows_in_file.to_f / batch_size).ceil }
+        .to_enum do
+          num_rows = count_of_rows_in_file
+          num_rows.nil? ? nil : (num_rows.to_f / batch_size).ceil
+        end
     end
 
     private
 
     def count_of_rows_in_file
-      # TODO: Remove rescue for NoMethodError when Ruby 2.6 is no longer supported.
-      begin
-        filepath = @csv.path
-      rescue NoMethodError
-        return
-      end
-
-      # Behaviour of CSV#path changed in Ruby 2.6.3 (returns nil instead of raising NoMethodError)
+      filepath = @csv.path
       return unless filepath
 
-      count = %x(wc -l < #{filepath}).strip.to_i
+      count = File.foreach(filepath).count
       count -= 1 if @csv.headers
       count
     end

data/lib/job-iteration/interruption_adapters/delayed_job_adapter.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+begin
+  require "delayed_job"
+rescue LoadError
+  # DelayedJobb is not available, no need to load the adapter
+  return
+end
+
+begin
+  # Delayed::Worker#stop? was introduced in DelayedJob 3.0.3
+  gem("delayed_job", ">= 3.0.3")
+rescue Gem::LoadError
+  warn("job-iteration's interruption adapter for DelayedJob requires DelayedJob 3.0.3 or newer")
+  return
+end
+
+module JobIteration
+  module InterruptionAdapters
+    module DelayedJobAdapter
+      class << self
+        attr_accessor :delayed_worker, :delayed_worker_started
+
+        def call
+          if delayed_worker_started
+            delayed_worker.nil? || delayed_worker.stop?
+          else
+            false
+          end
+        end
+      end
+
+      self.delayed_worker_started = false
+
+      class DelayedJobPlugin < Delayed::Plugin
+        callbacks do |lifecycle|
+          lifecycle.before(:execute) do |worker|
+            DelayedJobAdapter.delayed_worker = worker
+            DelayedJobAdapter.delayed_worker_started = true
+          end
+
+          lifecycle.after(:execute) do |_worker|
+            DelayedJobAdapter.delayed_worker = nil
+          end
+        end
+      end
+      private_constant :DelayedJobPlugin
+
+      Delayed::Worker.plugins << DelayedJobPlugin
+    end
+
+    register(:delayed_job, DelayedJobAdapter)
+  end
+end

data/lib/job-iteration/interruption_adapters.rb

@@ -4,7 +4,7 @@ require_relative "interruption_adapters/null_adapter"
 
 module JobIteration
   module InterruptionAdapters
-    BUNDLED_ADAPTERS = [:good_job, :resque, :sidekiq, :solid_queue, :sqs].freeze # @api private
+    BUNDLED_ADAPTERS = [:delayed_job, :good_job, :resque, :sidekiq, :solid_queue, :sqs].freeze # @api private
 
     class << self
       # Returns adapter for specified name.

data/lib/job-iteration/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module JobIteration
-  VERSION = "1.9.0"
+  VERSION = "1.11.0"
 end

data/lib/tapioca/dsl/compilers/job_iteration.rb

@@ -8,10 +8,11 @@ module Tapioca
     module Compilers
       class JobIteration < Compiler
         extend T::Sig
+        extend T::Generic
 
         ConstantType = type_member { { fixed: T.class_of(::JobIteration::Iteration) } }
         PARAM_TYPES_IN_ORDER = [
-          RBI::Param,
+          RBI::ReqParam,
           RBI::OptParam,
           RBI::RestParam,
           RBI::KwParam,
@@ -55,10 +56,18 @@ module Tapioca
             # Sorbet expects optional keyword arguments to be after required keyword arguments.
             expanded_parameters.sort_by! { |typed_param| PARAM_TYPES_IN_ORDER.index(typed_param.param.class) }
 
+            number_of_generic_type_members = Tapioca::Runtime::GenericTypeRegistry.lookup_type_variables(constant)&.size
+
+            returned_job_class = if number_of_generic_type_members&.nonzero?
+              "#{constant_name}[#{number_of_generic_type_members.times.map { "T.untyped" }.join(", ")}]"
+            else
+              constant_name
+            end
+
             job.create_method(
               "perform_later",
-              parameters: perform_later_parameters(expanded_parameters, constant_name),
-              return_type: "T.any(#{constant_name}, FalseClass)",
+              parameters: perform_later_parameters(expanded_parameters, returned_job_class),
+              return_type: "T.any(#{returned_job_class}, FalseClass)",
               class_method: true,
             )
 
@@ -83,15 +92,15 @@ module Tapioca
         sig do
           params(
             parameters: T::Array[RBI::TypedParam],
-            constant_name: T.nilable(String),
+            returned_job_class: String,
           ).returns(T::Array[RBI::TypedParam])
         end
-        def perform_later_parameters(parameters, constant_name)
+        def perform_later_parameters(parameters, returned_job_class)
           if ::Gem::Requirement.new(">= 7.0").satisfied_by?(::ActiveJob.gem_version)
             parameters.reject! { |typed_param| RBI::BlockParam === typed_param.param }
             parameters + [create_block_param(
               "block",
-              type: "T.nilable(T.proc.params(job: #{constant_name}).void)",
+              type: "T.nilable(T.proc.params(job: #{returned_job_class}).void)",
             )]
           else
             parameters

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: job-iteration
 version: !ruby/object:Gem::Version
-  version: 1.9.0
+  version: 1.11.0
 platform: ruby
 authors:
 - Shopify
 bindir: exe
 cert_chain: []
-date: 2025-02-03 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: activejob
@@ -15,14 +15,14 @@ dependencies:
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '5.2'
+        version: '6.1'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '5.2'
+        version: '6.1'
 description: Makes your background jobs interruptible and resumable.
 email:
 - ops-accounts+shipit@shopify.com
@@ -30,29 +30,9 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- ".github/dependabot.yml"
-- ".github/workflows/ci.yml"
-- ".github/workflows/cla.yml"
-- ".gitignore"
-- ".rubocop.yml"
-- ".ruby-version"
-- ".yardopts"
 - CHANGELOG.md
-- CODE_OF_CONDUCT.md
-- Gemfile
-- Gemfile.lock
 - LICENSE.txt
 - README.md
-- Rakefile
-- bin/setup
-- bin/test
-- dev.yml
-- gemfiles/rails_gems.gemfile
-- guides/argument-semantics.md
-- guides/best-practices.md
-- guides/custom-enumerator.md
-- guides/iteration-how-it-works.md
-- guides/throttling.md
 - job-iteration.gemspec
 - lib/job-iteration.rb
 - lib/job-iteration/active_record_batch_enumerator.rb
@@ -61,6 +41,7 @@ files:
 - lib/job-iteration/csv_enumerator.rb
 - lib/job-iteration/enumerator_builder.rb
 - lib/job-iteration/interruption_adapters.rb
+- lib/job-iteration/interruption_adapters/delayed_job_adapter.rb
 - lib/job-iteration/interruption_adapters/good_job_adapter.rb
 - lib/job-iteration/interruption_adapters/null_adapter.rb
 - lib/job-iteration/interruption_adapters/resque_adapter.rb
@@ -88,14 +69,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: '2.6'
+      version: '3.0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.6.3
+rubygems_version: 3.6.9
 specification_version: 4
 summary: Makes your background jobs interruptible and resumable.
 test_files: []

data/.github/dependabot.yml

@@ -1,16 +0,0 @@
-version: 2
-
-updates:
-
-  - package-ecosystem: bundler
-    directory: '/'
-    versioning-strategy: increase
-    open-pull-requests-limit: 100
-    insecure-external-code-execution: allow
-    schedule:
-      interval: weekly
-
-  - package-ecosystem: github-actions
-    directory: '/'
-    schedule:
-      interval: daily

data/.github/workflows/ci.yml

@@ -1,98 +0,0 @@
-name: CI
-
-on: [push, pull_request]
-
-jobs:
-  build:
-    runs-on: ubuntu-latest
-    name: Ruby ${{ matrix.ruby }} | Rails ${{ matrix.rails }} | Gemfile ${{ matrix.gemfile }}
-    continue-on-error: ${{ matrix.rails == 'edge' }}
-    services:
-      redis:
-        image: redis
-        ports:
-          - 6379:6379
-    strategy:
-      fail-fast: false
-      matrix:
-        ruby: ["2.6", "2.7", "3.0", "3.1", "3.2", "3.3"]
-        rails: ["5.2", "6.0", "6.1", "7.0", "7.1", "7.2", "edge"]
-        gemfile: [rails_gems]
-        exclude:
-          - ruby: "2.6"
-            rails: "7.0"
-          - ruby: "2.6"
-            rails: "7.1"
-          - ruby: "2.6"
-            rails: "7.2"
-          - ruby: "2.6"
-            rails: "edge"
-          - ruby: "2.7"
-            rails: "7.1"
-          - ruby: "2.7"
-            rails: "7.2"
-          - ruby: "2.7"
-            rails: "edge"
-          - ruby: "3.0"
-            rails: "5.2"
-          - ruby: "3.0"
-            rails: "7.1"
-          - ruby: "3.0"
-            rails: "7.2"
-          - ruby: "3.0"
-            rails: "edge"
-          - ruby: "3.1"
-            rails: "5.2"
-          - ruby: "3.1"
-            rails: "6.0"
-          - ruby: "3.1"
-            rails: "edge"
-          - ruby: "3.2"
-            rails: "5.2"
-          - ruby: "3.2"
-            rails: "6.0"
-          - ruby: "3.2"
-            rails: "6.1"
-          - ruby: "3.3"
-            rails: "5.2"
-          - ruby: "3.3"
-            rails: "6.0"
-          - ruby: "3.3"
-            rails: "6.1"
-
-        include:
-          - ruby: head
-            rails: "edge"
-            gemfile: rails_gems
-    env:
-      BUNDLE_GEMFILE: gemfiles/${{ matrix.gemfile }}.gemfile
-      RAILS_VERSION: ${{ matrix.rails }}
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v4
-      - name: Set up Ruby ${{ matrix.ruby }}
-        uses: ruby/setup-ruby@v1
-        with:
-          ruby-version: ${{ matrix.ruby }}
-          bundler-cache: true
-      - name: Start MySQL and create DB
-        run: |
-          sudo systemctl start mysql.service
-          mysql -uroot -h localhost -proot -e "CREATE DATABASE job_iteration_test;"
-      - name: Ruby tests
-        run: bundle exec rake test
-
-  lint:
-    runs-on: ubuntu-latest
-    name: Lint
-    steps:
-      - name: Check out code
-        uses: actions/checkout@v4
-      - name: Set up Ruby
-        uses: ruby/setup-ruby@v1
-        with:
-          bundler-cache: true
-      - name: Rubocop
-        run: bundle exec rubocop
-      - name: Documentation correctly written
-        run: bundle exec yardoc --no-output --no-save --no-stats --fail-on-warning

data/.github/workflows/cla.yml

@@ -1,22 +0,0 @@
-name: Contributor License Agreement (CLA)
-
-on:
-  pull_request_target:
-    types: [opened, synchronize]
-  issue_comment:
-    types: [created]
-
-jobs:
-  cla:
-    runs-on: ubuntu-latest
-    if: |
-      (github.event.issue.pull_request 
-        && !github.event.issue.pull_request.merged_at
-        && contains(github.event.comment.body, 'signed')
-      ) 
-      || (github.event.pull_request && !github.event.pull_request.merged)
-    steps:
-      - uses: Shopify/shopify-cla-action@v1
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cla-token: ${{ secrets.CLA_TOKEN }}

data/.gitignore

@@ -1,11 +0,0 @@
-/.bundle/
-/.yardoc
-/_yardoc/
-/coverage/
-/doc/
-/pkg/
-/spec/reports/
-/tmp/
-.rubocop-http---shopify-github-io-ruby-style-guide-rubocop-yml
-gemfiles/*.lock
-dump.rdb

data/.rubocop.yml

@@ -1,16 +0,0 @@
-inherit_gem:
-  rubocop-shopify: rubocop.yml
-
-inherit_mode:
-  merge:
-    - Include
-
-AllCops:
-  Include:
-    - '**/*.gemfile'
-Lint/SuppressedException:
-  Exclude:
-    - lib/job-iteration.rb
-Naming/FileName:
-  Exclude:
-    - lib/job-iteration.rb

data/.ruby-version

@@ -1 +0,0 @@
-3.3.3

data/.yardopts

@@ -1,3 +0,0 @@
---no-private
---embed-mixins
---markup markdown

data/CODE_OF_CONDUCT.md

@@ -1,74 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, gender identity and expression, level of experience,
-nationality, personal appearance, race, religion, or sexual identity and
-orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
-advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
-  address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
-  professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at shatrov@me.com. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at [http://contributor-covenant.org/version/1/4][version]
-
-[homepage]: http://contributor-covenant.org
-[version]: http://contributor-covenant.org/version/1/4/

data/Gemfile

@@ -1,42 +0,0 @@
-# frozen_string_literal: true
-
-source "https://rubygems.org"
-
-git_source(:github) { |repo_name| "https://github.com/#{repo_name}" }
-
-# Specify your gem's dependencies in job-iteration.gemspec
-gemspec
-
-# for integration testing
-gem "sidekiq"
-gem "resque"
-
-if defined?(@rails_gems_requirements) && @rails_gems_requirements
-  # We avoid the `gem "..."` syntax here so Dependabot doesn't try to update these gems.
-  [
-    "activejob",
-    "activerecord",
-  ].each { |name| gem name, @rails_gems_requirements }
-else
-  # gem "activejob" # Set in gemspec
-  gem "activerecord"
-end
-
-gem "mysql2", github: "brianmario/mysql2"
-gem "globalid"
-gem "i18n"
-gem "redis"
-
-gem "pry"
-gem "mocha"
-
-gem "rubocop-shopify", require: false
-gem "yard"
-gem "rake"
-gem "csv" # required for Ruby 3.4+
-
-# for unit testing optional sorbet support
-gem "sorbet-runtime"
-gem "tapioca"
-
-gem "logger"