resque 2.7.0 → 3.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7419ce3cc8734b05527a47b0c5fb4a9eea0a5d33265a6e3214bb429980c37681
-  data.tar.gz: 4216e18e121883e0350275e00c19e1d6c6f0965915d4b48a8a0a29f07e652317
+  metadata.gz: e48882e2b37c0256d7fb756dd585af487b5fd3d1161aa8e9aa0e06780ab51840
+  data.tar.gz: dd5cf203cb498beb074acf2d024343a948ad6658d52e9fc645c01450369176b9
 SHA512:
-  metadata.gz: 7301e154b368d012161ae22d69cb8e4400977a3099392f7789369ca37b04a413d0abec092e60dd9dddee01bda81c55dbf182da5cede4a8c2beff94f8ed44bc94
-  data.tar.gz: 85d0c05d98502b0da28ae60ca6117d3b4a47bbfc981b48c4168f6bf727bac8c4c325dcebf604f8916d0e04c97bf933651b6077b55ead117e7ecb1cc2cf4655f0
+  metadata.gz: 259841b283671d350964b422526e1ef05ef789f72a2c03afea5a74cd15d7f162dfd32ce2dce54d058cc1d62ea859603ba0f6f6e45a7da017d0f2a36322212dcc
+  data.tar.gz: 4bc3a3159deef3f51ba4546c8c23cc2203e0b127558a6beb84f502d8668122feaf2e61ba7640a1292d1c790a1b77c9590d62ad1afbabad950a660feb55dc9351

data/HISTORY.md

@@ -1,3 +1,37 @@
+## 3.0.1
+
+### Fixed
+
+* resque-web: Fix reflected XSS in the key sets endpoint (#1942)
+* Address multi_json class name deprecation warning (#1940)
+* Update the Rails initializer docs to load Redis config with `config_for` (#1936)
+* Add tests covering queue priority ordering with wildcards (#1930)
+
+## 3.0.0
+
+### Breaking Changes
+
+* **Minimum Ruby version is now 3.0.0** - Ruby 2.x is no longer supported
+* **Minimum Sinatra version is now 2.0** - old Sinatra versions (0.9-1.x) are no longer supported
+* **Rack 1.x is no longer supported** - Resque now requires Rack 2.x or greater
+* **Minimum Rails version is now 7.2** - You can use Resque without Rails, but if you use it with Rails it must be a version that is still supported
+
+### Added
+
+* Rack 3 support and switched default Rack handler to Puma (#1912)
+* Ruby 3.4 and Rails 8.1 support (#1912 and #1906)
+* Upstream Active Job adapter from Rails (#1906)
+* Explicit `redis` (>= 4.0) and `base64` (~> 0.1) gem dependencies (#1922)
+* Added `shutdown` hooks - allows plugins and adapters to know when a worker is shutting down, enabling features like Active Job Continuations (#1916, #1924)
+* Added support for dynamic interval sleep - workers can now use a faster polling interval that automatically backs off when idle, reducing Redis load (#1920)
+
+### Fixed
+
+* Fix circular require from Rails ActiveJob adapter (#1906)
+* Fix failed job `retried_at` timezone data (#1918)
+* Fix double Redis get when checking for paused workers (#1919)
+* Documentation cleanup (#1903, #1917)
+
 ## 2.7.0
 
 ### Fixed

data/README.markdown

@@ -39,15 +39,23 @@ The Resque frontend tells you what workers are doing, what workers are
 not doing, what queues you're using, what's in those queues, provides
 general usage stats, and helps you track failures.
 
-Resque now supports Ruby 2.3.0 and above.
-We will also only be supporting Redis 3.0 and above going forward.
+Resque 3.0 requires Ruby 3.0.0 or newer.
 
-### Note on the future of Resque
+**Version Support:**
+- Ruby: 3.0, 3.1, 3.2, 3.3, 3.4+
+- Redis gem: 4.0+
+- Rack: 2.x or 3.x
+- Rails (for ActiveJob): 7.2+ (requires Ruby 3.1+ for Rails 8.0+)
 
-Would you like to be involved in Resque? Do you have thoughts about what
-Resque should be and do going forward? There's currently an [open discussion here](https://github.com/resque/resque/issues/1759)
-on just that topic, so please feel free to join in. We'd love to hear your thoughts
-and/or have people volunteer to be a part of the project!
+### Resque 3.0
+
+Resque 3.0 is a major release that modernizes the codebase with updated dependency requirements:
+- Requires Ruby 3.0+ (drops support for Ruby 2.x)
+- Requires redis gem 4.0+ (tested with redis gem 4.x and 5.x)
+- Requires Sinatra 2.0+ (with Rack 2.x or 3.x support via appropriate Sinatra version)
+- Maintains compatibility with Rails 7.2+ and ActiveJob
+
+If you need Ruby 2.x support, please use Resque 2.x.
 
 Example
 -------
@@ -133,15 +141,6 @@ require 'your/app' # Include this line if you want your workers to have access t
 
 #### Rails
 
-##### Rails 7.1
-
-If you're using Rails 7.1.x, use [rack-session version 1.0.2](https://rubygems.org/gems/rack-session/versions/1.0.2). This is because resque-web's dependency, [Sinatra, isn't compatible with rack 3.0](https://github.com/sinatra/sinatra/issues/1797), which is [required by rack-session 2.0.0](https://rubygems.org/gems/rack-session/versions/2.0.0).
-
-```
-  gem 'rails', '~> 7.1'
-  gem 'rack-session', '~> 1.0', '>= 1.0.2'
-```
-
 To make resque specific changes, you can override the `resque:setup` job in `lib/tasks` (ex: `lib/tasks/resque.rake`). GitHub's setup task looks like this:
 
 ``` ruby
@@ -496,12 +495,14 @@ Here's our `config/resque.yml`:
 And our initializer:
 
 ``` ruby
-rails_root = ENV['RAILS_ROOT'] || File.dirname(__FILE__) + '/../..'
-rails_env = ENV['RAILS_ENV'] || 'development'
-config_file = rails_root + '/config/resque.yml'
+config_file = 'config/resque.yml'
+rails_env = ENV['RAILS_ENV'] || Rails.env
 
-resque_config = YAML::load(ERB.new(IO.read(config_file)).result)
+resque_config = YAML::load_file(config_file)
 Resque.redis = resque_config[rails_env]
+
+# OR auto load with current env
+Resque.redis = Rails.application.config_for(:resque)
 ```
 
 Easy peasy! Why not just use `RAILS_ROOT` and `RAILS_ENV`? Because
@@ -633,7 +634,7 @@ Choose DelayedJob if:
 * You like numeric priorities
 * You're not doing a gigantic amount of jobs each day
 * Your queue stays small and nimble
-* There is not a lot failure / chaos
+* There is not a lot of failure / chaos
 * You want to easily throw anything on the queue
 * You don't want to setup Redis
 

data/Rakefile

@@ -28,7 +28,33 @@ Rake::TestTask.new do |test|
   test.verbose = true
   test.libs << "test"
   test.libs << "lib"
-  test.test_files = FileList['test/**/*_test.rb']
+  test.test_files = FileList['test/**/*_test.rb'].exclude('test/active_job/**/*')
+end
+
+Rake::TestTask.new('test:activejob') do |test|
+  test.verbose = true
+  test.libs << "test"
+  test.libs << "lib"
+  test.libs << "test/active_job"
+  test.test_files = FileList['test/active_job/cases/*_test.rb']
+end
+
+task "env:aj_integration" do
+  ENV["AJ_INTEGRATION_TESTS"] = "1"
+end
+
+Rake::TestTask.new('test:activejob:integration' => 'env:aj_integration') do |t|
+  if Gem::Version.new(RUBY_VERSION) < Gem::Version.new("3.1")
+    puts "Integration tests require Ruby 3.1 or later"
+    exit 0
+  end
+
+  t.description = "Run integration tests for Resque::ActiveJob::Adapter"
+  t.libs << "test"
+  t.libs << "test/active_job"
+  t.test_files = FileList["test/active_job/integration/**/*_test.rb"]
+  t.verbose = true
+  t.warning = true
 end
 
 if command? :kicker

data/lib/active_job/queue_adapters/resque_adapter.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require "active_support/core_ext/enumerable"
+require "active_support/core_ext/array/access"
+
+begin
+  require "resque-scheduler"
+rescue LoadError
+  begin
+    require "resque_scheduler"
+  rescue LoadError
+    false
+  end
+end
+
+module ActiveJob
+  module QueueAdapters
+    remove_const(:ResqueAdapter) if defined?(ResqueAdapter)
+
+    # = Resque adapter for Active Job
+    #
+    # Resque (pronounced like "rescue") is a Redis-backed library for creating
+    # background jobs, placing those jobs on multiple queues, and processing
+    # them later.
+    #
+    # Read more about Resque {here}[https://github.com/resque/resque].
+    #
+    # To use Resque set the queue_adapter config to +:resque+.
+    #
+    #   Rails.application.config.active_job.queue_adapter = :resque
+    class ResqueAdapter < ::ActiveJob::QueueAdapters::AbstractAdapter
+      def enqueue(job) # :nodoc:
+        JobWrapper.instance_variable_set(:@queue, job.queue_name)
+        Resque.enqueue_to job.queue_name, JobWrapper, job.serialize
+      end
+
+      def enqueue_at(job, timestamp) # :nodoc:
+        unless Resque.respond_to?(:enqueue_at_with_queue)
+          raise NotImplementedError, "To be able to schedule jobs with Resque you need the " \
+            "resque-scheduler gem. Please add it to your Gemfile and run bundle install"
+        end
+        Resque.enqueue_at_with_queue job.queue_name, timestamp, JobWrapper, job.serialize
+      end
+
+      class JobWrapper # :nodoc:
+        class << self
+          def perform(job_data)
+            ::ActiveJob::Base.execute job_data
+          end
+        end
+      end
+    end
+  end
+end

data/lib/resque/failure/redis.rb

@@ -78,7 +78,7 @@ module Resque
       def self.requeue(id, queue = nil)
         check_queue(queue)
         item = all(id)
-        item['retried_at'] = Time.now.strftime("%Y/%m/%d %H:%M:%S")
+        item['retried_at'] = UTF8Util.clean(Time.now.strftime("%Y/%m/%d %H:%M:%S %Z"))
         data_store.update_item_in_failed_queue(id,Resque.encode(item))
         Job.create(item['queue'], item['payload']['class'], *item['payload']['args'])
       end

data/lib/resque/helpers.rb

@@ -1,14 +1,19 @@
 require 'multi_json'
 
-# OkJson won't work because it doesn't serialize symbols
-# in the same way yajl and json do.
-if MultiJson.respond_to?(:adapter)
-  raise "Please install the yajl-ruby or json gem" if MultiJson.adapter.to_s == 'MultiJson::Adapters::OkJson'
-elsif MultiJson.respond_to?(:engine)
-  raise "Please install the yajl-ruby or json gem" if MultiJson.engine.to_s == 'MultiJson::Engines::OkJson'
-end
-
 module Resque
+  # multi_json renamed it's top-level constant from MultiJson to MultiJSON,
+  # keeping MultiJson as a deprecated alias until v2.0. Reference whichever name the
+  # installed version exposes so we stay compatible with future versions of multi_json
+  MultiJson = defined?(::MultiJSON) ? ::MultiJSON : ::MultiJson
+
+  # OkJson won't work because it doesn't serialize symbols
+  # in the same way yajl and json do.
+  if MultiJson.respond_to?(:adapter)
+    raise "Please install the yajl-ruby or json gem" if MultiJson.adapter.to_s.end_with?('::OkJson')
+  elsif MultiJson.respond_to?(:engine)
+    raise "Please install the yajl-ruby or json gem" if MultiJson.engine.to_s.end_with?('::OkJson')
+  end
+
   # Methods used by various classes in Resque.
   module Helpers
     class DecodeException < StandardError; end

data/lib/resque/railtie.rb

@@ -6,5 +6,11 @@ module Resque
       # redefine ths task to load the rails env
       task "resque:setup" => :environment
     end
+
+    initializer "resque.active_job" do
+      ActiveSupport.on_load(:active_job) do
+        require "active_job/queue_adapters/resque_adapter"
+      end
+    end
   end
 end

data/lib/resque/server/views/key_sets.erb

@@ -2,7 +2,7 @@
 
   <p class='sub'><%= page_entries_info start = params[:start].to_i, start + 20, size = redis_get_size(key) %></p>
 
-  <h1>Key "<%= key %>" is a <%= resque.redis.type key %></h1>
+  <h1>Key "<%= escape_html(key) %>" is a <%= resque.redis.type key %></h1>
   <table>
     <% for row in redis_get_value_as_array(key, start) %>
     <tr>

data/lib/resque/tasks.rb

@@ -17,7 +17,11 @@ namespace :resque do
 
     worker.prepare
     worker.log "Starting worker #{worker}"
-    worker.work(ENV['INTERVAL'] || 5) # interval, will block
+    # will block until a shutdown signal is received
+    worker.work(ENV["INTERVAL"],
+                max_interval:     ENV['MAX_INTERVAL'],
+                min_interval:     ENV['MIN_INTERVAL'],
+                backoff_interval: ENV['BACKOFF_INTERVAL'])
   end
 
   desc "Start multiple Resque workers. Should only be used in dev mode."

data/lib/resque/version.rb

@@ -1,3 +1,3 @@
 module Resque
-  VERSION = '2.7.0'
+  VERSION = '3.0.1'
 end

data/lib/resque/web_runner.rb

@@ -3,6 +3,10 @@ require 'logger'
 require 'optparse'
 require 'fileutils'
 require 'rack'
+begin
+  require 'rackup'
+rescue LoadError
+end
 require 'resque/server'
 
 # only used with `bin/resque-web`
@@ -30,7 +34,7 @@ module Resque
 
       @args = load_options(runtime_args)
 
-      @rack_handler = (s = options[:rack_handler]) ? Rack::Handler.get(s) : setup_rack_handler
+      @rack_handler = (s = options[:rack_handler]) ? self.class.get_rackup_or_rack_handler.get(s) : setup_rack_handler
 
       case option_parser.command
       when :help
@@ -270,6 +274,10 @@ module Resque
       self.class.logger
     end
 
+    def self.get_rackup_or_rack_handler
+      defined?(::Rackup::Handler) && ::Rack.release >= '3' ? ::Rackup::Handler : ::Rack::Handler
+    end
+
   private
     def setup_rack_handler
       # First try to set Rack handler via a special hook we honor
@@ -284,24 +292,25 @@ module Resque
           handler = nil
           @app.server.each do |server|
             begin
-              handler = Rack::Handler.get(server)
+              handler = self.class.get_rackup_or_rack_handler.get(server)
               break
             rescue LoadError, NameError
               next
             end
           end
-          raise 'No available Rack handler (e.g. WEBrick, Thin, Puma, etc.) was found.' if handler.nil?
+          raise 'No available Rack handler (e.g. WEBrick, Puma, etc.) was found.' if handler.nil?
 
           handler
 
         # :server might be set explicitly to a single option like "mongrel"
         else
-          Rack::Handler.get(@app.server)
+          self.class.get_rackup_or_rack_handler.get(@app.server)
         end
 
-      # If all else fails, we'll use Thin
+      # If all else fails, we'll use Puma
       else
-        JRUBY ? Rack::Handler::WEBrick : Rack::Handler::Thin
+        rack_server = JRUBY ? 'webrick' : 'puma'
+        self.class.get_rackup_or_rack_handler.get(rack_server)
       end
     end
 

data/lib/resque/worker.rb

@@ -143,6 +143,7 @@ module Resque
     def initialize(*queues)
       @shutdown = nil
       @paused = nil
+      @cached_pause_value = nil
       @before_first_fork_hook_ran = false
 
       @heartbeat_thread = nil
@@ -237,20 +238,43 @@ module Resque
     # The default is 5 seconds, but for a semi-active site you may
     # want to use a smaller value.
     #
+    # Can also be passed 3 interval floats: max, min, backoff.
+    # The actual sleep amount will float between these min/max
+    # bounds.
+    #
+    # If a job is picked up we sleep for the minimum amount of
+    # time, but as the queues empty we increase the backoff to the
+    # max interval. This prevents idle workers from hammering the
+    # redis server with lpop requests.
+    #
     # Also accepts a block which will be passed the job as soon as it
     # has completed processing. Useful for testing.
-    def work(interval = 5.0, &block)
-      interval = Float(interval)
+    def work(interval = 5.0,
+             min_interval: nil,     # defaults to interval
+             max_interval: nil,     # defaults to interval
+             backoff_interval: nil, # defaults to 0.1
+             &block)
+      interval = Float(interval || 5.0)
+      max_interval = Float(max_interval || interval)
+      min_interval = Float(min_interval || interval).clamp(nil, max_interval)
+      backoff_interval = Float(backoff_interval || 0.1).clamp(nil, max_interval)
+      interval = interval.clamp(min_interval, max_interval)
       startup
 
       loop do
         break if shutdown?
 
-        unless work_one_job(&block)
+        if work_one_job(&block)
+          interval = min_interval
+        else
           state_change
           break if interval.zero?
+
+          interval = (interval + backoff_interval)
+            .clamp(nil, max_interval)
+
           log_with_severity :debug, "Sleeping for #{interval} seconds"
-          procline paused? ? "Paused" : "Waiting for #{queues.join(',')}"
+          procline @cached_pause_value ? "Paused" : "Waiting for #{queues.join(',')}"
           sleep interval
         end
       end
@@ -260,6 +284,7 @@ module Resque
     rescue Exception => exception
       return if exception.class == SystemExit && !@child && run_at_exit_hooks
       log_with_severity :error, "Failed to start worker : #{exception.inspect}"
+      log_with_severity :error, exception.backtrace.join("\n")
       unregister_worker(exception)
       run_hook :worker_exit
     end
@@ -441,6 +466,8 @@ module Resque
     def shutdown
       log_with_severity :info, 'Exiting...'
       @shutdown = true
+      run_hook :shutdown
+      true
     end
 
     # Kill the child and shutdown immediately.
@@ -579,7 +606,7 @@ module Resque
 
     # are we paused?
     def paused?
-      @paused || redis.get('pause-all-workers').to_s.strip.downcase == 'true'
+      @cached_pause_value = @paused || redis.get('pause-all-workers').to_s.strip.downcase == 'true'
     end
 
     # Stop processing jobs after the current one has completed (if we're

data/lib/resque.rb

@@ -49,7 +49,7 @@ module Resque
       else
         MultiJson.decode object
       end
-    rescue ::MultiJson::DecodeError => e
+    rescue MultiJson::DecodeError => e
       raise Helpers::DecodeException, e.message, e.backtrace
     end
   end
@@ -316,6 +316,20 @@ module Resque
     register_hook(:worker_exit, block)
   end
 
+  # The `shutdown` hook will be run in the **child** process
+  # when the worker has received a signal to stop processing
+  #
+  # Call with a block to register a hook.
+  # Call with no arguments to return all registered hooks.
+  def shutdown(&block)
+    block ? register_hook(:shutdown, block) : hooks(:shutdown)
+  end
+
+  # Register a shutdown proc.
+  def shutdown=(block)
+    register_hook(:shutdown, block)
+  end
+
   def to_s
     "Resque Client connected to #{redis_id}"
   end

metadata

@@ -1,18 +1,45 @@
 --- !ruby/object:Gem::Specification
 name: resque
 version: !ruby/object:Gem::Version
-  version: 2.7.0
+  version: 3.0.1
 platform: ruby
 authors:
 - Chris Wanstrath
 - Steve Klabnik
 - Terence Lee
 - Michael Bianco
-autorequire: 
 bindir: bin
 cert_chain: []
-date: 2024-12-30 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: base64
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+- !ruby/object:Gem::Dependency
+  name: logger
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: redis-namespace
   requirement: !ruby/object:Gem::Requirement
@@ -27,20 +54,34 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '1.6'
+- !ruby/object:Gem::Dependency
+  name: redis
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.0'
 - !ruby/object:Gem::Dependency
   name: sinatra
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.9.2
+        version: '2.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.9.2
+        version: '2.0'
 - !ruby/object:Gem::Dependency
   name: multi_json
   requirement: !ruby/object:Gem::Requirement
@@ -70,7 +111,7 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '1'
 - !ruby/object:Gem::Dependency
-  name: thin
+  name: puma
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -127,6 +168,7 @@ files:
 - Rakefile
 - bin/resque
 - bin/resque-web
+- lib/active_job/queue_adapters/resque_adapter.rb
 - lib/resque.rb
 - lib/resque/data_store.rb
 - lib/resque/errors.rb
@@ -186,7 +228,6 @@ licenses:
 metadata:
   changelog_uri: https://github.com/resque/resque/blob/master/HISTORY.md
   rubygems_mfa_required: 'true'
-post_install_message: 
 rdoc_options:
 - "--charset=UTF-8"
 require_paths:
@@ -195,15 +236,14 @@ required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
-      version: 2.3.0
+      version: 3.0.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.3.1
-signing_key: 
+rubygems_version: 4.0.6
 specification_version: 4
 summary: Resque is a Redis-backed queueing system.
 test_files: []