sidekiq 8.0.2 → 8.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a2128fafa84e48bfd4cad6155e9dcddaab93bc68152b28d6c27156cf84490cf9
-  data.tar.gz: 346e2bff46bcae590efbdab4bf4c0942994a70fd4360cb395efffaac02e9ee02
+  metadata.gz: 63658318932ac6b7045211590b07f84ed9e1e6398f908ace790a0df20d5ac1f0
+  data.tar.gz: 0c3f5e69ada7529bdac7392acb1c56ef2a19b8ed13210be8c36aac19858c6df4
 SHA512:
-  metadata.gz: 318665d5dd278811f1ce98923fac64fbc79d78f38bde1e54c92f975f4bce24a24ef43a7c15eda03fc54caa39f1bddd1e932aa98f866a6983d4d92cf2ec587a6d
-  data.tar.gz: 16b597d7ff91c948bd4868c85386387a4f306f6dfce260d9141db485e9252845c38c2195ee6ec7b4490e94c950d66de43a4a429feaacf37c9b06d3be4ed9ed14
+  metadata.gz: 9a5e95a2faccbbf42f8e651c4882c9495ad72b0500bb8fe77fa2a93fb4aa6f8cb96fc820621cd1910fb4b00accf35d75a99d548b3cc8ccdd41a18b5620a9ad73
+  data.tar.gz: 1bc66490ee07c548c71f680fb280e876efa7eeb8c6f234277df73211ce05c5ae29f5c9bc13064972da945568d1df6634252a6c22a7c5298bdd436b7266246c53

data/Changes.md

@@ -2,6 +2,114 @@
 
 [Sidekiq Changes](https://github.com/sidekiq/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/sidekiq/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/sidekiq/sidekiq/blob/main/Ent-Changes.md)
 
+8.1.0
+----------
+
+- `retry_for` and `retry` are now mutually exclusive [#6878, Saidbek]
+- `perform_inline` now enforces `strict_args!` [#6718, Saidbek]
+- Integrate Herb linting for ERB templates [#6760, Saidbek]
+- Remove CSRF code, use `Sec-Fetch-Site` header [#6874, deve1212]
+- Allow custom Web UI `assets_path` for CDN purposes [#6865, stanhu]
+- Upgrade to connection_pool 3.0
+- Allow idle connection reaping after N seconds.
+You can activate this **beta** feature like below.
+Feedback requested: is this feature stable and useful for you in production?
+This feature may or may not be enabled by default in Sidekiq 9.0.
+```ruby
+Sidekiq.configure_server do |cfg|
+  cfg.reap_idle_redis_connections(60)
+end
+```
+
+8.0.10
+----------
+
+- Add confirm dialog for Delete All buttons in Web UI [#6853]
+- Adjust scheduler to run closer to poll average [#6866]
+- Forward compatibility changes for connection_pool 3.0.0
+- Backwards compatibility fix for <8.0.9 process data in Redis [#6870]
+- Backtrace dump can now be triggered with the INFO signal, since Puma uses the
+  same signal [#6857]
+
+8.0.9
+----------
+
+- Implement idle Redis connection reaping, will be activated in 8.1 [#6663]
+- Updated `Sidekiq::Process` API to provide capsule data. The `queues` and `weights` 
+  data will be removed from Redis in Sidekiq 8.1, as this data can now be found in the
+  `capsules` element. [#6295]
+- Restore bulk action buttons on Scheduled, Retry and Dead tabs [#6833, deve1212]
+- Support logging additional job attributes [#6846, bschrag620]
+- Fix display of long job args [#6836]
+- Create development lifecycle (`docs/sdlc.md`) and security (`docs/SECURITY.md`) policy
+  documentation for Sidekiq's current workflows
+
+8.0.8
+----------
+
+- Allow an optional global iteration max runtime. After executing for this length of time,
+  Sidekiq will re-queue the job to continue execution at a later time [#6819, fatkodima]
+```ruby
+Sidekiq.configure_server do |cfg|
+  cfg[:max_iteration_runtime] = 600 # ten minutes
+end
+```
+- Add `discarded_at` attribute when discarding a job so death handlers can distinguish between
+  a job which was killed and one that was discarded. [#6820, gstokkink]
+- `perform_bulk` now accepts an `:at` array of times to schedule each job at the corresponding time.
+  `perform_bulk(args: [[1], [2]], at: [Time.now, Time.now + 1])` [#6790, fatkodima]
+- `perform_bulk` now accepts a `:spread_interval` value to schedule jobs over
+  the next N seconds. `perform_bulk(..., spread_interval: 60)` [#6792, fatkodima]
+- Fix unintended display of flash messages in the Web UI due to session key collision
+- Add support for lazy load hooks [#6825]
+
+8.0.7
+----------
+
+- The `:discard` option for `sidekiq_retries_exhausted` and `sidekiq_retry_in`
+  now calls death handlers, otherwise it could break other Sidekiq
+  functionality. [#6741]
+- Provide a Plain log formatter which does not colorize output [#6778]
+- Job iteration now exposes `current_object` for easy access within the `around_iteration` callback [#6774]
+- Fix JS race condition which could skip confirmation dialogs when Live Polling [#6768]
+- Fix edge case which could lose CurrentAttributes [#6767]
+- Update UK locale [#6776]
+
+8.0.6
+----------
+
+- Adjust transactional client to use ActiveRecord 7.2's support for
+  `after_all_transactions_commit` when available. [#6765, rewritten]
+- Fix Rails 7.0 and 7.1 compatibility [#6746, mlarraz]
+- Flush metrics at `:exit` [#6764]
+
+8.0.5
+----------
+
+- Add `stopping?` method to AJ adapter for compatibility with the new AJ::Continuations feature [#6732]
+- Further improvements to Rails boot compatibility [#6710]
+- Add ability to disable CSRF middleware. SameSite cookies prevent
+  CSRF in a cleaner manner and are default in most browsers now.
+  CSRF code will be removed in Sidekiq 9.0. [#6739]
+
+8.0.4
+----------
+
+- Adjust Rails integration for various edge cases [6713]
+- Flush job iteration state when an error is raised [#6704]
+- Update Accept-Language parsing in Web UI [#6721]
+- Remove fixed-width in Web UI [#6686]
+- Adjust CSRF middleware ordering [#6688]
+- Support proxies when POSTing profiles to profiler.firefox.com [#6687]
+- Dont swallow NoMethodErrors in CurrentAttributes [#6685]
+
+8.0.3
+----------
+
+- Configure Vernier output directory [#6674]
+- Rework Rails integration [#6669]
+- Implement flash messages for the Web UI [#6675]
+
 8.0.2
 ----------
 

data/README.md

@@ -97,6 +97,21 @@ Contributing
 
 See [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
 
+### ERB Linting with HERB
+
+This project uses [HERB](https://herb-tools.dev/) for ERB file linting and formatting. All ERB files have been renamed to use the `.html.erb` extension for better tooling support.
+
+**Local Development:**
+```bash
+# Run HERB linting
+bundle exec rake lint:herb
+# or
+bin/lint-herb
+```
+
+**CI Integration:**
+HERB linting is automatically run in CI to ensure all ERB files are properly formatted and free of parse errors.
+
 License
 -----------------
 

data/bin/lint-herb

@@ -0,0 +1,13 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+# HERB Linting Script
+# Run this script to lint all ERB files in the project
+# Usage: bin/lint-herb
+
+require "bundler/setup"
+
+puts "🔍 Running HERB linting on ERB files..."
+puts
+
+exec("bundle exec herb analyze web/views -n --no-log-file")

data/lib/active_job/queue_adapters/sidekiq_adapter.rb

@@ -1,75 +1,121 @@
 # frozen_string_literal: true
 
-module ActiveJob
-  module QueueAdapters
-    # Explicitly remove the implementation existing in older Rails.
-    remove_const(:SidekiqAdapter) if const_defined?(:SidekiqAdapter)
+begin
+  gem "activejob", ">= 7.0"
+  require "active_job"
 
-    # Sidekiq adapter for Active Job
-    #
-    # To use Sidekiq set the queue_adapter config to +:sidekiq+.
-    #
-    #   Rails.application.config.active_job.queue_adapter = :sidekiq
-    class SidekiqAdapter
-      # Defines whether enqueuing should happen implicitly to after commit when called
-      # from inside a transaction.
+  module Sidekiq
+    module ActiveJob
       # @api private
-      def enqueue_after_transaction_commit?
-        true
-      end
+      class Wrapper
+        include Sidekiq::Job
 
-      # @api private
-      def enqueue(job)
-        job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
-          wrapped: job.class,
-          queue: job.queue_name
-        ).perform_async(job.serialize)
+        def perform(job_data)
+          ::ActiveJob::Base.execute(job_data.merge("provider_job_id" => jid))
+        end
       end
+    end
+  end
 
-      # @api private
-      def enqueue_at(job, timestamp)
-        job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
-          wrapped: job.class,
-          queue: job.queue_name
-        ).perform_at(timestamp, job.serialize)
-      end
+  ActiveSupport.on_load(:active_job) do
+    # By including the Options module, we allow AJs to directly control sidekiq features
+    # via the *sidekiq_options* class method and, for instance, not use AJ's retry system.
+    # AJ retries don't show up in the Sidekiq UI Retries tab, don't save any error data, can't be
+    # manually retried, don't automatically die, etc.
+    #
+    #   class SomeJob < ActiveJob::Base
+    #     queue_as :default
+    #     sidekiq_options retry: 3, backtrace: 10
+    #     def perform
+    #     end
+    #   end
+    include Sidekiq::Job::Options unless respond_to?(:sidekiq_options)
+  end
 
-      # @api private
-      def enqueue_all(jobs)
-        enqueued_count = 0
-        jobs.group_by(&:class).each do |job_class, same_class_jobs|
-          same_class_jobs.group_by(&:queue_name).each do |queue, same_class_and_queue_jobs|
-            immediate_jobs, scheduled_jobs = same_class_and_queue_jobs.partition { |job| job.scheduled_at.nil? }
-
-            if immediate_jobs.any?
-              jids = Sidekiq::Client.push_bulk(
-                "class" => Sidekiq::ActiveJob::Wrapper,
-                "wrapped" => job_class,
-                "queue" => queue,
-                "args" => immediate_jobs.map { |job| [job.serialize] }
-              )
-              enqueued_count += jids.compact.size
-            end
+  # Patch the ActiveJob module
+  module ActiveJob
+    module QueueAdapters
+      # Explicitly remove the implementation existing in older Rails.
+      remove_const(:SidekiqAdapter) if const_defined?(:SidekiqAdapter)
+
+      # Sidekiq adapter for Active Job
+      #
+      # To use Sidekiq set the queue_adapter config to +:sidekiq+.
+      #
+      #   Rails.application.config.active_job.queue_adapter = :sidekiq
+      parent = const_defined?(:AbstractAdapter) ? AbstractAdapter : Object
+      class SidekiqAdapter < parent
+        @@stopping = false
 
-            if scheduled_jobs.any?
-              jids = Sidekiq::Client.push_bulk(
-                "class" => Sidekiq::ActiveJob::Wrapper,
-                "wrapped" => job_class,
-                "queue" => queue,
-                "args" => scheduled_jobs.map { |job| [job.serialize] },
-                "at" => scheduled_jobs.map { |job| job.scheduled_at&.to_f }
-              )
-              enqueued_count += jids.compact.size
+        callback = -> { @@stopping = true }
+
+        Sidekiq.configure_client { |config| config.on(:quiet, &callback) }
+        Sidekiq.configure_server { |config| config.on(:quiet, &callback) }
+
+        # Defines whether enqueuing should happen implicitly to after commit when called
+        # from inside a transaction.
+        # @api private
+        def enqueue_after_transaction_commit?
+          true
+        end
+
+        # @api private
+        def enqueue(job)
+          job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
+            wrapped: job.class,
+            queue: job.queue_name
+          ).perform_async(job.serialize)
+        end
+
+        # @api private
+        def enqueue_at(job, timestamp)
+          job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
+            wrapped: job.class,
+            queue: job.queue_name
+          ).perform_at(timestamp, job.serialize)
+        end
+
+        # @api private
+        def enqueue_all(jobs)
+          enqueued_count = 0
+          jobs.group_by(&:class).each do |job_class, same_class_jobs|
+            same_class_jobs.group_by(&:queue_name).each do |queue, same_class_and_queue_jobs|
+              immediate_jobs, scheduled_jobs = same_class_and_queue_jobs.partition { |job| job.scheduled_at.nil? }
+
+              if immediate_jobs.any?
+                jids = Sidekiq::Client.push_bulk(
+                  "class" => Sidekiq::ActiveJob::Wrapper,
+                  "wrapped" => job_class,
+                  "queue" => queue,
+                  "args" => immediate_jobs.map { |job| [job.serialize] }
+                )
+                enqueued_count += jids.compact.size
+              end
+
+              if scheduled_jobs.any?
+                jids = Sidekiq::Client.push_bulk(
+                  "class" => Sidekiq::ActiveJob::Wrapper,
+                  "wrapped" => job_class,
+                  "queue" => queue,
+                  "args" => scheduled_jobs.map { |job| [job.serialize] },
+                  "at" => scheduled_jobs.map { |job| job.scheduled_at&.to_f }
+                )
+                enqueued_count += jids.compact.size
+              end
             end
           end
+          enqueued_count
         end
-        enqueued_count
-      end
 
-      # Defines a class alias for backwards compatibility with enqueued Active Job jobs.
-      # @api private
-      class JobWrapper < Sidekiq::ActiveJob::Wrapper
+        # @api private
+        def stopping? = !!@@stopping
+
+        # Defines a class alias for backwards compatibility with enqueued Active Job jobs.
+        # @api private
+        JobWrapper = Sidekiq::ActiveJob::Wrapper
       end
     end
   end
+rescue Gem::LoadError
+  # ActiveJob not available or version requirement not met
 end

data/lib/generators/sidekiq/templates/job.rb.erb

@@ -6,4 +6,4 @@ class <%= class_name %>Job
     # Do something
   end
 end
-<% end -%>
+<% end -%>

data/lib/sidekiq/api.rb

@@ -1059,9 +1059,9 @@ module Sidekiq
   #   'started_at' => <process start time>,
   #   'pid' => 12345,
   #   'tag' => 'myapp'
-  #   'concurrency' => 25,
-  #   'queues' => ['default', 'low'],
-  #   'busy' => 10,
+  #   'concurrency' => 5,
+  #   'capsules' => {"default" => {"mode" => "weighted", "concurrency" => 5, "weights" => {"default" => 2, "low" => 1}}},
+  #   'busy' => 3,
   #   'beat' => <last heartbeat>,
   #   'identity' => <unique string identifying the process>,
   #   'embedded' => true,
@@ -1089,12 +1089,35 @@ module Sidekiq
       self["identity"]
     end
 
+    # deprecated, use capsules below
     def queues
-      self["queues"]
+      # Backwards compatibility with <8.0.8
+      if !self["capsules"]
+        self["queues"]
+      else
+        capsules.values.flat_map { |x| x["weights"].keys }.uniq
+      end
     end
 
+    # deprecated, use capsules below
     def weights
-      self["weights"]
+      # Backwards compatibility with <8.0.8
+      if !self["capsules"]
+        self["weights"]
+      else
+        hash = {}
+        capsules.values.each do |cap|
+          # Note: will lose data if two capsules are processing the same named queue
+          cap["weights"].each_pair do |queue, weight|
+            hash[queue] = weight
+          end
+        end
+        hash
+      end
+    end
+
+    def capsules
+      self["capsules"]
     end
 
     def version
@@ -1168,7 +1191,6 @@ module Sidekiq
   #      # thread_id is a unique identifier per thread
   #      # work is a `Sidekiq::Work` instance that has the following accessor methods.
   #      # [work.queue, work.run_at, work.payload]
-  #      # run_at is an epoch Integer.
   #    end
   #
   class WorkSet
@@ -1322,3 +1344,5 @@ module Sidekiq
     end
   end
 end
+
+Sidekiq.loader.run_load_hooks(:api)

data/lib/sidekiq/capsule.rb

@@ -38,6 +38,10 @@ module Sidekiq
       @mode = :strict
     end
 
+    def to_h
+      {concurrency: concurrency, mode: mode, weights: weights}
+    end
+
     def fetcher
       @fetcher ||= begin
         instance = (config[:fetch_class] || Sidekiq::BasicFetch).new(self)

data/lib/sidekiq/cli.rb

@@ -49,7 +49,7 @@ module Sidekiq # :nodoc:
       logger.info "Booted Rails #{::Rails.version} application in #{environment} environment" if rails_app?
 
       self_read, self_write = IO.pipe
-      sigs = %w[INT TERM TTIN TSTP]
+      sigs = %w[INT TERM INFO TTIN TSTP]
       # USR1 and USR2 don't work on the JVM
       sigs << "USR2" if Sidekiq.pro? && !jruby?
       sigs.each do |sig|
@@ -201,7 +201,19 @@ module Sidekiq # :nodoc:
         cli.logger.info "Received TSTP, no longer accepting new work"
         cli.launcher.quiet
       },
+      # deprecated, use INFO
       "TTIN" => ->(cli) {
+        cli.logger.error { "DEPRECATED: Please use the INFO signal for backtraces, support for TTIN will be removed in Sidekiq 9.0." }
+        Thread.list.each do |thread|
+          cli.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread.name}"
+          if thread.backtrace
+            cli.logger.warn thread.backtrace.join("\n")
+          else
+            cli.logger.warn "<no backtrace available>"
+          end
+        end
+      },
+      "INFO" => ->(cli) {
         Thread.list.each do |thread|
           cli.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread.name}"
           if thread.backtrace
@@ -212,12 +224,12 @@ module Sidekiq # :nodoc:
         end
       }
     }
-    UNHANDLED_SIGNAL_HANDLER = ->(cli) { cli.logger.info "No signal handler registered, ignoring" }
-    SIGNAL_HANDLERS.default = UNHANDLED_SIGNAL_HANDLER
 
     def handle_signal(sig)
       logger.debug "Got #{sig} signal"
-      SIGNAL_HANDLERS[sig].call(self)
+      hndlr = SIGNAL_HANDLERS[sig]
+      hndlr ? hndlr.call(self) :
+        logger.warn("No #{sig} signal handler registered, ignoring")
     end
 
     private

data/lib/sidekiq/client.rb

@@ -117,6 +117,9 @@ module Sidekiq
     # larger than 1000 but YMMV based on network quality, size of job args, etc.
     # A large number of jobs can cause a bit of Redis command processing latency.
     #
+    # Accepts an additional `:spread_interval` option (in seconds) to randomly spread
+    # the jobs schedule times over the specified interval.
+    #
     # Takes the same arguments as #push except that args is expected to be
     # an Array of Arrays.  All other keys are duplicated for each job.  Each job
     # is run through the client middleware pipeline and each job gets its own Job ID
@@ -131,13 +134,24 @@ module Sidekiq
     def push_bulk(items)
       batch_size = items.delete(:batch_size) || items.delete("batch_size") || 1_000
       args = items["args"]
-      at = items.delete("at")
+      at = items.delete("at") || items.delete(:at)
       raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all? { |entry| entry.is_a?(Numeric) })
       raise ArgumentError, "Job 'at' Array must have same size as 'args' Array" if at.is_a?(Array) && at.size != args.size
 
       jid = items.delete("jid")
       raise ArgumentError, "Explicitly passing 'jid' when pushing more than one job is not supported" if jid && args.size > 1
 
+      spread_interval = items.delete(:spread_interval) || items.delete("spread_interval")
+      raise ArgumentError, "Jobs 'spread_interval' must be a positive Numeric" if spread_interval && (!spread_interval.is_a?(Numeric) || spread_interval <= 0)
+      raise ArgumentError, "Only one of 'at' or 'spread_interval' can be provided" if at && spread_interval
+
+      if !at && spread_interval
+        # Do not use spread interval smaller than pooling interval.
+        spread_interval = [spread_interval, 5].max
+        now = Time.now.to_f
+        at = args.map { now + rand * spread_interval }
+      end
+
       normed = normalize_item(items)
       slice_index = 0
       result = args.each_slice(batch_size).flat_map do |slice|

data/lib/sidekiq/component.rb

@@ -19,7 +19,8 @@ module Sidekiq
   DEFAULT_THREAD_PRIORITY = -1
 
   ##
-  # Sidekiq::Component assumes a config instance is available at @config
+  # Sidekiq::Component provides a set of utility methods depending only
+  # on Sidekiq::Config. It assumes a config instance is available at @config.
   module Component # :nodoc:
     attr_reader :config
 

data/lib/sidekiq/config.rb

@@ -17,10 +17,9 @@ module Sidekiq
       poll_interval_average: nil,
       average_scheduled_poll_interval: 5,
       on_complex_arguments: :raise,
-      iteration: {
-        max_job_runtime: nil,
-        retry_backoff: 0
-      },
+      # if the Iterable job runs longer than this value (in seconds), then the job
+      # will be interrupted after the current iteration and re-enqueued at the back of the queue
+      max_iteration_runtime: nil,
       error_handlers: [],
       death_handlers: [],
       lifecycle_events: {
@@ -36,7 +35,9 @@ module Sidekiq
       dead_max_jobs: 10_000,
       dead_timeout_in_seconds: 180 * 24 * 60 * 60, # 6 months
       reloader: proc { |&block| block.call },
-      backtrace_cleaner: ->(backtrace) { backtrace }
+      backtrace_cleaner: ->(backtrace) { backtrace },
+      logged_job_attributes: ["bid", "tags"],
+      redis_idle_timeout: nil
     }
 
     ERROR_HANDLER = ->(ex, ctx, cfg = Sidekiq.default_configuration) {
@@ -145,11 +146,15 @@ module Sidekiq
       @redis_config = @redis_config.merge(hash)
     end
 
+    def reap_idle_redis_connections(timeout = 60)
+      self[:redis_idle_timeout] = timeout
+    end
+
     def redis_pool
       Thread.current[:sidekiq_redis_pool] || Thread.current[:sidekiq_capsule]&.redis_pool || local_redis_pool
     end
 
-    private def local_redis_pool
+    def local_redis_pool
       # this is our internal client/housekeeping pool. each capsule has its
       # own pool for executing threads.
       @redis ||= new_redis_pool(10, "internal")

data/lib/sidekiq/fetch.rb

@@ -7,6 +7,7 @@ require "sidekiq/capsule"
 module Sidekiq # :nodoc:
   class BasicFetch
     include Sidekiq::Component
+
     # We want the fetch operation to timeout every few seconds so the thread
     # can check if the process is shutting down.
     TIMEOUT = 2