sidekiq 7.3.9 → 8.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dccc402e8f4e309a47a2adb62418fa29989012e8c94f34aa11200b098dc375f2
-  data.tar.gz: 71b64a582d5971b045d60bae38dc26da265b60ff9a099ca3b2f9c7df9035e805
+  metadata.gz: 5862bbba5b859c1983788b6aeb49193fad724a1186426084c1e42747cb226ba2
+  data.tar.gz: 11fbaba14b814b87d09d809173ca59f9477c119f9ab6d556e4a9baa8e4e83d63
 SHA512:
-  metadata.gz: d9148b613a222ca9617ceebe25bb04a5d086edacdafad4a426b6232662d3b583db462dc5e0491297f7859037c5166bcb541bddcd3949f6d3b5a1c2e3fc572b65
-  data.tar.gz: 93a797cefd1a68adb236c7538c28571467c328d6965af0bc5ce505a1391d6f5b6e2ae2c87b42110e837dcfa8d788aff55309806bbb1bcd181e205c3bc66adc29
+  metadata.gz: 67d640ff21778c036ba15db434bafe0582a61e1b462d1bee2502fa7a9b8978208b4099c46d8ad6bd6214f198558de8985deeb11b6e7d99f1e603b15f087f0215
+  data.tar.gz: 4becc4ea36c062f52667fea91d4f49b5bfb759fe43f75ef6427c1aeacdc36764408f442cafccc3595585cf13ef2ff71da98212d7db0fcb89ff198ace0cd19caa

data/Changes.md

@@ -2,10 +2,67 @@
 
 [Sidekiq Changes](https://github.com/sidekiq/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/sidekiq/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/sidekiq/sidekiq/blob/main/Ent-Changes.md)
 
+HEAD
+----------
+
+- Configure Vernier output directory [#6674]
+- Rework Rails integration [#6669]
+- Implement flash messages for the Web UI [#6675]
+
+8.0.2
+----------
+
+- Add `on(:exit)` event to run code right before the Sidekiq process exits [#6637]
+- Metrics page crashes with Rack 3.1+ [#6646]
+
+8.0.1
+----------
+
+- Relax Redis requirement to 7.0 for compatibility with AWS and Ubuntu 24.04 LTS. [#6630]
+
+8.0.0
+----------
+
+- **WARNING** The underlying class name for Active Jobs has changed from `ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper` to `Sidekiq::ActiveJob::Wrapper`.
+  The old name will still work in 8.x.
+- **WARNING** The `created_at`, `enqueued_at`, `failed_at` and `retried_at` attributes are now stored as epoch milliseconds, rather than epoch floats.
+  This is meant to avoid precision issues with JSON and JavaScript's 53-bit Floats.
+  Example: `"created_at" => 1234567890.123456` -> `"created_at" => 1234567890123`.
+- **NEW FEATURE** Job Profiling is now supported with [Vernier](https://vernier.prof)
+  which makes it really easy to performance tune your slow jobs.
+  The Web UI contains a new **Profiles** tab to view any collected profile data.
+  Please read the new [Profiling](https://github.com/sidekiq/sidekiq/wiki/Profiling) wiki page for details.
+- **NEW FEATURE** Job Metrics now store up to 72 hours of data and the Web UI allows display of 24/48/72 hours. [#6614]
+- CurrentAttribute support now uses `ActiveJob::Arguments` to serialize the context object, supporting Symbols and GlobalID.
+  The change should be backwards compatible. [#6510]
+- Freshen up `Sidekiq::Web` to simplify the code and improve security [#6532]
+  The CSS has been rewritten from scratch to remove the Bootstrap framework.
+- Add `on_cancel` callback for iterable jobs [#6607]
+- Add `cursor` reader to get the current cursor inside iterable jobs [#6606]
+- Default error logging has been modified to use Ruby's `Exception#detailed_message` and `#full_message` APIs.
+- CI now runs against Redis, Dragonfly and Valkey.
+- Job tags now allow custom CSS display [#6595]
+- The Web UI's language picker now shows options in the native language
+- Remove global variable usage within the codebase
+- Colorize and adjust logging for easier reading
+- Adjust Sidekiq's default thread priority to -1 for a 50ms timeslice.
+  This can help avoid TimeoutErrors when Sidekiq is overloaded. [#6543]
+- Use `Logger#with_level`, remove Sidekiq's custom impl
+- Remove `base64` gem dependency
+- Support: (Dragonfly 1.27+, Valkey 7.2+, Redis 7.2+), Ruby 3.2+, Rails 7.0+
+
+7.3.10
+----------
+
+- Deprecate Redis :password as a String to avoid log disclosure. [#6625]
+  Use a Proc instead: `config.redis = { password: ->(username) { "password" } }`
+
 7.3.9
 ----------
 
 - Only require activejob if necessary [#6584]
+  You might get `uninitialized constant Sidekiq::ActiveJob` if you
+  `require 'sidekiq'` before `require 'rails'`.
 - Fix iterable job cancellation [#6589]
 - Web UI accessibility improvements [#6604]
 

data/README.md

@@ -4,21 +4,23 @@ Sidekiq
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
 ![Build](https://github.com/sidekiq/sidekiq/workflows/CI/badge.svg)
 
-Simple, efficient background processing for Ruby.
+Simple, efficient background jobs for Ruby.
 
 Sidekiq uses threads to handle many jobs at the same time in the
-same process.  It does not require Rails but will integrate tightly with
-Rails to make background processing dead simple.
+same process. Sidekiq can be used by any Ruby application.
 
 
 Requirements
 -----------------
 
-- Redis: Redis 6.2+ or Dragonfly 1.13+
-- Ruby: MRI 2.7+ or JRuby 9.3+.
+- Redis: Redis 7.0+, Valkey 7.2+ or Dragonfly 1.27+
+- Ruby: MRI 3.2+ or JRuby 9.4+.
 
-Sidekiq 7.0 supports Rails 6.0+ but does not require it.
-As of 7.2, Sidekiq supports Dragonfly as an alternative to Redis for data storage.
+Sidekiq 8.0 supports Rails and Active Job 7.0+.
+
+Sidekiq supports [Valkey](https://valkey.io) and [Dragonfly](https://www.dragonflydb.io) as Redis alternatives.
+Redis 7.2.4 is considered to be the canonical implementation.
+Incompatibilities with that version are considered bugs.
 
 Installation
 -----------------
@@ -42,6 +44,7 @@ The benchmark in `bin/sidekiqload` creates 500,000 no-op jobs and drains them as
 This requires a lot of Redis network I/O and JSON parsing.
 This benchmark is IO-bound so we increase the concurrency to 25.
 If your application is sending lots of emails or performing other network-intensive work, you could see a similar benefit but be careful not to saturate the CPU.
+Real world applications will rarely if ever need to use concurrency greater than 10.
 
 Version | Time to process 500k jobs | Throughput (jobs/sec) | Ruby | Concurrency | Job Type
 -----------------|------|---------|---------|------------------------|---
@@ -62,7 +65,7 @@ Want to Upgrade?
 Use `bundle up sidekiq` to upgrade Sidekiq and all its dependencies.
 Upgrade notes between each major version can be found in the `docs/` directory.
 
-I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
+I also sell [Sidekiq Pro](https://billing.contribsys.com/spro/) and [Sidekiq Enterprise](https://billing.contribsys.com/sent/new.cgi), extensions to Sidekiq which provide more
 features, a commercial-friendly license and allow you to support high
 quality open source development all at the same time.  Please see the
 [Sidekiq](https://sidekiq.org/) homepage for more detail.
@@ -71,7 +74,7 @@ quality open source development all at the same time.  Please see the
 Problems?
 -----------------
 
-**Please do not directly email any Sidekiq committers with questions or problems.**
+**Do not directly email any Sidekiq committers with questions or problems.**
 A community is best served when discussions are held in public.
 
 If you have a problem, please review the [FAQ](https://github.com/sidekiq/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/sidekiq/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
@@ -86,21 +89,21 @@ Useful resources:
 * Occasional announcements are made to the [@sidekiq](https://ruby.social/@sidekiq) Mastodon account.
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
 
-Every Thursday morning is Sidekiq office hour: I video chat and answer questions.
+Every Thursday morning is Sidekiq Office Hour: I video chat and answer questions.
 See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
 
 Contributing
 -----------------
 
-Please see [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
+See [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
 
 License
 -----------------
 
-Please see [LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/LICENSE.txt) for licensing details.
+See [LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/LICENSE.txt) for licensing details.
 The license for Sidekiq Pro and Sidekiq Enterprise can be found in [COMM-LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.txt).
 
 Author
 -----------------
 
-Mike Perham, [@getajobmike](https://ruby.social/@getajobmike) / [@sidekiq](https://ruby.social/@sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
+Mike Perham, [mastodon](https://ruby.social/@getajobmike), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)

data/bin/sidekiqload

@@ -28,7 +28,7 @@ require "ruby-prof" if ENV["PROFILE"]
 require "bundler/setup"
 Bundler.require(:default, :load_test)
 
-latency = Integer(ENV["LATENCY"] || 1)
+latency = Integer(ENV["LATENCY"] || 0)
 if latency > 0
   # brew tap shopify/shopify
   # brew install toxiproxy
@@ -86,7 +86,7 @@ class Loader
   def initialize
     @iter = ENV["GC"] ? 10 : 500
     @count = Integer(ENV["COUNT"] || 1_000)
-    @latency = Integer(ENV["LATENCY"] || 1)
+    @latency = Integer(ENV["LATENCY"] || 0)
   end
 
   def configure
@@ -137,7 +137,7 @@ class Loader
   end
 
   def setup
-    Sidekiq.logger.error("Setup RSS: #{Process.rss}")
+    Sidekiq.logger.warn("Setup RSS: #{Process.rss}")
     Sidekiq.redis { |c| c.flushdb }
     start = Time.now
     if ENV["AJ"]
@@ -167,9 +167,9 @@ class Loader
         if total == 0
           ending = Time.now - @start
           size = @iter * @count
-          Sidekiq.logger.error("Done, #{size} jobs in #{ending} sec, #{(size / ending).to_i} jobs/sec")
-          Sidekiq.logger.error("Ending RSS: #{Process.rss}")
-          Sidekiq.logger.error("Now here's the latency for three jobs")
+          Sidekiq.logger.warn("Done, #{size} jobs in #{ending} sec, #{(size / ending).to_i} jobs/sec")
+          Sidekiq.logger.warn("Ending RSS: #{Process.rss}")
+          Sidekiq.logger.warn("Now here's the latency for three jobs")
 
           if ENV["AJ"]
             LoadJob.perform_later("", 1, {}, Time.now.to_f)
@@ -191,7 +191,7 @@ class Loader
   end
 
   def with_latency(latency, &block)
-    Sidekiq.logger.error "Simulating #{latency}ms of latency between Sidekiq and redis"
+    Sidekiq.logger.warn "Simulating #{latency}ms of latency between Sidekiq and redis"
     if latency > 0
       Toxiproxy[:redis].downstream(:latency, latency: latency).apply(&block)
     else
@@ -204,8 +204,8 @@ class Loader
     monitor
 
     if ENV["PROFILE"]
-      RubyProf.exclude_threads = [@monitor]
-      RubyProf.start
+      $profile = RubyProf::Profile.new(exclude_threads: [@monitor])
+      $profile.start
     elsif ENV["GC"]
       GC.start
       GC.compact
@@ -236,7 +236,7 @@ class Loader
     Sidekiq.logger.error("GC End RSS: #{Process.rss}") if ENV["GC"]
     if ENV["PROFILE"]
       Sidekiq.logger.error("Profiling...")
-      result = RubyProf.stop
+      result = $profile.stop
       printer = RubyProf::GraphHtmlPrinter.new(result)
       printer.print(File.new("output.html", "w"), min_percent: 1)
     end

data/bin/webload

@@ -0,0 +1,69 @@
+#!/usr/bin/env ruby
+
+require 'bundler'
+Bundler.setup
+
+# This skeleton allows you to run Sidekiq::Web page rendering
+# through Vernier for tuning.
+require "sidekiq/web"
+require "rack/test"
+require "vernier"
+
+Sidekiq::Web.configure do |config|
+  config.middlewares.clear # remove csrf
+end
+
+class SomeJob
+  include Sidekiq::Job
+end
+
+class BenchWeb
+  include Rack::Test::Methods
+
+  def app
+    Sidekiq::Web.new
+  end
+
+  def warmup(page = "/scheduled")
+    # Sidekiq.redis {|c| c.flushdb }
+
+    # 100.times do |idx|
+    #   SomeJob.perform_at(idx, 1, 3, "mike", {"foo" => "bar"})
+    # end
+
+    100.times do
+      get page
+    end
+  end
+
+  def load(page = "/scheduled", count = 10_000)
+    profile do
+      count.times do
+        get page
+        raise last_response.inspect unless last_response.status == 200
+      end
+    end
+  end
+
+  def profile(&)
+    if ENV["PROF"]
+      Vernier.profile(out: "profile.json.gz", &)
+    else
+      yield
+    end
+  end
+end
+
+def timer(name="block", count = 10_000)
+  a = Time.now
+  yield count
+  b = Time.now
+  puts "#{name} in #{b - a} sec"
+end
+
+page = "/busy"
+b = BenchWeb.new
+b.warmup(page)
+timer(page) do |count|
+  b.load(page, count)
+end

data/lib/active_job/queue_adapters/sidekiq_adapter.rb

@@ -1,75 +1,111 @@
 # frozen_string_literal: true
 
-module ActiveJob
-  module QueueAdapters
-    # Explicitly remove the implementation existing in older rails'.
-    remove_const(:SidekiqAdapter) if const_defined?(:SidekiqAdapter)
+begin
+  gem "activejob", ">= 7.0"
+  require "active_job"
 
-    # Sidekiq adapter for Active Job
-    #
-    # To use Sidekiq set the queue_adapter config to +:sidekiq+.
-    #
-    #   Rails.application.config.active_job.queue_adapter = :sidekiq
-    class SidekiqAdapter
-      # Defines whether enqueuing should happen implicitly to after commit when called
-      # from inside a transaction.
+  module Sidekiq
+    module ActiveJob
       # @api private
-      def enqueue_after_transaction_commit?
-        true
-      end
+      class Wrapper
+        include Sidekiq::Job
 
-      # @api private
-      def enqueue(job)
-        job.provider_job_id = JobWrapper.set(
-          wrapped: job.class,
-          queue: job.queue_name
-        ).perform_async(job.serialize)
+        def perform(job_data)
+          ::ActiveJob::Base.execute(job_data.merge("provider_job_id" => jid))
+        end
       end
+    end
+  end
 
-      # @api private
-      def enqueue_at(job, timestamp)
-        job.provider_job_id = JobWrapper.set(
-          wrapped: job.class,
-          queue: job.queue_name
-        ).perform_at(timestamp, job.serialize)
-      end
+  unless ActiveJob::Base.respond_to?(:sidekiq_options)
+    # By including the Options module, we allow AJs to directly control sidekiq features
+    # via the *sidekiq_options* class method and, for instance, not use AJ's retry system.
+    # AJ retries don't show up in the Sidekiq UI Retries tab, don't save any error data, can't be
+    # manually retried, don't automatically die, etc.
+    #
+    #   class SomeJob < ActiveJob::Base
+    #     queue_as :default
+    #     sidekiq_options retry: 3, backtrace: 10
+    #     def perform
+    #     end
+    #   end
+    ActiveJob::Base.include Sidekiq::Job::Options
+  end
 
-      # @api private
-      def enqueue_all(jobs)
-        enqueued_count = 0
-        jobs.group_by(&:class).each do |job_class, same_class_jobs|
-          same_class_jobs.group_by(&:queue_name).each do |queue, same_class_and_queue_jobs|
-            immediate_jobs, scheduled_jobs = same_class_and_queue_jobs.partition { |job| job.scheduled_at.nil? }
+  # Patch the ActiveJob module
+  module ActiveJob
+    module QueueAdapters
+      # Explicitly remove the implementation existing in older Rails.
+      remove_const(:SidekiqAdapter) if const_defined?(:SidekiqAdapter)
 
-            if immediate_jobs.any?
-              jids = Sidekiq::Client.push_bulk(
-                "class" => JobWrapper,
-                "wrapped" => job_class,
-                "queue" => queue,
-                "args" => immediate_jobs.map { |job| [job.serialize] }
-              )
-              enqueued_count += jids.compact.size
-            end
+      # Sidekiq adapter for Active Job
+      #
+      # To use Sidekiq set the queue_adapter config to +:sidekiq+.
+      #
+      #   Rails.application.config.active_job.queue_adapter = :sidekiq
+      class SidekiqAdapter
+        # Defines whether enqueuing should happen implicitly to after commit when called
+        # from inside a transaction.
+        # @api private
+        def enqueue_after_transaction_commit?
+          true
+        end
+
+        # @api private
+        def enqueue(job)
+          job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
+            wrapped: job.class,
+            queue: job.queue_name
+          ).perform_async(job.serialize)
+        end
+
+        # @api private
+        def enqueue_at(job, timestamp)
+          job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
+            wrapped: job.class,
+            queue: job.queue_name
+          ).perform_at(timestamp, job.serialize)
+        end
 
-            if scheduled_jobs.any?
-              jids = Sidekiq::Client.push_bulk(
-                "class" => JobWrapper,
-                "wrapped" => job_class,
-                "queue" => queue,
-                "args" => scheduled_jobs.map { |job| [job.serialize] },
-                "at" => scheduled_jobs.map { |job| job.scheduled_at&.to_f }
-              )
-              enqueued_count += jids.compact.size
+        # @api private
+        def enqueue_all(jobs)
+          enqueued_count = 0
+          jobs.group_by(&:class).each do |job_class, same_class_jobs|
+            same_class_jobs.group_by(&:queue_name).each do |queue, same_class_and_queue_jobs|
+              immediate_jobs, scheduled_jobs = same_class_and_queue_jobs.partition { |job| job.scheduled_at.nil? }
+
+              if immediate_jobs.any?
+                jids = Sidekiq::Client.push_bulk(
+                  "class" => Sidekiq::ActiveJob::Wrapper,
+                  "wrapped" => job_class,
+                  "queue" => queue,
+                  "args" => immediate_jobs.map { |job| [job.serialize] }
+                )
+                enqueued_count += jids.compact.size
+              end
+
+              if scheduled_jobs.any?
+                jids = Sidekiq::Client.push_bulk(
+                  "class" => Sidekiq::ActiveJob::Wrapper,
+                  "wrapped" => job_class,
+                  "queue" => queue,
+                  "args" => scheduled_jobs.map { |job| [job.serialize] },
+                  "at" => scheduled_jobs.map { |job| job.scheduled_at&.to_f }
+                )
+                enqueued_count += jids.compact.size
+              end
             end
           end
+          enqueued_count
         end
-        enqueued_count
-      end
 
-      # Defines a class alias for backwards compatibility with enqueued Active Job jobs.
-      # @api private
-      class JobWrapper < Sidekiq::ActiveJob::Wrapper
+        # Defines a class alias for backwards compatibility with enqueued Active Job jobs.
+        # @api private
+        class JobWrapper < Sidekiq::ActiveJob::Wrapper
+        end
       end
     end
   end
+rescue Gem::LoadError
+  # ActiveJob not available or version requirement not met
 end