sidekiq 7.3.9 → 8.0.0.beta1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dccc402e8f4e309a47a2adb62418fa29989012e8c94f34aa11200b098dc375f2
-  data.tar.gz: 71b64a582d5971b045d60bae38dc26da265b60ff9a099ca3b2f9c7df9035e805
+  metadata.gz: f7280923fe38a56b42bb33c639f9e0a365c1f585f928e8cdfd3ce06fe989fc7e
+  data.tar.gz: e8c6a01e393390f1a3ab3a48c04b3621efd5621c6b184240c3e8316602e24e9c
 SHA512:
-  metadata.gz: d9148b613a222ca9617ceebe25bb04a5d086edacdafad4a426b6232662d3b583db462dc5e0491297f7859037c5166bcb541bddcd3949f6d3b5a1c2e3fc572b65
-  data.tar.gz: 93a797cefd1a68adb236c7538c28571467c328d6965af0bc5ce505a1391d6f5b6e2ae2c87b42110e837dcfa8d788aff55309806bbb1bcd181e205c3bc66adc29
+  metadata.gz: 076a8f70170dfdc6ec78430ca41b182ed986453bdfbcfe24b7dd5ef1d8f2321b560cb94201c2f2c3a30c2ce42a31e38ce14151f0ebbee90bc97a2463b427747a
+  data.tar.gz: 8cc4828931a73da67ae40edb14508b82624781b9732280ce9e8db615c8bed1039b89b8b50db54c35bbffd4a010f904598be2bb4aa037228bbd4b4b325b2b9421

data/Changes.md

@@ -2,12 +2,29 @@
 
 [Sidekiq Changes](https://github.com/sidekiq/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/sidekiq/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/sidekiq/sidekiq/blob/main/Ent-Changes.md)
 
-7.3.9
+HEAD / main
 ----------
 
-- Only require activejob if necessary [#6584]
-- Fix iterable job cancellation [#6589]
-- Web UI accessibility improvements [#6604]
+- **WARNING** The underlying class name for Active Jobs has changed from `ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper` to `Sidekiq::ActiveJob::Wrapper`.
+- **WARNING** The `created_at` and `enqueued_at` attributes are now stored as
+  integer milliseconds, rather than epoch floats. This is meant to avoid precision
+  issues with JSON and JavaScript's 53-bit Floats. Example:
+  `"created_at" => 1234567890.123456` -> `"created_at" => 1234567890123`.
+- **NEW FEATURE** Job Profiling is now supported with [Vernier](https://vernier.prof)
+  which makes it really easy to performance tune your slow jobs.
+  The Web UI contains a new **Profiles** tab to view any collected profile data.
+  Please read the new [Profiling](https://github.com/sidekiq/sidekiq/wiki/Profiling) wiki page for details.
+- CurrentAttribute support now uses `ActiveJob::Arguments` to serialize the context object, supporting Symbols and GlobalID.
+  The change should be backwards compatible. [#6510]
+- Freshen up `Sidekiq::Web` to simplify the code and improve security [#6532]
+  The CSS has been rewritten from scratch to remove the Bootstrap framework.
+- Default error logging has been modified to use Ruby's `Exception#detailed_message` and `#full_message` APIs.
+- CI now runs against Redis, Dragonfly and Valkey.
+- The Web UI's language picker now shows options in the native language
+- Remove global variable usage within the codebase
+- Adjust Sidekiq's default thread priority to -1 for a 50ms timeslice.
+  This can help avoid TimeoutErrors when Sidekiq is overloaded. [#6543]
+- Support: Redis 7.2+, Ruby 3.2+, Rails 7.0+
 
 7.3.8
 ----------

data/README.md

@@ -4,21 +4,23 @@ Sidekiq
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
 ![Build](https://github.com/sidekiq/sidekiq/workflows/CI/badge.svg)
 
-Simple, efficient background processing for Ruby.
+Simple, efficient background jobs for Ruby.
 
 Sidekiq uses threads to handle many jobs at the same time in the
-same process.  It does not require Rails but will integrate tightly with
-Rails to make background processing dead simple.
+same process. Sidekiq can be used by any Ruby application.
 
 
 Requirements
 -----------------
 
-- Redis: Redis 6.2+ or Dragonfly 1.13+
-- Ruby: MRI 2.7+ or JRuby 9.3+.
+- Redis: Redis 7.2+, Valkey 7.2+ or Dragonfly 1.13+
+- Ruby: MRI 3.2+ or JRuby 9.4+.
 
-Sidekiq 7.0 supports Rails 6.0+ but does not require it.
-As of 7.2, Sidekiq supports Dragonfly as an alternative to Redis for data storage.
+Sidekiq 8.0 supports Rails and Active Job 7.0+.
+
+Sidekiq supports [Valkey](https://valkey.io) and [Dragonfly](https://www.dragonflydb.io) as Redis alternatives.
+Redis 7.2.4 is considered to be the canonical implementation.
+Incompatibilities with that version are considered bugs.
 
 Installation
 -----------------
@@ -42,6 +44,7 @@ The benchmark in `bin/sidekiqload` creates 500,000 no-op jobs and drains them as
 This requires a lot of Redis network I/O and JSON parsing.
 This benchmark is IO-bound so we increase the concurrency to 25.
 If your application is sending lots of emails or performing other network-intensive work, you could see a similar benefit but be careful not to saturate the CPU.
+Real world applications will rarely if ever need to use concurrency greater than 10.
 
 Version | Time to process 500k jobs | Throughput (jobs/sec) | Ruby | Concurrency | Job Type
 -----------------|------|---------|---------|------------------------|---
@@ -62,7 +65,7 @@ Want to Upgrade?
 Use `bundle up sidekiq` to upgrade Sidekiq and all its dependencies.
 Upgrade notes between each major version can be found in the `docs/` directory.
 
-I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
+I also sell [Sidekiq Pro](https://billing.contribsys.com/spro/) and [Sidekiq Enterprise](https://billing.contribsys.com/sent/new.cgi), extensions to Sidekiq which provide more
 features, a commercial-friendly license and allow you to support high
 quality open source development all at the same time.  Please see the
 [Sidekiq](https://sidekiq.org/) homepage for more detail.
@@ -71,7 +74,7 @@ quality open source development all at the same time.  Please see the
 Problems?
 -----------------
 
-**Please do not directly email any Sidekiq committers with questions or problems.**
+**Do not directly email any Sidekiq committers with questions or problems.**
 A community is best served when discussions are held in public.
 
 If you have a problem, please review the [FAQ](https://github.com/sidekiq/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/sidekiq/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
@@ -86,21 +89,21 @@ Useful resources:
 * Occasional announcements are made to the [@sidekiq](https://ruby.social/@sidekiq) Mastodon account.
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
 
-Every Thursday morning is Sidekiq office hour: I video chat and answer questions.
+Every Thursday morning is Sidekiq Office Hour: I video chat and answer questions.
 See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
 
 Contributing
 -----------------
 
-Please see [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
+See [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
 
 License
 -----------------
 
-Please see [LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/LICENSE.txt) for licensing details.
+See [LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/LICENSE.txt) for licensing details.
 The license for Sidekiq Pro and Sidekiq Enterprise can be found in [COMM-LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.txt).
 
 Author
 -----------------
 
-Mike Perham, [@getajobmike](https://ruby.social/@getajobmike) / [@sidekiq](https://ruby.social/@sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
+Mike Perham, [mastodon](https://ruby.social/@getajobmike), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)

data/bin/sidekiqload

@@ -28,7 +28,7 @@ require "ruby-prof" if ENV["PROFILE"]
 require "bundler/setup"
 Bundler.require(:default, :load_test)
 
-latency = Integer(ENV["LATENCY"] || 1)
+latency = Integer(ENV["LATENCY"] || 0)
 if latency > 0
   # brew tap shopify/shopify
   # brew install toxiproxy
@@ -86,7 +86,7 @@ class Loader
   def initialize
     @iter = ENV["GC"] ? 10 : 500
     @count = Integer(ENV["COUNT"] || 1_000)
-    @latency = Integer(ENV["LATENCY"] || 1)
+    @latency = Integer(ENV["LATENCY"] || 0)
   end
 
   def configure
@@ -137,7 +137,7 @@ class Loader
   end
 
   def setup
-    Sidekiq.logger.error("Setup RSS: #{Process.rss}")
+    Sidekiq.logger.warn("Setup RSS: #{Process.rss}")
     Sidekiq.redis { |c| c.flushdb }
     start = Time.now
     if ENV["AJ"]
@@ -167,9 +167,9 @@ class Loader
         if total == 0
           ending = Time.now - @start
           size = @iter * @count
-          Sidekiq.logger.error("Done, #{size} jobs in #{ending} sec, #{(size / ending).to_i} jobs/sec")
-          Sidekiq.logger.error("Ending RSS: #{Process.rss}")
-          Sidekiq.logger.error("Now here's the latency for three jobs")
+          Sidekiq.logger.warn("Done, #{size} jobs in #{ending} sec, #{(size / ending).to_i} jobs/sec")
+          Sidekiq.logger.warn("Ending RSS: #{Process.rss}")
+          Sidekiq.logger.warn("Now here's the latency for three jobs")
 
           if ENV["AJ"]
             LoadJob.perform_later("", 1, {}, Time.now.to_f)
@@ -191,7 +191,7 @@ class Loader
   end
 
   def with_latency(latency, &block)
-    Sidekiq.logger.error "Simulating #{latency}ms of latency between Sidekiq and redis"
+    Sidekiq.logger.warn "Simulating #{latency}ms of latency between Sidekiq and redis"
     if latency > 0
       Toxiproxy[:redis].downstream(:latency, latency: latency).apply(&block)
     else
@@ -204,8 +204,8 @@ class Loader
     monitor
 
     if ENV["PROFILE"]
-      RubyProf.exclude_threads = [@monitor]
-      RubyProf.start
+      $profile = RubyProf::Profile.new(exclude_threads: [@monitor])
+      $profile.start
     elsif ENV["GC"]
       GC.start
       GC.compact
@@ -236,7 +236,7 @@ class Loader
     Sidekiq.logger.error("GC End RSS: #{Process.rss}") if ENV["GC"]
     if ENV["PROFILE"]
       Sidekiq.logger.error("Profiling...")
-      result = RubyProf.stop
+      result = $profile.stop
       printer = RubyProf::GraphHtmlPrinter.new(result)
       printer.print(File.new("output.html", "w"), min_percent: 1)
     end

data/bin/webload

@@ -0,0 +1,69 @@
+#!/usr/bin/env ruby
+
+require 'bundler'
+Bundler.setup
+
+# This skeleton allows you to run Sidekiq::Web page rendering
+# through Vernier for tuning.
+require "sidekiq/web"
+require "rack/test"
+require "vernier"
+
+Sidekiq::Web.configure do |config|
+  config.middlewares.clear # remove csrf
+end
+
+class SomeJob
+  include Sidekiq::Job
+end
+
+class BenchWeb
+  include Rack::Test::Methods
+
+  def app
+    Sidekiq::Web.new
+  end
+
+  def warmup(page = "/scheduled")
+    # Sidekiq.redis {|c| c.flushdb }
+
+    # 100.times do |idx|
+    #   SomeJob.perform_at(idx, 1, 3, "mike", {"foo" => "bar"})
+    # end
+
+    100.times do
+      get page
+    end
+  end
+
+  def load(page = "/scheduled", count = 10_000)
+    profile do
+      count.times do
+        get page
+        raise last_response.inspect unless last_response.status == 200
+      end
+    end
+  end
+
+  def profile(&)
+    if ENV["PROF"]
+      Vernier.profile(out: "profile.json.gz", &)
+    else
+      yield
+    end
+  end
+end
+
+def timer(name="block", count = 10_000)
+  a = Time.now
+  yield count
+  b = Time.now
+  puts "#{name} in #{b - a} sec"
+end
+
+page = "/busy"
+b = BenchWeb.new
+b.warmup(page)
+timer(page) do |count|
+  b.load(page, count)
+end

data/lib/active_job/queue_adapters/sidekiq_adapter.rb

@@ -2,7 +2,7 @@
 
 module ActiveJob
   module QueueAdapters
-    # Explicitly remove the implementation existing in older rails'.
+    # Explicitly remove the implementation existing in older Rails.
     remove_const(:SidekiqAdapter) if const_defined?(:SidekiqAdapter)
 
     # Sidekiq adapter for Active Job
@@ -20,7 +20,7 @@ module ActiveJob
 
       # @api private
       def enqueue(job)
-        job.provider_job_id = JobWrapper.set(
+        job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
           wrapped: job.class,
           queue: job.queue_name
         ).perform_async(job.serialize)
@@ -28,7 +28,7 @@ module ActiveJob
 
       # @api private
       def enqueue_at(job, timestamp)
-        job.provider_job_id = JobWrapper.set(
+        job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(
           wrapped: job.class,
           queue: job.queue_name
         ).perform_at(timestamp, job.serialize)
@@ -43,7 +43,7 @@ module ActiveJob
 
             if immediate_jobs.any?
               jids = Sidekiq::Client.push_bulk(
-                "class" => JobWrapper,
+                "class" => Sidekiq::ActiveJob::Wrapper,
                 "wrapped" => job_class,
                 "queue" => queue,
                 "args" => immediate_jobs.map { |job| [job.serialize] }
@@ -53,7 +53,7 @@ module ActiveJob
 
             if scheduled_jobs.any?
               jids = Sidekiq::Client.push_bulk(
-                "class" => JobWrapper,
+                "class" => Sidekiq::ActiveJob::Wrapper,
                 "wrapped" => job_class,
                 "queue" => queue,
                 "args" => scheduled_jobs.map { |job| [job.serialize] },

data/lib/sidekiq/api.rb

@@ -1,10 +1,8 @@
 # frozen_string_literal: true
 
-require "sidekiq"
-
 require "zlib"
-require "set"
 
+require "sidekiq"
 require "sidekiq/metrics/query"
 
 #
@@ -101,11 +99,22 @@ module Sidekiq
         rescue
           {}
         end
-        now = Time.now.to_f
-        thence = job["enqueued_at"] || now
-        now - thence
+
+        enqueued_at = job["enqueued_at"]
+        if enqueued_at
+          if enqueued_at.is_a?(Float)
+            # old format
+            now = Time.now.to_f
+            now - enqueued_at
+          else
+            now = ::Process.clock_gettime(::Process::CLOCK_REALTIME, :millisecond)
+            (now - enqueued_at) / 1000.0
+          end
+        else
+          0.0
+        end
       else
-        0
+        0.0
       end
 
       @stats = {
@@ -265,11 +274,22 @@ module Sidekiq
       entry = Sidekiq.redis { |conn|
         conn.lindex(@rname, -1)
       }
-      return 0 unless entry
+      return 0.0 unless entry
+
       job = Sidekiq.load_json(entry)
-      now = Time.now.to_f
-      thence = job["enqueued_at"] || now
-      now - thence
+      enqueued_at = job["enqueued_at"]
+      if enqueued_at
+        if enqueued_at.is_a?(Float)
+          # old format
+          now = Time.now.to_f
+          now - enqueued_at
+        else
+          now = ::Process.clock_gettime(::Process::CLOCK_REALTIME, :millisecond)
+          (now - enqueued_at) / 1000.0
+        end
+      else
+        0.0
+      end
     end
 
     def each
@@ -422,11 +442,13 @@ module Sidekiq
     end
 
     def enqueued_at
-      self["enqueued_at"] ? Time.at(self["enqueued_at"]).utc : nil
+      if self["enqueued_at"]
+        time_from_timestamp(self["enqueued_at"])
+      end
     end
 
     def created_at
-      Time.at(self["created_at"] || self["enqueued_at"] || 0).utc
+      time_from_timestamp(self["created_at"] || self["enqueued_at"] || 0)
     end
 
     def tags
@@ -444,8 +466,17 @@ module Sidekiq
     end
 
     def latency
-      now = Time.now.to_f
-      now - (@item["enqueued_at"] || @item["created_at"] || now)
+      timestamp = @item["enqueued_at"] || @item["created_at"]
+      if timestamp
+        if timestamp.is_a?(Float)
+          # old format
+          Time.now.to_f - timestamp
+        else
+          (::Process.clock_gettime(::Process::CLOCK_REALTIME, :millisecond) - timestamp) / 1000.0
+        end
+      else
+        0.0
+      end
     end
 
     # Remove this job from the queue
@@ -494,6 +525,15 @@ module Sidekiq
       uncompressed = Zlib::Inflate.inflate(strict_base64_decoded)
       Sidekiq.load_json(uncompressed)
     end
+
+    def time_from_timestamp(timestamp)
+      if timestamp.is_a?(Float)
+        # old format, timestamps were stored as fractional seconds since the epoch
+        Time.at(timestamp).utc
+      else
+        Time.at(timestamp / 1000, timestamp % 1000, :millisecond)
+      end
+    end
   end
 
   # Represents a job within a Redis sorted set where the score
@@ -1142,7 +1182,7 @@ module Sidekiq
         end
       end
 
-      results.sort_by { |(_, _, hsh)| hsh.raw("run_at") }.each(&block)
+      results.sort_by { |(_, _, work)| work.run_at }.each(&block)
     end
 
     # Note that #size is only as accurate as Sidekiq's heartbeat,
@@ -1172,13 +1212,14 @@ module Sidekiq
     #
     # @param jid [String] the job identifier
     # @return [Sidekiq::Work] the work or nil
-    def find_work_by_jid(jid)
+    def find_work(jid)
       each do |_process_id, _thread_id, work|
         job = work.job
         return work if job.jid == jid
       end
       nil
     end
+    alias_method :find_work_by_jid, :find_work
   end
 
   # Sidekiq::Work represents a job which is currently executing.
@@ -1208,33 +1249,64 @@ module Sidekiq
     def payload
       @hsh["payload"]
     end
+  end
 
-    # deprecated
-    def [](key)
-      kwargs = {uplevel: 1}
-      kwargs[:category] = :deprecated if RUBY_VERSION > "3.0" # TODO
-      warn("Direct access to `Sidekiq::Work` attributes is deprecated, please use `#payload`, `#queue`, `#run_at` or `#job` instead", **kwargs)
+  # Since "worker" is a nebulous term, we've deprecated the use of this class name.
+  # Is "worker" a process, a type of job, a thread? Undefined!
+  # WorkSet better describes the data.
+  Workers = WorkSet
 
-      @hsh[key]
-    end
+  class ProfileSet
+    include Enumerable
 
-    # :nodoc:
-    # @api private
-    def raw(name)
-      @hsh[name]
+    # This is a point in time/snapshot API, you'll need to instantiate a new instance
+    # if you want to fetch newer records.
+    def initialize
+      @records = Sidekiq.redis do |c|
+        # This throws away expired profiles
+        c.zremrangebyscore("profiles", "-inf", Time.now.to_f.to_s)
+        # retreive records, newest to oldest
+        c.zrange("profiles", "+inf", 0, "byscore", "rev")
+      end
     end
 
-    def method_missing(*all)
-      @hsh.send(*all)
+    def size
+      @records.size
     end
 
-    def respond_to_missing?(name, *args)
-      @hsh.respond_to?(name)
+    def each(&block)
+      fetch_keys = %w[started_at jid type token size elapsed].freeze
+      arrays = Sidekiq.redis do |c|
+        c.pipelined do |p|
+          @records.each do |key|
+            p.hmget(key, *fetch_keys)
+          end
+        end
+      end
+
+      arrays.compact.map { |arr| ProfileRecord.new(arr) }.each(&block)
     end
   end
 
-  # Since "worker" is a nebulous term, we've deprecated the use of this class name.
-  # Is "worker" a process, a type of job, a thread? Undefined!
-  # WorkSet better describes the data.
-  Workers = WorkSet
+  class ProfileRecord
+    attr_reader :started_at, :jid, :type, :token, :size, :elapsed
+
+    def initialize(arr)
+      # Must be same order as fetch_keys above
+      @started_at = Time.at(Integer(arr[0]))
+      @jid = arr[1]
+      @type = arr[2]
+      @token = arr[3]
+      @size = Integer(arr[4])
+      @elapsed = Float(arr[5])
+    end
+
+    def key
+      "#{token}-#{jid}"
+    end
+
+    def data
+      Sidekiq.redis { |c| c.hget(key, "data") }
+    end
+  end
 end

data/lib/sidekiq/capsule.rb

@@ -27,7 +27,7 @@ module Sidekiq
     attr_reader :mode
     attr_reader :weights
 
-    def_delegators :@config, :[], :[]=, :fetch, :key?, :has_key?, :merge!, :dig
+    def_delegators :@config, :[], :[]=, :fetch, :key?, :has_key?, :merge!, :dig, :thread_priority
 
     def initialize(name, config)
       @name = name

data/lib/sidekiq/cli.rb

@@ -3,7 +3,6 @@
 $stdout.sync = true
 
 require "yaml"
-require "singleton"
 require "optparse"
 require "erb"
 require "fileutils"
@@ -17,7 +16,6 @@ require "sidekiq/launcher"
 module Sidekiq # :nodoc:
   class CLI
     include Sidekiq::Component
-    include Singleton unless $TESTING
 
     attr_accessor :launcher
     attr_accessor :environment
@@ -31,6 +29,10 @@ module Sidekiq # :nodoc:
       validate!
     end
 
+    def self.instance
+      @instance ||= new
+    end
+
     def jruby?
       defined?(::JRUBY_VERSION)
     end
@@ -74,7 +76,7 @@ module Sidekiq # :nodoc:
       # fire startup and start multithreading.
       info = @config.redis_info
       ver = Gem::Version.new(info["redis_version"])
-      raise "You are connecting to Redis #{ver}, Sidekiq requires Redis 6.2.0 or greater" if ver < Gem::Version.new("6.2.0")
+      raise "You are connected to Redis #{ver}, Sidekiq requires Redis 7.2.0 or greater" if ver < Gem::Version.new("7.2.0")
 
       maxmemory_policy = info["maxmemory_policy"]
       if maxmemory_policy != "noeviction" && maxmemory_policy != ""
@@ -298,29 +300,18 @@ module Sidekiq # :nodoc:
 
       if File.directory?(@config[:require])
         require "rails"
-        if ::Rails::VERSION::MAJOR < 6
-          warn "Sidekiq #{Sidekiq::VERSION} only supports Rails 6+"
+        if ::Rails::VERSION::MAJOR < 7
+          warn "Sidekiq #{Sidekiq::VERSION} only supports Rails 7+"
         end
         require "sidekiq/rails"
         require File.expand_path("#{@config[:require]}/config/environment.rb")
-        @config[:tag] ||= default_tag
+        @config[:tag] ||= default_tag(::Rails.root)
       else
         require @config[:require]
+        @config[:tag] ||= default_tag
       end
     end
 
-    def default_tag
-      dir = ::Rails.root
-      name = File.basename(dir)
-      prevdir = File.dirname(dir) # Capistrano release directory?
-      if name.to_i != 0 && prevdir
-        if File.basename(prevdir) == "releases"
-          return File.basename(File.dirname(prevdir))
-        end
-      end
-      name
-    end
-
     def validate!
       if !File.exist?(@config[:require]) ||
           (File.directory?(@config[:require]) && !File.exist?("#{@config[:require]}/config/application.rb"))
@@ -395,7 +386,12 @@ module Sidekiq # :nodoc:
     end
 
     def initialize_logger
-      @config.logger.level = ::Logger::DEBUG if @config[:verbose]
+      if @config[:verbose] || ENV["DEBUG_INVOCATION"] == "1"
+        # DEBUG_INVOCATION is a systemd-ism triggered by
+        # RestartMode=debug. We turn on debugging when the
+        # sidekiq process crashes and is restarted with this flag.
+        @config.logger.level = ::Logger::DEBUG
+      end
     end
 
     def parse_config(path)

data/lib/sidekiq/client.rb

@@ -67,9 +67,7 @@ module Sidekiq
         c.pipelined do |p|
           p.hsetnx(key, "cancelled", Time.now.to_i)
           p.hget(key, "cancelled")
-          p.expire(key, Sidekiq::Job::Iterable::STATE_TTL)
-          # TODO When Redis 7.2 is required
-          # p.expire(key, Sidekiq::Job::Iterable::STATE_TTL, "nx")
+          p.expire(key, Sidekiq::Job::Iterable::STATE_TTL, "nx")
         end
       end
       result.to_i
@@ -266,22 +264,21 @@ module Sidekiq
       if payloads.first.key?("at")
         conn.zadd("schedule", payloads.flat_map { |hash|
           at = hash["at"].to_s
-          # ActiveJob sets this but the job has not been enqueued yet
-          hash.delete("enqueued_at")
-          # TODO: Use hash.except("at") when support for Ruby 2.7 is dropped
-          hash = hash.dup
-          hash.delete("at")
+          # ActiveJob sets enqueued_at but the job has not been enqueued yet
+          hash = hash.except("enqueued_at", "at")
           [at, Sidekiq.dump_json(hash)]
         })
       else
-        queue = payloads.first["queue"]
-        now = Time.now.to_f
-        to_push = payloads.map { |entry|
-          entry["enqueued_at"] = now
-          Sidekiq.dump_json(entry)
-        }
-        conn.sadd("queues", [queue])
-        conn.lpush("queue:#{queue}", to_push)
+        now = ::Process.clock_gettime(::Process::CLOCK_REALTIME, :millisecond) # milliseconds since the epoch
+        grouped_queues = payloads.group_by { |job| job["queue"] }
+        conn.sadd("queues", grouped_queues.keys)
+        grouped_queues.each do |queue, grouped_payloads|
+          to_push = grouped_payloads.map { |entry|
+            entry["enqueued_at"] = now
+            Sidekiq.dump_json(entry)
+          }
+          conn.lpush("queue:#{queue}", to_push)
+        end
       end
     end
   end