sidekiq 5.2.7 → 8.0.5

data/LICENSE.txt

@@ -0,0 +1,9 @@
+Copyright (c) Contributed Systems LLC
+
+Sidekiq is an Open Source project licensed under the terms of
+the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
+for license text.
+
+Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
+You can find the commercial license in COMM-LICENSE.txt.
+Please see https://sidekiq.org for purchasing options.

data/README.md

@@ -2,108 +2,108 @@ Sidekiq
 ==============
 
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
-[![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
-[![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
-[![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
+![Build](https://github.com/sidekiq/sidekiq/workflows/CI/badge.svg)
 
-
-Simple, efficient background processing for Ruby.
+Simple, efficient background jobs for Ruby.
 
 Sidekiq uses threads to handle many jobs at the same time in the
-same process.  It does not require Rails but will integrate tightly with
-Rails to make background processing dead simple.
-
-Sidekiq is compatible with Resque.  It uses the exact same
-message format as Resque so it can integrate into an existing Resque processing farm.
-You can have Sidekiq and Resque run side-by-side at the same time and
-use the Resque client to enqueue jobs in Redis to be processed by Sidekiq.
-
-Performance
----------------
-
-Version |	Latency | Garbage created for 10,000 jobs	| Time to process 100,000 jobs |	Throughput
------------------|------|---------|---------|------------------------
-Sidekiq 4.0.0    | 10ms	| 151 MB  | 22 sec  | **4500 jobs/sec**
-Sidekiq 3.5.1    | 22ms	| 1257 MB | 125 sec | 800 jobs/sec
-Resque 1.25.2    |  -	  | -       | 420 sec | 240 jobs/sec
-DelayedJob 4.1.1 |  -   | -       | 465 sec | 215 jobs/sec
+same process. Sidekiq can be used by any Ruby application.
 
-<small>This benchmark can be found in `bin/sidekiqload`.</small>
 
 Requirements
 -----------------
 
-Sidekiq supports CRuby 2.2.2+ and JRuby 9k.
+- Redis: Redis 7.0+, Valkey 7.2+ or Dragonfly 1.27+
+- Ruby: MRI 3.2+ or JRuby 9.4+.
 
-All Rails releases >= 4.0 are officially supported.
-
-Redis 2.8 or greater is required.  3.0.3+ is recommended for large
-installations with thousands of worker threads.
+Sidekiq 8.0 supports Rails and Active Job 7.0+.
 
+Sidekiq supports [Valkey](https://valkey.io) and [Dragonfly](https://www.dragonflydb.io) as Redis alternatives.
+Redis 7.2.4 is considered to be the canonical implementation.
+Incompatibilities with that version are considered bugs.
 
 Installation
 -----------------
 
-    gem install sidekiq
+    bundle add sidekiq
 
 
 Getting Started
 -----------------
 
-See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
-You can watch [this Youtube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
+See the [Getting Started wiki page](https://github.com/sidekiq/sidekiq/wiki/Getting-Started) and follow the simple setup process.
+You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
 Sidekiq and see its features in action.  Here's the Web UI:
 
-![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
+![Web UI](https://github.com/sidekiq/sidekiq/raw/main/examples/web-ui.png)
 
+Performance
+---------------
+
+The benchmark in `bin/sidekiqload` creates 500,000 no-op jobs and drains them as fast as possible, assuming a fixed Redis network latency of 1ms.
+This requires a lot of Redis network I/O and JSON parsing.
+This benchmark is IO-bound so we increase the concurrency to 25.
+If your application is sending lots of emails or performing other network-intensive work, you could see a similar benefit but be careful not to saturate the CPU.
+Real world applications will rarely if ever need to use concurrency greater than 10.
+
+Version | Time to process 500k jobs | Throughput (jobs/sec) | Ruby | Concurrency | Job Type
+-----------------|------|---------|---------|------------------------|---
+Sidekiq 7.0.3 | 21.3 sec| 23,500 | 3.2.0+yjit | 30 | Sidekiq::Job
+Sidekiq 7.0.3 | 33.8 sec| 14,700 | 3.2.0+yjit | 30 | ActiveJob 7.0.4
+Sidekiq 7.0.3 | 23.5 sec| 21,300 | 3.2.0 | 30 | Sidekiq::Job
+Sidekiq 7.0.3 | 46.5 sec| 10,700 | 3.2.0 | 30 | ActiveJob 7.0.4
+Sidekiq 7.0.3 | 23.0 sec| 21,700 | 2.7.5 | 30 | Sidekiq::Job
+Sidekiq 7.0.3 | 46.5 sec| 10,850 | 2.7.5 | 30 | ActiveJob 7.0.4
+
+Most of Sidekiq's overhead is Redis network I/O.
+ActiveJob adds a notable amount of CPU overhead due to argument deserialization and callbacks.
+Concurrency of 30 was determined experimentally to maximize one CPU without saturating it.
 
 Want to Upgrade?
 -------------------
 
-I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
+Use `bundle up sidekiq` to upgrade Sidekiq and all its dependencies.
+Upgrade notes between each major version can be found in the `docs/` directory.
+
+I also sell [Sidekiq Pro](https://billing.contribsys.com/spro/) and [Sidekiq Enterprise](https://billing.contribsys.com/sent/new.cgi), extensions to Sidekiq which provide more
 features, a commercial-friendly license and allow you to support high
 quality open source development all at the same time.  Please see the
-[Sidekiq](http://sidekiq.org/) homepage for more detail.
-
-Subscribe to the **[quarterly newsletter](https://tinyletter.com/sidekiq)** to stay informed about the latest
-features and changes to Sidekiq and its bigger siblings.
+[Sidekiq](https://sidekiq.org/) homepage for more detail.
 
 
 Problems?
 -----------------
 
-**Please do not directly email any Sidekiq committers with questions or problems.**  A community is best served when discussions are held in public.
+**Do not directly email any Sidekiq committers with questions or problems.**
+A community is best served when discussions are held in public.
 
-If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
-Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
+If you have a problem, please review the [FAQ](https://github.com/sidekiq/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/sidekiq/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
+Searching the [issues](https://github.com/sidekiq/sidekiq/issues) for your problem is also a good idea.
 
-Sidekiq Pro and Sidekiq Enterprise customers get private email support.  You can purchase at http://sidekiq.org; email support@contribsys.com for help.
+Sidekiq Pro and Sidekiq Enterprise customers get private email support.
+You can purchase at https://sidekiq.org; email support@contribsys.com for help.
 
 Useful resources:
 
-* Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
-* Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
+* Product documentation is in the [wiki](https://github.com/sidekiq/sidekiq/wiki).
+* Occasional announcements are made to the [@sidekiq](https://ruby.social/@sidekiq) Mastodon account.
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q &amp; A.
 
-**No support via Twitter**
+Every Thursday morning is Sidekiq Office Hour: I video chat and answer questions.
+See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
 
-Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
-See the [Sidekiq support page](http://sidekiq.org/support.html) for details.
-
-Thanks
+Contributing
 -----------------
 
-Sidekiq stays fast by using the [JProfiler java profiler](http://www.ej-technologies.com/products/jprofiler/overview.html) to find and fix
-performance problems on JRuby.  Unfortunately MRI does not have good multithreaded profiling tools.
-
+See [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
 
 License
 -----------------
 
-Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for licensing details.
-
+See [LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/LICENSE.txt) for licensing details.
+The license for Sidekiq Pro and Sidekiq Enterprise can be found in [COMM-LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.txt).
 
 Author
 -----------------
 
-Mike Perham, [@mperham](https://twitter.com/mperham) / [@sidekiq](https://twitter.com/sidekiq), [http://www.mikeperham.com](http://www.mikeperham.com) / [http://www.contribsys.com](http://www.contribsys.com)
+Mike Perham, [mastodon](https://ruby.social/@getajobmike), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)

data/bin/multi_queue_bench

@@ -0,0 +1,271 @@
+#!/usr/bin/env ruby
+
+#
+# bin/bench is a helpful script to load test and
+# performance tune Sidekiq's core. It's a configurable script,
+# which accepts the following parameters as ENV variables.
+#
+# QUEUES
+# Number of queues to consume from. Default is 8
+#
+# PROCESSES
+# The number of processes this benchmark will create. Each process, consumes
+# from one of the available queues. When processes are more than the number of
+# queues, they are distributed to processes in round robin. Default is 8
+#
+# ELEMENTS
+# Number of jobs to push to each queue. Default is 1000
+#
+# ITERATIONS
+# Each queue pushes ITERATIONS times ELEMENTS jobs. Default is 1000
+#
+# PORT
+# The port of the Dragonfly instance. Default is 6379
+#
+# IP
+# The ip of the Dragonfly instance. Default is 127.0.0.1
+#
+# Example Usage:
+#
+# > RUBY_YJIT_ENABLE=1 THREADS=10 PROCESSES=8 QUEUES=8 bin/multi_queue_bench
+#
+# None of this script is considered a public API and may change over time.
+#
+
+# Quiet some warnings we see when running in warning mode:
+# RUBYOPT=-w bundle exec sidekiq
+$TESTING = false
+puts RUBY_DESCRIPTION
+
+require "bundler/setup"
+Bundler.require(:default, :load_test)
+
+class LoadWorker
+  include Sidekiq::Job
+  sidekiq_options retry: 1
+  sidekiq_retry_in do |x|
+    1
+  end
+
+  def perform(idx, ts = nil)
+    puts(Time.now.to_f - ts) if !ts.nil?
+    # raise idx.to_s if idx % 100 == 1
+  end
+end
+
+def Process.rss
+  `ps -o rss= -p #{Process.pid}`.chomp.to_i
+end
+
+$iterations = ENV["ITERATIONS"] ? Integer(ENV["ITERATIONS"]) : 1_000
+$elements = ENV["ELEMENTS"] ? Integer(ENV["ELEMENTS"]) : 1_000
+$port = ENV["PORT"] ? Integer(ENV["PORT"]) : 6379
+$ip = ENV["IP"] ? String(ENV["IP"]) : "127.0.0.1"
+
+class Loader
+  def initialize
+    @iter =  $iterations
+    @count = $elements
+  end
+
+  def configure(queue)
+    @x = Sidekiq.configure_embed do |config|
+      config.redis = {db: 0, host: $ip, port: $port}
+      config.concurrency = Integer(ENV.fetch("THREADS", "30"))
+      config.queues = queue
+      config.logger.level = Logger::WARN
+      config.average_scheduled_poll_interval = 2
+      config.reliable! if defined?(Sidekiq::Pro)
+    end
+
+    @self_read, @self_write = IO.pipe
+    %w[INT TERM TSTP TTIN].each do |sig|
+      trap sig do
+        @self_write.puts(sig)
+      end
+    rescue ArgumentError
+      puts "Signal #{sig} not supported"
+    end
+  end
+
+  def handle_signal(sig)
+    launcher = @x
+    Sidekiq.logger.debug "Got #{sig} signal"
+    case sig
+    when "INT"
+      # Handle Ctrl-C in JRuby like MRI
+      # http://jira.codehaus.org/browse/JRUBY-4637
+      raise Interrupt
+    when "TERM"
+      # Heroku sends TERM and then waits 30 seconds for process to exit.
+      raise Interrupt
+    when "TSTP"
+      Sidekiq.logger.info "Received TSTP, no longer accepting new work"
+      launcher.quiet
+    when "TTIN"
+      Thread.list.each do |thread|
+        Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread["label"]}"
+        if thread.backtrace
+          Sidekiq.logger.warn thread.backtrace.join("\n")
+        else
+          Sidekiq.logger.warn "<no backtrace available>"
+        end
+      end
+    end
+  end
+
+  def setup(queue)
+    Sidekiq.logger.error("Setup RSS: #{Process.rss}")
+    Sidekiq.logger.error("Pushing work to queue: #{queue}")
+    start = Time.now
+    @iter.times do
+      arr = Array.new(@count) { |idx| [idx] }
+      # Sidekiq always prepends "queue:" to the queue name,
+      # that's why we pass 'q1', 'q2', etc instead of 'queue:q1'
+      Sidekiq::Client.push_bulk("class" => LoadWorker, "args" => arr, "queue" => queue)
+      $stdout.write "."
+    end
+    puts "Done"
+  end
+
+  def monitor_single(queue)
+    q = "queue:#{queue}"
+    @monitor_single = Thread.new do
+      GC.start
+      loop do
+        sleep 0.2
+        total = Sidekiq.redis do |conn|
+          conn.llen q
+        end
+
+        if total == 0
+          sleep 0.1
+          @x.stop
+          Process.kill("INT", $$)
+          break
+        end
+
+      end
+    end
+  end
+
+  def monitor_all(queues)
+    @monitor_all = Thread.new do
+      GC.start
+      loop do
+        sleep 0.2
+        qsize = 0
+        queues.each do |q|
+          tmp = Sidekiq.redis do |conn|
+            conn.llen q
+          end
+          qsize = qsize + tmp
+        end
+        total = qsize
+
+        if total == 0
+          ending = Time.now - @start
+          size = @iter * @count * queues.length()
+          Sidekiq.logger.error("Done, #{size} jobs in #{ending} sec, #{(size / ending).to_i} jobs/sec")
+          Sidekiq.logger.error("Ending RSS: #{Process.rss}")
+
+          sleep 0.1
+          @x.stop
+          Process.kill("INT", $$)
+          break
+        end
+      end
+    end
+  end
+
+  def run(queues, queue, monitor_all_queues)
+    Sidekiq.logger.warn("Consuming from #{queue}")
+    if monitor_all_queues
+      monitor_all(queues)
+    else
+      monitor_single(queue)
+    end
+
+    @start = Time.now
+    @x.run
+
+    while (readable_io = IO.select([@self_read]))
+      signal = readable_io.first[0].gets.strip
+      handle_signal(signal)
+    end
+    # normal
+  rescue Interrupt
+  rescue => e
+    raise e if $DEBUG
+    warn e.message
+    warn e.backtrace.join("\n")
+    exit 1
+  ensure
+    @x.stop
+  end
+end
+
+def setup(queue)
+  ll = Loader.new
+  ll.configure(queue)
+  ll.setup(queue)
+end
+
+def consume(queues, queue, monitor_all_queues)
+  ll = Loader.new
+  ll.configure(queue)
+  ll.run(queues, queue, monitor_all_queues)
+end
+
+# We assign one queue to each sidekiq process
+def run(number_of_processes, total_queues)
+  read_stream, write_stream = IO.pipe
+
+  queues = []
+  (0..total_queues-1).each do |idx|
+    queues.push("queue:q#{idx}")
+  end
+
+  Sidekiq.logger.info("Queues are: #{queues}")
+
+  # Produce
+  start = Time.now
+  (0..total_queues-1).each do |idx|
+    Process.fork do
+      queue_num = "q#{idx}"
+      setup(queue_num)
+    end
+  end
+
+  queue_sz = $iterations * $elements * total_queues
+  Process.waitall
+
+  ending = Time.now - start
+  #Sidekiq.logger.info("Pushed #{queue_sz} in #{ending} secs")
+
+  # Consume
+  (0..number_of_processes-1).each do |idx|
+    Process.fork do
+      # First process only consumes from it's own queue but monitors all queues.
+      # It works as a synchronization point. Once all processes finish
+      # (that is, when all queues are emptied) it prints the the stats.
+      if idx == 0
+        queue = "q#{idx}"
+        consume(queues, queue, true)
+      else
+        queue = "q#{idx % total_queues}"
+        consume(queues, queue, false)
+      end
+    end
+  end
+
+  Process.waitall
+  write_stream.close
+  results = read_stream.read
+  read_stream.close
+end
+
+$total_processes = ENV["PROCESSES"] ? Integer(ENV["PROCESSES"]) : 8;
+$total_queues = ENV["QUEUES"] ? Integer(ENV["QUEUES"]) : 8;
+
+run($total_processes, $total_queues)

data/bin/sidekiq

@@ -4,15 +4,34 @@
 # RUBYOPT=-w bundle exec sidekiq
 $TESTING = false
 
-require_relative '../lib/sidekiq/cli'
+require_relative "../lib/sidekiq/cli"
+
+def integrate_with_systemd
+  return unless ENV["NOTIFY_SOCKET"]
+
+  Sidekiq.configure_server do |config|
+    config.logger.info "Enabling systemd notification integration"
+    require "sidekiq/sd_notify"
+    config.on(:startup) do
+      Sidekiq::SdNotify.ready
+    end
+    config.on(:shutdown) do
+      Sidekiq::SdNotify.stopping
+    end
+    Sidekiq.start_watchdog if Sidekiq::SdNotify.watchdog?
+  end
+end
 
 begin
   cli = Sidekiq::CLI.instance
   cli.parse
+
+  integrate_with_systemd
+
   cli.run
 rescue => e
   raise e if $DEBUG
-  STDERR.puts e.message
-  STDERR.puts e.backtrace.join("\n")
+  warn e.message
+  warn e.backtrace.join("\n")
   exit 1
 end