sidekiq 0.10.0 → 7.2.0

data/LICENSE.txt

@@ -0,0 +1,9 @@
+Copyright (c) Contributed Systems LLC
+
+Sidekiq is an Open Source project licensed under the terms of
+the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
+for license text.
+
+Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
+You can find the commercial license in COMM-LICENSE.txt.
+Please see https://sidekiq.org for purchasing options.

data/README.md

@@ -1,60 +1,106 @@
 Sidekiq
 ==============
 
-Simple, efficient message processing for Ruby.
+[![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
+![Build](https://github.com/sidekiq/sidekiq/workflows/CI/badge.svg)
 
-Sidekiq is compatible with Resque.  It uses the exact same
-message format as Resque so it can integrate into an existing Resque processing farm.
-You can have Sidekiq and Resque run side-by-side at the same time and
-use the Resque client to enqueue messages in Redis to be processed by Sidekiq.
+Simple, efficient background processing for Ruby.
 
-At the same time, Sidekiq uses multithreading so it much more memory efficient than Resque (which forks a new process for every job).
-You'll find that you might need 50 200MB resque processes to peg your CPU
-whereas one 300MB Sidekiq process will peg the same CPU and perform the
-same amount of work.  Please see [my blog post on Resque's memory
-efficiency](http://blog.carbonfive.com/2011/09/16/improving-resques-memory-efficiency/)
- and how I was able to shrink a Carbon Five client's resque processing farm
-from 9 machines to 1 machine.
-
-In sum, if your jobs are well-behaved and threadsafe, Sidekiq is probably a good replacement for Resque. If your jobs are not thread-safe or they leak memory, you may want to continue using Resque, because its forking model gives you more protection.
+Sidekiq uses threads to handle many jobs at the same time in the
+same process.  It does not require Rails but will integrate tightly with
+Rails to make background processing dead simple.
 
 
 Requirements
 -----------------
 
-I test on Ruby 1.9.3 and JRuby 1.6.5 in 1.9 mode.  Other versions/VMs are
-untested but I will do my best to support them.  Ruby 1.8 is not supported.
+- Redis: 6.2+
+- Ruby: MRI 2.7+ or JRuby 9.3+.
+
+Sidekiq 7.0 supports Rails 6.0+ but does not require it.
 
 
 Installation
 -----------------
 
-   gem install sidekiq
+    bundle add sidekiq
 
 
 Getting Started
 -----------------
 
-See the [sidekiq home page](http://mperham.github.com/sidekiq) for the simple 4-step process.
+See the [Getting Started wiki page](https://github.com/sidekiq/sidekiq/wiki/Getting-Started) and follow the simple setup process.
+You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
+Sidekiq and see its features in action.  Here's the Web UI:
+
+![Web UI](https://github.com/sidekiq/sidekiq/raw/main/examples/web-ui.png)
+
+Performance
+---------------
+
+The benchmark in `bin/sidekiqload` creates 500,000 no-op jobs and drains them as fast as possible, assuming a fixed Redis network latency of 1ms.
+This requires a lot of Redis network I/O and JSON parsing.
+This benchmark is IO-bound so we increase the concurrency to 25.
+If your application is sending lots of emails or performing other network-intensive work, you could see a similar benefit but be careful not to saturate the CPU.
+
+Version | Time to process 500k jobs | Throughput (jobs/sec) | Ruby | Concurrency | Job Type
+-----------------|------|---------|---------|------------------------|---
+Sidekiq 7.0.3 | 21.3 sec| 23,500 | 3.2.0+yjit | 30 | Sidekiq::Job
+Sidekiq 7.0.3 | 33.8 sec| 14,700 | 3.2.0+yjit | 30 | ActiveJob 7.0.4
+Sidekiq 7.0.3 | 23.5 sec| 21,300 | 3.2.0 | 30 | Sidekiq::Job
+Sidekiq 7.0.3 | 46.5 sec| 10,700 | 3.2.0 | 30 | ActiveJob 7.0.4
+Sidekiq 7.0.3 | 23.0 sec| 21,700 | 2.7.5 | 30 | Sidekiq::Job
+Sidekiq 7.0.3 | 46.5 sec| 10,850 | 2.7.5 | 30 | ActiveJob 7.0.4
+
+Most of Sidekiq's overhead is Redis network I/O.
+ActiveJob adds a notable amount of CPU overhead due to argument deserialization and callbacks.
+Concurrency of 30 was determined experimentally to maximize one CPU without saturating it.
+
+Want to Upgrade?
+-------------------
+
+Use `bundle up sidekiq` to upgrade Sidekiq and all its dependencies.
+Upgrade notes between each major version can be found in the `docs/` directory.
+
+I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
+features, a commercial-friendly license and allow you to support high
+quality open source development all at the same time.  Please see the
+[Sidekiq](https://sidekiq.org/) homepage for more detail.
 
 
-More Information
+Problems?
 -----------------
 
-Please see the [sidekiq wiki](https://github.com/mperham/sidekiq/wiki) for more information.
-[#sidekiq on irc.freenode.net](irc://irc.freenode.net/#sidekiq) is dedicated to this project,
-but bug reports or feature requests suggestions should still go through [issues on Github](https://github.com/mperham/sidekiq/issues).
+**Please do not directly email any Sidekiq committers with questions or problems.**
+A community is best served when discussions are held in public.
 
+If you have a problem, please review the [FAQ](https://github.com/sidekiq/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/sidekiq/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
+Searching the [issues](https://github.com/sidekiq/sidekiq/issues) for your problem is also a good idea.
 
-License
+Sidekiq Pro and Sidekiq Enterprise customers get private email support.
+You can purchase at https://sidekiq.org; email support@contribsys.com for help.
+
+Useful resources:
+
+* Product documentation is in the [wiki](https://github.com/sidekiq/sidekiq/wiki).
+* Occasional announcements are made to the [@sidekiq](https://ruby.social/@sidekiq) Mastodon account.
+* The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
+
+Every Friday morning is Sidekiq office hour: I video chat and answer questions.
+See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
+
+Contributing
 -----------------
 
-Please see LICENSE for licensing details.
+Please see [the contributing guidelines](https://github.com/sidekiq/sidekiq/blob/main/.github/contributing.md).
 
-<a href='http://www.pledgie.com/campaigns/16623'><img alt='Click here to lend your support to Open Source and make a donation at www.pledgie.com !' src='http://www.pledgie.com/campaigns/16623.png?skin_name=chrome' border='0' /></a>
+License
+-----------------
+
+Please see [LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/LICENSE.txt) for licensing details.
+The license for Sidekiq Pro and Sidekiq Enterprise can be found in [COMM-LICENSE.txt](https://github.com/sidekiq/sidekiq/blob/main/COMM-LICENSE.txt).
 
 Author
 -----------------
 
-Mike Perham, [@mperham](https://twitter.com/mperham), [http://mikeperham.com](http://mikeperham.com)
-
+Mike Perham, [@getajobmike](https://ruby.social/@getajobmike) / [@sidekiq](https://ruby.social/@sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)

data/bin/sidekiq

@@ -1,21 +1,37 @@
 #!/usr/bin/env ruby
 
-begin
-  require 'sidekiq/cli'
-rescue LoadError
-  # Better way to do this in dev?
-  dir = File.expand_path(File.dirname(__FILE__) + '/../lib') 
-  $LOAD_PATH << dir if File.exist?("#{dir}/sidekiq.rb")
-  require 'sidekiq/cli'
+# Quiet some warnings we see when running in warning mode:
+# RUBYOPT=-w bundle exec sidekiq
+$TESTING = false
+
+require_relative "../lib/sidekiq/cli"
+
+def integrate_with_systemd
+  return unless ENV["NOTIFY_SOCKET"]
+
+  Sidekiq.configure_server do |config|
+    config.logger.info "Enabling systemd notification integration"
+    require "sidekiq/sd_notify"
+    config.on(:startup) do
+      Sidekiq::SdNotify.ready
+    end
+    config.on(:shutdown) do
+      Sidekiq::SdNotify.stopping
+    end
+    Sidekiq.start_watchdog if Sidekiq::SdNotify.watchdog?
+  end
 end
 
 begin
   cli = Sidekiq::CLI.instance
   cli.parse
+
+  integrate_with_systemd
+
   cli.run
 rescue => e
   raise e if $DEBUG
-  STDERR.puts e.message
-  STDERR.puts e.backtrace.join("\n")
+  warn e.message
+  warn e.backtrace.join("\n")
   exit 1
 end

data/bin/sidekiqload

@@ -0,0 +1,247 @@
+#!/usr/bin/env ruby
+
+#
+# bin/sidekiqload is a helpful script to load test and
+# performance tune Sidekiq's core. It creates 500,000 no-op
+# jobs and executes them as fast as possible.
+# Example Usage:
+#
+# > RUBY_YJIT_ENABLE=1 LATENCY=0 THREADS=10 bin/sidekiqload
+# Result: Done, 500000 jobs in 20.264945 sec, 24673 jobs/sec
+#
+# Use LATENCY=1 to get a more real world network setup
+# but you'll need to setup and start toxiproxy as noted below.
+#
+# Use AJ=1 to test ActiveJob instead of plain old Sidekiq::Jobs so
+# you can see the runtime performance difference between the two APIs.
+#
+# None of this script is considered a public API and may change over time.
+#
+
+# Quiet some warnings we see when running in warning mode:
+# RUBYOPT=-w bundle exec sidekiq
+$TESTING = false
+puts RUBY_DESCRIPTION
+puts(%w[THREADS LATENCY AJ PROFILE].map { |x| "#{x}: #{ENV[x] || "nil"}" }.join(", "))
+
+require "ruby-prof" if ENV["PROFILE"]
+require "bundler/setup"
+Bundler.require(:default, :load_test)
+
+latency = Integer(ENV["LATENCY"] || 1)
+if latency > 0
+  # brew tap shopify/shopify
+  # brew install toxiproxy
+  # run `toxiproxy-server` in a separate terminal window.
+  require "toxiproxy"
+  # simulate a non-localhost network for realer-world conditions.
+  # adding 1ms of network latency has an ENORMOUS impact on benchmarks
+  Toxiproxy.populate([{
+    name: "redis",
+    listen: "127.0.0.1:6380",
+    upstream: "127.0.0.1:6379"
+  }])
+end
+
+if ENV["AJ"]
+  require "active_job"
+  puts "Using ActiveJob #{ActiveJob::VERSION::STRING}"
+  ActiveJob::Base.queue_adapter = :sidekiq
+  ActiveJob::Base.logger.level = Logger::WARN
+
+  class LoadJob < ActiveJob::Base
+    def perform(idx, ts = nil)
+      puts(Time.now.to_f - ts) if !ts.nil?
+    end
+  end
+end
+
+class LoadWorker
+  include Sidekiq::Job
+  sidekiq_options retry: 1
+  sidekiq_retry_in do |x|
+    1
+  end
+
+  def perform(idx, ts = nil)
+    puts(Time.now.to_f - ts) if !ts.nil?
+    # raise idx.to_s if idx % 100 == 1
+  end
+end
+
+def Process.rss
+  `ps -o rss= -p #{Process.pid}`.chomp.to_i
+end
+
+class Loader
+  def initialize
+    @iter = ENV["GC"] ? 10 : 500
+    @count = Integer(ENV["COUNT"] || 1_000)
+    @latency = Integer(ENV["LATENCY"] || 1)
+  end
+
+  def configure
+    @x = Sidekiq.configure_embed do |config|
+      config.redis = {db: 13, port: ((@latency > 0) ? 6380 : 6379)}
+      config.concurrency = Integer(ENV.fetch("THREADS", "10"))
+      # config.redis = { db: 13, port: 6380, driver: :hiredis}
+      config.queues = %w[default]
+      config.logger.level = Logger::WARN
+      config.average_scheduled_poll_interval = 2
+      config.reliable! if defined?(Sidekiq::Pro)
+    end
+
+    @self_read, @self_write = IO.pipe
+    %w[INT TERM TSTP TTIN].each do |sig|
+      trap sig do
+        @self_write.puts(sig)
+      end
+    rescue ArgumentError
+      puts "Signal #{sig} not supported"
+    end
+  end
+
+  def handle_signal(sig)
+    launcher = @x
+    Sidekiq.logger.debug "Got #{sig} signal"
+    case sig
+    when "INT"
+      # Handle Ctrl-C in JRuby like MRI
+      # http://jira.codehaus.org/browse/JRUBY-4637
+      raise Interrupt
+    when "TERM"
+      # Heroku sends TERM and then waits 30 seconds for process to exit.
+      raise Interrupt
+    when "TSTP"
+      Sidekiq.logger.info "Received TSTP, no longer accepting new work"
+      launcher.quiet
+    when "TTIN"
+      Thread.list.each do |thread|
+        Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread["label"]}"
+        if thread.backtrace
+          Sidekiq.logger.warn thread.backtrace.join("\n")
+        else
+          Sidekiq.logger.warn "<no backtrace available>"
+        end
+      end
+    end
+  end
+
+  def setup
+    Sidekiq.logger.error("Setup RSS: #{Process.rss}")
+    Sidekiq.redis { |c| c.flushdb }
+    start = Time.now
+    if ENV["AJ"]
+      @iter.times do
+        @count.times do |idx|
+          LoadJob.perform_later(idx)
+        end
+      end
+    else
+      @iter.times do
+        arr = Array.new(@count) { |idx| [idx] }
+        Sidekiq::Client.push_bulk("class" => LoadWorker, "args" => arr)
+      end
+    end
+    Sidekiq.logger.warn "Created #{@count * @iter} jobs in #{Time.now - start} sec"
+  end
+
+  def monitor
+    @monitor = Thread.new do
+      GC.start
+      loop do
+        sleep 0.2
+        qsize = Sidekiq.redis do |conn|
+          conn.llen "queue:default"
+        end
+        total = qsize
+        if total == 0
+          ending = Time.now - @start
+          size = @iter * @count
+          Sidekiq.logger.error("Done, #{size} jobs in #{ending} sec, #{(size / ending).to_i} jobs/sec")
+          Sidekiq.logger.error("Ending RSS: #{Process.rss}")
+          Sidekiq.logger.error("Now here's the latency for three jobs")
+
+          if ENV["AJ"]
+            LoadJob.perform_later(1, Time.now.to_f)
+            LoadJob.perform_later(2, Time.now.to_f)
+            LoadJob.perform_later(3, Time.now.to_f)
+          else
+            LoadWorker.perform_async(1, Time.now.to_f)
+            LoadWorker.perform_async(2, Time.now.to_f)
+            LoadWorker.perform_async(3, Time.now.to_f)
+          end
+
+          sleep 0.1
+          @x.stop
+          Process.kill("INT", $$)
+          break
+        end
+      end
+    end
+  end
+
+  def with_latency(latency, &block)
+    Sidekiq.logger.error "Simulating #{latency}ms of latency between Sidekiq and redis"
+    if latency > 0
+      Toxiproxy[:redis].downstream(:latency, latency: latency).apply(&block)
+    else
+      yield
+    end
+  end
+
+  def run(name)
+    Sidekiq.logger.warn("Starting #{name}")
+    monitor
+
+    if ENV["PROFILE"]
+      RubyProf.exclude_threads = [@monitor]
+      RubyProf.start
+    elsif ENV["GC"]
+      GC.start
+      GC.compact
+      GC.disable
+      Sidekiq.logger.error("GC Start RSS: #{Process.rss}")
+    end
+    @start = Time.now
+    with_latency(@latency) do
+      @x.run
+
+      while (readable_io = IO.select([@self_read]))
+        signal = readable_io.first[0].gets.strip
+        handle_signal(signal)
+      end
+    end
+    # normal
+  rescue Interrupt
+  rescue => e
+    raise e if $DEBUG
+    warn e.message
+    warn e.backtrace.join("\n")
+    exit 1
+  ensure
+    @x.stop
+  end
+
+  def done
+    Sidekiq.logger.error("GC End RSS: #{Process.rss}") if ENV["GC"]
+    if ENV["PROFILE"]
+      Sidekiq.logger.error("Profiling...")
+      result = RubyProf.stop
+      printer = RubyProf::GraphHtmlPrinter.new(result)
+      printer.print(File.new("output.html", "w"), min_percent: 1)
+    end
+  end
+end
+
+ll = Loader.new
+ll.configure
+
+if ENV["WARM"]
+  ll.setup
+  ll.run("warmup")
+end
+
+ll.setup
+ll.run("load")
+ll.done

data/bin/sidekiqmon

@@ -0,0 +1,11 @@
+#!/usr/bin/env ruby
+
+require "sidekiq/monitor"
+
+# disable the Redis connection pool logging
+Sidekiq.default_configuration.logger.level = :warn
+
+section = "all"
+section = ARGV[0] if ARGV.size == 1
+
+Sidekiq::Monitor::Status.new.display(section)

data/lib/generators/sidekiq/job_generator.rb

@@ -0,0 +1,57 @@
+require "rails/generators/named_base"
+
+module Sidekiq
+  module Generators # :nodoc:
+    class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
+      desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
+
+      check_class_collision suffix: "Job"
+
+      def self.default_generator_root
+        File.dirname(__FILE__)
+      end
+
+      def create_job_file
+        template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
+      end
+
+      def create_test_file
+        return unless test_framework
+
+        if test_framework == :rspec
+          create_job_spec
+        else
+          create_job_test
+        end
+      end
+
+      private
+
+      def create_job_spec
+        template_file = File.join(
+          "spec/sidekiq",
+          class_path,
+          "#{file_name}_job_spec.rb"
+        )
+        template "job_spec.rb.erb", template_file
+      end
+
+      def create_job_test
+        template_file = File.join(
+          "test/sidekiq",
+          class_path,
+          "#{file_name}_job_test.rb"
+        )
+        template "job_test.rb.erb", template_file
+      end
+
+      def file_name
+        @_file_name ||= super.sub(/_?job\z/i, "")
+      end
+
+      def test_framework
+        ::Rails.application.config.generators.options[:rails][:test_framework]
+      end
+    end
+  end
+end

data/lib/generators/sidekiq/templates/job.rb.erb

@@ -0,0 +1,9 @@
+<% module_namespacing do -%>
+class <%= class_name %>Job
+  include Sidekiq::Job
+
+  def perform(*args)
+    # Do something
+  end
+end
+<% end -%>

data/lib/generators/sidekiq/templates/job_spec.rb.erb

@@ -0,0 +1,6 @@
+require 'rails_helper'
+<% module_namespacing do -%>
+RSpec.describe <%= class_name %>Job, type: :job do
+  pending "add some examples to (or delete) #{__FILE__}"
+end
+<% end -%>

data/lib/generators/sidekiq/templates/job_test.rb.erb

@@ -0,0 +1,8 @@
+require 'test_helper'
+<% module_namespacing do -%>
+class <%= class_name %>JobTest < Minitest::Test
+  def test_example
+    skip "add some examples to (or delete) #{__FILE__}"
+  end
+end
+<% end -%>