sidekiq 5.2.3 → 6.0.0

data/Pro-5.0-Upgrade.md

@@ -0,0 +1,25 @@
+# Welcome to Sidekiq Pro 5.0!
+
+Sidekiq Pro 5.0 is mainly a cleanup release for Sidekiq 6.0. The
+migration should be as close to trivial as a major version bump can be.
+Note that Sidekiq 6.0 does have major breaking changes.
+
+## What's New
+
+* New localizations for the Sidekiq Pro Web UI: ES, ZH, PT, JA, RU
+* Removed deprecated APIs and warnings.
+* Various changes for Sidekiq 6.0
+* Requires Ruby 2.5+ and Redis 4.0+
+* Requires Sidekiq 6.0+.
+
+## Upgrade
+
+* Upgrade to the latest Sidekiq Pro 4.x.
+```ruby
+gem 'sidekiq-pro', '< 5'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 5.x.
+```ruby
+gem 'sidekiq-pro', '< 6'
+```

data/Pro-Changes.md

@@ -4,10 +4,27 @@
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
-HEAD
+5.0.0
 ---------
 
-- Add Japanese and Russian translations, see [issue #3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language
+- There is no significant migration from Sidekiq Pro 4.0 to 5.0
+  but make sure you read the [update notes for Sidekiq
+6.0](/mperham/sidekiq/blob/master/6.0-Upgrade.md).
+- Removed various deprecated APIs and associated warnings.
+- **BREAKING CHANGE** Remove the `Sidekiq::Batch::Status#dead_jobs` API in favor of
+  `Sidekiq::Batch::Status#dead_jids`. [#4217]
+- Update Sidekiq Pro codebase to use StandardRB formatting
+- Fix lingering "b-XXX-died" elements in Redis which could cause
+  excessive memory usage. [#4217]
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+4.0.5
+---------
+
+- Increase super\_fetch retriever thread count from 1 to 2 to make it
+  less sensitive to Redis latency.
+- Better handling of invalid job JSON by reliable scheduler [#4053]
+- Added ZH, PT, JA and RU translations.
 
 4.0.4
 ---------

data/README.md

@@ -3,7 +3,7 @@ Sidekiq
 
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
 [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
-[![Build Status](https://travis-ci.org/mperham/sidekiq.svg)](https://travis-ci.org/mperham/sidekiq)
+[![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
 [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
 
 
@@ -13,32 +13,26 @@ Sidekiq uses threads to handle many jobs at the same time in the
 same process.  It does not require Rails but will integrate tightly with
 Rails to make background processing dead simple.
 
-Sidekiq is compatible with Resque.  It uses the exact same
-message format as Resque so it can integrate into an existing Resque processing farm.
-You can have Sidekiq and Resque run side-by-side at the same time and
-use the Resque client to enqueue jobs in Redis to be processed by Sidekiq.
-
 Performance
 ---------------
 
-Version |	Latency | Garbage created for 10,000 jobs	| Time to process 100,000 jobs |	Throughput
------------------|------|---------|---------|------------------------
-Sidekiq 4.0.0    | 10ms	| 151 MB  | 22 sec  | **4500 jobs/sec**
-Sidekiq 3.5.1    | 22ms	| 1257 MB | 125 sec | 800 jobs/sec
-Resque 1.25.2    |  -	  | -       | 420 sec | 240 jobs/sec
-DelayedJob 4.1.1 |  -   | -       | 465 sec | 215 jobs/sec
+Version |	Latency | Garbage created for 10k jobs	| Time to process 100k jobs |	Throughput | Ruby
+-----------------|------|---------|---------|------------------------|-----
+Sidekiq 6.0.0    | 3 ms	| 156 MB  | 19 sec  | **5200 jobs/sec** | MRI 2.6.3
+Sidekiq 4.0.0    | 10 ms	| 151 MB  | 22 sec  | 4500 jobs/sec |
+Sidekiq 3.5.1    | 22 ms	| 1257 MB | 125 sec | 800 jobs/sec |
+Resque 1.25.2    |  -	  | -       | 420 sec | 240 jobs/sec |
+DelayedJob 4.1.1 |  -   | -       | 465 sec | 215 jobs/sec |
 
-<small>This benchmark can be found in `bin/sidekiqload`.</small>
+This benchmark can be found in `bin/sidekiqload` and assumes a Redis network latency of 1ms.
 
 Requirements
 -----------------
 
-Sidekiq supports CRuby 2.2.2+ and JRuby 9k.
-
-All Rails releases >= 4.0 are officially supported.
+- Redis: 4.0+
+- Ruby: MRI 2.5+ or JRuby 9.2+.
 
-Redis 2.8 or greater is required.  3.0.3+ is recommended for large
-installations with thousands of worker threads.
+Sidekiq 6.0 supports Rails 5.0+ but does not require it.
 
 
 Installation
@@ -63,7 +57,7 @@ Want to Upgrade?
 I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
 features, a commercial-friendly license and allow you to support high
 quality open source development all at the same time.  Please see the
-[Sidekiq](http://sidekiq.org/) homepage for more detail.
+[Sidekiq](https://sidekiq.org/) homepage for more detail.
 
 Subscribe to the **[quarterly newsletter](https://tinyletter.com/sidekiq)** to stay informed about the latest
 features and changes to Sidekiq and its bigger siblings.
@@ -77,24 +71,16 @@ Problems?
 If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
 Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
 
-Sidekiq Pro and Sidekiq Enterprise customers get private email support.  You can purchase at http://sidekiq.org; email support@contribsys.com for help.
+Sidekiq Pro and Sidekiq Enterprise customers get private email support.  You can purchase at https://sidekiq.org; email support@contribsys.com for help.
 
 Useful resources:
 
 * Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
-* Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
+* Occasional announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q &amp; A.
 
-**No support via Twitter**
-
 Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
-See the [Sidekiq support page](http://sidekiq.org/support.html) for details.
-
-Thanks
------------------
-
-Sidekiq stays fast by using the [JProfiler java profiler](http://www.ej-technologies.com/products/jprofiler/overview.html) to find and fix
-performance problems on JRuby.  Unfortunately MRI does not have good multithreaded profiling tools.
+See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
 
 
 License
@@ -106,4 +92,4 @@ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for
 Author
 -----------------
 
-Mike Perham, [@mperham](https://twitter.com/mperham) / [@sidekiq](https://twitter.com/sidekiq), [http://www.mikeperham.com](http://www.mikeperham.com) / [http://www.contribsys.com](http://www.contribsys.com)
+Mike Perham, [@mperham@mastodon.xyz](https://mastodon.xyz/@mperham) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)

data/Rakefile

@@ -1,8 +1,10 @@
-require 'bundler/gem_tasks'
-require 'rake/testtask'
+require "bundler/gem_tasks"
+require "rake/testtask"
+require "standard/rake"
+
 Rake::TestTask.new(:test) do |test|
   test.warning = true
-  test.pattern = 'test/**/test_*.rb'
+  test.pattern = "test/**/test_*.rb"
 end
 
-task :default => :test
+task default: [:standard, :test]

data/bin/sidekiqload

@@ -13,8 +13,8 @@ require_relative '../lib/sidekiq/launcher'
 include Sidekiq::Util
 
 Sidekiq.configure_server do |config|
-  #config.options[:concurrency] = 1
-  config.redis = { db: 13 }
+  config.options[:concurrency] = 10
+  config.redis = { db: 13, port: 6380, driver: :hiredis }
   config.options[:queues] << 'default'
   config.logger.level = Logger::ERROR
   config.average_scheduled_poll_interval = 2
@@ -28,7 +28,8 @@ class LoadWorker
     1
   end
 
-  def perform(idx)
+  def perform(idx, ts=nil)
+    puts(Time.now.to_f - ts) if ts != nil
     #raise idx.to_s if idx % 100 == 1
   end
 end
@@ -36,14 +37,15 @@ end
 # brew tap shopify/shopify
 # brew install toxiproxy
 # gem install toxiproxy
-#require 'toxiproxy'
+# run `toxiproxy-server` in a separate terminal window.
+require 'toxiproxy'
 # simulate a non-localhost network for realer-world conditions.
 # adding 1ms of network latency has an ENORMOUS impact on benchmarks
-#Toxiproxy.populate([{
-    #"name": "redis",
-    #"listen": "127.0.0.1:6380",
-    #"upstream": "127.0.0.1:6379"
-#}])
+Toxiproxy.populate([{
+    "name": "redis",
+    "listen": "127.0.0.1:6380",
+    "upstream": "127.0.0.1:6379"
+}])
 
 self_read, self_write = IO.pipe
 %w(INT TERM TSTP TTIN).each do |sig|
@@ -65,7 +67,7 @@ def handle_signal(launcher, sig)
     # http://jira.codehaus.org/browse/JRUBY-4637
     raise Interrupt
   when 'TERM'
-    # Heroku sends TERM and then waits 10 seconds for process to exit.
+    # Heroku sends TERM and then waits 30 seconds for process to exit.
     raise Interrupt
   when 'TSTP'
     Sidekiq.logger.info "Received TSTP, no longer accepting new work"
@@ -103,18 +105,20 @@ Sidekiq.logger.error "Created #{count*iter} jobs"
 Monitoring = Thread.new do
   watchdog("monitor thread") do
     while true
-      sleep 1
-      qsize, retries = Sidekiq.redis do |conn|
-        conn.pipelined do
-          conn.llen "queue:default"
-          conn.zcard "retry"
-        end
-      end.map(&:to_i)
-      total = qsize + retries
-      #GC.start
+      sleep 0.5
+      qsize = Sidekiq.redis do |conn|
+        conn.llen "queue:default"
+      end
+      total = qsize
       Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
       if total == 0
-        Sidekiq.logger.error("Done")
+        Sidekiq.logger.error("Done, now here's the latency for three jobs")
+
+        LoadWorker.perform_async(1, Time.now.to_f)
+        LoadWorker.perform_async(2, Time.now.to_f)
+        LoadWorker.perform_async(3, Time.now.to_f)
+
+        sleep 0.2
         exit(0)
       end
     end
@@ -125,8 +129,8 @@ begin
   #RubyProf::exclude_threads = [ Monitoring ]
   #RubyProf.start
   fire_event(:startup)
-  #Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
-  #Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
+  Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
+  Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
     launcher = Sidekiq::Launcher.new(Sidekiq.options)
     launcher.run
 
@@ -134,7 +138,7 @@ begin
       signal = readable_io.first[0].gets.strip
       handle_signal(launcher, signal)
     end
-  #end
+  end
 rescue SystemExit => e
   #Sidekiq.logger.error("Profiling...")
   #result = RubyProf.stop

data/bin/sidekiqmon

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+require 'sidekiq/monitor'
+
+if ARGV[0] == 'status'
+  Sidekiq::Monitor::Status.new.display(ARGV[1])
+else
+  Sidekiq::Monitor.print_usage
+end

data/lib/generators/sidekiq/templates/worker_test.rb.erb

@@ -1,6 +1,6 @@
 require 'test_helper'
 <% module_namespacing do -%>
-class <%= class_name %>WorkerTest < <% if defined? Minitest::Test %>Minitest::Test<% else %>MiniTest::Unit::TestCase<% end %>
+class <%= class_name %>WorkerTest < Minitest::Test
   def test_example
     skip "add some examples to (or delete) #{__FILE__}"
   end

data/lib/generators/sidekiq/worker_generator.rb

@@ -1,18 +1,18 @@
-require 'rails/generators/named_base'
+require "rails/generators/named_base"
 
 module Sidekiq
   module Generators # :nodoc:
     class WorkerGenerator < ::Rails::Generators::NamedBase # :nodoc:
-      desc 'This generator creates a Sidekiq Worker in app/workers and a corresponding test'
+      desc "This generator creates a Sidekiq Worker in app/workers and a corresponding test"
 
-      check_class_collision suffix: 'Worker'
+      check_class_collision suffix: "Worker"
 
       def self.default_generator_root
         File.dirname(__FILE__)
       end
 
       def create_worker_file
-        template 'worker.rb.erb', File.join('app/workers', class_path, "#{file_name}_worker.rb")
+        template "worker.rb.erb", File.join("app/workers", class_path, "#{file_name}_worker.rb")
       end
 
       def create_test_file
@@ -27,23 +27,21 @@ module Sidekiq
 
       def create_worker_spec
         template_file = File.join(
-            'spec/workers',
-            class_path,
-            "#{file_name}_worker_spec.rb"
+          "spec/workers",
+          class_path,
+          "#{file_name}_worker_spec.rb"
         )
-        template 'worker_spec.rb.erb', template_file
+        template "worker_spec.rb.erb", template_file
       end
 
       def create_worker_test
         template_file = File.join(
-            'test/workers',
-            class_path,
-            "#{file_name}_worker_test.rb"
+          "test/workers",
+          class_path,
+          "#{file_name}_worker_test.rb"
         )
-        template 'worker_test.rb.erb', template_file
+        template "worker_test.rb.erb", template_file
       end
-
-
     end
   end
 end

data/lib/sidekiq.rb

@@ -1,26 +1,27 @@
 # frozen_string_literal: true
-require 'sidekiq/version'
-fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.2.2." if RUBY_PLATFORM != 'java' && Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.2.2')
 
-require 'sidekiq/logging'
-require 'sidekiq/client'
-require 'sidekiq/worker'
-require 'sidekiq/redis_connection'
-require 'sidekiq/delay'
+require "sidekiq/version"
+fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.5.0." if RUBY_PLATFORM != "java" && Gem::Version.new(RUBY_VERSION) < Gem::Version.new("2.5.0")
 
-require 'json'
+require "sidekiq/logger"
+require "sidekiq/client"
+require "sidekiq/worker"
+require "sidekiq/redis_connection"
+require "sidekiq/delay"
+
+require "json"
 
 module Sidekiq
-  NAME = 'Sidekiq'
-  LICENSE = 'See LICENSE and the LGPL-3.0 for licensing details.'
+  NAME = "Sidekiq"
+  LICENSE = "See LICENSE and the LGPL-3.0 for licensing details."
 
   DEFAULTS = {
     queues: [],
     labels: [],
     concurrency: 10,
-    require: '.',
+    require: ".",
     environment: nil,
-    timeout: 8,
+    timeout: 25,
     poll_interval_average: nil,
     average_scheduled_poll_interval: 5,
     error_handlers: [],
@@ -37,8 +38,8 @@ module Sidekiq
   }
 
   DEFAULT_WORKER_OPTIONS = {
-    'retry' => true,
-    'queue' => 'default'
+    "retry" => true,
+    "queue" => "default",
   }
 
   FAKE_INFO = {
@@ -46,7 +47,7 @@ module Sidekiq
     "uptime_in_days" => "9999",
     "connected_clients" => "9999",
     "used_memory_human" => "9P",
-    "used_memory_peak_human" => "9P"
+    "used_memory_peak_human" => "9P",
   }
 
   def self.❨╯°□°❩╯︵┻━┻
@@ -56,6 +57,7 @@ module Sidekiq
   def self.options
     @options ||= DEFAULTS.dup
   end
+
   def self.options=(opts)
     @options = opts
   end
@@ -94,9 +96,13 @@ module Sidekiq
       begin
         yield conn
       rescue Redis::CommandError => ex
-        #2550 Failover can cause the server to become a slave, need
-        # to disconnect and reopen the socket to get back to the master.
-        (conn.disconnect!; retryable = false; retry) if retryable && ex.message =~ /READONLY/
+        # 2550 Failover can cause the server to become a replica, need
+        # to disconnect and reopen the socket to get back to the primary.
+        if retryable && ex.message =~ /READONLY/
+          conn.disconnect!
+          retryable = false
+          retry
+        end
         raise
       end
     end
@@ -104,19 +110,17 @@ module Sidekiq
 
   def self.redis_info
     redis do |conn|
-      begin
-        # admin commands can't go through redis-namespace starting
-        # in redis-namespace 2.0
-        if conn.respond_to?(:namespace)
-          conn.redis.info
-        else
-          conn.info
-        end
-      rescue Redis::CommandError => ex
-        #2850 return fake version when INFO command has (probably) been renamed
-        raise unless ex.message =~ /unknown command/
-        FAKE_INFO
+      # admin commands can't go through redis-namespace starting
+      # in redis-namespace 2.0
+      if conn.respond_to?(:namespace)
+        conn.redis.info
+      else
+        conn.info
       end
+    rescue Redis::CommandError => ex
+      # 2850 return fake version when INFO command has (probably) been renamed
+      raise unless /unknown command/.match?(ex.message)
+      FAKE_INFO
     end
   end
 
@@ -150,18 +154,13 @@ module Sidekiq
 
   def self.default_worker_options=(hash)
     # stringify
-    @default_worker_options = default_worker_options.merge(Hash[hash.map{|k, v| [k.to_s, v]}])
+    @default_worker_options = default_worker_options.merge(Hash[hash.map { |k, v| [k.to_s, v] }])
   end
+
   def self.default_worker_options
     defined?(@default_worker_options) ? @default_worker_options : DEFAULT_WORKER_OPTIONS
   end
 
-  def self.default_retries_exhausted=(prok)
-    logger.info { "default_retries_exhausted is deprecated, please use `config.death_handlers << -> {|job, ex| }`" }
-    return nil unless prok
-    death_handlers << prok
-  end
-
   ##
   # Death handlers are called when all retries for a job have been exhausted and
   # the job dies.  It's the notification to your application
@@ -178,15 +177,29 @@ module Sidekiq
   def self.load_json(string)
     JSON.parse(string)
   end
+
   def self.dump_json(object)
     JSON.generate(object)
   end
 
+  def self.log_formatter
+    @log_formatter ||= if ENV["DYNO"]
+      Sidekiq::Logger::Formatters::WithoutTimestamp.new
+    else
+      Sidekiq::Logger::Formatters::Pretty.new
+    end
+  end
+
+  def self.log_formatter=(log_formatter)
+    @log_formatter = log_formatter
+  end
+
   def self.logger
-    Sidekiq::Logging.logger
+    @logger ||= Sidekiq::Logger.new(STDOUT, level: Logger::INFO)
   end
-  def self.logger=(log)
-    Sidekiq::Logging.logger = log
+
+  def self.logger=(logger)
+    @logger = logger
   end
 
   # How frequently Redis should be checked by a random Sidekiq process for
@@ -195,7 +208,7 @@ module Sidekiq
   #
   # See sidekiq/scheduled.rb for an in-depth explanation of this value
   def self.average_scheduled_poll_interval=(interval)
-    self.options[:average_scheduled_poll_interval] = interval
+    options[:average_scheduled_poll_interval] = interval
   end
 
   # Register a proc to handle any error which occurs within the Sidekiq process.
@@ -206,7 +219,7 @@ module Sidekiq
   #
   # The default error handler logs errors to Sidekiq.logger.
   def self.error_handlers
-    self.options[:error_handlers]
+    options[:error_handlers]
   end
 
   # Register a block to run at a point in the Sidekiq lifecycle.
@@ -232,4 +245,4 @@ module Sidekiq
   class Shutdown < Interrupt; end
 end
 
-require 'sidekiq/rails' if defined?(::Rails::Engine)
+require "sidekiq/rails" if defined?(::Rails::Engine)