sidekiq 4.2.10 → 5.2.7

data/Gemfile

@@ -1,29 +1,23 @@
 source 'https://rubygems.org'
+
 gemspec
 
-gem 'rails', '>= 5.0.1'
-gem "hiredis"
-gem 'simplecov'
-gem 'minitest'
-#gem 'minitest-utils'
-gem 'toxiproxy'
+gem 'rake'
+gem 'redis-namespace'
+gem 'rails', '~> 5.2'
+gem 'sqlite3', '~> 1.3.6', platforms: :ruby
+gem 'activerecord-jdbcsqlite3-adapter', platforms: :jruby
 
-platforms :rbx do
-  gem 'rubysl', '~> 2.0'         # if using anything in the ruby standard library
-  gem 'psych'                    # if using yaml
-  gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
+group :test do
+  gem 'minitest'
+  gem 'simplecov'
 end
 
-platforms :ruby do
-  gem 'sqlite3'
+group :development, :test do
+  gem 'pry-byebug', platforms: :mri
 end
 
-platforms :mri do
-  gem 'pry-byebug'
-  gem 'ruby-prof'
+group :load_test do
+  gem 'hiredis'
+  gem 'toxiproxy'
 end
-
-#platforms :jruby do
-  #gem 'jruby-openssl'
-  #gem 'activerecord-jdbcsqlite3-adapter'
-#end

data/LICENSE

@@ -5,5 +5,5 @@ the LGPLv3 license.  Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
 for license text.
 
 Sidekiq Pro has a commercial-friendly license allowing private forks
-and modifications of Sidekiq.  Please see http://sidekiq.org/pro/ for
+and modifications of Sidekiq.  Please see https://sidekiq.org/products/pro.html for
 more detail.  You can find the commercial license terms in COMM-LICENSE.

data/Pro-4.0-Upgrade.md

@@ -0,0 +1,35 @@
+# Welcome to Sidekiq Pro 4.0!
+
+Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
+
+## What's New
+
+* Batches now "die" if any of their jobs die.  You can enumerate the set
+  of dead batches and their associated dead jobs.  The success callback
+  for a dead batch will never fire unless these jobs are fixed.
+```ruby
+Sidekiq::Batch::DeadSet.new.each do |status|
+  status.dead? # => true
+  status.dead_jobs # => [...]
+end
+```
+This API allows you to enumerate the batches which need help.
+If you fix the issue and the dead jobs succeed, the batch will succeed.
+* The older `reliable_fetch` and `timed_fetch` algorithms have been
+  removed.  Only super\_fetch is available in 4.0.
+* The statsd middleware has been tweaked to remove support for legacy,
+  pre-3.6.0 configuration and add relevant tags.
+* Requires Sidekiq 5.0.5+.
+
+## Upgrade
+
+* Upgrade to the latest Sidekiq Pro 3.x.
+```ruby
+gem 'sidekiq-pro', '< 4'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 4.x.
+```ruby
+gem 'sidekiq-pro', '< 5'
+```
+

data/Pro-Changes.md

@@ -1,11 +1,142 @@
-Sidekiq Pro Changelog
-=======================
+# Sidekiq Pro Changelog
+
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
 HEAD
 ---------
 
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+4.0.5
+---------
+
+- Increase super\_fetch retriever thread count from 1 to 2 to make it
+  less sensitive to Redis latency.
+- Better handling of invalid job JSON by reliable scheduler [#4053]
+- Added ZH, PT, JA and RU translations.
+
+4.0.4
+---------
+
+- Update Sidekiq::Client patches to work with new Module#prepend
+  mechanism in Sidekiq 5.2.0. [#3930]
+
+4.0.3
+---------
+
+- Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
+- Implement batch death callback.  This is fired the first time a job within a batch dies. [#3841]
+```ruby
+batch = Sidekiq::Batch.new
+batch.on(:death, ...)
+```
+
+4.0.2
+---------
+
+- Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
+  call and resulting latency [#3790]
+- Fix possible bad statsd metric call on super\_fetch startup
+- Remove superfluous `freeze` calls on Strings [#3759]
+
+4.0.1
+---------
+
+- Fix incompatibility with the statsd-ruby gem [#3740]
+- Add tags to Statsd metrics when using Datadog [#3744]
+
+4.0.0
+---------
+
+- See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
+
+
+3.7.1
+---------
+
+- Deprecate timed\_fetch.  Switch to super\_fetch:
+```ruby
+config.super_fetch!
+```
+
+
+3.7.0
+---------
+
+- Refactor batch job success/failure to gracefully handle several edge
+  cases with regard to Sidekiq::Shutdown.  This should greatly reduce
+  the chances of seeing the long-standing "negative pending count" problem. [#3710]
+
+
+3.6.1
+---------
+
+- Add support for Datadog::Statsd, it is the recommended Statsd client.  [#3699]
+```ruby
+Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
+```
+- Size the statsd connection pool based on Sidekiq's concurrency [#3700]
+
+
+3.6.0
+---------
+
+This release overhauls the Statsd metrics support and adds more
+metrics for tracking Pro feature usage.  In your initializer:
+```ruby
+Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
+```
+Sidekiq Pro will emit more metrics to Statsd:
+```
+jobs.expired - when a job is expired
+jobs.recovered.push - when a job is recovered by reliable_push after network outage
+jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
+batch.created - when a batch is created
+batch.complete - when a batch is completed
+batch.success - when a batch is successful
+```
+Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
+Everything should be backwards compatible with one deprecation notice.
+
+
+3.5.4
+---------
+
+- Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
+- Fix case where TimedFetch might not recover some pending jobs
+- Fix edge case in Batch::Status#poll leading to premature completion [#3640]
+- Adjust scan API to check 100 elements at a time, to minimize network round trips
+  when scanning large sets.
+
+3.5.3
+---------
+
+- Restore error check for super\_fetch's job ack [#3601]
+- Trim error messages saved in Batch's failure hash, preventing huge
+  messages from bloating Redis. [#3570]
+
+3.5.2
+---------
+
+- Fix `Status#completed?` when run against a Batch that had succeeded
+  and was deleted. [#3519]
+
+3.5.1
+---------
+
+- Work with Sidekiq 5.0.2+
+- Improve performance of super\_fetch with weighted queues [#3489]
+
+3.5.0
+---------
+
+- Add queue pause/unpause endpoints for scripting via curl [#3445]
+- Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
+- Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
+- Batch page displays Pending JIDs if less than 10 [#3130]
+- Batch page has a Search button to find associated Retries [#3130]
 - Make Batch UI progress bar more friendly to the colorblind [#3387]
 
 3.4.5

data/README.md

@@ -3,7 +3,7 @@ Sidekiq
 
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
 [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
-[![Build Status](https://travis-ci.org/mperham/sidekiq.svg)](https://travis-ci.org/mperham/sidekiq)
+[![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
 [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
 
 
@@ -28,13 +28,14 @@ Sidekiq 3.5.1    | 22ms	| 1257 MB | 125 sec | 800 jobs/sec
 Resque 1.25.2    |  -	  | -       | 420 sec | 240 jobs/sec
 DelayedJob 4.1.1 |  -   | -       | 465 sec | 215 jobs/sec
 
+<small>This benchmark can be found in `bin/sidekiqload`.</small>
 
 Requirements
 -----------------
 
-Sidekiq supports CRuby 2.0+ and JRuby 9k.
+Sidekiq supports CRuby 2.2.2+ and JRuby 9k.
 
-All Rails releases >= 3.2 are officially supported.
+All Rails releases >= 4.0 are officially supported.
 
 Redis 2.8 or greater is required.  3.0.3+ is recommended for large
 installations with thousands of worker threads.
@@ -76,17 +77,18 @@ Problems?
 If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
 Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
 
+Sidekiq Pro and Sidekiq Enterprise customers get private email support.  You can purchase at http://sidekiq.org; email support@contribsys.com for help.
+
 Useful resources:
 
 * Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
 * Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
-* Here's a [Reddit forum](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q &amp; A.
 
-**No support via Twitter, 140 characters is not enough.**
+**No support via Twitter**
 
 Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
-See the [Sidekiq support page](http://sidekiq.org/support) for details.
+See the [Sidekiq support page](http://sidekiq.org/support.html) for details.
 
 Thanks
 -----------------

data/Rakefile

@@ -1,12 +1,9 @@
 require 'bundler/gem_tasks'
 require 'rake/testtask'
+
 Rake::TestTask.new(:test) do |test|
   test.warning = true
   test.pattern = 'test/**/test_*.rb'
 end
 
-task :default => :test
-
-task :appraise do
-  exec("cd myapp && rake appraise")
-end
+task default: :test

data/bin/sidekiqctl

@@ -1,99 +1,20 @@
 #!/usr/bin/env ruby
 
 require 'fileutils'
+require 'sidekiq/api'
+require 'sidekiq/ctl'
 
-class Sidekiqctl
-  DEFAULT_KILL_TIMEOUT = 10
-
-  attr_reader :stage, :pidfile, :kill_timeout
-
-  def self.print_usage
-    puts "#{File.basename($0)} - stop a Sidekiq process from the command line."
-    puts
-    puts "Usage: #{File.basename($0)} <command> <pidfile> <kill_timeout>"
-    puts " where <command> is either 'quiet' or 'stop'"
-    puts "       <pidfile> is path to a pidfile"
-    puts "       <kill_timeout> is number of seconds to wait until Sidekiq exits"
-    puts "       (default: #{Sidekiqctl::DEFAULT_KILL_TIMEOUT}), after which Sidekiq will be KILL'd"
-    puts
-    puts "Be sure to set the kill_timeout LONGER than Sidekiq's -t timeout.  If you want"
-    puts "to wait 60 seconds for jobs to finish, use `sidekiq -t 60` and `sidekiqctl stop"
-    puts " path_to_pidfile 61`"
-    puts
-  end
-
-  def initialize(stage, pidfile, timeout)
-    @stage = stage
-    @pidfile = pidfile
-    @kill_timeout = timeout
-
-    done('No pidfile given', :error) if !pidfile
-    done("Pidfile #{pidfile} does not exist", :warn) if !File.exist?(pidfile)
-    done('Invalid pidfile content', :error) if pid == 0
-
-    fetch_process
-
-    begin
-      send(stage)
-    rescue NoMethodError
-      done "Invalid command: #{stage}", :error
-    end
-  end
-
-  def fetch_process
-    Process.kill(0, pid)
-  rescue Errno::ESRCH
-    done "Process doesn't exist", :error
-  # We were not allowed to send a signal, but the process must have existed
-  # when Process.kill() was called.
-  rescue Errno::EPERM
-    return pid
-  end
-
-  def done(msg, error = nil)
-    puts msg
-    exit(exit_signal(error))
-  end
-
-  def exit_signal(error)
-    (error == :error) ? 1 : 0
-  end
-
-  def pid
-    @pid ||= File.read(pidfile).to_i
-  end
-
-  def quiet
-    `kill -USR1 #{pid}`
-  end
-
-  def stop
-    `kill -TERM #{pid}`
-    kill_timeout.times do
-      begin
-        Process.kill(0, pid)
-      rescue Errno::ESRCH
-        FileUtils.rm_f pidfile
-        done 'Sidekiq shut down gracefully.'
-      rescue Errno::EPERM
-        done 'Not permitted to shut down Sidekiq.'
-      end
-      sleep 1
-    end
-    `kill -9 #{pid}`
-    FileUtils.rm_f pidfile
-    done 'Sidekiq shut down forcefully.'
-  end
-  alias_method :shutdown, :stop
-end
-
-if ARGV.length < 2
-  Sidekiqctl.print_usage
+if ARGV[0] == 'status'
+  Sidekiq::Ctl::Status.new.display(ARGV[1])
 else
-  stage = ARGV[0]
-  pidfile = ARGV[1]
-  timeout = ARGV[2].to_i
-  timeout = Sidekiqctl::DEFAULT_KILL_TIMEOUT if timeout == 0
+  if ARGV.length < 2
+    Sidekiq::Ctl.print_usage
+  else
+    stage = ARGV[0]
+    pidfile = ARGV[1]
+    timeout = ARGV[2].to_i
+    timeout = Sidekiq::Ctl::DEFAULT_KILL_TIMEOUT if timeout == 0
 
-  Sidekiqctl.new(stage, pidfile, timeout)
+    Sidekiq::Ctl.new(stage, pidfile, timeout)
+  end
 end

data/bin/sidekiqload

@@ -46,7 +46,7 @@ end
 #}])
 
 self_read, self_write = IO.pipe
-%w(INT TERM USR1 USR2 TTIN).each do |sig|
+%w(INT TERM TSTP TTIN).each do |sig|
   begin
     trap sig do
       self_write.puts(sig)
@@ -65,19 +65,14 @@ def handle_signal(launcher, sig)
     # http://jira.codehaus.org/browse/JRUBY-4637
     raise Interrupt
   when 'TERM'
-    # Heroku sends TERM and then waits 10 seconds for process to exit.
+    # Heroku sends TERM and then waits 30 seconds for process to exit.
     raise Interrupt
-  when 'USR1'
-    Sidekiq.logger.info "Received USR1, no longer accepting new work"
+  when 'TSTP'
+    Sidekiq.logger.info "Received TSTP, no longer accepting new work"
     launcher.quiet
-  when 'USR2'
-    if Sidekiq.options[:logfile]
-      Sidekiq.logger.info "Received USR2, reopening log file"
-      Sidekiq::Logging.reopen_logs
-    end
   when 'TTIN'
     Thread.list.each do |thread|
-      Sidekiq.logger.warn "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}"
+      Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread['label']}"
       if thread.backtrace
         Sidekiq.logger.warn thread.backtrace.join("\n")
       else

data/lib/generators/sidekiq/templates/worker_spec.rb.erb

@@ -1,6 +1,6 @@
 require 'rails_helper'
 <% module_namespacing do -%>
 RSpec.describe <%= class_name %>Worker, type: :worker do
-    pending "add some examples to (or delete) #{__FILE__}"
+  pending "add some examples to (or delete) #{__FILE__}"
 end
 <% end -%>