sidekiq 5.0.1 → 5.2.9

data/Pro-Changes.md

@@ -4,6 +4,131 @@
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+HEAD
+---------
+
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+4.0.5
+---------
+
+- Increase super\_fetch retriever thread count from 1 to 2 to make it
+  less sensitive to Redis latency.
+- Better handling of invalid job JSON by reliable scheduler [#4053]
+- Added ZH, PT, JA and RU translations.
+
+4.0.4
+---------
+
+- Update Sidekiq::Client patches to work with new Module#prepend
+  mechanism in Sidekiq 5.2.0. [#3930]
+
+4.0.3
+---------
+
+- Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
+- Implement batch death callback.  This is fired the first time a job within a batch dies. [#3841]
+```ruby
+batch = Sidekiq::Batch.new
+batch.on(:death, ...)
+```
+
+4.0.2
+---------
+
+- Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
+  call and resulting latency [#3790]
+- Fix possible bad statsd metric call on super\_fetch startup
+- Remove superfluous `freeze` calls on Strings [#3759]
+
+4.0.1
+---------
+
+- Fix incompatibility with the statsd-ruby gem [#3740]
+- Add tags to Statsd metrics when using Datadog [#3744]
+
+4.0.0
+---------
+
+- See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
+
+
+3.7.1
+---------
+
+- Deprecate timed\_fetch.  Switch to super\_fetch:
+```ruby
+config.super_fetch!
+```
+
+
+3.7.0
+---------
+
+- Refactor batch job success/failure to gracefully handle several edge
+  cases with regard to Sidekiq::Shutdown.  This should greatly reduce
+  the chances of seeing the long-standing "negative pending count" problem. [#3710]
+
+
+3.6.1
+---------
+
+- Add support for Datadog::Statsd, it is the recommended Statsd client.  [#3699]
+```ruby
+Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
+```
+- Size the statsd connection pool based on Sidekiq's concurrency [#3700]
+
+
+3.6.0
+---------
+
+This release overhauls the Statsd metrics support and adds more
+metrics for tracking Pro feature usage.  In your initializer:
+```ruby
+Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
+```
+Sidekiq Pro will emit more metrics to Statsd:
+```
+jobs.expired - when a job is expired
+jobs.recovered.push - when a job is recovered by reliable_push after network outage
+jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
+batch.created - when a batch is created
+batch.complete - when a batch is completed
+batch.success - when a batch is successful
+```
+Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
+Everything should be backwards compatible with one deprecation notice.
+
+
+3.5.4
+---------
+
+- Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
+- Fix case where TimedFetch might not recover some pending jobs
+- Fix edge case in Batch::Status#poll leading to premature completion [#3640]
+- Adjust scan API to check 100 elements at a time, to minimize network round trips
+  when scanning large sets.
+
+3.5.3
+---------
+
+- Restore error check for super\_fetch's job ack [#3601]
+- Trim error messages saved in Batch's failure hash, preventing huge
+  messages from bloating Redis. [#3570]
+
+3.5.2
+---------
+
+- Fix `Status#completed?` when run against a Batch that had succeeded
+  and was deleted. [#3519]
+
+3.5.1
+---------
+
+- Work with Sidekiq 5.0.2+
+- Improve performance of super\_fetch with weighted queues [#3489]
+
 3.5.0
 ---------
 

data/README.md

@@ -3,7 +3,7 @@ Sidekiq
 
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
 [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
-[![Build Status](https://travis-ci.org/mperham/sidekiq.svg)](https://travis-ci.org/mperham/sidekiq)
+[![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
 [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
 
 
@@ -77,16 +77,18 @@ Problems?
 If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
 Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
 
+Sidekiq Pro and Sidekiq Enterprise customers get private email support.  You can purchase at http://sidekiq.org; email support@contribsys.com for help.
+
 Useful resources:
 
 * Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
 * Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
 
-**No support via Twitter, 140 characters is not enough.**
+**No support via Twitter**
 
 Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
-See the [Sidekiq support page](http://sidekiq.org/support) for details.
+See the [Sidekiq support page](http://sidekiq.org/support.html) for details.
 
 Thanks
 -----------------

data/Rakefile

@@ -1,12 +1,9 @@
 require 'bundler/gem_tasks'
 require 'rake/testtask'
+
 Rake::TestTask.new(:test) do |test|
   test.warning = true
   test.pattern = 'test/**/test_*.rb'
 end
 
-task :default => :test
-
-task :appraise do
-  exec("cd myapp && rake appraise")
-end
+task default: :test

data/bin/sidekiqctl

@@ -1,99 +1,20 @@
 #!/usr/bin/env ruby
 
 require 'fileutils'
+require 'sidekiq/api'
+require 'sidekiq/ctl'
 
-class Sidekiqctl
-  DEFAULT_KILL_TIMEOUT = 10
-
-  attr_reader :stage, :pidfile, :kill_timeout
-
-  def self.print_usage
-    puts "#{File.basename($0)} - stop a Sidekiq process from the command line."
-    puts
-    puts "Usage: #{File.basename($0)} <command> <pidfile> <kill_timeout>"
-    puts " where <command> is either 'quiet' or 'stop'"
-    puts "       <pidfile> is path to a pidfile"
-    puts "       <kill_timeout> is number of seconds to wait until Sidekiq exits"
-    puts "       (default: #{Sidekiqctl::DEFAULT_KILL_TIMEOUT}), after which Sidekiq will be KILL'd"
-    puts
-    puts "Be sure to set the kill_timeout LONGER than Sidekiq's -t timeout.  If you want"
-    puts "to wait 60 seconds for jobs to finish, use `sidekiq -t 60` and `sidekiqctl stop"
-    puts " path_to_pidfile 61`"
-    puts
-  end
-
-  def initialize(stage, pidfile, timeout)
-    @stage = stage
-    @pidfile = pidfile
-    @kill_timeout = timeout
-
-    done('No pidfile given', :error) if !pidfile
-    done("Pidfile #{pidfile} does not exist", :warn) if !File.exist?(pidfile)
-    done('Invalid pidfile content', :error) if pid == 0
-
-    fetch_process
-
-    begin
-      send(stage)
-    rescue NoMethodError
-      done "Invalid command: #{stage}", :error
-    end
-  end
-
-  def fetch_process
-    Process.kill(0, pid)
-  rescue Errno::ESRCH
-    done "Process doesn't exist", :error
-  # We were not allowed to send a signal, but the process must have existed
-  # when Process.kill() was called.
-  rescue Errno::EPERM
-    return pid
-  end
-
-  def done(msg, error = nil)
-    puts msg
-    exit(exit_signal(error))
-  end
-
-  def exit_signal(error)
-    (error == :error) ? 1 : 0
-  end
-
-  def pid
-    @pid ||= File.read(pidfile).to_i
-  end
-
-  def quiet
-    `kill -TSTP #{pid}`
-  end
-
-  def stop
-    `kill -TERM #{pid}`
-    kill_timeout.times do
-      begin
-        Process.kill(0, pid)
-      rescue Errno::ESRCH
-        FileUtils.rm_f pidfile
-        done 'Sidekiq shut down gracefully.'
-      rescue Errno::EPERM
-        done 'Not permitted to shut down Sidekiq.'
-      end
-      sleep 1
-    end
-    `kill -9 #{pid}`
-    FileUtils.rm_f pidfile
-    done 'Sidekiq shut down forcefully.'
-  end
-  alias_method :shutdown, :stop
-end
-
-if ARGV.length < 2
-  Sidekiqctl.print_usage
+if ARGV[0] == 'status'
+  Sidekiq::Ctl::Status.new.display(ARGV[1])
 else
-  stage = ARGV[0]
-  pidfile = ARGV[1]
-  timeout = ARGV[2].to_i
-  timeout = Sidekiqctl::DEFAULT_KILL_TIMEOUT if timeout == 0
+  if ARGV.length < 2
+    Sidekiq::Ctl.print_usage
+  else
+    stage = ARGV[0]
+    pidfile = ARGV[1]
+    timeout = ARGV[2].to_i
+    timeout = Sidekiq::Ctl::DEFAULT_KILL_TIMEOUT if timeout == 0
 
-  Sidekiqctl.new(stage, pidfile, timeout)
+    Sidekiq::Ctl.new(stage, pidfile, timeout)
+  end
 end

data/bin/sidekiqload

@@ -65,14 +65,14 @@ def handle_signal(launcher, sig)
     # http://jira.codehaus.org/browse/JRUBY-4637
     raise Interrupt
   when 'TERM'
-    # Heroku sends TERM and then waits 10 seconds for process to exit.
+    # Heroku sends TERM and then waits 30 seconds for process to exit.
     raise Interrupt
   when 'TSTP'
     Sidekiq.logger.info "Received TSTP, no longer accepting new work"
     launcher.quiet
   when 'TTIN'
     Thread.list.each do |thread|
-      Sidekiq.logger.warn "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}"
+      Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread['label']}"
       if thread.backtrace
         Sidekiq.logger.warn thread.backtrace.join("\n")
       else

data/lib/sidekiq.rb

@@ -1,7 +1,7 @@
-# encoding: utf-8
 # frozen_string_literal: true
+
 require 'sidekiq/version'
-fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.2.2." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.2.2'
+fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.2.2." if RUBY_PLATFORM != 'java' && Gem::Version.new(RUBY_VERSION) < Gem::Version.new('2.2.2')
 
 require 'sidekiq/logging'
 require 'sidekiq/client'
@@ -12,19 +12,20 @@ require 'sidekiq/delay'
 require 'json'
 
 module Sidekiq
-  NAME = 'Sidekiq'.freeze
+  NAME = 'Sidekiq'
   LICENSE = 'See LICENSE and the LGPL-3.0 for licensing details.'
 
   DEFAULTS = {
     queues: [],
     labels: [],
-    concurrency: 25,
+    concurrency: 10,
     require: '.',
     environment: nil,
     timeout: 8,
     poll_interval_average: nil,
-    average_scheduled_poll_interval: 15,
+    average_scheduled_poll_interval: 5,
     error_handlers: [],
+    death_handlers: [],
     lifecycle_events: {
       startup: [],
       quiet: [],
@@ -47,7 +48,7 @@ module Sidekiq
     "connected_clients" => "9999",
     "used_memory_human" => "9P",
     "used_memory_peak_human" => "9P"
-  }.freeze
+  }
 
   def self.❨╯°□°❩╯︵┻━┻
     puts "Calm down, yo."
@@ -56,6 +57,7 @@ module Sidekiq
   def self.options
     @options ||= DEFAULTS.dup
   end
+
   def self.options=(opts)
     @options = opts
   end
@@ -94,8 +96,8 @@ module Sidekiq
       begin
         yield conn
       rescue Redis::CommandError => ex
-        #2550 Failover can cause the server to become a slave, need
-        # to disconnect and reopen the socket to get back to the master.
+        #2550 Failover can cause the server to become a replica, need
+        # to disconnect and reopen the socket to get back to the primary.
         (conn.disconnect!; retryable = false; retry) if retryable && ex.message =~ /READONLY/
         raise
       end
@@ -156,16 +158,23 @@ module Sidekiq
     defined?(@default_worker_options) ? @default_worker_options : DEFAULT_WORKER_OPTIONS
   end
 
+  def self.default_retries_exhausted=(prok)
+    logger.info { "default_retries_exhausted is deprecated, please use `config.death_handlers << -> {|job, ex| }`" }
+    return nil unless prok
+    death_handlers << prok
+  end
+
+  ##
+  # Death handlers are called when all retries for a job have been exhausted and
+  # the job dies.  It's the notification to your application
+  # that this job will not succeed without manual intervention.
+  #
   # Sidekiq.configure_server do |config|
-  #   config.default_retries_exhausted = -> (job, ex) do
+  #   config.death_handlers << ->(job, ex) do
   #   end
   # end
-  def self.default_retries_exhausted=(prok)
-    @default_retries_exhausted = prok
-  end
-  @default_retries_exhausted = ->(job, ex) { }
-  def self.default_retries_exhausted
-    @default_retries_exhausted
+  def self.death_handlers
+    options[:death_handlers]
   end
 
   def self.load_json(string)

data/lib/sidekiq/api.rb

@@ -1,9 +1,24 @@
-# encoding: utf-8
 # frozen_string_literal: true
 require 'sidekiq'
 
 module Sidekiq
+
+  module RedisScanner
+    def sscan(conn, key)
+      cursor = '0'
+      result = []
+      loop do
+        cursor, values = conn.sscan(key, cursor)
+        result.push(*values)
+        break if cursor == '0'
+      end
+      result
+    end
+  end
+
   class Stats
+    include RedisScanner
+
     def initialize
       fetch_stats!
     end
@@ -51,33 +66,39 @@ module Sidekiq
     def fetch_stats!
       pipe1_res = Sidekiq.redis do |conn|
         conn.pipelined do
-          conn.get('stat:processed'.freeze)
-          conn.get('stat:failed'.freeze)
-          conn.zcard('schedule'.freeze)
-          conn.zcard('retry'.freeze)
-          conn.zcard('dead'.freeze)
-          conn.scard('processes'.freeze)
-          conn.lrange('queue:default'.freeze, -1, -1)
-          conn.smembers('processes'.freeze)
-          conn.smembers('queues'.freeze)
+          conn.get('stat:processed')
+          conn.get('stat:failed')
+          conn.zcard('schedule')
+          conn.zcard('retry')
+          conn.zcard('dead')
+          conn.scard('processes')
+          conn.lrange('queue:default', -1, -1)
         end
       end
 
+      processes = Sidekiq.redis do |conn|
+        sscan(conn, 'processes')
+      end
+
+      queues = Sidekiq.redis do |conn|
+        sscan(conn, 'queues')
+      end
+
       pipe2_res = Sidekiq.redis do |conn|
         conn.pipelined do
-          pipe1_res[7].each {|key| conn.hget(key, 'busy'.freeze) }
-          pipe1_res[8].each {|queue| conn.llen("queue:#{queue}") }
+          processes.each {|key| conn.hget(key, 'busy') }
+          queues.each {|queue| conn.llen("queue:#{queue}") }
         end
       end
 
-      s = pipe1_res[7].size
+      s = processes.size
       workers_size = pipe2_res[0...s].map(&:to_i).inject(0, &:+)
       enqueued     = pipe2_res[s..-1].map(&:to_i).inject(0, &:+)
 
       default_queue_latency = if (entry = pipe1_res[6].first)
                                 job = Sidekiq.load_json(entry) rescue {}
                                 now = Time.now.to_f
-                                thence = job['enqueued_at'.freeze] || now
+                                thence = job['enqueued_at'] || now
                                 now - thence
                               else
                                 0
@@ -117,9 +138,11 @@ module Sidekiq
     end
 
     class Queues
+      include RedisScanner
+
       def lengths
         Sidekiq.redis do |conn|
-          queues = conn.smembers('queues'.freeze)
+          queues = sscan(conn, 'queues')
 
           lengths = conn.pipelined do
             queues.each do |queue|
@@ -163,7 +186,7 @@ module Sidekiq
 
         while i < @days_previous
           date = @start_date - i
-          datestr = date.strftime("%Y-%m-%d".freeze)
+          datestr = date.strftime("%Y-%m-%d")
           keys << "stat:#{stat}:#{datestr}"
           dates << datestr
           i += 1
@@ -199,18 +222,19 @@ module Sidekiq
   #
   class Queue
     include Enumerable
+    extend RedisScanner
 
     ##
     # Return all known queues within Redis.
     #
     def self.all
-      Sidekiq.redis { |c| c.smembers('queues'.freeze) }.sort.map { |q| Sidekiq::Queue.new(q) }
+      Sidekiq.redis { |c| sscan(c, 'queues') }.sort.map { |q| Sidekiq::Queue.new(q) }
     end
 
     attr_reader :name
 
     def initialize(name="default")
-      @name = name
+      @name = name.to_s
       @rname = "queue:#{name}"
     end
 
@@ -273,7 +297,7 @@ module Sidekiq
       Sidekiq.redis do |conn|
         conn.multi do
           conn.del(@rname)
-          conn.srem("queues".freeze, name)
+          conn.srem("queues", name)
         end
       end
     end
@@ -335,7 +359,7 @@ module Sidekiq
 
     def display_args
       # Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
-      @args ||= case klass
+      @display_args ||= case klass
                 when /\ASidekiq::Extensions::Delayed/
                   safe_load(args[0], args) do |_, _, arg|
                     arg
@@ -349,9 +373,9 @@ module Sidekiq
                     job_args
                   end
                 else
-                  if self['encrypt'.freeze]
+                  if self['encrypt']
                     # no point in showing 150+ bytes of random garbage
-                    args[-1] = '[encrypted data]'.freeze
+                    args[-1] = '[encrypted data]'
                   end
                   args
                 end
@@ -458,14 +482,7 @@ module Sidekiq
     # Place job in the dead set
     def kill
       remove_job do |message|
-        now = Time.now.to_f
-        Sidekiq.redis do |conn|
-          conn.multi do
-            conn.zadd('dead', now, message)
-            conn.zremrangebyscore('dead', '-inf', now - DeadSet.timeout)
-            conn.zremrangebyrank('dead', 0, - DeadSet.max_jobs)
-          end
-        end
+        DeadSet.new.kill(message)
       end
     end
 
@@ -555,7 +572,7 @@ module Sidekiq
         end
         break if elements.empty?
         page -= 1
-        elements.each do |element, score|
+        elements.reverse.each do |element, score|
           yield SortedEntry.new(self, score, element)
         end
         offset_size = initial_size - @_size
@@ -653,6 +670,12 @@ module Sidekiq
         each(&:retry)
       end
     end
+
+    def kill_all
+      while size > 0
+        each(&:kill)
+      end
+    end
   end
 
   ##
@@ -663,6 +686,27 @@ module Sidekiq
       super 'dead'
     end
 
+    def kill(message, opts={})
+      now = Time.now.to_f
+      Sidekiq.redis do |conn|
+        conn.multi do
+          conn.zadd(name, now.to_s, message)
+          conn.zremrangebyscore(name, '-inf', now - self.class.timeout)
+          conn.zremrangebyrank(name, 0, - self.class.max_jobs)
+        end
+      end
+
+      if opts[:notify_failure] != false
+        job = Sidekiq.load_json(message)
+        r = RuntimeError.new("Job killed by API")
+        r.set_backtrace(caller)
+        Sidekiq.death_handlers.each do |handle|
+          handle.call(job, r)
+        end
+      end
+      true
+    end
+
     def retry_all
       while size > 0
         each(&:retry)
@@ -687,17 +731,18 @@ module Sidekiq
   #
   class ProcessSet
     include Enumerable
+    include RedisScanner
 
     def initialize(clean_plz=true)
-      self.class.cleanup if clean_plz
+      cleanup if clean_plz
     end
 
     # Cleans up dead processes recorded in Redis.
     # Returns the number of processes cleaned.
-    def self.cleanup
+    def cleanup
       count = 0
       Sidekiq.redis do |conn|
-        procs = conn.smembers('processes').sort
+        procs = sscan(conn, 'processes').sort
         heartbeats = conn.pipelined do
           procs.each do |key|
             conn.hget(key, 'info')
@@ -717,7 +762,7 @@ module Sidekiq
     end
 
     def each
-      procs = Sidekiq.redis { |conn| conn.smembers('processes') }.sort
+      procs = Sidekiq.redis { |conn| sscan(conn, 'processes') }.sort
 
       Sidekiq.redis do |conn|
         # We're making a tradeoff here between consuming more memory instead of
@@ -852,10 +897,11 @@ module Sidekiq
   #
   class Workers
     include Enumerable
+    include RedisScanner
 
     def each
       Sidekiq.redis do |conn|
-        procs = conn.smembers('processes')
+        procs = sscan(conn, 'processes')
         procs.sort.each do |key|
           valid, workers = conn.pipelined do
             conn.exists(key)
@@ -877,7 +923,7 @@ module Sidekiq
     # which can easily get out of sync with crashy processes.
     def size
       Sidekiq.redis do |conn|
-        procs = conn.smembers('processes')
+        procs = sscan(conn, 'processes')
         if procs.empty?
           0
         else