sidekiq 3.4.1 → 4.0.0

data/README.md

@@ -4,7 +4,6 @@ Sidekiq
 [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
 [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
 [![Build Status](https://travis-ci.org/mperham/sidekiq.svg)](https://travis-ci.org/mperham/sidekiq)
-[![Coverage Status](https://coveralls.io/repos/mperham/sidekiq/badge.svg?branch=master)](https://coveralls.io/r/mperham/sidekiq)
 [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
 
 
@@ -33,7 +32,7 @@ are untested but might work fine.  MRI 1.9 is no longer supported.
 
 All Rails releases starting from 3.2 are officially supported.
 
-Redis 2.4 or greater is required.
+Redis 2.8 or greater is required.
 
 
 Installation
@@ -45,7 +44,7 @@ Installation
 Getting Started
 -----------------
 
-See the [sidekiq home page](http://sidekiq.org) for the simple 3-step process.
+See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
 You can watch [Railscast #366](http://railscasts.com/episodes/366-sidekiq) to see Sidekiq in action.  If you do everything right, you should see this:
 
 ![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
@@ -54,10 +53,10 @@ You can watch [Railscast #366](http://railscasts.com/episodes/366-sidekiq) to se
 Want to Upgrade?
 -------------------
 
-I also sell Sidekiq Pro, an extension to Sidekiq which provides more
-features, a commercial-friendly license and allows you to support high
+I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
+features, a commercial-friendly license and allow you to support high
 quality open source development all at the same time.  Please see the
-[Sidekiq Pro](http://sidekiq.org/pro) homepage for more detail.
+[Sidekiq](http://sidekiq.org/) homepage for more detail.
 
 
 More Information
@@ -67,7 +66,7 @@ Please see the [sidekiq wiki](https://github.com/mperham/sidekiq/wiki) for the o
 [mperham/sidekiq on Gitter](https://gitter.im/mperham/sidekiq) is dedicated to this project,
 but bug reports or feature requests suggestions should still go through [issues on Github](https://github.com/mperham/sidekiq/issues).  Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
 
-You may also find useful a [Google Group](https://groups.google.com/forum/#!forum/sidekiq) dedicated to Sidekiq discussion and [a Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow.
+You may also find useful a [Reddit area](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion and [a Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow.
 
 
 Problems?
@@ -76,14 +75,14 @@ Problems?
 **Please do not directly email any Sidekiq committers with questions or problems.**  A community is best served when discussions are held in public.
 
 If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages. Searching the issues for your problem is also a good idea.  If that doesn't help, feel free to email the Sidekiq mailing list, chat in Gitter, or open a new issue.
-The mailing list is the preferred place to ask questions on usage. If you are encountering what you think is a bug, please open an issue.
+StackOverflow or Reddit is the preferred place to ask questions on usage. If you are encountering what you think is a bug, please open an issue.
 
 
 Thanks
 -----------------
 
 Sidekiq stays fast by using the [JProfiler java profiler](http://www.ej-technologies.com/products/jprofiler/overview.html) to find and fix
-performance problems on JRuby.  Unfortunately MRI does not have good profile tooling.
+performance problems on JRuby.  Unfortunately MRI does not have good multithreaded profiling tools.
 
 
 License

data/bin/sidekiq

@@ -1,5 +1,10 @@
 #!/usr/bin/env ruby
 
+# Quiet some warnings we see when running in warning mode:
+# RUBYOPT=-w bundle exec sidekiq
+$TESTING = false
+$CELLULOID_DEBUG = false
+
 require_relative '../lib/sidekiq/cli'
 
 begin

data/bin/sidekiqctl

@@ -41,9 +41,13 @@ class Sidekiqctl
   end
 
   def fetch_process
-    Process.getpgid(pid)
+    Process.kill(0, pid)
   rescue Errno::ESRCH
     done "Process doesn't exist", :error
+  # We were not allowed to send a signal, but the process must have existed
+  # when Process.kill() was called.
+  rescue Errno::EPERM
+    return pid
   end
 
   def done(msg, error = nil)
@@ -67,10 +71,12 @@ class Sidekiqctl
     `kill -TERM #{pid}`
     kill_timeout.times do
       begin
-        Process.getpgid(pid)
+        Process.kill(0, pid)
       rescue Errno::ESRCH
         FileUtils.rm_f pidfile
         done 'Sidekiq shut down gracefully.'
+      rescue Errno::EPERM
+        done 'Not permitted to shut down Sidekiq.'
       end
       sleep 1
     end

data/bin/sidekiqload

@@ -0,0 +1,167 @@
+#!/usr/bin/env ruby
+
+# Quiet some warnings we see when running in warning mode:
+# RUBYOPT=-w bundle exec sidekiq
+$TESTING = false
+
+#require 'ruby-prof'
+Bundler.require(:default)
+
+require_relative '../lib/sidekiq/cli'
+require_relative '../lib/sidekiq/launcher'
+
+include Sidekiq::Util
+
+# brew tap shopify/shopify
+# brew install toxiproxy
+# gem install toxiproxy
+require 'toxiproxy'
+# simulate a non-localhost network for realer-world conditions.
+# adding 1ms of network latency has an ENORMOUS impact on benchmarks
+Toxiproxy.populate([{
+    "name": "redis",
+    "listen": "127.0.0.1:6380",
+    "upstream": "127.0.0.1:6379"
+}])
+
+
+Sidekiq.configure_server do |config|
+  config.redis = { db: 13, port: 6380 }
+  #config.redis = { db: 13 }
+  config.options[:queues] << 'default'
+  config.logger.level = Logger::ERROR
+  config.average_scheduled_poll_interval = 2
+  config.reliable! if defined?(Sidekiq::Pro)
+end
+
+class LoadWorker
+  include Sidekiq::Worker
+  sidekiq_options retry: 1
+  sidekiq_retry_in do |x|
+    1
+  end
+
+  def perform(idx)
+    #raise idx.to_s if idx % 100 == 1
+  end
+end
+
+# brew tap shopify/shopify
+# brew install toxiproxy
+# gem install toxiproxy
+require 'toxiproxy'
+# simulate a non-localhost network for realer-world conditions.
+# adding 1ms of network latency has an ENORMOUS impact on benchmarks
+Toxiproxy.populate([{
+    "name": "redis",
+    "listen": "127.0.0.1:6380",
+    "upstream": "127.0.0.1:6379"
+}])
+
+self_read, self_write = IO.pipe
+%w(INT TERM USR1 USR2 TTIN).each do |sig|
+  begin
+    trap sig do
+      self_write.puts(sig)
+    end
+  rescue ArgumentError
+    puts "Signal #{sig} not supported"
+  end
+end
+
+Sidekiq.redis {|c| c.flushdb}
+def handle_signal(launcher, sig)
+  Sidekiq.logger.debug "Got #{sig} signal"
+  case sig
+  when 'INT'
+    # Handle Ctrl-C in JRuby like MRI
+    # http://jira.codehaus.org/browse/JRUBY-4637
+    raise Interrupt
+  when 'TERM'
+    # Heroku sends TERM and then waits 10 seconds for process to exit.
+    raise Interrupt
+  when 'USR1'
+    Sidekiq.logger.info "Received USR1, no longer accepting new work"
+    launcher.quiet
+  when 'USR2'
+    if Sidekiq.options[:logfile]
+      Sidekiq.logger.info "Received USR2, reopening log file"
+      Sidekiq::Logging.reopen_logs
+    end
+  when 'TTIN'
+    Thread.list.each do |thread|
+      Sidekiq.logger.warn "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}"
+      if thread.backtrace
+        Sidekiq.logger.warn thread.backtrace.join("\n")
+      else
+        Sidekiq.logger.warn "<no backtrace available>"
+      end
+    end
+  end
+end
+
+def Process.rss
+  `ps -o rss= -p #{Process.pid}`.chomp.to_i
+end
+
+iter = 10
+count = 10_000
+
+iter.times do
+  arr = Array.new(count) do
+    []
+  end
+  count.times do |idx|
+    arr[idx][0] = idx
+  end
+  Sidekiq::Client.push_bulk('class' => LoadWorker, 'args' => arr)
+end
+Sidekiq.logger.error "Created #{count*iter} jobs"
+
+Monitoring = Thread.new do
+  watchdog("monitor thread") do
+    while true
+      sleep 2
+      qsize, retries = Sidekiq.redis do |conn|
+        conn.pipelined do
+          conn.llen "queue:default"
+          conn.zcard "retry"
+        end
+      end.map(&:to_i)
+      total = qsize + retries
+      #GC.start
+      Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
+      if total == 0
+        Sidekiq.logger.error("Done")
+        exit(0)
+      end
+    end
+  end
+end
+
+begin
+  #RubyProf::exclude_threads = [ Monitoring ]
+  #RubyProf.start
+  fire_event(:startup)
+  Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
+  Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
+    launcher = Sidekiq::Launcher.new(Sidekiq.options)
+    launcher.run
+
+    while readable_io = IO.select([self_read])
+      signal = readable_io.first[0].gets.strip
+      handle_signal(launcher, signal)
+    end
+  end
+rescue SystemExit => e
+  #Sidekiq.logger.error("Profiling...")
+  #result = RubyProf.stop
+  #printer = RubyProf::GraphHtmlPrinter.new(result)
+  #printer.print(File.new("output.html", "w"), :min_percent => 1)
+  # normal
+rescue => e
+  raise e if $DEBUG
+  STDERR.puts e.message
+  STDERR.puts e.backtrace.join("\n")
+  exit 1
+end

data/lib/sidekiq/api.rb

@@ -159,14 +159,15 @@ module Sidekiq
 
         while i < @days_previous
           date = @start_date - i
-          keys << "stat:#{stat}:#{date}"
-          dates << date
+          datestr = date.strftime("%Y-%m-%d".freeze)
+          keys << "stat:#{stat}:#{datestr}"
+          dates << datestr
           i += 1
         end
 
         Sidekiq.redis do |conn|
           conn.mget(keys).each_with_index do |value, idx|
-            stat_hash[dates[idx].to_s] = value ? value.to_i : 0
+            stat_hash[dates[idx]] = value ? value.to_i : 0
           end
         end
 
@@ -224,7 +225,7 @@ module Sidekiq
       page = 0
       page_size = 50
 
-      loop do
+      while true do
         range_start = page * page_size - deleted_size
         range_end   = page * page_size - deleted_size + (page_size - 1)
         entries = Sidekiq.redis do |conn|
@@ -262,7 +263,6 @@ module Sidekiq
   # removed from the queue via Job#delete.
   #
   class Job
-    KNOWN_WRAPPERS = [/\ASidekiq::Extensions::Delayed/, "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"]
     attr_reader :item
 
     def initialize(item, queue_name=nil)
@@ -312,7 +312,7 @@ module Sidekiq
     end
 
     def enqueued_at
-      Time.at(@item['enqueued_at'] || 0).utc
+      @item['enqueued_at'] ? Time.at(@item['enqueued_at']).utc : nil
     end
 
     def created_at
@@ -369,11 +369,15 @@ module Sidekiq
     end
 
     def delete
-      @parent.delete(score, jid)
+      if @value
+        @parent.delete_by_value(@parent.name, @value)
+      else
+        @parent.delete_by_jid(score, jid)
+      end
     end
 
     def reschedule(at)
-      @parent.delete(score, jid)
+      delete
       @parent.schedule(at, item)
     end
 
@@ -484,7 +488,7 @@ module Sidekiq
       page = -1
       page_size = 50
 
-      loop do
+      while true do
         range_start = page * page_size + offset_size
         range_end   = page * page_size + offset_size + (page_size - 1)
         elements = Sidekiq.redis do |conn|
@@ -519,36 +523,30 @@ module Sidekiq
       self.detect { |j| j.jid == jid }
     end
 
-    def delete(score, jid = nil)
-      if jid
-        elements = Sidekiq.redis do |conn|
-          conn.zrangebyscore(name, score, score)
-        end
+    def delete_by_value(name, value)
+      Sidekiq.redis do |conn|
+        ret = conn.zrem(name, value)
+        @_size -= 1 if ret
+        ret
+      end
+    end
 
-        elements_with_jid = elements.map do |element|
+    def delete_by_jid(score, jid)
+      Sidekiq.redis do |conn|
+        elements = conn.zrangebyscore(name, score, score)
+        elements.each do |element|
           message = Sidekiq.load_json(element)
-
           if message["jid"] == jid
-            _, @_size = Sidekiq.redis do |conn|
-              conn.multi do
-                conn.zrem(name, element)
-                conn.zcard name
-              end
-            end
-          end
-        end
-        elements_with_jid.count != 0
-      else
-        count, @_size = Sidekiq.redis do |conn|
-          conn.multi do
-            conn.zremrangebyscore(name, score, score)
-            conn.zcard name
+            ret = conn.zrem(name, element)
+            @_size -= 1 if ret
+            break ret
           end
+          false
         end
-        count != 0
       end
     end
 
+    alias_method :delete, :delete_by_jid
   end
 
   ##

data/lib/sidekiq/cli.rb

@@ -11,18 +11,18 @@ require 'sidekiq'
 require 'sidekiq/util'
 
 module Sidekiq
-  # We are shutting down Sidekiq but what about workers that
-  # are working on some long job?  This error is
-  # raised in workers that have not finished within the hard
-  # timeout limit.  This is needed to rollback db transactions,
-  # otherwise Ruby's Thread#kill will commit.  See #377.
-  # DO NOT RESCUE THIS ERROR.
-  class Shutdown < Interrupt; end
-
   class CLI
     include Util
     include Singleton unless $TESTING
 
+    PROCTITLES = [
+      proc { 'sidekiq'.freeze },
+      proc { Sidekiq::VERSION },
+      proc { |me, data| data['tag'] },
+      proc { |me, data| "[#{Processor::WORKER_STATE.size} of #{data['concurrency']} busy]" },
+      proc { |me, data| "stopping" if me.stopping? },
+    ]
+
     # Used for CLI testing
     attr_accessor :code
     attr_accessor :launcher
@@ -40,7 +40,6 @@ module Sidekiq
       validate!
       daemonize
       write_pid
-      load_celluloid
     end
 
     # Code within this method is not tested because it alters
@@ -64,15 +63,23 @@ module Sidekiq
 
       logger.info "Running in #{RUBY_DESCRIPTION}"
       logger.info Sidekiq::LICENSE
-      logger.info "Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org/pro" unless defined?(::Sidekiq::Pro)
-
-      fire_event(:startup)
+      logger.info "Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org" unless defined?(::Sidekiq::Pro)
 
       Sidekiq.redis do |conn|
         # touch the connection pool so it is created before we
-        # launch the actors.
+        # fire startup and start multithreading.
+        ver = conn.info['redis_version']
+        raise "You are using Redis v#{ver}, Sidekiq requires Redis v2.8.0 or greater" if ver < '2.8'
       end
 
+      # Before this point, the process is initializing with just the main thread.
+      # Starting here the process will now have multiple threads running.
+      fire_event(:startup)
+
+      logger.debug {
+        "Middleware: #{Sidekiq.server_middleware.map(&:klass).join(', ')}"
+      }
+
       if !options[:daemon]
         logger.info 'Starting processing, hit Ctrl-C to stop'
       end
@@ -90,26 +97,28 @@ module Sidekiq
       rescue Interrupt
         logger.info 'Shutting down'
         launcher.stop
-        fire_event(:shutdown, true)
         # Explicitly exit so busy Processor threads can't block
         # process shutdown.
+        logger.info "Bye!"
         exit(0)
       end
     end
 
     def self.banner
-%q{         s
-          ss
-     sss  sss         ss
-     s  sss s   ssss sss   ____  _     _      _    _
-     s     sssss ssss     / ___|(_) __| | ___| | _(_) __ _
-    s         sss         \___ \| |/ _` |/ _ \ |/ / |/ _` |
-    s sssss  s             ___) | | (_| |  __/   <| | (_| |
-    ss    s  s            |____/|_|\__,_|\___|_|\_\_|\__, |
-    s     s s                                           |_|
-          s s
-         sss
-         sss }
+%q{
+         m,
+         `$b
+    .ss,  $$:         .,d$
+    `$$P,d$P'    .,md$P"'
+     ,$$$$$bmmd$$$P^'
+   .d$$$$$$$$$$P'
+   $$^' `"^$$$'       ____  _     _      _    _
+   $:     ,$$:       / ___|(_) __| | ___| | _(_) __ _
+   `b     :$$        \___ \| |/ _` |/ _ \ |/ / |/ _` |
+          $$:         ___) | | (_| |  __/   <| | (_| |
+          $$         |____/|_|\__,_|\___|_|\_\_|\__, |
+        .d$$                                       |_|
+}
     end
 
     def handle_signal(sig)
@@ -124,8 +133,7 @@ module Sidekiq
         raise Interrupt
       when 'USR1'
         Sidekiq.logger.info "Received USR1, no longer accepting new work"
-        launcher.manager.async.stop
-        fire_event(:quiet, true)
+        launcher.quiet
       when 'USR2'
         if Sidekiq.options[:logfile]
           Sidekiq.logger.info "Received USR2, reopening log file"
@@ -154,19 +162,6 @@ module Sidekiq
       end
     end
 
-    def load_celluloid
-      raise "Celluloid cannot be required until here, or it will break Sidekiq's daemonization" if defined?(::Celluloid) && options[:daemon]
-
-      # Celluloid can't be loaded until after we've daemonized
-      # because it spins up threads and creates locks which get
-      # into a very bad state if forked.
-      require 'celluloid/autostart'
-      Celluloid.logger = (options[:verbose] ? Sidekiq.logger : nil)
-
-      require 'sidekiq/manager'
-      require 'sidekiq/scheduled'
-    end
-
     def daemonize
       return unless options[:daemon]
 
@@ -340,7 +335,11 @@ module Sidekiq
         die 1
       end
       @parser.parse!(argv)
-      opts[:config_file] ||= 'config/sidekiq.yml' if File.exist?('config/sidekiq.yml')
+
+      %w[config/sidekiq.yml config/sidekiq.yml.erb].each do |filename|
+        opts[:config_file] ||= filename if File.exist?(filename)
+      end
+
       opts
     end
 

data/lib/sidekiq/client.rb

@@ -119,16 +119,12 @@ module Sidekiq
 
     class << self
 
-      def default
-        @default ||= new
-      end
-
       def push(item)
-        default.push(item)
+        new.push(item)
       end
 
       def push_bulk(items)
-        default.push_bulk(items)
+        new.push_bulk(items)
       end
 
       # Resque compatibility helpers.  Note all helpers
@@ -210,10 +206,9 @@ module Sidekiq
     end
 
     def normalize_item(item)
-      raise(ArgumentError, "Message must be a Hash of the form: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash)
-      raise(ArgumentError, "Message must include a class and set of arguments: #{item.inspect}") if !item['class'] || !item['args']
-      raise(ArgumentError, "Message args must be an Array") unless item['args'].is_a?(Array)
-      raise(ArgumentError, "Message class must be either a Class or String representation of the class name") unless item['class'].is_a?(Class) || item['class'].is_a?(String)
+      raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash) && item.has_key?('class'.freeze) && item.has_key?('args'.freeze)
+      raise(ArgumentError, "Job args must be an Array") unless item['args'].is_a?(Array)
+      raise(ArgumentError, "Job class must be either a Class or String representation of the class name") unless item['class'.freeze].is_a?(Class) || item['class'.freeze].is_a?(String)
 
       normalized_hash(item['class'.freeze])
         .each{ |key, value| item[key] = value if item[key].nil? }