sidekiq 2.17.8 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 01d6264503e20644482c431b9381fb01b3a37733
-  data.tar.gz: a61f928dd7d5f0994959fcddaa0f04e837d685b2
+  metadata.gz: 6924f9ac5a08e3bfb8b20204262b281082a590c9
+  data.tar.gz: db0c1c65e0f8ca499771c8f68174cc889a977e79
 SHA512:
-  metadata.gz: f92ff4829fb64f95480838fdab6f23d297a67a4acb6efe4da61684824d4f745c01969b371c96c63d050b618fb11c690293e50807b7e90b972f750eb4080abb29
-  data.tar.gz: 7544d61d9eb35f0983beaaf3e99b20dff42a2b1fbafdee572eb717da81bfb5bdb2e9be54e7359456c8d7f64b1412b9d9a2d15d07cda7edbbf5fdb2c893ae37ab
+  metadata.gz: 541f6b4546da89be32045f56d4e5dd41aa9d97d03860ee4fdde696cfb88d9b6ca68d4fb5f83377696d4ee1a9fe572d129afe8c5ddfebb61809302ed0cd54f88c
+  data.tar.gz: d396cad4b53238a174c722a92e485fb8657664d1965ea826c264c2aab1e5e25991023edce4556e3ceff05250bccf02892190b9fa69a1bc1c40944a71cb20d445

data/3.0-Upgrade.md

@@ -0,0 +1,63 @@
+# Upgrading to Sidekiq 3.0
+
+Sidekiq 3.0 brings several new features but also removes old APIs and
+changes a few data elements in Redis.  To upgrade cleanly:
+
+* Upgrade to the latest Sidekiq 2.x and run it for a few weeks.
+  `gem 'sidekiq', '< 3'`
+  This is only needed if you have retries pending.
+* 3rd party gems which use **client-side middleware** will need to update
+  due to an API change.  The Redis connection for a particular job is
+  passed thru the middleware to handle sharding where jobs can
+  be pushed to different redis server instances.
+
+  `def call(worker_class, msg, queue, redis_pool)`
+
+  Client-side middleware should use `redis_pool.with { |conn| ... }` to
+  perform Redis operations and **not** `Sidekiq.redis`.
+* If you used the capistrano integration, you'll need to pull in the
+  new [capistrano-sidekiq](https://github.com/seuros/capistrano-sidekiq)
+  gem and use it in your deploy.rb.
+* API changes:
+  - `Sidekiq::Client.registered_workers` replaced by `Sidekiq::Workers.new`
+  - `Sidekiq::Client.registered_queues` replaced by `Sidekiq::Queue.all`
+  - `Sidekiq::Worker#retries_exhausted` replaced by `Sidekiq::Worker.sidekiq_retries_exhausted`
+  - `Sidekiq::Workers#each` has changed significantly with a reworking
+    of Sidekiq's internal process/thread data model.
+* Redis-to-Go is no longer transparently activated on Heroku so as to not play
+  favorites with any particular Redis service. You need to set a config option
+  for your app:
+  `heroku config:set REDIS_PROVIDER=REDISTOGO_URL`
+* Anyone using Airbrake, Honeybadger, Exceptional or ExceptionNotifier
+  will need to update their error gem version to the latest to pull in
+  Sidekiq support.  Sidekiq will not provide explicit support for these
+  services so as to not play favorites with any particular error service.
+* MRI 1.9 is no longer officially supported.  Sidekiq's official
+  support policy is to support the current and previous major releases
+  of MRI and Rails.  As of February 2014, that's MRI 2.1, MRI 2.0, JRuby 1.7, Rails 4.0
+  and Rails 3.2.  I will consider PRs to fix issues found by users for
+  other platforms/versions.
+
+## Error Service Providers
+
+If you previously provided a middleware to capture job errors, you
+should instead provide a global error handler with Sidekiq 3.0.  This
+ensures **any** error within Sidekiq will be logged appropriately, not
+just during job execution.
+
+```ruby
+if Sidekiq::VERSION < '3'
+  # old behavior
+  Sidekiq.configure_server do |config|
+    config.server_middleware do |chain|
+      chain.add MyErrorService::Middleware
+    end
+  end
+else
+  Sidekiq.configure_server do |config|
+    config.error_handlers << Proc.new {|ex,context| MyErrorService.notify(ex, context) }
+  end
+end
+```
+
+Your error handler must respond to `call(exception, context_hash)`.

data/Changes.md

@@ -1,8 +1,71 @@
-2.17.8
+3.0.0
 -----------
 
-- Tighten allowed gem versions to prevent future incompatibility.  Fixes
-  breakage with Celluloid 0.16.
+**Not yet released**
+
+Please see [3.0-Upgrade.md](3.0-Upgrade.md) for more comprehensive upgrade notes.
+
+- **Dead Job Queue** - jobs which run out of retries are now moved to a dead
+  job queue.  These jobs must be retried manually or they will expire
+  after 6 months or 10,000 jobs.  The Web UI contains a "Dead" tab
+  exposing these jobs.  Use `sidekiq_options :retry => false` if you
+don't wish jobs to be retried or put in the DJQ.  Use
+`sidekiq_options :retry => 0` if you don't want jobs to retry but go
+straight to the DJQ.
+- **Process Lifecycle Events** - you can now register blocks to run at
+  certain points during the Sidekiq process lifecycle: startup, quiet and
+  shutdown.
+```ruby
+Sidekiq.configure_server do |config|
+  config.on(:startup) do
+    # do something
+  end
+end
+```
+- **Global Error Handlers** - blocks of code which handle errors that
+  occur anywhere within Sidekiq, not just within middleware.
+```ruby
+Sidekiq.configure_server do |config|
+  config.error_handlers << Proc.new {|ex,ctx| ... }
+end
+```
+- **Process Heartbeat** - each Sidekiq process will ping Redis every 5
+  seconds to give a summary of the Sidekiq population at work.
+- The Workers tab is now renamed to Busy and contains a list of live
+  Sidekiq processes and jobs in progress based on the heartbeat.
+- **Shardable Client** - Sidekiq::Client instances can use a custom
+  Redis connection pool, allowing very large Sidekiq installations to scale by
+  sharding: sending different jobs to different Redis instances.
+```ruby
+client = Sidekiq::Client.new(ConnectionPool.new { Redis.new })
+client.push(...)
+```
+```ruby
+Sidekiq::Client.via(ConnectionPool.new { Redis.new }) do
+  FooWorker.perform_async
+  BarWorker.perform_async
+end
+```
+  **Sharding support does require a breaking change to client-side
+middleware, see 3.0-Upgrade.md.**
+- New Chinese, Greek, Swedish and Czech translations for the Web UI.
+- Updated most languages translations for the new UI features.
+- **Remove official Capistrano integration** - this integration has been
+  moved into the [capistrano-sidekiq](https://github.com/seuros/capistrano-sidekiq) gem.
+- **Remove official support for MRI 1.9** - Things still might work but
+  I no longer actively test on it.
+- **Remove built-in support for Redis-to-Go**.
+  Heroku users: `heroku config:set REDIS_PROVIDER=REDISTOGO_URL`
+- **Remove built-in error integration for Airbrake, Honeybadger, ExceptionNotifier and Exceptional**.
+  Each error gem should provide its own Sidekiq integration.  Update your error gem to the latest
+  version to pick up Sidekiq support.
+- Upgrade to connection\_pool 2.0 which now creates connections lazily.
+- Remove deprecated Sidekiq::Client.registered\_\* APIs
+- Remove deprecated support for the old Sidekiq::Worker#retries\_exhausted method.
+- Removed 'sidekiq/yaml\_patch', this was never documented or recommended.
+- Removed --profile option, #1592
+- Remove usage of the term 'Worker' in the UI for clarity.  Users would call both threads and
+  processes 'workers'.  Instead, use "Thread", "Process" or "Job".
 
 2.17.7
 -----------

data/Contributing.md

@@ -1,7 +1,6 @@
 # Contributing
 
-First of all, thank you for even opening this file up!  I hope you find
-it worthwhile to help out with Sidekiq.
+First of all, thank you!  I hope you find it worthwhile to help out with Sidekiq.
 
 ## Issues
 
@@ -21,7 +20,6 @@ fix or new functionality.  Functionality must meet my design goals and
 vision for the project to be accepted; I would be happy to discuss how
 your idea can best fit into Sidekiq.
 
-
 ## Sponsorship
 
 If you've got more money than time and want to sponsor Sidekiq's continued support, your company can buy [Sidekiq Pro](http://sidekiq.org/pro).  You get great functionality, I continue to fix bugs and enhance Sidekiq for years to come.

data/Pro-Changes.md

@@ -3,6 +3,24 @@ Sidekiq Pro Changelog
 
 Please see [http://sidekiq.org/pro](http://sidekiq.org/pro) for more details and how to buy.
 
+HEAD
+-----------
+
+- Compatible with Sidekiq 3.
+
+1.5.1
+-----------
+
+- Due to a breaking API change in Sidekiq 3.0, this version is limited
+  to Sidekiq 2.x.
+
+1.5.0
+-----------
+
+- Compatible with upcoming Sidekiq 3.0 release
+- Fix issue on Heroku where reliable fetch could orphan jobs [#1573]
+
+
 1.4.3
 -----------
 

data/README.md

@@ -17,14 +17,14 @@ use the Resque client to enqueue jobs in Redis to be processed by Sidekiq.
 
 At the same time, Sidekiq uses multithreading so it is much more memory efficient than
 Resque (which forks a new process for every job).  You'll find that you might need
-50 200MB resque processes to peg your CPU whereas one 300MB Sidekiq process will peg
+10 200MB resque processes to peg your CPU whereas one 300MB Sidekiq process will peg
 the same CPU and perform the same amount of work.
 
 
 Requirements
 -----------------
 
-I test with the latest Ruby (2.0) and JRuby versions (1.7).  Other versions/VMs
+I test with the latest MRI (2.1, 2.0) and JRuby versions (1.7).  Other versions/VMs
 are untested but might work fine.
 
 The last two major Rails releases (3.2 and 4.0) are officially supported, other

data/bin/sidekiqctl

@@ -3,9 +3,19 @@
 require 'fileutils'
 
 class Sidekiqctl
+  DEFAULT_TIMEOUT = 10
 
   attr_reader :stage, :pidfile, :timeout
 
+  def self.print_usage
+    puts
+    puts "Usage: #{File.basename($0)} <command> <pidfile> <timeout>"
+    puts " where <command> is either 'quiet', 'stop' or 'shutdown'"
+    puts "       <pidfile> is path to a pidfile"
+    puts "       <timeout> is number of seconds to wait till Sidekiq exits (default: #{Sidekiqctl::DEFAULT_TIMEOUT})"
+    puts
+  end
+
   def initialize(stage, pidfile, timeout)
     @stage = stage
     @pidfile = pidfile
@@ -67,12 +77,15 @@ class Sidekiqctl
     quiet
     stop
   end
-
 end
 
-stage = ARGV[0]
-pidfile = ARGV[1]
-timeout = ARGV[2].to_i
-timeout = 10 if timeout == 0
+if ARGV.length < 2
+  Sidekiqctl.print_usage
+else
+  stage = ARGV[0]
+  pidfile = ARGV[1]
+  timeout = ARGV[2].to_i
+  timeout = Sidekiqctl::DEFAULT_TIMEOUT if timeout == 0
 
-Sidekiqctl.new(stage, pidfile, timeout)
+  Sidekiqctl.new(stage, pidfile, timeout)
+end

data/lib/sidekiq.rb

@@ -4,8 +4,6 @@ require 'sidekiq/logging'
 require 'sidekiq/client'
 require 'sidekiq/worker'
 require 'sidekiq/redis_connection'
-require 'sidekiq/util'
-require 'sidekiq/api'
 
 require 'json'
 
@@ -19,7 +17,12 @@ module Sidekiq
     :require => '.',
     :environment => nil,
     :timeout => 8,
-    :profile => false,
+    :error_handlers => [],
+    :lifecycle_events => {
+      :startup => [],
+      :quiet => [],
+      :shutdown => [],
+    },
   }
 
   def self.❨╯°□°❩╯︵┻━┻
@@ -63,17 +66,18 @@ module Sidekiq
 
   def self.redis(&block)
     raise ArgumentError, "requires a block" if !block
-    @redis ||= Sidekiq::RedisConnection.create(@hash || {})
-    @redis.with(&block)
+    redis_pool.with(&block)
   end
 
-  def self.redis=(hash)
-    return @redis = hash if hash.is_a?(ConnectionPool)
+  def self.redis_pool
+    @redis ||= Sidekiq::RedisConnection.create
+  end
 
-    if hash.is_a?(Hash)
-      @hash = hash
+  def self.redis=(hash)
+    @redis = if hash.is_a?(ConnectionPool)
+      hash
     else
-      raise ArgumentError, "redis= requires a Hash or ConnectionPool"
+      Sidekiq::RedisConnection.create(hash)
     end
   end
 
@@ -94,7 +98,7 @@ module Sidekiq
   end
 
   def self.default_worker_options
-    @default_worker_options || { 'retry' => true, 'queue' => 'default' }
+    defined?(@default_worker_options) ? @default_worker_options : { 'retry' => true, 'queue' => 'default' }
   end
 
   def self.load_json(string)
@@ -117,6 +121,44 @@ module Sidekiq
     self.options[:poll_interval] = interval
   end
 
+  # Register a proc to handle any error which occurs within the Sidekiq process.
+  #
+  #   Sidekiq.configure_server do |config|
+  #     config.error_handlers << Proc.new {|ex,ctx_hash| MyErrorService.notify(ex, ctx_hash) }
+  #   end
+  #
+  # The default error handler logs errors to Sidekiq.logger.
+  def self.error_handlers
+    self.options[:error_handlers]
+  end
+
+  # Register a block to run at a point in the Sidekiq lifecycle.
+  # :startup, :quiet or :shutdown are valid events.
+  #
+  #   Sidekiq.configure_server do |config|
+  #     config.on(:shutdown) do
+  #       puts "Goodbye cruel world!"
+  #     end
+  #   end
+  def self.on(event, &block)
+    raise ArgumentError, "Symbols only please: #{event}" if !event.is_a?(Symbol)
+    raise ArgumentError, "Invalid event name: #{event}" if !options[:lifecycle_events].keys.include?(event)
+    options[:lifecycle_events][event] << block
+  end
+
+  BANNER = %q{         s
+        ss
+   sss  sss         ss
+   s  sss s   ssss sss   ____  _     _      _    _
+   s     sssss ssss     / ___|(_) __| | ___| | _(_) __ _
+  s         sss         \___ \| |/ _` |/ _ \ |/ / |/ _` |
+  s sssss  s             ___) | | (_| |  __/   <| | (_| |
+  ss    s  s            |____/|_|\__,_|\___|_|\_\_|\__, |
+  s     s s                                           |_|
+        s s
+       sss
+       sss }
+
 end
 
 require 'sidekiq/extensions/class_methods'

data/lib/sidekiq/actor.rb

@@ -16,6 +16,7 @@ module Sidekiq
       def after(interval)
       end
       def alive?
+        @dead = false unless defined?(@dead)
         !@dead
       end
       def terminate

data/lib/sidekiq/api.rb

@@ -1,3 +1,4 @@
+# encoding: utf-8
 require 'sidekiq'
 
 module Sidekiq
@@ -14,8 +15,13 @@ module Sidekiq
       all   = %w(failed processed)
       stats = stats.empty? ? all : all & stats.flatten.compact.map(&:to_s)
 
+      mset_args = []
+      stats.each do |stat|
+        mset_args << "stat:#{stat}"
+        mset_args << 0
+      end
       Sidekiq.redis do |conn|
-        stats.each { |stat| conn.set("stat:#{stat}", 0) }
+        conn.mset(*mset_args)
       end
     end
 
@@ -23,8 +29,16 @@ module Sidekiq
       Sidekiq.redis do |conn|
         queues = conn.smembers('queues')
 
+        lengths = conn.pipelined do
+          queues.each do |queue|
+            conn.llen("queue:#{queue}")
+          end
+        end
+
+        i = 0
         array_of_arrays = queues.inject({}) do |memo, queue|
-          memo[queue] = conn.llen("queue:#{queue}")
+          memo[queue] = lengths[i]
+          i += 1
           memo
         end.sort_by { |_, size| size }
 
@@ -44,6 +58,10 @@ module Sidekiq
       Sidekiq.redis {|c| c.zcard('retry') }
     end
 
+    def dead_size
+      Sidekiq.redis {|c| c.zcard('dead') }
+    end
+
     class History
       def initialize(days_previous, start_date = nil)
         @days_previous = days_previous
@@ -63,15 +81,19 @@ module Sidekiq
       def date_stat_hash(stat)
         i = 0
         stat_hash = {}
+        keys = []
+        dates = []
+
+        while i < @days_previous
+          date = @start_date - i
+          keys << "stat:#{stat}:#{date}"
+          dates << date
+          i += 1
+        end
 
         Sidekiq.redis do |conn|
-          while i < @days_previous
-            date = @start_date - i
-            value = conn.get("stat:#{stat}:#{date}")
-
-            stat_hash[date.to_s] = value ? value.to_i : 0
-
-            i += 1
+          conn.mget(keys).each_with_index do |value, i|
+            stat_hash[dates[i].to_s] = value ? value.to_i : 0
           end
         end
 
@@ -151,6 +173,7 @@ module Sidekiq
         end
       end
     end
+    alias_method :💣, :clear
   end
 
   ##
@@ -209,6 +232,7 @@ module Sidekiq
 
   class SortedEntry < Job
     attr_reader :score
+    attr_reader :parent
 
     def initialize(parent, score, item)
       super(item)
@@ -243,11 +267,11 @@ module Sidekiq
     end
 
     def retry
-      raise "Retry not available on jobs not in the Retry queue." unless item["failed_at"]
+      raise "Retry not available on jobs which have not failed" unless item["failed_at"]
       Sidekiq.redis do |conn|
         results = conn.multi do
-          conn.zrangebyscore('retry', score, score)
-          conn.zremrangebyscore('retry', score, score)
+          conn.zrangebyscore(parent.name, score, score)
+          conn.zremrangebyscore(parent.name, score, score)
         end.first
         results.map do |message|
           msg = Sidekiq.load_json(message)
@@ -272,6 +296,16 @@ module Sidekiq
       Sidekiq.redis {|c| c.zcard(name) }
     end
 
+    def clear
+      Sidekiq.redis do |conn|
+        conn.del(name)
+      end
+    end
+    alias_method :💣, :clear
+  end
+
+  class JobSet < SortedSet
+
     def schedule(timestamp, message)
       Sidekiq.redis do |conn|
         conn.zadd(name, timestamp.to_f.to_s, Sidekiq.dump_json(message))
@@ -349,11 +383,6 @@ module Sidekiq
       end
     end
 
-    def clear
-      Sidekiq.redis do |conn|
-        conn.del(name)
-      end
-    end
   end
 
   ##
@@ -368,7 +397,7 @@ module Sidekiq
   #     retri.args[0] == 'User' &&
   #     retri.args[1] == 'setup_new_subscriber'
   #   end.map(&:delete)
-  class ScheduledSet < SortedSet
+  class ScheduledSet < JobSet
     def initialize
       super 'schedule'
     end
@@ -386,7 +415,7 @@ module Sidekiq
   #     retri.args[0] == 'User' &&
   #     retri.args[1] == 'setup_new_subscriber'
   #   end.map(&:delete)
-  class RetrySet < SortedSet
+  class RetrySet < JobSet
     def initialize
       super 'retry'
     end
@@ -398,6 +427,76 @@ module Sidekiq
     end
   end
 
+  class DeadSet < JobSet
+    def initialize
+      super 'dead'
+    end
+
+    def retry_all
+      while size > 0
+        each(&:retry)
+      end
+    end
+  end
+
+  ##
+  # Enumerates the set of Sidekiq processes which are actively working
+  # right now.  Each process send a heartbeat to Redis every 5 seconds
+  # so this set should be relatively accurate, barring network partitions.
+  #
+  # Yields a hash of data which looks something like this:
+  #
+  # {
+  #   'hostname' => 'app-1.example.com',
+  #   'started_at' => <process start time>,
+  #   'pid' => 12345,
+  #   'tag' => 'myapp'
+  #   'concurrency' => 25,
+  #   'queues' => ['default', 'low'],
+  #   'busy' => 10,
+  #   'beat' => <last heartbeat>,
+  # }
+
+  class ProcessSet
+    include Enumerable
+
+    def each(&block)
+      procs = Sidekiq.redis { |conn| conn.smembers('processes') }
+
+      to_prune = []
+      sorted = procs.sort
+      Sidekiq.redis do |conn|
+        # We're making a tradeoff here between consuming more memory instead of
+        # making more roundtrips to Redis, but if you have hundreds or thousands of workers,
+        # you'll be happier this way
+        result = conn.pipelined do
+          sorted.each do |key|
+            conn.hmget(key, 'info', 'busy', 'beat')
+          end
+        end
+
+        result.each_with_index do |(info, busy, at_s), i|
+          # the hash named key has an expiry of 60 seconds.
+          # if it's not found, that means the process has not reported
+          # in to Redis and probably died.
+          (to_prune << sorted[i]; next) if info.nil?
+          hash = Sidekiq.load_json(info)
+          yield hash.merge('busy' => busy.to_i, 'beat' => at_s.to_f)
+        end
+      end
+
+      Sidekiq.redis {|conn| conn.srem('processes', to_prune) } unless to_prune.empty?
+      nil
+    end
+
+    # This method is not guaranteed accurate since it does not prune the set
+    # based on current heartbeat.  #each does that and ensures the set only
+    # contains Sidekiq processes which have sent a heartbeat within the last
+    # 60 seconds.
+    def size
+      Sidekiq.redis { |conn| conn.scard('processes') }
+    end
+  end
 
   ##
   # Programmatic access to the current active worker set.
@@ -410,62 +509,50 @@ module Sidekiq
   #
   #    workers = Sidekiq::Workers.new
   #    workers.size => 2
-  #    workers.each do |name, work, started_at|
-  #      # name is a unique identifier per worker
+  #    workers.each do |process_id, thread_id, work|
+  #      # process_id is a unique identifier per Sidekiq process
+  #      # thread_id is a unique identifier per thread
   #      # work is a Hash which looks like:
   #      # { 'queue' => name, 'run_at' => timestamp, 'payload' => msg }
-  #      # started_at is a String rep of the time when the worker started working on the job
+  #      # run_at is an epoch Integer.
   #    end
-
+  #
   class Workers
     include Enumerable
 
     def each(&block)
       Sidekiq.redis do |conn|
-        workers = conn.smembers("workers")
-        workers.each do |w|
-          json = conn.get("worker:#{w}")
-          next unless json
-          msg = Sidekiq.load_json(json)
-          block.call(w, msg, Time.at(msg['run_at']).to_s)
+        procs = conn.smembers('processes')
+        procs.sort.each do |key|
+          valid, workers = conn.pipelined do
+            conn.exists(key)
+            conn.hgetall("#{key}:workers")
+          end
+          next unless valid
+          workers.each_pair do |tid, json|
+            yield key, tid, Sidekiq.load_json(json)
+          end
         end
       end
     end
 
+    # Note that #size is only as accurate as Sidekiq's heartbeat,
+    # which happens every 5 seconds.  It is NOT real-time.
+    #
+    # Not very efficient if you have lots of Sidekiq
+    # processes but the alternative is a global counter
+    # which can easily get out of sync with crashy processes.
     def size
       Sidekiq.redis do |conn|
-        conn.scard("workers")
-      end.to_i
-    end
+        procs = conn.smembers('processes')
+        return 0 if procs.empty?
 
-    # Prune old worker entries from the Busy set.  Worker entries
-    # can be orphaned if Sidekiq hard crashes while processing jobs.
-    # Default is to delete worker entries older than one hour.
-    #
-    # Returns the number of records removed.
-    def prune(older_than=60*60)
-      to_rem = []
-      Sidekiq.redis do |conn|
-        conn.smembers('workers').each do |w|
-          msg = conn.get("worker:#{w}")
-          if !msg
-            to_rem << w
-          else
-            m = Sidekiq.load_json(msg)
-            run_at = Time.at(m['run_at'])
-            # prune jobs older than one hour
-            if run_at < (Time.now - older_than)
-              to_rem << w
-            else
-            end
+        conn.pipelined do
+          procs.each do |key|
+            conn.hget(key, 'busy')
           end
-        end
-      end
-
-      if to_rem.size > 0
-        Sidekiq.redis { |conn| conn.srem('workers', to_rem) }
+        end.map(&:to_i).inject(:+)
       end
-      to_rem.size
     end
   end