sidekiq 3.3.0 → 3.3.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: dea6136861d61b466dbeec68ffc57022140dddeb
-  data.tar.gz: a8eae3c0c21c3006571739e68f5978bffd1312c6
+  metadata.gz: 5ebae4a31efb6367e41c8b9a808252448ed678bb
+  data.tar.gz: f32133f60e2b52b83653b83a0bb034a9fb649ec6
 SHA512:
-  metadata.gz: a5c6decc852e45640a6b9526b38f1457fb75fe5fb439f53f0d30b8108f50c4817e20398833188ebfdbabc0f88e15f362654256f1ef3457f7ac2554ce083ff87a
-  data.tar.gz: 692dbde9798b2bca0d0db7c62cd47912ba43e6422ba2d159009715a904c9ca277875d69b7271aeefbc01be4f3c23e7a2ca7098a76843c97b027a31d46a34300b
+  metadata.gz: d439e1b3ac4a44c5f9d59a91289041241eb66659952b80cf10f1bce3c5a32b3d7972821e62430a21ab4a4ec20fdf0a6546dbbe5a127437a24e24355381cc6f39
+  data.tar.gz: 4d199feb0af0d1c099f137f743b38ec56cd98110d10a1b8d2bad4dc6cbdc7afe1057a368ef7079eb92582edb9518a3584e4e1436a420419a1150a3b6510a1bf8

data/.travis.yml

@@ -1,18 +1,16 @@
 language: ruby
+sudo: false
+cache: bundler
 services:
   - redis-server
 rvm:
-  - jruby-19mode
-  - rbx-2
-  - 2.0.0
   - 2.1
+  - 2.0.0
   - 2.2
+  - jruby-19mode
+  - jruby
+  - rbx-2
 matrix:
-  global:
-    - BUNDLE_JOBS=4
   allow_failures:
     - rvm: rbx-2
-before_install:
-  # Only use 1 job until Travis fixes the rbx --jobs issue.
-  - if [ "$TRAVIS_RUBY_VERSION" == "rbx-2" ] ; then export BUNDLE_JOBS=1 ; fi
-install: bundle install --retry=3
+    - rvm: jruby

data/Changes.md

@@ -1,3 +1,19 @@
+3.3.1
+-----------
+
+- Dumb down ActionMailer integration so it tries to deliver if possible [#2149]
+- Stringify Sidekiq.default_worker_options's keys [#2126]
+- Add random integer to process identity [#2113, michaeldiscala]
+- Log Sidekiq Pro's Batch ID if available [#2076]
+- Refactor Processor Redis usage to avoid redis/redis-rb#490 [#2094]
+- Move /dashboard/stats to /stats.  Add /stats/queues. [moserke, #2099]
+- Add processes count to /stats [ismaelga, #2141]
+- Greatly improve speed of Sidekiq::Stats [ismaelga, #2142]
+- Add better usage text for `sidekiqctl`.
+- `Sidekiq::Logging.with_context` is now a stack so you can set your
+  own job context for logging purposes [grosser, #2110]
+- Remove usage of Google Fonts in Web UI so it loads in China [#2144]
+
 3.3.0
 -----------
 

data/Pro-2.0-Upgrade.md

@@ -0,0 +1,102 @@
+# Upgrading to Sidekiq Pro 2.0
+
+Sidekiq Pro 2.0 allows nested batches for more complex job workflows.
+It also removes deprecated APIs, changes the batch data format and
+how features are activated.  Read carefully to ensure your upgrade goes
+smoothly.
+
+## Nested Batches
+
+Batches can now be nested within the `jobs` method.
+This feature enables Sidekiq Pro to handle workflow processing of any size
+and complexity!
+
+```ruby
+a = Sidekiq::Batch.new
+a.on(:success, SomeCallback)
+a.jobs do
+  SomeWork.perform_async
+
+  b = Sidekiq::Batch.new
+  b.on(:success, MyCallback)
+  b.jobs do
+    OtherWork.perform_async
+  end
+end
+```
+
+Parent batch callbacks are not processed until any child batch callbacks have
+run successfully.  In the example above, `MyCallback` will always fire
+before `SomeCallback` because `b` is considered a child of `a`.
+
+Of course you can dynamically add child batches while a batch job is executing.
+
+```ruby
+def perform(*args)
+  do_something(args)
+
+  if more_work?
+    # Sidekiq::Worker#batch returns the Batch this job is part of.
+    batch.jobs do
+      b = Sidekiq::Batch.new
+      b.on(:success, MyCallback)
+      b.jobs do
+        OtherWork.perform_async
+      end
+    end
+  end
+end
+```
+
+## Batch Data
+
+The batch data model was overhauled.  Batch data should take
+significantly less space in Redis now.  A simple benchmark shows 25%
+savings but real world savings should be even greater.
+
+* Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
+  "vTF1-9QvLPnREQ".  Batch 1.x BIDs were 16 character hex-encoded
+  strings, e.g. "4a3fc67d30370edf".
+* In 1.x, batch data was not removed until it naturally expired in Redis.
+  In 2.x, all data for a batch is removed from Redis once the batch has
+  run any success callbacks.
+* Because of the former point, batch expiry is no longer a concern.
+  Batch expiry is hardcoded to 30 days and is no longer user-tunable.
+* Failed batch jobs no longer automatically store any associated
+  backtrace in Redis.
+
+**There's no data migration required.  Sidekiq Pro 2.0 transparently handles
+both old and new format.**
+
+**Note that you CANNOT go back to Pro 1.x once you've created batches
+with 2.x.  The new batches will not process correctly with 1.x.**
+
+## Reliability
+
+You no longer need to require anything to use Reliability features.
+
+* Activate reliable fetch:
+```ruby
+Sidekiq.configure_server do |config|
+  config.reliable_fetch!
+end
+```
+* Activate reliable push:
+```ruby
+Sidekiq::Client.reliable_push!
+```
+
+## Other Changes
+
+* You must require `sidekiq/notifications` if you want to use the
+  existing notification schemes.  I don't recommend using them as the
+  newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
+* Several classes have been renamed.  Generally these classes are ones
+  you should not need to require/use in your own code, e.g. the Batch
+  middleware.
+* The Web UI now shows the Sidekiq Pro version in the footer.
+
+## Thanks
+
+Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
+feedback during the release process.  Thank you guys!

data/Pro-Changes.md

@@ -3,6 +3,20 @@ Sidekiq Pro Changelog
 
 Please see [http://sidekiq.org/pro](http://sidekiq.org/pro) for more details and how to buy.
 
+
+2.0.0 (see mperham/sidekiq#2130 for progress)
+-----------
+
+- See [[Pro-2.0-Upgrade.md]] for thorough release notes.
+
+1.9.2
+-----------
+
+- As of 1/1/2015, Sidekiq Pro is hosted on a new dedicated server.
+  Happy new year and let's hope for 100% uptime!
+- Fix bug in reliable\_fetch where jobs could be duplicated if a Sidekiq
+  process crashed and you were using weighted queues. [#2120]
+
 1.9.1
 -----------
 

data/README.md

@@ -24,7 +24,7 @@ the same CPU and perform the same amount of work.
 Requirements
 -----------------
 
-I test with the latest MRI (2.1, 2.0) and JRuby versions (1.7).  Other versions/VMs
+I test with the latest MRI (2.2, 2.1 and 2.0) and JRuby versions (1.7).  Other versions/VMs
 are untested but might work fine.  MRI 1.9 is no longer supported.
 
 All Rails releases starting from 3.2 are officially supported.
@@ -41,7 +41,7 @@ Installation
 Getting Started
 -----------------
 
-See the [sidekiq home page](http://mperham.github.com/sidekiq) for the simple 3-step process.
+See the [sidekiq home page](http://sidekiq.org) for the simple 3-step process.
 You can watch [Railscast #366](http://railscasts.com/episodes/366-sidekiq) to see Sidekiq in action.  If you do everything right, you should see this: 
 
 ![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
@@ -91,4 +91,4 @@ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for
 Author
 -----------------
 
-Mike Perham, [@mperham](https://twitter.com/mperham) / [@sidekiq](https://twitter.com/sidekiq), [http://mikeperham.com](http://mikeperham.com) / [http://www.contribsys.com](http://www.contribsys.com)
+Mike Perham, [@mperham](https://twitter.com/mperham) / [@sidekiq](https://twitter.com/sidekiq), [http://www.mikeperham.com](http://www.mikeperham.com) / [http://www.contribsys.com](http://www.contribsys.com)

data/Rakefile

@@ -1,7 +1,6 @@
 require 'bundler/gem_tasks'
 require 'rake/testtask'
 Rake::TestTask.new(:test) do |test|
-  test.libs << 'test'
   #SO MUCH NOISE
   #test.warning = true
   test.pattern = 'test/**/test_*.rb'

data/bin/sidekiqctl

@@ -3,23 +3,29 @@
 require 'fileutils'
 
 class Sidekiqctl
-  DEFAULT_TIMEOUT = 10
+  DEFAULT_KILL_TIMEOUT = 10
 
-  attr_reader :stage, :pidfile, :timeout
+  attr_reader :stage, :pidfile, :kill_timeout
 
   def self.print_usage
+    puts "#{File.basename($0)} - stop a Sidekiq process from the command line."
     puts
-    puts "Usage: #{File.basename($0)} <command> <pidfile> <timeout>"
-    puts " where <command> is either 'quiet', 'stop' or 'shutdown'"
+    puts "Usage: #{File.basename($0)} <command> <pidfile> <kill_timeout>"
+    puts " where <command> is either 'quiet' or 'stop'"
     puts "       <pidfile> is path to a pidfile"
-    puts "       <timeout> is number of seconds to wait till Sidekiq exits (default: #{Sidekiqctl::DEFAULT_TIMEOUT})"
+    puts "       <kill_timeout> is number of seconds to wait until Sidekiq exits"
+    puts "       (default: #{Sidekiqctl::DEFAULT_KILL_TIMEOUT}), after which Sidekiq will be KILL'd"
+    puts
+    puts "Be sure to set the kill_timeout LONGER than Sidekiq's -t timeout.  If you want"
+    puts "to wait 60 seconds for jobs to finish, use `sidekiq -t 60` and `sidekiqctl stop"
+    puts " path_to_pidfile 61`"
     puts
   end
 
   def initialize(stage, pidfile, timeout)
     @stage = stage
     @pidfile = pidfile
-    @timeout = timeout
+    @kill_timeout = timeout
 
     done('No pidfile given', :error) if !pidfile
     done("Pidfile #{pidfile} does not exist", :warn) if !File.exist?(pidfile)
@@ -30,7 +36,7 @@ class Sidekiqctl
     begin
       send(stage)
     rescue NoMethodError
-      done 'Invalid control command', :error
+      done "Invalid command: #{stage}", :error
     end
   end
 
@@ -59,7 +65,7 @@ class Sidekiqctl
 
   def stop
     `kill -TERM #{pid}`
-    timeout.times do
+    kill_timeout.times do
       begin
         Process.getpgid(pid)
       rescue Errno::ESRCH
@@ -72,11 +78,7 @@ class Sidekiqctl
     FileUtils.rm_f pidfile
     done 'Sidekiq shut down forcefully.'
   end
-
-  def shutdown
-    quiet
-    stop
-  end
+  alias_method :shutdown, :stop
 end
 
 if ARGV.length < 2
@@ -85,7 +87,7 @@ else
   stage = ARGV[0]
   pidfile = ARGV[1]
   timeout = ARGV[2].to_i
-  timeout = Sidekiqctl::DEFAULT_TIMEOUT if timeout == 0
+  timeout = Sidekiqctl::DEFAULT_KILL_TIMEOUT if timeout == 0
 
   Sidekiqctl.new(stage, pidfile, timeout)
 end

data/lib/sidekiq.rb

@@ -10,21 +10,21 @@ require 'sidekiq/redis_connection'
 require 'json'
 
 module Sidekiq
-  NAME = "Sidekiq"
+  NAME = 'Sidekiq'
   LICENSE = 'See LICENSE and the LGPL-3.0 for licensing details.'
 
   DEFAULTS = {
-    :queues => [],
-    :labels => [],
-    :concurrency => 25,
-    :require => '.',
-    :environment => nil,
-    :timeout => 8,
-    :error_handlers => [],
-    :lifecycle_events => {
-      :startup => [],
-      :quiet => [],
-      :shutdown => [],
+    queues: [],
+    labels: [],
+    concurrency: 25,
+    require: '.',
+    environment: nil,
+    timeout: 8,
+    error_handlers: [],
+    lifecycle_events: {
+      startup: [],
+      quiet: [],
+      shutdown: [],
     }
   }
 
@@ -97,7 +97,7 @@ module Sidekiq
   end
 
   def self.default_worker_options=(hash)
-    @default_worker_options = default_worker_options.merge(hash)
+    @default_worker_options = default_worker_options.merge(hash.stringify_keys)
   end
 
   def self.default_worker_options
@@ -146,7 +146,7 @@ module Sidekiq
   #   end
   def self.on(event, &block)
     raise ArgumentError, "Symbols only please: #{event}" unless event.is_a?(Symbol)
-    raise ArgumentError, "Invalid event name: #{event}" unless options[:lifecycle_events].keys.include?(event)
+    raise ArgumentError, "Invalid event name: #{event}" unless options[:lifecycle_events].key?(event)
     options[:lifecycle_events][event] << block
   end
 end

data/lib/sidekiq/api.rb

@@ -3,12 +3,89 @@ require 'sidekiq'
 
 module Sidekiq
   class Stats
+    def initialize
+      fetch_stats!
+    end
+
     def processed
-      Sidekiq.redis { |conn| conn.get("stat:processed") }.to_i
+      stat :processed
     end
 
     def failed
-      Sidekiq.redis { |conn| conn.get("stat:failed") }.to_i
+      stat :failed
+    end
+
+    def scheduled_size
+      stat :scheduled_size
+    end
+
+    def retry_size
+      stat :retry_size
+    end
+
+    def dead_size
+      stat :dead_size
+    end
+
+    def enqueued
+      stat :enqueued
+    end
+
+    def processes_size
+      stat :processes_size
+    end
+
+    def workers_size
+      stat :workers_size
+    end
+
+    def default_queue_latency
+      stat :default_queue_latency
+    end
+
+    def fetch_stats!
+      pipe1_res = Sidekiq.redis do |conn|
+        conn.pipelined do
+          conn.get('stat:processed'.freeze)
+          conn.get('stat:failed'.freeze)
+          conn.zcard('schedule'.freeze)
+          conn.zcard('retry'.freeze)
+          conn.zcard('dead'.freeze)
+          conn.scard('processes'.freeze)
+          conn.lrange('queue:default'.freeze, -1, -1)
+          conn.smembers('processes'.freeze)
+          conn.smembers('queues'.freeze)
+        end
+      end
+
+      pipe2_res = Sidekiq.redis do |conn|
+        conn.pipelined do
+          pipe1_res[7].each {|key| conn.hget(key, 'busy'.freeze) }
+          pipe1_res[8].each {|queue| conn.llen("queue:#{queue}") }
+        end
+      end
+
+      s = pipe1_res[7].size
+      workers_size = pipe2_res[0...s].map(&:to_i).inject(0, &:+)
+      enqueued     = pipe2_res[s..-1].map(&:to_i).inject(0, &:+)
+
+      default_queue_latency = if (entry = pipe1_res[6].first)
+                                Time.now.to_f - Sidekiq.load_json(entry)['enqueued_at'.freeze]
+                              else
+                                0
+                              end
+      @stats = {
+        processed:             pipe1_res[0].to_i,
+        failed:                pipe1_res[1].to_i,
+        scheduled_size:        pipe1_res[2],
+        retry_size:            pipe1_res[3],
+        dead_size:             pipe1_res[4],
+        processes_size:        pipe1_res[5],
+
+        default_queue_latency: default_queue_latency,
+        workers_size:          workers_size,
+        enqueued:              enqueued
+      }
     end
 
     def reset(*stats)
@@ -25,41 +102,33 @@ module Sidekiq
       end
     end
 
-    def queues
-      Sidekiq.redis do |conn|
-        queues = conn.smembers('queues')
-
-        lengths = conn.pipelined do
-          queues.each do |queue|
-            conn.llen("queue:#{queue}")
-          end
-        end
-
-        i = 0
-        array_of_arrays = queues.inject({}) do |memo, queue|
-          memo[queue] = lengths[i]
-          i += 1
-          memo
-        end.sort_by { |_, size| size }
+    private
 
-        Hash[array_of_arrays.reverse]
-      end
+    def stat(s)
+      @stats[s]
     end
 
-    def enqueued
-      queues.values.inject(&:+) || 0
-    end
+    class Queues
+      def lengths
+        Sidekiq.redis do |conn|
+          queues = conn.smembers('queues'.freeze)
 
-    def scheduled_size
-      Sidekiq.redis {|c| c.zcard('schedule') }
-    end
+          lengths = conn.pipelined do
+            queues.each do |queue|
+              conn.llen("queue:#{queue}")
+            end
+          end
 
-    def retry_size
-      Sidekiq.redis {|c| c.zcard('retry') }
-    end
+          i = 0
+          array_of_arrays = queues.inject({}) do |memo, queue|
+            memo[queue] = lengths[i]
+            i += 1
+            memo
+          end.sort_by { |_, size| size }
 
-    def dead_size
-      Sidekiq.redis {|c| c.zcard('dead') }
+          Hash[array_of_arrays.reverse]
+        end
+      end
     end
 
     class History
@@ -92,8 +161,8 @@ module Sidekiq
         end
 
         Sidekiq.redis do |conn|
-          conn.mget(keys).each_with_index do |value, i|
-            stat_hash[dates[i].to_s] = value ? value.to_i : 0
+          conn.mget(keys).each_with_index do |value, idx|
+            stat_hash[dates[idx].to_s] = value ? value.to_i : 0
           end
         end
 
@@ -118,7 +187,7 @@ module Sidekiq
     include Enumerable
 
     def self.all
-      Sidekiq.redis {|c| c.smembers('queues') }.sort.map {|q| Sidekiq::Queue.new(q) }
+      Sidekiq.redis {|c| c.smembers('queues'.freeze) }.sort.map {|q| Sidekiq::Queue.new(q) }
     end
 
     attr_reader :name
@@ -174,7 +243,7 @@ module Sidekiq
       Sidekiq.redis do |conn|
         conn.multi do
           conn.del(@rname)
-          conn.srem("queues", name)
+          conn.srem("queues".freeze, name)
         end
       end
     end
@@ -267,7 +336,7 @@ module Sidekiq
 
     def safe_load(content, default)
       begin
-        yield *YAML.load(content)
+        yield(*YAML.load(content))
       rescue ::ArgumentError => ex
         # #1761 in dev mode, it's possible to have jobs enqueued which haven't been loaded into
         # memory yet so the YAML can't be loaded.
@@ -382,7 +451,7 @@ module Sidekiq
     end
 
     def size
-      Sidekiq.redis {|c| c.zcard(name) }
+      Sidekiq.redis { |c| c.zcard(name) }
     end
 
     def clear
@@ -411,7 +480,7 @@ module Sidekiq
         range_start = page * page_size + offset_size
         range_end   = page * page_size + offset_size + (page_size - 1)
         elements = Sidekiq.redis do |conn|
-          conn.zrange name, range_start, range_end, :with_scores => true
+          conn.zrange name, range_start, range_end, with_scores: true
         end
         break if elements.empty?
         page -= 1
@@ -586,7 +655,7 @@ module Sidekiq
           end
         end
 
-        result.each_with_index do |(info, busy, at_s), i|
+        result.each do |info, busy, at_s|
           hash = Sidekiq.load_json(info)
           yield Process.new(hash.merge('busy' => busy.to_i, 'beat' => at_s.to_f))
         end
@@ -616,6 +685,7 @@ module Sidekiq
   #   'queues' => ['default', 'low'],
   #   'busy' => 10,
   #   'beat' => <last heartbeat>,
+  #   'identity' => <unique string identifying the process>,
   # }
   class Process
     def initialize(hash)
@@ -655,7 +725,7 @@ module Sidekiq
     end
 
     def identity
-      @id ||= "#{self['hostname']}:#{self['pid']}"
+      self['identity']
     end
   end