sidekiq 0.11.1 → 0.11.2

data/Changes.md

@@ -1,4 +1,15 @@
-HEAD
+0.11.2
+-----------
+
+- Implement "safe shutdown".  The messages for any workers that
+  are stil busy when we hit the TERM timeout will be requeued in
+  Redis so the messages are not lost when the Sidekiq process exits.
+  [#110]
+- Work around Celluloid's small 4kb stack limit [#115]
+- Add support for a custom Capistrano role to limit Sidekiq to
+  a set of machines. [#113]
+
+0.11.1
 -----------
 
 - Fix fetch breaking retry when used with Redis namespaces. [#109]
@@ -13,10 +24,7 @@ HEAD
   As a side effect of this change, the client API works on Ruby 1.8.
   It's not officially supported but should work [#103]
 - NO POLL!  Sidekiq no longer polls Redis, leading to lower network
-  utilization and lower latency for message processing.  As a side
-  effect of this change, queue weights are no longer supported. If you
-  wish to process multiple queues, list them in the order you want
-  them processed: `sidekiq -q critical -q high -q default -q low`
+  utilization and lower latency for message processing.
 - Add --version CLI option
 
 0.10.1

data/README.md

@@ -3,26 +3,28 @@ Sidekiq
 
 Simple, efficient message processing for Ruby.
 
+Sidekiq uses threads to handle many messages at the same time in the
+same process.  It integrates tightly with Rails 3 to make background
+message processing dead simple.
+
 Sidekiq is compatible with Resque.  It uses the exact same
 message format as Resque so it can integrate into an existing Resque processing farm.
 You can have Sidekiq and Resque run side-by-side at the same time and
 use the Resque client to enqueue messages in Redis to be processed by Sidekiq.
 
-At the same time, Sidekiq uses multithreading so it much more memory efficient than Resque (which forks a new process for every job).
-You'll find that you might need 50 200MB resque processes to peg your CPU
-whereas one 300MB Sidekiq process will peg the same CPU and perform the
-same amount of work.  Please see [my blog post on Resque's memory
+At the same time, Sidekiq uses multithreading so it much more memory efficient than
+Resque (which forks a new process for every job).  You'll find that you might need
+50 200MB resque processes to peg your CPU whereas one 300MB Sidekiq process will peg
+the same CPU and perform the same amount of work.  Please see [my blog post on Resque's memory
 efficiency](http://blog.carbonfive.com/2011/09/16/improving-resques-memory-efficiency/)
  and how I was able to shrink a Carbon Five client's resque processing farm
 from 9 machines to 1 machine.
 
-In sum, if your jobs are well-behaved and threadsafe, Sidekiq is probably a good replacement for Resque. If your jobs are not thread-safe or they leak memory, you may want to continue using Resque, because its forking model gives you more protection.
-
 
 Requirements
 -----------------
 
-I test on Ruby 1.9.3 and JRuby 1.6.5 in 1.9 mode.  Other versions/VMs are
+I test on Ruby 1.9.3 and JRuby 1.6.x in 1.9 mode.  Other versions/VMs are
 untested but I will do my best to support them.  Ruby 1.8 is not supported.
 
 
@@ -57,4 +59,3 @@ Author
 -----------------
 
 Mike Perham, [@mperham](https://twitter.com/mperham), [http://mikeperham.com](http://mikeperham.com)
-

data/lib/sidekiq.rb

@@ -15,7 +15,7 @@ module Sidekiq
     :concurrency => 25,
     :require => '.',
     :environment => nil,
-    :timeout => 5,
+    :timeout => 8,
     :enable_rails_extensions => true,
   }
 

data/lib/sidekiq/capistrano.rb

@@ -3,27 +3,28 @@ Capistrano::Configuration.instance.load do
   after "deploy", "sidekiq:restart"
 
   _cset(:sidekiq_timeout) { 10 }
+  _cset(:sidekiq_role) { :app }
 
   namespace :sidekiq do
 
     desc "Quiet sidekiq (stop accepting new work)"
-    task :quiet do
+    task :quiet, :roles => lambda { fetch(:sidekiq_role) } do
       run "cd #{current_path} && if [ -f #{current_path}/tmp/pids/sidekiq.pid ]; then bundle exec sidekiqctl quiet #{current_path}/tmp/pids/sidekiq.pid ; fi"
     end
 
     desc "Stop sidekiq"
-    task :stop do
+    task :stop, :roles => lambda { fetch(:sidekiq_role) } do
       run "cd #{current_path} && if [ -f #{current_path}/tmp/pids/sidekiq.pid ]; then bundle exec sidekiqctl stop #{current_path}/tmp/pids/sidekiq.pid #{fetch :sidekiq_timeout} ; fi"
     end
 
     desc "Start sidekiq"
-    task :start do
+    task :start, :roles => lambda { fetch(:sidekiq_role) } do
       rails_env = fetch(:rails_env, "production")
       run "cd #{current_path} ; nohup bundle exec sidekiq -e #{rails_env} -C #{current_path}/config/sidekiq.yml -P #{current_path}/tmp/pids/sidekiq.pid >> #{current_path}/log/sidekiq.log 2>&1 &"
     end
 
     desc "Restart sidekiq"
-    task :restart do
+    task :restart, :roles => lambda { fetch(:sidekiq_role) } do
       stop
       start
     end

data/lib/sidekiq/cli.rb

@@ -67,6 +67,7 @@ module Sidekiq
         poller.terminate
         @manager.stop!(:shutdown => true, :timeout => options[:timeout])
         @manager.wait(:shutdown)
+        exit(0)
       end
     end
 

data/lib/sidekiq/fetch.rb

@@ -16,7 +16,7 @@ module Sidekiq
     def initialize(mgr, queues)
       @mgr = mgr
       @queues = queues.map { |q| "queue:#{q}" }
-      @num_queues = queues.uniq.size
+      @unique_queues = @queues.uniq
     end
 
     # Fetching is straightforward: the Manager makes a fetch
@@ -49,9 +49,9 @@ module Sidekiq
     # recreate the queue command each time we invoke Redis#blpop
     # to honor weights and avoid queue starvation.
     def queues_cmd
-      cmd = @queues.sample(@num_queues)
-      cmd << TIMEOUT
-      cmd
+      queues = @queues.sample(@unique_queues.size).uniq
+      queues.concat(@unique_queues - queues)
+      queues << TIMEOUT
     end
   end
 end

data/lib/sidekiq/manager.rb

@@ -25,6 +25,7 @@ module Sidekiq
       @count = options[:concurrency] || 25
       @done_callback = nil
 
+      @in_progress = {}
       @done = false
       @busy = []
       @fetcher = Fetcher.new(current_actor, options[:queues])
@@ -39,29 +40,22 @@ module Sidekiq
         @done = true
 
         @fetcher.terminate if @fetcher.alive?
+
+        logger.info { "Shutting down #{@ready.size} quiet workers" }
         @ready.each { |x| x.terminate if x.alive? }
         @ready.clear
 
-        redis do |conn|
+        logger.debug { "Clearing workers in redis" }
+        Sidekiq.redis do |conn|
           workers = conn.smembers('workers')
           workers.each do |name|
             conn.srem('workers', name) if name =~ /:#{process_id}-/
           end
         end
 
-        if shutdown
-          if @busy.empty?
-            # after(0) needed to avoid deadlock in Celluoid after USR1 + TERM
-            return after(0) { signal(:shutdown) }
-          else
-            logger.info { "Pausing #{timeout} seconds to allow workers to finish..." }
-          end
-
-          after(timeout) do
-            @busy.each { |x| x.terminate if x.alive? }
-            signal(:shutdown)
-          end
-        end
+        return after(0) { signal(:shutdown) } if @busy.empty?
+        logger.info { "Pausing up to #{timeout} seconds to allow workers to finish..." }
+        hard_shutdown_in(timeout) if shutdown
       end
     end
 
@@ -76,6 +70,7 @@ module Sidekiq
     def processor_done(processor)
       watchdog('Manager#processor_done died') do
         @done_callback.call(processor) if @done_callback
+        @in_progress.delete(processor.object_id)
         @busy.delete(processor)
         if stopped?
           processor.terminate if processor.alive?
@@ -89,6 +84,7 @@ module Sidekiq
 
     def processor_died(processor, reason)
       watchdog("Manager#processor_died died") do
+        @in_progress.delete(processor.object_id)
         @busy.delete(processor)
 
         unless stopped?
@@ -103,6 +99,7 @@ module Sidekiq
     def assign(msg, queue)
       watchdog("Manager#assign died") do
         processor = @ready.pop
+        @in_progress[processor.object_id] = [msg, queue]
         @busy << processor
         processor.process!(MultiJson.decode(msg), queue)
       end
@@ -110,6 +107,29 @@ module Sidekiq
 
     private
 
+    def hard_shutdown_in(delay)
+      watchdog("Manager#watch_for_shutdown died") do
+        after(delay) do
+          # We've reached the timeout and we still have busy workers.
+          # They must die but their messages shall live on.
+          logger.info("Still waiting for #{@busy.size} busy workers")
+
+          Sidekiq.redis do |conn|
+            @busy.each do |processor|
+              # processor is an actor proxy and we can't call any methods
+              # that would go to the actor (since it's busy).  Instead
+              # we'll use the object_id to track the worker's data here.
+              msg, queue = @in_progress[processor.object_id]
+              conn.lpush("queue:#{queue}", msg)
+            end
+          end
+          logger.info("Pushed #{@busy.size} messages back to Redis")
+
+          after(0) { signal(:shutdown) }
+        end
+      end
+    end
+
     def dispatch
       return if stopped?
       # This is a safety check to ensure we haven't leaked

data/lib/sidekiq/processor.rb

@@ -30,9 +30,11 @@ module Sidekiq
     def process(msg, queue)
       klass  = constantize(msg['class'])
       worker = klass.new
-      stats(worker, msg, queue) do
-        Sidekiq.server_middleware.invoke(worker, msg, queue) do
-          worker.perform(*msg['args'])
+      defer do
+        stats(worker, msg, queue) do
+          Sidekiq.server_middleware.invoke(worker, msg, queue) do
+            worker.perform(*msg['args'])
+          end
         end
       end
       @boss.processor_done!(current_actor)

data/lib/sidekiq/redis_connection.rb

@@ -6,7 +6,9 @@ module Sidekiq
   class RedisConnection
     def self.create(options={})
       url = options[:url] || ENV['REDISTOGO_URL'] || 'redis://localhost:6379/0'
-      size = (options[:size] || Sidekiq.options[:concurrency] || 25)
+      # need a connection for Fetcher and Retry
+      size = options[:size] || (Sidekiq.options[:concurrency] + 2)
+
       ConnectionPool.new(:timeout => 1, :size => size) do
         build_client(url, options[:namespace])
       end

data/lib/sidekiq/testing.rb

@@ -19,6 +19,10 @@ module Sidekiq
     #   assert_equal 1, HardWorker.jobs.size
     #   assert_equal :something, HardWorker.jobs[0]['args'][0]
     #
+    #   assert_equal 0, Sidekiq::Extensions::DelayedMailer.jobs.size
+    #   MyMailer.delayed.send_welcome_email('foo@example.com')
+    #   assert_equal 1, Sidekiq::Extensions::DelayedMailer.jobs.size
+    #
     module ClassMethods
       alias_method :perform_async_old, :perform_async
       def perform_async(*args)

data/lib/sidekiq/version.rb

@@ -1,3 +1,3 @@
 module Sidekiq
-  VERSION = "0.11.1"
+  VERSION = "0.11.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq
 version: !ruby/object:Gem::Version
-  version: 0.11.1
+  version: 0.11.2
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-04-04 00:00:00.000000000 Z
+date: 2012-04-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
-  requirement: &70115690910620 !ruby/object:Gem::Requirement
+  requirement: &70143968211720 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70115690910620
+  version_requirements: *70143968211720
 - !ruby/object:Gem::Dependency
   name: redis-namespace
-  requirement: &70115690910180 !ruby/object:Gem::Requirement
+  requirement: &70143968227640 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70115690910180
+  version_requirements: *70143968227640
 - !ruby/object:Gem::Dependency
   name: connection_pool
-  requirement: &70115690926040 !ruby/object:Gem::Requirement
+  requirement: &70143968227140 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -43,10 +43,10 @@ dependencies:
         version: 0.9.0
   type: :runtime
   prerelease: false
-  version_requirements: *70115690926040
+  version_requirements: *70143968227140
 - !ruby/object:Gem::Dependency
   name: celluloid
-  requirement: &70115690925540 !ruby/object:Gem::Requirement
+  requirement: &70143968226640 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -54,10 +54,10 @@ dependencies:
         version: 0.10.0
   type: :runtime
   prerelease: false
-  version_requirements: *70115690925540
+  version_requirements: *70143968226640
 - !ruby/object:Gem::Dependency
   name: multi_json
-  requirement: &70115690925160 !ruby/object:Gem::Requirement
+  requirement: &70143968226260 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70115690925160
+  version_requirements: *70143968226260
 - !ruby/object:Gem::Dependency
   name: minitest
-  requirement: &70115690924700 !ruby/object:Gem::Requirement
+  requirement: &70143968225800 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,10 +76,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70115690924700
+  version_requirements: *70143968225800
 - !ruby/object:Gem::Dependency
   name: sinatra
-  requirement: &70115690924280 !ruby/object:Gem::Requirement
+  requirement: &70143968225380 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -87,10 +87,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70115690924280
+  version_requirements: *70143968225380
 - !ruby/object:Gem::Dependency
   name: slim
-  requirement: &70115690923860 !ruby/object:Gem::Requirement
+  requirement: &70143968224960 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -98,10 +98,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70115690923860
+  version_requirements: *70143968224960
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70115690923440 !ruby/object:Gem::Requirement
+  requirement: &70143968224540 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -109,10 +109,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70115690923440
+  version_requirements: *70143968224540
 - !ruby/object:Gem::Dependency
   name: actionmailer
-  requirement: &70115690922940 !ruby/object:Gem::Requirement
+  requirement: &70143968224040 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -120,10 +120,10 @@ dependencies:
         version: '3'
   type: :development
   prerelease: false
-  version_requirements: *70115690922940
+  version_requirements: *70143968224040
 - !ruby/object:Gem::Dependency
   name: activerecord
-  requirement: &70115690922440 !ruby/object:Gem::Requirement
+  requirement: &70143968223540 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
@@ -131,7 +131,7 @@ dependencies:
         version: '3'
   type: :development
   prerelease: false
-  version_requirements: *70115690922440
+  version_requirements: *70143968223540
 description: Simple, efficient message processing for Ruby
 email:
 - mperham@gmail.com