sidekiq 2.9.0 → 2.10.0

data/.gitignore

@@ -1,4 +1,5 @@
 .rvmrc
+.ruby-version
 tags
 Gemfile.lock
 *.swp

data/Changes.md

@@ -1,3 +1,15 @@
+2.10.0
+-----------
+
+- Refactor algorithm for putting scheduled jobs onto the queue [#843]
+- Fix scheduler thread dying due to incorrect error handling [#839]
+- Fix issue which left stale workers if Sidekiq wasn't shutdown while
+quiet. [#840]
+- I18n for web UI.  Please submit translations of `web/locales/en.yml` for
+your own language. [#811]
+- 'sinatra', 'slim' and 'i18n' are now gem dependencies for Sidekiq.
+
+
 2.9.0
 -----------
 

data/README.md

@@ -1,10 +1,7 @@
 Sidekiq
 ==============
 
-  - [![Gem Version](https://badge.fury.io/rb/sidekiq.png)](https://rubygems.org/gems/sidekiq)
-  - [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.png)](https://codeclimate.com/github/mperham/sidekiq)
-  - [![Build Status](https://travis-ci.org/mperham/sidekiq.png)](https://travis-ci.org/mperham/sidekiq)
-  - [![Coverage Status](https://coveralls.io/repos/mperham/sidekiq/badge.png?branch=master)](https://coveralls.io/r/mperham/sidekiq)
+[![Gem Version](https://badge.fury.io/rb/sidekiq.png)](https://rubygems.org/gems/sidekiq) [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.png)](https://codeclimate.com/github/mperham/sidekiq) [![Build Status](https://travis-ci.org/mperham/sidekiq.png)](https://travis-ci.org/mperham/sidekiq) [![Coverage Status](https://coveralls.io/repos/mperham/sidekiq/badge.png?branch=master)](https://coveralls.io/r/mperham/sidekiq)
 
 
 Simple, efficient message processing for Ruby.
@@ -21,16 +18,13 @@ use the Resque client to enqueue messages in Redis to be processed by Sidekiq.
 At the same time, Sidekiq uses multithreading so it is much more memory efficient than
 Resque (which forks a new process for every job).  You'll find that you might need
 50 200MB resque processes to peg your CPU whereas one 300MB Sidekiq process will peg
-the same CPU and perform the same amount of work.  Please see [my blog post on Resque's memory
-efficiency](http://blog.carbonfive.com/2011/09/16/improving-resques-memory-efficiency/)
- and how I was able to shrink a Carbon Five client's resque processing farm
-from 9 machines to 1 machine.
+the same CPU and perform the same amount of work.
 
 
 Requirements
 -----------------
 
-I test on Ruby 1.9.3 and JRuby 1.6.x in 1.9 mode.  Other versions/VMs are
+I test on Ruby 1.9.3 and JRuby 1.7.x.  Other versions/VMs are
 untested but I will do my best to support them.  Ruby 1.8 is not supported.
 
 Redis 2.0 or greater is required.
@@ -45,12 +39,20 @@ Installation
 Getting Started
 -----------------
 
-See the [sidekiq home page](http://mperham.github.com/sidekiq) for the simple 4-step process.
+See the [sidekiq home page](http://mperham.github.com/sidekiq) for the simple 3-step process.
 You can watch [Railscast #366](http://railscasts.com/episodes/366-sidekiq) to see Sidekiq in action.  If you do everything right, you should see this: 
 
 ![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
 
 
+Want to Upgrade?
+-------------------
+
+I also sell Sidekiq Pro, an extension to Sidekiq which provides more
+features, a commercial-friendly license and allows you to support high
+quality open source development all at the same time.  Please see the
+[Sidekiq Pro](http://sidekiq.org/pro) homepage for more detail.
+
 
 More Information
 -----------------

data/lib/sidekiq/manager.rb

@@ -42,8 +42,9 @@ module Sidekiq
         @ready.each { |x| x.terminate if x.alive? }
         @ready.clear
 
+        clear_worker_set
+
         return after(0) { signal(:shutdown) } if @busy.empty?
-        logger.info { "Pausing up to #{timeout} seconds to allow workers to finish..." }
         hard_shutdown_in timeout if shutdown
       end
     end
@@ -108,7 +109,24 @@ module Sidekiq
 
     private
 
+    def clear_worker_set
+      # Clearing workers in Redis
+      # NOTE: we do this before terminating worker threads because the
+      # process will likely receive a hard shutdown soon anyway, which
+      # means the threads will killed.
+      logger.debug { "Clearing workers in redis" }
+      Sidekiq.redis do |conn|
+        workers = conn.smembers('workers')
+        workers_to_remove = workers.select do |worker_name|
+          worker_name =~ /:#{process_id}-/
+        end
+        conn.srem('workers', workers_to_remove) if !workers_to_remove.empty?
+      end
+    end
+
     def hard_shutdown_in(delay)
+      logger.info { "Pausing up to #{delay} seconds to allow workers to finish..." }
+
       after(delay) do
         watchdog("Manager#hard_shutdown_in died") do
           # We've reached the timeout and we still have busy workers.
@@ -123,19 +141,6 @@ module Sidekiq
           # it is worse to lose a job than to run it twice.
           Sidekiq::Fetcher.strategy.bulk_requeue(@in_progress.values)
 
-          # Clearing workers in Redis
-          # NOTE: we do this before terminating worker threads because the
-          # process will likely receive a hard shutdown soon anyway, which
-          # means the threads will killed.
-          logger.debug { "Clearing workers in redis" }
-          Sidekiq.redis do |conn|
-            workers = conn.smembers('workers')
-            workers_to_remove = workers.select do |worker_name|
-              worker_name =~ /:#{process_id}-/
-            end
-            conn.srem('workers', workers_to_remove) if !workers_to_remove.empty?
-          end
-
           logger.debug { "Terminating worker threads" }
           @busy.each do |processor|
             t = processor.bare_object.actual_work_thread

data/lib/sidekiq/scheduled.rb

@@ -28,25 +28,30 @@ module Sidekiq
             now = Time.now.to_f.to_s
             Sidekiq.redis do |conn|
               SETS.each do |sorted_set|
-                (messages, _) = conn.multi do
-                  conn.zrangebyscore(sorted_set, '-inf', now)
-                  conn.zremrangebyscore(sorted_set, '-inf', now)
-                end
-
-                messages.each do |message|
-                  logger.debug { "enqueued #{sorted_set}: #{message}" }
+                # Get the next item in the queue if it's score (time to execute) is <= now.
+                # We need to go through the list one at a time to reduce the risk of something
+                # going wrong between the time jobs are popped from the scheduled queue and when
+                # they are pushed onto a work queue and losing the jobs.
+                while message = conn.zrangebyscore(sorted_set, '-inf', now, :limit => [0, 1]).first do
                   msg = Sidekiq.load_json(message)
-                  conn.multi do
-                    conn.sadd('queues', msg['queue'])
-                    conn.rpush("queue:#{msg['queue']}", message)
+                  # Pop item off the queue and add it to the work queue. If the job can't be popped from
+                  # the queue, it's because another process already popped it so we can move on to the
+                  # next one.
+                  if conn.zrem(sorted_set, message)
+                    conn.multi do
+                      conn.sadd('queues', msg['queue'])
+                      conn.rpush("queue:#{msg['queue']}", message)
+                    end
+                    logger.debug("enqueued #{sorted_set}: #{message}") if logger.debug?
                   end
                 end
               end
             end
-          rescue SystemCallError, Redis::TimeoutError, Redis::ConnectionError => ex
-            # ECONNREFUSED, etc.  Most likely a problem with
-            # redis networking.  Punt and try again at the next interval
-            logger.warn ex.message
+          rescue => ex
+            # Most likely a problem with redis networking.
+            # Punt and try again at the next interval
+            logger.error ex.message
+            logger.error(ex.backtrace.first)
           end
 
           after(poll_interval) { poll }

data/lib/sidekiq/version.rb

@@ -1,3 +1,3 @@
 module Sidekiq
-  VERSION = "2.9.0"
+  VERSION = "2.10.0"
 end

data/lib/sidekiq/web.rb

@@ -1,18 +1,28 @@
 require 'sinatra/base'
 require 'slim'
 require 'sidekiq/paginator'
+require 'i18n'
 
 module Sidekiq
   class Web < Sinatra::Base
     include Sidekiq::Paginator
 
     dir = File.expand_path(File.dirname(__FILE__) + "/../../web")
+    I18n.load_path += Dir[File.join(dir, 'locales', '*.yml').to_s]
+
     set :public_folder, "#{dir}/assets"
     set :views,  "#{dir}/views"
     set :root, "#{dir}/public"
     set :slim, :pretty => true
 
     helpers do
+      def get_locale
+        (request.env["HTTP_ACCEPT_LANGUAGE"] || 'en')[0,2]
+      end
+
+      def t(msg, options={})
+        I18n.t(msg, options.merge(:locale => get_locale))
+      end
 
       def reset_worker_list
         Sidekiq.redis do |conn|

data/sidekiq.gemspec

@@ -19,9 +19,10 @@ Gem::Specification.new do |gem|
   gem.add_dependency                  'connection_pool', '~> 1.0'
   gem.add_dependency                  'celluloid', '~> 0.12.0'
   gem.add_dependency                  'multi_json', '~> 1'
+  gem.add_dependency                  'sinatra'
+  gem.add_dependency                  'slim'
+  gem.add_dependency                  'i18n'
   gem.add_development_dependency      'minitest', '~> 4'
-  gem.add_development_dependency      'sinatra'
-  gem.add_development_dependency      'slim'
   gem.add_development_dependency      'rake'
   gem.add_development_dependency      'actionmailer', '~> 3'
   gem.add_development_dependency      'activerecord', '~> 3'

data/test/test_retry.rb

@@ -199,26 +199,4 @@ class TestRetry < MiniTest::Unit::TestCase
     end
   end
 
-  describe 'poller' do
-    before do
-      @redis = MiniTest::Mock.new
-      Sidekiq.instance_variable_set(:@redis, @redis)
-
-      def @redis.with; yield self; end
-    end
-
-    it 'should poll like a bad mother...SHUT YO MOUTH' do
-      fake_msg = Sidekiq.dump_json({ 'class' => 'Bob', 'args' => [1,2], 'queue' => 'someq' })
-      @redis.expect :multi, [[fake_msg], 1], []
-      @redis.expect :multi, [[], nil], []
-      @redis.expect :multi, [[], nil], []
-      @redis.expect :multi, [[], nil], []
-
-      inst = Sidekiq::Scheduled::Poller.new
-      inst.poll
-
-      @redis.verify
-    end
-  end
-
 end

data/test/test_scheduled.rb

@@ -0,0 +1,48 @@
+require 'helper'
+require 'sidekiq/scheduled'
+
+class TestScheduled < MiniTest::Unit::TestCase
+  class ScheduledWorker
+    include Sidekiq::Worker
+    def perform(x)
+    end
+  end
+  
+  describe 'poller' do
+    before do
+      Sidekiq.redis = REDIS
+      Sidekiq.redis do |conn|
+        conn.flushdb
+      end
+    end
+
+    it 'should empty the retry and scheduled queues up to the current time' do
+      Sidekiq.redis do |conn|
+        error_1 = Sidekiq.dump_json('class' => ScheduledWorker.name, 'args' => ["error_1"], 'queue' => 'queue_1')
+        error_2 = Sidekiq.dump_json('class' => ScheduledWorker.name, 'args' => ["error_2"], 'queue' => 'queue_2')
+        error_3 = Sidekiq.dump_json('class' => ScheduledWorker.name, 'args' => ["error_3"], 'queue' => 'queue_3')
+        future_1 = Sidekiq.dump_json('class' => ScheduledWorker.name, 'args' => ["future_1"], 'queue' => 'queue_4')
+        future_2 = Sidekiq.dump_json('class' => ScheduledWorker.name, 'args' => ["future_2"], 'queue' => 'queue_5')
+        future_3 = Sidekiq.dump_json('class' => ScheduledWorker.name, 'args' => ["future_3"], 'queue' => 'queue_6')
+
+        conn.zadd("retry", (Time.now - 60).to_f.to_s, error_1)
+        conn.zadd("retry", (Time.now - 50).to_f.to_s, error_2)
+        conn.zadd("retry", (Time.now + 60).to_f.to_s, error_3)
+        conn.zadd("schedule", (Time.now - 60).to_f.to_s, future_1)
+        conn.zadd("schedule", (Time.now - 50).to_f.to_s, future_2)
+        conn.zadd("schedule", (Time.now + 60).to_f.to_s, future_3)
+
+        poller = Sidekiq::Scheduled::Poller.new
+        poller.poll
+        poller.terminate
+        
+        assert_equal [error_1], conn.lrange("queue:queue_1", 0, -1)
+        assert_equal [error_2], conn.lrange("queue:queue_2", 0, -1)
+        assert_equal [error_3], conn.zrange("retry", 0, -1)
+        assert_equal [future_1], conn.lrange("queue:queue_4", 0, -1)
+        assert_equal [future_2], conn.lrange("queue:queue_5", 0, -1)
+        assert_equal [future_3], conn.zrange("schedule", 0, -1)
+      end
+    end
+  end
+end

data/web/locales/da.yml

@@ -0,0 +1,59 @@
+# elements like %{queue} are variables and should not be translated
+da:
+  Dashboard: Instrumentbræt
+  Status: Status
+  Time: Tid
+  Namespace: Namespace
+  Realtime: Real-time
+  History: Historik
+  Busy: Travl
+  Processed: Processeret
+  Failed: Fejlet
+  Scheduled: Planlagt
+  Retries: Forsøg
+  Enqueued: I kø
+  Worker: Arbejder
+  Workers: Arbejdere
+  LivePoll: Live Poll
+  StopPolling: Stop Polling
+  Queue: Kø
+  Class: Klasse
+  Job: Job
+  Arguments: Argumenter
+  Started: Startet
+  ShowAll: Vis alle
+  CurrentMessagesInQueue: Nuværende beskeder i <span class='title'>%{queue}</span>
+  Delete: Slet
+  AreYouSureDeleteJob: Er du sikker på at du vil slette dette job?
+  AreYouSureDeleteQueue: Er du sikker på at du vil slette %{queue} køen?
+  Queues: Køer
+  Size: Størrelse
+  Actions: Actions
+  NextRetry: Næste forsøg
+  RetryCount: Antal forsøg
+  RetryNow: Prøv igen nu
+  LastRetry: Sidste forsøg
+  OriginallyFailed: Oprindeligt fejlet
+  AreYouSure: Er du sikker?
+  DeleteAll: Slet alle
+  RetryAll: Forsøg alle
+  NoRetriesFound: Ingen gen-forsøg var fundet
+  Error: Fejl
+  ErrorClass: Fejl klasse
+  ErrorMessage: Fejl besked
+  ErrorBacktrace: Fejl backtrace
+  GoBack: ← Tilbage
+  NoScheduledFound: Ingen jobs i kø fundet
+  When: Når
+  ScheduledJobs: Jobs i kø
+  idle: idle
+  active: aktiv
+  Version: Version
+  Connections: Forbindelser
+  MemoryUsage: RAM forbrug
+  PeakMemoryUsage: Peak RAM forbrug
+  Uptime: Oppetid (dage)
+  OneWeek: 1 uge
+  OneMonth: 1 måned
+  ThreeMonths: 3 måneder
+  SixMonths: 6 måneder

data/web/locales/de.yml

@@ -0,0 +1,59 @@
+# elements like %{queue} are variables and should not be translated
+de:
+  Dashboard: Dashboard
+  Status: Status
+  Time: Zeit
+  Namespace: Namensraum
+  Realtime: Echtzeit
+  History: Verlauf
+  Busy: Beschäftigt
+  Processed: Verarbeitet
+  Failed: Fehlgeschlagen
+  Scheduled: Geplant
+  Retries: Versuche
+  Enqueued: In der Warteschlange
+  Worker: Arbeiter
+  Workers: Arbeiter
+  LivePoll: Live Poll
+  StopPolling: Abfrage stoppen
+  Queue: Warteschlange
+  Class: Klasse
+  Job: Job
+  Arguments: Argumente
+  Started: Startet
+  ShowAll: Alle anzeigen
+  CurrentMessagesInQueue: Aktuelle Nachrichten in <span class='title'>%{queue}</span>
+  Delete: Löschen
+  AreYouSureDeleteJob: Möchtest du diesen Job wirklich löschen?
+  AreYouSureDeleteQueue: Möchtest du %{queue} wirklich löschen?
+  Queues: Warteschlangen
+  Size: Größe
+  Actions: Aktionen
+  NextRetry: Nächster Versuch
+  RetryCount: Anzahl der Versuche
+  RetryNow: Jetzt erneut versuchen
+  LastRetry: Letzter Versuch
+  OriginallyFailed: Ursprünglich Fehlgeschlagen
+  AreYouSure: Bist du sicher?
+  DeleteAll: Alle löschen
+  RetryAll: Alle erneut versuchen
+  NoRetriesFound: Keine erneuten Versuche gefunden
+  Error: Fehler
+  ErrorClass: Fehler klasse
+  ErrorMessage: Fehler-Nachricht
+  ErrorBacktrace: Fehler-Backtrace
+  GoBack: ← Zurück
+  NoScheduledFound: Keine geplanten Jobs gefunden
+  When: Wann
+  ScheduledJobs: Jobs in Wartschlange
+  idle: inaktiv
+  active: aktiv
+  Version: Version
+  Connections: Verbindungen
+  MemoryUsage: RAM-Nutzung
+  PeakMemoryUsage: Maximale RAM-Nutzung
+  Uptime: Laufzeit
+  OneWeek: 1 Woche
+  OneMonth: 1 Monat
+  ThreeMonths: 3 Monate
+  SixMonths: 6 Monate

data/web/locales/en.yml

@@ -0,0 +1,61 @@
+# elements like %{queue} are variables and should not be translated
+en:
+  Dashboard: Dashboard
+  Status: Status
+  Time: Time
+  Namespace: Namespace
+  Realtime: Real-time
+  History: History
+  Busy: Busy
+  Processed: Processed
+  Failed: Failed
+  Scheduled: Scheduled
+  Retries: Retries
+  Enqueued: Enqueued
+  ClearWorkerList: Clear workers list
+  Worker: Worker
+  Workers: Workers
+  LivePoll: Live Poll
+  StopPolling: Stop Polling
+  Queue: Queue
+  Class: Class
+  Job: Job
+  Arguments: Arguments
+  Started: Started
+  ShowAll: Show All
+  CurrentMessagesInQueue: Current messages in <span class='title'>%{queue}</span>
+  Delete: Delete
+  AreYouSureDeleteJob: Are you sure you want to delete this job?
+  AreYouSureDeleteQueue: Are you sure you want to delete the %{queue} queue?
+  Queues: Queues
+  Size: Size
+  Actions: Actions
+  NextRetry: Next Retry
+  RetryCount: Retry Count
+  RetryNow: Retry Now
+  LastRetry: Last Retry
+  OriginallyFailed: Originally Failed
+  AreYouSure: Are you sure?
+  DeleteAll: Delete All
+  RetryAll: Retry All
+  NoRetriesFound: No retries were found
+  Error: Error
+  ErrorClass: Error Class
+  ErrorMessage: Error Message
+  ErrorBacktrace: Error Backtrace
+  GoBack: ← Back
+  NoScheduledFound: No scheduled jobs were found
+  When: When
+  ScheduledJobs: Scheduled Jobs
+  idle: idle
+  active: active
+  Version: Version
+  Connections: Connections
+  MemoryUsage: Memory Usage
+  PeakMemoryUsage: Peak Memory Usage
+  Uptime: Uptime (days)
+  OneWeek: 1 week
+  OneMonth: 1 month
+  ThreeMonths: 3 months
+  SixMonths: 6 months
+  Batches: Batches