sidekiq 3.3.1 → 3.3.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 5ebae4a31efb6367e41c8b9a808252448ed678bb
-  data.tar.gz: f32133f60e2b52b83653b83a0bb034a9fb649ec6
+  metadata.gz: 66911189b7648f87bf3ce6e11f5917d6b242ddcc
+  data.tar.gz: e21cbc6d3f73a9a25df7f2771248205ba457f453
 SHA512:
-  metadata.gz: d439e1b3ac4a44c5f9d59a91289041241eb66659952b80cf10f1bce3c5a32b3d7972821e62430a21ab4a4ec20fdf0a6546dbbe5a127437a24e24355381cc6f39
-  data.tar.gz: 4d199feb0af0d1c099f137f743b38ec56cd98110d10a1b8d2bad4dc6cbdc7afe1057a368ef7079eb92582edb9518a3584e4e1436a420419a1150a3b6510a1bf8
+  metadata.gz: 16d3851a53e70a270cfd7277817953fa54897dabf0f4f527308bcbcb769d4232d98466bac1477f820472a7cac361326da5563e696481f7a2963471c6cd3119b8
+  data.tar.gz: c687ab3be80e972fd2b98c6d2939c7509506a8516495eabb81d4ae92c6ec939203a28e4bfea99925bd8469f8035f8024010cc6e992aef97e9adc74a256b4b9c4

data/Changes.md

@@ -1,8 +1,15 @@
+3.3.2
+-----------
+
+- Add Sidekiq::Stats#queues back
+- Allows configuration of dead job set size and timeout [#2173, jonhyman]
+- Refactor scheduler enqueuing so Sidekiq Pro can override it. [#2159]
+
 3.3.1
 -----------
 
 - Dumb down ActionMailer integration so it tries to deliver if possible [#2149]
-- Stringify Sidekiq.default_worker_options's keys [#2126]
+- Stringify Sidekiq.default\_worker\_options's keys [#2126]
 - Add random integer to process identity [#2113, michaeldiscala]
 - Log Sidekiq Pro's Batch ID if available [#2076]
 - Refactor Processor Redis usage to avoid redis/redis-rb#490 [#2094]

data/Contributing.md

@@ -1,19 +1,16 @@
 # Contributing
 
-First of all, thank you!  I hope you find it worthwhile to help out with Sidekiq.
-
 ## Issues
 
-When opening an issue, please include a **backtrace** with your error.
+When opening an issue:
 
-## Code
+* include the full **backtrace** with your error
+* include your Sidekiq initializer
+* list versions you are using: Ruby, Rails, Sidekiq, OS, etc.
 
-If you're interested in helping or contributing to Sidekiq, here's
-some known areas which need improvement:
+It's always better to include more info rather than less.
 
-* The Web UI and sidekiq.org website could always use more polish.  If you have an eye for design and an idea for improvement, please open an issue and let us know.
-* The [Sidekiq issue tracker](https://github.com/mperham/sidekiq/issues) usually has a few ideas
-* Your brilliant idea which I haven't listed!
+## Code
 
 It's always best to open an issue before investing a lot of time into a
 fix or new functionality.  Functionality must meet my design goals and
@@ -22,8 +19,8 @@ your idea can best fit into Sidekiq.
 
 ## Legal
 
-By submitting a Pull Request, you disavow any rights or
-claims to any changes submitted to the Sidekiq project and assign the copyright of
+By submitting a Pull Request, you disavow any rights or claims to any changes
+submitted to the Sidekiq project and assign the copyright of
 those changes to Contributed Systems LLC.
 
 If you cannot or do not want to reassign those rights (your employment
@@ -31,5 +28,5 @@ contract for your employer may not allow this), you should not submit a PR.
 Open an issue and someone else can do the work.
 
 This is a legal way of saying "If you submit a PR to us, that code becomes ours".
-99.9% of the time that's what you intend anyways; we hope it doesn't scare you away from contributing!
-
+99.9% of the time that's what you intend anyways; we hope it doesn't scare you
+away from contributing.

data/Pro-2.0-Upgrade.md

@@ -1,10 +1,20 @@
 # Upgrading to Sidekiq Pro 2.0
 
-Sidekiq Pro 2.0 allows nested batches for more complex job workflows.
+Sidekiq Pro 2.0 allows nested batches for more complex job workflows
+and provides a new reliable scheduler which uses Lua to guarantee
+atomicity and much higher performance.
+
 It also removes deprecated APIs, changes the batch data format and
 how features are activated.  Read carefully to ensure your upgrade goes
 smoothly.
 
+Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater.  Redis 2.8 is
+recommended; Redis 2.4 or 2.6 will work but some functionality will not be
+available.
+
+**Note that you CANNOT go back to Pro 1.x once you've created batches
+with 2.x.  The new batches will not process correctly with 1.x.**
+
 ## Nested Batches
 
 Batches can now be nested within the `jobs` method.
@@ -68,17 +78,22 @@ savings but real world savings should be even greater.
 **There's no data migration required.  Sidekiq Pro 2.0 transparently handles
 both old and new format.**
 
-**Note that you CANNOT go back to Pro 1.x once you've created batches
-with 2.x.  The new batches will not process correctly with 1.x.**
-
 ## Reliability
 
-You no longer need to require anything to use Reliability features.
+2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
+scheduled jobs is atomic.  Benchmarks show it 50x faster when enqueuing
+lots of jobs.  **One caveat**: client-side middleware is not executed
+for each job when enqueued with the reliable scheduler.  No Sidekiq or
+Sidekiq Pro functionality is affected by this change but some 3rd party
+plugins might be.
+
+**You no longer require anything to use the Reliability features.**
 
-* Activate reliable fetch:
+* Activate reliable fetch and/or the new reliable scheduler:
 ```ruby
 Sidekiq.configure_server do |config|
   config.reliable_fetch!
+  config.reliable_scheduler!
 end
 ```
 * Activate reliable push:
@@ -88,7 +103,7 @@ Sidekiq::Client.reliable_push!
 
 ## Other Changes
 
-* You must require `sidekiq/notifications` if you want to use the
+* You must require `sidekiq/pro/notifications` if you want to use the
   existing notification schemes.  I don't recommend using them as the
   newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
 * Several classes have been renamed.  Generally these classes are ones

data/Pro-Changes.md

@@ -4,10 +4,10 @@ Sidekiq Pro Changelog
 Please see [http://sidekiq.org/pro](http://sidekiq.org/pro) for more details and how to buy.
 
 
-2.0.0 (see mperham/sidekiq#2130 for progress)
+2.0.0 (unreleased)
 -----------
 
-- See [[Pro-2.0-Upgrade.md]] for thorough release notes.
+- See [the Upgrade Notes](Pro-2.0-Upgrade.md) for detailed notes.
 
 1.9.2
 -----------

data/README.md

@@ -1,7 +1,11 @@
 Sidekiq
 ==============
 
-[![Gem Version](https://badge.fury.io/rb/sidekiq.png)](https://rubygems.org/gems/sidekiq) [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.png)](https://codeclimate.com/github/mperham/sidekiq) [![Build Status](https://travis-ci.org/mperham/sidekiq.png)](https://travis-ci.org/mperham/sidekiq) [![Coverage Status](https://coveralls.io/repos/mperham/sidekiq/badge.png?branch=master)](https://coveralls.io/r/mperham/sidekiq)
+[![Gem Version](https://badge.fury.io/rb/sidekiq.png)](https://rubygems.org/gems/sidekiq) 
+[![Code Climate](https://codeclimate.com/github/mperham/sidekiq.png)](https://codeclimate.com/github/mperham/sidekiq) 
+[![Build Status](https://travis-ci.org/mperham/sidekiq.png)](https://travis-ci.org/mperham/sidekiq) 
+[![Coverage Status](https://coveralls.io/repos/mperham/sidekiq/badge.png?branch=master)](https://coveralls.io/r/mperham/sidekiq) 
+[![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.png)](https://gitter.im/mperham/sidekiq)
 
 
 Simple, efficient background processing for Ruby.
@@ -60,7 +64,7 @@ More Information
 -----------------
 
 Please see the [sidekiq wiki](https://github.com/mperham/sidekiq/wiki) for the official documentation.
-[#sidekiq on irc.freenode.net](irc://irc.freenode.net/#sidekiq) is dedicated to this project,
+[mperham/sidekiq on Gitter](https://gitter.im/mperham/sidekiq) is dedicated to this project,
 but bug reports or feature requests suggestions should still go through [issues on Github](https://github.com/mperham/sidekiq/issues).  Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
 
 You may also find useful a [Google Group](https://groups.google.com/forum/#!forum/sidekiq) dedicated to Sidekiq discussion and [a Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow.
@@ -71,7 +75,7 @@ Problems?
 
 **Please do not directly email any Sidekiq committers with questions or problems.**  A community is best served when discussions are held in public.
 
-If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages. Searching the issues for your problem is also a good idea.  If that doesn't help, feel free to email the Sidekiq mailing list or open a new issue.
+If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages. Searching the issues for your problem is also a good idea.  If that doesn't help, feel free to email the Sidekiq mailing list, chat in Gitter, or open a new issue.
 The mailing list is the preferred place to ask questions on usage. If you are encountering what you think is a bug, please open an issue.
 
 

data/lib/sidekiq.rb

@@ -25,7 +25,9 @@ module Sidekiq
       startup: [],
       quiet: [],
       shutdown: [],
-    }
+    },
+    dead_max_jobs: 10_000,
+    dead_timeout_in_seconds: 180 * 24 * 60 * 60 # 6 months
   }
 
   def self.❨╯°□°❩╯︵┻━┻

data/lib/sidekiq/api.rb

@@ -43,6 +43,10 @@ module Sidekiq
       stat :default_queue_latency
     end
 
+    def queues
+      Sidekiq::Stats::Queues.new.lengths
+    end
+
     def fetch_stats!
       pipe1_res = Sidekiq.redis do |conn|
         conn.pipelined do
@@ -395,8 +399,8 @@ module Sidekiq
         Sidekiq.redis do |conn|
           conn.multi do
             conn.zadd('dead', now, message)
-            conn.zremrangebyscore('dead', '-inf', now - DeadSet::TIMEOUT)
-            conn.zremrangebyrank('dead', 0, - DeadSet::MAX_JOBS)
+            conn.zremrangebyscore('dead', '-inf', now - DeadSet.timeout)
+            conn.zremrangebyrank('dead', 0, - DeadSet.max_jobs)
           end
         end
       end
@@ -589,9 +593,6 @@ module Sidekiq
   # Allows enumeration of dead jobs within Sidekiq.
   #
   class DeadSet < JobSet
-    TIMEOUT = 180 * 24 * 60 * 60 # 6 months
-    MAX_JOBS = 10_000
-
     def initialize
       super 'dead'
     end
@@ -601,6 +602,14 @@ module Sidekiq
         each(&:retry)
       end
     end
+
+    def self.max_jobs
+      Sidekiq.options[:dead_max_jobs]
+    end
+
+    def self.timeout
+      Sidekiq.options[:dead_timeout_in_seconds]
+    end
   end
 
   ##

data/lib/sidekiq/middleware/server/retry_jobs.rb

@@ -156,8 +156,8 @@ module Sidekiq
           Sidekiq.redis do |conn|
             conn.multi do
               conn.zadd('dead', now, payload)
-              conn.zremrangebyscore('dead', '-inf', now - DeadSet::TIMEOUT)
-              conn.zremrangebyrank('dead', 0, -DeadSet::MAX_JOBS)
+              conn.zremrangebyscore('dead', '-inf', now - DeadSet.timeout)
+              conn.zremrangebyrank('dead', 0, -DeadSet.max_jobs)
             end
           end
         end

data/lib/sidekiq/scheduled.rb

@@ -5,8 +5,32 @@ require 'sidekiq/api'
 
 module Sidekiq
   module Scheduled
+    SETS = %w(retry schedule)
 
-    INITIAL_WAIT = 10
+    class Enq
+      def enqueue_jobs(now=Time.now.to_f.to_s, sorted_sets=SETS)
+        # A job's "score" in Redis is the time at which it should be processed.
+        # Just check Redis for the set of jobs with a timestamp before now.
+        Sidekiq.redis do |conn|
+          sorted_sets.each do |sorted_set|
+            # Get the next item in the queue if it's score (time to execute) is <= now.
+            # We need to go through the list one at a time to reduce the risk of something
+            # going wrong between the time jobs are popped from the scheduled queue and when
+            # they are pushed onto a work queue and losing the jobs.
+            while job = conn.zrangebyscore(sorted_set, '-inf', now, :limit => [0, 1]).first do
+
+              # Pop item off the queue and add it to the work queue. If the job can't be popped from
+              # the queue, it's because another process already popped it so we can move on to the
+              # next one.
+              if conn.zrem(sorted_set, job)
+                Sidekiq::Client.push(Sidekiq.load_json(job))
+                Sidekiq::Logging.logger.debug { "enqueued #{sorted_set}: #{job}" }
+              end
+            end
+          end
+        end
+      end
+    end
 
     ##
     # The Poller checks Redis every N seconds for jobs in the retry or scheduled
@@ -17,34 +41,18 @@ module Sidekiq
       include Util
       include Actor
 
-      SETS = %w(retry schedule)
+      INITIAL_WAIT = 10
+
+      def initialize
+        @enq = (Sidekiq.options[:scheduled_enq] || Sidekiq::Scheduled::Enq).new
+      end
 
       def poll(first_time=false)
         watchdog('scheduling poller thread died!') do
           initial_wait if first_time
 
           begin
-            # A job's "score" in Redis is the time at which it should be processed.
-            # Just check Redis for the set of jobs with a timestamp before now.
-            now = Time.now.to_f.to_s
-            Sidekiq.redis do |conn|
-              SETS.each do |sorted_set|
-                # Get the next item in the queue if it's score (time to execute) is <= now.
-                # We need to go through the list one at a time to reduce the risk of something
-                # going wrong between the time jobs are popped from the scheduled queue and when
-                # they are pushed onto a work queue and losing the jobs.
-                while job = conn.zrangebyscore(sorted_set, '-inf', now, :limit => [0, 1]).first do
-
-                  # Pop item off the queue and add it to the work queue. If the job can't be popped from
-                  # the queue, it's because another process already popped it so we can move on to the
-                  # next one.
-                  if conn.zrem(sorted_set, job)
-                    Sidekiq::Client.push(Sidekiq.load_json(job))
-                    logger.debug { "enqueued #{sorted_set}: #{job}" }
-                  end
-                end
-              end
-            end
+            @enq.enqueue_jobs
           rescue => ex
             # Most likely a problem with redis networking.
             # Punt and try again at the next interval
@@ -85,10 +93,10 @@ module Sidekiq
 
       def initial_wait
         begin
-          # Have all processes sleep between 10-15 seconds.  10 seconds
-          # to give time for the heartbeat to register and 5 random seconds
-          # to ensure they don't all hit Redis at the same time.
-          sleep(INITIAL_WAIT)
+          # Have all processes sleep between 5-15 seconds.  10 seconds
+          # to give time for the heartbeat to register (if the poll interval is going to be calculated by the number
+          # of workers), and 5 random seconds to ensure they don't all hit Redis at the same time.
+          sleep(INITIAL_WAIT) unless Sidekiq.options[:poll_interval]
           sleep(5 * rand)
         rescue Celluloid::Task::TerminatedError
           # Hit Ctrl-C when Sidekiq is finished booting and we have a chance

data/lib/sidekiq/version.rb

@@ -1,3 +1,3 @@
 module Sidekiq
-  VERSION = "3.3.1"
+  VERSION = "3.3.2"
 end

data/test/test_api.rb

@@ -76,6 +76,10 @@ class TestApi < Sidekiq::Test
     end
 
     describe "queues" do
+      it "returns all queues" do
+        assert_equal Sidekiq::Stats.new.queues, Sidekiq::Stats::Queues.new.lengths
+      end
+
       it "is initially empty" do
         s = Sidekiq::Stats::Queues.new
         assert_equal 0, s.lengths.size

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: sidekiq
 version: !ruby/object:Gem::Version
-  version: 3.3.1
+  version: 3.3.2
 platform: ruby
 authors:
 - Mike Perham
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-01-27 00:00:00.000000000 Z
+date: 2015-02-05 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis