sidekiq 3.5.4 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d09c0753f21b76f14daf02a88f5490fc1b06858a
-  data.tar.gz: e7edc849a6a2cd7d471976d770b0f2937eafff98
+  metadata.gz: 154fd4d987e5f7c32fad42a20176d27e958bd49f
+  data.tar.gz: a33449473a2210e1f79e3f814c50f6fe982ff101
 SHA512:
-  metadata.gz: 50ba2a1930ba127f8f9bebe250184c456ccc5fec7ed64df338fab31ce741bfe518e594e3b21eced6fe88d87ed622214ee5a99382f7d13f02ef782c25a25f798b
-  data.tar.gz: 06c9547502b98dbc17ab95d32e6313d7121fa6072e4fb78c63b84380390a4dec17cff796fa3b57a4f27437ea011b999beca61b59e5be1cd2f28a56df58cde918
+  metadata.gz: 264d2c5a210a978ca8ce9f2a78962cc4f4d62966444788b7d942644881baec212133123f3434ffafdcc21d5bd9c0130eef007195a945f86771a33da5b2d1fbea
+  data.tar.gz: 6e651a3d38dd4518a3db01abd5febdb9a4c739d16873f2ce3ffa4998bd5f5a2285303437cc6f489cb43d032590bf741006d50170470c333ed31cfa62c1168178

data/4.0-Upgrade.md

@@ -0,0 +1,50 @@
+# Welcome to Sidekiq 4.0!
+
+Sidekiq 4.0 contains a redesigned, more efficient core with less overhead per job.
+See my blog for [an overview of Sidekiq 4's higher performance](http://www.mikeperham.com/2015/10/14/optimizing-sidekiq/).
+
+## What's New
+
+* Sidekiq no longer uses Celluloid.  If your application code uses Celluloid,
+  you will need to pull it in yourself.
+
+* `redis-namespace` has been removed from Sidekiq's gem dependencies. If
+  you want to use namespacing ([and I strongly urge you not to](http://www.mikeperham.com/2015/09/24/storing-data-with-redis/)), you'll need to add the gem to your Gemfile:
+```ruby
+gem 'redis-namespace'
+```
+
+* **Redis 2.8.0 or greater is required.**  Redis 2.8 was released two years
+  ago and contains **many** useful features which Sidekiq couldn't
+  leverage until now.  **Redis 3.0.3 or greater is recommended** for large
+  scale use.
+
+* Jobs are now fetched from Redis in parallel, making Sidekiq more
+  resilient to high network latency.  This means that Sidekiq requires
+  more Redis connections per process.  You must have a minimum of
+  `concurrency + 2` connections in your pool or Sidekiq will exit.
+  When in doubt, let Sidekiq size the connection pool for you.
+
+* There's a new testing API based off the `Sidekiq::Queues` namespace. All
+  assertions made against the Worker class still work as expected.
+```ruby
+assert_equal 0, Sidekiq::Queues["default"].size
+HardWorker.perform_async("log")
+assert_equal 1, Sidekiq::Queues["default"].size
+assert_equal "log", Sidekiq::Queues["default"].first['args'][0]
+Sidekiq::Queues.clear_all
+```
+
+## Upgrade
+
+First, make sure you are using Redis 2.8 or greater. Next:
+
+* Upgrade to the latest Sidekiq 3.x.
+```ruby
+gem 'sidekiq', '< 4'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 4.x.
+```ruby
+gem 'sidekiq', '< 5'
+```

data/Changes.md

@@ -1,10 +1,22 @@
 # Sidekiq Changes
 
-3.5.4
+4.0.0.pre1
 -----------
 
-- Ensure exception message is a string [#2707]
-- Revert racy Process.kill usage in sidekiqctl
+- Sidekiq's internals have been completely overhauled for performance
+  and to remove dependencies.  This has resulted in major speedups, as
+  [detailed on my blog](http://www.mikeperham.com/2015/10/14/optimizing-sidekiq/).
+- See the [4.0 upgrade notes](4.0-Upgrade.md) for more detail.
+- There's a new testing API based off the `Sidekiq::Queues` namespace. All
+  assertions made against the Worker class still work as expected.
+  [#2659, brandonhilkert]
+```ruby
+assert_equal 0, Sidekiq::Queues["default"].size
+HardWorker.perform_async("log")
+assert_equal 1, Sidekiq::Queues["default"].size
+assert_equal "log", Sidekiq::Queues["default"].first['args'][0]
+Sidekiq::Queues.clear_all
+```
 
 3.5.3
 -----------

data/Ent-Changes.md

@@ -3,6 +3,33 @@ Sidekiq Enterprise Changelog
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+1.0.0.pre1
+----------
+
+- Enterprise 1.x targets Sidekiq 4.x.
+- Rewrite several features to remove Celluloid dependency.  No
+  functional changes.
+
+0.7.8
+----------
+
+- Fix `unique_for: false` [#2658]
+
+
+0.7.7
+----------
+
+- Enterprise 0.x targets Sidekiq 3.x.
+- Fix racy shutdown event which could lead to disappearing periodic
+  jobs, requires Sidekiq >= 3.5.3.
+- Add new :leader event which is fired when a process gains leadership.
+
+0.7.6
+----------
+
+- Redesign how overrated jobs are rescheduled to avoid creating new
+  jobs. [#2619]
+
 0.7.5
 ----------
 

data/Gemfile

@@ -4,12 +4,12 @@ gemspec
 gem 'rails', '~> 4.2'
 gem 'simplecov'
 gem 'minitest'
+gem 'minitest-utils'
 gem 'toxiproxy'
 
 platforms :rbx do
   gem 'rubysl', '~> 2.0'         # if using anything in the ruby standard library
   gem 'psych'                    # if using yaml
-  gem 'minitest'                 # if using minitest
   gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
 end
 
@@ -19,6 +19,7 @@ end
 
 platforms :mri do
   gem 'pry-byebug'
+  gem 'ruby-prof'
 end
 
 platforms :jruby do

data/Pro-3.0-Upgrade.md

@@ -0,0 +1,46 @@
+# Welcome to Sidekiq Pro 3.0!
+
+Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
+
+## What's New
+
+* **Redis 2.8.0 or greater is required.**  Redis 2.8 was released two years
+  ago and contains **many** useful features which Sidekiq couldn't
+  leverage until now.  **Redis 3.0.3 or greater is recommended** for large
+  scale use.
+
+* Sidekiq Pro no longer uses Celluloid.  If your application code uses Celluloid,
+  you will need to pull it in yourself.
+
+* Pausing and unpausing queues is now instantaneous, no more polling!
+
+* Reliable fetch has been re-implemented due to the fetch changes in
+  Sidekiq 4.0.
+
+* Support for platforms without persistent hostnames.  Since reliable fetch
+  normally requires a persistent hostname, you may disable hostname usage on
+  platforms like Heroku and Docker:
+```ruby
+Sidekiq.configure_server do |config|
+  config.options[:ephemeral_hostname] = true
+  config.reliable_fetch!
+end
+```
+  This option is enabled automatically if Heroku's DYNO environment variable is present.
+  Without a persistent hostname, each Sidekiq process **must** have its own unique index.
+
+* The old 'sidekiq/notifications' features have been removed.
+
+## Upgrade
+
+First, make sure you are using Redis 2.8 or greater. Next:
+
+* Upgrade to the latest Sidekiq Pro 2.x.
+```ruby
+gem 'sidekiq-pro', '< 3'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 3.x.
+```ruby
+gem 'sidekiq-pro', '< 4'
+```

data/Pro-Changes.md

@@ -3,6 +3,27 @@ Sidekiq Pro Changelog
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+3.0.0.pre2
+-----------
+
+- Fix excessive connection usage by reliable fetch.
+
+3.0.0.pre1
+-----------
+
+- See the [Pro 3.0 release notes](Pro-3.0-Upgrade.md).
+
+2.1.3
+-----------
+
+- Don't enable strict priority if using weighted queueing like `-q a,1 -q b,1`
+- Safer JSON mangling in Lua [#2639]
+
+2.1.2
+-----------
+
+- Lock Sidekiq Pro 2.x to Sidekiq 3.x.
+
 2.1.1
 -----------
 

data/README.md

@@ -32,7 +32,7 @@ are untested but might work fine.  MRI 1.9 is no longer supported.
 
 All Rails releases starting from 3.2 are officially supported.
 
-Redis 2.4 or greater is required.
+Redis 2.8 or greater is required.
 
 
 Installation
@@ -66,7 +66,7 @@ Please see the [sidekiq wiki](https://github.com/mperham/sidekiq/wiki) for the o
 [mperham/sidekiq on Gitter](https://gitter.im/mperham/sidekiq) is dedicated to this project,
 but bug reports or feature requests suggestions should still go through [issues on Github](https://github.com/mperham/sidekiq/issues).  Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
 
-You may also find useful a [Google Group](https://groups.google.com/forum/#!forum/sidekiq) dedicated to Sidekiq discussion and [a Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow.
+You may also find useful a [Reddit area](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion and [a Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow.
 
 
 Problems?
@@ -75,14 +75,14 @@ Problems?
 **Please do not directly email any Sidekiq committers with questions or problems.**  A community is best served when discussions are held in public.
 
 If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages. Searching the issues for your problem is also a good idea.  If that doesn't help, feel free to email the Sidekiq mailing list, chat in Gitter, or open a new issue.
-The mailing list is the preferred place to ask questions on usage. If you are encountering what you think is a bug, please open an issue.
+StackOverflow or Reddit is the preferred place to ask questions on usage. If you are encountering what you think is a bug, please open an issue.
 
 
 Thanks
 -----------------
 
 Sidekiq stays fast by using the [JProfiler java profiler](http://www.ej-technologies.com/products/jprofiler/overview.html) to find and fix
-performance problems on JRuby.  Unfortunately MRI does not have good profile tooling.
+performance problems on JRuby.  Unfortunately MRI does not have good multithreaded profiling tools.
 
 
 License

data/bin/sidekiqctl

@@ -41,9 +41,13 @@ class Sidekiqctl
   end
 
   def fetch_process
-    Process.getpgid(pid)
+    Process.kill(0, pid)
   rescue Errno::ESRCH
     done "Process doesn't exist", :error
+  # We were not allowed to send a signal, but the process must have existed
+  # when Process.kill() was called.
+  rescue Errno::EPERM
+    return pid
   end
 
   def done(msg, error = nil)
@@ -67,10 +71,12 @@ class Sidekiqctl
     `kill -TERM #{pid}`
     kill_timeout.times do
       begin
-        Process.getpgid(pid)
+        Process.kill(0, pid)
       rescue Errno::ESRCH
         FileUtils.rm_f pidfile
         done 'Sidekiq shut down gracefully.'
+      rescue Errno::EPERM
+        done 'Not permitted to shut down Sidekiq.'
       end
       sleep 1
     end

data/bin/sidekiqload

@@ -3,21 +3,35 @@
 # Quiet some warnings we see when running in warning mode:
 # RUBYOPT=-w bundle exec sidekiq
 $TESTING = false
-$CELLULOID_DEBUG = false
 
-require 'celluloid/current'
-puts Celluloid::VERSION
+#require 'ruby-prof'
+Bundler.require(:default)
+
 require_relative '../lib/sidekiq/cli'
 require_relative '../lib/sidekiq/launcher'
-Celluloid.logger = nil
 
 include Sidekiq::Util
 
+# brew tap shopify/shopify
+# brew install toxiproxy
+# gem install toxiproxy
+require 'toxiproxy'
+# simulate a non-localhost network for realer-world conditions.
+# adding 1ms of network latency has an ENORMOUS impact on benchmarks
+Toxiproxy.populate([{
+    "name": "redis",
+    "listen": "127.0.0.1:6380",
+    "upstream": "127.0.0.1:6379"
+}])
+
+
 Sidekiq.configure_server do |config|
   config.redis = { db: 13, port: 6380 }
+  #config.redis = { db: 13 }
   config.options[:queues] << 'default'
   config.logger.level = Logger::ERROR
   config.average_scheduled_poll_interval = 2
+  config.reliable! if defined?(Sidekiq::Pro)
 end
 
 class LoadWorker
@@ -68,8 +82,7 @@ def handle_signal(launcher, sig)
     raise Interrupt
   when 'USR1'
     Sidekiq.logger.info "Received USR1, no longer accepting new work"
-    launcher.manager.async.stop
-    #fire_event(:quiet, true)
+    launcher.quiet
   when 'USR2'
     if Sidekiq.options[:logfile]
       Sidekiq.logger.info "Received USR2, reopening log file"
@@ -129,6 +142,7 @@ end
 begin
   #RubyProf::exclude_threads = [ Monitoring ]
   #RubyProf.start
+  fire_event(:startup)
   Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
   Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
     launcher = Sidekiq::Launcher.new(Sidekiq.options)

data/lib/sidekiq.rb

@@ -110,11 +110,25 @@ module Sidekiq
   end
 
   def self.server_middleware
-    @server_chain ||= Processor.default_middleware
+    @server_chain ||= default_server_middleware
     yield @server_chain if block_given?
     @server_chain
   end
 
+  def self.default_server_middleware
+    require 'sidekiq/middleware/server/retry_jobs'
+    require 'sidekiq/middleware/server/logging'
+
+    Middleware::Chain.new do |m|
+      m.add Middleware::Server::Logging
+      m.add Middleware::Server::RetryJobs
+      if defined?(::ActiveRecord::Base)
+        require 'sidekiq/middleware/server/active_record'
+        m.add Sidekiq::Middleware::Server::ActiveRecord
+      end
+    end
+  end
+
   def self.default_worker_options=(hash)
     @default_worker_options = default_worker_options.merge(hash.stringify_keys)
   end
@@ -139,16 +153,6 @@ module Sidekiq
     Sidekiq::Logging.logger = log
   end
 
-  # When set, overrides Sidekiq.options[:average_scheduled_poll_interval] and sets
-  # the average interval that this process will delay before checking for
-  # scheduled jobs or job retries that are ready to run.
-  #
-  # See sidekiq/scheduled.rb for an in-depth explanation of this value
-  def self.poll_interval=(interval)
-    $stderr.puts "DEPRECATION: `config.poll_interval = #{interval}` will be removed in Sidekiq 4. Please update to `config.average_scheduled_poll_interval = #{interval}`."
-    self.options[:poll_interval_average] = interval
-  end
-
   # How frequently Redis should be checked by a random Sidekiq process for
   # scheduled and retriable jobs. Each individual process will take turns by
   # waiting some multiple of this value.
@@ -182,6 +186,15 @@ module Sidekiq
     raise ArgumentError, "Invalid event name: #{event}" unless options[:lifecycle_events].key?(event)
     options[:lifecycle_events][event] << block
   end
+
+  # We are shutting down Sidekiq but what about workers that
+  # are working on some long job?  This error is
+  # raised in workers that have not finished within the hard
+  # timeout limit.  This is needed to rollback db transactions,
+  # otherwise Ruby's Thread#kill will commit.  See #377.
+  # DO NOT RESCUE THIS ERROR IN YOUR WORKERS
+  class Shutdown < Interrupt; end
+
 end
 
 require 'sidekiq/extensions/class_methods'

data/lib/sidekiq/api.rb

@@ -225,7 +225,7 @@ module Sidekiq
       page = 0
       page_size = 50
 
-      loop do
+      while true do
         range_start = page * page_size - deleted_size
         range_end   = page * page_size - deleted_size + (page_size - 1)
         entries = Sidekiq.redis do |conn|
@@ -488,7 +488,7 @@ module Sidekiq
       page = -1
       page_size = 50
 
-      loop do
+      while true do
         range_start = page * page_size + offset_size
         range_end   = page * page_size + offset_size + (page_size - 1)
         elements = Sidekiq.redis do |conn|

data/lib/sidekiq/cli.rb

@@ -11,18 +11,18 @@ require 'sidekiq'
 require 'sidekiq/util'
 
 module Sidekiq
-  # We are shutting down Sidekiq but what about workers that
-  # are working on some long job?  This error is
-  # raised in workers that have not finished within the hard
-  # timeout limit.  This is needed to rollback db transactions,
-  # otherwise Ruby's Thread#kill will commit.  See #377.
-  # DO NOT RESCUE THIS ERROR.
-  class Shutdown < Interrupt; end
-
   class CLI
     include Util
     include Singleton unless $TESTING
 
+    PROCTITLES = [
+      proc { 'sidekiq'.freeze },
+      proc { Sidekiq::VERSION },
+      proc { |me, data| data['tag'] },
+      proc { |me, data| "[#{Processor::WORKER_STATE.size} of #{data['concurrency']} busy]" },
+      proc { |me, data| "stopping" if me.stopping? },
+    ]
+
     # Used for CLI testing
     attr_accessor :code
     attr_accessor :launcher
@@ -40,7 +40,6 @@ module Sidekiq
       validate!
       daemonize
       write_pid
-      load_celluloid
     end
 
     # Code within this method is not tested because it alters
@@ -66,17 +65,21 @@ module Sidekiq
       logger.info Sidekiq::LICENSE
       logger.info "Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org" unless defined?(::Sidekiq::Pro)
 
+      Sidekiq.redis do |conn|
+        # touch the connection pool so it is created before we
+        # fire startup and start multithreading.
+        ver = conn.info['redis_version']
+        raise "You are using Redis v#{ver}, Sidekiq requires Redis v2.8.0 or greater" if ver < '2.8'
+      end
+
+      # Before this point, the process is initializing with just the main thread.
+      # Starting here the process will now have multiple threads running.
       fire_event(:startup)
 
       logger.debug {
         "Middleware: #{Sidekiq.server_middleware.map(&:klass).join(', ')}"
       }
 
-      Sidekiq.redis do |conn|
-        # touch the connection pool so it is created before we
-        # launch the actors.
-      end
-
       if !options[:daemon]
         logger.info 'Starting processing, hit Ctrl-C to stop'
       end
@@ -96,6 +99,7 @@ module Sidekiq
         launcher.stop
         # Explicitly exit so busy Processor threads can't block
         # process shutdown.
+        logger.info "Bye!"
         exit(0)
       end
     end
@@ -129,8 +133,7 @@ module Sidekiq
         raise Interrupt
       when 'USR1'
         Sidekiq.logger.info "Received USR1, no longer accepting new work"
-        launcher.manager.async.stop
-        fire_event(:quiet, true)
+        launcher.quiet
       when 'USR2'
         if Sidekiq.options[:logfile]
           Sidekiq.logger.info "Received USR2, reopening log file"
@@ -159,19 +162,6 @@ module Sidekiq
       end
     end
 
-    def load_celluloid
-      raise "Celluloid cannot be required until here, or it will break Sidekiq's daemonization" if defined?(::Celluloid) && options[:daemon]
-
-      # Celluloid can't be loaded until after we've daemonized
-      # because it spins up threads and creates locks which get
-      # into a very bad state if forked.
-      require 'celluloid/current'
-      Celluloid.logger = (options[:verbose] ? Sidekiq.logger : nil)
-
-      require 'sidekiq/manager'
-      require 'sidekiq/scheduled'
-    end
-
     def daemonize
       return unless options[:daemon]