sidekiq 4.1.1 → 4.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f3831fb91606790c4912c4fb8d96a560e49aa9b5
-  data.tar.gz: 74e50d973aab4a58f00f7176ea640dd52f37e218
+  metadata.gz: db0f25c2fe7ea87588364c3eb86e3b929197139c
+  data.tar.gz: 8fa8c056e6c5fad50ce5d49838515aca61573458
 SHA512:
-  metadata.gz: b5b098a51e90fff82508ddfa1b565d1e06758e99de904870795919314987439231aeeb1b315212eed9dcc771a9464909fa437e6990b799e8c245c52506b66eee
-  data.tar.gz: e1a1ae8f6c3217333b69187041ac8e1c974ebc29aa862b9abab284ae312d8680d960191defb1e67d7a3e6ee95c9d9a924cfc308f5e96e06a38157e0e0dd45ebe
+  metadata.gz: 4a801ac5f3d70ca6e1856cb7369f557d5bb8d1c579e775f6a4f5ed97d534af2d289f884979de978f3a7cab2d1261fabeef6b7fc9e73b1684c7b1daabbd25e0e7
+  data.tar.gz: 37c617e0ea241fe8cea7c85e6479e2259e06f1b5bfb06ce811b2529d73adc8a09a67655dbdcaff028ba4cd3226169a79f636f813f7848d7fe8eaa084ea9d5739

data/.github/issue_template.md

@@ -0,0 +1,4 @@
+Ruby version:
+Sidekiq / Pro / Enterprise version(s):
+
+If relevant, please include your initializer and any error message with the full backtrace.

data/.travis.yml

@@ -3,13 +3,10 @@ sudo: false
 cache: bundler
 services:
   - redis-server
+before_install:
+  - gem install bundler
+  - gem update bundler
 rvm:
-  - 2.0.0
-  - 2.1.8
   - 2.2.4
   - 2.3.0
   - jruby-head
-  - rbx-2
-matrix:
-  allow_failures:
-    - rvm: rbx-2

data/4.0-Upgrade.md

@@ -25,6 +25,9 @@ gem 'redis-namespace'
   `concurrency + 2` connections in your pool or Sidekiq will exit.
   When in doubt, let Sidekiq size the connection pool for you.
 
+* Worker data is no longer updated in real-time but rather upon every
+  heartbeat.  Don't expect the `Sidekiq::Workers` API to be millisecond-precise.
+
 * There's a new testing API based off the `Sidekiq::Queues` namespace. All
   assertions made against the Worker class still work as expected.
 ```ruby

data/COMM-LICENSE

@@ -28,7 +28,7 @@ In order to use the Software under this Agreement, you must receive a “Source
 
 3.1 You shall not (and shall not allow any third party to): (a) decompile, disassemble, or otherwise reverse engineer the Software or attempt to reconstruct or discover any source code, underlying ideas, algorithms, file formats or programming interfaces of the Software by any means whatsoever (except and only to the extent that applicable law prohibits or restricts reverse engineering restrictions); (b) distribute, sell, sublicense, rent, lease or use the Software for time sharing, hosting, service provider or like purposes, except as expressly permitted under this Agreement; (c) redistribute the Software or Modifications other than by including the Software or a portion thereof within your own product, which must have substantially different functionality than the Software or Modifications and must not allow any third party to use the Software or Modifications, or any portions thereof, for software development or application development purposes; (d) redistribute the Software as part of a product, "appliance" or "virtual server"; (e) redistribute the Software on any server which is not directly under your control; (f) remove any product identification, proprietary, copyright or other notices contained in the Software; (g) modify any part of the Software, create a derivative work of any part of the Software (except as permitted in Section 4), or incorporate the Software, except to the extent expressly authorized in writing by Contributed Systems; (h) publicly disseminate performance information or analysis (including, without limitation, benchmarks) from any source relating to the Software; (i) utilize any equipment, device, software, or other means designed to circumvent or remove any form of Source URL or copy protection used by Contributed Systems in connection with the Software, or use the Software together with any authorization code, Source URL, serial number, or other copy protection device not supplied by Contributed Systems; (j) use the Software to develop a product which is competitive with any Contributed Systems product offerings; or (k) use unauthorized Source URLS or keycode(s) or distribute or publish Source URLs or keycode(s), except as may be expressly permitted by Contributed Systems in writing. If your unique Source URL is ever published, Contributed Systems reserves the right to terminate your access without notice.
 
-3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE FOR A PRODUCT THAT IS INTENDED FOR SOFTWARE OR APPLICATION DEVELOPMENT PURPOSES.
+3.2 UNDER NO CIRCUMSTANCES MAY YOU USE THE SOFTWARE AS PART OF A PRODUCT OR SERVICE THAT PROVIDES SIMILAR FUNCTIONALITY TO THE SOFTWARE ITSELF.
 
 The Open Source version of the Software (“LGPL Version”) is licensed
 under the terms of the GNU Lesser General Public License versions 3.0

data/Changes.md

@@ -1,5 +1,74 @@
 # Sidekiq Changes
 
+4.2.0
+-----------
+
+- Enable development-mode code reloading.  **With Rails 5.0+, you don't need
+  to restart Sidekiq to pick up your Sidekiq::Worker changes anymore!** [#2457]
+- **Remove Sinatra dependency**.  Sidekiq's Web UI now uses Rack directly.
+  Thank you to Sidekiq's newest committer, **badosu**, for writing the code
+  and doing a lot of testing to ensure compatibility with many different
+  3rd party plugins.  If your Web UI works with 4.1.4 but fails with
+  4.2.0, please open an issue. [#3075]
+- Allow tuning of concurrency with the `RAILS_MAX_THREADS` env var. [#2985]
+  This is the same var used by Puma so you can tune all of your systems
+  the same way:
+```sh
+web: RAILS_MAX_THREADS=5 bundle exec puma ...
+worker: RAILS_MAX_THREADS=10 bundle exec sidekiq ...
+```
+Using `-c` or `config/sidekiq.yml` overrides this setting.  I recommend
+adjusting your `config/database.yml` to use it too so connections are
+auto-scaled:
+```yaml
+  pool: <%= ENV['RAILS_MAX_THREADS'] || 5 %>
+```
+
+4.1.4
+-----------
+
+- Unlock Sinatra so a Rails 5.0 compatible version may be used [#3048]
+- Fix race condition on startup with JRuby [#3043]
+
+
+4.1.3
+-----------
+
+- Please note the Redis 3.3.0 gem has a [memory leak](https://github.com/redis/redis-rb/issues/612),
+  Redis 3.2.2 is recommended until that issue is fixed.
+- Sinatra 1.4.x is now a required dependency, avoiding cryptic errors
+  and old bugs due to people not upgrading Sinatra for years. [#3042]
+- Fixed race condition in heartbeat which could rarely lead to lingering
+  processes on the Busy tab. [#2982]
+```ruby
+# To clean up lingering processes, modify this as necessary to connect to your Redis.
+# After 60 seconds, lingering processes should disappear from the Busy page.
+
+require 'redis'
+r = Redis.new(url: "redis://localhost:6379/0")
+# uncomment if you need a namespace
+#require 'redis-namespace'
+#r = Redis::Namespace.new("foo", r)
+r.smembers("processes").each do |pro|
+  r.expire(pro, 60)
+  r.expire("#{pro}:workers", 60)
+end
+```
+
+
+4.1.2
+-----------
+
+- Fix Redis data leak with worker data when a busy Sidekiq process
+  crashes.  You can find and expire leaked data in Redis with this
+script:
+```bash
+$ redis-cli keys  "*:workers" | while read LINE ; do TTL=`redis-cli expire "$LINE" 60`; echo "$LINE"; done;
+```
+  Please note that `keys` can be dangerous to run on a large, busy Redis.  Caveat runner.
+- Freeze all string literals with Ruby 2.3. [#2741]
+- Client middleware can now stop bulk job push. [#2887]
+
 4.1.1
 -----------
 

data/Ent-Changes.md

@@ -3,6 +3,46 @@ Sidekiq Enterprise Changelog
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+1.3.2
+-------------
+
+- Upgrade encryption to use OpenSSL's more secure GCM mode. [#3060]
+
+1.3.1
+-------------
+
+- Fix multi-process memory monitoring on CentOS 6.x [#3063]
+- Polish the new encryption feature a bit.
+
+1.3.0
+-------------
+
+- **BETA** [New encryption feature](https://github.com/mperham/sidekiq/wiki/Ent-Encryption)
+  which automatically encrypts the last argument of a Worker, aka the secret bag.
+
+1.2.4
+-------------
+
+- Fix issue causing some minutely jobs to execute every other minute.
+- Log a warning if slow periodic processing causes us to miss a clock tick.
+
+1.2.3
+-------------
+
+- Periodic jobs could stop executing until process restart if Redis goes down [#3047]
+
+1.2.2
+-------------
+
+- Add API to check if a unique lock is present. See [#2932] for details.
+- Tune concurrent limiters to minimize thread thrashing under heavy contention. [#2944]
+- Add option for tuning which Bundler groups get preloaded with `sidekiqswarm` [#3025]
+```
+SIDEKIQ_PRELOAD=default,production bin/sidekiqswarm ...
+# Use an empty value for maximum application compatibility
+SIDEKIQ_PRELOAD= bin/sidekiqswarm ...
+```
+
 1.2.1
 -------------
 

data/Gemfile

@@ -1,7 +1,8 @@
 source 'https://rubygems.org'
 gemspec
 
-gem 'rails', '~> 4.2'
+gem 'rails', '5.0.0'
+gem "hiredis"
 gem 'simplecov'
 gem 'minitest'
 gem 'minitest-utils'
@@ -22,7 +23,7 @@ platforms :mri do
   gem 'ruby-prof'
 end
 
-platforms :jruby do
-  gem 'jruby-openssl'
-  gem 'activerecord-jdbcsqlite3-adapter'
-end
+#platforms :jruby do
+  #gem 'jruby-openssl'
+  #gem 'activerecord-jdbcsqlite3-adapter'
+#end

data/Pro-3.0-Upgrade.md

@@ -17,17 +17,15 @@ Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
 * Reliable fetch has been re-implemented due to the fetch changes in
   Sidekiq 4.0.
 
-* Support for platforms without persistent hostnames.  Since reliable fetch
-  normally requires a persistent hostname, you may disable hostname usage on
-  platforms like Heroku and Docker:
+* Support for platforms without persistent hostnames.  Since the reliable\_fetch
+  algorithm requires a persistent hostname, an alternative reliability
+algorithm is now available for platforms like Heroku and Docker:
 ```ruby
 Sidekiq.configure_server do |config|
-  config.options[:ephemeral_hostname] = true
-  config.reliable_fetch!
+  config.timed_fetch!
 end
 ```
-  This option is enabled automatically if Heroku's DYNO environment variable is present.
-  Without a persistent hostname, each Sidekiq process **must** have its own unique index.
+  The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
 
 * The old 'sidekiq/notifications' features have been removed.
 

data/Pro-Changes.md

@@ -3,6 +3,72 @@ Sidekiq Pro Changelog
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+3.3.3
+---------
+
+- Update Web UI extension to work with Sidekiq 4.2.0's new Web UI. [#3075]
+
+3.3.2
+---------
+
+- Minimize batch memory usage after success [#3083]
+- Extract batch's 24 hr linger expiry to a LINGER constant so it can be tuned. [#3011]
+
+3.3.1
+---------
+
+- If environment is unset, treat it as development so reliable\_fetch works as before 3.2.2.
+
+3.3.0
+---------
+
+- Don't delete batches immediately upon success but set a 24 hr expiry, this allows
+  Sidekiq::Batch::Status#poll to work, even after batch success. [#3011]
+- New `Sidekiq::PendingSet#destroy(jid)` API to remove poison pill jobs [#3015]
+
+3.2.2
+---------
+
+- A default value for -i is only set in development now, staging or
+  other environments must set an index if you wish to use reliable\_fetch. [#2971]
+- Fix nil dereference when checking for jobs over timeout in timed\_fetch
+
+
+3.2.1
+---------
+
+- timed\_fetch now works with namespaces.  [ryansch]
+
+
+3.2.0
+---------
+
+- Fixed detection of missing batches, `NoSuchBatch` should be raised
+  properly now if `Sidekiq::Batch.new(bid)` is called on a batch no
+  longer in Redis.
+- Remove support for Pro 1.x format batches.  This version will no
+  longer seamlessly process batches created with Sidekiq Pro 1.x.
+  As always, upgrade one major version at a time to ensure a smooth
+  transition.
+- Fix edge case where a parent batch could expire before a child batch
+  was finished processing, leading to missing batches [#2889]
+
+2.1.5
+---------
+
+- Fix edge case where a parent batch could expire before a child batch
+  was finished processing, leading to missing batches [#2889]
+
+3.1.0
+---------
+
+- New container-friendly fetch algorithm: `timed_fetch`.  See the
+  [wiki documentation](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server)
+  for trade offs between the two reliability options.  You should
+  use this if you are on Heroku, Docker, Amazon ECS or EBS or
+  another container-based system.
+
+
 3.0.6
 ---------
 

data/README.md

@@ -18,7 +18,8 @@ message format as Resque so it can integrate into an existing Resque processing
 You can have Sidekiq and Resque run side-by-side at the same time and
 use the Resque client to enqueue jobs in Redis to be processed by Sidekiq.
 
-Sidekiq is fast.
+Performance
+---------------
 
 Version |	Latency | Garbage created for 10,000 jobs	| Time to process 100,000 jobs |	Throughput
 -----------------|------|---------|---------|------------------------

data/bin/sidekiqload

@@ -26,8 +26,8 @@ Toxiproxy.populate([{
 
 
 Sidekiq.configure_server do |config|
-  config.redis = { db: 13, port: 6380 }
-  #config.redis = { db: 13 }
+  #config.options[:concurrency] = 1
+  config.redis = { driver: :hiredis, db: 13, port: 6380 }
   config.options[:queues] << 'default'
   config.logger.level = Logger::ERROR
   config.average_scheduled_poll_interval = 2

data/lib/sidekiq.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 # encoding: utf-8
 require 'sidekiq/version'
 fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.0.0." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.0.0'
@@ -27,9 +28,11 @@ module Sidekiq
       startup: [],
       quiet: [],
       shutdown: [],
+      heartbeat: [],
     },
     dead_max_jobs: 10_000,
-    dead_timeout_in_seconds: 180 * 24 * 60 * 60 # 6 months
+    dead_timeout_in_seconds: 180 * 24 * 60 * 60, # 6 months
+    reloader: proc { |&block| block.call },
   }
 
   DEFAULT_WORKER_OPTIONS = {
@@ -168,6 +171,7 @@ module Sidekiq
   def self.default_retries_exhausted=(prok)
     @default_retries_exhausted = prok
   end
+  @default_retries_exhausted = ->(job, ex) { }
   def self.default_retries_exhausted
     @default_retries_exhausted
   end

data/lib/sidekiq/api.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 # encoding: utf-8
 require 'sidekiq'
 
@@ -195,7 +196,7 @@ module Sidekiq
     # Return all known queues within Redis.
     #
     def self.all
-      Sidekiq.redis {|c| c.smembers('queues'.freeze) }.sort.map {|q| Sidekiq::Queue.new(q) }
+      Sidekiq.redis { |c| c.smembers('queues'.freeze) }.sort.map { |q| Sidekiq::Queue.new(q) }
     end
 
     attr_reader :name
@@ -277,6 +278,7 @@ module Sidekiq
   #
   class Job
     attr_reader :item
+    attr_reader :value
 
     def initialize(item, queue_name=nil)
       @value = item

data/lib/sidekiq/cli.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 # encoding: utf-8
 $stdout.sync = true
 
@@ -70,6 +71,9 @@ module Sidekiq
       ver = Sidekiq.redis_info['redis_version']
       raise "You are using Redis v#{ver}, Sidekiq requires Redis v2.8.0 or greater" if ver < '2.8'
 
+      # Touch middleware so it isn't lazy loaded by multiple threads, #3043
+      Sidekiq.server_middleware
+
       # Before this point, the process is initializing with just the main thread.
       # Starting here the process will now have multiple threads running.
       fire_event(:startup)
@@ -204,6 +208,8 @@ module Sidekiq
       opts = parse_config(cfile).merge(opts) if cfile
 
       opts[:strict] = true if opts[:strict].nil?
+      opts[:concurrency] = Integer(ENV["RAILS_MAX_THREADS"]) if !opts[:concurrency] && ENV["RAILS_MAX_THREADS"]
+      opts[:identity] = identity
 
       options.merge!(opts)
     end
@@ -223,7 +229,7 @@ module Sidekiq
           require 'sidekiq/rails'
           require File.expand_path("#{options[:require]}/config/environment.rb")
           ::Rails.application.eager_load!
-        else
+        elsif ::Rails::VERSION::MAJOR == 4
           # Painful contortions, see 1791 for discussion
           require File.expand_path("#{options[:require]}/config/application.rb")
           ::Rails::Application.initializer "sidekiq.eager_load" do
@@ -231,10 +237,17 @@ module Sidekiq
           end
           require 'sidekiq/rails'
           require File.expand_path("#{options[:require]}/config/environment.rb")
+        else
+          require 'sidekiq/rails'
+          require File.expand_path("#{options[:require]}/config/environment.rb")
+          Sidekiq.options[:reloader] = Sidekiq::Rails::Reloader.new
         end
         options[:tag] ||= default_tag
       else
-        require options[:require]
+        not_required_message = "#{options[:require]} was not required, you should use an explicit path: " +
+            "./#{options[:require]} or /path/to/#{options[:require]}"
+
+        require(options[:require]) || raise(ArgumentError, not_required_message)
       end
     end
 

data/lib/sidekiq/client.rb

@@ -1,3 +1,4 @@
+# frozen_string_literal: true
 require 'securerandom'
 require 'sidekiq/middleware/chain'
 
@@ -82,10 +83,15 @@ module Sidekiq
     # Returns an array of the of pushed jobs' jids.  The number of jobs pushed can be less
     # than the number given if the middleware stopped processing for one or more jobs.
     def push_bulk(items)
+      arg = items['args'].first
+      return [] unless arg # no jobs to push
+      raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" if !arg.is_a?(Array)
+
       normed = normalize_item(items)
       payloads = items['args'].map do |args|
-        raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" if !args.is_a?(Array)
-        process_single(items['class'], normed.merge('args' => args, 'jid' => SecureRandom.hex(12), 'enqueued_at' => Time.now.to_f))
+        copy = normed.merge('args' => args, 'jid' => SecureRandom.hex(12), 'enqueued_at' => Time.now.to_f)
+        result = process_single(items['class'], copy)
+        result ? result : nil
       end.compact
 
       raw_push(payloads) if !payloads.empty?
@@ -106,7 +112,8 @@ module Sidekiq
     # you cannot scale any other way (e.g. splitting your app into smaller apps).
     def self.via(pool)
       raise ArgumentError, "No pool given" if pool.nil?
-      raise RuntimeError, "Sidekiq::Client.via is not re-entrant" if x = Thread.current[:sidekiq_via_pool] && x != pool
+      current_sidekiq_pool = Thread.current[:sidekiq_via_pool]
+      raise RuntimeError, "Sidekiq::Client.via is not re-entrant" if current_sidekiq_pool && current_sidekiq_pool != pool
       Thread.current[:sidekiq_via_pool] = pool
       yield
     ensure
@@ -205,6 +212,7 @@ module Sidekiq
       raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash) && item.has_key?('class'.freeze) && item.has_key?('args'.freeze)
       raise(ArgumentError, "Job args must be an Array") unless item['args'].is_a?(Array)
       raise(ArgumentError, "Job class must be either a Class or String representation of the class name") unless item['class'.freeze].is_a?(Class) || item['class'.freeze].is_a?(String)
+      #raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
 
       normalized_hash(item['class'.freeze])
         .each{ |key, value| item[key] = value if item[key].nil? }