sidekiq 4.2.10 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 595500a3810c25e8e04e86c644f966ff01cb035f
-  data.tar.gz: 3f92a9f61e61848bc2c0639469ce3588f3848fc0
+  metadata.gz: f2a9cd7270e53981aaf7b3f2639596ff042f5ae3
+  data.tar.gz: 7b6e4fbfee794de34d9b6b15772cbf085e05a65b
 SHA512:
-  metadata.gz: c4e97c3665d1e902f47cfb579fda437dc834ac4b584606f1ca3b8fd9225e8502cacf5fb687673b4735b359de6af179589759a34369f1c74e904b37f8fe70a4ee
-  data.tar.gz: a4e6d41067bf6b88c77577def0c1f9ecc3f5d46f0bdfb0821952c26867e91a71c64231aab09a25096d22c3272dea2dcc37547bc6096dd7622a192fa00e0a4e50
+  metadata.gz: dfaecc27ce4071e36b5ab994c27f59d85b4cf7f16ca05a5b786c79125c311f58fbb38b94f7e1c71e4dbf90e6645cb358102e8940a94881981f7bbc2578fd18ff
+  data.tar.gz: a98e9b87e5f4abe2580ea5fcb6f20ef1d28623eae9be4ca758d5f5b52c65dbef7a6b5df74f673449177f36494b9075573315c7ef6cd6b757ba6a59e825737833

data/.gitignore

@@ -10,3 +10,4 @@ vendor/
 .bundle/
 .sass-cache/
 tmp/
+pkg/*.gem

data/5.0-Upgrade.md

@@ -0,0 +1,56 @@
+# Welcome to Sidekiq 5.0!
+
+Sidekiq 5.0 contains a reworked job dispatch and execution core to integrate
+better with the new Rails 5.0 Executor.  It also drops support for older
+versions of Ruby and Rails and adds support for RTL languages in the Web UI.
+
+## What's New
+
+* Integrate job logging and retry logic directly in with the job
+  execution logic in Sidekiq::Processor.  Previously this logic was
+  defined as middleware.  In Rails 5.0, ActiveSupport::Executor handles ActiveRecord
+  connection management, job callbacks, development mode class loading,
+  etc.  Because of its extensive responsibilities, the Executor can't be
+  integrated as Sidekiq middleware; the logging/retry logic had to be pulled out
+  too.  Sidekiq 4.2 had a hack to make it work but this redesign provides
+  a cleaner integration. [#3235]
+* The Delayed Extensions `delay`, `delay_in` and `delay_until` APIs are
+  no longer available by default.  The extensions allow you to marshal
+  job arguments as YAML, leading to cases where job payloads could be many
+  100s of KB or larger if not careful, leading to Redis networking
+  timeouts or other problems.  As noted in the Best Practices wiki page,
+  Sidekiq is designed for jobs with small, simple arguments.
+
+  Add this line to your initializer to re-enable them and get the old behavior:
+  ```ruby
+  Sidekiq::Extensions.enable_delay!
+  ```
+  The old `Sidekiq.remove_delay!` API has been removed as it is now the default. [#3299]
+* Sidekiq's quiet signal is now `TSTP` (think of it as **T**hread
+  **ST**o**P**) instead of USR1 as USR1 is not available on JRuby.
+  USR1 will continue to be supported in Sidekiq 5.x for backwards
+  compatibility and will be removed in Sidekiq 6.x. [#3302]
+* The Web UI is now bi-directional - it can render either LTR
+  (left-to-right) or RTL languages. With this change, **Farsi, Arabic,
+  Hebrew and Urdu** are officially supported. [#3381]
+* Jobs which can't be parsed due to invalid JSON are now pushed
+  immediately to the Dead set since they require manual intervention and
+  will never execute successfully as is.  The Web UI has been updated to
+  more gracefully display these jobs. [#3296]
+* **Rails 3.2** is no longer supported.
+* **Ruby 2.0 and Ruby 2.1** are no longer supported. Ruby 2.2.2+ is required.
+
+## Upgrade
+
+As always, please upgrade Sidekiq **one major version at a time**.
+If you are already running Sidekiq 4.x, then:
+
+* Upgrade to the latest Sidekiq 4.x.
+```ruby
+gem 'sidekiq', '< 5'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 5.x.
+```ruby
+gem 'sidekiq', '< 6'
+```

data/Changes.md

@@ -1,6 +1,29 @@
 # Sidekiq Changes
 
-HEAD
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
+
+5.0.0
+-----------
+
+- **BREAKING CHANGE** Job dispatch was refactored for safer integration with
+  Rails 5.  The **Logging** and **RetryJobs** server middleware were removed and
+  functionality integrated directly into Sidekiq::Processor.  These aren't
+  commonly used public APIs so this shouldn't impact most users.
+```
+Sidekiq::Middleware::Server::RetryJobs -> Sidekiq::JobRetry
+Sidekiq::Middleware::Server::Logging -> Sidekiq::JobLogging
+```
+- Quieting Sidekiq is now done via the TSTP signal, the USR1 signal is deprecated.
+- The `delay` extension APIs are no longer available by default, you
+  must opt into them.
+- The Web UI is now BiDi and can render RTL languages like Arabic, Farsi and Hebrew.
+- Rails 3.2 and Ruby 2.0 and 2.1 are no longer supported.
+- The `SomeWorker.set(options)` API was re-written to avoid thread-local state. [#2152]
+- Sidekiq Enterprise's encrypted jobs now display "[encrypted data]" in the Web UI instead
+  of random hex bytes.
+- Please see the [5.0 Upgrade notes](5.0-Upgrade.md) for more detail.
+
+4.2.10
 -----------
 
 - Scheduled jobs can now be moved directly to the Dead queue via API [#3390]

data/Ent-Changes.md

@@ -1,5 +1,6 @@
-Sidekiq Enterprise Changelog
-=======================
+# Sidekiq Enterprise Changelog
+
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 

data/Pro-Changes.md

@@ -1,11 +1,15 @@
-Sidekiq Pro Changelog
-=======================
+# Sidekiq Pro Changelog
+
+[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
 HEAD
 ---------
 
+- Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
+- Batch page displays Pending JIDs if less than 10 [#3130]
+- Batch page has a Search button to find associated Retries [#3130]
 - Make Batch UI progress bar more friendly to the colorblind [#3387]
 
 3.4.5

data/README.md

@@ -32,9 +32,9 @@ DelayedJob 4.1.1 |  -   | -       | 465 sec | 215 jobs/sec
 Requirements
 -----------------
 
-Sidekiq supports CRuby 2.0+ and JRuby 9k.
+Sidekiq supports CRuby 2.2.2+ and JRuby 9k.
 
-All Rails releases >= 3.2 are officially supported.
+All Rails releases >= 4.0 are officially supported.
 
 Redis 2.8 or greater is required.  3.0.3+ is recommended for large
 installations with thousands of worker threads.

data/bin/sidekiqctl

@@ -64,7 +64,7 @@ class Sidekiqctl
   end
 
   def quiet
-    `kill -USR1 #{pid}`
+    `kill -TSTP #{pid}`
   end
 
   def stop

data/bin/sidekiqload

@@ -46,7 +46,7 @@ end
 #}])
 
 self_read, self_write = IO.pipe
-%w(INT TERM USR1 USR2 TTIN).each do |sig|
+%w(INT TERM TSTP TTIN).each do |sig|
   begin
     trap sig do
       self_write.puts(sig)
@@ -67,14 +67,9 @@ def handle_signal(launcher, sig)
   when 'TERM'
     # Heroku sends TERM and then waits 10 seconds for process to exit.
     raise Interrupt
-  when 'USR1'
-    Sidekiq.logger.info "Received USR1, no longer accepting new work"
+  when 'TSTP'
+    Sidekiq.logger.info "Received TSTP, no longer accepting new work"
     launcher.quiet
-  when 'USR2'
-    if Sidekiq.options[:logfile]
-      Sidekiq.logger.info "Received USR2, reopening log file"
-      Sidekiq::Logging.reopen_logs
-    end
   when 'TTIN'
     Thread.list.each do |thread|
       Sidekiq.logger.warn "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}"

data/lib/sidekiq/api.rb

@@ -75,7 +75,7 @@ module Sidekiq
       enqueued     = pipe2_res[s..-1].map(&:to_i).inject(0, &:+)
 
       default_queue_latency = if (entry = pipe1_res[6].first)
-                                job = Sidekiq.load_json(entry)
+                                job = Sidekiq.load_json(entry) rescue {}
                                 now = Time.now.to_f
                                 thence = job['enqueued_at'.freeze] || now
                                 now - thence
@@ -287,13 +287,25 @@ module Sidekiq
     attr_reader :value
 
     def initialize(item, queue_name=nil)
+      @args = nil
       @value = item
-      @item = item.is_a?(Hash) ? item : Sidekiq.load_json(item)
+      @item = item.is_a?(Hash) ? item : parse(item)
       @queue = queue_name || @item['queue']
     end
 
+    def parse(item)
+      Sidekiq.load_json(item)
+    rescue JSON::ParserError
+      # If the job payload in Redis is invalid JSON, we'll load
+      # the item as an empty hash and store the invalid JSON as
+      # the job 'args' for display in the Web UI.
+      @invalid = true
+      @args = [item]
+      {}
+    end
+
     def klass
-      @item['class']
+      self['class']
     end
 
     def display_class
@@ -324,32 +336,36 @@ module Sidekiq
                     arg
                   end
                 when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
-                  job_args = @item['wrapped'] ? args[0]["arguments"] : []
-                  if 'ActionMailer::DeliveryJob' == (@item['wrapped'] || args[0])
-                   # remove MailerClass, mailer_method and 'deliver_now'
-                   job_args.drop(3)
+                  job_args = self['wrapped'] ? args[0]["arguments"] : []
+                  if 'ActionMailer::DeliveryJob' == (self['wrapped'] || args[0])
+                    # remove MailerClass, mailer_method and 'deliver_now'
+                    job_args.drop(3)
                   else
-                   job_args
+                    job_args
                   end
                 else
+                  if self['encrypt'.freeze]
+                    # no point in showing 150+ bytes of random garbage
+                    args[-1] = '[encrypted data]'.freeze
+                  end
                   args
                 end
     end
 
     def args
-      @item['args']
+      @args || @item['args']
     end
 
     def jid
-      @item['jid']
+      self['jid']
     end
 
     def enqueued_at
-      @item['enqueued_at'] ? Time.at(@item['enqueued_at']).utc : nil
+      self['enqueued_at'] ? Time.at(self['enqueued_at']).utc : nil
     end
 
     def created_at
-      Time.at(@item['created_at'] || @item['enqueued_at'] || 0).utc
+      Time.at(self['created_at'] || self['enqueued_at'] || 0).utc
     end
 
     def queue
@@ -371,7 +387,10 @@ module Sidekiq
     end
 
     def [](name)
-      @item[name]
+      # nil will happen if the JSON fails to parse.
+      # We don't guarantee Sidekiq will work with bad job JSON but we should
+      # make a best effort to minimize the damage.
+      @item ? @item[name] : nil
     end
 
     private
@@ -756,7 +775,7 @@ module Sidekiq
     end
 
     def quiet!
-      signal('USR1')
+      signal('TSTP')
     end
 
     def stop!

data/lib/sidekiq/cli.rb

@@ -43,6 +43,10 @@ module Sidekiq
       write_pid
     end
 
+    def jruby?
+      defined?(::JRUBY_VERSION)
+    end
+
     # Code within this method is not tested because it alters
     # global process state irreversibly.  PRs which improve the
     # test coverage of Sidekiq::CLI are welcomed.
@@ -51,8 +55,14 @@ module Sidekiq
       print_banner
 
       self_read, self_write = IO.pipe
+      sigs = %w(INT TERM TTIN TSTP)
+      # USR1 and USR2 don't work on the JVM
+      if !jruby?
+        sigs << 'USR1'
+        sigs << 'USR2'
+      end
 
-      %w(INT TERM USR1 USR2 TTIN TSTP).each do |sig|
+      sigs.each do |sig|
         begin
           trap sig do
             self_write.puts(sig)
@@ -230,9 +240,7 @@ module Sidekiq
       if File.directory?(options[:require])
         require 'rails'
         if ::Rails::VERSION::MAJOR < 4
-          require 'sidekiq/rails'
-          require File.expand_path("#{options[:require]}/config/environment.rb")
-          ::Rails.application.eager_load!
+          raise "Sidekiq no longer supports this version of Rails"
         elsif ::Rails::VERSION::MAJOR == 4
           # Painful contortions, see 1791 for discussion
           # No autoloading, we want to force eager load for everything.
@@ -243,7 +251,6 @@ module Sidekiq
           require 'sidekiq/rails'
           require File.expand_path("#{options[:require]}/config/environment.rb")
         else
-          # Rails 5+ && development mode, use Reloader
           require 'sidekiq/rails'
           require File.expand_path("#{options[:require]}/config/environment.rb")
         end

data/lib/sidekiq/client.rb

@@ -48,6 +48,7 @@ module Sidekiq
     #   queue - the named queue to use, default 'default'
     #   class - the worker class to call, required
     #   args - an array of simple arguments to the perform method, must be JSON-serializable
+    #   at - timestamp to schedule the job (optional), must be Numeric (e.g. Time.now.to_f)
     #   retry - whether to retry this job if it fails, default true or an integer number of retries
     #   backtrace - whether to save any error backtrace, default false
     #
@@ -62,11 +63,11 @@ module Sidekiq
     #
     def push(item)
       normed = normalize_item(item)
-      payload = process_single(item['class'], normed)
+      payload = process_single(item['class'.freeze], normed)
 
       if payload
         raw_push([payload])
-        payload['jid']
+        payload['jid'.freeze]
       end
     end
 
@@ -83,19 +84,19 @@ module Sidekiq
     # Returns an array of the of pushed jobs' jids.  The number of jobs pushed can be less
     # than the number given if the middleware stopped processing for one or more jobs.
     def push_bulk(items)
-      arg = items['args'].first
+      arg = items['args'.freeze].first
       return [] unless arg # no jobs to push
       raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" if !arg.is_a?(Array)
 
       normed = normalize_item(items)
-      payloads = items['args'].map do |args|
-        copy = normed.merge('args' => args, 'jid' => SecureRandom.hex(12), 'enqueued_at' => Time.now.to_f)
-        result = process_single(items['class'], copy)
+      payloads = items['args'.freeze].map do |args|
+        copy = normed.merge('args'.freeze => args, 'jid'.freeze => SecureRandom.hex(12), 'enqueued_at'.freeze => Time.now.to_f)
+        result = process_single(items['class'.freeze], copy)
         result ? result : nil
       end.compact
 
       raw_push(payloads) if !payloads.empty?
-      payloads.collect { |payload| payload['jid'] }
+      payloads.collect { |payload| payload['jid'.freeze] }
     end
 
     # Allows sharding of jobs across any number of Redis instances.  All jobs
@@ -139,14 +140,14 @@ module Sidekiq
       # Messages are enqueued to the 'default' queue.
       #
       def enqueue(klass, *args)
-        klass.client_push('class' => klass, 'args' => args)
+        klass.client_push('class'.freeze => klass, 'args'.freeze => args)
       end
 
       # Example usage:
       #   Sidekiq::Client.enqueue_to(:queue_name, MyWorker, 'foo', 1, :bat => 'bar')
       #
       def enqueue_to(queue, klass, *args)
-        klass.client_push('queue' => queue, 'class' => klass, 'args' => args)
+        klass.client_push('queue'.freeze => queue, 'class'.freeze => klass, 'args'.freeze => args)
       end
 
       # Example usage:
@@ -157,7 +158,7 @@ module Sidekiq
         now = Time.now.to_f
         ts = (int < 1_000_000_000 ? now + int : int)
 
-        item = { 'class' => klass, 'args' => args, 'at' => ts, 'queue' => queue }
+        item = { 'class'.freeze => klass, 'args'.freeze => args, 'at'.freeze => ts, 'queue'.freeze => queue }
         item.delete('at'.freeze) if ts <= now
 
         klass.client_push(item)
@@ -183,13 +184,13 @@ module Sidekiq
     end
 
     def atomic_push(conn, payloads)
-      if payloads.first['at']
+      if payloads.first['at'.freeze]
         conn.zadd('schedule'.freeze, payloads.map do |hash|
           at = hash.delete('at'.freeze).to_s
           [at, Sidekiq.dump_json(hash)]
         end)
       else
-        q = payloads.first['queue']
+        q = payloads.first['queue'.freeze]
         now = Time.now.to_f
         to_push = payloads.map do |entry|
           entry['enqueued_at'.freeze] = now
@@ -201,7 +202,7 @@ module Sidekiq
     end
 
     def process_single(worker_class, item)
-      queue = item['queue']
+      queue = item['queue'.freeze]
 
       middleware.invoke(worker_class, item, queue, @redis_pool) do
         item
@@ -212,6 +213,7 @@ module Sidekiq
       raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash) && item.has_key?('class'.freeze) && item.has_key?('args'.freeze)
       raise(ArgumentError, "Job args must be an Array") unless item['args'].is_a?(Array)
       raise(ArgumentError, "Job class must be either a Class or String representation of the class name") unless item['class'.freeze].is_a?(Class) || item['class'.freeze].is_a?(String)
+      raise(ArgumentError, "Job 'at' must be a Numeric timestamp") if item.has_key?('at'.freeze) && !item['at'].is_a?(Numeric)
       #raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
 
       normalized_hash(item['class'.freeze])

data/lib/sidekiq/delay.rb

@@ -0,0 +1,21 @@
+module Sidekiq
+  module Extensions
+
+    def self.enable_delay!
+      if defined?(::ActiveSupport)
+        ActiveSupport.on_load(:active_record) do
+          require 'sidekiq/extensions/active_record'
+          include Sidekiq::Extensions::ActiveRecord
+        end
+        ActiveSupport.on_load(:action_mailer) do
+          require 'sidekiq/extensions/action_mailer'
+          extend Sidekiq::Extensions::ActionMailer
+        end
+      end
+
+      require 'sidekiq/extensions/class_methods'
+      Module.__send__(:include, Sidekiq::Extensions::Klass)
+    end
+
+  end
+end

data/lib/sidekiq/extensions/generic_proxy.rb

@@ -3,6 +3,8 @@ require 'yaml'
 
 module Sidekiq
   module Extensions
+    SIZE_LIMIT = 8_192
+
     class Proxy < BasicObject
       def initialize(performable, target, options={})
         @performable = performable
@@ -17,7 +19,11 @@ module Sidekiq
         # to JSON and then deserialized on the other side back into a
         # Ruby object.
         obj = [@target, name, args]
-        @performable.client_push({ 'class' => @performable, 'args' => [::YAML.dump(obj)] }.merge(@opts))
+        marshalled = ::YAML.dump(obj)
+        if marshalled.size > SIZE_LIMIT
+          ::Sidekiq.logger.warn { "#{@target}.#{name} job argument is #{marshalled.bytesize} bytes, you should refactor it to reduce the size" }
+        end
+        @performable.client_push({ 'class' => @performable, 'args' => [marshalled] }.merge(@opts))
       end
     end
 

data/lib/sidekiq/job_logger.rb

@@ -0,0 +1,27 @@
+module Sidekiq
+  class JobLogger
+
+    def call(item, queue)
+      begin
+        start = Time.now
+        logger.info("start".freeze)
+        yield
+        logger.info("done: #{elapsed(start)} sec")
+      rescue Exception
+        logger.info("fail: #{elapsed(start)} sec")
+        raise
+      end
+    end
+
+    private
+
+    def elapsed(start)
+      (Time.now - start).round(3)
+    end
+
+    def logger
+      Sidekiq.logger
+    end
+  end
+end
+