sidekiq 4.2.10 → 5.0.5

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

Files changed (57) hide show
  1. checksums.yaml +4 -4
  2. data/.gitignore +1 -0
  3. data/.travis.yml +5 -10
  4. data/5.0-Upgrade.md +56 -0
  5. data/COMM-LICENSE +1 -1
  6. data/Changes.md +60 -0
  7. data/Ent-Changes.md +29 -2
  8. data/Gemfile +3 -0
  9. data/Pro-Changes.md +28 -3
  10. data/README.md +3 -3
  11. data/bin/sidekiqctl +1 -1
  12. data/bin/sidekiqload +3 -8
  13. data/lib/generators/sidekiq/templates/worker_spec.rb.erb +1 -1
  14. data/lib/sidekiq.rb +6 -15
  15. data/lib/sidekiq/api.rb +77 -31
  16. data/lib/sidekiq/cli.rb +15 -6
  17. data/lib/sidekiq/client.rb +20 -13
  18. data/lib/sidekiq/core_ext.rb +1 -119
  19. data/lib/sidekiq/delay.rb +41 -0
  20. data/lib/sidekiq/extensions/generic_proxy.rb +7 -1
  21. data/lib/sidekiq/job_logger.rb +24 -0
  22. data/lib/sidekiq/job_retry.rb +228 -0
  23. data/lib/sidekiq/launcher.rb +1 -7
  24. data/lib/sidekiq/logging.rb +12 -0
  25. data/lib/sidekiq/middleware/server/active_record.rb +9 -0
  26. data/lib/sidekiq/processor.rb +63 -33
  27. data/lib/sidekiq/rails.rb +1 -73
  28. data/lib/sidekiq/redis_connection.rb +14 -3
  29. data/lib/sidekiq/testing.rb +12 -3
  30. data/lib/sidekiq/util.rb +1 -2
  31. data/lib/sidekiq/version.rb +1 -1
  32. data/lib/sidekiq/web/action.rb +0 -4
  33. data/lib/sidekiq/web/application.rb +8 -13
  34. data/lib/sidekiq/web/helpers.rb +54 -16
  35. data/lib/sidekiq/worker.rb +99 -16
  36. data/sidekiq.gemspec +3 -7
  37. data/web/assets/javascripts/dashboard.js +17 -12
  38. data/web/assets/stylesheets/application-rtl.css +246 -0
  39. data/web/assets/stylesheets/application.css +336 -4
  40. data/web/assets/stylesheets/bootstrap-rtl.min.css +9 -0
  41. data/web/locales/ar.yml +80 -0
  42. data/web/locales/fa.yml +1 -0
  43. data/web/locales/he.yml +79 -0
  44. data/web/locales/ur.yml +80 -0
  45. data/web/views/_footer.erb +2 -2
  46. data/web/views/_nav.erb +1 -1
  47. data/web/views/_paging.erb +1 -1
  48. data/web/views/busy.erb +9 -5
  49. data/web/views/dashboard.erb +1 -1
  50. data/web/views/layout.erb +10 -1
  51. data/web/views/morgue.erb +4 -4
  52. data/web/views/queue.erb +7 -7
  53. data/web/views/retries.erb +5 -5
  54. data/web/views/scheduled.erb +2 -2
  55. metadata +20 -83
  56. data/lib/sidekiq/middleware/server/logging.rb +0 -31
  57. data/lib/sidekiq/middleware/server/retry_jobs.rb +0 -205
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 595500a3810c25e8e04e86c644f966ff01cb035f
4
- data.tar.gz: 3f92a9f61e61848bc2c0639469ce3588f3848fc0
3
+ metadata.gz: 96202101069aaee71e462e2f3b91f5d4c58e952f
4
+ data.tar.gz: ad75e0b5dc019bb8d1058b124535190f00b06f60
5
5
  SHA512:
6
- metadata.gz: c4e97c3665d1e902f47cfb579fda437dc834ac4b584606f1ca3b8fd9225e8502cacf5fb687673b4735b359de6af179589759a34369f1c74e904b37f8fe70a4ee
7
- data.tar.gz: a4e6d41067bf6b88c77577def0c1f9ecc3f5d46f0bdfb0821952c26867e91a71c64231aab09a25096d22c3272dea2dcc37547bc6096dd7622a192fa00e0a4e50
6
+ metadata.gz: 4de121e1a00611b97f5c8838c5beb1d5a4fa5e6fb37e7cd7b2ac2074239d8a48d1d9526d4a8c0a94fcfcebe7ee3507209735569c78309f18d29363fbab24b992
7
+ data.tar.gz: 239a247dd69bab58148229f64d7b8ff12a658bb696c08c9f00b9ee9170c5d40ea052891eef5d89742633282d99095b4eb6004df298aa2f06b5f114fbbcfb10de
data/.gitignore CHANGED
@@ -10,3 +10,4 @@ vendor/
10
10
  .bundle/
11
11
  .sass-cache/
12
12
  tmp/
13
+ pkg/*.gem
@@ -6,13 +6,8 @@ services:
6
6
  before_install:
7
7
  - gem install bundler
8
8
  - gem update bundler
9
- matrix:
10
- include:
11
- - rvm: 2.2.4
12
- env: "PERCY_ENABLE=0"
13
- - rvm: 2.3.0
14
- env: "PERCY_ENABLE=0"
15
- - rvm: 2.4.0
16
- env: "PERCY_ENABLE=1"
17
- - rvm: jruby-9.1.6.0
18
- env: "PERCY_ENABLE=0"
9
+ rvm:
10
+ - 2.2.4
11
+ - 2.3.0
12
+ - 2.4.0
13
+ - jruby-9.1.6.0
@@ -0,0 +1,56 @@
1
+ # Welcome to Sidekiq 5.0!
2
+
3
+ Sidekiq 5.0 contains a reworked job dispatch and execution core to integrate
4
+ better with the new Rails 5.0 Executor. It also drops support for older
5
+ versions of Ruby and Rails and adds support for RTL languages in the Web UI.
6
+
7
+ ## What's New
8
+
9
+ * Integrate job logging and retry logic directly in with the job
10
+ execution logic in Sidekiq::Processor. Previously this logic was
11
+ defined as middleware. In Rails 5.0, ActiveSupport::Executor handles ActiveRecord
12
+ connection management, job callbacks, development mode class loading,
13
+ etc. Because of its extensive responsibilities, the Executor can't be
14
+ integrated as Sidekiq middleware; the logging/retry logic had to be pulled out
15
+ too. Sidekiq 4.2 had a hack to make it work but this redesign provides
16
+ a cleaner integration. [#3235]
17
+ * The Delayed Extensions `delay`, `delay_in` and `delay_until` APIs are
18
+ no longer available by default. The extensions allow you to marshal
19
+ job arguments as YAML, leading to cases where job payloads could be many
20
+ 100s of KB or larger if not careful, leading to Redis networking
21
+ timeouts or other problems. As noted in the Best Practices wiki page,
22
+ Sidekiq is designed for jobs with small, simple arguments.
23
+
24
+ Add this line to your initializer to re-enable them and get the old behavior:
25
+ ```ruby
26
+ Sidekiq::Extensions.enable_delay!
27
+ ```
28
+ The old `Sidekiq.remove_delay!` API has been removed as it is now the default. [#3299]
29
+ * Sidekiq's quiet signal is now `TSTP` (think of it as **T**hread
30
+ **ST**o**P**) instead of USR1 as USR1 is not available on JRuby.
31
+ USR1 will continue to be supported in Sidekiq 5.x for backwards
32
+ compatibility and will be removed in Sidekiq 6.x. [#3302]
33
+ * The Web UI is now bi-directional - it can render either LTR
34
+ (left-to-right) or RTL languages. With this change, **Farsi, Arabic,
35
+ Hebrew and Urdu** are officially supported. [#3381]
36
+ * Jobs which can't be parsed due to invalid JSON are now pushed
37
+ immediately to the Dead set since they require manual intervention and
38
+ will never execute successfully as is. The Web UI has been updated to
39
+ more gracefully display these jobs. [#3296]
40
+ * **Rails 3.2** is no longer supported.
41
+ * **Ruby 2.0 and Ruby 2.1** are no longer supported. Ruby 2.2.2+ is required.
42
+
43
+ ## Upgrade
44
+
45
+ As always, please upgrade Sidekiq **one major version at a time**.
46
+ If you are already running Sidekiq 4.x, then:
47
+
48
+ * Upgrade to the latest Sidekiq 4.x.
49
+ ```ruby
50
+ gem 'sidekiq', '< 5'
51
+ ```
52
+ * Fix any deprecation warnings you see.
53
+ * Upgrade to 5.x.
54
+ ```ruby
55
+ gem 'sidekiq', '< 6'
56
+ ```
@@ -38,7 +38,7 @@ under the terms of the GNU Lesser General Public License versions 3.0
38
38
 
39
39
  5. Fees and Payment. The Software license fees will be due and payable in full as set forth in the applicable invoice or at the time of purchase. If the Software does not function properly within two weeks of purchase, please contact us within those two weeks for a refund. You shall be responsible for all taxes, withholdings, duties and levies arising from the order (excluding taxes based on the net income of Contributed Systems).
40
40
 
41
- 6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the Sidekiq Pro support page (https://github.com/mperham/sidekiq/wiki/Pro-Support), support and maintenance services may be included with the purchase of your license subscription.
41
+ 6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the Sidekiq Pro support page (https://github.com/mperham/sidekiq/wiki/Commercial-Support), support and maintenance services may be included with the purchase of your license subscription.
42
42
 
43
43
  7. Term of Agreement.
44
44
 
data/Changes.md CHANGED
@@ -1,8 +1,68 @@
1
1
  # Sidekiq Changes
2
2
 
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
4
+
3
5
  HEAD
4
6
  -----------
5
7
 
8
+ - Update gemspec to allow newer versions of the Redis gem [#3617]
9
+ - Refactor Worker.set so it can be memoized [#3602]
10
+ - Fix display of Redis URL in web footer, broken in 5.0.3 [#3560]
11
+
12
+ 5.0.4
13
+ -----------
14
+
15
+ - Fix "slow startup" performance regression from 5.0.2. [#3525]
16
+ - Allow users to disable ID generation since some redis providers disable the CLIENT command. [#3521]
17
+
18
+ 5.0.3
19
+ -----------
20
+
21
+ - Fix overriding `class_attribute` core extension from ActiveSupport with Sidekiq one [PikachuEXE, #3499]
22
+ - Allow job logger to be overridden [AlfonsoUceda, #3502]
23
+ - Set a default Redis client identifier for debugging [#3516]
24
+ - Fix "Uninitialized constant" errors on startup with the delayed extensions [#3509]
25
+
26
+ 5.0.2
27
+ -----------
28
+
29
+ - fix broken release, thanks @nateberkopec
30
+
31
+ 5.0.1
32
+ -----------
33
+
34
+ - Fix incorrect server identity when daemonizing [jwilm, #3496]
35
+ - Work around error running Web UI against Redis Cluster [#3492]
36
+ - Remove core extensions, Sidekiq is now monkeypatch-free! [#3474]
37
+ - Reimplement Web UI's HTTP\_ACCEPT\_LANGUAGE parsing because the spec is utterly
38
+ incomprehensible for various edge cases. [johanlunds, natematykiewicz, #3449]
39
+ - Update `class_attribute` core extension to avoid warnings
40
+ - Expose `job_hash_context` from `Sidekiq::Logging` to support log customization
41
+
42
+ 5.0.0
43
+ -----------
44
+
45
+ - **BREAKING CHANGE** Job dispatch was refactored for safer integration with
46
+ Rails 5. The **Logging** and **RetryJobs** server middleware were removed and
47
+ functionality integrated directly into Sidekiq::Processor. These aren't
48
+ commonly used public APIs so this shouldn't impact most users.
49
+ ```
50
+ Sidekiq::Middleware::Server::RetryJobs -> Sidekiq::JobRetry
51
+ Sidekiq::Middleware::Server::Logging -> Sidekiq::JobLogger
52
+ ```
53
+ - Quieting Sidekiq is now done via the TSTP signal, the USR1 signal is deprecated.
54
+ - The `delay` extension APIs are no longer available by default, you
55
+ must opt into them.
56
+ - The Web UI is now BiDi and can render RTL languages like Arabic, Farsi and Hebrew.
57
+ - Rails 3.2 and Ruby 2.0 and 2.1 are no longer supported.
58
+ - The `SomeWorker.set(options)` API was re-written to avoid thread-local state. [#2152]
59
+ - Sidekiq Enterprise's encrypted jobs now display "[encrypted data]" in the Web UI instead
60
+ of random hex bytes.
61
+ - Please see the [5.0 Upgrade notes](5.0-Upgrade.md) for more detail.
62
+
63
+ 4.2.10
64
+ -----------
65
+
6
66
  - Scheduled jobs can now be moved directly to the Dead queue via API [#3390]
7
67
  - Fix edge case leading to job duplication when using Sidekiq Pro's
8
68
  reliability feature [#3388]
@@ -1,8 +1,35 @@
1
- Sidekiq Enterprise Changelog
2
- =======================
1
+ # Sidekiq Enterprise Changelog
2
+
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
3
4
 
4
5
  Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
5
6
 
7
+ 1.6.1
8
+ -------------
9
+
10
+ - Fix crash in rate limiter middleware when used with custom exceptions [#3604]
11
+
12
+ 1.6.0
13
+ -------------
14
+
15
+ - Show process "leader" tag on Busy page, requires Sidekiq 5.0.2 [#2867]
16
+ - Capture custom metrics with the `save_history` API. [#2815]
17
+ - Implement new `unique_until: 'start'` policy option. [#3471]
18
+
19
+ 1.5.4
20
+ -------------
21
+
22
+ - Fix broken Cron page in Web UI [#3458]
23
+
24
+ 1.5.3
25
+ -------------
26
+
27
+ - Remove dependency on the algorithms gem [#3446]
28
+ - Allow user to specify max memory in megabytes with SIDEKIQ\_MAXMEM\_MB [#3451]
29
+ - Implement logic to detect app startup failure, sidekiqswarm will exit
30
+ rather than try to restart the app forever [#3450]
31
+ - Another fix for doubly-encrypted arguments [#3368]
32
+
6
33
  1.5.2
7
34
  -------------
8
35
 
data/Gemfile CHANGED
@@ -8,6 +8,9 @@ gem 'minitest'
8
8
  #gem 'minitest-utils'
9
9
  gem 'toxiproxy'
10
10
 
11
+ # For Redis 4.0 support. Unreleased 9cb81bf.
12
+ gem 'redis-namespace', git: 'https://github.com/resque/redis-namespace'
13
+
11
14
  platforms :rbx do
12
15
  gem 'rubysl', '~> 2.0' # if using anything in the ruby standard library
13
16
  gem 'psych' # if using yaml
@@ -1,11 +1,36 @@
1
- Sidekiq Pro Changelog
2
- =======================
1
+ # Sidekiq Pro Changelog
2
+
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
3
4
 
4
5
  Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
5
6
 
6
- HEAD
7
+ 3.5.3
8
+ ---------
9
+
10
+ - Restore error check for super\_fetch's job ack [#3601]
11
+ - Trim error messages saved in Batch's failure hash, preventing huge
12
+ messages from bloating Redis. [#3570]
13
+
14
+ 3.5.2
15
+ ---------
16
+
17
+ - Fix `Status#completed?` when run against a Batch that had succeeded
18
+ and was deleted. [#3519]
19
+
20
+ 3.5.1
21
+ ---------
22
+
23
+ - Work with Sidekiq 5.0.2+
24
+ - Improve performance of super\_fetch with weighted queues [#3489]
25
+
26
+ 3.5.0
7
27
  ---------
8
28
 
29
+ - Add queue pause/unpause endpoints for scripting via curl [#3445]
30
+ - Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
31
+ - Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
32
+ - Batch page displays Pending JIDs if less than 10 [#3130]
33
+ - Batch page has a Search button to find associated Retries [#3130]
9
34
  - Make Batch UI progress bar more friendly to the colorblind [#3387]
10
35
 
11
36
  3.4.5
data/README.md CHANGED
@@ -28,13 +28,14 @@ Sidekiq 3.5.1 | 22ms | 1257 MB | 125 sec | 800 jobs/sec
28
28
  Resque 1.25.2 | - | - | 420 sec | 240 jobs/sec
29
29
  DelayedJob 4.1.1 | - | - | 465 sec | 215 jobs/sec
30
30
 
31
+ <small>This benchmark can be found in `bin/sidekiqload`.</small>
31
32
 
32
33
  Requirements
33
34
  -----------------
34
35
 
35
- Sidekiq supports CRuby 2.0+ and JRuby 9k.
36
+ Sidekiq supports CRuby 2.2.2+ and JRuby 9k.
36
37
 
37
- All Rails releases >= 3.2 are officially supported.
38
+ All Rails releases >= 4.0 are officially supported.
38
39
 
39
40
  Redis 2.8 or greater is required. 3.0.3+ is recommended for large
40
41
  installations with thousands of worker threads.
@@ -80,7 +81,6 @@ Useful resources:
80
81
 
81
82
  * Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
82
83
  * Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
83
- * Here's a [Reddit forum](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion
84
84
  * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q &amp; A.
85
85
 
86
86
  **No support via Twitter, 140 characters is not enough.**
@@ -64,7 +64,7 @@ class Sidekiqctl
64
64
  end
65
65
 
66
66
  def quiet
67
- `kill -USR1 #{pid}`
67
+ `kill -TSTP #{pid}`
68
68
  end
69
69
 
70
70
  def stop
@@ -46,7 +46,7 @@ end
46
46
  #}])
47
47
 
48
48
  self_read, self_write = IO.pipe
49
- %w(INT TERM USR1 USR2 TTIN).each do |sig|
49
+ %w(INT TERM TSTP TTIN).each do |sig|
50
50
  begin
51
51
  trap sig do
52
52
  self_write.puts(sig)
@@ -67,14 +67,9 @@ def handle_signal(launcher, sig)
67
67
  when 'TERM'
68
68
  # Heroku sends TERM and then waits 10 seconds for process to exit.
69
69
  raise Interrupt
70
- when 'USR1'
71
- Sidekiq.logger.info "Received USR1, no longer accepting new work"
70
+ when 'TSTP'
71
+ Sidekiq.logger.info "Received TSTP, no longer accepting new work"
72
72
  launcher.quiet
73
- when 'USR2'
74
- if Sidekiq.options[:logfile]
75
- Sidekiq.logger.info "Received USR2, reopening log file"
76
- Sidekiq::Logging.reopen_logs
77
- end
78
73
  when 'TTIN'
79
74
  Thread.list.each do |thread|
80
75
  Sidekiq.logger.warn "Thread TID-#{thread.object_id.to_s(36)} #{thread['label']}"
@@ -1,6 +1,6 @@
1
1
  require 'rails_helper'
2
2
  <% module_namespacing do -%>
3
3
  RSpec.describe <%= class_name %>Worker, type: :worker do
4
- pending "add some examples to (or delete) #{__FILE__}"
4
+ pending "add some examples to (or delete) #{__FILE__}"
5
5
  end
6
6
  <% end -%>
@@ -1,17 +1,18 @@
1
1
  # encoding: utf-8
2
2
  # frozen_string_literal: true
3
3
  require 'sidekiq/version'
4
- fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.0.0." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.0.0'
4
+ fail "Sidekiq #{Sidekiq::VERSION} does not support Ruby versions below 2.2.2." if RUBY_PLATFORM != 'java' && RUBY_VERSION < '2.2.2'
5
5
 
6
6
  require 'sidekiq/logging'
7
7
  require 'sidekiq/client'
8
8
  require 'sidekiq/worker'
9
9
  require 'sidekiq/redis_connection'
10
+ require 'sidekiq/delay'
10
11
 
11
12
  require 'json'
12
13
 
13
14
  module Sidekiq
14
- NAME = 'Sidekiq'
15
+ NAME = 'Sidekiq'.freeze
15
16
  LICENSE = 'See LICENSE and the LGPL-3.0 for licensing details.'
16
17
 
17
18
  DEFAULTS = {
@@ -33,7 +34,6 @@ module Sidekiq
33
34
  dead_max_jobs: 10_000,
34
35
  dead_timeout_in_seconds: 180 * 24 * 60 * 60, # 6 months
35
36
  reloader: proc { |&block| block.call },
36
- executor: proc { |&block| block.call },
37
37
  }
38
38
 
39
39
  DEFAULT_WORKER_OPTIONS = {
@@ -145,17 +145,12 @@ module Sidekiq
145
145
  end
146
146
 
147
147
  def self.default_server_middleware
148
- require 'sidekiq/middleware/server/retry_jobs'
149
- require 'sidekiq/middleware/server/logging'
150
-
151
- Middleware::Chain.new do |m|
152
- m.add Middleware::Server::RetryJobs
153
- m.add Middleware::Server::Logging
154
- end
148
+ Middleware::Chain.new
155
149
  end
156
150
 
157
151
  def self.default_worker_options=(hash)
158
- @default_worker_options = default_worker_options.merge(hash.stringify_keys)
152
+ # stringify
153
+ @default_worker_options = default_worker_options.merge(Hash[hash.map{|k, v| [k.to_s, v]}])
159
154
  end
160
155
  def self.default_worker_options
161
156
  defined?(@default_worker_options) ? @default_worker_options : DEFAULT_WORKER_OPTIONS
@@ -228,10 +223,6 @@ module Sidekiq
228
223
  # otherwise Ruby's Thread#kill will commit. See #377.
229
224
  # DO NOT RESCUE THIS ERROR IN YOUR WORKERS
230
225
  class Shutdown < Interrupt; end
231
-
232
226
  end
233
227
 
234
- require 'sidekiq/extensions/class_methods'
235
- require 'sidekiq/extensions/action_mailer'
236
- require 'sidekiq/extensions/active_record'
237
228
  require 'sidekiq/rails' if defined?(::Rails::Engine)
@@ -75,7 +75,7 @@ module Sidekiq
75
75
  enqueued = pipe2_res[s..-1].map(&:to_i).inject(0, &:+)
76
76
 
77
77
  default_queue_latency = if (entry = pipe1_res[6].first)
78
- job = Sidekiq.load_json(entry)
78
+ job = Sidekiq.load_json(entry) rescue {}
79
79
  now = Time.now.to_f
80
80
  thence = job['enqueued_at'.freeze] || now
81
81
  now - thence
@@ -146,11 +146,11 @@ module Sidekiq
146
146
  end
147
147
 
148
148
  def processed
149
- date_stat_hash("processed")
149
+ @processed ||= date_stat_hash("processed")
150
150
  end
151
151
 
152
152
  def failed
153
- date_stat_hash("failed")
153
+ @failed ||= date_stat_hash("failed")
154
154
  end
155
155
 
156
156
  private
@@ -169,10 +169,15 @@ module Sidekiq
169
169
  i += 1
170
170
  end
171
171
 
172
- Sidekiq.redis do |conn|
173
- conn.mget(keys).each_with_index do |value, idx|
174
- stat_hash[dates[idx]] = value ? value.to_i : 0
172
+ begin
173
+ Sidekiq.redis do |conn|
174
+ conn.mget(keys).each_with_index do |value, idx|
175
+ stat_hash[dates[idx]] = value ? value.to_i : 0
176
+ end
175
177
  end
178
+ rescue Redis::CommandError
179
+ # mget will trigger a CROSSSLOT error when run against a Cluster
180
+ # TODO Someone want to add Cluster support?
176
181
  end
177
182
 
178
183
  stat_hash
@@ -287,13 +292,25 @@ module Sidekiq
287
292
  attr_reader :value
288
293
 
289
294
  def initialize(item, queue_name=nil)
295
+ @args = nil
290
296
  @value = item
291
- @item = item.is_a?(Hash) ? item : Sidekiq.load_json(item)
297
+ @item = item.is_a?(Hash) ? item : parse(item)
292
298
  @queue = queue_name || @item['queue']
293
299
  end
294
300
 
301
+ def parse(item)
302
+ Sidekiq.load_json(item)
303
+ rescue JSON::ParserError
304
+ # If the job payload in Redis is invalid JSON, we'll load
305
+ # the item as an empty hash and store the invalid JSON as
306
+ # the job 'args' for display in the Web UI.
307
+ @invalid = true
308
+ @args = [item]
309
+ {}
310
+ end
311
+
295
312
  def klass
296
- @item['class']
313
+ self['class']
297
314
  end
298
315
 
299
316
  def display_class
@@ -318,38 +335,42 @@ module Sidekiq
318
335
 
319
336
  def display_args
320
337
  # Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
321
- @args ||= case klass
338
+ @display_args ||= case klass
322
339
  when /\ASidekiq::Extensions::Delayed/
323
340
  safe_load(args[0], args) do |_, _, arg|
324
341
  arg
325
342
  end
326
343
  when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
327
- job_args = @item['wrapped'] ? args[0]["arguments"] : []
328
- if 'ActionMailer::DeliveryJob' == (@item['wrapped'] || args[0])
329
- # remove MailerClass, mailer_method and 'deliver_now'
330
- job_args.drop(3)
344
+ job_args = self['wrapped'] ? args[0]["arguments"] : []
345
+ if 'ActionMailer::DeliveryJob' == (self['wrapped'] || args[0])
346
+ # remove MailerClass, mailer_method and 'deliver_now'
347
+ job_args.drop(3)
331
348
  else
332
- job_args
349
+ job_args
333
350
  end
334
351
  else
352
+ if self['encrypt'.freeze]
353
+ # no point in showing 150+ bytes of random garbage
354
+ args[-1] = '[encrypted data]'.freeze
355
+ end
335
356
  args
336
357
  end
337
358
  end
338
359
 
339
360
  def args
340
- @item['args']
361
+ @args || @item['args']
341
362
  end
342
363
 
343
364
  def jid
344
- @item['jid']
365
+ self['jid']
345
366
  end
346
367
 
347
368
  def enqueued_at
348
- @item['enqueued_at'] ? Time.at(@item['enqueued_at']).utc : nil
369
+ self['enqueued_at'] ? Time.at(self['enqueued_at']).utc : nil
349
370
  end
350
371
 
351
372
  def created_at
352
- Time.at(@item['created_at'] || @item['enqueued_at'] || 0).utc
373
+ Time.at(self['created_at'] || self['enqueued_at'] || 0).utc
353
374
  end
354
375
 
355
376
  def queue
@@ -371,7 +392,10 @@ module Sidekiq
371
392
  end
372
393
 
373
394
  def [](name)
374
- @item[name]
395
+ # nil will happen if the JSON fails to parse.
396
+ # We don't guarantee Sidekiq will work with bad job JSON but we should
397
+ # make a best effort to minimize the damage.
398
+ @item ? @item[name] : nil
375
399
  end
376
400
 
377
401
  private
@@ -434,14 +458,7 @@ module Sidekiq
434
458
  # Place job in the dead set
435
459
  def kill
436
460
  remove_job do |message|
437
- now = Time.now.to_f
438
- Sidekiq.redis do |conn|
439
- conn.multi do
440
- conn.zadd('dead', now, message)
441
- conn.zremrangebyscore('dead', '-inf', now - DeadSet.timeout)
442
- conn.zremrangebyrank('dead', 0, - DeadSet.max_jobs)
443
- end
444
- end
461
+ DeadSet.new.kill(message)
445
462
  end
446
463
  end
447
464
 
@@ -639,6 +656,17 @@ module Sidekiq
639
656
  super 'dead'
640
657
  end
641
658
 
659
+ def kill(message)
660
+ now = Time.now.to_f
661
+ Sidekiq.redis do |conn|
662
+ conn.multi do
663
+ conn.zadd(name, now.to_s, message)
664
+ conn.zremrangebyscore(name, '-inf', now - self.class.timeout)
665
+ conn.zremrangebyrank(name, 0, - self.class.max_jobs)
666
+ end
667
+ end
668
+ end
669
+
642
670
  def retry_all
643
671
  while size > 0
644
672
  each(&:retry)
@@ -706,6 +734,11 @@ module Sidekiq
706
734
  end
707
735
 
708
736
  result.each do |info, busy, at_s, quiet|
737
+ # If a process is stopped between when we query Redis for `procs` and
738
+ # when we query for `result`, we will have an item in `result` that is
739
+ # composed of `nil` values.
740
+ next if info.nil?
741
+
709
742
  hash = Sidekiq.load_json(info)
710
743
  yield Process.new(hash.merge('busy' => busy.to_i, 'beat' => at_s.to_f, 'quiet' => quiet))
711
744
  end
@@ -721,6 +754,18 @@ module Sidekiq
721
754
  def size
722
755
  Sidekiq.redis { |conn| conn.scard('processes') }
723
756
  end
757
+
758
+ # Returns the identity of the current cluster leader or "" if no leader.
759
+ # This is a Sidekiq Enterprise feature, will always return "" in Sidekiq
760
+ # or Sidekiq Pro.
761
+ def leader
762
+ @leader ||= begin
763
+ x = Sidekiq.redis {|c| c.get("dear-leader") }
764
+ # need a non-falsy value so we can memoize
765
+ x = "" unless x
766
+ x
767
+ end
768
+ end
724
769
  end
725
770
 
726
771
  #
@@ -755,8 +800,12 @@ module Sidekiq
755
800
  @attribs[key]
756
801
  end
757
802
 
803
+ def identity
804
+ self['identity']
805
+ end
806
+
758
807
  def quiet!
759
- signal('USR1')
808
+ signal('TSTP')
760
809
  end
761
810
 
762
811
  def stop!
@@ -783,9 +832,6 @@ module Sidekiq
783
832
  end
784
833
  end
785
834
 
786
- def identity
787
- self['identity']
788
- end
789
835
  end
790
836
 
791
837
  ##