sidekiq 6.0.6 → 6.2.2

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

Files changed (84) hide show
  1. checksums.yaml +4 -4
  2. data/Changes.md +100 -1
  3. data/LICENSE +1 -1
  4. data/README.md +2 -6
  5. data/bin/sidekiq +26 -2
  6. data/lib/sidekiq/api.rb +101 -60
  7. data/lib/sidekiq/cli.rb +24 -8
  8. data/lib/sidekiq/client.rb +16 -15
  9. data/lib/sidekiq/extensions/action_mailer.rb +3 -2
  10. data/lib/sidekiq/extensions/active_record.rb +4 -3
  11. data/lib/sidekiq/extensions/class_methods.rb +5 -4
  12. data/lib/sidekiq/extensions/generic_proxy.rb +3 -1
  13. data/lib/sidekiq/fetch.rb +29 -21
  14. data/lib/sidekiq/job.rb +8 -0
  15. data/lib/sidekiq/job_logger.rb +1 -1
  16. data/lib/sidekiq/job_retry.rb +4 -7
  17. data/lib/sidekiq/launcher.rb +92 -30
  18. data/lib/sidekiq/logger.rb +3 -2
  19. data/lib/sidekiq/manager.rb +4 -4
  20. data/lib/sidekiq/middleware/chain.rb +6 -4
  21. data/lib/sidekiq/processor.rb +4 -4
  22. data/lib/sidekiq/rails.rb +16 -18
  23. data/lib/sidekiq/redis_connection.rb +18 -13
  24. data/lib/sidekiq/scheduled.rb +7 -1
  25. data/lib/sidekiq/sd_notify.rb +1 -1
  26. data/lib/sidekiq/systemd.rb +1 -15
  27. data/lib/sidekiq/testing.rb +2 -4
  28. data/lib/sidekiq/util.rb +28 -0
  29. data/lib/sidekiq/version.rb +1 -1
  30. data/lib/sidekiq/web/action.rb +2 -2
  31. data/lib/sidekiq/web/application.rb +15 -9
  32. data/lib/sidekiq/web/csrf_protection.rb +180 -0
  33. data/lib/sidekiq/web/helpers.rb +34 -17
  34. data/lib/sidekiq/web/router.rb +5 -2
  35. data/lib/sidekiq/web.rb +36 -72
  36. data/lib/sidekiq/worker.rb +2 -5
  37. data/lib/sidekiq.rb +5 -3
  38. data/sidekiq.gemspec +11 -4
  39. data/web/assets/images/apple-touch-icon.png +0 -0
  40. data/web/assets/javascripts/application.js +3 -8
  41. data/web/assets/stylesheets/application-dark.css +65 -40
  42. data/web/assets/stylesheets/application.css +35 -135
  43. data/web/locales/ar.yml +8 -2
  44. data/web/locales/en.yml +3 -0
  45. data/web/locales/es.yml +18 -2
  46. data/web/locales/fr.yml +10 -3
  47. data/web/locales/ja.yml +3 -0
  48. data/web/locales/lt.yml +1 -1
  49. data/web/locales/pl.yml +4 -4
  50. data/web/locales/ru.yml +4 -0
  51. data/web/locales/vi.yml +83 -0
  52. data/web/views/_job_info.erb +1 -1
  53. data/web/views/busy.erb +50 -19
  54. data/web/views/dashboard.erb +14 -6
  55. data/web/views/dead.erb +1 -1
  56. data/web/views/layout.erb +1 -0
  57. data/web/views/morgue.erb +6 -6
  58. data/web/views/queue.erb +1 -1
  59. data/web/views/queues.erb +4 -4
  60. data/web/views/retries.erb +7 -7
  61. data/web/views/retry.erb +1 -1
  62. data/web/views/scheduled.erb +1 -1
  63. metadata +19 -45
  64. data/.circleci/config.yml +0 -60
  65. data/.github/contributing.md +0 -32
  66. data/.github/issue_template.md +0 -11
  67. data/.gitignore +0 -13
  68. data/.standard.yml +0 -20
  69. data/3.0-Upgrade.md +0 -70
  70. data/4.0-Upgrade.md +0 -53
  71. data/5.0-Upgrade.md +0 -56
  72. data/6.0-Upgrade.md +0 -72
  73. data/COMM-LICENSE +0 -97
  74. data/Ent-2.0-Upgrade.md +0 -37
  75. data/Ent-Changes.md +0 -256
  76. data/Gemfile +0 -24
  77. data/Gemfile.lock +0 -208
  78. data/Pro-2.0-Upgrade.md +0 -138
  79. data/Pro-3.0-Upgrade.md +0 -44
  80. data/Pro-4.0-Upgrade.md +0 -35
  81. data/Pro-5.0-Upgrade.md +0 -25
  82. data/Pro-Changes.md +0 -782
  83. data/Rakefile +0 -10
  84. data/code_of_conduct.md +0 -50
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: a9cb4face6040bd8635e465de260ac2ee28813fceea44bb0c8974989a723d799
4
- data.tar.gz: 3537c0f75e1a005359d184c63a462806a661c3c8192ea6e9907f321d2f87221b
3
+ metadata.gz: 1b0da541124d097e05936cc860a93009ba170d714cbd8dbc760c1245818e8ed7
4
+ data.tar.gz: 63995ccbb57fa16e4954cda4ab9610506b0a8ff35084492b520fc18608face35
5
5
  SHA512:
6
- metadata.gz: f54b4c103a10f5a82716e8bc38ef599bf29e63c713e047703d5af5fc1aafcbde8502ceffd81d1ed7211fe28c8d15f1531fd1fe5881881739dec50acf26b87d7b
7
- data.tar.gz: 352bd20c250c604327cf2aefd599915c96990d1d53a4d1d0d611f08ebe35db7dca6862bbccd22a9aa3709529fc7889046cc2775c1a14de1fc4bd2bb59026bac5
6
+ metadata.gz: f3b864db74530840437c0af43475ed0e12a876d6d2cbbd808ecc72722adddbd8d5548c9b777d4e34af5deddf94871567717518f70d787ea7f6a089f2f5b2f4ed
7
+ data.tar.gz: 4569eed5baa10134e99a0b1404c213e3b35c08ec158e2fa8b006918298750919e8d34074aa10d3b4dfb621bb18cbe68be467d1e7822fb855012cff0f64dcdc7b
data/Changes.md CHANGED
@@ -2,6 +2,99 @@
2
2
 
3
3
  [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
4
4
 
5
+ 6.2.2
6
+ ---------
7
+
8
+ - Reduce retry jitter, add jitter to `sidekiq_retry_in` values [#4957]
9
+ - Minimize scheduler load on Redis at scale [#4882]
10
+ - Improve logging of delay jobs [#4904, BuonOno]
11
+ - Minor CSS improvements for buttons and tables, design PRs always welcome!
12
+ - Tweak Web UI `Cache-Control` header [#4966]
13
+ - Rename internal API class `Sidekiq::Job` to `Sidekiq::JobRecord` [#4955]
14
+
15
+ 6.2.1
16
+ ---------
17
+
18
+ - Update RTT warning logic to handle transient RTT spikes [#4851]
19
+ - Fix very low priority CVE on unescaped queue name [#4852]
20
+ - Add note about sessions and Rails apps in API mode
21
+
22
+ 6.2.0
23
+ ---------
24
+
25
+ - Store Redis RTT and log if poor [#4824]
26
+ - Add process/thread stats to Busy page [#4806]
27
+ - Improve Web UI on mobile devices [#4840]
28
+ - **Refactor Web UI session usage** [#4804]
29
+ Numerous people have hit "Forbidden" errors and struggled with Sidekiq's
30
+ Web UI session requirement. If you have code in your initializer for
31
+ Web sessions, it's quite possible it will need to be removed. Here's
32
+ an overview:
33
+ ```
34
+ Sidekiq::Web needs a valid Rack session for CSRF protection. If this is a Rails app,
35
+ make sure you mount Sidekiq::Web *inside* your routes in `config/routes.rb` so
36
+ Sidekiq can reuse the Rails session:
37
+
38
+ Rails.application.routes.draw do
39
+ mount Sidekiq::Web => "/sidekiq"
40
+ ....
41
+ end
42
+
43
+ If this is a bare Rack app, use a session middleware before Sidekiq::Web:
44
+
45
+ # first, use IRB to create a shared secret key for sessions and commit it
46
+ require 'securerandom'; File.open(".session.key", "w") {|f| f.write(SecureRandom.hex(32)) }
47
+
48
+ # now, update your Rack app to include the secret with a session cookie middleware
49
+ use Rack::Session::Cookie, secret: File.read(".session.key"), same_site: true, max_age: 86400
50
+ run Sidekiq::Web
51
+
52
+ If this is a Rails app in API mode, you need to enable sessions.
53
+
54
+ https://guides.rubyonrails.org/api_app.html#using-session-middlewares
55
+ ```
56
+
57
+ 6.1.3
58
+ ---------
59
+
60
+ - Warn if Redis is configured to evict data under memory pressure [#4752]
61
+ - Add process RSS on the Busy page [#4717]
62
+
63
+ 6.1.2
64
+ ---------
65
+
66
+ - Improve readability in dark mode Web UI [#4674]
67
+ - Fix Web UI crash with corrupt session [#4672]
68
+ - Allow middleware to yield arguments [#4673, @eugeneius]
69
+ - Migrate CI from CircleCI to GitHub Actions [#4677]
70
+
71
+ 6.1.1
72
+ ---------
73
+
74
+ - Jobs are now sorted by age in the Busy Workers table. [#4641]
75
+ - Fix "check all" JS logic in Web UI [#4619]
76
+
77
+ 6.1.0
78
+ ---------
79
+
80
+ - Web UI - Dark Mode fixes [#4543, natematykiewicz]
81
+ - Ensure `Rack::ContentLength` is loaded as middleware for correct Web UI responses [#4541]
82
+ - Avoid exception dumping SSL store in Redis connection logging [#4532]
83
+ - Better error messages in Sidekiq::Client [#4549]
84
+ - Remove rack-protection, reimplement CSRF protection [#4588]
85
+ - Require redis-rb 4.2 [#4591]
86
+ - Update to jquery 1.12.4 [#4593]
87
+ - Refactor internal fetch logic and API [#4602]
88
+
89
+ 6.0.7
90
+ ---------
91
+
92
+ - Refactor systemd integration to work better with custom binaries [#4511]
93
+ - Don't connect to Redis at process exit if not needed [#4502]
94
+ - Remove Redis connection naming [#4479]
95
+ - Fix Redis Sentinel password redaction [#4499]
96
+ - Add Vietnamese locale (vi) [#4528]
97
+
5
98
  6.0.6
6
99
  ---------
7
100
 
@@ -115,7 +208,7 @@ assert_equal 1, Sidekiq::Extensions::DelayedMailer.jobs_for(FooMailer).size
115
208
 
116
209
  This release has major breaking changes. Read and test carefully in production.
117
210
 
118
- - With Rails 6.0.1+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
211
+ - With Rails 6.0.2+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
119
212
  features/internals like the retry subsystem. [#4213, pirj]
120
213
  ```ruby
121
214
  class MyJob < ActiveJob::Base
@@ -151,6 +244,12 @@ See the [Logging wiki page](https://github.com/mperham/sidekiq/wiki/Logging) for
151
244
  - Integrate the StandardRB code formatter to ensure consistent code
152
245
  styling. [#4114, gearnode]
153
246
 
247
+ 5.2.9
248
+ ---------
249
+
250
+ - Release Rack lock due to a cascade of CVEs. [#4566]
251
+ Pro-tip: don't lock Rack.
252
+
154
253
  5.2.8
155
254
  ---------
156
255
 
data/LICENSE CHANGED
@@ -6,4 +6,4 @@ for license text.
6
6
 
7
7
  Sidekiq Pro has a commercial-friendly license allowing private forks
8
8
  and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
9
- more detail. You can find the commercial license terms in COMM-LICENSE.
9
+ more detail. You can find the commercial license terms in COMM-LICENSE.txt.
data/README.md CHANGED
@@ -2,11 +2,7 @@ Sidekiq
2
2
  ==============
3
3
 
4
4
  [![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
5
- [![Code Climate](https://codeclimate.com/github/mperham/sidekiq.svg)](https://codeclimate.com/github/mperham/sidekiq)
6
- [![Test Coverage](https://codeclimate.com/github/mperham/sidekiq/badges/coverage.svg)](https://codeclimate.com/github/mperham/sidekiq/coverage)
7
- [![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
8
- [![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
9
-
5
+ ![Build](https://github.com/mperham/sidekiq/workflows/CI/badge.svg)
10
6
 
11
7
  Simple, efficient background processing for Ruby.
12
8
 
@@ -94,4 +90,4 @@ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for
94
90
  Author
95
91
  -----------------
96
92
 
97
- Mike Perham, [@mperham@mastodon.xyz](https://mastodon.xyz/@mperham) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
93
+ Mike Perham, [@getajobmike](https://twitter.com/getajobmike) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
data/bin/sidekiq CHANGED
@@ -6,13 +6,37 @@ $TESTING = false
6
6
 
7
7
  require_relative '../lib/sidekiq/cli'
8
8
 
9
+ def integrate_with_systemd
10
+ return unless ENV["NOTIFY_SOCKET"]
11
+
12
+ Sidekiq.configure_server do |config|
13
+ Sidekiq.logger.info "Enabling systemd notification integration"
14
+ require "sidekiq/sd_notify"
15
+ config.on(:startup) do
16
+ Sidekiq::SdNotify.ready
17
+ end
18
+ config.on(:shutdown) do
19
+ Sidekiq::SdNotify.stopping
20
+ end
21
+ Sidekiq.start_watchdog if Sidekiq::SdNotify.watchdog?
22
+ end
23
+ end
24
+
9
25
  begin
10
26
  cli = Sidekiq::CLI.instance
11
27
  cli.parse
28
+
29
+ integrate_with_systemd
30
+
12
31
  cli.run
13
32
  rescue => e
14
33
  raise e if $DEBUG
15
- STDERR.puts e.message
16
- STDERR.puts e.backtrace.join("\n")
34
+ if Sidekiq.error_handlers.length == 0
35
+ STDERR.puts e.message
36
+ STDERR.puts e.backtrace.join("\n")
37
+ else
38
+ cli.handle_exception e
39
+ end
40
+
17
41
  exit 1
18
42
  end
data/lib/sidekiq/api.rb CHANGED
@@ -8,7 +8,7 @@ require "base64"
8
8
  module Sidekiq
9
9
  class Stats
10
10
  def initialize
11
- fetch_stats!
11
+ fetch_stats_fast!
12
12
  end
13
13
 
14
14
  def processed
@@ -51,7 +51,8 @@ module Sidekiq
51
51
  Sidekiq::Stats::Queues.new.lengths
52
52
  end
53
53
 
54
- def fetch_stats!
54
+ # O(1) redis calls
55
+ def fetch_stats_fast!
55
56
  pipe1_res = Sidekiq.redis { |conn|
56
57
  conn.pipelined do
57
58
  conn.get("stat:processed")
@@ -64,6 +65,33 @@ module Sidekiq
64
65
  end
65
66
  }
66
67
 
68
+ default_queue_latency = if (entry = pipe1_res[6].first)
69
+ job = begin
70
+ Sidekiq.load_json(entry)
71
+ rescue
72
+ {}
73
+ end
74
+ now = Time.now.to_f
75
+ thence = job["enqueued_at"] || now
76
+ now - thence
77
+ else
78
+ 0
79
+ end
80
+
81
+ @stats = {
82
+ processed: pipe1_res[0].to_i,
83
+ failed: pipe1_res[1].to_i,
84
+ scheduled_size: pipe1_res[2],
85
+ retry_size: pipe1_res[3],
86
+ dead_size: pipe1_res[4],
87
+ processes_size: pipe1_res[5],
88
+
89
+ default_queue_latency: default_queue_latency
90
+ }
91
+ end
92
+
93
+ # O(number of processes + number of queues) redis calls
94
+ def fetch_stats_slow!
67
95
  processes = Sidekiq.redis { |conn|
68
96
  conn.sscan_each("processes").to_a
69
97
  }
@@ -83,30 +111,13 @@ module Sidekiq
83
111
  workers_size = pipe2_res[0...s].sum(&:to_i)
84
112
  enqueued = pipe2_res[s..-1].sum(&:to_i)
85
113
 
86
- default_queue_latency = if (entry = pipe1_res[6].first)
87
- job = begin
88
- Sidekiq.load_json(entry)
89
- rescue
90
- {}
91
- end
92
- now = Time.now.to_f
93
- thence = job["enqueued_at"] || now
94
- now - thence
95
- else
96
- 0
97
- end
98
- @stats = {
99
- processed: pipe1_res[0].to_i,
100
- failed: pipe1_res[1].to_i,
101
- scheduled_size: pipe1_res[2],
102
- retry_size: pipe1_res[3],
103
- dead_size: pipe1_res[4],
104
- processes_size: pipe1_res[5],
114
+ @stats[:workers_size] = workers_size
115
+ @stats[:enqueued] = enqueued
116
+ end
105
117
 
106
- default_queue_latency: default_queue_latency,
107
- workers_size: workers_size,
108
- enqueued: enqueued
109
- }
118
+ def fetch_stats!
119
+ fetch_stats_fast!
120
+ fetch_stats_slow!
110
121
  end
111
122
 
112
123
  def reset(*stats)
@@ -126,7 +137,8 @@ module Sidekiq
126
137
  private
127
138
 
128
139
  def stat(s)
129
- @stats[s]
140
+ fetch_stats_slow! if @stats[s].nil?
141
+ @stats[s] || raise(ArgumentError, "Unknown stat #{s}")
130
142
  end
131
143
 
132
144
  class Queues
@@ -141,7 +153,7 @@ module Sidekiq
141
153
  }
142
154
 
143
155
  array_of_arrays = queues.zip(lengths).sort_by { |_, size| -size }
144
- Hash[array_of_arrays]
156
+ array_of_arrays.to_h
145
157
  end
146
158
  end
147
159
  end
@@ -255,7 +267,7 @@ module Sidekiq
255
267
  break if entries.empty?
256
268
  page += 1
257
269
  entries.each do |entry|
258
- yield Job.new(entry, @name)
270
+ yield JobRecord.new(entry, @name)
259
271
  end
260
272
  deleted_size = initial_size - size
261
273
  end
@@ -265,7 +277,7 @@ module Sidekiq
265
277
  # Find the job with the given JID within this queue.
266
278
  #
267
279
  # This is a slow, inefficient operation. Do not use under
268
- # normal conditions. Sidekiq Pro contains a faster version.
280
+ # normal conditions.
269
281
  def find_job(jid)
270
282
  detect { |j| j.jid == jid }
271
283
  end
@@ -286,9 +298,9 @@ module Sidekiq
286
298
  # sorted set.
287
299
  #
288
300
  # The job should be considered immutable but may be
289
- # removed from the queue via Job#delete.
301
+ # removed from the queue via JobRecord#delete.
290
302
  #
291
- class Job
303
+ class JobRecord
292
304
  attr_reader :item
293
305
  attr_reader :value
294
306
 
@@ -316,21 +328,23 @@ module Sidekiq
316
328
 
317
329
  def display_class
318
330
  # Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
319
- @klass ||= case klass
320
- when /\ASidekiq::Extensions::Delayed/
321
- safe_load(args[0], klass) do |target, method, _|
322
- "#{target}.#{method}"
323
- end
324
- when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
325
- job_class = @item["wrapped"] || args[0]
326
- if job_class == "ActionMailer::DeliveryJob" || job_class == "ActionMailer::MailDeliveryJob"
327
- # MailerClass#mailer_method
328
- args[0]["arguments"][0..1].join("#")
329
- else
330
- job_class
331
- end
332
- else
333
- klass
331
+ @klass ||= self["display_class"] || begin
332
+ case klass
333
+ when /\ASidekiq::Extensions::Delayed/
334
+ safe_load(args[0], klass) do |target, method, _|
335
+ "#{target}.#{method}"
336
+ end
337
+ when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
338
+ job_class = @item["wrapped"] || args[0]
339
+ if job_class == "ActionMailer::DeliveryJob" || job_class == "ActionMailer::MailDeliveryJob"
340
+ # MailerClass#mailer_method
341
+ args[0]["arguments"][0..1].join("#")
342
+ else
343
+ job_class
344
+ end
345
+ else
346
+ klass
347
+ end
334
348
  end
335
349
  end
336
350
 
@@ -443,7 +457,7 @@ module Sidekiq
443
457
  end
444
458
  end
445
459
 
446
- class SortedEntry < Job
460
+ class SortedEntry < JobRecord
447
461
  attr_reader :score
448
462
  attr_reader :parent
449
463
 
@@ -791,19 +805,23 @@ module Sidekiq
791
805
  # you'll be happier this way
792
806
  conn.pipelined do
793
807
  procs.each do |key|
794
- conn.hmget(key, "info", "busy", "beat", "quiet")
808
+ conn.hmget(key, "info", "busy", "beat", "quiet", "rss", "rtt_us")
795
809
  end
796
810
  end
797
811
  }
798
812
 
799
- result.each do |info, busy, at_s, quiet|
813
+ result.each do |info, busy, at_s, quiet, rss, rtt|
800
814
  # If a process is stopped between when we query Redis for `procs` and
801
815
  # when we query for `result`, we will have an item in `result` that is
802
816
  # composed of `nil` values.
803
817
  next if info.nil?
804
818
 
805
819
  hash = Sidekiq.load_json(info)
806
- yield Process.new(hash.merge("busy" => busy.to_i, "beat" => at_s.to_f, "quiet" => quiet))
820
+ yield Process.new(hash.merge("busy" => busy.to_i,
821
+ "beat" => at_s.to_f,
822
+ "quiet" => quiet,
823
+ "rss" => rss.to_i,
824
+ "rtt_us" => rtt.to_i))
807
825
  end
808
826
  end
809
827
 
@@ -815,6 +833,18 @@ module Sidekiq
815
833
  Sidekiq.redis { |conn| conn.scard("processes") }
816
834
  end
817
835
 
836
+ # Total number of threads available to execute jobs.
837
+ # For Sidekiq Enterprise customers this number (in production) must be
838
+ # less than or equal to your licensed concurrency.
839
+ def total_concurrency
840
+ sum { |x| x["concurrency"].to_i }
841
+ end
842
+
843
+ def total_rss_in_kb
844
+ sum { |x| x["rss"].to_i }
845
+ end
846
+ alias_method :total_rss, :total_rss_in_kb
847
+
818
848
  # Returns the identity of the current cluster leader or "" if no leader.
819
849
  # This is a Sidekiq Enterprise feature, will always return "" in Sidekiq
820
850
  # or Sidekiq Pro.
@@ -864,6 +894,10 @@ module Sidekiq
864
894
  self["identity"]
865
895
  end
866
896
 
897
+ def queues
898
+ self["queues"]
899
+ end
900
+
867
901
  def quiet!
868
902
  signal("TSTP")
869
903
  end
@@ -894,8 +928,8 @@ module Sidekiq
894
928
  end
895
929
 
896
930
  ##
897
- # A worker is a thread that is currently processing a job.
898
- # Programmatic access to the current active worker set.
931
+ # The WorkSet stores the work being done by this Sidekiq cluster.
932
+ # It tracks the process and thread working on each job.
899
933
  #
900
934
  # WARNING WARNING WARNING
901
935
  #
@@ -903,25 +937,26 @@ module Sidekiq
903
937
  # If you call #size => 5 and then expect #each to be
904
938
  # called 5 times, you're going to have a bad time.
905
939
  #
906
- # workers = Sidekiq::Workers.new
907
- # workers.size => 2
908
- # workers.each do |process_id, thread_id, work|
940
+ # works = Sidekiq::WorkSet.new
941
+ # works.size => 2
942
+ # works.each do |process_id, thread_id, work|
909
943
  # # process_id is a unique identifier per Sidekiq process
910
944
  # # thread_id is a unique identifier per thread
911
945
  # # work is a Hash which looks like:
912
- # # { 'queue' => name, 'run_at' => timestamp, 'payload' => msg }
946
+ # # { 'queue' => name, 'run_at' => timestamp, 'payload' => job_hash }
913
947
  # # run_at is an epoch Integer.
914
948
  # end
915
949
  #
916
- class Workers
950
+ class WorkSet
917
951
  include Enumerable
918
952
 
919
- def each
953
+ def each(&block)
954
+ results = []
920
955
  Sidekiq.redis do |conn|
921
956
  procs = conn.sscan_each("processes").to_a
922
957
  procs.sort.each do |key|
923
958
  valid, workers = conn.pipelined {
924
- conn.exists(key)
959
+ conn.exists?(key)
925
960
  conn.hgetall("#{key}:workers")
926
961
  }
927
962
  next unless valid
@@ -930,10 +965,12 @@ module Sidekiq
930
965
  p = hsh["payload"]
931
966
  # avoid breaking API, this is a side effect of the JSON optimization in #4316
932
967
  hsh["payload"] = Sidekiq.load_json(p) if p.is_a?(String)
933
- yield key, tid, hsh
968
+ results << [key, tid, hsh]
934
969
  end
935
970
  end
936
971
  end
972
+
973
+ results.sort_by { |(_, _, hsh)| hsh["run_at"] }.each(&block)
937
974
  end
938
975
 
939
976
  # Note that #size is only as accurate as Sidekiq's heartbeat,
@@ -957,4 +994,8 @@ module Sidekiq
957
994
  end
958
995
  end
959
996
  end
997
+ # Since "worker" is a nebulous term, we've deprecated the use of this class name.
998
+ # Is "worker" a process, a type of job, a thread? Undefined!
999
+ # WorkSet better describes the data.
1000
+ Workers = WorkSet
960
1001
  end
data/lib/sidekiq/cli.rb CHANGED
@@ -33,8 +33,9 @@ module Sidekiq
33
33
  # Code within this method is not tested because it alters
34
34
  # global process state irreversibly. PRs which improve the
35
35
  # test coverage of Sidekiq::CLI are welcomed.
36
- def run
37
- boot_system
36
+ def run(boot_app: true)
37
+ boot_application if boot_app
38
+
38
39
  if environment == "development" && $stdout.tty? && Sidekiq.log_formatter.is_a?(Sidekiq::Logger::Formatters::Pretty)
39
40
  print_banner
40
41
  end
@@ -43,7 +44,7 @@ module Sidekiq
43
44
  self_read, self_write = IO.pipe
44
45
  sigs = %w[INT TERM TTIN TSTP]
45
46
  # USR1 and USR2 don't work on the JVM
46
- sigs << "USR2" unless jruby?
47
+ sigs << "USR2" if Sidekiq.pro? && !jruby?
47
48
  sigs.each do |sig|
48
49
  trap sig do
49
50
  self_write.puts(sig)
@@ -54,13 +55,26 @@ module Sidekiq
54
55
 
55
56
  logger.info "Running in #{RUBY_DESCRIPTION}"
56
57
  logger.info Sidekiq::LICENSE
57
- logger.info "Upgrade to Sidekiq Pro for more features and support: http://sidekiq.org" unless defined?(::Sidekiq::Pro)
58
+ logger.info "Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org" unless defined?(::Sidekiq::Pro)
58
59
 
59
60
  # touch the connection pool so it is created before we
60
61
  # fire startup and start multithreading.
61
- ver = Sidekiq.redis_info["redis_version"]
62
+ info = Sidekiq.redis_info
63
+ ver = info["redis_version"]
62
64
  raise "You are connecting to Redis v#{ver}, Sidekiq requires Redis v4.0.0 or greater" if ver < "4"
63
65
 
66
+ maxmemory_policy = info["maxmemory_policy"]
67
+ if maxmemory_policy != "noeviction"
68
+ logger.warn <<~EOM
69
+
70
+
71
+ WARNING: Your Redis instance will evict Sidekiq data under heavy load.
72
+ The 'noeviction' maxmemory policy is recommended (current policy: '#{maxmemory_policy}').
73
+ See: https://github.com/mperham/sidekiq/wiki/Using-Redis#memory
74
+
75
+ EOM
76
+ end
77
+
64
78
  # Since the user can pass us a connection pool explicitly in the initializer, we
65
79
  # need to verify the size is large enough or else Sidekiq's performance is dramatically slowed.
66
80
  cursize = Sidekiq.redis_pool.size
@@ -228,8 +242,7 @@ module Sidekiq
228
242
  opts = parse_config(opts[:config_file]).merge(opts) if opts[:config_file]
229
243
 
230
244
  # set defaults
231
- opts[:queues] = ["default"] if opts[:queues].nil? || opts[:queues].empty?
232
- opts[:strict] = true if opts[:strict].nil?
245
+ opts[:queues] = ["default"] if opts[:queues].nil?
233
246
  opts[:concurrency] = Integer(ENV["RAILS_MAX_THREADS"]) if opts[:concurrency].nil? && ENV["RAILS_MAX_THREADS"]
234
247
 
235
248
  # merge with defaults
@@ -240,7 +253,7 @@ module Sidekiq
240
253
  Sidekiq.options
241
254
  end
242
255
 
243
- def boot_system
256
+ def boot_application
244
257
  ENV["RACK_ENV"] = ENV["RAILS_ENV"] = environment
245
258
 
246
259
  if File.directory?(options[:require])
@@ -368,6 +381,8 @@ module Sidekiq
368
381
  end
369
382
 
370
383
  opts = opts.merge(opts.delete(environment.to_sym) || {})
384
+ opts.delete(:strict)
385
+
371
386
  parse_queues(opts, opts.delete(:queues) || [])
372
387
 
373
388
  opts
@@ -379,6 +394,7 @@ module Sidekiq
379
394
 
380
395
  def parse_queue(opts, queue, weight = nil)
381
396
  opts[:queues] ||= []
397
+ opts[:strict] = true if opts[:strict].nil?
382
398
  raise ArgumentError, "queues: #{queue} cannot be defined twice" if opts[:queues].include?(queue)
383
399
  [weight.to_i, 1].max.times { opts[:queues] << queue }
384
400
  opts[:strict] = false if weight.to_i > 0
@@ -19,7 +19,7 @@ module Sidekiq
19
19
  #
20
20
  def middleware(&block)
21
21
  @chain ||= Sidekiq.client_middleware
22
- if block_given?
22
+ if block
23
23
  @chain = @chain.dup
24
24
  yield @chain
25
25
  end
@@ -90,16 +90,17 @@ module Sidekiq
90
90
  # Returns an array of the of pushed jobs' jids. The number of jobs pushed can be less
91
91
  # than the number given if the middleware stopped processing for one or more jobs.
92
92
  def push_bulk(items)
93
- arg = items["args"].first
94
- return [] unless arg # no jobs to push
95
- raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" unless arg.is_a?(Array)
93
+ args = items["args"]
94
+ raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" unless args.is_a?(Array) && args.all?(Array)
95
+ return [] if args.empty? # no jobs to push
96
96
 
97
97
  at = items.delete("at")
98
98
  raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all?(Numeric))
99
+ raise ArgumentError, "Job 'at' Array must have same size as 'args' Array" if at.is_a?(Array) && at.size != args.size
99
100
 
100
101
  normed = normalize_item(items)
101
- payloads = items["args"].map.with_index { |args, index|
102
- copy = normed.merge("args" => args, "jid" => SecureRandom.hex(12), "enqueued_at" => Time.now.to_f)
102
+ payloads = args.map.with_index { |job_args, index|
103
+ copy = normed.merge("args" => job_args, "jid" => SecureRandom.hex(12), "enqueued_at" => Time.now.to_f)
103
104
  copy["at"] = (at.is_a?(Array) ? at[index] : at) if at
104
105
 
105
106
  result = process_single(items["class"], copy)
@@ -218,16 +219,16 @@ module Sidekiq
218
219
  end
219
220
  end
220
221
 
222
+ def validate(item)
223
+ raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
224
+ raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
225
+ raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
226
+ raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
227
+ raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
228
+ end
229
+
221
230
  def normalize_item(item)
222
- # 6.0.0 push_bulk bug, #4321
223
- # TODO Remove after a while...
224
- item.delete("at") if item.key?("at") && item["at"].nil?
225
-
226
- raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
227
- raise(ArgumentError, "Job args must be an Array") unless item["args"].is_a?(Array)
228
- raise(ArgumentError, "Job class must be either a Class or String representation of the class name") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
229
- raise(ArgumentError, "Job 'at' must be a Numeric timestamp") if item.key?("at") && !item["at"].is_a?(Numeric)
230
- raise(ArgumentError, "Job tags must be an Array") if item["tags"] && !item["tags"].is_a?(Array)
231
+ validate(item)
231
232
  # raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
232
233
 
233
234
  # merge in the default sidekiq_options for the item's class and/or wrapped element
@@ -5,9 +5,10 @@ require "sidekiq/extensions/generic_proxy"
5
5
  module Sidekiq
6
6
  module Extensions
7
7
  ##
8
- # Adds 'delay', 'delay_for' and `delay_until` methods to ActionMailer to offload arbitrary email
9
- # delivery to Sidekiq. Example:
8
+ # Adds +delay+, +delay_for+ and +delay_until+ methods to ActionMailer to offload arbitrary email
9
+ # delivery to Sidekiq.
10
10
  #
11
+ # @example
11
12
  # UserMailer.delay.send_welcome_email(new_user)
12
13
  # UserMailer.delay_for(5.days).send_welcome_email(new_user)
13
14
  # UserMailer.delay_until(5.days.from_now).send_welcome_email(new_user)