sidekiq 6.2.1 → 6.4.0

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

Files changed (65) hide show
  1. checksums.yaml +4 -4
  2. data/Changes.md +79 -1
  3. data/LICENSE +3 -3
  4. data/README.md +3 -3
  5. data/lib/generators/sidekiq/job_generator.rb +57 -0
  6. data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
  7. data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
  8. data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
  9. data/lib/sidekiq/api.rb +86 -60
  10. data/lib/sidekiq/cli.rb +13 -3
  11. data/lib/sidekiq/client.rb +5 -39
  12. data/lib/sidekiq/delay.rb +2 -0
  13. data/lib/sidekiq/extensions/action_mailer.rb +2 -2
  14. data/lib/sidekiq/extensions/active_record.rb +2 -2
  15. data/lib/sidekiq/extensions/class_methods.rb +2 -2
  16. data/lib/sidekiq/extensions/generic_proxy.rb +5 -3
  17. data/lib/sidekiq/fetch.rb +5 -4
  18. data/lib/sidekiq/job.rb +13 -0
  19. data/lib/sidekiq/job_logger.rb +1 -1
  20. data/lib/sidekiq/job_retry.rb +9 -11
  21. data/lib/sidekiq/job_util.rb +65 -0
  22. data/lib/sidekiq/launcher.rb +18 -18
  23. data/lib/sidekiq/manager.rb +7 -9
  24. data/lib/sidekiq/middleware/chain.rb +5 -3
  25. data/lib/sidekiq/middleware/current_attributes.rb +57 -0
  26. data/lib/sidekiq/rails.rb +11 -0
  27. data/lib/sidekiq/redis_connection.rb +4 -6
  28. data/lib/sidekiq/scheduled.rb +51 -16
  29. data/lib/sidekiq/testing.rb +1 -3
  30. data/lib/sidekiq/util.rb +13 -0
  31. data/lib/sidekiq/version.rb +1 -1
  32. data/lib/sidekiq/web/action.rb +1 -1
  33. data/lib/sidekiq/web/application.rb +9 -6
  34. data/lib/sidekiq/web/helpers.rb +9 -21
  35. data/lib/sidekiq/web.rb +4 -3
  36. data/lib/sidekiq/worker.rb +127 -7
  37. data/lib/sidekiq.rb +8 -1
  38. data/sidekiq.gemspec +1 -1
  39. data/web/assets/javascripts/application.js +82 -61
  40. data/web/assets/javascripts/dashboard.js +51 -51
  41. data/web/assets/stylesheets/application-dark.css +28 -45
  42. data/web/assets/stylesheets/application-rtl.css +0 -4
  43. data/web/assets/stylesheets/application.css +23 -237
  44. data/web/locales/ar.yml +8 -2
  45. data/web/locales/en.yml +4 -1
  46. data/web/locales/es.yml +18 -2
  47. data/web/locales/fr.yml +7 -0
  48. data/web/locales/ja.yml +3 -0
  49. data/web/locales/lt.yml +1 -1
  50. data/web/views/_footer.erb +1 -1
  51. data/web/views/_job_info.erb +1 -1
  52. data/web/views/_poll_link.erb +2 -5
  53. data/web/views/_summary.erb +7 -7
  54. data/web/views/busy.erb +5 -5
  55. data/web/views/dashboard.erb +22 -14
  56. data/web/views/dead.erb +1 -1
  57. data/web/views/layout.erb +1 -1
  58. data/web/views/morgue.erb +6 -6
  59. data/web/views/queue.erb +10 -10
  60. data/web/views/queues.erb +3 -3
  61. data/web/views/retries.erb +7 -7
  62. data/web/views/retry.erb +1 -1
  63. data/web/views/scheduled.erb +1 -1
  64. metadata +10 -7
  65. data/lib/generators/sidekiq/worker_generator.rb +0 -57
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 298711914bcb8534a9599c47b00b7410467ce324619ee70e7050d15c42f4c329
4
- data.tar.gz: '007900de7a1558633520c61870a58eff341e9c11009441dbabe0fbc177e4ed99'
3
+ metadata.gz: 9622b2851203b0c5a80695ab7801ca77e15dc63a641ae79132cda9a2fcbe0cc6
4
+ data.tar.gz: dd943a02d2cf910f51866d02254f3a7845cb159735bd54d223dd7127a1fbd2fa
5
5
  SHA512:
6
- metadata.gz: 592ecc114de13f0e43bba9193e1ffd3a973c89a43fac3ed1b750b6a70e29b5bf128a05657baf3fc2ccb77134f092efac055651907f01c0ed6d3c00d45a5ebdc9
7
- data.tar.gz: a7baed1f1df451e8bd5183fec4631e49c0761e700c4c95fc070389894894a5fb90103d0adce29da797bc1cae72f8a1f21e71da0279e8c18cb62e3b3b5ae05f0a
6
+ metadata.gz: 65bcd542866d8699ecf5958d81a15fd322cd1c51dfc1dbc6a8b15402b70862510c134e2b0ed9f9dbcdbb4dea3a59c1d12878ff926145e503079a59896f279ff3
7
+ data.tar.gz: 36143e85dc7fd4611f8a43fe39788dc590725ac63a827fbc174916da2d59fdadb3fb64fa0d819c4ef90694f6f292ea15f7fb58fb29f368c0a06c5b90f1b4bca7
data/Changes.md CHANGED
@@ -1,6 +1,84 @@
1
1
  # Sidekiq Changes
2
2
 
3
- [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
4
+
5
+ 6.4.0
6
+ ---------
7
+
8
+ - **SECURITY**: Validate input to avoid possible DoS in Web UI.
9
+ - Add **strict argument checking** [#5071]
10
+ Sidekiq will now log a warning if JSON-unsafe arguments are passed to `perform_async`.
11
+ Add `Sidekiq.strict_args!(false)` to your initializer to disable this warning.
12
+ This warning will switch to an exception in Sidekiq 7.0.
13
+ - Note that Delayed Extensions will be removed in Sidekiq 7.0 [#5076]
14
+ - Add `perform_{inline,sync}` in Sidekiq::Job to run a job synchronously [#5061, hasan-ally]
15
+ ```ruby
16
+ SomeJob.perform_async(args...)
17
+ SomeJob.perform_sync(args...)
18
+ SomeJob.perform_inline(args...)
19
+ ```
20
+ You can also dynamically redirect a job to run synchronously:
21
+ ```ruby
22
+ SomeJob.set("sync": true).perform_async(args...) # will run via perform_inline
23
+ ```
24
+ - Replace Sidekiq::Worker `app/workers` generator with Sidekiq::Job `app/sidekiq` generator [#5055]
25
+ ```
26
+ bin/rails generate sidekiq:job ProcessOrderJob
27
+ ```
28
+ - Fix job retries losing CurrentAttributes [#5090]
29
+ - Tweak shutdown to give long-running threads time to cleanup [#5095]
30
+ - Add keyword arguments support in extensions
31
+
32
+ 6.3.1
33
+ ---------
34
+
35
+ - Fix keyword arguments error with CurrentAttributes on Ruby 3.0 [#5048]
36
+
37
+ 6.3.0
38
+ ---------
39
+
40
+ - **BREAK**: The Web UI has been refactored to remove jQuery. Any UI extensions
41
+ which use jQuery will break.
42
+ - **FEATURE**: Sidekiq.logger has been enhanced so any `Rails.logger`
43
+ output in jobs now shows up in the Sidekiq console. Remove any logger
44
+ hacks in your initializer and see if it Just Works™ now. [#5021]
45
+ - **FEATURE**: Add `Sidekiq::Job` alias for `Sidekiq::Worker`, to better
46
+ reflect industry standard terminology. You can now do this:
47
+ ```ruby
48
+ class MyJob
49
+ include Sidekiq::Job
50
+ sidekiq_options ...
51
+ def perform(args)
52
+ end
53
+ end
54
+ ```
55
+ - **FEATURE**: Support for serializing ActiveSupport::CurrentAttributes into each job. [#4982]
56
+ ```ruby
57
+ # config/initializers/sidekiq.rb
58
+ require "sidekiq/middleware/current_attributes"
59
+ Sidekiq::CurrentAttributes.persist(Myapp::Current) # Your AS::CurrentAttributes singleton
60
+ ```
61
+ - **FEATURE**: Add `Sidekiq::Worker.perform_bulk` for enqueuing jobs in bulk,
62
+ similar to `Sidekiq::Client.push_bulk` [#5042]
63
+ ```ruby
64
+ MyJob.perform_bulk([[1], [2], [3]])
65
+ ```
66
+ - Implement `queue_as`, `wait` and `wait_until` for ActiveJob compatibility [#5003]
67
+ - Scheduler now uses Lua to reduce Redis load and network roundtrips [#5044]
68
+ - Retry Redis operation if we get an `UNBLOCKED` Redis error [#4985]
69
+ - Run existing signal traps, if any, before running Sidekiq's trap [#4991]
70
+ - Fix fetch bug when using weighted queues which caused Sidekiq to stop
71
+ processing queues randomly [#5031]
72
+
73
+ 6.2.2
74
+ ---------
75
+
76
+ - Reduce retry jitter, add jitter to `sidekiq_retry_in` values [#4957]
77
+ - Minimize scheduler load on Redis at scale [#4882]
78
+ - Improve logging of delay jobs [#4904, BuonOno]
79
+ - Minor CSS improvements for buttons and tables, design PRs always welcome!
80
+ - Tweak Web UI `Cache-Control` header [#4966]
81
+ - Rename internal API class `Sidekiq::Job` to `Sidekiq::JobRecord` [#4955]
4
82
 
5
83
  6.2.1
6
84
  ---------
data/LICENSE CHANGED
@@ -4,6 +4,6 @@ Sidekiq is an Open Source project licensed under the terms of
4
4
  the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
5
5
  for license text.
6
6
 
7
- Sidekiq Pro has a commercial-friendly license allowing private forks
8
- and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
9
- more detail. You can find the commercial license terms in COMM-LICENSE.
7
+ Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
8
+ You can find the commercial license in COMM-LICENSE.txt.
9
+ Please see https://sidekiq.org for purchasing options.
data/README.md CHANGED
@@ -43,10 +43,10 @@ Getting Started
43
43
  -----------------
44
44
 
45
45
  See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
46
- You can watch [this Youtube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
46
+ You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
47
47
  Sidekiq and see its features in action. Here's the Web UI:
48
48
 
49
- ![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
49
+ ![Web UI](https://github.com/mperham/sidekiq/raw/main/examples/web-ui.png)
50
50
 
51
51
 
52
52
  Want to Upgrade?
@@ -84,7 +84,7 @@ See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
84
84
  License
85
85
  -----------------
86
86
 
87
- Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for licensing details.
87
+ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/main/LICENSE) for licensing details.
88
88
 
89
89
 
90
90
  Author
@@ -0,0 +1,57 @@
1
+ require "rails/generators/named_base"
2
+
3
+ module Sidekiq
4
+ module Generators # :nodoc:
5
+ class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
6
+ desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
7
+
8
+ check_class_collision suffix: "Job"
9
+
10
+ def self.default_generator_root
11
+ File.dirname(__FILE__)
12
+ end
13
+
14
+ def create_job_file
15
+ template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
16
+ end
17
+
18
+ def create_test_file
19
+ return unless test_framework
20
+
21
+ if test_framework == :rspec
22
+ create_job_spec
23
+ else
24
+ create_job_test
25
+ end
26
+ end
27
+
28
+ private
29
+
30
+ def create_job_spec
31
+ template_file = File.join(
32
+ "spec/sidekiq",
33
+ class_path,
34
+ "#{file_name}_job_spec.rb"
35
+ )
36
+ template "job_spec.rb.erb", template_file
37
+ end
38
+
39
+ def create_job_test
40
+ template_file = File.join(
41
+ "test/sidekiq",
42
+ class_path,
43
+ "#{file_name}_job_test.rb"
44
+ )
45
+ template "job_test.rb.erb", template_file
46
+ end
47
+
48
+ def file_name
49
+ @_file_name ||= super.sub(/_?job\z/i, "")
50
+ end
51
+
52
+ def test_framework
53
+ ::Rails.application.config.generators.options[:rails][:test_framework]
54
+ end
55
+ end
56
+ end
57
+ end
@@ -1,6 +1,6 @@
1
1
  <% module_namespacing do -%>
2
- class <%= class_name %>Worker
3
- include Sidekiq::Worker
2
+ class <%= class_name %>Job
3
+ include Sidekiq::Job
4
4
 
5
5
  def perform(*args)
6
6
  # Do something
@@ -1,6 +1,6 @@
1
1
  require 'rails_helper'
2
2
  <% module_namespacing do -%>
3
- RSpec.describe <%= class_name %>Worker, type: :worker do
3
+ RSpec.describe <%= class_name %>Job, type: :job do
4
4
  pending "add some examples to (or delete) #{__FILE__}"
5
5
  end
6
6
  <% end -%>
@@ -1,6 +1,6 @@
1
1
  require 'test_helper'
2
2
  <% module_namespacing do -%>
3
- class <%= class_name %>WorkerTest < Minitest::Test
3
+ class <%= class_name %>JobTest < Minitest::Test
4
4
  def test_example
5
5
  skip "add some examples to (or delete) #{__FILE__}"
6
6
  end
data/lib/sidekiq/api.rb CHANGED
@@ -8,7 +8,7 @@ require "base64"
8
8
  module Sidekiq
9
9
  class Stats
10
10
  def initialize
11
- fetch_stats!
11
+ fetch_stats_fast!
12
12
  end
13
13
 
14
14
  def processed
@@ -51,7 +51,8 @@ module Sidekiq
51
51
  Sidekiq::Stats::Queues.new.lengths
52
52
  end
53
53
 
54
- def fetch_stats!
54
+ # O(1) redis calls
55
+ def fetch_stats_fast!
55
56
  pipe1_res = Sidekiq.redis { |conn|
56
57
  conn.pipelined do
57
58
  conn.get("stat:processed")
@@ -64,25 +65,6 @@ module Sidekiq
64
65
  end
65
66
  }
66
67
 
67
- processes = Sidekiq.redis { |conn|
68
- conn.sscan_each("processes").to_a
69
- }
70
-
71
- queues = Sidekiq.redis { |conn|
72
- conn.sscan_each("queues").to_a
73
- }
74
-
75
- pipe2_res = Sidekiq.redis { |conn|
76
- conn.pipelined do
77
- processes.each { |key| conn.hget(key, "busy") }
78
- queues.each { |queue| conn.llen("queue:#{queue}") }
79
- end
80
- }
81
-
82
- s = processes.size
83
- workers_size = pipe2_res[0...s].sum(&:to_i)
84
- enqueued = pipe2_res[s..-1].sum(&:to_i)
85
-
86
68
  default_queue_latency = if (entry = pipe1_res[6].first)
87
69
  job = begin
88
70
  Sidekiq.load_json(entry)
@@ -95,6 +77,7 @@ module Sidekiq
95
77
  else
96
78
  0
97
79
  end
80
+
98
81
  @stats = {
99
82
  processed: pipe1_res[0].to_i,
100
83
  failed: pipe1_res[1].to_i,
@@ -103,10 +86,39 @@ module Sidekiq
103
86
  dead_size: pipe1_res[4],
104
87
  processes_size: pipe1_res[5],
105
88
 
106
- default_queue_latency: default_queue_latency,
107
- workers_size: workers_size,
108
- enqueued: enqueued
89
+ default_queue_latency: default_queue_latency
90
+ }
91
+ end
92
+
93
+ # O(number of processes + number of queues) redis calls
94
+ def fetch_stats_slow!
95
+ processes = Sidekiq.redis { |conn|
96
+ conn.sscan_each("processes").to_a
97
+ }
98
+
99
+ queues = Sidekiq.redis { |conn|
100
+ conn.sscan_each("queues").to_a
109
101
  }
102
+
103
+ pipe2_res = Sidekiq.redis { |conn|
104
+ conn.pipelined do
105
+ processes.each { |key| conn.hget(key, "busy") }
106
+ queues.each { |queue| conn.llen("queue:#{queue}") }
107
+ end
108
+ }
109
+
110
+ s = processes.size
111
+ workers_size = pipe2_res[0...s].sum(&:to_i)
112
+ enqueued = pipe2_res[s..-1].sum(&:to_i)
113
+
114
+ @stats[:workers_size] = workers_size
115
+ @stats[:enqueued] = enqueued
116
+ @stats
117
+ end
118
+
119
+ def fetch_stats!
120
+ fetch_stats_fast!
121
+ fetch_stats_slow!
110
122
  end
111
123
 
112
124
  def reset(*stats)
@@ -126,7 +138,8 @@ module Sidekiq
126
138
  private
127
139
 
128
140
  def stat(s)
129
- @stats[s]
141
+ fetch_stats_slow! if @stats[s].nil?
142
+ @stats[s] || raise(ArgumentError, "Unknown stat #{s}")
130
143
  end
131
144
 
132
145
  class Queues
@@ -141,13 +154,15 @@ module Sidekiq
141
154
  }
142
155
 
143
156
  array_of_arrays = queues.zip(lengths).sort_by { |_, size| -size }
144
- Hash[array_of_arrays]
157
+ array_of_arrays.to_h
145
158
  end
146
159
  end
147
160
  end
148
161
 
149
162
  class History
150
163
  def initialize(days_previous, start_date = nil)
164
+ # we only store five years of data in Redis
165
+ raise ArgumentError if days_previous < 1 || days_previous > (5 * 365)
151
166
  @days_previous = days_previous
152
167
  @start_date = start_date || Time.now.utc.to_date
153
168
  end
@@ -255,7 +270,7 @@ module Sidekiq
255
270
  break if entries.empty?
256
271
  page += 1
257
272
  entries.each do |entry|
258
- yield Job.new(entry, @name)
273
+ yield JobRecord.new(entry, @name)
259
274
  end
260
275
  deleted_size = initial_size - size
261
276
  end
@@ -265,7 +280,7 @@ module Sidekiq
265
280
  # Find the job with the given JID within this queue.
266
281
  #
267
282
  # This is a slow, inefficient operation. Do not use under
268
- # normal conditions. Sidekiq Pro contains a faster version.
283
+ # normal conditions.
269
284
  def find_job(jid)
270
285
  detect { |j| j.jid == jid }
271
286
  end
@@ -286,9 +301,9 @@ module Sidekiq
286
301
  # sorted set.
287
302
  #
288
303
  # The job should be considered immutable but may be
289
- # removed from the queue via Job#delete.
304
+ # removed from the queue via JobRecord#delete.
290
305
  #
291
- class Job
306
+ class JobRecord
292
307
  attr_reader :item
293
308
  attr_reader :value
294
309
 
@@ -316,21 +331,23 @@ module Sidekiq
316
331
 
317
332
  def display_class
318
333
  # Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
319
- @klass ||= case klass
320
- when /\ASidekiq::Extensions::Delayed/
321
- safe_load(args[0], klass) do |target, method, _|
322
- "#{target}.#{method}"
323
- end
324
- when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
325
- job_class = @item["wrapped"] || args[0]
326
- if job_class == "ActionMailer::DeliveryJob" || job_class == "ActionMailer::MailDeliveryJob"
327
- # MailerClass#mailer_method
328
- args[0]["arguments"][0..1].join("#")
329
- else
330
- job_class
331
- end
332
- else
333
- klass
334
+ @klass ||= self["display_class"] || begin
335
+ case klass
336
+ when /\ASidekiq::Extensions::Delayed/
337
+ safe_load(args[0], klass) do |target, method, _|
338
+ "#{target}.#{method}"
339
+ end
340
+ when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
341
+ job_class = @item["wrapped"] || args[0]
342
+ if job_class == "ActionMailer::DeliveryJob" || job_class == "ActionMailer::MailDeliveryJob"
343
+ # MailerClass#mailer_method
344
+ args[0]["arguments"][0..1].join("#")
345
+ else
346
+ job_class
347
+ end
348
+ else
349
+ klass
350
+ end
334
351
  end
335
352
  end
336
353
 
@@ -443,7 +460,7 @@ module Sidekiq
443
460
  end
444
461
  end
445
462
 
446
- class SortedEntry < Job
463
+ class SortedEntry < JobRecord
447
464
  attr_reader :score
448
465
  attr_reader :parent
449
466
 
@@ -804,10 +821,10 @@ module Sidekiq
804
821
 
805
822
  hash = Sidekiq.load_json(info)
806
823
  yield Process.new(hash.merge("busy" => busy.to_i,
807
- "beat" => at_s.to_f,
808
- "quiet" => quiet,
809
- "rss" => rss.to_i,
810
- "rtt_us" => rtt.to_i))
824
+ "beat" => at_s.to_f,
825
+ "quiet" => quiet,
826
+ "rss" => rss.to_i,
827
+ "rtt_us" => rtt.to_i))
811
828
  end
812
829
  end
813
830
 
@@ -823,12 +840,13 @@ module Sidekiq
823
840
  # For Sidekiq Enterprise customers this number (in production) must be
824
841
  # less than or equal to your licensed concurrency.
825
842
  def total_concurrency
826
- sum { |x| x["concurrency"] }
843
+ sum { |x| x["concurrency"].to_i }
827
844
  end
828
845
 
829
- def total_rss
830
- sum { |x| x["rss"] || 0 }
846
+ def total_rss_in_kb
847
+ sum { |x| x["rss"].to_i }
831
848
  end
849
+ alias_method :total_rss, :total_rss_in_kb
832
850
 
833
851
  # Returns the identity of the current cluster leader or "" if no leader.
834
852
  # This is a Sidekiq Enterprise feature, will always return "" in Sidekiq
@@ -879,6 +897,10 @@ module Sidekiq
879
897
  self["identity"]
880
898
  end
881
899
 
900
+ def queues
901
+ self["queues"]
902
+ end
903
+
882
904
  def quiet!
883
905
  signal("TSTP")
884
906
  end
@@ -909,8 +931,8 @@ module Sidekiq
909
931
  end
910
932
 
911
933
  ##
912
- # A worker is a thread that is currently processing a job.
913
- # Programmatic access to the current active worker set.
934
+ # The WorkSet stores the work being done by this Sidekiq cluster.
935
+ # It tracks the process and thread working on each job.
914
936
  #
915
937
  # WARNING WARNING WARNING
916
938
  #
@@ -918,17 +940,17 @@ module Sidekiq
918
940
  # If you call #size => 5 and then expect #each to be
919
941
  # called 5 times, you're going to have a bad time.
920
942
  #
921
- # workers = Sidekiq::Workers.new
922
- # workers.size => 2
923
- # workers.each do |process_id, thread_id, work|
943
+ # works = Sidekiq::WorkSet.new
944
+ # works.size => 2
945
+ # works.each do |process_id, thread_id, work|
924
946
  # # process_id is a unique identifier per Sidekiq process
925
947
  # # thread_id is a unique identifier per thread
926
948
  # # work is a Hash which looks like:
927
- # # { 'queue' => name, 'run_at' => timestamp, 'payload' => msg }
949
+ # # { 'queue' => name, 'run_at' => timestamp, 'payload' => job_hash }
928
950
  # # run_at is an epoch Integer.
929
951
  # end
930
952
  #
931
- class Workers
953
+ class WorkSet
932
954
  include Enumerable
933
955
 
934
956
  def each(&block)
@@ -975,4 +997,8 @@ module Sidekiq
975
997
  end
976
998
  end
977
999
  end
1000
+ # Since "worker" is a nebulous term, we've deprecated the use of this class name.
1001
+ # Is "worker" a process, a type of job, a thread? Undefined!
1002
+ # WorkSet better describes the data.
1003
+ Workers = WorkSet
978
1004
  end
data/lib/sidekiq/cli.rb CHANGED
@@ -46,7 +46,15 @@ module Sidekiq
46
46
  # USR1 and USR2 don't work on the JVM
47
47
  sigs << "USR2" if Sidekiq.pro? && !jruby?
48
48
  sigs.each do |sig|
49
- trap sig do
49
+ old_handler = Signal.trap(sig) do
50
+ if old_handler.respond_to?(:call)
51
+ begin
52
+ old_handler.call
53
+ rescue Exception => exc
54
+ # signal handlers can't use Logger so puts only
55
+ puts ["Error in #{sig} handler", exc].inspect
56
+ end
57
+ end
50
58
  self_write.puts(sig)
51
59
  end
52
60
  rescue ArgumentError
@@ -372,7 +380,9 @@ module Sidekiq
372
380
  end
373
381
 
374
382
  def parse_config(path)
375
- opts = YAML.load(ERB.new(File.read(path)).result) || {}
383
+ erb = ERB.new(File.read(path))
384
+ erb.filename = File.expand_path(path)
385
+ opts = YAML.load(erb.result) || {}
376
386
 
377
387
  if opts.respond_to? :deep_symbolize_keys!
378
388
  opts.deep_symbolize_keys!
@@ -396,7 +406,7 @@ module Sidekiq
396
406
  opts[:queues] ||= []
397
407
  opts[:strict] = true if opts[:strict].nil?
398
408
  raise ArgumentError, "queues: #{queue} cannot be defined twice" if opts[:queues].include?(queue)
399
- [weight.to_i, 1].max.times { opts[:queues] << queue }
409
+ [weight.to_i, 1].max.times { opts[:queues] << queue.to_s }
400
410
  opts[:strict] = false if weight.to_i > 0
401
411
  end
402
412
 
@@ -2,9 +2,12 @@
2
2
 
3
3
  require "securerandom"
4
4
  require "sidekiq/middleware/chain"
5
+ require "sidekiq/job_util"
5
6
 
6
7
  module Sidekiq
7
8
  class Client
9
+ include Sidekiq::JobUtil
10
+
8
11
  ##
9
12
  # Define client-side middleware:
10
13
  #
@@ -95,7 +98,7 @@ module Sidekiq
95
98
  return [] if args.empty? # no jobs to push
96
99
 
97
100
  at = items.delete("at")
98
- raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all?(Numeric))
101
+ raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all? { |entry| entry.is_a?(Numeric) })
99
102
  raise ArgumentError, "Job 'at' Array must have same size as 'args' Array" if at.is_a?(Array) && at.size != args.size
100
103
 
101
104
  normed = normalize_item(items)
@@ -186,7 +189,7 @@ module Sidekiq
186
189
 
187
190
  def raw_push(payloads)
188
191
  @redis_pool.with do |conn|
189
- conn.multi do
192
+ conn.pipelined do
190
193
  atomic_push(conn, payloads)
191
194
  end
192
195
  end
@@ -218,42 +221,5 @@ module Sidekiq
218
221
  item
219
222
  end
220
223
  end
221
-
222
- def validate(item)
223
- raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
224
- raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
225
- raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
226
- raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
227
- raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
228
- end
229
-
230
- def normalize_item(item)
231
- validate(item)
232
- # raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
233
-
234
- # merge in the default sidekiq_options for the item's class and/or wrapped element
235
- # this allows ActiveJobs to control sidekiq_options too.
236
- defaults = normalized_hash(item["class"])
237
- defaults = defaults.merge(item["wrapped"].get_sidekiq_options) if item["wrapped"].respond_to?("get_sidekiq_options")
238
- item = defaults.merge(item)
239
-
240
- raise(ArgumentError, "Job must include a valid queue name") if item["queue"].nil? || item["queue"] == ""
241
-
242
- item["class"] = item["class"].to_s
243
- item["queue"] = item["queue"].to_s
244
- item["jid"] ||= SecureRandom.hex(12)
245
- item["created_at"] ||= Time.now.to_f
246
-
247
- item
248
- end
249
-
250
- def normalized_hash(item_class)
251
- if item_class.is_a?(Class)
252
- raise(ArgumentError, "Message must include a Sidekiq::Worker class, not class name: #{item_class.ancestors.inspect}") unless item_class.respond_to?("get_sidekiq_options")
253
- item_class.get_sidekiq_options
254
- else
255
- Sidekiq.default_worker_options
256
- end
257
- end
258
224
  end
259
225
  end
data/lib/sidekiq/delay.rb CHANGED
@@ -3,6 +3,8 @@
3
3
  module Sidekiq
4
4
  module Extensions
5
5
  def self.enable_delay!
6
+ Sidekiq.logger.error "Sidekiq's Delayed Extensions will be removed in Sidekiq 7.0. #{caller(1..1).first}"
7
+
6
8
  if defined?(::ActiveSupport)
7
9
  require "sidekiq/extensions/active_record"
8
10
  require "sidekiq/extensions/action_mailer"
@@ -16,8 +16,8 @@ module Sidekiq
16
16
  include Sidekiq::Worker
17
17
 
18
18
  def perform(yml)
19
- (target, method_name, args) = YAML.load(yml)
20
- msg = target.public_send(method_name, *args)
19
+ (target, method_name, args, kwargs) = YAML.load(yml)
20
+ msg = kwargs.empty? ? target.public_send(method_name, *args) : target.public_send(method_name, *args, **kwargs)
21
21
  # The email method can return nil, which causes ActionMailer to return
22
22
  # an undeliverable empty message.
23
23
  if msg
@@ -18,8 +18,8 @@ module Sidekiq
18
18
  include Sidekiq::Worker
19
19
 
20
20
  def perform(yml)
21
- (target, method_name, args) = YAML.load(yml)
22
- target.__send__(method_name, *args)
21
+ (target, method_name, args, kwargs) = YAML.load(yml)
22
+ kwargs.empty? ? target.__send__(method_name, *args) : target.__send__(method_name, *args, **kwargs)
23
23
  end
24
24
  end
25
25
 
@@ -16,8 +16,8 @@ module Sidekiq
16
16
  include Sidekiq::Worker
17
17
 
18
18
  def perform(yml)
19
- (target, method_name, args) = YAML.load(yml)
20
- target.__send__(method_name, *args)
19
+ (target, method_name, args, kwargs) = YAML.load(yml)
20
+ kwargs.empty? ? target.__send__(method_name, *args) : target.__send__(method_name, *args, **kwargs)
21
21
  end
22
22
  end
23
23