sidekiq 6.2.1 → 6.4.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/Changes.md +79 -1
- data/LICENSE +3 -3
- data/README.md +3 -3
- data/lib/generators/sidekiq/job_generator.rb +57 -0
- data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
- data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
- data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
- data/lib/sidekiq/api.rb +86 -60
- data/lib/sidekiq/cli.rb +13 -3
- data/lib/sidekiq/client.rb +5 -39
- data/lib/sidekiq/delay.rb +2 -0
- data/lib/sidekiq/extensions/action_mailer.rb +2 -2
- data/lib/sidekiq/extensions/active_record.rb +2 -2
- data/lib/sidekiq/extensions/class_methods.rb +2 -2
- data/lib/sidekiq/extensions/generic_proxy.rb +5 -3
- data/lib/sidekiq/fetch.rb +5 -4
- data/lib/sidekiq/job.rb +13 -0
- data/lib/sidekiq/job_logger.rb +1 -1
- data/lib/sidekiq/job_retry.rb +9 -11
- data/lib/sidekiq/job_util.rb +65 -0
- data/lib/sidekiq/launcher.rb +18 -18
- data/lib/sidekiq/manager.rb +7 -9
- data/lib/sidekiq/middleware/chain.rb +5 -3
- data/lib/sidekiq/middleware/current_attributes.rb +57 -0
- data/lib/sidekiq/rails.rb +11 -0
- data/lib/sidekiq/redis_connection.rb +4 -6
- data/lib/sidekiq/scheduled.rb +51 -16
- data/lib/sidekiq/testing.rb +1 -3
- data/lib/sidekiq/util.rb +13 -0
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/action.rb +1 -1
- data/lib/sidekiq/web/application.rb +9 -6
- data/lib/sidekiq/web/helpers.rb +9 -21
- data/lib/sidekiq/web.rb +4 -3
- data/lib/sidekiq/worker.rb +127 -7
- data/lib/sidekiq.rb +8 -1
- data/sidekiq.gemspec +1 -1
- data/web/assets/javascripts/application.js +82 -61
- data/web/assets/javascripts/dashboard.js +51 -51
- data/web/assets/stylesheets/application-dark.css +28 -45
- data/web/assets/stylesheets/application-rtl.css +0 -4
- data/web/assets/stylesheets/application.css +23 -237
- data/web/locales/ar.yml +8 -2
- data/web/locales/en.yml +4 -1
- data/web/locales/es.yml +18 -2
- data/web/locales/fr.yml +7 -0
- data/web/locales/ja.yml +3 -0
- data/web/locales/lt.yml +1 -1
- data/web/views/_footer.erb +1 -1
- data/web/views/_job_info.erb +1 -1
- data/web/views/_poll_link.erb +2 -5
- data/web/views/_summary.erb +7 -7
- data/web/views/busy.erb +5 -5
- data/web/views/dashboard.erb +22 -14
- data/web/views/dead.erb +1 -1
- data/web/views/layout.erb +1 -1
- data/web/views/morgue.erb +6 -6
- data/web/views/queue.erb +10 -10
- data/web/views/queues.erb +3 -3
- data/web/views/retries.erb +7 -7
- data/web/views/retry.erb +1 -1
- data/web/views/scheduled.erb +1 -1
- metadata +10 -7
- data/lib/generators/sidekiq/worker_generator.rb +0 -57
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9622b2851203b0c5a80695ab7801ca77e15dc63a641ae79132cda9a2fcbe0cc6
|
4
|
+
data.tar.gz: dd943a02d2cf910f51866d02254f3a7845cb159735bd54d223dd7127a1fbd2fa
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 65bcd542866d8699ecf5958d81a15fd322cd1c51dfc1dbc6a8b15402b70862510c134e2b0ed9f9dbcdbb4dea3a59c1d12878ff926145e503079a59896f279ff3
|
7
|
+
data.tar.gz: 36143e85dc7fd4611f8a43fe39788dc590725ac63a827fbc174916da2d59fdadb3fb64fa0d819c4ef90694f6f292ea15f7fb58fb29f368c0a06c5b90f1b4bca7
|
data/Changes.md
CHANGED
@@ -1,6 +1,84 @@
|
|
1
1
|
# Sidekiq Changes
|
2
2
|
|
3
|
-
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/
|
3
|
+
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
|
4
|
+
|
5
|
+
6.4.0
|
6
|
+
---------
|
7
|
+
|
8
|
+
- **SECURITY**: Validate input to avoid possible DoS in Web UI.
|
9
|
+
- Add **strict argument checking** [#5071]
|
10
|
+
Sidekiq will now log a warning if JSON-unsafe arguments are passed to `perform_async`.
|
11
|
+
Add `Sidekiq.strict_args!(false)` to your initializer to disable this warning.
|
12
|
+
This warning will switch to an exception in Sidekiq 7.0.
|
13
|
+
- Note that Delayed Extensions will be removed in Sidekiq 7.0 [#5076]
|
14
|
+
- Add `perform_{inline,sync}` in Sidekiq::Job to run a job synchronously [#5061, hasan-ally]
|
15
|
+
```ruby
|
16
|
+
SomeJob.perform_async(args...)
|
17
|
+
SomeJob.perform_sync(args...)
|
18
|
+
SomeJob.perform_inline(args...)
|
19
|
+
```
|
20
|
+
You can also dynamically redirect a job to run synchronously:
|
21
|
+
```ruby
|
22
|
+
SomeJob.set("sync": true).perform_async(args...) # will run via perform_inline
|
23
|
+
```
|
24
|
+
- Replace Sidekiq::Worker `app/workers` generator with Sidekiq::Job `app/sidekiq` generator [#5055]
|
25
|
+
```
|
26
|
+
bin/rails generate sidekiq:job ProcessOrderJob
|
27
|
+
```
|
28
|
+
- Fix job retries losing CurrentAttributes [#5090]
|
29
|
+
- Tweak shutdown to give long-running threads time to cleanup [#5095]
|
30
|
+
- Add keyword arguments support in extensions
|
31
|
+
|
32
|
+
6.3.1
|
33
|
+
---------
|
34
|
+
|
35
|
+
- Fix keyword arguments error with CurrentAttributes on Ruby 3.0 [#5048]
|
36
|
+
|
37
|
+
6.3.0
|
38
|
+
---------
|
39
|
+
|
40
|
+
- **BREAK**: The Web UI has been refactored to remove jQuery. Any UI extensions
|
41
|
+
which use jQuery will break.
|
42
|
+
- **FEATURE**: Sidekiq.logger has been enhanced so any `Rails.logger`
|
43
|
+
output in jobs now shows up in the Sidekiq console. Remove any logger
|
44
|
+
hacks in your initializer and see if it Just Works™ now. [#5021]
|
45
|
+
- **FEATURE**: Add `Sidekiq::Job` alias for `Sidekiq::Worker`, to better
|
46
|
+
reflect industry standard terminology. You can now do this:
|
47
|
+
```ruby
|
48
|
+
class MyJob
|
49
|
+
include Sidekiq::Job
|
50
|
+
sidekiq_options ...
|
51
|
+
def perform(args)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
```
|
55
|
+
- **FEATURE**: Support for serializing ActiveSupport::CurrentAttributes into each job. [#4982]
|
56
|
+
```ruby
|
57
|
+
# config/initializers/sidekiq.rb
|
58
|
+
require "sidekiq/middleware/current_attributes"
|
59
|
+
Sidekiq::CurrentAttributes.persist(Myapp::Current) # Your AS::CurrentAttributes singleton
|
60
|
+
```
|
61
|
+
- **FEATURE**: Add `Sidekiq::Worker.perform_bulk` for enqueuing jobs in bulk,
|
62
|
+
similar to `Sidekiq::Client.push_bulk` [#5042]
|
63
|
+
```ruby
|
64
|
+
MyJob.perform_bulk([[1], [2], [3]])
|
65
|
+
```
|
66
|
+
- Implement `queue_as`, `wait` and `wait_until` for ActiveJob compatibility [#5003]
|
67
|
+
- Scheduler now uses Lua to reduce Redis load and network roundtrips [#5044]
|
68
|
+
- Retry Redis operation if we get an `UNBLOCKED` Redis error [#4985]
|
69
|
+
- Run existing signal traps, if any, before running Sidekiq's trap [#4991]
|
70
|
+
- Fix fetch bug when using weighted queues which caused Sidekiq to stop
|
71
|
+
processing queues randomly [#5031]
|
72
|
+
|
73
|
+
6.2.2
|
74
|
+
---------
|
75
|
+
|
76
|
+
- Reduce retry jitter, add jitter to `sidekiq_retry_in` values [#4957]
|
77
|
+
- Minimize scheduler load on Redis at scale [#4882]
|
78
|
+
- Improve logging of delay jobs [#4904, BuonOno]
|
79
|
+
- Minor CSS improvements for buttons and tables, design PRs always welcome!
|
80
|
+
- Tweak Web UI `Cache-Control` header [#4966]
|
81
|
+
- Rename internal API class `Sidekiq::Job` to `Sidekiq::JobRecord` [#4955]
|
4
82
|
|
5
83
|
6.2.1
|
6
84
|
---------
|
data/LICENSE
CHANGED
@@ -4,6 +4,6 @@ Sidekiq is an Open Source project licensed under the terms of
|
|
4
4
|
the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
|
5
5
|
for license text.
|
6
6
|
|
7
|
-
Sidekiq Pro
|
8
|
-
|
9
|
-
|
7
|
+
Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
|
8
|
+
You can find the commercial license in COMM-LICENSE.txt.
|
9
|
+
Please see https://sidekiq.org for purchasing options.
|
data/README.md
CHANGED
@@ -43,10 +43,10 @@ Getting Started
|
|
43
43
|
-----------------
|
44
44
|
|
45
45
|
See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
|
46
|
-
You can watch [this
|
46
|
+
You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
|
47
47
|
Sidekiq and see its features in action. Here's the Web UI:
|
48
48
|
|
49
|
-
![Web UI](https://github.com/mperham/sidekiq/raw/
|
49
|
+
![Web UI](https://github.com/mperham/sidekiq/raw/main/examples/web-ui.png)
|
50
50
|
|
51
51
|
|
52
52
|
Want to Upgrade?
|
@@ -84,7 +84,7 @@ See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
|
|
84
84
|
License
|
85
85
|
-----------------
|
86
86
|
|
87
|
-
Please see [LICENSE](https://github.com/mperham/sidekiq/blob/
|
87
|
+
Please see [LICENSE](https://github.com/mperham/sidekiq/blob/main/LICENSE) for licensing details.
|
88
88
|
|
89
89
|
|
90
90
|
Author
|
@@ -0,0 +1,57 @@
|
|
1
|
+
require "rails/generators/named_base"
|
2
|
+
|
3
|
+
module Sidekiq
|
4
|
+
module Generators # :nodoc:
|
5
|
+
class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
|
6
|
+
desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
|
7
|
+
|
8
|
+
check_class_collision suffix: "Job"
|
9
|
+
|
10
|
+
def self.default_generator_root
|
11
|
+
File.dirname(__FILE__)
|
12
|
+
end
|
13
|
+
|
14
|
+
def create_job_file
|
15
|
+
template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
|
16
|
+
end
|
17
|
+
|
18
|
+
def create_test_file
|
19
|
+
return unless test_framework
|
20
|
+
|
21
|
+
if test_framework == :rspec
|
22
|
+
create_job_spec
|
23
|
+
else
|
24
|
+
create_job_test
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
private
|
29
|
+
|
30
|
+
def create_job_spec
|
31
|
+
template_file = File.join(
|
32
|
+
"spec/sidekiq",
|
33
|
+
class_path,
|
34
|
+
"#{file_name}_job_spec.rb"
|
35
|
+
)
|
36
|
+
template "job_spec.rb.erb", template_file
|
37
|
+
end
|
38
|
+
|
39
|
+
def create_job_test
|
40
|
+
template_file = File.join(
|
41
|
+
"test/sidekiq",
|
42
|
+
class_path,
|
43
|
+
"#{file_name}_job_test.rb"
|
44
|
+
)
|
45
|
+
template "job_test.rb.erb", template_file
|
46
|
+
end
|
47
|
+
|
48
|
+
def file_name
|
49
|
+
@_file_name ||= super.sub(/_?job\z/i, "")
|
50
|
+
end
|
51
|
+
|
52
|
+
def test_framework
|
53
|
+
::Rails.application.config.generators.options[:rails][:test_framework]
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
data/lib/sidekiq/api.rb
CHANGED
@@ -8,7 +8,7 @@ require "base64"
|
|
8
8
|
module Sidekiq
|
9
9
|
class Stats
|
10
10
|
def initialize
|
11
|
-
|
11
|
+
fetch_stats_fast!
|
12
12
|
end
|
13
13
|
|
14
14
|
def processed
|
@@ -51,7 +51,8 @@ module Sidekiq
|
|
51
51
|
Sidekiq::Stats::Queues.new.lengths
|
52
52
|
end
|
53
53
|
|
54
|
-
|
54
|
+
# O(1) redis calls
|
55
|
+
def fetch_stats_fast!
|
55
56
|
pipe1_res = Sidekiq.redis { |conn|
|
56
57
|
conn.pipelined do
|
57
58
|
conn.get("stat:processed")
|
@@ -64,25 +65,6 @@ module Sidekiq
|
|
64
65
|
end
|
65
66
|
}
|
66
67
|
|
67
|
-
processes = Sidekiq.redis { |conn|
|
68
|
-
conn.sscan_each("processes").to_a
|
69
|
-
}
|
70
|
-
|
71
|
-
queues = Sidekiq.redis { |conn|
|
72
|
-
conn.sscan_each("queues").to_a
|
73
|
-
}
|
74
|
-
|
75
|
-
pipe2_res = Sidekiq.redis { |conn|
|
76
|
-
conn.pipelined do
|
77
|
-
processes.each { |key| conn.hget(key, "busy") }
|
78
|
-
queues.each { |queue| conn.llen("queue:#{queue}") }
|
79
|
-
end
|
80
|
-
}
|
81
|
-
|
82
|
-
s = processes.size
|
83
|
-
workers_size = pipe2_res[0...s].sum(&:to_i)
|
84
|
-
enqueued = pipe2_res[s..-1].sum(&:to_i)
|
85
|
-
|
86
68
|
default_queue_latency = if (entry = pipe1_res[6].first)
|
87
69
|
job = begin
|
88
70
|
Sidekiq.load_json(entry)
|
@@ -95,6 +77,7 @@ module Sidekiq
|
|
95
77
|
else
|
96
78
|
0
|
97
79
|
end
|
80
|
+
|
98
81
|
@stats = {
|
99
82
|
processed: pipe1_res[0].to_i,
|
100
83
|
failed: pipe1_res[1].to_i,
|
@@ -103,10 +86,39 @@ module Sidekiq
|
|
103
86
|
dead_size: pipe1_res[4],
|
104
87
|
processes_size: pipe1_res[5],
|
105
88
|
|
106
|
-
default_queue_latency: default_queue_latency
|
107
|
-
|
108
|
-
|
89
|
+
default_queue_latency: default_queue_latency
|
90
|
+
}
|
91
|
+
end
|
92
|
+
|
93
|
+
# O(number of processes + number of queues) redis calls
|
94
|
+
def fetch_stats_slow!
|
95
|
+
processes = Sidekiq.redis { |conn|
|
96
|
+
conn.sscan_each("processes").to_a
|
97
|
+
}
|
98
|
+
|
99
|
+
queues = Sidekiq.redis { |conn|
|
100
|
+
conn.sscan_each("queues").to_a
|
109
101
|
}
|
102
|
+
|
103
|
+
pipe2_res = Sidekiq.redis { |conn|
|
104
|
+
conn.pipelined do
|
105
|
+
processes.each { |key| conn.hget(key, "busy") }
|
106
|
+
queues.each { |queue| conn.llen("queue:#{queue}") }
|
107
|
+
end
|
108
|
+
}
|
109
|
+
|
110
|
+
s = processes.size
|
111
|
+
workers_size = pipe2_res[0...s].sum(&:to_i)
|
112
|
+
enqueued = pipe2_res[s..-1].sum(&:to_i)
|
113
|
+
|
114
|
+
@stats[:workers_size] = workers_size
|
115
|
+
@stats[:enqueued] = enqueued
|
116
|
+
@stats
|
117
|
+
end
|
118
|
+
|
119
|
+
def fetch_stats!
|
120
|
+
fetch_stats_fast!
|
121
|
+
fetch_stats_slow!
|
110
122
|
end
|
111
123
|
|
112
124
|
def reset(*stats)
|
@@ -126,7 +138,8 @@ module Sidekiq
|
|
126
138
|
private
|
127
139
|
|
128
140
|
def stat(s)
|
129
|
-
@stats[s]
|
141
|
+
fetch_stats_slow! if @stats[s].nil?
|
142
|
+
@stats[s] || raise(ArgumentError, "Unknown stat #{s}")
|
130
143
|
end
|
131
144
|
|
132
145
|
class Queues
|
@@ -141,13 +154,15 @@ module Sidekiq
|
|
141
154
|
}
|
142
155
|
|
143
156
|
array_of_arrays = queues.zip(lengths).sort_by { |_, size| -size }
|
144
|
-
|
157
|
+
array_of_arrays.to_h
|
145
158
|
end
|
146
159
|
end
|
147
160
|
end
|
148
161
|
|
149
162
|
class History
|
150
163
|
def initialize(days_previous, start_date = nil)
|
164
|
+
# we only store five years of data in Redis
|
165
|
+
raise ArgumentError if days_previous < 1 || days_previous > (5 * 365)
|
151
166
|
@days_previous = days_previous
|
152
167
|
@start_date = start_date || Time.now.utc.to_date
|
153
168
|
end
|
@@ -255,7 +270,7 @@ module Sidekiq
|
|
255
270
|
break if entries.empty?
|
256
271
|
page += 1
|
257
272
|
entries.each do |entry|
|
258
|
-
yield
|
273
|
+
yield JobRecord.new(entry, @name)
|
259
274
|
end
|
260
275
|
deleted_size = initial_size - size
|
261
276
|
end
|
@@ -265,7 +280,7 @@ module Sidekiq
|
|
265
280
|
# Find the job with the given JID within this queue.
|
266
281
|
#
|
267
282
|
# This is a slow, inefficient operation. Do not use under
|
268
|
-
# normal conditions.
|
283
|
+
# normal conditions.
|
269
284
|
def find_job(jid)
|
270
285
|
detect { |j| j.jid == jid }
|
271
286
|
end
|
@@ -286,9 +301,9 @@ module Sidekiq
|
|
286
301
|
# sorted set.
|
287
302
|
#
|
288
303
|
# The job should be considered immutable but may be
|
289
|
-
# removed from the queue via
|
304
|
+
# removed from the queue via JobRecord#delete.
|
290
305
|
#
|
291
|
-
class
|
306
|
+
class JobRecord
|
292
307
|
attr_reader :item
|
293
308
|
attr_reader :value
|
294
309
|
|
@@ -316,21 +331,23 @@ module Sidekiq
|
|
316
331
|
|
317
332
|
def display_class
|
318
333
|
# Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
|
319
|
-
@klass ||=
|
320
|
-
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
|
328
|
-
|
329
|
-
|
330
|
-
|
331
|
-
|
332
|
-
|
333
|
-
|
334
|
+
@klass ||= self["display_class"] || begin
|
335
|
+
case klass
|
336
|
+
when /\ASidekiq::Extensions::Delayed/
|
337
|
+
safe_load(args[0], klass) do |target, method, _|
|
338
|
+
"#{target}.#{method}"
|
339
|
+
end
|
340
|
+
when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
|
341
|
+
job_class = @item["wrapped"] || args[0]
|
342
|
+
if job_class == "ActionMailer::DeliveryJob" || job_class == "ActionMailer::MailDeliveryJob"
|
343
|
+
# MailerClass#mailer_method
|
344
|
+
args[0]["arguments"][0..1].join("#")
|
345
|
+
else
|
346
|
+
job_class
|
347
|
+
end
|
348
|
+
else
|
349
|
+
klass
|
350
|
+
end
|
334
351
|
end
|
335
352
|
end
|
336
353
|
|
@@ -443,7 +460,7 @@ module Sidekiq
|
|
443
460
|
end
|
444
461
|
end
|
445
462
|
|
446
|
-
class SortedEntry <
|
463
|
+
class SortedEntry < JobRecord
|
447
464
|
attr_reader :score
|
448
465
|
attr_reader :parent
|
449
466
|
|
@@ -804,10 +821,10 @@ module Sidekiq
|
|
804
821
|
|
805
822
|
hash = Sidekiq.load_json(info)
|
806
823
|
yield Process.new(hash.merge("busy" => busy.to_i,
|
807
|
-
|
808
|
-
|
809
|
-
|
810
|
-
|
824
|
+
"beat" => at_s.to_f,
|
825
|
+
"quiet" => quiet,
|
826
|
+
"rss" => rss.to_i,
|
827
|
+
"rtt_us" => rtt.to_i))
|
811
828
|
end
|
812
829
|
end
|
813
830
|
|
@@ -823,12 +840,13 @@ module Sidekiq
|
|
823
840
|
# For Sidekiq Enterprise customers this number (in production) must be
|
824
841
|
# less than or equal to your licensed concurrency.
|
825
842
|
def total_concurrency
|
826
|
-
sum { |x| x["concurrency"] }
|
843
|
+
sum { |x| x["concurrency"].to_i }
|
827
844
|
end
|
828
845
|
|
829
|
-
def
|
830
|
-
sum { |x| x["rss"]
|
846
|
+
def total_rss_in_kb
|
847
|
+
sum { |x| x["rss"].to_i }
|
831
848
|
end
|
849
|
+
alias_method :total_rss, :total_rss_in_kb
|
832
850
|
|
833
851
|
# Returns the identity of the current cluster leader or "" if no leader.
|
834
852
|
# This is a Sidekiq Enterprise feature, will always return "" in Sidekiq
|
@@ -879,6 +897,10 @@ module Sidekiq
|
|
879
897
|
self["identity"]
|
880
898
|
end
|
881
899
|
|
900
|
+
def queues
|
901
|
+
self["queues"]
|
902
|
+
end
|
903
|
+
|
882
904
|
def quiet!
|
883
905
|
signal("TSTP")
|
884
906
|
end
|
@@ -909,8 +931,8 @@ module Sidekiq
|
|
909
931
|
end
|
910
932
|
|
911
933
|
##
|
912
|
-
#
|
913
|
-
#
|
934
|
+
# The WorkSet stores the work being done by this Sidekiq cluster.
|
935
|
+
# It tracks the process and thread working on each job.
|
914
936
|
#
|
915
937
|
# WARNING WARNING WARNING
|
916
938
|
#
|
@@ -918,17 +940,17 @@ module Sidekiq
|
|
918
940
|
# If you call #size => 5 and then expect #each to be
|
919
941
|
# called 5 times, you're going to have a bad time.
|
920
942
|
#
|
921
|
-
#
|
922
|
-
#
|
923
|
-
#
|
943
|
+
# works = Sidekiq::WorkSet.new
|
944
|
+
# works.size => 2
|
945
|
+
# works.each do |process_id, thread_id, work|
|
924
946
|
# # process_id is a unique identifier per Sidekiq process
|
925
947
|
# # thread_id is a unique identifier per thread
|
926
948
|
# # work is a Hash which looks like:
|
927
|
-
# # { 'queue' => name, 'run_at' => timestamp, 'payload' =>
|
949
|
+
# # { 'queue' => name, 'run_at' => timestamp, 'payload' => job_hash }
|
928
950
|
# # run_at is an epoch Integer.
|
929
951
|
# end
|
930
952
|
#
|
931
|
-
class
|
953
|
+
class WorkSet
|
932
954
|
include Enumerable
|
933
955
|
|
934
956
|
def each(&block)
|
@@ -975,4 +997,8 @@ module Sidekiq
|
|
975
997
|
end
|
976
998
|
end
|
977
999
|
end
|
1000
|
+
# Since "worker" is a nebulous term, we've deprecated the use of this class name.
|
1001
|
+
# Is "worker" a process, a type of job, a thread? Undefined!
|
1002
|
+
# WorkSet better describes the data.
|
1003
|
+
Workers = WorkSet
|
978
1004
|
end
|
data/lib/sidekiq/cli.rb
CHANGED
@@ -46,7 +46,15 @@ module Sidekiq
|
|
46
46
|
# USR1 and USR2 don't work on the JVM
|
47
47
|
sigs << "USR2" if Sidekiq.pro? && !jruby?
|
48
48
|
sigs.each do |sig|
|
49
|
-
trap
|
49
|
+
old_handler = Signal.trap(sig) do
|
50
|
+
if old_handler.respond_to?(:call)
|
51
|
+
begin
|
52
|
+
old_handler.call
|
53
|
+
rescue Exception => exc
|
54
|
+
# signal handlers can't use Logger so puts only
|
55
|
+
puts ["Error in #{sig} handler", exc].inspect
|
56
|
+
end
|
57
|
+
end
|
50
58
|
self_write.puts(sig)
|
51
59
|
end
|
52
60
|
rescue ArgumentError
|
@@ -372,7 +380,9 @@ module Sidekiq
|
|
372
380
|
end
|
373
381
|
|
374
382
|
def parse_config(path)
|
375
|
-
|
383
|
+
erb = ERB.new(File.read(path))
|
384
|
+
erb.filename = File.expand_path(path)
|
385
|
+
opts = YAML.load(erb.result) || {}
|
376
386
|
|
377
387
|
if opts.respond_to? :deep_symbolize_keys!
|
378
388
|
opts.deep_symbolize_keys!
|
@@ -396,7 +406,7 @@ module Sidekiq
|
|
396
406
|
opts[:queues] ||= []
|
397
407
|
opts[:strict] = true if opts[:strict].nil?
|
398
408
|
raise ArgumentError, "queues: #{queue} cannot be defined twice" if opts[:queues].include?(queue)
|
399
|
-
[weight.to_i, 1].max.times { opts[:queues] << queue }
|
409
|
+
[weight.to_i, 1].max.times { opts[:queues] << queue.to_s }
|
400
410
|
opts[:strict] = false if weight.to_i > 0
|
401
411
|
end
|
402
412
|
|
data/lib/sidekiq/client.rb
CHANGED
@@ -2,9 +2,12 @@
|
|
2
2
|
|
3
3
|
require "securerandom"
|
4
4
|
require "sidekiq/middleware/chain"
|
5
|
+
require "sidekiq/job_util"
|
5
6
|
|
6
7
|
module Sidekiq
|
7
8
|
class Client
|
9
|
+
include Sidekiq::JobUtil
|
10
|
+
|
8
11
|
##
|
9
12
|
# Define client-side middleware:
|
10
13
|
#
|
@@ -95,7 +98,7 @@ module Sidekiq
|
|
95
98
|
return [] if args.empty? # no jobs to push
|
96
99
|
|
97
100
|
at = items.delete("at")
|
98
|
-
raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all?(Numeric))
|
101
|
+
raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all? { |entry| entry.is_a?(Numeric) })
|
99
102
|
raise ArgumentError, "Job 'at' Array must have same size as 'args' Array" if at.is_a?(Array) && at.size != args.size
|
100
103
|
|
101
104
|
normed = normalize_item(items)
|
@@ -186,7 +189,7 @@ module Sidekiq
|
|
186
189
|
|
187
190
|
def raw_push(payloads)
|
188
191
|
@redis_pool.with do |conn|
|
189
|
-
conn.
|
192
|
+
conn.pipelined do
|
190
193
|
atomic_push(conn, payloads)
|
191
194
|
end
|
192
195
|
end
|
@@ -218,42 +221,5 @@ module Sidekiq
|
|
218
221
|
item
|
219
222
|
end
|
220
223
|
end
|
221
|
-
|
222
|
-
def validate(item)
|
223
|
-
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
|
224
|
-
raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
|
225
|
-
raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
|
226
|
-
raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
|
227
|
-
raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
|
228
|
-
end
|
229
|
-
|
230
|
-
def normalize_item(item)
|
231
|
-
validate(item)
|
232
|
-
# raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
|
233
|
-
|
234
|
-
# merge in the default sidekiq_options for the item's class and/or wrapped element
|
235
|
-
# this allows ActiveJobs to control sidekiq_options too.
|
236
|
-
defaults = normalized_hash(item["class"])
|
237
|
-
defaults = defaults.merge(item["wrapped"].get_sidekiq_options) if item["wrapped"].respond_to?("get_sidekiq_options")
|
238
|
-
item = defaults.merge(item)
|
239
|
-
|
240
|
-
raise(ArgumentError, "Job must include a valid queue name") if item["queue"].nil? || item["queue"] == ""
|
241
|
-
|
242
|
-
item["class"] = item["class"].to_s
|
243
|
-
item["queue"] = item["queue"].to_s
|
244
|
-
item["jid"] ||= SecureRandom.hex(12)
|
245
|
-
item["created_at"] ||= Time.now.to_f
|
246
|
-
|
247
|
-
item
|
248
|
-
end
|
249
|
-
|
250
|
-
def normalized_hash(item_class)
|
251
|
-
if item_class.is_a?(Class)
|
252
|
-
raise(ArgumentError, "Message must include a Sidekiq::Worker class, not class name: #{item_class.ancestors.inspect}") unless item_class.respond_to?("get_sidekiq_options")
|
253
|
-
item_class.get_sidekiq_options
|
254
|
-
else
|
255
|
-
Sidekiq.default_worker_options
|
256
|
-
end
|
257
|
-
end
|
258
224
|
end
|
259
225
|
end
|
data/lib/sidekiq/delay.rb
CHANGED
@@ -3,6 +3,8 @@
|
|
3
3
|
module Sidekiq
|
4
4
|
module Extensions
|
5
5
|
def self.enable_delay!
|
6
|
+
Sidekiq.logger.error "Sidekiq's Delayed Extensions will be removed in Sidekiq 7.0. #{caller(1..1).first}"
|
7
|
+
|
6
8
|
if defined?(::ActiveSupport)
|
7
9
|
require "sidekiq/extensions/active_record"
|
8
10
|
require "sidekiq/extensions/action_mailer"
|
@@ -16,8 +16,8 @@ module Sidekiq
|
|
16
16
|
include Sidekiq::Worker
|
17
17
|
|
18
18
|
def perform(yml)
|
19
|
-
(target, method_name, args) = YAML.load(yml)
|
20
|
-
msg = target.public_send(method_name, *args)
|
19
|
+
(target, method_name, args, kwargs) = YAML.load(yml)
|
20
|
+
msg = kwargs.empty? ? target.public_send(method_name, *args) : target.public_send(method_name, *args, **kwargs)
|
21
21
|
# The email method can return nil, which causes ActionMailer to return
|
22
22
|
# an undeliverable empty message.
|
23
23
|
if msg
|
@@ -18,8 +18,8 @@ module Sidekiq
|
|
18
18
|
include Sidekiq::Worker
|
19
19
|
|
20
20
|
def perform(yml)
|
21
|
-
(target, method_name, args) = YAML.load(yml)
|
22
|
-
target.__send__(method_name, *args)
|
21
|
+
(target, method_name, args, kwargs) = YAML.load(yml)
|
22
|
+
kwargs.empty? ? target.__send__(method_name, *args) : target.__send__(method_name, *args, **kwargs)
|
23
23
|
end
|
24
24
|
end
|
25
25
|
|
@@ -16,8 +16,8 @@ module Sidekiq
|
|
16
16
|
include Sidekiq::Worker
|
17
17
|
|
18
18
|
def perform(yml)
|
19
|
-
(target, method_name, args) = YAML.load(yml)
|
20
|
-
target.__send__(method_name, *args)
|
19
|
+
(target, method_name, args, kwargs) = YAML.load(yml)
|
20
|
+
kwargs.empty? ? target.__send__(method_name, *args) : target.__send__(method_name, *args, **kwargs)
|
21
21
|
end
|
22
22
|
end
|
23
23
|
|