sidekiq 6.3.1 → 6.4.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/Changes.md +27 -0
- data/LICENSE +3 -3
- data/README.md +1 -1
- data/lib/generators/sidekiq/job_generator.rb +57 -0
- data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
- data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
- data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
- data/lib/sidekiq/api.rb +6 -4
- data/lib/sidekiq/cli.rb +3 -1
- data/lib/sidekiq/client.rb +3 -37
- data/lib/sidekiq/delay.rb +2 -0
- data/lib/sidekiq/extensions/action_mailer.rb +2 -2
- data/lib/sidekiq/extensions/active_record.rb +2 -2
- data/lib/sidekiq/extensions/class_methods.rb +2 -2
- data/lib/sidekiq/extensions/generic_proxy.rb +2 -2
- data/lib/sidekiq/job_retry.rb +6 -4
- data/lib/sidekiq/job_util.rb +65 -0
- data/lib/sidekiq/manager.rb +7 -9
- data/lib/sidekiq/middleware/current_attributes.rb +6 -1
- data/lib/sidekiq/rails.rb +1 -1
- data/lib/sidekiq/scheduled.rb +13 -3
- data/lib/sidekiq/util.rb +13 -0
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/application.rb +5 -2
- data/lib/sidekiq/worker.rb +58 -5
- data/lib/sidekiq.rb +5 -0
- data/web/assets/stylesheets/application-dark.css +13 -17
- data/web/assets/stylesheets/application.css +3 -5
- metadata +7 -6
- data/lib/generators/sidekiq/worker_generator.rb +0 -57
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9622b2851203b0c5a80695ab7801ca77e15dc63a641ae79132cda9a2fcbe0cc6
|
4
|
+
data.tar.gz: dd943a02d2cf910f51866d02254f3a7845cb159735bd54d223dd7127a1fbd2fa
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 65bcd542866d8699ecf5958d81a15fd322cd1c51dfc1dbc6a8b15402b70862510c134e2b0ed9f9dbcdbb4dea3a59c1d12878ff926145e503079a59896f279ff3
|
7
|
+
data.tar.gz: 36143e85dc7fd4611f8a43fe39788dc590725ac63a827fbc174916da2d59fdadb3fb64fa0d819c4ef90694f6f292ea15f7fb58fb29f368c0a06c5b90f1b4bca7
|
data/Changes.md
CHANGED
@@ -2,6 +2,33 @@
|
|
2
2
|
|
3
3
|
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
|
4
4
|
|
5
|
+
6.4.0
|
6
|
+
---------
|
7
|
+
|
8
|
+
- **SECURITY**: Validate input to avoid possible DoS in Web UI.
|
9
|
+
- Add **strict argument checking** [#5071]
|
10
|
+
Sidekiq will now log a warning if JSON-unsafe arguments are passed to `perform_async`.
|
11
|
+
Add `Sidekiq.strict_args!(false)` to your initializer to disable this warning.
|
12
|
+
This warning will switch to an exception in Sidekiq 7.0.
|
13
|
+
- Note that Delayed Extensions will be removed in Sidekiq 7.0 [#5076]
|
14
|
+
- Add `perform_{inline,sync}` in Sidekiq::Job to run a job synchronously [#5061, hasan-ally]
|
15
|
+
```ruby
|
16
|
+
SomeJob.perform_async(args...)
|
17
|
+
SomeJob.perform_sync(args...)
|
18
|
+
SomeJob.perform_inline(args...)
|
19
|
+
```
|
20
|
+
You can also dynamically redirect a job to run synchronously:
|
21
|
+
```ruby
|
22
|
+
SomeJob.set("sync": true).perform_async(args...) # will run via perform_inline
|
23
|
+
```
|
24
|
+
- Replace Sidekiq::Worker `app/workers` generator with Sidekiq::Job `app/sidekiq` generator [#5055]
|
25
|
+
```
|
26
|
+
bin/rails generate sidekiq:job ProcessOrderJob
|
27
|
+
```
|
28
|
+
- Fix job retries losing CurrentAttributes [#5090]
|
29
|
+
- Tweak shutdown to give long-running threads time to cleanup [#5095]
|
30
|
+
- Add keyword arguments support in extensions
|
31
|
+
|
5
32
|
6.3.1
|
6
33
|
---------
|
7
34
|
|
data/LICENSE
CHANGED
@@ -4,6 +4,6 @@ Sidekiq is an Open Source project licensed under the terms of
|
|
4
4
|
the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
|
5
5
|
for license text.
|
6
6
|
|
7
|
-
Sidekiq Pro
|
8
|
-
|
9
|
-
|
7
|
+
Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
|
8
|
+
You can find the commercial license in COMM-LICENSE.txt.
|
9
|
+
Please see https://sidekiq.org for purchasing options.
|
data/README.md
CHANGED
@@ -43,7 +43,7 @@ Getting Started
|
|
43
43
|
-----------------
|
44
44
|
|
45
45
|
See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
|
46
|
-
You can watch [this
|
46
|
+
You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
|
47
47
|
Sidekiq and see its features in action. Here's the Web UI:
|
48
48
|
|
49
49
|
![Web UI](https://github.com/mperham/sidekiq/raw/main/examples/web-ui.png)
|
@@ -0,0 +1,57 @@
|
|
1
|
+
require "rails/generators/named_base"
|
2
|
+
|
3
|
+
module Sidekiq
|
4
|
+
module Generators # :nodoc:
|
5
|
+
class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
|
6
|
+
desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
|
7
|
+
|
8
|
+
check_class_collision suffix: "Job"
|
9
|
+
|
10
|
+
def self.default_generator_root
|
11
|
+
File.dirname(__FILE__)
|
12
|
+
end
|
13
|
+
|
14
|
+
def create_job_file
|
15
|
+
template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
|
16
|
+
end
|
17
|
+
|
18
|
+
def create_test_file
|
19
|
+
return unless test_framework
|
20
|
+
|
21
|
+
if test_framework == :rspec
|
22
|
+
create_job_spec
|
23
|
+
else
|
24
|
+
create_job_test
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
private
|
29
|
+
|
30
|
+
def create_job_spec
|
31
|
+
template_file = File.join(
|
32
|
+
"spec/sidekiq",
|
33
|
+
class_path,
|
34
|
+
"#{file_name}_job_spec.rb"
|
35
|
+
)
|
36
|
+
template "job_spec.rb.erb", template_file
|
37
|
+
end
|
38
|
+
|
39
|
+
def create_job_test
|
40
|
+
template_file = File.join(
|
41
|
+
"test/sidekiq",
|
42
|
+
class_path,
|
43
|
+
"#{file_name}_job_test.rb"
|
44
|
+
)
|
45
|
+
template "job_test.rb.erb", template_file
|
46
|
+
end
|
47
|
+
|
48
|
+
def file_name
|
49
|
+
@_file_name ||= super.sub(/_?job\z/i, "")
|
50
|
+
end
|
51
|
+
|
52
|
+
def test_framework
|
53
|
+
::Rails.application.config.generators.options[:rails][:test_framework]
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
data/lib/sidekiq/api.rb
CHANGED
@@ -161,6 +161,8 @@ module Sidekiq
|
|
161
161
|
|
162
162
|
class History
|
163
163
|
def initialize(days_previous, start_date = nil)
|
164
|
+
# we only store five years of data in Redis
|
165
|
+
raise ArgumentError if days_previous < 1 || days_previous > (5 * 365)
|
164
166
|
@days_previous = days_previous
|
165
167
|
@start_date = start_date || Time.now.utc.to_date
|
166
168
|
end
|
@@ -819,10 +821,10 @@ module Sidekiq
|
|
819
821
|
|
820
822
|
hash = Sidekiq.load_json(info)
|
821
823
|
yield Process.new(hash.merge("busy" => busy.to_i,
|
822
|
-
|
823
|
-
|
824
|
-
|
825
|
-
|
824
|
+
"beat" => at_s.to_f,
|
825
|
+
"quiet" => quiet,
|
826
|
+
"rss" => rss.to_i,
|
827
|
+
"rtt_us" => rtt.to_i))
|
826
828
|
end
|
827
829
|
end
|
828
830
|
|
data/lib/sidekiq/cli.rb
CHANGED
@@ -380,7 +380,9 @@ module Sidekiq
|
|
380
380
|
end
|
381
381
|
|
382
382
|
def parse_config(path)
|
383
|
-
|
383
|
+
erb = ERB.new(File.read(path))
|
384
|
+
erb.filename = File.expand_path(path)
|
385
|
+
opts = YAML.load(erb.result) || {}
|
384
386
|
|
385
387
|
if opts.respond_to? :deep_symbolize_keys!
|
386
388
|
opts.deep_symbolize_keys!
|
data/lib/sidekiq/client.rb
CHANGED
@@ -2,9 +2,12 @@
|
|
2
2
|
|
3
3
|
require "securerandom"
|
4
4
|
require "sidekiq/middleware/chain"
|
5
|
+
require "sidekiq/job_util"
|
5
6
|
|
6
7
|
module Sidekiq
|
7
8
|
class Client
|
9
|
+
include Sidekiq::JobUtil
|
10
|
+
|
8
11
|
##
|
9
12
|
# Define client-side middleware:
|
10
13
|
#
|
@@ -218,42 +221,5 @@ module Sidekiq
|
|
218
221
|
item
|
219
222
|
end
|
220
223
|
end
|
221
|
-
|
222
|
-
def validate(item)
|
223
|
-
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
|
224
|
-
raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
|
225
|
-
raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
|
226
|
-
raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
|
227
|
-
raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
|
228
|
-
end
|
229
|
-
|
230
|
-
def normalize_item(item)
|
231
|
-
validate(item)
|
232
|
-
# raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
|
233
|
-
|
234
|
-
# merge in the default sidekiq_options for the item's class and/or wrapped element
|
235
|
-
# this allows ActiveJobs to control sidekiq_options too.
|
236
|
-
defaults = normalized_hash(item["class"])
|
237
|
-
defaults = defaults.merge(item["wrapped"].get_sidekiq_options) if item["wrapped"].respond_to?("get_sidekiq_options")
|
238
|
-
item = defaults.merge(item)
|
239
|
-
|
240
|
-
raise(ArgumentError, "Job must include a valid queue name") if item["queue"].nil? || item["queue"] == ""
|
241
|
-
|
242
|
-
item["class"] = item["class"].to_s
|
243
|
-
item["queue"] = item["queue"].to_s
|
244
|
-
item["jid"] ||= SecureRandom.hex(12)
|
245
|
-
item["created_at"] ||= Time.now.to_f
|
246
|
-
|
247
|
-
item
|
248
|
-
end
|
249
|
-
|
250
|
-
def normalized_hash(item_class)
|
251
|
-
if item_class.is_a?(Class)
|
252
|
-
raise(ArgumentError, "Message must include a Sidekiq::Worker class, not class name: #{item_class.ancestors.inspect}") unless item_class.respond_to?("get_sidekiq_options")
|
253
|
-
item_class.get_sidekiq_options
|
254
|
-
else
|
255
|
-
Sidekiq.default_worker_options
|
256
|
-
end
|
257
|
-
end
|
258
224
|
end
|
259
225
|
end
|
data/lib/sidekiq/delay.rb
CHANGED
@@ -3,6 +3,8 @@
|
|
3
3
|
module Sidekiq
|
4
4
|
module Extensions
|
5
5
|
def self.enable_delay!
|
6
|
+
Sidekiq.logger.error "Sidekiq's Delayed Extensions will be removed in Sidekiq 7.0. #{caller(1..1).first}"
|
7
|
+
|
6
8
|
if defined?(::ActiveSupport)
|
7
9
|
require "sidekiq/extensions/active_record"
|
8
10
|
require "sidekiq/extensions/action_mailer"
|
@@ -16,8 +16,8 @@ module Sidekiq
|
|
16
16
|
include Sidekiq::Worker
|
17
17
|
|
18
18
|
def perform(yml)
|
19
|
-
(target, method_name, args) = YAML.load(yml)
|
20
|
-
msg = target.public_send(method_name, *args)
|
19
|
+
(target, method_name, args, kwargs) = YAML.load(yml)
|
20
|
+
msg = kwargs.empty? ? target.public_send(method_name, *args) : target.public_send(method_name, *args, **kwargs)
|
21
21
|
# The email method can return nil, which causes ActionMailer to return
|
22
22
|
# an undeliverable empty message.
|
23
23
|
if msg
|
@@ -18,8 +18,8 @@ module Sidekiq
|
|
18
18
|
include Sidekiq::Worker
|
19
19
|
|
20
20
|
def perform(yml)
|
21
|
-
(target, method_name, args) = YAML.load(yml)
|
22
|
-
target.__send__(method_name, *args)
|
21
|
+
(target, method_name, args, kwargs) = YAML.load(yml)
|
22
|
+
kwargs.empty? ? target.__send__(method_name, *args) : target.__send__(method_name, *args, **kwargs)
|
23
23
|
end
|
24
24
|
end
|
25
25
|
|
@@ -16,8 +16,8 @@ module Sidekiq
|
|
16
16
|
include Sidekiq::Worker
|
17
17
|
|
18
18
|
def perform(yml)
|
19
|
-
(target, method_name, args) = YAML.load(yml)
|
20
|
-
target.__send__(method_name, *args)
|
19
|
+
(target, method_name, args, kwargs) = YAML.load(yml)
|
20
|
+
kwargs.empty? ? target.__send__(method_name, *args) : target.__send__(method_name, *args, **kwargs)
|
21
21
|
end
|
22
22
|
end
|
23
23
|
|
@@ -13,13 +13,13 @@ module Sidekiq
|
|
13
13
|
@opts = options
|
14
14
|
end
|
15
15
|
|
16
|
-
def method_missing(name, *args)
|
16
|
+
def method_missing(name, *args, **kwargs)
|
17
17
|
# Sidekiq has a limitation in that its message must be JSON.
|
18
18
|
# JSON can't round trip real Ruby objects so we use YAML to
|
19
19
|
# serialize the objects to a String. The YAML will be converted
|
20
20
|
# to JSON and then deserialized on the other side back into a
|
21
21
|
# Ruby object.
|
22
|
-
obj = [@target, name, args]
|
22
|
+
obj = [@target, name, args, kwargs]
|
23
23
|
marshalled = ::YAML.dump(obj)
|
24
24
|
if marshalled.size > SIZE_LIMIT
|
25
25
|
::Sidekiq.logger.warn { "#{@target}.#{name} job argument is #{marshalled.bytesize} bytes, you should refactor it to reduce the size" }
|
data/lib/sidekiq/job_retry.rb
CHANGED
@@ -34,9 +34,10 @@ module Sidekiq
|
|
34
34
|
# The job will be retried this number of times before giving up. (If simply
|
35
35
|
# 'true', Sidekiq retries 25 times)
|
36
36
|
#
|
37
|
-
#
|
37
|
+
# Relevant options for job retries:
|
38
38
|
#
|
39
|
-
# * 'queue' - the queue
|
39
|
+
# * 'queue' - the queue for the initial job
|
40
|
+
# * 'retry_queue' - if job retries should be pushed to a different (e.g. lower priority) queue
|
40
41
|
# * 'retry_count' - number of times we've retried so far.
|
41
42
|
# * 'error_message' - the message from the exception
|
42
43
|
# * 'error_class' - the exception class
|
@@ -52,11 +53,12 @@ module Sidekiq
|
|
52
53
|
#
|
53
54
|
# Sidekiq.options[:max_retries] = 7
|
54
55
|
#
|
55
|
-
# or limit the number of retries for a particular worker
|
56
|
+
# or limit the number of retries for a particular worker and send retries to
|
57
|
+
# a low priority queue with:
|
56
58
|
#
|
57
59
|
# class MyWorker
|
58
60
|
# include Sidekiq::Worker
|
59
|
-
# sidekiq_options :
|
61
|
+
# sidekiq_options retry: 10, retry_queue: 'low'
|
60
62
|
# end
|
61
63
|
#
|
62
64
|
class JobRetry
|
@@ -0,0 +1,65 @@
|
|
1
|
+
require "securerandom"
|
2
|
+
require "time"
|
3
|
+
|
4
|
+
module Sidekiq
|
5
|
+
module JobUtil
|
6
|
+
# These functions encapsulate various job utilities.
|
7
|
+
# They must be simple and free from side effects.
|
8
|
+
|
9
|
+
def validate(item)
|
10
|
+
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
|
11
|
+
raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
|
12
|
+
raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
|
13
|
+
raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
|
14
|
+
raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
|
15
|
+
|
16
|
+
if Sidekiq.options[:on_complex_arguments] == :raise
|
17
|
+
msg = <<~EOM
|
18
|
+
Job arguments to #{item["class"]} must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices.
|
19
|
+
To disable this error, remove `Sidekiq.strict_args!` from your initializer.
|
20
|
+
EOM
|
21
|
+
raise(ArgumentError, msg) unless json_safe?(item)
|
22
|
+
elsif Sidekiq.options[:on_complex_arguments] == :warn
|
23
|
+
Sidekiq.logger.warn <<~EOM unless json_safe?(item)
|
24
|
+
Job arguments to #{item["class"]} do not serialize to JSON safely. This will raise an error in
|
25
|
+
Sidekiq 7.0. See https://github.com/mperham/sidekiq/wiki/Best-Practices or raise an error today
|
26
|
+
by calling `Sidekiq.strict_args!` during Sidekiq initialization.
|
27
|
+
EOM
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
def normalize_item(item)
|
32
|
+
validate(item)
|
33
|
+
|
34
|
+
# merge in the default sidekiq_options for the item's class and/or wrapped element
|
35
|
+
# this allows ActiveJobs to control sidekiq_options too.
|
36
|
+
defaults = normalized_hash(item["class"])
|
37
|
+
defaults = defaults.merge(item["wrapped"].get_sidekiq_options) if item["wrapped"].respond_to?(:get_sidekiq_options)
|
38
|
+
item = defaults.merge(item)
|
39
|
+
|
40
|
+
raise(ArgumentError, "Job must include a valid queue name") if item["queue"].nil? || item["queue"] == ""
|
41
|
+
|
42
|
+
item["class"] = item["class"].to_s
|
43
|
+
item["queue"] = item["queue"].to_s
|
44
|
+
item["jid"] ||= SecureRandom.hex(12)
|
45
|
+
item["created_at"] ||= Time.now.to_f
|
46
|
+
|
47
|
+
item
|
48
|
+
end
|
49
|
+
|
50
|
+
def normalized_hash(item_class)
|
51
|
+
if item_class.is_a?(Class)
|
52
|
+
raise(ArgumentError, "Message must include a Sidekiq::Worker class, not class name: #{item_class.ancestors.inspect}") unless item_class.respond_to?(:get_sidekiq_options)
|
53
|
+
item_class.get_sidekiq_options
|
54
|
+
else
|
55
|
+
Sidekiq.default_worker_options
|
56
|
+
end
|
57
|
+
end
|
58
|
+
|
59
|
+
private
|
60
|
+
|
61
|
+
def json_safe?(item)
|
62
|
+
JSON.parse(JSON.dump(item["args"])) == item["args"]
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
data/lib/sidekiq/manager.rb
CHANGED
@@ -55,9 +55,6 @@ module Sidekiq
|
|
55
55
|
fire_event(:quiet, reverse: true)
|
56
56
|
end
|
57
57
|
|
58
|
-
# hack for quicker development / testing environment #2774
|
59
|
-
PAUSE_TIME = $stdout.tty? ? 0.1 : 0.5
|
60
|
-
|
61
58
|
def stop(deadline)
|
62
59
|
quiet
|
63
60
|
fire_event(:shutdown, reverse: true)
|
@@ -69,12 +66,7 @@ module Sidekiq
|
|
69
66
|
return if @workers.empty?
|
70
67
|
|
71
68
|
logger.info { "Pausing to allow workers to finish..." }
|
72
|
-
|
73
|
-
while remaining > PAUSE_TIME
|
74
|
-
return if @workers.empty?
|
75
|
-
sleep PAUSE_TIME
|
76
|
-
remaining = deadline - ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
|
77
|
-
end
|
69
|
+
wait_for(deadline) { @workers.empty? }
|
78
70
|
return if @workers.empty?
|
79
71
|
|
80
72
|
hard_shutdown
|
@@ -130,6 +122,12 @@ module Sidekiq
|
|
130
122
|
cleanup.each do |processor|
|
131
123
|
processor.kill
|
132
124
|
end
|
125
|
+
|
126
|
+
# when this method returns, we immediately call `exit` which may not give
|
127
|
+
# the remaining threads time to run `ensure` blocks, etc. We pause here up
|
128
|
+
# to 3 seconds to give threads a minimal amount of time to run `ensure` blocks.
|
129
|
+
deadline = ::Process.clock_gettime(::Process::CLOCK_MONOTONIC) + 3
|
130
|
+
wait_for(deadline) { @workers.empty? }
|
133
131
|
end
|
134
132
|
end
|
135
133
|
end
|
data/lib/sidekiq/rails.rb
CHANGED
@@ -42,7 +42,7 @@ module Sidekiq
|
|
42
42
|
# This is the integration code necessary so that if code uses `Rails.logger.info "Hello"`,
|
43
43
|
# it will appear in the Sidekiq console with all of the job context. See #5021 and
|
44
44
|
# https://github.com/rails/rails/blob/b5f2b550f69a99336482739000c58e4e04e033aa/railties/lib/rails/commands/server/server_command.rb#L82-L84
|
45
|
-
unless ::ActiveSupport::Logger.logger_outputs_to?(::Rails.logger, $stdout)
|
45
|
+
unless ::Rails.logger == ::Sidekiq.logger || ::ActiveSupport::Logger.logger_outputs_to?(::Rails.logger, $stdout)
|
46
46
|
::Rails.logger.extend(::ActiveSupport::Logger.broadcast(::Sidekiq.logger))
|
47
47
|
end
|
48
48
|
end
|
data/lib/sidekiq/scheduled.rb
CHANGED
@@ -19,10 +19,11 @@ module Sidekiq
|
|
19
19
|
LUA
|
20
20
|
|
21
21
|
def initialize
|
22
|
+
@done = false
|
22
23
|
@lua_zpopbyscore_sha = nil
|
23
24
|
end
|
24
25
|
|
25
|
-
def enqueue_jobs(
|
26
|
+
def enqueue_jobs(sorted_sets = SETS)
|
26
27
|
# A job's "score" in Redis is the time at which it should be processed.
|
27
28
|
# Just check Redis for the set of jobs with a timestamp before now.
|
28
29
|
Sidekiq.redis do |conn|
|
@@ -31,7 +32,7 @@ module Sidekiq
|
|
31
32
|
# We need to go through the list one at a time to reduce the risk of something
|
32
33
|
# going wrong between the time jobs are popped from the scheduled queue and when
|
33
34
|
# they are pushed onto a work queue and losing the jobs.
|
34
|
-
while (job = zpopbyscore(conn, keys: [sorted_set], argv: [now]))
|
35
|
+
while !@done && (job = zpopbyscore(conn, keys: [sorted_set], argv: [Time.now.to_f.to_s]))
|
35
36
|
Sidekiq::Client.push(Sidekiq.load_json(job))
|
36
37
|
Sidekiq.logger.debug { "enqueued #{sorted_set}: #{job}" }
|
37
38
|
end
|
@@ -39,10 +40,17 @@ module Sidekiq
|
|
39
40
|
end
|
40
41
|
end
|
41
42
|
|
43
|
+
def terminate
|
44
|
+
@done = true
|
45
|
+
end
|
46
|
+
|
42
47
|
private
|
43
48
|
|
44
49
|
def zpopbyscore(conn, keys: nil, argv: nil)
|
45
|
-
|
50
|
+
if @lua_zpopbyscore_sha.nil?
|
51
|
+
raw_conn = conn.respond_to?(:redis) ? conn.redis : conn
|
52
|
+
@lua_zpopbyscore_sha = raw_conn.script(:load, LUA_ZPOPBYSCORE)
|
53
|
+
end
|
46
54
|
|
47
55
|
conn.evalsha(@lua_zpopbyscore_sha, keys: keys, argv: argv)
|
48
56
|
rescue Redis::CommandError => e
|
@@ -74,6 +82,8 @@ module Sidekiq
|
|
74
82
|
# Shut down this instance, will pause until the thread is dead.
|
75
83
|
def terminate
|
76
84
|
@done = true
|
85
|
+
@enq.terminate if @enq.respond_to?(:terminate)
|
86
|
+
|
77
87
|
if @thread
|
78
88
|
t = @thread
|
79
89
|
@thread = nil
|
data/lib/sidekiq/util.rb
CHANGED
@@ -39,6 +39,19 @@ module Sidekiq
|
|
39
39
|
module Util
|
40
40
|
include ExceptionHandler
|
41
41
|
|
42
|
+
# hack for quicker development / testing environment #2774
|
43
|
+
PAUSE_TIME = $stdout.tty? ? 0.1 : 0.5
|
44
|
+
|
45
|
+
# Wait for the orblock to be true or the deadline passed.
|
46
|
+
def wait_for(deadline, &condblock)
|
47
|
+
remaining = deadline - ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
|
48
|
+
while remaining > PAUSE_TIME
|
49
|
+
return if condblock.call
|
50
|
+
sleep PAUSE_TIME
|
51
|
+
remaining = deadline - ::Process.clock_gettime(::Process::CLOCK_MONOTONIC)
|
52
|
+
end
|
53
|
+
end
|
54
|
+
|
42
55
|
def watchdog(last_words)
|
43
56
|
yield
|
44
57
|
rescue Exception => ex
|
data/lib/sidekiq/version.rb
CHANGED
@@ -50,7 +50,10 @@ module Sidekiq
|
|
50
50
|
|
51
51
|
get "/" do
|
52
52
|
@redis_info = redis_info.select { |k, v| REDIS_KEYS.include? k }
|
53
|
-
|
53
|
+
days = (params["days"] || 30).to_i
|
54
|
+
return halt(401) if days < 1 || days > 180
|
55
|
+
|
56
|
+
stats_history = Sidekiq::Stats::History.new(days)
|
54
57
|
@processed_history = stats_history.processed
|
55
58
|
@failed_history = stats_history.failed
|
56
59
|
|
@@ -299,7 +302,7 @@ module Sidekiq
|
|
299
302
|
return [404, {"Content-Type" => "text/plain", "X-Cascade" => "pass"}, ["Not Found"]] unless action
|
300
303
|
|
301
304
|
app = @klass
|
302
|
-
resp = catch(:halt) do
|
305
|
+
resp = catch(:halt) do
|
303
306
|
self.class.run_befores(app, action)
|
304
307
|
action.instance_exec env, &action.block
|
305
308
|
ensure
|
data/lib/sidekiq/worker.rb
CHANGED
@@ -171,6 +171,8 @@ module Sidekiq
|
|
171
171
|
# SomeWorker.set(queue: 'foo').perform_async(....)
|
172
172
|
#
|
173
173
|
class Setter
|
174
|
+
include Sidekiq::JobUtil
|
175
|
+
|
174
176
|
def initialize(klass, opts)
|
175
177
|
@klass = klass
|
176
178
|
@opts = opts
|
@@ -188,13 +190,57 @@ module Sidekiq
|
|
188
190
|
end
|
189
191
|
|
190
192
|
def perform_async(*args)
|
191
|
-
@
|
193
|
+
if @opts["sync"] == true
|
194
|
+
perform_inline(*args)
|
195
|
+
else
|
196
|
+
@klass.client_push(@opts.merge("args" => args, "class" => @klass))
|
197
|
+
end
|
198
|
+
end
|
199
|
+
|
200
|
+
# Explicit inline execution of a job. Returns nil if the job did not
|
201
|
+
# execute, true otherwise.
|
202
|
+
def perform_inline(*args)
|
203
|
+
raw = @opts.merge("args" => args, "class" => @klass).transform_keys(&:to_s)
|
204
|
+
|
205
|
+
# validate and normalize payload
|
206
|
+
item = normalize_item(raw)
|
207
|
+
queue = item["queue"]
|
208
|
+
|
209
|
+
# run client-side middleware
|
210
|
+
result = Sidekiq.client_middleware.invoke(item["class"], item, queue, Sidekiq.redis_pool) do
|
211
|
+
item
|
212
|
+
end
|
213
|
+
return nil unless result
|
214
|
+
|
215
|
+
# round-trip the payload via JSON
|
216
|
+
msg = Sidekiq.load_json(Sidekiq.dump_json(item))
|
217
|
+
|
218
|
+
# prepare the job instance
|
219
|
+
klass = msg["class"].constantize
|
220
|
+
job = klass.new
|
221
|
+
job.jid = msg["jid"]
|
222
|
+
job.bid = msg["bid"] if job.respond_to?(:bid)
|
223
|
+
|
224
|
+
# run the job through server-side middleware
|
225
|
+
result = Sidekiq.server_middleware.invoke(job, msg, msg["queue"]) do
|
226
|
+
# perform it
|
227
|
+
job.perform(*msg["args"])
|
228
|
+
true
|
229
|
+
end
|
230
|
+
return nil unless result
|
231
|
+
# jobs do not return a result. they should store any
|
232
|
+
# modified state.
|
233
|
+
true
|
192
234
|
end
|
235
|
+
alias_method :perform_sync, :perform_inline
|
193
236
|
|
194
237
|
def perform_bulk(args, batch_size: 1_000)
|
195
|
-
|
196
|
-
|
238
|
+
hash = @opts.transform_keys(&:to_s)
|
239
|
+
result = args.each_slice(batch_size).flat_map do |slice|
|
240
|
+
Sidekiq::Client.push_bulk(hash.merge("class" => @klass, "args" => slice))
|
197
241
|
end
|
242
|
+
|
243
|
+
result.is_a?(Enumerator::Lazy) ? result.force : result
|
198
244
|
end
|
199
245
|
|
200
246
|
# +interval+ must be a timestamp, numeric or something that acts
|
@@ -238,7 +284,12 @@ module Sidekiq
|
|
238
284
|
end
|
239
285
|
|
240
286
|
def perform_async(*args)
|
241
|
-
|
287
|
+
Setter.new(self, {}).perform_async(*args)
|
288
|
+
end
|
289
|
+
|
290
|
+
# Inline execution of job's perform method after passing through Sidekiq.client_middleware and Sidekiq.server_middleware
|
291
|
+
def perform_inline(*args)
|
292
|
+
Setter.new(self, {}).perform_inline(*args)
|
242
293
|
end
|
243
294
|
|
244
295
|
##
|
@@ -262,9 +313,11 @@ module Sidekiq
|
|
262
313
|
# SomeWorker.perform_bulk([[1], [2], [3]])
|
263
314
|
#
|
264
315
|
def perform_bulk(items, batch_size: 1_000)
|
265
|
-
items.each_slice(batch_size).flat_map do |slice|
|
316
|
+
result = items.each_slice(batch_size).flat_map do |slice|
|
266
317
|
Sidekiq::Client.push_bulk("class" => self, "args" => slice)
|
267
318
|
end
|
319
|
+
|
320
|
+
result.is_a?(Enumerator::Lazy) ? result.force : result
|
268
321
|
end
|
269
322
|
|
270
323
|
# +interval+ must be a timestamp, numeric or something that acts
|
data/lib/sidekiq.rb
CHANGED
@@ -26,6 +26,7 @@ module Sidekiq
|
|
26
26
|
timeout: 25,
|
27
27
|
poll_interval_average: nil,
|
28
28
|
average_scheduled_poll_interval: 5,
|
29
|
+
on_complex_arguments: :warn,
|
29
30
|
error_handlers: [],
|
30
31
|
death_handlers: [],
|
31
32
|
lifecycle_events: {
|
@@ -252,6 +253,10 @@ module Sidekiq
|
|
252
253
|
options[:lifecycle_events][event] << block
|
253
254
|
end
|
254
255
|
|
256
|
+
def self.strict_args!(mode = :raise)
|
257
|
+
options[:on_complex_arguments] = mode
|
258
|
+
end
|
259
|
+
|
255
260
|
# We are shutting down Sidekiq but what about workers that
|
256
261
|
# are working on some long job? This error is
|
257
262
|
# raised in workers that have not finished within the hard
|
@@ -1,6 +1,6 @@
|
|
1
1
|
html, body {
|
2
|
-
background-color: #
|
3
|
-
color: #
|
2
|
+
background-color: #171717 !important;
|
3
|
+
color: #DEDEDE;
|
4
4
|
}
|
5
5
|
|
6
6
|
a,
|
@@ -30,15 +30,15 @@ span.current-interval,
|
|
30
30
|
|
31
31
|
.navbar-inverse {
|
32
32
|
background-color: #222;
|
33
|
-
border-color: #
|
33
|
+
border-color: #444;
|
34
34
|
}
|
35
35
|
|
36
36
|
table {
|
37
|
-
background-color: #
|
37
|
+
background-color: #1D1D1D;
|
38
38
|
}
|
39
39
|
|
40
40
|
.table-striped > tbody > tr:nth-of-type(odd) {
|
41
|
-
background-color: #
|
41
|
+
background-color: #2E2E2E;
|
42
42
|
}
|
43
43
|
|
44
44
|
.table-bordered,
|
@@ -48,7 +48,7 @@ table {
|
|
48
48
|
.table-bordered > tfoot > tr > th,
|
49
49
|
.table-bordered > thead > tr > td,
|
50
50
|
.table-bordered > thead > tr > th {
|
51
|
-
border: 1px solid #
|
51
|
+
border: 1px solid #444;
|
52
52
|
}
|
53
53
|
|
54
54
|
.table-hover > tbody > tr:hover {
|
@@ -72,9 +72,7 @@ table {
|
|
72
72
|
background-color: #31708f;
|
73
73
|
}
|
74
74
|
|
75
|
-
a:link,
|
76
|
-
a:active,
|
77
|
-
a:hover {
|
75
|
+
a:link, a:active, a:hover, a:visited {
|
78
76
|
color: #ddd;
|
79
77
|
}
|
80
78
|
|
@@ -85,15 +83,13 @@ input {
|
|
85
83
|
}
|
86
84
|
|
87
85
|
.summary_bar .summary {
|
88
|
-
background-color: #
|
89
|
-
border: 1px solid #
|
90
|
-
|
91
|
-
box-shadow: 0 0 5px rgba(255, 255, 255, .5);
|
86
|
+
background-color: #232323;
|
87
|
+
border: 1px solid #444;
|
92
88
|
}
|
93
89
|
|
94
90
|
.navbar-default {
|
95
|
-
background-color: #
|
96
|
-
border-color: #
|
91
|
+
background-color: #0F0F0F;
|
92
|
+
border-color: #444;
|
97
93
|
}
|
98
94
|
|
99
95
|
.navbar-default .navbar-nav > .active > a,
|
@@ -112,7 +108,7 @@ input {
|
|
112
108
|
.pagination > li > span {
|
113
109
|
color: #ddd;
|
114
110
|
background-color: #333;
|
115
|
-
border-color: #
|
111
|
+
border-color: #444;
|
116
112
|
}
|
117
113
|
.pagination > .disabled > a,
|
118
114
|
.pagination > .disabled > a:focus,
|
@@ -122,7 +118,7 @@ input {
|
|
122
118
|
.pagination > .disabled > span:hover {
|
123
119
|
color: #ddd;
|
124
120
|
background-color: #333;
|
125
|
-
border-color: #
|
121
|
+
border-color: #444;
|
126
122
|
}
|
127
123
|
|
128
124
|
.stat {
|
@@ -21,7 +21,7 @@ body {
|
|
21
21
|
a {
|
22
22
|
color: #b1003e;
|
23
23
|
}
|
24
|
-
a:active, a:hover {
|
24
|
+
a:active, a:hover, a:focus {
|
25
25
|
color: #4b001a;
|
26
26
|
}
|
27
27
|
|
@@ -89,11 +89,10 @@ header.row .pagination {
|
|
89
89
|
.summary_bar .summary {
|
90
90
|
margin-top: 12px;
|
91
91
|
background-color: #fff;
|
92
|
-
box-shadow: 0 0 5px rgba(50, 50, 50, 0.25);
|
93
92
|
border-radius: 4px;
|
93
|
+
border: 1px solid rgba(0, 0, 0, 0.1);
|
94
94
|
padding: 8px;
|
95
95
|
margin-bottom: 10px;
|
96
|
-
border-width: 0;
|
97
96
|
}
|
98
97
|
.poll-wrapper {
|
99
98
|
margin: 9px;
|
@@ -350,6 +349,7 @@ img.smallogo {
|
|
350
349
|
text-align: center;
|
351
350
|
margin-right: 20px;
|
352
351
|
border: 1px solid rgba(0, 0, 0, 0.1);
|
352
|
+
border-radius: 4px;
|
353
353
|
padding: 5px;
|
354
354
|
width: 150px;
|
355
355
|
margin-bottom: 20px;
|
@@ -405,8 +405,6 @@ span.current-interval {
|
|
405
405
|
|
406
406
|
div.interval-slider input {
|
407
407
|
width: 160px;
|
408
|
-
height: 3px;
|
409
|
-
margin-top: 5px;
|
410
408
|
border-radius: 2px;
|
411
409
|
background: currentcolor;
|
412
410
|
}
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 6.
|
4
|
+
version: 6.4.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Mike Perham
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2022-01-20 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: redis
|
@@ -67,10 +67,10 @@ files:
|
|
67
67
|
- bin/sidekiq
|
68
68
|
- bin/sidekiqload
|
69
69
|
- bin/sidekiqmon
|
70
|
-
- lib/generators/sidekiq/
|
71
|
-
- lib/generators/sidekiq/templates/
|
72
|
-
- lib/generators/sidekiq/templates/
|
73
|
-
- lib/generators/sidekiq/
|
70
|
+
- lib/generators/sidekiq/job_generator.rb
|
71
|
+
- lib/generators/sidekiq/templates/job.rb.erb
|
72
|
+
- lib/generators/sidekiq/templates/job_spec.rb.erb
|
73
|
+
- lib/generators/sidekiq/templates/job_test.rb.erb
|
74
74
|
- lib/sidekiq.rb
|
75
75
|
- lib/sidekiq/api.rb
|
76
76
|
- lib/sidekiq/cli.rb
|
@@ -85,6 +85,7 @@ files:
|
|
85
85
|
- lib/sidekiq/job.rb
|
86
86
|
- lib/sidekiq/job_logger.rb
|
87
87
|
- lib/sidekiq/job_retry.rb
|
88
|
+
- lib/sidekiq/job_util.rb
|
88
89
|
- lib/sidekiq/launcher.rb
|
89
90
|
- lib/sidekiq/logger.rb
|
90
91
|
- lib/sidekiq/manager.rb
|
@@ -1,57 +0,0 @@
|
|
1
|
-
require "rails/generators/named_base"
|
2
|
-
|
3
|
-
module Sidekiq
|
4
|
-
module Generators # :nodoc:
|
5
|
-
class WorkerGenerator < ::Rails::Generators::NamedBase # :nodoc:
|
6
|
-
desc "This generator creates a Sidekiq Worker in app/workers and a corresponding test"
|
7
|
-
|
8
|
-
check_class_collision suffix: "Worker"
|
9
|
-
|
10
|
-
def self.default_generator_root
|
11
|
-
File.dirname(__FILE__)
|
12
|
-
end
|
13
|
-
|
14
|
-
def create_worker_file
|
15
|
-
template "worker.rb.erb", File.join("app/workers", class_path, "#{file_name}_worker.rb")
|
16
|
-
end
|
17
|
-
|
18
|
-
def create_test_file
|
19
|
-
return unless test_framework
|
20
|
-
|
21
|
-
if test_framework == :rspec
|
22
|
-
create_worker_spec
|
23
|
-
else
|
24
|
-
create_worker_test
|
25
|
-
end
|
26
|
-
end
|
27
|
-
|
28
|
-
private
|
29
|
-
|
30
|
-
def create_worker_spec
|
31
|
-
template_file = File.join(
|
32
|
-
"spec/workers",
|
33
|
-
class_path,
|
34
|
-
"#{file_name}_worker_spec.rb"
|
35
|
-
)
|
36
|
-
template "worker_spec.rb.erb", template_file
|
37
|
-
end
|
38
|
-
|
39
|
-
def create_worker_test
|
40
|
-
template_file = File.join(
|
41
|
-
"test/workers",
|
42
|
-
class_path,
|
43
|
-
"#{file_name}_worker_test.rb"
|
44
|
-
)
|
45
|
-
template "worker_test.rb.erb", template_file
|
46
|
-
end
|
47
|
-
|
48
|
-
def file_name
|
49
|
-
@_file_name ||= super.sub(/_?worker\z/i, "")
|
50
|
-
end
|
51
|
-
|
52
|
-
def test_framework
|
53
|
-
::Rails.application.config.generators.options[:rails][:test_framework]
|
54
|
-
end
|
55
|
-
end
|
56
|
-
end
|
57
|
-
end
|