sidekiq 6.3.1 → 6.4.2
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/Changes.md +56 -0
- data/LICENSE +3 -3
- data/README.md +7 -2
- data/bin/sidekiq +3 -3
- data/bin/sidekiqload +57 -65
- data/bin/sidekiqmon +1 -1
- data/lib/generators/sidekiq/job_generator.rb +57 -0
- data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
- data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
- data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
- data/lib/sidekiq/api.rb +68 -62
- data/lib/sidekiq/cli.rb +16 -6
- data/lib/sidekiq/client.rb +44 -64
- data/lib/sidekiq/delay.rb +2 -0
- data/lib/sidekiq/extensions/generic_proxy.rb +1 -1
- data/lib/sidekiq/fetch.rb +2 -2
- data/lib/sidekiq/job_logger.rb +15 -27
- data/lib/sidekiq/job_retry.rb +28 -26
- data/lib/sidekiq/job_util.rb +67 -0
- data/lib/sidekiq/launcher.rb +37 -36
- data/lib/sidekiq/logger.rb +8 -18
- data/lib/sidekiq/manager.rb +13 -15
- data/lib/sidekiq/middleware/chain.rb +4 -4
- data/lib/sidekiq/middleware/current_attributes.rb +6 -1
- data/lib/sidekiq/middleware/i18n.rb +4 -4
- data/lib/sidekiq/monitor.rb +1 -1
- data/lib/sidekiq/paginator.rb +8 -8
- data/lib/sidekiq/processor.rb +27 -27
- data/lib/sidekiq/rails.rb +11 -4
- data/lib/sidekiq/redis_connection.rb +2 -2
- data/lib/sidekiq/scheduled.rb +13 -3
- data/lib/sidekiq/testing/inline.rb +4 -4
- data/lib/sidekiq/testing.rb +36 -35
- data/lib/sidekiq/util.rb +13 -0
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/application.rb +5 -2
- data/lib/sidekiq/web/csrf_protection.rb +2 -2
- data/lib/sidekiq/web/helpers.rb +4 -4
- data/lib/sidekiq/web.rb +3 -3
- data/lib/sidekiq/worker.rb +71 -16
- data/lib/sidekiq.rb +27 -15
- data/web/assets/javascripts/application.js +58 -26
- data/web/assets/stylesheets/application-dark.css +13 -17
- data/web/assets/stylesheets/application.css +4 -5
- data/web/views/_summary.erb +1 -1
- data/web/views/busy.erb +3 -3
- metadata +8 -7
- data/lib/generators/sidekiq/worker_generator.rb +0 -57
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 70d899b76bdd764a2ec7e5f23b6d056494b58897daaf49db672daad3e0f60237
|
4
|
+
data.tar.gz: daae0430ecaf8eb7f172c70e628581c20a17e1a7cb1a58170b9082fc3dde1ea2
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 930a0feb7ff47473eb995f8ba695468498afecb311c25296c93042ecfa00cd2bf48e19efd5ba775b4895c065705a344043e0c0e5d7b4115c2dceb1ec7cf341f9
|
7
|
+
data.tar.gz: 14ab645609e7c9f3fa954432097cd46d6e9e74e5bddc9ed1541697d13f3399add407ffb5af23c89c82651793c97e2c54b8fd4f845ca20287ce6bf033fd9caed2
|
data/Changes.md
CHANGED
@@ -2,6 +2,55 @@
|
|
2
2
|
|
3
3
|
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
|
4
4
|
|
5
|
+
6.4.2
|
6
|
+
---------
|
7
|
+
|
8
|
+
- Strict argument checking now runs after client-side middleware [#5246]
|
9
|
+
- Fix page events with live polling [#5184]
|
10
|
+
- Many under-the-hood changes to remove all usage of the term "worker"
|
11
|
+
from the Sidekiq codebase and APIs. This mostly involved RDoc and local
|
12
|
+
variable names but a few constants and public APIs were changed. The old
|
13
|
+
APIs will be removed in Sidekiq 7.0.
|
14
|
+
```
|
15
|
+
Sidekiq::DEFAULT_WORKER_OPTIONS -> Sidekiq.default_job_options
|
16
|
+
Sidekiq.default_worker_options -> Sidekiq.default_job_options
|
17
|
+
Sidekiq::Queues["default"].jobs_by_worker(HardJob) -> Sidekiq::Queues["default"].jobs_by_class(HardJob)
|
18
|
+
```
|
19
|
+
|
20
|
+
6.4.1
|
21
|
+
---------
|
22
|
+
|
23
|
+
- Fix pipeline/multi deprecations in redis-rb 4.6
|
24
|
+
- Fix sidekiq.yml YAML load errors on Ruby 3.1 [#5141]
|
25
|
+
- Sharding support for `perform_bulk` [#5129]
|
26
|
+
- Refactor job logger for SPEEEEEEED
|
27
|
+
|
28
|
+
6.4.0
|
29
|
+
---------
|
30
|
+
|
31
|
+
- **SECURITY**: Validate input to avoid possible DoS in Web UI.
|
32
|
+
- Add **strict argument checking** [#5071]
|
33
|
+
Sidekiq will now log a warning if JSON-unsafe arguments are passed to `perform_async`.
|
34
|
+
Add `Sidekiq.strict_args!(false)` to your initializer to disable this warning.
|
35
|
+
This warning will switch to an exception in Sidekiq 7.0.
|
36
|
+
- Note that Delayed Extensions will be removed in Sidekiq 7.0 [#5076]
|
37
|
+
- Add `perform_{inline,sync}` in Sidekiq::Job to run a job synchronously [#5061, hasan-ally]
|
38
|
+
```ruby
|
39
|
+
SomeJob.perform_async(args...)
|
40
|
+
SomeJob.perform_sync(args...)
|
41
|
+
SomeJob.perform_inline(args...)
|
42
|
+
```
|
43
|
+
You can also dynamically redirect a job to run synchronously:
|
44
|
+
```ruby
|
45
|
+
SomeJob.set("sync": true).perform_async(args...) # will run via perform_inline
|
46
|
+
```
|
47
|
+
- Replace Sidekiq::Worker `app/workers` generator with Sidekiq::Job `app/sidekiq` generator [#5055]
|
48
|
+
```
|
49
|
+
bin/rails generate sidekiq:job ProcessOrderJob
|
50
|
+
```
|
51
|
+
- Fix job retries losing CurrentAttributes [#5090]
|
52
|
+
- Tweak shutdown to give long-running threads time to cleanup [#5095]
|
53
|
+
|
5
54
|
6.3.1
|
6
55
|
---------
|
7
56
|
|
@@ -285,6 +334,13 @@ See the [Logging wiki page](https://github.com/mperham/sidekiq/wiki/Logging) for
|
|
285
334
|
- Integrate the StandardRB code formatter to ensure consistent code
|
286
335
|
styling. [#4114, gearnode]
|
287
336
|
|
337
|
+
5.2.10
|
338
|
+
---------
|
339
|
+
|
340
|
+
- Backport fix for CVE-2022-23837.
|
341
|
+
- Migrate to `exists?` for redis-rb.
|
342
|
+
- Lock redis-rb to <4.6 to avoid deprecations.
|
343
|
+
|
288
344
|
5.2.9
|
289
345
|
---------
|
290
346
|
|
data/LICENSE
CHANGED
@@ -4,6 +4,6 @@ Sidekiq is an Open Source project licensed under the terms of
|
|
4
4
|
the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
|
5
5
|
for license text.
|
6
6
|
|
7
|
-
Sidekiq Pro
|
8
|
-
|
9
|
-
|
7
|
+
Sidekiq Pro and Sidekiq Enterprise have a commercial-friendly license.
|
8
|
+
You can find the commercial license in COMM-LICENSE.txt.
|
9
|
+
Please see https://sidekiq.org for purchasing options.
|
data/README.md
CHANGED
@@ -36,14 +36,14 @@ Sidekiq 6.0 supports Rails 5.0+ but does not require it.
|
|
36
36
|
Installation
|
37
37
|
-----------------
|
38
38
|
|
39
|
-
|
39
|
+
bundle add sidekiq
|
40
40
|
|
41
41
|
|
42
42
|
Getting Started
|
43
43
|
-----------------
|
44
44
|
|
45
45
|
See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
|
46
|
-
You can watch [this
|
46
|
+
You can watch [this YouTube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
|
47
47
|
Sidekiq and see its features in action. Here's the Web UI:
|
48
48
|
|
49
49
|
![Web UI](https://github.com/mperham/sidekiq/raw/main/examples/web-ui.png)
|
@@ -80,6 +80,11 @@ Useful resources:
|
|
80
80
|
Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
|
81
81
|
See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
|
82
82
|
|
83
|
+
Contributing
|
84
|
+
-----------------
|
85
|
+
|
86
|
+
Please see [the contributing guidelines](https://github.com/mperham/sidekiq/blob/main/.github/contributing.md).
|
87
|
+
|
83
88
|
|
84
89
|
License
|
85
90
|
-----------------
|
data/bin/sidekiq
CHANGED
@@ -4,7 +4,7 @@
|
|
4
4
|
# RUBYOPT=-w bundle exec sidekiq
|
5
5
|
$TESTING = false
|
6
6
|
|
7
|
-
require_relative
|
7
|
+
require_relative "../lib/sidekiq/cli"
|
8
8
|
|
9
9
|
def integrate_with_systemd
|
10
10
|
return unless ENV["NOTIFY_SOCKET"]
|
@@ -32,8 +32,8 @@ begin
|
|
32
32
|
rescue => e
|
33
33
|
raise e if $DEBUG
|
34
34
|
if Sidekiq.error_handlers.length == 0
|
35
|
-
|
36
|
-
|
35
|
+
warn e.message
|
36
|
+
warn e.backtrace.join("\n")
|
37
37
|
else
|
38
38
|
cli.handle_exception e
|
39
39
|
end
|
data/bin/sidekiqload
CHANGED
@@ -4,19 +4,18 @@
|
|
4
4
|
# RUBYOPT=-w bundle exec sidekiq
|
5
5
|
$TESTING = false
|
6
6
|
|
7
|
-
#require
|
8
|
-
require
|
7
|
+
# require "ruby-prof"
|
8
|
+
require "bundler/setup"
|
9
9
|
Bundler.require(:default, :load_test)
|
10
10
|
|
11
|
-
require_relative
|
12
|
-
require_relative
|
13
|
-
|
14
|
-
include Sidekiq::Util
|
11
|
+
require_relative "../lib/sidekiq/cli"
|
12
|
+
require_relative "../lib/sidekiq/launcher"
|
15
13
|
|
16
14
|
Sidekiq.configure_server do |config|
|
17
15
|
config.options[:concurrency] = 10
|
18
|
-
config.redis = {
|
19
|
-
config.
|
16
|
+
config.redis = {db: 13, port: 6380}
|
17
|
+
# config.redis = { db: 13, port: 6380, driver: :hiredis}
|
18
|
+
config.options[:queues] << "default"
|
20
19
|
config.logger.level = Logger::ERROR
|
21
20
|
config.average_scheduled_poll_interval = 2
|
22
21
|
config.reliable! if defined?(Sidekiq::Pro)
|
@@ -29,53 +28,50 @@ class LoadWorker
|
|
29
28
|
1
|
30
29
|
end
|
31
30
|
|
32
|
-
def perform(idx, ts=nil)
|
33
|
-
puts(Time.now.to_f - ts) if ts
|
34
|
-
#raise idx.to_s if idx % 100 == 1
|
31
|
+
def perform(idx, ts = nil)
|
32
|
+
puts(Time.now.to_f - ts) if !ts.nil?
|
33
|
+
# raise idx.to_s if idx % 100 == 1
|
35
34
|
end
|
36
35
|
end
|
37
36
|
|
38
37
|
# brew tap shopify/shopify
|
39
38
|
# brew install toxiproxy
|
40
|
-
# gem install toxiproxy
|
41
39
|
# run `toxiproxy-server` in a separate terminal window.
|
42
|
-
require
|
40
|
+
require "toxiproxy"
|
43
41
|
# simulate a non-localhost network for realer-world conditions.
|
44
42
|
# adding 1ms of network latency has an ENORMOUS impact on benchmarks
|
45
43
|
Toxiproxy.populate([{
|
46
|
-
|
47
|
-
|
48
|
-
|
44
|
+
name: "redis",
|
45
|
+
listen: "127.0.0.1:6380",
|
46
|
+
upstream: "127.0.0.1:6379"
|
49
47
|
}])
|
50
48
|
|
51
49
|
self_read, self_write = IO.pipe
|
52
|
-
%w
|
53
|
-
|
54
|
-
|
55
|
-
self_write.puts(sig)
|
56
|
-
end
|
57
|
-
rescue ArgumentError
|
58
|
-
puts "Signal #{sig} not supported"
|
50
|
+
%w[INT TERM TSTP TTIN].each do |sig|
|
51
|
+
trap sig do
|
52
|
+
self_write.puts(sig)
|
59
53
|
end
|
54
|
+
rescue ArgumentError
|
55
|
+
puts "Signal #{sig} not supported"
|
60
56
|
end
|
61
57
|
|
62
|
-
Sidekiq.redis {|c| c.flushdb}
|
58
|
+
Sidekiq.redis { |c| c.flushdb }
|
63
59
|
def handle_signal(launcher, sig)
|
64
60
|
Sidekiq.logger.debug "Got #{sig} signal"
|
65
61
|
case sig
|
66
|
-
when
|
62
|
+
when "INT"
|
67
63
|
# Handle Ctrl-C in JRuby like MRI
|
68
64
|
# http://jira.codehaus.org/browse/JRUBY-4637
|
69
65
|
raise Interrupt
|
70
|
-
when
|
66
|
+
when "TERM"
|
71
67
|
# Heroku sends TERM and then waits 30 seconds for process to exit.
|
72
68
|
raise Interrupt
|
73
|
-
when
|
69
|
+
when "TSTP"
|
74
70
|
Sidekiq.logger.info "Received TSTP, no longer accepting new work"
|
75
71
|
launcher.quiet
|
76
|
-
when
|
72
|
+
when "TTIN"
|
77
73
|
Thread.list.each do |thread|
|
78
|
-
Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread[
|
74
|
+
Sidekiq.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread["label"]}"
|
79
75
|
if thread.backtrace
|
80
76
|
Sidekiq.logger.warn thread.backtrace.join("\n")
|
81
77
|
else
|
@@ -89,50 +85,46 @@ def Process.rss
|
|
89
85
|
`ps -o rss= -p #{Process.pid}`.chomp.to_i
|
90
86
|
end
|
91
87
|
|
92
|
-
iter =
|
88
|
+
iter = 50
|
93
89
|
count = 10_000
|
94
90
|
|
95
91
|
iter.times do
|
96
|
-
arr = Array.new(count)
|
97
|
-
|
98
|
-
end
|
99
|
-
count.times do |idx|
|
100
|
-
arr[idx][0] = idx
|
101
|
-
end
|
102
|
-
Sidekiq::Client.push_bulk('class' => LoadWorker, 'args' => arr)
|
92
|
+
arr = Array.new(count) { |idx| [idx] }
|
93
|
+
Sidekiq::Client.push_bulk("class" => LoadWorker, "args" => arr)
|
103
94
|
end
|
104
|
-
Sidekiq.logger.error "Created #{count*iter} jobs"
|
95
|
+
Sidekiq.logger.error "Created #{count * iter} jobs"
|
105
96
|
|
106
97
|
start = Time.now
|
107
98
|
|
108
99
|
Monitoring = Thread.new do
|
109
|
-
|
110
|
-
|
100
|
+
while true
|
101
|
+
sleep 0.2
|
102
|
+
qsize = Sidekiq.redis do |conn|
|
103
|
+
conn.llen "queue:default"
|
104
|
+
end
|
105
|
+
total = qsize
|
106
|
+
# Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
|
107
|
+
if total == 0
|
108
|
+
Sidekiq.logger.error("Done, #{iter * count} jobs in #{Time.now - start} sec")
|
109
|
+
Sidekiq.logger.error("Now here's the latency for three jobs")
|
110
|
+
|
111
|
+
LoadWorker.perform_async(1, Time.now.to_f)
|
112
|
+
LoadWorker.perform_async(2, Time.now.to_f)
|
113
|
+
LoadWorker.perform_async(3, Time.now.to_f)
|
114
|
+
|
111
115
|
sleep 0.2
|
112
|
-
|
113
|
-
conn.llen "queue:default"
|
114
|
-
end
|
115
|
-
total = qsize
|
116
|
-
#Sidekiq.logger.error("RSS: #{Process.rss} Pending: #{total}")
|
117
|
-
if total == 0
|
118
|
-
Sidekiq.logger.error("Done, #{iter * count} jobs in #{Time.now - start} sec")
|
119
|
-
Sidekiq.logger.error("Now here's the latency for three jobs")
|
120
|
-
|
121
|
-
LoadWorker.perform_async(1, Time.now.to_f)
|
122
|
-
LoadWorker.perform_async(2, Time.now.to_f)
|
123
|
-
LoadWorker.perform_async(3, Time.now.to_f)
|
124
|
-
|
125
|
-
sleep 0.2
|
126
|
-
exit(0)
|
127
|
-
end
|
116
|
+
exit(0)
|
128
117
|
end
|
129
118
|
end
|
130
119
|
end
|
131
120
|
|
132
121
|
begin
|
133
|
-
#RubyProf::exclude_threads = [ Monitoring ]
|
134
|
-
#RubyProf.start
|
135
|
-
|
122
|
+
# RubyProf::exclude_threads = [ Monitoring ]
|
123
|
+
# RubyProf.start
|
124
|
+
events = Sidekiq.options[:lifecycle_events][:startup]
|
125
|
+
events.each(&:call)
|
126
|
+
events.clear
|
127
|
+
|
136
128
|
Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
|
137
129
|
Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
|
138
130
|
launcher = Sidekiq::Launcher.new(Sidekiq.options)
|
@@ -144,14 +136,14 @@ begin
|
|
144
136
|
end
|
145
137
|
end
|
146
138
|
rescue SystemExit => e
|
147
|
-
#Sidekiq.logger.error("Profiling...")
|
148
|
-
#result = RubyProf.stop
|
149
|
-
#printer = RubyProf::GraphHtmlPrinter.new(result)
|
150
|
-
#printer.print(File.new("output.html", "w"), :min_percent => 1)
|
139
|
+
# Sidekiq.logger.error("Profiling...")
|
140
|
+
# result = RubyProf.stop
|
141
|
+
# printer = RubyProf::GraphHtmlPrinter.new(result)
|
142
|
+
# printer.print(File.new("output.html", "w"), :min_percent => 1)
|
151
143
|
# normal
|
152
144
|
rescue => e
|
153
145
|
raise e if $DEBUG
|
154
|
-
|
155
|
-
|
146
|
+
warn e.message
|
147
|
+
warn e.backtrace.join("\n")
|
156
148
|
exit 1
|
157
149
|
end
|
data/bin/sidekiqmon
CHANGED
@@ -0,0 +1,57 @@
|
|
1
|
+
require "rails/generators/named_base"
|
2
|
+
|
3
|
+
module Sidekiq
|
4
|
+
module Generators # :nodoc:
|
5
|
+
class JobGenerator < ::Rails::Generators::NamedBase # :nodoc:
|
6
|
+
desc "This generator creates a Sidekiq Job in app/sidekiq and a corresponding test"
|
7
|
+
|
8
|
+
check_class_collision suffix: "Job"
|
9
|
+
|
10
|
+
def self.default_generator_root
|
11
|
+
File.dirname(__FILE__)
|
12
|
+
end
|
13
|
+
|
14
|
+
def create_job_file
|
15
|
+
template "job.rb.erb", File.join("app/sidekiq", class_path, "#{file_name}_job.rb")
|
16
|
+
end
|
17
|
+
|
18
|
+
def create_test_file
|
19
|
+
return unless test_framework
|
20
|
+
|
21
|
+
if test_framework == :rspec
|
22
|
+
create_job_spec
|
23
|
+
else
|
24
|
+
create_job_test
|
25
|
+
end
|
26
|
+
end
|
27
|
+
|
28
|
+
private
|
29
|
+
|
30
|
+
def create_job_spec
|
31
|
+
template_file = File.join(
|
32
|
+
"spec/sidekiq",
|
33
|
+
class_path,
|
34
|
+
"#{file_name}_job_spec.rb"
|
35
|
+
)
|
36
|
+
template "job_spec.rb.erb", template_file
|
37
|
+
end
|
38
|
+
|
39
|
+
def create_job_test
|
40
|
+
template_file = File.join(
|
41
|
+
"test/sidekiq",
|
42
|
+
class_path,
|
43
|
+
"#{file_name}_job_test.rb"
|
44
|
+
)
|
45
|
+
template "job_test.rb.erb", template_file
|
46
|
+
end
|
47
|
+
|
48
|
+
def file_name
|
49
|
+
@_file_name ||= super.sub(/_?job\z/i, "")
|
50
|
+
end
|
51
|
+
|
52
|
+
def test_framework
|
53
|
+
::Rails.application.config.generators.options[:rails][:test_framework]
|
54
|
+
end
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
data/lib/sidekiq/api.rb
CHANGED
@@ -54,14 +54,14 @@ module Sidekiq
|
|
54
54
|
# O(1) redis calls
|
55
55
|
def fetch_stats_fast!
|
56
56
|
pipe1_res = Sidekiq.redis { |conn|
|
57
|
-
conn.pipelined do
|
58
|
-
|
59
|
-
|
60
|
-
|
61
|
-
|
62
|
-
|
63
|
-
|
64
|
-
|
57
|
+
conn.pipelined do |pipeline|
|
58
|
+
pipeline.get("stat:processed")
|
59
|
+
pipeline.get("stat:failed")
|
60
|
+
pipeline.zcard("schedule")
|
61
|
+
pipeline.zcard("retry")
|
62
|
+
pipeline.zcard("dead")
|
63
|
+
pipeline.scard("processes")
|
64
|
+
pipeline.lrange("queue:default", -1, -1)
|
65
65
|
end
|
66
66
|
}
|
67
67
|
|
@@ -101,9 +101,9 @@ module Sidekiq
|
|
101
101
|
}
|
102
102
|
|
103
103
|
pipe2_res = Sidekiq.redis { |conn|
|
104
|
-
conn.pipelined do
|
105
|
-
processes.each { |key|
|
106
|
-
queues.each { |queue|
|
104
|
+
conn.pipelined do |pipeline|
|
105
|
+
processes.each { |key| pipeline.hget(key, "busy") }
|
106
|
+
queues.each { |queue| pipeline.llen("queue:#{queue}") }
|
107
107
|
end
|
108
108
|
}
|
109
109
|
|
@@ -147,9 +147,9 @@ module Sidekiq
|
|
147
147
|
Sidekiq.redis do |conn|
|
148
148
|
queues = conn.sscan_each("queues").to_a
|
149
149
|
|
150
|
-
lengths = conn.pipelined {
|
150
|
+
lengths = conn.pipelined { |pipeline|
|
151
151
|
queues.each do |queue|
|
152
|
-
|
152
|
+
pipeline.llen("queue:#{queue}")
|
153
153
|
end
|
154
154
|
}
|
155
155
|
|
@@ -161,6 +161,8 @@ module Sidekiq
|
|
161
161
|
|
162
162
|
class History
|
163
163
|
def initialize(days_previous, start_date = nil)
|
164
|
+
# we only store five years of data in Redis
|
165
|
+
raise ArgumentError if days_previous < 1 || days_previous > (5 * 365)
|
164
166
|
@days_previous = days_previous
|
165
167
|
@start_date = start_date || Time.now.utc.to_date
|
166
168
|
end
|
@@ -285,9 +287,9 @@ module Sidekiq
|
|
285
287
|
|
286
288
|
def clear
|
287
289
|
Sidekiq.redis do |conn|
|
288
|
-
conn.multi do
|
289
|
-
|
290
|
-
|
290
|
+
conn.multi do |transaction|
|
291
|
+
transaction.unlink(@rname)
|
292
|
+
transaction.srem("queues", name)
|
291
293
|
end
|
292
294
|
end
|
293
295
|
end
|
@@ -352,27 +354,31 @@ module Sidekiq
|
|
352
354
|
def display_args
|
353
355
|
# Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
|
354
356
|
@display_args ||= case klass
|
355
|
-
|
356
|
-
|
357
|
-
|
358
|
-
|
359
|
-
|
360
|
-
|
361
|
-
|
362
|
-
|
363
|
-
|
364
|
-
|
365
|
-
|
366
|
-
|
367
|
-
|
368
|
-
|
369
|
-
|
370
|
-
|
371
|
-
|
372
|
-
|
373
|
-
|
374
|
-
|
375
|
-
|
357
|
+
when /\ASidekiq::Extensions::Delayed/
|
358
|
+
safe_load(args[0], args) do |_, _, arg, kwarg|
|
359
|
+
if !kwarg || kwarg.empty?
|
360
|
+
arg
|
361
|
+
else
|
362
|
+
[arg, kwarg]
|
363
|
+
end
|
364
|
+
end
|
365
|
+
when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
|
366
|
+
job_args = self["wrapped"] ? args[0]["arguments"] : []
|
367
|
+
if (self["wrapped"] || args[0]) == "ActionMailer::DeliveryJob"
|
368
|
+
# remove MailerClass, mailer_method and 'deliver_now'
|
369
|
+
job_args.drop(3)
|
370
|
+
elsif (self["wrapped"] || args[0]) == "ActionMailer::MailDeliveryJob"
|
371
|
+
# remove MailerClass, mailer_method and 'deliver_now'
|
372
|
+
job_args.drop(3).first["args"]
|
373
|
+
else
|
374
|
+
job_args
|
375
|
+
end
|
376
|
+
else
|
377
|
+
if self["encrypt"]
|
378
|
+
# no point in showing 150+ bytes of random garbage
|
379
|
+
args[-1] = "[encrypted data]"
|
380
|
+
end
|
381
|
+
args
|
376
382
|
end
|
377
383
|
end
|
378
384
|
|
@@ -517,9 +523,9 @@ module Sidekiq
|
|
517
523
|
|
518
524
|
def remove_job
|
519
525
|
Sidekiq.redis do |conn|
|
520
|
-
results = conn.multi {
|
521
|
-
|
522
|
-
|
526
|
+
results = conn.multi { |transaction|
|
527
|
+
transaction.zrangebyscore(parent.name, score, score)
|
528
|
+
transaction.zremrangebyscore(parent.name, score, score)
|
523
529
|
}.first
|
524
530
|
|
525
531
|
if results.size == 1
|
@@ -540,9 +546,9 @@ module Sidekiq
|
|
540
546
|
yield msg if msg
|
541
547
|
|
542
548
|
# push the rest back onto the sorted set
|
543
|
-
conn.multi do
|
549
|
+
conn.multi do |transaction|
|
544
550
|
nonmatched.each do |message|
|
545
|
-
|
551
|
+
transaction.zadd(parent.name, score.to_f.to_s, message)
|
546
552
|
end
|
547
553
|
end
|
548
554
|
end
|
@@ -729,10 +735,10 @@ module Sidekiq
|
|
729
735
|
def kill(message, opts = {})
|
730
736
|
now = Time.now.to_f
|
731
737
|
Sidekiq.redis do |conn|
|
732
|
-
conn.multi do
|
733
|
-
|
734
|
-
|
735
|
-
|
738
|
+
conn.multi do |transaction|
|
739
|
+
transaction.zadd(name, now.to_s, message)
|
740
|
+
transaction.zremrangebyscore(name, "-inf", now - self.class.timeout)
|
741
|
+
transaction.zremrangebyrank(name, 0, - self.class.max_jobs)
|
736
742
|
end
|
737
743
|
end
|
738
744
|
|
@@ -780,9 +786,9 @@ module Sidekiq
|
|
780
786
|
count = 0
|
781
787
|
Sidekiq.redis do |conn|
|
782
788
|
procs = conn.sscan_each("processes").to_a.sort
|
783
|
-
heartbeats = conn.pipelined {
|
789
|
+
heartbeats = conn.pipelined { |pipeline|
|
784
790
|
procs.each do |key|
|
785
|
-
|
791
|
+
pipeline.hget(key, "info")
|
786
792
|
end
|
787
793
|
}
|
788
794
|
|
@@ -804,9 +810,9 @@ module Sidekiq
|
|
804
810
|
# We're making a tradeoff here between consuming more memory instead of
|
805
811
|
# making more roundtrips to Redis, but if you have hundreds or thousands of workers,
|
806
812
|
# you'll be happier this way
|
807
|
-
conn.pipelined do
|
813
|
+
conn.pipelined do |pipeline|
|
808
814
|
procs.each do |key|
|
809
|
-
|
815
|
+
pipeline.hmget(key, "info", "busy", "beat", "quiet", "rss", "rtt_us")
|
810
816
|
end
|
811
817
|
end
|
812
818
|
}
|
@@ -819,10 +825,10 @@ module Sidekiq
|
|
819
825
|
|
820
826
|
hash = Sidekiq.load_json(info)
|
821
827
|
yield Process.new(hash.merge("busy" => busy.to_i,
|
822
|
-
|
823
|
-
|
824
|
-
|
825
|
-
|
828
|
+
"beat" => at_s.to_f,
|
829
|
+
"quiet" => quiet,
|
830
|
+
"rss" => rss.to_i,
|
831
|
+
"rtt_us" => rtt.to_i))
|
826
832
|
end
|
827
833
|
end
|
828
834
|
|
@@ -920,9 +926,9 @@ module Sidekiq
|
|
920
926
|
def signal(sig)
|
921
927
|
key = "#{identity}-signals"
|
922
928
|
Sidekiq.redis do |c|
|
923
|
-
c.multi do
|
924
|
-
|
925
|
-
|
929
|
+
c.multi do |transaction|
|
930
|
+
transaction.lpush(key, sig)
|
931
|
+
transaction.expire(key, 60)
|
926
932
|
end
|
927
933
|
end
|
928
934
|
end
|
@@ -956,9 +962,9 @@ module Sidekiq
|
|
956
962
|
Sidekiq.redis do |conn|
|
957
963
|
procs = conn.sscan_each("processes").to_a
|
958
964
|
procs.sort.each do |key|
|
959
|
-
valid, workers = conn.pipelined {
|
960
|
-
|
961
|
-
|
965
|
+
valid, workers = conn.pipelined { |pipeline|
|
966
|
+
pipeline.exists?(key)
|
967
|
+
pipeline.hgetall("#{key}:work")
|
962
968
|
}
|
963
969
|
next unless valid
|
964
970
|
workers.each_pair do |tid, json|
|
@@ -986,9 +992,9 @@ module Sidekiq
|
|
986
992
|
if procs.empty?
|
987
993
|
0
|
988
994
|
else
|
989
|
-
conn.pipelined {
|
995
|
+
conn.pipelined { |pipeline|
|
990
996
|
procs.each do |key|
|
991
|
-
|
997
|
+
pipeline.hget(key, "busy")
|
992
998
|
end
|
993
999
|
}.sum(&:to_i)
|
994
1000
|
end
|