sidekiq 6.0.6 → 6.2.2
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/Changes.md +100 -1
- data/LICENSE +1 -1
- data/README.md +2 -6
- data/bin/sidekiq +26 -2
- data/lib/sidekiq/api.rb +101 -60
- data/lib/sidekiq/cli.rb +24 -8
- data/lib/sidekiq/client.rb +16 -15
- data/lib/sidekiq/extensions/action_mailer.rb +3 -2
- data/lib/sidekiq/extensions/active_record.rb +4 -3
- data/lib/sidekiq/extensions/class_methods.rb +5 -4
- data/lib/sidekiq/extensions/generic_proxy.rb +3 -1
- data/lib/sidekiq/fetch.rb +29 -21
- data/lib/sidekiq/job.rb +8 -0
- data/lib/sidekiq/job_logger.rb +1 -1
- data/lib/sidekiq/job_retry.rb +4 -7
- data/lib/sidekiq/launcher.rb +92 -30
- data/lib/sidekiq/logger.rb +3 -2
- data/lib/sidekiq/manager.rb +4 -4
- data/lib/sidekiq/middleware/chain.rb +6 -4
- data/lib/sidekiq/processor.rb +4 -4
- data/lib/sidekiq/rails.rb +16 -18
- data/lib/sidekiq/redis_connection.rb +18 -13
- data/lib/sidekiq/scheduled.rb +7 -1
- data/lib/sidekiq/sd_notify.rb +1 -1
- data/lib/sidekiq/systemd.rb +1 -15
- data/lib/sidekiq/testing.rb +2 -4
- data/lib/sidekiq/util.rb +28 -0
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/action.rb +2 -2
- data/lib/sidekiq/web/application.rb +15 -9
- data/lib/sidekiq/web/csrf_protection.rb +180 -0
- data/lib/sidekiq/web/helpers.rb +34 -17
- data/lib/sidekiq/web/router.rb +5 -2
- data/lib/sidekiq/web.rb +36 -72
- data/lib/sidekiq/worker.rb +2 -5
- data/lib/sidekiq.rb +5 -3
- data/sidekiq.gemspec +11 -4
- data/web/assets/images/apple-touch-icon.png +0 -0
- data/web/assets/javascripts/application.js +3 -8
- data/web/assets/stylesheets/application-dark.css +65 -40
- data/web/assets/stylesheets/application.css +35 -135
- data/web/locales/ar.yml +8 -2
- data/web/locales/en.yml +3 -0
- data/web/locales/es.yml +18 -2
- data/web/locales/fr.yml +10 -3
- data/web/locales/ja.yml +3 -0
- data/web/locales/lt.yml +1 -1
- data/web/locales/pl.yml +4 -4
- data/web/locales/ru.yml +4 -0
- data/web/locales/vi.yml +83 -0
- data/web/views/_job_info.erb +1 -1
- data/web/views/busy.erb +50 -19
- data/web/views/dashboard.erb +14 -6
- data/web/views/dead.erb +1 -1
- data/web/views/layout.erb +1 -0
- data/web/views/morgue.erb +6 -6
- data/web/views/queue.erb +1 -1
- data/web/views/queues.erb +4 -4
- data/web/views/retries.erb +7 -7
- data/web/views/retry.erb +1 -1
- data/web/views/scheduled.erb +1 -1
- metadata +19 -45
- data/.circleci/config.yml +0 -60
- data/.github/contributing.md +0 -32
- data/.github/issue_template.md +0 -11
- data/.gitignore +0 -13
- data/.standard.yml +0 -20
- data/3.0-Upgrade.md +0 -70
- data/4.0-Upgrade.md +0 -53
- data/5.0-Upgrade.md +0 -56
- data/6.0-Upgrade.md +0 -72
- data/COMM-LICENSE +0 -97
- data/Ent-2.0-Upgrade.md +0 -37
- data/Ent-Changes.md +0 -256
- data/Gemfile +0 -24
- data/Gemfile.lock +0 -208
- data/Pro-2.0-Upgrade.md +0 -138
- data/Pro-3.0-Upgrade.md +0 -44
- data/Pro-4.0-Upgrade.md +0 -35
- data/Pro-5.0-Upgrade.md +0 -25
- data/Pro-Changes.md +0 -782
- data/Rakefile +0 -10
- data/code_of_conduct.md +0 -50
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1b0da541124d097e05936cc860a93009ba170d714cbd8dbc760c1245818e8ed7
|
4
|
+
data.tar.gz: 63995ccbb57fa16e4954cda4ab9610506b0a8ff35084492b520fc18608face35
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: f3b864db74530840437c0af43475ed0e12a876d6d2cbbd808ecc72722adddbd8d5548c9b777d4e34af5deddf94871567717518f70d787ea7f6a089f2f5b2f4ed
|
7
|
+
data.tar.gz: 4569eed5baa10134e99a0b1404c213e3b35c08ec158e2fa8b006918298750919e8d34074aa10d3b4dfb621bb18cbe68be467d1e7822fb855012cff0f64dcdc7b
|
data/Changes.md
CHANGED
@@ -2,6 +2,99 @@
|
|
2
2
|
|
3
3
|
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
|
4
4
|
|
5
|
+
6.2.2
|
6
|
+
---------
|
7
|
+
|
8
|
+
- Reduce retry jitter, add jitter to `sidekiq_retry_in` values [#4957]
|
9
|
+
- Minimize scheduler load on Redis at scale [#4882]
|
10
|
+
- Improve logging of delay jobs [#4904, BuonOno]
|
11
|
+
- Minor CSS improvements for buttons and tables, design PRs always welcome!
|
12
|
+
- Tweak Web UI `Cache-Control` header [#4966]
|
13
|
+
- Rename internal API class `Sidekiq::Job` to `Sidekiq::JobRecord` [#4955]
|
14
|
+
|
15
|
+
6.2.1
|
16
|
+
---------
|
17
|
+
|
18
|
+
- Update RTT warning logic to handle transient RTT spikes [#4851]
|
19
|
+
- Fix very low priority CVE on unescaped queue name [#4852]
|
20
|
+
- Add note about sessions and Rails apps in API mode
|
21
|
+
|
22
|
+
6.2.0
|
23
|
+
---------
|
24
|
+
|
25
|
+
- Store Redis RTT and log if poor [#4824]
|
26
|
+
- Add process/thread stats to Busy page [#4806]
|
27
|
+
- Improve Web UI on mobile devices [#4840]
|
28
|
+
- **Refactor Web UI session usage** [#4804]
|
29
|
+
Numerous people have hit "Forbidden" errors and struggled with Sidekiq's
|
30
|
+
Web UI session requirement. If you have code in your initializer for
|
31
|
+
Web sessions, it's quite possible it will need to be removed. Here's
|
32
|
+
an overview:
|
33
|
+
```
|
34
|
+
Sidekiq::Web needs a valid Rack session for CSRF protection. If this is a Rails app,
|
35
|
+
make sure you mount Sidekiq::Web *inside* your routes in `config/routes.rb` so
|
36
|
+
Sidekiq can reuse the Rails session:
|
37
|
+
|
38
|
+
Rails.application.routes.draw do
|
39
|
+
mount Sidekiq::Web => "/sidekiq"
|
40
|
+
....
|
41
|
+
end
|
42
|
+
|
43
|
+
If this is a bare Rack app, use a session middleware before Sidekiq::Web:
|
44
|
+
|
45
|
+
# first, use IRB to create a shared secret key for sessions and commit it
|
46
|
+
require 'securerandom'; File.open(".session.key", "w") {|f| f.write(SecureRandom.hex(32)) }
|
47
|
+
|
48
|
+
# now, update your Rack app to include the secret with a session cookie middleware
|
49
|
+
use Rack::Session::Cookie, secret: File.read(".session.key"), same_site: true, max_age: 86400
|
50
|
+
run Sidekiq::Web
|
51
|
+
|
52
|
+
If this is a Rails app in API mode, you need to enable sessions.
|
53
|
+
|
54
|
+
https://guides.rubyonrails.org/api_app.html#using-session-middlewares
|
55
|
+
```
|
56
|
+
|
57
|
+
6.1.3
|
58
|
+
---------
|
59
|
+
|
60
|
+
- Warn if Redis is configured to evict data under memory pressure [#4752]
|
61
|
+
- Add process RSS on the Busy page [#4717]
|
62
|
+
|
63
|
+
6.1.2
|
64
|
+
---------
|
65
|
+
|
66
|
+
- Improve readability in dark mode Web UI [#4674]
|
67
|
+
- Fix Web UI crash with corrupt session [#4672]
|
68
|
+
- Allow middleware to yield arguments [#4673, @eugeneius]
|
69
|
+
- Migrate CI from CircleCI to GitHub Actions [#4677]
|
70
|
+
|
71
|
+
6.1.1
|
72
|
+
---------
|
73
|
+
|
74
|
+
- Jobs are now sorted by age in the Busy Workers table. [#4641]
|
75
|
+
- Fix "check all" JS logic in Web UI [#4619]
|
76
|
+
|
77
|
+
6.1.0
|
78
|
+
---------
|
79
|
+
|
80
|
+
- Web UI - Dark Mode fixes [#4543, natematykiewicz]
|
81
|
+
- Ensure `Rack::ContentLength` is loaded as middleware for correct Web UI responses [#4541]
|
82
|
+
- Avoid exception dumping SSL store in Redis connection logging [#4532]
|
83
|
+
- Better error messages in Sidekiq::Client [#4549]
|
84
|
+
- Remove rack-protection, reimplement CSRF protection [#4588]
|
85
|
+
- Require redis-rb 4.2 [#4591]
|
86
|
+
- Update to jquery 1.12.4 [#4593]
|
87
|
+
- Refactor internal fetch logic and API [#4602]
|
88
|
+
|
89
|
+
6.0.7
|
90
|
+
---------
|
91
|
+
|
92
|
+
- Refactor systemd integration to work better with custom binaries [#4511]
|
93
|
+
- Don't connect to Redis at process exit if not needed [#4502]
|
94
|
+
- Remove Redis connection naming [#4479]
|
95
|
+
- Fix Redis Sentinel password redaction [#4499]
|
96
|
+
- Add Vietnamese locale (vi) [#4528]
|
97
|
+
|
5
98
|
6.0.6
|
6
99
|
---------
|
7
100
|
|
@@ -115,7 +208,7 @@ assert_equal 1, Sidekiq::Extensions::DelayedMailer.jobs_for(FooMailer).size
|
|
115
208
|
|
116
209
|
This release has major breaking changes. Read and test carefully in production.
|
117
210
|
|
118
|
-
- With Rails 6.0.
|
211
|
+
- With Rails 6.0.2+, ActiveJobs can now use `sidekiq_options` directly to configure Sidekiq
|
119
212
|
features/internals like the retry subsystem. [#4213, pirj]
|
120
213
|
```ruby
|
121
214
|
class MyJob < ActiveJob::Base
|
@@ -151,6 +244,12 @@ See the [Logging wiki page](https://github.com/mperham/sidekiq/wiki/Logging) for
|
|
151
244
|
- Integrate the StandardRB code formatter to ensure consistent code
|
152
245
|
styling. [#4114, gearnode]
|
153
246
|
|
247
|
+
5.2.9
|
248
|
+
---------
|
249
|
+
|
250
|
+
- Release Rack lock due to a cascade of CVEs. [#4566]
|
251
|
+
Pro-tip: don't lock Rack.
|
252
|
+
|
154
253
|
5.2.8
|
155
254
|
---------
|
156
255
|
|
data/LICENSE
CHANGED
@@ -6,4 +6,4 @@ for license text.
|
|
6
6
|
|
7
7
|
Sidekiq Pro has a commercial-friendly license allowing private forks
|
8
8
|
and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
|
9
|
-
more detail. You can find the commercial license terms in COMM-LICENSE.
|
9
|
+
more detail. You can find the commercial license terms in COMM-LICENSE.txt.
|
data/README.md
CHANGED
@@ -2,11 +2,7 @@ Sidekiq
|
|
2
2
|
==============
|
3
3
|
|
4
4
|
[![Gem Version](https://badge.fury.io/rb/sidekiq.svg)](https://rubygems.org/gems/sidekiq)
|
5
|
-
|
6
|
-
[![Test Coverage](https://codeclimate.com/github/mperham/sidekiq/badges/coverage.svg)](https://codeclimate.com/github/mperham/sidekiq/coverage)
|
7
|
-
[![Build Status](https://circleci.com/gh/mperham/sidekiq/tree/master.svg?style=svg)](https://circleci.com/gh/mperham/sidekiq/tree/master)
|
8
|
-
[![Gitter Chat](https://badges.gitter.im/mperham/sidekiq.svg)](https://gitter.im/mperham/sidekiq)
|
9
|
-
|
5
|
+
![Build](https://github.com/mperham/sidekiq/workflows/CI/badge.svg)
|
10
6
|
|
11
7
|
Simple, efficient background processing for Ruby.
|
12
8
|
|
@@ -94,4 +90,4 @@ Please see [LICENSE](https://github.com/mperham/sidekiq/blob/master/LICENSE) for
|
|
94
90
|
Author
|
95
91
|
-----------------
|
96
92
|
|
97
|
-
Mike Perham, [@
|
93
|
+
Mike Perham, [@getajobmike](https://twitter.com/getajobmike) / [@sidekiq](https://twitter.com/sidekiq), [https://www.mikeperham.com](https://www.mikeperham.com) / [https://www.contribsys.com](https://www.contribsys.com)
|
data/bin/sidekiq
CHANGED
@@ -6,13 +6,37 @@ $TESTING = false
|
|
6
6
|
|
7
7
|
require_relative '../lib/sidekiq/cli'
|
8
8
|
|
9
|
+
def integrate_with_systemd
|
10
|
+
return unless ENV["NOTIFY_SOCKET"]
|
11
|
+
|
12
|
+
Sidekiq.configure_server do |config|
|
13
|
+
Sidekiq.logger.info "Enabling systemd notification integration"
|
14
|
+
require "sidekiq/sd_notify"
|
15
|
+
config.on(:startup) do
|
16
|
+
Sidekiq::SdNotify.ready
|
17
|
+
end
|
18
|
+
config.on(:shutdown) do
|
19
|
+
Sidekiq::SdNotify.stopping
|
20
|
+
end
|
21
|
+
Sidekiq.start_watchdog if Sidekiq::SdNotify.watchdog?
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
9
25
|
begin
|
10
26
|
cli = Sidekiq::CLI.instance
|
11
27
|
cli.parse
|
28
|
+
|
29
|
+
integrate_with_systemd
|
30
|
+
|
12
31
|
cli.run
|
13
32
|
rescue => e
|
14
33
|
raise e if $DEBUG
|
15
|
-
|
16
|
-
|
34
|
+
if Sidekiq.error_handlers.length == 0
|
35
|
+
STDERR.puts e.message
|
36
|
+
STDERR.puts e.backtrace.join("\n")
|
37
|
+
else
|
38
|
+
cli.handle_exception e
|
39
|
+
end
|
40
|
+
|
17
41
|
exit 1
|
18
42
|
end
|
data/lib/sidekiq/api.rb
CHANGED
@@ -8,7 +8,7 @@ require "base64"
|
|
8
8
|
module Sidekiq
|
9
9
|
class Stats
|
10
10
|
def initialize
|
11
|
-
|
11
|
+
fetch_stats_fast!
|
12
12
|
end
|
13
13
|
|
14
14
|
def processed
|
@@ -51,7 +51,8 @@ module Sidekiq
|
|
51
51
|
Sidekiq::Stats::Queues.new.lengths
|
52
52
|
end
|
53
53
|
|
54
|
-
|
54
|
+
# O(1) redis calls
|
55
|
+
def fetch_stats_fast!
|
55
56
|
pipe1_res = Sidekiq.redis { |conn|
|
56
57
|
conn.pipelined do
|
57
58
|
conn.get("stat:processed")
|
@@ -64,6 +65,33 @@ module Sidekiq
|
|
64
65
|
end
|
65
66
|
}
|
66
67
|
|
68
|
+
default_queue_latency = if (entry = pipe1_res[6].first)
|
69
|
+
job = begin
|
70
|
+
Sidekiq.load_json(entry)
|
71
|
+
rescue
|
72
|
+
{}
|
73
|
+
end
|
74
|
+
now = Time.now.to_f
|
75
|
+
thence = job["enqueued_at"] || now
|
76
|
+
now - thence
|
77
|
+
else
|
78
|
+
0
|
79
|
+
end
|
80
|
+
|
81
|
+
@stats = {
|
82
|
+
processed: pipe1_res[0].to_i,
|
83
|
+
failed: pipe1_res[1].to_i,
|
84
|
+
scheduled_size: pipe1_res[2],
|
85
|
+
retry_size: pipe1_res[3],
|
86
|
+
dead_size: pipe1_res[4],
|
87
|
+
processes_size: pipe1_res[5],
|
88
|
+
|
89
|
+
default_queue_latency: default_queue_latency
|
90
|
+
}
|
91
|
+
end
|
92
|
+
|
93
|
+
# O(number of processes + number of queues) redis calls
|
94
|
+
def fetch_stats_slow!
|
67
95
|
processes = Sidekiq.redis { |conn|
|
68
96
|
conn.sscan_each("processes").to_a
|
69
97
|
}
|
@@ -83,30 +111,13 @@ module Sidekiq
|
|
83
111
|
workers_size = pipe2_res[0...s].sum(&:to_i)
|
84
112
|
enqueued = pipe2_res[s..-1].sum(&:to_i)
|
85
113
|
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
rescue
|
90
|
-
{}
|
91
|
-
end
|
92
|
-
now = Time.now.to_f
|
93
|
-
thence = job["enqueued_at"] || now
|
94
|
-
now - thence
|
95
|
-
else
|
96
|
-
0
|
97
|
-
end
|
98
|
-
@stats = {
|
99
|
-
processed: pipe1_res[0].to_i,
|
100
|
-
failed: pipe1_res[1].to_i,
|
101
|
-
scheduled_size: pipe1_res[2],
|
102
|
-
retry_size: pipe1_res[3],
|
103
|
-
dead_size: pipe1_res[4],
|
104
|
-
processes_size: pipe1_res[5],
|
114
|
+
@stats[:workers_size] = workers_size
|
115
|
+
@stats[:enqueued] = enqueued
|
116
|
+
end
|
105
117
|
|
106
|
-
|
107
|
-
|
108
|
-
|
109
|
-
}
|
118
|
+
def fetch_stats!
|
119
|
+
fetch_stats_fast!
|
120
|
+
fetch_stats_slow!
|
110
121
|
end
|
111
122
|
|
112
123
|
def reset(*stats)
|
@@ -126,7 +137,8 @@ module Sidekiq
|
|
126
137
|
private
|
127
138
|
|
128
139
|
def stat(s)
|
129
|
-
@stats[s]
|
140
|
+
fetch_stats_slow! if @stats[s].nil?
|
141
|
+
@stats[s] || raise(ArgumentError, "Unknown stat #{s}")
|
130
142
|
end
|
131
143
|
|
132
144
|
class Queues
|
@@ -141,7 +153,7 @@ module Sidekiq
|
|
141
153
|
}
|
142
154
|
|
143
155
|
array_of_arrays = queues.zip(lengths).sort_by { |_, size| -size }
|
144
|
-
|
156
|
+
array_of_arrays.to_h
|
145
157
|
end
|
146
158
|
end
|
147
159
|
end
|
@@ -255,7 +267,7 @@ module Sidekiq
|
|
255
267
|
break if entries.empty?
|
256
268
|
page += 1
|
257
269
|
entries.each do |entry|
|
258
|
-
yield
|
270
|
+
yield JobRecord.new(entry, @name)
|
259
271
|
end
|
260
272
|
deleted_size = initial_size - size
|
261
273
|
end
|
@@ -265,7 +277,7 @@ module Sidekiq
|
|
265
277
|
# Find the job with the given JID within this queue.
|
266
278
|
#
|
267
279
|
# This is a slow, inefficient operation. Do not use under
|
268
|
-
# normal conditions.
|
280
|
+
# normal conditions.
|
269
281
|
def find_job(jid)
|
270
282
|
detect { |j| j.jid == jid }
|
271
283
|
end
|
@@ -286,9 +298,9 @@ module Sidekiq
|
|
286
298
|
# sorted set.
|
287
299
|
#
|
288
300
|
# The job should be considered immutable but may be
|
289
|
-
# removed from the queue via
|
301
|
+
# removed from the queue via JobRecord#delete.
|
290
302
|
#
|
291
|
-
class
|
303
|
+
class JobRecord
|
292
304
|
attr_reader :item
|
293
305
|
attr_reader :value
|
294
306
|
|
@@ -316,21 +328,23 @@ module Sidekiq
|
|
316
328
|
|
317
329
|
def display_class
|
318
330
|
# Unwrap known wrappers so they show up in a human-friendly manner in the Web UI
|
319
|
-
@klass ||=
|
320
|
-
|
321
|
-
|
322
|
-
|
323
|
-
|
324
|
-
|
325
|
-
|
326
|
-
|
327
|
-
|
328
|
-
|
329
|
-
|
330
|
-
|
331
|
-
|
332
|
-
|
333
|
-
|
331
|
+
@klass ||= self["display_class"] || begin
|
332
|
+
case klass
|
333
|
+
when /\ASidekiq::Extensions::Delayed/
|
334
|
+
safe_load(args[0], klass) do |target, method, _|
|
335
|
+
"#{target}.#{method}"
|
336
|
+
end
|
337
|
+
when "ActiveJob::QueueAdapters::SidekiqAdapter::JobWrapper"
|
338
|
+
job_class = @item["wrapped"] || args[0]
|
339
|
+
if job_class == "ActionMailer::DeliveryJob" || job_class == "ActionMailer::MailDeliveryJob"
|
340
|
+
# MailerClass#mailer_method
|
341
|
+
args[0]["arguments"][0..1].join("#")
|
342
|
+
else
|
343
|
+
job_class
|
344
|
+
end
|
345
|
+
else
|
346
|
+
klass
|
347
|
+
end
|
334
348
|
end
|
335
349
|
end
|
336
350
|
|
@@ -443,7 +457,7 @@ module Sidekiq
|
|
443
457
|
end
|
444
458
|
end
|
445
459
|
|
446
|
-
class SortedEntry <
|
460
|
+
class SortedEntry < JobRecord
|
447
461
|
attr_reader :score
|
448
462
|
attr_reader :parent
|
449
463
|
|
@@ -791,19 +805,23 @@ module Sidekiq
|
|
791
805
|
# you'll be happier this way
|
792
806
|
conn.pipelined do
|
793
807
|
procs.each do |key|
|
794
|
-
conn.hmget(key, "info", "busy", "beat", "quiet")
|
808
|
+
conn.hmget(key, "info", "busy", "beat", "quiet", "rss", "rtt_us")
|
795
809
|
end
|
796
810
|
end
|
797
811
|
}
|
798
812
|
|
799
|
-
result.each do |info, busy, at_s, quiet|
|
813
|
+
result.each do |info, busy, at_s, quiet, rss, rtt|
|
800
814
|
# If a process is stopped between when we query Redis for `procs` and
|
801
815
|
# when we query for `result`, we will have an item in `result` that is
|
802
816
|
# composed of `nil` values.
|
803
817
|
next if info.nil?
|
804
818
|
|
805
819
|
hash = Sidekiq.load_json(info)
|
806
|
-
yield Process.new(hash.merge("busy" => busy.to_i,
|
820
|
+
yield Process.new(hash.merge("busy" => busy.to_i,
|
821
|
+
"beat" => at_s.to_f,
|
822
|
+
"quiet" => quiet,
|
823
|
+
"rss" => rss.to_i,
|
824
|
+
"rtt_us" => rtt.to_i))
|
807
825
|
end
|
808
826
|
end
|
809
827
|
|
@@ -815,6 +833,18 @@ module Sidekiq
|
|
815
833
|
Sidekiq.redis { |conn| conn.scard("processes") }
|
816
834
|
end
|
817
835
|
|
836
|
+
# Total number of threads available to execute jobs.
|
837
|
+
# For Sidekiq Enterprise customers this number (in production) must be
|
838
|
+
# less than or equal to your licensed concurrency.
|
839
|
+
def total_concurrency
|
840
|
+
sum { |x| x["concurrency"].to_i }
|
841
|
+
end
|
842
|
+
|
843
|
+
def total_rss_in_kb
|
844
|
+
sum { |x| x["rss"].to_i }
|
845
|
+
end
|
846
|
+
alias_method :total_rss, :total_rss_in_kb
|
847
|
+
|
818
848
|
# Returns the identity of the current cluster leader or "" if no leader.
|
819
849
|
# This is a Sidekiq Enterprise feature, will always return "" in Sidekiq
|
820
850
|
# or Sidekiq Pro.
|
@@ -864,6 +894,10 @@ module Sidekiq
|
|
864
894
|
self["identity"]
|
865
895
|
end
|
866
896
|
|
897
|
+
def queues
|
898
|
+
self["queues"]
|
899
|
+
end
|
900
|
+
|
867
901
|
def quiet!
|
868
902
|
signal("TSTP")
|
869
903
|
end
|
@@ -894,8 +928,8 @@ module Sidekiq
|
|
894
928
|
end
|
895
929
|
|
896
930
|
##
|
897
|
-
#
|
898
|
-
#
|
931
|
+
# The WorkSet stores the work being done by this Sidekiq cluster.
|
932
|
+
# It tracks the process and thread working on each job.
|
899
933
|
#
|
900
934
|
# WARNING WARNING WARNING
|
901
935
|
#
|
@@ -903,25 +937,26 @@ module Sidekiq
|
|
903
937
|
# If you call #size => 5 and then expect #each to be
|
904
938
|
# called 5 times, you're going to have a bad time.
|
905
939
|
#
|
906
|
-
#
|
907
|
-
#
|
908
|
-
#
|
940
|
+
# works = Sidekiq::WorkSet.new
|
941
|
+
# works.size => 2
|
942
|
+
# works.each do |process_id, thread_id, work|
|
909
943
|
# # process_id is a unique identifier per Sidekiq process
|
910
944
|
# # thread_id is a unique identifier per thread
|
911
945
|
# # work is a Hash which looks like:
|
912
|
-
# # { 'queue' => name, 'run_at' => timestamp, 'payload' =>
|
946
|
+
# # { 'queue' => name, 'run_at' => timestamp, 'payload' => job_hash }
|
913
947
|
# # run_at is an epoch Integer.
|
914
948
|
# end
|
915
949
|
#
|
916
|
-
class
|
950
|
+
class WorkSet
|
917
951
|
include Enumerable
|
918
952
|
|
919
|
-
def each
|
953
|
+
def each(&block)
|
954
|
+
results = []
|
920
955
|
Sidekiq.redis do |conn|
|
921
956
|
procs = conn.sscan_each("processes").to_a
|
922
957
|
procs.sort.each do |key|
|
923
958
|
valid, workers = conn.pipelined {
|
924
|
-
conn.exists(key)
|
959
|
+
conn.exists?(key)
|
925
960
|
conn.hgetall("#{key}:workers")
|
926
961
|
}
|
927
962
|
next unless valid
|
@@ -930,10 +965,12 @@ module Sidekiq
|
|
930
965
|
p = hsh["payload"]
|
931
966
|
# avoid breaking API, this is a side effect of the JSON optimization in #4316
|
932
967
|
hsh["payload"] = Sidekiq.load_json(p) if p.is_a?(String)
|
933
|
-
|
968
|
+
results << [key, tid, hsh]
|
934
969
|
end
|
935
970
|
end
|
936
971
|
end
|
972
|
+
|
973
|
+
results.sort_by { |(_, _, hsh)| hsh["run_at"] }.each(&block)
|
937
974
|
end
|
938
975
|
|
939
976
|
# Note that #size is only as accurate as Sidekiq's heartbeat,
|
@@ -957,4 +994,8 @@ module Sidekiq
|
|
957
994
|
end
|
958
995
|
end
|
959
996
|
end
|
997
|
+
# Since "worker" is a nebulous term, we've deprecated the use of this class name.
|
998
|
+
# Is "worker" a process, a type of job, a thread? Undefined!
|
999
|
+
# WorkSet better describes the data.
|
1000
|
+
Workers = WorkSet
|
960
1001
|
end
|
data/lib/sidekiq/cli.rb
CHANGED
@@ -33,8 +33,9 @@ module Sidekiq
|
|
33
33
|
# Code within this method is not tested because it alters
|
34
34
|
# global process state irreversibly. PRs which improve the
|
35
35
|
# test coverage of Sidekiq::CLI are welcomed.
|
36
|
-
def run
|
37
|
-
|
36
|
+
def run(boot_app: true)
|
37
|
+
boot_application if boot_app
|
38
|
+
|
38
39
|
if environment == "development" && $stdout.tty? && Sidekiq.log_formatter.is_a?(Sidekiq::Logger::Formatters::Pretty)
|
39
40
|
print_banner
|
40
41
|
end
|
@@ -43,7 +44,7 @@ module Sidekiq
|
|
43
44
|
self_read, self_write = IO.pipe
|
44
45
|
sigs = %w[INT TERM TTIN TSTP]
|
45
46
|
# USR1 and USR2 don't work on the JVM
|
46
|
-
sigs << "USR2"
|
47
|
+
sigs << "USR2" if Sidekiq.pro? && !jruby?
|
47
48
|
sigs.each do |sig|
|
48
49
|
trap sig do
|
49
50
|
self_write.puts(sig)
|
@@ -54,13 +55,26 @@ module Sidekiq
|
|
54
55
|
|
55
56
|
logger.info "Running in #{RUBY_DESCRIPTION}"
|
56
57
|
logger.info Sidekiq::LICENSE
|
57
|
-
logger.info "Upgrade to Sidekiq Pro for more features and support:
|
58
|
+
logger.info "Upgrade to Sidekiq Pro for more features and support: https://sidekiq.org" unless defined?(::Sidekiq::Pro)
|
58
59
|
|
59
60
|
# touch the connection pool so it is created before we
|
60
61
|
# fire startup and start multithreading.
|
61
|
-
|
62
|
+
info = Sidekiq.redis_info
|
63
|
+
ver = info["redis_version"]
|
62
64
|
raise "You are connecting to Redis v#{ver}, Sidekiq requires Redis v4.0.0 or greater" if ver < "4"
|
63
65
|
|
66
|
+
maxmemory_policy = info["maxmemory_policy"]
|
67
|
+
if maxmemory_policy != "noeviction"
|
68
|
+
logger.warn <<~EOM
|
69
|
+
|
70
|
+
|
71
|
+
WARNING: Your Redis instance will evict Sidekiq data under heavy load.
|
72
|
+
The 'noeviction' maxmemory policy is recommended (current policy: '#{maxmemory_policy}').
|
73
|
+
See: https://github.com/mperham/sidekiq/wiki/Using-Redis#memory
|
74
|
+
|
75
|
+
EOM
|
76
|
+
end
|
77
|
+
|
64
78
|
# Since the user can pass us a connection pool explicitly in the initializer, we
|
65
79
|
# need to verify the size is large enough or else Sidekiq's performance is dramatically slowed.
|
66
80
|
cursize = Sidekiq.redis_pool.size
|
@@ -228,8 +242,7 @@ module Sidekiq
|
|
228
242
|
opts = parse_config(opts[:config_file]).merge(opts) if opts[:config_file]
|
229
243
|
|
230
244
|
# set defaults
|
231
|
-
opts[:queues] = ["default"] if opts[:queues].nil?
|
232
|
-
opts[:strict] = true if opts[:strict].nil?
|
245
|
+
opts[:queues] = ["default"] if opts[:queues].nil?
|
233
246
|
opts[:concurrency] = Integer(ENV["RAILS_MAX_THREADS"]) if opts[:concurrency].nil? && ENV["RAILS_MAX_THREADS"]
|
234
247
|
|
235
248
|
# merge with defaults
|
@@ -240,7 +253,7 @@ module Sidekiq
|
|
240
253
|
Sidekiq.options
|
241
254
|
end
|
242
255
|
|
243
|
-
def
|
256
|
+
def boot_application
|
244
257
|
ENV["RACK_ENV"] = ENV["RAILS_ENV"] = environment
|
245
258
|
|
246
259
|
if File.directory?(options[:require])
|
@@ -368,6 +381,8 @@ module Sidekiq
|
|
368
381
|
end
|
369
382
|
|
370
383
|
opts = opts.merge(opts.delete(environment.to_sym) || {})
|
384
|
+
opts.delete(:strict)
|
385
|
+
|
371
386
|
parse_queues(opts, opts.delete(:queues) || [])
|
372
387
|
|
373
388
|
opts
|
@@ -379,6 +394,7 @@ module Sidekiq
|
|
379
394
|
|
380
395
|
def parse_queue(opts, queue, weight = nil)
|
381
396
|
opts[:queues] ||= []
|
397
|
+
opts[:strict] = true if opts[:strict].nil?
|
382
398
|
raise ArgumentError, "queues: #{queue} cannot be defined twice" if opts[:queues].include?(queue)
|
383
399
|
[weight.to_i, 1].max.times { opts[:queues] << queue }
|
384
400
|
opts[:strict] = false if weight.to_i > 0
|
data/lib/sidekiq/client.rb
CHANGED
@@ -19,7 +19,7 @@ module Sidekiq
|
|
19
19
|
#
|
20
20
|
def middleware(&block)
|
21
21
|
@chain ||= Sidekiq.client_middleware
|
22
|
-
if
|
22
|
+
if block
|
23
23
|
@chain = @chain.dup
|
24
24
|
yield @chain
|
25
25
|
end
|
@@ -90,16 +90,17 @@ module Sidekiq
|
|
90
90
|
# Returns an array of the of pushed jobs' jids. The number of jobs pushed can be less
|
91
91
|
# than the number given if the middleware stopped processing for one or more jobs.
|
92
92
|
def push_bulk(items)
|
93
|
-
|
94
|
-
|
95
|
-
|
93
|
+
args = items["args"]
|
94
|
+
raise ArgumentError, "Bulk arguments must be an Array of Arrays: [[1], [2]]" unless args.is_a?(Array) && args.all?(Array)
|
95
|
+
return [] if args.empty? # no jobs to push
|
96
96
|
|
97
97
|
at = items.delete("at")
|
98
98
|
raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all?(Numeric))
|
99
|
+
raise ArgumentError, "Job 'at' Array must have same size as 'args' Array" if at.is_a?(Array) && at.size != args.size
|
99
100
|
|
100
101
|
normed = normalize_item(items)
|
101
|
-
payloads =
|
102
|
-
copy = normed.merge("args" =>
|
102
|
+
payloads = args.map.with_index { |job_args, index|
|
103
|
+
copy = normed.merge("args" => job_args, "jid" => SecureRandom.hex(12), "enqueued_at" => Time.now.to_f)
|
103
104
|
copy["at"] = (at.is_a?(Array) ? at[index] : at) if at
|
104
105
|
|
105
106
|
result = process_single(items["class"], copy)
|
@@ -218,16 +219,16 @@ module Sidekiq
|
|
218
219
|
end
|
219
220
|
end
|
220
221
|
|
222
|
+
def validate(item)
|
223
|
+
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: `#{item}`") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
|
224
|
+
raise(ArgumentError, "Job args must be an Array: `#{item}`") unless item["args"].is_a?(Array)
|
225
|
+
raise(ArgumentError, "Job class must be either a Class or String representation of the class name: `#{item}`") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
|
226
|
+
raise(ArgumentError, "Job 'at' must be a Numeric timestamp: `#{item}`") if item.key?("at") && !item["at"].is_a?(Numeric)
|
227
|
+
raise(ArgumentError, "Job tags must be an Array: `#{item}`") if item["tags"] && !item["tags"].is_a?(Array)
|
228
|
+
end
|
229
|
+
|
221
230
|
def normalize_item(item)
|
222
|
-
|
223
|
-
# TODO Remove after a while...
|
224
|
-
item.delete("at") if item.key?("at") && item["at"].nil?
|
225
|
-
|
226
|
-
raise(ArgumentError, "Job must be a Hash with 'class' and 'args' keys: { 'class' => SomeWorker, 'args' => ['bob', 1, :foo => 'bar'] }") unless item.is_a?(Hash) && item.key?("class") && item.key?("args")
|
227
|
-
raise(ArgumentError, "Job args must be an Array") unless item["args"].is_a?(Array)
|
228
|
-
raise(ArgumentError, "Job class must be either a Class or String representation of the class name") unless item["class"].is_a?(Class) || item["class"].is_a?(String)
|
229
|
-
raise(ArgumentError, "Job 'at' must be a Numeric timestamp") if item.key?("at") && !item["at"].is_a?(Numeric)
|
230
|
-
raise(ArgumentError, "Job tags must be an Array") if item["tags"] && !item["tags"].is_a?(Array)
|
231
|
+
validate(item)
|
231
232
|
# raise(ArgumentError, "Arguments must be native JSON types, see https://github.com/mperham/sidekiq/wiki/Best-Practices") unless JSON.load(JSON.dump(item['args'])) == item['args']
|
232
233
|
|
233
234
|
# merge in the default sidekiq_options for the item's class and/or wrapped element
|
@@ -5,9 +5,10 @@ require "sidekiq/extensions/generic_proxy"
|
|
5
5
|
module Sidekiq
|
6
6
|
module Extensions
|
7
7
|
##
|
8
|
-
# Adds
|
9
|
-
# delivery to Sidekiq.
|
8
|
+
# Adds +delay+, +delay_for+ and +delay_until+ methods to ActionMailer to offload arbitrary email
|
9
|
+
# delivery to Sidekiq.
|
10
10
|
#
|
11
|
+
# @example
|
11
12
|
# UserMailer.delay.send_welcome_email(new_user)
|
12
13
|
# UserMailer.delay_for(5.days).send_welcome_email(new_user)
|
13
14
|
# UserMailer.delay_until(5.days.from_now).send_welcome_email(new_user)
|