sidekiq 8.1.1 → 8.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/Changes.md +49 -2
- data/README.md +1 -1
- data/bin/kiq +17 -0
- data/lib/active_job/queue_adapters/sidekiq_adapter.rb +8 -8
- data/lib/sidekiq/api.rb +110 -35
- data/lib/sidekiq/capsule.rb +0 -1
- data/lib/sidekiq/cli.rb +2 -3
- data/lib/sidekiq/client.rb +8 -1
- data/lib/sidekiq/component.rb +3 -0
- data/lib/sidekiq/config.rb +4 -8
- data/lib/sidekiq/job.rb +14 -1
- data/lib/sidekiq/job_retry.rb +1 -1
- data/lib/sidekiq/launcher.rb +1 -1
- data/lib/sidekiq/manager.rb +2 -1
- data/lib/sidekiq/paginator.rb +8 -2
- data/lib/sidekiq/profiler.rb +1 -1
- data/lib/sidekiq/redis_client_adapter.rb +6 -2
- data/lib/sidekiq/scheduled.rb +2 -5
- data/lib/sidekiq/testing.rb +1 -0
- data/lib/sidekiq/tui/controls.rb +53 -0
- data/lib/sidekiq/tui/filtering.rb +53 -0
- data/lib/sidekiq/tui/tabs/base_tab.rb +204 -0
- data/lib/sidekiq/tui/tabs/busy.rb +118 -0
- data/lib/sidekiq/tui/tabs/dead.rb +19 -0
- data/lib/sidekiq/tui/tabs/home.rb +144 -0
- data/lib/sidekiq/tui/tabs/metrics.rb +131 -0
- data/lib/sidekiq/tui/tabs/queues.rb +95 -0
- data/lib/sidekiq/tui/tabs/retries.rb +19 -0
- data/lib/sidekiq/tui/tabs/scheduled.rb +19 -0
- data/lib/sidekiq/tui/tabs/set_tab.rb +96 -0
- data/lib/sidekiq/tui/tabs.rb +15 -0
- data/lib/sidekiq/tui.rb +276 -913
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/action.rb +2 -2
- data/lib/sidekiq/web/application.rb +14 -1
- data/lib/sidekiq/web/helpers.rb +34 -9
- data/lib/sidekiq/web/router.rb +2 -2
- data/lib/sidekiq.rb +1 -1
- data/sidekiq.gemspec +2 -2
- data/web/assets/stylesheets/style.css +2 -0
- data/web/locales/ar.yml +1 -1
- data/web/locales/fa.yml +1 -1
- data/web/locales/gd.yml +12 -2
- data/web/locales/he.yml +1 -1
- data/web/locales/pt-BR.yml +1 -1
- data/web/locales/ur.yml +1 -1
- data/web/locales/zh-TW.yml +1 -1
- data/web/views/_job_info.html.erb +8 -0
- data/web/views/_paging.html.erb +1 -1
- data/web/views/busy.html.erb +49 -40
- data/web/views/retries.html.erb +3 -0
- metadata +17 -4
- data/bin/tui +0 -5
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 1e36b05d6e25f174a0ff8b8a21b4c097a108e4ccf98af88a56ae767cfea1186a
|
|
4
|
+
data.tar.gz: c58b4f89a1466bac6c9de05cf5ea2aee0152ac42184946b4d1646c6938b351a8
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 3b8690078001bb93fc6e4fb8fe4d52091d89c317d8dcef56d24247882f4c85dbe986a05ae5451aaf70b9ff25c4acbc89b2d15e909763550bbcb13f36e73b2f7f
|
|
7
|
+
data.tar.gz: 3cabd5a3a2861af5faf70c5f67d8eb93c578d58cb7825dc063e350f30ef37dc3e2c51060a58dcf2fca9915df1bb7fcbfd00d81996ba651c29882c8432545b03a
|
data/Changes.md
CHANGED
|
@@ -2,16 +2,63 @@
|
|
|
2
2
|
|
|
3
3
|
[Sidekiq Changes](https://github.com/sidekiq/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/sidekiq/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/sidekiq/sidekiq/blob/main/Ent-Changes.md)
|
|
4
4
|
|
|
5
|
+
8.1.6
|
|
6
|
+
----------
|
|
7
|
+
|
|
8
|
+
- Fix reported thread/memory leak when jobs fail [#7006]
|
|
9
|
+
- Users can limit data displayed on Busy page with the `only` parameter `/busy?only=(jobs|processes)` [#6992]
|
|
10
|
+
- Replace Rack::Utils usage with standard library APIs
|
|
11
|
+
- Several minor fixes from AI scanners.
|
|
12
|
+
|
|
13
|
+
8.1.5
|
|
14
|
+
----------
|
|
15
|
+
|
|
16
|
+
- Fix sub-second precision when computing the `retry_for` deadline [#7003]
|
|
17
|
+
- Identify Sidekiq connnections in Redis with `CLIENT SETINFO` [#6986]
|
|
18
|
+
- Fix edge case where Web UI could show an empty Batch set [#6987]
|
|
19
|
+
|
|
20
|
+
8.1.4
|
|
21
|
+
----------
|
|
22
|
+
|
|
23
|
+
- The TTIN signal is undeprecated as the INFO signal is not supported on Linux
|
|
24
|
+
- Show iteration job state on Busy page [#6978]
|
|
25
|
+
|
|
26
|
+
8.1.3
|
|
27
|
+
----------
|
|
28
|
+
|
|
29
|
+
- Fix edge case leading to duplicate, concurrent execution [#6379]
|
|
30
|
+
If 2 Capsules process jobs from the same queue, long-running
|
|
31
|
+
jobs could run in parallel during process shutdown.
|
|
32
|
+
- [SECURITY] Remove as much YAML usage as possible. [#6950]
|
|
33
|
+
Localization files in `web/locales` are now manually parsed.
|
|
34
|
+
Sidekiq::CLI will now only require YAML if you use a `-C` .yml file.
|
|
35
|
+
|
|
36
|
+
8.1.2
|
|
37
|
+
----------
|
|
38
|
+
|
|
39
|
+
- Initial release for `kiq`, Sidekiq's official terminal UI:
|
|
40
|
+
```
|
|
41
|
+
bundle exec kiq
|
|
42
|
+
```
|
|
43
|
+
Use REDIS_URL or REDIS_PROVIDER to point `kiq` to Redis.
|
|
44
|
+
- Mutation during iteration in `SortedSet#each` caused it to miss half of the jobs [#6936]
|
|
45
|
+
- Fix edge case resulting in nil crash on /busy page [#6954]
|
|
46
|
+
|
|
5
47
|
8.1.1
|
|
6
48
|
----------
|
|
7
49
|
|
|
8
|
-
-
|
|
9
|
-
|
|
50
|
+
- **DEPRECATION** `require 'sidekiq/testing'` and
|
|
51
|
+
`require 'sidekiq/testing/inline'`.
|
|
52
|
+
Add new `Sidekiq.testing!(mode)` API [#6931]
|
|
53
|
+
Requiring code should not enable process-wide changes.
|
|
10
54
|
```ruby
|
|
11
55
|
# Old, implicit
|
|
12
56
|
require "sidekiq/testing"
|
|
57
|
+
require "sidekiq/testing/inline"
|
|
13
58
|
# New, more explicit
|
|
59
|
+
require "sidekiq"
|
|
14
60
|
Sidekiq.testing!(:fake)
|
|
61
|
+
Sidekiq.testing!(:inline)
|
|
15
62
|
```
|
|
16
63
|
- Fix race condition with Stop button in UI [#6935]
|
|
17
64
|
- Fix javascript error handler [#6893]
|
data/README.md
CHANGED
|
@@ -90,7 +90,7 @@ Useful resources:
|
|
|
90
90
|
* The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
|
|
91
91
|
|
|
92
92
|
Every Thursday morning is Sidekiq Office Hour: I video chat and answer questions.
|
|
93
|
-
See the [Sidekiq support page](https://sidekiq.org/support
|
|
93
|
+
See the [Sidekiq support page](https://sidekiq.org/support/) for details.
|
|
94
94
|
|
|
95
95
|
Contributing
|
|
96
96
|
-----------------
|
data/bin/kiq
ADDED
|
@@ -0,0 +1,17 @@
|
|
|
1
|
+
#!/usr/bin/env ruby
|
|
2
|
+
|
|
3
|
+
# This requires the default gemset so Sidekiq Pro
|
|
4
|
+
# and Sidekiq Enterprise can load any code extensions.
|
|
5
|
+
Bundler.require(:default, :tui)
|
|
6
|
+
|
|
7
|
+
require_relative "../lib/sidekiq/tui"
|
|
8
|
+
|
|
9
|
+
# Run any load hooks registered during Bundler.require
|
|
10
|
+
Sidekiq.loader.run_load_hooks(:tui)
|
|
11
|
+
|
|
12
|
+
tt = Sidekiq::TUI.new(Sidekiq.default_configuration)
|
|
13
|
+
|
|
14
|
+
RatatuiRuby.run do |tui|
|
|
15
|
+
tt.prepare(tui)
|
|
16
|
+
tt.run_loop
|
|
17
|
+
end
|
|
@@ -63,19 +63,19 @@ begin
|
|
|
63
63
|
def enqueue(job)
|
|
64
64
|
# NB: Active Job only serializes keys it recognizes. We
|
|
65
65
|
# cannot set arbitrary key/values here.
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
)
|
|
66
|
+
options = {wrapped: job.class, queue: job.queue_name}
|
|
67
|
+
options[:profile] = job.profile if job.respond_to?(:profile) && !job.profile.nil?
|
|
68
|
+
|
|
69
|
+
wrapper = Sidekiq::ActiveJob::Wrapper.set(options)
|
|
70
70
|
job.provider_job_id = wrapper.perform_async(job.serialize)
|
|
71
71
|
end
|
|
72
72
|
|
|
73
73
|
# @api private
|
|
74
74
|
def enqueue_at(job, timestamp)
|
|
75
|
-
job.
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
).perform_at(timestamp, job.serialize)
|
|
75
|
+
options = {wrapped: job.class, queue: job.queue_name}
|
|
76
|
+
options[:profile] = job.profile if job.respond_to?(:profile) && !job.profile.nil?
|
|
77
|
+
|
|
78
|
+
job.provider_job_id = Sidekiq::ActiveJob::Wrapper.set(options).perform_at(timestamp, job.serialize)
|
|
79
79
|
end
|
|
80
80
|
|
|
81
81
|
# @api private
|
data/lib/sidekiq/api.rb
CHANGED
|
@@ -478,6 +478,10 @@ module Sidekiq
|
|
|
478
478
|
self["jid"]
|
|
479
479
|
end
|
|
480
480
|
|
|
481
|
+
def iterable_state
|
|
482
|
+
@iterable_state ||= Sidekiq::IterableJobQuery.new(jid)[jid]
|
|
483
|
+
end
|
|
484
|
+
|
|
481
485
|
def bid
|
|
482
486
|
self["bid"]
|
|
483
487
|
end
|
|
@@ -657,38 +661,8 @@ module Sidekiq
|
|
|
657
661
|
|
|
658
662
|
private
|
|
659
663
|
|
|
660
|
-
def remove_job
|
|
661
|
-
|
|
662
|
-
results = conn.multi { |transaction|
|
|
663
|
-
transaction.zrange(parent.name, score, score, "BYSCORE")
|
|
664
|
-
transaction.zremrangebyscore(parent.name, score, score)
|
|
665
|
-
}.first
|
|
666
|
-
|
|
667
|
-
if results.size == 1
|
|
668
|
-
yield results.first
|
|
669
|
-
else
|
|
670
|
-
# multiple jobs with the same score
|
|
671
|
-
# find the one with the right JID and push it
|
|
672
|
-
matched, nonmatched = results.partition { |message|
|
|
673
|
-
if message.index(jid)
|
|
674
|
-
msg = Sidekiq.load_json(message)
|
|
675
|
-
msg["jid"] == jid
|
|
676
|
-
else
|
|
677
|
-
false
|
|
678
|
-
end
|
|
679
|
-
}
|
|
680
|
-
|
|
681
|
-
msg = matched.first
|
|
682
|
-
yield msg if msg
|
|
683
|
-
|
|
684
|
-
# push the rest back onto the sorted set
|
|
685
|
-
conn.multi do |transaction|
|
|
686
|
-
nonmatched.each do |message|
|
|
687
|
-
transaction.zadd(parent.name, score.to_f.to_s, message)
|
|
688
|
-
end
|
|
689
|
-
end
|
|
690
|
-
end
|
|
691
|
-
end
|
|
664
|
+
def remove_job(&)
|
|
665
|
+
parent.remove_job(self, &)
|
|
692
666
|
end
|
|
693
667
|
end
|
|
694
668
|
|
|
@@ -857,6 +831,46 @@ module Sidekiq
|
|
|
857
831
|
nil
|
|
858
832
|
end
|
|
859
833
|
|
|
834
|
+
def remove_job(entry)
|
|
835
|
+
score = entry.score
|
|
836
|
+
jid = entry.jid
|
|
837
|
+
Sidekiq.redis do |conn|
|
|
838
|
+
results = conn.multi { |transaction|
|
|
839
|
+
transaction.zrange(name, score, score, "BYSCORE")
|
|
840
|
+
transaction.zremrangebyscore(name, score, score)
|
|
841
|
+
}.first
|
|
842
|
+
|
|
843
|
+
if results.size == 1
|
|
844
|
+
yield results.first
|
|
845
|
+
@_size -= 1
|
|
846
|
+
else
|
|
847
|
+
# multiple jobs with the same score
|
|
848
|
+
# find the one with the right JID and push it
|
|
849
|
+
matched, nonmatched = results.partition { |message|
|
|
850
|
+
if message.index(jid)
|
|
851
|
+
msg = Sidekiq.load_json(message)
|
|
852
|
+
msg["jid"] == jid
|
|
853
|
+
else
|
|
854
|
+
false
|
|
855
|
+
end
|
|
856
|
+
}
|
|
857
|
+
|
|
858
|
+
msg = matched.first
|
|
859
|
+
if msg
|
|
860
|
+
yield msg
|
|
861
|
+
@_size -= 1
|
|
862
|
+
end
|
|
863
|
+
|
|
864
|
+
# push the rest back onto the sorted set
|
|
865
|
+
conn.multi do |transaction|
|
|
866
|
+
nonmatched.each do |message|
|
|
867
|
+
transaction.zadd(name, score.to_f.to_s, message)
|
|
868
|
+
end
|
|
869
|
+
end
|
|
870
|
+
end
|
|
871
|
+
end
|
|
872
|
+
end
|
|
873
|
+
|
|
860
874
|
# :nodoc:
|
|
861
875
|
# @api private
|
|
862
876
|
def delete_by_value(name, value)
|
|
@@ -1030,19 +1044,20 @@ module Sidekiq
|
|
|
1030
1044
|
# you'll be happier this way
|
|
1031
1045
|
conn.pipelined do |pipeline|
|
|
1032
1046
|
procs.each do |key|
|
|
1033
|
-
pipeline.hmget(key, "info", "busy", "beat", "quiet", "rss", "rtt_us")
|
|
1047
|
+
pipeline.hmget(key, "info", "concurrency", "busy", "beat", "quiet", "rss", "rtt_us")
|
|
1034
1048
|
end
|
|
1035
1049
|
end
|
|
1036
1050
|
}
|
|
1037
1051
|
|
|
1038
|
-
result.each do |info, busy, beat, quiet, rss, rtt_us|
|
|
1052
|
+
result.each do |info, concurrency, busy, beat, quiet, rss, rtt_us|
|
|
1039
1053
|
# If a process is stopped between when we query Redis for `procs` and
|
|
1040
1054
|
# when we query for `result`, we will have an item in `result` that is
|
|
1041
1055
|
# composed of `nil` values.
|
|
1042
1056
|
next if info.nil?
|
|
1043
1057
|
|
|
1044
1058
|
hash = Sidekiq.load_json(info)
|
|
1045
|
-
yield Process.new(hash.merge("
|
|
1059
|
+
yield Process.new(hash.merge("concurrency" => concurrency.to_i,
|
|
1060
|
+
"busy" => busy.to_i,
|
|
1046
1061
|
"beat" => beat.to_f,
|
|
1047
1062
|
"quiet" => quiet,
|
|
1048
1063
|
"rss" => rss.to_i,
|
|
@@ -1386,6 +1401,66 @@ module Sidekiq
|
|
|
1386
1401
|
Sidekiq.redis { |c| c.hget(key, "data") }
|
|
1387
1402
|
end
|
|
1388
1403
|
end
|
|
1404
|
+
|
|
1405
|
+
# Persisted iteration state from Redis for jobs using Sidekiq::IterableJob.
|
|
1406
|
+
class IterableJobQuery
|
|
1407
|
+
def initialize(jids)
|
|
1408
|
+
@cache = bulk_fetch(jids)
|
|
1409
|
+
end
|
|
1410
|
+
|
|
1411
|
+
def [](jid)
|
|
1412
|
+
@cache[jid]
|
|
1413
|
+
end
|
|
1414
|
+
|
|
1415
|
+
private
|
|
1416
|
+
|
|
1417
|
+
# Bulk-fetch iteration state for multiple JIDs in a single Redis pipeline.
|
|
1418
|
+
# Returns a Hash of { jid => IterableJobState } for JIDs that have iteration state.
|
|
1419
|
+
def bulk_fetch(jids)
|
|
1420
|
+
raise ArgumentError unless jids
|
|
1421
|
+
jids_to_fetch = Array(jids).compact.uniq
|
|
1422
|
+
return {} if jids_to_fetch.empty?
|
|
1423
|
+
|
|
1424
|
+
results = Sidekiq.redis do |conn|
|
|
1425
|
+
conn.pipelined do |pipe|
|
|
1426
|
+
jids_to_fetch.each { |jid| pipe.hgetall("it-#{jid}") }
|
|
1427
|
+
end
|
|
1428
|
+
end
|
|
1429
|
+
|
|
1430
|
+
# TODO Requires Ruby 4
|
|
1431
|
+
# states = ::Hash.new(capacity: jids_to_fetch.size)
|
|
1432
|
+
states = {}
|
|
1433
|
+
jids_to_fetch.each_with_index do |jid, i|
|
|
1434
|
+
raw = results[i]
|
|
1435
|
+
next if raw.nil? || raw.empty?
|
|
1436
|
+
|
|
1437
|
+
states[jid] = State.new(jid, raw)
|
|
1438
|
+
end
|
|
1439
|
+
states
|
|
1440
|
+
end
|
|
1441
|
+
|
|
1442
|
+
State = Struct.new(:jid, :raw) do
|
|
1443
|
+
def executions
|
|
1444
|
+
raw["ex"].to_i
|
|
1445
|
+
end
|
|
1446
|
+
|
|
1447
|
+
def runtime
|
|
1448
|
+
raw["rt"].to_f
|
|
1449
|
+
end
|
|
1450
|
+
|
|
1451
|
+
def cursor
|
|
1452
|
+
@cursor ||= begin
|
|
1453
|
+
Sidekiq.load_json(raw["c"])
|
|
1454
|
+
rescue JSON::ParserError
|
|
1455
|
+
@raw["c"]
|
|
1456
|
+
end
|
|
1457
|
+
end
|
|
1458
|
+
|
|
1459
|
+
def cancelled
|
|
1460
|
+
raw["cancelled"]&.to_i
|
|
1461
|
+
end
|
|
1462
|
+
end
|
|
1463
|
+
end
|
|
1389
1464
|
end
|
|
1390
1465
|
|
|
1391
1466
|
Sidekiq.loader.run_load_hooks(:api)
|
data/lib/sidekiq/capsule.rb
CHANGED
data/lib/sidekiq/cli.rb
CHANGED
|
@@ -2,7 +2,6 @@
|
|
|
2
2
|
|
|
3
3
|
$stdout.sync = true
|
|
4
4
|
|
|
5
|
-
require "yaml"
|
|
6
5
|
require "optparse"
|
|
7
6
|
require "erb"
|
|
8
7
|
require "fileutils"
|
|
@@ -201,9 +200,7 @@ module Sidekiq # :nodoc:
|
|
|
201
200
|
cli.logger.info "Received TSTP, no longer accepting new work"
|
|
202
201
|
cli.launcher.quiet
|
|
203
202
|
},
|
|
204
|
-
# deprecated, use INFO
|
|
205
203
|
"TTIN" => ->(cli) {
|
|
206
|
-
cli.logger.error { "DEPRECATED: Please use the INFO signal for backtraces, support for TTIN will be removed in Sidekiq 9.0." }
|
|
207
204
|
Thread.list.each do |thread|
|
|
208
205
|
cli.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread.name}"
|
|
209
206
|
if thread.backtrace
|
|
@@ -214,6 +211,7 @@ module Sidekiq # :nodoc:
|
|
|
214
211
|
end
|
|
215
212
|
},
|
|
216
213
|
"INFO" => ->(cli) {
|
|
214
|
+
cli.logger.error { "DEPRECATED: the INFO signal does not work on Linux, use TTIN instead." }
|
|
217
215
|
Thread.list.each do |thread|
|
|
218
216
|
cli.logger.warn "Thread TID-#{(thread.object_id ^ ::Process.pid).to_s(36)} #{thread.name}"
|
|
219
217
|
if thread.backtrace
|
|
@@ -409,6 +407,7 @@ module Sidekiq # :nodoc:
|
|
|
409
407
|
def parse_config(path)
|
|
410
408
|
erb = ERB.new(File.read(path), trim_mode: "-")
|
|
411
409
|
erb.filename = File.expand_path(path)
|
|
410
|
+
require "yaml"
|
|
412
411
|
opts = YAML.safe_load(erb.result, permitted_classes: [Symbol], aliases: true) || {}
|
|
413
412
|
|
|
414
413
|
if opts.respond_to? :deep_symbolize_keys!
|
data/lib/sidekiq/client.rb
CHANGED
|
@@ -117,6 +117,11 @@ module Sidekiq
|
|
|
117
117
|
# larger than 1000 but YMMV based on network quality, size of job args, etc.
|
|
118
118
|
# A large number of jobs can cause a bit of Redis command processing latency.
|
|
119
119
|
#
|
|
120
|
+
# Accepts an `:at` option to schedule the jobs for future execution. It
|
|
121
|
+
# accepts either a single Numeric timestamp (or seconds-from-now) applied
|
|
122
|
+
# to every job, or an Array of Numeric values with the same size as `args`
|
|
123
|
+
# to schedule each job at its corresponding time.
|
|
124
|
+
#
|
|
120
125
|
# Accepts an additional `:spread_interval` option (in seconds) to randomly spread
|
|
121
126
|
# the jobs schedule times over the specified interval.
|
|
122
127
|
#
|
|
@@ -132,12 +137,14 @@ module Sidekiq
|
|
|
132
137
|
# push_bulk('class' => MyJob, 'args' => (1..100_000).to_a, batch_size: 1_000)
|
|
133
138
|
#
|
|
134
139
|
def push_bulk(items)
|
|
135
|
-
batch_size = items.delete(:batch_size) || items.delete("batch_size") || 1_000
|
|
136
140
|
args = items["args"]
|
|
137
141
|
at = items.delete("at") || items.delete(:at)
|
|
138
142
|
raise ArgumentError, "Job 'at' must be a Numeric or an Array of Numeric timestamps" if at && (Array(at).empty? || !Array(at).all? { |entry| entry.is_a?(Numeric) })
|
|
139
143
|
raise ArgumentError, "Job 'at' Array must have same size as 'args' Array" if at.is_a?(Array) && at.size != args.size
|
|
140
144
|
|
|
145
|
+
# Use a smaller batch size by default for scheduled jobs since adding to sorted sets is more costly.
|
|
146
|
+
batch_size = items.delete(:batch_size) || items.delete("batch_size") || (at ? 100 : 1_000)
|
|
147
|
+
|
|
141
148
|
jid = items.delete("jid")
|
|
142
149
|
raise ArgumentError, "Explicitly passing 'jid' when pushing more than one job is not supported" if jid && args.size > 1
|
|
143
150
|
|
data/lib/sidekiq/component.rb
CHANGED
|
@@ -57,6 +57,9 @@ module Sidekiq
|
|
|
57
57
|
end
|
|
58
58
|
|
|
59
59
|
def tid
|
|
60
|
+
# We XOR with PID to ensure Thread IDs changes after fork.
|
|
61
|
+
# I'm unclear why we don't multiply the two values to better guarantee
|
|
62
|
+
# a unique value but it's been this way for quite a while now. #3685
|
|
60
63
|
Thread.current["sidekiq_tid"] ||= (Thread.current.object_id ^ ::Process.pid).to_s(36)
|
|
61
64
|
end
|
|
62
65
|
|
data/lib/sidekiq/config.rb
CHANGED
|
@@ -255,7 +255,7 @@ module Sidekiq
|
|
|
255
255
|
# Register a proc to handle any error which occurs within the Sidekiq process.
|
|
256
256
|
#
|
|
257
257
|
# Sidekiq.configure_server do |config|
|
|
258
|
-
# config.error_handlers << proc {|ex,ctx_hash| MyErrorService.notify(ex, ctx_hash) }
|
|
258
|
+
# config.error_handlers << proc {|ex,ctx_hash,config| MyErrorService.notify(ex, ctx_hash) }
|
|
259
259
|
# end
|
|
260
260
|
#
|
|
261
261
|
# The default error handler logs errors to @logger.
|
|
@@ -297,15 +297,11 @@ module Sidekiq
|
|
|
297
297
|
@logger = logger
|
|
298
298
|
end
|
|
299
299
|
|
|
300
|
-
private def parameter_size(handler)
|
|
301
|
-
target = handler.is_a?(Proc) ? handler : handler.method(:call)
|
|
302
|
-
target.parameters.size
|
|
303
|
-
end
|
|
304
|
-
|
|
305
300
|
# INTERNAL USE ONLY
|
|
306
301
|
def handle_exception(ex, ctx = {})
|
|
307
|
-
if @options[:error_handlers].
|
|
308
|
-
|
|
302
|
+
if @options[:error_handlers].empty?
|
|
303
|
+
logger.error { "No error handlers configured, logging exception directly" }
|
|
304
|
+
logger.error { ex }
|
|
309
305
|
end
|
|
310
306
|
@options[:error_handlers].each do |handler|
|
|
311
307
|
handler.call(ex, ctx, self)
|
data/lib/sidekiq/job.rb
CHANGED
|
@@ -196,7 +196,7 @@ module Sidekiq
|
|
|
196
196
|
|
|
197
197
|
def set(options)
|
|
198
198
|
hash = options.transform_keys(&:to_s)
|
|
199
|
-
interval = hash.delete("wait_until") ||
|
|
199
|
+
interval = hash.delete("wait_until") || hash.delete("wait")
|
|
200
200
|
@opts.merge!(hash)
|
|
201
201
|
at(interval) if interval
|
|
202
202
|
self
|
|
@@ -313,6 +313,11 @@ module Sidekiq
|
|
|
313
313
|
#
|
|
314
314
|
# +items+ must be an Array of Arrays.
|
|
315
315
|
#
|
|
316
|
+
# The +:at+ option schedules the jobs for future execution. It accepts
|
|
317
|
+
# either a single Numeric timestamp (or seconds-from-now) applied to every
|
|
318
|
+
# job, or an Array of Numeric values with the same size as +items+ to
|
|
319
|
+
# schedule each job at its corresponding time.
|
|
320
|
+
#
|
|
316
321
|
# For finer-grained control, use `Sidekiq::Client.push_bulk` directly.
|
|
317
322
|
#
|
|
318
323
|
# Example (3 Redis round trips):
|
|
@@ -325,6 +330,14 @@ module Sidekiq
|
|
|
325
330
|
#
|
|
326
331
|
# SomeJob.perform_bulk([[1], [2], [3]])
|
|
327
332
|
#
|
|
333
|
+
# Scheduling every job 60 seconds from now (single Numeric +:at+):
|
|
334
|
+
#
|
|
335
|
+
# SomeJob.perform_bulk([[1], [2], [3]], at: 60)
|
|
336
|
+
#
|
|
337
|
+
# Scheduling each job at its own time (Array +:at+):
|
|
338
|
+
#
|
|
339
|
+
# SomeJob.perform_bulk([[1], [2]], at: [Time.now.to_f + 30, Time.now.to_f + 60])
|
|
340
|
+
#
|
|
328
341
|
def perform_bulk(*args, **kwargs)
|
|
329
342
|
Setter.new(self, {}).perform_bulk(*args, **kwargs)
|
|
330
343
|
end
|
data/lib/sidekiq/job_retry.rb
CHANGED
data/lib/sidekiq/launcher.rb
CHANGED
|
@@ -176,6 +176,7 @@ module Sidekiq
|
|
|
176
176
|
transaction.sadd("processes", [key])
|
|
177
177
|
transaction.exists(key)
|
|
178
178
|
transaction.hset(key, "info", to_json,
|
|
179
|
+
"concurrency", @config.total_concurrency,
|
|
179
180
|
"busy", curstate.size,
|
|
180
181
|
"beat", Time.now.to_f,
|
|
181
182
|
"rtt_us", rtt,
|
|
@@ -257,7 +258,6 @@ module Sidekiq
|
|
|
257
258
|
"started_at" => Time.now.to_f,
|
|
258
259
|
"pid" => ::Process.pid,
|
|
259
260
|
"tag" => @config[:tag] || "",
|
|
260
|
-
"concurrency" => @config.total_concurrency,
|
|
261
261
|
"capsules" => @config.capsules.each_with_object({}) { |(name, cap), memo|
|
|
262
262
|
memo[name] = cap.to_h
|
|
263
263
|
},
|
data/lib/sidekiq/manager.rb
CHANGED
|
@@ -69,12 +69,13 @@ module Sidekiq
|
|
|
69
69
|
def processor_result(processor, reason = nil)
|
|
70
70
|
@plock.synchronize do
|
|
71
71
|
@workers.delete(processor)
|
|
72
|
-
|
|
72
|
+
if !@done && @count > @workers.size
|
|
73
73
|
p = Processor.new(@config, &method(:processor_result))
|
|
74
74
|
@workers << p
|
|
75
75
|
p.start
|
|
76
76
|
end
|
|
77
77
|
end
|
|
78
|
+
nil
|
|
78
79
|
end
|
|
79
80
|
|
|
80
81
|
def stopped?
|
data/lib/sidekiq/paginator.rb
CHANGED
|
@@ -23,7 +23,8 @@ module Sidekiq
|
|
|
23
23
|
elsif key.start_with?("queue:")
|
|
24
24
|
type = TYPE_CACHE[key] = "list"
|
|
25
25
|
else
|
|
26
|
-
type =
|
|
26
|
+
type = conn.type(key)
|
|
27
|
+
TYPE_CACHE[key] = type unless type == "none"
|
|
27
28
|
end
|
|
28
29
|
rev = opts && opts[:reverse]
|
|
29
30
|
|
|
@@ -62,7 +63,12 @@ module Sidekiq
|
|
|
62
63
|
pageidx = current_page - 1
|
|
63
64
|
starting = pageidx * page_size
|
|
64
65
|
items = items.to_a
|
|
65
|
-
|
|
66
|
+
total_size = items.size
|
|
67
|
+
if starting > total_size
|
|
68
|
+
starting = 0
|
|
69
|
+
current_page = 1
|
|
70
|
+
end
|
|
71
|
+
[current_page, total_size, items[starting, page_size]]
|
|
66
72
|
end
|
|
67
73
|
end
|
|
68
74
|
end
|
data/lib/sidekiq/profiler.rb
CHANGED
|
@@ -41,8 +41,8 @@ module Sidekiq
|
|
|
41
41
|
# this allows us to use methods like `conn.hmset(...)` instead of having to use
|
|
42
42
|
# redis-client's native `conn.call("hmset", ...)`
|
|
43
43
|
def method_missing(*args, &block)
|
|
44
|
-
warn("[sidekiq#5788] Redis has deprecated the `#{args.first}`command, called at #{caller(1..1)}") if DEPRECATED_COMMANDS.include?(args.first)
|
|
45
|
-
@client.call(*args,
|
|
44
|
+
warn("[sidekiq#5788] Redis has deprecated the `#{args.first}` command, called at #{caller(1..1)}") if DEPRECATED_COMMANDS.include?(args.first)
|
|
45
|
+
@client.call(*args, &block)
|
|
46
46
|
end
|
|
47
47
|
ruby2_keywords :method_missing if respond_to?(:ruby2_keywords, true)
|
|
48
48
|
|
|
@@ -107,6 +107,10 @@ module Sidekiq
|
|
|
107
107
|
# on by default.
|
|
108
108
|
opts[:reconnect_attempts] ||= 1
|
|
109
109
|
|
|
110
|
+
# Identify ourselves to Redis via CLIENT SETINFO so connections
|
|
111
|
+
# are distinguishable in CLIENT LIST / CLIENT INFO output.
|
|
112
|
+
opts[:driver_info] ||= "sidekiq_v#{Sidekiq::VERSION}"
|
|
113
|
+
|
|
110
114
|
opts
|
|
111
115
|
end
|
|
112
116
|
end
|
data/lib/sidekiq/scheduled.rb
CHANGED
|
@@ -117,9 +117,7 @@ module Sidekiq
|
|
|
117
117
|
private
|
|
118
118
|
|
|
119
119
|
def wait
|
|
120
|
-
@sleeper.pop(timeout: random_poll_interval)
|
|
121
|
-
rescue Timeout::Error
|
|
122
|
-
# TODO move to exception: false
|
|
120
|
+
@sleeper.pop(timeout: random_poll_interval, exception: false)
|
|
123
121
|
rescue => ex
|
|
124
122
|
# if poll_interval_average hasn't been calculated yet, we can
|
|
125
123
|
# raise an error trying to reach Redis.
|
|
@@ -225,8 +223,7 @@ module Sidekiq
|
|
|
225
223
|
total += INITIAL_WAIT unless @config[:poll_interval_average]
|
|
226
224
|
total += (5 * rand)
|
|
227
225
|
|
|
228
|
-
@sleeper.pop(timeout: total)
|
|
229
|
-
rescue Timeout::Error
|
|
226
|
+
@sleeper.pop(timeout: total, exception: false)
|
|
230
227
|
ensure
|
|
231
228
|
# periodically clean out the `processes` set in Redis which can collect
|
|
232
229
|
# references to dead processes over time. The process count affects how
|
data/lib/sidekiq/testing.rb
CHANGED
|
@@ -0,0 +1,53 @@
|
|
|
1
|
+
module Sidekiq
|
|
2
|
+
class TUI
|
|
3
|
+
module Controls
|
|
4
|
+
# Defines data for input handling and for displaying controls.
|
|
5
|
+
# :code is the key code for input handling.
|
|
6
|
+
# :display and :description are shown in the controls area, with different
|
|
7
|
+
# styling between them. If :display is omitted, :code is displayed instead.
|
|
8
|
+
# :action is a lambda to execute when the control is triggered.
|
|
9
|
+
# :refresh means the action requires immediate refreshing of data
|
|
10
|
+
#
|
|
11
|
+
# Conventions: dangerous/irreversible actions should use UPPERCASE codes.
|
|
12
|
+
# The Shift button means "I'm sure".
|
|
13
|
+
GLOBAL = [
|
|
14
|
+
{code: "?", display: "?", description: "Help", action: ->(tui, tab) { tui.show_help }},
|
|
15
|
+
{code: "left", display: "←/→", description: "Select Tab", action: ->(tui, tab) { tui.navigate(:left) }, refresh: true},
|
|
16
|
+
{code: "right", action: ->(tui, tab) { tui.navigate(:right) }, refresh: true},
|
|
17
|
+
{code: "q", description: "Quit", action: ->(tui, tab) { :quit }},
|
|
18
|
+
{code: "c", modifiers: ["ctrl"], action: ->(tui, tab) { :quit }}
|
|
19
|
+
].freeze
|
|
20
|
+
|
|
21
|
+
SHARED = {
|
|
22
|
+
pageable: [
|
|
23
|
+
{code: "h", display: "h/l", description: "Prev/Next Page",
|
|
24
|
+
action: ->(tui, tab) { tab.prev_page }, refresh: true},
|
|
25
|
+
{code: "l", action: ->(tui, tab) { tab.next_page }, refresh: true}
|
|
26
|
+
],
|
|
27
|
+
selectable: [
|
|
28
|
+
{code: "k", display: "j/k", description: "Prev/Next Row",
|
|
29
|
+
action: ->(tui, tab) { tab.navigate_row(:up) }},
|
|
30
|
+
{code: "j", action: ->(tui, tab) { tab.navigate_row(:down) }},
|
|
31
|
+
{code: "x", description: "Select", action: ->(tui, tab) { tab.toggle_select }},
|
|
32
|
+
{code: "A", modifiers: ["shift"], display: "A", description: "Select All",
|
|
33
|
+
action: ->(tui, tab) { tab.toggle_select(:all) }}
|
|
34
|
+
],
|
|
35
|
+
filterable: [
|
|
36
|
+
{code: "/", display: "/", description: "Filter", action: ->(tui, tab) { tab.start_filtering }},
|
|
37
|
+
{code: "backspace", action: ->(tui, tab) { tab.remove_last_char_from_filter }, refresh: true},
|
|
38
|
+
{code: "enter", action: ->(tui, tab) { tab.stop_filtering }, refresh: true},
|
|
39
|
+
{code: "esc", action: ->(tui, tab) { tab.stop_and_clear_filtering }, refresh: true}
|
|
40
|
+
]
|
|
41
|
+
}.freeze
|
|
42
|
+
|
|
43
|
+
# Returns an array of symbols for functionality which this tab implements
|
|
44
|
+
def features
|
|
45
|
+
[]
|
|
46
|
+
end
|
|
47
|
+
|
|
48
|
+
def controls
|
|
49
|
+
GLOBAL + SHARED.slice(*features).values.flatten
|
|
50
|
+
end
|
|
51
|
+
end
|
|
52
|
+
end
|
|
53
|
+
end
|