sidekiq 5.2.4 → 7.2.4
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Changes.md +672 -8
- data/LICENSE.txt +9 -0
- data/README.md +48 -51
- data/bin/multi_queue_bench +271 -0
- data/bin/sidekiq +22 -3
- data/bin/sidekiqload +213 -115
- data/bin/sidekiqmon +11 -0
- data/lib/generators/sidekiq/job_generator.rb +57 -0
- data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
- data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
- data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
- data/lib/sidekiq/api.rb +623 -352
- data/lib/sidekiq/capsule.rb +127 -0
- data/lib/sidekiq/cli.rb +214 -229
- data/lib/sidekiq/client.rb +127 -102
- data/lib/sidekiq/component.rb +68 -0
- data/lib/sidekiq/config.rb +287 -0
- data/lib/sidekiq/deploy.rb +62 -0
- data/lib/sidekiq/embedded.rb +61 -0
- data/lib/sidekiq/fetch.rb +49 -42
- data/lib/sidekiq/job.rb +374 -0
- data/lib/sidekiq/job_logger.rb +33 -7
- data/lib/sidekiq/job_retry.rb +157 -108
- data/lib/sidekiq/job_util.rb +107 -0
- data/lib/sidekiq/launcher.rb +206 -106
- data/lib/sidekiq/logger.rb +131 -0
- data/lib/sidekiq/manager.rb +43 -46
- data/lib/sidekiq/metrics/query.rb +156 -0
- data/lib/sidekiq/metrics/shared.rb +95 -0
- data/lib/sidekiq/metrics/tracking.rb +140 -0
- data/lib/sidekiq/middleware/chain.rb +113 -56
- data/lib/sidekiq/middleware/current_attributes.rb +95 -0
- data/lib/sidekiq/middleware/i18n.rb +7 -7
- data/lib/sidekiq/middleware/modules.rb +21 -0
- data/lib/sidekiq/monitor.rb +146 -0
- data/lib/sidekiq/paginator.rb +28 -16
- data/lib/sidekiq/processor.rb +126 -117
- data/lib/sidekiq/rails.rb +52 -38
- data/lib/sidekiq/redis_client_adapter.rb +111 -0
- data/lib/sidekiq/redis_connection.rb +41 -112
- data/lib/sidekiq/ring_buffer.rb +29 -0
- data/lib/sidekiq/scheduled.rb +112 -50
- data/lib/sidekiq/sd_notify.rb +149 -0
- data/lib/sidekiq/systemd.rb +24 -0
- data/lib/sidekiq/testing/inline.rb +6 -5
- data/lib/sidekiq/testing.rb +91 -90
- data/lib/sidekiq/transaction_aware_client.rb +51 -0
- data/lib/sidekiq/version.rb +3 -1
- data/lib/sidekiq/web/action.rb +20 -11
- data/lib/sidekiq/web/application.rb +202 -80
- data/lib/sidekiq/web/csrf_protection.rb +183 -0
- data/lib/sidekiq/web/helpers.rb +165 -114
- data/lib/sidekiq/web/router.rb +23 -19
- data/lib/sidekiq/web.rb +68 -107
- data/lib/sidekiq/worker_compatibility_alias.rb +13 -0
- data/lib/sidekiq.rb +92 -182
- data/sidekiq.gemspec +25 -16
- data/web/assets/images/apple-touch-icon.png +0 -0
- data/web/assets/javascripts/application.js +152 -61
- data/web/assets/javascripts/base-charts.js +106 -0
- data/web/assets/javascripts/chart.min.js +13 -0
- data/web/assets/javascripts/chartjs-plugin-annotation.min.js +7 -0
- data/web/assets/javascripts/dashboard-charts.js +182 -0
- data/web/assets/javascripts/dashboard.js +35 -293
- data/web/assets/javascripts/metrics.js +298 -0
- data/web/assets/stylesheets/application-dark.css +147 -0
- data/web/assets/stylesheets/application-rtl.css +10 -93
- data/web/assets/stylesheets/application.css +124 -522
- data/web/assets/stylesheets/bootstrap.css +1 -1
- data/web/locales/ar.yml +71 -65
- data/web/locales/cs.yml +62 -62
- data/web/locales/da.yml +60 -53
- data/web/locales/de.yml +65 -53
- data/web/locales/el.yml +43 -24
- data/web/locales/en.yml +86 -66
- data/web/locales/es.yml +70 -54
- data/web/locales/fa.yml +65 -65
- data/web/locales/fr.yml +83 -62
- data/web/locales/gd.yml +99 -0
- data/web/locales/he.yml +65 -64
- data/web/locales/hi.yml +59 -59
- data/web/locales/it.yml +53 -53
- data/web/locales/ja.yml +75 -64
- data/web/locales/ko.yml +52 -52
- data/web/locales/lt.yml +83 -0
- data/web/locales/nb.yml +61 -61
- data/web/locales/nl.yml +52 -52
- data/web/locales/pl.yml +45 -45
- data/web/locales/pt-br.yml +83 -55
- data/web/locales/pt.yml +51 -51
- data/web/locales/ru.yml +68 -63
- data/web/locales/sv.yml +53 -53
- data/web/locales/ta.yml +60 -60
- data/web/locales/uk.yml +62 -61
- data/web/locales/ur.yml +64 -64
- data/web/locales/vi.yml +83 -0
- data/web/locales/zh-cn.yml +43 -16
- data/web/locales/zh-tw.yml +42 -8
- data/web/views/_footer.erb +18 -3
- data/web/views/_job_info.erb +21 -4
- data/web/views/_metrics_period_select.erb +12 -0
- data/web/views/_nav.erb +1 -1
- data/web/views/_paging.erb +2 -0
- data/web/views/_poll_link.erb +3 -6
- data/web/views/_summary.erb +7 -7
- data/web/views/busy.erb +79 -29
- data/web/views/dashboard.erb +48 -18
- data/web/views/dead.erb +3 -3
- data/web/views/filtering.erb +7 -0
- data/web/views/layout.erb +3 -1
- data/web/views/metrics.erb +91 -0
- data/web/views/metrics_for_job.erb +59 -0
- data/web/views/morgue.erb +14 -15
- data/web/views/queue.erb +33 -24
- data/web/views/queues.erb +19 -5
- data/web/views/retries.erb +16 -17
- data/web/views/retry.erb +3 -3
- data/web/views/scheduled.erb +17 -15
- metadata +71 -72
- data/.github/contributing.md +0 -32
- data/.github/issue_template.md +0 -11
- data/.gitignore +0 -15
- data/.travis.yml +0 -17
- data/3.0-Upgrade.md +0 -70
- data/4.0-Upgrade.md +0 -53
- data/5.0-Upgrade.md +0 -56
- data/Appraisals +0 -9
- data/COMM-LICENSE +0 -95
- data/Ent-Changes.md +0 -225
- data/Gemfile +0 -29
- data/LICENSE +0 -9
- data/Pro-2.0-Upgrade.md +0 -138
- data/Pro-3.0-Upgrade.md +0 -44
- data/Pro-4.0-Upgrade.md +0 -35
- data/Pro-Changes.md +0 -752
- data/Rakefile +0 -9
- data/bin/sidekiqctl +0 -237
- data/code_of_conduct.md +0 -50
- data/gemfiles/rails_4.gemfile +0 -31
- data/gemfiles/rails_5.gemfile +0 -31
- data/lib/generators/sidekiq/worker_generator.rb +0 -49
- data/lib/sidekiq/core_ext.rb +0 -1
- data/lib/sidekiq/delay.rb +0 -42
- data/lib/sidekiq/exception_handler.rb +0 -29
- data/lib/sidekiq/extensions/action_mailer.rb +0 -57
- data/lib/sidekiq/extensions/active_record.rb +0 -40
- data/lib/sidekiq/extensions/class_methods.rb +0 -40
- data/lib/sidekiq/extensions/generic_proxy.rb +0 -31
- data/lib/sidekiq/logging.rb +0 -122
- data/lib/sidekiq/middleware/server/active_record.rb +0 -23
- data/lib/sidekiq/util.rb +0 -66
- data/lib/sidekiq/worker.rb +0 -215
data/Pro-2.0-Upgrade.md
DELETED
@@ -1,138 +0,0 @@
|
|
1
|
-
# Upgrading to Sidekiq Pro 2.0
|
2
|
-
|
3
|
-
Sidekiq Pro 2.0 allows nested batches for more complex job workflows
|
4
|
-
and provides a new reliable scheduler which uses Lua to guarantee
|
5
|
-
atomicity and much higher performance.
|
6
|
-
|
7
|
-
It also removes deprecated APIs, changes the batch data format and
|
8
|
-
how features are activated. Read carefully to ensure your upgrade goes
|
9
|
-
smoothly.
|
10
|
-
|
11
|
-
Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
|
12
|
-
recommended; Redis 2.4 or 2.6 will work but some functionality will not be
|
13
|
-
available.
|
14
|
-
|
15
|
-
**Note that you CANNOT go back to Pro 1.x once you've created batches
|
16
|
-
with 2.x. The new batches will not process correctly with 1.x.**
|
17
|
-
|
18
|
-
**If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
|
19
|
-
latest 1.x version and run it for a week before upgrading to 2.0.**
|
20
|
-
|
21
|
-
## Nested Batches
|
22
|
-
|
23
|
-
Batches can now be nested within the `jobs` method.
|
24
|
-
This feature enables Sidekiq Pro to handle workflow processing of any size
|
25
|
-
and complexity!
|
26
|
-
|
27
|
-
```ruby
|
28
|
-
a = Sidekiq::Batch.new
|
29
|
-
a.on(:success, SomeCallback)
|
30
|
-
a.jobs do
|
31
|
-
SomeWork.perform_async
|
32
|
-
|
33
|
-
b = Sidekiq::Batch.new
|
34
|
-
b.on(:success, MyCallback)
|
35
|
-
b.jobs do
|
36
|
-
OtherWork.perform_async
|
37
|
-
end
|
38
|
-
end
|
39
|
-
```
|
40
|
-
|
41
|
-
Parent batch callbacks are not processed until all child batch callbacks have
|
42
|
-
run successfully. In the example above, `MyCallback` will always fire
|
43
|
-
before `SomeCallback` because `b` is considered a child of `a`.
|
44
|
-
|
45
|
-
Of course you can dynamically add child batches while a batch job is executing.
|
46
|
-
|
47
|
-
```ruby
|
48
|
-
def perform(*args)
|
49
|
-
do_something(args)
|
50
|
-
|
51
|
-
if more_work?
|
52
|
-
# Sidekiq::Worker#batch returns the Batch this job is part of.
|
53
|
-
batch.jobs do
|
54
|
-
b = Sidekiq::Batch.new
|
55
|
-
b.on(:success, MyCallback)
|
56
|
-
b.jobs do
|
57
|
-
OtherWork.perform_async
|
58
|
-
end
|
59
|
-
end
|
60
|
-
end
|
61
|
-
end
|
62
|
-
```
|
63
|
-
|
64
|
-
More context: [#1485]
|
65
|
-
|
66
|
-
## Batch Data
|
67
|
-
|
68
|
-
The batch data model was overhauled. Batch data should take
|
69
|
-
significantly less space in Redis now. A simple benchmark shows 25%
|
70
|
-
savings but real world savings should be even greater.
|
71
|
-
|
72
|
-
* Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
|
73
|
-
"vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
|
74
|
-
strings, e.g. "4a3fc67d30370edf".
|
75
|
-
* In 1.x, batch data was not removed until it naturally expired in Redis.
|
76
|
-
In 2.x, all data for a batch is removed from Redis once the batch has
|
77
|
-
run any success callbacks.
|
78
|
-
* Because of the former point, batch expiry is no longer a concern.
|
79
|
-
Batch expiry is hardcoded to 30 days and is no longer user-tunable.
|
80
|
-
* Failed batch jobs no longer automatically store any associated
|
81
|
-
backtrace in Redis.
|
82
|
-
|
83
|
-
**There's no data migration required. Sidekiq Pro 2.0 transparently handles
|
84
|
-
both old and new format.**
|
85
|
-
|
86
|
-
More context: [#2130]
|
87
|
-
|
88
|
-
## Reliability
|
89
|
-
|
90
|
-
2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
|
91
|
-
scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
|
92
|
-
lots of jobs.
|
93
|
-
|
94
|
-
**Two caveats**:
|
95
|
-
- Client-side middleware is not executed
|
96
|
-
for each job when enqueued with the reliable scheduler. No Sidekiq or
|
97
|
-
Sidekiq Pro functionality is affected by this change but some 3rd party
|
98
|
-
plugins might be.
|
99
|
-
- The Lua script used inside the reliable scheduler is not safe for use
|
100
|
-
with Redis Cluster or other multi-master Redis solutions.
|
101
|
-
It is safe to use with Redis Sentinel or a typical master/slave replication setup.
|
102
|
-
|
103
|
-
**You no longer require anything to use the Reliability features.**
|
104
|
-
|
105
|
-
* Activate reliable fetch and/or the new reliable scheduler:
|
106
|
-
```ruby
|
107
|
-
Sidekiq.configure_server do |config|
|
108
|
-
config.reliable_fetch!
|
109
|
-
config.reliable_scheduler!
|
110
|
-
end
|
111
|
-
```
|
112
|
-
* Activate reliable push:
|
113
|
-
```ruby
|
114
|
-
Sidekiq::Client.reliable_push!
|
115
|
-
```
|
116
|
-
|
117
|
-
More context: [#2130]
|
118
|
-
|
119
|
-
## Other Changes
|
120
|
-
|
121
|
-
* You must require `sidekiq/pro/notifications` if you want to use the
|
122
|
-
existing notification schemes. I don't recommend using them as the
|
123
|
-
newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
|
124
|
-
* Several classes have been renamed. Generally these classes are ones
|
125
|
-
you should not need to require/use in your own code, e.g. the Batch
|
126
|
-
middleware.
|
127
|
-
* You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
|
128
|
-
Pro will set it to the jid of the callback job. [#2178]
|
129
|
-
* There's now an official API to iterate all known Batches [#2191]
|
130
|
-
```ruby
|
131
|
-
Sidekiq::BatchSet.new.each {|status| p status.bid }
|
132
|
-
```
|
133
|
-
* The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
|
134
|
-
|
135
|
-
## Thanks
|
136
|
-
|
137
|
-
Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
|
138
|
-
feedback during the release process. Thank you guys!
|
data/Pro-3.0-Upgrade.md
DELETED
@@ -1,44 +0,0 @@
|
|
1
|
-
# Welcome to Sidekiq Pro 3.0!
|
2
|
-
|
3
|
-
Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
|
4
|
-
|
5
|
-
## What's New
|
6
|
-
|
7
|
-
* **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
|
8
|
-
ago and contains **many** useful features which Sidekiq couldn't
|
9
|
-
leverage until now. **Redis 3.0.3 or greater is recommended** for large
|
10
|
-
scale use.
|
11
|
-
|
12
|
-
* Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
|
13
|
-
you will need to pull it in yourself.
|
14
|
-
|
15
|
-
* Pausing and unpausing queues is now instantaneous, no more polling!
|
16
|
-
|
17
|
-
* Reliable fetch has been re-implemented due to the fetch changes in
|
18
|
-
Sidekiq 4.0.
|
19
|
-
|
20
|
-
* Support for platforms without persistent hostnames. Since the reliable\_fetch
|
21
|
-
algorithm requires a persistent hostname, an alternative reliability
|
22
|
-
algorithm is now available for platforms like Heroku and Docker:
|
23
|
-
```ruby
|
24
|
-
Sidekiq.configure_server do |config|
|
25
|
-
config.timed_fetch!
|
26
|
-
end
|
27
|
-
```
|
28
|
-
The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
|
29
|
-
|
30
|
-
* The old 'sidekiq/notifications' features have been removed.
|
31
|
-
|
32
|
-
## Upgrade
|
33
|
-
|
34
|
-
First, make sure you are using Redis 2.8 or greater. Next:
|
35
|
-
|
36
|
-
* Upgrade to the latest Sidekiq Pro 2.x.
|
37
|
-
```ruby
|
38
|
-
gem 'sidekiq-pro', '< 3'
|
39
|
-
```
|
40
|
-
* Fix any deprecation warnings you see.
|
41
|
-
* Upgrade to 3.x.
|
42
|
-
```ruby
|
43
|
-
gem 'sidekiq-pro', '< 4'
|
44
|
-
```
|
data/Pro-4.0-Upgrade.md
DELETED
@@ -1,35 +0,0 @@
|
|
1
|
-
# Welcome to Sidekiq Pro 4.0!
|
2
|
-
|
3
|
-
Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
|
4
|
-
|
5
|
-
## What's New
|
6
|
-
|
7
|
-
* Batches now "die" if any of their jobs die. You can enumerate the set
|
8
|
-
of dead batches and their associated dead jobs. The success callback
|
9
|
-
for a dead batch will never fire unless these jobs are fixed.
|
10
|
-
```ruby
|
11
|
-
Sidekiq::Batch::DeadSet.new.each do |status|
|
12
|
-
status.dead? # => true
|
13
|
-
status.dead_jobs # => [...]
|
14
|
-
end
|
15
|
-
```
|
16
|
-
This API allows you to enumerate the batches which need help.
|
17
|
-
If you fix the issue and the dead jobs succeed, the batch will succeed.
|
18
|
-
* The older `reliable_fetch` and `timed_fetch` algorithms have been
|
19
|
-
removed. Only super\_fetch is available in 4.0.
|
20
|
-
* The statsd middleware has been tweaked to remove support for legacy,
|
21
|
-
pre-3.6.0 configuration and add relevant tags.
|
22
|
-
* Requires Sidekiq 5.0.5+.
|
23
|
-
|
24
|
-
## Upgrade
|
25
|
-
|
26
|
-
* Upgrade to the latest Sidekiq Pro 3.x.
|
27
|
-
```ruby
|
28
|
-
gem 'sidekiq-pro', '< 4'
|
29
|
-
```
|
30
|
-
* Fix any deprecation warnings you see.
|
31
|
-
* Upgrade to 4.x.
|
32
|
-
```ruby
|
33
|
-
gem 'sidekiq-pro', '< 5'
|
34
|
-
```
|
35
|
-
|