sidekiq 6.1.1 → 6.5.10
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Changes.md +250 -3
- data/LICENSE +3 -3
- data/README.md +10 -6
- data/bin/sidekiq +3 -3
- data/bin/sidekiqload +70 -66
- data/bin/sidekiqmon +1 -1
- data/lib/generators/sidekiq/job_generator.rb +57 -0
- data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
- data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
- data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
- data/lib/sidekiq/api.rb +352 -156
- data/lib/sidekiq/cli.rb +86 -41
- data/lib/sidekiq/client.rb +49 -73
- data/lib/sidekiq/{util.rb → component.rb} +12 -14
- data/lib/sidekiq/delay.rb +3 -1
- data/lib/sidekiq/extensions/action_mailer.rb +3 -2
- data/lib/sidekiq/extensions/active_record.rb +1 -1
- data/lib/sidekiq/extensions/generic_proxy.rb +4 -2
- data/lib/sidekiq/fetch.rb +31 -20
- data/lib/sidekiq/job.rb +13 -0
- data/lib/sidekiq/job_logger.rb +16 -28
- data/lib/sidekiq/job_retry.rb +79 -59
- data/lib/sidekiq/job_util.rb +71 -0
- data/lib/sidekiq/launcher.rb +126 -65
- data/lib/sidekiq/logger.rb +11 -20
- data/lib/sidekiq/manager.rb +35 -34
- data/lib/sidekiq/metrics/deploy.rb +47 -0
- data/lib/sidekiq/metrics/query.rb +153 -0
- data/lib/sidekiq/metrics/shared.rb +94 -0
- data/lib/sidekiq/metrics/tracking.rb +134 -0
- data/lib/sidekiq/middleware/chain.rb +88 -42
- data/lib/sidekiq/middleware/current_attributes.rb +63 -0
- data/lib/sidekiq/middleware/i18n.rb +6 -4
- data/lib/sidekiq/middleware/modules.rb +21 -0
- data/lib/sidekiq/monitor.rb +2 -2
- data/lib/sidekiq/paginator.rb +17 -9
- data/lib/sidekiq/processor.rb +47 -41
- data/lib/sidekiq/rails.rb +32 -4
- data/lib/sidekiq/redis_client_adapter.rb +154 -0
- data/lib/sidekiq/redis_connection.rb +84 -55
- data/lib/sidekiq/ring_buffer.rb +29 -0
- data/lib/sidekiq/scheduled.rb +96 -32
- data/lib/sidekiq/testing/inline.rb +4 -4
- data/lib/sidekiq/testing.rb +38 -39
- data/lib/sidekiq/transaction_aware_client.rb +45 -0
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/action.rb +3 -3
- data/lib/sidekiq/web/application.rb +41 -16
- data/lib/sidekiq/web/csrf_protection.rb +32 -5
- data/lib/sidekiq/web/helpers.rb +52 -30
- data/lib/sidekiq/web/router.rb +4 -1
- data/lib/sidekiq/web.rb +38 -78
- data/lib/sidekiq/worker.rb +142 -16
- data/lib/sidekiq.rb +114 -31
- data/sidekiq.gemspec +12 -4
- data/web/assets/images/apple-touch-icon.png +0 -0
- data/web/assets/javascripts/application.js +114 -60
- data/web/assets/javascripts/chart.min.js +13 -0
- data/web/assets/javascripts/chartjs-plugin-annotation.min.js +7 -0
- data/web/assets/javascripts/dashboard.js +50 -67
- data/web/assets/javascripts/graph.js +16 -0
- data/web/assets/javascripts/metrics.js +262 -0
- data/web/assets/stylesheets/application-dark.css +61 -51
- data/web/assets/stylesheets/application-rtl.css +0 -4
- data/web/assets/stylesheets/application.css +84 -243
- data/web/locales/ar.yml +8 -2
- data/web/locales/el.yml +43 -19
- data/web/locales/en.yml +11 -1
- data/web/locales/es.yml +18 -2
- data/web/locales/fr.yml +8 -1
- data/web/locales/ja.yml +10 -0
- data/web/locales/lt.yml +1 -1
- data/web/locales/pt-br.yml +27 -9
- data/web/locales/ru.yml +4 -0
- data/web/locales/zh-cn.yml +36 -11
- data/web/locales/zh-tw.yml +32 -7
- data/web/views/_footer.erb +1 -1
- data/web/views/_job_info.erb +1 -1
- data/web/views/_nav.erb +1 -1
- data/web/views/_poll_link.erb +2 -5
- data/web/views/_summary.erb +7 -7
- data/web/views/busy.erb +57 -21
- data/web/views/dashboard.erb +23 -14
- data/web/views/dead.erb +1 -1
- data/web/views/layout.erb +2 -1
- data/web/views/metrics.erb +69 -0
- data/web/views/metrics_for_job.erb +87 -0
- data/web/views/morgue.erb +6 -6
- data/web/views/queue.erb +15 -11
- data/web/views/queues.erb +4 -4
- data/web/views/retries.erb +7 -7
- data/web/views/retry.erb +1 -1
- data/web/views/scheduled.erb +1 -1
- metadata +52 -39
- data/.circleci/config.yml +0 -71
- data/.github/contributing.md +0 -32
- data/.github/issue_template.md +0 -11
- data/.gitignore +0 -13
- data/.standard.yml +0 -20
- data/3.0-Upgrade.md +0 -70
- data/4.0-Upgrade.md +0 -53
- data/5.0-Upgrade.md +0 -56
- data/6.0-Upgrade.md +0 -72
- data/COMM-LICENSE +0 -97
- data/Ent-2.0-Upgrade.md +0 -37
- data/Ent-Changes.md +0 -275
- data/Gemfile +0 -24
- data/Gemfile.lock +0 -208
- data/Pro-2.0-Upgrade.md +0 -138
- data/Pro-3.0-Upgrade.md +0 -44
- data/Pro-4.0-Upgrade.md +0 -35
- data/Pro-5.0-Upgrade.md +0 -25
- data/Pro-Changes.md +0 -795
- data/Rakefile +0 -10
- data/code_of_conduct.md +0 -50
- data/lib/generators/sidekiq/worker_generator.rb +0 -57
- data/lib/sidekiq/exception_handler.rb +0 -27
data/Pro-2.0-Upgrade.md
DELETED
@@ -1,138 +0,0 @@
|
|
1
|
-
# Upgrading to Sidekiq Pro 2.0
|
2
|
-
|
3
|
-
Sidekiq Pro 2.0 allows nested batches for more complex job workflows
|
4
|
-
and provides a new reliable scheduler which uses Lua to guarantee
|
5
|
-
atomicity and much higher performance.
|
6
|
-
|
7
|
-
It also removes deprecated APIs, changes the batch data format and
|
8
|
-
how features are activated. Read carefully to ensure your upgrade goes
|
9
|
-
smoothly.
|
10
|
-
|
11
|
-
Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
|
12
|
-
recommended; Redis 2.4 or 2.6 will work but some functionality will not be
|
13
|
-
available.
|
14
|
-
|
15
|
-
**Note that you CANNOT go back to Pro 1.x once you've created batches
|
16
|
-
with 2.x. The new batches will not process correctly with 1.x.**
|
17
|
-
|
18
|
-
**If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
|
19
|
-
latest 1.x version and run it for a week before upgrading to 2.0.**
|
20
|
-
|
21
|
-
## Nested Batches
|
22
|
-
|
23
|
-
Batches can now be nested within the `jobs` method.
|
24
|
-
This feature enables Sidekiq Pro to handle workflow processing of any size
|
25
|
-
and complexity!
|
26
|
-
|
27
|
-
```ruby
|
28
|
-
a = Sidekiq::Batch.new
|
29
|
-
a.on(:success, SomeCallback)
|
30
|
-
a.jobs do
|
31
|
-
SomeWork.perform_async
|
32
|
-
|
33
|
-
b = Sidekiq::Batch.new
|
34
|
-
b.on(:success, MyCallback)
|
35
|
-
b.jobs do
|
36
|
-
OtherWork.perform_async
|
37
|
-
end
|
38
|
-
end
|
39
|
-
```
|
40
|
-
|
41
|
-
Parent batch callbacks are not processed until all child batch callbacks have
|
42
|
-
run successfully. In the example above, `MyCallback` will always fire
|
43
|
-
before `SomeCallback` because `b` is considered a child of `a`.
|
44
|
-
|
45
|
-
Of course you can dynamically add child batches while a batch job is executing.
|
46
|
-
|
47
|
-
```ruby
|
48
|
-
def perform(*args)
|
49
|
-
do_something(args)
|
50
|
-
|
51
|
-
if more_work?
|
52
|
-
# Sidekiq::Worker#batch returns the Batch this job is part of.
|
53
|
-
batch.jobs do
|
54
|
-
b = Sidekiq::Batch.new
|
55
|
-
b.on(:success, MyCallback)
|
56
|
-
b.jobs do
|
57
|
-
OtherWork.perform_async
|
58
|
-
end
|
59
|
-
end
|
60
|
-
end
|
61
|
-
end
|
62
|
-
```
|
63
|
-
|
64
|
-
More context: [#1485]
|
65
|
-
|
66
|
-
## Batch Data
|
67
|
-
|
68
|
-
The batch data model was overhauled. Batch data should take
|
69
|
-
significantly less space in Redis now. A simple benchmark shows 25%
|
70
|
-
savings but real world savings should be even greater.
|
71
|
-
|
72
|
-
* Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
|
73
|
-
"vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
|
74
|
-
strings, e.g. "4a3fc67d30370edf".
|
75
|
-
* In 1.x, batch data was not removed until it naturally expired in Redis.
|
76
|
-
In 2.x, all data for a batch is removed from Redis once the batch has
|
77
|
-
run any success callbacks.
|
78
|
-
* Because of the former point, batch expiry is no longer a concern.
|
79
|
-
Batch expiry is hardcoded to 30 days and is no longer user-tunable.
|
80
|
-
* Failed batch jobs no longer automatically store any associated
|
81
|
-
backtrace in Redis.
|
82
|
-
|
83
|
-
**There's no data migration required. Sidekiq Pro 2.0 transparently handles
|
84
|
-
both old and new format.**
|
85
|
-
|
86
|
-
More context: [#2130]
|
87
|
-
|
88
|
-
## Reliability
|
89
|
-
|
90
|
-
2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
|
91
|
-
scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
|
92
|
-
lots of jobs.
|
93
|
-
|
94
|
-
**Two caveats**:
|
95
|
-
- Client-side middleware is not executed
|
96
|
-
for each job when enqueued with the reliable scheduler. No Sidekiq or
|
97
|
-
Sidekiq Pro functionality is affected by this change but some 3rd party
|
98
|
-
plugins might be.
|
99
|
-
- The Lua script used inside the reliable scheduler is not safe for use
|
100
|
-
with Redis Cluster or other multi-master Redis solutions.
|
101
|
-
It is safe to use with Redis Sentinel or a typical master/slave replication setup.
|
102
|
-
|
103
|
-
**You no longer require anything to use the Reliability features.**
|
104
|
-
|
105
|
-
* Activate reliable fetch and/or the new reliable scheduler:
|
106
|
-
```ruby
|
107
|
-
Sidekiq.configure_server do |config|
|
108
|
-
config.reliable_fetch!
|
109
|
-
config.reliable_scheduler!
|
110
|
-
end
|
111
|
-
```
|
112
|
-
* Activate reliable push:
|
113
|
-
```ruby
|
114
|
-
Sidekiq::Client.reliable_push!
|
115
|
-
```
|
116
|
-
|
117
|
-
More context: [#2130]
|
118
|
-
|
119
|
-
## Other Changes
|
120
|
-
|
121
|
-
* You must require `sidekiq/pro/notifications` if you want to use the
|
122
|
-
existing notification schemes. I don't recommend using them as the
|
123
|
-
newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
|
124
|
-
* Several classes have been renamed. Generally these classes are ones
|
125
|
-
you should not need to require/use in your own code, e.g. the Batch
|
126
|
-
middleware.
|
127
|
-
* You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
|
128
|
-
Pro will set it to the jid of the callback job. [#2178]
|
129
|
-
* There's now an official API to iterate all known Batches [#2191]
|
130
|
-
```ruby
|
131
|
-
Sidekiq::BatchSet.new.each {|status| p status.bid }
|
132
|
-
```
|
133
|
-
* The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
|
134
|
-
|
135
|
-
## Thanks
|
136
|
-
|
137
|
-
Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
|
138
|
-
feedback during the release process. Thank you guys!
|
data/Pro-3.0-Upgrade.md
DELETED
@@ -1,44 +0,0 @@
|
|
1
|
-
# Welcome to Sidekiq Pro 3.0!
|
2
|
-
|
3
|
-
Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
|
4
|
-
|
5
|
-
## What's New
|
6
|
-
|
7
|
-
* **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
|
8
|
-
ago and contains **many** useful features which Sidekiq couldn't
|
9
|
-
leverage until now. **Redis 3.0.3 or greater is recommended** for large
|
10
|
-
scale use.
|
11
|
-
|
12
|
-
* Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
|
13
|
-
you will need to pull it in yourself.
|
14
|
-
|
15
|
-
* Pausing and unpausing queues is now instantaneous, no more polling!
|
16
|
-
|
17
|
-
* Reliable fetch has been re-implemented due to the fetch changes in
|
18
|
-
Sidekiq 4.0.
|
19
|
-
|
20
|
-
* Support for platforms without persistent hostnames. Since the reliable\_fetch
|
21
|
-
algorithm requires a persistent hostname, an alternative reliability
|
22
|
-
algorithm is now available for platforms like Heroku and Docker:
|
23
|
-
```ruby
|
24
|
-
Sidekiq.configure_server do |config|
|
25
|
-
config.timed_fetch!
|
26
|
-
end
|
27
|
-
```
|
28
|
-
The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
|
29
|
-
|
30
|
-
* The old 'sidekiq/notifications' features have been removed.
|
31
|
-
|
32
|
-
## Upgrade
|
33
|
-
|
34
|
-
First, make sure you are using Redis 2.8 or greater. Next:
|
35
|
-
|
36
|
-
* Upgrade to the latest Sidekiq Pro 2.x.
|
37
|
-
```ruby
|
38
|
-
gem 'sidekiq-pro', '< 3'
|
39
|
-
```
|
40
|
-
* Fix any deprecation warnings you see.
|
41
|
-
* Upgrade to 3.x.
|
42
|
-
```ruby
|
43
|
-
gem 'sidekiq-pro', '< 4'
|
44
|
-
```
|
data/Pro-4.0-Upgrade.md
DELETED
@@ -1,35 +0,0 @@
|
|
1
|
-
# Welcome to Sidekiq Pro 4.0!
|
2
|
-
|
3
|
-
Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
|
4
|
-
|
5
|
-
## What's New
|
6
|
-
|
7
|
-
* Batches now "die" if any of their jobs die. You can enumerate the set
|
8
|
-
of dead batches and their associated dead jobs. The success callback
|
9
|
-
for a dead batch will never fire unless these jobs are fixed.
|
10
|
-
```ruby
|
11
|
-
Sidekiq::Batch::DeadSet.new.each do |status|
|
12
|
-
status.dead? # => true
|
13
|
-
status.dead_jobs # => [...]
|
14
|
-
end
|
15
|
-
```
|
16
|
-
This API allows you to enumerate the batches which need help.
|
17
|
-
If you fix the issue and the dead jobs succeed, the batch will succeed.
|
18
|
-
* The older `reliable_fetch` and `timed_fetch` algorithms have been
|
19
|
-
removed. Only super\_fetch is available in 4.0.
|
20
|
-
* The statsd middleware has been tweaked to remove support for legacy,
|
21
|
-
pre-3.6.0 configuration and add relevant tags.
|
22
|
-
* Requires Sidekiq 5.0.5+.
|
23
|
-
|
24
|
-
## Upgrade
|
25
|
-
|
26
|
-
* Upgrade to the latest Sidekiq Pro 3.x.
|
27
|
-
```ruby
|
28
|
-
gem 'sidekiq-pro', '< 4'
|
29
|
-
```
|
30
|
-
* Fix any deprecation warnings you see.
|
31
|
-
* Upgrade to 4.x.
|
32
|
-
```ruby
|
33
|
-
gem 'sidekiq-pro', '< 5'
|
34
|
-
```
|
35
|
-
|
data/Pro-5.0-Upgrade.md
DELETED
@@ -1,25 +0,0 @@
|
|
1
|
-
# Welcome to Sidekiq Pro 5.0!
|
2
|
-
|
3
|
-
Sidekiq Pro 5.0 is mainly a cleanup release for Sidekiq 6.0. The
|
4
|
-
migration should be as close to trivial as a major version bump can be.
|
5
|
-
Note that Sidekiq 6.0 does have major breaking changes.
|
6
|
-
|
7
|
-
## What's New
|
8
|
-
|
9
|
-
* New localizations for the Sidekiq Pro Web UI: ES, ZH, PT, JA, RU
|
10
|
-
* Removed deprecated APIs and warnings.
|
11
|
-
* Various changes for Sidekiq 6.0
|
12
|
-
* Requires Ruby 2.5+ and Redis 4.0+
|
13
|
-
* Requires Sidekiq 6.0+.
|
14
|
-
|
15
|
-
## Upgrade
|
16
|
-
|
17
|
-
* Upgrade to the latest Sidekiq Pro 4.x.
|
18
|
-
```ruby
|
19
|
-
gem 'sidekiq-pro', '< 5'
|
20
|
-
```
|
21
|
-
* Fix any deprecation warnings you see.
|
22
|
-
* Upgrade to 5.x.
|
23
|
-
```ruby
|
24
|
-
gem 'sidekiq-pro', '< 6'
|
25
|
-
```
|