sidekiq 4.2.4 → 5.2.0
Sign up to get free protection for your applications and to get access to all the features.
Potentially problematic release.
This version of sidekiq might be problematic. Click here for more details.
- checksums.yaml +4 -4
- data/.github/issue_template.md +8 -1
- data/.gitignore +1 -0
- data/.travis.yml +5 -3
- data/5.0-Upgrade.md +56 -0
- data/COMM-LICENSE +1 -1
- data/Changes.md +151 -0
- data/Ent-Changes.md +77 -2
- data/Gemfile +10 -25
- data/LICENSE +1 -1
- data/Pro-4.0-Upgrade.md +35 -0
- data/Pro-Changes.md +156 -2
- data/README.md +9 -6
- data/Rakefile +1 -2
- data/bin/sidekiqctl +1 -1
- data/bin/sidekiqload +15 -33
- data/lib/generators/sidekiq/templates/worker_spec.rb.erb +1 -1
- data/lib/generators/sidekiq/templates/worker_test.rb.erb +1 -1
- data/lib/sidekiq/api.rb +157 -67
- data/lib/sidekiq/cli.rb +71 -26
- data/lib/sidekiq/client.rb +25 -18
- data/lib/sidekiq/core_ext.rb +1 -106
- data/lib/sidekiq/delay.rb +42 -0
- data/lib/sidekiq/exception_handler.rb +2 -4
- data/lib/sidekiq/extensions/generic_proxy.rb +7 -1
- data/lib/sidekiq/fetch.rb +1 -1
- data/lib/sidekiq/job_logger.rb +25 -0
- data/lib/sidekiq/job_retry.rb +241 -0
- data/lib/sidekiq/launcher.rb +45 -37
- data/lib/sidekiq/logging.rb +18 -2
- data/lib/sidekiq/manager.rb +3 -4
- data/lib/sidekiq/middleware/server/active_record.rb +10 -0
- data/lib/sidekiq/processor.rb +91 -34
- data/lib/sidekiq/rails.rb +15 -51
- data/lib/sidekiq/redis_connection.rb +31 -5
- data/lib/sidekiq/scheduled.rb +35 -8
- data/lib/sidekiq/testing.rb +24 -7
- data/lib/sidekiq/util.rb +6 -2
- data/lib/sidekiq/version.rb +1 -1
- data/lib/sidekiq/web/action.rb +2 -6
- data/lib/sidekiq/web/application.rb +28 -21
- data/lib/sidekiq/web/helpers.rb +67 -23
- data/lib/sidekiq/web/router.rb +14 -10
- data/lib/sidekiq/web.rb +4 -4
- data/lib/sidekiq/worker.rb +97 -14
- data/lib/sidekiq.rb +23 -24
- data/sidekiq.gemspec +7 -10
- data/web/assets/javascripts/application.js +0 -0
- data/web/assets/javascripts/dashboard.js +18 -13
- data/web/assets/stylesheets/application-rtl.css +246 -0
- data/web/assets/stylesheets/application.css +336 -4
- data/web/assets/stylesheets/bootstrap-rtl.min.css +9 -0
- data/web/assets/stylesheets/bootstrap.css +2 -2
- data/web/locales/ar.yml +80 -0
- data/web/locales/en.yml +1 -0
- data/web/locales/es.yml +4 -3
- data/web/locales/fa.yml +80 -0
- data/web/locales/he.yml +79 -0
- data/web/locales/ja.yml +5 -3
- data/web/locales/ur.yml +80 -0
- data/web/views/_footer.erb +5 -2
- data/web/views/_job_info.erb +1 -1
- data/web/views/_nav.erb +1 -1
- data/web/views/_paging.erb +1 -1
- data/web/views/busy.erb +9 -5
- data/web/views/dashboard.erb +3 -3
- data/web/views/layout.erb +11 -2
- data/web/views/morgue.erb +14 -10
- data/web/views/queue.erb +10 -10
- data/web/views/queues.erb +4 -2
- data/web/views/retries.erb +13 -11
- data/web/views/retry.erb +1 -1
- data/web/views/scheduled.erb +2 -2
- metadata +26 -160
- data/lib/sidekiq/middleware/server/logging.rb +0 -40
- data/lib/sidekiq/middleware/server/retry_jobs.rb +0 -205
- data/test/config.yml +0 -9
- data/test/env_based_config.yml +0 -11
- data/test/fake_env.rb +0 -1
- data/test/fixtures/en.yml +0 -2
- data/test/helper.rb +0 -75
- data/test/test_actors.rb +0 -138
- data/test/test_api.rb +0 -528
- data/test/test_cli.rb +0 -418
- data/test/test_client.rb +0 -266
- data/test/test_exception_handler.rb +0 -56
- data/test/test_extensions.rb +0 -127
- data/test/test_fetch.rb +0 -50
- data/test/test_launcher.rb +0 -95
- data/test/test_logging.rb +0 -35
- data/test/test_manager.rb +0 -50
- data/test/test_middleware.rb +0 -158
- data/test/test_processor.rb +0 -235
- data/test/test_rails.rb +0 -22
- data/test/test_redis_connection.rb +0 -132
- data/test/test_retry.rb +0 -326
- data/test/test_retry_exhausted.rb +0 -149
- data/test/test_scheduled.rb +0 -115
- data/test/test_scheduling.rb +0 -58
- data/test/test_sidekiq.rb +0 -107
- data/test/test_testing.rb +0 -143
- data/test/test_testing_fake.rb +0 -357
- data/test/test_testing_inline.rb +0 -94
- data/test/test_util.rb +0 -13
- data/test/test_web.rb +0 -726
- data/test/test_web_helpers.rb +0 -54
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 7d22391d1c7f5f4a237554a77f1648b42849a472
|
4
|
+
data.tar.gz: 4d1c7e75fec7f25a571868172aade268ab95a0b3
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: e70c87a39e46a8250ffde65b66f550e1849df7353940ed916765eb843a99c01fece8ce720d7eb08aa9ec5d8084363413e433628d852f42116670be423daaa138
|
7
|
+
data.tar.gz: bee6d32074c1a94dd5d22d87567a1499a1a655c371b011b2ea0c8bac3943eff3f6b3c82b91507bceb92b4e4e3265a9ef05560eabd95649361e60124b693fcdfa
|
data/.github/issue_template.md
CHANGED
@@ -1,4 +1,11 @@
|
|
1
1
|
Ruby version:
|
2
2
|
Sidekiq / Pro / Enterprise version(s):
|
3
3
|
|
4
|
-
|
4
|
+
Please include your initializer and any error message with the full backtrace.
|
5
|
+
|
6
|
+
Are you using an old version?
|
7
|
+
Have you checked the changelogs to see if your issue has been fixed in a later version?
|
8
|
+
|
9
|
+
https://github.com/mperham/sidekiq/blob/master/Changes.md
|
10
|
+
https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md
|
11
|
+
https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md
|
data/.gitignore
CHANGED
data/.travis.yml
CHANGED
data/5.0-Upgrade.md
ADDED
@@ -0,0 +1,56 @@
|
|
1
|
+
# Welcome to Sidekiq 5.0!
|
2
|
+
|
3
|
+
Sidekiq 5.0 contains a reworked job dispatch and execution core to integrate
|
4
|
+
better with the new Rails 5.0 Executor. It also drops support for older
|
5
|
+
versions of Ruby and Rails and adds support for RTL languages in the Web UI.
|
6
|
+
|
7
|
+
## What's New
|
8
|
+
|
9
|
+
* Integrate job logging and retry logic directly in with the job
|
10
|
+
execution logic in Sidekiq::Processor. Previously this logic was
|
11
|
+
defined as middleware. In Rails 5.0, ActiveSupport::Executor handles ActiveRecord
|
12
|
+
connection management, job callbacks, development mode class loading,
|
13
|
+
etc. Because of its extensive responsibilities, the Executor can't be
|
14
|
+
integrated as Sidekiq middleware; the logging/retry logic had to be pulled out
|
15
|
+
too. Sidekiq 4.2 had a hack to make it work but this redesign provides
|
16
|
+
a cleaner integration. [#3235]
|
17
|
+
* The Delayed Extensions `delay`, `delay_in` and `delay_until` APIs are
|
18
|
+
no longer available by default. The extensions allow you to marshal
|
19
|
+
job arguments as YAML, leading to cases where job payloads could be many
|
20
|
+
100s of KB or larger if not careful, leading to Redis networking
|
21
|
+
timeouts or other problems. As noted in the Best Practices wiki page,
|
22
|
+
Sidekiq is designed for jobs with small, simple arguments.
|
23
|
+
|
24
|
+
Add this line to your initializer to re-enable them and get the old behavior:
|
25
|
+
```ruby
|
26
|
+
Sidekiq::Extensions.enable_delay!
|
27
|
+
```
|
28
|
+
The old `Sidekiq.remove_delay!` API has been removed as it is now the default. [#3299]
|
29
|
+
* Sidekiq's quiet signal is now `TSTP` (think of it as **T**hread
|
30
|
+
**ST**o**P**) instead of USR1 as USR1 is not available on JRuby.
|
31
|
+
USR1 will continue to be supported in Sidekiq 5.x for backwards
|
32
|
+
compatibility and will be removed in Sidekiq 6.x. [#3302]
|
33
|
+
* The Web UI is now bi-directional - it can render either LTR
|
34
|
+
(left-to-right) or RTL languages. With this change, **Farsi, Arabic,
|
35
|
+
Hebrew and Urdu** are officially supported. [#3381]
|
36
|
+
* Jobs which can't be parsed due to invalid JSON are now pushed
|
37
|
+
immediately to the Dead set since they require manual intervention and
|
38
|
+
will never execute successfully as is. The Web UI has been updated to
|
39
|
+
more gracefully display these jobs. [#3296]
|
40
|
+
* **Rails 3.2** is no longer supported.
|
41
|
+
* **Ruby 2.0 and Ruby 2.1** are no longer supported. Ruby 2.2.2+ is required.
|
42
|
+
|
43
|
+
## Upgrade
|
44
|
+
|
45
|
+
As always, please upgrade Sidekiq **one major version at a time**.
|
46
|
+
If you are already running Sidekiq 4.x, then:
|
47
|
+
|
48
|
+
* Upgrade to the latest Sidekiq 4.x.
|
49
|
+
```ruby
|
50
|
+
gem 'sidekiq', '< 5'
|
51
|
+
```
|
52
|
+
* Fix any deprecation warnings you see.
|
53
|
+
* Upgrade to 5.x.
|
54
|
+
```ruby
|
55
|
+
gem 'sidekiq', '< 6'
|
56
|
+
```
|
data/COMM-LICENSE
CHANGED
@@ -38,7 +38,7 @@ under the terms of the GNU Lesser General Public License versions 3.0
|
|
38
38
|
|
39
39
|
5. Fees and Payment. The Software license fees will be due and payable in full as set forth in the applicable invoice or at the time of purchase. If the Software does not function properly within two weeks of purchase, please contact us within those two weeks for a refund. You shall be responsible for all taxes, withholdings, duties and levies arising from the order (excluding taxes based on the net income of Contributed Systems).
|
40
40
|
|
41
|
-
6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the Sidekiq Pro support page (https://github.com/mperham/sidekiq/wiki/
|
41
|
+
6. Support, Maintenance and Services. Subject to the terms and conditions of this Agreement, as set forth in your invoice, and as set forth on the Sidekiq Pro support page (https://github.com/mperham/sidekiq/wiki/Commercial-Support), support and maintenance services may be included with the purchase of your license subscription.
|
42
42
|
|
43
43
|
7. Term of Agreement.
|
44
44
|
|
data/Changes.md
CHANGED
@@ -1,5 +1,155 @@
|
|
1
1
|
# Sidekiq Changes
|
2
2
|
|
3
|
+
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
|
4
|
+
|
5
|
+
HEAD
|
6
|
+
-----------
|
7
|
+
|
8
|
+
- **Decrease default concurrency from 25 to 10** [#3892]
|
9
|
+
- Verify connection pool sizing upon startup [#3917]
|
10
|
+
- Smoother scheduling for large Sidekiq clusters [#3889]
|
11
|
+
- Switch Sidekiq::Testing impl from alias\_method to Module#prepend, for resiliency [#3852]
|
12
|
+
- Update Sidekiq APIs to use SCAN for scalability [#3848, ffiller]
|
13
|
+
- Remove concurrent-ruby gem dependency [#3830]
|
14
|
+
- Optimize Web UI's bootstrap.css [#3914]
|
15
|
+
|
16
|
+
5.1.3
|
17
|
+
-----------
|
18
|
+
|
19
|
+
- Fix version comparison so Ruby 2.2.10 works. [#3808, nateberkopec]
|
20
|
+
|
21
|
+
5.1.2
|
22
|
+
-----------
|
23
|
+
|
24
|
+
- Add link to docs in Web UI footer
|
25
|
+
- Fix crash on Ctrl-C in Windows [#3775, Bernica]
|
26
|
+
- Remove `freeze` calls on String constants. This is superfluous with Ruby
|
27
|
+
2.3+ and `frozen_string_literal: true`. [#3759]
|
28
|
+
- Fix use of AR middleware outside of Rails [#3787]
|
29
|
+
- Sidekiq::Worker `sidekiq_retry_in` block can now return nil or 0 to use
|
30
|
+
the default backoff delay [#3796, dsalahutdinov]
|
31
|
+
|
32
|
+
5.1.1
|
33
|
+
-----------
|
34
|
+
|
35
|
+
- Fix Web UI incompatibility with Redis 3.x gem [#3749]
|
36
|
+
|
37
|
+
5.1.0
|
38
|
+
-----------
|
39
|
+
|
40
|
+
- **NEW** Global death handlers - called when your job exhausts all
|
41
|
+
retries and dies. Now you can take action when a job fails permanently. [#3721]
|
42
|
+
- **NEW** Enable ActiveRecord query cache within jobs by default [#3718, sobrinho]
|
43
|
+
This will prevent duplicate SELECTS; cache is cleared upon any UPDATE/INSERT/DELETE.
|
44
|
+
See the issue for how to bypass the cache or disable it completely.
|
45
|
+
- Scheduler timing is now more accurate, 15 -> 5 seconds [#3734]
|
46
|
+
- Exceptions during the :startup event will now kill the process [#3717]
|
47
|
+
- Make `Sidekiq::Client.via` reentrant [#3715]
|
48
|
+
- Fix use of Sidekiq logger outside of the server process [#3714]
|
49
|
+
- Tweak `constantize` to better match Rails class lookup. [#3701, caffeinated-tech]
|
50
|
+
|
51
|
+
5.0.5
|
52
|
+
-----------
|
53
|
+
|
54
|
+
- Update gemspec to allow newer versions of the Redis gem [#3617]
|
55
|
+
- Refactor Worker.set so it can be memoized [#3602]
|
56
|
+
- Fix display of Redis URL in web footer, broken in 5.0.3 [#3560]
|
57
|
+
- Update `Sidekiq::Job#display_args` to avoid mutation [#3621]
|
58
|
+
|
59
|
+
5.0.4
|
60
|
+
-----------
|
61
|
+
|
62
|
+
- Fix "slow startup" performance regression from 5.0.2. [#3525]
|
63
|
+
- Allow users to disable ID generation since some redis providers disable the CLIENT command. [#3521]
|
64
|
+
|
65
|
+
5.0.3
|
66
|
+
-----------
|
67
|
+
|
68
|
+
- Fix overriding `class_attribute` core extension from ActiveSupport with Sidekiq one [PikachuEXE, #3499]
|
69
|
+
- Allow job logger to be overridden [AlfonsoUceda, #3502]
|
70
|
+
- Set a default Redis client identifier for debugging [#3516]
|
71
|
+
- Fix "Uninitialized constant" errors on startup with the delayed extensions [#3509]
|
72
|
+
|
73
|
+
5.0.2
|
74
|
+
-----------
|
75
|
+
|
76
|
+
- fix broken release, thanks @nateberkopec
|
77
|
+
|
78
|
+
5.0.1
|
79
|
+
-----------
|
80
|
+
|
81
|
+
- Fix incorrect server identity when daemonizing [jwilm, #3496]
|
82
|
+
- Work around error running Web UI against Redis Cluster [#3492]
|
83
|
+
- Remove core extensions, Sidekiq is now monkeypatch-free! [#3474]
|
84
|
+
- Reimplement Web UI's HTTP\_ACCEPT\_LANGUAGE parsing because the spec is utterly
|
85
|
+
incomprehensible for various edge cases. [johanlunds, natematykiewicz, #3449]
|
86
|
+
- Update `class_attribute` core extension to avoid warnings
|
87
|
+
- Expose `job_hash_context` from `Sidekiq::Logging` to support log customization
|
88
|
+
|
89
|
+
5.0.0
|
90
|
+
-----------
|
91
|
+
|
92
|
+
- **BREAKING CHANGE** Job dispatch was refactored for safer integration with
|
93
|
+
Rails 5. The **Logging** and **RetryJobs** server middleware were removed and
|
94
|
+
functionality integrated directly into Sidekiq::Processor. These aren't
|
95
|
+
commonly used public APIs so this shouldn't impact most users.
|
96
|
+
```
|
97
|
+
Sidekiq::Middleware::Server::RetryJobs -> Sidekiq::JobRetry
|
98
|
+
Sidekiq::Middleware::Server::Logging -> Sidekiq::JobLogger
|
99
|
+
```
|
100
|
+
- Quieting Sidekiq is now done via the TSTP signal, the USR1 signal is deprecated.
|
101
|
+
- The `delay` extension APIs are no longer available by default, you
|
102
|
+
must opt into them.
|
103
|
+
- The Web UI is now BiDi and can render RTL languages like Arabic, Farsi and Hebrew.
|
104
|
+
- Rails 3.2 and Ruby 2.0 and 2.1 are no longer supported.
|
105
|
+
- The `SomeWorker.set(options)` API was re-written to avoid thread-local state. [#2152]
|
106
|
+
- Sidekiq Enterprise's encrypted jobs now display "[encrypted data]" in the Web UI instead
|
107
|
+
of random hex bytes.
|
108
|
+
- Please see the [5.0 Upgrade notes](5.0-Upgrade.md) for more detail.
|
109
|
+
|
110
|
+
4.2.10
|
111
|
+
-----------
|
112
|
+
|
113
|
+
- Scheduled jobs can now be moved directly to the Dead queue via API [#3390]
|
114
|
+
- Fix edge case leading to job duplication when using Sidekiq Pro's
|
115
|
+
reliability feature [#3388]
|
116
|
+
- Fix error class name display on retry page [#3348]
|
117
|
+
- More robust latency calculation [#3340]
|
118
|
+
|
119
|
+
4.2.9
|
120
|
+
-----------
|
121
|
+
|
122
|
+
- Rollback [#3303] which broke Heroku Redis users [#3311]
|
123
|
+
- Add support for TSTP signal, for Sidekiq 5.0 forward compatibility. [#3302]
|
124
|
+
|
125
|
+
4.2.8
|
126
|
+
-----------
|
127
|
+
|
128
|
+
- Fix rare edge case with Redis driver that can create duplicate jobs [#3303]
|
129
|
+
- Fix Rails 5 loading issue [#3275]
|
130
|
+
- Restore missing tooltips to timestamps in Web UI [#3310]
|
131
|
+
- Work on **Sidekiq 5.0** is now active! [#3301]
|
132
|
+
|
133
|
+
4.2.7
|
134
|
+
-----------
|
135
|
+
|
136
|
+
- Add new integration testing to verify code loading and job execution
|
137
|
+
in development and production modes with Rails 4 and 5 [#3241]
|
138
|
+
- Fix delayed extensions in development mode [#3227, DarthSim]
|
139
|
+
- Use Worker's `retry` default if job payload does not have a retry
|
140
|
+
attribute [#3234, mlarraz]
|
141
|
+
|
142
|
+
4.2.6
|
143
|
+
-----------
|
144
|
+
|
145
|
+
- Run Rails Executor when in production [#3221, eugeneius]
|
146
|
+
|
147
|
+
4.2.5
|
148
|
+
-----------
|
149
|
+
|
150
|
+
- Re-enable eager loading of all code when running non-development Rails 5. [#3203]
|
151
|
+
- Better root URL handling for zany web servers [#3207]
|
152
|
+
|
3
153
|
4.2.4
|
4
154
|
-----------
|
5
155
|
|
@@ -19,6 +169,7 @@ Sidekiq::Web.set :sessions, false
|
|
19
169
|
- Fix Web UI sharding support broken in 4.2.2. [#3169]
|
20
170
|
- Fix timestamps not updating during UI polling [#3193, shaneog]
|
21
171
|
- Relax rack-protection version to >= 1.5.0
|
172
|
+
- Provide consistent interface to exception handlers, changing the structure of the context hash. [#3161]
|
22
173
|
|
23
174
|
4.2.2
|
24
175
|
-----------
|
data/Ent-Changes.md
CHANGED
@@ -1,8 +1,83 @@
|
|
1
|
-
Sidekiq Enterprise Changelog
|
2
|
-
|
1
|
+
# Sidekiq Enterprise Changelog
|
2
|
+
|
3
|
+
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
|
3
4
|
|
4
5
|
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
|
5
6
|
|
7
|
+
HEAD
|
8
|
+
-------------
|
9
|
+
|
10
|
+
- Add support for sidekiqswarm memory monitoring on FreeBSD [#3884]
|
11
|
+
|
12
|
+
1.7.1
|
13
|
+
-------------
|
14
|
+
|
15
|
+
- Fix Lua error in concurrent rate limiter under heavy contention
|
16
|
+
- Remove superfluous `freeze` calls on Strings [#3759]
|
17
|
+
|
18
|
+
1.7.0
|
19
|
+
-------------
|
20
|
+
|
21
|
+
- **NEW FEATURE** [Rolling restarts](https://github.com/mperham/sidekiq/wiki/Ent-Rolling-Restarts) - great for long running jobs!
|
22
|
+
- Adjust middleware so unique jobs that don't push aren't registered in a Batch [#3662]
|
23
|
+
- Add new unlimited rate limiter, useful for testing [#3743]
|
24
|
+
```ruby
|
25
|
+
limiter = Sidekiq::Limiter.unlimited(...any args...)
|
26
|
+
```
|
27
|
+
|
28
|
+
1.6.1
|
29
|
+
-------------
|
30
|
+
|
31
|
+
- Fix crash in rate limiter middleware when used with custom exceptions [#3604]
|
32
|
+
|
33
|
+
1.6.0
|
34
|
+
-------------
|
35
|
+
|
36
|
+
- Show process "leader" tag on Busy page, requires Sidekiq 5.0.2 [#2867]
|
37
|
+
- Capture custom metrics with the `save_history` API. [#2815]
|
38
|
+
- Implement new `unique_until: 'start'` policy option. [#3471]
|
39
|
+
|
40
|
+
1.5.4
|
41
|
+
-------------
|
42
|
+
|
43
|
+
- Fix broken Cron page in Web UI [#3458]
|
44
|
+
|
45
|
+
1.5.3
|
46
|
+
-------------
|
47
|
+
|
48
|
+
- Remove dependency on the algorithms gem [#3446]
|
49
|
+
- Allow user to specify max memory in megabytes with SIDEKIQ\_MAXMEM\_MB [#3451]
|
50
|
+
- Implement logic to detect app startup failure, sidekiqswarm will exit
|
51
|
+
rather than try to restart the app forever [#3450]
|
52
|
+
- Another fix for doubly-encrypted arguments [#3368]
|
53
|
+
|
54
|
+
1.5.2
|
55
|
+
-------------
|
56
|
+
|
57
|
+
- Fix encrypted arguments double-encrypted by retry or rate limiting [#3368]
|
58
|
+
- Fix leak in concurrent rate limiter, run this in Rails console to clean up existing data [#3323]
|
59
|
+
```ruby
|
60
|
+
expiry = 1.month.to_i; Sidekiq::Limiter.redis { |c| c.scan_each(match: "lmtr-cfree-*") { |key| c.expire(key, expiry) } }
|
61
|
+
```
|
62
|
+
|
63
|
+
1.5.1
|
64
|
+
-------------
|
65
|
+
|
66
|
+
- Fix issue with census startup when not using Bundler configuration for
|
67
|
+
source credentials.
|
68
|
+
|
69
|
+
1.5.0
|
70
|
+
-------------
|
71
|
+
|
72
|
+
- Add new web authorization API [#3251]
|
73
|
+
- Update all sidekiqswarm env vars to use SIDEKIQ\_ prefix [#3218]
|
74
|
+
- Add census reporting, the leader will ping contribsys nightly with aggregate usage metrics
|
75
|
+
|
76
|
+
1.4.0
|
77
|
+
-------------
|
78
|
+
|
79
|
+
- No functional changes, require latest Sidekiq and Sidekiq Pro versions
|
80
|
+
|
6
81
|
1.3.2
|
7
82
|
-------------
|
8
83
|
|
data/Gemfile
CHANGED
@@ -1,29 +1,14 @@
|
|
1
1
|
source 'https://rubygems.org'
|
2
2
|
gemspec
|
3
3
|
|
4
|
-
|
5
|
-
gem "hiredis"
|
6
|
-
gem '
|
7
|
-
|
8
|
-
|
9
|
-
gem '
|
10
|
-
|
11
|
-
|
12
|
-
gem '
|
13
|
-
gem '
|
14
|
-
gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
|
15
|
-
end
|
16
|
-
|
17
|
-
platforms :ruby do
|
18
|
-
gem 'sqlite3'
|
4
|
+
# load testing
|
5
|
+
#gem "hiredis"
|
6
|
+
#gem 'toxiproxy'
|
7
|
+
|
8
|
+
group :test do
|
9
|
+
gem 'rails', '>= 5.0.1'
|
10
|
+
gem 'minitest'
|
11
|
+
gem 'rake'
|
12
|
+
gem 'redis-namespace'
|
13
|
+
gem 'simplecov'
|
19
14
|
end
|
20
|
-
|
21
|
-
platforms :mri do
|
22
|
-
gem 'pry-byebug'
|
23
|
-
gem 'ruby-prof'
|
24
|
-
end
|
25
|
-
|
26
|
-
#platforms :jruby do
|
27
|
-
#gem 'jruby-openssl'
|
28
|
-
#gem 'activerecord-jdbcsqlite3-adapter'
|
29
|
-
#end
|
data/LICENSE
CHANGED
@@ -5,5 +5,5 @@ the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
|
|
5
5
|
for license text.
|
6
6
|
|
7
7
|
Sidekiq Pro has a commercial-friendly license allowing private forks
|
8
|
-
and modifications of Sidekiq. Please see
|
8
|
+
and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
|
9
9
|
more detail. You can find the commercial license terms in COMM-LICENSE.
|
data/Pro-4.0-Upgrade.md
ADDED
@@ -0,0 +1,35 @@
|
|
1
|
+
# Welcome to Sidekiq Pro 4.0!
|
2
|
+
|
3
|
+
Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
|
4
|
+
|
5
|
+
## What's New
|
6
|
+
|
7
|
+
* Batches now "die" if any of their jobs die. You can enumerate the set
|
8
|
+
of dead batches and their associated dead jobs. The success callback
|
9
|
+
for a dead batch will never fire unless these jobs are fixed.
|
10
|
+
```ruby
|
11
|
+
Sidekiq::Batch::DeadSet.new.each do |status|
|
12
|
+
status.dead? # => true
|
13
|
+
status.dead_jobs # => [...]
|
14
|
+
end
|
15
|
+
```
|
16
|
+
This API allows you to enumerate the batches which need help.
|
17
|
+
If you fix the issue and the dead jobs succeed, the batch will succeed.
|
18
|
+
* The older `reliable_fetch` and `timed_fetch` algorithms have been
|
19
|
+
removed. Only super\_fetch is available in 4.0.
|
20
|
+
* The statsd middleware has been tweaked to remove support for legacy,
|
21
|
+
pre-3.6.0 configuration and add relevant tags.
|
22
|
+
* Requires Sidekiq 5.0.5+.
|
23
|
+
|
24
|
+
## Upgrade
|
25
|
+
|
26
|
+
* Upgrade to the latest Sidekiq Pro 3.x.
|
27
|
+
```ruby
|
28
|
+
gem 'sidekiq-pro', '< 4'
|
29
|
+
```
|
30
|
+
* Fix any deprecation warnings you see.
|
31
|
+
* Upgrade to 4.x.
|
32
|
+
```ruby
|
33
|
+
gem 'sidekiq-pro', '< 5'
|
34
|
+
```
|
35
|
+
|
data/Pro-Changes.md
CHANGED
@@ -1,8 +1,162 @@
|
|
1
|
-
Sidekiq Pro Changelog
|
2
|
-
|
1
|
+
# Sidekiq Pro Changelog
|
2
|
+
|
3
|
+
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
|
3
4
|
|
4
5
|
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
|
5
6
|
|
7
|
+
4.0.3
|
8
|
+
---------
|
9
|
+
|
10
|
+
- Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
|
11
|
+
- Implement batch death callback. This is fired the first time a job within a batch dies. [#3841]
|
12
|
+
```ruby
|
13
|
+
batch = Sidekiq::Batch.new
|
14
|
+
batch.on(:death, ...)
|
15
|
+
```
|
16
|
+
|
17
|
+
4.0.2
|
18
|
+
---------
|
19
|
+
|
20
|
+
- Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
|
21
|
+
call and resulting latency [#3790]
|
22
|
+
- Fix possible bad statsd metric call on super\_fetch startup
|
23
|
+
- Remove superfluous `freeze` calls on Strings [#3759]
|
24
|
+
|
25
|
+
4.0.1
|
26
|
+
---------
|
27
|
+
|
28
|
+
- Fix incompatibility with the statsd-ruby gem [#3740]
|
29
|
+
|
30
|
+
4.0.0
|
31
|
+
---------
|
32
|
+
|
33
|
+
- See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
|
34
|
+
|
35
|
+
|
36
|
+
3.7.1
|
37
|
+
---------
|
38
|
+
|
39
|
+
- Deprecate timed\_fetch. Switch to super\_fetch:
|
40
|
+
```ruby
|
41
|
+
config.super_fetch!
|
42
|
+
```
|
43
|
+
|
44
|
+
|
45
|
+
3.7.0
|
46
|
+
---------
|
47
|
+
|
48
|
+
- Refactor batch job success/failure to gracefully handle several edge
|
49
|
+
cases with regard to Sidekiq::Shutdown. This should greatly reduce
|
50
|
+
the chances of seeing the long-standing "negative pending count" problem. [#3710]
|
51
|
+
|
52
|
+
|
53
|
+
3.6.1
|
54
|
+
---------
|
55
|
+
|
56
|
+
- Add support for Datadog::Statsd, it is the recommended Statsd client. [#3699]
|
57
|
+
```ruby
|
58
|
+
Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
|
59
|
+
```
|
60
|
+
- Size the statsd connection pool based on Sidekiq's concurrency [#3700]
|
61
|
+
|
62
|
+
|
63
|
+
3.6.0
|
64
|
+
---------
|
65
|
+
|
66
|
+
This release overhauls the Statsd metrics support and adds more
|
67
|
+
metrics for tracking Pro feature usage. In your initializer:
|
68
|
+
```ruby
|
69
|
+
Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
|
70
|
+
```
|
71
|
+
Sidekiq Pro will emit more metrics to Statsd:
|
72
|
+
```
|
73
|
+
jobs.expired - when a job is expired
|
74
|
+
jobs.recovered.push - when a job is recovered by reliable_push after network outage
|
75
|
+
jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
|
76
|
+
batch.created - when a batch is created
|
77
|
+
batch.complete - when a batch is completed
|
78
|
+
batch.success - when a batch is successful
|
79
|
+
```
|
80
|
+
Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
|
81
|
+
Everything should be backwards compatible with one deprecation notice.
|
82
|
+
|
83
|
+
|
84
|
+
3.5.4
|
85
|
+
---------
|
86
|
+
|
87
|
+
- Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
|
88
|
+
- Fix case where TimedFetch might not recover some pending jobs
|
89
|
+
- Fix edge case in Batch::Status#poll leading to premature completion [#3640]
|
90
|
+
- Adjust scan API to check 100 elements at a time, to minimize network round trips
|
91
|
+
when scanning large sets.
|
92
|
+
|
93
|
+
3.5.3
|
94
|
+
---------
|
95
|
+
|
96
|
+
- Restore error check for super\_fetch's job ack [#3601]
|
97
|
+
- Trim error messages saved in Batch's failure hash, preventing huge
|
98
|
+
messages from bloating Redis. [#3570]
|
99
|
+
|
100
|
+
3.5.2
|
101
|
+
---------
|
102
|
+
|
103
|
+
- Fix `Status#completed?` when run against a Batch that had succeeded
|
104
|
+
and was deleted. [#3519]
|
105
|
+
|
106
|
+
3.5.1
|
107
|
+
---------
|
108
|
+
|
109
|
+
- Work with Sidekiq 5.0.2+
|
110
|
+
- Improve performance of super\_fetch with weighted queues [#3489]
|
111
|
+
|
112
|
+
3.5.0
|
113
|
+
---------
|
114
|
+
|
115
|
+
- Add queue pause/unpause endpoints for scripting via curl [#3445]
|
116
|
+
- Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
|
117
|
+
- Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
|
118
|
+
- Batch page displays Pending JIDs if less than 10 [#3130]
|
119
|
+
- Batch page has a Search button to find associated Retries [#3130]
|
120
|
+
- Make Batch UI progress bar more friendly to the colorblind [#3387]
|
121
|
+
|
122
|
+
3.4.5
|
123
|
+
---------
|
124
|
+
|
125
|
+
- Fix potential job loss with reliable scheduler when lots of jobs are scheduled
|
126
|
+
at precisely the same time. Thanks to raivil for his hard work in
|
127
|
+
reproducing the bug. [#3371]
|
128
|
+
|
129
|
+
3.4.4
|
130
|
+
---------
|
131
|
+
|
132
|
+
- Optimize super\_fetch shutdown to restart jobs quicker [#3249]
|
133
|
+
|
134
|
+
3.4.3
|
135
|
+
---------
|
136
|
+
|
137
|
+
- Limit reliable scheduler to enqueue up to 100 jobs per call, minimizing Redis latency [#3332]
|
138
|
+
- Fix bug in super\_fetch logic for queues with `_` in the name [#3339]
|
139
|
+
|
140
|
+
3.4.2
|
141
|
+
---------
|
142
|
+
|
143
|
+
- Add `Batch::Status#invalidated?` API which returns true if any/all
|
144
|
+
JIDs were invalidated within the batch. [#3326]
|
145
|
+
|
146
|
+
3.4.1
|
147
|
+
---------
|
148
|
+
|
149
|
+
- Allow super\_fetch's orphan job check to happen as often as every hour [#3273]
|
150
|
+
- Officially deprecate reliable\_fetch algorithm, I now recommend you use `super_fetch` instead:
|
151
|
+
```ruby
|
152
|
+
Sidekiq.configure_server do |config|
|
153
|
+
config.super_fetch!
|
154
|
+
end
|
155
|
+
```
|
156
|
+
Also note that Sidekiq's `-i/--index` option is no longer used/relevant with super\_fetch.
|
157
|
+
- Don't display "Delete/Retry All" buttons when filtering in Web UI [#3243]
|
158
|
+
- Reimplement Sidekiq::JobSet#find\_job with ZSCAN [#3197]
|
159
|
+
|
6
160
|
3.4.0
|
7
161
|
---------
|
8
162
|
|
data/README.md
CHANGED
@@ -28,13 +28,14 @@ Sidekiq 3.5.1 | 22ms | 1257 MB | 125 sec | 800 jobs/sec
|
|
28
28
|
Resque 1.25.2 | - | - | 420 sec | 240 jobs/sec
|
29
29
|
DelayedJob 4.1.1 | - | - | 465 sec | 215 jobs/sec
|
30
30
|
|
31
|
+
<small>This benchmark can be found in `bin/sidekiqload`.</small>
|
31
32
|
|
32
33
|
Requirements
|
33
34
|
-----------------
|
34
35
|
|
35
|
-
Sidekiq supports CRuby 2.
|
36
|
+
Sidekiq supports CRuby 2.2.2+ and JRuby 9k.
|
36
37
|
|
37
|
-
All Rails releases >=
|
38
|
+
All Rails releases >= 4.0 are officially supported.
|
38
39
|
|
39
40
|
Redis 2.8 or greater is required. 3.0.3+ is recommended for large
|
40
41
|
installations with thousands of worker threads.
|
@@ -50,7 +51,8 @@ Getting Started
|
|
50
51
|
-----------------
|
51
52
|
|
52
53
|
See the [Getting Started wiki page](https://github.com/mperham/sidekiq/wiki/Getting-Started) and follow the simple setup process.
|
53
|
-
You can watch [
|
54
|
+
You can watch [this Youtube playlist](https://www.youtube.com/playlist?list=PLjeHh2LSCFrWGT5uVjUuFKAcrcj5kSai1) to learn all about
|
55
|
+
Sidekiq and see its features in action. Here's the Web UI:
|
54
56
|
|
55
57
|
![Web UI](https://github.com/mperham/sidekiq/raw/master/examples/web-ui.png)
|
56
58
|
|
@@ -75,17 +77,18 @@ Problems?
|
|
75
77
|
If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
|
76
78
|
Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
|
77
79
|
|
80
|
+
Sidekiq Pro and Sidekiq Enterprise customers get private email support. You can purchase at http://sidekiq.org; email support@contribsys.com for help.
|
81
|
+
|
78
82
|
Useful resources:
|
79
83
|
|
80
84
|
* Product documentation is in the [wiki](https://github.com/mperham/sidekiq/wiki).
|
81
85
|
* Release announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
|
82
|
-
* Here's a [Reddit forum](https://reddit.com/r/sidekiq) dedicated to Sidekiq discussion
|
83
86
|
* The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
|
84
87
|
|
85
|
-
**No support via Twitter
|
88
|
+
**No support via Twitter**
|
86
89
|
|
87
90
|
Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
|
88
|
-
See the [Sidekiq support page](http://sidekiq.org/support) for details.
|
91
|
+
See the [Sidekiq support page](http://sidekiq.org/support.html) for details.
|
89
92
|
|
90
93
|
Thanks
|
91
94
|
-----------------
|
data/Rakefile
CHANGED