sidekiq 6.1.1

Sign up to get free protection for your applications and to get access to all the features.

Potentially problematic release.


This version of sidekiq might be problematic. Click here for more details.

Files changed (127) hide show
  1. checksums.yaml +7 -0
  2. data/.circleci/config.yml +71 -0
  3. data/.github/contributing.md +32 -0
  4. data/.github/issue_template.md +11 -0
  5. data/.gitignore +13 -0
  6. data/.standard.yml +20 -0
  7. data/3.0-Upgrade.md +70 -0
  8. data/4.0-Upgrade.md +53 -0
  9. data/5.0-Upgrade.md +56 -0
  10. data/6.0-Upgrade.md +72 -0
  11. data/COMM-LICENSE +97 -0
  12. data/Changes.md +1724 -0
  13. data/Ent-2.0-Upgrade.md +37 -0
  14. data/Ent-Changes.md +275 -0
  15. data/Gemfile +24 -0
  16. data/Gemfile.lock +208 -0
  17. data/LICENSE +9 -0
  18. data/Pro-2.0-Upgrade.md +138 -0
  19. data/Pro-3.0-Upgrade.md +44 -0
  20. data/Pro-4.0-Upgrade.md +35 -0
  21. data/Pro-5.0-Upgrade.md +25 -0
  22. data/Pro-Changes.md +795 -0
  23. data/README.md +94 -0
  24. data/Rakefile +10 -0
  25. data/bin/sidekiq +42 -0
  26. data/bin/sidekiqload +157 -0
  27. data/bin/sidekiqmon +8 -0
  28. data/code_of_conduct.md +50 -0
  29. data/lib/generators/sidekiq/templates/worker.rb.erb +9 -0
  30. data/lib/generators/sidekiq/templates/worker_spec.rb.erb +6 -0
  31. data/lib/generators/sidekiq/templates/worker_test.rb.erb +8 -0
  32. data/lib/generators/sidekiq/worker_generator.rb +57 -0
  33. data/lib/sidekiq.rb +262 -0
  34. data/lib/sidekiq/api.rb +963 -0
  35. data/lib/sidekiq/cli.rb +396 -0
  36. data/lib/sidekiq/client.rb +263 -0
  37. data/lib/sidekiq/delay.rb +41 -0
  38. data/lib/sidekiq/exception_handler.rb +27 -0
  39. data/lib/sidekiq/extensions/action_mailer.rb +47 -0
  40. data/lib/sidekiq/extensions/active_record.rb +43 -0
  41. data/lib/sidekiq/extensions/class_methods.rb +43 -0
  42. data/lib/sidekiq/extensions/generic_proxy.rb +31 -0
  43. data/lib/sidekiq/fetch.rb +80 -0
  44. data/lib/sidekiq/job_logger.rb +63 -0
  45. data/lib/sidekiq/job_retry.rb +262 -0
  46. data/lib/sidekiq/launcher.rb +206 -0
  47. data/lib/sidekiq/logger.rb +165 -0
  48. data/lib/sidekiq/manager.rb +135 -0
  49. data/lib/sidekiq/middleware/chain.rb +160 -0
  50. data/lib/sidekiq/middleware/i18n.rb +40 -0
  51. data/lib/sidekiq/monitor.rb +133 -0
  52. data/lib/sidekiq/paginator.rb +47 -0
  53. data/lib/sidekiq/processor.rb +280 -0
  54. data/lib/sidekiq/rails.rb +50 -0
  55. data/lib/sidekiq/redis_connection.rb +146 -0
  56. data/lib/sidekiq/scheduled.rb +173 -0
  57. data/lib/sidekiq/sd_notify.rb +149 -0
  58. data/lib/sidekiq/systemd.rb +24 -0
  59. data/lib/sidekiq/testing.rb +344 -0
  60. data/lib/sidekiq/testing/inline.rb +30 -0
  61. data/lib/sidekiq/util.rb +67 -0
  62. data/lib/sidekiq/version.rb +5 -0
  63. data/lib/sidekiq/web.rb +213 -0
  64. data/lib/sidekiq/web/action.rb +93 -0
  65. data/lib/sidekiq/web/application.rb +357 -0
  66. data/lib/sidekiq/web/csrf_protection.rb +153 -0
  67. data/lib/sidekiq/web/helpers.rb +333 -0
  68. data/lib/sidekiq/web/router.rb +101 -0
  69. data/lib/sidekiq/worker.rb +244 -0
  70. data/sidekiq.gemspec +20 -0
  71. data/web/assets/images/favicon.ico +0 -0
  72. data/web/assets/images/logo.png +0 -0
  73. data/web/assets/images/status.png +0 -0
  74. data/web/assets/javascripts/application.js +90 -0
  75. data/web/assets/javascripts/dashboard.js +296 -0
  76. data/web/assets/stylesheets/application-dark.css +133 -0
  77. data/web/assets/stylesheets/application-rtl.css +246 -0
  78. data/web/assets/stylesheets/application.css +1158 -0
  79. data/web/assets/stylesheets/bootstrap-rtl.min.css +9 -0
  80. data/web/assets/stylesheets/bootstrap.css +5 -0
  81. data/web/locales/ar.yml +81 -0
  82. data/web/locales/cs.yml +78 -0
  83. data/web/locales/da.yml +68 -0
  84. data/web/locales/de.yml +81 -0
  85. data/web/locales/el.yml +68 -0
  86. data/web/locales/en.yml +83 -0
  87. data/web/locales/es.yml +70 -0
  88. data/web/locales/fa.yml +80 -0
  89. data/web/locales/fr.yml +78 -0
  90. data/web/locales/he.yml +79 -0
  91. data/web/locales/hi.yml +75 -0
  92. data/web/locales/it.yml +69 -0
  93. data/web/locales/ja.yml +83 -0
  94. data/web/locales/ko.yml +68 -0
  95. data/web/locales/lt.yml +83 -0
  96. data/web/locales/nb.yml +77 -0
  97. data/web/locales/nl.yml +68 -0
  98. data/web/locales/pl.yml +59 -0
  99. data/web/locales/pt-br.yml +68 -0
  100. data/web/locales/pt.yml +67 -0
  101. data/web/locales/ru.yml +78 -0
  102. data/web/locales/sv.yml +68 -0
  103. data/web/locales/ta.yml +75 -0
  104. data/web/locales/uk.yml +76 -0
  105. data/web/locales/ur.yml +80 -0
  106. data/web/locales/vi.yml +83 -0
  107. data/web/locales/zh-cn.yml +68 -0
  108. data/web/locales/zh-tw.yml +68 -0
  109. data/web/views/_footer.erb +20 -0
  110. data/web/views/_job_info.erb +89 -0
  111. data/web/views/_nav.erb +52 -0
  112. data/web/views/_paging.erb +23 -0
  113. data/web/views/_poll_link.erb +7 -0
  114. data/web/views/_status.erb +4 -0
  115. data/web/views/_summary.erb +40 -0
  116. data/web/views/busy.erb +101 -0
  117. data/web/views/dashboard.erb +75 -0
  118. data/web/views/dead.erb +34 -0
  119. data/web/views/layout.erb +41 -0
  120. data/web/views/morgue.erb +78 -0
  121. data/web/views/queue.erb +55 -0
  122. data/web/views/queues.erb +38 -0
  123. data/web/views/retries.erb +83 -0
  124. data/web/views/retry.erb +34 -0
  125. data/web/views/scheduled.erb +57 -0
  126. data/web/views/scheduled_job_info.erb +8 -0
  127. metadata +212 -0
data/LICENSE ADDED
@@ -0,0 +1,9 @@
1
+ Copyright (c) Contributed Systems LLC
2
+
3
+ Sidekiq is an Open Source project licensed under the terms of
4
+ the LGPLv3 license. Please see <http://www.gnu.org/licenses/lgpl-3.0.html>
5
+ for license text.
6
+
7
+ Sidekiq Pro has a commercial-friendly license allowing private forks
8
+ and modifications of Sidekiq. Please see https://sidekiq.org/products/pro.html for
9
+ more detail. You can find the commercial license terms in COMM-LICENSE.
@@ -0,0 +1,138 @@
1
+ # Upgrading to Sidekiq Pro 2.0
2
+
3
+ Sidekiq Pro 2.0 allows nested batches for more complex job workflows
4
+ and provides a new reliable scheduler which uses Lua to guarantee
5
+ atomicity and much higher performance.
6
+
7
+ It also removes deprecated APIs, changes the batch data format and
8
+ how features are activated. Read carefully to ensure your upgrade goes
9
+ smoothly.
10
+
11
+ Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
12
+ recommended; Redis 2.4 or 2.6 will work but some functionality will not be
13
+ available.
14
+
15
+ **Note that you CANNOT go back to Pro 1.x once you've created batches
16
+ with 2.x. The new batches will not process correctly with 1.x.**
17
+
18
+ **If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
19
+ latest 1.x version and run it for a week before upgrading to 2.0.**
20
+
21
+ ## Nested Batches
22
+
23
+ Batches can now be nested within the `jobs` method.
24
+ This feature enables Sidekiq Pro to handle workflow processing of any size
25
+ and complexity!
26
+
27
+ ```ruby
28
+ a = Sidekiq::Batch.new
29
+ a.on(:success, SomeCallback)
30
+ a.jobs do
31
+ SomeWork.perform_async
32
+
33
+ b = Sidekiq::Batch.new
34
+ b.on(:success, MyCallback)
35
+ b.jobs do
36
+ OtherWork.perform_async
37
+ end
38
+ end
39
+ ```
40
+
41
+ Parent batch callbacks are not processed until all child batch callbacks have
42
+ run successfully. In the example above, `MyCallback` will always fire
43
+ before `SomeCallback` because `b` is considered a child of `a`.
44
+
45
+ Of course you can dynamically add child batches while a batch job is executing.
46
+
47
+ ```ruby
48
+ def perform(*args)
49
+ do_something(args)
50
+
51
+ if more_work?
52
+ # Sidekiq::Worker#batch returns the Batch this job is part of.
53
+ batch.jobs do
54
+ b = Sidekiq::Batch.new
55
+ b.on(:success, MyCallback)
56
+ b.jobs do
57
+ OtherWork.perform_async
58
+ end
59
+ end
60
+ end
61
+ end
62
+ ```
63
+
64
+ More context: [#1485]
65
+
66
+ ## Batch Data
67
+
68
+ The batch data model was overhauled. Batch data should take
69
+ significantly less space in Redis now. A simple benchmark shows 25%
70
+ savings but real world savings should be even greater.
71
+
72
+ * Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
73
+ "vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
74
+ strings, e.g. "4a3fc67d30370edf".
75
+ * In 1.x, batch data was not removed until it naturally expired in Redis.
76
+ In 2.x, all data for a batch is removed from Redis once the batch has
77
+ run any success callbacks.
78
+ * Because of the former point, batch expiry is no longer a concern.
79
+ Batch expiry is hardcoded to 30 days and is no longer user-tunable.
80
+ * Failed batch jobs no longer automatically store any associated
81
+ backtrace in Redis.
82
+
83
+ **There's no data migration required. Sidekiq Pro 2.0 transparently handles
84
+ both old and new format.**
85
+
86
+ More context: [#2130]
87
+
88
+ ## Reliability
89
+
90
+ 2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
91
+ scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
92
+ lots of jobs.
93
+
94
+ **Two caveats**:
95
+ - Client-side middleware is not executed
96
+ for each job when enqueued with the reliable scheduler. No Sidekiq or
97
+ Sidekiq Pro functionality is affected by this change but some 3rd party
98
+ plugins might be.
99
+ - The Lua script used inside the reliable scheduler is not safe for use
100
+ with Redis Cluster or other multi-master Redis solutions.
101
+ It is safe to use with Redis Sentinel or a typical master/slave replication setup.
102
+
103
+ **You no longer require anything to use the Reliability features.**
104
+
105
+ * Activate reliable fetch and/or the new reliable scheduler:
106
+ ```ruby
107
+ Sidekiq.configure_server do |config|
108
+ config.reliable_fetch!
109
+ config.reliable_scheduler!
110
+ end
111
+ ```
112
+ * Activate reliable push:
113
+ ```ruby
114
+ Sidekiq::Client.reliable_push!
115
+ ```
116
+
117
+ More context: [#2130]
118
+
119
+ ## Other Changes
120
+
121
+ * You must require `sidekiq/pro/notifications` if you want to use the
122
+ existing notification schemes. I don't recommend using them as the
123
+ newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
124
+ * Several classes have been renamed. Generally these classes are ones
125
+ you should not need to require/use in your own code, e.g. the Batch
126
+ middleware.
127
+ * You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
128
+ Pro will set it to the jid of the callback job. [#2178]
129
+ * There's now an official API to iterate all known Batches [#2191]
130
+ ```ruby
131
+ Sidekiq::BatchSet.new.each {|status| p status.bid }
132
+ ```
133
+ * The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
134
+
135
+ ## Thanks
136
+
137
+ Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
138
+ feedback during the release process. Thank you guys!
@@ -0,0 +1,44 @@
1
+ # Welcome to Sidekiq Pro 3.0!
2
+
3
+ Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
4
+
5
+ ## What's New
6
+
7
+ * **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
8
+ ago and contains **many** useful features which Sidekiq couldn't
9
+ leverage until now. **Redis 3.0.3 or greater is recommended** for large
10
+ scale use.
11
+
12
+ * Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
13
+ you will need to pull it in yourself.
14
+
15
+ * Pausing and unpausing queues is now instantaneous, no more polling!
16
+
17
+ * Reliable fetch has been re-implemented due to the fetch changes in
18
+ Sidekiq 4.0.
19
+
20
+ * Support for platforms without persistent hostnames. Since the reliable\_fetch
21
+ algorithm requires a persistent hostname, an alternative reliability
22
+ algorithm is now available for platforms like Heroku and Docker:
23
+ ```ruby
24
+ Sidekiq.configure_server do |config|
25
+ config.timed_fetch!
26
+ end
27
+ ```
28
+ The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
29
+
30
+ * The old 'sidekiq/notifications' features have been removed.
31
+
32
+ ## Upgrade
33
+
34
+ First, make sure you are using Redis 2.8 or greater. Next:
35
+
36
+ * Upgrade to the latest Sidekiq Pro 2.x.
37
+ ```ruby
38
+ gem 'sidekiq-pro', '< 3'
39
+ ```
40
+ * Fix any deprecation warnings you see.
41
+ * Upgrade to 3.x.
42
+ ```ruby
43
+ gem 'sidekiq-pro', '< 4'
44
+ ```
@@ -0,0 +1,35 @@
1
+ # Welcome to Sidekiq Pro 4.0!
2
+
3
+ Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
4
+
5
+ ## What's New
6
+
7
+ * Batches now "die" if any of their jobs die. You can enumerate the set
8
+ of dead batches and their associated dead jobs. The success callback
9
+ for a dead batch will never fire unless these jobs are fixed.
10
+ ```ruby
11
+ Sidekiq::Batch::DeadSet.new.each do |status|
12
+ status.dead? # => true
13
+ status.dead_jobs # => [...]
14
+ end
15
+ ```
16
+ This API allows you to enumerate the batches which need help.
17
+ If you fix the issue and the dead jobs succeed, the batch will succeed.
18
+ * The older `reliable_fetch` and `timed_fetch` algorithms have been
19
+ removed. Only super\_fetch is available in 4.0.
20
+ * The statsd middleware has been tweaked to remove support for legacy,
21
+ pre-3.6.0 configuration and add relevant tags.
22
+ * Requires Sidekiq 5.0.5+.
23
+
24
+ ## Upgrade
25
+
26
+ * Upgrade to the latest Sidekiq Pro 3.x.
27
+ ```ruby
28
+ gem 'sidekiq-pro', '< 4'
29
+ ```
30
+ * Fix any deprecation warnings you see.
31
+ * Upgrade to 4.x.
32
+ ```ruby
33
+ gem 'sidekiq-pro', '< 5'
34
+ ```
35
+
@@ -0,0 +1,25 @@
1
+ # Welcome to Sidekiq Pro 5.0!
2
+
3
+ Sidekiq Pro 5.0 is mainly a cleanup release for Sidekiq 6.0. The
4
+ migration should be as close to trivial as a major version bump can be.
5
+ Note that Sidekiq 6.0 does have major breaking changes.
6
+
7
+ ## What's New
8
+
9
+ * New localizations for the Sidekiq Pro Web UI: ES, ZH, PT, JA, RU
10
+ * Removed deprecated APIs and warnings.
11
+ * Various changes for Sidekiq 6.0
12
+ * Requires Ruby 2.5+ and Redis 4.0+
13
+ * Requires Sidekiq 6.0+.
14
+
15
+ ## Upgrade
16
+
17
+ * Upgrade to the latest Sidekiq Pro 4.x.
18
+ ```ruby
19
+ gem 'sidekiq-pro', '< 5'
20
+ ```
21
+ * Fix any deprecation warnings you see.
22
+ * Upgrade to 5.x.
23
+ ```ruby
24
+ gem 'sidekiq-pro', '< 6'
25
+ ```
@@ -0,0 +1,795 @@
1
+ # Sidekiq Pro Changelog
2
+
3
+ [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
4
+
5
+ Please see [sidekiq.org](https://sidekiq.org/) for more details and how to buy.
6
+
7
+ 5.1.1
8
+ ---------
9
+
10
+ - Fix broken basic fetcher [#4616]
11
+
12
+ 5.1.0
13
+ ---------
14
+
15
+ - Remove old Statsd metrics with `WorkerName` in the name [#4377]
16
+ ```
17
+ job.WorkerName.count -> job.count with tag worker:WorkerName
18
+ job.WorkerName.perform -> job.perform with tag worker:WorkerName
19
+ job.WorkerName.failure -> job.failure with tag worker:WorkerName
20
+ ```
21
+ - Remove `concurrent-ruby` gem dependency [#4586]
22
+ - Update `constantize` for batch callbacks. [#4469]
23
+ - Add queue tag to `jobs.recovered.fetch` metric [#4594]
24
+ - Refactor Pro's fetch infrastructure [#4602]
25
+
26
+ 5.0.1
27
+ ---------
28
+
29
+ - Rejigger batch failures UI to add direct links to retries and scheduled jobs [#4209]
30
+ - Delete batch data with `UNLINK` [#4155]
31
+ - Fix bug where a scheduled job can lose its scheduled time when using reliable push [#4267]
32
+ - Sidekiq::JobSet#scan and #find_job APIs have been promoted to Sidekiq OSS. [#4259]
33
+
34
+ 5.0.0
35
+ ---------
36
+
37
+ - There is no significant migration from Sidekiq Pro 4.0 to 5.0
38
+ but make sure you read the [update notes for Sidekiq
39
+ 6.0](https://github.com/mperham/sidekiq/blob/master/6.0-Upgrade.md).
40
+ - Removed various deprecated APIs and associated warnings.
41
+ - **BREAKING CHANGE** Remove the `Sidekiq::Batch::Status#dead_jobs` API in favor of
42
+ `Sidekiq::Batch::Status#dead_jids`. [#4217]
43
+ - Update Sidekiq Pro codebase to use StandardRB formatting
44
+ - Fix lingering "b-XXX-died" elements in Redis which could cause
45
+ excessive memory usage. [#4217]
46
+ - Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
47
+
48
+ 4.0.5
49
+ ---------
50
+
51
+ - Increase super\_fetch retriever thread count from 1 to 2 to make it
52
+ less sensitive to Redis latency.
53
+ - Better handling of invalid job JSON by reliable scheduler [#4053]
54
+ - Added ZH, PT, JA and RU translations.
55
+
56
+ 4.0.4
57
+ ---------
58
+
59
+ - Update Sidekiq::Client patches to work with new Module#prepend
60
+ mechanism in Sidekiq 5.2.0. [#3930]
61
+
62
+ 4.0.3
63
+ ---------
64
+
65
+ - Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
66
+ - Implement batch death callback. This is fired the first time a job within a batch dies. [#3841]
67
+ ```ruby
68
+ batch = Sidekiq::Batch.new
69
+ batch.on(:death, ...)
70
+ ```
71
+
72
+ 4.0.2
73
+ ---------
74
+
75
+ - Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
76
+ call and resulting latency [#3790]
77
+ - Fix possible bad statsd metric call on super\_fetch startup
78
+ - Remove superfluous `freeze` calls on Strings [#3759]
79
+
80
+ 4.0.1
81
+ ---------
82
+
83
+ - Fix incompatibility with the statsd-ruby gem [#3740]
84
+ - Add tags to Statsd metrics when using Datadog [#3744]
85
+
86
+ 4.0.0
87
+ ---------
88
+
89
+ - See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
90
+
91
+
92
+ 3.7.1
93
+ ---------
94
+
95
+ - Deprecate timed\_fetch. Switch to super\_fetch:
96
+ ```ruby
97
+ config.super_fetch!
98
+ ```
99
+
100
+
101
+ 3.7.0
102
+ ---------
103
+
104
+ - Refactor batch job success/failure to gracefully handle several edge
105
+ cases with regard to Sidekiq::Shutdown. This should greatly reduce
106
+ the chances of seeing the long-standing "negative pending count" problem. [#3710]
107
+
108
+
109
+ 3.6.1
110
+ ---------
111
+
112
+ - Add support for Datadog::Statsd, it is the recommended Statsd client. [#3699]
113
+ ```ruby
114
+ Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
115
+ ```
116
+ - Size the statsd connection pool based on Sidekiq's concurrency [#3700]
117
+
118
+
119
+ 3.6.0
120
+ ---------
121
+
122
+ This release overhauls the Statsd metrics support and adds more
123
+ metrics for tracking Pro feature usage. In your initializer:
124
+ ```ruby
125
+ Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
126
+ ```
127
+ Sidekiq Pro will emit more metrics to Statsd:
128
+ ```
129
+ jobs.expired - when a job is expired
130
+ jobs.recovered.push - when a job is recovered by reliable_push after network outage
131
+ jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
132
+ batch.created - when a batch is created
133
+ batch.complete - when a batch is completed
134
+ batch.success - when a batch is successful
135
+ ```
136
+ Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
137
+ Everything should be backwards compatible with one deprecation notice.
138
+
139
+
140
+ 3.5.4
141
+ ---------
142
+
143
+ - Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
144
+ - Fix case where TimedFetch might not recover some pending jobs
145
+ - Fix edge case in Batch::Status#poll leading to premature completion [#3640]
146
+ - Adjust scan API to check 100 elements at a time, to minimize network round trips
147
+ when scanning large sets.
148
+
149
+ 3.5.3
150
+ ---------
151
+
152
+ - Restore error check for super\_fetch's job ack [#3601]
153
+ - Trim error messages saved in Batch's failure hash, preventing huge
154
+ messages from bloating Redis. [#3570]
155
+
156
+ 3.5.2
157
+ ---------
158
+
159
+ - Fix `Status#completed?` when run against a Batch that had succeeded
160
+ and was deleted. [#3519]
161
+
162
+ 3.5.1
163
+ ---------
164
+
165
+ - Work with Sidekiq 5.0.2+
166
+ - Improve performance of super\_fetch with weighted queues [#3489]
167
+
168
+ 3.5.0
169
+ ---------
170
+
171
+ - Add queue pause/unpause endpoints for scripting via curl [#3445]
172
+ - Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
173
+ - Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
174
+ - Batch page displays Pending JIDs if less than 10 [#3130]
175
+ - Batch page has a Search button to find associated Retries [#3130]
176
+ - Make Batch UI progress bar more friendly to the colorblind [#3387]
177
+
178
+ 3.4.5
179
+ ---------
180
+
181
+ - Fix potential job loss with reliable scheduler when lots of jobs are scheduled
182
+ at precisely the same time. Thanks to raivil for his hard work in
183
+ reproducing the bug. [#3371]
184
+
185
+ 3.4.4
186
+ ---------
187
+
188
+ - Optimize super\_fetch shutdown to restart jobs quicker [#3249]
189
+
190
+ 3.4.3
191
+ ---------
192
+
193
+ - Limit reliable scheduler to enqueue up to 100 jobs per call, minimizing Redis latency [#3332]
194
+ - Fix bug in super\_fetch logic for queues with `_` in the name [#3339]
195
+
196
+ 3.4.2
197
+ ---------
198
+
199
+ - Add `Batch::Status#invalidated?` API which returns true if any/all
200
+ JIDs were invalidated within the batch. [#3326]
201
+
202
+ 3.4.1
203
+ ---------
204
+
205
+ - Allow super\_fetch's orphan job check to happen as often as every hour [#3273]
206
+ - Officially deprecate reliable\_fetch algorithm, I now recommend you use `super_fetch` instead:
207
+ ```ruby
208
+ Sidekiq.configure_server do |config|
209
+ config.super_fetch!
210
+ end
211
+ ```
212
+ Also note that Sidekiq's `-i/--index` option is no longer used/relevant with super\_fetch.
213
+ - Don't display "Delete/Retry All" buttons when filtering in Web UI [#3243]
214
+ - Reimplement Sidekiq::JobSet#find\_job with ZSCAN [#3197]
215
+
216
+ 3.4.0
217
+ ---------
218
+
219
+ - Introducing the newest reliable fetching algorithm: `super_fetch`! This
220
+ algorithm will replace reliable\_fetch in Pro 4.0. super\_fetch is
221
+ bullet-proof across all environments, no longer requiring stable
222
+ hostnames or an index to be set per-process. [#3077]
223
+ ```ruby
224
+ Sidekiq.configure_server do |config|
225
+ config.super_fetch!
226
+ end
227
+ ```
228
+ Thank you to @jonhyman for code review and the Sidekiq Pro customers that
229
+ beta tested super\_fetch.
230
+
231
+ 3.3.3
232
+ ---------
233
+
234
+ - Update Web UI extension to work with Sidekiq 4.2.0's new Web UI. [#3075]
235
+
236
+ 3.3.2
237
+ ---------
238
+
239
+ - Minimize batch memory usage after success [#3083]
240
+ - Extract batch's 24 hr linger expiry to a LINGER constant so it can be tuned. [#3011]
241
+
242
+ 3.3.1
243
+ ---------
244
+
245
+ - If environment is unset, treat it as development so reliable\_fetch works as before 3.2.2.
246
+
247
+ 3.3.0
248
+ ---------
249
+
250
+ - Don't delete batches immediately upon success but set a 24 hr expiry, this allows
251
+ Sidekiq::Batch::Status#poll to work, even after batch success. [#3011]
252
+ - New `Sidekiq::PendingSet#destroy(jid)` API to remove poison pill jobs [#3015]
253
+
254
+ 3.2.2
255
+ ---------
256
+
257
+ - A default value for -i is only set in development now, staging or
258
+ other environments must set an index if you wish to use reliable\_fetch. [#2971]
259
+ - Fix nil dereference when checking for jobs over timeout in timed\_fetch
260
+
261
+
262
+ 3.2.1
263
+ ---------
264
+
265
+ - timed\_fetch now works with namespaces. [ryansch]
266
+
267
+
268
+ 3.2.0
269
+ ---------
270
+
271
+ - Fixed detection of missing batches, `NoSuchBatch` should be raised
272
+ properly now if `Sidekiq::Batch.new(bid)` is called on a batch no
273
+ longer in Redis.
274
+ - Remove support for Pro 1.x format batches. This version will no
275
+ longer seamlessly process batches created with Sidekiq Pro 1.x.
276
+ As always, upgrade one major version at a time to ensure a smooth
277
+ transition.
278
+ - Fix edge case where a parent batch could expire before a child batch
279
+ was finished processing, leading to missing batches [#2889]
280
+
281
+ 2.1.5
282
+ ---------
283
+
284
+ - Fix edge case where a parent batch could expire before a child batch
285
+ was finished processing, leading to missing batches [#2889]
286
+
287
+ 3.1.0
288
+ ---------
289
+
290
+ - New container-friendly fetch algorithm: `timed_fetch`. See the
291
+ [wiki documentation](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server)
292
+ for trade offs between the two reliability options. You should
293
+ use this if you are on Heroku, Docker, Amazon ECS or EBS or
294
+ another container-based system.
295
+
296
+
297
+ 3.0.6
298
+ ---------
299
+
300
+ - Fix race condition on reliable fetch shutdown
301
+
302
+ 3.0.5
303
+ ---------
304
+
305
+ - Statsd metrics now account for ActiveJob class names
306
+ - Allow reliable fetch internals to be overridden [jonhyman]
307
+
308
+ 3.0.4
309
+ ---------
310
+
311
+ - Queue pausing no longer requires reliable fetch. [#2786]
312
+
313
+ 3.0.3, 2.1.4
314
+ ------------
315
+
316
+ - Convert Lua-based `Sidekiq::Queue#delete_by_class` to Ruby-based, to
317
+ avoid O(N^2) performance and possible Redis failure. [#2806]
318
+
319
+ 3.0.2
320
+ -----------
321
+
322
+ - Make job registration with batch part of the atomic push so batch
323
+ metadata can't get out of sync with the job data. [#2714]
324
+
325
+ 3.0.1
326
+ -----------
327
+
328
+ - Remove a number of Redis version checks since we can assume 2.8+ now.
329
+ - Fix expiring jobs client middleware not loaded on server
330
+
331
+ 3.0.0
332
+ -----------
333
+
334
+ - See the [Pro 3.0 release notes](Pro-3.0-Upgrade.md).
335
+
336
+ 2.1.3
337
+ -----------
338
+
339
+ - Don't enable strict priority if using weighted queueing like `-q a,1 -q b,1`
340
+ - Safer JSON mangling in Lua [#2639]
341
+
342
+ 2.1.2
343
+ -----------
344
+
345
+ - Lock Sidekiq Pro 2.x to Sidekiq 3.x.
346
+
347
+ 2.1.1
348
+ -----------
349
+
350
+ - Make ShardSet lazier so Redis can first be initialized at startup. [#2603]
351
+
352
+
353
+ 2.1.0
354
+ -----------
355
+
356
+ - Explicit support for sharding batches. You list your Redis shards and
357
+ Sidekiq Pro will randomly spread batches across the shards. The BID
358
+ will indicate which shard contains the batch data. Jobs within a
359
+ batch may be spread across all shards too. [#2548, jonhyman]
360
+ - Officially deprecate Sidekiq::Notifications code. Notifications have
361
+ been undocumented for months now. [#2575]
362
+
363
+
364
+ 2.0.8
365
+ -----------
366
+
367
+ - Fix reliable scheduler mangling large numeric arguments. Lua's CJSON
368
+ library cannot accurately encode numbers larger than 14 digits! [#2478]
369
+
370
+ 2.0.7
371
+ -----------
372
+
373
+ - Optimize delete of enormous batches (100,000s of jobs) [#2458]
374
+
375
+ 2.0.6, 1.9.3
376
+ --------------
377
+
378
+ - CSRF protection in Sidekiq 3.4.2 broke job filtering in the Web UI [#2442]
379
+ - Sidekiq Pro 1.x is now limited to Sidekiq < 3.5.0.
380
+
381
+ 2.0.5
382
+ -----------
383
+
384
+ - Atomic scheduler now sets `enqueued_at` [#2414]
385
+ - Batches now account for jobs which are stopped by client middleware [#2406]
386
+ - Ignore redundant calls to `Sidekiq::Client.reliable_push!` [#2408]
387
+
388
+ 2.0.4
389
+ -----------
390
+
391
+ - Reliable push now supports sharding [#2409]
392
+ - Reliable push now only catches Redis exceptions [#2307]
393
+
394
+ 2.0.3
395
+ -----------
396
+
397
+ - Display Batch callback data on the Batch details page. [#2347]
398
+ - Fix incompatibility with Pro Web and Rack middleware. [#2344] Thank
399
+ you to Jason Clark for the tip on how to fix it.
400
+
401
+ 2.0.2
402
+ -----------
403
+
404
+ - Multiple Web UIs can now run in the same process. [#2267] If you have
405
+ multiple Redis shards, you can mount UIs for all in the same process:
406
+ ```ruby
407
+ POOL1 = ConnectionPool.new { Redis.new(:url => "redis://localhost:6379/0") }
408
+ POOL2 = ConnectionPool.new { Redis.new(:url => "redis://localhost:6378/0") }
409
+
410
+ mount Sidekiq::Pro::Web => '/sidekiq' # default
411
+ mount Sidekiq::Pro::Web.with(redis_pool: POOL1), at: '/sidekiq1', as: 'sidekiq1' # shard1
412
+ mount Sidekiq::Pro::Web.with(redis_pool: POOL2), at: '/sidekiq2', as: 'sidekiq2' # shard2
413
+ ```
414
+ - **SECURITY** Fix batch XSS in error data. Thanks to moneybird.com for
415
+ reporting the issue.
416
+
417
+ 2.0.1
418
+ -----------
419
+
420
+ - Add `batch.callback_queue` so batch callbacks can use a higher
421
+ priority queue than jobs. [#2200]
422
+ - Gracefully recover if someone runs `SCRIPT FLUSH` on Redis. [#2240]
423
+ - Ignore errors when attempting `bulk_requeue`, allowing clean shutdown
424
+
425
+ 2.0.0
426
+ -----------
427
+
428
+ - See [the Upgrade Notes](Pro-2.0-Upgrade.md) for detailed notes.
429
+
430
+ 1.9.2
431
+ -----------
432
+
433
+ - As of 1/1/2015, Sidekiq Pro is hosted on a new dedicated server.
434
+ Happy new year and let's hope for 100% uptime!
435
+ - Fix bug in reliable\_fetch where jobs could be duplicated if a Sidekiq
436
+ process crashed and you were using weighted queues. [#2120]
437
+
438
+ 1.9.1
439
+ -----------
440
+
441
+ - **SECURITY** Fix XSS in batch description, thanks to intercom.io for reporting the
442
+ issue. If you don't use batch descriptions, you don't need the fix.
443
+
444
+ 1.9.0
445
+ -----------
446
+
447
+ - Add new expiring jobs feature [#1982]
448
+ - Show batch expiration on Batch details page [#1981]
449
+ - Add '$' batch success token to the pubsub support. [#1953]
450
+
451
+
452
+ 1.8.0
453
+ -----------
454
+
455
+ - Fix race condition where Batches can complete
456
+ before they have been fully defined or only half-defined. Requires
457
+ Sidekiq 3.2.3. [#1919]
458
+
459
+
460
+ 1.7.6
461
+ -----------
462
+
463
+ - Quick release to verify #1919
464
+
465
+
466
+ 1.7.5
467
+ -----------
468
+
469
+ - Fix job filtering within the Dead tab.
470
+ - Add APIs and wiki documentation for invalidating jobs within a batch.
471
+
472
+
473
+ 1.7.4
474
+ -----------
475
+
476
+ - Awesome ANSI art startup banner!
477
+
478
+
479
+ 1.7.3
480
+ -----------
481
+
482
+ - Batch callbacks should use the same queue as the associated jobs.
483
+
484
+ 1.7.2
485
+ -----------
486
+
487
+ - **DEPRECATION** Use `Batch#on(:complete)` instead of `Batch#notify`.
488
+ The specific Campfire, HipChat, email and other notification schemes
489
+ will be removed in 2.0.0.
490
+ - Remove batch from UI when successful. [#1745]
491
+ - Convert batch callbacks to be asynchronous jobs for error handling [#1744]
492
+
493
+ 1.7.1
494
+ -----------
495
+
496
+ - Fix for paused queues being processed for a few seconds when starting
497
+ a new Sidekiq process.
498
+ - Add a 5 sec delay when starting reliable fetch on Heroku to minimize
499
+ any duplicate job processing with another process shutting down.
500
+
501
+ 1.7.0
502
+ -----------
503
+
504
+ - Add ability to pause reliable queues via API.
505
+ ```ruby
506
+ q = Sidekiq::Queue.new("critical")
507
+ q.pause!
508
+ q.paused? # => true
509
+ q.unpause!
510
+ ```
511
+
512
+ Sidekiq polls Redis every 10 seconds for paused queues so pausing will take
513
+ a few seconds to take effect.
514
+
515
+ 1.6.0
516
+ -----------
517
+
518
+ - Compatible with Sidekiq 3.
519
+
520
+ 1.5.1
521
+ -----------
522
+
523
+ - Due to a breaking API change in Sidekiq 3.0, this version is limited
524
+ to Sidekiq 2.x.
525
+
526
+ 1.5.0
527
+ -----------
528
+
529
+ - Fix issue on Heroku where reliable fetch could orphan jobs [#1573]
530
+
531
+
532
+ 1.4.3
533
+ -----------
534
+
535
+ - Reverse sorting of Batches in Web UI [#1098]
536
+ - Refactoring for Sidekiq 3.0, Pro now requires Sidekiq 2.17.5
537
+
538
+ 1.4.2
539
+ -----------
540
+
541
+ - Tolerate expired Batches in the web UI.
542
+ - Fix 100% CPU usage when using weighted queues and reliable fetch.
543
+
544
+ 1.4.1
545
+ -----------
546
+
547
+ - Add batch progress bar to batch detail page. [#1398]
548
+ - Fix race condition in initializing Lua scripts
549
+
550
+
551
+ 1.4.0
552
+ -----------
553
+
554
+ - Default batch expiration has been extended to 3 days, from 1 day previously.
555
+ - Batches now sort in the Web UI according to expiry time, not creation time.
556
+ - Add user-configurable batch expiry. If your batches might take longer
557
+ than 72 hours to process, you can extend the expiration date.
558
+
559
+ ```ruby
560
+ b = Sidekiq::Batch.new
561
+ b.expires_in 5.days
562
+ ...
563
+ ```
564
+
565
+ 1.3.2
566
+ -----------
567
+
568
+ - Lazy load Lua scripts so a Redis connection is not required on bootup.
569
+
570
+ 1.3.1
571
+ -----------
572
+
573
+ - Fix a gemspec packaging issue which broke the Batch UI.
574
+
575
+ 1.3.0
576
+ -----------
577
+
578
+ Thanks to @jonhyman for his contributions to this Sidekiq Pro release.
579
+
580
+ This release includes new functionality based on the SCAN command newly
581
+ added to Redis 2.8. Pro still works with Redis 2.4 but some
582
+ functionality will be unavailable.
583
+
584
+ - Job Filtering in the Web UI!
585
+ You can now filter retries and scheduled jobs in the Web UI so you
586
+ only see the jobs relevant to your needs. Queues cannot be filtered;
587
+ Redis does not provide the same SCAN operation on the LIST type.
588
+ **Redis 2.8**
589
+ ![Filtering](https://f.cloud.github.com/assets/2911/1619465/f47529f2-5657-11e3-8cd1-33899eb72aad.png)
590
+ - SCAN support in the Sidekiq::SortedSet API. Here's an example that
591
+ finds all jobs which contain the substring "Warehouse::OrderShip"
592
+ and deletes all matching retries. If the set is large, this API
593
+ will be **MUCH** faster than standard iteration using each.
594
+ **Redis 2.8**
595
+ ```ruby
596
+ Sidekiq::RetrySet.new.scan("Warehouse::OrderShip") do |job|
597
+ job.delete
598
+ end
599
+ ```
600
+
601
+ - Sidekiq::Batch#jobs now returns the set of JIDs added to the batch.
602
+ - Sidekiq::Batch#jids returns the complete set of JIDs associated with the batch.
603
+ - Sidekiq::Batch#remove\_jobs(jid, jid, ...) removes JIDs from the set, allowing early termination of jobs if they become irrelevant according to application logic.
604
+ - Sidekiq::Batch#include?(jid) allows jobs to check if they are still
605
+ relevant to a Batch and exit early if not.
606
+ - Sidekiq::SortedSet#find\_job(jid) now uses server-side Lua if possible **Redis 2.6** [jonhyman]
607
+ - The statsd integration now sets global job counts:
608
+ ```ruby
609
+ jobs.count
610
+ jobs.success
611
+ jobs.failure
612
+ ```
613
+
614
+ - Change shutdown logic to push leftover jobs in the private queue back
615
+ into the public queue when shutting down with Reliable Fetch. This
616
+ allows the safe decommission of a Sidekiq Pro process when autoscaling. [jonhyman]
617
+ - Add support for weighted random fetching with Reliable Fetch [jonhyman]
618
+ - Pro now requires Sidekiq 2.17.0
619
+
620
+ 1.2.5
621
+ -----------
622
+
623
+ - Convert Batch UI to use Sidekiq 2.16's support for extension localization.
624
+ - Update reliable\_push to work with Sidekiq::Client refactoring in 2.16
625
+ - Pro now requires Sidekiq 2.16.0
626
+
627
+ 1.2.4
628
+ -----------
629
+
630
+ - Convert Batch UI to Bootstrap 3
631
+ - Pro now requires Sidekiq 2.15.0
632
+ - Add Sidekiq::Batch::Status#delete [#1205]
633
+
634
+ 1.2.3
635
+ -----------
636
+
637
+ - Pro now requires Sidekiq 2.14.0
638
+ - Fix bad exception handling in batch callbacks [#1134]
639
+ - Convert Batch UI to ERB
640
+
641
+ 1.2.2
642
+ -----------
643
+
644
+ - Problem with reliable fetch which could lead to lost jobs when Sidekiq
645
+ is shut down normally. Thanks to MikaelAmborn for the report. [#1109]
646
+
647
+ 1.2.1
648
+ -----------
649
+
650
+ - Forgot to push paging code necessary for `delete_job` performance.
651
+
652
+ 1.2.0
653
+ -----------
654
+
655
+ - **LEAK** Fix batch key which didn't expire in Redis. Keys match
656
+ /b-[a-f0-9]{16}-pending/, e.g. "b-4f55163ddba10aa0-pending" [#1057]
657
+ - **Reliable fetch now supports multiple queues**, using the algorithm spec'd
658
+ by @jackrg [#1102]
659
+ - Fix issue with reliable\_push where it didn't return the JID for a pushed
660
+ job when sending previously cached jobs to Redis.
661
+ - Add fast Sidekiq::Queue#delete\_job(jid) API which leverages Lua so job lookup is
662
+ 100% server-side. Benchmark vs Sidekiq's Job#delete API. **Redis 2.6**
663
+
664
+ ```
665
+ Sidekiq Pro API
666
+ 0.030000 0.020000 0.050000 ( 1.640659)
667
+ Sidekiq API
668
+ 17.250000 2.220000 19.470000 ( 22.193300)
669
+ ```
670
+
671
+ - Add fast Sidekiq::Queue#delete\_by\_class(klass) API to remove all
672
+ jobs of a given type. Uses server-side Lua for performance. **Redis 2.6**
673
+
674
+ 1.1.0
675
+ -----------
676
+
677
+ - New `sidekiq/pro/reliable_push` which makes Sidekiq::Client resiliant
678
+ to Redis network failures. [#793]
679
+ - Move `sidekiq/reliable_fetch` to `sidekiq/pro/reliable_fetch`
680
+
681
+
682
+ 1.0.0
683
+ -----------
684
+
685
+ - Sidekiq Pro changelog moved to mperham/sidekiq for public visibility.
686
+ - Add new Rack endpoint for easy polling of batch status via JavaScript. See `sidekiq/rack/batch_status`
687
+
688
+ 0.9.3
689
+ -----------
690
+
691
+ - Fix bad /batches path in Web UI
692
+ - Fix Sinatra conflict with sidekiq-failures
693
+
694
+ 0.9.2
695
+ -----------
696
+
697
+ - Fix issue with lifecycle notifications not firing.
698
+
699
+ 0.9.1
700
+ -----------
701
+
702
+ - Update due to Sidekiq API changes.
703
+
704
+ 0.9.0
705
+ -----------
706
+
707
+ - Rearchitect Sidekiq's Fetch code to support different fetch
708
+ strategies. Add a ReliableFetch strategy which works with Redis'
709
+ RPOPLPUSH to ensure we don't lose messages, even when the Sidekiq
710
+ process crashes unexpectedly. [mperham/sidekiq#607]
711
+
712
+ 0.8.2
713
+ -----------
714
+
715
+ - Reimplement existing notifications using batch on_complete events.
716
+
717
+ 0.8.1
718
+ -----------
719
+
720
+ - Rejigger batch callback notifications.
721
+
722
+
723
+ 0.8.0
724
+ -----------
725
+
726
+ - Add new Batch 'callback' notification support, for in-process
727
+ notification.
728
+ - Symbolize option keys passed to Pony [mperham/sidekiq#603]
729
+ - Batch no longer requires the Web UI since Web UI usage is optional.
730
+ You must require is manually in your Web process:
731
+
732
+ ```ruby
733
+ require 'sidekiq/web'
734
+ require 'sidekiq/batch/web'
735
+ mount Sidekiq::Web => '/sidekiq'
736
+ ```
737
+
738
+
739
+ 0.7.1
740
+ -----------
741
+
742
+ - Worker instances can access the associated jid and bid via simple
743
+ accessors.
744
+ - Batches can now be modified while being processed so, e.g. a batch
745
+ job can add additional jobs to its own batch.
746
+
747
+ ```ruby
748
+ def perform(...)
749
+ batch = Sidekiq::Batch.new(bid) # instantiate batch associated with this job
750
+ batch.jobs do
751
+ SomeWorker.perform_async # add another job
752
+ end
753
+ end
754
+ ```
755
+
756
+ - Save error backtraces in batch's failure info for display in Web UI.
757
+ - Clean up email notification a bit.
758
+
759
+
760
+ 0.7.0
761
+ -----------
762
+
763
+ - Add optional batch description
764
+ - Mutable batches. Batches can now be modified to add additional jobs
765
+ at runtime. Example would be a batch job which needs to create more
766
+ jobs based on the data it is processing.
767
+
768
+ ```ruby
769
+ batch = Sidekiq::Batch.new(bid)
770
+ batch.jobs do
771
+ # define more jobs here
772
+ end
773
+ ```
774
+ - Fix issues with symbols vs strings in option hashes
775
+
776
+
777
+ 0.6.1
778
+ -----------
779
+
780
+ - Webhook notification support
781
+
782
+
783
+ 0.6
784
+ -----------
785
+
786
+ - Redis pubsub
787
+ - Email polish
788
+
789
+
790
+ 0.5
791
+ -----------
792
+
793
+ - Batches
794
+ - Notifications
795
+ - Statsd middleware