sidekiq_cleaner 5.3.6
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/.circleci/config.yml +61 -0
- data/.github/contributing.md +32 -0
- data/.github/issue_template.md +11 -0
- data/.gitignore +15 -0
- data/.travis.yml +11 -0
- data/3.0-Upgrade.md +70 -0
- data/4.0-Upgrade.md +53 -0
- data/5.0-Upgrade.md +56 -0
- data/COMM-LICENSE +97 -0
- data/Changes.md +1536 -0
- data/Ent-Changes.md +238 -0
- data/Gemfile +23 -0
- data/LICENSE +9 -0
- data/Pro-2.0-Upgrade.md +138 -0
- data/Pro-3.0-Upgrade.md +44 -0
- data/Pro-4.0-Upgrade.md +35 -0
- data/Pro-Changes.md +759 -0
- data/README.md +55 -0
- data/Rakefile +9 -0
- data/bin/sidekiq +18 -0
- data/bin/sidekiqctl +20 -0
- data/bin/sidekiqload +149 -0
- data/cleaner/assets/images/favicon.ico +0 -0
- data/cleaner/assets/images/logo.png +0 -0
- data/cleaner/assets/images/status.png +0 -0
- data/cleaner/assets/javascripts/application.js +172 -0
- data/cleaner/assets/javascripts/dashboard.js +315 -0
- data/cleaner/assets/stylesheets/application-rtl.css +246 -0
- data/cleaner/assets/stylesheets/application.css +1144 -0
- data/cleaner/assets/stylesheets/bootstrap-rtl.min.css +9 -0
- data/cleaner/assets/stylesheets/bootstrap.css +5 -0
- data/cleaner/locales/ar.yml +81 -0
- data/cleaner/locales/cs.yml +78 -0
- data/cleaner/locales/da.yml +68 -0
- data/cleaner/locales/de.yml +69 -0
- data/cleaner/locales/el.yml +68 -0
- data/cleaner/locales/en.yml +81 -0
- data/cleaner/locales/es.yml +70 -0
- data/cleaner/locales/fa.yml +80 -0
- data/cleaner/locales/fr.yml +78 -0
- data/cleaner/locales/he.yml +79 -0
- data/cleaner/locales/hi.yml +75 -0
- data/cleaner/locales/it.yml +69 -0
- data/cleaner/locales/ja.yml +80 -0
- data/cleaner/locales/ko.yml +68 -0
- data/cleaner/locales/nb.yml +77 -0
- data/cleaner/locales/nl.yml +68 -0
- data/cleaner/locales/pl.yml +59 -0
- data/cleaner/locales/pt-br.yml +68 -0
- data/cleaner/locales/pt.yml +67 -0
- data/cleaner/locales/ru.yml +78 -0
- data/cleaner/locales/sv.yml +68 -0
- data/cleaner/locales/ta.yml +75 -0
- data/cleaner/locales/uk.yml +76 -0
- data/cleaner/locales/ur.yml +80 -0
- data/cleaner/locales/zh-cn.yml +68 -0
- data/cleaner/locales/zh-tw.yml +68 -0
- data/cleaner/views/_footer.erb +20 -0
- data/cleaner/views/_job_info.erb +88 -0
- data/cleaner/views/_nav.erb +52 -0
- data/cleaner/views/_paging.erb +23 -0
- data/cleaner/views/_poll_link.erb +7 -0
- data/cleaner/views/_status.erb +4 -0
- data/cleaner/views/_summary.erb +40 -0
- data/cleaner/views/busy.erb +98 -0
- data/cleaner/views/dashboard.erb +75 -0
- data/cleaner/views/dead.erb +34 -0
- data/cleaner/views/errors.erb +84 -0
- data/cleaner/views/layout.erb +40 -0
- data/cleaner/views/morgue.erb +75 -0
- data/cleaner/views/queue.erb +46 -0
- data/cleaner/views/queues.erb +30 -0
- data/cleaner/views/retries.erb +80 -0
- data/cleaner/views/retry.erb +34 -0
- data/cleaner/views/scheduled.erb +54 -0
- data/cleaner/views/scheduled_job_info.erb +8 -0
- data/cleaner-stats.png +0 -0
- data/cleaner.png +0 -0
- data/code_of_conduct.md +50 -0
- data/lib/generators/sidekiq/templates/worker.rb.erb +9 -0
- data/lib/generators/sidekiq/templates/worker_spec.rb.erb +6 -0
- data/lib/generators/sidekiq/templates/worker_test.rb.erb +8 -0
- data/lib/generators/sidekiq/worker_generator.rb +49 -0
- data/lib/sidekiq/api.rb +940 -0
- data/lib/sidekiq/cleaner/action.rb +89 -0
- data/lib/sidekiq/cleaner/application.rb +385 -0
- data/lib/sidekiq/cleaner/helpers.rb +325 -0
- data/lib/sidekiq/cleaner/router.rb +100 -0
- data/lib/sidekiq/cleaner.rb +214 -0
- data/lib/sidekiq/cli.rb +445 -0
- data/lib/sidekiq/client.rb +243 -0
- data/lib/sidekiq/core_ext.rb +1 -0
- data/lib/sidekiq/ctl.rb +221 -0
- data/lib/sidekiq/delay.rb +42 -0
- data/lib/sidekiq/exception_handler.rb +29 -0
- data/lib/sidekiq/extensions/action_mailer.rb +57 -0
- data/lib/sidekiq/extensions/active_record.rb +40 -0
- data/lib/sidekiq/extensions/class_methods.rb +40 -0
- data/lib/sidekiq/extensions/generic_proxy.rb +31 -0
- data/lib/sidekiq/fetch.rb +81 -0
- data/lib/sidekiq/job_logger.rb +25 -0
- data/lib/sidekiq/job_retry.rb +262 -0
- data/lib/sidekiq/launcher.rb +173 -0
- data/lib/sidekiq/logging.rb +122 -0
- data/lib/sidekiq/manager.rb +137 -0
- data/lib/sidekiq/middleware/chain.rb +150 -0
- data/lib/sidekiq/middleware/i18n.rb +42 -0
- data/lib/sidekiq/middleware/server/active_record.rb +23 -0
- data/lib/sidekiq/paginator.rb +43 -0
- data/lib/sidekiq/processor.rb +279 -0
- data/lib/sidekiq/rails.rb +58 -0
- data/lib/sidekiq/redis_connection.rb +144 -0
- data/lib/sidekiq/scheduled.rb +174 -0
- data/lib/sidekiq/testing/inline.rb +29 -0
- data/lib/sidekiq/testing.rb +333 -0
- data/lib/sidekiq/util.rb +66 -0
- data/lib/sidekiq/version.rb +4 -0
- data/lib/sidekiq/worker.rb +220 -0
- data/lib/sidekiq.rb +237 -0
- data/sidekiq_cleaner.gemspec +21 -0
- metadata +235 -0
data/Pro-Changes.md
ADDED
@@ -0,0 +1,759 @@
|
|
1
|
+
# Sidekiq Pro Changelog
|
2
|
+
|
3
|
+
[Sidekiq Changes](https://github.com/mperham/sidekiq/blob/master/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/master/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/master/Ent-Changes.md)
|
4
|
+
|
5
|
+
Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
|
6
|
+
|
7
|
+
HEAD
|
8
|
+
---------
|
9
|
+
|
10
|
+
- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
|
11
|
+
|
12
|
+
4.0.5
|
13
|
+
---------
|
14
|
+
|
15
|
+
- Increase super\_fetch retriever thread count from 1 to 2 to make it
|
16
|
+
less sensitive to Redis latency.
|
17
|
+
- Better handling of invalid job JSON by reliable scheduler [#4053]
|
18
|
+
- Added ZH, PT, JA and RU translations.
|
19
|
+
|
20
|
+
4.0.4
|
21
|
+
---------
|
22
|
+
|
23
|
+
- Update Sidekiq::Client patches to work with new Module#prepend
|
24
|
+
mechanism in Sidekiq 5.2.0. [#3930]
|
25
|
+
|
26
|
+
4.0.3
|
27
|
+
---------
|
28
|
+
|
29
|
+
- Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
|
30
|
+
- Implement batch death callback. This is fired the first time a job within a batch dies. [#3841]
|
31
|
+
```ruby
|
32
|
+
batch = Sidekiq::Batch.new
|
33
|
+
batch.on(:death, ...)
|
34
|
+
```
|
35
|
+
|
36
|
+
4.0.2
|
37
|
+
---------
|
38
|
+
|
39
|
+
- Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
|
40
|
+
call and resulting latency [#3790]
|
41
|
+
- Fix possible bad statsd metric call on super\_fetch startup
|
42
|
+
- Remove superfluous `freeze` calls on Strings [#3759]
|
43
|
+
|
44
|
+
4.0.1
|
45
|
+
---------
|
46
|
+
|
47
|
+
- Fix incompatibility with the statsd-ruby gem [#3740]
|
48
|
+
- Add tags to Statsd metrics when using Datadog [#3744]
|
49
|
+
|
50
|
+
4.0.0
|
51
|
+
---------
|
52
|
+
|
53
|
+
- See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
|
54
|
+
|
55
|
+
|
56
|
+
3.7.1
|
57
|
+
---------
|
58
|
+
|
59
|
+
- Deprecate timed\_fetch. Switch to super\_fetch:
|
60
|
+
```ruby
|
61
|
+
config.super_fetch!
|
62
|
+
```
|
63
|
+
|
64
|
+
|
65
|
+
3.7.0
|
66
|
+
---------
|
67
|
+
|
68
|
+
- Refactor batch job success/failure to gracefully handle several edge
|
69
|
+
cases with regard to Sidekiq::Shutdown. This should greatly reduce
|
70
|
+
the chances of seeing the long-standing "negative pending count" problem. [#3710]
|
71
|
+
|
72
|
+
|
73
|
+
3.6.1
|
74
|
+
---------
|
75
|
+
|
76
|
+
- Add support for Datadog::Statsd, it is the recommended Statsd client. [#3699]
|
77
|
+
```ruby
|
78
|
+
Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
|
79
|
+
```
|
80
|
+
- Size the statsd connection pool based on Sidekiq's concurrency [#3700]
|
81
|
+
|
82
|
+
|
83
|
+
3.6.0
|
84
|
+
---------
|
85
|
+
|
86
|
+
This release overhauls the Statsd metrics support and adds more
|
87
|
+
metrics for tracking Pro feature usage. In your initializer:
|
88
|
+
```ruby
|
89
|
+
Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
|
90
|
+
```
|
91
|
+
Sidekiq Pro will emit more metrics to Statsd:
|
92
|
+
```
|
93
|
+
jobs.expired - when a job is expired
|
94
|
+
jobs.recovered.push - when a job is recovered by reliable_push after network outage
|
95
|
+
jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
|
96
|
+
batch.created - when a batch is created
|
97
|
+
batch.complete - when a batch is completed
|
98
|
+
batch.success - when a batch is successful
|
99
|
+
```
|
100
|
+
Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
|
101
|
+
Everything should be backwards compatible with one deprecation notice.
|
102
|
+
|
103
|
+
|
104
|
+
3.5.4
|
105
|
+
---------
|
106
|
+
|
107
|
+
- Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
|
108
|
+
- Fix case where TimedFetch might not recover some pending jobs
|
109
|
+
- Fix edge case in Batch::Status#poll leading to premature completion [#3640]
|
110
|
+
- Adjust scan API to check 100 elements at a time, to minimize network round trips
|
111
|
+
when scanning large sets.
|
112
|
+
|
113
|
+
3.5.3
|
114
|
+
---------
|
115
|
+
|
116
|
+
- Restore error check for super\_fetch's job ack [#3601]
|
117
|
+
- Trim error messages saved in Batch's failure hash, preventing huge
|
118
|
+
messages from bloating Redis. [#3570]
|
119
|
+
|
120
|
+
3.5.2
|
121
|
+
---------
|
122
|
+
|
123
|
+
- Fix `Status#completed?` when run against a Batch that had succeeded
|
124
|
+
and was deleted. [#3519]
|
125
|
+
|
126
|
+
3.5.1
|
127
|
+
---------
|
128
|
+
|
129
|
+
- Work with Sidekiq 5.0.2+
|
130
|
+
- Improve performance of super\_fetch with weighted queues [#3489]
|
131
|
+
|
132
|
+
3.5.0
|
133
|
+
---------
|
134
|
+
|
135
|
+
- Add queue pause/unpause endpoints for scripting via curl [#3445]
|
136
|
+
- Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
|
137
|
+
- Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
|
138
|
+
- Batch page displays Pending JIDs if less than 10 [#3130]
|
139
|
+
- Batch page has a Search button to find associated Retries [#3130]
|
140
|
+
- Make Batch UI progress bar more friendly to the colorblind [#3387]
|
141
|
+
|
142
|
+
3.4.5
|
143
|
+
---------
|
144
|
+
|
145
|
+
- Fix potential job loss with reliable scheduler when lots of jobs are scheduled
|
146
|
+
at precisely the same time. Thanks to raivil for his hard work in
|
147
|
+
reproducing the bug. [#3371]
|
148
|
+
|
149
|
+
3.4.4
|
150
|
+
---------
|
151
|
+
|
152
|
+
- Optimize super\_fetch shutdown to restart jobs quicker [#3249]
|
153
|
+
|
154
|
+
3.4.3
|
155
|
+
---------
|
156
|
+
|
157
|
+
- Limit reliable scheduler to enqueue up to 100 jobs per call, minimizing Redis latency [#3332]
|
158
|
+
- Fix bug in super\_fetch logic for queues with `_` in the name [#3339]
|
159
|
+
|
160
|
+
3.4.2
|
161
|
+
---------
|
162
|
+
|
163
|
+
- Add `Batch::Status#invalidated?` API which returns true if any/all
|
164
|
+
JIDs were invalidated within the batch. [#3326]
|
165
|
+
|
166
|
+
3.4.1
|
167
|
+
---------
|
168
|
+
|
169
|
+
- Allow super\_fetch's orphan job check to happen as often as every hour [#3273]
|
170
|
+
- Officially deprecate reliable\_fetch algorithm, I now recommend you use `super_fetch` instead:
|
171
|
+
```ruby
|
172
|
+
Sidekiq.configure_server do |config|
|
173
|
+
config.super_fetch!
|
174
|
+
end
|
175
|
+
```
|
176
|
+
Also note that Sidekiq's `-i/--index` option is no longer used/relevant with super\_fetch.
|
177
|
+
- Don't display "Delete/Retry All" buttons when filtering in Web UI [#3243]
|
178
|
+
- Reimplement Sidekiq::JobSet#find\_job with ZSCAN [#3197]
|
179
|
+
|
180
|
+
3.4.0
|
181
|
+
---------
|
182
|
+
|
183
|
+
- Introducing the newest reliable fetching algorithm: `super_fetch`! This
|
184
|
+
algorithm will replace reliable\_fetch in Pro 4.0. super\_fetch is
|
185
|
+
bullet-proof across all environments, no longer requiring stable
|
186
|
+
hostnames or an index to be set per-process. [#3077]
|
187
|
+
```ruby
|
188
|
+
Sidekiq.configure_server do |config|
|
189
|
+
config.super_fetch!
|
190
|
+
end
|
191
|
+
```
|
192
|
+
Thank you to @jonhyman for code review and the Sidekiq Pro customers that
|
193
|
+
beta tested super\_fetch.
|
194
|
+
|
195
|
+
3.3.3
|
196
|
+
---------
|
197
|
+
|
198
|
+
- Update Web UI extension to work with Sidekiq 4.2.0's new Web UI. [#3075]
|
199
|
+
|
200
|
+
3.3.2
|
201
|
+
---------
|
202
|
+
|
203
|
+
- Minimize batch memory usage after success [#3083]
|
204
|
+
- Extract batch's 24 hr linger expiry to a LINGER constant so it can be tuned. [#3011]
|
205
|
+
|
206
|
+
3.3.1
|
207
|
+
---------
|
208
|
+
|
209
|
+
- If environment is unset, treat it as development so reliable\_fetch works as before 3.2.2.
|
210
|
+
|
211
|
+
3.3.0
|
212
|
+
---------
|
213
|
+
|
214
|
+
- Don't delete batches immediately upon success but set a 24 hr expiry, this allows
|
215
|
+
Sidekiq::Batch::Status#poll to work, even after batch success. [#3011]
|
216
|
+
- New `Sidekiq::PendingSet#destroy(jid)` API to remove poison pill jobs [#3015]
|
217
|
+
|
218
|
+
3.2.2
|
219
|
+
---------
|
220
|
+
|
221
|
+
- A default value for -i is only set in development now, staging or
|
222
|
+
other environments must set an index if you wish to use reliable\_fetch. [#2971]
|
223
|
+
- Fix nil dereference when checking for jobs over timeout in timed\_fetch
|
224
|
+
|
225
|
+
|
226
|
+
3.2.1
|
227
|
+
---------
|
228
|
+
|
229
|
+
- timed\_fetch now works with namespaces. [ryansch]
|
230
|
+
|
231
|
+
|
232
|
+
3.2.0
|
233
|
+
---------
|
234
|
+
|
235
|
+
- Fixed detection of missing batches, `NoSuchBatch` should be raised
|
236
|
+
properly now if `Sidekiq::Batch.new(bid)` is called on a batch no
|
237
|
+
longer in Redis.
|
238
|
+
- Remove support for Pro 1.x format batches. This version will no
|
239
|
+
longer seamlessly process batches created with Sidekiq Pro 1.x.
|
240
|
+
As always, upgrade one major version at a time to ensure a smooth
|
241
|
+
transition.
|
242
|
+
- Fix edge case where a parent batch could expire before a child batch
|
243
|
+
was finished processing, leading to missing batches [#2889]
|
244
|
+
|
245
|
+
2.1.5
|
246
|
+
---------
|
247
|
+
|
248
|
+
- Fix edge case where a parent batch could expire before a child batch
|
249
|
+
was finished processing, leading to missing batches [#2889]
|
250
|
+
|
251
|
+
3.1.0
|
252
|
+
---------
|
253
|
+
|
254
|
+
- New container-friendly fetch algorithm: `timed_fetch`. See the
|
255
|
+
[wiki documentation](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server)
|
256
|
+
for trade offs between the two reliability options. You should
|
257
|
+
use this if you are on Heroku, Docker, Amazon ECS or EBS or
|
258
|
+
another container-based system.
|
259
|
+
|
260
|
+
|
261
|
+
3.0.6
|
262
|
+
---------
|
263
|
+
|
264
|
+
- Fix race condition on reliable fetch shutdown
|
265
|
+
|
266
|
+
3.0.5
|
267
|
+
---------
|
268
|
+
|
269
|
+
- Statsd metrics now account for ActiveJob class names
|
270
|
+
- Allow reliable fetch internals to be overridden [jonhyman]
|
271
|
+
|
272
|
+
3.0.4
|
273
|
+
---------
|
274
|
+
|
275
|
+
- Queue pausing no longer requires reliable fetch. [#2786]
|
276
|
+
|
277
|
+
3.0.3, 2.1.4
|
278
|
+
------------
|
279
|
+
|
280
|
+
- Convert Lua-based `Sidekiq::Queue#delete_by_class` to Ruby-based, to
|
281
|
+
avoid O(N^2) performance and possible Redis failure. [#2806]
|
282
|
+
|
283
|
+
3.0.2
|
284
|
+
-----------
|
285
|
+
|
286
|
+
- Make job registration with batch part of the atomic push so batch
|
287
|
+
metadata can't get out of sync with the job data. [#2714]
|
288
|
+
|
289
|
+
3.0.1
|
290
|
+
-----------
|
291
|
+
|
292
|
+
- Remove a number of Redis version checks since we can assume 2.8+ now.
|
293
|
+
- Fix expiring jobs client middleware not loaded on server
|
294
|
+
|
295
|
+
3.0.0
|
296
|
+
-----------
|
297
|
+
|
298
|
+
- See the [Pro 3.0 release notes](Pro-3.0-Upgrade.md).
|
299
|
+
|
300
|
+
2.1.3
|
301
|
+
-----------
|
302
|
+
|
303
|
+
- Don't enable strict priority if using weighted queueing like `-q a,1 -q b,1`
|
304
|
+
- Safer JSON mangling in Lua [#2639]
|
305
|
+
|
306
|
+
2.1.2
|
307
|
+
-----------
|
308
|
+
|
309
|
+
- Lock Sidekiq Pro 2.x to Sidekiq 3.x.
|
310
|
+
|
311
|
+
2.1.1
|
312
|
+
-----------
|
313
|
+
|
314
|
+
- Make ShardSet lazier so Redis can first be initialized at startup. [#2603]
|
315
|
+
|
316
|
+
|
317
|
+
2.1.0
|
318
|
+
-----------
|
319
|
+
|
320
|
+
- Explicit support for sharding batches. You list your Redis shards and
|
321
|
+
Sidekiq Pro will randomly spread batches across the shards. The BID
|
322
|
+
will indicate which shard contains the batch data. Jobs within a
|
323
|
+
batch may be spread across all shards too. [#2548, jonhyman]
|
324
|
+
- Officially deprecate Sidekiq::Notifications code. Notifications have
|
325
|
+
been undocumented for months now. [#2575]
|
326
|
+
|
327
|
+
|
328
|
+
2.0.8
|
329
|
+
-----------
|
330
|
+
|
331
|
+
- Fix reliable scheduler mangling large numeric arguments. Lua's CJSON
|
332
|
+
library cannot accurately encode numbers larger than 14 digits! [#2478]
|
333
|
+
|
334
|
+
2.0.7
|
335
|
+
-----------
|
336
|
+
|
337
|
+
- Optimize delete of enormous batches (100,000s of jobs) [#2458]
|
338
|
+
|
339
|
+
2.0.6, 1.9.3
|
340
|
+
--------------
|
341
|
+
|
342
|
+
- CSRF protection in Sidekiq 3.4.2 broke job filtering in the Web UI [#2442]
|
343
|
+
- Sidekiq Pro 1.x is now limited to Sidekiq < 3.5.0.
|
344
|
+
|
345
|
+
2.0.5
|
346
|
+
-----------
|
347
|
+
|
348
|
+
- Atomic scheduler now sets `enqueued_at` [#2414]
|
349
|
+
- Batches now account for jobs which are stopped by client middleware [#2406]
|
350
|
+
- Ignore redundant calls to `Sidekiq::Client.reliable_push!` [#2408]
|
351
|
+
|
352
|
+
2.0.4
|
353
|
+
-----------
|
354
|
+
|
355
|
+
- Reliable push now supports sharding [#2409]
|
356
|
+
- Reliable push now only catches Redis exceptions [#2307]
|
357
|
+
|
358
|
+
2.0.3
|
359
|
+
-----------
|
360
|
+
|
361
|
+
- Display Batch callback data on the Batch details page. [#2347]
|
362
|
+
- Fix incompatibility with Pro Web and Rack middleware. [#2344] Thank
|
363
|
+
you to Jason Clark for the tip on how to fix it.
|
364
|
+
|
365
|
+
2.0.2
|
366
|
+
-----------
|
367
|
+
|
368
|
+
- Multiple Web UIs can now run in the same process. [#2267] If you have
|
369
|
+
multiple Redis shards, you can mount UIs for all in the same process:
|
370
|
+
```ruby
|
371
|
+
POOL1 = ConnectionPool.new { Redis.new(:url => "redis://localhost:6379/0") }
|
372
|
+
POOL2 = ConnectionPool.new { Redis.new(:url => "redis://localhost:6378/0") }
|
373
|
+
|
374
|
+
mount Sidekiq::Pro::Web => '/sidekiq' # default
|
375
|
+
mount Sidekiq::Pro::Web.with(redis_pool: POOL1), at: '/sidekiq1', as: 'sidekiq1' # shard1
|
376
|
+
mount Sidekiq::Pro::Web.with(redis_pool: POOL2), at: '/sidekiq2', as: 'sidekiq2' # shard2
|
377
|
+
```
|
378
|
+
- **SECURITY** Fix batch XSS in error data. Thanks to moneybird.com for
|
379
|
+
reporting the issue.
|
380
|
+
|
381
|
+
2.0.1
|
382
|
+
-----------
|
383
|
+
|
384
|
+
- Add `batch.callback_queue` so batch callbacks can use a higher
|
385
|
+
priority queue than jobs. [#2200]
|
386
|
+
- Gracefully recover if someone runs `SCRIPT FLUSH` on Redis. [#2240]
|
387
|
+
- Ignore errors when attempting `bulk_requeue`, allowing clean shutdown
|
388
|
+
|
389
|
+
2.0.0
|
390
|
+
-----------
|
391
|
+
|
392
|
+
- See [the Upgrade Notes](Pro-2.0-Upgrade.md) for detailed notes.
|
393
|
+
|
394
|
+
1.9.2
|
395
|
+
-----------
|
396
|
+
|
397
|
+
- As of 1/1/2015, Sidekiq Pro is hosted on a new dedicated server.
|
398
|
+
Happy new year and let's hope for 100% uptime!
|
399
|
+
- Fix bug in reliable\_fetch where jobs could be duplicated if a Sidekiq
|
400
|
+
process crashed and you were using weighted queues. [#2120]
|
401
|
+
|
402
|
+
1.9.1
|
403
|
+
-----------
|
404
|
+
|
405
|
+
- **SECURITY** Fix XSS in batch description, thanks to intercom.io for reporting the
|
406
|
+
issue. If you don't use batch descriptions, you don't need the fix.
|
407
|
+
|
408
|
+
1.9.0
|
409
|
+
-----------
|
410
|
+
|
411
|
+
- Add new expiring jobs feature [#1982]
|
412
|
+
- Show batch expiration on Batch details page [#1981]
|
413
|
+
- Add '$' batch success token to the pubsub support. [#1953]
|
414
|
+
|
415
|
+
|
416
|
+
1.8.0
|
417
|
+
-----------
|
418
|
+
|
419
|
+
- Fix race condition where Batches can complete
|
420
|
+
before they have been fully defined or only half-defined. Requires
|
421
|
+
Sidekiq 3.2.3. [#1919]
|
422
|
+
|
423
|
+
|
424
|
+
1.7.6
|
425
|
+
-----------
|
426
|
+
|
427
|
+
- Quick release to verify #1919
|
428
|
+
|
429
|
+
|
430
|
+
1.7.5
|
431
|
+
-----------
|
432
|
+
|
433
|
+
- Fix job filtering within the Dead tab.
|
434
|
+
- Add APIs and wiki documentation for invalidating jobs within a batch.
|
435
|
+
|
436
|
+
|
437
|
+
1.7.4
|
438
|
+
-----------
|
439
|
+
|
440
|
+
- Awesome ANSI art startup banner!
|
441
|
+
|
442
|
+
|
443
|
+
1.7.3
|
444
|
+
-----------
|
445
|
+
|
446
|
+
- Batch callbacks should use the same queue as the associated jobs.
|
447
|
+
|
448
|
+
1.7.2
|
449
|
+
-----------
|
450
|
+
|
451
|
+
- **DEPRECATION** Use `Batch#on(:complete)` instead of `Batch#notify`.
|
452
|
+
The specific Campfire, HipChat, email and other notification schemes
|
453
|
+
will be removed in 2.0.0.
|
454
|
+
- Remove batch from UI when successful. [#1745]
|
455
|
+
- Convert batch callbacks to be asynchronous jobs for error handling [#1744]
|
456
|
+
|
457
|
+
1.7.1
|
458
|
+
-----------
|
459
|
+
|
460
|
+
- Fix for paused queues being processed for a few seconds when starting
|
461
|
+
a new Sidekiq process.
|
462
|
+
- Add a 5 sec delay when starting reliable fetch on Heroku to minimize
|
463
|
+
any duplicate job processing with another process shutting down.
|
464
|
+
|
465
|
+
1.7.0
|
466
|
+
-----------
|
467
|
+
|
468
|
+
- Add ability to pause reliable queues via API.
|
469
|
+
```ruby
|
470
|
+
q = Sidekiq::Queue.new("critical")
|
471
|
+
q.pause!
|
472
|
+
q.paused? # => true
|
473
|
+
q.unpause!
|
474
|
+
```
|
475
|
+
|
476
|
+
Sidekiq polls Redis every 10 seconds for paused queues so pausing will take
|
477
|
+
a few seconds to take effect.
|
478
|
+
|
479
|
+
1.6.0
|
480
|
+
-----------
|
481
|
+
|
482
|
+
- Compatible with Sidekiq 3.
|
483
|
+
|
484
|
+
1.5.1
|
485
|
+
-----------
|
486
|
+
|
487
|
+
- Due to a breaking API change in Sidekiq 3.0, this version is limited
|
488
|
+
to Sidekiq 2.x.
|
489
|
+
|
490
|
+
1.5.0
|
491
|
+
-----------
|
492
|
+
|
493
|
+
- Fix issue on Heroku where reliable fetch could orphan jobs [#1573]
|
494
|
+
|
495
|
+
|
496
|
+
1.4.3
|
497
|
+
-----------
|
498
|
+
|
499
|
+
- Reverse sorting of Batches in Web UI [#1098]
|
500
|
+
- Refactoring for Sidekiq 3.0, Pro now requires Sidekiq 2.17.5
|
501
|
+
|
502
|
+
1.4.2
|
503
|
+
-----------
|
504
|
+
|
505
|
+
- Tolerate expired Batches in the web UI.
|
506
|
+
- Fix 100% CPU usage when using weighted queues and reliable fetch.
|
507
|
+
|
508
|
+
1.4.1
|
509
|
+
-----------
|
510
|
+
|
511
|
+
- Add batch progress bar to batch detail page. [#1398]
|
512
|
+
- Fix race condition in initializing Lua scripts
|
513
|
+
|
514
|
+
|
515
|
+
1.4.0
|
516
|
+
-----------
|
517
|
+
|
518
|
+
- Default batch expiration has been extended to 3 days, from 1 day previously.
|
519
|
+
- Batches now sort in the Web UI according to expiry time, not creation time.
|
520
|
+
- Add user-configurable batch expiry. If your batches might take longer
|
521
|
+
than 72 hours to process, you can extend the expiration date.
|
522
|
+
|
523
|
+
```ruby
|
524
|
+
b = Sidekiq::Batch.new
|
525
|
+
b.expires_in 5.days
|
526
|
+
...
|
527
|
+
```
|
528
|
+
|
529
|
+
1.3.2
|
530
|
+
-----------
|
531
|
+
|
532
|
+
- Lazy load Lua scripts so a Redis connection is not required on bootup.
|
533
|
+
|
534
|
+
1.3.1
|
535
|
+
-----------
|
536
|
+
|
537
|
+
- Fix a gemspec packaging issue which broke the Batch UI.
|
538
|
+
|
539
|
+
1.3.0
|
540
|
+
-----------
|
541
|
+
|
542
|
+
Thanks to @jonhyman for his contributions to this Sidekiq Pro release.
|
543
|
+
|
544
|
+
This release includes new functionality based on the SCAN command newly
|
545
|
+
added to Redis 2.8. Pro still works with Redis 2.4 but some
|
546
|
+
functionality will be unavailable.
|
547
|
+
|
548
|
+
- Job Filtering in the Web UI!
|
549
|
+
You can now filter retries and scheduled jobs in the Web UI so you
|
550
|
+
only see the jobs relevant to your needs. Queues cannot be filtered;
|
551
|
+
Redis does not provide the same SCAN operation on the LIST type.
|
552
|
+
**Redis 2.8**
|
553
|
+
![Filtering](https://f.cloud.github.com/assets/2911/1619465/f47529f2-5657-11e3-8cd1-33899eb72aad.png)
|
554
|
+
- SCAN support in the Sidekiq::SortedSet API. Here's an example that
|
555
|
+
finds all jobs which contain the substring "Warehouse::OrderShip"
|
556
|
+
and deletes all matching retries. If the set is large, this API
|
557
|
+
will be **MUCH** faster than standard iteration using each.
|
558
|
+
**Redis 2.8**
|
559
|
+
```ruby
|
560
|
+
Sidekiq::RetrySet.new.scan("Warehouse::OrderShip") do |job|
|
561
|
+
job.delete
|
562
|
+
end
|
563
|
+
```
|
564
|
+
|
565
|
+
- Sidekiq::Batch#jobs now returns the set of JIDs added to the batch.
|
566
|
+
- Sidekiq::Batch#jids returns the complete set of JIDs associated with the batch.
|
567
|
+
- Sidekiq::Batch#remove\_jobs(jid, jid, ...) removes JIDs from the set, allowing early termination of jobs if they become irrelevant according to application logic.
|
568
|
+
- Sidekiq::Batch#include?(jid) allows jobs to check if they are still
|
569
|
+
relevant to a Batch and exit early if not.
|
570
|
+
- Sidekiq::SortedSet#find\_job(jid) now uses server-side Lua if possible **Redis 2.6** [jonhyman]
|
571
|
+
- The statsd integration now sets global job counts:
|
572
|
+
```ruby
|
573
|
+
jobs.count
|
574
|
+
jobs.success
|
575
|
+
jobs.failure
|
576
|
+
```
|
577
|
+
|
578
|
+
- Change shutdown logic to push leftover jobs in the private queue back
|
579
|
+
into the public queue when shutting down with Reliable Fetch. This
|
580
|
+
allows the safe decommission of a Sidekiq Pro process when autoscaling. [jonhyman]
|
581
|
+
- Add support for weighted random fetching with Reliable Fetch [jonhyman]
|
582
|
+
- Pro now requires Sidekiq 2.17.0
|
583
|
+
|
584
|
+
1.2.5
|
585
|
+
-----------
|
586
|
+
|
587
|
+
- Convert Batch UI to use Sidekiq 2.16's support for extension localization.
|
588
|
+
- Update reliable\_push to work with Sidekiq::Client refactoring in 2.16
|
589
|
+
- Pro now requires Sidekiq 2.16.0
|
590
|
+
|
591
|
+
1.2.4
|
592
|
+
-----------
|
593
|
+
|
594
|
+
- Convert Batch UI to Bootstrap 3
|
595
|
+
- Pro now requires Sidekiq 2.15.0
|
596
|
+
- Add Sidekiq::Batch::Status#delete [#1205]
|
597
|
+
|
598
|
+
1.2.3
|
599
|
+
-----------
|
600
|
+
|
601
|
+
- Pro now requires Sidekiq 2.14.0
|
602
|
+
- Fix bad exception handling in batch callbacks [#1134]
|
603
|
+
- Convert Batch UI to ERB
|
604
|
+
|
605
|
+
1.2.2
|
606
|
+
-----------
|
607
|
+
|
608
|
+
- Problem with reliable fetch which could lead to lost jobs when Sidekiq
|
609
|
+
is shut down normally. Thanks to MikaelAmborn for the report. [#1109]
|
610
|
+
|
611
|
+
1.2.1
|
612
|
+
-----------
|
613
|
+
|
614
|
+
- Forgot to push paging code necessary for `delete_job` performance.
|
615
|
+
|
616
|
+
1.2.0
|
617
|
+
-----------
|
618
|
+
|
619
|
+
- **LEAK** Fix batch key which didn't expire in Redis. Keys match
|
620
|
+
/b-[a-f0-9]{16}-pending/, e.g. "b-4f55163ddba10aa0-pending" [#1057]
|
621
|
+
- **Reliable fetch now supports multiple queues**, using the algorithm spec'd
|
622
|
+
by @jackrg [#1102]
|
623
|
+
- Fix issue with reliable\_push where it didn't return the JID for a pushed
|
624
|
+
job when sending previously cached jobs to Redis.
|
625
|
+
- Add fast Sidekiq::Queue#delete\_job(jid) API which leverages Lua so job lookup is
|
626
|
+
100% server-side. Benchmark vs Sidekiq's Job#delete API. **Redis 2.6**
|
627
|
+
|
628
|
+
```
|
629
|
+
Sidekiq Pro API
|
630
|
+
0.030000 0.020000 0.050000 ( 1.640659)
|
631
|
+
Sidekiq API
|
632
|
+
17.250000 2.220000 19.470000 ( 22.193300)
|
633
|
+
```
|
634
|
+
|
635
|
+
- Add fast Sidekiq::Queue#delete\_by\_class(klass) API to remove all
|
636
|
+
jobs of a given type. Uses server-side Lua for performance. **Redis 2.6**
|
637
|
+
|
638
|
+
1.1.0
|
639
|
+
-----------
|
640
|
+
|
641
|
+
- New `sidekiq/pro/reliable_push` which makes Sidekiq::Client resiliant
|
642
|
+
to Redis network failures. [#793]
|
643
|
+
- Move `sidekiq/reliable_fetch` to `sidekiq/pro/reliable_fetch`
|
644
|
+
|
645
|
+
|
646
|
+
1.0.0
|
647
|
+
-----------
|
648
|
+
|
649
|
+
- Sidekiq Pro changelog moved to mperham/sidekiq for public visibility.
|
650
|
+
- Add new Rack endpoint for easy polling of batch status via JavaScript. See `sidekiq/rack/batch_status`
|
651
|
+
|
652
|
+
0.9.3
|
653
|
+
-----------
|
654
|
+
|
655
|
+
- Fix bad /batches path in Web UI
|
656
|
+
- Fix Sinatra conflict with sidekiq-failures
|
657
|
+
|
658
|
+
0.9.2
|
659
|
+
-----------
|
660
|
+
|
661
|
+
- Fix issue with lifecycle notifications not firing.
|
662
|
+
|
663
|
+
0.9.1
|
664
|
+
-----------
|
665
|
+
|
666
|
+
- Update due to Sidekiq API changes.
|
667
|
+
|
668
|
+
0.9.0
|
669
|
+
-----------
|
670
|
+
|
671
|
+
- Rearchitect Sidekiq's Fetch code to support different fetch
|
672
|
+
strategies. Add a ReliableFetch strategy which works with Redis'
|
673
|
+
RPOPLPUSH to ensure we don't lose messages, even when the Sidekiq
|
674
|
+
process crashes unexpectedly. [mperham/sidekiq#607]
|
675
|
+
|
676
|
+
0.8.2
|
677
|
+
-----------
|
678
|
+
|
679
|
+
- Reimplement existing notifications using batch on_complete events.
|
680
|
+
|
681
|
+
0.8.1
|
682
|
+
-----------
|
683
|
+
|
684
|
+
- Rejigger batch callback notifications.
|
685
|
+
|
686
|
+
|
687
|
+
0.8.0
|
688
|
+
-----------
|
689
|
+
|
690
|
+
- Add new Batch 'callback' notification support, for in-process
|
691
|
+
notification.
|
692
|
+
- Symbolize option keys passed to Pony [mperham/sidekiq#603]
|
693
|
+
- Batch no longer requires the Web UI since Web UI usage is optional.
|
694
|
+
You must require is manually in your Web process:
|
695
|
+
|
696
|
+
```ruby
|
697
|
+
require 'sidekiq/web'
|
698
|
+
require 'sidekiq/batch/web'
|
699
|
+
mount Sidekiq::Web => '/sidekiq'
|
700
|
+
```
|
701
|
+
|
702
|
+
|
703
|
+
0.7.1
|
704
|
+
-----------
|
705
|
+
|
706
|
+
- Worker instances can access the associated jid and bid via simple
|
707
|
+
accessors.
|
708
|
+
- Batches can now be modified while being processed so, e.g. a batch
|
709
|
+
job can add additional jobs to its own batch.
|
710
|
+
|
711
|
+
```ruby
|
712
|
+
def perform(...)
|
713
|
+
batch = Sidekiq::Batch.new(bid) # instantiate batch associated with this job
|
714
|
+
batch.jobs do
|
715
|
+
SomeWorker.perform_async # add another job
|
716
|
+
end
|
717
|
+
end
|
718
|
+
```
|
719
|
+
|
720
|
+
- Save error backtraces in batch's failure info for display in Web UI.
|
721
|
+
- Clean up email notification a bit.
|
722
|
+
|
723
|
+
|
724
|
+
0.7.0
|
725
|
+
-----------
|
726
|
+
|
727
|
+
- Add optional batch description
|
728
|
+
- Mutable batches. Batches can now be modified to add additional jobs
|
729
|
+
at runtime. Example would be a batch job which needs to create more
|
730
|
+
jobs based on the data it is processing.
|
731
|
+
|
732
|
+
```ruby
|
733
|
+
batch = Sidekiq::Batch.new(bid)
|
734
|
+
batch.jobs do
|
735
|
+
# define more jobs here
|
736
|
+
end
|
737
|
+
```
|
738
|
+
- Fix issues with symbols vs strings in option hashes
|
739
|
+
|
740
|
+
|
741
|
+
0.6.1
|
742
|
+
-----------
|
743
|
+
|
744
|
+
- Webhook notification support
|
745
|
+
|
746
|
+
|
747
|
+
0.6
|
748
|
+
-----------
|
749
|
+
|
750
|
+
- Redis pubsub
|
751
|
+
- Email polish
|
752
|
+
|
753
|
+
|
754
|
+
0.5
|
755
|
+
-----------
|
756
|
+
|
757
|
+
- Batches
|
758
|
+
- Notifications
|
759
|
+
- Statsd middleware
|