sidekiq-throttled 0.18.0 → 1.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.adoc +314 -0
- data/lib/sidekiq/throttled/config.rb +44 -0
- data/lib/sidekiq/throttled/cooldown.rb +55 -0
- data/lib/sidekiq/throttled/expirable_set.rb +70 -0
- data/lib/sidekiq/throttled/job.rb +4 -4
- data/lib/sidekiq/throttled/middlewares/server.rb +28 -0
- data/lib/sidekiq/throttled/patches/basic_fetch.rb +53 -0
- data/lib/sidekiq/throttled/registry.rb +4 -7
- data/lib/sidekiq/throttled/strategy/concurrency.rb +2 -4
- data/lib/sidekiq/throttled/strategy/threshold.rb +2 -4
- data/lib/sidekiq/throttled/strategy.rb +10 -10
- data/lib/sidekiq/throttled/strategy_collection.rb +2 -3
- data/lib/sidekiq/throttled/version.rb +1 -1
- data/lib/sidekiq/throttled/web.rb +2 -45
- data/lib/sidekiq/throttled/worker.rb +1 -1
- data/lib/sidekiq/throttled.rb +45 -57
- metadata +22 -67
- data/.coveralls.yml +0 -1
- data/.github/dependabot.yml +0 -12
- data/.github/workflows/ci.yml +0 -52
- data/.gitignore +0 -12
- data/.rspec +0 -5
- data/.rubocop.yml +0 -20
- data/.rubocop_todo.yml +0 -68
- data/.travis.yml +0 -37
- data/.yardopts +0 -1
- data/Appraisals +0 -9
- data/CHANGES.md +0 -318
- data/Gemfile +0 -34
- data/Guardfile +0 -25
- data/README.md +0 -297
- data/Rakefile +0 -27
- data/gemfiles/sidekiq_6.4.gemfile +0 -33
- data/gemfiles/sidekiq_6.5.gemfile +0 -33
- data/lib/sidekiq/throttled/communicator/callbacks.rb +0 -72
- data/lib/sidekiq/throttled/communicator/exception_handler.rb +0 -25
- data/lib/sidekiq/throttled/communicator/listener.rb +0 -109
- data/lib/sidekiq/throttled/communicator.rb +0 -116
- data/lib/sidekiq/throttled/configuration.rb +0 -50
- data/lib/sidekiq/throttled/expirable_list.rb +0 -70
- data/lib/sidekiq/throttled/fetch/unit_of_work.rb +0 -83
- data/lib/sidekiq/throttled/fetch.rb +0 -94
- data/lib/sidekiq/throttled/middleware.rb +0 -22
- data/lib/sidekiq/throttled/patches/queue.rb +0 -18
- data/lib/sidekiq/throttled/queue_name.rb +0 -46
- data/lib/sidekiq/throttled/queues_pauser.rb +0 -152
- data/lib/sidekiq/throttled/testing.rb +0 -12
- data/lib/sidekiq/throttled/utils.rb +0 -19
- data/lib/sidekiq/throttled/web/queues.html.erb +0 -49
- data/lib/sidekiq/throttled/web/summary_fix.js +0 -10
- data/lib/sidekiq/throttled/web/summary_fix.rb +0 -35
- data/rubocop/layout.yml +0 -24
- data/rubocop/lint.yml +0 -41
- data/rubocop/metrics.yml +0 -4
- data/rubocop/performance.yml +0 -25
- data/rubocop/rspec.yml +0 -3
- data/rubocop/style.yml +0 -84
- data/sidekiq-throttled.gemspec +0 -36
- /data/{LICENSE.md → LICENSE.txt} +0 -0
data/.yardopts
DELETED
@@ -1 +0,0 @@
|
|
1
|
-
--no-private - LICENSE.md
|
data/Appraisals
DELETED
data/CHANGES.md
DELETED
@@ -1,318 +0,0 @@
|
|
1
|
-
## 0.18.0 (2023-04-10)
|
2
|
-
|
3
|
-
* Fix redis-prescription dependency usage
|
4
|
-
|
5
|
-
## 0.17.0 (2022-09-14)
|
6
|
-
|
7
|
-
* [#13](https://github.com/ixti/sidekiq-throttled/pull/13)
|
8
|
-
Fix redis-4.8+ deprecation warnings
|
9
|
-
|
10
|
-
* [#8](https://github.com/ixti/sidekiq-throttled/pull/8)
|
11
|
-
Drop Sidekiq < 6.4 support
|
12
|
-
|
13
|
-
## 0.16.2 (2022-07-19)
|
14
|
-
|
15
|
-
* [#5](https://github.com/ixti/sidekiq-throttled/issues/5)
|
16
|
-
[#6](https://github.com/ixti/sidekiq-throttled/pull/6)
|
17
|
-
Add Sidekiq 6.5+ support
|
18
|
-
|
19
|
-
## 0.16.1 (2022-06-14)
|
20
|
-
|
21
|
-
* [#2](https://github.com/ixti/sidekiq-throttled/pull/2)
|
22
|
-
Support custom ActiveJob adapters.
|
23
|
-
([@longkt90])
|
24
|
-
|
25
|
-
* [#107](https://github.com/sensortower/sidekiq-throttled/pull/107)
|
26
|
-
Log error when key sufix extraction fails.
|
27
|
-
([@pjungwir])
|
28
|
-
|
29
|
-
## 0.16.0 (2022-06-13)
|
30
|
-
|
31
|
-
* Drop Ruby 2.6 support.
|
32
|
-
|
33
|
-
* Drop Sidekiq 5.X support.
|
34
|
-
|
35
|
-
* [#121](https://github.com/sensortower/sidekiq-throttled/pull/121)
|
36
|
-
Sidekiq 6.5 compatibility.
|
37
|
-
([@hieuk09])
|
38
|
-
|
39
|
-
* [#116](https://github.com/sensortower/sidekiq-throttled/pull/116)
|
40
|
-
Unwrap ActiveJob JobWrapper
|
41
|
-
([@holstvoogd])
|
42
|
-
|
43
|
-
* [#115](https://github.com/sensortower/sidekiq-throttled/pull/115)
|
44
|
-
Fix Redis 4.6 deprecation warnings.
|
45
|
-
([@dbackeus])
|
46
|
-
|
47
|
-
* [#113](https://github.com/sensortower/sidekiq-throttled/pull/113)
|
48
|
-
Add Ruby 3.1 to CI.
|
49
|
-
([@petergoldstein])
|
50
|
-
|
51
|
-
* [#103](https://github.com/sensortower/sidekiq-throttled/pull/103)
|
52
|
-
Rename Sidekiq::Throttled::Worker to Sidekiq::Throttled::Job, and alias it
|
53
|
-
as Sidekiq::Throttled::Worker.
|
54
|
-
([@CHTJonas])
|
55
|
-
|
56
|
-
## 0.15.0 (2021-12-16)
|
57
|
-
|
58
|
-
* [#102](https://github.com/sensortower/sidekiq-throttled/pull/102)
|
59
|
-
Support Ruby 3.0 and Sidekiq 6.2+.
|
60
|
-
([@ybiquitous])
|
61
|
-
|
62
|
-
* [#97](https://github.com/sensortower/sidekiq-throttled/pull/97)
|
63
|
-
Fix kwargs usage in strategy collection.
|
64
|
-
([@baptistejub])
|
65
|
-
|
66
|
-
## 0.14.0 (2021-09-21)
|
67
|
-
|
68
|
-
* [#98](https://github.com/sensortower/sidekiq-throttled/pull/98)
|
69
|
-
Remove warning for strategy override.
|
70
|
-
([@mattiagiuffrida-st])
|
71
|
-
|
72
|
-
* [#65](https://github.com/sensortower/sidekiq-throttled/pull/65)
|
73
|
-
Support composite (multi-key) strategies.
|
74
|
-
([@holyketzer])
|
75
|
-
|
76
|
-
* [#89](https://github.com/sensortower/sidekiq-throttled/pull/89),
|
77
|
-
[#96](https://github.com/sensortower/sidekiq-throttled/pull/96), and
|
78
|
-
[#93](https://github.com/sensortower/sidekiq-throttled/pull/93)
|
79
|
-
Improve documentation.
|
80
|
-
([@hubertjakubiak], [@khaile], and [@kylerippey])
|
81
|
-
|
82
|
-
## 0.13.0 (2020-07-28)
|
83
|
-
|
84
|
-
* [#85](https://github.com/sensortower/sidekiq-throttled/pull/85)
|
85
|
-
Add Sidekiq 6.1+ support.
|
86
|
-
([@hmaack])
|
87
|
-
|
88
|
-
## 0.12.0 (2020-06-22)
|
89
|
-
|
90
|
-
* [#80](https://github.com/sensortower/sidekiq-throttled/pull/80)
|
91
|
-
Allow override cooldown timeout of queues with throttled jobs.
|
92
|
-
([@vaot])
|
93
|
-
|
94
|
-
* [#76](https://github.com/sensortower/sidekiq-throttled/pull/76)
|
95
|
-
Fix warnings on Ruby 2.7
|
96
|
-
([@lenon])
|
97
|
-
|
98
|
-
|
99
|
-
## 0.11.0 (2019-08-24)
|
100
|
-
|
101
|
-
* [#59](https://github.com/sensortower/sidekiq-throttled/pull/59)
|
102
|
-
Add throttling observer.
|
103
|
-
([@ogins57])
|
104
|
-
|
105
|
-
|
106
|
-
## 0.10.0 (2019-06-22)
|
107
|
-
|
108
|
-
* [#60](https://github.com/sensortower/sidekiq-throttled/pull/60)
|
109
|
-
Skip throttling check in redis if limit is 0.
|
110
|
-
([@mstruve])
|
111
|
-
|
112
|
-
* [#58](https://github.com/sensortower/sidekiq-throttled/pull/58)
|
113
|
-
Improve documentation bout TTL.
|
114
|
-
([@ziaulrehman40])
|
115
|
-
|
116
|
-
* Improve reliability of paused queues, by resyncing list of paused queues
|
117
|
-
on schedule.
|
118
|
-
([@ixti])
|
119
|
-
|
120
|
-
|
121
|
-
## 0.9.0 (2018-09-11)
|
122
|
-
|
123
|
-
* Add support of Sidekiq 5.2.x
|
124
|
-
([@ixti])
|
125
|
-
|
126
|
-
* Drop support of Sidekiq 4.x
|
127
|
-
([@ixti])
|
128
|
-
|
129
|
-
* Add concurrent-ruby as dependency which is no longer a dependency of
|
130
|
-
Sidekiq as 5.2+ version.
|
131
|
-
([@ixti])
|
132
|
-
|
133
|
-
|
134
|
-
## 0.8.2 (2018-02-14)
|
135
|
-
|
136
|
-
* Extract redis LUA scripts stored procedures runner to redis-prescription gem.
|
137
|
-
([@ixti])
|
138
|
-
|
139
|
-
* Switch to Concurrent.monotonic_time to expire elements of ExpirableList.
|
140
|
-
([@ixti])
|
141
|
-
|
142
|
-
|
143
|
-
## 0.8.1 (2017-11-02)
|
144
|
-
|
145
|
-
* Preload job class constant prior trying to get it's throttling strategy.
|
146
|
-
([@ixti])
|
147
|
-
|
148
|
-
|
149
|
-
## 0.8.0 (2017-10-11)
|
150
|
-
|
151
|
-
* Refactor concurrency throttling internals to use sorted sets in order to avoid
|
152
|
-
starvation in case when finalize! was not called (OOM / redis issues).
|
153
|
-
([@ixti])
|
154
|
-
|
155
|
-
|
156
|
-
## 0.7.3 (2017-06-26)
|
157
|
-
|
158
|
-
* [#34](https://github.com/sensortower/sidekiq-throttled/issues/34)
|
159
|
-
Fix reset button for sidekiq `>= 4.2`.
|
160
|
-
([@ixti])
|
161
|
-
|
162
|
-
|
163
|
-
## 0.7.2 (2017-04-02)
|
164
|
-
|
165
|
-
* Fix summary bar fixer on sidekiq 4.2+.
|
166
|
-
([@ixti])
|
167
|
-
|
168
|
-
* Fix regexp used to fix summay bar queues link when ui was enhanced.
|
169
|
-
([@ixti])
|
170
|
-
|
171
|
-
|
172
|
-
## 0.7.1 (2017-03-30)
|
173
|
-
|
174
|
-
* Fix summary bar queues link when queue ui was enhanced.
|
175
|
-
([@ixti])
|
176
|
-
|
177
|
-
* [#31](https://github.com/sensortower/sidekiq-throttled/pull/31)
|
178
|
-
[#30](https://github.com/sensortower/sidekiq-throttled/issues/30)
|
179
|
-
Do not throttle if limit is `nil`.
|
180
|
-
([@ixti])
|
181
|
-
|
182
|
-
|
183
|
-
## 0.7.0 (2017-03-22)
|
184
|
-
|
185
|
-
* Expose pause/resume queues hidden feature to UI. This was available via API
|
186
|
-
since v0.6.0 and today it's finally got it's UI.
|
187
|
-
([@ixti])
|
188
|
-
|
189
|
-
|
190
|
-
## 0.6.7 (2017-03-21)
|
191
|
-
|
192
|
-
* Fix fetcher causing workers starvation upon low concurrency thresholds.
|
193
|
-
([@ixti])
|
194
|
-
|
195
|
-
|
196
|
-
## 0.6.6 (2016-10-16)
|
197
|
-
|
198
|
-
* [#24](https://github.com/sensortower/sidekiq-throttled/pull/24)
|
199
|
-
Fix dynamic `:key_suffix` issue.
|
200
|
-
([@iporsut])
|
201
|
-
|
202
|
-
|
203
|
-
## 0.6.5 (2016-09-04)
|
204
|
-
|
205
|
-
* Fix concurrency throttling when redis-namespace is used.
|
206
|
-
([@ixti])
|
207
|
-
|
208
|
-
|
209
|
-
## 0.6.4 (2016-09-02)
|
210
|
-
|
211
|
-
* Rename UnitOfWork throttled requeue to `#requeue_throttled`.
|
212
|
-
([@ixti])
|
213
|
-
|
214
|
-
|
215
|
-
## 0.6.3 (2016-09-02)
|
216
|
-
|
217
|
-
* Enrich internal API to allow better extensibility.
|
218
|
-
([@ixti])
|
219
|
-
|
220
|
-
|
221
|
-
## 0.6.2 (2016-09-01)
|
222
|
-
|
223
|
-
* Add `Fetch.bulk_requeue` used by Sidekiq upon termination.
|
224
|
-
([@ixti])
|
225
|
-
|
226
|
-
|
227
|
-
## 0.6.1 (2016-08-30)
|
228
|
-
|
229
|
-
* Trivial internal API change: extracted queues list builder of `Fetch` into
|
230
|
-
dedicated internal method, allowing to enhance it with extra custom filters.
|
231
|
-
([@ixti])
|
232
|
-
|
233
|
-
|
234
|
-
## 0.6.0 (2016-08-27)
|
235
|
-
|
236
|
-
* [#21](https://github.com/sensortower/sidekiq-throttled/pull/21)
|
237
|
-
Allow pause/unpause queues.
|
238
|
-
([@ixti])
|
239
|
-
|
240
|
-
|
241
|
-
## 0.5.0 (2016-08-18)
|
242
|
-
|
243
|
-
* Drop Sidekiq 3.x support.
|
244
|
-
([@ixti])
|
245
|
-
|
246
|
-
|
247
|
-
## 0.4.1 (2016-08-18)
|
248
|
-
|
249
|
-
* [#15](https://github.com/sensortower/sidekiq-throttled/pull/15)
|
250
|
-
Fix throttled web UI on older versions of sidekiq.
|
251
|
-
([@palanglung])
|
252
|
-
|
253
|
-
|
254
|
-
## 0.4.0 (2016-05-17)
|
255
|
-
|
256
|
-
* [#14](https://github.com/sensortower/sidekiq-throttled/pull/14)
|
257
|
-
Support dynamic configuration of limits and periods.
|
258
|
-
([@azach], [@ixti])
|
259
|
-
|
260
|
-
|
261
|
-
## 0.3.2 (2016-05-16)
|
262
|
-
|
263
|
-
* [#13](https://github.com/sensortower/sidekiq-throttled/issues/13)
|
264
|
-
Fix throttled BasicFetch with strictly ordered queues on sidekiq 4.
|
265
|
-
([@palanglung], [@ixti])
|
266
|
-
|
267
|
-
|
268
|
-
## 0.3.1 (2016-05-15)
|
269
|
-
|
270
|
-
* Precalculate LUA script digests to reduce bandwidth upon nodes reload
|
271
|
-
_(which might (and might not) happen if you run thousands of nodes)_.
|
272
|
-
([@ixti])
|
273
|
-
|
274
|
-
|
275
|
-
## 0.3.0 (2016-05-02)
|
276
|
-
|
277
|
-
* [#1](https://github.com/sensortower/sidekiq-throttled/issues/1):
|
278
|
-
Add Sidekiq 4.0 support.
|
279
|
-
([@ixti])
|
280
|
-
|
281
|
-
|
282
|
-
## 0.2.0 (2016-02-29)
|
283
|
-
|
284
|
-
* [#6](https://github.com/sensortower/sidekiq-throttled/pull/6):
|
285
|
-
Add dynamic key suffix functionality.
|
286
|
-
([@fhwang])
|
287
|
-
|
288
|
-
|
289
|
-
## 0.1.0 (2015-11-03)
|
290
|
-
|
291
|
-
* Initial release.
|
292
|
-
|
293
|
-
|
294
|
-
[@ixti]: https://github.com/ixti
|
295
|
-
[@fhwang]: https://github.com/fhwang
|
296
|
-
[@palanglung]: https://github.com/palanglung
|
297
|
-
[@azach]: https://github.com/azach
|
298
|
-
[@iporsut]: https://github.com/iporsut
|
299
|
-
[@mstruve]: https://github.com/mstruve
|
300
|
-
[@ziaulrehman40]: https://github.com/ziaulrehman40
|
301
|
-
[@ogins57]: https://github.com/ogins57
|
302
|
-
[@lenon]: https://github.com/lenon
|
303
|
-
[@vaot]: https://github.com/vaot
|
304
|
-
[@hmaack]: https://github.com/hmaack
|
305
|
-
[@holyketzer]: https://github.com/holyketzer
|
306
|
-
[@hubertjakubiak]: https://github.com/hubertjakubiak
|
307
|
-
[@kylerippey]: https://github.com/kylerippey
|
308
|
-
[@khaile]: https://github.com/khaile
|
309
|
-
[@mattiagiuffrida-st]: https://github.com/mattiagiuffrida-st
|
310
|
-
[@baptistejub]: https://github.com/baptistejub
|
311
|
-
[@ybiquitous]: https://github.com/ybiquitous
|
312
|
-
[@hieuk09]: https://github.com/hieuk09
|
313
|
-
[@petergoldstein]: https://github.com/petergoldstein
|
314
|
-
[@dbackeus]: https://github.com/dbackeus
|
315
|
-
[@holstvoogd]: https://github.com/holstvoogd
|
316
|
-
[@CHTJonas]: https://github.com/CHTJonas
|
317
|
-
[@pjungwir]: https://github.com/pjungwir
|
318
|
-
[@longkt90]: https://github.com/longkt90
|
data/Gemfile
DELETED
@@ -1,34 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
source "https://rubygems.org"
|
4
|
-
|
5
|
-
gem "appraisal"
|
6
|
-
gem "rake"
|
7
|
-
gem "rspec"
|
8
|
-
gem "sidekiq"
|
9
|
-
|
10
|
-
group :development do
|
11
|
-
gem "byebug"
|
12
|
-
gem "guard", :require => false
|
13
|
-
gem "guard-rspec", :require => false
|
14
|
-
gem "guard-rubocop", :require => false
|
15
|
-
end
|
16
|
-
|
17
|
-
group :test do
|
18
|
-
gem "apparition"
|
19
|
-
gem "capybara"
|
20
|
-
gem "puma"
|
21
|
-
gem "rack-test"
|
22
|
-
gem "sinatra"
|
23
|
-
gem "timecop"
|
24
|
-
end
|
25
|
-
|
26
|
-
group :lint do
|
27
|
-
gem "rubocop", :require => false
|
28
|
-
gem "rubocop-performance", :require => false
|
29
|
-
gem "rubocop-rake", :require => false
|
30
|
-
gem "rubocop-rspec", :require => false
|
31
|
-
end
|
32
|
-
|
33
|
-
# Specify your gem's dependencies in sidekiq-throttled.gemspec
|
34
|
-
gemspec
|
data/Guardfile
DELETED
@@ -1,25 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
group :red_green_refactor, :halt_on_fail => true do
|
4
|
-
guard :rspec, :cmd => "bundle exec rspec --no-profile" do
|
5
|
-
require "guard/rspec/dsl"
|
6
|
-
dsl = Guard::RSpec::Dsl.new(self)
|
7
|
-
|
8
|
-
# Feel free to open issues for suggestions and improvements
|
9
|
-
|
10
|
-
# RSpec files
|
11
|
-
rspec = dsl.rspec
|
12
|
-
watch(rspec.spec_helper) { rspec.spec_dir }
|
13
|
-
watch(rspec.spec_support) { rspec.spec_dir }
|
14
|
-
watch(rspec.spec_files)
|
15
|
-
|
16
|
-
# Ruby files
|
17
|
-
ruby = dsl.ruby
|
18
|
-
dsl.watch_spec_files_for(ruby.lib_files)
|
19
|
-
end
|
20
|
-
|
21
|
-
guard :rubocop, :all_on_start => false do
|
22
|
-
watch(%r{.+\.rb$})
|
23
|
-
watch(%r{(?:.+/)?\.rubocop\.yml$}) { |m| File.dirname(m[0]) }
|
24
|
-
end
|
25
|
-
end
|
data/README.md
DELETED
@@ -1,297 +0,0 @@
|
|
1
|
-
# Sidekiq::Throttled
|
2
|
-
|
3
|
-
[![CI Status](https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml/badge.svg)](https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml)
|
4
|
-
[![Latest Version](https://badge.fury.io/rb/sidekiq-throttled.svg)](http://rubygems.org/gems/sidekiq-throttled)
|
5
|
-
[![API Docs](https://img.shields.io/badge/yard-docs-blue.svg)](http://www.rubydoc.info/gems/sidekiq-throttled)
|
6
|
-
|
7
|
-
Concurrency and threshold throttling for [Sidekiq][sidekiq].
|
8
|
-
|
9
|
-
## Installation
|
10
|
-
|
11
|
-
Add this line to your application's Gemfile:
|
12
|
-
|
13
|
-
``` ruby
|
14
|
-
gem "sidekiq-throttled"
|
15
|
-
```
|
16
|
-
|
17
|
-
And then execute:
|
18
|
-
|
19
|
-
$ bundle
|
20
|
-
|
21
|
-
Or install it yourself as:
|
22
|
-
|
23
|
-
$ gem install sidekiq-throttled
|
24
|
-
|
25
|
-
|
26
|
-
## Usage
|
27
|
-
|
28
|
-
Add somewhere in your app's bootstrap (e.g. `config/initializers/sidekiq.rb` if
|
29
|
-
you are using Rails):
|
30
|
-
|
31
|
-
``` ruby
|
32
|
-
require "sidekiq/throttled"
|
33
|
-
Sidekiq::Throttled.setup!
|
34
|
-
```
|
35
|
-
|
36
|
-
Load order can be an issue if you are using other Sidekiq plugins and/or middleware.
|
37
|
-
To prevent any problems, add the `.setup!` call to the bottom of your init file.
|
38
|
-
|
39
|
-
Once you've done that you can include `Sidekiq::Throttled::Job` to your
|
40
|
-
job classes and configure throttling:
|
41
|
-
|
42
|
-
``` ruby
|
43
|
-
class MyJob
|
44
|
-
include Sidekiq::Job
|
45
|
-
include Sidekiq::Throttled::Job
|
46
|
-
|
47
|
-
sidekiq_options :queue => :my_queue
|
48
|
-
|
49
|
-
sidekiq_throttle(
|
50
|
-
# Allow maximum 10 concurrent jobs of this class at a time.
|
51
|
-
:concurrency => { :limit => 10 },
|
52
|
-
# Allow maximum 1K jobs being processed within one hour window.
|
53
|
-
:threshold => { :limit => 1_000, :period => 1.hour }
|
54
|
-
)
|
55
|
-
|
56
|
-
def perform
|
57
|
-
# ...
|
58
|
-
end
|
59
|
-
end
|
60
|
-
```
|
61
|
-
|
62
|
-
**NOTE:** `Sidekiq::Throttled::Job` is aliased as `Sidekiq::Throttled::Worker`,
|
63
|
-
thus if you're using Sidekiq prior 6.3.0 version, or you using `Sidekiq::Worker`
|
64
|
-
naming convention, you can use the alias for consistency:
|
65
|
-
|
66
|
-
``` ruby
|
67
|
-
class MyWorker
|
68
|
-
include Sidekiq::Worker
|
69
|
-
include Sidekiq::Throttled::Worker
|
70
|
-
|
71
|
-
# ...
|
72
|
-
end
|
73
|
-
```
|
74
|
-
|
75
|
-
|
76
|
-
### Observer
|
77
|
-
|
78
|
-
You can specify an observer that will be called on throttling. To do so pass an
|
79
|
-
`:observer` option with callable object:
|
80
|
-
|
81
|
-
``` ruby
|
82
|
-
class MyJob
|
83
|
-
include Sidekiq::Job
|
84
|
-
include Sidekiq::Throttled::Job
|
85
|
-
|
86
|
-
MY_OBSERVER = lambda do |strategy, *args|
|
87
|
-
# do something
|
88
|
-
end
|
89
|
-
|
90
|
-
sidekiq_options :queue => :my_queue
|
91
|
-
|
92
|
-
sidekiq_throttle(
|
93
|
-
:concurrency => { :limit => 10 },
|
94
|
-
:threshold => { :limit => 100, :period => 1.hour }
|
95
|
-
:observer => MY_OBSERVER
|
96
|
-
)
|
97
|
-
|
98
|
-
def perform(*args)
|
99
|
-
# ...
|
100
|
-
end
|
101
|
-
end
|
102
|
-
```
|
103
|
-
|
104
|
-
Observer will receive `strategy, *args` arguments, where `strategy` is a Symbol
|
105
|
-
`:concurrency` or `:threshold`, and `*args` are the arguments that were passed
|
106
|
-
to the job.
|
107
|
-
|
108
|
-
|
109
|
-
### Dynamic throttling
|
110
|
-
|
111
|
-
You can throttle jobs dynamically with `:key_suffix` option:
|
112
|
-
|
113
|
-
``` ruby
|
114
|
-
class MyJob
|
115
|
-
include Sidekiq::Job
|
116
|
-
include Sidekiq::Throttled::Job
|
117
|
-
|
118
|
-
sidekiq_options :queue => :my_queue
|
119
|
-
|
120
|
-
sidekiq_throttle(
|
121
|
-
# Allow maximum 10 concurrent jobs per user at a time.
|
122
|
-
:concurrency => { :limit => 10, :key_suffix => -> (user_id) { user_id } }
|
123
|
-
)
|
124
|
-
|
125
|
-
def perform(user_id)
|
126
|
-
# ...
|
127
|
-
end
|
128
|
-
end
|
129
|
-
```
|
130
|
-
|
131
|
-
You can also supply dynamic values for limits and periods by supplying a proc
|
132
|
-
for these values. The proc will be evaluated at the time the job is fetched
|
133
|
-
and will receive the same arguments that are passed to the job.
|
134
|
-
|
135
|
-
``` ruby
|
136
|
-
class MyJob
|
137
|
-
include Sidekiq::Job
|
138
|
-
include Sidekiq::Throttled::Job
|
139
|
-
|
140
|
-
sidekiq_options :queue => :my_queue
|
141
|
-
|
142
|
-
sidekiq_throttle(
|
143
|
-
# Allow maximum 1000 concurrent jobs of this class at a time for VIPs and 10 for all other users.
|
144
|
-
:concurrency => {
|
145
|
-
:limit => ->(user_id) { User.vip?(user_id) ? 1_000 : 10 },
|
146
|
-
:key_suffix => ->(user_id) { User.vip?(user_id) ? "vip" : "std" }
|
147
|
-
},
|
148
|
-
# Allow 1000 jobs/hour to be processed for VIPs and 10/day for all others
|
149
|
-
:threshold => {
|
150
|
-
:limit => ->(user_id) { User.vip?(user_id) ? 1_000 : 10 },
|
151
|
-
:period => ->(user_id) { User.vip?(user_id) ? 1.hour : 1.day },
|
152
|
-
:key_suffix => ->(user_id) { User.vip?(user_id) ? "vip" : "std" }
|
153
|
-
)
|
154
|
-
|
155
|
-
def perform(user_id)
|
156
|
-
# ...
|
157
|
-
end
|
158
|
-
end
|
159
|
-
```
|
160
|
-
|
161
|
-
You also can use several different keys to throttle one worker.
|
162
|
-
|
163
|
-
``` ruby
|
164
|
-
class MyJob
|
165
|
-
include Sidekiq::Job
|
166
|
-
include Sidekiq::Throttled::Job
|
167
|
-
|
168
|
-
sidekiq_options :queue => :my_queue
|
169
|
-
|
170
|
-
sidekiq_throttle(
|
171
|
-
# Allow maximum 10 concurrent jobs per project at a time and maximum 2 jobs per user
|
172
|
-
:concurrency => [
|
173
|
-
{ :limit => 10, :key_suffix => -> (project_id, user_id) { project_id } },
|
174
|
-
{ :limit => 2, :key_suffix => -> (project_id, user_id) { user_id } }
|
175
|
-
]
|
176
|
-
# For :threshold it works the same
|
177
|
-
)
|
178
|
-
|
179
|
-
def perform(project_id, user_id)
|
180
|
-
# ...
|
181
|
-
end
|
182
|
-
end
|
183
|
-
```
|
184
|
-
|
185
|
-
**NB** Don't forget to specify `:key_suffix` and make it return different values
|
186
|
-
if you are using dynamic limit/period options. Otherwise you risk getting into
|
187
|
-
some trouble.
|
188
|
-
|
189
|
-
|
190
|
-
### Concurrency throttling fine-tuning
|
191
|
-
|
192
|
-
Concurrency throttling is based on distributed locks. Those locks have default
|
193
|
-
time to live (TTL) set to 15 minutes. If your job takes more than 15 minutes
|
194
|
-
to finish, lock will be released and you might end up with more jobs running
|
195
|
-
concurrently than you expect.
|
196
|
-
|
197
|
-
This is done to avoid deadlocks - when by any reason (e.g. Sidekiq process was
|
198
|
-
OOM-killed) cleanup middleware wasn't executed and locks were not released.
|
199
|
-
|
200
|
-
If your job takes more than 15 minutes to complete, you can tune concurrency
|
201
|
-
lock TTL to fit your needs:
|
202
|
-
|
203
|
-
``` ruby
|
204
|
-
# Set concurrency strategy lock TTL to 1 hour.
|
205
|
-
sidekiq_throttle(:concurrency => { :limit => 20, :ttl => 1.hour.to_i })
|
206
|
-
```
|
207
|
-
|
208
|
-
|
209
|
-
## Enhanced Queues list
|
210
|
-
|
211
|
-
This gem provides ability to pause/resume queues from processing by workers.
|
212
|
-
So you may simply pause particular queue without need to stop and reconfigure
|
213
|
-
workers by simply pushing a button on sidekiq web UI.
|
214
|
-
|
215
|
-
By default we add *Enhanced Queues* tab with this functionality. But if you
|
216
|
-
want you can override default *Queues* tab completely (notice that page will
|
217
|
-
still be available using it's URL, but tab will be pointing enhanced version).
|
218
|
-
To do so, just call `Sidekiq::Throttled::Web.enhance_queues_tab!` somewhere
|
219
|
-
in your initializer/bootstrap code. If you are using rails, you might want to
|
220
|
-
add it right into your `config/routes.rb` file:
|
221
|
-
|
222
|
-
``` ruby
|
223
|
-
# file config/routes.rb
|
224
|
-
|
225
|
-
require "sidekiq/web"
|
226
|
-
require "sidekiq/throttled/web"
|
227
|
-
|
228
|
-
Rails.application.routes.draw do
|
229
|
-
# ...
|
230
|
-
|
231
|
-
# Replace Sidekiq Queues with enhanced version!
|
232
|
-
Sidekiq::Throttled::Web.enhance_queues_tab!
|
233
|
-
|
234
|
-
# Mount Sidekiq Web UI to `/sidekiq` endpoint
|
235
|
-
mount Sidekiq::Web => "/sidekiq"
|
236
|
-
|
237
|
-
# ...
|
238
|
-
end
|
239
|
-
```
|
240
|
-
|
241
|
-
|
242
|
-
## Supported Ruby Versions
|
243
|
-
|
244
|
-
This library aims to support and is [tested against][ci] the following Ruby
|
245
|
-
versions:
|
246
|
-
|
247
|
-
* Ruby 2.7.x
|
248
|
-
* Ruby 3.0.x
|
249
|
-
* Ruby 3.1.x
|
250
|
-
|
251
|
-
If something doesn't work on one of these versions, it's a bug.
|
252
|
-
|
253
|
-
This library may inadvertently work (or seem to work) on other Ruby versions,
|
254
|
-
however support will only be provided for the versions listed above.
|
255
|
-
|
256
|
-
If you would like this library to support another Ruby version or
|
257
|
-
implementation, you may volunteer to be a maintainer. Being a maintainer
|
258
|
-
entails making sure all tests run and pass on that implementation. When
|
259
|
-
something breaks on your implementation, you will be responsible for providing
|
260
|
-
patches in a timely fashion. If critical issues for a particular implementation
|
261
|
-
exist at the time of a major release, support for that Ruby version may be
|
262
|
-
dropped.
|
263
|
-
|
264
|
-
|
265
|
-
## Supported Sidekiq Versions
|
266
|
-
|
267
|
-
This library aims to support work with following [Sidekiq][sidekiq] versions:
|
268
|
-
|
269
|
-
* Sidekiq 6.4.x
|
270
|
-
* Sidekiq 6.5.x
|
271
|
-
|
272
|
-
|
273
|
-
## Contributing
|
274
|
-
|
275
|
-
* Fork sidekiq-throttled on GitHub
|
276
|
-
* Make your changes
|
277
|
-
* Ensure all tests pass (`bundle exec rake`)
|
278
|
-
* Send a pull request
|
279
|
-
* If we like them we'll merge them
|
280
|
-
* If we've accepted a patch, feel free to ask for commit access!
|
281
|
-
|
282
|
-
|
283
|
-
## Development
|
284
|
-
|
285
|
-
```
|
286
|
-
bundle update
|
287
|
-
bundle exec appraisal install # install dependencies for all gemfiles
|
288
|
-
bundle exec appraisal update # update dependencies for all gemfiles
|
289
|
-
bundle exec appraisal rspec # run rspec against each gemfile
|
290
|
-
bundle exec rubocop # run static code analysis
|
291
|
-
```
|
292
|
-
|
293
|
-
Don't forget to run `appraisal update` after any changes to `Gemfile`.
|
294
|
-
|
295
|
-
|
296
|
-
[ci]: https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml
|
297
|
-
[sidekiq]: https://github.com/mperham/sidekiq
|
data/Rakefile
DELETED
@@ -1,27 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require "bundler/gem_tasks"
|
4
|
-
|
5
|
-
require "rspec/core/rake_task"
|
6
|
-
RSpec::Core::RakeTask.new(:spec)
|
7
|
-
|
8
|
-
desc "Run RuboCop"
|
9
|
-
task :rubocop do
|
10
|
-
require "rubocop"
|
11
|
-
result = RuboCop::CLI.new.run([])
|
12
|
-
abort("RuboCop failed!") if result.nonzero?
|
13
|
-
end
|
14
|
-
|
15
|
-
namespace :rubocop do
|
16
|
-
desc "Auto-correct RuboCop offenses"
|
17
|
-
task :autocorrect do
|
18
|
-
require "rubocop"
|
19
|
-
result = RuboCop::CLI.new.run(["--auto-correct"])
|
20
|
-
abort("RuboCop failed!") if result.nonzero?
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
default_suite = ENV["CI"] ? :spec : %i[spec rubocop]
|
25
|
-
named_suites = { "rubocop" => :rubocop, "rspec" => :spec }
|
26
|
-
|
27
|
-
task :default => named_suites.fetch(ENV.fetch("SUITE", nil), default_suite)
|