sidekiq-throttled 0.18.0 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.adoc +314 -0
- data/lib/sidekiq/throttled/config.rb +44 -0
- data/lib/sidekiq/throttled/cooldown.rb +55 -0
- data/lib/sidekiq/throttled/expirable_set.rb +70 -0
- data/lib/sidekiq/throttled/job.rb +4 -4
- data/lib/sidekiq/throttled/middlewares/server.rb +28 -0
- data/lib/sidekiq/throttled/patches/basic_fetch.rb +53 -0
- data/lib/sidekiq/throttled/registry.rb +4 -7
- data/lib/sidekiq/throttled/strategy/concurrency.rb +2 -4
- data/lib/sidekiq/throttled/strategy/threshold.rb +2 -4
- data/lib/sidekiq/throttled/strategy.rb +10 -10
- data/lib/sidekiq/throttled/strategy_collection.rb +2 -3
- data/lib/sidekiq/throttled/version.rb +1 -1
- data/lib/sidekiq/throttled/web.rb +2 -45
- data/lib/sidekiq/throttled/worker.rb +1 -1
- data/lib/sidekiq/throttled.rb +45 -57
- metadata +22 -67
- data/.coveralls.yml +0 -1
- data/.github/dependabot.yml +0 -12
- data/.github/workflows/ci.yml +0 -52
- data/.gitignore +0 -12
- data/.rspec +0 -5
- data/.rubocop.yml +0 -20
- data/.rubocop_todo.yml +0 -68
- data/.travis.yml +0 -37
- data/.yardopts +0 -1
- data/Appraisals +0 -9
- data/CHANGES.md +0 -318
- data/Gemfile +0 -34
- data/Guardfile +0 -25
- data/README.md +0 -297
- data/Rakefile +0 -27
- data/gemfiles/sidekiq_6.4.gemfile +0 -33
- data/gemfiles/sidekiq_6.5.gemfile +0 -33
- data/lib/sidekiq/throttled/communicator/callbacks.rb +0 -72
- data/lib/sidekiq/throttled/communicator/exception_handler.rb +0 -25
- data/lib/sidekiq/throttled/communicator/listener.rb +0 -109
- data/lib/sidekiq/throttled/communicator.rb +0 -116
- data/lib/sidekiq/throttled/configuration.rb +0 -50
- data/lib/sidekiq/throttled/expirable_list.rb +0 -70
- data/lib/sidekiq/throttled/fetch/unit_of_work.rb +0 -83
- data/lib/sidekiq/throttled/fetch.rb +0 -94
- data/lib/sidekiq/throttled/middleware.rb +0 -22
- data/lib/sidekiq/throttled/patches/queue.rb +0 -18
- data/lib/sidekiq/throttled/queue_name.rb +0 -46
- data/lib/sidekiq/throttled/queues_pauser.rb +0 -152
- data/lib/sidekiq/throttled/testing.rb +0 -12
- data/lib/sidekiq/throttled/utils.rb +0 -19
- data/lib/sidekiq/throttled/web/queues.html.erb +0 -49
- data/lib/sidekiq/throttled/web/summary_fix.js +0 -10
- data/lib/sidekiq/throttled/web/summary_fix.rb +0 -35
- data/rubocop/layout.yml +0 -24
- data/rubocop/lint.yml +0 -41
- data/rubocop/metrics.yml +0 -4
- data/rubocop/performance.yml +0 -25
- data/rubocop/rspec.yml +0 -3
- data/rubocop/style.yml +0 -84
- data/sidekiq-throttled.gemspec +0 -36
- /data/{LICENSE.md → LICENSE.txt} +0 -0
data/.yardopts
DELETED
@@ -1 +0,0 @@
|
|
1
|
-
--no-private - LICENSE.md
|
data/Appraisals
DELETED
data/CHANGES.md
DELETED
@@ -1,318 +0,0 @@
|
|
1
|
-
## 0.18.0 (2023-04-10)
|
2
|
-
|
3
|
-
* Fix redis-prescription dependency usage
|
4
|
-
|
5
|
-
## 0.17.0 (2022-09-14)
|
6
|
-
|
7
|
-
* [#13](https://github.com/ixti/sidekiq-throttled/pull/13)
|
8
|
-
Fix redis-4.8+ deprecation warnings
|
9
|
-
|
10
|
-
* [#8](https://github.com/ixti/sidekiq-throttled/pull/8)
|
11
|
-
Drop Sidekiq < 6.4 support
|
12
|
-
|
13
|
-
## 0.16.2 (2022-07-19)
|
14
|
-
|
15
|
-
* [#5](https://github.com/ixti/sidekiq-throttled/issues/5)
|
16
|
-
[#6](https://github.com/ixti/sidekiq-throttled/pull/6)
|
17
|
-
Add Sidekiq 6.5+ support
|
18
|
-
|
19
|
-
## 0.16.1 (2022-06-14)
|
20
|
-
|
21
|
-
* [#2](https://github.com/ixti/sidekiq-throttled/pull/2)
|
22
|
-
Support custom ActiveJob adapters.
|
23
|
-
([@longkt90])
|
24
|
-
|
25
|
-
* [#107](https://github.com/sensortower/sidekiq-throttled/pull/107)
|
26
|
-
Log error when key sufix extraction fails.
|
27
|
-
([@pjungwir])
|
28
|
-
|
29
|
-
## 0.16.0 (2022-06-13)
|
30
|
-
|
31
|
-
* Drop Ruby 2.6 support.
|
32
|
-
|
33
|
-
* Drop Sidekiq 5.X support.
|
34
|
-
|
35
|
-
* [#121](https://github.com/sensortower/sidekiq-throttled/pull/121)
|
36
|
-
Sidekiq 6.5 compatibility.
|
37
|
-
([@hieuk09])
|
38
|
-
|
39
|
-
* [#116](https://github.com/sensortower/sidekiq-throttled/pull/116)
|
40
|
-
Unwrap ActiveJob JobWrapper
|
41
|
-
([@holstvoogd])
|
42
|
-
|
43
|
-
* [#115](https://github.com/sensortower/sidekiq-throttled/pull/115)
|
44
|
-
Fix Redis 4.6 deprecation warnings.
|
45
|
-
([@dbackeus])
|
46
|
-
|
47
|
-
* [#113](https://github.com/sensortower/sidekiq-throttled/pull/113)
|
48
|
-
Add Ruby 3.1 to CI.
|
49
|
-
([@petergoldstein])
|
50
|
-
|
51
|
-
* [#103](https://github.com/sensortower/sidekiq-throttled/pull/103)
|
52
|
-
Rename Sidekiq::Throttled::Worker to Sidekiq::Throttled::Job, and alias it
|
53
|
-
as Sidekiq::Throttled::Worker.
|
54
|
-
([@CHTJonas])
|
55
|
-
|
56
|
-
## 0.15.0 (2021-12-16)
|
57
|
-
|
58
|
-
* [#102](https://github.com/sensortower/sidekiq-throttled/pull/102)
|
59
|
-
Support Ruby 3.0 and Sidekiq 6.2+.
|
60
|
-
([@ybiquitous])
|
61
|
-
|
62
|
-
* [#97](https://github.com/sensortower/sidekiq-throttled/pull/97)
|
63
|
-
Fix kwargs usage in strategy collection.
|
64
|
-
([@baptistejub])
|
65
|
-
|
66
|
-
## 0.14.0 (2021-09-21)
|
67
|
-
|
68
|
-
* [#98](https://github.com/sensortower/sidekiq-throttled/pull/98)
|
69
|
-
Remove warning for strategy override.
|
70
|
-
([@mattiagiuffrida-st])
|
71
|
-
|
72
|
-
* [#65](https://github.com/sensortower/sidekiq-throttled/pull/65)
|
73
|
-
Support composite (multi-key) strategies.
|
74
|
-
([@holyketzer])
|
75
|
-
|
76
|
-
* [#89](https://github.com/sensortower/sidekiq-throttled/pull/89),
|
77
|
-
[#96](https://github.com/sensortower/sidekiq-throttled/pull/96), and
|
78
|
-
[#93](https://github.com/sensortower/sidekiq-throttled/pull/93)
|
79
|
-
Improve documentation.
|
80
|
-
([@hubertjakubiak], [@khaile], and [@kylerippey])
|
81
|
-
|
82
|
-
## 0.13.0 (2020-07-28)
|
83
|
-
|
84
|
-
* [#85](https://github.com/sensortower/sidekiq-throttled/pull/85)
|
85
|
-
Add Sidekiq 6.1+ support.
|
86
|
-
([@hmaack])
|
87
|
-
|
88
|
-
## 0.12.0 (2020-06-22)
|
89
|
-
|
90
|
-
* [#80](https://github.com/sensortower/sidekiq-throttled/pull/80)
|
91
|
-
Allow override cooldown timeout of queues with throttled jobs.
|
92
|
-
([@vaot])
|
93
|
-
|
94
|
-
* [#76](https://github.com/sensortower/sidekiq-throttled/pull/76)
|
95
|
-
Fix warnings on Ruby 2.7
|
96
|
-
([@lenon])
|
97
|
-
|
98
|
-
|
99
|
-
## 0.11.0 (2019-08-24)
|
100
|
-
|
101
|
-
* [#59](https://github.com/sensortower/sidekiq-throttled/pull/59)
|
102
|
-
Add throttling observer.
|
103
|
-
([@ogins57])
|
104
|
-
|
105
|
-
|
106
|
-
## 0.10.0 (2019-06-22)
|
107
|
-
|
108
|
-
* [#60](https://github.com/sensortower/sidekiq-throttled/pull/60)
|
109
|
-
Skip throttling check in redis if limit is 0.
|
110
|
-
([@mstruve])
|
111
|
-
|
112
|
-
* [#58](https://github.com/sensortower/sidekiq-throttled/pull/58)
|
113
|
-
Improve documentation bout TTL.
|
114
|
-
([@ziaulrehman40])
|
115
|
-
|
116
|
-
* Improve reliability of paused queues, by resyncing list of paused queues
|
117
|
-
on schedule.
|
118
|
-
([@ixti])
|
119
|
-
|
120
|
-
|
121
|
-
## 0.9.0 (2018-09-11)
|
122
|
-
|
123
|
-
* Add support of Sidekiq 5.2.x
|
124
|
-
([@ixti])
|
125
|
-
|
126
|
-
* Drop support of Sidekiq 4.x
|
127
|
-
([@ixti])
|
128
|
-
|
129
|
-
* Add concurrent-ruby as dependency which is no longer a dependency of
|
130
|
-
Sidekiq as 5.2+ version.
|
131
|
-
([@ixti])
|
132
|
-
|
133
|
-
|
134
|
-
## 0.8.2 (2018-02-14)
|
135
|
-
|
136
|
-
* Extract redis LUA scripts stored procedures runner to redis-prescription gem.
|
137
|
-
([@ixti])
|
138
|
-
|
139
|
-
* Switch to Concurrent.monotonic_time to expire elements of ExpirableList.
|
140
|
-
([@ixti])
|
141
|
-
|
142
|
-
|
143
|
-
## 0.8.1 (2017-11-02)
|
144
|
-
|
145
|
-
* Preload job class constant prior trying to get it's throttling strategy.
|
146
|
-
([@ixti])
|
147
|
-
|
148
|
-
|
149
|
-
## 0.8.0 (2017-10-11)
|
150
|
-
|
151
|
-
* Refactor concurrency throttling internals to use sorted sets in order to avoid
|
152
|
-
starvation in case when finalize! was not called (OOM / redis issues).
|
153
|
-
([@ixti])
|
154
|
-
|
155
|
-
|
156
|
-
## 0.7.3 (2017-06-26)
|
157
|
-
|
158
|
-
* [#34](https://github.com/sensortower/sidekiq-throttled/issues/34)
|
159
|
-
Fix reset button for sidekiq `>= 4.2`.
|
160
|
-
([@ixti])
|
161
|
-
|
162
|
-
|
163
|
-
## 0.7.2 (2017-04-02)
|
164
|
-
|
165
|
-
* Fix summary bar fixer on sidekiq 4.2+.
|
166
|
-
([@ixti])
|
167
|
-
|
168
|
-
* Fix regexp used to fix summay bar queues link when ui was enhanced.
|
169
|
-
([@ixti])
|
170
|
-
|
171
|
-
|
172
|
-
## 0.7.1 (2017-03-30)
|
173
|
-
|
174
|
-
* Fix summary bar queues link when queue ui was enhanced.
|
175
|
-
([@ixti])
|
176
|
-
|
177
|
-
* [#31](https://github.com/sensortower/sidekiq-throttled/pull/31)
|
178
|
-
[#30](https://github.com/sensortower/sidekiq-throttled/issues/30)
|
179
|
-
Do not throttle if limit is `nil`.
|
180
|
-
([@ixti])
|
181
|
-
|
182
|
-
|
183
|
-
## 0.7.0 (2017-03-22)
|
184
|
-
|
185
|
-
* Expose pause/resume queues hidden feature to UI. This was available via API
|
186
|
-
since v0.6.0 and today it's finally got it's UI.
|
187
|
-
([@ixti])
|
188
|
-
|
189
|
-
|
190
|
-
## 0.6.7 (2017-03-21)
|
191
|
-
|
192
|
-
* Fix fetcher causing workers starvation upon low concurrency thresholds.
|
193
|
-
([@ixti])
|
194
|
-
|
195
|
-
|
196
|
-
## 0.6.6 (2016-10-16)
|
197
|
-
|
198
|
-
* [#24](https://github.com/sensortower/sidekiq-throttled/pull/24)
|
199
|
-
Fix dynamic `:key_suffix` issue.
|
200
|
-
([@iporsut])
|
201
|
-
|
202
|
-
|
203
|
-
## 0.6.5 (2016-09-04)
|
204
|
-
|
205
|
-
* Fix concurrency throttling when redis-namespace is used.
|
206
|
-
([@ixti])
|
207
|
-
|
208
|
-
|
209
|
-
## 0.6.4 (2016-09-02)
|
210
|
-
|
211
|
-
* Rename UnitOfWork throttled requeue to `#requeue_throttled`.
|
212
|
-
([@ixti])
|
213
|
-
|
214
|
-
|
215
|
-
## 0.6.3 (2016-09-02)
|
216
|
-
|
217
|
-
* Enrich internal API to allow better extensibility.
|
218
|
-
([@ixti])
|
219
|
-
|
220
|
-
|
221
|
-
## 0.6.2 (2016-09-01)
|
222
|
-
|
223
|
-
* Add `Fetch.bulk_requeue` used by Sidekiq upon termination.
|
224
|
-
([@ixti])
|
225
|
-
|
226
|
-
|
227
|
-
## 0.6.1 (2016-08-30)
|
228
|
-
|
229
|
-
* Trivial internal API change: extracted queues list builder of `Fetch` into
|
230
|
-
dedicated internal method, allowing to enhance it with extra custom filters.
|
231
|
-
([@ixti])
|
232
|
-
|
233
|
-
|
234
|
-
## 0.6.0 (2016-08-27)
|
235
|
-
|
236
|
-
* [#21](https://github.com/sensortower/sidekiq-throttled/pull/21)
|
237
|
-
Allow pause/unpause queues.
|
238
|
-
([@ixti])
|
239
|
-
|
240
|
-
|
241
|
-
## 0.5.0 (2016-08-18)
|
242
|
-
|
243
|
-
* Drop Sidekiq 3.x support.
|
244
|
-
([@ixti])
|
245
|
-
|
246
|
-
|
247
|
-
## 0.4.1 (2016-08-18)
|
248
|
-
|
249
|
-
* [#15](https://github.com/sensortower/sidekiq-throttled/pull/15)
|
250
|
-
Fix throttled web UI on older versions of sidekiq.
|
251
|
-
([@palanglung])
|
252
|
-
|
253
|
-
|
254
|
-
## 0.4.0 (2016-05-17)
|
255
|
-
|
256
|
-
* [#14](https://github.com/sensortower/sidekiq-throttled/pull/14)
|
257
|
-
Support dynamic configuration of limits and periods.
|
258
|
-
([@azach], [@ixti])
|
259
|
-
|
260
|
-
|
261
|
-
## 0.3.2 (2016-05-16)
|
262
|
-
|
263
|
-
* [#13](https://github.com/sensortower/sidekiq-throttled/issues/13)
|
264
|
-
Fix throttled BasicFetch with strictly ordered queues on sidekiq 4.
|
265
|
-
([@palanglung], [@ixti])
|
266
|
-
|
267
|
-
|
268
|
-
## 0.3.1 (2016-05-15)
|
269
|
-
|
270
|
-
* Precalculate LUA script digests to reduce bandwidth upon nodes reload
|
271
|
-
_(which might (and might not) happen if you run thousands of nodes)_.
|
272
|
-
([@ixti])
|
273
|
-
|
274
|
-
|
275
|
-
## 0.3.0 (2016-05-02)
|
276
|
-
|
277
|
-
* [#1](https://github.com/sensortower/sidekiq-throttled/issues/1):
|
278
|
-
Add Sidekiq 4.0 support.
|
279
|
-
([@ixti])
|
280
|
-
|
281
|
-
|
282
|
-
## 0.2.0 (2016-02-29)
|
283
|
-
|
284
|
-
* [#6](https://github.com/sensortower/sidekiq-throttled/pull/6):
|
285
|
-
Add dynamic key suffix functionality.
|
286
|
-
([@fhwang])
|
287
|
-
|
288
|
-
|
289
|
-
## 0.1.0 (2015-11-03)
|
290
|
-
|
291
|
-
* Initial release.
|
292
|
-
|
293
|
-
|
294
|
-
[@ixti]: https://github.com/ixti
|
295
|
-
[@fhwang]: https://github.com/fhwang
|
296
|
-
[@palanglung]: https://github.com/palanglung
|
297
|
-
[@azach]: https://github.com/azach
|
298
|
-
[@iporsut]: https://github.com/iporsut
|
299
|
-
[@mstruve]: https://github.com/mstruve
|
300
|
-
[@ziaulrehman40]: https://github.com/ziaulrehman40
|
301
|
-
[@ogins57]: https://github.com/ogins57
|
302
|
-
[@lenon]: https://github.com/lenon
|
303
|
-
[@vaot]: https://github.com/vaot
|
304
|
-
[@hmaack]: https://github.com/hmaack
|
305
|
-
[@holyketzer]: https://github.com/holyketzer
|
306
|
-
[@hubertjakubiak]: https://github.com/hubertjakubiak
|
307
|
-
[@kylerippey]: https://github.com/kylerippey
|
308
|
-
[@khaile]: https://github.com/khaile
|
309
|
-
[@mattiagiuffrida-st]: https://github.com/mattiagiuffrida-st
|
310
|
-
[@baptistejub]: https://github.com/baptistejub
|
311
|
-
[@ybiquitous]: https://github.com/ybiquitous
|
312
|
-
[@hieuk09]: https://github.com/hieuk09
|
313
|
-
[@petergoldstein]: https://github.com/petergoldstein
|
314
|
-
[@dbackeus]: https://github.com/dbackeus
|
315
|
-
[@holstvoogd]: https://github.com/holstvoogd
|
316
|
-
[@CHTJonas]: https://github.com/CHTJonas
|
317
|
-
[@pjungwir]: https://github.com/pjungwir
|
318
|
-
[@longkt90]: https://github.com/longkt90
|
data/Gemfile
DELETED
@@ -1,34 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
source "https://rubygems.org"
|
4
|
-
|
5
|
-
gem "appraisal"
|
6
|
-
gem "rake"
|
7
|
-
gem "rspec"
|
8
|
-
gem "sidekiq"
|
9
|
-
|
10
|
-
group :development do
|
11
|
-
gem "byebug"
|
12
|
-
gem "guard", :require => false
|
13
|
-
gem "guard-rspec", :require => false
|
14
|
-
gem "guard-rubocop", :require => false
|
15
|
-
end
|
16
|
-
|
17
|
-
group :test do
|
18
|
-
gem "apparition"
|
19
|
-
gem "capybara"
|
20
|
-
gem "puma"
|
21
|
-
gem "rack-test"
|
22
|
-
gem "sinatra"
|
23
|
-
gem "timecop"
|
24
|
-
end
|
25
|
-
|
26
|
-
group :lint do
|
27
|
-
gem "rubocop", :require => false
|
28
|
-
gem "rubocop-performance", :require => false
|
29
|
-
gem "rubocop-rake", :require => false
|
30
|
-
gem "rubocop-rspec", :require => false
|
31
|
-
end
|
32
|
-
|
33
|
-
# Specify your gem's dependencies in sidekiq-throttled.gemspec
|
34
|
-
gemspec
|
data/Guardfile
DELETED
@@ -1,25 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
group :red_green_refactor, :halt_on_fail => true do
|
4
|
-
guard :rspec, :cmd => "bundle exec rspec --no-profile" do
|
5
|
-
require "guard/rspec/dsl"
|
6
|
-
dsl = Guard::RSpec::Dsl.new(self)
|
7
|
-
|
8
|
-
# Feel free to open issues for suggestions and improvements
|
9
|
-
|
10
|
-
# RSpec files
|
11
|
-
rspec = dsl.rspec
|
12
|
-
watch(rspec.spec_helper) { rspec.spec_dir }
|
13
|
-
watch(rspec.spec_support) { rspec.spec_dir }
|
14
|
-
watch(rspec.spec_files)
|
15
|
-
|
16
|
-
# Ruby files
|
17
|
-
ruby = dsl.ruby
|
18
|
-
dsl.watch_spec_files_for(ruby.lib_files)
|
19
|
-
end
|
20
|
-
|
21
|
-
guard :rubocop, :all_on_start => false do
|
22
|
-
watch(%r{.+\.rb$})
|
23
|
-
watch(%r{(?:.+/)?\.rubocop\.yml$}) { |m| File.dirname(m[0]) }
|
24
|
-
end
|
25
|
-
end
|
data/README.md
DELETED
@@ -1,297 +0,0 @@
|
|
1
|
-
# Sidekiq::Throttled
|
2
|
-
|
3
|
-
[](https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml)
|
4
|
-
[](http://rubygems.org/gems/sidekiq-throttled)
|
5
|
-
[](http://www.rubydoc.info/gems/sidekiq-throttled)
|
6
|
-
|
7
|
-
Concurrency and threshold throttling for [Sidekiq][sidekiq].
|
8
|
-
|
9
|
-
## Installation
|
10
|
-
|
11
|
-
Add this line to your application's Gemfile:
|
12
|
-
|
13
|
-
``` ruby
|
14
|
-
gem "sidekiq-throttled"
|
15
|
-
```
|
16
|
-
|
17
|
-
And then execute:
|
18
|
-
|
19
|
-
$ bundle
|
20
|
-
|
21
|
-
Or install it yourself as:
|
22
|
-
|
23
|
-
$ gem install sidekiq-throttled
|
24
|
-
|
25
|
-
|
26
|
-
## Usage
|
27
|
-
|
28
|
-
Add somewhere in your app's bootstrap (e.g. `config/initializers/sidekiq.rb` if
|
29
|
-
you are using Rails):
|
30
|
-
|
31
|
-
``` ruby
|
32
|
-
require "sidekiq/throttled"
|
33
|
-
Sidekiq::Throttled.setup!
|
34
|
-
```
|
35
|
-
|
36
|
-
Load order can be an issue if you are using other Sidekiq plugins and/or middleware.
|
37
|
-
To prevent any problems, add the `.setup!` call to the bottom of your init file.
|
38
|
-
|
39
|
-
Once you've done that you can include `Sidekiq::Throttled::Job` to your
|
40
|
-
job classes and configure throttling:
|
41
|
-
|
42
|
-
``` ruby
|
43
|
-
class MyJob
|
44
|
-
include Sidekiq::Job
|
45
|
-
include Sidekiq::Throttled::Job
|
46
|
-
|
47
|
-
sidekiq_options :queue => :my_queue
|
48
|
-
|
49
|
-
sidekiq_throttle(
|
50
|
-
# Allow maximum 10 concurrent jobs of this class at a time.
|
51
|
-
:concurrency => { :limit => 10 },
|
52
|
-
# Allow maximum 1K jobs being processed within one hour window.
|
53
|
-
:threshold => { :limit => 1_000, :period => 1.hour }
|
54
|
-
)
|
55
|
-
|
56
|
-
def perform
|
57
|
-
# ...
|
58
|
-
end
|
59
|
-
end
|
60
|
-
```
|
61
|
-
|
62
|
-
**NOTE:** `Sidekiq::Throttled::Job` is aliased as `Sidekiq::Throttled::Worker`,
|
63
|
-
thus if you're using Sidekiq prior 6.3.0 version, or you using `Sidekiq::Worker`
|
64
|
-
naming convention, you can use the alias for consistency:
|
65
|
-
|
66
|
-
``` ruby
|
67
|
-
class MyWorker
|
68
|
-
include Sidekiq::Worker
|
69
|
-
include Sidekiq::Throttled::Worker
|
70
|
-
|
71
|
-
# ...
|
72
|
-
end
|
73
|
-
```
|
74
|
-
|
75
|
-
|
76
|
-
### Observer
|
77
|
-
|
78
|
-
You can specify an observer that will be called on throttling. To do so pass an
|
79
|
-
`:observer` option with callable object:
|
80
|
-
|
81
|
-
``` ruby
|
82
|
-
class MyJob
|
83
|
-
include Sidekiq::Job
|
84
|
-
include Sidekiq::Throttled::Job
|
85
|
-
|
86
|
-
MY_OBSERVER = lambda do |strategy, *args|
|
87
|
-
# do something
|
88
|
-
end
|
89
|
-
|
90
|
-
sidekiq_options :queue => :my_queue
|
91
|
-
|
92
|
-
sidekiq_throttle(
|
93
|
-
:concurrency => { :limit => 10 },
|
94
|
-
:threshold => { :limit => 100, :period => 1.hour }
|
95
|
-
:observer => MY_OBSERVER
|
96
|
-
)
|
97
|
-
|
98
|
-
def perform(*args)
|
99
|
-
# ...
|
100
|
-
end
|
101
|
-
end
|
102
|
-
```
|
103
|
-
|
104
|
-
Observer will receive `strategy, *args` arguments, where `strategy` is a Symbol
|
105
|
-
`:concurrency` or `:threshold`, and `*args` are the arguments that were passed
|
106
|
-
to the job.
|
107
|
-
|
108
|
-
|
109
|
-
### Dynamic throttling
|
110
|
-
|
111
|
-
You can throttle jobs dynamically with `:key_suffix` option:
|
112
|
-
|
113
|
-
``` ruby
|
114
|
-
class MyJob
|
115
|
-
include Sidekiq::Job
|
116
|
-
include Sidekiq::Throttled::Job
|
117
|
-
|
118
|
-
sidekiq_options :queue => :my_queue
|
119
|
-
|
120
|
-
sidekiq_throttle(
|
121
|
-
# Allow maximum 10 concurrent jobs per user at a time.
|
122
|
-
:concurrency => { :limit => 10, :key_suffix => -> (user_id) { user_id } }
|
123
|
-
)
|
124
|
-
|
125
|
-
def perform(user_id)
|
126
|
-
# ...
|
127
|
-
end
|
128
|
-
end
|
129
|
-
```
|
130
|
-
|
131
|
-
You can also supply dynamic values for limits and periods by supplying a proc
|
132
|
-
for these values. The proc will be evaluated at the time the job is fetched
|
133
|
-
and will receive the same arguments that are passed to the job.
|
134
|
-
|
135
|
-
``` ruby
|
136
|
-
class MyJob
|
137
|
-
include Sidekiq::Job
|
138
|
-
include Sidekiq::Throttled::Job
|
139
|
-
|
140
|
-
sidekiq_options :queue => :my_queue
|
141
|
-
|
142
|
-
sidekiq_throttle(
|
143
|
-
# Allow maximum 1000 concurrent jobs of this class at a time for VIPs and 10 for all other users.
|
144
|
-
:concurrency => {
|
145
|
-
:limit => ->(user_id) { User.vip?(user_id) ? 1_000 : 10 },
|
146
|
-
:key_suffix => ->(user_id) { User.vip?(user_id) ? "vip" : "std" }
|
147
|
-
},
|
148
|
-
# Allow 1000 jobs/hour to be processed for VIPs and 10/day for all others
|
149
|
-
:threshold => {
|
150
|
-
:limit => ->(user_id) { User.vip?(user_id) ? 1_000 : 10 },
|
151
|
-
:period => ->(user_id) { User.vip?(user_id) ? 1.hour : 1.day },
|
152
|
-
:key_suffix => ->(user_id) { User.vip?(user_id) ? "vip" : "std" }
|
153
|
-
)
|
154
|
-
|
155
|
-
def perform(user_id)
|
156
|
-
# ...
|
157
|
-
end
|
158
|
-
end
|
159
|
-
```
|
160
|
-
|
161
|
-
You also can use several different keys to throttle one worker.
|
162
|
-
|
163
|
-
``` ruby
|
164
|
-
class MyJob
|
165
|
-
include Sidekiq::Job
|
166
|
-
include Sidekiq::Throttled::Job
|
167
|
-
|
168
|
-
sidekiq_options :queue => :my_queue
|
169
|
-
|
170
|
-
sidekiq_throttle(
|
171
|
-
# Allow maximum 10 concurrent jobs per project at a time and maximum 2 jobs per user
|
172
|
-
:concurrency => [
|
173
|
-
{ :limit => 10, :key_suffix => -> (project_id, user_id) { project_id } },
|
174
|
-
{ :limit => 2, :key_suffix => -> (project_id, user_id) { user_id } }
|
175
|
-
]
|
176
|
-
# For :threshold it works the same
|
177
|
-
)
|
178
|
-
|
179
|
-
def perform(project_id, user_id)
|
180
|
-
# ...
|
181
|
-
end
|
182
|
-
end
|
183
|
-
```
|
184
|
-
|
185
|
-
**NB** Don't forget to specify `:key_suffix` and make it return different values
|
186
|
-
if you are using dynamic limit/period options. Otherwise you risk getting into
|
187
|
-
some trouble.
|
188
|
-
|
189
|
-
|
190
|
-
### Concurrency throttling fine-tuning
|
191
|
-
|
192
|
-
Concurrency throttling is based on distributed locks. Those locks have default
|
193
|
-
time to live (TTL) set to 15 minutes. If your job takes more than 15 minutes
|
194
|
-
to finish, lock will be released and you might end up with more jobs running
|
195
|
-
concurrently than you expect.
|
196
|
-
|
197
|
-
This is done to avoid deadlocks - when by any reason (e.g. Sidekiq process was
|
198
|
-
OOM-killed) cleanup middleware wasn't executed and locks were not released.
|
199
|
-
|
200
|
-
If your job takes more than 15 minutes to complete, you can tune concurrency
|
201
|
-
lock TTL to fit your needs:
|
202
|
-
|
203
|
-
``` ruby
|
204
|
-
# Set concurrency strategy lock TTL to 1 hour.
|
205
|
-
sidekiq_throttle(:concurrency => { :limit => 20, :ttl => 1.hour.to_i })
|
206
|
-
```
|
207
|
-
|
208
|
-
|
209
|
-
## Enhanced Queues list
|
210
|
-
|
211
|
-
This gem provides ability to pause/resume queues from processing by workers.
|
212
|
-
So you may simply pause particular queue without need to stop and reconfigure
|
213
|
-
workers by simply pushing a button on sidekiq web UI.
|
214
|
-
|
215
|
-
By default we add *Enhanced Queues* tab with this functionality. But if you
|
216
|
-
want you can override default *Queues* tab completely (notice that page will
|
217
|
-
still be available using it's URL, but tab will be pointing enhanced version).
|
218
|
-
To do so, just call `Sidekiq::Throttled::Web.enhance_queues_tab!` somewhere
|
219
|
-
in your initializer/bootstrap code. If you are using rails, you might want to
|
220
|
-
add it right into your `config/routes.rb` file:
|
221
|
-
|
222
|
-
``` ruby
|
223
|
-
# file config/routes.rb
|
224
|
-
|
225
|
-
require "sidekiq/web"
|
226
|
-
require "sidekiq/throttled/web"
|
227
|
-
|
228
|
-
Rails.application.routes.draw do
|
229
|
-
# ...
|
230
|
-
|
231
|
-
# Replace Sidekiq Queues with enhanced version!
|
232
|
-
Sidekiq::Throttled::Web.enhance_queues_tab!
|
233
|
-
|
234
|
-
# Mount Sidekiq Web UI to `/sidekiq` endpoint
|
235
|
-
mount Sidekiq::Web => "/sidekiq"
|
236
|
-
|
237
|
-
# ...
|
238
|
-
end
|
239
|
-
```
|
240
|
-
|
241
|
-
|
242
|
-
## Supported Ruby Versions
|
243
|
-
|
244
|
-
This library aims to support and is [tested against][ci] the following Ruby
|
245
|
-
versions:
|
246
|
-
|
247
|
-
* Ruby 2.7.x
|
248
|
-
* Ruby 3.0.x
|
249
|
-
* Ruby 3.1.x
|
250
|
-
|
251
|
-
If something doesn't work on one of these versions, it's a bug.
|
252
|
-
|
253
|
-
This library may inadvertently work (or seem to work) on other Ruby versions,
|
254
|
-
however support will only be provided for the versions listed above.
|
255
|
-
|
256
|
-
If you would like this library to support another Ruby version or
|
257
|
-
implementation, you may volunteer to be a maintainer. Being a maintainer
|
258
|
-
entails making sure all tests run and pass on that implementation. When
|
259
|
-
something breaks on your implementation, you will be responsible for providing
|
260
|
-
patches in a timely fashion. If critical issues for a particular implementation
|
261
|
-
exist at the time of a major release, support for that Ruby version may be
|
262
|
-
dropped.
|
263
|
-
|
264
|
-
|
265
|
-
## Supported Sidekiq Versions
|
266
|
-
|
267
|
-
This library aims to support work with following [Sidekiq][sidekiq] versions:
|
268
|
-
|
269
|
-
* Sidekiq 6.4.x
|
270
|
-
* Sidekiq 6.5.x
|
271
|
-
|
272
|
-
|
273
|
-
## Contributing
|
274
|
-
|
275
|
-
* Fork sidekiq-throttled on GitHub
|
276
|
-
* Make your changes
|
277
|
-
* Ensure all tests pass (`bundle exec rake`)
|
278
|
-
* Send a pull request
|
279
|
-
* If we like them we'll merge them
|
280
|
-
* If we've accepted a patch, feel free to ask for commit access!
|
281
|
-
|
282
|
-
|
283
|
-
## Development
|
284
|
-
|
285
|
-
```
|
286
|
-
bundle update
|
287
|
-
bundle exec appraisal install # install dependencies for all gemfiles
|
288
|
-
bundle exec appraisal update # update dependencies for all gemfiles
|
289
|
-
bundle exec appraisal rspec # run rspec against each gemfile
|
290
|
-
bundle exec rubocop # run static code analysis
|
291
|
-
```
|
292
|
-
|
293
|
-
Don't forget to run `appraisal update` after any changes to `Gemfile`.
|
294
|
-
|
295
|
-
|
296
|
-
[ci]: https://github.com/ixti/sidekiq-throttled/actions/workflows/ci.yml
|
297
|
-
[sidekiq]: https://github.com/mperham/sidekiq
|
data/Rakefile
DELETED
@@ -1,27 +0,0 @@
|
|
1
|
-
# frozen_string_literal: true
|
2
|
-
|
3
|
-
require "bundler/gem_tasks"
|
4
|
-
|
5
|
-
require "rspec/core/rake_task"
|
6
|
-
RSpec::Core::RakeTask.new(:spec)
|
7
|
-
|
8
|
-
desc "Run RuboCop"
|
9
|
-
task :rubocop do
|
10
|
-
require "rubocop"
|
11
|
-
result = RuboCop::CLI.new.run([])
|
12
|
-
abort("RuboCop failed!") if result.nonzero?
|
13
|
-
end
|
14
|
-
|
15
|
-
namespace :rubocop do
|
16
|
-
desc "Auto-correct RuboCop offenses"
|
17
|
-
task :autocorrect do
|
18
|
-
require "rubocop"
|
19
|
-
result = RuboCop::CLI.new.run(["--auto-correct"])
|
20
|
-
abort("RuboCop failed!") if result.nonzero?
|
21
|
-
end
|
22
|
-
end
|
23
|
-
|
24
|
-
default_suite = ENV["CI"] ? :spec : %i[spec rubocop]
|
25
|
-
named_suites = { "rubocop" => :rubocop, "rspec" => :spec }
|
26
|
-
|
27
|
-
task :default => named_suites.fetch(ENV.fetch("SUITE", nil), default_suite)
|