sidekiq-rate-limiter 0.0.0 → 0.0.1
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +1 -0
- data/.travis.yml +0 -1
- data/README.md +108 -1
- data/lib/sidekiq-rate-limiter.rb +1 -0
- data/lib/sidekiq-rate-limiter/fetch.rb +54 -0
- data/lib/sidekiq-rate-limiter/server.rb +6 -0
- data/lib/sidekiq-rate-limiter/version.rb +1 -1
- data/sidekiq-rate-limiter.gemspec +3 -0
- data/spec/sidekiq-rate-limiter/fetch_spec.rb +64 -0
- data/spec/sidekiq-rate-limiter/server_spec.rb +20 -0
- data/spec/spec_helper.rb +88 -0
- metadata +58 -2
data/.gitignore
CHANGED
data/.travis.yml
CHANGED
data/README.md
CHANGED
@@ -1,4 +1,111 @@
|
|
1
1
|
sidekiq-rate-limiter
|
2
2
|
====================
|
3
3
|
|
4
|
-
|
4
|
+
[![Build Status](https://secure.travis-ci.org/enova/sidekiq-rate-limiter.png)](http://travis-ci.org/enova/sidekiq-rate-limiter)
|
5
|
+
|
6
|
+
Adds to sidekiq the ability to rate limit job execution on a per-worker basis in
|
7
|
+
a redis-backed fashion.
|
8
|
+
|
9
|
+
## Compatibility
|
10
|
+
|
11
|
+
sidekiq-rate-limiter is actively tested against MRI versions 2.0.0 and 1.9.3.
|
12
|
+
|
13
|
+
sidekiq-rate-limiter works by using a custom fetch class, the class responsible
|
14
|
+
for pulling work from the queue stored in redis. Consequently you'll want to be
|
15
|
+
careful about using other gems that use a same strategy ([sidekiq-priority](https://github.com/socialpandas/sidekiq-priority)
|
16
|
+
being one example.
|
17
|
+
|
18
|
+
I've attempted to support the same options as used by [sidekiq-throttler](/gevans/sidekiq-throttler). So, if
|
19
|
+
your worker already looks like this example I lifted from the sidekiq-throttler wiki:
|
20
|
+
|
21
|
+
```ruby
|
22
|
+
class MyWorker
|
23
|
+
include Sidekiq::Worker
|
24
|
+
|
25
|
+
sidekiq_options throttle: { threshold: 50, period: 1.hour }
|
26
|
+
|
27
|
+
def perform(user_id)
|
28
|
+
# Do some heavy API interactions.
|
29
|
+
end
|
30
|
+
end
|
31
|
+
```
|
32
|
+
|
33
|
+
Then you wouldn't need to change anything.
|
34
|
+
|
35
|
+
## Installation
|
36
|
+
|
37
|
+
Add this line to your application's Gemfile:
|
38
|
+
|
39
|
+
gem 'sidekiq-rate-limiter'
|
40
|
+
|
41
|
+
And then execute:
|
42
|
+
|
43
|
+
$ bundle
|
44
|
+
|
45
|
+
Or install it yourself as:
|
46
|
+
|
47
|
+
$ gem install sidekiq-rate-limiter
|
48
|
+
|
49
|
+
## Configuration
|
50
|
+
|
51
|
+
See [server.rb](lib/sidekiq-rate-limiter/server.rb) for an example of how to
|
52
|
+
configure sidekiq-rate-limiter. Alternatively you can add the following to your
|
53
|
+
initializer or what-have-you:
|
54
|
+
|
55
|
+
```ruby
|
56
|
+
require 'sidekiq-rate-limiter/server'
|
57
|
+
```
|
58
|
+
|
59
|
+
Or, if you prefer, amend your Gemfile like so:
|
60
|
+
|
61
|
+
gem 'sidekiq-rate-limiter', :require => 'sidekiq-rate-limiter/server'
|
62
|
+
|
63
|
+
By default the limiter uses the name 'sidekiq-rate-limiter'. You can define the
|
64
|
+
constant ```Sidekiq::RateLimiter::DEFAULT_LIMIT_NAME``` prior to requiring to
|
65
|
+
change this. Alternatively, you can include a 'name' parameter in the configuration
|
66
|
+
hash included in sidekiq_options
|
67
|
+
|
68
|
+
For example, the following:
|
69
|
+
|
70
|
+
```ruby
|
71
|
+
class Job
|
72
|
+
include Sidekiq::Worker
|
73
|
+
|
74
|
+
sidekiq_options :queue => 'some_silly_queue',
|
75
|
+
:rate => {
|
76
|
+
:name => 'my_super_awesome_rate_limit',
|
77
|
+
:limit => 50,
|
78
|
+
:period => 3600, ## An hour
|
79
|
+
}
|
80
|
+
|
81
|
+
def perform(*args)
|
82
|
+
## do stuff
|
83
|
+
## ...
|
84
|
+
```
|
85
|
+
|
86
|
+
The configuration above would result in any jobs beyond the first 50 in a one
|
87
|
+
hour period being delayed. The server will continue to fetch items from redis, &
|
88
|
+
will place any items that are beyond the threshold at the back of their queue.
|
89
|
+
|
90
|
+
## Motivation
|
91
|
+
|
92
|
+
Sidekiq::Throttler is great for smaller quantities of jobs, but falls down a bit
|
93
|
+
for larger queues (see [issue #8](/gevans/sidekiq-throttler/issues/8). In addition, jobs that are
|
94
|
+
limited multiple times are counted as 'processed' each time, so the stats baloon quickly.
|
95
|
+
|
96
|
+
## TODO
|
97
|
+
|
98
|
+
* Most or all of the configuration options should support procs
|
99
|
+
* While it subclasses instead of monkey patching, setting Sidekiq.options[:fetch]
|
100
|
+
is still asking for interaction issues. It would be better for this to be directly
|
101
|
+
in sidekiq or to use some other means to accomplish this goal.
|
102
|
+
|
103
|
+
## Contributing
|
104
|
+
|
105
|
+
1. Fork
|
106
|
+
2. Commit
|
107
|
+
5. Pull Request
|
108
|
+
|
109
|
+
## License
|
110
|
+
|
111
|
+
MIT. See [LICENSE](LICENSE) for details.
|
data/lib/sidekiq-rate-limiter.rb
CHANGED
@@ -0,0 +1,54 @@
|
|
1
|
+
require 'celluloid'
|
2
|
+
require 'sidekiq/fetch'
|
3
|
+
require 'redis_rate_limiter'
|
4
|
+
|
5
|
+
module Sidekiq::RateLimiter
|
6
|
+
DEFAULT_LIMIT_NAME =
|
7
|
+
'sidekiq-rate-limit'.freeze unless defined?(DEFAULT_LIMIT_NAME)
|
8
|
+
|
9
|
+
class Fetch < Sidekiq::BasicFetch
|
10
|
+
def retrieve_work
|
11
|
+
limit(super)
|
12
|
+
end
|
13
|
+
|
14
|
+
def limit(work)
|
15
|
+
message = JSON.parse(work.message) rescue {}
|
16
|
+
|
17
|
+
klass = message['class']
|
18
|
+
rate = message['rate'] || message['throttle'] || {}
|
19
|
+
limit = rate['limit'] || rate['threshold']
|
20
|
+
interval = rate['period'] || rate['interval']
|
21
|
+
name = rate['name'] || DEFAULT_LIMIT_NAME
|
22
|
+
|
23
|
+
return work unless !!(klass && limit && interval)
|
24
|
+
|
25
|
+
options = {
|
26
|
+
:limit => limit,
|
27
|
+
:interval => interval,
|
28
|
+
:name => name,
|
29
|
+
}
|
30
|
+
|
31
|
+
Sidekiq.redis do |conn|
|
32
|
+
lim = Limit.new(conn, options)
|
33
|
+
if lim.exceeded?(klass)
|
34
|
+
conn.lpush("queue:#{work.queue_name}", work.message)
|
35
|
+
nil
|
36
|
+
else
|
37
|
+
lim.add(klass)
|
38
|
+
work
|
39
|
+
end
|
40
|
+
end
|
41
|
+
end
|
42
|
+
end
|
43
|
+
|
44
|
+
class Limit < RedisRateLimiter
|
45
|
+
def initialize(redis, options = {})
|
46
|
+
options = options.dup
|
47
|
+
name = options.delete('name') ||
|
48
|
+
options.delete(:name)
|
49
|
+
|
50
|
+
super(name, redis, options)
|
51
|
+
end
|
52
|
+
end
|
53
|
+
|
54
|
+
end
|
@@ -24,6 +24,9 @@ Gem::Specification.new do |s|
|
|
24
24
|
s.add_development_dependency "rake"
|
25
25
|
s.add_development_dependency "rspec"
|
26
26
|
s.add_development_dependency "simplecov"
|
27
|
+
s.add_development_dependency "simplecov-rcov"
|
27
28
|
|
29
|
+
s.add_dependency "redis"
|
28
30
|
s.add_dependency "sidekiq"
|
31
|
+
s.add_dependency "redis_rate_limiter"
|
29
32
|
end
|
@@ -0,0 +1,64 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
require 'sidekiq'
|
3
|
+
|
4
|
+
describe Sidekiq::RateLimiter::Fetch do
|
5
|
+
before(:all) do
|
6
|
+
class Job
|
7
|
+
include Sidekiq::Worker
|
8
|
+
sidekiq_options 'queue' => 'basic',
|
9
|
+
'retry' => false,
|
10
|
+
'rate' => {
|
11
|
+
'limit' => 1,
|
12
|
+
'period' => 1
|
13
|
+
}
|
14
|
+
def perform(*args); end
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
let(:options) { { queues: [queue, another_queue, another_queue] } }
|
19
|
+
let(:queue) { 'basic' }
|
20
|
+
let(:another_queue) { 'some_other_queue' }
|
21
|
+
let(:args) { ['I am some args'] }
|
22
|
+
let(:worker) { Job }
|
23
|
+
let(:redis_class) { Sidekiq.redis { |conn| conn.class } }
|
24
|
+
|
25
|
+
it 'should inherit from Sidekiq::BasicFetch' do
|
26
|
+
described_class.should < Sidekiq::BasicFetch
|
27
|
+
end
|
28
|
+
|
29
|
+
it 'should retrieve work with strict setting' do
|
30
|
+
fetch = described_class.new options.merge(:strict => true)
|
31
|
+
fetch.queues_cmd.should eql(["queue:#{queue}", "queue:#{another_queue}", 1])
|
32
|
+
end
|
33
|
+
|
34
|
+
it 'should retrieve work', queuing: true do
|
35
|
+
worker.perform_async(*args)
|
36
|
+
fetch = described_class.new(options)
|
37
|
+
work = fetch.retrieve_work
|
38
|
+
parsed = JSON.parse(work.message)
|
39
|
+
|
40
|
+
work.should_not be_nil
|
41
|
+
work.queue_name.should eql(queue)
|
42
|
+
work.acknowledge.should be_nil
|
43
|
+
|
44
|
+
parsed.should include(worker.get_sidekiq_options)
|
45
|
+
parsed.should include("class" => worker.to_s, "args" => args)
|
46
|
+
parsed.should include("jid", "enqueued_at")
|
47
|
+
|
48
|
+
q = Sidekiq::Queue.new(queue)
|
49
|
+
q.size.should == 0
|
50
|
+
end
|
51
|
+
|
52
|
+
it 'should place rate-limited work at the back of the queue', queuing: true do
|
53
|
+
worker.perform_async(*args)
|
54
|
+
Sidekiq::RateLimiter::Limit.any_instance.should_receive(:exceeded?).and_return(true)
|
55
|
+
redis_class.any_instance.should_receive(:lpush).exactly(:once).and_call_original
|
56
|
+
|
57
|
+
fetch = described_class.new(options)
|
58
|
+
fetch.retrieve_work.should be_nil
|
59
|
+
|
60
|
+
q = Sidekiq::Queue.new(queue)
|
61
|
+
q.size.should == 1
|
62
|
+
end
|
63
|
+
|
64
|
+
end
|
@@ -0,0 +1,20 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
describe Sidekiq::RateLimiter, 'server configuration' do
|
4
|
+
before do
|
5
|
+
Sidekiq.stub(:server? => true)
|
6
|
+
require 'sidekiq-rate-limiter/server'
|
7
|
+
end
|
8
|
+
|
9
|
+
it 'should set Sidekiq.options[:fetch] as desired' do
|
10
|
+
Sidekiq.configure_server do |config|
|
11
|
+
Sidekiq.options[:fetch].should eql(Sidekiq::RateLimiter::Fetch)
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
it 'should inherit from Sidekiq::BasicFetch' do
|
16
|
+
Sidekiq.configure_server do |config|
|
17
|
+
Sidekiq.options[:fetch].should < Sidekiq::BasicFetch
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
data/spec/spec_helper.rb
CHANGED
@@ -1,3 +1,51 @@
|
|
1
|
+
require 'sidekiq'
|
2
|
+
require 'sidekiq/testing'
|
3
|
+
|
4
|
+
## Confirming presence of redis server executable
|
5
|
+
abort "## `redis-server` not in path" if %x(which redis-server).empty?
|
6
|
+
redis_dir = "#{File.dirname(__FILE__)}/support/redis"
|
7
|
+
|
8
|
+
## Redis configuration
|
9
|
+
REDIS_CONFIG = <<-CONF
|
10
|
+
daemonize yes
|
11
|
+
pidfile #{redis_dir}/test.pid
|
12
|
+
port 6380
|
13
|
+
timeout 300
|
14
|
+
save 900 1
|
15
|
+
save 300 10
|
16
|
+
save 60 10000
|
17
|
+
dbfilename test.rdb
|
18
|
+
dir #{redis_dir}
|
19
|
+
loglevel warning
|
20
|
+
logfile stdout
|
21
|
+
databases 1
|
22
|
+
CONF
|
23
|
+
|
24
|
+
%x(echo '#{REDIS_CONFIG}' > #{redis_dir}/test.conf)
|
25
|
+
redis_command = "redis-server #{redis_dir}/test.conf"
|
26
|
+
%x[ #{redis_command} ]
|
27
|
+
##
|
28
|
+
|
29
|
+
## Configuring sidekiq
|
30
|
+
options = {
|
31
|
+
logger: nil,
|
32
|
+
redis: { :url => "redis://localhost:6380/0" }
|
33
|
+
}
|
34
|
+
|
35
|
+
Sidekiq.configure_client do |config|
|
36
|
+
options.each do |option, value|
|
37
|
+
config.send("#{option}=", value)
|
38
|
+
end
|
39
|
+
end
|
40
|
+
|
41
|
+
Sidekiq.configure_server do |config|
|
42
|
+
options.each do |option, value|
|
43
|
+
config.send("#{option}=", value)
|
44
|
+
end
|
45
|
+
end
|
46
|
+
##
|
47
|
+
|
48
|
+
## Configuring simplecov
|
1
49
|
require 'simplecov'
|
2
50
|
|
3
51
|
SimpleCov.start do
|
@@ -6,3 +54,43 @@ SimpleCov.start do
|
|
6
54
|
end
|
7
55
|
|
8
56
|
require File.expand_path("../../lib/sidekiq-rate-limiter", __FILE__)
|
57
|
+
##
|
58
|
+
|
59
|
+
## Hook to set Sidekiq::Testing mode using rspec tags
|
60
|
+
RSpec.configure do |config|
|
61
|
+
config.before(:each) do
|
62
|
+
## Use metadata to determine testing behavior
|
63
|
+
## for queuing.
|
64
|
+
|
65
|
+
case example.metadata[:queuing].to_s
|
66
|
+
when 'enable', 'enabled', 'on', 'true'
|
67
|
+
Sidekiq::Testing.disable!
|
68
|
+
when 'fake', 'mock'
|
69
|
+
Sidekiq::Testing.fake!
|
70
|
+
when 'inline'
|
71
|
+
Sidekiq::Testing.inline!
|
72
|
+
else
|
73
|
+
defined?(Redis::Connection::Memory) ?
|
74
|
+
Sidekiq::Testing.disable! : Sidekiq::Testing.inline!
|
75
|
+
end
|
76
|
+
|
77
|
+
if Sidekiq::Testing.disabled?
|
78
|
+
Sidekiq.redis { |conn| conn.flushdb }
|
79
|
+
elsif Sidekiq::Testing.fake?
|
80
|
+
Sidekiq::Worker.clear_all
|
81
|
+
end
|
82
|
+
|
83
|
+
end
|
84
|
+
|
85
|
+
config.after(:all) do
|
86
|
+
## Stopping Redis
|
87
|
+
ps = %x(ps -A -o pid,command | grep '#{redis_command}' | grep -v grep).split($/)
|
88
|
+
pids = ps.map { |p| p.split(/\s+/).reject(&:empty?).first.to_i }
|
89
|
+
pids.each { |pid| Process.kill("TERM", pid) }
|
90
|
+
|
91
|
+
## Cleaning up
|
92
|
+
sleep 0.1
|
93
|
+
%x(rm -rf #{redis_dir}/*)
|
94
|
+
end
|
95
|
+
end
|
96
|
+
##
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-rate-limiter
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.1
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -10,7 +10,7 @@ authors:
|
|
10
10
|
autorequire:
|
11
11
|
bindir: bin
|
12
12
|
cert_chain: []
|
13
|
-
date: 2013-12-
|
13
|
+
date: 2013-12-03 00:00:00.000000000 Z
|
14
14
|
dependencies:
|
15
15
|
- !ruby/object:Gem::Dependency
|
16
16
|
name: pry
|
@@ -76,6 +76,38 @@ dependencies:
|
|
76
76
|
- - ! '>='
|
77
77
|
- !ruby/object:Gem::Version
|
78
78
|
version: '0'
|
79
|
+
- !ruby/object:Gem::Dependency
|
80
|
+
name: simplecov-rcov
|
81
|
+
requirement: !ruby/object:Gem::Requirement
|
82
|
+
none: false
|
83
|
+
requirements:
|
84
|
+
- - ! '>='
|
85
|
+
- !ruby/object:Gem::Version
|
86
|
+
version: '0'
|
87
|
+
type: :development
|
88
|
+
prerelease: false
|
89
|
+
version_requirements: !ruby/object:Gem::Requirement
|
90
|
+
none: false
|
91
|
+
requirements:
|
92
|
+
- - ! '>='
|
93
|
+
- !ruby/object:Gem::Version
|
94
|
+
version: '0'
|
95
|
+
- !ruby/object:Gem::Dependency
|
96
|
+
name: redis
|
97
|
+
requirement: !ruby/object:Gem::Requirement
|
98
|
+
none: false
|
99
|
+
requirements:
|
100
|
+
- - ! '>='
|
101
|
+
- !ruby/object:Gem::Version
|
102
|
+
version: '0'
|
103
|
+
type: :runtime
|
104
|
+
prerelease: false
|
105
|
+
version_requirements: !ruby/object:Gem::Requirement
|
106
|
+
none: false
|
107
|
+
requirements:
|
108
|
+
- - ! '>='
|
109
|
+
- !ruby/object:Gem::Version
|
110
|
+
version: '0'
|
79
111
|
- !ruby/object:Gem::Dependency
|
80
112
|
name: sidekiq
|
81
113
|
requirement: !ruby/object:Gem::Requirement
|
@@ -92,6 +124,22 @@ dependencies:
|
|
92
124
|
- - ! '>='
|
93
125
|
- !ruby/object:Gem::Version
|
94
126
|
version: '0'
|
127
|
+
- !ruby/object:Gem::Dependency
|
128
|
+
name: redis_rate_limiter
|
129
|
+
requirement: !ruby/object:Gem::Requirement
|
130
|
+
none: false
|
131
|
+
requirements:
|
132
|
+
- - ! '>='
|
133
|
+
- !ruby/object:Gem::Version
|
134
|
+
version: '0'
|
135
|
+
type: :runtime
|
136
|
+
prerelease: false
|
137
|
+
version_requirements: !ruby/object:Gem::Requirement
|
138
|
+
none: false
|
139
|
+
requirements:
|
140
|
+
- - ! '>='
|
141
|
+
- !ruby/object:Gem::Version
|
142
|
+
version: '0'
|
95
143
|
description: Rate-limit Sidekiq fetches by worker class
|
96
144
|
email:
|
97
145
|
- bwthomas@gmail.com
|
@@ -108,9 +156,14 @@ files:
|
|
108
156
|
- README.md
|
109
157
|
- Rakefile
|
110
158
|
- lib/sidekiq-rate-limiter.rb
|
159
|
+
- lib/sidekiq-rate-limiter/fetch.rb
|
160
|
+
- lib/sidekiq-rate-limiter/server.rb
|
111
161
|
- lib/sidekiq-rate-limiter/version.rb
|
112
162
|
- sidekiq-rate-limiter.gemspec
|
163
|
+
- spec/sidekiq-rate-limiter/fetch_spec.rb
|
164
|
+
- spec/sidekiq-rate-limiter/server_spec.rb
|
113
165
|
- spec/spec_helper.rb
|
166
|
+
- spec/support/redis/.keep
|
114
167
|
homepage: https://github.com/enova/sidekiq-rate-limiter
|
115
168
|
licenses:
|
116
169
|
- MIT
|
@@ -137,5 +190,8 @@ signing_key:
|
|
137
190
|
specification_version: 3
|
138
191
|
summary: Rate-limit Sidekiq fetches by worker class
|
139
192
|
test_files:
|
193
|
+
- spec/sidekiq-rate-limiter/fetch_spec.rb
|
194
|
+
- spec/sidekiq-rate-limiter/server_spec.rb
|
140
195
|
- spec/spec_helper.rb
|
196
|
+
- spec/support/redis/.keep
|
141
197
|
has_rdoc:
|