sidekiq-multi-redis-client 0.9.0
Sign up to get free protection for your applications and to get access to all the features.
- data/.gitignore +8 -0
- data/.travis.yml +18 -0
- data/CHANGELOG.md +4 -0
- data/Gemfile +2 -0
- data/LICENSE +22 -0
- data/README.md +74 -0
- data/Rakefile +11 -0
- data/lib/sidekiq-multi-redis-client.rb +10 -0
- data/lib/sidekiq-multi-redis-client/config.rb +54 -0
- data/lib/sidekiq-multi-redis-client/middleware.rb +16 -0
- data/lib/sidekiq-multi-redis-client/middleware/client/multi_redis.rb +64 -0
- data/lib/sidekiq-multi-redis-client/middleware/server/multi_redis.rb +21 -0
- data/lib/sidekiq-multi-redis-client/version.rb +3 -0
- data/sidekiq-multi-redis-client.gemspec +21 -0
- data/test/helper.rb +125 -0
- data/test/lib/sidekiq/test_configuration.rb +76 -0
- data/test/lib/sidekiq/test_failed_connections.rb +17 -0
- data/test/lib/sidekiq/test_middleware.rb +30 -0
- data/test/lib/sidekiq/test_redis_selection.rb +104 -0
- metadata +143 -0
data/.gitignore
ADDED
data/.travis.yml
ADDED
@@ -0,0 +1,18 @@
|
|
1
|
+
language: ruby
|
2
|
+
rvm:
|
3
|
+
- 1.9.3
|
4
|
+
- jruby-19mode
|
5
|
+
- rbx-19mode
|
6
|
+
- 2.0.0
|
7
|
+
branches:
|
8
|
+
only:
|
9
|
+
- master
|
10
|
+
notifications:
|
11
|
+
email:
|
12
|
+
recipients:
|
13
|
+
- kbedell@gmail.com
|
14
|
+
matrix:
|
15
|
+
allow_failures:
|
16
|
+
- rvm: jruby-19mode
|
17
|
+
- rvm: rbx-19mode
|
18
|
+
- rvm: 2.0.0
|
data/CHANGELOG.md
ADDED
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2012 Kevin Bedell
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,74 @@
|
|
1
|
+
# SidekiqMultiRedisClient [![Build Status](https://travis-ci.org/kevinzen/sidekiq-multi-redis-client.png?branch=master)](https://travis-ci.org/kevinzen/sidekiq-multi-redis-client)
|
2
|
+
|
3
|
+
The use case for this gem is to provide automatic load-balancing and fail-over of jobs submitted to multiple redis instances, each with its own Sidekiq workers.
|
4
|
+
|
5
|
+
In our case, we had two data centers -- each of which needs to be available as backup to the other in case the data center becomes unavailable. This gem allows all jobs to automatically get submitted to the remaining data center if one becomes unavailable -- or even if just one of the redis instances goes away. If one of the redis endpoints goes away, then jobs will be submitted to the other one.
|
6
|
+
|
7
|
+
It also spreads processing across both the redis instances.
|
8
|
+
|
9
|
+
The initial version of this gem/plugin as it now stands requires you to specify two redis instances (each of which will have its own worker(s)). The gem then 'round robins' submitting jobs to the two redis instances alternating between them.
|
10
|
+
|
11
|
+
|
12
|
+
NOTE on unit tests failing. While this sentence remains here in the readme, there are failing unit tests because the travis-ci environment does not have any redis instances connected to it for testing. The tests work on my local machine when I have multiple redis instances available. The tests (for now) assume that redis is available. This will be addressed in a future release soon.
|
13
|
+
|
14
|
+
|
15
|
+
## Installation
|
16
|
+
|
17
|
+
Add this line to your application's Gemfile:
|
18
|
+
|
19
|
+
gem 'sidekiq-multi-redis-client'
|
20
|
+
|
21
|
+
And then execute:
|
22
|
+
|
23
|
+
$ bundle
|
24
|
+
|
25
|
+
Or install it yourself as:
|
26
|
+
|
27
|
+
$ gem install sidekiq-multi-redis-client
|
28
|
+
|
29
|
+
## Usage
|
30
|
+
|
31
|
+
All that is required is that you specifically set the sidekiq option for *multi-redis-job* to true like below:
|
32
|
+
|
33
|
+
```ruby
|
34
|
+
|
35
|
+
class MultiRedisJob
|
36
|
+
include Sidekiq::Worker
|
37
|
+
sidekiq_options :multi_redis_job => true # The client code will now submit jobs to two different redis instances!
|
38
|
+
def perform(x)
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
```
|
43
|
+
|
44
|
+
Not including the ```:multi_redis_job => true``` line causes Sidekiq to function as it normally would.
|
45
|
+
|
46
|
+
Requiring the gem in your gemfile should be sufficient to enable multi-redis client capability.
|
47
|
+
|
48
|
+
## Configuring the two redis endpoints
|
49
|
+
|
50
|
+
This gem requires you to specify an ARRAY of redis Connections in your redis initializer, like so:
|
51
|
+
|
52
|
+
```
|
53
|
+
REDIS_1 = Sidekiq::RedisConnection.create(:url => "redis://localhost:6379", :namespace => 'testy')
|
54
|
+
REDIS_2 = Sidekiq::RedisConnection.create(:url => "redis://localhost:6380", :namespace => 'testy')
|
55
|
+
REDIS_CONNECTION_POOLS = [REDIS_1, REDIS_2]
|
56
|
+
|
57
|
+
SidekiqMultiRedisClient::Config.clear_redi_params
|
58
|
+
SidekiqMultiRedisClient::Config.redi = REDIS_1_CONNECTION_POOLS
|
59
|
+
```
|
60
|
+
|
61
|
+
This version of the gem requires two redis endpoints and 'round robins' jobs between them. Future versions of this gem will add more flexibility.
|
62
|
+
|
63
|
+
|
64
|
+
## Contributing
|
65
|
+
|
66
|
+
1. Fork it
|
67
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
68
|
+
3. Commit your changes (`git commit -am 'Add some feature'`)
|
69
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
70
|
+
5. Create new Pull Request
|
71
|
+
|
72
|
+
## Contributors
|
73
|
+
|
74
|
+
- https://github.com/kevinzen
|
data/Rakefile
ADDED
@@ -0,0 +1,10 @@
|
|
1
|
+
require 'yaml' if RUBY_VERSION.include?('2.0.0')
|
2
|
+
require 'sidekiq-multi-redis-client/middleware'
|
3
|
+
require 'sidekiq-multi-redis-client/version'
|
4
|
+
require 'sidekiq-multi-redis-client/config'
|
5
|
+
|
6
|
+
module SidekiqMultiRedisClient
|
7
|
+
def self.config
|
8
|
+
SidekiqMultiRedisClient::Config
|
9
|
+
end
|
10
|
+
end
|
@@ -0,0 +1,54 @@
|
|
1
|
+
module SidekiqMultiRedisClient
|
2
|
+
class Config
|
3
|
+
def self.multi_redis_job=(multi_redis_job)
|
4
|
+
@multi_redis_job = multi_redis_job
|
5
|
+
end
|
6
|
+
|
7
|
+
def self.multi_redis_job
|
8
|
+
@multi_redis_job
|
9
|
+
end
|
10
|
+
|
11
|
+
# Redises? An Array of redis ConnectionPools
|
12
|
+
#
|
13
|
+
def self.redi=(array_of_redis_connection_pools)
|
14
|
+
@current_redis = array_of_redis_connection_pools.nil? ? nil : array_of_redis_connection_pools[0]
|
15
|
+
@redi = array_of_redis_connection_pools
|
16
|
+
end
|
17
|
+
|
18
|
+
def self.redi
|
19
|
+
@redi
|
20
|
+
end
|
21
|
+
|
22
|
+
def self.current_redis
|
23
|
+
@current_redis
|
24
|
+
end
|
25
|
+
|
26
|
+
def self.current_redis=(redis_connection_pool)
|
27
|
+
@current_redis = redis_connection_pool
|
28
|
+
end
|
29
|
+
|
30
|
+
def self.next_redis_connection
|
31
|
+
if @redi.size == 1
|
32
|
+
return @redi.first
|
33
|
+
end
|
34
|
+
@current_redis = @redi.reject {|redis_conn| @current_redis == redis_conn}.first
|
35
|
+
end
|
36
|
+
|
37
|
+
def self.clear_redi_params
|
38
|
+
@current_redis = @redi = nil
|
39
|
+
end
|
40
|
+
|
41
|
+
def self.error_count
|
42
|
+
@error_count
|
43
|
+
end
|
44
|
+
|
45
|
+
def self.inc_error_count
|
46
|
+
@error_count = @error_count + 1
|
47
|
+
end
|
48
|
+
|
49
|
+
def self.reset_error_count
|
50
|
+
@error_count = 0
|
51
|
+
end
|
52
|
+
|
53
|
+
end
|
54
|
+
end
|
@@ -0,0 +1,16 @@
|
|
1
|
+
require 'sidekiq'
|
2
|
+
|
3
|
+
Sidekiq.configure_server do |config|
|
4
|
+
config.client_middleware do |chain|
|
5
|
+
require 'sidekiq-multi-redis-client/middleware/client/multi_redis'
|
6
|
+
chain.add SidekiqMultiRedisClient::Middleware::Client::MultiRedis
|
7
|
+
end
|
8
|
+
|
9
|
+
end
|
10
|
+
|
11
|
+
Sidekiq.configure_client do |config|
|
12
|
+
config.client_middleware do |chain|
|
13
|
+
require 'sidekiq-multi-redis-client/middleware/client/multi_redis'
|
14
|
+
chain.add SidekiqMultiRedisClient::Middleware::Client::MultiRedis
|
15
|
+
end
|
16
|
+
end
|
@@ -0,0 +1,64 @@
|
|
1
|
+
require 'digest'
|
2
|
+
|
3
|
+
module SidekiqMultiRedisClient
|
4
|
+
module Middleware
|
5
|
+
module Client
|
6
|
+
class MultiRedis
|
7
|
+
|
8
|
+
def call(worker_class, item, queue)
|
9
|
+
|
10
|
+
klass = worker_class_constantize(worker_class)
|
11
|
+
enabled = klass.get_sidekiq_options['multi_redis_job']
|
12
|
+
|
13
|
+
if enabled
|
14
|
+
|
15
|
+
point_to_next_redis
|
16
|
+
|
17
|
+
# If the redis we just pointed to isn't available, move to the next one.
|
18
|
+
# If it's not there, we error. Right now we handle only 1 or 2 redis endpoints.
|
19
|
+
# If they're both gone, erroring is the right thing to do.
|
20
|
+
if is_redis_conn_there? == false
|
21
|
+
point_to_next_redis
|
22
|
+
end
|
23
|
+
|
24
|
+
yield
|
25
|
+
|
26
|
+
else
|
27
|
+
yield
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
|
32
|
+
protected
|
33
|
+
|
34
|
+
def worker_class_constantize(worker_class)
|
35
|
+
if worker_class.is_a?(String)
|
36
|
+
worker_class.constantize rescue worker_class
|
37
|
+
else
|
38
|
+
worker_class
|
39
|
+
end
|
40
|
+
end
|
41
|
+
|
42
|
+
def is_redis_conn_there?
|
43
|
+
is_there = false
|
44
|
+
begin
|
45
|
+
Sidekiq.redis do |c|
|
46
|
+
is_there = (c.ping == 'PONG')
|
47
|
+
end
|
48
|
+
rescue
|
49
|
+
Sidekiq.logger.debug { "Sidekiq - MultiRedis: Redis endpoint errored responding to ping." }
|
50
|
+
end
|
51
|
+
is_there
|
52
|
+
end
|
53
|
+
|
54
|
+
def point_to_next_redis
|
55
|
+
Sidekiq.configure_client do |config|
|
56
|
+
next_redis_config = SidekiqMultiRedisClient::Config.next_redis_connection
|
57
|
+
config.redis = next_redis_config
|
58
|
+
end
|
59
|
+
end
|
60
|
+
|
61
|
+
end
|
62
|
+
end
|
63
|
+
end
|
64
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
require 'digest'
|
2
|
+
|
3
|
+
module SidekiqMultiRedisClient
|
4
|
+
module Middleware
|
5
|
+
module Server
|
6
|
+
class MultiRedis
|
7
|
+
def call(worker, item, queue)
|
8
|
+
yield
|
9
|
+
ensure
|
10
|
+
end
|
11
|
+
|
12
|
+
protected
|
13
|
+
|
14
|
+
def logger
|
15
|
+
Sidekiq.logger
|
16
|
+
end
|
17
|
+
|
18
|
+
end
|
19
|
+
end
|
20
|
+
end
|
21
|
+
end
|
@@ -0,0 +1,21 @@
|
|
1
|
+
# -*- encoding: utf-8 -*-
|
2
|
+
require File.expand_path('../lib/sidekiq-multi-redis-client/version', __FILE__)
|
3
|
+
|
4
|
+
Gem::Specification.new do |gem|
|
5
|
+
gem.authors = ["Kevin Bedell"]
|
6
|
+
gem.email = ["kbedell@gmail.com"]
|
7
|
+
gem.description = gem.summary = "Allow sidekiq clients to send jobs to multiple redis instances"
|
8
|
+
gem.homepage = "http://kbedell.github.com/sidekiq-multi-redis-client"
|
9
|
+
gem.license = "MIT"
|
10
|
+
|
11
|
+
# gem.executables = ['']
|
12
|
+
gem.files = `git ls-files`.split("\n")
|
13
|
+
gem.test_files = `git ls-files -- test/*`.split("\n")
|
14
|
+
gem.name = "sidekiq-multi-redis-client"
|
15
|
+
gem.require_paths = ["lib"]
|
16
|
+
gem.version = SidekiqMultiRedisClient::VERSION
|
17
|
+
gem.add_dependency 'sidekiq', '~> 2.6'
|
18
|
+
gem.add_development_dependency 'minitest', '~> 3'
|
19
|
+
gem.add_development_dependency 'rake'
|
20
|
+
gem.add_development_dependency 'simplecov'
|
21
|
+
end
|
data/test/helper.rb
ADDED
@@ -0,0 +1,125 @@
|
|
1
|
+
ENV['RACK_ENV'] = ENV['RAILS_ENV'] = 'test'
|
2
|
+
if ENV.has_key?("SIMPLECOV")
|
3
|
+
require 'simplecov'
|
4
|
+
SimpleCov.start
|
5
|
+
end
|
6
|
+
require 'minitest/unit'
|
7
|
+
require 'minitest/pride'
|
8
|
+
require 'minitest/autorun'
|
9
|
+
|
10
|
+
require "sidekiq"
|
11
|
+
require 'sidekiq/util'
|
12
|
+
require 'sidekiq/redis_connection'
|
13
|
+
|
14
|
+
require 'sidekiq-multi-redis-client'
|
15
|
+
|
16
|
+
Sidekiq.logger.level = Logger::ERROR
|
17
|
+
|
18
|
+
class MultiRedisJob
|
19
|
+
include Sidekiq::Worker
|
20
|
+
sidekiq_options :multi_redis_job => true
|
21
|
+
def perform(x)
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
class PlainRedisJob
|
26
|
+
include Sidekiq::Worker
|
27
|
+
def perform(x)
|
28
|
+
end
|
29
|
+
end
|
30
|
+
|
31
|
+
class SingleRedisJob
|
32
|
+
include Sidekiq::Worker
|
33
|
+
sidekiq_options :multi_redis_job => false
|
34
|
+
def perform(x)
|
35
|
+
end
|
36
|
+
end
|
37
|
+
|
38
|
+
def worker_class_constantize(worker_class)
|
39
|
+
if worker_class.is_a?(String)
|
40
|
+
worker_class.constantize rescue worker_class
|
41
|
+
else
|
42
|
+
worker_class
|
43
|
+
end
|
44
|
+
end
|
45
|
+
|
46
|
+
def mock_redis
|
47
|
+
@redis = Minitest::Mock.new
|
48
|
+
def @redis.multi; [yield] * 2 if block_given?; end
|
49
|
+
def @redis.set(*); true; end
|
50
|
+
def @redis.sadd(*); true; end
|
51
|
+
def @redis.srem(*); true; end
|
52
|
+
def @redis.get(*); nil; end
|
53
|
+
def @redis.del(*); nil; end
|
54
|
+
def @redis.incrby(*); nil; end
|
55
|
+
def @redis.setex(*); true; end
|
56
|
+
def @redis.expire(*); true; end
|
57
|
+
def @redis.watch(*); true; end
|
58
|
+
def @redis.with_connection; yield self; end
|
59
|
+
def @redis.with; yield self; end
|
60
|
+
def @redis.exec; true; end
|
61
|
+
def @redis.ping; 'PONG'; end
|
62
|
+
Sidekiq.instance_variable_set(:@redis, @redis)
|
63
|
+
end
|
64
|
+
|
65
|
+
def unmock_redis
|
66
|
+
Sidekiq.instance_variable_set(:@redis, REDIS)
|
67
|
+
end
|
68
|
+
|
69
|
+
#############
|
70
|
+
#
|
71
|
+
# Below here setup the redis connections you are testing with. Please don't check in
|
72
|
+
# changes to this section.
|
73
|
+
|
74
|
+
# I use these for testing. Change for your situation if needed.
|
75
|
+
REDIS_1 = Sidekiq::RedisConnection.create(:url => "redis://localhost:6379/15", :namespace => 'testy')
|
76
|
+
REDIS_2 = Sidekiq::RedisConnection.create(:url => "redis://localhost:6380/15", :namespace => 'testy')
|
77
|
+
|
78
|
+
# This one doesn't exist
|
79
|
+
BAD_REDIS = Sidekiq::RedisConnection.create(:url => "redis://http://10.10.10.1:9999/1", :namespace => 'testy')
|
80
|
+
|
81
|
+
REDIS_1_CONNECTION_POOLS = [REDIS_1]
|
82
|
+
REDIS_2_CONNECTION_POOLS = [REDIS_1, REDIS_2]
|
83
|
+
|
84
|
+
REDIS_ONE_BAD = [BAD_REDIS]
|
85
|
+
REDIS_ONE_GOOD_ONE_BAD = [REDIS_1, BAD_REDIS]
|
86
|
+
|
87
|
+
def initialize_redis
|
88
|
+
Sidekiq.configure_client do |config|
|
89
|
+
next_redis_config = SidekiqMultiRedisClient::Config.next_redis_connection
|
90
|
+
config.redis = next_redis_config
|
91
|
+
end
|
92
|
+
end
|
93
|
+
|
94
|
+
def setup_one_redis_conn
|
95
|
+
SidekiqMultiRedisClient::Config.clear_redi_params
|
96
|
+
SidekiqMultiRedisClient::Config.redi = REDIS_1_CONNECTION_POOLS
|
97
|
+
end
|
98
|
+
|
99
|
+
def setup_two_redis_conns
|
100
|
+
SidekiqMultiRedisClient::Config.clear_redi_params
|
101
|
+
SidekiqMultiRedisClient::Config.redi = REDIS_2_CONNECTION_POOLS
|
102
|
+
initialize_redis
|
103
|
+
end
|
104
|
+
|
105
|
+
def setup_redis_one_bad
|
106
|
+
SidekiqMultiRedisClient::Config.clear_redi_params
|
107
|
+
SidekiqMultiRedisClient::Config.redi = REDIS_ONE_BAD
|
108
|
+
initialize_redis
|
109
|
+
end
|
110
|
+
|
111
|
+
def setup_redis_one_good_one_bad
|
112
|
+
SidekiqMultiRedisClient::Config.clear_redi_params
|
113
|
+
SidekiqMultiRedisClient::Config.redi = REDIS_ONE_GOOD_ONE_BAD
|
114
|
+
initialize_redis
|
115
|
+
end
|
116
|
+
|
117
|
+
# Method for flushing redis queues between tests.
|
118
|
+
# 'rescue' is needed in case they don't really exist.
|
119
|
+
def flush_redis_queues (array_of_connection_pools)
|
120
|
+
#array_of_connection_pools.each { |redis|
|
121
|
+
# Sidekiq.redis = redis
|
122
|
+
#Sidekiq.redis {|c| c.flushdb rescue nil}
|
123
|
+
#}
|
124
|
+
end
|
125
|
+
|
@@ -0,0 +1,76 @@
|
|
1
|
+
require 'helper'
|
2
|
+
|
3
|
+
class TestConfiguration < MiniTest::Unit::TestCase
|
4
|
+
describe 'with real redis' do
|
5
|
+
before do
|
6
|
+
end
|
7
|
+
|
8
|
+
describe "configuration options should be identifiable" do
|
9
|
+
it 'determines if the job is tagged to use multi_redis_job' do
|
10
|
+
enabled = MultiRedisJob.get_sidekiq_options['multi_redis_job']
|
11
|
+
assert enabled
|
12
|
+
end
|
13
|
+
|
14
|
+
it 'determines if the job is tagged to use multi_redis_job' do
|
15
|
+
enabled = SingleRedisJob.get_sidekiq_options['multi_redis_job']
|
16
|
+
assert !enabled
|
17
|
+
end
|
18
|
+
|
19
|
+
it "determines jobs that don't include it arent multi_redis_job" do
|
20
|
+
enabled = PlainRedisJob.get_sidekiq_options['multi_redis_job']
|
21
|
+
assert !enabled
|
22
|
+
end
|
23
|
+
end
|
24
|
+
|
25
|
+
describe "figure out how many REDIS endpoints there are" do
|
26
|
+
before "reset" do
|
27
|
+
SidekiqMultiRedisClient::Config.redi = nil
|
28
|
+
end
|
29
|
+
|
30
|
+
it 'knows when there are no redis instances setup' do
|
31
|
+
refute SidekiqMultiRedisClient::Config.redi
|
32
|
+
end
|
33
|
+
|
34
|
+
it 'correctly identifies the number of redis instances after setup' do
|
35
|
+
setup_one_redis_conn
|
36
|
+
assert SidekiqMultiRedisClient::Config.redi.size, 1
|
37
|
+
|
38
|
+
setup_two_redis_conns
|
39
|
+
assert SidekiqMultiRedisClient::Config.redi.size, 2
|
40
|
+
end
|
41
|
+
|
42
|
+
it 'allows you to set / reset config parameters' do
|
43
|
+
setup_two_redis_conns
|
44
|
+
|
45
|
+
redi = SidekiqMultiRedisClient::Config.redi
|
46
|
+
assert redi.size, 2
|
47
|
+
|
48
|
+
SidekiqMultiRedisClient::Config.redi = nil
|
49
|
+
refute SidekiqMultiRedisClient::Config.redi
|
50
|
+
end
|
51
|
+
|
52
|
+
it 'allows you to reset config parameters directly' do
|
53
|
+
setup_two_redis_conns
|
54
|
+
|
55
|
+
redi = SidekiqMultiRedisClient::Config.clear_redi_params
|
56
|
+
refute SidekiqMultiRedisClient::Config.redi
|
57
|
+
end
|
58
|
+
|
59
|
+
end
|
60
|
+
|
61
|
+
describe "configuration options should be identifiable from the string name of the class" do
|
62
|
+
it 'determines if the job is tagged to use multi_redis_job' do
|
63
|
+
klass = worker_class_constantize("MultiRedisJob")
|
64
|
+
enabled = klass.get_sidekiq_options['multi_redis_job']
|
65
|
+
assert enabled
|
66
|
+
end
|
67
|
+
|
68
|
+
it 'determines if the job is tagged to use multi_redis_job' do
|
69
|
+
klass = worker_class_constantize("PlainRedisJob")
|
70
|
+
enabled = klass.get_sidekiq_options['multi_redis_job']
|
71
|
+
refute enabled
|
72
|
+
end
|
73
|
+
end
|
74
|
+
|
75
|
+
end
|
76
|
+
end
|
@@ -0,0 +1,17 @@
|
|
1
|
+
require 'helper'
|
2
|
+
|
3
|
+
class TestFailedConnections < MiniTest::Unit::TestCase
|
4
|
+
describe 'with real redis that fails sometimes' do
|
5
|
+
before do
|
6
|
+
end
|
7
|
+
|
8
|
+
describe "should correctly know when one of the redis conns is bad" do
|
9
|
+
it 'should know there are two redis connections' do
|
10
|
+
setup_redis_one_bad
|
11
|
+
mw = SidekiqMultiRedisClient::Middleware::Client::MultiRedis.new
|
12
|
+
mw.call(MultiRedisJob, nil, nil) { }
|
13
|
+
end
|
14
|
+
end
|
15
|
+
|
16
|
+
end
|
17
|
+
end
|
@@ -0,0 +1,30 @@
|
|
1
|
+
describe 'multi redis middleware' do
|
2
|
+
before do
|
3
|
+
mock_redis
|
4
|
+
setup_two_redis_conns
|
5
|
+
end
|
6
|
+
|
7
|
+
after do
|
8
|
+
#unmock_redis
|
9
|
+
end
|
10
|
+
|
11
|
+
it 'changes the redis client for each call' do
|
12
|
+
redis_1 = Sidekiq.redis { |c| c.client.location }
|
13
|
+
|
14
|
+
mw = SidekiqMultiRedisClient::Middleware::Client::MultiRedis.new
|
15
|
+
mw.call(MultiRedisJob, nil, nil) { }
|
16
|
+
|
17
|
+
redis_2 = Sidekiq.redis { |c| c.client.location }
|
18
|
+
refute_equal redis_1, redis_2
|
19
|
+
end
|
20
|
+
it 'changes the redis client for each call over and over' do
|
21
|
+
mw = SidekiqMultiRedisClient::Middleware::Client::MultiRedis.new
|
22
|
+
redis_counts = {"localhost:6379" => 0, "localhost:6380" => 0}
|
23
|
+
100.times {
|
24
|
+
mw.call(MultiRedisJob, nil, nil) { }
|
25
|
+
new_redis = Sidekiq.redis { |c| c.client.location }
|
26
|
+
redis_counts[new_redis] = redis_counts[new_redis] + 1
|
27
|
+
}
|
28
|
+
assert_equal redis_counts["localhost:6379"], redis_counts["localhost:6380"]
|
29
|
+
end
|
30
|
+
end
|
@@ -0,0 +1,104 @@
|
|
1
|
+
require 'helper'
|
2
|
+
|
3
|
+
class TestRedisSelection < MiniTest::Unit::TestCase
|
4
|
+
describe 'with real redis' do
|
5
|
+
before do
|
6
|
+
end
|
7
|
+
|
8
|
+
|
9
|
+
describe "should correctly work with a single endpoint" do
|
10
|
+
|
11
|
+
before do
|
12
|
+
setup_one_redis_conn
|
13
|
+
end
|
14
|
+
|
15
|
+
it 'should know there is only one redis connections' do
|
16
|
+
assert SidekiqMultiRedisClient::Config.redi.size, 1
|
17
|
+
end
|
18
|
+
|
19
|
+
it 'should set the first redis instance as the current one' do
|
20
|
+
assert SidekiqMultiRedisClient::Config.current_redis, REDIS_1_CONNECTION_POOLS[0]
|
21
|
+
end
|
22
|
+
|
23
|
+
it "should get the correct values for both of the redis instances" do
|
24
|
+
assert SidekiqMultiRedisClient::Config.redi[0], REDIS_1_CONNECTION_POOLS[0]
|
25
|
+
end
|
26
|
+
|
27
|
+
it "should know when a new redis is fetched that it is the current one " do
|
28
|
+
next_conn = SidekiqMultiRedisClient::Config.next_redis_connection
|
29
|
+
curr_conn = SidekiqMultiRedisClient::Config.current_redis
|
30
|
+
|
31
|
+
assert next_conn, curr_conn
|
32
|
+
end
|
33
|
+
|
34
|
+
it "should always return the same redis when asked" do
|
35
|
+
1..100.times do
|
36
|
+
|
37
|
+
# do some extras to make sure we're not hitting different boundaries
|
38
|
+
(1 + rand(6)).times { SidekiqMultiRedisClient::Config.next_redis_connection }
|
39
|
+
|
40
|
+
# then grab the next two
|
41
|
+
conn1 = SidekiqMultiRedisClient::Config.next_redis_connection
|
42
|
+
conn2 = SidekiqMultiRedisClient::Config.next_redis_connection
|
43
|
+
|
44
|
+
# Make sure they are in the array we initialized with
|
45
|
+
assert_includes REDIS_1_CONNECTION_POOLS, conn1
|
46
|
+
assert_includes REDIS_1_CONNECTION_POOLS, conn2
|
47
|
+
|
48
|
+
# Make sure they are the same
|
49
|
+
assert conn1, conn2
|
50
|
+
end
|
51
|
+
end
|
52
|
+
|
53
|
+
end
|
54
|
+
|
55
|
+
describe "should correctly identify different redis connections" do
|
56
|
+
|
57
|
+
before do
|
58
|
+
setup_two_redis_conns
|
59
|
+
end
|
60
|
+
|
61
|
+
it 'should know there are two redis connections' do
|
62
|
+
assert SidekiqMultiRedisClient::Config.redi.size, 2
|
63
|
+
end
|
64
|
+
|
65
|
+
it 'should set the first redis instance as the current one' do
|
66
|
+
assert SidekiqMultiRedisClient::Config.current_redis, REDIS_2_CONNECTION_POOLS[0]
|
67
|
+
end
|
68
|
+
|
69
|
+
it "should get the correct values for both of the redis instances" do
|
70
|
+
assert SidekiqMultiRedisClient::Config.redi[0], REDIS_2_CONNECTION_POOLS[0]
|
71
|
+
assert SidekiqMultiRedisClient::Config.redi[1], REDIS_2_CONNECTION_POOLS[1]
|
72
|
+
end
|
73
|
+
|
74
|
+
it "should know when a new redis is fetched that it is the current one " do
|
75
|
+
next_conn = SidekiqMultiRedisClient::Config.next_redis_connection
|
76
|
+
curr_conn = SidekiqMultiRedisClient::Config.current_redis
|
77
|
+
|
78
|
+
assert next_conn, curr_conn
|
79
|
+
end
|
80
|
+
|
81
|
+
it "should get alternately return different ones when asked" do
|
82
|
+
1..100.times do
|
83
|
+
|
84
|
+
# do some extras to make sure we're not hitting different boundaries
|
85
|
+
(1 + rand(6)).times { SidekiqMultiRedisClient::Config.next_redis_connection }
|
86
|
+
|
87
|
+
# then grab the next two
|
88
|
+
conn1 = SidekiqMultiRedisClient::Config.next_redis_connection
|
89
|
+
conn2 = SidekiqMultiRedisClient::Config.next_redis_connection
|
90
|
+
|
91
|
+
# Make sure they are in the array we initialized with
|
92
|
+
assert_includes REDIS_2_CONNECTION_POOLS, conn1
|
93
|
+
assert_includes REDIS_2_CONNECTION_POOLS, conn2
|
94
|
+
|
95
|
+
# Make sure they are different
|
96
|
+
refute_equal conn1, conn2
|
97
|
+
end
|
98
|
+
end
|
99
|
+
|
100
|
+
end
|
101
|
+
|
102
|
+
|
103
|
+
end
|
104
|
+
end
|
metadata
ADDED
@@ -0,0 +1,143 @@
|
|
1
|
+
--- !ruby/object:Gem::Specification
|
2
|
+
name: sidekiq-multi-redis-client
|
3
|
+
version: !ruby/object:Gem::Version
|
4
|
+
prerelease:
|
5
|
+
version: 0.9.0
|
6
|
+
platform: ruby
|
7
|
+
authors:
|
8
|
+
- Kevin Bedell
|
9
|
+
autorequire:
|
10
|
+
bindir: bin
|
11
|
+
cert_chain: []
|
12
|
+
date: 2013-12-17 00:00:00.000000000 Z
|
13
|
+
dependencies:
|
14
|
+
- !ruby/object:Gem::Dependency
|
15
|
+
name: sidekiq
|
16
|
+
version_requirements: !ruby/object:Gem::Requirement
|
17
|
+
requirements:
|
18
|
+
- - "~>"
|
19
|
+
- !ruby/object:Gem::Version
|
20
|
+
version: '2.6'
|
21
|
+
none: false
|
22
|
+
requirement: !ruby/object:Gem::Requirement
|
23
|
+
requirements:
|
24
|
+
- - "~>"
|
25
|
+
- !ruby/object:Gem::Version
|
26
|
+
version: '2.6'
|
27
|
+
none: false
|
28
|
+
prerelease: false
|
29
|
+
type: :runtime
|
30
|
+
- !ruby/object:Gem::Dependency
|
31
|
+
name: minitest
|
32
|
+
version_requirements: !ruby/object:Gem::Requirement
|
33
|
+
requirements:
|
34
|
+
- - "~>"
|
35
|
+
- !ruby/object:Gem::Version
|
36
|
+
version: !binary |-
|
37
|
+
Mw==
|
38
|
+
none: false
|
39
|
+
requirement: !ruby/object:Gem::Requirement
|
40
|
+
requirements:
|
41
|
+
- - "~>"
|
42
|
+
- !ruby/object:Gem::Version
|
43
|
+
version: !binary |-
|
44
|
+
Mw==
|
45
|
+
none: false
|
46
|
+
prerelease: false
|
47
|
+
type: :development
|
48
|
+
- !ruby/object:Gem::Dependency
|
49
|
+
name: rake
|
50
|
+
version_requirements: !ruby/object:Gem::Requirement
|
51
|
+
requirements:
|
52
|
+
- - ">="
|
53
|
+
- !ruby/object:Gem::Version
|
54
|
+
version: !binary |-
|
55
|
+
MA==
|
56
|
+
none: false
|
57
|
+
requirement: !ruby/object:Gem::Requirement
|
58
|
+
requirements:
|
59
|
+
- - ">="
|
60
|
+
- !ruby/object:Gem::Version
|
61
|
+
version: !binary |-
|
62
|
+
MA==
|
63
|
+
none: false
|
64
|
+
prerelease: false
|
65
|
+
type: :development
|
66
|
+
- !ruby/object:Gem::Dependency
|
67
|
+
name: simplecov
|
68
|
+
version_requirements: !ruby/object:Gem::Requirement
|
69
|
+
requirements:
|
70
|
+
- - ">="
|
71
|
+
- !ruby/object:Gem::Version
|
72
|
+
version: !binary |-
|
73
|
+
MA==
|
74
|
+
none: false
|
75
|
+
requirement: !ruby/object:Gem::Requirement
|
76
|
+
requirements:
|
77
|
+
- - ">="
|
78
|
+
- !ruby/object:Gem::Version
|
79
|
+
version: !binary |-
|
80
|
+
MA==
|
81
|
+
none: false
|
82
|
+
prerelease: false
|
83
|
+
type: :development
|
84
|
+
description: Allow sidekiq clients to send jobs to multiple redis instances
|
85
|
+
email:
|
86
|
+
- kbedell@gmail.com
|
87
|
+
executables: []
|
88
|
+
extensions: []
|
89
|
+
extra_rdoc_files: []
|
90
|
+
files:
|
91
|
+
- ".gitignore"
|
92
|
+
- ".travis.yml"
|
93
|
+
- CHANGELOG.md
|
94
|
+
- Gemfile
|
95
|
+
- LICENSE
|
96
|
+
- README.md
|
97
|
+
- Rakefile
|
98
|
+
- lib/sidekiq-multi-redis-client.rb
|
99
|
+
- lib/sidekiq-multi-redis-client/config.rb
|
100
|
+
- lib/sidekiq-multi-redis-client/middleware.rb
|
101
|
+
- lib/sidekiq-multi-redis-client/middleware/client/multi_redis.rb
|
102
|
+
- lib/sidekiq-multi-redis-client/middleware/server/multi_redis.rb
|
103
|
+
- lib/sidekiq-multi-redis-client/version.rb
|
104
|
+
- sidekiq-multi-redis-client.gemspec
|
105
|
+
- test/helper.rb
|
106
|
+
- test/lib/sidekiq/test_configuration.rb
|
107
|
+
- test/lib/sidekiq/test_failed_connections.rb
|
108
|
+
- test/lib/sidekiq/test_middleware.rb
|
109
|
+
- test/lib/sidekiq/test_redis_selection.rb
|
110
|
+
homepage: http://kbedell.github.com/sidekiq-multi-redis-client
|
111
|
+
licenses:
|
112
|
+
- MIT
|
113
|
+
post_install_message:
|
114
|
+
rdoc_options: []
|
115
|
+
require_paths:
|
116
|
+
- lib
|
117
|
+
required_ruby_version: !ruby/object:Gem::Requirement
|
118
|
+
requirements:
|
119
|
+
- - ">="
|
120
|
+
- !ruby/object:Gem::Version
|
121
|
+
version: !binary |-
|
122
|
+
MA==
|
123
|
+
none: false
|
124
|
+
required_rubygems_version: !ruby/object:Gem::Requirement
|
125
|
+
requirements:
|
126
|
+
- - ">="
|
127
|
+
- !ruby/object:Gem::Version
|
128
|
+
version: !binary |-
|
129
|
+
MA==
|
130
|
+
none: false
|
131
|
+
requirements: []
|
132
|
+
rubyforge_project:
|
133
|
+
rubygems_version: 1.8.24
|
134
|
+
signing_key:
|
135
|
+
specification_version: 3
|
136
|
+
summary: Allow sidekiq clients to send jobs to multiple redis instances
|
137
|
+
test_files:
|
138
|
+
- test/helper.rb
|
139
|
+
- test/lib/sidekiq/test_configuration.rb
|
140
|
+
- test/lib/sidekiq/test_failed_connections.rb
|
141
|
+
- test/lib/sidekiq/test_middleware.rb
|
142
|
+
- test/lib/sidekiq/test_redis_selection.rb
|
143
|
+
has_rdoc:
|