redis_failover 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/.gitignore +17 -0
- data/Changes.md +4 -0
- data/Gemfile +2 -0
- data/LICENSE +22 -0
- data/README.md +106 -0
- data/Rakefile +9 -0
- data/bin/redis_failover_server +6 -0
- data/lib/redis_failover.rb +16 -0
- data/lib/redis_failover/cli.rb +47 -0
- data/lib/redis_failover/client.rb +198 -0
- data/lib/redis_failover/errors.rb +25 -0
- data/lib/redis_failover/node.rb +101 -0
- data/lib/redis_failover/node_manager.rb +151 -0
- data/lib/redis_failover/node_watcher.rb +46 -0
- data/lib/redis_failover/runner.rb +31 -0
- data/lib/redis_failover/server.rb +13 -0
- data/lib/redis_failover/util.rb +29 -0
- data/lib/redis_failover/version.rb +3 -0
- data/redis_failover.gemspec +25 -0
- data/spec/cli_spec.rb +39 -0
- data/spec/client_spec.rb +62 -0
- data/spec/node_manager_spec.rb +76 -0
- data/spec/node_spec.rb +76 -0
- data/spec/node_watcher_spec.rb +43 -0
- data/spec/spec_helper.rb +15 -0
- data/spec/support/node_manager_stub.rb +43 -0
- data/spec/support/redis_stub.rb +82 -0
- data/spec/util_spec.rb +11 -0
- metadata +157 -0
data/.gitignore
ADDED
data/Changes.md
ADDED
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,22 @@
|
|
1
|
+
Copyright (c) 2012 Ryan LeCompte
|
2
|
+
|
3
|
+
MIT License
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining
|
6
|
+
a copy of this software and associated documentation files (the
|
7
|
+
"Software"), to deal in the Software without restriction, including
|
8
|
+
without limitation the rights to use, copy, modify, merge, publish,
|
9
|
+
distribute, sublicense, and/or sell copies of the Software, and to
|
10
|
+
permit persons to whom the Software is furnished to do so, subject to
|
11
|
+
the following conditions:
|
12
|
+
|
13
|
+
The above copyright notice and this permission notice shall be
|
14
|
+
included in all copies or substantial portions of the Software.
|
15
|
+
|
16
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
17
|
+
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
|
18
|
+
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
19
|
+
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
|
20
|
+
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
|
21
|
+
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
|
22
|
+
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
|
data/README.md
ADDED
@@ -0,0 +1,106 @@
|
|
1
|
+
# Redis Failover Client/Server
|
2
|
+
|
3
|
+
Redis Failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not provide
|
4
|
+
an automatic failover capability when configured for master/slave replication. When the master node dies,
|
5
|
+
a new master must be manually brought online and assigned as the slave's new master. This manual
|
6
|
+
switch-over is not desirable in high traffic sites where Redis is a critical part of the overall
|
7
|
+
architecture. The existing standard Redis client for Ruby also only supports configuration for a single
|
8
|
+
Redis server. When using master/slave replication, it is desirable to have all writes go to the
|
9
|
+
master, and all reads go to one of the N configured slaves.
|
10
|
+
|
11
|
+
This gem attempts to address both the server and client problems. A redis failover server runs as a background
|
12
|
+
daemon and monitors all of your configured master/slave nodes. When the server starts up, it
|
13
|
+
automatically discovers who is the master and who are the slaves. Watchers are setup for each of
|
14
|
+
the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unreachable" state.
|
15
|
+
If the node that went offline was the master, then one of the slaves will be promoted as the new master.
|
16
|
+
All existing slaves will be automatically reconfigured to point to the new master for replication.
|
17
|
+
All nodes marked as unreachable will be periodically checked to see if they have been brought back online.
|
18
|
+
If so, the newly reachable nodes will be configured as slaves and brought back into the list of live
|
19
|
+
servers. Note that detection of a node going down should be nearly instantaneous, since the mechanism
|
20
|
+
used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
|
21
|
+
immediately when the node actually goes offline.
|
22
|
+
|
23
|
+
This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
|
24
|
+
with a single host/port pair that points to redis failover server. The client will automatically
|
25
|
+
connect to the server to find out the current state of the world (i.e., who's the current master and
|
26
|
+
who are the current slaves). The client also acts as a load balancer in that it will automatically
|
27
|
+
dispatch Redis read operations to one of N slaves, and Redis write operations to the master.
|
28
|
+
If it fails to communicate with any node, it will go back and ask the server for the current list of
|
29
|
+
available servers, and then optionally retry the operation.
|
30
|
+
|
31
|
+
## Installation
|
32
|
+
|
33
|
+
Add this line to your application's Gemfile:
|
34
|
+
|
35
|
+
gem 'redis_failover'
|
36
|
+
|
37
|
+
And then execute:
|
38
|
+
|
39
|
+
$ bundle
|
40
|
+
|
41
|
+
Or install it yourself as:
|
42
|
+
|
43
|
+
$ gem install redis_failover
|
44
|
+
|
45
|
+
## Server Usage
|
46
|
+
|
47
|
+
The redis failover server is a simple process that should be run as a background daemon. The server supports the
|
48
|
+
following options:
|
49
|
+
|
50
|
+
Usage: redis_failover_server [OPTIONS]
|
51
|
+
-P, --port port Server port
|
52
|
+
-p, --password password Redis password
|
53
|
+
-n, --nodes nodes Comma-separated redis host:port pairs
|
54
|
+
--max-failures count Max failures before server marks node unreachable (default 3)
|
55
|
+
-h, --help Display all options
|
56
|
+
|
57
|
+
To start the server for a simple master/slave configuration, use the following:
|
58
|
+
|
59
|
+
redis_failover_server -P 3000 -n localhost:6379,localhost:6380
|
60
|
+
|
61
|
+
The server will automatically figure out who is the master and who is the slave upon startup. Note that it is
|
62
|
+
a good idea to monitor the redis failover server process with a tool like Monit to ensure that it is restarted
|
63
|
+
in the case of a failure.
|
64
|
+
|
65
|
+
## Client Usage
|
66
|
+
|
67
|
+
The redis failover client must be used in conjunction with a running redis failover server. The
|
68
|
+
client supports various configuration options, however the two mandatory options are the host
|
69
|
+
and port of the redis failover server:
|
70
|
+
|
71
|
+
client = RedisFailover::Client.new(:host => 'localhost', :port => 3000)
|
72
|
+
|
73
|
+
The client actually employs the common redis and redis-namespace gems underneath, so this should be
|
74
|
+
a drop-in replacement for your existing pure redis client usage.
|
75
|
+
|
76
|
+
The full set of options that can be passed to RedisFailover::Client are:
|
77
|
+
|
78
|
+
:host - redis failover server host (required)
|
79
|
+
:port - redis failover server port (required)
|
80
|
+
:password - optional password for redis nodes
|
81
|
+
:namespace - optional namespace for redis nodes
|
82
|
+
:logger - optional logger override
|
83
|
+
:retry_failure - indicate if failures should be retried (default true)
|
84
|
+
:max_retries - max retries for a failure (default 3)
|
85
|
+
|
86
|
+
## Resources
|
87
|
+
|
88
|
+
To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).
|
89
|
+
|
90
|
+
## License
|
91
|
+
|
92
|
+
Please see LICENSE for licensing details.
|
93
|
+
|
94
|
+
## Author
|
95
|
+
|
96
|
+
Ryan LeCompte
|
97
|
+
|
98
|
+
[@ryanlecompte](https://twitter.com/ryanlecompte)
|
99
|
+
|
100
|
+
## Contributing
|
101
|
+
|
102
|
+
1. Fork it
|
103
|
+
2. Create your feature branch (`git checkout -b my-new-feature`)
|
104
|
+
3. Commit your changes (`git commit -am 'Added some feature'`)
|
105
|
+
4. Push to the branch (`git push origin my-new-feature`)
|
106
|
+
5. Create new Pull Request
|
data/Rakefile
ADDED
@@ -0,0 +1,16 @@
|
|
1
|
+
require 'redis'
|
2
|
+
require 'thread'
|
3
|
+
require 'logger'
|
4
|
+
require 'multi_json'
|
5
|
+
require 'securerandom'
|
6
|
+
|
7
|
+
require 'redis_failover/cli'
|
8
|
+
require 'redis_failover/util'
|
9
|
+
require 'redis_failover/node'
|
10
|
+
require 'redis_failover/errors'
|
11
|
+
require 'redis_failover/client'
|
12
|
+
require 'redis_failover/server'
|
13
|
+
require 'redis_failover/runner'
|
14
|
+
require 'redis_failover/version'
|
15
|
+
require 'redis_failover/node_manager'
|
16
|
+
require 'redis_failover/node_watcher'
|
@@ -0,0 +1,47 @@
|
|
1
|
+
module RedisFailover
|
2
|
+
# Parses server command-line arguments.
|
3
|
+
class CLI
|
4
|
+
def self.parse(source)
|
5
|
+
return {} if source.empty?
|
6
|
+
|
7
|
+
options = {}
|
8
|
+
parser = OptionParser.new do |opts|
|
9
|
+
opts.banner = "Usage: redis_failover_server [OPTIONS]"
|
10
|
+
|
11
|
+
opts.on('-P', '--port port', 'Server port') do |port|
|
12
|
+
options[:port] = Integer(port)
|
13
|
+
end
|
14
|
+
|
15
|
+
opts.on('-p', '--password password', 'Redis password') do |password|
|
16
|
+
options[:password] = password.strip
|
17
|
+
end
|
18
|
+
|
19
|
+
opts.on('-n', '--nodes nodes', 'Comma-separated redis host:port pairs') do |nodes|
|
20
|
+
# turns 'host1:port,host2:port' => [{:host => host, :port => port}, ...]
|
21
|
+
options[:nodes] = nodes.split(',').map do |node|
|
22
|
+
Hash[[:host, :port].zip(node.strip.split(':'))]
|
23
|
+
end
|
24
|
+
end
|
25
|
+
|
26
|
+
opts.on('--max-failures count',
|
27
|
+
'Max failures before server marks node unreachable (default 3)') do |max|
|
28
|
+
options[:max_failures] = Integer(max)
|
29
|
+
end
|
30
|
+
|
31
|
+
opts.on('-h', '--help', 'Display all options') do
|
32
|
+
puts opts
|
33
|
+
exit
|
34
|
+
end
|
35
|
+
end
|
36
|
+
|
37
|
+
parser.parse(source)
|
38
|
+
|
39
|
+
# assume password is same for all redis nodes
|
40
|
+
if password = options[:password]
|
41
|
+
options[:nodes].each { |opts| opts.update(:password => password) }
|
42
|
+
end
|
43
|
+
|
44
|
+
options
|
45
|
+
end
|
46
|
+
end
|
47
|
+
end
|
@@ -0,0 +1,198 @@
|
|
1
|
+
require 'set'
|
2
|
+
require 'open-uri'
|
3
|
+
|
4
|
+
module RedisFailover
|
5
|
+
# Redis failover-aware client.
|
6
|
+
class Client
|
7
|
+
include Util
|
8
|
+
|
9
|
+
REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }.freeze
|
10
|
+
REDIS_READ_OPS = Set[
|
11
|
+
:dbsize,
|
12
|
+
:echo,
|
13
|
+
:exists,
|
14
|
+
:get,
|
15
|
+
:getbit,
|
16
|
+
:getrange,
|
17
|
+
:hexists,
|
18
|
+
:hget,
|
19
|
+
:hgetall,
|
20
|
+
:hkeys,
|
21
|
+
:hlen,
|
22
|
+
:hmget,
|
23
|
+
:hvals,
|
24
|
+
:info,
|
25
|
+
:keys,
|
26
|
+
:lastsave,
|
27
|
+
:lindex,
|
28
|
+
:llen,
|
29
|
+
:lrange,
|
30
|
+
:mapped_hmget,
|
31
|
+
:mapped_mget,
|
32
|
+
:mget,
|
33
|
+
:ping,
|
34
|
+
:scard,
|
35
|
+
:sdiff,
|
36
|
+
:select,
|
37
|
+
:sinter,
|
38
|
+
:sismember,
|
39
|
+
:smembers,
|
40
|
+
:srandmember,
|
41
|
+
:strlen,
|
42
|
+
:sunion,
|
43
|
+
:ttl,
|
44
|
+
:type,
|
45
|
+
:zcard,
|
46
|
+
:zcount,
|
47
|
+
:zrange,
|
48
|
+
:zrangebyscore,
|
49
|
+
:zrank,
|
50
|
+
:zrevrange,
|
51
|
+
:zrevrangebyscore,
|
52
|
+
:zrevrank,
|
53
|
+
:zscore
|
54
|
+
].freeze
|
55
|
+
|
56
|
+
# Performance optimization: to avoid unnecessary method_missing calls,
|
57
|
+
# we proactively define methods that dispatch to the underlying redis
|
58
|
+
# calls.
|
59
|
+
Redis.public_instance_methods(false).each do |method|
|
60
|
+
define_method(method) do |*args, &block|
|
61
|
+
dispatch(method, *args, &block)
|
62
|
+
end
|
63
|
+
end
|
64
|
+
|
65
|
+
# Creates a new failover redis client.
|
66
|
+
#
|
67
|
+
# Options:
|
68
|
+
#
|
69
|
+
# :host - redis failover server host (required)
|
70
|
+
# :port - redis failover server port (required)
|
71
|
+
# :password - optional password for redis nodes
|
72
|
+
# :namespace - optional namespace for redis nodes
|
73
|
+
# :logger - optional logger override
|
74
|
+
# :retry_failure - indicate if failures should be retried (default true)
|
75
|
+
# :max_retries - max retries for a failure (default 5)
|
76
|
+
#
|
77
|
+
def initialize(options = {})
|
78
|
+
unless options.values_at(:host, :port).all?
|
79
|
+
raise ArgumentError, ':host and :port options required'
|
80
|
+
end
|
81
|
+
|
82
|
+
Util.logger = options[:logger] if options[:logger]
|
83
|
+
@namespace = options[:namespace]
|
84
|
+
@password = options[:password]
|
85
|
+
@retry = options[:retry_failure] || true
|
86
|
+
@max_retries = options[:max_retries] || 3
|
87
|
+
@registry_url = "http://#{options[:host]}:#{options[:port]}/redis_servers"
|
88
|
+
@redis_servers = nil
|
89
|
+
@master = nil
|
90
|
+
@slaves = []
|
91
|
+
@lock = Mutex.new
|
92
|
+
build_clients
|
93
|
+
end
|
94
|
+
|
95
|
+
def method_missing(method, *args, &block)
|
96
|
+
if redis_operation?(method)
|
97
|
+
dispatch(method, *args, &block)
|
98
|
+
else
|
99
|
+
super
|
100
|
+
end
|
101
|
+
end
|
102
|
+
|
103
|
+
def respond_to?(method)
|
104
|
+
redis_operation?(method) || super
|
105
|
+
end
|
106
|
+
|
107
|
+
def inspect
|
108
|
+
"#<RedisFailover::Client - master: #{master_info}, slaves: #{slaves_info})>"
|
109
|
+
end
|
110
|
+
alias_method :to_s, :inspect
|
111
|
+
|
112
|
+
private
|
113
|
+
|
114
|
+
def redis_operation?(method)
|
115
|
+
Redis.public_instance_methods(false).include?(method)
|
116
|
+
end
|
117
|
+
|
118
|
+
def dispatch(method, *args, &block)
|
119
|
+
tries = 0
|
120
|
+
begin
|
121
|
+
if REDIS_READ_OPS.include?(method)
|
122
|
+
# send read operations to a slave
|
123
|
+
slave.send(method, *args, &block)
|
124
|
+
else
|
125
|
+
# direct everything else to master
|
126
|
+
master.send(method, *args, &block)
|
127
|
+
end
|
128
|
+
rescue NoMasterError, *REDIS_ERRORS
|
129
|
+
logger.error("No suitable node available for operation `#{method}.`")
|
130
|
+
sleep(3)
|
131
|
+
build_clients
|
132
|
+
|
133
|
+
if @retry && tries < @max_retries
|
134
|
+
tries += 1
|
135
|
+
retry
|
136
|
+
end
|
137
|
+
|
138
|
+
raise
|
139
|
+
end
|
140
|
+
end
|
141
|
+
|
142
|
+
def master
|
143
|
+
@master or raise NoMasterError
|
144
|
+
end
|
145
|
+
|
146
|
+
def slave
|
147
|
+
# pick a slave, if none available fallback to master
|
148
|
+
@slaves.sample || master
|
149
|
+
end
|
150
|
+
|
151
|
+
def build_clients
|
152
|
+
@lock.synchronize do
|
153
|
+
begin
|
154
|
+
logger.info('Attempting to fetch nodes and build redis clients.')
|
155
|
+
servers = fetch_redis_servers
|
156
|
+
master = new_clients_for(servers[:master]).first if servers[:master]
|
157
|
+
slaves = new_clients_for(*servers[:slaves])
|
158
|
+
|
159
|
+
# once clients are successfully created, swap the references
|
160
|
+
@master = master
|
161
|
+
@slaves = slaves
|
162
|
+
rescue => ex
|
163
|
+
logger.error("Failed to fetch servers from #{@registry_url} - #{ex.message}")
|
164
|
+
logger.error(ex.backtrace.join("\n"))
|
165
|
+
end
|
166
|
+
end
|
167
|
+
end
|
168
|
+
|
169
|
+
def fetch_redis_servers
|
170
|
+
open(@registry_url) do |io|
|
171
|
+
servers = symbolize_keys(MultiJson.decode(io))
|
172
|
+
logger.info("Fetched servers: #{servers}")
|
173
|
+
servers
|
174
|
+
end
|
175
|
+
end
|
176
|
+
|
177
|
+
def new_clients_for(*nodes)
|
178
|
+
nodes.map do |node|
|
179
|
+
host, port = node.split(':')
|
180
|
+
client = Redis.new(:host => host, :port => port, :password => @password)
|
181
|
+
if @namespace
|
182
|
+
client = Redis::Namespace.new(@namespace, :redis => client)
|
183
|
+
end
|
184
|
+
client
|
185
|
+
end
|
186
|
+
end
|
187
|
+
|
188
|
+
def master_info
|
189
|
+
return "none" unless @master
|
190
|
+
"#{@master.client.host}:#{@master.client.port}"
|
191
|
+
end
|
192
|
+
|
193
|
+
def slaves_info
|
194
|
+
return "none" if @slaves.empty?
|
195
|
+
@slaves.map { |s| "#{s.client.host}:#{s.client.port}" }.join(', ')
|
196
|
+
end
|
197
|
+
end
|
198
|
+
end
|
@@ -0,0 +1,25 @@
|
|
1
|
+
module RedisFailover
|
2
|
+
class Error < StandardError
|
3
|
+
end
|
4
|
+
|
5
|
+
class InvalidNodeError < Error
|
6
|
+
end
|
7
|
+
|
8
|
+
class InvalidNodeStateError < Error
|
9
|
+
def initialize(node, state)
|
10
|
+
super("Invalid state change `#{state}` for node #{node}")
|
11
|
+
end
|
12
|
+
end
|
13
|
+
|
14
|
+
class NodeUnreachableError < Error
|
15
|
+
def initialize(node)
|
16
|
+
super("Node: #{node}")
|
17
|
+
end
|
18
|
+
end
|
19
|
+
|
20
|
+
class NoMasterError < Error
|
21
|
+
end
|
22
|
+
|
23
|
+
class NoSlaveError < Error
|
24
|
+
end
|
25
|
+
end
|