redis_failover 0.4.0 → 0.5.0

data/Changes.md

@@ -1,3 +1,7 @@
+0.5.0
+-----------
+- redis_failover is now built on top of ZooKeeper! This means redis_failover enjoys all of the reliability, redundancy, and data consistency offered by ZooKeeper. The old fragile HTTP-based approach has been removed and will no longer be supported in favor of ZooKeeper. This does mean that in order to use redis_failover, you must have ZooKeeper installed and running. Please see the README for steps on how to do this if you don't already have ZooKeeper running in your production environment.
+
 0.4.0
 -----------
 - No longer force newly available slaves to master if already slaves of that master

data/README.md

@@ -1,8 +1,8 @@
-# Automatic Redis Failover Client/Server
+# Automatic Redis Failover Client / Node Manager
 
 [![Build Status](https://secure.travis-ci.org/ryanlecompte/redis_failover.png?branch=master)](http://travis-ci.org/ryanlecompte/redis_failover)
 
-Redis Failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not provide
+redis_failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not provide
 an automatic failover capability when configured for master/slave replication. When the master node dies,
 a new master must be manually brought online and assigned as the slave's new master. This manual
 switch-over is not desirable in high traffic sites where Redis is a critical part of the overall
@@ -10,30 +10,34 @@ architecture. The existing standard Redis client for Ruby also only supports con
 Redis server. When using master/slave replication, it is desirable to have all writes go to the
 master, and all reads go to one of the N configured slaves.
 
-This gem attempts to address both the server and client problems. A redis failover server runs as a background
-daemon and monitors all of your configured master/slave nodes. When the server starts up, it
-automatically discovers who is the master and who are the slaves. Watchers are setup for each of
+This gem attempts to address these failover scenarios. A redis failover Node Manager daemon runs as a background
+process and monitors all of your configured master/slave nodes. When the daemon starts up, it
+automatically discovers the current master/slaves. Background watchers are setup for each of
 the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unavailable" state.
 If the node that went offline was the master, then one of the slaves will be promoted as the new master.
 All existing slaves will be automatically reconfigured to point to the new master for replication.
 All nodes marked as unavailable will be periodically checked to see if they have been brought back online.
-If so, the newly available nodes will be configured as slaves and brought back into the list of live
-servers. Note that detection of a node going down should be nearly instantaneous, since the mechanism
+If so, the newly available nodes will be configured as slaves and brought back into the list of available
+nodes. Note that detection of a node going down should be nearly instantaneous, since the mechanism
 used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
 immediately when the node actually goes offline. To avoid false positives (i.e., intermittent flaky
-network interruption), the server will only mark a node as unavailable if it fails to communicate with
+network interruption), the Node Manager will only mark a node as unavailable if it fails to communicate with
 it 3 times (this is configurable via --max-failures, see configuration options below).
 
 This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
-with a single host/port pair that points to redis failover server. The client will automatically
-connect to the server to find out the current state of the world (i.e., who's the current master and
-who are the current slaves). The client also acts as a load balancer in that it will automatically
+with a list of ZooKeeper servers. The client will automatically contact the ZooKeeper cluster to find out
+the current state of the world (i.e., who is the current master and who are the current slaves). The client
+also sets up a ZooKeeper watcher for the set of redis nodes controlled by the Node Manager daemon. When the daemon
+promotes a new master or detects a node as going down, ZooKeeper will notify the client near-instantaneously so
+that it can rebuild its set of Redis connections. The client also acts as a load balancer in that it will automatically
 dispatch Redis read operations to one of N slaves, and Redis write operations to the master.
-If it fails to communicate with any node, it will go back and ask the server for the current list of
-available servers, and then optionally retry the operation.
+If it fails to communicate with any node, it will go back and fetch the current list of available servers, and then
+optionally retry the operation.
 
 ## Installation
 
+redis_failover has an external dependency on ZooKeeper. You must have a running ZooKeeper cluster already available in order to use redis_failover. ZooKeeper provides redis_failover with its high availability and data consistency between Redis::Failover clients and the Node Manager daemon. Please see the requirements section below for more information on installing and setting up ZooKeeper if you don't have it running already.
+
 Add this line to your application's Gemfile:
 
     gem 'redis_failover'
@@ -46,69 +50,68 @@ Or install it yourself as:
 
     $ gem install redis_failover
 
-## Server Usage
+## Node Manager Daemon Usage
 
-The redis failover server is a simple process that should be run as a background daemon. The server supports the
+The Node Manager is a simple process that should be run as a background daemon. The daemon supports the
 following options:
 
-    Usage: redis_failover_server [OPTIONS]
-        -P, --port port                  Server port
-        -p, --password password          Redis password
-        -n, --nodes nodes                Comma-separated redis host:port pairs
-            --max-failures count         Max failures before server marks node unavailable (default 3)
+    Usage: redis_node_manager [OPTIONS]
+        -p, --password password          Redis password (optional)
+        -n, --nodes redis nodes          Comma-separated redis host:port pairs (required)
+        -z zookeeper servers,            Comma-separated zookeeper host:port pairs (required)
+            --zkservers
+            --znode-path path            Znode path override for storing redis server list (optional)
+            --max-failures count         Max failures before manager marks node unavailable (default 3)
         -h, --help                       Display all options
 
-To start the server for a simple master/slave configuration, use the following:
+To start the daemon for a simple master/slave configuration, use the following:
 
-    redis_failover_server -P 3000 -n localhost:6379,localhost:6380
+    redis_node_manager -n localhost:6379,localhost:6380 -z localhost:2181,localhost:2182,localhost:2183
 
-The server will automatically figure out who is the master and who is the slave upon startup. Note that it is
-a good idea to monitor the redis failover server process with a tool like Monit to ensure that it is restarted
+The Node Manager will automatically discover the master/slaves upon startup. Note that it is
+a good idea to monitor the redis Node Manager daemon process with a tool like Monit to ensure that it is restarted
 in the case of a failure.
 
 ## Client Usage
 
-The redis failover client must be used in conjunction with a running redis failover server. The
-client supports various configuration options, however the two mandatory options are the host
-and port of the redis failover server:
+The redis failover client must be used in conjunction with a running Node Manager daemon. The
+client supports various configuration options, however the only mandatory option is the list of
+ZooKeeper servers:
 
-    client = RedisFailover::Client.new(:host => 'localhost', :port => 3000)
+    client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
 
 The client actually employs the common redis and redis-namespace gems underneath, so this should be
 a drop-in replacement for your existing pure redis client usage.
 
 The full set of options that can be passed to RedisFailover::Client are:
 
-     :host          - redis failover server host (required)
-     :port          - redis failover server port (required)
-     :password      - optional password for redis nodes
-     :namespace     - optional namespace for redis nodes
-     :logger        - optional logger override
+     :zkservers     - comma-separated zookeeper host:port pairs (required)
+     :znode_path    - the Znode path override for redis server list (optional)
+     :password      - password for redis nodes (optional)
+     :namespace     - namespace for redis nodes (optional)
+     :logger        - logger override (optional)
      :retry_failure - indicate if failures should be retried (default true)
      :max_retries   - max retries for a failure (default 3)
 
 ## Requirements
 
-redis_failover is actively tested against MRI 1.9.2/1.9.3. Other rubies may work, although I don't actively test against them. 1.8 is not supported.
+- redis_failover is actively tested against MRI 1.9.2/1.9.3. Other rubies may work, although I don't actively test against them. 1.8 is not supported.
+- redis_failover requires a ZooKeeper service cluster to ensure reliability and data consistency. ZooKeeper is very simple and easy to get up and running. Please refer to this [Zookeper Guide](http://zookeeper.apache.org/doc/trunk/zookeeperStarted.html) to get up and running quickly if you don't already have ZooKeeper as a part of your environment.
 
 ## Considerations
 
-Note that by default the failover server will mark slaves that are currently syncing with their master as "available" based on the configuration value set for "slave-serve-stale-data" in redis.conf. By default this value is set to "yes" in the configuration, which means that slaves still syncing with their master will be available for servicing read requests. If you don't want this behavior, just set "slave-serve-stale-data" to "no" in your redis.conf file.
-
-## Limitations
+- Note that by default the Node Manager will mark slaves that are currently syncing with their master as "available" based on the configuration value set for "slave-serve-stale-data" in redis.conf. By default this value is set to "yes" in the configuration, which means that slaves still syncing with their master will be available for servicing read requests. If you don't want this behavior, just set "slave-serve-stale-data" to "no" in your redis.conf file.
 
-The redis_failover gem currently has limitations. It currently does not gracefully handle network partitions. In cases where
-the network splits, it is possible that more than one master could exist until the failover server sees all of the nodes again.
-If the failover client gets split from the failover server, it's also possible that it could be talking to a stale master. This would get corrected once the client could successfully reach the failover server again to fetch the latest set of master/slave nodes. This is a limitation that I hope to address in a future release. The gem can not guarantee data consistencies until this is addressed.
+- Note that it's still possible for the RedisFailover::Client instances to see a stale list of servers for a very small window. In most cases this will not be the case due to how ZooKeeper handles distributed communication, but you should be aware that in the worst case the client could write to a "stale" master for a small period of time until the next watch event is received by the client via ZooKeeper.
 
 ## TODO
 
-- Integrate with ZooKeeper for full reliability / data consistency.
-- Rework specs to work against a set of real Redis nodes as opposed to stubs.
+- Rework specs to work against a set of real Redis/ZooKeeper nodes as opposed to stubs.
 
 ## Resources
 
-To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).
+- To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).
+- To learn more about ZooKeeper, see the official [ZooKeeper](http://zookeeper.apache.org/) site.
 
 ## License
 

data/lib/redis_failover.rb

@@ -1,3 +1,4 @@
+require 'zk'
 require 'redis'
 require 'thread'
 require 'logger'
@@ -11,8 +12,8 @@ require 'redis_failover/util'
 require 'redis_failover/node'
 require 'redis_failover/errors'
 require 'redis_failover/client'
-require 'redis_failover/server'
 require 'redis_failover/runner'
 require 'redis_failover/version'
+require 'redis_failover/zk_client'
 require 'redis_failover/node_manager'
 require 'redis_failover/node_watcher'

data/lib/redis_failover/cli.rb

@@ -6,25 +6,32 @@ module RedisFailover
 
       options = {}
       parser = OptionParser.new do |opts|
-        opts.banner = "Usage: redis_failover_server [OPTIONS]"
+        opts.banner = "Usage: redis_node_manager [OPTIONS]"
 
-        opts.on('-P', '--port port', 'Server port') do |port|
-          options[:port] = Integer(port)
-        end
-
-        opts.on('-p', '--password password', 'Redis password') do |password|
+        opts.on('-p', '--password password', 'Redis password (optional)') do |password|
           options[:password] = password.strip
         end
 
-        opts.on('-n', '--nodes nodes', 'Comma-separated redis host:port pairs') do |nodes|
+        opts.on('-n', '--nodes redis nodes',
+          'Comma-separated redis host:port pairs (required)') do |nodes|
           # turns 'host1:port,host2:port' => [{:host => host, :port => port}, ...]
           options[:nodes] = nodes.split(',').map do |node|
             Hash[[:host, :port].zip(node.strip.split(':'))]
           end
         end
 
+        opts.on('-z', '--zkservers zookeeper servers',
+          'Comma-separated zookeeper host:port pairs (required)') do |servers|
+          options[:zkservers] = servers
+        end
+
+        opts.on('--znode-path path',
+          'Znode path override for storing redis server list (optional)') do |path|
+          options[:znode_path] = path
+        end
+
         opts.on('--max-failures count',
-          'Max failures before server marks node unavailable (default 3)') do |max|
+          'Max failures before manager marks node unavailable (default 3)') do |max|
           options[:max_failures] = Integer(max)
         end
 
@@ -35,7 +42,6 @@ module RedisFailover
       end
 
       parser.parse(source)
-
       # assume password is same for all redis nodes
       if password = options[:password]
         options[:nodes].each { |opts| opts.update(:password => password) }

data/lib/redis_failover/client.rb

@@ -1,5 +1,4 @@
 require 'set'
-require 'open-uri'
 
 module RedisFailover
   # Redis failover-aware client.
@@ -7,7 +6,6 @@ module RedisFailover
     include Util
 
     RETRY_WAIT_TIME = 3
-    REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }.freeze
     REDIS_READ_OPS = Set[
       :echo,
       :exists,
@@ -67,29 +65,27 @@ module RedisFailover
     #
     # Options:
     #
-    #   :host - redis failover server host (required)
-    #   :port - redis failover server port (required)
-    #   :password - optional password for redis nodes
-    #   :namespace - optional namespace for redis nodes
-    #   :logger - optional logger override
+    #   :zkservers - comma-separated zookeeper host:port pairs (required)
+    #   :znode_path - the Znode path override for redis server list (optional)
+    #   :password - password for redis nodes (optional)
+    #   :namespace - namespace for redis nodes (optional)
+    #   :logger - logger override (optional)
     #   :retry_failure - indicate if failures should be retried (default true)
     #   :max_retries - max retries for a failure (default 5)
     #
     def initialize(options = {})
-      unless options.values_at(:host, :port).all?
-        raise ArgumentError, ':host and :port options required'
-      end
-
       Util.logger = options[:logger] if options[:logger]
+      @zkservers = options.fetch(:zkservers) { raise ArgumentError, ':zkservers required'}
+      @znode = options[:znode_path] || Util::DEFAULT_ZNODE_PATH
       @namespace = options[:namespace]
       @password = options[:password]
       @retry = options[:retry_failure] || true
       @max_retries = @retry ? options.fetch(:max_retries, 3) : 0
-      @server_url = "http://#{options[:host]}:#{options[:port]}/redis_servers"
       @master = nil
       @slaves = []
+      @lock = Mutex.new
+      setup_zookeeper_client
       build_clients
-      start_background_monitor
     end
 
     def method_missing(method, *args, &block)
@@ -111,6 +107,30 @@ module RedisFailover
 
     private
 
+    def setup_zookeeper_client
+      @zkclient = ZkClient.new(@zkservers)
+
+      # when session expires / we are disconnected, purge client list
+      @zkclient.on_session_expiration do
+        @lock.synchronize { purge_clients }
+      end
+      @zkclient.event_handler.register_state_handler(:connecting) do
+        @lock.synchronize { purge_clients }
+      end
+
+      # register a watcher for future changes
+      @zkclient.watcher.register(@znode) do |event|
+        if event.node_created? || event.node_changed?
+          build_clients
+        elsif event.node_deleted?
+          @zkclient.stat( @znode, :watch => true)
+          @lock.synchronize { purge_clients }
+        else
+          logger.error("Unknown ZK node event: #{event.inspect}")
+        end
+      end
+    end
+
     def redis_operation?(method)
       Redis.public_instance_methods(false).include?(method)
     end
@@ -127,12 +147,11 @@ module RedisFailover
           # direct everything else to master
           master.send(method, *args, &block)
         end
-      rescue Error, *REDIS_ERRORS
-        logger.error("No suitable node available for operation `#{method}.`")
-        build_clients
-
+      rescue *ALL_ERRORS => ex
+        logger.error("Error while handling operation `#{method}` - #{ex.message}")
         if tries < @max_retries
           tries += 1
+          build_clients
           sleep(RETRY_WAIT_TIME) && retry
         end
 
@@ -141,7 +160,7 @@ module RedisFailover
     end
 
     def master
-      master = @master
+      master = @lock.synchronize { @master }
       if master
         verify_role!(master, :master)
         return master
@@ -151,7 +170,7 @@ module RedisFailover
 
     def slave
       # pick a slave, if none available fallback to master
-      if slave = @slaves.sample
+      if slave = @lock.synchronize { @slaves.sample }
         verify_role!(slave, :slave)
         return slave
       end
@@ -159,39 +178,40 @@ module RedisFailover
     end
 
     def build_clients
-      tries = 0
-
-      begin
-        logger.info('Checking for new redis nodes.')
-        nodes = fetch_nodes
-        return unless nodes_changed?(nodes)
-
-        logger.info('Node change detected, rebuilding clients.')
-        master = new_clients_for(nodes[:master]).first if nodes[:master]
-        slaves = new_clients_for(*nodes[:slaves])
-
-        # once clients are successfully created, swap the references
-        @master = master
-        @slaves = slaves
-      rescue => ex
-        logger.error("Failed to fetch nodes from #{@server_url} - #{ex.message}")
-        logger.error(ex.backtrace.join("\n"))
+      @lock.synchronize do
+        tries = 0
+
+        begin
+          nodes = fetch_nodes
+          return unless nodes_changed?(nodes)
+
+          purge_clients
+          logger.info("Building new clients for nodes #{nodes}")
+          new_master = new_clients_for(nodes[:master]).first if nodes[:master]
+          new_slaves = new_clients_for(*nodes[:slaves])
+          @master = new_master
+          @slaves = new_slaves
+        rescue *ALL_ERRORS => ex
+          purge_clients
+          logger.error("Failed to fetch nodes from #{@zkservers} - #{ex.message}")
+          logger.error(ex.backtrace.join("\n"))
+
+          if tries < @max_retries
+            tries += 1
+            sleep(RETRY_WAIT_TIME) && retry
+          end
 
-        if tries < @max_retries
-          tries += 1
-          sleep(RETRY_WAIT_TIME) && retry
+          raise
         end
-
-        raise FailoverServerUnavailableError.new(@server_url)
       end
     end
 
     def fetch_nodes
-      open(@server_url) do |io|
-        nodes = symbolize_keys(MultiJson.decode(io))
-        logger.info("Fetched nodes: #{nodes}")
-        nodes
-      end
+      data = @zkclient.get(@znode, :watch => true).first
+      nodes = symbolize_keys(decode(data))
+      logger.debug("Fetched nodes: #{nodes}")
+
+      nodes
     end
 
     def new_clients_for(*nodes)
@@ -243,20 +263,23 @@ module RedisFailover
       false
     end
 
-    # Spawns a background thread to periodically fetch the latest
-    # set of nodes from the redis failover server.
-    def start_background_monitor
-      Thread.new do
-        loop do
-          sleep(10)
+    def disconnect(*connections)
+      connections.each do |conn|
+        if conn
           begin
-            build_clients
-          rescue => ex
-            logger.error("Failed to poll for new nodes from #{@server_url} - #{ex.message}")
-            logger.error(ex.backtrace.join("\n"))
+            conn.client.disconnect
+          rescue
+            # best effort
           end
         end
       end
     end
+
+    def purge_clients
+      logger.info("Purging current redis clients")
+      disconnect(@master, *@slaves)
+      @master = nil
+      @slaves = []
+    end
   end
 end

data/lib/redis_failover/errors.rb

@@ -28,12 +28,6 @@ module RedisFailover
   class NoSlaveError < Error
   end
 
-  class FailoverServerUnavailableError < Error
-    def initialize(failover_server_url)
-      super("Unable to access #{failover_server_url}")
-    end
-  end
-
   class InvalidNodeRoleError < Error
     def initialize(node, assumed, actual)
       super("Invalid role detected for node #{node}, client thought " +

data/lib/redis_failover/node.rb

@@ -52,6 +52,7 @@ module RedisFailover
       perform_operation do |redis|
         unless slave_of?(master)
           redis.slaveof(master.host, master.port)
+          logger.info("#{self} now a slave of master #{master}")
           wakeup
         end
       end
@@ -61,6 +62,7 @@ module RedisFailover
       perform_operation do |redis|
         unless master?
           redis.slaveof('no', 'one')
+          logger.info("#{self} is now master")
           wakeup
         end
       end

data/lib/redis_failover/node_manager.rb

@@ -5,13 +5,15 @@ module RedisFailover
 
     def initialize(options)
       @options = options
-      @master, @slaves = parse_nodes
+      @zkclient = ZkClient.new(@options[:zkservers])
+      @znode = @options[:znode_path] || Util::DEFAULT_ZNODE_PATH
       @unavailable = []
       @queue = Queue.new
-      @lock = Mutex.new
+      discover_nodes
     end
 
     def start
+      initialize_path
       spawn_watchers
       handle_state_changes
     end
@@ -20,16 +22,6 @@ module RedisFailover
       @queue << [node, state]
     end
 
-    def nodes
-      @lock.synchronize do
-        {
-          :master => @master ? @master.to_s : nil,
-          :slaves => @slaves.map(&:to_s),
-          :unavailable => @unavailable.map(&:to_s)
-        }
-      end
-    end
-
     def shutdown
       @watchers.each(&:shutdown)
     end
@@ -38,20 +30,20 @@ module RedisFailover
 
     def handle_state_changes
       while state_change = @queue.pop
-        @lock.synchronize do
+        begin
           node, state = state_change
-          begin
-            case state
-            when :unavailable then handle_unavailable(node)
-            when :available   then handle_available(node)
-            when :syncing     then handle_syncing(node)
-            else raise InvalidNodeStateError.new(node, state)
-            end
-          rescue NodeUnavailableError
-            # node suddenly became unavailable, silently
-            # handle since the watcher will take care of
-            # keeping track of the node
+          case state
+          when :unavailable then handle_unavailable(node)
+          when :available   then handle_available(node)
+          when :syncing     then handle_syncing(node)
+          else raise InvalidNodeStateError.new(node, state)
           end
+
+          # flush current state
+          write_state
+        rescue *ALL_ERRORS => ex
+          logger.error("Error while handling #{state_change.inspect}: #{ex.message}")
+          logger.error(ex.backtrace.join("\n"))
         end
       end
     end
@@ -104,6 +96,7 @@ module RedisFailover
     end
 
     def promote_new_master(node = nil)
+      delete_path
       @master = nil
 
       # make a specific node or slave the new master
@@ -116,18 +109,21 @@ module RedisFailover
       candidate.make_master!
       @master = candidate
       redirect_slaves_to_master
+      create_path
+      write_state
       logger.info("Successfully promoted #{candidate} to master.")
     end
 
-    def parse_nodes
+    def discover_nodes
       nodes = @options[:nodes].map { |opts| Node.new(opts) }.uniq
-      raise NoMasterError unless master = find_master(nodes)
-      slaves = nodes - [master]
+      raise NoMasterError unless @master = find_master(nodes)
+      @slaves = nodes - [@master]
 
-      logger.info("Managing master (#{master}) and slaves" +
-        " (#{slaves.map(&:to_s).join(', ')})")
+      # ensure that slaves are correctly pointing to this master
+      redirect_slaves_to_master
 
-      [master, slaves]
+      logger.info("Managing master (#{@master}) and slaves" +
+        " (#{@slaves.map(&:to_s).join(', ')})")
     end
 
     def spawn_watchers
@@ -169,6 +165,7 @@ module RedisFailover
       return if @master == node && node.master?
       return if @master && node.slave_of?(@master)
 
+      logger.info("Reconciling node #{node}")
       if @master == node && !node.master?
         # we think the node is a master, but the node doesn't
         node.make_master!
@@ -180,5 +177,37 @@ module RedisFailover
         node.make_slave!(@master)
       end
     end
+
+    def current_nodes
+      {
+        :master => @master ? @master.to_s : nil,
+        :slaves => @slaves.map(&:to_s),
+        :unavailable => @unavailable.map(&:to_s)
+      }
+    end
+
+    def delete_path
+      if @zkclient.stat(@znode).exists?
+        @zkclient.delete(@znode)
+        logger.info("Deleted zookeeper node #{@znode}")
+      end
+    end
+
+    def create_path
+      # create nodes path if it doesn't already exist in ZK
+      unless @zkclient.stat(@znode).exists?
+        @zkclient.create(@znode, encode(current_nodes))
+        logger.info("Created zookeeper node #{@znode}")
+      end
+    end
+
+    def initialize_path
+      create_path
+      write_state
+    end
+
+    def write_state
+      @zkclient.set(@znode, encode(current_nodes))
+    end
   end
 end

data/lib/redis_failover/runner.rb

@@ -1,14 +1,13 @@
 module RedisFailover
-  # Runner is responsible for bootstrapping the redis failover server.
+  # Runner is responsible for bootstrapping the redis Node Manager.
   class Runner
     def self.run(options)
       options = CLI.parse(options)
-      Util.logger.info("Redis Failover Server starting on port #{options[:port]}")
-      Server.set(:port, options[:port])
       @node_manager = NodeManager.new(options)
-      server_thread = Thread.new { Server.run! { |server| trap_signals } }
+      trap_signals
       node_manager_thread = Thread.new { @node_manager.start }
-      [server_thread, node_manager_thread].each(&:join)
+      Util.logger.info("Redis Node Manager successfully started.")
+      node_manager_thread.join
     end
 
     def self.node_manager

data/lib/redis_failover/util.rb

@@ -1,8 +1,18 @@
+require 'redis_failover/errors'
+
 module RedisFailover
   # Common utiilty methods.
   module Util
     extend self
 
+    DEFAULT_ZNODE_PATH = '/redis_failover_nodes'
+    REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }
+    ALL_ERRORS = [
+      RedisFailover::Error,
+      ZookeeperExceptions::ZookeeperException,
+      REDIS_ERRORS,
+      StandardError].flatten
+
     def symbolize_keys(hash)
       Hash[hash.map { |k, v| [k.to_sym, v] }]
     end
@@ -29,5 +39,14 @@ module RedisFailover
     def logger
       Util.logger
     end
+
+    def encode(data)
+      MultiJson.encode(data)
+    end
+
+    def decode(data)
+      return unless data
+      MultiJson.decode(data)
+    end
   end
 end

data/lib/redis_failover/version.rb

@@ -1,3 +1,3 @@
 module RedisFailover
-  VERSION = "0.4.0"
+  VERSION = "0.5.0"
 end

data/lib/redis_failover/zk_client.rb

@@ -0,0 +1,78 @@
+module RedisFailover
+  # ZkClient is a thin wrapper over the ZK client to gracefully handle reconnects
+  # when a session expires.
+  class ZkClient
+    include Util
+
+    MAX_RECONNECTS = 3
+
+    def initialize(servers)
+      @servers = servers
+      @lock = Mutex.new
+      build_client
+    end
+
+    def on_session_expiration(&block)
+      @client.on_expired_session { block.call }
+      @on_session_expiration = block
+    end
+
+    def get(*args, &block)
+      perform_with_reconnect { @client.get(*args, &block) }
+    end
+
+    def set(*args, &block)
+      perform_with_reconnect { @client.set(*args, &block) }
+    end
+
+    def watcher(*args, &block)
+      perform_with_reconnect { @client.watcher(*args, &block) }
+    end
+
+    def event_handler(*args, &block)
+      perform_with_reconnect { @client.event_handler(*args, &block) }
+    end
+
+    def stat(*args, &block)
+      perform_with_reconnect { @client.stat(*args, &block) }
+    end
+
+    def create(*args, &block)
+      perform_with_reconnect { @client.create(*args, &block) }
+    end
+
+    def delete(*args, &block)
+      perform_with_reconnect { @client.delete(*args, &block) }
+    end
+
+    private
+
+    def perform_with_reconnect
+      tries = 0
+      begin
+        yield
+      rescue ZookeeperExceptions::ZookeeperException::SessionExpired
+        logger.info("Zookeeper client session expired, rebuilding client.")
+        if tries < MAX_RECONNECTS
+          tries += 1
+          build_client
+          @on_session_expiration.call if @on_session_expiration
+          sleep(2) && retry
+        end
+
+        raise
+      end
+    end
+
+    def build_client
+      @lock.synchronize do
+        if @client
+          @client.reopen
+        else
+          @client = ZK.new(@servers)
+        end
+        logger.info("Communicating with zookeeper servers #{@servers}")
+      end
+    end
+  end
+end

data/redis_failover.gemspec

@@ -4,8 +4,8 @@ require File.expand_path('../lib/redis_failover/version', __FILE__)
 Gem::Specification.new do |gem|
   gem.authors       = ["Ryan LeCompte"]
   gem.email         = ["lecompte@gmail.com"]
-  gem.description   = %(Redis Failover provides a full automatic master/slave failover solution for Ruby)
-  gem.summary       = %(Redis Failover provides a full automatic master/slave failover solution for Ruby)
+  gem.description   = %(Redis Failover is a ZooKeeper-based automatic master/slave failover solution for Ruby)
+  gem.summary       = %(Redis Failover is a ZooKeeper-based automatic master/slave failover solution for Ruby)
   gem.homepage      = "http://github.com/ryanlecompte/redis_failover"
 
   gem.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
@@ -18,7 +18,7 @@ Gem::Specification.new do |gem|
   gem.add_dependency('redis')
   gem.add_dependency('redis-namespace')
   gem.add_dependency('multi_json')
-  gem.add_dependency('sinatra')
+  gem.add_dependency('zk')
 
   gem.add_development_dependency('rake')
   gem.add_development_dependency('rspec')

data/spec/cli_spec.rb

@@ -7,11 +7,6 @@ module RedisFailover
         CLI.parse({}).should == {}
       end
 
-      it 'properly parses a server port' do
-        opts = CLI.parse(['-P 2222'])
-        opts.should == {:port => 2222}
-      end
-
       it 'properly parses redis nodes' do
         opts = CLI.parse(['-n host1:1,host2:2,host3:3'])
         opts[:nodes].should == [

data/spec/client_spec.rb

@@ -18,10 +18,12 @@ module RedisFailover
         :unavailable => []
       }
     end
+
+    def setup_zookeeper_client; end
   end
 
   describe Client do
-    let(:client) { ClientStub.new(:host => 'localhost', :port => 3000) }
+    let(:client) { ClientStub.new(:zkservers => 'localhost:9281') }
 
     describe '#build_clients' do
       it 'properly parses master' do
@@ -31,18 +33,6 @@ module RedisFailover
       it 'properly parses slaves' do
         client.current_slaves.first.to_s.should == 'localhost:1111'
       end
-
-      it 'does not rebuild clients if hosts have not changed' do
-        class << client
-          attr_reader :built_new_client
-          def new_clients_for(*)
-            @built_new_client = true
-          end
-        end
-
-        5.times { client.send(:build_clients) }
-        client.built_new_client.should_not be_true
-      end
     end
 
     describe '#dispatch' do
@@ -57,7 +47,7 @@ module RedisFailover
         client.get('foo')
       end
 
-      it 'reconnects with redis failover server when node is unavailable' do
+      it 'reconnects when node is unavailable' do
         class << client
           attr_reader :reconnected
           def build_clients
@@ -80,12 +70,6 @@ module RedisFailover
         client.reconnected.should be_true
       end
 
-      it 'fails hard when the failover server is unavailable' do
-        expect do
-          Client.new(:host => 'foo', :port => 123445)
-        end.to raise_error(FailoverServerUnavailableError)
-      end
-
       it 'properly detects when a node has changed roles' do
         client.current_master.change_role_to('slave')
         expect { client.send(:master) }.to raise_error(InvalidNodeRoleError)

data/spec/node_manager_spec.rb

@@ -6,7 +6,7 @@ module RedisFailover
 
     describe '#nodes' do
       it 'returns current master and slave nodes' do
-        manager.nodes.should == {
+        manager.current_nodes.should == {
           :master => 'master:6379',
           :slaves => ['slave:6379'],
           :unavailable => []
@@ -19,7 +19,7 @@ module RedisFailover
         it 'moves slave to unavailable list' do
           slave = manager.slaves.first
           manager.force_unavailable(slave)
-          manager.nodes[:unavailable].should include(slave.to_s)
+          manager.current_nodes[:unavailable].should include(slave.to_s)
         end
       end
 
@@ -35,7 +35,7 @@ module RedisFailover
         end
 
         it 'moves master to unavailable list' do
-          manager.nodes[:unavailable].should include(@master.to_s)
+          manager.current_nodes[:unavailable].should include(@master.to_s)
         end
       end
     end
@@ -50,8 +50,8 @@ module RedisFailover
       context 'slave node with a master present' do
         it 'removes slave from unavailable list' do
           manager.force_available(@slave)
-          manager.nodes[:unavailable].should be_empty
-          manager.nodes[:slaves].should include(@slave.to_s)
+          manager.current_nodes[:unavailable].should be_empty
+          manager.current_nodes[:slaves].should include(@slave.to_s)
         end
 
         it 'makes node a slave of new master' do
@@ -76,13 +76,12 @@ module RedisFailover
         end
 
         it 'promotes slave to master' do
-          manager.master.should be_nil
           manager.force_available(@slave)
           manager.master.should == @slave
         end
 
         it 'slaves list remains empty' do
-          manager.nodes[:slaves].should be_empty
+          manager.current_nodes[:slaves].should be_empty
         end
       end
     end
@@ -92,7 +91,7 @@ module RedisFailover
         it 'adds node to unavailable list' do
           slave = manager.slaves.first
           manager.force_syncing(slave, false)
-          manager.nodes[:unavailable].should include(slave.to_s)
+          manager.current_nodes[:unavailable].should include(slave.to_s)
         end
       end
 
@@ -100,8 +99,8 @@ module RedisFailover
         it 'makes node available' do
           slave = manager.slaves.first
           manager.force_syncing(slave, true)
-          manager.nodes[:unavailable].should_not include(slave.to_s)
-          manager.nodes[:slaves].should include(slave.to_s)
+          manager.current_nodes[:unavailable].should_not include(slave.to_s)
+          manager.current_nodes[:slaves].should include(slave.to_s)
         end
       end
     end

data/spec/spec_helper.rb

@@ -9,7 +9,10 @@ class NullObject
   end
 end
 
-RedisFailover::Util.logger = NullObject.new
+module RedisFailover
+  Util.logger = NullObject.new
+  def ZkClient.new(*args); NullObject.new; end
+end
 
 RSpec.configure do |config|
 end

data/spec/support/node_manager_stub.rb

@@ -1,14 +1,16 @@
 module RedisFailover
   class NodeManagerStub < NodeManager
     attr_accessor :master
+    public :current_nodes
 
-    def parse_nodes
+    def discover_nodes
       master = Node.new(:host => 'master')
       slave = Node.new(:host => 'slave')
       [master, slave].each { |node| node.extend(RedisStubSupport) }
       master.make_master!
       slave.make_slave!(master)
-      [master, [slave]]
+      @master = master
+      @slaves = [slave]
     end
 
     def slaves
@@ -44,5 +46,10 @@ module RedisFailover
       notify_state_change(node, :syncing)
       stop_processing
     end
+
+    def initialize_path; end
+    def delete_path; end
+    def create_path; end
+    def write_state; end
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redis_failover
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.5.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-04-15 00:00:00.000000000 Z
+date: 2012-04-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
-  requirement: &70210559246440 !ruby/object:Gem::Requirement
+  requirement: &70300698370600 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70210559246440
+  version_requirements: *70300698370600
 - !ruby/object:Gem::Dependency
   name: redis-namespace
-  requirement: &70210559246000 !ruby/object:Gem::Requirement
+  requirement: &70300698370180 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70210559246000
+  version_requirements: *70300698370180
 - !ruby/object:Gem::Dependency
   name: multi_json
-  requirement: &70210559245580 !ruby/object:Gem::Requirement
+  requirement: &70300698369740 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70210559245580
+  version_requirements: *70300698369740
 - !ruby/object:Gem::Dependency
-  name: sinatra
-  requirement: &70210559245160 !ruby/object:Gem::Requirement
+  name: zk
+  requirement: &70300698369320 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70210559245160
+  version_requirements: *70300698369320
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70210559244740 !ruby/object:Gem::Requirement
+  requirement: &70300698368900 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70210559244740
+  version_requirements: *70300698368900
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70210559244320 !ruby/object:Gem::Requirement
+  requirement: &70300698368480 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,13 +76,13 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70210559244320
-description: Redis Failover provides a full automatic master/slave failover solution
+  version_requirements: *70300698368480
+description: Redis Failover is a ZooKeeper-based automatic master/slave failover solution
   for Ruby
 email:
 - lecompte@gmail.com
 executables:
-- redis_failover_server
+- redis_node_manager
 extensions: []
 extra_rdoc_files: []
 files:
@@ -93,7 +93,7 @@ files:
 - LICENSE
 - README.md
 - Rakefile
-- bin/redis_failover_server
+- bin/redis_node_manager
 - lib/redis_failover.rb
 - lib/redis_failover/cli.rb
 - lib/redis_failover/client.rb
@@ -102,9 +102,9 @@ files:
 - lib/redis_failover/node_manager.rb
 - lib/redis_failover/node_watcher.rb
 - lib/redis_failover/runner.rb
-- lib/redis_failover/server.rb
 - lib/redis_failover/util.rb
 - lib/redis_failover/version.rb
+- lib/redis_failover/zk_client.rb
 - redis_failover.gemspec
 - spec/cli_spec.rb
 - spec/client_spec.rb
@@ -129,7 +129,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -4081812195470581448
+      hash: -1870245960706248512
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -138,14 +138,14 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -4081812195470581448
+      hash: -1870245960706248512
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.16
 signing_key: 
 specification_version: 3
-summary: Redis Failover provides a full automatic master/slave failover solution for
-  Ruby
+summary: Redis Failover is a ZooKeeper-based automatic master/slave failover solution
+  for Ruby
 test_files:
 - spec/cli_spec.rb
 - spec/client_spec.rb

data/lib/redis_failover/server.rb

@@ -1,13 +0,0 @@
-require 'sinatra'
-
-module RedisFailover
-  # Serves as an endpoint for discovering the current redis master and slaves.
-  class Server < Sinatra::Base
-    disable :logging
-
-    get '/redis_servers' do
-      content_type :json
-      MultiJson.encode(Runner.node_manager.nodes)
-    end
-  end
-end