redis_failover 0.3.0 → 0.4.0

data/Changes.md

@@ -1,8 +1,17 @@
+0.4.0
+-----------
+- No longer force newly available slaves to master if already slaves of that master
+- Honor a node's slave-serve-stale-data configuration option; do not mark a sync-with-master-in-progress slave as available if its slave-serve-stale-data is disabled
+- Change reachable/unreachable wording to available/unavailable
+- Added node reconciliation, i.e. if a node comes back up, make sure that the node manager and the node agree on current role
+- More efficient use of redis client connections
+- Raise proper error for unsupported operations (i.e., those that don't make sense for a failover client)
+- Properly handle any hanging node operations in the failover server
+
 0.3.0
 -----------
 - Integrated travis-ci
-- Added background monitor to client for proactively detecting
-  changes to current set of redis nodes
+- Added background monitor to client for proactively detecting changes to current set of redis nodes
 
 0.2.0
 -----------

data/README.md

@@ -13,15 +13,15 @@ master, and all reads go to one of the N configured slaves.
 This gem attempts to address both the server and client problems. A redis failover server runs as a background
 daemon and monitors all of your configured master/slave nodes. When the server starts up, it
 automatically discovers who is the master and who are the slaves. Watchers are setup for each of
-the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unreachable" state.
+the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unavailable" state.
 If the node that went offline was the master, then one of the slaves will be promoted as the new master.
 All existing slaves will be automatically reconfigured to point to the new master for replication.
-All nodes marked as unreachable will be periodically checked to see if they have been brought back online.
-If so, the newly reachable nodes will be configured as slaves and brought back into the list of live
+All nodes marked as unavailable will be periodically checked to see if they have been brought back online.
+If so, the newly available nodes will be configured as slaves and brought back into the list of live
 servers. Note that detection of a node going down should be nearly instantaneous, since the mechanism
 used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
 immediately when the node actually goes offline. To avoid false positives (i.e., intermittent flaky
-network interruption), the server will only mark a node as unreachable if it fails to communicate with
+network interruption), the server will only mark a node as unavailable if it fails to communicate with
 it 3 times (this is configurable via --max-failures, see configuration options below).
 
 This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
@@ -55,7 +55,7 @@ following options:
         -P, --port port                  Server port
         -p, --password password          Redis password
         -n, --nodes nodes                Comma-separated redis host:port pairs
-            --max-failures count         Max failures before server marks node unreachable (default 3)
+            --max-failures count         Max failures before server marks node unavailable (default 3)
         -h, --help                       Display all options
 
 To start the server for a simple master/slave configuration, use the following:
@@ -87,6 +87,25 @@ The full set of options that can be passed to RedisFailover::Client are:
      :retry_failure - indicate if failures should be retried (default true)
      :max_retries   - max retries for a failure (default 3)
 
+## Requirements
+
+redis_failover is actively tested against MRI 1.9.2/1.9.3. Other rubies may work, although I don't actively test against them. 1.8 is not supported.
+
+## Considerations
+
+Note that by default the failover server will mark slaves that are currently syncing with their master as "available" based on the configuration value set for "slave-serve-stale-data" in redis.conf. By default this value is set to "yes" in the configuration, which means that slaves still syncing with their master will be available for servicing read requests. If you don't want this behavior, just set "slave-serve-stale-data" to "no" in your redis.conf file.
+
+## Limitations
+
+The redis_failover gem currently has limitations. It currently does not gracefully handle network partitions. In cases where
+the network splits, it is possible that more than one master could exist until the failover server sees all of the nodes again.
+If the failover client gets split from the failover server, it's also possible that it could be talking to a stale master. This would get corrected once the client could successfully reach the failover server again to fetch the latest set of master/slave nodes. This is a limitation that I hope to address in a future release. The gem can not guarantee data consistencies until this is addressed.
+
+## TODO
+
+- Integrate with ZooKeeper for full reliability / data consistency.
+- Rework specs to work against a set of real Redis nodes as opposed to stubs.
+
 ## Resources
 
 To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).

data/lib/redis_failover.rb

@@ -1,6 +1,7 @@
 require 'redis'
 require 'thread'
 require 'logger'
+require 'timeout'
 require 'optparse'
 require 'multi_json'
 require 'securerandom'

data/lib/redis_failover/cli.rb

@@ -24,7 +24,7 @@ module RedisFailover
         end
 
         opts.on('--max-failures count',
-          'Max failures before server marks node unreachable (default 3)') do |max|
+          'Max failures before server marks node unavailable (default 3)') do |max|
           options[:max_failures] = Integer(max)
         end
 

data/lib/redis_failover/client.rb

@@ -9,7 +9,6 @@ module RedisFailover
     RETRY_WAIT_TIME = 3
     REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }.freeze
     REDIS_READ_OPS = Set[
-      :dbsize,
       :echo,
       :exists,
       :get,
@@ -22,26 +21,21 @@ module RedisFailover
       :hlen,
       :hmget,
       :hvals,
-      :info,
       :keys,
-      :lastsave,
       :lindex,
       :llen,
       :lrange,
       :mapped_hmget,
       :mapped_mget,
       :mget,
-      :ping,
       :scard,
       :sdiff,
-      :select,
       :sinter,
       :sismember,
       :smembers,
       :srandmember,
       :strlen,
       :sunion,
-      :ttl,
       :type,
       :zcard,
       :zcount,
@@ -54,6 +48,12 @@ module RedisFailover
       :zscore
     ].freeze
 
+    UNSUPPORTED_OPS = Set[
+      :select,
+      :ttl,
+      :dbsize,
+    ].freeze
+
     # Performance optimization: to avoid unnecessary method_missing calls,
     # we proactively define methods that dispatch to the underlying redis
     # calls.
@@ -88,7 +88,6 @@ module RedisFailover
       @server_url = "http://#{options[:host]}:#{options[:port]}/redis_servers"
       @master = nil
       @slaves = []
-      @lock = Mutex.new
       build_clients
       start_background_monitor
     end
@@ -117,6 +116,7 @@ module RedisFailover
     end
 
     def dispatch(method, *args, &block)
+      verify_supported!(method)
       tries = 0
 
       begin
@@ -159,32 +159,30 @@ module RedisFailover
     end
 
     def build_clients
-      @lock.synchronize do
-        tries = 0
-
-        begin
-          logger.info('Checking for new redis nodes.')
-          nodes = fetch_nodes
-          return unless nodes_changed?(nodes)
-
-          logger.info('Node change detected, rebuilding clients.')
-          master = new_clients_for(nodes[:master]).first if nodes[:master]
-          slaves = new_clients_for(*nodes[:slaves])
-
-          # once clients are successfully created, swap the references
-          @master = master
-          @slaves = slaves
-        rescue => ex
-          logger.error("Failed to fetch nodes from #{@server_url} - #{ex.message}")
-          logger.error(ex.backtrace.join("\n"))
-
-          if tries < @max_retries
-            tries += 1
-            sleep(RETRY_WAIT_TIME) && retry
-          end
+      tries = 0
 
-          raise FailoverServerUnreachableError.new(@server_url)
+      begin
+        logger.info('Checking for new redis nodes.')
+        nodes = fetch_nodes
+        return unless nodes_changed?(nodes)
+
+        logger.info('Node change detected, rebuilding clients.')
+        master = new_clients_for(nodes[:master]).first if nodes[:master]
+        slaves = new_clients_for(*nodes[:slaves])
+
+        # once clients are successfully created, swap the references
+        @master = master
+        @slaves = slaves
+      rescue => ex
+        logger.error("Failed to fetch nodes from #{@server_url} - #{ex.message}")
+        logger.error(ex.backtrace.join("\n"))
+
+        if tries < @max_retries
+          tries += 1
+          sleep(RETRY_WAIT_TIME) && retry
         end
+
+        raise FailoverServerUnavailableError.new(@server_url)
       end
     end
 
@@ -224,6 +222,12 @@ module RedisFailover
       role
     end
 
+    def verify_supported!(method)
+      if UNSUPPORTED_OPS.include?(method)
+        raise UnsupportedOperationError.new(method)
+      end
+    end
+
     def addresses_for(nodes)
       nodes.map { |node| address_for(node) }
     end

data/lib/redis_failover/errors.rb

@@ -1,5 +1,10 @@
 module RedisFailover
   class Error < StandardError
+    attr_reader :original
+    def initialize(msg = nil, original = $!)
+      super(msg)
+      @original = original
+    end
   end
 
   class InvalidNodeError < Error
@@ -11,7 +16,7 @@ module RedisFailover
     end
   end
 
-  class NodeUnreachableError < Error
+  class NodeUnavailableError < Error
     def initialize(node)
       super("Node: #{node}")
     end
@@ -23,9 +28,9 @@ module RedisFailover
   class NoSlaveError < Error
   end
 
-  class FailoverServerUnreachableError < Error
+  class FailoverServerUnavailableError < Error
     def initialize(failover_server_url)
-      super("Unable to reach #{failover_server_url}")
+      super("Unable to access #{failover_server_url}")
     end
   end
 
@@ -35,4 +40,10 @@ module RedisFailover
         "it was a #{assumed}, but it's now a #{actual}")
     end
   end
+
+  class UnsupportedOperationError < Error
+    def initialize(operation)
+      super("Operation `#{operation}` is currently unsupported")
+    end
+  end
 end

data/lib/redis_failover/node.rb

@@ -3,6 +3,8 @@ module RedisFailover
   class Node
     include Util
 
+    MAX_OP_WAIT_TIME = 5
+
     attr_reader :host, :port
 
     def initialize(options = {})
@@ -11,12 +13,6 @@ module RedisFailover
       @password = options[:password]
     end
 
-    def ping
-      perform_operation do
-        redis.ping
-      end
-    end
-
     def master?
       role == 'master'
     end
@@ -25,29 +21,48 @@ module RedisFailover
       !master?
     end
 
-    def wait_until_unreachable
-      perform_operation do
-        redis.blpop(wait_key, 0)
+    def slave_of?(master)
+      current_master == master
+    end
+
+    def current_master
+      info = fetch_info
+      return unless info[:role] == 'slave'
+      Node.new(:host => info[:master_host], :port => info[:master_port].to_i)
+    end
+
+    # Waits until something interesting happens. If the connection
+    # with this node dies, the blpop call will raise an error. If
+    # the blpop call returns without error, then this will be due to
+    # a graceful shutdown signaled by #wakeup or a timeout.
+    def wait
+      perform_operation do |redis|
+        redis.blpop(wait_key, MAX_OP_WAIT_TIME - 3)
         redis.del(wait_key)
       end
     end
 
-    def stop_waiting
-      perform_operation do
+    def wakeup
+      perform_operation do |redis|
         redis.lpush(wait_key, '1')
       end
     end
 
     def make_slave!(master)
-      perform_operation do
-        redis.slaveof(master.host, master.port)
+      perform_operation do |redis|
+        unless slave_of?(master)
+          redis.slaveof(master.host, master.port)
+          wakeup
+        end
       end
     end
 
     def make_master!
-      perform_operation do
-        # yes, this is a real redis operation!
-        redis.slaveof('no', 'one')
+      perform_operation do |redis|
+        unless master?
+          redis.slaveof('no', 'one')
+          wakeup
+        end
       end
     end
 
@@ -70,32 +85,55 @@ module RedisFailover
       to_s.hash
     end
 
-    private
+    def fetch_info
+      perform_operation do |redis|
+        symbolize_keys(redis.info)
+      end
+    end
+    alias_method :ping, :fetch_info
 
-    def role
-      fetch_info[:role]
+    def prohibits_stale_reads?
+      perform_operation do |redis|
+        redis.config('get', 'slave-serve-stale-data').last == 'no'
+      end
     end
 
-    def fetch_info
-      perform_operation do
-        symbolize_keys(redis.info)
+    def syncing_with_master?
+      perform_operation do |redis|
+        fetch_info[:master_sync_in_progress] == '1'
       end
     end
 
+    private
+
+    def role
+      fetch_info[:role]
+    end
+
     def wait_key
       @wait_key ||= "_redis_failover_#{SecureRandom.hex(32)}"
     end
 
-    def redis
+    def new_client
       Redis.new(:host => @host, :password => @password, :port => @port)
-    rescue
-      raise NodeUnreachableError.new(self)
     end
 
     def perform_operation
-      yield
+      redis = nil
+      Timeout.timeout(MAX_OP_WAIT_TIME) do
+        redis = new_client
+        yield redis
+      end
     rescue
-      raise NodeUnreachableError.new(self)
+      raise NodeUnavailableError.new(self)
+    ensure
+      if redis
+        begin
+          redis.client.disconnect
+        rescue
+          raise NodeUnavailableError.new(self)
+        end
+      end
     end
   end
 end

data/lib/redis_failover/node_manager.rb

@@ -6,7 +6,7 @@ module RedisFailover
     def initialize(options)
       @options = options
       @master, @slaves = parse_nodes
-      @unreachable = []
+      @unavailable = []
       @queue = Queue.new
       @lock = Mutex.new
     end
@@ -25,7 +25,7 @@ module RedisFailover
         {
           :master => @master ? @master.to_s : nil,
           :slaves => @slaves.map(&:to_s),
-          :unreachable => @unreachable.map(&:to_s)
+          :unavailable => @unavailable.map(&:to_s)
         }
       end
     end
@@ -42,12 +42,13 @@ module RedisFailover
           node, state = state_change
           begin
             case state
-            when :unreachable then handle_unreachable(node)
-            when :reachable then handle_reachable(node)
+            when :unavailable then handle_unavailable(node)
+            when :available   then handle_available(node)
+            when :syncing     then handle_syncing(node)
             else raise InvalidNodeStateError.new(node, state)
             end
-          rescue NodeUnreachableError
-            # node suddenly became unreachable, silently
+          rescue NodeUnavailableError
+            # node suddenly became unavailable, silently
             # handle since the watcher will take care of
             # keeping track of the node
           end
@@ -55,25 +56,27 @@ module RedisFailover
       end
     end
 
-    def handle_unreachable(node)
+    def handle_unavailable(node)
       # no-op if we already know about this node
-      return if @unreachable.include?(node)
-      logger.info("Handling unreachable node: #{node}")
+      return if @unavailable.include?(node)
+      logger.info("Handling unavailable node: #{node}")
 
-      @unreachable << node
+      @unavailable << node
       # find a new master if this node was a master
       if node == @master
-        logger.info("Demoting currently unreachable master #{node}.")
+        logger.info("Demoting currently unavailable master #{node}.")
         promote_new_master
       else
         @slaves.delete(node)
       end
     end
 
-    def handle_reachable(node)
+    def handle_available(node)
+      reconcile(node)
+
       # no-op if we already know about this node
       return if @master == node || @slaves.include?(node)
-      logger.info("Handling reachable node: #{node}")
+      logger.info("Handling available node: #{node}")
 
       if @master
         # master already exists, make a slave
@@ -84,7 +87,20 @@ module RedisFailover
         promote_new_master(node)
       end
 
-      @unreachable.delete(node)
+      @unavailable.delete(node)
+    end
+
+    def handle_syncing(node)
+      reconcile(node)
+
+      if node.prohibits_stale_reads?
+        logger.info("Node #{node} not ready yet, still syncing with master.")
+        @unavailable << node
+        return
+      end
+
+      # otherwise, we can use this node
+      handle_available(node)
     end
 
     def promote_new_master(node = nil)
@@ -104,7 +120,7 @@ module RedisFailover
     end
 
     def parse_nodes
-      nodes = @options[:nodes].map { |opts| Node.new(opts) }
+      nodes = @options[:nodes].map { |opts| Node.new(opts) }.uniq
       raise NoMasterError unless master = find_master(nodes)
       slaves = nodes - [master]
 
@@ -127,7 +143,7 @@ module RedisFailover
       nodes.find do |node|
         begin
           node.master?
-        rescue NodeUnreachableError
+        rescue NodeUnavailableError
           # will eventually be handled by watcher
           false
         end
@@ -139,10 +155,30 @@ module RedisFailover
       @slaves.each do |slave|
         begin
           slave.make_slave!(@master)
-        rescue NodeUnreachableError
+        rescue NodeUnavailableError
           # will also be detected by watcher
         end
       end
     end
+
+    # It's possible that a newly available node may have been restarted
+    # and completely lost its dynamically set run-time role by the node
+    # manager. This method ensures that the node resumes its role as
+    # determined by the manager.
+    def reconcile(node)
+      return if @master == node && node.master?
+      return if @master && node.slave_of?(@master)
+
+      if @master == node && !node.master?
+        # we think the node is a master, but the node doesn't
+        node.make_master!
+        return
+      end
+
+      # verify that node is a slave for the current master
+      if @master && !node.slave_of?(@master)
+        node.make_slave!(@master)
+      end
+    end
   end
 end

data/lib/redis_failover/node_watcher.rb

@@ -1,6 +1,10 @@
 module RedisFailover
-  # Watches a specific redis node for its reachability.
+  # Watches a specific redis node for its availability.
   class NodeWatcher
+    include Util
+
+    WATCHER_SLEEP_TIME = 3
+
     def initialize(manager, node, max_failures)
       @manager = manager
       @node = node
@@ -16,7 +20,7 @@ module RedisFailover
 
     def shutdown
       @done = true
-      @node.stop_waiting
+      @node.wakeup
       @monitor_thread.join if @monitor_thread
     rescue
       # best effort
@@ -27,20 +31,31 @@ module RedisFailover
     def monitor_node
       failures = 0
 
-      begin
-        return if @done
-        @node.ping
-        failures = 0
-        @manager.notify_state_change(@node, :reachable)
-        @node.wait_until_unreachable
-      rescue NodeUnreachableError
-        failures += 1
-        if failures >= @max_failures
-          @manager.notify_state_change(@node, :unreachable)
+      loop do
+        begin
+          return if @done
+          sleep(WATCHER_SLEEP_TIME)
+          @node.ping
           failures = 0
+
+          if @node.syncing_with_master?
+            notify(:syncing)
+          else
+            notify(:available)
+            @node.wait
+          end
+        rescue NodeUnavailableError
+          failures += 1
+          if failures >= @max_failures
+            notify(:unavailable)
+            failures = 0
+          end
         end
-        sleep(3) && retry
       end
     end
+
+    def notify(state)
+      @manager.notify_state_change(@node, state)
+    end
   end
 end

data/lib/redis_failover/version.rb

@@ -1,3 +1,3 @@
 module RedisFailover
-  VERSION = "0.3.0"
+  VERSION = "0.4.0"
 end

data/spec/client_spec.rb

@@ -15,7 +15,7 @@ module RedisFailover
       {
         :master => 'localhost:6379',
         :slaves => ['localhost:1111'],
-        :unreachable => []
+        :unavailable => []
       }
     end
   end
@@ -57,7 +57,7 @@ module RedisFailover
         client.get('foo')
       end
 
-      it 'reconnects with redis failover server when node is unreachable' do
+      it 'reconnects with redis failover server when node is unavailable' do
         class << client
           attr_reader :reconnected
           def build_clients
@@ -70,26 +70,30 @@ module RedisFailover
             {
               :master => "localhost:222#{@calls += 1}",
               :slaves => ['localhost:1111'],
-              :unreachable => []
+              :unavailable => []
             }
           end
         end
 
-        client.current_master.make_unreachable!
+        client.current_master.make_unavailable!
         client.del('foo')
         client.reconnected.should be_true
       end
 
-      it 'fails hard when the failover server is unreachable' do
+      it 'fails hard when the failover server is unavailable' do
         expect do
           Client.new(:host => 'foo', :port => 123445)
-        end.to raise_error(FailoverServerUnreachableError)
+        end.to raise_error(FailoverServerUnavailableError)
       end
 
       it 'properly detects when a node has changed roles' do
         client.current_master.change_role_to('slave')
         expect { client.send(:master) }.to raise_error(InvalidNodeRoleError)
       end
+
+      it 'raises error for unsupported operations' do
+        expect { client.select }.to raise_error(UnsupportedOperationError)
+      end
     end
   end
 end

data/spec/node_manager_spec.rb

@@ -9,17 +9,17 @@ module RedisFailover
         manager.nodes.should == {
           :master => 'master:6379',
           :slaves => ['slave:6379'],
-          :unreachable => []
+          :unavailable => []
         }
       end
     end
 
-    describe '#handle_unreachable' do
+    describe '#handle_unavailable' do
       context 'slave dies' do
-        it 'moves slave to unreachable list' do
+        it 'moves slave to unavailable list' do
           slave = manager.slaves.first
-          manager.force_unreachable(slave)
-          manager.nodes[:unreachable].should include(slave.to_s)
+          manager.force_unavailable(slave)
+          manager.nodes[:unavailable].should include(slave.to_s)
         end
       end
 
@@ -27,43 +27,57 @@ module RedisFailover
         before(:each) do
           @slave = manager.slaves.first
           @master = manager.master
-          manager.force_unreachable(@master)
+          manager.force_unavailable(@master)
         end
 
         it 'promotes slave to master' do
           manager.master.should == @slave
         end
 
-        it 'moves master to unreachable list' do
-          manager.nodes[:unreachable].should include(@master.to_s)
+        it 'moves master to unavailable list' do
+          manager.nodes[:unavailable].should include(@master.to_s)
         end
       end
     end
 
-    describe '#handle_reachable' do
+    describe '#handle_available' do
       before(:each) do
-        # force to be unreachable first
+        # force to be unavailable first
         @slave = manager.slaves.first
-        manager.force_unreachable(@slave)
+        manager.force_unavailable(@slave)
       end
 
       context 'slave node with a master present' do
-        it 'removes slave from unreachable list' do
-          manager.force_reachable(@slave)
-          manager.nodes[:unreachable].should be_empty
+        it 'removes slave from unavailable list' do
+          manager.force_available(@slave)
+          manager.nodes[:unavailable].should be_empty
           manager.nodes[:slaves].should include(@slave.to_s)
         end
+
+        it 'makes node a slave of new master' do
+          manager.master = Node.new(:host => 'foo', :port => '7892')
+          manager.force_available(@slave)
+          @slave.fetch_info.should == {
+            :role => 'slave',
+            :master_host => 'foo',
+            :master_port => '7892'}
+        end
+
+        it 'does not invoke slaveof operation if master has not changed' do
+          @slave.redis.should_not_receive(:slaveof)
+          manager.force_available(@slave)
+        end
       end
 
       context 'slave node with no master present' do
         before(:each) do
           @master = manager.master
-          manager.force_unreachable(@master)
+          manager.force_unavailable(@master)
         end
 
         it 'promotes slave to master' do
           manager.master.should be_nil
-          manager.force_reachable(@slave)
+          manager.force_available(@slave)
           manager.master.should == @slave
         end
 
@@ -72,5 +86,24 @@ module RedisFailover
         end
       end
     end
+
+    describe '#handle_syncing' do
+      context 'prohibits stale reads' do
+        it 'adds node to unavailable list' do
+          slave = manager.slaves.first
+          manager.force_syncing(slave, false)
+          manager.nodes[:unavailable].should include(slave.to_s)
+        end
+      end
+
+      context 'allows stale reads' do
+        it 'makes node available' do
+          slave = manager.slaves.first
+          manager.force_syncing(slave, true)
+          manager.nodes[:unavailable].should_not include(slave.to_s)
+          manager.nodes[:slaves].should include(slave.to_s)
+        end
+      end
+    end
   end
 end

data/spec/node_spec.rb

@@ -20,13 +20,13 @@ module RedisFailover
     end
 
     describe '#ping' do
-      it 'responds properly if node is reachable' do
+      it 'responds properly if node is available' do
         expect { node.ping }.to_not raise_error
       end
 
-      it 'responds properly if node is unreachable' do
-        node.redis.make_unreachable!
-        expect { node.ping }.to raise_error(NodeUnreachableError)
+      it 'responds properly if node is unavailable' do
+        node.redis.make_unavailable!
+        expect { node.ping }.to raise_error(NodeUnavailableError)
       end
     end
 
@@ -53,24 +53,32 @@ module RedisFailover
       end
     end
 
-    describe '#wait_until_unreachable' do
+    describe '#wait' do
       it 'should wait until node dies' do
-        thread = Thread.new { node.wait_until_unreachable }
+        thread = Thread.new { node.wait }
         thread.should be_alive
-        node.redis.make_unreachable!
+        node.redis.make_unavailable!
         expect { thread.value }.to raise_error
       end
     end
 
-    describe '#stop_waiting' do
+    describe '#wakeup' do
       it 'should gracefully stop waiting' do
-        thread = Thread.new { node.wait_until_unreachable }
+        thread = Thread.new { node.wait }
         thread.should be_alive
-        node.stop_waiting
+        node.wakeup
         sleep 0.2
         thread.should_not be_alive
         thread.value.should be_nil
       end
     end
+
+    describe '#perform_operation' do
+      it 'raises error for any operation that hangs for too long' do
+        expect do
+          node.send(:perform_operation) { 1_000_000.times { sleep 0.1 } }
+        end.to raise_error(NodeUnavailableError)
+      end
+    end
   end
 end

data/spec/node_watcher_spec.rb

@@ -20,23 +20,38 @@ module RedisFailover
     let(:node) { Node.new(:host => 'host', :port => 123).extend(RedisStubSupport) }
 
     describe '#watch' do
-      it 'properly informs manager of unreachable node' do
-        watcher = NodeWatcher.new(node_manager, node, 1)
-        watcher.watch
-        sleep(3)
-        node.redis.make_unreachable!
-        sleep(3)
-        watcher.shutdown
-        node_manager.state_for(node).should == :unreachable
+      context 'node is not syncing with master' do
+        it 'properly informs manager of unavailable node' do
+          watcher = NodeWatcher.new(node_manager, node, 1)
+          watcher.watch
+          sleep(3)
+          node.redis.make_unavailable!
+          sleep(3)
+          watcher.shutdown
+          node_manager.state_for(node).should == :unavailable
+        end
+
+        it 'properly informs manager of available node' do
+          node_manager.notify_state_change(node, :unavailable)
+          watcher = NodeWatcher.new(node_manager, node, 1)
+          watcher.watch
+          sleep(3)
+          watcher.shutdown
+          node_manager.state_for(node).should == :available
+        end
       end
 
-      it 'properly informs manager of reachable node' do
-        node_manager.notify_state_change(node, :unreachable)
-        watcher = NodeWatcher.new(node_manager, node, 1)
-        watcher.watch
-        sleep(3)
-        watcher.shutdown
-        node_manager.state_for(node).should == :reachable
+      context 'node is syncing with master' do
+        it 'properly informs manager of syncing node' do
+          node_manager.notify_state_change(node, :unavailable)
+          node.redis.slaveof('masterhost', 9876)
+          node.redis.force_sync_with_master(true)
+          watcher = NodeWatcher.new(node_manager, node, 1)
+          watcher.watch
+          sleep(3)
+          watcher.shutdown
+          node_manager.state_for(node).should == :syncing
+        end
       end
     end
   end

data/spec/support/node_manager_stub.rb

@@ -1,5 +1,7 @@
 module RedisFailover
   class NodeManagerStub < NodeManager
+    attr_accessor :master
+
     def parse_nodes
       master = Node.new(:host => 'master')
       slave = Node.new(:host => 'slave')
@@ -9,10 +11,6 @@ module RedisFailover
       [master, [slave]]
     end
 
-    def master
-      @master
-    end
-
     def slaves
       @slaves
     end
@@ -26,17 +24,24 @@ module RedisFailover
       @thread.value
     end
 
-    def force_unreachable(node)
+    def force_unavailable(node)
+      start_processing
+      node.redis.make_unavailable!
+      notify_state_change(node, :unavailable)
+      stop_processing
+    end
+
+    def force_available(node)
       start_processing
-      node.redis.make_unreachable!
-      notify_state_change(node, :unreachable)
+      node.redis.make_available!
+      notify_state_change(node, :available)
       stop_processing
     end
 
-    def force_reachable(node)
+    def force_syncing(node, serve_stale_reads)
       start_processing
-      node.redis.make_reachable!
-      notify_state_change(node, :reachable)
+      node.redis.force_sync_with_master(serve_stale_reads)
+      notify_state_change(node, :syncing)
       stop_processing
     end
   end

data/spec/support/redis_stub.rb

@@ -6,6 +6,7 @@ module RedisFailover
     class Proxy
       def initialize(queue, opts = {})
         @info = {'role' => 'master'}
+        @config = {'slave-serve-stale-data' => 'yes'}
         @queue = queue
       end
 
@@ -25,8 +26,12 @@ module RedisFailover
       def slaveof(host, port)
         if host == 'no' && port == 'one'
           @info['role'] = 'master'
+          @info.delete('master_host')
+          @info.delete('master_port')
         else
           @info['role'] = 'slave'
+          @info['master_host'] = host
+          @info['master_port'] = port.to_s
         end
       end
 
@@ -34,26 +39,35 @@ module RedisFailover
         @info.dup
       end
 
-      def ping
-        'pong'
-      end
-
       def change_role_to(role)
         @info['role'] = role
       end
+
+      def config(action, attribute)
+        [action, @config[attribute]]
+      end
+
+      def force_sync_with_master(serve_stale_reads)
+        @config['slave-serve-stale-data'] = serve_stale_reads ? 'yes' : 'no'
+        @info['master_sync_in_progress'] = '1'
+      end
+
+      def force_sync_done
+        @info['master_sync_in_progress'] = '0'
+      end
     end
 
-    attr_reader :host, :port, :reachable
+    attr_reader :host, :port, :available
     def initialize(opts = {})
       @host = opts[:host]
       @port = Integer(opts[:port])
       @queue = Queue.new
       @proxy = Proxy.new(@queue, opts)
-      @reachable = true
+      @available = true
     end
 
     def method_missing(method, *args, &block)
-      if @reachable
+      if @available
         @proxy.send(method, *args, &block)
       else
         raise Errno::ECONNREFUSED
@@ -64,13 +78,13 @@ module RedisFailover
       @proxy.change_role_to(role)
     end
 
-    def make_reachable!
-      @reachable = true
+    def make_available!
+      @available = true
     end
 
-    def make_unreachable!
+    def make_unavailable!
       @queue << Errno::ECONNREFUSED
-      @reachable = false
+      @available = false
     end
 
     def to_s
@@ -86,5 +100,6 @@ module RedisFailover
     def redis
       @redis ||= RedisStub.new(:host => @host, :port => @port)
     end
+    alias_method :new_client, :redis
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redis_failover
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-04-14 00:00:00.000000000 Z
+date: 2012-04-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
-  requirement: &70202974273640 !ruby/object:Gem::Requirement
+  requirement: &70210559246440 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70202974273640
+  version_requirements: *70210559246440
 - !ruby/object:Gem::Dependency
   name: redis-namespace
-  requirement: &70202974273200 !ruby/object:Gem::Requirement
+  requirement: &70210559246000 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,10 +32,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70202974273200
+  version_requirements: *70210559246000
 - !ruby/object:Gem::Dependency
   name: multi_json
-  requirement: &70202974272780 !ruby/object:Gem::Requirement
+  requirement: &70210559245580 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -43,10 +43,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70202974272780
+  version_requirements: *70210559245580
 - !ruby/object:Gem::Dependency
   name: sinatra
-  requirement: &70202974272360 !ruby/object:Gem::Requirement
+  requirement: &70210559245160 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -54,10 +54,10 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70202974272360
+  version_requirements: *70210559245160
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70202974271940 !ruby/object:Gem::Requirement
+  requirement: &70210559244740 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +65,10 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70202974271940
+  version_requirements: *70210559244740
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70202974271520 !ruby/object:Gem::Requirement
+  requirement: &70210559244320 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,7 +76,7 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70202974271520
+  version_requirements: *70210559244320
 description: Redis Failover provides a full automatic master/slave failover solution
   for Ruby
 email:
@@ -129,7 +129,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1781006209108187327
+      hash: -4081812195470581448
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -138,7 +138,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1781006209108187327
+      hash: -4081812195470581448
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.16