redis_failover 0.5.4 → 0.6.0

data/.travis.yml

@@ -2,3 +2,4 @@ language: ruby
 rvm:
   - 1.9.2
   - 1.9.3
+  - jruby-19mode

data/Changes.md

@@ -1,3 +1,9 @@
+0.6.0
+-----------
+- Add support for running multiple Node Manager processes for added redundancy (#4)
+- Add support for specifying a redis database in RedisFailover::Client (#5)
+- Improved Node Manager command-line option parsing
+
 0.5.4
 -----------
 - No longer use problematic ZK#reopen.

data/README.md

@@ -22,7 +22,8 @@ nodes. Note that detection of a node going down should be nearly instantaneous,
 used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
 immediately when the node actually goes offline. To avoid false positives (i.e., intermittent flaky
 network interruption), the Node Manager will only mark a node as unavailable if it fails to communicate with
-it 3 times (this is configurable via --max-failures, see configuration options below).
+it 3 times (this is configurable via --max-failures, see configuration options below). Note that you can
+deploy multiple Node Manager daemons for added redundancy.
 
 This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
 with a list of ZooKeeper servers. The client will automatically contact the ZooKeeper cluster to find out
@@ -56,12 +57,13 @@ The Node Manager is a simple process that should be run as a background daemon.
 following options:
 
     Usage: redis_node_manager [OPTIONS]
-        -p, --password password          Redis password (optional)
-        -n, --nodes redis nodes          Comma-separated redis host:port pairs (required)
-        -z zookeeper servers,            Comma-separated ZooKeeper host:port pairs (required)
-            --zkservers
-            --znode-path path            Znode path override for storing redis server list (optional)
-            --max-failures count         Max failures before manager marks node unavailable (default 3)
+
+    Specific options:
+        -n, --nodes NODES                Comma-separated redis host:port pairs
+        -z, --zkservers SERVERS          Comma-separated ZooKeeper host:port pairs
+        -p, --password [PASSWORD]        Redis password
+            --znode-path [PATH]          Znode path override for storing redis server list
+            --max-failures [COUNT]       Max failures before manager marks node unavailable
         -h, --help                       Display all options
 
 To start the daemon for a simple master/slave configuration, use the following:
@@ -69,8 +71,11 @@ To start the daemon for a simple master/slave configuration, use the following:
     redis_node_manager -n localhost:6379,localhost:6380 -z localhost:2181,localhost:2182,localhost:2183
 
 The Node Manager will automatically discover the master/slaves upon startup. Note that it is
-a good idea to monitor the redis Node Manager daemon process with a tool like Monit to ensure that it is restarted
-in the case of a failure.
+a good idea to run more than one instance of the Node Manager daemon in your environment. At
+any moment, a single Node Manager process will be designated to monitor the redis servers. If
+this Node Manager process dies or becomes partitioned from the network, another Node Manager
+will be promoted as the primary monitor of redis servers. You can run as many Node Manager
+processes as you'd like for added redundancy.
 
 ## Client Usage
 
@@ -88,6 +93,7 @@ The full set of options that can be passed to RedisFailover::Client are:
      :zkservers     - comma-separated ZooKeeper host:port pairs (required)
      :znode_path    - the Znode path override for redis server list (optional)
      :password      - password for redis nodes (optional)
+     :db            - db to use for redis nodes (optional)
      :namespace     - namespace for redis nodes (optional)
      :logger        - logger override (optional)
      :retry_failure - indicate if failures should be retried (default true)
@@ -95,7 +101,7 @@ The full set of options that can be passed to RedisFailover::Client are:
 
 ## Requirements
 
-- redis_failover is actively tested against MRI 1.9.2/1.9.3. Other rubies may work, although I don't actively test against them. 1.8 is not supported.
+- redis_failover is actively tested against MRI 1.9.2/1.9.3 and JRuby 1.6.7 (1.9 mode only). Other rubies may work, although I don't actively test against them. 1.8 is not supported.
 - redis_failover requires a ZooKeeper service cluster to ensure reliability and data consistency. ZooKeeper is very simple and easy to get up and running. Please refer to this [Quick ZooKeeper Guide](https://github.com/ryanlecompte/redis_failover/wiki/Quick-ZooKeeper-Guide) to get up and running quickly if you don't already have ZooKeeper as a part of your environment.
 
 ## Considerations
@@ -104,11 +110,6 @@ The full set of options that can be passed to RedisFailover::Client are:
 
 - Note that it's still possible for the RedisFailover::Client instances to see a stale list of servers for a very small window. In most cases this will not be the case due to how ZooKeeper handles distributed communication, but you should be aware that in the worst case the client could write to a "stale" master for a small period of time until the next watch event is received by the client via ZooKeeper.
 
-## TODO
-
-- Rework specs to work against a set of real Redis/ZooKeeper nodes as opposed to stubs.
-- Add support for running more than one Node Manager.
-
 ## Resources
 
 - To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).

data/lib/redis_failover.rb

@@ -1,4 +1,5 @@
 require 'zk'
+require 'set'
 require 'redis'
 require 'thread'
 require 'logger'

data/lib/redis_failover/cli.rb

@@ -2,36 +2,36 @@ module RedisFailover
   # Parses server command-line arguments.
   class CLI
     def self.parse(source)
-      return {} if source.empty?
-
       options = {}
       parser = OptionParser.new do |opts|
         opts.banner = "Usage: redis_node_manager [OPTIONS]"
+        opts.separator ""
+        opts.separator "Specific options:"
 
-        opts.on('-p', '--password password', 'Redis password (optional)') do |password|
-          options[:password] = password.strip
-        end
-
-        opts.on('-n', '--nodes redis nodes',
-          'Comma-separated redis host:port pairs (required)') do |nodes|
+        opts.on('-n', '--nodes NODES',
+          'Comma-separated redis host:port pairs') do |nodes|
           # turns 'host1:port,host2:port' => [{:host => host, :port => port}, ...]
           options[:nodes] = nodes.split(',').map do |node|
             Hash[[:host, :port].zip(node.strip.split(':'))]
           end
         end
 
-        opts.on('-z', '--zkservers ZooKeeper servers',
-          'Comma-separated ZooKeeper host:port pairs (required)') do |servers|
-          options[:zkservers] = servers
+        opts.on('-z', '--zkservers SERVERS',
+          'Comma-separated ZooKeeper host:port pairs') do |servers|
+          options[:zkservers] = servers.strip
         end
 
-        opts.on('--znode-path path',
-          'Znode path override for storing redis server list (optional)') do |path|
+        opts.on('-p', '--password [PASSWORD]', 'Redis password') do |password|
+          options[:password] = password.strip
+        end
+
+        opts.on('--znode-path [PATH]',
+          'Znode path override for storing redis server list') do |path|
           options[:znode_path] = path
         end
 
-        opts.on('--max-failures count',
-          'Max failures before manager marks node unavailable (default 3)') do |max|
+        opts.on('--max-failures [COUNT]',
+          'Max failures before manager marks node unavailable') do |max|
           options[:max_failures] = Integer(max)
         end
 
@@ -42,6 +42,11 @@ module RedisFailover
       end
 
       parser.parse(source)
+      if required_options_missing?(options)
+        puts parser
+        exit
+      end
+
       # assume password is same for all redis nodes
       if password = options[:password]
         options[:nodes].each { |opts| opts.update(:password => password) }
@@ -49,5 +54,11 @@ module RedisFailover
 
       options
     end
+
+    def self.required_options_missing?(options)
+      return true if options.empty?
+      return true unless options.values_at(:nodes, :zkservers).all?
+      false
+    end
   end
 end

data/lib/redis_failover/client.rb

@@ -1,12 +1,32 @@
-require 'set'
-
 module RedisFailover
-  # Redis failover-aware client.
+  # Redis failover-aware client. RedisFailover::Client is a wrapper over a set of underlying redis
+  # clients, which means all normal redis operations can be performed on an instance of this class.
+  # The class only requires a set of ZooKeeper server addresses to function properly. The client
+  # will automatically retry failed operations, and handle failover to a new master. The client
+  # registers and listens for watcher events from the Node Manager. When these events are received,
+  # the client fetches the latest set of redis nodes from ZooKeeper and rebuilds its internal
+  # Redis clients appropriately. RedisFailover::Client also directs write operations to the master,
+  # and all read operations to the slaves.
+  #
+  # Examples
+  #
+  #   client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
+  #   client.set('foo', 1) # will be directed to master
+  #   client.get('foo') # will be directed to a slave
+  #
   class Client
     include Util
 
+    # Maximum allowed elapsed time between notifications from the Node Manager.
+    # When this timeout is reached, the client will raise a NoNodeManagerError
+    # and purge its internal redis clients.
     ZNODE_UPDATE_TIMEOUT = 9
+
+    # Amount of time to sleep before retrying a failed operation.
     RETRY_WAIT_TIME = 3
+
+    # Redis read operations that are automatically dispatched to slaves. Any
+    # operation not listed here will be dispatched to the master.
     REDIS_READ_OPS = Set[
       :echo,
       :exists,
@@ -47,6 +67,8 @@ module RedisFailover
       :zscore
     ].freeze
 
+    # Unsupported Redis operations. These don't make sense in a client
+    # that abstracts the master/slave servers.
     UNSUPPORTED_OPS = Set[
       :select,
       :ttl,
@@ -66,13 +88,14 @@ module RedisFailover
     #
     # Options:
     #
-    #   :zkservers - comma-separated ZooKeeper host:port pairs (required)
-    #   :znode_path - the Znode path override for redis server list (optional)
-    #   :password - password for redis nodes (optional)
-    #   :namespace - namespace for redis nodes (optional)
-    #   :logger - logger override (optional)
+    #   :zkservers     - comma-separated ZooKeeper host:port pairs (required)
+    #   :znode_path    - the Znode path override for redis server list (optional)
+    #   :password      - password for redis nodes (optional)
+    #   :db            - db to use for redis nodes (optional)
+    #   :namespace     - namespace for redis nodes (optional)
+    #   :logger        - logger override (optional)
     #   :retry_failure - indicate if failures should be retried (default true)
-    #   :max_retries - max retries for a failure (default 3)
+    #   :max_retries   - max retries for a failure (default 3)
     #
     def initialize(options = {})
       Util.logger = options[:logger] if options[:logger]
@@ -80,6 +103,7 @@ module RedisFailover
       @znode = options[:znode_path] || Util::DEFAULT_ZNODE_PATH
       @namespace = options[:namespace]
       @password = options[:password]
+      @db = options[:db]
       @retry = options[:retry_failure] || true
       @max_retries = @retry ? options.fetch(:max_retries, 3) : 1
       @master = nil
@@ -89,6 +113,7 @@ module RedisFailover
       build_clients
     end
 
+    # Dispatches redis operations to master/slaves.
     def method_missing(method, *args, &block)
       if redis_operation?(method)
         dispatch(method, *args, &block)
@@ -102,7 +127,7 @@ module RedisFailover
     end
 
     def inspect
-      "#<RedisFailover::Client - master: #{master_name}, slaves: #{slave_names})>"
+      "#<RedisFailover::Client (master: #{master_name}, slaves: #{slave_names})>"
     end
     alias_method :to_s, :inspect
 
@@ -163,13 +188,14 @@ module RedisFailover
           master.send(method, *args, &block)
         end
       rescue *CONNECTIVITY_ERRORS => ex
-        logger.error("Error while handling operation `#{method}` - #{ex.inspect}")
+        logger.error("Error while handling `#{method}` - #{ex.inspect}")
         logger.error(ex.backtrace.join("\n"))
 
         if tries < @max_retries
           tries += 1
           build_clients
-          sleep(RETRY_WAIT_TIME) && retry
+          sleep(RETRY_WAIT_TIME)
+          retry
         end
 
         raise
@@ -214,7 +240,8 @@ module RedisFailover
 
           if tries < @max_retries
             tries += 1
-            sleep(RETRY_WAIT_TIME) && retry
+            sleep(RETRY_WAIT_TIME)
+            retry
           end
 
           raise
@@ -233,7 +260,10 @@ module RedisFailover
     def new_clients_for(*nodes)
       nodes.map do |node|
         host, port = node.split(':')
-        client = Redis.new(:host => host, :port => port, :password => @password)
+        opts = {:host => host, :port => port}
+        opts.update(:db => @db) if @db
+        opts.update(:password => @password) if @password
+        client = Redis.new(opts)
         if @namespace
           client = Redis::Namespace.new(@namespace, :redis => client)
         end

data/lib/redis_failover/node.rb

@@ -1,8 +1,13 @@
 module RedisFailover
-  # Represents a redis node (master or slave).
+  # Represents a redis node (master or slave). Instances of this class
+  # are used by the NodeManager and NodeWatcher to manipulate real redis
+  # servers.
   class Node
     include Util
 
+    # Maximum amount of time given for any redis operation to complete.
+    # If a redis operation doesn't complete in the alotted time, a
+    # NodeUnavailableError will be raised.
     MAX_OP_WAIT_TIME = 5
 
     attr_reader :host, :port

data/lib/redis_failover/node_manager.rb

@@ -1,24 +1,51 @@
 module RedisFailover
-  # NodeManager manages a list of redis nodes.
+  # NodeManager manages a list of redis nodes. Upon startup, the NodeManager
+  # will discover the current redis master and slaves. Each redis node is
+  # monitored by a NodeWatcher instance. The NodeWatchers periodically
+  # report the current state of the redis node it's watching to the
+  # NodeManager via an asynchronous queue. The NodeManager processes the
+  # state reports and reacts appropriately by handling stale/dead nodes,
+  # and promoting a new redis master if it sees fit to do so.
   class NodeManager
     include Util
 
+    # Name for the znode that handles exclusive locking between multiple
+    # Node Manager processes. Whoever holds the lock will be considered
+    # the "master" Node Manager, and will be responsible for monitoring
+    # the redis nodes. When a Node Manager that holds the lock disappears
+    # or fails, another Node Manager process will grab the lock and
+    # become the master.
+    LOCK_PATH = 'master_node_manager'
+
+    # Number of seconds to wait before retrying bootstrap process.
+    TIMEOUT = 3
+
     def initialize(options)
+      logger.info("Redis Node Manager v#{VERSION} starting (#{RUBY_DESCRIPTION})")
       @options = options
-      @zkclient = ZkClient.new(@options[:zkservers])
       @znode = @options[:znode_path] || Util::DEFAULT_ZNODE_PATH
       @unavailable = []
       @queue = Queue.new
-      discover_nodes
     end
 
     def start
-      initialize_path
-      spawn_watchers
-      handle_state_changes
+      @zkclient = ZkClient.new(@options[:zkservers])
+      logger.info('Waiting to become master Node Manager ...')
+      @zkclient.with_lock(LOCK_PATH) do
+        logger.info('Acquired master Node Manager lock')
+        discover_nodes
+        initialize_path
+        spawn_watchers
+        handle_state_reports
+      end
+    rescue *CONNECTIVITY_ERRORS => ex
+      logger.error("Error while attempting to manage nodes: #{ex.inspect}")
+      logger.error(ex.backtrace.join("\n"))
+      sleep(TIMEOUT)
+      retry
     end
 
-    def notify_state_change(node, state)
+    def notify_state(node, state)
       @queue << [node, state]
     end
 
@@ -29,10 +56,10 @@ module RedisFailover
 
     private
 
-    def handle_state_changes
-      while state_change = @queue.pop
+    def handle_state_reports
+      while state_report = @queue.pop
         begin
-          node, state = state_change
+          node, state = state_report
           case state
           when :unavailable then handle_unavailable(node)
           when :available   then handle_available(node)
@@ -43,7 +70,7 @@ module RedisFailover
           # flush current state
           write_state
         rescue StandardError, *CONNECTIVITY_ERRORS => ex
-          logger.error("Error while handling #{state_change.inspect}: #{ex.inspect}")
+          logger.error("Error handling #{state_report.inspect}: #{ex.inspect}")
           logger.error(ex.backtrace.join("\n"))
         end
       end

data/lib/redis_failover/node_watcher.rb

@@ -1,8 +1,11 @@
 module RedisFailover
-  # Watches a specific redis node for its availability.
+  # NodeWatcher periodically monitors a specific redis node for its availability.
+  # NodeWatcher instances periodically report a redis node's current state
+  # to the NodeManager for proper handling.
   class NodeWatcher
     include Util
 
+    # Time to sleep before checking on the monitored node's status.
     WATCHER_SLEEP_TIME = 2
 
     def initialize(manager, node, max_failures)
@@ -55,7 +58,7 @@ module RedisFailover
     end
 
     def notify(state)
-      @manager.notify_state_change(@node, state)
+      @manager.notify_state(@node, state)
     end
   end
 end

data/lib/redis_failover/runner.rb

@@ -6,7 +6,6 @@ module RedisFailover
       @node_manager = NodeManager.new(options)
       trap_signals
       node_manager_thread = Thread.new { @node_manager.start }
-      Util.logger.info("Redis Node Manager v#{VERSION} successfully started (#{RUBY_DESCRIPTION})")
       node_manager_thread.join
     end
 

data/lib/redis_failover/util.rb

@@ -1,12 +1,17 @@
 require 'redis_failover/errors'
 
 module RedisFailover
-  # Common utiilty methods.
+  # Common utiilty methods and constants.
   module Util
     extend self
 
+    # Default node in ZooKeeper that contains the current list of available redis nodes.
     DEFAULT_ZNODE_PATH = '/redis_failover_nodes'.freeze
+
+    # Connectivity errors that the redis client raises.
     REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }.freeze
+
+    # Full set of errors related to connectivity.
     CONNECTIVITY_ERRORS = [
       RedisFailover::Error,
       ZK::Exceptions::KeeperException,

data/lib/redis_failover/version.rb

@@ -1,3 +1,3 @@
 module RedisFailover
-  VERSION = "0.5.4"
+  VERSION = "0.6.0"
 end

data/lib/redis_failover/zk_client.rb

@@ -4,7 +4,13 @@ module RedisFailover
   class ZkClient
     include Util
 
+    # Time to sleep before retrying a failed operation.
+    TIMEOUT = 2
+
+    # Maximum reconnect attempts.
     MAX_RECONNECTS = 3
+
+    # Errors that are candidates for rebuilding the underlying ZK client.
     RECONNECTABLE_ERRORS = [
       ZookeeperExceptions::ZookeeperException::SessionExpired,
       ZookeeperExceptions::ZookeeperException::SystemError,
@@ -15,6 +21,8 @@ module RedisFailover
       ZookeeperExceptions::ZookeeperException::ConnectionClosed,
       ZookeeperExceptions::ZookeeperException::NotConnected
     ].freeze
+
+    # ZK methods that are wrapped with reconnect logic.
     WRAPPED_ZK_METHODS = [
       :get,
       :set,
@@ -22,7 +30,8 @@ module RedisFailover
       :event_handler,
       :stat,
       :create,
-      :delete].freeze
+      :delete,
+      :with_lock].freeze
 
     def initialize(servers, &setup_block)
       @servers = servers
@@ -42,7 +51,7 @@ module RedisFailover
     end
 
     WRAPPED_ZK_METHODS.each do |zk_method|
-      class_eval <<-RUBY
+      class_eval(<<-RUBY, __FILE__, __LINE__ + 1)
         def #{zk_method}(*args, &block)
           perform_with_reconnect do
             @client.#{zk_method}(*args, &block)
@@ -58,14 +67,15 @@ module RedisFailover
       begin
         yield
       rescue *RECONNECTABLE_ERRORS => ex
-        logger.error("ZooKeeper client connection error - rebuilding client: #{ex.inspect}")
+        logger.error("ZooKeeper connection error, rebuilding client: #{ex.inspect}")
         logger.error(ex.backtrace.join("\n"))
         if tries < MAX_RECONNECTS
           tries += 1
           @on_session_expiration.call if @on_session_expiration
           build_client
           @on_session_recovered.call if @on_session_recovered
-          sleep(2) && retry
+          sleep(TIMEOUT)
+          retry
         end
 
         raise

data/redis_failover.gemspec

@@ -17,8 +17,8 @@ Gem::Specification.new do |gem|
 
   gem.add_dependency('redis')
   gem.add_dependency('redis-namespace')
-  gem.add_dependency('multi_json')
-  gem.add_dependency('zk')
+  gem.add_dependency('multi_json', '>= 1.0', '< 1.3')
+  gem.add_dependency('zk', '~> 0.8.8')
 
   gem.add_development_dependency('rake')
   gem.add_development_dependency('rspec')

data/spec/cli_spec.rb

@@ -3,12 +3,8 @@ require 'spec_helper'
 module RedisFailover
   describe CLI do
     describe '.parse' do
-      it 'returns empty result for empty options' do
-        CLI.parse({}).should == {}
-      end
-
       it 'properly parses redis nodes' do
-        opts = CLI.parse(['-n host1:1,host2:2,host3:3'])
+        opts = CLI.parse(['-n host1:1,host2:2,host3:3', '-z localhost:1111'])
         opts[:nodes].should == [
           {:host => 'host1', :port => '1'},
           {:host => 'host2', :port => '2'},
@@ -16,8 +12,13 @@ module RedisFailover
         ]
       end
 
+      it 'properly parses ZooKeeper servers' do
+        opts = CLI.parse(['-n host1:1,host2:2,host3:3', '-z localhost:1111'])
+        opts[:zkservers].should == 'localhost:1111'
+      end
+
       it 'properly parses a redis password' do
-        opts = CLI.parse(['-n host:port', '-p redis_pass'])
+        opts = CLI.parse(['-n host:port', '-z localhost:1111', '-p redis_pass'])
         opts[:nodes].should == [{
           :host => 'host',
           :port => 'port',
@@ -26,7 +27,12 @@ module RedisFailover
       end
 
       it 'properly parses max node failures' do
-        opts = CLI.parse(['-n host:port', '-p redis_pass', '--max-failures', '1'])
+        opts = CLI.parse([
+          '-n host:port',
+          '-z localhost:1111',
+          '-p redis_pass',
+          '--max-failures',
+          '1'])
         opts[:max_failures].should == 1
       end
     end

data/spec/node_manager_spec.rb

@@ -4,6 +4,10 @@ module RedisFailover
   describe NodeManager do
     let(:manager) { NodeManagerStub.new({}) }
 
+    before(:each) do
+      manager.discover_nodes
+    end
+
     describe '#nodes' do
       it 'returns current master and slave nodes' do
         manager.current_nodes.should == {

data/spec/node_watcher_spec.rb

@@ -6,7 +6,7 @@ module RedisFailover
       @node_states = {}
     end
 
-    def notify_state_change(node, state)
+    def notify_state(node, state)
       @node_states[node] = state
     end
 
@@ -32,7 +32,7 @@ module RedisFailover
         end
 
         it 'properly informs manager of available node' do
-          node_manager.notify_state_change(node, :unavailable)
+          node_manager.notify_state(node, :unavailable)
           watcher = NodeWatcher.new(node_manager, node, 1)
           watcher.watch
           sleep(3)
@@ -43,7 +43,7 @@ module RedisFailover
 
       context 'node is syncing with master' do
         it 'properly informs manager of syncing node' do
-          node_manager.notify_state_change(node, :unavailable)
+          node_manager.notify_state(node, :unavailable)
           node.redis.slaveof('masterhost', 9876)
           node.redis.force_sync_with_master(true)
           watcher = NodeWatcher.new(node_manager, node, 1)

data/spec/spec_helper.rb

@@ -5,6 +5,7 @@ Dir["#{File.dirname(__FILE__)}/support/**/*.rb"].each { |f| require f }
 
 class NullObject
   def method_missing(method, *args, &block)
+    yield if block_given?
     self
   end
 end

data/spec/support/node_manager_stub.rb

@@ -3,7 +3,20 @@ module RedisFailover
     attr_accessor :master
     public :current_nodes
 
+    def initialize(options)
+      super
+      @zklock = Object.new
+      @zklock.instance_eval do
+        def with_lock
+          yield
+        end
+      end
+    end
+
     def discover_nodes
+      # only discover nodes once in testing
+      return if @nodes_discovered
+
       master = Node.new(:host => 'master')
       slave = Node.new(:host => 'slave')
       [master, slave].each { |node| node.extend(RedisStubSupport) }
@@ -11,6 +24,7 @@ module RedisFailover
       slave.make_slave!(master)
       @master = master
       @slaves = [slave]
+      @nodes_discovered = true
     end
 
     def slaves
@@ -29,21 +43,21 @@ module RedisFailover
     def force_unavailable(node)
       start_processing
       node.redis.make_unavailable!
-      notify_state_change(node, :unavailable)
+      notify_state(node, :unavailable)
       stop_processing
     end
 
     def force_available(node)
       start_processing
       node.redis.make_available!
-      notify_state_change(node, :available)
+      notify_state(node, :available)
       stop_processing
     end
 
     def force_syncing(node, serve_stale_reads)
       start_processing
       node.redis.force_sync_with_master(serve_stale_reads)
-      notify_state_change(node, :syncing)
+      notify_state(node, :syncing)
       stop_processing
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redis_failover
 version: !ruby/object:Gem::Version
-  version: 0.5.4
+  version: 0.6.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-04-18 00:00:00.000000000 Z
+date: 2012-04-21 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
-  requirement: &70362366935580 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,15 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70362366935580
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: redis-namespace
-  requirement: &70362366935160 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,32 +37,53 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70362366935160
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: multi_json
-  requirement: &70362366934740 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
       - !ruby/object:Gem::Version
-        version: '0'
+        version: '1.0'
+    - - <
+      - !ruby/object:Gem::Version
+        version: '1.3'
   type: :runtime
   prerelease: false
-  version_requirements: *70362366934740
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '1.0'
+    - - <
+      - !ruby/object:Gem::Version
+        version: '1.3'
 - !ruby/object:Gem::Dependency
   name: zk
-  requirement: &70362366934320 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.8.8
   type: :runtime
   prerelease: false
-  version_requirements: *70362366934320
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: 0.8.8
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70362366933900 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -65,10 +91,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70362366933900
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70362366933480 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -76,7 +107,12 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70362366933480
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Redis Failover is a ZooKeeper-based automatic master/slave failover solution
   for Ruby
 email:
@@ -129,7 +165,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 610999762978001666
+      hash: 2836022562258381397
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -138,10 +174,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 610999762978001666
+      hash: 2836022562258381397
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.16
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Redis Failover is a ZooKeeper-based automatic master/slave failover solution