redis_failover 0.8.9 → 0.9.0

data/Changes.md

@@ -1,3 +1,14 @@
+0.9.0
+-----------
+- Make Node Manager's lock path vary with its main znode. (Bira)
+- Node Manager's znode for holding current list of redis nodes is no longer ephemeral. This is unnecessary since the current master should only be changed by redis_failover.
+- Introduce :master_only option for RedisFailover::Client (disabled by default). This option configures the client to direct all read/write operations to the master.
+- Introduce :safe_mode option (enabled by default). This option configures the client to purge its redis clients when a ZK session expires or when the client hasn't recently heard from the node manager.
+- Introduce RedisFailover::Client#on_node_change callback notification for when the currently known list of master/slave redis nodes changes.
+- Added #current_master and #current_slaves to RedisFailover::Client. This is useful for programmatically doing things based on the current master/slaves.
+- redis_node_manager should start if no redis servers are available (#29)
+- Better handling of ZK session expirations in Node Manager.
+
 0.8.9
 -----------
 - Handle errors raised by redis 3.x client (tsilen) 

data/README.md

@@ -129,6 +129,16 @@ The full set of options that can be passed to RedisFailover::Client are:
      :logger        - logger override (optional)
      :retry_failure - indicate if failures should be retried (default true)
      :max_retries   - max retries for a failure (default 3)
+     :safe_mode     - indicates if safe mode is used or not (default true)
+     :master_only   - indicates if only redis master is used (default false)
+
+The RedisFailover::Client also supports a custom callback that will be invoked whenever the list of redis clients changes. Example usage:
+
+    RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183') do |client|
+      client.on_node_change do |master, slaves|
+        logger.info("Nodes changed! master: #{master}, slaves: #{slaves}")
+      end
+    end
 
 ## Manual Failover
 
@@ -160,11 +170,13 @@ redis_failover uses YARD for its API documentation. Refer to the generated [API
 
 ## Resources
 
+- Check out Steve Whittaker's [redis-failover-test](https://github.com/swhitt/redis-failover-test) project which shows how to test redis_failover in a non-trivial configuration using Vagrant/Chef.
 - To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).
 - To learn more about ZooKeeper, see the official [ZooKeeper](http://zookeeper.apache.org/) site.
 - See the [Quick ZooKeeper Guide](https://github.com/ryanlecompte/redis_failover/wiki/Quick-ZooKeeper-Guide) for a quick guide to getting ZooKeeper up and running with redis_failover.
 - To learn more about how ZooKeeper handles network partitions, see [ZooKeeper Failure Scenarios](http://wiki.apache.org/hadoop/ZooKeeper/FailureScenarios)
 
+
 ## License
 
 Please see LICENSE for licensing details.

data/lib/redis_failover/client.rb

@@ -27,52 +27,6 @@ module RedisFailover
     # Amount of time to sleep before retrying a failed operation.
     RETRY_WAIT_TIME = 3
 
-    # Redis read operations that are automatically dispatched to slaves. Any
-    # operation not listed here will be dispatched to the master.
-    REDIS_READ_OPS = Set[
-      :echo,
-      :exists,
-      :get,
-      :getbit,
-      :getrange,
-      :hexists,
-      :hget,
-      :hgetall,
-      :hkeys,
-      :hlen,
-      :hmget,
-      :hvals,
-      :keys,
-      :lindex,
-      :llen,
-      :lrange,
-      :mapped_hmget,
-      :mapped_mget,
-      :mget,
-      :scard,
-      :sdiff,
-      :sinter,
-      :sismember,
-      :smembers,
-      :srandmember,
-      :strlen,
-      :sunion,
-      :type,
-      :zcard,
-      :zcount,
-      :zrange,
-      :zrangebyscore,
-      :zrank,
-      :zrevrange,
-      :zrevrangebyscore,
-      :zrevrank,
-      :zscore
-    ].freeze
-
-    # Unsupported Redis operations. These don't make sense in a client
-    # that abstracts the master/slave servers.
-    UNSUPPORTED_OPS = Set[:select, :dbsize].freeze
-
     # Performance optimization: to avoid unnecessary method_missing calls,
     # we proactively define methods that dispatch to the underlying redis
     # calls.
@@ -93,25 +47,37 @@ module RedisFailover
     # @option options [Logger] :logger logger override
     # @option options [Boolean] :retry_failure indicates if failures are retried
     # @option options [Integer] :max_retries max retries for a failure
+    # @option options [Boolean] :safe_mode indicates if safe mode is used or not
+    # @option options [Boolean] :master_only indicates if only redis master is used
     # @return [RedisFailover::Client]
     def initialize(options = {})
       Util.logger = options[:logger] if options[:logger]
-      @zkservers = options.fetch(:zkservers) { raise ArgumentError, ':zkservers required'}
-      @znode = options[:znode_path] || Util::DEFAULT_ZNODE_PATH
-      @namespace = options[:namespace]
-      @password = options[:password]
-      @db = options[:db]
-      @retry = options[:retry_failure] || true
-      @max_retries = @retry ? options.fetch(:max_retries, 3) : 0
       @master = nil
       @slaves = []
       @node_addresses = {}
       @lock = Monitor.new
       @current_client_key = "current-client-#{self.object_id}"
+      yield self if block_given?
+
+      parse_options(options)
       setup_zk
       build_clients
     end
 
+    # Specifies a callback to invoke when the current redis node list changes.
+    #
+    # @param [Proc] a callback with current master and slaves as arguments
+    #
+    # @example Usage
+    #   RedisFailover::Client.new(:zkservers => zk_servers) do |client|
+    #     client.on_node_change do |master, slaves|
+    #       logger.info("Nodes changed! master: #{master}, slaves: #{slaves}")
+    #     end
+    #   end
+    def on_node_change(&callback)
+      @on_node_change = callback
+    end
+
     # Dispatches redis operations to master/slaves.
     def method_missing(method, *args, &block)
       if redis_operation?(method)
@@ -168,13 +134,31 @@ module RedisFailover
       build_clients
     end
 
+    # Retrieves the current redis master.
+    #
+    # @return [String] the host/port of the current master
+    def current_master
+      master = @lock.synchronize { @master }
+      address_for(master)
+    end
+
+    # Retrieves the current redis slaves.
+    #
+    # @return [Array<String>] an array of known slave host/port addresses
+    def current_slaves
+      slaves = @lock.synchronize { @slaves }
+      addresses_for(slaves)
+    end
+
     private
 
     # Sets up the underlying ZooKeeper connection.
     def setup_zk
       @zk = ZK.new(@zkservers)
       @zk.watcher.register(@znode) { |event| handle_zk_event(event) }
-      @zk.on_expired_session { purge_clients }
+      if @safe_mode
+        @zk.on_expired_session { purge_clients }
+      end
       @zk.on_connected { @zk.stat(@znode, :watch => true) }
       @zk.stat(@znode, :watch => true)
       update_znode_timestamp
@@ -210,7 +194,7 @@ module RedisFailover
     # @param [Proc] block an optional block to pass to the method
     # @return [Object] the result of dispatching the command
     def dispatch(method, *args, &block)
-      unless recently_heard_from_node_manager?
+      if @safe_mode && !recently_heard_from_node_manager?
         build_clients
       end
 
@@ -277,10 +261,26 @@ module RedisFailover
         rescue
           purge_clients
           raise
+        ensure
+          if should_notify?
+           @on_node_change.call(current_master, current_slaves)
+           @last_notified_master = current_master
+           @last_notified_slaves = current_slaves
+          end
         end
       end
     end
 
+    # Determines if the on_node_change callback should be invoked.
+    #
+    # @return [Boolean] true if callback should be invoked, false otherwise
+    def should_notify?
+      return false unless @on_node_change
+      return true if @last_notified_master != current_master
+      return true if different?(Array(@last_notified_slaves), current_slaves)
+      false
+    end
+
     # Fetches the known redis nodes from ZooKeeper.
     #
     # @return [Hash] the known master/slave redis servers
@@ -424,7 +424,16 @@ module RedisFailover
     #   nested blocks (e.g., block passed to multi).
     def client_for(method)
       stack = Thread.current[@current_client_key] ||= []
-      client = stack.last || (REDIS_READ_OPS.include?(method) ? slave : master)
+      client = if stack.last
+        stack.last
+      elsif @master_only
+        master
+      elsif REDIS_READ_OPS.include?(method)
+        slave
+      else
+        master
+      end
+
       stack << client
       client
     end
@@ -436,5 +445,20 @@ module RedisFailover
       end
       nil
     end
+
+    # Parses the configuration operations.
+    #
+    # @param [Hash] options the configuration options
+    def parse_options(options)
+      @zkservers = options.fetch(:zkservers) { raise ArgumentError, ':zkservers required'}
+      @znode = options.fetch(:znode_path, Util::DEFAULT_ZNODE_PATH)
+      @namespace = options[:namespace]
+      @password = options[:password]
+      @db = options[:db]
+      @retry = options.fetch(:retry_failure, true)
+      @max_retries = @retry ? options.fetch(:max_retries, 3) : 0
+      @safe_mode = options.fetch(:safe_mode, true)
+      @master_only = options.fetch(:master_only, false)
+    end
   end
 end

data/lib/redis_failover/errors.rb

@@ -44,4 +44,8 @@ module RedisFailover
       super("Operation `#{operation}` is currently unsupported")
     end
   end
+
+  # Raised when we detect an expired ZK session.
+  class ZKDisconnectedError < Error
+  end
 end

data/lib/redis_failover/node_manager.rb

@@ -9,14 +9,6 @@ module RedisFailover
   class NodeManager
     include Util
 
-    # Name for the znode that handles exclusive locking between multiple
-    # Node Manager processes. Whoever holds the lock will be considered
-    # the "master" Node Manager, and will be responsible for monitoring
-    # the redis nodes. When a Node Manager that holds the lock disappears
-    # or fails, another Node Manager process will grab the lock and
-    # become the master.
-    LOCK_PATH = 'master_node_manager'
-
     # Number of seconds to wait before retrying bootstrap process.
     TIMEOUT = 5
 
@@ -34,6 +26,14 @@ module RedisFailover
       @znode = @options[:znode_path] || Util::DEFAULT_ZNODE_PATH
       @manual_znode = ManualFailover::ZNODE_PATH
       @mutex = Mutex.new
+
+      # Name for the znode that handles exclusive locking between multiple
+      # Node Manager processes. Whoever holds the lock will be considered
+      # the "master" Node Manager, and will be responsible for monitoring
+      # the redis nodes. When a Node Manager that holds the lock disappears
+      # or fails, another Node Manager process will grab the lock and
+      # become the
+      @lock_path = "#{@znode}_lock".freeze
     end
 
     # Starts the node manager.
@@ -44,7 +44,7 @@ module RedisFailover
       @leader = false
       setup_zk
       logger.info('Waiting to become master Node Manager ...')
-      @zk.with_lock(LOCK_PATH) do
+      @zk.with_lock(@lock_path) do
         @leader = true
         logger.info('Acquired master Node Manager lock')
         discover_nodes
@@ -52,7 +52,7 @@ module RedisFailover
         spawn_watchers
         handle_state_reports
       end
-    rescue ZK::Exceptions::InterruptedSession => ex
+    rescue ZK::Exceptions::InterruptedSession, ZKDisconnectedError => ex
       logger.error("ZK error while attempting to manage nodes: #{ex.inspect}")
       logger.error(ex.backtrace.join("\n"))
       shutdown
@@ -83,6 +83,7 @@ module RedisFailover
     def setup_zk
       @zk.close! if @zk
       @zk = ZK.new("#{@options[:zkservers]}#{@options[:chroot] || ''}")
+      @zk.on_expired_session { notify_state(:zk_disconnected, nil) }
 
       @zk.register(@manual_znode) do |event|
         @mutex.synchronize do
@@ -106,12 +107,13 @@ module RedisFailover
           when :available       then handle_available(node)
           when :syncing         then handle_syncing(node)
           when :manual_failover then handle_manual_failover(node)
+          when :zk_disconnected then raise ZKDisconnectedError
           else raise InvalidNodeStateError.new(node, state)
           end
 
           # flush current state
           write_state
-        rescue ZK::Exceptions::InterruptedSession
+        rescue ZK::Exceptions::InterruptedSession, ZKDisconnectedError
           # fail hard if this is a ZK connection-related error
           raise
         rescue => ex
@@ -219,19 +221,19 @@ module RedisFailover
     def discover_nodes
       @unavailable = []
       nodes = @options[:nodes].map { |opts| Node.new(opts) }.uniq
-      raise NoMasterError unless @master = find_master(nodes)
+      @master = find_master(nodes)
       @slaves = nodes - [@master]
       logger.info("Managing master (#{@master}) and slaves" +
         " (#{@slaves.map(&:to_s).join(', ')})")
 
       # ensure that slaves are correctly pointing to this master
-      redirect_slaves_to(@master)
+      redirect_slaves_to(@master) if @master
     end
 
     # Spawns the {RedisFailover::NodeWatcher} instances for each managed node.
     def spawn_watchers
       @watchers = [@master, @slaves, @unavailable].flatten.map do |node|
-        NodeWatcher.new(self, node,  @options[:max_failures] || 3)
+        NodeWatcher.new(self, node, @options[:max_failures] || 3)
       end
       @watchers.each(&:watch)
     end
@@ -314,8 +316,10 @@ module RedisFailover
 
     # Creates the znode path containing the redis nodes.
     def create_path
-      @zk.create(@znode, encode(current_nodes), :ephemeral => true)
-      logger.info("Created ZooKeeper node #{@znode}")
+      unless @zk.exists?(@znode)
+        @zk.create(@znode, encode(current_nodes))
+        logger.info("Created ZooKeeper node #{@znode}")
+      end
     rescue ZK::Exceptions::NodeExists
       # best effort
     end

data/lib/redis_failover/runner.rb

@@ -10,8 +10,8 @@ module RedisFailover
       options = CLI.parse(options)
       @node_manager = NodeManager.new(options)
       trap_signals
-      node_manager_thread = Thread.new { @node_manager.start }
-      node_manager_thread.join
+      @node_manager_thread = Thread.new { @node_manager.start }
+      @node_manager_thread.join
     end
 
     # Traps shutdown signals.
@@ -20,6 +20,7 @@ module RedisFailover
         trap(signal) do
           Util.logger.info('Shutting down ...')
           @node_manager.shutdown
+          @node_manager_thread.join
           exit(0)
         end
       end

data/lib/redis_failover/util.rb

@@ -5,6 +5,52 @@ module RedisFailover
   module Util
     extend self
 
+    # Redis read operations that are automatically dispatched to slaves. Any
+    # operation not listed here will be dispatched to the master.
+    REDIS_READ_OPS = Set[
+      :echo,
+      :exists,
+      :get,
+      :getbit,
+      :getrange,
+      :hexists,
+      :hget,
+      :hgetall,
+      :hkeys,
+      :hlen,
+      :hmget,
+      :hvals,
+      :keys,
+      :lindex,
+      :llen,
+      :lrange,
+      :mapped_hmget,
+      :mapped_mget,
+      :mget,
+      :scard,
+      :sdiff,
+      :sinter,
+      :sismember,
+      :smembers,
+      :srandmember,
+      :strlen,
+      :sunion,
+      :type,
+      :zcard,
+      :zcount,
+      :zrange,
+      :zrangebyscore,
+      :zrank,
+      :zrevrange,
+      :zrevrangebyscore,
+      :zrevrank,
+      :zscore
+    ].freeze
+
+    # Unsupported Redis operations. These don't make sense in a client
+    # that abstracts the master/slave servers.
+    UNSUPPORTED_OPS = Set[:select, :dbsize].freeze
+
     # Default node in ZK that contains the current list of available redis nodes.
     DEFAULT_ZNODE_PATH = '/redis_failover_nodes'.freeze
 
@@ -12,7 +58,7 @@ module RedisFailover
     REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }
 
     # Connectivity errors that the redis (>3.x) client raises.
-    REDIS_ERRORS << Redis::BaseError if Redis.const_defined?("BaseError")
+    REDIS_ERRORS << Redis::BaseError if Redis.const_defined?('BaseError')
     REDIS_ERRORS.freeze
 
     # Full set of errors related to connectivity.

data/lib/redis_failover/version.rb

@@ -1,3 +1,3 @@
 module RedisFailover
-  VERSION = '0.8.9'
+  VERSION = '0.9.0'
 end

data/redis_failover.gemspec

@@ -4,8 +4,8 @@ require File.expand_path('../lib/redis_failover/version', __FILE__)
 Gem::Specification.new do |gem|
   gem.authors       = ["Ryan LeCompte"]
   gem.email         = ["lecompte@gmail.com"]
-  gem.description   = %(Redis Failover is a ZooKeeper-based automatic master/slave failover solution for Ruby)
-  gem.summary       = %(Redis Failover is a ZooKeeper-based automatic master/slave failover solution for Ruby)
+  gem.description   = %(redis_failover is a ZooKeeper-based automatic master/slave failover solution for Ruby)
+  gem.summary       = %(redis_failover is a ZooKeeper-based automatic master/slave failover solution for Ruby)
   gem.homepage      = "http://github.com/ryanlecompte/redis_failover"
 
   gem.executables   = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
@@ -15,7 +15,7 @@ Gem::Specification.new do |gem|
   gem.require_paths = ["lib"]
   gem.version       = RedisFailover::VERSION
 
-  gem.add_dependency('redis', '~> 3')
+  gem.add_dependency('redis', ['>= 2.2', '< 4'])
   gem.add_dependency('redis-namespace')
   gem.add_dependency('multi_json', '~> 1')
   gem.add_dependency('zk', '~> 1.6')

data/spec/client_spec.rb

@@ -26,7 +26,7 @@ module RedisFailover
   end
 
   describe Client do
-    let(:client) { ClientStub.new(:zkservers => 'localhost:9281') }
+    let(:client) { ClientStub.new(:zkservers => 'localhost:9281', :safe_mode => true) }
 
     describe '#build_clients' do
       it 'properly parses master' do
@@ -48,14 +48,28 @@ module RedisFailover
         called.should be_true
       end
 
-      it 'routes read operations to a slave' do
-        called = false
-        client.current_slaves.first.change_role_to('slave')
-        client.current_slaves.first.define_singleton_method(:get) do |*args|
-          called = true
+      context 'with :master_only false' do
+        it 'routes read operations to a slave' do
+          called = false
+          client.current_slaves.first.change_role_to('slave')
+          client.current_slaves.first.define_singleton_method(:get) do |*args|
+            called = true
+          end
+          client.get('foo')
+          called.should be_true
+        end
+      end
+
+      context 'with :master_only true' do
+        it 'routes read operations to master' do
+          client = ClientStub.new(:zkservers => 'localhost:9281', :master_only => true)
+          called = false
+          client.current_master.define_singleton_method(:get) do |*args|
+            called = true
+          end
+          client.get('foo')
+          called.should be_true
         end
-        client.get('foo')
-        called.should be_true
       end
 
       it 'reconnects when node is unavailable' do
@@ -90,10 +104,21 @@ module RedisFailover
         expect { client.select }.to raise_error(UnsupportedOperationError)
       end
 
-      it 'attempts ZK reconnect when no communication from Node Manager within certain time window' do
-        client.instance_variable_set(:@last_znode_timestamp, Time.at(0))
-        client.should_receive(:build_clients)
-        client.del('foo')
+      context 'with :safe_mode enabled' do
+        it 'rebuilds clients when no communication from Node Manager within certain time window' do
+          client.instance_variable_set(:@last_znode_timestamp, Time.at(0))
+          client.should_receive(:build_clients)
+          client.del('foo')
+        end
+      end
+
+      context 'with :safe_mode disabled' do
+        it 'does not rebuild clients when no communication from Node Manager within certain time window' do
+          client = ClientStub.new(:zkservers => 'localhost:9281', :safe_mode => false)
+          client.instance_variable_set(:@last_znode_timestamp, Time.at(0))
+          client.should_not_receive(:build_clients)
+          client.del('foo')
+        end
       end
     end
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redis_failover
 version: !ruby/object:Gem::Version
-  version: 0.8.9
+  version: 0.9.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,24 +9,30 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-07-17 00:00:00.000000000 Z
+date: 2012-08-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ~>
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '2.2'
+    - - <
       - !ruby/object:Gem::Version
-        version: '3'
+        version: '4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ~>
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '2.2'
+    - - <
       - !ruby/object:Gem::Version
-        version: '3'
+        version: '4'
 - !ruby/object:Gem::Dependency
   name: redis-namespace
   requirement: !ruby/object:Gem::Requirement
@@ -123,7 +129,7 @@ dependencies:
     - - ! '>='
       - !ruby/object:Gem::Version
         version: '0'
-description: Redis Failover is a ZooKeeper-based automatic master/slave failover solution
+description: redis_failover is a ZooKeeper-based automatic master/slave failover solution
   for Ruby
 email:
 - lecompte@gmail.com
@@ -183,7 +189,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1059043847699226647
+      hash: 827412647750283458
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -192,13 +198,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1059043847699226647
+      hash: 827412647750283458
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
-summary: Redis Failover is a ZooKeeper-based automatic master/slave failover solution
+summary: redis_failover is a ZooKeeper-based automatic master/slave failover solution
   for Ruby
 test_files:
 - spec/cli_spec.rb