redis_failover 0.9.7.2 → 1.0.0

data/Changes.md

@@ -1,3 +1,17 @@
+1.0.0
+-----------
+** NOTE: This version of redis_failover requires that you upgrade your clients and Node Managers at the same time.
+
+- redis_failover now supports distributed monitoring among the Node Managers! Previously, the Node Managers were only used
+as a means of redundancy in case a particular node manager crashed. Starting with version 1.0 of redis_failover, the Node
+Managers will all periodically report their health report/snapshots. The primary Node Manager will utilize a configurable
+"node strategy" to determine if a particular node is available or unavailable.
+- redis_failover now supports a configurable "failover strategy" that's consulted when performing a failover. Currently,
+a single strategy is provided that takes into account the average latency of the last health check to the redis server.
+- Improved handling of underlying ZK client connection in RedisFailover::NodeManager
+- Add support for passing in an existing ZK client instance to RedisFailover::Cient.new
+- Reduce unnecessary writes to ZK
+
 0.9.7.2
 -----------
 - Add support for Redis#client's location method. Fixes a compatibility issue with redis_failover and Sidekiq.

data/README.md

@@ -2,7 +2,7 @@
 
 [![Build Status](https://secure.travis-ci.org/ryanlecompte/redis_failover.png?branch=master)](http://travis-ci.org/ryanlecompte/redis_failover)
 
-redis_failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not provide
+redis_failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not currently provide
 an automatic failover capability when configured for master/slave replication. When the master node dies,
 a new master must be manually brought online and assigned as the slave's new master. This manual
 switch-over is not desirable in high traffic sites where Redis is a critical part of the overall
@@ -10,8 +10,8 @@ architecture. The existing standard Redis client for Ruby also only supports con
 Redis server. When using master/slave replication, it is desirable to have all writes go to the
 master, and all reads go to one of the N configured slaves.
 
-This gem (built using [ZK][]) attempts to address these failover scenarios. A redis failover Node Manager daemon runs as a background
-process and monitors all of your configured master/slave nodes. When the daemon starts up, it
+This gem (built using [ZK][]) attempts to address these failover scenarios. One or more Node Manager daemons run as background
+processes and monitor all of your configured master/slave nodes. When the daemon starts up, it
 automatically discovers the current master/slaves. Background watchers are setup for each of
 the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unavailable" state.
 If the node that went offline was the master, then one of the slaves will be promoted as the new master.
@@ -22,8 +22,10 @@ nodes. Note that detection of a node going down should be nearly instantaneous,
 used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
 immediately when the node actually goes offline. To avoid false positives (i.e., intermittent flaky
 network interruption), the Node Manager will only mark a node as unavailable if it fails to communicate with
-it 3 times (this is configurable via --max-failures, see configuration options below). Note that you can
-deploy multiple Node Manager daemons for added redundancy.
+it 3 times (this is configurable via --max-failures, see configuration options below). Note that you can (and should)
+deploy multiple Node Manager daemons since they each report periodic health reports/snapshots of the redis servers. A
+"node strategy" is used to determine if a node is actually unavailable. By default a majority strategy is used, but
+you can also configure "consensus" or "single" as well.
 
 This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
 with a list of ZooKeeper servers. The client will automatically contact the ZooKeeper cluster to find out
@@ -64,15 +66,20 @@ following options:
 
     Usage: redis_node_manager [OPTIONS]
 
+
     Specific options:
-        -n, --nodes NODES                Comma-separated redis host:port pairs
-        -z, --zkservers SERVERS          Comma-separated ZooKeeper host:port pairs
-        -p, --password PASSWORD          Redis password
-            --znode-path PATH            Znode path override for storing redis server list
-            --max-failures COUNT         Max failures before manager marks node unavailable
-        -C, --config PATH                Path to YAML configuration file
-        -E, --environment ENV            Config environment to use
-        -h, --help                       Display all options
+        -n, --nodes NODES                  Comma-separated redis host:port pairs
+        -z, --zkservers SERVERS            Comma-separated ZooKeeper host:port pairs
+        -p, --password PASSWORD            Redis password
+            --znode-path PATH              Znode path override for storing redis server list
+            --max-failures COUNT           Max failures before manager marks node unavailable
+        -C, --config PATH                  Path to YAML config file
+            --with-chroot ROOT             Path to ZooKeepers chroot
+        -E, --environment ENV              Config environment to use
+            --node-strategy STRATEGY       Strategy used when determining availability of nodes (default: majority)
+            --failover-strategy STRATEGY   Strategy used when failing over to a new node (default: latency)
+            --required-node-managers COUNT Required Node Managers that must be reachable to determine node state (default: 1)
+        -h, --help                         Display all options
 
 To start the daemon for a simple master/slave configuration, use the following:
 
@@ -83,6 +90,9 @@ would look like the following:
 
     ---
     :max_failures: 2
+    :node_strategy: majority
+    :failover_strategy: latency
+    :required_node_managers: 2
     :nodes:
       - localhost:6379
       - localhost:1111
@@ -103,25 +113,35 @@ directory for configuration file samples.
 
 The Node Manager will automatically discover the master/slaves upon startup. Note that it is
 a good idea to run more than one instance of the Node Manager daemon in your environment. At
-any moment, a single Node Manager process will be designated to monitor the redis servers. If
+any moment, a single Node Manager process will be designated to manage the redis servers. If
 this Node Manager process dies or becomes partitioned from the network, another Node Manager
-will be promoted as the primary monitor of redis servers. You can run as many Node Manager
-processes as you'd like for added redundancy.
+will be promoted as the primary manager of redis servers. You can run as many Node Manager
+processes as you'd like. Every Node Manager periodically records health "snapshots" which the
+primary/master Node Manager consults when determining if it should officially mark a redis 
+server as unavailable. By default, a majority strategy is used. Also, when a failover
+happens, the primary Node Manager will consult the node snapshots to determine the best
+node to use as the new master.
 
 ## Client Usage
 
 The redis failover client must be used in conjunction with a running Node Manager daemon. The
 client supports various configuration options, however the only mandatory option is the list of
-ZooKeeper servers:
+ZooKeeper servers OR an existing ZK client instance:
 
+    # Explicitly specify the ZK servers
     client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
 
+    # Explicitly specify an existing ZK client instance (useful if using a connection pool, etc)
+    zk = ZK.new('localhost:2181,localhost:2182,localhost:2183')
+    client = RedisFailover::Client.new(:zk => zk)
+
 The client actually employs the common redis and redis-namespace gems underneath, so this should be
 a drop-in replacement for your existing pure redis client usage.
 
 The full set of options that can be passed to RedisFailover::Client are:
 
-     :zkservers     - comma-separated ZooKeeper host:port pairs (required)
+     :zk            - an existing ZK client instance
+     :zkservers     - comma-separated ZooKeeper host:port pairs
      :znode_path    - the Znode path override for redis server list (optional)
      :password      - password for redis nodes (optional)
      :db            - db to use for redis nodes (optional)
@@ -149,6 +169,25 @@ server passed to #manual_failover, or it will pick a random slave to become the
     client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
     client.manual_failover(:host => 'localhost', :port => 2222)
 
+## Node & Failover Strategies
+
+As of redis_failover version 1.0, the notion of "node" and "failover" strategies exists. All running Node Managers will periodically record
+"snapshots" of their view of the redis nodes. The primary Node Manager will process these snapshots from all of the Node Managers by running a configurable
+node strategy. By default, a majority strategy is used. This means that if a majority of Node Managers indicate that a node is unavailable, then the primary
+Node Manager will officially mark it as unavailable. Other strategies exist:
+
+- consensus (all Node Managers must agree that the node is unavailable)
+- single (at least one Node Manager saying the node is unavailable will cause the node to be marked as such)
+
+When a failover happens, the primary Node Manager will now consult a "failover strategy" to determine which candidate node should be used. Currently only a single
+strategy is provided by redis_failover: latency. This strategy simply selects a node that is both marked as available by all Node Managers and has the lowest
+average latency for its last health check.
+
+Note that you should set the "required_node_managers" configuration option appropriately. This value (defaults to 1) is used to determine if enough Node
+Managers have reported their view of a node's state. For example, if you have deployed 5 Node Managers, then you should set this value to 5 if you only
+want to accept a node's availability when all 5 Node Managers are part of the snapshot. To give yourself flexibility, you may want to set this value to 3
+instead. This would give you flexibility to take down 2 Node Managers, while still allowing the cluster to be managed appropriately.
+
 ## Documentation
 
 redis_failover uses YARD for its API documentation. Refer to the generated [API documentation](http://rubydoc.info/github/ryanlecompte/redis_failover/master/frames) for full coverage.
@@ -166,8 +205,6 @@ redis_failover uses YARD for its API documentation. Refer to the generated [API
 
 - Note that it's still possible for the RedisFailover::Client instances to see a stale list of servers for a very small window. In most cases this will not be the case due to how ZooKeeper handles distributed communication, but you should be aware that in the worst case the client could write to a "stale" master for a small period of time until the next watch event is received by the client via ZooKeeper.
 
-- Note that currently multiple Node Managers are currently used for redundancy purposes only. The Node Managers do not communicate with each other to perform any type of election or voting to determine if they all agree on promoting a new master. Right now Node Managers that are not "active" just sit and wait until they can grab the lock to become the single decision-maker for which Redis servers are available or not. This means that a scenario could present itself where a Node Manager thinks the Redis master is available, however the actual RedisFailover::Client instances think they can't reach the Redis master (either due to network partitions or the Node Manager flapping due to machine failure, etc). We are exploring ways to improve this situation.
-
 ## Resources
 
 - Check out Steve Whittaker's [redis-failover-test](https://github.com/swhitt/redis-failover-test) project which shows how to test redis_failover in a non-trivial configuration using Vagrant/Chef.

data/examples/config.yml

@@ -2,6 +2,9 @@
 #   redis_node_manager -C config.yml
 ---
 :max_failures: 2
+:node_strategy: majority
+:failover_strategy: latency
+:required_node_managers: 2
 :nodes:
   - localhost:6379
   - localhost:1111

data/lib/redis_failover.rb

@@ -6,6 +6,7 @@ require 'thread'
 require 'logger'
 require 'timeout'
 require 'optparse'
+require 'benchmark'
 require 'multi_json'
 require 'securerandom'
 
@@ -16,7 +17,9 @@ require 'redis_failover/errors'
 require 'redis_failover/client'
 require 'redis_failover/runner'
 require 'redis_failover/version'
+require 'redis_failover/node_strategy'
 require 'redis_failover/node_manager'
 require 'redis_failover/node_watcher'
+require 'redis_failover/node_snapshot'
 require 'redis_failover/manual_failover'
-
+require 'redis_failover/failover_strategy'

data/lib/redis_failover/cli.rb

@@ -48,6 +48,21 @@ module RedisFailover
           options[:config_environment] = config_env
         end
 
+        opts.on('--node-strategy STRATEGY',
+         'Strategy used when determining availability of nodes (default: majority)') do |strategy|
+          options[:node_strategy] = strategy
+        end
+
+        opts.on('--failover-strategy STRATEGY',
+         'Strategy used when failing over to a new node (default: latency)') do |strategy|
+          options[:failover_strategy] = strategy
+        end
+
+        opts.on('--required-node-managers COUNT',
+         'Required Node Managers that must be reachable to determine node state (default: 1)') do |count|
+          options[:required_node_managers] = Integer(count)
+        end
+
         opts.on('-h', '--help', 'Display all options') do
           puts opts
           exit
@@ -59,7 +74,7 @@ module RedisFailover
         options = from_file(config_file, options[:config_environment])
       end
 
-      if required_options_missing?(options)
+      if invalid_options?(options)
         puts parser
         exit
       end
@@ -68,7 +83,7 @@ module RedisFailover
     end
 
     # @return [Boolean] true if required options missing, false otherwise
-    def self.required_options_missing?(options)
+    def self.invalid_options?(options)
       return true if options.empty?
       return true unless options.values_at(:nodes, :zkservers).all?
       false
@@ -113,6 +128,14 @@ module RedisFailover
         options[:nodes].each { |opts| opts.update(:password => password) }
       end
 
+      if node_strategy = options[:node_strategy]
+        options[:node_strategy] = node_strategy.to_sym
+      end
+
+      if failover_strategy = options[:failover_strategy]
+        options[:failover_strategy] = failover_strategy.to_sym
+      end
+
       options
     end
   end

data/lib/redis_failover/client.rb

@@ -40,6 +40,7 @@ module RedisFailover
     #
     # @param [Hash] options the options used to initialize the client instance
     # @option options [String] :zkservers comma-separated ZooKeeper host:port
+    # @option options [String] :zk an existing ZK client connection instance
     # @option options [String] :znode_path znode path override for redis nodes
     # @option options [String] :password password for redis nodes
     # @option options [String] :db database to use for redis nodes
@@ -49,6 +50,7 @@ module RedisFailover
     # @option options [Integer] :max_retries max retries for a failure
     # @option options [Boolean] :safe_mode indicates if safe mode is used or not
     # @option options [Boolean] :master_only indicates if only redis master is used
+    # @note Use either :zkservers or :zk
     # @return [RedisFailover::Client]
     def initialize(options = {})
       Util.logger = options[:logger] if options[:logger]
@@ -133,7 +135,7 @@ module RedisFailover
     # @option options [String] :host the host of the failover candidate
     # @option options [String] :port the port of the failover candidate
     def manual_failover(options = {})
-      ManualFailover.new(@zk, options).perform
+      ManualFailover.new(@zk, @root_znode, options).perform
       self
     end
 
@@ -177,13 +179,13 @@ module RedisFailover
 
     # Sets up the underlying ZooKeeper connection.
     def setup_zk
-      @zk = ZK.new(@zkservers)
-      @zk.watcher.register(@znode) { |event| handle_zk_event(event) }
+      @zk = ZK.new(@zkservers) if @zkservers
+      @zk.register(redis_nodes_path) { |event| handle_zk_event(event) }
       if @safe_mode
         @zk.on_expired_session { purge_clients }
       end
-      @zk.on_connected { @zk.stat(@znode, :watch => true) }
-      @zk.stat(@znode, :watch => true)
+      @zk.on_connected { @zk.stat(redis_nodes_path, :watch => true) }
+      @zk.stat(redis_nodes_path, :watch => true)
       update_znode_timestamp
     end
 
@@ -196,12 +198,12 @@ module RedisFailover
         build_clients
       elsif event.node_deleted?
         purge_clients
-        @zk.stat(@znode, :watch => true)
+        @zk.stat(redis_nodes_path, :watch => true)
       else
         logger.error("Unknown ZK node event: #{event.inspect}")
       end
     ensure
-      @zk.stat(@znode, :watch => true)
+      @zk.stat(redis_nodes_path, :watch => true)
     end
 
     # Determines if a method is a known redis operation.
@@ -310,7 +312,7 @@ module RedisFailover
     #
     # @return [Hash] the known master/slave redis servers
     def fetch_nodes
-      data = @zk.get(@znode, :watch => true).first
+      data = @zk.get(redis_nodes_path, :watch => true).first
       nodes = symbolize_keys(decode(data))
       logger.debug("Fetched nodes: #{nodes.inspect}")
 
@@ -319,6 +321,10 @@ module RedisFailover
       logger.debug { "Caught #{ex.class} '#{ex.message}' - reopening ZK client" }
       @zk.reopen
       retry
+    rescue *ZK_ERRORS => ex
+      logger.warn { "Caught #{ex.class} '#{ex.message}' - retrying" }
+      sleep(RETRY_WAIT_TIME)
+      retry
     end
 
     # Builds new Redis clients for the specified nodes.
@@ -475,8 +481,12 @@ module RedisFailover
     #
     # @param [Hash] options the configuration options
     def parse_options(options)
-      @zkservers = options.fetch(:zkservers) { raise ArgumentError, ':zkservers required'}
-      @znode = options.fetch(:znode_path, Util::DEFAULT_ZNODE_PATH)
+      @zk, @zkservers = options.values_at(:zk, :zkservers)
+      if [@zk, @zkservers].all? || [@zk, @zkservers].none?
+        raise ArgumentError, 'must specify :zk or :zkservers'
+      end
+
+      @root_znode = options.fetch(:znode_path, Util::DEFAULT_ROOT_ZNODE_PATH)
       @namespace = options[:namespace]
       @password = options[:password]
       @db = options[:db]
@@ -485,5 +495,10 @@ module RedisFailover
       @safe_mode = options.fetch(:safe_mode, true)
       @master_only = options.fetch(:master_only, false)
     end
+
+    # @return [String] the znode path for the master redis nodes config
+    def redis_nodes_path
+      "#{@root_znode}/nodes"
+    end
   end
 end

data/lib/redis_failover/errors.rb

@@ -51,8 +51,4 @@ module RedisFailover
       super("Operation `#{operation}` is currently unsupported")
     end
   end
-
-  # Raised when we detect an expired ZK session.
-  class ZKDisconnectedError < Error
-  end
 end

data/lib/redis_failover/failover_strategy.rb

@@ -0,0 +1,25 @@
+module RedisFailover
+  # Base class for strategies that determine which node is used during failover.
+  class FailoverStrategy
+    include Util
+
+    # Loads a strategy based on the given name.
+    #
+    # @param [String, Symbol] name the strategy name
+    # @return [Object] a new strategy instance
+    def self.for(name)
+      require "redis_failover/failover_strategy/#{name.downcase}"
+      const_get(name.capitalize).new
+    rescue LoadError, NameError
+      raise "Failed to find failover strategy: #{name}"
+    end
+
+    # Returns a candidate node as determined by this strategy.
+    #
+    # @param [Hash<Node, NodeSnapshot>] snapshots the node snapshots
+    # @return [Node] the candidate node or nil if one couldn't be found
+    def find_candidate(snapshots)
+      raise NotImplementedError
+    end
+  end
+end

data/lib/redis_failover/failover_strategy/latency.rb

@@ -0,0 +1,21 @@
+module RedisFailover
+  class FailoverStrategy
+    # Failover strategy that selects an available node that is both seen by all
+    # node managers and has the lowest reported health check latency.
+    class Latency < FailoverStrategy
+      # @see RedisFailover::FailoverStrategy#find_candidate
+      def find_candidate(snapshots)
+        candidates = {}
+        snapshots.each do |node, snapshot|
+          if snapshot.all_available?
+            candidates[node] = snapshot.avg_latency
+          end
+        end
+
+        if candidate = candidates.min_by(&:last)
+          candidate.first
+        end
+      end
+    end
+  end
+end

data/lib/redis_failover/manual_failover.rb

@@ -2,37 +2,49 @@ module RedisFailover
   # Provides manual failover support to a new master.
   class ManualFailover
     # Path for manual failover communication.
-    ZNODE_PATH = '/redis_failover_manual'.freeze
+    ZNODE_PATH = 'manual_failover'.freeze
 
     # Denotes that any slave can be used as a candidate for promotion.
     ANY_SLAVE = "ANY_SLAVE".freeze
 
+    def self.path(root_znode)
+      "#{root_znode}/#{ZNODE_PATH}"
+    end
+
     # Creates a new instance.
     #
     # @param [ZK] zk the ZooKeeper client
+    # @param [ZNode] root_znode the root ZK node
     # @param [Hash] options the options used for manual failover
     # @option options [String] :host the host of the failover candidate
     # @option options [String] :port the port of the failover candidate
     # @note
     #   If options is empty, a random slave will be used
     #   as a failover candidate.
-    def initialize(zk, options = {})
+    def initialize(zk, root_znode, options = {})
       @zk = zk
+      @root_znode = root_znode
       @options = options
+
+      unless @options.empty?
+        port = Integer(@options[:port]) rescue nil
+        raise ArgumentError, ':host not properly specified' if @options[:host].to_s.empty?
+        raise ArgumentError, ':port not properly specified' if port.nil?
+      end
     end
 
     # Performs a manual failover.
     def perform
       create_path
       node = @options.empty? ? ANY_SLAVE : "#{@options[:host]}:#{@options[:port]}"
-      @zk.set(ZNODE_PATH, node)
+      @zk.set(self.class.path(@root_znode), node)
     end
 
     private
 
     # Creates the znode path used for coordinating manual failovers.
     def create_path
-      @zk.create(ZNODE_PATH)
+      @zk.create(self.class.path(@root_znode))
     rescue ZK::Exceptions::NodeExists
       # best effort
     end