redis_failover 0.7.0 → 0.8.0

data/Changes.md

@@ -1,3 +1,8 @@
+0.8.0
+-----------
+- Added manual failover support (can be initiated via RedisFailover::Client#manual_failover)
+- Misc. cleanup
+
 0.7.0
 -----------
 - When new master promotion occurs, make existing slaves point to new candidate before promoting new master.

data/README.md

@@ -10,7 +10,7 @@ architecture. The existing standard Redis client for Ruby also only supports con
 Redis server. When using master/slave replication, it is desirable to have all writes go to the
 master, and all reads go to one of the N configured slaves.
 
-This gem attempts to address these failover scenarios. A redis failover Node Manager daemon runs as a background
+This gem (built using [ZK][]) attempts to address these failover scenarios. A redis failover Node Manager daemon runs as a background
 process and monitors all of your configured master/slave nodes. When the daemon starts up, it
 automatically discovers the current master/slaves. Background watchers are setup for each of
 the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unavailable" state.
@@ -35,6 +35,8 @@ dispatch Redis read operations to one of N slaves, and Redis write operations to
 If it fails to communicate with any node, it will go back and fetch the current list of available servers, and then
 optionally retry the operation.
 
+[ZK]: https://github.com/slyphon/zk
+
 ## Installation
 
 redis_failover has an external dependency on ZooKeeper. You must have a running ZooKeeper cluster already available in order to use redis_failover. ZooKeeper provides redis_failover with its high availability and data consistency between Redis::Failover clients and the Node Manager daemon. Please see the requirements section below for more information on installing and setting up ZooKeeper if you don't have it running already.
@@ -120,6 +122,15 @@ The full set of options that can be passed to RedisFailover::Client are:
      :retry_failure - indicate if failures should be retried (default true)
      :max_retries   - max retries for a failure (default 3)
 
+## Manual Failover
+
+Manual failover can be initiated via RedisFailover::Client#manual_failover. This schedules a manual failover with the
+currently active Node Manager. Once the Node Manager receives the request, it will either failover to the specific
+server passed to #manual_failover, or it will pick a random slave to become the new master. Here's an example:
+
+    client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
+    client.manual_failover(:host => 'localhost', :port => 2222)
+
 ## Requirements
 
 - redis_failover is actively tested against MRI 1.9.2/1.9.3 and JRuby 1.6.7 (1.9 mode only). Other rubies may work, although I don't actively test against them. 1.8 is not supported.
@@ -129,8 +140,12 @@ The full set of options that can be passed to RedisFailover::Client are:
 
 - Note that by default the Node Manager will mark slaves that are currently syncing with their master as "available" based on the configuration value set for "slave-serve-stale-data" in redis.conf. By default this value is set to "yes" in the configuration, which means that slaves still syncing with their master will be available for servicing read requests. If you don't want this behavior, just set "slave-serve-stale-data" to "no" in your redis.conf file.
 
+## Limitations
+
 - Note that it's still possible for the RedisFailover::Client instances to see a stale list of servers for a very small window. In most cases this will not be the case due to how ZooKeeper handles distributed communication, but you should be aware that in the worst case the client could write to a "stale" master for a small period of time until the next watch event is received by the client via ZooKeeper.
 
+- Note that currently multiple Node Managers are currently used for redundancy purposes only. The Node Managers do not communicate with each other to perform any type of election or voting to determine if they all agree on promoting a new master. Right now Node Managers that are not "active" just sit and wait until they can grab the lock to become the single decision-maker for which Redis servers are available or not. This means that a scenario could present itself where a Node Manager thinks the Redis master is available, however the actual RedisFailover::Client instances think they can't reach the Redis master (either due to network partitions or the Node Manager flapping due to machine failure, etc). We are exploring ways to improve this situation.
+
 ## Resources
 
 - To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).

data/lib/redis_failover.rb

@@ -15,6 +15,7 @@ require 'redis_failover/node'
 require 'redis_failover/errors'
 require 'redis_failover/client'
 require 'redis_failover/runner'
+require 'redis_failover/manual'
 require 'redis_failover/version'
 require 'redis_failover/node_manager'
 require 'redis_failover/node_watcher'

data/lib/redis_failover/client.rb

@@ -87,7 +87,6 @@ module RedisFailover
     # Creates a new failover redis client.
     #
     # Options:
-    #
     #   :zkservers     - comma-separated ZooKeeper host:port pairs (required)
     #   :znode_path    - the Znode path override for redis server list (optional)
     #   :password      - password for redis nodes (optional)
@@ -96,7 +95,6 @@ module RedisFailover
     #   :logger        - logger override (optional)
     #   :retry_failure - indicate if failures should be retried (default true)
     #   :max_retries   - max retries for a failure (default 3)
-    #
     def initialize(options = {})
       Util.logger = options[:logger] if options[:logger]
       @zkservers = options.fetch(:zkservers) { raise ArgumentError, ':zkservers required'}
@@ -132,6 +130,18 @@ module RedisFailover
     end
     alias_method :to_s, :inspect
 
+    # Force a manual failover to a new server. A specific server can be specified
+    # via options. If no options are passed, a random slave will be selected as
+    # the candidate for the new master.
+    #
+    # Options:
+    #   :host - the host of the failover candidate
+    #   :port - the port of the failover candidate
+    def manual_failover(options = {})
+      Manual.failover(zk, options)
+      self
+    end
+
     private
 
     def zk
@@ -141,13 +151,15 @@ module RedisFailover
     def start_zk
       @delivery_thread ||= Thread.new do
         while event = @queue.pop
-          if event.is_a?(Proc)
-            event.call
-          else
-            handle_zk_event(event)
+          begin
+            Proc === event ? event.call : handle_zk_event(event)
+          rescue => ex
+            logger.error("Error while handling event: #{ex.inspect}")
+            logger.error(ex.backtrace.join("\n"))
           end
         end
       end
+
       reconnect_zk
     end
 
@@ -165,6 +177,7 @@ module RedisFailover
         @zk.on_connected do
           @zk.stat(@znode, :watch => true)
         end
+        @zk.stat(@znode, :watch => true)
       end
     end
 
@@ -181,13 +194,13 @@ module RedisFailover
     end
 
     def reconnect_zk
-      handle_lost_connection
       @lock.synchronize do
+        handle_lost_connection
         @zk.close! if @zk
         @zk = ZK.new(@zkservers)
+        handle_session_established
+        update_znode_timestamp
       end
-      handle_session_established
-      update_znode_timestamp
     end
 
     def handle_lost_connection
@@ -200,8 +213,10 @@ module RedisFailover
 
     def dispatch(method, *args, &block)
       unless recently_heard_from_node_manager?
-        purge_clients
-        raise MissingNodeManagerError.new(ZNODE_UPDATE_TIMEOUT)
+        @lock.synchronize do
+          reconnect_zk
+          build_clients
+        end
       end
 
       verify_supported!(method)

data/lib/redis_failover/errors.rb

@@ -40,10 +40,4 @@ module RedisFailover
       super("Operation `#{operation}` is currently unsupported")
     end
   end
-
-  class MissingNodeManagerError < Error
-    def initialize(timeout)
-      super("Failed to hear from Node Manager within #{timeout} seconds")
-    end
-  end
 end

data/lib/redis_failover/manual.rb

@@ -0,0 +1,26 @@
+module RedisFailover
+  # Provides manual failover support to a new master.
+  module Manual
+    extend self
+
+    # Path for manual failover communication.
+    ZNODE_PATH = '/redis_failover_manual'.freeze
+
+    # Denotes that any slave can be used as a candidate for promotion.
+    ANY_SLAVE = "ANY_SLAVE".freeze
+
+    def failover(zk, options = {})
+      create_path(zk)
+      node = options.empty? ? ANY_SLAVE : "#{options[:host]}:#{options[:port]}"
+      zk.set(ZNODE_PATH, node)
+    end
+
+    private
+
+    def create_path(zk)
+      zk.create(ZNODE_PATH)
+    rescue ZK::Exceptions::NodeExists
+      # best effort
+    end
+  end
+end

data/lib/redis_failover/node.rb

@@ -82,7 +82,7 @@ module RedisFailover
     end
 
     def ==(other)
-      return false unless other.is_a?(Node)
+      return false unless Node === other
       return true if self.equal?(other)
       [host, port] == [other.host, other.port]
     end
@@ -132,13 +132,13 @@ module RedisFailover
         yield redis
       end
     rescue
-      raise NodeUnavailableError.new(self)
+      raise NodeUnavailableError, self, caller
     ensure
       if redis
         begin
           redis.client.disconnect
         rescue
-          raise NodeUnavailableError.new(self)
+          raise NodeUnavailableError, self, caller
         end
       end
     end

data/lib/redis_failover/node_manager.rb

@@ -24,13 +24,17 @@ module RedisFailover
       logger.info("Redis Node Manager v#{VERSION} starting (#{RUBY_DESCRIPTION})")
       @options = options
       @znode = @options[:znode_path] || Util::DEFAULT_ZNODE_PATH
+      @manual_znode = Manual::ZNODE_PATH
+      @mutex = Mutex.new
     end
 
     def start
       @queue = Queue.new
-      @zk = ZK.new(@options[:zkservers])
+      @leader = false
+      setup_zk
       logger.info('Waiting to become master Node Manager ...')
       @zk.with_lock(LOCK_PATH) do
+        @leader = true
         logger.info('Acquired master Node Manager lock')
         discover_nodes
         initialize_path
@@ -50,6 +54,7 @@ module RedisFailover
     end
 
     def shutdown
+      @queue.clear
       @queue << nil
       @watchers.each(&:shutdown) if @watchers
       @zk.close! if @zk
@@ -57,14 +62,33 @@ module RedisFailover
 
     private
 
+    def setup_zk
+      @zk.close! if @zk
+      @zk = ZK.new(@options[:zkservers])
+
+      @zk.register(@manual_znode) do |event|
+        @mutex.synchronize do
+          if event.node_changed?
+            schedule_manual_failover
+          end
+        end
+      end
+
+      @zk.on_connected do
+        @zk.stat(@manual_znode, :watch => true)
+      end
+      @zk.stat(@manual_znode, :watch => true)
+    end
+
     def handle_state_reports
       while state_report = @queue.pop
         begin
           node, state = state_report
           case state
-          when :unavailable then handle_unavailable(node)
-          when :available   then handle_available(node)
-          when :syncing     then handle_syncing(node)
+          when :unavailable     then handle_unavailable(node)
+          when :available       then handle_available(node)
+          when :syncing         then handle_syncing(node)
+          when :manual_failover then handle_manual_failover(node)
           else raise InvalidNodeStateError.new(node, state)
           end
 
@@ -127,6 +151,17 @@ module RedisFailover
       handle_available(node)
     end
 
+    def handle_manual_failover(node)
+      # no-op if node to be failed over is already master
+      return if @master == node
+      logger.info("Handling manual failover")
+
+      # make current master a slave, and promote new master
+      @slaves << @master
+      @slaves.delete(node)
+      promote_new_master(node)
+    end
+
     def promote_new_master(node = nil)
       delete_path
       @master = nil
@@ -244,5 +279,19 @@ module RedisFailover
       create_path
       @zk.set(@znode, encode(current_nodes))
     end
+
+    def schedule_manual_failover
+      return unless @leader
+      new_master = @zk.get(@manual_znode, :watch => true).first
+      logger.info("Received manual failover request for: #{new_master}")
+
+      node = if new_master == Manual::ANY_SLAVE
+        @slaves.sample
+      else
+        host, port = new_master.split(':', 2)
+        Node.new(:host => host, :port => port, :password => @options[:password])
+      end
+      notify_state(node, :manual_failover) if node
+    end
   end
 end

data/lib/redis_failover/node_watcher.rb

@@ -17,7 +17,7 @@ module RedisFailover
     end
 
     def watch
-      @monitor_thread = Thread.new { monitor_node }
+      @monitor_thread ||= Thread.new { monitor_node }
       self
     end
 

data/lib/redis_failover/runner.rb

@@ -9,17 +9,10 @@ module RedisFailover
       node_manager_thread.join
     end
 
-    def self.node_manager
-      @node_manager
-    end
-
     def self.trap_signals
       [:INT, :TERM].each do |signal|
-        previous_signal = trap(signal) do
+        trap(signal) do
           Util.logger.info('Shutting down ...')
-          if previous_signal && previous_signal.respond_to?(:call)
-            previous_signal.call
-          end
           @node_manager.shutdown
           exit(0)
         end

data/lib/redis_failover/version.rb

@@ -1,3 +1,3 @@
 module RedisFailover
-  VERSION = "0.7.0"
+  VERSION = "0.8.0"
 end

data/redis_failover.gemspec

@@ -17,8 +17,8 @@ Gem::Specification.new do |gem|
 
   gem.add_dependency('redis')
   gem.add_dependency('redis-namespace')
-  gem.add_dependency('multi_json', '>= 1.0', '< 1.3')
-  gem.add_dependency('zk', '~> 0.9')
+  gem.add_dependency('multi_json', '~> 1')
+  gem.add_dependency('zk', '~> 1.0')
 
   gem.add_development_dependency('rake')
   gem.add_development_dependency('rspec')

data/spec/client_spec.rb

@@ -89,9 +89,10 @@ module RedisFailover
         expect { client.select }.to raise_error(UnsupportedOperationError)
       end
 
-      it 'raises error when no communication from Node Manager within certain time window' do
+      it 'attempts ZK reconnect when no communication from Node Manager within certain time window' do
         client.instance_variable_set(:@last_znode_timestamp, Time.at(0))
-        expect { client.del('foo') }.to raise_error(MissingNodeManagerError)
+        client.should_receive(:reconnect_zk)
+        client.del('foo')
       end
     end
   end

data/spec/support/node_manager_stub.rb

@@ -3,16 +3,6 @@ module RedisFailover
     attr_accessor :master
     public :current_nodes
 
-    def initialize(options)
-      super
-      @zklock = Object.new
-      @zklock.instance_eval do
-        def with_lock
-          yield
-        end
-      end
-    end
-
     def discover_nodes
       # only discover nodes once in testing
       return if @nodes_discovered
@@ -28,6 +18,10 @@ module RedisFailover
       @nodes_discovered = true
     end
 
+    def setup_zk
+      @zk = NullObject.new
+    end
+
     def slaves
       @slaves
     end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: redis_failover
 version: !ruby/object:Gem::Version
-  version: 0.7.0
+  version: 0.8.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-04-24 00:00:00.000000000 Z
+date: 2012-05-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: redis
-  requirement: &70345784320360 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -21,10 +21,15 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70345784320360
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: redis-namespace
-  requirement: &70345784319940 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -32,35 +37,47 @@ dependencies:
         version: '0'
   type: :runtime
   prerelease: false
-  version_requirements: *70345784319940
-- !ruby/object:Gem::Dependency
-  name: multi_json
-  requirement: &70345784319420 !ruby/object:Gem::Requirement
+  version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
       - !ruby/object:Gem::Version
-        version: '1.0'
-    - - <
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: multi_json
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
       - !ruby/object:Gem::Version
-        version: '1.3'
+        version: '1'
   type: :runtime
   prerelease: false
-  version_requirements: *70345784319420
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1'
 - !ruby/object:Gem::Dependency
   name: zk
-  requirement: &70345784318660 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: '0.9'
+        version: '1.0'
   type: :runtime
   prerelease: false
-  version_requirements: *70345784318660
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ~>
+      - !ruby/object:Gem::Version
+        version: '1.0'
 - !ruby/object:Gem::Dependency
   name: rake
-  requirement: &70345784318280 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -68,10 +85,15 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70345784318280
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: rspec
-  requirement: &70345784317820 !ruby/object:Gem::Requirement
+  requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - ! '>='
@@ -79,7 +101,12 @@ dependencies:
         version: '0'
   type: :development
   prerelease: false
-  version_requirements: *70345784317820
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Redis Failover is a ZooKeeper-based automatic master/slave failover solution
   for Ruby
 email:
@@ -102,6 +129,7 @@ files:
 - lib/redis_failover/cli.rb
 - lib/redis_failover/client.rb
 - lib/redis_failover/errors.rb
+- lib/redis_failover/manual.rb
 - lib/redis_failover/node.rb
 - lib/redis_failover/node_manager.rb
 - lib/redis_failover/node_watcher.rb
@@ -132,7 +160,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -4131531819785678189
+      hash: -3688012640966712276
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -141,10 +169,10 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -4131531819785678189
+      hash: -3688012640966712276
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.16
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Redis Failover is a ZooKeeper-based automatic master/slave failover solution