redis_failover 0.1.0

data/.gitignore

@@ -0,0 +1,17 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/Changes.md

@@ -0,0 +1,4 @@
+0.1.0
+-----------
+
+- First release

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Ryan LeCompte
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,106 @@
+# Redis Failover Client/Server
+
+Redis Failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not provide
+an automatic failover capability when configured for master/slave replication. When the master node dies,
+a new master must be manually brought online and assigned as the slave's new master. This manual
+switch-over is not desirable in high traffic sites where Redis is a critical part of the overall
+architecture. The existing standard Redis client for Ruby also only supports configuration for a single
+Redis server. When using master/slave replication, it is desirable to have all writes go to the
+master, and all reads go to one of the N configured slaves.
+
+This gem attempts to address both the server and client problems. A redis failover server runs as a background
+daemon and monitors all of your configured master/slave nodes. When the server starts up, it
+automatically discovers who is the master and who are the slaves. Watchers are setup for each of
+the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unreachable" state.
+If the node that went offline was the master, then one of the slaves will be promoted as the new master.
+All existing slaves will be automatically reconfigured to point to the new master for replication.
+All nodes marked as unreachable will be periodically checked to see if they have been brought back online.
+If so, the newly reachable nodes will be configured as slaves and brought back into the list of live
+servers. Note that detection of a node going down should be nearly instantaneous, since the mechanism
+used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
+immediately when the node actually goes offline.
+
+This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
+with a single host/port pair that points to redis failover server. The client will automatically
+connect to the server to find out the current state of the world (i.e., who's the current master and
+who are the current slaves). The client also acts as a load balancer in that it will automatically
+dispatch Redis read operations to one of N slaves, and Redis write operations to the master.
+If it fails to communicate with any node, it will go back and ask the server for the current list of
+available servers, and then optionally retry the operation.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+    gem 'redis_failover'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install redis_failover
+
+## Server Usage
+
+The redis failover server is a simple process that should be run as a background daemon. The server supports the
+following options:
+
+    Usage: redis_failover_server [OPTIONS]
+        -P, --port port                  Server port
+        -p, --password password          Redis password
+        -n, --nodes nodes                Comma-separated redis host:port pairs
+            --max-failures count         Max failures before server marks node unreachable (default 3)
+        -h, --help                       Display all options
+
+To start the server for a simple master/slave configuration, use the following:
+
+    redis_failover_server -P 3000 -n localhost:6379,localhost:6380
+
+The server will automatically figure out who is the master and who is the slave upon startup. Note that it is
+a good idea to monitor the redis failover server process with a tool like Monit to ensure that it is restarted
+in the case of a failure.
+
+## Client Usage
+
+The redis failover client must be used in conjunction with a running redis failover server. The
+client supports various configuration options, however the two mandatory options are the host
+and port of the redis failover server:
+
+    client = RedisFailover::Client.new(:host => 'localhost', :port => 3000)
+
+The client actually employs the common redis and redis-namespace gems underneath, so this should be
+a drop-in replacement for your existing pure redis client usage.
+
+The full set of options that can be passed to RedisFailover::Client are:
+
+     :host          - redis failover server host (required)
+     :port          - redis failover server port (required)
+     :password      - optional password for redis nodes
+     :namespace     - optional namespace for redis nodes
+     :logger        - optional logger override
+     :retry_failure - indicate if failures should be retried (default true)
+     :max_retries   - max retries for a failure (default 3)
+
+## Resources
+
+To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).
+
+## License
+
+Please see LICENSE for licensing details.
+
+## Author
+
+Ryan LeCompte
+
+[@ryanlecompte](https://twitter.com/ryanlecompte)
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = %w(--format progress)
+end
+
+task :default => [:spec]

data/bin/redis_failover_server

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require 'redis_failover'
+
+RedisFailover::Runner.run(ARGV)
+

data/lib/redis_failover.rb

@@ -0,0 +1,16 @@
+require 'redis'
+require 'thread'
+require 'logger'
+require 'multi_json'
+require 'securerandom'
+
+require 'redis_failover/cli'
+require 'redis_failover/util'
+require 'redis_failover/node'
+require 'redis_failover/errors'
+require 'redis_failover/client'
+require 'redis_failover/server'
+require 'redis_failover/runner'
+require 'redis_failover/version'
+require 'redis_failover/node_manager'
+require 'redis_failover/node_watcher'

data/lib/redis_failover/cli.rb

@@ -0,0 +1,47 @@
+module RedisFailover
+  # Parses server command-line arguments.
+  class CLI
+    def self.parse(source)
+      return {} if source.empty?
+
+      options = {}
+      parser = OptionParser.new do |opts|
+        opts.banner = "Usage: redis_failover_server [OPTIONS]"
+
+        opts.on('-P', '--port port', 'Server port') do |port|
+          options[:port] = Integer(port)
+        end
+
+        opts.on('-p', '--password password', 'Redis password') do |password|
+          options[:password] = password.strip
+        end
+
+        opts.on('-n', '--nodes nodes', 'Comma-separated redis host:port pairs') do |nodes|
+          # turns 'host1:port,host2:port' => [{:host => host, :port => port}, ...]
+          options[:nodes] = nodes.split(',').map do |node|
+            Hash[[:host, :port].zip(node.strip.split(':'))]
+          end
+        end
+
+        opts.on('--max-failures count',
+          'Max failures before server marks node unreachable (default 3)') do |max|
+          options[:max_failures] = Integer(max)
+        end
+
+        opts.on('-h', '--help', 'Display all options') do
+          puts opts
+          exit
+        end
+      end
+
+      parser.parse(source)
+
+      # assume password is same for all redis nodes
+      if password = options[:password]
+        options[:nodes].each { |opts| opts.update(:password => password) }
+      end
+
+      options
+    end
+  end
+end

data/lib/redis_failover/client.rb

@@ -0,0 +1,198 @@
+require 'set'
+require 'open-uri'
+
+module RedisFailover
+  # Redis failover-aware client.
+  class Client
+    include Util
+
+    REDIS_ERRORS = Errno.constants.map { |c| Errno.const_get(c) }.freeze
+    REDIS_READ_OPS = Set[
+      :dbsize,
+      :echo,
+      :exists,
+      :get,
+      :getbit,
+      :getrange,
+      :hexists,
+      :hget,
+      :hgetall,
+      :hkeys,
+      :hlen,
+      :hmget,
+      :hvals,
+      :info,
+      :keys,
+      :lastsave,
+      :lindex,
+      :llen,
+      :lrange,
+      :mapped_hmget,
+      :mapped_mget,
+      :mget,
+      :ping,
+      :scard,
+      :sdiff,
+      :select,
+      :sinter,
+      :sismember,
+      :smembers,
+      :srandmember,
+      :strlen,
+      :sunion,
+      :ttl,
+      :type,
+      :zcard,
+      :zcount,
+      :zrange,
+      :zrangebyscore,
+      :zrank,
+      :zrevrange,
+      :zrevrangebyscore,
+      :zrevrank,
+      :zscore
+    ].freeze
+
+    # Performance optimization: to avoid unnecessary method_missing calls,
+    # we proactively define methods that dispatch to the underlying redis
+    # calls.
+    Redis.public_instance_methods(false).each do |method|
+      define_method(method) do |*args, &block|
+        dispatch(method, *args, &block)
+      end
+    end
+
+    # Creates a new failover redis client.
+    #
+    # Options:
+    #
+    #   :host - redis failover server host (required)
+    #   :port - redis failover server port (required)
+    #   :password - optional password for redis nodes
+    #   :namespace - optional namespace for redis nodes
+    #   :logger - optional logger override
+    #   :retry_failure - indicate if failures should be retried (default true)
+    #   :max_retries - max retries for a failure (default 5)
+    #
+    def initialize(options = {})
+      unless options.values_at(:host, :port).all?
+        raise ArgumentError, ':host and :port options required'
+      end
+
+      Util.logger = options[:logger] if options[:logger]
+      @namespace = options[:namespace]
+      @password = options[:password]
+      @retry = options[:retry_failure] || true
+      @max_retries = options[:max_retries] || 3
+      @registry_url = "http://#{options[:host]}:#{options[:port]}/redis_servers"
+      @redis_servers = nil
+      @master = nil
+      @slaves = []
+      @lock = Mutex.new
+      build_clients
+    end
+
+    def method_missing(method, *args, &block)
+      if redis_operation?(method)
+        dispatch(method, *args, &block)
+      else
+        super
+      end
+    end
+
+    def respond_to?(method)
+      redis_operation?(method) || super
+    end
+
+    def inspect
+      "#<RedisFailover::Client - master: #{master_info}, slaves: #{slaves_info})>"
+    end
+    alias_method :to_s, :inspect
+
+    private
+
+    def redis_operation?(method)
+      Redis.public_instance_methods(false).include?(method)
+    end
+
+    def dispatch(method, *args, &block)
+      tries = 0
+      begin
+        if REDIS_READ_OPS.include?(method)
+          # send read operations to a slave
+          slave.send(method, *args, &block)
+        else
+          # direct everything else to master
+          master.send(method, *args, &block)
+        end
+      rescue NoMasterError, *REDIS_ERRORS
+        logger.error("No suitable node available for operation `#{method}.`")
+        sleep(3)
+        build_clients
+
+        if @retry && tries < @max_retries
+          tries += 1
+          retry
+        end
+
+        raise
+      end
+    end
+
+    def master
+      @master or raise NoMasterError
+    end
+
+    def slave
+      # pick a slave, if none available fallback to master
+      @slaves.sample || master
+    end
+
+    def build_clients
+      @lock.synchronize do
+        begin
+          logger.info('Attempting to fetch nodes and build redis clients.')
+          servers = fetch_redis_servers
+          master = new_clients_for(servers[:master]).first if servers[:master]
+          slaves = new_clients_for(*servers[:slaves])
+
+          # once clients are successfully created, swap the references
+          @master = master
+          @slaves = slaves
+        rescue => ex
+          logger.error("Failed to fetch servers from #{@registry_url} - #{ex.message}")
+          logger.error(ex.backtrace.join("\n"))
+        end
+      end
+    end
+
+    def fetch_redis_servers
+      open(@registry_url) do |io|
+        servers = symbolize_keys(MultiJson.decode(io))
+        logger.info("Fetched servers: #{servers}")
+        servers
+      end
+    end
+
+    def new_clients_for(*nodes)
+      nodes.map do |node|
+        host, port = node.split(':')
+        client = Redis.new(:host => host, :port => port, :password => @password)
+        if @namespace
+          client = Redis::Namespace.new(@namespace, :redis => client)
+        end
+        client
+      end
+    end
+
+    def master_info
+      return "none" unless @master
+      "#{@master.client.host}:#{@master.client.port}"
+    end
+
+    def slaves_info
+      return "none" if @slaves.empty?
+      @slaves.map { |s| "#{s.client.host}:#{s.client.port}" }.join(', ')
+    end
+  end
+end

data/lib/redis_failover/errors.rb

@@ -0,0 +1,25 @@
+module RedisFailover
+  class Error < StandardError
+  end
+
+  class InvalidNodeError < Error
+  end
+
+  class InvalidNodeStateError < Error
+    def initialize(node, state)
+      super("Invalid state change `#{state}` for node #{node}")
+    end
+  end
+
+  class NodeUnreachableError < Error
+    def initialize(node)
+      super("Node: #{node}")
+    end
+  end
+
+  class NoMasterError < Error
+  end
+
+  class NoSlaveError < Error
+  end
+end