nogara-redis_failover 0.8.9

data/.gitignore

@@ -0,0 +1,19 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+tags
+

data/.travis.yml

@@ -0,0 +1,5 @@
+language: ruby
+rvm:
+  - 1.9.2
+  - 1.9.3
+  - jruby-19mode

data/.yardopts

@@ -0,0 +1,6 @@
+-m markdown
+--readme README.md
+-
+Changes.md
+LICENSE
+

data/Changes.md

@@ -0,0 +1,116 @@
+0.8.9
+-----------
+- Handle errors raised by redis 3.x client (tsilen) 
+
+0.8.8
+-----------
+- Use a stack for handling nested blocks in RedisFailover::Client (inspired by connection_pool gem)
+- Fix an issue with #multi and Redis 3.x.
+
+0.8.7
+-----------
+- Support TTL operation (#24)
+
+0.8.6
+-----------
+- No longer buffer output (kyohsuke)
+- Update redis/zk gem versions to latest (rudionrails)
+
+0.8.5
+-----------
+- Lock-down gemspec to version 1.1.x of redis-namespace to play nicely with redis 2.2.x.
+- Fix RedisFailover::Client#manual_failover regression (oleriesenberg)
+
+0.8.4
+-----------
+- Lock-down gemspec to redis 2.2.x in light of upcoming redis 3.x release. Once sufficient testing
+has been done with the 3.x release, I will relax the constraint in the gemspec.
+- Add environment-scoped configuration file support (oleriesenberg)
+
+0.8.3
+-----------
+- Added a way to gracefully shutdown/reconnect a RedisFailover::Client. (#13)
+- Upgraded to latest ZK version that supports forking.
+- Handle case where the same RedisFailover::Client is referenced by a #multi block (#14)
+
+0.8.2
+-----------
+- Fix method signature for RedisFailover::Client#respond_to_missing? (#12)
+
+0.8.1
+-----------
+- Added YARD documentation.
+- Improve ZooKeeper client connection management.
+- Upgrade to latest ZK gem stable release.
+
+0.8.0
+-----------
+- Added manual failover support (can be initiated via RedisFailover::Client#manual_failover)
+- Misc. cleanup
+
+0.7.0
+-----------
+- When new master promotion occurs, make existing slaves point to new candidate before promoting new master.
+- Add support for specifying command-line options in a config.yml file for Node Manager.
+- Upgrade to 0.9 version of ZK client and cleanup ZK connection error handling.
+
+0.6.0
+-----------
+- Add support for running multiple Node Manager processes for added redundancy (#4)
+- Add support for specifying a redis database in RedisFailover::Client (#5)
+- Improved Node Manager command-line option parsing
+
+0.5.4
+-----------
+- No longer use problematic ZK#reopen.
+
+0.5.3
+-----------
+- Handle more ZK exceptions as candidates for reconnecting the client on error.
+- Add safety check to actively purge redis clients if a RedisFailover::Client hasn't heard from the Node Manager in a certain time window.
+
+0.5.2
+-----------
+- Always try to create path before setting current state in Node Manager.
+- More explicit rescuing of exceptions.
+
+0.5.1
+-----------
+- More logging around exceptions
+- Handle re-watching on client session expirations / disconnections
+- Use an ephemeral node for the list of redis servers
+
+0.5.0
+-----------
+- redis_failover is now built on top of ZooKeeper! This means redis_failover enjoys all of the reliability, redundancy, and data consistency offered by ZooKeeper. The old fragile HTTP-based approach has been removed and will no longer be supported in favor of ZooKeeper. This does mean that in order to use redis_failover, you must have ZooKeeper installed and running. Please see the README for steps on how to do this if you don't already have ZooKeeper running in your production environment.
+
+0.4.0
+-----------
+- No longer force newly available slaves to master if already slaves of that master
+- Honor a node's slave-serve-stale-data configuration option; do not mark a sync-with-master-in-progress slave as available if its slave-serve-stale-data is disabled
+- Change reachable/unreachable wording to available/unavailable
+- Added node reconciliation, i.e. if a node comes back up, make sure that the node manager and the node agree on current role
+- More efficient use of redis client connections
+- Raise proper error for unsupported operations (i.e., those that don't make sense for a failover client)
+- Properly handle any hanging node operations in the failover server
+
+0.3.0
+-----------
+- Integrated travis-ci
+- Added background monitor to client for proactively detecting changes to current set of redis nodes
+
+0.2.0
+-----------
+- Added retry support for contacting failover server from client
+- Client now verifies proper master/slave role before attempting operation
+- General edge case cleanup for NodeManager
+
+0.1.1
+-----------
+
+- Fix option parser require
+
+0.1.0
+-----------
+
+- First release

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Ryan LeCompte
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,190 @@
+# Automatic Redis Failover
+
+[![Build Status](https://secure.travis-ci.org/ryanlecompte/redis_failover.png?branch=master)](http://travis-ci.org/ryanlecompte/redis_failover)
+
+redis_failover attempts to provides a full automatic master/slave failover solution for Ruby. Redis does not provide
+an automatic failover capability when configured for master/slave replication. When the master node dies,
+a new master must be manually brought online and assigned as the slave's new master. This manual
+switch-over is not desirable in high traffic sites where Redis is a critical part of the overall
+architecture. The existing standard Redis client for Ruby also only supports configuration for a single
+Redis server. When using master/slave replication, it is desirable to have all writes go to the
+master, and all reads go to one of the N configured slaves.
+
+This gem (built using [ZK][]) attempts to address these failover scenarios. A redis failover Node Manager daemon runs as a background
+process and monitors all of your configured master/slave nodes. When the daemon starts up, it
+automatically discovers the current master/slaves. Background watchers are setup for each of
+the redis nodes. As soon as a node is detected as being offline, it will be moved to an "unavailable" state.
+If the node that went offline was the master, then one of the slaves will be promoted as the new master.
+All existing slaves will be automatically reconfigured to point to the new master for replication.
+All nodes marked as unavailable will be periodically checked to see if they have been brought back online.
+If so, the newly available nodes will be configured as slaves and brought back into the list of available
+nodes. Note that detection of a node going down should be nearly instantaneous, since the mechanism
+used to keep tabs on a node is via a blocking Redis BLPOP call (no polling). This call fails nearly
+immediately when the node actually goes offline. To avoid false positives (i.e., intermittent flaky
+network interruption), the Node Manager will only mark a node as unavailable if it fails to communicate with
+it 3 times (this is configurable via --max-failures, see configuration options below). Note that you can
+deploy multiple Node Manager daemons for added redundancy.
+
+This gem provides a RedisFailover::Client wrapper that is master/slave aware. The client is configured
+with a list of ZooKeeper servers. The client will automatically contact the ZooKeeper cluster to find out
+the current state of the world (i.e., who is the current master and who are the current slaves). The client
+also sets up a ZooKeeper watcher for the set of redis nodes controlled by the Node Manager daemon. When the daemon
+promotes a new master or detects a node as going down, ZooKeeper will notify the client near-instantaneously so
+that it can rebuild its set of Redis connections. The client also acts as a load balancer in that it will automatically
+dispatch Redis read operations to one of N slaves, and Redis write operations to the master.
+If it fails to communicate with any node, it will go back and fetch the current list of available servers, and then
+optionally retry the operation.
+
+[ZK]: https://github.com/slyphon/zk
+
+## Architecture Diagram
+
+![redis_failover architecture diagram](https://github.com/ryanlecompte/redis_failover/raw/master/misc/redis_failover.png)
+
+## Installation
+
+redis_failover has an external dependency on ZooKeeper. You must have a running ZooKeeper cluster already available in order to use redis_failover. ZooKeeper provides redis_failover with its high availability and data consistency between Redis::Failover clients and the Node Manager daemon. Please see the requirements section below for more information on installing and setting up ZooKeeper if you don't have it running already.
+
+Add this line to your application's Gemfile:
+
+    gem 'redis_failover'
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install redis_failover
+
+## Node Manager Daemon Usage
+
+The Node Manager is a simple process that should be run as a background daemon. The daemon supports the
+following options:
+
+    Usage: redis_node_manager [OPTIONS]
+
+    Specific options:
+        -n, --nodes NODES                Comma-separated redis host:port pairs
+        -z, --zkservers SERVERS          Comma-separated ZooKeeper host:port pairs
+        -p, --password PASSWORD          Redis password
+            --znode-path PATH            Znode path override for storing redis server list
+            --max-failures COUNT         Max failures before manager marks node unavailable
+        -C, --config PATH                Path to YAML configuration file
+        -E, --environment ENV            Config environment to use
+        -h, --help                       Display all options
+
+To start the daemon for a simple master/slave configuration, use the following:
+
+    redis_node_manager -n localhost:6379,localhost:6380 -z localhost:2181,localhost:2182,localhost:2183
+
+The configuration parameters can also be specified in a config.yml file. An example configuration
+would look like the following:
+
+    ---
+    :max_failures: 2
+    :nodes:
+      - localhost:6379
+      - localhost:1111
+      - localhost:2222
+      - localhost:3333
+    :zkservers:
+      - localhost:2181
+      - localhost:2182
+      - localhost:2183
+    :password: foobar
+
+You would then simpy start the Node Manager via the following:
+
+    redis_node_manager -C config.yml
+
+You can also scope the configuration to a particular environment (e.g., staging/development). See the examples
+directory for configuration file samples.
+
+The Node Manager will automatically discover the master/slaves upon startup. Note that it is
+a good idea to run more than one instance of the Node Manager daemon in your environment. At
+any moment, a single Node Manager process will be designated to monitor the redis servers. If
+this Node Manager process dies or becomes partitioned from the network, another Node Manager
+will be promoted as the primary monitor of redis servers. You can run as many Node Manager
+processes as you'd like for added redundancy.
+
+## Client Usage
+
+The redis failover client must be used in conjunction with a running Node Manager daemon. The
+client supports various configuration options, however the only mandatory option is the list of
+ZooKeeper servers:
+
+    client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
+
+The client actually employs the common redis and redis-namespace gems underneath, so this should be
+a drop-in replacement for your existing pure redis client usage.
+
+The full set of options that can be passed to RedisFailover::Client are:
+
+     :zkservers     - comma-separated ZooKeeper host:port pairs (required)
+     :znode_path    - the Znode path override for redis server list (optional)
+     :password      - password for redis nodes (optional)
+     :db            - db to use for redis nodes (optional)
+     :namespace     - namespace for redis nodes (optional)
+     :logger        - logger override (optional)
+     :retry_failure - indicate if failures should be retried (default true)
+     :max_retries   - max retries for a failure (default 3)
+
+## Manual Failover
+
+Manual failover can be initiated via RedisFailover::Client#manual_failover. This schedules a manual failover with the
+currently active Node Manager. Once the Node Manager receives the request, it will either failover to the specific
+server passed to #manual_failover, or it will pick a random slave to become the new master. Here's an example:
+
+    client = RedisFailover::Client.new(:zkservers => 'localhost:2181,localhost:2182,localhost:2183')
+    client.manual_failover(:host => 'localhost', :port => 2222)
+
+## Documentation
+
+redis_failover uses YARD for its API documentation. Refer to the generated [API documentation](http://rubydoc.info/github/ryanlecompte/redis_failover/master/frames) for full coverage.
+
+## Requirements
+
+- redis_failover is actively tested against MRI 1.9.2/1.9.3 and JRuby 1.6.7 (1.9 mode only). Other rubies may work, although I don't actively test against them.
+- redis_failover requires a ZooKeeper service cluster to ensure reliability and data consistency. ZooKeeper is very simple and easy to get up and running. Please refer to this [Quick ZooKeeper Guide](https://github.com/ryanlecompte/redis_failover/wiki/Quick-ZooKeeper-Guide) to get up and running quickly if you don't already have ZooKeeper as a part of your environment.
+
+## Considerations
+
+- Note that by default the Node Manager will mark slaves that are currently syncing with their master as "available" based on the configuration value set for "slave-serve-stale-data" in redis.conf. By default this value is set to "yes" in the configuration, which means that slaves still syncing with their master will be available for servicing read requests. If you don't want this behavior, just set "slave-serve-stale-data" to "no" in your redis.conf file.
+
+## Limitations
+
+- Note that it's still possible for the RedisFailover::Client instances to see a stale list of servers for a very small window. In most cases this will not be the case due to how ZooKeeper handles distributed communication, but you should be aware that in the worst case the client could write to a "stale" master for a small period of time until the next watch event is received by the client via ZooKeeper.
+
+- Note that currently multiple Node Managers are currently used for redundancy purposes only. The Node Managers do not communicate with each other to perform any type of election or voting to determine if they all agree on promoting a new master. Right now Node Managers that are not "active" just sit and wait until they can grab the lock to become the single decision-maker for which Redis servers are available or not. This means that a scenario could present itself where a Node Manager thinks the Redis master is available, however the actual RedisFailover::Client instances think they can't reach the Redis master (either due to network partitions or the Node Manager flapping due to machine failure, etc). We are exploring ways to improve this situation.
+
+## Resources
+
+- Check out Steve Whittaker's [redis-failover-test](https://github.com/swhitt/redis-failover-test) project which shows how to test redis_failover in a non-trivial configuration using Vagrant/Chef.
+- To learn more about Redis master/slave replication, see the [Redis documentation](http://redis.io/topics/replication).
+- To learn more about ZooKeeper, see the official [ZooKeeper](http://zookeeper.apache.org/) site.
+- See the [Quick ZooKeeper Guide](https://github.com/ryanlecompte/redis_failover/wiki/Quick-ZooKeeper-Guide) for a quick guide to getting ZooKeeper up and running with redis_failover.
+- To learn more about how ZooKeeper handles network partitions, see [ZooKeeper Failure Scenarios](http://wiki.apache.org/hadoop/ZooKeeper/FailureScenarios)
+
+
+## License
+
+Please see LICENSE for licensing details.
+
+## Author
+
+Ryan LeCompte
+
+[@ryanlecompte](https://twitter.com/ryanlecompte)
+
+## Acknowledgements
+
+Special thanks to [Eric Lindvall](https://github.com/eric) and [Jonathan Simms](https://github.com/slyphon) for their invaluable ZooKeeper advice and guidance!
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,9 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = %w(--format progress)
+end
+
+task :default => [:spec]

data/bin/redis_node_manager

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+$stdout.sync = true
+require 'redis_failover'
+
+RedisFailover::Runner.run(ARGV)
+

data/examples/config.yml

@@ -0,0 +1,14 @@
+# Sample configuration file for Node Manager.
+#   redis_node_manager -C config.yml
+---
+:max_failures: 2
+:nodes:
+  - localhost:6379
+  - localhost:1111
+  - localhost:2222
+  - localhost:3333
+:zkservers:
+  - localhost:2181
+  - localhost:2182
+  - localhost:2183
+:password: foobar

data/examples/multiple_environments_config.yml

@@ -0,0 +1,15 @@
+:development:
+  :nodes:
+    - localhost:6379
+    - localhost:6389
+  :zkservers:
+    - localhost:2181
+
+:staging:
+  :nodes:
+    - redis01:6379
+    - redis02:6379
+  :zkservers:
+    - zk01:2181
+    - zk02:2181
+    - zk03:2181

data/lib/redis_failover.rb

@@ -0,0 +1,22 @@
+require 'zk'
+require 'set'
+require 'yaml'
+require 'redis'
+require 'thread'
+require 'logger'
+require 'timeout'
+require 'optparse'
+require 'multi_json'
+require 'securerandom'
+
+require 'redis_failover/cli'
+require 'redis_failover/util'
+require 'redis_failover/node'
+require 'redis_failover/errors'
+require 'redis_failover/client'
+require 'redis_failover/runner'
+require 'redis_failover/version'
+require 'redis_failover/node_manager'
+require 'redis_failover/node_watcher'
+require 'redis_failover/manual_failover'
+