beetle 0.2.3 → 0.2.5

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 XING AG
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/REDIS_AUTO_FAILOVER.rdoc

@@ -0,0 +1,124 @@
+= Automatic Redis Failover for Beetle
+
+== Introduction
+
+Redis is used as the persistence layer in the AMQP message deduplication
+process. Because it is such a critical piece in our infrastructure, it is
+essential that a failure of this service is as unlikely as possible. As our
+AMQP workers are working in a highly distributed manner, all accessing the same
+Redis server, a automatic failover to another Redis server has to be very
+defensive and ensure that every worker in the system will switch to the new
+server at the same time. If the new server would not get accepted from every
+worker, a switch would not be possible. This ensures that even in the case of a
+partitioned network it is impossible that two different workers use two
+different Redis servers for message deduplication.
+
+== Our goals
+
+* opt-in, no need to use the redis-failover solution
+* no single point of failure
+* automatic switch in case of redis-master failure
+* switch should not cause inconsistent data on the redis servers
+* workers should be able to determine the current redis-master without asking
+  another process (as long as the redis servers are working)
+
+== How it works
+
+To ensure consistency, a service (the Redis Configuration Server - RCS) is
+constantly checking the availability and configuration of the currently
+configured Redis master server. If this service detects that the Redis master
+is no longer available, it tries to find an alternative server (one of the
+slaves) which could be promoted to be the new Redis master.
+
+On every worker server runs another daemon, the Redis Configuration Client
+(RCC) which listens to messages sent by the RCS.
+
+If the RCS finds another potential Redis Master, it sends out a message to see
+if all known RCCs are still available (once again to eliminate the risk of a
+partitioned network) and if they agree to the master switch. 
+
+If all RCCs have answered to that message, the RCS sends out a message which 
+tells the RCCs to invalidate the current master.
+
+This happens by deleting the contents of a special file which is used
+by the workers to store the current Redis master (the content of that file is
+the hostname:port of the currently active Redis master). By doing that, it is
+ensured that no operations are done to the old Redis master server anymore, because the
+AMQP workers check this file's mtime and reads its contents in case that the
+file changed, before every Redis operation. When the file has been emptied, the
+RCCs respond to the "invalidate" message of the RCS. When all RCCs have
+responded, the RCS knows for sure that it is safe to switch the Redis master
+now. It sends a "reconfigure" message with the new Redis master hostname:port
+to the RCCs, which then write that value into their redis master file.
+
+Additionally, the RCS sends reconfigure messages with the current Redis master
+periodically, to allow new RCCs to pick up the current master. Plus it turns
+all other redis servers into slaves of the current master. 
+
+=== Prerequisites
+
+* one redis-configuration-server process ("RCS", on one server), one redis-configuration-client process ("RCC") on every worker server
+* the RCS knows about all possible RCCs using a list of client ids
+* the RCS and RCCs exchange messages via a "system queue"
+
+=== Flow of actions
+
+* on startup, an RCC can consult its redis master file to determine the current master without the help of the RCS by checking that it's still a master (or wait for the periodic reconfigure message with the current master from the RCS)
+* when the RCS finds the master to be down, it will retry a couple of times before starting a reconfiguration round
+* the RCS sends all RCCs a "ping" message to check if every client is there and able to to answer
+* the RCCs acknowledge via a "pong" message if they can confirm the current master to be unavailable
+* the RCS waits for *all* RCCs to reply via pong
+* the RCS tells all RCCs to stop using the master by sending an "invalidate" message
+* the RCCs acknowledge via an "invalidated" message if they can still confirm the current master to be unavailable
+* the RCS waits for *all* RCCs to acknowledge the invalidation
+* the RCS promotes the former slave to become the new master (by sending SLAVEOF no one)
+* the RCS sends a "reconfigure" message containing the new master to every RCC
+* the RCCs write the new master to their redis master file
+
+=== Configuration
+
+See Beetle::Configuration for setting redis configuration server and client options.
+
+Please note:
+Beetle::Configuration#redis_server must be a file path (not a redis host:port string) to use the redis failover. The RCS and RCCs store the current redis master in that file, and the handlers read from it.
+
+== How to use it
+
+This example uses two worker servers, identified by rcc-1 and rcc-2.
+
+Please note:
+All command line options can also be given as a yaml configuration file via the --config-file option.
+
+=== On one server
+
+Start the Redis Configuration Server:
+
+  beetle configuration_server start -- --redis-servers redis-1:6379,redis-2:6379 --client-ids rcc-1,rcc-2
+
+Get help for starting/stopping the server:
+
+  beetle configuration_server -h
+
+Get help for server options:
+
+  beetle configuration_server start -- -h
+
+=== On every worker server
+
+Start the Redis Configuration Client:
+
+On first worker server:
+
+  beetle configuration_client start -- --client-id rcc-1
+
+On second worker server:
+
+  beetle configuration_client start -- --client-id rcc-2
+
+Get help for starting/stopping the client:
+
+  beetle configuration_client -h
+
+Get help for client options:
+
+  beetle configuration_client start -- -h

data/RELEASE_NOTES.rdoc

@@ -0,0 +1,50 @@
+= Release Notes
+
+== Version 0.2.5
+
+Added missing files to gem and rdoc
+
+== Version 0.2.4
+
+Log and send a system notification when pong message from unknown client received.
+
+== Version 0.2.2
+
+Patch release which upgrades to redis-rb 2.0.4. This enables us to drop our redis monkey
+patch which enabled connection timeouts for earlier redis versions. Note that earlier
+Beetle versions are not compatible with redis 2.0.4.
+
+== Version 0.2.1
+
+Improved error message when no rabbitmq broker is available.
+
+== Version 0.2
+
+This version adds support for automatic redis deduplication store failover (see separate
+file REDIS_AUTO_FAILOVER.rdoc).
+
+=== User visible changes
+
+* it's possible to register auto deleted queues and exchanges
+* Beetle::Client#configure returns self in order to simplify client setup
+* it's possible to trace specific messages (see Beetle::Client#trace)
+* default message handler timeout is 10 minutes now
+* system wide configuration values can be specified via a yml formatted configuration
+  file (Beetle::Configuration#config_file)
+* the config value redis_server specifies either a single server or a file path (used
+  by the automatic redis failover logic)
+
+=== Fugs Bixed
+
+* handle active_support seconds notation for handler timeouts correctly
+* error handler was erroneously called for expired messages
+* subscribers would block when some non beetle process posts an undecodable message
+
+=== Gem Dependency Changes
+
+* redis needs to be at least version 2.0.3
+* we make use of the SystemTimer gem for ruby 1.8.7
+
+== Version 0.1
+
+Initial Release

data/Rakefile

@@ -0,0 +1,113 @@
+require 'rake'
+require 'rake/testtask'
+require 'rcov/rcovtask'
+require 'cucumber/rake/task'
+
+# 1.8/1.9 compatible way of loading lib/beetle.rb
+$:.unshift 'lib'
+require 'beetle'
+
+namespace :test do
+  namespace :coverage do
+    desc "Delete aggregate coverage data."
+    task(:clean) { rm_f "coverage.data" }
+  end
+
+  desc 'Aggregate code coverage'
+  task :coverage => "test:coverage:clean"
+
+  Rcov::RcovTask.new(:coverage) do |t|
+    t.libs << "test"
+    t.test_files = FileList["test/**/*_test.rb"]
+    t.output_dir = "test/coverage"
+    t.verbose = true
+    t.rcov_opts << "--exclude '.*' --include-file 'lib/beetle/'"
+  end
+  task :coverage do
+    system 'open test/coverage/index.html'
+  end if RUBY_PLATFORM =~ /darwin/
+end
+
+
+namespace :beetle do
+  task :test do
+    Beetle::Client.new.test
+  end
+
+  task :trace do
+    trap('INT'){ EM.stop_event_loop }
+    Beetle::Client.new.trace
+  end
+end
+
+namespace :rabbit do
+  def start(node_name, port)
+    script = File.expand_path(File.dirname(__FILE__)+"/script/start_rabbit")
+    puts "starting rabbit #{node_name} on port #{port}"
+    puts "type ^C a RETURN to abort"
+    sleep 1
+    exec "sudo #{script} #{node_name} #{port}"
+  end
+  desc "start rabbit instance 1"
+  task :start1 do
+    start "rabbit1", 5672
+  end
+  desc "start rabbit instance 2"
+  task :start2 do
+    start "rabbit2", 5673
+  end
+  desc "reset rabbit instances (deletes all data!)"
+  task :reset do
+     ["rabbit1", "rabbit2"].each do |node|
+       `sudo rabbitmqctl -n #{node} stop_app`
+       `sudo rabbitmqctl -n #{node} reset`
+       `sudo rabbitmqctl -n #{node} start_app`
+     end
+  end
+end
+
+namespace :redis do
+  def config_file(suffix)
+    File.expand_path(File.dirname(__FILE__)+"/etc/redis-#{suffix}.conf")
+  end
+  desc "start main redis"
+  task :start1 do
+    exec "redis-server #{config_file(:master)}"
+  end
+  desc "start slave redis"
+  task :start2 do
+    exec "redis-server #{config_file(:slave)}"
+  end
+end
+
+Cucumber::Rake::Task.new(:cucumber) do |t|
+  t.cucumber_opts = "features --format progress"
+end
+
+task :default do
+  Rake::Task[:test].invoke
+  Rake::Task[:cucumber].invoke
+end
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/**/*_test.rb']
+  t.verbose = true
+end
+
+require 'rake/rdoctask'
+
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'site/rdoc'
+  rdoc.title    = 'Beetle'
+  rdoc.main     = 'README.rdoc'
+  rdoc.options << '--line-numbers' << '--inline-source' << '--quiet'
+  rdoc.rdoc_files.include('**/*.rdoc')
+  rdoc.rdoc_files.include('MIT-LICENSE')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+desc "build the beetle gem"
+task :build do
+  system("gem build beetle.gemspec")
+end

data/beetle.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name    = "beetle"
-  s.version = "0.2.3"
+  s.version = "0.2.5"
 
   s.required_rubygems_version = ">= 1.3.1"
   s.authors            = ["Stefan Kaes", "Pascal Friederich", "Ali Jelveh", "Sebastian Roebke"]
@@ -10,8 +10,8 @@ Gem::Specification.new do |s|
   s.summary            = "High Availability AMQP Messaging with Redundant Queues"
   s.email              = "developers@xing.com"
   s.executables        = ["beetle"]
-  s.extra_rdoc_files   = ["README.rdoc"]
-  s.files              = Dir['{examples,ext,lib}/**/*.rb'] + %w(beetle.gemspec examples/README.rdoc)
+  s.extra_rdoc_files   = Dir['**/*.rdoc'] + %w(MIT-LICENSE)
+  s.files              = Dir['{examples,ext,lib}/**/*.rb'] + Dir['{features,script}/**/*'] + %w(beetle.gemspec Rakefile)
   s.extensions         = 'ext/mkrf_conf.rb'
   s.homepage           = "http://xing.github.com/beetle/"
   s.rdoc_options       = ["--charset=UTF-8"]

data/features/README.rdoc

@@ -0,0 +1,23 @@
+=== Cucumber
+
+Beetle ships with a cucumber feature to test the automatic redis failover
+as an integration test.
+
+To run it, you have to start a RabbitMQ.
+
+The top level Rakefile comes with targets to start several RabbitMQ instances locally.
+Make sure the corresponding binaries are in your search path. Open a new shell
+and execute the following command:
+
+    rake rabbit:start1
+
+Then you can run the cucumber feature by running:
+
+    cucumber
+
+or
+
+    rake cucumber
+
+
+Note: Cucumber will automatically run after the unit test when you run rake.

data/features/redis_auto_failover.feature

@@ -0,0 +1,105 @@
+Feature: Redis auto failover
+  In order to eliminate a single point of failure
+  Beetle handlers should automatically switch to a new redis master in case of a redis master failure
+
+  Background:
+    Given a redis server "redis-1" exists as master
+    And a redis server "redis-2" exists as slave of "redis-1"
+
+  Scenario: Successful redis master switch
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And a redis configuration client "rc-client-2" using redis servers "redis-1,redis-2" exists
+    And a beetle handler using the redis-master file from "rc-client-1" exists
+    And redis server "redis-1" is down
+    And the retry timeout for the redis master check is reached
+    Then a system notification for "redis-1" not being available should be sent
+    And the role of redis server "redis-2" should be "master"
+    And the redis master of "rc-client-1" should be "redis-2"
+    And the redis master of "rc-client-2" should be "redis-2"
+    And the redis master of the beetle handler should be "redis-2"
+    And a system notification for switching from "redis-1" to "redis-2" should be sent
+    Given a redis server "redis-1" exists as master
+    Then the role of redis server "redis-1" should be "slave"
+
+  Scenario: Redis master only temporarily down (no switch necessary)
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And a redis configuration client "rc-client-2" using redis servers "redis-1,redis-2" exists
+    And a beetle handler using the redis-master file from "rc-client-1" exists
+    And redis server "redis-1" is down for less seconds than the retry timeout for the redis master check
+    And the retry timeout for the redis master check is reached
+    Then the role of redis server "redis-1" should be "master"
+    Then the role of redis server "redis-2" should be "slave"
+    And the redis master of "rc-client-1" should be "redis-1"
+    And the redis master of "rc-client-2" should be "redis-1"
+    And the redis master of the beetle handler should be "redis-1"
+
+  Scenario: Not all redis configuration clients available (no switch possible)
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And redis server "redis-1" is down
+    And the retry timeout for the redis master check is reached
+    Then the role of redis server "redis-2" should be "slave"
+
+  Scenario: No redis slave available to become new master (no switch possible)
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And a redis configuration client "rc-client-2" using redis servers "redis-1,redis-2" exists
+    And redis server "redis-1" is down
+    And redis server "redis-2" is down
+    And the retry timeout for the redis master check is reached
+    Then the redis master of "rc-client-1" should be "redis-1"
+    And the redis master of "rc-client-2" should be "redis-1"
+    And a system notification for no slave available to become new master should be sent
+
+  Scenario: Redis configuration client starts while no redis master available
+    Given redis server "redis-1" is down
+    And redis server "redis-2" is down
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And the retry timeout for the redis master determination is reached
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while no redis master available but master file exists
+    Given redis server "redis-1" is down
+    And redis server "redis-2" is down
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And the retry timeout for the redis master determination is reached
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while both redis servers are master
+    Given redis server "redis-2" is master
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while both redis servers are master but master file exists
+    Given redis server "redis-2" is master
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be "redis-1"
+
+  Scenario: Redis configuration client starts while both redis servers are slave
+    Given a redis server "redis-3" exists as master
+    And redis server "redis-1" is slave of "redis-3"
+    And redis server "redis-2" is slave of "redis-3"
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while both redis servers are slave but master file exists
+    Given a redis server "redis-3" exists as master
+    And redis server "redis-1" is slave of "redis-3"
+    And redis server "redis-2" is slave of "redis-3"
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while there is a redis master but no slave
+    Given redis server "redis-2" is down
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while there is a redis master but no slave but master file exists
+    Given redis server "redis-2" is down
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be "redis-1"