beetle 0.2.4 → 0.2.5

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright (c) 2010 XING AG
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/REDIS_AUTO_FAILOVER.rdoc

@@ -0,0 +1,124 @@
+= Automatic Redis Failover for Beetle
+
+== Introduction
+
+Redis is used as the persistence layer in the AMQP message deduplication
+process. Because it is such a critical piece in our infrastructure, it is
+essential that a failure of this service is as unlikely as possible. As our
+AMQP workers are working in a highly distributed manner, all accessing the same
+Redis server, a automatic failover to another Redis server has to be very
+defensive and ensure that every worker in the system will switch to the new
+server at the same time. If the new server would not get accepted from every
+worker, a switch would not be possible. This ensures that even in the case of a
+partitioned network it is impossible that two different workers use two
+different Redis servers for message deduplication.
+
+== Our goals
+
+* opt-in, no need to use the redis-failover solution
+* no single point of failure
+* automatic switch in case of redis-master failure
+* switch should not cause inconsistent data on the redis servers
+* workers should be able to determine the current redis-master without asking
+  another process (as long as the redis servers are working)
+
+== How it works
+
+To ensure consistency, a service (the Redis Configuration Server - RCS) is
+constantly checking the availability and configuration of the currently
+configured Redis master server. If this service detects that the Redis master
+is no longer available, it tries to find an alternative server (one of the
+slaves) which could be promoted to be the new Redis master.
+
+On every worker server runs another daemon, the Redis Configuration Client
+(RCC) which listens to messages sent by the RCS.
+
+If the RCS finds another potential Redis Master, it sends out a message to see
+if all known RCCs are still available (once again to eliminate the risk of a
+partitioned network) and if they agree to the master switch. 
+
+If all RCCs have answered to that message, the RCS sends out a message which 
+tells the RCCs to invalidate the current master.
+
+This happens by deleting the contents of a special file which is used
+by the workers to store the current Redis master (the content of that file is
+the hostname:port of the currently active Redis master). By doing that, it is
+ensured that no operations are done to the old Redis master server anymore, because the
+AMQP workers check this file's mtime and reads its contents in case that the
+file changed, before every Redis operation. When the file has been emptied, the
+RCCs respond to the "invalidate" message of the RCS. When all RCCs have
+responded, the RCS knows for sure that it is safe to switch the Redis master
+now. It sends a "reconfigure" message with the new Redis master hostname:port
+to the RCCs, which then write that value into their redis master file.
+
+Additionally, the RCS sends reconfigure messages with the current Redis master
+periodically, to allow new RCCs to pick up the current master. Plus it turns
+all other redis servers into slaves of the current master. 
+
+=== Prerequisites
+
+* one redis-configuration-server process ("RCS", on one server), one redis-configuration-client process ("RCC") on every worker server
+* the RCS knows about all possible RCCs using a list of client ids
+* the RCS and RCCs exchange messages via a "system queue"
+
+=== Flow of actions
+
+* on startup, an RCC can consult its redis master file to determine the current master without the help of the RCS by checking that it's still a master (or wait for the periodic reconfigure message with the current master from the RCS)
+* when the RCS finds the master to be down, it will retry a couple of times before starting a reconfiguration round
+* the RCS sends all RCCs a "ping" message to check if every client is there and able to to answer
+* the RCCs acknowledge via a "pong" message if they can confirm the current master to be unavailable
+* the RCS waits for *all* RCCs to reply via pong
+* the RCS tells all RCCs to stop using the master by sending an "invalidate" message
+* the RCCs acknowledge via an "invalidated" message if they can still confirm the current master to be unavailable
+* the RCS waits for *all* RCCs to acknowledge the invalidation
+* the RCS promotes the former slave to become the new master (by sending SLAVEOF no one)
+* the RCS sends a "reconfigure" message containing the new master to every RCC
+* the RCCs write the new master to their redis master file
+
+=== Configuration
+
+See Beetle::Configuration for setting redis configuration server and client options.
+
+Please note:
+Beetle::Configuration#redis_server must be a file path (not a redis host:port string) to use the redis failover. The RCS and RCCs store the current redis master in that file, and the handlers read from it.
+
+== How to use it
+
+This example uses two worker servers, identified by rcc-1 and rcc-2.
+
+Please note:
+All command line options can also be given as a yaml configuration file via the --config-file option.
+
+=== On one server
+
+Start the Redis Configuration Server:
+
+  beetle configuration_server start -- --redis-servers redis-1:6379,redis-2:6379 --client-ids rcc-1,rcc-2
+
+Get help for starting/stopping the server:
+
+  beetle configuration_server -h
+
+Get help for server options:
+
+  beetle configuration_server start -- -h
+
+=== On every worker server
+
+Start the Redis Configuration Client:
+
+On first worker server:
+
+  beetle configuration_client start -- --client-id rcc-1
+
+On second worker server:
+
+  beetle configuration_client start -- --client-id rcc-2
+
+Get help for starting/stopping the client:
+
+  beetle configuration_client -h
+
+Get help for client options:
+
+  beetle configuration_client start -- -h

data/RELEASE_NOTES.rdoc

@@ -0,0 +1,50 @@
+= Release Notes
+
+== Version 0.2.5
+
+Added missing files to gem and rdoc
+
+== Version 0.2.4
+
+Log and send a system notification when pong message from unknown client received.
+
+== Version 0.2.2
+
+Patch release which upgrades to redis-rb 2.0.4. This enables us to drop our redis monkey
+patch which enabled connection timeouts for earlier redis versions. Note that earlier
+Beetle versions are not compatible with redis 2.0.4.
+
+== Version 0.2.1
+
+Improved error message when no rabbitmq broker is available.
+
+== Version 0.2
+
+This version adds support for automatic redis deduplication store failover (see separate
+file REDIS_AUTO_FAILOVER.rdoc).
+
+=== User visible changes
+
+* it's possible to register auto deleted queues and exchanges
+* Beetle::Client#configure returns self in order to simplify client setup
+* it's possible to trace specific messages (see Beetle::Client#trace)
+* default message handler timeout is 10 minutes now
+* system wide configuration values can be specified via a yml formatted configuration
+  file (Beetle::Configuration#config_file)
+* the config value redis_server specifies either a single server or a file path (used
+  by the automatic redis failover logic)
+
+=== Fugs Bixed
+
+* handle active_support seconds notation for handler timeouts correctly
+* error handler was erroneously called for expired messages
+* subscribers would block when some non beetle process posts an undecodable message
+
+=== Gem Dependency Changes
+
+* redis needs to be at least version 2.0.3
+* we make use of the SystemTimer gem for ruby 1.8.7
+
+== Version 0.1
+
+Initial Release

data/Rakefile

@@ -0,0 +1,113 @@
+require 'rake'
+require 'rake/testtask'
+require 'rcov/rcovtask'
+require 'cucumber/rake/task'
+
+# 1.8/1.9 compatible way of loading lib/beetle.rb
+$:.unshift 'lib'
+require 'beetle'
+
+namespace :test do
+  namespace :coverage do
+    desc "Delete aggregate coverage data."
+    task(:clean) { rm_f "coverage.data" }
+  end
+
+  desc 'Aggregate code coverage'
+  task :coverage => "test:coverage:clean"
+
+  Rcov::RcovTask.new(:coverage) do |t|
+    t.libs << "test"
+    t.test_files = FileList["test/**/*_test.rb"]
+    t.output_dir = "test/coverage"
+    t.verbose = true
+    t.rcov_opts << "--exclude '.*' --include-file 'lib/beetle/'"
+  end
+  task :coverage do
+    system 'open test/coverage/index.html'
+  end if RUBY_PLATFORM =~ /darwin/
+end
+
+
+namespace :beetle do
+  task :test do
+    Beetle::Client.new.test
+  end
+
+  task :trace do
+    trap('INT'){ EM.stop_event_loop }
+    Beetle::Client.new.trace
+  end
+end
+
+namespace :rabbit do
+  def start(node_name, port)
+    script = File.expand_path(File.dirname(__FILE__)+"/script/start_rabbit")
+    puts "starting rabbit #{node_name} on port #{port}"
+    puts "type ^C a RETURN to abort"
+    sleep 1
+    exec "sudo #{script} #{node_name} #{port}"
+  end
+  desc "start rabbit instance 1"
+  task :start1 do
+    start "rabbit1", 5672
+  end
+  desc "start rabbit instance 2"
+  task :start2 do
+    start "rabbit2", 5673
+  end
+  desc "reset rabbit instances (deletes all data!)"
+  task :reset do
+     ["rabbit1", "rabbit2"].each do |node|
+       `sudo rabbitmqctl -n #{node} stop_app`
+       `sudo rabbitmqctl -n #{node} reset`
+       `sudo rabbitmqctl -n #{node} start_app`
+     end
+  end
+end
+
+namespace :redis do
+  def config_file(suffix)
+    File.expand_path(File.dirname(__FILE__)+"/etc/redis-#{suffix}.conf")
+  end
+  desc "start main redis"
+  task :start1 do
+    exec "redis-server #{config_file(:master)}"
+  end
+  desc "start slave redis"
+  task :start2 do
+    exec "redis-server #{config_file(:slave)}"
+  end
+end
+
+Cucumber::Rake::Task.new(:cucumber) do |t|
+  t.cucumber_opts = "features --format progress"
+end
+
+task :default do
+  Rake::Task[:test].invoke
+  Rake::Task[:cucumber].invoke
+end
+
+Rake::TestTask.new do |t|
+  t.libs << "test"
+  t.test_files = FileList['test/**/*_test.rb']
+  t.verbose = true
+end
+
+require 'rake/rdoctask'
+
+Rake::RDocTask.new do |rdoc|
+  rdoc.rdoc_dir = 'site/rdoc'
+  rdoc.title    = 'Beetle'
+  rdoc.main     = 'README.rdoc'
+  rdoc.options << '--line-numbers' << '--inline-source' << '--quiet'
+  rdoc.rdoc_files.include('**/*.rdoc')
+  rdoc.rdoc_files.include('MIT-LICENSE')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+desc "build the beetle gem"
+task :build do
+  system("gem build beetle.gemspec")
+end

data/beetle.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name    = "beetle"
-  s.version = "0.2.4"
+  s.version = "0.2.5"
 
   s.required_rubygems_version = ">= 1.3.1"
   s.authors            = ["Stefan Kaes", "Pascal Friederich", "Ali Jelveh", "Sebastian Roebke"]
@@ -10,8 +10,8 @@ Gem::Specification.new do |s|
   s.summary            = "High Availability AMQP Messaging with Redundant Queues"
   s.email              = "developers@xing.com"
   s.executables        = ["beetle"]
-  s.extra_rdoc_files   = ["README.rdoc"]
-  s.files              = Dir['{examples,ext,lib}/**/*.rb'] + %w(beetle.gemspec examples/README.rdoc)
+  s.extra_rdoc_files   = Dir['**/*.rdoc'] + %w(MIT-LICENSE)
+  s.files              = Dir['{examples,ext,lib}/**/*.rb'] + Dir['{features,script}/**/*'] + %w(beetle.gemspec Rakefile)
   s.extensions         = 'ext/mkrf_conf.rb'
   s.homepage           = "http://xing.github.com/beetle/"
   s.rdoc_options       = ["--charset=UTF-8"]

data/features/README.rdoc

@@ -0,0 +1,23 @@
+=== Cucumber
+
+Beetle ships with a cucumber feature to test the automatic redis failover
+as an integration test.
+
+To run it, you have to start a RabbitMQ.
+
+The top level Rakefile comes with targets to start several RabbitMQ instances locally.
+Make sure the corresponding binaries are in your search path. Open a new shell
+and execute the following command:
+
+    rake rabbit:start1
+
+Then you can run the cucumber feature by running:
+
+    cucumber
+
+or
+
+    rake cucumber
+
+
+Note: Cucumber will automatically run after the unit test when you run rake.

data/features/redis_auto_failover.feature

@@ -0,0 +1,105 @@
+Feature: Redis auto failover
+  In order to eliminate a single point of failure
+  Beetle handlers should automatically switch to a new redis master in case of a redis master failure
+
+  Background:
+    Given a redis server "redis-1" exists as master
+    And a redis server "redis-2" exists as slave of "redis-1"
+
+  Scenario: Successful redis master switch
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And a redis configuration client "rc-client-2" using redis servers "redis-1,redis-2" exists
+    And a beetle handler using the redis-master file from "rc-client-1" exists
+    And redis server "redis-1" is down
+    And the retry timeout for the redis master check is reached
+    Then a system notification for "redis-1" not being available should be sent
+    And the role of redis server "redis-2" should be "master"
+    And the redis master of "rc-client-1" should be "redis-2"
+    And the redis master of "rc-client-2" should be "redis-2"
+    And the redis master of the beetle handler should be "redis-2"
+    And a system notification for switching from "redis-1" to "redis-2" should be sent
+    Given a redis server "redis-1" exists as master
+    Then the role of redis server "redis-1" should be "slave"
+
+  Scenario: Redis master only temporarily down (no switch necessary)
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And a redis configuration client "rc-client-2" using redis servers "redis-1,redis-2" exists
+    And a beetle handler using the redis-master file from "rc-client-1" exists
+    And redis server "redis-1" is down for less seconds than the retry timeout for the redis master check
+    And the retry timeout for the redis master check is reached
+    Then the role of redis server "redis-1" should be "master"
+    Then the role of redis server "redis-2" should be "slave"
+    And the redis master of "rc-client-1" should be "redis-1"
+    And the redis master of "rc-client-2" should be "redis-1"
+    And the redis master of the beetle handler should be "redis-1"
+
+  Scenario: Not all redis configuration clients available (no switch possible)
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And redis server "redis-1" is down
+    And the retry timeout for the redis master check is reached
+    Then the role of redis server "redis-2" should be "slave"
+
+  Scenario: No redis slave available to become new master (no switch possible)
+    Given a redis configuration server using redis servers "redis-1,redis-2" with clients "rc-client-1,rc-client-2" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And a redis configuration client "rc-client-2" using redis servers "redis-1,redis-2" exists
+    And redis server "redis-1" is down
+    And redis server "redis-2" is down
+    And the retry timeout for the redis master check is reached
+    Then the redis master of "rc-client-1" should be "redis-1"
+    And the redis master of "rc-client-2" should be "redis-1"
+    And a system notification for no slave available to become new master should be sent
+
+  Scenario: Redis configuration client starts while no redis master available
+    Given redis server "redis-1" is down
+    And redis server "redis-2" is down
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And the retry timeout for the redis master determination is reached
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while no redis master available but master file exists
+    Given redis server "redis-1" is down
+    And redis server "redis-2" is down
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    And the retry timeout for the redis master determination is reached
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while both redis servers are master
+    Given redis server "redis-2" is master
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while both redis servers are master but master file exists
+    Given redis server "redis-2" is master
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be "redis-1"
+
+  Scenario: Redis configuration client starts while both redis servers are slave
+    Given a redis server "redis-3" exists as master
+    And redis server "redis-1" is slave of "redis-3"
+    And redis server "redis-2" is slave of "redis-3"
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while both redis servers are slave but master file exists
+    Given a redis server "redis-3" exists as master
+    And redis server "redis-1" is slave of "redis-3"
+    And redis server "redis-2" is slave of "redis-3"
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while there is a redis master but no slave
+    Given redis server "redis-2" is down
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be undefined
+
+  Scenario: Redis configuration client starts while there is a redis master but no slave but master file exists
+    Given redis server "redis-2" is down
+    And an old redis master file for "rc-client-1" with master "redis-1" exists
+    And a redis configuration client "rc-client-1" using redis servers "redis-1,redis-2" exists
+    Then the redis master of "rc-client-1" should be "redis-1"

data/features/step_definitions/redis_auto_failover_steps.rb

@@ -0,0 +1,133 @@
+Given /^a redis server "([^\"]*)" exists as master$/ do |redis_name|
+  TestDaemons::Redis[redis_name].start
+  TestDaemons::Redis[redis_name].master
+end
+
+Given /^a redis server "([^\"]*)" exists as slave of "([^\"]*)"$/ do |redis_name, redis_master_name|
+  TestDaemons::Redis[redis_name].start
+  Given "redis server \"#{redis_name}\" is slave of \"#{redis_master_name}\""
+end
+
+Given /^redis server "([^\"]*)" is master$/ do |redis_name|
+  TestDaemons::Redis[redis_name].master
+end
+
+Given /^redis server "([^\"]*)" is slave of "([^\"]*)"$/ do |redis_name, redis_master_name|
+  TestDaemons::Redis[redis_name].slave_of(TestDaemons::Redis[redis_master_name].port)
+  master = TestDaemons::Redis[redis_master_name].redis
+  slave = TestDaemons::Redis[redis_name].redis
+  begin
+    sleep 1
+  end while !slave.slave_of?(master.host, master.port)
+end
+
+Given /^a redis configuration server using redis servers "([^\"]*)" with clients "([^\"]*)" exists$/ do |redis_names, redis_configuration_client_names|
+  redis_servers = redis_names.split(",").map { |redis_name| TestDaemons::Redis[redis_name].ip_with_port }.join(",")
+  TestDaemons::RedisConfigurationServer.start(redis_servers, redis_configuration_client_names)
+end
+
+Given /^a redis configuration client "([^\"]*)" using redis servers "([^\"]*)" exists$/ do |redis_configuration_client_name, redis_names|
+  redis_servers = redis_names.split(",").map do |redis_name|
+    TestDaemons::Redis[redis_name].ip_with_port
+  end
+  TestDaemons::RedisConfigurationClient[redis_configuration_client_name].start
+end
+
+Given /^redis server "([^\"]*)" is down$/ do |redis_name|
+  TestDaemons::Redis[redis_name].stop
+end
+
+Given /^the retry timeout for the redis master check is reached$/ do
+  basetime = Time.now
+  i = 0
+  while (i <= 10.0) do
+    break if TestDaemons::RedisConfigurationClient.instances.values.all? {|instance| File.mtime(instance.redis_master_file) > basetime rescue false}
+    i += 0.1
+    sleep(0.1)
+  end
+  sleep 1 # give it time to switch because the modified mtime might be because of the initial invalidation and not the switch
+end
+
+Given /^a beetle handler using the redis-master file from "([^\"]*)" exists$/ do |redis_configuration_client_name|
+  master_file = redis_master_file(redis_configuration_client_name)
+  `ruby features/support/beetle_handler start -- --redis-master-file=#{master_file}`
+  assert File.exist?(master_file), "file #{master_file} does not exist"
+end
+
+Given /^redis server "([^\"]*)" is down for less seconds than the retry timeout for the redis master check$/ do |redis_name|
+  TestDaemons::Redis[redis_name].restart(1)
+end
+
+Given /^the retry timeout for the redis master determination is reached$/ do
+  sleep 1
+end
+
+Given /^redis server "([^\"]*)" is coming back$/ do |redis_name|
+  TestDaemons::Redis[redis_name].start
+end
+
+Given /^an old redis master file for "([^\"]*)" with master "([^\"]*)" exists$/ do |redis_configuration_client_name, redis_name|
+  master_file = redis_master_file(redis_configuration_client_name)
+  File.open(master_file, 'w') do |f|
+    f.puts TestDaemons::Redis[redis_name].ip_with_port
+  end
+end
+
+
+Then /^the role of redis server "([^\"]*)" should be "(master|slave)"$/ do |redis_name, role|
+  expected_role = false
+  10.times do
+    expected_role = true and break if TestDaemons::Redis[redis_name].__send__ "#{role}?"
+    sleep 1
+  end
+  assert expected_role, "#{redis_name} is not a #{role}"
+end
+
+Then /^the redis master of "([^\"]*)" should be "([^\"]*)"$/ do |redis_configuration_client_name, redis_name|
+  master_file = redis_master_file(redis_configuration_client_name)
+  master = false
+  server_info = nil
+  10.times do
+    server_info = File.read(master_file).chomp if File.exist?(master_file)
+    master = true and break if TestDaemons::Redis[redis_name].ip_with_port == server_info
+    sleep 1
+  end
+  assert master, "#{redis_name} is not master of #{redis_configuration_client_name}, master file content: #{server_info.inspect}"
+end
+
+Then /^the redis master of "([^\"]*)" should be undefined$/ do |redis_configuration_client_name|
+  master_file = redis_master_file(redis_configuration_client_name)
+  empty = false
+  server_info = nil
+  10.times do
+    server_info = File.read(master_file).chomp if File.exist?(master_file)
+    empty = server_info == ""
+    break if empty
+    sleep 1
+  end
+  assert empty, "master file is not empty: #{server_info}"
+end
+
+Then /^the redis master of the beetle handler should be "([^\"]*)"$/ do |redis_name|
+  Beetle.config.servers = "localhost:5672" # rabbitmq
+  Beetle.config.logger.level = Logger::INFO
+  client = Beetle::Client.new
+  client.register_queue(:echo)
+  client.register_message(:echo)
+  assert_equal TestDaemons::Redis[redis_name].ip_with_port, client.rpc(:echo, 'nil').second
+end
+
+Then /^a system notification for "([^\"]*)" not being available should be sent$/ do |redis_name|
+  text = "Redis master '#{TestDaemons::Redis[redis_name].ip_with_port}' not available"
+  assert_match /#{text}/, File.readlines(system_notification_log_path).last
+end
+
+Then /^a system notification for switching from "([^\"]*)" to "([^\"]*)" should be sent$/ do |old_redis_master_name, new_redis_master_name|
+  text = "Setting redis master to '#{TestDaemons::Redis[new_redis_master_name].ip_with_port}' (was '#{TestDaemons::Redis[old_redis_master_name].ip_with_port}')"
+  assert_match /#{Regexp.escape(text)}/, File.read(system_notification_log_path)
+end
+
+Then /^a system notification for no slave available to become new master should be sent$/ do
+  text = "Redis master could not be switched, no slave available to become new master"
+  assert_match /#{text}/, File.readlines(system_notification_log_path).last
+end

data/features/support/beetle_handler

@@ -0,0 +1,32 @@
+#! /usr/bin/env ruby
+
+require "rubygems"
+require "daemons"
+require "optparse"
+require File.expand_path("../../lib/beetle", File.dirname(__FILE__))
+
+tmp_path = File.expand_path("../../tmp", File.dirname(__FILE__))
+
+Daemons.run_proc("beetle_handler", :log_output => true, :dir_mode => :normal, :dir => tmp_path) do
+  opts = OptionParser.new
+
+  opts.on("-f", "--redis-master-file path", String) do |val|
+    Beetle.config.redis_server = val
+  end
+
+  opts.parse!(ARGV - ["start", "--"])
+
+  Beetle.config.servers = "localhost:5672" # rabbitmq
+
+  # set Beetle log level to info, less noisy than debug
+  Beetle.config.logger.level = Logger::INFO
+
+  client = Beetle::Client.new.configure :auto_delete => true do |config|
+    config.queue(:echo)
+    config.message(:echo)
+    config.handler(:echo) {|message| client.deduplication_store.redis.server rescue "no redis master"}
+  end
+  client.listen do
+    puts "Started beetle handler"
+  end
+end

data/features/support/env.rb

@@ -0,0 +1,48 @@
+require File.expand_path('../../../lib/beetle', __FILE__)
+
+# Allow using Test::Unit for step assertions
+# See http://wiki.github.com/aslakhellesoy/cucumber/using-testunit
+require 'test/unit/assertions'
+World(Test::Unit::Assertions)
+
+Before do
+  `ruby features/support/system_notification_logger start`
+end
+
+After do
+  cleanup_test_env
+end
+
+at_exit do
+  cleanup_test_env
+end
+
+def cleanup_test_env
+  TestDaemons::RedisConfigurationClient.stop_all
+  TestDaemons::RedisConfigurationServer.stop
+
+  `ruby features/support/beetle_handler stop`
+  redis_master_files = tmp_path + "/redis-master-*"
+  `rm -f #{redis_master_files}`
+
+  `ruby features/support/system_notification_logger stop`
+  `rm -f #{system_notification_log_path}`
+
+  TestDaemons::Redis.stop_all
+end
+
+def redis_master_file(client_name)
+  tmp_path + "/redis-master-#{client_name}"
+end
+
+def first_redis_configuration_client_pid
+  File.read("redis_configuration_client0.pid").chomp.to_i
+end
+
+def system_notification_log_path
+  tmp_path + "/system_notifications.log"
+end
+
+def tmp_path
+  File.expand_path("../../../tmp", __FILE__)
+end

data/features/support/system_notification_logger

@@ -0,0 +1,31 @@
+#! /usr/bin/env ruby
+
+require "rubygems"
+require "daemons"
+require File.expand_path("../../lib/beetle", File.dirname(__FILE__))
+
+tmp_path = File.expand_path("../../tmp", File.dirname(__FILE__))
+system_notification_log_file_path = "#{tmp_path}/system_notifications.log"
+
+Daemons.run_proc("system_notification_logger", :log_output => true, :dir_mode => :normal, :dir => tmp_path) do
+  Beetle.config.servers = "localhost:5672, localhost:5673" # rabbitmq
+
+  # set Beetle log level to info, less noisy than debug
+  Beetle.config.logger.level = Logger::DEBUG
+
+  client = Beetle::Client.new
+  client.configure :exchange => :system, :auto_delete => true do |config|
+    config.message  :system_notification
+    config.queue    :system_notification
+    config.handler  :system_notification do |message|
+      payload = ActiveSupport::JSON.decode(message.data)
+      text = payload["message"]
+      puts "Writing message to #{system_notification_log_file_path}: #{text}"
+      File.open(system_notification_log_file_path, "a+") do |f|
+        f << text
+      end
+    end
+  end
+  puts "Started system notification logger"
+  client.listen
+end

data/features/support/test_daemons/redis.conf.erb

@@ -0,0 +1,189 @@
+# Redis configuration file example
+
+# By default Redis does not run as a daemon. Use 'yes' if you need it.
+# Note that Redis will write a pid file in /var/run/redis.pid when daemonized.
+daemonize yes
+
+# When run as a daemon, Redis write a pid file in /var/run/redis.pid by default.
+# You can specify a custom pid file location here.
+pidfile <%= pid_file %>
+
+# Accept connections on the specified port, default is 6379
+port <%= port %>
+
+# If you want you can bind a single interface, if the bind option is not
+# specified all the interfaces will listen for connections.
+#
+# bind 127.0.0.1
+
+# Close the connection after a client is idle for N seconds (0 to disable)
+timeout 300
+
+# Set server verbosity to 'debug'
+# it can be one of:
+# debug (a lot of information, useful for development/testing)
+# notice (moderately verbose, what you want in production probably)
+# warning (only very important / critical messages are logged)
+loglevel debug
+
+# Specify the log file name. Also 'stdout' can be used to force
+# the demon to log on the standard output. Note that if you use standard
+# output for logging but daemonize, logs will be sent to /dev/null
+logfile <%= log_file %>
+
+# Set the number of databases. The default database is DB 0, you can select
+# a different one on a per-connection basis using SELECT <dbid> where
+# dbid is a number between 0 and 'databases'-1
+databases 16
+
+################################ SNAPSHOTTING  #################################
+#
+# Save the DB on disk:
+#
+#   save <seconds> <changes>
+#
+#   Will save the DB if both the given number of seconds and the given
+#   number of write operations against the DB occurred.
+#
+#   In the example below the behaviour will be to save:
+#   after 900 sec (15 min) if at least 1 key changed
+#   after 300 sec (5 min) if at least 10 keys changed
+#   after 60 sec if at least 10000 keys changed
+save 900 1
+save 300 10
+save 60 10000
+
+# Compress string objects using LZF when dump .rdb databases?
+# For default that's set to 'yes' as it's almost always a win.
+# If you want to save some CPU in the saving child set it to 'no' but
+# the dataset will likely be bigger if you have compressible values or keys.
+rdbcompression yes
+
+# The filename where to dump the DB
+dbfilename dump.rdb
+
+# For default save/load DB in/from the working directory
+# Note that you must specify a directory not a file name.
+dir <%= dir %>
+
+################################# REPLICATION #################################
+
+# Master-Slave replication. Use slaveof to make a Redis instance a copy of
+# another Redis server. Note that the configuration is local to the slave
+# so for example it is possible to configure the slave to save the DB with a
+# different interval, or to listen to another port, and so on.
+#
+# slaveof <masterip> <masterport>
+
+# If the master is password protected (using the "requirepass" configuration
+# directive below) it is possible to tell the slave to authenticate before
+# starting the replication synchronization process, otherwise the master will
+# refuse the slave request.
+#
+# masterauth <master-password>
+
+################################## SECURITY ###################################
+
+# Require clients to issue AUTH <PASSWORD> before processing any other
+# commands.  This might be useful in environments in which you do not trust
+# others with access to the host running redis-server.
+#
+# This should stay commented out for backward compatibility and because most
+# people do not need auth (e.g. they run their own servers).
+#
+# requirepass foobared
+
+################################### LIMITS ####################################
+
+# Set the max number of connected clients at the same time. By default there
+# is no limit, and it's up to the number of file descriptors the Redis process
+# is able to open. The special value '0' means no limts.
+# Once the limit is reached Redis will close all the new connections sending
+# an error 'max number of clients reached'.
+#
+# maxclients 128
+
+# Don't use more memory than the specified amount of bytes.
+# When the memory limit is reached Redis will try to remove keys with an
+# EXPIRE set. It will try to start freeing keys that are going to expire
+# in little time and preserve keys with a longer time to live.
+# Redis will also try to remove objects from free lists if possible.
+#
+# If all this fails, Redis will start to reply with errors to commands
+# that will use more memory, like SET, LPUSH, and so on, and will continue
+# to reply to most read-only commands like GET.
+#
+# WARNING: maxmemory can be a good idea mainly if you want to use Redis as a
+# 'state' server or cache, not as a real DB. When Redis is used as a real
+# database the memory usage will grow over the weeks, it will be obvious if
+# it is going to use too much memory in the long run, and you'll have the time
+# to upgrade. With maxmemory after the limit is reached you'll start to get
+# errors for write operations, and this may even lead to DB inconsistency.
+#
+# maxmemory <bytes>
+
+############################## APPEND ONLY MODE ###############################
+
+# By default Redis asynchronously dumps the dataset on disk. If you can live
+# with the idea that the latest records will be lost if something like a crash
+# happens this is the preferred way to run Redis. If instead you care a lot
+# about your data and don't want to that a single record can get lost you should
+# enable the append only mode: when this mode is enabled Redis will append
+# every write operation received in the file appendonly.log. This file will
+# be read on startup in order to rebuild the full dataset in memory.
+#
+# Note that you can have both the async dumps and the append only file if you
+# like (you have to comment the "save" statements above to disable the dumps).
+# Still if append only mode is enabled Redis will load the data from the
+# log file at startup ignoring the dump.rdb file.
+#
+# The name of the append only file is "appendonly.log"
+#
+# IMPORTANT: Check the BGREWRITEAOF to check how to rewrite the append
+# log file in background when it gets too big.
+
+appendonly yes
+
+# The fsync() call tells the Operating System to actually write data on disk
+# instead to wait for more data in the output buffer. Some OS will really flush 
+# data on disk, some other OS will just try to do it ASAP.
+#
+# Redis supports three different modes:
+#
+# no: don't fsync, just let the OS flush the data when it wants. Faster.
+# always: fsync after every write to the append only log . Slow, Safest.
+# everysec: fsync only if one second passed since the last fsync. Compromise.
+#
+# The default is "always" that's the safer of the options. It's up to you to
+# understand if you can relax this to "everysec" that will fsync every second
+# or to "no" that will let the operating system flush the output buffer when
+# it want, for better performances (but if you can live with the idea of
+# some data loss consider the default persistence mode that's snapshotting).
+
+# appendfsync always
+appendfsync everysec
+# appendfsync no
+
+############################### ADVANCED CONFIG ###############################
+
+# Glue small output buffers together in order to send small replies in a
+# single TCP packet. Uses a bit more CPU but most of the times it is a win
+# in terms of number of queries per second. Use 'yes' if unsure.
+glueoutputbuf yes
+
+# Use object sharing. Can save a lot of memory if you have many common
+# string in your dataset, but performs lookups against the shared objects
+# pool so it uses more CPU and can be a bit slower. Usually it's a good
+# idea.
+#
+# When object sharing is enabled (shareobjects yes) you can use
+# shareobjectspoolsize to control the size of the pool used in order to try
+# object sharing. A bigger pool size will lead to better sharing capabilities.
+# In general you want this value to be at least the double of the number of
+# very common strings you have in your dataset.
+#
+# WARNING: object sharing is experimental, don't enable this feature
+# in production before of Redis 1.0-stable. Still please try this feature in
+# your development environment so that we can test it better.
+shareobjects no
+shareobjectspoolsize 1024

data/features/support/test_daemons/redis.rb

@@ -0,0 +1,186 @@
+require 'rubygems'
+require 'fileutils'
+require 'erb'
+require 'redis'
+require 'lib/beetle/redis_ext'
+require 'daemon_controller'
+
+module TestDaemons
+  class Redis
+
+    @@instances = {}
+    @@next_available_port = 6381
+
+    attr_reader :name, :port
+
+    def initialize(name)
+      @name = name
+      @port = @@next_available_port
+      
+      @@next_available_port += 1
+      @@instances[name] = self
+    end
+
+    class << self
+      def find_or_initialize_by_name(name)
+        @@instances[name] ||= new(name)
+      end
+      alias_method :[], :find_or_initialize_by_name
+
+      def stop_all
+        @@instances.values.each{|i| i.stop}
+      end
+    end
+
+    def start
+      create_dir
+      create_config
+      daemon_controller.start
+    end
+
+    def restart(delay=1)
+      redis.shutdown rescue Errno::ECONNREFUSED
+      sleep delay
+      `redis-server #{config_filename}`
+    end
+
+    def stop
+      # TODO: Might need to be moved into RedisConfigurationServer
+      #     10.times do
+      #       break if (redis.info["bgsave_in_progress"]) == 0 rescue false
+      #       sleep 1
+      #     end
+      daemon_controller.stop
+    ensure
+      cleanup
+    end
+
+    def cleanup
+      remove_dir
+      remove_config
+      remove_pid_file
+    end
+
+    # TODO: The retry logic must be moved into RedisConfigurationServer
+    def master
+      tries = 0
+      begin
+        redis.master!
+      rescue Errno::ECONNREFUSED, Errno::EAGAIN
+        puts "master role setting for #{name} failed: #{$!}"
+        sleep 1
+        retry if (tries+=1) > 5
+        raise "could not setup master #{name} #{$!}"
+      end
+    end
+
+    def running?
+      cmd = "ps aux | grep 'redis-server #{config_filename}' | grep -v grep"
+      res = `#{cmd}`
+      x = res.chomp.split("\n")
+      x.size == 1
+    end
+
+    def available?
+      redis.available?
+    end
+
+    def master?
+      redis.master?
+    end
+
+    def slave?
+      redis.slave?
+    end
+
+    # TODO: Move to redis_ext
+    def slave_of(master_port)
+      tries = 0
+      begin
+        redis.slave_of!("127.0.0.1", master_port)
+      rescue Errno::ECONNREFUSED, Errno::EAGAIN
+        puts "slave role setting for #{name} failed: #{$!}"
+        sleep 1
+        retry if (tries+=1) > 5
+        raise "could not setup slave #{name}: #{$!}"
+      end
+    end
+
+    def ip_with_port
+      "127.0.0.1:#{port}"
+    end
+  
+    def redis
+      @redis ||= ::Redis.new(:host => "127.0.0.1", :port => port)
+    end
+
+    private
+
+    def create_dir
+      FileUtils.mkdir(dir) unless File.exists?(dir)
+    end
+
+    def remove_dir
+      FileUtils.rm_r(dir) if File.exists?(dir)
+    end
+
+    def create_config
+      File.open(config_filename, "w") do |file|
+        file.puts config_content
+      end
+    end
+
+    def remove_config
+      FileUtils.rm(config_filename) if File.exists?(config_filename)
+    end
+
+    def remove_pid_file
+      FileUtils.rm(pid_file) if File.exists?(pid_file)
+    end
+
+    def tmp_path
+      File.expand_path(File.dirname(__FILE__) + "/../../../tmp")
+    end
+
+    def config_filename
+      tmp_path + "/redis-test-server-#{name}.conf"
+    end
+
+    def config_content
+      template = ERB.new(File.read(config_template_filename))
+      template.result(binding)
+    end
+
+    def config_template_filename
+      File.dirname(__FILE__) + "/redis.conf.erb"
+    end
+
+    def pid_file
+      tmp_path + "/redis-test-server-#{name}.pid"
+    end
+
+    def pid
+      File.read(pid_file).chomp.to_i
+    end
+
+    def log_file
+      tmp_path + "/redis-test-server-#{name}.log"
+    end
+
+    def dir
+      tmp_path + "/redis-test-server-#{name}/"
+    end
+
+    def daemon_controller
+      @daemon_controller ||= DaemonController.new(
+         :identifier    => "Redis test server",
+         :start_command => "redis-server #{config_filename}",
+         :ping_command  => lambda { running? && available? },
+         :pid_file      => pid_file,
+         :log_file      => log_file,
+         :start_timeout => 5
+      )
+    end
+  
+  end
+end

data/features/support/test_daemons/redis_configuration_client.rb

@@ -0,0 +1,64 @@
+require 'daemon_controller'
+
+module TestDaemons
+  class RedisConfigurationClient
+    cattr_reader :instances
+    @@next_daemon_id = 0
+    @@instances = {}
+
+    def initialize(name)
+      @name = name
+      @daemon_id = @@next_daemon_id
+
+      @@next_daemon_id += 1
+      @@instances[name] = self
+    end
+
+    class << self
+      def find_or_initialize_by_name(name)
+        @@instances[name] ||= new(name)
+      end
+      alias_method :[], :find_or_initialize_by_name
+
+      def stop_all
+        @@instances.values.each{|i| i.stop}
+      end
+    end
+
+    def start
+      daemon_controller.start
+    end
+
+    def stop
+      daemon_controller.stop
+    end
+
+    def daemon_controller
+      @daemon_controller ||= DaemonController.new(
+         :identifier    => "Redis configuration test client",
+         :start_command => "ruby bin/beetle configuration_client start -- -v --redis-master-file #{redis_master_file} --id #{@name} --pid-dir #{tmp_path} --amqp-servers 127.0.0.1:5672",
+         :ping_command  => lambda{ true },
+         :pid_file      => pid_file,
+         :log_file      => log_file,
+         :start_timeout => 5
+      )
+    end
+
+    def redis_master_file
+      "#{tmp_path}/redis-master-#{@name}"
+    end
+
+    def pid_file
+      "#{tmp_path}/redis_configuration_client#{@daemon_id}.pid"
+    end
+
+    def log_file
+      "#{tmp_path}/redis_configuration_client.output"
+    end
+
+    def tmp_path
+      File.expand_path(File.dirname(__FILE__) + "/../../../tmp")
+    end
+
+  end
+end

data/features/support/test_daemons/redis_configuration_server.rb

@@ -0,0 +1,52 @@
+require 'daemon_controller'
+
+module TestDaemons
+  class RedisConfigurationServer
+
+    # At the moment, we need only one, so we implement the methods
+    # as class methods
+
+    @@redis_servers = ""
+    @@redis_configuration_clients = ""
+
+    def self.start(redis_servers, redis_configuration_clients)
+      stop
+      @@redis_servers = redis_servers
+      @@redis_configuration_clients = redis_configuration_clients
+      daemon_controller.start
+    end
+
+    def self.stop
+      daemon_controller.stop
+    end
+
+    def self.daemon_controller
+      clients_parameter_string = @@redis_configuration_clients.blank? ? "" : "--client-ids #{@@redis_configuration_clients}"
+      DaemonController.new(
+         :identifier    => "Redis configuration test server",
+         :start_command => "ruby bin/beetle configuration_server start -- -v --redis-master-file #{redis_master_file} --redis-servers #{@@redis_servers} #{clients_parameter_string} --redis-retry-interval 1 --pid-dir #{tmp_path} --amqp-servers 127.0.0.1:5672",
+         :ping_command  => lambda{ true },
+         :pid_file      => pid_file,
+         :log_file      => log_file,
+         :start_timeout => 5
+      )
+    end
+
+    def self.redis_master_file
+      "#{tmp_path}/redis-master-rc-server"
+    end
+
+    def self.pid_file
+      "#{tmp_path}/redis_configuration_server.pid"
+    end
+
+    def self.log_file
+      "#{tmp_path}/redis_configuration_server.output"
+    end
+
+    def self.tmp_path
+      File.expand_path(File.dirname(__FILE__) + "/../../../tmp")
+    end
+
+  end
+end

data/script/console

@@ -0,0 +1,28 @@
+#!/usr/bin/env ruby
+require 'rubygems'
+require 'optparse'
+
+options = { }
+OptionParser.new do |opt|
+  opt.banner = "Usage: console [options]"
+  opt.on("--debugger", 'Enable ruby-debugging for the console.') { |v| options[:debugger] = v }
+  opt.parse!(ARGV)
+end
+
+libs =  " -r irb/completion"
+libs << %( -r ubygems)
+libs << %( -r #{File.expand_path("../../lib/beetle.rb",__FILE__)})
+
+if options[:debugger]
+  begin
+    require 'ruby-debug'
+    libs << " -r ruby-debug"
+    puts "=> Debugger enabled"
+  rescue Exception
+    puts "You need to install ruby-debug to run the console in debugging mode. With gems, use 'gem install ruby-debug'"
+    exit
+  end
+end
+
+irb = RUBY_PLATFORM =~ /(:?mswin|mingw)/ ? 'irb.bat' : 'irb'
+exec "#{irb} #{libs} --simple-prompt"

data/script/start_rabbit

@@ -0,0 +1,29 @@
+#!/bin/bash
+
+# export RABBITMQ_MNESIA_BASE=/var/lib/rabbitmq/mnesia2
+# Defaults to /var/lib/rabbitmq/mnesia. Set this to the directory where Mnesia
+# database files should be placed.
+
+# export RABBITMQ_LOG_BASE
+# Defaults to /var/log/rabbitmq. Log files generated by the server will be placed
+# in this directory.
+
+export RABBITMQ_NODENAME=$1
+# Defaults to rabbit. This can be useful if you want to run more than one node
+# per machine - RABBITMQ_NODENAME should be unique per erlang-node-and-machine
+# combination. See clustering on a single machine guide at <http://www.rab-
+# bitmq.com/clustering.html#single-machine> for details.
+
+# RABBITMQ_NODE_IP_ADDRESS
+# Defaults to 0.0.0.0. This can be changed if you only want to bind to one net-
+# work interface.
+
+export RABBITMQ_NODE_PORT=$2
+# Defaults to 5672.
+
+# RABBITMQ_CLUSTER_CONFIG_FILE
+# Defaults to /etc/rabbitmq/rabbitmq_cluster.config. If this file is present it
+# is used by the server to auto-configure a RabbitMQ cluster.  See the clustering
+# guide at <http://www.rabbitmq.com/clustering.html> for details.
+
+rabbitmq-server

data/test/beetle/deduplication_store_test.rb

@@ -73,7 +73,9 @@ module Beetle
 
     private
     def redis_test_master_file(server_string)
-      path = File.expand_path("../../../tmp/redis-master-for-unit-tests", __FILE__)
+      tmp_dir = File.expand_path("../../../tmp", __FILE__)
+      Dir.mkdir(tmp_dir) unless File.exists?(tmp_dir)
+      path = tmp_dir + "/redis-master-for-unit-tests"
       File.open(path, "w"){|f| f.puts server_string}
       path
     end

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: beetle
 version: !ruby/object:Gem::Version 
-  hash: 31
+  hash: 29
   prerelease: false
   segments: 
   - 0
   - 2
-  - 4
-  version: 0.2.4
+  - 5
+  version: 0.2.5
 platform: ruby
 authors: 
 - Stefan Kaes
@@ -18,7 +18,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2010-08-06 00:00:00 +02:00
+date: 2010-08-23 00:00:00 +02:00
 default_executable: beetle
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -182,7 +182,12 @@ executables:
 extensions: 
 - ext/mkrf_conf.rb
 extra_rdoc_files: 
+- examples/README.rdoc
+- features/README.rdoc
 - README.rdoc
+- REDIS_AUTO_FAILOVER.rdoc
+- RELEASE_NOTES.rdoc
+- MIT-LICENSE
 files: 
 - examples/attempts.rb
 - examples/handler_class.rb
@@ -212,9 +217,25 @@ files:
 - lib/beetle/redis_server_info.rb
 - lib/beetle/subscriber.rb
 - lib/beetle.rb
+- features/README.rdoc
+- features/redis_auto_failover.feature
+- features/step_definitions/redis_auto_failover_steps.rb
+- features/support/beetle_handler
+- features/support/env.rb
+- features/support/system_notification_logger
+- features/support/test_daemons/redis.conf.erb
+- features/support/test_daemons/redis.rb
+- features/support/test_daemons/redis_configuration_client.rb
+- features/support/test_daemons/redis_configuration_server.rb
+- script/console
+- script/start_rabbit
 - beetle.gemspec
+- Rakefile
 - examples/README.rdoc
 - README.rdoc
+- REDIS_AUTO_FAILOVER.rdoc
+- RELEASE_NOTES.rdoc
+- MIT-LICENSE
 - test/beetle/base_test.rb
 - test/beetle/client_test.rb
 - test/beetle/configuration_test.rb