bigwig 0.2

data/CHANGELOG

@@ -0,0 +1,3 @@
+v0.2. added bigwig-ping command to place ping messages on the queue
+
+v0.1. first public release

data/LICENSE

@@ -0,0 +1,11 @@
+Released under the MIT License
+ 
+Copyright (c) 2009 Brightbox Systems Ltd
+ 
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+ 
+* The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+ 
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+ 
+See http://www.brightbox.co.uk/ for contact details.

data/Manifest

@@ -0,0 +1,16 @@
+CHANGELOG
+LICENSE
+Manifest
+README.markdown
+Rakefile
+bigwig.yml
+bigwig.yml.example
+bin/bigwig
+bin/bigwig-ping
+lib/bigwig.rb
+lib/bigwig/internal_plugins/ping_plugin.rb
+lib/bigwig/job.rb
+lib/bigwig/pinger.rb
+lib/bigwig/plugin.rb
+lib/bigwig/plugins.rb
+lib/bigwig/runner.rb

data/README.markdown

@@ -0,0 +1,92 @@
+Bigwig
+======
+
+Bigwig is a daemon process that listens to a RabbitMQ queue and processes messages that it receives.  
+
+Typical Usage
+-------------
+
+* Install the bigwig gem
+* Start an AMQP server
+* Create a folder of plugins
+* Create a bigwig configuration file
+* Start bigwig - `bigwig start -c config -l logs`
+
+Configuration
+-------------
+
+Bigwig expects you to give it a configuration file detailing how it is to connect to the AMQP server and how it should respond to incoming messages.  
+
+When invoking bigwig, it looks for a file called bigwig.yml in the current folder.  Alternatively you can specify the configuration file using the `-c` command line option (note that you must supply an absolute path to the file, relative paths will fail).  
+
+A typical configuration file looks like this: 
+
+        user: rabbit-user
+        password: crunchycarrots
+        vhost: /
+        server: my.amqpserver.com
+        port: 5672
+        queue: myqueue
+        warren_logging: false
+        plugins_folder: /full/path/to/a/folder
+
+The first six items tell bigwig how to connect to the AMQP server.  The next item specifies whether you want warren (the lower-level AMQP processor) to log its output.  Lastly, you tell bigwig where to find its plugins.  
+
+Logging
+-------
+
+When invoking bigwig you should pass it the full path to a folder in which to write its log and pid files.  If not specified, it will use the current folder.  This is done using the `-l` command line option.  
+
+Messages
+--------
+
+Bigwig expects messages to be a string which can be deserialised into a Ruby Hash.  The Hash is split into two sections - the "header" and the "data".  
+
+A typical message will look something like this: 
+
+        message = { 
+          :id => '123', 
+          :method => 'my_command', 
+          :data => { 
+            :field => 'value', 
+            :another_field => 'another_value'
+          }
+        }
+
+The most important of these is the `:method` key.  Bigwig unpacks this and uses that to lookup which plugin it should invoke (see below).  
+
+Plugins
+-------
+
+When bigwig starts it looks through the plugins folder specified within the configuration file (and all its sub-directories).  It then loads all plugins that it finds.  
+
+For example, a plugin, which we shall call LoggingPlugin, would look like this: 
+
+* it should live in a file called `logging_plugin.rb` that is somewhere within your plugins folder
+* it should define a class called `LoggingPlugin` that descends from `BigWig::Plugin`
+* it should define a class method called `method`
+* it should define a class method called `call`
+
+        class LoggingPlugin < BigWig::Plugin
+          def self.method
+            "logging"
+          end
+          
+          def self.call(task_id, args)
+            BigWig.logger.info "LoggingPlugin was invoked with #{args.inspect} as parameters"
+          end
+        end
+
+When bigwig receives a message where the `:method` parameter == 'logging' it will then invoke `LoggingPlugin#call`, passing it the `:id` and `:data` from the original message as task_id and args respectively.  
+
+Pings
+-----
+
+There is a plugin built-in to bigwig called PingPlugin, that registers itself under the name "ping".  If you place a message onto the queue with :method => 'ping', the PingPlugin responds by writing a message to the log file.  This is useful for monitoring bigwig - another system places ping messages onto the queue at regular intervals and we watch to ensure that the log file's update time is changing.  
+
+License
+-------
+
+(c) 2009 Brightbox Systems Ltd.  Released under the MIT License - see LICENSE for more details.  
+
+

data/Rakefile

@@ -0,0 +1,8 @@
+require 'echoe'
+Echoe.new 'bigwig' do | gem | 
+  gem.author = 'David Smalley, Caius Durling, Rahoul Baruah'
+  gem.summary = 'A daemon that listens to an AMQP queue and responds to messages by invoking commands from a set of plugins'
+  gem.url = 'http://www.brightbox.co.uk/'
+  gem.runtime_dependencies = ["brightbox-warren >=0.9.0", "daemons >=1.0.10"]
+  gem.install_message = 'Welcome to bigwig.  Please set up a configuration file and a plugins folder before starting bigwig...'
+end

data/bigwig.gemspec

@@ -0,0 +1,38 @@
+# -*- encoding: utf-8 -*-
+
+Gem::Specification.new do |s|
+  s.name = %q{bigwig}
+  s.version = "0.2"
+
+  s.required_rubygems_version = Gem::Requirement.new(">= 1.2") if s.respond_to? :required_rubygems_version=
+  s.authors = ["David Smalley, Caius Durling, Rahoul Baruah"]
+  s.date = %q{2009-10-06}
+  s.description = %q{A daemon that listens to an AMQP queue and responds to messages by invoking commands from a set of plugins}
+  s.email = %q{}
+  s.executables = ["bigwig", "bigwig-ping"]
+  s.extra_rdoc_files = ["CHANGELOG", "LICENSE", "README.markdown", "bin/bigwig", "bin/bigwig-ping", "lib/bigwig.rb", "lib/bigwig/internal_plugins/ping_plugin.rb", "lib/bigwig/job.rb", "lib/bigwig/pinger.rb", "lib/bigwig/plugin.rb", "lib/bigwig/plugins.rb", "lib/bigwig/runner.rb"]
+  s.files = ["CHANGELOG", "LICENSE", "Manifest", "README.markdown", "Rakefile", "bigwig.yml", "bigwig.yml.example", "bin/bigwig", "bin/bigwig-ping", "lib/bigwig.rb", "lib/bigwig/internal_plugins/ping_plugin.rb", "lib/bigwig/job.rb", "lib/bigwig/pinger.rb", "lib/bigwig/plugin.rb", "lib/bigwig/plugins.rb", "lib/bigwig/runner.rb", "bigwig.gemspec"]
+  s.homepage = %q{http://www.brightbox.co.uk/}
+  s.post_install_message = %q{Welcome to bigwig.  Please set up a configuration file and a plugins folder before starting bigwig...}
+  s.rdoc_options = ["--line-numbers", "--inline-source", "--title", "Bigwig", "--main", "README.markdown"]
+  s.require_paths = ["lib"]
+  s.rubyforge_project = %q{bigwig}
+  s.rubygems_version = %q{1.3.5}
+  s.summary = %q{A daemon that listens to an AMQP queue and responds to messages by invoking commands from a set of plugins}
+
+  if s.respond_to? :specification_version then
+    current_version = Gem::Specification::CURRENT_SPECIFICATION_VERSION
+    s.specification_version = 3
+
+    if Gem::Version.new(Gem::RubyGemsVersion) >= Gem::Version.new('1.2.0') then
+      s.add_runtime_dependency(%q<brightbox-warren>, [">= 0.9.0"])
+      s.add_runtime_dependency(%q<daemons>, [">= 1.0.10"])
+    else
+      s.add_dependency(%q<brightbox-warren>, [">= 0.9.0"])
+      s.add_dependency(%q<daemons>, [">= 1.0.10"])
+    end
+  else
+    s.add_dependency(%q<brightbox-warren>, [">= 0.9.0"])
+    s.add_dependency(%q<daemons>, [">= 1.0.10"])
+  end
+end

data/bigwig.yml

@@ -0,0 +1,8 @@
+user: guest
+password: guest
+vhost: /
+server: localhost
+port: 5672
+queue: myqueue
+warren_logging: false
+plugins_folder: /Volumes/bb/bigwig-plugins

data/bigwig.yml.example

@@ -0,0 +1,8 @@
+user: rabbit-user
+password: crunchycarrots
+vhost: /
+server: my.amqpserver.com
+port: 5672
+queue: myqueue
+warren_logging: false
+plugins_folder: /full/path/to/a/folder

data/bin/bigwig

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+require File.join(File.dirname(__FILE__), "..", "lib", "bigwig")
+BigWig::Runner.new(ARGV).run

data/bin/bigwig-ping

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+require File.join(File.dirname(__FILE__), "..", "lib", "bigwig")
+BigWig::Pinger.new(ARGV).run

data/lib/bigwig.rb

@@ -0,0 +1,33 @@
+require 'rubygems'
+
+BIGWIG_ROOT = File.expand_path(File.join(File.dirname(__FILE__), ".."))
+
+module BigWig
+  require 'logger'
+
+  def self.logger
+    @logger ||= begin
+      l = Logger.new(STDERR)
+      l.level = Logger::DEBUG
+      l
+    end
+  end
+
+  def self.logger=(l)
+    @logger = l
+  end
+  
+  def self.connect_using config
+    env = ENV['WARREN_ENV'] || 'development'
+    
+    h = {:user => config["user"], :pass => config["password"], :vhost => config["vhost"], :default_queue => config["queue"], :host => config["server"], :logging => config["warren_logging"]}
+  
+    params = { env => h }
+    
+    Warren::Queue.logger = BigWig::logger
+    Warren::Queue.connection = params
+  end
+end
+
+Dir["#{BIGWIG_ROOT}/lib/bigwig/*.rb"].each {|r| require r }
+Dir["#{BIGWIG_ROOT}/lib/bigwig/internal_plugins/*.rb"].each {|r| require r }

data/lib/bigwig/internal_plugins/ping_plugin.rb

@@ -0,0 +1,11 @@
+class PingPlugin < BigWig::Plugin
+
+  def self.method
+    "ping"
+  end
+  
+  def self.call(task_id, args)
+    BigWig.logger.info "#{Time.now.to_s}: ping received"
+  end
+
+end

data/lib/bigwig/job.rb

@@ -0,0 +1,48 @@
+module BigWig
+  require 'benchmark'
+  # Dispatches jobs as they are received from the queue to the relevant plugin.  
+  # Each message on the queue is expected to take a form similar to: 
+  #   {
+  #     :method => 'my_method_name', 
+  #     :id => 'optional arbitrary task identifier', 
+  #     :data => { :hash => :of, :relevant => :data }
+  #   }
+  # The dispatch message asks for the Plugin that knows how to deal with the given method name and invokes it with the relevant id and data
+  class Job
+    
+    # Dispatch the given message to the correct plugin
+    def self.dispatch(msg)
+      new(msg).run
+    end
+    
+    def initialize(msg)
+      @method = msg[:method]
+      @task_id = msg[:id]
+      @data = msg[:data] || {}
+    end
+    
+    # Invoke the plugin with the given data and task id
+    def run
+      jobid = @task_id || rand(0xfffffff).to_s(16).upcase
+      BigWig::logger.info("Received #{@method} (job #{jobid}) with #{@data.inspect}")
+      
+      result = nil
+      benchmark = begin
+        Benchmark.measure do
+          result = plugin_for(@method).call(@task_id, @data) 
+        end
+      rescue StandardError => e
+        BigWig::logger.error("Bigwig Job id #{jobid} failed with exception #{e}: #{e.message}")
+        raise e
+      end
+
+      BigWig::logger.info("Bigwig Job id #{jobid} completed in #{benchmark.real.round} seconds")
+      return result
+    end
+    
+  private
+    def plugin_for(method)
+      BigWig::Plugins.plugin_for(method)
+    end
+  end
+end

data/lib/bigwig/pinger.rb

@@ -0,0 +1,60 @@
+require 'warren'
+require 'warren/adapters/bunny_adapter'
+require 'daemons'
+require 'optparse'
+require 'ostruct'
+require 'benchmark'
+require 'timeout'
+
+module BigWig
+  class Pinger
+    attr_reader :config
+    attr_reader :options
+    
+    def initialize(command_line)
+      load_options_from command_line
+    end
+    
+    def run
+      BigWig::logger.info("Starting bigwig pinger...")
+      @config = YAML.load(File.open(@options.config))
+      BigWig::connect_using @config
+
+      begin
+        Timeout::timeout(@options.timeout) do
+          begin
+            Warren::Queue.publish :default, :method => 'ping'
+          rescue Exception => ex
+            BigWig::logger.error("...ping failed with #{ex}")
+            exit 2
+          end
+        end
+      rescue Timeout::Error => te
+        BigWig::logger.error("...ping timed out: #{te}")
+        exit 1
+      end
+
+      BigWig::logger.info("...bigwig pinger finished")
+      exit 0
+    end
+    
+  protected
+    def load_options_from command_line
+      @options = OpenStruct.new(:config => File.expand_path('./bigwig.yml'), :timeout => 5) 
+      opts = OptionParser.new do |opts|
+        opts.banner = "Usage: #{File.basename($0)} [options]"
+        opts.on('-h', '--help', 'Show this message') do
+          puts opts
+          exit 1
+        end
+        opts.on('-c', '--config=file', 'Path to the config file; if not supplied then bigwig looks for bigwig.yml in the current folder') do |conf|
+          @options.config = conf
+        end
+        opts.on('-t', '--timeout=value', 'Timeout value in seconds') do | conf | 
+          @options.timeout = conf.to_i
+        end 
+      end
+      @args = opts.parse!(command_line)
+    end
+  end
+end

data/lib/bigwig/plugin.rb

@@ -0,0 +1,16 @@
+module BigWig
+  class Plugin
+    # Each plugin should override Plugin::method and return the name of the command that 
+    def self.method
+      raise "Plugins must override Plugin::method"
+    end
+
+    def self.register
+      { self.method => self }
+    end
+    
+    def self.call(task_id, args)
+      BigWig.logger.warn "NotImplemented: Called #{self} with #{args} for task #{task_id}"
+    end
+  end
+end

data/lib/bigwig/plugins.rb

@@ -0,0 +1,59 @@
+module BigWig
+  # loads all the plugins from the BigWig::Plugins.root folder
+  class Plugins 
+    class << self
+      attr_accessor :root
+      
+      # load all plugins in the BigWig::Plugins.root folder
+      def load_all
+        register_plugins
+      end
+      
+      # find the plugin with the given name
+      def plugin_for(key)
+        PLUGINS[key] || BigWig::Plugin
+      end
+      
+    private
+      PLUGINS = {} unless defined? PLUGINS
+
+      # list of all known plugin files
+      def all_plugins
+        Dir["#{Plugins.root}/**/*_plugin.rb"]
+      end
+
+      # go through each plugin and register it
+      def register_plugins
+        all_plugins.each do |plugin|
+          BigWig.logger.info "Registering plugin: #{plugin}"
+          require plugin
+          plugin_class = constantize(camelize(File.basename(plugin, ".rb")))
+          PLUGINS.merge!(plugin_class.register)
+        end
+        # and add the inbuilt Ping plugin
+        PLUGINS.merge!(PingPlugin.register)
+        PLUGINS.freeze
+      end
+
+      # Helper method stolen from Rails :)
+      def constantize(camel_cased_word)
+        names = camel_cased_word.split('::')
+        names.shift if names.empty? || names.first.empty?
+        constant = Object
+        names.each do |name|
+         constant = constant.const_defined?(name) ? constant.const_get(name) : constant.const_missing(name)
+        end
+        constant
+      end
+      
+      # Helper method stolen from Rails :)
+      def camelize(lower_case_and_underscored_word, first_letter_in_uppercase = true)
+        if first_letter_in_uppercase
+          lower_case_and_underscored_word.to_s.gsub(/\/(.?)/) { "::#{$1.upcase}" }.gsub(/(?:^|_)(.)/) { $1.upcase }
+        else
+          lower_case_and_underscored_word.first.downcase + camelize(lower_case_and_underscored_word)[1..-1]
+        end
+      end
+    end
+  end
+end

data/lib/bigwig/runner.rb

@@ -0,0 +1,119 @@
+require 'warren'
+require 'warren/adapters/bunny_adapter'
+require 'daemons'
+require 'optparse'
+require 'ostruct'
+require 'benchmark'
+
+module BigWig
+  class Runner
+    attr_reader :args
+    attr_reader :options
+    attr_reader :config
+    
+    # start the bigwig process
+    # use a default config file of bigwig.yml in the current folder, unless one is provided on the command line
+    def initialize(command_line_options)
+      load_options_from command_line_options
+    end
+  
+    def run
+      Daemons.run_proc('bigwig', :dir => @options.log_folder, :dir_mode => :normal, :log_output => true,  :ARGV => self.args) do |*args|
+        @config = YAML.load(File.open(@options.config))
+        load_plugins_from @config["plugins_folder"]
+        begin
+          BigWig::logger.info("Starting Bigwig job worker")
+
+          trap_signals
+          BigWig::connect_using config
+
+          last_successful_subscription = Time.now
+          last_connection_failed = false
+          loop do
+            message_received = false
+            break if $exit 
+
+            begin
+              Warren::Queue.subscribe(:default) do |msg|
+                BigWig::Job.dispatch(msg)
+                message_received = true
+              end
+              
+              last_successful_subscription = Time.now # record the time in case we have a failure next time round
+              if last_connection_failed
+                BigWig.logger.info('...reconnected successfully')
+                last_connection_failed = false
+              end
+              sleep(0.5) unless message_received # if the queue was empty, go to sleep before checking for the next message
+              
+            rescue Exception => ex # if there was an error connecting to the queue, wait a bit and then retry TODO: Change this exception
+              BigWig::logger.error("Error when subscribing to the queue: #{ex}")
+              retry_time = calculate_retry_time_based_upon last_successful_subscription
+              sleep(retry_time)
+              last_connection_failed = true
+            end
+            
+            break if $exit
+          end
+
+          BigWig::logger.info("Bigwig job worker stopped")
+        rescue => e
+          BigWig::logger.fatal("Exiting due to exception #{e}, #{e.message}")
+          exit 1
+        end
+      end
+    end
+
+  private
+    def load_options_from command_line_options
+      @options = OpenStruct.new(:config => File.expand_path('./bigwig.yml'), :log_folder => File.expand_path('.')) 
+      opts = OptionParser.new do |opts|
+        opts.banner = "Usage: #{File.basename($0)} [options] start|stop|restart|run"
+        opts.on('-h', '--help', 'Show this message') do
+          puts opts
+          exit 1
+        end
+        opts.on('-c', '--config=file', 'Path to the config file; if not supplied then bigwig looks for bigwig.yml in the current folder') do |conf|
+          @options.config = conf
+        end
+        opts.on('-l', '--log=folder', 'Path to a folder for the log file; if not supplied then bigwig will write its PID and log to the current folder') do |conf|
+          @options.log_folder = conf
+        end
+      end
+      @args = opts.parse!(command_line_options)
+    end
+    
+    def load_plugins_from folder
+      BigWig::Plugins.root = folder
+      BigWig::Plugins.load_all
+    end
+    
+    def trap_signals
+      trap('TERM') do
+        BigWig::logger.info('Bigwig exiting due to TERM signal')
+        $exit = true
+      end
+      trap('INT') do
+        BigWig::logger.info('Bigwig exiting due to INT signal')
+        $exit = true
+      end
+    end
+    
+    def calculate_retry_time_based_upon last_connection_time
+      seconds_since_last_connection = (Time.now - last_connection_time).to_i
+      minutes = (seconds_since_last_connection.to_f / 60.0)
+      minutes_squared = minutes * minutes
+      
+      # use a quadratic equation seconds_till_retry = a.x.x + b.x + c
+      # where x is the time in minutes since last connection
+      scale_down_factor = 0.5 # a in our equation
+      scale_movement_factor = 5 # b in our equation
+      constant_factor = 20.0 # c in our equation
+      
+      seconds_till_retry = (minutes_squared * scale_down_factor) + (minutes * scale_movement_factor) + constant_factor
+      BigWig::logger.info("#{minutes.to_i} minutes since last successful connection: retrying in #{seconds_till_retry.to_i} seconds...")
+      
+      return seconds_till_retry.to_i
+    end
+  end
+end

metadata

@@ -0,0 +1,107 @@
+--- !ruby/object:Gem::Specification 
+name: bigwig
+version: !ruby/object:Gem::Version 
+  version: "0.2"
+platform: ruby
+authors: 
+- David Smalley, Caius Durling, Rahoul Baruah
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-10-06 00:00:00 +01:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: brightbox-warren
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 0.9.0
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: daemons
+  type: :runtime
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.0.10
+    version: 
+description: A daemon that listens to an AMQP queue and responds to messages by invoking commands from a set of plugins
+email: ""
+executables: 
+- bigwig
+- bigwig-ping
+extensions: []
+
+extra_rdoc_files: 
+- CHANGELOG
+- LICENSE
+- README.markdown
+- bin/bigwig
+- bin/bigwig-ping
+- lib/bigwig.rb
+- lib/bigwig/internal_plugins/ping_plugin.rb
+- lib/bigwig/job.rb
+- lib/bigwig/pinger.rb
+- lib/bigwig/plugin.rb
+- lib/bigwig/plugins.rb
+- lib/bigwig/runner.rb
+files: 
+- CHANGELOG
+- LICENSE
+- Manifest
+- README.markdown
+- Rakefile
+- bigwig.yml
+- bigwig.yml.example
+- bin/bigwig
+- bin/bigwig-ping
+- lib/bigwig.rb
+- lib/bigwig/internal_plugins/ping_plugin.rb
+- lib/bigwig/job.rb
+- lib/bigwig/pinger.rb
+- lib/bigwig/plugin.rb
+- lib/bigwig/plugins.rb
+- lib/bigwig/runner.rb
+- bigwig.gemspec
+has_rdoc: true
+homepage: http://www.brightbox.co.uk/
+licenses: []
+
+post_install_message: Welcome to bigwig.  Please set up a configuration file and a plugins folder before starting bigwig...
+rdoc_options: 
+- --line-numbers
+- --inline-source
+- --title
+- Bigwig
+- --main
+- README.markdown
+require_paths: 
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "1.2"
+  version: 
+requirements: []
+
+rubyforge_project: bigwig
+rubygems_version: 1.3.5
+signing_key: 
+specification_version: 3
+summary: A daemon that listens to an AMQP queue and responds to messages by invoking commands from a set of plugins
+test_files: []
+